{"thread":{"id":"1193","subject":"[PATCH 1/2] Management of packs not yet installed","startedAt":"2005-07-10T19:53:18Z","lastAt":"2005-07-12T13:35:54Z","messageCount":15,"participants":["Daniel Barkalow","Junio C Hamano","Linus Torvalds","Darrin Thompson"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"5906","messageId":"Pine.LNX.4.21.0507101539220.30848-100000@iabervon.org","threadId":"1193","inReplyTo":null,"subject":"[PATCH 0/2] Support for packs in HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-10T19:53:18Z","receivedAt":"2005-07-10T19:53:18Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This series has one patch which is ready to go in and one that's not\n(although it's a reasonable phony for the current state of the git world).\n\n 1: Several additional functions are needed in the library to support\n progressively getting pack data from some remote location and using it\n to determine what else to get.\n\n 2: git-http-pull can get packs as appropriate by getting all the index\n files first, and then using them to figure out whether the object it's\n looking for is in some pack it could get.\n\nCurrently, there's no sane way to figure out what pack/index files are\navailable from an HTTP server. But there only seems to be one pack file\navailable on an HTTP server at the moment, so this tries to get that\none.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"5904","messageId":"Pine.LNX.4.21.0507101553270.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507101539220.30848-100000@iabervon.org","subject":"[PATCH 1/2] Management of packs not yet installed","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-10T19:55:22Z","receivedAt":"2005-07-10T19:55:22Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"Support for parsing index files without pack files, installing pack\nfiles while running, and checking what pack files are available.\n\nSigned-off-by: Daniel Barkalow \n\n---\ncommit b686d7a0377c24e05dbed0dafe909dda6c3dfb48\ntree ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1\nparent 167a4a3308f4a1606e268c2204c98d6999046ae0\nauthor Daniel Barkalow 1121024924 -0400\ncommitter Daniel Barkalow 1121024924 -0400\n\nIndex: cache.h\n===================================================================\n--- dbae854c7c91182c8a124d0b85d802945d1c6223/cache.h (mode:100644 sha1:84d43d366c6145a30865aa65d92ada88ab95bb9f)\n+++ ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1/cache.h (mode:100644 sha1:719a77dfabb24e58abd21b7f3a4b846a114e000a)\n@@ -161,6 +161,8 @@\n extern char *mkpath(const char *fmt, ...);\n extern char *git_path(const char *fmt, ...);\n extern char *sha1_file_name(const unsigned char *sha1);\n+extern char *sha1_pack_name(const unsigned char *sha1);\n+extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n int safe_create_leading_directories(char *path);\n \n@@ -189,6 +191,9 @@\n extern int has_sha1_pack(const unsigned char *sha1);\n extern int has_sha1_file(const unsigned char *sha1);\n \n+extern int has_pack_file(const unsigned char *sha1);\n+extern int has_pack_index(const unsigned char *sha1);\n+\n /* Convert to/from hex/sha1 representation */\n extern int get_sha1(const char *str, unsigned char *sha1);\n extern int get_sha1_hex(const char *hex, unsigned char *sha1);\n@@ -260,6 +265,7 @@\n \tvoid *pack_base;\n \tunsigned int pack_last_used;\n \tunsigned int pack_use_cnt;\n+\tunsigned char sha1[20];\n \tchar pack_name[0]; /* something like \".git/objects/pack/xxxxx.pack\" */\n } *packed_git;\n \n@@ -274,7 +280,14 @@\n extern int path_match(const char *path, int nr, char **match);\n extern int get_ack(int fd, unsigned char *result_sha1);\n \n+extern struct packed_git *parse_pack_index(unsigned char *sha1);\n+\n extern void prepare_packed_git(void);\n+extern void install_packed_git(struct packed_git *pack);\n+\n+struct packed_git *find_sha1_pack(const unsigned char *sha1, \n+\t\t\t\t struct packed_git *packs);\n+\n extern int use_packed_git(struct packed_git *);\n extern void unuse_packed_git(struct packed_git *);\n extern struct packed_git *add_packed_git(char *, int);\nIndex: sha1_file.c\n===================================================================\n--- dbae854c7c91182c8a124d0b85d802945d1c6223/sha1_file.c (mode:100644 sha1:b2914dd2ea629ae974fd4b4c272e77cb04e5c0e0)\n+++ ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1/sha1_file.c (mode:100644 sha1:27136fdba0fbf2dd943f2634cb49660cdbf95ec4)\n@@ -200,6 +200,56 @@\n \treturn base;\n }\n \n+char *sha1_pack_name(const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tstatic char *name, *base, *buf;\n+\tint i;\n+\n+\tif (!base) {\n+\t\tconst char *sha1_file_directory = get_object_directory();\n+\t\tint len = strlen(sha1_file_directory);\n+\t\tbase = xmalloc(len + 60);\n+\t\tsprintf(base, \"%s/pack/pack-1234567890123456789012345678901234567890.pack\", sha1_file_directory);\n+\t\tname = base + len + 11;\n+\t}\n+\n+\tbuf = name;\n+\n+\tfor (i = 0; i < 20; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t\n+\treturn base;\n+}\n+\n+char *sha1_pack_index_name(const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tstatic char *name, *base, *buf;\n+\tint i;\n+\n+\tif (!base) {\n+\t\tconst char *sha1_file_directory = get_object_directory();\n+\t\tint len = strlen(sha1_file_directory);\n+\t\tbase = xmalloc(len + 60);\n+\t\tsprintf(base, \"%s/pack/pack-1234567890123456789012345678901234567890.idx\", sha1_file_directory);\n+\t\tname = base + len + 11;\n+\t}\n+\n+\tbuf = name;\n+\n+\tfor (i = 0; i < 20; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t\n+\treturn base;\n+}\n+\n struct alternate_object_database *alt_odb;\n \n /*\n@@ -360,6 +410,14 @@\n \n int use_packed_git(struct packed_git *p)\n {\n+\tif (!p->pack_size) {\n+\t\tstruct stat st;\n+\t\t// We created the struct before we had the pack\n+\t\tstat(p->pack_name, &st);\n+\t\tif (!S_ISREG(st.st_mode))\n+\t\t\tdie(\"packfile %s not a regular file\", p->pack_name);\n+\t\tp->pack_size = st.st_size;\n+\t}\n \tif (!p->pack_base) {\n \t\tint fd;\n \t\tstruct stat st;\n@@ -387,8 +445,10 @@\n \t\t * this is cheap.\n \t\t */\n \t\tif (memcmp((char*)(p->index_base) + p->index_size - 40,\n-\t\t\t p->pack_base + p->pack_size - 20, 20))\n+\t\t\t p->pack_base + p->pack_size - 20, 20)) {\n+\t\t\t \n \t\t\tdie(\"packfile %s does not match index.\", p->pack_name);\n+\t\t}\n \t}\n \tp->pack_last_used = pack_used_ctr++;\n \tp->pack_use_cnt++;\n@@ -426,6 +486,37 @@\n \treturn p;\n }\n \n+struct packed_git *parse_pack_index(unsigned char *sha1)\n+{\n+\tstruct packed_git *p;\n+\tunsigned long idx_size;\n+\tvoid *idx_map;\n+\tchar *path = sha1_pack_index_name(sha1);\n+\n+\tif (check_packed_git_idx(path, &idx_size, &idx_map))\n+\t\treturn NULL;\n+\n+\tpath = sha1_pack_name(sha1);\n+\n+\tp = xmalloc(sizeof(*p) + strlen(path) + 2);\n+\tstrcpy(p->pack_name, path);\n+\tp->index_size = idx_size;\n+\tp->pack_size = 0;\n+\tp->index_base = idx_map;\n+\tp->next = NULL;\n+\tp->pack_base = NULL;\n+\tp->pack_last_used = 0;\n+\tp->pack_use_cnt = 0;\n+\tmemcpy(p->sha1, sha1, 20);\n+\treturn p;\n+}\n+\n+void install_packed_git(struct packed_git *pack)\n+{\n+\tpack->next = packed_git;\n+\tpacked_git = pack;\n+}\n+\n static void prepare_packed_git_one(char *objdir)\n {\n \tchar path[PATH_MAX];\n@@ -999,6 +1090,20 @@\n \treturn 0;\n }\n \n+struct packed_git *find_sha1_pack(const unsigned char *sha1, \n+\t\t\t\t struct packed_git *packs)\n+{\n+\tstruct packed_git *p;\n+\tstruct pack_entry e;\n+\n+\tfor (p = packs; p; p = p->next) {\n+\t\tif (find_pack_entry_one(sha1, &e, p))\n+\t\t\treturn p;\n+\t}\n+\treturn NULL;\n+\t\n+}\n+\n int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep)\n {\n \tint status;\n@@ -1283,6 +1388,22 @@\n \treturn 0;\n }\n \n+int has_pack_index(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_index_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n+int has_pack_file(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n int has_sha1_pack(const unsigned char *sha1)\n {\n \tstruct pack_entry e;\n"},{"id":"5908","messageId":"Pine.LNX.4.21.0507101555271.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507101539220.30848-100000@iabervon.org","subject":"[PATCH 2/2] Demo support for packs via HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-10T19:56:07Z","receivedAt":"2005-07-10T19:56:07Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"Support for downloading the pack file\n\"e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135\" when appropriate. (Will\nsupport other pack files when the repository has a list of them.)\n\nSigned-off-by: Daniel Barkalow \n\n---\ncommit 74132562a2f6cfce9690a5091de7e85bd51d88af\ntree c0ae9cb936abac4412aa4a89928f4609d111fd2c\nparent b686d7a0377c24e05dbed0dafe909dda6c3dfb48\nauthor Daniel Barkalow 1121024943 -0400\ncommitter Daniel Barkalow 1121024943 -0400\n\nIndex: http-pull.c\n===================================================================\n--- ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1/http-pull.c (mode:100644 sha1:1f9d60b9b1d5eed85b24d96c240666bbfc5a22ed)\n+++ c0ae9cb936abac4412aa4a89928f4609d111fd2c/http-pull.c (mode:100644 sha1:2a8d7e71d9447483668cb4a1eb01a096e736f8e3)\n@@ -56,6 +56,126 @@\n \treturn size;\n }\n \n+static int got_indices = 0;\n+\n+static struct packed_git *packs = NULL;\n+\n+static int fetch_index(unsigned char *sha1)\n+{\n+\tchar *filename;\n+\tchar *url;\n+\n+\tFILE *indexfile;\n+\n+\tif (has_pack_index(sha1))\n+\t\treturn 0;\n+\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting index for pack %s\\n\",\n+\t\t\tsha1_to_hex(sha1));\n+\t\n+\turl = xmalloc(strlen(base) + 64);\n+\tsprintf(url, \"%s/objects/pack/pack-%s.idx\",\n+\t\tbase, sha1_to_hex(sha1));\n+\t\n+\tfilename = sha1_pack_index_name(sha1);\n+\tindexfile = fopen(filename, \"w\");\n+\tif (!indexfile)\n+\t\treturn error(\"Unable to open local file %s for pack index\",\n+\t\t\t filename);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\t\n+\tif (curl_easy_perform(curl)) {\n+\t\tfclose(indexfile);\n+\t\treturn error(\"Unable to get pack index %s\", url);\n+\t}\n+\n+\tfclose(indexfile);\n+\treturn 0;\n+}\n+\n+static int setup_index(unsigned char *sha1)\n+{\n+\tstruct packed_git *new_pack;\n+\tif (has_pack_file(sha1))\n+\t\treturn 0; // don't list this as something we can get\n+\n+\tif (fetch_index(sha1))\n+\t\treturn -1;\n+\n+\tnew_pack = parse_pack_index(sha1);\n+\tnew_pack->next = packs;\n+\tpacks = new_pack;\n+\treturn 0;\n+}\n+\n+static int fetch_indices(void)\n+{\n+\tunsigned char sha1[20];\n+\tif (got_indices)\n+\t\treturn 0;\n+\tget_sha1_hex(\"e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135\", sha1);\n+\tsetup_index(sha1);\n+\tgot_indices = 1;\n+\treturn 0;\n+}\n+\n+static int fetch_pack(unsigned char *sha1)\n+{\n+\tchar *url;\n+\tstruct packed_git *target;\n+\tstruct packed_git **lst;\n+\tFILE *packfile;\n+\tchar *filename;\n+\n+\tif (fetch_indices())\n+\t\treturn -1;\n+\ttarget = find_sha1_pack(sha1, packs);\n+\tif (!target)\n+\t\treturn error(\"Couldn't get %s: not separate or in any pack\",\n+\t\t\t sha1_to_hex(sha1));\n+\n+\tif (get_verbosely) {\n+\t\tfprintf(stderr, \"Getting pack %s\\n\",\n+\t\t\tsha1_to_hex(target->sha1));\n+\t\tfprintf(stderr, \" which contains %s\\n\",\n+\t\t\tsha1_to_hex(sha1));\n+\t}\n+\n+\turl = xmalloc(strlen(base) + 65);\n+\tsprintf(url, \"%s/objects/pack/pack-%s.pack\",\n+\t\tbase, sha1_to_hex(target->sha1));\n+\n+\tfilename = sha1_pack_name(target->sha1);\n+\tpackfile = fopen(filename, \"w\");\n+\tif (!packfile)\n+\t\treturn error(\"Unable to open local file %s for pack\",\n+\t\t\t filename);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, packfile);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\t\n+\tif (curl_easy_perform(curl)) {\n+\t\tfclose(packfile);\n+\t\treturn error(\"Unable to get pack file %s\", url);\n+\t}\n+\n+\tfclose(packfile);\n+\n+\tinstall_packed_git(target);\n+\n+\tlst = &packs;\n+\twhile (*lst != target)\n+\t\tlst = &((*lst)->next);\n+\t*lst = (*lst)->next;\n+\n+\treturn 0;\n+}\n+\n int fetch(unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n@@ -67,7 +187,7 @@\n \tlocal = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);\n \n \tif (local < 0)\n-\t\treturn error(\"Couldn't open %s\\n\", filename);\n+\t\treturn error(\"Couldn't open local object %s\\n\", filename);\n \n \tmemset(&stream, 0, sizeof(stream));\n \n@@ -75,6 +195,7 @@\n \n \tSHA1_Init(&c);\n \n+\tcurl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);\n \tcurl_easy_setopt(curl, CURLOPT_FILE, NULL);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \n@@ -90,8 +211,12 @@\n \n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n \n-\tif (curl_easy_perform(curl))\n-\t\treturn error(\"Couldn't get %s for %s\\n\", url, hex);\n+\tif (curl_easy_perform(curl)) {\n+\t\tunlink(filename);\n+\t\tif (fetch_pack(sha1))\n+\t\t\treturn error(\"Tried %s\", url);\n+\t\treturn 0;\n+\t}\n \n \tclose(local);\n \tinflateEnd(&stream);\n"},{"id":"5920","messageId":"7vzmsus1eu.fsf@assigned-by-dhcp.cox.net","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507101539220.30848-100000@iabervon.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-10T23:10:17Z","receivedAt":"2005-07-10T23:10:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow writes:\n\n> This series has one patch which is ready to go in and one that's not\n> (although it's a reasonable phony for the current state of the git world).\n\nI like the general direction in which this patch is leading us.\n\nBut before going further, I'd like to see a consensus on the\npack naming convention. The \"sha1 of packed object names\" was\noriginally introduced to easily avoid the pack name collisions,\nbut not enforced, so a user could do the following and still\nexpect things to work:\n\n $ n=`git-pack-objects pk \n> So I would suggest either:\n> \n> - droping the packname parameter from git-pack-objects. Make\n> the packs always named pack-X{40}.pack (or just X{40}.pack);\n\nWell, regardless, we want to be able to specify which directory to write \nthem to. We don't necessarily want to write them to the current working \ndirectory, nor do we want to write them to their eventual destination in \n.git/objects/pack.\n\nIn fact, the main current user (\"git repack\") really wants to write them \nto a temporary file, and one that isn't even called \"pack-xxx\", since it \nends up doing cleanup with \n\n\trm -f .tmp-pack-*\n\nin case a previous re-pack was interrupted (in which case it simply cannor\nknow what the exact name was supposed to be).\n\nSo the \"basename\" ends up being necessary and meaningful regardless. We do \n_not_ want to remove that capability.\n\n> also have verify-pack to verify the name of the packfile,\n> and make sure X{40} part of the name matches what it claims\n> to contain;\n\nNow, that would be fine, but it can't be done. Not the way things are laid \nout. A SHA1 checksum depends on the order the data was checksummed in, and \nwe don't even save that.\n\n> - or drop sha1_pack_name() and let the user name the pack any\n> way he wants.\n\nNo, I do want to use a SHA1, because I want to make sure that you can mix \npacks in a rsync/wget environment where if two files are named the same, \nthey'll have the same contents.\n\nSo we can make verify-pack check that the pack-name matches the style\n\"pack-xxxxx.\" naming convention, but we can't match up the sha1 with\nanything.\n\n\t\tLinus\n"},{"id":"5922","messageId":"Pine.LNX.4.21.0507101959410.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"Pine.LNX.4.58.0507101637390.17536@g5.osdl.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-11T00:20:27Z","receivedAt":"2005-07-11T00:20:27Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 10 Jul 2005, Linus Torvalds wrote:\n\n> On Sun, 10 Jul 2005, Junio C Hamano wrote:\n> >\n> > So I would suggest either:\n> > \n> > - droping the packname parameter from git-pack-objects. Make\n> > the packs always named pack-X{40}.pack (or just X{40}.pack);\n> \n> Well, regardless, we want to be able to specify which directory to write \n> them to. We don't necessarily want to write them to the current working \n> directory, nor do we want to write them to their eventual destination in \n> .git/objects/pack.\n> \n> In fact, the main current user (\"git repack\") really wants to write them \n> to a temporary file, and one that isn't even called \"pack-xxx\", since it \n> ends up doing cleanup with \n> \n> \trm -f .tmp-pack-*\n> \n> in case a previous re-pack was interrupted (in which case it simply cannor\n> know what the exact name was supposed to be).\n> \n> So the \"basename\" ends up being necessary and meaningful regardless. We do \n> _not_ want to remove that capability.\n\nShouldn't we do the same thing we do with object files? I don't see any\ndifference in desired behavior.\n\n> > also have verify-pack to verify the name of the packfile,\n> > and make sure X{40} part of the name matches what it claims\n> > to contain;\n> \n> Now, that would be fine, but it can't be done. Not the way things are laid \n> out. A SHA1 checksum depends on the order the data was checksummed in, and \n> we don't even save that.\n\nWhy not checksum it in a predictable order, either that of the pack file\nor the index? We do care that it's something verifiable, so that people\ncan't cause intentional collisions (for a DoS) just by naming their packs\nafter existing packs that users might not have downloaded yet.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"5923","messageId":"Pine.LNX.4.58.0507101731330.17536@g5.osdl.org","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507101959410.30848-100000@iabervon.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-11T00:34:26Z","receivedAt":"2005-07-11T00:34:26Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 10 Jul 2005, Daniel Barkalow wrote:\n\n> On Sun, 10 Jul 2005, Linus Torvalds wrote:\n> > \n> > Well, regardless, we want to be able to specify which directory to write \n> > them to. We don't necessarily want to write them to the current working \n> > directory, nor do we want to write them to their eventual destination in \n> > .git/objects/pack.\n> > \n> > In fact, the main current user (\"git repack\") really wants to write them \n> > to a temporary file, and one that isn't even called \"pack-xxx\", since it \n> > ends up doing cleanup with \n> > \n> > \trm -f .tmp-pack-*\n> > \n> > in case a previous re-pack was interrupted (in which case it simply cannor\n> > know what the exact name was supposed to be).\n> > \n> > So the \"basename\" ends up being necessary and meaningful regardless. We do \n> > _not_ want to remove that capability.\n> \n> Shouldn't we do the same thing we do with object files? I don't see any\n> difference in desired behavior.\n\nWell, the main difference is that pack-files can be used for many things.\n\nFor example, a web interface for getting a pack-file between two releases: \nsay you knew you had version xyzzy, and you want to get version xyzzy+1, \nyou could do that through webgit some way even with a \"stupid\" interface. \nKay already had some patch to generate pack-files for something.\n\nThe point being that pack-files are _not_ like objects. Pack-files are \nmeant for communication. Having them in .git/objects/pack is just one \nspecial case.\n\n> Why not checksum it in a predictable order, either that of the pack file\n> or the index? We do care that it's something verifiable, so that people\n> can't cause intentional collisions (for a DoS) just by naming their packs\n> after existing packs that users might not have downloaded yet.\n\nWe could sha1-sum the \"sorted by SHA1\" list, I guess.\n\n\t\tLinus\n"},{"id":"5927","messageId":"Pine.LNX.4.21.0507102253270.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"Pine.LNX.4.58.0507101731330.17536@g5.osdl.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-11T03:22:44Z","receivedAt":"2005-07-11T03:22:44Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 10 Jul 2005, Linus Torvalds wrote:\n\n> \n> \n> On Sun, 10 Jul 2005, Daniel Barkalow wrote:\n> \n> > On Sun, 10 Jul 2005, Linus Torvalds wrote:\n> > > \n> > > Well, regardless, we want to be able to specify which directory to write \n> > > them to. We don't necessarily want to write them to the current working \n> > > directory, nor do we want to write them to their eventual destination in \n> > > .git/objects/pack.\n> > > \n> > > In fact, the main current user (\"git repack\") really wants to write them \n> > > to a temporary file, and one that isn't even called \"pack-xxx\", since it \n> > > ends up doing cleanup with \n> > > \n> > > \trm -f .tmp-pack-*\n> > > \n> > > in case a previous re-pack was interrupted (in which case it simply cannor\n> > > know what the exact name was supposed to be).\n> > > \n> > > So the \"basename\" ends up being necessary and meaningful regardless. We do \n> > > _not_ want to remove that capability.\n> > \n> > Shouldn't we do the same thing we do with object files? I don't see any\n> > difference in desired behavior.\n> \n> Well, the main difference is that pack-files can be used for many things.\n> \n> For example, a web interface for getting a pack-file between two releases: \n> say you knew you had version xyzzy, and you want to get version xyzzy+1, \n> you could do that through webgit some way even with a \"stupid\" interface. \n> Kay already had some patch to generate pack-files for something.\n> \n> The point being that pack-files are _not_ like objects. Pack-files are \n> meant for communication. Having them in .git/objects/pack is just one \n> special case.\n\nOkay, I can see the use for them getting written to arbitrary paths; but I\nthink that it's worth having a canonical location for a pack that's being\nused by the system (either not having been sent anywhere, or after having\nbeen received). Perhaps git-pack-objects should have the base as a\noptional argument, with a default of the filename in $GIT_DIR/objects/pack\nand an option for sending just the pack file to stdout? I think that\ncovers everything in order of usefulness, and means that the program deals\nwith any filename that the user doesn't know in advance.\n\n> > Why not checksum it in a predictable order, either that of the pack file\n> > or the index? We do care that it's something verifiable, so that people\n> > can't cause intentional collisions (for a DoS) just by naming their packs\n> > after existing packs that users might not have downloaded yet.\n> \n> We could sha1-sum the \"sorted by SHA1\" list, I guess.\n\nThat'd be good; then git-http-pull can validate the hash on the index and\nbe sure that a matching pack file from a different location still has the\nsame contents.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"5928","messageId":"Pine.LNX.4.58.0507102034460.17536@g5.osdl.org","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507102253270.30848-100000@iabervon.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-11T03:37:12Z","receivedAt":"2005-07-11T03:37:12Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 10 Jul 2005, Daniel Barkalow wrote:\n> \n> Perhaps git-pack-objects should have the base as a optional argument,\n> with a default of the filename in $GIT_DIR/objects/pack and an option\n> for sending just the pack file to stdout?\n\nYou really _mustn't_ try to create the pack directly to the\n$GIT_DIR/objects/pack subdirectory - that would make git itself start\npossibly using that pack before the index is all done, and that would be\njust wrong and nasty.\n\nSo you really should _always_ generate the pack somewhere else, and then \nmove it (pack file first, index file second).\n\nWhich is, btw, exactly what \"git repack\" does, so the solution to the \nproblem is to just never use git-pack-objects directly if you don't like \nthe semantics..\n\n\t\t\tLinus\n"},{"id":"5935","messageId":"Pine.LNX.4.21.0507102337160.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"Pine.LNX.4.58.0507102034460.17536@g5.osdl.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-11T04:07:10Z","receivedAt":"2005-07-11T04:07:10Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 10 Jul 2005, Linus Torvalds wrote:\n\n> On Sun, 10 Jul 2005, Daniel Barkalow wrote:\n> > \n> > Perhaps git-pack-objects should have the base as a optional argument,\n> > with a default of the filename in $GIT_DIR/objects/pack and an option\n> > for sending just the pack file to stdout?\n> \n> You really _mustn't_ try to create the pack directly to the\n> $GIT_DIR/objects/pack subdirectory - that would make git itself start\n> possibly using that pack before the index is all done, and that would be\n> just wrong and nasty.\n>\n> So you really should _always_ generate the pack somewhere else, and then \n> move it (pack file first, index file second).\n\nIt's currently fine ignoring index files without corresponding\npack files (sha1_file.c, line 470). Do you want to make the constraint\nthat the pack/ directory doesn't have index files for packs that aren't\nalso there? (I've been putting the index files for packs that might be\npossibile to get there, and relying on the above check to make sure that\nthey don't affect anything if it hasn't fetched the pack.)\n\nOf course, we should never write to files in locations that anything looks\nat; we want everything to appear atomically, completely written and\nverified. But there's nothing wrong with having the C code place the\nobjects, which is certainly going to be necessary in the case of\ndownloading them by HTTP, since the program will want to place them and\nenable them while running.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"5953","messageId":"Pine.LNX.4.58.0507111040251.17536@g5.osdl.org","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507102337160.30848-100000@iabervon.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-11T17:44:33Z","receivedAt":"2005-07-11T17:44:33Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 11 Jul 2005, Daniel Barkalow wrote:\n> On Sun, 10 Jul 2005, Linus Torvalds wrote:\n> \n> > \n> > You really _mustn't_ try to create the pack directly to the\n> > $GIT_DIR/objects/pack subdirectory - that would make git itself start\n> > possibly using that pack before the index is all done, and that would be\n> > just wrong and nasty.\n> >\n> > So you really should _always_ generate the pack somewhere else, and then \n> > move it (pack file first, index file second).\n> \n> It's currently fine ignoring index files without corresponding\n> pack files (sha1_file.c, line 470).\n\nThat doesn't help.\n\nRedgardless of which order you write them (and you _will_ write the \npack-file first), you'll find that at some point you have both files, but \none or the other isn't fully written, ie they are unusable.\n\nAnd yes, you can handle that by always checking the SHA1 of the files when \nyou open them, but the fact is, you shouldn't need to, just to use it. \nChecking the SHA1 of the pack-file in particular is very expensive (since \nit's potentially a huge file, and you don't even want to read all of it).\n\nSo you could have rules like: pack-file must get populated first, and \nindex file must be SHA1-checked by all users.\n\nOr, you could just have a saner rule: create the pack-files somewhere \nelse, and move them atomically to the right place. Problem solved.\n\nSo that's what I decided the rule is: never ever have a partial file, and \nthus you can by definition use them immediately when you see both files.\n\nBut that requires that you write them under another name than the final \none. And since you want that _anyway_ for other uses, you don't hide that \ninside \"git-pack-objects\", but you make it an exported interface.\n\n\t\tLinus\n"},{"id":"5984","messageId":"Pine.LNX.4.21.0507111519380.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"Pine.LNX.4.58.0507111040251.17536@g5.osdl.org","subject":"Re: [PATCH 0/2] Support for packs in HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-11T20:03:56Z","receivedAt":"2005-07-11T20:03:56Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 11 Jul 2005, Linus Torvalds wrote:\n\n> \n> \n> On Mon, 11 Jul 2005, Daniel Barkalow wrote:\n> > On Sun, 10 Jul 2005, Linus Torvalds wrote:\n> > \n> > > \n> > > You really _mustn't_ try to create the pack directly to the\n> > > $GIT_DIR/objects/pack subdirectory - that would make git itself start\n> > > possibly using that pack before the index is all done, and that would be\n> > > just wrong and nasty.\n> > >\n> > > So you really should _always_ generate the pack somewhere else, and then \n> > > move it (pack file first, index file second).\n> > \n> > It's currently fine ignoring index files without corresponding\n> > pack files (sha1_file.c, line 470).\n> \n> That doesn't help.\n\nWell, it means that the order you move them doesn't matter, because it\nwill ignore the pair if either hasn't been moved.\n\n> Redgardless of which order you write them (and you _will_ write the \n> pack-file first), you'll find that at some point you have both files, but \n> one or the other isn't fully written, ie they are unusable.\n\n(Off topic: note that git-http-pull writes the _index_ first, because it\nfetches it to determine if it should fetch the pack)\n\n> And yes, you can handle that by always checking the SHA1 of the files when \n> you open them, but the fact is, you shouldn't need to, just to use it. \n> Checking the SHA1 of the pack-file in particular is very expensive (since \n> it's potentially a huge file, and you don't even want to read all of it).\n\nIIRC, we check the size of the pack file and there are hashes around the\nends of the two files which have to match; but this is a die() check, not\nan ignore check, so we just crash with a clear error message rather than\ndoing crazy stuff (like reading from beyond the end of the mmap).\n\n> So that's what I decided the rule is: never ever have a partial file, and \n> thus you can by definition use them immediately when you see both files.\n> \n> But that requires that you write them under another name than the final \n> one. And since you want that _anyway_ for other uses, you don't hide that \n> inside \"git-pack-objects\", but you make it an exported interface.\n\nWe should never write anything under the final name, anyway, for just this\nreason; we already use open/write/close/rename for objects, refs, and\ncache (maybe not working directory files, though). I think we're actually\nagreeing on this.\n\nMy position is that the temporary location should be something like\n{final-name}.part, such that it doesn't match *.idx or *.pack beforehand\n(so it doesn't look like a complete file that you might want to send to\nsomeone) and it doesn't have to worry about EXDEV on the rename. Also, I\nwould ideally like to be able to resume an interrupted download, which\nmeans that it would have to find the partial file in a predictable\nlocation, given what it's supposed to contain.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"5968","messageId":"1121118544.3939.6.camel@localhost.localdomain","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507101555271.30848-100000@iabervon.org","subject":"Re: [PATCH 2/2] Demo support for packs via HTTP","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-07-11T21:49:04Z","receivedAt":"2005-07-11T21:49:04Z","isPatch":true,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"On Sun, 2005-07-10 at 15:56 -0400, Daniel Barkalow wrote:\n> +\tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n> +\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n> +\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n\nI was hoping to send in a patch which would turn on user auth and turn\noff ssl peer verification.\n\nYour (preliminary obviously) patch puts curl handling in two places. Is\nthere a place were I can safely start working on adding the needed\nsetopts?\n\n--\nDarrin\n"},{"id":"5998","messageId":"Pine.LNX.4.21.0507112147400.30848-100000@iabervon.org","threadId":"1193","inReplyTo":"1121118544.3939.6.camel@localhost.localdomain","subject":"Re: [PATCH 2/2] Demo support for packs via HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-12T01:54:00Z","receivedAt":"2005-07-12T01:54:00Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 11 Jul 2005, Darrin Thompson wrote:\n\n> On Sun, 2005-07-10 at 15:56 -0400, Daniel Barkalow wrote:\n> > +\tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n> > +\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n> > +\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n> \n> I was hoping to send in a patch which would turn on user auth and turn\n> off ssl peer verification.\n> \n> Your (preliminary obviously) patch puts curl handling in two places. Is\n> there a place were I can safely start working on adding the needed\n> setopts?\n\nIf I understand the curl documentation, you should be able to set options \non the curl object when it has just been created, if those options aren't\ngoing to change between requests. Note that I make requests from multiple\nplaces, but I use the same curl object for all of them.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"6034","messageId":"42D3C73A.9010305@progeny.com","threadId":"1193","inReplyTo":"Pine.LNX.4.21.0507112147400.30848-100000@iabervon.org","subject":"Re: [PATCH 2/2] Demo support for packs via HTTP","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-07-12T13:35:54Z","receivedAt":"2005-07-12T13:35:54Z","isPatch":true,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"Daniel Barkalow wrote:\n> If I understand the curl documentation, you should be able to set options \n> on the curl object when it has just been created, if those options aren't\n> going to change between requests. Note that I make requests from multiple\n> places, but I use the same curl object for all of them.\n> \n\nI didn't realize that it was the same object. Great!\n\n--\nDarrin\n"}]}