{"thread":{"id":"2038","subject":"Create object subdirectories on demand","startedAt":"2005-10-06T23:23:48Z","lastAt":"2005-10-09T10:42:28Z","messageCount":6,"participants":["Linus Torvalds","Junio C Hamano","Daniel Barkalow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"9782","messageId":"Pine.LNX.4.64.0510061612080.31407@g5.osdl.org","threadId":"2038","inReplyTo":null,"subject":"Create object subdirectories on demand","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-06T23:23:48Z","receivedAt":"2005-10-06T23:23:48Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nThis makes it possible to have a \"sparse\" git object subdirectory \nstructure, something that has become much more attractive now that people \nuse pack-files all the time.\n\nAs a result of pack-files, a git object directory doesn't necessarily have \nany individual objects lying around, and in that case it's just wasting \nspace to keep the empty first-level object directories around: on many \nfilesystems the 256 empty directories will be aboue 1MB of diskspace.\n\nEven more importantly, after you re-pack a project that _used_ to be \nunpacked, you could be left with huge directories that no longer contain \nanything, but that waste space and take time to look through.\n\nWith this change, \"git prune-packed\" can just do an rmdir() on the \ndirectories, and they'll get removed if empty, and re-created on demand.\n\nThis patch also tries to fix up \"write_sha1_from_fd()\" to use the new \ncommon infrastructure for creating the object files, closing a hole where \nwe might otherwise leave half-written objects in the object database.\n\nSigned-off-by: Linus Torvalds <torvalds@osdl.org>\n---\n\n\nThis has _not_ gotten a lot of testing, but I verified that basic things \nseem to work, and that packing an archive properly removes the unnecessary \nsubdirectories.\n\nI'd suggest it sit in \"pu\" for a while.\n\nThe primary reason for this was that a big git archive (the historic one) \nends up with lots of objects, and the object subdirectories end up being \nabout 150kB in size. Even repacking the archive did nothing for it - you \nstill ended up with 256 of these monster directories, never mind that they \nwere empty. Walking through them in the cold-cache case was actually \nreally painful - it would walk through about 40MB of disk very very \ninefficiently, for no actual gain.\n\nHowever, somebody should really check my code carefully before merging \nthis. In particular, I didn't test \"git-ssh-pull\" at all, so I'm not sure \nI actually fixed the \"write temp-file\" thing properly.\n\n\ndiff --git a/daemon.c b/daemon.c\n--- a/daemon.c\n+++ b/daemon.c\n@@ -142,7 +142,7 @@ static int upload(char *dir, int dirlen)\n \t * is ok with us doing this.\n \t */\n \tif ((!export_all_trees && access(\"git-daemon-export-ok\", F_OK)) ||\n-\t    access(\"objects/00\", X_OK) ||\n+\t    access(\"objects/\", X_OK) ||\n \t    access(\"HEAD\", R_OK)) {\n \t\tlogerror(\"Not a valid git-daemon-enabled repository: '%s'\", dir);\n \t\treturn -1;\ndiff --git a/fsck-objects.c b/fsck-objects.c\n--- a/fsck-objects.c\n+++ b/fsck-objects.c\n@@ -329,9 +329,8 @@ static int fsck_dir(int i, char *path)\n \tDIR *dir = opendir(path);\n \tstruct dirent *de;\n \n-\tif (!dir) {\n-\t\treturn error(\"missing sha1 directory '%s'\", path);\n-\t}\n+\tif (!dir)\n+\t\treturn 0;\n \n \twhile ((de = readdir(dir)) != NULL) {\n \t\tchar name[100];\ndiff --git a/git-rename.perl b/git-rename.perl\n--- a/git-rename.perl\n+++ b/git-rename.perl\n@@ -15,7 +15,7 @@ sub usage($);\n my $GIT_DIR = $ENV{'GIT_DIR'} || \".git\";\n \n unless ( -d $GIT_DIR && -d $GIT_DIR . \"/objects\" && \n-\t-d $GIT_DIR . \"/objects/00\" && -d $GIT_DIR . \"/refs\") {\n+\t-d $GIT_DIR . \"/objects/\" && -d $GIT_DIR . \"/refs\") {\n \tusage(\"Git repository not found.\");\n }\n \ndiff --git a/git-sh-setup.sh b/git-sh-setup.sh\n--- a/git-sh-setup.sh\n+++ b/git-sh-setup.sh\n@@ -22,4 +22,4 @@ refs/*)\t: ;;\n *)\tfalse ;;\n esac &&\n [ -d \"$GIT_DIR/refs\" ] &&\n-[ -d \"$GIT_OBJECT_DIRECTORY/00\" ]\n+[ -d \"$GIT_OBJECT_DIRECTORY/\" ]\ndiff --git a/init-db.c b/init-db.c\n--- a/init-db.c\n+++ b/init-db.c\n@@ -244,10 +244,6 @@ int main(int argc, char **argv)\n \tmemcpy(path, sha1_dir, len);\n \n \tsafe_create_dir(sha1_dir);\n-\tfor (i = 0; i < 256; i++) {\n-\t\tsprintf(path+len, \"/%02x\", i);\n-\t\tsafe_create_dir(path);\n-\t}\n \tstrcpy(path+len, \"/pack\");\n \tsafe_create_dir(path);\n \tstrcpy(path+len, \"/info\");\ndiff --git a/prune-packed.c b/prune-packed.c\n--- a/prune-packed.c\n+++ b/prune-packed.c\n@@ -26,6 +26,8 @@ static void prune_dir(int i, DIR *dir, c\n \t\telse if (unlink(pathname) < 0)\n \t\t\terror(\"unable to unlink %s\", pathname);\n \t}\n+\tpathname[len] = 0;\n+\trmdir(pathname);\n }\n \n static void prune_packed_objects(void)\n@@ -46,7 +48,7 @@ static void prune_packed_objects(void)\n \t\tsprintf(pathname + len, \"%02x/\", i);\n \t\td = opendir(pathname);\n \t\tif (!d)\n-\t\t\tdie(\"unable to open %s\", pathname);\n+\t\t\tcontinue;\n \t\tprune_dir(i, d, pathname, len + 3);\n \t\tclosedir(d);\n \t}\ndiff --git a/sha1_file.c b/sha1_file.c\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1248,6 +1248,73 @@ char *write_sha1_file_prepare(void *buf,\n \treturn sha1_file_name(sha1);\n }\n \n+/*\n+ * Link the tempfile to the final place, possibly creating the\n+ * last directory level as you do so.\n+ *\n+ * Returns the errno on failure, 0 on success.\n+ */\n+static int link_temp_to_file(const char *tmpfile, char *filename)\n+{\n+\tint ret;\n+\n+\tif (!link(tmpfile, filename))\n+\t\treturn 0;\n+\n+\t/*\n+\t * Try to mkdir the last path component if that failed\n+\t * with an ENOENT.\n+\t *\n+\t * Re-try the \"link()\" regardless of whether the mkdir\n+\t * succeeds, since a race might mean that somebody\n+\t * else succeeded.\n+\t */\n+\tret = errno;\n+\tif (ret == ENOENT) {\n+\t\tchar *dir = strrchr(filename, '/');\n+\t\tif (dir) {\n+\t\t\t*dir = 0;\n+\t\t\tmkdir(filename, 0777);\n+\t\t\t*dir = '/';\n+\t\t\tif (!link(tmpfile, filename))\n+\t\t\t\treturn 0;\n+\t\t\tret = errno;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n+/*\n+ * Move the just written object into its final resting place\n+ */\n+static int move_temp_to_file(const char *tmpfile, char *filename)\n+{\n+\tint ret = link_temp_to_file(tmpfile, filename);\n+\tif (ret) {\n+\t\t/*\n+\t\t * Coda hack - coda doesn't like cross-directory links,\n+\t\t * so we fall back to a rename, which will mean that it\n+\t\t * won't be able to check collisions, but that's not a\n+\t\t * big deal.\n+\t\t *\n+\t\t * When this succeeds, we just return 0. We have nothing\n+\t\t * left to unlink.\n+\t\t */\n+\t\tif (ret == EXDEV && !rename(tmpfile, filename))\n+\t\t\treturn 0;\n+\t}\n+\tunlink(tmpfile);\n+\tif (ret) {\n+\t\tif (ret != EEXIST) {\n+\t\t\tfprintf(stderr, \"unable to write sha1 filename %s: %s\", filename, strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\t/* FIXME!!! Collision check here ? */\n+\t}\n+\n+\treturn 0;\n+}\n+\n int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n {\n \tint size;\n@@ -1257,7 +1324,7 @@ int write_sha1_file(void *buf, unsigned \n \tchar *filename;\n \tstatic char tmpfile[PATH_MAX];\n \tunsigned char hdr[50];\n-\tint fd, hdrlen, ret;\n+\tint fd, hdrlen;\n \n \t/* Normally if we have it in the pack then we do not bother writing\n \t * it out into .git/objects/??/?{38} file.\n@@ -1320,32 +1387,7 @@ int write_sha1_file(void *buf, unsigned \n \tclose(fd);\n \tfree(compressed);\n \n-\tret = link(tmpfile, filename);\n-\tif (ret < 0) {\n-\t\tret = errno;\n-\n-\t\t/*\n-\t\t * Coda hack - coda doesn't like cross-directory links,\n-\t\t * so we fall back to a rename, which will mean that it\n-\t\t * won't be able to check collisions, but that's not a\n-\t\t * big deal.\n-\t\t *\n-\t\t * When this succeeds, we just return 0. We have nothing\n-\t\t * left to unlink.\n-\t\t */\n-\t\tif (ret == EXDEV && !rename(tmpfile, filename))\n-\t\t\treturn 0;\n-\t}\n-\tunlink(tmpfile);\n-\tif (ret) {\n-\t\tif (ret != EEXIST) {\n-\t\t\tfprintf(stderr, \"unable to write sha1 filename %s: %s\", filename, strerror(ret));\n-\t\t\treturn -1;\n-\t\t}\n-\t\t/* FIXME!!! Collision check here ? */\n-\t}\n-\n-\treturn 0;\n+\treturn move_temp_to_file(tmpfile, filename);\n }\n \n int write_sha1_to_fd(int fd, const unsigned char *sha1)\n@@ -1420,8 +1462,7 @@ int write_sha1_to_fd(int fd, const unsig\n int write_sha1_from_fd(const unsigned char *sha1, int fd, char *buffer,\n \t\t       size_t bufsize, size_t *bufposn)\n {\n-\tchar *filename = sha1_file_name(sha1);\n-\n+\tchar tmpfile[PATH_MAX];\n \tint local;\n \tz_stream stream;\n \tunsigned char real_sha1[20];\n@@ -1429,10 +1470,11 @@ int write_sha1_from_fd(const unsigned ch\n \tint ret;\n \tSHA_CTX c;\n \n-\tlocal = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\tsnprintf(tmpfile, sizeof(tmpfile), \"%s/obj_XXXXXX\", get_object_directory());\n \n+\tlocal = mkstemp(tmpfile);\n \tif (local < 0)\n-\t\treturn error(\"Couldn't open %s\\n\", filename);\n+\t\treturn error(\"Couldn't open %s for %s\\n\", tmpfile, sha1_to_hex(sha1));\n \n \tmemset(&stream, 0, sizeof(stream));\n \n@@ -1462,7 +1504,7 @@ int write_sha1_from_fd(const unsigned ch\n \t\tsize = read(fd, buffer + *bufposn, bufsize - *bufposn);\n \t\tif (size <= 0) {\n \t\t\tclose(local);\n-\t\t\tunlink(filename);\n+\t\t\tunlink(tmpfile);\n \t\t\tif (!size)\n \t\t\t\treturn error(\"Connection closed?\");\n \t\t\tperror(\"Reading from connection\");\n@@ -1475,15 +1517,15 @@ int write_sha1_from_fd(const unsigned ch\n \tclose(local);\n \tSHA1_Final(real_sha1, &c);\n \tif (ret != Z_STREAM_END) {\n-\t\tunlink(filename);\n+\t\tunlink(tmpfile);\n \t\treturn error(\"File %s corrupted\", sha1_to_hex(sha1));\n \t}\n \tif (memcmp(sha1, real_sha1, 20)) {\n-\t\tunlink(filename);\n+\t\tunlink(tmpfile);\n \t\treturn error(\"File %s has bad hash\\n\", sha1_to_hex(sha1));\n \t}\n-\t\n-\treturn 0;\n+\n+\treturn move_temp_to_file(tmpfile, sha1_file_name(sha1));\n }\n \n int has_pack_index(const unsigned char *sha1)\n"},{"id":"9792","messageId":"7vfyrd7jll.fsf@assigned-by-dhcp.cox.net","threadId":"2038","inReplyTo":"Pine.LNX.4.64.0510061612080.31407@g5.osdl.org","subject":"Re: Create object subdirectories on demand","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T09:38:14Z","receivedAt":"2005-10-07T09:38:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@osdl.org> writes:\n\n> This has _not_ gotten a lot of testing, but I verified that basic things \n> seem to work, and that packing an archive properly removes the unnecessary \n> subdirectories.\n>\n> I'd suggest it sit in \"pu\" for a while.\n\nI typically use the tip of \"pu\" myself, which resulted in an\ninteresting case (nothing grave).  After running 'git prune', I\nswitched to another topic branch that did not include this\npatch, built it, and tried it out -- and got complaint because\nthe earlier 'git prune' happened to remove \".git/objects/00\".\nNow I was not in a valid git repository anymore ;-).\n\nThis needs to wait until everybody's git get this update,\nespecially, I cannot use this version in my $HOME/bin/ on\nkernel.org right now.\n\nBut then once everybody else updates, the repository their git\ncreates cannot be read by my git -- an interesting chicken and\negg problem.\n\nMaybe successful rmdir() immediately followed by mkdir(), if we\nare willing to waste 4KB or so per empty directory, trading\nspace for safety and ease of transition?  That is, phase I tools\ndo not complain if objects/00 is missing, lazily creates\nobject/??/ is missing, and makes sure empty directories are\npruned but still recreates them for safety.  Then phase II tools\nthen stops the recreating part.\n\n> However, somebody should really check my code carefully before merging \n> this. In particular, I didn't test \"git-ssh-pull\" at all, so I'm not sure \n> I actually fixed the \"write temp-file\" thing properly.\n\nWell, the thing is, I do not use commit walkers over ssh myself\nand honestly consider them having outlived their usefulness.\nBut you are right -- I should look at this one again before\nplacing in the master.\n"},{"id":"9795","messageId":"Pine.LNX.4.64.0510070745370.31407@g5.osdl.org","threadId":"2038","inReplyTo":"7vfyrd7jll.fsf@assigned-by-dhcp.cox.net","subject":"Re: Create object subdirectories on demand","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-07T14:56:29Z","receivedAt":"2005-10-07T14:56:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 7 Oct 2005, Junio C Hamano wrote:\n> \n> But then once everybody else updates, the repository their git\n> creates cannot be read by my git -- an interesting chicken and\n> egg problem.\n\nWell, you can do it in two (or three) phases:\n\n (a) remove the tests for \"GIT_DIR/objects/00\" from the \"is this a \n     git dir\" tests and allow git-fsck-objects to skip missing \n     directories.\n\nThis makes read-only operations work fine in sparse directories\n\n (b) apply my sha1_file.c part of the patch that knows how to do the mkdir \n     and also fixes (untested!) the write_sha1_from_fd() problem.\n\nThis makes all operations work fine in sparse object directories\n\n (c) remove the \"for(i = 0; i < 256; i++) mkdir(i)\" thing from git-init-db \n     and add the \"rmdir()\" to git-prune-packed.\n\nThis is the part that actually generates the sparse object directories.\n\nSo if you apply (a+b) to mainline, and wait with (c) until a later date, \nat least we can then do (c) at any point. No hurry.\n\nAppended is a suggested split, if you want it explicitly. I haven't \nactually tested this split and did it by hand, but it looks correct.\n\n\t\t\tLinus\n\n---- stage (a) ----\n\nPhase 1: don't require the object subdirectories to be there if empty\n\ndiff --git a/daemon.c b/daemon.c\n--- a/daemon.c\n+++ b/daemon.c\n@@ -142,7 +142,7 @@ static int upload(char *dir, int dirlen)\n \t * is ok with us doing this.\n \t */\n \tif ((!export_all_trees && access(\"git-daemon-export-ok\", F_OK)) ||\n-\t    access(\"objects/00\", X_OK) ||\n+\t    access(\"objects/\", X_OK) ||\n \t    access(\"HEAD\", R_OK)) {\n \t\tlogerror(\"Not a valid git-daemon-enabled repository: '%s'\", dir);\n \t\treturn -1;\ndiff --git a/fsck-objects.c b/fsck-objects.c\n--- a/fsck-objects.c\n+++ b/fsck-objects.c\n@@ -329,9 +329,8 @@ static int fsck_dir(int i, char *path)\n \tDIR *dir = opendir(path);\n \tstruct dirent *de;\n \n-\tif (!dir) {\n-\t\treturn error(\"missing sha1 directory '%s'\", path);\n-\t}\n+\tif (!dir)\n+\t\treturn 0;\n \n \twhile ((de = readdir(dir)) != NULL) {\n \t\tchar name[100];\ndiff --git a/git-rename.perl b/git-rename.perl\n--- a/git-rename.perl\n+++ b/git-rename.perl\n@@ -15,7 +15,7 @@ sub usage($);\n my $GIT_DIR = $ENV{'GIT_DIR'} || \".git\";\n \n unless ( -d $GIT_DIR && -d $GIT_DIR . \"/objects\" && \n-\t-d $GIT_DIR . \"/objects/00\" && -d $GIT_DIR . \"/refs\") {\n+\t-d $GIT_DIR . \"/objects/\" && -d $GIT_DIR . \"/refs\") {\n \tusage(\"Git repository not found.\");\n }\n \ndiff --git a/git-sh-setup.sh b/git-sh-setup.sh\n--- a/git-sh-setup.sh\n+++ b/git-sh-setup.sh\n@@ -22,4 +22,4 @@ refs/*)\t: ;;\n *)\tfalse ;;\n esac &&\n [ -d \"$GIT_DIR/refs\" ] &&\n-[ -d \"$GIT_OBJECT_DIRECTORY/00\" ]\n+[ -d \"$GIT_OBJECT_DIRECTORY/\" ]\ndiff --git a/prune-packed.c b/prune-packed.c\n--- a/prune-packed.c\n+++ b/prune-packed.c\n@@ -46,7 +48,7 @@ static void prune_packed_objects(void)\n \t\tsprintf(pathname + len, \"%02x/\", i);\n \t\td = opendir(pathname);\n \t\tif (!d)\n-\t\t\tdie(\"unable to open %s\", pathname);\n+\t\t\tcontinue;\n \t\tprune_dir(i, d, pathname, len + 3);\n \t\tclosedir(d);\n \t}\n\n---- stage (b) ----\n\nPhase 2: create object subdirectories on demand\n\ndiff --git a/sha1_file.c b/sha1_file.c\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1248,6 +1248,73 @@ char *write_sha1_file_prepare(void *buf,\n \treturn sha1_file_name(sha1);\n }\n \n+/*\n+ * Link the tempfile to the final place, possibly creating the\n+ * last directory level as you do so.\n+ *\n+ * Returns the errno on failure, 0 on success.\n+ */\n+static int link_temp_to_file(const char *tmpfile, char *filename)\n+{\n+\tint ret;\n+\n+\tif (!link(tmpfile, filename))\n+\t\treturn 0;\n+\n+\t/*\n+\t * Try to mkdir the last path component if that failed\n+\t * with an ENOENT.\n+\t *\n+\t * Re-try the \"link()\" regardless of whether the mkdir\n+\t * succeeds, since a race might mean that somebody\n+\t * else succeeded.\n+\t */\n+\tret = errno;\n+\tif (ret == ENOENT) {\n+\t\tchar *dir = strrchr(filename, '/');\n+\t\tif (dir) {\n+\t\t\t*dir = 0;\n+\t\t\tmkdir(filename, 0777);\n+\t\t\t*dir = '/';\n+\t\t\tif (!link(tmpfile, filename))\n+\t\t\t\treturn 0;\n+\t\t\tret = errno;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n+/*\n+ * Move the just written object into its final resting place\n+ */\n+static int move_temp_to_file(const char *tmpfile, char *filename)\n+{\n+\tint ret = link_temp_to_file(tmpfile, filename);\n+\tif (ret) {\n+\t\t/*\n+\t\t * Coda hack - coda doesn't like cross-directory links,\n+\t\t * so we fall back to a rename, which will mean that it\n+\t\t * won't be able to check collisions, but that's not a\n+\t\t * big deal.\n+\t\t *\n+\t\t * When this succeeds, we just return 0. We have nothing\n+\t\t * left to unlink.\n+\t\t */\n+\t\tif (ret == EXDEV && !rename(tmpfile, filename))\n+\t\t\treturn 0;\n+\t}\n+\tunlink(tmpfile);\n+\tif (ret) {\n+\t\tif (ret != EEXIST) {\n+\t\t\tfprintf(stderr, \"unable to write sha1 filename %s: %s\", filename, strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\t/* FIXME!!! Collision check here ? */\n+\t}\n+\n+\treturn 0;\n+}\n+\n int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n {\n \tint size;\n@@ -1257,7 +1324,7 @@ int write_sha1_file(void *buf, unsigned \n \tchar *filename;\n \tstatic char tmpfile[PATH_MAX];\n \tunsigned char hdr[50];\n-\tint fd, hdrlen, ret;\n+\tint fd, hdrlen;\n \n \t/* Normally if we have it in the pack then we do not bother writing\n \t * it out into .git/objects/??/?{38} file.\n@@ -1320,32 +1387,7 @@ int write_sha1_file(void *buf, unsigned \n \tclose(fd);\n \tfree(compressed);\n \n-\tret = link(tmpfile, filename);\n-\tif (ret < 0) {\n-\t\tret = errno;\n-\n-\t\t/*\n-\t\t * Coda hack - coda doesn't like cross-directory links,\n-\t\t * so we fall back to a rename, which will mean that it\n-\t\t * won't be able to check collisions, but that's not a\n-\t\t * big deal.\n-\t\t *\n-\t\t * When this succeeds, we just return 0. We have nothing\n-\t\t * left to unlink.\n-\t\t */\n-\t\tif (ret == EXDEV && !rename(tmpfile, filename))\n-\t\t\treturn 0;\n-\t}\n-\tunlink(tmpfile);\n-\tif (ret) {\n-\t\tif (ret != EEXIST) {\n-\t\t\tfprintf(stderr, \"unable to write sha1 filename %s: %s\", filename, strerror(ret));\n-\t\t\treturn -1;\n-\t\t}\n-\t\t/* FIXME!!! Collision check here ? */\n-\t}\n-\n-\treturn 0;\n+\treturn move_temp_to_file(tmpfile, filename);\n }\n \n int write_sha1_to_fd(int fd, const unsigned char *sha1)\n@@ -1420,8 +1462,7 @@ int write_sha1_to_fd(int fd, const unsig\n int write_sha1_from_fd(const unsigned char *sha1, int fd, char *buffer,\n \t\t       size_t bufsize, size_t *bufposn)\n {\n-\tchar *filename = sha1_file_name(sha1);\n-\n+\tchar tmpfile[PATH_MAX];\n \tint local;\n \tz_stream stream;\n \tunsigned char real_sha1[20];\n@@ -1429,10 +1470,11 @@ int write_sha1_from_fd(const unsigned ch\n \tint ret;\n \tSHA_CTX c;\n \n-\tlocal = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\tsnprintf(tmpfile, sizeof(tmpfile), \"%s/obj_XXXXXX\", get_object_directory());\n \n+\tlocal = mkstemp(tmpfile);\n \tif (local < 0)\n-\t\treturn error(\"Couldn't open %s\\n\", filename);\n+\t\treturn error(\"Couldn't open %s for %s\\n\", tmpfile, sha1_to_hex(sha1));\n \n \tmemset(&stream, 0, sizeof(stream));\n \n@@ -1462,7 +1504,7 @@ int write_sha1_from_fd(const unsigned ch\n \t\tsize = read(fd, buffer + *bufposn, bufsize - *bufposn);\n \t\tif (size <= 0) {\n \t\t\tclose(local);\n-\t\t\tunlink(filename);\n+\t\t\tunlink(tmpfile);\n \t\t\tif (!size)\n \t\t\t\treturn error(\"Connection closed?\");\n \t\t\tperror(\"Reading from connection\");\n@@ -1475,15 +1517,15 @@ int write_sha1_from_fd(const unsigned ch\n \tclose(local);\n \tSHA1_Final(real_sha1, &c);\n \tif (ret != Z_STREAM_END) {\n-\t\tunlink(filename);\n+\t\tunlink(tmpfile);\n \t\treturn error(\"File %s corrupted\", sha1_to_hex(sha1));\n \t}\n \tif (memcmp(sha1, real_sha1, 20)) {\n-\t\tunlink(filename);\n+\t\tunlink(tmpfile);\n \t\treturn error(\"File %s has bad hash\\n\", sha1_to_hex(sha1));\n \t}\n-\t\n-\treturn 0;\n+\n+\treturn move_temp_to_file(tmpfile, sha1_file_name(sha1));\n }\n \n int has_pack_index(const unsigned char *sha1)\n\n---- stage (c) ----\n\nPhase 3: actually remove object subdirectories\n\ndiff --git a/init-db.c b/init-db.c\n--- a/init-db.c\n+++ b/init-db.c\n@@ -244,10 +244,6 @@ int main(int argc, char **argv)\n \tmemcpy(path, sha1_dir, len);\n \n \tsafe_create_dir(sha1_dir);\n-\tfor (i = 0; i < 256; i++) {\n-\t\tsprintf(path+len, \"/%02x\", i);\n-\t\tsafe_create_dir(path);\n-\t}\n \tstrcpy(path+len, \"/pack\");\n \tsafe_create_dir(path);\n \tstrcpy(path+len, \"/info\");\ndiff --git a/prune-packed.c b/prune-packed.c\n--- a/prune-packed.c\n+++ b/prune-packed.c\n@@ -26,6 +26,8 @@ static void prune_dir(int i, DIR *dir, c\n \t\telse if (unlink(pathname) < 0)\n \t\t\terror(\"unable to unlink %s\", pathname);\n \t}\n+\tpathname[len] = 0;\n+\trmdir(pathname);\n }\n \n static void prune_packed_objects(void)\n"},{"id":"9799","messageId":"7vll155lxj.fsf@assigned-by-dhcp.cox.net","threadId":"2038","inReplyTo":"7vfyrd7jll.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] Create object subdirectories on demand (phase II)","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T16:30:48Z","receivedAt":"2005-10-07T16:30:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This removes the unoptimization.  The previous round does not mind\nmissing fan-out directories, but still makes sure they exist, lest\nolder versions choke on a repository created/packed by it.\n\nThis round does not play that nicely anymore -- empty fan-out\ndirectories are not created by init-db, and will stay removed by\nprune-packed.  The prune command also removes empty fan-out directories.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n    Here is what I did.  I unoptimized your original and that\n    sits near the tip of \"pu\" branch.  This patch is to revert\n    the unoptimization.  In addition, git-prune is told about\n    the empty fan-out removal, although it may not matter that\n    much.  Hopefully, the phase I can be included in the next\n    official release, and after everybody updates, this patch\n    can go in.\n\n git-prune.sh     |    1 +\n init-db.c        |    4 ----\n prune-packed.c   |    3 +--\n t/t0000-basic.sh |    8 ++++----\n 4 files changed, 6 insertions(+), 10 deletions(-)\n\napplies-to: d672cb30d513553ea3ca92a933563a25df1d7865\n93638ce0a569543d0b41eedfac6873a541f5c753\ndiff --git a/git-prune.sh b/git-prune.sh\nindex 9657dbf..b28630c 100755\n--- a/git-prune.sh\n+++ b/git-prune.sh\n@@ -22,6 +22,7 @@ sed -ne '/unreachable /{\n }' | {\n \tcd \"$GIT_OBJECT_DIRECTORY\" || exit\n \txargs $echo rm -f\n+\trmdir 2>/dev/null [0-9a-f][0-9a-f]\n }\n \n git-prune-packed $dryrun\ndiff --git a/init-db.c b/init-db.c\nindex aabc09f..921df9b 100644\n--- a/init-db.c\n+++ b/init-db.c\n@@ -244,10 +244,6 @@ int main(int argc, char **argv)\n \tmemcpy(path, sha1_dir, len);\n \n \tsafe_create_dir(sha1_dir);\n-\tfor (i = 0; i < 256; i++) {\n-\t\tsprintf(path+len, \"/%02x\", i);\n-\t\tsafe_create_dir(path);\n-\t}\n \tstrcpy(path+len, \"/pack\");\n \tsafe_create_dir(path);\n \tstrcpy(path+len, \"/info\");\ndiff --git a/prune-packed.c b/prune-packed.c\nindex 73f0f3a..16685d1 100644\n--- a/prune-packed.c\n+++ b/prune-packed.c\n@@ -27,8 +27,7 @@ static void prune_dir(int i, DIR *dir, c\n \t\t\terror(\"unable to unlink %s\", pathname);\n \t}\n \tpathname[len] = 0;\n-\tif (rmdir(pathname))\n-\t\tmkdir(pathname, 0777);\n+\trmdir(pathname);\n }\n \n static void prune_packed_objects(void)\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 5c5f854..dff7d69 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -28,12 +28,12 @@ test_expect_success \\\n     '.git/objects should be empty after git-init-db in an empty repo.' \\\n     'cmp -s /dev/null should-be-empty' \n \n-# also it should have 258 subdirectories; 256 fan-out anymore, pack, and info.\n-# 259 is counting \"objects\" itself\n+# also it should have 2 subdirectories; no fan-out anymore, pack, and info.\n+# 3 is counting \"objects\" itself\n find .git/objects -type d -print >full-of-directories\n test_expect_success \\\n-    '.git/objects should have 258 subdirectories.' \\\n-    'test $(wc -l < full-of-directories) = 259'\n+    '.git/objects should have 3 subdirectories.' \\\n+    'test $(wc -l < full-of-directories) = 3'\n \n ################################################################\n # Basics of the basics\n---\n0.99.8.GIT\n"},{"id":"9827","messageId":"Pine.LNX.4.63.0510072124290.23242@iabervon.org","threadId":"2038","inReplyTo":"Pine.LNX.4.64.0510061612080.31407@g5.osdl.org","subject":"Re: Create object subdirectories on demand","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-08T01:45:32Z","receivedAt":"2005-10-08T01:45:32Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Thu, 6 Oct 2005, Linus Torvalds wrote:\n\n> This patch also tries to fix up \"write_sha1_from_fd()\" to use the new \n> common infrastructure for creating the object files, closing a hole where \n> we might otherwise leave half-written objects in the object database.\n\nThis looks right to me, but it would be nice to also split out and \nshare the temp file creation. Also, http-fetch.c writes object files and \nneeds at least move_temp_to_file() if it's going to do special stuff.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9855","messageId":"7vd5mfezu3.fsf@assigned-by-dhcp.cox.net","threadId":"2038","inReplyTo":"Pine.LNX.4.63.0510072124290.23242@iabervon.org","subject":"Use the same move_temp_to_file in git-http-fetch.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-09T10:42:28Z","receivedAt":"2005-10-09T10:42:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The http commit walker cannot use the same temporary file\ncreation code because it needs to use predictable temporary\nfilename for partial fetch continuation purposes, but the code\nto move the temporary file to the final location should be\nusable from the ordinary object creation case.\n\nExport move_temp_to_file from sha1_file.c and use it, while\nlosing the custom relink_or_rename function from http-fetch.c.\n\nAlso the temporary object file creation part needs to make sure\nthe leading path exists, in preparation of the really lazy\nfan-out directory creation.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n    Daniel Barkalow <barkalow@iabervon.org> writes:\n\n    > This looks right to me, but it would be nice to also split out and \n    > share the temp file creation. Also, http-fetch.c writes object files and \n    > needs at least move_temp_to_file() if it's going to do special stuff.\n\n    I agree.  The proposed updates branch has this patch.\n\n cache.h      |    1 +\n http-fetch.c |   47 ++++++++++++++++++-----------------------------\n sha1_file.c  |    2 +-\n 3 files changed, 20 insertions(+), 30 deletions(-)\n\napplies-to: f3741b89b850db8d343d3cde58416e076964c952\n07d374bffd6fcd0c67c671902dde9402335776f5\ndiff --git a/cache.h b/cache.h\nindex 5987d4c..28077fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -221,6 +221,7 @@ extern int read_tree(void *buffer, unsig\n extern int write_sha1_from_fd(const unsigned char *sha1, int fd, char *buffer,\n \t\t\t      size_t bufsize, size_t *bufposn);\n extern int write_sha1_to_fd(int fd, const unsigned char *sha1);\n+extern int move_temp_to_file(const char *tmpfile, char *filename);\n \n extern int has_sha1_pack(const unsigned char *sha1);\n extern int has_sha1_file(const unsigned char *sha1);\ndiff --git a/http-fetch.c b/http-fetch.c\nindex 5d0e3e3..c617583 100644\n--- a/http-fetch.c\n+++ b/http-fetch.c\n@@ -138,25 +138,6 @@ static size_t fwrite_sha1_file(void *ptr\n \treturn size;\n }\n \n-int relink_or_rename(char *old, char *new) {\n-\tint ret;\n-\n-\tret = link(old, new);\n-\tif (ret < 0) {\n-\t\t/* Same Coda hack as in write_sha1_file(sha1_file.c) */\n-\t\tret = errno;\n-\t\tif (ret == EXDEV && !rename(old, new))\n-\t\t\treturn 0;\n-\t}\n-\tunlink(old);\n-\tif (ret) {\n-\t\tif (ret != EEXIST)\n-\t\t\treturn ret;\n-\t}\n-\n-\treturn 0;\n-}\n-\n #ifdef USE_CURL_MULTI\n void process_curl_messages();\n void process_request_queue();\n@@ -295,6 +276,20 @@ void start_request(struct transfer_reque\n \n \trequest->local = open(request->tmpfile,\n \t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t/* This could have failed due to the \"lazy directory creation\";\n+\t * try to mkdir the last path component.\n+\t */\n+\tif (request->local < 0 && errno == ENOENT) {\n+\t\tchar *dir = strrchr(request->tmpfile, '/');\n+\t\tif (dir) {\n+\t\t\t*dir = 0;\n+\t\t\tmkdir(request->tmpfile, 0777);\n+\t\t\t*dir = '/';\n+\t\t}\n+\t\trequest->local = open(request->tmpfile,\n+\t\t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t}\n+\n \tif (request->local < 0) {\n \t\trequest->state = ABORTED;\n \t\terror(\"Couldn't create temporary file %s for %s: %s\\n\",\n@@ -408,7 +403,7 @@ void finish_request(struct transfer_requ\n \t\treturn;\n \t}\n \trequest->rename =\n-\t\trelink_or_rename(request->tmpfile, request->filename);\n+\t\tmove_temp_to_file(request->tmpfile, request->filename);\n \n \tif (request->rename == 0)\n \t\tpull_say(\"got %s\\n\", sha1_to_hex(request->sha1));\n@@ -599,12 +594,7 @@ static int fetch_index(struct alt_base *\n \n \tfclose(indexfile);\n \n-\tret = relink_or_rename(tmpfile, filename);\n-\tif (ret)\n-\t\treturn error(\"unable to write index filename %s: %s\",\n-\t\t\t     filename, strerror(ret));\n-\n-\treturn 0;\n+\treturn move_temp_to_file(tmpfile, filename);\n }\n \n static int setup_index(struct alt_base *repo, unsigned char *sha1)\n@@ -869,10 +859,9 @@ static int fetch_pack(struct alt_base *r\n \n \tfclose(packfile);\n \n-\tret = relink_or_rename(tmpfile, filename);\n+\tret = move_temp_to_file(tmpfile, filename);\n \tif (ret)\n-\t\treturn error(\"unable to write pack filename %s: %s\",\n-\t\t\t     filename, strerror(ret));\n+\t\treturn ret;\n \n \tlst = &repo->packs;\n \twhile (*lst != target)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex baaa4c0..6e3ea23 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1287,7 +1287,7 @@ static int link_temp_to_file(const char \n /*\n  * Move the just written object into its final resting place\n  */\n-static int move_temp_to_file(const char *tmpfile, char *filename)\n+int move_temp_to_file(const char *tmpfile, char *filename)\n {\n \tint ret = link_temp_to_file(tmpfile, filename);\n \tif (ret) {\n---\n0.99.8.GIT\n"}]}