{"thread":{"id":"23827","subject":"Checking out on a different+partial directory","startedAt":"2010-05-17T10:53:55Z","lastAt":"2010-05-27T05:10:12Z","messageCount":6,"participants":["Eli Barzilay","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"141803","messageId":"19441.8259.634019.348195@winooski.ccs.neu.edu","threadId":"23827","inReplyTo":null,"subject":"Checking out on a different+partial directory","fromName":"Eli Barzilay","fromEmail":"eli@barzilay.org","sentAt":"2010-05-17T10:53:55Z","receivedAt":"2010-05-17T10:53:55Z","isPatch":false,"sender":{"key":"eli@barzilay.org","avatar":"https://avatars.githubusercontent.com/u/185905?v=4"},"body":"Say that I have a repository at /some/dir, and a tree that contains\nmost of its files at /another/dir (the second is a build directory,\nmade with `git archive', so some files removed due to export-ignore\nattributes and some built files are present).\n\nIs there a convenient way to make /some/dir usable as a repository?\nTwo things that I tried are\n\n  git --work-tree=/another/dir reset --hard master\n\nwhich one time, but then failed with \"fatal: unable to read tree...\",\nand another is\n\n  cp -a /some/dir/.git /another/dir\n  cd /another/dir\n  git reset --hard master\n\nwhich looks like it can suffer from the same problem.\n\n(It would be especially nice if there's a way to have only different\nfiles touched in /another/dir.)\n\n-- \n          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:\n                    http://barzilay.org/                   Maze is Life!\n"},{"id":"142081","messageId":"20100522054852.GB29118@coredump.intra.peff.net","threadId":"23827","inReplyTo":"19441.8259.634019.348195@winooski.ccs.neu.edu","subject":"Re: Checking out on a different+partial directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-22T05:48:52Z","receivedAt":"2010-05-22T05:48:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 17, 2010 at 06:53:55AM -0400, Eli Barzilay wrote:\n\n> Say that I have a repository at /some/dir, and a tree that contains\n> most of its files at /another/dir (the second is a build directory,\n> made with `git archive', so some files removed due to export-ignore\n> attributes and some built files are present).\n> \n> Is there a convenient way to make /some/dir usable as a repository?\n\nDid you mean /another/dir? /some/dir is already a repository as per your\ndescription above (is it a bare repository or a regular one?).\n\n> Two things that I tried are\n> \n>   git --work-tree=/another/dir reset --hard master\n> \n> which one time, but then failed with \"fatal: unable to read tree...\",\n> and another is\n\nI would have thought that worked, assuming you were in /some/dir. And\noddly, _some_ stuff works. I tried:\n\n  mkdir repo && cd repo && git init\n  echo content >file && git add . && git commit -m file\n  git archive --format=tar --prefix=work/ HEAD | tar -C .. -xf -\n\nto create the situation. Running\n\n  cd repo\n  git --work-tree=$PWD/../work diff-files\n  git --work-tree=$PWD/../work diff\n\nworks as expected (the first one notices stat-dirtiness, and the second\nshows an empty diff). But then I get:\n\n  $ git --work-tree=$PWD/../work reset --hard\n  fatal: unable to read tree ea394696ee1f7f5b55ca0d97d37748933cf39095\n\nSo there is clearly a bug. I'll investigate.\n\n>   cp -a /some/dir/.git /another/dir\n>   cd /another/dir\n>   git reset --hard master\n> \n> which looks like it can suffer from the same problem.\n\nThat should work, too.\n\n> (It would be especially nice if there's a way to have only different\n> files touched in /another/dir.)\n\nOnly different files will be rewritten, but git will have to read all of\nthe files to determine their sha1 (usually it avoids this by checking\nstat info, but obviously your exported files will not match the stat\ninfo in the /another/dir's index).\n\n-Peff\n"},{"id":"142082","messageId":"20100522064308.GA11335@coredump.intra.peff.net","threadId":"23827","inReplyTo":"20100522054852.GB29118@coredump.intra.peff.net","subject":"[PATCH] fix over-eager caching in sha1_file_name","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-22T06:43:08Z","receivedAt":"2010-05-22T06:43:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 22, 2010 at 01:48:52AM -0400, Jeff King wrote:\n\n> I would have thought that worked, assuming you were in /some/dir. And\n> oddly, _some_ stuff works. I tried:\n> \n>   mkdir repo && cd repo && git init\n>   echo content >file && git add . && git commit -m file\n>   git archive --format=tar --prefix=work/ HEAD | tar -C .. -xf -\n> \n> to create the situation. Running\n> \n>   cd repo\n>   git --work-tree=$PWD/../work diff-files\n>   git --work-tree=$PWD/../work diff\n> \n> works as expected (the first one notices stat-dirtiness, and the second\n> shows an empty diff). But then I get:\n> \n>   $ git --work-tree=$PWD/../work reset --hard\n>   fatal: unable to read tree ea394696ee1f7f5b55ca0d97d37748933cf39095\n> \n> So there is clearly a bug. I'll investigate.\n\nUgh. This code dates back to the first two weeks of git, but didn't\nbecome a bug until sometime in the intervening 5 years. See below for\nthe explanation and fix.\n\n-- >8 --\nSubject: [PATCH] fix over-eager caching in sha1_file_name\n\nThis function takes a sha1 and produces a loose object\nfilename. It caches the location of the object directory so\nthat it can fill the sha1 information directly without\nallocating a new buffer (and in its original incarnation,\nwithout calling getenv(), though these days we cache that\nwith the code in environment.c).\n\nThis cached base directory can become stale, however, if in\na single process git changes the location of the object\ndirectory (e.g., by running setup_work_tree, which will\nchdir to the new worktree).\n\nIn most cases this isn't a problem, because we tend to set\nup the git repository location and do any chdir()s before\nactually looking up any objects, so the first lookup will\ncache the correct location. In the case of reset --hard,\nhowever, we do something like:\n\n  1. look up the commit object\n\n  2. notice we are doing --hard, run setup_work_tree\n\n  3. look up the tree object to reset\n\nStep (3) fails because our cache object directory value is\nbogus.\n\nThis patch takes the minimalist fix. It retains the caching,\nbut checks the validity of our object directory against the\none cached in environment.c, which adds only a single\nfunction call and a pointer comparison to the fast path.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nAs I noted above, this is the minimal fix. I think it would be more\nreadable, though, to simply remove this caching layer altogether and use\na static buffer. I suspect the original was just trying to avoid the\nslow getenv() call, which is no longer an issue now. We can probably\nafford an snprintf. I'll post that patch shortly.\n\n sha1_file.c |    6 ++++--\n 1 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d8e61a6..f047b3c 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -103,10 +103,12 @@ static void fill_sha1_path(char *pathbuf, const unsigned char *sha1)\n char *sha1_file_name(const unsigned char *sha1)\n {\n \tstatic char *name, *base;\n+\tstatic const char *sha1_file_directory;\n \n-\tif (!base) {\n-\t\tconst char *sha1_file_directory = get_object_directory();\n+\tif (!base || sha1_file_directory != get_object_directory()) {\n+\t\tsha1_file_directory = get_object_directory();\n \t\tint len = strlen(sha1_file_directory);\n+\t\tfree(base);\n \t\tbase = xmalloc(len + 60);\n \t\tmemcpy(base, sha1_file_directory, len);\n \t\tmemset(base+len, 0, 60);\n-- \n1.7.1.352.gb63a7.dirty\n"},{"id":"142083","messageId":"20100522065942.GB11335@coredump.intra.peff.net","threadId":"23827","inReplyTo":"20100522064308.GA11335@coredump.intra.peff.net","subject":"[PATCH] remove over-eager caching in sha1_file_name","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-22T06:59:42Z","receivedAt":"2010-05-22T06:59:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 22, 2010 at 02:43:08AM -0400, Jeff King wrote:\n\n> This patch takes the minimalist fix. It retains the caching,\n> but checks the validity of our object directory against the\n> one cached in environment.c, which adds only a single\n> function call and a pointer comparison to the fast path.\n> \n> Signed-off-by: Jeff King <peff@peff.net>\n> ---\n> As I noted above, this is the minimal fix. I think it would be more\n> readable, though, to simply remove this caching layer altogether and use\n> a static buffer. I suspect the original was just trying to avoid the\n> slow getenv() call, which is no longer an issue now. We can probably\n> afford an snprintf. I'll post that patch shortly.\n\nActually, thinking on it more, micro-optimizing this is really\npointless. I was thinking that it would get called a lot, so we need to\ncare. But the next step is almost certainly to open, mmap, and zlib\ndecompress the resulting object, which is way more expensive.\n\nSo here it is with caching ripped out. More readable and more robust,\nand I'm sure we can afford an extra strlen() and memcpy() on each object\nlookup.\n\n-- >8 --\nSubject: [PATCH] remove over-eager caching in sha1_file_name\n\nThis function takes a sha1 and produces a loose object\nfilename. It caches the location of the object directory so\nthat it can fill the sha1 information directly without\nallocating a new buffer (and in its original incarnation,\nwithout calling getenv(), though these days we cache that\nwith the code in environment.c).\n\nThis cached base directory can become stale, however, if in\na single process git changes the location of the object\ndirectory (e.g., by running setup_work_tree, which will\nchdir to the new worktree).\n\nIn most cases this isn't a problem, because we tend to set\nup the git repository location and do any chdir()s before\nactually looking up any objects, so the first lookup will\ncache the correct location. In the case of reset --hard,\nhowever, we do something like:\n\n  1. look up the commit object\n\n  2. notice we are doing --hard, run setup_work_tree\n\n  3. look up the tree object to reset\n\nStep (3) fails because our cache object directory value is\nbogus.\n\nThis patch simply removes the caching. We use a static\nbuffer instead of allocating one each time (the original\nversion treated the malloc'd buffer as a static, so there is\nno change in calling semantics).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n sha1_file.c |   28 +++++++++++++++-------------\n 1 files changed, 15 insertions(+), 13 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d8e61a6..e42ef96 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -102,20 +102,22 @@ static void fill_sha1_path(char *pathbuf, const unsigned char *sha1)\n  */\n char *sha1_file_name(const unsigned char *sha1)\n {\n-\tstatic char *name, *base;\n+\tstatic char buf[PATH_MAX];\n+\tconst char *objdir;\n+\tint len;\n \n-\tif (!base) {\n-\t\tconst char *sha1_file_directory = get_object_directory();\n-\t\tint len = strlen(sha1_file_directory);\n-\t\tbase = xmalloc(len + 60);\n-\t\tmemcpy(base, sha1_file_directory, len);\n-\t\tmemset(base+len, 0, 60);\n-\t\tbase[len] = '/';\n-\t\tbase[len+3] = '/';\n-\t\tname = base + len + 1;\n-\t}\n-\tfill_sha1_path(name, sha1);\n-\treturn base;\n+\tobjdir = get_object_directory();\n+\tlen = strlen(objdir);\n+\n+\t/* '/' + sha1(2) + '/' + sha1(38) + '\\0' */\n+\tif (len + 43 > PATH_MAX)\n+\t\tdie(\"insanely long object directory %s\", objdir);\n+\tmemcpy(buf, objdir, len);\n+\tbuf[len] = '/';\n+\tbuf[len+3] = '/';\n+\tbuf[len+42] = '\\0';\n+\tfill_sha1_path(buf + len + 1, sha1);\n+\treturn buf;\n }\n \n static char *sha1_get_pack_name(const unsigned char *sha1,\n-- \n1.7.1.227.ge187a.dirty\n"},{"id":"142306","messageId":"7vtypvqmim.fsf@alter.siamese.dyndns.org","threadId":"23827","inReplyTo":"20100522065942.GB11335@coredump.intra.peff.net","subject":"Re: [PATCH] remove over-eager caching in sha1_file_name","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-05-26T05:07:13Z","receivedAt":"2010-05-26T05:07:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Actually, thinking on it more, micro-optimizing this is really\n> pointless. I was thinking that it would get called a lot, so we need to\n> care. But the next step is almost certainly to open, mmap, and zlib\n> decompress the resulting object, which is way more expensive.\n>\n> So here it is with caching ripped out. More readable and more robust,\n> and I'm sure we can afford an extra strlen() and memcpy() on each object\n> lookup.\n\nMakes sense; thanks.\n"},{"id":"142398","messageId":"m3d3wikk0b.fsf@winooski.ccs.neu.edu","threadId":"23827","inReplyTo":"20100522054852.GB29118@coredump.intra.peff.net","subject":"Re: Checking out on a different+partial directory","fromName":"Eli Barzilay","fromEmail":"eli@barzilay.org","sentAt":"2010-05-27T05:10:12Z","receivedAt":"2010-05-27T05:10:12Z","isPatch":false,"sender":{"key":"eli@barzilay.org","avatar":"https://avatars.githubusercontent.com/u/185905?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Mon, May 17, 2010 at 06:53:55AM -0400, Eli Barzilay wrote:\n>\n>> Say that I have a repository at /some/dir, and a tree that contains\n>> most of its files at /another/dir (the second is a build directory,\n>> made with `git archive', so some files removed due to export-ignore\n>> attributes and some built files are present).\n>> \n>> Is there a convenient way to make /some/dir usable as a repository?\n>\n> Did you mean /another/dir? /some/dir is already a repository as per your\n> description above\n\nYes, sorry for the confusion...\n\n\n> (is it a bare repository or a regular one?).\n\nIt's a regular one.\n\n\n>> Two things that I tried are\n>> \n>>   git --work-tree=/another/dir reset --hard master\n>> \n>> which one time, but then failed with \"fatal: unable to read tree...\",\n>> and another is\n>\n> I would have thought that worked, assuming you were in /some/dir. And\n> oddly, _some_ stuff works. I tried:\n>\n> [...]\n>\n> So there is clearly a bug. I'll investigate.\n\nThanks!\n\n\n>>   cp -a /some/dir/.git /another/dir\n>>   cd /another/dir\n>>   git reset --hard master\n>> \n>> which looks like it can suffer from the same problem.\n>\n> That should work, too.\n\nHeh, I just assumed that I'm doing something similarly wrong in both\ncases...  In any case, I finally settled on doing things the other\nway: grab all of the files that were created by the built tree and add\nthem to the repository directory (that is -- get stuff from\n/another/dir into /some/dir).  It's a little less convenient for me,\nso I'll probably switch back to the above, given that it should work.\n\n\n>> (It would be especially nice if there's a way to have only\n>> different files touched in /another/dir.)\n>\n> Only different files will be rewritten, but git will have to read all of\n> the files to determine their sha1 (usually it avoids this by checking\n> stat info, but obviously your exported files will not match the stat\n> info in the /another/dir's index).\n\nYes, I expected this cost...\n\n-- \n          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:\n                    http://barzilay.org/                   Maze is Life!\n"}]}