{"thread":{"id":"10237","subject":"git branch performance problem?","startedAt":"2007-10-10T20:22:29Z","lastAt":"2007-10-12T17:19:19Z","messageCount":25,"participants":["Han-Wen Nienhuys","Lars Hjemli","J. Bruce Fields","Johannes Schindelin","Brandon Casey","Linus Torvalds","Alex Riesen","Mike Ralphson","Salikh Zakirov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"55408","messageId":"f329bf540710101322xdea6210x5576779f2efd89b7@mail.gmail.com","threadId":"10237","inReplyTo":null,"subject":"git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-10T20:22:29Z","receivedAt":"2007-10-10T20:22:29Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"Hello,\n\nI'm seeing very slow performance with 'git-branch'.  Is this the\ncanonical way to find out the current branch? ( I know I can look into\n.git/HEAD, but how likely is that to break in the future?)\n\nhanwen@lilypond:/tmp/z$ time git branch\n* foo\n  master\n\nreal    0m0.307s\nuser    0m0.232s\nsys     0m0.038s\n\nhanwen@lilypond:/tmp/z$ git --version\ngit version 1.5.1.rc1.949.g322bc\n\n\nOn NFS this takes 5 seconds. Note that I have a humongous amount of\nremotes, but those should not be examined without -r, right?\n\nhanwen@lilypond:/tmp/z$ find .git/refs/remotes | wc -l\n1856\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55409","messageId":"8c5c35580710101344t3aed4214h4f999072483c4cb5@mail.gmail.com","threadId":"10237","inReplyTo":"f329bf540710101322xdea6210x5576779f2efd89b7@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2007-10-10T20:44:24Z","receivedAt":"2007-10-10T20:44:24Z","isPatch":false,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On 10/10/07, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:\n> I'm seeing very slow performance with 'git-branch'.  Is this the\n> canonical way to find out the current branch?\n\nYou could also try 'git symbolic-ref HEAD', but see below...\n\n> hanwen@lilypond:/tmp/z$ find .git/refs/remotes | wc -l\n> 1856\n\nYou probably want to run 'git gc' (which will run 'git pack-refs',\ni.e. put all files currently under .git/refs into a single file). This\nshould speed up 'git branch' (and quite possibly other commands too).\n\n--\nlarsh\n"},{"id":"55411","messageId":"f329bf540710101417w640b2421v73279cc8e34449b8@mail.gmail.com","threadId":"10237","inReplyTo":"8c5c35580710101344t3aed4214h4f999072483c4cb5@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-10T21:17:06Z","receivedAt":"2007-10-10T21:17:06Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/10, Lars Hjemli <hjemli@gmail.com>:\n> On 10/10/07, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:\n> > I'm seeing very slow performance with 'git-branch'.  Is this the\n> > canonical way to find out the current branch?\n>\n> You could also try 'git symbolic-ref HEAD', but see below...\n>\n> > hanwen@lilypond:/tmp/z$ find .git/refs/remotes | wc -l\n> > 1856\n>\n> You probably want to run 'git gc' (which will run 'git pack-refs',\n> i.e. put all files currently under .git/refs into a single file). This\n> should speed up 'git branch' (and quite possibly other commands too).\n\nThis seems rather unuseful. After running gc pack-refs --all, I lost my HEAD,\n\nhanwen@lilypond:~/vc/git5$ git show HEAD\nfatal: ambiguous argument 'HEAD': unknown revision or path not in the\nworking tree.\nUse '--' to separate paths from revisions\n\nIs there a way to only pack refs under a certain subdirectory of .git/refs ?\n(I'm thinking of .git/refs/remotes )\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55414","messageId":"f329bf540710101424q22309489sada99907e94b2cd0@mail.gmail.com","threadId":"10237","inReplyTo":"f329bf540710101417w640b2421v73279cc8e34449b8@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-10T21:24:57Z","receivedAt":"2007-10-10T21:24:57Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/10, Han-Wen Nienhuys <hanwenn@gmail.com>:\n> > You probably want to run 'git gc' (which will run 'git pack-refs',\n> > i.e. put all files currently under .git/refs into a single file). This\n> > should speed up 'git branch' (and quite possibly other commands too).\n>\n> This seems rather unuseful. After running gc pack-refs --all, I lost my HEAD,\n>\n> hanwen@lilypond:~/vc/git5$ git show HEAD\n> fatal: ambiguous argument 'HEAD': unknown revision or path not in the\n> working tree.\n\nMore to the point, I seemed to have lost my entire repository. This is\nthe type of surprise  I don't enjoy.\n\nNow, can someone explain why 'git branch' takes forever if there are\nonly two non-remote branches ?\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55417","messageId":"f329bf540710101430i63926b25q7d55976af96b891d@mail.gmail.com","threadId":"10237","inReplyTo":"f329bf540710101424q22309489sada99907e94b2cd0@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-10T21:30:02Z","receivedAt":"2007-10-10T21:30:02Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/10, Han-Wen Nienhuys <hanwenn@gmail.com>:\n> More to the point, I seemed to have lost my entire repository. This is\n> the type of surprise  I don't enjoy.\n>\n> Now, can someone explain why 'git branch' takes forever if there are\n> only two non-remote branches ?\n\nSo,\n\nHere is a question:  I would like to share commitishes between two checkouts\nof a repository. The reason for this is that I want to easily cherry\npick back and forth between the two. The files of in one of them\nshould be continually available, since I am running out of that\ndirectory.\n\nThe way I solved that, was to have both repositories pointing to each\nother, using alternates.\n\nNow, after a couple of gc and pack-refs iterations, I am greeted by\n\nhanwen@lilypond:~/vc/git6$ git fsck\nmissing tree 12b00ec3190f7b46a5fe0a3235445bead4c9645b\nbroken link from    tree 1718d09e0394d113c162e4a3471e7a1f20914a94\n              to    blob 635e2802568b85017007698c0e6dd4d28dca496f\nbroken link from    tree 926899798fce75038e24f8fa1838f6da8bcf105f\n              to    tree f1b852d270ebbaaf95d8ddc06c52763bad11ff25\nmissing blob 99f0c0d63276fce444e3a200167b636236784c52\nmissing tree f1b852d270ebbaaf95d8ddc06c52763bad11ff25\nmissing blob 236962a87fafae8ca2dce2dc550d344aa7a8884a\nmissing blob 7d69ca297f392a954c4cdcb62bb4c8a90ddb862b\nmissing blob 9e39be8f5cb4eeff97fcfd6eb77fefeda02f0e71\ndangling blob f3a93f023080ce9fc6becb397e366cc4ceb192f5\n\n\ncould it be that GC does not handle cyclic alternates correctly?\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55419","messageId":"8c5c35580710101434n3a4f77edm50d205d53fbc9200@mail.gmail.com","threadId":"10237","inReplyTo":"f329bf540710101424q22309489sada99907e94b2cd0@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2007-10-10T21:34:49Z","receivedAt":"2007-10-10T21:34:49Z","isPatch":false,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On 10/10/07, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:\n> 2007/10/10, Han-Wen Nienhuys <hanwenn@gmail.com>:\n> > > You probably want to run 'git gc' (which will run 'git pack-refs',\n> > > i.e. put all files currently under .git/refs into a single file). This\n> > > should speed up 'git branch' (and quite possibly other commands too).\n> >\n> > This seems rather unuseful. After running gc pack-refs --all, I lost my HEAD,\n> >\n> > hanwen@lilypond:~/vc/git5$ git show HEAD\n> > fatal: ambiguous argument 'HEAD': unknown revision or path not in the\n> > working tree.\n>\n> More to the point, I seemed to have lost my entire repository. This is\n> the type of surprise  I don't enjoy.\n\nYeah, this is bad, I'm sorry to have caused you trouble. But I fail to\nsee how 'git pack-refs --all' could possibly trash your repository. A\nfew questions:\n\nWhat version of git are you using?\nWhat's the output from these commands:\n$ cat .git/packed-refs\n$ cat .git/HEAD\n$ find .git/refs -type f | wc -l\n\n> Now, can someone explain why 'git branch' takes forever if there are\n> only two non-remote branches ?\n\nThat's because git-branch always traverses the complete directory tree\nbelow .git/refs, even if you only want to see the 'local' branches (I\nhave a patch cooking to fix this).\n\n--\nlarsh\n"},{"id":"55421","messageId":"20071010213925.GB2963@fieldses.org","threadId":"10237","inReplyTo":"f329bf540710101430i63926b25q7d55976af96b891d@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2007-10-10T21:39:25Z","receivedAt":"2007-10-10T21:39:25Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Wed, Oct 10, 2007 at 06:30:02PM -0300, Han-Wen Nienhuys wrote:\n> 2007/10/10, Han-Wen Nienhuys <hanwenn@gmail.com>:\n> > More to the point, I seemed to have lost my entire repository. This is\n> > the type of surprise  I don't enjoy.\n> >\n> > Now, can someone explain why 'git branch' takes forever if there are\n> > only two non-remote branches ?\n> \n> So,\n> \n> Here is a question:  I would like to share commitishes between two checkouts\n> of a repository. The reason for this is that I want to easily cherry\n> pick back and forth between the two. The files of in one of them\n> should be continually available, since I am running out of that\n> directory.\n> \n> The way I solved that, was to have both repositories pointing to each\n> other, using alternates.\n> \n> Now, after a couple of gc and pack-refs iterations, I am greeted by\n> \n> hanwen@lilypond:~/vc/git6$ git fsck\n> missing tree 12b00ec3190f7b46a5fe0a3235445bead4c9645b\n> broken link from    tree 1718d09e0394d113c162e4a3471e7a1f20914a94\n>               to    blob 635e2802568b85017007698c0e6dd4d28dca496f\n> broken link from    tree 926899798fce75038e24f8fa1838f6da8bcf105f\n>               to    tree f1b852d270ebbaaf95d8ddc06c52763bad11ff25\n> missing blob 99f0c0d63276fce444e3a200167b636236784c52\n> missing tree f1b852d270ebbaaf95d8ddc06c52763bad11ff25\n> missing blob 236962a87fafae8ca2dce2dc550d344aa7a8884a\n> missing blob 7d69ca297f392a954c4cdcb62bb4c8a90ddb862b\n> missing blob 9e39be8f5cb4eeff97fcfd6eb77fefeda02f0e71\n> dangling blob f3a93f023080ce9fc6becb397e366cc4ceb192f5\n> \n> \n> could it be that GC does not handle cyclic alternates correctly?\n\nDoes it handle alternates at all?  If you run git-gc on a repository\nwhich other repositories get objects from, then my impression was that\nbad things happen.\n\n--b.\n"},{"id":"55422","messageId":"8c5c35580710101445h232f9a67jd0c326b3b97ae3dd@mail.gmail.com","threadId":"10237","inReplyTo":"20071010213925.GB2963@fieldses.org","subject":"Re: git branch performance problem?","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2007-10-10T21:45:46Z","receivedAt":"2007-10-10T21:45:46Z","isPatch":false,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On 10/10/07, J. Bruce Fields <bfields@fieldses.org> wrote:\n> On Wed, Oct 10, 2007 at 06:30:02PM -0300, Han-Wen Nienhuys wrote:\n> > could it be that GC does not handle cyclic alternates correctly?\n>\n> Does it handle alternates at all?  If you run git-gc on a repository\n> which other repositories get objects from, then my impression was that\n> bad things happen.\n>\n\nAFAIK 'git gc' is safe, while 'git gc --prune' will remove loose\n(unreferenced) objects.\n\n-- \nlarsh\n"},{"id":"55423","messageId":"f329bf540710101449oad9c9dg85f3821f55fb85ea@mail.gmail.com","threadId":"10237","inReplyTo":"8c5c35580710101445h232f9a67jd0c326b3b97ae3dd@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-10T21:49:19Z","receivedAt":"2007-10-10T21:49:19Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/10, Lars Hjemli <hjemli@gmail.com>:\n> On 10/10/07, J. Bruce Fields <bfields@fieldses.org> wrote:\n> > On Wed, Oct 10, 2007 at 06:30:02PM -0300, Han-Wen Nienhuys wrote:\n> > > could it be that GC does not handle cyclic alternates correctly?\n> >\n> > Does it handle alternates at all?  If you run git-gc on a repository\n> > which other repositories get objects from, then my impression was that\n> > bad things happen.\n> >\n>\n> AFAIK 'git gc' is safe, while 'git gc --prune' will remove loose\n> (unreferenced) objects.\n\nYes, I think that in this case, gc --prune was run accidentally, but\ngiven that the history of the program invoking git just died, I'm not\nsure how to figure that out.\n\nMaybe gc --prune could follow the alternates and abort if a cycle was detected?\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55424","messageId":"20071010215317.GC2963@fieldses.org","threadId":"10237","inReplyTo":"f329bf540710101449oad9c9dg85f3821f55fb85ea@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2007-10-10T21:53:17Z","receivedAt":"2007-10-10T21:53:17Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Wed, Oct 10, 2007 at 06:49:19PM -0300, Han-Wen Nienhuys wrote:\n> 2007/10/10, Lars Hjemli <hjemli@gmail.com>:\n> > On 10/10/07, J. Bruce Fields <bfields@fieldses.org> wrote:\n> > > On Wed, Oct 10, 2007 at 06:30:02PM -0300, Han-Wen Nienhuys wrote:\n> > > > could it be that GC does not handle cyclic alternates correctly?\n> > >\n> > > Does it handle alternates at all?  If you run git-gc on a repository\n> > > which other repositories get objects from, then my impression was that\n> > > bad things happen.\n> > >\n> >\n> > AFAIK 'git gc' is safe, while 'git gc --prune' will remove loose\n> > (unreferenced) objects.\n> \n> Yes, I think that in this case, gc --prune was run accidentally, but\n> given that the history of the program invoking git just died, I'm not\n> sure how to figure that out.\n> \n> Maybe gc --prune could follow the alternates and abort if a cycle was detected?\n\nDon't the alternates point in the wrong direction?  You'd need pointers\nback from the main repository to the repositories that depend on it for\nobjects.\n\nWhich would be nice....\n\n--b.\n"},{"id":"55425","messageId":"Pine.LNX.4.64.0710102251230.4174@racer.site","threadId":"10237","inReplyTo":"f329bf540710101449oad9c9dg85f3821f55fb85ea@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-10-10T21:53:29Z","receivedAt":"2007-10-10T21:53:29Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 10 Oct 2007, Han-Wen Nienhuys wrote:\n\n> 2007/10/10, Lars Hjemli <hjemli@gmail.com>:\n> > On 10/10/07, J. Bruce Fields <bfields@fieldses.org> wrote:\n> > > On Wed, Oct 10, 2007 at 06:30:02PM -0300, Han-Wen Nienhuys wrote:\n> > > > could it be that GC does not handle cyclic alternates correctly?\n> > >\n> > > Does it handle alternates at all?  If you run git-gc on a repository \n> > > which other repositories get objects from, then my impression was \n> > > that bad things happen.\n> > >\n> >\n> > AFAIK 'git gc' is safe, while 'git gc --prune' will remove loose\n> > (unreferenced) objects.\n> \n> Yes, I think that in this case, gc --prune was run accidentally, but\n> given that the history of the program invoking git just died, I'm not\n> sure how to figure that out.\n> \n> Maybe gc --prune could follow the alternates and abort if a cycle was \n> detected?\n\nI think we talked about this quite some time ago, and the resolution was \nthat it is too hard.\n\nNow that it bit somebody in real life, I think we have to try harder.\n\nAnd probably the best place to check would be git-prune, not git-gc, since \nthat is the program (called by gc) that most probably killed your repo.\n\nCome to think of it, it should probably be part of git-repack, too.\n\nWill try to cobble up a patch,\nDscho\n"},{"id":"55426","messageId":"1192053283-2351-1-git-send-email-hjemli@gmail.com","threadId":"10237","inReplyTo":"f329bf540710101424q22309489sada99907e94b2cd0@mail.gmail.com","subject":"[PATCH] git-branch: only traverse the requested refs","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2007-10-10T21:54:43Z","receivedAt":"2007-10-10T21:54:43Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"This avoids looking at every single file below .git/refs when git-branch\nis fetching the list of refs to display.\n\nSigned-off-by: Lars Hjemli <hjemli@gmail.com>\n---\n\nThis patch should make git-branch much more efficient when there exists\nmany files below .git/refs, but it does require two passes through\n.git/packed-refs when -a is specified.\n\nNo benchmarking performed...\n\n builtin-branch.c |   28 +++++++++-------------------\n 1 files changed, 9 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin-branch.c b/builtin-branch.c\nindex 3da8b55..466e1e0 100644\n--- a/builtin-branch.c\n+++ b/builtin-branch.c\n@@ -185,25 +185,8 @@ static int append_ref(const char *refname, const unsigned char *sha1, int flags,\n {\n \tstruct ref_list *ref_list = (struct ref_list*)(cb_data);\n \tstruct ref_item *newitem;\n-\tint kind = REF_UNKNOWN_TYPE;\n \tint len;\n \n-\t/* Detect kind */\n-\tif (!prefixcmp(refname, \"refs/heads/\")) {\n-\t\tkind = REF_LOCAL_BRANCH;\n-\t\trefname += 11;\n-\t} else if (!prefixcmp(refname, \"refs/remotes/\")) {\n-\t\tkind = REF_REMOTE_BRANCH;\n-\t\trefname += 13;\n-\t} else if (!prefixcmp(refname, \"refs/tags/\")) {\n-\t\tkind = REF_TAG;\n-\t\trefname += 10;\n-\t}\n-\n-\t/* Don't add types the caller doesn't want */\n-\tif ((kind & ref_list->kinds) == 0)\n-\t\treturn 0;\n-\n \t/* Resize buffer */\n \tif (ref_list->index >= ref_list->alloc) {\n \t\tref_list->alloc = alloc_nr(ref_list->alloc);\n@@ -214,7 +197,7 @@ static int append_ref(const char *refname, const unsigned char *sha1, int flags,\n \t/* Record the new item */\n \tnewitem = &(ref_list->list[ref_list->index++]);\n \tnewitem->name = xstrdup(refname);\n-\tnewitem->kind = kind;\n+\tnewitem->kind = ref_list->kinds;\n \thashcpy(newitem->sha1, sha1);\n \tlen = strlen(newitem->name);\n \tif (len > ref_list->maxwidth)\n@@ -296,8 +279,15 @@ static void print_ref_list(int kinds, int detached, int verbose, int abbrev)\n \tstruct ref_list ref_list;\n \n \tmemset(&ref_list, 0, sizeof(ref_list));\n+\tif (kinds & REF_LOCAL_BRANCH) {\n+\t\tref_list.kinds = REF_LOCAL_BRANCH;\n+\t\tfor_each_branch_ref(append_ref, &ref_list);\n+\t}\n+\tif (kinds & REF_REMOTE_BRANCH) {\n+\t\tref_list.kinds = REF_REMOTE_BRANCH;\n+\t\tfor_each_remote_ref(append_ref, &ref_list);\n+\t}\n \tref_list.kinds = kinds;\n-\tfor_each_ref(append_ref, &ref_list);\n \n \tqsort(ref_list.list, ref_list.index, sizeof(struct ref_item), ref_cmp);\n \n-- \n1.5.3.4.206.g58ba4\n"},{"id":"55428","messageId":"f329bf540710101501q7f7297ebqd6365a51775d17d3@mail.gmail.com","threadId":"10237","inReplyTo":"20071010215317.GC2963@fieldses.org","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-10T22:01:48Z","receivedAt":"2007-10-10T22:01:48Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/10, J. Bruce Fields <bfields@fieldses.org>:\n> > Maybe gc --prune could follow the alternates and abort if a cycle was detected?\n>\n> Don't the alternates point in the wrong direction?  You'd need pointers\n> back from the main repository to the repositories that depend on it for\n> objects.\n>\n> Which would be nice....\n\nThe development repo was cloned from the main repo; then sometimes I\ncherry pick from development into the main repo. Hence alternates in 2\ndirections.\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55431","messageId":"470D585D.1030808@nrlssc.navy.mil","threadId":"10237","inReplyTo":"8c5c35580710101445h232f9a67jd0c326b3b97ae3dd@mail.gmail.com","subject":"Re: Spam: Re: git branch performance problem?","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2007-10-10T22:55:25Z","receivedAt":"2007-10-10T22:55:25Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Lars Hjemli wrote:\n> On 10/10/07, J. Bruce Fields <bfields@fieldses.org> wrote:\n>> On Wed, Oct 10, 2007 at 06:30:02PM -0300, Han-Wen Nienhuys wrote:\n>>> could it be that GC does not handle cyclic alternates correctly?\n>> Does it handle alternates at all?  If you run git-gc on a repository\n>> which other repositories get objects from, then my impression was that\n>> bad things happen.\n>>\n> \n> AFAIK 'git gc' is safe, while 'git gc --prune' will remove loose\n> (unreferenced) objects.\n\nNo, this is not the case, unless something has changed very recently\nin git-gc or git-repack. Even git-gc with no arguments is unsafe if\nthe repository being gc'ed is listed in another's alternates.\n\ngit-gc calls repack with -a and -d. which causes a new pack to be\ncreated which only contains the objects required by the local repository.\nThe other packs are then deleted. Objects contained in those packs and\nrequired by a \"sharing\" repository (one using the alternates mechanism)\nwill be deleted if the local repository no longer references them.\n\nMaybe git-gc should make use of repack's new -A option by default and\nonly use -a (and not -A) when --prune is specified...\n\n-brandon\n"},{"id":"55432","messageId":"Pine.LNX.4.64.0710102358110.4174@racer.site","threadId":"10237","inReplyTo":"1192053283-2351-1-git-send-email-hjemli@gmail.com","subject":"Re: [PATCH] git-branch: only traverse the requested refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-10-10T23:00:08Z","receivedAt":"2007-10-10T23:00:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 10 Oct 2007, Lars Hjemli wrote:\n\n> This avoids looking at every single file below .git/refs when git-branch \n> is fetching the list of refs to display.\n> \n> [...]\n>\n> +\tif (kinds & REF_LOCAL_BRANCH) {\n> +\t\tref_list.kinds = REF_LOCAL_BRANCH;\n> +\t\tfor_each_branch_ref(append_ref, &ref_list);\n> +\t}\n\nThe function for_each_branch_ref() calls do_for_each_ref(), which in turn \ncalls get_loose_refs(), which calls get_ref_dir() to read all loose refs, \nif they have not yet been read.\n\nSo I think that your patch (unfortunately) will no help Han-Wen's \nsituation.\n\nCiao,\nDscho\n"},{"id":"55434","messageId":"8c5c35580710101630r36cff79ax6358416dd4851f48@mail.gmail.com","threadId":"10237","inReplyTo":"Pine.LNX.4.64.0710102358110.4174@racer.site","subject":"Re: [PATCH] git-branch: only traverse the requested refs","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2007-10-10T23:30:42Z","receivedAt":"2007-10-10T23:30:42Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On 10/11/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> On Wed, 10 Oct 2007, Lars Hjemli wrote:\n> > +     if (kinds & REF_LOCAL_BRANCH) {\n> > +             ref_list.kinds = REF_LOCAL_BRANCH;\n> > +             for_each_branch_ref(append_ref, &ref_list);\n> > +     }\n>\n> The function for_each_branch_ref() calls do_for_each_ref(), which in turn\n> calls get_loose_refs(), which calls get_ref_dir() to read all loose refs,\n> if they have not yet been read.\n\nOk, I'll see if get_loose_refs() could take 'const char *base' and\npass this on to get_ref_dir(), which should solve the problem.\n\nThanks for noticing.\n\n-- \nlarsh\n"},{"id":"55438","messageId":"alpine.LFD.0.999.0710101610230.20690@woody.linux-foundation.org","threadId":"10237","inReplyTo":"f329bf540710101430i63926b25q7d55976af96b891d@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-10-10T23:39:19Z","receivedAt":"2007-10-10T23:39:19Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 10 Oct 2007, Han-Wen Nienhuys wrote:\n> \n> The way I solved that, was to have both repositories pointing to each\n> other, using alternates.\n\nOuch. Double un-good. Not a good idea. Especially not if you do \ndevelopment in both and pull and push between them. \n\nWhat will happen is that if you do alternates pointing both ways, you \nbasically end up having a \"shared pool of objects\". So it's pretty much \nequivalent to just using a shared object directory, and it has *exactly* \nthe same issues with object reachability and references: you have a shared \npool of objects, but you only ever see *one* set of references, so garbage \ncollection cannot work - because it will always see just a subset of the \nreal references, while it sees essentially all objects.\n\n> could it be that GC does not handle cyclic alternates correctly?\n\nIt's not about cyclic per se: it's about the fact that GC will do garbage \ncollection based on reachability with the local references.\n\nWhich is normally fine. It's normally fine, because the object tree is \n\"local\" too. But when doing alternates:\n\n - the tree that is being used as an alternate *has* to be totally stable. \n   It must *never* have been re-based, or have any GC'able objects in the \n   first place. IOW, doing a \"git gc\" on it will be safe, because there is \n   no way any objects that the other alternate depends on could be pruned.\n\n - You definitely must *not* do a two-way alternate, because that violates \n   another rule: the rule that the \"alternate base\" (which is now *both*\n   of the repositories) is self-sufficient. Since they both point to each \n   other, there's no way to know whether they are self-sufficient or not: \n   they may be re-using each others objects *and* packs!\n\nAnd in the above, the \"*and* packs\" is important, and probably the cause \nof your problems. Because \"git repack -a -d -l\" (which is what \"git gc\" \ndoes) will always gather up any loose objects even from remote sites, but \nthe \"-l\" means that it will not do so for alternate packed objects.\n\nSo what happens is that if one of the repositories can reach some object \nthat is in a pack in the other repository, \"git gc\" will still *leave* it \ndependent on a pack in the other repository. But maybe that object isn't \neven reachable in the other repo any more (for whatever reason - a rebase, \nwhatever), then when you repack the other repository, now all the packs \nwill be replaced by one new pack - and the one new pack will only contain \nthe objects reachable from the other repo.\n\nIOW: alternates are dangerous. A shared object directory is dangerous. You \nshould basically only do it under very controlled circumstances, and \notherwise you should use either hardlinks or if you want added safety, \ntotally separate repositories.\n\nBasically, here's an example of badness, with A and B being repos that \npoint to each other.\n\n - do something in A\n - pull it into B - this leaves the objects in A, because of the \n   alternates link.\n - rebase A\n - \"git gc\" in A: this removes unreachable objects from A, and now B is \n   screwed.\n\nSo the rule really is: never *ever* do anything but fast-forward in a repo \nthat is an alternate for another one. If you do a circular link, I think \nit's still safe if you follow that rule, but now obviously the rule holds \nfor *both* repos (and quite frankly, I'd worry so much that I'd never do \nit even then).\n\nThere should be another rule too: git on its own is not a backup system. \nYou can use git *as* a backup system, but you need to do so by mirroring \nthe whole repository, and not on the same disk.\n\n(ie, for me, git *is* a backup system, but that's only because I push my \nrepos to other sites - a single git repo on its own has zero redundancy)\n\n\t\tLinus\n"},{"id":"55447","messageId":"f329bf540710101926vedf8b19p52e3eeb193203d03@mail.gmail.com","threadId":"10237","inReplyTo":"alpine.LFD.0.999.0710101610230.20690@woody.linux-foundation.org","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-11T02:26:24Z","receivedAt":"2007-10-11T02:26:24Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/10, Linus Torvalds <torvalds@linux-foundation.org>:\n\n> IOW: alternates are dangerous. A shared object directory is dangerous. You\n> should basically only do it under very controlled circumstances, and\n> otherwise you should use either hardlinks or if you want added safety,\n> totally separate repositories.\n\nI recall reading a few months ago that it was \"clone -l\" that gave you\nthe jeebies, rather than \"clone -s\".\n\n\n> So the rule really is: never *ever* do anything but fast-forward in a repo\n>[..]\n\nMethinks this is all too difficult. I will use clone -l henceforth. Is\nthere any reason to prefer -s over -l? Given your lengthy exposition\non the dangers of alternates, I would say this is a features that\ndeserves to be buried or at least deemphasized in the documentation.\n\nFor cherrypicking convenience, I would still appreciate it if there\nwas a mechanism similar to alternates that would allow me to view\nobjects from an alternate repo; objects found through this mechanism\nshould never be assumed to be present in the database, of course.\n\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55450","messageId":"20071011064126.GA3649@steel.home","threadId":"10237","inReplyTo":"f329bf540710101926vedf8b19p52e3eeb193203d03@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2007-10-11T06:41:26Z","receivedAt":"2007-10-11T06:41:26Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Han-Wen Nienhuys, Thu, Oct 11, 2007 04:26:24 +0200:\n> > So the rule really is: never *ever* do anything but fast-forward in a repo\n> >[..]\n> \n> Methinks this is all too difficult. I will use clone -l henceforth.\n\nIt is current default for local clones\n"},{"id":"55458","messageId":"e2b179460710110241i445bc61ep8ae60e421c19c0f0@mail.gmail.com","threadId":"10237","inReplyTo":"470D585D.1030808@nrlssc.navy.mil","subject":"Re: Spam: Re: git branch performance problem?","fromName":"Mike Ralphson","fromEmail":"mike.ralphson@gmail.com","sentAt":"2007-10-11T09:41:06Z","receivedAt":"2007-10-11T09:41:06Z","isPatch":false,"sender":{"key":"mike.ralphson@gmail.com","avatar":"https://avatars.githubusercontent.com/u/21603?v=4"},"body":"On 10/10/07, Brandon Casey <casey@nrlssc.navy.mil> wrote:\n> No, this is not the case, unless something has changed very recently\n> in git-gc or git-repack. Even git-gc with no arguments is unsafe if\n> the repository being gc'ed is listed in another's alternates.\n>\n> git-gc calls repack with -a and -d. which causes a new pack to be\n> created which only contains the objects required by the local repository.\n> The other packs are then deleted. Objects contained in those packs and\n> required by a \"sharing\" repository (one using the alternates mechanism)\n> will be deleted if the local repository no longer references them.\n\nIt's not something I've really looked into, but there seems to be a\nreflogs mechanism which can temporarily pin an otherwise unreferenced\nobject so it doesn't get deleted. Would it be possible to populate the\nremote's view of referenced objects into this, at the point of clone,\npush or pull, which would seem to be the points at which this might be\nchanging.\n\nObviously this is of no use if you're 'anonymously' poncing off a\nthird repo to save clone time, but if you're in control of both repo's\nit might be useful.\n\nMike\n"},{"id":"55459","messageId":"Pine.LNX.4.64.0710111145090.4174@racer.site","threadId":"10237","inReplyTo":"f329bf540710101926vedf8b19p52e3eeb193203d03@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-10-11T10:46:28Z","receivedAt":"2007-10-11T10:46:28Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 10 Oct 2007, Han-Wen Nienhuys wrote:\n\n> For cherrypicking convenience, I would still appreciate it if there was \n> a mechanism similar to alternates that would allow me to view objects \n> from an alternate repo; objects found through this mechanism should \n> never be assumed to be present in the database, of course.\n\nSilly question: why don't you just\n\n\tgit remote add -f other <url>\n\nand then review the changes with \"git log\", \"git diff\" and \"git show\"?\n\nCiao,\nDscho\n"},{"id":"55460","messageId":"Pine.LNX.4.64.0710111148000.4174@racer.site","threadId":"10237","inReplyTo":"e2b179460710110241i445bc61ep8ae60e421c19c0f0@mail.gmail.com","subject":"Re: Spam: Re: git branch performance problem?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-10-11T10:58:31Z","receivedAt":"2007-10-11T10:58:31Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 11 Oct 2007, Mike Ralphson wrote:\n\n> It's not something I've really looked into, but there seems to be a\n> reflogs mechanism which can temporarily pin an otherwise unreferenced\n> object so it doesn't get deleted. Would it be possible to populate the\n> remote's view of referenced objects into this, at the point of clone,\n> push or pull, which would seem to be the points at which this might be\n> changing.\n> \n> Obviously this is of no use if you're 'anonymously' poncing off a\n> third repo to save clone time, but if you're in control of both repo's\n> it might be useful.\n\nI cannot really allege that I understood what you were trying to say, but \nI guess you want to use clone to get rid of objects you just threw out by \neither filter-branch or deleting a branch.\n\nThe answer is that the file:// as well as the git:// protocol will do \nthat.  For local clones, they are not the default, since they are slower \nthan hardlinking.\n\nHth,\nDscho\n"},{"id":"55463","messageId":"f329bf540710110611j7ea39cfdx416b43b902fcc9e4@mail.gmail.com","threadId":"10237","inReplyTo":"Pine.LNX.4.64.0710111145090.4174@racer.site","subject":"Re: git branch performance problem?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2007-10-11T13:11:41Z","receivedAt":"2007-10-11T13:11:41Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"2007/10/11, Johannes Schindelin <Johannes.Schindelin@gmx.de>:\n> > For cherrypicking convenience, I would still appreciate it if there was\n> > a mechanism similar to alternates that would allow me to view objects\n> > from an alternate repo; objects found through this mechanism should\n> > never be assumed to be present in the database, of course.\n>\n> Silly question: why don't you just\n>\n>         git remote add -f other <url>\n>\n> and then review the changes with \"git log\", \"git diff\" and \"git show\"?\n\nThank for the tip; I'll look into it.\n\n-- \nHan-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"55478","messageId":"alpine.LFD.0.999.0710110807170.20690@woody.linux-foundation.org","threadId":"10237","inReplyTo":"f329bf540710101926vedf8b19p52e3eeb193203d03@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-10-11T15:16:47Z","receivedAt":"2007-10-11T15:16:47Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 10 Oct 2007, Han-Wen Nienhuys wrote:\n> \n> I recall reading a few months ago that it was \"clone -l\" that gave you\n> the jeebies, rather than \"clone -s\".\n\nYes, \"clone -l\" gives me the jeebies, because I'm a totally anal person \nwhen it comes to disk corruption and a worry-wart. I've just had it happen \ntoo many times (usually because a disk simply goes bad), and \"git clone \n-l\" basically means that if one repository gets corrupted, then so does \nthe other one.\n\nBut clone -s gives me even *more* jeebies, although I think it's in some \nrespect also more useful. The alternates thing is really useful for \nservers in particular, where you basically want to have multiple \n\"branches\" maintained by lots of people, but all based on some expected \nbase version.\n\nSo if you think of alternates as a \"kernel.org\" or \"repo.or.cz\" thing, \nwhere you might have a hundred different repositories all based on the \nsame \"standard\" version, then I think you basically have the right model. \nIn that situation, \"git clone -l\" doesn't work that well, since the \nrepositories just start out sharing data, but don't do it long term.\n\nSo \"git clone -l\" (which is the default now - my jeebies really are my \npersonal psychological problem) is really useful for latency reasons for a \nlocal clone, and has basically no real downsides. It's not useful for \n*backups*, but it's useful for development.\n\n> > So the rule really is: never *ever* do anything but fast-forward in a repo\n> >[..]\n> \n> Methinks this is all too difficult. I will use clone -l henceforth. Is\n> there any reason to prefer -s over -l?\n\nGood. And no, for actual *development* there is no reason to prefer -s \nover -l (and as mentioned, '-l' is the default in modern versions).\n\nFor a git *server* setup, -s is better, since it's more long-term. But in \nthat situation, it also requires that the server maintainer have some \nrules (ie only use \"-s\" for stable base trees and/or use extra care when \nrepacking the base).\n\n> Given your lengthy exposition on the dangers of alternates, I would say \n> this is a features that deserves to be buried or at least deemphasized \n> in the documentation.\n\nI do agree. We should make the dangers very clear.\n\n> For cherrypicking convenience, I would still appreciate it if there\n> was a mechanism similar to alternates that would allow me to view\n> objects from an alternate repo; objects found through this mechanism\n> should never be assumed to be present in the database, of course.\n\nWell, the way that really should work is that you \"git fetch remote\" and \nwork on the end result in a \"remote branch\".\n\nThat *will* make the objects present in the database, but not in your \nactual branches (until you cherry-pick), but there really are no real \ndownsides. If the remote is truly related to your local tree, it all \ndelta's so well that the disk space issues should basically be none.\n\n\t\tLinus\n"},{"id":"55568","messageId":"470FAC97.5070904@gmail.com","threadId":"10237","inReplyTo":"f329bf540710101926vedf8b19p52e3eeb193203d03@mail.gmail.com","subject":"Re: git branch performance problem?","fromName":"Salikh Zakirov","fromEmail":"salikh@gmail.com","sentAt":"2007-10-12T17:19:19Z","receivedAt":"2007-10-12T17:19:19Z","isPatch":false,"sender":{"key":"salikh@gmail.com","avatar":"https://gravatar.com/avatar/952c102bb1dcf721dab8de4f5a11d276756a65d301d021f755e265cc3251efae?d=mp&s=160"},"body":"Han-Wen Nienhuys wrote:\n> For cherrypicking convenience, I would still appreciate it if there\n> was a mechanism similar to alternates that would allow me to view\n> objects from an alternate repo; objects found through this mechanism\n> should never be assumed to be present in the database, of course.\n\nThere exist a script contrib/workdir/git-new-workdir,\nwhich creates a new working copy that literally shares the same object store.\nIt will share both object store and branches, so some care must be taken:\nbranch which checkout out in one shared working directory must never be updated\n(committed or pulled into) from the other shared working directory.\n\nSaid that, I personally find this trick very useful for browsing alternate\nbranch code and quick bug fixing.\n"}]}