{"thread":{"id":"12553","subject":"Re: warning: no common commits - slow pull","startedAt":"2008-03-07T01:35:46Z","lastAt":"2008-03-10T17:40:53Z","messageCount":9,"participants":["David Brownell","Daniel Barkalow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"71300","messageId":"200803061735.47674.david-b@pacbell.net","threadId":"12553","inReplyTo":null,"subject":"Re: warning: no common commits - slow pull","fromName":"David Brownell","fromEmail":"david-b@pacbell.net","sentAt":"2008-03-07T01:35:46Z","receivedAt":"2008-03-07T01:35:46Z","isPatch":false,"sender":{"key":"david-b@pacbell.net","avatar":null},"body":"Any progress on fixing this?\n\nI'll report that with git version 1.5.4.3.447.gc95b3.dirty\n(just a couple days old) I've observed this when updating\na clone by pulling from\n\n (a) a parent on the same disk partition\n (b) a parent on a non-mirrored network server\n\nSo that would seem to trash the assumptions that this is\nrelated to version mismatch between mirrors, and that the\nfix can (or should!) wait till 1.6.0 ...\n\n\nI was glad to see the \"^C\" workaround, that seems to work.\n\nWhen I \"git pull\" it first fetches a bunch of files, then\nconcludes (wrongly) \"no common commits\", then starts a\nsecond fetch of a *HUGE* number of files ... 400 MB is too\nmuch to pay when updating from rc3-last-week to rc4.  But\nif I interrupt that second one with ^C, it seems that the\nfirst one fetched enough to make the next \"git pull\" go\npretty quickly.\n\n\nThis does seem appear to show up more often lately because\nof RC4 having been tagged ... but I don't know for sure.\n\nI've got a couple kernel workspaces still on last Friday's\nversion, which -- if this holds true to form! -- will show\nthis bug when I \"git pull\".  So if there are experiments\nthat would help nail down what's going on here, please\nspell them out to me (\"this command, then this ... send\nthis output...\").\n\n- Dave\n"},{"id":"71391","messageId":"alpine.LNX.1.00.0803072006550.19665@iabervon.org","threadId":"12553","inReplyTo":"200803061735.47674.david-b@pacbell.net","subject":"Re: warning: no common commits - slow pull","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-03-08T01:22:01Z","receivedAt":"2008-03-08T01:22:01Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Thu, 6 Mar 2008, David Brownell wrote:\n\n> Any progress on fixing this?\n> \n> I'll report that with git version 1.5.4.3.447.gc95b3.dirty\n> (just a couple days old) I've observed this when updating\n> a clone by pulling from\n> \n>  (a) a parent on the same disk partition\n>  (b) a parent on a non-mirrored network server\n> \n> So that would seem to trash the assumptions that this is\n> related to version mismatch between mirrors, and that the\n> fix can (or should!) wait till 1.6.0 ...\n> \n> I was glad to see the \"^C\" workaround, that seems to work.\n> \n> When I \"git pull\" it first fetches a bunch of files, then\n> concludes (wrongly) \"no common commits\", then starts a\n> second fetch of a *HUGE* number of files ... 400 MB is too\n> much to pay when updating from rc3-last-week to rc4.  But\n> if I interrupt that second one with ^C, it seems that the\n> first one fetched enough to make the next \"git pull\" go\n> pretty quickly.\n> \n> \n> This does seem appear to show up more often lately because\n> of RC4 having been tagged ... but I don't know for sure.\n> \n> I've got a couple kernel workspaces still on last Friday's\n> version, which -- if this holds true to form! -- will show\n> this bug when I \"git pull\".  So if there are experiments\n> that would help nail down what's going on here, please\n> spell them out to me (\"this command, then this ... send\n> this output...\").\n\nActually, if you can make a tarball of the .git directory of one of those \nworkspaces, and see if the bug is reproducable with that .git directory \nevery time (particularly when pulling a local repository), it would be \nreally helpful to have a reliable test case.\n\nThere's a debugging thing that would help, but it's not in your version. \nIt's in next as\n\n49aaddd102aff Teach upload-pack to log the received need lines to an fd\n\nWith that commit, you should be able to do:\n\nGIT_DEBUG_SEND_PACK=3 git pull 3>UPLOAD_LOG\n\nand get a file UPLOAD_LOG that will show what it's doing, although there's \na reasonable chance that it'll only demonstrate that it's doing nothing \nhelpful, which we already pretty much know.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"71455","messageId":"200803081448.47000.david-b@pacbell.net","threadId":"12553","inReplyTo":"alpine.LNX.1.00.0803072006550.19665@iabervon.org","subject":"Re: warning: no common commits - slow pull","fromName":"David Brownell","fromEmail":"david-b@pacbell.net","sentAt":"2008-03-08T22:48:46Z","receivedAt":"2008-03-08T22:48:46Z","isPatch":false,"sender":{"key":"david-b@pacbell.net","avatar":null},"body":"On Friday 07 March 2008, Daniel Barkalow wrote:\n> On Thu, 6 Mar 2008, David Brownell wrote:\n> \n> > When I \"git pull\" it first fetches a bunch of files, then\n> > concludes (wrongly) \"no common commits\", then starts a\n> > second fetch of a *HUGE* number of files ... 400 MB is too\n> > much to pay when updating from rc3-last-week to rc4.  But\n> > if I interrupt that second one with ^C, it seems that the\n> > first one fetched enough to make the next \"git pull\" go\n> > pretty quickly.\n> \n> Actually, if you can make a tarball of the .git directory of one of those \n> workspaces, and see if the bug is reproducable with that .git directory \n> every time (particularly when pulling a local repository), it would be \n> really helpful to have a reliable test case.\n\nSeems to be.  Let me know where I can stash a ~300 MB tar.bz2 file\nfor you ...\n\n\n> There's a debugging thing that would help, but it's not in your version.\n\nSo I updated.  :)\n\n\n> It's in next as\n> \n> 49aaddd102aff Teach upload-pack to log the received need lines to an fd\n> \n> With that commit, you should be able to do:\n> \n> GIT_DEBUG_SEND_PACK=3 git pull 3>UPLOAD_LOG\n> \n> and get a file UPLOAD_LOG that will show what it's doing, although there's \n> a reasonable chance that it'll only demonstrate that it's doing nothing \n\nThat file has always been empty.\n\n\n> helpful, which we already pretty much know.\n\nOddly, a few times when I tried that the bug didn't reproduce.\nOne factor may be workspaces cloned a long time ago with early\nversions of GIT (or cogito).\n\n- Dave\n\n"},{"id":"71456","messageId":"alpine.LNX.1.00.0803081750580.19665@iabervon.org","threadId":"12553","inReplyTo":"200803081448.47000.david-b@pacbell.net","subject":"Re: warning: no common commits - slow pull","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-03-08T22:58:33Z","receivedAt":"2008-03-08T22:58:33Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sat, 8 Mar 2008, David Brownell wrote:\n\n> On Friday 07 March 2008, Daniel Barkalow wrote:\n> > On Thu, 6 Mar 2008, David Brownell wrote:\n> > \n> > > When I \"git pull\" it first fetches a bunch of files, then\n> > > concludes (wrongly) \"no common commits\", then starts a\n> > > second fetch of a *HUGE* number of files ... 400 MB is too\n> > > much to pay when updating from rc3-last-week to rc4.  But\n> > > if I interrupt that second one with ^C, it seems that the\n> > > first one fetched enough to make the next \"git pull\" go\n> > > pretty quickly.\n> > \n> > Actually, if you can make a tarball of the .git directory of one of those \n> > workspaces, and see if the bug is reproducable with that .git directory \n> > every time (particularly when pulling a local repository), it would be \n> > really helpful to have a reliable test case.\n> \n> Seems to be.  Let me know where I can stash a ~300 MB tar.bz2 file\n> for you ...\n\nWherever you've got web space, I guess. It shouldn't be a problem for me \non the download side.\n\n> > There's a debugging thing that would help, but it's not in your version.\n> \n> So I updated.  :)\n> \n> \n> > It's in next as\n> > \n> > 49aaddd102aff Teach upload-pack to log the received need lines to an fd\n> > \n> > With that commit, you should be able to do:\n> > \n> > GIT_DEBUG_SEND_PACK=3 git pull 3>UPLOAD_LOG\n> > \n> > and get a file UPLOAD_LOG that will show what it's doing, although there's \n> > a reasonable chance that it'll only demonstrate that it's doing nothing \n> \n> That file has always been empty.\n\nHmm; if the testing code is working, you should get at least one pair of \n\"#B/#E\" lines. You probably need to either test with current \"next\" or \ncherry-pick that commit, since I don't think that commit's in a particular \nhurry to get to \"master\"\n\n> > helpful, which we already pretty much know.\n> \n> Oddly, a few times when I tried that the bug didn't reproduce.\n> One factor may be workspaces cloned a long time ago with early\n> versions of GIT (or cogito).\n\nIs it a few times on different workspaces, or with copies of the same \nworkspace? I think that there's a dependance on how your current state and \nthe new state happen to line up.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"71475","messageId":"200803081525.08181.david-b@pacbell.net","threadId":"12553","inReplyTo":"alpine.LNX.1.00.0803081750580.19665@iabervon.org","subject":"Re: warning: no common commits - slow pull","fromName":"David Brownell","fromEmail":"david-b@pacbell.net","sentAt":"2008-03-08T23:25:07Z","receivedAt":"2008-03-08T23:25:07Z","isPatch":false,"sender":{"key":"david-b@pacbell.net","avatar":null},"body":"On Saturday 08 March 2008, Daniel Barkalow wrote:\n> On Sat, 8 Mar 2008, David Brownell wrote:\n> > Oddly, a few times when I tried that the bug didn't reproduce.\n> > One factor may be workspaces cloned a long time ago with early\n> > versions of GIT (or cogito).\n> \n> Is it a few times on different workspaces, or with copies of the same \n> workspace?\n\nI had several workspaces that weren't updated ... which by previous\nexperience should *all* have done an extra update.  I did the pull\nwith the GIT_DEBUG on a few of them, and the bogus one didn't happen.\n\nThis seemed odd to me so the next time I tried without the GIT_DEBUG\nthing ... and got the bogus pull with \"no common commits\".  This was\nunfortunately *after* I deleted tarballs of the previous workspaces,\nso that exact experiment can't be reproduced.  Those were, however,\n\"newer\" workspaces -- probably created last summer, rather than\na few years ago.\n\n\n> I think that there's a dependance on how your current state and  \n> the new state happen to line up.\n\nThe workspace in the tarball I can send you doesn't have any issues\nwith reproducibility ... now.  \n\n- Dave\n"},{"id":"71478","messageId":"alpine.LNX.1.00.0803081826080.19665@iabervon.org","threadId":"12553","inReplyTo":"200803081525.08181.david-b@pacbell.net","subject":"Re: warning: no common commits - slow pull","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-03-08T23:27:55Z","receivedAt":"2008-03-08T23:27:55Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sat, 8 Mar 2008, David Brownell wrote:\n\n> On Saturday 08 March 2008, Daniel Barkalow wrote:\n> > I think that there's a dependance on how your current state and  \n> > the new state happen to line up.\n> \n> The workspace in the tarball I can send you doesn't have any issues\n> with reproducibility ... now.  \n\nLet me see if I can write up a patch that you should be able to test \nwithout moving the big file around. I may run out of time for tonight, \nthough.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"71519","messageId":"alpine.LNX.1.00.0803091443440.19665@iabervon.org","threadId":"12553","inReplyTo":"alpine.LNX.1.00.0803081826080.19665@iabervon.org","subject":"Re: warning: no common commits - slow pull","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-03-09T18:47:08Z","receivedAt":"2008-03-09T18:47:08Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sat, 8 Mar 2008, Daniel Barkalow wrote:\n\n> On Sat, 8 Mar 2008, David Brownell wrote:\n> \n> > On Saturday 08 March 2008, Daniel Barkalow wrote:\n> > > I think that there's a dependance on how your current state and  \n> > > the new state happen to line up.\n> > \n> > The workspace in the tarball I can send you doesn't have any issues\n> > with reproducibility ... now.  \n> \n> Let me see if I can write up a patch that you should be able to test \n> without moving the big file around. I may run out of time for tonight, \n> though.\n\nTry this. I'm not at all sure that it's doing what I want, but it passes \nall the current tests, and it should only affect your test case if it's \nactually right.\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 29b38e4..37cae35 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -26,6 +26,8 @@ static const char fetch_pack_usage[] =\n #define SEEN\t\t(1U << 3)\n #define POPPED\t\t(1U << 4)\n \n+static int marked;\n+\n /*\n  * After sending this many \"have\"s if we do not get any new ACK , we\n  * give up traversing our history.\n@@ -61,6 +63,16 @@ static int rev_list_insert_ref(const char *path, const unsigned char *sha1, int\n \treturn 0;\n }\n \n+static int clear_marks(const char *path, const unsigned char *sha1, int flag, void *cb_data)\n+{\n+\tstruct object *o = deref_tag(parse_object(sha1), path, 0);\n+\n+\tif (o && o->type == OBJ_COMMIT)\n+\t\tclear_commit_marks((struct commit *)o, \n+\t\t\t\t   COMMON | COMMON_REF | SEEN | POPPED);\n+\treturn 0;\n+}\n+\n /*\n    This function marks a rev and its ancestors as common.\n    In some cases, it is desirable to mark only the ancestors (for example\n@@ -153,6 +165,10 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tunsigned in_vain = 0;\n \tint got_continue = 0;\n \n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n+\n \tfor_each_ref(rev_list_insert_ref, NULL);\n \n \tfetching = 0;\n"},{"id":"71596","messageId":"200803101018.09282.david-b@pacbell.net","threadId":"12553","inReplyTo":"alpine.LNX.1.00.0803091443440.19665@iabervon.org","subject":"Re: warning: no common commits - slow pull","fromName":"David Brownell","fromEmail":"david-b@pacbell.net","sentAt":"2008-03-10T17:18:09Z","receivedAt":"2008-03-10T17:18:09Z","isPatch":false,"sender":{"key":"david-b@pacbell.net","avatar":null},"body":"On Sunday 09 March 2008, Daniel Barkalow wrote:\n> Try this. I'm not at all sure that it's doing what I want, but it passes \n> all the current tests, and it should only affect your test case if it's \n> actually right.\n\nSeems to resolve that problem for me ... well timed, in terms of RC5!\n\nBut some stuff still looks a bit fishy.  See this RC4 --> RC5 pull:\n\n  remote: Counting objects: 1329, done.\n  remote: Compressing objects: 100% (276/276), done.\n  remote: Total 908 (delta 749), reused 760 (delta 631)\n  Receiving objects: 100% (908/908), 146.35 KiB | 40 KiB/s, done.\n  Resolving deltas: 100% (749/749), completed with 287 local objects.\n  remote: Counting objects: 1330, done.\n  remote: Compressing objects: 100% (277/277), done.\n  remote: Total 909 (delta 749), reused 760 (delta 631)\n  Receiving objects: 100% (909/909), 146.63 KiB | 38 KiB/s, done.\n  Resolving deltas: 100% (749/749), completed with 287 local objects.\n  From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6\n   * [new tag]         v2.6.25-rc5 -> v2.6.25-rc5\n  Updating 29e8c3c..cdeeeae\n  Fast forward\n  Auto packing your repository for optimum performance. You may also\n  run \"git gc\" manually. See \"git help gc\" for more information.\n  ...\n\nThat's pretty typical:  two count/compress/receive/resolve passes,\nwith almost an identical shape and size.  Every time I see that, I\nsuspect that most of the second one should not be needed...\n\nHowever, the grace of not re-fetching the entire GIT database (at\nseveral hundred MBytes) the second time is much appreciated!\n\n- Dave\n\n\n> --- a/builtin-fetch-pack.c\n> +++ b/builtin-fetch-pack.c\n> @@ -26,6 +26,8 @@ static const char fetch_pack_usage[] =\n>  #define SEEN\t\t(1U << 3)\n>  #define POPPED\t\t(1U << 4)\n>  \n> +static int marked;\n> +\n>  /*\n>   * After sending this many \"have\"s if we do not get any new ACK , we\n>   * give up traversing our history.\n> @@ -61,6 +63,16 @@ static int rev_list_insert_ref(const char *path, const unsigned char *sha1, int\n>  \treturn 0;\n>  }\n>  \n> +static int clear_marks(const char *path, const unsigned char *sha1, int flag, void *cb_data)\n> +{\n> +\tstruct object *o = deref_tag(parse_object(sha1), path, 0);\n> +\n> +\tif (o && o->type == OBJ_COMMIT)\n> +\t\tclear_commit_marks((struct commit *)o, \n> +\t\t\t\t   COMMON | COMMON_REF | SEEN | POPPED);\n> +\treturn 0;\n> +}\n> +\n>  /*\n>     This function marks a rev and its ancestors as common.\n>     In some cases, it is desirable to mark only the ancestors (for example\n> @@ -153,6 +165,10 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n>  \tunsigned in_vain = 0;\n>  \tint got_continue = 0;\n>  \n> +\tif (marked)\n> +\t\tfor_each_ref(clear_marks, NULL);\n> +\tmarked = 1;\n> +\n>  \tfor_each_ref(rev_list_insert_ref, NULL);\n>  \n>  \tfetching = 0;\n> \n"},{"id":"71597","messageId":"alpine.LNX.1.00.0803101327050.19665@iabervon.org","threadId":"12553","inReplyTo":"200803101018.09282.david-b@pacbell.net","subject":"Re: warning: no common commits - slow pull","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-03-10T17:40:53Z","receivedAt":"2008-03-10T17:40:53Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 10 Mar 2008, David Brownell wrote:\n\n> On Sunday 09 March 2008, Daniel Barkalow wrote:\n> > Try this. I'm not at all sure that it's doing what I want, but it passes \n> > all the current tests, and it should only affect your test case if it's \n> > actually right.\n> \n> Seems to resolve that problem for me ... well timed, in terms of RC5!\n> \n> But some stuff still looks a bit fishy.  See this RC4 --> RC5 pull:\n> \n>   remote: Counting objects: 1329, done.\n>   remote: Compressing objects: 100% (276/276), done.\n>   remote: Total 908 (delta 749), reused 760 (delta 631)\n>   Receiving objects: 100% (908/908), 146.35 KiB | 40 KiB/s, done.\n>   Resolving deltas: 100% (749/749), completed with 287 local objects.\n>   remote: Counting objects: 1330, done.\n>   remote: Compressing objects: 100% (277/277), done.\n>   remote: Total 909 (delta 749), reused 760 (delta 631)\n>   Receiving objects: 100% (909/909), 146.63 KiB | 38 KiB/s, done.\n>   Resolving deltas: 100% (749/749), completed with 287 local objects.\n>   From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6\n>    * [new tag]         v2.6.25-rc5 -> v2.6.25-rc5\n>   Updating 29e8c3c..cdeeeae\n>   Fast forward\n>   Auto packing your repository for optimum performance. You may also\n>   run \"git gc\" manually. See \"git help gc\" for more information.\n>   ...\n> \n> That's pretty typical:  two count/compress/receive/resolve passes,\n> with almost an identical shape and size.  Every time I see that, I\n> suspect that most of the second one should not be needed...\n\nAh, I bet it's not rereading the list of current refs when it updates \nthem, and therefore only says it has the stuff that it had before. That \nwould actually explain somewhat why the other bug was actually mattering.\n\nShould be easy to fix, but I'm not sure when I'll get a chance to work on \nit.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"}]}