{"thread":{"id":"47003","subject":"\"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","startedAt":"2017-10-20T01:44:04Z","lastAt":"2017-11-07T12:32:16Z","messageCount":11,"participants":["Stefan Beller","Jeff King","Kaartic Sivaraam"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"330707","messageId":"CAGZ79kYP0z1G_H3nwfmSHraWHMBOcik5LepUXKj0nveeBrihiw@mail.gmail.com","threadId":"47003","inReplyTo":null,"subject":"\"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-20T01:43:57Z","receivedAt":"2017-10-20T01:44:04Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"So I ran \"git fetch --all\" inside git.git that you are all familiar with.\nAll fetches failed with a similar error as:\nFetching kernelorg\nfatal: bad object HEAD\nerror: https://kernel.googlesource.com/pub/scm/git/git did not send\nall necessary objects\n\nerror: Could not fetch kernelorg\n\nWorking with an older version of Git (2.13.0) gives me\n\n$ GIT_TRACE=/u/trace GIT_TRACE_PACKET=/u/out git fetch git://github.com/git/git\nFrom git://github.com/git/git\n * branch HEAD -> FETCH_HEAD\n\nAfter fiddling around and some printf debugging,\nclean fsck(!)\nI could not make sense of it, so bisecting\n\n$ git bisect bad\nd0c39a49ccb5dfe7feba4325c3374d99ab123c59 is the first bad commit\ncommit d0c39a49ccb5dfe7feba4325c3374d99ab123c59\nAuthor: Nguyễn Thái Ngọc Duy \nDate: Wed Aug 23 19:36:59 2017 +0700\n\n revision.c: --all adds HEAD from all worktrees\n\n Unless single_worktree is set, --all now adds HEAD from all worktrees.\n\n Since reachable.c code does not use setup_revisions(), we need to call\n other_head_refs_submodule() explicitly there to have the same effect on\n \"git prune\", so that we won't accidentally delete objects needed by some\n other HEADs.\n\n A new FIXME is added because we would need something like\n\n int refs_other_head_refs(struct ref_store *, each_ref_fn, cb_data);\n\n in addition to other_head_refs() to handle it, which might require\n\n int get_submodule_worktrees(const char *submodule, int flags);\n\n It could be a separate topic to reduce the scope of this one.\n\n Signed-off-by: Nguyễn Thái Ngọc Duy \n Signed-off-by: Junio C Hamano \n\nAnd here we go, I do use worktrees nowadays!\n$ git worktree list\n/usr/local/google/home/sbeller/OSS/git 117ddefdb4\n(detached HEAD)\n/u/git d0c39a49cc\n(detached HEAD)\n/usr/local/google/home/sbeller/OSS/git_origin_master b14f27f917\n(detached HEAD)\n/usr/local/google/home/sbeller/OSS/submodule_remote_dot 3d9025bd69\n(detached HEAD)\n\n$ git show 3d9025bd69\nfatal: ambiguous argument '3d9025bd69': unknown revision or path not\nin the working tree.\n\nok, so I presume I just delete that working tree to \"fix my copy of git\"\nsbeller@sbeller:/u/git$ rm -rf\n/usr/local/google/home/sbeller/OSS/submodule_remote_dot\nsbeller@sbeller:/u/git$ git worktree prune\nsbeller@sbeller:/u/git$ git worktree list\n/usr/local/google/home/sbeller/OSS/git 117ddefdb4 (detached HEAD)\n/u/git d0c39a49cc (detached HEAD)\n/usr/local/google/home/sbeller/OSS/git_origin_master b14f27f917 (detached HEAD)\n$ git bisect reset\nPrevious HEAD position was d0c39a49cc... revision.c: --all adds HEAD\nfrom all worktrees\nHEAD is now at 660fb3dfa8... Sync with maint\n$ make install\n$ git fetch --all\n# works fine!\n\nAny idea which direction we need to heading for a real fix?\n"},{"id":"330710","messageId":"20171020031630.44zvzh3d2vlhglv4@sigill.intra.peff.net","threadId":"47003","inReplyTo":"CAGZ79kYP0z1G_H3nwfmSHraWHMBOcik5LepUXKj0nveeBrihiw@mail.gmail.com","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-10-20T03:16:30Z","receivedAt":"2017-10-20T03:16:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 19, 2017 at 06:43:57PM -0700, Stefan Beller wrote:\n\n> So I ran \"git fetch --all\" inside git.git that you are all familiar with.\n> All fetches failed with a similar error as:\n> Fetching kernelorg\n> fatal: bad object HEAD\n> error: https://kernel.googlesource.com/pub/scm/git/git did not send\n> all necessary objects\n> \n> error: Could not fetch kernelorg\n\nI ran into this a while back with worktrees, too. In my case I had a\nworktree which was \"owned\" by multiple repositories[1]. One would write\nthe HEAD, but the other did not necessarily have those objects.\n\nI don't quite recall how I ended up there. Doing \"git worktree add\"\nmultiple times does seem to complain about the directory already\nexisting. My case did involve relative symlinks in both repositories, so\nmaybe that confused things. Anyway, I can't seem to replicate it now.\n\n> And here we go, I do use worktrees nowadays!\n> $ git worktree list\n> /usr/local/google/home/sbeller/OSS/git 117ddefdb4\n> (detached HEAD)\n> /u/git d0c39a49cc\n> (detached HEAD)\n> /usr/local/google/home/sbeller/OSS/git_origin_master b14f27f917\n> (detached HEAD)\n> /usr/local/google/home/sbeller/OSS/submodule_remote_dot 3d9025bd69\n> (detached HEAD)\n> \n> $ git show 3d9025bd69\n> fatal: ambiguous argument '3d9025bd69': unknown revision or path not\n> in the working tree.\n\nSo maybe you've ended up in the same situation and somehow \"reused\"\nsubmodule_remote_dot.\n\nBut I think until that commit you mentioned (d0c39a49cc), \"gc\" in the\nparent repo could drop objects accessible only from the worktree. So if\nthat happened _before_ d0c39a49cc, then at that point your worktree was\ncorrupted. And now of course further operations will complain.\n\n> ok, so I presume I just delete that working tree to \"fix my copy of git\"\n> sbeller@sbeller:/u/git$ rm -rf\n> /usr/local/google/home/sbeller/OSS/submodule_remote_dot\n> sbeller@sbeller:/u/git$ git worktree prune\n> sbeller@sbeller:/u/git$ git worktree list\n> /usr/local/google/home/sbeller/OSS/git 117ddefdb4 (detached HEAD)\n> /u/git d0c39a49cc (detached HEAD)\n> /usr/local/google/home/sbeller/OSS/git_origin_master b14f27f917 (detached HEAD)\n\nSo that's the right immediate workaround (and hoping there was nothing\nvaluable in that worktree!).\n\n> Any idea which direction we need to heading for a real fix?\n\nIf my analysis above is correct, then it's already fixed. You just had\nleftover corruption.\n\n-Peff\n"},{"id":"330711","messageId":"CAGZ79kZc+O9gM97bVZETE3sgkmc-t78Nf_Hq6=K4Gf2yaE79zA@mail.gmail.com","threadId":"47003","inReplyTo":"20171020031630.44zvzh3d2vlhglv4@sigill.intra.peff.net","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-20T05:27:28Z","receivedAt":"2017-10-20T05:27:35Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Oct 19, 2017 at 8:16 PM, Jeff King wrote:\n> On Thu, Oct 19, 2017 at 06:43:57PM -0700, Stefan Beller wrote:\n\n>\n> So that's the right immediate workaround (and hoping there was nothing\n> valuable in that worktree!).\n\nHow would I know? ;)\n\nWithout the commit available I do not even know if the code there\nis well reasoned about. (My commit messages usually tell my future\nself how much thought I put into the code)\n\n>\n>> Any idea which direction we need to heading for a real fix?\n>\n> If my analysis above is correct, then it's already fixed. You just had\n> leftover corruption.\n\nWell fetching yesterday worked and the commit in question is from\n8/23, the merge 8a044c7f1d56cef657be342e40de0795d688e882\noccurred 9/18, so I suspect there is something else at play.\n(I do not remember having a gc between yesterday and today.\nThough maybe one in the background?)\n\n\nI am curious how you can have a worktree owned by multiple\nrepositories [1] (?).\n\nThanks,\nStefan\n"},{"id":"330714","messageId":"20171020060443.l6v74ik4v4jdt4ky@sigill.intra.peff.net","threadId":"47003","inReplyTo":"CAGZ79kZc+O9gM97bVZETE3sgkmc-t78Nf_Hq6=K4Gf2yaE79zA@mail.gmail.com","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-10-20T06:04:43Z","receivedAt":"2017-10-20T06:04:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 19, 2017 at 10:27:28PM -0700, Stefan Beller wrote:\n\n> > If my analysis above is correct, then it's already fixed. You just had\n> > leftover corruption.\n> \n> Well fetching yesterday worked and the commit in question is from\n> 8/23, the merge 8a044c7f1d56cef657be342e40de0795d688e882\n> occurred 9/18, so I suspect there is something else at play.\n> (I do not remember having a gc between yesterday and today.\n> Though maybe one in the background?)\n\nEven a gc between yesterday and today should have used the new code,\nwhich would have been safe. So yeah, maybe it is something else\nentirely.\n\n> I am curious how you can have a worktree owned by multiple\n> repositories [1] (?).\n\nSorry, I forgot my footnote. I saw this with my \"ci\" script:\n\n https://github.com/peff/git/blob/7905ff395adecdd2bb7ab045a24223dfb103e0e9/ci\n\nI check out the contents of my \"meta\" branch as \"Meta\", and it contains\nthat script. It basically just waits for ref updates, then walks over\nall the commits and runs \"make test\" on them in the background (caching\nthe results, thanks to the git-test[1] script). So I kick off \"Meta/ci\"\nin a terminal and forget about it, and magically it builds my commits in\nthe background as I work.\n\nIt operates in a worktree inside the Meta directory (Meta/tmp-ci), so as\nnot to disturb what I'm doing. So far so good.\n\nBut I actually have _two_ clones of Git on my system. One on which I do\nmost of my work, and then the other which has the fork we use in\nproduction at GitHub. I symlink the Meta directory from the first into\nthe latter, which means they both see the same worktree directory. And\nsomehow running \"Meta/ci\" in the second corrupted things.\n\nI can get some funniness now, but I think it's mostly caused by the\nscript being confused about the worktree existing but not having access\nto our branches. That's not a corruption, just a confusion. I _think_ I\nhad a bogus HEAD in the worktree at one point, but I may be\nmis-remembering. I can't seem to trigger it now.\n\n-Peff\n\n[1] https://github.com/mhagger/git-test\n"},{"id":"330741","messageId":"CAGZ79kaeJsahVuWgFsJfyGahciT4xBeM3m59F5crGy4+ZRJMCw@mail.gmail.com","threadId":"47003","inReplyTo":"20171020060443.l6v74ik4v4jdt4ky@sigill.intra.peff.net","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-20T20:45:31Z","receivedAt":"2017-10-20T20:45:36Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Oct 19, 2017 at 11:04 PM, Jeff King wrote:\n> On Thu, Oct 19, 2017 at 10:27:28PM -0700, Stefan Beller wrote:\n>\n>> > If my analysis above is correct, then it's already fixed. You just had\n>> > leftover corruption.\n>>\n>> Well fetching yesterday worked and the commit in question is from\n>> 8/23, the merge 8a044c7f1d56cef657be342e40de0795d688e882\n>> occurred 9/18, so I suspect there is something else at play.\n>> (I do not remember having a gc between yesterday and today.\n>> Though maybe one in the background?)\n>\n> Even a gc between yesterday and today should have used the new code,\n> which would have been safe. So yeah, maybe it is something else\n> entirely.\n\nOh, yeah.\n\n>\n>> I am curious how you can have a worktree owned by multiple\n>> repositories [1] (?).\n>\n> Sorry, I forgot my footnote. I saw this with my \"ci\" script:\n>\n> https://github.com/peff/git/blob/7905ff395adecdd2bb7ab045a24223dfb103e0e9/ci\n>\n> I check out the contents of my \"meta\" branch as \"Meta\", and it contains\n> that script. It basically just waits for ref updates, then walks over\n> all the commits and runs \"make test\" on them in the background (caching\n> the results, thanks to the git-test[1] script). So I kick off \"Meta/ci\"\n> in a terminal and forget about it, and magically it builds my commits in\n> the background as I work.\n>\n> It operates in a worktree inside the Meta directory (Meta/tmp-ci), so as\n> not to disturb what I'm doing. So far so good.\n>\n> But I actually have _two_ clones of Git on my system. One on which I do\n> most of my work, and then the other which has the fork we use in\n> production at GitHub. I symlink the Meta directory from the first into\n> the latter, which means they both see the same worktree directory. And\n> somehow running \"Meta/ci\" in the second corrupted things.\n>\n> I can get some funniness now, but I think it's mostly caused by the\n> script being confused about the worktree existing but not having access\n> to our branches. That's not a corruption, just a confusion. I _think_ I\n> had a bogus HEAD in the worktree at one point, but I may be\n> mis-remembering. I can't seem to trigger it now.\n\nThanks for these insights.\n\nI played around with Meta a bit, but I did not feel it would enhance\nmy workflow enough as I am not involved with any maintainance\nof git using git.\n\nThe git-test from Michael sounds intriguing. Initially I put off using\nit as I had my main working dir (or rather test dir) on a spinning\ndisk, still. Now I test in memory only, which is a lot faster, so I could\ntry if git-test can keep up with my local commit pace.\n\nThanks,\nStefan\n"},{"id":"330803","messageId":"1508682297.6715.15.camel@gmail.com","threadId":"47003","inReplyTo":"CAGZ79kaeJsahVuWgFsJfyGahciT4xBeM3m59F5crGy4+ZRJMCw@mail.gmail.com","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2017-10-22T14:24:57Z","receivedAt":"2017-10-22T14:25:08Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:\n> \n> The git-test from Michael sounds intriguing. Initially I put off using\n> it as I had my main working dir (or rather test dir) on a spinning\n> disk, still. Now I test in memory only, which is a lot faster, so I could\n> try if git-test can keep up with my local commit pace.\n> \n\nExcuse my ignorance but I don't get your statement correctly. What do\nyou mean by \"I test in memory only\"? How do you do that?\n\n---\nKaartic\n"},{"id":"330811","messageId":"20171023003647.4pvzmhk7v5t6zqm5@sigill.intra.peff.net","threadId":"47003","inReplyTo":"1508682297.6715.15.camel@gmail.com","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-10-23T00:36:48Z","receivedAt":"2017-10-23T00:37:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Oct 22, 2017 at 07:54:57PM +0530, Kaartic Sivaraam wrote:\n\n> On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:\n> > \n> > The git-test from Michael sounds intriguing. Initially I put off using\n> > it as I had my main working dir (or rather test dir) on a spinning\n> > disk, still. Now I test in memory only, which is a lot faster, so I could\n> > try if git-test can keep up with my local commit pace.\n> > \n> \n> Excuse my ignorance but I don't get your statement correctly. What do\n> you mean by \"I test in memory only\"? How do you do that?\n\nI assume it means pointing \"--root\" at a tmpfs or ram disk in\nGIT_TEST_OPTS. That makes the test suite run _way_ faster.\n\n-Peff\n"},{"id":"330844","messageId":"CAGZ79kYAn0Hi0qaG8P5zvVOFTsfXYrtvrR7oiT0KwwWeKLqZgw@mail.gmail.com","threadId":"47003","inReplyTo":"20171023003647.4pvzmhk7v5t6zqm5@sigill.intra.peff.net","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-23T17:37:55Z","receivedAt":"2017-10-23T17:38:06Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Oct 22, 2017 at 5:36 PM, Jeff King wrote:\n> On Sun, Oct 22, 2017 at 07:54:57PM +0530, Kaartic Sivaraam wrote:\n>\n>> On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:\n>> >\n>> > The git-test from Michael sounds intriguing. Initially I put off using\n>> > it as I had my main working dir (or rather test dir) on a spinning\n>> > disk, still. Now I test in memory only, which is a lot faster, so I could\n>> > try if git-test can keep up with my local commit pace.\n>> >\n>>\n>> Excuse my ignorance but I don't get your statement correctly. What do\n>> you mean by \"I test in memory only\"? How do you do that?\n>\n> I assume it means pointing \"--root\" at a tmpfs or ram disk in\n> GIT_TEST_OPTS. That makes the test suite run _way_ faster.\n\nExactly. By memory I mean volatile RAM (as opposed to\nmemory on a spinning disk).\n\nUsing GIT_TEST_OPTS has had some issues (I remember vaguely\nthere was an inconsistency between the output of `make test` and prove),\nso I put my entire working tree on a tmpfs, I run roughly this script\nafter booting my computer:\n\n sudo mount -t tmpfs -o size=16g tmpfs /u\n mkdir /u/git\n echo \"gitdir:\n/usr/local/google/home/sbeller/OSS/git/.git/worktrees/git\"\n>/u/git/.git\n git -C /u/git checkout -f HEAD\n\n cat </u/config.mak\n DEVELOPER=1\n DEVELOPERS=1\n CFLAGS += -g -O2\n CFLAGS += -DFLEX_ARRAY=2048\n #CFLAGS += -Wno-unused-value\n EOF\n\nThe downside (or upside?) is that there will be no files cluttering\nmy git directory, but any untracked file is wiped periodically. So if I want\nto really preserve a thing I am forcing myself to put it somewhere else\nthan in an untracked file inside the git repo.\n\nThe test suite (excluding t9*) runs in less than 50 seconds on the ram\ndisk.\n\nThanks,\nStefan\n\n>\n> -Peff\n"},{"id":"331766","messageId":"0f1f8317-9d82-5e61-5332-f877ca886056@gmail.com","threadId":"47003","inReplyTo":"CAGZ79kYAn0Hi0qaG8P5zvVOFTsfXYrtvrR7oiT0KwwWeKLqZgw@mail.gmail.com","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2017-11-03T09:32:52Z","receivedAt":"2017-11-03T09:33:04Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On Monday 23 October 2017 11:07 PM, Stefan Beller wrote:\n> Exactly. By memory I mean volatile RAM (as opposed to\n> memory on a spinning disk).\n> \n> Using GIT_TEST_OPTS has had some issues (I remember vaguely\n> there was an inconsistency between the output of `make test` and prove),\n> so I put my entire working tree on a tmpfs, I run roughly this script\n> after booting my computer:\n> \n> sudo mount -t tmpfs -o size=16g tmpfs /u\n> mkdir /u/git\n> echo \"gitdir:\n> /usr/local/google/home/sbeller/OSS/git/.git/worktrees/git\"\n>> /u/git/.git\n> git -C /u/git checkout -f HEAD\n> \n> cat </u/config.mak\n> DEVELOPER=1\n> DEVELOPERS=1\n> CFLAGS += -g -O2\n> CFLAGS += -DFLEX_ARRAY=2048\n> #CFLAGS += -Wno-unused-value\n> EOF\n\nDid I thank you for a good explanation? If not, thanks that was \ninteresting and enlightening.\n\n> The test suite (excluding t9*) runs in less than 50 seconds on the ram\n> disk.\n> \n\nBTW, this is what I call _way way_ faster. Unfortunately due to the \nlimited configuration of my system, the test suite has following timing\n\n real 3m14.482s\n user 2m10.556s\n sys 1m12.328s\n\nAnyways it's better than the timing I get for running it on a spinning \ndisk which takes\n\n real 4m37.585s\n user 2m17.244s\n sys 1m37.136s\n\nThanks,\nKaartic\n"},{"id":"331800","messageId":"CAGZ79kY+R_Kd7BbmnRb0C1H94HvZ3TGRvWKFOX8XEStjBQMB-A@mail.gmail.com","threadId":"47003","inReplyTo":"0f1f8317-9d82-5e61-5332-f877ca886056@gmail.com","subject":"Re: \"Cannot fetch git.git\" (worktrees at fault? or origin/HEAD) ?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-03T19:11:43Z","receivedAt":"2017-11-03T19:11:50Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 3, 2017 at 2:32 AM, Kaartic Sivaraam\n wrote:\n> On Monday 23 October 2017 11:07 PM, Stefan Beller wrote:\n>>\n>> Exactly. By memory I mean volatile RAM (as opposed to\n>> memory on a spinning disk).\n>>\n>> Using GIT_TEST_OPTS has had some issues (I remember vaguely\n>> there was an inconsistency between the output of `make test` and prove),\n>> so I put my entire working tree on a tmpfs, I run roughly this script\n>> after booting my computer:\n>>\n>> sudo mount -t tmpfs -o size=16g tmpfs /u\n>> mkdir /u/git\n>> echo \"gitdir:\n>> /usr/local/google/home/sbeller/OSS/git/.git/worktrees/git\"\n>>>\n>>> /u/git/.git\n>>\n>> git -C /u/git checkout -f HEAD\n>>\n>> cat </u/config.mak\n>> DEVELOPER=1\n>> DEVELOPERS=1\n>> CFLAGS += -g -O2\n>> CFLAGS += -DFLEX_ARRAY=2048\n>> #CFLAGS += -Wno-unused-value\n>> EOF\n>\n>\n> Did I thank you for a good explanation? If not, thanks that was interesting\n> and enlightening.\n>\n>> The test suite (excluding t9*) runs in less than 50 seconds on the ram\n>> disk.\n>>\n\nJust tested again, I meant to say 70s instead of 50s.\n\n\n> BTW, this is what I call _way way_ faster. Unfortunately due to the limited\n> configuration of my system, the test suite has following timing\n>\n> real 3m14.482s\n> user 2m10.556s\n> sys 1m12.328s\n\nThis sounds to me as if it is running with just one thread\n(because sys+user = real); I usually run with\n\n cd t\n prove --jobs 25 t[0-8][0-9]*.sh\n\nThe multithreading can be seen in the timing as well\n real 1m9.913s\n user 1m50.796s\n sys 0m54.092s\nas user > real already.\n\nIf you run tests via 'make test' or 'cd t && make', you can also give\na --jobs \nto make it faster. I have no good answer for how many, but I have the impression\noverloading the system is no big deal. (I only have a few cores in this machine,\n4 or 6, but still run with --jobs 25; 'git grep sleep' returns a\ncouple of lines,\nand such threads sleeping definitely don't need a CPU)\n\nThanks,\nStefan\n"},{"id":"331989","messageId":"1510057755.8228.4.camel@gmail.com","threadId":"47003","inReplyTo":"CAGZ79kY+R_Kd7BbmnRb0C1H94HvZ3TGRvWKFOX8XEStjBQMB-A@mail.gmail.com","subject":"Re: \"Cannot fetch git.git\" ( iworktrees at fault? or origin/HEAD) ?","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2017-11-07T12:29:15Z","receivedAt":"2017-11-07T12:32:16Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On Fri, 2017-11-03 at 12:11 -0700, Stefan Beller wrote:\n> On Fri, Nov 3, 2017 at 2:32 AM, Kaartic Sivaraam\n> wrote:\n> > BTW, this is what I call _way way_ faster. Unfortunately due to the limited\n> > configuration of my system, the test suite has following timing\n> > \n> > real 3m14.482s\n> > user 2m10.556s\n> > sys 1m12.328s\n> \n> This sounds to me as if it is running with just one thread\n> (because sys+user = real); I usually run with\n> \n> cd t\n> prove --jobs 25 t[0-8][0-9]*.sh\n> \n> The multithreading can be seen in the timing as well\n> real 1m9.913s\n> user 1m50.796s\n> sys 0m54.092s\n> as user > real already.\n> \n> If you run tests via 'make test' or 'cd t && make', you can also give\n> a --jobs \n> to make it faster. I have no good answer for how many, but I have the impression\n> overloading the system is no big deal. (I only have a few cores in this machine,\n> 4 or 6, but still run with --jobs 25; 'git grep sleep' returns a\n> couple of lines,\n> and such threads sleeping definitely don't need a CPU)\n> \n\nI usually use the following command to build and run the test suite\nfrom the root of the working directory,\n\n make NO_GETTEXT=1 DEVELOPER=1 DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS=\"--timer --jobs 16\" test\n\n I beleived that the DEFAULT_TST_TARGE and GIT_PROVE_OPTS options would\n be seen by the Makefile in t/ but didn't verify it. Your statement made\n me suspicious and I wanted to know whether my assumption was correct or\n not. So, I did this,\n\n\n 1. I first tried running the above command while seeing how much time\n on the 'next' branch when it was pointing at,\n\n 9a519c715 (Merge branch 'jk/rebase-i-exec-gitdir-fix' into next, 2017-11-02)\n\n First, I observed that at least part of my assumption held on observing\n that the output was similar to that of 'prove'. This shows that\n t/Makefile recognizes DEFAULT_TEST_TARGET.\n\n Next, to verify if it recognizes GIT_PROVE_OPTS I ran the above command\n with 'time' and it gave back the following statistics,\n\n real \t 4m55.707s\n user \t 2m29.924s\n sys \t 1m48.072s\n\n which is in line with the statistics for the spinning disk in the\n previous mail. (The time doesn't include the build time).\n\n (real)-(user+sys) = 38s (approx.)\n\n\n 2. Now I switched to the t/ directory, I tried running the following,\n\n make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS=\"--timer --jobs 16\"\n\n 'time' gave the statistics for the above command as follows,\n\n real \t 5m19.523s\n user \t 2m26.764s\n sys \t 1m45.240s\n\n (real)-(user+sys) =  68s (approx.)\nNot a big improvement from the previous case to assure that\nGIT_PROVE_OPTS wasn't recognized there.\n\n\n3. Staying in the t/ directory, I tried running the following just to\nbe pretty sure that the \"--jobs\" was indeed sent to 'prove',\n\n prove --jobs 16 t[0-9][0-9]*.sh\n\n'time' gave the statistics for the above command as follows,\n\n real \t 4m49.241s\n user \t 2m29.220s\n sys \t 1m47.828s\n\n (real)-(user+sys) = 34s (approx.)\n\n This seems to be identical with the first case.\n\n So, it does seem to be a limitation of my system (Intel i3 with 2\n cores, FWIW).\n\n\n 4. To see if things improve with a higher number of jobs I tried,\n\n prove --jobs 25 t[0-9][0-9]*.sh\n\n 'time' gave the statistics for the above command as follows,\n\n real \t 5m21.229s\n user \t 2m25.704s\n sys \t 1m46.744s\n\n (real)-(user+sys) = 70s (approx.)\n\n This ensured the limitation once more!\n\n Anyways, thanks for the explanation and information which was\n enlightening. :-)\n\n-- \nKaartic\n"}]}