{"thread":{"id":"20461","subject":"Re: git gc expanding packed data?","startedAt":"2009-08-08T01:11:35Z","lastAt":"2009-09-28T04:18:07Z","messageCount":26,"participants":["Andreas Schwab","Hin-Tak Leung","Nicolas Pitre","Jason Merrill","Matthieu Moy","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"119944","messageId":"m2tz0j154o.fsf@igel.home","threadId":"20461","inReplyTo":null,"subject":"Re: git gc expanding packed data?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2009-08-08T01:11:35Z","receivedAt":"2009-08-08T01:11:35Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Nicolas Pitre writes:\n\n> It appears that the git installation serving clone requests for\n> git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I\n> just cloned it and the pack I was sent contains 1383356 objects (can be\n> determined with 'git show-index < .git/objects/pack/*.idx | wc -l').\n> However, there are only 978501 actually referenced objects in that\n> cloned repository ( 'git rev-list --all --objects | wc -l'). That makes\n> for 404855 useless objects in the cloned repository.\n\nThose objects are not useless. They are referenced by the remote refs\non the remote side, which are not fetched by default. If you clone a\nmirror of the repository you'll see no unreferenced objects.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"119984","messageId":"3ace41890908080605k4ec6661bmcb4c87e10bc5fd87@mail.gmail.com","threadId":"20461","inReplyTo":"m2tz0j154o.fsf@igel.home","subject":"Re: git gc expanding packed data?","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-08-08T13:05:22Z","receivedAt":"2009-08-08T13:05:22Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On Sat, Aug 8, 2009 at 2:11 AM, Andreas Schwab wrote:\n> Nicolas Pitre writes:\n>\n>> It appears that the git installation serving clone requests for\n>> git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I\n>> just cloned it and the pack I was sent contains 1383356 objects (can be\n>> determined with 'git show-index < .git/objects/pack/*.idx | wc -l').\n>> However, there are only 978501 actually referenced objects in that\n>> cloned repository ( 'git rev-list --all --objects | wc -l').  That makes\n>> for 404855 useless objects in the cloned repository.\n>\n> Those objects are not useless.  They are referenced by the remote refs\n> on the remote side, which are not fetched by default.  If you clone a\n> mirror of the repository you'll see no unreferenced objects.\n>\n> Andreas.\n>\n> --\n> Andreas Schwab, schwab@linux-m68k.org\n> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n> \"And now for something completely different.\"\n>\n\nThanks... It is a difference between svn and git mentality probably -\none only pushes reasonably reliable code to a public git repository,\nwhereas anything transient is recorded in svn - I think many of the\nunreferenced objects are svn user-branches (which are probably of use\nto people who intend to work on gcc for fairly extended periods,\nrather than casual users like me).\nThe case with gcc is probably quite extreme - many user branches, and\nvery large code base - but is there anything on the git side with git\ngc which can lessen this kind of pathological behavior (expanding\npacks)?\n\nThanks a lot for the explanation and the discussion.\n\nHin-Tak\n"},{"id":"119985","messageId":"m21vnm8mk8.fsf@igel.home","threadId":"20461","inReplyTo":"3ace41890908080605k4ec6661bmcb4c87e10bc5fd87@mail.gmail.com","subject":"Re: git gc expanding packed data?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2009-08-08T13:25:27Z","receivedAt":"2009-08-08T13:25:27Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Hin-Tak Leung writes:\n\n> Thanks... It is a difference between svn and git mentality probably -\n\nIt is just that the remote is a git-svn tree, with only a few branches\ncreated as local branches.\n\n> The case with gcc is probably quite extreme - many user branches, and\n> very large code base - but is there anything on the git side with git\n> gc which can lessen this kind of pathological behavior (expanding\n> packs)?\n\nIf you fetch all refs, not only refs/heads/*, all objects will be\nreferenced.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"120026","messageId":"alpine.LFD.2.00.0908082246020.440@xanadu.home","threadId":"20461","inReplyTo":"m2tz0j154o.fsf@igel.home","subject":"Re: git gc expanding packed data?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-09T02:56:57Z","receivedAt":"2009-08-09T02:56:57Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 8 Aug 2009, Andreas Schwab wrote:\n\n> Nicolas Pitre writes:\n> \n> > It appears that the git installation serving clone requests for\n> > git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I\n> > just cloned it and the pack I was sent contains 1383356 objects (can be\n> > determined with 'git show-index < .git/objects/pack/*.idx | wc -l').\n> > However, there are only 978501 actually referenced objects in that\n> > cloned repository ( 'git rev-list --all --objects | wc -l'). That makes\n> > for 404855 useless objects in the cloned repository.\n> \n> Those objects are not useless. They are referenced by the remote refs\n> on the remote side, which are not fetched by default. If you clone a\n> mirror of the repository you'll see no unreferenced objects.\n\nIf you do a clone using the git:// protocol and the server sends you \nonly the ref for the trunk branch, then it should send you only objects \nreachable from that branch. Any extra objects sent by the server are \nuseless to me and wastes my and everyone else's bandwidth, and on my \nnext repack those objects are pruned anyway. The point of the git \nprotocol is _not_ necessarily to send a copy of the remote pack file \nover, even during a clone.\n\n\nNicolas\n"},{"id":"120036","messageId":"m2k51dzb39.fsf@linux-m68k.org","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0908082246020.440@xanadu.home","subject":"Re: git gc expanding packed data?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2009-08-09T07:43:22Z","receivedAt":"2009-08-09T07:43:22Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Nicolas Pitre writes:\n\n> If you do a clone using the git:// protocol and the server sends you \n> only the ref for the trunk branch,\n\nA clone will fetch all branches from refs/heads/*.\n\n> then it should send you only objects reachable from that branch.\n\nApparantly this does not work. I'd guess the extra objects are needed\ndue to the delta compression.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"123790","messageId":"4ABD0669.7050309@redhat.com","threadId":"20461","inReplyTo":"m2k51dzb39.fsf@linux-m68k.org","subject":"Re: git clone sending unneeded objects (was : git gc expanding packed data?)","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-25T18:05:29Z","receivedAt":"2009-09-25T18:05:29Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"On 08/09/2009 03:43 AM, Andreas Schwab wrote:\n> Nicolas Pitre writes:\n>\n>> If you do a clone using the git:// protocol and the server sends you\n>> only the ref for the trunk branch,\n>\n> A clone will fetch all branches from refs/heads/*.\n>\n>> then it should send you only objects reachable from that branch.\n>\n> Apparantly this does not work. I'd guess the extra objects are needed\n> due to the delta compression.\n\nI just tried doing a clone of the GCC repository, then git gc \n--prune=now, and another clone specifying --reference to the first, and \nit wanted to download all the unreachable objects again. So it doesn't \nseem to be a compression issue.\n\nThis is with git 1.6.4 on both ends.\n\nJason\n"},{"id":"123797","messageId":"vpqvdj6izt6.fsf@bauges.imag.fr","threadId":"20461","inReplyTo":"4ABD0669.7050309@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2009-09-25T19:34:13Z","receivedAt":"2009-09-25T19:34:13Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jason Merrill writes:\n\n> On 08/09/2009 03:43 AM, Andreas Schwab wrote:\n>> Nicolas Pitre writes:\n>>\n>>> If you do a clone using the git:// protocol and the server sends you\n>>> only the ref for the trunk branch,\n>>\n>> A clone will fetch all branches from refs/heads/*.\n>>\n>>> then it should send you only objects reachable from that branch.\n>>\n>> Apparantly this does not work. I'd guess the extra objects are needed\n>> due to the delta compression.\n>\n> I just tried doing a clone of the GCC repository, then git gc\n> --prune=now, and another clone specifying --reference to the first,\n> and it wanted to download all the unreachable objects again. So it\n> doesn't seem to be a compression issue.\n>\n> This is with git 1.6.4 on both ends.\n\nWhich protocol did you use?\n\nIf you use git:// or ssh://, it's normally a security feature that Git\nsends you only reachable objects. If it doesn't, it's a serious bug.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"123798","messageId":"4ABD1D6D.2070404@redhat.com","threadId":"20461","inReplyTo":"vpqvdj6izt6.fsf@bauges.imag.fr","subject":"Re: git clone sending unneeded objects","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-25T19:43:41Z","receivedAt":"2009-09-25T19:43:41Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"On 09/25/2009 03:34 PM, Matthieu Moy wrote:\n> Which protocol did you use?\n\ngit://\n\nJason\n"},{"id":"123799","messageId":"alpine.LFD.2.00.0909251551290.4997@xanadu.home","threadId":"20461","inReplyTo":"vpqvdj6izt6.fsf@bauges.imag.fr","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-25T19:53:09Z","receivedAt":"2009-09-25T19:53:09Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 25 Sep 2009, Matthieu Moy wrote:\n\n> Jason Merrill writes:\n> \n> > On 08/09/2009 03:43 AM, Andreas Schwab wrote:\n> >> Nicolas Pitre writes:\n> >>\n> >>> If you do a clone using the git:// protocol and the server sends you\n> >>> only the ref for the trunk branch,\n> >>\n> >> A clone will fetch all branches from refs/heads/*.\n> >>\n> >>> then it should send you only objects reachable from that branch.\n> >>\n> >> Apparantly this does not work. I'd guess the extra objects are needed\n> >> due to the delta compression.\n> >\n> > I just tried doing a clone of the GCC repository, then git gc\n> > --prune=now, and another clone specifying --reference to the first,\n> > and it wanted to download all the unreachable objects again. So it\n> > doesn't seem to be a compression issue.\n> >\n> > This is with git 1.6.4 on both ends.\n> \n> Which protocol did you use?\n> \n> If you use git:// or ssh://, it's normally a security feature that Git\n> sends you only reachable objects. If it doesn't, it's a serious bug.\n\nI did reproduce the issue with git:// back when this discussion started. \nI also asked for more information about the remote which didn't come \nforth.\n\n\nNicolas\n"},{"id":"123801","messageId":"4ABD25FE.2040902@redhat.com","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0909251551290.4997@xanadu.home","subject":"Re: git clone sending unneeded objects","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-25T20:20:14Z","receivedAt":"2009-09-25T20:20:14Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"On 09/25/2009 03:53 PM, Nicolas Pitre wrote:\n> I did reproduce the issue with git:// back when this discussion started.\n> I also asked for more information about the remote which didn't come\n> forth.\n\nLooking back, I only see you asking about the git version on the server, \nwhich is 1.6.4.\n\nSo again:\n\ngit clone git://gcc.gnu.org/git/gcc.git\n (1399509 objects, ~600MB .git dir)\ngit gc --prune=now (988906 objects, ~450MB .git dir)\n\n...then\n\ngit clone git://gcc.gnu.org/git/gcc.git --reference $firstclone\n (573401 objects, ~550MB .git dir)\ngit fsck (clean)\ngit gc --prune=now (5 objects, ~7MB .git dir)\n\nWhat's going on here?\n\nJason\n"},{"id":"123802","messageId":"alpine.LFD.2.00.0909251629330.4997@xanadu.home","threadId":"20461","inReplyTo":"4ABD25FE.2040902@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-25T20:47:52Z","receivedAt":"2009-09-25T20:47:52Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 25 Sep 2009, Jason Merrill wrote:\n\n> On 09/25/2009 03:53 PM, Nicolas Pitre wrote:\n> > I did reproduce the issue with git:// back when this discussion started.\n> > I also asked for more information about the remote which didn't come\n> > forth.\n> \n> Looking back, I only see you asking about the git version on the server, which\n> is 1.6.4.\n> \n> So again:\n> \n> git clone git://gcc.gnu.org/git/gcc.git\n> (1399509 objects, ~600MB .git dir)\n> git gc --prune=now (988906 objects, ~450MB .git dir)\n> \n> ...then\n> \n> git clone git://gcc.gnu.org/git/gcc.git --reference $firstclone\n> (573401 objects, ~550MB .git dir)\n> git fsck (clean)\n> git gc --prune=now (5 objects, ~7MB .git dir)\n> \n> What's going on here?\n\nSome screw up.\n\nDo you have access to the remote machine? Is it possible to have a \ntarball of the gcc.git directory from there?\n\n\nNicolas\n"},{"id":"123823","messageId":"4ABD4F7B.4030701@redhat.com","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0909251629330.4997@xanadu.home","subject":"Re: git clone sending unneeded objects","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-25T23:17:15Z","receivedAt":"2009-09-25T23:17:15Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"On 09/25/2009 04:47 PM, Nicolas Pitre wrote:\n> Do you have access to the remote machine? Is it possible to have a\n> tarball of the gcc.git directory from there?\n\nhttp://gcc.gnu.org/gcc-git.tar.gz\n\nI'll leave it there for a few days.\n\nJason\n"},{"id":"123828","messageId":"3ace41890909251743v7a51027agb039514dd0636058@mail.gmail.com","threadId":"20461","inReplyTo":"4ABD25FE.2040902@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-09-26T00:43:57Z","receivedAt":"2009-09-26T00:43:57Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On Fri, Sep 25, 2009 at 9:20 PM, Jason Merrill wrote:\n> On 09/25/2009 03:53 PM, Nicolas Pitre wrote:\n>>\n>> I did reproduce the issue with git:// back when this discussion started.\n>> I also asked for more information about the remote which didn't come\n>> forth.\n>\n> Looking back, I only see you asking about the git version on the server,\n> which is 1.6.4.\n\nHmm, I was under the impression from the previous thread that the\nserver is a bit older and/or have more backward compatible settings to\ncater for older git clients?\n\n>\n> So again:\n>\n> git clone git://gcc.gnu.org/git/gcc.git\n> (1399509 objects, ~600MB .git dir)\n> git gc --prune=now (988906 objects, ~450MB .git dir)\n>\n> ...then\n>\n> git clone git://gcc.gnu.org/git/gcc.git --reference $firstclone\n> (573401 objects, ~550MB .git dir)\n> git fsck (clean)\n> git gc --prune=now (5 objects, ~7MB .git dir)\n>\n> What's going on here?\n\nFWIW, I still have my clone (git://) and do my periodic 'git fetch'\nand 'git gc prune=now' (learned my lessons!) and it is currently .git\ndir is about 350MB. (from previous discussion the optimal at the time\nwas about 300MB, so it has grown a bit in the last couple of months).\n\nAnd thanks everybody for all the discussion and advice. git is a great\ntool. (and I have essentially stopped using svn, prefering git-svn!).\n"},{"id":"123829","messageId":"alpine.LFD.2.00.0909252045290.4997@xanadu.home","threadId":"20461","inReplyTo":"4ABD4F7B.4030701@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-26T00:49:27Z","receivedAt":"2009-09-26T00:49:27Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 25 Sep 2009, Jason Merrill wrote:\n\n> On 09/25/2009 04:47 PM, Nicolas Pitre wrote:\n> > Do you have access to the remote machine? Is it possible to have a\n> > tarball of the gcc.git directory from there?\n> \n> http://gcc.gnu.org/gcc-git.tar.gz\n> \n> I'll leave it there for a few days.\n\nThanks, I got it now. And I was able to reproduce the issue locally.\n\nCloning the original repository does transfer objects which become \nunreferenced in the clone. But cloning that cloned repository (before \npruning the unreferenced objects) does not transfer those objects again. \n\nJust need to find out why.\n\n\nNicolas\n"},{"id":"123832","messageId":"alpine.LFD.2.00.0909252314260.4997@xanadu.home","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0909252045290.4997@xanadu.home","subject":"[PATCH] make 'git clone' ask the remote only for objects it cares about","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-26T03:54:42Z","receivedAt":"2009-09-26T03:54:42Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Current behavior of 'git clone' when not using --mirror is to fetch \neverything from the peer, and then filter out unwanted refs just before \nwriting them out to the cloned repository. This may become highly \ninefficient if the peer has an unusual ref namespace, or if it simply \nhas \"remotes\" refs of its own, and those locally unwanted refs are \nconnecting to a large set of objects which becomes unreferenced as soon \nas they are fetched.\n\nLet's filter out those unwanted refs from the peer _before_ asking it \nwhat refs we want to fetch instead, which is the most logical thing to \ndo anyway.\n\nSigned-off-by: Nicolas Pitre \n---\n\nOn Fri, 25 Sep 2009, Nicolas Pitre wrote:\n\n> On Fri, 25 Sep 2009, Jason Merrill wrote:\n> \n> > On 09/25/2009 04:47 PM, Nicolas Pitre wrote:\n> > > Do you have access to the remote machine? Is it possible to have a\n> > > tarball of the gcc.git directory from there?\n> > \n> > http://gcc.gnu.org/gcc-git.tar.gz\n> > \n> > I'll leave it there for a few days.\n> \n> Thanks, I got it now. And I was able to reproduce the issue locally.\n> \n> Cloning the original repository does transfer objects which become \n> unreferenced in the clone. But cloning that cloned repository (before \n> pruning the unreferenced objects) does not transfer those objects again. \n> \n> Just need to find out why.\n\nAnd the \"why\" is described above. The problem was actually on the \nclient side and was affecting clones of any repository containing \nanything outside refs/heads and refs/tags.\n\nThe fact that the git repository on gcc.gnu.org has lots of stuff in \n\"remote\" branches that don't get cloned by default is a separate \nconfiguration/policy issue on that server which might need (or not) to \nbe looked into. For instance at least, as a bare repository, it should \nhave all the git files in gcc.git/ directly instead of gcc.git/.git/.\n\ndiff --git a/builtin-clone.c b/builtin-clone.c\nindex bab2d84..edf7c7f 100644\n--- a/builtin-clone.c\n+++ b/builtin-clone.c\n@@ -329,24 +329,28 @@ static void remove_junk_on_signal(int signo)\n \traise(signo);\n }\n \n-static struct ref *write_remote_refs(const struct ref *refs,\n-\t\tstruct refspec *refspec, const char *reflog)\n+static struct ref *wanted_peer_refs(const struct ref *refs,\n+\t\tstruct refspec *refspec)\n {\n \tstruct ref *local_refs = NULL;\n \tstruct ref **tail = &local_refs;\n-\tstruct ref *r;\n \n \tget_fetch_map(refs, refspec, &tail, 0);\n \tif (!option_mirror)\n \t\tget_fetch_map(refs, tag_refspec, &tail, 0);\n \n+\treturn local_refs;\n+}\n+\n+static void write_remote_refs(const struct ref *local_refs, const char *reflog)\n+{\n+\tconst struct ref *r;\n+\n \tfor (r = local_refs; r; r = r->next)\n \t\tadd_extra_ref(r->peer_ref->name, r->old_sha1, 0);\n \n \tpack_refs(PACK_REFS_ALL);\n \tclear_extra_refs();\n-\n-\treturn local_refs;\n }\n \n int cmd_clone(int argc, const char **argv, const char *prefix)\n@@ -495,9 +499,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tstrbuf_reset(&value);\n \n-\tif (path && !is_bundle)\n+\tif (path && !is_bundle) {\n \t\trefs = clone_local(path, git_dir);\n-\telse {\n+\t\tmapped_refs = wanted_peer_refs(refs, refspec);\n+\t} else {\n \t\tstruct remote *remote = remote_get(argv[0]);\n \t\ttransport = transport_get(remote, remote->url[0]);\n \n@@ -520,14 +525,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t option_upload_pack);\n \n \t\trefs = transport_get_remote_refs(transport);\n-\t\tif (refs)\n-\t\t\ttransport_fetch_refs(transport, refs);\n+\t\tif (refs) {\n+\t\t\tmapped_refs = wanted_peer_refs(refs, refspec);\n+\t\t\ttransport_fetch_refs(transport, mapped_refs);\n+\t\t}\n \t}\n \n \tif (refs) {\n \t\tclear_extra_refs();\n \n-\t\tmapped_refs = write_remote_refs(refs, refspec, reflog_msg.buf);\n+\t\twrite_remote_refs(mapped_refs, reflog_msg.buf);\n \n \t\tremote_head = find_ref_by_name(refs, \"HEAD\");\n \t\tremote_head_points_at =\n"},{"id":"123833","messageId":"4ABD9C2C.60800@redhat.com","threadId":"20461","inReplyTo":"4ABD4F7B.4030701@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-26T04:44:28Z","receivedAt":"2009-09-26T04:44:28Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"Incidentally, somewhat related to this issue, I've noticed that if I \nfetch a branch which I don't currently have in my repository, and I have \nmost of the commits on that branch in my object store (or in an \nalternate repository) but not the most recent commit, git fetch isn't \nsmart enough to only grab the commits I'm actually missing, it wants to \nfetch much more.\n\nI would expect that since the clone pulled down everything in the \ngcc.git repository, I could then do\n\ngit config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'\ngit fetch\n\nand have all the branches, not just the ones in refs/heads. But when I \ndo this git fetch wants to fetch some 500k redundant objects.\n\nJason\n"},{"id":"123834","messageId":"m2iqf62mu4.fsf@whitebox.home","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0909252314260.4997@xanadu.home","subject":"Re: [PATCH] make 'git clone' ask the remote only for objects it cares about","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2009-09-26T07:21:07Z","receivedAt":"2009-09-26T07:21:07Z","isPatch":true,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Nicolas Pitre writes:\n\n> The fact that the git repository on gcc.gnu.org has lots of stuff in \n> \"remote\" branches that don't get cloned by default is a separate \n> configuration/policy issue on that server which might need (or not) to \n> be looked into. For instance at least, as a bare repository, it should \n> have all the git files in gcc.git/ directly instead of gcc.git/.git/.\n\nThe remote is just a git-svn tree.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"123842","messageId":"4ABE1818.6010007@redhat.com","threadId":"20461","inReplyTo":"4ABD9C2C.60800@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-26T13:33:12Z","receivedAt":"2009-09-26T13:33:12Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"On 09/26/2009 12:44 AM, Jason Merrill wrote:\n> git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'\n> git fetch\n\ngit count-objects -v before:\n\ncount: 44\nsize: 1768\nin-pack: 1399509\npacks: 1\nsize-pack: 600456\nprune-packable: 0\ngarbage: 0\n\nand after (transferred 278MB):\n\ncount: 44\nsize: 1768\nin-pack: 1947339\npacks: 2\nsize-pack: 1178408\nprune-packable: 8\ngarbage: 0\n\nand then after git gc --prune=now:\n\ncount: 0\nsize: 0\nin-pack: 1399613\npacks: 1\nsize-pack: 839900\nprune-packable: 0\ngarbage: 0\n\nSo I only actually needed 104 more objects, but fetch wasn't clever \nenough to see that, and my new pack is much less efficient.\n\nI've run into the same issue using alternates to set up multiple working \ndirectories for different branches; if the alternate directory isn't \ncompletely up-to-date, fetch wants to pull down lots of data again \nrather than use what I have and only fetch the last one or two commits.\n\nJason\n"},{"id":"123859","messageId":"20090926195039.GG14660@spearce.org","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0909252314260.4997@xanadu.home","subject":"Re: [PATCH] make 'git clone' ask the remote only for objects it cares about","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-09-26T19:50:39Z","receivedAt":"2009-09-26T19:50:39Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> Current behavior of 'git clone' when not using --mirror is to fetch \n> everything from the peer, and then filter out unwanted refs just before \n> writing them out to the cloned repository. This may become highly \n> inefficient if the peer has an unusual ref namespace, or if it simply \n> has \"remotes\" refs of its own, and those locally unwanted refs are \n> connecting to a large set of objects which becomes unreferenced as soon \n> as they are fetched.\n...\n> +static void write_remote_refs(const struct ref *local_refs, const char *reflog)\n\nHere reflog is now unused. I'm going to squash this in.\n\ndiff --git a/builtin-clone.c b/builtin-clone.c\nindex edf7c7f..4992c25 100644\n--- a/builtin-clone.c\n+++ b/builtin-clone.c\n@@ -342,7 +342,7 @@ static struct ref *wanted_peer_refs(const struct ref *refs,\n \treturn local_refs;\n }\n \n-static void write_remote_refs(const struct ref *local_refs, const char *reflog)\n+static void write_remote_refs(const struct ref *local_refs)\n {\n \tconst struct ref *r;\n \n@@ -534,7 +534,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs) {\n \t\tclear_extra_refs();\n \n-\t\twrite_remote_refs(mapped_refs, reflog_msg.buf);\n+\t\twrite_remote_refs(mapped_refs);\n \n \t\tremote_head = find_ref_by_name(refs, \"HEAD\");\n \t\tremote_head_points_at =\n\n-- \nShawn.\n"},{"id":"123877","messageId":"alpine.LFD.2.00.0909262023070.4997@xanadu.home","threadId":"20461","inReplyTo":"20090926195039.GG14660@spearce.org","subject":"Re: [PATCH] make 'git clone' ask the remote only for objects it cares about","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-27T00:26:49Z","receivedAt":"2009-09-27T00:26:49Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 26 Sep 2009, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > Current behavior of 'git clone' when not using --mirror is to fetch \n> > everything from the peer, and then filter out unwanted refs just before \n> > writing them out to the cloned repository. This may become highly \n> > inefficient if the peer has an unusual ref namespace, or if it simply \n> > has \"remotes\" refs of its own, and those locally unwanted refs are \n> > connecting to a large set of objects which becomes unreferenced as soon \n> > as they are fetched.\n> ...\n> > +static void write_remote_refs(const struct ref *local_refs, const char *reflog)\n> \n> Here reflog is now unused. I'm going to squash this in.\n\nYeah, I noticed. Since I didn't know what was the original intent for \nit, I just left it there.\n\n\nNicolas\n"},{"id":"123879","messageId":"alpine.LFD.2.00.0909262059520.4997@xanadu.home","threadId":"20461","inReplyTo":"4ABD9C2C.60800@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-27T01:27:13Z","receivedAt":"2009-09-27T01:27:13Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 26 Sep 2009, Jason Merrill wrote:\n\n> Incidentally, somewhat related to this issue, I've noticed that if I fetch a\n> branch which I don't currently have in my repository, and I have most of the\n> commits on that branch in my object store (or in an alternate repository) but\n> not the most recent commit, git fetch isn't smart enough to only grab the\n> commits I'm actually missing, it wants to fetch much more.\n> \n> I would expect that since the clone pulled down everything in the gcc.git\n> repository, I could then do\n> \n> git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'\n> git fetch\n> \n> and have all the branches, not just the ones in refs/heads. But when I do\n> this git fetch wants to fetch some 500k redundant objects.\n\nWell... Assuming a fixed git using the patch I posted yesterday, my \nclone of gcc.git has 988941 objects. The source repository used for the \nclone has 1399551 objects. Of course the source repo has more objects \nbecause it has extra branches in the refs/remotes/ namespace that the \nclone didn't fetch. If you wish to also fetch those branches as you \nillustrated above then you'll get the difference i.e. 410610 additional \nobjects.\n\nAnd even if the broken clone (before my patch) did pull everything from \ngcc.git, in the cloned repository those 410610 extra objects are \nconsidered as garbage because nothing actually reference them. So even \nif you decide to fetch the extra branches that the initial clone didn't \npick up, or if you do reference that repository with \"garbage\" objects \nfor another clone to which you want to add those extra branches, git has \nno way to know that it already had access to those objects locally and \n\"ungarbage\" them as they aren't referenced. Result is a useless fetch \nof 410610 objects that you already have, but that you weren't supposed \nto have in the first place.\n\n\nNicolas\n"},{"id":"123880","messageId":"20090927020409.GK14660@spearce.org","threadId":"20461","inReplyTo":"alpine.LFD.2.00.0909262059520.4997@xanadu.home","subject":"Re: git clone sending unneeded objects","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-09-27T02:04:10Z","receivedAt":"2009-09-27T02:04:10Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> And even if the broken clone (before my patch) did pull everything from \n> gcc.git, in the cloned repository those 410610 extra objects are \n> considered as garbage because nothing actually reference them. So even \n> if you decide to fetch the extra branches that the initial clone didn't \n> pick up, or if you do reference that repository with \"garbage\" objects \n> for another clone to which you want to add those extra branches, git has \n> no way to know that it already had access to those objects locally and \n> \"ungarbage\" them as they aren't referenced. Result is a useless fetch \n> of 410610 objects that you already have, but that you weren't supposed \n> to have in the first place.\n\nJust to clarify a minor nit:\n\nActually, if those refs have not changed, quickfetch should kick in\nand realize that all 410610 objects are reachable locally without\nerrors, permitting the client to avoid the object transfer.\n\nHowever, if *ANY* of those refs were to change to something you\ndon't actually have, quickfetch would fail, and we would need to\nfetch all 410610 objects.\n\n-- \nShawn.\n"},{"id":"123881","messageId":"alpine.LFD.2.00.0909262140280.4997@xanadu.home","threadId":"20461","inReplyTo":"4ABE1818.6010007@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-27T02:26:32Z","receivedAt":"2009-09-27T02:26:32Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 26 Sep 2009, Jason Merrill wrote:\n\n> On 09/26/2009 12:44 AM, Jason Merrill wrote:\n> > git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'\n> > git fetch\n> \n> git count-objects -v before:\n> \n> count: 44\n> size: 1768\n> in-pack: 1399509\n> packs: 1\n> size-pack: 600456\n> prune-packable: 0\n> garbage: 0\n\nI'm sure if you had done 'git rev-list --all --objects | wc -l' at that \npoint, the result would have been something around 900000. That's the \nactual number of objects git had a reference to, compared to the total \nobjects contained in the object store.\n\n> and after (transferred 278MB):\n> \n> count: 44\n> size: 1768\n> in-pack: 1947339\n> packs: 2\n> size-pack: 1178408\n> prune-packable: 8\n> garbage: 0\n\nAnd those 500000 extra objects or so (minus a couple dozens which were \nprobably used to \"complete\" the fetched thin pack and are duplicates of \nlocal objects -- the fetch progress message gave the exact number) were \nobtained from the remote repository because git has no way to tell the \nremote it already had them. That's what I was explaining in my previous \nemail.\n\n> and then after git gc --prune=now:\n> \n> count: 0\n> size: 0\n> in-pack: 1399613\n> packs: 1\n> size-pack: 839900\n> prune-packable: 0\n> garbage: 0\n> \n> So I only actually needed 104 more objects, but fetch wasn't clever enough to\n> see that, and my new pack is much less efficient.\n\nLike I said, it's not that the fetch wasn't clever enough. Rather that \nyour initial clone asked for way too many objects in the first place. \nThat's what my patch fixed.\n\nNow the pack efficiency can be explained as well. A single pack is \nalways going to be more efficient than 2 packs. Problem is when you do \na gc, by default git does the least costly operation which consists of \ncopying as much data from existing packs without extra processing. \nThat means that many objects were copied from the second (newly \nreceived) pack although a better delta representation was most probably \navailable in the other larger pack (remember that most objects from that \nsecond pack already existed in the first pack). Git do select the \nsecond pack in preference to the other pack because it is more recent, \nand normally more recent packs contains more recent objects which is a \ngood heuristic to optimizes the object enumeration. In this case this \ndidn't produce a good result, but again we're talking about a scenario \nwhich is bogus from the start and shouldn't be.\n\nSo if you do a 'git gc --aggressive' and let it run for a while, you \nshould get back a smaller pack, possibly even much smaller than the \noriginal \none.\n\n\nNicolas\n"},{"id":"123882","messageId":"alpine.LFD.2.00.0909262228210.4997@xanadu.home","threadId":"20461","inReplyTo":"20090927020409.GK14660@spearce.org","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-27T02:31:44Z","receivedAt":"2009-09-27T02:31:44Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 26 Sep 2009, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > And even if the broken clone (before my patch) did pull everything from \n> > gcc.git, in the cloned repository those 410610 extra objects are \n> > considered as garbage because nothing actually reference them. So even \n> > if you decide to fetch the extra branches that the initial clone didn't \n> > pick up, or if you do reference that repository with \"garbage\" objects \n> > for another clone to which you want to add those extra branches, git has \n> > no way to know that it already had access to those objects locally and \n> > \"ungarbage\" them as they aren't referenced. Result is a useless fetch \n> > of 410610 objects that you already have, but that you weren't supposed \n> > to have in the first place.\n> \n> Just to clarify a minor nit:\n> \n> Actually, if those refs have not changed, quickfetch should kick in\n> and realize that all 410610 objects are reachable locally without\n> errors, permitting the client to avoid the object transfer.\n> \n> However, if *ANY* of those refs were to change to something you\n> don't actually have, quickfetch would fail, and we would need to\n> fetch all 410610 objects.\n\nRight. But since we're talking about a git mirror for the gcc svn repo \nand gcc is a rather active project, the likelyhood of any ref to change \nat any time is rather high.\n\n\nNicolas\n"},{"id":"123883","messageId":"4ABEEB92.1020307@redhat.com","threadId":"20461","inReplyTo":"20090927020409.GK14660@spearce.org","subject":"Re: git clone sending unneeded objects","fromName":"Jason Merrill","fromEmail":"jason@redhat.com","sentAt":"2009-09-27T04:35:30Z","receivedAt":"2009-09-27T04:35:30Z","isPatch":false,"sender":{"key":"jason@redhat.com","avatar":"https://avatars.githubusercontent.com/u/266146?v=4"},"body":"On 09/26/2009 10:04 PM, Shawn O. Pearce wrote:\n> Actually, if those refs have not changed, quickfetch should kick in\n> and realize that all 410610 objects are reachable locally without\n> errors, permitting the client to avoid the object transfer.\n>\n> However, if *ANY* of those refs were to change to something you\n> don't actually have, quickfetch would fail, and we would need to\n> fetch all 410610 objects.\n\nRight. That seems unfortunate to me; couldn't fetch do a bit more \nchecking before it decides to download the whole world again?\n\nJason\n"},{"id":"123915","messageId":"alpine.LFD.2.00.0909280009250.4997@xanadu.home","threadId":"20461","inReplyTo":"4ABEEB92.1020307@redhat.com","subject":"Re: git clone sending unneeded objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2009-09-28T04:18:07Z","receivedAt":"2009-09-28T04:18:07Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 27 Sep 2009, Jason Merrill wrote:\n\n> On 09/26/2009 10:04 PM, Shawn O. Pearce wrote:\n> > Actually, if those refs have not changed, quickfetch should kick in\n> > and realize that all 410610 objects are reachable locally without\n> > errors, permitting the client to avoid the object transfer.\n> > \n> > However, if *ANY* of those refs were to change to something you\n> > don't actually have, quickfetch would fail, and we would need to\n> > fetch all 410610 objects.\n> \n> Right. That seems unfortunate to me; couldn't fetch do a bit more checking\n> before it decides to download the whole world again?\n\nThe quickfetch test could be turned into a filter so refs that are \nalready available locally could simply not be fetched on a per ref \nbasis. But that would be a rather expensive test which couldn't keep \nits \"quick\" qualifier anymore, and so for a case that shouldn't have \nhappened normally anyway if git didn't have a bug with its clone \noperation as I've explained already.\n\n\nNicolas\n"}]}