{"thread":{"id":"3421","subject":"How to not download objects more than needed?","startedAt":"2006-02-21T19:38:42Z","lastAt":"2006-02-22T21:19:42Z","messageCount":10,"participants":["Radoslaw Szkodzinski","sean","Junio C Hamano","Linus Torvalds","Jan Harkes","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"16516","messageId":"43FB6C42.5000208@gorzow.mm.pl","threadId":"3421","inReplyTo":null,"subject":"How to not download objects more than needed?","fromName":"Radoslaw Szkodzinski","fromEmail":"astralstorm@gorzow.mm.pl","sentAt":"2006-02-21T19:38:42Z","receivedAt":"2006-02-21T19:38:42Z","isPatch":false,"sender":{"key":"astralstorm@gorzow.mm.pl","avatar":null},"body":"I have a pecuilar, but common use case for git.\n\nI have linux-2.6 repository pulled and I'd like to download some branch\n(say, netdev-2.6), which uses many of the same objects,\nbut not to get all the objects from the git server.\n\nI've already tried certain commands, but still can't do it,\nand my bandwidth isn't too happy about it.\n\nIt seems to require some kind of HEAD rewinding,\nor maybe fetching to another branch, I don't know.\n\nAnyone cares to help?\n\n-- \nGPG Key id: 0xD1F10BA2\nFingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\n\nAstralStorm\n\n"},{"id":"16524","messageId":"BAYC1-PASMTP03A58A4F389365AC85DA68AEFC0@CEZ.ICE","threadId":"3421","inReplyTo":"43FB6C42.5000208@gorzow.mm.pl","subject":"Re: How to not download objects more than needed?","fromName":"sean","fromEmail":"seanlkml@sympatico.ca","sentAt":"2006-02-21T21:13:40Z","receivedAt":"2006-02-21T21:13:40Z","isPatch":false,"sender":{"key":"seanlkml@sympatico.ca","avatar":"https://gravatar.com/avatar/f92923f54fc08c401fc59b71829d4b89e9b8087fbba45ff87c82e6a83aee02ae?d=mp&s=160"},"body":"On Tue, 21 Feb 2006 20:38:42 +0100\nRadoslaw Szkodzinski wrote:\n\n> I have a pecuilar, but common use case for git.\n\nIt's not really that peculiar.\n\n> I have linux-2.6 repository pulled and I'd like to download some branch\n> (say, netdev-2.6), which uses many of the same objects,\n> but not to get all the objects from the git server.\n\nJust make sure you're not using the rsync protocol. Using the\nnative git protocol would be best.\n\n> I've already tried certain commands, but still can't do it,\n> and my bandwidth isn't too happy about it.\n\nFor instance, make sure your current linus repository is up to date \nwith a \"git pull\" and then:\n\ngit fetch \\\n git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/netdev-2.6.git \\\n upstream:netdev\n\nwill take the \"upstream\" branch from the netdev repository and name it \nnetdev in your local repository.\n\nSean\n"},{"id":"16525","messageId":"7vek1wwg8d.fsf@assigned-by-dhcp.cox.net","threadId":"3421","inReplyTo":"43FB6C42.5000208@gorzow.mm.pl","subject":"Re: How to not download objects more than needed?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-21T21:32:02Z","receivedAt":"2006-02-21T21:32:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Radoslaw Szkodzinski writes:\n\n> I have a pecuilar, but common use case for git.\n>\n> I have linux-2.6 repository pulled and I'd like to download some branch\n> (say, netdev-2.6), which uses many of the same objects,\n> but not to get all the objects from the git server.\n>\n> I've already tried certain commands, but still can't do it,\n> and my bandwidth isn't too happy about it.\n>\n> It seems to require some kind of HEAD rewinding,\n> or maybe fetching to another branch, I don't know.\n>\n> Anyone cares to help?\n\nIt is not peculiar at all. The tools already should do what you\nwant:\n\n o---o---o---...---o (netdev-2.6)\n /\n / < netdev forked some time ago.\n /\n ---o---o---o---o---...---o---o---o (linus tip)\n ^v2.6.16-rc3 ^v2.6.16-rc4 \n\nSuppose the \"global\" ancestry graph was like the above. And\nnetdev-2.6 has not been merged into Linus tree.\n\nWhat you have, already pulled from Linus, is:\n\n ---o---o---o---o---...---o---o---o (linus tip)\n ^v2.6.16-rc3 ^v2.6.16-rc4 \n\nAnd suppose what the netdev tree has is something like this:\n\n o---o---o---...---o (netdev-2.6)\n /\n / < netdev forked some time ago.\n /\n ---o---o---o\n ^v2.6.16-rc3\n\nThe point being that the netdev tree does not know about Linus\ntip you have.\n\nWhen you \"git fetch git://.../netdev-2.6.git/\", a program that\nruns on your end (git-fetch-pack) and another program that runs\non the other end (git-upload-pack) discuss to find out what both\nof you have in common. Your side starts from Linus tip and go\nbackwards, telling the other end \"I have this, I have that,\n...\". At first, netdev side will not see what it knows about,\nbut after a while, it will see a commit both of you have\n(i.e. where the branch forked from). After they find that out,\nyour side tells the other side \"I want your netdev-2.6 head\".\n\nThe other side sends the objects needed to complete the chain up\nto the requested head, assuming that your side has objects to\ncomplete the common ancestor point (again, the fork point, but\nit could be some revs after that if the graph looked like the\nabove picture). Objects behind the fork point does not need to\nbe sent.\n"},{"id":"16547","messageId":"Pine.LNX.4.64.0602211635450.30245@g5.osdl.org","threadId":"3421","inReplyTo":"BAYC1-PASMTP03A58A4F389365AC85DA68AEFC0@CEZ.ICE","subject":"Re: How to not download objects more than needed?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-02-22T00:42:34Z","receivedAt":"2006-02-22T00:42:34Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 21 Feb 2006, sean wrote:\n> \n> > I have linux-2.6 repository pulled and I'd like to download some branch\n> > (say, netdev-2.6), which uses many of the same objects,\n> > but not to get all the objects from the git server.\n> \n> Just make sure you're not using the rsync protocol. Using the\n> native git protocol would be best.\n\nSide note: the \"automatic tag following\" is broken wrt pulling unnecessary \nobjects, even with the git protocol.\n\nWhat happens is that if you don't explicitly have a branch for what you \nare pulling, and you do something like\n\n\tgit pull git://git.kernel.org/....\n\nand the automatic tag following kicks in, it will first have fetched the \nobjects once, and then when it tries to fetch the tag objects, it will \nfetch the objects it already fetched _again_ (plus the tags), because it \nwill do the same object pull, but the temporary branch (to be merged) will \nnever have been written as a branch head.\n\nSo you'll see something like\n\n\tGenerating pack...\n\tDone counting objects.\n\tPacking objects.......................\n\tUnpacking objects\n\t 100% (/) done\n\tAuto-following refs/tags/v1.2.2\n\tGenerating pack...\n\tDone counting objects.\n\tPacking objects.......................\n\tUnpacking objects\n\t 100% (/) done\n\t* refs/tags/v1.2.2: storing tag 'v1.2.2' of master.kernel.org:/pub/scm/git/git\n\njust because we hadn't updated any refs before we started re-fetching more \nobjects.\n\nSo we do have cases where we fetch unnecessarily even with the native \nprotocol.\n\n\t\tLinus\n"},{"id":"16550","messageId":"20060222011338.GL5000@delft.aura.cs.cmu.edu","threadId":"3421","inReplyTo":"Pine.LNX.4.64.0602211635450.30245@g5.osdl.org","subject":"Re: How to not download objects more than needed?","fromName":"Jan Harkes","fromEmail":"jaharkes@cs.cmu.edu","sentAt":"2006-02-22T01:13:38Z","receivedAt":"2006-02-22T01:13:38Z","isPatch":false,"sender":{"key":"jaharkes@cs.cmu.edu","avatar":"https://gravatar.com/avatar/cf95aecd150ca8ef33d6edc337ac4bb9e13aa4246fc3679257d578c7fddc1633?d=mp&s=160"},"body":"On Tue, Feb 21, 2006 at 04:42:34PM -0800, Linus Torvalds wrote:\n> \n> \tgit pull git://git.kernel.org/....\n> \n> and the automatic tag following kicks in, it will first have fetched the \n> objects once, and then when it tries to fetch the tag objects, it will \n> fetch the objects it already fetched _again_ (plus the tags), because it \n> will do the same object pull, but the temporary branch (to be merged) will \n> never have been written as a branch head.\n\nIsn't this easily avoided by fetching the tags first?\n\nJan\n\n\ndiff --git a/git-fetch.sh b/git-fetch.sh\nindex b4325d9..9c6748f 100755\n--- a/git-fetch.sh\n+++ b/git-fetch.sh\n@@ -363,8 +363,6 @@ fetch_main () {\n \n }\n \n-fetch_main \"$reflist\"\n-\n # automated tag following\n case \"$no_tags$tags\" in\n '')\n@@ -389,6 +387,8 @@ case \"$no_tags$tags\" in\n \tesac\n esac\n \n+fetch_main \"$reflist\"\n+\n # If the original head was empty (i.e. no \"master\" yet), or\n # if we were told not to worry, we do not have to check.\n case \",$update_head_ok,$orig_head,\" in\n"},{"id":"16555","messageId":"7v3bicupgb.fsf@assigned-by-dhcp.cox.net","threadId":"3421","inReplyTo":"20060222011338.GL5000@delft.aura.cs.cmu.edu","subject":"Re: How to not download objects more than needed?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-22T01:55:48Z","receivedAt":"2006-02-22T01:55:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jan Harkes writes:\n\n> On Tue, Feb 21, 2006 at 04:42:34PM -0800, Linus Torvalds wrote:\n>> \n>> \tgit pull git://git.kernel.org/....\n>> \n>> and the automatic tag following kicks in, it will first have fetched the \n>> objects once, and then when it tries to fetch the tag objects, it will \n>> fetch the objects it already fetched _again_ (plus the tags), because it \n>> will do the same object pull, but the temporary branch (to be merged) will \n>> never have been written as a branch head.\n>\n> Isn't this easily avoided by fetching the tags first?\n\nI do not think so.\n\nNotice how the tag following code uses cat-file to determine if\nthe main fetch likely has slurped the object they point at.\n"},{"id":"16559","messageId":"20060222031136.GN5000@delft.aura.cs.cmu.edu","threadId":"3421","inReplyTo":"7v3bicupgb.fsf@assigned-by-dhcp.cox.net","subject":"Re: How to not download objects more than needed?","fromName":"Jan Harkes","fromEmail":"jaharkes@cs.cmu.edu","sentAt":"2006-02-22T03:11:36Z","receivedAt":"2006-02-22T03:11:36Z","isPatch":false,"sender":{"key":"jaharkes@cs.cmu.edu","avatar":"https://gravatar.com/avatar/cf95aecd150ca8ef33d6edc337ac4bb9e13aa4246fc3679257d578c7fddc1633?d=mp&s=160"},"body":"On Tue, Feb 21, 2006 at 05:55:48PM -0800, Junio C Hamano wrote:\n> Jan Harkes writes:\n> > On Tue, Feb 21, 2006 at 04:42:34PM -0800, Linus Torvalds wrote:\n> >> \n> >> \tgit pull git://git.kernel.org/....\n> >> \n> >> and the automatic tag following kicks in, it will first have fetched the \n> >> objects once, and then when it tries to fetch the tag objects, it will \n> >> fetch the objects it already fetched _again_ (plus the tags), because it \n> >> will do the same object pull, but the temporary branch (to be merged) will \n> >> never have been written as a branch head.\n> >\n> > Isn't this easily avoided by fetching the tags first?\n> \n> I do not think so.\n> \n> Notice how the tag following code uses cat-file to determine if\n> the main fetch likely has slurped the object they point at.\n\nNeat, it only fetches tags that refer to things we already have. Hadn't\nchecked what the automatic tag fetcher was doing.\n\nSo either introduce temporary local refs that can be removed once the\ntags have been fetched, or else fix it in fetch-pack with the following\nchange that might do the trick for this case as well. However that one\nalready got shot down because of possible consistency problems.\n\n http://marc.theaimsgroup.com/?l=git&m=113030081014456&w=2\n\nJan\n"},{"id":"16560","messageId":"7vacckt6vr.fsf@assigned-by-dhcp.cox.net","threadId":"3421","inReplyTo":"20060222031136.GN5000@delft.aura.cs.cmu.edu","subject":"Re: How to not download objects more than needed?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-22T03:22:16Z","receivedAt":"2006-02-22T03:22:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jan Harkes writes:\n\n> Neat, it only fetches tags that refer to things we already have. Hadn't\n> checked what the automatic tag fetcher was doing.\n>\n> So either introduce temporary local refs that can be removed once the\n> tags have been fetched,...\n\nI think it is enough just to disable tag following when you are\npromiscuously fetching. That is, do the tag following only if\nthe main fetch is going to store a ref because it has tracking\nbranch for the remote side. Otherwise the remote tags do not\nmatter and if you really care about them you can ask with --tags.\n"},{"id":"16586","messageId":"7vfymbm72x.fsf_-_@assigned-by-dhcp.cox.net","threadId":"3421","inReplyTo":"7vacckt6vr.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] git-fetch: follow tag only when tracking remote branch.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-22T21:12:06Z","receivedAt":"2006-02-22T21:12:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Unless --no-tags flag was given, git-fetch tried to always\nfollow remote tags that point at the commits we picked up.\n\nIt is not very useful to pick up tags from remote unless storing\nthe fetched branch head in a local tracking branch. This is\nespecially true if the fetch is done to merge the remote branch\ninto our current branch as one-shot basis (i.e. \"please pull\"),\nand is even harmful if the remote repository has many irrelevant\ntags.\n\nThis proposed update disables the automated tag following unless\nwe are storing the a fetched branch head in a local tracking\nbranch.\n\nSigned-off-by: Junio C Hamano \n\n---\n\n * Likes, dislikes?\n\n git-fetch.sh | 33 +++++++++++++++++++--------------\n 1 files changed, 19 insertions(+), 14 deletions(-)\n\nf41b73b4c2e0313d1638261ed87f0921e47904b2\ndiff --git a/git-fetch.sh b/git-fetch.sh\nindex b4325d9..fcc24f8 100755\n--- a/git-fetch.sh\n+++ b/git-fetch.sh\n@@ -368,20 +368,25 @@ fetch_main \"$reflist\"\n # automated tag following\n case \"$no_tags$tags\" in\n '')\n-\ttaglist=$(IFS=\" \" &&\n-\tgit-ls-remote $upload_pack --tags \"$remote\" |\n-\tsed -ne 's|^\\([0-9a-f]*\\)[ \t]\\(refs/tags/.*\\)^{}$|\\1 \\2|p' |\n-\twhile read sha1 name\n-\tdo\n-\t\ttest -f \"$GIT_DIR/$name\" && continue\n-\t \tgit-check-ref-format \"$name\" || {\n-\t\t\techo >&2 \"warning: tag ${name} ignored\"\n-\t\t\tcontinue\n-\t\t}\n-\t\tgit-cat-file -t \"$sha1\" >/dev/null 2>&1 || continue\n-\t\techo >&2 \"Auto-following $name\"\n-\t\techo \".${name}:${name}\"\n-\tdone)\n+\tcase \"$reflist\" in\n+\t*:refs/*)\n+\t\t# effective only when we are following remote branch\n+\t\t# using local tracking branch.\n+\t\ttaglist=$(IFS=\" \" &&\n+\t\tgit-ls-remote $upload_pack --tags \"$remote\" |\n+\t\tsed -ne 's|^\\([0-9a-f]*\\)[ \t]\\(refs/tags/.*\\)^{}$|\\1 \\2|p' |\n+\t\twhile read sha1 name\n+\t\tdo\n+\t\t\ttest -f \"$GIT_DIR/$name\" && continue\n+\t\t\tgit-check-ref-format \"$name\" || {\n+\t\t\t\techo >&2 \"warning: tag ${name} ignored\"\n+\t\t\t\tcontinue\n+\t\t\t}\n+\t\t\tgit-cat-file -t \"$sha1\" >/dev/null 2>&1 || continue\n+\t\t\techo >&2 \"Auto-following $name\"\n+\t\t\techo \".${name}:${name}\"\n+\t\tdone)\n+\tesac\n \tcase \"$taglist\" in\n \t'') ;;\n \t?*)\n-- \n1.2.2.ga35e\n"},{"id":"16587","messageId":"43FCD56E.5090804@op5.se","threadId":"3421","inReplyTo":"7vfymbm72x.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] git-fetch: follow tag only when tracking remote branch.","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-02-22T21:19:42Z","receivedAt":"2006-02-22T21:19:42Z","isPatch":true,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> Unless --no-tags flag was given, git-fetch tried to always\n> follow remote tags that point at the commits we picked up.\n> \n> It is not very useful to pick up tags from remote unless storing\n> the fetched branch head in a local tracking branch. This is\n> especially true if the fetch is done to merge the remote branch\n> into our current branch as one-shot basis (i.e. \"please pull\"),\n> and is even harmful if the remote repository has many irrelevant\n> tags.\n> \n> This proposed update disables the automated tag following unless\n> we are storing the a fetched branch head in a local tracking\n> branch.\n> \n> Signed-off-by: Junio C Hamano \n> \n> ---\n> \n> * Likes, dislikes?\n> \n\nLikes a lot. This is a Good Thing.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"}]}