{"thread":{"id":"40670","subject":"git-fetch pulls already-pulled objects?","startedAt":"2015-10-28T23:28:24Z","lastAt":"2015-10-29T19:52:27Z","messageCount":5,"participants":["Matt Glazar","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"272472","messageId":"D256A718.1373A%strager@fb.com","threadId":"40670","inReplyTo":null,"subject":"git-fetch pulls already-pulled objects?","fromName":"Matt Glazar","fromEmail":"strager@fb.com","sentAt":"2015-10-28T23:28:24Z","receivedAt":"2015-10-28T23:28:24Z","isPatch":false,"sender":{"key":"strager@fb.com","avatar":null},"body":"On a remote, I have two Git commit objects which point to the same tree\nobject (created with git commit-tree). If I fetch one of the commits, the\ncommit object (including the tree object) is fetched. If I then fetch the\nother commit, the tree object (and its dependencies) is fetched *again* (I\nthink). I don't watch the tree object downloaded again, because it is\nlarge (multi-gigabyte). Because the tree object exists locally, I think it\nshould not be downloaded.\n\nIs this a bug in Git, or is this by design? How can I confirm that the\ntree object (and dependencies) are downloaded twice? Is there are more\ncomplicated git-fetch (or similar) command I can execute to not download\nthe already-downloaded tree objects? (I have the hash of the tree object\nwhich would be potentially re-downloaded, if that helps.)\n\nSequence of commands to reproduce:\n\n# Replace this with the URL to an empty Git repository.\nremote=ssh://foo/bar.git\n\n# Create some random data to exaggerate git-fetch times.\n# If you have a slow remote, reduce 'count'.\nmkdir minimal\ncd minimal\ndd if=/dev/urandom of=random bs=65536 count=4096\n\n# Create our two commits (master and master2).\ngit init\ngit add random\ngit commit -m 'Random data (commit 1)'\ngit branch master2 \\\n \"$(echo 'Random data (commit 2)' \\\n | git commit-tree 'HEAD^{tree}')\"\n\n# Push our commits. Expected to take some time.\ngit remote add origin \"${remote}\"\ngit push origin \\\n master:refs/heads/master \\\n master2:refs/heads/master2\n\n# Clone master. Expected to take some time.\ncd ..\nmkdir minimal-clone\ngit clone --single-branch --branch master \"${remote}\"\n\n# Fetch master2. Should be nearly instant, but takes some\n# time. Seems to be download everything again.\ncd minimal-clone\ngit fetch origin master2\n\n# Try again. git-fetch takes a while, but shouldn't.\nrm -f .git/FETCH_HEAD\ngit gc --prune=all\ngit fetch origin master2\n\nInfo about my system:\n\n\nLocal (pusher):\nOS: OS X 10.10.5\ngit: git version 2.0.1\nssh: OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011\n\n\nRemote (server):\nOS: Linux 4.0.9 (CentOS 6)\ngit: git version 2.4.6\nsshd: OpenSSH_6.7p1-hpn14v5, OpenSSL 1.0.1e-fips 11 Feb 2013\n\n"},{"id":"272488","messageId":"xmqqd1vxshrd.fsf@gitster.mtv.corp.google.com","threadId":"40670","inReplyTo":"D256A718.1373A%strager@fb.com","subject":"Re: git-fetch pulls already-pulled objects?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-29T17:32:06Z","receivedAt":"2015-10-29T17:32:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Glazar writes:\n\n> On a remote, I have two Git commit objects which point to the same tree\n> object (created with git commit-tree).\n\nWhat you are expecting _could_ be implemented by exchanging all\ntree and blob objects sending and receiving sides have and computing\nthe set difference, but the sender and the receiver do not exchange\nsuch a huge list.\n\nThe object transfer is done by first finding the common ancestor of\nhistories of the sending and the receiving sides, which allows the\nsender to enumerate commits that the sender has but the receiver\ndoesn't. From there, all objects [*1*] that are referenced by these\ncommits that need to be sent.\n\n\n[Footnote]\n\n*1* There is an optimization to exclude the trees and blobs that can\nbe cheaply proven to exist on the receiving end. If the receiving\nend has a commit that the sending end does *not* have, and that\ncommit happens to record a tree the sending end needs to send,\nhowever, the sending end cannot prove that the tree does not have to\nbe sent without first fetching that commit from the receiving end,\nwhich fails \"can be cheaply proven to exist\" test.\n"},{"id":"272492","messageId":"D257AC36.1376F%strager@fb.com","threadId":"40670","inReplyTo":"xmqqd1vxshrd.fsf@gitster.mtv.corp.google.com","subject":"Re: git-fetch pulls already-pulled objects?","fromName":"Matt Glazar","fromEmail":"strager@fb.com","sentAt":"2015-10-29T18:08:44Z","receivedAt":"2015-10-29T18:08:44Z","isPatch":false,"sender":{"key":"strager@fb.com","avatar":null},"body":"> What you are expecting _could_ be implemented by exchanging all\n> tree and blob objects sending and receiving sides have and computing\n> the set difference, but the sender and the receiver do not exchange\n> such a huge list.\n\nIn my case, I only want to exchange the tree object hash pointed directly\nby the commit object; I don't care about all subtrees and blobs reachable\nfrom the commit. I think a naive approach would only double the number of\nhashes sent worst case.\n\nWould negotiating the tree object hashes be possible on the client without\nserver changes? Is the protocol that flexible?\n\n\nIf what I want is *not* possible, is it possible to explicitly put a tree\n(and its descendants) into its own pack? I think that will speed up the\ngit-fetch a bit by doing this on the server. (I know what trees/commits\nwill be sent ahead of time.) (The server does less work pulling the\nobjects out of an existing pack and repacking them for the client. (Or\nmaybe my mental model of git packs is wrong?))\n\n> The object transfer is done by first finding the common ancestor of\n> histories of the sending and the receiving sides, which allows the\n> sender to enumerate commits that the sender has but the receiver\n> doesn't. From there, all objects [*1*] that are referenced by these\n> commits that need to be sent.\n\nThanks for clarifying.\n\n> *1* There is an optimization to exclude the trees and blobs that can\n> be cheaply proven to exist on the receiving end.\n\nThat makes sense (especially for 'git revert HEAD' situations).\n\nThank you for your reply, Junio.\n\n-----Original Message-----\nFrom: Junio C Hamano \nDate: Thursday, October 29, 2015 at 10:32 AM\nTo: Matt Glazer \nCc: \"git@vger.kernel.org\" \nSubject: Re: git-fetch pulls already-pulled objects?\n\n>Matt Glazar writes:\n>\n>> On a remote, I have two Git commit objects which point to the same tree\n>> object (created with git commit-tree).\n>\n>What you are expecting _could_ be implemented by exchanging all\n>tree and blob objects sending and receiving sides have and computing\n>the set difference, but the sender and the receiver do not exchange\n>such a huge list.\n>\n>The object transfer is done by first finding the common ancestor of\n>histories of the sending and the receiving sides, which allows the\n>sender to enumerate commits that the sender has but the receiver\n>doesn't. From there, all objects [*1*] that are referenced by these\n>commits that need to be sent.\n>\n>\n>[Footnote]\n>\n>*1* There is an optimization to exclude the trees and blobs that can\n>be cheaply proven to exist on the receiving end. If the receiving\n>end has a commit that the sending end does *not* have, and that\n>commit happens to record a tree the sending end needs to send,\n>however, the sending end cannot prove that the tree does not have to\n>be sent without first fetching that commit from the receiving end,\n>which fails \"can be cheaply proven to exist\" test.\n>\n\n"},{"id":"272494","messageId":"xmqqwpu5qzxs.fsf@gitster.mtv.corp.google.com","threadId":"40670","inReplyTo":"D257AC36.1376F%strager@fb.com","subject":"Re: git-fetch pulls already-pulled objects?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-29T18:42:23Z","receivedAt":"2015-10-29T18:42:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Glazar writes:\n\n> Would negotiating the tree object hashes be possible on the client without\n> server changes? Is the protocol that flexible?\n\nThe protocol is strictly \"find common ancestor in the commit\nhistory\". Everything else is done on the sender.\n\n>>The object transfer is done by first finding the common ancestor of\n>>histories of the sending and the receiving sides, which allows the\n>>sender to enumerate commits that the sender has but the receiver\n>>doesn't. From there, all objects [*1*] that are referenced by these\n>>commits that need to be sent.\n\n>>[Footnote]\n>>\n>>*1* There is an optimization to exclude the trees and blobs that can\n>>be cheaply proven to exist on the receiving end. If the receiving\n>>end has a commit that the sending end does *not* have, and that\n>>commit happens to record a tree the sending end needs to send,\n>>however, the sending end cannot prove that the tree does not have to\n>>be sent without first fetching that commit from the receiving end,\n>>which fails \"can be cheaply proven to exist\" test.\n\nI forgot to mention the recent \"pack bitmap\" addition. It makes the\nset of \"can be cheaply proven to exist\" a lot larger.\n\nIf for example the sender needs to send one commit C because it\ndetermined that the receiver has history up to commit C~1, without\nthe bitmap, even when C^{tree} (i.e. the tree of C) is identical to\nC~2^{tree} (i.e. the tree of C~2), it would have sent that tree\nobject because \"proving that the receiver already has it\" would\nrequire the sender to dig its history back, starting from C~1\n(i.e. the commit that is known to exist at the receiver), to\nenumerate the objects contained in the common part of the history,\nwhich fails the \"can be cheaply proven to exist\" test.\n\nThe \"pack bitmap\" pre-computes what commits, trees and blobs should\nalready exist in the repository given a commit for which bitmap\nexists. Using the bitmap, from C~1 (i.e. the commit known to exist\nat the receiving end), it can be proven cheaply that C^{tree} that\nhappens to be identical to C~2^{tree} already exists over there, and\nthe sender can use this knowledge to reduce the transfer.\n\nThe \"pack bitmap\" however does not change the fundamental structure.\nIf your receiver has a commit that is not known to the sender, and\nthat commit happens to record the same tree recorded in the commit\nthat needs to be sent, there is no way for the sender to know that\nthe receiver has it, exactly because the exchange between them is\npurely \"find common ancestor in history\".\n"},{"id":"272496","messageId":"D257C4CB.1378A%strager@fb.com","threadId":"40670","inReplyTo":"xmqqwpu5qzxs.fsf@gitster.mtv.corp.google.com","subject":"Re: git-fetch pulls already-pulled objects?","fromName":"Matt Glazar","fromEmail":"strager@fb.com","sentAt":"2015-10-29T19:52:27Z","receivedAt":"2015-10-29T19:52:27Z","isPatch":false,"sender":{"key":"strager@fb.com","avatar":null},"body":"> I forgot to mention the recent \"pack bitmap\" addition. It makes the\n> set of \"can be cheaply proven to exist\" a lot larger.\n\nCool! I tried this feature, and it worked! (At least, it worked for my\nsmall test case.)\n\nI ran on the server (after pushing the objects):\n\ngit config repack.writeBitmaps true\ngit repack -Ad\n\nAfter this, the 'git fetch origin master2' was super quick.\n\nThanks for your help!\n\nAside: This test case is using (normal, C/sh) Git. My production\nenvironment uses JGit on the server. I haven't tested this with JGit.\n\n-----Original Message-----\nFrom: Junio C Hamano \nDate: Thursday, October 29, 2015 at 11:42 AM\nTo: Matt Glazer \nCc: \"git@vger.kernel.org\" \nSubject: Re: git-fetch pulls already-pulled objects?\n\n>Matt Glazar writes:\n>\n>> Would negotiating the tree object hashes be possible on the client\n>>without\n>> server changes? Is the protocol that flexible?\n>\n>The protocol is strictly \"find common ancestor in the commit\n>history\". Everything else is done on the sender.\n>\n>>>The object transfer is done by first finding the common ancestor of\n>>>histories of the sending and the receiving sides, which allows the\n>>>sender to enumerate commits that the sender has but the receiver\n>>>doesn't. From there, all objects [*1*] that are referenced by these\n>>>commits that need to be sent.\n>\n>>>[Footnote]\n>>>\n>>>*1* There is an optimization to exclude the trees and blobs that can\n>>>be cheaply proven to exist on the receiving end. If the receiving\n>>>end has a commit that the sending end does *not* have, and that\n>>>commit happens to record a tree the sending end needs to send,\n>>>however, the sending end cannot prove that the tree does not have to\n>>>be sent without first fetching that commit from the receiving end,\n>>>which fails \"can be cheaply proven to exist\" test.\n>\n>I forgot to mention the recent \"pack bitmap\" addition. It makes the\n>set of \"can be cheaply proven to exist\" a lot larger.\n>\n>If for example the sender needs to send one commit C because it\n>determined that the receiver has history up to commit C~1, without\n>the bitmap, even when C^{tree} (i.e. the tree of C) is identical to\n>C~2^{tree} (i.e. the tree of C~2), it would have sent that tree\n>object because \"proving that the receiver already has it\" would\n>require the sender to dig its history back, starting from C~1\n>(i.e. the commit that is known to exist at the receiver), to\n>enumerate the objects contained in the common part of the history,\n>which fails the \"can be cheaply proven to exist\" test.\n>\n>The \"pack bitmap\" pre-computes what commits, trees and blobs should\n>already exist in the repository given a commit for which bitmap\n>exists. Using the bitmap, from C~1 (i.e. the commit known to exist\n>at the receiving end), it can be proven cheaply that C^{tree} that\n>happens to be identical to C~2^{tree} already exists over there, and\n>the sender can use this knowledge to reduce the transfer.\n>\n>The \"pack bitmap\" however does not change the fundamental structure.\n>If your receiver has a commit that is not known to the sender, and\n>that commit happens to record the same tree recorded in the commit\n>that needs to be sent, there is no way for the sender to know that\n>the receiver has it, exactly because the exchange between them is\n>purely \"find common ancestor in history\".\n\n"}]}