{"thread":{"id":"52698","subject":"git-clone --single-branch clones objects outside of branch","startedAt":"2020-01-26T12:40:13Z","lastAt":"2020-01-29T02:23:57Z","messageCount":6,"participants":["Chris Jerdonek","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"390510","messageId":"CAOTb1wfoMf338VPRYcjAv+EaMYQxKJ=kP6-qMykG-LfxPrWuag@mail.gmail.com","threadId":"52698","inReplyTo":null,"subject":"git-clone --single-branch clones objects outside of branch","fromName":"Chris Jerdonek","fromEmail":"chris.jerdonek@gmail.com","sentAt":"2020-01-26T12:39:52Z","receivedAt":"2020-01-26T12:40:13Z","isPatch":false,"sender":{"key":"chris.jerdonek@gmail.com","avatar":"https://gravatar.com/avatar/86b2cff752baeaabcfaf26ff30fd8f74c5caf8657fdb7990dc3c787d542f26a3?d=mp&s=160"},"body":"Hi,\n\nI'm reporting some git-clone behavior regarding --single-branch that I\nfound unexpected after reading the docs. I'm using git 2.25.0.\n\nThe git-clone docs for --single-branch say:\n\n> Clone only the history leading to the tip of a single branch, either specified by the --branch option or the primary branch remote’s HEAD points at.\n\n(from: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---no-single-branch\n)\n\nHowever, when I attempted this with a local repo, I found that objects\nlocated only in branches other than the branch I specified are also\ncloned. Also, this is true even if the remote repo has only loose\nobjects (i.e. no pack files). So it doesn't appear to be doing this\nonly to avoid creating new files.\n\nIn contrast, git-fetch behaves as expected (including locally).\ngit-fetch appears to fetch only objects in the given branch, when a\nbranch is specified.\n\nBelow are some commands to assist with reproducing this situation (but\nyou will need to update the path in the `git-remote add` invocation\nbelow). At the least, it seems like the docs should clarify the\nbehavior. (The Python commands were for when I was doing some\nexperiments with pack files.)\n\nmkdir repo1\ncd repo1\ngit init\npython -c \"print('a\\n' + 10 * 'x\\n')\" > a.txt\ngit add a.txt\ngit commit -m \"add a\"\n# Get object id to check existence with `git cat-file -t` below.\ngit hash-object a.txt\n\ngit checkout -b dev\npython -c \"print('b\\n' + 10 * 'x\\n')\" > b.txt\ngit add b.txt\ngit commit -m \"add b\"\n# Get object id to check existence with `git cat-file -t` below.\ngit hash-object b.txt\n\ngit checkout master\n\ncd ..\nmkdir repo2\ncd repo2\ngit init\ngit remote add other file:///<path-to-repo1>\ngit fetch other master\n\ncd ..\ngit clone --branch master --single-branch repo1 repo3\n\nThanks,\n--Chris\n"},{"id":"390531","messageId":"20200127055509.GA12108@coredump.intra.peff.net","threadId":"52698","inReplyTo":"CAOTb1wfoMf338VPRYcjAv+EaMYQxKJ=kP6-qMykG-LfxPrWuag@mail.gmail.com","subject":"Re: git-clone --single-branch clones objects outside of branch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-27T05:55:09Z","receivedAt":"2020-01-27T05:55:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 26, 2020 at 04:39:52AM -0800, Chris Jerdonek wrote:\n\n> However, when I attempted this with a local repo, I found that objects\n> located only in branches other than the branch I specified are also\n> cloned. Also, this is true even if the remote repo has only loose\n> objects (i.e. no pack files). So it doesn't appear to be doing this\n> only to avoid creating new files.\n> \n> In contrast, git-fetch behaves as expected (including locally).\n> git-fetch appears to fetch only objects in the given branch, when a\n> branch is specified.\n\nThis is the expected outcome, because in your example you're cloning on\nthe local filesystem. By default that enables some optimizations, one of\nwhich is to hard-link the object files into the destination repository.\nThat avoids the cost of copying and re-hashing them (which a normal\ncross-system clone would do). And it even avoids traversing the objects\nto find which are necessary, instead just hard-linking everything.\n\nSo with:\n\n> git clone --branch master --single-branch repo1 repo3\n\nYou should be able to see with \"git for-each-ref\" in repo3 that you only\ngot the \"master\" branch, but not \"dev\". But those extra objects are\navailable to you, because of the hard-links.\n\nIf you do:\n\n  git clone --branch master --single-branch --no-local repo1 repo4\n\nthen repo4 will not have the objects (we really will send a packfile\nacross a pipe, just as we would across the network for a cross-system\nclone).\n\n> cd repo2\n> git init\n> git remote add other file:///<path-to-repo1>\n> git fetch other master\n\nThis one behaves as you expected because git-fetch does not perform the\nsame optimizations (it wouldn't make as much sense there, as generally\nin a fetch we already have most of the objects from the other side\nanyway, so hard-linking would just give us duplicates).\n\n-Peff\n"},{"id":"390533","messageId":"CAOTb1wd9D3YytevTt0cGnw1o-9cN1-yxCqbuH4oLH1KB6mzEeA@mail.gmail.com","threadId":"52698","inReplyTo":"20200127055509.GA12108@coredump.intra.peff.net","subject":"Re: git-clone --single-branch clones objects outside of branch","fromName":"Chris Jerdonek","fromEmail":"chris.jerdonek@gmail.com","sentAt":"2020-01-27T06:46:07Z","receivedAt":"2020-01-27T06:46:22Z","isPatch":false,"sender":{"key":"chris.jerdonek@gmail.com","avatar":"https://gravatar.com/avatar/86b2cff752baeaabcfaf26ff30fd8f74c5caf8657fdb7990dc3c787d542f26a3?d=mp&s=160"},"body":"On Sun, Jan 26, 2020 at 9:55 PM Jeff King <peff@peff.net> wrote:\n> On Sun, Jan 26, 2020 at 04:39:52AM -0800, Chris Jerdonek wrote:\n> > However, when I attempted this with a local repo, I found that objects\n> > located only in branches other than the branch I specified are also\n> > cloned. Also, this is true even if the remote repo has only loose\n> > objects (i.e. no pack files). So it doesn't appear to be doing this\n> > only to avoid creating new files.\n>\n> This is the expected outcome, because in your example you're cloning on\n> the local filesystem. By default that enables some optimizations, one of\n> which is to hard-link the object files into the destination repository.\n> That avoids the cost of copying and re-hashing them (which a normal\n> cross-system clone would do). And it even avoids traversing the objects\n> to find which are necessary, instead just hard-linking everything.\n\nThanks for the reply. It's okay for that to be the expected behavior.\nMy suggestion would just be that the documentation for --single-branch\nbe updated to clarify that objects unreachable from the specified\nbranch can still be in the cloned repo when run using the --local\noptimizations. For example, it can matter for security if one is\ntrying to create a clone of a repo that doesn't include data from\nbranches with sensitive info (e.g. in following Git's advice to create\na separate repo if security of private data is desired:\nhttps://git-scm.com/docs/gitnamespaces#_security ).\n\nI'm guessing other flags also don't apply when --local is being used.\nFor example, I'm guessing --reference is also ignored when using\n--local, but I haven't checked yet to confirm. It would be nice if the\ndocumentation gave a heads up in cases like these. Even if hard links\nare being used, it's not clear from the docs whether the objects are\nfiltered first, prior to hard linking, when flags like --single-branch\nand --reference are passed.\n\n> This one behaves as you expected because git-fetch does not perform the\n> same optimizations (it wouldn't make as much sense there, as generally\n> in a fetch we already have most of the objects from the other side\n> anyway, so hard-linking would just give us duplicates).\n\nIncidentally, here's a thread from 2010 requesting that this\noptimization be available in the git-fetch case:\nhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=573909\n(I don't know how reports on that Debian list relate to this list.)\n\n--Chris\n"},{"id":"390618","messageId":"20200128094801.GC574544@coredump.intra.peff.net","threadId":"52698","inReplyTo":"CAOTb1wd9D3YytevTt0cGnw1o-9cN1-yxCqbuH4oLH1KB6mzEeA@mail.gmail.com","subject":"Re: git-clone --single-branch clones objects outside of branch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-28T09:48:01Z","receivedAt":"2020-01-28T09:48:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 26, 2020 at 10:46:07PM -0800, Chris Jerdonek wrote:\n\n> Thanks for the reply. It's okay for that to be the expected behavior.\n> My suggestion would just be that the documentation for --single-branch\n> be updated to clarify that objects unreachable from the specified\n> branch can still be in the cloned repo when run using the --local\n> optimizations. For example, it can matter for security if one is\n> trying to create a clone of a repo that doesn't include data from\n> branches with sensitive info (e.g. in following Git's advice to create\n> a separate repo if security of private data is desired:\n> https://git-scm.com/docs/gitnamespaces#_security ).\n\nI think it would make sense to talk about that under \"--local\". There\nare other subtle reasons you might want \"--no-local\", too: it will\nperform more consistency checks, which could be valuable if you're\nthinking about deleting the old copy.\n\nI'm not sure how much Git guarantees in general that you won't get extra\nobjects. It's true that we try to avoid sending objects that aren't\nneeded, but it's mostly as an optimization. If we later modified\npack-objects to sometimes send unneeded objects (say, because we're able\nto compute the set more efficiently if we use an approximation that errs\non the conservative side), then I think that's something we'd consider.\nAnd I suppose it's already possible with the dumb-http protocol, which\nhas to fetch whole packfiles. And the same would be true of recently\nproposed schemes to clients to a pre-generated packfile URL.\n\n> I'm guessing other flags also don't apply when --local is being used.\n> For example, I'm guessing --reference is also ignored when using\n> --local, but I haven't checked yet to confirm. It would be nice if the\n> documentation gave a heads up in cases like these. Even if hard links\n> are being used, it's not clear from the docs whether the objects are\n> filtered first, prior to hard linking, when flags like --single-branch\n> and --reference are passed.\n\nNo, \"--reference\" behaves as usual. However, \"--depth\" is ignored (and\nissues a warning). I don't think it would be wrong to issue a warning\nwhen --single-branch is used locally (though it would not be \"single\nbranch is ignored, since it does impact which refs are copied). But I\nkind of wonder if it would be annoying for people who don't care about\nhaving the extra objects reachable.\n\n> > This one behaves as you expected because git-fetch does not perform the\n> > same optimizations (it wouldn't make as much sense there, as generally\n> > in a fetch we already have most of the objects from the other side\n> > anyway, so hard-linking would just give us duplicates).\n> \n> Incidentally, here's a thread from 2010 requesting that this\n> optimization be available in the git-fetch case:\n> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=573909\n> (I don't know how reports on that Debian list relate to this list.)\n\nSometimes they get forwarded here, and sometimes not. :)\n\nI think there are subtle issues that make naively using the optimization\na bad idea, as it could actually backfire and cause more disk usage.\nE.g., consider a sequence like this:\n\n  1. Repo A has a 100MB packfile.\n\n  2. \"git clone A B\" uses hardlinks. Now we have two copies of the repo,\n     storing 100MB.\n\n  3. Repo A adds a few more commits, and then does a \"git gc\", breaking\n     the hardlinks. Now we have ~200MB used. We're no worse off than if\n     we hadn't done the hardlinks in the first place.\n\n  4. Repo B fetches from A. It wants the new commits, but they're in\n     repo A's big packfile. So it hardlinks that. We have ~200MB in repo\n     B, but half of that is hardlinked and shared with A. So we're still\n     using ~200MB. So far so good.\n\n  5. Repo A repacks again, breaking the hardlinks. Now it's using\n     ~100MB, but repo B is still using ~300MB. We're worse off than we\n     would be without the optimization.\n\nIf you really want to keep sharing objects over time, I think using\n\"clone -s\" is a better choice (though it comes with its own\ncomplications and dangers, too; see the git-clone documentation).\n\n-Peff\n"},{"id":"390679","messageId":"CAOTb1wecnwXrOqJa-dq1zyS1ydgQ-c_b_GX7FkWzJPtGk6xL3g@mail.gmail.com","threadId":"52698","inReplyTo":"20200128094801.GC574544@coredump.intra.peff.net","subject":"Re: git-clone --single-branch clones objects outside of branch","fromName":"Chris Jerdonek","fromEmail":"chris.jerdonek@gmail.com","sentAt":"2020-01-29T01:59:54Z","receivedAt":"2020-01-29T02:03:18Z","isPatch":false,"sender":{"key":"chris.jerdonek@gmail.com","avatar":"https://gravatar.com/avatar/86b2cff752baeaabcfaf26ff30fd8f74c5caf8657fdb7990dc3c787d542f26a3?d=mp&s=160"},"body":"On Tue, Jan 28, 2020 at 1:48 AM Jeff King <peff@peff.net> wrote:\n> On Sun, Jan 26, 2020 at 10:46:07PM -0800, Chris Jerdonek wrote:\n> > I'm guessing other flags also don't apply when --local is being used.\n> > For example, I'm guessing --reference is also ignored when using\n> > --local, but I haven't checked yet to confirm. It would be nice if the\n> > documentation gave a heads up in cases like these. Even if hard links\n> > are being used, it's not clear from the docs whether the objects are\n> > filtered first, prior to hard linking, when flags like --single-branch\n> > and --reference are passed.\n>\n> No, \"--reference\" behaves as usual.\n\nOn this, I found that --reference does behave differently in the way\nthat I suspected. For example, when run with the default --local, I\nfound that git-clone will create hard links in the new repo to loose\nobjects, even if those objects already exist in the reference\nrepository. When run with --non-local, the objects in the reference\nrepository weren't copied (I didn't find them in the cloned repo's\npack file).\n\nSo in addition to --single-branch, this seems to be another case where\n`git-clone --local` will ignore the provided options when deciding\nwhat files inside .git/objects/ to hard-link. It just hard-links\neverything. This is another example of something that I think would be\nworth mentioning in the docs in some form. Currently, the\ndocumentation for --reference suggests that objects won't be created\nin the new repo if they already exist in the reference repository.\n\n--Chris\n\n> However, \"--depth\" is ignored (and\n> issues a warning). I don't think it would be wrong to issue a warning\n> when --single-branch is used locally (though it would not be \"single\n> branch is ignored, since it does impact which refs are copied). But I\n> kind of wonder if it would be annoying for people who don't care about\n> having the extra objects reachable.\n"},{"id":"390681","messageId":"20200129022354.GA596285@coredump.intra.peff.net","threadId":"52698","inReplyTo":"CAOTb1wecnwXrOqJa-dq1zyS1ydgQ-c_b_GX7FkWzJPtGk6xL3g@mail.gmail.com","subject":"Re: git-clone --single-branch clones objects outside of branch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-29T02:23:54Z","receivedAt":"2020-01-29T02:23:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 28, 2020 at 05:59:54PM -0800, Chris Jerdonek wrote:\n\n> > No, \"--reference\" behaves as usual.\n> \n> On this, I found that --reference does behave differently in the way\n> that I suspected. For example, when run with the default --local, I\n> found that git-clone will create hard links in the new repo to loose\n> objects, even if those objects already exist in the reference\n> repository. When run with --non-local, the objects in the reference\n> repository weren't copied (I didn't find them in the cloned repo's\n> pack file).\n\nSure, but I'd just consider that the same issue: hardlinking takes\neverything in the object database without regard to what's needed. So\n--reference still works (and when you repack and break the hardlinks, it\nwill drop any duplicates as usual), but obviously you have access to the\nextra hardlinked objects in the meantime.\n\nIOW, I don't think it's worth calling this out in the --reference\ndocumentation, but could be in the --local documentation. Because the\nunusual implication is the same whether --reference is used or not.\n\n> So in addition to --single-branch, this seems to be another case where\n> `git-clone --local` will ignore the provided options when deciding\n> what files inside .git/objects/ to hard-link. It just hard-links\n> everything. This is another example of something that I think would be\n> worth mentioning in the docs in some form. Currently, the\n> documentation for --reference suggests that objects won't be created\n> in the new repo if they already exist in the reference repository.\n\nIt says:\n\n  Using an already existing repository as an alternate will require\n  fewer objects to be copied from the repository being cloned, reducing\n  network and local storage costs.\n\nwhich I think is still technically true. The hardlinks are free in terms\nof storage (and there's no network cost by definition). So at worst\n--reference is doing nothing to start with. And the repository has been\nsetup in such a way that it may yield benefits later (after you repack\nand the hardlinks are broken).\n\nThis is maybe splitting hairs a bit. The thing that matters, I think, is\nwhere a documentation change should go (and what it should say).\n\n-Peff\n"}]}