{"thread":{"id":"17467","subject":"'git clone' doesn't use alternates automatically?","startedAt":"2009-01-30T22:12:42Z","lastAt":"2009-02-03T06:06:08Z","messageCount":13,"participants":["James Pickens","Jeff King","Jakub Narebski","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"102647","messageId":"885649360901301412jd5c6b0dne1eff1ff00dd043e@mail.gmail.com","threadId":"17467","inReplyTo":null,"subject":"'git clone' doesn't use alternates automatically?","fromName":"James Pickens","fromEmail":"jepicken@gmail.com","sentAt":"2009-01-30T22:12:42Z","receivedAt":"2009-01-30T22:12:42Z","isPatch":false,"sender":{"key":"jepicken@gmail.com","avatar":null},"body":"Hi,\n\nI have a central, shared Git repository on an NFS drive at path\n$central. I have added \"$central/objects\" to\n$central/objects/info/alternates. I see that when I clone this\nrepository with Git 1.6.1, the alternates file is automatically copied\nto the clone, but so are all the pack files and loose objects. If I\nthen cd to the clone and run 'git gc', it removes the redundant local\nobjects.\n\nI thought I tested this setup a few months back, and 'git clone'\nautomatically used the alternates file to avoid copying the redundant\nobjects into the clone. Has this behavior changed, or is my memory\nbad?\n\nJames\n"},{"id":"102682","messageId":"20090131071238.GC3033@coredump.intra.peff.net","threadId":"17467","inReplyTo":"885649360901301412jd5c6b0dne1eff1ff00dd043e@mail.gmail.com","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-01-31T07:12:38Z","receivedAt":"2009-01-31T07:12:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 30, 2009 at 03:12:42PM -0700, James Pickens wrote:\n\n> I have a central, shared Git repository on an NFS drive at path\n> $central. I have added \"$central/objects\" to\n> $central/objects/info/alternates. I see that when I clone this\n> repository with Git 1.6.1, the alternates file is automatically copied\n> to the clone, but so are all the pack files and loose objects. If I\n> then cd to the clone and run 'git gc', it removes the redundant local\n> objects.\n\nYes, we don't set up alternates to an origin by default. If it's a local\nclone, we do hardlink by default:\n\n $ ls -i git/.git/objects/pack\n 7639155 pack-0651ae7e35ffde1921db158a3292e1c81153be1a.idx\n 7638782 pack-0651ae7e35ffde1921db158a3292e1c81153be1a.pack\n $ git clone git foo\n ...\n $ ls -i foo/.git/objects/pack\n 7639155 pack-0651ae7e35ffde1921db158a3292e1c81153be1a.idx\n 7638782 pack-0651ae7e35ffde1921db158a3292e1c81153be1a.pack\n\nbut presumably in your example the second clone is _not_ on the NFS\nmount, and therefore can't hardlink.\n\nSo you can try \"git clone -s\" to specify that you definitely want\nalternates.\n\n> I thought I tested this setup a few months back, and 'git clone'\n> automatically used the alternates file to avoid copying the redundant\n> objects into the clone. Has this behavior changed, or is my memory\n> bad?\n\nI don't recall clone ever being that clever, but I could be wrong (it is\nnot an area of the code that I am too familiar with).\n\nCan you try a test with a few different versions to see if it ever\nbehaved as you expected (and if it does, bisect to find the breakage)?\n\n-Peff\n"},{"id":"102708","messageId":"885649360901311208s4bc17ae3me2062b07b302291e@mail.gmail.com","threadId":"17467","inReplyTo":"20090131071238.GC3033@coredump.intra.peff.net","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"James Pickens","fromEmail":"jepicken@gmail.com","sentAt":"2009-01-31T20:08:16Z","receivedAt":"2009-01-31T20:08:16Z","isPatch":false,"sender":{"key":"jepicken@gmail.com","avatar":null},"body":"On Sat, Jan 31, 2009 at 12:12 AM, Jeff King wrote:\n> but presumably in your example the second clone is _not_ on the NFS\n> mount, and therefore can't hardlink.\n\nThat's correct.\n\n> So you can try \"git clone -s\" to specify that you definitely want\n> alternates.\n\nWell, the clone gets the alternates either way. It just doesn't\nuse them to avoid copying the data unless I give -s. More\nimportantly, if 'git clone' worked the way I thought, then when I\nclone a remote repository for which I have a local mirror, I\ncould avoid typing '--reference ' by adding\n/objects to the alternates file in the\nremote repository.\n\n> I don't recall clone ever being that clever, but I could be wrong (it is\n> not an area of the code that I am too familiar with).\n>\n> Can you try a test with a few different versions to see if it ever\n> behaved as you expected (and if it does, bisect to find the breakage)?\n\nDamn. I was hoping the response would be \"it's a regression, and\nhere's a patch to fix it\". I went ahead and tested a few old\nversions and they all behave the same way.\n\nSo, is there any reason 'git clone' shouldn't automatically use\nthe alternates that it copied into the new repository? I might\nlook into writing a patch if nobody objects.\n\nJames\n"},{"id":"102712","messageId":"m3tz7fw5ek.fsf@localhost.localdomain","threadId":"17467","inReplyTo":"885649360901311208s4bc17ae3me2062b07b302291e@mail.gmail.com","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-01-31T21:08:02Z","receivedAt":"2009-01-31T21:08:02Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"James Pickens writes:\n\n> So, is there any reason 'git clone' shouldn't automatically use\n> the alternates that it copied into the new repository? I might\n> look into writing a patch if nobody objects.\n\nAlternates are fragile with respect to garbage collecting in the\nrepository you borrow objects from.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"102714","messageId":"885649360901311343r2a9bd1b5l5f8226218000dbe8@mail.gmail.com","threadId":"17467","inReplyTo":"m3tz7fw5ek.fsf@localhost.localdomain","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"James Pickens","fromEmail":"jepicken@gmail.com","sentAt":"2009-01-31T21:43:13Z","receivedAt":"2009-01-31T21:43:13Z","isPatch":false,"sender":{"key":"jepicken@gmail.com","avatar":null},"body":"On Sat, Jan 31, 2009 at 2:08 PM, Jakub Narebski wrote:\n> James Pickens writes:\n>\n>> So, is there any reason 'git clone' shouldn't automatically use\n>> the alternates that it copied into the new repository? I might\n>> look into writing a patch if nobody objects.\n>\n> Alternates are fragile with respect to garbage collecting in the\n> repository you borrow objects from.\n\nI think that's irrelevant in this case. The scenario is that I\nclone repo A, which is borrowing objects from repo B. So repo A\nwas already assuming that it's safe to borrow from B.\n\nThe current behavior is that the clone of A also borrows from B\nautomatically. What I am asking is whether 'git clone' should\ntake advantage of that to avoid copying redundant objects from A\ninto the clone. They will get deleted the first time I run 'git\ngc' in the clone anyways.\n\nJames\n"},{"id":"102723","messageId":"20090131215514.GB9415@coredump.intra.peff.net","threadId":"17467","inReplyTo":"885649360901311208s4bc17ae3me2062b07b302291e@mail.gmail.com","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-01-31T21:55:14Z","receivedAt":"2009-01-31T21:55:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 31, 2009 at 01:08:16PM -0700, James Pickens wrote:\n\n> Well, the clone gets the alternates either way. It just doesn't\n> use them to avoid copying the data unless I give -s. More\n\nThe other key change is that you don't depend on the origin in your\nalternates when you don't use \"-s\".\n\n> So, is there any reason 'git clone' shouldn't automatically use\n> the alternates that it copied into the new repository? I might\n> look into writing a patch if nobody objects.\n\nI think the reason \"-s\" isn't the default is that alternates are fragile\n(as Jakub mentioned), and we don't want ot set them up without the user\nasking to do so.\n\nSo from what you've posted (but I haven't double checked or looked at\nthe code), it sounds like the current behavior is:\n\n - with \"-s\", add the origin as an alternate, and use alternates while\n cloning\n\n - \"with --reference\", add some other repo as an alternate, and use\n alternates while cloning\n\n - without either, copy alternates from origin, but _don't_ use\n alternates while cloning\n\nThe last one seems a little silly. Why bother setting up the alternates\nif you're not going to use them? I guess because we might not be able to\nget the objects at all, otherwise, and we need to know where to copy\nthem from. But either:\n\n - that is an implementation-specific detail of clone, and those\n alternates should go away after we clone\n\n or\n\n - we should fully respect those alternates\n\nThe only downside to the latter is that now somebody who has cloned a\nrepository with alternates now has an alternates-based repository and\nmight not know it (i.e., they might have been the one who set up\nalternates in the origin).\n\n-Peff\n"},{"id":"102729","messageId":"7vtz7f2cy6.fsf@gitster.siamese.dyndns.org","threadId":"17467","inReplyTo":"885649360901311208s4bc17ae3me2062b07b302291e@mail.gmail.com","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-01T00:55:13Z","receivedAt":"2009-02-01T00:55:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"James Pickens writes:\n\n> So, is there any reason 'git clone' shouldn't automatically use\n> the alternates that it copied into the new repository?\n\nWhen you say \"git clone\" without -s, you are saying \"I do not want to use\nthe repository I am cloning from as my alternate, because I do not know if\nwill stay stable. I do not trust it.\"\n\nThis would be a very sensible way to clone, if you were cloning my\nrepository whose 'pu' and its constituent topic branches are subject to\nrewinding at any time. After I rebase some of the branches and rebuild\n'pu', and prune the unnecessary objects from my repository, the objects\nyou may have been borrowing from me will be gone from my repository. Of\ncourse, I can remove my repository altogether any time, and when that\nhappens, your repository will have many missing objects.\n\nThat is why \"-s\" is not the default.\n\nOnly when you positively know that the other repository will not drop\nbranches or rewind them, perhaps because you control that repository\nyourself, it is safe to use it as your alternate, and you use commands\nlike \"git clone -s\" and/or \"git clone --reference\" to do so.\n\n Side note. People on k.org are encouraged to use Linus's\n repository as an alternate to save space on the k.org machine,\n because it is known that Linus's repository will never rewind its\n branches.\n\nNow, if you are cloning from a local filesystem, by default we will copy\nthe objects/info/alternates from the source repository to the new one. It\nmay be debatable if this is a sensible thing to do. On one hand, because\nyou are saying you don't trust if the objects in the source repository\nwill stay stable by not giving \"-s\", it might be sensible not to trust its\nchoice of alternates either. But in such a case, you can always use file://\nURL when cloning to get a full freestanding copy.\n\nI suspect you are trying to improve the other extreme end: trusting all\nthe other repositories involved in the cloning process a lot more than the\ncode currently does.\n\nI do not think it is a bad thing to do per-se.\n\nI haven't looked at the codepaths involved recently, but if I recall\ncorrectly, optimizing of cloning from a repository that uses alternates\nitself was never a part of the initial design considerations. I suspect\nthere may be an ample room for you to optimize things.\n"},{"id":"102732","messageId":"7v7i4b2bto.fsf@gitster.siamese.dyndns.org","threadId":"17467","inReplyTo":"20090131215514.GB9415@coredump.intra.peff.net","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-01T01:19:31Z","receivedAt":"2009-02-01T01:19:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> - without either, copy alternates from origin, but _don't_ use\n> alternates while cloning\n\nAre you talking about a local clone optimization that does hardlink from\nthe source repository?\n\nI am fairly certain that copying alternates from the source repository was\nnot an intended behaviour but was a consequence of lazy coding of how we\ncopy (or link) everything from it. The original was literally the simple\nmatter of:\n\n find objects ! -type d -print | cpio $cpio_quiet_flag -pumd$l \"$GIT_DIR/\"\n\nwhose intention was to copy objects/?? and objects/pack/. and it wasn't\neven part of the design consideration to worry about what would happen to\nthe alternates the source repository might have in objects/info/.\n"},{"id":"102736","messageId":"885649360901311732t6e4f8651p6c3955f018b9a36c@mail.gmail.com","threadId":"17467","inReplyTo":"7vtz7f2cy6.fsf@gitster.siamese.dyndns.org","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"James Pickens","fromEmail":"jepicken@gmail.com","sentAt":"2009-02-01T01:32:06Z","receivedAt":"2009-02-01T01:32:06Z","isPatch":false,"sender":{"key":"jepicken@gmail.com","avatar":null},"body":"On Sat, Jan 31, 2009, Junio C Hamano wrote:\n> When you say \"git clone\" without -s, you are saying \"I do not want to use\n> the repository I am cloning from as my alternate, because I do not know if\n> will stay stable. I do not trust it.\"\n\nYes, I'm aware of the caveats of -s. I was talking about what\nhappens when I *don't* use -s.\n\n> Now, if you are cloning from a local filesystem, by default we will copy\n> the objects/info/alternates from the source repository to the new one. It\n\nCrap, I didn't realize the alternates were only copied when you\nclone from the local filesystem. I wanted to use this when\ncloning over ssh from site A to site B, to automatically add a\nmirror at site B as an alternate. Sounds like I have no choice\nbut to use --reference for that.\n\n> I suspect you are trying to improve the other extreme end: trusting all\n> the other repositories involved in the cloning process a lot more than the\n> code currently does.\n\nWhat I was suggesting did not involve trusting anything any more\nthan the current code does. It just meant taking immediate\nadvantage of the trust that was already there.\n\nThanks for your input,\nJames\n"},{"id":"102740","messageId":"7vocxnylzz.fsf@gitster.siamese.dyndns.org","threadId":"17467","inReplyTo":"885649360901311732t6e4f8651p6c3955f018b9a36c@mail.gmail.com","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-01T01:38:40Z","receivedAt":"2009-02-01T01:38:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"James Pickens writes:\n\n> ... Sounds like I have no choice\n> but to use --reference for that.\n\nAs --reference was invented exactly for that use case, I think using it to\ninstruct where to borrow your object from is a very sensible thing to do.\n"},{"id":"102854","messageId":"20090202130755.GA8487@sigio.peff.net","threadId":"17467","inReplyTo":"7v7i4b2bto.fsf@gitster.siamese.dyndns.org","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-02-02T13:07:55Z","receivedAt":"2009-02-02T13:07:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 31, 2009 at 05:19:31PM -0800, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > - without either, copy alternates from origin, but _don't_ use\n> > alternates while cloning\n> \n> Are you talking about a local clone optimization that does hardlink from\n> the source repository?\n\nSorry, I was wrong about what was happening. From reading James' posts\nand not doing any experimenting or looking, I had the impression that\ndoing this:\n\n # plain repo\n mkdir repo1 &&\n (cd repo1 && git init &&\n echo content >file && git add . && git commit -m one)\n\n # repo with alternates, but extra content\n git clone -s repo1 repo2 &&\n (cd repo2 &&\n echo content >>file && git commit -a -m two)\n\n # clone of repo w/ alternates\n git clone repo2 repo3\n\nwould cause the final clone to set up the alternate to repo1, but still\npull in the objects. But that isn't the case, of course. Either:\n\n 1. It is a local hardlink clone, in which case we just pull in the\n objects from repo2.\n\n 2. It isn't, in which case we don't copy over the alternates.\n\n> I am fairly certain that copying alternates from the source repository was\n> not an intended behaviour but was a consequence of lazy coding of how we\n> copy (or link) everything from it. The original was literally the simple\n> matter of:\n> \n> find objects ! -type d -print | cpio $cpio_quiet_flag -pumd$l \"$GIT_DIR/\"\n> \n> whose intention was to copy objects/?? and objects/pack/. and it wasn't\n> even part of the design consideration to worry about what would happen to\n> the alternates the source repository might have in objects/info/.\n\nRight, I think that is what is going on. And what I was suggesting in my\nother email is that it is actively harmful to have this behavior,\nbecause now repo3 depends on repo1, without the user having explicitly\nasked for such a relationship (and they might not even be aware of\nrepo1).\n\nI was tempted to suggest avoiding copying the alternates from repo2\nto repo3. But you can't do that: repo2 is _missing_ objects that repo3\nwon't have. Without the alternates file pointing to repo1, repo3 is\ncorrupt. So simply avoiding copying the alternates file doesn't work;\none would have to actually pull the missing objects in from the\nalternate before doing so.\n\nBut actually, I think there is even more breakage in hardlinking the\nalternates file: alternates files can be relative paths. So if repo2\npoints to \"../../../repo1/.git/objects\" (which it doesn't in the example\nabove, as \"clone -s\" uses absolute paths -- but it is easy enough to\nconstruct a broken case), then repo3 will gain that alternate pointer,\nbut may be in a totally different directory where that relative path is\nbroken. And then repo3 is corrupt. So the alternates must be copied and\nany relative paths munged for it to work reliably.\n\nThe hardlink code operates by default because it was thought to be a\nsafe optimization that couldn't bite people. But it interacts badly with\nthe concept of alternates. So I think a sane fix would be to disable\nhardlinking if the parent repo is using alternates at all. Then a\nvanilla \"git clone repo2 repo3\" will do the safe but more costly\nbehavior of actually copying the objects. If the user wants to accept\nthe risks of alternates, then he can give \"-s\" explicitly, and git will\ntrack the alternates recursively through repo2 to repo1 at runtime.\n\n-Peff\n"},{"id":"102913","messageId":"7v7i48f8gj.fsf@gitster.siamese.dyndns.org","threadId":"17467","inReplyTo":"20090202130755.GA8487@sigio.peff.net","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-03T04:30:36Z","receivedAt":"2009-02-03T04:30:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> The hardlink code operates by default because it was thought to be a\n> safe optimization that couldn't bite people. But it interacts badly with\n> the concept of alternates.\n\nYes, you are right.\n\nTo be fair, I think it was proposed/implemented by somebody who almost\nnever uses alternates himself, and certainly never a relative alternates.\nThe intention of hardlinking was while saving tons of disk space, still be\nindependent from the original repository.\n\nBack when e95ab1e ([PATCH] Short-circuit git-clone-pack while cloning\nlocally (take 2)., 2005-07-06) was done, the packfile implementation was\nstill only a week old, and hardlinking made a lot of sense from space\nsaving's point of view. These days, if you make a local hardlinked clone,\nwork a little there and then repack it, most of the space saving will be\ngone; there isn't much point in the hardlink optimization anymore from\nthat angle, even though it still is a good compromise between the clone\nspeed and safety, especially when no alternates are involved.\n\nI think a possible fix would be not to copy alternates file literally, but\ninstall an alternates file to directly borrow from the same repositories\nthe clone-source repository borrows from ourselves, taking relative paths\ninto account. Another would be to look at the alternates and hardlink the\nobjects and packs while cloning, and if the repositories involved reside\nacross filesystem boundaries, we need to fall back to copying.\n"},{"id":"102923","messageId":"20090203060608.GA21367@sigill.intra.peff.net","threadId":"17467","inReplyTo":"7v7i48f8gj.fsf@gitster.siamese.dyndns.org","subject":"Re: 'git clone' doesn't use alternates automatically?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-02-03T06:06:08Z","receivedAt":"2009-02-03T06:06:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 02, 2009 at 08:30:36PM -0800, Junio C Hamano wrote:\n\n> To be fair, I think it was proposed/implemented by somebody who almost\n> never uses alternates himself, and certainly never a relative alternates.\n> The intention of hardlinking was while saving tons of disk space, still be\n> independent from the original repository.\n> \n> Back when e95ab1e ([PATCH] Short-circuit git-clone-pack while cloning\n> locally (take 2)., 2005-07-06) was done, the packfile implementation was\n\nWell, in your defense, relative alternates didn't come about until two\nmonths later, in ccfd3e9. So you can blame the author of that patch for\nscrewing up your existing work. :)\n\n> still only a week old, and hardlinking made a lot of sense from space\n> saving's point of view. These days, if you make a local hardlinked clone,\n> work a little there and then repack it, most of the space saving will be\n> gone; there isn't much point in the hardlink optimization anymore from\n> that angle, even though it still is a good compromise between the clone\n> speed and safety, especially when no alternates are involved.\n\nTrue. But I still think the hardlinks are nice for one-off repositories.\nEvery once in a while I want to start a new topic or experiment while my\nrepository is a mess; it's nice to just \"git clone git foo\", hack around\nin the work directory, and blow it away. And the hardlinks make that\nfirst step a _lot_ faster.\n\nBut I also don't mind having to add a command-line option to get the\nspeed. And for my use case, there really isn't a benefit to hardlinks\nover alternates.\n\n> I think a possible fix would be not to copy alternates file literally, but\n> install an alternates file to directly borrow from the same repositories\n> the clone-source repository borrows from ourselves, taking relative paths\n> into account. Another would be to look at the alternates and hardlink the\n> objects and packs while cloning, and if the repositories involved reside\n> across filesystem boundaries, we need to fall back to copying.\n\nYes, either of those would work. But I wonder if it is really worth the\ncomplexity. When I suggested just ditching hardlinks if the remote uses\nalternates, my thought was that most people won't really care. Either\nthey use alternates, in which case they should be providing \"-s\" and\nnot doing hardlinks, or they don't, in which case things will happen as\nusual.\n\nBut reading your response, I wonder if it is worth keeping the hardlink\noptimization around at all; getting rid of it would simplify the code\nand the explanation of why \"git clone foo\" is different from \"git clone\nfile://$PWD/foo\". If people want a fast, dependent clone, they can use\n\"-s\". I guess hardlinks are also useful for a fast \"git clone foo bar;\nrm -rf foo\". But I'm not sure how common that is.\n\n-Peff\n"}]}