{"thread":{"id":"15394","subject":"Alternates and push","startedAt":"2008-09-06T12:42:50Z","lastAt":"2008-09-09T14:57:14Z","messageCount":20,"participants":["Jon Smirl","Theodore Tso","Junio C Hamano","Shawn O. Pearce","Jan Hudec","Daniel Barkalow","Petr Baudis"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"89903","messageId":"9e4733910809060542s5ede6d6m5bdb894c958ea8b7@mail.gmail.com","threadId":"15394","inReplyTo":null,"subject":"Alternates and push","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2008-09-06T12:42:50Z","receivedAt":"2008-09-06T12:42:50Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"At github my repo, digispeaker, has an alternate pointing to github's\nlocal copy of Linus' tree. I ignored my tree for a month and then\npushed to it including 200MB of objects from Linus' tree. These 200MB\nof objects were pushed up to the server, but these objects were\nalready in the alternates repository.\n\nWhat's supposed to happen? Is something broken in github's setup, or\ndoes pushing not take into account alternates?\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"89910","messageId":"20080906162030.GT9129@mit.edu","threadId":"15394","inReplyTo":"9e4733910809060542s5ede6d6m5bdb894c958ea8b7@mail.gmail.com","subject":"Re: Alternates and push","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2008-09-06T16:20:30Z","receivedAt":"2008-09-06T16:20:30Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sat, Sep 06, 2008 at 08:42:50AM -0400, Jon Smirl wrote:\n> At github my repo, digispeaker, has an alternate pointing to github's\n> local copy of Linus' tree. I ignored my tree for a month and then\n> pushed to it including 200MB of objects from Linus' tree. These 200MB\n> of objects were pushed up to the server, but these objects were\n> already in the alternates repository.\n> \n> What's supposed to happen? Is something broken in github's setup, or\n> does pushing not take into account alternates?\n\nLong-standing mis-feature in git's logic in deciding what to push.\nIt's been reported a few times, but apparently it's hard to fix, or at\nleast it never hsa been fixed as far as I know.\n\nI work around it by ssh'ing into master.kernel.org and doing a \"git\nbranch -f origin <commit-id-of-master-on-linus's-tree>\".  As long as\nthere is one branch which is up-to-date, git will avoid pushing a huge\nnumber of objects to master.kernel.org.  Of course, this workaround\nonly works if you have shell access....\n\n\t\t\t\t\t\t- Ted\n"},{"id":"89915","messageId":"7viqt9rvwm.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"20080906162030.GT9129@mit.edu","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-06T18:06:49Z","receivedAt":"2008-09-06T18:06:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@MIT.EDU> writes:\n\n> Long-standing mis-feature in git's logic in deciding what to push.\n> It's been reported a few times, but apparently it's hard to fix, or at\n> least it never hsa been fixed as far as I know.\n\nThis comes from an early (mis)design of git.\n\nBackground.\n\n * A git repository and the object store it uses can be separate.  From\n   the beginning, you can have a\"objects/\" directory (aka \"object store\")\n   that is shared by more than one repositories.  There is no Porcelain\n   level support to set up two repositories that physically share the same\n   object store, but the result of \"git init; rm -rf .git/objects; ln -s\n   $other/.git/objects .git/objects\" was supposed to work (and it still\n   largely works, until you gc) in the original design.\n\n   The alternate object store does not even have to be a git repository,\n   which makes things worse.  You can have everybody pointing at\n   /var/cache/objects, and /var/cache does not have to be a git repository\n   (i.e. no var/cache/refs).\n\n * The existing alternates mechanism is not about alternate repositories.\n   It is about alternate object stores.  That is why each line of this\n   file points at \"objects\" directory elsewhere, not the \".git\" directory\n   that is typically at one level above that \"objects\" directory.\n\n   The fact your repository's object store points at the object store that\n   happens to be inside Linus's repository does not imply that Linus's\n   object store is associated with refs in Linus's repository in any way\n   (that's the early _mis_design part).\n\n * An existing ref in a git repository is meant to be a guarantee that all\n   objects the object referenced by the ref is found somewhere in the\n   object store(s) the repository uses.  Object transfers in git\n   (i.e. fetch and push) use this guarantee to tell what a repository has\n   to the other side.\n\n   What happens in your case is that github end knows that the repository\n   you are pushing into have up to the refs you have there.  Alternate may\n   point at object store that holds objects from Linus's repository, but\n   there is no information as to what the latest commits you do not see in\n   your refs namespace (namely, \"what's Linus's latest\" is not something\n   you can learn from your repository that has alternates).\n\nA possible fix would involve:\n\n - Deprecate objects/info/alternates file, and GIT_OBJECT_DIRECTORY and\n   GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables;\n\n - Introduce info/alternates that points at alternate _repositories_ (as\n   opposed to objects/info/alternates that points at alternate object\n   stores);\n\n - Teach fetch and push to include refs from alternate _repositories_ into\n   what local side considers complete.\n\nThe above won't break existing setups, but it won't help them either.  All\nthe borrowing repositoies need to be converted if we go that route.\n\nWe could instead redefine the semantics of the existing alternates\nmechanism.  This technically *breaks* backward compatibility, but I\nsuspect it won't hurt many existing installations:\n\n - Declare that a freestanding object store is illegal.  In other words,\n   if a directory \"$D/objects\" is (1) used as $GIT_OBJECT_DIRECTORY's\n   value, (2) pointed by some repository's \"alternates\" file, or (3)\n   listed in $GIT_ALTERNATE_OBJECT_DIRECTORIES's value, this change makes\n   it illegal for \"$D\" not being a proper git repository.\n\n   This will not break your example of your repository's object store\n   borrowing from the object store inside Linus's repository.\n\n - When you have \"$D/objects\" in alternates, start relying on \"$D/refs\"\n   being correct (i.e. repository $D is not corrupt).  This technically\n   makes the system slightly less robust, as we are depending on _other\n   people's_ good behaviour even more when you use alternates, but you are\n   already depending on them having good objects in $D/objects anyway, so\n   it is not a big deal.\n\n - Now that we declared that everything reachable from \"$D/refs\" do not\n   have to be transferred from elsewhere when a push sends things into us\n   (or a fetch gets things from elsewhere into us) when you have\n   \"$D/objects\" in your alternates.  In your \"borrowing from Linus\"\n   example, Linus's latest will be reachable from somewhere in \"$D/refs\",\n   when you are borrowing from him by having \"$D/objects\" in your\n   alternates. \n"},{"id":"89919","messageId":"9e4733910809061124r3759ea79sda98b549df3e6e58@mail.gmail.com","threadId":"15394","inReplyTo":"7viqt9rvwm.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2008-09-06T18:24:59Z","receivedAt":"2008-09-06T18:24:59Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 9/6/08, Junio C Hamano <gitster@pobox.com> wrote:\n> Theodore Tso <tytso@MIT.EDU> writes:\n>\n>  > Long-standing mis-feature in git's logic in deciding what to push.\n>  > It's been reported a few times, but apparently it's hard to fix, or at\n>  > least it never hsa been fixed as far as I know.\n>\n>\n> This comes from an early (mis)design of git.\n\nWhat happens if I set a remote in my github digispeaker repo pointing\nto github's linus repo and the set a chron script to fetch it once a\nday? The alternates link is still in place.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"89925","messageId":"20080906192106.GB18631@spearce.org","threadId":"15394","inReplyTo":"9e4733910809061124r3759ea79sda98b549df3e6e58@mail.gmail.com","subject":"Re: Alternates and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-09-06T19:21:06Z","receivedAt":"2008-09-06T19:21:06Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl <jonsmirl@gmail.com> wrote:\n> On 9/6/08, Junio C Hamano <gitster@pobox.com> wrote:\n> > Theodore Tso <tytso@MIT.EDU> writes:\n> >\n> >  > Long-standing mis-feature in git's logic in deciding what to push.\n> >  > It's been reported a few times, but apparently it's hard to fix, or at\n> >  > least it never hsa been fixed as far as I know.\n> >\n> >\n> > This comes from an early (mis)design of git.\n> \n> What happens if I set a remote in my github digispeaker repo pointing\n> to github's linus repo and the set a chron script to fetch it once a\n> day? The alternates link is still in place.\n\ngithub should do what repo.or.cz does:\n\n\tln -s .../linus.git/refs digispeaker.git/refs/forkee\n\nThat way the refs available in Linus' tree are also available\nin your tree as Git will transparently follow the symlink.\n\nHowever you have to be careful to make sure `git pack-refs` isn't\nrun with `--all --prune` as it will delete the refs from linus.git\nwhen executed in digispeaker.git.  Fun times when I did that to my\nown repository one day.  ;-)\n\nThough we probably should fix Git to be somewhat smarter.  But\nuntil that stable binary is available, the symlink trick above\nis a good work around.\n\n-- \nShawn.\n"},{"id":"89991","messageId":"20080907184922.GA3909@efreet.light.src","threadId":"15394","inReplyTo":"7viqt9rvwm.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-09-07T18:49:22Z","receivedAt":"2008-09-07T18:49:22Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Sat, Sep 06, 2008 at 11:06:49 -0700, Junio C Hamano wrote:\n> Theodore Tso <tytso@MIT.EDU> writes:\n> \n> > Long-standing mis-feature in git's logic in deciding what to push.\n> > It's been reported a few times, but apparently it's hard to fix, or at\n> > least it never hsa been fixed as far as I know.\n> \n> This comes from an early (mis)design of git.\n> [...]\n>  * The existing alternates mechanism is not about alternate repositories.\n>    It is about alternate object stores.  That is why each line of this\n>    file points at \"objects\" directory elsewhere, not the \".git\" directory\n>    that is typically at one level above that \"objects\" directory.\n> \n>    The fact your repository's object store points at the object store that\n>    happens to be inside Linus's repository does not imply that Linus's\n>    object store is associated with refs in Linus's repository in any way\n>    (that's the early _mis_design part).\n\nWhy is this a *mis*design? Couldn't push be fixed by redesigning it's\nprotocol along the lines of:\n - clients sends a list of sha1s it wants to push, from the tip down\n - server stops it when it sees an object it has -- this check can be done\n   against the object store without having a ref for it.\n\nRegards,\nJan\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec <bulb@ucw.cz>\n"},{"id":"89992","messageId":"7vhc8rn5sl.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"20080907184922.GA3909@efreet.light.src","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-07T18:56:42Z","receivedAt":"2008-09-07T18:56:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jan Hudec <bulb@ucw.cz> writes:\n\n> On Sat, Sep 06, 2008 at 11:06:49 -0700, Junio C Hamano wrote:\n>> Theodore Tso <tytso@MIT.EDU> writes:\n>> \n>> > Long-standing mis-feature in git's logic in deciding what to push.\n>> > It's been reported a few times, but apparently it's hard to fix, or at\n>> > least it never hsa been fixed as far as I know.\n>> \n>> This comes from an early (mis)design of git.\n>> [...]\n>>  * The existing alternates mechanism is not about alternate repositories.\n>>    It is about alternate object stores.  That is why each line of this\n>>    file points at \"objects\" directory elsewhere, not the \".git\" directory\n>>    that is typically at one level above that \"objects\" directory.\n>> \n>>    The fact your repository's object store points at the object store that\n>>    happens to be inside Linus's repository does not imply that Linus's\n>>    object store is associated with refs in Linus's repository in any way\n>>    (that's the early _mis_design part).\n>\n> Why is this a *mis*design? Couldn't push be fixed by redesigning it's\n> protocol along the lines of:\n>  - clients sends a list of sha1s it wants to push, from the tip down\n>  - server stops it when it sees an object it has -- this check can be done\n>    against the object store without having a ref for it.\n\nBecause your second step is *BROKEN*.\n\nThink of a case where an earlier commit walker started fetching into that\n\"server\" end, which got newer commits and their associated objects first\nand then older ones, and then got killed before reaching to the objects it\nalready had.  In such a case, the commit walker will *not* update the refs\non the server end (and for a very good reason).\n\nAfter that, the server end would have:\n\n * refs that point at some older commits, all objects from whom are\n   guaranteed to be in the repository (that's the \"ref\" guarantee);\n\n * newer commits and their objects, but if you follow them you will hit\n   some objects that are *NOT* in the repository.\n\nNow imagine starting your broken procedure to serve clients from such a\nrepository.  Your second step would cause the server to attempt to pack\nthe difference from the latter classes of incomplete objects and makes the\npack generation fail.\n"},{"id":"89994","messageId":"20080907191740.GA29055@efreet.light.src","threadId":"15394","inReplyTo":"7vhc8rn5sl.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-09-07T19:17:40Z","receivedAt":"2008-09-07T19:17:40Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Sun, Sep 07, 2008 at 11:56:42 -0700, Junio C Hamano wrote:\n> Jan Hudec <bulb@ucw.cz> writes:\n> \n> > On Sat, Sep 06, 2008 at 11:06:49 -0700, Junio C Hamano wrote:\n> >> Theodore Tso <tytso@MIT.EDU> writes:\n> >> \n> >> > Long-standing mis-feature in git's logic in deciding what to push.\n> >> > It's been reported a few times, but apparently it's hard to fix, or at\n> >> > least it never hsa been fixed as far as I know.\n> >> \n> >> This comes from an early (mis)design of git.\n> >> [...]\n> >>  * The existing alternates mechanism is not about alternate repositories.\n> >>    It is about alternate object stores.  That is why each line of this\n> >>    file points at \"objects\" directory elsewhere, not the \".git\" directory\n> >>    that is typically at one level above that \"objects\" directory.\n> >> \n> >>    The fact your repository's object store points at the object store that\n> >>    happens to be inside Linus's repository does not imply that Linus's\n> >>    object store is associated with refs in Linus's repository in any way\n> >>    (that's the early _mis_design part).\n> >\n> > Why is this a *mis*design? Couldn't push be fixed by redesigning it's\n> > protocol along the lines of:\n> >  - clients sends a list of sha1s it wants to push, from the tip down\n> >  - server stops it when it sees an object it has -- this check can be done\n> >    against the object store without having a ref for it.\n> \n> Because your second step is *BROKEN*.\n> \n> Think of a case where an earlier commit walker started fetching into that\n> \"server\" end, which got newer commits and their associated objects first\n> and then older ones, and then got killed before reaching to the objects it\n\nIs there really any transport that stores newer objects first? And can't be\nfixed to only store the objects when they are good. My proposal definitely\ndoes not need to do that (the list goes from newest to older, but the data in\nthird step can come from oldest to newer).\n\n> already had.  In such a case, the commit walker will *not* update the refs\n> on the server end (and for a very good reason).\n> \n> After that, the server end would have:\n> \n>  * refs that point at some older commits, all objects from whom are\n>    guaranteed to be in the repository (that's the \"ref\" guarantee);\n> \n>  * newer commits and their objects, but if you follow them you will hit\n>    some objects that are *NOT* in the repository.\n> \n> Now imagine starting your broken procedure to serve clients from such a\n> repository.  Your second step would cause the server to attempt to pack\n> the difference from the latter classes of incomplete objects and makes the\n> pack generation fail.\n\nYes, it would add a requirement that objects are only placed in the object\nstore after all objects they reference, but I can't imagine good reason this\ncouldn't be done.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec <bulb@ucw.cz>\n"},{"id":"89996","messageId":"7v8wu3n4t1.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"7vhc8rn5sl.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-07T19:18:02Z","receivedAt":"2008-09-07T19:18:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Jan Hudec <bulb@ucw.cz> writes:\n> ...\n>> Why is this a *mis*design? Couldn't push be fixed by redesigning it's\n>> protocol along the lines of:\n>>  - clients sends a list of sha1s it wants to push, from the tip down\n>>  - server stops it when it sees an object it has -- this check can be done\n>>    against the object store without having a ref for it.\n>\n> Because your second step is *BROKEN*.\n>\n> Think of a case where an earlier commit walker started fetching into that\n> \"server\" end, which got newer commits and their associated objects first\n> and then older ones, and then got killed before reaching to the objects it\n> already had.  In such a case, the commit walker will *not* update the refs\n> on the server end (and for a very good reason).\n>\n> After that, the server end would have:\n>\n>  * refs that point at some older commits, all objects from whom are\n\ns/from whom/reachable from which/;\n\n>    guaranteed to be in the repository (that's the \"ref\" guarantee);\n>\n>  * newer commits and their objects, but if you follow them you will hit\n>    some objects that are *NOT* in the repository.\n\nTo visualize, the server object store and refs would be like this:\n\n    ---o---o---A...x...x...x...x...o---o---X\n               ^ ref\n\nCommits 'x' are all missing because the commit walker fetched commit X,\ninspected its tree and got the necessary tree and blob objects, went back\nto get X's parent, did the same, then its parent, attempted to do the same\nbut got killed before connecting the history fully to A.\n\nIf you accepted history on top of X before guaranteeing that you have\neverything reachable from X already in this round of push will give you this:\n\n\n    ---o---o---A...x...x...x...x...o---o---X---o---o---Y\n               ^ ref =========== (wrong) ============> ^ ref\n\nand if you update the ref to point at Y, then you cannot satisfy requests\nfrom other people who want the history that leads to Y, because somewhere\nbetween A and X there are commit that you do not even have to begin with.\n\nSo you may even be able accept objects between X..Y, but you cannot update\nthe ref from A to Y after accepting such a push, which is pointless.\n\nYou could try a variant of it to unbreak your trick, though.  When you see\nan object that you have, say 'X' above, you traverse down from there until\nreaching some ref (in this case, A) and make sure that you have everything\nin between (not just commits but also associated trees and blobs that are\nneeded).  This is quite similar to what is happening when the commit\nwalker says \"walk deadbeef...\" in its progress output.  So it _could_ be\ndone, but it would be somewhat expensive.\n"},{"id":"90025","messageId":"20080907234118.GA8161@mit.edu","threadId":"15394","inReplyTo":"7viqt9rvwm.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2008-09-07T23:41:18Z","receivedAt":"2008-09-07T23:41:18Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sat, Sep 06, 2008 at 11:06:49AM -0700, Junio C Hamano wrote:\n> We could instead redefine the semantics of the existing alternates\n> mechanism.  This technically *breaks* backward compatibility, but I\n> suspect it won't hurt many existing installations:\n> \n>  - Declare that a freestanding object store is illegal.  In other words,\n>    if a directory \"$D/objects\" is (1) used as $GIT_OBJECT_DIRECTORY's\n>    value, (2) pointed by some repository's \"alternates\" file, or (3)\n>    listed in $GIT_ALTERNATE_OBJECT_DIRECTORIES's value, this change makes\n>    it illegal for \"$D\" not being a proper git repository.\n> \n>    This will not break your example of your repository's object store\n>    borrowing from the object store inside Linus's repository.\n> \n>  - When you have \"$D/objects\" in alternates, start relying on \"$D/refs\"\n>    being correct (i.e. repository $D is not corrupt).  This technically\n>    makes the system slightly less robust, as we are depending on _other\n>    people's_ good behaviour even more when you use alternates, but you are\n>    already depending on them having good objects in $D/objects anyway, so\n>    it is not a big deal.\n\nOne way that wouldn't break backwards compatibility would be to use\n$D/refs if it exists, but if it isn't, fall back to existing behavior\n(which is to say, only use the refs in the repository, not in the\nborrowed repository/object store).  Is there a reason why this would\nbe problematic?\n\n\t\t\t\t\t\t\t- Ted\n"},{"id":"90029","messageId":"7vd4jfjyiq.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"20080907234118.GA8161@mit.edu","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-08T00:02:05Z","receivedAt":"2008-09-08T00:02:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@MIT.EDU> writes:\n\n> On Sat, Sep 06, 2008 at 11:06:49AM -0700, Junio C Hamano wrote:\n>> We could instead redefine the semantics of the existing alternates\n>> mechanism.  This technically *breaks* backward compatibility, but I\n>> suspect it won't hurt many existing installations:\n>> \n>>  - Declare that a freestanding object store is illegal.  In other words,\n>>    if a directory \"$D/objects\" is (1) used as $GIT_OBJECT_DIRECTORY's\n>>    value, (2) pointed by some repository's \"alternates\" file, or (3)\n>>    listed in $GIT_ALTERNATE_OBJECT_DIRECTORIES's value, this change makes\n>>    it illegal for \"$D\" not being a proper git repository.\n>> \n>>    This will not break your example of your repository's object store\n>>    borrowing from the object store inside Linus's repository.\n>> \n>>  - When you have \"$D/objects\" in alternates, start relying on \"$D/refs\"\n>>    being correct (i.e. repository $D is not corrupt).  This technically\n>>    makes the system slightly less robust, as we are depending on _other\n>>    people's_ good behaviour even more when you use alternates, but you are\n>>    already depending on them having good objects in $D/objects anyway, so\n>>    it is not a big deal.\n>\n> One way that wouldn't break backwards compatibility would be to use\n> $D/refs if it exists, but if it isn't, fall back to existing behavior\n> (which is to say, only use the refs in the repository, not in the\n> borrowed repository/object store).  Is there a reason why this would\n> be problematic?\n\nI think you just reiterated what I said in \"we could instead\", and I think\nwe are in agreement.\n\nI already stated the reason this could be problematic and also I said I do\nnot think it is a big deal in the above.\n\nI think the question to ask at this point is not \"is there a reason why\nthis would be problematic\", but \"is it really not a big deal as Junio\nclaims?\" and \"aren't there _other_ reasons than the above that makes it\nproblematic?\".\n\nThe arguments to make are \"Junio is worrying too much; depending on the\nother repository's ref is no worse than depending on the objects the other\nrepository uses, and here is a proof that it is not just 'not a big deal'\nbut 'no problem at all'\", \"I've polled the userbase and there is no\nexisting configuration that will be broken by this change\", and \"I have\nthis configuration that will be broken by above change, don't do it\".\n"},{"id":"90033","messageId":"20080908004137.GB8161@mit.edu","threadId":"15394","inReplyTo":"7vd4jfjyiq.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2008-09-08T00:41:37Z","receivedAt":"2008-09-08T00:41:37Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sun, Sep 07, 2008 at 05:02:05PM -0700, Junio C Hamano wrote:\n> \n> I think you just reiterated what I said in \"we could instead\", and I think\n> we are in agreement.\n> \n\nSorry, I misunderstood what you were saying.  I thought you were\nproposing that if $D/refs didn't exist, then if alternates pointed at\n$D/objects, git would reject using it.  But that makes no sense, which\nis why I didn't understand why you proposed it.  (Turns out I\nmisunderstood you.  :-)\n\n> The arguments to make are \"Junio is worrying too much; depending on the\n> other repository's ref is no worse than depending on the objects the other\n> repository uses, and here is a proof that it is not just 'not a big deal'\n> but 'no problem at all'\", \"I've polled the userbase and there is no\n> existing configuration that will be broken by this change\", and \"I have\n> this configuration that will be broken by above change, don't do it\".\n\nSo the only configuration I can think of that would be broken by this\nis where $D/refs exists, but is insane.  (i.e., such that git fsck for\n$D would result in errors).  That seems pretty unlikely...\n\n   \t \t   \t     \t  \t       - Ted\n"},{"id":"90039","messageId":"7vljy3ic09.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"20080908004137.GB8161@mit.edu","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-08T02:53:42Z","receivedAt":"2008-09-08T02:53:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@MIT.EDU> writes:\n\n>> The arguments to make are \"Junio is worrying too much; depending on the\n>> other repository's ref is no worse than depending on the objects the other\n>> repository uses, and here is a proof that it is not just 'not a big deal'\n>> but 'no problem at all'\", \"I've polled the userbase and there is no\n>> existing configuration that will be broken by this change\", and \"I have\n>> this configuration that will be broken by above change, don't do it\".\n>\n> So the only configuration I can think of that would be broken by this\n> is where $D/refs exists, but is insane.  (i.e., such that git fsck for\n> $D would result in errors).  That seems pretty unlikely...\n\nI was more worried about \"once $D/ started its life as an actively used\nand maintained repository, but it itself has been abandoned and its refs\nare not maintained anymore; only $D/objects are still in use\", but even in\nthat case, it won't be \"$D/refs exists but insane\" case, so we should be\nOk.\n"},{"id":"90047","messageId":"7v4p4ri5t2.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"20080907234118.GA8161@mit.edu","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-08T05:07:37Z","receivedAt":"2008-09-08T05:07:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@MIT.EDU> writes:\n\n> One way that wouldn't break backwards compatibility would be to use\n> $D/refs if it exists, but if it isn't, fall back to existing behavior\n> (which is to say, only use the refs in the repository, not in the\n> borrowed repository/object store).  Is there a reason why this would\n> be problematic?\n\nThis does not answer \"is this safe enough change?\" question, but the code\nto implement this should look like this (Area expert Daniel CC'ed).\n\n Makefile                                 |    2 +-\n receive-pack.c => builtin-receive-pack.c |   60 ++++++++++++++++++++++++++++-\n builtin.h                                |    1 +\n cache.h                                  |    2 +\n git.c                                    |    1 +\n sha1_file.c                              |   10 +++++\n 6 files changed, 72 insertions(+), 4 deletions(-)\n\ndiff --git c/Makefile w/Makefile\nindex dfed7ba..badf27a 100644\n--- c/Makefile\n+++ w/Makefile\n@@ -294,7 +294,6 @@ PROGRAMS += git-mktag$X\n PROGRAMS += git-mktree$X\n PROGRAMS += git-pack-redundant$X\n PROGRAMS += git-patch-id$X\n-PROGRAMS += git-receive-pack$X\n PROGRAMS += git-send-pack$X\n PROGRAMS += git-show-index$X\n PROGRAMS += git-unpack-file$X\n@@ -543,6 +542,7 @@ BUILTIN_OBJS += builtin-prune-packed.o\n BUILTIN_OBJS += builtin-prune.o\n BUILTIN_OBJS += builtin-push.o\n BUILTIN_OBJS += builtin-read-tree.o\n+BUILTIN_OBJS += builtin-receive-pack.o\n BUILTIN_OBJS += builtin-reflog.o\n BUILTIN_OBJS += builtin-remote.o\n BUILTIN_OBJS += builtin-rerere.o\ndiff --git c/receive-pack.c w/builtin-receive-pack.c\nsimilarity index 88%\nrename from receive-pack.c\nrename to builtin-receive-pack.c\nindex b81678a..c9bb722 100644\n--- c/receive-pack.c\n+++ w/builtin-receive-pack.c\n@@ -6,6 +6,8 @@\n #include \"exec_cmd.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"remote.h\"\n+#include \"transport.h\"\n \n static const char receive_pack_usage[] = \"git-receive-pack <git-dir>\";\n \n@@ -462,14 +464,64 @@ static int delete_only(struct command *cmd)\n \treturn 1;\n }\n \n-int main(int argc, char **argv)\n+/* Need to move this to path.c or somewhere, and remove the same from builtin-clone.c */\n+static int is_directory(const char *path)\n+{\n+\tstruct stat buf;\n+\n+\treturn !stat(path, &buf) && S_ISDIR(buf.st_mode);\n+}\n+\n+static int add_refs_from_alternate(struct alternate_object_database *e, void *unused)\n+{\n+\tchar *other = xstrdup(make_absolute_path(e->base));\n+\tsize_t len = strlen(other);\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tconst struct ref *extra;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\twhile (other[len-1] == '/')\n+\t\tother[--len] = '\\0';\n+\tif (len < 8 || memcmp(other + len - 8, \"/objects\", 8))\n+\t\treturn 0;\n+\t/* Is this a git repository with refs? */\n+\tmemcpy(other + len - 8, \"/refs\", 6);\n+\tif (!is_directory(other))\n+\t\treturn 0;\n+\tother[len - 8] = '\\0';\n+\tremote = remote_get(other);\n+\ttransport = transport_get(remote, other);\n+\tfor (extra = transport_get_remote_refs(transport);\n+\t     extra;\n+\t     extra = extra->next) {\n+\t\t/*\n+\t\t * Make sure this is named uniquely but with invalid name, so\n+\t\t * that we can catch the other end trying to do anything funky\n+\t\t * with it.\n+\t\t */\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"refs/borrowed-ref/%s\", extra->name);\n+\t\tadd_extra_ref(buf.buf, extra->old_sha1, 0);\n+\t}\n+\ttransport_disconnect(transport);\n+\tfree(other);\n+\treturn 0;\n+}\n+\n+static void add_alternate_refs(void)\n+{\n+\tforeach_alt_odb(add_refs_from_alternate, NULL);\n+}\n+\n+int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tchar *dir = NULL;\n \n \targv++;\n \tfor (i = 1; i < argc; i++) {\n-\t\tchar *arg = *argv++;\n+\t\tconst char *arg = *argv++;\n \n \t\tif (*arg == '-') {\n \t\t\t/* Do flag handling here */\n@@ -477,7 +529,7 @@ int main(int argc, char **argv)\n \t\t}\n \t\tif (dir)\n \t\t\tusage(receive_pack_usage);\n-\t\tdir = arg;\n+\t\tdir = xstrdup(arg);\n \t}\n \tif (!dir)\n \t\tusage(receive_pack_usage);\n@@ -497,7 +549,9 @@ int main(int argc, char **argv)\n \telse if (0 <= receive_unpack_limit)\n \t\tunpack_limit = receive_unpack_limit;\n \n+\tadd_alternate_refs();\n \twrite_head_info();\n+\tclear_extra_refs();\n \n \t/* EOF */\n \tpacket_flush(1);\ndiff --git c/builtin.h w/builtin.h\nindex f3502d3..ade9a12 100644\n--- c/builtin.h\n+++ w/builtin.h\n@@ -78,6 +78,7 @@ extern int cmd_prune(int argc, const char **argv, const char *prefix);\n extern int cmd_prune_packed(int argc, const char **argv, const char *prefix);\n extern int cmd_push(int argc, const char **argv, const char *prefix);\n extern int cmd_read_tree(int argc, const char **argv, const char *prefix);\n+extern int cmd_receive_pack(int argc, const char **argv, const char *prefix);\n extern int cmd_reflog(int argc, const char **argv, const char *prefix);\n extern int cmd_remote(int argc, const char **argv, const char *prefix);\n extern int cmd_config(int argc, const char **argv, const char *prefix);\ndiff --git c/cache.h w/cache.h\nindex de8c2b6..ddea8d1 100644\n--- c/cache.h\n+++ w/cache.h\n@@ -640,6 +640,8 @@ extern struct alternate_object_database {\n } *alt_odb_list;\n extern void prepare_alt_odb(void);\n extern void add_to_alternates_file(const char *reference);\n+typedef int alt_odb_fn(struct alternate_object_database *, void *);\n+extern void foreach_alt_odb(alt_odb_fn, void*);\n \n struct pack_window {\n \tstruct pack_window *next;\ndiff --git c/git.c w/git.c\nindex fdb0f71..bf1ac1b 100644\n--- c/git.c\n+++ w/git.c\n@@ -328,6 +328,7 @@ static void handle_internal_command(int argc, const char **argv)\n \t\t{ \"prune-packed\", cmd_prune_packed, RUN_SETUP },\n \t\t{ \"push\", cmd_push, RUN_SETUP },\n \t\t{ \"read-tree\", cmd_read_tree, RUN_SETUP },\n+\t\t{ \"receive-pack\", cmd_receive_pack },\n \t\t{ \"reflog\", cmd_reflog, RUN_SETUP },\n \t\t{ \"remote\", cmd_remote, RUN_SETUP },\n \t\t{ \"repo-config\", cmd_config },\ndiff --git c/sha1_file.c w/sha1_file.c\nindex 9ee1ed1..6bd0ef0 100644\n--- c/sha1_file.c\n+++ w/sha1_file.c\n@@ -394,6 +394,16 @@ void add_to_alternates_file(const char *reference)\n \t\tlink_alt_odb_entries(alt, alt + strlen(alt), '\\n', NULL, 0);\n }\n \n+void foreach_alt_odb(alt_odb_fn fn, void *cb)\n+{\n+\tstruct alternate_object_database *ent;\n+\n+\tprepare_alt_odb();\n+\tfor (ent = alt_odb_list; ent; ent = ent->next)\n+\t\tif (fn(ent, cb))\n+\t\t\treturn;\n+}\n+\n void prepare_alt_odb(void)\n {\n \tconst char *alt;\n"},{"id":"90049","messageId":"7viqt7gmtq.fsf@gitster.siamese.dyndns.org","threadId":"15394","inReplyTo":"7v4p4ri5t2.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-08T06:42:57Z","receivedAt":"2008-09-08T06:42:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Theodore Tso <tytso@MIT.EDU> writes:\n>\n>> One way that wouldn't break backwards compatibility would be to use\n>> $D/refs if it exists, but if it isn't, fall back to existing behavior\n>> (which is to say, only use the refs in the repository, not in the\n>> borrowed repository/object store).  Is there a reason why this would\n>> be problematic?\n>\n> This does not answer \"is this safe enough change?\" question, but the code\n> to implement this should look like this (Area expert Daniel CC'ed).\n\nOops, I forgot to Cc Daniel.  Shawn also CC'ed because he will have the\nsame issue with the smart CGI based http-push he is building.\n\n> +static int add_refs_from_alternate(struct alternate_object_database *e, void *unused)\n> +{\n> +\tchar *other = xstrdup(make_absolute_path(e->base));\n> +...\n> +\tremote = remote_get(other);\n> +\ttransport = transport_get(remote, other);\n> +\tfor (extra = transport_get_remote_refs(transport);\n> +\t     extra;\n> +\t     extra = extra->next) {\n> +\t\t/*\n> +\t\t * Make sure this is named uniquely but with invalid name, so\n> +\t\t * that we can catch the other end trying to do anything funky\n> +\t\t * with it.\n> +\t\t */\n> +\t\tstrbuf_reset(&buf);\n> +\t\tstrbuf_addf(&buf, \"refs/borrowed-ref/%s\", extra->name);\n> +\t\tadd_extra_ref(buf.buf, extra->old_sha1, 0);\n> +\t}\n> +\ttransport_disconnect(transport);\n> +\tfree(other);\n> +\treturn 0;\n> +}\n\nI need to clarify the comment in the above function.\n\nWhen you are running \"git push\", the protocol employed is the send-pack\nprotocol, which begins by the receiving end advertising what refs it has\nand where they point at.  The above code is in the receiving end and adds\nextra refs from alternate repositories to this advertisement.\n\nThe advertisement serves two different purposes.  One is the immediate\nissue we are tackling --- to tell the sending end what objects are not\nneeded in the pack it sends.  Objects that are reachable from the objects\nin this advertisement do not have to be sent.  When the sending end wants\nto send its 'master', for example, it uses this information to compute the\nset of objects by doing (roughly):\n\n\tgit rev-list --objects master --not $sha1_1 $sha1_2 ...\n\nwhere $sha1_N are the objects we advertise here.  The more we assure the\nsender we have, the less it has to send to us.\n\nBut there is another purpose the sender uses this information for.  It is\nused to determine which refs to push (when \"git push\" is run without any\nexplicit refspecs, it does \"matching refs\" -- it learns the set of refs\nthe receiving end has from this advertisement, and updates the refs of the\nsame name the sending end also has), and which branches on the receiving\nend should be removed (when \"git push --mirror\" is run, it will send NULL\nupdates to refs the receiving end has but the sender doesn't).\n\nWe do want to show the extra object names to the sender so that it can\nexclude more objects from the transfer, but for the latter purpose, we\nreally do not want these phoney refs to be understood as refs we have by\nthe sending end.  In the original version of my patch, I actually had\n\"refs/borrowed*refs%s/%s\" as the format string (notice the asterisk) to\nmake it an invalid refname, and (notice the extra %s) to make them unique\nby including the full-pathname to the repository as part of this string,\nso that we can pretend we have different 'master' from two alternate\nrepositories.\n\nThe sending end, however, has a safety to silently ignore malformed refs\nit learns from the receiving end over the wire.  So the current sender\nwon't work with asterisk in there '*', nor full-pathname, because\ntypically it contains \"/.git/\" in the string, which would make it an\ninvalid refname to be ignored.\n\nAlso, the sending end remembers the objects we have per refname, so\nsending two records with the same refname pointing at two different\nobjects will not work well either (the current code is loose and does not\ncheck duplicates, but that is not a feature by design but is an accident).\nIf the receiving repository borrows from two alternate repositories, both\nof which have 'master' branch, we would be sending two records, each of\nwhich claims that it is \"refs/borrowed-ref/refs/heads/master\" but pointing\nat a different commit.\n\nThat is the \"unique and invalid\" comment above is about (iow, the above\nhunk that does not send \"unique and invalid\" refs cannot be the final form\nof this enhancement).\n\nWhat this means is that we also need to update the sending side in order\nto enable this enhancement.  connect.c::get_remote_heads() is the relevant\ncode and we would probably need to add an option to keep all refs (and\ninvalid refs) in the returned list.  Both remote.c::match_refs() and\nbuiltin-send-pack.c::do_send_pack() need to be taught to ignore the\ninvalid refs in the list, while builtin-send-pack.c::pack_objects() should\nutilize all the refs (and invalid ones) when excluding the objects the\nreceiving end claims to have.  While at it, we probably should declare\nthat sending duplicate invalid refs is not an error (and we can use\nsomething like \"*borrowed*\" as the phoney refname --- we do not need the\nuniqueness for them, nor we need to tell the other end what ref we are\nborrowing from whom).\n\nWe can start sending invalid refs (just replace - with * in the patch I\nsent) from the receiving end without waiting for the sender end program to\nget updated.  The sending end will ignore it and nothing (other than the\nextra startup overhead the additional code has) bad should happen.  When\nthe sender is updated to keep the invalid refs, it will start to take\nnotice and your push will suddenly get smaller.\n"},{"id":"90054","messageId":"alpine.LNX.1.00.0809080246520.19665@iabervon.org","threadId":"15394","inReplyTo":"7viqt7gmtq.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-09-08T07:24:56Z","receivedAt":"2008-09-08T07:24:56Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 7 Sep 2008, Junio C Hamano wrote:\n\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > Theodore Tso <tytso@MIT.EDU> writes:\n> >\n> >> One way that wouldn't break backwards compatibility would be to use\n> >> $D/refs if it exists, but if it isn't, fall back to existing behavior\n> >> (which is to say, only use the refs in the repository, not in the\n> >> borrowed repository/object store).  Is there a reason why this would\n> >> be problematic?\n> >\n> > This does not answer \"is this safe enough change?\" question, but the code\n> > to implement this should look like this (Area expert Daniel CC'ed).\n> \n> Oops, I forgot to Cc Daniel.  Shawn also CC'ed because he will have the\n> same issue with the smart CGI based http-push he is building.\n\nThe use of transport in general looks good to me; I'm not really an area \nexpert for the native protocol.\n\n> > +static int add_refs_from_alternate(struct alternate_object_database *e, void *unused)\n> > +{\n> > +\tchar *other = xstrdup(make_absolute_path(e->base));\n> > +...\n> > +\tremote = remote_get(other);\n> > +\ttransport = transport_get(remote, other);\n> > +\tfor (extra = transport_get_remote_refs(transport);\n> > +\t     extra;\n> > +\t     extra = extra->next) {\n> > +\t\t/*\n> > +\t\t * Make sure this is named uniquely but with invalid name, so\n> > +\t\t * that we can catch the other end trying to do anything funky\n> > +\t\t * with it.\n> > +\t\t */\n> > +\t\tstrbuf_reset(&buf);\n> > +\t\tstrbuf_addf(&buf, \"refs/borrowed-ref/%s\", extra->name);\n> > +\t\tadd_extra_ref(buf.buf, extra->old_sha1, 0);\n> > +\t}\n> > +\ttransport_disconnect(transport);\n> > +\tfree(other);\n> > +\treturn 0;\n> > +}\n> \n> I need to clarify the comment in the above function.\n> \n> When you are running \"git push\", the protocol employed is the send-pack\n> protocol, which begins by the receiving end advertising what refs it has\n> and where they point at.  The above code is in the receiving end and adds\n> extra refs from alternate repositories to this advertisement.\n> \n> The advertisement serves two different purposes.  One is the immediate\n> issue we are tackling --- to tell the sending end what objects are not\n> needed in the pack it sends.  Objects that are reachable from the objects\n> in this advertisement do not have to be sent.  When the sending end wants\n> to send its 'master', for example, it uses this information to compute the\n> set of objects by doing (roughly):\n> \n> \tgit rev-list --objects master --not $sha1_1 $sha1_2 ...\n> \n> where $sha1_N are the objects we advertise here.  The more we assure the\n> sender we have, the less it has to send to us.\n> \n> But there is another purpose the sender uses this information for.  It is\n> used to determine which refs to push (when \"git push\" is run without any\n> explicit refspecs, it does \"matching refs\" -- it learns the set of refs\n> the receiving end has from this advertisement, and updates the refs of the\n> same name the sending end also has), and which branches on the receiving\n> end should be removed (when \"git push --mirror\" is run, it will send NULL\n> updates to refs the receiving end has but the sender doesn't).\n> \n> We do want to show the extra object names to the sender so that it can\n> exclude more objects from the transfer, but for the latter purpose, we\n> really do not want these phoney refs to be understood as refs we have by\n> the sending end.  In the original version of my patch, I actually had\n> \"refs/borrowed*refs%s/%s\" as the format string (notice the asterisk) to\n> make it an invalid refname, and (notice the extra %s) to make them unique\n> by including the full-pathname to the repository as part of this string,\n> so that we can pretend we have different 'master' from two alternate\n> repositories.\n> \n> The sending end, however, has a safety to silently ignore malformed refs\n> it learns from the receiving end over the wire.  So the current sender\n> won't work with asterisk in there '*', nor full-pathname, because\n> typically it contains \"/.git/\" in the string, which would make it an\n> invalid refname to be ignored.\n>\n> Also, the sending end remembers the objects we have per refname, so\n> sending two records with the same refname pointing at two different\n> objects will not work well either (the current code is loose and does not\n> check duplicates, but that is not a feature by design but is an accident).\n> If the receiving repository borrows from two alternate repositories, both\n> of which have 'master' branch, we would be sending two records, each of\n> which claims that it is \"refs/borrowed-ref/refs/heads/master\" but pointing\n> at a different commit.\n> \n> That is the \"unique and invalid\" comment above is about (iow, the above\n> hunk that does not send \"unique and invalid\" refs cannot be the final form\n> of this enhancement).\n> \n> What this means is that we also need to update the sending side in order\n> to enable this enhancement.  connect.c::get_remote_heads() is the relevant\n> code and we would probably need to add an option to keep all refs (and\n> invalid refs) in the returned list.  Both remote.c::match_refs() and\n> builtin-send-pack.c::do_send_pack() need to be taught to ignore the\n> invalid refs in the list, while builtin-send-pack.c::pack_objects() should\n> utilize all the refs (and invalid ones) when excluding the objects the\n> receiving end claims to have.  While at it, we probably should declare\n> that sending duplicate invalid refs is not an error (and we can use\n> something like \"*borrowed*\" as the phoney refname --- we do not need the\n> uniqueness for them, nor we need to tell the other end what ref we are\n> borrowing from whom).\n\nI think we should use a really invalid ref name, like \"^\" or something \nlike that, so that it's clearly not an actually available ref, but just a \nway for the remote to mention that it has the object, and the remote \ndoesn't have to try to make it unique.\n\n> We can start sending invalid refs (just replace - with * in the patch I\n> sent) from the receiving end without waiting for the sender end program to\n> get updated.  The sending end will ignore it and nothing (other than the\n> extra startup overhead the additional code has) bad should happen.  When\n> the sender is updated to keep the invalid refs, it will start to take\n> notice and your push will suddenly get smaller.\n\nAgreed.\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"90087","messageId":"20080908145654.GC10252@spearce.org","threadId":"15394","inReplyTo":"alpine.LNX.1.00.0809080246520.19665@iabervon.org","subject":"Re: Alternates and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-09-08T14:56:54Z","receivedAt":"2008-09-08T14:56:54Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> wrote:\n> On Sun, 7 Sep 2008, Junio C Hamano wrote:\n> > \n> > Oops, I forgot to Cc Daniel.  Shawn also CC'ed because he will have the\n> > same issue with the smart CGI based http-push he is building.\n\nNot just the smart CGI based http-push thingy.  JGit is impacted\nanytime we start talking about network transport.  ;-)\n \n> > The sending end, however, has a safety to silently ignore malformed refs\n> > it learns from the receiving end over the wire.  So the current sender\n> > won't work with asterisk in there '*', nor full-pathname, because\n> > typically it contains \"/.git/\" in the string, which would make it an\n> > invalid refname to be ignored.\n> > \n> > Also, the sending end remembers the objects we have per refname, so\n> > sending two records with the same refname pointing at two different\n> > objects will not work well either (the current code is loose and does not\n> > check duplicates, but that is not a feature by design but is an accident).\n\nCute. Learn something new every day.\n\nJGit doesn't do the check_ref_format(), but it does do duplicate\ndetection.  If the remote side sends us duplicate advertisements\nJGit freaks out and aborts the transfer.  This is largely just a\nfunction of JGit hashing the advertised refs for O(1) access later\non during processing.  We avoid a lot of ugly O(N^2) loops that way.\n\n> I think we should use a really invalid ref name, like \"^\" or something \n> like that, so that it's clearly not an actually available ref, but just a \n> way for the remote to mention that it has the object, and the remote \n> doesn't have to try to make it unique.\n\nI agree.  How about just calling all of these \".have\"?\n\nAccording to check_ref_format() it fails, so older clients will\njust ignore it.  Attempts by bad clients to push or delete \".have\"\nshould also fail automatically in receive-pack's own test at the\nstart of the update() function.\n\nUsing a constant name for all instances hides what the underlying\nalternates really have, in case their branches aren't public, but\ntheir databases are.  It also makes it easier for hashing clients\nlike JGit to collect these object IDs into a table of known objects,\nbut ignoring the name component.\n\nSadly no matter what we do here JGit's duplicate detection is in\ndirect opposite behavior with git.git's ignoring check_ref_format()\nfailures.  So we'll have to patch JGit no matter what we do.\n\n-- \nShawn.\n"},{"id":"90111","messageId":"20080908175606.GA3466@efreet.light.src","threadId":"15394","inReplyTo":"7v8wu3n4t1.fsf@gitster.siamese.dyndns.org","subject":"Re: Alternates and push","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-09-08T17:56:06Z","receivedAt":"2008-09-08T17:56:06Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Sun, Sep 07, 2008 at 12:18:02 -0700, Junio C Hamano wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > Jan Hudec <bulb@ucw.cz> writes:\n> > ...\n> >> Why is this a *mis*design? Couldn't push be fixed by redesigning it's\n> >> protocol along the lines of:\n> >>  - clients sends a list of sha1s it wants to push, from the tip down\n> >>  - server stops it when it sees an object it has -- this check can be done\n> >>    against the object store without having a ref for it.\n> >\n> > Because your second step is *BROKEN*.\n> >\n> > Think of a case where an earlier commit walker started fetching into that\n> > \"server\" end, which got newer commits and their associated objects first\n> > and then older ones, and then got killed before reaching to the objects it\n> > already had.  In such a case, the commit walker will *not* update the refs\n> > on the server end (and for a very good reason).\n> >\n> > After that, the server end would have:\n> >\n> >  * refs that point at some older commits, all objects from whom are\n> \n> s/from whom/reachable from which/;\n> \n> >    guaranteed to be in the repository (that's the \"ref\" guarantee);\n> >\n> >  * newer commits and their objects, but if you follow them you will hit\n> >    some objects that are *NOT* in the repository.\n> \n> To visualize, the server object store and refs would be like this:\n> \n>     ---o---o---A...x...x...x...x...o---o---X\n>                ^ ref\n> \n> Commits 'x' are all missing because the commit walker fetched commit X,\n> inspected its tree and got the necessary tree and blob objects, went back\n> to get X's parent, did the same, then its parent, attempted to do the same\n> but got killed before connecting the history fully to A.\n\nThe problem was I didn't realize how this could happen. Now when you said\n/walker/, it's obvious which way of adding objects does it. I'd however\nargue that that, rather than having the object store independent in the first\nplace, is misdesign.\n\n> If you accepted history on top of X before guaranteeing that you have\n> everything reachable from X already in this round of push will give you this:\n> \n> \n>     ---o---o---A...x...x...x...x...o---o---X---o---o---Y\n>                ^ ref =========== (wrong) ============> ^ ref\n> \n> and if you update the ref to point at Y, then you cannot satisfy requests\n> from other people who want the history that leads to Y, because somewhere\n> between A and X there are commit that you do not even have to begin with.\n> \n> So you may even be able accept objects between X..Y, but you cannot update\n> the ref from A to Y after accepting such a push, which is pointless.\n> \n> You could try a variant of it to unbreak your trick, though.  When you see\n> an object that you have, say 'X' above, you traverse down from there until\n> reaching some ref (in this case, A) and make sure that you have everything\n> in between (not just commits but also associated trees and blobs that are\n> needed).  This is quite similar to what is happening when the commit\n> walker says \"walk deadbeef...\" in its progress output.  So it _could_ be\n> done, but it would be somewhat expensive.\n\nNo. I would vote for unbreaking it at the walker instead. Instead of putting\nthe downloaded packs directly into the object store, it could put them in\nsome staging area and only move them in place when all dependencies are\ndownloaded. Still makes the solution comparably complex to the other ones.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec <bulb@ucw.cz>\n"},{"id":"90188","messageId":"20080909083551.GA10544@machine.or.cz","threadId":"15394","inReplyTo":"20080906192106.GB18631@spearce.org","subject":"Re: Alternates and push","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-09-09T08:35:51Z","receivedAt":"2008-09-09T08:35:51Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Sat, Sep 06, 2008 at 12:21:06PM -0700, Shawn O. Pearce wrote:\n> github should do what repo.or.cz does:\n> \n> \tln -s .../linus.git/refs digispeaker.git/refs/forkee\n> \n> That way the refs available in Linus' tree are also available\n> in your tree as Git will transparently follow the symlink.\n> \n> However you have to be careful to make sure `git pack-refs` isn't\n> run with `--all --prune` as it will delete the refs from linus.git\n> when executed in digispeaker.git.  Fun times when I did that to my\n> own repository one day.  ;-)\n> \n> Though we probably should fix Git to be somewhat smarter.  But\n> until that stable binary is available, the symlink trick above\n> is a good work around.\n\nBut it should be used only in controlled environment. If you happen to\nhave permissions to write to the forkee, you can wipe out its refs with\ngit fetch --mirror (and if you don't happen to have permissions, it will\njust fail, so currently you cannot use this on repo.or.cz forks,\nunfortunately). If you don't make sure refs are never packed (I do on\nrepo.or.cz, historically because of dumb transports - do they support\npacked refs by now?), this won't work either. Maybe there are other\nconsiderations too.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nThe next generation of interesting software will be done\non the Macintosh, not the IBM PC.  -- Bill Gates\n"},{"id":"90225","messageId":"20080909145714.GD10015@spearce.org","threadId":"15394","inReplyTo":"20080909083551.GA10544@machine.or.cz","subject":"Re: Alternates and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-09-09T14:57:14Z","receivedAt":"2008-09-09T14:57:14Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Petr Baudis <pasky@suse.cz> wrote:\n> On Sat, Sep 06, 2008 at 12:21:06PM -0700, Shawn O. Pearce wrote:\n> > github should do what repo.or.cz does:\n> > \n> > \tln -s .../linus.git/refs digispeaker.git/refs/forkee\n> \n> But it should be used only in controlled environment. If you happen to\n> have permissions to write to the forkee, you can wipe out its refs with\n> git fetch --mirror (and if you don't happen to have permissions, it will\n> just fail, so currently you cannot use this on repo.or.cz forks,\n> unfortunately). If you don't make sure refs are never packed (I do on\n> repo.or.cz, historically because of dumb transports - do they support\n> packed refs by now?), this won't work either. Maybe there are other\n> considerations too.\n\nAll very good points.\n\nThe dumb transports do support packed-refs in more modern versions.\nI forget when they learned it though.  Being nice to them by not\npacking refs is still good if you can.\n\n-- \nShawn.\n"}]}