{"thread":{"id":"25828","subject":"'git replace' and pushing","startedAt":"2010-11-24T04:33:59Z","lastAt":"2010-11-27T17:54:17Z","messageCount":10,"participants":["Cory Fields","Michael J Gruber","Jonathan Nieder","Junio C Hamano","Martin von Zweigbergk"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"156465","messageId":"AANLkTinzPCeCJ486cysmk981HE61=dv9MS7E8Ap4rQ5r@mail.gmail.com","threadId":"25828","inReplyTo":null,"subject":"'git replace' and pushing","fromName":"Cory Fields","fromEmail":"foss@atlastechnologiesinc.com","sentAt":"2010-11-24T04:33:59Z","receivedAt":"2010-11-24T04:33:59Z","isPatch":false,"sender":{"key":"foss@atlastechnologiesinc.com","avatar":"https://gravatar.com/avatar/62a322461af616bb53aba899996f7e3d5f05b7054f5edfa56de7d5c0dca326e3?d=mp&s=160"},"body":"I am having some trouble understanding how a replaced object (commit)\nshould behave when pushed to a remote repo. Here's my scenario:\n\nWe are moving from svn to git. Our svn repo is huge, and most of the\nhistory is useless. To save space, I would like to do a 50/50 split so\nthat when the repo is cloned, 50% is seen by default, and the\nhistorical 50% can be seen by fetching the replacement history. I've\ndone this by creating a phony snapshot at 3 then using a 'replace' to\nput the others on top. The history is purely linear.\n\n1---2---3---4---5\n                 \\---4---5\n\nWhen the replacement is in place, the repo is half size (commit-wise)\nas expected. The problem is that 'git push' does not honor the\nreplace. So when I push, all objects go with it, which defeats the\npurpose. The only way that seams to work is doing a filter-branch and\nreplacing the other way.\n\nIs this by design? I would really like a way to split the repo without\nbreaking hashes for the developers that have already begun using git\nsvn.\n\nThanks,\nCory\n"},{"id":"156535","messageId":"4CEE2060.4020507@drmicha.warpmail.net","threadId":"25828","inReplyTo":"AANLkTinzPCeCJ486cysmk981HE61=dv9MS7E8Ap4rQ5r@mail.gmail.com","subject":"Re: 'git replace' and pushing","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-11-25T08:37:52Z","receivedAt":"2010-11-25T08:37:52Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Cory Fields venit, vidit, dixit 24.11.2010 05:33:\n> I am having some trouble understanding how a replaced object (commit)\n> should behave when pushed to a remote repo. Here's my scenario:\n> \n> We are moving from svn to git. Our svn repo is huge, and most of the\n> history is useless. To save space, I would like to do a 50/50 split so\n> that when the repo is cloned, 50% is seen by default, and the\n> historical 50% can be seen by fetching the replacement history. I've\n> done this by creating a phony snapshot at 3 then using a 'replace' to\n> put the others on top. The history is purely linear.\n> \n> 1---2---3---4---5\n> \\---4---5\n\nI assume the \"other\" 4 goes off 3 (you're not using a monospaced font,\nare you?).\n\nAlso, the other 4 should have no parent, otherwise you've not cut-off\nany history.\n\n> \n> When the replacement is in place, the repo is half size (commit-wise)\n> as expected. The problem is that 'git push' does not honor the\n> replace. So when I push, all objects go with it, which defeats the\n> purpose. The only way that seams to work is doing a filter-branch and\n> replacing the other way.\n> \n> Is this by design? I would really like a way to split the repo without\n> breaking hashes for the developers that have already begun using git\n> svn.\n\nIt is by design since a replace creates a \"fake history\", and this\nshould not be created behind a users back.\n\nThe 5 is not rewritten, and it's ancestry contains the whole history. If\nthat is the commit your developers have already and that you want to\npreserve then there's not much you can do.\n\nYou could try to push or pull your replacement refs first (refs/replace)\nbut I don't think this will change what objects the push of 5 will\ntransfer. Just have a try.\n\nMichael\n"},{"id":"156711","messageId":"alpine.DEB.1.10.1011262114410.17721@debian","threadId":"25828","inReplyTo":"AANLkTi=FjSFLsbXf2Rp_Onm26yyxX+xSPrh2pB=_f5RU@mail.gmail.com","subject":"Re: 'git replace' and pushing","fromName":"Martin von Zweigbergk","fromEmail":"martin.von.zweigbergk@gmail.com","sentAt":"2010-11-26T20:29:30Z","receivedAt":"2010-11-26T20:29:30Z","isPatch":false,"sender":{"key":"martinvonz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/891642?v=4"},"body":"On Fri, 26 Nov 2010, Cory Fields wrote:\n\n> A shallow clone does not fit for us, because we want the default clone\n> to only pull half.\n> Having a public 1gb repository that will be cloned quite often is\n> bound to make our host\n> unhappy, so we're doing everything we can to get the size down.\n\nAt the GitTogether last month, my colleague brought up the subject of\nhow to cope with repositories growing over time. The conclusion from\nthe discussions was that shallow clones would proabably be the best\noption in general. FYI, even though it may not help you right now,\nhaving a default shallow clone depth configured in the repository on\nthe server was also discussed.\n\n/Martin\n"},{"id":"156693","messageId":"AANLkTimQ3fjPb+YVJ5i8EAgui+gd5rfnXMvdQPJPeUtA@mail.gmail.com","threadId":"25828","inReplyTo":"4CEE2060.4020507@drmicha.warpmail.net","subject":"Re: 'git replace' and pushing","fromName":"Cory Fields","fromEmail":"foss@atlastechnologiesinc.com","sentAt":"2010-11-26T21:16:39Z","receivedAt":"2010-11-26T21:16:39Z","isPatch":false,"sender":{"key":"foss@atlastechnologiesinc.com","avatar":"https://gravatar.com/avatar/62a322461af616bb53aba899996f7e3d5f05b7054f5edfa56de7d5c0dca326e3?d=mp&s=160"},"body":"On Thu, Nov 25, 2010 at 3:37 AM, Michael J Gruber\n wrote:\n> Cory Fields venit, vidit, dixit 24.11.2010 05:33:\n>> I am having some trouble understanding how a replaced object (commit)\n>> should behave when pushed to a remote repo. Here's my scenario:\n>>\n>> We are moving from svn to git. Our svn repo is huge, and most of the\n>> history is useless. To save space, I would like to do a 50/50 split so\n>> that when the repo is cloned, 50% is seen by default, and the\n>> historical 50% can be seen by fetching the replacement history. I've\n>> done this by creating a phony snapshot at 3 then using a 'replace' to\n>> put the others on top. The history is purely linear.\n>>\n>> 1---2---3---4---5\n>>                  \\---4---5\n>\n> I assume the \"other\" 4 goes off 3 (you're not using a monospaced font,\n> are you?).\n>\n\nI used a monospace font, but gmail decided not to use it. Sorry for that.\n\n> Also, the other 4 should have no parent, otherwise you've not cut-off\n> any history.\n\nI created a \"fake\" 4 that consists of the full working tree at 4 with no parent.\nAs I mentioned, everything looks fine locally.\n\n>\n>>\n>> When the replacement is in place, the repo is half size (commit-wise)\n>> as expected. The problem is that 'git push' does not honor the\n>> replace. So when I push, all objects go with it, which defeats the\n>> purpose. The only way that seams to work is doing a filter-branch and\n>> replacing the other way.\n>>\n>> Is this by design? I would really like a way to split the repo without\n>> breaking hashes for the developers that have already begun using git\n>> svn.\n>\n> It is by design since a replace creates a \"fake history\", and this\n> should not be created behind a users back.\n>\n> The 5 is not rewritten, and it's ancestry contains the whole history. If\n> that is the commit your developers have already and that you want to\n> preserve then there's not much you can do.\n>\n> You could try to push or pull your replacement refs first (refs/replace)\n> but I don't think this will change what objects the push of 5 will\n> transfer. Just have a try.\n\nI tried this to no avail.\n\nI realize that allowing replacements to be pushed \"behind users backs\", so\nI guess not respecting it makes sense.\n\nBut is there no way that I can pull this off without rewriting hashes?\n\nThanks,\nCory\n"},{"id":"156695","messageId":"20101126214325.GC23462@burratino","threadId":"25828","inReplyTo":"AANLkTimQ3fjPb+YVJ5i8EAgui+gd5rfnXMvdQPJPeUtA@mail.gmail.com","subject":"Re: 'git replace' and pushing","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-26T21:43:25Z","receivedAt":"2010-11-26T21:43:25Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Cory,\n\nCory Fields wrote:\n\n> I realize that allowing replacements to be pushed \"behind users backs\", so\n> I guess not respecting it makes sense.\n>\n> But is there no way that I can pull this off without rewriting hashes?\n\nThe usual way to accomplish what you are talking about would be like\nthis:\n\n Real history\n ------------\n 4' --- 5 --- 6\n\n 1 --- 2 --- 3 --- 4\n\n Fake history\n ------------\n 1 --- 2 --- 3 --- 4 --- 5 --- 6\n\n Replacement ref\n ---------------\n 4' --> 4\n\nThis way, a person a person can fetch either piece of real history\nwithout trouble, and if they fetch the replacement ref, too, the\nhistory is pasted together.\n\nIt is not possible in git to push a commit without its ancestors;\nreplacement refs do not change that. However, it is sort of possible\nto fetch a commit without its ancestors using the --depth option to\nclone and fetch.\n\nHope that helps,\nJonathan\n"},{"id":"156698","messageId":"7vvd3jptch.fsf@alter.siamese.dyndns.org","threadId":"25828","inReplyTo":"20101126214325.GC23462@burratino","subject":"Re: 'git replace' and pushing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-11-26T23:18:06Z","receivedAt":"2010-11-26T23:18:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Real history\n> ------------\n> 4' --- 5 --- 6\n>\n> 1 --- 2 --- 3 --- 4\n>\n> Fake history\n> ------------\n> 1 --- 2 --- 3 --- 4 --- 5 --- 6\n>\n> Replacement ref\n> ---------------\n> 4' --> 4\n>\n> This way, a person a person can fetch either piece of real history\n> without trouble, and if they fetch the replacement ref, too, the\n> history is pasted together.\n>\n> It is not possible in git to push a commit without its ancestors;\n> replacement refs do not change that.\n\nTrue, but I suspect the above picture pretty much satisfies Cory's initial\nwish, no? You can fetch recent 4'--5---6 history as if 4' were the root\ncommit, and if you fetched replacement that tells us to pretend that 4'\nhas 3 as its parent (and the history leading to 3), you will get a deeper\nhistory.\n"},{"id":"156709","messageId":"AANLkTi=FjSFLsbXf2Rp_Onm26yyxX+xSPrh2pB=_f5RU@mail.gmail.com","threadId":"25828","inReplyTo":"7vvd3jptch.fsf@alter.siamese.dyndns.org","subject":"Re: 'git replace' and pushing","fromName":"Cory Fields","fromEmail":"foss@atlastechnologiesinc.com","sentAt":"2010-11-27T01:58:03Z","receivedAt":"2010-11-27T01:58:03Z","isPatch":false,"sender":{"key":"foss@atlastechnologiesinc.com","avatar":"https://gravatar.com/avatar/62a322461af616bb53aba899996f7e3d5f05b7054f5edfa56de7d5c0dca326e3?d=mp&s=160"},"body":"On Fri, Nov 26, 2010 at 6:18 PM, Junio C Hamano wrote:\n> Jonathan Nieder writes:\n>\n>>  Real history\n>>  ------------\n>>  4' --- 5 --- 6\n>>\n>>  1 --- 2 --- 3 --- 4\n>>\n>>  Fake history\n>>  ------------\n>>  1 --- 2 --- 3 --- 4 --- 5 --- 6\n>>\n>>  Replacement ref\n>>  ---------------\n>>  4' --> 4\n>>\n>> This way, a person a person can fetch either piece of real history\n>> without trouble, and if they fetch the replacement ref, too, the\n>> history is pasted together.\n>>\n>> It is not possible in git to push a commit without its ancestors;\n>> replacement refs do not change that.\n>\n> True, but I suspect the above picture pretty much satisfies Cory's initial\n> wish, no?  You can fetch recent 4'--5---6 history as if 4' were the root\n> commit, and if you fetched replacement that tells us to pretend that 4'\n> has 3 as its parent (and the history leading to 3), you will get a deeper\n> history.\n>\n\nYes, both of these can be accomplished. I've managed to get that part\nworking, where a\ndefault clone pulls in half history, and fetching refs/replace gives\nyou the rest. The only\nproblem is that it requires a filter-branch before pushing. Otherwise,\n4 gets pushed rather\nthan 4', meaning that clones will require all the objects. So it\nworks, but I'll have to spend\nquite a while making it 'perfect' so that I only have to rewrite history once.\n\nA shallow clone does not fit for us, because we want the default clone\nto only pull half.\nHaving a public 1gb repository that will be cloned quite often is\nbound to make our host\nunhappy, so we're doing everything we can to get the size down.\n\nAlso, maybe I haven't made this clear... the \"real\" commit IDs need to\nmatch the \"fake\"\nones in order to prevent confusion. I think that's the part that makes\nthis so difficult.\nOtherwise, something like this [1] would work just fine (probably\nexactly what Junio was\nsuggesting)\n\nAny other suggestions? Or do I just have to face the fact that I'm\ngoing to have to break\nhashes?\n\n[1] http://progit.org/2010/03/17/replace.html\n\nCory\n"},{"id":"156710","messageId":"AANLkTikvbZwt0=Om98WZzPGCC71mLpXAz=N18x4ZQA2p@mail.gmail.com","threadId":"25828","inReplyTo":"AANLkTi=FjSFLsbXf2Rp_Onm26yyxX+xSPrh2pB=_f5RU@mail.gmail.com","subject":"Re: 'git replace' and pushing","fromName":"Cory Fields","fromEmail":"foss@atlastechnologiesinc.com","sentAt":"2010-11-27T01:59:16Z","receivedAt":"2010-11-27T01:59:16Z","isPatch":false,"sender":{"key":"foss@atlastechnologiesinc.com","avatar":"https://gravatar.com/avatar/62a322461af616bb53aba899996f7e3d5f05b7054f5edfa56de7d5c0dca326e3?d=mp&s=160"},"body":"On Fri, Nov 26, 2010 at 8:58 PM, Cory Fields\n wrote:\n> On Fri, Nov 26, 2010 at 6:18 PM, Junio C Hamano wrote:\n>> Jonathan Nieder writes:\n>>\n>>>  Real history\n>>>  ------------\n>>>  4' --- 5 --- 6\n>>>\n>>>  1 --- 2 --- 3 --- 4\n>>>\n>>>  Fake history\n>>>  ------------\n>>>  1 --- 2 --- 3 --- 4 --- 5 --- 6\n>>>\n>>>  Replacement ref\n>>>  ---------------\n>>>  4' --> 4\n>>>\n>>> This way, a person a person can fetch either piece of real history\n>>> without trouble, and if they fetch the replacement ref, too, the\n>>> history is pasted together.\n>>>\n>>> It is not possible in git to push a commit without its ancestors;\n>>> replacement refs do not change that.\n>>\n>> True, but I suspect the above picture pretty much satisfies Cory's initial\n>> wish, no?  You can fetch recent 4'--5---6 history as if 4' were the root\n>> commit, and if you fetched replacement that tells us to pretend that 4'\n>> has 3 as its parent (and the history leading to 3), you will get a deeper\n>> history.\n>>\n>\n> Yes, both of these can be accomplished. I've managed to get that part\n> working, where a\n> default clone pulls in half history, and fetching refs/replace gives\n> you the rest. The only\n> problem is that it requires a filter-branch before pushing. Otherwise,\n> 4 gets pushed rather\n> than 4', meaning that clones will require all the objects. So it\n> works, but I'll have to spend\n> quite a while making it 'perfect' so that I only have to rewrite history once.\n>\n> A shallow clone does not fit for us, because we want the default clone\n> to only pull half.\n> Having a public 1gb repository that will be cloned quite often is\n> bound to make our host\n> unhappy, so we're doing everything we can to get the size down.\n>\n> Also, maybe I haven't made this clear... the \"real\" commit IDs need to\n> match the \"fake\"\n> ones in order to prevent confusion. I think that's the part that makes\n> this so difficult.\n> Otherwise, something like this [1] would work just fine (probably\n> exactly what Junio was\n> suggesting)\n>\n> Any other suggestions? Or do I just have to face the fact that I'm\n> going to have to break\n> hashes?\n>\n> [1] http://progit.org/2010/03/17/replace.html\n>\n> Cory\n>\n\nSorry for the stupid wrapping.. gmail and I are not getting along in\nthis thread!\n\nCory\n"},{"id":"156718","messageId":"20101127075236.GB24433@burratino","threadId":"25828","inReplyTo":"AANLkTi=FjSFLsbXf2Rp_Onm26yyxX+xSPrh2pB=_f5RU@mail.gmail.com","subject":"Re: 'git replace' and pushing","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-27T07:52:36Z","receivedAt":"2010-11-27T07:52:36Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Cory Fields wrote:\n> On Fri, Nov 26, 2010 at 6:18 PM, Junio C Hamano wrote:\n\n>> True, but I suspect the above picture pretty much satisfies Cory's initial\n>> wish, no?  You can fetch recent 4'--5---6 history as if 4' were the root\n>> commit, and if you fetched replacement that tells us to pretend that 4'\n>> has 3 as its parent (and the history leading to 3), you will get a deeper\n>> history.\n>\n> Yes, both of these can be accomplished. I've managed to get that part\n> working, where a default clone pulls in half history, and fetching\n> refs/replace gives you the rest. The only problem is that it requires a\n> filter-branch before pushing.\n\nThat's a one-time thing, not per-push, right? A filter-branch would\nindeed be needed to transform the history\n\n 1 --- 2 --- 3 --- 4 --- 5' --- 6'\n\ninto\n\n 1 --- 2 --- 3 --- 4\n 4' --- 5 --- 6\n\nand that is unavoidable: the object names encode the entire list of\nancestors, you cannot push an object without its ancestors, etc.\nBut afterwards you can build on the history rooted at 4' and all\nshould be well, and you can use checkout --orphan to get a new\nroot when the current line of history is about to grow too long.\n\nIn other words, the distinction between real history and fake history\nis very relevant. Object transport only cares about the real history\n(barring bugs); if you want to tweak what objects get transferred, you\nreally need to rewrite the real history (or use --depth).\n\n> A shallow clone does not fit for us, because we want the default clone to\n> only pull half. Having a public 1gb repository that will be cloned quite\n> often is bound to make our host unhappy, so we're doing everything we can to\n> get the size down.\n\nWhy not publish a \"git bundle\" of the first 1gb using HTTP,\nBitTorrent, or some other cache-friendly protocol and use a hook to\nreject attempts to fetch too many objects at once from the host?\n\n> Also, maybe I haven't made this clear... the \"real\" commit IDs need to\n> match the \"fake\" ones in order to prevent confusion.\n\nNot sure what this means. But commit IDs are defined based on\ncontent, and for simplicity and sanity the object transport machinery\ndeliberately does not look beyond that.\n\nRegards,\nJonathan\n"},{"id":"156734","messageId":"AANLkTikQfQsGrh27Bsegm97E706BVUto7Bp5QQ=sr0J2@mail.gmail.com","threadId":"25828","inReplyTo":"20101127075236.GB24433@burratino","subject":"Re: 'git replace' and pushing","fromName":"Cory Fields","fromEmail":"foss@atlastechnologiesinc.com","sentAt":"2010-11-27T17:54:17Z","receivedAt":"2010-11-27T17:54:17Z","isPatch":false,"sender":{"key":"foss@atlastechnologiesinc.com","avatar":"https://gravatar.com/avatar/62a322461af616bb53aba899996f7e3d5f05b7054f5edfa56de7d5c0dca326e3?d=mp&s=160"},"body":"On Sat, Nov 27, 2010 at 2:52 AM, Jonathan Nieder wrote:\n> Cory Fields wrote:\n>> On Fri, Nov 26, 2010 at 6:18 PM, Junio C Hamano wrote:\n>\n>>> True, but I suspect the above picture pretty much satisfies Cory's initial\n>>> wish, no?  You can fetch recent 4'--5---6 history as if 4' were the root\n>>> commit, and if you fetched replacement that tells us to pretend that 4'\n>>> has 3 as its parent (and the history leading to 3), you will get a deeper\n>>> history.\n>>\n>> Yes, both of these can be accomplished. I've managed to get that part\n>> working, where a default clone pulls in half history, and fetching\n>> refs/replace gives you the rest. The only problem is that it requires a\n>> filter-branch before pushing.\n>\n> That's a one-time thing, not per-push, right?  A filter-branch would\n> indeed be needed to transform the history\n>\n>  1 --- 2 --- 3 --- 4 --- 5' --- 6'\n>\n> into\n>\n>  1 --- 2 --- 3 --- 4\n>  4' --- 5 --- 6\n>\n> and that is unavoidable: the object names encode the entire list of\n> ancestors, you cannot push an object without its ancestors, etc.\n> But afterwards you can build on the history rooted at 4' and all\n> should be well, and you can use checkout --orphan to get a new\n> root when the current line of history is about to grow too long.\n>\n> In other words, the distinction between real history and fake history\n> is very relevant.  Object transport only cares about the real history\n> (barring bugs); if you want to tweak what objects get transferred, you\n> really need to rewrite the real history (or use --depth).\n>\n>> A shallow clone does not fit for us, because we want the default clone to\n>> only pull half.  Having a public 1gb repository that will be cloned quite\n>> often is bound to make our host unhappy, so we're doing everything we can to\n>> get the size down.\n>\n> Why not publish a \"git bundle\" of the first 1gb using HTTP,\n> BitTorrent, or some other cache-friendly protocol and use a hook to\n> reject attempts to fetch too many objects at once from the host?\n>\n>> Also, maybe I haven't made this clear... the \"real\" commit IDs need to\n>> match the \"fake\" ones in order to prevent confusion.\n>\n> Not sure what this means.  But commit IDs are defined based on\n> content, and for simplicity and sanity the object transport machinery\n> deliberately does not look beyond that.\n>\n> Regards,\n> Jonathan\n>\n\nI think a one-time filter-branch is going to be our best bet. I had\nassumed that this was the case, I just wanted reassurance that it was\nnecessary. I have that now. Thanks to all for the responses.\n\nMartin: That sounds very interesting indeed. However, the docs make\nshallow clones sound scary. From the docs: \"A shallow repository has a\nnumber of limitations (you cannot clone or fetch from it, nor push\nfrom nor into it)\"\n\nI suppose these limitations would need to be addressed if/when looking\ninto serverside depth defaults?\n\nCory\n"}]}