{"thread":{"id":"37845","subject":"Joining historical repository using grafts or replace","startedAt":"2014-10-30T15:39:56Z","lastAt":"2014-11-01T15:03:06Z","messageCount":8,"participants":["Dmitry Oksenchuk","W. Trevor King","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"251217","messageId":"CA+POfmvCiNBF=P-OvQBTROVhaLtOdgNTDgPNyS=97bupSGk=4g@mail.gmail.com","threadId":"37845","inReplyTo":null,"subject":"Joining historical repository using grafts or replace","fromName":"Dmitry Oksenchuk","fromEmail":"oksenchuk89@gmail.com","sentAt":"2014-10-30T15:39:56Z","receivedAt":"2014-10-30T15:39:56Z","isPatch":false,"sender":{"key":"oksenchuk89@gmail.com","avatar":null},"body":"Hello,\n\nWe're in the middle of conversion of a large CVS repository (20 years,\n70K commits, 1K branches, 10K tags) to Git and considering two\nseparate Git repositories: \"historical\" with CVS history and \"working\"\ncreated without history from heads of active branches (10 active\nbranches). This allows us to have small fast \"working\" repository for\ndevelopers who don't want to have full history locally and ability to\nrewrite history in \"historical\" repository (for example, to add\nparents to merge commits or to fix conversion mistakes) without\naffecting commit hashes in \"working\" repository (the hashes can be\nstored in bug tracker or in the code).\n\nThe first idea was to use grafs to join branch roots in \"working\"\nrepository with branches in \"historical\" repository like in linux\nrepository but it seems that grafts are known as a \"horrible hack\" (\nhttp://marc.info/?l=git&m=131127600030310&w=2\nhttp://permalink.gmane.org/gmane.comp.version-control.git/177153 )\n\nSince Git 1.6.5 \"replace\" can also be used to join the histories by\nreplacing branch roots in \"working\" repository with branch heads in\n\"historical\" repository.\n\nBoth grafts and replace will be used locally. Grafts is a bit easier\nto distribute (simple copying, replaces should be created via bash\nscript).\n\nAre there any disadvantages of using grafts and replace? Will both of\nthem be supported in future versions of Git?\n\nThank you,\nDmitry\n"},{"id":"251218","messageId":"20141030154426.GU15443@odin.tremily.us","threadId":"37845","inReplyTo":"CA+POfmvCiNBF=P-OvQBTROVhaLtOdgNTDgPNyS=97bupSGk=4g@mail.gmail.com","subject":"Re: Joining historical repository using grafts or replace","fromName":"W. Trevor King","fromEmail":"wking@tremily.us","sentAt":"2014-10-30T15:44:26Z","receivedAt":"2014-10-30T15:44:26Z","isPatch":false,"sender":{"key":"wking@tremily.us","avatar":"https://avatars.githubusercontent.com/u/209920?v=4"},"body":"On Thu, Oct 30, 2014 at 06:39:56PM +0300, Dmitry Oksenchuk wrote:\n> We're in the middle of conversion of a large CVS repository (20\n> years, 70K commits, 1K branches, 10K tags) to Git and considering\n> two separate Git repositories: \"historical\" with CVS history and\n> \"working\" created without history from heads of active branches (10\n> active branches). This allows us to have small fast \"working\"\n> repository for developers who don't want to have full history\n> locally and ability to rewrite history in \"historical\" repository\n> (for example, to add parents to merge commits or to fix conversion\n> mistakes) without affecting commit hashes in \"working\" repository\n> (the hashes can be stored in bug tracker or in the code).\n\nA number of projects have done something like this (e.g. Linux).\nModern Gits have good support for shallow repositories though, so I'd\njust make one full repository and leave it to developers to decide how\ndeep they want their local copy to be.\n\nCheers,\nTrevor\n\n-- \nThis email may be signed or encrypted with GnuPG (http://www.gnupg.org).\nFor more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy\n"},{"id":"251221","messageId":"CAP8UFD3_fAWRdxQgAbfxYZSzrmy1Aza=nuZh-uSJsKOdRj+LVA@mail.gmail.com","threadId":"37845","inReplyTo":"CA+POfmvCiNBF=P-OvQBTROVhaLtOdgNTDgPNyS=97bupSGk=4g@mail.gmail.com","subject":"Re: Joining historical repository using grafts or replace","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2014-10-30T16:54:54Z","receivedAt":"2014-10-30T16:54:54Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Thu, Oct 30, 2014 at 4:39 PM, Dmitry Oksenchuk wrote:\n> Hello,\n>\n> We're in the middle of conversion of a large CVS repository (20 years,\n> 70K commits, 1K branches, 10K tags) to Git and considering two\n> separate Git repositories: \"historical\" with CVS history and \"working\"\n> created without history from heads of active branches (10 active\n> branches). This allows us to have small fast \"working\" repository for\n> developers who don't want to have full history locally and ability to\n> rewrite history in \"historical\" repository (for example, to add\n> parents to merge commits or to fix conversion mistakes) without\n> affecting commit hashes in \"working\" repository (the hashes can be\n> stored in bug tracker or in the code).\n\nThis might be a good idea. Did you already test that the small\nrepository is really faster than the full repository?\n\n> The first idea was to use grafs to join branch roots in \"working\"\n> repository with branches in \"historical\" repository like in linux\n> repository but it seems that grafts are known as a \"horrible hack\" (\n> http://marc.info/?l=git&m=131127600030310&w=2\n> http://permalink.gmane.org/gmane.comp.version-control.git/177153 )\n>\n> Since Git 1.6.5 \"replace\" can also be used to join the histories by\n> replacing branch roots in \"working\" repository with branch heads in\n> \"historical\" repository.\n>\n> Both grafts and replace will be used locally. Grafts is a bit easier\n> to distribute (simple copying, replaces should be created via bash\n> script).\n\nFirst, you might want to have a look at:\n\nhttp://git-scm.com/book/en/v2/Git-Tools-Replace\n\nas it looks like it describes your use case very well.\n\n> Are there any disadvantages of using grafts and replace? Will both of\n> them be supported in future versions of Git?\n\nMy opinion is that grafts have no advantage compared to replace refs.\n\nOnce you have created your replace refs, they can be managed like\nother git refs, so they are easier to distribute.\n\nBasically if you want to get the full history on a computer you just need to do:\n\ngit fetch 'refs/replace/*:refs/replace/*'\n\nBest,\nChristian.\n"},{"id":"251223","messageId":"CA+POfmvXEjDV9Vap6NDX7HvOMjEVG4mVe1uWFSTQy5g_c+vJnw@mail.gmail.com","threadId":"37845","inReplyTo":"CAP8UFD3_fAWRdxQgAbfxYZSzrmy1Aza=nuZh-uSJsKOdRj+LVA@mail.gmail.com","subject":"Re: Joining historical repository using grafts or replace","fromName":"Dmitry Oksenchuk","fromEmail":"oksenchuk89@gmail.com","sentAt":"2014-10-30T17:41:37Z","receivedAt":"2014-10-30T17:41:37Z","isPatch":false,"sender":{"key":"oksenchuk89@gmail.com","avatar":null},"body":"Hi Christian,\n\nThanks for your reply.\n\n2014-10-30 19:54 GMT+03:00 Christian Couder :\n> On Thu, Oct 30, 2014 at 4:39 PM, Dmitry Oksenchuk wrote:\n>> We're in the middle of conversion of a large CVS repository (20 years,\n>> 70K commits, 1K branches, 10K tags) to Git and considering two\n>> separate Git repositories: \"historical\" with CVS history and \"working\"\n>> created without history from heads of active branches (10 active\n>> branches). This allows us to have small fast \"working\" repository for\n>> developers who don't want to have full history locally and ability to\n>> rewrite history in \"historical\" repository (for example, to add\n>> parents to merge commits or to fix conversion mistakes) without\n>> affecting commit hashes in \"working\" repository (the hashes can be\n>> stored in bug tracker or in the code).\n>\n> This might be a good idea. Did you already test that the small\n> repository is really faster than the full repository?\n\nYes, because of such amount of refs, push in \"historical\" repository\ntakes 12 sec, push in \"working\" repository takes 0.4 sec, push in\n\"joined\" repository takes 2 sec. Local operations with history like\nlog and blame work with the same speed in \"joined\" repository as in\n\"historical\" repository.\n\n>> Are there any disadvantages of using grafts and replace? Will both of\n>> them be supported in future versions of Git?\n>\n> My opinion is that grafts have no advantage compared to replace refs.\n>\n> Once you have created your replace refs, they can be managed like\n> other git refs, so they are easier to distribute.\n>\n> Basically if you want to get the full history on a computer you just need to do:\n>\n> git fetch 'refs/replace/*:refs/replace/*'\n\nThat's true but you still need to have another remote with full\nhistory because it has lots of tags and branches that will be cloned\nby initial clone.\n\nRegards,\nDmitry\n"},{"id":"251225","messageId":"CA+POfms4+fPnEv2nBPJx+hPRSMZyR66wpvfDf+SFy3X_8t2H-Q@mail.gmail.com","threadId":"37845","inReplyTo":"20141030154426.GU15443@odin.tremily.us","subject":"Re: Joining historical repository using grafts or replace","fromName":"Dmitry Oksenchuk","fromEmail":"oksenchuk89@gmail.com","sentAt":"2014-10-30T17:56:54Z","receivedAt":"2014-10-30T17:56:54Z","isPatch":false,"sender":{"key":"oksenchuk89@gmail.com","avatar":null},"body":"2014-10-30 18:44 GMT+03:00 W. Trevor King :\n> On Thu, Oct 30, 2014 at 06:39:56PM +0300, Dmitry Oksenchuk wrote:\n>> We're in the middle of conversion of a large CVS repository (20\n>> years, 70K commits, 1K branches, 10K tags) to Git and considering\n>> two separate Git repositories: \"historical\" with CVS history and\n>> \"working\" created without history from heads of active branches (10\n>> active branches). This allows us to have small fast \"working\"\n>> repository for developers who don't want to have full history\n>> locally and ability to rewrite history in \"historical\" repository\n>> (for example, to add parents to merge commits or to fix conversion\n>> mistakes) without affecting commit hashes in \"working\" repository\n>> (the hashes can be stored in bug tracker or in the code).\n>\n> A number of projects have done something like this (e.g. Linux).\n> Modern Gits have good support for shallow repositories though, so I'd\n> just make one full repository and leave it to developers to decide how\n> deep they want their local copy to be.\n\nGood point. Shallow clone allows a developer to have a small fast\nrepository if history is not needed.\nBut having new history in one repository with CVS history prevents us\nfrom rewriting it in case of conversion mistakes or desire to restore\nparents in merge commits.\n\nThanks,\nDmitry\n"},{"id":"251261","messageId":"CAP8UFD0ZN1Dq3kAs=nQE3fY8RydqUhSDvYksy7J2igTpt9p5WQ@mail.gmail.com","threadId":"37845","inReplyTo":"CA+POfmvXEjDV9Vap6NDX7HvOMjEVG4mVe1uWFSTQy5g_c+vJnw@mail.gmail.com","subject":"Re: Joining historical repository using grafts or replace","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2014-10-31T08:45:43Z","receivedAt":"2014-10-31T08:45:43Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi Dmitry,\n\nOn Thu, Oct 30, 2014 at 6:41 PM, Dmitry Oksenchuk wrote:\n> 2014-10-30 19:54 GMT+03:00 Christian Couder :\n>>\n>> This might be a good idea. Did you already test that the small\n>> repository is really faster than the full repository?\n>\n> Yes, because of such amount of refs, push in \"historical\" repository\n> takes 12 sec, push in \"working\" repository takes 0.4 sec, push in\n> \"joined\" repository takes 2 sec. Local operations with history like\n> log and blame work with the same speed in \"joined\" repository as in\n> \"historical\" repository.\n\nWhat does \"joined\" mean? Does it mean joined using grafts? Or joined\nusing replace refs? Or just the unsplit full repository?\n\nAlso what is interesting is if local operations work with the same\nspeed in the small \"working\" repository as in the unsplit full\nrepository.\n\n>>> Are there any disadvantages of using grafts and replace? Will both of\n>>> them be supported in future versions of Git?\n>>\n>> My opinion is that grafts have no advantage compared to replace refs.\n>>\n>> Once you have created your replace refs, they can be managed like\n>> other git refs, so they are easier to distribute.\n>>\n>> Basically if you want to get the full history on a computer you just need to do:\n>>\n>> git fetch 'refs/replace/*:refs/replace/*'\n\nBy the way the above should be:\n\ngit fetch origin 'refs/replace/*:refs/replace/*'\n\n> That's true but you still need to have another remote with full\n> history because it has lots of tags and branches that will be cloned\n> by initial clone.\n\nYeah, you might want to have another remote for that reason, but this\nis true with both grafts and replace refs.\n\nBest,\nChristian.\n"},{"id":"251267","messageId":"CA+POfmt9cQcKshc3LgGpBqHdbsgjoN7Ou7eN3sGqJcDeXwgPyA@mail.gmail.com","threadId":"37845","inReplyTo":"CAP8UFD0ZN1Dq3kAs=nQE3fY8RydqUhSDvYksy7J2igTpt9p5WQ@mail.gmail.com","subject":"Re: Joining historical repository using grafts or replace","fromName":"Dmitry Oksenchuk","fromEmail":"oksenchuk89@gmail.com","sentAt":"2014-10-31T15:47:40Z","receivedAt":"2014-10-31T15:47:40Z","isPatch":false,"sender":{"key":"oksenchuk89@gmail.com","avatar":null},"body":"Hi Christian,\n\n> On Thu, Oct 30, 2014 at 6:41 PM, Dmitry Oksenchuk wrote:\n>> 2014-10-30 19:54 GMT+03:00 Christian Couder :\n>>>\n>>> This might be a good idea. Did you already test that the small\n>>> repository is really faster than the full repository?\n>>\n>> Yes, because of such amount of refs, push in \"historical\" repository\n>> takes 12 sec, push in \"working\" repository takes 0.4 sec, push in\n>> \"joined\" repository takes 2 sec. Local operations with history like\n>> log and blame work with the same speed in \"joined\" repository as in\n>> \"historical\" repository.\n>\n> What does \"joined\" mean? Does it mean joined using grafts? Or joined\n> using replace refs? Or just the unsplit full repository?\n\nIt's joined using grafts or replace. In both cases performance is the same.\n\n> Also what is interesting is if local operations work with the same\n> speed in the small \"working\" repository as in the unsplit full\n> repository.\n\nSpeed of operations like git diff, git add, git commit is exactly the\nsame in both repositories.\nOperations like git log and git blame work much faster in repository\nwithout history (not surprisingly :)\nFor example, git log in small repository takes 0.2 sec, in full\nrepository - 0.8 sec. git blame in full repository can take up to 9\nsec for large files with long history.\n\nRegards,\nDmitry\n"},{"id":"251287","messageId":"CAP8UFD17DOajgFWTTDC12qz3m4wKJTn2XLn+MWUE8Fc_XyTBTQ@mail.gmail.com","threadId":"37845","inReplyTo":"CA+POfmt9cQcKshc3LgGpBqHdbsgjoN7Ou7eN3sGqJcDeXwgPyA@mail.gmail.com","subject":"Re: Joining historical repository using grafts or replace","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2014-11-01T15:03:06Z","receivedAt":"2014-11-01T15:03:06Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi Dmitry,\n\nOn Fri, Oct 31, 2014 at 4:47 PM, Dmitry Oksenchuk wrote:\n> Hi Christian,\n>\n>> On Thu, Oct 30, 2014 at 6:41 PM, Dmitry Oksenchuk wrote:\n>>>\n>>> Yes, because of such amount of refs, push in \"historical\" repository\n>>> takes 12 sec, push in \"working\" repository takes 0.4 sec, push in\n>>> \"joined\" repository takes 2 sec. Local operations with history like\n>>> log and blame work with the same speed in \"joined\" repository as in\n>>> \"historical\" repository.\n>>\n>> What does \"joined\" mean? Does it mean joined using grafts? Or joined\n>> using replace refs? Or just the unsplit full repository?\n>\n> It's joined using grafts or replace. In both cases performance is the same.\n>\n>> Also what is interesting is if local operations work with the same\n>> speed in the small \"working\" repository as in the unsplit full\n>> repository.\n>\n> Speed of operations like git diff, git add, git commit is exactly the\n> same in both repositories.\n> Operations like git log and git blame work much faster in repository\n> without history (not surprisingly :)\n> For example, git log in small repository takes 0.2 sec, in full\n> repository - 0.8 sec. git blame in full repository can take up to 9\n> sec for large files with long history.\n\nOk, thanks for the information. I think it shows that indeed it makes\nsense to split your repo.\n\nBest,\nChristian.\n"}]}