{"thread":{"id":"15544","subject":"Grafts workflow for a \"shallow\" repository...?","startedAt":"2008-09-16T05:09:14Z","lastAt":"2008-09-16T23:12:32Z","messageCount":9,"participants":["Martin Langhoff","Shawn O. Pearce","Junio C Hamano","Björn Steinbrink","Michael J Gruber","Sergio","Sverre Rabbelier","Dmitry Potapov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"90801","messageId":"46a038f90809152209l2230d9e3o442dac1f5047d2bd@mail.gmail.com","threadId":"15544","inReplyTo":null,"subject":"Grafts workflow for a \"shallow\" repository...?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2008-09-16T05:09:14Z","receivedAt":"2008-09-16T05:09:14Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Here is my attempt at a \"let's publish a shallow repository for branch\nof moodle\". Let me show you what I did...\n\n # before we start\n git --version\n git version 1.5.5.1\n\n cat /etc/redhat-release\n Fedora release 9 (Sulphur)\n\n # clone the 'full' repo ~200MB fully packed\n git clone git://git.catalyst.net.nz/moodle-r2.git\n\n # 1.7 was a significant release, anything earlier than that\n # is just not interesting -- even for pickaxe/annotate purposes\n # so add a graft point right at the branching point.\n cd moodle-r2\n echo `git merge-base origin/cvshead origin/MOODLE_17_STABLE` >>\n.git/info/grafts\n\n # push the branches I care about to a newly created repo on a different server\n # this is approx 50MB... (as opposed to 200MB)\n git push git+ssh://dev.laptop.org/home/martin/public_git/moodle.git \\\n origin/MOODLE_19_STABLE:refs/heads/MOODLE_19_STABLE\n git push git+ssh://dev.laptop.org/home/martin/public_git/moodle.git \\\n origin/cvshead:refs/heads/cvshead\n # fixed up the server-side repo to have HEAD pointing to a new branch...\n\n # clone to see it works...\n git clone git://dev.laptop.org/git/users/martin/moodle.git\n\n # and now gitk bails out on me. Not entirely unexpected...\n gitk --all\n\nIs this kind of workflow (or a variation of it) supported? For this to\nwork, we should communicate the grafts in some push operations and\nread them in clone ops - and perhaps in fetch too.\n\ncheers,\n\n\n\nmartin\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"90802","messageId":"20080916052425.GA14644@spearce.org","threadId":"15544","inReplyTo":"46a038f90809152209l2230d9e3o442dac1f5047d2bd@mail.gmail.com","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-09-16T05:24:25Z","receivedAt":"2008-09-16T05:24:25Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Martin Langhoff wrote:\n> Here is my attempt at a \"let's publish a shallow repository for branch\n> of moodle\". Let me show you what I did...\n...\n> # 1.7 was a significant release, anything earlier than that\n> # is just not interesting -- even for pickaxe/annotate purposes\n> # so add a graft point right at the branching point.\n...\n> Is this kind of workflow (or a variation of it) supported? For this to\n> work, we should communicate the grafts in some push operations and\n> read them in clone ops - and perhaps in fetch too.\n\nCurrently the grafts file isn't transferred over any transport\nprotocol as it is considered to be local only to the repository.\n\nFor one thing, grafts are a security risk. Any user can graft\nanything in at any position and log/blame operations will honor\nthe graft, rather than what is stored in the signed commit chain.\nIts a low risk, but it allows a peer to lie to you and give you\nsomething other than what you asked for.\n\nPushing (or fetching) a graft just seems ugly. You don't want\nto fetch a graft if you have no grafts because you have the full\nhistory. Nor do you want to fetch a graft that might possibly\noverwrite/replace a graft you already have. You might not want to\npush all of your available grafts to a remote. Etc.\n\nI think that in this case the best thing to do is give users\na shell script that does roughly:\n\n\tgit init\n\techo $BASE >.git/info/grafts\n\tgit remote add -f origin $url\n\tgit checkout -b master origin/master\n\nSign the script, and let users verify it before executing. You may\nalso want a script to drag in the history behind by removing the\ngraft and fetching $BASE^, but that is hard because your repository\nalready \"has\" that.\n\n-- \nShawn.\n"},{"id":"90808","messageId":"7v3ak08v5l.fsf@gitster.siamese.dyndns.org","threadId":"15544","inReplyTo":"20080916052425.GA14644@spearce.org","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-16T06:25:10Z","receivedAt":"2008-09-16T06:25:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Martin Langhoff wrote:\n>> Here is my attempt at a \"let's publish a shallow repository for branch\n>> of moodle\". Let me show you what I did...\n> ...\n>> # 1.7 was a significant release, anything earlier than that\n>> # is just not interesting -- even for pickaxe/annotate purposes\n>> # so add a graft point right at the branching point.\n> ...\n>> Is this kind of workflow (or a variation of it) supported? For this to\n>> work, we should communicate the grafts in some push operations and\n>> read them in clone ops - and perhaps in fetch too.\n> ...\n> I think that in this case the best thing to do is give users\n> a shell script that does roughly:\n>\n> \tgit init\n> \techo $BASE >.git/info/grafts\n> \tgit remote add -f origin $url\n> \tgit checkout -b master origin/master\n>\n> Sign the script, and let users verify it before executing. You may\n> also want a script to drag in the history behind by removing the\n> graft and fetching $BASE^, but that is hard because your repository\n> already \"has\" that.\n\nWhy not just filter-branch _once at the origin_ and publish the result?\n"},{"id":"90819","messageId":"20080916080908.GA14272@atjola.homenet","threadId":"15544","inReplyTo":"7v3ak08v5l.fsf@gitster.siamese.dyndns.org","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-09-16T08:09:08Z","receivedAt":"2008-09-16T08:09:08Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > Martin Langhoff wrote:\n> >> Here is my attempt at a \"let's publish a shallow repository for branch\n> >> of moodle\". Let me show you what I did...\n> > ...\n> >> # 1.7 was a significant release, anything earlier than that\n> >> # is just not interesting -- even for pickaxe/annotate purposes\n> >> # so add a graft point right at the branching point.\n> > ...\n> >> Is this kind of workflow (or a variation of it) supported? For this to\n> >> work, we should communicate the grafts in some push operations and\n> >> read them in clone ops - and perhaps in fetch too.\n> > ...\n> > I think that in this case the best thing to do is give users\n> > a shell script that does roughly:\n> >\n> > \tgit init\n> > \techo $BASE >.git/info/grafts\n> > \tgit remote add -f origin $url\n> > \tgit checkout -b master origin/master\n> >\n> > Sign the script, and let users verify it before executing. You may\n> > also want a script to drag in the history behind by removing the\n> > graft and fetching $BASE^, but that is hard because your repository\n> > already \"has\" that.\n> \n> Why not just filter-branch _once at the origin_ and publish the result?\n\nI think the idea was to have a shallow clone starting at a certain\npoint, as opposed to the --depth option, where you cannot specify a\nstarting point, but only the depth of the clone.\n\nBjörn\n"},{"id":"90832","messageId":"48CFB44F.8060609@drmicha.warpmail.net","threadId":"15544","inReplyTo":"20080916080908.GA14272@atjola.homenet","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-09-16T13:27:43Z","receivedAt":"2008-09-16T13:27:43Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Björn Steinbrink venit, vidit, dixit 16.09.2008 10:09:\n> On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:\n>> \"Shawn O. Pearce\" writes:\n>>\n>>> Martin Langhoff wrote:\n>>>> Here is my attempt at a \"let's publish a shallow repository for branch\n>>>> of moodle\". Let me show you what I did...\n>>> ...\n>>>> # 1.7 was a significant release, anything earlier than that\n>>>> # is just not interesting -- even for pickaxe/annotate purposes\n>>>> # so add a graft point right at the branching point.\n>>> ...\n>>>> Is this kind of workflow (or a variation of it) supported? For this to\n>>>> work, we should communicate the grafts in some push operations and\n>>>> read them in clone ops - and perhaps in fetch too.\n>>> ...\n>>> I think that in this case the best thing to do is give users\n>>> a shell script that does roughly:\n>>>\n>>> \tgit init\n>>> \techo $BASE >.git/info/grafts\n>>> \tgit remote add -f origin $url\n>>> \tgit checkout -b master origin/master\n>>>\n>>> Sign the script, and let users verify it before executing. You may\n>>> also want a script to drag in the history behind by removing the\n>>> graft and fetching $BASE^, but that is hard because your repository\n>>> already \"has\" that.\n>> Why not just filter-branch _once at the origin_ and publish the result?\n> \n> I think the idea was to have a shallow clone starting at a certain\n> point, as opposed to the --depth option, where you cannot specify a\n> starting point, but only the depth of the clone.\n\nThat's what Junio suggests:\n\nchop the history by introducing an appropriate graft\nmake it \"permanent\" by filter-branching (and remove from info/grafts).\n\nNow you have a disconnect dag. clone/push/whatever works on the\n\"components\" given by connected branches.\n\nNote that in this approach all history after the \"chopping point\" will\nbe rewritten, i.e. get new sha1's.\n\nMichael\n"},{"id":"90835","messageId":"20080916135022.GA19466@atjola.homenet","threadId":"15544","inReplyTo":"48CFB44F.8060609@drmicha.warpmail.net","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-09-16T13:50:22Z","receivedAt":"2008-09-16T13:50:22Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.09.16 15:27:43 +0200, Michael J Gruber wrote:\n> Björn Steinbrink venit, vidit, dixit 16.09.2008 10:09:\n> > On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:\n> >> \"Shawn O. Pearce\" writes:\n> >>\n> >>> Martin Langhoff wrote:\n> >>>> Here is my attempt at a \"let's publish a shallow repository for branch\n> >>>> of moodle\". Let me show you what I did...\n> >>> ...\n> >>>> # 1.7 was a significant release, anything earlier than that\n> >>>> # is just not interesting -- even for pickaxe/annotate purposes\n> >>>> # so add a graft point right at the branching point.\n> >>> ...\n> >>>> Is this kind of workflow (or a variation of it) supported? For this to\n> >>>> work, we should communicate the grafts in some push operations and\n> >>>> read them in clone ops - and perhaps in fetch too.\n> >>> ...\n> >>> I think that in this case the best thing to do is give users\n> >>> a shell script that does roughly:\n> >>>\n> >>> \tgit init\n> >>> \techo $BASE >.git/info/grafts\n> >>> \tgit remote add -f origin $url\n> >>> \tgit checkout -b master origin/master\n> >>>\n> >>> Sign the script, and let users verify it before executing. You may\n> >>> also want a script to drag in the history behind by removing the\n> >>> graft and fetching $BASE^, but that is hard because your repository\n> >>> already \"has\" that.\n> >> Why not just filter-branch _once at the origin_ and publish the result?\n> > \n> > I think the idea was to have a shallow clone starting at a certain\n> > point, as opposed to the --depth option, where you cannot specify a\n> > starting point, but only the depth of the clone.\n> \n> That's what Junio suggests:\n> \n> chop the history by introducing an appropriate graft\n> make it \"permanent\" by filter-branching (and remove from info/grafts).\n> \n> Now you have a disconnect dag. clone/push/whatever works on the\n> \"components\" given by connected branches.\n> \n> Note that in this approach all history after the \"chopping point\" will\n> be rewritten, i.e. get new sha1's.\n\nBut shallow clones using --depth don't get alterted hashed. They have\n\"graft-like\" entries in .git/shallow (or whatever it is called) instead.\n\nAssuming that using a shallow clone is worth the hassle (ie. you have to\nfetch significantly less data than with a full clone), I can imagine\nsome use-cases for that, for example bisecting a bug of which you know\nthat it was introduced after the last release. Just shallow clone the\nrepo from that version's tag onwards and bisect locally, while still\nbeing able to use the original hash to report the broken version. Or\ngenrally anything for which you need a recent part of the history and\nwhich requires you to talk about certain commits. The hashes are useful\nfor that, and filter-branch'ing breaks that.\n\nMaybe instead of providing \"pre-shallowed\" repos to clone from, that\nshould have been an RFE to support shallow clones starting at a given\ncommit in the DAG? I have no idea whether that's feasible though.\n\nBjörn\n"},{"id":"90858","messageId":"loom.20080916T190745-332@post.gmane.org","threadId":"15544","inReplyTo":"20080916052425.GA14644@spearce.org","subject":"Re: Grafts workflow for a","fromName":"Sergio","fromEmail":"sergio.callegari@gmail.com","sentAt":"2008-09-16T19:12:51Z","receivedAt":"2008-09-16T19:12:51Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Shawn O. Pearce spearce.org> writes:\n\n> \n> Martin Langhoff gmail.com> wrote:\n> > Here is my attempt at a \"let's publish a shallow repository for branch\n> > of moodle\". Let me show you what I did...\n> ...\n> > # 1.7 was a significant release, anything earlier than that\n> > # is just not interesting -- even for pickaxe/annotate purposes\n> > # so add a graft point right at the branching point.\n> ...\n> > Is this kind of workflow (or a variation of it) supported? For this to\n> > work, we should communicate the grafts in some push operations and\n> > read them in clone ops - and perhaps in fetch too.\n> \n> Currently the grafts file isn't transferred over any transport\n> protocol as it is considered to be local only to the repository.\n> \n> For one thing, grafts are a security risk. Any user can graft\n> anything in at any position and log/blame operations will honor\n> the graft, rather than what is stored in the signed commit chain.\n> Its a low risk, but it allows a peer to lie to you and give you\n> something other than what you asked for.\n> \n\nWould it make sense to differentiate two types of grafts, just like we\ndifferentiate two kind of tags?\n\nThen there could be \"annotated grafts objects\" that could be optionally signed\nand that would get transferred on clones, fetches, pushes, etc.\n\nSergio\n"},{"id":"90866","messageId":"bd6139dc0809161502t121c5aa5la53784bb8ff273a2@mail.gmail.com","threadId":"15544","inReplyTo":"20080916135022.GA19466@atjola.homenet","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Sverre Rabbelier","fromEmail":"alturin@gmail.com","sentAt":"2008-09-16T22:02:25Z","receivedAt":"2008-09-16T22:02:25Z","isPatch":false,"sender":{"key":"alturin@gmail.com","avatar":null},"body":"On Tue, Sep 16, 2008 at 15:50, Björn Steinbrink wrote:\n> Maybe instead of providing \"pre-shallowed\" repos to clone from, that\n> should have been an RFE to support shallow clones starting at a given\n> commit in the DAG? I have no idea whether that's feasible though.\n\nWould it be like grafts, only the graft is set up by the fetcher,\ninstead of the host? (E.g., the graft is created on clone, instead of\n-before- the clone, by means of the --depth parameter?) Which means\nthe mentioned security risk is not there (what with the fetcher\nsetting it up himself).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"90872","messageId":"20080916231232.GK28210@dpotapov.dyndns.org","threadId":"15544","inReplyTo":"bd6139dc0809161502t121c5aa5la53784bb8ff273a2@mail.gmail.com","subject":"Re: Grafts workflow for a \"shallow\" repository...?","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-09-16T23:12:32Z","receivedAt":"2008-09-16T23:12:32Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, Sep 17, 2008 at 12:02:25AM +0200, Sverre Rabbelier wrote:\n> On Tue, Sep 16, 2008 at 15:50, Björn Steinbrink wrote:\n> > Maybe instead of providing \"pre-shallowed\" repos to clone from, that\n> > should have been an RFE to support shallow clones starting at a given\n> > commit in the DAG? I have no idea whether that's feasible though.\n> \n> Would it be like grafts, only the graft is set up by the fetcher,\n> instead of the host?\n\nNo, the idea is to have it in the $GIT_DIR/shallow format, which differs\nfrom grafts in that it cannot add new parents. So, it can only truncate\nhistory, while grafts adding new parents can modify history in different\nways.\n\n> (E.g., the graft is created on clone, instead of\n> -before- the clone, by means of the --depth parameter?) Which means\n> the mentioned security risk is not there (what with the fetcher\n> setting it up himself).\n\nThe important difference between grafts and shallow is that the latter\ndoes not allow you to add new parents. So, you cannot modify history but\nonly truncate existing one. So I am not sure about what security risk\nyou are talking about. Obviously, that git blame will attribute all\nchanges that were done before to the earliest downloaded commit. Other\nthan that, I don't think that you can fake anything here. All downloaded\nhistory will be correct and that can be verified based on SHA-1.\n\nAlso, I believe that the default mode should be to ignore this shallow\nfile on server unless the user specified the corresponding option to\nmake a shallow copy.\n\nBTW, the current implementation of shallow copy has some limitations.\nSee Documentation/technical/shallow.txt for details.\n\nDmitry\n"}]}