threads / discuss / 15544

Grafts workflow for a "shallow" repository...?

Subject: Grafts workflow for a "shallow" repository...?

## tl;dr

9 messages between Sep 16, 2008 and Sep 16, 2008.

replies: 8people: 8as markdown or json

Martin Langhoff· Sep 16, 2008, 05:09 UTC · lore

Here is my attempt at a "let's publish a shallow repository for branch of moodle". Let me show you what I did...

 # before we start
 git --version
 git version 1.5.5.1
 cat /etc/redhat-release
 Fedora release 9 (Sulphur)
 # clone the 'full' repo ~200MB fully packed
 git clone git://git.catalyst.net.nz/moodle-r2.git
 # 1.7 was a significant release, anything earlier than that
 # is just not interesting -- even for pickaxe/annotate purposes
 # so add a graft point right at the branching point.
 cd moodle-r2
 echo `git merge-base origin/cvshead origin/MOODLE_17_STABLE` >>
.git/info/grafts
 # push the branches I care about to a newly created repo on a different server
 # this is approx 50MB... (as opposed to 200MB)
 git push git+ssh://dev.laptop.org/home/martin/public_git/moodle.git \
    origin/MOODLE_19_STABLE:refs/heads/MOODLE_19_STABLE
 git push git+ssh://dev.laptop.org/home/martin/public_git/moodle.git \
    origin/cvshead:refs/heads/cvshead
 # fixed up the server-side repo to have HEAD pointing to a new branch...
 # clone to see it works...
 git clone  git://dev.laptop.org/git/users/martin/moodle.git
 # and now gitk bails out on me. Not entirely unexpected...
 gitk --all

Is this kind of workflow (or a variation of it) supported? For this to work, we should communicate the grafts in some push operations and read them in clone ops - and perhaps in fetch too.

cheers,
martin
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff
Shawn O. Pearce· Sep 16, 2008, 05:24 UTC · re: Martin Langhoff · lore

Re: Grafts workflow for a "shallow" repository...?

Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Here is my attempt at a "let's publish a shallow repository for branch
> of moodle". Let me show you what I did...
...
>  # 1.7 was a significant release, anything earlier than that
>  # is just not interesting -- even for pickaxe/annotate purposes
>  # so add a graft point right at the branching point.
...
> Is this kind of workflow (or a variation of it) supported? For this to
> work, we should communicate the grafts in some push operations and
> read them in clone ops - and perhaps in fetch too.

Currently the grafts file isn't transferred over any transport protocol as it is considered to be local only to the repository.

For one thing, grafts are a security risk. Any user can graft anything in at any position and log/blame operations will honor the graft, rather than what is stored in the signed commit chain. Its a low risk, but it allows a peer to lie to you and give you something other than what you asked for.

Pushing (or fetching) a graft just seems ugly. You don't want to fetch a graft if you have no grafts because you have the full history. Nor do you want to fetch a graft that might possibly overwrite/replace a graft you already have. You might not want to push all of your available grafts to a remote. Etc.

I think that in this case the best thing to do is give users a shell script that does roughly:

	git init
	echo $BASE >.git/info/grafts
	git remote add -f origin $url
	git checkout -b master origin/master

Sign the script, and let users verify it before executing. You may also want a script to drag in the history behind by removing the graft and fetching $BASE^, but that is hard because your repository already "has" that.

-- 
Shawn.
Junio C Hamano· Sep 16, 2008, 06:25 UTC · re: Shawn O. Pearce · lore

Re: Grafts workflow for a "shallow" repository...?

"Shawn O. Pearce" <spearce@spearce.org> writes:
Show 24 quoted lines
> Martin Langhoff <martin.langhoff@gmail.com> wrote:
>> Here is my attempt at a "let's publish a shallow repository for branch
>> of moodle". Let me show you what I did...
> ...
>>  # 1.7 was a significant release, anything earlier than that
>>  # is just not interesting -- even for pickaxe/annotate purposes
>>  # so add a graft point right at the branching point.
> ...
>> Is this kind of workflow (or a variation of it) supported? For this to
>> work, we should communicate the grafts in some push operations and
>> read them in clone ops - and perhaps in fetch too.
> ...
> I think that in this case the best thing to do is give users
> a shell script that does roughly:
>
> 	git init
> 	echo $BASE >.git/info/grafts
> 	git remote add -f origin $url
> 	git checkout -b master origin/master
>
> Sign the script, and let users verify it before executing.  You may
> also want a script to drag in the history behind by removing the
> graft and fetching $BASE^, but that is hard because your repository
> already "has" that.
Why not just filter-branch _once at the origin_ and publish the result?
Björn Steinbrink· Sep 16, 2008, 08:09 UTC · re: Junio C Hamano · lore

Re: Grafts workflow for a "shallow" repository...?

On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:
Show 28 quoted lines
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> 
> > Martin Langhoff <martin.langhoff@gmail.com> wrote:
> >> Here is my attempt at a "let's publish a shallow repository for branch
> >> of moodle". Let me show you what I did...
> > ...
> >>  # 1.7 was a significant release, anything earlier than that
> >>  # is just not interesting -- even for pickaxe/annotate purposes
> >>  # so add a graft point right at the branching point.
> > ...
> >> Is this kind of workflow (or a variation of it) supported? For this to
> >> work, we should communicate the grafts in some push operations and
> >> read them in clone ops - and perhaps in fetch too.
> > ...
> > I think that in this case the best thing to do is give users
> > a shell script that does roughly:
> >
> > 	git init
> > 	echo $BASE >.git/info/grafts
> > 	git remote add -f origin $url
> > 	git checkout -b master origin/master
> >
> > Sign the script, and let users verify it before executing.  You may
> > also want a script to drag in the history behind by removing the
> > graft and fetching $BASE^, but that is hard because your repository
> > already "has" that.
> 
> Why not just filter-branch _once at the origin_ and publish the result?

I think the idea was to have a shallow clone starting at a certain point, as opposed to the --depth option, where you cannot specify a starting point, but only the depth of the clone.

Björn
Michael J Gruber· Sep 16, 2008, 13:27 UTC · re: Björn Steinbrink · lore

Re: Grafts workflow for a "shallow" repository...?

Björn Steinbrink venit, vidit, dixit 16.09.2008 10:09:
Show 32 quoted lines
> On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:
>> "Shawn O. Pearce" <spearce@spearce.org> writes:
>>
>>> Martin Langhoff <martin.langhoff@gmail.com> wrote:
>>>> Here is my attempt at a "let's publish a shallow repository for branch
>>>> of moodle". Let me show you what I did...
>>> ...
>>>>  # 1.7 was a significant release, anything earlier than that
>>>>  # is just not interesting -- even for pickaxe/annotate purposes
>>>>  # so add a graft point right at the branching point.
>>> ...
>>>> Is this kind of workflow (or a variation of it) supported? For this to
>>>> work, we should communicate the grafts in some push operations and
>>>> read them in clone ops - and perhaps in fetch too.
>>> ...
>>> I think that in this case the best thing to do is give users
>>> a shell script that does roughly:
>>>
>>> 	git init
>>> 	echo $BASE >.git/info/grafts
>>> 	git remote add -f origin $url
>>> 	git checkout -b master origin/master
>>>
>>> Sign the script, and let users verify it before executing.  You may
>>> also want a script to drag in the history behind by removing the
>>> graft and fetching $BASE^, but that is hard because your repository
>>> already "has" that.
>> Why not just filter-branch _once at the origin_ and publish the result?
> 
> I think the idea was to have a shallow clone starting at a certain
> point, as opposed to the --depth option, where you cannot specify a
> starting point, but only the depth of the clone.
That's what Junio suggests:

chop the history by introducing an appropriate graft make it "permanent" by filter-branching (and remove from info/grafts).

Now you have a disconnect dag. clone/push/whatever works on the "components" given by connected branches.

Note that in this approach all history after the "chopping point" will be rewritten, i.e. get new sha1's.

Michael
Björn Steinbrink· Sep 16, 2008, 13:50 UTC · re: Michael J Gruber · lore

Re: Grafts workflow for a "shallow" repository...?

On 2008.09.16 15:27:43 +0200, Michael J Gruber wrote:
Show 44 quoted lines
> Björn Steinbrink venit, vidit, dixit 16.09.2008 10:09:
> > On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:
> >> "Shawn O. Pearce" <spearce@spearce.org> writes:
> >>
> >>> Martin Langhoff <martin.langhoff@gmail.com> wrote:
> >>>> Here is my attempt at a "let's publish a shallow repository for branch
> >>>> of moodle". Let me show you what I did...
> >>> ...
> >>>>  # 1.7 was a significant release, anything earlier than that
> >>>>  # is just not interesting -- even for pickaxe/annotate purposes
> >>>>  # so add a graft point right at the branching point.
> >>> ...
> >>>> Is this kind of workflow (or a variation of it) supported? For this to
> >>>> work, we should communicate the grafts in some push operations and
> >>>> read them in clone ops - and perhaps in fetch too.
> >>> ...
> >>> I think that in this case the best thing to do is give users
> >>> a shell script that does roughly:
> >>>
> >>> 	git init
> >>> 	echo $BASE >.git/info/grafts
> >>> 	git remote add -f origin $url
> >>> 	git checkout -b master origin/master
> >>>
> >>> Sign the script, and let users verify it before executing.  You may
> >>> also want a script to drag in the history behind by removing the
> >>> graft and fetching $BASE^, but that is hard because your repository
> >>> already "has" that.
> >> Why not just filter-branch _once at the origin_ and publish the result?
> > 
> > I think the idea was to have a shallow clone starting at a certain
> > point, as opposed to the --depth option, where you cannot specify a
> > starting point, but only the depth of the clone.
> 
> That's what Junio suggests:
> 
> chop the history by introducing an appropriate graft
> make it "permanent" by filter-branching (and remove from info/grafts).
> 
> Now you have a disconnect dag. clone/push/whatever works on the
> "components" given by connected branches.
> 
> Note that in this approach all history after the "chopping point" will
> be rewritten, i.e. get new sha1's.

But shallow clones using --depth don't get alterted hashed. They have "graft-like" entries in .git/shallow (or whatever it is called) instead.

Assuming that using a shallow clone is worth the hassle (ie. you have to fetch significantly less data than with a full clone), I can imagine some use-cases for that, for example bisecting a bug of which you know that it was introduced after the last release. Just shallow clone the repo from that version's tag onwards and bisect locally, while still being able to use the original hash to report the broken version. Or genrally anything for which you need a recent part of the history and which requires you to talk about certain commits. The hashes are useful for that, and filter-branch'ing breaks that.

Maybe instead of providing "pre-shallowed" repos to clone from, that should have been an RFE to support shallow clones starting at a given commit in the DAG? I have no idea whether that's feasible though.

Björn
Sverre Rabbelier· Sep 16, 2008, 22:02 UTC · re: Björn Steinbrink · lore

Re: Grafts workflow for a "shallow" repository...?

On Tue, Sep 16, 2008 at 15:50, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> Maybe instead of providing "pre-shallowed" repos to clone from, that
> should have been an RFE to support shallow clones starting at a given
> commit in the DAG? I have no idea whether that's feasible though.

Would it be like grafts, only the graft is set up by the fetcher, instead of the host? (E.g., the graft is created on clone, instead of -before- the clone, by means of the --depth parameter?) Which means the mentioned security risk is not there (what with the fetcher setting it up himself).

-- 
Cheers,

Sverre Rabbelier
Dmitry Potapov· Sep 16, 2008, 23:12 UTC · re: Sverre Rabbelier · lore

Re: Grafts workflow for a "shallow" repository...?

On Wed, Sep 17, 2008 at 12:02:25AM +0200, Sverre Rabbelier wrote:
Show 7 quoted lines
> On Tue, Sep 16, 2008 at 15:50, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> > Maybe instead of providing "pre-shallowed" repos to clone from, that
> > should have been an RFE to support shallow clones starting at a given
> > commit in the DAG? I have no idea whether that's feasible though.
> 
> Would it be like grafts, only the graft is set up by the fetcher,
> instead of the host?

No, the idea is to have it in the $GIT_DIR/shallow format, which differs from grafts in that it cannot add new parents. So, it can only truncate history, while grafts adding new parents can modify history in different ways.

> (E.g., the graft is created on clone, instead of
> -before- the clone, by means of the --depth parameter?) Which means
> the mentioned security risk is not there (what with the fetcher
> setting it up himself).

The important difference between grafts and shallow is that the latter does not allow you to add new parents. So, you cannot modify history but only truncate existing one. So I am not sure about what security risk you are talking about. Obviously, that git blame will attribute all changes that were done before to the earliest downloaded commit. Other than that, I don't think that you can fake anything here. All downloaded history will be correct and that can be verified based on SHA-1.

Also, I believe that the default mode should be to ignore this shallow file on server unless the user specified the corresponding option to make a shallow copy.

BTW, the current implementation of shallow copy has some limitations. See Documentation/technical/shallow.txt for details.

Dmitry
Sergio· Sep 16, 2008, 19:12 UTC · re: Shawn O. Pearce · lore

Re: Grafts workflow for a

Shawn O. Pearce <spearce <at> spearce.org> writes:
Show 22 quoted lines
> 
> Martin Langhoff <martin.langhoff <at> gmail.com> wrote:
> > Here is my attempt at a "let's publish a shallow repository for branch
> > of moodle". Let me show you what I did...
> ...
> >  # 1.7 was a significant release, anything earlier than that
> >  # is just not interesting -- even for pickaxe/annotate purposes
> >  # so add a graft point right at the branching point.
> ...
> > Is this kind of workflow (or a variation of it) supported? For this to
> > work, we should communicate the grafts in some push operations and
> > read them in clone ops - and perhaps in fetch too.
> 
> Currently the grafts file isn't transferred over any transport
> protocol as it is considered to be local only to the repository.
> 
> For one thing, grafts are a security risk.  Any user can graft
> anything in at any position and log/blame operations will honor
> the graft, rather than what is stored in the signed commit chain.
> Its a low risk, but it allows a peer to lie to you and give you
> something other than what you asked for.
>

Would it make sense to differentiate two types of grafts, just like we differentiate two kind of tags?

Then there could be "annotated grafts objects" that could be optionally signed and that would get transferred on clones, fetches, pushes, etc.

Sergio

← back to recent threads