threads / discuss / 18748

Submodules can't work recursively because Git implements policy?

Subject: Submodules can't work recursively because Git implements policy?

## tl;dr

5 messages between Apr 6, 2009 and Apr 6, 2009.

replies: 4people: 3as markdown or json

Klas Lindberg· Apr 6, 2009, 13:42 UTC · lore

On Mon, Apr 6, 2009 at 3:16 PM, Finn Arne Gangstad <finnag@pvv.org> wrote:

Show 7 quoted lines

> git submodule update just does "git fetch" and hopes that the required
> commit appears. In practice this means that you (may) need to invent a
> tag or a branch for all the submodules, otherwise they are not
> fetchable.
>
> This bit us pretty hard when we tried to use submodules earlier, so we
> gave up. Maybe some day...

It "hopes" to find them? This is actually my other reason for bringing the whole SHA key fetching thing up. From what I can see, it is not possible to implement submodules sensibly without support for fetching SHA keys. I.e. I want fetch, checkout and every other command to recurse as needed in the presence of submodules. This limitation forces me to implement a whole CM tool where none should be necessary.

It appears to me that the security concern (being able to hide commits by making them unreachable from a named reference) is actually a policy decision and not a technical one. On what grounds does Git decide for me how to handle security concerns? It just seems more important to be able to have recursive submodule behaviour than to provide band aid for careless users.

Out of curiosity: Is it really possible to change the value of an already pushed tag? Can you only do the hiding trick with branches?

BR / Klas

Finn Arne Gangstad· Apr 6, 2009, 13:56 UTC · re: Klas Lindberg · lore

Re: Submodules can't work recursively because Git implements policy?

On Mon, Apr 06, 2009 at 03:42:31PM +0200, Klas Lindberg wrote:

Show 11 quoted lines

> On Mon, Apr 6, 2009 at 3:16 PM, Finn Arne Gangstad <finnag@pvv.org> wrote:
> 
> > git submodule update just does "git fetch" and hopes that the required
> > commit appears. In practice this means that you (may) need to invent a
> > tag or a branch for all the submodules, otherwise they are not
> > fetchable.
> >
> > This bit us pretty hard when we tried to use submodules earlier, so we
> > gave up. Maybe some day...
> 
> It "hopes" to find them?

Perhaps "hopes" is a loaded word, "expects" at least. The code just does the equivalent of "git fetch; git checkout <sha-1> || die .. "

Show 6 quoted lines

>  This is actually my other reason for bringing
> the whole SHA key fetching thing up. From what I can see, it is not
> possible to implement submodules sensibly without support for fetching
> SHA keys. I.e. I want fetch, checkout and every other command to
> recurse as needed in the presence of submodules. This limitation
> forces me to implement a whole CM tool where none should be necessary.

Yes, I could not agree more. You may also end up writing some really complicated wrappers around git push to get things going (where do you push, for example). We made some interesting "concept art" around this last year at $dayjob, but decided to drop it.

Show 6 quoted lines

> It appears to me that the security concern (being able to hide commits
> by making them unreachable from a named reference) is actually a
> policy decision and not a technical one. On what grounds does Git
> decide for me how to handle security concerns? It just seems more
> important to be able to have recursive submodule behaviour than to
> provide band aid for careless users.

Maybe the security concerns could be handled by adding some functionality to (quickly) get rid of unwanted commits?

> Out of curiosity: Is it really possible to change the value of an
> already pushed tag? Can you only do the hiding trick with branches?

Yes, but if you modify a tag, you get additional complications. In particular, no one will ever try to refetch the tag, so everyone who has already fetched it will have a permanently broken tag.

- Finn Arne

Klas Lindberg· Apr 6, 2009, 14:47 UTC · re: Finn Arne Gangstad · lore

Re: Submodules can't work recursively because Git implements policy?

On Mon, Apr 6, 2009 at 3:56 PM, Finn Arne Gangstad <finnag@pvv.org> wrote:

> Yes, I could not agree more.  You may also end up writing some really
> complicated wrappers around git push to get things going (where do you
> push, for example). We made some interesting "concept art" around this
> last year at $dayjob, but decided to drop it.

I don't see how pushing could work at all without recursion.

> Maybe the security concerns could be handled by adding some
> functionality to (quickly) get rid of unwanted commits?

Why not simply allow users with write permissions to "pop" revisions from the top of the history DAG in a way that actually really deletes the them? Or at least moves those commits to a separate, locked down DAG that cannot be read by people without write permissions?

But anyway: If I implement support for fetching SHA keys and full recursive behaviour in the presence of submodules; would my patches automatically be rejected because of the rationale for the current behaviour?

/Klas

Shawn O. Pearce· Apr 6, 2009, 14:51 UTC · re: Klas Lindberg · lore

Re: Submodules can't work recursively because Git implements policy?

Klas Lindberg <klas.lindberg@gmail.com> wrote:

Show 8 quoted lines

> On Mon, Apr 6, 2009 at 3:56 PM, Finn Arne Gangstad <finnag@pvv.org> wrote:
> > Maybe the security concerns could be handled by adding some
> > functionality to (quickly) get rid of unwanted commits?
> 
> Why not simply allow users with write permissions to "pop" revisions
> from the top of the history DAG in a way that actually really deletes
> the them? Or at least moves those commits to a separate, locked down
> DAG that cannot be read by people without write permissions?

What, like a secret shadow repository that you move the objects into?

That could be very expensive in terms of disk IO if those objects
are in large packs.  You'd need to break the pack apart into the
"ok" and "sekret" parts.  Ick.

> But anyway: If I implement support for fetching SHA keys and full
> recursive behaviour in the presence of submodules; would my patches
> automatically be rejected because of the rationale for the current
> behaviour?

See my recent email (like ~10-15 minutes ago). It will be rejected due to the issue that unreachable objects are subjected to GC and you'd easily see your repository delete that data on the next "git gc" invocation. Automatic data destruction is not something that users come to git for.

-- 
Shawn.

Klas Lindberg· Apr 6, 2009, 16:29 UTC · re: Shawn O. Pearce · lore

Re: Submodules can't work recursively because Git implements policy?

On Mon, Apr 6, 2009 at 4:51 PM, Shawn O. Pearce <spearce@spearce.org> wrote:

Show 5 quoted lines

> What, like a secret shadow repository that you move the objects into?
>
> That could be very expensive in terms of disk IO if those objects
> are in large packs.  You'd need to break the pack apart into the
> "ok" and "sekret" parts.  Ick.

Well, ok. Just popping then? Or adding the wrong publication to a forbidden fetch list?

Show 10 quoted lines

>> But anyway: If I implement support for fetching SHA keys and full
>> recursive behaviour in the presence of submodules; would my patches
>> automatically be rejected because of the rationale for the current
>> behaviour?
>
> See my recent email (like ~10-15 minutes ago).  It will be rejected
> due to the issue that unreachable objects are subjected to GC and
> you'd easily see your repository delete that data on the next "git
> gc" invocation.  Automatic data destruction is not something that
> users come to git for.

Indeed not. But I don't want to suggest that named references shouldn't be necessary. Just that it should be possible to fetch based on the SHA key if that commit is (still) available on the remote end.

BR / Klas