threads / discuss / 12622

[QUESTION] Selective fetch possible?

Subject: [QUESTION] Selective fetch possible?

## tl;dr

7 messages between Mar 10, 2008 and Mar 11, 2008.

replies: 6people: 4as markdown or json

Filippo Zangheri· Mar 10, 2008, 22:02 UTC · lore
Hi everybody,
Is it possible to git-fetch only a portion of the tree
of the specified repository, say, fetch only one directory or a
subset of files matching some regular expression? This is currently
- to my knowledge - only possible via wget iff the GIT repository
has gitweb enabled. But that's just a workaround.

Thanks for the great instrument that GIT is. Best regards.

PS: please CC me in replies, because I'm not subscribed to git ML.
-- 
Filippo Zangheri

GPG key ID: 0xE1D879FA
Key fingerprint: 816B CE57 D43C 0A47 EF35 3378 EA5F A72A E1D8 79FA
Key server: pgp.mit.edu

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GE d- s+:- a-- C++ UL+++ P+ L+++ E-- W+ N* o-- K- w--- O-- M--
V- PS++ PE+ Y+ PGP++ t 5-- X++ R* tv b+ DI-- D---- G-- e++ h--
r++ z*
------END GEEK CODE BLOCK------
Shawn O. Pearce· Mar 10, 2008, 22:53 UTC · re: Filippo Zangheri · lore

Re: [QUESTION] Selective fetch possible?

Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
Show 5 quoted lines
> Is it possible to git-fetch only a portion of the tree
> of the specified repository, say, fetch only one directory or a
> subset of files matching some regular expression? This is currently
> - to my knowledge - only possible via wget iff the GIT repository
> has gitweb enabled. But that's just a workaround.
No.

You can use a shallow clone to fetch only X commits back into history on any branch, and you can also manually configure the fetch specification in .git/config to only fetch specific branches, but you must fetch the entire tree to get any of the files in it.

If the repository is available by git:// protocol you may be able
to use git-archive to obtain a tarfile for just the directory you
want (service has to be enabled on the remote side) but that is
just a raw UNIX tar; there is no Git repository and no ability to
commit/fetch/push/diff/apply/log/etc.
 
-- 
Shawn.
Jakub Narebski· Mar 10, 2008, 23:34 UTC · re: Shawn O. Pearce · lore

Re: [QUESTION] Selective fetch possible?

"Shawn O. Pearce" <spearce@spearce.org> writes:
Show 20 quoted lines
> Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
> >
> > Is it possible to git-fetch only a portion of the tree
> > of the specified repository, say, fetch only one directory or a
> > subset of files matching some regular expression? This is currently
> > - to my knowledge - only possible via wget iff the GIT repository
> > has gitweb enabled. But that's just a workaround.
> 
> No.
> 
> You can use a shallow clone to fetch only X commits back into
> history on any branch, and you can also manually configure the
> fetch specification in .git/config to only fetch specific branches,
> but you must fetch the entire tree to get any of the files in it.
> 
> If the repository is available by git:// protocol you may be able
> to use git-archive to obtain a tarfile for just the directory you
> want (service has to be enabled on the remote side) but that is
> just a raw UNIX tar; there is no Git repository and no ability to
> commit/fetch/push/diff/apply/log/etc.
Note that what you wanted is, I guess, something called partial
checkout or subtree checkout.  This feature appears now and then in
feature requests; lately Nguyen Thai Ngoc Duy (pclouds) offered to do
this in "on subtree checkout" thread:
  http://thread.gmane.org/gmane.comp.version-control.git/74915

The problem is twofold, as far as I understand it. First, what to do if there is merge conflicts outside checked out (selected) directory? Second, how to make repository contain only relevant objects: git in many places assumes full connectivity, and that if it has an object it hass all objects depending on it.

-- 
Jakub Narebski
Poland
ShadeHawk on #git
Rogan Dawes· Mar 11, 2008, 07:26 UTC · re: Jakub Narebski · lore

Re: [QUESTION] Selective fetch possible?

Jakub Narebski wrote:
Show 26 quoted lines
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> 
>> Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
>>> Is it possible to git-fetch only a portion of the tree
>>> of the specified repository, say, fetch only one directory or a
>>> subset of files matching some regular expression? This is currently
>>> - to my knowledge - only possible via wget iff the GIT repository
>>> has gitweb enabled. But that's just a workaround.
>> No.
>>
>> You can use a shallow clone to fetch only X commits back into
>> history on any branch, and you can also manually configure the
>> fetch specification in .git/config to only fetch specific branches,
>> but you must fetch the entire tree to get any of the files in it.
>>
>> If the repository is available by git:// protocol you may be able
>> to use git-archive to obtain a tarfile for just the directory you
>> want (service has to be enabled on the remote side) but that is
>> just a raw UNIX tar; there is no Git repository and no ability to
>> commit/fetch/push/diff/apply/log/etc.
> 
> Note that what you wanted is, I guess, something called partial
> checkout or subtree checkout.  This feature appears now and then in
> feature requests; lately Nguyen Thai Ngoc Duy (pclouds) offered to do
> this in "on subtree checkout" thread:
>   http://thread.gmane.org/gmane.comp.version-control.git/74915

I still believe that it could be done fairly easily by simply recording the SHA1's of the files and directories that are *not* checked out somewhere in the .git directory, and just reusing those when checking the working tree. i.e. rather than stat-ing a tree that was never checked out, get the known SHA1 for that tree from where it was recorded on checkout.

> The problem is twofold, as far as I understand it.  First, what to do
> if there is merge conflicts outside checked out (selected) directory?

This is something that has been repeated many times, and I fail to see how it can be an issue. How can there be a conflict in a directory that is not, and never has been, checked out, and therefore cannot have been modified?

The only possibility that I can see is if the directory has been renamed elsewhere, but in that case, it *is* effectively checked out (just with a different directory name).

> Second, how to make repository contain only relevant objects: git in
> many places assumes full connectivity, and that if it has an object it
> hass all objects depending on it.
> 
Yes, this is the big problem as I see it.
Rogan
Shawn O. Pearce· Mar 11, 2008, 07:50 UTC · re: Rogan Dawes · lore

Re: [QUESTION] Selective fetch possible?

Rogan Dawes <lists@dawes.za.net> wrote:
Show 15 quoted lines
> Jakub Narebski wrote:
> >"Shawn O. Pearce" <spearce@spearce.org> writes:
> >
> >>Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
> >>>Is it possible to git-fetch only a portion of the tree
> >>>of the specified repository, say, fetch only one directory or a
> >>>subset of files matching some regular expression?
> 
> >The problem is twofold, as far as I understand it.  First, what to do
> >if there is merge conflicts outside checked out (selected) directory?
> 
> This is something that has been repeated many times, and I fail to see 
> how it can be an issue. How can there be a conflict in a directory that 
> is not, and never has been, checked out, and therefore cannot have been 
> modified?
Given two branches:
	code
	docs

and the code people checkout the "src/" subdirectory and the docs people checkout the "Documentation/" subdirectory, and they *only* every work in that subdirectory, things are fine.

Until one day some developer also checks out "Documentation/" and fixes something in the documentation as part of the same commit that makes a code change. The push this to the code branch.

Someday in the future a documentation writer merges the code branch over to the docs branch, "just keeping it current".

Now there arises a possiblity of a merge conflict in a part of the tree that you do not have checked out.

If you want to say "don't ever modify stuff outside of your branch's purpose" then why aren't you just using submodules (one for docs and one for code) and using a supermodule to tie everything together into a "release package"?

Show 6 quoted lines
> >Second, how to make repository contain only relevant objects: git in
> >many places assumes full connectivity, and that if it has an object it
> >hass all objects depending on it.
> >
> 
> Yes, this is the big problem as I see it.

This is easy enough that if the above problem could be resolved sufficiently to the git gurus' satisfaction you would be able to get some advice on how to solve it. Its not difficult, just damn annoying. We already do it (to some extent) with grafts and shallow clones.

-- 
Shawn.
Rogan Dawes· Mar 11, 2008, 09:07 UTC · re: Shawn O. Pearce · lore

Re: [QUESTION] Selective fetch possible?

Shawn O. Pearce wrote:
Show 39 quoted lines
> Rogan Dawes <lists@dawes.za.net> wrote:
>> Jakub Narebski wrote:
>>> "Shawn O. Pearce" <spearce@spearce.org> writes:
>>>
>>>> Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
>>>>> Is it possible to git-fetch only a portion of the tree
>>>>> of the specified repository, say, fetch only one directory or a
>>>>> subset of files matching some regular expression?
>>> The problem is twofold, as far as I understand it.  First, what to do
>>> if there is merge conflicts outside checked out (selected) directory?
>> This is something that has been repeated many times, and I fail to see 
>> how it can be an issue. How can there be a conflict in a directory that 
>> is not, and never has been, checked out, and therefore cannot have been 
>> modified?
> 
> Given two branches:
> 
> 	code
> 	docs
> 
> and the code people checkout the "src/" subdirectory and the docs
> people checkout the "Documentation/" subdirectory, and they *only*
> every work in that subdirectory, things are fine.
> 
> Until one day some developer also checks out "Documentation/" and
> fixes something in the documentation as part of the same commit
> that makes a code change.  The push this to the code branch.
> 
> Someday in the future a documentation writer merges the code branch
> over to the docs branch, "just keeping it current".
> 
> Now there arises a possiblity of a merge conflict in a part of the
> tree that you do not have checked out.
> 
> 
> If you want to say "don't ever modify stuff outside of your branch's
> purpose" then why aren't you just using submodules (one for docs and
> one for code) and using a supermodule to tie everything together into
> a "release package"?
Ok, fair enough. Thanks for the example.

I think that one should not *expect* to be able to complete merges with only a partial checkout, though. It *may* work in cases where there are no conflicts, but I think it would be a perfectly valid error path to fail if there is a conflicting merge in a part of the tree that has not been checked out.

So, for a user working on partial trees, they would be able to modify their partial tree, and check in their changes, but merges would have to be done by someone with a complete checkout. For the given examples where partial trees make sense (documentation workers), this seems like a reasonable compromise.

Show 11 quoted lines
>>> Second, how to make repository contain only relevant objects: git in
>>> many places assumes full connectivity, and that if it has an object it
>>> hass all objects depending on it.
>>>
>> Yes, this is the big problem as I see it.
> 
> This is easy enough that if the above problem could be resolved
> sufficiently to the git gurus' satisfaction you would be able
> to get some advice on how to solve it.  Its not difficult, just
> damn annoying.  We already do it (to some extent) with grafts and
> shallow clones.
How's my suggestion above?
Rogan
Filippo Zangheri· Mar 11, 2008, 12:29 UTC · re: Rogan Dawes · lore

Re: [QUESTION] Selective fetch possible?

Rogan Dawes ha scritto: (...)

Show 11 quoted lines
> I think that one should not *expect* to be able to complete merges with 
> only a partial checkout, though. It *may* work in cases where there are 
> no conflicts, but I think it would be a perfectly valid error path to 
> fail if there is a conflicting merge in a part of the tree that has not 
> been checked out.
> 
> So, for a user working on partial trees, they would be able to modify 
> their partial tree, and check in their changes, but merges would have to 
> be done by someone with a complete checkout. For the given examples 
> where partial trees make sense (documentation workers), this seems like 
> a reasonable compromise.

I think this is what every reasonable developer should have in mind when he's working on just a project subset :). And I also think, this is not a valid reason for forbidding/not implementing such a partial (or subtree) checkout.

-- 
Filippo Zangheri

GPG key ID: 0xE1D879FA
Key fingerprint: 816B CE57 D43C 0A47 EF35 3378 EA5F A72A E1D8 79FA
Key server: pgp.mit.edu

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GE d- s+:- a-- C++ UL+++ P+ L+++ E-- W+ N* o-- K- w--- O-- M--
V- PS++ PE+ Y+ PGP++ t 5-- X++ R* tv b+ DI-- D---- G-- e++ h--
r++ z*
------END GEEK CODE BLOCK------

← back to recent threads