threads / discuss / 19459

partial checkouts

Subject: partial checkouts

## tl;dr

9 messages between May 23, 2009 and May 25, 2009.

replies: 8people: 6as markdown or json

Chani· May 23, 2009, 21:00 UTC · lore

so I'm trying to convert KDE's translation helper scripts to support git repositories for kde modules. the translators themselves are staying in svn for now, but we want to start migrating kde modules over to git.

I've hit a bit of a problem, though. inside the translators' folder, there's a folder called documentation/ - and inside that is a bunch of svn externals pointing to the doc/ folder of every kde module. (this is, then, all of kde's english documentation.) translators occasionally need to run a script that reads from the documentation/ folder.

now, what happens when one of those modules switches to git? first, the svn external goes away. that's ok, I can work around that by having the update_xml script (the one translators need to run) fetch the doc folder itself, right? but where does it fetch it from? doc/ is now a subdirectory in a git repository, and git doesn't support checking out only one folder, not even if all I want to do is read the files in it.

checking out the entire git repo isn't really feasible, because once KDE finishes switching to git that'll mean checking out all of KDE. I don't think that turning every doc folder into a submodule is feasible, either - they sound kinda awkward to work with, and I can imagine lots of people getting confused and messing them up...

right now all I've thought of is one ugly hack: have a server that checks out all the kde git repos, pulls daily, copies all the doc/ folders into a documentation folder, and offers that folder up on the interwebs so that update_xml can rsync from it or download a tgz of it or something. there appear to be lots of images in the documentation, so it's not a small download - 200mb and growing. it still hasn't finished downloading all the externals...

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo that only tracks the doc/ folder for a module - but I've no idea whether it'd have to be recreated from scratch every time someone changes something in the real repo's doc/

can anyone think of a less ugly solution? what are the chances of git supporting this kind of partial checkout someday?

-- 
This message brought to you by eevil bananas and the number 3.
www.chani3.com
Johan Herland· May 23, 2009, 23:34 UTC · re: Chani · lore

Re: partial checkouts

On Saturday 23 May 2009, Chani wrote:
Show 9 quoted lines
> [...]
>
> right now all I've thought of is one ugly hack: have a server that checks
> out all the kde git repos, pulls daily, copies all the doc/ folders into
> a documentation folder, and offers that folder up on the interwebs so
> that update_xml can rsync from it or download a tgz of it or something.
> there appear to be lots of images in the documentation, so it's not a
> small download - 200mb and growing. it still hasn't finished downloading
> all the externals...

Do you need the doc/ folders from _all_ kde git repos, or just from those repos that you have currently checked out? In the latter case, you could solve this by adding symlinks to all the doc/ folders inside the documentation/ folder, and then make sure the software that traverse the documentation/ folder recognize and skips symlinks. Of course, this won't work if the translations project need _all_ doc/ folders accessible, but not all the kde git repos.

Show 8 quoted lines
> I'm kinda wondering if there'd be a way to use git-filter-branch to make
> a repo that only tracks the doc/ folder for a module - but I've no idea
> whether it'd have to be recreated from scratch every time someone changes
> something in the real repo's doc/
>
> can anyone think of a less ugly solution?
> what are the chances of git supporting this kind of partial checkout
> someday?

Check out git-subtree. It can split out a subdirectory into its own repo, and re-integrate it back into the "parent" repo at a later date. git-subtree has been posted as a patch to this list a couple of times without much response, but it looks like an interesting alternative to submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30

If a lot of people find git-subtree useful, who knows, it might be included in a future git version.

Have fun! :)
...Johan
-- 
Johan Herland, <johan@herland.net>
www.herland.net
Chani· May 24, 2009, 00:26 UTC · re: Johan Herland · lore

Re: partial checkouts

On May 23, 2009 16:34:53 Johan Herland wrote:
Show 18 quoted lines
> On Saturday 23 May 2009, Chani wrote:
> > [...]
> >
> > right now all I've thought of is one ugly hack: have a server that checks
> > out all the kde git repos, pulls daily, copies all the doc/ folders into
> > a documentation folder, and offers that folder up on the interwebs so
> > that update_xml can rsync from it or download a tgz of it or something.
> > there appear to be lots of images in the documentation, so it's not a
> > small download - 200mb and growing. it still hasn't finished downloading
> > all the externals...
>
> Do you need the doc/ folders from _all_ kde git repos, or just from those
> repos that you have currently checked out? In the latter case, you could
> solve this by adding symlinks to all the doc/ folders inside the
> documentation/ folder, and then make sure the software that traverse the
> documentation/ folder recognize and skips symlinks. Of course, this won't
> work if the translations project need _all_ doc/ folders accessible, but
> not all the kde git repos.

nope, the translators may not have checked out *any* of them but the script they want to run needs *all* the docs. :( however, I've been told they also don't want to have to change their workflow in any way at all no matter how small, so we may be stuck in svn-land anyways, because you can't make an svn external out of something that's not in svn, and having anything other than svn externals would change their workflow :(

Show 18 quoted lines
>
> > I'm kinda wondering if there'd be a way to use git-filter-branch to make
> > a repo that only tracks the doc/ folder for a module - but I've no idea
> > whether it'd have to be recreated from scratch every time someone changes
> > something in the real repo's doc/
> >
> > can anyone think of a less ugly solution?
> > what are the chances of git supporting this kind of partial checkout
> > someday?
>
> Check out git-subtree. It can split out a subdirectory into its own repo,
> and re-integrate it back into the "parent" repo at a later date.
> git-subtree has been posted as a patch to this list a couple of times
> without much response, but it looks like an interesting alternative to
> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
>
> If a lot of people find git-subtree useful, who knows, it might be included
> in a future git version.

looks interesting. might have been a solution until I heard about this requirement to not change workflow at all. :/

however, my friend told me about a project to make a git-svnserver that serves git repos as svn repos, and *that* would allow the translators to stay where they are without holding everyone else there too. know anything about that?

mm, google turns up an email from someone claiming they have a partial implementation in python...

-- 
This message brought to you by eevil bananas and the number 3.
www.chani3.com
Avery Pennarun· May 24, 2009, 19:45 UTC · re: Chani · lore

Re: partial checkouts

On Sat, May 23, 2009 at 8:26 PM, Chani <chanika@gmail.com> wrote:
Show 19 quoted lines
> On May 23, 2009 16:34:53 Johan Herland wrote:
>> On Saturday 23 May 2009, Chani wrote:
>> > I'm kinda wondering if there'd be a way to use git-filter-branch to make
>> > a repo that only tracks the doc/ folder for a module - but I've no idea
>> > whether it'd have to be recreated from scratch every time someone changes
>> > something in the real repo's doc/
>> >
>> > can anyone think of a less ugly solution?
>> > what are the chances of git supporting this kind of partial checkout
>> > someday?
>>
>> Check out git-subtree. It can split out a subdirectory into its own repo,
>> and re-integrate it back into the "parent" repo at a later date.
>> git-subtree has been posted as a patch to this list a couple of times
>> without much response, but it looks like an interesting alternative to
>> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
>
> looks interesting. might have been a solution until I heard about this
> requirement to not change workflow at all. :/

Here's something that might work. Write a script that runs on a server somewhere. It checks out *all* of the git-managed KDE components, including documentation (it's okay to do this from *one* place, right?). Then it loops through all the documentation subdirectories and splits them out into individual projects/branches with 'git subtree split'.

You could then make a *new* project, kde-docs for example, that pulls those new doc projects back together into a single project, using 'git subtree merge'.

People could then checkout the single kde-docs project and get *all* the docs, then make changes and push them back upstream to you. As far as the average person is concerned, this is probably even easier than svn:externals - they don't have to remember to commit in each subdirectory separately, as git-subtree rips it all apart later. Also note that *end users* never have to operate the git subtree command; all that happens behind the scenes on your server.

Later, you use 'git subtree split' to rip it apart back into the individual projects, and 'git subtree merge' those back into their original git repositories.

Since git-subtree is my own little toy and it's designed for exactly this kind of workflow, I'd be happy to help you debug a script for this if you like. Probably I'll learn something.

Disclaimer: git-subtree hasn't been used by too many people yet and we
may end up finding some bugs.
Have fun,
Avery
Chani· May 25, 2009, 03:07 UTC · re: Avery Pennarun · lore

Re: partial checkouts

Show 17 quoted lines
> >>
> >> Check out git-subtree. It can split out a subdirectory into its own
> >> repo, and re-integrate it back into the "parent" repo at a later date.
> >> git-subtree has been posted as a patch to this list a couple of times
> >> without much response, but it looks like an interesting alternative to
> >> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
> >
> > looks interesting. might have been a solution until I heard about this
> > requirement to not change workflow at all. :/
>
> Here's something that might work.  Write a script that runs on a
> server somewhere.  It checks out *all* of the git-managed KDE
> components, including documentation (it's okay to do this from *one*
> place, right?).  Then it loops through all the documentation
> subdirectories and splits them out into individual projects/branches
> with 'git subtree split'.
>

I like this idea. I like the idea of git subtree in general. :) however, we've decided to go the "copy all documentation into svn" route for now (so that the translators can keep using only svn). I'll be back when we run into the next svn-external issue (there's sure to be more). :)

-- 
This message brought to you by eevil bananas and the number 3.
www.chani3.com
Aidan Van Dyk· May 25, 2009, 14:51 UTC · re: Avery Pennarun · lore

Re: partial checkouts

If all you're looking for is a "single set of read-only docs" conglomeration of
all the KDE submodules projects documentation so you can translate them, what
about using "archive"?  Since archive takes a "tree-ish", and work on remote
upload-pack servers (if it's enabled), you can get an un-versioned "remote
partial checkout" with something like do things like:
	git archive --format=tar --prefix=git/Documentation/ --remote=git://repo.or.cz/git.git HEAD:Documentation | tar tf - | head
	git archive --format=tar --prefix=/opt/git/share/man/ --remote=git://repo.or.cz/git.git man | tar tf -
So, this could be simply scripted to give you a repo with *all* modules docs/ inside a single repo...
	for m in $MODULES
	do
		git archive --format=tar --prefix="$m/$DOCDIR/" --remote=$REMOTE_BASE/$m $BRANCH:$DOCDIR | tar xf -
		git add $m/$DOCDIR
		git commit $m/$DOCDIR -m "$m: Automatic import"
	done

Do that in a central location an dlet translators pull that... Everythings "duplicated", but git's object store will help out there... You loose the history of the original docs, but for translations, you you're not translating the history anyways, your'e translation the docs at a point in time...

Or just dump all thd docs in a .gitignore'd subdirctory and have translators run a script to update them.

a.
Show 11 quoted lines
> On Sat, May 23, 2009 at 8:26 PM, Chani <chanika@gmail.com> wrote:
> > On May 23, 2009 16:34:53 Johan Herland wrote:
> >> On Saturday 23 May 2009, Chani wrote:
> >> > I'm kinda wondering if there'd be a way to use git-filter-branch to make
> >> > a repo that only tracks the doc/ folder for a module - but I've no idea
> >> > whether it'd have to be recreated from scratch every time someone changes
> >> > something in the real repo's doc/
> >> >
> >> > can anyone think of a less ugly solution?
> >> > what are the chances of git supporting this kind of partial checkout
> >> > someday?
Show 34 quoted lines
> Here's something that might work.  Write a script that runs on a
> server somewhere.  It checks out all of the git-managed KDE
> components, including documentation (it's okay to do this from one
> place, right?).  Then it loops through all the documentation
> subdirectories and splits them out into individual projects/branches
> with 'git subtree split'.
> 
> You could then make a new project, kde-docs for example, that pulls
> those new doc projects back together into a single project, using 'git
> subtree merge'.
> 
> People could then checkout the single kde-docs project and get all
> the docs, then make changes and push them back upstream to you.  As
> far as the average person is concerned, this is probably even easier
> than svn:externals - they don't have to remember to commit in each
> subdirectory separately, as git-subtree rips it all apart later.  Also
> note that *end users* never have to operate the git subtree command;
> all that happens behind the scenes on your server.
> 
> Later, you use 'git subtree split' to rip it apart back into the
> individual projects, and 'git subtree merge' those back into their
> original git repositories.
> 
> Since git-subtree is my own little toy and it's designed for exactly
> this kind of workflow, I'd be happy to help you debug a script for
> this if you like.  Probably I'll learn something.
> 
> Disclaimer: git-subtree hasn't been used by too many people yet and we
> may end up finding some bugs.
> 
> Have fun,
> 
> Avery
>
Nguyen Thai Ngoc Duy· May 24, 2009, 02:07 UTC · re: Chani · lore

Re: partial checkouts

On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:
Show 5 quoted lines
> checking out the entire git repo isn't really feasible, because once KDE
> finishes switching to git that'll mean checking out all of KDE.
> I don't think that turning every doc folder into a submodule is feasible,
> either - they sound kinda awkward to work with, and I can imagine lots of
> people getting confused and messing them up...

If you don't mind cloning the entire git repo (which may be quite large in KDE case, I guess), then you may want to try "sparse checkout". The patch series is available as a topic branch "tp/sco" in this branch:

http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst

Or I can send the patch series to you. It lets you specify what part of the tree you want to checkout.

-- 
Duy
Thomas Adam· May 24, 2009, 15:57 UTC · re: Nguyen Thai Ngoc Duy · lore

Re: partial checkouts

2009/5/24 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
Show 16 quoted lines
> On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:
>> checking out the entire git repo isn't really feasible, because once KDE
>> finishes switching to git that'll mean checking out all of KDE.
>> I don't think that turning every doc folder into a submodule is feasible,
>> either - they sound kinda awkward to work with, and I can imagine lots of
>> people getting confused and messing them up...
>
> If you don't mind cloning the entire git repo (which may be quite
> large in KDE case, I guess), then you may want to try "sparse
> checkout". The patch series is available as a topic branch "tp/sco" in
> this branch:
>
> http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
>
> Or I can send the patch series to you. It lets you specify what part
> of the tree you want to checkout.

I recall following this series when you first announced it. Do you have any plans to try and get it included in mainline? ISTR there was some more work needed to be done on it before then.

-- Thomas Adam
Nguyen Thai Ngoc Duy· May 25, 2009, 01:51 UTC · re: Thomas Adam · lore

Re: partial checkouts

On Mon, May 25, 2009 at 1:57 AM, Thomas Adam <thomas.adam22@gmail.com> wrote:
Show 21 quoted lines
> 2009/5/24 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>> On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:
>>> checking out the entire git repo isn't really feasible, because once KDE
>>> finishes switching to git that'll mean checking out all of KDE.
>>> I don't think that turning every doc folder into a submodule is feasible,
>>> either - they sound kinda awkward to work with, and I can imagine lots of
>>> people getting confused and messing them up...
>>
>> If you don't mind cloning the entire git repo (which may be quite
>> large in KDE case, I guess), then you may want to try "sparse
>> checkout". The patch series is available as a topic branch "tp/sco" in
>> this branch:
>>
>> http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
>>
>> Or I can send the patch series to you. It lets you specify what part
>> of the tree you want to checkout.
>
> I recall following this series when you first announced it.  Do you
> have any plans to try and get it included in mainline?  ISTR there was
> some more work needed to be done on it before then.
It was pulled off because of big, intrusive and not well-designed patches.
http://mid.gmane.org/20081128192033.GF23984@spearce.org

I may submit it again when I get code insertions down to ~500 lines or something like that ;-) Anyway you could always help by giving more use cases to test against the current design.

-- 
Duy

← back to recent threads