{"thread":{"id":"19459","subject":"partial checkouts","startedAt":"2009-05-23T21:00:55Z","lastAt":"2009-05-25T14:51:31Z","messageCount":9,"participants":["Chani","Johan Herland","Nguyen Thai Ngoc Duy","Thomas Adam","Avery Pennarun","Aidan Van Dyk"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"114538","messageId":"200905231401.11651.chanika@gmail.com","threadId":"19459","inReplyTo":null,"subject":"partial checkouts","fromName":"Chani","fromEmail":"chanika@gmail.com","sentAt":"2009-05-23T21:00:55Z","receivedAt":"2009-05-23T21:00:55Z","isPatch":false,"sender":{"key":"chanika@gmail.com","avatar":"https://gravatar.com/avatar/118096b4e8fe9991f3026ffc7aa39a4a925d265856f037a5012098f9e5d1a993?d=mp&s=160"},"body":"so I'm trying to convert KDE's translation helper scripts to support git \nrepositories for kde modules. the translators themselves are staying in svn \nfor now, but we want to start migrating kde modules over to git.\n\nI've hit a bit of a problem, though.\ninside the translators' folder, there's a folder called documentation/ - and \ninside that is a bunch of svn externals pointing to the doc/ folder of every \nkde module. (this is, then, all of kde's english documentation.)\ntranslators occasionally need to run a script that reads from the \ndocumentation/ folder.\n\nnow, what happens when one of those modules switches to git? first, the svn \nexternal goes away. that's ok, I can work around that by having the update_xml \nscript (the one translators need to run) fetch the doc folder itself, right?\nbut where does it fetch it from? doc/ is now a subdirectory in a git \nrepository, and git doesn't support checking out only one folder, not even if \nall I want to do is read the files in it.\n\nchecking out the entire git repo isn't really feasible, because once KDE \nfinishes switching to git that'll mean checking out all of KDE.\nI don't think that turning every doc folder into a submodule is feasible, \neither - they sound kinda awkward to work with, and I can imagine lots of \npeople getting confused and messing them up...\n\nright now all I've thought of is one ugly hack: have a server that checks out \nall the kde git repos, pulls daily, copies all the doc/ folders into a \ndocumentation folder, and offers that folder up on the interwebs so that \nupdate_xml can rsync from it or download a tgz of it or something.\nthere appear to be lots of images in the documentation, so it's not a small \ndownload - 200mb and growing. it still hasn't finished downloading all the \nexternals...\n\nI'm kinda wondering if there'd be a way to use git-filter-branch to make a repo \nthat only tracks the doc/ folder for a module - but I've no idea whether it'd \nhave to be recreated from scratch every time someone changes something in the \nreal repo's doc/\n\ncan anyone think of a less ugly solution?\nwhat are the chances of git supporting this kind of partial checkout someday?\n\n-- \nThis message brought to you by eevil bananas and the number 3.\nwww.chani3.com\n"},{"id":"114541","messageId":"200905240134.53387.johan@herland.net","threadId":"19459","inReplyTo":"200905231401.11651.chanika@gmail.com","subject":"Re: partial checkouts","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-05-23T23:34:53Z","receivedAt":"2009-05-23T23:34:53Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Saturday 23 May 2009, Chani wrote:\n> [...]\n>\n> right now all I've thought of is one ugly hack: have a server that checks\n> out all the kde git repos, pulls daily, copies all the doc/ folders into\n> a documentation folder, and offers that folder up on the interwebs so\n> that update_xml can rsync from it or download a tgz of it or something.\n> there appear to be lots of images in the documentation, so it's not a\n> small download - 200mb and growing. it still hasn't finished downloading\n> all the externals...\n\nDo you need the doc/ folders from _all_ kde git repos, or just from those \nrepos that you have currently checked out? In the latter case, you could \nsolve this by adding symlinks to all the doc/ folders inside the \ndocumentation/ folder, and then make sure the software that traverse the \ndocumentation/ folder recognize and skips symlinks. Of course, this won't \nwork if the translations project need _all_ doc/ folders accessible, but not \nall the kde git repos.\n\n> I'm kinda wondering if there'd be a way to use git-filter-branch to make\n> a repo that only tracks the doc/ folder for a module - but I've no idea\n> whether it'd have to be recreated from scratch every time someone changes\n> something in the real repo's doc/\n>\n> can anyone think of a less ugly solution?\n> what are the chances of git supporting this kind of partial checkout\n> someday?\n\nCheck out git-subtree. It can split out a subdirectory into its own repo, \nand re-integrate it back into the \"parent\" repo at a later date. git-subtree \nhas been posted as a patch to this list a couple of times without much \nresponse, but it looks like an interesting alternative to submodules:\nhttp://alumnit.ca/~apenwarr/log/?m=200904#30\n\nIf a lot of people find git-subtree useful, who knows, it might be included \nin a future git version.\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"114543","messageId":"200905231726.10361.chanika@gmail.com","threadId":"19459","inReplyTo":"200905240134.53387.johan@herland.net","subject":"Re: partial checkouts","fromName":"Chani","fromEmail":"chanika@gmail.com","sentAt":"2009-05-24T00:26:09Z","receivedAt":"2009-05-24T00:26:09Z","isPatch":false,"sender":{"key":"chanika@gmail.com","avatar":"https://gravatar.com/avatar/118096b4e8fe9991f3026ffc7aa39a4a925d265856f037a5012098f9e5d1a993?d=mp&s=160"},"body":"On May 23, 2009 16:34:53 Johan Herland wrote:\n> On Saturday 23 May 2009, Chani wrote:\n> > [...]\n> >\n> > right now all I've thought of is one ugly hack: have a server that checks\n> > out all the kde git repos, pulls daily, copies all the doc/ folders into\n> > a documentation folder, and offers that folder up on the interwebs so\n> > that update_xml can rsync from it or download a tgz of it or something.\n> > there appear to be lots of images in the documentation, so it's not a\n> > small download - 200mb and growing. it still hasn't finished downloading\n> > all the externals...\n>\n> Do you need the doc/ folders from _all_ kde git repos, or just from those\n> repos that you have currently checked out? In the latter case, you could\n> solve this by adding symlinks to all the doc/ folders inside the\n> documentation/ folder, and then make sure the software that traverse the\n> documentation/ folder recognize and skips symlinks. Of course, this won't\n> work if the translations project need _all_ doc/ folders accessible, but\n> not all the kde git repos.\n\nnope, the translators may not have checked out *any* of them but the script \nthey want to run needs *all* the docs. :( however, I've been told they also \ndon't want to have to change their workflow in any way at all no matter how \nsmall, so we may be stuck in svn-land anyways, because you can't make an svn \nexternal out of something that's not in svn, and having anything other than \nsvn externals would change their workflow :(\n\n>\n> > I'm kinda wondering if there'd be a way to use git-filter-branch to make\n> > a repo that only tracks the doc/ folder for a module - but I've no idea\n> > whether it'd have to be recreated from scratch every time someone changes\n> > something in the real repo's doc/\n> >\n> > can anyone think of a less ugly solution?\n> > what are the chances of git supporting this kind of partial checkout\n> > someday?\n>\n> Check out git-subtree. It can split out a subdirectory into its own repo,\n> and re-integrate it back into the \"parent\" repo at a later date.\n> git-subtree has been posted as a patch to this list a couple of times\n> without much response, but it looks like an interesting alternative to\n> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30\n>\n> If a lot of people find git-subtree useful, who knows, it might be included\n> in a future git version.\n\nlooks interesting. might have been a solution until I heard about this \nrequirement to not change workflow at all. :/\n\nhowever, my friend told me about a project to make a git-svnserver that serves \ngit repos as svn repos, and *that* would allow the translators to stay where \nthey are without holding everyone else there too. know anything about that? \n\nmm, google turns up an email from someone claiming they have a partial \nimplementation in python...\n\n-- \nThis message brought to you by eevil bananas and the number 3.\nwww.chani3.com\n"},{"id":"114545","messageId":"fcaeb9bf0905231907q16160ad1t8aa8ef71e2adc8b0@mail.gmail.com","threadId":"19459","inReplyTo":"200905231401.11651.chanika@gmail.com","subject":"Re: partial checkouts","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-24T02:07:37Z","receivedAt":"2009-05-24T02:07:37Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:\n> checking out the entire git repo isn't really feasible, because once KDE\n> finishes switching to git that'll mean checking out all of KDE.\n> I don't think that turning every doc folder into a submodule is feasible,\n> either - they sound kinda awkward to work with, and I can imagine lots of\n> people getting confused and messing them up...\n\nIf you don't mind cloning the entire git repo (which may be quite\nlarge in KDE case, I guess), then you may want to try \"sparse\ncheckout\". The patch series is available as a topic branch \"tp/sco\" in\nthis branch:\n\nhttp://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst\n\nOr I can send the patch series to you. It lets you specify what part\nof the tree you want to checkout.\n-- \nDuy\n"},{"id":"114576","messageId":"18071eea0905240857x2433e7a1x5776def3209afbf5@mail.gmail.com","threadId":"19459","inReplyTo":"fcaeb9bf0905231907q16160ad1t8aa8ef71e2adc8b0@mail.gmail.com","subject":"Re: partial checkouts","fromName":"Thomas Adam","fromEmail":"thomas.adam22@gmail.com","sentAt":"2009-05-24T15:57:52Z","receivedAt":"2009-05-24T15:57:52Z","isPatch":false,"sender":{"key":"thomas.adam22@gmail.com","avatar":"https://gravatar.com/avatar/137f9858bc6bfd5b2f743aefd988c81ce0cbd306248889df80e269519cfc8741?d=mp&s=160"},"body":"2009/5/24 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:\n> On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:\n>> checking out the entire git repo isn't really feasible, because once KDE\n>> finishes switching to git that'll mean checking out all of KDE.\n>> I don't think that turning every doc folder into a submodule is feasible,\n>> either - they sound kinda awkward to work with, and I can imagine lots of\n>> people getting confused and messing them up...\n>\n> If you don't mind cloning the entire git repo (which may be quite\n> large in KDE case, I guess), then you may want to try \"sparse\n> checkout\". The patch series is available as a topic branch \"tp/sco\" in\n> this branch:\n>\n> http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst\n>\n> Or I can send the patch series to you. It lets you specify what part\n> of the tree you want to checkout.\n\nI recall following this series when you first announced it.  Do you\nhave any plans to try and get it included in mainline?  ISTR there was\nsome more work needed to be done on it before then.\n\n-- Thomas Adam\n"},{"id":"114587","messageId":"32541b130905241245m7139c766v73f6fe2aeda1fcf9@mail.gmail.com","threadId":"19459","inReplyTo":"200905231726.10361.chanika@gmail.com","subject":"Re: partial checkouts","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-05-24T19:45:12Z","receivedAt":"2009-05-24T19:45:12Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Sat, May 23, 2009 at 8:26 PM, Chani <chanika@gmail.com> wrote:\n> On May 23, 2009 16:34:53 Johan Herland wrote:\n>> On Saturday 23 May 2009, Chani wrote:\n>> > I'm kinda wondering if there'd be a way to use git-filter-branch to make\n>> > a repo that only tracks the doc/ folder for a module - but I've no idea\n>> > whether it'd have to be recreated from scratch every time someone changes\n>> > something in the real repo's doc/\n>> >\n>> > can anyone think of a less ugly solution?\n>> > what are the chances of git supporting this kind of partial checkout\n>> > someday?\n>>\n>> Check out git-subtree. It can split out a subdirectory into its own repo,\n>> and re-integrate it back into the \"parent\" repo at a later date.\n>> git-subtree has been posted as a patch to this list a couple of times\n>> without much response, but it looks like an interesting alternative to\n>> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30\n>\n> looks interesting. might have been a solution until I heard about this\n> requirement to not change workflow at all. :/\n\nHere's something that might work.  Write a script that runs on a\nserver somewhere.  It checks out *all* of the git-managed KDE\ncomponents, including documentation (it's okay to do this from *one*\nplace, right?).  Then it loops through all the documentation\nsubdirectories and splits them out into individual projects/branches\nwith 'git subtree split'.\n\nYou could then make a *new* project, kde-docs for example, that pulls\nthose new doc projects back together into a single project, using 'git\nsubtree merge'.\n\nPeople could then checkout the single kde-docs project and get *all*\nthe docs, then make changes and push them back upstream to you.  As\nfar as the average person is concerned, this is probably even easier\nthan svn:externals - they don't have to remember to commit in each\nsubdirectory separately, as git-subtree rips it all apart later.  Also\nnote that *end users* never have to operate the git subtree command;\nall that happens behind the scenes on your server.\n\nLater, you use 'git subtree split' to rip it apart back into the\nindividual projects, and 'git subtree merge' those back into their\noriginal git repositories.\n\nSince git-subtree is my own little toy and it's designed for exactly\nthis kind of workflow, I'd be happy to help you debug a script for\nthis if you like.  Probably I'll learn something.\n\nDisclaimer: git-subtree hasn't been used by too many people yet and we\nmay end up finding some bugs.\n\nHave fun,\n\nAvery\n"},{"id":"114618","messageId":"fcaeb9bf0905241851i768b4c5bl9bca9faeb12ff4a@mail.gmail.com","threadId":"19459","inReplyTo":"18071eea0905240857x2433e7a1x5776def3209afbf5@mail.gmail.com","subject":"Re: partial checkouts","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-25T01:51:46Z","receivedAt":"2009-05-25T01:51:46Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 25, 2009 at 1:57 AM, Thomas Adam <thomas.adam22@gmail.com> wrote:\n> 2009/5/24 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:\n>> On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:\n>>> checking out the entire git repo isn't really feasible, because once KDE\n>>> finishes switching to git that'll mean checking out all of KDE.\n>>> I don't think that turning every doc folder into a submodule is feasible,\n>>> either - they sound kinda awkward to work with, and I can imagine lots of\n>>> people getting confused and messing them up...\n>>\n>> If you don't mind cloning the entire git repo (which may be quite\n>> large in KDE case, I guess), then you may want to try \"sparse\n>> checkout\". The patch series is available as a topic branch \"tp/sco\" in\n>> this branch:\n>>\n>> http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst\n>>\n>> Or I can send the patch series to you. It lets you specify what part\n>> of the tree you want to checkout.\n>\n> I recall following this series when you first announced it.  Do you\n> have any plans to try and get it included in mainline?  ISTR there was\n> some more work needed to be done on it before then.\n\nIt was pulled off because of big, intrusive and not well-designed patches.\n\nhttp://mid.gmane.org/20081128192033.GF23984@spearce.org\n\nI may submit it again when I get code insertions down to ~500 lines or\nsomething like that ;-) Anyway you could always help by giving more\nuse cases to test against the current design.\n-- \nDuy\n"},{"id":"114619","messageId":"200905242007.26575.chanika@gmail.com","threadId":"19459","inReplyTo":"32541b130905241245m7139c766v73f6fe2aeda1fcf9@mail.gmail.com","subject":"Re: partial checkouts","fromName":"Chani","fromEmail":"chanika@gmail.com","sentAt":"2009-05-25T03:07:18Z","receivedAt":"2009-05-25T03:07:18Z","isPatch":false,"sender":{"key":"chanika@gmail.com","avatar":"https://gravatar.com/avatar/118096b4e8fe9991f3026ffc7aa39a4a925d265856f037a5012098f9e5d1a993?d=mp&s=160"},"body":"> >>\n> >> Check out git-subtree. It can split out a subdirectory into its own\n> >> repo, and re-integrate it back into the \"parent\" repo at a later date.\n> >> git-subtree has been posted as a patch to this list a couple of times\n> >> without much response, but it looks like an interesting alternative to\n> >> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30\n> >\n> > looks interesting. might have been a solution until I heard about this\n> > requirement to not change workflow at all. :/\n>\n> Here's something that might work.  Write a script that runs on a\n> server somewhere.  It checks out *all* of the git-managed KDE\n> components, including documentation (it's okay to do this from *one*\n> place, right?).  Then it loops through all the documentation\n> subdirectories and splits them out into individual projects/branches\n> with 'git subtree split'.\n>\n\nI like this idea. I like the idea of git subtree in general. :)\nhowever, we've decided to go the \"copy all documentation into svn\" route for \nnow (so that the translators can keep using only svn).\nI'll be back when we run into the next svn-external issue (there's sure to be \nmore). :)\n\n-- \nThis message brought to you by eevil bananas and the number 3.\nwww.chani3.com\n"},{"id":"114672","messageId":"20090525145131.GG6295@yugib.highrise.ca","threadId":"19459","inReplyTo":"32541b130905241245m7139c766v73f6fe2aeda1fcf9@mail.gmail.com","subject":"Re: partial checkouts","fromName":"Aidan Van Dyk","fromEmail":"aidan@highrise.ca","sentAt":"2009-05-25T14:51:31Z","receivedAt":"2009-05-25T14:51:31Z","isPatch":false,"sender":{"key":"aidan@highrise.ca","avatar":"https://gravatar.com/avatar/853c50d90cce753dc1c390fdc6cbed558f5f969bd43fa4f5cb0118d8f71316f6?d=mp&s=160"},"body":"If all you're looking for is a \"single set of read-only docs\" conglomeration of\nall the KDE submodules projects documentation so you can translate them, what\nabout using \"archive\"?  Since archive takes a \"tree-ish\", and work on remote\nupload-pack servers (if it's enabled), you can get an un-versioned \"remote\npartial checkout\" with something like do things like:\n\tgit archive --format=tar --prefix=git/Documentation/ --remote=git://repo.or.cz/git.git HEAD:Documentation | tar tf - | head\n\tgit archive --format=tar --prefix=/opt/git/share/man/ --remote=git://repo.or.cz/git.git man | tar tf -\n\n\n\nSo, this could be simply scripted to give you a repo with *all* modules docs/ inside a single repo...\n\tfor m in $MODULES\n\tdo\n\t\tgit archive --format=tar --prefix=\"$m/$DOCDIR/\" --remote=$REMOTE_BASE/$m $BRANCH:$DOCDIR | tar xf -\n\t\tgit add $m/$DOCDIR\n\t\tgit commit $m/$DOCDIR -m \"$m: Automatic import\"\n\tdone\n\nDo that in a central location an dlet translators pull that...  Everythings\n\"duplicated\", but git's object store will help out there...  You loose the\nhistory of the original docs, but for translations, you you're not translating\nthe history anyways, your'e translation the docs at a point in time...\n\nOr just dump all thd docs in a .gitignore'd subdirctory and have translators\nrun a script to update them.\n\na.\n\n\n> On Sat, May 23, 2009 at 8:26 PM, Chani <chanika@gmail.com> wrote:\n> > On May 23, 2009 16:34:53 Johan Herland wrote:\n> >> On Saturday 23 May 2009, Chani wrote:\n> >> > I'm kinda wondering if there'd be a way to use git-filter-branch to make\n> >> > a repo that only tracks the doc/ folder for a module - but I've no idea\n> >> > whether it'd have to be recreated from scratch every time someone changes\n> >> > something in the real repo's doc/\n> >> >\n> >> > can anyone think of a less ugly solution?\n> >> > what are the chances of git supporting this kind of partial checkout\n> >> > someday?\n \n> Here's something that might work.  Write a script that runs on a\n> server somewhere.  It checks out all of the git-managed KDE\n> components, including documentation (it's okay to do this from one\n> place, right?).  Then it loops through all the documentation\n> subdirectories and splits them out into individual projects/branches\n> with 'git subtree split'.\n> \n> You could then make a new project, kde-docs for example, that pulls\n> those new doc projects back together into a single project, using 'git\n> subtree merge'.\n> \n> People could then checkout the single kde-docs project and get all\n> the docs, then make changes and push them back upstream to you.  As\n> far as the average person is concerned, this is probably even easier\n> than svn:externals - they don't have to remember to commit in each\n> subdirectory separately, as git-subtree rips it all apart later.  Also\n> note that *end users* never have to operate the git subtree command;\n> all that happens behind the scenes on your server.\n> \n> Later, you use 'git subtree split' to rip it apart back into the\n> individual projects, and 'git subtree merge' those back into their\n> original git repositories.\n> \n> Since git-subtree is my own little toy and it's designed for exactly\n> this kind of workflow, I'd be happy to help you debug a script for\n> this if you like.  Probably I'll learn something.\n> \n> Disclaimer: git-subtree hasn't been used by too many people yet and we\n> may end up finding some bugs.\n> \n> Have fun,\n> \n> Avery\n> \n"}]}