svn:externals using git submodules

> Hello,
> 
> I've done this by hand as a proof of concept I suspect it would need loads of 
> work in git-svn to do it properly.  However, I thought I'd mention as part of 
> my "success with submodules" reports.

> ffmpeg is managed with svn; I like to track its development with git-svn.  
> Works wonderfully except for one problem: they've made use of svn:externals 
> for one component, libswscale.  Previously I just regularly updated the 
> libswscale subdirectory by checking out the latest copy (which is all that 
> subversion does) and committing it to my own branch off upstream.
> 
> With submodule support in git, it makes it possible to do a much better job.  
> What I did was have two svn-remote sections in the config:
> 
> [svn-remote "ffmpeg"]
>     url = svn://svn.mplayerhq.hu/ffmpeg
>     fetch = trunk:refs/remotes/ffmpeg-svn
> 
> [svn-remote "libswscale"]
>     url = svn://svn.mplayerhq.hu/mplayer
>     fetch = trunk/libswscale:refs/remotes/libswscale-svn
> 
> After running git-svn fetch; there are two independent branches in my 
> repository:
> 
>   -- * -- * -- * -- * -- * (ffmpeg-svn)
>   ---- * ----- * ------- * (libswscale-svn)  
> 
> Now, we fork from ffmpeg-svn and libswscale-svn to make non-tracking branches 
> that can be committed to:
> 
>  $ git checkout -b master-ffm ffmpeg-svn
>  $ git branch master-sws libswscale-svn
> 
> Next, we create a shared clone of the repository as a subdirectory in that 
> repository.
> 
>  $ git clone -s . libswscale
> 
> Now we want that clone to be even more strongly linked to the parent - to the 
> extent that they share the same refs, etc:
> 
>  $ cd libswscale
>  $ rm -rf .git/refs .git/logs .git/info description config
>  $ ln -s ../../.git/refs .git/refs
>  $ ln -s ../../.git/logs .git/logs
>  $ ln -s ../../.git/info .git/info
>  $ ln -s ../../.git/config .git/config
>  $ ln -s ../../.git/description .git/description
> 
> Only HEAD and index are independent.  Next we switch from the ffmpeg branch to 
> the libswscale branch in this subdirectory:
> 
>  $ git checkout master-sws
> 
> Now, we make the subdirectory a submodule in the parent:
> 
>  $ cd ..
>  $ git add libswscale
>  $ git commit -m "libswscale is now a submodule"
> 
> How dangerous is this?  I've made the repository it's own submodule and it 
> shares the same refs, info and logs.  LIVING ON THE EDGE MAN!
> 
> You have to run two git-svn commands to sync with upstream:
> 
>  $ git-svn fetch ffmpeg
>  $ git-svn fetch libswscale
> 
> Then of course you would merge
> 
>  $ git merge ffmpeg-svn
>  $ cd libswscale; git merge libswscale-svn; cd ..
>  $ git commit -m "Sync with upstream"
> 
> Personally I think that's pretty cool, this is significantly better than 
> svn:externals because the particular revision of libswscale in use is 
> recorded.  Seriously - someone show me another VCS that can do that - I think 
> git has actual magic powers :-)
> 
> I dare say that git-svn could do much better because it could reconstruct the 
> submodule history based on the repository dates and create the link in the 
> tracking branch rather than having to do it manually at the end as I've done 
> here.  That would mean that the recorded submodule was right for all 
> history - again, not the case for svn:externals, if you check out a previous 
> version the external remains current.
> 
> 
> 
> Andy
> -- 
> Dr Andy Parkins, M Eng (hons), MIET
> andyparkins@gmail.com
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

> On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote:
> > Hello,
> > 
> > I've done this by hand as a proof of concept I suspect it would need loads of 
> > work in git-svn to do it properly.  However, I thought I'd mention as part of 
> > my "success with submodules" reports.
> 
> For my part, I wonder if it can be simplified somehow; and I suspect
> it doesn't work well with svn:externals that specify a particular
> revision.

> Chris Shoemaker <c.shoemaker@cox.net> wrote:
> > On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote:
> > > Hello,
> > > 
> > > I've done this by hand as a proof of concept I suspect it would need loads of 
> > > work in git-svn to do it properly.  However, I thought I'd mention as part of 
> > > my "success with submodules" reports.
> > 
> > For my part, I wonder if it can be simplified somehow; and I suspect
> > it doesn't work well with svn:externals that specify a particular
> > revision.
> 
> Actually that is an interesting point that Chris makes.  Isn't the
> svn:externals property revision controlled on the parent directory?
> So each change to it is actually recorded in the revision history
> of the parent project.  

> On Tue, May 01, 2007 at 11:22:28AM -0400, Shawn O. Pearce wrote:
> > And if every svn:externals URL included the
> > exact version of the other project to include, aren't svn:externals
> > then more-or-less like the subproject link support, except they
> > also include the URL?
> 
> Just to clarify, my point was just that Andy's setup seems to assume
> that the externals don't specify a revision.  If they do, maybe
> git-svn can map the externals into subprojects.  Is this what
> you're thinking?

> > Actually that is an interesting point that Chris makes.  Isn't the
> > svn:externals property revision controlled on the parent directory?
> > So each change to it is actually recorded in the revision history
> > of the parent project.
>
> Yes and yes.

> > And if every svn:externals URL included the
> > exact version of the other project to include, aren't svn:externals
> > then more-or-less like the subproject link support, except they
> > also include the URL?
>
> Just to clarify, my point was just that Andy's setup seems to assume
> that the externals don't specify a revision.  If they do, maybe

> On Tuesday 2007, May 01, Chris Shoemaker wrote:
> 
> > > Actually that is an interesting point that Chris makes.  Isn't the
> > > svn:externals property revision controlled on the parent directory?
> > > So each change to it is actually recorded in the revision history
> > > of the parent project.
> >
> > Yes and yes.
> 
> Yes and no.  Think of svn:externals as a file in the parent repository; 
> it contains
> 
>  directory-name URL
> 
> Now, changes to that file _are_ tracked, in that if I changed the URL 
> that change would be recorded in the parent repository.  However, 
> nowhere is the revision of the external recorded.  Subversion always 
> fetches the latest revision at that URL.

> 
> > > And if every svn:externals URL included the
> > > exact version of the other project to include, aren't svn:externals
> > > then more-or-less like the subproject link support, except they
> > > also include the URL?
> >
> > Just to clarify, my point was just that Andy's setup seems to assume
> > that the externals don't specify a revision.  If they do, maybe
> 
> They don't.  If they did, they'd be just as useful as git's submodules.
>
> > git-svn can map the externals into subprojects.  Is this what
> > you're thinking?
> 
> Well, I'm thinking that that information /can/ be reconstructed from the 
> revision date information - kind of - the problem is that there is no 
> way to know when the parent updated the module.   svn:externals really 
> is just a quick way of doing
>  $ cd submodule
>  $ svn update
> That's it.  That's all you get.  We could guess that when the parent 
> module was at date YYYY-MM-DD, that the submodule would be at that same 
> date - but who knows?

> On Tuesday 2007, May 01, Chris Shoemaker wrote:
> 
> > That's only true when the revision is not specified in the external.
> > The repo you track may not do that, but it's not uncommon to do so.
> 
> It's been a while since I used subversion, and even longer since I used 
> externals - is that a new feature?  

> I used subversion since before 
> version 1.0, so I often missed new features when they arrived. 
> 
> > And, as I think you're pointing out, it's the only way to get any
> > sort of reliable information about the relationship between the
> > parent and the external.
> 
> Does subversion automatically update that fixed attachment when you 
> update the submodule?  I would have found that quite useful back then.

> > I think it would probably be undesirable for git-svn to attempt to
> > convert "floating" externals into well-versioned submodules, since
> > they're not even well-versioned in the svn repo.  However, handling
> > the "locked-down" externals is quite another thing.
> 
> Absolutely.  If the information is available, then git is certainly 
> capable of recording it.  It sounds like subversion has a facility I 
> didn't know exist, so I've been bad mouthing it more than I should.  Oh 
> well :-)
> 

> 
> Making git-svn handle svn:externals with specified revisions would be
> _quite_ useful.  There's a special-case of this that I use personally:
> svn:externals that point to other paths (and other revisions) of the
> parent repo.

> On Tue, 1 May 2007, Chris Shoemaker wrote:
>> 
>> Making git-svn handle svn:externals with specified revisions would be
>> _quite_ useful.  There's a special-case of this that I use personally:
>> svn:externals that point to other paths (and other revisions) of the
>> parent repo.
>
> Side note: even _without_ a specified revision, I think it's quite sane to 
> have the rule that a submodule hash of all zeroes is "unversioned".

> But while I'm encouraged that the whole gitlink thing seems to be working 
> for Andy, and some others are playing with it too, I'm also a bit 
> discouraged by the fact that there hasn't been any noise or work on the 
> porcelain side. I was obviously optimistic and hoping we'd see support in 
> checkout/diff, but I haven't heard anybody talk about actually 
> implementing .gitmodules and the porcelain support that uses them..

> Linus Torvalds <torvalds@linux-foundation.org> writes:
> > But while I'm encouraged that the whole gitlink thing seems to be working 
> > for Andy, and some others are playing with it too, I'm also a bit 
> > discouraged by the fact that there hasn't been any noise or work on the 
> > porcelain side. I was obviously optimistic and hoping we'd see support in 
> > checkout/diff, but I haven't heard anybody talk about actually 
> > implementing .gitmodules and the porcelain support that uses them..
> 
> The thing is, almost all the core git people happen to be busy
> at the same time at this moment.  Johannes has just moved, Shawn
> and I are deep in day-jobs to the neck, ...

> 
> Hello,
> 
> I've done this by hand as a proof of concept I suspect it would need loads of 
> work in git-svn to do it properly.  However, I thought I'd mention as part of 
> my "success with submodules" reports.
> 
> ... 
> Now we want that clone to be even more strongly linked to the parent - to the 
> extent that they share the same refs, etc:
> 
>  $ cd libswscale
>  $ rm -rf .git/refs .git/logs .git/info description config
>  $ ln -s ../../.git/refs .git/refs
>  $ ln -s ../../.git/logs .git/logs
>  $ ln -s ../../.git/info .git/info
>  $ ln -s ../../.git/config .git/config
>  $ ln -s ../../.git/description .git/description
> 
> ...
> Andy