# svn:externals using git submodules 14 messages from 2007-05-01 to 2010-01-20. Participants: Andy Parkins, Chris Shoemaker, Shawn O. Pearce, Linus Torvalds, Junio C Hamano, Michel Jouvin. Thread: https://gitlist.dev/t/7926 ## Andy Parkins, 2007-05-01 10:21 Subject: svn:externals using git submodules Message-ID: <200705011121.17172.andyparkins@gmail.com> URL: https://gitlist.dev/e/200705011121.17172.andyparkins%40gmail.com ``` Hello, I've done this by hand as a proof of concept I suspect it would need loads of work in git-svn to do it properly. However, I thought I'd mention as part of my "success with submodules" reports. ffmpeg is managed with svn; I like to track its development with git-svn. Works wonderfully except for one problem: they've made use of svn:externals for one component, libswscale. Previously I just regularly updated the libswscale subdirectory by checking out the latest copy (which is all that subversion does) and committing it to my own branch off upstream. With submodule support in git, it makes it possible to do a much better job. What I did was have two svn-remote sections in the config: [svn-remote "ffmpeg"] url = svn://svn.mplayerhq.hu/ffmpeg fetch = trunk:refs/remotes/ffmpeg-svn [svn-remote "libswscale"] url = svn://svn.mplayerhq.hu/mplayer fetch = trunk/libswscale:refs/remotes/libswscale-svn After running git-svn fetch; there are two independent branches in my repository: -- * -- * -- * -- * -- * (ffmpeg-svn) ---- * ----- * ------- * (libswscale-svn) Now, we fork from ffmpeg-svn and libswscale-svn to make non-tracking branches that can be committed to: $ git checkout -b master-ffm ffmpeg-svn $ git branch master-sws libswscale-svn Next, we create a shared clone of the repository as a subdirectory in that repository. $ git clone -s . libswscale Now we want that clone to be even more strongly linked to the parent - to the extent that they share the same refs, etc: $ cd libswscale $ rm -rf .git/refs .git/logs .git/info description config $ ln -s ../../.git/refs .git/refs $ ln -s ../../.git/logs .git/logs $ ln -s ../../.git/info .git/info $ ln -s ../../.git/config .git/config $ ln -s ../../.git/description .git/description Only HEAD and index are independent. Next we switch from the ffmpeg branch to the libswscale branch in this subdirectory: $ git checkout master-sws Now, we make the subdirectory a submodule in the parent: $ cd .. $ git add libswscale $ git commit -m "libswscale is now a submodule" How dangerous is this? I've made the repository it's own submodule and it shares the same refs, info and logs. LIVING ON THE EDGE MAN! You have to run two git-svn commands to sync with upstream: $ git-svn fetch ffmpeg $ git-svn fetch libswscale Then of course you would merge $ git merge ffmpeg-svn $ cd libswscale; git merge libswscale-svn; cd .. $ git commit -m "Sync with upstream" Personally I think that's pretty cool, this is significantly better than svn:externals because the particular revision of libswscale in use is recorded. Seriously - someone show me another VCS that can do that - I think git has actual magic powers :-) I dare say that git-svn could do much better because it could reconstruct the submodule history based on the repository dates and create the link in the tracking branch rather than having to do it manually at the end as I've done here. That would mean that the recorded submodule was right for all history - again, not the case for svn:externals, if you check out a previous version the external remains current. Andy -- Dr Andy Parkins, M Eng (hons), MIET andyparkins@gmail.com ``` ## Chris Shoemaker, 2007-05-01 15:07 Subject: Re: svn:externals using git submodules Message-ID: <20070501150724.GA20797@pe.Belkin> URL: https://gitlist.dev/e/20070501150724.GA20797%40pe.Belkin In-Reply-To: <200705011121.17172.andyparkins@gmail.com> ``` On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote: > Hello, > > I've done this by hand as a proof of concept I suspect it would need loads of > work in git-svn to do it properly. However, I thought I'd mention as part of > my "success with submodules" reports. That's truly interesting, Andy. Thanks for the encouraging report. Please do keep us informed of anything more you conclude about this approach. I'm sure some of the experts around here can respond with the pros and cons of this technique. For my part, I wonder if it can be simplified somehow; and I suspect it doesn't work well with svn:externals that specify a particular revision. -chris > ffmpeg is managed with svn; I like to track its development with git-svn. > Works wonderfully except for one problem: they've made use of svn:externals > for one component, libswscale. Previously I just regularly updated the > libswscale subdirectory by checking out the latest copy (which is all that > subversion does) and committing it to my own branch off upstream. > > With submodule support in git, it makes it possible to do a much better job. > What I did was have two svn-remote sections in the config: > > [svn-remote "ffmpeg"] > url = svn://svn.mplayerhq.hu/ffmpeg > fetch = trunk:refs/remotes/ffmpeg-svn > > [svn-remote "libswscale"] > url = svn://svn.mplayerhq.hu/mplayer > fetch = trunk/libswscale:refs/remotes/libswscale-svn > > After running git-svn fetch; there are two independent branches in my > repository: > > -- * -- * -- * -- * -- * (ffmpeg-svn) > ---- * ----- * ------- * (libswscale-svn) > > Now, we fork from ffmpeg-svn and libswscale-svn to make non-tracking branches > that can be committed to: > > $ git checkout -b master-ffm ffmpeg-svn > $ git branch master-sws libswscale-svn > > Next, we create a shared clone of the repository as a subdirectory in that > repository. > > $ git clone -s . libswscale > > Now we want that clone to be even more strongly linked to the parent - to the > extent that they share the same refs, etc: > > $ cd libswscale > $ rm -rf .git/refs .git/logs .git/info description config > $ ln -s ../../.git/refs .git/refs > $ ln -s ../../.git/logs .git/logs > $ ln -s ../../.git/info .git/info > $ ln -s ../../.git/config .git/config > $ ln -s ../../.git/description .git/description > > Only HEAD and index are independent. Next we switch from the ffmpeg branch to > the libswscale branch in this subdirectory: > > $ git checkout master-sws > > Now, we make the subdirectory a submodule in the parent: > > $ cd .. > $ git add libswscale > $ git commit -m "libswscale is now a submodule" > > How dangerous is this? I've made the repository it's own submodule and it > shares the same refs, info and logs. LIVING ON THE EDGE MAN! > > You have to run two git-svn commands to sync with upstream: > > $ git-svn fetch ffmpeg > $ git-svn fetch libswscale > > Then of course you would merge > > $ git merge ffmpeg-svn > $ cd libswscale; git merge libswscale-svn; cd .. > $ git commit -m "Sync with upstream" > > Personally I think that's pretty cool, this is significantly better than > svn:externals because the particular revision of libswscale in use is > recorded. Seriously - someone show me another VCS that can do that - I think > git has actual magic powers :-) > > I dare say that git-svn could do much better because it could reconstruct the > submodule history based on the repository dates and create the link in the > tracking branch rather than having to do it manually at the end as I've done > here. That would mean that the recorded submodule was right for all > history - again, not the case for svn:externals, if you check out a previous > version the external remains current. > > > > Andy > -- > Dr Andy Parkins, M Eng (hons), MIET > andyparkins@gmail.com > - > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ``` ## Shawn O. Pearce, 2007-05-01 15:22 Subject: Re: svn:externals using git submodules Message-ID: <20070501152228.GF5942@spearce.org> URL: https://gitlist.dev/e/20070501152228.GF5942%40spearce.org In-Reply-To: <20070501150724.GA20797@pe.Belkin> ``` Chris Shoemaker wrote: > On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote: > > Hello, > > > > I've done this by hand as a proof of concept I suspect it would need loads of > > work in git-svn to do it properly. However, I thought I'd mention as part of > > my "success with submodules" reports. > > For my part, I wonder if it can be simplified somehow; and I suspect > it doesn't work well with svn:externals that specify a particular > revision. Actually that is an interesting point that Chris makes. Isn't the svn:externals property revision controlled on the parent directory? So each change to it is actually recorded in the revision history of the parent project. And if every svn:externals URL included the exact version of the other project to include, aren't svn:externals then more-or-less like the subproject link support, except they also include the URL? -- Shawn. ``` ## Chris Shoemaker, 2007-05-01 15:36 Subject: Re: svn:externals using git submodules Message-ID: <20070501153626.GA21182@pe.Belkin> URL: https://gitlist.dev/e/20070501153626.GA21182%40pe.Belkin In-Reply-To: <20070501152228.GF5942@spearce.org> ``` On Tue, May 01, 2007 at 11:22:28AM -0400, Shawn O. Pearce wrote: > Chris Shoemaker wrote: > > On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote: > > > Hello, > > > > > > I've done this by hand as a proof of concept I suspect it would need loads of > > > work in git-svn to do it properly. However, I thought I'd mention as part of > > > my "success with submodules" reports. > > > > For my part, I wonder if it can be simplified somehow; and I suspect > > it doesn't work well with svn:externals that specify a particular > > revision. > > Actually that is an interesting point that Chris makes. Isn't the > svn:externals property revision controlled on the parent directory? > So each change to it is actually recorded in the revision history > of the parent project. Yes and yes. > And if every svn:externals URL included the > exact version of the other project to include, aren't svn:externals > then more-or-less like the subproject link support, except they > also include the URL? Just to clarify, my point was just that Andy's setup seems to assume that the externals don't specify a revision. If they do, maybe git-svn can map the externals into subprojects. Is this what you're thinking? -chris > > -- > Shawn. ``` ## Shawn O. Pearce, 2007-05-01 15:40 Subject: Re: svn:externals using git submodules Message-ID: <20070501154056.GH5942@spearce.org> URL: https://gitlist.dev/e/20070501154056.GH5942%40spearce.org In-Reply-To: <20070501153626.GA21182@pe.Belkin> ``` Chris Shoemaker wrote: > On Tue, May 01, 2007 at 11:22:28AM -0400, Shawn O. Pearce wrote: > > And if every svn:externals URL included the > > exact version of the other project to include, aren't svn:externals > > then more-or-less like the subproject link support, except they > > also include the URL? > > Just to clarify, my point was just that Andy's setup seems to assume > that the externals don't specify a revision. If they do, maybe > git-svn can map the externals into subprojects. Is this what > you're thinking? Yes. ;-) -- Shawn. ``` ## Andy Parkins, 2007-05-01 18:36 Subject: Re: svn:externals using git submodules Message-ID: <200705011936.14345.andyparkins@gmail.com> URL: https://gitlist.dev/e/200705011936.14345.andyparkins%40gmail.com In-Reply-To: <20070501153626.GA21182@pe.Belkin> ``` On Tuesday 2007, May 01, Chris Shoemaker wrote: > > Actually that is an interesting point that Chris makes. Isn't the > > svn:externals property revision controlled on the parent directory? > > So each change to it is actually recorded in the revision history > > of the parent project. > > Yes and yes. Yes and no. Think of svn:externals as a file in the parent repository; it contains directory-name URL Now, changes to that file _are_ tracked, in that if I changed the URL that change would be recorded in the parent repository. However, nowhere is the revision of the external recorded. Subversion always fetches the latest revision at that URL. > > And if every svn:externals URL included the > > exact version of the other project to include, aren't svn:externals > > then more-or-less like the subproject link support, except they > > also include the URL? > > Just to clarify, my point was just that Andy's setup seems to assume > that the externals don't specify a revision. If they do, maybe They don't. If they did, they'd be just as useful as git's submodules. > git-svn can map the externals into subprojects. Is this what > you're thinking? Well, I'm thinking that that information /can/ be reconstructed from the revision date information - kind of - the problem is that there is no way to know when the parent updated the module. svn:externals really is just a quick way of doing $ cd submodule $ svn update That's it. That's all you get. We could guess that when the parent module was at date YYYY-MM-DD, that the submodule would be at that same date - but who knows? Andy -- Dr Andy Parkins, M Eng (hons), MIET andyparkins@gmail.com ``` ## Andy Parkins, 2007-05-01 18:39 Subject: Re: svn:externals using git submodules Message-ID: <200705011939.17846.andyparkins@gmail.com> URL: https://gitlist.dev/e/200705011939.17846.andyparkins%40gmail.com In-Reply-To: <200705011936.14345.andyparkins@gmail.com> ``` On Tuesday 2007, May 01, Andy Parkins wrote: > Now, changes to that file _are_ tracked, in that if I changed the URL > that change would be recorded in the parent repository. However, > nowhere is the revision of the external recorded. Subversion always > fetches the latest revision at that URL. I meant to add as well that this is absolutely NOT the thing that you want to be tracked. There are any number of times while using externals that I reorganised a directory only to have to change the svn:externals in the parent. That change is then tracked, so if you check out an earlier version not only do you not get a particular revision you also don't get the right URL, so subverion doesn't even fetch the current version. Gah! Andy -- Dr Andy Parkins, M Eng (hons), MIET andyparkins@gmail.com ``` ## Chris Shoemaker, 2007-05-01 19:17 Subject: Re: svn:externals using git submodules Message-ID: <20070501191703.GA25287@pe.Belkin> URL: https://gitlist.dev/e/20070501191703.GA25287%40pe.Belkin In-Reply-To: <200705011936.14345.andyparkins@gmail.com> ``` On Tue, May 01, 2007 at 07:36:11PM +0100, Andy Parkins wrote: > On Tuesday 2007, May 01, Chris Shoemaker wrote: > > > > Actually that is an interesting point that Chris makes. Isn't the > > > svn:externals property revision controlled on the parent directory? > > > So each change to it is actually recorded in the revision history > > > of the parent project. > > > > Yes and yes. > > Yes and no. Think of svn:externals as a file in the parent repository; > it contains > > directory-name URL > > Now, changes to that file _are_ tracked, in that if I changed the URL > that change would be recorded in the parent repository. However, > nowhere is the revision of the external recorded. Subversion always > fetches the latest revision at that URL. That's only true when the revision is not specified in the external. The repo you track may not do that, but it's not uncommon to do so. And, as I think you're pointing out, it's the only way to get any sort of reliable information about the relationship between the parent and the external. I think it would probably be undesirable for git-svn to attempt to convert "floating" externals into well-versioned submodules, since they're not even well-versioned in the svn repo. However, handling the "locked-down" externals is quite another thing. > > > > And if every svn:externals URL included the > > > exact version of the other project to include, aren't svn:externals > > > then more-or-less like the subproject link support, except they > > > also include the URL? > > > > Just to clarify, my point was just that Andy's setup seems to assume > > that the externals don't specify a revision. If they do, maybe > > They don't. If they did, they'd be just as useful as git's submodules. > > > git-svn can map the externals into subprojects. Is this what > > you're thinking? > > Well, I'm thinking that that information /can/ be reconstructed from the > revision date information - kind of - the problem is that there is no > way to know when the parent updated the module. svn:externals really > is just a quick way of doing > $ cd submodule > $ svn update > That's it. That's all you get. We could guess that when the parent > module was at date YYYY-MM-DD, that the submodule would be at that same > date - but who knows? svn users who want the externals to meaningfully define the version relationship between the parent and the project already have to use externals that specify a revision. -chris ``` ## Andy Parkins, 2007-05-01 19:48 Subject: Re: svn:externals using git submodules Message-ID: <200705012048.04817.andyparkins@gmail.com> URL: https://gitlist.dev/e/200705012048.04817.andyparkins%40gmail.com In-Reply-To: <20070501191703.GA25287@pe.Belkin> ``` On Tuesday 2007, May 01, Chris Shoemaker wrote: > That's only true when the revision is not specified in the external. > The repo you track may not do that, but it's not uncommon to do so. It's been a while since I used subversion, and even longer since I used externals - is that a new feature? I used subversion since before version 1.0, so I often missed new features when they arrived. > And, as I think you're pointing out, it's the only way to get any > sort of reliable information about the relationship between the > parent and the external. Does subversion automatically update that fixed attachment when you update the submodule? I would have found that quite useful back then. > I think it would probably be undesirable for git-svn to attempt to > convert "floating" externals into well-versioned submodules, since > they're not even well-versioned in the svn repo. However, handling > the "locked-down" externals is quite another thing. Absolutely. If the information is available, then git is certainly capable of recording it. It sounds like subversion has a facility I didn't know exist, so I've been bad mouthing it more than I should. Oh well :-) Andy -- Dr Andy Parkins, M Eng (hons), MIET andyparkins@gmail.com ``` ## Chris Shoemaker, 2007-05-01 20:23 Subject: Re: svn:externals using git submodules Message-ID: <20070501202356.GA25531@pe.Belkin> URL: https://gitlist.dev/e/20070501202356.GA25531%40pe.Belkin In-Reply-To: <200705012048.04817.andyparkins@gmail.com> ``` On Tue, May 01, 2007 at 08:48:01PM +0100, Andy Parkins wrote: > On Tuesday 2007, May 01, Chris Shoemaker wrote: > > > That's only true when the revision is not specified in the external. > > The repo you track may not do that, but it's not uncommon to do so. > > It's been a while since I used subversion, and even longer since I used > externals - is that a new feature? I don't know, but I would guess that it's no newer than externals in general, as it's not a particularly special case. > I used subversion since before > version 1.0, so I often missed new features when they arrived. > > > And, as I think you're pointing out, it's the only way to get any > > sort of reliable information about the relationship between the > > parent and the external. > > Does subversion automatically update that fixed attachment when you > update the submodule? I would have found that quite useful back then. No, you have to manage the revision in the svn:external property manually. > > I think it would probably be undesirable for git-svn to attempt to > > convert "floating" externals into well-versioned submodules, since > > they're not even well-versioned in the svn repo. However, handling > > the "locked-down" externals is quite another thing. > > Absolutely. If the information is available, then git is certainly > capable of recording it. It sounds like subversion has a facility I > didn't know exist, so I've been bad mouthing it more than I should. Oh > well :-) > Making git-svn handle svn:externals with specified revisions would be _quite_ useful. There's a special-case of this that I use personally: svn:externals that point to other paths (and other revisions) of the parent repo. I'm curious if people think that teaching git-svn to handle this special case is more or less difficult than handling the general case. -chris ``` ## Linus Torvalds, 2007-05-01 22:19 Subject: Re: svn:externals using git submodules Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.98.0705011512300.3808%40woody.linux-foundation.org In-Reply-To: <20070501202356.GA25531@pe.Belkin> ``` On Tue, 1 May 2007, Chris Shoemaker wrote: > > Making git-svn handle svn:externals with specified revisions would be > _quite_ useful. There's a special-case of this that I use personally: > svn:externals that point to other paths (and other revisions) of the > parent repo. Side note: even _without_ a specified revision, I think it's quite sane to have the rule that a submodule hash of all zeroes is "unversioned". Such a submodule is still _useful_: while the tree itself contains no information (and it SHOULD NOT do so, since the actual location of the external module may not be globally stable or visible!), it would basically act like subversion externals together with the ".gitmodules" file that contains that information. So while the git submodule thing was designed to specify specific revisions, there's nothing that really technically _requires_ it. The exact SHA1 details in the submodule link are going to be up to the higher-level user anyway. (Of course, if you actually have a "all zeroes" gitlink entry, and then have a checked-out git tree at that entry, "git status" and "git diff" would show it as needing update. I think that's _correct_, but if we want to shut it up for the special case of all-zero SHA1, we trivially could). But while I'm encouraged that the whole gitlink thing seems to be working for Andy, and some others are playing with it too, I'm also a bit discouraged by the fact that there hasn't been any noise or work on the porcelain side. I was obviously optimistic and hoping we'd see support in checkout/diff, but I haven't heard anybody talk about actually implementing .gitmodules and the porcelain support that uses them.. Linus ``` ## Junio C Hamano, 2007-05-01 22:37 Subject: Re: svn:externals using git submodules Message-ID: <7vzm4o41bz.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vzm4o41bz.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > On Tue, 1 May 2007, Chris Shoemaker wrote: >> >> Making git-svn handle svn:externals with specified revisions would be >> _quite_ useful. There's a special-case of this that I use personally: >> svn:externals that point to other paths (and other revisions) of the >> parent repo. > > Side note: even _without_ a specified revision, I think it's quite sane to > have the rule that a submodule hash of all zeroes is "unversioned". Yup. > But while I'm encouraged that the whole gitlink thing seems to be working > for Andy, and some others are playing with it too, I'm also a bit > discouraged by the fact that there hasn't been any noise or work on the > porcelain side. I was obviously optimistic and hoping we'd see support in > checkout/diff, but I haven't heard anybody talk about actually > implementing .gitmodules and the porcelain support that uses them.. The thing is, almost all the core git people happen to be busy at the same time at this moment. Johannes has just moved, Shawn and I are deep in day-jobs to the neck, ... Don't worry, it eventually will come. ``` ## Shawn O. Pearce, 2007-05-01 23:16 Subject: Re: svn:externals using git submodules Message-ID: <20070501231628.GJ5942@spearce.org> URL: https://gitlist.dev/e/20070501231628.GJ5942%40spearce.org In-Reply-To: <7vzm4o41bz.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano wrote: > Linus Torvalds writes: > > But while I'm encouraged that the whole gitlink thing seems to be working > > for Andy, and some others are playing with it too, I'm also a bit > > discouraged by the fact that there hasn't been any noise or work on the > > porcelain side. I was obviously optimistic and hoping we'd see support in > > checkout/diff, but I haven't heard anybody talk about actually > > implementing .gitmodules and the porcelain support that uses them.. > > The thing is, almost all the core git people happen to be busy > at the same time at this moment. Johannes has just moved, Shawn > and I are deep in day-jobs to the neck, ... Heh, very true. My short-term focus (this week) is git-gui, then pack v4, and probably while working on pack v4 start at least playing with Linus' gitlink thing and prototyping porcleain over it. The gitlink work interests me, and I want to spend time on it, but as Junio said, I'm overbooked... -- Shawn. ``` ## Michel Jouvin, 2010-01-20 13:58 Subject: Re: svn:externals using git submodules Message-ID: URL: https://gitlist.dev/e/loom.20100120T145348-193%40post.gmane.org In-Reply-To: <200705011121.17172.andyparkins@gmail.com> ``` Andy Parkins gmail.com> writes: > > Hello, > > I've done this by hand as a proof of concept I suspect it would need loads of > work in git-svn to do it properly. However, I thought I'd mention as part of > my "success with submodules" reports. > > ... > Now we want that clone to be even more strongly linked to the parent - to the > extent that they share the same refs, etc: > > $ cd libswscale > $ rm -rf .git/refs .git/logs .git/info description config > $ ln -s ../../.git/refs .git/refs > $ ln -s ../../.git/logs .git/logs > $ ln -s ../../.git/info .git/info > $ ln -s ../../.git/config .git/config > $ ln -s ../../.git/description .git/description > > ... > Andy I don't know if it's a good idea to follow-up on such an old entry... but I used your trick, Andy, and it works pretty well. With one exception: I am using Git on Windows and there is no symlinks on this OS. That means that you need to basically "recreate" the symlinks everytime you want to update the submodules. Not very handy but acceptable as they are not updated very often. I have to check the list if there has been some new ways to achieve this. But I'm clearly in the case of svn:externals not tighten to a particular revision, which seems the difficult case for Git. Cheers, Michel ```