Re: [RFC] New type of remote helpers
- From
Sverre Rabbelier <srabbelier@gmail.com>
- Date
- Oct 3, 2010, 13:56 UTC
- Message-ID
- <AANLkTikQyVLyH-O-OH2yZ0B3_UKDqzcnNgtqefSCN68t@mail.gmail.com>
- In-Reply-To
- <4CA86A12.6080905@dbservice.com>
Heya,
On Sun, Oct 3, 2010 at 13:33, Tomas Carnecky <tom@dbservice.com> wrote:
> My work has the goal of making interaction with foreign SCMs more > natural. The work that was done on remote helpers is the right > direction. But the 'import' and 'export' commands are the wrong approach > I think.
I'm not convinced that they are, but we'll see.
> The problem I have with 'import' is that updating the refs is > left up to the remote helper (or git-fast-import). So you lose the nice > output from ls-remote/fetch: non-ff and other warnings etc.
This is a good point, and I like the patches addressing this.
Show 5 quoted lines
> I slightly > modified how the remote helpers (and fast-import) work, now they behave > exactly like 'core' git when fetching: Git tells the remote helper to > fetch some refs, the helper does that and creates a pack and git then > updates the refs (or not, depending on fast-forward etc).
Again, these patches I like, and would like to see them included. I'll probably pick them up and send them out as part of my next git-remote-hg reroll if nothing happens with them.
> To test this > approach I created a simple remote helper for svn.
I guess it suffices as a POC, but I'd have preferred to see collaboration with the people working on git-remote-svn instead (cc-ed).
Show 8 quoted lines
> $ git ls-remote svn::/Volumes/Dump/Source/Mirror/Transmission/ > r1017 (impure) trunk > r919 (impure) branches/nat-traversal > r480 (impure) branches/0.6 > > Git learned to understand version numbers from foreign SCMs. Git > displays those as 'impure' because it knows that version exists but does > not know yet which git commit that version maps to.
Very interesting. This is a useful feature, I approve.
Show 6 quoted lines
> $ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/ > *:refs/remotes/svn/* > From svn::/Volumes/Dump/Source/Mirror/Transmission > * [new branch] trunk -> svn/trunk > * [new branch] branches/nat-traversal -> svn/branches/nat-traversal > * [new branch] branches/0.6 -> svn/branches/0.6
Interesting, if you do 'git remote add svn svn::/Volumes/Dump/Source/Mirror/Transmission/' and then do 'git fetch svn', do you get (more or less) the same output?
Show 6 quoted lines
> Git tells the remote helper to 'fetch r1017 trunk'. The remote helper > does that, creates the pack and then tells git that it imported r1017 as > commit c5fed7ec. This is done with a new reply to the 'fetch' command: > 'map r1017 c5fed7ec'. The remote helper can use that to inform core git > as which git commit the impure ref was imported. Git can then update the > refs. At no point does the remote helper manipulate refs directly.
I love this. Very elegant.
> The pack is created by a heavily modified git-fast-import. The existing > fast-import not only creates the pack but also updates the refs. This is > no longer desired as git is in charge of updating the refs.
NAK. I object against forking git fast-import just for this purpose. I'd much rather just modify git fast-import to learn to learn not update refs, which should be easy enough.
Show 5 quoted lines
> My modified > fast-import works like this: After creating a commit, it writes it's git > object name to stdout. That way the remote helper can figure out as > which git commits the svn revisions were imported and relay that back to > core git using the above described 'map' reply.
Work has been underway to teach git fast-import to do just this (courtesy of Jonathan), no need to fork git fast-import to achieve that.
Show 9 quoted lines
> $ git show --show-notes=svn svn/trunk > commit c5fed7ecc318363523d3ea2045e1c16a378bb10c > Author: livings124 <livings124@localhost> > Date: Wed Oct 18 13:57:19 2006 +0000 > > more traditional toolbar icons for those afraid of change > > Notes (svn): > b4697c4a-7d4c-4a30-bd92-6745580d73b3/trunk@1017
Very nice! I'm definitely stealing the note-generating code for git-remote-hg.
> The svn helper needs to be able to map svn revisions to git commits. > git-svn does this by adding the 'git-svn-id' line to each commit > message. I'm using git notes for that and it seems to work just fine. > The note contains the repo UUID, path within the repo and revision.
Very elegant.
Show 8 quoted lines
> There was a challenge how to update the notes ref (refs/notes/svn). As > with fast-import, I did not want the remote helper to do it. Neither the > remote helper nor fast-import should be writing any refs. But core git > can only update refs which were discovered during transport->fetch(). I > modified the remote helper 'fetch' command and the transport->fetch() > function to return an optional list of refs. These are the refs that the > remote helper wants to update but which should not be presented to the > user (because these are internally used refs, such as my svn notes).
Nice.
Show 8 quoted lines
> So the whole session between git and my svn remote helper looks like this: >> list > < :r1017 trunk >> fetch :r1017 trunk > [helper creates the pack including history up to r1017 and associated > svn notes] > < map r1017 <commit corresponding to r1017> > < silent refs/notes/svn <new commit which stores the updated svn notes>
Looks good.
Show 12 quoted lines
> $ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/ > *:refs/remotes/svn/* > From svn::/Volumes/Dump/Source/Mirror/Transmission > c5fed7e..228eaf3 trunk -> svn/trunk > > $ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/ > *:refs/remotes/svn/* > From svn::/Volumes/Dump/Source/Mirror/Transmission > 228eaf3..207e5e5 trunk -> svn/trunk > * [new branch] branches/scrape -> svn/branches/scrape > * [new branch] branches/multitracker -> svn/branches/multitracker > * [new branch] branches/io -> svn/branches/io
Note to other reviewers: the repository was updated in between successive calls to 'git fetch', see below.
Show 5 quoted lines
> Updating the svn branches works like expected. The remote helper > automatically detects which branches it already imported (by going > through all refs and the attached svn notes) and creates a new pack with > the new commits. New branches are also detected. The svn notes are > updated accordingly.
I assume this only works for regular svn repositories? I guess it doesn't really matter to the rest of the series, since the 'git-remote-svn' helper is more of a POC I think?
Most of your extensions to the helper protocol make sense. However, after re-reading your series I think we _should_ keep the 'import' and 'export' command, so that helpers don't have to invoke 'git fast-import' or 'git fast-import' themselves. I suspect it will be more efficient than your approach as well. Speed _is_ a very important concern, to have decent support for foreign remotes, imports/exports should be as fast as possible. Also, this series doesn't address pushing back to the foreign scm, which is very convenient through the 'export' command.
Either way, thank you very much for working on this!
-- Cheers, Sverre Rabbelier