threads / discuss / 9066

Another question about importing SVN with fast-import

Subject: Another question about importing SVN with fast-import

## tl;dr

11 messages between Jul 16, 2007 and Aug 7, 2007.

replies: 10people: 4as markdown or json

Julian Phillips· Jul 16, 2007, 21:11 UTC · lore

First off, I would just like to say fast-import rocks. It's well named too ...

Now the question. Shawn recently added C and R operations - almost as soon as they were asked for too. However, how do you copy a file from a particular revision? I have just hit a point where someone deleted a directory, and then copied one of the files from that directory back from an old revision (as two separate commits). Since I'm not tracking any branch contents in my front-end, and the copy operation only works from the current branch head I seem to be stuck ... or have I missed something?

-- 
Julian

  ---
What I want to find out is -- do parrots know much about Astro-Turf?
Shawn O. Pearce· Jul 17, 2007, 03:38 UTC · re: Julian Phillips · lore

Re: Another question about importing SVN with fast-import

Julian Phillips <julian@quantumfyre.co.uk> wrote:
> First off, I would just like to say fast-import rocks.  It's well named 
> too ...
I'm glad you like it.  ;-)
 
> Now the question.  Shawn recently added C and R operations - almost as 
> soon as they were asked for too.
Sometimes I'm responsive...
> However, how do you copy a file from a 
> particular revision?

and sometimes not so much. You can't do that right now. I've wanted to open up the data subcommand to allow another form that lets you specify data from a branch and file path (thus selecting a blob from another revision) but I haven't gotten around to it. I also don't have time to do it during the earlier part of this week. Maybe I'll get to it later near the end of the week.

It shouldn't be too difficult now with the tree_content_get() function that I recently defined for the C/R commands. The SHA-1 comes back in the tree_entry leaf, but that codepath is only valid for the tip of a branch that fast-import knows about in memory. If it doesn't then you probably need to fallback into raw tree parsing. Ugh.

The way prior frontends have handled this is they assigned marks to every blob, and then had a translation table within the frontend of revision->mark, so that anytime it needed a given revision of a file it knew what mark to send to fast-import. This does require that the frontend maintain basically everything...

Show 5 quoted lines
> I have just hit a point where someone deleted a 
> directory, and then copied one of the files from that directory back from 
> an old revision (as two separate commits).  Since I'm not tracking any 
> branch contents in my front-end, and the copy operation only works from 
> the current branch head I seem to be stuck ... or have I missed something?

Yea, I got nothing. I can't think of any method to make that work aside from using marks. Or teaching fast-import how to do tree lookup, like I described above.

-- 
Shawn.
Julian Phillips· Aug 6, 2007, 13:13 UTC · re: Shawn O. Pearce · lore

Re: Another question about importing SVN with fast-import

On Mon, 16 Jul 2007, Shawn O. Pearce wrote:
Show 10 quoted lines
> Julian Phillips <julian@quantumfyre.co.uk> wrote:
>> However, how do you copy a file from a
>> particular revision?
>
> and sometimes not so much.  You can't do that right now.  I've wanted
> to open up the data subcommand to allow another form that lets you
> specify data from a branch and file path (thus selecting a blob
> from another revision) but I haven't gotten around to it.  I also
> don't have time to do it during the earlier part of this week.
> Maybe I'll get to it later near the end of the week.

I was wondering if this was still a vague todo item, or if it was being worked on? It really is the biggest thing blocking my import at the moment and I'm willing to have a bash at it myself, but I didn't want to start messing around inside the fast-import code if someone else was already there ...

Show 13 quoted lines
>
> It shouldn't be too difficult now with the tree_content_get()
> function that I recently defined for the C/R commands.  The SHA-1
> comes back in the tree_entry leaf, but that codepath is only valid
> for the tip of a branch that fast-import knows about in memory.
> If it doesn't then you probably need to fallback into raw tree
> parsing.  Ugh.
>
> The way prior frontends have handled this is they assigned marks
> to every blob, and then had a translation table within the frontend
> of revision->mark, so that anytime it needed a given revision of a
> file it knew what mark to send to fast-import.  This does require
> that the frontend maintain basically everything...

Not sure how that would apply to trees ... and I would rather minimise the amount of state retained in the frontend.

-- 
Julian

  ---
Disco is to music what Etch-A-Sketch is to art.
Shawn O. Pearce· Aug 7, 2007, 01:08 UTC · re: Julian Phillips · lore

Re: Another question about importing SVN with fast-import

Julian Phillips <julian@quantumfyre.co.uk> wrote:
Show 18 quoted lines
> On Mon, 16 Jul 2007, Shawn O. Pearce wrote:
> 
> >Julian Phillips <julian@quantumfyre.co.uk> wrote:
> >>However, how do you copy a file from a
> >>particular revision?
> >
> >and sometimes not so much.  You can't do that right now.  I've wanted
> >to open up the data subcommand to allow another form that lets you
> >specify data from a branch and file path (thus selecting a blob
> >from another revision) but I haven't gotten around to it.  I also
> >don't have time to do it during the earlier part of this week.
> >Maybe I'll get to it later near the end of the week.
> 
> I was wondering if this was still a vague todo item, or if it was being 
> worked on?  It really is the biggest thing blocking my import at the 
> moment and I'm willing to have a bash at it myself, but I didn't want to 
> start messing around inside the fast-import code if someone else was 
> already there ...
No, its not vague.  I'm still working on it.  Actually its on
my to-do list for tonight, and if I don't finish it tonight then
Thursday.
 
-- 
Shawn.
David Frech· Jul 19, 2007, 07:09 UTC · re: Julian Phillips · lore

Re: Another question about importing SVN with fast-import

On 7/16/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
Show 7 quoted lines
> Now the question.  Shawn recently added C and R operations - almost as
> soon as they were asked for too.  However, how do you copy a file from a
> particular revision?  I have just hit a point where someone deleted a
> directory, and then copied one of the files from that directory back from
> an old revision (as two separate commits).  Since I'm not tracking any
> branch contents in my front-end, and the copy operation only works from
> the current branch head I seem to be stuck ... or have I missed something?
I have to second Julian's question.

The only things I have left to implement (in my svn dump to fast-import translator) before I can call my code "complete" are branch copies (which will generate "from" lines in commits so fast-import can properly initialize the new branch) and a generic "copy <path> from <rev>" that Julian also needs.

I mark my commits and keep track of the mapping from svn revs to git commits, so when the svn dump asks for a <path> from a specific <svn rev> I can correlate that to a mark (ie, a git commit). But I can't tell fast-import to *do* anything with it.

This is kind of a request and a comment/question. The request is: there is no way to do *reasonably* in the front end what fast-import can do somewhat reasonably: namely, copy a <path> (file or directory!) from an arbitrary previously committed revision/mark to the current branch.

The comment/question is: how different is this, really, from being able to specify a "from" line in a commit? In both cases I'm asking fast-import to reach into its memory (or the repo) and pull out a tree, and to add (some or all of it) to my current branch. Isn't the kind of generic C command that Julian and I are asking for the same thing, only instead of taking the whole tree (from the specified commit) it takes a single file or directory?

I hope I haven't missed the point entirely.

Lastly, do we really need "R"? With this generic copy - and I think there should be *only* a generic version, not a "streamlined local copy" version and a "reach into history arbitrarily" version - we can, as an earlier poster pointed out, do R by doing a C and then a D. This is, in fact, how svn dump files represent file and directory renames.

It would be nice to keep the fast-import command set small and orthogonal.
My few cents.
Cheers,
- David
-- 
If I have not seen farther, it is because I have stood in the
footsteps of giants.
Shawn O. Pearce· Jul 20, 2007, 05:11 UTC · re: David Frech · lore

Re: Another question about importing SVN with fast-import

David Frech <david@nimblemachines.com> wrote:
Show 6 quoted lines
> On 7/16/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
> >Now the question.  Shawn recently added C and R operations - almost as
> >soon as they were asked for too.  However, how do you copy a file from a
> >particular revision?
> 
> I have to second Julian's question.
I'm getting to it.  Maybe this weekend.  Right now I have git-gui
work to finish, and that work is more important to me this week than
fast-import is.  Not that fast-import isn't important to me either,
its just the way things are this week.
 
Show 5 quoted lines
> This is kind of a request and a comment/question. The request is:
> there is no way to do *reasonably* in the front end what fast-import
> can do somewhat reasonably: namely, copy a <path> (file or directory!)
> from an arbitrary previously committed revision/mark to the current
> branch.
I agree.
 
Show 7 quoted lines
> The comment/question is: how different is this, really, from being
> able to  specify a "from" line in a commit? In both cases I'm asking
> fast-import to reach into its memory (or the repo) and pull out a
> tree, and to add (some or all of it) to my current branch. Isn't the
> kind of generic C command that Julian and I are asking for the same
> thing, only instead of taking the whole tree (from the specified
> commit) it takes a single file or directory?

It is possible. I'm just not sure what the syntax for it should be. Suggestions? I really want to stay backwards compatible with the current "C" command, so:

	'C' SP commit SP path SP path

is out because its ambiguous with the current meaning where the second (destination) path can contain SP without being quoted by the frontend.

Show 5 quoted lines
> Lastly, do we really need "R"? With this generic copy - and I think
> there should be *only* a generic version, not a "streamlined local
> copy" version and a "reach into history arbitrarily" version - we can,
> as an earlier poster pointed out, do R by doing a C and then a D. This
> is, in fact, how svn dump files represent file and directory renames.
The code for "R" is so short that I just don't see a need to remove it.
Its also already out in the wild, as it has been in Junio's master for
a little while now.
 
-- 
Shawn.
Julian Phillips· Jul 20, 2007, 05:50 UTC · re: Shawn O. Pearce · lore

Re: Another question about importing SVN with fast-import

On Fri, 20 Jul 2007, Shawn O. Pearce wrote:
Show 12 quoted lines
> David Frech <david@nimblemachines.com> wrote:
>> On 7/16/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
>>> Now the question.  Shawn recently added C and R operations - almost as
>>> soon as they were asked for too.  However, how do you copy a file from a
>>> particular revision?
>>
>> I have to second Julian's question.
>
> I'm getting to it.  Maybe this weekend.  Right now I have git-gui
> work to finish, and that work is more important to me this week than
> fast-import is.  Not that fast-import isn't important to me either,
> its just the way things are this week.

No complaints from me - git-gui is one of the reasons I want to be able to use fast-import ;)

Show 17 quoted lines
>> The comment/question is: how different is this, really, from being
>> able to  specify a "from" line in a commit? In both cases I'm asking
>> fast-import to reach into its memory (or the repo) and pull out a
>> tree, and to add (some or all of it) to my current branch. Isn't the
>> kind of generic C command that Julian and I are asking for the same
>> thing, only instead of taking the whole tree (from the specified
>> commit) it takes a single file or directory?
>
> It is possible.  I'm just not sure what the syntax for it should be.
> Suggestions?  I really want to stay backwards compatible with the
> current "C" command, so:
>
> 	'C' SP commit SP path SP path
>
> is out because its ambiguous with the current meaning where the
> second (destination) path can contain SP without being quoted by
> the frontend.
You could always make it part of the 'M' command?

'M' sp mode sp 'copy' sp path_str lf (ref_str | hexsha1 | sha1exp_str | idnum) SP path_str;

Or just make it a new command, O (for other) or E (for existing) maybe? :S
-- 
Julian

  ---
It is easier to change the specification to fit the program than vice versa.
David Frech· Jul 20, 2007, 06:54 UTC · re: Julian Phillips · lore

Re: Another question about importing SVN with fast-import

On 7/19/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
Show 18 quoted lines
> On Fri, 20 Jul 2007, Shawn O. Pearce wrote:
>
> > It is possible.  I'm just not sure what the syntax for it should be.
> > Suggestions?  I really want to stay backwards compatible with the
> > current "C" command, so:
> >
> >       'C' SP commit SP path SP path
> >
> > is out because its ambiguous with the current meaning where the
> > second (destination) path can contain SP without being quoted by
> > the frontend.
>
> You could always make it part of the 'M' command?
>
> 'M' sp mode sp 'copy' sp path_str lf (ref_str | hexsha1 | sha1exp_str |
> idnum) SP path_str;
>
> Or just make it a new command, O (for other) or E (for existing) maybe? :S

Since we'll be referring to past commits via marks (with start with ":") how about this:

'C' SP srcpath:mark SP dstpath

In the case of making it a new command I can't think of any really compelling one letter names. ;-)

- David
Show 7 quoted lines
>
> --
> Julian
>
>   ---
> It is easier to change the specification to fit the program than vice versa.
>
-- 
If I have not seen farther, it is because I have stood in the
footsteps of giants.
Julian Phillips· Jul 20, 2007, 07:01 UTC · re: David Frech · lore

Re: Another question about importing SVN with fast-import

On Thu, 19 Jul 2007, David Frech wrote:
Show 24 quoted lines
> On 7/19/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
>>  On Fri, 20 Jul 2007, Shawn O. Pearce wrote:
>> 
>> >  It is possible.  I'm just not sure what the syntax for it should be.
>> >  Suggestions?  I really want to stay backwards compatible with the
>> >  current "C" command, so:
>> > 
>> >        'C' SP commit SP path SP path
>> > 
>> >  is out because its ambiguous with the current meaning where the
>> >  second (destination) path can contain SP without being quoted by
>> >  the frontend.
>>
>>  You could always make it part of the 'M' command?
>>
>>  'M' sp mode sp 'copy' sp path_str lf (ref_str | hexsha1 | sha1exp_str |
>>  idnum) SP path_str;
>>
>>  Or just make it a new command, O (for other) or E (for existing) maybe? :S
>
> Since we'll be referring to past commits via marks (with start with
> ":") how about this:
>
> 'C' SP srcpath:mark SP dstpath

Why only via marks? That may be what you happen to want, but no other command restricts you to _only_ using marks ...

Show 16 quoted lines
>
> In the case of making it a new command I can't think of any really
> compelling one letter names. ;-)
>
> - David
>
>>
>>  --
>>  Julian
>>
>>    ---
>>  It is easier to change the specification to fit the program than vice
>>  versa.
>> 
>
>
-- 
Julian

  ---
What say you, Mr. American Ambassador?

Fuck Canada!
David Frech· Jul 20, 2007, 07:16 UTC · re: Julian Phillips · lore

Re: Another question about importing SVN with fast-import

On 7/20/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
Show 8 quoted lines
> On Thu, 19 Jul 2007, David Frech wrote:
> > Since we'll be referring to past commits via marks (with start with
> > ":") how about this:
> >
> > 'C' SP srcpath:mark SP dstpath
>
> Why only via marks?  That may be what you happen to want, but no other
> command restricts you to _only_ using marks ...
I was using the royal "we". ;-)
Yes, you're right of course. My bad.
- David
Show 8 quoted lines
> --
> Julian
>
>   ---
> What say you, Mr. American Ambassador?
>
> Fuck Canada!
>
-- 
If I have not seen farther, it is because I have stood in the
footsteps of giants.
Jan Hudec· Jul 23, 2007, 18:06 UTC · re: Shawn O. Pearce · lore

Re: Another question about importing SVN with fast-import

On Fri, Jul 20, 2007 at 01:11:42 -0400, Shawn O. Pearce wrote:
Show 9 quoted lines
> It is possible.  I'm just not sure what the syntax for it should be.
> Suggestions?  I really want to stay backwards compatible with the
> current "C" command, so:
> 
> 	'C' SP commit SP path SP path
> 
> is out because its ambiguous with the current meaning where the
> second (destination) path can contain SP without being quoted by
> the frontend.
I'd suggest one of two variants:
 1) 'M' SP <mode> SP <dataref> SP <path> LF
    where <dataref> would be extended to understand the
    {tag-id|commit-id|tree-id}:path notation git-ref-parse understands plus
    mark:path where mark points to commit.
 2) 'C' SP <dataref> SP <path> LF
    where again <dataref> can be {tag-id|commit-id|tree-id|mark}:path -- or
    just path which implies current head.
-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>

← back to recent threads