RFD: fast-import is picky with author names (and maybe it should - but how much so?)

> It seems that our fast-import is super picky with regards to author
> names. I've encountered author names like
> 
> Foo Bar<foo.bar@dev.null>
> Foo Bar <foo.bar@dev.null
> foo.bar@dev.null
> 
> in the self-hosting repo of some other dvcs, and the question is how to
> translate them faithfully into a git author name.

> In general, we try to do
> 
> fullotherdvcsname <none@none>
> 
> if the other system's entry does not parse as a git author name, but
> fast-import does not accept either of
> 
> Foo Bar<foo.bar@dev.null> <none@none>
> "Foo Bar<foo.bar@dev.null>" <none@none>
> 
> because of the way it parses for <>. While the above could be easily
> turned into
> 
> Foo Bar <foo.bar@dev.null>
> 
> it would not be a faithful representation of the original commit in the
> other dvcs.

> On Fri, Nov 02, 2012 at 03:43:24PM +0100, Michael J Gruber wrote:
> 
>> It seems that our fast-import is super picky with regards to author
>> names. I've encountered author names like
>>
>> Foo Bar<foo.bar@dev.null>
>> Foo Bar <foo.bar@dev.null
>> foo.bar@dev.null
>>
>> in the self-hosting repo of some other dvcs, and the question is how to
>> translate them faithfully into a git author name.
> 
> It is not just fast-import. Git's author field looks like an rfc822
> address, but it's much simpler. It fundamentally does not allow angle
> brackets in the "name" field, regardless of any quoting. As you noted in
> your followup, we strip them out if you provide them via
> GIT_AUTHOR_NAME.
> 
> I doubt this will change anytime soon due to the compatibility fallout.
> So it is up to generators of fast-import streams to decide how to encode
> what they get from another system (you could come up with an encoding
> scheme that represents angle brackets).

>> In general, we try to do
>>
>> fullotherdvcsname <none@none>
>>
>> if the other system's entry does not parse as a git author name, but
>> fast-import does not accept either of
>>
>> Foo Bar<foo.bar@dev.null> <none@none>
>> "Foo Bar<foo.bar@dev.null>" <none@none>
>>
>> because of the way it parses for <>. While the above could be easily
>> turned into
>>
>> Foo Bar <foo.bar@dev.null>
>>
>> it would not be a faithful representation of the original commit in the
>> other dvcs.
> 
> I'd think that if a remote system has names with angle brackets and
> email-looking things inside them, we would do better to stick them in
> the email field rather than putting in a useless <none@none>. The latter
> should only be used for systems that lack the information.
> 
> But that is a quality-of-implementation issue for the import scripts
> (and they may even want to have options, just like git-cvsimport allows
> mapping cvs usernames into full identities).

> Hg seems to store just anything in the author field ("committer"). The
> various interfaces that are floating around do some behind-the-back
> conversion to git format. The more conversions they do, the better they
> seem to work (no erroring out) but I'm wondering whether it's really a
> good thing, or whether we should encourage a more diligent approach
> which requires a user to map non-conforming author names wilfully.

> On Fri, Nov 9, 2012 at 10:28 AM, Michael J Gruber
> <git@drmicha.warpmail.net> wrote:
> 
>> Hg seems to store just anything in the author field ("committer"). The
>> various interfaces that are floating around do some behind-the-back
>> conversion to git format. The more conversions they do, the better they
>> seem to work (no erroring out) but I'm wondering whether it's really a
>> good thing, or whether we should encourage a more diligent approach
>> which requires a user to map non-conforming author names wilfully.
> 
> So you propose that when somebody does 'git clone hg::hg hg-git' the
> thing should fail. I hope you don't think it's too unbecoming for me
> to say that I disagree.

> IMO it should be git fast-import the one that converts these bad
> authors, not every single tool out there. Maybe throw a warning, but
> that's all. Or maybe generate a list of bad authors ready to be filled
> out. That way when a project is doing a real conversion, say, when
> moving to git, they can run the conversion once and see which authors
> are bad and not multiple times, each try taking longer than the next.

> Felipe Contreras venit, vidit, dixit 09.11.2012 15:34:
>> On Fri, Nov 9, 2012 at 10:28 AM, Michael J Gruber
>> <git@drmicha.warpmail.net> wrote:
>>
>>> Hg seems to store just anything in the author field ("committer"). The
>>> various interfaces that are floating around do some behind-the-back
>>> conversion to git format. The more conversions they do, the better they
>>> seem to work (no erroring out) but I'm wondering whether it's really a
>>> good thing, or whether we should encourage a more diligent approach
>>> which requires a user to map non-conforming author names wilfully.
>>
>> So you propose that when somebody does 'git clone hg::hg hg-git' the
>> thing should fail. I hope you don't think it's too unbecoming for me
>> to say that I disagree.
>
> There is no need to disagree with a proposal I haven't made. I would
> disagree with the proposal that I haven't made, too.

>> IMO it should be git fast-import the one that converts these bad
>> authors, not every single tool out there. Maybe throw a warning, but
>> that's all. Or maybe generate a list of bad authors ready to be filled
>> out. That way when a project is doing a real conversion, say, when
>> moving to git, they can run the conversion once and see which authors
>> are bad and not multiple times, each try taking longer than the next.
>
> As Jeff pointed out, git-fast-import expects output conforming to a
> certain standard, and that's not going to change. import is agnostic to
> where its import stream is coming from. Only the producer of that stream
> can have additional information about the provenience of the stream's
> data which may aid (possibly together with user input or choices) in
> transforming that into something conforming.

> On Sat, Nov 10, 2012 at 6:28 PM, Michael J Gruber
> <git@drmicha.warpmail.net>  wrote:
>> Felipe Contreras venit, vidit, dixit 09.11.2012 15:34:
>>> On Fri, Nov 9, 2012 at 10:28 AM, Michael J Gruber
>>> <git@drmicha.warpmail.net>  wrote:
>>>
>>>> Hg seems to store just anything in the author field ("committer"). The
>>>> various interfaces that are floating around do some behind-the-back
>>>> conversion to git format. The more conversions they do, the better they
>>>> seem to work (no erroring out) but I'm wondering whether it's really a
>>>> good thing, or whether we should encourage a more diligent approach
>>>> which requires a user to map non-conforming author names wilfully.
>>>
>>> So you propose that when somebody does 'git clone hg::hg hg-git' the
>>> thing should fail. I hope you don't think it's too unbecoming for me
>>> to say that I disagree.
>>
>> There is no need to disagree with a proposal I haven't made. I would
>> disagree with the proposal that I haven't made, too.
>
> All right, we shouldn't encourage a more diligent approach which
> requires a user to map author names then.
>
>>> IMO it should be git fast-import the one that converts these bad
>>> authors, not every single tool out there. Maybe throw a warning, but
>>> that's all. Or maybe generate a list of bad authors ready to be filled
>>> out. That way when a project is doing a real conversion, say, when
>>> moving to git, they can run the conversion once and see which authors
>>> are bad and not multiple times, each try taking longer than the next.
>>
>> As Jeff pointed out, git-fast-import expects output conforming to a
>> certain standard, and that's not going to change. import is agnostic to
>> where its import stream is coming from. Only the producer of that stream
>> can have additional information about the provenience of the stream's
>> data which may aid (possibly together with user input or choices) in
>> transforming that into something conforming.
>
> We already know where the import of those streams come from:
> mercurial, bazaar, etc. There's absolutely nothing the tools exporting
> data from those repositories can do, except try to convert all kind of
> weird names--and many tools do it poorly.
>
> So, the options are:
>
> a) Leave the name conversion to the export tools, and when they miss
> some weird corner case, like 'Author<email', let the user face the
> consequences, perhaps after an hour of the process.
>
> We know there are sources of data that don't have git-formatted author
> names, so we know every tool out there must do this checking.
>
> In addition to that, let the export tool decide what to do when one of
> these bad names appear, which in many cases probably means do nothing,
> so the user would not even see that such a bad name was there, which
> might not be what they want.
>
> b) Do the name conversion in fast-import itself, perhaps optionally,
> so if a tool missed some weird corner case, the user does not have to
> face the consequences.
>
> The tool writers don't have to worry about this, so we would not have
> tools out there doing a half-assed job of this.
>
> And what happens when such bad names end up being consistent: warning,
> a scaffold mapping of bad names, etc.
>
>
> One is bad for the users, and the tools writers, only disadvantages,
> the other is good for the users and the tools writers, only
> advantages.
>

>> So, the options are:
>>
>> a) Leave the name conversion to the export tools, and when they miss
>> some weird corner case, like 'Author<email', let the user face the
>> consequences, perhaps after an hour of the process.
>>
>> We know there are sources of data that don't have git-formatted author
>> names, so we know every tool out there must do this checking.
>>
>> In addition to that, let the export tool decide what to do when one of
>> these bad names appear, which in many cases probably means do nothing,
>> so the user would not even see that such a bad name was there, which
>> might not be what they want.
>>
>> b) Do the name conversion in fast-import itself, perhaps optionally,
>> so if a tool missed some weird corner case, the user does not have to
>> face the consequences.
>>
>> The tool writers don't have to worry about this, so we would not have
>> tools out there doing a half-assed job of this.
>>
>> And what happens when such bad names end up being consistent: warning,
>> a scaffold mapping of bad names, etc.
>>
>>
>> One is bad for the users, and the tools writers, only disadvantages,
>> the other is good for the users and the tools writers, only
>> advantages.
>>
>
> c) Do the name conversion, and whatever other cleanup and manipulations
> you're interesting in, in a filter between the exporter and git-fast-import.

> On Sat, Nov 10, 2012 at 8:25 PM, A Large Angry SCM<gitzilla@gmail.com>  wrote:
>> On 11/10/2012 01:43 PM, Felipe Contreras wrote:
>
>>> So, the options are:
>>>
>>> a) Leave the name conversion to the export tools, and when they miss
>>> some weird corner case, like 'Author<email', let the user face the
>>> consequences, perhaps after an hour of the process.
>>>
>>> We know there are sources of data that don't have git-formatted author
>>> names, so we know every tool out there must do this checking.
>>>
>>> In addition to that, let the export tool decide what to do when one of
>>> these bad names appear, which in many cases probably means do nothing,
>>> so the user would not even see that such a bad name was there, which
>>> might not be what they want.
>>>
>>> b) Do the name conversion in fast-import itself, perhaps optionally,
>>> so if a tool missed some weird corner case, the user does not have to
>>> face the consequences.
>>>
>>> The tool writers don't have to worry about this, so we would not have
>>> tools out there doing a half-assed job of this.
>>>
>>> And what happens when such bad names end up being consistent: warning,
>>> a scaffold mapping of bad names, etc.
>>>
>>>
>>> One is bad for the users, and the tools writers, only disadvantages,
>>> the other is good for the users and the tools writers, only
>>> advantages.
>>>
>>
>> c) Do the name conversion, and whatever other cleanup and manipulations
>> you're interesting in, in a filter between the exporter and git-fast-import.
>
> Such a filter would probably be quite complicated, and would decrease
> performance.
>

> >>>a) Leave the name conversion to the export tools, and when they miss
> >>>some weird corner case, like 'Author<email', let the user face the
> >>>consequences, perhaps after an hour of the process.
> [...]
> >>>b) Do the name conversion in fast-import itself, perhaps optionally,
> >>>so if a tool missed some weird corner case, the user does not have to
> >>>face the consequences.
> [...]
> >>c) Do the name conversion, and whatever other cleanup and manipulations
> >>you're interesting in, in a filter between the exporter and git-fast-import.
> >
> >Such a filter would probably be quite complicated, and would decrease
> >performance.
> >
> 
> Really?
> 
> The fast import stream protocol is pretty simple. All the filter
> really needs to do is pass through everything that isn't a 'commit'
> command. And for the 'commit' command, it only needs to do something
> with the 'author' and 'committer' lines; passing through everything
> else.
> 
> I agree that an additional filter _may_ decrease performance somewhat
> if you are already CPU constrained. But I suspect that the effect
> would be negligible compared to the all of the SHA-1 calculations.

> If there is a standard filter, then what is the advantage in doing it as
> a pipe? Why not just teach fast-import the same trick (and possibly make
> it optional)? That would be simpler, more efficient, and it would make
> it easier for remote helpers to turn it on (they use a command-line
> switch rather than setting up an extra process).

> So if we receive:
>
>   Foo Bar<foo.bar@example.com> <none@none>
>
> or:
>
>   Foo Bar<foo.bar@example.com <none@none>
>
> or:
>
>   Foo Bar<foo.bar@example.com
>
> what do we do with it? Is the first part a malformed name/email pair,
> and the second part is crap added by a lazy exporter? Or does the
> exporter want to keep the angle brackets as part of the name field? Is
> there a malformed email in the last one, or no email at all?

> > If there is a standard filter, then what is the advantage in doing it as
> > a pipe? Why not just teach fast-import the same trick (and possibly make
> > it optional)? That would be simpler, more efficient, and it would make
> > it easier for remote helpers to turn it on (they use a command-line
> > switch rather than setting up an extra process).
> 
> Right, but instead of a command-line switch it probably should be
> enabled on the stream:
> 
>   feature clean-authors
> 
> Or something.

> > We can clean up and normalize
> > things like whitespace (and we probably should if we do not do so
> > already). But beyond that, we have no context about the name; only the
> > exporter has that.
> 
> There is no context.

> On Sun, Nov 11, 2012 at 06:45:32PM +0100, Felipe Contreras wrote:
>
>> > If there is a standard filter, then what is the advantage in doing it as
>> > a pipe? Why not just teach fast-import the same trick (and possibly make
>> > it optional)? That would be simpler, more efficient, and it would make
>> > it easier for remote helpers to turn it on (they use a command-line
>> > switch rather than setting up an extra process).
>>
>> Right, but instead of a command-line switch it probably should be
>> enabled on the stream:
>>
>>   feature clean-authors
>>
>> Or something.
>
> Yeah, I was thinking it would need a feature switch to the remote helper
> to turn on the command-line, but I forgot that fast-import can take
> feature lines directly.
>
>> > We can clean up and normalize
>> > things like whitespace (and we probably should if we do not do so
>> > already). But beyond that, we have no context about the name; only the
>> > exporter has that.
>>
>> There is no context.
>
> There may not be a lot, but there is some:
>
>> These are exactly the same questions every exporter must answer. And
>> there's no answer, because the field is not a git author, it's a
>> mercurial user, or a bazaar committer, or who knows what.
>
> The exporter knows that the field is a mercurial user (or whatever).
> Fast-import does not even know that, and cannot apply any rules or
> heuristics about the format of a mercurial user string, what is common
> in the mercurial world, etc. It may not be a lot of context in some
> cases (I do not know anything about mercurial's formats, so I can't say
> what knowledge is available). But at least the exporter has a chance at
> domain-specific interpretation of the string. Fast-import has no chance,
> because it does not know the domain.
>
> I've snipped the rest of your argument, which is basically that
> mercurial does not have any context at all, and knowing that it is a
> mercurial author is useless.  I am not sure that is true; even knowing
> that it is a free-form field versus something structured (e.g., we know
> CVS authors are usernames on the server server) is useful.

> But I would agree there are probably multiple systems that are like
> mercurial in that the author field is usually something like "name
> <email>", but may be arbitrary text (I assume bzr is the same way, but
> you would know better than me).  So it may make sense to have some stock
> algorithm to try to convert arbitrary almost-name-and-email text into
> name and email to reduce duplication between exporters, but:

> >   3. Exporters should not use it if they have any broken-down
> >      representation at all. Even knowing that the first half is a human
> >      name and the second half is something else would give it a better
> >      shot at cleaning than fast-import would get.
> 
> I'm not sure what you mean by this. If they have name and email, then
> sure, it's easy.

> On Sun, Nov 11, 2012 at 07:48:14PM +0100, Felipe Contreras wrote:
>
>> >   3. Exporters should not use it if they have any broken-down
>> >      representation at all. Even knowing that the first half is a human
>> >      name and the second half is something else would give it a better
>> >      shot at cleaning than fast-import would get.
>>
>> I'm not sure what you mean by this. If they have name and email, then
>> sure, it's easy.
>
> But not as easy as just printing it. What if you have this:
>
>   name="Peff <angle brackets> King"
>   email="<peff@peff.net>"
>
> Concatenating them does not produce a valid git author name. Sending the
> concatenation through fast-import's cleanup function would lose
> information (namely, the location of the boundary between name and
> email).

> On Mon, Nov 12, 2012 at 10:41 PM, Jeff King <peff@peff.net> wrote:
>> On Sun, Nov 11, 2012 at 07:48:14PM +0100, Felipe Contreras wrote:
>>
>>>>   3. Exporters should not use it if they have any broken-down
>>>>      representation at all. Even knowing that the first half is a human
>>>>      name and the second half is something else would give it a better
>>>>      shot at cleaning than fast-import would get.
>>>
>>> I'm not sure what you mean by this. If they have name and email, then
>>> sure, it's easy.
>>
>> But not as easy as just printing it. What if you have this:
>>
>>   name="Peff <angle brackets> King"
>>   email="<peff@peff.net>"
>>
>> Concatenating them does not produce a valid git author name. Sending the
>> concatenation through fast-import's cleanup function would lose
>> information (namely, the location of the boundary between name and
>> email).
> 
> Right. Unfortunately I'm not aware of any DSCM that does that.
> 
>> Similarly, one might have other structured data (e.g., CVS username)
>> where the structure is a useful hint, but some conversion to name+email
>> is still necessary.
> 
> CVS might be the only one that has such structured data. I think in
> subversion the username has no meaning. A 'felipec' subversion
> username is as bad as a mercurial 'felipec' username.

> Felipe Contreras venit, vidit, dixit 12.11.2012 23:47:
>> On Mon, Nov 12, 2012 at 10:41 PM, Jeff King <peff@peff.net> wrote:
>>> On Sun, Nov 11, 2012 at 07:48:14PM +0100, Felipe Contreras wrote:
>>>
>>>>>   3. Exporters should not use it if they have any broken-down
>>>>>      representation at all. Even knowing that the first half is a human
>>>>>      name and the second half is something else would give it a better
>>>>>      shot at cleaning than fast-import would get.
>>>>
>>>> I'm not sure what you mean by this. If they have name and email, then
>>>> sure, it's easy.
>>>
>>> But not as easy as just printing it. What if you have this:
>>>
>>>   name="Peff <angle brackets> King"
>>>   email="<peff@peff.net>"
>>>
>>> Concatenating them does not produce a valid git author name. Sending the
>>> concatenation through fast-import's cleanup function would lose
>>> information (namely, the location of the boundary between name and
>>> email).
>>
>> Right. Unfortunately I'm not aware of any DSCM that does that.
>>
>>> Similarly, one might have other structured data (e.g., CVS username)
>>> where the structure is a useful hint, but some conversion to name+email
>>> is still necessary.
>>
>> CVS might be the only one that has such structured data. I think in
>> subversion the username has no meaning. A 'felipec' subversion
>> username is as bad as a mercurial 'felipec' username.
>
> In subversion, the username has the clearly defined meaning of being a
> username on the subversion host. If the host is, e.g., a sourceforge
> site then I can easily look up the user profile and convert the username
> into a valid e-mail address (<username>@users.sf.net). That is the
> advantage that the exporter (together with user knowledge) has over the
> importer.
>
> If the initial clone process aborts after every single "unknown" user
> it's no fun, of course. On the other hand, if an incremental clone
> (fetch) let's commits with unknown author sneak in it's no fun either
> (because I may want to fetch in crontab and publish that converted beast
> automatically). That is why I proposed neither approach.
>
> Most conveniently, the export side of a remote helper would
>
> - do "obvious" automatic lossless transformations
> - use an author map for other names

> If the cloning involves a "foreign clone" (like the hg clone behind the
> scene) then the runtime of the second pass should be much smaller. In
> principle, one could even store all blobs and trees on the first run and
> skip that step on the second, but that would rely on immutability on the
> foreign side, so I dunno. (And to check the sha1, we have to get the
> blob anyways.)

> On Sun, Nov 11, 2012 at 12:00:44PM -0500, A Large Angry SCM wrote:
>
>>>>> a) Leave the name conversion to the export tools, and when they miss
>>>>> some weird corner case, like 'Author<email', let the user face the
>>>>> consequences, perhaps after an hour of the process.
>> [...]
>>>>> b) Do the name conversion in fast-import itself, perhaps optionally,
>>>>> so if a tool missed some weird corner case, the user does not have to
>>>>> face the consequences.
>> [...]
>>>> c) Do the name conversion, and whatever other cleanup and manipulations
>>>> you're interesting in, in a filter between the exporter and git-fast-import.
>>>
>>> Such a filter would probably be quite complicated, and would decrease
>>> performance.
>>>
>>
>> Really?
>>
>> The fast import stream protocol is pretty simple. All the filter
>> really needs to do is pass through everything that isn't a 'commit'
>> command. And for the 'commit' command, it only needs to do something
>> with the 'author' and 'committer' lines; passing through everything
>> else.
>>
>> I agree that an additional filter _may_ decrease performance somewhat
>> if you are already CPU constrained. But I suspect that the effect
>> would be negligible compared to the all of the SHA-1 calculations.
>
> It might be measurable, as you are passing every byte of every version
> of every file in the repo through an extra pipe. But more importantly, I
> don't think it helps.
>
> If there is not a standard filter for fixing up names, we do not need to
> care. The user can use "sed" or whatever and pay the performance penalty
> (and deal with the possibility of errors from being lazy about parsing
> the fast-import stream).
>
> If there is a standard filter, then what is the advantage in doing it as
> a pipe? Why not just teach fast-import the same trick (and possibly make
> it optional)? That would be simpler, more efficient, and it would make
> it easier for remote helpers to turn it on (they use a command-line
> switch rather than setting up an extra process).
>
> But what I don't understand is: what would such a standard filter look
> like? Fast-import (or a filter) would already receive the exporter's
> best attempt at a git-like ident string. We can clean up and normalize
> things like whitespace (and we probably should if we do not do so
> already). But beyond that, we have no context about the name; only the
> exporter has that.
>
> So if we receive:
>
>    Foo Bar<foo.bar@example.com>  <none@none>
>
> or:
>
>    Foo Bar<foo.bar@example.com<none@none>
>
> or:
>
>    Foo Bar<foo.bar@example.com
>
> what do we do with it? Is the first part a malformed name/email pair,
> and the second part is crap added by a lazy exporter? Or does the
> exporter want to keep the angle brackets as part of the name field? Is
> there a malformed email in the last one, or no email at all?
>
> The exporter is the only program that actually knows where the data came
> from, how it should be broken down, and what is appropriate for pulling
> data out of its particular source system. For that reason, the exporter
> has to be the place where we come up with a syntactically correct and
> unambiguous ident.
>
> I am not opposed to adding a mailmap-like feature to fast-import to map
> identities, but it has to start with sane, unambiguous output from the
> exporter.

>> Such a filter would probably be quite complicated, and would decrease
>> performance.
>
> Really?
>
> The fast import stream protocol is pretty simple. All the filter really
> needs to do is pass through everything that isn't a 'commit' command. And
> for the 'commit' command, it only needs to do something with the 'author'
> and 'committer' lines; passing through everything else.

> On Sun, Nov 11, 2012 at 6:00 PM, A Large Angry SCM<gitzilla@gmail.com>  wrote:
>> On 11/11/2012 07:41 AM, Felipe Contreras wrote:
>
>>> Such a filter would probably be quite complicated, and would decrease
>>> performance.
>>
>> Really?
>>
>> The fast import stream protocol is pretty simple. All the filter really
>> needs to do is pass through everything that isn't a 'commit' command. And
>> for the 'commit' command, it only needs to do something with the 'author'
>> and 'committer' lines; passing through everything else.
>
> And how do you propose to find the commit commands without parsing all
> the other commands? If you randomly look for lines that begin with
> 'commit /refs' you might end up in the middle of a commit message or
> the contents of a file.

>
>> I agree that an additional filter _may_ decrease performance somewhat if you
>> are already CPU constrained. But I suspect that the effect would be
>> negligible compared to the all of the SHA-1 calculations.
>
> Well. If it's so easy surely you can write one quickly, and I can measure it.

>> And how do you propose to find the commit commands without parsing all
>> the other commands? If you randomly look for lines that begin with
>> 'commit /refs' you might end up in the middle of a commit message or
>> the contents of a file.
>
> I didn't say you didn't have to parse the protocol. I said that the protocol
> is pretty simple.

>>> I agree that an additional filter _may_ decrease performance somewhat if
>>> you
>>> are already CPU constrained. But I suspect that the effect would be
>>> negligible compared to the all of the SHA-1 calculations.
>>
>> Well. If it's so easy surely you can write one quickly, and I can measure
>> it.
>
> Not my itch; You care, you do it.

> On 11/11/2012 07:41 AM, Felipe Contreras wrote:
>> On Sat, Nov 10, 2012 at 8:25 PM, A Large Angry SCM<gitzilla@gmail.com>  wrote:
>>> On 11/10/2012 01:43 PM, Felipe Contreras wrote:
>>
>>>> So, the options are:
>>>>
>>>> a) Leave the name conversion to the export tools, and when they miss
>>>> some weird corner case, like 'Author<email', let the user face the
>>>> consequences, perhaps after an hour of the process.
>>>>
>>>> We know there are sources of data that don't have git-formatted author
>>>> names, so we know every tool out there must do this checking.
>>>>
>>>> In addition to that, let the export tool decide what to do when one of
>>>> these bad names appear, which in many cases probably means do nothing,
>>>> so the user would not even see that such a bad name was there, which
>>>> might not be what they want.
>>>>
>>>> b) Do the name conversion in fast-import itself, perhaps optionally,
>>>> so if a tool missed some weird corner case, the user does not have to
>>>> face the consequences.
>>>>
>>>> The tool writers don't have to worry about this, so we would not have
>>>> tools out there doing a half-assed job of this.
>>>>
>>>> And what happens when such bad names end up being consistent: warning,
>>>> a scaffold mapping of bad names, etc.
>>>>
>>>>
>>>> One is bad for the users, and the tools writers, only disadvantages,
>>>> the other is good for the users and the tools writers, only
>>>> advantages.
>>>>
>>>
>>> c) Do the name conversion, and whatever other cleanup and manipulations
>>> you're interesting in, in a filter between the exporter and git-fast-import.
>>
>> Such a filter would probably be quite complicated, and would decrease
>> performance.
>>
>
> Really?
>
> The fast import stream protocol is pretty simple. All the filter
> really needs to do is pass through everything that isn't a 'commit'
> command. And for the 'commit' command, it only needs to do something
> with the 'author' and 'committer' lines; passing through everything
> else.
>
> I agree that an additional filter _may_ decrease performance somewhat
> if you are already CPU constrained. But I suspect that the effect
> would be negligible compared to the all of the SHA-1 calculations.

> A Large Angry SCM <gitzilla@gmail.com> writes:
>
>> On 11/11/2012 07:41 AM, Felipe Contreras wrote:
>>> On Sat, Nov 10, 2012 at 8:25 PM, A Large Angry SCM<gitzilla@gmail.com>  wrote:
>>>> On 11/10/2012 01:43 PM, Felipe Contreras wrote:
>>>
>>>>> So, the options are:
>>>>>
>>>>> a) Leave the name conversion to the export tools, and when they miss
>>>>> some weird corner case, like 'Author<email', let the user face the
>>>>> consequences, perhaps after an hour of the process.
>>>>>
>>>>> We know there are sources of data that don't have git-formatted author
>>>>> names, so we know every tool out there must do this checking.
>>>>>
>>>>> In addition to that, let the export tool decide what to do when one of
>>>>> these bad names appear, which in many cases probably means do nothing,
>>>>> so the user would not even see that such a bad name was there, which
>>>>> might not be what they want.
>>>>>
>>>>> b) Do the name conversion in fast-import itself, perhaps optionally,
>>>>> so if a tool missed some weird corner case, the user does not have to
>>>>> face the consequences.
>>>>>
>>>>> The tool writers don't have to worry about this, so we would not have
>>>>> tools out there doing a half-assed job of this.
>>>>>
>>>>> And what happens when such bad names end up being consistent: warning,
>>>>> a scaffold mapping of bad names, etc.
>>>>>
>>>>>
>>>>> One is bad for the users, and the tools writers, only disadvantages,
>>>>> the other is good for the users and the tools writers, only
>>>>> advantages.
>>>>>
>>>>
>>>> c) Do the name conversion, and whatever other cleanup and manipulations
>>>> you're interesting in, in a filter between the exporter and git-fast-import.
>>>
>>> Such a filter would probably be quite complicated, and would decrease
>>> performance.
>>>
>>
>> Really?
>>
>> The fast import stream protocol is pretty simple. All the filter
>> really needs to do is pass through everything that isn't a 'commit'
>> command. And for the 'commit' command, it only needs to do something
>> with the 'author' and 'committer' lines; passing through everything
>> else.
>>
>> I agree that an additional filter _may_ decrease performance somewhat
>> if you are already CPU constrained. But I suspect that the effect
>> would be negligible compared to the all of the SHA-1 calculations.
>
> More importantly, which do users prefer: quickly produce an
> incorrect result, or spend some more time to get it right?

> Because the exporting tool has a lot more intimate knowledge about
> how the names are represented in the history of the original SCM,
> canonicalization of the names, if done at that point, would likely
> to give us more useful results, than a canonicalization done at the
> beginning of the importer, which lacks SCM specific details.  So in
> that sense, (a) is more preferrable than (b).

> On the other hand, we would want consistency across the converted
> results no matter what SCM the history was originally in.  E.g. a
> name without email that came from CVS or SVN would consistently want
> to become "name <noname@noname>" or "name <name>" or whatever, and
> letting exporting tools responsible for the canonicalization will
> lead them to create their own garbage.  In that sense, (b) can be
> better than (a).