threads / discuss / 16573

fast-import problem importing dos format files under cygwin

Subject: fast-import problem importing dos format files under cygwin

## tl;dr

7 messages between Dec 3, 2008 and Dec 3, 2008.

replies: 6people: 4as markdown or json

Jan Hudec· Dec 3, 2008, 07:51 UTC · lore

Hello folks,

I have been playing with fast-import in cygwin and I have problems importing files with CR/LF line-endings. The size in data command is calculated including the CRs and than the file is copied binary to the fast-import input stream. However fast-import skips the CRs when reading, overreads by that number of bytes and fails when it tries to read the next command from the middle.

Attached is a test input stream and crash report generated by fast-import when reading it. In case mail system damages it along the way despite being attached to prevent that, the file should be in unix format -- that is what my cygwin perl outputs by default -- and has CRs only on lines 15 and 16. The unix.txt and dos.txt should only differ that the '.'s in former are replaced by '^M's in the later (so the data commands are otherwise same).

Note, that when I convert the file to dos format, it is read as intended. However, that is inconsistent with rest of the cygwin environment which generated and expects files in unix format. I use binary mounts (not converting) and CYGWIN environment variable is empty. My git version is 1.6.0.4 from official Cygwin package.

Is this behaviour intentional workaround for something or a bug?

-- 
                                        - Jan Hudec <bulb@ucw.cz>

Jan Hudec· Dec 3, 2008, 09:10 UTC · re: Jan Hudec · lore

Ad: fast-import problem importing dos format files under cygwin

On 3 December 2008, 08:51, Jan Hudec wrote:

Show 8 quoted lines

> Hello folks,
>
> I have been playing with fast-import in cygwin and I have problems
> importing files with CR/LF line-endings. The size in data command is
> calculated including the CRs and than the file is copied binary to the
> fast-import input stream. However fast-import skips the CRs when reading,
> overreads by that number of bytes and fails when it tries to read the
> next command from the middle.

One addition:

I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the test, as it was, except it didn't like 'refs/heads/master' as branchname (and accepted bare 'master', but that created '.git/master').

-- 
                                        - Jan Hudec <bulb@ucw.cz>

Johannes Sixt· Dec 3, 2008, 12:18 UTC · re: Jan Hudec · lore

Re: Ad: fast-import problem importing dos format files under cygwin

Jan Hudec schrieb:

Show 15 quoted lines

> On 3 December 2008, 08:51, Jan Hudec wrote:
>> Hello folks,
>>
>> I have been playing with fast-import in cygwin and I have problems
>> importing files with CR/LF line-endings. The size in data command is
>> calculated including the CRs and than the file is copied binary to the
>> fast-import input stream. However fast-import skips the CRs when reading,
>> overreads by that number of bytes and fails when it tries to read the
>> next command from the middle.
> 
> One addition:
> 
> I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the
> test, as it was, except it didn't like 'refs/heads/master' as branchname
> (and accepted bare 'master', but that created '.git/master').

With my current version of MinGW git the import is successful after I edited test1.gfi to match your description (it had CR on all lines; I removed all except on lines 15 and 16). The repository content is as one would it expect given the input. master is b8ad21c3dc271d43a6e43c261909d6be725fa5b8.

Do you happen to have core.autocrlf set in some way and could it make a difference for fast-import? I have it unset.

-- Hannes

Jan Hudec· Dec 3, 2008, 13:43 UTC · re: Johannes Sixt · lore

Re: Ad: fast-import problem importing dos format files under cygwin

Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):

Show 14 quoted lines

> Jan Hudec schrieb:
>> On 3 December 2008, 08:51, Jan Hudec wrote:
>>> Hello folks,
>>>
>>> I have been playing with fast-import in cygwin and I have problems
>>> importing files with CR/LF line-endings. The size in data command is
>>> calculated including the CRs and than the file is copied binary to the
>>> fast-import input stream. However fast-import skips the CRs when
>>> reading,
>>> overreads by that number of bytes and fails when it tries to read the
>>> next command from the middle.
>
> Do you happen to have core.autocrlf set in some way and could it make a
> difference for fast-import? I have it unset.

I have it set to false explicitly in global config. Tried with not having it set at all and gives the same problem. Since the previous version of MSys Git worked for me, I suspect it's somehow cygwin-related.

-- 
                                        - Jan Hudec <bulb@ucw.cz>

Shawn O. Pearce· Dec 3, 2008, 16:04 UTC · re: Jan Hudec · lore

Re: Ad: fast-import problem importing dos format files under cygwin

Jan Hudec <bulb@ucw.cz> wrote:

Show 19 quoted lines

> Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> > Jan Hudec schrieb:
> >> On 3 December 2008, 08:51, Jan Hudec wrote:
> >>> Hello folks,
> >>>
> >>> I have been playing with fast-import in cygwin and I have problems
> >>> importing files with CR/LF line-endings. The size in data command is
> >>> calculated including the CRs and than the file is copied binary to the
> >>> fast-import input stream. However fast-import skips the CRs when
> >>> reading,
> >>> overreads by that number of bytes and fails when it tries to read the
> >>> next command from the middle.
> >
> > Do you happen to have core.autocrlf set in some way and could it make a
> > difference for fast-import? I have it unset.
> 
> I have it set to false explicitly in global config. Tried with not having
> it set at all and gives the same problem. Since the previous version of
> MSys Git worked for me, I suspect it's somehow cygwin-related.

Huh. So fast-import *never* does auto-CRLF conversion, even if the property is set. It just doesn't make those calls internally. It blindly copies data from the input stream into the pack. No exceptions.

fast-import under-reading near CRs and getting misaligned on its input indicates that the stdio library has given us a FILE* for stdin which is converting CRLF pairs into LFs, even within an fread() call.

My guess here is fast-import's stdin is set in text mode, but it really needs to be in binary mode. fast-import.c never attempts to correct that when it starts, so on DOS based systems we are probably totally screwed from the beginning...

-- 
Shawn.

Johannes Schindelin· Dec 3, 2008, 16:20 UTC · re: Shawn O. Pearce · lore

Re: Ad: fast-import problem importing dos format files under cygwin

Hi,

On Wed, 3 Dec 2008, Shawn O. Pearce wrote:

Show 32 quoted lines

> Jan Hudec <bulb@ucw.cz> wrote:
> > Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> > > Jan Hudec schrieb:
> > >> On 3 December 2008, 08:51, Jan Hudec wrote:
> > >>>
> > >>> I have been playing with fast-import in cygwin and I have problems 
> > >>> importing files with CR/LF line-endings. The size in data command 
> > >>> is calculated including the CRs and than the file is copied binary 
> > >>> to the fast-import input stream. However fast-import skips the CRs 
> > >>> when reading, overreads by that number of bytes and fails when it 
> > >>> tries to read the next command from the middle.
> > >
> > > Do you happen to have core.autocrlf set in some way and could it 
> > > make a difference for fast-import? I have it unset.
> > 
> > I have it set to false explicitly in global config. Tried with not 
> > having it set at all and gives the same problem. Since the previous 
> > version of MSys Git worked for me, I suspect it's somehow 
> > cygwin-related.
> 
> Huh.  So fast-import *never* does auto-CRLF conversion, even if the 
> property is set.  It just doesn't make those calls internally. It 
> blindly copies data from the input stream into the pack. No exceptions.
> 
> fast-import under-reading near CRs and getting misaligned on its input 
> indicates that the stdio library has given us a FILE* for stdin which is 
> converting CRLF pairs into LFs, even within an fread() call.
> 
> My guess here is fast-import's stdin is set in text mode, but it really 
> needs to be in binary mode.  fast-import.c never attempts to correct 
> that when it starts, so on DOS based systems we are probably totally 
> screwed from the beginning...

I think you need to set the environment variable

	CYGWIN=binmode

Hth, Dscho

Jan Hudec· Dec 3, 2008, 18:05 UTC · re: Johannes Schindelin · lore

Re: Ad: fast-import problem importing dos format files under cygwin

Hi,

On Wed, Dec 03, 2008 at 17:20:54 +0100, Johannes Schindelin wrote:

Show 23 quoted lines

> Hi,
> On Wed, 3 Dec 2008, Shawn O. Pearce wrote:
> > Jan Hudec <bulb@ucw.cz> wrote:
> > > Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> > > > Jan Hudec schrieb:
> > > >> On 3 December 2008, 08:51, Jan Hudec wrote:
> > > >>>
> > > >>> I have been playing with fast-import in cygwin and I have problems 
> > > >>> importing files with CR/LF line-endings. The size in data command 
> > > >>> is calculated including the CRs and than the file is copied binary 
> > > >>> to the fast-import input stream. However fast-import skips the CRs 
> > > >>> when reading, overreads by that number of bytes and fails when it 
> > > >>> tries to read the next command from the middle.
> >
> > [...] 
> > fast-import under-reading near CRs and getting misaligned on its input 
> > indicates that the stdio library has given us a FILE* for stdin which is 
> > converting CRLF pairs into LFs, even within an fread() call.
> > 
> > My guess here is fast-import's stdin is set in text mode, but it really 
> > needs to be in binary mode.  fast-import.c never attempts to correct 
> > that when it starts, so on DOS based systems we are probably totally 
> > screwed from the beginning...

Yes, it does indeed sound so. Strange thing is why it would be that way, when it does not seem to be the case for any other process (eg. the shell will complain loudly if I feed it a DOS formatted script). The standard input is simple shell redirect from a file on a binary-mounted filesystem. I'll do some more cross-checks tomorrow.

> I think you need to set the environment variable
> 
> 	CYGWIN=binmode

Will try. Thanks.

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>