{"thread":{"id":"16573","subject":"fast-import problem importing dos format files under cygwin","startedAt":"2008-12-03T07:51:40Z","lastAt":"2008-12-03T18:05:50Z","messageCount":7,"participants":["Jan Hudec","Johannes Sixt","Shawn O. Pearce","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"97032","messageId":"43827.194.138.12.144.1228290700.squirrel@artax.karlin.mff.cuni.cz","threadId":"16573","inReplyTo":null,"subject":"fast-import problem importing dos format files under cygwin","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-12-03T07:51:40Z","receivedAt":"2008-12-03T07:51:40Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"Hello folks,\n\nI have been playing with fast-import in cygwin and I have problems\nimporting files with CR/LF line-endings. The size in data command is\ncalculated including the CRs and than the file is copied binary to the\nfast-import input stream. However fast-import skips the CRs when reading,\noverreads by that number of bytes and fails when it tries to read the next\ncommand from the middle.\n\nAttached is a test input stream and crash report generated by fast-import\nwhen reading it. In case mail system damages it along the way despite\nbeing attached to prevent that, the file should be in unix format -- that\nis what my cygwin perl outputs by default -- and has CRs only on lines 15\nand 16. The unix.txt and dos.txt should only differ that the '.'s in\nformer are replaced by '^M's in the later (so the data commands are\notherwise same).\n\nNote, that when I convert the file to dos format, it is read as intended.\nHowever, that is inconsistent with rest of the cygwin environment which\ngenerated and expects files in unix format. I use binary mounts (not\nconverting) and CYGWIN environment variable is empty. My git version is\n1.6.0.4 from official Cygwin package.\n\nIs this behaviour intentional workaround for something or a bug?\n\n-- \n - Jan Hudec "},{"id":"97038","messageId":"43270.194.138.12.144.1228295417.squirrel@artax.karlin.mff.cuni.cz","threadId":"16573","inReplyTo":"43827.194.138.12.144.1228290700.squirrel@artax.karlin.mff.cuni.cz","subject":"Ad: fast-import problem importing dos format files under cygwin","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-12-03T09:10:17Z","receivedAt":"2008-12-03T09:10:17Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On 3 December 2008, 08:51, Jan Hudec wrote:\n> Hello folks,\n>\n> I have been playing with fast-import in cygwin and I have problems\n> importing files with CR/LF line-endings. The size in data command is\n> calculated including the CRs and than the file is copied binary to the\n> fast-import input stream. However fast-import skips the CRs when reading,\n> overreads by that number of bytes and fails when it tries to read the\n> next command from the middle.\n\nOne addition:\n\nI have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the\ntest, as it was, except it didn't like 'refs/heads/master' as branchname\n(and accepted bare 'master', but that created '.git/master').\n\n-- \n - Jan Hudec \n"},{"id":"97053","messageId":"49367909.8070605@viscovery.net","threadId":"16573","inReplyTo":"43270.194.138.12.144.1228295417.squirrel@artax.karlin.mff.cuni.cz","subject":"Re: Ad: fast-import problem importing dos format files under cygwin","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-12-03T12:18:17Z","receivedAt":"2008-12-03T12:18:17Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Jan Hudec schrieb:\n> On 3 December 2008, 08:51, Jan Hudec wrote:\n>> Hello folks,\n>>\n>> I have been playing with fast-import in cygwin and I have problems\n>> importing files with CR/LF line-endings. The size in data command is\n>> calculated including the CRs and than the file is copied binary to the\n>> fast-import input stream. However fast-import skips the CRs when reading,\n>> overreads by that number of bytes and fails when it tries to read the\n>> next command from the middle.\n> \n> One addition:\n> \n> I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the\n> test, as it was, except it didn't like 'refs/heads/master' as branchname\n> (and accepted bare 'master', but that created '.git/master').\n\nWith my current version of MinGW git the import is successful after I\nedited test1.gfi to match your description (it had CR on all lines; I\nremoved all except on lines 15 and 16). The repository content is as one\nwould it expect given the input. master is\nb8ad21c3dc271d43a6e43c261909d6be725fa5b8.\n\nDo you happen to have core.autocrlf set in some way and could it make a\ndifference for fast-import? I have it unset.\n\n-- Hannes\n"},{"id":"97067","messageId":"51143.194.138.12.144.1228311791.squirrel@artax.karlin.mff.cuni.cz","threadId":"16573","inReplyTo":"49367909.8070605@viscovery.net","subject":"Re: Ad: fast-import problem importing dos format files under cygwin","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-12-03T13:43:11Z","receivedAt":"2008-12-03T13:43:11Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"\nDne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):\n> Jan Hudec schrieb:\n>> On 3 December 2008, 08:51, Jan Hudec wrote:\n>>> Hello folks,\n>>>\n>>> I have been playing with fast-import in cygwin and I have problems\n>>> importing files with CR/LF line-endings. The size in data command is\n>>> calculated including the CRs and than the file is copied binary to the\n>>> fast-import input stream. However fast-import skips the CRs when\n>>> reading,\n>>> overreads by that number of bytes and fails when it tries to read the\n>>> next command from the middle.\n>\n> Do you happen to have core.autocrlf set in some way and could it make a\n> difference for fast-import? I have it unset.\n\nI have it set to false explicitly in global config. Tried with not having\nit set at all and gives the same problem. Since the previous version of\nMSys Git worked for me, I suspect it's somehow cygwin-related.\n\n-- \n - Jan Hudec \n"},{"id":"97075","messageId":"20081203160446.GY23984@spearce.org","threadId":"16573","inReplyTo":"51143.194.138.12.144.1228311791.squirrel@artax.karlin.mff.cuni.cz","subject":"Re: Ad: fast-import problem importing dos format files under cygwin","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-12-03T16:04:46Z","receivedAt":"2008-12-03T16:04:46Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jan Hudec wrote:\n> Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):\n> > Jan Hudec schrieb:\n> >> On 3 December 2008, 08:51, Jan Hudec wrote:\n> >>> Hello folks,\n> >>>\n> >>> I have been playing with fast-import in cygwin and I have problems\n> >>> importing files with CR/LF line-endings. The size in data command is\n> >>> calculated including the CRs and than the file is copied binary to the\n> >>> fast-import input stream. However fast-import skips the CRs when\n> >>> reading,\n> >>> overreads by that number of bytes and fails when it tries to read the\n> >>> next command from the middle.\n> >\n> > Do you happen to have core.autocrlf set in some way and could it make a\n> > difference for fast-import? I have it unset.\n> \n> I have it set to false explicitly in global config. Tried with not having\n> it set at all and gives the same problem. Since the previous version of\n> MSys Git worked for me, I suspect it's somehow cygwin-related.\n\nHuh. So fast-import *never* does auto-CRLF conversion, even if the\nproperty is set. It just doesn't make those calls internally.\nIt blindly copies data from the input stream into the pack.\nNo exceptions.\n\nfast-import under-reading near CRs and getting misaligned on its\ninput indicates that the stdio library has given us a FILE* for stdin\nwhich is converting CRLF pairs into LFs, even within an fread() call.\n\nMy guess here is fast-import's stdin is set in text mode, but it\nreally needs to be in binary mode. fast-import.c never attempts\nto correct that when it starts, so on DOS based systems we are\nprobably totally screwed from the beginning...\n\n-- \nShawn.\n"},{"id":"97077","messageId":"alpine.DEB.1.00.0812031718350.28112@intel-tinevez-2-302","threadId":"16573","inReplyTo":"20081203160446.GY23984@spearce.org","subject":"Re: Ad: fast-import problem importing dos format files under cygwin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-12-03T16:20:54Z","receivedAt":"2008-12-03T16:20:54Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 3 Dec 2008, Shawn O. Pearce wrote:\n\n> Jan Hudec wrote:\n> > Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):\n> > > Jan Hudec schrieb:\n> > >> On 3 December 2008, 08:51, Jan Hudec wrote:\n> > >>>\n> > >>> I have been playing with fast-import in cygwin and I have problems \n> > >>> importing files with CR/LF line-endings. The size in data command \n> > >>> is calculated including the CRs and than the file is copied binary \n> > >>> to the fast-import input stream. However fast-import skips the CRs \n> > >>> when reading, overreads by that number of bytes and fails when it \n> > >>> tries to read the next command from the middle.\n> > >\n> > > Do you happen to have core.autocrlf set in some way and could it \n> > > make a difference for fast-import? I have it unset.\n> > \n> > I have it set to false explicitly in global config. Tried with not \n> > having it set at all and gives the same problem. Since the previous \n> > version of MSys Git worked for me, I suspect it's somehow \n> > cygwin-related.\n> \n> Huh. So fast-import *never* does auto-CRLF conversion, even if the \n> property is set. It just doesn't make those calls internally. It \n> blindly copies data from the input stream into the pack. No exceptions.\n> \n> fast-import under-reading near CRs and getting misaligned on its input \n> indicates that the stdio library has given us a FILE* for stdin which is \n> converting CRLF pairs into LFs, even within an fread() call.\n> \n> My guess here is fast-import's stdin is set in text mode, but it really \n> needs to be in binary mode. fast-import.c never attempts to correct \n> that when it starts, so on DOS based systems we are probably totally \n> screwed from the beginning...\n\nI think you need to set the environment variable\n\n\tCYGWIN=binmode\n\nHth,\nDscho\n"},{"id":"97081","messageId":"20081203180550.GA5149@efreet.light.src","threadId":"16573","inReplyTo":"alpine.DEB.1.00.0812031718350.28112@intel-tinevez-2-302","subject":"Re: Ad: fast-import problem importing dos format files under cygwin","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-12-03T18:05:50Z","receivedAt":"2008-12-03T18:05:50Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"Hi,\n\nOn Wed, Dec 03, 2008 at 17:20:54 +0100, Johannes Schindelin wrote:\n> Hi,\n> On Wed, 3 Dec 2008, Shawn O. Pearce wrote:\n> > Jan Hudec wrote:\n> > > Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):\n> > > > Jan Hudec schrieb:\n> > > >> On 3 December 2008, 08:51, Jan Hudec wrote:\n> > > >>>\n> > > >>> I have been playing with fast-import in cygwin and I have problems \n> > > >>> importing files with CR/LF line-endings. The size in data command \n> > > >>> is calculated including the CRs and than the file is copied binary \n> > > >>> to the fast-import input stream. However fast-import skips the CRs \n> > > >>> when reading, overreads by that number of bytes and fails when it \n> > > >>> tries to read the next command from the middle.\n> >\n> > [...] \n> > fast-import under-reading near CRs and getting misaligned on its input \n> > indicates that the stdio library has given us a FILE* for stdin which is \n> > converting CRLF pairs into LFs, even within an fread() call.\n> > \n> > My guess here is fast-import's stdin is set in text mode, but it really \n> > needs to be in binary mode. fast-import.c never attempts to correct \n> > that when it starts, so on DOS based systems we are probably totally \n> > screwed from the beginning...\n\nYes, it does indeed sound so. Strange thing is why it would be that way, when\nit does not seem to be the case for any other process (eg. the shell will\ncomplain loudly if I feed it a DOS formatted script). The standard input is\nsimple shell redirect from a file on a binary-mounted filesystem. I'll do\nsome more cross-checks tomorrow.\n\n> I think you need to set the environment variable\n> \n> \tCYGWIN=binmode\n\nWill try. Thanks.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"}]}