{"thread":{"id":"47713","subject":"\"git fast-import\" crashes parsing output from \"fossil export --git\"","startedAt":"2018-01-29T16:48:41Z","lastAt":"2018-01-29T20:29:02Z","messageCount":3,"participants":["Rowan Thorpe","SZEDER Gábor"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"337667","messageId":"CACgDUr6XG+dZ+GJcf9+11Edf=q-=QLSkmgpGf=XmondaLaescA@mail.gmail.com","threadId":"47713","inReplyTo":null,"subject":"\"git fast-import\" crashes parsing output from \"fossil export --git\"","fromName":"Rowan Thorpe","fromEmail":"rowan@rowanthorpe.com","sentAt":"2018-01-29T16:47:50Z","receivedAt":"2018-01-29T16:48:41Z","isPatch":false,"sender":{"key":"rowan@rowanthorpe.com","avatar":null},"body":"\"git fast-import\" fails while trying to import the latest \"Fossil\"\nfossil-sources (it dog-foods its own source-code). Namely, the below\npasted commands caused the crash report below them (I've trimmed\nirrelevant noise from the report for brevity). One of the tests\nincluded in Fossil uses example-files to test its handling of\nfilenames with special characters - in this case an embedded newline\nin \"abc[NEWLINE]def.txt\" - and \"git fast-import\" causes the second\npart of that filename (post-newline) to spill over as the \"command\" on\nthe next line of its input when transforming the \"fossil export\"\noutput into usable input. Considering git can handle such filenames, I\nguess \"git fast-import\" not handling them counts as a bug - unless a\ndesign-decision was made to not be as rigorous for the sake of\nsimplicity in the import protocol? Or is \"fossil export\" perhaps\noutputting the filename incorrectly for the \"fast-import\" protocol\n(maybe it needs some kind of quoting)?\n\n* Commands which cause the crash:\n----8<----\nfossil clone https://www.fossil-scm.org/xfer fossil.fossil\ngit init fossil-git\ncd fossil-git\nfossil export --git ../fossil.fossil | git fast-import\n----8<----\n\n* The part of \"fossil export\" output which breaks \"git fast-import\":\n----8<----\n...\n\ncommit refs/heads/test-ticket-d17d6e5b17\nmark :26782\ncommitter jan.nijtmans 1353531216 +0000\ndata 82\nJust commit some weird filenames, even one with a newline in it, to\ntest the code.\nfrom :26779\nM 100644 :427 :abc\nM 100644 :10049 abc\ndef.txt\nM 100644 :427 str\"i\"ng.h\nM 100644 :427 str[ing.txt\nM 100644 :427 xyz<5.x\n\n...\n----8<----\n\nCrash report [trimmed]:\n----8<----\nfast-import crash report:\n fast-import process: 23836\n parent process : 23835\n at 2018-01-29 15:12:07 +0000\n\nfatal: Unsupported command: def.txt\n\nMost Recent Commands Before Crash\n---------------------------------\n M 100644 :10003 src/tkt.c\n M 100644 :9992 src/update.c\n M 100644 :9838 src/wiki.c\n M 100644 :9986 src/wikiformat.c\n M 100644 :9995 src/xfer.c\n M 100644 :10007 src/th_main.c\n\n commit refs/heads/trunk\n mark :26772\n committer drh 1353449638 +0000\n data 115\n from :26770\n M 100644 :9981 src/attach.c\n M 100644 :10028 src/captcha.c\n M 100644 :10012 src/info.c\n M 100644 :9906 src/login.c\n M 100644 :9965 src/style.c\n M 100644 :10017 src/tkt.c\n M 100644 :10025 src/wiki.c\n\n commit refs/heads/th1Hooks\n mark :26773\n committer mistachkin 1353449970 +0000\n data 48\n from :26771\n M 100644 :10008 src/main.c\n\n commit refs/heads/trunk\n mark :26774\n committer mistachkin 1353450526 +0000\n data 70\n from :26772\n M 100644 :9245 src/captcha.c\n M 100644 :10030 src/login.c\n\n commit refs/heads/th1Hooks\n mark :26775\n committer mistachkin 1353452400 +0000\n data 51\n from :26773\n M 100644 :10036 src/main.c\n\n commit refs/heads/trunk\n mark :26776\n committer drh 1353459061 +0000\n data 141\n from :26774\n M 100644 :10027 src/attach.c\n M 100644 :9833 src/captcha.c\n M 100644 :9904 src/setup.c\n M 100644 :10031 src/style.c\n M 100644 :10032 src/tkt.c\n M 100644 :10033 src/wiki.c\n\n commit refs/heads/trunk\n mark :26777\n committer drh 1353460136 +0000\n data 56\n from :26776\n M 100644 :10037 src/captcha.c\n\n commit refs/heads/ticket-d17d6e5b17\n mark :26778\n committer jan.nijtmans 1353489155 +0000\n data 80\n from :26767\n M 100644 :10019 src/file.c\n M 100644 :9846 src/rebuild.c\n M 100644 :9677 src/vfile.c\n\n commit refs/heads/ticket-d17d6e5b17\n mark :26779\n committer jan.nijtmans 1353489615 +0000\n data 41\n from :26778\n M 100644 :10043 src/file.c\n\n commit refs/heads/trunk\n mark :26780\n committer drh 1353510669 +0000\n data 64\n from :26777\n M 100644 :9850 src/encode.c\n M 100644 :10018 src/wikiformat.c\n\n commit refs/heads/trunk\n mark :26781\n committer drh 1353515283 +0000\n data 51\n from :26780\n M 100644 :10047 src/wikiformat.c\n\n commit refs/heads/test-ticket-d17d6e5b17\n mark :26782\n committer jan.nijtmans 1353531216 +0000\n data 82\n from :26779\n M 100644 :427 :abc\n M 100644 :10049 abc\n* def.txt\n\nActive Branch LRU\n-----------------\n active_branches = 5 cur, 5 max\n\n pos clock name\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n 1) 4672 refs/heads/test-ticket-d17d6e5b17\n 2) 4669 refs/heads/ticket-d17d6e5b17\n 3) 4654 refs/heads/dbReOpenConfigAttach\n 4) 4665 refs/heads/th1Hooks\n 5) 4671 refs/heads/trunk\n\nInactive Branches\n-----------------\n...[loads of trimmed entries]...\n\nMarks\n-----\n...[loads of trimmed entries]...\n\n-------------------\nEND OF CRASH REPORT\n----8<----\n"},{"id":"337669","messageId":"20180129171127.17097-1-szeder.dev@gmail.com","threadId":"47713","inReplyTo":"CACgDUr6XG+dZ+GJcf9+11Edf=q-=QLSkmgpGf=XmondaLaescA@mail.gmail.com","subject":"\"git fast-import\" crashes parsing output from \"fossil export --git\"","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-29T17:11:27Z","receivedAt":"2018-01-29T17:11:45Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> \"git fast-import\" fails while trying to import the latest \"Fossil\"\n> fossil-sources (it dog-foods its own source-code). Namely, the below\n> pasted commands caused the crash report below them (I've trimmed\n> irrelevant noise from the report for brevity). One of the tests\n> included in Fossil uses example-files to test its handling of\n> filenames with special characters - in this case an embedded newline\n> in \"abc[NEWLINE]def.txt\" - and \"git fast-import\" causes the second\n> part of that filename (post-newline) to spill over as the \"command\" on\n> the next line of its input when transforming the \"fossil export\"\n> output into usable input. Considering git can handle such filenames, I\n> guess \"git fast-import\" not handling them counts as a bug - unless a\n> design-decision was made to not be as rigorous for the sake of\n> simplicity in the import protocol? Or is \"fossil export\" perhaps\n> outputting the filename incorrectly for the \"fast-import\" protocol\n> (maybe it needs some kind of quoting)?\n> \n> * Commands which cause the crash:\n> ----8<----\n> fossil clone https://www.fossil-scm.org/xfer fossil.fossil\n> git init fossil-git\n> cd fossil-git\n> fossil export --git ../fossil.fossil | git fast-import\n> ----8<----\n> \n> * The part of \"fossil export\" output which breaks \"git fast-import\":\n> ----8<----\n> ...\n> \n> commit refs/heads/test-ticket-d17d6e5b17\n> mark :26782\n> committer jan.nijtmans 1353531216 +0000\n> data 82\n> Just commit some weird filenames, even one with a newline in it, to\n> test the code.\n> from :26779\n> M 100644 :427 :abc\n> M 100644 :10049 abc\n> def.txt\n\nA path like this must be quoted. Quoting from 'git fast-import'\nmanpage:\n\n A string must use UNIX-style directory separators (forward\n slash /), may contain any byte other than LF, and must not start\n with double quote (\").\n\n A path can use C-style string quoting; this is accepted in all\n cases and mandatory if the filename starts with double quote or\n contains LF. In C-style quoting, the complete name should be\n surrounded with double quotes, and any LF, backslash, or double\n quote characters must be escaped by preceding them with a backslash\n (e.g., \"path/with\\n, \\\\ and \\\" in it\").\n \n"},{"id":"337693","messageId":"CACgDUr6_LCMJaMcBBTfqTEH6nJfLKXRqu-HKMDanhqdLfHzbPA@mail.gmail.com","threadId":"47713","inReplyTo":"20180129171127.17097-1-szeder.dev@gmail.com","subject":"Re: \"git fast-import\" crashes parsing output from \"fossil export --git\"","fromName":"Rowan Thorpe","fromEmail":"rowan@rowanthorpe.com","sentAt":"2018-01-29T20:15:22Z","receivedAt":"2018-01-29T20:29:02Z","isPatch":false,"sender":{"key":"rowan@rowanthorpe.com","avatar":null},"body":"On 29 January 2018 at 19:11, SZEDER Gábor wrote:\n> > ..[snip]..\n> > Just commit some weird filenames, even one with a newline in it, to\n> > test the code.\n> > from :26779\n> > M 100644 :427 :abc\n> > M 100644 :10049 abc\n> > def.txt\n>\n> A path like this must be quoted. Quoting from 'git fast-import'\n> manpage:\n>\n> A string must use UNIX-style directory separators (forward\n> slash /), may contain any byte other than LF, and must not start\n> with double quote (\").\n>\n> A path can use C-style string quoting; this is accepted in all\n> cases and mandatory if the filename starts with double quote or\n> contains LF. In C-style quoting, the complete name should be\n> surrounded with double quotes, and any LF, backslash, or double\n> quote characters must be escaped by preceding them with a backslash\n> (e.g., \"path/with\\n, \\\\ and \\\" in it\").\n\nAh, thanks. I had skimmed that manpage quickly but obviously too\nquickly and somehow missed that paragraph. I will post the bug-report\nat Fossil, where it belongs. Sorry for the noise.\n"}]}