threads / discuss / 47713

"git fast-import" crashes parsing output from "fossil export --git"

Subject: "git fast-import" crashes parsing output from "fossil export --git"

## tl;dr

3 messages between Jan 29, 2018 and Jan 29, 2018.

replies: 2people: 2as markdown or json

Rowan Thorpe· Jan 29, 2018, 16:47 UTC · lore

"git fast-import" fails while trying to import the latest "Fossil" fossil-sources (it dog-foods its own source-code). Namely, the below pasted commands caused the crash report below them (I've trimmed irrelevant noise from the report for brevity). One of the tests included in Fossil uses example-files to test its handling of filenames with special characters - in this case an embedded newline in "abc[NEWLINE]def.txt" - and "git fast-import" causes the second part of that filename (post-newline) to spill over as the "command" on the next line of its input when transforming the "fossil export" output into usable input. Considering git can handle such filenames, I guess "git fast-import" not handling them counts as a bug - unless a design-decision was made to not be as rigorous for the sake of simplicity in the import protocol? Or is "fossil export" perhaps outputting the filename incorrectly for the "fast-import" protocol (maybe it needs some kind of quoting)?

* Commands which cause the crash:
----8<----
fossil clone https://www.fossil-scm.org/xfer fossil.fossil
git init fossil-git
cd fossil-git
fossil export --git ../fossil.fossil | git fast-import
----8<----
* The part of "fossil export" output which breaks "git fast-import":
----8<----
...

commit refs/heads/test-ticket-d17d6e5b17 mark :26782 committer jan.nijtmans <jan.nijtmans> 1353531216 +0000 data 82 Just commit some weird filenames, even one with a newline in it, to test the code. from :26779 M 100644 :427 :abc M 100644 :10049 abc def.txt M 100644 :427 str"i"ng.h M 100644 :427 str[ing.txt M 100644 :427 xyz<5.x

... ----8<----

Crash report [trimmed]:
----8<----
fast-import crash report:
    fast-import process: 23836
    parent process     : 23835
    at 2018-01-29 15:12:07 +0000
fatal: Unsupported command: def.txt
Most Recent Commands Before Crash
---------------------------------
  M 100644 :10003 src/tkt.c
  M 100644 :9992 src/update.c
  M 100644 :9838 src/wiki.c
  M 100644 :9986 src/wikiformat.c
  M 100644 :9995 src/xfer.c
  M 100644 :10007 src/th_main.c
  commit refs/heads/trunk
  mark :26772
  committer drh <drh> 1353449638 +0000
  data 115
  from :26770
  M 100644 :9981 src/attach.c
  M 100644 :10028 src/captcha.c
  M 100644 :10012 src/info.c
  M 100644 :9906 src/login.c
  M 100644 :9965 src/style.c
  M 100644 :10017 src/tkt.c
  M 100644 :10025 src/wiki.c
  commit refs/heads/th1Hooks
  mark :26773
  committer mistachkin <mistachkin> 1353449970 +0000
  data 48
  from :26771
  M 100644 :10008 src/main.c
  commit refs/heads/trunk
  mark :26774
  committer mistachkin <mistachkin> 1353450526 +0000
  data 70
  from :26772
  M 100644 :9245 src/captcha.c
  M 100644 :10030 src/login.c
  commit refs/heads/th1Hooks
  mark :26775
  committer mistachkin <mistachkin> 1353452400 +0000
  data 51
  from :26773
  M 100644 :10036 src/main.c
  commit refs/heads/trunk
  mark :26776
  committer drh <drh> 1353459061 +0000
  data 141
  from :26774
  M 100644 :10027 src/attach.c
  M 100644 :9833 src/captcha.c
  M 100644 :9904 src/setup.c
  M 100644 :10031 src/style.c
  M 100644 :10032 src/tkt.c
  M 100644 :10033 src/wiki.c
  commit refs/heads/trunk
  mark :26777
  committer drh <drh> 1353460136 +0000
  data 56
  from :26776
  M 100644 :10037 src/captcha.c
  commit refs/heads/ticket-d17d6e5b17
  mark :26778
  committer jan.nijtmans <jan.nijtmans> 1353489155 +0000
  data 80
  from :26767
  M 100644 :10019 src/file.c
  M 100644 :9846 src/rebuild.c
  M 100644 :9677 src/vfile.c
  commit refs/heads/ticket-d17d6e5b17
  mark :26779
  committer jan.nijtmans <jan.nijtmans> 1353489615 +0000
  data 41
  from :26778
  M 100644 :10043 src/file.c
  commit refs/heads/trunk
  mark :26780
  committer drh <drh> 1353510669 +0000
  data 64
  from :26777
  M 100644 :9850 src/encode.c
  M 100644 :10018 src/wikiformat.c
  commit refs/heads/trunk
  mark :26781
  committer drh <drh> 1353515283 +0000
  data 51
  from :26780
  M 100644 :10047 src/wikiformat.c
  commit refs/heads/test-ticket-d17d6e5b17
  mark :26782
  committer jan.nijtmans <jan.nijtmans> 1353531216 +0000
  data 82
  from :26779
  M 100644 :427 :abc
  M 100644 :10049 abc
* def.txt
Active Branch LRU
-----------------
    active_branches = 5 cur, 5 max
  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1)   4672 refs/heads/test-ticket-d17d6e5b17
   2)   4669 refs/heads/ticket-d17d6e5b17
   3)   4654 refs/heads/dbReOpenConfigAttach
   4)   4665 refs/heads/th1Hooks
   5)   4671 refs/heads/trunk

Inactive Branches ----------------- ...[loads of trimmed entries]...

Marks ----- ...[loads of trimmed entries]...

------------------- END OF CRASH REPORT ----8<----

SZEDER Gábor· Jan 29, 2018, 17:11 UTC · re: Rowan Thorpe · lore
Show 38 quoted lines
> "git fast-import" fails while trying to import the latest "Fossil"
> fossil-sources (it dog-foods its own source-code). Namely, the below
> pasted commands caused the crash report below them (I've trimmed
> irrelevant noise from the report for brevity). One of the tests
> included in Fossil uses example-files to test its handling of
> filenames with special characters - in this case an embedded newline
> in "abc[NEWLINE]def.txt" - and "git fast-import" causes the second
> part of that filename (post-newline) to spill over as the "command" on
> the next line of its input when transforming the "fossil export"
> output into usable input. Considering git can handle such filenames, I
> guess "git fast-import" not handling them counts as a bug - unless a
> design-decision was made to not be as rigorous for the sake of
> simplicity in the import protocol? Or is "fossil export" perhaps
> outputting the filename incorrectly for the "fast-import" protocol
> (maybe it needs some kind of quoting)?
> 
> * Commands which cause the crash:
> ----8<----
> fossil clone https://www.fossil-scm.org/xfer fossil.fossil
> git init fossil-git
> cd fossil-git
> fossil export --git ../fossil.fossil | git fast-import
> ----8<----
> 
> * The part of "fossil export" output which breaks "git fast-import":
> ----8<----
> ...
> 
> commit refs/heads/test-ticket-d17d6e5b17
> mark :26782
> committer jan.nijtmans <jan.nijtmans> 1353531216 +0000
> data 82
> Just commit some weird filenames, even one with a newline in it, to
> test the code.
> from :26779
> M 100644 :427 :abc
> M 100644 :10049 abc
> def.txt

A path like this must be quoted. Quoting from 'git fast-import' manpage:

  A <path> string must use UNIX-style directory separators (forward
  slash /), may contain any byte other than LF, and must not start
  with double quote (").
  A path can use C-style string quoting; this is accepted in all
  cases and mandatory if the filename starts with double quote or
  contains LF. In C-style quoting, the complete name should be
  surrounded with double quotes, and any LF, backslash, or double
  quote characters must be escaped by preceding them with a backslash
  (e.g., "path/with\n, \\ and \" in it").
 
Rowan Thorpe· Jan 29, 2018, 20:15 UTC · re: SZEDER Gábor · lore

Re: "git fast-import" crashes parsing output from "fossil export --git"

On 29 January 2018 at 19:11, SZEDER Gábor <szeder.dev@gmail.com> wrote:
Show 21 quoted lines
> > ..[snip]..
> > Just commit some weird filenames, even one with a newline in it, to
> > test the code.
> > from :26779
> > M 100644 :427 :abc
> > M 100644 :10049 abc
> > def.txt
>
> A path like this must be quoted.  Quoting from 'git fast-import'
> manpage:
>
>   A <path> string must use UNIX-style directory separators (forward
>   slash /), may contain any byte other than LF, and must not start
>   with double quote (").
>
>   A path can use C-style string quoting; this is accepted in all
>   cases and mandatory if the filename starts with double quote or
>   contains LF. In C-style quoting, the complete name should be
>   surrounded with double quotes, and any LF, backslash, or double
>   quote characters must be escaped by preceding them with a backslash
>   (e.g., "path/with\n, \\ and \" in it").

Ah, thanks. I had skimmed that manpage quickly but obviously too quickly and somehow missed that paragraph. I will post the bug-report at Fossil, where it belongs. Sorry for the noise.

← back to recent threads