git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Efficient parsing of `status -z` output

From
Matthew Rothenberg <mrothenberg@gmail.com>
Date
Mar 7, 2015, 23:00 UTC
Message-ID
<CAMJduDuxBDoJ9_ETY8FCRoANf+taAS7-1acf5CFRGXDFyL72Rg@mail.gmail.com>

I've been working on a utility that parses the output of `git status --porcelain` as a fundamental part of its operation.

Since I would like for this tool to be as robust as possible (and cross-platform compatibility is a goal), I am currently trying to migrate it from parsing the output of `--porcelain` to using the output of `-z`, to quote from the documentation:

    > There is also an alternate -z format recommended for machine parsing. In
    > that format, the status field is the same, but some other things change.
    > First, the -> is omitted from rename entries and the field order is
    > reversed (e.g from -> to becomes to from). Second, a NUL (ASCII 0) follows
    > each filename, replacing space as a field separator and the terminating
    > newline (but a space still separates the status field from the first
    > filename). Third, filenames containing special characters are not
    > specially formatted; no quoting or backslash-escaping is performed.

I am encountering some significant issues with using this because of one detail. In particular, parsing output using NUL as *both* the entry terminator and the filename separator for entries that contain multiple files is problematic. Because of this, one cannot know in advance how many NULs to read from the buffer until considering an entry to be in memory for parsing.

There are two workarounds I've considered:
 1. Reading the *entire* buffer into memory, and then using a regular
expression (yikes) to split the entries. This is something I would
obviously like to avoid for performance reasons.
 2. Read from buffer until the first NUL, parse the entry status
codes, and if the entry status code represents a status that *should*
have multiple filenames, read from buffer until a second NUL is found,
and then reparse that entry with both filenames. The issues I see with
this approach:
   a.) One has to know exactly which status code combinations will end
up with two filenames, and this list has to be exhaustive. As far as I
can tell, there is no canonical documentation for this?
   b.) It seems a bit brittle, because if the logic from the above is
wrong and we miss an extended entry or ask for one when it doesn't
exist we will leave the buffer an essentially corrupt state for future
reads.

My understanding is the goal of `-z` is to make machine parsing status from a binary stream *more* reliable, so perhaps (likely!) I am missing something obvious?

Thanks for any assistance!
Next: Junio C Hamano
Message 1 of 7 in “Efficient parsing of `status -z` output”
  1. Matthew RothenbergMar 7, 2015
  2. Junio C HamanoMar 8, 2015
  3. Matthew RothenbergMar 9, 2015
  4. Jeff KingMar 9, 2015
  5. Jeff KingMar 9, 2015
  6. Matthew RothenbergMar 9, 2015
  7. Jeff KingMar 10, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.