git/list[1] front-page[2] threads[3] people[4] search[5] about
 

force deterministic trees on git push - exact sort-order of filenames

From
milan hauth <milahu@gmail.com>
Date
Oct 15, 2021, 16:04 UTC
Message-ID
<CAGiEHCtdjA+cVXjN43NPbSZfrDtr-kDtPMN4x_VTGSJuPZ8bcg@mail.gmail.com>
fact: the sort-order of filenames in a tree is not strictly regulated

proposal: enforce the exact sort-order of tree-items, to make trees deterministic and reproducible. the git server could refuse a 'git push', when the tree is invalid

use case: reconstruct a git tree only from files.

current situation: blobs are deterministic, trees are non-deterministic

backward compatibility: rewriting git history is usually not desired. so this new rule would apply only to new commits after a certain 'deadline', set by the git server

sample trees from [1] and [2]:

git cat-file -p 2b75a7dbb76138587dbe50a5a6af8a6eedbaf66b | grep id_ed25519 100644 blob f914b3f712fc56ab212b53cb9055e1291b5c77a2 id_ed25519 100644 blob 40de4a8ac6027f64ac85f687bea7049467b428a2 id_ed25519.pub

git cat-file -p c8a72e628d0ca0a174a1a4241e6c7314a4660f0f | grep example 100644 blob fde6f3cbd19addb8ce84ffe32ab4d040e8b09c18 example.pem 040000 tree 6b0ee97865059ac965590e0ff5272fb76b6fd2c8 example

the first tree is sorted as expected

printf 'id_ed25519\nid_ed25519.pub\n' | sort id_ed25519 id_ed25519.pub

the expected sort-order for the second tree:

printf 'example.pem\nexample\n' | sort example example.pem

the "unexpected" sort-order can be produced with [3]:
```py
import functools
def cmp(s, t):
  """
  Alter lexicographic sort order
  to make longer keys go *before* any of their prefixes
  """
  for p, q in zip(s, t):
    if p < q: return -1
    if q < p: return 1
  if len(s) > len(t): return -1
  elif len(t) > len(s): return 1
  return 0

arr = [ 'example', 'example.pem' ] print("\n".join(sorted(arr, key=functools.cmp_to_key(cmp)))) ```

result:

example.pem example

[1] sample tree 1 https://api.github.com/repos/NixOS/nixpkgs/git/trees/2b75a7dbb76138587dbe50a5a6af8a6eedbaf66b

[2] sample tree 2 https://api.github.com/repos/NixOS/nixpkgs/git/trees/c8a72e628d0ca0a174a1a4241e6c7314a4660f0f

[3] python impl of unexpected sort https://stackoverflow.com/questions/42899405/sort-a-list-with-longest-items-first

[4] related issue (wontfix) https://github.com/dulwich/dulwich/issues/905

Next: René Scharfe
Message 1 of 6 in “force deterministic trees on git push - exact sort-order of filenames”
  1. milan hauthOct 15, 2021
  2. René ScharfeOct 15, 2021
  3. Junio C HamanoOct 15, 2021
  4. milan hauthOct 15, 2021
  5. Ævar Arnfjörð BjarmasonOct 17, 2021
  6. Junio C HamanoOct 18, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.