force deterministic trees on git push - exact sort-order of filenames
- From
milan hauth <milahu@gmail.com>
- Date
- Oct 15, 2021, 16:04 UTC
- Message-ID
- <CAGiEHCtdjA+cVXjN43NPbSZfrDtr-kDtPMN4x_VTGSJuPZ8bcg@mail.gmail.com>
fact: the sort-order of filenames in a tree is not strictly regulated
proposal: enforce the exact sort-order of tree-items, to make trees deterministic and reproducible. the git server could refuse a 'git push', when the tree is invalid
use case: reconstruct a git tree only from files.
current situation: blobs are deterministic, trees are non-deterministic
backward compatibility: rewriting git history is usually not desired. so this new rule would apply only to new commits after a certain 'deadline', set by the git server
sample trees from [1] and [2]:
git cat-file -p 2b75a7dbb76138587dbe50a5a6af8a6eedbaf66b | grep id_ed25519 100644 blob f914b3f712fc56ab212b53cb9055e1291b5c77a2 id_ed25519 100644 blob 40de4a8ac6027f64ac85f687bea7049467b428a2 id_ed25519.pub
git cat-file -p c8a72e628d0ca0a174a1a4241e6c7314a4660f0f | grep example 100644 blob fde6f3cbd19addb8ce84ffe32ab4d040e8b09c18 example.pem 040000 tree 6b0ee97865059ac965590e0ff5272fb76b6fd2c8 example
the first tree is sorted as expected
printf 'id_ed25519\nid_ed25519.pub\n' | sort id_ed25519 id_ed25519.pub
the expected sort-order for the second tree:
printf 'example.pem\nexample\n' | sort example example.pem
the "unexpected" sort-order can be produced with [3]:
```py
import functools
def cmp(s, t):
"""
Alter lexicographic sort order
to make longer keys go *before* any of their prefixes
"""
for p, q in zip(s, t):
if p < q: return -1
if q < p: return 1
if len(s) > len(t): return -1
elif len(t) > len(s): return 1
return 0arr = [ 'example', 'example.pem' ] print("\n".join(sorted(arr, key=functools.cmp_to_key(cmp)))) ```
result:
example.pem example
[1] sample tree 1 https://api.github.com/repos/NixOS/nixpkgs/git/trees/2b75a7dbb76138587dbe50a5a6af8a6eedbaf66b
[2] sample tree 2 https://api.github.com/repos/NixOS/nixpkgs/git/trees/c8a72e628d0ca0a174a1a4241e6c7314a4660f0f
[3] python impl of unexpected sort https://stackoverflow.com/questions/42899405/sort-a-list-with-longest-items-first
[4] related issue (wontfix) https://github.com/dulwich/dulwich/issues/905