Re: git --archive

From: René Scharfe <l.s.r@web.de>
Date: Sep 25, 2022, 08:17 UTC
Message-ID: <164df3bc-a7e1-70b5-e2b3-3f6c68494f0a@web.de>
In-Reply-To: <b0e84636-d856-eb52-dec0-07727f297b62@web.de>

Am 24.09.22 um 20:07 schrieb René Scharfe:

Show 36 quoted lines

> Am 24.09.22 um 15:19 schrieb René Scharfe:
>> It could be done by relying on the randomness of the object IDs and
>> partitioning by a sub-string.  Or perhaps using pseudo-random numbers
>> is sufficient:
>>
>>    git ls-tree -r HEAD |
>>    awk '{print $3}' |
>
> No need for awk here, of course; "git ls-tree -r --object-only HEAD"
> does the same.  Just saying.
>
>> Here's an idea after all: Using "git ls-tree" without "-r" and handling
>> recursing in the prefetch script would allow traversing trees in a
>> different order and even in parallel.  Not sure how to limit parallelism
>> to a sane degree.
>
> How about something like this?  xargs -P provides a controlled degree of
> parallelism.  Sorting by object ID (i.e. hash value) should provide a
> fairly random order.  Does this thing work for you?
>
>
> treeish=HEAD
> parallelism=8
>
> dir=$(mktemp -d)
> echo "$treeish" >"$dir/trees"
>
> # Traverse all sub-trees in randomized order and collect all blob IDs.
> while test -s "$dir/trees"
> do
>         sort <"$dir/trees" >"$dir/trees.current"
>         rm "$dir/trees"
>         xargs -P "$parallelism" -L 1 git ls-tree <"$dir/trees.current" |
>         awk -v dir="$dir" -v pieces="$parallelism" '
>                 $2 == "tree" {print $3 > (dir "/trees")}
>                 $2 == "blob" {print $3 >> (dir "/blobs" int(rand() * pieces))}

This will exhaust the file descriptor limit (ulimit -n) for really high degrees of parallelism. Here's a version that scales beyond that. It takes ca. five seconds on my machine against Git's own repository on an SSD, so its process overhead should still be bearable for your purpose.

treeish=HEAD parallelism=1000

dir=$(mktemp -d) echo "$treeish" >"$dir/trees"

while test -s "$dir/trees"
do
        sort <"$dir/trees" >"$dir/trees.current"
        rm "$dir/trees"
        xargs -P "$parallelism" -L 1 git ls-tree <"$dir/trees.current" |
        awk -v dir="$dir" -v pieces="$parallelism" '
                $2 == "tree" {print $3 > (dir "/trees")}
                $2 == "blob" {print int(rand()*pieces)+1, $3 >> (dir "/blobs")}
        '
done

replstr="%" command="awk '\$1 == \"%\" {print \$2}' | sort | git cat-file --batch >/dev/null" seq "$parallelism" | xargs -P "$parallelism" -I "$replstr" -L 1 sh -c "$command"

rm "$dir/trees.current" "$dir/blobs" rmdir "$dir"