git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Repository corruption if objects pushed in the middle of repack

From
Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Date
Jun 13, 2022, 20:31 UTC
Message-ID
<20220613203145.wbpi2m3ys3hchw6c@meerkat.local>
Hi, all:

I'm trying to figure the cause of repository corruption in a very specific case. Here's the setup:

1. the repository is several GB in size, full of automatically generated pushes
   (https://git.yoctoproject.org/poky-buildhistory/)
2. this repository has no alternates or other clever things -- just your old
   boring repository
2. the builders check out this repository with --depth 1 during the build
   stage, then add new logs to the repository, commit and push
Admittedly, this is a bad use of git, but let's use that outside the scope.

Every weekend we run a set of maintenance tasks and if we find that there are lots of new loose objects (which there usually are), we fire off a routine repack:

1. first, repack runs with the following flags (-f if deemed necessary):
        git repack -n --window-memory=1g -a -b --unpack-unreachable=yesterday -f --pack-kept-objects -d
   Since the repository is large, this usually takes a long time (3+ hours)
2. next, we generate a fresh commit-graph:
        git commit-graph write
3. next, we run pack-refs:
        git pack-refs --all
4. after that, we run prune:
        git prune --expire=yesterday

In the case of this particular repository, we regularly run into repository corruption, reported during the prune stage:

    fsck[10362] 2022-05-09 01:00:06,378 - INFO - /var/lib/gitolite3/repositories/poky-buildhistory.git:
    fsck[10362] 2022-05-09 01:00:06,700 - INFO -    repack: performing a full repack for optimal deltas
    fsck[10362] 2022-05-09 01:00:06,701 - INFO -    repack: repacking with "-n --window-memory=1g -a -b --unpack-unreachable=yesterday -f --pack-kept-objects -d"
    fsck[10362] 2022-05-09 03:19:15,825 - INFO -     graph: generating commit-graph
    fsck[10362] 2022-05-09 03:19:20,830 - INFO -  packrefs: repacking all refs
    fsck[10362] 2022-05-09 03:19:20,842 - INFO -     prune: pruning
    fsck[10362] 2022-05-09 03:19:21,622 - CRITICAL - /var/lib/gitolite3/repositories/poky-buildhistory.git reports errors:
    fsck[10362] 2022-05-09 03:19:21,625 - CRITICAL -        fatal: bad tree object ace77888c63e5c4e545f1bd7a3ee5934e35f56e9
    fsck[10362] 2022-05-09 03:19:21,626 - WARNING - Repacking /var/lib/gitolite3/repositories/poky-buildhistory.git was unsuccessful
The tree object in question came in during the repack stage:
    2022-05-09.02:36:33     11129   update  poky-buildhistory buildhistory    W       refs/heads/poky/master/qemuppc  5aad6c8130370bf22f5639162bbbfeaefd0fcd5e ea4e65d72a6161fece5734f7b111e31af77c7578        refs/.*

As far as I know, the maintenance steps we are running shouldn't result in any missing objects, so I'm curious if it's something we're doing wrong (using unsafe flags) or if git isn't properly accounting for some objects that come in during the repack stage. We're seeing this happen fairly routinely, so it's not just a random fluke.

Git version 2.36.1 (and earlier).

Thanks, Konstantin

Next: Taylor Blau
Message 1 of 7 in “Repository corruption if objects pushed in the middle of repack”
  1. Konstantin RyabitsevJun 13, 2022
  2. Taylor BlauJun 13, 2022
  3. Taylor BlauJun 13, 2022
  4. Konstantin RyabitsevJun 13, 2022
  5. Taylor BlauJun 13, 2022
  6. Konstantin RyabitsevJun 13, 2022
  7. Chris TorekJun 13, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.