auto packing with simultaneous pushes: "error: Could not stat 'objects/[…]/[…]'"

Ivan D Vasin· Jan 23, 2013, 23:50 UTC · lore

my coworker and i just pushed some commits seconds apart from each other. both of our pushes triggered a server-side auto pack (presumably as part of ``receive-pack``, as a result of the setting ``receive.autogc = true``). each auto pack produced multiple errors of the form ``error: Could not stat 'objects/[…]/[…]'``. there appears to be no overlap between the two auto packs' lists of failed object hashes.

my suspicion is that the two auto pack processes each generated a list of objects to pack, then proceeded to pack them, removing the corresponding files as they went along. as each one encountered an entry in its list that was previously handled by the other process, it found that it was unable to stat the corresponding file because it had been removed by the other process.

my suggestion is that an auto pack should lock the repository, preventing at least other auto packs (and perhaps other operations) from running until it is finished. alternatively, the auto pack can skip over objects that were already packed by another process; however, i imagine this would be much less efficient if done correctly.

here is the console output from my push:

$ git push
X11 forwarding request failed on channel 0
Counting objects: 32, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 2.70 KiB, done.
Total 18 (delta 12), reused 0 (delta 0)
Auto packing the repository for optimum performance.
error: Could not stat 'objects/00/
c706ccf0ef06092134ccb1f9fdf0e7d39a9d5f'
error: Could not stat 'objects/02/7c2932047c9c9c7bacd6e1abdc3810866161bc'
error: Could not stat 'objects/13/b2c3f0a55e8de099553cad6901a150f19b83c2'
error: Could not stat 'objects/2a/3992fa9c096f7514a62781a3c3923a996a7073'
error: Could not stat 'objects/3d/cb6e9055b9b51911aae52f4bb0ca8d8719d645'
error: Could not stat 'objects/50/c2f6d8e0de1eba988256597cc1d6e1a387902c'
error: Could not stat 'objects/50/9f494f6b60741957a45cf51f8d097f0578c89a'
error: Could not stat 'objects/52/6597c12d730c5903d46f598c6ec438cfabfe0f'
error: Could not stat 'objects/52/a6b4ee4bebb043d21858455fd0cd33b4812a42'
error: Could not stat 'objects/53/71c6f99023efecb3a4c16daeeee63f534d63b2'
error: Could not stat 'objects/5f/bd49df3c4abdc93c7118f762ac770a0fcd6ce4'
error: Could not stat 'objects/67/900e6e598e8e1c7bde5776fd220f7daf147cde'
error: Could not stat 'objects/72/42ecd310d0286da6c3967985c6446cf105b0bd'
error: Could not stat 'objects/79/4afecb2d348dde5ad4d24cf62c8fcd4db3b63a'
error: Could not stat 'objects/7e/4c612c51c5d76f03f12c04241d3b3c5aa212e4'
error: Could not stat 'objects/84/fd1e038d4345bac1fa2b33edb73fd73813ac4f'
error: Could not stat 'objects/89/c32fdbf3400c797d01659f9ee644aa8a713574'
error: Could not stat 'objects/8b/cbe10b0d09325b49876416341d6cf4bccc9e35'
error: Could not stat 'objects/8d/6dcc95d6bea57cfafe8a08fed7947561682b68'
error: Could not stat 'objects/b9/67af0f23deb87f1708689d159ee6444503462a'
error: Could not stat 'objects/bb/9d6fafe3ba2189582ac48088ee6f732755dddb'
error: Could not stat 'objects/e0/549c20e56d6ec270ec3ffee6c93a6d6cb8f6b5'
error: Could not stat 'objects/f2/737f0d8635265a56d57c27a3ddf155da48b07a'
error: Could not stat 'objects/fc/8b2f0837aefa21d86de65f6230c86f64addb8d'
error: Could not stat 'objects/fc/ac21d190770fdee23e5e69cdba36eefd6e001a'
error: Could not stat 'objects/fd/ae521d6a3404ceabff223385d7b5b5c783f0fe'
To git@code.adverplex.com:syseng
   fb49a86..e161cf5  HEAD -> uber_release_alpha

here is the console output from my coworker's push:

$ git push
Counting objects: 181, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (154/154), done.
Writing objects: 100% (159/159), 1.81 MiB, done.
Total 159 (delta 74), reused 0 (delta 0)
Auto packing the repository for optimum performance.
error: Could not stat 'objects/06/b2aee0b4adca015d072be4ef6a655c118b6af3'
error: Could not stat 'objects/0d/66f41003c9112b8093dfee5fd56e8db25a142b'
error: Could not stat 'objects/0d/6c3e296fb7b83998677d207f9df6a69e144168'
error: Could not stat 'objects/14/64e2bd863a15e08be7aa2f33b8df5b5f695608'
error: Could not stat 'objects/14/19cad4e2a38720186eb36b3c756d05e1963877'
error: Could not stat 'objects/16/1b9e65004a9120ea250ef8ef8292d5416ed984'
error: Could not stat 'objects/23/b6bc50c16fe3740393c8a08cf57cc07a56c778'
error: Could not stat 'objects/60/1d7c562cafaf6c71c57dbbd0bab55c3698d0c7'
error: Could not stat 'objects/6c/c5c7ededba86ce291b531def840e757ce3eccc'
error: Could not stat 'objects/74/8d6bece2b4ccec41584d22152d0fd903e01908'
error: Could not stat 'objects/9a/6c21779e7c063b87bbf055e4c6b6baecc8a3e1'
error: Could not stat 'objects/9a/6519e7bb625f681ead373e9a9226f862ee1550'
error: Could not stat 'objects/9b/7027422cbce1f5f2c5929ae1313ecee2cd3537'
error: Could not stat 'objects/a4/7e0b5906c2b552158a1e5943063e983532988f'
error: Could not stat 'objects/a4/6db00ea7b7d2e618731d7b4f1e59b5f0276aff'
error: Could not stat 'objects/ab/b3a8fd9585af5a7f6dbd609e6b4ed8c4359ca6'
error: Could not stat 'objects/ad/0c98f2674807beee093cead1179d95756c1788'
error: Could not stat 'objects/bf/4fd1b9cc617d3fc832350783064418e0d7fdfd'
error: Could not stat 'objects/bf/c65d6bc705dfc5018c3d25228a4469d069f3b8'
error: Could not stat 'objects/c0/adb5b39efd31698789ee8aefcedf91913fc7f1'
error: Could not stat 'objects/ca/f4ee60f223039d2ec6185adbb4c28ab7df461e'
error: Could not stat 'objects/ca/e3b7c60b4b3a34bba4b4a5738edf0f7786a784'
error: Could not stat 'objects/cb/4a470752088b708290713b0a513e22bb0dcb15'
error: Could not stat 'objects/d3/d44ba75332e3e7e21a89eafe724e682cf124dd'
error: Could not stat 'objects/d3/8a635a420af938f680ab30c6252fdddd448a5d'
error: Could not stat 'objects/d8/f102045fc608b65cb1d75ad8de11a673ebd656'
error: Could not stat 'objects/da/97ac29d9024f589b4aec46bc0ea1f597b00802'
error: Could not stat 'objects/e5/3e9d05c4e9a3e336453af3b43fe06dfa5c6d57'
error: Could not stat 'objects/ea/b586bcc6925bcb34cdeede1562a851d5b58d90'
error: Could not stat 'objects/ec/f2cf1dd8d6045242676cd22320902c085ecbb3'
error: Could not stat 'objects/f3/819de13ed535bdd92580e2f2b01580714188e1'
To git@code.adverplex.com:syseng
   fb49a86..a39c897  master -> master

``git fsck`` is successful on both of our repos and on the bare repo to which we pushed.

Ivan D Vasin· Jan 24, 2013, 01:39 UTC · lore

Re: auto packing with simultaneous pushes: "error: Could not stat 'objects/[…]/[…]'"

On Wed, Jan 23, 2013 at 7:28 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 10 quoted lines

> Ivan D Vasin <nisavid@gmail.com> writes:
>
>> my suggestion is that an auto pack should lock the repository,
>> preventing at least other auto packs (and perhaps other operations)
>> ...
>>
>> ``git fsck`` is successful on both of our repos and on the bare repo
>> to which we pushed.
>
> Successful after you pushed, or before you pushed, or both?

both

Show 9 quoted lines

>
> I suspect both.
>
> I do not think such a lock is necessary for correctness of the
> operation, but running two auto packing sumultaneously is wasteful,
> so it would help performance.  But that would produce a larger
> problem.  What if your modified auto-packer takes a lock and then
> dies without relinquishing the lock?  The repository will never be
> repacked after such an event forever?

perhaps the lock could contain the PID of the auto pack process. if that PID has gone away, the lock is ignored and replaced with a new one.

that's what comes to my mind. of course, there could be other ways to handle this that i'm not thinking of.

in any case, the error messages, though spurious, are alarming to the uninformed user. it looks like Git is saying that there is actual data loss, where in fact there is none. if Git doesn't prevent these messages from appearing (via locking behavior or otherwise), then it should at least annotate them with a message that describes their possibly spurious nature and perhaps instructs the user to verify everything with ``git fsck``.