{"thread":{"id":"40557","subject":"repository corruption when pushing commits to a repository running 'git gc --prune='","startedAt":"2015-10-14T13:44:59Z","lastAt":"2015-10-14T21:39:11Z","messageCount":10,"participants":["Jan Smets","Matthieu Moy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"271645","messageId":"561E5C5B.3090207@alcatel-lucent.com","threadId":"40557","inReplyTo":null,"subject":"repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Jan Smets","fromEmail":"jan.smets@alcatel-lucent.com","sentAt":"2015-10-14T13:44:59Z","receivedAt":"2015-10-14T13:44:59Z","isPatch":false,"sender":{"key":"jan.smets@alcatel-lucent.com","avatar":null},"body":"Hi\n\nI've recently expired my reflog to prune loose objects. On a live, bare, \nrepository I ran 'git gc --prune=now'\n\nAll clients ended up having problems, they would report:\n error: refs/heads/master does not point to a valid object!\nRunning 'git log' on the bare repo gave : fatal: bad object HEAD\n\n\n\nSo I did a couple of tries reproducing the issue and with this little \ntest I got a couple of interesting different/similar outcomes\n\nA: mkdir -p /tmp/test; cd /tmp/test ; git init --bare bare ;\nB: cd /tmp/test ; git clone bare 1 ; cd 1; touch test; git add test; \nwhile true; do date >> test ; git commit -m \"$(date)\" test ; git push || \nbreak; done\nA: cd bare; git gc --prune=now (or run it in a loop)\n\n\n1)\n\nfatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69\nfatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69\nTo /tmp/test/bare\n ! [remote rejected] master -> master (missing necessary objects)\n\n2)\n\nremote: error: unable to write sha1 filename \nobjects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed: No such file or directory\nremote: fatal: failed to write object\nerror: unpack failed: unpack-objects abnormal exit\nTo /tmp/test/bare\n ! [remote rejected] master -> master (unpacker error)\n\n3)\n\nerror: Ref refs/heads/master is at \ne992810e70949e797d33041bf6bc961c9fa4f3e5 but expected \n0000000000000000000000000000000000000000\nremote: error: Cannot lock ref 'refs/heads/master':\nTo /tmp/test/bare\n ! [remote rejected] master -> master (failed to update ref)\n\n4)\n\nremote: error: cannot lock ref 'refs/heads/master': Unable to create \n'/tmp/test/bare/refs/heads/master.lock': File exists.\nremote:\nremote: If no other git process is currently running, this probably means a\nremote: git process crashed in this repository earlier. Make sure no \nother git\nremote: process is running and remove the file manually to continue.\nTo /tmp/test/bare\n ! [remote rejected] master -> master (failed to update ref)\nerror: failed to push some refs to '/tmp/test/bare'\n\n\nAnd eventually running 'git gc --prune=now/all' on the bare repository \ncould end with this message:\n\n$ git gc --prune=all\nerror: refs/heads/master does not point to a valid object!\nfatal: bad object refs/heads/master\nerror: failed to run repack\n$ git log -1\nfatal: bad object HEAD\n\n\nThis behaviour has been observed with version 2.4.4, 2.5.[12] and 2.6.1\n\nThank you\n\n- Jan\n"},{"id":"271647","messageId":"vpq4mhtbjma.fsf@grenoble-inp.fr","threadId":"40557","inReplyTo":"561E5C5B.3090207@alcatel-lucent.com","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2015-10-14T14:38:37Z","receivedAt":"2015-10-14T14:38:37Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jan Smets writes:\n\n> Hi\n>\n> I've recently expired my reflog to prune loose objects. On a live,\n> bare, repository I ran 'git gc --prune=now'\n>\n> All clients ended up having problems, they would report:\n> error: refs/heads/master does not point to a valid object!\n> Running 'git log' on the bare repo gave : fatal: bad object HEAD\n[...]\n> fatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69\n> fatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69\n> To /tmp/test/bare\n> ! [remote rejected] master -> master (missing necessary objects)\n\nI think this is the expected behavior. push will create new objects that\nare not referenced until the ref is updated (at the very end). prune can\nrun concurrently since creating and deleting objects is done in a\nlockless way (only the ref update needs a lock).\n\nStill, this is not the *documented* behavior, and an easy way to corrupt\na repo should be very explicitly documented as very dangerous, and the\nprecautions to take when using it should be explained clearly.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"271654","messageId":"xmqqoag18i6w.fsf@gitster.mtv.corp.google.com","threadId":"40557","inReplyTo":"vpq4mhtbjma.fsf@grenoble-inp.fr","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-14T17:37:43Z","receivedAt":"2015-10-14T17:37:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> I think this is the expected behavior. push will create new objects that\n> are not referenced until the ref is updated (at the very end). prune can\n> run concurrently since creating and deleting objects is done in a\n> lockless way (only the ref update needs a lock).\n>\n> Still, this is not the *documented* behavior, and an easy way to corrupt\n> a repo should be very explicitly documented as very dangerous, and the\n> precautions to take when using it should be explained clearly.\n\nYeah, I think this paragraph in the user-manual is the only thing\nthat mentions it:\n\n Anyway, once you are sure that you're not interested in any dangling\n state, you can just prune all unreachable objects:\n\n ------------------------------------------------\n $ git prune\n ------------------------------------------------\n\n and they'll be gone. (You should only run `git prune` on a quiescent\n repository--it's kind of like doing a filesystem fsck recovery: you\n don't want to do that while the filesystem is mounted.\n\nSomething along this line, perhaps?\n\n Documentation/git-gc.txt | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 5223498..fa15104 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -63,8 +63,11 @@ automatic consolidation of packs.\n --prune=::\n \tPrune loose objects older than date (default is 2 weeks ago,\n \toverridable by the config variable `gc.pruneExpire`).\n-\t--prune=all prunes loose objects regardless of their age.\n-\t--prune is on by default.\n+\t--prune=all prunes loose objects regardless of their age (do\n+\tnot use --prune=all unless you know exactly what you are doing.\n+\tUnless the repository is quiescent, you will lose newly created\n+\tobjects that haven't been anchored with the refs and end up\n+\tcorrupting your repository). --prune is on by default.\n \n --no-prune::\n \tDo not prune any loose objects.\n"},{"id":"271656","messageId":"vpqmvvl8fb1.fsf@grenoble-inp.fr","threadId":"40557","inReplyTo":"xmqqoag18i6w.fsf@gitster.mtv.corp.google.com","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2015-10-14T18:40:02Z","receivedAt":"2015-10-14T18:40:02Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n> index 5223498..fa15104 100644\n> --- a/Documentation/git-gc.txt\n> +++ b/Documentation/git-gc.txt\n> @@ -63,8 +63,11 @@ automatic consolidation of packs.\n> --prune=::\n> \tPrune loose objects older than date (default is 2 weeks ago,\n> \toverridable by the config variable `gc.pruneExpire`).\n> -\t--prune=all prunes loose objects regardless of their age.\n> -\t--prune is on by default.\n> +\t--prune=all prunes loose objects regardless of their age (do\n> +\tnot use --prune=all unless you know exactly what you are doing.\n> +\tUnless the repository is quiescent, you will lose newly created\n> +\tobjects that haven't been anchored with the refs and end up\n> +\tcorrupting your repository). --prune is on by default.\n\nLooks good to me. I think the same should be added to git-prune.txt\nunder --expire.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"271657","messageId":"vpqbnc18ezo.fsf@grenoble-inp.fr","threadId":"40557","inReplyTo":"561E5C5B.3090207@alcatel-lucent.com","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2015-10-14T18:46:51Z","receivedAt":"2015-10-14T18:46:51Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jan Smets writes:\n\n> 2)\n>\n> remote: error: unable to write sha1 filename\n> objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed: No such file or\n> directory\n> remote: fatal: failed to write object\n\nIf I read correctly, this happens when we move the temp file to its\npermanent position.\n\nThis looks like a race between deleting objects/05/ and creating\nobjects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed.\n\nI don't understand how this is possible, since the temporary file and\nthe final one are in the same directory, so deleting the directory\nshould fail.\n\nWhat am I missing?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"271658","messageId":"xmqq7fmp8bcf.fsf@gitster.mtv.corp.google.com","threadId":"40557","inReplyTo":"vpqbnc18ezo.fsf@grenoble-inp.fr","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-14T20:05:36Z","receivedAt":"2015-10-14T20:05:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> This looks like a race between deleting objects/05/ and creating\n> objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed.\n>\n> I don't understand how this is possible, since the temporary file and\n> the final one are in the same directory, so deleting the directory\n> should fail.\n>\n> What am I missing?\n\n 1. tmp_obj_* gets created in objects/05/ by \"git unpack-objects\";\n\n 2. prune_cruft() comes, notices tmp_obj_* that is sufficiently old,\n decides to remove in \"git prune\".\n\n 3. prune_subdir() comes, notices objects/05/ is now empty, does\n rmdir().\n\n 4. \"git unpack-objects\" closes the file whose inode was originally\n associated with the path objects/05/tmp_obj_*;\n\n 5. \"git unpack-objects\" calls finalize_object_file() and tries to\n link(2) or rename(2) it to its final name. It does not find the\n source of the link or rename and says \"oops, no such file\".\n"},{"id":"271659","messageId":"vpq37xd8b7f.fsf@grenoble-inp.fr","threadId":"40557","inReplyTo":"xmqq7fmp8bcf.fsf@gitster.mtv.corp.google.com","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2015-10-14T20:08:36Z","receivedAt":"2015-10-14T20:08:36Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> Matthieu Moy writes:\n>\n>> This looks like a race between deleting objects/05/ and creating\n>> objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed.\n>>\n>> I don't understand how this is possible, since the temporary file and\n>> the final one are in the same directory, so deleting the directory\n>> should fail.\n>>\n>> What am I missing?\n>\n> 1. tmp_obj_* gets created in objects/05/ by \"git unpack-objects\";\n>\n> 2. prune_cruft() comes, notices tmp_obj_* that is sufficiently old,\n> decides to remove in \"git prune\".\n\nAh, OK, the real problem is the temporary file removal. Thanks.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"271693","messageId":"xmqqtwpt6usi.fsf@gitster.mtv.corp.google.com","threadId":"40557","inReplyTo":"vpqmvvl8fb1.fsf@grenoble-inp.fr","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-14T20:48:29Z","receivedAt":"2015-10-14T20:48:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> Looks good to me. I think the same should be added to git-prune.txt\n> under --expire.\n\nI thought about it, but decided against it, as the command is not\neven recommended to end users.\n"},{"id":"271698","messageId":"vpqtwpt6tsv.fsf@grenoble-inp.fr","threadId":"40557","inReplyTo":"xmqqtwpt6usi.fsf@gitster.mtv.corp.google.com","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2015-10-14T21:09:52Z","receivedAt":"2015-10-14T21:09:52Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> Matthieu Moy writes:\n>\n>> Looks good to me. I think the same should be added to git-prune.txt\n>> under --expire.\n>\n> I thought about it, but decided against it, as the command is not\n> even recommended to end users.\n\nEven non-\"end users\" can shoot themselves in the foot. Someone writting\na script using \"git prune\" may actually shoot in other people's foot ...\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"271701","messageId":"xmqqfv1d6sg0.fsf@gitster.mtv.corp.google.com","threadId":"40557","inReplyTo":"vpqtwpt6tsv.fsf@grenoble-inp.fr","subject":"Re: repository corruption when pushing commits to a repository running 'git gc --prune='","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-14T21:39:11Z","receivedAt":"2015-10-14T21:39:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> Junio C Hamano writes:\n>\n>> Matthieu Moy writes:\n>>\n>>> Looks good to me. I think the same should be added to git-prune.txt\n>>> under --expire.\n>>\n>> I thought about it, but decided against it, as the command is not\n>> even recommended to end users.\n>\n> Even non-\"end users\" can shoot themselves in the foot. Someone writting\n> a script using \"git prune\" may actually shoot in other people's foot ...\n\nSure, patches welcome.\n\nTo be honest, I felt that\n\n --expire