{"thread":{"id":"34200","subject":"Fixing the git-repack replacement gap?","startedAt":"2013-06-18T16:52:50Z","lastAt":"2013-06-18T17:17:24Z","messageCount":2,"participants":["Martin Fick","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"221260","messageId":"201306181052.50490.mfick@codeaurora.org","threadId":"34200","inReplyTo":null,"subject":"Fixing the git-repack replacement gap?","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2013-06-18T16:52:50Z","receivedAt":"2013-06-18T16:52:50Z","isPatch":false,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"I have been trying to think of ways to fix git-repack so \nthat it no longer momentarily makes the objects in a repo \ninaccessible to all processes when it replaces packfiles \nwith the same objects in them as an already existing pack \nfile.  To be more explicit, I am talking about the way it \nmoves the existing pack file (and index) to old-<sha1>.pack \nbefore moving the new packfile in place.  During this moment \nin time the objects in that packfile are simply not \navailable to anyone using the repo.  This can be \nparticularly problematic for busy servers.\n\nThere likely are at lest 2 ways that the fundamental design \nof packfiles, their indexes, and their names have led to \nthis issue.  If the packfile and index were stored in a \nsingle file, they could have been replaced atomically and \nthus it would potentially avoid the issue of them being \ntemporarily inaccessible (although admittedly that might not \nwork anyway on some filesystems).  Alternatively, if the \npack file were named after the sha1 of the packed contents \nof the file instead of the sha1 of the objects in the sha1, \nthen the replacement would never need to happen since it \nmakes no sense to replace a file with another file with the \nexact same contents (unless, of course the first one is \ncorrupt, but then you aren't likely making the repo \ntemporarily worse, you are fixing a broken repo).\n\nI suspect these 2 ideas have been discussed before, but \nsince they are fundamental changes to the way pack files \nwork (and thus would not be backwards compatible), they are \nnot likely to get implemented soon.  This got me wondering \nif there wasn't an easier backwards compatible solution to \navoid making the objects inaccessible?\n\nIt seems like the problem could be avoided if we could \nsimply change the name of the pack file when a replacement \nwould be needed?  Of course, if we just changed the name, \nthen the name would not match the sha1 of the contained \nobjects and would likely be considered bad by git?  So, what \nif we could simply add a dummy object to the file to cause \nit to deserve a name change?\n\nSo the idea would be, have git-repack detect the conflict in \nfilenames and have it repack the new file with an additional \ndummy (unused) object in it, and then deliver the new file \nwhich no longer conflicts.  Would this be possible?  If so, \nwhat sort of other problems would this cause?  It would \nlikely cause an unreferenced object and likely cause it to \nwant to get pruned by the next git-repack?  Is that OK, \nmaybe you want it to get pruned because then the pack file \nwill get repacked once again without the dummy object later \nand avoid the temporarily inaccessible period for objects in \nthe file?  \n\nHmm, but then maybe that could even be done in a single git-\nrepack run (at the expense of extra disk space)?  \n\n1) Detect the conflict, \n2) Save the replacement file \n3) Create a new packfile with the dummy object\n4) Put the new file with the dummy object into service\n5) Remove the old conflicting file (no gap)\n6) Place the new conflicting file in service (no dummy)\n7) Remove the new file with dummy object (no gap again)\n\ndone?  Would it work?\n\nIf so, is there an easy way to create the dummy file?  Can \nany object simply be added at the end of a pack file after \nthe fact (and then added to the index too)?  Also, what \nshould the dummy object be?  Is there some sort of null \nobject that would be tiny and that would never already be in \nthe pack?\n\nThanks for any thoughts,\n\n-Martin\n"},{"id":"221272","messageId":"7v8v275oez.fsf@alter.siamese.dyndns.org","threadId":"34200","inReplyTo":"201306181052.50490.mfick@codeaurora.org","subject":"Re: Fixing the git-repack replacement gap?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-06-18T17:17:24Z","receivedAt":"2013-06-18T17:17:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Fick <mfick@codeaurora.org> writes:\n\n> ... So, what \n> if we could simply add a dummy object to the file to cause \n> it to deserve a name change?\n>\n> So the idea would be, have git-repack detect the conflict in \n> filenames and have it repack the new file with an additional \n> dummy (unused) object in it, and then deliver the new file \n> which no longer conflicts.  Would this be possible?\n\nSounds like a fun exercise.  I do not think it breaks anything, and\nbecause we have the list of objects to be placed in the resulting\npack fairly early in the process, this sequence would be possible:\n\n    (1) enumerate the objects;\n    (2) compute the resulting packname;\n    (3) notice it is the same as an existing one;\n    (4) add another dummy object and go back to (2);\n    (5) do the heavy-lifting of delitify;\n    (6) write out the resulting pack.\n\ninside pack-objects.\n\nI do not know if the loop between (2) and (4) is the only necessary\nthing to completely avoid the race you are worrying about, though.\n"}]}