{"thread":{"id":"40586","subject":"problem with modification time of packfiles","startedAt":"2015-10-18T21:37:55Z","lastAt":"2015-10-19T23:52:30Z","messageCount":5,"participants":["Andreas Amann","brian m. carlson","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"271885","messageId":"87io63503w.fsf@msstf091.ucc.ie","threadId":"40586","inReplyTo":null,"subject":"problem with modification time of packfiles","fromName":"Andreas Amann","fromEmail":"a.amann@ucc.ie","sentAt":"2015-10-18T21:37:55Z","receivedAt":"2015-10-18T21:37:55Z","isPatch":false,"sender":{"key":"a.amann@ucc.ie","avatar":null},"body":"git (2.6.1) sometimes updates the modification time of a packfile, even if it\nhas not changed at all.\n\nOn my system this triggers quite expensive an d unnecessary backup\noperations, which I would prefer to avoid. Is there a simple way to\nkeep the mtime of packfiles fixed, once they are created?\n\nApparently the undesired mtime update is done in\nsha1_file.c:freshen_file() which is called (indirectly) by\nwrite_sha1_file(). However I did not understand, why this is done.\n\nAny clarification and pointers, how mtime can be kept constant would be\nappreciated.\n\nThanks,\nAndreas\n"},{"id":"271888","messageId":"20151019025727.GA23404@vauxhall.crustytoothpaste.net","threadId":"40586","inReplyTo":"87io63503w.fsf@msstf091.ucc.ie","subject":"Re: problem with modification time of packfiles","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2015-10-19T02:57:28Z","receivedAt":"2015-10-19T02:57:28Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Oct 18, 2015 at 10:37:55PM +0100, Andreas Amann wrote:\n> git (2.6.1) sometimes updates the modification time of a packfile, even if it\n> has not changed at all.\n> \n> On my system this triggers quite expensive an d unnecessary backup\n> operations, which I would prefer to avoid. Is there a simple way to\n> keep the mtime of packfiles fixed, once they are created?\n> \n> Apparently the undesired mtime update is done in\n> sha1_file.c:freshen_file() which is called (indirectly) by\n> write_sha1_file(). However I did not understand, why this is done.\n> \n> Any clarification and pointers, how mtime can be kept constant would be\n> appreciated.\n\nThis is required to avoid deleting items that might still be needed.\nThe commit message for the commit that introduced that function is as\nfollows:\n\n write_sha1_file: freshen existing objects\n \n When we try to write a loose object file, we first check\n whether that object already exists. If so, we skip the\n write as an optimization. However, this can interfere with\n prune's strategy of using mtimes to mark files in progress.\n \n For example, if a branch contains a particular tree object\n and is deleted, that tree object may become unreachable, and\n have an old mtime. If a new operation then tries to write\n the same tree, this ends up as a noop; we notice we\n already have the object and do nothing. A prune running\n simultaneously with this operation will see the object as\n old, and may delete it.\n \n We can solve this by \"freshening\" objects that we avoid\n writing by updating their mtime. The algorithm for doing so\n is essentially the same as that of has_sha1_file. Therefore\n we provide a new (static) interface \"check_and_freshen\",\n which finds and optionally freshens the object. It's trivial\n to implement freshening and simple checking by tweaking a\n single parameter.\n \n Signed-off-by: Jeff King \n Signed-off-by: Junio C Hamano \n\nPerhaps implementing a backup strategy based on content instead of mtime\nwould be more successful.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"271949","messageId":"87a8re4oks.fsf@msstf091.ucc.ie","threadId":"40586","inReplyTo":"20151019025727.GA23404@vauxhall.crustytoothpaste.net","subject":"Re: problem with modification time of packfiles","fromName":"Andreas Amann","fromEmail":"a.amann@ucc.ie","sentAt":"2015-10-19T19:59:15Z","receivedAt":"2015-10-19T19:59:15Z","isPatch":false,"sender":{"key":"a.amann@ucc.ie","avatar":null},"body":"\"brian m. carlson\" writes:\n> On Sun, Oct 18, 2015 at 10:37:55PM +0100, Andreas Amann wrote:\n>> git (2.6.1) sometimes updates the modification time of a packfile, even if it\n>> has not changed at all.\n>> \n>> On my system this triggers quite expensive an d unnecessary backup\n>> operations, which I would prefer to avoid. Is there a simple way to\n>> keep the mtime of packfiles fixed, once they are created?\n>> \n>> Apparently the undesired mtime update is done in\n>> sha1_file.c:freshen_file() which is called (indirectly) by\n>> write_sha1_file(). However I did not understand, why this is done.\n>> \n>> Any clarification and pointers, how mtime can be kept constant would be\n>> appreciated.\n>\n> This is required to avoid deleting items that might still be needed.\n> The commit message for the commit that introduced that function is as\n> follows:\n>\n> write_sha1_file: freshen existing objects\n> \n> When we try to write a loose object file, we first check\n> whether that object already exists. If so, we skip the\n> write as an optimization. However, this can interfere with\n> prune's strategy of using mtimes to mark files in progress.\n> \n\nThank you for your answer. However, this reasoning only applies to loose\nobjects and not packfiles.\n\nMy understanding is that \"git prune\" will not prune any pack files\n(except those starting with tmp_). Only \"git repack\" should do that.\nRepack seems to be however mtime agnostic and therefore it does not seem\nto be necessary to freshen packfiles.\n\nIt therefore seems that git freshens packfiles unnecessarily, which can\nlead to expensive and unnecessary backup operations. \n\nGiven this, would a trivial patch to remove the freshening of packfiles\nbe acceptable?\n\nAlternatively, maybe it would be preferrable to use ctime instead of\nmtime to mark recently used packfiles and loose objects? This might be\nmore natural, as mtime is usually associated with a \"modification\" of\nthe file itself, which does not happen here. ctime on the other hand\nindicates attribute changes. (Instead of utime() in this case chmod()\ncould be used to update ctime.)\n"},{"id":"271955","messageId":"20151019230919.GA60184@vauxhall.crustytoothpaste.net","threadId":"40586","inReplyTo":"87a8re4oks.fsf@msstf091.ucc.ie","subject":"Re: problem with modification time of packfiles","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2015-10-19T23:09:19Z","receivedAt":"2015-10-19T23:09:19Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Oct 19, 2015 at 08:59:15PM +0100, Andreas Amann wrote:\n> Thank you for your answer. However, this reasoning only applies to loose\n> objects and not packfiles.\n> \n> My understanding is that \"git prune\" will not prune any pack files\n> (except those starting with tmp_). Only \"git repack\" should do that.\n> Repack seems to be however mtime agnostic and therefore it does not seem\n> to be necessary to freshen packfiles.\n> \n> It therefore seems that git freshens packfiles unnecessarily, which can\n> lead to expensive and unnecessary backup operations.\n> \n> Given this, would a trivial patch to remove the freshening of packfiles\n> be acceptable?\n\nI'm not familiar enough with the code to say for certain, but it looks\nlike you're right. Peff, Junio, do you think this is safe, or is there\nsomething we're missing?\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"271957","messageId":"20151019235230.GA14800@sigill.intra.peff.net","threadId":"40586","inReplyTo":"20151019230919.GA60184@vauxhall.crustytoothpaste.net","subject":"Re: problem with modification time of packfiles","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-10-19T23:52:30Z","receivedAt":"2015-10-19T23:52:30Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 19, 2015 at 11:09:19PM +0000, brian m. carlson wrote:\n\n> On Mon, Oct 19, 2015 at 08:59:15PM +0100, Andreas Amann wrote:\n> > Thank you for your answer. However, this reasoning only applies to loose\n> > objects and not packfiles.\n> > \n> > My understanding is that \"git prune\" will not prune any pack files\n> > (except those starting with tmp_). Only \"git repack\" should do that.\n> > Repack seems to be however mtime agnostic and therefore it does not seem\n> > to be necessary to freshen packfiles.\n> > \n> > It therefore seems that git freshens packfiles unnecessarily, which can\n> > lead to expensive and unnecessary backup operations.\n> > \n> > Given this, would a trivial patch to remove the freshening of packfiles\n> > be acceptable?\n> \n> I'm not familiar enough with the code to say for certain, but it looks\n> like you're right. Peff, Junio, do you think this is safe, or is there\n> something we're missing?\n\nNo, it's not safe. When doing a full repack, we pack only reachable\nobjects. Unreachable ones are either loosened and given the mtime of the\npackfile (from which they can then be pruned), or discarded if the pack\nmtime is already old (as an optimization to avoid writing and then\nimmediately pruning).\n\nSee builtin/pack-objects.c:loosen_unused_packed_objects.\n\n-Peff\n"}]}