{"thread":{"id":"13490","subject":"Why repository grows after \"git gc\"? / Purpose of *.keep files?","startedAt":"2008-05-12T12:29:00Z","lastAt":"2008-05-14T23:24:01Z","messageCount":35,"participants":["Teemu Likonen","Johannes Schindelin","David Tweed","Nicolas Pitre","Govind Salinas","Mike Hommey","Shawn O. Pearce","Junio C Hamano","Paolo Bonzini","Stephen R. van den Berg","Juergen Ruehle","Linus Torvalds","A Large Angry SCM"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"76699","messageId":"20080512122900.GA13050@mithlond.arda.local","threadId":"13490","inReplyTo":null,"subject":"Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-12T12:29:00Z","receivedAt":"2008-05-12T12:29:00Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"I have noticed that after cloning a repository (via git protocol) the\nrepo is packed pretty tightly and takes relatively small amount of disk\nspace. After using it a while and running \"git gc\" the repo sometimes\ngrows 25% or something like that.\n\nFor testing purposes I deleted objects/pack/*.keep file(s) and ran \"git\ngc\" again. The repo resulted in small again, just like after the initial\nclone. I don't have disk space problems but a repo growing about 25%\nafter manual \"git gc\" seems weird. What's the purpose of these *.keep\nfiles? They just contain text like \"fetch-pack on \".\n\n\nPS. I have merged Brandon Casey's new git-gc/repack patches. In case it\n has some effect. See the \"pu\" branch or \"git log 9e7d5019\".\n"},{"id":"76721","messageId":"20080512155243.GA3592@mithlond.arda.local","threadId":"13490","inReplyTo":"20080512122900.GA13050@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-12T15:52:43Z","receivedAt":"2008-05-12T15:52:43Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"Teemu Likonen wrote (2008-05-12 15:29 +0300):\n\n> For testing purposes I deleted objects/pack/*.keep file(s) and ran\n> \"git gc\" again. The repo resulted in small again, just like after the\n> initial clone.\n\nAfter playing with test repo a while it seems that \"git gc\" never\ntouches pack files which have accompanying .keep file around. (And it's\ncommon to have a .keep file after \"git clone\".) This makes gc perform\nfaster. A side effect seems to be that objects which later become\nunreferenced in those pack-files-with-.keep are never pruned. *.keep\nfiles also seem to prevent from really aggressively optimizing the\nrepository's size.\n\nProbably a crazy idea: What if \"gc --aggressive\" first removed *.keep\nfiles and after packing and garbage-collecting and whatever it does it\nwould add a .keep file for the newly created pack?\n"},{"id":"76725","messageId":"alpine.DEB.1.00.0805121810501.30431@racer","threadId":"13490","inReplyTo":"20080512155243.GA3592@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-05-12T17:13:17Z","receivedAt":"2008-05-12T17:13:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 12 May 2008, Teemu Likonen wrote:\n\n> Probably a crazy idea: What if \"gc --aggressive\" first removed *.keep \n> files and after packing and garbage-collecting and whatever it does it \n> would add a .keep file for the newly created pack?\n\nMost .keep files are not meant to be removed by git-gc. Usually, .keep \nfiles are only created interactively (if you _want_ to keep a pack, e.g. \nwhen it has been optimally packed and is big), or by git-index-pack while \nit is writing a pack (IIRC).\n\nSo I think it would be wrong for \"gc --aggressive\" to remove the .keep \nfiles.\n\nCiao,\nDscho\n"},{"id":"76726","messageId":"e1dab3980805121017u4c244d25s76b39cf015f6c5c5@mail.gmail.com","threadId":"13490","inReplyTo":"20080512155243.GA3592@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"David Tweed","fromEmail":"david.tweed@gmail.com","sentAt":"2008-05-12T17:17:05Z","receivedAt":"2008-05-12T17:17:05Z","isPatch":false,"sender":{"key":"david.tweed@gmail.com","avatar":null},"body":"On Mon, May 12, 2008 at 4:52 PM, Teemu Likonen wrote:\n> Teemu Likonen wrote (2008-05-12 15:29 +0300):\n> Probably a crazy idea: What if \"gc --aggressive\" first removed *.keep\n> files and after packing and garbage-collecting and whatever it does it\n> would add a .keep file for the newly created pack?\n\nMy understanding is that the repacking with -a redoes the computation\nto repack ALL the objects in every pack and loose objects, whereas\nwhat would be preferred is to try to delta new objects (loose and\npacked) against the existing .keep pack (extending it with the new\nobjects) but not trying to re-deltify objects in the .keep pack. This\nis because .keep files are primarily for those who are cloning onto a\nmachine that isn't powerful (maybe even a laptop/palmtop) but who are\ncloning from a powerful server, so that you wouldn't necessarily want\nto apply your strategy unconditionally.\n\n-- \ncheers, dave tweed__________________________\ndavid.tweed@gmail.com\nRm 124, School of Systems Engineering, University of Reading.\n\"while having code so boring anyone can maintain it, use Python.\" --\nattempted insult seen on slashdot\n"},{"id":"76743","messageId":"20080512184334.GB5160@mithlond.arda.local","threadId":"13490","inReplyTo":"alpine.DEB.1.00.0805121810501.30431@racer","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-12T18:43:34Z","receivedAt":"2008-05-12T18:43:34Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"Johannes Schindelin wrote (2008-05-12 18:13 +0100):\n\n> On Mon, 12 May 2008, Teemu Likonen wrote:\n> \n> > Probably a crazy idea: What if \"gc --aggressive\" first removed\n> > *.keep files and after packing and garbage-collecting and whatever\n> > it does it would add a .keep file for the newly created pack?\n> \n> Most .keep files are not meant to be removed by git-gc. Usually,\n> .keep files are only created interactively (if you _want_ to keep\n> a pack, e.g. when it has been optimally packed and is big), or by\n> git-index-pack while it is writing a pack (IIRC).\n> \n> So I think it would be wrong for \"gc --aggressive\" to remove the .keep\n> files.\n\nI guess you're right. Maybe \"gc --aggressive\" could delete only certain\nmachine-generated .keep files which have an identifier inside?\n\nWell, I don't really have any problems with the current behaviour; it\njust feels a bit strange that, for example, Linus's kernel repository\ngrew about 90MB after just one update pull and gc. Also, dangling\nobjects are kept forever in .keep packs (which are created with \"git\nclone\", for example).\n"},{"id":"76749","messageId":"alpine.LFD.1.10.0805121453250.23581@xanadu.home","threadId":"13490","inReplyTo":"20080512184334.GB5160@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-12T18:56:28Z","receivedAt":"2008-05-12T18:56:28Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 May 2008, Teemu Likonen wrote:\n\n> Well, I don't really have any problems with the current behaviour; it\n> just feels a bit strange that, for example, Linus's kernel repository\n> grew about 90MB after just one update pull and gc.\n\nThat looks really odd. Sure the repo might grow a bit, but 90MB seems \nreally excessive. How many time did pass between the initial clone and \nthat subsequent pull?\n\n> Also, dangling\n> objects are kept forever in .keep packs (which are created with \"git\n> clone\", for example).\n\nA pack obtained via 'git clone' will never contain any dangling objects.\n\n\nNicolas\n"},{"id":"76753","messageId":"20080512190946.GC5160@mithlond.arda.local","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805121453250.23581@xanadu.home","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-12T19:09:46Z","receivedAt":"2008-05-12T19:09:46Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"Nicolas Pitre wrote (2008-05-12 14:56 -0400):\n\n> On Mon, 12 May 2008, Teemu Likonen wrote:\n> \n> > Well, I don't really have any problems with the current behaviour;\n> > it just feels a bit strange that, for example, Linus's kernel\n> > repository grew about 90MB after just one update pull and gc.\n> \n> That looks really odd. Sure the repo might grow a bit, but 90MB seems\n> really excessive. How many time did pass between the initial clone\n> and that subsequent pull?\n\nAs I used the kernel repo just for testing this behaviour in question\nI did both things today. Timestamps tell that there were six hours\nbetween the initial .keep pack and the new pack created by manual \"git\ngc\".\n\n> > Also, dangling objects are kept forever in .keep packs (which are\n> > created with \"git clone\", for example).\n> \n> A pack obtained via 'git clone' will never contain any dangling\n> objects.\n\nI think it can contain at some later point. For example, if a user first\nfetches all the branches but later decides to track only one branch.\nAfter deleting unneeded tracking branches and expiring the reflog\nthere'll be dangling objects in the original .keep pack created with\n\"git clone\".\n"},{"id":"76757","messageId":"alpine.LFD.1.10.0805121527550.23581@xanadu.home","threadId":"13490","inReplyTo":"20080512190946.GC5160@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-12T19:36:13Z","receivedAt":"2008-05-12T19:36:13Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 May 2008, Teemu Likonen wrote:\n\n> Nicolas Pitre wrote (2008-05-12 14:56 -0400):\n> \n> > On Mon, 12 May 2008, Teemu Likonen wrote:\n> > \n> > > Well, I don't really have any problems with the current behaviour;\n> > > it just feels a bit strange that, for example, Linus's kernel\n> > > repository grew about 90MB after just one update pull and gc.\n> > \n> > That looks really odd. Sure the repo might grow a bit, but 90MB seems\n> > really excessive. How many time did pass between the initial clone\n> > and that subsequent pull?\n> \n> As I used the kernel repo just for testing this behaviour in question\n> I did both things today. Timestamps tell that there were six hours\n> between the initial .keep pack and the new pack created by manual \"git\n> gc\".\n\nThis is way too big a difference. Something is going on.\n\nWhat git version is this? And can you send me the content of your \n.git/logs directory?\n\n> > > Also, dangling objects are kept forever in .keep packs (which are\n> > > created with \"git clone\", for example).\n> > \n> > A pack obtained via 'git clone' will never contain any dangling\n> > objects.\n> \n> I think it can contain at some later point. For example, if a user first\n> fetches all the branches but later decides to track only one branch.\n> After deleting unneeded tracking branches and expiring the reflog\n> there'll be dangling objects in the original .keep pack created with\n> \"git clone\".\n\nSure. But to decide to track only one branch and exclude the others \nrequire some higher level of git knowledge already. At that point if \nyou really care about top packing performances you certainly can deal \nwith the .keep file as well.\n\n\nNicolas\n"},{"id":"76760","messageId":"5d46db230805121310q5a0c6037p769ea091c26284d0@mail.gmail.com","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805121527550.23581@xanadu.home","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Govind Salinas","fromEmail":"govind@sophiasuchtig.com","sentAt":"2008-05-12T20:10:30Z","receivedAt":"2008-05-12T20:10:30Z","isPatch":false,"sender":{"key":"govind@sophiasuchtig.com","avatar":null},"body":"On Mon, May 12, 2008 at 2:36 PM, Nicolas Pitre wrote:\n> On Mon, 12 May 2008, Teemu Likonen wrote:\n>\n> > Nicolas Pitre wrote (2008-05-12 14:56 -0400):\n> >\n> > > On Mon, 12 May 2008, Teemu Likonen wrote:\n> > >\n> > > > Well, I don't really have any problems with the current behaviour;\n> > > > it just feels a bit strange that, for example, Linus's kernel\n> > > > repository grew about 90MB after just one update pull and gc.\n> > >\n> > > That looks really odd. Sure the repo might grow a bit, but 90MB seems\n> > > really excessive. How many time did pass between the initial clone\n> > > and that subsequent pull?\n> >\n> > As I used the kernel repo just for testing this behaviour in question\n> > I did both things today. Timestamps tell that there were six hours\n> > between the initial .keep pack and the new pack created by manual \"git\n> > gc\".\n>\n> This is way too big a difference. Something is going on.\n>\n> What git version is this? And can you send me the content of your\n> .git/logs directory?\n>\n>\n> > > > Also, dangling objects are kept forever in .keep packs (which are\n> > > > created with \"git clone\", for example).\n> > >\n> > > A pack obtained via 'git clone' will never contain any dangling\n> > > objects.\n> >\n> > I think it can contain at some later point. For example, if a user first\n> > fetches all the branches but later decides to track only one branch.\n> > After deleting unneeded tracking branches and expiring the reflog\n> > there'll be dangling objects in the original .keep pack created with\n> > \"git clone\".\n>\n> Sure. But to decide to track only one branch and exclude the others\n> require some higher level of git knowledge already. At that point if\n> you really care about top packing performances you certainly can deal\n> with the .keep file as well.\n>\n>\n\nI have had some similar problems with .keep files. I cloned a repo I\ncreated that had a branch that I wasn't interested in. I deleted the\nbranch and then I could never get rid of the (large) number of objects\nin that pack until I deleted the .keep and repacked. I think there\nshould be some way of forcing git to fix this sort of thing.\n\nIt gets even worse, I had pushed up the branch I wanted to get rid of\nto my hosted server and there was no way to get git to release that\ndisk space. I had to have the hosting admin send me a tarball\nof the repo, extract it, delete the .keep file and repack it then send\nit back to him. I was fortunate enough to have a service that would\nlet me do that.\n\nThanks,\nGovind.\n"},{"id":"76786","messageId":"20080512202414.GA8620@mithlond.arda.local","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805121527550.23581@xanadu.home","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-12T20:24:14Z","receivedAt":"2008-05-12T20:24:14Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"Nicolas Pitre wrote (2008-05-12 15:36 -0400):\n\n> On Mon, 12 May 2008, Teemu Likonen wrote:\n> \n> > > On Mon, 12 May 2008, Teemu Likonen wrote:\n> > > \n> > > > Well, I don't really have any problems with the current\n> > > > behaviour; it just feels a bit strange that, for example,\n> > > > Linus's kernel repository grew about 90MB after just one update\n> > > > pull and gc.\n\n> > As I used the kernel repo just for testing this behaviour in\n> > question I did both things today. Timestamps tell that there were\n> > six hours between the initial .keep pack and the new pack created by\n> > manual \"git gc\".\n> \n> This is way too big a difference. Something is going on.\n> \n> What git version is this? And can you send me the content of your\n> .git/logs directory?\n\nI'm using Git from the \"master\" branch; compiled it today. I have the\nfollowing gc/repack-related patches applied from the \"pu\" branch:\n\n builtin-gc.c: deprecate --prune, it now really has no effect\n git-gc: always use -A when manually repacking\n repack: modify behavior of -A option to leave unreferenced objects unpacked\n\nBut I have experienced the same earlier with some other post-1.5.5\nversion so I believe you can reproduce this yourself. After cloning\nLinus's linux-2.6 repo its .git directory weights 209MB. After single\n\"git pull\" and \"git gc\" it was 298MB in my test.\n\nI'll send you the .git/logs directory but I'm afraid it doesn't tell\nmuch. There are just three files:\n\n .git/logs/HEAD\n .git/logs/refs/heads/master\n .git/logs/refs/remotes/origin/master\n\nThey containt one line for the initial clone and one line for\nthe fast-forward pull.\n\n> > I think it can contain at some later point. For example, if a user\n> > first fetches all the branches but later decides to track only one\n> > branch. After deleting unneeded tracking branches and expiring the\n> > reflog there'll be dangling objects in the original .keep pack\n> > created with \"git clone\".\n> \n> Sure. But to decide to track only one branch and exclude the others\n> require some higher level of git knowledge already. At that point if\n> you really care about top packing performances you certainly can deal\n> with the .keep file as well.\n\nPerhaps so. Although I don't consider this very high level Git\nknowledge:\n\n $ git remote rm origin\n $ git remote add -t wanted_branch origin git://...\n\nThe first command removes all the tracking branches. The latter starts\nto track only one branch.\n"},{"id":"76792","messageId":"20080512210304.GA17352@glandium.org","threadId":"13490","inReplyTo":"20080512202414.GA8620@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2008-05-12T21:03:04Z","receivedAt":"2008-05-12T21:03:04Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:\n> But I have experienced the same earlier with some other post-1.5.5\n> version so I believe you can reproduce this yourself. After cloning\n> Linus's linux-2.6 repo its .git directory weights 209MB. After single\n> \"git pull\" and \"git gc\" it was 298MB in my test.\n\nI noticed that a while ago: when repacking multiple packs when one has a\n.keep file, the resulting additional pack contains too many blobs and\ntrees, contrary to when only packing loose objects:\n\n$ git init\n$ echo a > a; git add a; git commit -m a\n$ git gc\nCounting objects: 3, done.\nWriting objects: 100% (3/3), done.\nTotal 3 (delta 0), reused 0 (delta 0)\n$ git verify-pack -v .git/objects/pack/pack-b87e61e2dc18ff37624d7f996f1270f923411530.pack\n4bba7c0583de30efff4097299f89b199ab4a6dff commit 160 116 12\n78981922613b2afb6025042ff6bd878ac1994e85 blob 2 11 167\naaff74984cccd156a469afa7d9ab10e4777beb24 tree 29 39 128\n.git/objects/pack/pack-b87e61e2dc18ff37624d7f996f1270f923411530.pack: ok\n\n$ touch .git/objects/pack/pack-b87e61e2dc18ff37624d7f996f1270f923411530.keep\n$ echo b > b; git add b; git commit -m b\n$ git gc\nCounting objects: 3, done.\nCompressing objects: 100% (2/2), done.\nWriting objects: 100% (3/3), done.\nTotal 3 (delta 0), reused 0 (delta 0)\n$ git verify-pack -v\n.git/objects/pack/pack-aa817046e43f278d67c6b85962676246f57bb855.pack\n3683f870be446c7cc05ffaef9fa06415276e1828 tree 58 65 158\n61780798228d17af2d34fce4cfbdf35556832472 blob 2 11 223\n647aed0360e964adc5cedb12e0719fb8bfc05867 commit 208 146 12\n.git/objects/pack/pack-aa817046e43f278d67c6b85962676246f57bb855.pack: ok\n\n$ git gc\nCounting objects: 4, done.\nCompressing objects: 100% (2/2), done.\nWriting objects: 100% (4/4), done.\nTotal 4 (delta 0), reused 4 (delta 0)\n$ git verify-pack -v\n.git/objects/pack/pack-5f692a665e062dedad7b4baf692517adec37899d.pack\n3683f870be446c7cc05ffaef9fa06415276e1828 tree 58 65 158\n61780798228d17af2d34fce4cfbdf35556832472 blob 2 11 234\n647aed0360e964adc5cedb12e0719fb8bfc05867 commit 208 146 12\n78981922613b2afb6025042ff6bd878ac1994e85 blob 2 11 223\n.git/objects/pack/pack-5f692a665e062dedad7b4baf692517adec37899d.pack: ok\n\nMike\n"},{"id":"76793","messageId":"alpine.LFD.1.10.0805121704140.23581@xanadu.home","threadId":"13490","inReplyTo":"5d46db230805121310q5a0c6037p769ea091c26284d0@mail.gmail.com","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-12T21:06:08Z","receivedAt":"2008-05-12T21:06:08Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 May 2008, Govind Salinas wrote:\n\n> On Mon, May 12, 2008 at 2:36 PM, Nicolas Pitre wrote:\n> > Sure. But to decide to track only one branch and exclude the others\n> > require some higher level of git knowledge already. At that point if\n> > you really care about top packing performances you certainly can deal\n> > with the .keep file as well.\n> \n> I have had some similar problems with .keep files. I cloned a repo I\n> created that had a branch that I wasn't interested in. I deleted the\n> branch and then I could never get rid of the (large) number of objects\n> in that pack until I deleted the .keep and repacked.\n\nBut as soon as you just \"git pull\" you'll get the deleted branch back.\n\n\nNicolas\n"},{"id":"76794","messageId":"alpine.LFD.1.10.0805121706260.23581@xanadu.home","threadId":"13490","inReplyTo":"20080512202414.GA8620@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-12T21:07:44Z","receivedAt":"2008-05-12T21:07:44Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 May 2008, Teemu Likonen wrote:\n\n> I'll send you the .git/logs directory but I'm afraid it doesn't tell\n> much. There are just three files:\n> \n> .git/logs/HEAD\n> .git/logs/refs/heads/master\n> .git/logs/refs/remotes/origin/master\n> \n> They containt one line for the initial clone and one line for\n> the fast-forward pull.\n\nThat's what I want. This way I should be able to reproduce your exact \ncase.\n\n\nNicolas\n"},{"id":"76795","messageId":"5d46db230805121407pdb3f6fdnd41898d3dde6862b@mail.gmail.com","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805121704140.23581@xanadu.home","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Govind Salinas","fromEmail":"govind@sophiasuchtig.com","sentAt":"2008-05-12T21:07:56Z","receivedAt":"2008-05-12T21:07:56Z","isPatch":false,"sender":{"key":"govind@sophiasuchtig.com","avatar":null},"body":"On Mon, May 12, 2008 at 4:06 PM, Nicolas Pitre wrote:\n> On Mon, 12 May 2008, Govind Salinas wrote:\n>\n> > On Mon, May 12, 2008 at 2:36 PM, Nicolas Pitre wrote:\n>\n> > > Sure. But to decide to track only one branch and exclude the others\n> > > require some higher level of git knowledge already. At that point if\n> > > you really care about top packing performances you certainly can deal\n> > > with the .keep file as well.\n> >\n> > I have had some similar problems with .keep files. I cloned a repo I\n> > created that had a branch that I wasn't interested in. I deleted the\n> > branch and then I could never get rid of the (large) number of objects\n> > in that pack until I deleted the .keep and repacked.\n>\n> But as soon as you just \"git pull\" you'll get the deleted branch back.\n>\n>\nIf you read the rest of my mail, you will see where I removed it from the\nhosted server as well. But with difficulty.\n\nThanks,\nGovind.\n"},{"id":"76796","messageId":"20080512210807.GA22221@glandium.org","threadId":"13490","inReplyTo":"20080512210304.GA17352@glandium.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2008-05-12T21:08:07Z","receivedAt":"2008-05-12T21:08:07Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:\n> On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:\n> > But I have experienced the same earlier with some other post-1.5.5\n> > version so I believe you can reproduce this yourself. After cloning\n> > Linus's linux-2.6 repo its .git directory weights 209MB. After single\n> > \"git pull\" and \"git gc\" it was 298MB in my test.\n> \n> I noticed that a while ago: when repacking multiple packs when one has a\n> .keep file, the resulting additional pack contains too many blobs and\n> trees, contrary to when only packing loose objects:\n(...)\n\nThat is, it seems to also contain all the blobs and subtrees for all the\ncommits the pack contains, even when they already are in the pack having\na .keep file.\n\nMike\n"},{"id":"76810","messageId":"20080512234906.GX29038@spearce.org","threadId":"13490","inReplyTo":"e1dab3980805121017u4c244d25s76b39cf015f6c5c5@mail.gmail.com","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-12T23:49:06Z","receivedAt":"2008-05-12T23:49:06Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"David Tweed wrote:\n> On Mon, May 12, 2008 at 4:52 PM, Teemu Likonen wrote:\n> > Teemu Likonen wrote (2008-05-12 15:29 +0300):\n> > Probably a crazy idea: What if \"gc --aggressive\" first removed *.keep\n> > files and after packing and garbage-collecting and whatever it does it\n> > would add a .keep file for the newly created pack?\n> \n> My understanding is that the repacking with -a redoes the computation\n> to repack ALL the objects in every pack and loose objects,\n\nNo. -a means repack all objects in all packs which do not have a\n.keep on them. Without -a we only repack loose objects.\n\n> whereas\n> what would be preferred is to try to delta new objects (loose and\n> packed) against the existing .keep pack (extending it with the new\n> objects) but not trying to re-deltify objects in the .keep pack.\n\nWe cannot do that. Deltas in pack A may not reference base objects\nin pack B. This is a simplification rule that prevents us from\nneeding to worry about damaging a pack when we repack and delete\nanother pack.\n\n> This\n> is because .keep files are primarily for those who are cloning onto a\n> machine that isn't powerful (maybe even a laptop/palmtop) but who are\n> cloning from a powerful server, so that you wouldn't necessarily want\n> to apply your strategy unconditionally.\n\nYes, sort of. We use .keep for two reasons:\n\n - As a \"lock file\" to prevent a pack that was just created by a\n git-fetch or git-recieve-pack from being deleted by a concurrent\n git-repack before the objects it contains are linked into the\n refs space and thus considered reachable;\n\n - As a way to avoid _huge_ packs (say >1G) that would take a lot\n of disk IO just to copy with 100% delta reuse from an old pack\n to a new pack each time the user runs git-gc.\n\nI think git-clone marking a 150M linux-2.6 pack with .keep is wrong;\nmost users working with the linux-2.6 sources have sufficient\nhardware to deal with the disk IO required to copy that with 100%\ndelta reuse. But I have a repository at day-job with a 600M pack,\nthat's starting to head into the realm where git-gc while running\non battery on a laptop would prefer to have that .keep.\n\n-- \nShawn.\n"},{"id":"76811","messageId":"7vod7bw03a.fsf@gitster.siamese.dyndns.org","threadId":"13490","inReplyTo":"20080512234906.GX29038@spearce.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-12T23:53:44Z","receivedAt":"2008-05-12T23:53:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> David Tweed wrote:\n>> On Mon, May 12, 2008 at 4:52 PM, Teemu Likonen wrote:\n>> > Teemu Likonen wrote (2008-05-12 15:29 +0300):\n>> > Probably a crazy idea: What if \"gc --aggressive\" first removed *.keep\n>> > files and after packing and garbage-collecting and whatever it does it\n>> > would add a .keep file for the newly created pack?\n>> \n>> My understanding is that the repacking with -a redoes the computation\n>> to repack ALL the objects in every pack and loose objects,\n>\n> No. -a means repack all objects in all packs which do not have a\n> .keep on them. Without -a we only repack loose objects.\n>\n>> whereas\n>> what would be preferred is to try to delta new objects (loose and\n>> packed) against the existing .keep pack (extending it with the new\n>> objects) but not trying to re-deltify objects in the .keep pack.\n>\n> We cannot do that. Deltas in pack A may not reference base objects\n> in pack B. This is a simplification rule that prevents us from\n> needing to worry about damaging a pack when we repack and delete\n> another pack.\n>\n>> This\n>> is because .keep files are primarily for those who are cloning onto a\n>> machine that isn't powerful (maybe even a laptop/palmtop) but who are\n>> cloning from a powerful server, so that you wouldn't necessarily want\n>> to apply your strategy unconditionally.\n>\n> Yes, sort of. We use .keep for two reasons:\n>\n> - As a \"lock file\" to prevent a pack that was just created by a\n> git-fetch or git-recieve-pack from being deleted by a concurrent\n> git-repack before the objects it contains are linked into the\n> refs space and thus considered reachable;\n>\n> - As a way to avoid _huge_ packs (say >1G) that would take a lot\n> of disk IO just to copy with 100% delta reuse from an old pack\n> to a new pack each time the user runs git-gc.\n>\n> I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;\n> most users working with the linux-2.6 sources have sufficient\n> hardware to deal with the disk IO required to copy that with 100%\n> delta reuse. But I have a repository at day-job with a 600M pack,\n> that's starting to head into the realm where git-gc while running\n> on battery on a laptop would prefer to have that .keep.\n\nPerhaps clone can decide to keep the .keep file depending on the size of\nthe pack then?\n"},{"id":"76815","messageId":"20080513000925.GA29038@spearce.org","threadId":"13490","inReplyTo":"7vod7bw03a.fsf@gitster.siamese.dyndns.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-13T00:09:25Z","receivedAt":"2008-05-13T00:09:25Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> >\n> > I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;\n> > most users working with the linux-2.6 sources have sufficient\n> > hardware to deal with the disk IO required to copy that with 100%\n> > delta reuse. But I have a repository at day-job with a 600M pack,\n> > that's starting to head into the realm where git-gc while running\n> > on battery on a laptop would prefer to have that .keep.\n> \n> Perhaps clone can decide to keep the .keep file depending on the size of\n> the pack then?\n\nYea, I think that's the better thing to do here. I'm not sure where\nthe cut-off is, maybe its <512M delete the .keep once the refs are\ninplace and the objects are ensured to be reachable.\n\nOf course this does not fix the issue Nico was looking at.\nWe shouldn't be seeing a 98M explosion with objects duplicated\nfrom the .keep pack into the new pack.\n\n-- \nShawn.\n"},{"id":"76816","messageId":"20080513001252.GB29038@spearce.org","threadId":"13490","inReplyTo":"20080512210807.GA22221@glandium.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-13T00:12:52Z","receivedAt":"2008-05-13T00:12:52Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Mike Hommey wrote:\n> On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:\n> > On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:\n> > > But I have experienced the same earlier with some other post-1.5.5\n> > > version so I believe you can reproduce this yourself. After cloning\n> > > Linus's linux-2.6 repo its .git directory weights 209MB. After single\n> > > \"git pull\" and \"git gc\" it was 298MB in my test.\n> > \n> > I noticed that a while ago: when repacking multiple packs when one has a\n> > .keep file, the resulting additional pack contains too many blobs and\n> > trees, contrary to when only packing loose objects:\n> (...)\n> \n> That is, it seems to also contain all the blobs and subtrees for all the\n> commits the pack contains, even when they already are in the pack having\n> a .keep file.\n\nI've noticed this too. Like since day 1 when we added .keep.\nBut uh, nobody else complained and I forgot about it.\n\nMy theory (totally unproven) is that the new pack has objects we\ncopied from the .keep pack, because those objects were the best\ndelta-bases for the loose objects we have deltafied and want to\nstore in the new pack. Except they aren't yet packed in the new\npack, so we pack them too. Tada, duplicates. :-\\\n\nSuddenly your repository nearly doubles in size if we have most\nfiles/trees change, as those delta bases are copied whole into the\nnew pack.\n\n-- \nShawn.\n"},{"id":"76830","messageId":"48292243.3050307@gnu.org","threadId":"13490","inReplyTo":"20080513000925.GA29038@spearce.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Paolo Bonzini","fromEmail":"bonzini@gnu.org","sentAt":"2008-05-13T05:08:19Z","receivedAt":"2008-05-13T05:08:19Z","isPatch":false,"sender":{"key":"bonzini@gnu.org","avatar":"https://avatars.githubusercontent.com/u/42082?v=4"},"body":"Shawn O. Pearce wrote:\n> Junio C Hamano wrote:\n>> \"Shawn O. Pearce\" writes:\n>>> I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;\n>>> most users working with the linux-2.6 sources have sufficient\n>>> hardware to deal with the disk IO required to copy that with 100%\n>>> delta reuse. But I have a repository at day-job with a 600M pack,\n>>> that's starting to head into the realm where git-gc while running\n>>> on battery on a laptop would prefer to have that .keep.\n>> Perhaps clone can decide to keep the .keep file depending on the size of\n>> the pack then?\n> \n> Yea, I think that's the better thing to do here. I'm not sure where\n> the cut-off is, maybe its <512M delete the .keep once the refs are\n> inplace and the objects are ensured to be reachable.\n\nI think separate cutoffs should be in place for file size and number of \nobjects. Very tight packs probably require hours to repack as efficiently.\n\nBy the way, another scenario where I used pack files is when I can only \ndistribute via http because of firewalls. I make a clone of the \noriginal repository and mark the pack as keep; then I push to the \ndistribution site, gc, and mark the pack as keep; then I have every day \na cron job that does git-gc. This way I know that the user will only \nhave to download the third pack. I think I'll modify the cron job to \nmark as keep the packs that exceed 2 megabytes or something like that.\n\nThinking about both use cases, the best would be to have options (common \nto git-clone, git-remote add, git-gc at least; and available via config \nkeys too) like\n\n --keep-packs[=THRES1,THRES2,...]\n\nwhere:\n\n- one threshold would be enough to mark a pack as keep\n- thresholds could be in the form \"\\d+[kmg]?b\" for file size, \n\"\\d+[kmg]?\" for number of objects.\n- if no threshold is given, the default could be --keep-packs=100k,512MB \nor whatever is in the config.\n- to mark all packs, use --keep-packs=0\n\n\nPaolo\n"},{"id":"76831","messageId":"20080513052232.GH29038@spearce.org","threadId":"13490","inReplyTo":"48292243.3050307@gnu.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-13T05:22:32Z","receivedAt":"2008-05-13T05:22:32Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Paolo Bonzini wrote:\n> Shawn O. Pearce wrote:\n> >Junio C Hamano wrote:\n> >>Perhaps clone can decide to keep the .keep file depending on the size of\n> >>the pack then?\n> >\n> >Yea, I think that's the better thing to do here. I'm not sure where\n> >the cut-off is, maybe its <512M delete the .keep once the refs are\n> >inplace and the objects are ensured to be reachable.\n> \n> I think separate cutoffs should be in place for file size and number of \n> objects. Very tight packs probably require hours to repack as efficiently.\n\nSo long as you don't use `gc --aggressive` or `repack -f` the\ntightness of a pack doesn't matter; delta reuse means we copy the\ntight delta from the source pack to the new destination pack.\n\nHowever, you are correct that the more objects in the source pack\nthe longer it will take to compute what is reachable, which does\nextend the time needed for even a simple git-gc.\n \n-- \nShawn.\n"},{"id":"76832","messageId":"20080513053306.GA18232@glandium.org","threadId":"13490","inReplyTo":"20080513001252.GB29038@spearce.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2008-05-13T05:33:06Z","receivedAt":"2008-05-13T05:33:06Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Mon, May 12, 2008 at 08:12:52PM -0400, Shawn O. Pearce wrote:\n> Mike Hommey wrote:\n> > On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:\n> > > On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:\n> > > > But I have experienced the same earlier with some other post-1.5.5\n> > > > version so I believe you can reproduce this yourself. After cloning\n> > > > Linus's linux-2.6 repo its .git directory weights 209MB. After single\n> > > > \"git pull\" and \"git gc\" it was 298MB in my test.\n> > > \n> > > I noticed that a while ago: when repacking multiple packs when one has a\n> > > .keep file, the resulting additional pack contains too many blobs and\n> > > trees, contrary to when only packing loose objects:\n> > (...)\n> > \n> > That is, it seems to also contain all the blobs and subtrees for all the\n> > commits the pack contains, even when they already are in the pack having\n> > a .keep file.\n> \n> I've noticed this too. Like since day 1 when we added .keep.\n> But uh, nobody else complained and I forgot about it.\n> \n> My theory (totally unproven) is that the new pack has objects we\n> copied from the .keep pack, because those objects were the best\n> delta-bases for the loose objects we have deltafied and want to\n> store in the new pack. Except they aren't yet packed in the new\n> pack, so we pack them too. Tada, duplicates. :-\\\n\nWell, that does not seem delta related, since my testcase doesn't show\ndeltas in the second pack.\n\nMike\n"},{"id":"76853","messageId":"20080513092237.GA4413@mithlond.arda.local","threadId":"13490","inReplyTo":"48292243.3050307@gnu.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-13T09:22:37Z","receivedAt":"2008-05-13T09:22:37Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"Paolo Bonzini wrote (2008-05-13 07:08 +0200):\n\n> I think separate cutoffs should be in place for file size and number\n> of objects. Very tight packs probably require hours to repack as\n> efficiently.\n[...]\n> Thinking about both use cases, the best would be to have options\n> (common to git-clone, git-remote add, git-gc at least; and available\n> via config keys too) like\n>\n> --keep-packs[=THRES1,THRES2,...]\n\nSome thoughts from user interface's point of view. Two assumptions:\n\n - gc is daily or weekly operation\n - gc --aggressive is more like weekly or monthly operation.\n\nIn big repositories gc can feel pretty slow if there are not any .keep\npacks and user runs the command daily. So I think there's a point in\nhaving a .keep pack in repositories the size of linux-2.6 for example.\nBut at the same time I think it would be nice to have an easy UI-way to\nrepack with better disk space optimization.\n\nThis started as a crazy idea but maybe it's not so crazy so I'll\nrephrase my previous suggestion. At final stage the command gc\n--aggressive would add new .keep file which contains an identifier like\n\n This .keep file was added by \"gc --aggressive\" and\n will be automatically deleted at next run.\n\n(Or something like that, you get the idea.)\n\nAt first gc --aggressive looks for .keep files with such identifier and\ndeletes them if found. Then it proceeds normally and finally adds new\n.keep file with the same identifier.\n\nThis way the \"daily\" gc would operate very fast (as it leaves .keep\npacks alone), and with gc --aggressive user could easily decide when to\ncreate new landmark .keep packs (and also prune possible dangling\nobjects inside previous .keep packs). Normal user don't need to know the\ndetails. Just run gc occasionally and maybe gc --aggressive when better\noptimization is needed.\n\nHow does this sound?\n"},{"id":"76888","messageId":"20080513214638.GA14930@cuci.nl","threadId":"13490","inReplyTo":"20080513092237.GA4413@mithlond.arda.local","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2008-05-13T21:46:38Z","receivedAt":"2008-05-13T21:46:38Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Teemu Likonen wrote:\n>This way the \"daily\" gc would operate very fast (as it leaves .keep\n>packs alone), and with gc --aggressive user could easily decide when to\n>create new landmark .keep packs (and also prune possible dangling\n>objects inside previous .keep packs). Normal user don't need to know the\n>details. Just run gc occasionally and maybe gc --aggressive when better\n>optimization is needed.\n\n>How does this sound?\n\nIt sounds sound :-).\nI like the simplicity.\n-- \nSincerely, srb@cuci.nl\n Stephen R. van den Berg.\n"},{"id":"76899","messageId":"alpine.LFD.1.10.0805132005550.23581@xanadu.home","threadId":"13490","inReplyTo":"20080513001252.GB29038@spearce.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-14T01:03:31Z","receivedAt":"2008-05-14T01:03:31Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 May 2008, Shawn O. Pearce wrote:\n\n> Mike Hommey wrote:\n> > On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:\n> > > On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:\n> > > > But I have experienced the same earlier with some other post-1.5.5\n> > > > version so I believe you can reproduce this yourself. After cloning\n> > > > Linus's linux-2.6 repo its .git directory weights 209MB. After single\n> > > > \"git pull\" and \"git gc\" it was 298MB in my test.\n> > > \n> > > I noticed that a while ago: when repacking multiple packs when one has a\n> > > .keep file, the resulting additional pack contains too many blobs and\n> > > trees, contrary to when only packing loose objects:\n> > (...)\n> > \n> > That is, it seems to also contain all the blobs and subtrees for all the\n> > commits the pack contains, even when they already are in the pack having\n> > a .keep file.\n> \n> I've noticed this too. Like since day 1 when we added .keep.\n> But uh, nobody else complained and I forgot about it.\n\nWell, now that I've reproduced Teemu Likonen's test case, I can confirm \nthis is actually a problem. Here I get:\n\n|remote: Counting objects: 523, done.\n|remote: Compressing objects: 100% (57/57), done.\n|remote: Total 362 (delta 305), reused 362 (delta 305)\n|Receiving objects: 100% (362/362), 65.37 KiB, done.\n|Resolving deltas: 100% (305/305), completed with 105 local objects.\n|From ../test1\n| 492c2e4..9404ef0 master -> master\n\nThe received pack is 449135 bytes large. This is much larger than the \nactually received data which is 65.37 KiB, but we're completing a thin \npack with 105 undeltified objects accounting for the size increase which \nis expected. So far so good.\n\nNow, in theory, running 'git gc' should only repack those 362 + 105 \nobjects, since the remaining ones are all found in the .keep flagged \npack. But that's not what's happening at all:\n\n|Counting objects: 26559, done.\n|Compressing objects: 100% (24708/24708), done.\n|Writing objects: 100% (26559/26559), done.\n|Total 26559 (delta 3054), reused 14011 (delta 1613)\n\nSo... there is something definitively wrong here. The expectation was \nto get a pack in the same size range as the one received during the \npack, or somewhat smaller due to a better delta compression of the added \nobjects. But instead we get a pack containing 26559 objects!!! And in \nthat lot, only 3054 (11%) are deltas. That makes for a pack that \nstarted from 449135 bytes and grew to 72395940 bytes.\n\n> My theory (totally unproven) is that the new pack has objects we\n> copied from the .keep pack, because those objects were the best\n> delta-bases for the loose objects we have deltafied and want to\n> store in the new pack. Except they aren't yet packed in the new\n> pack, so we pack them too. Tada, duplicates. :-\\\n\nWell, not exactly.\n\nLet's see what happens here even before any packing is attempted\n\n|$ git rev-list --objects 492c2e4..9404ef0\n|362\n|\n|$ git rev-list --objects --all \\\n| --unpacked=pack-6a3438b2702be06697023d80b77e67a73a0b0b5c.pack |\n|\twc -l\n|26559\n\nSo this --unpacked= argument (which undocumented semantics I still have \nissues with) is certainly not doing what is expected.\n\n\nNicolas\n"},{"id":"77047","messageId":"20080514054245.GA3147@mithlond.arda.local","threadId":"13490","inReplyTo":"20080513214638.GA14930@cuci.nl","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2008-05-14T05:42:45Z","receivedAt":"2008-05-14T05:42:45Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"Stephen R. van den Berg wrote (2008-05-14 00:46 +0300):\n\n> Teemu Likonen wrote:\n> >This way the \"daily\" gc would operate very fast (as it leaves .keep\n> >packs alone), and with gc --aggressive user could easily decide when to\n> >create new landmark .keep packs (and also prune possible dangling\n> >objects inside previous .keep packs). Normal user don't need to know the\n> >details. Just run gc occasionally and maybe gc --aggressive when better\n> >optimization is needed.\n> \n> >How does this sound?\n> \n> It sounds sound :-).\n> I like the simplicity.\n\nIt turned out that gc --aggressive is not what I thought it was, i.e.\n\"pack aggressively and efficiently\". So my suggestion implies the\nsemantics that --aggressive would do effective compressing.\n"},{"id":"76921","messageId":"7vy76dperf.fsf@gitster.siamese.dyndns.org","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805132005550.23581@xanadu.home","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-14T06:43:16Z","receivedAt":"2008-05-14T06:43:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> Let's see what happens here even before any packing is attempted\n>\n> |$ git rev-list --objects 492c2e4..9404ef0\n> |362\n> |\n> |$ git rev-list --objects --all \\\n> | --unpacked=pack-6a3438b2702be06697023d80b77e67a73a0b0b5c.pack |\n> |\twc -l\n> |26559\n>\n> So this --unpacked= argument (which undocumented semantics I still have \n> issues with) is certainly not doing what is expected.\n\nThe output from rev-list is not surprising. --unpacked=$this.pack implies\nthe usual --unpacked behaviour (i.e. only show unpacked objects by not\ntraversing into commits that are packed) and at the same time pretends\nthat objects in $this.pack are loose.\n\nIt was meant to be used for a partial incremental repacking. If you have\na pack to be kept (perhaps a highly packed deep pack that holds the\nearlier parts of the history), marked with .keep, and a handful young\npacks, you would give these young ones with --unpacked, so that the\nresulting single pack contains all that are loose or in these young\npacks. After that, you can remove all the young packs and loose objects.\n\nAt least that is the idea.\n\nI am not sure where that rev-list experiment you showed fits in the bigger\npicture, but if that is used for repacking the young packs, perhaps the\nissue is that after the repacking the code forgets to remove the young\nones whose objects are now moved into the new pack?\n"},{"id":"76934","messageId":"18474.44155.823000.368851@lapjr.intranet.kiel.bmiag.de","threadId":"13490","inReplyTo":"7vy76dperf.fsf@gitster.siamese.dyndns.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Juergen Ruehle","fromEmail":"j.ruehle@bmiag.de","sentAt":"2008-05-14T09:10:19Z","receivedAt":"2008-05-14T09:10:19Z","isPatch":false,"sender":{"key":"j.ruehle@bmiag.de","avatar":null},"body":"Junio C Hamano writes:\n > The output from rev-list is not surprising. --unpacked=$this.pack implies\n > the usual --unpacked behaviour (i.e. only show unpacked objects by not\n > traversing into commits that are packed)\n\nThe problem is unconditional traversing into commits that are\nunpacked. This behavior is immediately obvious if the packed blob in\nthe .keep pack is large. I've been using the following since the large\nobject discussion with Dana, but it might be completely broken (though\nthe test case is probably correct).\n\n--\n\nPreviously --unpacked would filter on the commit level, ignoring whether the\nobjects comprising the commit actually were packed or unpacked.\n\nThis makes it impossible to store e.g. excessively large blobs in\ndifferent packs from the commits referencing them, since the next repack of\nsuch a commit will suck all referenced blobs into the same pack.\n\nThis change moves the unpacked check to the output stage and no longer checks\nthe flag during commit traversal and adds a trivial test demonstrating the\nproblem.\n---\n Note that t6009 is already taken, so it might be better to merge the test\n into one of the other rev-list tests.\n\n list-objects.c | 6 ++++--\n revision.c | 2 --\n t/t6009-rev-list-unpacked.sh | 32 ++++++++++++++++++++++++++++++++\n 3 files changed, 36 insertions(+), 4 deletions(-)\n create mode 100644 t/t6009-rev-list-unpacked.sh\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c8b8375..b378c0f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -146,7 +146,8 @@ void traverse_commit_list(struct rev_info *revs,\n \n \twhile ((commit = get_revision(revs)) != NULL) {\n \t\tprocess_tree(revs, commit->tree, &objects, NULL, \"\");\n-\t\tshow_commit(commit);\n+\t\tif (!revs->unpacked || !has_sha1_pack(commit->object.sha1, revs->ignore_packed))\n+\t\t\tshow_commit(commit);\n \t}\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = revs->pending.objects + i;\n@@ -173,7 +174,8 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t sha1_to_hex(obj->sha1), name);\n \t}\n \tfor (i = 0; i < objects.nr; i++)\n-\t\tshow_object(&objects.objects[i]);\n+\t\tif (!revs->unpacked || !has_sha1_pack(objects.objects[i].item->sha1, revs->ignore_packed))\n+\t\t\tshow_object(&objects.objects[i]);\n \tfree(objects.objects);\n \tif (revs->pending.nr) {\n \t\tfree(revs->pending.objects);\ndiff --git a/revision.c b/revision.c\nindex 4231ea2..0e90d3b 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1508,8 +1508,6 @@ enum commit_action simplify_commit(struct rev_info *revs, struct commit *commit)\n {\n \tif (commit->object.flags & SHOWN)\n \t\treturn commit_ignore;\n-\tif (revs->unpacked && has_sha1_pack(commit->object.sha1, revs->ignore_packed))\n-\t\treturn commit_ignore;\n \tif (revs->show_all)\n \t\treturn commit_show;\n \tif (commit->object.flags & UNINTERESTING)\ndiff --git a/t/t6009-rev-list-unpacked.sh b/t/t6009-rev-list-unpacked.sh\nnew file mode 100644\nindex 0000000..6b65e83\n--- /dev/null\n+++ b/t/t6009-rev-list-unpacked.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='test git rev-list --unpacked --objects'\n+\n+. ./test-lib.sh\n+\n+# Create an unpacked commit that references a packed object.\n+\n+test_expect_success setup '\n+\techo Hallo > foo &&\n+\tgit add foo &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+ git gc &&\n+\techo Cello > bar &&\n+\tgit add bar &&\n+\ttest_tick &&\n+\tgit commit -m \"B\"\n+'\n+\n+test_expect_success \\\n+ 'object list should contain foo' '\n+ git rev-list --all --objects | grep -q \"foo\"\n+'\n+\n+test_expect_success \\\n+ 'unpacked object list should not contain foo' '\n+ test_must_fail \"git rev-list --all --unpacked --objects | grep -q \\\"foo\\\"\"\n+'\n+\n+\n+test_done\n-- \n1.5.5.1.382.g7d84c\n"},{"id":"76943","messageId":"alpine.LFD.1.10.0805141017210.23581@xanadu.home","threadId":"13490","inReplyTo":"18474.44155.823000.368851@lapjr.intranet.kiel.bmiag.de","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-14T14:24:17Z","receivedAt":"2008-05-14T14:24:17Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 14 May 2008, Juergen Ruehle wrote:\n\n> Junio C Hamano writes:\n> > The output from rev-list is not surprising. --unpacked=$this.pack implies\n> > the usual --unpacked behaviour (i.e. only show unpacked objects by not\n> > traversing into commits that are packed)\n> \n> The problem is unconditional traversing into commits that are\n> unpacked. This behavior is immediately obvious if the packed blob in\n> the .keep pack is large. \n\nThat's what I was suspecting too. And because the Linux repo contains \nmany files, then a single commit will fetch a large bunch of objects \nindeed.\n\n> I've been using the following since the large\n> object discussion with Dana, but it might be completely broken (though\n> the test case is probably correct).\n\nThis is not some part of git code I'm familiar with, so I can't tell if \nthe patch is broken or not. What I can do is repeat my simple test \nwhich produces the following results with your patch:\n\n|$ git rev-list --objects 492c2e4..9404ef0\n|362\n|\n|$ git rev-list --objects --all \\\n| --unpacked=pack-6a3438b2702be06697023d80b77e67a73a0b0b5c.pack |\n| wc -l\n|362\n\nThat's exactly what is expected.\n\n\nNicolas\n"},{"id":"76955","messageId":"7vd4noq0ms.fsf@gitster.siamese.dyndns.org","threadId":"13490","inReplyTo":"18474.44155.823000.368851@lapjr.intranet.kiel.bmiag.de","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-14T17:03:07Z","receivedAt":"2008-05-14T17:03:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Juergen Ruehle writes:\n\n> Previously --unpacked would filter on the commit level, ignoring whether the\n> objects comprising the commit actually were packed or unpacked.\n>\n> This makes it impossible to store e.g. excessively large blobs in\n> different packs from the commits referencing them, since the next repack of\n> such a commit will suck all referenced blobs into the same pack.\n\nDoesn't this patch essentially make the --unpacked option to rev-list and\nthe --incremental option to pack-objects the same thing?\n\nThe semantics of the --unpacked has been defined that way from the very\nbeginning, and I've always wondered how the option and --incremental\nshould interact with each other. I think the approach your patch takes\nmakes sense.\n\n> This change moves the unpacked check to the output stage and no longer checks\n> the flag during commit traversal and adds a trivial test demonstrating the\n> problem.\n\nSign-off?\n\n> diff --git a/t/t6009-rev-list-unpacked.sh b/t/t6009-rev-list-unpacked.sh\n> new file mode 100644\n> index 0000000..6b65e83\n> --- /dev/null\n> +++ b/t/t6009-rev-list-unpacked.sh\n> @@ -0,0 +1,32 @@\n> ...\n> +test_expect_success \\\n> + 'unpacked object list should not contain foo' '\n> + test_must_fail \"git rev-list --all --unpacked --objects | grep -q \\\"foo\\\"\"\n> +'\n\nAhhh. Ugly but don't you mean \"! (rev-list | grep)\"?\n"},{"id":"76984","messageId":"alpine.LFD.1.10.0805141247560.3019@woody.linux-foundation.org","threadId":"13490","inReplyTo":"18474.44155.823000.368851@lapjr.intranet.kiel.bmiag.de","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-05-14T20:06:06Z","receivedAt":"2008-05-14T20:06:06Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 14 May 2008, Juergen Ruehle wrote:\n> \n> Previously --unpacked would filter on the commit level, ignoring whether the\n> objects comprising the commit actually were packed or unpacked.\n\nI think this patch is correct, but I wonder why you removed the pruning \nfrom revision.c? Why do we want to process trees for commits that aren't \ngoing to be shown? This is going to slow down things a lot, and we've long \nhad the rule that commits have to be complete in the packs that are kept \n(ie you should never have a pack-file that points to an unpacked object).\n\nSo I'd suggest a slightly less intrusive patch (untested!!) instead, which \nleaves the commit object logic alone.\n\n(Your test-case should obviously be merged regardless)\n\n\t\tLinus\n\n---\n list-objects.c | 8 ++++++--\n 1 files changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c8b8375..8cb05ca 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -172,8 +172,12 @@ void traverse_commit_list(struct rev_info *revs,\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t sha1_to_hex(obj->sha1), name);\n \t}\n-\tfor (i = 0; i < objects.nr; i++)\n-\t\tshow_object(&objects.objects[i]);\n+\tfor (i = 0; i < objects.nr; i++) {\n+\t\tstruct object_array_entry *entry = &objects.objects[i];\n+\t\tif (revs->unpacked && has_sha1_pack(entry->item->sha1, revs->ignore_packed))\n+\t\t\tcontinue;\n+\t\tshow_object(entry);\n+\t}\n \tfree(objects.objects);\n \tif (revs->pending.nr) {\n \t\tfree(revs->pending.objects);\n"},{"id":"76986","messageId":"alpine.LFD.1.10.0805141314440.3019@woody.linux-foundation.org","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805141247560.3019@woody.linux-foundation.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-05-14T20:19:08Z","receivedAt":"2008-05-14T20:19:08Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 14 May 2008, Linus Torvalds wrote:\n> \n> I think this patch is correct, but I wonder why you removed the pruning \n> from revision.c?\n\nIn fact, it might be a good idea to not just keep it in revision.c, but \nmove it up a bit, so that a commit that is packed and should be ignored \nwon't even have its parents put on the list (which means that we not only \nignore the trees in that commit, but also all parents).\n\nOf course, the more aggressively we prune, the more we end up having to \ndepend on the fact that a commit that is in a pack that is marked \"keep\" \nmust *always* have everything that leads to it in that pack or others also \nmarked \"keep\". We effectively have that already (because we've always \npruned away the commits early), but it's a thing to keep in mind whenever \nwe prune even more aggressively.\n\n\t\tLinus\n"},{"id":"76987","messageId":"alpine.LFD.1.10.0805141626070.23581@xanadu.home","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805141314440.3019@woody.linux-foundation.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-14T20:29:31Z","receivedAt":"2008-05-14T20:29:31Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 14 May 2008, Linus Torvalds wrote:\n\n> Of course, the more aggressively we prune, the more we end up having to \n> depend on the fact that a commit that is in a pack that is marked \"keep\" \n> must *always* have everything that leads to it in that pack or others also \n> marked \"keep\". We effectively have that already (because we've always \n> pruned away the commits early), but it's a thing to keep in mind whenever \n> we prune even more aggressively.\n\nI wonder if this is a good thing. Such a rule would effectively put \nrestrictions on how objects like big blobs could be distributed amongst \nmany .keep packs. I just wish we're not painting ourselves in a corner.\n\n\nNicolas\n"},{"id":"76989","messageId":"alpine.LFD.1.10.0805141333050.3019@woody.linux-foundation.org","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805141626070.23581@xanadu.home","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-05-14T20:36:39Z","receivedAt":"2008-05-14T20:36:39Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 14 May 2008, Nicolas Pitre wrote:\n\n> On Wed, 14 May 2008, Linus Torvalds wrote:\n> \n> > Of course, the more aggressively we prune, the more we end up having to \n> > depend on the fact that a commit that is in a pack that is marked \"keep\" \n> > must *always* have everything that leads to it in that pack or others also \n> > marked \"keep\". We effectively have that already (because we've always \n> > pruned away the commits early), but it's a thing to keep in mind whenever \n> > we prune even more aggressively.\n> \n> I wonder if this is a good thing. Such a rule would effectively put \n> restrictions on how objects like big blobs could be distributed amongst \n> many .keep packs. I just wish we're not painting ourselves in a corner.\n\nYou can distribute big objects arbitrarily among many .keep packs, but \nwhat you can *NOT* do (and which has _always_ been a bug to do) is to have \na *.keep pack that refers to objects that are not in a .keep pack!\n\nSo keep<->keep you can do anything you want, and distribute objects any \nway.\n\nBut a keep pack must only refer to objects in itself or in other keep \npacks.\n\nBecause otherwise, if we ever hit an object in a keep pack, we'll stop \neven looking further when we use --unpacked. And that has always been true \n(admittedly only for \"commit\" objects, but those are the ones that most \ncommonly refer to other objects, so ..)\n\n\t\t\tLinus\n"},{"id":"77003","messageId":"482B7491.30608@gmail.com","threadId":"13490","inReplyTo":"alpine.LFD.1.10.0805141333050.3019@woody.linux-foundation.org","subject":"Re: Why repository grows after \"git gc\"? / Purpose of *.keep files?","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2008-05-14T23:24:01Z","receivedAt":"2008-05-14T23:24:01Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Linus Torvalds wrote:\n> \n> On Wed, 14 May 2008, Nicolas Pitre wrote:\n> \n>> On Wed, 14 May 2008, Linus Torvalds wrote:\n>>\n>>> Of course, the more aggressively we prune, the more we end up having to \n>>> depend on the fact that a commit that is in a pack that is marked \"keep\" \n>>> must *always* have everything that leads to it in that pack or others also \n>>> marked \"keep\". We effectively have that already (because we've always \n>>> pruned away the commits early), but it's a thing to keep in mind whenever \n>>> we prune even more aggressively.\n>> I wonder if this is a good thing. Such a rule would effectively put \n>> restrictions on how objects like big blobs could be distributed amongst \n>> many .keep packs. I just wish we're not painting ourselves in a corner.\n> \n> You can distribute big objects arbitrarily among many .keep packs, but \n> what you can *NOT* do (and which has _always_ been a bug to do) is to have \n> a *.keep pack that refers to objects that are not in a .keep pack!\n> \n> So keep<->keep you can do anything you want, and distribute objects any \n> way.\n> \n> But a keep pack must only refer to objects in itself or in other keep \n> packs.\n> \n> Because otherwise, if we ever hit an object in a keep pack, we'll stop \n> even looking further when we use --unpacked. And that has always been true \n> (admittedly only for \"commit\" objects, but those are the ones that most \n> commonly refer to other objects, so ..)\n\nSounds like git-fsck needs to start checking for this.\n"}]}