{"thread":{"id":"24500","subject":"object/pack size x5 larger than a fresh clone?","startedAt":"2010-07-24T21:57:47Z","lastAt":"2010-07-27T21:15:31Z","messageCount":6,"participants":["Hin-Tak Leung","Andreas Ericsson","Junio C Hamano","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"146215","messageId":"AANLkTimL+wfu+yMPutq2VHD6vO2AtaF_7FpWt8aZPm1c@mail.gmail.com","threadId":"24500","inReplyTo":null,"subject":"object/pack size x5 larger than a fresh clone?","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2010-07-24T21:57:47Z","receivedAt":"2010-07-24T21:57:47Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"Is there any reason why a fresh git clone has a object pack around\n140MB but one that has been updated over the years has it over 700MB?\n(even with git gc --aggressive --prune=now and git fsck?)\n\n$ du .git/objects/\n711364\t.git/objects/pack\n\n$ du *wine/.git/objects/pack\n144692\tgit-wine/.git/objects/pack\n144604\twine/.git/objects/pack\n\nI had a problem with git fetch \"Cannot obtain needed object\" from\nwine's git repository (which seems to be something to do with http\nproxy, although AFAIK I don't have one) since about 2 weeks ago which\nobviously does not apply to anybody else as I would have heard from\nwine-devel.\n\nEditing .git/config to switch from a http url to git url cure it...\nbut in the course of investigating, I git clone fresh (there are only\nabout 3 local changes so I could just git-format-patch them and move\nthem)\n\nhttp://source.winehq.org/git/wine.git\ngit://source.winehq.org/git/wine.git\n\nand I am a bit surprised that the new clones are so much smaller than\nthe one I have been working on these last few years. (I have had the\nold one for at least 3-4 years).\n"},{"id":"146353","messageId":"4C4D42BA.6070900@op5.se","threadId":"24500","inReplyTo":"AANLkTimL+wfu+yMPutq2VHD6vO2AtaF_7FpWt8aZPm1c@mail.gmail.com","subject":"Re: object/pack size x5 larger than a fresh clone?","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2010-07-26T08:09:30Z","receivedAt":"2010-07-26T08:09:30Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"On 07/24/2010 11:57 PM, Hin-Tak Leung wrote:\n> Is there any reason why a fresh git clone has a object pack around\n> 140MB but one that has been updated over the years has it over 700MB?\n> (even with git gc --aggressive --prune=now and git fsck?)\n> \n> $ du .git/objects/\n> 711364\t.git/objects/pack\n> \n> $ du *wine/.git/objects/pack\n> 144692\tgit-wine/.git/objects/pack\n> 144604\twine/.git/objects/pack\n> \n> I had a problem with git fetch \"Cannot obtain needed object\" from\n> wine's git repository (which seems to be something to do with http\n> proxy, although AFAIK I don't have one) since about 2 weeks ago which\n> obviously does not apply to anybody else as I would have heard from\n> wine-devel.\n> \n> Editing .git/config to switch from a http url to git url cure it...\n> but in the course of investigating, I git clone fresh (there are only\n> about 3 local changes so I could just git-format-patch them and move\n> them)\n> \n> http://source.winehq.org/git/wine.git\n> git://source.winehq.org/git/wine.git\n> \n> and I am a bit surprised that the new clones are so much smaller than\n> the one I have been working on these last few years. (I have had the\n> old one for at least 3-4 years).\n\nTo make a fair comparison, try\n git repack -a -f -d && git prune --expire=now\n\nin your old repository. Be warned that this will remove all commits\nreachable from reflogs but not from branch heads or tags though.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"146414","messageId":"AANLkTimLAROADKgWvEXa9QyyDGbWQGP+9BdbcN4fMHJ0@mail.gmail.com","threadId":"24500","inReplyTo":"4C4D42BA.6070900@op5.se","subject":"Re: object/pack size x5 larger than a fresh clone?","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2010-07-26T18:42:13Z","receivedAt":"2010-07-26T18:42:13Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On 7/26/10, Andreas Ericsson wrote:\n> On 07/24/2010 11:57 PM, Hin-Tak Leung wrote:\n>> Is there any reason why a fresh git clone has a object pack around\n>> 140MB but one that has been updated over the years has it over 700MB?\n>> (even with git gc --aggressive --prune=now and git fsck?)\n>>\n>> $ du .git/objects/\n>> 711364\t.git/objects/pack\n>>\n>> $ du *wine/.git/objects/pack\n>> 144692\tgit-wine/.git/objects/pack\n>> 144604\twine/.git/objects/pack\n>>\n>> I had a problem with git fetch \"Cannot obtain needed object\" from\n>> wine's git repository (which seems to be something to do with http\n>> proxy, although AFAIK I don't have one) since about 2 weeks ago which\n>> obviously does not apply to anybody else as I would have heard from\n>> wine-devel.\n>>\n>> Editing .git/config to switch from a http url to git url cure it...\n>> but in the course of investigating, I git clone fresh (there are only\n>> about 3 local changes so I could just git-format-patch them and move\n>> them)\n>>\n>> http://source.winehq.org/git/wine.git\n>> git://source.winehq.org/git/wine.git\n>>\n>> and I am a bit surprised that the new clones are so much smaller than\n>> the one I have been working on these last few years. (I have had the\n>> old one for at least 3-4 years).\n>\n> To make a fair comparison, try\n> git repack -a -f -d && git prune --expire=now\n>\n> in your old repository. Be warned that this will remove all commits\n> reachable from reflogs but not from branch heads or tags though.\n\nI have tried as you said, and it has gone slightly worse -\n$ du .git/objects/\n741172\t.git/objects/pack\n16\t.git/objects/info\n741196\t.git/objects/\n\nHowever, I think I found one very big anormaly - in most of my git\nclones (I have ~20 projects I track) , .git/object/pack consists of\npairs of files like this:\n\npack-.idx\npack-. pack\n\nwith the occasional 3rd member, \"pack-. keep\" . I did not look\nbefore I did the above, but now the strange repository consists of one\nsuch pair to about 147MB, which has a very new time stamp, but a lot\nof singular pack-.idx without a corresponding pack-.pack ,\nand some of them quite large, and many of them has time stamps going\nback to Mar 2008. I have got almost 500 of them, and they vary from\nabout 1.2k to 12MB, so it adds up to over 550MB.\n\nSo I guess these *.idx without a corresponding *.pack are safe to\ndelete? But git gc or one of the other house keeping commands should\nget rid of them though, I think.\n\nShould I file a bug etc for this? Thanks a lot any how.\n\nHin-Tak\n"},{"id":"146493","messageId":"7v7hkg982j.fsf@alter.siamese.dyndns.org","threadId":"24500","inReplyTo":"AANLkTimLAROADKgWvEXa9QyyDGbWQGP+9BdbcN4fMHJ0@mail.gmail.com","subject":"Re: object/pack size x5 larger than a fresh clone?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-07-27T16:57:08Z","receivedAt":"2010-07-27T16:57:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Hin-Tak Leung writes:\n\n> So I guess these *.idx without a corresponding *.pack are safe to\n> delete? But git gc or one of the other house keeping commands should\n> get rid of them though, I think.\n\nI agree. I think the dumb transports like http:// grab *.idx files\nwithout downloading corresponding *.pack files when they encounter an\nobject that is not found loose in the originating repository to see which\npackfile to fetch, but after they are done (or when they are interrupted,\nfor that matter), these *.idx files may not be getting garbage-collected.\n\nAnd they should be, perhaps with or without some grace period (I don't\nknow which offhand---I didn't think this through).\n"},{"id":"146494","messageId":"20100727170317.GC25268@spearce.org","threadId":"24500","inReplyTo":"7v7hkg982j.fsf@alter.siamese.dyndns.org","subject":"Re: object/pack size x5 larger than a fresh clone?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-07-27T17:03:17Z","receivedAt":"2010-07-27T17:03:17Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> Hin-Tak Leung writes:\n> \n> > So I guess these *.idx without a corresponding *.pack are safe to\n> > delete? But git gc or one of the other house keeping commands should\n> > get rid of them though, I think.\n> \n> I agree. I think the dumb transports like http:// grab *.idx files\n> without downloading corresponding *.pack files when they encounter an\n> object that is not found loose in the originating repository to see which\n> packfile to fetch, but after they are done (or when they are interrupted,\n> for that matter), these *.idx files may not be getting garbage-collected.\n> \n> And they should be, perhaps with or without some grace period (I don't\n> know which offhand---I didn't think this through).\n\nWe should GC these, but only after a grace period.\n\nLong ago when I used dumb http it really helped to have the *.idx\nfiles cached. If the upstream only did an incremental repack holding\nonto the *.idx files locally meant I didn't need to redownload\nthem in order to rule-out those packs as onces interesting for the\ncurrent fetch.\n\nMaybe we just prune those during git fetch if they don't have a\nlocal *.pack and they don't match a pack listed by the remote's\nobjects/info/packs file?\n\n-- \nShawn.\n"},{"id":"146534","messageId":"AANLkTikDGzc5d=1w43eJFyO4JdcHEqYYaEm=1P-LpqCj@mail.gmail.com","threadId":"24500","inReplyTo":"20100727170317.GC25268@spearce.org","subject":"Re: object/pack size x5 larger than a fresh clone?","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2010-07-27T21:15:31Z","receivedAt":"2010-07-27T21:15:31Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On 7/27/10, Shawn O. Pearce wrote:\n> Junio C Hamano wrote:\n>> Hin-Tak Leung writes:\n>>\n>> > So I guess these *.idx without a corresponding *.pack are safe to\n>> > delete? But git gc or one of the other house keeping commands should\n>> > get rid of them though, I think.\n>>\n>> I agree. I think the dumb transports like http:// grab *.idx files\n>> without downloading corresponding *.pack files when they encounter an\n>> object that is not found loose in the originating repository to see which\n>> packfile to fetch, but after they are done (or when they are interrupted,\n>> for that matter), these *.idx files may not be getting garbage-collected.\n>>\n>> And they should be, perhaps with or without some grace period (I don't\n>> know which offhand---I didn't think this through).\n>\n> We should GC these, but only after a grace period.\n>\n> Long ago when I used dumb http it really helped to have the *.idx\n> files cached. If the upstream only did an incremental repack holding\n> onto the *.idx files locally meant I didn't need to redownload\n> them in order to rule-out those packs as onces interesting for the\n> current fetch.\n>\n> Maybe we just prune those during git fetch if they don't have a\n> local *.pack and they don't match a pack listed by the remote's\n> objects/info/packs file?\n\nOkay, so they are left-overs from using http:// for fetching but\nserves a useful purpose for a limited period. The usual gc --prune\ndefaults to 2 weeks, is that good enough? Or should there be a longer\ngrace period?\n\nI only switch over to git:// these last few days after it failing to\nfetch for two weeks and it looks like only I have this problem and\naround the web, failure-to-fetch seems to indicate an http proxy\nproblem.\n\nFWIW, my left-over files seems to co-incide with the 2-3 week snapshot\nrelease schedule of wine. I don't know if any of you is familiar with\nwine, but only one person (AJ) has commit rights and he reviews all\npatches and does a periodic push, usually just before or after a\nsnapshot release but occasionally more frequent. My left-overs seems\nto co-incide with the first fetch after such a push. My most recent\n\"permanent\" fetch failure is due to the long-awaited wine 1.2 release,\nI think.\n\nThanks for all the insights, and I hope a future git release will\nprune some of these left-over files after a period.\n\nHin-Tak\n"}]}