{"thread":{"id":"26664","subject":"why is my repo size ten times bigger than what I expected?","startedAt":"2011-03-05T10:05:24Z","lastAt":"2011-03-09T21:06:19Z","messageCount":10,"participants":["Ruben Laguna","Pascal Obry","Jonathan del Strother","Andreas Schwab","Phillip Susi","Tor Arvid Lund"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"162810","messageId":"AANLkTi=V3nEamocowbHovvV0U69nZgD70fysu1CQOwrR@mail.gmail.com","threadId":"26664","inReplyTo":"AANLkTimi+OnpdX+Y7jx1JaOmGbZc_XEgJFeK0PKLpu2o@mail.gmail.com","subject":"why is my repo size ten times bigger than what I expected?","fromName":"Ruben Laguna","fromEmail":"ruben.laguna@gmail.com","sentAt":"2011-03-05T10:05:24Z","receivedAt":"2011-03-05T10:05:24Z","isPatch":false,"sender":{"key":"ruben.laguna@gmail.com","avatar":"https://gravatar.com/avatar/ab5036870c2456b61092d6e238eed916343a69d86721685c7012b50c39d95a61?d=mp&s=160"},"body":"Hi,\n\nI had a repo which was big 143MB because it contained a bunch of jar\nfiles. So I decided to remove those completely from the history.\n\nIn short I used the git-large-blob [1] to find all the jars and used\nthe git-remove-history script [2] which does the filter-branch thing,\nprune, etc.\n\nI did this on all branches (that I know of) and now I can see that the\njars are gone because I can't find them with git-large-blob.  and the\nrepo size has dropped from 143Mb to 87Mb.\n\nMy concern is that 87Mb is still really big taking into account he\nsize of the project. in fact if I run \"git diff-tree -r -p $commit\n|wc -c\" for each commit and sum all I get 5.5Mb.\n\n\nI also ran the git-rev-size [3] script that I found in this mailing\nlist and I only see that the size grows steadly from commit to commit\nup to 1482731 bytes. So again how come the .git directory is 87MB?\n\n\nSo, Can anybody tell me if this repository size is \"normal\" for a\nproject with 1.4MB source and 352 commits?\nIs there a better way to calculate the size (in bytes) of each commit?\n\nIs there any other thing I could do to reduce and audit  the repository size?\n\n\nThanks in advance!\nRubén\n\n---\n[1] http://stackoverflow.com/questions/298314/find-files-in-git-repo-over-x-megabytes-that-dont-exist-in-head\n[2] http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/\n[3] http://markmail.org/message/762zzg5zckbiq2i7\n"},{"id":"162811","messageId":"4D721552.4090205@obry.net","threadId":"26664","inReplyTo":"AANLkTi=V3nEamocowbHovvV0U69nZgD70fysu1CQOwrR@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Pascal Obry","fromEmail":"pascal@obry.net","sentAt":"2011-03-05T10:49:54Z","receivedAt":"2011-03-05T10:49:54Z","isPatch":false,"sender":{"key":"pascal@obry.net","avatar":"https://avatars.githubusercontent.com/u/467069?v=4"},"body":"Le 05/03/2011 11:05, Ruben Laguna a écrit :\n> Is there any other thing I could do to reduce and audit the repository size?\n\n $ git gc\n\n?\n\n-- \n\n--|------------------------------------------------------\n--| Pascal Obry Team-Ada Member\n--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE\n--|------------------------------------------------------\n--| http://www.obry.net - http://v2p.fr.eu.org\n--| \"The best way to travel is by means of imagination\"\n--|\n--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B\n"},{"id":"162812","messageId":"AANLkTimp8B5Lv15qhGOwOzh+kqOS0g3Xwvgib8vyk+m+@mail.gmail.com","threadId":"26664","inReplyTo":"AANLkTi=V3nEamocowbHovvV0U69nZgD70fysu1CQOwrR@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Jonathan del Strother","fromEmail":"maillist@steelskies.com","sentAt":"2011-03-05T10:49:58Z","receivedAt":"2011-03-05T10:49:58Z","isPatch":false,"sender":{"key":"jon.delstrother@bestbefore.tv","avatar":"https://gravatar.com/avatar/754e21ab701c00e2d21fc261187254c34b2a1c0b959d9ee5be1a295990be3081?d=mp&s=160"},"body":"On 5 March 2011 10:05, Ruben Laguna wrote:\n> Hi,\n>\n> I had a repo which was big 143MB because it contained a bunch of jar\n> files. So I decided to remove those completely from the history.\n>\n> In short I used the git-large-blob [1] to find all the jars and used\n> the git-remove-history script [2] which does the filter-branch thing,\n> prune, etc.\n>\n> I did this on all branches (that I know of) and now I can see that the\n> jars are gone because I can't find them with git-large-blob.  and the\n> repo size has dropped from 143Mb to 87Mb.\n>\n> My concern is that 87Mb is still really big taking into account he\n> size of the project.  in fact if I run \"git diff-tree -r -p $commit\n> |wc -c\" for each commit and sum all I get 5.5Mb.\n>\n>\n> I also ran the git-rev-size [3] script that I found in this mailing\n> list and I only see that the size grows steadly from commit to commit\n> up to 1482731 bytes. So again how come the .git directory is 87MB?\n>\n>\n> So, Can anybody tell me if this repository size is \"normal\" for a\n> project with 1.4MB source and 352 commits?\n> Is there a better way to calculate the size (in bytes) of each commit?\n>\n> Is there any other thing I could do to reduce and audit  the repository size?\n>\n>\n> Thanks in advance!\n> Rubén\n>\n> ---\n> [1] http://stackoverflow.com/questions/298314/find-files-in-git-repo-over-x-megabytes-that-dont-exist-in-head\n> [2] http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/\n> [3] http://markmail.org/message/762zzg5zckbiq2i7\n\nWhat happens if you clone that repo?\ngit-gc will only pruned unused objects that're older than 2 weeks by\ndefault, so it's possible that your repo size will suddenly shrink in\n2 weeks time (or sooner, if you run git-gc with the appropriate\noptions)\n"},{"id":"162816","messageId":"AANLkTinwHMULqPZSguYtJztuA4Oy6-s6Ah3_tcVVO7D9@mail.gmail.com","threadId":"26664","inReplyTo":"AANLkTimp8B5Lv15qhGOwOzh+kqOS0g3Xwvgib8vyk+m+@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Ruben Laguna","fromEmail":"ruben.laguna@gmail.com","sentAt":"2011-03-05T11:41:22Z","receivedAt":"2011-03-05T11:41:22Z","isPatch":false,"sender":{"key":"ruben.laguna@gmail.com","avatar":"https://gravatar.com/avatar/ab5036870c2456b61092d6e238eed916343a69d86721685c7012b50c39d95a61?d=mp&s=160"},"body":"well, the git-remove-history script does\n\nrm -rf .git/refs/original/\ngit reflog expire --expire=now --all\ngit fsck --unreachable\ngit gc --prune=now\ngit gc --aggressive --prune=now\n\n\nafter filter-branch so I don't think it's that.\n\nalso cloning the repo doesn't change a thing\n\n$ git clone en4j en4j_xx\nCloning into en4j_xx...\ndone.\n$ cd en4j_xx\n$ du -sh .git\n 87M .git\n\nany other idea?\n\nOn Sat, Mar 5, 2011 at 11:49 AM, Jonathan del Strother\n wrote:\n> On 5 March 2011 10:05, Ruben Laguna wrote:\n>> Hi,\n>>\n>> I had a repo which was big 143MB because it contained a bunch of jar\n>> files. So I decided to remove those completely from the history.\n>>\n>> In short I used the git-large-blob [1] to find all the jars and used\n>> the git-remove-history script [2] which does the filter-branch thing,\n>> prune, etc.\n>>\n>> I did this on all branches (that I know of) and now I can see that the\n>> jars are gone because I can't find them with git-large-blob.  and the\n>> repo size has dropped from 143Mb to 87Mb.\n>>\n>> My concern is that 87Mb is still really big taking into account he\n>> size of the project.  in fact if I run \"git diff-tree -r -p $commit\n>> |wc -c\" for each commit and sum all I get 5.5Mb.\n>>\n>>\n>> I also ran the git-rev-size [3] script that I found in this mailing\n>> list and I only see that the size grows steadly from commit to commit\n>> up to 1482731 bytes. So again how come the .git directory is 87MB?\n>>\n>>\n>> So, Can anybody tell me if this repository size is \"normal\" for a\n>> project with 1.4MB source and 352 commits?\n>> Is there a better way to calculate the size (in bytes) of each commit?\n>>\n>> Is there any other thing I could do to reduce and audit  the repository size?\n>>\n>>\n>> Thanks in advance!\n>> Rubén\n>>\n>> ---\n>> [1] http://stackoverflow.com/questions/298314/find-files-in-git-repo-over-x-megabytes-that-dont-exist-in-head\n>> [2] http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/\n>> [3] http://markmail.org/message/762zzg5zckbiq2i7\n>\n> What happens if you clone that repo?\n> git-gc will only pruned unused objects that're older than 2 weeks by\n> default, so it's possible that your repo size will suddenly shrink in\n> 2 weeks time (or sooner, if you run git-gc with the appropriate\n> options)\n>\n\n\n\n-- \n/Rubén\n"},{"id":"162819","messageId":"m2zkp9wwqe.fsf@igel.home","threadId":"26664","inReplyTo":"AANLkTinwHMULqPZSguYtJztuA4Oy6-s6Ah3_tcVVO7D9@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2011-03-05T12:57:45Z","receivedAt":"2011-03-05T12:57:45Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Ruben Laguna writes:\n\n> also cloning the repo doesn't change a thing\n>\n> $ git clone en4j en4j_xx\n> Cloning into en4j_xx...\n> done.\n> $ cd en4j_xx\n> $ du -sh .git\n> 87M .git\n>\n> any other idea?\n\nPlease use file://$PWD/en4j as URL, otherwise git clone just hard links\neverything.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"162899","messageId":"AANLkTinJZFGVNSQVfCipo33h5uPpK0pFY10E203oTfhU@mail.gmail.com","threadId":"26664","inReplyTo":"m2zkp9wwqe.fsf@igel.home","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Ruben Laguna","fromEmail":"ruben.laguna@gmail.com","sentAt":"2011-03-07T09:59:29Z","receivedAt":"2011-03-07T09:59:29Z","isPatch":false,"sender":{"key":"ruben.laguna@gmail.com","avatar":"https://gravatar.com/avatar/ab5036870c2456b61092d6e238eed916343a69d86721685c7012b50c39d95a61?d=mp&s=160"},"body":"Cloning it that way didn't help either,\n\nBut I have more info\n\nIf I set a bare repo and push my four branches to it (master, develop,\ngh-pages and experimental) the total size of the repo is 2.4MB\n(instead of 87MB)\n\n\n$ git init --bare en4j_xx\n$ cd en4j\n$ git checkout master\n$ git push file://$PWD/../en4j_xx master\n$ git checkout develop\n$ git push file://$PWD/../en4j_xx develop\n$ git checkout experimental\n$ git push file://$PWD/../en4j_xx experimental\n$ git checkout gh-pages\n$ git push file://$PWD/../en4j_xx gh-pages\n$ $ du -sh ../en4j_xx\n2.3M\t../en4j_xx\n\nSo, how can I find the contents present in en4j that are not present in en4j_xx?\n\n\n\n\n\n\nOn Sat, Mar 5, 2011 at 1:57 PM, Andreas Schwab wrote:\n> Ruben Laguna writes:\n>\n>> also cloning the repo doesn't change a thing\n>>\n>> $ git clone en4j en4j_xx\n>> Cloning into en4j_xx...\n>> done.\n>> $ cd en4j_xx\n>> $ du -sh .git\n>>  87M    .git\n>>\n>> any other idea?\n>\n> Please use file://$PWD/en4j as URL, otherwise git clone just hard links\n> everything.\n>\n> Andreas.\n>\n> --\n> Andreas Schwab, schwab@linux-m68k.org\n> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n> \"And now for something completely different.\"\n>\n\n\n\n-- \n/Rubén\n"},{"id":"163026","messageId":"4D769EBE.40008@cfl.rr.com","threadId":"26664","inReplyTo":"AANLkTinJZFGVNSQVfCipo33h5uPpK0pFY10E203oTfhU@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Phillip Susi","fromEmail":"psusi@cfl.rr.com","sentAt":"2011-03-08T21:25:18Z","receivedAt":"2011-03-08T21:25:18Z","isPatch":false,"sender":{"key":"psusi@cfl.rr.com","avatar":null},"body":"On 3/7/2011 4:59 AM, Ruben Laguna wrote:\n> Cloning it that way didn't help either,\n> \n> But I have more info\n> \n> If I set a bare repo and push my four branches to it (master, develop,\n> gh-pages and experimental) the total size of the repo is 2.4MB\n> (instead of 87MB)\n\nThen there are other branches using that space. Run git branch -a and\nsee what else is there.\n"},{"id":"163030","messageId":"AANLkTikuuzHZ897kOY2u0Sv=0JTDffo0UhcxkyynVQAZ@mail.gmail.com","threadId":"26664","inReplyTo":"AANLkTi=V3nEamocowbHovvV0U69nZgD70fysu1CQOwrR@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Tor Arvid Lund","fromEmail":"torarvid@gmail.com","sentAt":"2011-03-08T21:44:00Z","receivedAt":"2011-03-08T21:44:00Z","isPatch":false,"sender":{"key":"torarvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/439758?v=4"},"body":"On Sat, Mar 5, 2011 at 11:05 AM, Ruben Laguna wrote:\n> Hi,\n>\n> I had a repo which was big 143MB because it contained a bunch of jar\n> files. So I decided to remove those completely from the history.\n>\n> In short I used the git-large-blob [1] to find all the jars and used\n> the git-remove-history script [2] which does the filter-branch thing,\n> prune, etc.\n>\n> I did this on all branches (that I know of) and now I can see that the\n> jars are gone because I can't find them with git-large-blob.  and the\n> repo size has dropped from 143Mb to 87Mb.\n\nI just thought I'd mention that the git-remove-history script that you\nmention does filter-branch on HEAD, and not using the --all parameter.\nI thought --all was the best way to \"catch all\" branches in one go...\n\n -- Tor Arvid\n\n> My concern is that 87Mb is still really big taking into account he\n> size of the project.  in fact if I run \"git diff-tree -r -p $commit\n> |wc -c\" for each commit and sum all I get 5.5Mb.\n>\n>\n> I also ran the git-rev-size [3] script that I found in this mailing\n> list and I only see that the size grows steadly from commit to commit\n> up to 1482731 bytes. So again how come the .git directory is 87MB?\n>\n>\n> So, Can anybody tell me if this repository size is \"normal\" for a\n> project with 1.4MB source and 352 commits?\n> Is there a better way to calculate the size (in bytes) of each commit?\n>\n> Is there any other thing I could do to reduce and audit  the repository size?\n>\n>\n> Thanks in advance!\n> Rubén\n>\n> ---\n> [1] http://stackoverflow.com/questions/298314/find-files-in-git-repo-over-x-megabytes-that-dont-exist-in-head\n> [2] http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/\n> [3] http://markmail.org/message/762zzg5zckbiq2i7\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"163061","messageId":"AANLkTikxPPo4pGqrvo4rdDUOwp1PYYEdcERVfXfLnsVh@mail.gmail.com","threadId":"26664","inReplyTo":"AANLkTikuuzHZ897kOY2u0Sv=0JTDffo0UhcxkyynVQAZ@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Ruben Laguna","fromEmail":"ruben.laguna@gmail.com","sentAt":"2011-03-09T16:35:50Z","receivedAt":"2011-03-09T16:35:50Z","isPatch":false,"sender":{"key":"ruben.laguna@gmail.com","avatar":"https://gravatar.com/avatar/ab5036870c2456b61092d6e238eed916343a69d86721685c7012b50c39d95a61?d=mp&s=160"},"body":"> I just thought I'd mention that the git-remove-history script that you\n> mention does filter-branch on HEAD, and not using the --all parameter.\n> I thought --all was the best way to \"catch all\" branches in one go...\n>\n>    -- Tor Arvid\n>\n\nMuch faster this way, thanks Tor,\n\nBut it still gives the same result 88MB\n\n\n$ git branch -a\n* develop\n master\n remotes/origin/HEAD -> origin/develop\n remotes/origin/develop\n remotes/origin/experimental\n remotes/origin/gh-pages\n remotes/origin/master\n\nFinally I have deleted my public repo on github, created a new one and\npushed master and develop to the new empty one.\n\n\n-- \n/Rubén\n"},{"id":"163085","messageId":"AANLkTimYE8qCz98u3r2HRmeCx7k0wd_cWp-tR+tSoKbD@mail.gmail.com","threadId":"26664","inReplyTo":"AANLkTikxPPo4pGqrvo4rdDUOwp1PYYEdcERVfXfLnsVh@mail.gmail.com","subject":"Re: why is my repo size ten times bigger than what I expected?","fromName":"Tor Arvid Lund","fromEmail":"torarvid@gmail.com","sentAt":"2011-03-09T21:06:19Z","receivedAt":"2011-03-09T21:06:19Z","isPatch":false,"sender":{"key":"torarvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/439758?v=4"},"body":"On Wed, Mar 9, 2011 at 5:35 PM, Ruben Laguna wrote:\n>> I just thought I'd mention that the git-remove-history script that you\n>> mention does filter-branch on HEAD, and not using the --all parameter.\n>> I thought --all was the best way to \"catch all\" branches in one go...\n>>\n>>    -- Tor Arvid\n>>\n>\n> Much faster this way, thanks Tor,\n>\n> But it still gives the same result 88MB\n>\n>\n> $ git branch -a\n> * develop\n>  master\n>  remotes/origin/HEAD -> origin/develop\n>  remotes/origin/develop\n>  remotes/origin/experimental\n>  remotes/origin/gh-pages\n>  remotes/origin/master\n>\n> Finally I have deleted my public repo on github, created a new one and\n> pushed master and develop to the new empty one.\n\nAh, that's why I got only 3.6M when i cloned just now ;)\n\nFWIW (if you still want to figure it out...) - Whatever refs that your\norigin branches point to - their history and objects will *not* get\ndeleted by git gc/prune/whatever. So if they point to commits which\nhave these big jars in the history, that may be the cause. Also, when\nI do filter-branch, it saves the old refs in .git/refs/original so\nthat I can revert it all those times when I screw it up ;)\n\nBasically - since your \"new\" repo is so small, there is something in\nyour original repo that refers to your large objects.\n\nHave a good night.\n\n -- Tor Arvid\n"}]}