{"thread":{"id":"20071","subject":"[script] find largest pack objects","startedAt":"2009-07-10T01:16:50Z","lastAt":"2009-08-31T13:25:57Z","messageCount":4,"participants":["Antony Stubbs","Nicolas Pitre","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"117742","messageId":"A67AA762-487D-4CFB-B555-718C88C5787D@gmail.com","threadId":"20071","inReplyTo":null,"subject":"[script] find largest pack objects","fromName":"Antony Stubbs","fromEmail":"antony.stubbs@gmail.com","sentAt":"2009-07-10T01:16:50Z","receivedAt":"2009-07-10T01:16:50Z","isPatch":false,"sender":{"key":"antony.stubbs@gmail.com","avatar":"https://gravatar.com/avatar/c5404b8c9bc4a7eea80a093cbab861fc087b6ff46cf5fd9f719e4860302fd69b?d=mp&s=160"},"body":"Blog post about git pruning history and finding large objects in your  \nrepo: http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n\nThis is a script I put together after migrating the Spring Modules  \nproject from CVS, using git-cvsimport (which I also had to patch, to  \nget to work on OS X / MacPorts). I wrote it because I wanted to get  \nrid of all the large jar files, and documentation etc, that had been  \nput into source control. However, if _large files_ are deleted in the  \nlatest revision, then they can be hard to track down.\n\n#!/bin/bash\n#set -x\n\n# Shows you the largest objects in your repo's pack file.\n# Written for osx.\n#\n# @see http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n# @author Antony Stubbs\n\n# set the internal field spereator to line break, so that we can  \niterate easily over the verify-pack output\nIFS=$'\\n';\n\n# list all objects including their size, sort by size, take top 10\nobjects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v  \nchain | sort -k3nr | head`\n\necho \"All sizes are in kB's. The pack column is the size of the  \nobject, compressed, inside the pack file.\"\n\noutput=\"size,pack,SHA,location\"\nfor y in $objects\ndo\n\t# extract the size in bytes\n\tsize=$((`echo $y | cut -f 5 -d ' '`/1024))\n\t# extract the compressed size in bytes\n\tcompressedSize=$((`echo $y | cut -f 6 -d ' '`/1024))\n\t# extract the SHA\n\tsha=`echo $y | cut -f 1 -d ' '`\n\t# find the objects location in the repository tree\n\tother=`git rev-list --all --objects | grep $sha`\n\t#lineBreak=`echo -e \"\\n\"`\n\toutput=\"${output}\\n${size},${compressedSize},${other}\"\ndone\n\necho -e $output | column -t -s ', '\n\nMore info on the blog post: http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n\nRegards,\nAntony Stubbs\n\nTalk to me about Wicket, Spring, Maven consulting, small scale  \noutsourcing to Australasia and India and Open Source development!\n\nCheck out the Spring Modules fork at http://wiki.github.com/astubbs/spring-modules \n  ! We've just done the first release of the project in over a year!\n\nWebsite: http://sharca.com\nBlog: http://stubbisms.wordpress.com\nLinked In: http://www.linkedin.com/in/antonystubbs\nPodcast: http://www.illegalargument.com\n"},{"id":"117748","messageId":"alpine.LFD.2.00.0907092332030.21547@xanadu.home","threadId":"20071","inReplyTo":"A67AA762-487D-4CFB-B555-718C88C5787D@gmail.com","subject":"Re: [script] find largest pack objects","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-07-10T03:34:26Z","receivedAt":"2009-07-10T03:34:26Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 10 Jul 2009, Antony Stubbs wrote:\n\n> Blog post about git pruning history and finding large objects in your repo:\n> http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n> \n> This is a script I put together after migrating the Spring Modules project\n> from CVS, using git-cvsimport (which I also had to patch, to get to work on OS\n> X / MacPorts). I wrote it because I wanted to get rid of all the large jar\n> files, and documentation etc, that had been put into source control. However,\n> if _large files_ are deleted in the latest revision, then they can be hard to\n> track down.\n> \n> #!/bin/bash\n> #set -x\n> \n> # Shows you the largest objects in your repo's pack file.\n> # Written for osx.\n> #\n> # @see\n> http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n> # @author Antony Stubbs\n> \n> # set the internal field spereator to line break, so that we can iterate\n> easily over the verify-pack output\n> IFS=$'\\n';\n> \n> # list all objects including their size, sort by size, take top 10\n> objects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v chain |\n> sort -k3nr | head`\n> \n> echo \"All sizes are in kB's. The pack column is the size of the object,\n> compressed, inside the pack file.\"\n> \n> output=\"size,pack,SHA,location\"\n> for y in $objects\n> do\n> \t# extract the size in bytes\n> \tsize=$((`echo $y | cut -f 5 -d ' '`/1024))\n> \t# extract the compressed size in bytes\n> \tcompressedSize=$((`echo $y | cut -f 6 -d ' '`/1024))\n> \t# extract the SHA\n> \tsha=`echo $y | cut -f 1 -d ' '`\n> \t# find the objects location in the repository tree\n> \tother=`git rev-list --all --objects | grep $sha`\n> \t#lineBreak=`echo -e \"\\n\"`\n> \toutput=\"${output}\\n${size},${compressedSize},${other}\"\n> done\n> \n> echo -e $output | column -t -s ', '\n\nThis is certainly useful.  Mind submitting a patch adding this script to \ncontrib/stats/ ?\n\n\nNicolas\n"},{"id":"117765","messageId":"20090710114316.GA6880@atjola.homenet","threadId":"20071","inReplyTo":"A67AA762-487D-4CFB-B555-718C88C5787D@gmail.com","subject":"Re: [script] find largest pack objects","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2009-07-10T11:43:16Z","receivedAt":"2009-07-10T11:43:16Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2009.07.10 13:16:50 +1200, Antony Stubbs wrote:\n> Blog post about git pruning history and finding large objects in\n> your repo: http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n> \n> This is a script I put together after migrating the Spring Modules\n> project from CVS, using git-cvsimport (which I also had to patch, to\n> get to work on OS X / MacPorts). I wrote it because I wanted to get\n> rid of all the large jar files, and documentation etc, that had been\n> put into source control. However, if _large files_ are deleted in\n> the latest revision, then they can be hard to track down.\n\nHere's my script, basically for the same purpose, but instead of looking\nat the packfiles, it looks at the rev-list output to find those objects\nthat aren't prunable (ignoring the reflog). I'm also using some kind of\nugly sed invocation to run rev-list only twice, regardless of the number\nof objects to be shown, which greatly reduces the time required to run\nthe script.\n\n#!/bin/sh\ngit rev-list --all --objects |\n\tsed -n $(git rev-list --objects --all |\n\t\tcut -f1 -d' ' | git cat-file --batch-check | grep blob |\n\t\tsort -n -k3 | tail -n$1 | while read hash type size;\n\t\tdo\n\t\t\techo -n \"-e s/$hash/$size/p \";\n\t\tdone) |\n\tsort -n -k1\n\nIt takes the number of objects to be shown as an argument, so for the\ntop ten run as \"git find-large 10\" (assuming that the script is in $PATH\nand called git-find-large).\n\nIt doesn't list as much information as yours does, e.g. the compressed\nsize is missing, but it's good enough for me, and speed was far more\nimportant for me, especially since the \"rev-list --all --objects\" trick\ngets you only a single filename for the blob, so if there were renames,\nyou may need to run it again after having deleted one version via\nfilter-branch.\n\nSomething similar applies to deltified stuff. As verify-pack shows the\nsize of the delta, your script might miss some file B if that is a\ncurrently stored as a delta against some other large file A. Only after\nthe blob for A got deleted, B will be shown (as it is no longer\ndeltified).\n\nOTOH, this means that the output of my script is likely to have the same\nfilename over and over again. If that gets out of hand, I usually do\nsomething like:\ngit find-large 100 | cut -d' ' -f2 | sort -u\n\nSo I get just the filenames, hoping that the top 100 include all\ninteresting things ;-)\n\nMaybe this helps someone to come up with a smart combination of our\nscripts.\n\nBjörn\n"},{"id":"122186","messageId":"C0EFE78F-97D6-4287-832A-568AA07031A7@gmail.com","threadId":"20071","inReplyTo":"alpine.LFD.2.00.0907092332030.21547@xanadu.home","subject":"Re: [script] find largest pack objects","fromName":"Antony Stubbs","fromEmail":"antony.stubbs@gmail.com","sentAt":"2009-08-31T13:25:57Z","receivedAt":"2009-08-31T13:25:57Z","isPatch":false,"sender":{"key":"antony.stubbs@gmail.com","avatar":"https://gravatar.com/avatar/c5404b8c9bc4a7eea80a093cbab861fc087b6ff46cf5fd9f719e4860302fd69b?d=mp&s=160"},"body":"Sorry Nicolas,\nCompletely missed your message amongst the torrent of the git mailing  \nlist - which I'm now unsubscribed from.. But sure, I'll ad this to my  \ntodo list :)\n\nCheers,\nAntony Stubbs,\n\nsharca.com\n\nOn 10/07/2009, at 5:34 AM, Nicolas Pitre wrote:\n\n> On Fri, 10 Jul 2009, Antony Stubbs wrote:\n>\n>> Blog post about git pruning history and finding large objects in  \n>> your repo:\n>> http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n>>\n>> This is a script I put together after migrating the Spring Modules  \n>> project\n>> from CVS, using git-cvsimport (which I also had to patch, to get to  \n>> work on OS\n>> X / MacPorts). I wrote it because I wanted to get rid of all the  \n>> large jar\n>> files, and documentation etc, that had been put into source  \n>> control. However,\n>> if _large files_ are deleted in the latest revision, then they can  \n>> be hard to\n>> track down.\n>>\n>> #!/bin/bash\n>> #set -x\n>>\n>> # Shows you the largest objects in your repo's pack file.\n>> # Written for osx.\n>> #\n>> # @see\n>> http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/\n>> # @author Antony Stubbs\n>>\n>> # set the internal field spereator to line break, so that we can  \n>> iterate\n>> easily over the verify-pack output\n>> IFS=$'\\n';\n>>\n>> # list all objects including their size, sort by size, take top 10\n>> objects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v  \n>> chain |\n>> sort -k3nr | head`\n>>\n>> echo \"All sizes are in kB's. The pack column is the size of the  \n>> object,\n>> compressed, inside the pack file.\"\n>>\n>> output=\"size,pack,SHA,location\"\n>> for y in $objects\n>> do\n>> \t# extract the size in bytes\n>> \tsize=$((`echo $y | cut -f 5 -d ' '`/1024))\n>> \t# extract the compressed size in bytes\n>> \tcompressedSize=$((`echo $y | cut -f 6 -d ' '`/1024))\n>> \t# extract the SHA\n>> \tsha=`echo $y | cut -f 1 -d ' '`\n>> \t# find the objects location in the repository tree\n>> \tother=`git rev-list --all --objects | grep $sha`\n>> \t#lineBreak=`echo -e \"\\n\"`\n>> \toutput=\"${output}\\n${size},${compressedSize},${other}\"\n>> done\n>>\n>> echo -e $output | column -t -s ', '\n>\n> This is certainly useful.  Mind submitting a patch adding this  \n> script to\n> contrib/stats/ ?\n>\n>\n> Nicolas\n"}]}