{"thread":{"id":"1676","subject":"[PATCH] git-repack-script: Add option to repack all objects","startedAt":"2005-08-27T08:41:11Z","lastAt":"2005-08-29T19:44:06Z","messageCount":10,"participants":["Frank Sorenson","Junio C Hamano","A Large Angry SCM"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"7842","messageId":"43102727.2050206@tuxrocks.com","threadId":"1676","inReplyTo":null,"subject":"[PATCH] git-repack-script: Add option to repack all objects","fromName":"Frank Sorenson","fromEmail":"frank@tuxrocks.com","sentAt":"2005-08-27T08:41:11Z","receivedAt":"2005-08-27T08:41:11Z","isPatch":true,"sender":{"key":"frank@tuxrocks.com","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nThis patch adds an option to git-repack-script to repack all objects,\nincluding both packed and unpacked.  This allows a full repack of\na git archive (current cogito packs from 39MB to 4.5MB, and git packs\nfrom 4.4MB to 3.8MB).\n\nSigned-off-by: Frank Sorenson <frank@tuxrocks.com>\n\ndiff --git a/git-repack-script b/git-repack-script\n- --- a/git-repack-script\n+++ b/git-repack-script\n@@ -5,10 +5,12 @@\n \n . git-sh-setup-script || die \"Not a git archive\"\n \t\n+repack_all=\n no_update_info=\n while case \"$#\" in 0) break ;; esac\n do\n \tcase \"$1\" in\n+\t--all)\trepack_all=t ;;\n \t-n)\tno_update_info=t ;;\n \t*)\tbreak ;;\n \tesac\n@@ -16,13 +18,22 @@ do\n done\n \n rm -f .tmp-pack-*\n- -packname=$(git-rev-list --unpacked --objects $(git-rev-parse --all) |\n- -\tgit-pack-objects --non-empty --incremental .tmp-pack) ||\n- -\texit 1\n- -if [ -z \"$packname\" ]; then\n- -\techo Nothing new to pack\n- -\texit 0\n- -fi\n+case \"$repack_all\" in\n+t)\tpackname=$(git-rev-list --objects $(git-rev-parse --all) |\n+\t\tgit-pack-objects .tmp-pack) ||\n+\t\texit 1\n+\tfind \"$GIT_OBJECT_DIRECTORY/\"?? -type f | xargs rm -f\n+\tfind \"$GIT_OBJECT_DIRECTORY/pack\" -type f | xargs rm -f\n+\t;;\n+*)\tpackname=$(git-rev-list --unpacked --objects $(git-rev-parse --all) |\n+\t\tgit-pack-objects --non-empty --incremental .tmp-pack) ||\n+\t\texit 1\n+\tif [ -z \"$packname\" ]; then\n+\t\techo Nothing new to pack\n+\t\texit 0\n+\tfi\n+\t;;\n+esac\n \n mkdir -p \"$GIT_OBJECT_DIRECTORY/pack\" &&\n mv .tmp-pack-$packname.pack \"$GIT_OBJECT_DIRECTORY/pack/pack-$packname.pack\" &&\n\n\nFrank\n- -- \nFrank Sorenson - KD7TZK\nSystems Manager, Computer Science Department\nBrigham Young University\nfrank@tuxrocks.com\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.1 (GNU/Linux)\nComment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org\n\niD8DBQFDECcnaI0dwg4A47wRAsigAKDEItbKTKAeO+PO8VV0dtMvFl0qfgCffyDc\nhL0nAUB0HxeDlDoh9fv2m4o=\n=r4gM\n-----END PGP SIGNATURE-----\n"},{"id":"7881","messageId":"7vbr3hlqjs.fsf@assigned-by-dhcp.cox.net","threadId":"1676","inReplyTo":"43102727.2050206@tuxrocks.com","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-28T21:06:31Z","receivedAt":"2005-08-28T21:06:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Frank Sorenson <frank@tuxrocks.com> writes:\n\n> This patch adds an option to git-repack-script to repack all objects,\n> including both packed and unpacked.  This allows a full repack of\n> a git archive (current cogito packs from 39MB to 4.5MB, and git packs\n> from 4.4MB to 3.8MB).\n>\n> Signed-off-by: Frank Sorenson <frank@tuxrocks.com>\n\nWhile I agree that giving more flexibility to repack objects is\na good idea, I am not sure rolling all existing objects into one\npack and removing the existing one is a good way to go.\n\nI'd do this slightly differently.  I do not think removing\nexisting pack belongs to this command.  We would probably want a\nseparate tool to find extra/redundant packs and remove them, or\nmore generally optimize packs by selectively exploding them and\nrepacking them (\"pack optimizer\").\n"},{"id":"7889","messageId":"4312BC27.9010604@tuxrocks.com","threadId":"1676","inReplyTo":"7vbr3hlqjs.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects","fromName":"Frank Sorenson","fromEmail":"frank@tuxrocks.com","sentAt":"2005-08-29T07:41:27Z","receivedAt":"2005-08-29T07:41:27Z","isPatch":true,"sender":{"key":"frank@tuxrocks.com","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nJunio C Hamano wrote:\n> Frank Sorenson <frank@tuxrocks.com> writes:\n> \n>>This patch adds an option to git-repack-script to repack all objects,\n>>including both packed and unpacked.  This allows a full repack of\n>>a git archive (current cogito packs from 39MB to 4.5MB, and git packs\n>>from 4.4MB to 3.8MB).\n>>\n>>Signed-off-by: Frank Sorenson <frank@tuxrocks.com>\n> \n> \n> While I agree that giving more flexibility to repack objects is\n> a good idea, I am not sure rolling all existing objects into one\n> pack and removing the existing one is a good way to go.\n\nIt reduces the disk space requirement significantly (linux packs from\n135MB to 73MB), and I'm seeing speed improvements as well (probably\nbecause cache-cold operation requires far less seeking, and the caching\nrequirements are smaller).\n\nWhat are the benefits to keeping old packs?\n\n> I'd do this slightly differently.  I do not think removing\n> existing pack belongs to this command.  We would probably want a\n> separate tool to find extra/redundant packs and remove them, or\n> more generally optimize packs by selectively exploding them and\n> repacking them (\"pack optimizer\").\n\nI disagree about not removing old packs.  When you \"repack\" your\nsuitcase, you take everything out and put it back in again, so a command\nnamed \"repack\" should remove all existing objects, and put them back again.\n\nOkay, so the pack algorithm could be better, but that only means that\nrepacking the entire set of objects would improve things more, making\nsome sort of \"git-repack-all\" an even more valuable operation.\n\nFrank\n- --\nFrank Sorenson - KD7TZK\nSystems Manager, Computer Science Department\nBrigham Young University\nfrank@tuxrocks.com\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.1 (GNU/Linux)\nComment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org\n\niD8DBQFDErwnaI0dwg4A47wRAkVGAKDqDjQ5IBTO+DC/nKpYl+69w7RESgCg6omQ\nxwbQqnXJnfxITC1TAjRtLSk=\n=tCyP\n-----END PGP SIGNATURE-----\n"},{"id":"7902","messageId":"7vvf1obsfc.fsf@assigned-by-dhcp.cox.net","threadId":"1676","inReplyTo":"4312BC27.9010604@tuxrocks.com","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-29T16:48:23Z","receivedAt":"2005-08-29T16:48:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Frank Sorenson <frank@tuxrocks.com> writes:\n\n> It reduces the disk space requirement significantly (linux packs from\n> 135MB to 73MB), and I'm seeing speed improvements as well (probably\n> because cache-cold operation requires far less seeking, and the caching\n> requirements are smaller).\n>\n> What are the benefits to keeping old packs?\n\nFor a private repository where one does development in and does\npush to public repositories from, packing everything into one\npack and pruning everything else (including old packs) is always\nthe optimum thing, if one can afford the time to repack.\n\nThere are no benefits to _keeping_ old packs, but there may be\nbenefits not to pack everything into one huge one when other\npeople are involved.\n\nSuppose I have currently three packs (one since the beginning of\ntime to some time ago, one incremental on top of it, another\nincremental on top of the other two).  Somebody cloned from my\nrepository reasonably early in the project timeline (he has only\nthe first pack), somebody else cloned yesterday (has all three\npacks).  And \"git count-objects\" reports many other objects are\nunpacked and I decide it is a time to repack.\n\nAt this point I could create everything into one new big pack\nand remove old packs.  Or I could create the fourth incremental.\nAnother possibility, and which is what I currently do by hand,\nis to create a pack that is incremental on top of the first two,\nand replace the latest incremental with it.\n\nNow these two people want to fetch from my repository while the\nthird person wants to clone from scratch.  Which repacking\nstrategy gives the best transfer to these three people?  Having\na single huge pack favors the newcomer and penalizes the old\ntimers.  Especially, the current http-pull does not have a smart\nto pick a better pack when an object is found in more than one\npacks, so leaving old packs around would not help.\n\nLeaving the old packs around could help all of them.  In the\nabove example, I could create the fourth incremental _and_ a\nsuperpack that has everything in it.  The newcomer would slurp\nin the superpack, the one with only the first pack can use one\nof the second+third+fourth or the superpack, and the one with\nall three can use the fourth pack.\n\nHaving said that, the packing has an interesting compression\ncharacteristics.  Repacking the three existing packs (from the\nexample) along with the unpacked objects into one pack would\nresult in a very small pack, compared to the sum of three\nexisting packs, depending on how often you repack.  In that\nsense, it may not be such a big deal to force everybody to\nre-fetch everything even if most of them are already locally\navailable, by repacking everything into one.\n\n> I disagree about not removing old packs.\n\nI am not saying we should not remove old pack.  I am saying that\nrepacking, choosing which pack to remove and doing the actual\nremoving should be kept as separate steps and in separate\ncommands, perhaps the latter two as part of \"git prune\".\n\n-jc\n"},{"id":"7904","messageId":"7vll2kbqa4.fsf_-_@assigned-by-dhcp.cox.net","threadId":"1676","inReplyTo":"7vvf1obsfc.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] git-repack-script: Add option to repack all objects.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-29T17:34:43Z","receivedAt":"2005-08-29T17:34:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This originally came from Frank Sorenson but with a bit of\nrework to allow future enhancement to the command without\nchanging the external interface for removal part.\n\nWith the '-a' option, all objects in the current repository are\npacked into a single pack.  When the '-d' option is given at the\nsame time, existing packs that were made redundant by this round\nof repacking are deleted.\n\nSince we currently have only two repacking strategies, one '-a'\n(everything into one) and the other not '-a' (incrementally pack\nonly the unpacked ones), '-d' is meaningful only used with '-a'\nand removes all the existing packs before repacking for now.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n    Junio C Hamano <junkio@cox.net> writes:\n    > I am not saying we should not remove old pack.  I am saying that\n    > repacking, choosing which pack to remove and doing the actual\n    > removing should be kept as separate steps and in separate\n    > commands, perhaps the latter two as part of \"git prune\".\n\n    Frank, this is what I meant by the above.  When we have pack\n    redundancy detection and removal in \"git prune\", probably we\n    would call it when '-d' is given instead of rolling our own\n    here.  That is, \"git repack [-a] -d\" would be just a\n    shorthand to say \"repack\" without '-d' immediately followed\n    by \"git prune --redundant-packs\".\n\n    Pack optimization idea itself might turn out to be not worth\n    it, in which case this version would suffice.  I don't know..\n\n git-repack-script |   51 +++++++++++++++++++++++++++++++++++++++++++--------\n 1 files changed, 43 insertions(+), 8 deletions(-)\n\n0c3d34bce4c44640a606e47c6346c400bc353604\ndiff --git a/git-repack-script b/git-repack-script\n--- a/git-repack-script\n+++ b/git-repack-script\n@@ -5,28 +5,63 @@\n \n . git-sh-setup-script || die \"Not a git archive\"\n \t\n-no_update_info=\n+no_update_info= all_into_one= remove_redundant=\n while case \"$#\" in 0) break ;; esac\n do\n \tcase \"$1\" in\n \t-n)\tno_update_info=t ;;\n+\t-a)\tall_into_one=t ;;\n+\t-d)\tremove_redandant=t ;;\n \t*)\tbreak ;;\n \tesac\n \tshift\n done\n \n rm -f .tmp-pack-*\n-packname=$(git-rev-list --unpacked --objects $(git-rev-parse --all) |\n-\tgit-pack-objects --non-empty --incremental .tmp-pack) ||\n+PACKDIR=\"$GIT_OBJECT_DIRECTORY/pack\"\n+\n+# There will be more repacking strategies to come...\n+case \",$all_into_one,\" in\n+,,)\n+\trev_list='--unpacked'\n+\trev_parse='--all'\n+\tpack_objects='--incremental'\n+\t;;\n+,t,)\n+\trev_list=\n+\trev_parse='--all'\n+\tpack_objects=\n+\t# This part is a stop-gap until we have proper pack redundancy\n+\t# checker.\n+\texisting=`cd \"$PACKDIR\" && \\\n+\t    find . -type f \\( -name '*.pack' -o -name '*.idx' \\) -print`\n+\t;;\n+esac\n+name=$(git-rev-list --objects $rev_list $(git-rev-parse $rev_parse) |\n+\tgit-pack-objects --non-empty $pack_objects .tmp-pack) ||\n \texit 1\n-if [ -z \"$packname\" ]; then\n-\techo Nothing new to pack\n+if [ -z \"$name\" ]; then\n+\techo Nothing new to pack.\n \texit 0\n fi\n+echo \"Pack pack-$name created.\"\n+\n+mkdir -p \"$PACKDIR\" || exit\n+\n+mv .tmp-pack-$name.pack \"$PACKDIR/pack-$name.pack\" &&\n+mv .tmp-pack-$name.idx  \"$PACKDIR/pack-$name.idx\" ||\n+exit\n+\n+if test \"$remove_redandant\" = t\n+then\n+\t# We know $existing are all redandant only when\n+\t# all-into-one is used.\n+\tif test \"$all_into_one\" != '' && test \"$existing\" != ''\n+\tthen\n+\t\t( cd \"$PACKDIR\" && rm -f $existing )\n+\tfi\n+fi\n \n-mkdir -p \"$GIT_OBJECT_DIRECTORY/pack\" &&\n-mv .tmp-pack-$packname.pack \"$GIT_OBJECT_DIRECTORY/pack/pack-$packname.pack\" &&\n-mv .tmp-pack-$packname.idx  \"$GIT_OBJECT_DIRECTORY/pack/pack-$packname.idx\" &&\n case \"$no_update_info\" in\n t) : ;;\n *) git-update-server-info ;;\n"},{"id":"7912","messageId":"43135414.6080400@gmail.com","threadId":"1676","inReplyTo":"7vll2kbqa4.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects.","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2005-08-29T18:29:40Z","receivedAt":"2005-08-29T18:29:40Z","isPatch":true,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> This originally came from Frank Sorenson but with a bit of\n> rework to allow future enhancement to the command without\n> changing the external interface for removal part.\n> \n> With the '-a' option, all objects in the current repository are\n> packed into a single pack.  When the '-d' option is given at the\n> same time, existing packs that were made redundant by this round\n> of repacking are deleted.\n> \n> Since we currently have only two repacking strategies, one '-a'\n> (everything into one) and the other not '-a' (incrementally pack\n> only the unpacked ones), '-d' is meaningful only used with '-a'\n> and removes all the existing packs before repacking for now.\n> \n[Rest of updated patch snipped]\n\nFrank,\n\nCan you produce a patch to update the git-repack-script documentation to \nreflect the new functionality?\n"},{"id":"7914","messageId":"7v3bosbn1o.fsf@assigned-by-dhcp.cox.net","threadId":"1676","inReplyTo":"43135414.6080400@gmail.com","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-29T18:44:35Z","receivedAt":"2005-08-29T18:44:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> writes:\n\n> Frank,\n>\n> Can you produce a patch to update the git-repack-script documentation to \n> reflect the new functionality?\n\nNot including the doc changes in the patch was my fault, but the\nmessage was meant primarily as an explanation of what I meant,\nnot for immediate inclusion in the master branch.\n\nI have some other documentation updates sitting in the proposed\nupdates, so I'd do it myself along with other manual pages if\nyou and Frank do not mind.\n\nIn any case, I first would like to make sure that the proposed\npatch you are replying to is something Frank agrees to.\n"},{"id":"7915","messageId":"43135AAA.4070406@gmail.com","threadId":"1676","inReplyTo":"7v3bosbn1o.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects.","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2005-08-29T18:57:46Z","receivedAt":"2005-08-29T18:57:46Z","isPatch":true,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> A Large Angry SCM <gitzilla@gmail.com> writes:\n> \n>>Frank,\n>>\n>>Can you produce a patch to update the git-repack-script documentation to \n>>reflect the new functionality?\n> \n> Not including the doc changes in the patch was my fault, but the\n> message was meant primarily as an explanation of what I meant,\n> not for immediate inclusion in the master branch.\n> \n> I have some other documentation updates sitting in the proposed\n> updates, so I'd do it myself along with other manual pages if\n> you and Frank do not mind.\n> \n> In any case, I first would like to make sure that the proposed\n> patch you are replying to is something Frank agrees to.\n> \n\nI sent my request to Frank because he was/is the sponsor of the change \nbut anyone can provide the documentation. :-)\n\nI think it'd be a good idea for documentation updates to accompany all \npatches (and for the maintainer to not be shy about asking for them).\n\nJust my $0.02 as I look at which commands have no documentation.\n"},{"id":"7916","messageId":"43135AF7.6070705@tuxrocks.com","threadId":"1676","inReplyTo":"7vll2kbqa4.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects.","fromName":"Frank Sorenson","fromEmail":"frank@tuxrocks.com","sentAt":"2005-08-29T18:59:03Z","receivedAt":"2005-08-29T18:59:03Z","isPatch":true,"sender":{"key":"frank@tuxrocks.com","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nJunio C Hamano wrote:\n> This originally came from Frank Sorenson but with a bit of\n> rework to allow future enhancement to the command without\n> changing the external interface for removal part.\n> \n> With the '-a' option, all objects in the current repository are\n> packed into a single pack.  When the '-d' option is given at the\n> same time, existing packs that were made redundant by this round\n> of repacking are deleted.\n> \n> Since we currently have only two repacking strategies, one '-a'\n> (everything into one) and the other not '-a' (incrementally pack\n> only the unpacked ones), '-d' is meaningful only used with '-a'\n> and removes all the existing packs before repacking for now.\n\nThank you for explaining the reasoning, and reworking the patch.  This\ndoes make more sense, and I can see the logic for leaving around the\npacks.  Coming from the perspective of the end user, I would probably\nwant to repack quite a bit more often to take advantage of the size and\nspeed advantages, while large public repositories will probably want to\nrepack at much longer periods.  Thanks for seeing both perspectives.  I\nlike your updated patch.\n\nFrank\n- --\nFrank Sorenson - KD7TZK\nSystems Manager, Computer Science Department\nBrigham Young University\nfrank@tuxrocks.com\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.2.6 (GNU/Linux)\nComment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org\n\niD8DBQFDE1r3aI0dwg4A47wRAldNAJ9J7wmyQMsMm5G0FgvOggc+QDtg/QCg0T+w\ny6A/46LYEr1zhFgxK6uKX0I=\n=z8uM\n-----END PGP SIGNATURE-----\n"},{"id":"7918","messageId":"7vslwsa5q1.fsf@assigned-by-dhcp.cox.net","threadId":"1676","inReplyTo":"43135AAA.4070406@gmail.com","subject":"Re: [PATCH] git-repack-script: Add option to repack all objects.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-29T19:44:06Z","receivedAt":"2005-08-29T19:44:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> writes:\n\n> ... (and for the maintainer to not be shy about asking for\n> them).\n\nPoint taken.\n"}]}