{"thread":{"id":"8469","subject":"[PATCH] [RFC] Generational repacking","startedAt":"2007-06-06T11:08:25Z","lastAt":"2007-06-07T22:51:53Z","messageCount":13,"participants":["Sam Vilain","Junio C Hamano","Dana How","Nicolas Pitre","Martin Langhoff"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"44158","messageId":"11811281053874-git-send-email-sam.vilain@catalyst.net.nz","threadId":"8469","inReplyTo":null,"subject":"[PATCH] [RFC] Generational repacking","fromName":"Sam Vilain","fromEmail":"sam.vilain@catalyst.net.nz","sentAt":"2007-06-06T11:08:25Z","receivedAt":"2007-06-06T11:08:25Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"This is a quick hack at generational repacking. The idea is that you\nsuccessively do larger repack runs as the number of packs accumulates.\n\nThe commandline interface for this should be considered development\ngrade only, and of course there are no tests and very verbose output\n:)\n\nThe useful invocation of this is git-repack -d -g\n\nThe -a option then becomes a degenerate case of generative repacking.\n\nThe intention is that this should end up light enough to be triggered\nautomatically whenever the (approximate) count of loose objects hits a\nthreshold, like 100 or 1000 - making git repositories \"maintenance\nfree\".\n---\n git-repack.sh | 79 ++++++++++++++++++++++++++++++++++++++++++++++++--------\n 1 files changed, 67 insertions(+), 12 deletions(-)\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 8c32724..b26ca2a 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -3,19 +3,21 @@\n # Copyright (c) 2005 Linus Torvalds\n #\n \n-USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--max-pack-size=N] [--window=N] [--depth=N]'\n+USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [-g] [--max-pack-size=N] [--window=N] [--depth=N]'\n SUBDIRECTORY_OK='Yes'\n . git-sh-setup\n \n-no_update_info= all_into_one= remove_redundant=\n-local= quiet= no_reuse= extra=\n+no_update_info= generations= remove_redundant=\n+local= quiet= no_reuse= extra= generation_width=\n while case \"$#\" in 0) break ;; esac\n do\n \tcase \"$1\" in\n \t-n)\tno_update_info=t ;;\n-\t-a)\tall_into_one=t ;;\n+\t-a)\tgenerations=0 ;;\n \t-d)\tremove_redundant=t ;;\n \t-q)\tquiet=-q ;;\n+\t-g)\tgenerations=3 generation_width=10 ;;\n+\t-G)\tgenerations=$2; generation_width=5; shift ;;\n \t-f)\tno_reuse=--no-reuse-object ;;\n \t-l)\tlocal=--local ;;\n \t--max-pack-size=*) extra=\"$extra $1\" ;;\n@@ -40,24 +42,74 @@ PACKTMP=\"$GIT_OBJECT_DIRECTORY/.tmp-$$-pack\"\n rm -f \"$PACKTMP\"-*\n trap 'rm -f \"$PACKTMP\"-*' 0 1 2 3 15\n \n+generation=\n+redundant=\n+\n # There will be more repacking strategies to come...\n-case \",$all_into_one,\" in\n+case \",$generations,\" in\n ,,)\n \targs='--unpacked --incremental'\n \t;;\n-,t,)\n+,*,)\n \tif [ -d \"$PACKDIR\" ]; then\n+\t\tmax_gen=0\n+\t\techo \"Scanning for packs... generations: $generations, generation_width: $generation_width\"\n \t\tfor e in `cd \"$PACKDIR\" && find . -type f -name '*.pack' \\\n \t\t\t| sed -e 's/^\\.\\///' -e 's/\\.pack$//'`\n \t\tdo\n+\t\t\techo -n \"what about $e ? \"\n \t\t\tif [ -e \"$PACKDIR/$e.keep\" ]; then\n \t\t\t\t: keep\n+\t\t\t\techo \"keep it - .keep exists\"\n \t\t\telse\n-\t\t\t\targs=\"$args --unpacked=$e.pack\"\n \t\t\t\texisting=\"$existing $e\"\n+\t\t\t\tif [ -e \"$PACKDIR/$e.gen\" ]; then\n+\t\t\t\t\tgen=`cat $PACKDIR/$e.gen`\n+\t\t\t\telse\n+\t\t\t\t\techo -n \"assuming \"\n+\t\t\t\t\tgen=1\n+\t\t\t\tfi\n+\t\t\t\techo \"generation $gen\"\n+\t\t\t\t[ \"$max_gen\" -lt $gen ] && max_gen=$gen\n+\t\t\t\teval \"gen_${gen}=\\\"\\$gen_${gen} $e\\\"\";\n+\t\t\t\teval \"c_gen_${gen}=\\$((\\$c_gen_${gen} + 1))\";\n \t\t\tfi\n \t\tdone\n+\t\ti=$max_gen\n+\t\tpacking=\n+\t\twhile [ $i -gt 0 ]\n+\t\tdo\n+\t\t\tc_gen=`eval \"echo \\\\\\$c_gen_$i\"`\n+\t\t\tpacks=`eval \"echo \\\\\\$gen_$i\"`\n+\t\t\tif [ -n \"$c_gen\" -a $i -gt \"$generations\" ]\n+\t\t\tthen\n+\t\t\t\techo \"saw $c_gen packs at generation $i\"\n+\t\t\t\techo \"therefore, repacking everything\"\n+\t\t\t\tpacking=1\n+\t\t\t\t[ -z \"$generation\" ] && generation=$(($i + 1))\n+\t\t\telif [ -n \"$c_gen\" -a \"$c_gen\" -ge \"$generation_width\" -a \"$i\" -lt \"$generations\" ]\n+\t\t\tthen\n+\t\t\t\techo -n \"generation $i has too many packs \"\n+\t\t\t\techo \"($c_gen >= $generation_width)\"\n+\t\t\t\techo \"repacking at this level and below\"\n+\t\t\t\tpacking=1\n+\t\t\t\t[ -z \"$generation\" ] && generation=$(($i + 1))\n+\t\t\tfi\n+\t\t\tif [ -n \"$packing\" ]\n+\t\t\tthen\n+\t\t\t\tfor x in $packs; do\n+\t\t\t\t\targs=\"$args --unpacked=$x.pack\"\n+\t\t\t\t\tredundant=\"$redundant $x\"\n+\t\t\t\tdone\n+\t\t\tfi\n+\t\t\ti=$(($i - 1))\n+\t\tdone\n+\t\tif [ -n \"$generation\" ]; then\n+\t\t\t[ \"$generation\" -gt \"$generations\" ] && generation=$generations\n+\t\t\t[ \"$generation\" -eq 0 ] && generation=1\n+\t\tfi\n \tfi\n+\n \t[ -z \"$args\" ] && args='--unpacked --incremental'\n \t;;\n esac\n@@ -95,20 +147,23 @@ for name in $names ; do\n \t\texit 1\n \t}\n \trm -f \"$PACKDIR/old-pack-$name.pack\" \"$PACKDIR/old-pack-$name.idx\"\n+\t[ -n \"$generation\" ] && echo $generation > \"$PACKDIR/pack-$name.gen\"\n done\n \n if test \"$remove_redundant\" = t\n then\n-\t# We know $existing are all redundant.\n-\tif [ -n \"$existing\" ]\n+ \techo \"removing redundant packs\"\n+\t# We know $redundant are all redundant.\n+\tif [ -n \"$redundant\" ]\n \tthen\n \t\tsync\n \t\t( cd \"$PACKDIR\" &&\n-\t\t for e in $existing\n+\t\t for e in $redundant\n \t\t do\n \t\t\tcase \" $fullbases \" in\n-\t\t\t*\" $e \"*) ;;\n-\t\t\t*)\trm -f \"$e.pack\" \"$e.idx\" \"$e.keep\" ;;\n+\t\t\t*\" $e \"*) echo \"ignoring $e\" ;;\n+\t\t\t*)\techo \"removing $e.pack etc\";\n+\t\t\t\trm -f \"$e.pack\" \"$e.idx\" \"$e.keep\" ;;\n \t\t\tesac\n \t\t done\n \t\t)\n-- \n1.5.2.0.45.gfea6d-dirty\n"},{"id":"44204","messageId":"7vlkewya4s.fsf@assigned-by-dhcp.cox.net","threadId":"8469","inReplyTo":"11811281053874-git-send-email-sam.vilain@catalyst.net.nz","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-06T22:46:11Z","receivedAt":"2007-06-06T22:46:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sam Vilain writes:\n\n> +\t\t\tc_gen=`eval \"echo \\\\\\$c_gen_$i\"`\n> +\t\t\tpacks=`eval \"echo \\\\\\$gen_$i\"`\n\nI used to write something like these myself when I was young\n;-), but it is enough to write:\n\n\teval 'c_gen=$c_gen_'$i\n\teval \"packs=\\$gen_$i\"\n"},{"id":"44207","messageId":"46673AF5.1070901@vilain.net","threadId":"8469","inReplyTo":"7vlkewya4s.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-06-06T22:53:41Z","receivedAt":"2007-06-06T22:53:41Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> Sam Vilain writes:\n> \n>> +\t\t\tc_gen=`eval \"echo \\\\\\$c_gen_$i\"`\n>> +\t\t\tpacks=`eval \"echo \\\\\\$gen_$i\"`\n>> \n>\n> I used to write something like these myself when I was young\n> ;-), but it is enough to write:\n>\n> \teval 'c_gen=$c_gen_'$i\n> \teval \"packs=\\$gen_$i\"\n> \n\nWould've re-written it in Perl ages ago :P\n\nSam.\n"},{"id":"44211","messageId":"56b7f5510706061704r34692c49v994ff368bbc12d05@mail.gmail.com","threadId":"8469","inReplyTo":"11811281053874-git-send-email-sam.vilain@catalyst.net.nz","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-06-07T00:04:46Z","receivedAt":"2007-06-07T00:04:46Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 6/6/07, Sam Vilain wrote:\n> This is a quick hack at generational repacking. The idea is that you\n> successively do larger repack runs as the number of packs accumulates.\n>\n> The commandline interface for this should be considered development\n> grade only, and of course there are no tests and very verbose output\n> :)\n>\n> The useful invocation of this is git-repack -d -g\n>\n> The -a option then becomes a degenerate case of generative repacking.\n>\n> The intention is that this should end up light enough to be triggered\n> automatically whenever the (approximate) count of loose objects hits a\n> threshold, like 100 or 1000 - making git repositories \"maintenance\n> free\".\n\nThis patch complicates git-repack.sh quite a bit and\nI'm unclear on what _problem_ you're addressing.\nThe recent LRU preferred pack patch\nreduces much of the value in keeping a repository tidy\n(\"tidy\" == \"few pack files\").\n\nAlready git-gc calls git-repack -a -d. How do you plan to change this?\nI wonder if you should be making git-gc more intelligent instead.\n\nAlso, you introduce a new pack properties file (.gen) which seems\nawkward to me.\n\nPerhaps something like this would be useful on a huge repository\nunder active use. But delta re-use makes full repacking quite quick for\na reasonably-sized repository already, and I don't see this being very useful\nfor a repository which is large due to large objects.\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"44218","messageId":"46676D44.7070703@vilain.net","threadId":"8469","inReplyTo":"56b7f5510706061704r34692c49v994ff368bbc12d05@mail.gmail.com","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-06-07T02:28:20Z","receivedAt":"2007-06-07T02:28:20Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Dana How wrote:\n> This patch complicates git-repack.sh quite a bit and\n> I'm unclear on what _problem_ you're addressing.\n\nThe problem is simple, and it is partially in the eye of the beholder.\n\nThat is;\n\n 1. without repacking, you get a lot of loose objects.\n - unnecessary disk space usage\n - bad performance on many OSes\n\n 2. repack takes too long to run very regularly; it's an occasional\n command.\n\n 3. the perception that git repositories are not maintenance free.\n\nWhat I'm aiming for is something which is light enough that it might\neven win back the performance loss you got from 1), and to solve the\nperception problem of 3).\n\nMuch as users who don't like automatic database maintenance turn it off\nand run it at the best time, advanced git users will want to disable\nthis feature in ~/.gitrc and run repack themselves when it suits them,\nor via cron or whatever. Or it's disabled by default and users that\nwhine get told to turn it on, it really doesn't matter. I can already\ndo it with a commit hook, so I'm quite happy.\n\n> The recent LRU preferred pack patch\n> reduces much of the value in keeping a repository tidy\n> (\"tidy\" == \"few pack files\").\n\nGreat, that is a good thing.\n\nPack files are an almost indistinguishable concept from database\npartitions. In terms of that, scaling problems with lots of partitions\ncan be managed, certainly.\n\nFor instance with database partitioning you would expect your query\nplanner (in this case, read_packed_sha1()) to be able to select the\nright partition (pack) to go to first to avoid excessive index lookups.\n That a strategy for picking the best pack quickly N% of the time exists\nfor git is an excellent measure to reduce the impact of a large number\nof pack files. I think you would probably find measurable wins by\nensuring that the gross number of packs is kept limited.\n\nConsider that I'm thinking of running this generational repack somewhere\nsuch as a commit hook, if it found >100 loose objects, so that the first\ngeneration repack is very quick and doesn't annoy me - and the second\ngeneration will similarly be fairly quick as many deltas will already be\ncomputed. The exact behaviour will probably require tuning to get a\ngood balance between good delta computation and minimal interruption to\ncommit flow. Someone on IRC floated the idea of making the first\ngeneration do no delta computation to make it lightning fast.\n\nNote that if you had 3 pack generations, only the first two levels will\never be repacked - you'll end up with an unlimited number of third\ngeneration packs, which will also end up in LRP* order.\n\n> Already git-gc calls git-repack -a -d. How do you plan to change this?\n> I wonder if you should be making git-gc more intelligent instead.\n> \n> Also, you introduce a new pack properties file (.gen) which seems\n> awkward to me.\n\nThis implementation is a simple demonstration of the logic which was\ndesigned to communicate the idea and stimulate discussion. I think the\nlogic could probably go elsewhere too, and yes the new file is a bit of\na hack.\n\nIt might be better to base the \"generation\" assessment of the file on\nthe actual size of the pack, for instance - ie, Instead of the number of\nloose objects, the size of the loose objects, call 1st generation = <1MB\npack, 2nd generation = <5MB, etc. When the combined size of 1st\ngeneration packs gets above 5MB then that generation is full and a new\n2nd generation pack is made. Then no state file is required.\n\n> Perhaps something like this would be useful on a huge repository\n> under active use. But delta re-use makes full repacking quite quick for\n> a reasonably-sized repository already, and I don't see this being very useful\n> for a repository which is large due to large objects.\n\nI agree with your point of view, however I think if the feature is out\nthere but disabled by default then this can be found through experience.\n As you can see all of the elements to implement it are already there -\nand as you mention, combining packs is already quick.\n\nSam.\n\n* Last Recently Packed ;)\n"},{"id":"44220","messageId":"alpine.LFD.0.99.0706062301210.12885@xanadu.home","threadId":"8469","inReplyTo":"56b7f5510706061704r34692c49v994ff368bbc12d05@mail.gmail.com","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-07T03:05:55Z","receivedAt":"2007-06-07T03:05:55Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 6 Jun 2007, Dana How wrote:\n\n> On 6/6/07, Sam Vilain wrote:\n> > This is a quick hack at generational repacking. The idea is that you\n> > successively do larger repack runs as the number of packs accumulates.\n> > \n> This patch complicates git-repack.sh quite a bit and\n> I'm unclear on what _problem_ you're addressing.\n> The recent LRU preferred pack patch\n> reduces much of the value in keeping a repository tidy\n> (\"tidy\" == \"few pack files\").\n[...]\n\nFor the record... I didn't comment on this patch because I couldn't \nform \na clear opinion about it.\n\nBut until its usefulness can be demonstrated I should side with Dana \nhere.\n\n\nNicolas\n"},{"id":"44221","messageId":"alpine.LFD.0.99.0706062314410.12885@xanadu.home","threadId":"8469","inReplyTo":"46676D44.7070703@vilain.net","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-07T03:20:59Z","receivedAt":"2007-06-07T03:20:59Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 7 Jun 2007, Sam Vilain wrote:\n\n> Dana How wrote:\n> > This patch complicates git-repack.sh quite a bit and\n> > I'm unclear on what _problem_ you're addressing.\n> \n> The problem is simple, and it is partially in the eye of the beholder.\n> \n> That is;\n> \n> 1. without repacking, you get a lot of loose objects.\n> - unnecessary disk space usage\n> - bad performance on many OSes\n\nNo argument.\n\n> 2. repack takes too long to run very regularly; it's an occasional\n> command.\n\nIt doesn't take long at all when you don't use -a.\n\n> 3. the perception that git repositories are not maintenance free.\n\nBut this perception is true! It is possible to automate it, but some \nmaintenance is necessary at some point.\n\n> What I'm aiming for is something which is light enough that it might\n> even win back the performance loss you got from 1), and to solve the\n> perception problem of 3).\n\nRun git-repack without -a from some hook. You can even launch it in the \nbackground.\n\nOr what am I missing?\n\n\nNicolas\n"},{"id":"44228","messageId":"466793EB.2060102@vilain.net","threadId":"8469","inReplyTo":"alpine.LFD.0.99.0706062314410.12885@xanadu.home","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-06-07T05:13:15Z","receivedAt":"2007-06-07T05:13:15Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n>> 2. repack takes too long to run very regularly; it's an occasional\n>> command.\n> It doesn't take long at all when you don't use -a.\n\nWell that depends how many loose objects there are :) I heard about on\nWindows a case where packing 30k loose objects took over an hour.\n\n>> What I'm aiming for is something which is light enough that it might\n>> even win back the performance loss you got from 1), and to solve the\n>> perception problem of 3).\n> \n> Run git-repack without -a from some hook. You can even launch it in the \n> background.\n> \n> Or what am I missing?\n\nIf you repack every 100 objects without -a, sure it will be fast, but\nyou'll end up with too many packs.\n\nSam.\n"},{"id":"44243","messageId":"alpine.LFD.0.99.0706070932000.12885@xanadu.home","threadId":"8469","inReplyTo":"466793EB.2060102@vilain.net","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-07T13:38:30Z","receivedAt":"2007-06-07T13:38:30Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 7 Jun 2007, Sam Vilain wrote:\n\n> Nicolas Pitre wrote:\n> >> 2. repack takes too long to run very regularly; it's an occasional\n> >> command.\n> > It doesn't take long at all when you don't use -a.\n> \n> Well that depends how many loose objects there are :) I heard about on\n> Windows a case where packing 30k loose objects took over an hour.\n\nAnd your patch cannot change anything to that, right?\nYou shouldn't wait until 30k loose objects accumulate before repacking.\n\n> >> What I'm aiming for is something which is light enough that it might\n> >> even win back the performance loss you got from 1), and to solve the\n> >> perception problem of 3).\n> > \n> > Run git-repack without -a from some hook. You can even launch it in the \n> > background.\n> > \n> > Or what am I missing?\n> \n> If you repack every 100 objects without -a, sure it will be fast, but\n> you'll end up with too many packs.\n\nYou just need to adjust this treshold of 100 objects.\n\nAnd latest GIT behaves _much_ better with lots of packs, almost like if \nthere was only one pack. See the test results I posted to the list.\n\n\nNicolas\n"},{"id":"44257","messageId":"46a038f90706071246y7dd14f55t199b8ed4e7617b68@mail.gmail.com","threadId":"8469","inReplyTo":"alpine.LFD.0.99.0706062314410.12885@xanadu.home","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2007-06-07T19:46:26Z","receivedAt":"2007-06-07T19:46:26Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 6/7/07, Nicolas Pitre wrote:\n> Run git-repack without -a from some hook. You can even launch it in the\n> background.\n\nI posted an RFC patch a while ago doing exactly that, and Linus shot\nit down, indicating we should instead print a message suggesting\nrepack to the user.\n\nRelevant thread around\nhttp://www.gelato.unsw.edu.au/archives/git/0606/22977.html\n\n> Or what am I missing?\n\nI don't think people were comfortable at the time with concurrent\nrepacks -- though the semantics are safe if we don't hit any bug. My\nguess is that noone wants to risk the .001% chances of data corruption\nfor this nice-to-have.\n\nIt was also probably a bad idea in my patch that it said -a -- it should just be\n\n git repack -l -q &\n\n[And I generally agree with the concerns about possibly broken\nsemantics, unexpected user actions and racing repacks. One of the main\nreasons I've never advocated SVN is that the early stories of data\ncorruption using BDB backends made me very very very wary of it.\nPerhaps because I lot months of work in the past to disk corruption in\na cvs repo, and have a good mental picture of the nervous breakdown\nthat followed.]\n\ncheers,\n\n\nm\n"},{"id":"44264","messageId":"4668789C.7010104@vilain.net","threadId":"8469","inReplyTo":"alpine.LFD.0.99.0706070932000.12885@xanadu.home","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-06-07T21:29:00Z","receivedAt":"2007-06-07T21:29:00Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n> >> Well that depends how many loose objects there are :) I heard about on\n>> Windows a case where packing 30k loose objects took over an hour.\n> > And your patch cannot change anything to that, right?\n> You shouldn't wait until 30k loose objects accumulate before repacking.\n\nAbsolutely. But the generational approach would make it practical to\nrepack very often, in fact automatically, and keep the time of that\nrepack reasonably bounded. As I said in the original patch, the\nintention is that this style of repacking works best when applied\nautomatically, perhaps on commit.\n\n>> If you repack every 100 objects without -a, sure it will be fast, but\n>> you'll end up with too many packs.\n> You just need to adjust this treshold of 100 objects.\n> And latest GIT behaves _much_ better with lots of packs, almost like if \n> there was only one pack. See the test results I posted to the list.\n\nThe test run with 66 packs? What about a repository with 300000 objects\nthat was repacked every 100 new objects, would it work well with with\n3000 packs? Surely there is some value in conglomerating older packs,\neven though your LRU patch might make it very small (which I do find\nimpressive). Do you really need me to prove this with a benchmark?\n\n\"Adjust the threshold\" you have already said. But the problem with that\nis what to? I want the automatic repack to be *fast*. So making it\nlarger will make it slower. But making it too small will make the packs\nexcessive.\n\nSam.\n"},{"id":"44265","messageId":"46687A73.6080100@vilain.net","threadId":"8469","inReplyTo":"46a038f90706071246y7dd14f55t199b8ed4e7617b68@mail.gmail.com","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-06-07T21:36:51Z","receivedAt":"2007-06-07T21:36:51Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Martin Langhoff wrote:\n> On 6/7/07, Nicolas Pitre wrote:\n>> Run git-repack without -a from some hook. You can even launch it in the\n>> background.\n> \n> I posted an RFC patch a while ago doing exactly that, and Linus shot\n> it down, indicating we should instead print a message suggesting\n> repack to the user.\n> \n> Relevant thread around\n> http://www.gelato.unsw.edu.au/archives/git/0606/22977.html\n> \n>> Or what am I missing?\n> \n> I don't think people were comfortable at the time with concurrent\n> repacks -- though the semantics are safe if we don't hit any bug. My\n> guess is that noone wants to risk the .001% chances of data corruption\n> for this nice-to-have.\n\nOk. But if repack is generational you probably don't mind waiting for\nit on commit so don't need to background it.\n\nSam.\n"},{"id":"44274","messageId":"46a038f90706071551w45592471g59a35abb871749bd@mail.gmail.com","threadId":"8469","inReplyTo":"46687A73.6080100@vilain.net","subject":"Re: [PATCH] [RFC] Generational repacking","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2007-06-07T22:51:53Z","receivedAt":"2007-06-07T22:51:53Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 6/8/07, Sam Vilain wrote:\n> Ok. But if repack is generational you probably don't mind waiting for\n> it on commit so don't need to background it.\n\nAgreed. The patch I posted back then was on fetch as a signle fetch is\nmore likely to bring a lot of new objects in. (Moreso than a commit)\n\nRepack-on-commit if there's more than 1k unpacked objects makes sense.\nAre fetches nowaways always using the \"keep pack\" and \"fatten thin\npack\" semantics? If yes, then\n\ncheers,\n\n\n\nmartin\n"}]}