{"thread":{"id":"2471","subject":"Remove unneeded packs","startedAt":"2005-11-12T13:04:23Z","lastAt":"2005-11-13T23:13:12Z","messageCount":18,"participants":["Marcel Holtmann","Andreas Ericsson","Craig Schlenter","Petr Baudis","Lukas Sandström","Junio C Hamano","Sergey Vlasov","Josef Weidendorfer"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"11674","messageId":"1131800663.29461.11.camel@blade","threadId":"2471","inReplyTo":null,"subject":"Remove unneeded packs","fromName":"Marcel Holtmann","fromEmail":"marcel@holtmann.org","sentAt":"2005-11-12T13:04:23Z","receivedAt":"2005-11-12T13:04:23Z","isPatch":false,"sender":{"key":"marcel@holtmann.org","avatar":null},"body":"Hi guys,\n\nevery time Linus re-creates the pack for his linux-2.6 tree, I end up\nwith another pack. I use HTTP as transport and thus the new pack will be\ndownload (which is almost 100 MB), but that is fine. However it seems\nthat the old (previous) pack will never be deleted. For the no longer\nneeded object files I can use git-prune-packed, but the old pack I have\nto identify and delete by myself. Exists an easy and nice way to get rid\nof old unneeded packs? Can't git-prune-packed also do this job?\n\nRegards\n\nMarcel\n"},{"id":"11675","messageId":"4375EA80.7070405@op5.se","threadId":"2471","inReplyTo":"1131800663.29461.11.camel@blade","subject":"Re: Remove unneeded packs","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2005-11-12T13:13:36Z","receivedAt":"2005-11-12T13:13:36Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Marcel Holtmann wrote:\n> Hi guys,\n> \n> every time Linus re-creates the pack for his linux-2.6 tree, I end up\n> with another pack. I use HTTP as transport and thus the new pack will be\n> download (which is almost 100 MB), but that is fine. However it seems\n> that the old (previous) pack will never be deleted. For the no longer\n> needed object files I can use git-prune-packed, but the old pack I have\n> to identify and delete by myself. Exists an easy and nice way to get rid\n> of old unneeded packs? Can't git-prune-packed also do this job?\n> \n\nA patchset was posted to the list 2005-11-09 by Lukas Sandström, adding \n\"git-pack-intersect\" which was subsequently renamed to the more \nappropriate \"git-pack-redundant\".\n\nIf I remember the commit messages and understand your question correctly \nit does what you want.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"11676","messageId":"1131802238.29461.18.camel@blade","threadId":"2471","inReplyTo":"4375EA80.7070405@op5.se","subject":"Re: Remove unneeded packs","fromName":"Marcel Holtmann","fromEmail":"marcel@holtmann.org","sentAt":"2005-11-12T13:30:38Z","receivedAt":"2005-11-12T13:30:38Z","isPatch":false,"sender":{"key":"marcel@holtmann.org","avatar":null},"body":"Hi Andreas,\n\n> > every time Linus re-creates the pack for his linux-2.6 tree, I end up\n> > with another pack. I use HTTP as transport and thus the new pack will be\n> > download (which is almost 100 MB), but that is fine. However it seems\n> > that the old (previous) pack will never be deleted. For the no longer\n> > needed object files I can use git-prune-packed, but the old pack I have\n> > to identify and delete by myself. Exists an easy and nice way to get rid\n> > of old unneeded packs? Can't git-prune-packed also do this job?\n> > \n> \n> A patchset was posted to the list 2005-11-09 by Lukas Sandström, adding \n> \"git-pack-intersect\" which was subsequently renamed to the more \n> appropriate \"git-pack-redundant\".\n> \n> If I remember the commit messages and understand your question correctly \n> it does what you want.\n\nyou are right. It is exactly what I was looking for. I just saw it some\nminutes ago, when I pulled the latest git tree. However to make an old\nGCC 2.95 happy, the attached patch is needed.\n\nI am not sure if it is fully working. It deletes a lot of old packs, but\nin case of the linux-2.6 tree it leaves on additional behind.\n\n.git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.idx\n.git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.pack\n.git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.pack\n.git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.idx\n\nThe 4d76... is the current pack, but the b3c6... is an old one that is\nnot needed anymore.\n\nRegards\n\nMarcel\n\n\n\ndiff --git a/pack-redundant.c b/pack-redundant.c\nindex 1f8c577..4ed974e 100644\n--- a/pack-redundant.c\n+++ b/pack-redundant.c\n@@ -358,11 +358,11 @@ size_t sizeof_union(struct packed_git *p\n size_t get_pack_redundancy(struct pack_list *pl)\n {\n \tstruct pack_list *subset;\n+\tsize_t ret = 0;\n \n \tif (pl == NULL)\n \t\treturn 0;\n \n-\tsize_t ret = 0;\n \twhile ((subset = pl->next)) {\n \t\twhile(subset) {\n \t\t\tret += sizeof_union(pl->pack, subset->pack);\n"},{"id":"11678","messageId":"cae2e895f6598781f4f22b76e781684b@codefountain.com","threadId":"2471","inReplyTo":"1131800663.29461.11.camel@blade","subject":"Re: Remove unneeded packs","fromName":"Craig Schlenter","fromEmail":"craig@codefountain.com","sentAt":"2005-11-12T13:40:50Z","receivedAt":"2005-11-12T13:40:50Z","isPatch":false,"sender":{"key":"craig@codefountain.com","avatar":null},"body":"On 12 Nov 2005, at 3:04 PM, Marcel Holtmann wrote:\n\n> every time Linus re-creates the pack for his linux-2.6 tree, I end up\n> with another pack. I use HTTP as transport and thus the new pack will \n> be\n> download (which is almost 100 MB), but that is fine.\n> [snip]\n\nThe 100MB situation is not cool for those of us on a tight bandwidth\nbudget or slow links. Can anyone tell me if the native git protocol is\nany better at this stuff please?\n\nThanks,\n\n--Craig\n"},{"id":"11679","messageId":"20051112135947.GC30496@pasky.or.cz","threadId":"2471","inReplyTo":"cae2e895f6598781f4f22b76e781684b@codefountain.com","subject":"Balanced packing strategy","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-12T13:59:47Z","receivedAt":"2005-11-12T13:59:47Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Nov 12, 2005 at 02:40:50PM CET, I got a letter\nwhere Craig Schlenter said that...\n> On 12 Nov 2005, at 3:04 PM, Marcel Holtmann wrote:\n> \n> >every time Linus re-creates the pack for his linux-2.6 tree, I end up\n> >with another pack. I use HTTP as transport and thus the new pack will \n> >be\n> >download (which is almost 100 MB), but that is fine.\n> >[snip]\n> \n> The 100MB situation is not cool for those of us on a tight bandwidth\n> budget or slow links. Can anyone tell me if the native git protocol is\n> any better at this stuff please?\n\nYes, the native GIT protocol transfers only the objects you need.\n\nBut the 100MB situation is still bad. FWIW, this is my proposal I sent\nabout a month ago to some packs-related discussion at the kernel.org\nmailing list (ok, I updated it a little):\n\n\nThe repacking should be done in such a way to minimize the overhead for\nthe dumb transport users. Ideal for this is some structure like (at the\nend of october):\n\n\tyear2003.pack\n\tyear2004.pack\n\thalfyear2004-2.pack\n\thalfyear2005-1.pack\n\tmonth4.pack\n\tmonth5.pack\n\tmonth6.pack\n\tmonth7.pack\n\tmonth8.pack\n\tmonth9.pack\n\tweek37.pack\n\tweek38.pack\n\tweek39.pack\n\tweek40.pack\n\tweek41.pack\n\tweek42.pack\n\tweek43.pack\n\t\n\n\nThis has the property that the second half of given pack is covered by\nobjects with precision lower by one. This is a relatively high overload\n(this can be balanced by only keeping the last third or whatever), but\nit designed to reduce the overhead of fetching packs over dumb\ntransport. E.g. if it's almost the end of July and you last fetched at\nthe start of June, you will not have to get the whole halfyear2005-1\npack, but be able to catch up by just fetching month6 pack, and then few\nweek-packs.\n\nFor the autopacker (which should be ideally ran by some cronjob), this\nmeans packing new week each week and getting rid of a week worth of\nobjects, packing new month each month and getting rid of a month worth\nof objects, etc.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"11681","messageId":"b6abcb70496730705046934973221b93@codefountain.com","threadId":"2471","inReplyTo":"20051112135947.GC30496@pasky.or.cz","subject":"Re: Balanced packing strategy","fromName":"Craig Schlenter","fromEmail":"craig@codefountain.com","sentAt":"2005-11-12T15:14:36Z","receivedAt":"2005-11-12T15:14:36Z","isPatch":false,"sender":{"key":"craig@codefountain.com","avatar":null},"body":"On 12 Nov 2005, at 3:59 PM, Petr Baudis wrote:\n\n> Dear diary, on Sat, Nov 12, 2005 at 02:40:50PM CET, I got a letter\n> where Craig Schlenter said that...\n>> The 100MB situation is not cool for those of us on a tight bandwidth\n>> budget or slow links. Can anyone tell me if the native git protocol is\n>> any better at this stuff please?\n>\n> Yes, the native GIT protocol transfers only the objects you need.\n\nAh, magic, thanks!\n\n> But the 100MB situation is still bad. FWIW, this is my proposal I sent\n> about a month ago to some packs-related discussion at the kernel.org\n> mailing list (ok, I updated it a little):\n\nIt would be nice if there was some meaningful automatic packing that\ndidn't hurt \"non-git-aware protocol\" users.\n\nDoes the pack index file contain enough information to enable a client\nto send http byte range requests to grab individual objects from a pack?\nIt does seem to store object offsets but maybe I'm missing something ...\n\nThank you,\n\n--Craig\n"},{"id":"11702","messageId":"43766687.2000007@etek.chalmers.se","threadId":"2471","inReplyTo":"1131802238.29461.18.camel@blade","subject":"Re: Remove unneeded packs","fromName":"Lukas Sandström","fromEmail":"lukass@etek.chalmers.se","sentAt":"2005-11-12T22:02:47Z","receivedAt":"2005-11-12T22:02:47Z","isPatch":false,"sender":{"key":"luksan@gmail.com","avatar":"https://avatars.githubusercontent.com/u/152281?v=4"},"body":"Marcel Holtmann wrote:\n> you are right. It is exactly what I was looking for. I just saw it some\n> minutes ago, when I pulled the latest git tree. However to make an old\n> GCC 2.95 happy, the attached patch is needed.\n> \n> I am not sure if it is fully working. It deletes a lot of old packs, but\n> in case of the linux-2.6 tree it leaves on additional behind.\n> \n> .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.idx\n> .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.pack\n> .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.pack\n> .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.idx\n> \n> The 4d76... is the current pack, but the b3c6... is an old one that is\n> not needed anymore.\n> \n> Regards\n> \n> Marcel\n\nThis is most likley because the pack b3c6... contains unreachable objects.\ngit-pack-redundant only makes sure that all objects present in packfiles\nstill are present in packfiles after the redundant packs have been removed.\n\nThus, unreachable objects will also be considered as required.\n\nNote that I haven't checked if this is the cause in this particular case,\nbut I have the same packfiles (I use the HTTP transport too).\n\nI'm thinking of the possibility passing a list of objects to be ignored\non stdin to git-pack-redundant. This would hopefully solve this problem.\n\n/Lukas Sandström\n"},{"id":"11704","messageId":"1131833595.25203.10.camel@blade","threadId":"2471","inReplyTo":"43766687.2000007@etek.chalmers.se","subject":"Re: Remove unneeded packs","fromName":"Marcel Holtmann","fromEmail":"marcel@holtmann.org","sentAt":"2005-11-12T22:13:15Z","receivedAt":"2005-11-12T22:13:15Z","isPatch":false,"sender":{"key":"marcel@holtmann.org","avatar":null},"body":"Hi Lukas,\n\n> > you are right. It is exactly what I was looking for. I just saw it some\n> > minutes ago, when I pulled the latest git tree. However to make an old\n> > GCC 2.95 happy, the attached patch is needed.\n> > \n> > I am not sure if it is fully working. It deletes a lot of old packs, but\n> > in case of the linux-2.6 tree it leaves on additional behind.\n> > \n> > .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.idx\n> > .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.pack\n> > .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.pack\n> > .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.idx\n> > \n> > The 4d76... is the current pack, but the b3c6... is an old one that is\n> > not needed anymore.\n> \n> This is most likley because the pack b3c6... contains unreachable objects.\n> git-pack-redundant only makes sure that all objects present in packfiles\n> still are present in packfiles after the redundant packs have been removed.\n> \n> Thus, unreachable objects will also be considered as required.\n> \n> Note that I haven't checked if this is the cause in this particular case,\n> but I have the same packfiles (I use the HTTP transport too).\n\nmaybe these packs are from a previous bad update. The cloned repository\nI found it, is actually quite old. When I checked it with some others it\nseems that it works perfect.\n\n> I'm thinking of the possibility passing a list of objects to be ignored\n> on stdin to git-pack-redundant. This would hopefully solve this problem.\n\nSounds good, but I don't even know what objects are involved in this\ncase and stops it from being marked as redundant.\n\nRegards\n\nMarcel\n"},{"id":"11714","messageId":"7vveyxcm3p.fsf@assigned-by-dhcp.cox.net","threadId":"2471","inReplyTo":"b6abcb70496730705046934973221b93@codefountain.com","subject":"Re: Balanced packing strategy","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-13T02:34:02Z","receivedAt":"2005-11-13T02:34:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Craig Schlenter writes:\n\n> Does the pack index file contain enough information to enable a client\n> to send http byte range requests to grab individual objects from a pack?\n> It does seem to store object offsets...\n\nYes, it is certainly doable; there is enough information. I am\nnot sure if it is worth the complexity, though.\n\nMany objects are stored delitified, so your byte range requests\nwould return delta and base object name. After you read what\nwas returned and find out the base object name, you would need\nto get it, which can be another delta against its base object.\nThis would make tangling a delta chain would become a serialized\nsequence of requests.\n"},{"id":"11715","messageId":"7voe4pclwm.fsf@assigned-by-dhcp.cox.net","threadId":"2471","inReplyTo":"43766687.2000007@etek.chalmers.se","subject":"Re: Remove unneeded packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-13T02:38:17Z","receivedAt":"2005-11-13T02:38:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lukas Sandström writes:\n\n> This is most likley because the pack b3c6... contains unreachable objects.\n> git-pack-redundant only makes sure that all objects present in packfiles\n> still are present in packfiles after the redundant packs have been removed.\n> ...\n> I'm thinking of the possibility passing a list of objects to be ignored\n> on stdin to git-pack-redundant. This would hopefully solve this problem.\n\nBut once you go down that path, wouldn't doing 'repack -a -d'\nbecome looking simpler and more attractive, I wonder?\n"},{"id":"11722","messageId":"43771C43.7000104@etek.chalmers.se","threadId":"2471","inReplyTo":"7voe4pclwm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Remove unneeded packs","fromName":"Lukas Sandström","fromEmail":"lukass@etek.chalmers.se","sentAt":"2005-11-13T10:58:11Z","receivedAt":"2005-11-13T10:58:11Z","isPatch":false,"sender":{"key":"luksan@gmail.com","avatar":"https://avatars.githubusercontent.com/u/152281?v=4"},"body":"Junio C Hamano wrote:\n> Lukas Sandström writes:\n>>This is most likley because the pack b3c6... contains unreachable objects.\n>>git-pack-redundant only makes sure that all objects present in packfiles\n>>still are present in packfiles after the redundant packs have been removed.\n>>...\n>>I'm thinking of the possibility passing a list of objects to be ignored\n>>on stdin to git-pack-redundant. This would hopefully solve this problem.\n> \n> \n> But once you go down that path, wouldn't doing 'repack -a -d'\n> become looking simpler and more attractive, I wonder?\n> \n> \n\nIt depends on how expensive git-fsck-objects --full --unreacahble is versus\na full repack.\n\nHowerver, if I read the source correctly git-fsck-objects doesn't currently test \nthe reachablility of packed objects. This would have to change, and I'm not certain\nof how to do that properly.\n\nNote that the following patch is reqired if git-repack -a -d is to work as expected.\n(Remove all packs except the new one)\n\nBtw, I'm sending this patch in utf8, let's see if it works...\n\n----\nSubject: [PATCH] Make sure all old packfiles are removed when doing a full repack\n\nThis is nessecary because unrachable objects in packfiles makes git-pack-redundant\nflag them as non-redundant.\n\nSigned-off-by: Lukas Sandström \n\n---\n\n git-repack.sh | 16 +++++++++++++++-\n 1 files changed, 15 insertions(+), 1 deletions(-)\n\napplies-to: 9a0f0c748316751fbf593a21f2b16bcdd975095a\n08df1f641bd3f98a607a8413d647667adc18a633\ndiff --git a/git-repack.sh b/git-repack.sh\nindex f347207..293bb50 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -32,6 +32,8 @@ case \",$all_into_one,\" in\n \trev_list=\n \trev_parse='--all'\n \tpack_objects=\n+\texisting=`cd \"$PACKDIR\" && \\\n+\t find . -type f \\( -name '*.pack' -o -name '*.idx' \\) -print`\n \t;;\n esac\n if [ \"$local\" ]; then\n@@ -60,7 +62,19 @@ mv .tmp-pack-$name.pack \"$PACKDIR/pack-$\n mv .tmp-pack-$name.idx \"$PACKDIR/pack-$name.idx\" ||\n exit\n \n-if test \"$remove_redandant\" = t\n+if test \"$all_into_one\" = t\n+then\n+\tsync\n+\t( cd \"$PACKDIR\" &&\n+\t\tfor e in $existing\n+\t\tdo\n+\t\tcase \"$e\" in\n+\t\t./pack-$name.pack | ./pack-$name.idx) ;;\n+\t\t*)\trm -f $e ;;\n+\t\tesac\n+\t\tdone\n+\t)\n+else if test \"$remove_redandant\" = t\n then\n \tsync\n \tredundant=$(git-pack-redundant --all)\n---\n0.99.9.GIT\n"},{"id":"11724","messageId":"20051113110024.GM30496@pasky.or.cz","threadId":"2471","inReplyTo":"7vveyxcm3p.fsf@assigned-by-dhcp.cox.net","subject":"Re: Balanced packing strategy","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-13T11:00:24Z","receivedAt":"2005-11-13T11:00:24Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sun, Nov 13, 2005 at 03:34:02AM CET, I got a letter\nwhere Junio C Hamano said that...\n> Craig Schlenter writes:\n> \n> > Does the pack index file contain enough information to enable a client\n> > to send http byte range requests to grab individual objects from a pack?\n> > It does seem to store object offsets...\n> \n> Yes, it is certainly doable; there is enough information. I am\n> not sure if it is worth the complexity, though.\n\nI think we need either the balanced packing or this.\n\n> Many objects are stored delitified, so your byte range requests\n> would return delta and base object name. After you read what\n> was returned and find out the base object name, you would need\n> to get it, which can be another delta against its base object.\n> This would make tangling a delta chain would become a serialized\n> sequence of requests.\n\nSort the objects topologically, then get everything from the old heads\non. Obviously, this will not work so well when we get multiple heads in\nsingle pack, but either don't do that (would it be actually so bad if we\nwould create one pack per head?), or:\n\n (i) objects are topologically sorted\n (ii) objects introduced by a commit/tree are right after the commit or\n tree in the pack file\n (iii) index file contains parents list for each commit\n\nThis way, you can possibly run through the gaps, or if the gap is big\nenough, restart the request. You still will miss objects introduced by\ncommits in different branches, but in case of trees you can slurp the\ntrees at once again, and pick the individual objects otherwise; while\ndoing this second pass, you can apply the gaps strategy again.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"11726","messageId":"20051113150051.4a10365d.vsu@altlinux.ru","threadId":"2471","inReplyTo":"43771C43.7000104@etek.chalmers.se","subject":"Re: Remove unneeded packs","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-11-13T12:00:51Z","receivedAt":"2005-11-13T12:00:51Z","isPatch":false,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Sun, 13 Nov 2005 11:58:11 +0100 Lukas Sandstr__m wrote:\n\n> Subject: [PATCH] Make sure all old packfiles are removed when doing a full repack\n> \n> This is nessecary because unrachable objects in packfiles makes git-pack-redundant\n> flag them as non-redundant.\n> \n> Signed-off-by: Lukas Sandstr____m \n> \n> ---\n> \n> git-repack.sh | 16 +++++++++++++++-\n> 1 files changed, 15 insertions(+), 1 deletions(-)\n> \n> applies-to: 9a0f0c748316751fbf593a21f2b16bcdd975095a\n> 08df1f641bd3f98a607a8413d647667adc18a633\n> diff --git a/git-repack.sh b/git-repack.sh\n> index f347207..293bb50 100755\n> --- a/git-repack.sh\n> +++ b/git-repack.sh\n> @@ -32,6 +32,8 @@ case \",$all_into_one,\" in\n> \trev_list=\n> \trev_parse='--all'\n> \tpack_objects=\n> +\texisting=`cd \"$PACKDIR\" && \\\n> +\t find . -type f \\( -name '*.pack' -o -name '*.idx' \\) -print`\n> \t;;\n> esac\n> if [ \"$local\" ]; then\n> @@ -60,7 +62,19 @@ mv .tmp-pack-$name.pack \"$PACKDIR/pack-$\n> mv .tmp-pack-$name.idx \"$PACKDIR/pack-$name.idx\" ||\n> exit\n> \n> -if test \"$remove_redandant\" = t\n> +if test \"$all_into_one\" = t\n\nThis should be\n\nif test \"$all_into_one$remove_redandant\" = tt\n\n(otherwise \"git repack -a\" becomes the same as \"git repack -a -d\").\n\n> +then\n> +\tsync\n> +\t( cd \"$PACKDIR\" &&\n> +\t\tfor e in $existing\n> +\t\tdo\n> +\t\tcase \"$e\" in\n> +\t\t./pack-$name.pack | ./pack-$name.idx) ;;\n> +\t\t*)\trm -f $e ;;\n> +\t\tesac\n> +\t\tdone\n> +\t)\n> +else if test \"$remove_redandant\" = t\n> then\n> \tsync\n> \tredundant=$(git-pack-redundant --all)\n> ---\n> 0.99.9.GIT\n"},{"id":"11727","messageId":"43772C96.9030805@etek.chalmers.se","threadId":"2471","inReplyTo":"20051113150051.4a10365d.vsu@altlinux.ru","subject":"Re: Remove unneeded packs","fromName":"Lukas Sandström","fromEmail":"lukass@etek.chalmers.se","sentAt":"2005-11-13T12:07:50Z","receivedAt":"2005-11-13T12:07:50Z","isPatch":false,"sender":{"key":"luksan@gmail.com","avatar":"https://avatars.githubusercontent.com/u/152281?v=4"},"body":"Sergey Vlasov wrote:\n> On Sun, 13 Nov 2005 11:58:11 +0100 Lukas Sandström wrote:\n> \n> \n>>Subject: [PATCH] Make sure all old packfiles are removed when doing a full repack\n>>\n>>This is nessecary because unrachable objects in packfiles makes git-pack-redundant\n>>flag them as non-redundant.\n>>\n>>Signed-off-by: Lukas Sandström \n>>\n>>---\n>>\n>> git-repack.sh | 16 +++++++++++++++-\n>> 1 files changed, 15 insertions(+), 1 deletions(-)\n>>\n>>applies-to: 9a0f0c748316751fbf593a21f2b16bcdd975095a\n>>08df1f641bd3f98a607a8413d647667adc18a633\n>>diff --git a/git-repack.sh b/git-repack.sh\n>>index f347207..293bb50 100755\n>>--- a/git-repack.sh\n>>+++ b/git-repack.sh\n>>@@ -32,6 +32,8 @@ case \",$all_into_one,\" in\n>> \trev_list=\n>> \trev_parse='--all'\n>> \tpack_objects=\n>>+\texisting=`cd \"$PACKDIR\" && \\\n>>+\t find . -type f \\( -name '*.pack' -o -name '*.idx' \\) -print`\n>> \t;;\n>> esac\n>> if [ \"$local\" ]; then\n>>@@ -60,7 +62,19 @@ mv .tmp-pack-$name.pack \"$PACKDIR/pack-$\n>> mv .tmp-pack-$name.idx \"$PACKDIR/pack-$name.idx\" ||\n>> exit\n>> \n>>-if test \"$remove_redandant\" = t\n>>+if test \"$all_into_one\" = t\n> \n> \n> This should be\n> \n> if test \"$all_into_one$remove_redandant\" = tt\n> \n> (otherwise \"git repack -a\" becomes the same as \"git repack -a -d\").\n> \n> \n\nThis was the behaviour before git-pack-redundant, I just restored it.\nSomeone else gets to decide if git repack -a implies \"remove all old packs\".\n"},{"id":"11728","messageId":"20051113122017.GA9996@procyon.home","threadId":"2471","inReplyTo":"43772C96.9030805@etek.chalmers.se","subject":"Re: Remove unneeded packs","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-11-13T12:20:18Z","receivedAt":"2005-11-13T12:20:18Z","isPatch":false,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Sun, Nov 13, 2005 at 01:07:50PM +0100, Lukas Sandstr?m wrote:\n> Sergey Vlasov wrote:\n> > On Sun, 13 Nov 2005 11:58:11 +0100 Lukas Sandstr?m wrote:\n\n> >>-if test \"$remove_redandant\" = t\n> >>+if test \"$all_into_one\" = t\n> > \n> > \n> > This should be\n> > \n> > if test \"$all_into_one$remove_redandant\" = tt\n> > \n> > (otherwise \"git repack -a\" becomes the same as \"git repack -a -d\").\n> > \n> > \n> \n> This was the behaviour before git-pack-redundant, I just restored it.\n\nBut the old code was:\n\nif test \"$remove_redandant\" = t\nthen\n\t# We know $existing are all redandant only when\n\t# all-into-one is used.\n\tif test \"$all_into_one\" != '' && test \"$existing\" != ''\n\tthen\n\t\tsync\n\t\t( cd \"$PACKDIR\" &&\n\t\t for e in $existing\n\t\t do\n\t\t\tcase \"$e\" in\n\t\t\t./pack-$name.pack | ./pack-$name.idx) ;;\n\t\t\t*)\trm -f $e ;;\n\t\t\tesac\n\t\t done\n\t\t)\n\tfi\nfi\n\nSo without the -d option nothing was removed, even with -a.\n\n(And test \"$existing\" != '' might also be needed for some shells which\nare confused by the empty list in the for statement.)\n\n> Someone else gets to decide if git repack -a implies \"remove all old packs\".\n\nIf there is a separate -d option for this, just using -a probably\nshould not remove anything.\n"},{"id":"11729","messageId":"4377323B.4000203@etek.chalmers.se","threadId":"2471","inReplyTo":"20051113122017.GA9996@procyon.home","subject":"Re: Remove unneeded packs","fromName":"Lukas Sandström","fromEmail":"lukass@etek.chalmers.se","sentAt":"2005-11-13T12:31:55Z","receivedAt":"2005-11-13T12:31:55Z","isPatch":false,"sender":{"key":"luksan@gmail.com","avatar":"https://avatars.githubusercontent.com/u/152281?v=4"},"body":"Sergey Vlasov wrote:\n> On Sun, Nov 13, 2005 at 01:07:50PM +0100, Lukas Sandstr?m wrote:\n> \n>>Sergey Vlasov wrote:\n>>\n>>>On Sun, 13 Nov 2005 11:58:11 +0100 Lukas Sandstr?m wrote:\n> \n> \n>>>>-if test \"$remove_redandant\" = t\n>>>>+if test \"$all_into_one\" = t\n>>>\n>>>\n>>>This should be\n>>>\n>>>if test \"$all_into_one$remove_redandant\" = tt\n>>>\n>>>(otherwise \"git repack -a\" becomes the same as \"git repack -a -d\").\n>>>\n>>>\n>>\n>>This was the behaviour before git-pack-redundant, I just restored it.\n> \n> \n> But the old code was:\n> \n> if test \"$remove_redandant\" = t\n> then\n> \t# We know $existing are all redandant only when\n> \t# all-into-one is used.\n> \tif test \"$all_into_one\" != '' && test \"$existing\" != ''\n> \tthen\n> \t\tsync\n> \t\t( cd \"$PACKDIR\" &&\n> \t\t for e in $existing\n> \t\t do\n> \t\t\tcase \"$e\" in\n> \t\t\t./pack-$name.pack | ./pack-$name.idx) ;;\n> \t\t\t*)\trm -f $e ;;\n> \t\t\tesac\n> \t\t done\n> \t\t)\n> \tfi\n> fi\n> \n> So without the -d option nothing was removed, even with -a.\n> \nTrue. I forgot to look at the context around the changed lines...\nBtw, remove_redundant is misspellt.\n> (And test \"$existing\" != '' might also be needed for some shells which\n> are confused by the empty list in the for statement.)\n> \n> \n>>Someone else gets to decide if git repack -a implies \"remove all old packs\".\n> \n> \n> If there is a separate -d option for this, just using -a probably\n> should not remove anything.\n\nTrue, but you will have trouble removing stale packfiles if they contain\nunreachable objects unless you remove them when you create the -a pack.\n\nAnyway, ignore the patch above.\n"},{"id":"11739","messageId":"200511132106.29841.Josef.Weidendorfer@gmx.de","threadId":"2471","inReplyTo":"20051112135947.GC30496@pasky.or.cz","subject":"Re: Balanced packing strategy","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2005-11-13T20:06:29Z","receivedAt":"2005-11-13T20:06:29Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Saturday 12 November 2005 14:59, Petr Baudis wrote:\n> The repacking should be done in such a way to minimize the overhead for\n> the dumb transport users. Ideal for this is some structure like (at the\n> end of october):\n> \n> \tyear2003.pack\n> \tyear2004.pack\n> ...\n> \tweek42.pack\n> \tweek43.pack\n> \t\n\nI am not sure if it is really beneficial, as packs have the requirement\nto be self contained, so you get a lot of objects undeltified which could\nbe deltified in a better scheme (as eg. in git native protocol).\n\nAFAICS, the git native protocol (which is nothing more than a pack itself\nfor each transfer) even has this problem, too: If you are updating every\nday via git native, the sum of transfered bytes in a month will be a\nmultiple of one git transfer for all the month's changes.\n\nTo keep the pack self-containment property, but work better with dumb\ntransfers, we could introduce incremental packs:\n\nInstead of fully repacking, create a new pack by only appendending new\nobjects at the end of the pack. Thus, most objects will be appended in\ndeltified form, making the incremental addition quite small. The outcome\nwould be a totally new package.\n\nUnfortunately, I do not know the package format in detail, and hope that\nthis is possible at all.\n\nFor dumb protocols to take advantage of this, the information that the\nfirst part of a package is actually the same as another package has to\nbe stored somewhere visible.\nIf a client detects that it has the first part of a pack already locally,\nit would be enough to fetch only some the second part. \n\nThis is more or less the same as Pasky's solution, but by using incremental\npacks instead. I think that such incremental packing will not even take\nmuch more space that fully repacking.\n\nJosef\n"},{"id":"11746","messageId":"7vbr0o87lj.fsf@assigned-by-dhcp.cox.net","threadId":"2471","inReplyTo":"200511132106.29841.Josef.Weidendorfer@gmx.de","subject":"Re: Balanced packing strategy","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-13T23:13:12Z","receivedAt":"2005-11-13T23:13:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Petr Baudis writes:\n\n> This has the property that the second half of given pack is covered by\n> objects with precision lower by one. This is a relatively high overload\n> (this can be balanced by only keeping the last third or whatever), but\n> it designed to reduce the overhead of fetching packs over dumb\n> transport.\n\nI have a feeling that you would be better off if instead do the\nrepacking on the server side to prepare multiple packs, each of\nwhich has all the necessary objects to bring people who was\nup-to-date at various timerange ago, to arrange that you would\nneed only one patch fetch with individual objects near the tip.\n\nThis obviously needs smarter client-side support.\n\nSuppose we are somewhere after releasing v1.8 and inching\ntowards v1.9:\n\nIn your proposal, the object ranges each pack contains would\nlook like this:\n\n v1.0..v1.5 --------\n v1.5..v1.6 -----\n v1.6..v1.7 -------\n v1.7..v1.8 -----\n individual objects ....\n\nThat is, there are slight overlaps but you would do multiple\npacks if you are really behind.\n\nInstead, you could do this:\n\n v1.0..v1.8 ----------------------\n v1.5..v1.8 --------------\n v1.6..v1.8 ----------\n v1.7..v1.8 ----\n individual objects ....\n\nEverybody starts from the tip, fetching individual objects, and\nwhen the last repack boundary (the time we released 1.8) is\nreached, the dumb protocol downloader now faces a choice. The\nindices are fairly small, so you fetch all of them and see how\nmany objects you are lacking from each pack. If you were\nup-to-date very long time ago, say at v1.2, you would obviously\nneed to fetch the longest pack. If you were up-to-date\nrecently, say after v1.6 was released, you need to fetch smaller\npack.\n\nGiven the self containedness requirements, any path that is\ntouched once in a period needs at least one full copy of it in\neach pack (all other revisions could be deltified), and I\nsuspect in practice the oldest pack (v1.0..v1.5 pack in your\nscheme) would not save much space by not having v1.5..v1.8\nhistory. We could tweak things further to do something like\nthis:\n\n v1.0..v1.8 ------------------\n v1.5..v1.8 ----------\n v1.6..v1.8 ------\n v1.7..v1.8 ----\n individual objects ....\n\nto also account for a fact that the recent ones cover shorter\ntime range and not many paths are touched.\n"}]}