{"thread":{"id":"11665","subject":"git-svn should default to --repack","startedAt":"2008-01-18T12:17:55Z","lastAt":"2008-01-23T19:22:23Z","messageCount":27,"participants":["Kevin Ballard","Karl Hasselström","Junio C Hamano","Harvey Harrison","Eric Wong","Sam Vilain","Mike Hommey","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"65865","messageId":"96C7A3A5-D750-43AB-A8A6-8A3A6D09AF4E@sb.org","threadId":"11665","inReplyTo":null,"subject":"git-svn should default to --repack","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-01-18T12:17:55Z","receivedAt":"2008-01-18T12:17:55Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"I was very surprised to find that git-svn does not in fact default to  \n--repack. I firmly believe it should. Here's an example as to why it  \nshould.\n\nI used git-svn to import a repository with 33000 revisions and about  \n7500 files. It took about 18 hours to import. When it was done,  \nmy .git folder had 242001 files that comprised 2.0GB. I ran `git gc -- \nagressive --prune` and let that sit overnight (I wish it was more  \nverbose, it went for over an hour without printing anything), and that  \nmanaged to compress the repo down to 334 files and 64MB.\n\nNow I have to figure out how to delete the .git folder from my regular  \nbackups.\n\nhttp://skitch.com/kballard/r7mn/results-of-git-gc-ono-macports-repo\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n\n\n"},{"id":"65890","messageId":"20080118155607.GA21236@diana.vm.bytemark.co.uk","threadId":"11665","inReplyTo":"96C7A3A5-D750-43AB-A8A6-8A3A6D09AF4E@sb.org","subject":"Re: git-svn should default to --repack","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-01-18T15:56:07Z","receivedAt":"2008-01-18T15:56:07Z","isPatch":false,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-01-18 07:17:55 -0500, Kevin Ballard wrote:\n\n> I was very surprised to find that git-svn does not in fact default\n> to --repack. I firmly believe it should.\n\nI believe so too. And nowadays there's \"git gc --auto\", which was made\nfor occasions such as this, so it should be a breeze to implement. The\noverhead might be low enough that it can be called after _every_\nimported revision.\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"65918","messageId":"7vfxwudftz.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"20080118155607.GA21236@diana.vm.bytemark.co.uk","subject":"Re: git-svn should default to --repack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-18T20:44:08Z","receivedAt":"2008-01-18T20:44:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karl Hasselström <kha@treskal.com> writes:\n\n> On 2008-01-18 07:17:55 -0500, Kevin Ballard wrote:\n>\n>> I was very surprised to find that git-svn does not in fact default\n>> to --repack. I firmly believe it should.\n>\n> I believe so too. And nowadays there's \"git gc --auto\", which was made\n> for occasions such as this, so it should be a breeze to implement. The\n> overhead might be low enough that it can be called after _every_\n> imported revision.\n\nCareful.  I made the same mistake and it had to be corrected\nwith e0cd252eb0ba6453acd64762625b004aa4cc162b.\n\n\"gc --auto\" after every 1000 or so feels like a good default and\nI would agree that would be a real fix to a real usability bug.\n\nPatches?\n"},{"id":"65960","messageId":"20080119123557.GA30778@diana.vm.bytemark.co.uk","threadId":"11665","inReplyTo":"7vfxwudftz.fsf@gitster.siamese.dyndns.org","subject":"Re: git-svn should default to --repack","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-01-19T12:35:57Z","receivedAt":"2008-01-19T12:35:57Z","isPatch":false,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-01-18 12:44:08 -0800, Junio C Hamano wrote:\n\n> Karl Hasselström <kha@treskal.com> writes:\n>\n> > I believe so too. And nowadays there's \"git gc --auto\", which was\n> > made for occasions such as this, so it should be a breeze to\n> > implement. The overhead might be low enough that it can be called\n> > after _every_ imported revision.\n>\n> Careful. I made the same mistake and it had to be corrected with\n> e0cd252eb0ba6453acd64762625b004aa4cc162b.\n>\n> \"gc --auto\" after every 1000 or so feels like a good default and I\n> would agree that would be a real fix to a real usability bug.\n\nI think 1000 might be too high; considering that (at least in my\nexperience) it takes on the order of 250-500 ms to import a commit,\nthe gc --auto overhead of maybe 10 ms isn't so bad.\n\nA good compromise might be to run gc --auto after every 10-100\ncommits, _and_ when the import is done.\n\nHowever, if gc --auto always takes a lot of time without accomplishing\nanything in the presence of too many unreachable loose objects it\nmight not be a good idea to run it at all, since the use of git-svn\ninvolves frequent rebasing.\n\n> Patches?\n\nJust hot air and noise for now from my end. Sorry.\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"65967","messageId":"BB932B28-08F3-4243-8E3B-ACC4527F947E@sb.org","threadId":"11665","inReplyTo":"20080119123557.GA30778@diana.vm.bytemark.co.uk","subject":"Re: git-svn should default to --repack","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-01-19T15:05:01Z","receivedAt":"2008-01-19T15:05:01Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"Note: CC list pruned as, once again, my Mail client decided to send  \nthe original message as HTML and it got bounced from the list.\nOriginal CC list: kha@treskal.com, gitster@pobox.com\n\nOn Jan 19, 2008, at 7:35 AM, Karl Hasselström wrote:\n\n> On 2008-01-18 12:44:08 -0800, Junio C Hamano wrote:\n>\n>> Karl Hasselström <kha@treskal.com> writes:\n>>\n>>> I believe so too. And nowadays there's \"git gc --auto\", which was\n>>> made for occasions such as this, so it should be a breeze to\n>>> implement. The overhead might be low enough that it can be called\n>>> after _every_ imported revision.\n>>\n>> Careful. I made the same mistake and it had to be corrected with\n>> e0cd252eb0ba6453acd64762625b004aa4cc162b.\n>>\n>> \"gc --auto\" after every 1000 or so feels like a good default and I\n>> would agree that would be a real fix to a real usability bug.\n>\n> I think 1000 might be too high; considering that (at least in my\n> experience) it takes on the order of 250-500 ms to import a commit,\n> the gc --auto overhead of maybe 10 ms isn't so bad.\n>\n> A good compromise might be to run gc --auto after every 10-100\n> commits, _and_ when the import is done.\n>\n> However, if gc --auto always takes a lot of time without accomplishing\n> anything in the presence of too many unreachable loose objects it\n> might not be a good idea to run it at all, since the use of git-svn\n> involves frequent rebasing.\n\nI don't know much about how this works, so if git gc --auto might have  \na problem, it seems the simplest fix for now would be to default git- \nsvn to having --repack=1000 on.\n\n>> Patches?\n>\n> Just hot air and noise for now from my end. Sorry.\n\nSame. I don't know Perl. Sorry.\n\n-Kevin Ballard\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n\n\n"},{"id":"65991","messageId":"20080119223249.8227.31460.stgit@yoghurt","threadId":"11665","inReplyTo":"20080119123557.GA30778@diana.vm.bytemark.co.uk","subject":"[PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-01-19T22:36:30Z","receivedAt":"2008-01-19T22:36:30Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Let \"git svn\" run \"git gc --auto\" every 100 imported commits, to\nreduce the number of loose objects.\n\nTo handle the common use case of frequent imports, where each\ninvocation typically fetches less than 100 commits, randomly set the\ncounter to something in the range 1-100 on initialization. It's almost\nas good as saving the counter, and much less of a hassle.\n\nOh, and 100 is just my best guess at a reasonable number. It could\nconceivably need tweaking.\n\nSigned-off-by: Karl Hasselström <kha@treskal.com>\n\n---\n\nOn 2008-01-19 13:35:57 +0100, Karl Hasselström wrote:\n\n> On 2008-01-18 12:44:08 -0800, Junio C Hamano wrote:\n> \n> > Patches?\n> \n> Just hot air and noise for now from my end. Sorry.\n\nOK, it didn't feel good saying that. So here's my attempt at being a\nmodel citizen. (It's not hard with a change this small ...)\n\nI'm not quite sure how this should interact with the --repack flag.\nRight now they just coexist, except for never running right after one\nanother, but conceivably we should do something cleverer. Eric?\n\n git-svn.perl |    7 ++++++-\n 1 files changed, 6 insertions(+), 1 deletions(-)\n\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 9f2b587..89e1d61 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1247,7 +1247,7 @@ use File::Path qw/mkpath/;\n use File::Copy qw/copy/;\n use IPC::Open3;\n \n-my $_repack_nr;\n+my ($_repack_nr, $_gc_nr, $_gc_period);\n # properties that we do not log:\n my %SKIP_PROP;\n BEGIN {\n@@ -1413,6 +1413,8 @@ sub init_vars {\n \t\t$_repack_nr = $_repack;\n \t\t$_repack_flags ||= '-d';\n \t}\n+\t$_gc_period = 100;\n+\t$_gc_nr = int(rand($_gc_period)) + 1;\n }\n \n sub verify_remotes_sanity {\n@@ -2157,6 +2159,9 @@ sub do_git_commit {\n \t\tprint \"Running git repack $_repack_flags ...\\n\";\n \t\tcommand_noisy('repack', split(/\\s+/, $_repack_flags));\n \t\tprint \"Done repacking\\n\";\n+\t} elsif (--$_gc_nr == 0) {\n+\t\t$_gc_nr = $_gc_period;\n+\t\tcommand_noisy('gc', '--auto');\n \t}\n \treturn $commit;\n }\n"},{"id":"65993","messageId":"1200783050.5724.196.camel@brick","threadId":"11665","inReplyTo":"20080119223249.8227.31460.stgit@yoghurt","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Harvey Harrison","fromEmail":"harvey.harrison@gmail.com","sentAt":"2008-01-19T22:50:50Z","receivedAt":"2008-01-19T22:50:50Z","isPatch":true,"sender":{"key":"harvey.harrison@gmail.com","avatar":null},"body":"On Sat, 2008-01-19 at 23:36 +0100, Karl Hasselström wrote:\n> Let \"git svn\" run \"git gc --auto\" every 100 imported commits, to\n> reduce the number of loose objects.\n\nI found 100 was a bit too low when doing some large repos, I've\nbeen using 1000.  I'd argue that --repack=1000 should be done by\ndefault.\n\n> I'm not quite sure how this should interact with the --repack flag.\n> Right now they just coexist, except for never running right after one\n> another, but conceivably we should do something cleverer. Eric?\n> \n\nHow about git gc always gets run at the very end of a git svn fetch?\n\nJust a thought.\n\nHarvey\n"},{"id":"66012","messageId":"20080120033737.GA7767@soma","threadId":"11665","inReplyTo":"1200783050.5724.196.camel@brick","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-01-20T03:37:37Z","receivedAt":"2008-01-20T03:37:37Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Harvey Harrison <harvey.harrison@gmail.com> wrote:\n> On Sat, 2008-01-19 at 23:36 +0100, Karl Hasselström wrote:\n> > Let \"git svn\" run \"git gc --auto\" every 100 imported commits, to\n> > reduce the number of loose objects.\n> \n> I found 100 was a bit too low when doing some large repos, I've\n> been using 1000.  I'd argue that --repack=1000 should be done by\n> default.\n\nI've found 100 for repack too low in the past, too, which is why\nrepack defaults to 1000 if no number is specified.  I think it\nshould hold for gc --auto, too.\n\n> > I'm not quite sure how this should interact with the --repack flag.\n> > Right now they just coexist, except for never running right after one\n> > another, but conceivably we should do something cleverer. Eric?\n\nI consider --repack is out-of-date now that we have gc --auto.  I'm in\nfavor of ripping out repack support in git-svn and just using gc --auto.\n\n> How about git gc always gets run at the very end of a git svn fetch?\n\nI'd much prefer that we run gc --auto at the end of every fetch instead\nof doing so randomly for small fetches.\n\n-- \nEric Wong\n"},{"id":"66024","messageId":"20080120093436.GA10924@diana.vm.bytemark.co.uk","threadId":"11665","inReplyTo":"20080120033737.GA7767@soma","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-01-20T09:34:36Z","receivedAt":"2008-01-20T09:34:36Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-01-19 19:37:37 -0800, Eric Wong wrote:\n\n> Harvey Harrison <harvey.harrison@gmail.com> wrote:\n>\n> > I found 100 was a bit too low when doing some large repos, I've\n> > been using 1000. I'd argue that --repack=1000 should be done by\n> > default.\n>\n> I've found 100 for repack too low in the past, too, which is why\n> repack defaults to 1000 if no number is specified. I think it should\n> hold for gc --auto, too.\n\nOK, I'll change it. But remember, gc --auto doesn't do _anything_\nunless it's deemed necessary, so it should behave much better than\njust plain repack. In theory at least.\n\n> I consider --repack is out-of-date now that we have gc --auto. I'm\n> in favor of ripping out repack support in git-svn and just using gc\n> --auto.\n\nWill do. What should I do with the repack commadline options? Keep\nthem for backwards compatibility but ignore them?\n\n> > How about git gc always gets run at the very end of a git svn\n> > fetch?\n>\n> I'd much prefer that we run gc --auto at the end of every fetch\n> instead of doing so randomly for small fetches.\n\nOK, will do. I'll just have to find a good spot to call it from. Hints\nwelcome.\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"66042","messageId":"7vlk6k8fyp.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"20080120093436.GA10924@diana.vm.bytemark.co.uk","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-20T19:17:02Z","receivedAt":"2008-01-20T19:17:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karl Hasselström <kha@treskal.com> writes:\n\n> On 2008-01-19 19:37:37 -0800, Eric Wong wrote:\n>\n>> Harvey Harrison <harvey.harrison@gmail.com> wrote:\n>>\n>> > I found 100 was a bit too low when doing some large repos, I've\n>> > been using 1000. I'd argue that --repack=1000 should be done by\n>> > default.\n>>\n>> I've found 100 for repack too low in the past, too, which is why\n>> repack defaults to 1000 if no number is specified. I think it should\n>> hold for gc --auto, too.\n>\n> OK, I'll change it. But remember, gc --auto doesn't do _anything_\n> unless it's deemed necessary, so it should behave much better than\n> just plain repack. In theory at least.\n\nCareful. I made the same mistake and it had to be corrected with\ne0cd252eb0ba6453acd64762625b004aa4cc162b.\n\nI think defaulting to --repack=1000 is a sane first step and you\nguys already have most code for it so that is a very safe thing.\n\nSwitching to \"gc --auto\" can be done early post 1.5.4, right?\n"},{"id":"66051","messageId":"20080120213847.9679.56653.stgit@yoghurt","threadId":"11665","inReplyTo":"20080120093436.GA10924@diana.vm.bytemark.co.uk","subject":"[PATCH 1/2] git-svn: Don't call git-repack anymore","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-01-20T21:39:51Z","receivedAt":"2008-01-20T21:39:51Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"In a moment, we'll start calling git-gc --auto instead, since it is a\nbetter fit to what we're trying to accomplish.\n\nThe command line options are still accepted, but don't have any\neffect, and we warn the user about that.\n\nSigned-off-by: Karl Hasselström <kha@treskal.com>\n\n---\n\nIs this close enough to what you intended?\n\n git-svn.perl |   14 ++------------\n 1 files changed, 2 insertions(+), 12 deletions(-)\n\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 9f2b587..988d8f6 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1247,7 +1247,6 @@ use File::Path qw/mkpath/;\n use File::Copy qw/copy/;\n use IPC::Open3;\n \n-my $_repack_nr;\n # properties that we do not log:\n my %SKIP_PROP;\n BEGIN {\n@@ -1408,10 +1407,8 @@ sub read_all_remotes {\n }\n \n sub init_vars {\n-\tif (defined $_repack) {\n-\t\t$_repack = 1000 if ($_repack <= 0);\n-\t\t$_repack_nr = $_repack;\n-\t\t$_repack_flags ||= '-d';\n+\tif (defined $_repack || defined $_repack_flags) {\n+               warn \"Repack options are obsolete; they have no effect.\\n\";\n \t}\n }\n \n@@ -2151,13 +2148,6 @@ sub do_git_commit {\n \t\t                   0, $self->svm_uuid);\n \t}\n \tprint \" = $commit ($self->{ref_id})\\n\";\n-\tif (defined $_repack && (--$_repack_nr == 0)) {\n-\t\t$_repack_nr = $_repack;\n-\t\t# repack doesn't use any arguments with spaces in them, does it?\n-\t\tprint \"Running git repack $_repack_flags ...\\n\";\n-\t\tcommand_noisy('repack', split(/\\s+/, $_repack_flags));\n-\t\tprint \"Done repacking\\n\";\n-\t}\n \treturn $commit;\n }\n \n"},{"id":"66052","messageId":"20080120214008.9679.69776.stgit@yoghurt","threadId":"11665","inReplyTo":"20080120093436.GA10924@diana.vm.bytemark.co.uk","subject":"[PATCH 2/2] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-01-20T21:40:41Z","receivedAt":"2008-01-20T21:40:41Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Let \"git svn\" run \"git gc --auto\" every 1000 imported commits to\nreduce the number of loose objects.\n\nTo handle the common use case of frequent imports, where each\ninvocation typically fetches much less than 1000 commits, also run gc\nunconditionally at the end of the import.\n\n\"1000\" is the same number that was used by default when we called\ngit-repack. It isn't necessarily still the best choice.\n\nSigned-off-by: Karl Hasselström <kha@treskal.com>\n\n---\n\n git-svn.perl |   12 ++++++++++++\n 1 files changed, 12 insertions(+), 0 deletions(-)\n\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 988d8f6..be4105c 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1247,6 +1247,8 @@ use File::Path qw/mkpath/;\n use File::Copy qw/copy/;\n use IPC::Open3;\n \n+my ($_gc_nr, $_gc_period);\n+\n # properties that we do not log:\n my %SKIP_PROP;\n BEGIN {\n@@ -1407,6 +1409,7 @@ sub read_all_remotes {\n }\n \n sub init_vars {\n+\t$_gc_nr = $_gc_period = 1000;\n \tif (defined $_repack || defined $_repack_flags) {\n                warn \"Repack options are obsolete; they have no effect.\\n\";\n \t}\n@@ -2095,6 +2098,10 @@ sub restore_commit_header_env {\n \t}\n }\n \n+sub gc {\n+\tcommand_noisy('gc', '--auto');\n+};\n+\n sub do_git_commit {\n \tmy ($self, $log_entry) = @_;\n \tmy $lr = $self->last_rev;\n@@ -2148,6 +2155,10 @@ sub do_git_commit {\n \t\t                   0, $self->svm_uuid);\n \t}\n \tprint \" = $commit ($self->{ref_id})\\n\";\n+\tif (--$_gc_nr == 0) {\n+\t\t$_gc_nr = $_gc_period;\n+\t\tgc();\n+\t}\n \treturn $commit;\n }\n \n@@ -3975,6 +3986,7 @@ sub gs_fetch_loop_common {\n \t\t$max += $inc;\n \t\t$max = $head if ($max > $head);\n \t}\n+\tGit::SVN::gc();\n }\n \n sub match_globs {\n"},{"id":"66222","messageId":"20080121224818.GA8872@untitled","threadId":"11665","inReplyTo":"7vlk6k8fyp.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-01-21T22:48:28Z","receivedAt":"2008-01-21T22:48:28Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> wrote:\n> Karl Hasselström <kha@treskal.com> writes:\n> \n> > On 2008-01-19 19:37:37 -0800, Eric Wong wrote:\n> >\n> >> Harvey Harrison <harvey.harrison@gmail.com> wrote:\n> >>\n> >> > I found 100 was a bit too low when doing some large repos, I've\n> >> > been using 1000. I'd argue that --repack=1000 should be done by\n> >> > default.\n> >>\n> >> I've found 100 for repack too low in the past, too, which is why\n> >> repack defaults to 1000 if no number is specified. I think it should\n> >> hold for gc --auto, too.\n> >\n> > OK, I'll change it. But remember, gc --auto doesn't do _anything_\n> > unless it's deemed necessary, so it should behave much better than\n> > just plain repack. In theory at least.\n> \n> Careful. I made the same mistake and it had to be corrected with\n> e0cd252eb0ba6453acd64762625b004aa4cc162b.\n> \n> I think defaulting to --repack=1000 is a sane first step and you\n> guys already have most code for it so that is a very safe thing.\n> \n> Switching to \"gc --auto\" can be done early post 1.5.4, right?\n\nSorry for the latency[1], ack on both of Karl's patches for post-1.5.4.\n\nHere's a conservative change for 1.5.4 (not at all tested):\n\nFrom dbccd8081c6422569a9ca1211e27f56a24fdf3f3 Mon Sep 17 00:00:00 2001\nFrom: Eric Wong <normalperson@yhbt.net>\nDate: Mon, 21 Jan 2008 14:37:41 -0800\nSubject: [PATCH] git-svn: default to repacking every 1000 commits\n\nThis should reduce disk space usage when doing large imports.\nWe'll be switching to \"gc --auto\" post-1.5.4 to handle\nrepacking for us.\n\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n git-svn.perl |    8 +++-----\n 1 files changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 9f2b587..12745d5 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1408,11 +1408,9 @@ sub read_all_remotes {\n }\n \n sub init_vars {\n-\tif (defined $_repack) {\n-\t\t$_repack = 1000 if ($_repack <= 0);\n-\t\t$_repack_nr = $_repack;\n-\t\t$_repack_flags ||= '-d';\n-\t}\n+\t$_repack = 1000 unless (defined $_repack && $_repack > 0);\n+\t$_repack_nr = $_repack;\n+\t$_repack_flags ||= '-d';\n }\n \n sub verify_remotes_sanity {\n-- \nEric Wong\n\n[1] - I've been busy with other things and will also be traveling\n      this week, too.\n"},{"id":"66235","messageId":"7vr6gawvkt.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"20080121224818.GA8872@untitled","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-22T00:30:26Z","receivedAt":"2008-01-22T00:30:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong <normalperson@yhbt.net> writes:\n\n> Here's a conservative change for 1.5.4 (not at all tested):\n>\n> From dbccd8081c6422569a9ca1211e27f56a24fdf3f3 Mon Sep 17 00:00:00 2001\n> From: Eric Wong <normalperson@yhbt.net>\n> Date: Mon, 21 Jan 2008 14:37:41 -0800\n> Subject: [PATCH] git-svn: default to repacking every 1000 commits\n>\n> This should reduce disk space usage when doing large imports.\n> We'll be switching to \"gc --auto\" post-1.5.4 to handle\n> repacking for us.\n>\n> Signed-off-by: Eric Wong <normalperson@yhbt.net>\n> ---\n>  git-svn.perl |    8 +++-----\n>  1 files changed, 3 insertions(+), 5 deletions(-)\n>\n> diff --git a/git-svn.perl b/git-svn.perl\n> index 9f2b587..12745d5 100755\n> --- a/git-svn.perl\n> +++ b/git-svn.perl\n> @@ -1408,11 +1408,9 @@ sub read_all_remotes {\n>  }\n>  \n>  sub init_vars {\n> -\tif (defined $_repack) {\n> -\t\t$_repack = 1000 if ($_repack <= 0);\n> -\t\t$_repack_nr = $_repack;\n> -\t\t$_repack_flags ||= '-d';\n> -\t}\n> +\t$_repack = 1000 unless (defined $_repack && $_repack > 0);\n> +\t$_repack_nr = $_repack;\n> +\t$_repack_flags ||= '-d';\n>  }\n>  \n>  sub verify_remotes_sanity {\n\nThanks, but I think you need to do something about this part:\n\n2154:\tif (defined $_repack && (--$_repack_nr == 0)) {\n\nI'd say \n\n\tif ($_repack && (--$_repack_nr == 0)) {\n"},{"id":"66237","messageId":"20080122003911.GA16453@hand.yhbt.net","threadId":"11665","inReplyTo":"7vr6gawvkt.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-01-22T00:39:11Z","receivedAt":"2008-01-22T00:39:11Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> wrote:\n> Eric Wong <normalperson@yhbt.net> writes:\n> \n> > Here's a conservative change for 1.5.4 (not at all tested):\n> >\n> > From dbccd8081c6422569a9ca1211e27f56a24fdf3f3 Mon Sep 17 00:00:00 2001\n> > From: Eric Wong <normalperson@yhbt.net>\n> > Date: Mon, 21 Jan 2008 14:37:41 -0800\n> > Subject: [PATCH] git-svn: default to repacking every 1000 commits\n> >\n> > This should reduce disk space usage when doing large imports.\n> > We'll be switching to \"gc --auto\" post-1.5.4 to handle\n> > repacking for us.\n> >\n> > Signed-off-by: Eric Wong <normalperson@yhbt.net>\n> > ---\n> >  git-svn.perl |    8 +++-----\n> >  1 files changed, 3 insertions(+), 5 deletions(-)\n> >\n> > diff --git a/git-svn.perl b/git-svn.perl\n> > index 9f2b587..12745d5 100755\n> > --- a/git-svn.perl\n> > +++ b/git-svn.perl\n> > @@ -1408,11 +1408,9 @@ sub read_all_remotes {\n> >  }\n> >  \n> >  sub init_vars {\n> > -\tif (defined $_repack) {\n> > -\t\t$_repack = 1000 if ($_repack <= 0);\n> > -\t\t$_repack_nr = $_repack;\n> > -\t\t$_repack_flags ||= '-d';\n> > -\t}\n> > +\t$_repack = 1000 unless (defined $_repack && $_repack > 0);\n> > +\t$_repack_nr = $_repack;\n> > +\t$_repack_flags ||= '-d';\n> >  }\n> >  \n> >  sub verify_remotes_sanity {\n> \n> Thanks, but I think you need to do something about this part:\n> \n> 2154:\tif (defined $_repack && (--$_repack_nr == 0)) {\n> \n> I'd say \n> \n> \tif ($_repack && (--$_repack_nr == 0)) {\n\ninit_vars() is called unconditionally, and always defines $_repack.\nIt could actually just be:\n\n\tif (--$_repack_nr == 0) {\n\n-- \nEric Wong\n"},{"id":"66255","messageId":"7vtzl6vd7v.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"20080122003911.GA16453@hand.yhbt.net","subject":"Re: [PATCH] Let \"git svn\" run \"git gc --auto\" occasionally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-22T01:52:20Z","receivedAt":"2008-01-22T01:52:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong <normalperson@yhbt.net> writes:\n\n>> >  sub init_vars {\n>> > -\tif (defined $_repack) {\n>> > -\t\t$_repack = 1000 if ($_repack <= 0);\n>> > -\t\t$_repack_nr = $_repack;\n>> > -\t\t$_repack_flags ||= '-d';\n>> > -\t}\n>> > +\t$_repack = 1000 unless (defined $_repack && $_repack > 0);\n>> > +\t$_repack_nr = $_repack;\n>> > +\t$_repack_flags ||= '-d';\n>> >  }\n>> >  \n>> >  sub verify_remotes_sanity {\n>> \n>> Thanks, but I think you need to do something about this part:\n>> \n>> 2154:\tif (defined $_repack && (--$_repack_nr == 0)) {\n>> \n>> I'd say \n>> \n>> \tif ($_repack && (--$_repack_nr == 0)) {\n>\n> init_vars() is called unconditionally, and always defines $_repack.\n> It could actually just be:\n>\n> \tif (--$_repack_nr == 0) {\n\nBut that means predecremented --$_repack_nr will count -1, -2, ...\nuntil it wraps around when the user said \"--repack=0\", meaning\n\"never repack\".  Instead you made it \"do not repack for a many\nmany many rounds\".\n\nWhich would be perfectly fine in practice but somehow feels a\nbit dirty to me.\n"},{"id":"66353","messageId":"BE604744-0D26-4A39-85CE-B5C0C8C00F9E@sb.org","threadId":"11665","inReplyTo":"7vtzl6vd7v.fsf@gitster.siamese.dyndns.org","subject":"git filter-branch should run git gc --auto","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-01-23T02:43:15Z","receivedAt":"2008-01-23T02:43:15Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"I just glanced at git-filter-branch.sh (and I must say I was  \nincredibly surprised to find out it was a shell script) and it seems  \nit never runs git-gc or git-repack. Doesn't that end up with the same  \nproblems as git-svn sans git-repack when filtering a large number of  \ncommits? I was just thinking, if I were to git-filter-branch on my  \nmassive repo (in fact, the same repo that started this thread, with  \nover 33000 commits in the upstream svn repo), even if I just do  \nsomething as simple as change the commit msg wont I end up with  \nthousands of unreachable objects? I shudder to think how many  \nunreachable objects I would have if I pruned the entire dports  \ndirectory off of the tree.\n\nAm I missing something, or does git-filter-branch really not do any  \ngarbage collection? I tried reading the source, but complex bash  \nscripts are almost as bad as perl in terms of readability.\n\n-Kevin Ballard\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n\n\n"},{"id":"66354","messageId":"7v1w89qmw3.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"BE604744-0D26-4A39-85CE-B5C0C8C00F9E@sb.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-23T02:46:52Z","receivedAt":"2008-01-23T02:46:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kevin Ballard <kevin@sb.org> writes:\n\n> I just glanced at git-filter-branch.sh (and I must say I was\n> incredibly surprised to find out it was a shell script) and it seems\n> it never runs git-gc or git-repack. Doesn't that end up with the same\n> problems as git-svn sans git-repack when filtering a large number of\n> commits? I was just thinking, if I were to git-filter-branch on my\n> massive repo (in fact, the same repo that started this thread, with\n> over 33000 commits in the upstream svn repo), even if I just do\n> something as simple as change the commit msg wont I end up with\n> thousands of unreachable objects? I shudder to think how many\n> unreachable objects I would have if I pruned the entire dports\n> directory off of the tree.\n>\n> Am I missing something, or does git-filter-branch really not do any\n> garbage collection? I tried reading the source, but complex bash\n> scripts are almost as bad as perl in terms of readability.\n\nTheoretically yes, and it largely depends on what you do, but\nfilter-branch goes over the objects that already exists in your\nrepository, and hopefully you won't be rewriting majority of\nthem.\n\nSo the impact of not repacking is probably much less painful in\npractice.\n\nBut again as I said, it largely depends on what you do in your\nfilter.  If you are upcasing (or convert to NFD ;-)) the\ncontents of all of your blob objects, you would certainly want\nto repack every once in a while.\n"},{"id":"66356","messageId":"7vwsq1p82i.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"7v1w89qmw3.fsf@gitster.siamese.dyndns.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-23T02:52:21Z","receivedAt":"2008-01-23T02:52:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Kevin Ballard <kevin@sb.org> writes:\n>\n>> I just glanced at git-filter-branch.sh (and I must say I was\n>> incredibly surprised to find out it was a shell script) and it seems\n>> it never runs git-gc or git-repack. Doesn't that end up with the same\n>> problems as git-svn sans git-repack when filtering a large number of\n>> commits? I was just thinking, if I were to git-filter-branch on my\n>> massive repo (in fact, the same repo that started this thread, with\n>> over 33000 commits in the upstream svn repo), even if I just do\n>> something as simple as change the commit msg wont I end up with\n>> thousands of unreachable objects? I shudder to think how many\n>> unreachable objects I would have if I pruned the entire dports\n>> directory off of the tree.\n\nAnother thing I forgot to say in my previous message.  The old\nrefs are kept in reflogs and also in refs/original/, so you will\nnot be creating new unreachables even if you rewrite many objects.\n\n>> Am I missing something, or does git-filter-branch really not do any\n>> garbage collection? I tried reading the source, but complex bash\n>> scripts are almost as bad as perl in terms of readability.\n>\n> Theoretically yes, and it largely depends on what you do, but\n> filter-branch goes over the objects that already exists in your\n> repository, and hopefully you won't be rewriting majority of\n> them.\n>\n> So the impact of not repacking is probably much less painful in\n> practice.\n>\n> But again as I said, it largely depends on what you do in your\n> filter.  If you are upcasing (or convert to NFD ;-)) the\n> contents of all of your blob objects, you would certainly want\n> to repack every once in a while.\n\nSomething like this, perhaps?\n\n git-filter-branch.sh |    6 ++++++\n 1 files changed, 6 insertions(+), 0 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex ebf05ca..8e44001 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -299,6 +299,12 @@ while read commit parents; do\n \t\t\tdie \"msg filter failed: $filter_msg\"\n \tsh -c \"$filter_commit\" \"git commit-tree\" \\\n \t\t$(git write-tree) $parentstr < ../message > ../map/$commit\n+\n+\tif test $(( $i % 512 )) = 0\n+\tthen\n+\t\tgit gc --auto\n+\tfi\n+\n done <../revs\n \n # In case of a subdirectory filter, it is possible that a specified head\n"},{"id":"66357","messageId":"1201056848.16972.102.camel@brick","threadId":"11665","inReplyTo":"7v1w89qmw3.fsf@gitster.siamese.dyndns.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Harvey Harrison","fromEmail":"harvey.harrison@gmail.com","sentAt":"2008-01-23T02:54:08Z","receivedAt":"2008-01-23T02:54:08Z","isPatch":false,"sender":{"key":"harvey.harrison@gmail.com","avatar":null},"body":"On Tue, 2008-01-22 at 18:46 -0800, Junio C Hamano wrote:\n> Kevin Ballard <kevin@sb.org> writes:\n> \n> > I just glanced at git-filter-branch.sh (and I must say I was\n> > incredibly surprised to find out it was a shell script) and it seems\n> > it never runs git-gc or git-repack. Doesn't that end up with the same\n> > problems as git-svn sans git-repack when filtering a large number of\n> > commits? I was just thinking, if I were to git-filter-branch on my\n> > massive repo (in fact, the same repo that started this thread, with\n> > over 33000 commits in the upstream svn repo), even if I just do\n> > something as simple as change the commit msg wont I end up with\n> > thousands of unreachable objects? I shudder to think how many\n> > unreachable objects I would have if I pruned the entire dports\n> > directory off of the tree.\n> >\n> > Am I missing something, or does git-filter-branch really not do any\n> > garbage collection? I tried reading the source, but complex bash\n> > scripts are almost as bad as perl in terms of readability.\n> \n> Theoretically yes, and it largely depends on what you do, but\n> filter-branch goes over the objects that already exists in your\n> repository, and hopefully you won't be rewriting majority of\n> them.\n> \n> So the impact of not repacking is probably much less painful in\n> practice.\n\nAnd afterwards, you'll probably want to check the rewritten history\nto make sure it is acceptable before doing a git gc --prune.\n\nCheers,\n\nHarvey\n"},{"id":"66359","messageId":"60D6C8F5-EF88-48F0-92CA-8E49838C0CB9@sb.org","threadId":"11665","inReplyTo":"7v1w89qmw3.fsf@gitster.siamese.dyndns.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-01-23T02:58:09Z","receivedAt":"2008-01-23T02:58:09Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Jan 22, 2008, at 9:46 PM, Junio C Hamano wrote:\n\n> Kevin Ballard <kevin@sb.org> writes:\n>\n>> I just glanced at git-filter-branch.sh (and I must say I was\n>> incredibly surprised to find out it was a shell script) and it seems\n>> it never runs git-gc or git-repack. Doesn't that end up with the same\n>> problems as git-svn sans git-repack when filtering a large number of\n>> commits? I was just thinking, if I were to git-filter-branch on my\n>> massive repo (in fact, the same repo that started this thread, with\n>> over 33000 commits in the upstream svn repo), even if I just do\n>> something as simple as change the commit msg wont I end up with\n>> thousands of unreachable objects? I shudder to think how many\n>> unreachable objects I would have if I pruned the entire dports\n>> directory off of the tree.\n>>\n>> Am I missing something, or does git-filter-branch really not do any\n>> garbage collection? I tried reading the source, but complex bash\n>> scripts are almost as bad as perl in terms of readability.\n>\n> Theoretically yes, and it largely depends on what you do, but\n> filter-branch goes over the objects that already exists in your\n> repository, and hopefully you won't be rewriting majority of\n> them.\n>\n> So the impact of not repacking is probably much less painful in\n> practice.\n>\n> But again as I said, it largely depends on what you do in your\n> filter.  If you are upcasing (or convert to NFD ;-)) the\n> contents of all of your blob objects, you would certainly want\n> to repack every once in a while.\n\n\nI'm actually considering what the cost would be of switching macports  \nto git (not that it will ever happen - too many anonymous people pull  \nfrom svn trunk). Right now the svn trunk contains a subfolder for the  \nsource code and another subfolder for all ~4400+ Portfiles. In such a  \ntheoretical move, I'd want to split that up, probably into two  \nunrelated branches. Doing so would mean running git-filter-branch over  \na linear commit history that's 31580 objects long, with a tree filter  \nto prune the dports directory away and a msg filter to remove the svn- \nid stuff that git-svn left behind. This means that every single commit  \nobjects would be changed, as well as the root tree object for every  \nsingle commit. That would be about 63160 objects. I'd also have to  \nfigure out some way to remove the commit objects entirely that only  \nreference the dports directory. Then I'd have to do it again with the  \nopposite tree filter (to prune everything but the dports directory and  \nmove the contents of the dports directory up one level) and same msg  \nfilter. Granted, if I do the first action in a branch, that leaves no  \nunreachable objects (since the originals are still referenced), but  \nthe second operation definitely would leave unreachable objects, and  \nwere I to clone the repository instead and do the operations in the  \ndifferent repos (which is perfectly legitimate - otherwise I'd have to  \nclone it after everything else and then delete branches) then both  \nactions would leave thousands of objects unreachable.\n\nI'd suggest a patch to run git gc --auto, but it looks like you just  \ndid in a subsequent email. As for your comments about the reflogs,  \ncan't I disable recording those, at least temporarily? I'd rather  \nclean up after myself as I work rather than balloon the repository and  \ncollapse it in a single operation at the end.\n\n-Kevin Ballard\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n\n\n"},{"id":"66361","messageId":"175DB3F7-2E39-4450-945F-E33D83EF2792@sb.org","threadId":"11665","inReplyTo":"7vwsq1p82i.fsf@gitster.siamese.dyndns.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-01-23T03:03:30Z","receivedAt":"2008-01-23T03:03:30Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Jan 22, 2008, at 9:52 PM, Junio C Hamano wrote:\n\n> Junio C Hamano <gitster@pobox.com> writes:\n>\n>> Kevin Ballard <kevin@sb.org> writes:\n>>\n>>> Am I missing something, or does git-filter-branch really not do any\n>>> garbage collection? I tried reading the source, but complex bash\n>>> scripts are almost as bad as perl in terms of readability.\n>>\n>> Theoretically yes, and it largely depends on what you do, but\n>> filter-branch goes over the objects that already exists in your\n>> repository, and hopefully you won't be rewriting majority of\n>> them.\n>>\n>> So the impact of not repacking is probably much less painful in\n>> practice.\n>>\n>> But again as I said, it largely depends on what you do in your\n>> filter.  If you are upcasing (or convert to NFD ;-)) the\n>> contents of all of your blob objects, you would certainly want\n>> to repack every once in a while.\n>\n> Something like this, perhaps?\n>\n> git-filter-branch.sh |    6 ++++++\n> 1 files changed, 6 insertions(+), 0 deletions(-)\n>\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> index ebf05ca..8e44001 100755\n> --- a/git-filter-branch.sh\n> +++ b/git-filter-branch.sh\n> @@ -299,6 +299,12 @@ while read commit parents; do\n> \t\t\tdie \"msg filter failed: $filter_msg\"\n> \tsh -c \"$filter_commit\" \"git commit-tree\" \\\n> \t\t$(git write-tree) $parentstr < ../message > ../map/$commit\n> +\n> +\tif test $(( $i % 512 )) = 0\n> +\tthen\n> +\t\tgit gc --auto\n> +\tfi\n> +\n> done <../revs\n>\n> # In case of a subdirectory filter, it is possible that a specified  \n> head\n>\n\n\nOffhand that looks good, but we'd probably want to unilaterally do  \nanother git-gc when we're done.\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex ebf05ca..32274a6 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -299,8 +299,16 @@ while read commit parents; do\n  \t\t\tdie \"msg filter failed: $filter_msg\"\n  \tsh -c \"$filter_commit\" \"git commit-tree\" \\\n  \t\t$(git write-tree) $parentstr < ../message > ../map/$commit\n+\n+\tif test $(( $i % 512 )) = 0\n+\tthen\n+\t\tgit gc --auto\n+\tfi\n+\n  done <../revs\n\n+git gc --auto\n+\n  # In case of a subdirectory filter, it is possible that a specified  \nhead\n  # is not in the set of rewritten commits, because it was pruned by the\n  # revision walker.  Fix it by mapping these heads to the next  \nrewritten\n\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n\n\n"},{"id":"66368","messageId":"4796CB78.2070607@vilain.net","threadId":"11665","inReplyTo":"60D6C8F5-EF88-48F0-92CA-8E49838C0CB9@sb.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-01-23T05:07:04Z","receivedAt":"2008-01-23T05:07:04Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Kevin Ballard wrote:\n> I'm actually considering what the cost would be of switching macports  \n> to git (not that it will ever happen - too many anonymous people pull  \n> from svn trunk). Right now the svn trunk contains a subfolder for the  \n> source code and another subfolder for all ~4400+ Portfiles. In such a  \n> theoretical move, I'd want to split that up, probably into two  \n> unrelated branches. Doing so would mean running git-filter-branch over  \n> a linear commit history that's 31580 objects long, with a tree filter  \n> to prune the dports directory away and a msg filter to remove the svn- \n> id stuff that git-svn left behind.\n\nYou could have used git-svn --no-metadata :)\n\nUsing a commit filter to implement the pruning will be much faster;\nyou'll need to make a temporary index, use git-read-tree, git-rm, then\ngit-commit.  This way you avoid the expense of checking out the files\njust to delete them in your rewrite hook.\n\n> I'd also have to  \n> figure out some way to remove the commit objects entirely that only  \n> reference the dports directory. \n\nThis can be done with a parent filter.\n\n> I'd suggest a patch to run git gc --auto, but it looks like you just  \n> did in a subsequent email. As for your comments about the reflogs,  \n> can't I disable recording those, at least temporarily? I'd rather  \n> clean up after myself as I work rather than balloon the repository and  \n> collapse it in a single operation at the end.\n\nHonestly, the optimisation I mention above will save you much more time.\n Note that you can run git-repack -d every half hour out of cron, it is\nsafe and will let it clean as you go.\n\nSam.\n"},{"id":"66372","messageId":"20080123064430.GD16297@glandium.org","threadId":"11665","inReplyTo":"7v1w89qmw3.fsf@gitster.siamese.dyndns.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2008-01-23T06:44:30Z","receivedAt":"2008-01-23T06:44:30Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Jan 22, 2008 at 06:46:52PM -0800, Junio C Hamano wrote:\n> Kevin Ballard <kevin@sb.org> writes:\n> \n> > I just glanced at git-filter-branch.sh (and I must say I was\n> > incredibly surprised to find out it was a shell script) and it seems\n> > it never runs git-gc or git-repack. Doesn't that end up with the same\n> > problems as git-svn sans git-repack when filtering a large number of\n> > commits? I was just thinking, if I were to git-filter-branch on my\n> > massive repo (in fact, the same repo that started this thread, with\n> > over 33000 commits in the upstream svn repo), even if I just do\n> > something as simple as change the commit msg wont I end up with\n> > thousands of unreachable objects? I shudder to think how many\n> > unreachable objects I would have if I pruned the entire dports\n> > directory off of the tree.\n> >\n> > Am I missing something, or does git-filter-branch really not do any\n> > garbage collection? I tried reading the source, but complex bash\n> > scripts are almost as bad as perl in terms of readability.\n> \n> Theoretically yes, and it largely depends on what you do, but\n> filter-branch goes over the objects that already exists in your\n> repository, and hopefully you won't be rewriting majority of\n> them.\n> \n> So the impact of not repacking is probably much less painful in\n> practice.\n> \n> But again as I said, it largely depends on what you do in your\n> filter.  If you are upcasing (or convert to NFD ;-)) the\n> contents of all of your blob objects, you would certainly want\n> to repack every once in a while.\n\nI wonder if it wouldn't be possible to have filter-branch use\nfast-import, so that it would create a pack instead of a lot of loose\nobjects.\n\nMike\n"},{"id":"66376","messageId":"C1A3D16A-1797-4895-ABE3-CB6A48C27F1F@sb.org","threadId":"11665","inReplyTo":"4796CB78.2070607@vilain.net","subject":"Re: git filter-branch should run git gc --auto","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-01-23T08:18:32Z","receivedAt":"2008-01-23T08:18:32Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Jan 23, 2008, at 12:07 AM, Sam Vilain wrote:\n\n> Kevin Ballard wrote:\n>> I'm actually considering what the cost would be of switching macports\n>> to git (not that it will ever happen - too many anonymous people pull\n>> from svn trunk). Right now the svn trunk contains a subfolder for the\n>> source code and another subfolder for all ~4400+ Portfiles. In such a\n>> theoretical move, I'd want to split that up, probably into two\n>> unrelated branches. Doing so would mean running git-filter-branch  \n>> over\n>> a linear commit history that's 31580 objects long, with a tree filter\n>> to prune the dports directory away and a msg filter to remove the  \n>> svn-\n>> id stuff that git-svn left behind.\n>\n> You could have used git-svn --no-metadata :)\n\nSure, except I imported the svn repo with the intention of continuing  \nto track it. I'm only floating the idea now of converting the upstream  \nrepo to git, but as I said before we have enough anonymous checkouts  \nof people tracking trunk that we probably can't justify switching  \nVCSs, especially when svn is now bundled on Leopard but git isn't.\n\n> Using a commit filter to implement the pruning will be much faster;\n> you'll need to make a temporary index, use git-read-tree, git-rm, then\n> git-commit.  This way you avoid the expense of checking out the files\n> just to delete them in your rewrite hook.\n\nI suspect an index filter would be simpler, and that's really what I  \nmeant when I said tree filter.\n\n>> I'd also have to\n>> figure out some way to remove the commit objects entirely that only\n>> reference the dports directory.\n>\n> This can be done with a parent filter.\n\nGood to know.\n\n>> I'd suggest a patch to run git gc --auto, but it looks like you just\n>> did in a subsequent email. As for your comments about the reflogs,\n>> can't I disable recording those, at least temporarily? I'd rather\n>> clean up after myself as I work rather than balloon the repository  \n>> and\n>> collapse it in a single operation at the end.\n>\n> Honestly, the optimisation I mention above will save you much more  \n> time.\n> Note that you can run git-repack -d every half hour out of cron, it is\n> safe and will let it clean as you go.\n\nThat's a reasonable suggestion. And I'm still just thinking about  \nthis, so I have no idea if I'll ever actually have to run git-filter- \nbranch on this massive history.\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n\n\n"},{"id":"66395","messageId":"alpine.LSU.1.00.0801231256400.5731@racer.site","threadId":"11665","inReplyTo":"20080123064430.GD16297@glandium.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-01-23T13:00:37Z","receivedAt":"2008-01-23T13:00:37Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jan 2008, Mike Hommey wrote:\n\n> On Tue, Jan 22, 2008 at 06:46:52PM -0800, Junio C Hamano wrote:\n> > Kevin Ballard <kevin@sb.org> writes:\n> > \n> > > I just glanced at git-filter-branch.sh (and I must say I was \n> > > incredibly surprised to find out it was a shell script) and it seems \n> > > it never runs git-gc or git-repack. Doesn't that end up with the \n> > > same problems as git-svn sans git-repack when filtering a large \n> > > number of commits? I was just thinking, if I were to \n> > > git-filter-branch on my massive repo (in fact, the same repo that \n> > > started this thread, with over 33000 commits in the upstream svn \n> > > repo), even if I just do something as simple as change the commit \n> > > msg wont I end up with thousands of unreachable objects? I shudder \n> > > to think how many unreachable objects I would have if I pruned the \n> > > entire dports directory off of the tree.\n> > >\n> > > Am I missing something, or does git-filter-branch really not do any \n> > > garbage collection? I tried reading the source, but complex bash \n> > > scripts are almost as bad as perl in terms of readability.\n> > \n> > Theoretically yes, and it largely depends on what you do, but \n> > filter-branch goes over the objects that already exists in your \n> > repository, and hopefully you won't be rewriting majority of them.\n> > \n> > So the impact of not repacking is probably much less painful in \n> > practice.\n> > \n> > But again as I said, it largely depends on what you do in your filter.  \n> > If you are upcasing (or convert to NFD ;-)) the contents of all of \n> > your blob objects, you would certainly want to repack every once in a \n> > while.\n> \n> I wonder if it wouldn't be possible to have filter-branch use \n> fast-import, so that it would create a pack instead of a lot of loose \n> objects.\n\nNot really; the filters are very much tuned to the index-modification and \ncommit process.\n\nAnd I doubt that the gc --auto would help much; git-filter-branch creates \ngazillions of files, and that is likely to bring performance down.  If, \nthat is, you choose _not_ to heed the comment in \nDocumentation/git-filter-branch.txt lines 44-46:\n\n\tNote that since this operation is extensively I/O expensive, it \n\tmight be a good idea to redirect the temporary directory off-disk \n\twith the '-d' option, e.g. on tmpfs.  Reportedly the speedup is \n\tvery noticeable.\n\nCiao,\nDscho\n"},{"id":"66423","messageId":"7vfxwoibyo.fsf@gitster.siamese.dyndns.org","threadId":"11665","inReplyTo":"20080123064430.GD16297@glandium.org","subject":"Re: git filter-branch should run git gc --auto","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-23T19:22:23Z","receivedAt":"2008-01-23T19:22:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mike Hommey <mh@glandium.org> writes:\n\n> I wonder if it wouldn't be possible to have filter-branch use\n> fast-import, so that it would create a pack instead of a lot of loose\n> objects.\n\nI do not think it will help.  The objects in packs fast-import\ncreates cannot be accessed from outside fast-import.  Not even\nthe rest of the core routines running inside that fast-import\nprocess cannot access them via the usual read_sha1_file()\ninterface, as described in detail in a recent thread [*1*].  The\nonly way to make it available while you are still feeding new\ndata to fast-import is to explicitly tell it to finalize the\ncurrent pack by issuing a 'mark' command (and fast-import will\nstart writing to a new pack).\n\nAnd filters need to be able to read the objects previous steps\nproduced to do their work.\n\nWhich means that instead of having to deal with many loose\nobjects, you will now face many little packs, each contains data\nchanged perhaps at most one commit's worth.  You would need to\n\"repack -a -d\" to consolidate these little packs every once in a\nwhile, and I suspect more often than you would need to repack\nloose objects, as handling many packs is much more expensive\nthan handling many loose objects.\n\n[Reference]\n\n*1* http://thread.gmane.org/gmane.comp.version-control.git/70964/focus=71076\n"}]}