{"thread":{"id":"10150","subject":"[PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","startedAt":"2007-10-05T00:15:28Z","lastAt":"2007-10-06T08:15:52Z","messageCount":10,"participants":["Steven Grimm","Andreas Ericsson","Peter Baumann","Johannes Schindelin","Eric Wong"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"54891","messageId":"20071005001528.GA13029@midwinter.com","threadId":"10150","inReplyTo":null,"subject":"[PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-10-05T00:15:28Z","receivedAt":"2007-10-05T00:15:28Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"git-svn dcommit, by virtue of rewriting history to insert svn revision IDs,\nleaves old commits dangling.  Since dcommit is already unsafe to run\nconcurrently with other git commands, no additional risk is introduced\nby making it prune those old objects as needed.\n\nSigned-off-by: Steven Grimm <koreth@midwinter.com>\n---\n\nThis is in response to a colleague who complained that, after I\ninstalled the latest git release, he was getting lots of \"too many\nunreachable loose objects\" errors from the new \"git gc --auto\" run.\nThose objects turned out to be dangling commits from a year's worth of\ngit-svn usage, since every git-svn commit will abandon at least one\nexisting commit in order to rewrite it with the svn version data.\n\n git-svn.perl |    6 ++++++\n 1 files changed, 6 insertions(+), 0 deletions(-)\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 777e436..be62ee1 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -441,6 +441,12 @@ sub cmd_dcommit {\n \t\t\t}\n \t\t\tcommand_noisy(@finish, $gs->refname);\n \t\t\t$last_rev = $cmt_rev;\n+\n+\t\t\t# rebase will have made the just-committed revisions\n+\t\t\t# unreachable; over time that can build up lots of\n+\t\t\t# loose objects in the repo. prune is unsafe to run\n+\t\t\t# concurrently but so is dcommit.\n+\t\t\tcommand_noisy(qw/gc --auto --prune/);\n \t\t}\n \t}\n }\n-- \n1.5.3.4.203.gcc61a\n"},{"id":"54913","messageId":"4705EFF2.9090506@op5.se","threadId":"10150","inReplyTo":"20071005001528.GA13029@midwinter.com","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-10-05T08:04:02Z","receivedAt":"2007-10-05T08:04:02Z","isPatch":true,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Steven Grimm wrote:\n> git-svn dcommit, by virtue of rewriting history to insert svn revision IDs,\n> leaves old commits dangling.  Since dcommit is already unsafe to run\n> concurrently with other git commands, no additional risk is introduced\n> by making it prune those old objects as needed.\n> \n> Signed-off-by: Steven Grimm <koreth@midwinter.com>\n> ---\n> \n> This is in response to a colleague who complained that, after I\n> installed the latest git release, he was getting lots of \"too many\n> unreachable loose objects\" errors from the new \"git gc --auto\" run.\n> Those objects turned out to be dangling commits from a year's worth of\n> git-svn usage, since every git-svn commit will abandon at least one\n> existing commit in order to rewrite it with the svn version data.\n> \n>  git-svn.perl |    6 ++++++\n>  1 files changed, 6 insertions(+), 0 deletions(-)\n> \n> diff --git a/git-svn.perl b/git-svn.perl\n> index 777e436..be62ee1 100755\n> --- a/git-svn.perl\n> +++ b/git-svn.perl\n> @@ -441,6 +441,12 @@ sub cmd_dcommit {\n>  \t\t\t}\n>  \t\t\tcommand_noisy(@finish, $gs->refname);\n>  \t\t\t$last_rev = $cmt_rev;\n> +\n> +\t\t\t# rebase will have made the just-committed revisions\n> +\t\t\t# unreachable; over time that can build up lots of\n> +\t\t\t# loose objects in the repo. prune is unsafe to run\n> +\t\t\t# concurrently but so is dcommit.\n> +\t\t\tcommand_noisy(qw/gc --auto --prune/);\n>  \t\t}\n>  \t}\n>  }\n\nI'd be surprised if this would ever prune anything, as git doesn't throw out\nobjects reachable by reflog (or, I assume, any of the objects reachable from\nobjects reachable from reflog).\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n"},{"id":"54916","messageId":"20071005082110.GA4797@xp.machine.xx","threadId":"10150","inReplyTo":"20071005001528.GA13029@midwinter.com","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Peter Baumann","fromEmail":"waste.manager@gmx.de","sentAt":"2007-10-05T08:21:10Z","receivedAt":"2007-10-05T08:21:10Z","isPatch":true,"sender":{"key":"waste.manager@gmx.de","avatar":null},"body":"On Thu, Oct 04, 2007 at 05:15:28PM -0700, Steven Grimm wrote:\n> git-svn dcommit, by virtue of rewriting history to insert svn revision IDs,\n> leaves old commits dangling.  Since dcommit is already unsafe to run\n> concurrently with other git commands, no additional risk is introduced\n> by making it prune those old objects as needed.\n> \n> Signed-off-by: Steven Grimm <koreth@midwinter.com>\n> ---\n> \n> This is in response to a colleague who complained that, after I\n> installed the latest git release, he was getting lots of \"too many\n> unreachable loose objects\" errors from the new \"git gc --auto\" run.\n> Those objects turned out to be dangling commits from a year's worth of\n> git-svn usage, since every git-svn commit will abandon at least one\n> existing commit in order to rewrite it with the svn version data.\n> \n\nI don't like the automatic prune. What if someone has other objects in\nthere which shouldn't be pruned? Making git svn dcommit doing the prune\nwould be at least suprising, because how is one supposed to know that\ndoing a commit into svn will prune all your precious objects?\n\nSure, I can unterstand from where you are coming from, but I'd prefere\nif this could be specified on a case by case basis, e.g. from the\ncmdline or as a config option.\n\n-Peter\n\n\n>  git-svn.perl |    6 ++++++\n>  1 files changed, 6 insertions(+), 0 deletions(-)\n> \n> diff --git a/git-svn.perl b/git-svn.perl\n> index 777e436..be62ee1 100755\n> --- a/git-svn.perl\n> +++ b/git-svn.perl\n> @@ -441,6 +441,12 @@ sub cmd_dcommit {\n>  \t\t\t}\n>  \t\t\tcommand_noisy(@finish, $gs->refname);\n>  \t\t\t$last_rev = $cmt_rev;\n> +\n> +\t\t\t# rebase will have made the just-committed revisions\n> +\t\t\t# unreachable; over time that can build up lots of\n> +\t\t\t# loose objects in the repo. prune is unsafe to run\n> +\t\t\t# concurrently but so is dcommit.\n> +\t\t\tcommand_noisy(qw/gc --auto --prune/);\n>  \t\t}\n>  \t}\n>  }\n> -- \n> 1.5.3.4.203.gcc61a\n> \n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"54918","messageId":"Pine.LNX.4.64.0710050926160.4174@racer.site","threadId":"10150","inReplyTo":"4705EFF2.9090506@op5.se","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-10-05T08:27:20Z","receivedAt":"2007-10-05T08:27:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 5 Oct 2007, Andreas Ericsson wrote:\n\n> Steven Grimm wrote:\n> > git-svn dcommit, by virtue of rewriting history to insert svn revision IDs,\n> > leaves old commits dangling.  Since dcommit is already unsafe to run\n> > concurrently with other git commands, no additional risk is introduced\n> > by making it prune those old objects as needed.\n> > \n> > Signed-off-by: Steven Grimm <koreth@midwinter.com>\n> > ---\n> > \n> > This is in response to a colleague who complained that, after I\n> > installed the latest git release, he was getting lots of \"too many\n> > unreachable loose objects\" errors from the new \"git gc --auto\" run.\n> > Those objects turned out to be dangling commits from a year's worth of\n> > git-svn usage, since every git-svn commit will abandon at least one\n> > existing commit in order to rewrite it with the svn version data.\n> > \n> >  git-svn.perl |    6 ++++++\n> >  1 files changed, 6 insertions(+), 0 deletions(-)\n> > \n> > diff --git a/git-svn.perl b/git-svn.perl\n> > index 777e436..be62ee1 100755\n> > --- a/git-svn.perl\n> > +++ b/git-svn.perl\n> > @@ -441,6 +441,12 @@ sub cmd_dcommit {\n> >  \t\t\t}\n> >  \t\t\tcommand_noisy(@finish, $gs->refname);\n> >  \t\t\t$last_rev = $cmt_rev;\n> > +\n> > +\t\t\t# rebase will have made the just-committed revisions\n> > +\t\t\t# unreachable; over time that can build up lots of\n> > +\t\t\t# loose objects in the repo. prune is unsafe to run\n> > +\t\t\t# concurrently but so is dcommit.\n> > +\t\t\tcommand_noisy(qw/gc --auto --prune/);\n> >  \t\t}\n> >  \t}\n> >  }\n> \n> I'd be surprised if this would ever prune anything, as git doesn't throw \n> out objects reachable by reflog (or, I assume, any of the objects \n> reachable from objects reachable from reflog).\n\nIt will so, in due time.  Reflogs have an expiry date, and will be culled \nby git gc --auto.  So if you dcommit often (which I do), the objects will \nbe pruned, eventually.\n\nCiao,\nDscho\n"},{"id":"54970","messageId":"47066255.6080500@midwinter.com","threadId":"10150","inReplyTo":"20071005082110.GA4797@xp.machine.xx","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-10-05T16:12:05Z","receivedAt":"2007-10-05T16:12:05Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Peter Baumann wrote:\n> I don't like the automatic prune. What if someone has other objects in\n> there which shouldn't be pruned? Making git svn dcommit doing the prune\n> would be at least suprising, because how is one supposed to know that\n> doing a commit into svn will prune all your precious objects?\n>   \n\n\"git commit\" already does garbage collection, so we've already set a \nprecedent for a commit operation also doing some cleanup at the end. \nHowever, you're correct that this cleanup behavior (and the way to turn \nit off) should be documented so that there's some way to know about it. \nDoc patch forthcoming.\n\n> Sure, I can unterstand from where you are coming from, but I'd prefere\n> if this could be specified on a case by case basis, e.g. from the\n> cmdline or as a config option.\n>   \n\nThis code (by virtue of only doing the prune if the \"too many loose \nobjects\" test succeeds) will obey the existing gc.auto config option. So \nit's already possible to turn off as is. I'll note that in the doc patch.\n\n-Steve\n"},{"id":"54971","messageId":"20071005161522.GA12545@midwinter.com","threadId":"10150","inReplyTo":"47066255.6080500@midwinter.com","subject":"[PATCH 3/2] Document the fact that git-svn now runs git-gc","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-10-05T16:15:22Z","receivedAt":"2007-10-05T16:15:22Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Signed-off-by: Steven Grimm <koreth@midwinter.com>\n---\n Documentation/git-svn.txt |   10 +++++++++-\n 1 files changed, 9 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex e157c6a..26f0f39 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -125,7 +125,15 @@ and have no uncommitted changes.\n \talternative to HEAD.\n \tThis is advantageous over 'set-tree' (below) because it produces\n \tcleaner, more linear history.\n-+\n+\n+When the commit is finished, gitlink:git-gc[1] is run with the\n+`--prune` and `--auto` options to clean up the git object database,\n+including removing old unreachable objects (some of which are\n+created by the process of committing to SVN.) Set the `gc.auto`\n+config option to 0 if you don't want your repository to be cleaned,\n+e.g., because you are intentionally keeping unreachable objects in\n+your repository.\n+\n --no-rebase;;\n \tAfter committing, do not rebase or reset.\n --\n-- \n1.5.3.4.203.gcc61a\n"},{"id":"54981","messageId":"20071005164912.GE4797@xp.machine.xx","threadId":"10150","inReplyTo":"47066255.6080500@midwinter.com","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Peter Baumann","fromEmail":"waste.manager@gmx.de","sentAt":"2007-10-05T16:49:12Z","receivedAt":"2007-10-05T16:49:12Z","isPatch":true,"sender":{"key":"waste.manager@gmx.de","avatar":null},"body":"On Fri, Oct 05, 2007 at 09:12:05AM -0700, Steven Grimm wrote:\n> Peter Baumann wrote:\n>> I don't like the automatic prune. What if someone has other objects in\n>> there which shouldn't be pruned? Making git svn dcommit doing the prune\n>> would be at least suprising, because how is one supposed to know that\n>> doing a commit into svn will prune all your precious objects?\n>>   \n>\n> \"git commit\" already does garbage collection, so we've already set a \n> precedent for a commit operation also doing some cleanup at the end. \n> However, you're correct that this cleanup behavior (and the way to turn it \n> off) should be documented so that there's some way to know about it. Doc \n> patch forthcoming.\n>\n\nThat's new to me. Glancing over git-commit.sh, I could only find a\n'git-gc --auto', but no prune. I am not against doing a 'git gc --auto',\nbut I am against the --prune, because this could make shared\nrepositories unfunctional.\n\n-Peter\n"},{"id":"54988","messageId":"470678ED.8050407@midwinter.com","threadId":"10150","inReplyTo":"20071005164912.GE4797@xp.machine.xx","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-10-05T17:48:29Z","receivedAt":"2007-10-05T17:48:29Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Peter Baumann wrote:\n> That's new to me. Glancing over git-commit.sh, I could only find a\n> 'git-gc --auto', but no prune. I am not against doing a 'git gc --auto',\n> but I am against the --prune, because this could make shared\n> repositories unfunctional.\n>   \n\nDoes anyone run \"git svn dcommit\" from a shared repository? That is the \nonly command that will trigger this code path.\n\nGiven that you lose all the svn metadata if you do \"git clone\" (or \"git \nclone -s\") on a git-svn-managed repository, it's not clear to me that \nanyone would ever be bitten by this. Counterexamples welcome, of course.\n\nHow would you feel about a separate config option to specifically enable \nauto-pruning, and having \"git svn clone\" set that option by default? \nPresumably anyone who is setting up a shared git-svn repository will be \nup to the task of disabling the option.\n\n-Steve\n"},{"id":"55006","messageId":"20071005235453.GB16849@untitled","threadId":"10150","inReplyTo":"20071005001528.GA13029@midwinter.com","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2007-10-05T23:54:53Z","receivedAt":"2007-10-05T23:54:53Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Steven Grimm <koreth@midwinter.com> wrote:\n> git-svn dcommit, by virtue of rewriting history to insert svn revision IDs,\n> leaves old commits dangling.  Since dcommit is already unsafe to run\n> concurrently with other git commands, no additional risk is introduced\n> by making it prune those old objects as needed.\n> \n> Signed-off-by: Steven Grimm <koreth@midwinter.com>\n> ---\n> \n> This is in response to a colleague who complained that, after I\n> installed the latest git release, he was getting lots of \"too many\n> unreachable loose objects\" errors from the new \"git gc --auto\" run.\n> Those objects turned out to be dangling commits from a year's worth of\n> git-svn usage, since every git-svn commit will abandon at least one\n> existing commit in order to rewrite it with the svn version data.\n\nI'm not a fan of automatic gc in general, but I understand it can\nhelp new users.  So as long as clueful users can easily disable it,\nthen it's fine by me...\n\n>  git-svn.perl |    6 ++++++\n>  1 files changed, 6 insertions(+), 0 deletions(-)\n> \n> diff --git a/git-svn.perl b/git-svn.perl\n> index 777e436..be62ee1 100755\n> --- a/git-svn.perl\n> +++ b/git-svn.perl\n> @@ -441,6 +441,12 @@ sub cmd_dcommit {\n>  \t\t\t}\n>  \t\t\tcommand_noisy(@finish, $gs->refname);\n>  \t\t\t$last_rev = $cmt_rev;\n> +\n> +\t\t\t# rebase will have made the just-committed revisions\n> +\t\t\t# unreachable; over time that can build up lots of\n> +\t\t\t# loose objects in the repo. prune is unsafe to run\n> +\t\t\t# concurrently but so is dcommit.\n> +\t\t\tcommand_noisy(qw/gc --auto --prune/);\n>  \t\t}\n>  \t}\n>  }\n\nThis is better called outside of this loop.  We now do a rebase after\nevery revision committed (which gets us even more dangling commits);\nbut we only want to call git-gc after everything is committed.\n\nIt'll be faster since git-gc is only invoked once, and if git-gc takes a\nvery long time to repack, we won't have to worry about timing out a SVN\nnetwork connection.  It'll also reduce the window for somebody else to\ncommit a conflicting change that'll cause dcommit to fail midway\nthrough.\n\n\nAs far as Peter's concerns for shared repositories go, I'm not sure...\n\nI've never been comfortable with shared repositories myself (even in a\npure git environment without git-svn) and always just preferred using\nfull clones or copies[1] myself so I could rm -r any working directory\nand not worry about any other repositories relying on it.\n\n[1] - I usually go about using cp -al + libflcow :)\n\n-- \nEric Wong\n"},{"id":"55009","messageId":"20071006081552.GF4797@xp.machine.xx","threadId":"10150","inReplyTo":"470678ED.8050407@midwinter.com","subject":"Re: [PATCH 2/2] Run garbage collection with loose object pruning after svn dcommit","fromName":"Peter Baumann","fromEmail":"waste.manager@gmx.de","sentAt":"2007-10-06T08:15:52Z","receivedAt":"2007-10-06T08:15:52Z","isPatch":true,"sender":{"key":"waste.manager@gmx.de","avatar":null},"body":"On Fri, Oct 05, 2007 at 10:48:29AM -0700, Steven Grimm wrote:\n> Peter Baumann wrote:\n>> That's new to me. Glancing over git-commit.sh, I could only find a\n>> 'git-gc --auto', but no prune. I am not against doing a 'git gc --auto',\n>> but I am against the --prune, because this could make shared\n>> repositories unfunctional.\n>>   \n>\n> Does anyone run \"git svn dcommit\" from a shared repository? That is the \n> only command that will trigger this code path.\n>\n> Given that you lose all the svn metadata if you do \"git clone\" (or \"git \n> clone -s\") on a git-svn-managed repository, it's not clear to me that \n> anyone would ever be bitten by this. Counterexamples welcome, of course.\n>\n> How would you feel about a separate config option to specifically enable \n> auto-pruning, and having \"git svn clone\" set that option by default? \n> Presumably anyone who is setting up a shared git-svn repository will be up \n> to the task of disabling the option.\n>\n\nSorry, I looked at 'git commit' (as you said in your mail) and not\n'git-svn dcommit'. Looking now at git-svn, I could see the there is only\ndone a git-repack if the user *explicitly* asked for it on the cmdline\nspecifying --repack. For this repack run, the default parameter includes\n-d and no --prune, so I do not think that we are doing a --prune run if\nwe where not _explicitly_ asked for it. As I said, I am totaly fine with\ndoing a 'git-gc --auto', but I am a little worried about the --prune.\n\nWe advertise everywhere that GIT adds only new content/objects/data to the\nrepository and *never* deletes anything itself in the repo and now you\nwant to do a --prune, wich obviously *does* delete data behind the users\nback in a dcommit/fetch operation, which no one would think of that these\ncommands do have anything in common with deleting data. And this worries me.\n\n-Peter\n"}]}