{"thread":{"id":"13440","subject":"git gc & deleted branches","startedAt":"2008-05-08T17:45:31Z","lastAt":"2008-05-11T18:39:12Z","messageCount":50,"participants":["Guido Ostkamp","Jeff King","Brandon Casey","Nicolas Pitre","Junio C Hamano","Geert Bosch","Chris Frey","Jeremy Maitin-Shepard","Shawn O. Pearce","drafnel@gmail.com","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"76389","messageId":"alpine.LSU.1.10.0805081920160.8678@bianca.dialin.t-online.de","threadId":"13440","inReplyTo":null,"subject":"git gc & deleted branches","fromName":"Guido Ostkamp","fromEmail":"git@ostkamp.fastmail.fm","sentAt":"2008-05-08T17:45:31Z","receivedAt":"2008-05-08T17:45:31Z","isPatch":false,"sender":{"key":"git@ostkamp.fastmail.fm","avatar":null},"body":"Hello,\n\nI'm trying to reclaim space from an abandoned branch (never involved in \nany merge) using 'git gc', but it doesn't appear to work:\n\n mkdir testrepo\n cd testrepo\n git init\n dd if=/dev/urandom bs=1024k count=10 of=file\n git add file\n git commit -a -m 'initial checkin'\n git checkout -b test\n dd if=/dev/urandom bs=1024k count=10 of=file\n git commit -a -m 'branch checkin'\n git checkout master\n du -s . # returns 30960\n git branch -D test\n git gc\n du -s . # returns 30916\n\nHere I had expected ~20000 since the branch uses ~10000.\n\nMy config is\n\n[gc]\n reflogExpire = 0\n reflogExpireUnreachable = 0\n rerereresolved = 0\n rerereunresolved = 0\n packrefs = 1\n\nI also tried 'git-pack-refs --all' or 'git-pack-refs --prune' but to no \navail.\n\nWhat am I doing wrong?\n\nThanks for any hints.\n\nRegards\n\nGuido\n"},{"id":"76394","messageId":"20080508183926.GA30613@sigill.intra.peff.net","threadId":"13440","inReplyTo":"alpine.LSU.1.10.0805081920160.8678@bianca.dialin.t-online.de","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T18:39:26Z","receivedAt":"2008-05-08T18:39:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 07:45:31PM +0200, Guido Ostkamp wrote:\n\n> [gc]\n> reflogExpire = 0\n> reflogExpireUnreachable = 0\n> rerereresolved = 0\n> rerereunresolved = 0\n> packrefs = 1\n\ngit-gc uses a \"safe\" pruning mode, where it only prunes unreferenced\nobjects that are older than a certain period (this makes it safe to run\ngit-gc, even if other processes are creating objects at the same time).\n\nSo try\n\n[gc]\n pruneExpire = now\n\nAlternatively, you can just run 'git prune' manually instead of 'git\ngc'.\n\n> I also tried 'git-pack-refs --all' or 'git-pack-refs --prune' but to no \n> avail.\n\nThose won't help at all; they are purely about moving refs from\nindividual files into the 'packed-refs' file.\n\n-Peff\n"},{"id":"76396","messageId":"alpine.LSU.1.10.0805082051210.10981@bianca.dialin.t-online.de","threadId":"13440","inReplyTo":"20080508183926.GA30613@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Guido Ostkamp","fromEmail":"git@ostkamp.fastmail.fm","sentAt":"2008-05-08T18:55:50Z","receivedAt":"2008-05-08T18:55:50Z","isPatch":false,"sender":{"key":"git@ostkamp.fastmail.fm","avatar":null},"body":"On Thu, 8 May 2008, Jeff King wrote:\n> On Thu, May 08, 2008 at 07:45:31PM +0200, Guido Ostkamp wrote:\n>\n>> [gc]\n>> reflogExpire = 0\n>> reflogExpireUnreachable = 0\n>> rerereresolved = 0\n>> rerereunresolved = 0\n>> packrefs = 1\n>\n> git-gc uses a \"safe\" pruning mode, where it only prunes unreferenced\n> objects that are older than a certain period (this makes it safe to run\n> git-gc, even if other processes are creating objects at the same time).\n>\n> So try\n>\n> [gc]\n> pruneExpire = now\n>\n> Alternatively, you can just run 'git prune' manually instead of 'git\n> gc'.\n\nJeff, I tried it, but it has no effect (see below). There is only the \nmaster branch left, and only one commit therein, still it uses the space \nformer occupied by the branch. I'm using git version 1.5.5.1.147.g867f.\n\nAny further ideas?\n\n\n$ git config -l\ncore.repositoryformatversion=0\ncore.filemode=true\ncore.bare=false\ncore.logallrefupdates=true\ngc.reflogexpire=0\ngc.reflogexpireunreachable=0\ngc.rerereresolved=0\ngc.rerereunresolved=0\ngc.packrefs=1\ngc.pruneexpire=now\n\n$ git gc\nCounting objects: 6, done.\nCompressing objects: 100% (4/4), done.\nWriting objects: 100% (6/6), done.\nTotal 6 (delta 0), reused 6 (delta 0)\n\n$ git prune\n\n$ git branch\n* master\n\n$ du -s .\n30820 .\n\n$ git log\ncommit 9717437cdcb2a4457f28f41db5f6fad9ca55b54e\nAuthor: Testuser \nDate: Thu May 8 19:40:06 2008 +0200\n\n initial checkin\n\n$ ls -l\ntotal 10240\n-rw-r--r-- 1 testuser users 10485760 May 8 19:40 file\n\nRegards\n\nGuido\n"},{"id":"76399","messageId":"48235D99.2040407@nrlssc.navy.mil","threadId":"13440","inReplyTo":"alpine.LSU.1.10.0805082051210.10981@bianca.dialin.t-online.de","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-08T20:07:53Z","receivedAt":"2008-05-08T20:07:53Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Guido Ostkamp wrote:\n> On Thu, 8 May 2008, Jeff King wrote:\n>> On Thu, May 08, 2008 at 07:45:31PM +0200, Guido Ostkamp wrote:\n>>\n>>> [gc]\n>>> reflogExpire = 0\n>>> reflogExpireUnreachable = 0\n>>> rerereresolved = 0\n>>> rerereunresolved = 0\n>>> packrefs = 1\n>>\n>> git-gc uses a \"safe\" pruning mode, where it only prunes unreferenced\n>> objects that are older than a certain period (this makes it safe to run\n>> git-gc, even if other processes are creating objects at the same time).\n>>\n>> So try\n>>\n>> [gc]\n>> pruneExpire = now\n>>\n>> Alternatively, you can just run 'git prune' manually instead of 'git\n>> gc'.\n> \n> Jeff, I tried it, but it has no effect (see below). There is only the\n> master branch left, and only one commit therein, still it uses the space\n> former occupied by the branch. I'm using git version 1.5.5.1.147.g867f.\n> \n> Any further ideas?\n\n\nPossibly that object got packed? git-prune only removes loose objects.\nTry 'git gc --prune' which will call git-repack with the -a option.\n\nbtw, this is _really_ a non-issue. It seems to keep coming up on the list.\n\nJust know that each one of the config options that you set to zero, including\nthe one Jeff suggested setting to \"now\", is a safety mechanism that is there\nto ensure that you never ever lose data and that mistakes are recoverable.\n\nAnd be assured that the objects referenced by a deleted branch will be removed\nfrom the repository eventually as long as 'git gc --prune' is run periodically.\n\n-brandon\n"},{"id":"76404","messageId":"20080508205129.GA32762@sigill.intra.peff.net","threadId":"13440","inReplyTo":"alpine.LSU.1.10.0805082051210.10981@bianca.dialin.t-online.de","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T20:51:29Z","receivedAt":"2008-05-08T20:51:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 08:55:50PM +0200, Guido Ostkamp wrote:\n\n> Jeff, I tried it, but it has no effect (see below). There is only the \n> master branch left, and only one commit therein, still it uses the space \n> former occupied by the branch. I'm using git version 1.5.5.1.147.g867f.\n\nIt worked fine for me; it's possible, as Brandon mentioned, that it is\nin a pack already, and only a \"repack -a\" would get rid of it. FWIW, my\nsteps were:\n\n # ...same as you for repo and branch creation\n git branch -D test\n git config gc.reflogexpire 0\n git config gc.reflogexpireunreachable 0\n git config gc.pruneexpire now\n git gc\n du -s .git ;# shows 10396\n\n-Peff\n"},{"id":"76405","messageId":"alpine.LSU.1.10.0805082232070.4260@bianca.dialin.t-online.de","threadId":"13440","inReplyTo":"48235D99.2040407@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Guido Ostkamp","fromEmail":"git@ostkamp.fastmail.fm","sentAt":"2008-05-08T20:52:19Z","receivedAt":"2008-05-08T20:52:19Z","isPatch":false,"sender":{"key":"git@ostkamp.fastmail.fm","avatar":null},"body":"On Thu, 8 May 2008, Brandon Casey wrote:\n> Possibly that object got packed? git-prune only removes loose objects. \n> Try 'git gc --prune' which will call git-repack with the -a option.\n\nThanks, Brandon - that did the trick :-)\n\n> Just know that each one of the config options that you set to zero, \n> including the one Jeff suggested setting to \"now\", is a safety mechanism \n> that is there to ensure that you never ever lose data and that mistakes \n> are recoverable.\n\nI am aware of this. However, at work I am unfortunately bound to a very \nrestrictive filesystem quota on central development servers, so every \nsingle byte counts in (our official versioning control system is ClearCase \nwhere less space is required due to working tree and history being \nsupplied through virtual filesystems).\n\n> And be assured that the objects referenced by a deleted branch will be \n> removed from the repository eventually as long as 'git gc --prune' is \n> run periodically.\n\nOk. I did not know about the 'prune' option yet as it neither mentioned in \nthe \"Git Tutorial\" nor \"Everyday Git\", there only 'git gc' is used with no \noptions.\n\nRegards\n\nGuido\n"},{"id":"76406","messageId":"20080508205652.GB32762@sigill.intra.peff.net","threadId":"13440","inReplyTo":"48235D99.2040407@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T20:56:53Z","receivedAt":"2008-05-08T20:56:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 03:07:53PM -0500, Brandon Casey wrote:\n\n> btw, this is _really_ a non-issue. It seems to keep coming up on the list.\n> \n> Just know that each one of the config options that you set to zero, including\n> the one Jeff suggested setting to \"now\", is a safety mechanism that is there\n> to ensure that you never ever lose data and that mistakes are recoverable.\n\nYes, I want to chime in since I have been giving advice in such threads:\nPlease don't construe my help as any sort of endorsement of this\nbehavior. Git tries hard not to lose your data, and it is almost always\na bad idea to try to override these safety checks unless you really know\nwhat you are doing.\n\nAnd even then, try to consider balancing a bit of freed disk space (and\ngenerally _no_ performance gain, because git is very good about not\nlooking at objects that aren't necessary to the current operation)\nversus thinking \"oops, I wish I still had that data\" in a few days.\n\nI can think offhand of only one time when it was truly useful for me to\nprune aggressively, and it was a very special case: a pathologically\nlarge repo for which I was doing a one-shot conversion from another\nformat (and I wanted to prune failed attempts).\n\n-Peff\n"},{"id":"76407","messageId":"20080508210125.GC32762@sigill.intra.peff.net","threadId":"13440","inReplyTo":"alpine.LSU.1.10.0805082232070.4260@bianca.dialin.t-online.de","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T21:01:25Z","receivedAt":"2008-05-08T21:01:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 10:52:19PM +0200, Guido Ostkamp wrote:\n\n>> And be assured that the objects referenced by a deleted branch will be \n>> removed from the repository eventually as long as 'git gc --prune' is \n>> run periodically.\n>\n> Ok. I did not know about the 'prune' option yet as it neither mentioned in \n> the \"Git Tutorial\" nor \"Everyday Git\", there only 'git gc' is used with no \n> options.\n\nIt is deprecated; see 25ee9731.\n\nAccording to that commit message, prune is now a no-op. However, it\nlooks like it is still used for trigger a \"repack -a\" rather than\n\"repack -A\". I don't know if it is worth making that behavior available\nthrough some more sane command line option (I would think people who\nreally know that they want \"repack -a\" would just call it).\n\n-Peff\n"},{"id":"76409","messageId":"alpine.LFD.1.10.0805081712270.23581@xanadu.home","threadId":"13440","inReplyTo":"20080508210125.GC32762@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-08T21:15:34Z","receivedAt":"2008-05-08T21:15:34Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 8 May 2008, Jeff King wrote:\n\n> On Thu, May 08, 2008 at 10:52:19PM +0200, Guido Ostkamp wrote:\n> \n> >> And be assured that the objects referenced by a deleted branch will be \n> >> removed from the repository eventually as long as 'git gc --prune' is \n> >> run periodically.\n> >\n> > Ok. I did not know about the 'prune' option yet as it neither mentioned in \n> > the \"Git Tutorial\" nor \"Everyday Git\", there only 'git gc' is used with no \n> > options.\n> \n> It is deprecated; see 25ee9731.\n> \n> According to that commit message, prune is now a no-op. However, it\n> looks like it is still used for trigger a \"repack -a\" rather than\n> \"repack -A\". I don't know if it is worth making that behavior available\n> through some more sane command line option (I would think people who\n> really know that they want \"repack -a\" would just call it).\n\nWell, actually this is a problem.\n\nI think it is a good thing to deprecate gc --prune. but if that means \nthat repack -a is never used then unreferenced and expired objects will \nnever be pruned if they're packed if one is always using 'git gc' as we \nare advocating.\n\n\nNicolas\n"},{"id":"76410","messageId":"20080508211734.GA819@sigill.intra.peff.net","threadId":"13440","inReplyTo":"alpine.LFD.1.10.0805081712270.23581@xanadu.home","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T21:17:34Z","receivedAt":"2008-05-08T21:17:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 05:15:34PM -0400, Nicolas Pitre wrote:\n\n> > According to that commit message, prune is now a no-op. However, it\n> > looks like it is still used for trigger a \"repack -a\" rather than\n> > \"repack -A\". I don't know if it is worth making that behavior available\n> \n> Well, actually this is a problem.\n> \n> I think it is a good thing to deprecate gc --prune. but if that means \n> that repack -a is never used then unreferenced and expired objects will \n> never be pruned if they're packed if one is always using 'git gc' as we \n> are advocating.\n\nI thought that -A would eventually put them all into a single pack,\nkilling off the old packs.\n\n-Peff\n"},{"id":"76411","messageId":"48236F69.2060900@nrlssc.navy.mil","threadId":"13440","inReplyTo":"20080508211734.GA819@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-08T21:23:53Z","receivedAt":"2008-05-08T21:23:53Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Jeff King wrote:\n> On Thu, May 08, 2008 at 05:15:34PM -0400, Nicolas Pitre wrote:\n> \n>>> According to that commit message, prune is now a no-op. However, it\n>>> looks like it is still used for trigger a \"repack -a\" rather than\n>>> \"repack -A\". I don't know if it is worth making that behavior available\n>> Well, actually this is a problem.\n>>\n>> I think it is a good thing to deprecate gc --prune. but if that means \n>> that repack -a is never used then unreferenced and expired objects will \n>> never be pruned if they're packed if one is always using 'git gc' as we \n>> are advocating.\n> \n> I thought that -A would eventually put them all into a single pack,\n> killing off the old packs.\n\n'-a' puts everything in a single pack and kills off old packs. Anything that\nwas unreachable is not repacked in the new pack.\n\n'-A' does the same thing but it also repacks the unreachable objects that were\npreviously packed.\n\nSo if something gets packed that subsequently becomes unreachable it will never\nbe removed unless 'repack -a' is used.\n\nPossibly --keep-unreachable should instead unpack the unreachable items which would\nallow them to eventually be pruned based on pruneExpire. Then we could indeed\nget rid of the --prune option to git-gc.\n\n-brandon\n"},{"id":"76413","messageId":"20080508213107.GA1016@sigill.intra.peff.net","threadId":"13440","inReplyTo":"48236F69.2060900@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T21:31:07Z","receivedAt":"2008-05-08T21:31:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 04:23:53PM -0500, Brandon Casey wrote:\n\n> > I thought that -A would eventually put them all into a single pack,\n> > killing off the old packs.\n> \n> '-a' puts everything in a single pack and kills off old packs. Anything that\n> was unreachable is not repacked in the new pack.\n> \n> '-A' does the same thing but it also repacks the unreachable objects that were\n> previously packed.\n\nAh, indeed. I hadn't looked closely at the -A behavior before. So yes,\nwe are never killing off prunable packed objects. Probably we could use\nthe same solution as \"git prune --expire\"; perhaps a\n\"--keep-unreachable=2.weeks.ago\"?\n\n-Peff\n"},{"id":"76414","messageId":"alpine.LSU.1.10.0805082330520.4850@bianca.dialin.t-online.de","threadId":"13440","inReplyTo":"20080508210125.GC32762@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Guido Ostkamp","fromEmail":"git@ostkamp.fastmail.fm","sentAt":"2008-05-08T21:33:39Z","receivedAt":"2008-05-08T21:33:39Z","isPatch":false,"sender":{"key":"git@ostkamp.fastmail.fm","avatar":null},"body":"On Thu, 8 May 2008, Jeff King wrote:\n> According to that commit message, prune is now a no-op. However, it \n> looks like it is still used for trigger a \"repack -a\" rather than \n> \"repack -A\".\n\nI tried to look at this but found this option '-A' to be undocumented in \nthe manpage (git/Documentation/git-repack.txt and what is generated from \nit).\n\nRegards\n\nGuido\n"},{"id":"76415","messageId":"48237344.6070405@nrlssc.navy.mil","threadId":"13440","inReplyTo":"20080508213107.GA1016@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-08T21:40:20Z","receivedAt":"2008-05-08T21:40:20Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Jeff King wrote:\n> On Thu, May 08, 2008 at 04:23:53PM -0500, Brandon Casey wrote:\n> \n>>> I thought that -A would eventually put them all into a single pack,\n>>> killing off the old packs.\n>> '-a' puts everything in a single pack and kills off old packs. Anything that\n>> was unreachable is not repacked in the new pack.\n>>\n>> '-A' does the same thing but it also repacks the unreachable objects that were\n>> previously packed.\n> \n> Ah, indeed. I hadn't looked closely at the -A behavior before. So yes,\n> we are never killing off prunable packed objects. Probably we could use\n> the same solution as \"git prune --expire\"; perhaps a\n> \"--keep-unreachable=2.weeks.ago\"?\n\nThe 'prune --expire' behavior is based on object mtime (i.e. file modification time).\nThat is lost once something is packed right?\n\nI was thinking that either repack or pack-objects could be modified to unpack those\nunreachable objects and leave them loose, and also give them the timestamp of the\npack file they came from. Then the --expire behavior of git-prune could work normally\nand remove them. This seems like it would work nicely since prune follows repack in\ngit-gc.\n\n-brandon\n"},{"id":"76416","messageId":"20080508214454.GA1939@sigill.intra.peff.net","threadId":"13440","inReplyTo":"48237344.6070405@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T21:44:54Z","receivedAt":"2008-05-08T21:44:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 04:40:20PM -0500, Brandon Casey wrote:\n\n> The 'prune --expire' behavior is based on object mtime (i.e. file\n> modification time). That is lost once something is packed right?\n\nYes. You would have to use the pack mtime. But of course you would have\nto actually _leave_ them in a pack, or they would just keep getting\nadded to the new pack.\n\n> I was thinking that either repack or pack-objects could be modified to\n> unpack those unreachable objects and leave them loose, and also give\n> them the timestamp of the pack file they came from. Then the --expire\n> behavior of git-prune could work normally and remove them. This seems\n> like it would work nicely since prune follows repack in git-gc.\n\nThat is sensible, I think.\n\n-Peff\n"},{"id":"76417","messageId":"48237650.5060008@nrlssc.navy.mil","threadId":"13440","inReplyTo":"20080508214454.GA1939@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-08T21:53:20Z","receivedAt":"2008-05-08T21:53:20Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Jeff King wrote:\n> On Thu, May 08, 2008 at 04:40:20PM -0500, Brandon Casey wrote:\n> \n>> The 'prune --expire' behavior is based on object mtime (i.e. file\n>> modification time). That is lost once something is packed right?\n> \n> Yes. You would have to use the pack mtime. But of course you would have\n> to actually _leave_ them in a pack, or they would just keep getting\n> added to the new pack.\n\nI had the impression that unreachable objects would not be packed. Maybe it\nwas more of an assumption.\n\n-brandon\n"},{"id":"76418","messageId":"20080508224827.GA2938@sigill.intra.peff.net","threadId":"13440","inReplyTo":"48237650.5060008@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-08T22:48:27Z","receivedAt":"2008-05-08T22:48:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2008 at 04:53:20PM -0500, Brandon Casey wrote:\n\n> > Yes. You would have to use the pack mtime. But of course you would have\n> > to actually _leave_ them in a pack, or they would just keep getting\n> > added to the new pack.\n> \n> I had the impression that unreachable objects would not be packed. Maybe it\n> was more of an assumption.\n\nLook in builtin-pack-objects.c:1981-1982. We basically just say \"if it's\nin a pack now, then it should go into the new pack.\"\n\n-Peff\n"},{"id":"76427","messageId":"loom.20080509T011318-478@post.gmane.org","threadId":"13440","inReplyTo":"20080508224827.GA2938@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-09T01:41:30Z","receivedAt":"2008-05-09T01:41:30Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Jeff King peff.net> writes:\n\n> \n> On Thu, May 08, 2008 at 04:53:20PM -0500, Brandon Casey wrote:\n> \n> > > Yes. You would have to use the pack mtime. But of course you would have\n> > > to actually _leave_ them in a pack, or they would just keep getting\n> > > added to the new pack.\n> > \n> > I had the impression that unreachable objects would not be packed. Maybe it\n> > was more of an assumption.\n> \n> Look in builtin-pack-objects.c:1981-1982. We basically just say \"if it's\n> in a pack now, then it should go into the new pack.\"\n> \n> -Peff\n> \n\n\nHere's what I was thinking (posted using gmane):\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex e18eb3f..064c331 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -30,7 +30,7 @@ do\n -n) no_update_info=t ;;\n -a) all_into_one=t ;;\n -A) all_into_one=t\n- keep_unreachable=--keep-unreachable ;;\n+ keep_unreachable=t ;;\n -d) remove_redundant=t ;;\n -q) quiet=-q ;;\n -f) no_reuse=--no-reuse-object ;;\n@@ -78,9 +78,6 @@ case \",$all_into_one,\" in\n if test -z \"$args\"\n then\n args='--unpacked --incremental'\n- elif test -n \"$keep_unreachable\"\n- then\n- args=\"$args $keep_unreachable\"\n fi\n ;;\n esac\n@@ -116,7 +113,15 @@ for name in $names ; do\n echo >&2 \"old-pack-$name.{pack,idx} in $PACKDIR.\"\n exit 1\n }\n- rm -f \"$PACKDIR/old-pack-$name.pack\" \"$PACKDIR/old-pack-$name.idx\"\n+ rm -f \"$PACKDIR/old-pack-$name.idx\"\n+ test -z \"$keep_unreachable\" ||\n+ ! test -f \"$PACKDIR/old-pack-$name.pack\" ||\n+ git unpack-objects < \"$PACKDIR/old-pack-$name.pack\" || {\n+ echo >&2 \"Failed unpacking unreachable objects from old pack\"\n+ echo >&2 \"saved as old-pack-$name.pack in $PACKDIR.\"\n+ exit 1\n+ }\n+ rm -f \"$PACKDIR/old-pack-$name.pack\"\n done\n \n if test \"$remove_redundant\" = t\n@@ -130,7 +135,18 @@ then\n do\n case \" $fullbases \" in\n *\" $e \"*) ;;\n- *) rm -f \"$e.pack\" \"$e.idx\" \"$e.keep\" ;;\n+ *)\n+ rm -f \"$e.idx\" \"$e.keep\"\n+ if test -n \"$keep_unreachable\" &&\n+ test -f \"$e.pack\"\n+ then\n+ git unpack-objects < \"$e.pack\" || {\n+ echo >&2 \"Fail AVOID GMANE WRAP\"\n+ exit 1\n+ }\n+ fi\n+ rm -f \"$e.pack\"\n+ ;;\n esac\n done\n )\n\n\nIs the first invocation of unpack-objects necessary? pack-objects has created\na pack which hashes to the same name of a pack we already have, and we replace\nthe original with the new one. Is that what is happening? They will be identical\nright?\n\nOf course this won't set the timestamp on the created objects based on the\ntimestamp of the pack file, but this was easy. Setting the timestamp would be\nproper, but what's another two weeks. Besides, for those users not manually\nrunning git-gc, this code path won't even be executed until there are enough\npack files for git-gc to add -A to the repack options.\n\nThen, for git-gc it should be enough to just always use -A with repack when\nmanually running it. Then --prune can be deprecated.\n\n-brandon\n"},{"id":"76439","messageId":"7vabj0b1re.fsf@gitster.siamese.dyndns.org","threadId":"13440","inReplyTo":"loom.20080509T011318-478@post.gmane.org","subject":"Re: git gc & deleted branches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-09T03:21:57Z","receivedAt":"2008-05-09T03:21:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Casey writes:\n\n> @@ -116,7 +113,15 @@ for name in $names ; do\n> echo >&2 \"old-pack-$name.{pack,idx} in $PACKDIR.\"\n> exit 1\n> }\n> - rm -f \"$PACKDIR/old-pack-$name.pack\" \"$PACKDIR/old-pack-$name.idx\"\n> + rm -f \"$PACKDIR/old-pack-$name.idx\"\n> + test -z \"$keep_unreachable\" ||\n> + ! test -f \"$PACKDIR/old-pack-$name.pack\" ||\n> + git unpack-objects < \"$PACKDIR/old-pack-$name.pack\" || {\n> + echo >&2 \"Failed unpacking unreachable objects from old pack\"\n> + echo >&2 \"saved as old-pack-$name.pack in $PACKDIR.\"\n> + exit 1\n> + }\n\nNeat trick. Unreachable objects that are only in this pack will get\ncurrent timestamp and gets a new lease of life for two weeks and then will\ndisappear.\n"},{"id":"76440","messageId":"20080509041921.GA14773@sigill.intra.peff.net","threadId":"13440","inReplyTo":"loom.20080509T011318-478@post.gmane.org","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-09T04:19:21Z","receivedAt":"2008-05-09T04:19:21Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 09, 2008 at 01:41:30AM +0000, Brandon Casey wrote:\n\n> Here's what I was thinking (posted using gmane):\n> \n> diff --git a/git-repack.sh b/git-repack.sh\n> index e18eb3f..064c331 100755\n> --- a/git-repack.sh\n> +++ b/git-repack.sh\n\nI like it. It makes an easy rule to say \"packed objects _never_ get\npruned, they only get demoted to loose objects.\" And then of course\nwe have sane rules for pruning loose objects.\n\n> - rm -f \"$PACKDIR/old-pack-$name.pack\" \"$PACKDIR/old-pack-$name.idx\"\n> + rm -f \"$PACKDIR/old-pack-$name.idx\"\n> + test -z \"$keep_unreachable\" ||\n> + ! test -f \"$PACKDIR/old-pack-$name.pack\" ||\n> + git unpack-objects < \"$PACKDIR/old-pack-$name.pack\" || {\n> + echo >&2 \"Failed unpacking unreachable objects from old pack\"\n> + echo >&2 \"saved as old-pack-$name.pack in $PACKDIR.\"\n> + exit 1\n> + }\n> + rm -f \"$PACKDIR/old-pack-$name.pack\"\n> [...]\n> \n> Is the first invocation of unpack-objects necessary? pack-objects has created\n> a pack which hashes to the same name of a pack we already have, and we replace\n> the original with the new one. Is that what is happening? They will be\n> identical right?\n\nYeah, that's what it looks like to me (that the first unpack is\nunnecessary, because we will just be putting the new pack into place\nthat has all the same objects). AIUI, two packs with identical hashes\nmust contain the exact same objects.\n\n> Of course this won't set the timestamp on the created objects based on the\n> timestamp of the pack file, but this was easy. Setting the timestamp would be\n> proper, but what's another two weeks. Besides, for those users not manually\n> running git-gc, this code path won't even be executed until there are enough\n> pack files for git-gc to add -A to the repack options.\n\nI think the extra two weeks is fine.\n\n-Peff\n"},{"id":"76457","messageId":"E1B43061-69C7-43D7-9A57-34B7C55DF345@adacore.com","threadId":"13440","inReplyTo":"20080509041921.GA14773@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2008-05-09T15:00:59Z","receivedAt":"2008-05-09T15:00:59Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn May 9, 2008, at 00:19, Jeff King wrote:\n\n> I like it. It makes an easy rule to say \"packed objects _never_ get\n> pruned, they only get demoted to loose objects.\" And then of course\n> we have sane rules for pruning loose objects.\n\nIsn't there an issue with the \"git gc\" triggering because there\nmay be too many loose unreferenced objects?\nStill, I do like the approach.\n\nMaybe unreferenced objects and old refs should go to a .git/lost+found\ndirectory and be expired from there. This has a couple of benefits:\n\n - Easy to manually inspect or blow away any crud\n - One git-gc run can make one pack in lost+found,\n avoiding huge numbers of loose objects (and massive disk use)\n when trying to do a large cleanup (to possibly reclaim disk space)\n - Objects will not be accessible by ordinary git commands for a \nwhile,\n before they are really removed, avoiding surprises\n\nOnly some tools would look in the lost+found to restore stuff.\n\n -Geert\n"},{"id":"76458","messageId":"48246A44.7020303@nrlssc.navy.mil","threadId":"13440","inReplyTo":"E1B43061-69C7-43D7-9A57-34B7C55DF345@adacore.com","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-09T15:14:12Z","receivedAt":"2008-05-09T15:14:12Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Geert Bosch wrote:\n> \n> On May 9, 2008, at 00:19, Jeff King wrote:\n> \n>> I like it. It makes an easy rule to say \"packed objects _never_ get\n>> pruned, they only get demoted to loose objects.\" And then of course\n>> we have sane rules for pruning loose objects.\n> \n> Isn't there an issue with the \"git gc\" triggering because there\n> may be too many loose unreferenced objects?\n> Still, I do like the approach.\n\nThis would be an argument for going the extra mile and having the loose\nobjects adopt the timestamp of their pack file. In the normal case they\nwould probably be pruned immediately during the same git-gc run.\n\n> Maybe unreferenced objects and old refs should go to a .git/lost+found\n> directory and be expired from there. This has a couple of benefits:\n\n> - Objects will not be accessible by ordinary git commands for a while,\n> before they are really removed, avoiding surprises\n\nUnreferenced objects are sometimes used by other repositories which have\nthis repository listed as an alternate. So it may not be a good idea to\nmake the unreferenced objects inaccessible.\n\n-brandon\n"},{"id":"76459","messageId":"20080509155323.GA28966@sigill.intra.peff.net","threadId":"13440","inReplyTo":"48246A44.7020303@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-09T15:53:24Z","receivedAt":"2008-05-09T15:53:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 09, 2008 at 10:14:12AM -0500, Brandon Casey wrote:\n\n> > - Objects will not be accessible by ordinary git commands for a while,\n> > before they are really removed, avoiding surprises\n> \n> Unreferenced objects are sometimes used by other repositories which have\n> this repository listed as an alternate. So it may not be a good idea to\n> make the unreferenced objects inaccessible.\n\nBut that is precisely what we're going to do, but in two weeks. Isn't it\nbetter to have the dependent repo fail while the change is recoverable?\n\n-Peff\n"},{"id":"76460","messageId":"4824741E.4080807@nrlssc.navy.mil","threadId":"13440","inReplyTo":"20080509155323.GA28966@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-09T15:56:14Z","receivedAt":"2008-05-09T15:56:14Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Jeff King wrote:\n> On Fri, May 09, 2008 at 10:14:12AM -0500, Brandon Casey wrote:\n> \n>>> - Objects will not be accessible by ordinary git commands for a while,\n>>> before they are really removed, avoiding surprises\n>> Unreferenced objects are sometimes used by other repositories which have\n>> this repository listed as an alternate. So it may not be a good idea to\n>> make the unreferenced objects inaccessible.\n> \n> But that is precisely what we're going to do, but in two weeks. Isn't it\n> better to have the dependent repo fail while the change is recoverable?\n\ngood point.\n\n-b\n"},{"id":"76463","messageId":"alpine.LFD.1.10.0805091205580.23581@xanadu.home","threadId":"13440","inReplyTo":"48246A44.7020303@nrlssc.navy.mil","subject":"Re: git gc & deleted branches","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-09T16:12:22Z","receivedAt":"2008-05-09T16:12:22Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 9 May 2008, Brandon Casey wrote:\n\n> Geert Bosch wrote:\n> > \n> > On May 9, 2008, at 00:19, Jeff King wrote:\n> > \n> >> I like it. It makes an easy rule to say \"packed objects _never_ get\n> >> pruned, they only get demoted to loose objects.\" And then of course\n> >> we have sane rules for pruning loose objects.\n> > \n> > Isn't there an issue with the \"git gc\" triggering because there\n> > may be too many loose unreferenced objects?\n> > Still, I do like the approach.\n> \n> This would be an argument for going the extra mile and having the loose\n> objects adopt the timestamp of their pack file. In the normal case they\n> would probably be pruned immediately during the same git-gc run.\n\nWell, not necessarily. If you created a large branch yesterday and you \nare deleting it today, then if you repacked in between means that those \nloose objects won't be more than one day old. Yet there could be enough \nof them to trigger auto gc. But that auto gc won't pack those objects \nsince they are unreferenced. Hence auto gc will trigger all the time \nwithout making any progress.\n\n> > Maybe unreferenced objects and old refs should go to a .git/lost+found\n> > directory and be expired from there. This has a couple of benefits:\n> \n> > - Objects will not be accessible by ordinary git commands for a while,\n> > before they are really removed, avoiding surprises\n> \n> Unreferenced objects are sometimes used by other repositories which have\n> this repository listed as an alternate. So it may not be a good idea to\n> make the unreferenced objects inaccessible.\n\nNah. If this is really the case then you shouldn't be running gc at all \nin the first place.\n\n\nNicolas\n"},{"id":"76466","messageId":"482481B6.5010106@nrlssc.navy.mil","threadId":"13440","inReplyTo":"alpine.LFD.1.10.0805091205580.23581@xanadu.home","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-05-09T16:54:14Z","receivedAt":"2008-05-09T16:54:14Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Nicolas Pitre wrote:\n> On Fri, 9 May 2008, Brandon Casey wrote:\n> \n>> Geert Bosch wrote:\n>>> On May 9, 2008, at 00:19, Jeff King wrote:\n>>>\n>>>> I like it. It makes an easy rule to say \"packed objects _never_ get\n>>>> pruned, they only get demoted to loose objects.\" And then of course\n>>>> we have sane rules for pruning loose objects.\n>>> Isn't there an issue with the \"git gc\" triggering because there\n>>> may be too many loose unreferenced objects?\n>>> Still, I do like the approach.\n>> This would be an argument for going the extra mile and having the loose\n>> objects adopt the timestamp of their pack file. In the normal case they\n>> would probably be pruned immediately during the same git-gc run.\n> \n> Well, not necessarily. If you created a large branch yesterday and you \n> are deleting it today, then if you repacked in between means that those \n> loose objects won't be more than one day old. Yet there could be enough \n> of them to trigger auto gc. But that auto gc won't pack those objects \n> since they are unreferenced. Hence auto gc will trigger all the time \n> without making any progress.\n\nThat's true, but the intermediate repack is not the cause here. You'd be\nin the same situation if a large branch was created yesterday and then\ndeleted today even if packing had never occurred.\n\nI do see your point, but you should have said a large branch created a month\nago, deleted today, but repacked yesterday. :)\n\n-brandon\n"},{"id":"76474","messageId":"7vwsm39kft.fsf@gitster.siamese.dyndns.org","threadId":"13440","inReplyTo":"alpine.LFD.1.10.0805091205580.23581@xanadu.home","subject":"Re: git gc & deleted branches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-09T22:33:41Z","receivedAt":"2008-05-09T22:33:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> On Fri, 9 May 2008, Brandon Casey wrote:\n>\n>> Unreferenced objects are sometimes used by other repositories which have\n>> this repository listed as an alternate. So it may not be a good idea to\n>> make the unreferenced objects inaccessible.\n>\n> Nah. If this is really the case then you shouldn't be running gc at all \n> in the first place.\n\nTrue.\n\nI think the true motivation behind --keep-unreachable is not about the\nshared object store (aka \"alternates\") but about races between gc and\npush (or fetch). Before push (or fetch) finishes and updates refs, the\nnew objects they create would be dangling _and_ the objects these dangling\nobjects refer to may be packed but unreferenced. Repacking unreferenced\npacked objects was a way to avoid losing them.\n"},{"id":"76476","messageId":"20080509230950.GA2733@foursquare.net","threadId":"13440","inReplyTo":"7vwsm39kft.fsf@gitster.siamese.dyndns.org","subject":"[PATCH] Updating documentation to match Brandon Casey's proposed git-repack patch.","fromName":"Chris Frey","fromEmail":"cdfrey@foursquare.net","sentAt":"2008-05-09T23:09:50Z","receivedAt":"2008-05-09T23:09:50Z","isPatch":true,"sender":{"key":"cdfrey@foursquare.net","avatar":null},"body":"On Fri, May 09, 2008 at 03:33:41PM -0700, Junio C Hamano wrote:\n> I think the true motivation behind --keep-unreachable is not about the\n> shared object store (aka \"alternates\") but about races between gc and\n> push (or fetch). Before push (or fetch) finishes and updates refs, the\n> new objects they create would be dangling _and_ the objects these dangling\n> objects refer to may be packed but unreferenced. Repacking unreferenced\n> packed objects was a way to avoid losing them.\n\nThis is what the log history seems to indicate:\n\n\tgit log -p --grep=keep-unreach\n\nSo pack-objects --keep-unreachable was implemented in order to add repack -A,\nwhich now doesn't need --keep-unreachable, and can become obsolete.\n\nWhich is just as well, since --keep-unreachable never made it to the\nman pages. :-)\n\nIf I understand things correctly, there is no user-friendly way to add\nloose, unreachable objects to a pack. This whole architecture was just\nto prevent a repack from silently deleting things.\n\nIf this is right, the patch below updates the docs.\n\n- Chris\n\n\n>From 443b1201d54f0b7197d18779ce934823e9897b36 Mon Sep 17 00:00:00 2001\nFrom: Chris Frey \nDate: Fri, 9 May 2008 19:08:26 -0400\nSubject: [PATCH] Updating documentation to match Brandon Casey's proposed git-repack patch.\n\nThis patch clarifies the git-prune man page, documenting that it only\nprunes unpacked objects. git-repack is documented according to\nthe new git-repack -A behaviour, which does not depend on\ngit-pack-objects --keep-unreachable anymore.\n\nSigned-off-by: Chris Frey \n---\n Documentation/git-prune.txt | 5 ++++-\n Documentation/git-repack.txt | 14 +++++++++++++-\n 2 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-prune.txt b/Documentation/git-prune.txt\nindex f92bb8c..3178bc4 100644\n--- a/Documentation/git-prune.txt\n+++ b/Documentation/git-prune.txt\n@@ -18,12 +18,15 @@ git-prune. See the section \"NOTES\", below.\n \n This runs `git-fsck --unreachable` using all the refs\n available in `$GIT_DIR/refs`, optionally with additional set of\n-objects specified on the command line, and prunes all\n+objects specified on the command line, and prunes all unpacked\n objects unreachable from any of these head objects from the object database.\n In addition, it\n prunes the unpacked objects that are also found in packs by\n running `git prune-packed`.\n \n+Note that unreachable, packed objects will remain. If this is\n+not desired, see linkgit:git-repack[1].\n+\n OPTIONS\n -------\n \ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 3d95749..906d3c7 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository\n \n SYNOPSIS\n --------\n-'git-repack' [-a] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]\n+'git-repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]\n \n DESCRIPTION\n -----------\n@@ -37,6 +37,18 @@ OPTIONS\n \tleaves behind, but `git fsck --full` shows as\n \tdangling.\n \n+-A::\n+\tSame as `-a`, but any unreachable objects in a previous\n+\tpack become loose, unpacked objects, instead of being\n+\tleft in the old pack. Unreachable objects are never\n+\tintentionally added to a pack, even when repacking.\n+\tWhen used with '-d', this option\n+\tprevents unreachable objects from being immediately\n+\tdeleted by way of being left in the old pack and then\n+\tremoved. Instead, the loose unreachable objects\n+\twill be pruned according to normal expiry rules\n+\twith the next linkgit:git-gc[1].\n+\n -d::\n \tAfter packing, if the newly created packs make some\n \texisting packs redundant, remove the redundant packs.\n-- \n1.5.4.4\n"},{"id":"76479","messageId":"877ie3yqb3.fsf@jeremyms.com","threadId":"13440","inReplyTo":"7vwsm39kft.fsf@gitster.siamese.dyndns.org","subject":"Re: git gc & deleted branches","fromName":"Jeremy Maitin-Shepard","fromEmail":"jbms@cmu.edu","sentAt":"2008-05-10T00:07:44Z","receivedAt":"2008-05-10T00:07:44Z","isPatch":false,"sender":{"key":"jbms@cmu.edu","avatar":null},"body":"Junio C Hamano writes:\n\n> Nicolas Pitre writes:\n>> On Fri, 9 May 2008, Brandon Casey wrote:\n>> \n>>> Unreferenced objects are sometimes used by other repositories which have\n>>> this repository listed as an alternate. So it may not be a good idea to\n>>> make the unreferenced objects inaccessible.\n>> \n>> Nah. If this is really the case then you shouldn't be running gc at all \n>> in the first place.\n\n> True.\n\n> I think the true motivation behind --keep-unreachable is not about the\n> shared object store (aka \"alternates\") but about races between gc and\n> push (or fetch). Before push (or fetch) finishes and updates refs, the\n> new objects they create would be dangling _and_ the objects these dangling\n> objects refer to may be packed but unreferenced. Repacking unreferenced\n> packed objects was a way to avoid losing them.\n\nI feel like the current approach of (not very well) keeping track of\nwhich objects are still needed is very messy, not very well defined or\nbased on specific solid principles, and prone to errors and losing\nobjects.\n\nThings like git clone -shared can only really be used in extremely\nspecialized setups, or if pruning of unreferenced objects is completely\ndisabled in the source repository, or if specialized scripts are used to\ndo the garbage collection that take into account the references of the\n\"child\" repository. It is my impression that even repo.or.cz, while it\nhas some safe guards, does not even completely safely handle garbage\ncollection. Probably it would be very useful to examples of such\nscripts in contrib.\n\nI think that ultimately, some general purpose and reliable solution\nneeds to be found to handle the cases of (1) a repository having its\nobjects referenced by another via info/alternates; (2) a repository with\nmultiple working directories (presumably this should warn/error out\nunless given a force option/detach head and warn if you try to switch\nHEAD for some working directory to the same branch as some other working\ndirectory). It seems, btw, that a third type of clone, one which merely\nsymlinks the objects directory, would also be useful, once there is a\nsolution to the robustness issue. This would be a case (3) that needs\nto be handled as well.\n\nIt seems that clear that ultimately, to handle these three cases, every\nrepository needs to know about every other repository, probably via a\nsymlink to other repository's .git directory. Git gc would then also\nexamine any refs in this directory, making sure to avoid circular\nreferences that might result from following the symlinks. It should\nalso probably error out if it finds a symlink that doesn't point to a\nvalid git repository, because such a symlink either refers to a\nnow-deleted repository for which the symlink needs to be cleaned up, or\nit refers to a repository that was moved and therefore the symlink needs\nto be updated. Simply ignoring invalid symlinks could result in pruning\nobjects that need to be kept for repositories that have moved.\n\nIt is extremely cumbersome to have to worry about whether there are\nother concurrent accesses to the repository when running e.g. git gc.\nFor servers, you may never be able to guarantee that nothing else is\naccessing the repository concurrently. Here is a possible solution:\n\nEach git process creates a log file of the references that it has\ncreated. The log file should be named in some way with e.g. the process\nid and start time of the process, and simply consist of a list of\n20-byte sha1 hashes to be considered additional in-use references for\nthe purpose of garbage collection. The log file would be cleaned up\nwhen the process exits, and would also be deleted by any instance of git\ngc that notices a stale log file that doesn't correspond to a running\nprocess. To handle shell scripts that need to deal with git-hash-object\ndirectly, git hash-object could be passed maybe a file descriptor or\nfilename of a log file to use instead of creating one. Maybe the log\nfile format could be more complicated, and also support paths to\ne.g. alternate index files to also consider for references. Things\nwould need to be one so that race conditions do not occur, but I think\nsomething like this would work.\n\n-- \nJeremy Maitin-Shepard\n"},{"id":"76481","messageId":"20080510002014.GH29038@spearce.org","threadId":"13440","inReplyTo":"877ie3yqb3.fsf@jeremyms.com","subject":"Re: git gc & deleted branches","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-10T00:20:14Z","receivedAt":"2008-05-10T00:20:14Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jeremy Maitin-Shepard wrote:\n> It is extremely cumbersome to have to worry about whether there are\n> other concurrent accesses to the repository when running e.g. git gc.\n> For servers, you may never be able to guarantee that nothing else is\n> accessing the repository concurrently. Here is a possible solution:\n> \n> Each git process creates a log file of the references that it has\n> created. The log file should be named in some way with e.g. the process\n> id and start time of the process, and simply consist of a list of\n> 20-byte sha1 hashes to be considered additional in-use references for\n> the purpose of garbage collection.\n\nI believe we partially considered that in the past and discarded it\nas far too complex implementation-wise for the benefit it gives us.\n\nThe current approach of leaving unreachable loose objects around\nfor 2 weeks is good enough. Any Git process that has been running\nfor 2 weeks while still not linking everything it needs into the\nreachable refs of that repository is already braindamaged and\nshouldn't be running anymore.\n\nIf we are dealing with a pack file, those are protected by .keep\n\"lock files\" between the time they are created on disk and the\ntime that the git-fetch or git-receive-pack process has finished\nupdating the refs to anchor the pack's contents as reachable.\nEvery once in a while a stale .keep file gets left behind when a\nprocess gets killed by the OS, and its damn annoying to clean up.\n\nI'd hate to clean up logs from every little git-add or git-commit\nthat aborted in the middle uncleanly.\n\n-- \nShawn.\n"},{"id":"76482","messageId":"873aoryomv.fsf@jeremyms.com","threadId":"13440","inReplyTo":"20080510002014.GH29038@spearce.org","subject":"Re: git gc & deleted branches","fromName":"Jeremy Maitin-Shepard","fromEmail":"jbms@cmu.edu","sentAt":"2008-05-10T00:43:52Z","receivedAt":"2008-05-10T00:43:52Z","isPatch":false,"sender":{"key":"jbms@cmu.edu","avatar":null},"body":"\"Shawn O. Pearce\" writes:\n\n> Jeremy Maitin-Shepard wrote:\n>> It is extremely cumbersome to have to worry about whether there are\n>> other concurrent accesses to the repository when running e.g. git gc.\n>> For servers, you may never be able to guarantee that nothing else is\n>> accessing the repository concurrently. Here is a possible solution:\n>> \n>> Each git process creates a log file of the references that it has\n>> created. The log file should be named in some way with e.g. the process\n>> id and start time of the process, and simply consist of a list of\n>> 20-byte sha1 hashes to be considered additional in-use references for\n>> the purpose of garbage collection.\n\n> I believe we partially considered that in the past and discarded it\n> as far too complex implementation-wise for the benefit it gives us.\n\nIt doesn't seem all that complex, and I'd say that fundamentally it is\nthe _correct_ way to do things. Being sloppy is always easier in the\nshort run, but then either means the system is permanently broken or\nresults in a lot of \"fixing up\" work later. I think almost all of the\nwork of handling these log files could be done without impacting a lot\nof code that calls the relevant APIs that would actually use the log\nfiles. I think the biggest impact would be on non-C code, but even for\nthat code, appropriate wrapper could be used to avoid having to make\nmany changes.\n\n> The current approach of leaving unreachable loose objects around\n> for 2 weeks is good enough. Any Git process that has been running\n> for 2 weeks while still not linking everything it needs into the\n> reachable refs of that repository is already braindamaged and\n> shouldn't be running anymore.\n\nThis sort of reasoning just leads to an inherently unreliable system.\nSure, two weeks might seem good enough for nearly all cases, but why\n_shouldn't_ I be able to leave my editor open for two weeks before\ntyping in my commit message and finishing the commit, or wait for two\nweeks in the middle of a rebase (it seems that in the new\nimplementation, temporary refs are created basically to do the same\nthing as the log file I described.) I could easily be typing up my\ncommit message, then switch to something else, and happen not to come\nback to it for two weeks.\n\nBecause such a \"timeout\" based solution isn't really the \"correct\nsolution\" but will work most of the time, potential problems won't be\nnoticed while testing.\n\nAnother significant issue is that this timeout means that unreferenced\njunk has to stay around in the repository for two weeks for no (good)\nreason.\n\n> If we are dealing with a pack file, those are protected by .keep\n> \"lock files\" between the time they are created on disk and the\n> time that the git-fetch or git-receive-pack process has finished\n> updating the refs to anchor the pack's contents as reachable.\n> Every once in a while a stale .keep file gets left behind when a\n> process gets killed by the OS, and its damn annoying to clean up.\n\n> I'd hate to clean up logs from every little git-add or git-commit\n> that aborted in the middle uncleanly.\n\nFirst of all, merely exiting due to an error should not cause log files\nto be left around. The only thing that should cause log files to be\nleft around is kill -9 or a system crash. Second, by storing the\nprocess id and a timestamp of when the log file was created, it is\npossible to reliably determine if a log file is stale.\n\n-- \nJeremy Maitin-Shepard\n"},{"id":"76484","messageId":"7vskwr9coz.fsf@gitster.siamese.dyndns.org","threadId":"13440","inReplyTo":"20080510002014.GH29038@spearce.org","subject":"Re: git gc & deleted branches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-10T01:21:00Z","receivedAt":"2008-05-10T01:21:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Jeremy Maitin-Shepard wrote:\n>> It is extremely cumbersome to have to worry about whether there are\n>> other concurrent accesses to the repository when running e.g. git gc.\n>> For servers, you may never be able to guarantee that nothing else is\n>> accessing the repository concurrently. Here is a possible solution:\n>> \n>> Each git process creates a log file of the references that it has\n>> created. The log file should be named in some way with e.g. the process\n>> id and start time of the process, and simply consist of a list of\n>> 20-byte sha1 hashes to be considered additional in-use references for\n>> the purpose of garbage collection.\n\nHow would that solve the issue that you should not prune/gc the repository\n\"clone --shared\" aka \"alternates\" borrows from?\n\nBy the way, I do not think your \"git-commit stopped for two weeks due to a\nlong editing session of the commit message\" should result in any object\nlossage, as the new objects are all reachable from the index, and the new\ntree nor the new commit hasn't been built while you are typing (rather,\nnot typing) the log message.\n\nHmm, a partial commit that uses a temporary index file may lose, come to\nthink of it. Perhaps we should teach reachable.c about the temporary\nindex file as well. I dunno.\n \n"},{"id":"76485","messageId":"87y76jx6y4.fsf@jeremyms.com","threadId":"13440","inReplyTo":"7vskwr9coz.fsf@gitster.siamese.dyndns.org","subject":"Re: git gc & deleted branches","fromName":"Jeremy Maitin-Shepard","fromEmail":"jbms@cmu.edu","sentAt":"2008-05-10T01:51:15Z","receivedAt":"2008-05-10T01:51:15Z","isPatch":false,"sender":{"key":"jbms@cmu.edu","avatar":null},"body":"Junio C Hamano writes:\n\n> \"Shawn O. Pearce\" writes:\n>> Jeremy Maitin-Shepard wrote:\n>>> It is extremely cumbersome to have to worry about whether there are\n>>> other concurrent accesses to the repository when running e.g. git gc.\n>>> For servers, you may never be able to guarantee that nothing else is\n>>> accessing the repository concurrently. Here is a possible solution:\n>>> \n>>> Each git process creates a log file of the references that it has\n>>> created. The log file should be named in some way with e.g. the process\n>>> id and start time of the process, and simply consist of a list of\n>>> 20-byte sha1 hashes to be considered additional in-use references for\n>>> the purpose of garbage collection.\n\n> How would that solve the issue that you should not prune/gc the repository\n> \"clone --shared\" aka \"alternates\" borrows from?\n\nThe log files are only for handling in-progress commands editing the\nrepository. I also describe in first part of the e-mail a possible\nsolution to that issue as well as the issues created by having multiple\nworking directories:\n\nWhen you create a new working directory, you would also create in the\noriginal repository a symlink named\ne.g. orig_repo/.git/peers/ that\npoints to the .git directory of the newly created working directory.\ngit clone -shared would likewise create such a link in the original\nrepository. There could be a separate simple command to \"destroy\" a\nrepository created via clone -shared or via new-work-dir that would\nsimply remove this \"peer\" symlink from any repositories it shares from,\nand then rm -rf the target repository. The list of repositories that a\ngiven target repository shares from would be discovered using perhaps\nseveral different methods, depending on whether it is a new work dir, an\nactual separate repository, or the new type of \"shared\" repository I\nsuggested in my original e-mail, namely one that has its own refs but\ncompletely shares the object store of the original repository, e.g. via\na symlink to the original repository's objects directory In any case, I\nbelieve the information to go \"upstream\" is already available, and we\njust need to add those \"peer\" symlinks in order to be able to go\n\"downstream\".\n\nThere could also be a simple git command to move a repository that would\ntake care of updating all of the references that other repositories have\nto it. Currently it is not possible to write such a command, because\nthe \"downstream\" links are not stored, but with these added symlinks it\nwould be possible.\n\nAs I said in my previous e-mail, if git gc finds any broken symlinks\n(i.e. symlinks that point to invalid repositories), it would error out,\nbecause user attention is required to specify whether the symlinks\ncorrespond to deleted repositories, or to repositories that have been\nmoved without making the proper updates.\n\n> By the way, I do not think your \"git-commit stopped for two weeks due to a\n> long editing session of the commit message\" should result in any object\n> lossage, as the new objects are all reachable from the index, and the new\n> tree nor the new commit hasn't been built while you are typing (rather,\n> not typing) the log message.\n\n> Hmm, a partial commit that uses a temporary index file may lose, come to\n> think of it. Perhaps we should teach reachable.c about the temporary\n> index file as well. I dunno.\n\nWell, providing a generic mechanism for telling git about reachable\nthings other than the index and refs is precisely what these log files\nwould do, and also because they would record the process id and a\ntimestamp, stale log files would automatically get cleaned up. If each\nindividual git command has its own special way of trying to keep track\nof temporary references, it is just going to be more complicated and\nmore error prone.\n\n-- \nJeremy Maitin-Shepard\n"},{"id":"76487","messageId":"ee63ef30805092032h18fe7ff7mda4233b08ae431ea@mail.gmail.com","threadId":"13440","inReplyTo":"7v1w4bb291.fsf@gitster.siamese.dyndns.org","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-10T03:32:44Z","receivedAt":"2008-05-10T03:32:44Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"I didn't read your email until now. It may sound strange, but I can't\nread my personal email from work.\n\nAlso, looks like I inadvertantly took this message off-list.\n\nOn Fri, May 9, 2008 at 4:23 PM, Junio C Hamano wrote:\n> \"Brandon Casey\" writes:\n>\n>> Is this invocation of pack-objects that you commented on necessary\n>> though? This is the section that is replacing an existing pack file\n>> with a newly created pack file of the same name...\n>\n> Yuck, I failed to see it in the context. You are right. This won't do\n> anything I suspect.\n\nNow I'm confused. I hope you realized I said \"pack-objects\" when I\nmeant \"unpack-objects\".\n\nIt was only the first invocation of unpack-objects that I thought may\nbe unnecessary. This is because pack-objects (yes, pack) created a new\npack with\nthe same name as an existing pack which means (if I'm thinking about things\ncorrectly) that none of the objects inside the pack are unreachable. So trying\nto unpack-objects is a waste of time.\n\nThere is a second invocation of unpack-objects which is run on packs which\nwill be deleted. This one should unpack any unreferenced objects from each pack.\n\n-brandon\n\n\nThe remainder of your email is below since the list has never seen it.\n\n\n> Perhaps we would want a new option --eject that causes pack-object write\n> out unreachable packed objects in the loose format? This would require a\n> minor surgery to write_sha1_file() so that it won't pay attention to the\n> return value of has_sha1_file(sha1), perhaps like this.\n>\n> This is a lunch-time hack and needs more serious work, such as fixing\n> horrible \"write_sha1_file_1\" to something more sensible.\n>\n> Also it should add --eject as an independent and incompatible option\n> instead of dropping --keep-unpacked support.\n>\n> ---\n>\n> builtin-pack-objects.c | 56 ++++++++++++-----------------------------------\n> sha1_file.c | 9 ++++++-\n> 2 files changed, 22 insertions(+), 43 deletions(-)\n>\n> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n> index 777f272..8240a4b 100644\n> --- a/builtin-pack-objects.c\n> +++ b/builtin-pack-objects.c\n> @@ -1836,38 +1836,25 @@ struct in_pack {\n> struct in_pack_object *array;\n> };\n>\n> -static void mark_in_pack_object(struct object *object, struct packed_git *p, struct in_pack *in_pack)\n> +static void eject_object(struct object *obj)\n> {\n> - in_pack->array[in_pack->nr].offset = find_pack_entry_one(object->sha1, p);\n> - in_pack->array[in_pack->nr].object = object;\n> - in_pack->nr++;\n> -}\n> -\n> -/*\n> - * Compare the objects in the offset order, in order to emulate the\n> - * \"git-rev-list --objects\" output that produced the pack originally.\n> - */\n> -static int ofscmp(const void *a_, const void *b_)\n> -{\n> - struct in_pack_object *a = (struct in_pack_object *)a_;\n> - struct in_pack_object *b = (struct in_pack_object *)b_;\n> + enum object_type type;\n> + unsigned long size;\n> + void *data;\n> + unsigned char sha1[20];\n>\n> - if (a->offset < b->offset)\n> - return -1;\n> - else if (a->offset > b->offset)\n> - return 1;\n> - else\n> - return hashcmp(a->object->sha1, b->object->sha1);\n> + data = read_sha1_file(obj->sha1, &type, &size);\n> + assert(data && type == obj->type);\n> + if (write_sha1_file_1(data, size, typename(type), sha1, 1))\n> + die(\"cannot eject packed object %s\", sha1_to_hex(obj->sha1));\n> + assert(!memcmp(sha1, obj->sha1));\n> }\n>\n> -static void add_objects_in_unpacked_packs(struct rev_info *revs)\n> +static void eject_objects_in_unpacked_packs(struct rev_info *revs)\n> {\n> struct packed_git *p;\n> - struct in_pack in_pack;\n> uint32_t i;\n>\n> - memset(&in_pack, 0, sizeof(in_pack));\n> -\n> for (p = packed_git; p; p = p->next) {\n> const unsigned char *sha1;\n> struct object *o;\n> @@ -1881,28 +1868,15 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n> if (open_pack_index(p))\n> die(\"cannot open pack index\");\n>\n> - ALLOC_GROW(in_pack.array,\n> - in_pack.nr + p->num_objects,\n> - in_pack.alloc);\n> -\n> for (i = 0; i < p->num_objects; i++) {\n> sha1 = nth_packed_object_sha1(p, i);\n> o = lookup_unknown_object(sha1);\n> - if (!(o->flags & OBJECT_ADDED))\n> - mark_in_pack_object(o, p, &in_pack);\n> + if (o->flags & OBJECT_ADDED)\n> + continue;\n> o->flags |= OBJECT_ADDED;\n> + eject_object(o);\n> }\n> }\n> -\n> - if (in_pack.nr) {\n> - qsort(in_pack.array, in_pack.nr, sizeof(in_pack.array[0]),\n> - ofscmp);\n> - for (i = 0; i < in_pack.nr; i++) {\n> - struct object *o = in_pack.array[i].object;\n> - add_object_entry(o->sha1, o->type, \"\", 0);\n> - }\n> - }\n> - free(in_pack.array);\n> }\n>\n> static void get_object_list(int ac, const char **av)\n> @@ -1938,7 +1912,7 @@ static void get_object_list(int ac, const char **av)\n> traverse_commit_list(&revs, show_commit, show_object);\n>\n> if (keep_unreachable)\n> - add_objects_in_unpacked_packs(&revs);\n> + eject_objects_in_unpacked_packs(&revs);\n> }\n>\n> static int adjust_perm(const char *path, mode_t mode)\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 3516777..ce3a661 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -2102,7 +2102,7 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n> return 0;\n> }\n>\n> -int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n> +int write_sha1_file_1(void *buf, unsigned long len, const char *type, unsigned char *returnsha1, int refresh)\n> {\n> int size, ret;\n> unsigned char *compressed;\n> @@ -2120,7 +2120,7 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n> filename = sha1_file_name(sha1);\n> if (returnsha1)\n> hashcpy(returnsha1, sha1);\n> - if (has_sha1_file(sha1))\n> + if (!refresh && has_sha1_file(sha1))\n> return 0;\n> fd = open(filename, O_RDONLY);\n> if (fd >= 0) {\n> @@ -2185,6 +2185,11 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n> return move_temp_to_file(tmpfile, filename);\n> }\n>\n> +int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n> +{\n> + return write_sha1_file_1(buf, len, type, returnsha1, 0);\n> +}\n> +\n> /*\n> * We need to unpack and recompress the object for writing\n> * it out to a different file.\n>\n"},{"id":"76491","messageId":"5119086.1210392044037.JavaMail.teamon@b303.teamon.com","threadId":"13440","inReplyTo":"7vabj0b1re.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 0/3] leave unreferenced objects unpacked","fromName":"","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-10T04:01:54Z","receivedAt":"2008-05-10T04:01:54Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Here is a formal patch. I removed the first invocation of\nunpack-objects since I think it is unneccessary.\n\nFollowed up with mods to git-gc.\n\n-brandon\n"},{"id":"76490","messageId":"3927888.1210392047922.JavaMail.teamon@b303.teamon.com","threadId":"13440","inReplyTo":"7vabj0b1re.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 1/3] repack: modify behavior of -A option to leave unreferenced objects unpacked","fromName":"","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-10T04:01:55Z","receivedAt":"2008-05-10T04:01:55Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"From: Brandon Casey \n\nThe previous behavior of the -A option was to retain any previously\npacked objects which had become unreferenced, and place them into the newly\ncreated pack file. Since git-gc, when run automatically with the --auto\noption, calls repack with the -A option, this had the effect of retaining\nunreferenced packed objects indefinitely. To avoid this scenario, the\nuser was required to run git-gc with the little known --prune option or\nto manually run repack with the -a option.\n\nThis patch changes the behavior of the -A option so that unreferenced\nobjects that exist in any pack file being replaced, will be unpacked into\nthe repository. The unreferenced loose objects can then be garbage collected\nby git-gc (i.e. git-prune) based on the gc.pruneExpire setting.\n\nAlso add new tests for checking whether unreferenced objects which were\npreviously packed are properly left in the repository unpacked after\nrepacking.\n\nSigned-off-by: Brandon Casey \n---\n git-repack.sh | 18 +++++++++---\n t/t7701-repack-unpack-unreachable.sh | 47 ++++++++++++++++++++++++++++++++++\n 2 files changed, 60 insertions(+), 5 deletions(-)\n create mode 100755 t/t7701-repack-unpack-unreachable.sh\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex e18eb3f..a0e06ed 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -30,7 +30,7 @@ do\n \t-n)\tno_update_info=t ;;\n \t-a)\tall_into_one=t ;;\n \t-A)\tall_into_one=t\n-\t\tkeep_unreachable=--keep-unreachable ;;\n+\t\tkeep_unreachable=t ;;\n \t-d)\tremove_redundant=t ;;\n \t-q)\tquiet=-q ;;\n \t-f)\tno_reuse=--no-reuse-object ;;\n@@ -78,9 +78,6 @@ case \",$all_into_one,\" in\n \tif test -z \"$args\"\n \tthen\n \t\targs='--unpacked --incremental'\n-\telif test -n \"$keep_unreachable\"\n-\tthen\n-\t\targs=\"$args $keep_unreachable\"\n \tfi\n \t;;\n esac\n@@ -130,7 +127,18 @@ then\n \t\t do\n \t\t\tcase \" $fullbases \" in\n \t\t\t*\" $e \"*) ;;\n-\t\t\t*)\trm -f \"$e.pack\" \"$e.idx\" \"$e.keep\" ;;\n+\t\t\t*)\n+\t\t\t\trm -f \"$e.idx\" \"$e.keep\"\n+\t\t\t\tif test -n \"$keep_unreachable\" &&\n+\t\t\t\t test -f \"$e.pack\"\n+\t\t\t\tthen\n+\t\t\t\t\tgit unpack-objects < \"$e.pack\" || {\n+\t\t\t\t\t\techo >&2 \"Failed unpacking unreachable objects from redundant pack file $e.pack\"\n+\t\t\t\t\t\texit 1\n+\t\t\t\t\t}\n+\t\t\t\tfi\n+\t\t\t\trm -f \"$e.pack\"\n+\t\t\t;;\n \t\t\tesac\n \t\t done\n \t\t)\ndiff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh\nnew file mode 100755\nindex 0000000..6a5211f\n--- /dev/null\n+++ b/t/t7701-repack-unpack-unreachable.sh\n@@ -0,0 +1,47 @@\n+#!/bin/sh\n+\n+test_description='git-repack works correctly'\n+\n+. ./test-lib.sh\n+\n+test_expect_success '-A option leaves unreachable objects unpacked' '\n+\techo content > file1 &&\n+\tgit add . &&\n+\tgit commit -m initial_commit &&\n+\t# create a transient branch with unique content\n+\tgit checkout -b transient_branch &&\n+\techo more content >> file1 &&\n+\t# record the objects created in the database for file, commit, tree\n+\tfsha1=$(git hash-object file1) &&\n+\tgit commit -a -m more_content &&\n+\tcsha1=$(git rev-parse HEAD^{commit}) &&\n+\ttsha1=$(git rev-parse HEAD^{tree}) &&\n+\tgit checkout master &&\n+\techo even more content >> file1 &&\n+\tgit commit -a -m even_more_content &&\n+\t# delete the transient branch\n+\tgit branch -D transient_branch &&\n+\t# pack the repo\n+\tgit repack -A -d -l &&\n+\t# verify objects are packed in repository\n+\ttest 3 = $(git verify-pack -v -- .git/objects/pack/*.idx |\n+\t\t grep -e \"^$fsha1 \" -e \"^$csha1 \" -e \"^$tsha1 \" |\n+\t\t sort | uniq | wc -l) &&\n+\tgit show $fsha1 &&\n+\tgit show $csha1 &&\n+\tgit show $tsha1 &&\n+\t# now expire the reflog\n+\tsleep 1 &&\n+\tgit reflog expire --expire-unreachable=now --all &&\n+\t# and repack\n+\tgit repack -A -d -l &&\n+\t# verify objects are retained unpacked\n+\ttest 0 = $(git verify-pack -v -- .git/objects/pack/*.idx |\n+\t\t grep -e \"^$fsha1 \" -e \"^$csha1 \" -e \"^$tsha1 \" |\n+\t\t sort | uniq | wc -l) &&\n+\tgit show $fsha1 &&\n+\tgit show $csha1 &&\n+\tgit show $tsha1\n+'\n+\n+test_done\n-- \n1.5.5.67.g9a49\n"},{"id":"76489","messageId":"11461522.1210392051144.JavaMail.teamon@b303.teamon.com","threadId":"13440","inReplyTo":"7vabj0b1re.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 2/3] git-gc: always use -A when manually repacking","fromName":"","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-10T04:01:56Z","receivedAt":"2008-05-10T04:01:56Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"From: Brandon Casey \n\nNow that repack -A will leave unreferenced objects unpacked, there is\nno reason to use the -a option to repack (which will discard unreferenced\nobjects). The unpacked unreferenced objects will not be repacked by a\nsubsequent repack, and will eventually be pruned by git-gc based on the\ngc.pruneExpire config option.\n---\n builtin-gc.c | 13 ++-----------\n 1 files changed, 2 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex f99ebc7..6db2f51 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -256,17 +256,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\"performance. You may also\\n\"\n \t\t\t\"run \\\"git gc\\\" manually. See \"\n \t\t\t\"\\\"git help gc\\\" for more information.\\n\");\n-\t} else {\n-\t\t/*\n-\t\t * Use safer (for shared repos) \"-A\" option to\n-\t\t * repack when not pruning. Auto-gc makes its\n-\t\t * own decision.\n-\t\t */\n-\t\tif (prune)\n-\t\t\tappend_option(argv_repack, \"-a\", MAX_ADD);\n-\t\telse\n-\t\t\tappend_option(argv_repack, \"-A\", MAX_ADD);\n-\t}\n+\t} else\n+\t\tappend_option(argv_repack, \"-A\", MAX_ADD);\n \n \tif (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))\n \t\treturn error(FAILED_RUN, argv_pack_refs[0]);\n-- \n1.5.5.67.g9a49\n"},{"id":"76488","messageId":"14594477.1210392054360.JavaMail.teamon@b303.teamon.com","threadId":"13440","inReplyTo":"7vabj0b1re.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 3/3] builtin-gc.c: deprecate --prune, it now really has no effect","fromName":"","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-10T04:01:57Z","receivedAt":"2008-05-10T04:01:57Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"From: Brandon Casey \n\n---\n builtin-gc.c | 3 +--\n 1 files changed, 1 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 6db2f51..48f7d95 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -219,7 +219,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tchar buf[80];\n \n \tstruct option builtin_gc_options[] = {\n-\t\tOPT_BOOLEAN(0, \"prune\", &prune, \"prune unreferenced objects\"),\n+\t\tOPT_BOOLEAN(0, \"prune\", &prune, \"prune unreferenced objects (deprecated)\"),\n \t\tOPT_BOOLEAN(0, \"aggressive\", &aggressive, \"be more thorough (increased runtime)\"),\n \t\tOPT_BOOLEAN(0, \"auto\", &auto_gc, \"enable auto-gc mode\"),\n \t\tOPT_BOOLEAN('q', \"quiet\", &quiet, \"suppress progress reports\"),\n@@ -249,7 +249,6 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tprune = 0;\n \t\tif (!need_to_gc())\n \t\t\treturn 0;\n \t\tfprintf(stderr, \"Auto packing your repository for optimum \"\n-- \n1.5.5.67.g9a49\n"},{"id":"76492","messageId":"ee63ef30805092115i7f433f65p4b8ed3f2721fb456@mail.gmail.com","threadId":"13440","inReplyTo":"ee63ef30805092032h18fe7ff7mda4233b08ae431ea@mail.gmail.com","subject":"Re: git gc & deleted branches","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-10T04:15:11Z","receivedAt":"2008-05-10T04:15:11Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"On Fri, May 9, 2008 at 10:32 PM, Brandon Casey wrote:\n> On Fri, May 9, 2008 at 4:23 PM, Junio C Hamano wrote:\n> The remainder of your email is below since the list has never seen it.\n>\n>\n>> Perhaps we would want a new option --eject that causes pack-object write\n>> out unreachable packed objects in the loose format?\n\nTwo comments on this.\n\n1) If you decide to go in this direction, I think the option should be\n named --unpack-unreachable rather than --eject.\n2) There is some part of me that is against a program named\n pack-objects doing unpacking. But it is probably more efficient\n this way.\n\n-brandon\n"},{"id":"76493","messageId":"20080510052548.GA11556@sigill.intra.peff.net","threadId":"13440","inReplyTo":"87y76jx6y4.fsf@jeremyms.com","subject":"Re: git gc & deleted branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-10T05:25:49Z","receivedAt":"2008-05-10T05:25:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 09, 2008 at 09:51:15PM -0400, Jeremy Maitin-Shepard wrote:\n\n> When you create a new working directory, you would also create in the\n> original repository a symlink named\n> e.g. orig_repo/.git/peers/ that\n> points to the .git directory of the newly created working directory.\n\nThat assumes you _can_ write to the original repository. That may or may\nnot be the case, depending on your setup.\n\n-Peff\n"},{"id":"76494","messageId":"87tzh6yb2w.fsf@jeremyms.com","threadId":"13440","inReplyTo":"20080510052548.GA11556@sigill.intra.peff.net","subject":"Re: git gc & deleted branches","fromName":"Jeremy Maitin-Shepard","fromEmail":"jbms@cmu.edu","sentAt":"2008-05-10T05:36:39Z","receivedAt":"2008-05-10T05:36:39Z","isPatch":false,"sender":{"key":"jbms@cmu.edu","avatar":null},"body":"Jeff King writes:\n\n> On Fri, May 09, 2008 at 09:51:15PM -0400, Jeremy Maitin-Shepard wrote:\n>> When you create a new working directory, you would also create in the\n>> original repository a symlink named\n>> e.g. orig_repo/.git/peers/ that\n>> points to the .git directory of the newly created working directory.\n\n> That assumes you _can_ write to the original repository. That may or may\n> not be the case, depending on your setup.\n\nWell, I suppose in that case it could print a warning or maybe fail\nwithout some \"force\" option. If you can't write to the repository, then\nI think it is safe to say that it will never know or care about you, so\nyou will fundamentally have a fragile setup. I'd say that except in\nvery special circumstances, you are better off just not sharing it at\nall.\n\nConsider, for instance, that even if the repository that you are sharing\nform never deletes branches and never does non-fast-forward updates of\nreferences, it could very well happen to have, due to some temporary\noperation, some unreferenced object that happens to be exactly the same\nobject that you want to add to your repository. Because you've listed\nit in info/alternates, you won't write that object to your own\nrepository, but then the source repository will very likely garbage\ncollect the object at some later point, corrupting your repository.\n\n-- \nJeremy Maitin-Shepard\n"},{"id":"76496","messageId":"20080510060345.GC11556@sigill.intra.peff.net","threadId":"13440","inReplyTo":"3927888.1210392047922.JavaMail.teamon@b303.teamon.com","subject":"Re: [PATCH 1/3] repack: modify behavior of -A option to leave unreferenced objects unpacked","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-10T06:03:45Z","receivedAt":"2008-05-10T06:03:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 09, 2008 at 11:01:55PM -0500, drafnel@gmail.com wrote:\n\n> -\t\tkeep_unreachable=--keep-unreachable ;;\n> +\t\tkeep_unreachable=t ;;\n\nCan we call this something else (like unpack_unreachable) since it now\nhas nothing to do with the --keep-unreachable flag?\n\nAlso, should --keep-unreachable be deprecated / removed?\n\n> +\t\t\t*)\n> +\t\t\t\trm -f \"$e.idx\" \"$e.keep\"\n> +\t\t\t\tif test -n \"$keep_unreachable\" &&\n> +\t\t\t\t test -f \"$e.pack\"\n> +\t\t\t\tthen\n> +\t\t\t\t\tgit unpack-objects < \"$e.pack\" || {\n> +\t\t\t\t\t\techo >&2 \"Failed unpacking unreachable objects from redundant pack file $e.pack\"\n> +\t\t\t\t\t\texit 1\n> +\t\t\t\t\t}\n> +\t\t\t\tfi\n\nI still like Geert's suggestion of unpacking them to a _different_\nplace. That helps to avoid spurious \"gc --auto\" invocations caused by\ntoo many prunable objects. Though it certainly doesn't solve it, and\nmaybe that just needs to be fixed separately.\n\nPossibly the \"gc --auto\" test should be:\n\n - count objects; if too few, exit\n - count unreachable loose objects; if too few, exit\n - run gc\n\nThat means having a lot of unreachable objects will still incur some\nextra processing, but not as much as a full repack. And it won't bug the\nuser with a \"you need to repack\" message.\n\n-Peff\n"},{"id":"76501","messageId":"alpine.DEB.1.00.0805101003350.30431@racer","threadId":"13440","inReplyTo":"87tzh6yb2w.fsf@jeremyms.com","subject":"Re: git gc & deleted branches","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-05-10T09:04:27Z","receivedAt":"2008-05-10T09:04:27Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 10 May 2008, Jeremy Maitin-Shepard wrote:\n\n> Jeff King writes:\n> \n> > On Fri, May 09, 2008 at 09:51:15PM -0400, Jeremy Maitin-Shepard wrote:\n\n> >> When you create a new working directory, you would also create in the\n> >> original repository a symlink named\n> >> e.g. orig_repo/.git/peers/ that\n> >> points to the .git directory of the newly created working directory.\n> \n> > That assumes you _can_ write to the original repository. That may or may\n> > not be the case, depending on your setup.\n\nFWIW this argument can be found in the mailing list. It does not have to \nbe told over and over again, right?\n\n> Well, I suppose in that case it could print a warning or maybe fail \n> without some \"force\" option. If you can't write to the repository, then \n> I think it is safe to say that it will never know or care about you, so \n> you will fundamentally have a fragile setup. I'd say that except in \n> very special circumstances, you are better off just not sharing it at \n> all.\n\nCounterexample kernel.org. Counterexample repo.or.cz.\n\nHth,\nDscho\n"},{"id":"76531","messageId":"87prruxh3z.fsf@jeremyms.com","threadId":"13440","inReplyTo":"alpine.DEB.1.00.0805101003350.30431@racer","subject":"Re: git gc & deleted branches","fromName":"Jeremy Maitin-Shepard","fromEmail":"jbms@cmu.edu","sentAt":"2008-05-10T16:24:00Z","receivedAt":"2008-05-10T16:24:00Z","isPatch":false,"sender":{"key":"jbms@cmu.edu","avatar":null},"body":"Johannes Schindelin writes:\n\n> Hi,\n> On Sat, 10 May 2008, Jeremy Maitin-Shepard wrote:\n\n>> Jeff King writes:\n>> \n>> > On Fri, May 09, 2008 at 09:51:15PM -0400, Jeremy Maitin-Shepard wrote:\n\n>> >> When you create a new working directory, you would also create in the\n>> >> original repository a symlink named\n>> >> e.g. orig_repo/.git/peers/ that\n>> >> points to the .git directory of the newly created working directory.\n>> \n>> > That assumes you _can_ write to the original repository. That may or may\n>> > not be the case, depending on your setup.\n\n> FWIW this argument can be found in the mailing list. It does not have to \n> be told over and over again, right?\n\nMaybe you can point me at the relevant thread. Fundamentally, though,\nI'd say objects/info/alternates _cannot_ work reliably without the\nsource repository knowing about the objects that the sharing\nrepositories need. Otherwise, there is no way for it to know not to\nprune them. The only way for it to have that information in general is\nto write it in the repository. In a site-specific setting, it may\nindeed be possible to rely on some site-specific database, but that is\nnot particularly relevant.\n\nCurrently repository sharing seems to be used in many cases in quite\nunsafe ways. It may seem unfortunate that doing things the \"safe way\"\nis much more of a hassle and doesn't work in certain environments, but\nI'd say that is just the way things have to be.\n\nPerhaps you can point me to an existing thread that addresses this idea,\nthough.\n\n>> Well, I suppose in that case it could print a warning or maybe fail \n>> without some \"force\" option. If you can't write to the repository, then \n>> I think it is safe to say that it will never know or care about you, so \n>> you will fundamentally have a fragile setup. I'd say that except in \n>> very special circumstances, you are better off just not sharing it at \n>> all.\n\n> Counterexample kernel.org. Counterexample repo.or.cz.\n\nrepo.or.cz is not a counterexample. It is completely \"managed\", and\ncould quite easily implement the approach I described. I don't know\nexactly how kernel.org works, but I imagine likewise some setuid helper\nscript could be provided to write these symlinks.\n\nThere is the issue that these setuid helper scripts would mean at the\nvery least that if user A can \"fork\" user B's repository, then to some\nextent user B can make user A use large amounts of disk space\n(i.e. exceed his quota or something) by just referencing a bunch of\ntemporary objects that user A happens to have in his repository, and it\nwould take careful examination of the git repository to actually figure\nout that it is user B's fault. I don't think this would be a\nsignificant problem in practice, though.\n\n-- \nJeremy Maitin-Shepard\n"},{"id":"76565","messageId":"alpine.LFD.1.10.0805101157090.23581@xanadu.home","threadId":"13440","inReplyTo":"20080510060345.GC11556@sigill.intra.peff.net","subject":"Re: [PATCH 1/3] repack: modify behavior of -A option to leave unreferenced objects unpacked","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-11T01:10:53Z","receivedAt":"2008-05-11T01:10:53Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 10 May 2008, Jeff King wrote:\n\n> Also, should --keep-unreachable be deprecated / removed?\n\nDepends. If it has no maintenance cost then we might as well keep it \naround.\n\n> I still like Geert's suggestion of unpacking them to a _different_\n> place. That helps to avoid spurious \"gc --auto\" invocations caused by\n> too many prunable objects. Though it certainly doesn't solve it, and\n> maybe that just needs to be fixed separately.\n\nHaving a separate location for objects seems clunky to me.\n\nAnd the fundamental problem isn't solved indeed -- you may end up with \nmany non expired unreachable loose objects already without packing them.\n\n> Possibly the \"gc --auto\" test should be:\n> \n> - count objects; if too few, exit\n> - count unreachable loose objects; if too few, exit\n\nDetermining the number of unreachable objects is quite costly, packed or \nnot. So that isn't a good thing to do on every 'git gc --auto' \ninvokation.\n\n> - run gc\n> \n> That means having a lot of unreachable objects will still incur some\n> extra processing, but not as much as a full repack. And it won't bug the\n> user with a \"you need to repack\" message.\n\nThe auto gc performs incremental packing most of the time. And that is \nway faster than figuring out which objects are unreachable.\n\nFor example, running 'git prune' in my Linux repo takes 16 seconds, even \nwhen there is nothing to prune. Running 'git repack' (with no option so \nto perform an incremental repack) took less than 2 seconds to pack 541 \nreachable objects that happened to be loose.\n\nI'm now starting to wonder if there is a reason for keeping unreachable \nobjects that used to be packed. Putting --keep-unreachable aside for \nnow, the only way an unreachable object could have entered a pack is if \nit used to be reachable before through the commit history or reflog. \nSo if they're not reachable anymore, that's most probably because their \nreflog expired. So what's the point for keeping them even longer? \nWhat's the reasoning that led to the creation of --keep-unreachable in \nthe first place?\n\n\nNicolas\n"},{"id":"76566","messageId":"7v63tlab2e.fsf@gitster.siamese.dyndns.org","threadId":"13440","inReplyTo":"alpine.LFD.1.10.0805101157090.23581@xanadu.home","subject":"Re: [PATCH 1/3] repack: modify behavior of -A option to leave unreferenced objects unpacked","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-11T01:23:05Z","receivedAt":"2008-05-11T01:23:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> I'm now starting to wonder if there is a reason for keeping unreachable \n> objects that used to be packed. Putting --keep-unreachable aside for \n> now, the only way an unreachable object could have entered a pack is if \n> it used to be reachable before through the commit history or reflog. \n> So if they're not reachable anymore, that's most probably because their \n> reflog expired. So what's the point for keeping them even longer? \n> What's the reasoning that led to the creation of --keep-unreachable in \n> the first place?\n\nI think the logic went like this.\n\n(1) You may have rewound your head since you last repacked; blobs and\n trees in the rewound commit are already packed now.\n\n(2) Now you may be fetching (or somebody else may be pushing) a commit\n that contains such blobs and/or trees, and the fetch or push is small\n enough that it unpacks, but the packed and unreachable ones are not\n unpacked.\n\n(3) But before that fetch or push finishes to update the ref, you can race\n with a \"repack -a -d\".\n"},{"id":"76573","messageId":"ee63ef30805102116m68e83fadr8ef9afb080d26cf0@mail.gmail.com","threadId":"13440","inReplyTo":"20080510060345.GC11556@sigill.intra.peff.net","subject":"Re: [PATCH 1/3] repack: modify behavior of -A option to leave unreferenced objects unpacked","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-11T04:16:44Z","receivedAt":"2008-05-11T04:16:44Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"On Sat, May 10, 2008 at 1:03 AM, Jeff King wrote:\n> On Fri, May 09, 2008 at 11:01:55PM -0500, drafnel@gmail.com wrote:\n>\n>> - keep_unreachable=--keep-unreachable ;;\n>> + keep_unreachable=t ;;\n>\n> Can we call this something else (like unpack_unreachable) since it now\n> has nothing to do with the --keep-unreachable flag?\n\nActually I initially changed it to unpack_unreachable, and then\nchanged it back. The reason I did this is because I think\nkeep_unreachable still describes what is being accomplished, that\nunreachables are being kept. When -A is supplied along with -d,\nunreachables are kept by being unpacked. When -d is not supplied,\nunreachables are kept in their original pack file. If Geert's proposal\nor something else is implemented, keep_unreachable may still be\nappropriate. hmm?\n\n> Also, should --keep-unreachable be deprecated / removed?\n>\n>> + *)\n>> + rm -f \"$e.idx\" \"$e.keep\"\n>> + if test -n \"$keep_unreachable\" &&\n>> + test -f \"$e.pack\"\n>> + then\n>> + git unpack-objects < \"$e.pack\" || {\n>> + echo >&2 \"Failed unpacking unreachable objects from redundant pack file $e.pack\"\n>> + exit 1\n>> + }\n>> + fi\n>\n> I still like Geert's suggestion of unpacking them to a _different_\n> place. That helps to avoid spurious \"gc --auto\" invocations caused by\n> too many prunable objects. Though it certainly doesn't solve it, and\n> maybe that just needs to be fixed separately.\n\nThat was my thinking.\n\n>\n> Possibly the \"gc --auto\" test should be:\n>\n> - count objects; if too few, exit\n> - count unreachable loose objects; if too few, exit\n> - run gc\n>\n> That means having a lot of unreachable objects will still incur some\n> extra processing, but not as much as a full repack. And it won't bug the\n> user with a \"you need to repack\" message.\n\nI've got a thought. How about limiting how often auto repack repacks\nby looking at the timestamp of the most recent pack? Wouldn't the\npacks already be prepared in most cases i.e. prepare_packed_git()\n\n-brandon\n"},{"id":"76574","messageId":"ee63ef30805102151r54121e46n4aac3f951077b4fd@mail.gmail.com","threadId":"13440","inReplyTo":"ee63ef30805102116m68e83fadr8ef9afb080d26cf0@mail.gmail.com","subject":"Re: [PATCH 1/3] repack: modify behavior of -A option to leave unreferenced objects unpacked","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2008-05-11T04:51:41Z","receivedAt":"2008-05-11T04:51:41Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"On Sat, May 10, 2008 at 11:16 PM, Brandon Casey wrote:\n\n> I've got a thought. How about limiting how often auto repack repacks\n> by looking at the timestamp of the most recent pack? Wouldn't the\n> packs already be prepared in most cases i.e. prepare_packed_git()\n\ncompletely untested and hopefully not mangled by google...\n\nactually, this will do nothing for the case where there exists many\nloose unreachable objects and no loose reachable objects since we\nwon't create a new pack with an updated timestamp to compare against.\nSo git-gc will continue to spin its wheels without getting anywhere.\nCould we update the pack timestamp after running git-gc or use a\ntimestamp from someplace else?\n\n-brandon\n\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 48f7d95..16b1455 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -27,6 +27,7 @@ static int aggressive_window = -1;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n static char *prune_expire = \"2.weeks.ago\";\n+static time_t gc_auto_pack_frequency = 21600; /* 6 hours */\n\n #define MAX_ADD 10\n static const char *argv_pack_refs[] = {\"pack-refs\", \"--all\", \"--prune\", NULL};\n@@ -56,6 +57,10 @@ static int gc_config(const char *var, const char *value)\n gc_auto_pack_limit = git_config_int(var, value);\n return 0;\n }\n+ if (!strcmp(var, \"gc.autopackfrequency\")) {\n+ gc_auto_pack_frequency = git_config_ulong(var, value);\n+ return 0;\n+ }\n if (!strcmp(var, \"gc.pruneexpire\")) {\n if (!value)\n return config_error_nonbool(var);\n@@ -205,6 +210,14 @@ static int need_to_gc(void)\n else if (!too_many_loose_objects())\n return 0;\n\n+ if (gc_auto_pack_frequency) {\n+ prepare_packed_git();\n+ if (packed_git &&\n+ packed_git->mtime >\n+ approxidate(\"now\") - gc_auto_pack_frequency)\n+ return 0;\n+ }\n+\n if (run_hook())\n return 0;\n return 1;\n"},{"id":"76599","messageId":"alpine.DEB.1.00.0805111204580.30431@racer","threadId":"13440","inReplyTo":"87prruxh3z.fsf@jeremyms.com","subject":"Re: git gc & deleted branches","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-05-11T11:11:02Z","receivedAt":"2008-05-11T11:11:02Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 10 May 2008, Jeremy Maitin-Shepard wrote:\n\n> Johannes Schindelin writes:\n> \n> > On Sat, 10 May 2008, Jeremy Maitin-Shepard wrote:\n> \n> >> Jeff King writes:\n> >> \n> >> > On Fri, May 09, 2008 at 09:51:15PM -0400, Jeremy Maitin-Shepard \n> >> > wrote:\n> \n> >> >> When you create a new working directory, you would also create in \n> >> >> the original repository a symlink named e.g. \n> >> >> orig_repo/.git/peers/ \n> >> >> that points to the .git directory of the newly created working \n> >> >> directory.\n> >> \n> >> > That assumes you _can_ write to the original repository. That may \n> >> > or may not be the case, depending on your setup.\n> \n> > FWIW this argument can be found in the mailing list. It does not have \n> > to be told over and over again, right?\n> \n> Maybe you can point me at the relevant thread. Fundamentally, though,\n> I'd say objects/info/alternates _cannot_ work reliably without the\n> source repository knowing about the objects that the sharing\n> repositories need. Otherwise, there is no way for it to know not to\n> prune them. The only way for it to have that information in general is\n> to write it in the repository. In a site-specific setting, it may\n> indeed be possible to rely on some site-specific database, but that is\n> not particularly relevant.\n> \n> Currently repository sharing seems to be used in many cases in quite\n> unsafe ways. It may seem unfortunate that doing things the \"safe way\"\n> is much more of a hassle and doesn't work in certain environments, but\n> I'd say that is just the way things have to be.\n> \n> Perhaps you can point me to an existing thread that addresses this idea, \n> though.\n\nUnfortunately, a quick search did not turn up anything useful. Maybe you \ntry your luck yourself...\n\n> >> Well, I suppose in that case it could print a warning or maybe fail \n> >> without some \"force\" option. If you can't write to the repository, \n> >> then I think it is safe to say that it will never know or care about \n> >> you, so you will fundamentally have a fragile setup. I'd say that \n> >> except in very special circumstances, you are better off just not \n> >> sharing it at all.\n> \n> > Counterexample kernel.org. Counterexample repo.or.cz.\n> \n> repo.or.cz is not a counterexample. It is completely \"managed\", and \n> could quite easily implement the approach I described.\n\nHalf true... you said \"if you can't write to the repository...\" and on \nrepo.or.cz, the first part is true, the second part not.\n\n> There is the issue that these setuid helper scripts would mean at the \n> very least that if user A can \"fork\" user B's repository, then to some \n> extent user B can make user A use large amounts of disk space (i.e. \n> exceed his quota or something) by just referencing a bunch of temporary \n> objects that user A happens to have in his repository, and it would take \n> careful examination of the git repository to actually figure out that it \n> is user B's fault. I don't think this would be a significant problem in \n> practice, though.\n\nWell, I think that the setuid helper script would open a whole bunch of \nother issues.\n\nI think that the shared repository problem is rather a semantic one, i.e. \nit is only solvable between the owners of the repository by good-ole \ntalking, not something that can be solved by the tool (Git).\n\nCiao,\nDscho\n"},{"id":"76652","messageId":"7vy76g3ctr.fsf@gitster.siamese.dyndns.org","threadId":"13440","inReplyTo":"alpine.DEB.1.00.0805111204580.30431@racer","subject":"Re: git gc & deleted branches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-11T18:39:12Z","receivedAt":"2008-05-11T18:39:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Well, I think that the setuid helper script would open a whole bunch of \n> other issues.\n>\n> I think that the shared repository problem is rather a semantic one, i.e. \n> it is only solvable between the owners of the repository by good-ole \n> talking, not something that can be solved by the tool (Git).\n\nI very strongly agree with you that the suid helper would be the last\nditch thing we would want to avoid doing unless there is no other way. I\nalso agree with you that the owners of the repository need to be talking.\n\nBut I think the tool _could_ help them do their talking. It is\nconceivable that just like you can explicitly allow selected others to\npush into your own repository, you would want to explicitly allow some\nothers to ask you to keep objects their repositories borrow from you.\n\n\tOriginally, I wrote \"allow others to borrow from you\", but that is\n\tvery ill defined. If they can read from your repository they can\n\tunilaterally borrow from you without having any write permission\n\tto your repository that is needed to install backpointers.\n\nI am not fundamentally opposed to a backpointer that point at borrowers so\nthat the lender can protect the objects that are pointed by them.\nHowever, there are two technical issues in the solution of pointing at the\nborrower's .git/refs with a symlink from the repository that is borrowed\nfrom, as I pointed out when this came up last time.\n\n - There are systems without symbolic links.\n\n - A symref (e.g. refs/remotes/origin/HEAD of borrower that points at\n refs/remotes/origin/master of borrower) is relative to borrower's\n repository.\n"}]}