{"thread":{"id":"41310","subject":"git object-count differs between clones","startedAt":"2016-02-02T15:52:51Z","lastAt":"2016-02-03T15:21:41Z","messageCount":7,"participants":["Andrew Martin","Matthieu Moy","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"277237","messageId":"2142875754.710575.1454428371555.JavaMail.zimbra@xes-inc.com","threadId":"41310","inReplyTo":"57434188.709288.1454428054374.JavaMail.zimbra@xes-inc.com","subject":"git object-count differs between clones","fromName":"Andrew Martin","fromEmail":"amartin@xes-inc.com","sentAt":"2016-02-02T15:52:51Z","receivedAt":"2016-02-02T15:52:51Z","isPatch":false,"sender":{"key":"amartin@xes-inc.com","avatar":null},"body":"Hello,\n\nI am using git 2.7.0 on Ubuntu 14.04. I recently tried pushing a large (90,000+\ncommits) repository to a gogs server (https://gogs.io/) and then cloning the\nrepository back to my machine. After running \"git count-objects -v\", I see a\ndiscrepancy in the number of objects:\n\nOriginal Repository:\ncount: 0\nsize: 0\nin-pack: 1258300\npacks: 1\nsize-pack: 593889\nprune-packable: 0\ngarbage: 0\nsize-garbage: 0\n\n\nClone from gogs:\ncount: 0\nsize: 0\nin-pack: 1258270\npacks: 1\nsize-pack: 593884\nprune-packable: 0\ngarbage: 0\nsize-garbage: 0\n\n\nI ran \"git fsck\" on both, which reported no problems. Moreover, I ran \"git gc\"\nand made sure there were no objects pending garbage collection, but still I\ncannot account for this difference. Can someone explain why these numbers\ndiffer, and if this is a problem or not?\n\nThanks,\n\nAndrew\n"},{"id":"277238","messageId":"vpq7finm6v8.fsf@anie.imag.fr","threadId":"41310","inReplyTo":"2142875754.710575.1454428371555.JavaMail.zimbra@xes-inc.com","subject":"Re: git object-count differs between clones","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2016-02-02T16:09:31Z","receivedAt":"2016-02-02T16:09:31Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Andrew Martin writes:\n\n> I ran \"git fsck\" on both, which reported no problems. Moreover, I ran \"git gc\"\n> and made sure there were no objects pending garbage collection, \n\nIt's not sufficient: you may have objects reachable from your reflog,\nhence not candidate for garbage collection. Since the reflog is not\npropagated, pushing + cloning will not transfer these objects if the\nreflog is the only way to reach them.\n\nYou may try expiring your reflog and \"git gc\" again.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"277241","messageId":"203317792.714602.1454430077194.JavaMail.zimbra@xes-inc.com","threadId":"41310","inReplyTo":"vpq7finm6v8.fsf@anie.imag.fr","subject":"Re: git object-count differs between clones","fromName":"Andrew Martin","fromEmail":"amartin@xes-inc.com","sentAt":"2016-02-02T16:21:17Z","receivedAt":"2016-02-02T16:21:17Z","isPatch":false,"sender":{"key":"amartin@xes-inc.com","avatar":null},"body":"----- Original Message -----\n> From: \"Matthieu Moy\" \n> To: \"Andrew Martin\" \n> Cc: git@vger.kernel.org\n> Sent: Tuesday, February 2, 2016 10:09:31 AM\n> Subject: Re: git object-count differs between clones\n> \n> Andrew Martin writes:\n> \n> > I ran \"git fsck\" on both, which reported no problems. Moreover, I ran \"git\n> > gc\"\n> > and made sure there were no objects pending garbage collection,\n> \n> It's not sufficient: you may have objects reachable from your reflog,\n> hence not candidate for garbage collection. Since the reflog is not\n> propagated, pushing + cloning will not transfer these objects if the\n> reflog is the only way to reach them.\n> \n> You may try expiring your reflog and \"git gc\" again.\n> \nMatthieu,\n\nThanks, I found some commits that are not referenced in any branch. How can I\nremove these from the reflog? I tried running\n\"git reflog expire --expire=now --expire-unreachable=now --all\" followed by\n\"git gc\" but still the same number of objects remain.\n\nThanks,\n\nAndrew\n"},{"id":"277249","messageId":"20160202165246.GA17822@sigill.intra.peff.net","threadId":"41310","inReplyTo":"203317792.714602.1454430077194.JavaMail.zimbra@xes-inc.com","subject":"Re: git object-count differs between clones","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-02T16:52:46Z","receivedAt":"2016-02-02T16:52:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 02, 2016 at 10:21:17AM -0600, Andrew Martin wrote:\n\n> > You may try expiring your reflog and \"git gc\" again.\n> \n> Thanks, I found some commits that are not referenced in any branch. How can I\n> remove these from the reflog? I tried running\n> \"git reflog expire --expire=now --expire-unreachable=now --all\" followed by\n> \"git gc\" but still the same number of objects remain.\n\nAre the objects now loose, or still in packs? Git has a grace period for\npruning objects, so that we do not delete objects for an in-progress\noperation. The life cycle of an unreferenced object should be something\nlike:\n\n - reachable by reflogs, which are pruned after 30 days (or\n gc.reflogExpireUnreachable config). Objects will be repacked as\n normal during this time. Override with \"reflog expire\" as you did\n above.\n\n - after the reflog expires, the objects are now unreachable. During\n the next repack, they'll be ejected from the pack into loose\n objects, and their mtimes set to match the pack they came from\n (which is probably quite recent if you just repacked!).\n\n - after 2 weeks (or gc.pruneExpire), unreachable loose objects are\n dropped by \"git prune\", which is called as part of \"git gc\". This is\n based on the object mtime.\n\n You can accelerate this with \"git gc --prune=now\" (or\n \"--prune=5.minutes.ago\").\n\n-Peff\n"},{"id":"277253","messageId":"1978336183.727736.1454433728105.JavaMail.zimbra@xes-inc.com","threadId":"41310","inReplyTo":"20160202165246.GA17822@sigill.intra.peff.net","subject":"Re: git object-count differs between clones","fromName":"Andrew Martin","fromEmail":"amartin@xes-inc.com","sentAt":"2016-02-02T17:22:08Z","receivedAt":"2016-02-02T17:22:08Z","isPatch":false,"sender":{"key":"amartin@xes-inc.com","avatar":null},"body":"----- Original Message -----\n> From: \"Jeff King\" \n> To: \"Andrew Martin\" \n> Cc: \"Matthieu Moy\" , git@vger.kernel.org\n> Sent: Tuesday, February 2, 2016 10:52:46 AM\n> Subject: Re: git object-count differs between clones\n> \n> On Tue, Feb 02, 2016 at 10:21:17AM -0600, Andrew Martin wrote:\n> \n> > > You may try expiring your reflog and \"git gc\" again.\n> > \n> > Thanks, I found some commits that are not referenced in any branch. How can\n> > I\n> > remove these from the reflog? I tried running\n> > \"git reflog expire --expire=now --expire-unreachable=now --all\" followed by\n> > \"git gc\" but still the same number of objects remain.\n> \n> Are the objects now loose, or still in packs? Git has a grace period for\n> pruning objects, so that we do not delete objects for an in-progress\n> operation. The life cycle of an unreferenced object should be something\n> like:\n> \n> - reachable by reflogs, which are pruned after 30 days (or\n> gc.reflogExpireUnreachable config). Objects will be repacked as\n> normal during this time. Override with \"reflog expire\" as you did\n> above.\n> \n> - after the reflog expires, the objects are now unreachable. During\n> the next repack, they'll be ejected from the pack into loose\n> objects, and their mtimes set to match the pack they came from\n> (which is probably quite recent if you just repacked!).\n> \n> - after 2 weeks (or gc.pruneExpire), unreachable loose objects are\n> dropped by \"git prune\", which is called as part of \"git gc\". This is\n> based on the object mtime.\n> \n> You can accelerate this with \"git gc --prune=now\" (or\n> \"--prune=5.minutes.ago\").\n> \n> -Peff\n\nJeff,\n\nThanks for the clarification. I now ran \"git repack -A\" followed by \n\"git gc --prune=now\", however I am still seeing the same number of objects. What\nelse can I try to successfully mark these and unreachable and garbage collect them?\n\nThanks,\n\nAndrew\n"},{"id":"277298","messageId":"20160203043411.GB21561@sigill.intra.peff.net","threadId":"41310","inReplyTo":"1978336183.727736.1454433728105.JavaMail.zimbra@xes-inc.com","subject":"Re: git object-count differs between clones","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-03T04:34:12Z","receivedAt":"2016-02-03T04:34:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 02, 2016 at 11:22:08AM -0600, Andrew Martin wrote:\n\n> Thanks for the clarification. I now ran \"git repack -A\" followed by \n> \"git gc --prune=now\", however I am still seeing the same number of objects. What\n> else can I try to successfully mark these and unreachable and garbage collect them?\n\nThat should clear out any unreachable objects. Are we sure that the\nobjects in question are, in fact, unreachable?\n\nTry:\n\n git rev-list --objects --all --reflog | wc -l\n\nwhich should give a count of reachable objects. I'd expect that to line\nup with that \"git count-objects -v\" reports after having run your gc\nabove.\n\nIn your original email, the discrepancy was between your \"original\"\nrepository and the one that had round-tripped to a clone. Is it possible\nthere are refs in the original that did not get pushed? Try comparing\n\"git for-each-ref\" in each repository.\n\nWe also consider objects in the index to be reachable for packing. Could\nyour original perhaps have some uncommitted objects mentioned in the\nindex?\n\n-Peff\n"},{"id":"277337","messageId":"386465007.95896.1454512901907.JavaMail.zimbra@xes-inc.com","threadId":"41310","inReplyTo":"20160203043411.GB21561@sigill.intra.peff.net","subject":"Re: git object-count differs between clones","fromName":"Andrew Martin","fromEmail":"amartin@xes-inc.com","sentAt":"2016-02-03T15:21:41Z","receivedAt":"2016-02-03T15:21:41Z","isPatch":false,"sender":{"key":"amartin@xes-inc.com","avatar":null},"body":"----- Original Message -----\n> From: \"Jeff King\" \n> To: \"Andrew Martin\" \n> Cc: \"Matthieu Moy\" , git@vger.kernel.org\n> Sent: Tuesday, February 2, 2016 10:34:12 PM\n> Subject: Re: git object-count differs between clones\n> \n> On Tue, Feb 02, 2016 at 11:22:08AM -0600, Andrew Martin wrote:\n> \n> > Thanks for the clarification. I now ran \"git repack -A\" followed by\n> > \"git gc --prune=now\", however I am still seeing the same number of objects.\n> > What\n> > else can I try to successfully mark these and unreachable and garbage\n> > collect them?\n> \n> That should clear out any unreachable objects. Are we sure that the\n> objects in question are, in fact, unreachable?\n> \n> Try:\n> \n> git rev-list --objects --all --reflog | wc -l\n> \n> which should give a count of reachable objects. I'd expect that to line\n> up with that \"git count-objects -v\" reports after having run your gc\n> above.\n> \n> In your original email, the discrepancy was between your \"original\"\n> repository and the one that had round-tripped to a clone. Is it possible\n> there are refs in the original that did not get pushed? Try comparing\n> \"git for-each-ref\" in each repository.\n> \n> We also consider objects in the index to be reachable for packing. Could\n> your original perhaps have some uncommitted objects mentioned in the\n> index?\n> \n> -Peff\n> \n\nJeff,\n\nThis did it - those commits were still referenced by some remotes:\n$ git for-each-ref\n945c3a60dfb4d9ab774708d19f7aa74dd545db90 commit refs/heads/master\n945c3a60dfb4d9ab774708d19f7aa74dd545db90 commit refs/remotes/origin/brancha\n8b331e4bb42f6291c33eb0847c4481407e3d753c commit refs/remotes/origin/branchb\n\nI removed them:\n$ git update-ref -d refs/remotes/origin/brancha\n$ git update-ref -d refs/remotes/origin/branchb\n\nAnd then I was able to garbage collect and get the expected object count:\n$ git reflog expire --expire=now --expire-unreachable=now --all\n$ git gc --prune=now\n\nThanks for the help!\n\nAndrew\n"}]}