# git object-count differs between clones 7 messages from 2016-02-02 to 2016-02-03. Participants: Andrew Martin, Matthieu Moy, Jeff King. Thread: https://gitlist.dev/t/41310 ## Andrew Martin, 2016-02-02 15:52 Subject: git object-count differs between clones Message-ID: <2142875754.710575.1454428371555.JavaMail.zimbra@xes-inc.com> URL: https://gitlist.dev/e/2142875754.710575.1454428371555.JavaMail.zimbra%40xes-inc.com In-Reply-To: <57434188.709288.1454428054374.JavaMail.zimbra@xes-inc.com> ``` Hello, I am using git 2.7.0 on Ubuntu 14.04. I recently tried pushing a large (90,000+ commits) repository to a gogs server (https://gogs.io/) and then cloning the repository back to my machine. After running "git count-objects -v", I see a discrepancy in the number of objects: Original Repository: count: 0 size: 0 in-pack: 1258300 packs: 1 size-pack: 593889 prune-packable: 0 garbage: 0 size-garbage: 0 Clone from gogs: count: 0 size: 0 in-pack: 1258270 packs: 1 size-pack: 593884 prune-packable: 0 garbage: 0 size-garbage: 0 I ran "git fsck" on both, which reported no problems. Moreover, I ran "git gc" and made sure there were no objects pending garbage collection, but still I cannot account for this difference. Can someone explain why these numbers differ, and if this is a problem or not? Thanks, Andrew ``` ## Matthieu Moy, 2016-02-02 16:09 Subject: Re: git object-count differs between clones Message-ID: URL: https://gitlist.dev/e/vpq7finm6v8.fsf%40anie.imag.fr In-Reply-To: <2142875754.710575.1454428371555.JavaMail.zimbra@xes-inc.com> ``` Andrew Martin writes: > I ran "git fsck" on both, which reported no problems. Moreover, I ran "git gc" > and made sure there were no objects pending garbage collection, It's not sufficient: you may have objects reachable from your reflog, hence not candidate for garbage collection. Since the reflog is not propagated, pushing + cloning will not transfer these objects if the reflog is the only way to reach them. You may try expiring your reflog and "git gc" again. -- Matthieu Moy http://www-verimag.imag.fr/~moy/ ``` ## Andrew Martin, 2016-02-02 16:21 Subject: Re: git object-count differs between clones Message-ID: <203317792.714602.1454430077194.JavaMail.zimbra@xes-inc.com> URL: https://gitlist.dev/e/203317792.714602.1454430077194.JavaMail.zimbra%40xes-inc.com In-Reply-To: ``` ----- Original Message ----- > From: "Matthieu Moy" > To: "Andrew Martin" > Cc: git@vger.kernel.org > Sent: Tuesday, February 2, 2016 10:09:31 AM > Subject: Re: git object-count differs between clones > > Andrew Martin writes: > > > I ran "git fsck" on both, which reported no problems. Moreover, I ran "git > > gc" > > and made sure there were no objects pending garbage collection, > > It's not sufficient: you may have objects reachable from your reflog, > hence not candidate for garbage collection. Since the reflog is not > propagated, pushing + cloning will not transfer these objects if the > reflog is the only way to reach them. > > You may try expiring your reflog and "git gc" again. > Matthieu, Thanks, I found some commits that are not referenced in any branch. How can I remove these from the reflog? I tried running "git reflog expire --expire=now --expire-unreachable=now --all" followed by "git gc" but still the same number of objects remain. Thanks, Andrew ``` ## Jeff King, 2016-02-02 16:52 Subject: Re: git object-count differs between clones Message-ID: <20160202165246.GA17822@sigill.intra.peff.net> URL: https://gitlist.dev/e/20160202165246.GA17822%40sigill.intra.peff.net In-Reply-To: <203317792.714602.1454430077194.JavaMail.zimbra@xes-inc.com> ``` On Tue, Feb 02, 2016 at 10:21:17AM -0600, Andrew Martin wrote: > > You may try expiring your reflog and "git gc" again. > > Thanks, I found some commits that are not referenced in any branch. How can I > remove these from the reflog? I tried running > "git reflog expire --expire=now --expire-unreachable=now --all" followed by > "git gc" but still the same number of objects remain. Are the objects now loose, or still in packs? Git has a grace period for pruning objects, so that we do not delete objects for an in-progress operation. The life cycle of an unreferenced object should be something like: - reachable by reflogs, which are pruned after 30 days (or gc.reflogExpireUnreachable config). Objects will be repacked as normal during this time. Override with "reflog expire" as you did above. - after the reflog expires, the objects are now unreachable. During the next repack, they'll be ejected from the pack into loose objects, and their mtimes set to match the pack they came from (which is probably quite recent if you just repacked!). - after 2 weeks (or gc.pruneExpire), unreachable loose objects are dropped by "git prune", which is called as part of "git gc". This is based on the object mtime. You can accelerate this with "git gc --prune=now" (or "--prune=5.minutes.ago"). -Peff ``` ## Andrew Martin, 2016-02-02 17:22 Subject: Re: git object-count differs between clones Message-ID: <1978336183.727736.1454433728105.JavaMail.zimbra@xes-inc.com> URL: https://gitlist.dev/e/1978336183.727736.1454433728105.JavaMail.zimbra%40xes-inc.com In-Reply-To: <20160202165246.GA17822@sigill.intra.peff.net> ``` ----- Original Message ----- > From: "Jeff King" > To: "Andrew Martin" > Cc: "Matthieu Moy" , git@vger.kernel.org > Sent: Tuesday, February 2, 2016 10:52:46 AM > Subject: Re: git object-count differs between clones > > On Tue, Feb 02, 2016 at 10:21:17AM -0600, Andrew Martin wrote: > > > > You may try expiring your reflog and "git gc" again. > > > > Thanks, I found some commits that are not referenced in any branch. How can > > I > > remove these from the reflog? I tried running > > "git reflog expire --expire=now --expire-unreachable=now --all" followed by > > "git gc" but still the same number of objects remain. > > Are the objects now loose, or still in packs? Git has a grace period for > pruning objects, so that we do not delete objects for an in-progress > operation. The life cycle of an unreferenced object should be something > like: > > - reachable by reflogs, which are pruned after 30 days (or > gc.reflogExpireUnreachable config). Objects will be repacked as > normal during this time. Override with "reflog expire" as you did > above. > > - after the reflog expires, the objects are now unreachable. During > the next repack, they'll be ejected from the pack into loose > objects, and their mtimes set to match the pack they came from > (which is probably quite recent if you just repacked!). > > - after 2 weeks (or gc.pruneExpire), unreachable loose objects are > dropped by "git prune", which is called as part of "git gc". This is > based on the object mtime. > > You can accelerate this with "git gc --prune=now" (or > "--prune=5.minutes.ago"). > > -Peff Jeff, Thanks for the clarification. I now ran "git repack -A" followed by "git gc --prune=now", however I am still seeing the same number of objects. What else can I try to successfully mark these and unreachable and garbage collect them? Thanks, Andrew ``` ## Jeff King, 2016-02-03 04:34 Subject: Re: git object-count differs between clones Message-ID: <20160203043411.GB21561@sigill.intra.peff.net> URL: https://gitlist.dev/e/20160203043411.GB21561%40sigill.intra.peff.net In-Reply-To: <1978336183.727736.1454433728105.JavaMail.zimbra@xes-inc.com> ``` On Tue, Feb 02, 2016 at 11:22:08AM -0600, Andrew Martin wrote: > Thanks for the clarification. I now ran "git repack -A" followed by > "git gc --prune=now", however I am still seeing the same number of objects. What > else can I try to successfully mark these and unreachable and garbage collect them? That should clear out any unreachable objects. Are we sure that the objects in question are, in fact, unreachable? Try: git rev-list --objects --all --reflog | wc -l which should give a count of reachable objects. I'd expect that to line up with that "git count-objects -v" reports after having run your gc above. In your original email, the discrepancy was between your "original" repository and the one that had round-tripped to a clone. Is it possible there are refs in the original that did not get pushed? Try comparing "git for-each-ref" in each repository. We also consider objects in the index to be reachable for packing. Could your original perhaps have some uncommitted objects mentioned in the index? -Peff ``` ## Andrew Martin, 2016-02-03 15:21 Subject: Re: git object-count differs between clones Message-ID: <386465007.95896.1454512901907.JavaMail.zimbra@xes-inc.com> URL: https://gitlist.dev/e/386465007.95896.1454512901907.JavaMail.zimbra%40xes-inc.com In-Reply-To: <20160203043411.GB21561@sigill.intra.peff.net> ``` ----- Original Message ----- > From: "Jeff King" > To: "Andrew Martin" > Cc: "Matthieu Moy" , git@vger.kernel.org > Sent: Tuesday, February 2, 2016 10:34:12 PM > Subject: Re: git object-count differs between clones > > On Tue, Feb 02, 2016 at 11:22:08AM -0600, Andrew Martin wrote: > > > Thanks for the clarification. I now ran "git repack -A" followed by > > "git gc --prune=now", however I am still seeing the same number of objects. > > What > > else can I try to successfully mark these and unreachable and garbage > > collect them? > > That should clear out any unreachable objects. Are we sure that the > objects in question are, in fact, unreachable? > > Try: > > git rev-list --objects --all --reflog | wc -l > > which should give a count of reachable objects. I'd expect that to line > up with that "git count-objects -v" reports after having run your gc > above. > > In your original email, the discrepancy was between your "original" > repository and the one that had round-tripped to a clone. Is it possible > there are refs in the original that did not get pushed? Try comparing > "git for-each-ref" in each repository. > > We also consider objects in the index to be reachable for packing. Could > your original perhaps have some uncommitted objects mentioned in the > index? > > -Peff > Jeff, This did it - those commits were still referenced by some remotes: $ git for-each-ref 945c3a60dfb4d9ab774708d19f7aa74dd545db90 commit refs/heads/master 945c3a60dfb4d9ab774708d19f7aa74dd545db90 commit refs/remotes/origin/brancha 8b331e4bb42f6291c33eb0847c4481407e3d753c commit refs/remotes/origin/branchb I removed them: $ git update-ref -d refs/remotes/origin/brancha $ git update-ref -d refs/remotes/origin/branchb And then I was able to garbage collect and get the expected object count: $ git reflog expire --expire=now --expire-unreachable=now --all $ git gc --prune=now Thanks for the help! Andrew ```