# Re: fsck --full is Ok, but clones are not, "missing commits"?! 9 messages from 2008-04-16 to 2008-04-18. Participants: Brian Foster, Johannes Sixt, Brandon Casey, Dmitry Potapov. Thread: https://gitlist.dev/t/13139 ## Brian Foster, 2008-04-16 15:04 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: URL: https://gitlist.dev/e/a537dd660804160804t252d2900j33e30a01dc58979c%40mail.gmail.com In-Reply-To: <200804161626.44174.brian.foster@innova-card.com> ``` Johannes Sixt helpfully observes: > Brian Foster schrieb: > > Dmitry Potapov correctly deduced: > >> I suspect your original git repository has info/grafts > > bingo! YES, it does: >[ ... ] > > the goal is to put things into a sane state so any new > > clones are healthy. there's only one(?) existing clone, > > which may or may not be(? become?) an issue. > > Just move info/grafts out of the way and you *may* be all set. Don't > delete it - there might be a reason that the file exists. Hannes & Dmitry, thanks for the suggestions. I've made (and carefully protected) multiple backups, and am doing all my experiments on copies, and have taken other protective steps, so no worries about accidents. moving info/grafts out of the way causes `fsck --full' to issue the same complaints as it does for clones. conversely, copying info/grafts to a clone makes `fsck --full' happy. furthermore, close examination of that one pre-existing clone shows it has the identical info/grafts as the bare repository. so that seems to have been what was previously being done (I'm now attempting to confirm this, and also to find out WHY). however, no joy at all with the suggestion: > Your best bet is to run 'git filter-branch --tag-name-filter cat -- --all' > on *a copy* of your bare repository (with info/grafts in place). [ ... ] at toplevel of (a copy of) the bare repository, with info/grafts in-place, and `fsck --full' happy: $ git version 1.5.3 $ git filter-branch --tag-name-filter cat -- --all # at bare toplevel fatal: Not a git repository: '.git' You need to run this command from the toplevel of the working tree. $ at toplevel of a (not-bare) clone, with info/grafts in-place, and a happy `fsck -full' (same machine so same git version): $ git filter-branch --tag-name-filter cat -- --all # at not-bare toplevel Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168) Ref 'refs/heads/master' was rewritten Ref 'refs/remotes/origin/HEAD' was rewritten WARNING: Ref 'refs/remotes/origin/master' is unchanged Ref 'refs/tags/linux-2.6.15' was rewritten error: Ref refs/tags/linux-2.6.15 is at \ 26a33413c95dfda6c70ca4a83da49cddb7b236b9 but expected \ 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4 fatal: refs/tags/linux-2.6.15: cannot lock the ref Could not rewrite refs/tags/linux-2.6.15 $ in the later case, the info/grafts still existed at the end, and `fsck --full' was happy. (I suppose this isn't surprising since the filter-branch was not happy.) moving info/grafts out of the way caused the same complaints from `fsck --full'. the missing commits are all(? mostly?) similar to this example: $ git remote origin $ git remote show origin * remote origin URL: git://git.kernel.org/pub/scm/linux/kernel/git/ralf/linux.git [ ... ] $ git show dff364d8da commit dff364d8da15be0b856a174062fb785acb1c363e Merge: bc59a40... 427abfa... Author: Ralf Baechle Date: Sun Jun 18 03:42:49 2006 +0100 Merge tag 'v2.6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6 $ as such, is there some way of adding them back to the bare repository (if that even makes sense?), or whatever? (i.e., that have not been lost, is it possible to take advantage of that fact?) also (sorry!), does anyone recognise the development process that apparently was used? (the one pre-existing clone has few-to-no clews, since it was used for some fairly trivial local development, not for the "merging" (if I can call it that) with linux-mips repository.) cheers! -blf- -- "How many surrealists does it take to | Brian Foster change a lightbulb? Three. One calms | somewhere in south of France the warthog, and two fill the bathtub | Stop E$$o (ExxonMobil)! with brightly-coloured machine tools." | http://www.stopesso.com ``` ## Johannes Sixt, 2008-04-16 15:22 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: <480619AE.8010506@viscovery.net> URL: https://gitlist.dev/e/480619AE.8010506%40viscovery.net In-Reply-To: ``` Brian Foster schrieb: > at toplevel of a (not-bare) clone, with info/grafts in-place, > and a happy `fsck -full' (same machine so same git version): > > $ git filter-branch --tag-name-filter cat -- --all # at not-bare toplevel > Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168) > Ref 'refs/heads/master' was rewritten > Ref 'refs/remotes/origin/HEAD' was rewritten > WARNING: Ref 'refs/remotes/origin/master' is unchanged > Ref 'refs/tags/linux-2.6.15' was rewritten > error: Ref refs/tags/linux-2.6.15 is at \ > 26a33413c95dfda6c70ca4a83da49cddb7b236b9 but expected \ > 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4 > fatal: refs/tags/linux-2.6.15: cannot lock the ref > Could not rewrite refs/tags/linux-2.6.15 > $ Actually, I don't know how to overcome this situation; maybe forget about the tags and remove the '--tag-name-filter cat' part. They wouldn't have been rewritten correctly anyway (annotated tags loose the message and become unannotated). > as such, is there some way of adding them back to the bare > repository (if that even makes sense?), or whatever? (i.e., > that have not been lost, is it possible to take advantage > of that fact?) > > also (sorry!), does anyone recognise the development process > that apparently was used? (the one pre-existing clone has > few-to-no clews, since it was used for some fairly trivial > local development, not for the "merging" (if I can call it > that) with linux-mips repository.) In this case you might be able to salvage missing objects by cloning linux-mips. Just copy the objects/pack/* from that clone into your objects/pack, remove info/grafts, and maybe things "just work"? -- Hannes ``` ## Brandon Casey, 2008-04-16 16:11 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: <48062515.50903@nrlssc.navy.mil> URL: https://gitlist.dev/e/48062515.50903%40nrlssc.navy.mil In-Reply-To: <480619AE.8010506@viscovery.net> ``` Johannes Sixt wrote: > Brian Foster schrieb: >> at toplevel of a (not-bare) clone, with info/grafts in-place, >> and a happy `fsck -full' (same machine so same git version): >> >> $ git filter-branch --tag-name-filter cat -- --all # at not-bare toplevel >> Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168) >> Ref 'refs/heads/master' was rewritten >> Ref 'refs/remotes/origin/HEAD' was rewritten >> WARNING: Ref 'refs/remotes/origin/master' is unchanged >> Ref 'refs/tags/linux-2.6.15' was rewritten >> error: Ref refs/tags/linux-2.6.15 is at \ >> 26a33413c95dfda6c70ca4a83da49cddb7b236b9 but expected \ >> 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4 >> fatal: refs/tags/linux-2.6.15: cannot lock the ref >> Could not rewrite refs/tags/linux-2.6.15 >> $ > > Actually, I don't know how to overcome this situation; maybe forget about > the tags and remove the '--tag-name-filter cat' part. They wouldn't have > been rewritten correctly anyway (annotated tags loose the message and > become unannotated). I don't think the tag-name-filter section has been reached yet. I think the problem is the --all option to rev-parse. This adds everything under .git/refs to the list of heads to rewrite. Just a guess, but possibly the linux-2.6.15 is an annotated tag, which is dereferenced to a commit (the oldsha1), and then when update-ref is called with the newsha1 and the oldsha1, the oldsha1 doesn't match what is in refs/tags/linux-2.6.15 since that points to the sha1 of the tag object. Try --branches instead of --all. Or leave it blank if you just want to filter the current branch. Also, if you're interested in retaining the annotations from the tags, search the list for '[PATCH v4] filter-branch.sh: support nearly proper tag name filtering' or grab the patch from git's pu branch. -brandon ``` ## Dmitry Potapov, 2008-04-16 17:15 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: <20080416171547.GE3133@dpotapov.dyndns.org> URL: https://gitlist.dev/e/20080416171547.GE3133%40dpotapov.dyndns.org In-Reply-To: ``` On Wed, Apr 16, 2008 at 05:04:56PM +0200, Brian Foster wrote: > > in the later case, the info/grafts still existed at the end, > and `fsck --full' was happy. (I suppose this isn't surprising > since the filter-branch was not happy.) moving info/grafts > out of the way caused the same complaints from `fsck --full'. Do not remove info/grafts. Just clone this repository after filter-branch and then the _newly_ cloned repository should be fine. Dmitry ``` ## Brian Foster, 2008-04-17 14:53 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: URL: https://gitlist.dev/e/a537dd660804170753i4dcfd189m45a9581294ce5091%40mail.gmail.com In-Reply-To: <200804171643.15504.brian.foster@innova-card.com> ``` Hannes suggested: > Brian Foster schrieb: > >[ ... ] is there some way of adding [ the missing commits, > > all of which seem to be from linux-mips, ] back to the bare > > repository (if that even makes sense?), or whatever? (i.e., > > [ they ] have not been lost, [ so ] is it possible to take > > advantage of that fact?) [ ... ] > > In this case you might be able to salvage missing objects by cloning > linux-mips. Just copy the objects/pack/* from that clone into your > objects/pack, remove info/grafts, and maybe things "just work"? Hannes et al., thanks for the suggestion. bingo! that basically worked. after copying linux-mips objects/pack/* into my goofy bare repository, `fsck --full' (after moving grafts out of the way) found different issues. it complained about a zillion dangling tags (yawn), plus a handful of dangling commits. the dangling tags were the linux-mips refs/tags/*. copying them into my repository's ref/tags fixed that. (there were no name collisions with the existing tags, so this was easy.) the dangling commits were linux-mips refs/remotes/origin/*, and again could be safely/easily copied into my repository. `fsck --full' is now 100% happy. (yea!) the obvious missing thing (which I _think_ is easy to fix?) is the remote URL &tc is not in my now-not-so-goofy bare repository. hence `branch -r' shows origin/* branches, but (I speculate) a `pull' (e.g.) will be confused. at this point in time I've not done any extensive testing of the seemingly-fixed repository, but things are looking much better. before trying the copying suggestion, I played some more with `filter-branch'. I had no success at all. as one example, with `--branches' instead of `--all' (one of Brandon's suggestions) produced: $ git filter-branch --tag-name-filter cat -- --branches Which ref do you want to rewrite? $ also, Dmitry's suggestion of cloning (with grafts still in-place) after `filter-branch ... --all' accomplished nothing obvious: `fsck --full' of the new clone was still unhappy (same set of complaints). whilst I'm still trying to understand the rationale for why things were set-up(? left-in?) this weird state, browsing the history gives some vague hints. however, the intended usage model remains opaque. and that will soon be my next problem: what's a better (best?) usage model (for this project)? I need to do some more reading here .... ;-\ cheers! -blf- -- "How many surrealists does it take to | Brian Foster change a lightbulb? Three. One calms | somewhere in south of France the warthog, and two fill the bathtub | Stop E$$o (ExxonMobil)! with brightly-coloured machine tools." | http://www.stopesso.com ``` ## Brandon Casey, 2008-04-17 15:41 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: <48076F94.7030206@nrlssc.navy.mil> URL: https://gitlist.dev/e/48076F94.7030206%40nrlssc.navy.mil In-Reply-To: ``` Brian Foster wrote: > before trying the copying suggestion, I played some > more with `filter-branch'. I had no success at all. > as one example, with `--branches' instead of `--all' > (one of Brandon's suggestions) produced: > > $ git filter-branch --tag-name-filter cat -- --branches > Which ref do you want to rewrite? > $ That's strange. Are there any branches in this repository? Yes, looking at your earlier email: at toplevel of a (not-bare) clone, with info/grafts in-place, and a happy `fsck -full' (same machine so same git version): $ git filter-branch --tag-name-filter cat -- --all # at not-bare toplevel Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168) Ref 'refs/heads/master' was rewritten Ref 'refs/remotes/origin/HEAD' was rewritten [snip] Looks like there is a single branch named 'master'. The command within git-filter-branch.sh that is being called is git rev-parse --no-flags --revs-only --symbolic-full-name --default HEAD --branches In a trim git repository for example this command returns: refs/heads/master refs/heads/next Referring to two branches 'master' and 'next'. These are the references that exist in .git/refs/heads/*. I'd be interested to see what this command returns in your repo. You can compare this output to the output you get using --all and also relate that to the directory structure beneath .git/refs/ -brandon p.s. It's more polite to keep the people you are responding to in the cc list. Take a look at how Hannes did it (the email to which this one is replying). Usually it's enough to select 'Reply All'. ``` ## Brian Foster, 2008-04-18 08:41 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: URL: https://gitlist.dev/e/a537dd660804180141h25b7f8e2yf7283f63d49521e6%40mail.gmail.com In-Reply-To: <200804180943.20933.brian.foster@innova-card.com> ``` > Brian Foster wrote: > > $ git filter-branch --tag-name-filter cat -- --branches > > Which ref do you want to rewrite? > > $ > > That's strange. Are there any branches in this repository? [ ... ] > Looks like there is a single branch named 'master'. Brandon et al., correct. there is just the `master' branch. it is extremely clean and straightforward: O---O---O---O-- ... --O---O > The command within git-filter-branch.sh that is being called is > > git rev-parse --no-flags --revs-only --symbolic-full-name --default HEAD --branches > > In a trim git repository for example this command returns: > > refs/heads/master > refs/heads/next > > Referring to two branches 'master' and 'next'. These are the references > that exist in .git/refs/heads/*. > > I'd be interested to see what this command returns in your repo. below is a full example session, including, eventually, the above command `rev-parse ...' command, and several observations. all the outputs are quite short, so I include most in full: $ git version git version 1.5.3 $ $ git clone /pub/scm/linux/kernel/git/linux-2.6-usip.git linux-2.6-usip Initialized empty Git repository in /home/bfoster/usip-linux/linux-2.6-usip/.git/ 165531 blocks Checking 21723 files out... 100% (21723/21723) done $ $ cd linux-2.6-usip $ git fsck --full broken link from commit dd3f3c0636cfd50719c706b030db5473b0270add to commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb missing commit fb57c018d15005b60f104e57f198ff34a6035b99 missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb missing commit dff364d8da15be0b856a174062fb785acb1c363e broken link from commit 8700854c41a40d333e90104971c3abbbcf082e57 to commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551 missing commit ce33a5e8c32816f38862bc41560a1d646d0803a6 missing commit 91483db9b01b547ae9cc45c8c98b217642acb40a missing commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551 broken link from commit c87c46fe892211f8aa4fd363ccff4f667a9aaf7d to commit ce33a5e8c32816f38862bc41560a1d646d0803a6 broken link from commit fb5967688f7b464421cff28f266b64ad2a313a9e to commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c broken link from commit e5a60f1636cceac33777bb8098a0b7a4a136a56c to commit fb57c018d15005b60f104e57f198ff34a6035b99 broken link from commit 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4 to commit 91483db9b01b547ae9cc45c8c98b217642acb40a broken link from commit 0ff75b3afff6fb306bef221bf1823ccf5ffc568b to commit dff364d8da15be0b856a174062fb785acb1c363e $ $ git filter-branch --tag-name-filter cat -- --branches Which ref do you want to rewrite? $ $ ls -laR .git-rewrite .git-rewrite: total 16 drwxrwxr-x 3 bfoster bfoster 4096 Apr 18 10:11 . drwxrwxr-x 21 bfoster bfoster 4096 Apr 18 10:11 .. -rw-rw-r-- 1 bfoster bfoster 1864 Apr 18 10:11 backup-refs -rw-rw-r-- 1 bfoster bfoster 0 Apr 18 10:11 heads drwxrwxr-x 2 bfoster bfoster 4096 Apr 18 10:11 t .git-rewrite/t: total 8 drwxrwxr-x 2 bfoster bfoster 4096 Apr 18 10:11 . drwxrwxr-x 3 bfoster bfoster 4096 Apr 18 10:11 .. $ watching the execution of git-filter-branch with `bash -x', the key point seems to be that `check-ref-formats' is being called with what seems to be a bogus argument: + git check-ref-format 'refs/heads/master refs/remotes/origin/master' that prints nothing on stdout and exits with status 1. hence .git-rewrite/heads is empty, which is the proximate cause of the "Which ref ..." message. (more on this below.) $ git filter-branch --tag-name-filter cat -- --branches .git-rewrite already exists, please remove it $ $ git rev-parse --no-flags --revs-only \ --symbolic-full-name --default HEAD --branches 7df30811617517bc4d5ec7c190a435667228320c $ > You can compare this output to the output you get using --all and also > relate that to the directory structure beneath .git/refs/ $ git branch -a * master origin/HEAD origin/master $ that seemingly bogus `check-ref-format' argument comes from the pipeline (where ref=master): git for-each-ref --format='%(refname)' | grep /"$ref" i.e.: git for-each-ref --format='%(refname)' | grep /master so... $ git for-each-ref --format='%(refname)' refs/heads/master refs/remotes/origin/HEAD refs/remotes/origin/master refs/tags/linux-2.6.15 refs/tags/linux-usip-v2.6.15 [ ... ] $ note two names match the RE `/master', hence the bogus argument. the reason for that seems to be (this is a guess!) that `rev-parse' is *not* being run as: git rev-parse --no-flags --revs-only \ --symbolic-full-name --default HEAD --branches as I _think_ you are claiming it should be, but as: git rev-parse --revs-only --symbolic --branches the former outputs a list of SHA1s (see above). the later outputs just the name `master': $ git rev-parse --revs-only --symbolic --branches master $ just for completeness, below is the output with `--all': $ git rev-parse --no-flags --revs-only \ --symbolic-full-name --default HEAD --all 7df30811617517bc4d5ec7c190a435667228320c 7df30811617517bc4d5ec7c190a435667228320c 7df30811617517bc4d5ec7c190a435667228320c 26a33413c95dfda6c70ca4a83da49cddb7b236b9 3e1e17e642cd2fface5691ab136d79f709af9a1c 8b9e620aa7f6231e30ddf70cf38759aea7c98070 cd220dbc113774b420c634fe1907860387883aa2 acb1587f596e4bfe4c748be742b6bf1509fc88ed 938ed01b502fbf659de24c4ad1da2bb60e0a3347 d52b1d70e5f4c0509b6220070a1e585ac9f6d727 5893ac94b533aba4a9864b0a4a3c346830ec004f bf0dfbc784e2734c61818aa0683b716cf82f13e2 824a82029f8ebf2f72846490a9cd455ceaf2f5cc b07c23ef78c785168b579242f652b5f49749b02b 0ae7caf477cd6a8b265497f1fb2e4273378dcfb4 57545107a114f6f6fb0bcc9a0d655d84dc32113c db7e2bc1e16ab3c6611de4b84fda529e59cb378e df1ae0303a60ee0268012e926240720ac4d01de9 b7e5fe765a0afd315781fb536ffcbd32a6666421 1effee36b5784ebe5e3e7e7d34bfaf6f63be3afa b9a029d4c06eb18778f8907b359fe1616d1e2019 250210f21cadd70ea5cf459b6dca990d1faecbea dc8843f2fd0954c941aff8b06205cd15c1f0873f 36ee52578b10eb83d2f30e324117bfb9457e1e0e f8e0127a4cd3c1e42665a8e895e0c1a9dee052ce $ $ cat .git/refs/heads/master 7df30811617517bc4d5ec7c190a435667228320c $ this smells like a bug in this particular version (1.5.3) of `git', or perhaps in the installation? like the repository, both are inherited, and I do not know the history. ;-( cheers! -blf- -- "How many surrealists does it take to | Brian Foster change a lightbulb? Three. One calms | somewhere in south of France the warthog, and two fill the bathtub | Stop E$$o (ExxonMobil)! with brightly-coloured machine tools." | http://www.stopesso.com ``` ## Johannes Sixt, 2008-04-18 08:55 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: <48086200.5020600@viscovery.net> URL: https://gitlist.dev/e/48086200.5020600%40viscovery.net In-Reply-To: ``` Brian Foster schrieb: > that seemingly bogus `check-ref-format' argument comes > from the pipeline (where ref=master): > > git for-each-ref --format='%(refname)' | grep /"$ref" > i.e.: > git for-each-ref --format='%(refname)' | grep /master > > so... > $ git for-each-ref --format='%(refname)' > refs/heads/master > refs/remotes/origin/HEAD > refs/remotes/origin/master > refs/tags/linux-2.6.15 > refs/tags/linux-usip-v2.6.15 > [ ... ] > $ > > note two names match the RE `/master', hence the bogus > argument. the reason for that seems to be (this is a > guess!) that `rev-parse' is *not* being run as: > > git rev-parse --no-flags --revs-only \ > --symbolic-full-name --default HEAD --branches > > as I _think_ you are claiming it should be, but as: > > git rev-parse --revs-only --symbolic --branches One of the changes since v.1.5.3 was to get rid of this bogus grep. Can you install a newer git? -- Hannes ``` ## Brian Foster, 2008-04-18 09:29 Subject: Re: fsck --full is Ok, but clones are not, "missing commits"?! Message-ID: URL: https://gitlist.dev/e/a537dd660804180229i263fea0cxb7f7ac9e8f4780%40mail.gmail.com In-Reply-To: <200804181114.47067.brian.foster@innova-card.com> ``` Johannes Sixt asked: > One of the changes since v.1.5.3 was to get rid of this bogus grep > [ in `check-ref-format' ]. Can you install a newer git? Hannes, yes! about 10 minutes ago I was handed the keys to a brand new VM which is to be the company's git server. a very important point is it will be backed-up (unlike the former repository!). git on the VM is "my" problem, and one of the first things I must do is install it. the VM is running "CentOS release 5 (Final)". all things being equal, I'd prefer to install a reliable/stable pre-built package (RPM, I assume, since CentOS is based on Redhat). I confess I do not know (have not looked) where to find suitable RPMs of git. suggestions are most welcome! cheers! -blf- -- "How many surrealists does it take to | Brian Foster change a lightbulb? Three. One calms | somewhere in south of France the warthog, and two fill the bathtub | Stop E$$o (ExxonMobil)! with brightly-coloured machine tools." | http://www.stopesso.com ```