git update --prune issue

I have an unreliable problem when using "git remote update --prune". git claims that many refs from a particular remote do not point to a valid object, but only after finishing another update.  I've included a shortened version of the output from one particular instance of the issue.  Note that the errors are printed for every subsequent remote as well.  However, after the update is completed, everything is fine.

It seems to only happen when there are non-fast-forward changes (new branch, forced update, pruned branch).  The issue only happens with this particular remote, which I've tried removing and recreating, and is the same type of remote as all my others (another user's NFS-mounted home directory).  However, my remotes are all individual developers, and this developer is the only one who ever rebases her working branches.  (recloning the repo from origin and setting up my config and remotes all over again has also had no effect)

Unfortunately, I have been unable to reproduce the problem in any test repos - for example, though a forced update and a pruned branch in the problematic remote along with an update in another remote seems to fairly reliably produce the problem in this repo, recreating that situation in another repo doesn't cause any problems.  Sorry for the incomplete bug report, but perhaps this will be enough to go on!

Thanks, Jeffrey

I've seen the issue in previous versions built from git.git master, as well as v1.6.3.3, but for this particular one:

$ git --version git version 1.6.5.1.61.ge79999

$ git remote update --prune
Updating origin
remote: Counting objects: 42, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 15 (delta 9), reused 12 (delta 6)
Unpacking objects: 100% (15/15), done.
From /users/cxtfcm/CxTF_DB
   88b8613..d40f26d  2009.Q4    -> origin/2009.Q4
   d40f26d..56305b8  dev        -> origin/dev
Updating steph
remote: Counting objects: 299, done.
remote: Compressing objects: 100% (241/241), done.
remote: Total 276 (delta 186), reused 19 (delta 6)
Receiving objects: 100% (276/276), 41.09 KiB | 10 KiB/s, done.
Resolving deltas: 100% (186/186), completed with 17 local objects.
From /users/sdewet/CxTF_DEV/CxTF_DB
 + c2439dd...69cb5c3 beta_gc_dev -> steph/beta_gc_dev  (forced update)
 + fb25173...f0e4963 beta_veh_dev -> steph/beta_veh_dev  (forced update)
 * [new branch]      beta_veh_dev_old -> steph/beta_veh_dev_old
Pruning steph
URL: /users/sdewet/CxTF_DEV/CxTF_DB/
 * [pruned] steph/beta_gc_dev_old
Updating kevin
error: refs/remotes/steph/beta_gc_dev does not point to a valid object!
error: refs/remotes/steph/beta_veh_dev does not point to a valid object!
Updating jose
error: refs/remotes/steph/beta_gc_dev does not point to a valid object!
error: refs/remotes/steph/beta_veh_dev does not point to a valid object!
### many more remotes with the same errors ###

Jeffrey Middleton venit, vidit, dixit 26.10.2009 23:08:

Do you get the same problem if you do the steps individually, i.e.:

git remote update steph git remote prune steph git remote update kevin

Does it depend on the order, i.e.:

git remote update steph git remote update kevin git remote prune steph

Does "git fsck --full" say anything special?

Michael

On Tue, Oct 27, 2009 at 3:42 AM, Michael J Gruber <git@drmicha.warpmail.net> wrote:

I don't *think* I'll see it this way - I was doing essentially this prior to introduction of the --prune option, and never saw it then.

I've tried once and saw no problems.  I just realized I should be
saving off all remote refs before doing a remote update --prune again
- next time I see the problem I should be able to rule out everything
for sure.  Sorry I didn't do that sooner.  However, I'm still fairly
sure it's specific to the all-at-once operation of remote update
--prune since I never saw it before that feature, and because once
that command finishes, everything is okay.

Just 47 dangling blobs, 23 dangling trees, and 13 dangling commits.

On Tue, Oct 27, 2009 at 10:07:27AM -0500, Jeffrey Middleton wrote:

I suspect you are seeing git keeping an invalid cache of the set of refs within a single program. Your original report had:

The "pruned" line basically means that we are deleting that ref, and then we complain immediately about that very ref (which is probably pointing to the null sha1 or something at this point). We tend to cache packed refs, so that may be the cause.

So either we need to invalidate that ref from the cache when it gets deleted, or perhaps we are already invalidating it and we need to be respecting that invalidation in other parts of the code. I'll take a look.

-Peff

On Tue, Oct 27, 2009 at 11:29 AM, Jeff King <peff@peff.net> wrote:

Actually, it force updated two refs, created one, and deleted one, and then immediately complained about the force-updated refs - but only after moving on into the next remote update. My best guess would still be that as you suggest there's an out of date cache - it gets updated within the "update steph" part, but when it moves on to "update kevin" it doesn't have all the new information.

On Tue, Oct 27, 2009 at 12:29:36PM -0400, Jeff King wrote:

Actually, it's beta_gc_dev_old and beta_gc_dev, not the same refs.

Clemens

On Tue, Oct 27, 2009 at 05:40:40PM +0100, Clemens Buchacher wrote:

OK, I'm a bit blind. Thanks for pointing it out. I'll still see if I can replicate it.

-Peff

On Tue, Oct 27, 2009 at 12:50:55PM -0400, Jeff King wrote:

I'm not having any luck reproducing here with a simple case. :( I'm trying:

-- >8 -- #!/bin/sh

rm -rf parent* child

commit() {
  echo $1 >file && git add file && git commit -m $1
}

mkdir parent1 && (
  cd parent1 &&
  git init &&
  commit one &&
  commit two &&
  git checkout -b other
) &&
mkdir parent2 && (
  cd parent2 &&
  git init &&
  commit three
) &&
mkdir child && (
  cd child &&
  git init &&
  git remote add parent1 ../parent1 &&
  git remote add parent2 ../parent2 &&
  git remote update
) &&
(
  cd parent1 &&
  git checkout master &&
  git branch -d other &&
  git reset --hard HEAD^ &&
  commit four
) && (
  cd child &&
  git remote update --prune
)

On 2009.10.27 14:46:27 -0400, Jeff King wrote:

You need more parent repos, and you need packs. What I guess what happens is:

The first remote gets fetched from. The first remote gets pruned. This initializes the pack stuff. The second remote gets fetched from, and creates a _new_ _pack_. The second remote gets pruned (*) The third remote gets fetched from. The third remote gets pruned.

Now, this prune step calls for_each_ref, which looks through the new refs for the second remote. This fails, because the new pack was not present when the pack "system" got initialzed.

(*) I have no idea why this step doesn't fail... But hey, I have no idea about the whole pack handling either. I just recall that there's some initialization step...

This reproduces the issue here:

#!/bin/sh

rm -rf parent* child

commit() {
  echo $1 >file && git add file && git commit -m $1
}

mkdir parent0 && (
  cd parent0 &&
  git init &&
  commit one
) &&
mkdir parent1 && (
  cd parent1 &&
  git init &&
  commit two &&
  commit three &&
  git checkout -b other
) &&
mkdir parent2 && (
  cd parent2 &&
  git init &&
  commit four
) &&
mkdir child && (
  cd child &&
  git init &&
  git config fetch.unpackLimit 1
  git remote add parent0 file://$PWD/../parent0 &&
  git remote add parent1 file://$PWD/../parent1 &&
  git remote add parent2 file://$PWD/../parent2 &&
  git remote update
) && (
  cd parent1 &&
  git checkout master &&
  git branch -d other &&
  git reset --hard HEAD^ &&
  commit five
) && (
  cd child &&
  git remote update --prune
)