{"thread":{"id":"21364","subject":"git update --prune issue","startedAt":"2009-10-26T22:08:45Z","lastAt":"2009-10-27T23:30:56Z","messageCount":9,"participants":["Jeffrey Middleton","Michael J Gruber","Jeff King","Clemens Buchacher","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"125954","messageId":"4389ce950910261508p7eca354el77bad07305a80952@mail.gmail.com","threadId":"21364","inReplyTo":null,"subject":"git update --prune issue","fromName":"Jeffrey Middleton","fromEmail":"jefromi@gmail.com","sentAt":"2009-10-26T22:08:45Z","receivedAt":"2009-10-26T22:08:45Z","isPatch":false,"sender":{"key":"jefromi@gmail.com","avatar":null},"body":"I have an unreliable problem when using \"git remote update --prune\".\ngit claims that many refs from a particular remote do not point to a\nvalid object, but only after finishing another update.  I've included\na shortened version of the output from one particular instance of the\nissue.  Note that the errors are printed for every subsequent remote\nas well.  However, after the update is completed, everything is fine.\n\nIt seems to only happen when there are non-fast-forward changes (new\nbranch, forced update, pruned branch).  The issue only happens with\nthis particular remote, which I've tried removing and recreating, and\nis the same type of remote as all my others (another user's\nNFS-mounted home directory).  However, my remotes are all individual\ndevelopers, and this developer is the only one who ever rebases her\nworking branches.  (recloning the repo from origin and setting up my\nconfig and remotes all over again has also had no effect)\n\nUnfortunately, I have been unable to reproduce the problem in any test\nrepos - for example, though a forced update and a pruned branch in the\nproblematic remote along with an update in another remote seems to\nfairly reliably produce the problem in this repo, recreating that\nsituation in another repo doesn't cause any problems.  Sorry for the\nincomplete bug report, but perhaps this will be enough to go on!\n\nThanks,\nJeffrey\n\n\nI've seen the issue in previous versions built from git.git master, as\nwell as v1.6.3.3, but for this particular one:\n\n$ git --version\ngit version 1.6.5.1.61.ge79999\n\n$ git remote update --prune\nUpdating origin\nremote: Counting objects: 42, done.\nremote: Compressing objects: 100% (9/9), done.\nremote: Total 15 (delta 9), reused 12 (delta 6)\nUnpacking objects: 100% (15/15), done.\nFrom /users/cxtfcm/CxTF_DB\n   88b8613..d40f26d  2009.Q4    -> origin/2009.Q4\n   d40f26d..56305b8  dev        -> origin/dev\nUpdating steph\nremote: Counting objects: 299, done.\nremote: Compressing objects: 100% (241/241), done.\nremote: Total 276 (delta 186), reused 19 (delta 6)\nReceiving objects: 100% (276/276), 41.09 KiB | 10 KiB/s, done.\nResolving deltas: 100% (186/186), completed with 17 local objects.\nFrom /users/sdewet/CxTF_DEV/CxTF_DB\n + c2439dd...69cb5c3 beta_gc_dev -> steph/beta_gc_dev  (forced update)\n + fb25173...f0e4963 beta_veh_dev -> steph/beta_veh_dev  (forced update)\n * [new branch]      beta_veh_dev_old -> steph/beta_veh_dev_old\nPruning steph\nURL: /users/sdewet/CxTF_DEV/CxTF_DB/\n * [pruned] steph/beta_gc_dev_old\nUpdating kevin\nerror: refs/remotes/steph/beta_gc_dev does not point to a valid object!\nerror: refs/remotes/steph/beta_veh_dev does not point to a valid object!\nUpdating jose\nerror: refs/remotes/steph/beta_gc_dev does not point to a valid object!\nerror: refs/remotes/steph/beta_veh_dev does not point to a valid object!\n### many more remotes with the same errors ###\n"},{"id":"125995","messageId":"4AE6B28F.9010407@drmicha.warpmail.net","threadId":"21364","inReplyTo":"4389ce950910261508p7eca354el77bad07305a80952@mail.gmail.com","subject":"Re: git update --prune issue","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2009-10-27T08:42:55Z","receivedAt":"2009-10-27T08:42:55Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Jeffrey Middleton venit, vidit, dixit 26.10.2009 23:08:\n> I have an unreliable problem when using \"git remote update --prune\".\n> git claims that many refs from a particular remote do not point to a\n> valid object, but only after finishing another update.  I've included\n> a shortened version of the output from one particular instance of the\n> issue.  Note that the errors are printed for every subsequent remote\n> as well.  However, after the update is completed, everything is fine.\n> \n> It seems to only happen when there are non-fast-forward changes (new\n> branch, forced update, pruned branch).  The issue only happens with\n> this particular remote, which I've tried removing and recreating, and\n> is the same type of remote as all my others (another user's\n> NFS-mounted home directory).  However, my remotes are all individual\n> developers, and this developer is the only one who ever rebases her\n> working branches.  (recloning the repo from origin and setting up my\n> config and remotes all over again has also had no effect)\n> \n> Unfortunately, I have been unable to reproduce the problem in any test\n> repos - for example, though a forced update and a pruned branch in the\n> problematic remote along with an update in another remote seems to\n> fairly reliably produce the problem in this repo, recreating that\n> situation in another repo doesn't cause any problems.  Sorry for the\n> incomplete bug report, but perhaps this will be enough to go on!\n> \n> Thanks,\n> Jeffrey\n> \n> \n> I've seen the issue in previous versions built from git.git master, as\n> well as v1.6.3.3, but for this particular one:\n> \n> $ git --version\n> git version 1.6.5.1.61.ge79999\n> \n> $ git remote update --prune\n> Updating origin\n> remote: Counting objects: 42, done.\n> remote: Compressing objects: 100% (9/9), done.\n> remote: Total 15 (delta 9), reused 12 (delta 6)\n> Unpacking objects: 100% (15/15), done.\n> From /users/cxtfcm/CxTF_DB\n>    88b8613..d40f26d  2009.Q4    -> origin/2009.Q4\n>    d40f26d..56305b8  dev        -> origin/dev\n> Updating steph\n> remote: Counting objects: 299, done.\n> remote: Compressing objects: 100% (241/241), done.\n> remote: Total 276 (delta 186), reused 19 (delta 6)\n> Receiving objects: 100% (276/276), 41.09 KiB | 10 KiB/s, done.\n> Resolving deltas: 100% (186/186), completed with 17 local objects.\n> From /users/sdewet/CxTF_DEV/CxTF_DB\n>  + c2439dd...69cb5c3 beta_gc_dev -> steph/beta_gc_dev  (forced update)\n>  + fb25173...f0e4963 beta_veh_dev -> steph/beta_veh_dev  (forced update)\n>  * [new branch]      beta_veh_dev_old -> steph/beta_veh_dev_old\n> Pruning steph\n> URL: /users/sdewet/CxTF_DEV/CxTF_DB/\n>  * [pruned] steph/beta_gc_dev_old\n> Updating kevin\n> error: refs/remotes/steph/beta_gc_dev does not point to a valid object!\n> error: refs/remotes/steph/beta_veh_dev does not point to a valid object!\n> Updating jose\n> error: refs/remotes/steph/beta_gc_dev does not point to a valid object!\n> error: refs/remotes/steph/beta_veh_dev does not point to a valid object!\n> ### many more remotes with the same errors ###\n\nDo you get the same problem if you do the steps individually, i.e.:\n\ngit remote update steph\ngit remote prune steph\ngit remote update kevin\n\nDoes it depend on the order, i.e.:\n\ngit remote update steph\ngit remote update kevin\ngit remote prune steph\n\nDoes \"git fsck --full\" say anything special?\n\nMichael\n"},{"id":"126026","messageId":"4389ce950910270807o69d51155xb083f34bb31e1dae@mail.gmail.com","threadId":"21364","inReplyTo":"4AE6B28F.9010407@drmicha.warpmail.net","subject":"Re: git update --prune issue","fromName":"Jeffrey Middleton","fromEmail":"jefromi@gmail.com","sentAt":"2009-10-27T15:07:27Z","receivedAt":"2009-10-27T15:07:27Z","isPatch":false,"sender":{"key":"jefromi@gmail.com","avatar":null},"body":"On Tue, Oct 27, 2009 at 3:42 AM, Michael J Gruber\n<git@drmicha.warpmail.net> wrote:\n> Do you get the same problem if you do the steps individually, i.e.:\n>\n> git remote update steph\n> git remote prune steph\n> git remote update kevin\n\nI don't *think* I'll see it this way - I was doing essentially this\nprior to introduction of the --prune option, and never saw it then.\n\n> Does it depend on the order, i.e.:\n>\n> git remote update steph\n> git remote update kevin\n> git remote prune steph\n\nI've tried once and saw no problems.  I just realized I should be\nsaving off all remote refs before doing a remote update --prune again\n- next time I see the problem I should be able to rule out everything\nfor sure.  Sorry I didn't do that sooner.  However, I'm still fairly\nsure it's specific to the all-at-once operation of remote update\n--prune since I never saw it before that feature, and because once\nthat command finishes, everything is okay.\n\n> Does \"git fsck --full\" say anything special?\n\nJust 47 dangling blobs, 23 dangling trees, and 13 dangling commits.\n\n> Michael\n>\n"},{"id":"126032","messageId":"20091027162936.GA8130@sigill.intra.peff.net","threadId":"21364","inReplyTo":"4389ce950910270807o69d51155xb083f34bb31e1dae@mail.gmail.com","subject":"Re: git update --prune issue","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-10-27T16:29:36Z","receivedAt":"2009-10-27T16:29:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 27, 2009 at 10:07:27AM -0500, Jeffrey Middleton wrote:\n\n> > Do you get the same problem if you do the steps individually, i.e.:\n> >\n> > git remote update steph\n> > git remote prune steph\n> > git remote update kevin\n> \n> I don't *think* I'll see it this way - I was doing essentially this\n> prior to introduction of the --prune option, and never saw it then.\n\nI suspect you are seeing git keeping an invalid cache of the set of refs\nwithin a single program. Your original report had:\n\n> Pruning steph\n> URL: /users/sdewet/CxTF_DEV/CxTF_DB/\n>  * [pruned] steph/beta_gc_dev_old\n> Updating kevin\n> error: refs/remotes/steph/beta_gc_dev does not point to a valid object!\n> error: refs/remotes/steph/beta_veh_dev does not point to a valid object!\n\nThe \"pruned\" line basically means that we are deleting that ref, and\nthen we complain immediately about that very ref (which is probably\npointing to the null sha1 or something at this point). We tend to cache\npacked refs, so that may be the cause.\n\nSo either we need to invalidate that ref from the cache when it gets\ndeleted, or perhaps we are already invalidating it and we need to be\nrespecting that invalidation in other parts of the code. I'll take a\nlook.\n\n-Peff\n"},{"id":"126035","messageId":"4389ce950910270939u10fff402l6e9a5333913d7a74@mail.gmail.com","threadId":"21364","inReplyTo":"20091027162936.GA8130@sigill.intra.peff.net","subject":"Re: git update --prune issue","fromName":"Jeffrey Middleton","fromEmail":"jefromi@gmail.com","sentAt":"2009-10-27T16:39:14Z","receivedAt":"2009-10-27T16:39:14Z","isPatch":false,"sender":{"key":"jefromi@gmail.com","avatar":null},"body":"On Tue, Oct 27, 2009 at 11:29 AM, Jeff King <peff@peff.net> wrote:\n> The \"pruned\" line basically means that we are deleting that ref, and\n> then we complain immediately about that very ref (which is probably\n> pointing to the null sha1 or something at this point). We tend to cache\n> packed refs, so that may be the cause.\n\nActually, it force updated two refs, created one, and deleted one, and\nthen immediately complained about the force-updated refs - but only\nafter moving on into the next remote update.  My best guess would\nstill be that as you suggest there's an out of date cache - it gets\nupdated within the \"update steph\" part, but when it moves on to\n\"update kevin\" it doesn't have all the new information.\n\n> So either we need to invalidate that ref from the cache when it gets\n> deleted, or perhaps we are already invalidating it and we need to be\n> respecting that invalidation in other parts of the code. I'll take a\n> look.\n>\n> -Peff\n>\n"},{"id":"126036","messageId":"20091027164040.GB12464@localhost","threadId":"21364","inReplyTo":"20091027162936.GA8130@sigill.intra.peff.net","subject":"Re: git update --prune issue","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2009-10-27T16:40:40Z","receivedAt":"2009-10-27T16:40:40Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Tue, Oct 27, 2009 at 12:29:36PM -0400, Jeff King wrote:\n\n> > Pruning steph\n> > URL: /users/sdewet/CxTF_DEV/CxTF_DB/\n> >  * [pruned] steph/beta_gc_dev_old\n> > Updating kevin\n> > error: refs/remotes/steph/beta_gc_dev does not point to a valid object!\n> > error: refs/remotes/steph/beta_veh_dev does not point to a valid object!\n> \n> The \"pruned\" line basically means that we are deleting that ref, and\n> then we complain immediately about that very ref (which is probably\n> pointing to the null sha1 or something at this point).\n\nActually, it's beta_gc_dev_old and beta_gc_dev, not the same refs.\n\nClemens\n"},{"id":"126037","messageId":"20091027165054.GA8554@sigill.intra.peff.net","threadId":"21364","inReplyTo":"20091027164040.GB12464@localhost","subject":"Re: git update --prune issue","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-10-27T16:50:55Z","receivedAt":"2009-10-27T16:50:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 27, 2009 at 05:40:40PM +0100, Clemens Buchacher wrote:\n\n> On Tue, Oct 27, 2009 at 12:29:36PM -0400, Jeff King wrote:\n> \n> > > Pruning steph\n> > > URL: /users/sdewet/CxTF_DEV/CxTF_DB/\n> > >  * [pruned] steph/beta_gc_dev_old\n> > > Updating kevin\n> > > error: refs/remotes/steph/beta_gc_dev does not point to a valid object!\n> > > error: refs/remotes/steph/beta_veh_dev does not point to a valid object!\n> > \n> > The \"pruned\" line basically means that we are deleting that ref, and\n> > then we complain immediately about that very ref (which is probably\n> > pointing to the null sha1 or something at this point).\n> \n> Actually, it's beta_gc_dev_old and beta_gc_dev, not the same refs.\n\nOK, I'm a bit blind. Thanks for pointing it out. I'll still see if I can\nreplicate it.\n\n-Peff\n"},{"id":"126047","messageId":"20091027184627.GA19292@sigill.intra.peff.net","threadId":"21364","inReplyTo":"20091027165054.GA8554@sigill.intra.peff.net","subject":"Re: git update --prune issue","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-10-27T18:46:27Z","receivedAt":"2009-10-27T18:46:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 27, 2009 at 12:50:55PM -0400, Jeff King wrote:\n\n> > Actually, it's beta_gc_dev_old and beta_gc_dev, not the same refs.\n> \n> OK, I'm a bit blind. Thanks for pointing it out. I'll still see if I can\n> replicate it.\n\nI'm not having any luck reproducing here with a simple case. :( I'm\ntrying:\n\n-- >8 --\n#!/bin/sh\n\nrm -rf parent* child\n\ncommit() {\n  echo $1 >file && git add file && git commit -m $1\n}\n\nmkdir parent1 && (\n  cd parent1 &&\n  git init &&\n  commit one &&\n  commit two &&\n  git checkout -b other\n) &&\nmkdir parent2 && (\n  cd parent2 &&\n  git init &&\n  commit three\n) &&\nmkdir child && (\n  cd child &&\n  git init &&\n  git remote add parent1 ../parent1 &&\n  git remote add parent2 ../parent2 &&\n  git remote update\n) &&\n(\n  cd parent1 &&\n  git checkout master &&\n  git branch -d other &&\n  git reset --hard HEAD^ &&\n  commit four\n) && (\n  cd child &&\n  git remote update --prune\n)\n"},{"id":"126068","messageId":"20091027233056.GA25689@atjola.homenet","threadId":"21364","inReplyTo":"20091027184627.GA19292@sigill.intra.peff.net","subject":"Re: git update --prune issue","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2009-10-27T23:30:56Z","receivedAt":"2009-10-27T23:30:56Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2009.10.27 14:46:27 -0400, Jeff King wrote:\n> On Tue, Oct 27, 2009 at 12:50:55PM -0400, Jeff King wrote:\n> \n> > > Actually, it's beta_gc_dev_old and beta_gc_dev, not the same refs.\n> > \n> > OK, I'm a bit blind. Thanks for pointing it out. I'll still see if I can\n> > replicate it.\n> \n> I'm not having any luck reproducing here with a simple case. :( I'm\n> trying:\n\nYou need more parent repos, and you need packs. What I guess what\nhappens is:\n\nThe first remote gets fetched from.\nThe first remote gets pruned. This initializes the pack stuff.\nThe second remote gets fetched from, and creates a _new_ _pack_.\nThe second remote gets pruned (*)\nThe third remote gets fetched from.\nThe third remote gets pruned.\n\nNow, this prune step calls for_each_ref, which looks through the new\nrefs for the second remote. This fails, because the new pack was not\npresent when the pack \"system\" got initialzed.\n\n(*) I have no idea why this step doesn't fail... But hey, I have no idea\nabout the whole pack handling either. I just recall that there's some\ninitialization step...\n\nThis reproduces the issue here:\n\n#!/bin/sh\n\nrm -rf parent* child\n\ncommit() {\n  echo $1 >file && git add file && git commit -m $1\n}\n\nmkdir parent0 && (\n  cd parent0 &&\n  git init &&\n  commit one\n) &&\nmkdir parent1 && (\n  cd parent1 &&\n  git init &&\n  commit two &&\n  commit three &&\n  git checkout -b other\n) &&\nmkdir parent2 && (\n  cd parent2 &&\n  git init &&\n  commit four\n) &&\nmkdir child && (\n  cd child &&\n  git init &&\n  git config fetch.unpackLimit 1\n  git remote add parent0 file://$PWD/../parent0 &&\n  git remote add parent1 file://$PWD/../parent1 &&\n  git remote add parent2 file://$PWD/../parent2 &&\n  git remote update\n) && (\n  cd parent1 &&\n  git checkout master &&\n  git branch -d other &&\n  git reset --hard HEAD^ &&\n  commit five\n) && (\n  cd child &&\n  git remote update --prune\n)\n"}]}