"Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

So I ran "git fetch --all" inside git.git that you are all familiar with. All fetches failed with a similar error as: Fetching kernelorg fatal: bad object HEAD error: https://kernel.googlesource.com/pub/scm/git/git did not send all necessary objects

error: Could not fetch kernelorg

Working with an older version of Git (2.13.0) gives me

$ GIT_TRACE=/u/trace GIT_TRACE_PACKET=/u/out git fetch git://github.com/git/git
From git://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

After fiddling around and some printf debugging, clean fsck(!) I could not make sense of it, so bisecting

$ git bisect bad
d0c39a49ccb5dfe7feba4325c3374d99ab123c59 is the first bad commit
commit d0c39a49ccb5dfe7feba4325c3374d99ab123c59
Author: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date:   Wed Aug 23 19:36:59 2017 +0700

    revision.c: --all adds HEAD from all worktrees

    Unless single_worktree is set, --all now adds HEAD from all worktrees.

    Since reachable.c code does not use setup_revisions(), we need to call
    other_head_refs_submodule() explicitly there to have the same effect on
    "git prune", so that we won't accidentally delete objects needed by some
    other HEADs.

    A new FIXME is added because we would need something like

        int refs_other_head_refs(struct ref_store *, each_ref_fn, cb_data);

    in addition to other_head_refs() to handle it, which might require

        int get_submodule_worktrees(const char *submodule, int flags);

    It could be a separate topic to reduce the scope of this one.

    Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

And here we go, I do use worktrees nowadays! $ git worktree list /usr/local/google/home/sbeller/OSS/git 117ddefdb4 (detached HEAD) /u/git d0c39a49cc (detached HEAD) /usr/local/google/home/sbeller/OSS/git_origin_master b14f27f917 (detached HEAD) /usr/local/google/home/sbeller/OSS/submodule_remote_dot 3d9025bd69 (detached HEAD)

$ git show 3d9025bd69 fatal: ambiguous argument '3d9025bd69': unknown revision or path not in the working tree.

ok, so I presume I just delete that working tree to "fix my copy of git" sbeller@sbeller:/u/git$ rm -rf /usr/local/google/home/sbeller/OSS/submodule_remote_dot sbeller@sbeller:/u/git$ git worktree prune sbeller@sbeller:/u/git$ git worktree list /usr/local/google/home/sbeller/OSS/git 117ddefdb4 (detached HEAD) /u/git d0c39a49cc (detached HEAD) /usr/local/google/home/sbeller/OSS/git_origin_master b14f27f917 (detached HEAD) $ git bisect reset Previous HEAD position was d0c39a49cc... revision.c: --all adds HEAD from all worktrees HEAD is now at 660fb3dfa8... Sync with maint $ make install $ git fetch --all # works fine!

Any idea which direction we need to heading for a real fix?

On Thu, Oct 19, 2017 at 06:43:57PM -0700, Stefan Beller wrote:

I ran into this a while back with worktrees, too. In my case I had a worktree which was "owned" by multiple repositories[1]. One would write the HEAD, but the other did not necessarily have those objects.

I don't quite recall how I ended up there. Doing "git worktree add" multiple times does seem to complain about the directory already existing. My case did involve relative symlinks in both repositories, so maybe that confused things. Anyway, I can't seem to replicate it now.

So maybe you've ended up in the same situation and somehow "reused" submodule_remote_dot.

But I think until that commit you mentioned (d0c39a49cc), "gc" in the parent repo could drop objects accessible only from the worktree. So if that happened _before_ d0c39a49cc, then at that point your worktree was corrupted. And now of course further operations will complain.

So that's the right immediate workaround (and hoping there was nothing valuable in that worktree!).

If my analysis above is correct, then it's already fixed. You just had leftover corruption.

-Peff

On Thu, Oct 19, 2017 at 8:16 PM, Jeff King <peff@peff.net> wrote:

How would I know? ;)

Without the commit available I do not even know if the code there is well reasoned about. (My commit messages usually tell my future self how much thought I put into the code)

Well fetching yesterday worked and the commit in question is from 8/23, the merge 8a044c7f1d56cef657be342e40de0795d688e882 occurred 9/18, so I suspect there is something else at play. (I do not remember having a gc between yesterday and today. Though maybe one in the background?)

I am curious how you can have a worktree owned by multiple repositories [1] (?).

Thanks, Stefan

On Thu, Oct 19, 2017 at 10:27:28PM -0700, Stefan Beller wrote:

Even a gc between yesterday and today should have used the new code, which would have been safe. So yeah, maybe it is something else entirely.

Sorry, I forgot my footnote. I saw this with my "ci" script:

  https://github.com/peff/git/blob/7905ff395adecdd2bb7ab045a24223dfb103e0e9/ci

I check out the contents of my "meta" branch as "Meta", and it contains that script. It basically just waits for ref updates, then walks over all the commits and runs "make test" on them in the background (caching the results, thanks to the git-test[1] script). So I kick off "Meta/ci" in a terminal and forget about it, and magically it builds my commits in the background as I work.

It operates in a worktree inside the Meta directory (Meta/tmp-ci), so as not to disturb what I'm doing. So far so good.

But I actually have _two_ clones of Git on my system. One on which I do most of my work, and then the other which has the fork we use in production at GitHub. I symlink the Meta directory from the first into the latter, which means they both see the same worktree directory. And somehow running "Meta/ci" in the second corrupted things.

I can get some funniness now, but I think it's mostly caused by the script being confused about the worktree existing but not having access to our branches. That's not a corruption, just a confusion. I _think_ I had a bogus HEAD in the worktree at one point, but I may be mis-remembering. I can't seem to trigger it now.

-Peff

[1] https://github.com/mhagger/git-test

On Thu, Oct 19, 2017 at 11:04 PM, Jeff King <peff@peff.net> wrote:

Oh, yeah.

Thanks for these insights.

I played around with Meta a bit, but I did not feel it would enhance my workflow enough as I am not involved with any maintainance of git using git.

The git-test from Michael sounds intriguing. Initially I put off using it as I had my main working dir (or rather test dir) on a spinning disk, still. Now I test in memory only, which is a lot faster, so I could try if git-test can keep up with my local commit pace.

Thanks, Stefan

On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:

Excuse my ignorance but I don't get your statement correctly. What do you mean by "I test in memory only"? How do you do that?

--- Kaartic

On Sun, Oct 22, 2017 at 07:54:57PM +0530, Kaartic Sivaraam wrote:

I assume it means pointing "--root" at a tmpfs or ram disk in GIT_TEST_OPTS. That makes the test suite run _way_ faster.

-Peff

On Sun, Oct 22, 2017 at 5:36 PM, Jeff King <peff@peff.net> wrote:

Exactly. By memory I mean volatile RAM (as opposed to memory on a spinning disk).

Using GIT_TEST_OPTS has had some issues (I remember vaguely there was an inconsistency between the output of `make test` and prove), so I put my entire working tree on a tmpfs, I run roughly this script after booting my computer:

  sudo mount -t tmpfs -o size=16g tmpfs /u
  mkdir /u/git
  echo "gitdir:
/usr/local/google/home/sbeller/OSS/git/.git/worktrees/git"

  git -C /u/git checkout -f HEAD

  cat <<EOF >/u/config.mak
  DEVELOPER=1
  DEVELOPERS=1
  CFLAGS += -g -O2
  CFLAGS += -DFLEX_ARRAY=2048
  #CFLAGS += -Wno-unused-value
  EOF

The downside (or upside?) is that there will be no files cluttering my git directory, but any untracked file is wiped periodically. So if I want to really preserve a thing I am forcing myself to put it somewhere else than in an untracked file inside the git repo.

The test suite (excluding t9*) runs in less than 50 seconds on the ram disk.

Thanks, Stefan

On Monday 23 October 2017 11:07 PM, Stefan Beller wrote:

Did I thank you for a good explanation? If not, thanks that was interesting and enlightening.

BTW, this is what I call _way way_ faster. Unfortunately due to the limited configuration of my system, the test suite has following timing

     real    3m14.482s
     user    2m10.556s
     sys     1m12.328s

Anyways it's better than the timing I get for running it on a spinning disk which takes

     real    4m37.585s
     user    2m17.244s
     sys     1m37.136s

Thanks, Kaartic

On Fri, Nov 3, 2017 at 2:32 AM, Kaartic Sivaraam <kaartic.sivaraam@gmail.com> wrote:

Just tested again, I meant to say 70s instead of 50s.

This sounds to me as if it is running with just one thread (because sys+user = real); I usually run with

    cd t
    prove --jobs 25 t[0-8][0-9]*.sh

The multithreading can be seen in the timing as well
    real 1m9.913s
    user 1m50.796s
    sys 0m54.092s
as user > real already.

If you run tests via 'make test' or 'cd t && make', you can also give a --jobs <n> to make it faster. I have no good answer for how many, but I have the impression overloading the system is no big deal. (I only have a few cores in this machine, 4 or 6, but still run with --jobs 25; 'git grep sleep' returns a couple of lines, and such threads sleeping definitely don't need a CPU)

Thanks, Stefan

On Fri, 2017-11-03 at 12:11 -0700, Stefan Beller wrote:

I usually use the following command to build and run the test suite from the root of the working directory,

    make NO_GETTEXT=1 DEVELOPER=1 DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS="--timer --jobs 16" test

    I beleived that the DEFAULT_TST_TARGE and GIT_PROVE_OPTS options would
    be seen by the Makefile in t/ but didn't verify it. Your statement made
    me suspicious and I wanted to know whether my assumption was correct or
    not. So, I did this,

    1. I first tried running the above command while seeing how much time
    on the 'next' branch when it was pointing at,

    9a519c715 (Merge branch 'jk/rebase-i-exec-gitdir-fix' into next, 2017-11-02)

    First, I observed that at least part of my assumption held on observing
    that the output was similar to that of 'prove'. This shows that
    t/Makefile recognizes DEFAULT_TEST_TARGET.

    Next, to verify if it recognizes GIT_PROVE_OPTS I ran the above command
    with 'time' and it gave back the following statistics,

    real    	    4m55.707s
    user    	    2m29.924s
    sys    	    1m48.072s

    which is in line with the statistics for the spinning disk in the
    previous mail. (The time doesn't include the build time).

    (real)-(user+sys) = 38s (approx.)

    2. Now I switched to the t/ directory, I tried running the following,

    make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS="--timer --jobs 16"

    'time' gave the statistics for the above command as follows,

    real    	    5m19.523s
    user    	    2m26.764s
    sys    	    1m45.240s

    (real)-(user+sys) =     68s (approx.)
Not a big improvement from the previous case to assure that
GIT_PROVE_OPTS wasn't recognized there.

3. Staying in the t/ directory, I tried running the following just to
be pretty sure that the "--jobs" was indeed sent to 'prove',

    prove --jobs 16 t[0-9][0-9]*.sh

'time' gave the statistics for the above command as follows,

    real    	    4m49.241s
    user    	    2m29.220s
    sys    	    1m47.828s

    (real)-(user+sys) = 34s (approx.)

    This seems to be identical with the first case.

    So, it does seem to be a limitation of my system (Intel i3 with 2
    cores, FWIW).

    4. To see if things improve with a higher number of jobs I tried,

    prove --jobs 25 t[0-9][0-9]*.sh

    'time' gave the statistics for the above command as follows,

    real    	    5m21.229s
    user    	    2m25.704s
    sys    	    1m46.744s

        (real)-(user+sys) = 70s (approx.)

        This ensured the limitation once more!

        Anyways, thanks for the explanation and information which was
        enlightening. :-)

--  Kaartic