threads / discuss / 57974

Best way to update `HEAD` in mirrored repos

Subject: Best way to update `HEAD` in mirrored repos

## tl;dr

9 messages between Jun 8, 2022 and Jun 15, 2022.

replies: 8people: 3as markdown or json

Rodrigo Silva Mendoza· Jun 8, 2022, 20:36 UTC · lore
Hi all,

Given a repo cloned like so: `git clone --mirror someRemote`

And then with its origin updated like so `git remote set-url origin someRemote`

What would be the best way to update the `HEAD` ref in the clone, if some change happens to the remote HEAD? Like say, the default branch changing from `main` to `dev` or being renamed from `main` to `main_2`.

What I've got that I think works
1. Get the ref from the origin that points to `HEAD`. Extract the ref
from the output.
2. Manually update the `HEAD` ref with the extracted output prior step

Like so: ``` $ git ls-remote --symref origin HEAD ref: refs/heads/good_main_3 HEAD 0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821 HEAD

$ git symbolic-ref HEAD refs/heads/good_main_3 ```

Does this make sense?
The following are some of the things I've tried - they all fail to
update the `HEAD` file, or fail for some other reason.
`git fetch -p`
`git fetch -p -a`
`get fetch -p -a -u`
`git remote set-head origin -a`
   Fails with "error: Not a valid ref:
refs/remotes/origin/good_main_3". The ref it fails with is whatever
the remote HEAD is.
`git remote update --prune`

And my use case: I have a set of git repos cloned/mirrored. They are then indexed for search, using the `HEAD` ref. The indexer runs periodically, and before indexing it always runs `get fetch -p` to update repos. For each repo, to get the latest commit to index, I use the libgit2 equivalent of `git rev-parse HEAD^0`. Right now, previously cloned repos whose HEAD is updated after the initial clone will fail at this step, which causes them to not get indexed. Here's an example run, all for the same repo `x`.

1. `git clone & set origin for repo x` (indexer first clones repo if
it doesn't exist)
2. `git rev-parse HEAD^0` - successfully outputs some commit sha
3. Github default branch is renamed from `main` to `main_3`
4. `git fetch -p` - indexer runs this to update
5. `git rev-parse HEAD^0` - now outputs "fatal: ambiguous argument
'HEAD^0': unknown revision or path not in the working tree."
Thanks for your time,
Rodrigo
Junio C Hamano· Jun 8, 2022, 22:53 UTC · re: Rodrigo Silva Mendoza · lore

Re: Best way to update `HEAD` in mirrored repos

Rodrigo Silva Mendoza <rodrigosilvamendoza3@gmail.com> writes:
Show 15 quoted lines
> What I've got that I think works
> 1. Get the ref from the origin that points to `HEAD`. Extract the ref
> from the output.
> 2. Manually update the `HEAD` ref with the extracted output prior step
>
> Like so:
> ```
> $ git ls-remote --symref origin HEAD
> ref: refs/heads/good_main_3     HEAD
> 0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD
>
> $ git symbolic-ref HEAD refs/heads/good_main_3
> ```
>
> Does this make sense?
Yes.  That soudns like the right thing to do.
Rodrigo Silva Mendoza· Jun 9, 2022, 05:01 UTC · re: Junio C Hamano · lore

Re: Best way to update `HEAD` in mirrored repos

Great, thanks for the confirmation. Appreciate it!
Rodrigo
On Wed, Jun 8, 2022 at 3:54 PM Junio C Hamano <gitster@pobox.com> wrote:
Show 21 quoted lines
>
> Rodrigo Silva Mendoza <rodrigosilvamendoza3@gmail.com> writes:
>
> > What I've got that I think works
> > 1. Get the ref from the origin that points to `HEAD`. Extract the ref
> > from the output.
> > 2. Manually update the `HEAD` ref with the extracted output prior step
> >
> > Like so:
> > ```
> > $ git ls-remote --symref origin HEAD
> > ref: refs/heads/good_main_3     HEAD
> > 0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD
> >
> > $ git symbolic-ref HEAD refs/heads/good_main_3
> > ```
> >
> > Does this make sense?
>
> Yes.  That soudns like the right thing to do.
>
Johannes Schindelin· Jun 9, 2022, 08:02 UTC · re: Rodrigo Silva Mendoza · lore

Re: Best way to update `HEAD` in mirrored repos

Hi Rodrigo,
On Wed, 8 Jun 2022, Rodrigo Silva Mendoza wrote:
> `git remote set-head origin -a`
>    Fails with "error: Not a valid ref:
> refs/remotes/origin/good_main_3". The ref it fails with is whatever
> the remote HEAD is.

The reason is that `set-head` expects options to come before arguments, like so:

	git remote set-head -a origin

Ciao, Johannes

Rodrigo Silva Mendoza· Jun 9, 2022, 14:46 UTC · re: Johannes Schindelin · lore

Re: Best way to update `HEAD` in mirrored repos

Hi Johannes,

On Thu, Jun 9, 2022 at 1:02 AM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 9 quoted lines
>
>
> The reason is that `set-head` expects options to come before arguments,
> like so:
>
>         git remote set-head -a origin
>
> Ciao,
> Johannes
Hmm, that doesn't seem to work for me either - I get the same type of
error as before.
Here's a minimal repro:
    $ git clone git@github.com:git/gitscm-old.git --mirror
    $ cd gitscm-old.git
    $ git remote set-head -a origin
      error: Not a valid ref: refs/remotes/origin/master
Git version: git version 2.32.1 (Apple Git-133)
Let me know if I'm missing anything.

Cheers, Rodrigo

Johannes Schindelin· Jun 10, 2022, 21:46 UTC · re: Rodrigo Silva Mendoza · lore

Re: Best way to update `HEAD` in mirrored repos

Hi Rodrigo,
On Thu, 9 Jun 2022, Rodrigo Silva Mendoza wrote:
Show 12 quoted lines
> On Thu, Jun 9, 2022 at 1:02 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > The reason is that `set-head` expects options to come before arguments,
> > like so:
> >
> >         git remote set-head -a origin
>
> Hmm, that doesn't seem to work for me either - I get the same type of
> error as before.
> Here's a minimal repro:
>     $ git clone git@github.com:git/gitscm-old.git --mirror

Here, you set this up as a fetch mirror. That is something I've missed before. It is also important because it means that there is no `refs/remotes/origin/HEAD`.

>     $ cd gitscm-old.git
>     $ git remote set-head -a origin
>       error: Not a valid ref: refs/remotes/origin/master

... yet that's exactly what `set-head` assumes. The following patch fixes it for me:

-- snip --
diff --git a/builtin/remote.c b/builtin/remote.c
index eddd40c8f87..fead15adb97 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -1344,7 +1344,7 @@ static int show(int argc, const char **argv)

 static int set_head(int argc, const char **argv)
 {
-	int i, opt_a = 0, opt_d = 0, result = 0;
+	int i, opt_a = 0, opt_d = 0, result = 0, is_mirror = 0;
 	struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
 	char *head_name = NULL;

@@ -1357,8 +1357,16 @@ static int set_head(int argc, const char **argv)
 	};
 	argc = parse_options(argc, argv, NULL, options, builtin_remote_sethead_usage,
 			     0);
-	if (argc)
-		strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
+	if (argc) {
+		struct remote *remote = remote_get(argv[0]);
+
+		if (!remote || !remote->mirror)
+			strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
+		else {
+			is_mirror = 1;
+			strbuf_addstr(&buf, "HEAD");
+		}
+	}

 	if (!opt_a && !opt_d && argc == 2) {
 		head_name = xstrdup(argv[1]);
@@ -1383,7 +1391,10 @@ static int set_head(int argc, const char **argv)
 		usage_with_options(builtin_remote_sethead_usage, options);

 	if (head_name) {
-		strbuf_addf(&buf2, "refs/remotes/%s/%s", argv[0], head_name);
+		if (is_mirror)
+			strbuf_addf(&buf2, "refs/heads/%s", head_name);
+		else
+			strbuf_addf(&buf2, "refs/remotes/%s/%s", argv[0], head_name);
 		/* make sure it's valid */
 		if (!ref_exists(buf2.buf))
 			result |= error(_("Not a valid ref: %s"), buf2.buf);

-- snap --

Would you have a chance to build Git with this patch and verify that it
works for you, too?

Ciao,
Johannes
Junio C Hamano· Jun 10, 2022, 21:53 UTC · re: Johannes Schindelin · lore

Re: Best way to update `HEAD` in mirrored repos

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 29 quoted lines
> diff --git a/builtin/remote.c b/builtin/remote.c
> index eddd40c8f87..fead15adb97 100644
> --- a/builtin/remote.c
> +++ b/builtin/remote.c
> @@ -1344,7 +1344,7 @@ static int show(int argc, const char **argv)
>
>  static int set_head(int argc, const char **argv)
>  {
> -	int i, opt_a = 0, opt_d = 0, result = 0;
> +	int i, opt_a = 0, opt_d = 0, result = 0, is_mirror = 0;
>  	struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
>  	char *head_name = NULL;
>
> @@ -1357,8 +1357,16 @@ static int set_head(int argc, const char **argv)
>  	};
>  	argc = parse_options(argc, argv, NULL, options, builtin_remote_sethead_usage,
>  			     0);
> -	if (argc)
> -		strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +	if (argc) {
> +		struct remote *remote = remote_get(argv[0]);
> +
> +		if (!remote || !remote->mirror)
> +			strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +		else {
> +			is_mirror = 1;
> +			strbuf_addstr(&buf, "HEAD");
> +		}
> +	}
Good.

Regardless of the "mirror" issue, it makes tons of sense to ask the remote API how the remote-tracking refs for the given remote is set up, instead of assuming that it must be "refs/remotes/<remote>" blindly like in the original code.

That way, we could even handle a case like so:
	[remote "frotz"]
		fetch = +refs/heads/*:refs/remotes/nitfol/*

Their HEAD should be mapped to refs/remotes/nitfol/HEAD on our end, so set-head should be able to notice that, too, if we go further with your "do not assume, instead ask remote API" approach.

Thanks.
Rodrigo Silva Mendoza· Jun 15, 2022, 04:11 UTC · re: Junio C Hamano · lore

Re: Best way to update `HEAD` in mirrored repos

Hello,

Apologies for the delay, I'd never built Git from source so I put it off until I had a chunk of time, assuming it might take me a bit - thankfully it was painless.

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> Here, you set this up as a fetch mirror. That is something I've missed
> before. It is also important because it means that there is no
> `refs/remotes/origin/HEAD`.

Ahhh, the reason it wasn't working before clicked for me now, thanks for the explanation.

Show 29 quoted lines
> diff --git a/builtin/remote.c b/builtin/remote.c
> index eddd40c8f87..fead15adb97 100644
> --- a/builtin/remote.c
> +++ b/builtin/remote.c
> @@ -1344,7 +1344,7 @@ static int show(int argc, const char **argv)
>
>  static int set_head(int argc, const char **argv)
>  {
> -     int i, opt_a = 0, opt_d = 0, result = 0;
> +     int i, opt_a = 0, opt_d = 0, result = 0, is_mirror = 0;
>       struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
>       char *head_name = NULL;
>
> @@ -1357,8 +1357,16 @@ static int set_head(int argc, const char **argv)
>       };
>       argc = parse_options(argc, argv, NULL, options, builtin_remote_sethead_usage,
>                            0);
> -     if (argc)
> -             strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +     if (argc) {
> +             struct remote *remote = remote_get(argv[0]);
> +
> +             if (!remote || !remote->mirror)
> +                     strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +             else {
> +                     is_mirror = 1;
> +                     strbuf_addstr(&buf, "HEAD");
> +             }
> +     }
> Would you have a chance to build Git with this patch and verify that it
> works for you, too?

Good news Johannes, your patch worked for me as well! Here's a run with the patched code.

    $ git --version
    git version 2.37.0.rc0.dirty
    $ git clone git@github.com:xvandish/livegrep-fragment.git --mirror
    $ cd livegrep-fragment
    $ git symbolic-ref HEAD
    refs/heads/good_main_5
    $ // I changed the github default branch name at this moment
    $ git ls-remote --symref origin HEAD
    ref: refs/heads/good_main_5     HEAD
    0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD
    $ git fetch -p
    From github.com:xvandish/livegrep-fragment
    - [deleted]         (none)     -> good_main_4
    * [new branch]      good_main_5 -> good_main_5
    $ git remote set-head -a origin
    origin/HEAD set to good_main_5
    $ git symbolic-ref HEAD
    refs/heads/good_main_5
On Fri, Jun 10, 2022 at 2:53 PM Junio C Hamano <gitster@pobox.com> wrote:
Show 17 quoted lines
> Good.
>
> Regardless of the "mirror" issue, it makes tons of sense to ask the
> remote API how the remote-tracking refs for the given remote is set
> up, instead of assuming that it must be "refs/remotes/<remote>"
> blindly like in the original code.
>
> That way, we could even handle a case like so:
>
>         [remote "frotz"]
>                 fetch = +refs/heads/*:refs/remotes/nitfol/*
>
> Their HEAD should be mapped to refs/remotes/nitfol/HEAD on our end,
> so set-head should be able to notice that, too, if we go further
> with your "do not assume, instead ask remote API" approach.
>
> Thanks.

This makes sense to me as well, as a continuation of the "ask remote " style. I only have a very vague idea of how I'd implement this from a look at the codebase, so I'd be of more help testing things out. If this idea carries forward let me know if I can be of help testing, or if you'd like me to like me to try to put a patch up let me know.

Thanks to both of you for your time.
Rodrigo Silva Mendoza· Jun 15, 2022, 04:18 UTC · re: Rodrigo Silva Mendoza · lore

Re: Best way to update `HEAD` in mirrored repos

On Tue, Jun 14, 2022 at 9:11 PM Rodrigo Silva Mendoza <rodrigosilvamendoza3@gmail.com> wrote:

Show 9 quoted lines
> Good news Johannes, your patch worked for me as well! Here's a run
> with the patched code.
>
>     $ git --version
>     git version 2.37.0.rc0.dirty
>     $ git clone git@github.com:xvandish/livegrep-fragment.git --mirror
>     $ cd livegrep-fragment
>     $ git symbolic-ref HEAD
>     refs/heads/good_main_5
....

Whoops, botched the output of the first `git symbolic-ref HEAD` call. It should be:

    $ git symbolic-ref HEAD
    refs/heads/good_main_4

Cheers, Rodrigo

← back to recent threads