threads / discuss / 62055

`git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

Subject: `git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

## tl;dr

8 messages between Sep 4, 2024 and Sep 4, 2024.

replies: 7people: 3as markdown or json

Han Jiang· Sep 4, 2024, 04:09 UTC · lore

Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

cd '/'; cd '/'; rm --force --recursive -- './test_git'; mkdir "$_"; cd "$_"; mkdir --parents -- './server' './client'; git -C './server' init --bare './repo.git' git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty -m "$((++number))" git -C './server/repo.git' branch branch1; git -C './server/repo.git' branch branch2

git -C './client' init './repo' git -C './client/repo' remote add server 'file://'"$(realpath './server/repo.git')" git -C './client/repo' remote --verbose git -C './client/repo' config list --local --show-scope --show-origin git -C './client/repo' fetch --dry-run git -C './client/repo' fetch --dry-run --refmap='+refs/heads/*:refs/remotes/server/*' server branch1 branch2 git -C './client/repo' fetch --dry-run --refmap='+refs/heads/branch*:refs/remotes/server/branch*' server branch1 branch2 git -C './client/repo' fetch --dry-run --refmap='+refs/heads/branch1:refs/remotes/server/branch1' server branch1 branch2 git -C './client/repo' fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1' server branch1 branch2 git -C './client/repo' branch --list --all --verbose --verbose

What did you expect to happen? (Expected behavior)

In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1' server branch1 branch2`, `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` merges with `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`, so these should be what would be done: branch1 -> server/branch1 branch2 -> server/branch2

What happened instead? (Actual behavior)

In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1' server branch1 branch2`, `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`, so these are what would be done: branch1 -> server/branch1

What's different between what you expected and what actually happened?
Anything else you want to add:

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

[System Info]
git version:
git version 2.46.0.windows.1
cpu: x86_64
built from commit: 2e6a859ffc0471f60f79c1256f766042b0d5d17d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.9.0
OpenSSL: OpenSSL 3.2.2 4 Jun 2024
zlib: 1.3.1
uname: Windows 10.0 22631
compiler info: gnuc: 14.1
libc info: no libc information available
$SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe

[Enabled Hooks] not run from a git repository - no hooks to show

Johannes Sixt· Sep 4, 2024, 06:15 UTC · re: Han Jiang · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

Am 04.09.24 um 06:09 schrieb Han Jiang:
Show 6 quoted lines
> What did you expect to happen? (Expected behavior)
> 
> In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> server branch1 branch2`,
> `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` merges with
> `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,

The way I read the documentation of --refmap, I cannot infer any sort of merging of refspec, only that a replacement occurs.

Show 12 quoted lines
> so these should be what would be done:
> branch1 -> server/branch1
> branch2 -> server/branch2
> 
> What happened instead? (Actual behavior)
> 
> In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> server branch1 branch2`,
> `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> so these are what would be done:
> branch1 -> server/branch1

So, this seems the correct output, because it is no longer specified how branch2 should be stored locally.

-- Hannes
Han Jiang· Sep 4, 2024, 11:12 UTC · re: Johannes Sixt · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

1. If the doc says "providing an empty <refspec> to the --refmap
option causes Git to ignore the configured refspecs and rely entirely
on the refspecs supplied as command-line arguments" then it's
reasonable to guess that "providing a non-empty <refspec> will not do
those". (Otherwise the doc could just say "providing a <refspec> will
do those" without the word "empty".) This is both logical fallacy and
common sense.
2. Searching "empty string" "empty value" in
https://git-scm.com/docs/git-config shows that using empty value to
clear lower priority configurations (and non-empty values just stack
up) is common behavior of multi-valued configuration variables.
On Wed, Sep 4, 2024 at 6:15 PM Johannes Sixt <j6t@kdbg.org> wrote:
Show 30 quoted lines
>
> Am 04.09.24 um 06:09 schrieb Han Jiang:
> > What did you expect to happen? (Expected behavior)
> >
> > In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> > server branch1 branch2`,
> > `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` merges with
> > `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
>
> The way I read the documentation of --refmap, I cannot infer any sort of
> merging of refspec, only that a replacement occurs.
>
> > so these should be what would be done:
> > branch1 -> server/branch1
> > branch2 -> server/branch2
> >
> > What happened instead? (Actual behavior)
> >
> > In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> > server branch1 branch2`,
> > `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> > by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> > so these are what would be done:
> > branch1 -> server/branch1
>
> So, this seems the correct output, because it is no longer specified how
> branch2 should be stored locally.
>
> -- Hannes
>
Johannes Sixt· Sep 4, 2024, 16:38 UTC · re: Han Jiang · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

Am 04.09.24 um 13:12 schrieb Han Jiang:
Show 5 quoted lines
> 1. If the doc says "providing an empty <refspec> to the --refmap
> option causes Git to ignore the configured refspecs and rely entirely
> on the refspecs supplied as command-line arguments" then it's
> reasonable to guess that "providing a non-empty <refspec> will not do
> those".

Fair enough. But then in section "Configured Remote-tracking Branches" it is said that "The latter use of the remote.<repository>.fetch values can be overridden by giving the --refmap=<refspec> parameter(s) on the command line." So... I dunno. Just my 2c.

-- Hannes
Junio C Hamano· Sep 4, 2024, 16:47 UTC · re: Johannes Sixt · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

Johannes Sixt <j6t@kdbg.org> writes:
Show 11 quoted lines
> Am 04.09.24 um 13:12 schrieb Han Jiang:
>> 1. If the doc says "providing an empty <refspec> to the --refmap
>> option causes Git to ignore the configured refspecs and rely entirely
>> on the refspecs supplied as command-line arguments" then it's
>> reasonable to guess that "providing a non-empty <refspec> will not do
>> those".
>
> Fair enough. But then in section "Configured Remote-tracking Branches"
> it is said that "The latter use of the remote.<repository>.fetch values
> can be overridden by giving the --refmap=<refspec> parameter(s) on the
> command line." So... I dunno. Just my 2c.
This cuts both ways, I suspect.  It is so unusual to say
    git fetch --refmap="" foo bar baz

to use an empty string as <refspec> in "--refmap=<refspec>", simply because you would not say

    git fetch origin ""
and expect the empty string is a <refspec> in "git fetch origin <refspec>".
Any string other than an empty string, e.g. 
    git fetch --refmap="refs/heads/*:refs/prefetch/x/*" foo bar baz

would have an extra side effect of storing what gets fetched, when your primary focus is "I want the command to ignore the configured refspec".

For these reasons, I would actually say it is quite fair for the documentation to single out the empty string case as an idiom that may be hard for readers to figure out for themselves without being told.

I would not be opposed to add a sentence or two to clarify what happens when a non-empty <refspec> is used, though.

Thanks.
Han Jiang· Sep 4, 2024, 23:52 UTC · re: Junio C Hamano · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

> But then in section "Configured Remote-tracking Branches" it is said that

The problem with the doc is "how to override". From https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---refmapltrefspecgt it seems we need to `git fetch --refmap='' --refmap=<refspec>` to really override configuration variables. I guess the inner working is like this: first for-loop traverses configuration variable `remote.<remote>.fetch` collecting <refspec>s into a list, then second for-loop traverses `--refmap=<refspec>`s, clears the list when an empty string is encountered, appends to the list when a non-empty string is encountered. Maybe I'm just daydreaming...

> git fetch --refmap="" foo bar baz

I think the correct use case of empty string, is to specify more --refmap=<refspec>s after it (like in https://git-scm.com/docs/git-config#Documentation/git-config.txt-pushpushOption ), otherwise the mapping (to operate upon foo,bar and baz) would be empty.

Sorry that I didn't use examples to express my intended use scenarios clearly!

After I searched "empty value" or "empty string" in https://git-scm.com/docs/git-config yesterday, the most intuitive behavior of `git fetch --refmap` (for me) was: `git fetch <remote> <branch>...` uses `remote.<remote>.fetch`; `git fetch --refmap=<refspec> --refmap=<refspec> <remote> <branch>...` uses `remote.<remote>.fetch` and all the `--refmap=<refspec>`s; `git fetch --refmap='' --refmap=<refspec> --refmap=<refspec> <remote> <branch>...` uses all the `--refmap=<refspec>`s;

Here is an example of all of these being used at the same time.
1. Normally, user uses `fetch-safe` which uses `git fetch
--refmap=<refspec>` to use whatever mappings specified by
`remote.<remote>.fetch` and also let our backup logic sneak in.
2. Occasionally, upstream is rebased and our backup logic (refspec
without `+`) blocks the process because of `--atomic`.
2.1. When that happens, user uses `fetch-bak-current-time` which uses
`git fetch --refmap='' --refmap=<refspec>` to clear whatever mappings
specified by `remote.<remote>.fetch` and use our another backup logic
(refspec with `+`).
2.2. After old tips are saved, user uses `git fetch --force` to use
whatever mappings specified by `remote.<remote>.fetch`;

Currently [1.] is not working because --refmap=<not empty string!> clears `remote.<remote>.fetch` instead of merging with it. Hence this bug report.

cd '/'; cd '/'; rm --force --recursive -- './test_git2'; mkdir "$_"; cd "$_"; mkdir --parents -- './server' './client';

git -C './server' init --bare './repo.git' git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty -m "$((++number))" git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty -m "$((++number))" git -C './server/repo.git' branch branch1; git -C './server/repo.git' branch branch2

git -C './client' init './repo' git -C './client/repo' config set --local 'alias.fetch-safe' '!git_fetch_safe(){ server="$1"; shift; git fetch --atomic --refmap='\''refs/*:refs/namespaces/'\''"$server"'\''_backup/refs/*'\'' "$server" "$@"; };git_fetch_safe' git -C './client/repo' config set --local 'alias.fetch-bak-current-time' '!git_fetch_bak_current_time(){ server="$1"; shift; git -c '\''remote.'\''"$server"'\''_backup.url=.'\'' --namespace="$server"'\''_backup'\'' fetch --refmap='\''refs/*:refs/namespaces/'\''"$server"'\''_backup_'\''"$(date +'%Y%m%d%H%M%S')"'\''/refs/*'\'' "$server"'_backup' "$@"; git fetch --atomic --refmap='\'\'' --refmap='\''+refs/*:refs/namespaces/'\''"$server"'\''_backup/refs/*'\'' "$server" "$@"; };git_fetch_bak_current_time' git -C './client/repo' remote add server 'file://'"$(realpath './server/repo.git')" git -C './client/repo' remote --verbose git -C './client/repo' config list --local --show-scope --show-origin git -C './client/repo' fetch-safe server branch1 branch2 git -C './client/repo' show-ref git -C './client/repo' lg --all --decorate-refs='*'

git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty -m "$((++number))" git -C './server/repo.git' branch --force branch1; git -C './server/repo.git' branch --force branch2 git -C './client/repo' fetch-safe server branch1 branch2 git -C './client/repo' show-ref git -C './client/repo' lg --all --decorate-refs='*'

git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty -m "$((++number))" --amend git -C './server/repo.git' branch --force branch1; git -C './server/repo.git' branch --force branch2 git -C './client/repo' fetch-safe server branch1 branch2 git -C './client/repo' show-ref git -C './client/repo' lg --all --decorate-refs='*' git -C './client/repo' fetch-bak-current-time server branch1 branch2 git -C './client/repo' show-ref git -C './client/repo' lg --all --decorate-refs='*' git -C './client/repo' fetch --force server branch1 branch2 git -C './client/repo' show-ref git -C './client/repo' lg --all --decorate-refs='*'

On Thu, Sep 5, 2024 at 4:47 AM Junio C Hamano <gitster@pobox.com> wrote:
Show 44 quoted lines
>
> Johannes Sixt <j6t@kdbg.org> writes:
>
> > Am 04.09.24 um 13:12 schrieb Han Jiang:
> >> 1. If the doc says "providing an empty <refspec> to the --refmap
> >> option causes Git to ignore the configured refspecs and rely entirely
> >> on the refspecs supplied as command-line arguments" then it's
> >> reasonable to guess that "providing a non-empty <refspec> will not do
> >> those".
> >
> > Fair enough. But then in section "Configured Remote-tracking Branches"
> > it is said that "The latter use of the remote.<repository>.fetch values
> > can be overridden by giving the --refmap=<refspec> parameter(s) on the
> > command line." So... I dunno. Just my 2c.
>
> This cuts both ways, I suspect.  It is so unusual to say
>
>     git fetch --refmap="" foo bar baz
>
> to use an empty string as <refspec> in "--refmap=<refspec>", simply
> because you would not say
>
>     git fetch origin ""
>
> and expect the empty string is a <refspec> in "git fetch origin <refspec>".
>
> Any string other than an empty string, e.g.
>
>     git fetch --refmap="refs/heads/*:refs/prefetch/x/*" foo bar baz
>
> would have an extra side effect of storing what gets fetched, when
> your primary focus is "I want the command to ignore the configured
> refspec".
>
> For these reasons, I would actually say it is quite fair for the
> documentation to single out the empty string case as an idiom that
> may be hard for readers to figure out for themselves without being
> told.
>
> I would not be opposed to add a sentence or two to clarify what
> happens when a non-empty <refspec> is used, though.
>
> Thanks.
>
Junio C Hamano· Sep 4, 2024, 14:34 UTC · re: Han Jiang · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

Han Jiang <jhcarl0814@gmail.com> writes:
Show 8 quoted lines
> What happened instead? (Actual behavior)
>
> In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> server branch1 branch2`,
> `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> so these are what would be done:
> branch1 -> server/branch1
This is totally expected.

A refspec specifies "what to fetch" and "where to store locally what we fetched" at the same time. A refmap only specifies the "where the stuff is stored locally" part, and what to fetch is specified separately. You asked branch1 and branch2 to be fetched (peek into the .git/FETCH_HEAD file to see both objects are fetched) from the command line arguments, and your refmap specified where to store what you got from their refs/heads/branch1. As the refmap does not say anything about any other branches, they are not stored anywhere in your local refs/ namespace.

Typically --refmap is given with a glob, but the story is the same there. If you fetched refs/notes/commits from that same server with a refmap to map "refs/heads/*:refs/remotes/server/*", the notes commit is not mapped to anywhere with the refmap, so nothing will happen to it.

Han Jiang· Sep 4, 2024, 23:41 UTC · re: Junio C Hamano · lore

Re: `git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

@Junio C Hamano Thank you for the explanation! As you said, "refmap only specifies the where the stuff is stored locally" (what to fetch is specified by last command line arguments), but the issue with this report is a bit complicated because it's about whether "only refmap specifies the where the stuff is stored locally". After I search "empty value" or "empty string" in https://git-scm.com/docs/git-config , the most intuitive behavior of `git fetch --refmap` (for me) is: `git fetch <remote> <branch>...` uses `remote.<remote>.fetch`; `git fetch --refmap=<refspec> --refmap=<refspec> <remote> <branch>...` uses `remote.<remote>.fetch` and all the `--refmap=<refspec>`s; `git fetch --refmap='' --refmap=<refspec> --refmap=<refspec> <remote> <branch>...` uses all the `--refmap=<refspec>`s; I'll post an example of all of these being used at the same time to both of you.

On Thu, Sep 5, 2024 at 2:34 AM Junio C Hamano <gitster@pobox.com> wrote:
Show 31 quoted lines
>
> Han Jiang <jhcarl0814@gmail.com> writes:
>
> > What happened instead? (Actual behavior)
> >
> > In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> > server branch1 branch2`,
> > `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> > by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> > so these are what would be done:
> > branch1 -> server/branch1
>
> This is totally expected.
>
> A refspec specifies "what to fetch" and "where to store locally what
> we fetched" at the same time.  A refmap only specifies the "where
> the stuff is stored locally" part, and what to fetch is specified
> separately.  You asked branch1 and branch2 to be fetched (peek into
> the .git/FETCH_HEAD file to see both objects are fetched) from the
> command line arguments, and your refmap specified where to store
> what you got from their refs/heads/branch1.  As the refmap does not
> say anything about any other branches, they are not stored anywhere
> in your local refs/ namespace.
>
> Typically --refmap is given with a glob, but the story is the same
> there.  If you fetched refs/notes/commits from that same server with
> a refmap to map "refs/heads/*:refs/remotes/server/*", the notes commit
> is not mapped to anywhere with the refmap, so nothing will happen to
> it.
>
>

← back to recent threads