SEGV (detected by Address Sanitizer) when using `core.abbrev` option

c8aed5e8dadf (repository: stop setting SHA1 as the default object
hash, 2024-05-07) stopped initializing the_hash_algo, but config.c
references it when it observes a user setting core.abbrev, in two
ways:
- if core.abbrev is detected as a 'no' boolean value, then
default_abbrev is set to the_hash_algo->hexsz
- if core.abbrev is set to an integer, it verifies that it's within
range for the hash algorithm (specifically: it errors out if the value
is < minimum_abbrev or > the_hash_algo->hexsz).

Stack:
==2421488==ERROR: AddressSanitizer: SEGV on unknown address
0x000000000018 (pc 0x56344202585f bp 0x7fff9546fe10 sp 0x7fff9546fcb0
T0)
==2421488==The signal is caused by a READ memory access.
==2421488==Hint: address points to the zero page.
    #0 0x56344202585f in git_default_core_config git/config.c:1466
    #1 0x56344202585f in git_default_config git/config.c:1815
    #2 0x56344202064e in configset_iter git/config.c:2185
    #3 0x563441d531cb in cmd_clone builtin/clone.c:981
    #4 0x563441cebac2 in run_builtin git/git.c:474
    #5 0x563441cebac2 in handle_builtin git/git.c:729
    #6 0x563441ceed0a in run_argv git/git.c:793
    #7 0x563441cf0aea in cmd_main git/git.c:928
    #8 0x563441ce9323 in main git/common-main.c:62
    #9 0x7fa3228456c9 in __libc_start_call_main
../sysdeps/nptl/libc_start_call_main.h:58
    #10 0x7fa322845784 in __libc_start_main_impl ../csu/libc-start.c:360
    #11 0x563441ceb530 in _start (git/git+0x1e0530) (BuildId:
c0e4b09d5b212a201769f1eb8e7592cddbe3af1d)

AddressSanitizer can not provide additional info.

---

My first thought for a fix was to just cap it at 40, with the assumption that the code would handle it correctly in the unlikely event that the hash size ever decreased, but I don't think that does the right thing if `core.abbrev=no`. That's documented as a way of obtaining the full hashes (with no abbreviation), and if we're using sha256, capping that at 40 hex (160bits) is incorrect.

My second thought was that we could store the requested value and validate it on every usage. This complicates usage locations, and can lead to poor behavior (crashes in the middle of operation when we finally get around to checking the value).

My third thought was that we could store the requested value and validate when we have a repository that initializes the hash for us as part of that initialization. If we attempt to abbreviate some hashes without that setup, we act as if core.abbrev isn't set at all (so they get the default behavior). That seems like the best option overall. Exploring that further ...

I've looked at a semi-random collection of places where `default_abbrev` or `DEFAULT_ABBREV` are used, and they all seem to eventually go through `repo_find_unique_abbrev_r`, and they pass in the len. This also always has a repository available (otherwise it wouldn't be able to disambiguate). Furthermore, it has `if (len < 0)`. We could thus carry the "unvalidated" request in default_abbrev by having special magic values (auto=-1 like today, no=-2 (replaced with hexsz when we know it), other requests are <= -4, for a requested length of 4 or higher), or we could have another variable (requested_default_abbrev) that gets copied to default_abbrev when we have a repo.

The one potential issue I can think of with this is that setting `core.abbrev = no`, having that resolve to 64 (sha256) when we set up the repo, and then if we ever read from a repo that uses 40 hexsz (such as sha1), then we have to ensure that we tolerate a requested length greater than the current hash algorithm's maximum length. This likely wasn't a problem when sha1 was the default, because we're unlikely to go to a hash algorithm with <160 bits in the future. But if sha256 becomes the default, then this can be problematic.

On Mon, Jun 10, 2024 at 3:27 PM Kyle Lippincott <spectral@google.com> wrote:

I just realized I didn't give a reproduction command, sorry about that; here's the reproduction command provided by our user:

git config --global core.abbrev 12 git clone https://github.com/git/git.git

I realized this because Josh Steadmon informed me that config loading/parsing is lazy when I asked why `git bisect` still worked with this setting in my git config. So I think this might be a problem only for things that go through `git_default_config` (if we set up the repo object, we probably work just fine).

Kyle Lippincott <spectral@google.com> writes:

Or "git -c core.abbrev=12 clone https://any/thing".

Nicely spotted. And this is a curious one, as you would generally expect that having a fallback setting in the global space would help you instead of hurting you.

On Mon, Jun 10, 2024 at 03:27:11PM -0700, Kyle Lippincott wrote:

Good find.

Exactly, this doesn't feel like the right solution.

This feels more complex than needed indeed.

This also feels a bit too complex in my opinion.

I think this is the best solution. I don't quite see why we need to abort in the first place when the caller asks for something longer than GIT_MAX_HEXSZ. If they do, it's rather clear that the intent is to show the full object hash, so we can do that.

In other words, the following should be sufficient, shouldn't it?

The "core.abbrev" config allows the user to specify the minimum length when abbreviating object hashes. Next to the values "auto" and "no", this config also accepts a concrete length that needs to be bigger or equal to the minimum length and smaller or equal to the hash algorithm's hex length. While the former condition is trivial, the latter depends on the object format used by the current repository. It is thus a variable upper boundary that may either be 40 (SHA-1) or 64 (SHA-256).

This has two major downsides. First, the user that specifies this config must be aware of the object hashes that its repository use. If they want to configure the value globally, then they cannot pick any value in the range `[41, 64]` if they have any repository that uses SHA-1. If they did, Git would error out when parsing the config.

Second, and more importantly, parsing "core.abbrev" crashes when outside of a Git repository because we dereference `the_hash_algo` to figure out its hex length. Starting with c8aed5e8da (repository: stop setting SHA1 as the default object hash, 2024-05-07) though, we stopped initializing `the_hash_algo` outside of Git repositories.

Fix both of these issues by not making it an error anymore when the given length exceeds the hash length. Instead, if we have a repository, then we truncate the length to the maximum length of `the_hash_algo`. Otherwise, we simply leave the abbreviated length intact and store it as-is. This is equivalent to the logic in `parse_opt_abbrev_cb()` and is handled just fine by `repo_find_unique_abbrev_r()`. In practice, we should never even end up using `default_abbrev` without a repository anyway given that abbreviating object IDs to unique prefixes requires us to have access to an object database.

Reported-by: Kyle Lippincott <spectral@google.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 config.c         | 7 +++++--
 t/t4202-log.sh   | 6 ++++++
 t/t5601-clone.sh | 7 +++++++
 3 files changed, 18 insertions(+), 2 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

Makes sense.

We will need to have some code that further adjusts overly long default_abbrev when we really have to abbreviate (at which time, hopefully we are already aware of the real hash algorithm used in the repository, and that may be SHA-1) anyway.

So do we even need the conditional here? Can't we just set it to GIT_MAX_HEXSZ here unconditionally?

Interesting. We didn't have coverage for "we want to see full object names" case.

OK.

On Tue, Jun 11, 2024 at 10:54:23AM -0700, Junio C Hamano wrote:

Not really. I was erring on the safe side here to retain the status quo for all relevant cases. As explained in the commit message, the length is only relevant when we have a repository because we otherwise wouldn't be able to abbreviate anything anyway. So essentially, this change is a functional no-op.

There's also the question of the second commit, where we only handle `abbrev == the_hash_algo->size`, but not `abbrev > the_hash_algo->size`. It works, but is slower when not truncating the length until the second patch fixes it.

So yes, we can set this unconditionally to `GIT_MAX_HEXSZ`, but out of abundance of caution I decided to make this conditional.

Patrick

When given a length that equals the current hash algorithm's hex size, then `repo_find_unique_abbrev_r()` exits early without trying to find an abbreviation. This is only sensible because there is nothing to abbreviate in the first place, so searching through objects to find a unique prefix would be a waste of compute.

What we don't handle though is the case where the user passes a length greater than the hash length. This is fine in practice as we still compute the correct result. But at the very least, this is a waste of resources as we try to abbreviate a value that cannot be abbreviated, which causes us to hit the object database.

Start to explicitly handle values larger than hexsz to avoid this performance penalty, which leads to a measureable speedup. The following benchmark has been executed in linux.git:

  Benchmark 1: git -c core.abbrev=9000 log --abbrev-commit (revision = HEAD~)
    Time (mean ± σ):     12.812 s ±  0.040 s    [User: 12.225 s, System: 0.554 s]
    Range (min … max):   12.723 s … 12.857 s    10 runs

  Benchmark 2: git -c core.abbrev=9000 log --abbrev-commit (revision = HEAD)
    Time (mean ± σ):     11.095 s ±  0.029 s    [User: 10.546 s, System: 0.521 s]
    Range (min … max):   11.037 s … 11.122 s    10 runs

  Summary
    git -c core.abbrev=9000 log --abbrev-commit HEAD (revision = HEAD) ran
      1.15 ± 0.00 times faster than git -c core.abbrev=9000 log --abbrev-commit HEAD (revision = HEAD~)

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 object-name.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Hi,

this is the second version of my patch series that fixes a segfault when parsing "core.abbrev" without a repository.

Changes compared to v1:

  - Stop truncating the abbreviation length to the upper boundary
    completely. It's unnecessary, complicates the code and makes us
    dependent on `the_repository`.

  - Adapt `parse_opt_abbrev_cb()` to also stop truncating such that the
    behaviour of "core.abbrev" and `--abbrev` is the same.

  - Extend test coverage a bit.

Thanks!

Patrick

Patrick Steinhardt (3):
  config: fix segfault when parsing "core.abbrev" without repo
  parse-options-cb: stop clamping "--abbrev=" to hash length
  object-name: don't try to abbreviate to lengths greater than hexsz

 config.c           |  4 ++--
 object-name.c      |  2 +-
 parse-options-cb.c |  2 --
 t/t4202-log.sh     | 24 ++++++++++++++++++++++++
 t/t5601-clone.sh   |  7 +++++++
 5 files changed, 34 insertions(+), 5 deletions(-)

Range-diff against v1:
1:  7ded51bbce ! 1:  b48c50dd92 config: fix segfault when parsing "core.abbrev" without repo
    @@ Commit message
         `the_hash_algo` outside of Git repositories.
     
         Fix both of these issues by not making it an error anymore when the
    -    given length exceeds the hash length. Instead, if we have a repository,
    -    then we truncate the length to the maximum length of `the_hash_algo`.
    -    Otherwise, we simply leave the abbreviated length intact and store it
    -    as-is. This is equivalent to the logic in `parse_opt_abbrev_cb()` and is
    -    handled just fine by `repo_find_unique_abbrev_r()`. In practice, we
    -    should never even end up using `default_abbrev` without a repository
    -    anyway given that abbreviating object IDs to unique prefixes requires us
    -    to have access to an object database.
    +    given length exceeds the hash length. Instead, leave the abbreviated
    +    length intact. `repo_find_unique_abbrev_r()` handles this just fine
    +    except for a performance penalty which we will fix in a subsequent
    +    commit.
     
         Reported-by: Kyle Lippincott <spectral@google.com>
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
    @@ config.c: static int git_default_core_config(const char *var, const char *value,
      			default_abbrev = -1;
      		else if (!git_parse_maybe_bool_text(value))
     -			default_abbrev = the_hash_algo->hexsz;
    -+			default_abbrev = startup_info->have_repository ?
    -+				the_hash_algo->hexsz : GIT_MAX_HEXSZ;
    ++			default_abbrev = GIT_MAX_HEXSZ;
      		else {
      			int abbrev = git_config_int(var, value, ctx->kvi);
     -			if (abbrev < minimum_abbrev || abbrev > the_hash_algo->hexsz)
     +			if (abbrev < minimum_abbrev)
      				return error(_("abbrev length out of range: %d"), abbrev);
    -+			else if (startup_info->have_repository && abbrev > the_hash_algo->hexsz)
    -+				abbrev = the_hash_algo->hexsz;
      			default_abbrev = abbrev;
      		}
    - 		return 0;
     
      ## t/t4202-log.sh ##
     @@ t/t4202-log.sh: test_expect_success 'log.abbrevCommit configuration' '
      	test_cmp expect.whatchanged.full actual
      '
      
    -+test_expect_success 'log.abbrevCommit with --abbrev=9000' '
    ++test_expect_success '--abbrev-commit with core.abbrev=false' '
     +	git log --no-abbrev >expect &&
    -+	git log --abbrev-commit --abbrev=9000 >actual &&
    ++	git -c core.abbrev=false log --abbrev-commit >actual &&
    ++	test_cmp expect actual
    ++'
    ++
    ++test_expect_success '--abbrev-commit with core.abbrev=9000' '
    ++	git log --no-abbrev >expect &&
    ++	git -c core.abbrev=9000 log --abbrev-commit >actual &&
     +	test_cmp expect actual
     +'
     +
-:  ---------- > 2:  92860256a6 parse-options-cb: stop clamping "--abbrev=" to hash length
2:  31c0405f85 = 3:  0ccb8d8efa object-name: don't try to abbreviate to lengths greater than hexsz

The "core.abbrev" config allows the user to specify the minimum length when abbreviating object hashes. Next to the values "auto" and "no", this config also accepts a concrete length that needs to be bigger or equal to the minimum length and smaller or equal to the hash algorithm's hex length. While the former condition is trivial, the latter depends on the object format used by the current repository. It is thus a variable upper boundary that may either be 40 (SHA-1) or 64 (SHA-256).

This has two major downsides. First, the user that specifies this config must be aware of the object hashes that its repository use. If they want to configure the value globally, then they cannot pick any value in the range `[41, 64]` if they have any repository that uses SHA-1. If they did, Git would error out when parsing the config.

Second, and more importantly, parsing "core.abbrev" crashes when outside of a Git repository because we dereference `the_hash_algo` to figure out its hex length. Starting with c8aed5e8da (repository: stop setting SHA1 as the default object hash, 2024-05-07) though, we stopped initializing `the_hash_algo` outside of Git repositories.

Fix both of these issues by not making it an error anymore when the given length exceeds the hash length. Instead, leave the abbreviated length intact. `repo_find_unique_abbrev_r()` handles this just fine except for a performance penalty which we will fix in a subsequent commit.

Reported-by: Kyle Lippincott <spectral@google.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 config.c         |  4 ++--
 t/t4202-log.sh   | 12 ++++++++++++
 t/t5601-clone.sh |  7 +++++++
 3 files changed, 21 insertions(+), 2 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

So if the value is set to 'no'ish, we set it to the max value possible.

I was wondering if the documentation for 'core.abbrev' needs to be modified, but seems like we don't mention how the max value can error out in the first place so all good there.

The `OPT__ABBREV()` option allows the user to specify the length that object hashes shall be abbreviated to. This length needs to be in the range of `(MIN_ABBREV, the_hash_algo->hexsz)`, which is why we clamp the value as required. While this makes sense in the case of `MIN_ABBREV`, it is unnecessary for the upper boundary as the value is eventually passed down to `repo_find_unnique_abbrev_r()`, which handles values larger than the current hash length just fine.

In the preceding commit, we have changed parsing of the "core.abbrev" config to stop clamping to the upper boundary. Let's do the same here so that the code becomes simpler, we are consistent with how we treat the "core.abbrev" config and so that we stop depending on `the_repository`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 parse-options-cb.c |  2 --
 t/t4202-log.sh     | 12 ++++++++++++
 2 files changed, 12 insertions(+), 2 deletions(-)

When given a length that equals the current hash algorithm's hex size, then `repo_find_unique_abbrev_r()` exits early without trying to find an abbreviation. This is only sensible because there is nothing to abbreviate in the first place, so searching through objects to find a unique prefix would be a waste of compute.

What we don't handle though is the case where the user passes a length greater than the hash length. This is fine in practice as we still compute the correct result. But at the very least, this is a waste of resources as we try to abbreviate a value that cannot be abbreviated, which causes us to hit the object database.

Start to explicitly handle values larger than hexsz to avoid this performance penalty, which leads to a measureable speedup. The following benchmark has been executed in linux.git:

  Benchmark 1: git -c core.abbrev=9000 log --abbrev-commit (revision = HEAD~)
    Time (mean ± σ):     12.812 s ±  0.040 s    [User: 12.225 s, System: 0.554 s]
    Range (min … max):   12.723 s … 12.857 s    10 runs

  Benchmark 2: git -c core.abbrev=9000 log --abbrev-commit (revision = HEAD)
    Time (mean ± σ):     11.095 s ±  0.029 s    [User: 10.546 s, System: 0.521 s]
    Range (min … max):   11.037 s … 11.122 s    10 runs

  Summary
    git -c core.abbrev=9000 log --abbrev-commit HEAD (revision = HEAD) ran
      1.15 ± 0.00 times faster than git -c core.abbrev=9000 log --abbrev-commit HEAD (revision = HEAD~)

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 object-name.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Patrick Steinhardt <ps@pks.im> writes:

The changes look good, I wasn't expecting a benchmark in the last commit for such a small change, but that was nice to see. Thanks.

[snip]