threads / patch / 63013

patchupload-pack: no longer use hidden-refs as exclude_patterns

Subject: [PATCH] upload-pack: no longer use hidden-refs as exclude_patterns

## tl;dr

3 messages between Feb 27, 2025 and Feb 27, 2025. Diffs are folded; open one to read it.

replies: 2people: 3as markdown or json

SURA via GitGitGadget· Feb 27, 2025, 12:46 UTC · lore
From: SURA <sura907@hotmail.com>
Signed-off-by: SURA <sura907@hotmail.com>
---
    upload-pack: No longer use hidden-refs as exclude_patterns
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1866%2FSURA907%2Fmaster-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1866/SURA907/master-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1866
 upload-pack.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)
Show changes to upload-pack.c +3 −12
diff --git a/upload-pack.c b/upload-pack.c
index 728b2477fcc..9ae42a463a3 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -609,21 +609,12 @@ static int allow_hidden_refs(enum allow_uor allow_uor)
 static void for_each_namespaced_ref_1(each_ref_fn fn,
 				      struct upload_pack_data *data)
 {
-	const char **excludes = NULL;
 	/*
-	 * If `data->allow_uor` allows fetching hidden refs, we need to
-	 * mark all references (including hidden ones), to check in
-	 * `is_our_ref()` below.
-	 *
-	 * Otherwise, we only care about whether each reference's object
-	 * has the OUR_REF bit set or not, so do not need to visit
-	 * hidden references.
+	 * config transfer.hideRefs of upload-pack is diffient from arg exclude of for-each-ref,
+	 * We should not set exclude_patterns here
 	 */
-	if (allow_hidden_refs(data->allow_uor))
-		excludes = hidden_refs_to_excludes(&data->hidden_refs);
-
 	refs_for_each_namespaced_ref(get_main_ref_store(the_repository),
-				     excludes, fn, data);
+				     NULL, fn, data);
 }
 
 

base-commit: 08bdfd453584e489d5a551aecbdcb77584e1b958
-- 
gitgitgadget
Patrick Steinhardt· Feb 27, 2025, 14:54 UTC · re: SURA via GitGitGadget · lore

Re: [PATCH] upload-pack: no longer use hidden-refs as exclude_patterns

On Thu, Feb 27, 2025 at 12:46:11PM +0000, SURA via GitGitGadget wrote:
Show 40 quoted lines
> From: SURA <sura907@hotmail.com>
> 
> Signed-off-by: SURA <sura907@hotmail.com>
> ---
>     upload-pack: No longer use hidden-refs as exclude_patterns
> 
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1866%2FSURA907%2Fmaster-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1866/SURA907/master-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/1866
> 
>  upload-pack.c | 15 +++------------
>  1 file changed, 3 insertions(+), 12 deletions(-)
> 
> diff --git a/upload-pack.c b/upload-pack.c
> index 728b2477fcc..9ae42a463a3 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -609,21 +609,12 @@ static int allow_hidden_refs(enum allow_uor allow_uor)
>  static void for_each_namespaced_ref_1(each_ref_fn fn,
>  				      struct upload_pack_data *data)
>  {
> -	const char **excludes = NULL;
>  	/*
> -	 * If `data->allow_uor` allows fetching hidden refs, we need to
> -	 * mark all references (including hidden ones), to check in
> -	 * `is_our_ref()` below.
> -	 *
> -	 * Otherwise, we only care about whether each reference's object
> -	 * has the OUR_REF bit set or not, so do not need to visit
> -	 * hidden references.
> +	 * config transfer.hideRefs of upload-pack is diffient from arg exclude of for-each-ref,
> +	 * We should not set exclude_patterns here
>  	 */
> -	if (allow_hidden_refs(data->allow_uor))
> -		excludes = hidden_refs_to_excludes(&data->hidden_refs);
> -
>  	refs_for_each_namespaced_ref(get_main_ref_store(the_repository),
> -				     excludes, fn, data);
> +				     NULL, fn, data);
>  }

This message is missing any context _why_ we want to do this. For background: setting up these exclude patterns for hidden references is quite an important performance optimization in large repositories, so disabling it just like that is not an option without a good reason to do so.

So what is the issue that you see and why is this fix the solution for that issue?

Patrick
SURA· Feb 27, 2025, 15:51 UTC · re: Patrick Steinhardt · lore

Re: [PATCH] upload-pack: no longer use hidden-refs as exclude_patterns

在 2025/2/27 22:54, Patrick Steinhardt 写道:
Show 51 quoted lines
> On Thu, Feb 27, 2025 at 12:46:11PM +0000, SURA via GitGitGadget wrote:
>> From: SURA <sura907@hotmail.com>
>>
>> Signed-off-by: SURA <sura907@hotmail.com>
>> ---
>>      upload-pack: No longer use hidden-refs as exclude_patterns
>>
>> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1866%2FSURA907%2Fmaster-v1
>> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1866/SURA907/master-v1
>> Pull-Request: https://github.com/gitgitgadget/git/pull/1866
>>
>>   upload-pack.c | 15 +++------------
>>   1 file changed, 3 insertions(+), 12 deletions(-)
>>
>> diff --git a/upload-pack.c b/upload-pack.c
>> index 728b2477fcc..9ae42a463a3 100644
>> --- a/upload-pack.c
>> +++ b/upload-pack.c
>> @@ -609,21 +609,12 @@ static int allow_hidden_refs(enum allow_uor allow_uor)
>>   static void for_each_namespaced_ref_1(each_ref_fn fn,
>>   				      struct upload_pack_data *data)
>>   {
>> -	const char **excludes = NULL;
>>   	/*
>> -	 * If `data->allow_uor` allows fetching hidden refs, we need to
>> -	 * mark all references (including hidden ones), to check in
>> -	 * `is_our_ref()` below.
>> -	 *
>> -	 * Otherwise, we only care about whether each reference's object
>> -	 * has the OUR_REF bit set or not, so do not need to visit
>> -	 * hidden references.
>> +	 * config transfer.hideRefs of upload-pack is diffient from arg exclude of for-each-ref,
>> +	 * We should not set exclude_patterns here
>>   	 */
>> -	if (allow_hidden_refs(data->allow_uor))
>> -		excludes = hidden_refs_to_excludes(&data->hidden_refs);
>> -
>>   	refs_for_each_namespaced_ref(get_main_ref_store(the_repository),
>> -				     excludes, fn, data);
>> +				     NULL, fn, data);
>>   }
> This message is missing any context _why_ we want to do this. For
> background: setting up these exclude patterns for hidden references is
> quite an important performance optimization in large repositories, so
> disabling it just like that is not an option without a good reason to do
> so.
>
> So what is the issue that you see and why is this fix the solution for
> that issue?
>
> Patrick
Oh, so sorry, I should merge mail message
See: 
https://lore.kernel.org/git/CAD6AYr-ZC32VNfUfMB63H-rQRfTdV=VQfBm67i2mG+6GDCNxkQ@mail.gmail.com/T/#u
Copy message here
---
Hello everyone
OS: Linux Mint 22
git version: v2.48.1

I found that packed refs are excluded by the transfer.hideRefs front match, while loose refs use full match (when transfer.hideRefs ends with '/', it is prefix match, which is normal)

When the server uses git, after setting transfer.hideRefs, the references that the client can see before and after server repo gc are different

It seems that 59c35fa accidentally damaged upload-pack when optimizing git for-each-ref

It seems that there is no simple fix except rolling back this commit?

← back to recent threads