Re: Git is not scalable with too many refs/*

From: Junio C Hamano <gitster@pobox.com>
Date: Sep 26, 2011, 23:38 UTC
Message-ID: <7vsjnizxf5.fsf@alter.siamese.dyndns.org>
In-Reply-To: <ece30e6a1b74bcddde5634003408f61f@quantumfyre.co.uk>

Julian Phillips <julian@quantumfyre.co.uk> writes:

> Back when I made that change, I failed to notice that get_ref_dir was
> recursive for subdirectories ... sorry ...

Aha, I also was blind while I was watching this discussion from the sideline, and I thought I re-read the codepath involved X-<. Indeed we were sorting the list way too early and the patch looks correct.

Thanks.

Show 41 quoted lines

> Hopefully this should speed things up.  My test repo went from ~17m
> user time, to ~2.5s.
> Packing still make things much faster of course.
>
> diff --git a/refs.c b/refs.c
> index a615043..212e7ec 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -319,7 +319,7 @@ static struct ref_list *get_ref_dir(const char
> *submodule, c
>                 free(ref);
>                 closedir(dir);
>         }
> -       return sort_ref_list(list);
> +       return list;
>  }
>
>  struct warn_if_dangling_data {
> @@ -361,11 +361,13 @@ static struct ref_list *get_loose_refs(const
> char *submodu
>         if (submodule) {
>                 free_ref_list(submodule_refs.loose);
>                 submodule_refs.loose = get_ref_dir(submodule, "refs",
> NULL);
> +               submodule_refs.loose =
> sort_refs_list(submodule_refs.loose);
>                 return submodule_refs.loose;
>         }
>
>         if (!cached_refs.did_loose) {
>                 cached_refs.loose = get_ref_dir(NULL, "refs", NULL);
> +               cached_refs.loose = sort_refs_list(cached_refs.loose);
>                 cached_refs.did_loose = 1;
>         }
>         return cached_refs.loose;
>
>
>
>>
>>
>> -Martin