ls-refs protocol extension

Hi list, I have been working on a custom git server where the sha values required to respond to a ls-refs command can occasionally be expensive to compute. It would be a great improvement of user experience if it was possible to show progress to the user while this is happening. I was hoping that this could be solved by using the already existing sideband-all capability, but unfortunately it seems that even with sideband-all the sideband is not available during ls-refs. So my plan would be to start working on a patch to add sideband support to ls-refs, unless there is some other protocol feature that I have overlooked and that would enable my use case.

Any pointers to either alternative solutions and/or hints to potential pitfalls in the implementation of my suggestion would be greatly appreciated.

Best regards, Christian Schilling

On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:

Almost certainly the expensive part of ls-refs is actually traversing the references, not parsing the objects they point at and determining their hash.

Incidentally, we had a discussion recently [1] that resulted in some patches that make it so that ls-refs often only has to read through part of the refs in your repository, not all of them.

It's possible that that might help, but honestly I'd be surprised if there was a real use-case that needed it (especially after the patches that I mentioned which should make it fast enough that you don't have to care :-)).

Thanks, Taylor

[1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/

On Mon 25. Jan 2021 at 02:04, Taylor Blau <me@ttaylorr.com> wrote:

My use case is very different. I’m working on a custom git server that creates refs as well as the objects they are referring to on demand.

More specifically the references are derived from the repo url to implement partial cloning. So you can do: git clone http://repository-url.git:/any/subdir/you/want.git

The server will do something similar to git-filter-branch and deliver the result. Due to the way my own branch filtering implementation works this is usually fast enough for the users not to notice what is going on, but the fact remains that the shas are only known after the filtering is complete. Also my server acts as a proxy with the canonical backing repository hosted on a different machine. So to start filtering the proxy has to first ask the upstream for the current state of the repo and download any new changes. I have done a lot of optimizations to make all of this fast and in most cases it is, but for the occasional case, for example if a completely new repo is cloned for the first time, it leaves the users looking at a seemingly “frozen” command prompt not knowing if they made a mistake or are encountering some bug or network problem.

At first I also thought about suggesting a protocol extension that allows to start transferring pack data before all refs are known to reduce the total time needed, but the slow cases are infrequent enough so that this probably not worth it.

I had my own struggles with the “to many refs” problem, but this is unrelated :-)

Sorry for not being clearer about what I am trying to accomplish the first time, I hope I convinced you that this is a very different, but real, use case.

Thanks, Christian