Re: RFC: reverse history tree, for faster & background clones

From: Andres G. Aragoneses <knocte@gmail.com>
Date: Jun 14, 2015, 14:14 UTC
Message-ID: <mlk28u$mm4$1@ger.gmane.org>
In-Reply-To: <1434112410.5381.8.camel@kaarsemaker.net>

On 12/06/15 14:33, Dennis Kaarsemaker wrote:

Show 24 quoted lines

> On vr, 2015-06-12 at 13:39 +0200, Andres G. Aragoneses wrote:
>> On 12/06/15 13:33, Dennis Kaarsemaker wrote:
>>> On vr, 2015-06-12 at 13:26 +0200, Andres G. Aragoneses wrote:
>>>
>>>> AFAIU git stores the contents of a repo as a sequence of patches in the
>>>> .git metadata folder.
>>>
>>> It does not, it stores full snapshots of files.
>>
>> In bare repos too?
>
> Yes. A bare repo is nothing more than the .git dir of a non-bare repo
> with the core.bare variable set to True :)
>
>>>> 1. `git clone --depth 1` would be way faster, and without the need of
>>>> on-demand compressing of packfiles in the server side, correct me if I'm
>>>> wrong?
>>>
>>> You're wrong due to the misunderstanding of how git works :)
>>
>> Thanks for pointing this out, do you mind giving me a link of some docs
>> where I can correct my knowledge about this?
>
> http://git-scm.com/book/en/v2/Git-Internals-Git-Objects should help.

Wow, now I wonder if I should also propose a change to make git optionally not store the full snapshots, so save disk space. Thanks for pointing this out to me.

Show 22 quoted lines

>>>> 2. `git clone` would be able to allow a "fast operation, complete in the
>>>> background" mode that would allow you to download the first snapshot of
>>>> the repo very quickly, so that the user would be able to start working
>>>> on his working directory very quickly, while a "background job" keeps
>>>> retreiving the history data in the background.
>>>
>>> This could actually be a good thing, and can be emulated now with git
>>> clone --depth=1 and subsequent fetches in the background to deepen the
>>> history. I can see some value in clone doing this by itself, first doing
>>> a depth=1 fetch, then launching itself into the background, giving you a
>>> worktree to play with earlier.
>>
>> You're right, didn't think about the feature that converts a --depth=1
>> repo to a normal one. Then a patch that would create a --progressive
>> flag (for instance, didn't think of a better name yet) for the `clone`
>> command would actually be trivial to create, I assume, because it would
>> just use `depth=1` and then retrieve the rest of the history in the
>> background, right?
>
> A naive implementation that does just clone --depth=1 and then fetch
> --unshallow would probably not be too hard, no. But whether that would
> be the 'right' way of implementing it, I wouldn't know.

Ok, anyone else that can give an insight here?

I imagine that I would not get real feedback until I send a [PATCH]...

I guess I would use a user-facing message like this one:

Finished cloning the last snapshot of the repository. Auto downloading the rest of the history in background.

(Since there's already a similar "background" feature already wrt auto-packing the repository: `Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping.`.)

Thanks