Re: [StGit PATCH 03/14] Write to a stack log when stack is modified

From: Catalin Marinas <catalin.marinas@gmail.com>
Date: Jun 19, 2008, 09:24 UTC
Message-ID: <b0943d9e0806190224v1b6434fesd3a54443422edaeb@mail.gmail.com>
In-Reply-To: <20080618173246.GA1155@diana.vm.bytemark.co.uk>

2008/6/18 Karl Hasselström <kha@treskal.com>:

Show 6 quoted lines

> On 2008-06-18 17:16:10 +0100, Catalin Marinas wrote:
>> We could also easily use the commit message directly for metadata
>> instead of .git/patches/<branch>/.
>
> Yes. This is true for any log scheme, though. (But I agree -- longer
> term, it would be very nice to have _only_ the log.)

It would make collaboration using stgit stacks easier. I can start looking at this once we agree on the log structure.

Show 13 quoted lines

>> The time to create that tree and blobs worries me a bit, plus the
>> (in my view) complicated structure.
>
> You're right to worry. My log format makes things feel slightly slower
> when logging a 20-30 deep stack. If I can't make it faster, it's not
> viable. But I'm pretty sure I can -- it should be simple to reuse all
> the trees and blobs for the patches that weren't touched. (And
> operations that touch lots of patches, like rebase, have to write a
> lot of git objects anyway as part of their operation, so the relative
> slowdown should not be large.)
>
> Anyway, benchmarking is the way to go here. Talk will only get us that
> far.

In the past, I ran some tests because people complained that stgit was slow compared to quilt:

http://article.gmane.org/gmane.comp.version-control.git/9670

After profiling (the stg-prof file), as expected, most of the time was spent in external Git calls which I tried to keep to a minimum. With your approach, you probably add at least 4-5 calls to Git (just a guess, I haven't counted) and, with a big repository, it will be visible (I have about 15 branches on my Linux kernel clone going back to the 2.6.12 kernel and the .git size is over 1/2GB after git-gc).

Show 7 quoted lines

>> Making the first log entry special gets difficult with log pruning
>> (unless you prune the whole log rather than entries older than a
>> chosen time or number) since you might have to re-create all the
>> chained log entries as the first log's sha1 will probably change.
>
> You have to re-create all the commits anyway, since they all are
> immutable, and all have a back pointer.

Ah, OK. So, at least initially, we should only support the full log pruning.

Show 10 quoted lines

>> The applied patches are chained automatically via HEAD. For
>> unapplied patches, we could add the correponding commits as parents
>> of the logging commit (starting with the third parent as the first
>> two are used for log chaining and applied patches). Do we hit any OS
>> limit with the number of arguments?
>
> Not until long after we hit git limits to the number of parents of a
> commit. I believe the octopus merge refuses to create merges with more
> than about 25 parents, and we probably shouldn't do more than that
> either. We'll have to do a tree of octopuses.

For the first log only, we could chain the unapplied patches using commits with 2 parents. We just need to warn people not to stare at the <branch>.stgit directly :-)

Show 11 quoted lines

>> Since we only need the unapplied commits tracking for gc and pull
>> reasons, we could only create that commit that references them when
>> we prune the stack log and link it from the top one (but it won't be
>> used by stgit).
>
> Yes, we need to create an "unapplied octopus" if and only if we have
> unapplied patches that we can't prove are reachable from the stack top
> or the branch head (we have to save both, in case the user has done
> something such as git-committing on topp of the stack and caused head
> != top). Which is for the first log entry, and in situations such as
> "stg pick --unapplied", but not for "stg pop" and the like.

Yes, but just chain unapplied commits rather than using octopus (you could try to the -mm series with 500+ patches and see what happens).

> I do agree that we shouldn't try to use the octopuses to get hold of
> the commits, though -- just to keep them reachable. We save the sha1
> along with the patch name elsewhere in a more convenient form. (My
> proposed format does precisely this.)

Yes, but you still have to refer the tree objects corresponding to a patch and extra work to generate the simplified log.

> So. If I got it right, your proposal is:
>
>  * Tree: just take the HEAD tree.

Yes.

>  * Commit message: list the applied and unapplied patches with their
>    commit sha1s.

We don't even need to differentiate between applied and unapplied in the commit message as long as they are listed in order since one of the parents would represent the boundary between them. Since lib.stack is pretty well structured, we could later modify PatchOrder to use the log.

>  * Parents: the previous log entry; branch head; something that
>    (recusively) points to all unapplied commits, if necessary.

As you pointed below, "branch head" should probably be the "stack top". We don't need to track the "branch head" if different, just need to fix up the error and add the patches to the stack. And, anyway, if one modifies the HEAD using git directly, the log will still point to the top of the stack.

The third head would only be needed for the first log entry or when we use pick --unapplied (in the latter, it only points to the unapplied commit). The third head shouldn't be used at all in the log, just created when needed.

> I'd add to that:
>
>  * The stack top must be a parent too if head != top.

See above, branch head would actually be the stack top.

>  * The commit message must include a version number, and the branch
>    head sha1.

OK with the version number but the branch head (or stack top) is one of the parents already.

>  * I'm pretty sure we want the kind of "simplified" log I have in my
>    proposal. The full log in your proposal is going to look every bit
>    as ugly as the one in mine.

I agree it will look ugly but the simplified log adds an extra overhead on any stgit action. If we don't use stg log -g, a text only log command could show the diff. We can add it afterwards though if it is fast enough.

-- 
Catalin