Darcs-Git: upgrading to Git 0.99

[CC'd to the Git mailling list; please CC any replies to Darcs-Devel]

David, Ian,

I'd like to upgrade the Git code used in Darcs to 0.99 (we're currently using 0.6). There are two good reasons for that, the first of which is actually a showstopper:

 - the format of Git repositories has changed incompatibly, with a new
   kind of thing called the ``pack'' (a very neat performance hack, by
   the way); hence, Darcs-Git is unable to read recent Git repos,
   unless you use the Git tools to unpack them;

 - 0.99 actually exports usable interfaces, which will allow us to use
   pristine Git sources in Darcs.

Now I'm wondering how to do that. Currently, I'm using a nasty hack using the C preprocessor to include just the sources we need in Darcs. As 0.99 builds a ``libgit.a'', I'd like to use that instead.

There are three ways to do that:

  (1) require that the users put a suitable libgit.a in /usr/local/lib
      before building Darcs, and distribute a tarball of Git from
      darcs.net;

  (2) include just the sources needed for libgit.a in Darcs, and have
      the Darcs build build a local libgit

  (3) as (2), but include all of Git, including their
      ``user-friendly'' scripts.

Solution (2) will include 33 files totalling 167KB, while (3) is about a megabyte of source.

My personal favourite is solution (2), as it is simple for both the users and us. I'm not very keen on (1), as it will cause problems when the friendly Git folks change their interfaces, but have no strong dislike towards it if it's what you think is right. (3) is definitely overkill.

                                        Juliusz

On Sat, Jul 16, 2005 at 10:45:47PM +0200, Juliusz Chroboczek wrote:

Great!

I'd really prefer option (1), *if* the git folks can confirm that the API is at least intended to be stable. As an subtly different option, we could include a script that would download and untar the git sources and then build them. But it'd be great to allow users to upgrade their libgit without our intervention if a protocol or repository format change occurs that doesn't affect the API.

I guess the real question is whether the API is more or less stable than the protocols and disk formats. If the API is more stable, we'd rather link with an external libgit and be robust with respect to on-disk format changes (such as pack files). If the on-disk format is more stable, we'd rather include a copy of the source code and be robust with respect to API changes of libgit.

A fourth option would be to include git sources, but also include a configure flag that could be used to link with an external libgit. This is probably the most robust solution, but also the most complex solution (and thus probably not the best).

Juliusz Chroboczek wrote:

I was under the impression that the stablest interface to git was the command line; we use spawnvp in stacked git. I've been using it with repositories that make the Linux kernel look small.

I certainly don't think the lib interface is anywhere near stable: Linus accepted my change to index_fd far too easily.

 >
 >   (2) include just the sources needed for libgit.a in Darcs, and have
 >       the Darcs build build a local libgit
 >
 >   (3) as (2), but include all of Git, including their
 >       ``user-friendly'' scripts.

Ugh. That's what they do in the commercial world. We have it so much better here in Linux & BSD land: you just add a "depends libgit1" line to your package, and the right thing happens: minor updates happen automatically and changes that break the interface don't. Of course, keeping it that easy for the user requires active effort from us, the developers, and sometimes the pain spills over, but the benefits are nice.

I see too major long-term alternatives:

1) talk the darcs guys into using spawnvp
2) talk the git people into exporting a stable lib interface

It's my opinion that #1 is a non-starter. We want others to interact with us. I'm going to use spawnvp for my porcelain, but we should be inclusive.

The current 'libgit' probably contains more than Linus and Junio are comfortable locking down as a stable interface, but I'm sure that there's a subset that they'd be comfortable with once a relative amount of stability is achieved, or it may be achievable via some other method.

I propose that Darcs includes all of git for now. (I prefer this over partial inclusion; anybody should be able to take the darcs sources and easily drop in a later git). However, the long term goal is for a library. Darcs and git work together to determine the minimal amount that needs to go into libgit1.so. It gets blessed by being documented, and doesn't change until libgit2.so.

I'd like to see this added to Junio's list of "1.0" goals.

A similar 1.0 goal would be to document porcelain's use of the .git directory. For instance, stacked git uses .git/patches, .git/patchdescr.tmpl and .git/patchexport.tmpl. If Linus does not accept a patch documenting this usage, stacked git should use .stgit instead.

Bryan

Bryan Larsen <bryan.larsen@gmail.com> writes:

I should mention that I liked the libification work Brad Roberts did around late April. Unfortunately the things have been always in great flux for the last couple of months, and it was too hard to include that and maintain it back then.

I fully agree that supporting C-level linkage is worthy, and should be one of our longer term goals.

I agree that coordinating the namespace under $GIT_DIR among Porcelains is something we need (it was what prompted me to steal the branches/ convention from Cogito). The job of the core should be to help Porcelains avoid stepping on each other's toes.

The documentation of the internals for $GIT_DIR/patches is probably better left to StGIT documentation, though, at the moment. When other Porcelains start wishing to access the "series of patches expressed as a set of commit chain" expressed by StGIT there (e.g. show patch series in addition to regular commit chain in gitk), the core should help the Porcelains to work well with each other, to do things in a compatible way. This may involve moving some common things to core side and mention the convention for Porcelains to work well together in the core documentation.

However, I am slightly negative about suggesting these two to be part of the 1.0 goals. Linus wanted to make 1.0 how many weeks ago? I personally think that a usable baseline, stable enough to allow stripping out the core part currently shipped as part of Cogito, would be a good place to stop and declare 1.0. My list was meant to enumuerate what might be missing from the "usable baseline".

Junio C Hamano wrote:

Excellent.

I think we want the same thing, you're just expressing it explicitly: stgit's usage of the .git namespace should be mentioned in git documentation. actual details belong in stgit, either explicitly in documentation or implicitly in the code.

All I'm looking for is a statement like "once we're at 1.0, darcs doesn't break until 2.0". If we don't actually break out a blessed lib interface until 1.1, that's fine with me. To me, 1.0 implies core stability.

thanks, Bryan

Noted, thanks for the info.

(This makes a lot of sense, Git is evolving very fast. I haven't looked at Git since mid-April, and I'm very much impressed at the difference between 0.6 and 0.99.)

This is, of course, only possible when there are stable interfaces, which in turn make change problematic. (Which, as far as I understand, is the very reason why the Linux kernel tree contains everyone's and his brother's driver rather than having a stable module ABI, but that's besides the point.)

Hold on... Nobody is speaking about *binary* compatibility, it's source-level compatibility that we need. There is absolutely no reason to introduce the complexities of shared libraries into the picture.

                                        Juliusz

Source level compatibility and stability is the big deal. Compared to that, shared libraries are an implementation detail, in my opinion. Sometimes those details get "interesting", but they are soluble.

I could care less whether you use libgit.a or a libgit.so. Just as long as distros or anybody else can update their darcs if a major data-loss bug is found in git. A recompile is acceptable. Dealing with the addition of a parameter to index_fd() is not.

Bryan