Core and Not-So Core

I have been experimenting with pure-Java implementation of GIT concepts with a goal of eventually providing plugins to Eclipse to allow the Eclipse GUI to interact with GIT repositories.

One thing I noticed when doing this is that the present index/cache structure is rather arbitrary and the optimum index structure is determined by the structure of the tools that use a GIT repository rather than the structure of the GIT repository itself.

To give a concrete example: the cache currently contains most of the posix stat structure primarily to allow quick change detection. In the Java world, most of the posix stat structure is not directly accessible via the pure-Java file system abstractions. However, for most purposes detecting changes to files modification time and file size would be enough. Given this is the case, a Java-GIT client doesn't need to bother getting access to a posix stat structure and could therefore get away with a simpler index structure, provided it doesn't need to interoperate with a 'C'-GIT client that shared the same workspace. A Java-GIT client might also choose to represent an index cache as a complex serialized Java object graph or (perhaps) an XML document.

Another example: I can imagine a variant of the index file structure that recorded all the parents which have been merged into the cache and automatically include this information when performing the commit.

The point is that many different index file structures are possible and will be determined in part by the tooling created in the porcelain layer - there really is no one true index file format as there is a one true repository format. Different tools can use different index file formats and still interoperate at the repository level because only the repository format needs to have a solid, unchanging definition.

Currently the GIT stack is structured as follows:

cogito git-core

I think it would be worthwhile if care was taken to draw a distinction between the repository and the cache aspects of the git core, perhaps even going to the extreme of moving all knowledge of the cache into cogito itself. By clearly drawing this distinction, we will more easily enable the creation of different kind of tools sets atop the foundation of the GIT repository format.

e.g., either:

cogito git-cache git-respository

or:

cogito-tools cogito-cache git-repository

Anyway, I offer this as food for thought - chew or flame away as appropriate!

jon.

On Wed, 2005-05-11 at 01:00 +1000, Jon Seymour wrote:

It's not April 1st. Why would you want to reimplement it in Java instead of just using the existing implementation? Is this a religious issue?

He want's to make an eclipce plugin...

This must be done in Java.

On Tue, 2005-05-10 at 12:50 -0300, Eduardo Teixeira Dias wrote:

And there's no way for him to call execve()?

Hell, even the lisp nutters manage to make emacs invoke external programs occasionally. What makes Java programmers so much less sane?

Good point!

It can be done using execxxx().

But a Java version is better for this particular use (Eclipce Plugin):

  - Just a .jar download
  - Installation without external dependencies

El Tue, 10 May 2005 13:19:49 -0300 (BRT), "Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:

Someone who is going to hack the kernel can very well install more things. And anyway, git is "the linux SCM tool" so all distros will package it. Also, people who hacks the linux kernel usually runs it, so "git is not ported to win32" is not a big problem.

...and I don't think people who use eclipse wants to have the fastest tool on earth, so "java-invoking-C is slower than pure java" is not a great excuse either...

Code reuse == good. Java programmers should know that. Anyway, people is free to do whatever they want with their time :)

I'm a Java programmer and I was thinking like one (Culture thing).

The reuse argument is very Strong.

The git-core are ready and it's a very good idea to reuse it. Any change in the core or in the internal representation of git repository will benefit the Git-Java implementation...

I think that a growing utilization of GCJ and ClassPath will contribute to a Gradual Culture Change in the Java Land and promote a beter reutilization among all GCC languages...

Write Once Run Everywhere that GCJ Runs (Sounds Good).

Eclipse will be compiled with GCJ in Fedora 4. Let's see how it works...

Cheers!

[somewhat OT]

El Tue, 10 May 2005 19:33:14 -0300 (BRT), "Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:

GCJ in fact supports more architectures than Sun's java VM so with GCJ you _really_ can run your java code everywhere.

I'm a Java programmer and I was thinking like one (Culture thing).

The reuse argument is very Strong.

The git-core are ready and it's a very good idea to reuse it. Any change in the core or in the internal representation of git repository will benefit the Git-Java implementation...

I think that a growing utilization of GCJ and ClassPath will contribute to a Gradual Culture Change in the Java Land and promote a beter reutilization among all GCC languages...

Write Once Run Everywhere that GCJ Runs (Sounds Good).

Eclipse will be compiled with GCJ in Fedora 4. Let's see how it works...

Cheers!

I'm a Java programmer and I was thinking like one (Culture thing).

The reuse argument is very Strong.

The git-core are ready and it's a very good idea to reuse it. Any change in the core or in the internal representation of git repository will benefit the Git-Java implementation...

I think that a growing utilization of GCJ and ClassPath will contribute to a Gradual Culture Change in the Java Land and promote a beter reutilization among all GCC languages...

Write Once Run Everywhere that GCJ Runs (Sounds Good).

Eclipse will be compiled with GCJ in Fedora 4. Let's see how it works...

Cheers!

Dear diary, on Tue, May 10, 2005 at 11:45:01PM CEST, I got a letter where Diego Calleja <diegocg@gmail.com> told me that...

It's not like everything git is ever going to be used for is kernel and only the kernel.

Btw, I rewrote about 90% of git in python, just out of curiousity. Performance is pretty competitive, and I can optimize certain operations by keeping the index in memory temporarily. It also saves a lot of output parsing, simplifies error-handling (if the frontend is written in python too, which bit now is), and the best of all: its about 600 lines of python code. I am all for a Java implementation. After its done we will see whether it works and how well it works. We can still throw it away if it sucks.

Andreas

On Wed, 11 May 2005, Petr Baudis wrote:

On Wed, 2005-05-11 at 08:44, Petr Baudis wrote:

So does this mean I can keep working on my wxWidgets implementation? :)

On Tue, May 10, 2005 at 11:45:01PM +0200, Diego Calleja wrote:

Note that it's also not true. At least if you use the gcj-specific CNI mechanisms instead of Sun's braindead JNI.

On 5/11/05, Christoph Hellwig <hch@infradead.org> wrote:

What I actually claimed (contrary to Diego's misquote) is that "Java-invokes-C-executable" would be slower than pure-Java. I didn't make a claim one way or the other about the performance of a Java implementation invoking some form of native interface.

jon.

On 5/11/05, David Woodhouse <dwmw2@infradead.org> wrote:

Not really - in the Java world, things are simpler if you don't have to carry around a JNI library - that way, you can just run it wherever a Java interpreter exists and not worry about having someone recompiling the JNI part.

A pure-Java implementation will perform better than a Java-invokes-C-executable approach [ though not, of course, a C-only approach ].

Another benefit of playing around with a Java abstraction is that I can more easily experiment with abstractions than I can in C since the abstraction-facilities of Java more directly support such playing than does C. So, for example, I can easily create a virtual repository which layers a local repository over a remote repository and transparently populates the local repository from the remote repository "on-demand". Of course, this sort of thing can be done in C, but it requires much more "work" to set up the abstractions.

You could perhaps argue that the existing 'C' tools in some way encapsulate access to the repository format. Maybe, but the fact is the rapid adoption of GIT has already effectively fixed the GIT repository format in practice so any change in repository format will require considerable planning anyway - planning that will allow sufficient time for implementations in other languages to catch up.

So, no, it's not a religious issue. If anything, it is being dogmatic to insist that the sacred GIT repository structure only be manipulated by 'C' tools blessed by the hands of Linus.

The concepts in GIT are bigger than the programming language its toolsets are implemented in.

jon.

On Wed, 2005-05-11 at 02:22 +1000, Jon Seymour wrote:

Given the volatility of the structure -- at least the details if not the fundamentals -- it seems bizarre to want to reimplement it rather than just using the existing tools.

This is the same mentality which gives Eclipse a half-arsed SSH reimplementation which doesn't behave like normal SSH is configured to behave either, right?

On Wed, 11 May 2005, Jon Seymour wrote:

I'd guess that an Eclipse plugin wouldn't have a on-disk cache at all, since it expects to keep the same address space through a whole day's work.

The two clients should be able to interoperate trivially; the C version would just treat the Java one as a text editor whose modifications happen to match commits going into the same head; as soon as the C user does either read-tree or update-cache, everything will be seen to be consistent again.

I think this is nonsensical. The cache format is tied to the way in which the repository accessing code is written, so a git-core separate from the cache wouldn't have a useful set of code.

It might be worthwhile to produce a separate document describing the repository format, such that it could be accessed by different code, however. Of course, the index file wouldn't be part of this documentation, for the same reason that the public git repositories don't include index files.

	-Daniel
*This .sig left intentionally blank*

Dear diary, on Tue, May 10, 2005 at 05:00:33PM CEST, I got a letter where Jon Seymour <jon.seymour@gmail.com> told me that...

Yes. And that's how it should be - the directory cache is just that - a _cache_. It does not hold any permanent information, merely serves to record git tools' working state relative to the given working tree. So unlike the objects database which has well-defined format and is supposed to be "public", you should view the directory cache as internal git tools' structure. If you want to mess with it too, either use the proper level of abstraction and call the git tools, or don't mess with it at all. And you need to care about it only if you want the git tools working on the same tree properly too - so in that case use the git tools too.

 From your arguments, it's not clear to me what really is the big
problem with the git tools. They are _designed_ for automatic use
instead of human interaction - you can perceive them just as methods
with funny (but actually friendly to your programs) calling convention.

Kind regards,

On 5/11/05, Petr Baudis <pasky@ucw.cz> wrote:

No argument there.

I agree in principle, though I'd like users to be able to easily switch between the Eclipse and git tools view of the workspace if they want to - who am I to say how a user should work? Eclipse does this kind of thing quite well with CVS precisely because it shares the workspace structures with the CVS command line tools rather than "re-inventing" the wheel. Yes, separation of concerns has been lost (two implementations of a CVS client), but the big win is that the tools behave like the user wants them to behave.

The trickiest case here is when the user switches between toolsets mid-merge. I guess what I can do is this:

user switch from eclipse -> to git-tools:
    1. blow away existing git tools index
    2. use git-read-tree to repeat the merge executed in eclipse (my
workspace will track parents)
    3. use git-update-cache --add/--remove to reflect merge actions
that have occurred since the workspace deviated from the HEAD.

Alternatively, I can just make Eclipse reflect cache changing actions out onto the git-tools, via an exec of those tools, as and when they occur.

Making use of the git tools index going the other way isn't so easy to
achieve because the git tools workspace as it stands doesn't track the
merges that have occurred (i.e. which parents were used to form the
current cache). However, that's not necessarily a big problem. I just
rebuild my "cache" from scratch based on the merges I know about and
treat every other difference from the HEAD as a user edit to the
workspace.
 

I am not really arguing that there is a big problem with the existing git tools. However, what I am arguing is that the existing workspace tools are just one way to manage the workspace (Eclipse might be another, as an example) and it would be helpful to keep this in mind, particularly when/if libification ever happens.

jon.

Jon Seymour wrote:

I disagree here. My experience is that Eclipse does not work well with command line use of CVS within one of its workspaces. This is because Eclipse keeps too much "state" information internally instead of relying on CVS for all information about the state of the CVS playground. This is one of the reasons that you can't attach Eclipse to an already checked out CVS playground. The problem should be easily avoidable.

In spite of the above, Eclipse's SCM interface is the best that I've stumbled across and is the main reason that I use Eclipse. I particularly like the ability to compare a file in the workspace with those in different versions, branches, etc.

So I'm looking forward to a git plug-in.

Peter

On Wed, 11 May 2005, Jon Seymour wrote:

In this case you just need to manage to use the current index format somehow.

Honestly I don't think Linus might be interested into changing the index format just to make a Java implementation happier. So your best bet is probably to use it as is or ignore it entirely.

Nicolas

On 5/11/05, Nicolas Pitre <nico@cam.org> wrote:

Agreed - I am not proposing a change to the index format, merely highlighting that the index format is closely bound to the workspace-oriented toolset and that the workspace-oriented toolset is distinct from the repository-oriented toolset; different workspace-oriented toolsets will have different requirements for their index files and so their index files might quite rightly differ in both form and substance.

That said, their is no harm making my set of tools interoperable with the git tools, where that is possible so that the user can decide to use which ever toolset fits the job immediately at hand. With this in mind, a published specification of the index format would help, but I am not going insist on that. I am certainly not going to ask for changes to it! Life is too short :-)

jon.

Hi

Note that eclipse in particular has a fairly complicated repository provider interface. The subversion plugin developers (the subclipse project) took quite a while to implement their stuff. Basing your stuff off of their code would be a good idea.

Also, they worked in 2 stages - in the first stage, they created something called the JavaHL interface, behind which they talked to the subversion C libraries using JNI. Then they created a pure-java implemenation of the subversion C libraries which also implemented the JavaHL interface, allowing them to compare and contrast the behaviour.

Regards,
   Noel Grandin

Jon Seymour wrote:

NOTICE: Please note that this email, and the contents thereof, 
are subject to the standard Peralex email disclaimer, which may 
be found at: http://www.peralex.com/disclaimer.html

If you cannot access the disclaimer through the URL attached 
 and you wish to receive a copy thereof please send 
 an email to email@peralex.com

On 5/11/05, Noel Grandin <noel@peralex.com> wrote:

Noel,

Thanks for the info. I'll certainly have a look at what the subclipse folks did as I am sure it will be helpful to understand the strategy. I do think we are in a slightly different situation here as we don't quite have a stable library interface to git yet - Brad Roberts work notwithstanding.

The repository API in pure Java is almost a no-brainer since Linus has done such a good job in keeping the repository specification simple and unambiguous.

A Java workspace API can take advantage of the abstraction and GUI facilities that Java and Eclipse afford so will naturally be different in form to the existing command line tools for manipulating the git index. Certainly, there will be similarities - aspects of the 3-way merge, for example, but there will be differences too - workspace change detection will be somewhat assisted by the change notification framework in Eclipse and won't require as much manual intervention.

At this stage, my thoughts are to implement a listener pattern to keep the git index up to date, and I may well implement that by calling out the the C git-update-cache executable. This can run in a background thread so it needn't be a huge drag on interactive user performance.

Anyway, thanks for your input.

jon.

I've got exactly this problem in Darcs-git; and I ignore all of the cached data except the file size, mtime and sha1. I don't currently ever write to the cache.

There's nothing that prevents you from ignoring the Git cache and using your own cache instead.

                                        Juliusz

On 5/11/05, David Woodhouse <dwmw2@infradead.org> wrote:

I did consider wrapping it - I really did. But after thinking about it for a couple of weeks I eventually came to the conclusion it would be a sub-optimal solution.

And I don't agree that the structure is volatile. The actual structure of GIT repository has been rock-solid since Linus reversed the compression-signature order. Yes, there has been the excellent delta work that Nicholas has been working on, but a Java version doesn't need to use that anymore than Linus himself does [ and last time I checked (the mailing list), Linus was being rather conservative about that ].

What _has_ been changing at a great rate recently is the behaviour of the tooling layer.

So, if I want a stable foundation to build my stuff on, basing it on the output of the C tools would be a huge mistake. No, I think it would be far safer for me to build my tooling using assumptions that have proven to be rock-solid over the last few weeks - the structure of the GIT repository format itself.

David, I have nothing whatsoever to do with the Eclipse implementation of SSH, so what exactly is your point? All Java programmers are fundamentally brain-damaged? Grow up, please.

jon.

On Wed, 2005-05-11 at 03:15 +1000, Jon Seymour wrote:

That all programmers of _any_ persuasion who want to reimplement the world in their own language instead of using existing tools are fundamentally brain-damaged.

I'd say the same if the KDevelop people wanted to rewrite git and ssh in C++, too.

I don't think that's such a childish observation, surely? Piecing together tools which do one thing, and do it well, is part of the UNIX philosophy.

On 5/11/05, David Woodhouse <dwmw2@infradead.org> wrote:

This might surprise you, but everything in the world is not a unix command line. Using tools designed for use on a unix command line in a highly interactive environment like Eclipse would produce an abominably clunky result.

Presumably you think this is the Unix way.

I have already explained all the pragmatic reasons for doing a GIT implementation in Java but you are prepared to ignore all of those reasons. You have ignored all these reasons rather than lift a finger to compose a well-reasoned rebuttal.

Instead you cling to a dogmatic insistence that GIT respositories only be accessed via the C toolsets.

David, you are being extremely childish and extremely dogmatic. Why bother?

jon

On Wed, May 11, 2005 at 03:36:29AM +1000, Jon Seymour wrote:

The unix commandline is an highly interactive enviroment aswell, and supports scripting in addition :)

You tried to argue for re-inventing the wheel. Fortunately you are allowed to reinvent the wheel here (which isn't given anymore these days). Just don't expect any support from people who have been burnt by that before. And the Java world is re-inventing the wheel far to often - I suspect that'll be cured when the community gets more mature in a few years..

The C toolchain is a well-defined interface. One that makes a lot of sense.

On Wed, 11 May 2005, Jon Seymour wrote:

Can Java use a library that's implemented in C, like Python and Perl can? If that is the case, the C implementation of git simply needs the ability to be called as a library and you can implement Java bindings for it.

That should also make it easier to create front-ends in other languages...

On 5/11/05, Rik van Riel <riel@redhat.com> wrote:

It can in principle, yes. This is the so-called Java-Native-Interface (JNI).

I think in the longer term it would make sense to write a JNI layer but the GIT source code is probably a little too unstable for that now. What needs to happen first, I think, is that a solid and stable C API (e.g. a libgit) be proposed and developed.

One outcome of the work I am doing (a long time away, probably will never happen) might eventually be to propose an C API for GIT based, in part, on the prototyping work I do in Java. However, I suspect its Java-tainted heritage would mean instant death to it for purely dogmatic reasons so I think it I'll leave it to others to propose a C-API and then write a JNI wrapper for that when/if it ever happens.

Agreed.

jon.

Dear diary, on Wed, May 11, 2005 at 04:09:49AM CEST, I got a letter where Jon Seymour <jon.seymour@gmail.com> told me that...

FWIW, this is actually happening now (see Brad Roberts' posts in nearby thread).

On 5/11/05, Christoph Hellwig <hch@infradead.org> wrote:

I am _not_ re-inventing any wheel. Merely building a road through different territory upon which the GIT wheel may roll more freely.

Don't get me wrong. I appreciate the UNIX mindset of building small tools and composing them freely. But the UNIX way isn't the only way to interact with the world and it is being hopelessly dogmatic to insist otherwise.

jon. -- homepage: http://www.zeta.org.au/~jon/ blog: http://orwelliantremors.blogspot.com/

On Wed, 11 May 2005, Jon Seymour wrote:

Would you mind fixing your mailer before, so that your messages won't appear as new Subject: every time?

- Davide

I guess I agree it is somewhat nonsensical if one considers the current git toolset as a collection of programs - the exercise now might simply reduce to classifying git tools as index-using/non-index-using and nothing more. However, it might be worth keeping in mind when/if the "libification" of git happens so that there is a clean separation of layers in the API between the repository API and index/cache/workspace API.

And thanks for your constructive criticism.

jon. -- homepage: http://www.zeta.org.au/~jon/ blog: http://orwelliantremors.blogspot.com/

Dear diary, on Wed, May 11, 2005 at 01:05:55AM CEST, I got a letter where Jon Seymour <jon.seymour@gmail.com> told me that...

In that case the repository API's input/output is data structure equivalent to the index, and workspace API's input/output is the index too. That is what it really is - and it is kept on the disk only since the commands are invoked separately so it needs to keep the state around somewhere.

On 5/11/05, Petr Baudis <pasky@ucw.cz> wrote:

Not sure I agree...

The repository API would contain functionality equivalent to cat-file, ls-tree, most of fsck-cache, rev-list, rev-tree, diff-tree, most of the transport code - things that don't involve use of the index.

The workspace API would contain read-tree, write-tree, commit-tree, etc - things that do involve use of the the index.

jon.

On Wed, 11 May 2005, Jon Seymour wrote:

Unfortunately for this idea, you can't actually check files into or out of the repository using the git tools without the index (in memory at least, if not on disk). This is a bit like having a libc with all the system calls except read and write. Sure, there are a number of programs that would be fine that way, but it makes the API unintuitive, and most serious programs need the extension anyway.

	-Daniel
*This .sig left intentionally blank*

On 5/12/05, Daniel Barkalow <barkalow@iabervon.org> wrote:

Daniel,

Checking in/checking out is definitely a workspace API function that involves an index (almost certainly) an on-disk index, but reading and writing from and to the repository is purely a repository API function.

There is no reason why the workspace API can't make use of "lower-level" functions in the repository API. I guess I would argue that the repository API really shouldn't need much, if any knowledge, of the workspace API.

The repository API knows about the on-disk structure of repository objects - it knows how to write writing compressed, SHA1 signed objects, it also knows how to pack and unpack the various object types from their on-disk representation into structures manipulated by the workspace API.

On the otherhand the workspace API knows about managing index (workspace) state. It knows how to determine whether the pre-conditions for a commit have been reached. It knows how to read the workspace structures and coordinate read and writes from and to the repository.

I really don't think this makes the API unintuitive, but then I don't really understand why you think this is the case.

jon. -- homepage: http://www.zeta.org.au/~jon/ blog: http://orwelliantremors.blogspot.com