linking libgit.a in C++ projects

I'm writing a git gui for OS X using cocoa/Objective-C++, and rather than being lame and parsing the output the various git commands, I'm using libgit.a to provide all of the needed functionality for my app. However, the git source uses a few reserved C++ keywords; namely 'typename', and 'new'. So, I was wondering if it is worth submitting a patch to fix these issues... I'm asking because I'm new to the whole open source thing, and I don't want to get yelled at by the git maintainers for submitting stupid patches that no one in their right mind would accept :)

Thanks!

On Thu, Jul 31, 2008 at 02:53:37AM -0700, cte wrote:

Don't do that! libgit.a is an internal library used solely to build git binaries. It means that its interface can be cahnged at any time. Though, there is an idea of creating the real git library that other applications can use, but AFAIK no one is working on it. So parsing output is the only correct solution right now. In fact, it is not difficult to do, because most plumbing commands are rather flexibly in what they output and how.

Because this source code are meant to be compiled by C and not by C++! Even if we will have real git library for other applications to use, it still be compiled only by C. Thus, C++ keywords are not issue.

Dmitry

On Thu, Jul 31, 2008 at 3:57 AM, Dmitry Potapov <dpotapov@gmail.com> wrote:

I'm not worried about the interfaces changing; the gui is tied to a particular version of git, and I will update the code that calls into libgit I pull new changes from the mainline into my local clone. Also, who's to say that the output of the various commands won't change formats with future releases of git? There is no correct solution if you are worried about forward compatibility, unless a well defined API is created (which would be sweet btw, but is probably not a priority).

Clearly ;)

Fortunately, g++ can compile C programs and link static libraries that were compiled by C compilers, unless of course, they use C++ keywords. I don't think it is unreasonable to rename the _very few_ C++ keywords in git's source in the interest of allowing C++ projects to leverage libgit.

On Jul 31, 2008, at 12:10 PM, cte wrote:

[...]

I think the point Dmitry was trying to make is that you should compile libgit as C, using gcc, and then link it with your C++/Objective C code.

No patch is required to git, only to your makefile/xcode project file.

Best regards,

On Thu, Jul 31, 2008 at 12:16:45PM +0100, Pedro Melo wrote:

libgit has a certain (albeit currently unofficial and non-settled) API and the API is defined in header files that must be eatable by C++ compilers in order to do this.

				Petr "Pasky" Baudis

The git .h files must be in your include path, and must not contain C++ keywords in order to link against libgit.a.

On Thu, Jul 31, 2008 at 13:10, cte <cestreich@gmail.com> wrote:

You should be ;). Unless you are planning to learn a lot of C very fast, you should be worried about the interfaces changing. That is, if you want your GUI to be able to stay up to date with the current git version.

Junio is to say. Plumbing output format is git's API.

There is, use the plumbing, forward compatibility is 95% assured. With the exception of major releases, for which any plumbing output/behavior changes will be announced in the changelog, usually including an explanation on how to change your code to match.

In short, use the forc-... errr, plumbing ;).

On Thu, Jul 31, 2008 at 11:27 AM, Sverre Rabbelier <alturin@gmail.com> wrote:

That is the plan.

Using output from the command line utilities as an API has its own set of problems. For instance, check out some of the difficulties that gitk and qgit have had to deal with: http://kerneltrap.org/mailarchive/git/2007/11/2/379067. Digging into the git internals and reusing its core functions will always be more powerful and flexible than parsing command line output. Of course, it is not always easy; git wasn't written to be easily compiled into a library and reused (graceful error handling and memory management are problematic). But I think the right thing to do is to work towards making the awesome git internals easier to use for other developers so great tools can continue to be built on top of git.

95% assured != correct, IMO :)

On Thu, Jul 31, 2008 at 23:44, cte <cestreich@gmail.com> wrote:

I beg to differ. If I skimmed the topic correctly, the problems there were not related to having to parse git's output, but due to the fact that '--topo-order' is a post-processing operation, which takes long. Do read the recent discussion between Linus and Roman about that.

Sure, but is it worth it? What do you need in your GUI that you cannot get from the plumbing?

I do agree with that, libification of git would be really nice. Especially since that'd mean that integrating it into other languages (by means of wrappers), such as Python or Ruby, becomes a lot easier.

Why not? Junio has a very good reputation of keeping git backwards compatible. The 95% is of course not an actual figure but an expression meant to indicate "statement is true, minus a few rare case exceptions".

It's ok to remove text that you do not respond to, including signatures :P.

Sverre Rabbelier <alturin@gmail.com> wrote:

And actually if you try to use topo-order internally in C you still
have to wait for the post-processing.  Which is going to cause your
UI to lock up because it is single-threaded as both Git and your UI
toolkit are probably single threaded.  At least by forking out to
git-rev-list your UI can respond while the computation is happening
in a background process.
 

I'm going to be shot for saying this, but both Python and Ruby
have implementations that run on the JVM.  So does Git.  Want
to use Git and Python?  Use JGit and Jython.  :)
 

Too many people have scripts based upon plumbing to make incompatible changes. We'd have all of our users screaming. Remember many Git users are programmers themselves, they will make small home-grown scripts based upon Git plumbing to simplify their workflow and everyday tasks. They use plumbing precisely because they can trust it won't change or break on them.

On Thu, Jul 31, 2008 at 23:58, Shawn O. Pearce <spearce@spearce.org> wrote:

Aye, that's what I was reffering to, thanks for clarifying :).

Heheh, nice plug :P, but thanks but no thanks. I'd rather have something more native than "JGit + Jython", two levels of 'emulation' can't be good!

Exactly! :)

On Thu, Jul 31, 2008 at 11:58 PM, Shawn O. Pearce <spearce@spearce.org> wrote:

There is also the "stores" branch from pygit [1] that implements pack and object reading natively in python. Anyway implementing more than that (ie, packing and such things) natively is clearly harder and takes more work for less efficiency than making bindings around a libified git...

[1] http://code.istique.net/?p=pygit.git;a=shortlog;h=refs/heads/stores Don't mind the 50X error, just refresh... broken server.

Didn't mean to imply that somehow it is no longer a post-processing op if you aren't using git plumbing. The discussion shows, however, that if gitk was actually doing the revision traversals, then it would be able to trigger events that update the gui whenever it wanted, which would have allowed it to implement the early output feature without changing any of the git source. Instead, git-log had to be altered to address gitk's needs, and an option was added that users don't typically use. This is not exactly what I would consider spartan programming (http://www.codinghorror.com/blog/archives/001148.html), plus there are already too many options to remember! Anyways, I suppose it is pointless to argue about which approach is better, because both have trade-offs, and the correct path depends on your use case.

Definitely not questioning his ability to maintain backwards compatibility; it was merely an observation about your strange definition of correct. In school, when I completed 95% of a proof, it was never marked as "correct", and I was told that I hadn't actually proven anything. Those damn teachers :)

On Thu, 31 Jul 2008, cte wrote:

Actually, no. 

I ended up doing a hack to add "git log --early-output", and some quick hacks to make git use it. But I'm happy to say that gitk doesn't use it any more. Gitk just parses the normal git log output, although obviously it ends up needing enough data to be able to fill in the gaps (ie it uses the "--parents" flag to get the rewritten parenthood info and the merges to keep it together).

So yeah, we have some options that enable output that simply doesn't make _sense_ to humans, but does when you are post-processing it (git has always had those, since it very much was about scripting from day one), but no, at least gitk doesn't use any really odd ones.

It literally does

	git log --no-color -z --pretty=raw --parents --boundary

(plus the args the user gave it). The --no-color turns off color if it's enabled by default.

The -z makes it the git log output a bit easier to parse by using a NUL character between commits.

The --pretty=raw is just to give the full/raw commit info - like giving the timestamps in the native raw format that is much easier to parse than any human-readable format.

The --parents I already mentioned - it is what makes you able to stitch the commits together (and it's useful for other things too: any scripts that look for merges will tend to use it, for example)

And the --boundary is to show the commits that aren't part of the actual selected set in gray.

It's all pretty generic, in other words. The -z option is purely for machine parsing, the others _can_ actually be useful even for humans (eg, --boundary together with --left-right and --pretty=oneline actually is very readable if you are used to that format).

So it's very true that git in general is geared towards scripting (and we've often added things to make it even more so), but no, your particular complaint isn't true. gitk doesn't do anything really strange.

		Linus

Yeah, I guess that's why I like using git so much; a few piped commands and a script or two, and you can do some pretty rad stuff.

On Fri, Aug 1, 2008 at 1:44 AM, cte <cestreich@gmail.com> wrote:

There is no problem with parsing. If you want to receive the output in the specific order, Git has to read everything first, and that is *slow*. So, --topo-order is convenient but slow, and it is slow not because it is piping data, but because it takes some time to read the whole history.

"Flexible" is not a synonym of the word "useful". For instance, using core functions will not help you to overcome the aforementioned problem. Drawing a graph is NOT what git core functions about. You have to do that in your GUI, and to do that when revisions are given to you in arbitrary order is not easy. Yet, it is something what good GUI should be capable to handle, because otherwise the response time will be bad.

Dmitry

On Thu, Jul 31, 2008 at 02:57:27PM +0400, Dmitry Potapov wrote:

I don't think this is that big a problem; there are applications that are doing this already, e.g. cgit, and if you tie your application to a particular git version by for example making git a submodule of your source, this is pretty safe; it will just mean that you will have to do some non-trivial porting of your code to the new interface each time you update - but I think large changes in the interface are pretty rare in practice by now, and there shouldn't be much on the horizon either(?).

What would be the reason to disallow C++ users? The costs aren't that high, and (modulo, say, extern "C" { }) there should be no C-C++ compatibility issues, right?

Exactly. It works just great for me in XCode 3.1 on OS X.

On Thu, Jul 31, 2008 at 01:14:46PM +0200, Petr Baudis wrote:

What you see as large changes depend on how well you know git internals. Git develops very quickly and if someone who is trying to use libgit.a does not follow git development closely, it may happen pretty soon that even not so big changes will become a huge problem to accomadate them. As result, the program may stick with an old Git version, and that puts users of this program in the situation where they cannot use their favorite frontend with new repositories.

I mean that putting extern "C" { } around should be sufficient to use this library in C++. But I see now some current headers contains some C++ keywords and that causes the problem. So, yes, those headers should be corrected if they become part of external available API. I am not sure whether it makes sense to correct them now, but there are only three places where C++ keywords are used:

diff.h:135:extern int diff_tree_sha1(const unsigned char *old, const diff.h:137:extern int diff_root_tree_sha1(const unsigned char *new, object.h:38:extern const char *typename(unsigned int type);

So, the patch should not be large, and it is up to Junio to decide what to do about it.

Dmitry

cte wrote:

The practice of avoiding C++ keywords from public C headers is very welcome. You should send a patch and try to push it.

That said the problem can be easily avoided.

Produce a C file and header that defines some stable API to your GUI application, that does not expose any git internal headers. Then compile that, say git_api.c, with C compiler in Makefile and extern "C" link that file to your C++ application. This will completely insulate you from any git code.

This could also solve the other problem of API changing, only the git_api.c need change, your outer GUI code stays the same.

And if you do all that maybe you can submit it for inclusion as a: somewhat stable high-level library, for developers. Ala git-dev

Cheers Boaz

On Thu, Jul 31, 2008 at 04:04:50PM +0300, Boaz Harrosh wrote:

While the idea of creating of such a wrapper makes sense, it may not easy to implement properly in all cases, because some git functions do not free allocated memory as they rely on this memory being free at exit. There is a problem with die() as it is not enough to longjmp from it, you have to free all resources that were allocated (such as open files, memory). Thus writing a full functional library, which will not leak resources, is not an easy task. Of course, some of Git function can be used easily without risk that your application will leak resources.

Dmitry

Boaz Harrosh, Thu, Jul 31, 2008 15:04:50 +0200:

no, it wont. He still have to resolve name conflicts at the link time.

On 7/31/08, Alex Riesen <raa.lkml@gmail.com> wrote:

Language keywords (as opposed to function names) like 'new' and 'typename' are definitely not exported to the object files. Moreover, function parameter names aren't either.

Avery

Avery Pennarun, Thu, Jul 31, 2008 20:55:26 +0200:

Didn't mean them. Meant the globally visible names. libgit does not use a prefix for its exported symbols. They will clash with the symbols of the programs it is linked to.

Alex Riesen wrote:

But that's a problem for C programs, C++ programs will not have that problem, right? ;-)

With C programs these git symbols can be avoided.

Boaz

Excellent idea; thank you!