threads / discuss / 43117

Re: jgit performance update

Subject: Re: jgit performance update

## tl;dr

16 messages between Dec 3, 2006 and Dec 4, 2006.

replies: 15people: 6as markdown or json

Shawn Pearce· Dec 3, 2006, 04:59 UTC · lore

jgit performance update

With the help of Robin Rosenberg I've been able to make jgit's log operation run (on average) within a few milliseconds of core Git.

Walking the 50,000 most recent commits from the Mozilla trunk[1]:
  $ time git rev-list --max-count=50000 HEAD >/dev/null
  core Git:  1.882s (average)
  jgit:      1.932s (average)
  (times are with hot cache and from repeated executions)

I think that is actually pretty good given that jgit is written in Java using a fairly object-oriented design and has to deal with some of the limitations of the language.

One of the biggest annoyances has been the fact that although Java 1.4 offers a way to mmap a file into the process, the overhead to access that data seems to be far higher than just reading the file content into a very large byte array, especially if we are going to access that file content multiple times. So jgit performs worse than core Git early on while it copies everything from the OS buffer cache into the Java process, but then performs reasonably well once the internal cache is hot. On the other hand using the mmap call reduces early latency but hurts the access times so much that we're talking closer to 3s average read times for the same log operation.

Anyway, jgit is now hopefully fast enough that we can start to build some real functionality on top of it, and not need to wait several minutes for answers from those features while debugging them. :)

**1** This is the pack file from Jon Smirl's import attempt.
Robin Rosenberg· Dec 3, 2006, 13:55 UTC · re: Shawn Pearce · lore
söndag 03 december 2006 05:59 skrev Shawn Pearce:
Show 11 quoted lines
> With the help of Robin Rosenberg I've been able to make jgit's log
> operation run (on average) within a few milliseconds of core Git.
>
> Walking the 50,000 most recent commits from the Mozilla trunk[1]:
>
>   $ time git rev-list --max-count=50000 HEAD >/dev/null
>
>   core Git:  1.882s (average)
>   jgit:      1.932s (average)
>
>   (times are with hot cache and from repeated executions)
Nice indeed. That was a ten-fold improvement for getting my full history.

So, just go on to the next case. I added filtering on filenames (yes, CVS-induced brain damage, I should track the content. next version. filenames are so much handier to work with). That gives me 4.5s to retrieve a filtered history (from 10800 commits).Half of the time is spent in re-sorting tree entries. Is that really necessary?

> I think that is actually pretty good given that jgit is written
> in Java using a fairly object-oriented design and has to deal with
> some of the limitations of the language.

Most of java's slowness comes from the programmers using it. (Lutz Prechelt. Technical opinion: comparing Java vs. C/C++ efficiency differences to interpersonal differences. ACM, Vol 42,#10, 1999)

Show 10 quoted lines
> One of the biggest annoyances has been the fact that although Java 
> 1.4 offers a way to mmap a file into the process, the overhead to
> access that data seems to be far higher than just reading the file
> content into a very large byte array, especially if we are going
> to access that file content multiple times.  So jgit performs worse
> than core Git early on while it copies everything from the OS buffer
> cache into the Java process, but then performs reasonably well once
> the internal cache is hot.  On the other hand using the mmap call
> reduces early latency but hurts the access times so much that we're
> talking closer to 3s average read times for the same log operation.
Have you tried that with difference JVM's?
Jakub Narebski· Dec 3, 2006, 14:19 UTC · re: Robin Rosenberg · lore
Robin Rosenberg wrote:
Show 18 quoted lines
> söndag 03 december 2006 05:59 skrev Shawn Pearce:
>
>> With the help of Robin Rosenberg I've been able to make jgit's log
>> operation run (on average) within a few milliseconds of core Git.
>>
>> Walking the 50,000 most recent commits from the Mozilla trunk[1]:
>>
>>   $ time git rev-list --max-count=50000 HEAD >/dev/null
>>
>>   core Git:  1.882s (average)
>>   jgit:      1.932s (average)
>>
>>   (times are with hot cache and from repeated executions)
> Nice indeed. That was a ten-fold improvement for getting my full history. 
> 
> So, just go on to the next case. I added filtering on filenames (yes, 
> CVS-induced brain damage, I should track the content. next version. filenames 
> are so much handier to work with).

Git uses <path> as _revision limiter_, not as output filter. Shouldn't jgit do the same?

P.S. What is the status of --follow option?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Robin Rosenberg· Dec 3, 2006, 15:53 UTC · re: Jakub Narebski · lore
söndag 03 december 2006 15:19 skrev Jakub Narebski:
Show 6 quoted lines
> Robin Rosenberg wrote:
> > CVS-induced brain damage, I should track the content. next version.
> > filenames are so much handier to work with).
>
> Git uses <path> as _revision limiter_, not as output filter. Shouldn't
> jgit do the same?

It's egit, i.e. the eclipse plugin I'm referring to so it's a user interface thing and it uses the path name.

Shawn Pearce· Dec 3, 2006, 23:06 UTC · re: Robin Rosenberg · lore
Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
Show 9 quoted lines
> söndag 03 december 2006 15:19 skrev Jakub Narebski:
> > Robin Rosenberg wrote:
> > > CVS-induced brain damage, I should track the content. next version.
> > > filenames are so much handier to work with).
> >
> > Git uses <path> as _revision limiter_, not as output filter. Shouldn't
> > jgit do the same?
> It's egit, i.e. the eclipse plugin I'm referring to so it's a user interface 
> thing and it uses the path name.

Jakub's point is that "git log -- a" lists only the revisions which affect path 'a'. Once those revisions have been selected then output begins. The 'a' gets reapplied as an output filter to only show changes relevant to file 'a', but its very much a revision filter during the revision listing process.

Shawn Pearce· Dec 3, 2006, 22:59 UTC · re: Robin Rosenberg · lore
Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
Show 5 quoted lines
> So, just go on to the next case. I added filtering on filenames (yes, 
> CVS-induced brain damage, I should track the content. next version. filenames 
> are so much handier to work with). That gives me 4.5s to retrieve a filtered 
> history (from 10800 commits).Half of the time is spent in re-sorting tree 
> entries. Is that really necessary?

Yea, I was looking at that code while doing the other performance improvements and thought it might start to become a bottleneck. I guess I was right.

What is happening here is jgit wants to store the items in the tree in name ordering, but Git stores the items in the tree sorted such that subtrees sort with a '/' on the end of their name. This is a different ordering...

The reason I'm resorting them is so we can find an entry without
knowing what its type is first.  Looks like that's going to have
to change somehow.
 
> Most of java's slowness comes from the programmers using it. (Lutz Prechelt. 
> Technical opinion: comparing Java vs. C/C++ efficiency differences to 
> interpersonal differences. ACM, Vol 42,#10, 1999)
Yes, that was clearly the case here with jgit!  :-)
_This_ programmer made jgit slow.  Learned from the mistake, and
made it faster.
Show 12 quoted lines
> > One of the biggest annoyances has been the fact that although Java 
> > 1.4 offers a way to mmap a file into the process, the overhead to
> > access that data seems to be far higher than just reading the file
> > content into a very large byte array, especially if we are going
> > to access that file content multiple times.  So jgit performs worse
> > than core Git early on while it copies everything from the OS buffer
> > cache into the Java process, but then performs reasonably well once
> > the internal cache is hot.  On the other hand using the mmap call
> > reduces early latency but hurts the access times so much that we're
> > talking closer to 3s average read times for the same log operation.
> 
> Have you tried that with difference JVM's?

No, I'm on Mac OS X so I don't have a huge JVM selection (that I know of). And I haven't tried jgit or egit on any other system yet.

Linus Torvalds· Dec 3, 2006, 17:45 UTC · re: Shawn Pearce · lore
On Sat, 2 Dec 2006, Shawn Pearce wrote:
>
> With the help of Robin Rosenberg I've been able to make jgit's log
> operation run (on average) within a few milliseconds of core Git.
Very good. Are we any closer to actually having an eclipse plugin then?

Not that I've ever actually used eclipse, but maybe I should try it, just to see what those strange user-land people actually do. I'll be a veritable Jane Goodall..

Show 8 quoted lines
> Walking the 50,000 most recent commits from the Mozilla trunk[1]:
> 
>   $ time git rev-list --max-count=50000 HEAD >/dev/null
> 
>   core Git:  1.882s (average)
>   jgit:      1.932s (average)
> 
>   (times are with hot cache and from repeated executions)

Now, the _interesting_ case in many ways is not "--max-count", but the revision limiter. It _should_ be equally fast, but if you've done something wrong, it won't be.

IOW, try to find a point far enough back in time to get about the same number of commits, and then do

	time git rev-list <thatpoint>..HEAD >/dev/null

because one of the things you want to handle is ranges, more so than simple counts. And that is not only the much more common case, it also triggers a few cases that you probably didn't trigger with the regular "list the first 50 thousand commits" case.

Show 5 quoted lines
> One of the biggest annoyances has been the fact that although Java
> 1.4 offers a way to mmap a file into the process, the overhead to
> access that data seems to be far higher than just reading the file
> content into a very large byte array, especially if we are going
> to access that file content multiple times.

That must suck for big packed repositories. What JVM and other environment are you using?

Also, I have to say, one of the reasons I'm interested in your project is that I've never done any Java programming, because quite frankly, I've never had any reason what-so-ever to do so. But if there is some simple setup, and you have jgit exposed somewhere as a git archive, I'd love to take a look, if only to finally learn more about Java.

Jakub Narebski· Dec 3, 2006, 17:56 UTC · re: Linus Torvalds · lore
Linus Torvalds wrote:
Show 5 quoted lines
> Also, I have to say, one of the reasons I'm interested in your project is 
> that I've never done any Java programming, because quite frankly, I've 
> never had any reason what-so-ever to do so. But if there is some simple 
> setup, and you have jgit exposed somewhere as a git archive, I'd love to 
> take a look, if only to finally learn more about Java.
GitWiki tells us about egit/jgit repository at
  http://www.spearce.org/projects/scm/egit.git
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Juergen Stuber· Dec 3, 2006, 22:42 UTC · re: Jakub Narebski · lore
Hi Jakub,
Jakub Narebski <jnareb@gmail.com> writes:
>
> GitWiki tells us about egit/jgit repository at
>   http://www.spearce.org/projects/scm/egit.git
I tried to access that with git 1.4.4.1 from Debian but 
% git clone http://www.spearce.org/projects/scm/egit.git

hangs, the first time after "walk e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just before it.

There's also the following error shortly after the start:
error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50 (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
Jürgen
-- 
Jürgen Stuber <juergen@jstuber.net>
http://www.jstuber.net/
gnupg key fingerprint = 2767 CA3C 5680 58BA 9A91  23D9 BED6 9A7A AF9E 68B4
Robin Rosenberg· Dec 3, 2006, 23:39 UTC · re: Juergen Stuber · lore
söndag 03 december 2006 23:42 skrev Juergen Stuber:
Show 14 quoted lines
> Hi Jakub,
>
> Jakub Narebski <jnareb@gmail.com> writes:
> > GitWiki tells us about egit/jgit repository at
> >   http://www.spearce.org/projects/scm/egit.git
>
> I tried to access that with git 1.4.4.1 from Debian but
>
> % git clone http://www.spearce.org/projects/scm/egit.git
>
> hangs, the first time after "walk
> e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
> 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
> before it.
Works fine here. (git 1.4.4.gf05d).
Show 6 quoted lines
>
> There's also the following error shortly after the start:
>
> error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50
> (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41e
>bbd87d8f8968529e9c30) corrupt

Unfortunately, messages about corrupt objects are "normal" with clone over http. I'm not sure it has to be that way though. Run git-fsck-objects to make sure there are no errors. The hangs aren't normal.

-- robin
Jakub Narebski· Dec 3, 2006, 23:58 UTC · re: Robin Rosenberg · lore
Robin Rosenberg wrote:
Show 17 quoted lines
> söndag 03 december 2006 23:42 skrev Juergen Stuber:
>>
>> Jakub Narebski <jnareb@gmail.com> writes:
>>>
>>> GitWiki tells us about egit/jgit repository at
>>>   http://www.spearce.org/projects/scm/egit.git
>>
>> I tried to access that with git 1.4.4.1 from Debian but
>>
>> % git clone http://www.spearce.org/projects/scm/egit.git
>>
>> hangs, the first time after "walk
>> e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
>> 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
>> before it.
>
> Works fine here. (git 1.4.4.gf05d).
Works fine here. (git 1.4.4.1)
Show 9 quoted lines
>> There's also the following error shortly after the start:
>>
>> error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50
>> (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41e
>>bbd87d8f8968529e9c30) corrupt
> 
> Unfortunately, messages about corrupt objects are "normal" with clone over 
> http. I'm not sure it has to be that way though. Run git-fsck-objects to make 
> sure there are no errors. The hangs aren't normal.
I got:
$ git clone http://www.spearce.org/projects/scm/egit.git
[...]
 got 73ed47b2bb1fa5978f7368775979e5c85d354c5a
 error: File 2332eacf114debb7a27d138811197f06eb262551 
 (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
 Getting pack list for http://www.spearce.org/projects/scm/egit.git/
 got afefbe09bacc08adb75fb46200a973001c6b02de
[...]
 walk c1f287cb19b9910af19756cf29c08b1fda75da8c
 Some loose object were found to be corrupt, but they might be just
 a false '404 Not Found' error message sent with incorrect HTTP
 status code.  Suggest running git fsck-objects.
 got eab86de8ac23e2e77878835007724146fdd83796
$ git fsck-objects --unreachable --full --strict   ;# returns no errors
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Shawn Pearce· Dec 4, 2006, 00:46 UTC · re: Jakub Narebski · lore
Jakub Narebski <jnareb@gmail.com> wrote:
Show 15 quoted lines
> I got:
> $ git clone http://www.spearce.org/projects/scm/egit.git
> [...]
>  got 73ed47b2bb1fa5978f7368775979e5c85d354c5a
>  error: File 2332eacf114debb7a27d138811197f06eb262551 
>  (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
>  Getting pack list for http://www.spearce.org/projects/scm/egit.git/
>  got afefbe09bacc08adb75fb46200a973001c6b02de
> [...]
>  walk c1f287cb19b9910af19756cf29c08b1fda75da8c
>  Some loose object were found to be corrupt, but they might be just
>  a false '404 Not Found' error message sent with incorrect HTTP
>  status code.  Suggest running git fsck-objects.
>  got eab86de8ac23e2e77878835007724146fdd83796
> $ git fsck-objects --unreachable --full --strict   ;# returns no errors

Right. My web server doesn't send back 404 messages when an object isn't found (but is packed). Thus the error...

I've setup a project on repo.or.cz:
  http://repo.or.cz/w/egit.git
Juergen Stuber· Dec 4, 2006, 20:35 UTC · re: Robin Rosenberg · lore
Hej Robin,
Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:
Show 14 quoted lines
> söndag 03 december 2006 23:42 skrev Juergen Stuber:
>> Jakub Narebski <jnareb@gmail.com> writes:
>> > GitWiki tells us about egit/jgit repository at
>> >   http://www.spearce.org/projects/scm/egit.git
>>
>> I tried to access that with git 1.4.4.1 from Debian but
>>
>> % git clone http://www.spearce.org/projects/scm/egit.git
>>
>> hangs, the first time after "walk
>> e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
>> 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
>> before it.
> Works fine here. (git 1.4.4.gf05d).
now it works fine for me, too.
Tack
Jürgen
-- 
Jürgen Stuber <juergen@jstuber.net>
http://www.jstuber.net/
gnupg key fingerprint = 2767 CA3C 5680 58BA 9A91  23D9 BED6 9A7A AF9E 68B4
Shawn Pearce· Dec 3, 2006, 22:47 UTC · re: Linus Torvalds · lore
Linus Torvalds <torvalds@osdl.org> wrote:
> Very good. Are we any closer to actually having an eclipse plugin then?
We have some parts in place.  But nothing that's user-ready.
Show 5 quoted lines
> > Walking the 50,000 most recent commits from the Mozilla trunk[1]:
> 
> Now, the _interesting_ case in many ways is not "--max-count", but the 
> revision limiter. It _should_ be equally fast, but if you've done 
> something wrong, it won't be.

We haven't implemented a rev-list equivalent yet. That's a major feature which is missing. The --max-count test was simple to put together. I really need to start thinking about replicating some of the features of the revision functions in core Git.

Show 8 quoted lines
> > One of the biggest annoyances has been the fact that although Java
> > 1.4 offers a way to mmap a file into the process, the overhead to
> > access that data seems to be far higher than just reading the file
> > content into a very large byte array, especially if we are going
> > to access that file content multiple times.
> 
> That must suck for big packed repositories. What JVM and other environment 
> are you using?

Mac OS 10.4.8 / Java 1.4.2. It appears as though Sun isn't going to fix the mmap performance problems as they can't do it "securely". So its likely to be an issue anywhere jgit gets used...

Show 5 quoted lines
> Also, I have to say, one of the reasons I'm interested in your project is 
> that I've never done any Java programming, because quite frankly, I've 
> never had any reason what-so-ever to do so. But if there is some simple 
> setup, and you have jgit exposed somewhere as a git archive, I'd love to 
> take a look, if only to finally learn more about Java.
Of course its in Git.  :-)
There's no webpage here, but you can clone it:
  http://www.spearce.org/projects/scm/egit.git

The code you would recognize most is in org.spearce.jgit/.../lib. E.g. BinaryDelta.java came from patch-delta.c in core Git. Repository has some of the functionality of sha1_file.c, the pack reading code is in PackFile.java.

sf· Dec 3, 2006, 21:55 UTC · re: Shawn Pearce · lore

Shawn Pearce wrote: ...

> One of the biggest annoyances has been the fact that although Java
> 1.4 offers a way to mmap a file into the process,

Be careful with mmap: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038

Regards
Stephan
Shawn Pearce· Dec 3, 2006, 22:16 UTC · re: sf · lore
sf <sf-gmane@stephan-feder.de> wrote:
Show 7 quoted lines
> Shawn Pearce wrote:
> ...
> > One of the biggest annoyances has been the fact that although Java
> > 1.4 offers a way to mmap a file into the process,
> 
> Be careful with mmap:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038

Thanks. That particular bug has been of some discussion on #git before. I'm planning on making it a configuration flag in .git/config for the user to decide how much pain they want: mmap (with all its huge downsides) or read into byte[] (with all the memory footprint that requires).

← back to recent threads