threads / discuss / 11260

Efficient retrieval of commit log info

Subject: Efficient retrieval of commit log info

## tl;dr

8 messages between Dec 12, 2007 and Dec 12, 2007.

replies: 7people: 6as markdown or json

Eirik Bjørsnøs· Dec 12, 2007, 14:36 UTC · lore

Hi,

I'm developing a piece of software that grabs logs from various types of SCMs and presents (hopefully) useful information about the history.

My current approach with Git is do a "git clone --n" and then parse the output of "git log". To check for updates I do a "git pull" followed by a new "git log".

This approach works fine, but cloning the whole repository just to get the change log seems like a somewhat inefficient use of bandwidth and storage.

What I would like to do is to fetch just the change log information from the remote repository.

(Using the "CVS done right" tool I can do this with: "svn log --xml -v -r<last, HEAD> http://svn.example.com/")

I haven't found a way to do this using the Git command line tools.

Is it at all possible using the git network protocols to fetch just the commit log info, without transferring the contents?

I'd be happy to hear from anyone who has some insight into this.

Thanks,

Eirik.

Johannes Schindelin· Dec 12, 2007, 14:55 UTC · re: Eirik Bjørsnøs · lore

Re: Efficient retrieval of commit log info

Hi,

On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:

Show 18 quoted lines

> I'm developing a piece of software that grabs logs from various types of 
> SCMs and presents (hopefully) useful information about the history.
> 
> My current approach with Git is do a "git clone --n" and then parse the 
> output of "git log". To check for updates I do a "git pull" followed by 
> a new "git log".
> 
> This approach works fine, but cloning the whole repository just to get 
> the change log seems like a somewhat inefficient use of bandwidth and 
> storage.
> 
> What I would like to do is to fetch just the change log information
> from the remote repository.
> 
> (Using the "CVS done right" tool I can do this with: "svn log --xml -v
> -r<last, HEAD> http://svn.example.com/")
> 
> I haven't found a way to do this using the Git command line tools.

It is not possible to get just the metadata. Remember, svn can do it only since the repository is purely remote. And git is a SCM (source code management system), not a CMV (commit metadata viewer).

You might be able to cobble up something that works accessing gitweb, but it might be even more inefficient.

Ciao, Dscho

Nicolas Pitre· Dec 12, 2007, 15:04 UTC · re: Johannes Schindelin · lore

Re: Efficient retrieval of commit log info

On Wed, 12 Dec 2007, Johannes Schindelin wrote:

Show 29 quoted lines

> Hi,
> 
> On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:
> 
> > I'm developing a piece of software that grabs logs from various types of 
> > SCMs and presents (hopefully) useful information about the history.
> > 
> > My current approach with Git is do a "git clone --n" and then parse the 
> > output of "git log". To check for updates I do a "git pull" followed by 
> > a new "git log".
> > 
> > This approach works fine, but cloning the whole repository just to get 
> > the change log seems like a somewhat inefficient use of bandwidth and 
> > storage.
> > 
> > What I would like to do is to fetch just the change log information
> > from the remote repository.
> > 
> > (Using the "CVS done right" tool I can do this with: "svn log --xml -v
> > -r<last, HEAD> http://svn.example.com/")
> > 
> > I haven't found a way to do this using the Git command line tools.
> 
> It is not possible to get just the metadata.  Remember, svn can do it only 
> since the repository is purely remote.  And git is a SCM (source code 
> management system), not a CMV (commit metadata viewer).
> 
> You might be able to cobble up something that works accessing gitweb, but 
> it might be even more inefficient.

On the other hand, this is pretty trivial to extend the protocol so only commit objects are transferred. Using 'git log' on the client side would work, as long as you don't use any of the content walking options.

Nicolas

Eirik Bjørsnøs· Dec 12, 2007, 15:19 UTC · re: Nicolas Pitre · lore

Re: Efficient retrieval of commit log info

> On the other hand, this is pretty trivial to extend the protocol so only
> commit objects are transferred.  Using 'git log' on the client side
> would work, as long as you don't use any of the content walking options.

Nicolas,

So what you're suggesting is that:

1) git-clone could be changed such that the user could ask not to
fetch content, only commit objects.
2) git-daemon (or the protocol layer, again I'm not familiar with the
implementation) could be changed such that it only sends commit
objects on request.
3) Changes to git-log is not necessary.

Anybody have a clue how much work this would require and if this is something that would be generally useful for the community?

Eirik.

Jon Smirl· Dec 12, 2007, 15:34 UTC · re: Eirik Bjørsnøs · lore

Re: Efficient retrieval of commit log info

On 12/12/07, Eirik Bjørsnøs <eirbjo@gmail.com> wrote:

Show 7 quoted lines

> > On the other hand, this is pretty trivial to extend the protocol so only
> > commit objects are transferred.  Using 'git log' on the client side
> > would work, as long as you don't use any of the content walking options.
>
> Nicolas,
>
> So what you're suggesting is that:

Write a script the runs the command locally on the remote sever and emails you the info. Or ssh to the remote server and run the commands locally.

Show 17 quoted lines

>
> 1) git-clone could be changed such that the user could ask not to
> fetch content, only commit objects.
> 2) git-daemon (or the protocol layer, again I'm not familiar with the
> implementation) could be changed such that it only sends commit
> objects on request.
> 3) Changes to git-log is not necessary.
>
> Anybody have a clue how much work this would require and if this is
> something that would be generally useful for the community?
>
> Eirik.
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
Jon Smirl
jonsmirl@gmail.com

Eirik Bjørsnøs· Dec 12, 2007, 15:50 UTC · re: Jon Smirl · lore

Re: Efficient retrieval of commit log info

> Write a script the runs the command locally on the remote sever and
> emails you the info. Or ssh to the remote server and run the commands
> locally.

Hi Jon,

Unfortunately that won't work for my use case since I'm fetching change metadata from remote repositories that I don't have any kind of control over.

Eirik.

Santi Béjar· Dec 12, 2007, 15:34 UTC · re: Johannes Schindelin · lore

Re: Efficient retrieval of commit log info

On Dec 12, 2007 3:55 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

> Hi,
>
> On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:
>

[...]

> > What I would like to do is to fetch just the change log information
> > from the remote repository.
> >

[ ... ]

Show 7 quoted lines

>
> It is not possible to get just the metadata.  Remember, svn can do it only
> since the repository is purely remote.  And git is a SCM (source code
> management system), not a CMV (commit metadata viewer).
>
> You might be able to cobble up something that works accessing gitweb, but
> it might be even more inefficient.

You could see how it is done in the git-browser:

http://repo.or.cz/w/git-browser.git

It is a gitk-like browser written in javascript. But I suspect that it should be enabled in the web server.

Santi

Linus Torvalds· Dec 12, 2007, 16:24 UTC · re: Eirik Bjørsnøs · lore

Re: Efficient retrieval of commit log info

On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:

> 
> Is it at all possible using the git network protocols to fetch just
> the commit log info, without transferring the contents?

>From a _protocol_ angle that's trivial (the git network protocol doesn't

really care what it transfers, it just transfers objects), but no, we don't actually expose any capability like that. If you were to just want this privately, the quick hack is to just disable traversal of trees in builtin-pack-objects on the server side, but it sounds like you want to do this in general (not just with git and not just with a server you control), and if so, you're kind of screwed.

You could hack around it (very inefficiently) by parsing gitweb output, of course, but I bet that you'd probably get better performance just cloning the whole thing (which allows streaming, rather than trying to click through commits on gitweb)

		Linus