Summary of core GIT while you are away.

Hoping that you had a good time during your vacation, and at the same time also hoping to help you catch up with GIT traffic, here is my version of the summary of things that happened in the GIT community around core GIT [*1*].

 - Just when I was forced to discontinue git-jc tree because my
   ISP webspace was running out, Petr Baudis started git-pb tree
   hosted at kernel.org next to his cogito repository, promising
   to pick up and keep good bits from the mailing list traffic.

   He's been doing quite a good job at it and if you pull from
   his tree, you will not miss anything worth keeping.  I can
   personally vouch for the crap-free-ness of his tree up to
   commit 95649d6cf9ec68f05d1dc57ec1b989b8d263a7ae (not that I
   suspect Petr will put crap in the tree after that, but that
   is the head as of this writing).  They consist of half
   bugfixes and half enhancements; all of them help Porcelain
   layers and end users.  Old git-jc tree has all been sucked
   into git-pb before being shut down, so we did not lose
   anything from there either.

 - A notable change you need to adjust your scripts and .profile
   to is that GIT environment variable names have changed.  All
   of them now have GIT_ as prefix.  Old names continue to work
   for now, though.

 - Nicolas Pitre's deltification patch is still cooking and in
   its latest incarnation it looked quite promising to me (the
   impact to the core is kept reasonably small and cleanly
   isolated).  Once fsck is made aware of it in the next round,
   I expect it to be ready to merge.

 - Brad Roberts has been working on libification to remove the
   assumption of having a single active_cache array.  I have
   minor qualms about it (I think it should also be tied with
   the index.lock handling for updates), but other than that, it
   looked admirably cleanly done.  I expect it to be in a
   mergeable state in the next round if/when he catches up with
   the tip of git-pb tree.

 - David Greaves has done all the heavy lifting for core GIT
   documentation.  It now is hosted at kernel.org with a nice
   web interface.

 - I have sent out about a dozen patches Petr has not caught up,
   mostly because we work in different timezones.  Other than
   that I have not seen anything immediately mergeable to the
   core GIT in the mailing list traffic.

 - Also you see names of many people in the git-pb log who
   contributed bugfixes and enhancements.  I am not belittling
   their contributions, but just not going to repeat what you
   can get from the log yourself.

That's about it.  Now please start pulling from 

	rsync://rsync.kernel.org/pub/scm/cogito/git-pb.git/

[Footnote]

*1* Porcelain layer seems to have had many more interesting discussions, but I am not qualified to comment on them (I was not following most of them).

On Sat, 14 May 2005, Junio C Hamano wrote:

Thanks. I'm back and reading email, but have by no means caught up yet.

Anyway, everythign I've read so far makes sense, and it might make sense to continue git development using just git-pb. The only thing I personally think sucks is the author/committer matching of git-rev-list/tree, since it would seem like somebody might well like to match on an arbitrary part of a commit, and special-casing author/committer seems somewhat broken. But that's a fairly minor nit.

I personally suspect that both git-rev-list and git-rev-tree should have an alternate output format that could be more easily grepped by subsequent commands. For example, right now git-rev-list just outputs a list of commit ID's, and it might make sense to have a flag to just append the commit message to the output, and zero-terminate it (and if the commit message has a NUL byte in it, just truncate it at that point).

Then you could just do

	git-rev-list -v --header HEAD | grep -z 'author[^\n]*Linus'

to tell it to do the "verbose" thing (only showing the header of the commit, not the whole message), and grep for "Linus" in the author line.

Or, if you want to see all commits that are signed off by so-and-so, do:

	git-rev-list -v HEAD | grep -zi 'signed-off-by:[^\n]*so-and-so'

and it would still be pretty efficient, and a _lot_ more generic and flexible than the "--author" and "--committer" flags.

I also suspect that a lot of uses if git-rev-{tree|list} actually want to see the commit stuff anyway, ie things like gitweb etc currently likely end up doing a separate "git-cat-file commit <sha1>" to get the whole commit information, which the "-v" flag would just give directly.

The only question is whether you want to have it be human-readable by default (indent the message lines with a <tab>, and nonheaders with <tab>+4*<space> or something), and then have a "-z" flag to do the machine-readable version described above.

Hmm?

		Linus

LT> Anyway, everything I've read so far makes sense, and it LT> might make sense to continue git development using just LT> git-pb. The only thing I personally think sucks is the LT> author/committer matching of git-rev-list/tree, since it LT> would seem like somebody might well like to match on an LT> arbitrary part of a commit, and special-casing LT> author/committer seems somewhat broken.

Well, that author/committer thing is not in git-pb yet, if I am not mistaken [*1].

The only reason why I did it that way was because the strategy taken by "struct object" derivatives seemed to pick up bare absolute minimum to support actual callers that have immediate need for information stored in structural fields, as opposed to designing for helping yet to be written callers by adding fields to hold information of "having this might also help somebody in the future" type. And the author and committer names are in the structured fields while signed-off-by and others are not. Also when author / committer name strings are intern'ed like the way I did, the memory consumption even for a long sequence of commits are kept reasonably low. However,...

LT> I personally suspect that both git-rev-list and git-rev-tree LT> should have an alternate output format that could be more LT> easily grepped by subsequent commands. For example, right LT> now git-rev-list just outputs a list of commit ID's, and it LT> might make sense to have a flag to just append the commit LT> message to the output, and zero-terminate it (and if the LT> commit message has a NUL byte in it, just truncate it at LT> that point).

I think what you said here makes more sense [*2*]. The above implies to keep the unpacked raw data as a whole to be accessible to the callers for at least commit objects and if we go that route I think it would make more sense to do that uniformly for everything (probably except for pure "blob" objects for size concerns but we might as well do them while we are at it). On the other hand, the current lifetime rules being what it is, that strategy may introduce memory consumption problems when working on a huge project.

[Footnote]

*1* Petr has been applying quite good judgements. I would have polluted git-jc with that patch already if I were still running it. So far, I have been generally happy with his acceptance criteria for external patches. Anything he places on hold or just outright returns to me, I later find rooms for improvements myself, and the later rounds that eventually get accepted always turn out to be far cleaner, thanks to his comments.

*2* At least in principle. I am not quite sure what the output should look like for rev-tree.

Dear diary, on Mon, May 16, 2005 at 11:33:11PM CEST, I got a letter where Junio C Hamano <junkio@cox.net> told me that...

I like Linus' suggestion. At the very least, what about making the matching generic for the header? Something like --match-header or whatever.

Thanks for the praise :-), but I'm actually quite unhappy with putting patches "on hold" and it's not intentional. It's just that I don't feel right about the patch enough to apply it immediately, and I either don't have time to voice my concerns if they are non-trivial, or I just want some time to think about it.

On Mon, 16 May 2005, Junio C Hamano wrote:

I think if we had a good reason to make author and committer structured, there must be some reason we'll want them parsed out eventually, although we may not have reached the point at which it makes sense to do it.

Perhaps have a struct object field for the unpacked data, have it filled by a unpack_object(struct object *) function, have the parse functions use it if it's filled, and have a function to free it (and NULL the field).

So, if you want the actual contents, you'd call unpack_object(&commit->obj) before parse_commit(commit), and close_object(&commit_obj) when you're done, and the parse wouldn't duplicate the work. The lifetime rules only really care that the return value from lookup_*() stays the same for the duration of the program, not that the fields in it don't get cleared (aside from the flags and the hash).

This could be useful even for blobs; I'm working on a program to do diff/merge on commits and everything under them (taking advantage of all available information if the fast path leads to a conflict); it would be nice not to have to go through the filesystem with the blob data.

	-Daniel
*This .sig left intentionally blank*

DB> Perhaps have a struct object field for the unpacked data,... DB> ..., and the parse wouldn't duplicate the work.

Absolutely. That is what I wanted to see ever since I started looking at "struct object" derivatives.

Dear diary, on Mon, May 16, 2005 at 06:10:10PM CEST, I got a letter where Linus Torvalds <torvalds@osdl.org> told me that...

I'd prefer that, in order to be consistent with the diff tools and such.

On Mon, May 16, 2005 at 09:10:10AM -0700, Linus Torvalds wrote:

What happened to that idea? That's not already working in some other way I missed, right? The pickaxe stuff is nice and was easy to call from the cgi, but searching in commit messages would be nice too. If it's not going to happen in the git-binaries, I may do it just in the cgi itself.

Kay

KS> What happened to that idea? That's not already working in some other way I KS> missed, right? The pickaxe stuff is nice and was easy to call from the cgi, KS> but searching in commit messages would be nice too. KS> If it's not going to happen in the git-binaries, I may do it just in the KS> cgi itself.

I haven't done anything about this yet. I was kind of waiting for the blob retention API for "struct object" derivatives to come to the conclusion.

On Wed, 25 May 2005, Junio C Hamano wrote:

I decided on it, and it's a hacky (but reasonable) decision.

"parse_commit()" always retains the buffer for the commit. It's very commit-specific, since (a) that's the main data structure that you'd actually want to retain and (b) of all the core objects (ignoring tags) commits tend to be the smallest and fewest, so this fixed policy has the least impact memory-usage-wise.

If you want to free it, you can do

	free(commit->buffer);
	commit->buffer = NULL;

and you should be all good (but the only user I thought migh want to is "fsck", and that one doesn't need to, since it never calls "parse_commit()" at all - it calls "parse_commit_buffer()" which does _not_ do this, since the caller already knows the buffer and can choose to do it if it wants to.

Oh, thinking about that actually brings up a bug: mixing parse_commit() and parse_commit_buffer() is unsafe, since the state of the buffer is unclear. Normally that's ok, _except_ when we use just "parse_object()" (like we do in "lookup_commit_reference()"), when we really don't know. Hmm.

		Linus

On Thu, 26 May 2005, Kay Sievers wrote:

Ok, you twisted my arm. Checked in.

	git-rev-list --header HEAD | grep -z 'author[\n]*Linus'

and you will get output that is a series of commits by me (strictly speaking, that's not true, since the "author" thing might be in the non-header part).

The format is:

	<commit-id> '\n' <commit-msg> '\0'

and if the full commit message is binary and contains NUL bytes in itself (only crazy people do that, but let's keep in mind that crazy people exist), we naturally truncate it at the first NUL. This makes it easy to parse, and is what makes "grep -z" work, for example.

It also has a rudimentary "stop at commit x" function, but I'm not doing any reachability analysis, so it's purely based on the commit date, and is thus _not_ equivalent to "git-rev-tree START ^END". I reserve the right to try to change that behaviour, though, if I decide I (or sombody else ;) can do a nice incremental reacability thing.

So you can do

	git-rev-list --header HEAD v2.6.12-rc5

and it will print out all the commits in date order until it hits v2.6.12-rc5 (which it won't print out).

The output format is really optimized for something like "git log" or "gitweb". I suspect it's pretty much perfect for doing the "last 10" things, without having to do "git-cat-file" for each commit.

Oh, and I considered adding a "--header-lines=c" which limits the header printout to 'c' lines (probably starting at where the free-form thing starts, so that "--header-lines=0" would print out just the fixed-format header of the commit messages.

(This is slightly different from what I initially envisioned, which had "-v" and "--header", and I'm not sure it's better but it ended up being what I did. If you think my first idea was better, send me a (by now trivial) patch).

		Linus

On Wed, May 25, 2005 at 06:45:14PM -0700, Linus Torvalds wrote:

Nice, thanks. Here is a very first try. It searches the commit messages
and uses pickaxe to search in the file content:
  http://ehlo.org/~kay/gitweb.cgi?p=git/git.git;a=search;s=symlink

Before someone asks: kernel.org needs to wait until cogito has catched up and released a new version and it is installed on the machines.

Kay

KS> Before someone asks: kernel.org needs to wait until cogito has catched up and KS> released a new version and it is installed on the machines.

Just from curiosity, does anybody runs (need to run) Cogito on kernel.org machine(s)? Or is it that gitweb depends on Cogito, not running directly on top of Plumbing?

On Thu, May 26, 2005 at 01:51:36PM -0700, Junio C Hamano wrote:

The cgi doesn't need it. Cogito has the spec file and the Mekefile with support for it - that was the reason the RPM made it on the machines there. Splitting the git-core stuff completely out of Cogito and install Cogito with a dependency on the git package is the only sane option I think.

Kay

KS> The cgi doesn't need it. Cogito has the spec file and the KS> Mekefile with support for it - that was the reason the RPM KS> made it on the machines there.

Ah, just installation convenience.  I see.

Junio C Hamano wrote:

Not "just" an installation convenience. Right now cogito and git-core *conflict*. That's why cogito really needs to be broken out from git-core, so git-core can be updated independently.

	-hpa

HPA> Junio C Hamano wrote: KS> The cgi doesn't need it. Cogito has the spec file and the KS> Mekefile with support for it - that was the reason the RPM KS> made it on the machines there.

HPA> Not "just" an installation convenience. Right now cogito and git-core HPA> *conflict*. That's why cogito really needs to be broken out from HPA> git-core, so git-core can be updated independently.

What I learned from this thread are these:

 (1) CGI does not need Cogito but just needs core GIT.
 (2) Nobody said he uses Cogito on kernel.org machines.
 (3) But Cogito is installed nevertheless.
 (4) Installation of Cogito is done via RPM "with Makefiles and
     spec files" and both Cogito and core GIT comes bundled in.
 (5) I gather that there is no core GIT only RPM (nobody said
     this explicitly in this thread, though, so I may be
     mistaken about this point).

And that was where my comment about "just an installation convenience" came from.

If somebody said Cogito _is_ used on kernel.org machines, then I would not have said "just" nor "convenience". That would mean there is a real need to have Cogito and having just core GIT would not be enough.

Junio C Hamano wrote:

I, for one, use Cogito on the kernel.org machines.

	-hpa

HPA> I, for one, use Cogito on the kernel.org machines.

I see. I see cogito.git catching up to the Linus core, so this would not be much of a problem anymore.

Junio C Hamano wrote:

It would still be a lot nicer to have them decoupled.

	-hpa

HPA> Junio C Hamano wrote:

HPA> I, for one, use Cogito on the kernel.org machines.

HPA> It would still be a lot nicer to have them decoupled.

No question about it.  Let's give Petr some time.

On Thu, 26 May 2005, H. Peter Anvin wrote:

Yes, but git has been changing so much that I think it has made sense to have it together with cogito, so that you don't have version skew.

Have we gotten to the point where git is calming down? Judging by the raw diff etc changes, I'd say no, but on the other hand, those are largely over, and maybe we don't have any serious real incompatibilities pending any mroe...

The object data structures have been pretty stable (thank the Gods!), but command line switches and some data output formats certainly have not.

The one big change I see coming is that I really want to make "git-rev-list HEAD TAIL" work right (ie do a real reachabilty thing rather than the date-based thing), at which point you'd _usually_ not need to use the full logic of git-rev-tree that can do multiple parents etc. But that change should be compatible, I hope (ie old scripts will continue to use git-rev-tree and play games with the output, because they'd not know any better).

			Linus

Kay Sievers wrote:

Petr, it seems we need a new Cogito release soon. 0.10 really has fallen behind a lot. Also, separating out Cogito from git-core would be nice.

	-hpa

KS> Nice, thanks. Here is a very first try. It searches the commit messages KS> and uses pickaxe to search in the file content: KS> http://ehlo.org/~kay/gitweb.cgi?p=git/git.git;a=search;s=symlink

Nice, thanks for using pickaxe ;-). One request, knowing well that what you have on ehlo is your first cut.

The pickaxe really shines when you can paste a couple of lines of code in the version you are having trouble with (or just you are curious about) in order to see where they came from. For example, I did an equivalent of the following command back to figure out which patch and author to give credit to when I stole the tagged output format for git-ls-files from Cogito branch:

git-rev-list pasky | git-diff-tree -v -p --stdin \ -S'static const char *tag_cached = ""; static const char *tag_unmerged = ""; static const char *tag_removed = "";'

Since this is primarily about program code, forbidding '*' is rather nasty, and not being able to do multiple lines makes it quite less useful than it could be.

On Thu, May 26, 2005 at 03:25:17PM -0700, Junio C Hamano wrote:

Yeah, I like it a lot. Nice work.

Any idea, what's the reason, it doesn't find anything here in the
kernel tree:
  git-rev-list HEAD | git-diff-tree -S'TIMEOUT' --stdin

while:
  git-rev-list --header HEAD | grep -z TIMEOUT

prints the commit:
  6897089c5f7989603ccb9c696050470ba1dbd262

with the following change:
  git-diff-tree -p 6897089c5f7989603ccb9c696050470ba1dbd262 6f31e42221c7deae4527136ce0dd73990d8bc1d1
  diff --git a/drivers/base/firmware_class.c
  b/drivers/base/firmware_class.c
  --- a/drivers/base/firmware_class.c
  +++ b/drivers/base/firmware_class.c
  @@ -102,9 +102,6 @@ firmware_class_hotplug(struct class_devi
          if (add_hotplug_env_var(envp, num_envp, &i, buffer, buffer_size, &len,
                          "FIRMWARE=%s", fw_priv->fw_id))
                  return -ENOMEM;
  -       if (add_hotplug_env_var(envp, num_envp, &i, buffer, buffer_size,
          &len,
  -                       "TIMEOUT=%i", loading_timeout))
  -               return -ENOMEM;
   
          envp[i] = NULL;

Nice feature, maybe we can add it later (see next).

I see what you mean, but allow to pass any charater through a cgi to a invoked shell as a command-line option is a nightmare without being very carful. That's why only a few character allowed now. I'm not sure how to do that without creating possibly a big hole in the webserver. :)

Kay

KS> On Thu, May 26, 2005 at 03:25:17PM -0700, Junio C Hamano wrote:

KS> Nice, thanks. Here is a very first try. It searches the commit messages KS> and uses pickaxe to search in the file content: KS> http://ehlo.org/~kay/gitweb.cgi?p=git/git.git;a=search;s=symlink

KS> Yeah, I like it a lot. Nice work.

KS> Any idea, what's the reason, it doesn't find anything here in the KS> kernel tree: KS> git-rev-list HEAD | git-diff-tree -S'TIMEOUT' --stdin

Dies this work for you?  Notice "-r".

git-rev-list HEAD | git-diff-tree -r -S'TIMEOUT' --stdin

On Thu, May 26, 2005 at 05:16:42PM -0700, Junio C Hamano wrote:

Yes, works perfect! Sorry, I need to get used to repositories with subdirectories. :)

Thanks, Kay

KS> I see what you mean, but allow to pass any charater through a cgi to a KS> invoked shell as a command-line option is a nightmare without being very KS> carful.

You are absolutely right that you must be careful. But being careful is not that hard.

I do not know what you are using to parse the data coming from the browser (presumably application/x-www-form-urlencoded or somesuch), but once you got the raw data out of it in a variable in your host language (sorry I do not know what language you are writing in, either), quoting that value safely for shell command line is very simple and easy. You can emulate what sq_expand() in diff.c does.

Essentially you take advantage of single quote quoting which lets you pass anything other than single quotes as literals, and deal with single quotes by stepping out temporarily of the single quote environment every time you see a single quote, quote that single quote with a backslash, and immediately after that you go back into single quote environment again and continue.

  original     sq_expand     result
  name     ==> name      ==> 'name'
  a b      ==> a b       ==> 'a b'
  a'b      ==> a'\''b    ==> 'a'\''b'

So in shell, using sed, you would do something like this:

------------ #!/bin/sh

orig="foo '"'\bar b az'

script="s/'/'\\\\\\''/g" ;# change (') to ('\'')

sq_inside=`echo "$orig" | sed "$script"`

echo "'$sq_inside'" ;# and enclose the whole thing in sq pair ------------