Re: Git-commits mailing list feed.

On Thu, 2005-04-21 at 12:29 +0200, Arjan van de Ven wrote:

With BK this was not possible, but could you please provide your criticism in 'diff -up' form?

I've done 'perl -pi -e s/-u/-up/ gitdiff-do' as a quick hack to provide what you want, but a saner fix to make gitdiff-do obey the same GIT_DIFF_CMD and GIT_DIFF_OPTS environment variables as show-diff.c would be a more useful answer.

On Sun, 24 Apr 2005, David Woodhouse wrote:

The reason I've not done tags yet is that I haven't decided how to do them.

The git-pasky "just remember the tag name" approach certainly works, but I was literally thinking o fsetting up some signing system, so that a tag doesn't just say "commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 is v2.6.12-rc2", but it would actually give stronger guarantees, ie it would say "Linus says that commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 is his 2.6.12-rc2 release".

That's something fundamentally more powerful, and it's also something that I actually can integrate better into git.

In other words, I actually want to create "tag objects", the same way we have "commit objects". A tag object points to a commit object, but in addition it contains the tag name _and_ the digital signature of whoever created the tag.

Then you just distribute these tag objects along with all the other objects, and fsck-cache can pick them up even without any other knowledge, but normally you'd actually point to them some other way too, ie you could have the ".git/tags/xxx" files have the pointers, but now they are _validated_ pointers.

That was my plan, at least. But I haven't set up any signature generation thing, and this really isn't my area of expertise any more. But my _plan_ literally was to have the tag object look a lot like a commit object, but instead of pointing to the tree and the commit parents, it would point to the commit you are tagging. Somehting like

	commit a2755a80f40e5794ddc20e00f781af9d6320fafb
	tag v2.6.12-rc3
	signer Linus Torvalds

	This is my official original 2.6.12-rc2 release

	-----BEGIN PGP SIGNATURE-----
	....
	-----END PGP SIGNATURE-----

with a few fixed headers and then a place for free-form commentary, everything signed by the key (and then it ends up being encapsulated as an object with the object type "tag", and SHA1-csummed and compressed, ie it ends up being just another object as far as git is concerned, but now it's an object that tells you about _trust_)

(The "signer" field is just a way to easily figure out which public key to check the signature against, so that you don't have to try them all. Or something. My point being that I know what I want, but because I normally don't actually ever _use_ PGP etc, I don't know the scripts to create these, so I've been punting on it all).

If somebody writes a script to generate the above kind of thing (and tells me how to validate it), I'll do the rest, and start tagging things properly. Oh, and make sure the above sounds sane (ie if somebody has a better idea for how to more easily identify how to find the public key to check against, please speak up).

			Linus

On Sat, 23 Apr 2005, Linus Torvalds wrote:

Btw, in case it wasn't clear, one of the advantages of this is that these objects are really _not_ versioned themselves, and that they are totally independent of the objects that they actually tag.

They spread together with all the other objects, so they fit very well into the whole git infrastructure, but the real commit objects don't have any linkages to the tag and the tag objects themselves don't have any history amongst themselves, so you can create a tag at any (later) time, and it doesn't actually change the commit in any way or affect other tags in any way.

In particular, many different people can tag the same commit, and they don't even need to tage their _own_ commit - you can use this tag objects to show that you trust somebody elses commit. You can also throw the tag objects away, since nothing else depends on them and they have nothing linking to them - so you can make a "one-time" tag object that you can pass off to somebody else, and then delete it, and now it's just a "temporary tag" that tells the recipient _something_ about the commit you tagged, but that doesn't stay around in the archive.

That's important, because I actually want to have the ability for people who want me to pull from their archive to send me a message that says "pull from this archive, and btw, here's the tag that not only tells you which head to merge, but also proves that it was me who created it".

Will we use this? Maybe not. Quite frankly, I think human trust is much more important than automated trust through some technical means, but I think it's good to have the _support_ for this kind of trust mechanism built into the system. And I think it's a good way for distributors etc to say: "this is the source code we used to build the kernel that we released, and we tagged it 'v2.6.11-mm6-crazy-fixes-3.96'".

And if my key gets stolen, I can re-generate all the tags (from my archive of tags that I trust), and sign them with a new key, and revoke the trust of my old key. This is why it's important that tags don't have interdependencies, they are just a one-way "this key trusts that release and calls it xyzzy".

		Linus

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Samstag, 23. April 2005 19:31 schrieb Linus Torvalds:

To generate those you do:

# cat unsigned_tag

	commit a2755a80f40e5794ddc20e00f781af9d6320fafb
	tag v2.6.12-rc3
	signer Linus Torvalds
	This is my official original 2.6.12-rc2 release

# gpg --clearsign < unsigned_tag > signed_tag # gpg will ask here for the secret key phrase

To verify you do:

# gpg --verify < signed_tag

and check exit status.

Hope that helps,

cu

Fabian 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCaorzI0lSH7CXz7MRAr3QAJ45f2CQTgJ0sYfF9kRyrWHbsazVQQCeMqW7 HCsah/llt/I8sQ36dlDnRWg= =Fgq1 -----END PGP SIGNATURE-----

I would prefer a generic mechanism to sign _any_ object, not just tag objects:

- Introduce "signature objects" that contains an implementation-specific 
  signature. git doesn't care about the content, as long some script can 
  verify that the signature in the signature object matches the content of 
  the object(s) it references. The "name" of a signature object is the 
  SHA1 hash of the content (=gpg signature, for example).

- Referencing signatures in tags makes no sense IMO, because it would 
  require to change the (hash) name of tags when someone else wants to 
  co-sign it later on. I would just distribute two names for that (here is 
  tag xxxxx and its signature is yyyyy). Tags should only contain a
  symbolic name and the hash of the commit object they point to.

- A nice benefit of this is that we could sign unnamed commits (think 
  automatic signing of intermediate commit), or even sign individual
  files in the tree.

Just my 2c.

Andreas

On Sat, 23 Apr 2005, Fabian Franz wrote:

On Sat, April 23, 2005 1:31 pm, Linus Torvalds said:

tells me how to validate it), I'll do the rest, and start tagging things properly. Oh, and make sure the above sounds sane (ie if somebody has a better idea for how to more easily identify how to find the public key to

Hi Linus,

Why not leave tags open to being signed or unsigned? Anyone that wants to create a trusted tag could simply sign their cleartext entry in the tag object.

Ideally the SHA1 tree reference would be included in the text entry whether it was signed or not. Thus any script can pull the SHA1 out of the text entry. And a script that understands the signing method can verify it. But scripts that don't understand the signing method can still use the tag.

For presentation in the log or whatever, the script can look inside the clear text message, grab the SHA1 and display it in the header area; even though it's not really in the header, always just in the clear text area.

Sean

Hello,

I think that this is the idea anyway.

Having the SHA1 signature twice in would be confusing and error-prone when checking is done automated.

So establishing the infrastructure is a good thing. To use it for every commit is another issue.

	Thomas

On Sat, April 23, 2005 3:02 pm, Thomas Glanzmann said:

There's no need to have the SHA1 object reference twice. It will only be in the clear text, nowhere else. Of course scripts that display the log, could show the object reference in the header area for aesthetics. Another nice thing is that this works no matter cleartext signing methods emerge in the future.

Sean

On Sat, 23 Apr 2005, Sean wrote:

That is exactly what my proposal does, except I'd make the normal tags creation always sign.

But since _git_ won't care which is why I want the signature at the _end_, not "surrpunding" the thing, you could create a tag that just doesn't have the signature, and git will never even notice. The people who see the tag may say "hmm, why couldn't he be bothered to sign it", though.

		Linus

Hello,

there is no need to tell the verifier against what key to verify because the signature already contains this information.

# This creates the signature. gpg --clearsign < sign_this > signature

# And this verifies it. gpg --verify < signature && echo valid

	Thomas

On Sat, 23 Apr 2005, Thomas Glanzmann wrote:

This really doesn't work for me - I do not want to have the gpg header above it, only the signature below. Since I want git to actually understand the tags, but do _not_ want git to have to know about whatever signing method was used, I really want the resulting file to look like

	commit ....
	tag ...

	here goes comment
	here goes signature

and no headers.

Whether that can be faked by always forcing SHA1 as the hash, and then just removing the top lines, and re-inserting them when verifying, or whether there is some mode to make gpg not do the header crud at all, I don't know. Which is exactly why I never even got started.

		Linus

On Sat, April 23, 2005 2:30 pm, Linus Torvalds said:

above it, only the signature below. Since I want git to actually understand the tags, but do _not_ want git to have to know about whatever

just removing the top lines, and re-inserting them when verifying, or whether there is some mode to make gpg not do the header crud at all, I don't know. Which is exactly why I never even got started.

Linus,

A script that knows how to validate signed tags, can easly strip off all the signing overhead for display. Users of scripts that don't understand will see the cruft, but at least it will still be usable.

Sean

On Sat, 23 Apr 2005, Sean wrote:

NO.

Guys, I will say this once more: git will not look at the signature.

That means that we don't "strip them off", because dammit, they DO NOT EXIST as far as git is concerned. This is why a tag-file will _always_ start with

	commit <commit-sha1>
	tag <tag-name>

because that way we can use fsck and validate reachability and have things that want trees (or commits) take tag-files instead, and git will automatically look up the associated tree/commit. And it will do so _without_ having to understand about signing, since signing is for trust between _people_ not for git.

And that is why I from the very beginning tried to make ti very clear that the signature goes at the end. Not at the beginning, not in the middle, and not in a different file. IT GOES AT THE END.

		Linus

On Sat, April 23, 2005 3:38 pm, Linus Torvalds said:

Yes, totally agreed.

Okay now you're just being difficult <g> You're acting like it's impossible for git to grab the SHA1 out of the clear text message if there is signing overhead above the tag reference. That is nonesense. You simply state that tag must include a SHA1 object reference preceded by "REF:" in the comment. Git can surely use this regardless of what signing overhead is above, below or beside it. The suggestion for stripping out the signing overhead was for _human_ readability; git won't care a gnit.

Sean

On Sat, 23 Apr 2005, Sean wrote:

No. It's not "impossible" for git to parse crap. But git won't.

There are two ways you can write programs:
 - reliably
 - unreliably

and I do the first one. That means that a program I write does something _repeatable_. It does the same thing, regardless of whether a human happened to write "REF:" in the comment section, or anything else.

The thing is, great programs come not out of great coding, but out of great data structures. The whole git philosophy bases itself on getting the data structure right.

And what you are asking for is doing it _wrong_. So in git I don't just parse random free-form text and guess that a line that starts with REF: is a reference to a commit. It has very rigid and well-specified data structures, and that's how you make reliable programs.

I don't care what anybody else does on top of git, but dammit, I'll make sure that the core infrastructure is designed the right way.

And that means that we don't guess, and that we don't parse random ASCII blobs. It means that we have very very fixed formats so that programs can either do the right thing or unambiguously say "that's crap".

I've said it before, and I'll say it again: we have enough crap that calls itself SCM's out there already. I want git to be reliable and _simple_, not a collection of crap that just happens to work.

		Linus

LT> Guys, I will say this once more: git will not look at the signature.

LT> And that is why I from the very beginning tried to make ti very clear that LT> the signature goes at the end. Not at the beginning, not in the middle, LT> and not in a different file. IT GOES AT THE END.

If that is the case, can't you do it without introducing this new tag object, like this?

  1. Find existing commit-id that you want to tag.
  2. Sign that commit object:

     cat-file commit $commit |
     gpg --detach-sign --armor -u 'Linus Torvalds' >commit.sig

  3. Make another commit, making the original commit as its parent:

     {
         echo tag This is my tag.
         cat commit.sig 
     } | commit-tree $(cat-file commit $commit |
                       sed -e 's/tree //;d') -p $commit

Then you can publish the ID of this commit object, which attests that the original commit is what you vouch for. Am I missing something?

On Sat, 23 Apr 2005, Junio C Hamano wrote:

No, because I also want to sign the _name_ I gave it.

Otherwise somebody can take my "signed commit", and claim that I called it something else.

Just signing the commit is indeed sufficient to just say "I trust this commit". But I essentially what to also say what I trust it _for_ as well.

And sure, I could make a totally bogus "commit" object that just points to the original commit, uses the same "tree" from that original commit, and write what I want to trust into that commit. I then sign that, and create yet _another_ commit that has the signature (and the pointer to the just signed commit) in its commit message, and then I point to _that_ commit.

So yes, we can certainly do this with playing games with commits. That sounds singularly ugly, though, since just doing a "tag" object is a lot more straightforward, and really tells the world what's going on (and makes it easy for automated tools to just browse the object database and see "that's a tag").

			Linus

JCH> If that is the case, can't you do it without introducing this JCH> new tag object, like this?

Of course It Would Not Work. I am an idiot X-<. Sorry. What I suggested does not authenticate the tag itself.

On Sat, 23 Apr 2005, Linus Torvalds wrote:

Actually, can you make the signature be detached and a seperate object? Ie, add a signature object in its own right, distinct from tag. They could then:

- be used to sign any kind of object
- allow objects to be signed by multiple people

Ideally, there'd be an index of signature objects by the SHA-1 sum of the object they sign, as the signed object should not refer to the signature (or the second of the above is not possible).

The latter of the two points would, in combination with the former, allow for cryptographic 'signed-off-by' chains. If a 'commit' is signed by $RANDOM_CONTRIBUTOR and $SUBSYSTEM_MAINTAINER and $ANDREW, you know its time to pull it. Would also work for things like "fixes only" trees, where (say) a change must be approved by X/2+1 of a group of X hacker providing oversight -> looking up the commit object's signatures would tell you whether it was approved.

No idea whether this is possible or practical. :) But it would be good for future flexibility to avoid including the signature in the object being signed.

regards,

On Mon, 25 Apr 2005, Paul Jakma wrote:

Ah, this could (obviously) be done generally by providing a general index of 'referals' (if desirable).

I have no idea whether git already does this, I havn't checked it out yet but I'm very interested to see how git will mature and have been trying to follow its progress - I'm a frustrated admin of a CVS repository..

regards,

On Sat, 23 Apr 2005, Linus Torvalds wrote:

 >
 >> And that is why I from the very beginning tried to make ti very clear
 >> that the signature goes at the end. Not at the beginning, not in the
 >> middle, and not in a different file. IT GOES AT THE END.

It may be better to have them as simple detached signatures, which are completely separate files (see gpg --detached). Yeah, gpg currently implements detached signatures by repeating what gets signed, which is unfortunate, but the _idea_ is the right one.

Paul Jakma wrote:

Yes, and see my earlier posting.  It'd be easy to store signatures in
the current objects directory, of course.  The trick is to be able
to go from signed-object to the signature; this could be done
just by creating a subdirectory using a variant of
the name of the signed-object's file, and in that directory store the
hash values of the signatures.  E.G.:
  00/
     3b128932189018329839019          <- object to sign
     3b128932189018329839019.d/
     0143709289032890234323451
  01/
     43709289032890234323451          <- signature

Right. Lots of tricks you can do once the signatures are there, such as checking to counter repository subversion (did everything get signed), finding out who introduced a malicious line of code (& "proving" what key signed it first), etc. There are LOTS of reasons for storing signatures so that they can be checked later on, just like there are lots of reasons for storing old code... they give you evidence that the reputed history is true (and if you doubt it, they give you a way to limit the doubt).

--- David A. Wheeler

On Sun, 24 Apr 2005, David A. Wheeler wrote:

Hmm, what do you mean by "repeating what gets signed"?

Two ways:

1. An index of sigs to signed-object.

(or more generally: objects to referring-objects)

2. Just give people the URI of the signature, let them (or their
    tools) follow the 'parent' link to the object of interest

You could hack it in to the namespace somehow I guess. I'm not sure hacking it in would be a good thing though.

I think it might be more useful just to provide a general index to lookup 'referring' objects (if git does not already - I dont think it does, but I dont know enough to know for sure). So you could ask "which {commit,tag,signature,tree}(s) refer(s) to this object?" - that general concept will always work. If you wanted to make the implementation of this index use some kind of sub directory as in the above, fine..

See also method 2 above. Which would be more efficient for tools if, within a project, some developers sign their 'updates' and some dont.. (you never need to check whether there's a signature or not - you'll know it from the URI automatically).

Indeed.

Anyway, we shall see what Linus does. :)

(But I do hope at least that signatures are /not/ included inline using BEGIN PGP.. in the object that is signed.)

regards,

Paul Jakma wrote:

Forget it, irrelevant.  I vaguely remembered some problem with
gpg's detached signatures, but it was probably either a really
early alpha version or someone was using "--clearsign" instead
of "--armor".  I just did a quick check with:
  gpg --armor --detach -o junk.sig junk.c
and it worked "as expected"; no repeat of the data.

Right. I suggested putting it in the same directory as the objects, so that rsync users get them "for free", but a separate directory has its own advantages & that'd be fine too. In fact, the more I think about it, I think it'd be cleaner to have it separate. You could prepend on top of the signature (if signatures are separate from assertions) WHAT got signed so that the index could be recreated from scratch when desired.

If you mean "the signatures aren't stored with the objects", NO. Please don't! If the signatures are not stored in the database, then over time they'll get lost. It's important to me to store the record of trust, as well as what changed, so that ANYONE can later go back and verify that things are as they're supposed to be, or exactly who trusted what.

git definitely doesn't have this currently, though you could run the fsck tools which end up creating a lot of the info (but it's then thrown away).

 > So you could ask "which

Yes. The problem is that maintaining the index is a pain. It's probably worth it for signatures, because the primary use is the other direction ("who signed this?"); it's not clear that the other direction is common for other data.

--- David A. Wheeler

On Sun, 24 Apr 2005, David A. Wheeler wrote:

Well, i'm trying to play with git right now to see what would fit with how it abstracts things.

I think possibly:

- add the 'signature object' to the respository after the signed
   object

So a 'signed commit' turns into the

- tool preparing the commit object,
 	- get the user to sign it
 	- save the detached signature for later
- adding the commit object to the repository
- prepare the signing object and add to repository

The repository head then refers then to signature object, which could (handwaving) look something like:

 	Object		Signature
 	Signing 	<object ID, in this case of the commit object>
 	Sign-type 	GPG

 	<signature data>

Tools should then treat signature objects as 'stand ins' for the object they are signing (verify the signature - if desired - and then just retrieve the 'Signing' object ID and use that further).

I have no working knowledge of git though, other than following this list. So I have no idea whether above is at all appropriate or workable.

No more lost than anything else in the git 'fs'.

If someone prunes old objects, they'll lose the signed objects along with the signatures. If those files weren't replicated anywhere else, well they've just blown away history for good, both the history of the source and corresponding signatures.

See above.

Well, it could be retained then.

Possibly.

In CVS it is. If you 'cvs log' a file, you can get a report on which 
revisions of the file belong to which tags (which can be useful 
information sometimes: "ah, so that release had the buggy version" 
type of thing. Or as a sanity check to make sure you got a tag right 
- particularly when you have to move a wrong tag[1]). So, in addition 
to signatures, a general 'referrers of this object' index could be 
useful for reports.

1. This might be just a CVS thing, and not wanted for git -> the 
ability to tag historical revisions and indeed change what tags refer 
to.

regards,

Ah, to add to below..

If one wished, one could optionally store the actual signature data as a seperate blob object and refer to it in the signing object. Not needed really for a GPG ASCII clear-signed detached signature (tiny and they're ASCII obviously :) ), but who knows.

On Mon, 25 Apr 2005, Paul Jakma wrote:

   - adding the signature blob, if it is to stored as a blob

With either a 'Signature  <ID of signature data blob>' or else:

regards,

On Sun, 24 Apr 2005, David A. Wheeler wrote:

Actually, if we do totally separate files, then the detached thing is ok, and we migth decide to not call the objects at all, since that seems to be unnecessarily complex.

Maybe we'll just have signed tags by doing exactly that: just a collection of detached signature files. The question becomes one of how to name such things in a distributed tree. That is the thing that using an object for them would have solved very naturally.

		Linus

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Montag, 25. April 2005 03:50 schrieb Linus Torvalds:

What about just <sha1 hash of object>.sig or <sha1 hash of object>.asc?

Or would this violate the concept of the object database to just contain hashes?

cu

Fabian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCbFMsI0lSH7CXz7MRAof0AKCILjPE/M72cMSVNDC/DWYSzmrU/ACggOuS ogNPwUf2ASAwmbwixzSTuPs= =pW5D -----END PGP SIGNATURE-----

It may sound a little weird, but we could actually store the signature in the inode/filename. GPG signatures seem to be around 80 bytes of ASC, thats well below MAXPATH and should work even if your repository is somewhere/deep/in/your/filesystem/hierarchy.

1. Signed objects are named sha1-sig (sig is a 80 character signature 
here, not the three letters sig).

2. To make sure we can find objects without their signature, there is 
   always a soft link sha1 -> sha1-sig (fsck can check this and create 
   missing links).

3. To find a signature, just follow the link and look at the real name.

4. Files can be distributed without signature (content is unchanged) and
   you can sign them in your local tree with your own signature, 
   effectively throwing my signature away.

The only limitation is that each object can only be signed by one person. On the other hand, this might not be a limitation at all. If I create a file, I sign it. Nobody else. Same goes for trees and commits that I create. You can sign your own commit object when you merge my stuff, and then push that commit object out (along with your co-signature).

Andreas

On Mon, 25 Apr 2005, Fabian Franz wrote:

On Mon, 25 Apr 2005, Fabian Franz wrote:

Well, the SHA1 of an object really is not a very good name, unless you have something to manage it with. Again, the object database has something to manage and find those objects with - things like .git/HEAD, but also "fsck" to find dangling and unnamed objects.

Maybe we'll never have so many tags that we need to manage them, and yes, if so, we can just have ".git/signatures" be a directory with objects that are just named for their content SHA1, the same way the object database is, but separately (and probably just using a flat file structure, no need for the subdirectory fan-out that the object directory has).

No need for a ".sig" thing, since they'd be defined to be signatures just from their location.

This wouldn't be an object at all in that case, they'd be totally outside the scope of the git object model.

And yes, if they were to be git objects, they'd follow totally different rules: they'd have to have the "tag+length+'\0'" format, and they would be zlib-compressed.

If they are totally outside of git, then I don't care what the object format is, and then it could be just a regular text-file with a signature and content, and just happen to be named for the SHA1 hash so that there is no confusion about what happens when multiple people happen to create different tags with the same name.

		Linus

On Mon, 25 Apr 2005, Fabian Franz wrote:
 >> What about just <sha1 hash of object>.sig or <sha1 hash of object>.asc?

If you mean "hash of object being signed", the problem is that there may be more than one signature of a given object. Keys get stolen, for example, so you want to re-sign the objects. Yes, you could replace the files, but it's nicer to make it so there's never a need to replace files in the first place. That's one of the nice properties of the git object database; so if we can have that property everywhere, I think we should.

Instead, store the signatures in the normal object database, &
give it type "signature".  To speed access FROM a commit or tag
to a signature (and FROM a commit to a tag), create a
separate reverse directory that tells you what objects reference
a given object.  Like this:
.git/
   objects/
     00/
       0195297c2a6336c2007548f909769e0862b509  <= a commit object
     02/
       0395297c2a6336c2007548f909769e0862b509  <= signature of commit
     04/
       0595297c2a6336c2007548f909769e0862b509  <= a tag
     06/
       0795297c2a6336c2007548f909769e0862b509  <= signature of tag
   reverse/
     00/
       0195297c2a6336c2007548f909769e0862b509/
         020395297c2a6336c2007548f909769e0862b509  "this signs commit"
         .... other later signatures of this commit go here.
     04/
       0595297c2a6336c2007548f909769e0862b509/
         060795297c2a6336c2007548f909769e0862b509
         .... other later signatures of this tag go here.

The reverse directory's contents are basically the filenames. The files themselves could be symlinks back up, or not. Content-free files are probably more portable across filesystems, and it's probably also good for space efficiency (though I haven't examined that carefully).

"git"'s knowledge of signatures should be VERY limited, and not dependent on PGP. I think that'd be easy. You could prepend some signature data into the "signature" file to make it much easier to reconstruct the reverse directory and to make it easy to check things WITHOUT knowledge of PGP or whatever.

Here's potential output:

$ cat-file commit 000195297c2a6336c2007548f909769e0862b509 tree 2aaf94eae20acc451553766f3c063bc46cfa75c6 parent dc459bf85b3ff97333e759d641c5d18f4dad470d author Petr Baudis <pasky@ucw.cz> 1114303479 +0200 committer Petr Baudis <xpasky@machine.sinus.cz> 1114303479 +0200

    Added the whatsit flag.

$ cat-file signature 000195297c2a6336c2007548f909769e0862b509 signatureof commit 000195297c2a6336c2007548f909769e0862b509 signer Petr Baudis <pasky@ucw.cz>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQBCbFaRCxlT/+f+SU4RAgYSAKCWpPNlDKDkxuuA649zJop7WkQPnACdF1Fg JgXatbJU8YJ7JHqvgyGepRU= =Kttg -----END PGP SIGNATURE-----

$

--- David A. Wheeler

David A. Wheeler wrote:
  > $ cat-file signature 000195297c2a6336c2007548f909769e0862b509
minor comment, cat-file gives you raw access to the object data.

better: $ cat-file signature $(what-signs 000195297c2a6336c2007548f909769e0862b509)

David

Linus Torvalds wrote:

I agree, naming signatures using the same way other objects are named would be very clean. So, why not? It's perfectly reasonable to just store detached signatures as hashed objects, just like the rest; just create a new object type ("signature"). If 3 different keys are used to sign the same object, the detached signatures will have different hash values, so they'll get named easily.

Now you just have to FIND the signature of a signed object, i.e. efficiently go the "other way" from signed object to detached signature. A separate directory with this mapping, or embedding the mapping inside the object directory (HASH.d/<list>) both solve it.

The more I think about it, the more I think a separate "reverse"
index directory would be a better idea. It just needs to from
"me" to "who references me", at least so that you can quickly
find all signatures of a given object. If the reverse directory
gets wonky, anyone can just delete the reverse index directory
at any time & reconstruct it by iterating the objects.
Before "-----BEGIN PGP SIGNATURE-----" you should add:
  signatureof HASHVALUE
to make reconstruction easy; PGP processors ignore stuff
before "-----".  The PGP data does include a hash, but it's not
easy to get it out (I don't see a way to do it in gpg from the
command line), and it's quite possible that a signer won't
use SHA-1 when they sign something (they may not even
realize it; it depends on their implementation's configuration).
Better to include something about what was signed with the signature.

Hmm, probably worth backtracking to see what's needed. There needs to be a way to identify tags, and a way to sign that tag so that you can decide to trust some tags & not others. There needs to be a way to sign commits, and store that info for later. And really, these are special cases of general assertions about other things; you might want someone to be able to make other signed assertions (e.g., that it passed test suite XYZ).

If tags & commits are all you plan to sign for now, well, you already have commits. You can just add a "tag" type and a "signature" type of object (the "signature" is just a detached OpenPGP signature). "signature" can sign tag or commit types. I still like the idea of a more general "assertion" type, esp. for assertions that something passed a test suite on a certain date or was reviewed at a certain date by someone, but admittedly that could be added later in the same manner.

Then you need to be able to quickly find a signature, given a commit or tag. A "reverse" directory then does that nicely, and if you put enough information in front of the signature, you can regenerate the reverse directory whenever you wish.

--- David A. Wheeler

On Sun, 24 Apr 2005, David A. Wheeler wrote:

You dont even need it, see my other mail. If:

- the signature is an object and added after the commit object

- tools know that signatures are 'proxies of' or precursors to the
   objects they are signing (which makes sense, a signature by
   definition refers to something else)

- the signature object refers to the object it is signing (eg a
   'Signing <object ID>' header)

Then head can simply be the signature object and tools can find the commit by following the 'Signing' field of the signature (they dont even need to check the signature is valid). No index lookup needed.

You only need the index for historical verification really, and you can always generate an index if needs be. (and have the tools maintain it).

Oof, dont do this:

- makes assumptions about the format of the signature
 	- that it is ASCII
 	- that you can change it

Just add a git header which is independent of the signature data.

In lieu of the 'signature object as precursor' approach above, just have the tools maintain an index. It can be maintained as objects as added, and can always be blown away and recreated by inspection of the repository data.

regards,

On Mon, 25 Apr 2005, Paul Jakma wrote:

Uh, I have no idea whether verifying a signature of a commit object is sufficient, ie equivalent to signing each file.

commit refers to tree objects, which I presume lists the SHA-1 object IDs of files, but IIRC Linus already described why a signature of the commit object should not be used to trust the rest of commit.. (i'll have to find his mail). If so, an index is required.

regards,

On Mon, 25 Apr 2005, Paul Jakma wrote:

Ah, apparently it is sufficient:

Linus:

“Just signing the commit is indeed sufficient to just say "I trust this commit". But I essentially what to also say what I trust it _for_ as well.”

So this would work for commit objects.

It would also work for tag objects, if you pointed people at the signature object rather than the actual tag object.

regards,

This patch adds a new option -H to rpush/rpull to update the HEAD pointer when pushing a new release to a remote repository.

Signed-off-by: Andreas Gal <gal@uci.edu>

--- c27af2c2464de28732b8ad1fff3ed8a0804250d6/rpull.c +++ rpull.c

On Sun, 24 Apr 2005, Andreas Gal wrote:

Updating the head pointer (in either direction) should be instead of specifying a commit, and should also apply to http-pull. I've also suggested some changes to the organization of HEAD and related items, so the logical things to read and write are likely to change soon.

	-Daniel
*This .sig left intentionally blank*

Why? Updating HEAD right after writing the commit id and all its children to the object directory seems reasonable and prevents race conditions when the remote repository is shared via HTTP etc. rpull, in contrast, should never touch HEAD, because conflicts might force a merge that will set HEAD to something else. For the pull case we should let the script(tm) do that. Its local anyway. rpush is different because the script(tm) has to do some SSH magic to update HEAD. I will gladly supply a patch to fix what to read/write once you have figured out the final layout, but I really need a working rpush _NOW_ ;).

Andreas

On Mon, 25 Apr 2005, Daniel Barkalow wrote:

On Sun, 24 Apr 2005, Andreas Gal wrote:

Come to think of it, the option I was thinking of is orthogonal to your option; I was thinking of an option to read the head from the sending side's file, rather than from the command line.

In any case, if you're sharing the repository by HTTP, there's no hurry to update the HEAD right after, since the old head doesn't stop being valid (although it's obviously not going to be current in a moment).

For the rpull case, what you want is to just say "HEAD" (or something), and the remote server will send you the HEAD and you pull down that commit. You're probably right that you don't want to automatically write it to the local HEAD, though, although the future format should give us somewhere good to put the result.

That was actually my motice in getting rpush/rpull/http-pull in, too.

If you're going to do much serious with this, you should probably remove the if (has_sha1_file()) continue;" bit in process_commit(), so that it will make sure that the repository gets pulled completely with -a, even if some commits have already been pulled. (This will make things less efficient, but less error-prone, and we'll fix the inefficiency later.)

	-Daniel
*This .sig left intentionally blank*

On Sun, Apr 24, 2005 at 09:01:28PM -0400, David A. Wheeler wrote:

I solve this with two simple scripts, "sign" calls "cutsig".

-------------- sign

#!/bin/sh

DEFAULT_KEY="my-private-key-string"
CUTSIG=~/bin/cutsig.pl
usage()
{
    echo "usage: $0 filename"
    echo " produces filename.sign"
}

if [ $# -lt 1 ]; then
   usage
   exit 1;
fi

gpg --armor --clearsign --detach-sign --default-key "${DEFAULT_KEY} -v -v -o - ${1} | \ ${CUTSIG} > ${1}.sign

exit 0

----------------- cutsig

#!/usr/bin/perl -w

do {
    $line = <STDIN>;
} until $line =~ "-----BEGIN PGP SIGNATURE-----";

print $line;
while ( $line = <STDIN>) {
    print $line;
}

exit 0;

On Sun, Apr 24, 2005 at 09:34:20PM -0500, Matt Domsch wrote:

...

You could also just leave out the --clearsign option and it will DTRT.

Jan

Hello,

# This creates only the signature in Ascii Armor. gpg -a --detach-sign < to_sign > signature

	Thomas

Hello,

# And to verify: gpg --verify signature to_sign

	Thomas

On Sat, Apr 23, 2005 at 11:30:36AM -0700, Linus Torvalds wrote:

It is a bit more messy, but it can be done with a detached signature.

To sign,
    gpg -ab unsigned_commit
    cat unsigned_commit unsigned_commit.asc > signed_commit

To verify,
    cat signed_commit | sed '/-----BEGIN PGP/Q' | gpg --verify signed_commit -

Jan

On Sat, 23 Apr 2005, Jan Harkes wrote:

Ok, this looks more like it.

Except:

Except I think you'd need to searc for the "---BEGIN PGP" starting from the end, rather than the beginning.

Anyway, that should be workable. I'll whip something up.

		Linus

LT> I really want the resulting file to look like

LT> commit .... LT> tag ...

LT> here goes comment LT> here goes signature

LT> and no headers.

You can use --detach-sign with --armor, like this.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
#!/bin/sh

sq=s/\'/\''\\'\'\'/g
usage="usage: $0  [--signer=...] commit-id tag < message"
while case "$#" in 0) break;; esac
do
 case "$1" in
 -s=*|--s=*|--si=*|--sig=*|--sign=*|--signe=*|--signer=*)
  signer=`expr "$1" : '-[^=]*=\(.*\)'` ;;
 -s|--s|--si|--sig|--sign|--signe|--signer)
  case "$#" in 0 | 1) echo "$usage"; exit 1 ;; esac
  signer="${2?}"
  shift ;;
 --)
  shift
  break ;;
 -*)
  echo "$usage"
  exit 1 ;;
 *)
  break ;;
 esac
 shift
done

case "$#" in 2) echo >&2 "$usage"; exit 1 ;; esac commit="$1" tag="$2"

case "$signer" in '') signer_arg='' ;; ?*) signer_arg="--local-user '$(echo "$signer" | sed -e "$sq")'" ;; esac

tmp=.jit-tag.$$ trap 'rm -f $tmp-*' 0 1 2 3 15 tagblob=$tmp-tagblob tagsign=$tmp-tagsign

case $(cat-file -t "$commit" 2>/dev/null) in
commit) ;;
*) echo >&2 "$0: $commit is not a commit object"; exit 1 ;;
esac
{
    echo "commit $commit"
    echo "tag $tag"
    case "$signer" in
    '') ;;
    ?*) echo "signer $signer" ;;
    esac
    echo
    tty -s && echo >&2 "Type your tag message and end with ^D."
    cat
} >$tagblob || exit
gpgcmd="gpg $signer_arg -a --output $tagsign --detach-sign $tagblob"
eval "$gpgcmd" || exit
cat $tagblob $tagsign

On Sat, Apr 23, 2005 at 10:31:28AM -0700, Linus Torvalds wrote:

I see how we can use such a tag object to find a specific commit object in the tree. But if you put the tag objects in the tree as well we now have to figure out a way to find the tag objects.

Why not keep the tags object outside of the tree in the tags/ directory. That way it is easy to find them, and simple to validate all tags or update the signatures if you lost your key.

Others already mentioned the gpg clearsign and verify options, to find a public key that you haven't seen before it is probably easiest to use a keyserver. If verify complains that it doesn't know a key it will print a key-id that identifies it. That id can then be looked up as follows,

    gpg --keyserver wwwkeys.pgp.net --search-keys 0xA86B35C5
    gpg: searching for "0xA86B35C5" from hkp server wwwkeys.pgp.net
    (1)     Linus Torvalds <Linus.Torvalds@Helsinki.FI>
		1024 bit RSA key A86B35C5, created: 1996-06-08
    Keys 1-1 of 1 for "0xA86B35C5".  Enter number(s), N)ext, or Q)uit > q

Ofcourse trusting a key obtained this way is another thing altogether, and would probably depend on who signed it and such.

Jan

On Sat, 23 Apr 2005, Jan Harkes wrote:

Because then you have all those special cases with fetching them and with fsck, and with shared object directories. In other words: no.

You can have symlinks (or even better, just a single file with all the tags listed, which you can create with "fsck", for example) from the tags/ directory, but the thing is, objects go in the object directory and nowhere else.

			Linus

On Sat, Apr 23, 2005 at 12:30:38PM -0700, Linus Torvalds wrote:

I respectfully disagree,

rsync works fine for now, but people are already looking at implementing smarter (more efficient) ways to synchronize git repositories by grabbing missing commits, and from there fetching any missing tree and file blobs. However there is no such linkage to discover missing tag objects, only a full rsync would be able to get them and for that it has to send the name of every object in the repository to the other side to check for any missing ones.

So fetching tags is already going to be a special case.

And any form of validation of a tag is a special operation. In fact tags could be as simple as a the sha of an (like pasky's tags) followed by the detached pgp signature of the tagged object instead of trying to signing the tag itself. That also avoids having to strip the signature part from the tag when we want to validate it.

Jan

Jan Harkes wrote:

I think that one reason why rsync is inefficient for git is that it appears to need an acknowledgement after every file. (at least, that's what what the rhythm of the packets looked like when I sniffed it earlier, I don't know anything else about it) For a series of very small files this interacts badly with tcp's flow control mechanisms. Perhaps rsync could be modified for a "sliding file acknowledgement window".

Most "swarming protocols" (e.g BitTorrent, eDonkey) work well for one big file shared among multiple hosts, but poorly for lots of small files.

*Nothing* out there matches the simplicity of git's sha1 filename length... but

Something like robcast or fcast/flute might be of interest:

http://www.inrialpes.fr/planete/people/roca/mcl/mcl_in_short.html

Or one of the multicast netnews experiments:

"mcntp" http://mcntp.sourceforge.net/ "newscaster" http://www.dmn.tzi.org/en/newscaster.html

lastly, Monotone has it's own "netsync" protocol (via http://www.venge.net/monotone/faq.html)

"[netsync] is a bi-directional pipelined protocol for synchronizing collections using a tree of hashed indices. It allows any copy of monotone to function as either a client or a server, and rapidly synchronize or half-synchronize (push / pull) their database with another user. It is somewhat similar in flavor to rsync or Unison, in that it quickly and idempotently synchronizes information across the network without needing to store any local state; however, it is much more efficient than these protocols."

On Sat, Apr 23, 2005 at 02:28:59PM -0700, Mike Taht wrote:

Actually I just realized that I personally probably wouldn't care about most of the tags that people might add to their trees. Maybe once in a while, but the tag would probably be obtained through email or the web.

...

Interesting, probably something like any of these might end up useful to replace rsync for mirroring full git repositories.

I'm actually more selfish than that and am thinking on how I expect to use git.

See, I don't care about most of the objects in the repository. In practice I would probably pull only the latest 'head' once in a while look for missing commits to give me a quick overview of what has changed. Then if a diff between the new head and my current tree shows that anything might have changed in an area I actually do care about, such as the VFS, I'd want something that does a quick binary search to identify the commits where the changes occured. But for that I only need to look at a limited number of tree objects.

As long as I know that someone, somewhere is archiving the whole repository I can always come back later and fill in the blanks.

HTTP/1.1 with persistent connections and some request interleaving is probably the fastest and most server friendly way to grab those objects I really care about.

Jan

On Sat, 23 Apr 2005, Jan Harkes wrote:

Bit this is a _feature_.

Other people normally shouldn't be interested in your tags. I think it's a mistake to make everybody care.

So you normally would fetch only tags you _know_ about. For example, one of the reasons we've been _avoiding_ personal tags in teh BK trees is that it just gets really ugly really quickly because they get percolated up to everybody else. That means that in a BK tree, you can't sanely use tags for "private" stuff, like telling somebody else "please sync with this tag".

So having the tag in the object database means that fsck etc will notice these things, and can build up a list of tags you know about. It also means that you can have tag-aware synchronization tools, ie exactly the kind of tools that only grab missing commits can also then be used to select missing tags according to some _private_ understanding of what tags you might want to find..

		Linus

Linus Torvalds wrote:

I'm thinking out loud here, but maybe instead of just "tag objects", how about broadening them into "assertion objects" that include (1) a claim and (2) a signature. Monotone, for example, can do this -- it has a generalized mechanism for signed assertions.

A claim might be that some commit is a tag ("Linus Torvalds says this is tag 2.6.12-rc2") or even "I did this" ("Greg KH really committed this"). You could create the latter objects when you push (just sign the head commit), and that would make it really easy to do checks across an entire repository (who said what?). Currently people can send signed emails that they really DID commit something, but then that data's not available to everyone else. Signed assertions about commits would suddenly make it possible for arbitrary people to detect & counter subverted repositories (if they have the public key list); fsck-cache could check signatures as it went.

With a more generalized assertion mechanism, you could make the same assertion multiple times, e.g., if your key gets captured, you could go back and re-sign with a new key, simply creating a new assertion object. You can also make assertions about multiple objects (e.g., assert some relationship between far-removed commits). I'd expect assertion objects to be stored by their hash, just like any other object.

Yes, that makes sense. I think .git/tags/xxx should point to the assertion objects that claim they are tags; you can then check if you accept the assertion object's signature when you try to use it (and locally cache that acceptance once you've checked the signature).

For generalized assertion objects, you need a way to travel FROM the object(s) being described TO the assertion object. A simple method might be to create subdirectories with the name derived from the object(s) being described, and inside the subdirectory have a set of files that have the hashes of the assertion objects. E.G., this kind of structure

  00/
    10f32aca7ba78e2cd95dcfedee0e6329edb735      (commit object)
    10f32aca7ba78e2cd95dcfedee0e6329edb735.d/
      1038e8b8e04b287ec876594cbab9df4af09ce131 ->
                        ../../1038e8b8e04b287ec876594cbab9df4af09ce131
  10/
    38e8b8e04b287ec876594cbab9df4af09ce131     (assertion object)

There's no reason you can't point to assertions from multiple places, which would make them cheap to find when needed.

One problem with symlinks is that some dopey filesystems don't support them :-(. In this case, though, if they got copied multiple times they'd just waste space & not interfere with meaning, since they'd all be static read-only. An alternative would be 0-length files whose names are the hashes.

...

--- David A. Wheeler

Linus Torvalds wrote:

groovy

[tangent]

Any chance you'll have a tree tagged with older releases? Is someone with access to BK working on that?

I do a lot of patch merges where someone sends me a 2.6.10 patch. Presuming the fix is still valid, I'll clone to 2.6.10, merge the patch, pull 2.6.latest into the 2.6.10-based repo, then push the whole she-bang into one of my for-upstream repos.

	Jeff

On Sat, 2005-04-23 at 10:31 -0700, Linus Torvalds wrote:

I'm slightly concerned that to find a given tag by its name if we do _just_ the above would be a fairly slow process. I suspect you'll want a .git/tags/ directory _anyway_, but with named files which refer to tag objects, instead of directly to commit objects as in Petr's current implementation.

Other operations we might want to be at least _reasonably_ efficient would include 'show me the latest tag from Linus' and 'show me all extant tags'.