{"thread":{"id":"35629","subject":"Verifiable git archives?","startedAt":"2014-01-09T03:10:32Z","lastAt":"2014-01-25T21:56:53Z","messageCount":9,"participants":["Andy Lutomirski","Stefan Beller","Junio C Hamano","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"232941","messageId":"CALCETrU88evB6VQrE8=8vrc+HYXAX8_Zx7TsYZp6YXeE4dZdvg@mail.gmail.com","threadId":"35629","inReplyTo":null,"subject":"Verifiable git archives?","fromName":"Andy Lutomirski","fromEmail":"luto@amacapital.net","sentAt":"2014-01-09T03:10:32Z","receivedAt":"2014-01-09T03:10:32Z","isPatch":false,"sender":{"key":"luto@amacapital.net","avatar":null},"body":"It's possible, in principle, to shove enough metadata into the output\nof 'git archive' to allow anyone to verify (without cloning the repo)\nto verify that the archive is a correct copy of a given commit. Would\nthis be considered a useful feature?\n\nPresumably there would be a 'git untar' command that would report\nfailure if it fails to verify the archive contents.\n\nThis could be as simple as including copies of the commit object and\nall relevant tree objects and checking all of the hashes when\nuntarring.\n\n(Even better: allow subsets of the repository to be archived and\nverified as well.)\n\n--Andy\n"},{"id":"232961","messageId":"52CEF7E8.3060100@googlemail.com","threadId":"35629","inReplyTo":"CALCETrU88evB6VQrE8=8vrc+HYXAX8_Zx7TsYZp6YXeE4dZdvg@mail.gmail.com","subject":"Re: Verifiable git archives?","fromName":"Stefan Beller","fromEmail":"stefanbeller@googlemail.com","sentAt":"2014-01-09T19:26:32Z","receivedAt":"2014-01-09T19:26:32Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On 09.01.2014 04:10, Andy Lutomirski wrote:\n> It's possible, in principle, to shove enough metadata into the output\n> of 'git archive' to allow anyone to verify (without cloning the repo)\n> to verify that the archive is a correct copy of a given commit. Would\n> this be considered a useful feature?\n> \n\nDo you know git bundles?\n\n\n> Presumably there would be a 'git untar' command that would report\n> failure if it fails to verify the archive contents.\n> \n> This could be as simple as including copies of the commit object and\n> all relevant tree objects and checking all of the hashes when\n> untarring.\n> \n\nI thought the git archive rather had the purpose of creating plain\narchives not polluted with any gitish stuff.\n\n> (Even better: allow subsets of the repository to be archived and\n> verified as well.)\n\nStefan\n"},{"id":"232965","messageId":"xmqqsiswapav.fsf@gitster.dls.corp.google.com","threadId":"35629","inReplyTo":"CALCETrU88evB6VQrE8=8vrc+HYXAX8_Zx7TsYZp6YXeE4dZdvg@mail.gmail.com","subject":"Re: Verifiable git archives?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-01-09T20:11:20Z","receivedAt":"2014-01-09T20:11:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Lutomirski writes:\n\n> It's possible, in principle, to shove enough metadata into the output\n> of 'git archive' to allow anyone to verify (without cloning the repo)\n> to verify that the archive is a correct copy of a given commit. Would\n> this be considered a useful feature?\n>\n> Presumably there would be a 'git untar' command that would report\n> failure if it fails to verify the archive contents.\n>\n> This could be as simple as including copies of the commit object and\n> all relevant tree objects and checking all of the hashes when\n> untarring.\n\nYou only need the object name of the top-level tree. After \"untar\"\nthe archive into an empty directory, make it a new repository and\n\"git add . && git write-tree\"---the result should match the\ntop-level tree the archive was supposed to contain.\n\nOf course, you can write \"git verify-archive\" that does the same\ncomputation all in-core, without actually extracting the archive\ninto an empty directory.\n"},{"id":"232966","messageId":"CALCETrXTbvHQAWm2z13OwyHCmBZZNDSqyWCdiu6maHRp2mi-Pw@mail.gmail.com","threadId":"35629","inReplyTo":"xmqqsiswapav.fsf@gitster.dls.corp.google.com","subject":"Re: Verifiable git archives?","fromName":"Andy Lutomirski","fromEmail":"luto@amacapital.net","sentAt":"2014-01-09T20:51:22Z","receivedAt":"2014-01-09T20:51:22Z","isPatch":false,"sender":{"key":"luto@amacapital.net","avatar":null},"body":"On Thu, Jan 9, 2014 at 12:11 PM, Junio C Hamano wrote:\n> Andy Lutomirski writes:\n>\n>> It's possible, in principle, to shove enough metadata into the output\n>> of 'git archive' to allow anyone to verify (without cloning the repo)\n>> to verify that the archive is a correct copy of a given commit. Would\n>> this be considered a useful feature?\n>>\n>> Presumably there would be a 'git untar' command that would report\n>> failure if it fails to verify the archive contents.\n>>\n>> This could be as simple as including copies of the commit object and\n>> all relevant tree objects and checking all of the hashes when\n>> untarring.\n>\n> You only need the object name of the top-level tree. After \"untar\"\n> the archive into an empty directory, make it a new repository and\n> \"git add . && git write-tree\"---the result should match the\n> top-level tree the archive was supposed to contain.\n\nHmm. I didn't realize that there was enough metadata in the 'git\narchive' output to reproduce the final tree. If I can make it work,\nwould you accept a patch to add another extended pax header containing\nthe commit object and the top-level tree hash to the 'git archive'\ntarball output?\n\n>\n> Of course, you can write \"git verify-archive\" that does the same\n> computation all in-core, without actually extracting the archive\n> into an empty directory.\n\nHmm. I'll play with this.\n\n--Andy\n"},{"id":"232976","messageId":"xmqq38kwai3v.fsf@gitster.dls.corp.google.com","threadId":"35629","inReplyTo":"CALCETrXTbvHQAWm2z13OwyHCmBZZNDSqyWCdiu6maHRp2mi-Pw@mail.gmail.com","subject":"Re: Verifiable git archives?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-01-09T22:46:44Z","receivedAt":"2014-01-09T22:46:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Lutomirski writes:\n\n>> You only need the object name of the top-level tree. After \"untar\"\n>> the archive into an empty directory, make it a new repository and\n>> \"git add . && git write-tree\"---the result should match the\n>> top-level tree the archive was supposed to contain.\n>\n> Hmm. I didn't realize that there was enough metadata in the 'git\n> archive' output to reproduce the final tree.\n\nWe do record the commit object name in the extended header when\nwriting a tar archive already, but you have to grab the commit\nobject from somewhere in order to read the top-level tree object\nname, which we do not record.\n\nAlso, if you used keyword substitution and such when creating an\narchive, then the filesystem entities resulting from expanding it\nwould not match the original.\n"},{"id":"232977","messageId":"CALCETrWjHnpZxdsE3z_rZEUsNfdheDjSsZ2=yaxW8Hz4v05dLA@mail.gmail.com","threadId":"35629","inReplyTo":"xmqq38kwai3v.fsf@gitster.dls.corp.google.com","subject":"Re: Verifiable git archives?","fromName":"Andy Lutomirski","fromEmail":"luto@amacapital.net","sentAt":"2014-01-09T22:50:08Z","receivedAt":"2014-01-09T22:50:08Z","isPatch":false,"sender":{"key":"luto@amacapital.net","avatar":null},"body":"On Thu, Jan 9, 2014 at 2:46 PM, Junio C Hamano wrote:\n> Andy Lutomirski writes:\n>\n>>> You only need the object name of the top-level tree. After \"untar\"\n>>> the archive into an empty directory, make it a new repository and\n>>> \"git add . && git write-tree\"---the result should match the\n>>> top-level tree the archive was supposed to contain.\n>>\n>> Hmm. I didn't realize that there was enough metadata in the 'git\n>> archive' output to reproduce the final tree.\n>\n> We do record the commit object name in the extended header when\n> writing a tar archive already, but you have to grab the commit\n> object from somewhere in order to read the top-level tree object\n> name, which we do not record.\n\nThis could be changed :)\n\n>\n> Also, if you used keyword substitution and such when creating an\n> archive, then the filesystem entities resulting from expanding it\n> would not match the original.\n>\n\nIn the simple case, you'd need to have an archive with no prefix or\nfunny business (or at least a known prefix). In the fancy case, you\ncould at least verify that all the file contents really came from git,\nbut then you'd really need the tree objects.\n\nThe use case I have in mind is for projects to distribute archives but\nonly need to sign the tagged git commit id. I think this should be\ndoable without too much pain. (This assumes that the release doesn't\ncontain autogen output and such.)\n\n--Andy\n"},{"id":"233360","messageId":"52DB1DB4.7080806@alum.mit.edu","threadId":"35629","inReplyTo":"xmqqsiswapav.fsf@gitster.dls.corp.google.com","subject":"Re: Verifiable git archives?","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-01-19T00:35:00Z","receivedAt":"2014-01-19T00:35:00Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/09/2014 09:11 PM, Junio C Hamano wrote:\n> Andy Lutomirski writes:\n> \n>> It's possible, in principle, to shove enough metadata into the output\n>> of 'git archive' to allow anyone to verify (without cloning the repo)\n>> to verify that the archive is a correct copy of a given commit. Would\n>> this be considered a useful feature?\n>>\n>> Presumably there would be a 'git untar' command that would report\n>> failure if it fails to verify the archive contents.\n>>\n>> This could be as simple as including copies of the commit object and\n>> all relevant tree objects and checking all of the hashes when\n>> untarring.\n> \n> You only need the object name of the top-level tree. After \"untar\"\n> the archive into an empty directory, make it a new repository and\n> \"git add . && git write-tree\"---the result should match the\n> top-level tree the archive was supposed to contain.\n> [...]\n\nThis wouldn't work if any files were excluded from the archive using\ngitattribute \"export-ignore\" (or \"export-subst\", which you already\nmentioned in a follow-up email).\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"233453","messageId":"xmqqiotdunus.fsf@gitster.dls.corp.google.com","threadId":"35629","inReplyTo":"52DB1DB4.7080806@alum.mit.edu","subject":"Re: Verifiable git archives?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-01-21T19:38:35Z","receivedAt":"2014-01-21T19:38:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n> On 01/09/2014 09:11 PM, Junio C Hamano wrote:\n>> Andy Lutomirski writes:\n>> \n>>> It's possible, in principle, to shove enough metadata into the output\n>>> of 'git archive' to allow anyone to verify (without cloning the repo)\n>>> to verify that the archive is a correct copy of a given commit. Would\n>>> this be considered a useful feature?\n>>>\n>>> Presumably there would be a 'git untar' command that would report\n>>> failure if it fails to verify the archive contents.\n>>>\n>>> This could be as simple as including copies of the commit object and\n>>> all relevant tree objects and checking all of the hashes when\n>>> untarring.\n>> \n>> You only need the object name of the top-level tree. After \"untar\"\n>> the archive into an empty directory, make it a new repository and\n>> \"git add . && git write-tree\"---the result should match the\n>> top-level tree the archive was supposed to contain.\n>> [...]\n>\n> This wouldn't work if any files were excluded from the archive using\n> gitattribute \"export-ignore\" (or \"export-subst\", which you already\n> mentioned in a follow-up email).\n\nCorrect. By \"and such\" below, I meant any and all futzing that\nmakes the resulting working tree different from the tree object\nbeing archived ;-) That includes the line-ending configuration\nand other things as well.\n\n Also, if you used keyword substitution and such when creating an\n archive, then the filesystem entities resulting from expanding\n it would not match the original.\n"},{"id":"233757","messageId":"CALCETrVq_+o44EqvrOJNUfEuDMOtMketa+KrxrZ0tLwVErx7EQ@mail.gmail.com","threadId":"35629","inReplyTo":"xmqqiotdunus.fsf@gitster.dls.corp.google.com","subject":"Re: Verifiable git archives?","fromName":"Andy Lutomirski","fromEmail":"luto@amacapital.net","sentAt":"2014-01-25T21:56:53Z","receivedAt":"2014-01-25T21:56:53Z","isPatch":false,"sender":{"key":"luto@amacapital.net","avatar":null},"body":"Here's a rather hackish implementation of the write side. Any\nthoughts on the format? (Obviously the implementation needs work.\nFor example, it needs to be optional.\n\nThoughts so far:\n - I want to put the value of \"prefix\" into an extended header.\n - Should blobs have their sha1 hashes in an extended header? Pros:\nit makes figuring out substitutions easier. Cons: it adds 512 bytes\nper file.\n - I want to support tags as roots.\n - I (or someone) need to write a verifier / verified unpacker. Does\ngit accept Python code?\n\nThis thing is tested in the sense that GNU tar unpacks its output\nwithout any warnings or other fanfare.\n\n--Andy\n\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 719b629..c6bf7e4 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -2,6 +2,8 @@\n * Copyright (c) 2005, 2006 Rene Scharfe\n */\n #include \"cache.h\"\n+#include \"tree.h\"\n+#include \"object.h\"\n #include \"tar.h\"\n #include \"archive.h\"\n #include \"streaming.h\"\n@@ -200,6 +202,74 @@ static int write_extended_header(struct archiver_args *args,\n \treturn 0;\n }\n \n+/*\n+ * A GIT-SCM object header is a global extended header that embeds a single\n+ * git object. This object serves a purpose described by the \"purpose\"\n+ * field. Valid purposes include:\n+ *\n+ * - \"root\" -- an object that, by itself, in conjunction with other roots,\n+ * or in conjunction with external data, identifies a root to use to\n+ * verify this archive.\n+ * - \"vrfy\" -- an object that can be use to prove that the contents\n+ * of this archive are as described.\n+ *\n+ * There's one basic rule to observe: every \"vrfy\" object must hash to\n+ * a SHA-1 that matches something described in a \"root\", another \"vrfy\" object,\n+ * or something typed in by a user decoding the archive.\n+ *\n+ * (Of course, if you want the archive to be usefully verifiable, all of the\n+ * non-GIT-SCM contents should also be attributable to an appropriate\n+ * \"vrfy\" object.)\n+ *\n+ * The fields are:\n+ * GIT-SCM.obj.purpose: the purpose of the embedded object\n+ * GIT-SCM.obj.sha1: the sha1 of the embedded object\n+ * GIT-SCM.obj.type: the type of the embedded object\n+ * GIT-SCM.obj.data: the data in the embedded object\n+ *\n+ * The block header is intentionally unspecified, except that it must\n+ * have typeflag 'g'. (This is to allow some flexibility in trying to\n+ * preserve compatibility with old tar implementations.)\n+ */\n+static int write_gitscm_obj_header(struct archiver_args *args,\n+\t\t\t\t const char *purpose,\n+\t\t\t\t const unsigned char *sha1)\n+{\n+\tstruct strbuf ext_header = STRBUF_INIT;\n+\tstruct ustar_header header;\n+\tunsigned int mode;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *buffer;\n+\tconst char *typestr;\n+\tint err = 0;\n+\n+\tstrbuf_append_ext_header(&ext_header, \"GIT-SCM.obj.purpose\",\n+\t\t\t\t purpose, strlen(purpose));\n+\tstrbuf_append_ext_header(&ext_header, \"GIT-SCM.obj.sha1\",\n+\t\t\t\t sha1_to_hex(sha1), 40);\n+\n+\tbuffer = read_sha1_file(sha1, &type, &size);\n+\ttypestr = typename(type);\n+\n+\tstrbuf_append_ext_header(&ext_header, \"GIT-SCM.obj.type\",\n+\t\t\t\t typestr, strlen(typestr));\n+\tstrbuf_append_ext_header(&ext_header, \"GIT-SCM.obj.data\",\n+\t\t\t\t buffer, size);\n+\tfree(buffer);\n+\tbuffer = NULL;\n+\n+\tmemset(&header, 0, sizeof(header));\n+\t*header.typeflag = TYPEFLAG_GLOBAL_HEADER;\n+\tmode = 0100666;\n+\tstrcpy(header.name, \"pax_global_header\");\n+\tprepare_header(args, &header, mode, ext_header.len);\n+\twrite_blocked(&header, sizeof(header));\n+\twrite_blocked(ext_header.buf, ext_header.len);\n+\tstrbuf_release(&ext_header);\n+\treturn err;\n+}\n+\n static int write_tar_entry(struct archiver_args *args,\n \t\t\t const unsigned char *sha1,\n \t\t\t const char *path, size_t pathlen,\n@@ -212,6 +282,10 @@ static int write_tar_entry(struct archiver_args *args,\n \tvoid *buffer;\n \tint err = 0;\n \n+\tif (S_ISDIR(mode)) {\n+\t\twrite_gitscm_obj_header(args, \"vrfy\", sha1);\n+\t}\n+\n \tmemset(&header, 0, sizeof(header));\n \n \tif (S_ISDIR(mode) || S_ISGITLINK(mode)) {\n@@ -384,8 +458,11 @@ static int write_tar_archive(const struct archiver *ar,\n \n \tif (args->commit_sha1)\n \t\terr = write_global_extended_header(args);\n-\tif (!err)\n+\tif (!err) {\n+\t\twrite_gitscm_obj_header(args, \"root\", args->commit_sha1);\n+\t\twrite_gitscm_obj_header(args, \"vrfy\", args->tree->object.sha1);\n \t\terr = write_archive_entries(args, write_tar_entry);\n+\t}\n \tif (!err)\n \t\twrite_trailer();\n \treturn err;\n"}]}