{"thread":{"id":"41255","subject":"git archive should use vendor extension in pax header","startedAt":"2016-01-24T15:59:09Z","lastAt":"2016-02-15T20:25:40Z","messageCount":9,"participants":["fuz@fuz.su","René Scharfe","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"276664","messageId":"20160124155909.GA16847@fuz.su","threadId":"41255","inReplyTo":null,"subject":"git archive should use vendor extension in pax header","fromName":"","fromEmail":"fuz@fuz.su","sentAt":"2016-01-24T15:59:09Z","receivedAt":"2016-01-24T15:59:09Z","isPatch":false,"sender":{"key":"fuz@fuz.su","avatar":null},"body":"Right now, git archive creates a pax global header of the form\n\n    comment=57ca140635bf157354124e4e4b3c8e1bde2832f1\n\nin tar archives it creates. This is suboptimal as as comments are\nspecified to be ignored by extraction software. It is impossible to\nfind out in an automatic way (short of guessing) that this is supposed\nto be a commit hash. It would be much more useful if git created a\ncustom key. As per POSIX suggestions, something like this would be\nappropriate:\n\n    GIT.commit=57ca140635bf157354124e4e4b3c8e1bde2832f1\n\nPlease consider this suggestion.\n\nYours sincerely,\nRobert Clausecker\n\n-- \n()  ascii ribbon campaign - for an 8-bit clean world \n/\\  - against html email  - against proprietary attachments\n"},{"id":"276840","messageId":"56A7EDE1.1020909@web.de","threadId":"41255","inReplyTo":"20160124155909.GA16847@fuz.su","subject":"Re: git archive should use vendor extension in pax header","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2016-01-26T22:06:25Z","receivedAt":"2016-01-26T22:06:25Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 24.01.2016 um 16:59 schrieb fuz@fuz.su:\n> Right now, git archive creates a pax global header of the form\n>\n>      comment=57ca140635bf157354124e4e4b3c8e1bde2832f1\n>\n> in tar archives it creates. This is suboptimal as as comments are\n> specified to be ignored by extraction software. It is impossible to\n> find out in an automatic way (short of guessing) that this is supposed\n> to be a commit hash.\n\nThis is only a problem if you don't know how a given tar files was \ncreated (or modified later).  How did you get into this situation?  Or \nin other words: Please tell me more about your use case.\n\n> It would be much more useful if git created a\n> custom key. As per POSIX suggestions, something like this would be\n> appropriate:\n>\n>      GIT.commit=57ca140635bf157354124e4e4b3c8e1bde2832f1\n\nThis would be included in addition to the comment in order to avoid \nbreaking existing users, I guess.\n\nIf you have a random archive and want to know if it was generated by git \nthen your next question might be which options and substitutions were \nused.  That reminds me of this thread regarding verifiable archives:\n\n     http://article.gmane.org/gmane.comp.version-control.git/240244\n\nThanks,\nRené\n"},{"id":"276945","messageId":"20160127232534.GA5435@fuz.su","threadId":"41255","inReplyTo":"56A7EDE1.1020909@web.de","subject":"Re: git archive should use vendor extension in pax header","fromName":"","fromEmail":"fuz@fuz.su","sentAt":"2016-01-27T23:25:34Z","receivedAt":"2016-01-27T23:25:34Z","isPatch":false,"sender":{"key":"fuz@fuz.su","avatar":null},"body":"On Tue, Jan 26, 2016 at 11:06:25PM +0100, René Scharfe wrote:\n> Am 24.01.2016 um 16:59 schrieb fuz@fuz.su:\n> >Right now, git archive creates a pax global header of the form\n> >\n> >     comment=57ca140635bf157354124e4e4b3c8e1bde2832f1\n> >\n> >in tar archives it creates. This is suboptimal as as comments are\n> >specified to be ignored by extraction software. It is impossible to\n> >find out in an automatic way (short of guessing) that this is supposed\n> >to be a commit hash.\n> \n> This is only a problem if you don't know how a given tar files was\n> created (or modified later).  How did you get into this situation?\n> Or in other words: Please tell me more about your use case.\n\nMy situation is that I'm interested in knowing if an archive was created\nby git so I can find out where the corresponding repository is and find\nout which commit this archive was created from.  Right now the only way\nis to open a hex editor or as archiving software is instructed to ignore\nthe content of comment headers.  This is clearly a suboptimal situation.\n\n> >It would be much more useful if git created a\n> >custom key. As per POSIX suggestions, something like this would be\n> >appropriate:\n> >\n> >     GIT.commit=57ca140635bf157354124e4e4b3c8e1bde2832f1\n> \n> This would be included in addition to the comment in order to avoid\n> breaking existing users, I guess.\n\nGood point.  I'm not sure how many user use the comment header at all.\n\n> If you have a random archive and want to know if it was generated by\n> git then your next question might be which options and substitutions\n> were used.  That reminds me of this thread regarding verifiable\n> archives:\n> \n>     http://article.gmane.org/gmane.comp.version-control.git/240244\n\nGood point.  Something like this should be enough to be enough to have\nreproducable archives if archives with a tree ID were to have a time\nstamp of 0 (1970-01-01) instead of the current date:\n\n    comment=...    (for compatibility)\n    GIT.commit=... (like comment)\n    GIT.umask=...  (tar.umask)\n    GIT.prefix=... (--prefix=)\n    GIT.path=...   (see below\n    GIT.export-subst=1 (in extended header instead of global header)\n\nA different key such as GIT.treeish might be appropriate.  The\nGIT.export-subst key should be set only for those files where a\nsubstitution has taken place. Maybe there should also be an\nGIT.original-name key.\n\nAn option GIT.export-ignore is not required.  Instead it would be more\nuseful to have a special file type G (for git) with the convention that\nthe file name .gitattributes means “attributes that apply to this git\narchive.”\n\nThe GIT.path option holds the paths that are being archived. It is a bit\ntricky to get right.  The intent of POSIX pax headers is that each key\nis an attribute that applies to a series of files.  In the case of a\nglobal header, each key applies until it is overridden with a new\nheader or with a local header.  A GIT.path key should only apply to the\nfiles that correspond to this path operant to git archive.  Thus, a new\nGIT.path should be written frequently.  There should always be at least\none GIT.path.\n\nIt might be a good idea to be able to control the kind of metadata git\nadds to the archive as to be able to not leak any confidential\ninformation with git archive.  If you are interested I can try to make a\nspecification for these headers.\n\nYours sincerely,\nRobert Clausecker\n\n-- \n()  ascii ribbon campaign - for an 8-bit clean world \n/\\  - against html email  - against proprietary attachments\n"},{"id":"276947","messageId":"20160127234512.GA5453@fuz.su","threadId":"41255","inReplyTo":"56A92913.3030909@web.de","subject":"Re: git archive should use vendor extension in pax header","fromName":"","fromEmail":"fuz@fuz.su","sentAt":"2016-01-27T23:45:12Z","receivedAt":"2016-01-27T23:45:12Z","isPatch":false,"sender":{"key":"fuz@fuz.su","avatar":null},"body":"Hallo René,\n\nOn Wed, Jan 27, 2016 at 09:31:15PM +0100, René Scharfe wrote:\n> Hello Robert,\n> \n> it's customary to discuss in the open by copying the list.  Unless\n> there are secrets involved, but I don't see any below.  I kept it\n> private anyway in case I missed any, but please cc:\n> git@vger.kernel.org on your reply if possible.\n\nI'm sorry for miss-sending the last mail, I sent it to the list again.\n\n> >>This is only a problem if you don't know how a given tar files was\n> >>created (or modified later).  How did you get into this situation?\n> >>Or in other words: Please tell me more about your use case.\n> >\n> >My situation is that I'm interested in knowing if an archive was created\n> >by git so I can find out where the corresponding repository is and find\n> >out which commit this archive was created from.  Right now the only way\n> >is to open a hex editor or as archiving software is instructed to ignore\n> >the content of comment headers.  This is clearly a suboptimal situation.\n> \n> There is git get-tar-commit-id, which prints the commit ID if it\n> finds a comment entry which looks like a hexadecimal SHA-1 hash.\n> It's better than a hex editor at least. :)\n\nThis is incredibly fuzzy and can get wrong for a pleothora of reasons.\nI hope you agree though that the situation is suboptimal, git is doing\nthe equivalent of using a custom file format without an easily\nrecognizable magic number.\n\n> But I'm still interested how you got a collection of tar files with\n> unknown origin.  Just curious.\n\nEasy: Just download the (source) distribution archives of a distribution\nof choice and try to verify that the tarballs they use to compile their\npackages actually come from the project's public git repositories.\n\nThere are other reasons why an easily detectable git hash might be\nuseful.  For example, file(1) could show that the archive comes from\ngit.  Other utilities could use this to work around git-specific bugs.\nAn unpacker could add corresponding meta-data when unpacking the file.\n\n> >>>It would be much more useful if git created a\n> >>>custom key. As per POSIX suggestions, something like this would be\n> >>>appropriate:\n> >>>\n> >>>     GIT.commit=57ca140635bf157354124e4e4b3c8e1bde2832f1\n> >>\n> >>This would be included in addition to the comment in order to avoid\n> >>breaking existing users, I guess.\n> >\n> >Good point.  I'm not sure how many user use the comment header at all.\n> \n> Apart from git get-tar-commit-id I don't know any program for\n> extracting pax comments.  And I don't know how widely used that is,\n> but I assume there is *someone* out there, extracting commit IDs\n> with it.\n\nNeither do I.  But remember, POSIX explicitly specifies that programs\nthat parse pax file must ignore pax comments so an unpacker that\ninterpretes the content of such a comment in any way is in violation of\nthe pax specification.\n\n> >>If you have a random archive and want to know if it was generated by\n> >>git then your next question might be which options and substitutions\n> >>were used.  That reminds me of this thread regarding verifiable\n> >>archives:\n> >>\n> >>     http://article.gmane.org/gmane.comp.version-control.git/240244\n> >\n> >Good point.  Something like this should be enough to be enough to have\n> >reproducable archives if archives with a tree ID were to have a time\n> >stamp of 0 (1970-01-01) instead of the current date:\n> >\n> >     comment=...    (for compatibility)\n> >     GIT.commit=... (like comment)\n> >     GIT.umask=...  (tar.umask)\n> >     GIT.prefix=... (--prefix=)\n> >     GIT.path=...   (see below\n> >     GIT.export-subst=1 (in extended header instead of global header)\n> >\n> >A different key such as GIT.treeish might be appropriate.  The\n> >GIT.export-subst key should be set only for those files where a\n> >substitution has taken place.\n> \n> What would GIT.export-subst contain? There can be multiple\n> replacements in a file.\n\nGIT.export-subst would only contain a 1 if substitution is turned on.\nThe goal is to have reproduceable archives, not the ability to turn an\narchive back into a git repository.\n\n> >Maybe there should also be an\n> >GIT.original-name key.\n> \n> What would it be used for?\n\nIn case an export substition changes the file name so the implementation\ncan verify that the original file could plausibly have been substituted\ninto the current name.  Also for the case where multiple files\nsubstitute into the same name to tell which file git should check\nequivalency with.\n\n> >An option GIT.export-ignore is not required.  Instead it would be more\n> >useful to have a special file type G (for git) with the convention that\n> >the file name .gitattributes means “attributes that apply to this git\n> >archive.”\n> \n> That would be a non-standard extension.  Archivers would extract\n> these as regular files.  Storing a list of excluded paths (in\n> GIT.exclude or so) might be a better idea.\n\nNo, that's not a good idea as pax headers are interpreted as “attributes\npertaining to a file.”  A file doesn't have the attribute that other\nfiles have been omitted.  Making this a special file type is useful as\nit allows archivers that don't implement git extensions to recover this\ninformation in a useful way (after all, the .gitattributes file took\npart in creating the archive) and, more importantly, reserves a file\ntype for future git extensions.\n\n> >The GIT.path option holds the paths that are being archived. It is a bit\n> >tricky to get right.  The intent of POSIX pax headers is that each key\n> >is an attribute that applies to a series of files.  In the case of a\n> >global header, each key applies until it is overridden with a new\n> >header or with a local header.  A GIT.path key should only apply to the\n> >files that correspond to this path operant to git archive.  Thus, a new\n> >GIT.path should be written frequently.  There should always be at least\n> >one GIT.path.\n> \n> That's for the optional path parameters of git archive, right?  A\n> list of included paths (GIT.include) would be simpler and should\n> suffice, no?\n\nNo.  Again: An attribute in a pax header pertains a file.  It's metadata\nattached to a file, not metadata attached to the whole archive, even when\npart of a global header.  Thus each file should have attached what path\noperand it came from.  A file doesn't have the attribute what other path\noperands git received, only the path operand that caused the inclusion of\nthat one file is an attribute of the file.\n\n> >It might be a good idea to be able to control the kind of metadata git\n> >adds to the archive as to be able to not leak any confidential\n> >information with git archive.  If you are interested I can try to make a\n> >specification for these headers.\n> \n> Which of the field might be sensitive?\n\nThe existence of a git-specific pax header is sensitive as it proves\nthat a git archive of the source code exists.  This can be a problem if\nyou want to plausibly deny the possession of other versions of the\nsource code you distribute.  The existence of export-ignore meta data\nleaks information about what other files are in the repository the\narchive was created from and can be critical.  The existence of\npath-operand meta data can show what path structure the repository has\nwhich can be sensitive.  Basically the existence of any information\nbesides the information you want to add itself is sensitive.\n\n> Users can always go back to the original format.  At least I don't\n> expect this new format becoming the default too quickly.\n\nSure thing.  If this is going to be implemented, I would add options\nto choose what / what style of metadata to include.\n\n> An extractor is needed -- unlike the comment field (which is at\n> least menationed in the spec) I can't see any generic archiver to\n> add support for the git specific fields.\n\nFor most archivers, the support comprises ignoring them (and not warning\nabout the unrecognized fields).  But custom software could use them in\nuseful ways, e.g. to verify the validity of an archive.  The comment\nfield has the reliability problems outlined above.  It's like a file\nformat without a magic number.\n\n> René\n\nBest regards,\nRobert Clausecker\n\n-- \n()  ascii ribbon campaign - for an 8-bit clean world \n/\\  - against html email  - against proprietary attachments\n"},{"id":"276961","messageId":"alpine.DEB.2.20.1601280906340.2964@virtualbox","threadId":"41255","inReplyTo":"20160127234512.GA5453@fuz.su","subject":"Re: git archive should use vendor extension in pax header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-01-28T08:13:03Z","receivedAt":"2016-01-28T08:13:03Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Robert,\n\n[I am not going to re-Cc: the dropped email addresses; please note that it\nis pretty much frowned upon on this mailing list if you do not\nreply-to-all and might affect your conversation.]\n\nOn Thu, 28 Jan 2016, fuz@fuz.su wrote:\n\n> On Wed, Jan 27, 2016 at 09:31:15PM +0100, René Scharfe wrote:\n>\n> > Users can always go back to the original format.  At least I don't\n> > expect this new format becoming the default too quickly.\n\nThis is the most crucial issue here, as far as I am concerned: there are\nalready tons of .zip files out there that were created by git archive, and\nthere will inevitably be loads of tons more *having the current pax header\nformat*.\n\nSo tools wanting to deal with Git archives will have to handle those as\nwell, i.e. do *precisely* as René suggested and use get-tar-commit-id. As\nsuch, the value of changing the format *now* is a bit like closing the\nbarn's door after pretty much all of the horses left (except the old one\nthat has a few troubles getting up in the morning but that is too nice to\nthe kids to shoot).\n\n> Sure thing.  If this is going to be implemented, I would add options\n> to choose what / what style of metadata to include.\n\nWhy not put your money where your mouth is? I.e. get your head down into\nthe code and come up with a patch (because otherwise it is unlikely that\nyour idea will go anywhere)?\n\nCiao,\nJohannes"},{"id":"276979","messageId":"20160128091418.GA6875@fuz.su","threadId":"41255","inReplyTo":"alpine.DEB.2.20.1601280906340.2964@virtualbox","subject":"Re: git archive should use vendor extension in pax header","fromName":"","fromEmail":"fuz@fuz.su","sentAt":"2016-01-28T09:14:18Z","receivedAt":"2016-01-28T09:14:18Z","isPatch":false,"sender":{"key":"fuz@fuz.su","avatar":null},"body":"Hello,\n\n> > > Users can always go back to the original format.  At least I don't\n> > > expect this new format becoming the default too quickly.\n>\n> This is the most crucial issue here, as far as I am concerned: there are\n> already tons of .zip files out there that were created by git archive, and\n> there will inevitably be loads of tons more *having the current pax header\n> format*.\n> \n> So tools wanting to deal with Git archives will have to handle those as\n> well, i.e. do *precisely* as René suggested and use get-tar-commit-id. As\n> such, the value of changing the format *now* is a bit like closing the\n> barn's door after pretty much all of the horses left (except the old one\n> that has a few troubles getting up in the morning but that is too nice to\n> the kids to shoot).\n\nThat's not really an argument.  The situation you describes applies to\nall file formats and it always ends in the same way:  A new file format\nis designed and then slowly adopted by the rest of the users, in case of\ngit I imagine this to be a quick process taking maybe a year or two.\nNewly created files use the new file format and old files still hang\naround but their importance is dwindling until you can safely support\nonly the new format.  But to get there, a new file format has to be\nadopted in the first place.\n\n> > Sure thing.  If this is going to be implemented, I would add options\n> > to choose what / what style of metadata to include.\n> \n> Why not put your money where your mouth is? I.e. get your head down into\n> the code and come up with a patch (because otherwise it is unlikely that\n> your idea will go anywhere)?\n\nI'd love to but I prefer to ask if there is interest in such a change in\nthe first place.  I'm not going to waste my time implementing this and\nthen being told that the git project is not interested in this kind of\nfunctionality.  So can someone give me a clear signal?\n\n> Ciao,\n> Johannes\n\nYours sincerely,\nRobert Clausecker\n\n-- \n()  ascii ribbon campaign - for an 8-bit clean world \n/\\  - against html email  - against proprietary attachments\n"},{"id":"277651","messageId":"56B5F3BF.7080601@web.de","threadId":"41255","inReplyTo":"20160127234512.GA5453@fuz.su","subject":"Re: git archive should use vendor extension in pax header","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2016-02-06T13:23:11Z","receivedAt":"2016-02-06T13:23:11Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 28.01.2016 um 00:45 schrieb fuz@fuz.su:\n>> There is git get-tar-commit-id, which prints the commit ID if it\n>> finds a comment entry which looks like a hexadecimal SHA-1 hash.\n>> It's better than a hex editor at least. :)\n>\n> This is incredibly fuzzy and can get wrong for a pleothora of reasons.\n> I hope you agree though that the situation is suboptimal, git is doing\n> the equivalent of using a custom file format without an easily\n> recognizable magic number.\n\nIt is fuzzy in theory. But which other programs allow writing a comment \nheader?  I'm not aware of any, but I have to admit that I didn't look \ntoo hard.\n\n>> But I'm still interested how you got a collection of tar files with\n>> unknown origin.  Just curious.\n>\n> Easy: Just download the (source) distribution archives of a distribution\n> of choice and try to verify that the tarballs they use to compile their\n> packages actually come from the project's public git repositories.\n\nOK, that's easier than calculating checksums and comparing them with \nthose published by the respective projects, but also less trustworthy.\n\n> There are other reasons why an easily detectable git hash might be\n> useful.  For example, file(1) could show that the archive comes from\n> git.  Other utilities could use this to work around git-specific bugs.\n> An unpacker could add corresponding meta-data when unpacking the file.\n\nfile(1) could use the same heuristic as git get-tar-commit-id. \nSomething like this would work (the first line is already shipped with \nfile):\n\n\t257\tstring\tustar\\0 POSIX tar archive\n\t>156\tstring\tg\n\t>>512\tstring\t52\\ comment=\n\t>>>523\tregex\t[0-9a-f]{40}\t\\b, git commit %s\n\nNB: With Ian Darwin's file you need to use -e tar in order to turn off \nits internal tar test.\n\nI'm very interested in hearing about any git specific bugs.\n\n>>>>> It would be much more useful if git created a\n>>>>> custom key. As per POSIX suggestions, something like this would be\n>>>>> appropriate:\n>>>>>\n>>>>>      GIT.commit=57ca140635bf157354124e4e4b3c8e1bde2832f1\n>>>>\n>>>> This would be included in addition to the comment in order to avoid\n>>>> breaking existing users, I guess.\n>>>\n>>> Good point.  I'm not sure how many user use the comment header at all.\n>>\n>> Apart from git get-tar-commit-id I don't know any program for\n>> extracting pax comments.  And I don't know how widely used that is,\n>> but I assume there is *someone* out there, extracting commit IDs\n>> with it.\n>\n> Neither do I.  But remember, POSIX explicitly specifies that programs\n> that parse pax file must ignore pax comments so an unpacker that\n> interpretes the content of such a comment in any way is in violation of\n> the pax specification.\n\nAlmost right: The spec says that *pax* shall ignore comments.  Which is \ngood -- we can use this field to transport anything without pax complaining.\n\n>>>> If you have a random archive and want to know if it was generated by\n>>>> git then your next question might be which options and substitutions\n>>>> were used.  That reminds me of this thread regarding verifiable\n>>>> archives:\n>>>>\n>>>>      http://article.gmane.org/gmane.comp.version-control.git/240244\n>>>\n>>> Good point.  Something like this should be enough to be enough to have\n>>> reproducable archives if archives with a tree ID were to have a time\n>>> stamp of 0 (1970-01-01) instead of the current date:\n>>>\n>>>      comment=...    (for compatibility)\n>>>      GIT.commit=... (like comment)\n>>>      GIT.umask=...  (tar.umask)\n>>>      GIT.prefix=... (--prefix=)\n>>>      GIT.path=...   (see below\n>>>      GIT.export-subst=1 (in extended header instead of global header)\n>>>\n>>> A different key such as GIT.treeish might be appropriate.  The\n>>> GIT.export-subst key should be set only for those files where a\n>>> substitution has taken place.\n>>\n>> What would GIT.export-subst contain? There can be multiple\n>> replacements in a file.\n>\n> GIT.export-subst would only contain a 1 if substitution is turned on.\n> The goal is to have reproduceable archives, not the ability to turn an\n> archive back into a git repository.\n\nOK.\n\n>>> Maybe there should also be an\n>>> GIT.original-name key.\n>>\n>> What would it be used for?\n>\n> In case an export substition changes the file name so the implementation\n> can verify that the original file could plausibly have been substituted\n> into the current name.  Also for the case where multiple files\n> substitute into the same name to tell which file git should check\n> equivalency with.\n\nStupid question: Could you please provide an example?  The only \npossibility for name changes that I'm aware of is using --prefix.\n\n>>> An option GIT.export-ignore is not required.  Instead it would be more\n>>> useful to have a special file type G (for git) with the convention that\n>>> the file name .gitattributes means “attributes that apply to this git\n>>> archive.”\n>>\n>> That would be a non-standard extension.  Archivers would extract\n>> these as regular files.  Storing a list of excluded paths (in\n>> GIT.exclude or so) might be a better idea.\n>\n> No, that's not a good idea as pax headers are interpreted as “attributes\n> pertaining to a file.”  A file doesn't have the attribute that other\n> files have been omitted.  Making this a special file type is useful as\n> it allows archivers that don't implement git extensions to recover this\n> information in a useful way (after all, the .gitattributes file took\n> part in creating the archive) and, more importantly, reserves a file\n> type for future git extensions.\n\nWe can interpret our own keywords as we see fit.  Other programs will \nignore them (or at most print a warning).  There are precedents for \nglobal headers pertaining to the whole archive, e.g. SCHILY.archtype of \nstar by Jörg Schilling.\n\nLetting archivers extract meta data as regular files is annoying to \nthose that are not interested in it.  Extended headers themselves (type \ng) are bad enough already in this regard for those stuck with old tar \nversions.\n\n>>> The GIT.path option holds the paths that are being archived. It is a bit\n>>> tricky to get right.  The intent of POSIX pax headers is that each key\n>>> is an attribute that applies to a series of files.  In the case of a\n>>> global header, each key applies until it is overridden with a new\n>>> header or with a local header.  A GIT.path key should only apply to the\n>>> files that correspond to this path operant to git archive.  Thus, a new\n>>> GIT.path should be written frequently.  There should always be at least\n>>> one GIT.path.\n>>\n>> That's for the optional path parameters of git archive, right?  A\n>> list of included paths (GIT.include) would be simpler and should\n>> suffice, no?\n>\n> No.  Again: An attribute in a pax header pertains a file.  It's metadata\n> attached to a file, not metadata attached to the whole archive, even when\n> part of a global header.  Thus each file should have attached what path\n> operand it came from.  A file doesn't have the attribute what other path\n> operands git received, only the path operand that caused the inclusion of\n> that one file is an attribute of the file.\n\nNot an issue; we can make our own rules for our own keywords.\n\n>>> It might be a good idea to be able to control the kind of metadata git\n>>> adds to the archive as to be able to not leak any confidential\n>>> information with git archive.  If you are interested I can try to make a\n>>> specification for these headers.\n>>\n>> Which of the field might be sensitive?\n>\n> The existence of a git-specific pax header is sensitive as it proves\n> that a git archive of the source code exists.  This can be a problem if\n> you want to plausibly deny the possession of other versions of the\n> source code you distribute.  The existence of export-ignore meta data\n> leaks information about what other files are in the repository the\n> archive was created from and can be critical.  The existence of\n> path-operand meta data can show what path structure the repository has\n> which can be sensitive.  Basically the existence of any information\n> besides the information you want to add itself is sensitive.\n\nOK..\n\n>> Users can always go back to the original format.  At least I don't\n>> expect this new format becoming the default too quickly.\n>\n> Sure thing.  If this is going to be implemented, I would add options\n> to choose what / what style of metadata to include.\n\nAlright.  (An environment requiring these options sounds scary, though.)\n\nRené\n"},{"id":"277657","messageId":"20160206145726.GA27001@fuz.su","threadId":"41255","inReplyTo":"56B5F3BF.7080601@web.de","subject":"Re: git archive should use vendor extension in pax header","fromName":"","fromEmail":"fuz@fuz.su","sentAt":"2016-02-06T14:57:26Z","receivedAt":"2016-02-06T14:57:26Z","isPatch":false,"sender":{"key":"fuz@fuz.su","avatar":null},"body":"On Sat, Feb 06, 2016 at 02:23:11PM +0100, René Scharfe wrote:\n> Am 28.01.2016 um 00:45 schrieb fuz@fuz.su:\n> >>There is git get-tar-commit-id, which prints the commit ID if it\n> >>finds a comment entry which looks like a hexadecimal SHA-1 hash.\n> >>It's better than a hex editor at least. :)\n> >\n> >This is incredibly fuzzy and can get wrong for a pleothora of reasons.\n> >I hope you agree though that the situation is suboptimal, git is doing\n> >the equivalent of using a custom file format without an easily\n> >recognizable magic number.\n> \n> It is fuzzy in theory. But which other programs allow writing a\n> comment header?  I'm not aware of any, but I have to admit that I\n> didn't look too hard.\n\nWell, let's say what happens if the Mercurial folks were to implement\nthe same thing? Suddenly there is a conflict. Yes, of course, right now\nthere might be no program that uses the comment field for its own\npurpose but such design decisions tend to be not future proof. There is\na very good reason why file formats typically have magic numbers and\ndon't just rely on people knowing that the file has a certain type and\nthat is the same reason why git should mark its meta data in a unique\nfashion.\n\n> >>But I'm still interested how you got a collection of tar files with\n> >>unknown origin.  Just curious.\n> >\n> >Easy: Just download the (source) distribution archives of a distribution\n> >of choice and try to verify that the tarballs they use to compile their\n> >packages actually come from the project's public git repositories.\n> \n> OK, that's easier than calculating checksums and comparing them with\n> those published by the respective projects, but also less\n> trustworthy.\n\nIf you have a known trusted archive, you could use it directly, no need\nfor cross-verification. The intent is to be able to check if archives\ngenerated by someone from some sources could have plausibly been\ngenerated from these sources.\n\n> >There are other reasons why an easily detectable git hash might be\n> >useful.  For example, file(1) could show that the archive comes from\n> >git.  Other utilities could use this to work around git-specific bugs.\n> >An unpacker could add corresponding meta-data when unpacking the file.\n> \n> file(1) could use the same heuristic as git get-tar-commit-id.\n> Something like this would work (the first line is already shipped\n> with file):\n> \n> \t257\tstring\tustar\\0 POSIX tar archive\n> \t>156\tstring\tg\n> \t>>512\tstring\t52\\ comment=\n> \t>>>523\tregex\t[0-9a-f]{40}\t\\b, git commit %s\n> \n> NB: With Ian Darwin's file you need to use -e tar in order to turn\n> off its internal tar test.\n\nSame issues as above.\n\n> I'm very interested in hearing about any git specific bugs.\n\nI don't know any. Bugs tens to be known only after 1000s of buggy\narchives have been published (just as with some GNU tar bugs). It's\ngreat to have a way to detect that the archive might be affected by\na bug so you know that you need to work around it.\n\n> >>>>>It would be much more useful if git created a\n> >>>>>custom key. As per POSIX suggestions, something like this would be\n> >>>>>appropriate:\n> >>>>>\n> >>>>>     GIT.commit=57ca140635bf157354124e4e4b3c8e1bde2832f1\n> >>>>\n> >>>>This would be included in addition to the comment in order to avoid\n> >>>>breaking existing users, I guess.\n> >>>\n> >>>Good point.  I'm not sure how many user use the comment header at all.\n> >>\n> >>Apart from git get-tar-commit-id I don't know any program for\n> >>extracting pax comments.  And I don't know how widely used that is,\n> >>but I assume there is *someone* out there, extracting commit IDs\n> >>with it.\n> >\n> >Neither do I.  But remember, POSIX explicitly specifies that programs\n> >that parse pax file must ignore pax comments so an unpacker that\n> >interpretes the content of such a comment in any way is in violation of\n> >the pax specification.\n> \n> Almost right: The spec says that *pax* shall ignore comments.  Which\n> is good -- we can use this field to transport anything without pax\n> complaining.\n\nThe intent of the committee is that comments shall be ignored by all\nsoftware that processes tar files. Of course you can put metadata into\nthe comments, but that is just a perversion of the file format. POSIX\nexplicitly states that unknown keys in extended headers are to be\nignored by extractors, so using these should be fine, too. But you are\nright, some research needs to be done as to how different archivers deal\nwith unexpected keys in pax headers.\n\n> >>>Maybe there should also be an\n> >>>GIT.original-name key.\n> >>\n> >>What would it be used for?\n> >\n> >In case an export substition changes the file name so the implementation\n> >can verify that the original file could plausibly have been substituted\n> >into the current name.  Also for the case where multiple files\n> >substitute into the same name to tell which file git should check\n> >equivalency with.\n> \n> Stupid question: Could you please provide an example?  The only\n> possibility for name changes that I'm aware of is using --prefix.\n\nI was under the impression that export substitutions also apply to file\nnames but it seems like I'm wrong about this.\n\n> >>>An option GIT.export-ignore is not required.  Instead it would be more\n> >>>useful to have a special file type G (for git) with the convention that\n> >>>the file name .gitattributes means “attributes that apply to this git\n> >>>archive.”\n> >>\n> >>That would be a non-standard extension.  Archivers would extract\n> >>these as regular files.  Storing a list of excluded paths (in\n> >>GIT.exclude or so) might be a better idea.\n> >\n> >No, that's not a good idea as pax headers are interpreted as “attributes\n> >pertaining to a file.”  A file doesn't have the attribute that other\n> >files have been omitted.  Making this a special file type is useful as\n> >it allows archivers that don't implement git extensions to recover this\n> >information in a useful way (after all, the .gitattributes file took\n> >part in creating the archive) and, more importantly, reserves a file\n> >type for future git extensions.\n> \n> We can interpret our own keywords as we see fit.  Other programs\n> will ignore them (or at most print a warning).  There are precedents\n> for global headers pertaining to the whole archive, e.g.\n> SCHILY.archtype of star by Jörg Schilling.\n\nSince a tar archive is semantically a concatenation of individual file\nrecords, it makes sense that each file has the attribute “this file has\nstar extensions.”\n\nThinking about the problem a bit more and discussion with the\naforementioned Jörg Schilling we came to the conclusion that the best\nway to deal with an “file omitted” attribute is to attach it to the\ndirectory that would normally contain the omitted file.\n\n> Letting archivers extract meta data as regular files is annoying to\n> those that are not interested in it.  Extended headers themselves\n> (type g) are bad enough already in this regard for those stuck with\n> old tar versions.\n\nI think we can safely assume that systems support pax headers 15 years\nafter they have been standardized. I was actually unable to find a\nnon-historical version of a serious archiver that claims to support tar\narchives but doesn't support pax headers.\n\n> >>>The GIT.path option holds the paths that are being archived. It is a bit\n> >>>tricky to get right.  The intent of POSIX pax headers is that each key\n> >>>is an attribute that applies to a series of files.  In the case of a\n> >>>global header, each key applies until it is overridden with a new\n> >>>header or with a local header.  A GIT.path key should only apply to the\n> >>>files that correspond to this path operant to git archive.  Thus, a new\n> >>>GIT.path should be written frequently.  There should always be at least\n> >>>one GIT.path.\n> >>\n> >>That's for the optional path parameters of git archive, right?  A\n> >>list of included paths (GIT.include) would be simpler and should\n> >>suffice, no?\n> >\n> >No.  Again: An attribute in a pax header pertains a file.  It's metadata\n> >attached to a file, not metadata attached to the whole archive, even when\n> >part of a global header.  Thus each file should have attached what path\n> >operand it came from.  A file doesn't have the attribute what other path\n> >operands git received, only the path operand that caused the inclusion of\n> >that one file is an attribute of the file.\n> \n> Not an issue; we can make our own rules for our own keywords.\n\nWell, yes, but they should still stick to the semantic concept POSIX\nimposes for extended headers: headers pertain to files and the only\ndifference between a g header and an x header is that the former applies\nuntil it is revoked by a new g header or overridden by an x header.\nNot sticking to this concept can lead to weird problems with programs\nthat modify tar archives (like GNU-tar) and is not future proof. Better\nstick to the standard.\n\n> >>Users can always go back to the original format.  At least I don't\n> >>expect this new format becoming the default too quickly.\n> >\n> >Sure thing.  If this is going to be implemented, I would add options\n> >to choose what / what style of metadata to include.\n> \n> Alright.  (An environment requiring these options sounds scary, though.)\n\nAlways remember: https://xkcd.com/1172/\n\n> René\n\nYours,\nRobert Clausecker\n\n-- \n()  ascii ribbon campaign - for an 8-bit clean world \n/\\  - against html email  - against proprietary attachments\n"},{"id":"278188","messageId":"56C23444.5000405@web.de","threadId":"41255","inReplyTo":"20160206145726.GA27001@fuz.su","subject":"Re: git archive should use vendor extension in pax header","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2016-02-15T20:25:40Z","receivedAt":"2016-02-15T20:25:40Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 06.02.2016 um 15:57 schrieb fuz@fuz.su:\n> On Sat, Feb 06, 2016 at 02:23:11PM +0100, René Scharfe wrote:\n>> Am 28.01.2016 um 00:45 schrieb fuz@fuz.su:\n>>>> There is git get-tar-commit-id, which prints the commit ID if it\n>>>> finds a comment entry which looks like a hexadecimal SHA-1 hash.\n>>>> It's better than a hex editor at least. :)\n>>>\n>>> This is incredibly fuzzy and can get wrong for a pleothora of reasons.\n>>> I hope you agree though that the situation is suboptimal, git is doing\n>>> the equivalent of using a custom file format without an easily\n>>> recognizable magic number.\n>>\n>> It is fuzzy in theory. But which other programs allow writing a\n>> comment header?  I'm not aware of any, but I have to admit that I\n>> didn't look too hard.\n>\n> Well, let's say what happens if the Mercurial folks were to implement\n> the same thing? Suddenly there is a conflict. Yes, of course, right now\n> there might be no program that uses the comment field for its own\n> purpose but such design decisions tend to be not future proof. There is\n> a very good reason why file formats typically have magic numbers and\n> don't just rely on people knowing that the file has a certain type and\n> that is the same reason why git should mark its meta data in a unique\n> fashion.\n\nChances are good that Mercurial would do it in a way that doesn't \nconflict with git's tar comments.  I get your point, though, and agree \nthat it's not ideal.  However, so far it's just a potential problem.\n\n>>>> But I'm still interested how you got a collection of tar files with\n>>>> unknown origin.  Just curious.\n>>>\n>>> Easy: Just download the (source) distribution archives of a distribution\n>>> of choice and try to verify that the tarballs they use to compile their\n>>> packages actually come from the project's public git repositories.\n>>\n>> OK, that's easier than calculating checksums and comparing them with\n>> those published by the respective projects, but also less\n>> trustworthy.\n>\n> If you have a known trusted archive, you could use it directly, no need\n> for cross-verification. The intent is to be able to check if archives\n> generated by someone from some sources could have plausibly been\n> generated from these sources.\n\nIt's probably not too important, but I think I still don't fully \nunderstand.  So you have a tar file of unknown origin.  You hand it to \ngit get-tar-commit-id or a similar tool and get back \na08595f76159b09d57553e37a5123f1091bb13e7.  You can google this string \nand find out it's the commit ID for git v2.7.1.\n\nYour tar file could have been modified in various ways, though, e.g. \nwith tar u or tar --delete.  So you try to find a download site for the \nsoftware that includes file hashes for archives of this release, like in\nhttps://www.kernel.org/pub/software/scm/git/sha256sums.asc.\n\nIf the published hash and a hash of your file match then you can be \nreasonably sure the files are the same.  If they don't then it could be \ndue to variations added by the compressor.  You can download the \nauthoritative archive and compare it with yours.\n\nIs that how it goes?\n\n>> I'm very interested in hearing about any git specific bugs.\n>\n> I don't know any. Bugs tens to be known only after 1000s of buggy\n> archives have been published (just as with some GNU tar bugs). It's\n> great to have a way to detect that the archive might be affected by\n> a bug so you know that you need to work around it.\n\nThat requires a field containing the git version which was used to \ncreate the archive, no?\n\n> Thinking about the problem a bit more and discussion with the\n> aforementioned Jörg Schilling we came to the conclusion that the best\n> way to deal with an “file omitted” attribute is to attach it to the\n> directory that would normally contain the omitted file.\n\nSounds sensible, but the ordering can be a bit tricky.  If d/a is \nincluded and d/b is not then it would be easy to write d/, d/a and the \nextended header that says that d/b is excluded, in that order.  Writing \nthe extended header first is a bit harder and I'm not sure if it's \nneeded.  And it gets tricky if more than one entry is excluded per \ndirectory. (Just thinking out loud here.)\n\n>> Letting archivers extract meta data as regular files is annoying to\n>> those that are not interested in it.  Extended headers themselves\n>> (type g) are bad enough already in this regard for those stuck with\n>> old tar versions.\n>\n> I think we can safely assume that systems support pax headers 15 years\n> after they have been standardized. I was actually unable to find a\n> non-historical version of a serious archiver that claims to support tar\n> archives but doesn't support pax headers.\n\nWell, that depends on your definition of \"serious\".  Plan 9's tar \nperhaps doesn't fit it, but what about 7-Zip (http://www.7-zip.org/)?\n\nAnd there is no way (or did I overlook it?) to modify or display the \ncomment extended header using GNU tar.  That's actually surprising to \nme: I'd think the ability to add a human-readable description to a \nbackup on tape is quite important.  (But I didn't touch an actual tape \nfor quite a while, and I never used tar directly with them.)\n\n>>>>> The GIT.path option holds the paths that are being archived. It is a bit\n>>>>> tricky to get right.  The intent of POSIX pax headers is that each key\n>>>>> is an attribute that applies to a series of files.  In the case of a\n>>>>> global header, each key applies until it is overridden with a new\n>>>>> header or with a local header.  A GIT.path key should only apply to the\n>>>>> files that correspond to this path operant to git archive.  Thus, a new\n>>>>> GIT.path should be written frequently.  There should always be at least\n>>>>> one GIT.path.\n>>>>\n>>>> That's for the optional path parameters of git archive, right?  A\n>>>> list of included paths (GIT.include) would be simpler and should\n>>>> suffice, no?\n>>>\n>>> No.  Again: An attribute in a pax header pertains a file.  It's metadata\n>>> attached to a file, not metadata attached to the whole archive, even when\n>>> part of a global header.  Thus each file should have attached what path\n>>> operand it came from.  A file doesn't have the attribute what other path\n>>> operands git received, only the path operand that caused the inclusion of\n>>> that one file is an attribute of the file.\n>>\n>> Not an issue; we can make our own rules for our own keywords.\n>\n> Well, yes, but they should still stick to the semantic concept POSIX\n> imposes for extended headers: headers pertain to files and the only\n> difference between a g header and an x header is that the former applies\n> until it is revoked by a new g header or overridden by an x header.\n> Not sticking to this concept can lead to weird problems with programs\n> that modify tar archives (like GNU-tar) and is not future proof. Better\n> stick to the standard.\n\nIt's easy enough, I think: For each archive entry check if it is \nexplicity mentioned in the list of paths to archive and write an \nextended header with GIT.path before proceeding as usual, no?  And for \nthe common case without path specification (meaning all files are \nincluded) no such header would be needed.\n\nRené\n"}]}