{"thread":{"id":"53067","subject":"[PATCH] import-tars: ignore the global PAX header","startedAt":"2020-03-23T13:08:51Z","lastAt":"2020-03-25T18:43:29Z","messageCount":13,"participants":["Johannes Schindelin via GitGitGadget","René Scharfe","Junio C Hamano","Johannes Schindelin","brian m. carlson"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"393762","messageId":"pull.577.git.1584968924555.gitgitgadget@gmail.com","threadId":"53067","inReplyTo":null,"subject":"[PATCH] import-tars: ignore the global PAX header","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-23T13:08:44Z","receivedAt":"2020-03-23T13:08:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nGit's own `git archive` inserts that header, but it often gets into the\nway of `import-tars.perl` e.g. when a prefix was specified (for example\nvia `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\nGitHub releases): this prefix _should_ be stripped.\n\nLet's just skip it.\n\nSigned-off-by: Johannes Schindelin \n---\n Ignore the global PAX header in import-tars.perl\n \n This problem came up in Pacman-related work, where PKGBUILD definitions\n would reference the tarballs downloaded from GitHub, and patches would\n be applied on top. To work on those patches efficiently (e.g. when an\n upgrade to a new version of the project no longer lets those patches\n apply), I need to be able to import those tarballs into playground\n worktrees and work on them. I like to use \n contrib/fast-import/import-tars.perl for that purpose, but it really\n needs to strip the prefix, otherwise it is too tedious to work with it.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-577%2Fdscho%2Fimport-tars-skip-pax-header-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-577/dscho/import-tars-skip-pax-header-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/577\n\n contrib/fast-import/import-tars.perl | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\nindex e800d9f5c9c..d50ce26d5d9 100755\n--- a/contrib/fast-import/import-tars.perl\n+++ b/contrib/fast-import/import-tars.perl\n@@ -139,6 +139,8 @@\n \t\t\tprint FI \"\\n\";\n \t\t}\n \n+\t\tnext if ($typeflag eq 'g'); # ignore global header\n+\n \t\tmy $path;\n \t\tif ($prefix) {\n \t\t\t$path = \"$prefix/$name\";\n\nbase-commit: b4374e96c84ed9394fed363973eb540da308ed4f\n-- \ngitgitgadget\n"},{"id":"393784","messageId":"fce519db-5ad4-270f-abcf-0e26549486cb@web.de","threadId":"53067","inReplyTo":"pull.577.git.1584968924555.gitgitgadget@gmail.com","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-03-23T17:09:39Z","receivedAt":"2020-03-23T17:10:02Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.03.20 um 14:08 schrieb Johannes Schindelin via GitGitGadget:\n> From: Johannes Schindelin \n>\n> Git's own `git archive` inserts that header, but it often gets into the\n> way of `import-tars.perl` e.g. when a prefix was specified (for example\n> via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n> GitHub releases): this prefix _should_ be stripped.\n>\n> Let's just skip it.\n\ngit archive uses a global pax header to pass the ID of the archived\ncommit as a comment, and for mtime values after 2242-03-16. Ignoring it\nin a simple importer seems reasonable for now, but I don't understand\nhow this relates to prefixes. Is it because the header is treated as a\nregular file with the full path \"pax_global_header\" (independently from\nany prefix for actual files) and can thus be placed outside the expected\ndestination directory?\n\nThanks,\nRené\n"},{"id":"393787","messageId":"xmqqh7yf55q7.fsf@gitster.c.googlers.com","threadId":"53067","inReplyTo":"fce519db-5ad4-270f-abcf-0e26549486cb@web.de","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-23T17:41:20Z","receivedAt":"2020-03-23T17:41:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> Am 23.03.20 um 14:08 schrieb Johannes Schindelin via GitGitGadget:\n>> From: Johannes Schindelin \n>>\n>> Git's own `git archive` inserts that header, but it often gets into the\n>> way of `import-tars.perl` e.g. when a prefix was specified (for example\n>> via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n>> GitHub releases): this prefix _should_ be stripped.\n>>\n>> Let's just skip it.\n>\n> git archive uses a global pax header to pass the ID of the archived\n> commit as a comment, and for mtime values after 2242-03-16. Ignoring it\n> in a simple importer seems reasonable for now, but I don't understand\n> how this relates to prefixes. Is it because the header is treated as a\n> regular file with the full path \"pax_global_header\" (independently from\n> any prefix for actual files) and can thus be placed outside the expected\n> destination directory?\n\nThanks for asking the question, as I was also curious if we are\nthrowing away too much (perhaps \"prefix is given as a global pax\nheader, and ignoring all global pax headers is the most expedite\nway\" was the reason the patch was written that way?). I agree with\nyou that for the purpose of simple-minded importer, it probably is\nacceptable to take such a short-cut, but it would help future\ndevelopers if we clearly documented that it is a short-cut that\nthrows too much. That would welcome their effort to enhance the\nimporter, if they find it more useful to keep some other information\nfound in global headers, without breaking the intent of this change.\n\nHaving said all that, even before \"git archive\" existed, release\ntarballs by many projects had leading prefix so that a tarball\nextract would be made inside a versioned directory. To truly help\nusers of the importer, doesn't the logic to allow the user to say\n\"please strip one leading level of directory from all the tarballs I\nfeed you, as I know they are versioned directories\" belong to the\ncommand line option of the importer?\n\n"},{"id":"393818","messageId":"nycvar.QRO.7.76.6.2003232158160.46@tvgsbejvaqbjf.bet","threadId":"53067","inReplyTo":"fce519db-5ad4-270f-abcf-0e26549486cb@web.de","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-03-23T21:05:28Z","receivedAt":"2020-03-23T21:05:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\nOn Mon, 23 Mar 2020, René Scharfe wrote:\n\n> Am 23.03.20 um 14:08 schrieb Johannes Schindelin via GitGitGadget:\n> > From: Johannes Schindelin \n> >\n> > Git's own `git archive` inserts that header, but it often gets into the\n> > way of `import-tars.perl` e.g. when a prefix was specified (for example\n> > via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n> > GitHub releases): this prefix _should_ be stripped.\n> >\n> > Let's just skip it.\n>\n> git archive uses a global pax header to pass the ID of the archived\n> commit as a comment, and for mtime values after 2242-03-16. Ignoring it\n> in a simple importer seems reasonable for now, but I don't understand\n> how this relates to prefixes.\n\nThe tar importer in `contrib/fast-import/import-tars.perl` has a very\nconvenient feature that I took for public knowledge: if _all_ paths stored\nin the imported `.tar` start with a common prefix, e.g.\n`my-project-1.0.0/` as indicated in the commit message (or `git-2.26.0/`\nin the tar at https://github.com/git/git/archive/v2.26.0.tar.gz), then\nthis prefix is stripped.\n\nThis feature makes a ton of sense because it is relatively common to\nimport two or more revisions of the same project into Git, and obviously\nwe don't want all files to live in a tree whose name changes from revision\nto revision.\n\nNow, the problem with that feature is that it breaks down if there is a\n`pax_global_header` \"file\" located outside of said prefix.\n\n> Is it because the header is treated as a regular file with the full path\n> \"pax_global_header\" (independently from any prefix for actual files) and\n> can thus be placed outside the expected destination directory?\n\nExactly.\n\nThanks,\nDscho\n\n>\n> Thanks,\n> René\n>\n>\n"},{"id":"393819","messageId":"nycvar.QRO.7.76.6.2003232205580.46@tvgsbejvaqbjf.bet","threadId":"53067","inReplyTo":"xmqqh7yf55q7.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-03-23T21:08:36Z","receivedAt":"2020-03-23T21:08:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 23 Mar 2020, Junio C Hamano wrote:\n\n> René Scharfe writes:\n>\n> > Am 23.03.20 um 14:08 schrieb Johannes Schindelin via GitGitGadget:\n> >> From: Johannes Schindelin \n> >>\n> >> Git's own `git archive` inserts that header, but it often gets into the\n> >> way of `import-tars.perl` e.g. when a prefix was specified (for example\n> >> via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n> >> GitHub releases): this prefix _should_ be stripped.\n> >>\n> >> Let's just skip it.\n> >\n> > git archive uses a global pax header to pass the ID of the archived\n> > commit as a comment, and for mtime values after 2242-03-16. Ignoring it\n> > in a simple importer seems reasonable for now, but I don't understand\n> > how this relates to prefixes. Is it because the header is treated as a\n> > regular file with the full path \"pax_global_header\" (independently from\n> > any prefix for actual files) and can thus be placed outside the expected\n> > destination directory?\n>\n> Thanks for asking the question, as I was also curious if we are\n> throwing away too much (perhaps \"prefix is given as a global pax\n> header, and ignoring all global pax headers is the most expedite\n> way\" was the reason the patch was written that way?). I agree with\n> you that for the purpose of simple-minded importer, it probably is\n> acceptable to take such a short-cut, but it would help future\n> developers if we clearly documented that it is a short-cut that\n> throws too much. That would welcome their effort to enhance the\n> importer, if they find it more useful to keep some other information\n> found in global headers, without breaking the intent of this change.\n\nI don't think that we're throwing away anything because the PAX header is\nintended to be a _header_, not a _file_, yet\n`contrib/fast-import/import-tars.perl` currently treats PAX headers that\nway.\n\n> Having said all that, even before \"git archive\" existed, release\n> tarballs by many projects had leading prefix so that a tarball\n> extract would be made inside a versioned directory. To truly help\n> users of the importer, doesn't the logic to allow the user to say\n> \"please strip one leading level of directory from all the tarballs I\n> feed you, as I know they are versioned directories\" belong to the\n> command line option of the importer?\n\nI guess nobody needed an explicit way to strip path prefixes yet, since\nthe implicit way works so well.\n\nCiao,\nDscho\n"},{"id":"393832","messageId":"xmqqtv2e3fmu.fsf@gitster.c.googlers.com","threadId":"53067","inReplyTo":"nycvar.QRO.7.76.6.2003232205580.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-23T21:50:17Z","receivedAt":"2020-03-23T21:50:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> throws too much. That would welcome their effort to enhance the\n>> importer, if they find it more useful to keep some other information\n>> found in global headers, without breaking the intent of this change.\n>\n> I don't think that we're throwing away anything because the PAX header is\n> intended to be a _header_, not a _file_, yet\n> `contrib/fast-import/import-tars.perl` currently treats PAX headers that\n> way.\n\nWhat I meant (and wrote) was information contained within the\nheader. You could store such metadata (e.g. this tree came from\nthis commit from the upstream project) in the commit object while\nimporting, for example.\n\nAs I wrote, I do not think we need to implement such feature right\nnow. I am just saying that we should make sure we are not\nunintendely discouraging future developers from doing so by giving\nan impression that we are claiming \"a pax header is intended to be a\nheader and has no interesting information---never look at it\". If\nwe said \"We discard the headers because it is the most expedient way\nand we currently have no use for them\", that would make it clear\nthat it is OK for them (the future ourselves developers) to extend\nthe code not to lose the information, as long as they keep ignoring\nthe prefix thing alone, if they want to follow the course set by\nthis change.\n\n>> Having said all that, even before \"git archive\" existed, release\n>> tarballs by many projects had leading prefix so that a tarball\n>> extract would be made inside a versioned directory. To truly help\n>> users of the importer, doesn't the logic to allow the user to say\n>> \"please strip one leading level of directory from all the tarballs I\n>> feed you, as I know they are versioned directories\" belong to the\n>> command line option of the importer?\n>\n> I guess nobody needed an explicit way to strip path prefixes yet, since\n> the implicit way works so well.\n\nNow you confused me. If implicit way already works well, why are we\nadding this patch to make it implicitly ignore?\n"},{"id":"393833","messageId":"xmqqpnd23fhg.fsf@gitster.c.googlers.com","threadId":"53067","inReplyTo":"nycvar.QRO.7.76.6.2003232158160.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-23T21:53:31Z","receivedAt":"2020-03-23T21:53:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> The tar importer in `contrib/fast-import/import-tars.perl` has a very\n> convenient feature that I took for public knowledge: if _all_ paths stored\n> in the imported `.tar` start with a common prefix, e.g.\n> `my-project-1.0.0/` as indicated in the commit message (or `git-2.26.0/`\n> in the tar at https://github.com/git/git/archive/v2.26.0.tar.gz), then\n> this prefix is stripped.\n\nAhh, scratch what I said in my recent response. No, the above is\nnot public knowledge, and that was missing from the log message that\nconfused me.\n\nWith that spelled out, yes, it makes sense not to allow the phony\n\"file\" at the top level created out of the pax headers.\n\nSo, I think with a bit more explanation in the log, the patch would\nbe OK. We do not want _any_ header.\n\nThanks.\n"},{"id":"393838","messageId":"20200323232520.GE6499@camp.crustytoothpaste.net","threadId":"53067","inReplyTo":"pull.577.git.1584968924555.gitgitgadget@gmail.com","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-03-23T23:25:20Z","receivedAt":"2020-03-23T23:25:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-03-23 at 13:08:44, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> Git's own `git archive` inserts that header, but it often gets into the\n> way of `import-tars.perl` e.g. when a prefix was specified (for example\n> via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n> GitHub releases): this prefix _should_ be stripped.\n> \n> Let's just skip it.\n> \n> Signed-off-by: Johannes Schindelin \n> ---\n> Ignore the global PAX header in import-tars.perl\n> \n> This problem came up in Pacman-related work, where PKGBUILD definitions\n> would reference the tarballs downloaded from GitHub, and patches would\n> be applied on top. To work on those patches efficiently (e.g. when an\n> upgrade to a new version of the project no longer lets those patches\n> apply), I need to be able to import those tarballs into playground\n> worktrees and work on them. I like to use \n> contrib/fast-import/import-tars.perl for that purpose, but it really\n> needs to strip the prefix, otherwise it is too tedious to work with it.\n> \n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-577%2Fdscho%2Fimport-tars-skip-pax-header-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-577/dscho/import-tars-skip-pax-header-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/577\n> \n> contrib/fast-import/import-tars.perl | 2 ++\n> 1 file changed, 2 insertions(+)\n> \n> diff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\n> index e800d9f5c9c..d50ce26d5d9 100755\n> --- a/contrib/fast-import/import-tars.perl\n> +++ b/contrib/fast-import/import-tars.perl\n> @@ -139,6 +139,8 @@\n> \t\t\tprint FI \"\\n\";\n> \t\t}\n> \n> +\t\tnext if ($typeflag eq 'g'); # ignore global header\n> +\n\nIn general, it isn't safe to do this. A pax global header contains\nattributes that may live in a normal extended header at a lower\npriority. So it is valid, for example, to write an mtime field in the\nglobal header that applies to the entire archive and overrides the\nustar header block (and is overridden by a normal extended header).\n\nI think we need a different solution for this case.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"393874","messageId":"nycvar.QRO.7.76.6.2003241433380.46@tvgsbejvaqbjf.bet","threadId":"53067","inReplyTo":"20200323232520.GE6499@camp.crustytoothpaste.net","subject":"Re: [PATCH] import-tars: ignore the global PAX header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-03-24T13:35:55Z","receivedAt":"2020-03-24T13:36:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi brian,\n\nOn Mon, 23 Mar 2020, brian m. carlson wrote:\n\n> On 2020-03-23 at 13:08:44, Johannes Schindelin via GitGitGadget wrote:\n> > From: Johannes Schindelin \n> >\n> > Git's own `git archive` inserts that header, but it often gets into the\n> > way of `import-tars.perl` e.g. when a prefix was specified (for example\n> > via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n> > GitHub releases): this prefix _should_ be stripped.\n> >\n> > Let's just skip it.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > Ignore the global PAX header in import-tars.perl\n> >\n> > This problem came up in Pacman-related work, where PKGBUILD definitions\n> > would reference the tarballs downloaded from GitHub, and patches would\n> > be applied on top. To work on those patches efficiently (e.g. when an\n> > upgrade to a new version of the project no longer lets those patches\n> > apply), I need to be able to import those tarballs into playground\n> > worktrees and work on them. I like to use\n> > contrib/fast-import/import-tars.perl for that purpose, but it really\n> > needs to strip the prefix, otherwise it is too tedious to work with it.\n> >\n> > Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-577%2Fdscho%2Fimport-tars-skip-pax-header-v1\n> > Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-577/dscho/import-tars-skip-pax-header-v1\n> > Pull-Request: https://github.com/gitgitgadget/git/pull/577\n> >\n> > contrib/fast-import/import-tars.perl | 2 ++\n> > 1 file changed, 2 insertions(+)\n> >\n> > diff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\n> > index e800d9f5c9c..d50ce26d5d9 100755\n> > --- a/contrib/fast-import/import-tars.perl\n> > +++ b/contrib/fast-import/import-tars.perl\n> > @@ -139,6 +139,8 @@\n> > \t\t\tprint FI \"\\n\";\n> > \t\t}\n> >\n> > +\t\tnext if ($typeflag eq 'g'); # ignore global header\n> > +\n>\n> In general, it isn't safe to do this. A pax global header contains\n> attributes that may live in a normal extended header at a lower\n> priority. So it is valid, for example, to write an mtime field in the\n> global header that applies to the entire archive and overrides the\n> ustar header block (and is overridden by a normal extended header).\n>\n> I think we need a different solution for this case.\n\nI agree that we need a different solution for this case.\n\nAt the same time, I would like to point out that I am trying to address a\n_different_ problem than \"we're not using the information contained within\nthe PAX global header\": I want to prevent that header from polluting the\ntop-level tree.\n\nFor what it's worth, my patch does not prevent future patches from using\nthe information contained within the global header :-)\n\nThanks,\nDscho\n"},{"id":"393889","messageId":"pull.577.v2.git.1585078545448.gitgitgadget@gmail.com","threadId":"53067","inReplyTo":"pull.577.git.1584968924555.gitgitgadget@gmail.com","subject":"[PATCH v2] import-tars: ignore the global PAX header","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-24T19:35:45Z","receivedAt":"2020-03-24T19:35:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe tar importer in `contrib/fast-import/import-tars.perl` has a very\nconvenient feature: if _all_ paths stored in the imported `.tar` start\nwith a common prefix, e.g. `git-2.26.0/` in the tar at\nhttps://github.com/git/git/archive/v2.26.0.tar.gz, then this prefix is\nstripped.\n\nThis feature makes a ton of sense because it is relatively common to\nimport two or more revisions of the same project into Git, and obviously\nwe don't want all files to live in a tree whose name changes from\nrevision to revision.\n\nNow, the problem with that feature is that it breaks down if there is a\n`pax_global_header` \"file\" located outside of said prefix, at the top of\nthe tree. This is the case for `.tar` files generated by Git's very own\n`git archive` command: it inserts that header, and `git archive` allows\nspecifying a common prefix (that the header does _not_ share with the\nother files contained in the archive) via `--prefix=my-project-1.0.0/`.\n\nLet's just skip any global header when importing `.tar` files into Git.\n\nNote: this global header might contain useful information. For example,\nin the output of `git archive`, it lists the original commit, which _is_\nuseful information. A future improvement to the `import-tars.perl`\nscript might be to include that information in the commit message, or do\nother things with the information (e.g. use `mtime` information\ncontained in the global header as date of the commit). This patch does\nnot prevent any future patch from making that happen, it only prevents\nthe header from being treated as if it was a regular file.\n\nSigned-off-by: Johannes Schindelin \n---\n Ignore the global PAX header in import-tars.perl\n \n This problem came up in Pacman-related work, where PKGBUILD definitions\n would reference the tarballs downloaded from GitHub, and patches would\n be applied on top. To work on those patches efficiently (e.g. when an\n upgrade to a new version of the project no longer lets those patches\n apply), I need to be able to import those tarballs into playground\n worktrees and work on them. I like to use \n contrib/fast-import/import-tars.perl for that purpose, but it really\n needs to strip the prefix, otherwise it is too tedious to work with it.\n \n Changes since v1:\n \n * Mentioned the implicit prefix-stripping feature of import-tars.perl \n in the commit message; Without that context, it is really hard to\n understand the motivation for this patch.\n * Clarified in the commit message that this patch does not prevent any\n future patches that would use the information contained in the global\n header.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-577%2Fdscho%2Fimport-tars-skip-pax-header-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-577/dscho/import-tars-skip-pax-header-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/577\n\nRange-diff vs v1:\n\n 1: 718bde8f4a7 ! 1: 842dabe6128 import-tars: ignore the global PAX header\n @@ -2,12 +2,34 @@\n \n import-tars: ignore the global PAX header\n \n - Git's own `git archive` inserts that header, but it often gets into the\n - way of `import-tars.perl` e.g. when a prefix was specified (for example\n - via `--prefix=my-project-1.0.0/`, or when downloading a `.tar.gz` from\n - GitHub releases): this prefix _should_ be stripped.\n + The tar importer in `contrib/fast-import/import-tars.perl` has a very\n + convenient feature: if _all_ paths stored in the imported `.tar` start\n + with a common prefix, e.g. `git-2.26.0/` in the tar at\n + https://github.com/git/git/archive/v2.26.0.tar.gz, then this prefix is\n + stripped.\n \n - Let's just skip it.\n + This feature makes a ton of sense because it is relatively common to\n + import two or more revisions of the same project into Git, and obviously\n + we don't want all files to live in a tree whose name changes from\n + revision to revision.\n +\n + Now, the problem with that feature is that it breaks down if there is a\n + `pax_global_header` \"file\" located outside of said prefix, at the top of\n + the tree. This is the case for `.tar` files generated by Git's very own\n + `git archive` command: it inserts that header, and `git archive` allows\n + specifying a common prefix (that the header does _not_ share with the\n + other files contained in the archive) via `--prefix=my-project-1.0.0/`.\n +\n + Let's just skip any global header when importing `.tar` files into Git.\n +\n + Note: this global header might contain useful information. For example,\n + in the output of `git archive`, it lists the original commit, which _is_\n + useful information. A future improvement to the `import-tars.perl`\n + script might be to include that information in the commit message, or do\n + other things with the information (e.g. use `mtime` information\n + contained in the global header as date of the commit). This patch does\n + not prevent any future patch from making that happen, it only prevents\n + the header from being treated as if it was a regular file.\n \n Signed-off-by: Johannes Schindelin \n \n\n\n contrib/fast-import/import-tars.perl | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\nindex e800d9f5c9c..d50ce26d5d9 100755\n--- a/contrib/fast-import/import-tars.perl\n+++ b/contrib/fast-import/import-tars.perl\n@@ -139,6 +139,8 @@\n \t\t\tprint FI \"\\n\";\n \t\t}\n \n+\t\tnext if ($typeflag eq 'g'); # ignore global header\n+\n \t\tmy $path;\n \t\tif ($prefix) {\n \t\t\t$path = \"$prefix/$name\";\n\nbase-commit: b4374e96c84ed9394fed363973eb540da308ed4f\n-- \ngitgitgadget\n"},{"id":"393895","messageId":"xmqqd0911n3b.fsf@gitster.c.googlers.com","threadId":"53067","inReplyTo":"pull.577.v2.git.1585078545448.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] import-tars: ignore the global PAX header","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-24T21:04:24Z","receivedAt":"2020-03-24T21:04:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> + Note: this global header might contain useful information. For example,\n> + in the output of `git archive`, it lists the original commit, which _is_\n> + useful information. A future improvement to the `import-tars.perl`\n> + script might be to include that information in the commit message, or do\n> + other things with the information (e.g. use `mtime` information\n> + contained in the global header as date of the commit). This patch does\n> + not prevent any future patch from making that happen, it only prevents\n> + the header from being treated as if it was a regular file.\n> \n> Signed-off-by: Johannes Schindelin \n> \n>\n>\n> contrib/fast-import/import-tars.perl | 2 ++\n> 1 file changed, 2 insertions(+)\n>\n> diff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\n> index e800d9f5c9c..d50ce26d5d9 100755\n> --- a/contrib/fast-import/import-tars.perl\n> +++ b/contrib/fast-import/import-tars.perl\n> @@ -139,6 +139,8 @@\n> \t\t\tprint FI \"\\n\";\n> \t\t}\n> \n> +\t\tnext if ($typeflag eq 'g'); # ignore global header\n> +\n\nYeah, it is more like \"don't create a phony file out of global\nheader\" which is exactly the point of this fix, rather than \"ignore\nglobal header\", and if the contents of the header is used for any\nother purpose (e.g. metadata that will be added to the log message\nor mode bits that is forced on all files), that can be handled\nbefore this line. And the current code structure is already\nprepared for it: all that remains in the block after this point is\nto create a file at $path and store its contents.\n\nSo, this makes sense. I suspect that with an update to the comment\nin the direction, there probably is no need for the huge \"Note\" in\nthe log message.\n\nThanks.\n\n> \t\tmy $path;\n> \t\tif ($prefix) {\n> \t\t\t$path = \"$prefix/$name\";\n>\n> base-commit: b4374e96c84ed9394fed363973eb540da308ed4f\n"},{"id":"393989","messageId":"nycvar.QRO.7.76.6.2003251859070.46@tvgsbejvaqbjf.bet","threadId":"53067","inReplyTo":"xmqqd0911n3b.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2] import-tars: ignore the global PAX header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-03-25T17:59:14Z","receivedAt":"2020-03-25T18:00:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 24 Mar 2020, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n>\n> > + Note: this global header might contain useful information. For example,\n> > + in the output of `git archive`, it lists the original commit, which _is_\n> > + useful information. A future improvement to the `import-tars.perl`\n> > + script might be to include that information in the commit message, or do\n> > + other things with the information (e.g. use `mtime` information\n> > + contained in the global header as date of the commit). This patch does\n> > + not prevent any future patch from making that happen, it only prevents\n> > + the header from being treated as if it was a regular file.\n> >\n> > Signed-off-by: Johannes Schindelin \n> >\n> >\n> >\n> > contrib/fast-import/import-tars.perl | 2 ++\n> > 1 file changed, 2 insertions(+)\n> >\n> > diff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\n> > index e800d9f5c9c..d50ce26d5d9 100755\n> > --- a/contrib/fast-import/import-tars.perl\n> > +++ b/contrib/fast-import/import-tars.perl\n> > @@ -139,6 +139,8 @@\n> > \t\t\tprint FI \"\\n\";\n> > \t\t}\n> >\n> > +\t\tnext if ($typeflag eq 'g'); # ignore global header\n> > +\n>\n> Yeah, it is more like \"don't create a phony file out of global\n> header\" which is exactly the point of this fix, rather than \"ignore\n> global header\", and if the contents of the header is used for any\n> other purpose (e.g. metadata that will be added to the log message\n> or mode bits that is forced on all files), that can be handled\n> before this line. And the current code structure is already\n> prepared for it: all that remains in the block after this point is\n> to create a file at $path and store its contents.\n>\n> So, this makes sense. I suspect that with an update to the comment\n> in the direction, there probably is no need for the huge \"Note\" in\n> the log message.\n\nToo many of such in-code comments added by me became stale. I'd rather\nkeep this in the commit message where it explains the reasoning for the\ncurrent change.\n\nThanks,\nDscho\n\n>\n> Thanks.\n>\n> > \t\tmy $path;\n> > \t\tif ($prefix) {\n> > \t\t\t$path = \"$prefix/$name\";\n> >\n> > base-commit: b4374e96c84ed9394fed363973eb540da308ed4f\n>\n"},{"id":"393993","messageId":"xmqqlfnowa0o.fsf@gitster.c.googlers.com","threadId":"53067","inReplyTo":"nycvar.QRO.7.76.6.2003251859070.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2] import-tars: ignore the global PAX header","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-25T18:43:19Z","receivedAt":"2020-03-25T18:43:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Too many of such in-code comments added by me became stale. I'd rather\n> keep this in the commit message where it explains the reasoning for the\n> current change.\n\nI was updating this week's \"What's cooking\" report, and described\nthis one like so:\n\n * js/import-tars-do-not-make-phony-files-from-pax-headers (2020-03-24) 1 commit\n - import-tars: ignore the global PAX header\n\n The import-tars importer (in contrib/fast-import/) used to create\n phony files at the top-level of the repository when the archive\n contains global PAX headers, which made its own logic to detect and\n omit the common leading directory ineffective, which has been\n corrected.\n\nI guess we are ready to mark it as \"Will merge\".\n\nThanks. \n"}]}