{"thread":{"id":"55228","subject":"Round-tripping fast-export/import changes commit hashes","startedAt":"2021-02-27T12:32:45Z","lastAt":"2023-01-13T07:27:24Z","messageCount":21,"participants":["anatoly techtonik","Elijah Newren","Ævar Arnfjörð Bjarmason","Junio C Hamano","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"417943","messageId":"CAPkN8xK7JnhatkdurEb16bC0wb+=Khd=xJ51YQUXmf2H23YCGw@mail.gmail.com","threadId":"55228","inReplyTo":null,"subject":"Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-02-27T12:31:32Z","receivedAt":"2021-02-27T12:32:45Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"Hi.\n\nI can't get the same commit hashes after fast-export and then fast-import of\nthis repository without any edits https://github.com/simons-public/protonfixes\nI have no idea what causes this, and how to prevent it from happening. Are\nthere any workarounds?\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\n  #!/bin/bash\n\n  git clone https://github.com/simons-public/protonfixes.git\n  git -C protonfixes log --format=oneline | tail -n 4\n\n  git init protoimported\n  git -C protonfixes fast-export --all --reencode=no | (cd\nprotoimported && git fast-import)\n  git -C protoimported log --format=oneline | tail -n 4\n\nWhat did you expect to happen? (Expected behavior)\n\n  Expect imported repo to match exported.\n\nWhat happened instead? (Actual behavior)\n\n  All hashes are different, the exported repo diverged on the second commit.\n\nWhat's different between what you expected and what actually happened?\n\n  The log of hashes from initial repo:\n\n    + git -C protonfixes log --format=oneline\n    + tail -n 4\n    1c0cf2c8e742e673dba9fd1a09afd12a25c25571 Update README.md\n    367d61f9b2a799accbdaeed5d64f9be914ca0f7a Updated zip link\n    d3d24b63446c7d06586eaa51764ff0c619113f09 Update README.md\n    7a43ca89ff7a70127ac9ca0f10b6eaaa34f2f69c Initial commit\n\n  The log from imported repo:\n\n    + git -C protoimported log --format=oneline\n    + tail -n 4\n    a27ec5d2e4c562f40e693e0b4149959d2b69bf21 Update README.md\n    e59cf92be79c47984e9f94bfad912e5a29dfa5e0 Updated zip link\n    fb6498f62af783d2e943770f90bc642cf5c9ec9c Update README.md\n    7a43ca89ff7a70127ac9ca0f10b6eaaa34f2f69c Initial commit\n\n[System Info]\ngit version:\ngit version 2.31.0.rc0\ncpu: x86_64\nbuilt from commit: 225365fb5195e804274ab569ac3cc4919451dc7f\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 5.8.0-43-generic #49-Ubuntu SMP Fri Feb 5 03:01:28 UTC 2021 x86_64\ncompiler info: gnuc: 10.2\nlibc info: glibc: 2.32\n$SHELL (typically, interactive shell): /usr/bin/zsh\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n-- \nanatoly t.\n"},{"id":"417945","messageId":"CABPp-BGDB6jj+Et44D6D22KXprB89dNpyS_AAu3E8vOCtVaW1A@mail.gmail.com","threadId":"55228","inReplyTo":"CAPkN8xK7JnhatkdurEb16bC0wb+=Khd=xJ51YQUXmf2H23YCGw@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-27T17:48:56Z","receivedAt":"2021-02-27T17:53:35Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Sat, Feb 27, 2021 at 4:37 AM anatoly techtonik <techtonik@gmail.com> wrote:\n>\n> Hi.\n>\n> I can't get the same commit hashes after fast-export and then fast-import of\n> this repository without any edits https://github.com/simons-public/protonfixes\n> I have no idea what causes this, and how to prevent it from happening. Are\n> there any workarounds?\n\nYour second commit is signed.  Fast-export strips any extended headers\non commits, such as GPG signatures, because there's no way to keep\nthem in general.  In the special case that you aren't making *any*\nchanges to the repository and will import it as-is, you could\ntheoretically keep the signatures, but you don't need fast-export in\nsuch a case so no one ever bothered to implement commit signature\nhandling in fast-export and fast-import.  If you make any changes\nwhatsoever to the commits before the signature (including importing\nthem to a different system), then the signature would be invalid.\n\nYou probably don't want to hear this, but there are no workarounds.\n\nThere are also other things that will prevent a simple fast-export |\nfast-import pipeline from preserving your history as-is besides signed\ncommits (most of these are noted in the \"Inherited Limitations\"\nsection over at\nhttps://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html):\n\n  * any other form of extended header; fast-export only looks for the\nheaders it knows and exports those\n  * grafts and replace objects will just get rewritten (and if they\ncause any cycles, those cycles and anything depending on them are\ndropped)\n  * commits without an author will be given one matching the committer\n(hopefully you don't have these, but if you do...)\n  * tags that are missing a tagger are also a problem (hopefully you\ndon't have these, but if you do...)\n  * annotated or signed tags outside the refs/tags/ namespace will get\nrenamed weirdly\n  * commits by default are re-encoded into UTF-8, though I notice you\ndid pass --reencode=no to handle this\n\nHope that at least explains things for you, even if it doesn't give\nyou a workaround or a solution.\n"},{"id":"417963","messageId":"CAPkN8xK9__74a3aEFsevfdW_hQ-vzWE+c=QypRacTktuZOfdSw@mail.gmail.com","threadId":"55228","inReplyTo":"CABPp-BGDB6jj+Et44D6D22KXprB89dNpyS_AAu3E8vOCtVaW1A@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-02-28T10:00:20Z","receivedAt":"2021-02-28T10:01:35Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Sat, Feb 27, 2021 at 8:49 PM Elijah Newren <newren@gmail.com> wrote:\n>\n> Your second commit is signed.  Fast-export strips any extended headers\n> on commits, such as GPG signatures, because there's no way to keep\n> them in general.\n\nWhy is it not possible to encode them with base64 and insert into the\nstream?\n\n> There are also other things that will prevent a simple fast-export |\n> fast-import pipeline from preserving your history as-is besides signed\n> commits (most of these are noted in the \"Inherited Limitations\"\n> section over at\n> https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html):\n\nIs there any way to check what commits will be altered as a result of\n`fast-export` and why? Right now I don't see that it is reported.\n\n> Hope that at least explains things for you, even if it doesn't give\n> you a workaround or a solution.\n\nThanks. That is very helpful to know.\n\nThe reason I am asking is because I tried to merge two repos with\n`reposurgeon` which operates on `fast-export` data. It is basically\nmerging GitHub wiki into main repo,\n\nAfter successfully merging them I still can not send a PR, because\nit produces a huge amount of changes, because of the stripped info.\nIt can be seen here:\n\nhttps://github.com/simons-public/protonfixes/compare/master...techtonik:master\n\nI tracked this behaviour in `reposurgeon` in this issue\nhttps://gitlab.com/esr/reposurgeon/-/issues/344\n-- \nanatoly t.\n"},{"id":"417965","messageId":"87mtvolbuj.fsf@evledraar.gmail.com","threadId":"55228","inReplyTo":"CAPkN8xK9__74a3aEFsevfdW_hQ-vzWE+c=QypRacTktuZOfdSw@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-28T10:34:44Z","receivedAt":"2021-02-28T10:35:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Feb 28 2021, anatoly techtonik wrote:\n\n> On Sat, Feb 27, 2021 at 8:49 PM Elijah Newren <newren@gmail.com> wrote:\n>>\n>> Your second commit is signed.  Fast-export strips any extended headers\n>> on commits, such as GPG signatures, because there's no way to keep\n>> them in general.\n>\n> Why is it not possible to encode them with base64 and insert into the\n> stream?\n\nI think Elijah means that in the general case people are using fast\nexport/import to export/import between different systems or in\ncombination with a utility like git-filter-repo.\n\nIn those cases users are also changing the content of the repository, so\nthe hashes will change, invalidating signatures.\n\nBut there's also cases where e.g. you don't modify the history, or only\npart of it, and could then preserve these headers. I think there's no\ninherent reason not to do so, just that nobody's cared enough to submit\npatches etc.\n\n>> There are also other things that will prevent a simple fast-export |\n>> fast-import pipeline from preserving your history as-is besides signed\n>> commits (most of these are noted in the \"Inherited Limitations\"\n>> section over at\n>> https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html):\n>\n> Is there any way to check what commits will be altered as a result of\n> `fast-export` and why? Right now I don't see that it is reported.\n\nI don't think so, but not being very familiar with fast export/import I\ndon't see why it shouldn't have some option to not munge data like that,\nor to report it, if someone cared enough to track those issues & patch\nit...\n\n>> Hope that at least explains things for you, even if it doesn't give\n>> you a workaround or a solution.\n>\n> Thanks. That is very helpful to know.\n>\n> The reason I am asking is because I tried to merge two repos with\n> `reposurgeon` which operates on `fast-export` data. It is basically\n> merging GitHub wiki into main repo,\n>\n> After successfully merging them I still can not send a PR, because\n> it produces a huge amount of changes, because of the stripped info.\n> It can be seen here:\n>\n> https://github.com/simons-public/protonfixes/compare/master...techtonik:master\n>\n> I tracked this behaviour in `reposurgeon` in this issue\n> https://gitlab.com/esr/reposurgeon/-/issues/344\n\n"},{"id":"418005","messageId":"CAPkN8xLE68d5Ngpy+LOQ8SALNgfB-+q4F3mFK-QBD=+EOKZSVg@mail.gmail.com","threadId":"55228","inReplyTo":"87mtvolbuj.fsf@evledraar.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-03-01T07:44:12Z","receivedAt":"2021-03-01T07:45:19Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> I think Elijah means that in the general case people are using fast\n> export/import to export/import between different systems or in\n> combination with a utility like git-filter-repo.\n>\n> In those cases users are also changing the content of the repository, so\n> the hashes will change, invalidating signatures.\n>\n> But there's also cases where e.g. you don't modify the history, or only\n> part of it, and could then preserve these headers. I think there's no\n> inherent reason not to do so, just that nobody's cared enough to submit\n> patches etc.\n\nIs fast-export/import the only way to filter information in `git`? Maybe there\nis a slow json-export/import tool that gives a complete representation of all\nevents in a repository? Or API that can be used to serialize and import that\nstream?\n\nIf no, then I'd like to take a look at where header filtering and serialization\ntakes place. My C skills are at the \"hello world\" level, so I am not sure I can\nwrite a patch. But I can write the logic in Python and ask somebody to port\nthat.\n-- \nanatoly t.\n"},{"id":"418049","messageId":"xmqqblc2srq0.fsf@gitster.c.googlers.com","threadId":"55228","inReplyTo":"CAPkN8xLE68d5Ngpy+LOQ8SALNgfB-+q4F3mFK-QBD=+EOKZSVg@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-01T17:34:31Z","receivedAt":"2021-03-01T17:41:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"anatoly techtonik <techtonik@gmail.com> writes:\n\n> Is fast-export/import the only way to filter information in `git`? Maybe there\n> is a slow json-export/import tool that gives a complete representation of all\n> events in a repository? Or API that can be used to serialize and import that\n> stream?\n\nI do not think representation is a problem.\n\nIt is just that the output stream of fast-export is designed to be\n\"filtered\" and the expected use case is to modify the stream somehow\nbefore feeding it to fast-import.  And because every object name and\ncommit & tag signature depends on everything that they can reach,\neven a single bit change in an earlier part of the history will\ninvalidate any and all signatures on objects that can reach it.  So\ninstead of originally-signed objects whose signatures are now\ninvalid, \"fast-export | fast-import\" pipeline would give you\noriginally-signed objects whose signatures are stripped.\n\nAdmittedly, there is a narrow use case where such a signature\ninvalidation is not an issue.  If you run fast-export and feed that\nstraight into fast-import without doing any modification to the\nstream, then you are getting a bit-for-bit identical copy.\n\nBut \"git clone --mirror\" is a much better way to do get such a\nbit-for-bit identical history and objects.  And if you want to do so\nwith sneakernet, you can create a bundle file, sneakernet it to your\ndestination, and then clone from the bundle.\n\nSo...\n\n"},{"id":"418051","messageId":"CABPp-BE=9wzF6_VypoR-uEPHsLWdV7zyE13FOgLK0h8NOcMz3g@mail.gmail.com","threadId":"55228","inReplyTo":"CAPkN8xLE68d5Ngpy+LOQ8SALNgfB-+q4F3mFK-QBD=+EOKZSVg@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-01T18:06:43Z","receivedAt":"2021-03-01T18:11:11Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik <techtonik@gmail.com> wrote:\n>\n> On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n> >\n> > I think Elijah means that in the general case people are using fast\n> > export/import to export/import between different systems or in\n> > combination with a utility like git-filter-repo.\n> >\n> > In those cases users are also changing the content of the repository, so\n> > the hashes will change, invalidating signatures.\n> >\n> > But there's also cases where e.g. you don't modify the history, or only\n> > part of it, and could then preserve these headers. I think there's no\n> > inherent reason not to do so, just that nobody's cared enough to submit\n> > patches etc.\n>\n> Is fast-export/import the only way to filter information in `git`? Maybe there\n> is a slow json-export/import tool that gives a complete representation of all\n> events in a repository? Or API that can be used to serialize and import that\n> stream?\n>\n> If no, then I'd like to take a look at where header filtering and serialization\n> takes place. My C skills are at the \"hello world\" level, so I am not sure I can\n> write a patch. But I can write the logic in Python and ask somebody to port\n> that.\n\nIf you are intent on keeping signatures because you know they are\nstill valid, then you already know you aren't modifying any\nblobs/trees/commits leading up to those signatures.  If that is the\ncase, perhaps you should just avoid exporting the signature or\nanything it depends on, and just export the stuff after that point.\nYou can do this with fast-export's --reference-excluded-parents option\nand pass it an exclusion range.  For example:\n\n   git fast-export --reference-excluded-parents ^master~5 --all\n\nand then pipe that through fast-import.\n\n\nIn general, I think if fast-export or fast-import are lacking features\nyou want, we should add them there, but I don't see how adding\nsignature reading to fast-import and signature exporting to\nfast-export makes sense in general.  Even if you assume fast-import\ncan process all the bits it is sent (e.g. you extend it to support\ncommits without an author, tags without a tagger, signed objects, any\nother extended commit headers), and even if you add flags to\nfast-export to die if there are any bits it doesn't recognize and to\nexport all pieces of blobs/trees/tags (e.g. don't add missing authors,\ndon't re-encode messages in UTF-8, don't use grafts or replace\nobjects, keep extended headers such as signatures, etc.), then it\nstill couldn't possibly work in all cases in general.  For example, if\nyou had a repository with unusual objects made by ancient or broken\ngit versions (such as tree entries in the wrong sort order, or tree\nentries that recorded modes of 040000 instead of 40000 for trees or\nsomething with perms other than 100644 or 100755 for files), then when\nfast-import goes to recreate these objects using the canonical format\nthey will no longer have the same hash and your commit signatures will\nget invalidated.  Other git commands will also refuse to create\nobjects with those oddities, even if git accepts ancient objects that\nhave them.\n\nSo, it's basically impossible to have a \"complete representation of\nall events in a repository\" that do what you want except for the\n*original* binary format.  (But if you really want to see the original\nbinary format, maybe `git cat-file --batch` will be handy to you.)\n\nBut I think fast-export's --reference-excluded-parents might come in\nhandy for you and let you do what you want.\n"},{"id":"418063","messageId":"87im6ak5hd.fsf@evledraar.gmail.com","threadId":"55228","inReplyTo":"CAPkN8xLE68d5Ngpy+LOQ8SALNgfB-+q4F3mFK-QBD=+EOKZSVg@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-01T20:02:06Z","receivedAt":"2021-03-01T20:09:15Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 01 2021, anatoly techtonik wrote:\n\n> On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n>>\n>> I think Elijah means that in the general case people are using fast\n>> export/import to export/import between different systems or in\n>> combination with a utility like git-filter-repo.\n>>\n>> In those cases users are also changing the content of the repository, so\n>> the hashes will change, invalidating signatures.\n>>\n>> But there's also cases where e.g. you don't modify the history, or only\n>> part of it, and could then preserve these headers. I think there's no\n>> inherent reason not to do so, just that nobody's cared enough to submit\n>> patches etc.\n>\n> Is fast-export/import the only way to filter information in `git`? Maybe there\n> is a slow json-export/import tool that gives a complete representation of all\n> events in a repository? Or API that can be used to serialize and import that\n> stream?\n\nAside from other things mentioned & any issues in fast export/import in\nthis thread, if you want round-trip correctness you're not going to want\nJSON-anything. It's not capable of representing arbitrary binary data.\n\nBut in any case, it's not the fast-export format that's the issue, but\nhow the tools in git.git are munging/rewriting/omitting the repository\ndata in question...\n"},{"id":"418064","messageId":"87ft1ek5dg.fsf@evledraar.gmail.com","threadId":"55228","inReplyTo":"CABPp-BE=9wzF6_VypoR-uEPHsLWdV7zyE13FOgLK0h8NOcMz3g@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-01T20:04:27Z","receivedAt":"2021-03-01T20:09:16Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 01 2021, Elijah Newren wrote:\n\n> On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik <techtonik@gmail.com> wrote:\n>>\n>> On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason\n>> <avarab@gmail.com> wrote:\n>> >\n>> > I think Elijah means that in the general case people are using fast\n>> > export/import to export/import between different systems or in\n>> > combination with a utility like git-filter-repo.\n>> >\n>> > In those cases users are also changing the content of the repository, so\n>> > the hashes will change, invalidating signatures.\n>> >\n>> > But there's also cases where e.g. you don't modify the history, or only\n>> > part of it, and could then preserve these headers. I think there's no\n>> > inherent reason not to do so, just that nobody's cared enough to submit\n>> > patches etc.\n>>\n>> Is fast-export/import the only way to filter information in `git`? Maybe there\n>> is a slow json-export/import tool that gives a complete representation of all\n>> events in a repository? Or API that can be used to serialize and import that\n>> stream?\n>>\n>> If no, then I'd like to take a look at where header filtering and serialization\n>> takes place. My C skills are at the \"hello world\" level, so I am not sure I can\n>> write a patch. But I can write the logic in Python and ask somebody to port\n>> that.\n>\n> If you are intent on keeping signatures because you know they are\n> still valid, then you already know you aren't modifying any\n> blobs/trees/commits leading up to those signatures.  If that is the\n> case, perhaps you should just avoid exporting the signature or\n> anything it depends on, and just export the stuff after that point.\n> You can do this with fast-export's --reference-excluded-parents option\n> and pass it an exclusion range.  For example:\n>\n>    git fast-export --reference-excluded-parents ^master~5 --all\n>\n> and then pipe that through fast-import.\n>\n>\n> In general, I think if fast-export or fast-import are lacking features\n> you want, we should add them there, but I don't see how adding\n> signature reading to fast-import and signature exporting to\n> fast-export makes sense in general.  Even if you assume fast-import\n> can process all the bits it is sent (e.g. you extend it to support\n> commits without an author, tags without a tagger, signed objects, any\n> other extended commit headers), and even if you add flags to\n> fast-export to die if there are any bits it doesn't recognize and to\n> export all pieces of blobs/trees/tags (e.g. don't add missing authors,\n> don't re-encode messages in UTF-8, don't use grafts or replace\n> objects, keep extended headers such as signatures, etc.), then it\n> still couldn't possibly work in all cases in general.  For example, if\n> you had a repository with unusual objects made by ancient or broken\n> git versions (such as tree entries in the wrong sort order, or tree\n> entries that recorded modes of 040000 instead of 40000 for trees or\n> something with perms other than 100644 or 100755 for files), then when\n> fast-import goes to recreate these objects using the canonical format\n> they will no longer have the same hash and your commit signatures will\n> get invalidated.  Other git commands will also refuse to create\n> objects with those oddities, even if git accepts ancient objects that\n> have them.\n>\n> So, it's basically impossible to have a \"complete representation of\n> all events in a repository\" that do what you want except for the\n> *original* binary format.  (But if you really want to see the original\n> binary format, maybe `git cat-file --batch` will be handy to you.)\n>\n> But I think fast-export's --reference-excluded-parents might come in\n> handy for you and let you do what you want.\n\n...to add to that line of thinking, it's also a completely valid\ntechnique to just completele rewrite your repository, then (re-)push the\nold signed tags to refs/tags/*.\n\nBy default they won't be pulled down as they won't reference commits on\nbranches you're fetching, and you can also stick them somewhere else\nthan refs/tags/*, e.g. refs/legacy-tags/*.\n\nNone of the commit history will be the same, but the content (mostly)\nwill, which is usually what matters when checking out an old tag.\n\nOf course this hack has little benefit over just keeping a foo-old.git\nrepo around, and moving on with new history in your new foo.git.\n"},{"id":"418065","messageId":"CABPp-BHdtAKz_V2RhBqevMy+Hy_rHtQ7Y2chggpt1rZ9nRn8Zw@mail.gmail.com","threadId":"55228","inReplyTo":"87ft1ek5dg.fsf@evledraar.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-01T20:17:36Z","receivedAt":"2021-03-01T20:24:01Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Mar 1, 2021 at 12:04 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n>\n> On Mon, Mar 01 2021, Elijah Newren wrote:\n>\n> > On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik <techtonik@gmail.com> wrote:\n> >>\n> >> On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason\n> >> <avarab@gmail.com> wrote:\n> >> >\n> >> > I think Elijah means that in the general case people are using fast\n> >> > export/import to export/import between different systems or in\n> >> > combination with a utility like git-filter-repo.\n> >> >\n> >> > In those cases users are also changing the content of the repository, so\n> >> > the hashes will change, invalidating signatures.\n> >> >\n> >> > But there's also cases where e.g. you don't modify the history, or only\n> >> > part of it, and could then preserve these headers. I think there's no\n> >> > inherent reason not to do so, just that nobody's cared enough to submit\n> >> > patches etc.\n> >>\n> >> Is fast-export/import the only way to filter information in `git`? Maybe there\n> >> is a slow json-export/import tool that gives a complete representation of all\n> >> events in a repository? Or API that can be used to serialize and import that\n> >> stream?\n> >>\n> >> If no, then I'd like to take a look at where header filtering and serialization\n> >> takes place. My C skills are at the \"hello world\" level, so I am not sure I can\n> >> write a patch. But I can write the logic in Python and ask somebody to port\n> >> that.\n> >\n> > If you are intent on keeping signatures because you know they are\n> > still valid, then you already know you aren't modifying any\n> > blobs/trees/commits leading up to those signatures.  If that is the\n> > case, perhaps you should just avoid exporting the signature or\n> > anything it depends on, and just export the stuff after that point.\n> > You can do this with fast-export's --reference-excluded-parents option\n> > and pass it an exclusion range.  For example:\n> >\n> >    git fast-export --reference-excluded-parents ^master~5 --all\n> >\n> > and then pipe that through fast-import.\n> >\n> >\n> > In general, I think if fast-export or fast-import are lacking features\n> > you want, we should add them there, but I don't see how adding\n> > signature reading to fast-import and signature exporting to\n> > fast-export makes sense in general.  Even if you assume fast-import\n> > can process all the bits it is sent (e.g. you extend it to support\n> > commits without an author, tags without a tagger, signed objects, any\n> > other extended commit headers), and even if you add flags to\n> > fast-export to die if there are any bits it doesn't recognize and to\n> > export all pieces of blobs/trees/tags (e.g. don't add missing authors,\n> > don't re-encode messages in UTF-8, don't use grafts or replace\n> > objects, keep extended headers such as signatures, etc.), then it\n> > still couldn't possibly work in all cases in general.  For example, if\n> > you had a repository with unusual objects made by ancient or broken\n> > git versions (such as tree entries in the wrong sort order, or tree\n> > entries that recorded modes of 040000 instead of 40000 for trees or\n> > something with perms other than 100644 or 100755 for files), then when\n> > fast-import goes to recreate these objects using the canonical format\n> > they will no longer have the same hash and your commit signatures will\n> > get invalidated.  Other git commands will also refuse to create\n> > objects with those oddities, even if git accepts ancient objects that\n> > have them.\n> >\n> > So, it's basically impossible to have a \"complete representation of\n> > all events in a repository\" that do what you want except for the\n> > *original* binary format.  (But if you really want to see the original\n> > binary format, maybe `git cat-file --batch` will be handy to you.)\n> >\n> > But I think fast-export's --reference-excluded-parents might come in\n> > handy for you and let you do what you want.\n>\n> ...to add to that line of thinking, it's also a completely valid\n> technique to just completele rewrite your repository, then (re-)push the\n> old signed tags to refs/tags/*.\n\nThe repository in question didn't have any signed tags, just a signed commit.\n\n> By default they won't be pulled down as they won't reference commits on\n> branches you're fetching, and you can also stick them somewhere else\n> than refs/tags/*, e.g. refs/legacy-tags/*.\n>\n> None of the commit history will be the same, but the content (mostly)\n> will, which is usually what matters when checking out an old tag.\n>\n> Of course this hack has little benefit over just keeping a foo-old.git\n> repo around, and moving on with new history in your new foo.git.\n"},{"id":"418116","messageId":"CAPkN8xKM0zi-AB1xKRGp=whEQTZAbn78w0JjvUXfGfRDky0C=w@mail.gmail.com","threadId":"55228","inReplyTo":"xmqqblc2srq0.fsf@gitster.c.googlers.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-03-02T21:52:53Z","receivedAt":"2021-03-03T06:41:57Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Mon, Mar 1, 2021 at 8:34 PM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> It is just that the output stream of fast-export is designed to be\n> \"filtered\" and the expected use case is to modify the stream somehow\n> before feeding it to fast-import.  And because every object name and\n> commit & tag signature depends on everything that they can reach,\n> even a single bit change in an earlier part of the history will\n> invalidate any and all signatures on objects that can reach it.  So\n> instead of originally-signed objects whose signatures are now\n> invalid, \"fast-export | fast-import\" pipeline would give you\n> originally-signed objects whose signatures are stripped.\n\nI need to merge two unrelated repos and I am using `reposurgeon`\nhttp://www.catb.org/~esr/reposurgeon/repository-editing.html\nto do this preserving timestamps and commit order. The model of\noperation is that it reads revisions into memory from git using\nfast-export, operates on them, and then rebuild the stream back\ninto git repo with fast-import. The problem is that in the exported\ndump the information is already lost, and the resulting commits are\n\"not mergeable\". Basically all GitHub repositories where people\nedited `README.md` online are \"not mergeable\" after this point,\nbecause all GitHub edited commits are signed.\n\nFor my use case, where I just need to attach another branch in\ntime without altering original commits in any way, `reposurgeon`\ncan not be used.\n\n> Admittedly, there is a narrow use case where such a signature\n> invalidation is not an issue.  If you run fast-export and feed that\n> straight into fast-import without doing any modification to the\n> stream, then you are getting a bit-for-bit identical copy.\n\nI did just that and signatures got stripped, altering history.\n\ngit -C protonfixes fast-export --all --reencode=no | (cd\nprotoimported && git fast-import)\n\n-- \nanatoly t.\n"},{"id":"418117","messageId":"CAPkN8xLtVS4Xx86UMrRXD09P72NQc+DwQVG40gEXaEDRD53mBA@mail.gmail.com","threadId":"55228","inReplyTo":"87im6ak5hd.fsf@evledraar.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-03-02T22:23:59Z","receivedAt":"2021-03-03T06:41:58Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Mon, Mar 1, 2021 at 11:02 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> Aside from other things mentioned & any issues in fast export/import in\n> this thread, if you want round-trip correctness you're not going to want\n> JSON-anything. It's not capable of representing arbitrary binary data.\n\nYes, binary data needs to be explicitly represented in base64 or similar\nencoding. Just the same way as ordinary strings will need escape\nsymbols.\n-- \nanatoly t.\n"},{"id":"418119","messageId":"CAPkN8xKTAEfm+1-PfjUPYMf7RhjuwSEb3eESkTQjcWhUDi+7FA@mail.gmail.com","threadId":"55228","inReplyTo":"CABPp-BE=9wzF6_VypoR-uEPHsLWdV7zyE13FOgLK0h8NOcMz3g@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-03-02T22:12:05Z","receivedAt":"2021-03-03T06:41:58Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Mon, Mar 1, 2021 at 9:06 PM Elijah Newren <newren@gmail.com> wrote:\n> On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik <techtonik@gmail.com> wrote:\n> For example:\n>\n>    git fast-export --reference-excluded-parents ^master~5 --all\n>\n> and then pipe that through fast-import.\n\nThat may come in handy, but if certain parents are excluded, it will be\nimpossible to find them to reference and attach branches to them.\n\n> Other git commands will also refuse to create\n> objects with those oddities, even if git accepts ancient objects that\n> have them.\n\nAre there any `lint` commands that can detect and warn about those\noddities?\n\n> (But if you really want to see the original\n> binary format, maybe `git cat-file --batch` will be handy to you.)\n\nLooks good. Is there a way to import it back? And how hard it could\nbe to write a parser for it? Is there a specification for its fields?\n-- \nanatoly t.\n"},{"id":"418134","messageId":"08f31194-dce6-9434-c362-94d9a2d97563@kdbg.org","threadId":"55228","inReplyTo":"CAPkN8xKM0zi-AB1xKRGp=whEQTZAbn78w0JjvUXfGfRDky0C=w@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-03-03T07:13:13Z","receivedAt":"2021-03-04T00:22:41Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 02.03.21 um 22:52 schrieb anatoly techtonik:\n> For my use case, where I just need to attach another branch in\n> time without altering original commits in any way, `reposurgeon`\n> can not be used.\n\nWhat do you mean by \"attach another branch in time\"? Because if you\nreally do not want to alter original commits in any way, perhaps you\nonly want `git fetch /the/other/repository master:the-other-one-s-master`?\n\n-- Hannes\n"},{"id":"418190","messageId":"xmqqlfb3g2jp.fsf@gitster.c.googlers.com","threadId":"55228","inReplyTo":"08f31194-dce6-9434-c362-94d9a2d97563@kdbg.org","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-04T00:55:54Z","receivedAt":"2021-03-04T01:09:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt <j6t@kdbg.org> writes:\n\n> Am 02.03.21 um 22:52 schrieb anatoly techtonik:\n>> For my use case, where I just need to attach another branch in\n>> time without altering original commits in any way, `reposurgeon`\n>> can not be used.\n>\n> What do you mean by \"attach another branch in time\"? Because if you\n> really do not want to alter original commits in any way, perhaps you\n> only want `git fetch /the/other/repository master:the-other-one-s-master`?\n\nYeah, I had the same impression.  If a bit-for-bit identical copy of\nthe original history is needed, then fetching from the original\nrepository (either directly or via a bundle) would be a much simpler\nand performant way.\n\nThanks.\n"},{"id":"432307","messageId":"CAPkN8x+agKRRD0Zd-pxs_EuYO_Xm8EyE0nJLCWQB4KNuNkvK8Q@mail.gmail.com","threadId":"55228","inReplyTo":"xmqqlfb3g2jp.fsf@gitster.c.googlers.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-08-09T15:45:26Z","receivedAt":"2021-08-09T15:45:44Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Thu, Mar 4, 2021 at 3:56 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Johannes Sixt <j6t@kdbg.org> writes:\n>\n> > Am 02.03.21 um 22:52 schrieb anatoly techtonik:\n> >> For my use case, where I just need to attach another branch in\n> >> time without altering original commits in any way, `reposurgeon`\n> >> can not be used.\n> >\n> > What do you mean by \"attach another branch in time\"? Because if you\n> > really do not want to alter original commits in any way, perhaps you\n> > only want `git fetch /the/other/repository master:the-other-one-s-master`?\n>\n> Yeah, I had the same impression.  If a bit-for-bit identical copy of\n> the original history is needed, then fetching from the original\n> repository (either directly or via a bundle) would be a much simpler\n> and performant way.\n\nThe goal is to have an editable stream, which, if left without edits, would\nbe bit-by-bit identical, so that external tools like `reposurgeon` could\noperate on that stream and be audited.\n\nRight now, because the repository\nhttps://github.com/simons-public/protonfixes contains a signed commit\nright from the start, the simple fast-export and fast-import with git itself\nfails the check.\n\nI understand that patching `git` to add `--complete` to fast-import is\nrealistically beyond my coding abilities, and my only option is to parse\nthe binary stream produced by `git cat-file --batch`, which I also won't\nbe able to do without specification.\n\nP.S. I am resurrecting the old thread, because my problem with editing\nthe history of the repository with an external tool still can not be solved.\n-- \nanatoly t.\n"},{"id":"432323","messageId":"CABPp-BH5RhHR-KhhumuhZGy2F4ypUBoqgAatY5MKkQsB46KM4g@mail.gmail.com","threadId":"55228","inReplyTo":"CAPkN8x+agKRRD0Zd-pxs_EuYO_Xm8EyE0nJLCWQB4KNuNkvK8Q@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-09T18:15:44Z","receivedAt":"2021-08-09T18:15:59Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 9, 2021 at 8:45 AM anatoly techtonik <techtonik@gmail.com> wrote:\n>\n> On Thu, Mar 4, 2021 at 3:56 AM Junio C Hamano <gitster@pobox.com> wrote:\n> >\n> > Johannes Sixt <j6t@kdbg.org> writes:\n> >\n> > > Am 02.03.21 um 22:52 schrieb anatoly techtonik:\n> > >> For my use case, where I just need to attach another branch in\n> > >> time without altering original commits in any way, `reposurgeon`\n> > >> can not be used.\n> > >\n> > > What do you mean by \"attach another branch in time\"? Because if you\n> > > really do not want to alter original commits in any way, perhaps you\n> > > only want `git fetch /the/other/repository master:the-other-one-s-master`?\n> >\n> > Yeah, I had the same impression.  If a bit-for-bit identical copy of\n> > the original history is needed, then fetching from the original\n> > repository (either directly or via a bundle) would be a much simpler\n> > and performant way.\n>\n> The goal is to have an editable stream, which, if left without edits, would\n> be bit-by-bit identical, so that external tools like `reposurgeon` could\n> operate on that stream and be audited.\n\nThere were some patches proposed some months back[1] to make\nfast-import allow importing signed commits...except that they\nunconditionally kept the signatures and didn't do any validation,\nwhich would have resulted in invalid signatures if any edits happened.\nI suggested adding signature verification (which would allow options\nlike erroring out if they didn't match, or dropping signatures when\nthey didn't match but keeping them otherwise).  That'd help usecases\nlike yours.  The author wasn't interested in implementing that\nsuggestion (and it's a low priority for me that I may never get around\nto).  The series also wasn't pushed through and eventually was\ndropped.\n\nHowever, that wouldn't fully solve your stated goal.  As already\nmentioned earlier in this thread, I don't think your stated goal is\nrealistic; the only complete bit-for-bit identical representation of\nthe repository is the original binary format.\n\nYour stated goal here, however, isn't required for solving the usecase\nyou present.\n\n[1] https://lore.kernel.org/git/20210430232537.1131641-1-lukeshu@lukeshu.com/\n\n> Right now, because the repository\n> https://github.com/simons-public/protonfixes contains a signed commit\n> right from the start, the simple fast-export and fast-import with git itself\n> fails the check.\n\nYes, and I mentioned several other reasons why a round-trip from\nfast-export through fast-import cannot be relied upon to preserve\nobject hashes.\n\n> I understand that patching `git` to add `--complete` to fast-import is\n> realistically beyond my coding abilities, and my only option is to parse\n\nIt's more patching than that which would be required:\n(1) It'd be both fast-export and fast-import that would need patching,\nnot just fast-import.\n(2) --complete is a bit of a misnomer too, because it's not just\nget-all-the-data, it's keep-the-data-in-the-original-format.  If\nobjects had modes of 040000 instead of 40000, despite meaning the same\nthing, you'd have to prevent canonicalization and store them as the\noriginal recorded value or you'd get a different hash.  Ditto for\ncommit messages with extra data after a NUL byte, and a variety of\nother possible issues.\n(3) fast-export works by looking for the relevant bits it knows how to\nexport.  You'd have to redesign it to fully parse every bit of data in\neach object it looks at, throw errors if it didn't recognize any, and\nmake sure it exports all the bits.  That might be difficult since it's\nhard to know how to future proof it.  How do you guarantee you've\nprinted every field in a commit struct, when that struct might gain\nnew fields in the future?  (This is especially challenging since\nfast-export/fast-import might not be considered core tools, or at\nleast don't get as much attention as the \"truly core\" parts of git;\nsee https://lore.kernel.org/git/xmqq36mxdnpz.fsf@gitster-ct.c.googlers.com/)\n\n> the binary stream produced by `git cat-file --batch`, which I also won't\n> be able to do without specification.\n\nThe specification is already available in the manual.  Just run `git\ncat-file --help` to see it.  Let me quote part of it for you:\n\n       For example, --batch without a custom format would produce:\n\n           <sha1> SP <type> SP <size> LF\n           <contents> LF\n\n> P.S. I am resurrecting the old thread, because my problem with editing\n> the history of the repository with an external tool still can not be solved.\n\nSure it can, just use fast-export's --reference-excluded-parents\noption and don't export commits you know you won't need to change.\n\nOr, if for some reason you are really set on exporting everything and\nthen editing, then go ahead and create the full fast-export output,\nincluding with all your edits, and then post-process it manually\nbefore feeding to fast-import.  In particular, in the post-processing\nstep find the commits that were problematic that you know won't be\nmodified, such as your signed commit.  Then go edit that fast-export\ndump and (a) remove the dump of the no-longer-signed signed commit\n(because you don't want it), and (b) replace any references to the\nno-longer-signed-commit (e.g. \"from :12\") to instead use the hash of\nthe actual original signed commit (e.g. \"from\nd3d24b63446c7d06586eaa51764ff0c619113f09\").  If you do that, then git\nfast-import will just build the new commits on the existing signed\ncommit instead of on some new commit that is missing the signature.\nTechnically, you can even skip step (a), as all it will do is produce\nan extra commit in your repository that isn't used and thus will be\ngarbage collected later.\n"},{"id":"432383","messageId":"CAPkN8xLN_fKdbU8ugxLYJ1YeCJ8CxBWh+kdhAq1mR8hfAe-NAA@mail.gmail.com","threadId":"55228","inReplyTo":"CABPp-BH5RhHR-KhhumuhZGy2F4ypUBoqgAatY5MKkQsB46KM4g@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-08-10T15:51:40Z","receivedAt":"2021-08-10T15:52:01Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Mon, Aug 9, 2021 at 9:15 PM Elijah Newren <newren@gmail.com> wrote:\n>\n> The author wasn't interested in implementing that\n> suggestion (and it's a low priority for me that I may never get around\n> to).  The series also wasn't pushed through and eventually was\n> dropped.\n\nWhat it takes to validate the commit signature? Isn't it the same as\nvalidating commit tag? Is it possible to merge at least the `--fast-export`\npart? The effect of roundtrip would be the same, but at least external\ntools would be able to detect signed commits and warn users.\n\n> [1] https://lore.kernel.org/git/20210430232537.1131641-1-lukeshu@lukeshu.com/\n\n> Yes, and I mentioned several other reasons why a round-trip from\n> fast-export through fast-import cannot be relied upon to preserve\n> object hashes.\n\nYes, I understand that. What would be the recommended way to detect\nwhich commits would change as a result of the round-trip? It will then\nbe possible to warn users in `reposurgeon` `lint` command.\n\n> (3) fast-export works by looking for the relevant bits it knows how to\n> export.  You'd have to redesign it to fully parse every bit of data in\n> each object it looks at, throw errors if it didn't recognize any, and\n> make sure it exports all the bits.  That might be difficult since it's\n> hard to know how to future proof it.  How do you guarantee you've\n> printed every field in a commit struct, when that struct might gain\n> new fields in the future?  (This is especially challenging since\n> fast-export/fast-import might not be considered core tools, or at\n> least don't get as much attention as the \"truly core\" parts of git;\n> see https://lore.kernel.org/git/xmqq36mxdnpz.fsf@gitster-ct.c.googlers.com/)\n\nLooks like the only way to make it forward compatible is to introduce\nsome kind of versioning and a validation schema like protobuf. Otherwise\nwriting an importer and exporter for each and every thing that may\nencounter in a git stream may be unrealistic, yes.\n\n> > P.S. I am resurrecting the old thread, because my problem with editing\n> > the history of the repository with an external tool still can not be solved.\n>\n> Sure it can, just use fast-export's --reference-excluded-parents\n> option and don't export commits you know you won't need to change.\n\nHow does `--reference-excluded-parents` help to read signed commits?\n\n`reposurgeon` needs all commits to select those that are needed by\ndifferent criteria. It is hard to tell which commits are not important without\nreading and processing them first.\n\n> Or, if for some reason you are really set on exporting everything and\n> then editing, then go ahead and create the full fast-export output,\n> including with all your edits, and then post-process it manually\n> before feeding to fast-import.  In particular, in the post-processing\n> step find the commits that were problematic that you know won't be\n> modified, such as your signed commit.  Then go edit that fast-export\n> dump and (a) remove the dump of the no-longer-signed signed commit\n> (because you don't want it), and (b) replace any references to the\n> no-longer-signed-commit (e.g. \"from :12\") to instead use the hash of\n> the actual original signed commit (e.g. \"from\n> d3d24b63446c7d06586eaa51764ff0c619113f09\").  If you do that, then git\n> fast-import will just build the new commits on the existing signed\n> commit instead of on some new commit that is missing the signature.\n> Technically, you can even skip step (a), as all it will do is produce\n> an extra commit in your repository that isn't used and thus will be\n> garbage collected later.\n\nThe problem is to detect problematic signed commits, because as I\nunderstand `fast-export` doesn't give any signs if commits were signed\nbefore the export.\n-- \nanatoly t.\n"},{"id":"432390","messageId":"CABPp-BEqvmSaqVrK=nQsk-8PNXq6Pzq4Y-=RopYwTDjtyitAuw@mail.gmail.com","threadId":"55228","inReplyTo":"CAPkN8xLN_fKdbU8ugxLYJ1YeCJ8CxBWh+kdhAq1mR8hfAe-NAA@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-10T17:57:54Z","receivedAt":"2021-08-10T18:18:08Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 10, 2021 at 8:51 AM anatoly techtonik <techtonik@gmail.com> wrote:\n>\n> On Mon, Aug 9, 2021 at 9:15 PM Elijah Newren <newren@gmail.com> wrote:\n> >\n> > The author wasn't interested in implementing that\n> > suggestion (and it's a low priority for me that I may never get around\n> > to).  The series also wasn't pushed through and eventually was\n> > dropped.\n>\n> What it takes to validate the commit signature?\n\nI'm not familiar with any of the gpg libraries, and don't even have an\nactive gpg key.  So, I don't know.  Some quick grepping shows that we\nhave gpg-interface.[ch], so we have some functions we can apparently\ncall.\n\n> Isn't it the same as validating commit tag?\n\ngpg signatures of tags are somewhat different than gpg signatures of commits:\n\n* gpg signatures for tags are simply part of the annotated tag message\n* gpg signatures for commits are stored in a separate commit header,\nnot just as extra text at the end of the commit message\n\nThis gpg signature handling for tags means that fast-import isn't even\naware of whether the tag is signed; it simply sees a commit message\nand records it.  fast-export also would have been unaware and just\nexported them as-is if someone hadn't written some special parsing for\nit.  fast-import would need to do similar special parsing to become\naware of whether the tags are signed or not.  For now, fast-import\njust keeps any tag messages as-is, and thus potentially writes invalid\ntag signatures.  (The only way people have to control this is at the\nfast-export side with the --signed-tags flag, which gives you the\nchoices of abort, strip, or keep the signatures even though they'll\nlikely be wrong.)  If fast-import were to gain knowledge of tag\nsignatures and an ability to validate them, it could offer smarter\noptions like keep-if-valid-and-discard-otherwise.\n\nIn contrast, the fact that gpg signatures for commits have to be\nrecorded as a separate commit header means they cannot be recorded in\nfast-import without additional code changes.  And both the fast-export\nand fast-import sides have to be made aware of and specially handle\nthe commit signatures for them to even get propagated, let alone\nvalidated.\n\n> Is it possible to merge at least the `--fast-export`\n> part? The effect of roundtrip would be the same, but at least external\n> tools would be able to detect signed commits and warn users.\n\nThe fact that it wasn't merged suggests there was some issue raised in\nfeedback that wasn't addressed.  I don't remember if that was the case\nor not, but someone would have to find out, address any remaining\nissues pointed out by feedback, and champion it through.\n\nPersonally, I don't like shoving a half solution through and think\nthere needs to be validation on the fast-import side added at the same\ntime, but others may disagree with me.  I have plenty of other\nprojects to work on, though, so whoever does the work will more likely\nbe the ones to decide.\n\n> > [1] https://lore.kernel.org/git/20210430232537.1131641-1-lukeshu@lukeshu.com/\n>\n> > Yes, and I mentioned several other reasons why a round-trip from\n> > fast-export through fast-import cannot be relied upon to preserve\n> > object hashes.\n>\n> Yes, I understand that. What would be the recommended way to detect\n> which commits would change as a result of the round-trip? It will then\n> be possible to warn users in `reposurgeon` `lint` command.\n\nThere is no function or command that would check that kind of thing\nshort of doing the round-trip.  I provided a list of reasons IDs could\nchange as a starting point in case anyone wanted to try to write a\nfunction or command that could check, and to point out that it is a\nlong list and might grow in the future.\n\nI think practically, if you're doing a one-shot export (as I\noriginally assumed from your email), that you'd find out and then just\nmanually fix things up by hand.  If your goal is writing or changing a\ngeneral purpose filtering tool, then I'd suggest instead using the\nalternate technique I outlined in the other thread you started at [2].\n\n[2] https://lore.kernel.org/git/CABPp-BH4dcsW52immJpTjgY5LjaVfKrY9MaUOnKT3byi2tBPpg@mail.gmail.com/\n\n> > (3) fast-export works by looking for the relevant bits it knows how to\n> > export.  You'd have to redesign it to fully parse every bit of data in\n> > each object it looks at, throw errors if it didn't recognize any, and\n> > make sure it exports all the bits.  That might be difficult since it's\n> > hard to know how to future proof it.  How do you guarantee you've\n> > printed every field in a commit struct, when that struct might gain\n> > new fields in the future?  (This is especially challenging since\n> > fast-export/fast-import might not be considered core tools, or at\n> > least don't get as much attention as the \"truly core\" parts of git;\n> > see https://lore.kernel.org/git/xmqq36mxdnpz.fsf@gitster-ct.c.googlers.com/)\n>\n> Looks like the only way to make it forward compatible is to introduce\n> some kind of versioning and a validation schema like protobuf. Otherwise\n> writing an importer and exporter for each and every thing that may\n> encounter in a git stream may be unrealistic, yes.\n>\n> > > P.S. I am resurrecting the old thread, because my problem with editing\n> > > the history of the repository with an external tool still can not be solved.\n> >\n> > Sure it can, just use fast-export's --reference-excluded-parents\n> > option and don't export commits you know you won't need to change.\n>\n> How does `--reference-excluded-parents` help to read signed commits?\n\nIt doesn't.  I was assuming you were doing a one shot export, namely\nof the repository you linked to,\nhttps://github.com/simons-public/protonfixes, and that you already\nknew which commits were not going to be changed (because you pointed\nthem out in your email to the list) -- and in fact that it was only a\nsingle commit affected, as you mentioned.\n\nArmed with that knowledge, you could just export the parts of the\nrepository AFTER that commit, and use --reference-excluded-parents to\nmake sure the fast-export stream built upon them rather than squashing\nall changes up to that point into the first commit in the stream.\n\n> `reposurgeon` needs all commits to select those that are needed by\n> different criteria. It is hard to tell which commits are not important without\n> reading and processing them first.\n\nRight, so you aren't trying to just handle this one repository, but\nmodify/create a general purpose tool that does so.  See my response in\nthe other thread you started, again at [2] above.\n\n> > Or, if for some reason you are really set on exporting everything and\n> > then editing, then go ahead and create the full fast-export output,\n> > including with all your edits, and then post-process it manually\n> > before feeding to fast-import.  In particular, in the post-processing\n> > step find the commits that were problematic that you know won't be\n> > modified, such as your signed commit.  Then go edit that fast-export\n> > dump and (a) remove the dump of the no-longer-signed signed commit\n> > (because you don't want it), and (b) replace any references to the\n> > no-longer-signed-commit (e.g. \"from :12\") to instead use the hash of\n> > the actual original signed commit (e.g. \"from\n> > d3d24b63446c7d06586eaa51764ff0c619113f09\").  If you do that, then git\n> > fast-import will just build the new commits on the existing signed\n> > commit instead of on some new commit that is missing the signature.\n> > Technically, you can even skip step (a), as all it will do is produce\n> > an extra commit in your repository that isn't used and thus will be\n> > garbage collected later.\n>\n> The problem is to detect problematic signed commits, because as I\n> understand `fast-export` doesn't give any signs if commits were signed\n> before the export.\n\nSigned commits is just one issue, and you'll have to add special code\nto handle a bunch of other special cases if you go down this route.\nI'd rephrase the problem.  You want to know when _your tool_ (e.g.\nreposurgeon since you refer to it multiple times; I'm guessing you're\ncontributing to it?) has not modified a commit or any of its\nancestors, and when it hasn't, then _your tool_ should remove that\ncommit from the fast-export stream and replace any references to it by\nthe original commit's object id.  I outlined how to do this in [2],\nreferenced above, making use of the --show-original-ids flag to\nfast-export.  If you do that, then for any commits which you haven't\nmodified (including not modifying any of its ancestors), then you'll\nkeep the same commits as-is with no stripping of gpg-signatures or\ncanonicalization of objects, so that you'll have the exact same commit\nIDs.  Further, you can do this today, without any changes to git\nfast-export or git fast-import.\n"},{"id":"468869","messageId":"CAPkN8xJ_B_t7L6bErsFB+rp6fzy7PO55myWqyJWPNbTgiQ69ow@mail.gmail.com","threadId":"55228","inReplyTo":"CABPp-BEqvmSaqVrK=nQsk-8PNXq6Pzq4Y-=RopYwTDjtyitAuw@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2022-12-11T18:30:08Z","receivedAt":"2022-12-11T18:30:33Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"On Tue, Aug 10, 2021 at 8:58 PM Elijah Newren <newren@gmail.com> wrote:\n>\n> On Tue, Aug 10, 2021 at 8:51 AM anatoly techtonik <techtonik@gmail.com> wrote:\n> >\n> > On Mon, Aug 9, 2021 at 9:15 PM Elijah Newren <newren@gmail.com> wrote:\n> > >\n>\n> [2] https://lore.kernel.org/git/CABPp-BH4dcsW52immJpTjgY5LjaVfKrY9MaUOnKT3byi2tBPpg@mail.gmail.com/\n>\n> Signed commits is just one issue, and you'll have to add special code\n> to handle a bunch of other special cases if you go down this route.\n> I'd rephrase the problem.  You want to know when _your tool_ (e.g.\n> reposurgeon since you refer to it multiple times; I'm guessing you're\n> contributing to it?) has not modified a commit or any of its\n> ancestors, and when it hasn't, then _your tool_ should remove that\n> commit from the fast-export stream and replace any references to it by\n> the original commit's object id.  I outlined how to do this in [2],\n> referenced above, making use of the --show-original-ids flag to\n> fast-export.  If you do that, then for any commits which you haven't\n> modified (including not modifying any of its ancestors), then you'll\n> keep the same commits as-is with no stripping of gpg-signatures or\n> canonicalization of objects, so that you'll have the exact same commit\n> IDs.  Further, you can do this today, without any changes to git\n> fast-export or git fast-import.\n\nTook me a while to process the reply. Let's recap.\n\nI want to make a roundtrip export/import of\nhttps://github.com/simons-public/protonfixes which should get exactly\nthe same repository.\n\n# --- fast-export to exported.txt\ngit clone https://github.com/simons-public/protonfixes\ngit -C protonfixes fast-export --all > exported.txt\n# --- check revision of the repo\ngit -C protonfixes rev-parse HEAD\n# 681411ba8ceb5d2d790e674eb7a5b98951d426e6\n\n# --- fast-import into new repo\ngit init newrepo\ngit -C newrepo fast-import < exported.txt\n# --- checking revision of the new repo\ngit -C newrepo rev-parse HEAD\n# 9888762d7857d9721f0c354e7fc187a199754a4b\n\nHashes don't match. The roundtrip fails.\n\n\nLet's see if --reference-excluded-parents helps.\n\n# --- export below produces the same export stream as above\ngit -C protonfixes fast-export --reference-excluded-parents --all >\nexported_parents.txt\n\n\nBecause fast-import/fast-export don't work, you propose to keep the old\nrepo around until it is clear which commits I am going to modify. Then\nmake a new fast-export starting from the first commit I am going to\nmodify with --reference-excluded-parents flag. Is that correct so far?\n\nThen given this partial export and old repo, how to init the new repo\nthat fast-import can apply its tail there?\n\nWhat if I have multiple commits that I modify, but I don't know which\nof their parents was first? And when I touch commits from different\nbranches, how to recreate their parent history intact in one repo?\n\n-- \nanatoly t.\n"},{"id":"470243","messageId":"CABPp-BFRBc4sNHcmbEZOvXxA1wVBiQn2VQUkSkbFnpz++yT8hw@mail.gmail.com","threadId":"55228","inReplyTo":"CAPkN8xJ_B_t7L6bErsFB+rp6fzy7PO55myWqyJWPNbTgiQ69ow@mail.gmail.com","subject":"Re: Round-tripping fast-export/import changes commit hashes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-01-13T07:21:55Z","receivedAt":"2023-01-13T07:27:24Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Dec 11, 2022 at 10:30 AM anatoly techtonik <techtonik@gmail.com> wrote:\n>\n> On Tue, Aug 10, 2021 at 8:58 PM Elijah Newren <newren@gmail.com> wrote:\n> >\n> > On Tue, Aug 10, 2021 at 8:51 AM anatoly techtonik <techtonik@gmail.com> wrote:\n> > >\n> > > On Mon, Aug 9, 2021 at 9:15 PM Elijah Newren <newren@gmail.com> wrote:\n> > > >\n> >\n> > [2] https://lore.kernel.org/git/CABPp-BH4dcsW52immJpTjgY5LjaVfKrY9MaUOnKT3byi2tBPpg@mail.gmail.com/\n> >\n> > Signed commits is just one issue, and you'll have to add special code\n> > to handle a bunch of other special cases if you go down this route.\n> > I'd rephrase the problem.  You want to know when _your tool_ (e.g.\n> > reposurgeon since you refer to it multiple times; I'm guessing you're\n> > contributing to it?) has not modified a commit or any of its\n> > ancestors, and when it hasn't, then _your tool_ should remove that\n> > commit from the fast-export stream and replace any references to it by\n> > the original commit's object id.  I outlined how to do this in [2],\n> > referenced above, making use of the --show-original-ids flag to\n> > fast-export.  If you do that, then for any commits which you haven't\n> > modified (including not modifying any of its ancestors), then you'll\n> > keep the same commits as-is with no stripping of gpg-signatures or\n> > canonicalization of objects, so that you'll have the exact same commit\n> > IDs.  Further, you can do this today, without any changes to git\n> > fast-export or git fast-import.\n>\n> Took me a while to process the reply. Let's recap.\n>\n> I want to make a roundtrip export/import of\n> https://github.com/simons-public/protonfixes which should get exactly\n> the same repository.\n\nAs I've stated a few times in the thread, this request of yours is\nsimply impossible for general repositories ([1] contains the best\nsummary of the reasons).  For the specific repository in question, the\nonly relevant roadblocker is the presence of a signed commit which\nhappens to be a root commit.  That opens the door to some workarounds\nthat could be used with this specific repository.\n\n[1] https://lore.kernel.org/git/CABPp-BGDB6jj+Et44D6D22KXprB89dNpyS_AAu3E8vOCtVaW1A@mail.gmail.com/\n\nI provided two workarounds you could try to use for your specific case\nat [2] and [3], one of which you ask about below.\n\n[2] https://lore.kernel.org/git/CABPp-BE=9wzF6_VypoR-uEPHsLWdV7zyE13FOgLK0h8NOcMz3g@mail.gmail.com/\n[3]  https://lore.kernel.org/git/CABPp-BH4dcsW52immJpTjgY5LjaVfKrY9MaUOnKT3byi2tBPpg@mail.gmail.com/\n\n> # --- fast-export to exported.txt\n> git clone https://github.com/simons-public/protonfixes\n> git -C protonfixes fast-export --all > exported.txt\n> # --- check revision of the repo\n> git -C protonfixes rev-parse HEAD\n> # 681411ba8ceb5d2d790e674eb7a5b98951d426e6\n>\n> # --- fast-import into new repo\n> git init newrepo\n> git -C newrepo fast-import < exported.txt\n> # --- checking revision of the new repo\n> git -C newrepo rev-parse HEAD\n> # 9888762d7857d9721f0c354e7fc187a199754a4b\n>\n> Hashes don't match. The roundtrip fails.\n\nAs expected, given that one of your commits is signed.\n\n> Let's see if --reference-excluded-parents helps.\n>\n> # --- export below produces the same export stream as above\n> git -C protonfixes fast-export --reference-excluded-parents --all >\n> exported_parents.txt\n\n--reference-excluded-parents only has effect if there are excluded\nparents.  You didn't exclude any parents, so obviously adding this\nflag isn't going to change anything.  You should instead first\nclone/fetch the part of history up to the commits you want to keep\nintact (e.g. the signed commits), and then run a command like\n   git -C protonfixes fast-export --reference-excluded-parents\n^${BASECOMMIT1} ^${BASECOMMIT2} ^${BASECOMMITN} --all\n>exported_only_newer_history.txt | git -C newrepo fast-import\n\nNote that the examples I gave you (e.g. [2] above) all used some\nexcluded references (e.g. \"^master~5\").\n\n> Because fast-import/fast-export don't work\n\nYou have not yet identified a bug in either, so I disagree with this comment.\n\n>, you propose to keep the old\n> repo around until it is clear which commits I am going to modify.\n\n\nThis statement framing looks really weird to me.  You have posed your\nproblem in the form of doing some kind of export/import operation,\nwhich is fine.  However, in order to do an export operation, you\nobviously need the repository in order to export it.  So why are you\ncalling out that you keep the repo around until you run the\nfast-export command?\n\nAnyway, that aside...\n\nI was just saying that\n  (1) signed commits exist as a method to ensure to other users that\nthe commits have not been modified\n  (2) fast-export and fast-import exist to allow you to modify history\nin some fashion (and are separate steps so people can edit the stream\nbetween running the two commands)\n  (3) the above two imply that if you still want users to be able to\nverify the signed commits, that signed commits should NOT be sent\nthrough fast-export and fast-import\n  (4) therefore, if you want the signed commits kept as-is, you should\nsimply fetch the history up to and including those, and only send the\nremainder of the history through fast-export/fast-import.\n\nBut I will add here one additional thing:\n\nIf you're weaving repositories together, that likely changes the\nparent(s) of some of the commits.  Once you change the parent(s) of a\ncommit, that alone changes the commit and invalidates any signature it\nhas.  In your case you seem to only have a root commit that is signed,\nand if you keep that signed commit as a root commit, you can avoid\nthis problem.  But, in general, if signed commits are involved in the\nweaving such that they gain new parents, then what you want to do is\nsimply impossible; you will not be able to keep the signatures in such\na case (and the commit ids will change as well).\n\n> Then\n> make a new fast-export starting from the first commit I am going to\n> modify with --reference-excluded-parents flag. Is that correct so far?\n\nYou have the basic idea, but you are making things excessively complex\nwith one detail here -- it does not need to start with the first\ncommit you are going to modify; it can start earlier.  You can simply\nexport all commits after the one(s) you know you don't want to change.\nFor example, if the history looks like this:\n\nA---B---C---D---E---F\n\nand commits A and B are the only signed commits (which you want to\npreserve) and commit D is the first one you are going to modify, you\ncould still run fast-export on \"^A ^B F\" (i.e. C, D, E, and F in this\ncase) -- that will also include C, but C isn't signed and round-trips\nwithout problems, so it doesn't hurt to include it.\n\n> Then given this partial export and old repo, how to init the new repo\n> that fast-import can apply its tail there?\n\nFlag the signed commit(s) with a branch or branches of some sort, then\nfetch just those branches into the new repo.\n\n> What if I have multiple commits that I modify, but I don't know which\n> of their parents was first?\n\nI wouldn't bother trying to figure out which one(s) is/are first.  (I\nmean, you could do some revision walking to figure that out, in which\ncase you'd have to fetch more than just the history of the signed\ncommits you want to keep but everything prior to whatever first\ncommit(s) you want to modify.)\n\nInstead, I'd just do the easier thing I noted above -- use the signed\ncommits as exclusion markers.\n\n> And when I touch commits from different\n> branches, how to recreate their parent history intact in one repo?\n\nPlace temporary branches pointing directly to each of the signed\ncommits you want to keep intact (which also implies you are keeping\nall the history behind those commits intact as well), then run:\n\ngit -C newrepo fetch PATH_OR_URL_OF_OLD_REPO ${TEMPBRANCH1}\n${TEMPBRANCH2} ${TEMPBRANCHN}\n\nThen use the earlier suggestion of\n\ngit -C protonfixes fast-export --reference-excluded-parents\n^${TEMPBRANCH1} ^${TEMPBRANCH2} ^${TEMPBRANCHN} --all\n>exported_only_newer_history.txt | git -C newrepo fast-import\n\nto get the remainder of the history exported/imported.\n\n\n\nI will also add that since you are interested in attempting to\nround-trip through fast-export/fast-import and still end up with the\nsame hashes (ignoring a few fundamental shortcomings mentioned earlier\nin this thread that won't always permit this to work), you can at\nleast get closer by adding \"--reencode=no\" to fast-export (so that it\ndoesn't alter commit messages) and setting core.ignorecase=false for\nat least the fast-import invocation (so that fast-import doesn't make\nfiles which differ in case only clob each other while importing).\nBut, again, that only addresses like two issues out of half a dozen.\nAgain, see the link at [1] earlier in this email.\n"}]}