{"thread":{"id":"56241","subject":"Working with git binary stream","startedAt":"2021-08-09T16:13:54Z","lastAt":"2021-08-10T13:58:08Z","messageCount":3,"participants":["anatoly techtonik","Jeff King","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"432310","messageId":"CAPkN8xJqqnJfdUM5fEEMA00JoKsFcqnQo--_qbCLAx1qXSrgdQ@mail.gmail.com","threadId":"56241","inReplyTo":null,"subject":"Working with git binary stream","fromName":"anatoly techtonik","fromEmail":"techtonik@gmail.com","sentAt":"2021-08-09T16:12:13Z","receivedAt":"2021-08-09T16:13:54Z","isPatch":false,"sender":{"key":"techtonik@gmail.com","avatar":"https://gravatar.com/avatar/94c82f2698c78dcbc926e549dcfcfae48be4e89c6959287fca063cd461b75af7?d=mp&s=160"},"body":"Hi.\n\nIn https://lore.kernel.org/git/CAPkN8xK7JnhatkdurEb16bC0wb+=Khd=xJ51YQUXmf2H23YCGw@mail.gmail.com/T/#u\nit became clear that it is impossible to make fast-export followed\nby fast-import to get identical commit hashes for the resulting\nrepository (try https://github.com/simons-public/protonfixes).\nIt is also impossible to detect which commits would be altered\nas a result of this operation. Because fast-export/import does\nsome implicit commit normalization, fixing that probably requires\ntoo much effort.\n\nAs an alternative it appeared that that theres is also a\n\"git binary stream\" log that is produced by\n\ngit cat-file --batch --batch-all-objects\n\nIs there a way to reconstruct the repository given that stream?\nIs there documentation on how to read it?\n-- \nanatoly t.\n"},{"id":"432335","messageId":"YRGZIjD7MioTc+Gh@coredump.intra.peff.net","threadId":"56241","inReplyTo":"CAPkN8xJqqnJfdUM5fEEMA00JoKsFcqnQo--_qbCLAx1qXSrgdQ@mail.gmail.com","subject":"Re: Working with git binary stream","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-09T21:07:46Z","receivedAt":"2021-08-09T21:07:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 09, 2021 at 07:12:13PM +0300, anatoly techtonik wrote:\n\n> As an alternative it appeared that that theres is also a\n> \"git binary stream\" log that is produced by\n> \n> git cat-file --batch --batch-all-objects\n> \n> Is there a way to reconstruct the repository given that stream?\n\nYes, though it is probably not the easiest way to do so. Just dumping\nall of the object contents back into another repository will indeed give\nyou the same hashes, etc. But if you change one object, then all its\nhash will change, and all of the other objects pointing to it will need\nto change, etc. And that dump is in apparently-random order with respect\nto the actual graph structure and relationship between objects.\n\nYou'd probably do better to build a tool around rev-list, and only use\ncat-file to fetch the verbatim object contents. At some point your tool\nwould start to look a lot like fast-export/fast-import, and it may be\nless work to teach them whatever features you need to avoid any\nnormalization (e.g., retaining signatures, encodings, etc).\n\n> Is there documentation on how to read it?\n\nThe output format is described in the \"BATCH FORMAT\" section of \"git\nhelp cat-file\". Basically you get each object id, type, and size in\nbytes, followed by the object contents. You can use the size from the\nheader to know how many bytes to read.\n\nThere's no tool to accept the whole stream. You'd have to parse each\nentry and feed it to \"git hash-object\" with the appropriate type.\n\nHaving a mode to hash-object to read in a bunch of objects in \"cat-file\n--batch\" format wouldn't be unreasonable, but nobody has found a need\nfor it so far. It would also be quite slow (it writes out individual\nloose objects, whereas something like fast-import writes out a packfile,\nincluding at least a basic attempt at deltas).\n\n-Peff\n"},{"id":"432375","messageId":"CABPp-BH4dcsW52immJpTjgY5LjaVfKrY9MaUOnKT3byi2tBPpg@mail.gmail.com","threadId":"56241","inReplyTo":"CAPkN8xJqqnJfdUM5fEEMA00JoKsFcqnQo--_qbCLAx1qXSrgdQ@mail.gmail.com","subject":"Re: Working with git binary stream","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-10T13:57:53Z","receivedAt":"2021-08-10T13:58:08Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 9, 2021 at 9:16 AM anatoly techtonik wrote:\n>\n> Hi.\n>\n> In https://lore.kernel.org/git/CAPkN8xK7JnhatkdurEb16bC0wb+=Khd=xJ51YQUXmf2H23YCGw@mail.gmail.com/T/#u\n> it became clear that it is impossible to make fast-export followed\n> by fast-import to get identical commit hashes for the resulting\n> repository (try https://github.com/simons-public/protonfixes).\n> It is also impossible to detect which commits would be altered\n> as a result of this operation. Because fast-export/import does\n> some implicit commit normalization, fixing that probably requires\n> too much effort.\n>\n> As an alternative it appeared that that theres is also a\n> \"git binary stream\" log that is produced by\n>\n> git cat-file --batch --batch-all-objects\n>\n> Is there a way to reconstruct the repository given that stream?\n> Is there documentation on how to read it?\n\nPeff already responded about hash-object. And pointed you, again, to\nthe manual for cat-file.\n\nCan I suggest an alternative, even if it changes the problem statement\nslightly? For some reason you didn't like my\n--reference-excluded-parents suggestion, but there's another way to do\nthis as well with fast-export and fast-import as they exist today: use\nfast-export's --show-original-ids flag. With that flag, you'll know\nthe original hashes. And if your filtering process does not modify a\ncommit nor any of its ancestors, it can simply omit that commit (i.e.\nnot pass it along to fast-import) and replace any references to the\ncommit with a reference to the original hash. So, for example if the\n`git fast-export --show-original-ids ...` output looked as follows (a\nsimple repository with just three commits for demonstration purposes):\n\n\"\"\"\nreset refs/heads/main\ncommit refs/heads/main\nmark :1\noriginal-oid 81b642ea15a614e84cdd52514a963735426ab06c\nauthor Developer Name 1628603376 -0400\ncommitter Developer Name 1628603376 -0400\ndata 35\nFirst commit, which was gpg signed\nM 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 fileA\n\ncommit refs/heads/main\nmark :2\noriginal-oid 0024a18e9bfef3fd1091305cef4dd5a789164809\nauthor Developer Name 1628603396 -0400\ncommitter Developer Name 1628603396 -0400\ndata 14\nSecond commit\nfrom :1\nM 100644 f2e41136eac73c39554dede1fd7e67b12502d577 fileA\nM 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 fileB\n\ncommit refs/heads/main\nmark :3\noriginal-oid 96efb1173ad5c037f03f3639976f2465b1c58186\nauthor Developer Name 1628603422 -0400\ncommitter Developer Name 1628603422 -0400\ndata 13\nThird commit\nfrom :2\nM 100644 f15bf479158b73b9bb79e158ce93d75190bc9597 fileA\nM 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 fileC\n\"\"\"\n\nThen we'd parse the first commit, decide we didn't want to filter it,\nnote that we hadn't filtered it or any of its parents, and then decide\nto replace any references to \":1\" (the stream's name for the\nreplacement for that commit) with\n\"81b642ea15a614e84cdd52514a963735426ab06c\" (the original hash).\n\nThen we'd parse the second commit. Perhaps on this one we decide we\nwant to remove fileB. So we output it after removing the fileB line,\nand after replacing \":1\" with the appropriate hash.\n\nThen we'd parse the third commit. We decide we don't want to change\nthis one, but we did change the second commit (the one with \"mark\n:2\"), so we still have to output it. There are no direct references\nto :1, so we don't need to update those either.\n\nIn the end, we'd pass this stream to fast-import:\n\n\"\"\"\nreset refs/heads/main\ncommit refs/heads/main\nmark :2\noriginal-oid 0024a18e9bfef3fd1091305cef4dd5a789164809\nauthor Developer Name 1628603396 -0400\ncommitter Developer Name 1628603396 -0400\ndata 14\nSecond commit\nfrom 81b642ea15a614e84cdd52514a963735426ab06c\nM 100644 f2e41136eac73c39554dede1fd7e67b12502d577 fileA\nM 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 fileB\n\ncommit refs/heads/main\nmark :3\noriginal-oid 96efb1173ad5c037f03f3639976f2465b1c58186\nauthor Developer Name 1628603422 -0400\ncommitter Developer Name 1628603422 -0400\ndata 13\nThird commit\nfrom :2\nM 100644 f15bf479158b73b9bb79e158ce93d75190bc9597 fileA\nM 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 fileC\n\"\"\"\n\nand it'd recover the original commit as you wanted.\n\nThis does presume that you're importing into the original repository\n(or a clone --mirror of it), because it expects certain hashes to\nalready exist. And when importing into such a repo, you want to use\n--force with fast-import. But it should do what you're asking for,\nwithout needing to do any extra work in fast-export or fast-import.\n"}]}