{"thread":{"id":"66639","subject":"[PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3","startedAt":"2026-10-23T07:25:45Z","lastAt":"2026-10-27T12:53:28Z","messageCount":17,"participants":["Christian Couder","Ben Knoble","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":13},"messages":[{"id":"555958","messageId":"20261023072559.3204273-1-christian.couder@gmail.com","threadId":"66639","inReplyTo":null,"subject":"[PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:45Z","receivedAt":"2026-10-23T07:25:45Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This series is the next part of brian m. carlson's SHA-1/SHA-256\ninteroperability work.\n\nIt adds support for reading and writing pack index v3, which stores\nobject IDs in both the main and the compatibility hash\nalgorithms. Interoperability needs it so that packed objects can be\nmapped between algorithms, the same way the loose object map already\ndoes for loose objects.\n\n## Base\n\nThis MR targets `master` at 47fdf2208c (The 7th batch, 2026-10-19).\n\n## Origin of the patches\n\nPatches 1-9 and 11-12 are the first 11 commits of brian's\n`sha256-interop-part-2` branch (https://github.com/bk2204/git.git,\nb175380036), which brian suggested for upstreaming [1].\n\nPatch 10 is a new patch I inserted to validate pack index v3\ntables.\n\nPatch 13, which copies pack index v3 documentation, comes from his\nfull `sha256-interop` branch.\n\nAll patches except patch 10 are authored by brian.\n\n[1]: https://lore.kernel.org/git/asV1oB_avuEbgRVe@fruit.crustytoothpaste.net/\n\n## Previous work and naming of this series\n\nPreviously, brian sent the following to the mailing list:\n\n - A patch series called `bc/sha1-256-interop-01` merged at 98401c10fc\n (Merge branch 'bc/sha1-256-interop-01', 2025-10-22).\n\n - A patch series called `bc/sha1-256-interop-02` merged at 8194f1795b\n (Merge branch 'bc/sha1-256-interop-02', 2026-03-12).\n\nSo this series is called \"part 3\" even though it comes mostly from\nbrian's `sha256-interop-part-2` branch.\n\n## Overview of the patches\n\n - Patches 1-5 add `odb_map_oid()` and `odb_write_oid_map()`, backed\n by two new `struct odb_source` callbacks, `map_oid()` and\n `write_oid_map()`, which the \"files\", \"loose\" and \"packed\" backends\n implement.\n\n - Patches 6-12 implement pack index v3 writing (pack-write,\n index-pack) and reading (packfile), and use it to look up compat\n object IDs in packs.\n\n - Patch 13 copies the pack index v3 format documentation into\n `gitformat-pack(5)`.\n\n## Changes compared to brian's branch\n\nMy `Signed-off-by: ...` was added to all the patches. AI assistance\n(mostly Claude Opus 5.5) was used.\n\n - Patches 1 and 4 are adapted to `ps/odb-files-alternates`. There is\n now a single `odb->source`, so `odb_map_oid()` and\n `odb_write_oid_map()` just dispatch to it. The files backend checks\n each of its object directories in turn (main directory first, then\n alternates) and writes mappings into the main directory, like it\n does for objects. The `read_alternates` callbacks are dropped, as\n the topic removed them.\n\n - In patch 4, the commit message was improved to describe how the\n \"files\" backend looks up and writes mappings.\n\n - In patch 4, the `odb/source.h` comments about the flags of the\n `map_oid()` and `write_oid_map()` callbacks have been fixed. They\n wrongly said the flags were unused.\n\n - In patch 6, the `QSORT()` that was added to\n `builtin/pack-objects.c` is dropped. It sorted `to_pack.nr_objects`\n entries while only the first `nr_written` were set, and it was\n redundant anyway, as `write_idx_file()` always sorts the array by\n object ID before returning. The comment of `write_idx_file()` and\n the commit message now say so.\n\n - In patch 6, when asked to reject duplicate objects (`--strict`),\n the v3 writer now looks for them in the arrays sorted by object ID,\n where duplicates are adjacent: it compares main object IDs in the\n array sorted by main object ID, and compatibility object IDs in the\n array sorted by compatibility object ID. (Previously, the main\n object IDs were only compared between entries that are next to each\n other in pack order, so without a compatibility algorithm, a pack\n containing objects A, B, A, B was accepted. With a compatibility\n algorithm, the check over the array sorted by compatibility object\n ID compared main object IDs, so it did not detect two different\n objects with the same compatibility object ID.)\n\n - In patch 7, `index-pack` no longer passes `OBJ_MAP_WRITE` when\n recording the compat object IDs it computes, so they stay in memory\n as the commit message and the code comment say, like in brian's\n full `sha256-interop` branch.\n\n - In patches 7 and 8, the compat hash contexts are initialized with\n `git_hash_init()` instead of the raw `init_fn()` callback, as since\n 9e396aa553 (hash: check ctx->active flag in all wrapper functions,\n 2026-07-07), `git_hash_update()` refuses contexts that were not\n initialized that way. Without this,\n `git index-pack --index-version=3` and adding a large file both hit\n a `BUG()` in a repository with a compatibility algorithm.\n\n - Patch 7 adds a test to t5308 to check that `index-pack --strict`\n rejects duplicate objects when writing a v3 index.\n\n - Patches 7, 8, 11 and 12 now also add tests to t1016 for the\n problems fixed in them, which were not covered before. They use a\n new small `create_compat_repo()` helper introduced in patch 7.\n\n - In patch 9, a few fixes have been implemented:\n\n - `bsearch_pack_v3()` now reports an object as not found, instead of\n dividing by zero, when looking it up in an algorithm for which a\n v3 index has no data.\n \n - The `munmap()` calls on errors in `load_idx()` are removed, as the\n caller already unmaps the file, like for older index versions.\n\n - The main algorithm is retrieved from the hash size instead of the\n repository, so that it works without a repository, for example in\n the fuzzer.\n\n - Patch 10 is new. It validates pack index v3 tables, mostly in\n `load_idx()`:\n\n - The header length, the trailer offset, the format entries and the\n position of all tables are checked against the file size.\n \n - Duplicate formats and invalid short name lengths are rejected.\n\n - When building the reverse index from a v3 index, the 8-byte\n offsets are now checked to lie within the index.\n\n - The positions read from the order maps are also checked against\n the number of objects when they are used: an invalid one now makes\n Git die() with a \"corrupt index?\" error, or makes a lookup report\n the object as not found, instead of reading outside the index. A\n test for this is added to t5313.\n\n - In patch 11, mapping now skips packs that don't have a v3 index,\n and `nth_packed_object_id_algop()` fails instead of returning bytes\n from the wrong table when an index cannot provide the requested\n mapping. Previously, in a compat repo, a packed object whose\n mapping was not in the loose object map got a wrong compat object\n ID.\n\n - In patch 12, looking up objects by an abbreviated compatibility\n object ID now reads the object IDs, and the offsets, at the\n positions of the table sorted by that algorithm, through\n `nth_packed_object_id_algop()` and a new\n `nth_packed_object_offset_algop()`. (Previously the main object ID\n at that position was compared against the compat prefix, so such\n lookups returned wrong or missing results.)\n\n - In patch 12, the compatibility algorithm is also skipped for packs\n with an index older than v3, which only store main object IDs. So\n when looking up a prefix in any algorithm, the callback no longer\n fires twice for the same object in such packs.\n\n - Patch 13 now says that a v3 index has 1 or 2 object formats (both\n the reader and the writer accept a single one), and it points to\n the hash function transition document for the \"Loose objects and\n unreachable objects\" section, which this document doesn't have.\n\nExcept that I added my `Signed-off-by: ...` to them, patches 2, 3 and\n5 are unchanged.\n\n## Things not yet implemented\n\n - The test suite can't run with `GIT_TEST_DEFAULT_HASH=sha256:sha1`\n yet, because accepting `:` in `GIT_DEFAULT_HASH`\n comes later in brian's series.\n\n - Index v3 is not yet automatically selected in compat repos, and\n `show-index` can't read v3 yet. Both come later in brian's series.\n\n - In a compat repo, `git index-pack` can only compute the compat\n object ID of a commit, tree or tag if the objects it references\n already have a mapping. As packs usually list commits before their\n trees and parents, indexing a pack that brings new objects, for\n example from a fetch, fails with \"unable to map tree … in commit\n object\". brian's full `sha256-interop` branch handles this in later\n commits, starting with \"builtin/index-pack: map straggling\n objects\", by converting such objects after the whole pack has been\n read, in dependency order.\n\n - Resolving an abbreviated compatibility object ID only works with\n commands like `git cat-file` or\n `git rev-parse --output-object-format=`, not with a plain\n `git rev-parse`, and the packed `find_abbrev_len()` callback\n doesn't handle compatibility object IDs yet.\n\n - A multi-pack-index only stores main object IDs, so abbreviated\n compatibility object IDs of objects in packs covered by a\n multi-pack-index are not found. The t1016 test for abbreviated\n compatibility object IDs therefore disables\n `GIT_TEST_MULTI_PACK_INDEX`.\n\n - In the \"files\" backend, `odb_map_oid()` doesn't re-read\n info/alternates before retrying. `read_object_info()` doesn't\n either on master, but Patrick's \"PATCH 0/4 odb/files: reload\n alternates on retry again\" series [2] restores that for\n it. `odb_map_oid()` should follow the same logic once that series\n is merged. I have started working on this.\n\n - No protocol support yet.\n\n[2] https://lore.kernel.org/git/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im/\n\n## Changes compared to v1\n\nThanks to Junio for his comments on this series.\n\nThe only change compared to v1 is that I added my `Signed-off-by: ...`\nto patches 2, 3 and 5 (that are otherwise unchanged compared to\nbrian's original patches).\n\n## Range-diff against brian's original work\n\n 1: 4e1f77f06a ! 1: 1f24f793f0 loose: allow looking up in object map with source\n @@ Commit message\n future.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## loose.c ##\n @@ loose.c: int repo_add_loose_object_map(struct odb_source_loose *loose,\n @@ loose.c: int repo_add_loose_object_map(struct odb_source_loose *loose,\n \t\t\t const struct git_hash_algo *to,\n \t\t\t struct object_id *dest)\n {\n - \tstruct odb_source *source;\n + \tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->source);\n -\tkh_oid_map_t *map;\n -\tkhiter_t pos;\n \n - \tfor (source = repo->objects->sources; source; source = source->next) {\n - \t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n --\t\tstruct loose_object_map *loose_map = files->loose->map;\n + \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n +-\t\tstruct loose_object_map *loose_map = dir->loose->map;\n -\t\tif (!loose_map)\n +\t\tint ret;\n +\n -+\t\tif (!files->loose)\n ++\t\tif (!dir->loose)\n \t\t\tcontinue;\n -\t\tmap = (to == repo->compat_hash_algo) ?\n -\t\t\tloose_map->to_compat :\n @@ loose.c: int repo_add_loose_object_map(struct odb_source_loose *loose,\n -\t\t\toidcpy(dest, kh_value(map, pos));\n -\t\t\treturn 0;\n -\t\t}\n -+\t\tret = source_loose_object_map_oid(files->loose, src, to, dest);\n ++\t\tret = source_loose_object_map_oid(dir->loose, src, to, dest);\n +\t\tif (ret != -1)\n +\t\t\treturn ret;\n \t}\n 2: b36496537a ! 2: 042aed1783 loose: rename repo_add_loose_object_map\n @@ Commit message\n database abstraction in a future commit.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## loose.c ##\n @@ loose.c: static int write_one_object(struct odb_source_loose *loose,\n 3: cb123eb81b ! 3: 9da3b460cb loose: add a flag to control whether we write mapped OIDs\n @@ Commit message\n so pick 1 << 6 as the write bit to avoid conflicts.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## loose.c ##\n @@ loose.c: static int write_one_object(struct odb_source_loose *loose,\n 4: 4b74282d13 ! 4: 45fe8d6076 odb: implement interoperability functions\n @@ Metadata\n ## Commit message ##\n odb: implement interoperability functions\n \n - An object database running with a compatibility algorithm needs a way to\n - map object IDs from one algorithm to another and to write mappings into\n - the repository. Add these functions and implement them for our loose\n - and packed backends so that we can use them in the future.\n + An object database running with a compatibility algorithm needs a way\n + to map object IDs from one algorithm to another and to write mappings\n + into the repository. Add these functions and implement them for the\n + \"files\" backend and for the \"loose\" and \"packed\" sources it is made of,\n + so that we can use them in the future.\n \n - Note that the packed backends don't write any information, since that\n + Like when reading objects, the \"files\" backend looks up mappings in all\n + of its object directories. It starts with the main one and continues\n + with alternates, which is what the loose object map code already does.\n +\n + If a mapping is not found in a directory, the loose object map of that\n + directory is reloaded and checked again, as another process may have\n + written new mappings in the meantime. New mappings are only written\n + into the main object directory, which is also where new objects are\n + written.\n +\n + Note that the \"packed\" sources don't write any information, since that\n will be written when we index the pack, and they don't currently read\n any information, since we don't yet have support for pack index v3,\n which we'll need to read this properly. This functionality will come\n in a future commit.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## odb.c ##\n @@ odb.c: int odb_count_objects(struct object_database *odb,\n @@ odb.c: int odb_count_objects(struct object_database *odb,\n +\t\tstruct object_id *dest,\n +\t\tenum odb_map_oid_flags flags)\n +{\n -+\tstruct odb_source *source;\n -+\tint ret;\n -+\n -+\todb_prepare_alternates(odb, NULL);\n -+\tfor (source = odb->sources; source; source = source->next) {\n -+\t\tret = odb_source_map_oid(source, src, to, dest, flags);\n -+\t\tif (ret != -1)\n -+\t\t\treturn ret;\n -+\t}\n -+\treturn -1;\n ++\treturn odb_source_map_oid(odb->source, src, to, dest, flags);\n +}\n +\n +int odb_write_oid_map(struct object_database *odb,\n @@ odb.c: int odb_count_objects(struct object_database *odb,\n +\t\t const struct object_id *compat_oid,\n +\t\t enum odb_write_oid_map_flags flags)\n +{\n -+\tstruct odb_source *source;\n -+\n -+\todb_prepare_alternates(odb, NULL);\n -+\tfor (source = odb->sources; source; source = source->next)\n -+\t\tif (!odb_source_write_oid_map(source, oid, compat_oid, flags))\n -+\t\t\treturn 0;\n -+\treturn -1;\n ++\treturn odb_source_write_oid_map(odb->source, oid, compat_oid, flags);\n +}\n ++\n /*\n * Return the slot of the most-significant bit set in \"val\". There are various\n * ways to do this quickly with fls() or __builtin_clzl(), but speed is\n @@ odb/source-files.c\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/source.h\"\n -@@ odb/source-files.c: static int odb_source_files_fsck(struct odb_source *source,\n - \treturn ret;\n +@@ odb/source-files.c: struct odb_files_dir *odb_source_files_find_dir(struct object_database *odb, con\n + \treturn dir;\n }\n \n +static int odb_source_files_map_oid(struct odb_source *source,\n @@ odb/source-files.c: static int odb_source_files_fsck(struct odb_source *source,\n +\t\t\t\t enum odb_map_oid_flags flags)\n +{\n +\tstruct odb_source_files *files = odb_source_files_downcast(source);\n -+\tint ret;\n -+\tret = odb_source_map_oid(&files->loose->base, src, to, dest, flags);\n -+\tif (ret != -1)\n -+\t\treturn ret;\n -+\tret = odb_source_map_oid(&files->packed->base, src, to, dest, flags);\n -+\tif (ret != -1)\n -+\t\treturn ret;\n -+\treturn odb_source_map_oid(&files->loose->base, src, to, dest, flags | ODB_MAP_OID_RELOAD);\n ++\n ++\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n ++\t\tint ret;\n ++\n ++\t\tret = odb_source_map_oid(&dir->loose->base, src, to, dest, flags);\n ++\t\tif (ret != -1)\n ++\t\t\treturn ret;\n ++\t\tret = odb_source_map_oid(&dir->packed->base, src, to, dest, flags);\n ++\t\tif (ret != -1)\n ++\t\t\treturn ret;\n ++\t\tret = odb_source_map_oid(&dir->loose->base, src, to, dest,\n ++\t\t\t\t\t flags | ODB_MAP_OID_RELOAD);\n ++\t\tif (ret != -1)\n ++\t\t\treturn ret;\n ++\t}\n ++\n ++\treturn -1;\n +}\n +\n +static inline int odb_source_files_write_oid_map(struct odb_source *source,\n @@ odb/source-files.c: static int odb_source_files_fsck(struct odb_source *source,\n +\t\t\t\t\t\t enum odb_write_oid_map_flags flags)\n +{\n +\tstruct odb_source_files *files = odb_source_files_downcast(source);\n -+\treturn odb_source_write_oid_map(&files->loose->base, oid, compat_oid, flags);\n ++\treturn odb_source_write_oid_map(&files->dirs->loose->base, oid, compat_oid, flags);\n +}\n +\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n - \t\t\t\t\t const char *path,\n - \t\t\t\t\t bool local)\n + \t\t\t\t\t enum odb_new_flags flags)\n + {\n @@ odb/source-files.c: struct odb_source_files *odb_source_files_new(struct object_database *odb,\n + \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n \tfiles->base.begin_transaction = odb_source_files_begin_transaction;\n - \tfiles->base.read_alternates = odb_source_files_read_alternates;\n +\tfiles->base.map_oid = odb_source_files_map_oid;\n +\tfiles->base.write_oid_map = odb_source_files_write_oid_map;\n \tfiles->base.optimize = odb_source_files_optimize;\n @@ odb/source-loose.c: static void odb_source_loose_reparent(const char *old_cwd,\n +\t * let's reload the map to see if the object has appeared.\n +\t */\n +\tif (flags & ODB_MAP_OID_RELOAD)\n -+\t\trepo_read_loose_object_map(source->odb->repo);\n ++\t\tloose_object_map_load(loose);\n +\treturn source_loose_object_map_oid(loose, src, to, dest);\n +}\n +\n @@ odb/source-loose.c: static void odb_source_loose_reparent(const char *old_cwd,\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n @@ odb/source-loose.c: struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n + \tloose->base.write_object = odb_source_loose_write_object;\n \tloose->base.write_object_stream = odb_source_loose_write_object_stream;\n \tloose->base.begin_transaction = odb_source_loose_begin_transaction;\n - \tloose->base.read_alternates = odb_source_loose_read_alternates;\n +\tloose->base.map_oid = odb_source_loose_map_oid;\n +\tloose->base.write_oid_map = odb_source_loose_write_oid_map;\n \n @@ odb/source-packed.c: static void odb_source_packed_close(struct odb_source *sour\n {\n \tstruct odb_source_packed *packed = odb_source_packed_downcast(source);\n @@ odb/source-packed.c: struct odb_source_packed *odb_source_packed_new(struct object_database *odb,\n + \tpacked->base.write_object = odb_source_packed_write_object;\n \tpacked->base.write_object_stream = odb_source_packed_write_object_stream;\n \tpacked->base.begin_transaction = odb_source_packed_begin_transaction;\n - \tpacked->base.read_alternates = odb_source_packed_read_alternates;\n +\tpacked->base.map_oid = odb_source_packed_map_oid;\n +\tpacked->base.write_oid_map = odb_source_packed_write_oid_map;\n \n @@ odb/source.h: struct odb_source {\n +\t/*\n +\t * This callback is expected to provide the mapping of an object from\n +\t * the algorithm specified in src to the algorithm specified in to,\n -+\t * leaving the result in dest. flags is required, but so far unused; it\n -+\t * should be set to 0.\n ++\t * leaving the result in dest. flags is a combination of `enum\n ++\t * odb_map_oid_flags` values: with ODB_MAP_OID_RELOAD, the source should\n ++\t * reload its mapping data before the lookup, as another process may\n ++\t * have written new mappings in the meantime.\n +\t *\n +\t * The algo field in src must point to a valid algorithm (specifically,\n +\t * it must not be zero). to must likewise point to a valid algorithm;\n @@ odb/source.h: struct odb_source {\n +\n +\t/*\n +\t * This callback is expected to write a mapping of an object between oid\n -+\t * and compat_oid. flags is required, but so far unused; it should be\n -+\t * set to 0.\n ++\t * and compat_oid. flags is a combination of `enum\n ++\t * odb_write_oid_map_flags` values: with ODB_WRITE_OID_MAP_WRITE, the\n ++\t * mapping should be persisted to disk; otherwise it may only be kept in\n ++\t * memory.\n +\t *\n +\t * The algo fields in oid and compat_oid must point to valid algorithms\n +\t * (specifically, they must not be zero).\n 5: c5781631c6 ! 5: 69dc3ffea6 object-file-convert: implement mapping in terms of object DB\n @@ Commit message\n issues.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## object-file-convert.c ##\n @@\n 6: 885edd8636 ! 6: c2123917c7 pack-write: add support for pack index v3\n @@ Commit message\n doing so will break the tests. In a future commit, once we can read it,\n we'll enable it by default when a compatibility algorithm is enabled.\n \n - Since we're not sorting the array unconditionally in write_idx_file, be\n - sure to sort it where necessary.\n + Index v3 needs the objects both in pack order and sorted by object ID,\n + so when writing a v3 index, sort copies of the array instead of the\n + array itself. Callers, like the bitmap code in pack-objects, rely on\n + the array being sorted by object ID when write_idx_file() returns,\n + though, so sort it before returning in that case too.\n \n - Signed-off-by: brian m. carlson \n + When asked to reject duplicate objects using `--strict`, look for them\n + in the arrays sorted by object ID, where duplicates are adjacent.\n \n - ## builtin/pack-objects.c ##\n -@@\n - */\n - static struct packing_data to_pack;\n - \n -+static int oid_compare(const void *_a, const void *_b)\n -+{\n -+\tstruct pack_idx_entry *a = *(struct pack_idx_entry **)_a;\n -+\tstruct pack_idx_entry *b = *(struct pack_idx_entry **)_b;\n -+\treturn oidcmp(&a->oid, &b->oid);\n -+}\n -+\n - static inline struct object_entry *oe_delta(\n - \t\tconst struct packing_data *pack,\n - \t\tconst struct object_entry *e)\n -@@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\tif (write_bitmap_index) {\n - \t\t\t\tsize_t tmpname_len = tmpname.len;\n - \n -+\t\t\t\tQSORT(written_list, to_pack.nr_objects, oid_compare);\n -+\n - \t\t\t\tstrbuf_addstr(&tmpname, \"bitmap\");\n - \t\t\t\tstop_progress(&progress_state);\n - \n + Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## pack-write.c ##\n @@ pack-write.c: void reset_pack_idx_option(struct pack_idx_option *opts)\n @@ pack-write.c: static int need_large_offset(off_t offset, const struct pack_idx_o\n - * The *sha1 contains the pack content SHA1 hash.\n - * The objects array passed in will be sorted by SHA1 on exit.\n + * The *hash contains the pack content hash.\n -+ * The objects array passed in should be in pack order and may be sorted (or\n -+ * not) on return.\n ++ * The objects array passed in should be in pack order and will be sorted by\n ++ * object ID on return.\n */\n const char *write_idx_file(struct repository *repo,\n \t\t\t const char *index_name, struct pack_idx_entry **objects,\n @@ pack-write.c: const char *write_idx_file(struct repository *repo,\n +\t\tfor (i = 0; i < nr_objects; i++) {\n +\t\t\tstruct pack_idx_entry *obj = *list++;\n +\t\t\thashwrite(f, obj->oid.hash, short_name_len[0]);\n ++\t\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n ++\t\t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n ++\t\t\t\tdie(\"The same object %s appears twice in the pack\",\n ++\t\t\t\t oid_to_hex(&obj->oid));\n +\t\t}\n +\t} else {\n +\t\t/*\n @@ pack-write.c: const char *write_idx_file(struct repository *repo,\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_idx_entry *obj = *list++;\n \t\tif (index_version < 2)\n -@@ pack-write.c: const char *write_idx_file(struct repository *repo,\n + \t\t\thashwrite_be32(f, obj->offset);\n + \t\thashwrite(f, obj->oid.hash, repo->hash_algo->rawsz);\n +-\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n ++\t\tif ((opts->flags & WRITE_IDX_STRICT) && index_version < 3 &&\n + \t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n + \t\t\tdie(\"The same object %s appears twice in the pack\",\n \t\t\t oid_to_hex(&obj->oid));\n \t}\n \n @@ pack-write.c: const char *write_idx_file(struct repository *repo,\n +\t\t\tstruct pack_idx_entry *obj = *list++;\n +\t\t\thashwrite(f, obj->compat_oid.hash, short_name_len[1]);\n +\t\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n -+\t\t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n ++\t\t\t (i && oideq(&list[-2]->compat_oid, &obj->compat_oid)))\n +\t\t\t\tdie(\"The same object %s appears twice in the pack\",\n +\t\t\t\t oid_to_hex(&obj->compat_oid));\n +\t\t}\n @@ pack-write.c: const char *write_idx_file(struct repository *repo,\n +\t\tfor (i = 0; i < nr_objects; i++) {\n +\t\t\tstruct pack_idx_entry *obj = *list++;\n +\t\t\thashwrite(f, obj->compat_oid.hash, compat->rawsz);\n -+\t\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n -+\t\t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n -+\t\t\t\tdie(\"The same object %s appears twice in the pack\",\n -+\t\t\t\t oid_to_hex(&obj->compat_oid));\n +\t\t}\n +\n +\t\t/* write the sorted-to-pack index */\n 7: fe70d52c3a ! 7: 8ced7b7a0b builtin: populate pack index v3 fields when indexing\n @@ Commit message\n and release the mutex between the conversion and writing to the map\n because hashing may be slow.\n \n + Add a test in t1016 to check that index-pack can write a v3 index in a\n + repository with a compatibility algorithm, and a test that it rejects\n + packs with duplicate objects when writing a v3 index with `--strict`.\n +\n + Also add a create_compat_repo() helper in t1016 to refactor common\n + code.\n +\n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## builtin/index-pack.c ##\n @@\n @@ builtin/index-pack.c: static void *unpack_entry_data(off_t offset, size_t size,\n \t\tgit_hash_update(&c, hdr, hdrlen);\n -\t} else\n +\t\tif (compat && compat_oid) {\n -+\t\t\tcompat->init_fn(&c_compat);\n ++\t\t\tgit_hash_init(&c_compat, compat);\n +\t\t\tif (type == OBJ_BLOB)\n -+\t\t\t\tcompat->update_fn(&c_compat, hdr, hdrlen);\n ++\t\t\t\tgit_hash_update(&c_compat, hdr, hdrlen);\n +\t\t}\n +\t} else {\n \t\toid = NULL;\n @@ builtin/index-pack.c: static void *unpack_entry_data(off_t offset, size_t size,\n +\t\t}\n +\t\tgit_hash_final_oid(compat_oid, &c_compat);\n +\t\trepo_lock();\n -+\t\todb_write_oid_map(the_repository->objects, oid, compat_oid, OBJ_MAP_WRITE);\n ++\t\todb_write_oid_map(the_repository->objects, oid, compat_oid, 0);\n +\t\trepo_unlock();\n +\t}\n \treturn buf == fixed_buf ? NULL : buf;\n @@ builtin/index-pack.c: static struct base_data *resolve_delta(struct object_entry\n +\t\trepo_lock();\n +\t\todb_write_oid_map(the_repository->objects,\n +\t\t\t\t &delta_obj->idx.oid,\n -+\t\t\t\t &delta_obj->idx.compat_oid,\n -+\t\t\t\t OBJ_MAP_WRITE);\n ++\t\t\t\t &delta_obj->idx.compat_oid, 0);\n +\t\trepo_unlock();\n +\t\tstrbuf_release(&outbuf);\n +\t}\n @@ builtin/pack-objects.c: static int option_parse_index_version(const struct optio\n \t\tdie(_(\"unsupported index version %s\"), val);\n \tif (*c == ',' && c[1])\n \t\tpopts->off32_limit = strtoul(c+1, &c, 0);\n +\n + ## t/t1016-compatObjectFormat.sh ##\n +@@ t/t1016-compatObjectFormat.sh: compat_hash () {\n + \tesac\n + }\n + \n ++create_compat_repo () {\n ++\tlocal hash=\"$1\"\n ++\tlocal repo=\"$2\"\n ++\n ++\tgit init --object-format=\"$hash\" \"$repo\" &&\n ++\tgit -C \"$repo\" config set core.repositoryformatversion 1 &&\n ++\tgit -C \"$repo\" config set extensions.compatObjectFormat $(compat_hash \"$hash\")\n ++}\n ++\n + hello_oid () {\n + \tcase \"$1\" in\n + \t\"sha1\")\n +@@ t/t1016-compatObjectFormat.sh: do\n + \t\tfor repo in alt borrow\n + \t\tdo\n + \t\t\ttest_when_finished \"rm -rf $repo\" &&\n +-\t\t\tgit init --object-format=$hash $repo &&\n +-\t\t\tgit -C $repo config set core.repositoryformatversion 1 &&\n +-\t\t\tgit -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1\n ++\t\t\tcreate_compat_repo $hash $repo || exit 1\n + \t\tdone &&\n + \n + \t\tgit -C alt commit --allow-empty --message A &&\n +@@ t/t1016-compatObjectFormat.sh: do\n + \t\tgit -C borrow rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >actual &&\n + \t\ttest_cmp expect actual\n + \t'\n ++\n ++\ttest_expect_success 'index-pack writes pack index v3' '\n ++\t\ttest_when_finished \"rm -rf v3-repo v3.idx\" &&\n ++\t\tcreate_compat_repo $hash v3-repo &&\n ++\t\techo content >v3-repo/file &&\n ++\t\tgit -C v3-repo add file &&\n ++\t\tgit -C v3-repo commit --message A &&\n ++\t\tgit -C v3-repo commit --allow-empty --message B &&\n ++\t\tgit -C v3-repo repack -a -d &&\n ++\t\tpack=$(echo v3-repo/.git/objects/pack/pack-*.pack) &&\n ++\t\tgit -C v3-repo index-pack --index-version=3 \\\n ++\t\t\t-o \"$(pwd)/v3.idx\" \"$(pwd)/$pack\" &&\n ++\t\techo \"ff 74 4f 63 00 00 00 03 \" >expect &&\n ++\t\ttest_copy_bytes 8 actual &&\n ++\t\ttest_cmp expect actual\n ++\t'\n + done\n + cd \"$base\"\n + \n +\n + ## t/t5308-pack-detect-duplicates.sh ##\n +@@ t/t5308-pack-detect-duplicates.sh: test_expect_success 'index-pack can reject packs with duplicates' '\n + \ttest_expect_code 1 git cat-file -e $LO_SHA1\n + '\n + \n ++test_expect_success 'index-pack can reject packs with duplicates with index v3' '\n ++\tclear_packs &&\n ++\tcreate_pack dups.pack 2 &&\n ++\ttest_must_fail git index-pack --strict --index-version=3 --stdin \n + Signed-off-by: Christian Couder \n \n ## object-file.c ##\n @@ object-file.c: static int hash_stream(struct odb_stream *stream,\n @@ object-file.c: static int odb_transaction_files_write_object_stream(struct odb_t\n \tgit_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);\n \tgit_hash_update(&ctx, obuf, header_len);\n +\tif (compat) {\n -+\t\tcompat->init_fn(&compat_ctx);\n ++\t\tgit_hash_init(&compat_ctx, compat);\n +\t\tgit_hash_update(&compat_ctx, obuf, header_len);\n +\t}\n \n @@ object-file.c: static int odb_transaction_files_write_object_stream(struct odb_t\n \t\tALLOC_GROW(state->written,\n \t\t\t state->nr_written + 1,\n \t\t\t state->alloc_written);\n +\n + ## t/t1016-compatObjectFormat.sh ##\n +@@ t/t1016-compatObjectFormat.sh: do\n + \t\ttest_copy_bytes 8 actual &&\n + \t\ttest_cmp expect actual\n + \t'\n ++\n ++\ttest_expect_success 'add a large file streamed into a pack' '\n ++\t\ttest_when_finished \"rm -rf large-repo\" &&\n ++\t\tcreate_compat_repo $hash large-repo &&\n ++\t\tgit -C large-repo config set core.bigFileThreshold 1k &&\n ++\t\ttest-tool genrandom large 5000 >large-repo/large &&\n ++\t\tgit -C large-repo add large &&\n ++\t\tls large-repo/.git/objects/pack/pack-*.pack &&\n ++\t\tgit -C large-repo hash-object large >expect &&\n ++\t\tgit -C large-repo rev-parse :large >actual &&\n ++\t\ttest_cmp expect actual &&\n ++\t\tgit -C large-repo cat-file blob :large >actual &&\n ++\t\ttest_cmp large-repo/large actual\n ++\t'\n + done\n + cd \"$base\"\n + \n 9: c401efada9 ! 9: 036a8fa8cc packfile: add support for reading pack index v3\n @@ Commit message\n larger. We can then use it when finding the offset into the file, since\n it will be generic over both v2 and v3 indices.\n \n + A v3 index may contain data for only one hash algorithm, so just fail if\n + we look up an object ID in an algorithm for which the index has no data.\n +\n In the pack revindex, we use the order map for the algorithm in question\n to look up the proper offset into the pack index, since we're iterating\n over objects in the offset table, which is sorted by object ID, not pack\n @@ Commit message\n formats can't represent the SHA-1 values.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## builtin/index-pack.c ##\n @@ builtin/index-pack.c: static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)\n @@ packfile.c: int load_idx(const char *path, const unsigned int hashsz, void *idx_\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n \t\tp->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));\n +\t} else if (version == 3) {\n -+\t\tuint32_t hdrlen, num_formats, trailer_offset;\n ++\t\tuint32_t hdrlen, num_formats, trailer_offset, main_algo;\n +\t\tconst uint32_t *q = index;\n +\t\tint i;\n +\n +\t\thdrlen = ntohl(*q++);\n -+\t\tif (idx_size < hdrlen) {\n -+\t\t\tmunmap(idx_map, idx_size);\n ++\t\tif (idx_size < hdrlen)\n +\t\t\treturn error(\"v3 index has truncated header\");\n -+\t\t}\n +\n +\t\tnr = ntohl(*q++);\n +\t\tnum_formats = ntohl(*q++);\n -+\t\tif (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats)) {\n -+\t\t\tmunmap(idx_map, idx_size);\n ++\t\tif (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))\n +\t\t\treturn error(\"too many formats for v3 index\");\n -+\t\t}\n +\t\tmemset(p->formats, 0, sizeof(p->formats));\n +\t\tfor (i = 0; i < num_formats; i++) {\n +\t\t\tuint32_t format_id = ntohl(*q++);\n +\t\t\tuint32_t format = hash_algo_by_id(format_id);\n +\t\t\tstruct packed_git_format *pfp;\n +\n -+\t\t\tif (format == GIT_HASH_UNKNOWN) {\n -+\t\t\t\tmunmap(idx_map, idx_size);\n ++\t\t\tif (format == GIT_HASH_UNKNOWN)\n +\t\t\t\treturn error(\"v3 index with unknown format_id 0x%08x\", format_id);\n -+\t\t\t}\n +\t\t\t/*\n +\t\t\t * We store at the value of the constant minus 1 to\n +\t\t\t * avoid having to store a useless entry at offset 0.\n @@ packfile.c: int load_idx(const char *path, const unsigned int hashsz, void *idx_\n +\t\t\tif (!i)\n +\t\t\t\tp->crc_offset = pfp->order_map_offset + nr * 4;\n +\t\t}\n -+\t\tif (!p->formats[p->repo->hash_algo - hash_algos - 1].data_offset) {\n -+\t\t\tmunmap(idx_map, idx_size);\n ++\t\tmain_algo = hash_algo_by_length(hashsz);\n ++\t\tif (main_algo == GIT_HASH_UNKNOWN ||\n ++\t\t !p->formats[main_algo - 1].data_offset)\n +\t\t\treturn error(\"v3 index with missing data for this repository's algorithm\");\n -+\t\t}\n +\t\ttrailer_offset = get_be64(q);\n -+\t\tif (trailer_offset + 2 * hashsz != idx_size) {\n -+\t\t\tmunmap(idx_map, idx_size);\n ++\t\tif (trailer_offset + 2 * hashsz != idx_size)\n +\t\t\treturn error(\"v3 index with bad size\\n\");\n -+\t\t}\n \t}\n \n \tp->index_version = version;\n @@ packfile.c: void *unpack_entry(struct repository *r, struct packed_git *p, off_t\n +\tconst struct packed_git_format *pfp = &p->formats[algo - 1];\n +\tconst unsigned char *data = p->index_data;\n +\n ++\t/* This index has no data for the algorithm of the object ID. */\n ++\tif (!pfp->data_offset) {\n ++\t\tif (result)\n ++\t\t\t*result = 0;\n ++\t\treturn 0;\n ++\t}\n ++\n +\treturn bsearch_hash_v3(oid->hash, data + pfp->full_oid_offset,\n +\t\t\t (const uint32_t *)(data + pfp->order_map_offset),\n +\t\t\t data + pfp->data_offset,\n -: ---------- > 10: 5e8aaa8329 packfile: validate pack index v3 tables\n10: cbfbb56596 ! 11: 4d575f34ed loose: map object IDs using pack index v3\n @@ Commit message\n In order to do so, let's add a variant of nth_packed_object_id that\n handles an arbitrary hash algorithm.\n \n + Only pack index v3 stores object IDs in more than one algorithm, so\n + skip packs with an older index when mapping.\n +\n + Also make the new variant fail when the index cannot provide the\n + requested object ID, for example because it has no data for one of the\n + algorithms.\n +\n + Add a test in t1016 to check that packed objects are mapped using a v3\n + index, but not using a v2 index, when the loose object map has no entry\n + for them.\n +\n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## odb/source-packed.c ##\n @@ odb/source-packed.c: static void odb_source_packed_close(struct odb_source *source)\n @@ odb/source-packed.c: static void odb_source_packed_close(struct odb_source *sour\n +\n +\t\tif (open_pack_index(p) || !p->num_objects)\n +\t\t\tcontinue;\n ++\t\t/* Only v3 indices store object IDs in both algorithms. */\n ++\t\tif (p->index_version < 3)\n ++\t\t\tcontinue;\n +\t\tif (!bsearch_pack(src, p, &off))\n +\t\t\tcontinue;\n -+\t\treturn nth_packed_object_id_algop(dest, p, off, src_algo, to);\n ++\t\tif (nth_packed_object_id_algop(dest, p, off, src_algo, to))\n ++\t\t\tcontinue;\n ++\t\treturn 0;\n +\t}\n \treturn -1;\n }\n @@ packfile.c: int bsearch_pack(const struct object_id *oid, const struct packed_gi\n \t\tif (open_pack_index(p))\n \t\t\treturn -1;\n @@ packfile.c: int nth_packed_object_id(struct object_id *oid,\n + \t}\n + \tif (n >= p->num_objects)\n + \t\treturn -1;\n ++\tif (p->index_version < 3 && (src != dest || dest != p->repo->hash_algo))\n ++\t\treturn -1; /* older indices only store main object IDs */\n \tif (p->index_version == 1) {\n \t\tindex += 4 * 256;\n \t\toidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),\n @@ packfile.c: int nth_packed_object_id(struct object_id *oid,\n +\t\tint algo = hash_algo_by_ptr(dest);\n \t\tconst struct packed_git_format *pfp = &p->formats[algo-1];\n -\t\toidread(oid, index + pfp->full_oid_offset + (hashsz * n), p->repo->hash_algo);\n -+\t\tuint32_t pack_order_idx = nth_packed_object_pack_order_algop(p, n, src);\n ++\t\tuint32_t pack_order_idx;\n ++\n ++\t\tif (!pfp->data_offset ||\n ++\t\t !p->formats[hash_algo_by_ptr(src) - 1].data_offset)\n ++\t\t\treturn -1; /* no data for one of the algorithms */\n ++\t\tpack_order_idx = nth_packed_object_pack_order_algop(p, n, src);\n +\t\toidread(oid, index + st_add(pfp->full_oid_offset, st_mult(hashsz, pack_order_idx)), dest);\n \t}\n \treturn 0;\n @@ packfile.h: int bsearch_pack(const struct object_id *oid, const struct packed_gi\n /*\n * Return the offset of the nth object within the specified packfile.\n * The index must already be opened.\n +\n + ## t/t1016-compatObjectFormat.sh ##\n +@@ t/t1016-compatObjectFormat.sh: do\n + \t\tgit -C large-repo cat-file blob :large >actual &&\n + \t\ttest_cmp large-repo/large actual\n + \t'\n ++\n ++\ttest_expect_success 'packed objects are only mapped using pack index v3' '\n ++\t\ttest_when_finished \"rm -rf map-repo v3.idx loose-object-idx\" &&\n ++\t\tcreate_compat_repo $hash map-repo &&\n ++\t\techo content >map-repo/file &&\n ++\t\tgit -C map-repo add file &&\n ++\t\tgit -C map-repo commit --message A &&\n ++\t\toid=$(git -C map-repo rev-parse HEAD) &&\n ++\t\tgit -C map-repo rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >expect &&\n ++\t\tgit -C map-repo repack -a -d &&\n ++\t\tpack=$(echo map-repo/.git/objects/pack/pack-*.pack) &&\n ++\t\tgit -C map-repo index-pack --index-version=3 \\\n ++\t\t\t-o \"$(pwd)/v3.idx\" \"$(pwd)/$pack\" &&\n ++\t\tmv map-repo/.git/objects/loose-object-idx loose-object-idx &&\n ++\n ++\t\t# A v2 index only has object IDs in the main algorithm, so it\n ++\t\t# must not provide a mapping.\n ++\t\tgit -C map-repo rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >actual &&\n ++\t\t! test_cmp expect actual &&\n ++\t\ttest $(wc -c actual &&\n ++\t\ttest_cmp expect actual\n ++\t'\n + done\n + cd \"$base\"\n + \n11: f60e1e2c0d ! 12: ce5de5c8c5 packfile: allow looking up compat object IDs in packs\n @@ Commit message\n it is easier to understand as two statements. The compiler will\n optimize it appropriately.\n \n + When looking up an object ID, we find positions in the table sorted by\n + that object ID's algorithm. So read the object IDs there for that\n + algorithm with nth_packed_object_id_algop(), and add\n + nth_packed_object_offset_algop() to get their offsets.\n +\n + Skip the compatibility algorithm for packs with an index older than v3,\n + as these only store object IDs in the main algorithm.\n +\n + Add a test that looks up an abbreviated compatibility object ID of a\n + packed object.\n +\n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## odb/source-packed.c ##\n @@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(\n @@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(\n +\t\t (!p->repo->compat_hash_algo ||\n +\t\t algop != p->repo->compat_hash_algo))\n +\t\t\tcontinue;\n -+\n ++\t\tif (p->index_version < 3 && algop != p->repo->hash_algo)\n ++\t\t\tcontinue;\n +\n +\t\toidcpy(&wanted, opts->prefix);\n +\t\twanted.algo = algo;\n @@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(\n +\t\tfor (i = first; i < num; i++) {\n +\t\t\tstruct object_id oid;\n +\n -+\t\t\tnth_packed_object_id(&oid, p, i);\n -+\t\t\tif (!match_hash(len, opts->prefix->hash, oid.hash))\n ++\t\t\tif (nth_packed_object_id_algop(&oid, p, i, algop, algop) ||\n ++\t\t\t !match_hash(len, opts->prefix->hash, oid.hash))\n +\t\t\t\tbreak;\n +\n +\t\t\tif (data->request) {\n +\t\t\t\tstruct object_info oi = *data->request;\n -+\t\t\t\toff_t offset = nth_packed_object_offset(p, i);\n ++\t\t\t\toff_t offset = nth_packed_object_offset_algop(p, i, algop);\n +\n +\t\t\t\tret = packed_object_info(source, p, offset, &oi);\n +\t\t\t\tif (ret)\n @@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(\n \t\t}\n \t}\n \n +\n + ## packfile.c ##\n +@@ packfile.c: void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n + }\n + \n + off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n ++{\n ++\treturn nth_packed_object_offset_algop(p, n, p->repo->hash_algo);\n ++}\n ++\n ++off_t nth_packed_object_offset_algop(const struct packed_git *p, uint32_t n,\n ++\t\t\t\t const struct git_hash_algo *algop)\n + {\n + \tconst unsigned char *index = p->index_data;\n + \tconst unsigned int hashsz = p->repo->hash_algo->rawsz;\n ++\n ++\tif (p->index_version < 3 && algop != p->repo->hash_algo)\n ++\t\tBUG(\"pack index v%d only stores main object IDs\",\n ++\t\t p->index_version);\n ++\n + \tif (p->index_version == 1) {\n + \t\tindex += 4 * 256;\n + \t\treturn ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));\n + \t} else {\n + \t\tuint32_t off;\n + \t\tif (p->index_version == 3)\n +-\t\t\tn = nth_packed_object_pack_order_algop(p, n, p->repo->hash_algo);\n ++\t\t\tn = nth_packed_object_pack_order_algop(p, n, algop);\n + \t\tindex += st_add(p->crc_offset, st_mult(p->num_objects, 4));\n + \t\toff = ntohl(*((uint32_t *)(index + st_mult(4, n))));\n + \t\tif (!(off & 0x80000000))\n +\n + ## packfile.h ##\n +@@ packfile.h: int nth_packed_object_id_algop(struct object_id *, struct packed_git *,\n + */\n + off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n + \n ++/*\n ++ * The same as nth_packed_object_offset, but the nth object is counted in\n ++ * algop. For indices older than v3, algop must be the main algorithm.\n ++ */\n ++off_t nth_packed_object_offset_algop(const struct packed_git *, uint32_t n,\n ++\t\t\t\t const struct git_hash_algo *algop);\n ++\n + /*\n + * Return the index into pack order of the nth object within the specified\n + * packfile. The index must already be opened and it must be a v3 index.\n +\n + ## t/t1016-compatObjectFormat.sh ##\n +@@ t/t1016-compatObjectFormat.sh: do\n + \t\tgit -C map-repo rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >actual &&\n + \t\ttest_cmp expect actual\n + \t'\n ++\n ++\ttest_expect_success 'abbreviated compat object IDs are found using pack index v3' '\n ++\t\ttest_when_finished \"rm -rf abbrev-repo v3.idx\" &&\n ++\t\tcreate_compat_repo $hash abbrev-repo &&\n ++\t\techo content >abbrev-repo/file &&\n ++\t\tgit -C abbrev-repo add file &&\n ++\t\tgit -C abbrev-repo commit --message A &&\n ++\t\tcompat_oid=$(git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) HEAD) &&\n ++\t\tabbrev=$(echo $compat_oid | cut -c 1-12) &&\n ++\n ++\t\t# A multi-pack-index only stores object IDs in the main\n ++\t\t# algorithm.\n ++\t\tGIT_TEST_MULTI_PACK_INDEX=0 git -C abbrev-repo repack -a -d &&\n ++\t\tpack=$(echo abbrev-repo/.git/objects/pack/pack-*.pack) &&\n ++\t\tgit -C abbrev-repo index-pack --index-version=3 \\\n ++\t\t\t-o \"$(pwd)/v3.idx\" \"$(pwd)/$pack\" &&\n ++\t\trm abbrev-repo/.git/objects/loose-object-idx &&\n ++\n ++\t\t# A v2 index only has object IDs in the main algorithm.\n ++\t\ttest_must_fail git -C abbrev-repo cat-file -t $abbrev &&\n ++\n ++\t\trm -f \"${pack%.pack}.idx\" \"${pack%.pack}.rev\" &&\n ++\t\tmv v3.idx \"${pack%.pack}.idx\" &&\n ++\t\techo commit >expect &&\n ++\t\tgit -C abbrev-repo cat-file -t $abbrev >actual &&\n ++\t\ttest_cmp expect actual &&\n ++\t\techo $compat_oid >expect &&\n ++\t\tgit -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) $abbrev >actual &&\n ++\t\ttest_cmp expect actual\n ++\t'\n + done\n + cd \"$base\"\n + \n12: 6afaf2ff9e ! 13: 50f515b9aa doc: move pack index v3 documentation\n @@ Commit message\n compatibility.\n \n Signed-off-by: brian m. carlson \n + Signed-off-by: Christian Couder \n \n ## Documentation/gitformat-pack.adoc ##\n @@ Documentation/gitformat-pack.adoc: Pack file entry: <+\n @@ Documentation/gitformat-pack.adoc: Pack file entry: <+\n + * 4-byte length of the header section, including the signature and\n + version number\n + * 4-byte number of objects contained in the pack\n -+ * 4-byte number of object formats in this pack index: 2\n ++ * 4-byte number of object formats in this pack index: 1 or 2\n + * For each object format:\n + ** 4-byte format identifier (e.g., 'sha1' for SHA-1)\n + ** 4-byte length in bytes of shortened object names. This is the\n @@ Documentation/gitformat-pack.adoc: Pack file entry: <+\n + * 8-byte offset to the trailer from the beginning of this file.\n + * Zero or more additional key/value pairs (4-byte key, 4-byte\n + value). Only one key is supported: 'PSRC'. See the \"Loose objects\n -+ and unreachable objects\" section for supported values and how this\n -+ is used. All other keys are reserved. Readers must ignore\n ++ and unreachable objects\" section of the hash function transition\n ++ document (`Documentation/technical/hash-function-transition.adoc`)\n ++ for supported values and how this is used. All other keys are reserved. Readers must ignore\n + unrecognized keys.\n + - Zero or more NUL bytes. This can optionally be used to improve the\n + alignment of the full object name table below.\n\n\nChristian Couder (1):\n packfile: validate pack index v3 tables\n\nbrian m. carlson (12):\n loose: allow looking up in object map with source\n loose: rename repo_add_loose_object_map\n loose: add a flag to control whether we write mapped OIDs\n odb: implement interoperability functions\n object-file-convert: implement mapping in terms of object DB\n pack-write: add support for pack index v3\n builtin: populate pack index v3 fields when indexing\n object-file: handle compatibility algorithm\n packfile: add support for reading pack index v3\n loose: map object IDs using pack index v3\n packfile: allow looking up compat object IDs in packs\n doc: move pack index v3 documentation\n\n Documentation/gitformat-pack.adoc | 75 +++++++++\n builtin/index-pack.c | 107 +++++++++++-\n builtin/pack-objects.c | 8 +-\n hash-lookup.c | 50 +++++-\n hash-lookup.h | 19 ++-\n loose.c | 50 ++++--\n loose.h | 15 +-\n object-file-convert.c | 14 +-\n object-file.c | 18 +-\n odb.c | 17 ++\n odb.h | 24 +++\n odb/source-files.c | 38 +++++\n odb/source-loose.c | 31 +++-\n odb/source-packed.c | 124 ++++++++++----\n odb/source.h | 65 ++++++++\n pack-check.c | 4 +-\n pack-revindex.c | 24 ++-\n pack-write.c | 263 ++++++++++++++++++++++++++----\n pack.h | 2 +\n packfile.c | 185 +++++++++++++++++++--\n packfile.h | 33 ++++\n t/t1016-compatObjectFormat.sh | 100 +++++++++++-\n t/t5302-pack-index.sh | 71 ++++----\n t/t5308-pack-detect-duplicates.sh | 7 +\n t/t5313-pack-bounds-checks.sh | 17 ++\n 25 files changed, 1193 insertions(+), 168 deletions(-)\n\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555959","messageId":"20261023072559.3204273-2-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 01/13] loose: allow looking up in object map with source","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:46Z","receivedAt":"2026-10-23T07:25:46Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nIn preparation for wiring up object mapping into the ODB abstraction,\nadd a function which can map object IDs based on a struct\nodb_source_loose and use it in the existing function. This preserves\nthe same behavior, but allows us to make use of it elsewhere in the\nfuture.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n loose.c | 41 +++++++++++++++++++++++++++++------------\n loose.h | 5 +++++\n 2 files changed, 34 insertions(+), 12 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex d6a2bb5ba3..fe8f540979 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -225,27 +225,44 @@ int repo_add_loose_object_map(struct odb_source_loose *loose,\n \treturn 0;\n }\n \n+int source_loose_object_map_oid(struct odb_source_loose *loose,\n+\t\t\t\tconst struct object_id *src,\n+\t\t\t\tconst struct git_hash_algo *to,\n+\t\t\t\tstruct object_id *dest)\n+{\n+\tkh_oid_map_t *map;\n+\tkhiter_t pos;\n+\tstruct loose_object_map *loose_map = loose->map;\n+\tstruct repository *repo = loose->base.odb->repo;\n+\n+\tif (!loose_map)\n+\t\treturn -1;\n+\tmap = (to == repo->compat_hash_algo) ?\n+\t\tloose_map->to_compat :\n+\t\tloose_map->to_storage;\n+\tpos = kh_get_oid_map(map, *src);\n+\tif (pos < kh_end(map)) {\n+\t\toidcpy(dest, kh_value(map, pos));\n+\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n int repo_loose_object_map_oid(struct repository *repo,\n \t\t\t const struct object_id *src,\n \t\t\t const struct git_hash_algo *to,\n \t\t\t struct object_id *dest)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->source);\n-\tkh_oid_map_t *map;\n-\tkhiter_t pos;\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tstruct loose_object_map *loose_map = dir->loose->map;\n-\t\tif (!loose_map)\n+\t\tint ret;\n+\n+\t\tif (!dir->loose)\n \t\t\tcontinue;\n-\t\tmap = (to == repo->compat_hash_algo) ?\n-\t\t\tloose_map->to_compat :\n-\t\t\tloose_map->to_storage;\n-\t\tpos = kh_get_oid_map(map, *src);\n-\t\tif (pos < kh_end(map)) {\n-\t\t\toidcpy(dest, kh_value(map, pos));\n-\t\t\treturn 0;\n-\t\t}\n+\t\tret = source_loose_object_map_oid(dir->loose, src, to, dest);\n+\t\tif (ret != -1)\n+\t\t\treturn ret;\n \t}\n \treturn -1;\n }\ndiff --git a/loose.h b/loose.h\nindex ed663ac550..f3b1e47363 100644\n--- a/loose.h\n+++ b/loose.h\n@@ -24,4 +24,9 @@ int repo_add_loose_object_map(struct odb_source_loose *loose,\n int repo_read_loose_object_map(struct repository *repo);\n int repo_write_loose_object_map(struct repository *repo);\n \n+int source_loose_object_map_oid(struct odb_source_loose *loose,\n+\t\t\t\tconst struct object_id *src,\n+\t\t\t\tconst struct git_hash_algo *to,\n+\t\t\t\tstruct object_id *dest);\n+\n #endif\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555960","messageId":"20261023072559.3204273-3-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 02/13] loose: rename repo_add_loose_object_map","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:47Z","receivedAt":"2026-10-23T07:25:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nThe pointer it takes is not a struct repository, but a struct\nodb_source_loose. Let's rename this and we'll use it in our object\ndatabase abstraction in a future commit.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n loose.c | 6 +++---\n loose.h | 6 +++---\n odb/source-loose.c | 4 ++--\n 3 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex fe8f540979..8d7ea27448 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -210,9 +210,9 @@ static int write_one_object(struct odb_source_loose *loose,\n \treturn -1;\n }\n \n-int repo_add_loose_object_map(struct odb_source_loose *loose,\n-\t\t\t const struct object_id *oid,\n-\t\t\t const struct object_id *compat_oid)\n+int source_add_loose_object_map(struct odb_source_loose *loose,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tconst struct object_id *compat_oid)\n {\n \tint inserted = 0;\n \ndiff --git a/loose.h b/loose.h\nindex f3b1e47363..6f0c8554ca 100644\n--- a/loose.h\n+++ b/loose.h\n@@ -18,9 +18,6 @@ int repo_loose_object_map_oid(struct repository *repo,\n \t\t\t const struct object_id *src,\n \t\t\t const struct git_hash_algo *dest_algo,\n \t\t\t struct object_id *dest);\n-int repo_add_loose_object_map(struct odb_source_loose *loose,\n-\t\t\t const struct object_id *oid,\n-\t\t\t const struct object_id *compat_oid);\n int repo_read_loose_object_map(struct repository *repo);\n int repo_write_loose_object_map(struct repository *repo);\n \n@@ -28,5 +25,8 @@ int source_loose_object_map_oid(struct odb_source_loose *loose,\n \t\t\t\tconst struct object_id *src,\n \t\t\t\tconst struct git_hash_algo *to,\n \t\t\t\tstruct object_id *dest);\n+int source_add_loose_object_map(struct odb_source_loose *loose,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tconst struct object_id *compat_oid);\n \n #endif\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 0664634938..3c0c85f3e3 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -848,7 +848,7 @@ static int odb_source_loose_write_object(struct odb_source *source,\n \t\treturn -1;\n \n \tif (compat_oid)\n-\t\treturn repo_add_loose_object_map(loose, oid, compat_oid);\n+\t\treturn source_add_loose_object_map(loose, oid, compat_oid);\n \n \treturn 0;\n }\n@@ -970,7 +970,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \terr = finalize_object_file_flags(loose->base.odb->repo, tmp_file.buf, filename.buf,\n \t\t\t\t\t FOF_SKIP_COLLISION_CHECK);\n \tif (!err && compat)\n-\t\terr = repo_add_loose_object_map(loose, oid, &compat_oid);\n+\t\terr = source_add_loose_object_map(loose, oid, &compat_oid);\n cleanup:\n \tstrbuf_release(&tmp_file);\n \tstrbuf_release(&filename);\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555961","messageId":"20261023072559.3204273-4-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 03/13] loose: add a flag to control whether we write mapped OIDs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:48Z","receivedAt":"2026-10-23T07:25:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nRight now, we always write the object to the loose object map. However,\nin the future, we'll need to temporarily store some object ID mappings\nin memory so that we can rewrite other objects, such as when we're\nindexing packs. Let's make this easier by setting a flag indicating\nwhether we want to write this to disk.\n\nThe bottom six bits will be used for an object type in a future commit,\nso pick 1 << 6 as the write bit to avoid conflicts.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n loose.c | 5 +++--\n loose.h | 6 +++++-\n odb/source-loose.c | 4 ++--\n 3 files changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex 8d7ea27448..090681b4b3 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -212,7 +212,8 @@ static int write_one_object(struct odb_source_loose *loose,\n \n int source_add_loose_object_map(struct odb_source_loose *loose,\n \t\t\t\tconst struct object_id *oid,\n-\t\t\t\tconst struct object_id *compat_oid)\n+\t\t\t\tconst struct object_id *compat_oid,\n+\t\t\t\tint flags)\n {\n \tint inserted = 0;\n \n@@ -220,7 +221,7 @@ int source_add_loose_object_map(struct odb_source_loose *loose,\n \t\treturn 0;\n \n \tinserted = insert_loose_map(loose, oid, compat_oid);\n-\tif (inserted)\n+\tif (inserted && (flags & OBJ_MAP_WRITE))\n \t\treturn write_one_object(loose, oid, compat_oid);\n \treturn 0;\n }\ndiff --git a/loose.h b/loose.h\nindex 6f0c8554ca..19d3be26a3 100644\n--- a/loose.h\n+++ b/loose.h\n@@ -3,6 +3,9 @@\n \n #include \"khash.h\"\n \n+/* Should we write this object to disk? */\n+#define OBJ_MAP_WRITE (1 << 6)\n+\n struct repository;\n struct odb_source_loose;\n \n@@ -27,6 +30,7 @@ int source_loose_object_map_oid(struct odb_source_loose *loose,\n \t\t\t\tstruct object_id *dest);\n int source_add_loose_object_map(struct odb_source_loose *loose,\n \t\t\t\tconst struct object_id *oid,\n-\t\t\t\tconst struct object_id *compat_oid);\n+\t\t\t\tconst struct object_id *compat_oid,\n+\t\t\t\tint flags);\n \n #endif\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 3c0c85f3e3..b58184f31d 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -848,7 +848,7 @@ static int odb_source_loose_write_object(struct odb_source *source,\n \t\treturn -1;\n \n \tif (compat_oid)\n-\t\treturn source_add_loose_object_map(loose, oid, compat_oid);\n+\t\treturn source_add_loose_object_map(loose, oid, compat_oid, OBJ_MAP_WRITE);\n \n \treturn 0;\n }\n@@ -970,7 +970,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \terr = finalize_object_file_flags(loose->base.odb->repo, tmp_file.buf, filename.buf,\n \t\t\t\t\t FOF_SKIP_COLLISION_CHECK);\n \tif (!err && compat)\n-\t\terr = source_add_loose_object_map(loose, oid, &compat_oid);\n+\t\terr = source_add_loose_object_map(loose, oid, &compat_oid, OBJ_MAP_WRITE);\n cleanup:\n \tstrbuf_release(&tmp_file);\n \tstrbuf_release(&filename);\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555962","messageId":"20261023072559.3204273-5-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 04/13] odb: implement interoperability functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:49Z","receivedAt":"2026-10-23T07:25:49Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nAn object database running with a compatibility algorithm needs a way\nto map object IDs from one algorithm to another and to write mappings\ninto the repository. Add these functions and implement them for the\n\"files\" backend and for the \"loose\" and \"packed\" sources it is made of,\nso that we can use them in the future.\n\nLike when reading objects, the \"files\" backend looks up mappings in all\nof its object directories. It starts with the main one and continues\nwith alternates, which is what the loose object map code already does.\n\nIf a mapping is not found in a directory, the loose object map of that\ndirectory is reloaded and checked again, as another process may have\nwritten new mappings in the meantime. New mappings are only written\ninto the main object directory, which is also where new objects are\nwritten.\n\nNote that the \"packed\" sources don't write any information, since that\nwill be written when we index the pack, and they don't currently read\nany information, since we don't yet have support for pack index v3,\nwhich we'll need to read this properly. This functionality will come\nin a future commit.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n odb.c | 17 ++++++++++++\n odb.h | 24 +++++++++++++++++\n odb/source-files.c | 38 ++++++++++++++++++++++++++\n odb/source-loose.c | 27 +++++++++++++++++++\n odb/source-packed.c | 19 +++++++++++++\n odb/source.h | 65 +++++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 190 insertions(+)\n\ndiff --git a/odb.c b/odb.c\nindex 98c5c862f5..a0a1be7c7e 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -590,6 +590,23 @@ int odb_count_objects(struct object_database *odb,\n \treturn ret;\n }\n \n+int odb_map_oid(struct object_database *odb,\n+\t\tconst struct object_id *src,\n+\t\tconst struct git_hash_algo *to,\n+\t\tstruct object_id *dest,\n+\t\tenum odb_map_oid_flags flags)\n+{\n+\treturn odb_source_map_oid(odb->source, src, to, dest, flags);\n+}\n+\n+int odb_write_oid_map(struct object_database *odb,\n+\t\t const struct object_id *oid,\n+\t\t const struct object_id *compat_oid,\n+\t\t enum odb_write_oid_map_flags flags)\n+{\n+\treturn odb_source_write_oid_map(odb->source, oid, compat_oid, flags);\n+}\n+\n /*\n * Return the slot of the most-significant bit set in \"val\". There are various\n * ways to do this quickly with fls() or __builtin_clzl(), but speed is\ndiff --git a/odb.h b/odb.h\nindex 0ccc47ee73..70af2c7c78 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -2,6 +2,7 @@\n #define ODB_H\n \n #include \"hashmap.h\"\n+#include \"loose.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n #include \"oidset.h\"\n@@ -649,6 +650,29 @@ static inline int odb_write_object(struct object_database *odb,\n \n struct odb_stream;\n \n+enum odb_map_oid_flags {\n+\t/* Reload this source to obtain any new information. */\n+\tODB_MAP_OID_RELOAD = (1 << 0),\n+};\n+\n+int odb_map_oid(struct object_database *odb,\n+\t\tconst struct object_id *src,\n+\t\tconst struct git_hash_algo *to,\n+\t\tstruct object_id *dest,\n+\t\tenum odb_map_oid_flags flags);\n+\n+enum odb_write_oid_map_flags {\n+\t/*\n+\t * Serialize this data to disk instead of persisting it only in memory.\n+\t */\n+\tODB_WRITE_OID_MAP_WRITE = OBJ_MAP_WRITE,\n+};\n+\n+int odb_write_oid_map(struct object_database *odb,\n+\t\t const struct object_id *oid,\n+\t\t const struct object_id *compat_oid,\n+\t\t enum odb_write_oid_map_flags flags);\n+\n int odb_write_object_stream(struct object_database *odb,\n \t\t\t struct odb_stream *stream,\n \t\t\t struct object_id *oid);\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 455f409db0..4f4f7f1f87 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -6,6 +6,7 @@\n #include \"gettext.h\"\n #include \"hex.h\"\n #include \"lockfile.h\"\n+#include \"loose.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/source.h\"\n@@ -1234,6 +1235,41 @@ struct odb_files_dir *odb_source_files_find_dir(struct object_database *odb, con\n \treturn dir;\n }\n \n+static int odb_source_files_map_oid(struct odb_source *source,\n+\t\t\t\t const struct object_id *src,\n+\t\t\t\t const struct git_hash_algo *to,\n+\t\t\t\t struct object_id *dest,\n+\t\t\t\t enum odb_map_oid_flags flags)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tint ret;\n+\n+\t\tret = odb_source_map_oid(&dir->loose->base, src, to, dest, flags);\n+\t\tif (ret != -1)\n+\t\t\treturn ret;\n+\t\tret = odb_source_map_oid(&dir->packed->base, src, to, dest, flags);\n+\t\tif (ret != -1)\n+\t\t\treturn ret;\n+\t\tret = odb_source_map_oid(&dir->loose->base, src, to, dest,\n+\t\t\t\t\t flags | ODB_MAP_OID_RELOAD);\n+\t\tif (ret != -1)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static inline int odb_source_files_write_oid_map(struct odb_source *source,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t const struct object_id *compat_oid,\n+\t\t\t\t\t\t enum odb_write_oid_map_flags flags)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\treturn odb_source_write_oid_map(&files->dirs->loose->base, oid, compat_oid, flags);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t enum odb_new_flags flags)\n {\n@@ -1271,6 +1307,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n \tfiles->base.begin_transaction = odb_source_files_begin_transaction;\n+\tfiles->base.map_oid = odb_source_files_map_oid;\n+\tfiles->base.write_oid_map = odb_source_files_write_oid_map;\n \tfiles->base.optimize = odb_source_files_optimize;\n \tfiles->base.optimize_required = odb_source_files_optimize_required;\n \tfiles->base.generate_pack = odb_source_files_generate_pack;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex b58184f31d..22c46bd64a 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -1017,6 +1017,31 @@ static void odb_source_loose_reparent(const char *old_cwd,\n \tloose->base.path = path;\n }\n \n+static int odb_source_loose_map_oid(struct odb_source *source, const struct object_id *src,\n+\t\t\t\t const struct git_hash_algo *to, struct object_id *dest,\n+\t\t\t\t enum odb_map_oid_flags flags)\n+{\n+\tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n+\t/*\n+\t * We may have loaded the object map at repo initialization but\n+\t * another process (perhaps upstream of a pipe from us) may have\n+\t * written a new object into the map. If the object is missing,\n+\t * let's reload the map to see if the object has appeared.\n+\t */\n+\tif (flags & ODB_MAP_OID_RELOAD)\n+\t\tloose_object_map_load(loose);\n+\treturn source_loose_object_map_oid(loose, src, to, dest);\n+}\n+\n+static inline int odb_source_loose_write_oid_map(struct odb_source *source,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t const struct object_id *compat_oid,\n+\t\t\t\t\t\t enum odb_write_oid_map_flags flags)\n+{\n+\tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n+\treturn source_add_loose_object_map(loose, oid, compat_oid, flags);\n+}\n+\n static void odb_source_loose_free(struct odb_source *source)\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n@@ -1139,6 +1164,8 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n \tloose->base.write_object = odb_source_loose_write_object;\n \tloose->base.write_object_stream = odb_source_loose_write_object_stream;\n \tloose->base.begin_transaction = odb_source_loose_begin_transaction;\n+\tloose->base.map_oid = odb_source_loose_map_oid;\n+\tloose->base.write_oid_map = odb_source_loose_write_oid_map;\n \n \tif (!is_absolute_path(loose->base.path))\n \t\tchdir_notify_register(odb_source_loose_reparent, loose);\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 2226fbd313..ee1edfdb2b 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -863,6 +863,23 @@ static void odb_source_packed_close(struct odb_source *source)\n \tpacked->midx = NULL;\n }\n \n+static int odb_source_packed_map_oid(struct odb_source *source UNUSED,\n+\t\t\t\t const struct object_id *src UNUSED,\n+\t\t\t\t const struct git_hash_algo *to UNUSED,\n+\t\t\t\t struct object_id *dest UNUSED,\n+\t\t\t\t enum odb_map_oid_flags flags UNUSED)\n+{\n+\treturn -1;\n+}\n+\n+static inline int odb_source_packed_write_oid_map(struct odb_source *source UNUSED,\n+\t\t\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t\t\t const struct object_id *compat_oid UNUSED,\n+\t\t\t\t\t\t enum odb_write_oid_map_flags flags UNUSED)\n+{\n+\treturn -1;\n+}\n+\n static void odb_source_packed_free(struct odb_source *source)\n {\n \tstruct odb_source_packed *packed = odb_source_packed_downcast(source);\n@@ -1013,6 +1030,8 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,\n \tpacked->base.write_object = odb_source_packed_write_object;\n \tpacked->base.write_object_stream = odb_source_packed_write_object_stream;\n \tpacked->base.begin_transaction = odb_source_packed_begin_transaction;\n+\tpacked->base.map_oid = odb_source_packed_map_oid;\n+\tpacked->base.write_oid_map = odb_source_packed_write_oid_map;\n \n \tif (!is_absolute_path(path))\n \t\tchdir_notify_register(odb_source_packed_reparent, packed);\ndiff --git a/odb/source.h b/odb/source.h\nindex 1867347dfc..e1f01359ce 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -300,6 +300,42 @@ struct odb_source {\n \t */\n \tint (*fsck)(struct odb_source *source,\n \t\t struct odb_fsck_options *options);\n+\n+\t/*\n+\t * This callback is expected to provide the mapping of an object from\n+\t * the algorithm specified in src to the algorithm specified in to,\n+\t * leaving the result in dest. flags is a combination of `enum\n+\t * odb_map_oid_flags` values: with ODB_MAP_OID_RELOAD, the source should\n+\t * reload its mapping data before the lookup, as another process may\n+\t * have written new mappings in the meantime.\n+\t *\n+\t * The algo field in src must point to a valid algorithm (specifically,\n+\t * it must not be zero). to must likewise point to a valid algorithm;\n+\t * it must be non-NULL.\n+\t *\n+\t * The callback is expected to return 0 on success, -1 if the object is\n+\t * not found, and a different negative error code on error.\n+\t */\n+\tint (*map_oid)(struct odb_source *source, const struct object_id *src,\n+\t\t const struct git_hash_algo *to, struct object_id *dest,\n+\t\t enum odb_map_oid_flags flags);\n+\n+\t/*\n+\t * This callback is expected to write a mapping of an object between oid\n+\t * and compat_oid. flags is a combination of `enum\n+\t * odb_write_oid_map_flags` values: with ODB_WRITE_OID_MAP_WRITE, the\n+\t * mapping should be persisted to disk; otherwise it may only be kept in\n+\t * memory.\n+\t *\n+\t * The algo fields in oid and compat_oid must point to valid algorithms\n+\t * (specifically, they must not be zero).\n+\t *\n+\t * The callback is expected to return 0 on success, and a negative\n+\t * error code on error.\n+\t */\n+\tint (*write_oid_map)(struct odb_source *source, const struct object_id *oid,\n+\t\t\t const struct object_id *compat_oid,\n+\t\t\t enum odb_write_oid_map_flags flags);\n };\n \n /*\n@@ -555,4 +591,33 @@ static inline int odb_source_fsck(struct odb_source *source,\n \treturn source->fsck(source, opts);\n }\n \n+/*\n+ * Map the specified object ID into the specified algorithm.\n+ *\n+ * Returns 0 on success, -1 if the object is not found, and a different negative\n+ * error code otherwise.\n+ */\n+static inline int odb_source_map_oid(struct odb_source *source,\n+\t\t\t\t const struct object_id *src,\n+\t\t\t\t const struct git_hash_algo *to,\n+\t\t\t\t struct object_id *dest,\n+\t\t\t\t int flags)\n+{\n+\treturn source->map_oid(source, src, to, dest, flags);\n+}\n+\n+/*\n+ * Create a mapping between the two object IDs, which should be in the main and\n+ * compatibility hash algorithms.\n+ *\n+ * Returns 0 on success, a negative error otherwise.\n+ */\n+static inline int odb_source_write_oid_map(struct odb_source *source,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const struct object_id *compat_oid,\n+\t\t\t\t\t int flags)\n+{\n+\treturn source->write_oid_map(source, oid, compat_oid, flags);\n+}\n+\n #endif\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555963","messageId":"20261023072559.3204273-6-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 05/13] object-file-convert: implement mapping in terms of object DB","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:50Z","receivedAt":"2026-10-23T07:25:50Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nNow that we have odb_map_oid, let's implement repo_oid_to_algop in terms\nof it. We still need the latter because it does some cleanup that needs\nto happen so that our object database backend need not deal with these\nissues.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n object-file-convert.c | 14 ++------------\n 1 file changed, 2 insertions(+), 12 deletions(-)\n\ndiff --git a/object-file-convert.c b/object-file-convert.c\nindex 63ee18630b..90a75a6c21 100644\n--- a/object-file-convert.c\n+++ b/object-file-convert.c\n@@ -7,6 +7,7 @@\n #include \"repository.h\"\n #include \"hash.h\"\n #include \"object.h\"\n+#include \"odb.h\"\n #include \"loose.h\"\n #include \"commit.h\"\n #include \"gpg-interface.h\"\n@@ -35,18 +36,7 @@ int repo_oid_to_algop(struct repository *repo, const struct object_id *srcoid,\n \t\t\toidcpy(dest, src);\n \t\treturn 0;\n \t}\n-\tif (repo_loose_object_map_oid(repo, src, to, dest)) {\n-\t\t/*\n-\t\t * We may have loaded the object map at repo initialization but\n-\t\t * another process (perhaps upstream of a pipe from us) may have\n-\t\t * written a new object into the map. If the object is missing,\n-\t\t * let's reload the map to see if the object has appeared.\n-\t\t */\n-\t\trepo_read_loose_object_map(repo);\n-\t\tif (repo_loose_object_map_oid(repo, src, to, dest))\n-\t\t\treturn -1;\n-\t}\n-\treturn 0;\n+\treturn odb_map_oid(repo->objects, src, to, dest, 0);\n }\n \n static int decode_tree_entry_raw(struct object_id *oid, const char **path,\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555964","messageId":"20261023072559.3204273-7-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 06/13] pack-write: add support for pack index v3","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:51Z","receivedAt":"2026-10-23T07:25:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nAs part of the SHA-256 transition plan, we'll be adding support for pack\nindex v3. This version supports multiple hash algorithms so we can\nwrite both the SHA-256 and SHA-1 object names into the same repository.\n\nTo do so, let's add support for the compatibility object ID and the\nentry number into the index entry structure. We then use this\ninformation when writing the pack. There is no code to populate these\nfields yet; this functionality will be coming in a future commit.\n\nWe always choose to write the minimum amount of padding that is\nnecessary to cause the end of the short object name table to end on a\nmultiple of 4. Git reads many tables as 32-bit values and if we didn't\ndo this, the operations would fail on architectures which don't support\nunaligned access and would be slower on all other architectures.\n\nWe choose to always write the main hash algorithm first in the pack.\nThis optimizes for the case when we're interacting natively with the\nremote side by permitting us to use the CRC32 entries.\n\nNote that we never write index v3, since we cannot read it yet, and\ndoing so will break the tests. In a future commit, once we can read it,\nwe'll enable it by default when a compatibility algorithm is enabled.\n\nIndex v3 needs the objects both in pack order and sorted by object ID,\nso when writing a v3 index, sort copies of the array instead of the\narray itself. Callers, like the bitmap code in pack-objects, rely on\nthe array being sorted by object ID when write_idx_file() returns,\nthough, so sort it before returning in that case too.\n\nWhen asked to reject duplicate objects using `--strict`, look for them\nin the arrays sorted by object ID, where duplicates are adjacent.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n pack-write.c | 263 +++++++++++++++++++++++++++++++++++++++++++++------\n pack.h | 2 +\n 2 files changed, 235 insertions(+), 30 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 85674e4b72..1834394a77 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -22,13 +22,20 @@ void reset_pack_idx_option(struct pack_idx_option *opts)\n \topts->delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT;\n }\n \n-static int sha1_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tstruct pack_idx_entry *a = *(struct pack_idx_entry **)_a;\n \tstruct pack_idx_entry *b = *(struct pack_idx_entry **)_b;\n \treturn oidcmp(&a->oid, &b->oid);\n }\n \n+static int oid_compare_compat(const void *_a, const void *_b)\n+{\n+\tstruct pack_idx_entry *a = *(struct pack_idx_entry **)_a;\n+\tstruct pack_idx_entry *b = *(struct pack_idx_entry **)_b;\n+\treturn oidcmp(&a->compat_oid, &b->compat_oid);\n+}\n+\n static int cmp_uint32(const void *a_, const void *b_)\n {\n \tuint32_t a = *((uint32_t *)a_);\n@@ -50,30 +57,156 @@ static int need_large_offset(off_t offset, const struct pack_idx_option *opts)\n \t\t\t sizeof(ofsval), cmp_uint32);\n }\n \n+static inline int last_matching_offset(const struct object_id *a, const struct object_id *b,\n+\t\t\t\t const struct git_hash_algo *algop)\n+{\n+\tsize_t i;\n+\tfor (i = 0; i < algop->rawsz; i++)\n+\t\tif (a->hash[i] != b->hash[i])\n+\t\t\treturn i;\n+\t/* We should never hit this case. */\n+\treturn i;\n+}\n+\n+static const struct object_id *main_oid(const struct pack_idx_entry *ent)\n+{\n+\treturn &ent->oid;\n+}\n+\n+static const struct object_id *compat_oid(const struct pack_idx_entry *ent)\n+{\n+\treturn &ent->compat_oid;\n+}\n+\n+static uint32_t find_short_name_len(struct pack_idx_entry **sorted, size_t nr_objects,\n+\t\t\t\t const struct git_hash_algo *algop,\n+\t\t\t\t const struct object_id *(*fn)(const struct pack_idx_entry *))\n+{\n+\tsize_t i, len = 1;\n+\tfor (i = 1; i < nr_objects; i++) {\n+\t\tconst struct object_id *oid1 = fn(sorted[i - 1]);\n+\t\tconst struct object_id *oid2 = fn(sorted[i]);\n+\t\tsize_t offset = last_matching_offset(oid1, oid2, algop);\n+\n+\t\tif (offset >= len)\n+\t\t\tlen = offset + 1;\n+\t}\n+\treturn len > algop->rawsz ? algop->rawsz : len;\n+}\n+\n+struct pack_idx_format_header {\n+\tuint32_t id;\n+\tuint32_t short_name_len;\n+\tuint64_t offset;\n+};\n+\n+/* This struct contains fields beyond the signature and version number. */\n+struct pack_idx_header_v3 {\n+\tuint32_t hdr_len;\n+\tuint32_t nr_objects;\n+\tuint32_t nr_formats;\n+};\n+\n+static void write_v3_header(struct repository *repo,\n+\t\t\t int nr_objects, int nr_large_offset,\n+\t\t\t uint32_t short_name_len[2], int padding[2],\n+\t\t\t struct hashfile *f)\n+{\n+\tstruct pack_idx_header_v3 hdr;\n+\tstruct pack_idx_format_header format[2];\n+\tconst struct git_hash_algo *compat = repo->compat_hash_algo;\n+\tuint32_t nformats = compat ? 2 : 1;\n+\t/*\n+\t * signature (4) + version (4) + length (4) + nobjects(4) + nformats (4) +\n+\t * each format: identifier (4) + length of shortened names (4) +\n+\t *\t\ttable offset (8) +\n+\t * offset to trailer (8)\n+\t */\n+\tuint32_t hdrlen = 4 + 4 + 4 + 4 + 4 + nformats * (4 + 4 + 8) + 8;\n+\tuint64_t offset = hdrlen;\n+\tuint64_t short_table = short_name_len[0] * nr_objects;\n+\n+\thdr.hdr_len = htonl(hdrlen);\n+\thdr.nr_objects = htonl(nr_objects);\n+\thdr.nr_formats = htonl(nformats);\n+\thashwrite(f, &hdr, sizeof(hdr));\n+\n+\t/* padding */\n+\tif (short_table & 3) {\n+\t\tpadding[0] = 4 - (short_table & 3);\n+\t\toffset += padding[0];\n+\t}\n+\n+\tformat[0].id = htonl(repo->hash_algo->format_id);\n+\tformat[0].offset = htonll(offset);\n+\tformat[0].short_name_len = htonl(short_name_len[0]);\n+\thashwrite(f, &format[0], sizeof(format[0]));\n+\toffset += (uint64_t)(short_name_len[0] + repo->hash_algo->rawsz + 4 + 4 + 4) * nr_objects +\n+\t\t 8 * nr_large_offset;\n+\n+\tif (compat) {\n+\t\tshort_table = short_name_len[1] * nr_objects;\n+\t\t/* padding */\n+\t\tif (short_table & 3) {\n+\t\t\tpadding[1] = 4 - (short_table & 3);\n+\t\t\toffset += padding[1];\n+\t\t}\n+\n+\t\tformat[1].id = htonl(compat->format_id);\n+\t\tformat[1].offset = htonll(offset);\n+\t\tformat[1].short_name_len = htonl(short_name_len[1]);\n+\t\thashwrite(f, &format[1], sizeof(format[1]));\n+\t\toffset += (uint64_t)(short_name_len[1] + compat->rawsz + 4) * nr_objects;\n+\t}\n+\n+\toffset = htonll(offset);\n+\thashwrite(f, &offset, sizeof(offset));\n+}\n+\n /*\n- * The *sha1 contains the pack content SHA1 hash.\n- * The objects array passed in will be sorted by SHA1 on exit.\n+ * The *hash contains the pack content hash.\n+ * The objects array passed in should be in pack order and will be sorted by\n+ * object ID on return.\n */\n const char *write_idx_file(struct repository *repo,\n \t\t\t const char *index_name, struct pack_idx_entry **objects,\n \t\t\t int nr_objects, const struct pack_idx_option *opts,\n-\t\t\t const unsigned char *sha1)\n+\t\t\t const unsigned char *hash)\n {\n \tstruct hashfile *f;\n-\tstruct pack_idx_entry **sorted_by_sha, **list, **last;\n-\toff_t last_obj_offset = 0;\n+\tstruct pack_idx_entry **sorted_by_sha, **sorted_by_sha_compat = NULL, **list, **last;\n \tint i, fd;\n \tuint32_t index_version;\n+\tunsigned int nr_large_offset = 0;\n+\tconst struct git_hash_algo *compat = repo->compat_hash_algo;\n+\tuint32_t short_name_len[2] = { 0 };\n+\tint padding_size[2] = { 0 };\n+\tuint8_t zero[4] = { 0 };\n+\n+\tindex_version = opts->version;\n \n \tif (nr_objects) {\n \t\tsorted_by_sha = objects;\n \t\tlist = sorted_by_sha;\n \t\tlast = sorted_by_sha + nr_objects;\n \t\tfor (i = 0; i < nr_objects; ++i) {\n-\t\t\tif (objects[i]->offset > last_obj_offset)\n-\t\t\t\tlast_obj_offset = objects[i]->offset;\n+\t\t\tobjects[i]->idx = i;\n+\t\t\tif (need_large_offset(objects[i]->offset, opts))\n+\t\t\t\tnr_large_offset++;\n \t\t}\n-\t\tQSORT(sorted_by_sha, nr_objects, sha1_compare);\n+\t\t/*\n+\t\t * If we're using index v3, we need both pack order and sorted\n+\t\t * order.\n+\t\t */\n+\t\tif (index_version >= 3) {\n+\t\t\tDUP_ARRAY(sorted_by_sha, objects, nr_objects);\n+\t\t\tif (compat) {\n+\t\t\t\tDUP_ARRAY(sorted_by_sha_compat, objects,\n+\t\t\t\t\t nr_objects);\n+\t\t\t\tQSORT(sorted_by_sha_compat, nr_objects, oid_compare_compat);\n+\t\t\t}\n+\t\t}\n+\t\tQSORT(sorted_by_sha, nr_objects, oid_compare);\n \t}\n \telse\n \t\tsorted_by_sha = list = last = NULL;\n@@ -95,7 +228,8 @@ const char *write_idx_file(struct repository *repo,\n \t}\n \n \t/* if last object's offset is >= 2^31 we should use index V2 */\n-\tindex_version = need_large_offset(last_obj_offset, opts) ? 2 : opts->version;\n+\tif (index_version == 1 && nr_large_offset)\n+\t\tindex_version = 2;\n \n \t/* index versions 2 and above need a header */\n \tif (index_version >= 2) {\n@@ -105,50 +239,84 @@ const char *write_idx_file(struct repository *repo,\n \t\thashwrite(f, &hdr, sizeof(hdr));\n \t}\n \n-\t/*\n-\t * Write the first-level table (the list is sorted,\n-\t * but we use a 256-entry lookup to be able to avoid\n-\t * having to do eight extra binary search iterations).\n-\t */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct pack_idx_entry **next = list;\n-\t\twhile (next < last) {\n-\t\t\tstruct pack_idx_entry *obj = *next;\n-\t\t\tif (obj->oid.hash[0] != i)\n-\t\t\t\tbreak;\n-\t\t\tnext++;\n+\tif (index_version >= 3) {\n+\t\tshort_name_len[0] = find_short_name_len(sorted_by_sha,\n+\t\t\t\t\t\t\tnr_objects,\n+\t\t\t\t\t\t\trepo->hash_algo,\n+\t\t\t\t\t\t\tmain_oid);\n+\t\tif (compat)\n+\t\t\tshort_name_len[1] = find_short_name_len(sorted_by_sha_compat,\n+\t\t\t\t\t\t\t\tnr_objects,\n+\t\t\t\t\t\t\t\tcompat, compat_oid);\n+\t\twrite_v3_header(repo, nr_objects, nr_large_offset, short_name_len, padding_size, f);\n+\t\thashwrite(f, zero, padding_size[0]);\n+\n+\t\t/* Write the sorted abbreviated names */\n+\t\tlist = sorted_by_sha;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\thashwrite(f, obj->oid.hash, short_name_len[0]);\n+\t\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n+\t\t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n+\t\t\t\tdie(\"The same object %s appears twice in the pack\",\n+\t\t\t\t oid_to_hex(&obj->oid));\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Write the first-level table (the list is sorted,\n+\t\t * but we use a 256-entry lookup to be able to avoid\n+\t\t * having to do eight extra binary search iterations).\n+\t\t */\n+\t\tfor (i = 0; i < 256; i++) {\n+\t\t\tstruct pack_idx_entry **next = list;\n+\t\t\twhile (next < last) {\n+\t\t\t\tstruct pack_idx_entry *obj = *next;\n+\t\t\t\tif (obj->oid.hash[0] != i)\n+\t\t\t\t\tbreak;\n+\t\t\t\tnext++;\n+\t\t\t}\n+\t\t\thashwrite_be32(f, next - sorted_by_sha);\n+\t\t\tlist = next;\n \t\t}\n-\t\thashwrite_be32(f, next - sorted_by_sha);\n-\t\tlist = next;\n \t}\n \n \t/*\n \t * Write the actual SHA1 entries..\n \t */\n-\tlist = sorted_by_sha;\n+\tlist = objects;\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_idx_entry *obj = *list++;\n \t\tif (index_version < 2)\n \t\t\thashwrite_be32(f, obj->offset);\n \t\thashwrite(f, obj->oid.hash, repo->hash_algo->rawsz);\n-\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n+\t\tif ((opts->flags & WRITE_IDX_STRICT) && index_version < 3 &&\n \t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n \t\t\tdie(\"The same object %s appears twice in the pack\",\n \t\t\t oid_to_hex(&obj->oid));\n \t}\n \n+\tif (index_version >= 3) {\n+\t\t/* write the sorted-to-pack index */\n+\t\tlist = sorted_by_sha;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\tuint32_t index = htonl(obj->idx);\n+\t\t\thashwrite(f, &index, 4);\n+\t\t}\n+\t}\n+\n \tif (index_version >= 2) {\n \t\tunsigned int nr_large_offset = 0;\n \n \t\t/* write the crc32 table */\n-\t\tlist = sorted_by_sha;\n+\t\tlist = objects;\n \t\tfor (i = 0; i < nr_objects; i++) {\n \t\t\tstruct pack_idx_entry *obj = *list++;\n \t\t\thashwrite_be32(f, obj->crc32);\n \t\t}\n \n \t\t/* write the 32-bit offset table */\n-\t\tlist = sorted_by_sha;\n+\t\tlist = objects;\n \t\tfor (i = 0; i < nr_objects; i++) {\n \t\t\tstruct pack_idx_entry *obj = *list++;\n \t\t\tuint32_t offset;\n@@ -160,7 +328,7 @@ const char *write_idx_file(struct repository *repo,\n \t\t}\n \n \t\t/* write the large offset table */\n-\t\tlist = sorted_by_sha;\n+\t\tlist = objects;\n \t\twhile (nr_large_offset) {\n \t\t\tstruct pack_idx_entry *obj = *list++;\n \t\t\tuint64_t offset = obj->offset;\n@@ -172,10 +340,45 @@ const char *write_idx_file(struct repository *repo,\n \t\t}\n \t}\n \n-\thashwrite(f, sha1, repo->hash_algo->rawsz);\n+\tif (index_version >= 3 && compat) {\n+\t\thashwrite(f, zero, padding_size[1]);\n+\n+\t\t/* Write the sorted abbreviated names */\n+\t\tlist = sorted_by_sha_compat;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\thashwrite(f, obj->compat_oid.hash, short_name_len[1]);\n+\t\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n+\t\t\t (i && oideq(&list[-2]->compat_oid, &obj->compat_oid)))\n+\t\t\t\tdie(\"The same object %s appears twice in the pack\",\n+\t\t\t\t oid_to_hex(&obj->compat_oid));\n+\t\t}\n+\n+\t\tlist = objects;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\thashwrite(f, obj->compat_oid.hash, compat->rawsz);\n+\t\t}\n+\n+\t\t/* write the sorted-to-pack index */\n+\t\tlist = sorted_by_sha_compat;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\tuint32_t index = htonl(obj->idx);\n+\t\t\thashwrite(f, &index, 4);\n+\t\t}\n+\t}\n+\n+\thashwrite(f, hash, repo->hash_algo->rawsz);\n \tfinalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,\n \t\t\t CSUM_HASH_IN_STREAM | CSUM_CLOSE |\n \t\t\t ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));\n+\n+\tif (sorted_by_sha != objects) {\n+\t\tQSORT(objects, nr_objects, oid_compare);\n+\t\tfree(sorted_by_sha);\n+\t}\n+\tfree(sorted_by_sha_compat);\n \treturn index_name;\n }\n \ndiff --git a/pack.h b/pack.h\nindex ada506b5c5..86b2a8ba7b 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -79,7 +79,9 @@ struct pack_idx_header {\n */\n struct pack_idx_entry {\n \tstruct object_id oid;\n+\tstruct object_id compat_oid;\n \tuint32_t crc32;\n+\tuint32_t idx;\n \toff_t offset;\n };\n \n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555965","messageId":"20261023072559.3204273-8-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 07/13] builtin: populate pack index v3 fields when indexing","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:52Z","receivedAt":"2026-10-23T07:25:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nRight now, we have support for writing pack index v3. However, we need\nto populate the compatibility object ID in order to write it into the\nrepository. Let's do so by converting the objects from SHA-256 to SHA-1\nas part of our indexing process.\n\nTake advantage of the fact that a SHA-256 blob and its corresponding\nSHA-1 blob are identical by always hashing them at the same time. For\nnon-blobs, we convert the objects from SHA-256 to SHA-1 before hashing.\n\nSince, when converting an object, we may need to know about the\ncompatibility object IDs of objects earlier in the pack, on which they\ndepend, store the compatibility object IDs we generate in the loose\nobject map, but don't write them to disk. This is the simplest way to\nstore these objects and have them properly looked up by the same process\nwithout polluting the loose object index on disk with all of these\nobjects.\n\nBecause these operations may be threaded and modifying the loose object\nmap isn't thread safe, add a mutex and use it to protect this structure.\nNote that the object conversion routines use this structure. Acquire\nand release the mutex between the conversion and writing to the map\nbecause hashing may be slow.\n\nAdd a test in t1016 to check that index-pack can write a v3 index in a\nrepository with a compatibility algorithm, and a test that it rejects\npacks with duplicate objects when writing a v3 index with `--strict`.\n\nAlso add a create_compat_repo() helper in t1016 to refactor common\ncode.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n builtin/index-pack.c | 105 +++++++++++++++++++++++++++---\n builtin/pack-objects.c | 8 ++-\n t/t1016-compatObjectFormat.sh | 29 ++++++++-\n t/t5308-pack-detect-duplicates.sh | 7 ++\n 4 files changed, 137 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 0264c18c5f..723595ae03 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -31,6 +31,8 @@\n #include \"run-command.h\"\n #include \"setup.h\"\n #include \"strvec.h\"\n+#include \"loose.h\"\n+#include \"object-file-convert.h\"\n \n static const char index_pack_usage[] =\n \"git index-pack [-v] [-o ] [--keep | --keep=] [--[no-]rev-index] [--verify] [--strict[==...]] [--fsck-objects[==...]] ( | --stdin [--fix-thin] [])\";\n@@ -182,6 +184,10 @@ static pthread_mutex_t deepest_delta_mutex;\n #define deepest_delta_lock()\tlock_mutex(&deepest_delta_mutex)\n #define deepest_delta_unlock()\tunlock_mutex(&deepest_delta_mutex)\n \n+static pthread_mutex_t repo_mutex;\n+#define repo_lock()\tlock_mutex(&repo_mutex)\n+#define repo_unlock()\tunlock_mutex(&repo_mutex)\n+\n static pthread_key_t key;\n \n static inline void lock_mutex(pthread_mutex_t *mutex)\n@@ -205,6 +211,7 @@ static void init_thread(void)\n \tinit_recursive_mutex(&read_mutex);\n \tpthread_mutex_init(&counter_mutex, NULL);\n \tpthread_mutex_init(&work_mutex, NULL);\n+\tpthread_mutex_init(&repo_mutex, NULL);\n \tif (show_stat)\n \t\tpthread_mutex_init(&deepest_delta_mutex, NULL);\n \tpthread_key_create(&key, NULL);\n@@ -225,6 +232,7 @@ static void cleanup_thread(void)\n \tpthread_mutex_destroy(&read_mutex);\n \tpthread_mutex_destroy(&counter_mutex);\n \tpthread_mutex_destroy(&work_mutex);\n+\tpthread_mutex_destroy(&repo_mutex);\n \tif (show_stat)\n \t\tpthread_mutex_destroy(&deepest_delta_mutex);\n \tfor (i = 0; i < nr_threads; i++)\n@@ -469,13 +477,15 @@ static int is_delta_type(enum object_type type)\n }\n \n static void *unpack_entry_data(off_t offset, size_t size,\n-\t\t\t enum object_type type, struct object_id *oid)\n+\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t struct object_id *compat_oid)\n {\n \tstatic char fixed_buf[8192];\n \tint status;\n \tgit_zstream stream;\n \tvoid *buf;\n-\tstruct git_hash_ctx c;\n+\tstruct git_hash_ctx c, c_compat;\n+\tconst struct git_hash_algo *compat = the_repository->compat_hash_algo;\n \tchar hdr[32];\n \tint hdrlen;\n \n@@ -483,8 +493,15 @@ static void *unpack_entry_data(off_t offset, size_t size,\n \t\thdrlen = format_object_header(hdr, sizeof(hdr), type, size);\n \t\tgit_hash_init(&c, the_hash_algo);\n \t\tgit_hash_update(&c, hdr, hdrlen);\n-\t} else\n+\t\tif (compat && compat_oid) {\n+\t\t\tgit_hash_init(&c_compat, compat);\n+\t\t\tif (type == OBJ_BLOB)\n+\t\t\t\tgit_hash_update(&c_compat, hdr, hdrlen);\n+\t\t}\n+\t} else {\n \t\toid = NULL;\n+\t\tcompat_oid = NULL;\n+\t}\n \tif (type == OBJ_BLOB &&\n \t size > repo_settings_get_big_file_threshold(the_repository))\n \t\tbuf = fixed_buf;\n@@ -504,6 +521,8 @@ static void *unpack_entry_data(off_t offset, size_t size,\n \t\tuse(input_len - stream.avail_in);\n \t\tif (oid)\n \t\t\tgit_hash_update(&c, last_out, stream.next_out - last_out);\n+\t\tif (compat && compat_oid && type == OBJ_BLOB)\n+\t\t\tgit_hash_update(&c_compat, last_out, stream.next_out - last_out);\n \t\tif (buf == fixed_buf) {\n \t\t\tstream.next_out = buf;\n \t\t\tstream.avail_out = sizeof(fixed_buf);\n@@ -514,13 +533,43 @@ static void *unpack_entry_data(off_t offset, size_t size,\n \tgit_inflate_end(&stream);\n \tif (oid)\n \t\tgit_hash_final_oid(oid, &c);\n+\tif (compat && compat_oid) {\n+\t\tif (type != OBJ_BLOB) {\n+\t\t\t/*\n+\t\t\t * This is not a blob. Let's remap it to the proper\n+\t\t\t * algorithm.\n+\t\t\t */\n+\t\t\tstruct strbuf outbuf = STRBUF_INIT;\n+\t\t\tint ret;\n+\t\t\trepo_lock();\n+\t\t\tret = convert_object_file(the_repository, &outbuf,\n+\t\t\t\t\t\t the_repository->hash_algo,\n+\t\t\t\t\t\t compat,\n+\t\t\t\t\t\t buf, size, type, 0);\n+\t\t\trepo_unlock();\n+\t\t\tif (ret < 0)\n+\t\t\t\tbad_object(offset, _(\"could not convert object\"));\n+\n+\t\t\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %\"PRIuMAX,\n+\t\t\t\t\t type_name(type),(uintmax_t)outbuf.len) + 1;\n+\n+\t\t\tgit_hash_update(&c_compat, hdr, hdrlen);\n+\t\t\tgit_hash_update(&c_compat, outbuf.buf, outbuf.len);\n+\t\t\tstrbuf_release(&outbuf);\n+\t\t}\n+\t\tgit_hash_final_oid(compat_oid, &c_compat);\n+\t\trepo_lock();\n+\t\todb_write_oid_map(the_repository->objects, oid, compat_oid, 0);\n+\t\trepo_unlock();\n+\t}\n \treturn buf == fixed_buf ? NULL : buf;\n }\n \n static void *unpack_raw_entry(struct object_entry *obj,\n \t\t\t off_t *ofs_offset,\n \t\t\t struct object_id *ref_oid,\n-\t\t\t struct object_id *oid)\n+\t\t\t struct object_id *oid,\n+\t\t\t struct object_id *compat_oid)\n {\n \tunsigned char *p;\n \tsize_t size, c;\n@@ -582,7 +631,7 @@ static void *unpack_raw_entry(struct object_entry *obj,\n \t}\n \tobj->hdr_size = consumed_bytes - obj->idx.offset;\n \n-\tdata = unpack_entry_data(obj->idx.offset, obj->size, obj->type, oid);\n+\tdata = unpack_entry_data(obj->idx.offset, obj->size, obj->type, oid, compat_oid);\n \tobj->idx.crc32 = input_crc32;\n \treturn data;\n }\n@@ -1069,6 +1118,45 @@ static struct base_data *resolve_delta(struct object_entry *delta_obj,\n \t\tbad_object(delta_obj->idx.offset, _(\"failed to apply delta\"));\n \thash_object_file(the_hash_algo, result_data, result_size,\n \t\t\t delta_obj->real_type, &delta_obj->idx.oid);\n+\tif (the_repository->compat_hash_algo) {\n+\t\tstruct strbuf outbuf = STRBUF_INIT;\n+\t\tint ret = 0;\n+\t\tchar *buf;\n+\t\tsize_t len;\n+\n+\t\tif (delta_obj->real_type != OBJ_BLOB) {\n+\t\t\trepo_lock();\n+\t\t\tret = convert_object_file(the_repository, &outbuf,\n+\t\t\t\t\t\t the_repository->hash_algo,\n+\t\t\t\t\t\t the_repository->compat_hash_algo,\n+\t\t\t\t\t\t result_data, result_size,\n+\t\t\t\t\t\t delta_obj->real_type, 0);\n+\t\t\trepo_unlock();\n+\t\t\tbuf = outbuf.buf;\n+\t\t\tlen = outbuf.len;\n+\t\t} else {\n+\t\t\tbuf = result_data;\n+\t\t\tlen = result_size;\n+\t\t}\n+\t\tif (ret < 0)\n+\t\t\tbad_object(delta_obj->idx.offset,\n+\t\t\t\t _(\"could not convert object from delta\"));\n+\n+\t\thash_object_file(the_repository->compat_hash_algo, buf, len,\n+\t\t\t\t delta_obj->real_type, &delta_obj->idx.compat_oid);\n+\t\t/*\n+\t\t * We add this entry to the loose object map but do not write it\n+\t\t * to disk because we may need it available when mapping other\n+\t\t * objects. Storing it temporarily in the loose object map lets\n+\t\t * us keep it around for that purpose.\n+\t\t */\n+\t\trepo_lock();\n+\t\todb_write_oid_map(the_repository->objects,\n+\t\t\t\t &delta_obj->idx.oid,\n+\t\t\t\t &delta_obj->idx.compat_oid, 0);\n+\t\trepo_unlock();\n+\t\tstrbuf_release(&outbuf);\n+\t}\n \tsha1_object(result_data, NULL, result_size, delta_obj->real_type,\n \t\t &delta_obj->idx.oid);\n \n@@ -1265,7 +1353,8 @@ static void parse_pack_objects(unsigned char *hash)\n \t\tstruct object_entry *obj = &objects[i];\n \t\tvoid *data = unpack_raw_entry(obj, &ofs_delta->offset,\n \t\t\t\t\t &ref_delta_oid,\n-\t\t\t\t\t &obj->idx.oid);\n+\t\t\t\t\t &obj->idx.oid,\n+\t\t\t\t\t &obj->idx.compat_oid);\n \t\tobj->real_type = obj->type;\n \t\tif (obj->type == OBJ_OFS_DELTA) {\n \t\t\tnr_ofs_deltas++;\n@@ -1669,7 +1758,7 @@ static int git_index_pack_config(const char *k, const char *v,\n \n \tif (!strcmp(k, \"pack.indexversion\")) {\n \t\topts->version = git_config_int(k, v, ctx->kvi);\n-\t\tif (opts->version > 2)\n+\t\tif (opts->version > 3)\n \t\t\tdie(_(\"bad pack.indexVersion=%\"PRIu32), opts->version);\n \t\treturn 0;\n \t}\n@@ -1991,7 +2080,7 @@ int cmd_index_pack(int argc,\n \t\t\t} else if (starts_with(arg, \"--index-version=\")) {\n \t\t\t\tchar *c;\n \t\t\t\topts.version = strtoul(arg + 16, &c, 10);\n-\t\t\t\tif (opts.version > 2)\n+\t\t\t\tif (opts.version > 3)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t\tif (*c == ',')\n \t\t\t\t\topts.off32_limit = strtoul(c+1, &c, 0);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca3a891dfb..fc4adb6877 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -47,6 +47,9 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"path-walk.h\"\n+#include \"trace2.h\"\n+#include \"loose.h\"\n+#include \"object-file-convert.h\"\n \n /*\n * Objects we are going to pack are collected in the `to_pack` structure.\n@@ -1847,6 +1850,9 @@ static struct object_entry *create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid);\n \tentry->hash = hash;\n+\tif (repo_oid_to_algop(the_repository, &entry->idx.oid,\n+\t\t\t the_repository->compat_hash_algo, &entry->idx.compat_oid) < 0)\n+\t\tdie(_(\"can't map object %s while writing pack\"), oid_to_hex(oid));\n \toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n@@ -5039,7 +5045,7 @@ static int option_parse_index_version(const struct option *opt,\n \tBUG_ON_OPT_NEG(unset);\n \n \tpopts->version = strtoul(val, &c, 10);\n-\tif (popts->version > 2)\n+\tif (popts->version > 3)\n \t\tdie(_(\"unsupported index version %s\"), val);\n \tif (*c == ',' && c[1])\n \t\tpopts->off32_limit = strtoul(c+1, &c, 0);\ndiff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh\nindex 9cafcee509..1560b6a2e2 100755\n--- a/t/t1016-compatObjectFormat.sh\n+++ b/t/t1016-compatObjectFormat.sh\n@@ -45,6 +45,15 @@ compat_hash () {\n \tesac\n }\n \n+create_compat_repo () {\n+\tlocal hash=\"$1\"\n+\tlocal repo=\"$2\"\n+\n+\tgit init --object-format=\"$hash\" \"$repo\" &&\n+\tgit -C \"$repo\" config set core.repositoryformatversion 1 &&\n+\tgit -C \"$repo\" config set extensions.compatObjectFormat $(compat_hash \"$hash\")\n+}\n+\n hello_oid () {\n \tcase \"$1\" in\n \t\"sha1\")\n@@ -192,9 +201,7 @@ do\n \t\tfor repo in alt borrow\n \t\tdo\n \t\t\ttest_when_finished \"rm -rf $repo\" &&\n-\t\t\tgit init --object-format=$hash $repo &&\n-\t\t\tgit -C $repo config set core.repositoryformatversion 1 &&\n-\t\t\tgit -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1\n+\t\t\tcreate_compat_repo $hash $repo || exit 1\n \t\tdone &&\n \n \t\tgit -C alt commit --allow-empty --message A &&\n@@ -205,6 +212,22 @@ do\n \t\tgit -C borrow rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >actual &&\n \t\ttest_cmp expect actual\n \t'\n+\n+\ttest_expect_success 'index-pack writes pack index v3' '\n+\t\ttest_when_finished \"rm -rf v3-repo v3.idx\" &&\n+\t\tcreate_compat_repo $hash v3-repo &&\n+\t\techo content >v3-repo/file &&\n+\t\tgit -C v3-repo add file &&\n+\t\tgit -C v3-repo commit --message A &&\n+\t\tgit -C v3-repo commit --allow-empty --message B &&\n+\t\tgit -C v3-repo repack -a -d &&\n+\t\tpack=$(echo v3-repo/.git/objects/pack/pack-*.pack) &&\n+\t\tgit -C v3-repo index-pack --index-version=3 \\\n+\t\t\t-o \"$(pwd)/v3.idx\" \"$(pwd)/$pack\" &&\n+\t\techo \"ff 74 4f 63 00 00 00 03 \" >expect &&\n+\t\ttest_copy_bytes 8 actual &&\n+\t\ttest_cmp expect actual\n+\t'\n done\n cd \"$base\"\n \ndiff --git a/t/t5308-pack-detect-duplicates.sh b/t/t5308-pack-detect-duplicates.sh\nindex 0f84137867..7b43db870a 100755\n--- a/t/t5308-pack-detect-duplicates.sh\n+++ b/t/t5308-pack-detect-duplicates.sh\n@@ -79,4 +79,11 @@ test_expect_success 'index-pack can reject packs with duplicates' '\n \ttest_expect_code 1 git cat-file -e $LO_SHA1\n '\n \n+test_expect_success 'index-pack can reject packs with duplicates with index v3' '\n+\tclear_packs &&\n+\tcreate_pack dups.pack 2 &&\n+\ttest_must_fail git index-pack --strict --index-version=3 --stdin \n\nWhen we check in a large blob, we stream it directly to a pack and then\nwrite an index. However, in compatibility mode, we also need to make\nsure to compute the object ID for the compatibility algorithm as well as\nthe main algorithm or we will incorrectly store the null OID in the\nindex we create. Do so here to avoid corrupting the repository.\n\nAdd a test that adds a large file in a repository with a compatibility\nalgorithm.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n object-file.c | 18 ++++++++++++++++--\n t/t1016-compatObjectFormat.sh | 14 ++++++++++++++\n 2 files changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex c049bf13f7..b4c4a2ea0f 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -747,6 +747,7 @@ static int hash_stream(struct odb_stream *stream,\n */\n static void stream_to_pack(struct transaction_packfile *state,\n \t\t\t struct git_hash_ctx *ctx,\n+\t\t\t struct git_hash_ctx *compat_ctx,\n \t\t\t struct odb_stream *stream)\n {\n \tgit_zstream s;\n@@ -774,6 +775,9 @@ static void stream_to_pack(struct transaction_packfile *state,\n \t\t\t\tis_finished = true;\n \n \t\t\tgit_hash_update(ctx, ibuf, rsize);\n+\t\t\tif (compat_ctx)\n+\t\t\t\tgit_hash_update(compat_ctx,\n+\t\t\t\t\t\tibuf, rsize);\n \n \t\t\ts.next_in = ibuf;\n \t\t\ts.avail_in = rsize;\n@@ -882,16 +886,22 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \t\t\t\t\t\t\t\t struct odb_transaction_files,\n \t\t\t\t\t\t\t\t base);\n \tstruct transaction_packfile *state = &transaction->packfile;\n-\tstruct git_hash_ctx ctx;\n+\tstruct git_hash_ctx ctx, compat_ctx;\n+\tstruct object_id compat_oid;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n \tstruct hashfile_checkpoint checkpoint;\n \tstruct pack_idx_entry *idx;\n+\tconst struct git_hash_algo *compat = transaction->base.source->odb->repo->compat_hash_algo;\n \n \theader_len = format_object_header((char *)obuf, sizeof(obuf),\n \t\t\t\t\t stream->type, stream->size);\n \tgit_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);\n \tgit_hash_update(&ctx, obuf, header_len);\n+\tif (compat) {\n+\t\tgit_hash_init(&compat_ctx, compat);\n+\t\tgit_hash_update(&compat_ctx, obuf, header_len);\n+\t}\n \n \t/*\n \t * If writing another object to the packfile could result in it\n@@ -916,8 +926,10 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \thashfile_checkpoint(state->f, &checkpoint);\n \tidx->offset = state->offset;\n \tcrc32_begin(state->f);\n-\tstream_to_pack(state, &ctx, stream);\n+\tstream_to_pack(state, &ctx, compat ? &compat_ctx : NULL, stream);\n \tgit_hash_final_oid(result_oid, &ctx);\n+\tif (compat)\n+\t\tgit_hash_final_oid(&compat_oid, &compat_ctx);\n \n \tidx->crc32 = crc32_end(state->f);\n \tif (already_written(transaction, result_oid)) {\n@@ -926,6 +938,8 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \t\tfree(idx);\n \t} else {\n \t\toidcpy(&idx->oid, result_oid);\n+\t\tif (compat)\n+\t\t\toidcpy(&idx->compat_oid, &compat_oid);\n \t\tALLOC_GROW(state->written,\n \t\t\t state->nr_written + 1,\n \t\t\t state->alloc_written);\ndiff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh\nindex 1560b6a2e2..af9532aa1f 100755\n--- a/t/t1016-compatObjectFormat.sh\n+++ b/t/t1016-compatObjectFormat.sh\n@@ -228,6 +228,20 @@ do\n \t\ttest_copy_bytes 8 actual &&\n \t\ttest_cmp expect actual\n \t'\n+\n+\ttest_expect_success 'add a large file streamed into a pack' '\n+\t\ttest_when_finished \"rm -rf large-repo\" &&\n+\t\tcreate_compat_repo $hash large-repo &&\n+\t\tgit -C large-repo config set core.bigFileThreshold 1k &&\n+\t\ttest-tool genrandom large 5000 >large-repo/large &&\n+\t\tgit -C large-repo add large &&\n+\t\tls large-repo/.git/objects/pack/pack-*.pack &&\n+\t\tgit -C large-repo hash-object large >expect &&\n+\t\tgit -C large-repo rev-parse :large >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tgit -C large-repo cat-file blob :large >actual &&\n+\t\ttest_cmp large-repo/large actual\n+\t'\n done\n cd \"$base\"\n \n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555967","messageId":"20261023072559.3204273-10-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 09/13] packfile: add support for reading pack index v3","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:54Z","receivedAt":"2026-10-23T07:25:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nNow that we can write pack index v3, we'll probably also want to read\nit. Let's add support for that by making our bsearch function more\ngeneric to handle lookups of any size and then using that both for older\nversion and for pack index v3.\n\nTo make lookups easier, let's store several offsets we compute from\nvalues per format so we don't need to look them up when reading. Since\nour pack may be larger than 4 GiB, let's make the crc_offset field\nlarger. We can then use it when finding the offset into the file, since\nit will be generic over both v2 and v3 indices.\n\nA v3 index may contain data for only one hash algorithm, so just fail if\nwe look up an object ID in an algorithm for which the index has no data.\n\nIn the pack revindex, we use the order map for the algorithm in question\nto look up the proper offset into the pack index, since we're iterating\nover objects in the offset table, which is sorted by object ID, not pack\norder.\n\nLet's also update t5302 to generate this new format and disable the old\nformats when we're using a compatibility hash algorithm, since the old\nformats can't represent the SHA-1 values.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n builtin/index-pack.c | 2 +-\n hash-lookup.c | 47 +++++++++++++----\n hash-lookup.h | 19 ++++++-\n pack-check.c | 4 +-\n pack-revindex.c | 21 ++++++--\n packfile.c | 117 +++++++++++++++++++++++++++++++++++++-----\n packfile.h | 18 +++++++\n t/t5302-pack-index.sh | 71 +++++++++++++------------\n 8 files changed, 238 insertions(+), 61 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 723595ae03..4db3f18951 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1841,7 +1841,7 @@ static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)\n \t/* Read the attributes from the existing idx file */\n \topts->version = p->index_version;\n \n-\tif (opts->version == 2)\n+\tif (opts->version >= 2)\n \t\tread_v2_anomalous_offsets(p, opts);\n \n \t/*\ndiff --git a/hash-lookup.c b/hash-lookup.c\nindex 5f808caa51..a77ccad8b5 100644\n--- a/hash-lookup.c\n+++ b/hash-lookup.c\n@@ -104,18 +104,13 @@ int oid_pos(const struct object_id *oid, const void *table, size_t nr,\n \treturn index_pos_to_insert_pos(lo);\n }\n \n-int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,\n-\t\t const unsigned char *table, size_t stride, uint32_t *result)\n+static int bsearch_hash_1(const unsigned char *hash, const unsigned char *table,\n+\t\t\t size_t stride, size_t cmpsize, uint32_t hi,\n+\t\t\t uint32_t lo, uint32_t *result)\n {\n-\tuint32_t hi, lo;\n-\n-\thi = ntohl(fanout_nbo[*hash]);\n-\tlo = ((*hash == 0x0) ? 0 : ntohl(fanout_nbo[*hash - 1]));\n-\n \twhile (lo < hi) {\n \t\tunsigned mi = lo + (hi - lo) / 2;\n-\t\tint cmp = hashcmp(table + mi * stride, hash,\n-\t\t\t\t the_repository->hash_algo);\n+\t\tint cmp = memcmp(table + mi * stride, hash, cmpsize);\n \n \t\tif (!cmp) {\n \t\t\tif (result)\n@@ -132,3 +127,37 @@ int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,\n \t\t*result = lo;\n \treturn 0;\n }\n+\n+int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,\n+\t\t const unsigned char *table, size_t stride, uint32_t *result)\n+{\n+\tuint32_t hi, lo;\n+\n+\thi = ntohl(fanout_nbo[*hash]);\n+\tlo = ((*hash == 0x0) ? 0 : ntohl(fanout_nbo[*hash - 1]));\n+\n+\treturn bsearch_hash_1(hash, table, stride, the_hash_algo->rawsz, hi, lo, result);\n+}\n+\n+/* full_oid is assumed to follow directly behind short_oid in memory. */\n+int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,\n+\t\t const uint32_t *pack_map, const unsigned char *short_oid,\n+\t\t size_t stride, size_t hashsz, uint32_t *result)\n+{\n+\tuint32_t obj_off, index;\n+\tint res = bsearch_hash_1(hash, short_oid, stride, stride, (full_oid - short_oid) / stride,\n+\t\t\t\t 0, &obj_off);\n+\tif (result)\n+\t\t*result = obj_off;\n+\tif (!res)\n+\t\treturn 0;\n+\t/*\n+\t * obj_off is now the offset in the shortened list of object names that\n+\t * we've found. We now need to look it up in the mapping of pack order\n+\t * to find the full object ID. index is the offset in pack order.\n+\t */\n+\tindex = ntohl(pack_map[obj_off]);\n+\tif (!memcmp(hash, full_oid + (hashsz * index), hashsz))\n+\t\treturn 1;\n+\treturn 0;\n+}\ndiff --git a/hash-lookup.h b/hash-lookup.h\nindex dbd71ebaf7..03a9be4929 100644\n--- a/hash-lookup.h\n+++ b/hash-lookup.h\n@@ -20,7 +20,7 @@ int oid_pos(const struct object_id *oid,\n * first byte is less than or equal to i\n * - table: a sorted list of hashes with optional extra information in between\n * - stride: distance between two consecutive elements in table (should be\n- * GIT_MAX_RAWSZ or greater)\n+ * GIT_MAX_RAWSZ or less)\n * - result: if not NULL, this function stores the element index of the\n * position found (if the search is successful) or the index of the least\n * element that is greater than hash (if the search is not successful)\n@@ -29,4 +29,21 @@ int oid_pos(const struct object_id *oid,\n */\n int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,\n \t\t const unsigned char *table, size_t stride, uint32_t *result);\n+\n+/*\n+ * Performs a binary search for hash, returning 1 if it is found and 0 if not.\n+ *\n+ * - hash is the object ID to search for.\n+ * - short_oid is a sorted table of equal-length prefixes of the hashes (each of\n+ * which is stride bytes in length.\n+ * - full_oid, which must follow in memory directly behind short_oid, is a list\n+ * of full object IDs in an arbitrary order (usually pack order); each item is\n+ * hashsz bytes in length.\n+ * - pack_map is a 4-byte mapping of indices of items in short_oid to full_oid.\n+ * - result may be NULL; if not, it is the index of the item in the list of\n+ * object IDs if found and the least element greater than it otherwise.\n+ */\n+int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,\n+\t\t const uint32_t *pack_map, const unsigned char *short_oid,\n+\t\t size_t stride, size_t hashsz, uint32_t *result);\n #endif\ndiff --git a/pack-check.c b/pack-check.c\nindex c7275f4b92..f69a137a18 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -44,7 +44,9 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs,\n \t} while (len);\n \n \tindex_crc = p->index_data;\n-\tindex_crc += 2 + 256 + (size_t)p->num_objects * (p->repo->hash_algo->rawsz/4) + nr;\n+\tif (p->index_version == 3)\n+\t\tnr = nth_packed_object_pack_order_algop(p, nr, p->repo->hash_algo);\n+\tindex_crc += (p->crc_offset / 4) + nr;\n \n \treturn data_crc != ntohl(*index_crc);\n }\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 62387ae632..fb81db8033 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -138,11 +138,25 @@ static void create_pack_revindex(struct packed_git *p)\n \tconst unsigned hashsz = p->repo->hash_algo->rawsz;\n \n \tALLOC_ARRAY(p->revindex, num_ent + 1);\n-\tindex += 4 * 256;\n \n-\tif (p->index_version > 1) {\n+\tif (p->index_version == 3) {\n \t\tconst uint32_t *off_32 =\n-\t\t\t(uint32_t *)(index + 8 + (size_t)p->num_objects * (hashsz + 4));\n+\t\t\t(uint32_t *)(index + st_add(p->crc_offset, st_mult(p->num_objects, 4)));\n+\t\tconst uint32_t *off_64 = off_32 + p->num_objects;\n+\t\tfor (i = 0; i < num_ent; i++) {\n+\t\t\tconst uint32_t pack_order = nth_packed_object_pack_order_algop(p, i, p->repo->hash_algo);\n+\t\t\tconst uint32_t off = ntohl(off_32[pack_order]);\n+\t\t\tconst uint32_t lowoff = off & 0x7fffffff;\n+\t\t\tif (!(off & 0x80000000)) {\n+\t\t\t\tp->revindex[i].offset = off;\n+\t\t\t} else {\n+\t\t\t\tp->revindex[i].offset = get_be64(off_64 + st_mult(lowoff, 2));\n+\t\t\t}\n+\t\t\tp->revindex[i].nr = i;\n+\t\t}\n+\t} else if (p->index_version == 2) {\n+\t\tconst uint32_t *off_32 =\n+\t\t\t(uint32_t *)(index + st_add(p->crc_offset, st_mult(p->num_objects, 4)));\n \t\tconst uint32_t *off_64 = off_32 + p->num_objects;\n \t\tfor (i = 0; i < num_ent; i++) {\n \t\t\tconst uint32_t off = ntohl(*off_32++);\n@@ -155,6 +169,7 @@ static void create_pack_revindex(struct packed_git *p)\n \t\t\tp->revindex[i].nr = i;\n \t\t}\n \t} else {\n+\t\tindex += 4 * 256;\n \t\tfor (i = 0; i < num_ent; i++) {\n \t\t\tconst uint32_t hl = *((uint32_t *)(index + (hashsz + 4) * i));\n \t\t\tp->revindex[i].offset = ntohl(hl);\ndiff --git a/packfile.c b/packfile.c\nindex b32b4cd75d..95207c0e19 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -88,7 +88,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\treturn -1;\n \t}\n \tidx_size = xsize_t(st.st_size);\n-\tif (idx_size < 4 * 256 + hashsz + hashsz) {\n+\tif (idx_size < hashsz * 3) {\n \t\tclose(fd);\n \t\treturn error(\"index file %s is too small\", path);\n \t}\n@@ -109,14 +109,14 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \tstruct pack_idx_header *hdr = idx_map;\n \tuint32_t version, nr, i, *index;\n \n-\tif (idx_size < 4 * 256 + hashsz + hashsz)\n+\tif (idx_size < hashsz * 3)\n \t\treturn error(\"index file %s is too small\", path);\n \tif (!idx_map)\n \t\treturn error(\"empty data\");\n \n \tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n \t\tversion = ntohl(hdr->idx_version);\n-\t\tif (version < 2 || version > 2)\n+\t\tif (version < 2 || version > 3)\n \t\t\treturn error(\"index file %s is version %\"PRIu32\n \t\t\t\t \" and is not supported by this binary\"\n \t\t\t\t \" (try upgrading GIT to a newer version)\",\n@@ -124,15 +124,20 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \t} else\n \t\tversion = 1;\n \n+\tif (version < 3 && idx_size < 4 * 256 + hashsz + hashsz)\n+\t\treturn error(\"index file %s is too small\", path);\n+\n \tnr = 0;\n \tindex = idx_map;\n \tif (version > 1)\n \t\tindex += 2; /* skip index header */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tuint32_t n = ntohl(index[i]);\n-\t\tif (n < nr)\n-\t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\tnr = n;\n+\tif (version <= 2) {\n+\t\tfor (i = 0; i < 256; i++) {\n+\t\t\tuint32_t n = ntohl(index[i]);\n+\t\t\tif (n < nr)\n+\t\t\t\treturn error(\"non-monotonic index %s\", path);\n+\t\t\tnr = n;\n+\t\t}\n \t}\n \n \tif (version == 1) {\n@@ -174,6 +179,51 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \t\t (sizeof(off_t) <= 4))\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n \t\tp->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));\n+\t} else if (version == 3) {\n+\t\tuint32_t hdrlen, num_formats, trailer_offset, main_algo;\n+\t\tconst uint32_t *q = index;\n+\t\tint i;\n+\n+\t\thdrlen = ntohl(*q++);\n+\t\tif (idx_size < hdrlen)\n+\t\t\treturn error(\"v3 index has truncated header\");\n+\n+\t\tnr = ntohl(*q++);\n+\t\tnum_formats = ntohl(*q++);\n+\t\tif (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))\n+\t\t\treturn error(\"too many formats for v3 index\");\n+\t\tmemset(p->formats, 0, sizeof(p->formats));\n+\t\tfor (i = 0; i < num_formats; i++) {\n+\t\t\tuint32_t format_id = ntohl(*q++);\n+\t\t\tuint32_t format = hash_algo_by_id(format_id);\n+\t\t\tstruct packed_git_format *pfp;\n+\n+\t\t\tif (format == GIT_HASH_UNKNOWN)\n+\t\t\t\treturn error(\"v3 index with unknown format_id 0x%08x\", format_id);\n+\t\t\t/*\n+\t\t\t * We store at the value of the constant minus 1 to\n+\t\t\t * avoid having to store a useless entry at offset 0.\n+\t\t\t */\n+\t\t\tpfp = &p->formats[format-1];\n+\n+\t\t\tpfp->short_name_len = ntohl(*q++);\n+\t\t\tpfp->data_offset = get_be64(q);\n+\t\t\tq += 2;\n+\n+\t\t\tpfp->full_oid_offset = pfp->data_offset +\n+\t\t\t\t\t (pfp->short_name_len * nr);\n+\t\t\tpfp->order_map_offset = pfp->full_oid_offset +\n+\t\t\t\t\t\t(hash_algos[format].rawsz * nr);\n+\t\t\tif (!i)\n+\t\t\t\tp->crc_offset = pfp->order_map_offset + nr * 4;\n+\t\t}\n+\t\tmain_algo = hash_algo_by_length(hashsz);\n+\t\tif (main_algo == GIT_HASH_UNKNOWN ||\n+\t\t !p->formats[main_algo - 1].data_offset)\n+\t\t\treturn error(\"v3 index with missing data for this repository's algorithm\");\n+\t\ttrailer_offset = get_be64(q);\n+\t\tif (trailer_offset + 2 * hashsz != idx_size)\n+\t\t\treturn error(\"v3 index with bad size\\n\");\n \t}\n \n \tp->index_version = version;\n@@ -1730,6 +1780,26 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,\n \treturn data;\n }\n \n+static int bsearch_pack_v3(const struct object_id *oid, const struct packed_git *p, uint32_t *result)\n+{\n+\tint algo = oid->algo ? oid->algo : hash_algo_by_ptr(p->repo->hash_algo);\n+\tconst struct packed_git_format *pfp = &p->formats[algo - 1];\n+\tconst unsigned char *data = p->index_data;\n+\n+\t/* This index has no data for the algorithm of the object ID. */\n+\tif (!pfp->data_offset) {\n+\t\tif (result)\n+\t\t\t*result = 0;\n+\t\treturn 0;\n+\t}\n+\n+\treturn bsearch_hash_v3(oid->hash, data + pfp->full_oid_offset,\n+\t\t\t (const uint32_t *)(data + pfp->order_map_offset),\n+\t\t\t data + pfp->data_offset,\n+\t\t\t pfp->short_name_len,\n+\t\t\t hash_algos[algo].rawsz, result);\n+}\n+\n int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32_t *result)\n {\n \tconst unsigned char *index_fanout = p->index_data;\n@@ -1740,6 +1810,9 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32\n \tif (!index_fanout)\n \t\tBUG(\"bsearch_pack called without a valid pack-index\");\n \n+\tif (p->index_version == 3)\n+\t\treturn bsearch_pack_v3(oid, p, result);\n+\n \tindex_lookup = index_fanout + 4 * 256;\n \tif (p->index_version == 1) {\n \t\tindex_lookup_width = hashsz + 4;\n@@ -1767,13 +1840,17 @@ int nth_packed_object_id(struct object_id *oid,\n \t}\n \tif (n >= p->num_objects)\n \t\treturn -1;\n-\tindex += 4 * 256;\n \tif (p->index_version == 1) {\n+\t\tindex += 4 * 256;\n \t\toidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),\n \t\t\tp->repo->hash_algo);\n-\t} else {\n-\t\tindex += 8;\n+\t} else if (p->index_version == 2) {\n+\t\tindex += 4 * 256 + 8;\n \t\toidread(oid, index + st_mult(hashsz, n), p->repo->hash_algo);\n+\t} else {\n+\t\tint algo = hash_algo_by_ptr(p->repo->hash_algo);\n+\t\tconst struct packed_git_format *pfp = &p->formats[algo-1];\n+\t\toidread(oid, index + pfp->full_oid_offset + (hashsz * n), p->repo->hash_algo);\n \t}\n \treturn 0;\n }\n@@ -1796,12 +1873,14 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n {\n \tconst unsigned char *index = p->index_data;\n \tconst unsigned int hashsz = p->repo->hash_algo->rawsz;\n-\tindex += 4 * 256;\n \tif (p->index_version == 1) {\n+\t\tindex += 4 * 256;\n \t\treturn ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));\n \t} else {\n \t\tuint32_t off;\n-\t\tindex += st_add(8, st_mult(p->num_objects, hashsz + 4));\n+\t\tif (p->index_version == 3)\n+\t\t\tn = nth_packed_object_pack_order_algop(p, n, p->repo->hash_algo);\n+\t\tindex += st_add(p->crc_offset, st_mult(p->num_objects, 4));\n \t\toff = ntohl(*((uint32_t *)(index + st_mult(4, n))));\n \t\tif (!(off & 0x80000000))\n \t\t\treturn off;\n@@ -1812,6 +1891,18 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n \t}\n }\n \n+uint32_t nth_packed_object_pack_order_algop(const struct packed_git *p, uint32_t n,\n+\t\t\t\t\t const struct git_hash_algo *algop)\n+{\n+\tconst unsigned char *index = p->index_data;\n+\tconst unsigned int algo = hash_algo_by_ptr(algop);\n+\tconst struct packed_git_format *pfp = &p->formats[algo-1];\n+\n+\treturn ntohl(*(uint32_t *)(index +\n+\t\t\t\t st_add(pfp->order_map_offset,\n+\t\t\t\t\t st_mult(4, n))));\n+}\n+\n off_t find_pack_entry_one(const struct object_id *oid,\n \t\t\t struct packed_git *p)\n {\ndiff --git a/packfile.h b/packfile.h\nindex ed3378c457..22278cc6c6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -1,6 +1,7 @@\n #ifndef PACKFILE_H\n #define PACKFILE_H\n \n+#include \"hash.h\"\n #include \"list.h\"\n #include \"object.h\"\n #include \"odb.h\"\n@@ -14,12 +15,22 @@\n struct object_info;\n struct odb_stream;\n \n+struct packed_git_format {\n+\t/* This is also the location of the short OID table. */\n+\tuint64_t data_offset;\n+\tuint64_t full_oid_offset;\n+\tuint64_t order_map_offset;\n+\tuint32_t short_name_len;\n+};\n+\n struct packed_git {\n \tstruct pack_window *windows;\n \toff_t pack_size;\n \tconst void *index_data;\n \tsize_t index_size;\n \tuint32_t num_objects;\n+\tuint32_t num_formats;\n+\tstruct packed_git_format formats[GIT_HASH_NALGOS - 1];\n \tsize_t crc_offset;\n \tstruct oidset bad_objects;\n \tint index_version;\n@@ -284,6 +295,13 @@ int nth_packed_object_id(struct object_id *, struct packed_git *, uint32_t n);\n */\n off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n \n+/*\n+ * Return the index into pack order of the nth object within the specified\n+ * packfile. The index must already be opened and it must be a v3 index.\n+ */\n+uint32_t nth_packed_object_pack_order_algop(const struct packed_git *, uint32_t n,\n+\t\t\t\t\t const struct git_hash_algo *algop);\n+\n /*\n * If the object named by oid is present in the specified packfile,\n * return its offset within the packfile; otherwise, return 0.\ndiff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\nindex 735de1023e..9ea0ed487e 100755\n--- a/t/t5302-pack-index.sh\n+++ b/t/t5302-pack-index.sh\n@@ -36,50 +36,55 @@ test_expect_success 'setup' '\n \tgit update-ref HEAD $commit\n '\n \n-test_expect_success 'pack-objects with index version 1' '\n+test_expect_success !COMPAT_HASH 'pack-objects with index version 1' '\n \tpack1=$(git pack-objects --index-version=1 test-1 file_101_foo1\n '\n \n-test_expect_success \\\n+test_expect_success !COMPAT_HASH \\\n \t'[index v1] 3) corrupted delta happily returned wrong data' '\n \ttest -f file_101_foo1 && ! cmp file_101 file_101_foo1\n '\n \n-test_expect_success \\\n+test_expect_success !COMPAT_HASH \\\n \t'[index v1] 4) confirm that the pack is actually corrupted' '\n \ttest_must_fail git fsck --full $commit\n '\n \n-test_expect_success \\\n+test_expect_success !COMPAT_HASH \\\n \t'[index v1] 5) pack-objects happily reuses corrupted data' '\n \tpack4=$(git pack-objects test-4 file_101_foo2\n '\n \n-test_expect_success \\\n+test_expect_success !COMPAT_HASH \\\n \t'[index v2] 3) corrupted delta happily returned wrong data' '\n \ttest -f file_101_foo2 && ! cmp file_101 file_101_foo2\n '\n \n-test_expect_success \\\n+test_expect_success !COMPAT_HASH \\\n \t'[index v2] 4) confirm that the pack is actually corrupted' '\n \ttest_must_fail git fsck --full $commit\n '\n \n-test_expect_success \\\n+test_expect_success !COMPAT_HASH \\\n \t'[index v2] 5) pack-objects refuses to reuse corrupted data' '\n \ttest_must_fail git pack-objects test-5 err &&\n \ttest_grep \"^warning:.* expected .tagger. line\" err\n '\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555968","messageId":"20261023072559.3204273-11-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 10/13] packfile: validate pack index v3 tables","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:55Z","receivedAt":"2026-10-23T07:25:55Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented loading a pack index v3, but didn't\ncheck much of its format when doing so.\n\nTo avoid reading outside a v3 index, if it's corrupt or crafted, let's\ncheck that its header, its format entries and all its tables lie within\nthe file, when loading it.\n\nLet's also reject duplicate formats and short names longer than the\nfull object names.\n\nThe values in the order maps are checked when they are read, rather\nthan when loading the index, so that loading stays cheap. A new test in\nt5313 ensures that an invalid order map entry is detected.\n\nIn the pack revindex, let's also check that the 8-byte offsets we read\nlie within the index.\n\nSigned-off-by: Christian Couder \n---\n hash-lookup.c | 5 ++-\n pack-revindex.c | 5 ++-\n packfile.c | 71 ++++++++++++++++++++++++++---------\n t/t5313-pack-bounds-checks.sh | 17 +++++++++\n 4 files changed, 78 insertions(+), 20 deletions(-)\n\ndiff --git a/hash-lookup.c b/hash-lookup.c\nindex a77ccad8b5..3f0725f7ec 100644\n--- a/hash-lookup.c\n+++ b/hash-lookup.c\n@@ -144,8 +144,9 @@ int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,\n \t\t const uint32_t *pack_map, const unsigned char *short_oid,\n \t\t size_t stride, size_t hashsz, uint32_t *result)\n {\n+\tuint32_t nr = (full_oid - short_oid) / stride;\n \tuint32_t obj_off, index;\n-\tint res = bsearch_hash_1(hash, short_oid, stride, stride, (full_oid - short_oid) / stride,\n+\tint res = bsearch_hash_1(hash, short_oid, stride, stride, nr,\n \t\t\t\t 0, &obj_off);\n \tif (result)\n \t\t*result = obj_off;\n@@ -157,6 +158,8 @@ int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,\n \t * to find the full object ID. index is the offset in pack order.\n \t */\n \tindex = ntohl(pack_map[obj_off]);\n+\tif (index >= nr)\n+\t\treturn 0;\n \tif (!memcmp(hash, full_oid + (hashsz * index), hashsz))\n \t\treturn 1;\n \treturn 0;\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex fb81db8033..4e6923cfd3 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -150,7 +150,10 @@ static void create_pack_revindex(struct packed_git *p)\n \t\t\tif (!(off & 0x80000000)) {\n \t\t\t\tp->revindex[i].offset = off;\n \t\t\t} else {\n-\t\t\t\tp->revindex[i].offset = get_be64(off_64 + st_mult(lowoff, 2));\n+\t\t\t\tconst uint32_t *ptr = off_64 + st_mult(lowoff, 2);\n+\n+\t\t\t\tcheck_pack_index_ptr(p, ptr);\n+\t\t\t\tp->revindex[i].offset = get_be64(ptr);\n \t\t\t}\n \t\t\tp->revindex[i].nr = i;\n \t\t}\ndiff --git a/packfile.c b/packfile.c\nindex 95207c0e19..f6e2db45d5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -180,50 +180,81 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n \t\tp->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));\n \t} else if (version == 3) {\n-\t\tuint32_t hdrlen, num_formats, trailer_offset, main_algo;\n+\t\tuint32_t hdrlen, num_formats, main_algo;\n+\t\tuint64_t trailer_offset;\n \t\tconst uint32_t *q = index;\n-\t\tint i;\n \n+\t\t/* Fixed part of the header (after signature and version) */\n \t\thdrlen = ntohl(*q++);\n-\t\tif (idx_size < hdrlen)\n-\t\t\treturn error(\"v3 index has truncated header\");\n-\n \t\tnr = ntohl(*q++);\n \t\tnum_formats = ntohl(*q++);\n \t\tif (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))\n-\t\t\treturn error(\"too many formats for v3 index\");\n+\t\t\treturn error(\"v3 index %s has an invalid number of formats: %\"PRIu32,\n+\t\t\t\t path, num_formats);\n+\n+\t\t/* Then 16 bytes per format and the 8-byte offset of the trailer */\n+\t\tif (hdrlen < 20 + num_formats * (4 + 4 + 8) + 8 ||\n+\t\t hdrlen > idx_size - 2 * hashsz)\n+\t\t\treturn error(\"v3 index %s has an invalid header length\", path);\n+\n+\t\ttrailer_offset = get_be64(q + num_formats * 4);\n+\t\tif (trailer_offset != idx_size - 2 * hashsz)\n+\t\t\treturn error(\"v3 index %s has a bad size\", path);\n+\n \t\tmemset(p->formats, 0, sizeof(p->formats));\n \t\tfor (i = 0; i < num_formats; i++) {\n \t\t\tuint32_t format_id = ntohl(*q++);\n \t\t\tuint32_t format = hash_algo_by_id(format_id);\n \t\t\tstruct packed_git_format *pfp;\n+\t\t\tuint64_t end;\n \n \t\t\tif (format == GIT_HASH_UNKNOWN)\n-\t\t\t\treturn error(\"v3 index with unknown format_id 0x%08x\", format_id);\n+\t\t\t\treturn error(\"v3 index %s has an unknown format_id 0x%08\"PRIx32,\n+\t\t\t\t\t path, format_id);\n \t\t\t/*\n \t\t\t * We store at the value of the constant minus 1 to\n \t\t\t * avoid having to store a useless entry at offset 0.\n \t\t\t */\n \t\t\tpfp = &p->formats[format-1];\n+\t\t\tif (pfp->data_offset)\n+\t\t\t\treturn error(\"v3 index %s has duplicate format_id 0x%08\"PRIx32,\n+\t\t\t\t\t path, format_id);\n \n \t\t\tpfp->short_name_len = ntohl(*q++);\n \t\t\tpfp->data_offset = get_be64(q);\n \t\t\tq += 2;\n \n+\t\t\tif (!pfp->short_name_len ||\n+\t\t\t pfp->short_name_len > hash_algos[format].rawsz)\n+\t\t\t\treturn error(\"v3 index %s has an invalid short name length\",\n+\t\t\t\t\t path);\n+\n+\t\t\tif (pfp->data_offset < hdrlen ||\n+\t\t\t pfp->data_offset > trailer_offset)\n+\t\t\t\treturn error(\"v3 index %s has an invalid table offset\", path);\n+\n+\t\t\t/* No overflow as pfp->data_offset < idx_size and nr < 2^32 */\n \t\t\tpfp->full_oid_offset = pfp->data_offset +\n-\t\t\t\t\t (pfp->short_name_len * nr);\n+\t\t\t\t\t (uint64_t)pfp->short_name_len * nr;\n \t\t\tpfp->order_map_offset = pfp->full_oid_offset +\n-\t\t\t\t\t\t(hash_algos[format].rawsz * nr);\n-\t\t\tif (!i)\n-\t\t\t\tp->crc_offset = pfp->order_map_offset + nr * 4;\n+\t\t\t\t\t\t(uint64_t)hash_algos[format].rawsz * nr;\n+\t\t\tend = pfp->order_map_offset + (uint64_t)nr * 4;\n+\n+\t\t\t/* After the first format tables: CRC32 and 4-byte offset tables */\n+\t\t\tif (!i) {\n+\t\t\t\tp->crc_offset = end;\n+\t\t\t\tend += (uint64_t)nr * (4 + 4); /* CRC32 + offset */\n+\t\t\t}\n+\n+\t\t\tif (pfp->order_map_offset % 4 || end > trailer_offset)\n+\t\t\t\treturn error(\"v3 index %s has invalid tables\", path);\n \t\t}\n+\n \t\tmain_algo = hash_algo_by_length(hashsz);\n \t\tif (main_algo == GIT_HASH_UNKNOWN ||\n \t\t !p->formats[main_algo - 1].data_offset)\n-\t\t\treturn error(\"v3 index with missing data for this repository's algorithm\");\n-\t\ttrailer_offset = get_be64(q);\n-\t\tif (trailer_offset + 2 * hashsz != idx_size)\n-\t\t\treturn error(\"v3 index with bad size\\n\");\n+\t\t\treturn error(\"v3 index %s is missing data for the main hash algorithm\",\n+\t\t\t\t path);\n \t}\n \n \tp->index_version = version;\n@@ -1897,10 +1928,14 @@ uint32_t nth_packed_object_pack_order_algop(const struct packed_git *p, uint32_t\n \tconst unsigned char *index = p->index_data;\n \tconst unsigned int algo = hash_algo_by_ptr(algop);\n \tconst struct packed_git_format *pfp = &p->formats[algo-1];\n+\tuint32_t pos = ntohl(*(uint32_t *)(index +\n+\t\t\t\t\t st_add(pfp->order_map_offset,\n+\t\t\t\t\t\t st_mult(4, n))));\n \n-\treturn ntohl(*(uint32_t *)(index +\n-\t\t\t\t st_add(pfp->order_map_offset,\n-\t\t\t\t\t st_mult(4, n))));\n+\tif (pos >= p->num_objects)\n+\t\tdie(_(\"invalid object position in pack index for %s (corrupt index?)\"),\n+\t\t p->pack_name);\n+\treturn pos;\n }\n \n off_t find_pack_entry_one(const struct object_id *oid,\ndiff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh\nindex 5be01260d7..ab2b0bb9c6 100755\n--- a/t/t5313-pack-bounds-checks.sh\n+++ b/t/t5313-pack-bounds-checks.sh\n@@ -45,6 +45,14 @@ extended_table () {\n \techo $(($(ofs_table \"$1\") + 4*$1))\n }\n \n+# Offset in a single-format v3 .idx with no extended offsets to the\n+# start of its order map. $1 is its number of objects, nr.\n+# The order map (4*nr) is followed by the CRC32 table (4*nr), the\n+# 4-byte offset table (4*nr) and by the trailer (2*rawsz).\n+v3_order_map () {\n+\techo $(($(wc -c <\"$idx\") - 2*$(test_oid rawsz) - 12*$1))\n+}\n+\n test_expect_success 'setup' '\n \ttest_oid_cache <<-EOF\n \toid000 sha1:1485\n@@ -171,6 +179,15 @@ test_expect_success 'bogus offset inside v2 extended table' '\n \ttest_must_fail git index-pack --verify $pack\n '\n \n+test_expect_success 'bogus order map entry (v3)' '\n+\tdo_pack $object --index-version=3 &&\n+\tmunge $idx $(v3_order_map 1) \"\\377\\377\\377\\377\" &&\n+\tclear_base &&\n+\ttest_must_fail git cat-file blob $object &&\n+\ttest_must_fail git cat-file --batch-all-objects --batch-check 2>err &&\n+\ttest_grep \"corrupt index\" err\n+'\n+\n test_expect_success 'bogus OFS_DELTA in packfile' '\n \t# Generate a pack with a delta in it.\n \tbase=$(test-tool genrandom foo 3000 | git hash-object --stdin -w) &&\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555969","messageId":"20261023072559.3204273-12-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 11/13] loose: map object IDs using pack index v3","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:56Z","receivedAt":"2026-10-23T07:25:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nRight now, we can only map an object if it exists in the loose object\nmap. However, we have a perfectly good way to look up packed objects,\nwhich is the pack index. Let's use that to look up any packed objects\nif they're not in the loose object map.\n\nIn order to do so, let's add a variant of nth_packed_object_id that\nhandles an arbitrary hash algorithm.\n\nOnly pack index v3 stores object IDs in more than one algorithm, so\nskip packs with an older index when mapping.\n\nAlso make the new variant fail when the index cannot provide the\nrequested object ID, for example because it has no data for one of the\nalgorithms.\n\nAdd a test in t1016 to check that packed objects are mapped using a v3\nindex, but not using a v2 index, when the loose object map has no entry\nfor them.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n odb/source-packed.c | 30 ++++++++++++++++++++++++++----\n packfile.c | 28 +++++++++++++++++++++++-----\n packfile.h | 8 ++++++++\n t/t1016-compatObjectFormat.sh | 27 +++++++++++++++++++++++++++\n 4 files changed, 84 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex ee1edfdb2b..4ecc2de818 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -863,12 +863,34 @@ static void odb_source_packed_close(struct odb_source *source)\n \tpacked->midx = NULL;\n }\n \n-static int odb_source_packed_map_oid(struct odb_source *source UNUSED,\n-\t\t\t\t const struct object_id *src UNUSED,\n-\t\t\t\t const struct git_hash_algo *to UNUSED,\n-\t\t\t\t struct object_id *dest UNUSED,\n+static int odb_source_packed_map_oid(struct odb_source *source,\n+\t\t\t\t const struct object_id *src,\n+\t\t\t\t const struct git_hash_algo *to,\n+\t\t\t\t struct object_id *dest,\n \t\t\t\t enum odb_map_oid_flags flags UNUSED)\n {\n+\tstruct odb_source_packed *packed = odb_source_packed_downcast(source);\n+\tstruct repository *repo = packed->base.odb->repo;\n+\tstruct packfile_list_entry *e;\n+\n+\tfor (e = packfile_store_get_packs(packed); e; e = e->next) {\n+\t\tstruct packed_git *p = e->pack;\n+\t\tuint32_t off;\n+\t\tconst struct git_hash_algo *src_algo = src->algo ?\n+\t\t\t\t\t\t &hash_algos[src->algo] :\n+\t\t\t\t\t\t repo->hash_algo;\n+\n+\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\t\tcontinue;\n+\t\t/* Only v3 indices store object IDs in both algorithms. */\n+\t\tif (p->index_version < 3)\n+\t\t\tcontinue;\n+\t\tif (!bsearch_pack(src, p, &off))\n+\t\t\tcontinue;\n+\t\tif (nth_packed_object_id_algop(dest, p, off, src_algo, to))\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n \treturn -1;\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex f6e2db45d5..3bfd427c6c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1861,9 +1861,19 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32\n int nth_packed_object_id(struct object_id *oid,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t n)\n+{\n+\treturn nth_packed_object_id_algop(oid, p, n, p->repo->hash_algo,\n+\t\t\t\t\t p->repo->hash_algo);\n+}\n+\n+int nth_packed_object_id_algop(struct object_id *oid,\n+\t\t\t struct packed_git *p,\n+\t\t\t uint32_t n,\n+\t\t\t const struct git_hash_algo *src,\n+\t\t\t const struct git_hash_algo *dest)\n {\n \tconst unsigned char *index = p->index_data;\n-\tconst unsigned int hashsz = p->repo->hash_algo->rawsz;\n+\tconst unsigned int hashsz = dest->rawsz;\n \tif (!index) {\n \t\tif (open_pack_index(p))\n \t\t\treturn -1;\n@@ -1871,17 +1881,25 @@ int nth_packed_object_id(struct object_id *oid,\n \t}\n \tif (n >= p->num_objects)\n \t\treturn -1;\n+\tif (p->index_version < 3 && (src != dest || dest != p->repo->hash_algo))\n+\t\treturn -1; /* older indices only store main object IDs */\n \tif (p->index_version == 1) {\n \t\tindex += 4 * 256;\n \t\toidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),\n-\t\t\tp->repo->hash_algo);\n+\t\t\tdest);\n \t} else if (p->index_version == 2) {\n \t\tindex += 4 * 256 + 8;\n-\t\toidread(oid, index + st_mult(hashsz, n), p->repo->hash_algo);\n+\t\toidread(oid, index + st_mult(hashsz, n), dest);\n \t} else {\n-\t\tint algo = hash_algo_by_ptr(p->repo->hash_algo);\n+\t\tint algo = hash_algo_by_ptr(dest);\n \t\tconst struct packed_git_format *pfp = &p->formats[algo-1];\n-\t\toidread(oid, index + pfp->full_oid_offset + (hashsz * n), p->repo->hash_algo);\n+\t\tuint32_t pack_order_idx;\n+\n+\t\tif (!pfp->data_offset ||\n+\t\t !p->formats[hash_algo_by_ptr(src) - 1].data_offset)\n+\t\t\treturn -1; /* no data for one of the algorithms */\n+\t\tpack_order_idx = nth_packed_object_pack_order_algop(p, n, src);\n+\t\toidread(oid, index + st_add(pfp->full_oid_offset, st_mult(hashsz, pack_order_idx)), dest);\n \t}\n \treturn 0;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 22278cc6c6..73452b2a43 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -289,6 +289,14 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32\n */\n int nth_packed_object_id(struct object_id *, struct packed_git *, uint32_t n);\n \n+/*\n+ * The same as nth_packed_object_id, but the nth object is counted in src\n+ * and the object ID looked up is in dest.\n+ */\n+int nth_packed_object_id_algop(struct object_id *, struct packed_git *,\n+\t\t\t uint32_t n, const struct git_hash_algo *src,\n+\t\t\t const struct git_hash_algo *dest);\n+\n /*\n * Return the offset of the nth object within the specified packfile.\n * The index must already be opened.\ndiff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh\nindex af9532aa1f..030d6000b5 100755\n--- a/t/t1016-compatObjectFormat.sh\n+++ b/t/t1016-compatObjectFormat.sh\n@@ -242,6 +242,33 @@ do\n \t\tgit -C large-repo cat-file blob :large >actual &&\n \t\ttest_cmp large-repo/large actual\n \t'\n+\n+\ttest_expect_success 'packed objects are only mapped using pack index v3' '\n+\t\ttest_when_finished \"rm -rf map-repo v3.idx loose-object-idx\" &&\n+\t\tcreate_compat_repo $hash map-repo &&\n+\t\techo content >map-repo/file &&\n+\t\tgit -C map-repo add file &&\n+\t\tgit -C map-repo commit --message A &&\n+\t\toid=$(git -C map-repo rev-parse HEAD) &&\n+\t\tgit -C map-repo rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >expect &&\n+\t\tgit -C map-repo repack -a -d &&\n+\t\tpack=$(echo map-repo/.git/objects/pack/pack-*.pack) &&\n+\t\tgit -C map-repo index-pack --index-version=3 \\\n+\t\t\t-o \"$(pwd)/v3.idx\" \"$(pwd)/$pack\" &&\n+\t\tmv map-repo/.git/objects/loose-object-idx loose-object-idx &&\n+\n+\t\t# A v2 index only has object IDs in the main algorithm, so it\n+\t\t# must not provide a mapping.\n+\t\tgit -C map-repo rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >actual &&\n+\t\t! test_cmp expect actual &&\n+\t\ttest $(wc -c actual &&\n+\t\ttest_cmp expect actual\n+\t'\n done\n cd \"$base\"\n \n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555970","messageId":"20261023072559.3204273-13-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 12/13] packfile: allow looking up compat object IDs in packs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:57Z","receivedAt":"2026-10-23T07:25:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nNow that we have the ability to store compatibility object IDs in pack\nindex v3, we also want to be able to look up these object IDs. Our\ndisambiguation code can work with a particular hash algorithm or with\nany hash algorithm, the most notable instance of the latter case being\nwhen git rev-parse is invoked with --output-object-format.\n\nWe already handle the single algorithm case just fine, but we don't\ngracefully handle resolving an object ID in any algorithm. Solve this\nby iterating through the algorithms and, if the algorithm is both one of\nthe desired ones and one our repository supports, then try looking up\nthe object IDs for that algorithm. Use an intermediate variable so that\nwe can pass the correct algorithm in the object ID's algo member, which\nwill cause the correct tables to be looked up.\n\nWrite this check as two if statements because the logic is complex and\nit is easier to understand as two statements. The compiler will\noptimize it appropriately.\n\nWhen looking up an object ID, we find positions in the table sorted by\nthat object ID's algorithm. So read the object IDs there for that\nalgorithm with nth_packed_object_id_algop(), and add\nnth_packed_object_offset_algop() to get their offsets.\n\nSkip the compatibility algorithm for packs with an index older than v3,\nas these only store object IDs in the main algorithm.\n\nAdd a test that looks up an abbreviated compatibility object ID of a\npacked object.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n odb/source-packed.c | 83 +++++++++++++++++++++++------------\n packfile.c | 13 +++++-\n packfile.h | 7 +++\n t/t1016-compatObjectFormat.sh | 30 +++++++++++++\n 4 files changed, 104 insertions(+), 29 deletions(-)\n\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 4ecc2de818..8b16e8da29 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -295,37 +295,64 @@ static int for_each_prefixed_object_in_pack(\n \tint len = opts->prefix_hex_len > p->repo->hash_algo->hexsz ?\n \t\tp->repo->hash_algo->hexsz : opts->prefix_hex_len;\n \tint ret;\n+\tstruct object_id wanted;\n \n \tnum = p->num_objects;\n-\tbsearch_pack(opts->prefix, p, &first);\n \n-\t/*\n-\t * At this point, \"first\" is the location of the lowest object\n-\t * with an object name that could match \"bin_pfx\". See if we have\n-\t * 0, 1 or more objects that actually match(es).\n-\t */\n-\tfor (i = first; i < num; i++) {\n-\t\tstruct object_id oid;\n-\n-\t\tnth_packed_object_id(&oid, p, i);\n-\t\tif (!match_hash(len, opts->prefix->hash, oid.hash))\n-\t\t\tbreak;\n-\n-\t\tif (data->request) {\n-\t\t\tstruct object_info oi = *data->request;\n-\t\t\toff_t offset = nth_packed_object_offset(p, i);\n-\n-\t\t\tret = packed_object_info(source, p, offset, &oi);\n-\t\t\tif (ret)\n-\t\t\t\tgoto out;\n-\n-\t\t\tret = data->cb(&oid, &oi, data->cb_data);\n-\t\t\tif (ret)\n-\t\t\t\tgoto out;\n-\t\t} else {\n-\t\t\tret = data->cb(&oid, NULL, data->cb_data);\n-\t\t\tif (ret)\n-\t\t\t\tgoto out;\n+\tfor (uint32_t algo = 1; algo < GIT_HASH_NALGOS; algo++) {\n+\t\tconst struct git_hash_algo *algop = &hash_algos[algo];\n+\n+\t\t/*\n+\t\t * Skip this algorithm unless it's the desired one or we've\n+\t\t * asked for any algorithm.\n+\t\t */\n+\t\tif (opts->prefix->algo != GIT_HASH_UNKNOWN &&\n+\t\t opts->prefix->algo != algo)\n+\t\t\tcontinue;\n+\t\t/*\n+\t\t * Skip this algorithm unless it's the main algorithm or we have\n+\t\t * a compatibility algorithm and it's that algorithm.\n+\t\t */\n+\t\tif (algop != p->repo->hash_algo &&\n+\t\t (!p->repo->compat_hash_algo ||\n+\t\t algop != p->repo->compat_hash_algo))\n+\t\t\tcontinue;\n+\t\tif (p->index_version < 3 && algop != p->repo->hash_algo)\n+\t\t\tcontinue;\n+\n+\t\toidcpy(&wanted, opts->prefix);\n+\t\twanted.algo = algo;\n+\n+\t\tbsearch_pack(&wanted, p, &first);\n+\n+\t\t/*\n+\t\t * At this point, \"first\" is the location of the lowest object\n+\t\t * with an object name that could match \"bin_pfx\". See if we have\n+\t\t * 0, 1 or more objects that actually match(es).\n+\t\t */\n+\t\tfor (i = first; i < num; i++) {\n+\t\t\tstruct object_id oid;\n+\n+\t\t\tif (nth_packed_object_id_algop(&oid, p, i, algop, algop) ||\n+\t\t\t !match_hash(len, opts->prefix->hash, oid.hash))\n+\t\t\t\tbreak;\n+\n+\t\t\tif (data->request) {\n+\t\t\t\tstruct object_info oi = *data->request;\n+\t\t\t\toff_t offset = nth_packed_object_offset_algop(p, i, algop);\n+\n+\t\t\t\tret = packed_object_info(source, p, offset, &oi);\n+\t\t\t\tif (ret)\n+\t\t\t\t\tgoto out;\n+\n+\t\t\t\tret = data->cb(&oid, &oi, data->cb_data);\n+\t\t\t\tif (ret)\n+\t\t\t\t\tgoto out;\n+\t\t\t} else {\n+\t\t\t\tret = data->cb(&oid, NULL, data->cb_data);\n+\t\t\t\tif (ret)\n+\t\t\t\t\tgoto out;\n+\t\t\t}\n \t\t}\n \t}\n \ndiff --git a/packfile.c b/packfile.c\nindex 3bfd427c6c..3f3eb27e42 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1919,16 +1919,27 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n }\n \n off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n+{\n+\treturn nth_packed_object_offset_algop(p, n, p->repo->hash_algo);\n+}\n+\n+off_t nth_packed_object_offset_algop(const struct packed_git *p, uint32_t n,\n+\t\t\t\t const struct git_hash_algo *algop)\n {\n \tconst unsigned char *index = p->index_data;\n \tconst unsigned int hashsz = p->repo->hash_algo->rawsz;\n+\n+\tif (p->index_version < 3 && algop != p->repo->hash_algo)\n+\t\tBUG(\"pack index v%d only stores main object IDs\",\n+\t\t p->index_version);\n+\n \tif (p->index_version == 1) {\n \t\tindex += 4 * 256;\n \t\treturn ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));\n \t} else {\n \t\tuint32_t off;\n \t\tif (p->index_version == 3)\n-\t\t\tn = nth_packed_object_pack_order_algop(p, n, p->repo->hash_algo);\n+\t\t\tn = nth_packed_object_pack_order_algop(p, n, algop);\n \t\tindex += st_add(p->crc_offset, st_mult(p->num_objects, 4));\n \t\toff = ntohl(*((uint32_t *)(index + st_mult(4, n))));\n \t\tif (!(off & 0x80000000))\ndiff --git a/packfile.h b/packfile.h\nindex 73452b2a43..40760295ee 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -303,6 +303,13 @@ int nth_packed_object_id_algop(struct object_id *, struct packed_git *,\n */\n off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n \n+/*\n+ * The same as nth_packed_object_offset, but the nth object is counted in\n+ * algop. For indices older than v3, algop must be the main algorithm.\n+ */\n+off_t nth_packed_object_offset_algop(const struct packed_git *, uint32_t n,\n+\t\t\t\t const struct git_hash_algo *algop);\n+\n /*\n * Return the index into pack order of the nth object within the specified\n * packfile. The index must already be opened and it must be a v3 index.\ndiff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh\nindex 030d6000b5..97a299f088 100755\n--- a/t/t1016-compatObjectFormat.sh\n+++ b/t/t1016-compatObjectFormat.sh\n@@ -269,6 +269,36 @@ do\n \t\tgit -C map-repo rev-parse --output-object-format=$(compat_hash $hash) \"$oid\" >actual &&\n \t\ttest_cmp expect actual\n \t'\n+\n+\ttest_expect_success 'abbreviated compat object IDs are found using pack index v3' '\n+\t\ttest_when_finished \"rm -rf abbrev-repo v3.idx\" &&\n+\t\tcreate_compat_repo $hash abbrev-repo &&\n+\t\techo content >abbrev-repo/file &&\n+\t\tgit -C abbrev-repo add file &&\n+\t\tgit -C abbrev-repo commit --message A &&\n+\t\tcompat_oid=$(git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) HEAD) &&\n+\t\tabbrev=$(echo $compat_oid | cut -c 1-12) &&\n+\n+\t\t# A multi-pack-index only stores object IDs in the main\n+\t\t# algorithm.\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git -C abbrev-repo repack -a -d &&\n+\t\tpack=$(echo abbrev-repo/.git/objects/pack/pack-*.pack) &&\n+\t\tgit -C abbrev-repo index-pack --index-version=3 \\\n+\t\t\t-o \"$(pwd)/v3.idx\" \"$(pwd)/$pack\" &&\n+\t\trm abbrev-repo/.git/objects/loose-object-idx &&\n+\n+\t\t# A v2 index only has object IDs in the main algorithm.\n+\t\ttest_must_fail git -C abbrev-repo cat-file -t $abbrev &&\n+\n+\t\trm -f \"${pack%.pack}.idx\" \"${pack%.pack}.rev\" &&\n+\t\tmv v3.idx \"${pack%.pack}.idx\" &&\n+\t\techo commit >expect &&\n+\t\tgit -C abbrev-repo cat-file -t $abbrev >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\techo $compat_oid >expect &&\n+\t\tgit -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) $abbrev >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n done\n cd \"$base\"\n \n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"555971","messageId":"20261023072559.3204273-14-christian.couder@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"[PATCH v2 13/13] doc: move pack index v3 documentation","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-23T07:25:58Z","receivedAt":"2026-10-23T07:25:58Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: \"brian m. carlson\" \n\nRight now, we have our pack index v3 documentation in the hash function\ntransition document. However, now that we've implemented pack index v3,\npeople will look for the index documentation in the normal place, so\ncopy it there.\n\nWe preserve it in the hash function transition document for\ncompatibility.\n\nSigned-off-by: brian m. carlson \nSigned-off-by: Christian Couder \n---\n Documentation/gitformat-pack.adoc | 75 +++++++++++++++++++++++++++++++\n 1 file changed, 75 insertions(+)\n\ndiff --git a/Documentation/gitformat-pack.adoc b/Documentation/gitformat-pack.adoc\nindex 3416edceab..19afbc881e 100644\n--- a/Documentation/gitformat-pack.adoc\n+++ b/Documentation/gitformat-pack.adoc\n@@ -318,6 +318,81 @@ Pack file entry: <+\n \n Index checksum of all of the above.\n \n+== Version 3 pack-*.idx files have the format:\n+\n+ - A header appears at the beginning and consists of the following:\n+ * The 4-byte pack index signature: '\\377t0c'\n+ * 4-byte version number: 3\n+ * 4-byte length of the header section, including the signature and\n+ version number\n+ * 4-byte number of objects contained in the pack\n+ * 4-byte number of object formats in this pack index: 1 or 2\n+ * For each object format:\n+ ** 4-byte format identifier (e.g., 'sha1' for SHA-1)\n+ ** 4-byte length in bytes of shortened object names. This is the\n+ shortest possible length needed to make names in the shortened\n+ object name table unambiguous.\n+ ** 8-byte integer, recording where tables relating to this format\n+ are stored in this index file, as an offset from the beginning.\n+ * 8-byte offset to the trailer from the beginning of this file.\n+ * Zero or more additional key/value pairs (4-byte key, 4-byte\n+ value). Only one key is supported: 'PSRC'. See the \"Loose objects\n+ and unreachable objects\" section of the hash function transition\n+ document (`Documentation/technical/hash-function-transition.adoc`)\n+ for supported values and how this is used. All other keys are reserved. Readers must ignore\n+ unrecognized keys.\n+ - Zero or more NUL bytes. This can optionally be used to improve the\n+ alignment of the full object name table below.\n+ - Tables for the first object format:\n+ * A sorted table of shortened object names. These are prefixes of\n+ the names of all objects in this pack file, packed together\n+ without offset values to reduce the cache footprint of the binary\n+ search for a specific object name.\n+\n+ * A table of full object names in pack order. This allows resolving\n+ a reference to \"the nth object in the pack file\" (from a\n+ reachability bitmap or from the next table of another object\n+ format) to its object name.\n+\n+ * A table of 4-byte values mapping object name order to pack order.\n+ For an object in the table of sorted shortened object names, the\n+ value at the corresponding index in this table is the index in the\n+ previous table for that same object.\n+ This can be used to look up the object in reachability bitmaps or\n+ to look up its name in another object format.\n+\n+ * A table of 4-byte CRC32 values of the packed object data, in the\n+ order that the objects appear in the pack file. This is to allow\n+ compressed data to be copied directly from pack to pack during\n+ repacking without undetected data corruption.\n+\n+ * A table of 4-byte offset values. The index of this table in pack order\n+ indicates where that object can be found in the pack file. These are\n+ usually 31-bit pack file offsets, but large offsets are encoded as\n+ an index into the next table with the most significant bit set.\n+\n+ * A table of 8-byte offset entries (empty for pack files less than\n+ 2 GiB). Pack files are organized with heavily used objects toward\n+ the front, so most object references should not need to refer to\n+ this table.\n+ - Zero or more NUL bytes.\n+ - Tables for the second object format, with the same layout as above,\n+ up to and not including the table of CRC32 values.\n+ - Zero or more NUL bytes.\n+ - The trailer consists of the following:\n+ * A copy of the hash checksum at the end of the corresponding\n+ packfile.\n+\n+ * Hash checksum of all of the above.\n+\n+Note that the two hashes at the end of the index file are in whatever the\n+repository's main algorithm is. In the usual case when there are multiple\n+algorithms, the main algorithm will be SHA-256 and the compatibility algorithm\n+will be SHA-1. Git always writes the main algorithm first, but doesn't require this.\n+\n+All 4-byte numbers are in network order and must be 4-byte aligned in the file,\n+so the NUL padding may be required in some cases.\n+\n == pack-*.rev files have the format:\n \n - A 4-byte magic number '0x52494458' ('RIDX').\n-- \n2.56.0.215.g47fdf2208c\n\n\n"},{"id":"556001","messageId":"16FCA7B0-02D5-4887-B2B0-3B794ECE13D9@gmail.com","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"Re: [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3","fromName":"Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2026-10-23T12:43:50Z","receivedAt":"2026-10-23T12:43:50Z","isPatch":true,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"\n\n> Le 23 oct. 2026 à 03:30, Christian Couder a écrit :\n> \n> This series is the next part of brian m. carlson's SHA-1/SHA-256\n> interoperability work.\n> \n> ## Changes compared to v1\n> \n> Thanks to Junio for his comments on this series.\n> \n> The only change compared to v1 is that I added my `Signed-off-by: ...`\n> to patches 2, 3 and 5 (that are otherwise unchanged compared to\n> brian's original patches).\n\nIn such a long cover letter, it might be helpful to present the changes from prior versions much earlier ;)\n"},{"id":"556051","messageId":"xmqq1p9gbgwz.fsf@gitster.g","threadId":"66639","inReplyTo":"20261023072559.3204273-1-christian.couder@gmail.com","subject":"Re: [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-23T15:00:44Z","receivedAt":"2026-10-23T15:00:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> This series is the next part of brian m. carlson's SHA-1/SHA-256\n> interoperability work.\n>\n> It adds support for reading and writing pack index v3, which stores\n> object IDs in both the main and the compatibility hash\n> algorithms. Interoperability needs it so that packed objects can be\n> mapped between algorithms, the same way the loose object map already\n> does for loose objects.\n> ...\n> My `Signed-off-by: ...` was added to all the patches. AI assistance\n> (mostly Claude Opus 5.5) was used.\n> ...\n> Except that I added my `Signed-off-by: ...` to them, patches 2, 3 and\n> 5 are unchanged.\n> ## Changes compared to v1\n>\n> Thanks to Junio for his comments on this series.\n>\n> The only change compared to v1 is that I added my `Signed-off-by: ...`\n> to patches 2, 3 and 5 (that are otherwise unchanged compared to\n> brian's original patches).\n\nWill replace.\n\nNext time please make sure that the cover letter for a new\nround is a reply to the cover letter of either immediate previous\niteration or the first iteration (and all patches replies to the\ncover letter of the new round). When you send a new round as a\nnew thread unrelated that is unrelated to the previous round's\npatches, automation will not quite work well.\n\nThanks.\n\n"},{"id":"556503","messageId":"CAP8UFD3qSAnpNBxrBu0ow5roTv6wzqAudUc9_bOR9EK92ZYMiQ@mail.gmail.com","threadId":"66639","inReplyTo":"xmqq1p9gbgwz.fsf@gitster.g","subject":"Re: [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T12:53:28Z","receivedAt":"2026-10-27T12:53:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Oct 23, 2026 at 5:00 PM Junio C Hamano wrote:\n\n> Will replace.\n>\n> Next time please make sure that the cover letter for a new\n> round is a reply to the cover letter of either immediate previous\n> iteration or the first iteration (and all patches replies to the\n> cover letter of the new round). When you send a new round as a\n> new thread unrelated that is unrelated to the previous round's\n> patches, automation will not quite work well.\n\nYeah, sorry. I am going to try a new approach using a\nsendemail-validate hook and scripts (see\nhttps://gitlab.com/chriscool/check-send-email), which can also be used\nwith b4, to avoid most common mistakes.\n\n"}]}