{"thread":{"id":"66744","subject":"[PATCH 0/6] object-file: move logic related to the loose backend","startedAt":"2026-11-02T11:20:16Z","lastAt":"2026-11-02T11:20:22Z","messageCount":7,"participants":["Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"557158","messageId":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":null,"subject":"[PATCH 0/6] object-file: move logic related to the loose backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:16Z","receivedAt":"2026-11-02T11:20:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series moves logic that relates to the loose backend\nfrom \"object-file.c\" into \"odb/source-loose.c\". On the one hand this\nmakes the backend more self-contained. On the other hand, this also lets\nus quickly identify exactly those code units that still access internal\nimplementation details of the \"loose\" backend by grepping for those\nfiles that include \"odb/source-loose.h\".\n\nThe second reason is in fact the main driver. I've got two more patch\nseries pending that do the same for the \"packed\" and \"files\" backends.\nWith those changes in it becomes trivial to do a quick \"reality\" check\nof which subsystems are fully backend agnostic and which ones still need\nmore work.\n\nThe series is built on top of ebdc087dc99 (The 13th batch, 2026-11-01),\nbut it conflicts with bc/restrict-hex-to-lowercase. This conflict can be\neasily resolved by deleting the conflicting section in \"object-file.c\" and\nadding this diff in \"odb/source-loose.c\":\n\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 54f680d6d78..e92da0d8b2e 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -81,7 +81,7 @@ static int for_each_file_in_obj_subdir(unsigned int subdir_nr,\n \t\tstrbuf_add(path, de->d_name, namelen);\n \t\tif (namelen == algop->hexsz - 2 &&\n \t\t    !hex_to_bytes(oid.hash + 1, de->d_name,\n-\t\t\t\t  algop->rawsz - 1)) {\n+\t\t\t\t  algop->rawsz - 1, HEX_KIND_OID)) {\n \t\t\toid_set_algo(&oid, algop);\n \t\t\tmemset(oid.hash + algop->rawsz, 0,\n \t\t\t       GIT_MAX_RAWSZ - algop->rawsz);\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n      object-file: move logic to read loose objects into backend\n      object-file: move logic to derive loose object paths into backend\n      object-file: move iteration over loose files into backend\n      odb/source-loose: deduplicate logic to mmap objects\n      object-file: stop including \"odb/source-loose.h\"\n      object-file: stop including \"git-zlib.h\"\n\n apply.c                 |   1 +\n builtin/count-objects.c |   1 +\n builtin/fast-import.c   |   1 +\n builtin/gc.c            |   1 +\n builtin/index-pack.c    |   1 +\n builtin/pack-objects.c  |   2 +\n builtin/prune.c         |   1 +\n diff.c                  |   1 +\n http-walker.c           |   1 +\n http.c                  |   1 +\n loose.c                 |   1 +\n object-file.c           | 419 +---------------------------------------------\n object-file.h           | 100 +----------\n odb/source-loose.c      | 429 ++++++++++++++++++++++++++++++++++++++++++++++--\n odb/source-loose.h      |  41 +++++\n pack-check.c            |   1 +\n packfile.c              |   1 +\n prune-packed.c          |   1 +\n setup.c                 |   1 +\n 19 files changed, 476 insertions(+), 529 deletions(-)\n\n\n---\nbase-commit: ebdc087dc99d68bb340c282b2ba5f722def28d19\nchange-id: 20261013-pks-odb-loose-cleanups-fb7e5b890389\n\n\n"},{"id":"557159","messageId":"20261102-pks-odb-loose-cleanups-v1-1-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","subject":"[PATCH 1/6] object-file: move logic to read loose objects into backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:17Z","receivedAt":"2026-11-02T11:20:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The logic to read loose objects is still contained in \"object-file.c\"\neven though it's an implementation detail of the \"loose\" backend, and\nthat backend is nowadays also the only remaining caller.\n\nMove the logic around and remove the accompanying declarations.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n object-file.c      | 300 ---------------------------------------------------\n object-file.h      |  45 --------\n odb/source-loose.c | 306 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 306 insertions(+), 345 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 132aaab6f84..c1b78a25c9f 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -151,179 +151,6 @@ int stream_object_signature(struct repository *r,\n \treturn !oideq(oid, &real_oid) ? -1 : 0;\n }\n \n-/*\n- * Map and close the given loose object fd. The path argument is used for\n- * error reporting.\n- */\n-static void *map_fd(int fd, const char *path, unsigned long *size)\n-{\n-\tvoid *map = NULL;\n-\tstruct stat st;\n-\n-\tif (!fstat(fd, &st)) {\n-\t\t*size = xsize_t(st.st_size);\n-\t\tif (!*size) {\n-\t\t\t/* mmap() is forbidden on empty files */\n-\t\t\terror(_(\"object file %s is empty\"), path);\n-\t\t\tclose(fd);\n-\t\t\treturn NULL;\n-\t\t}\n-\t\tmap = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\t}\n-\tclose(fd);\n-\treturn map;\n-}\n-\n-enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n-\t\t\t\t\t\t    unsigned char *map,\n-\t\t\t\t\t\t    unsigned long mapsize,\n-\t\t\t\t\t\t    void *buffer,\n-\t\t\t\t\t\t    unsigned long bufsiz)\n-{\n-\tint status;\n-\n-\t/* Get the data stream */\n-\tmemset(stream, 0, sizeof(*stream));\n-\tstream->next_in = map;\n-\tstream->avail_in = mapsize;\n-\tstream->next_out = buffer;\n-\tstream->avail_out = bufsiz;\n-\n-\tgit_inflate_init(stream);\n-\tobj_read_unlock();\n-\tstatus = git_inflate(stream, 0);\n-\tobj_read_lock();\n-\tif (status != Z_OK && status != Z_STREAM_END)\n-\t\treturn ULHR_BAD;\n-\n-\t/*\n-\t * Check if entire header is unpacked in the first iteration.\n-\t */\n-\tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn ULHR_OK;\n-\n-\t/*\n-\t * We have a header longer than MAX_HEADER_LEN.\n-\t */\n-\treturn ULHR_TOO_LONG;\n-}\n-\n-void *unpack_loose_rest(git_zstream *stream,\n-\t\t\tvoid *buffer, unsigned long size,\n-\t\t\tconst struct object_id *oid)\n-{\n-\tsize_t bytes = strlen(buffer) + 1, n;\n-\tunsigned char *buf = xmallocz(size);\n-\tint status = Z_OK;\n-\n-\tn = stream->total_out - bytes;\n-\tif (n > size)\n-\t\tn = size;\n-\tmemcpy(buf, (char *) buffer + bytes, n);\n-\tbytes = n;\n-\tif (bytes <= size) {\n-\t\t/*\n-\t\t * The above condition must be (bytes <= size), not\n-\t\t * (bytes < size).  In other words, even though we\n-\t\t * expect no more output and set avail_out to zero,\n-\t\t * the input zlib stream may have bytes that express\n-\t\t * \"this concludes the stream\", and we *do* want to\n-\t\t * eat that input.\n-\t\t *\n-\t\t * Otherwise we would not be able to test that we\n-\t\t * consumed all the input to reach the expected size;\n-\t\t * we also want to check that zlib tells us that all\n-\t\t * went well with status == Z_STREAM_END at the end.\n-\t\t */\n-\t\tstream->next_out = buf + bytes;\n-\t\tstream->avail_out = size - bytes;\n-\t\twhile (status == Z_OK) {\n-\t\t\tobj_read_unlock();\n-\t\t\tstatus = git_inflate(stream, Z_FINISH);\n-\t\t\tobj_read_lock();\n-\t\t}\n-\t}\n-\n-\tif (status != Z_STREAM_END) {\n-\t\terror(_(\"corrupt loose object '%s'\"), oid_to_hex(oid));\n-\t\tFREE_AND_NULL(buf);\n-\t} else if (stream->avail_in) {\n-\t\terror(_(\"garbage at end of loose object '%s'\"),\n-\t\t      oid_to_hex(oid));\n-\t\tFREE_AND_NULL(buf);\n-\t}\n-\n-\treturn buf;\n-}\n-\n-/*\n- * parse_loose_header() parses the starting \"<type> <len>\\0\" of an\n- * object. If it doesn't follow that format -1 is returned. To check\n- * the validity of the <type> populate the \"typep\" in the \"struct\n- * object_info\". It will be OBJ_BAD if the object type is unknown. The\n- * parsed <len> can be retrieved via \"oi->sizep\", and from there\n- * passed to unpack_loose_rest().\n- *\n- * We used to just use \"sscanf()\", but that's actually way\n- * too permissive for what we want to check. So do an anal\n- * object header parse by hand.\n- */\n-int parse_loose_header(const char *hdr, struct object_info *oi)\n-{\n-\tconst char *type_buf = hdr;\n-\tsize_t size;\n-\tint type, type_len = 0;\n-\n-\t/*\n-\t * The type can be of any size but is followed by\n-\t * a space.\n-\t */\n-\tfor (;;) {\n-\t\tchar c = *hdr++;\n-\t\tif (!c)\n-\t\t\treturn -1;\n-\t\tif (c == ' ')\n-\t\t\tbreak;\n-\t\ttype_len++;\n-\t}\n-\n-\ttype = type_from_string_gently(type_buf, type_len, 1);\n-\tif (oi->typep)\n-\t\t*oi->typep = type;\n-\n-\t/*\n-\t * The length must follow immediately, and be in canonical\n-\t * decimal format (ie \"010\" is not valid).\n-\t */\n-\tsize = *hdr++ - '0';\n-\tif (size > 9)\n-\t\treturn -1;\n-\tif (size) {\n-\t\tfor (;;) {\n-\t\t\tunsigned long c = *hdr - '0';\n-\t\t\tif (c > 9)\n-\t\t\t\tbreak;\n-\t\t\thdr++;\n-\t\t\tsize = st_add(st_mult(size, 10), c);\n-\t\t}\n-\t}\n-\n-\tif (oi->sizep)\n-\t\t*oi->sizep = size;\n-\n-\t/*\n-\t * The length must be followed by a zero byte\n-\t */\n-\tif (*hdr)\n-\t\treturn -1;\n-\n-\t/*\n-\t * The format is valid, but the type may still be bogus. The\n-\t * Caller needs to check its oi->typep.\n-\t */\n-\treturn 0;\n-}\n-\n #define CHECK_COLLISION_DEST_VANISHED -2\n \n static int check_collision(const char *source, const char *dest)\n@@ -1127,133 +954,6 @@ int for_each_loose_file_in_source(struct odb_source *source,\n \treturn r;\n }\n \n-static int check_stream_oid(git_zstream *stream,\n-\t\t\t    const char *hdr,\n-\t\t\t    unsigned long size,\n-\t\t\t    const char *path,\n-\t\t\t    const struct object_id *expected_oid,\n-\t\t\t    const struct git_hash_algo *algop)\n-{\n-\tstruct git_hash_ctx c;\n-\tstruct object_id real_oid;\n-\tunsigned char buf[4096];\n-\tunsigned long total_read;\n-\tint status = Z_OK;\n-\n-\tgit_hash_init(&c, algop);\n-\tgit_hash_update(&c, hdr, stream->total_out);\n-\n-\t/*\n-\t * We already read some bytes into hdr, but the ones up to the NUL\n-\t * do not count against the object's content size.\n-\t */\n-\ttotal_read = stream->total_out - strlen(hdr) - 1;\n-\n-\t/*\n-\t * This size comparison must be \"<=\" to read the final zlib packets;\n-\t * see the comment in unpack_loose_rest for details.\n-\t */\n-\twhile (total_read <= size &&\n-\t       (status == Z_OK ||\n-\t\t(status == Z_BUF_ERROR && !stream->avail_out))) {\n-\t\tstream->next_out = buf;\n-\t\tstream->avail_out = sizeof(buf);\n-\t\tif (size - total_read < stream->avail_out)\n-\t\t\tstream->avail_out = size - total_read;\n-\t\tstatus = git_inflate(stream, Z_FINISH);\n-\t\tgit_hash_update(&c, buf, stream->next_out - buf);\n-\t\ttotal_read += stream->next_out - buf;\n-\t}\n-\n-\tif (status != Z_STREAM_END) {\n-\t\terror(_(\"corrupt loose object '%s'\"), oid_to_hex(expected_oid));\n-\t\tgit_hash_discard(&c);\n-\t\treturn -1;\n-\t}\n-\tif (stream->avail_in) {\n-\t\terror(_(\"garbage at end of loose object '%s'\"),\n-\t\t      oid_to_hex(expected_oid));\n-\t\tgit_hash_discard(&c);\n-\t\treturn -1;\n-\t}\n-\n-\tgit_hash_final_oid(&real_oid, &c);\n-\tif (!oideq(expected_oid, &real_oid)) {\n-\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t      oid_to_hex(expected_oid));\n-\t\treturn -1;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-int read_loose_object(struct repository *repo,\n-\t\t      const char *path,\n-\t\t      const struct object_id *expected_oid,\n-\t\t      struct object_id *real_oid,\n-\t\t      void **contents,\n-\t\t      struct object_info *oi)\n-{\n-\tint ret = -1;\n-\tint fd;\n-\tvoid *map = NULL;\n-\tunsigned long mapsize;\n-\tgit_zstream stream;\n-\tchar hdr[MAX_HEADER_LEN];\n-\tsize_t *size = oi->sizep;\n-\n-\tfd = git_open(path);\n-\tif (fd >= 0)\n-\t\tmap = map_fd(fd, path, &mapsize);\n-\tif (!map) {\n-\t\terror_errno(_(\"unable to mmap %s\"), path);\n-\t\tgoto out;\n-\t}\n-\n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) != ULHR_OK) {\n-\t\terror(_(\"unable to unpack header of %s\"), path);\n-\t\tgoto out_inflate;\n-\t}\n-\n-\tif (parse_loose_header(hdr, oi) < 0) {\n-\t\terror(_(\"unable to parse header of %s\"), path);\n-\t\tgoto out_inflate;\n-\t}\n-\n-\tif (*oi->typep < 0) {\n-\t\terror(_(\"unable to parse type from header '%s' of %s\"),\n-\t\t      hdr, path);\n-\t\tgoto out_inflate;\n-\t}\n-\n-\tif (*oi->typep == OBJ_BLOB &&\n-\t    *size > repo_settings_get_big_file_threshold(repo)) {\n-\t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid,\n-\t\t\t\t     repo->hash_algo) < 0)\n-\t\t\tgoto out_inflate;\n-\t} else {\n-\t\t*contents = unpack_loose_rest(&stream, hdr, *size, expected_oid);\n-\t\tif (!*contents) {\n-\t\t\terror(_(\"unable to unpack contents of %s\"), path);\n-\t\t\tgoto out_inflate;\n-\t\t}\n-\t\thash_object_file(repo->hash_algo,\n-\t\t\t\t *contents, *size,\n-\t\t\t\t *oi->typep, real_oid);\n-\t\tif (!oideq(expected_oid, real_oid))\n-\t\t\tgoto out_inflate;\n-\t}\n-\n-\tret = 0; /* everything checks out */\n-\n-out_inflate:\n-\tgit_inflate_end(&stream);\n-out:\n-\tif (map)\n-\t\tmunmap(map, mapsize);\n-\treturn ret;\n-}\n-\n static int write_pack_lockfile(struct odb_source *source, const char *pack_name,\n \t\t\t       const char *keep_msg, struct tempfile **out)\n {\ndiff --git a/object-file.h b/object-file.h\nindex f44758c4f8b..65ab44542b0 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -21,7 +21,6 @@ enum {\n int index_fd(struct index_state *istate, struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n int index_path(struct index_state *istate, struct object_id *oid, const char *path, struct stat *st, unsigned flags);\n \n-struct object_info;\n struct odb_source;\n \n /*\n@@ -122,50 +121,6 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n int check_and_freshen_file(const char *fn, int freshen,\n \t\t\t   const time_t *mtime);\n \n-/*\n- * Open the loose object at path, check its hash, and return the contents,\n- * use the \"oi\" argument to assert things about the object, or e.g. populate its\n- * type, and size. If the object is a blob, then \"contents\" may return NULL,\n- * to allow streaming of large blobs.\n- *\n- * Returns 0 on success, negative on error (details may be written to stderr).\n- */\n-int read_loose_object(struct repository *repo,\n-\t\t      const char *path,\n-\t\t      const struct object_id *expected_oid,\n-\t\t      struct object_id *real_oid,\n-\t\t      void **contents,\n-\t\t      struct object_info *oi);\n-\n-enum unpack_loose_header_result {\n-\tULHR_OK,\n-\tULHR_BAD,\n-\tULHR_TOO_LONG,\n-};\n-\n-/**\n- * unpack_loose_header() initializes the data stream needed to unpack\n- * a loose object header.\n- *\n- * Returns:\n- *\n- * - ULHR_OK on success\n- * - ULHR_BAD on error\n- * - ULHR_TOO_LONG if the header was too long\n- *\n- * It will only parse up to MAX_HEADER_LEN bytes.\n- */\n-enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n-\t\t\t\t\t\t    unsigned char *map,\n-\t\t\t\t\t\t    unsigned long mapsize,\n-\t\t\t\t\t\t    void *buffer,\n-\t\t\t\t\t\t    unsigned long bufsiz);\n-void *unpack_loose_rest(git_zstream *stream,\n-\t\t\tvoid *buffer, unsigned long size,\n-\t\t\tconst struct object_id *oid);\n-\n-int parse_loose_header(const char *hdr, struct object_info *oi);\n-\n struct odb_transaction;\n \n /*\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 50cb388c352..96fc2ed213f 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -64,6 +64,162 @@ static int quick_has_loose(struct odb_source_loose *loose,\n \treturn !!oidtree_contains(odb_source_loose_cache(loose, oid), oid);\n }\n \n+enum unpack_loose_header_result {\n+\tULHR_OK,\n+\tULHR_BAD,\n+\tULHR_TOO_LONG,\n+};\n+\n+static enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t\t   unsigned char *map,\n+\t\t\t\t\t\t\t   unsigned long mapsize,\n+\t\t\t\t\t\t\t   void *buffer,\n+\t\t\t\t\t\t\t   unsigned long bufsiz)\n+{\n+\tint status;\n+\n+\t/* Get the data stream */\n+\tmemset(stream, 0, sizeof(*stream));\n+\tstream->next_in = map;\n+\tstream->avail_in = mapsize;\n+\tstream->next_out = buffer;\n+\tstream->avail_out = bufsiz;\n+\n+\tgit_inflate_init(stream);\n+\tobj_read_unlock();\n+\tstatus = git_inflate(stream, 0);\n+\tobj_read_lock();\n+\tif (status != Z_OK && status != Z_STREAM_END)\n+\t\treturn ULHR_BAD;\n+\n+\t/*\n+\t * Check if entire header is unpacked in the first iteration.\n+\t */\n+\tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n+\t\treturn ULHR_OK;\n+\n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN.\n+\t */\n+\treturn ULHR_TOO_LONG;\n+}\n+\n+static void *unpack_loose_rest(git_zstream *stream,\n+\t\t\t       void *buffer, unsigned long size,\n+\t\t\t       const struct object_id *oid)\n+{\n+\tsize_t bytes = strlen(buffer) + 1, n;\n+\tunsigned char *buf = xmallocz(size);\n+\tint status = Z_OK;\n+\n+\tn = stream->total_out - bytes;\n+\tif (n > size)\n+\t\tn = size;\n+\tmemcpy(buf, (char *) buffer + bytes, n);\n+\tbytes = n;\n+\tif (bytes <= size) {\n+\t\t/*\n+\t\t * The above condition must be (bytes <= size), not\n+\t\t * (bytes < size).  In other words, even though we\n+\t\t * expect no more output and set avail_out to zero,\n+\t\t * the input zlib stream may have bytes that express\n+\t\t * \"this concludes the stream\", and we *do* want to\n+\t\t * eat that input.\n+\t\t *\n+\t\t * Otherwise we would not be able to test that we\n+\t\t * consumed all the input to reach the expected size;\n+\t\t * we also want to check that zlib tells us that all\n+\t\t * went well with status == Z_STREAM_END at the end.\n+\t\t */\n+\t\tstream->next_out = buf + bytes;\n+\t\tstream->avail_out = size - bytes;\n+\t\twhile (status == Z_OK) {\n+\t\t\tobj_read_unlock();\n+\t\t\tstatus = git_inflate(stream, Z_FINISH);\n+\t\t\tobj_read_lock();\n+\t\t}\n+\t}\n+\n+\tif (status != Z_STREAM_END) {\n+\t\terror(_(\"corrupt loose object '%s'\"), oid_to_hex(oid));\n+\t\tFREE_AND_NULL(buf);\n+\t} else if (stream->avail_in) {\n+\t\terror(_(\"garbage at end of loose object '%s'\"),\n+\t\t      oid_to_hex(oid));\n+\t\tFREE_AND_NULL(buf);\n+\t}\n+\n+\treturn buf;\n+}\n+\n+/*\n+ * parse_loose_header() parses the starting \"<type> <len>\\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the <type> populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed <len> can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ *\n+ * We used to just use \"sscanf()\", but that's actually way\n+ * too permissive for what we want to check. So do an anal\n+ * object header parse by hand.\n+ */\n+static int parse_loose_header(const char *hdr, struct object_info *oi)\n+{\n+\tconst char *type_buf = hdr;\n+\tsize_t size;\n+\tint type, type_len = 0;\n+\n+\t/*\n+\t * The type can be of any size but is followed by\n+\t * a space.\n+\t */\n+\tfor (;;) {\n+\t\tchar c = *hdr++;\n+\t\tif (!c)\n+\t\t\treturn -1;\n+\t\tif (c == ' ')\n+\t\t\tbreak;\n+\t\ttype_len++;\n+\t}\n+\n+\ttype = type_from_string_gently(type_buf, type_len, 1);\n+\tif (oi->typep)\n+\t\t*oi->typep = type;\n+\n+\t/*\n+\t * The length must follow immediately, and be in canonical\n+\t * decimal format (ie \"010\" is not valid).\n+\t */\n+\tsize = *hdr++ - '0';\n+\tif (size > 9)\n+\t\treturn -1;\n+\tif (size) {\n+\t\tfor (;;) {\n+\t\t\tunsigned long c = *hdr - '0';\n+\t\t\tif (c > 9)\n+\t\t\t\tbreak;\n+\t\t\thdr++;\n+\t\t\tsize = st_add(st_mult(size, 10), c);\n+\t\t}\n+\t}\n+\n+\tif (oi->sizep)\n+\t\t*oi->sizep = size;\n+\n+\t/*\n+\t * The length must be followed by a zero byte\n+\t */\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n+}\n+\n static int read_object_info_from_path(struct odb_source_loose *loose,\n \t\t\t\t      const char *path,\n \t\t\t\t      const struct object_id *oid,\n@@ -248,6 +404,29 @@ static int open_loose_object(struct odb_source_loose *loose,\n \treturn -1;\n }\n \n+/*\n+ * Map and close the given loose object fd. The path argument is used for\n+ * error reporting.\n+ */\n+static void *map_fd(int fd, const char *path, unsigned long *size)\n+{\n+\tvoid *map = NULL;\n+\tstruct stat st;\n+\n+\tif (!fstat(fd, &st)) {\n+\t\t*size = xsize_t(st.st_size);\n+\t\tif (!*size) {\n+\t\t\t/* mmap() is forbidden on empty files */\n+\t\t\terror(_(\"object file %s is empty\"), path);\n+\t\t\tclose(fd);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tmap = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\t}\n+\tclose(fd);\n+\treturn map;\n+}\n+\n static void *odb_source_loose_map_object(struct odb_source_loose *loose,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t unsigned long *size)\n@@ -1015,6 +1194,133 @@ static void odb_source_loose_free(struct odb_source *source)\n \tfree(loose);\n }\n \n+static int check_stream_oid(git_zstream *stream,\n+\t\t\t    const char *hdr,\n+\t\t\t    unsigned long size,\n+\t\t\t    const char *path,\n+\t\t\t    const struct object_id *expected_oid,\n+\t\t\t    const struct git_hash_algo *algop)\n+{\n+\tstruct git_hash_ctx c;\n+\tstruct object_id real_oid;\n+\tunsigned char buf[4096];\n+\tunsigned long total_read;\n+\tint status = Z_OK;\n+\n+\tgit_hash_init(&c, algop);\n+\tgit_hash_update(&c, hdr, stream->total_out);\n+\n+\t/*\n+\t * We already read some bytes into hdr, but the ones up to the NUL\n+\t * do not count against the object's content size.\n+\t */\n+\ttotal_read = stream->total_out - strlen(hdr) - 1;\n+\n+\t/*\n+\t * This size comparison must be \"<=\" to read the final zlib packets;\n+\t * see the comment in unpack_loose_rest for details.\n+\t */\n+\twhile (total_read <= size &&\n+\t       (status == Z_OK ||\n+\t\t(status == Z_BUF_ERROR && !stream->avail_out))) {\n+\t\tstream->next_out = buf;\n+\t\tstream->avail_out = sizeof(buf);\n+\t\tif (size - total_read < stream->avail_out)\n+\t\t\tstream->avail_out = size - total_read;\n+\t\tstatus = git_inflate(stream, Z_FINISH);\n+\t\tgit_hash_update(&c, buf, stream->next_out - buf);\n+\t\ttotal_read += stream->next_out - buf;\n+\t}\n+\n+\tif (status != Z_STREAM_END) {\n+\t\terror(_(\"corrupt loose object '%s'\"), oid_to_hex(expected_oid));\n+\t\tgit_hash_discard(&c);\n+\t\treturn -1;\n+\t}\n+\tif (stream->avail_in) {\n+\t\terror(_(\"garbage at end of loose object '%s'\"),\n+\t\t      oid_to_hex(expected_oid));\n+\t\tgit_hash_discard(&c);\n+\t\treturn -1;\n+\t}\n+\n+\tgit_hash_final_oid(&real_oid, &c);\n+\tif (!oideq(expected_oid, &real_oid)) {\n+\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n+\t\t      oid_to_hex(expected_oid));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int read_loose_object(struct repository *repo,\n+\t\t\t     const char *path,\n+\t\t\t     const struct object_id *expected_oid,\n+\t\t\t     struct object_id *real_oid,\n+\t\t\t     void **contents,\n+\t\t\t     struct object_info *oi)\n+{\n+\tint ret = -1;\n+\tint fd;\n+\tvoid *map = NULL;\n+\tunsigned long mapsize;\n+\tgit_zstream stream;\n+\tchar hdr[MAX_HEADER_LEN];\n+\tsize_t *size = oi->sizep;\n+\n+\tfd = git_open(path);\n+\tif (fd >= 0)\n+\t\tmap = map_fd(fd, path, &mapsize);\n+\tif (!map) {\n+\t\terror_errno(_(\"unable to mmap %s\"), path);\n+\t\tgoto out;\n+\t}\n+\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) != ULHR_OK) {\n+\t\terror(_(\"unable to unpack header of %s\"), path);\n+\t\tgoto out_inflate;\n+\t}\n+\n+\tif (parse_loose_header(hdr, oi) < 0) {\n+\t\terror(_(\"unable to parse header of %s\"), path);\n+\t\tgoto out_inflate;\n+\t}\n+\n+\tif (*oi->typep < 0) {\n+\t\terror(_(\"unable to parse type from header '%s' of %s\"),\n+\t\t      hdr, path);\n+\t\tgoto out_inflate;\n+\t}\n+\n+\tif (*oi->typep == OBJ_BLOB &&\n+\t    *size > repo_settings_get_big_file_threshold(repo)) {\n+\t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid,\n+\t\t\t\t     repo->hash_algo) < 0)\n+\t\t\tgoto out_inflate;\n+\t} else {\n+\t\t*contents = unpack_loose_rest(&stream, hdr, *size, expected_oid);\n+\t\tif (!*contents) {\n+\t\t\terror(_(\"unable to unpack contents of %s\"), path);\n+\t\t\tgoto out_inflate;\n+\t\t}\n+\t\thash_object_file(repo->hash_algo,\n+\t\t\t\t *contents, *size,\n+\t\t\t\t *oi->typep, real_oid);\n+\t\tif (!oideq(expected_oid, real_oid))\n+\t\t\tgoto out_inflate;\n+\t}\n+\n+\tret = 0; /* everything checks out */\n+\n+out_inflate:\n+\tgit_inflate_end(&stream);\n+out:\n+\tif (map)\n+\t\tmunmap(map, mapsize);\n+\treturn ret;\n+}\n+\n struct fsck_loose_data {\n \tstruct odb_source_loose *source;\n \tstruct odb_fsck_options *opts;\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557160","messageId":"20261102-pks-odb-loose-cleanups-v1-2-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","subject":"[PATCH 2/6] object-file: move logic to derive loose object paths into backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:18Z","receivedAt":"2026-11-02T11:20:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The logic to derive the filesystem path of a loose object is contained\nin \"object-file.c\" even though it's an implementation detail of the\n\"loose\" backend. Move the logic into the \"loose\" backend accordingly.\nWhile at it, inline the call to `fill_loose_path()`.\n\nIdeally, we'd be able to remove the externally-visible declaration. But\nunfortunately, we still require access to that function for use in the\ndumb transport so that it can derive the target path for downloaded\nobjects. This will need to get abstracted eventually, but for now we\nsimply continue to expose the function.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n object-file.c      | 25 -------------------------\n object-file.h      |  8 --------\n odb/source-loose.c | 21 +++++++++++++++++++++\n odb/source-loose.h |  8 ++++++++\n 4 files changed, 29 insertions(+), 33 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex c1b78a25c9f..ab919f03fef 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -43,31 +43,6 @@ static int get_conv_flags(unsigned flags)\n \t\treturn 0;\n }\n \n-static void fill_loose_path(struct strbuf *buf,\n-\t\t\t    const struct object_id *oid,\n-\t\t\t    const struct git_hash_algo *algop)\n-{\n-\tfor (size_t i = 0; i < algop->rawsz; i++) {\n-\t\tstatic char hex[] = \"0123456789abcdef\";\n-\t\tunsigned int val = oid->hash[i];\n-\t\tstrbuf_addch(buf, hex[val >> 4]);\n-\t\tstrbuf_addch(buf, hex[val & 0xf]);\n-\t\tif (!i)\n-\t\t\tstrbuf_addch(buf, '/');\n-\t}\n-}\n-\n-const char *odb_loose_path(struct odb_source_loose *loose,\n-\t\t\t   struct strbuf *buf,\n-\t\t\t   const struct object_id *oid)\n-{\n-\tstrbuf_reset(buf);\n-\tstrbuf_addstr(buf, loose->base.path);\n-\tstrbuf_addch(buf, '/');\n-\tfill_loose_path(buf, oid, loose->base.odb->repo->hash_algo);\n-\treturn buf->buf;\n-}\n-\n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n static int freshen_file(const char *fn, const time_t *mtime)\n {\ndiff --git a/object-file.h b/object-file.h\nindex 65ab44542b0..3cfcaefa420 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -23,14 +23,6 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa\n \n struct odb_source;\n \n-/*\n- * Put in `buf` the name of the file in the local object database that\n- * would be used to store a loose object with the specified oid.\n- */\n-const char *odb_loose_path(struct odb_source_loose *source,\n-\t\t\t   struct strbuf *buf,\n-\t\t\t   const struct object_id *oid);\n-\n /*\n  * Iterate over the files in the loose-object parts of the object\n  * directory \"path\", triggering the following callbacks:\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 96fc2ed213f..0f29162d752 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -18,6 +18,27 @@\n #include \"tempfile.h\"\n #include \"write-or-die.h\"\n \n+const char *odb_loose_path(struct odb_source_loose *loose,\n+\t\t\t   struct strbuf *buf,\n+\t\t\t   const struct object_id *oid)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\n+\tstrbuf_reset(buf);\n+\tstrbuf_addstr(buf, loose->base.path);\n+\tstrbuf_addch(buf, '/');\n+\n+\tfor (size_t i = 0; i < loose->base.odb->repo->hash_algo->rawsz; i++) {\n+\t\tunsigned int val = oid->hash[i];\n+\t\tstrbuf_addch(buf, hex[val >> 4]);\n+\t\tstrbuf_addch(buf, hex[val & 0xf]);\n+\t\tif (!i)\n+\t\t\tstrbuf_addch(buf, '/');\n+\t}\n+\n+\treturn buf->buf;\n+}\n+\n static int append_loose_object(const struct object_id *oid,\n \t\t\t       const char *path UNUSED,\n \t\t\t       void *data)\ndiff --git a/odb/source-loose.h b/odb/source-loose.h\nindex a6ae808662b..8d4cdac0d50 100644\n--- a/odb/source-loose.h\n+++ b/odb/source-loose.h\n@@ -49,4 +49,12 @@ static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_sour\n \treturn container_of(source, struct odb_source_loose, base);\n }\n \n+/*\n+ * Put in `buf` the name of the file in the local object database that\n+ * would be used to store a loose object with the specified oid.\n+ */\n+const char *odb_loose_path(struct odb_source_loose *source,\n+\t\t\t   struct strbuf *buf,\n+\t\t\t   const struct object_id *oid);\n+\n #endif\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557161","messageId":"20261102-pks-odb-loose-cleanups-v1-3-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","subject":"[PATCH 3/6] object-file: move iteration over loose files into backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:19Z","receivedAt":"2026-11-02T11:20:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The logic to iterate over files part of the \"loose\" backend is contained\nin \"object-file.c\" even though it's an implementation detail of the\nbackend itself. Move the logic into the backend accordingly.\n\nNote that the function is a bit on the weird side: it accepts a generic\n`struct odb_source` as input and doesn't cast it down to its backend at\nall. This is because all it requires access to is the `path` member of\nthe source, and from thereon it knows how to iterate the subdirectories\nof the \"loose\" backend.\n\nWe should adapt this function to accept `struct odb_source_loose` to\nproperly reflect that it expects a path to a loose object database. But\nthat would require changes to a bunch of callers without much benefit.\nSo for now, the function signature is left as-is. But eventually we may\nwant to change it once we have reworked a couple of its callers.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/count-objects.c |  1 +\n builtin/gc.c            |  1 +\n builtin/prune.c         |  1 +\n object-file.c           | 92 -------------------------------------------------\n object-file.h           | 44 -----------------------\n odb/source-loose.c      | 92 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-loose.h      | 33 ++++++++++++++++++\n prune-packed.c          |  1 +\n 8 files changed, 129 insertions(+), 136 deletions(-)\n\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex 3f7b20fde53..aae040567e3 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -14,6 +14,7 @@\n #include \"quote.h\"\n #include \"packfile.h\"\n #include \"object-file.h\"\n+#include \"odb/source-loose.h\"\n \n static unsigned long garbage;\n static off_t size_garbage;\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 81cecf6b6c3..d3ba6f0c9e9 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -31,6 +31,7 @@\n #include \"packfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n+#include \"odb/source-loose.h\"\n #include \"path.h\"\n #include \"reflog.h\"\n #include \"rerere.h\"\ndiff --git a/builtin/prune.c b/builtin/prune.c\nindex 97f4c1418bc..af2a547d931 100644\n--- a/builtin/prune.c\n+++ b/builtin/prune.c\n@@ -17,6 +17,7 @@\n #include \"object-file.h\"\n #include \"object-name.h\"\n #include \"odb.h\"\n+#include \"odb/source-loose.h\"\n #include \"shallow.h\"\n \n static const char * const prune_usage[] = {\ndiff --git a/object-file.c b/object-file.c\nindex ab919f03fef..d3ddcbf309c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -12,7 +12,6 @@\n #include \"git-compat-util.h\"\n #include \"config.h\"\n #include \"convert.h\"\n-#include \"dir.h\"\n #include \"environment.h\"\n #include \"fsck.h\"\n #include \"gettext.h\"\n@@ -838,97 +837,6 @@ int read_pack_header(int fd, struct pack_header *header)\n \treturn 0;\n }\n \n-int for_each_file_in_obj_subdir(unsigned int subdir_nr,\n-\t\t\t\tstruct strbuf *path,\n-\t\t\t\tconst struct git_hash_algo *algop,\n-\t\t\t\teach_loose_object_fn obj_cb,\n-\t\t\t\teach_loose_cruft_fn cruft_cb,\n-\t\t\t\teach_loose_subdir_fn subdir_cb,\n-\t\t\t\tvoid *data)\n-{\n-\tsize_t origlen, baselen;\n-\tDIR *dir;\n-\tstruct dirent *de;\n-\tint r = 0;\n-\tstruct object_id oid;\n-\n-\tif (subdir_nr > 0xff)\n-\t\tBUG(\"invalid loose object subdirectory: %x\", subdir_nr);\n-\n-\toriglen = path->len;\n-\tstrbuf_complete(path, '/');\n-\tstrbuf_addf(path, \"%02x\", subdir_nr);\n-\n-\tdir = opendir(path->buf);\n-\tif (!dir) {\n-\t\tif (errno != ENOENT)\n-\t\t\tr = error_errno(_(\"unable to open %s\"), path->buf);\n-\t\tstrbuf_setlen(path, origlen);\n-\t\treturn r;\n-\t}\n-\n-\toid.hash[0] = subdir_nr;\n-\tstrbuf_addch(path, '/');\n-\tbaselen = path->len;\n-\n-\twhile ((de = readdir_skip_dot_and_dotdot(dir))) {\n-\t\tsize_t namelen;\n-\n-\t\tnamelen = strlen(de->d_name);\n-\t\tstrbuf_setlen(path, baselen);\n-\t\tstrbuf_add(path, de->d_name, namelen);\n-\t\tif (namelen == algop->hexsz - 2 &&\n-\t\t    !hex_to_bytes(oid.hash + 1, de->d_name,\n-\t\t\t\t  algop->rawsz - 1)) {\n-\t\t\toid_set_algo(&oid, algop);\n-\t\t\tmemset(oid.hash + algop->rawsz, 0,\n-\t\t\t       GIT_MAX_RAWSZ - algop->rawsz);\n-\t\t\tif (obj_cb) {\n-\t\t\t\tr = obj_cb(&oid, path->buf, data);\n-\t\t\t\tif (r)\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (cruft_cb) {\n-\t\t\tr = cruft_cb(de->d_name, path->buf, data);\n-\t\t\tif (r)\n-\t\t\t\tbreak;\n-\t\t}\n-\t}\n-\tclosedir(dir);\n-\n-\tstrbuf_setlen(path, baselen - 1);\n-\tif (!r && subdir_cb)\n-\t\tr = subdir_cb(subdir_nr, path->buf, data);\n-\n-\tstrbuf_setlen(path, origlen);\n-\n-\treturn r;\n-}\n-\n-int for_each_loose_file_in_source(struct odb_source *source,\n-\t\t\t\t  each_loose_object_fn obj_cb,\n-\t\t\t\t  each_loose_cruft_fn cruft_cb,\n-\t\t\t\t  each_loose_subdir_fn subdir_cb,\n-\t\t\t\t  void *data)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tint r;\n-\n-\tstrbuf_addstr(&buf, source->path);\n-\tfor (int i = 0; i < 256; i++) {\n-\t\tr = for_each_file_in_obj_subdir(i, &buf, source->odb->repo->hash_algo,\n-\t\t\t\t\t\tobj_cb, cruft_cb, subdir_cb, data);\n-\t\tif (r)\n-\t\t\tbreak;\n-\t}\n-\n-\tstrbuf_release(&buf);\n-\treturn r;\n-}\n-\n static int write_pack_lockfile(struct odb_source *source, const char *pack_name,\n \t\t\t       const char *keep_msg, struct tempfile **out)\n {\ndiff --git a/object-file.h b/object-file.h\nindex 3cfcaefa420..83d3f38a540 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -23,50 +23,6 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa\n \n struct odb_source;\n \n-/*\n- * Iterate over the files in the loose-object parts of the object\n- * directory \"path\", triggering the following callbacks:\n- *\n- *  - loose_object is called for each loose object we find.\n- *\n- *  - loose_cruft is called for any files that do not appear to be\n- *    loose objects. Note that we only look in the loose object\n- *    directories \"objects/[0-9a-f]{2}/\", so we will not report\n- *    \"objects/foobar\" as cruft.\n- *\n- *  - loose_subdir is called for each top-level hashed subdirectory\n- *    of the object directory (e.g., \"$OBJDIR/f0\"). It is called\n- *    after the objects in the directory are processed.\n- *\n- * Any callback that is NULL will be ignored. Callbacks returning non-zero\n- * will end the iteration.\n- *\n- * In the \"buf\" variant, \"path\" is a strbuf which will also be used as a\n- * scratch buffer, but restored to its original contents before\n- * the function returns.\n- */\n-typedef int each_loose_object_fn(const struct object_id *oid,\n-\t\t\t\t const char *path,\n-\t\t\t\t void *data);\n-typedef int each_loose_cruft_fn(const char *basename,\n-\t\t\t\tconst char *path,\n-\t\t\t\tvoid *data);\n-typedef int each_loose_subdir_fn(unsigned int nr,\n-\t\t\t\t const char *path,\n-\t\t\t\t void *data);\n-int for_each_loose_file_in_source(struct odb_source *source,\n-\t\t\t\t  each_loose_object_fn obj_cb,\n-\t\t\t\t  each_loose_cruft_fn cruft_cb,\n-\t\t\t\t  each_loose_subdir_fn subdir_cb,\n-\t\t\t\t  void *data);\n-int for_each_file_in_obj_subdir(unsigned int subdir_nr,\n-\t\t\t\tstruct strbuf *path,\n-\t\t\t\tconst struct git_hash_algo *algop,\n-\t\t\t\teach_loose_object_fn obj_cb,\n-\t\t\t\teach_loose_cruft_fn cruft_cb,\n-\t\t\t\teach_loose_subdir_fn subdir_cb,\n-\t\t\t\tvoid *data);\n-\n /**\n  * format_object_header() is a thin wrapper around s xsnprintf() that\n  * writes the initial \"<type> <obj-len>\" part of the loose object\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 0f29162d752..a90268e200a 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -1,6 +1,7 @@\n #include \"git-compat-util.h\"\n #include \"abspath.h\"\n #include \"chdir-notify.h\"\n+#include \"dir.h\"\n #include \"gettext.h\"\n #include \"hex.h\"\n #include \"loose.h\"\n@@ -39,6 +40,97 @@ const char *odb_loose_path(struct odb_source_loose *loose,\n \treturn buf->buf;\n }\n \n+static int for_each_file_in_obj_subdir(unsigned int subdir_nr,\n+\t\t\t\t       struct strbuf *path,\n+\t\t\t\t       const struct git_hash_algo *algop,\n+\t\t\t\t       each_loose_object_fn obj_cb,\n+\t\t\t\t       each_loose_cruft_fn cruft_cb,\n+\t\t\t\t       each_loose_subdir_fn subdir_cb,\n+\t\t\t\t       void *data)\n+{\n+\tsize_t origlen, baselen;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tint r = 0;\n+\tstruct object_id oid;\n+\n+\tif (subdir_nr > 0xff)\n+\t\tBUG(\"invalid loose object subdirectory: %x\", subdir_nr);\n+\n+\toriglen = path->len;\n+\tstrbuf_complete(path, '/');\n+\tstrbuf_addf(path, \"%02x\", subdir_nr);\n+\n+\tdir = opendir(path->buf);\n+\tif (!dir) {\n+\t\tif (errno != ENOENT)\n+\t\t\tr = error_errno(_(\"unable to open %s\"), path->buf);\n+\t\tstrbuf_setlen(path, origlen);\n+\t\treturn r;\n+\t}\n+\n+\toid.hash[0] = subdir_nr;\n+\tstrbuf_addch(path, '/');\n+\tbaselen = path->len;\n+\n+\twhile ((de = readdir_skip_dot_and_dotdot(dir))) {\n+\t\tsize_t namelen;\n+\n+\t\tnamelen = strlen(de->d_name);\n+\t\tstrbuf_setlen(path, baselen);\n+\t\tstrbuf_add(path, de->d_name, namelen);\n+\t\tif (namelen == algop->hexsz - 2 &&\n+\t\t    !hex_to_bytes(oid.hash + 1, de->d_name,\n+\t\t\t\t  algop->rawsz - 1)) {\n+\t\t\toid_set_algo(&oid, algop);\n+\t\t\tmemset(oid.hash + algop->rawsz, 0,\n+\t\t\t       GIT_MAX_RAWSZ - algop->rawsz);\n+\t\t\tif (obj_cb) {\n+\t\t\t\tr = obj_cb(&oid, path->buf, data);\n+\t\t\t\tif (r)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (cruft_cb) {\n+\t\t\tr = cruft_cb(de->d_name, path->buf, data);\n+\t\t\tif (r)\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\tclosedir(dir);\n+\n+\tstrbuf_setlen(path, baselen - 1);\n+\tif (!r && subdir_cb)\n+\t\tr = subdir_cb(subdir_nr, path->buf, data);\n+\n+\tstrbuf_setlen(path, origlen);\n+\n+\treturn r;\n+}\n+\n+int for_each_loose_file_in_source(struct odb_source *source,\n+\t\t\t\t  each_loose_object_fn obj_cb,\n+\t\t\t\t  each_loose_cruft_fn cruft_cb,\n+\t\t\t\t  each_loose_subdir_fn subdir_cb,\n+\t\t\t\t  void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint r;\n+\n+\tstrbuf_addstr(&buf, source->path);\n+\tfor (int i = 0; i < 256; i++) {\n+\t\tr = for_each_file_in_obj_subdir(i, &buf, source->odb->repo->hash_algo,\n+\t\t\t\t\t\tobj_cb, cruft_cb, subdir_cb, data);\n+\t\tif (r)\n+\t\t\tbreak;\n+\t}\n+\n+\tstrbuf_release(&buf);\n+\treturn r;\n+}\n+\n static int append_loose_object(const struct object_id *oid,\n \t\t\t       const char *path UNUSED,\n \t\t\t       void *data)\ndiff --git a/odb/source-loose.h b/odb/source-loose.h\nindex 8d4cdac0d50..20f6ec7fa08 100644\n--- a/odb/source-loose.h\n+++ b/odb/source-loose.h\n@@ -57,4 +57,37 @@ const char *odb_loose_path(struct odb_source_loose *source,\n \t\t\t   struct strbuf *buf,\n \t\t\t   const struct object_id *oid);\n \n+/*\n+ * Iterate over the files in the loose-object parts of the object\n+ * directory \"path\", triggering the following callbacks:\n+ *\n+ *  - loose_object is called for each loose object we find.\n+ *\n+ *  - loose_cruft is called for any files that do not appear to be\n+ *    loose objects. Note that we only look in the loose object\n+ *    directories \"objects/[0-9a-f]{2}/\", so we will not report\n+ *    \"objects/foobar\" as cruft.\n+ *\n+ *  - loose_subdir is called for each top-level hashed subdirectory\n+ *    of the object directory (e.g., \"$OBJDIR/f0\"). It is called\n+ *    after the objects in the directory are processed.\n+ *\n+ * Any callback that is NULL will be ignored. Callbacks returning non-zero\n+ * will end the iteration.\n+ */\n+typedef int each_loose_object_fn(const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t void *data);\n+typedef int each_loose_cruft_fn(const char *basename,\n+\t\t\t\tconst char *path,\n+\t\t\t\tvoid *data);\n+typedef int each_loose_subdir_fn(unsigned int nr,\n+\t\t\t\t const char *path,\n+\t\t\t\t void *data);\n+int for_each_loose_file_in_source(struct odb_source *source,\n+\t\t\t\t  each_loose_object_fn obj_cb,\n+\t\t\t\t  each_loose_cruft_fn cruft_cb,\n+\t\t\t\t  each_loose_subdir_fn subdir_cb,\n+\t\t\t\t  void *data);\n+\n #endif\ndiff --git a/prune-packed.c b/prune-packed.c\nindex b6749b8d132..d0a8a062b97 100644\n--- a/prune-packed.c\n+++ b/prune-packed.c\n@@ -3,6 +3,7 @@\n #include \"git-compat-util.h\"\n #include \"gettext.h\"\n #include \"object-file.h\"\n+#include \"odb/source-loose.h\"\n #include \"packfile.h\"\n #include \"progress.h\"\n #include \"prune-packed.h\"\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557162","messageId":"20261102-pks-odb-loose-cleanups-v1-4-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","subject":"[PATCH 4/6] odb/source-loose: deduplicate logic to mmap objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:20Z","receivedAt":"2026-11-02T11:20:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The logic to mmap objects is duplicated across `map_fd()` and\n`odb_source_loose_map_object()`. The only difference is that the former\ntakes an arbitrary path, whereas the latter derives the path from an\nobject ID.\n\nDeduplicate the logic.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-loose.c | 23 +++--------------------\n 1 file changed, 3 insertions(+), 20 deletions(-)\n\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex a90268e200a..5cd679e802c 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -544,28 +544,11 @@ static void *odb_source_loose_map_object(struct odb_source_loose *loose,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t unsigned long *size)\n {\n-\tconst char *p;\n-\tint fd = open_loose_object(loose, oid, &p);\n-\tvoid *map = NULL;\n-\tstruct stat st;\n-\n+\tconst char *path;\n+\tint fd = open_loose_object(loose, oid, &path);\n \tif (fd < 0)\n \t\treturn NULL;\n-\n-\tif (!fstat(fd, &st)) {\n-\t\t*size = xsize_t(st.st_size);\n-\t\tif (!*size) {\n-\t\t\t/* mmap() is forbidden on empty files */\n-\t\t\terror(_(\"object file %s is empty\"), p);\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\tmap = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\t}\n-\n-out:\n-\tclose(fd);\n-\treturn map;\n+\treturn map_fd(fd, path, size);\n }\n \n struct odb_loose_read_stream {\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557163","messageId":"20261102-pks-odb-loose-cleanups-v1-5-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","subject":"[PATCH 5/6] object-file: stop including \"odb/source-loose.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:21Z","receivedAt":"2026-11-02T11:20:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have now moved all functionality that relate to the loose backend\nfrom \"object-file.c\" into the backend itself. Wrap up these refactorings\nby not including \"odb/source-loose.h\" anymore and instead include the\nfile explicitly in all compilation units that require access to it.\n\nLike this we can trivially identify exactly those files that still\ndepend on details of the loose backend itself and that thus require\nfurther work to become truly backend-agnostic.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c | 1 +\n http-walker.c          | 1 +\n http.c                 | 1 +\n loose.c                | 1 +\n object-file.c          | 1 +\n object-file.h          | 2 +-\n setup.c                | 1 +\n 7 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca3a891dfbc..e53c69af31f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -34,6 +34,7 @@\n #include \"object-file.h\"\n #include \"object-file-convert.h\"\n #include \"odb.h\"\n+#include \"odb/source-loose.h\"\n #include \"odb/streaming.h\"\n #include \"replace-object.h\"\n #include \"dir.h\"\ndiff --git a/http-walker.c b/http-walker.c\nindex 4f50dd9b8d0..13730a27847 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -11,6 +11,7 @@\n #include \"packfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n+#include \"odb/source-loose.h\"\n \n struct alt_base {\n \tchar *base;\ndiff --git a/http.c b/http.c\nindex d79678d01f0..25b5fc3b3b7 100644\n--- a/http.c\n+++ b/http.c\n@@ -21,6 +21,7 @@\n #include \"string-list.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n+#include \"odb/source-loose.h\"\n #include \"tempfile.h\"\n #include \"date.h\"\n #include \"trace2.h\"\ndiff --git a/loose.c b/loose.c\nindex d6a2bb5ba3c..64cd6f1c3c1 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -4,6 +4,7 @@\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/source-files.h\"\n+#include \"odb/source-loose.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n #include \"wrapper.h\"\ndiff --git a/object-file.c b/object-file.c\nindex d3ddcbf309c..917a65d5b33 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -20,6 +20,7 @@\n #include \"object-file-convert.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n+#include \"odb/source.h\"\n #include \"odb/streaming.h\"\n #include \"odb/transaction.h\"\n #include \"pack.h\"\ndiff --git a/object-file.h b/object-file.h\nindex 83d3f38a540..746356ceb51 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -4,7 +4,7 @@\n #include \"git-zlib.h\"\n #include \"object.h\"\n #include \"odb.h\"\n-#include \"odb/source-loose.h\"\n+#include \"odb/streaming.h\"\n #include \"odb/transaction.h\"\n \n /* The maximum size for an object header. */\ndiff --git a/setup.c b/setup.c\nindex 61bcdafe82a..bc94463a239 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -9,6 +9,7 @@\n #include \"hex.h\"\n #include \"object-file.h\"\n #include \"object-name.h\"\n+#include \"odb/source.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n #include \"config.h\"\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557164","messageId":"20261102-pks-odb-loose-cleanups-v1-6-fbbc86c29693@pks.im","threadId":"66744","inReplyTo":"20261102-pks-odb-loose-cleanups-v1-0-fbbc86c29693@pks.im","subject":"[PATCH 6/6] object-file: stop including \"git-zlib.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:20:22Z","receivedAt":"2026-11-02T11:20:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We no longer reference `git_zstream` in \"object-file.h\", but still\ninclude its header. Drop the header and add it to all the other users\nthat implicitly depend on it.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n apply.c                | 1 +\n builtin/fast-import.c  | 1 +\n builtin/index-pack.c   | 1 +\n builtin/pack-objects.c | 1 +\n diff.c                 | 1 +\n object-file.c          | 1 +\n object-file.h          | 1 -\n odb/source-loose.c     | 1 +\n pack-check.c           | 1 +\n packfile.c             | 1 +\n 10 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/apply.c b/apply.c\nindex a0b7119c639..c705ce06adb 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -21,6 +21,7 @@\n #include \"dir.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"xdiff-interface.h\"\n #include \"merge-ll.h\"\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 9db3f83eea8..c4d955b61f1 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -5,6 +5,7 @@\n #include \"abspath.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"config.h\"\n #include \"lockfile.h\"\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 0264c18c5f8..6d758a04419 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -6,6 +6,7 @@\n #include \"delta.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"pack.h\"\n #include \"csum-file.h\"\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e53c69af31f..f13c49090a1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4,6 +4,7 @@\n #include \"builtin.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"config.h\"\n #include \"attr.h\"\ndiff --git a/diff.c b/diff.c\nindex b4ac17f8efc..0309f91f5ab 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -18,6 +18,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"delta.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"xdiff-interface.h\"\n #include \"color.h\"\ndiff --git a/object-file.c b/object-file.c\nindex 917a65d5b33..cf859520949 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -15,6 +15,7 @@\n #include \"environment.h\"\n #include \"fsck.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"loose.h\"\n #include \"object-file-convert.h\"\ndiff --git a/object-file.h b/object-file.h\nindex 746356ceb51..70138a358de 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -1,7 +1,6 @@\n #ifndef OBJECT_FILE_H\n #define OBJECT_FILE_H\n \n-#include \"git-zlib.h\"\n #include \"object.h\"\n #include \"odb.h\"\n #include \"odb/streaming.h\"\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 5cd679e802c..737d583f033 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -3,6 +3,7 @@\n #include \"chdir-notify.h\"\n #include \"dir.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"loose.h\"\n #include \"object-file.h\"\ndiff --git a/pack-check.c b/pack-check.c\nindex 605811b7b09..c6c3d7c9c0c 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -2,6 +2,7 @@\n \n #include \"git-compat-util.h\"\n #include \"environment.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n #include \"pack.h\"\ndiff --git a/packfile.c b/packfile.c\nindex b32b4cd75da..59e0e0c1c91 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -3,6 +3,7 @@\n #include \"git-compat-util.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\n+#include \"git-zlib.h\"\n #include \"hex.h\"\n #include \"list.h\"\n #include \"pack.h\"\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"}]}