{"thread":{"id":"64600","subject":"[PATCH v3 0/2] improve --exclude-promisor-objects performance","startedAt":"2025-12-09T01:49:27Z","lastAt":"2025-12-09T01:49:33Z","messageCount":3,"participants":["Aaron Plattner"],"isPatch":true,"patchVersion":3,"patchTotal":2},"messages":[{"id":"531869","messageId":"20251209014900.402637-1-aplattner@nvidia.com","threadId":"64600","inReplyTo":null,"subject":"[PATCH v3 0/2] improve --exclude-promisor-objects performance","fromName":"Aaron Plattner","fromEmail":"aplattner@nvidia.com","sentAt":"2025-12-09T01:48:55Z","receivedAt":"2025-12-09T01:49:27Z","isPatch":true,"sender":{"key":"aplattner@nvidia.com","avatar":"https://avatars.githubusercontent.com/u/343551?v=4"},"body":"This series fixes the PARSE_OBJECT_SKIP_HASH_CHECK optimization in\nparse_object_with_flags() so that it applies to objects with their type set to\nOBJ_NONE too, and then uses that behavior significantly improve the performance\nof add_promisor_object().\n\nAaron Plattner (2):\n  object: apply skip_hash and discard_tree optimizations to unknown\n    blobs too\n  packfile: skip hash checks in add_promisor_object()\n\n object.c   | 4 ++--\n packfile.c | 3 ++-\n 2 files changed, 4 insertions(+), 3 deletions(-)\n\n-- \n2.52.0\n\n"},{"id":"531870","messageId":"20251209014900.402637-2-aplattner@nvidia.com","threadId":"64600","inReplyTo":"20251209014900.402637-1-aplattner@nvidia.com","subject":"[PATCH v3 1/2] object: apply skip_hash and discard_tree optimizations to unknown blobs too","fromName":"Aaron Plattner","fromEmail":"aplattner@nvidia.com","sentAt":"2025-12-09T01:48:56Z","receivedAt":"2025-12-09T01:49:29Z","isPatch":true,"sender":{"key":"aplattner@nvidia.com","avatar":"https://avatars.githubusercontent.com/u/343551?v=4"},"body":"parse_object_with_flags() has an optimization to skip parsing blobs if\nPARSE_OBJECT_SKIP_HASH_CHECK is set and the object hasn't been seen\nbefore or might be a blob but hasn't been parsed yet. The latter can\nhappen, for example, if add_tree_entries() walks a path that references\na blob object that hasn't been seen before: lookup_blob() marks the\nreferenced oid as being a blob, but does not provide any additional\ninformation about it until it is parsed.\n\nIt's possible for an object to be created without even a type, such as\nwhen prepare_revision_walk() uses mark_uninteresting() to mark all\npromisor objects as uninteresting. These objects have obj->parsed ==\nfalse and obj->type == OBJ_NONE.\n\nThe skip_hash optimization does not consider this kind of object, so\nparse_object_with_flags() proceeds to fully parse the object to\ndetermine its type.\n\nImprove the optimization by applying it to OBJ_NONE objects as well as\nOBJ_BLOB ones. Apply a similar fix for trees.\n\nFixes: 8db2dad7a045 (\"parse_object(): check on-disk type of suspected blob\")\nSigned-off-by: Aaron Plattner <aplattner@nvidia.com>\n---\n object.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex b08fc7a163..4669b8d65e 100644\n--- a/object.c\n+++ b/object.c\n@@ -328,7 +328,7 @@ struct object *parse_object_with_flags(struct repository *r,\n \t\t\treturn &commit->object;\n \t}\n \n-\tif ((!obj || obj->type == OBJ_BLOB) &&\n+\tif ((!obj || obj->type == OBJ_NONE || obj->type == OBJ_BLOB) &&\n \t    odb_read_object_info(r->objects, oid, NULL) == OBJ_BLOB) {\n \t\tif (!skip_hash && stream_object_signature(r, repl) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n@@ -344,7 +344,7 @@ struct object *parse_object_with_flags(struct repository *r,\n \t * have the on-disk object with the correct type.\n \t */\n \tif (skip_hash && discard_tree &&\n-\t    (!obj || obj->type == OBJ_TREE) &&\n+\t    (!obj || obj->type == OBJ_NONE || obj->type == OBJ_TREE) &&\n \t    odb_read_object_info(r->objects, oid, NULL) == OBJ_TREE) {\n \t\treturn &lookup_tree(r, oid)->object;\n \t}\n-- \n2.52.0\n\n"},{"id":"531871","messageId":"20251209014900.402637-3-aplattner@nvidia.com","threadId":"64600","inReplyTo":"20251209014900.402637-1-aplattner@nvidia.com","subject":"[PATCH v3 2/2] packfile: skip hash checks in add_promisor_object()","fromName":"Aaron Plattner","fromEmail":"aplattner@nvidia.com","sentAt":"2025-12-09T01:48:57Z","receivedAt":"2025-12-09T01:49:33Z","isPatch":true,"sender":{"key":"aplattner@nvidia.com","avatar":"https://avatars.githubusercontent.com/u/343551?v=4"},"body":"When is_promisor_object() is called for the first time, it lazily\ninitializes a set of all promisor objects by iterating through all\nobjects in promisor packs. For each object, add_promisor_object() calls\nparse_object(), which decompresses and hashes the entire object.\n\nFor repositories with large pack files, this can take an extremely long\ntime. For example, on a production repository with a 176 GB promisor\npack:\n\n $ time ~/git/git/git-rev-list --objects --all --exclude-promisor-objects --quiet\n ________________________________________________________\n Executed in   76.10 mins    fish           external\n    usr time   72.10 mins    1.83 millis   72.10 mins\n    sys time    3.56 mins    0.17 millis    3.56 mins\n\nadd_promisor_object() just wants to construct the set of all promisor\nobjects, so it doesn't really need to verify the hash of every object.\nSet PARSE_OBJECT_SKIP_HASH_CHECK to skip the hash check. This has the\nside effect of skipping decompression of blob objects completely, saving\na significant amount of time:\n\n $ time ~/git/git/git-rev-list --objects --all --exclude-promisor-objects --quiet\n ________________________________________________________\n Executed in  124.70 secs    fish           external\n    usr time   46.94 secs    0.00 millis   46.94 secs\n    sys time   43.11 secs    1.03 millis   43.11 secs\n\nSigned-off-by: Aaron Plattner <aplattner@nvidia.com>\n---\n packfile.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 3d8b994a61..d3014b6746 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2295,7 +2295,8 @@ static int add_promisor_object(const struct object_id *oid,\n \t\twe_parsed_object = 0;\n \t} else {\n \t\twe_parsed_object = 1;\n-\t\tobj = parse_object(pack->repo, oid);\n+\t\tobj = parse_object_with_flags(pack->repo, oid,\n+\t\t\t\t\t      PARSE_OBJECT_SKIP_HASH_CHECK);\n \t}\n \n \tif (!obj)\n-- \n2.52.0\n\n"}]}