{"thread":{"id":"62429","subject":"[PATCH 0/6] PATH WALK I: The path-walk API","startedAt":"2024-10-31T06:27:07Z","lastAt":"2024-12-27T14:18:50Z","messageCount":67,"participants":["Derrick Stolee via GitGitGadget","Derrick Stolee","karthik nayak","Taylor Blau","Jonathan Tan","Jeff King","Junio C Hamano","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"506358","messageId":"pull.1818.git.1730356023.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":null,"subject":"[PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:26:57Z","receivedAt":"2024-10-31T06:27:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nIntroduction and relation to prior series\n=========================================\n\nThis is a new series that rerolls the initial \"path-walk API\" patches of my\nRFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\npath-walk.h) presents a new way to walk objects such that trees and blobs\nare walked in batches according to their path.\n\nThis also replaces the previous version of ds/path-walk that was being\nreviewed in [2]. The consensus was that the series was too long/dense and\ncould use some reduction in size. This series takes the first few patches,\nbut also makes some updates (which will be described later).\n\n[1]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n[RFC] Path-walk API and applications\n\n[2]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n[PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n\nThis series only introduces the path-walk API, but does so to the full\ncomplexity required to later add the integration with git pack-objects to\nimprove packing compression in both time and space for repositories with\nmany name hash collisions. The compression also at least improves for other\nrepositories, but may not always have an improvement in time.\n\nSome of the changes that are present in this series that differ from the\nprevious version are motivated directly by discoveries made by testing the\nfeature in Git for Windows and microsoft/git forks that shipped these\nfeatures for fast delivery of these improvements to users who needed them.\nThat testing across many environments informed some things that needed to be\nchanged, and in this series those changes are checked by tests in the\nt6601-path-walk.sh test script and the test-tool path-walk test helper.\nThus, the code being introduced in this series is covered by tests even\nthough it is not integrated into the git executable.\n\n\nDiscussion of follow-up applications\n====================================\n\nBy splitting this series out into its own, I was able to reorganize the\npatches such that each application can be build independently off of this\nseries. These are available as pending PRs in gitgitgadget/git:\n\n * Better delta compression with 'git pack-objects' [3]: This application\n allows an option in 'git pack-objects' to change how objects are walked\n in order to group objects with the same path for early delta compression\n before using the name hash sort to look for cross-path deltas. This helps\n significantly in repositories with many name-hash collisions. This\n reduces the size of 'git push' pacifies via a config option and reduces\n the total repo size in 'git repack'.\n\n * The 'git backfill' command [4]: This command downloads missing blobs in a\n bloodless partial clone. In order to save space and network bandwidth, it\n assumes that objects at a common path are likely to delta well with each\n other, so it downloads missing blobs in batches via the path-walk API.\n This presents a way to use blobless clones as a pseudo-resumable clone,\n since the initial clone of commits and trees is a smaller initial\n download and the batch size allows downloading blobs incrementally. When\n pairing this command with the sparse-checkout feature, the path-walk API\n is adjusted to focus on the paths within the sparse-checkout. This allows\n the user to only download the files they are likely to need when\n inspecting history within their scope without downloading the entire\n repository history.\n\n * The 'git survey' command [5]. This application begins the work to mimic\n the behavior of git-sizer, but to use internal data structures for better\n performance and careful understanding of how objects are stored. Using\n the path-walk API, paths with many versions can be considered in a batch\n and sorted into a list to report the paths that contribute most to the\n size of the repository. A version of this command was used to help\n confirm the issues with the name hash collisions. It was also used to\n diagnose why some repacks using the --path-walk option were taking more\n space than without for some repositories. (More on this later.)\n\nQuestion for reviewers: I am prepped to send these three applications to the\nmailing list, but I'll refrain for now to avoid causing too much noise for\nfolks. Would you like to see them on-list while this series is under review?\nOr would you prefer to explore the PRs ([3] [4] and [5])?\n\n[3] https://github.com/gitgitgadget/git/pull/1819\nPATH WALK II: Add --path-walk option to 'git pack-objects'\n\n[4] https://github.com/gitgitgadget/git/pull/1820\nPATH WALK III: Add 'git backfill' command\n\n[5] https://github.com/gitgitgadget/git/pull/1821\nPATH WALK IV: Add 'git survey' command\n\n\nStructure of the Patch Series\n=============================\n\nThis patch series attempts to create the simplest version of the API in\npatch 1, then build functionality incrementally. During the process, each\nchange will introduce an update to:\n\n * The path-walk API itself in path-walk.c and path-walk.h.\n * The documentation of the API in\n Documentation/technical/api-path-walk.txt.\n * The test script t/t6601-path-walk.sh.\n\nThe core of the API relies on using a 'struct rev_info' to define an initial\nset of objects and some form of a commit walk to define what range of\nobjects to visit. Initially, only a subset of 'struct rev_info' options work\nas expected. For example:\n\n * Patch 1 assumes that only commit objects are starting positions, but the\n focus is on exploring trees and blobs.\n * Patch 3 allows users to specify object types, which includes outputting\n the visited commits in a batch.\n * Annotated tags and indexed objects are considered in Patch 4. These are\n grouped because they both exist within the 'pending' object list.\n * UNINTERESTING objects are not considered until Patch 5.\n\n\nChanges since previous version\n==============================\n\nThere are a few hard-won learnings from previous versions of this series due\nto testing this in the wild with many different repositories.\n\n * Initially, the 'git pack-objects --path-walk' feature was not tested with\n the '--shallow' option because it was expected that this option was for\n servers creating a pack containing shallow commits. However, this option\n is also used when pushing from a shallow clone, and this was a critical\n feature that we needed to reduce the size of commits pushed from\n automated environments that were bootstrapped by shallow clones. The crux\n of the change is in Patch 5 and how UNINTERESTING objects are handled. We\n no longer need to push the UNINTERESTING flag around the objects\n ourselves and can use existing logic in list-objects.c to do so. This\n allows using the --objects-edge-aggressive option when necessary to\n reduce the object count when pushing from a shallow clone. (The\n pack-objects series expands on tests to cover this integration point.)\n\n * When looking into cases where 'git repack -adf' outperformed 'git repack\n -adf --path-walk', I discovered that the issue did not reproduce in a\n bare repository. This is due to 'git repack' iterating over all indexed\n objects before walking commits. I had inadvertently put all indexed\n objects in their own category, leading to no good deltas with previous\n versions of those files; I had also not used the 'path' option from the\n pending list, so these objects had invalid name hash values. You will see\n in patch 4 that the pending list is handled quite differently and the\n '--indexed-objects' option is tested directly within t6601.\n\n * I added a new 'test_cmp_sorted' helper because I wanted to simplify some\n repeated sections of t6601.\n\n * Patch 1 has significantly more context than it did before.\n\n * Annotated tags are given a name of \"/tags\" to differentiate them slightly\n from root trees and commits.\n\nI will include a full range diff relative to the previous versions of these\npatches in [2] in a reply to this cover letter.\n\nThanks, -Stolee\n\nDerrick Stolee (6):\n path-walk: introduce an object walk by path\n test-lib-functions: add test_cmp_sorted\n t6601: add helper for testing path-walk API\n path-walk: allow consumer to specify object types\n path-walk: visit tags and cached objects\n path-walk: mark trees and blobs as UNINTERESTING\n\n Documentation/technical/api-path-walk.txt | 63 +++\n Makefile | 2 +\n path-walk.c | 529 ++++++++++++++++++++++\n path-walk.h | 65 +++\n t/helper/test-path-walk.c | 114 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 366 +++++++++++++++\n t/test-lib-functions.sh | 10 +\n 9 files changed, 1151 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\n\nbase-commit: e9356ba3ea2a6754281ff7697b3e5a1697b21e24\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1818%2Fderrickstolee%2Fapi-upstream-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1818/derrickstolee/api-upstream-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1818\n-- \ngitgitgadget\n"},{"id":"506360","messageId":"c71f0a0e3613555de83668473ce97db86aad659f.1730356023.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH 1/6] path-walk: introduce an object walk by path","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:26:58Z","receivedAt":"2024-10-31T06:27:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a few planned applications, introduce the most basic form\nof a path-walk API. It currently assumes that there are no UNINTERESTING\nobjects, and does not include any complicated filters. It calls a function\npointer on groups of tree and blob objects as grouped by path. This only\nincludes objects the first time they are discovered, so an object that\nappears at multiple paths will not be included in two batches.\n\nThese batches are collected in 'struct type_and_oid_list' objects, which\nstore an object type and an oid_array of objects.\n\nThe data structures are documented in 'struct path_walk_context', but in\nsummary the most important are:\n\n * 'paths_to_lists' is a strmap that connects a path to a\n type_and_oid_list for that path. To avoid conflicts in path names,\n we make sure that tree paths end in \"/\" (except the root path with\n is an empty string) and blob paths do not end in \"/\".\n\n * 'path_stack' is a string list that is added to in an append-only\n way. This stores the stack of our depth-first search on the heap\n instead of using recursion.\n\n * 'path_stack_pushed' is a strmap that stores path names that were\n already added to 'path_stack', to avoid repeating paths in the\n stack. Mostly, this saves us from quadratic lookups from doing\n unsorted checks into the string_list.\n\nThe coupling of 'path_stack' and 'path_stack_pushed' is protected by the\npush_to_stack() method. Call this instead of inserting into these\nstructures directly.\n\nThe walk_objects_by_path() method initializes these structures and\nstarts walking commits from the given rev_info struct. The commits are\nused to find the list of root trees which populate the start of our\ndepth-first search.\n\nThe core of our depth-first search is in a while loop that continues\nwhile we have not indicated an early exit and our 'path_stack' still has\nentries in it. The loop body pops a path off of the stack and \"visits\"\nthe path via the walk_path() method.\n\nThe walk_path() method gets the list of OIDs from the 'path_to_lists'\nstrmap and executes the callback method on that list with the given path\nand type. If the OIDs correspond to tree objects, then iterate over all\ntrees in the list and run add_children() to add the child objects to\ntheir own lists, adding new entries to the stack if necessary.\n\nIn testing, this depth-first search approach was the one that used the\nleast memory while iterating over the object lists. There is still a\nchance that repositories with too-wide path patterns could cause memory\npressure issues. Limiting the stack size could be done in the future by\nlimiting how many objects are being considered in-progress, or by\nvisiting blob paths earlier than trees.\n\nThere are many future adaptations that could be made, but they are left for\nfuture updates when consumers are ready to take advantage of those features.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 45 ++++\n Makefile | 1 +\n path-walk.c | 260 ++++++++++++++++++++++\n path-walk.h | 43 ++++\n 4 files changed, 349 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nnew file mode 100644\nindex 00000000000..c550c77ca30\n--- /dev/null\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -0,0 +1,45 @@\n+Path-Walk API\n+=============\n+\n+The path-walk API is used to walk reachable objects, but to visit objects\n+in batches based on a common path they appear in, or by type.\n+\n+For example, all reachable commits are visited in a group. All tags are\n+visited in a group. Then, all root trees are visited. At some point, all\n+blobs reachable via a path `my/dir/to/A` are visited. When there are\n+multiple paths possible to reach the same object, then only one of those\n+paths is used to visit the object.\n+\n+Basics\n+------\n+\n+To use the path-walk API, include `path-walk.h` and call\n+`walk_objects_by_path()` with a customized `path_walk_info` struct. The\n+struct is used to set all of the options for how the walk should proceed.\n+Let's dig into the different options and their use.\n+\n+`path_fn` and `path_fn_data`::\n+\tThe most important option is the `path_fn` option, which is a\n+\tfunction pointer to the callback that can execute logic on the\n+\tobject IDs for objects grouped by type and path. This function\n+\talso receives a `data` value that corresponds to the\n+\t`path_fn_data` member, for providing custom data structures to\n+\tthis callback function.\n+\n+`revs`::\n+\tTo configure the exact details of the reachable set of objects,\n+\tuse the `revs` member and initialize it using the revision\n+\tmachinery in `revision.h`. Initialize `revs` using calls such as\n+\t`setup_revisions()` or `parse_revision_opt()`. Do not call\n+\t`prepare_revision_walk()`, as that will be called within\n+\t`walk_objects_by_path()`.\n++\n+It is also important that you do not specify the `--objects` flag for the\n+`revs` struct. The revision walk should only be used to walk commits, and\n+the objects will be walked in a separate way based on those starting\n+commits.\n+\n+Examples\n+--------\n+\n+See example usages in future changes.\ndiff --git a/Makefile b/Makefile\nindex 7344a7f7257..d0d8d6888e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\n+LIB_OBJS += path-walk.o\n LIB_OBJS += pathspec.o\n LIB_OBJS += pkt-line.o\n LIB_OBJS += preload-index.o\ndiff --git a/path-walk.c b/path-walk.c\nnew file mode 100644\nindex 00000000000..9dc56aff88c\n--- /dev/null\n+++ b/path-walk.c\n@@ -0,0 +1,260 @@\n+/*\n+ * path-walk.c: implementation for path-based walks of the object graph.\n+ */\n+#include \"git-compat-util.h\"\n+#include \"path-walk.h\"\n+#include \"blob.h\"\n+#include \"commit.h\"\n+#include \"dir.h\"\n+#include \"hashmap.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"oid-array.h\"\n+#include \"revision.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+#include \"trace2.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+\n+struct type_and_oid_list\n+{\n+\tenum object_type type;\n+\tstruct oid_array oids;\n+};\n+\n+#define TYPE_AND_OID_LIST_INIT { \\\n+\t.type = OBJ_NONE, \t \\\n+\t.oids = OID_ARRAY_INIT\t \\\n+}\n+\n+struct path_walk_context {\n+\t/**\n+\t * Repeats of data in 'struct path_walk_info' for\n+\t * access with fewer characters.\n+\t */\n+\tstruct repository *repo;\n+\tstruct rev_info *revs;\n+\tstruct path_walk_info *info;\n+\n+\t/**\n+\t * Map a path to a 'struct type_and_oid_list'\n+\t * containing the objects discovered at that\n+\t * path.\n+\t */\n+\tstruct strmap paths_to_lists;\n+\n+\t/**\n+\t * Store the current list of paths in a stack, to\n+\t * facilitate depth-first-search without recursion.\n+\t *\n+\t * Use path_stack_pushed to indicate whether a path\n+\t * was previously added to path_stack.\n+\t */\n+\tstruct string_list path_stack;\n+\tstruct strset path_stack_pushed;\n+};\n+\n+static void push_to_stack(struct path_walk_context *ctx,\n+\t\t\t const char *path)\n+{\n+\tif (strset_contains(&ctx->path_stack_pushed, path))\n+\t\treturn;\n+\n+\tstrset_add(&ctx->path_stack_pushed, path);\n+\tstring_list_append(&ctx->path_stack, path);\n+}\n+\n+static int add_children(struct path_walk_context *ctx,\n+\t\t\tconst char *base_path,\n+\t\t\tstruct object_id *oid)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t base_len;\n+\tstruct tree *tree = lookup_tree(ctx->repo, oid);\n+\n+\tif (!tree) {\n+\t\terror(_(\"failed to walk children of tree %s: not found\"),\n+\t\t oid_to_hex(oid));\n+\t\treturn -1;\n+\t} else if (parse_tree_gently(tree, 1)) {\n+\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n+\t}\n+\n+\tstrbuf_addstr(&path, base_path);\n+\tbase_len = path.len;\n+\n+\tparse_tree(tree);\n+\tinit_tree_desc(&desc, &tree->object.oid, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tstruct type_and_oid_list *list;\n+\t\tstruct object *o;\n+\t\t/* Not actually true, but we will ignore submodules later. */\n+\t\tenum object_type type = S_ISDIR(entry.mode) ? OBJ_TREE : OBJ_BLOB;\n+\n+\t\t/* Skip submodules. */\n+\t\tif (S_ISGITLINK(entry.mode))\n+\t\t\tcontinue;\n+\n+\t\tif (type == OBJ_TREE) {\n+\t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else if (type == OBJ_BLOB) {\n+\t\t\tstruct blob *child = lookup_blob(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else {\n+\t\t\t/* Wrong type? */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!o) /* report error?*/\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, base_len);\n+\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n+\n+\t\t/*\n+\t\t * Trees will end with \"/\" for concatenation and distinction\n+\t\t * from blobs at the same path.\n+\t\t */\n+\t\tif (type == OBJ_TREE)\n+\t\t\tstrbuf_addch(&path, '/');\n+\n+\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {\n+\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\tlist->type = type;\n+\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n+\t\t}\n+\t\tpush_to_stack(ctx, path.buf);\n+\n+\t\t/* Skip this object if already seen. */\n+\t\tif (o->flags & SEEN)\n+\t\t\tcontinue;\n+\t\to->flags |= SEEN;\n+\t\toid_array_append(&list->oids, &entry.oid);\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+\tstrbuf_release(&path);\n+\treturn 0;\n+}\n+\n+/*\n+ * For each path in paths_to_explore, walk the trees another level\n+ * and add any found blobs to the batch (but only if they exist and\n+ * haven't been added yet).\n+ */\n+static int walk_path(struct path_walk_context *ctx,\n+\t\t const char *path)\n+{\n+\tstruct type_and_oid_list *list;\n+\tint ret = 0;\n+\n+\tlist = strmap_get(&ctx->paths_to_lists, path);\n+\n+\t/* Evaluate function pointer on this data. */\n+\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t ctx->info->path_fn_data);\n+\n+\t/* Expand data for children. */\n+\tif (list->type == OBJ_TREE) {\n+\t\tfor (size_t i = 0; i < list->oids.nr; i++) {\n+\t\t\tret |= add_children(ctx,\n+\t\t\t\t\t path,\n+\t\t\t\t\t &list->oids.oid[i]);\n+\t\t}\n+\t}\n+\n+\toid_array_clear(&list->oids);\n+\tstrmap_remove(&ctx->paths_to_lists, path, 1);\n+\treturn ret;\n+}\n+\n+static void clear_strmap(struct strmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\n+\thashmap_for_each_entry(&map->map, &iter, e, ent) {\n+\t\tstruct type_and_oid_list *list = e->value;\n+\t\toid_array_clear(&list->oids);\n+\t}\n+\tstrmap_clear(map, 1);\n+\tstrmap_init(map);\n+}\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info)\n+{\n+\tconst char *root_path = \"\";\n+\tint ret = 0;\n+\tsize_t commits_nr = 0, paths_nr = 0;\n+\tstruct commit *c;\n+\tstruct type_and_oid_list *root_tree_list;\n+\tstruct path_walk_context ctx = {\n+\t\t.repo = info->revs->repo,\n+\t\t.revs = info->revs,\n+\t\t.info = info,\n+\t\t.path_stack = STRING_LIST_INIT_DUP,\n+\t\t.path_stack_pushed = STRSET_INIT,\n+\t\t.paths_to_lists = STRMAP_INIT\n+\t};\n+\n+\ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\t/* Insert a single list for the root tree into the paths. */\n+\tCALLOC_ARRAY(root_tree_list, 1);\n+\troot_tree_list->type = OBJ_TREE;\n+\tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n+\tpush_to_stack(&ctx, root_path);\n+\n+\tif (prepare_revision_walk(info->revs))\n+\t\tdie(_(\"failed to setup revision walk\"));\n+\n+\twhile ((c = get_revision(info->revs))) {\n+\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct tree *t;\n+\t\tcommits_nr++;\n+\n+\t\toid = get_commit_tree_oid(c);\n+\t\tt = lookup_tree(info->revs->repo, oid);\n+\n+\t\tif (!t) {\n+\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (t->object.flags & SEEN)\n+\t\t\tcontinue;\n+\t\tt->object.flags |= SEEN;\n+\t\toid_array_append(&root_tree_list->oids, oid);\n+\t}\n+\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n+\ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n+\twhile (!ret && ctx.path_stack.nr) {\n+\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\tctx.path_stack.nr--;\n+\t\tpaths_nr++;\n+\n+\t\tret = walk_path(&ctx, path);\n+\n+\t\tfree(path);\n+\t}\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n+\ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n+\n+\tclear_strmap(&ctx.paths_to_lists);\n+\tstrset_clear(&ctx.path_stack_pushed);\n+\tstring_list_clear(&ctx.path_stack, 0);\n+\treturn ret;\n+}\ndiff --git a/path-walk.h b/path-walk.h\nnew file mode 100644\nindex 00000000000..c9e94a98bc8\n--- /dev/null\n+++ b/path-walk.h\n@@ -0,0 +1,43 @@\n+/*\n+ * path-walk.h : Methods and structures for walking the object graph in batches\n+ * by the paths that can reach those objects.\n+ */\n+#include \"object.h\" /* Required for 'enum object_type'. */\n+\n+struct rev_info;\n+struct oid_array;\n+\n+/**\n+ * The type of a function pointer for the method that is called on a list of\n+ * objects reachable at a given path.\n+ */\n+typedef int (*path_fn)(const char *path,\n+\t\t struct oid_array *oids,\n+\t\t enum object_type type,\n+\t\t void *data);\n+\n+struct path_walk_info {\n+\t/**\n+\t * revs provides the definitions for the commit walk, including\n+\t * which commits are UNINTERESTING or not.\n+\t */\n+\tstruct rev_info *revs;\n+\n+\t/**\n+\t * The caller wishes to execute custom logic on objects reachable at a\n+\t * given path. Every reachable object will be visited exactly once, and\n+\t * the first path to see an object wins. This may not be a stable choice.\n+\t */\n+\tpath_fn path_fn;\n+\tvoid *path_fn_data;\n+};\n+\n+#define PATH_WALK_INFO_INIT { 0 }\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info);\n-- \ngitgitgadget\n\n"},{"id":"506359","messageId":"4f9f898fec1d88d5107a10c4b953caaa98a51f17.1730356023.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH 2/6] test-lib-functions: add test_cmp_sorted","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:26:59Z","receivedAt":"2024-10-31T06:27:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis test helper will be helpful to reduce repeated logic in\nt6601-path-walk.sh, but may be helpful elsewhere, too.\n\nSigned-off-by: Derrick Stolee \n---\n t/test-lib-functions.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex fde9bf54fc3..16b70aebd60 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1267,6 +1267,16 @@ test_cmp () {\n \teval \"$GIT_TEST_CMP\" '\"$@\"'\n }\n \n+# test_cmp_sorted runs test_cmp on sorted versions of the two\n+# input files. Uses \"$1.sorted\" and \"$2.sorted\" as temp files.\n+\n+test_cmp_sorted () {\n+\tsort <\"$1\" >\"$1.sorted\" &&\n+\tsort <\"$2\" >\"$2.sorted\" &&\n+\ttest_cmp \"$1.sorted\" \"$2.sorted\" &&\n+\trm \"$1.sorted\" \"$2.sorted\"\n+}\n+\n # Check that the given config key has the expected value.\n #\n # test_cmp_config [-C ] \n-- \ngitgitgadget\n\n"},{"id":"506361","messageId":"6f93dff88e7cf81800c8f2c8d39175d962a0576f.1730356023.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:27:00Z","receivedAt":"2024-10-31T06:27:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests based on the current behavior, doing interesting checks\nfor different sets of branches, ranges, and the --boundary option. This\nsets a baseline for the behavior and we can extend it as new options are\nintroduced.\n\nIt is important to mention that the behavior of the API will change soon as\nwe start to handle UNINTERESTING objects differently, but these tests will\ndemonstrate the change in behavior.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 3 +-\n Makefile | 1 +\n t/helper/test-path-walk.c | 86 ++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 118 ++++++++++++++++++++++\n 6 files changed, 209 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex c550c77ca30..662162ec70b 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -42,4 +42,5 @@ commits.\n Examples\n --------\n \n-See example usages in future changes.\n+See example usages in:\n+\t`t/helper/test-path-walk.c`\ndiff --git a/Makefile b/Makefile\nindex d0d8d6888e3..50413d96492 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -818,6 +818,7 @@ TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n+TEST_BUILTINS_OBJS += test-path-walk.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\n TEST_BUILTINS_OBJS += test-proc-receive.o\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nnew file mode 100644\nindex 00000000000..3c48f017fa0\n--- /dev/null\n+++ b/t/helper/test-path-walk.c\n@@ -0,0 +1,86 @@\n+#define USE_THE_REPOSITORY_VARIABLE\n+\n+#include \"test-tool.h\"\n+#include \"environment.h\"\n+#include \"hex.h\"\n+#include \"object-name.h\"\n+#include \"object.h\"\n+#include \"pretty.h\"\n+#include \"revision.h\"\n+#include \"setup.h\"\n+#include \"parse-options.h\"\n+#include \"path-walk.h\"\n+#include \"oid-array.h\"\n+\n+static const char * const path_walk_usage[] = {\n+\tN_(\"test-tool path-walk -- \"),\n+\tNULL\n+};\n+\n+struct path_walk_test_data {\n+\tuintmax_t tree_nr;\n+\tuintmax_t blob_nr;\n+};\n+\n+static int emit_block(const char *path, struct oid_array *oids,\n+\t\t enum object_type type, void *data)\n+{\n+\tstruct path_walk_test_data *tdata = data;\n+\tconst char *typestr;\n+\n+\tswitch (type) {\n+\tcase OBJ_TREE:\n+\t\ttypestr = \"TREE\";\n+\t\ttdata->tree_nr += oids->nr;\n+\t\tbreak;\n+\n+\tcase OBJ_BLOB:\n+\t\ttypestr = \"BLOB\";\n+\t\ttdata->blob_nr += oids->nr;\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tBUG(\"we do not understand this type\");\n+\t}\n+\n+\tfor (size_t i = 0; i < oids->nr; i++)\n+\t\tprintf(\"%s:%s:%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]));\n+\n+\treturn 0;\n+}\n+\n+int cmd__path_walk(int argc, const char **argv)\n+{\n+\tint res;\n+\tstruct rev_info revs = REV_INFO_INIT;\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tstruct path_walk_test_data data = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tinitialize_repository(the_repository);\n+\tsetup_git_directory();\n+\trevs.repo = the_repository;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, path_walk_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT | PARSE_OPT_KEEP_ARGV0);\n+\n+\tif (argc > 1)\n+\t\tsetup_revisions(argc, argv, &revs, NULL);\n+\telse\n+\t\tusage(path_walk_usage[0]);\n+\n+\tinfo.revs = &revs;\n+\tinfo.path_fn = emit_block;\n+\tinfo.path_fn_data = &data;\n+\n+\tres = walk_objects_by_path(&info);\n+\n+\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\t \"blobs:%\" PRIuMAX \"\\n\",\n+\t data.tree_nr, data.blob_nr);\n+\n+\treturn res;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..43676e7b93a 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -52,6 +52,7 @@ static struct test_cmd cmds[] = {\n \t{ \"parse-subcommand\", cmd__parse_subcommand },\n \t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n+\t{ \"path-walk\", cmd__path_walk },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"proc-receive\", cmd__proc_receive },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..9cfc5da6e57 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__parse_pathspec_file(int argc, const char** argv);\n int cmd__parse_subcommand(int argc, const char **argv);\n int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n+int cmd__path_walk(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\n int cmd__proc_receive(int argc, const char **argv);\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nnew file mode 100755\nindex 00000000000..1f277b88291\n--- /dev/null\n+++ b/t/t6601-path-walk.sh\n@@ -0,0 +1,118 @@\n+#!/bin/sh\n+\n+test_description='direct path-walk API tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repository' '\n+\tgit checkout -b base &&\n+\n+\tmkdir left &&\n+\tmkdir right &&\n+\techo a >a &&\n+\techo b >left/b &&\n+\techo c >right/c &&\n+\tgit add . &&\n+\tgit commit -m \"first\" &&\n+\n+\techo d >right/d &&\n+\tgit add right &&\n+\tgit commit -m \"second\" &&\n+\n+\techo bb >left/b &&\n+\tgit commit -a -m \"third\" &&\n+\n+\tgit checkout -b topic HEAD~1 &&\n+\techo cc >right/c &&\n+\tgit commit -a -m \"topic\"\n+'\n+\n+test_expect_success 'all' '\n+\ttest-tool path-walk -- --all >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE::$(git rev-parse base^{tree})\n+\tTREE::$(git rev-parse base~1^{tree})\n+\tTREE::$(git rev-parse base~2^{tree})\n+\tTREE:left/:$(git rev-parse base:left)\n+\tTREE:left/:$(git rev-parse base~2:left)\n+\tTREE:right/:$(git rev-parse topic:right)\n+\tTREE:right/:$(git rev-parse base~1:right)\n+\tTREE:right/:$(git rev-parse base~2:right)\n+\ttrees:9\n+\tBLOB:a:$(git rev-parse base~2:a)\n+\tBLOB:left/b:$(git rev-parse base~2:left/b)\n+\tBLOB:left/b:$(git rev-parse base:left/b)\n+\tBLOB:right/c:$(git rev-parse base~2:right/c)\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n+\tblobs:6\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic only' '\n+\ttest-tool path-walk -- topic >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE::$(git rev-parse base~1^{tree})\n+\tTREE::$(git rev-parse base~2^{tree})\n+\tTREE:left/:$(git rev-parse base~2:left)\n+\tTREE:right/:$(git rev-parse topic:right)\n+\tTREE:right/:$(git rev-parse base~1:right)\n+\tTREE:right/:$(git rev-parse base~2:right)\n+\ttrees:7\n+\tBLOB:a:$(git rev-parse base~2:a)\n+\tBLOB:left/b:$(git rev-parse base~2:left/b)\n+\tBLOB:right/c:$(git rev-parse base~2:right/c)\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n+\tblobs:5\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base' '\n+\ttest-tool path-walk -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE:left/:$(git rev-parse topic:left)\n+\tTREE:right/:$(git rev-parse topic:right)\n+\ttrees:3\n+\tBLOB:a:$(git rev-parse topic:a)\n+\tBLOB:left/b:$(git rev-parse topic:left/b)\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse topic:right/d)\n+\tblobs:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, boundary' '\n+\ttest-tool path-walk -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE::$(git rev-parse base~1^{tree})\n+\tTREE:left/:$(git rev-parse base~1:left)\n+\tTREE:right/:$(git rev-parse topic:right)\n+\tTREE:right/:$(git rev-parse base~1:right)\n+\ttrees:5\n+\tBLOB:a:$(git rev-parse base~1:a)\n+\tBLOB:left/b:$(git rev-parse base~1:left/b)\n+\tBLOB:right/c:$(git rev-parse base~1:right/c)\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n+\tblobs:5\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"506362","messageId":"f4bf8be30b53730cd08f3e1bd85be85947f7ff8b.1730356023.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH 4/6] path-walk: allow consumer to specify object types","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:27:01Z","receivedAt":"2024-10-31T06:27:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe add the ability to filter the object types in the path-walk API so\nthe callback function is called fewer times.\n\nThis adds the ability to ask for the commits in a list, as well. We\nre-use the empty string for this set of objects because these are passed\ndirectly to the callback function instead of being part of the\n'path_stack'.\n\nFuture changes will add the ability to visit annotated tags.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 9 ++++\n path-walk.c | 33 ++++++++++--\n path-walk.h | 14 ++++-\n t/helper/test-path-walk.c | 17 +++++-\n t/t6601-path-walk.sh | 63 +++++++++++++++++++++++\n 5 files changed, 129 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 662162ec70b..dce553b6114 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,6 +39,15 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n+`commits`, `blobs`, `trees`::\n+\tBy default, these members are enabled and signal that the path-walk\n+\tAPI should call the `path_fn` on objects of these types. Specialized\n+\tapplications could disable some options to make it simpler to walk\n+\tthe objects or to have fewer calls to `path_fn`.\n++\n+While it is possible to walk only commits in this way, consumers would be\n+better off using the revision walk API instead.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex 9dc56aff88c..14ad322bdd2 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -98,6 +98,10 @@ static int add_children(struct path_walk_context *ctx,\n \t\tif (S_ISGITLINK(entry.mode))\n \t\t\tcontinue;\n \n+\t\t/* If the caller doesn't want blobs, then don't bother. */\n+\t\tif (!ctx->info->blobs && type == OBJ_BLOB)\n+\t\t\tcontinue;\n+\n \t\tif (type == OBJ_TREE) {\n \t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n \t\t\to = child ? &child->object : NULL;\n@@ -154,9 +158,11 @@ static int walk_path(struct path_walk_context *ctx,\n \n \tlist = strmap_get(&ctx->paths_to_lists, path);\n \n-\t/* Evaluate function pointer on this data. */\n-\tret = ctx->info->path_fn(path, &list->oids, list->type,\n-\t\t\t\t ctx->info->path_fn_data);\n+\t/* Evaluate function pointer on this data, if requested. */\n+\tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n+\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t\tctx->info->path_fn_data);\n \n \t/* Expand data for children. */\n \tif (list->type == OBJ_TREE) {\n@@ -198,6 +204,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n \tstruct type_and_oid_list *root_tree_list;\n+\tstruct type_and_oid_list *commit_list;\n \tstruct path_walk_context ctx = {\n \t\t.repo = info->revs->repo,\n \t\t.revs = info->revs,\n@@ -209,6 +216,9 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\tCALLOC_ARRAY(commit_list, 1);\n+\tcommit_list->type = OBJ_COMMIT;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n@@ -219,10 +229,18 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\tdie(_(\"failed to setup revision walk\"));\n \n \twhile ((c = get_revision(info->revs))) {\n-\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct object_id *oid;\n \t\tstruct tree *t;\n \t\tcommits_nr++;\n \n+\t\tif (info->commits)\n+\t\t\toid_array_append(&commit_list->oids,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t/* If we only care about commits, then skip trees. */\n+\t\tif (!info->trees && !info->blobs)\n+\t\t\tcontinue;\n+\n \t\toid = get_commit_tree_oid(c);\n \t\tt = lookup_tree(info->revs->repo, oid);\n \n@@ -240,6 +258,13 @@ int walk_objects_by_path(struct path_walk_info *info)\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n \ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\t/* Track all commits. */\n+\tif (info->commits)\n+\t\tret = info->path_fn(\"\", &commit_list->oids, OBJ_COMMIT,\n+\t\t\t\t info->path_fn_data);\n+\toid_array_clear(&commit_list->oids);\n+\tfree(commit_list);\n+\n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n \twhile (!ret && ctx.path_stack.nr) {\n \t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\ndiff --git a/path-walk.h b/path-walk.h\nindex c9e94a98bc8..2d2afc29b47 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -30,9 +30,21 @@ struct path_walk_info {\n \t */\n \tpath_fn path_fn;\n \tvoid *path_fn_data;\n+\n+\t/**\n+\t * Initialize which object types the path_fn should be called on. This\n+\t * could also limit the walk to skip blobs if not set.\n+\t */\n+\tint commits;\n+\tint trees;\n+\tint blobs;\n };\n \n-#define PATH_WALK_INFO_INIT { 0 }\n+#define PATH_WALK_INFO_INIT { \\\n+\t.blobs = 1,\t\t\\\n+\t.trees = 1,\t\t\\\n+\t.commits = 1,\t\t\\\n+}\n \n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 3c48f017fa0..37c5e3e31e8 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -18,6 +18,7 @@ static const char * const path_walk_usage[] = {\n };\n \n struct path_walk_test_data {\n+\tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n };\n@@ -29,6 +30,11 @@ static int emit_block(const char *path, struct oid_array *oids,\n \tconst char *typestr;\n \n \tswitch (type) {\n+\tcase OBJ_COMMIT:\n+\t\ttypestr = \"COMMIT\";\n+\t\ttdata->commit_nr += oids->nr;\n+\t\tbreak;\n+\n \tcase OBJ_TREE:\n \t\ttypestr = \"TREE\";\n \t\ttdata->tree_nr += oids->nr;\n@@ -56,6 +62,12 @@ int cmd__path_walk(int argc, const char **argv)\n \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n \tstruct path_walk_test_data data = { 0 };\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"blobs\", &info.blobs,\n+\t\t\t N_(\"toggle inclusion of blob objects\")),\n+\t\tOPT_BOOL(0, \"commits\", &info.commits,\n+\t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"trees\", &info.trees,\n+\t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -78,9 +90,10 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tres = walk_objects_by_path(&info);\n \n-\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\tprintf(\"commits:%\" PRIuMAX \"\\n\"\n+\t \"trees:%\" PRIuMAX \"\\n\"\n \t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.tree_nr, data.blob_nr);\n+\t data.commit_nr, data.tree_nr, data.blob_nr);\n \n \treturn res;\n }\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 1f277b88291..4b16a0a3c80 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -31,6 +31,11 @@ test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n \tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tCOMMIT::$(git rev-parse base)\n+\tCOMMIT::$(git rev-parse base~1)\n+\tCOMMIT::$(git rev-parse base~2)\n+\tcommits:4\n \tTREE::$(git rev-parse topic^{tree})\n \tTREE::$(git rev-parse base^{tree})\n \tTREE::$(git rev-parse base~1^{tree})\n@@ -57,6 +62,10 @@ test_expect_success 'topic only' '\n \ttest-tool path-walk -- topic >out &&\n \n \tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tCOMMIT::$(git rev-parse base~1)\n+\tCOMMIT::$(git rev-parse base~2)\n+\tcommits:3\n \tTREE::$(git rev-parse topic^{tree})\n \tTREE::$(git rev-parse base~1^{tree})\n \tTREE::$(git rev-parse base~2^{tree})\n@@ -80,6 +89,8 @@ test_expect_success 'topic, not base' '\n \ttest-tool path-walk -- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tcommits:1\n \tTREE::$(git rev-parse topic^{tree})\n \tTREE:left/:$(git rev-parse topic:left)\n \tTREE:right/:$(git rev-parse topic:right)\n@@ -94,10 +105,62 @@ test_expect_success 'topic, not base' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, only blobs' '\n+\ttest-tool path-walk --no-trees --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n+\ttrees:0\n+\tBLOB:a:$(git rev-parse topic:a)\n+\tBLOB:left/b:$(git rev-parse topic:left/b)\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse topic:right/d)\n+\tblobs:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+# No, this doesn't make a lot of sense for the path-walk API,\n+# but it is possible to do.\n+test_expect_success 'topic, not base, only commits' '\n+\ttest-tool path-walk --no-blobs --no-trees \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tcommits:1\n+\ttrees:0\n+\tblobs:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only trees' '\n+\ttest-tool path-walk --no-blobs --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE:left/:$(git rev-parse topic:left)\n+\tTREE:right/:$(git rev-parse topic:right)\n+\ttrees:3\n+\tblobs:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'topic, not base, boundary' '\n \ttest-tool path-walk -- --boundary topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tCOMMIT::$(git rev-parse base~1)\n+\tcommits:2\n \tTREE::$(git rev-parse topic^{tree})\n \tTREE::$(git rev-parse base~1^{tree})\n \tTREE:left/:$(git rev-parse base~1:left)\n-- \ngitgitgadget\n\n"},{"id":"506363","messageId":"3dc27658526784edf46f67661d1161a8eea3ea0a.1730356023.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH 5/6] path-walk: visit tags and cached objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:27:02Z","receivedAt":"2024-10-31T06:27:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe rev_info that is specified for a path-walk traversal may specify\nvisiting tag refs (both lightweight and annotated) and also may specify\nindexed objects (blobs and trees). Update the path-walk API to walk\nthese objects as well.\n\nWhen walking tags, we need to peel the annotated objects until reaching\na non-tag object. If we reach a commit, then we can add it to the\npending objects to make sure we visit in the commit walk portion. If we\nreach a tree, then we will assume that it is a root tree. If we reach a\nblob, then we have no good path name and so add it to a new list of\n\"tagged blobs\".\n\nWhen the rev_info includes the \"--indexed-objects\" flag, then the\npending set includes blobs and trees found in the cache entries and\ncache-tree. The cache entries are usually blobs, though they could be\ntrees in the case of a sparse index. The cache-tree stores\npreviously-hashed tree objects but these are cleared out when staging\nobjects below those paths. We add tests that demonstrate this.\n\nThe indexed objects come with a non-NULL 'path' value in the pending\nitem. This allows us to prepopulate the 'path_to_lists' strmap with\nlists for these paths.\n\nThe tricky thing about this walk is that we will want to combine the\nindexed objects walk with the commit walk, especially in the future case\nof walking objects during a command like 'git repack'.\n\nWhenever possible, we want the objects from the index to be grouped with\nsimilar objects in history. We don't want to miss any paths that appear\nonly in the index and not in the commit history.\n\nThus, we need to be careful to let the path stack be populated initially\nwith only the root tree path (and possibly tags and tagged blobs) and go\nthrough the normal depth-first search. Afterwards, if there are other\npaths that are remaining in the paths_to_lists strmap, we should then\niterate through the stack and visit those objects recursively.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 2 +-\n path-walk.c | 175 +++++++++++++++++++++-\n path-walk.h | 2 +\n t/helper/test-path-walk.c | 13 +-\n t/t6601-path-walk.sh | 154 ++++++++++++++++++-\n 5 files changed, 336 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex dce553b6114..6022c381b7c 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,7 +39,7 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n-`commits`, `blobs`, `trees`::\n+`commits`, `blobs`, `trees`, `tags`::\n \tBy default, these members are enabled and signal that the path-walk\n \tAPI should call the `path_fn` on objects of these types. Specialized\n \tapplications could disable some options to make it simpler to walk\ndiff --git a/path-walk.c b/path-walk.c\nindex 14ad322bdd2..eca0e5f3d5b 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -13,10 +13,13 @@\n #include \"revision.h\"\n #include \"string-list.h\"\n #include \"strmap.h\"\n+#include \"tag.h\"\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n \n+static const char *root_path = \"\";\n+\n struct type_and_oid_list\n {\n \tenum object_type type;\n@@ -158,9 +161,13 @@ static int walk_path(struct path_walk_context *ctx,\n \n \tlist = strmap_get(&ctx->paths_to_lists, path);\n \n+\tif (!list)\n+\t\tBUG(\"provided path '%s' that had no associated list\", path);\n+\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n-\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n+\t (list->type == OBJ_TAG && ctx->info->tags))\n \t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n \t\t\t\t\tctx->info->path_fn_data);\n \n@@ -191,6 +198,134 @@ static void clear_strmap(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static void setup_pending_objects(struct path_walk_info *info,\n+\t\t\t\t struct path_walk_context *ctx)\n+{\n+\tstruct type_and_oid_list *tags = NULL;\n+\tstruct type_and_oid_list *tagged_blobs = NULL;\n+\tstruct type_and_oid_list *root_tree_list = NULL;\n+\n+\tif (info->tags)\n+\t\tCALLOC_ARRAY(tags, 1);\n+\tif (info->blobs)\n+\t\tCALLOC_ARRAY(tagged_blobs, 1);\n+\tif (info->trees)\n+\t\troot_tree_list = strmap_get(&ctx->paths_to_lists, root_path);\n+\n+\t/*\n+\t * Pending objects include:\n+\t * * Commits at branch tips.\n+\t * * Annotated tags at tag tips.\n+\t * * Any kind of object at lightweight tag tips.\n+\t * * Trees and blobs in the index (with an associated path).\n+\t */\n+\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n+\t\tstruct object *obj = pending->item;\n+\n+\t\t/* Commits will be picked up by revision walk. */\n+\t\tif (obj->type == OBJ_COMMIT)\n+\t\t\tcontinue;\n+\n+\t\t/* Navigate annotated tag object chains. */\n+\t\twhile (obj->type == OBJ_TAG) {\n+\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n+\t\t\tif (!tag)\n+\t\t\t\tbreak;\n+\t\t\tif (tag->object.flags & SEEN)\n+\t\t\t\tbreak;\n+\t\t\ttag->object.flags |= SEEN;\n+\n+\t\t\tif (tags)\n+\t\t\t\toid_array_append(&tags->oids, &obj->oid);\n+\t\t\tobj = tag->tagged;\n+\t\t}\n+\n+\t\tif (obj->type == OBJ_TAG)\n+\t\t\tcontinue;\n+\n+\t\t/* We are now at a non-tag object. */\n+\t\tif (obj->flags & SEEN)\n+\t\t\tcontinue;\n+\t\tobj->flags |= SEEN;\n+\n+\t\tswitch (obj->type) {\n+\t\tcase OBJ_TREE:\n+\t\t\tif (!info->trees)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = *pending->path ? xstrfmt(\"%s/\", pending->path)\n+\t\t\t\t\t\t\t : xstrdup(\"\");\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_TREE;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t\tfree(path);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&root_tree_list->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (!info->blobs)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = pending->path;\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_BLOB;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&tagged_blobs->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_COMMIT:\n+\t\t\t/* Make sure it is in the object walk */\n+\t\t\tif (obj != pending->item)\n+\t\t\t\tadd_pending_object(info->revs, obj, \"\");\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"should not see any other type here\");\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Add tag objects and tagged blobs if they exist.\n+\t */\n+\tif (tagged_blobs) {\n+\t\tif (tagged_blobs->oids.nr) {\n+\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n+\t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\tpush_to_stack(ctx, tagged_blob_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n+\t\t} else {\n+\t\t\toid_array_clear(&tagged_blobs->oids);\n+\t\t\tfree(tagged_blobs);\n+\t\t}\n+\t}\n+\tif (tags) {\n+\t\tif (tags->oids.nr) {\n+\t\t\tconst char *tag_path = \"/tags\";\n+\t\t\ttags->type = OBJ_TAG;\n+\t\t\tpush_to_stack(ctx, tag_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n+\t\t} else {\n+\t\t\toid_array_clear(&tags->oids);\n+\t\t\tfree(tags);\n+\t\t}\n+\t}\n+}\n+\n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\n * call 'info->path_fn' on each discovered path.\n@@ -199,7 +334,6 @@ static void clear_strmap(struct strmap *map)\n */\n int walk_objects_by_path(struct path_walk_info *info)\n {\n-\tconst char *root_path = \"\";\n \tint ret = 0;\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n@@ -219,15 +353,31 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(commit_list, 1);\n \tcommit_list->type = OBJ_COMMIT;\n \n+\tif (info->tags)\n+\t\tinfo->revs->tag_objects = 1;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n+\t/*\n+\t * Set these values before preparing the walk to catch\n+\t * lightweight tags pointing to non-commits and indexed objects.\n+\t */\n+\tinfo->revs->blob_objects = info->blobs;\n+\tinfo->revs->tree_objects = info->trees;\n+\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n+\n+\ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\tsetup_pending_objects(info, &ctx);\n+\ttrace2_region_leave(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\n \twhile ((c = get_revision(info->revs))) {\n \t\tstruct object_id *oid;\n \t\tstruct tree *t;\n@@ -275,6 +425,27 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \t\tfree(path);\n \t}\n+\n+\t/* Are there paths remaining? Likely they are from indexed objects. */\n+\tif (!strmap_empty(&ctx.paths_to_lists)) {\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *entry;\n+\n+\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry) {\n+\t\t\tpush_to_stack(&ctx, entry->key);\n+\t\t}\n+\n+\t\twhile (!ret && ctx.path_stack.nr) {\n+\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\t\tctx.path_stack.nr--;\n+\t\t\tpaths_nr++;\n+\n+\t\t\tret = walk_path(&ctx, path);\n+\n+\t\t\tfree(path);\n+\t\t}\n+\t}\n+\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n \ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n \ndiff --git a/path-walk.h b/path-walk.h\nindex 2d2afc29b47..ca839f873e4 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -38,12 +38,14 @@ struct path_walk_info {\n \tint commits;\n \tint trees;\n \tint blobs;\n+\tint tags;\n };\n \n #define PATH_WALK_INFO_INIT { \\\n \t.blobs = 1,\t\t\\\n \t.trees = 1,\t\t\\\n \t.commits = 1,\t\t\\\n+\t.tags = 1,\t\t\\\n }\n \n /**\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 37c5e3e31e8..c6c60d68749 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -21,6 +21,7 @@ struct path_walk_test_data {\n \tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n+\tuintmax_t tag_nr;\n };\n \n static int emit_block(const char *path, struct oid_array *oids,\n@@ -45,6 +46,11 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->blob_nr += oids->nr;\n \t\tbreak;\n \n+\tcase OBJ_TAG:\n+\t\ttypestr = \"TAG\";\n+\t\ttdata->tag_nr += oids->nr;\n+\t\tbreak;\n+\n \tdefault:\n \t\tBUG(\"we do not understand this type\");\n \t}\n@@ -66,6 +72,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of blob objects\")),\n \t\tOPT_BOOL(0, \"commits\", &info.commits,\n \t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"tags\", &info.tags,\n+\t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n@@ -92,8 +100,9 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tprintf(\"commits:%\" PRIuMAX \"\\n\"\n \t \"trees:%\" PRIuMAX \"\\n\"\n-\t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.commit_nr, data.tree_nr, data.blob_nr);\n+\t \"blobs:%\" PRIuMAX \"\\n\"\n+\t \"tags:%\" PRIuMAX \"\\n\",\n+\t data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);\n \n \treturn res;\n }\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 4b16a0a3c80..5ed6c79fbd1 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -7,24 +7,55 @@ test_description='direct path-walk API tests'\n test_expect_success 'setup test repository' '\n \tgit checkout -b base &&\n \n+\t# Make some objects that will only be reachable\n+\t# via non-commit tags.\n+\tmkdir child &&\n+\techo file >child/file &&\n+\tgit add child &&\n+\tgit commit -m \"will abandon\" &&\n+\tgit tag -a -m \"tree\" tree-tag HEAD^{tree} &&\n+\techo file2 >file2 &&\n+\tgit add file2 &&\n+\tgit commit --amend -m \"will abandon\" &&\n+\tgit tag tree-tag2 HEAD^{tree} &&\n+\n+\techo blob >file &&\n+\tblob_oid=$(git hash-object -t blob -w --stdin file2 &&\n+\tblob2_oid=$(git hash-object -t blob -w --stdin a &&\n \techo b >left/b &&\n \techo c >right/c &&\n \tgit add . &&\n-\tgit commit -m \"first\" &&\n+\tgit commit --amend -m \"first\" &&\n+\tgit tag -m \"first\" first HEAD &&\n \n \techo d >right/d &&\n \tgit add right &&\n \tgit commit -m \"second\" &&\n+\tgit tag -a -m \"second (under)\" second.1 HEAD &&\n+\tgit tag -a -m \"second (top)\" second.2 second.1 &&\n \n+\t# Set up file/dir collision in history.\n+\trm a &&\n+\tmkdir a &&\n+\techo a >a/a &&\n \techo bb >left/b &&\n-\tgit commit -a -m \"third\" &&\n+\tgit add a left &&\n+\tgit commit -m \"third\" &&\n+\tgit tag -a -m \"third\" third &&\n \n \tgit checkout -b topic HEAD~1 &&\n \techo cc >right/c &&\n-\tgit commit -a -m \"topic\"\n+\tgit commit -a -m \"topic\" &&\n+\tgit tag -a -m \"fourth\" fourth\n '\n \n test_expect_success 'all' '\n@@ -40,19 +71,104 @@ test_expect_success 'all' '\n \tTREE::$(git rev-parse base^{tree})\n \tTREE::$(git rev-parse base~1^{tree})\n \tTREE::$(git rev-parse base~2^{tree})\n+\tTREE::$(git rev-parse refs/tags/tree-tag^{})\n+\tTREE::$(git rev-parse refs/tags/tree-tag2^{})\n+\tTREE:a/:$(git rev-parse base:a)\n \tTREE:left/:$(git rev-parse base:left)\n \tTREE:left/:$(git rev-parse base~2:left)\n \tTREE:right/:$(git rev-parse topic:right)\n \tTREE:right/:$(git rev-parse base~1:right)\n \tTREE:right/:$(git rev-parse base~2:right)\n-\ttrees:9\n+\tTREE:child/:$(git rev-parse refs/tags/tree-tag^{}:child)\n+\ttrees:13\n \tBLOB:a:$(git rev-parse base~2:a)\n+\tBLOB:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n \tBLOB:left/b:$(git rev-parse base~2:left/b)\n \tBLOB:left/b:$(git rev-parse base:left/b)\n \tBLOB:right/c:$(git rev-parse base~2:right/c)\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse base~1:right/d)\n-\tblobs:6\n+\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n+\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\tBLOB:child/file:$(git rev-parse refs/tags/tree-tag^{}:child/file)\n+\tblobs:10\n+\tTAG:/tags:$(git rev-parse refs/tags/first)\n+\tTAG:/tags:$(git rev-parse refs/tags/second.1)\n+\tTAG:/tags:$(git rev-parse refs/tags/second.2)\n+\tTAG:/tags:$(git rev-parse refs/tags/third)\n+\tTAG:/tags:$(git rev-parse refs/tags/fourth)\n+\tTAG:/tags:$(git rev-parse refs/tags/tree-tag)\n+\tTAG:/tags:$(git rev-parse refs/tags/blob-tag)\n+\ttags:7\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'indexed objects' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"left\" directory.\n+\techo bogus >left/c &&\n+\tgit add left &&\n+\n+\ttest-tool path-walk -- --indexed-objects >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n+\tTREE:right/:$(git rev-parse topic:right)\n+\ttrees:1\n+\tBLOB:a:$(git rev-parse HEAD:a)\n+\tBLOB:left/b:$(git rev-parse HEAD:left/b)\n+\tBLOB:left/c:$(git rev-parse :left/c)\n+\tBLOB:right/c:$(git rev-parse HEAD:right/c)\n+\tBLOB:right/d:$(git rev-parse HEAD:right/d)\n+\tblobs:5\n+\ttags:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'branches and indexed objects mix well' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"right\" directory.\n+\techo fake >right/d &&\n+\tgit add right &&\n+\n+\ttest-tool path-walk -- --indexed-objects --branches >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tCOMMIT::$(git rev-parse base)\n+\tCOMMIT::$(git rev-parse base~1)\n+\tCOMMIT::$(git rev-parse base~2)\n+\tcommits:4\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE::$(git rev-parse base^{tree})\n+\tTREE::$(git rev-parse base~1^{tree})\n+\tTREE::$(git rev-parse base~2^{tree})\n+\tTREE:a/:$(git rev-parse base:a)\n+\tTREE:left/:$(git rev-parse base:left)\n+\tTREE:left/:$(git rev-parse base~2:left)\n+\tTREE:right/:$(git rev-parse topic:right)\n+\tTREE:right/:$(git rev-parse base~1:right)\n+\tTREE:right/:$(git rev-parse base~2:right)\n+\ttrees:10\n+\tBLOB:a:$(git rev-parse base~2:a)\n+\tBLOB:left/b:$(git rev-parse base:left/b)\n+\tBLOB:left/b:$(git rev-parse base~2:left/b)\n+\tBLOB:right/c:$(git rev-parse base~2:right/c)\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n+\tBLOB:right/d:$(git rev-parse :right/d)\n+\tblobs:7\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -80,6 +196,7 @@ test_expect_success 'topic only' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse base~1:right/d)\n \tblobs:5\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -100,6 +217,7 @@ test_expect_success 'topic, not base' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse topic:right/d)\n \tblobs:4\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -117,6 +235,7 @@ test_expect_success 'topic, not base, only blobs' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse topic:right/d)\n \tblobs:4\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -133,6 +252,7 @@ test_expect_success 'topic, not base, only commits' '\n \tcommits:1\n \ttrees:0\n \tblobs:0\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -149,6 +269,7 @@ test_expect_success 'topic, not base, only trees' '\n \tTREE:right/:$(git rev-parse topic:right)\n \ttrees:3\n \tblobs:0\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -173,9 +294,32 @@ test_expect_success 'topic, not base, boundary' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse base~1:right/d)\n \tblobs:5\n+\ttags:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'trees are reported exactly once' '\n+\ttest_when_finished \"rm -rf unique-trees\" &&\n+\ttest_create_repo unique-trees &&\n+\t(\n+\t\tcd unique-trees &&\n+\t\tmkdir initial &&\n+\t\ttest_commit initial/file &&\n+\n+\t\tgit switch -c move-to-top &&\n+\t\tgit mv initial/file.t ./ &&\n+\t\ttest_tick &&\n+\t\tgit commit -m moved &&\n+\n+\t\tgit update-ref refs/heads/other HEAD\n+\t) &&\n+\n+\ttest-tool -C unique-trees path-walk -- --all >out &&\n+\ttree=$(git -C unique-trees rev-parse HEAD:) &&\n+\tgrep \"$tree\" out >out-filtered &&\n+\ttest_line_count = 1 out-filtered\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"506364","messageId":"0bb607e1fd31ad2c6c358a444bd4740c10e15d61.1730356023.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH 6/6] path-walk: mark trees and blobs as UNINTERESTING","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-10-31T06:27:03Z","receivedAt":"2024-10-31T06:27:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the input rev_info has UNINTERESTING starting points, we want to be\nsure that the UNINTERESTING flag is passed appropriately through the\nobjects. To match how this is done in places such as 'git pack-objects', we\nuse the mark_edges_uninteresting() method.\n\nThis method has an option for using the \"sparse\" walk, which is similar in\nspirit to the path-walk API's walk. To be sure to keep it independent, add a\nnew 'prune_all_uninteresting' option to the path_walk_info struct.\n\nTo check how the UNINTERSTING flag is spread through our objects, extend the\n'test-tool path-walk' command to output whether or not an object has that\nflag. This changes our tests significantly, including the removal of some\nobjects that were previously visited due to the incomplete implementation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 8 +++\n path-walk.c | 73 +++++++++++++++++++++\n path-walk.h | 8 +++\n t/helper/test-path-walk.c | 10 ++-\n t/t6601-path-walk.sh | 79 +++++++++++++++++------\n 5 files changed, 157 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 6022c381b7c..7075d0d5ab5 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -48,6 +48,14 @@ commits.\n While it is possible to walk only commits in this way, consumers would be\n better off using the revision walk API instead.\n \n+`prune_all_uninteresting`::\n+\tBy default, all reachable paths are emitted by the path-walk API.\n+\tThis option allows consumers to declare that they are not\n+\tinterested in paths where all included objects are marked with the\n+\t`UNINTERESTING` flag. This requires using the `boundary` option in\n+\tthe revision walk so that the walk emits commits marked with the\n+\t`UNINTERESTING` flag.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex eca0e5f3d5b..6f658c28307 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -8,6 +8,7 @@\n #include \"dir.h\"\n #include \"hashmap.h\"\n #include \"hex.h\"\n+#include \"list-objects.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n #include \"revision.h\"\n@@ -24,6 +25,7 @@ struct type_and_oid_list\n {\n \tenum object_type type;\n \tstruct oid_array oids;\n+\tint maybe_interesting;\n };\n \n #define TYPE_AND_OID_LIST_INIT { \\\n@@ -140,6 +142,9 @@ static int add_children(struct path_walk_context *ctx,\n \t\tif (o->flags & SEEN)\n \t\t\tcontinue;\n \t\to->flags |= SEEN;\n+\n+\t\tif (!(o->flags & UNINTERESTING))\n+\t\t\tlist->maybe_interesting = 1;\n \t\toid_array_append(&list->oids, &entry.oid);\n \t}\n \n@@ -164,6 +169,43 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list)\n \t\tBUG(\"provided path '%s' that had no associated list\", path);\n \n+\tif (ctx->info->prune_all_uninteresting) {\n+\t\t/*\n+\t\t * This is true if all objects were UNINTERESTING\n+\t\t * when added to the list.\n+\t\t */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\n+\t\t/*\n+\t\t * But it's still possible that the objects were set\n+\t\t * as UNINTERESTING after being added. Do a quick check.\n+\t\t */\n+\t\tlist->maybe_interesting = 0;\n+\t\tfor (size_t i = 0;\n+\t\t !list->maybe_interesting && i < list->oids.nr;\n+\t\t i++) {\n+\t\t\tif (list->type == OBJ_TREE) {\n+\t\t\t\tstruct tree *t = lookup_tree(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (t && !(t->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else if (list->type == OBJ_BLOB) {\n+\t\t\t\tstruct blob *b = lookup_blob(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (b && !(b->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else {\n+\t\t\t\t/* Tags are always interesting if visited. */\n+\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* We have confirmed that all objects are UNINTERESTING. */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\t}\n+\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n \t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n@@ -198,6 +240,26 @@ static void clear_strmap(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static struct repository *edge_repo;\n+static struct type_and_oid_list *edge_tree_list;\n+\n+static void show_edge(struct commit *commit)\n+{\n+\tstruct tree *t = repo_get_commit_tree(edge_repo, commit);\n+\n+\tif (!t)\n+\t\treturn;\n+\n+\tif (commit->object.flags & UNINTERESTING)\n+\t\tt->object.flags |= UNINTERESTING;\n+\n+\tif (t->object.flags & SEEN)\n+\t\treturn;\n+\tt->object.flags |= SEEN;\n+\n+\toid_array_append(&edge_tree_list->oids, &t->object.oid);\n+}\n+\n static void setup_pending_objects(struct path_walk_info *info,\n \t\t\t\t struct path_walk_context *ctx)\n {\n@@ -306,6 +368,7 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\tif (tagged_blobs->oids.nr) {\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\ttagged_blobs->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n \t\t} else {\n@@ -317,6 +380,7 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\tif (tags->oids.nr) {\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n+\t\t\ttags->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tag_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n \t\t} else {\n@@ -359,6 +423,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n+\troot_tree_list->maybe_interesting = 1;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n@@ -372,6 +437,14 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\t/* Walk trees to mark them as UNINTERESTING. */\n+\tedge_repo = info->revs->repo;\n+\tedge_tree_list = root_tree_list;\n+\tmark_edges_uninteresting(info->revs, show_edge,\n+\t\t\t\t info->prune_all_uninteresting);\n+\tedge_repo = NULL;\n+\tedge_tree_list = NULL;\n+\n \tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n \n \ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\ndiff --git a/path-walk.h b/path-walk.h\nindex ca839f873e4..de0db007dc9 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -39,6 +39,14 @@ struct path_walk_info {\n \tint trees;\n \tint blobs;\n \tint tags;\n+\n+\t/**\n+\t * When 'prune_all_uninteresting' is set and a path has all objects\n+\t * marked as UNINTERESTING, then the path-walk will not visit those\n+\t * objects. It will not call path_fn on those objects and will not\n+\t * walk the children of such trees.\n+\t */\n+\tint prune_all_uninteresting;\n };\n \n #define PATH_WALK_INFO_INIT { \\\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex c6c60d68749..06b103d8760 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -55,8 +55,12 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\tBUG(\"we do not understand this type\");\n \t}\n \n-\tfor (size_t i = 0; i < oids->nr; i++)\n-\t\tprintf(\"%s:%s:%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]));\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tstruct object *o = lookup_unknown_object(the_repository,\n+\t\t\t\t\t\t\t &oids->oid[i]);\n+\t\tprintf(\"%s:%s:%s%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]),\n+\t\t o->flags & UNINTERESTING ? \":UNINTERESTING\" : \"\");\n+\t}\n \n \treturn 0;\n }\n@@ -76,6 +80,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n+\t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n+\t\t\t N_(\"toggle pruning of uninteresting paths\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 5ed6c79fbd1..a561c21d484 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -209,13 +209,13 @@ test_expect_success 'topic, not base' '\n \tCOMMIT::$(git rev-parse topic)\n \tcommits:1\n \tTREE::$(git rev-parse topic^{tree})\n-\tTREE:left/:$(git rev-parse topic:left)\n+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n \tTREE:right/:$(git rev-parse topic:right)\n \ttrees:3\n-\tBLOB:a:$(git rev-parse topic:a)\n-\tBLOB:left/b:$(git rev-parse topic:left/b)\n+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n-\tBLOB:right/d:$(git rev-parse topic:right/d)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:4\n \ttags:0\n \tEOF\n@@ -223,6 +223,29 @@ test_expect_success 'topic, not base' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'fourth, blob-tag2, not base' '\n+\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tcommits:1\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\tTREE:right/:$(git rev-parse topic:right)\n+\ttrees:3\n+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\tblobs:5\n+\tTAG:/tags:$(git rev-parse fourth)\n+\ttags:1\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'topic, not base, only blobs' '\n \ttest-tool path-walk --no-trees --no-commits \\\n \t\t-- topic --not base >out &&\n@@ -230,10 +253,10 @@ test_expect_success 'topic, not base, only blobs' '\n \tcat >expect <<-EOF &&\n \tcommits:0\n \ttrees:0\n-\tBLOB:a:$(git rev-parse topic:a)\n-\tBLOB:left/b:$(git rev-parse topic:left/b)\n+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n-\tBLOB:right/d:$(git rev-parse topic:right/d)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:4\n \ttags:0\n \tEOF\n@@ -265,7 +288,7 @@ test_expect_success 'topic, not base, only trees' '\n \tcat >expect <<-EOF &&\n \tcommits:0\n \tTREE::$(git rev-parse topic^{tree})\n-\tTREE:left/:$(git rev-parse topic:left)\n+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n \tTREE:right/:$(git rev-parse topic:right)\n \ttrees:3\n \tblobs:0\n@@ -280,19 +303,19 @@ test_expect_success 'topic, not base, boundary' '\n \n \tcat >expect <<-EOF &&\n \tCOMMIT::$(git rev-parse topic)\n-\tCOMMIT::$(git rev-parse base~1)\n+\tCOMMIT::$(git rev-parse base~1):UNINTERESTING\n \tcommits:2\n \tTREE::$(git rev-parse topic^{tree})\n-\tTREE::$(git rev-parse base~1^{tree})\n-\tTREE:left/:$(git rev-parse base~1:left)\n+\tTREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n \tTREE:right/:$(git rev-parse topic:right)\n-\tTREE:right/:$(git rev-parse base~1:right)\n+\tTREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n \ttrees:5\n-\tBLOB:a:$(git rev-parse base~1:a)\n-\tBLOB:left/b:$(git rev-parse base~1:left/b)\n-\tBLOB:right/c:$(git rev-parse base~1:right/c)\n+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\tBLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n-\tBLOB:right/d:$(git rev-parse base~1:right/d)\n+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:5\n \ttags:0\n \tEOF\n@@ -300,6 +323,27 @@ test_expect_success 'topic, not base, boundary' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, boundary with pruning' '\n+\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tCOMMIT::$(git rev-parse topic)\n+\tCOMMIT::$(git rev-parse base~1):UNINTERESTING\n+\tcommits:2\n+\tTREE::$(git rev-parse topic^{tree})\n+\tTREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n+\tTREE:right/:$(git rev-parse topic:right)\n+\tTREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\ttrees:4\n+\tBLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n+\tBLOB:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\ttags:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n@@ -307,15 +351,12 @@ test_expect_success 'trees are reported exactly once' '\n \t\tcd unique-trees &&\n \t\tmkdir initial &&\n \t\ttest_commit initial/file &&\n-\n \t\tgit switch -c move-to-top &&\n \t\tgit mv initial/file.t ./ &&\n \t\ttest_tick &&\n \t\tgit commit -m moved &&\n-\n \t\tgit update-ref refs/heads/other HEAD\n \t) &&\n-\n \ttest-tool -C unique-trees path-walk -- --all >out &&\n \ttree=$(git -C unique-trees rev-parse HEAD:) &&\n \tgrep \"$tree\" out >out-filtered &&\n-- \ngitgitgadget\n"},{"id":"506388","messageId":"d467d241-4989-45aa-bf15-c2ead7b74240@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-10-31T12:36:02Z","receivedAt":"2024-10-31T12:36:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/31/24 2:26 AM, Derrick Stolee via GitGitGadget wrote:\n\n> This is a new series that rerolls the initial \"path-walk API\" patches of my\n> RFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\n> path-walk.h) presents a new way to walk objects such that trees and blobs\n> are walked in batches according to their path.\n> \n> This also replaces the previous version of ds/path-walk that was being\n> reviewed in [2]. The consensus was that the series was too long/dense and\n> could use some reduction in size. This series takes the first few patches,\n> but also makes some updates (which will be described later).\n> \n> [1]\n> https://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n> [RFC] Path-walk API and applications\n> \n> [2]\n> https://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n> [PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n...\n> I will include a full range diff relative to the previous versions of these\n> patches in [2] in a reply to this cover letter.\nHere is the promised range-diff:\n\n1: 98bdc94a773 ! 1: c71f0a0e361 path-walk: introduce an object walk by path\n @@ Commit message\n\n In anticipation of a few planned applications, introduce the most \nbasic form\n of a path-walk API. It currently assumes that there are no UNINTERESTING\n - objects and does not include any complicated filters. It calls a function\n + objects, and does not include any complicated filters. It calls a function\n pointer on groups of tree and blob objects as grouped by path. This only\n includes objects the first time they are discovered, so an object that\n appears at multiple paths will not be included in two batches.\n\n + These batches are collected in 'struct type_and_oid_list' objects, which\n + store an object type and an oid_array of objects.\n +\n + The data structures are documented in 'struct path_walk_context', but in\n + summary the most important are:\n +\n + * 'paths_to_lists' is a strmap that connects a path to a\n + type_and_oid_list for that path. To avoid conflicts in path names,\n + we make sure that tree paths end in \"/\" (except the root path with\n + is an empty string) and blob paths do not end in \"/\".\n +\n + * 'path_stack' is a string list that is added to in an append-only\n + way. This stores the stack of our depth-first search on the heap\n + instead of using recursion.\n +\n + * 'path_stack_pushed' is a strmap that stores path names that were\n + already added to 'path_stack', to avoid repeating paths in the\n + stack. Mostly, this saves us from quadratic lookups from doing\n + unsorted checks into the string_list.\n +\n + The coupling of 'path_stack' and 'path_stack_pushed' is protected by the\n + push_to_stack() method. Call this instead of inserting into these\n + structures directly.\n +\n + The walk_objects_by_path() method initializes these structures and\n + starts walking commits from the given rev_info struct. The commits are\n + used to find the list of root trees which populate the start of our\n + depth-first search.\n +\n + The core of our depth-first search is in a while loop that continues\n + while we have not indicated an early exit and our 'path_stack' still has\n + entries in it. The loop body pops a path off of the stack and \"visits\"\n + the path via the walk_path() method.\n +\n + The walk_path() method gets the list of OIDs from the 'path_to_lists'\n + strmap and executes the callback method on that list with the given path\n + and type. If the OIDs correspond to tree objects, then iterate over all\n + trees in the list and run add_children() to add the child objects to\n + their own lists, adding new entries to the stack if necessary.\n +\n + In testing, this depth-first search approach was the one that used the\n + least memory while iterating over the object lists. There is still a\n + chance that repositories with too-wide path patterns could cause memory\n + pressure issues. Limiting the stack size could be done in the future by\n + limiting how many objects are being considered in-progress, or by\n + visiting blob paths earlier than trees.\n +\n There are many future adaptations that could be made, but they are \nleft for\n future updates when consumers are ready to take advantage of those \nfeatures.\n\n @@ Documentation/technical/api-path-walk.txt (new)\n +multiple paths possible to reach the same object, then only one of those\n +paths is used to visit the object.\n +\n -+When walking a range of commits with some `UNINTERESTING` objects, the\n -+objects with the `UNINTERESTING` flag are included in these batches. In\n -+order to walk `UNINTERESTING` objects, the `--boundary` option must be\n -+used in the commit walk in order to visit `UNINTERESTING` commits.\n -+\n +Basics\n +------\n +\n @@ Documentation/technical/api-path-walk.txt (new)\n +`revs` struct. The revision walk should only be used to walk commits, and\n +the objects will be walked in a separate way based on those starting\n +commits.\n -++\n -+If you want the path-walk API to emit `UNINTERESTING` objects based on the\n -+commit walk's boundary, be sure to set `revs.boundary` so the boundary\n -+commits are emitted.\n +\n +Examples\n +--------\n @@ path-walk.c (new)\n +\t/**\n +\t * Store the current list of paths in a stack, to\n +\t * facilitate depth-first-search without recursion.\n ++\t *\n ++\t * Use path_stack_pushed to indicate whether a path\n ++\t * was previously added to path_stack.\n +\t */\n +\tstruct string_list path_stack;\n ++\tstruct strset path_stack_pushed;\n +};\n +\n ++static void push_to_stack(struct path_walk_context *ctx,\n ++\t\t\t const char *path)\n ++{\n ++\tif (strset_contains(&ctx->path_stack_pushed, path))\n ++\t\treturn;\n ++\n ++\tstrset_add(&ctx->path_stack_pushed, path);\n ++\tstring_list_append(&ctx->path_stack, path);\n ++}\n ++\n +static int add_children(struct path_walk_context *ctx,\n +\t\t\tconst char *base_path,\n +\t\t\tstruct object_id *oid)\n @@ path-walk.c (new)\n +\t\tif (!o) /* report error?*/\n +\t\t\tcontinue;\n +\n -+\t\t/* Skip this object if already seen. */\n -+\t\tif (o->flags & SEEN)\n -+\t\t\tcontinue;\n -+\t\to->flags |= SEEN;\n -+\n +\t\tstrbuf_setlen(&path, base_len);\n +\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n +\n @@ path-walk.c (new)\n +\t\t\tCALLOC_ARRAY(list, 1);\n +\t\t\tlist->type = type;\n +\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n -+\t\t\tstring_list_append(&ctx->path_stack, path.buf);\n +\t\t}\n ++\t\tpush_to_stack(ctx, path.buf);\n ++\n ++\t\t/* Skip this object if already seen. */\n ++\t\tif (o->flags & SEEN)\n ++\t\t\tcontinue;\n ++\t\to->flags |= SEEN;\n +\t\toid_array_append(&list->oids, &entry.oid);\n +\t}\n +\n @@ path-walk.c (new)\n +\t\t.revs = info->revs,\n +\t\t.info = info,\n +\t\t.path_stack = STRING_LIST_INIT_DUP,\n ++\t\t.path_stack_pushed = STRSET_INIT,\n +\t\t.paths_to_lists = STRMAP_INIT\n +\t};\n +\n @@ path-walk.c (new)\n +\tCALLOC_ARRAY(root_tree_list, 1);\n +\troot_tree_list->type = OBJ_TREE;\n +\tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n ++\tpush_to_stack(&ctx, root_path);\n +\n +\tif (prepare_revision_walk(info->revs))\n +\t\tdie(_(\"failed to setup revision walk\"));\n +\n +\twhile ((c = get_revision(info->revs))) {\n +\t\tstruct object_id *oid = get_commit_tree_oid(c);\n -+\t\tstruct tree *t = lookup_tree(info->revs->repo, oid);\n ++\t\tstruct tree *t;\n +\t\tcommits_nr++;\n +\n -+\t\tif (t) {\n -+\t\t\tif (t->object.flags & SEEN)\n -+\t\t\t\tcontinue;\n -+\t\t\tt->object.flags |= SEEN;\n -+\t\t\toid_array_append(&root_tree_list->oids, oid);\n -+\t\t} else {\n ++\t\toid = get_commit_tree_oid(c);\n ++\t\tt = lookup_tree(info->revs->repo, oid);\n ++\n ++\t\tif (!t) {\n +\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n ++\t\t\tcontinue;\n +\t\t}\n ++\n ++\t\tif (t->object.flags & SEEN)\n ++\t\t\tcontinue;\n ++\t\tt->object.flags |= SEEN;\n ++\t\toid_array_append(&root_tree_list->oids, oid);\n +\t}\n +\n +\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n +\ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n +\n -+\tstring_list_append(&ctx.path_stack, root_path);\n -+\n +\ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n +\twhile (!ret && ctx.path_stack.nr) {\n +\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n @@ path-walk.c (new)\n +\ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n +\n +\tclear_strmap(&ctx.paths_to_lists);\n ++\tstrset_clear(&ctx.path_stack_pushed);\n +\tstring_list_clear(&ctx.path_stack, 0);\n +\treturn ret;\n +}\n5: 6e89fb219b5 ! 2: 4f9f898fec1 revision: create mark_trees_uninteresting_dense()\n @@ Metadata\n Author: Derrick Stolee \n\n ## Commit message ##\n - revision: create mark_trees_uninteresting_dense()\n + test-lib-functions: add test_cmp_sorted\n\n - The sparse tree walk algorithm was created in d5d2e93577e (revision:\n - implement sparse algorithm, 2019-01-16) and involves using the\n - mark_trees_uninteresting_sparse() method. This method takes a repository\n - and an oidset of tree IDs, some of which have the UNINTERESTING flag and\n - some of which do not.\n -\n - Create a method that has an equivalent set of preconditions but uses a\n - \"dense\" walk (recursively visits all reachable trees, as long as they\n - have not previously been marked UNINTERESTING). This is an important\n - difference from mark_tree_uninteresting(), which short-circuits if the\n - given tree has the UNINTERESTING flag.\n -\n - A use of this method will be added in a later change, with a condition\n - set whether the sparse or dense approach should be used.\n + This test helper will be helpful to reduce repeated logic in\n + t6601-path-walk.sh, but may be helpful elsewhere, too.\n\n Signed-off-by: Derrick Stolee \n\n - ## revision.c ##\n -@@ revision.c: static void add_children_by_path(struct repository *r,\n - \tfree_tree_buffer(tree);\n + ## t/test-lib-functions.sh ##\n +@@ t/test-lib-functions.sh: test_cmp () {\n + \teval \"$GIT_TEST_CMP\" '\"$@\"'\n }\n\n -+void mark_trees_uninteresting_dense(struct repository *r,\n -+\t\t\t\t struct oidset *trees)\n -+{\n -+\tstruct object_id *oid;\n -+\tstruct oidset_iter iter;\n -+\n -+\toidset_iter_init(trees, &iter);\n -+\twhile ((oid = oidset_iter_next(&iter))) {\n -+\t\tstruct tree *tree = lookup_tree(r, oid);\n ++# test_cmp_sorted runs test_cmp on sorted versions of the two\n ++# input files. Uses \"$1.sorted\" and \"$2.sorted\" as temp files.\n +\n -+\t\tif (tree && (tree->object.flags & UNINTERESTING))\n -+\t\t\tmark_tree_contents_uninteresting(r, tree);\n -+\t}\n ++test_cmp_sorted () {\n ++\tsort <\"$1\" >\"$1.sorted\" &&\n ++\tsort <\"$2\" >\"$2.sorted\" &&\n ++\ttest_cmp \"$1.sorted\" \"$2.sorted\" &&\n ++\trm \"$1.sorted\" \"$2.sorted\"\n +}\n +\n - void mark_trees_uninteresting_sparse(struct repository *r,\n - \t\t\t\t struct oidset *trees)\n - {\n -\n - ## revision.h ##\n -@@ revision.h: void put_revision_mark(const struct rev_info *revs,\n -\n - void mark_parents_uninteresting(struct rev_info *revs, struct commit \n*commit);\n - void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n -+void mark_trees_uninteresting_dense(struct repository *r, struct oidset \n*trees);\n - void mark_trees_uninteresting_sparse(struct repository *r, struct oidset \n*trees);\n -\n - void show_object_with_name(FILE *, struct object *, const char *);\n + # Check that the given config key has the expected value.\n + #\n + # test_cmp_config [-C ] \n2: a00ab0c62c9 ! 3: 6f93dff88e7 t6601: add helper for testing path-walk API\n @@ Commit message\n sets a baseline for the behavior and we can extend it as new options are\n introduced.\n\n + It is important to mention that the behavior of the API will change \nsoon as\n + we start to handle UNINTERESTING objects differently, but these tests will\n + demonstrate the change in behavior.\n +\n Signed-off-by: Derrick Stolee \n\n ## Documentation/technical/api-path-walk.txt ##\n -@@ Documentation/technical/api-path-walk.txt: commits are emitted.\n +@@ Documentation/technical/api-path-walk.txt: commits.\n Examples\n --------\n\n @@ t/t6601-path-walk.sh (new)\n +\tblobs:6\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n +test_expect_success 'topic only' '\n @@ t/t6601-path-walk.sh (new)\n +\tblobs:5\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n +test_expect_success 'topic, not base' '\n @@ t/t6601-path-walk.sh (new)\n +\tblobs:4\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n +test_expect_success 'topic, not base, boundary' '\n @@ t/t6601-path-walk.sh (new)\n +\tblobs:5\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n +test_done\n3: 14375d19392 ! 4: f4bf8be30b5 path-walk: allow consumer to specify object types\n @@ Commit message\n We add the ability to filter the object types in the path-walk API so\n the callback function is called fewer times.\n\n - This adds the ability to ask for the commits in a list, as well. Future\n - changes will add the ability to visit annotated tags.\n + This adds the ability to ask for the commits in a list, as well. We\n + re-use the empty string for this set of objects because these are passed\n + directly to the callback function instead of being part of the\n + 'path_stack'.\n +\n + Future changes will add the ability to visit annotated tags.\n\n Signed-off-by: Derrick Stolee \n\n ## Documentation/technical/api-path-walk.txt ##\n -@@ Documentation/technical/api-path-walk.txt: If you want the path-walk \nAPI to emit `UNINTERESTING` objects based on the\n - commit walk's boundary, be sure to set `revs.boundary` so the boundary\n - commits are emitted.\n +@@ Documentation/technical/api-path-walk.txt: It is also important that \nyou do not specify the `--objects` flag for the\n + the objects will be walked in a separate way based on those starting\n + commits.\n\n +`commits`, `blobs`, `trees`::\n +\tBy default, these members are enabled and signal that the path-walk\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n - \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n --\n - \tif (prepare_revision_walk(info->revs))\n +@@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \t\tdie(_(\"failed to setup revision walk\"));\n\n \twhile ((c = get_revision(info->revs))) {\n -\t\tstruct object_id *oid = get_commit_tree_oid(c);\n --\t\tstruct tree *t = lookup_tree(info->revs->repo, oid);\n +\t\tstruct object_id *oid;\n -+\t\tstruct tree *t;\n + \t\tstruct tree *t;\n \t\tcommits_nr++;\n\n +\t\tif (info->commits)\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n +\t\tif (!info->trees && !info->blobs)\n +\t\t\tcontinue;\n +\n -+\t\toid = get_commit_tree_oid(c);\n -+\t\tt = lookup_tree(info->revs->repo, oid);\n -+\n - \t\tif (t) {\n - \t\t\tif (t->object.flags & SEEN)\n - \t\t\t\tcontinue;\n + \t\toid = get_commit_tree_oid(c);\n + \t\tt = lookup_tree(info->revs->repo, oid);\n +\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n \ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n +\toid_array_clear(&commit_list->oids);\n +\tfree(commit_list);\n +\n - \tstring_list_append(&ctx.path_stack, root_path);\n -\n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n + \twhile (!ret && ctx.path_stack.nr) {\n + \t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n\n ## path-walk.h ##\n @@ path-walk.h: struct path_walk_info {\n \t */\n \tpath_fn path_fn;\n \tvoid *path_fn_data;\n ++\n +\t/**\n +\t * Initialize which object types the path_fn should be called on. This\n +\t * could also limit the walk to skip blobs if not set.\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n \tTREE:left/:$(git rev-parse topic:left)\n \tTREE:right/:$(git rev-parse topic:right)\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n - \ttest_cmp expect.sorted out.sorted\n + \ttest_cmp_sorted expect out\n '\n\n +test_expect_success 'topic, not base, only blobs' '\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\tblobs:4\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n +# No, this doesn't make a lot of sense for the path-walk API,\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\tblobs:0\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n +test_expect_success 'topic, not base, only trees' '\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\tblobs:0\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n ++\ttest_cmp_sorted expect out\n +'\n +\n test_expect_success 'topic, not base, boundary' '\n4: c321f58c62d ! 5: dfd00b2bf0c path-walk: allow visiting tags\n @@ Metadata\n Author: Derrick Stolee \n\n ## Commit message ##\n - path-walk: allow visiting tags\n + path-walk: visit tags and cached objects\n\n - In anticipation of using the path-walk API to analyze tags or include\n - them in a pack-file, add the ability to walk the tags that were included\n - in the revision walk.\n + The rev_info that is specified for a path-walk traversal may specify\n + visiting tag refs (both lightweight and annotated) and also may specify\n + indexed objects (blobs and trees). Update the path-walk API to walk\n + these objects as well.\n\n - When these tag objects point to blobs or trees, we need to make sure\n - those objects are also visited. Treat tagged trees as root trees, but\n - put the tagged blobs in their own category.\n + When walking tags, we need to peel the annotated objects until reaching\n + a non-tag object. If we reach a commit, then we can add it to the\n + pending objects to make sure we visit in the commit walk portion. If we\n + reach a tree, then we will assume that it is a root tree. If we reach a\n + blob, then we have no good path name and so add it to a new list of\n + \"tagged blobs\".\n\n - Be careful about objects that are referred to by multiple references.\n + When the rev_info includes the \"--indexed-objects\" flag, then the\n + pending set includes blobs and trees found in the cache entries and\n + cache-tree. The cache entries are usually blobs, though they could be\n + trees in the case of a sparse index. The cache-tree stores\n + previously-hashed tree objects but these are cleared out when staging\n + objects below those paths. We add tests that demonstrate this.\n +\n + The indexed objects come with a non-NULL 'path' value in the pending\n + item. This allows us to prepopulate the 'path_to_lists' strmap with\n + lists for these paths.\n +\n + The tricky thing about this walk is that we will want to combine the\n + indexed objects walk with the commit walk, especially in the future case\n + of walking objects during a command like 'git repack'.\n +\n + Whenever possible, we want the objects from the index to be grouped with\n + similar objects in history. We don't want to miss any paths that appear\n + only in the index and not in the commit history.\n +\n + Thus, we need to be careful to let the path stack be populated initially\n + with only the root tree path (and possibly tags and tagged blobs) and go\n + through the normal depth-first search. Afterwards, if there are other\n + paths that are remaining in the paths_to_lists strmap, we should then\n + iterate through the stack and visit those objects recursively.\n\n - Co-authored-by: Johannes Schindelin \n - Signed-off-by: Johannes Schindelin \n Signed-off-by: Derrick Stolee \n\n ## Documentation/technical/api-path-walk.txt ##\n -@@ Documentation/technical/api-path-walk.txt: If you want the path-walk \nAPI to emit `UNINTERESTING` objects based on the\n - commit walk's boundary, be sure to set `revs.boundary` so the boundary\n - commits are emitted.\n +@@ Documentation/technical/api-path-walk.txt: It is also important that \nyou do not specify the `--objects` flag for the\n + the objects will be walked in a separate way based on those starting\n + commits.\n\n -`commits`, `blobs`, `trees`::\n +`commits`, `blobs`, `trees`, `tags`::\n @@ path-walk.c\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n +\n ++static const char *root_path = \"\";\n ++\n + struct type_and_oid_list\n + {\n + \tenum object_type type;\n +@@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n +\n + \tlist = strmap_get(&ctx->paths_to_lists, path);\n +\n ++\tif (!list)\n ++\t\tBUG(\"provided path '%s' that had no associated list\", path);\n ++\n + \t/* Evaluate function pointer on this data, if requested. */\n + \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n +-\t (list->type == OBJ_BLOB && ctx->info->blobs))\n ++\t (list->type == OBJ_BLOB && ctx->info->blobs)||\n ++\t (list->type == OBJ_TAG && ctx->info->tags))\n + \t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n + \t\t\t\t\tctx->info->path_fn_data);\n +\n +@@ path-walk.c: static void clear_strmap(struct strmap *map)\n + \tstrmap_init(map);\n + }\n +\n ++static void setup_pending_objects(struct path_walk_info *info,\n ++\t\t\t\t struct path_walk_context *ctx)\n ++{\n ++\tstruct type_and_oid_list *tags = NULL;\n ++\tstruct type_and_oid_list *tagged_blobs = NULL;\n ++\tstruct type_and_oid_list *root_tree_list = NULL;\n ++\n ++\tif (info->tags)\n ++\t\tCALLOC_ARRAY(tags, 1);\n ++\tif (info->blobs)\n ++\t\tCALLOC_ARRAY(tagged_blobs, 1);\n ++\tif (info->trees)\n ++\t\troot_tree_list = strmap_get(&ctx->paths_to_lists, root_path);\n ++\n ++\t/*\n ++\t * Pending objects include:\n ++\t * * Commits at branch tips.\n ++\t * * Annotated tags at tag tips.\n ++\t * * Any kind of object at lightweight tag tips.\n ++\t * * Trees and blobs in the index (with an associated path).\n ++\t */\n ++\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n ++\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n ++\t\tstruct object *obj = pending->item;\n ++\n ++\t\t/* Commits will be picked up by revision walk. */\n ++\t\tif (obj->type == OBJ_COMMIT)\n ++\t\t\tcontinue;\n ++\n ++\t\t/* Navigate annotated tag object chains. */\n ++\t\twhile (obj->type == OBJ_TAG) {\n ++\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n ++\t\t\tif (!tag)\n ++\t\t\t\tbreak;\n ++\t\t\tif (tag->object.flags & SEEN)\n ++\t\t\t\tbreak;\n ++\t\t\ttag->object.flags |= SEEN;\n ++\n ++\t\t\tif (tags)\n ++\t\t\t\toid_array_append(&tags->oids, &obj->oid);\n ++\t\t\tobj = tag->tagged;\n ++\t\t}\n ++\n ++\t\tif (obj->type == OBJ_TAG)\n ++\t\t\tcontinue;\n ++\n ++\t\t/* We are now at a non-tag object. */\n ++\t\tif (obj->flags & SEEN)\n ++\t\t\tcontinue;\n ++\t\tobj->flags |= SEEN;\n ++\n ++\t\tswitch (obj->type) {\n ++\t\tcase OBJ_TREE:\n ++\t\t\tif (!info->trees)\n ++\t\t\t\tcontinue;\n ++\t\t\tif (pending->path) {\n ++\t\t\t\tstruct type_and_oid_list *list;\n ++\t\t\t\tchar *path = *pending->path ? xstrfmt(\"%s/\", pending->path)\n ++\t\t\t\t\t\t\t : xstrdup(\"\");\n ++\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n ++\t\t\t\t\tCALLOC_ARRAY(list, 1);\n ++\t\t\t\t\tlist->type = OBJ_TREE;\n ++\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n ++\t\t\t\t}\n ++\t\t\t\toid_array_append(&list->oids, &obj->oid);\n ++\t\t\t\tfree(path);\n ++\t\t\t} else {\n ++\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n ++\t\t\t\toid_array_append(&root_tree_list->oids, &obj->oid);\n ++\t\t\t}\n ++\t\t\tbreak;\n ++\n ++\t\tcase OBJ_BLOB:\n ++\t\t\tif (!info->blobs)\n ++\t\t\t\tcontinue;\n ++\t\t\tif (pending->path) {\n ++\t\t\t\tstruct type_and_oid_list *list;\n ++\t\t\t\tchar *path = pending->path;\n ++\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n ++\t\t\t\t\tCALLOC_ARRAY(list, 1);\n ++\t\t\t\t\tlist->type = OBJ_BLOB;\n ++\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n ++\t\t\t\t}\n ++\t\t\t\toid_array_append(&list->oids, &obj->oid);\n ++\t\t\t} else {\n ++\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n ++\t\t\t\toid_array_append(&tagged_blobs->oids, &obj->oid);\n ++\t\t\t}\n ++\t\t\tbreak;\n ++\n ++\t\tcase OBJ_COMMIT:\n ++\t\t\t/* Make sure it is in the object walk */\n ++\t\t\tif (obj != pending->item)\n ++\t\t\t\tadd_pending_object(info->revs, obj, \"\");\n ++\t\t\tbreak;\n ++\n ++\t\tdefault:\n ++\t\t\tBUG(\"should not see any other type here\");\n ++\t\t}\n ++\t}\n ++\n ++\t/*\n ++\t * Add tag objects and tagged blobs if they exist.\n ++\t */\n ++\tif (tagged_blobs) {\n ++\t\tif (tagged_blobs->oids.nr) {\n ++\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n ++\t\t\ttagged_blobs->type = OBJ_BLOB;\n ++\t\t\tpush_to_stack(ctx, tagged_blob_path);\n ++\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n ++\t\t} else {\n ++\t\t\toid_array_clear(&tagged_blobs->oids);\n ++\t\t\tfree(tagged_blobs);\n ++\t\t}\n ++\t}\n ++\tif (tags) {\n ++\t\tif (tags->oids.nr) {\n ++\t\t\tconst char *tag_path = \"/tags\";\n ++\t\t\ttags->type = OBJ_TAG;\n ++\t\t\tpush_to_stack(ctx, tag_path);\n ++\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n ++\t\t} else {\n ++\t\t\toid_array_clear(&tags->oids);\n ++\t\t\tfree(tags);\n ++\t\t}\n ++\t}\n ++}\n ++\n + /**\n + * Given the configuration of 'info', walk the commits based on \n'info->revs' and\n + * call 'info->path_fn' on each discovered path.\n +@@ path-walk.c: static void clear_strmap(struct strmap *map)\n + */\n + int walk_objects_by_path(struct path_walk_info *info)\n + {\n +-\tconst char *root_path = \"\";\n + \tint ret = 0;\n + \tsize_t commits_nr = 0, paths_nr = 0;\n + \tstruct commit *c;\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(commit_list, 1);\n \tcommit_list->type = OBJ_COMMIT;\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n -+\n + \tpush_to_stack(&ctx, root_path);\n +\n +\t/*\n +\t * Set these values before preparing the walk to catch\n -+\t * lightweight tags pointing to non-commits.\n ++\t * lightweight tags pointing to non-commits and indexed objects.\n +\t */\n +\tinfo->revs->blob_objects = info->blobs;\n +\tinfo->revs->tree_objects = info->trees;\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n\n +\tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n +\n -+\tif (info->tags) {\n -+\t\tstruct oid_array tagged_blob_list = OID_ARRAY_INIT;\n -+\t\tstruct oid_array tags = OID_ARRAY_INIT;\n -+\n -+\t\ttrace2_region_enter(\"path-walk\", \"tag-walk\", info->revs->repo);\n -+\n -+\t\t/*\n -+\t\t * Walk any pending objects at this point, but they should only\n -+\t\t * be tags.\n -+\t\t */\n -+\t\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n -+\t\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n -+\t\t\tstruct object *obj = pending->item;\n -+\n -+\t\t\tif (obj->type == OBJ_COMMIT || obj->flags & SEEN)\n -+\t\t\t\tcontinue;\n -+\n -+\t\t\twhile (obj->type == OBJ_TAG) {\n -+\t\t\t\tstruct tag *tag = lookup_tag(info->revs->repo,\n -+\t\t\t\t\t\t\t &obj->oid);\n -+\t\t\t\tif (!(obj->flags & SEEN)) {\n -+\t\t\t\t\tobj->flags |= SEEN;\n -+\t\t\t\t\toid_array_append(&tags, &obj->oid);\n -+\t\t\t\t}\n -+\t\t\t\tobj = tag->tagged;\n -+\t\t\t}\n -+\n -+\t\t\tif ((obj->flags & SEEN))\n -+\t\t\t\tcontinue;\n -+\t\t\tobj->flags |= SEEN;\n ++\ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n ++\tsetup_pending_objects(info, &ctx);\n ++\ttrace2_region_leave(\"path-walk\", \"pending-walk\", info->revs->repo);\n +\n -+\t\t\tswitch (obj->type) {\n -+\t\t\tcase OBJ_TREE:\n -+\t\t\t\tif (info->trees)\n -+\t\t\t\t\toid_array_append(&root_tree_list->oids, &obj->oid);\n -+\t\t\t\tbreak;\n -+\n -+\t\t\tcase OBJ_BLOB:\n -+\t\t\t\tif (info->blobs)\n -+\t\t\t\t\toid_array_append(&tagged_blob_list, &obj->oid);\n -+\t\t\t\tbreak;\n + \twhile ((c = get_revision(info->revs))) {\n + \t\tstruct object_id *oid;\n + \t\tstruct tree *t;\n +@@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n +\n + \t\tfree(path);\n + \t}\n +\n -+\t\t\tcase OBJ_COMMIT:\n -+\t\t\t\t/* Make sure it is in the object walk */\n -+\t\t\t\tadd_pending_object(info->revs, obj, \"\");\n -+\t\t\t\tbreak;\n ++\t/* Are there paths remaining? Likely they are from indexed objects. */\n ++\tif (!strmap_empty(&ctx.paths_to_lists)) {\n ++\t\tstruct hashmap_iter iter;\n ++\t\tstruct strmap_entry *entry;\n +\n -+\t\t\tdefault:\n -+\t\t\t\tBUG(\"should not see any other type here\");\n -+\t\t\t}\n ++\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry) {\n ++\t\t\tpush_to_stack(&ctx, entry->key);\n +\t\t}\n +\n -+\t\tinfo->path_fn(\"\", &tags, OBJ_TAG, info->path_fn_data);\n ++\t\twhile (!ret && ctx.path_stack.nr) {\n ++\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n ++\t\t\tctx.path_stack.nr--;\n ++\t\t\tpaths_nr++;\n +\n -+\t\tif (tagged_blob_list.nr && info->blobs)\n -+\t\t\tinfo->path_fn(\"/tagged-blobs\", &tagged_blob_list, OBJ_BLOB,\n -+\t\t\t\t info->path_fn_data);\n ++\t\t\tret = walk_path(&ctx, path);\n +\n -+\t\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"tags\", tags.nr);\n -+\t\ttrace2_region_leave(\"path-walk\", \"tag-walk\", info->revs->repo);\n -+\t\toid_array_clear(&tags);\n -+\t\toid_array_clear(&tagged_blob_list);\n ++\t\t\tfree(path);\n ++\t\t}\n +\t}\n +\n - \twhile ((c = get_revision(info->revs))) {\n - \t\tstruct object_id *oid;\n - \t\tstruct tree *t;\n + \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n + \ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n +\n\n ## path-walk.h ##\n @@ path-walk.h: struct path_walk_info {\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n +\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n +\tBLOB:child/file:$(git rev-parse refs/tags/tree-tag^{}:child/file)\n +\tblobs:10\n -+\tTAG::$(git rev-parse refs/tags/first)\n -+\tTAG::$(git rev-parse refs/tags/second.1)\n -+\tTAG::$(git rev-parse refs/tags/second.2)\n -+\tTAG::$(git rev-parse refs/tags/third)\n -+\tTAG::$(git rev-parse refs/tags/fourth)\n -+\tTAG::$(git rev-parse refs/tags/tree-tag)\n -+\tTAG::$(git rev-parse refs/tags/blob-tag)\n ++\tTAG:/tags:$(git rev-parse refs/tags/first)\n ++\tTAG:/tags:$(git rev-parse refs/tags/second.1)\n ++\tTAG:/tags:$(git rev-parse refs/tags/second.2)\n ++\tTAG:/tags:$(git rev-parse refs/tags/third)\n ++\tTAG:/tags:$(git rev-parse refs/tags/fourth)\n ++\tTAG:/tags:$(git rev-parse refs/tags/tree-tag)\n ++\tTAG:/tags:$(git rev-parse refs/tags/blob-tag)\n +\ttags:7\n ++\tEOF\n ++\n ++\ttest_cmp_sorted expect out\n ++'\n ++\n ++test_expect_success 'indexed objects' '\n ++\ttest_when_finished git reset --hard &&\n ++\n ++\t# stage change into index, adding a blob but\n ++\t# also invalidating the cache-tree for the root\n ++\t# and the \"left\" directory.\n ++\techo bogus >left/c &&\n ++\tgit add left &&\n ++\n ++\ttest-tool path-walk -- --indexed-objects >out &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\tcommits:0\n ++\tTREE:right/:$(git rev-parse topic:right)\n ++\ttrees:1\n ++\tBLOB:a:$(git rev-parse HEAD:a)\n ++\tBLOB:left/b:$(git rev-parse HEAD:left/b)\n ++\tBLOB:left/c:$(git rev-parse :left/c)\n ++\tBLOB:right/c:$(git rev-parse HEAD:right/c)\n ++\tBLOB:right/d:$(git rev-parse HEAD:right/d)\n ++\tblobs:5\n ++\ttags:0\n ++\tEOF\n ++\n ++\ttest_cmp_sorted expect out\n ++'\n ++\n ++test_expect_success 'branches and indexed objects mix well' '\n ++\ttest_when_finished git reset --hard &&\n ++\n ++\t# stage change into index, adding a blob but\n ++\t# also invalidating the cache-tree for the root\n ++\t# and the \"right\" directory.\n ++\techo fake >right/d &&\n ++\tgit add right &&\n ++\n ++\ttest-tool path-walk -- --indexed-objects --branches >out &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\tCOMMIT::$(git rev-parse topic)\n ++\tCOMMIT::$(git rev-parse base)\n ++\tCOMMIT::$(git rev-parse base~1)\n ++\tCOMMIT::$(git rev-parse base~2)\n ++\tcommits:4\n ++\tTREE::$(git rev-parse topic^{tree})\n ++\tTREE::$(git rev-parse base^{tree})\n ++\tTREE::$(git rev-parse base~1^{tree})\n ++\tTREE::$(git rev-parse base~2^{tree})\n ++\tTREE:a/:$(git rev-parse base:a)\n ++\tTREE:left/:$(git rev-parse base:left)\n ++\tTREE:left/:$(git rev-parse base~2:left)\n ++\tTREE:right/:$(git rev-parse topic:right)\n ++\tTREE:right/:$(git rev-parse base~1:right)\n ++\tTREE:right/:$(git rev-parse base~2:right)\n ++\ttrees:10\n ++\tBLOB:a:$(git rev-parse base~2:a)\n ++\tBLOB:left/b:$(git rev-parse base:left/b)\n ++\tBLOB:left/b:$(git rev-parse base~2:left/b)\n ++\tBLOB:right/c:$(git rev-parse base~2:right/c)\n ++\tBLOB:right/c:$(git rev-parse topic:right/c)\n ++\tBLOB:right/d:$(git rev-parse base~1:right/d)\n ++\tBLOB:right/d:$(git rev-parse :right/d)\n ++\tblobs:7\n ++\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n + \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse base~1:right/d)\n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n +\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n + \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse topic:right/d)\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n + \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse topic:right/d)\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n +\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n + \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only commits' '\n \tcommits:1\n \ttrees:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only commits' '\n +\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n + \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n \tTREE:right/:$(git rev-parse topic:right)\n \ttrees:3\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n +\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n + \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \tBLOB:right/c:$(git rev-parse topic:right/c)\n \tBLOB:right/d:$(git rev-parse base~1:right/d)\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n +\ttags:0\n \tEOF\n\n - \tsort expect >expect.sorted &&\n -@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n - \ttest_cmp expect.sorted out.sorted\n + \ttest_cmp_sorted expect out\n '\n\n +test_expect_success 'trees are reported exactly once' '\n6: 238d7d95715 ! 6: 5252076d556 path-walk: add prune_all_uninteresting option\n @@ Metadata\n Author: Derrick Stolee \n\n ## Commit message ##\n - path-walk: add prune_all_uninteresting option\n + path-walk: mark trees and blobs as UNINTERESTING\n\n - This option causes the path-walk API to act like the sparse tree-walk\n - algorithm implemented by mark_trees_uninteresting_sparse() in\n - list-objects.c.\n + When the input rev_info has UNINTERESTING starting points, we want to be\n + sure that the UNINTERESTING flag is passed appropriately through the\n + objects. To match how this is done in places such as 'git \npack-objects', we\n + use the mark_edges_uninteresting() method.\n\n - Starting from the commits marked as UNINTERESTING, their root trees and\n - all objects reachable from those trees are UNINTERSTING, at least as we\n - walk path-by-path. When we reach a path where all objects associated\n - with that path are marked UNINTERESTING, then do no continue walking the\n - children of that path.\n + This method has an option for using the \"sparse\" walk, which is similar in\n + spirit to the path-walk API's walk. To be sure to keep it independent, \nadd a\n + new 'prune_all_uninteresting' option to the path_walk_info struct.\n\n - We need to be careful to pass the UNINTERESTING flag in a deep way on\n - the UNINTERESTING objects before we start the path-walk, or else the\n - depth-first search for the path-walk API may accidentally report some\n - objects as interesting.\n + To check how the UNINTERSTING flag is spread through our objects, \nextend the\n + 'test-tool path-walk' command to output whether or not an object has that\n + flag. This changes our tests significantly, including the removal of some\n + objects that were previously visited due to the incomplete implementation.\n\n Signed-off-by: Derrick Stolee \n\n ## Documentation/technical/api-path-walk.txt ##\n -@@ Documentation/technical/api-path-walk.txt: commits are emitted.\n +@@ Documentation/technical/api-path-walk.txt: commits.\n While it is possible to walk only commits in this way, consumers would be\n better off using the revision walk API instead.\n\n @@ Documentation/technical/api-path-walk.txt: commits are emitted.\n\n\n ## path-walk.c ##\n +@@\n + #include \"dir.h\"\n + #include \"hashmap.h\"\n + #include \"hex.h\"\n ++#include \"list-objects.h\"\n + #include \"object.h\"\n + #include \"oid-array.h\"\n + #include \"revision.h\"\n @@ path-walk.c: struct type_and_oid_list\n {\n \tenum object_type type;\n @@ path-walk.c: struct type_and_oid_list\n\n #define TYPE_AND_OID_LIST_INIT { \\\n @@ path-walk.c: static int add_children(struct path_walk_context *ctx,\n - \t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n - \t\t\tstring_list_append(&ctx->path_stack, path.buf);\n - \t\t}\n + \t\tif (o->flags & SEEN)\n + \t\t\tcontinue;\n + \t\to->flags |= SEEN;\n ++\n +\t\tif (!(o->flags & UNINTERESTING))\n +\t\t\tlist->maybe_interesting = 1;\n \t\toid_array_append(&list->oids, &entry.oid);\n \t}\n\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n -\n - \tlist = strmap_get(&ctx->paths_to_lists, path);\n + \tif (!list)\n + \t\tBUG(\"provided path '%s' that had no associated list\", path);\n\n +\tif (ctx->info->prune_all_uninteresting) {\n +\t\t/*\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n +\t\t\t\t\t\t\t &list->oids.oid[i]);\n +\t\t\t\tif (t && !(t->object.flags & UNINTERESTING))\n +\t\t\t\t\tlist->maybe_interesting = 1;\n -+\t\t\t} else {\n ++\t\t\t} else if (list->type == OBJ_BLOB) {\n +\t\t\t\tstruct blob *b = lookup_blob(ctx->repo,\n +\t\t\t\t\t\t\t &list->oids.oid[i]);\n +\t\t\t\tif (b && !(b->object.flags & UNINTERESTING))\n +\t\t\t\t\tlist->maybe_interesting = 1;\n ++\t\t\t} else {\n ++\t\t\t\t/* Tags are always interesting if visited. */\n ++\t\t\t\tlist->maybe_interesting = 1;\n +\t\t\t}\n +\t\t}\n +\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n +\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n - \t (list->type == OBJ_BLOB && ctx->info->blobs))\n + \t (list->type == OBJ_BLOB && ctx->info->blobs)||\n @@ path-walk.c: static void clear_strmap(struct strmap *map)\n - int walk_objects_by_path(struct path_walk_info *info)\n - {\n - \tconst char *root_path = \"\";\n --\tint ret = 0;\n -+\tint ret = 0, has_uninteresting = 0;\n - \tsize_t commits_nr = 0, paths_nr = 0;\n - \tstruct commit *c;\n - \tstruct type_and_oid_list *root_tree_list;\n -@@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n - \t\t.path_stack = STRING_LIST_INIT_DUP,\n - \t\t.paths_to_lists = STRMAP_INIT\n - \t};\n -+\tstruct oidset root_tree_set = OIDSET_INIT;\n -\n - \ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n + \tstrmap_init(map);\n + }\n\n ++static struct repository *edge_repo;\n ++static struct type_and_oid_list *edge_tree_list;\n ++\n ++static void show_edge(struct commit *commit)\n ++{\n ++\tstruct tree *t = repo_get_commit_tree(edge_repo, commit);\n ++\n ++\tif (!t)\n ++\t\treturn;\n ++\n ++\tif (commit->object.flags & UNINTERESTING)\n ++\t\tt->object.flags |= UNINTERESTING;\n ++\n ++\tif (t->object.flags & SEEN)\n ++\t\treturn;\n ++\tt->object.flags |= SEEN;\n ++\n ++\toid_array_append(&edge_tree_list->oids, &t->object.oid);\n ++}\n ++\n + static void setup_pending_objects(struct path_walk_info *info,\n + \t\t\t\t struct path_walk_context *ctx)\n + {\n +@@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n + \t\tif (tagged_blobs->oids.nr) {\n + \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n + \t\t\ttagged_blobs->type = OBJ_BLOB;\n ++\t\t\ttagged_blobs->maybe_interesting = 1;\n + \t\t\tpush_to_stack(ctx, tagged_blob_path);\n + \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n + \t\t} else {\n +@@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n + \t\tif (tags->oids.nr) {\n + \t\t\tconst char *tag_path = \"/tags\";\n + \t\t\ttags->type = OBJ_TAG;\n ++\t\t\ttags->maybe_interesting = 1;\n + \t\t\tpush_to_stack(ctx, tag_path);\n + \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n + \t\t} else {\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n +\troot_tree_list->maybe_interesting = 1;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n + \tpush_to_stack(&ctx, root_path);\n\n - \t/*\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n - \t\tt = lookup_tree(info->revs->repo, oid);\n + \tif (prepare_revision_walk(info->revs))\n + \t\tdie(_(\"failed to setup revision walk\"));\n\n - \t\tif (t) {\n -+\t\t\tif ((c->object.flags & UNINTERESTING)) {\n -+\t\t\t\tt->object.flags |= UNINTERESTING;\n -+\t\t\t\thas_uninteresting = 1;\n -+\t\t\t}\n ++\t/* Walk trees to mark them as UNINTERESTING. */\n ++\tedge_repo = info->revs->repo;\n ++\tedge_tree_list = root_tree_list;\n ++\tmark_edges_uninteresting(info->revs, show_edge,\n ++\t\t\t\t info->prune_all_uninteresting);\n ++\tedge_repo = NULL;\n ++\tedge_tree_list = NULL;\n +\n - \t\t\tif (t->object.flags & SEEN)\n - \t\t\t\tcontinue;\n - \t\t\tt->object.flags |= SEEN;\n --\t\t\toid_array_append(&root_tree_list->oids, oid);\n -+\t\t\tif (!oidset_insert(&root_tree_set, oid))\n -+\t\t\t\toid_array_append(&root_tree_list->oids, oid);\n - \t\t} else {\n - \t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n - \t\t}\n -@@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n - \toid_array_clear(&commit_list->oids);\n - \tfree(commit_list);\n + \tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n\n -+\t/*\n -+\t * Before performing a DFS of our paths and emitting them as interesting,\n -+\t * do a full walk of the trees to distribute the UNINTERESTING bit. Use\n -+\t * the sparse algorithm if prune_all_uninteresting was set.\n -+\t */\n -+\tif (has_uninteresting) {\n -+\t\ttrace2_region_enter(\"path-walk\", \"uninteresting-walk\", info->revs->repo);\n -+\t\tif (info->prune_all_uninteresting)\n -+\t\t\tmark_trees_uninteresting_sparse(ctx.repo, &root_tree_set);\n -+\t\telse\n -+\t\t\tmark_trees_uninteresting_dense(ctx.repo, &root_tree_set);\n -+\t\ttrace2_region_leave(\"path-walk\", \"uninteresting-walk\", info->revs->repo);\n -+\t}\n -+\toidset_clear(&root_tree_set);\n -+\n - \tstring_list_append(&ctx.path_stack, root_path);\n -\n - \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n + \ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n\n ## path-walk.h ##\n @@ path-walk.h: struct path_walk_info {\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n\n\n ## t/t6601-path-walk.sh ##\n +@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n + \tCOMMIT::$(git rev-parse topic)\n + \tcommits:1\n + \tTREE::$(git rev-parse topic^{tree})\n +-\tTREE:left/:$(git rev-parse topic:left)\n ++\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n + \tTREE:right/:$(git rev-parse topic:right)\n + \ttrees:3\n +-\tBLOB:a:$(git rev-parse topic:a)\n +-\tBLOB:left/b:$(git rev-parse topic:left/b)\n ++\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n ++\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n + \tBLOB:right/c:$(git rev-parse topic:right/c)\n +-\tBLOB:right/d:$(git rev-parse topic:right/d)\n ++\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n + \tblobs:4\n + \ttags:0\n + \tEOF\n +@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n + \ttest_cmp_sorted expect out\n + '\n +\n ++test_expect_success 'fourth, blob-tag2, not base' '\n ++\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\tCOMMIT::$(git rev-parse topic)\n ++\tcommits:1\n ++\tTREE::$(git rev-parse topic^{tree})\n ++\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n ++\tTREE:right/:$(git rev-parse topic:right)\n ++\ttrees:3\n ++\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n ++\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n ++\tBLOB:right/c:$(git rev-parse topic:right/c)\n ++\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n ++\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n ++\tblobs:5\n ++\tTAG:/tags:$(git rev-parse fourth)\n ++\ttags:1\n ++\tEOF\n ++\n ++\ttest_cmp_sorted expect out\n ++'\n ++\n + test_expect_success 'topic, not base, only blobs' '\n + \ttest-tool path-walk --no-trees --no-commits \\\n + \t\t-- topic --not base >out &&\n +@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n + \tcat >expect <<-EOF &&\n + \tcommits:0\n + \ttrees:0\n +-\tBLOB:a:$(git rev-parse topic:a)\n +-\tBLOB:left/b:$(git rev-parse topic:left/b)\n ++\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n ++\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n + \tBLOB:right/c:$(git rev-parse topic:right/c)\n +-\tBLOB:right/d:$(git rev-parse topic:right/d)\n ++\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n + \tblobs:4\n + \ttags:0\n + \tEOF\n +@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n + \tcat >expect <<-EOF &&\n + \tcommits:0\n + \tTREE::$(git rev-parse topic^{tree})\n +-\tTREE:left/:$(git rev-parse topic:left)\n ++\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n + \tTREE:right/:$(git rev-parse topic:right)\n + \ttrees:3\n + \tblobs:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n\n \tcat >expect <<-EOF &&\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \ttags:0\n \tEOF\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n - \ttest_cmp expect.sorted out.sorted\n + \ttest_cmp_sorted expect out\n '\n\n +-test_expect_success 'trees are reported exactly once' '\n +-\ttest_when_finished \"rm -rf unique-trees\" &&\n +-\ttest_create_repo unique-trees &&\n +-\t(\n +-\t\tcd unique-trees &&\n +-\t\tmkdir initial &&\n +-\t\ttest_commit initial/file &&\n +-\n +-\t\tgit switch -c move-to-top &&\n +-\t\tgit mv initial/file.t ./ &&\n +-\t\ttest_tick &&\n +-\t\tgit commit -m moved &&\n +-\n +-\t\tgit update-ref refs/heads/other HEAD\n +-\t) &&\n +-\n +-\ttest-tool -C unique-trees path-walk -- --all >out &&\n +-\ttree=$(git -C unique-trees rev-parse HEAD:) &&\n +-\tgrep \"$tree\" out >out-filtered &&\n +-\ttest_line_count = 1 out-filtered\n +test_expect_success 'topic, not base, boundary with pruning' '\n +\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n +\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n +\ttags:0\n +\tEOF\n +\n -+\tsort expect >expect.sorted &&\n -+\tsort out >out.sorted &&\n -+\n -+\ttest_cmp expect.sorted out.sorted\n -+'\n -+\n - test_expect_success 'trees are reported exactly once' '\n - \ttest_when_finished \"rm -rf unique-trees\" &&\n - \ttest_create_repo unique-trees &&\n ++\ttest_cmp_sorted expect out\n + '\n +\n + test_done\n\n"},{"id":"506442","messageId":"CAOLa=ZQnEL7i=QbqjHYjpjWv_uhG=u+1w1FUKTMgkfNYd3fefg@mail.gmail.com","threadId":"62429","inReplyTo":"c71f0a0e3613555de83668473ce97db86aad659f.1730356023.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/6] path-walk: introduce an object walk by path","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-01T13:12:16Z","receivedAt":"2024-11-01T13:12:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> In anticipation of a few planned applications, introduce the most basic form\n> of a path-walk API. It currently assumes that there are no UNINTERESTING\n> objects, and does not include any complicated filters. It calls a function\n> pointer on groups of tree and blob objects as grouped by path. This only\n> includes objects the first time they are discovered, so an object that\n> appears at multiple paths will not be included in two batches.\n>\n> These batches are collected in 'struct type_and_oid_list' objects, which\n> store an object type and an oid_array of objects.\n>\n> The data structures are documented in 'struct path_walk_context', but in\n> summary the most important are:\n>\n> * 'paths_to_lists' is a strmap that connects a path to a\n> type_and_oid_list for that path. To avoid conflicts in path names,\n> we make sure that tree paths end in \"/\" (except the root path with\n> is an empty string) and blob paths do not end in \"/\".\n>\n> * 'path_stack' is a string list that is added to in an append-only\n> way. This stores the stack of our depth-first search on the heap\n> instead of using recursion.\n>\n> * 'path_stack_pushed' is a strmap that stores path names that were\n> already added to 'path_stack', to avoid repeating paths in the\n> stack. Mostly, this saves us from quadratic lookups from doing\n> unsorted checks into the string_list.\n>\n> The coupling of 'path_stack' and 'path_stack_pushed' is protected by the\n> push_to_stack() method. Call this instead of inserting into these\n> structures directly.\n>\n> The walk_objects_by_path() method initializes these structures and\n> starts walking commits from the given rev_info struct. The commits are\n> used to find the list of root trees which populate the start of our\n> depth-first search.\n\nIsn't this more of breadth-first search? Reading through the code, the\nalgorithm seems something like:\n\n- For each commit in list of commits (from rev_info)\n - Tackle each root tree, add root path to the stack.\n- For each path in stack left\n - Call the callback provided by client.\n - Find all its first level children, add each to the stack.\n\nSo wouldn't this go through the tree in level by level basis? Making it\na BFS?\n\nApart from this, the patch itself looks solid. I ended up writing a\nsmall client to play with this API, and was very pleased how quickly I\ncould get it running.\n\n[snip]\n"},{"id":"506444","messageId":"CAOLa=ZRJd7CXNFi1V=4UN_CuNFHqSsOGE5S0AnVeBMvPJJy72w@mail.gmail.com","threadId":"62429","inReplyTo":"draft-87r07v14kl.fsf@archlinux.mail-host-address-is-not-set","subject":"Re: [PATCH 1/6] path-walk: introduce an object walk by path","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-01T13:42:11Z","receivedAt":"2024-11-01T13:42:17Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"karthik nayak writes:\n\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n>> From: Derrick Stolee \n>>\n>> In anticipation of a few planned applications, introduce the most basic form\n>> of a path-walk API. It currently assumes that there are no UNINTERESTING\n>> objects, and does not include any complicated filters. It calls a function\n>> pointer on groups of tree and blob objects as grouped by path. This only\n>> includes objects the first time they are discovered, so an object that\n>> appears at multiple paths will not be included in two batches.\n>>\n>> These batches are collected in 'struct type_and_oid_list' objects, which\n>> store an object type and an oid_array of objects.\n>>\n>> The data structures are documented in 'struct path_walk_context', but in\n>> summary the most important are:\n>>\n>> * 'paths_to_lists' is a strmap that connects a path to a\n>> type_and_oid_list for that path. To avoid conflicts in path names,\n>> we make sure that tree paths end in \"/\" (except the root path with\n>> is an empty string) and blob paths do not end in \"/\".\n>>\n>> * 'path_stack' is a string list that is added to in an append-only\n>> way. This stores the stack of our depth-first search on the heap\n>> instead of using recursion.\n>>\n>> * 'path_stack_pushed' is a strmap that stores path names that were\n>> already added to 'path_stack', to avoid repeating paths in the\n>> stack. Mostly, this saves us from quadratic lookups from doing\n>> unsorted checks into the string_list.\n>>\n>> The coupling of 'path_stack' and 'path_stack_pushed' is protected by the\n>> push_to_stack() method. Call this instead of inserting into these\n>> structures directly.\n>>\n>> The walk_objects_by_path() method initializes these structures and\n>> starts walking commits from the given rev_info struct. The commits are\n>> used to find the list of root trees which populate the start of our\n>> depth-first search.\n>\n> Isn't this more of breadth-first search? Reading through the code, the\n> algorithm seems something like:\n>\n> - For each commit in list of commits (from rev_info)\n> - Tackle each root tree, add root path to the stack.\n> - For each path in stack left\n> - Call the callback provided by client.\n> - Find all its first level children, add each to the stack.\n>\n> So wouldn't this go through the tree in level by level basis? Making it\n> a BFS?\n\nMy bad here, thinking more about it, it is DFS indeed. Although we add\nall the children of a level to the stack, we pop each of them from the\nstack and end up traversing down that level.\n\n>\n> Apart from this, the patch itself looks solid. I ended up writing a\n> small client to play with this API, and was very pleased how quickly I\n> could get it running.\n>\n> [snip]\n"},{"id":"506446","messageId":"541b89c7-5e4b-4251-a8e7-3a9cde381064@gmail.com","threadId":"62429","inReplyTo":"CAOLa=ZQnEL7i=QbqjHYjpjWv_uhG=u+1w1FUKTMgkfNYd3fefg@mail.gmail.com","subject":"Re: [PATCH 1/6] path-walk: introduce an object walk by path","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-01T13:44:12Z","receivedAt":"2024-11-01T13:44:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/24 9:12 AM, karthik nayak wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> The walk_objects_by_path() method initializes these structures and\n>> starts walking commits from the given rev_info struct. The commits are\n>> used to find the list of root trees which populate the start of our\n>> depth-first search.\n> \n> Isn't this more of breadth-first search? Reading through the code, the\n> algorithm seems something like:\n> \n> - For each commit in list of commits (from rev_info)\n> - Tackle each root tree, add root path to the stack.\n> - For each path in stack left\n> - Call the callback provided by client.\n> - Find all its first level children, add each to the stack.\n> \n> So wouldn't this go through the tree in level by level basis? Making it\n> a BFS?\n\nWhile we are adding all children to the stack, we only pop off the top\nof the stack, making it a DFS. (We do visit the paths in reverse-\nlexicographic order, though.)\n\nTo make it a BFS, we would need to visit the paths in the order they\nare added to the list. Instead, we visit them in Last-In First-Out\norder.\n\nI initially had built it as a BFS, but ran into memory issues when\nrunning it on very large repos.\n\nThanks,\n-Stolee\n\n"},{"id":"506447","messageId":"CAOLa=ZQfcroHv_V7YOD0MeTSvbedJvLNMR2RVQ-dWRe8ERpzdw@mail.gmail.com","threadId":"62429","inReplyTo":"6f93dff88e7cf81800c8f2c8d39175d962a0576f.1730356023.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-01T13:46:11Z","receivedAt":"2024-11-01T13:46:14Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n[snip]\n\n> diff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\n> new file mode 100755\n> index 00000000000..1f277b88291\n> --- /dev/null\n> +++ b/t/t6601-path-walk.sh\n> @@ -0,0 +1,118 @@\n> +#!/bin/sh\n> +\n> +test_description='direct path-walk API tests'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup test repository' '\n> +\tgit checkout -b base &&\n> +\n> +\tmkdir left &&\n> +\tmkdir right &&\n> +\techo a >a &&\n> +\techo b >left/b &&\n> +\techo c >right/c &&\n> +\tgit add . &&\n> +\tgit commit -m \"first\" &&\n> +\n> +\techo d >right/d &&\n> +\tgit add right &&\n> +\tgit commit -m \"second\" &&\n> +\n> +\techo bb >left/b &&\n> +\tgit commit -a -m \"third\" &&\n> +\n> +\tgit checkout -b topic HEAD~1 &&\n> +\techo cc >right/c &&\n> +\tgit commit -a -m \"topic\"\n> +'\n> +\n\nNit: Since the root level tree is already special cased out, we only\ncheck one level of path here, would be nice to add another level of tree\nto this.\n\n> +test_expect_success 'all' '\n> +\ttest-tool path-walk -- --all >out &&\n> +\n> +\tcat >expect <<-EOF &&\n> +\tTREE::$(git rev-parse topic^{tree})\n> +\tTREE::$(git rev-parse base^{tree})\n> +\tTREE::$(git rev-parse base~1^{tree})\n> +\tTREE::$(git rev-parse base~2^{tree})\n> +\tTREE:left/:$(git rev-parse base:left)\n> +\tTREE:left/:$(git rev-parse base~2:left)\n> +\tTREE:right/:$(git rev-parse topic:right)\n> +\tTREE:right/:$(git rev-parse base~1:right)\n> +\tTREE:right/:$(git rev-parse base~2:right)\n> +\ttrees:9\n> +\tBLOB:a:$(git rev-parse base~2:a)\n> +\tBLOB:left/b:$(git rev-parse base~2:left/b)\n> +\tBLOB:left/b:$(git rev-parse base:left/b)\n> +\tBLOB:right/c:$(git rev-parse base~2:right/c)\n> +\tBLOB:right/c:$(git rev-parse topic:right/c)\n> +\tBLOB:right/d:$(git rev-parse base~1:right/d)\n> +\tblobs:6\n> +\tEOF\n> +\n> +\ttest_cmp_sorted expect out\n> +'\n\nIsn't the order deterministic? Why do we need to sort it?\n"},{"id":"506448","messageId":"CAOLa=ZR1BjKzujxpA2icAfJHin-krqwkxTf5VPeGg14AVr-TcQ@mail.gmail.com","threadId":"62429","inReplyTo":"3dc27658526784edf46f67661d1161a8eea3ea0a.1730356023.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] path-walk: visit tags and cached objects","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-01T14:25:38Z","receivedAt":"2024-11-01T14:25:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The rev_info that is specified for a path-walk traversal may specify\n> visiting tag refs (both lightweight and annotated) and also may specify\n> indexed objects (blobs and trees). Update the path-walk API to walk\n> these objects as well.\n>\n> When walking tags, we need to peel the annotated objects until reaching\n> a non-tag object. If we reach a commit, then we can add it to the\n> pending objects to make sure we visit in the commit walk portion. If we\n\nNit: s/in/it in/\n\n[snip]\n\n> +\t\tcase OBJ_BLOB:\n> +\t\t\tif (!info->blobs)\n> +\t\t\t\tcontinue;\n> +\t\t\tif (pending->path) {\n> +\t\t\t\tstruct type_and_oid_list *list;\n> +\t\t\t\tchar *path = pending->path;\n> +\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n> +\t\t\t\t\tCALLOC_ARRAY(list, 1);\n> +\t\t\t\t\tlist->type = OBJ_BLOB;\n> +\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n> +\t\t\t\t}\n> +\t\t\t\toid_array_append(&list->oids, &obj->oid);\n> +\t\t\t} else {\n> +\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n\nShouldn't this comment be for tagged blobs?\n\n> +\t\t\t\toid_array_append(&tagged_blobs->oids, &obj->oid);\n> +\t\t\t}\n> +\t\t\tbreak;\n\n[snip]\n"},{"id":"506464","messageId":"ZyUqr/wb5K4Og9j9@nand.local","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-01T19:23:27Z","receivedAt":"2024-11-01T19:23:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Stolee,\n\nOn Thu, Oct 31, 2024 at 06:26:57AM +0000, Derrick Stolee via GitGitGadget wrote:\n>\n> Introduction and relation to prior series\n> =========================================\n>\n> This is a new series that rerolls the initial \"path-walk API\" patches of my\n> RFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\n> path-walk.h) presents a new way to walk objects such that trees and blobs\n> are walked in batches according to their path.\n>\n> This also replaces the previous version of ds/path-walk that was being\n> reviewed in [2]. The consensus was that the series was too long/dense and\n> could use some reduction in size. This series takes the first few patches,\n> but also makes some updates (which will be described later).\n>\n> [1]\n> https://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n> [RFC] Path-walk API and applications\n>\n> [2]\n> https://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n> [PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n\nI apologize for not having a better place to start discussing a topic\nwhich pertains to more than just this immediate patch series, but I\nfigure here is as good a place as any to do so.\n\nFrom our earlier discussion, it seems to stand that the path-walk API\nis fundamentally incompatible with reachability bitmaps and\ndelta-islands, making the series a non-starter in environments that\nrely significantly one or both of those features. My understanding as a\nresult is that the path-walk API and feature are more targeted towards\nimproving client-side repacks and push performance, where neither of the\naforementioned two features are used quite as commonly.\n\nI was discussing this a bit off-list with Peff (who I hope will join the\nthread and share his own thoughts), but I wonder if it was a mistake to\ndiscard your '--full-name-hash' idea (or something similar, which I'll\ndiscuss in a bit below) from earlier.\n\n(Repeating a few things that I am sure are obvious to you out loud so\nthat I can get a grasp on them for my own understanding):\n\nIt seems that the problems you've identified which result in poor repack\nperformance occur when you have files at the same path, but they get\npoorly sorted in the delta selection window due to other paths having\nthe same final 16 characters, so Git doesn't see that much better delta\nopportunities exist.\n\nYour series takes into account the full name when hashing, which seems\nto produce a clear win in many cases. I'm sure that there are some cases\nwhere it presents a modest regression in pack sizes, but I think that's\nfine and probably par for the course when making any changes like this,\nas there is probably no easy silver bullet here that uniformly improves\nall cases.\n\nI suspect that you could go even further and intern the full path at\nwhich each object occurs, and sort lexically by that. Just stringing\ntogether all of the paths in linux.git only takes 3.099 MiB on my clone.\n(Of course, that's unbounded in the number of objects and length of\ntheir pathnames, but you could at least bound the latter by taking only\nthe last, say, 128 characters, which would be more than good enough for\nthe kernel, whose longest path is only 102 characters).\n\nSome of the repositories that you've tested on I don't have easy access\nto, so I wonder if either doing (a) that, or (b) using some fancier\ncontext-sensitive hash (like SimHash or MinHash) would be beneficial.\n\nI realize that this is taking us back to an idea you've already\npresented to the list, but I think (to me, at least) the benefit and\nsimplicity of that approach has only become clear to me in hindsight\nwhen seeing some alternatives. I would like to apologize for the time\nyou spent reworking this series back and forth to have the response be\n\"maybe we should have just done the first thing you suggested\". Like I\nsaid, I think to me it was really only clear in hindsight.\n\nIn any event, the major benefit to doing --full-name-hash would be that\n*all* environments could benefit from the size reduction, not just those\nthat don't rely on certain other features.\n\nPerhaps just --full-name-hash isn't quite as good by itself as the\n--path-walk implementation that this series starts us off implementing.\nSo in that sense, maybe we want both, which I understand was the\noriginal approach. I see a couple of options here:\n\n - We take both, because doing --path-walk on top represents a\n significant enough improvement that we are collectively OK with\n taking on more code to improve a more narrow (but common) use-case.\n\n - Or we decide that either the benefit isn't significant enough to\n warrant an additional and relatively complex implementation, or in\n other words that --full-name-hash by itself is good enough.\n\nAgain, I apologize for not having a clearer picture of this all to start\nwith, and I want to tell you specifically and sincerely that I\nappreciate your patience as I wrap my head around all of this. I think\nthe benefit of --full-name-hash is much clearer and appealing to me now\nhaving had both more time and seeing the series approached in a couple\nof different ways. Let me know what you think.\n\nThanks,\nTaylor\n"},{"id":"506473","messageId":"20241101222334.1724050-1-jonathantanmy@google.com","threadId":"62429","inReplyTo":"6f93dff88e7cf81800c8f2c8d39175d962a0576f.1730356023.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-01T22:23:34Z","receivedAt":"2024-11-01T22:23:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I haven't looked thoroughly at the rest of the patches yet, but had a\ncomment about this test. Rearranging:\n\n\"Derrick Stolee via GitGitGadget\" writes:\n> +test_expect_success 'all' '\n> +\ttest-tool path-walk -- --all >out &&\n> +\n> +\tcat >expect <<-EOF &&\n> +\tTREE::$(git rev-parse topic^{tree})\n> +\tTREE::$(git rev-parse base^{tree})\n> +\tTREE::$(git rev-parse base~1^{tree})\n> +\tTREE::$(git rev-parse base~2^{tree})\n> +\tTREE:left/:$(git rev-parse base:left)\n> +\tTREE:left/:$(git rev-parse base~2:left)\n> +\tTREE:right/:$(git rev-parse topic:right)\n> +\tTREE:right/:$(git rev-parse base~1:right)\n> +\tTREE:right/:$(git rev-parse base~2:right)\n> +\ttrees:9\n\n[snip rest of \"expect\"]\n\nThe way you're testing this, wouldn't the tests pass even if the OIDs\naren't emitted in path order? (E.g. if topic:right and base~1:right\nwere somehow grouped into two different groups, even though they have\nthe same path.)\n\nI would have expected the test output to be something like:\n\n TREE:right/ $(rp :right topic base~1 base~2)\n\nwhere rp is a function that takes in a suffix and one or more prefixes -\nI haven't figured out its contents yet, but\n\n echo $(git rev-parse HEAD^^ HEAD^ HEAD | sort)\n\ngives us a space-separated list, so it doesn't seem too difficult to\ndefine such a function.\n\n> +static int emit_block(const char *path, struct oid_array *oids,\n> +\t\t enum object_type type, void *data)\n> +{\n> +\tstruct path_walk_test_data *tdata = data;\n> +\tconst char *typestr;\n> +\n> +\tswitch (type) {\n> +\tcase OBJ_TREE:\n> +\t\ttypestr = \"TREE\";\n> +\t\ttdata->tree_nr += oids->nr;\n> +\t\tbreak;\n> +\n> +\tcase OBJ_BLOB:\n> +\t\ttypestr = \"BLOB\";\n> +\t\ttdata->blob_nr += oids->nr;\n> +\t\tbreak;\n> +\n> +\tdefault:\n> +\t\tBUG(\"we do not understand this type\");\n> +\t}\n> +\n> +\tfor (size_t i = 0; i < oids->nr; i++)\n> +\t\tprintf(\"%s:%s:%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]));\n\nThen here, you would print typestr and path before the \"for\" loop. In\nthe \"for\" loop you would add oid_to_hex() results to a sorted string\nlist, have another \"for\" loop that prints each element preceded by a\nspace, then print a \"\\n\" after both \"for\" loops.\n"},{"id":"506543","messageId":"2d2940ef-0b26-4060-90b6-9b6969f23754@gmail.com","threadId":"62429","inReplyTo":"ZyUqr/wb5K4Og9j9@nand.local","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-04T15:48:49Z","receivedAt":"2024-11-04T15:48:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/24 3:23 PM, Taylor Blau wrote:\n> Hi Stolee,\n> \n> On Thu, Oct 31, 2024 at 06:26:57AM +0000, Derrick Stolee via GitGitGadget wrote:\n>>\n>> Introduction and relation to prior series\n>> =========================================\n>>\n>> This is a new series that rerolls the initial \"path-walk API\" patches of my\n>> RFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\n>> path-walk.h) presents a new way to walk objects such that trees and blobs\n>> are walked in batches according to their path.\n>>\n>> This also replaces the previous version of ds/path-walk that was being\n>> reviewed in [2]. The consensus was that the series was too long/dense and\n>> could use some reduction in size. This series takes the first few patches,\n>> but also makes some updates (which will be described later).\n>>\n>> [1]\n>> https://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n>> [RFC] Path-walk API and applications\n>>\n>> [2]\n>> https://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n>> [PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n> \n> I apologize for not having a better place to start discussing a topic\n> which pertains to more than just this immediate patch series, but I\n> figure here is as good a place as any to do so.\n> \n> From our earlier discussion, it seems to stand that the path-walk API\n> is fundamentally incompatible with reachability bitmaps and\n> delta-islands, making the series a non-starter in environments that\n> rely significantly one or both of those features. My understanding as a\n> result is that the path-walk API and feature are more targeted towards\n> improving client-side repacks and push performance, where neither of the\n> aforementioned two features are used quite as commonly.\n\nThis is correct. I would go even farther to say that this approach was\ndesigned first and foremost for Git clients and specifically their\nperformance while computing a thin packfile during \"git push\". The same\nlogic to help the push case happens to also help the \"git repack\" case\nsignificantly.\n\n> I was discussing this a bit off-list with Peff (who I hope will join the\n> thread and share his own thoughts), but I wonder if it was a mistake to\n> discard your '--full-name-hash' idea (or something similar, which I'll\n> discuss in a bit below) from earlier.\n\nI'd be happy to resurrect that series, adding in the learnings from\nworking on the path-walk feature. It helps that the current series adds\nthe path-walk API and has no conflicting changes in the pack-objects or\nrepack builtins. (I can handle those conflicts as things merge down.)\n\n> (Repeating a few things that I am sure are obvious to you out loud so\n> that I can get a grasp on them for my own understanding):\n> \n> It seems that the problems you've identified which result in poor repack\n> performance occur when you have files at the same path, but they get\n> poorly sorted in the delta selection window due to other paths having\n> the same final 16 characters, so Git doesn't see that much better delta\n> opportunities exist.\n> \n> Your series takes into account the full name when hashing, which seems\n> to produce a clear win in many cases. I'm sure that there are some cases\n> where it presents a modest regression in pack sizes, but I think that's\n> fine and probably par for the course when making any changes like this,\n> as there is probably no easy silver bullet here that uniformly improves\n> all cases.\n> \n> I suspect that you could go even further and intern the full path at\n> which each object occurs, and sort lexically by that. Just stringing\n> together all of the paths in linux.git only takes 3.099 MiB on my clone.\n> (Of course, that's unbounded in the number of objects and length of\n> their pathnames, but you could at least bound the latter by taking only\n> the last, say, 128 characters, which would be more than good enough for\n> the kernel, whose longest path is only 102 characters).\n\nWhen the optimization idea is to focus on the full path and not care\nabout the \"locality\" of the path name by its later bits, storing the\nfull name in a list and storing an index into that list would have a\nvery similar effect.\n\nI'd be interested to explore the idea of storing the full path name.\nBased on my exploration with the 'test-tool name-hash' test helper in\nthat series, I'm not sure that we will make significant gains by doing\nso. Worth trying.\n\n> Some of the repositories that you've tested on I don't have easy access\n> to, so I wonder if either doing (a) that, or (b) using some fancier\n> context-sensitive hash (like SimHash or MinHash) would be beneficial.\n\nI don't know too much about SimHash or MinHash, but based on what I\ncould gather from some initial reading I'm not sure that they would be\neffective without increasing the hash length. We'd also get a different\nkind of locality, such as the appearance of a common word would be more\nlikely to affect the locality than the end of the path.\n\nThe size of the hash could probably be mitigated by storing it in the\nlist of all full paths and accessing them from the index stored on each\nto-pack object.\n\n> I realize that this is taking us back to an idea you've already\n> presented to the list, but I think (to me, at least) the benefit and\n> simplicity of that approach has only become clear to me in hindsight\n> when seeing some alternatives. I would like to apologize for the time\n> you spent reworking this series back and forth to have the response be\n> \"maybe we should have just done the first thing you suggested\". Like I\n> said, I think to me it was really only clear in hindsight.\n\nI always assumed that we'd come back to it eventually. There is also the\nextra bit about making the change to the name-hash compatible with the\nway name-hashes are stored in the reachability bitmaps. That will need\nsome work before it is ready for prime time.\n\n> In any event, the major benefit to doing --full-name-hash would be that\n> *all* environments could benefit from the size reduction, not just those\n> that don't rely on certain other features.\n\nI disagree that all environments will prefer the --full-name-hash. I'm\ncurrently repeating the performance tests right now, and I've added one.\nThe issues are:\n\n 1. The --full-name-hash approach sometimes leads to a larger pack when\n using \"git push\" on the client, especially when the name-hash is\n already effective for compressing across paths.\n\n 2. A depth 1 shallow clone cannot use previous versions of a path, so\n those situations will want to use the normal name hash. This can be\n accomplished simply by disabling the --full-name-hash option when\n the --shallow option is present; a more detailed version could be\n used to check for a large depth before disabling it. This case also\n disables bitmaps, so that isn't something to worry about.\n\n> Perhaps just --full-name-hash isn't quite as good by itself as the\n> --path-walk implementation that this series starts us off implementing.\n> So in that sense, maybe we want both, which I understand was the\n> original approach. I see a couple of options here:\n> \n> - We take both, because doing --path-walk on top represents a\n> significant enough improvement that we are collectively OK with\n> taking on more code to improve a more narrow (but common) use-case.\n\nDoing both doesn't help at all, since the --path-walk approach already\nbatches by the full path name. The --path-walk approach has a significant\nbenefit by doing a second pass by the standard name-hash to pick up on the\ncross-path deltas. This is why the --path-walk approach with the standard\nname hash as consistently provided the most-compact pack-files in all\ntests.\n\n Aside: there were some initial tests that showed the --path-walk option\n led to slightly larger packfiles, but I've since discovered that those\n cases were due to an incorrect walking of indexed paths. This is fixed\n by the code in patch 5 of the current series and my WIP patches in [3]\n have the performance numbers with this change.\n\n[3] https://github.com/gitgitgadget/git/pull/1819\nPATH WALK II: Add --path-walk option to 'git pack-objects'\n\n> - Or we decide that either the benefit isn't significant enough to\n> warrant an additional and relatively complex implementation, or in\n> other words that --full-name-hash by itself is good enough.\n\nI hope that I've sufficiently communicated that --full-name-hash is not\ngood enough by itself.\n\nThe point I was trying to make by submitting it first was that I believed\nit was likely the easiest way for Git servers to gain 90% of the benefits\nthat the --path-walk approach provides while making it relatively easy to\nintegrate with other server-side features such as bitmaps and delta islands.\n\n(Maybe the --path-walk approach could also be extended to be compatible\nwith those features, but it would be a significant investment that rebuilds\nthose features within the context of the new object walk instead of relying\non the existing implementations. That could easily be a blocker.)\n\n> Again, I apologize for not having a clearer picture of this all to start\n> with, and I want to tell you specifically and sincerely that I\n> appreciate your patience as I wrap my head around all of this. I think\n> the benefit of --full-name-hash is much clearer and appealing to me now\n> having had both more time and seeing the series approached in a couple\n> of different ways. Let me know what you think.\nThanks for taking the time to engage with the patches. I'm currently\nrerunning my performance tests on a rebased copy of the --full-name-hash\npatches and will submit a new version when it's ready.\n\nThanks,\n-Stolee\n\n"},{"id":"506545","messageId":"58e2e41e-ff67-4d4b-98f0-48be5eb3df27@gmail.com","threadId":"62429","inReplyTo":"20241101222334.1724050-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-04T15:56:05Z","receivedAt":"2024-11-04T15:56:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/24 6:23 PM, Jonathan Tan wrote:\n> I haven't looked thoroughly at the rest of the patches yet, but had a\n> comment about this test. Rearranging:\n> \n> \"Derrick Stolee via GitGitGadget\" writes:\n>> +test_expect_success 'all' '\n>> +\ttest-tool path-walk -- --all >out &&\n>> +\n>> +\tcat >expect <<-EOF &&\n>> +\tTREE::$(git rev-parse topic^{tree})\n>> +\tTREE::$(git rev-parse base^{tree})\n>> +\tTREE::$(git rev-parse base~1^{tree})\n>> +\tTREE::$(git rev-parse base~2^{tree})\n>> +\tTREE:left/:$(git rev-parse base:left)\n>> +\tTREE:left/:$(git rev-parse base~2:left)\n>> +\tTREE:right/:$(git rev-parse topic:right)\n>> +\tTREE:right/:$(git rev-parse base~1:right)\n>> +\tTREE:right/:$(git rev-parse base~2:right)\n>> +\ttrees:9\n> \n> [snip rest of \"expect\"]\n> \n> The way you're testing this, wouldn't the tests pass even if the OIDs\n> aren't emitted in path order? (E.g. if topic:right and base~1:right\n> were somehow grouped into two different groups, even though they have\n> the same path.)\n\nYou are correct that if the path-walk API emitted multiple batches\nwith the same path name, then we would not detect that via the current\ntesting strategy.\n\nThe main reason to use the sort is to avoid adding a restriction on\nthe order in which objects appear within the batch.\n\nYour recommendation to group a batch into a single line does not\nstrike me as a suitable approach, because long lines become hard to\nread and difficult to parse diffs. (Also, the order within the batch\nbecomes baked in as a requirement.)\n\nThe biggest question I'd like to ask is this: do you see a risk of\na path being repeated? There are cases where it will happen, such as\nindexed objects that are not reachable anywhere else.\n\nThe way I would consider modifying these tests to reflect the batching\nwould be to associate each batch with a number, causing the order of\nthe paths to become hard-coded in the test. Something like\n\n 0:COMMIT::$(git rev-parse ...)\n 0:COMMIT::$(git rev-parse ...)\n 1:TREE::$(git rev-parse ...)\n 1:TREE::$(git rev-parse ...)\n 2:TREE:right/:$(git rev-parse ...)\n 3:BLOB:right/a:$(...)\n 4:TREE:left/:$(git rev-parse ...)\n 5:BLOB:left/b:$(...)\n\nThis would imply some amount of order that maybe should become a\nrequirement of the API.\n\nThanks,\n-Stolee\n"},{"id":"506546","messageId":"1b81a6a3-ba51-4d2c-8529-1063d045bad5@gmail.com","threadId":"62429","inReplyTo":"CAOLa=ZR1BjKzujxpA2icAfJHin-krqwkxTf5VPeGg14AVr-TcQ@mail.gmail.com","subject":"Re: [PATCH 5/6] path-walk: visit tags and cached objects","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-04T15:56:45Z","receivedAt":"2024-11-04T15:56:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/24 10:25 AM, karthik nayak wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> The rev_info that is specified for a path-walk traversal may specify\n>> visiting tag refs (both lightweight and annotated) and also may specify\n>> indexed objects (blobs and trees). Update the path-walk API to walk\n>> these objects as well.\n>>\n>> When walking tags, we need to peel the annotated objects until reaching\n>> a non-tag object. If we reach a commit, then we can add it to the\n>> pending objects to make sure we visit in the commit walk portion. If we\n> \n> Nit: s/in/it in/\n\nthanks!\n\n>> +\t\tcase OBJ_BLOB:\n>> +\t\t\tif (!info->blobs)\n>> +\t\t\t\tcontinue;\n>> +\t\t\tif (pending->path) {\n>> +\t\t\t\tstruct type_and_oid_list *list;\n>> +\t\t\t\tchar *path = pending->path;\n>> +\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n>> +\t\t\t\t\tCALLOC_ARRAY(list, 1);\n>> +\t\t\t\t\tlist->type = OBJ_BLOB;\n>> +\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n>> +\t\t\t\t}\n>> +\t\t\t\toid_array_append(&list->oids, &obj->oid);\n>> +\t\t\t} else {\n>> +\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n> \n> Shouldn't this comment be for tagged blobs?\n\nYes. This is a copy-paste error.\n\nThanks for the careful reading.\n-Stolee\n"},{"id":"506547","messageId":"20241104172533.GA2985568@coredump.intra.peff.net","threadId":"62429","inReplyTo":"2d2940ef-0b26-4060-90b6-9b6969f23754@gmail.com","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-04T17:25:33Z","receivedAt":"2024-11-04T17:25:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 04, 2024 at 10:48:49AM -0500, Derrick Stolee wrote:\n\n> > I was discussing this a bit off-list with Peff (who I hope will join the\n> > thread and share his own thoughts), but I wonder if it was a mistake to\n> > discard your '--full-name-hash' idea (or something similar, which I'll\n> > discuss in a bit below) from earlier.\n> \n> I'd be happy to resurrect that series, adding in the learnings from\n> working on the path-walk feature. It helps that the current series adds\n> the path-walk API and has no conflicting changes in the pack-objects or\n> repack builtins. (I can handle those conflicts as things merge down.)\n\nAdding my two cents, the discussion we had came after reading this post:\n\n https://www.jonathancreamer.com/how-we-shrunk-our-git-repo-size-by-94-percent/\n\nI think a few of the low-level details in there are confusing, but it\nseemed to me that most of the improvement he mentions is just about\nfinding better delta candidates. And it seems obvious that our current\npack_name_hash() is pretty rudimentary as context-sensitive hashing\ngoes and won't do well for long paths with similar endings.\n\nSo just swapping that out for something better seems like an easy thing\nto do regardless of whether we pursue --path-walk. It doesn't\ndrastically change how we choose delta pairs so it's not much code and\nit shouldn't conflict with other features. And the risk of making\nanything worse should be pretty low.\n\nI wouldn't at all be surprised if --path-walk can do better, but if we\ndo the easy thing first then I think it gives us a better idea of the\ncost/benefit it's providing.\n\nI suspect there's room for both in the long run. You seem to be focused\non push size and cost, whereas I think Taylor and I are more interested\nin overall repo size and cost of serving bitmapped fetches.\n\n> When the optimization idea is to focus on the full path and not care\n> about the \"locality\" of the path name by its later bits, storing the\n> full name in a list and storing an index into that list would have a\n> very similar effect.\n> \n> I'd be interested to explore the idea of storing the full path name.\n> Based on my exploration with the 'test-tool name-hash' test helper in\n> that series, I'm not sure that we will make significant gains by doing\n> so. Worth trying.\n\nThe way I look at it is a possible continuum. We want to use pathnames\nas a way to sort delta candidates near each other, since we expect them\nto have high locality with respect to delta-able contents. The current\nname_hash uses a very small bit of that path information and throws away\nmost of it. The other extreme end is holding the whole path. We may want\nto end up in the middle for two reasons:\n\n 1. Dealing with whole paths might be costly (though I'm not yet\n convinced of that; outside of pathological cases, the number of\n paths in a repo tends to pale in comparison to the number of\n objects, and the per-object costs dominate during repacking).\n\n 2. It's possible that over-emphasizing the path might be a slightly\n worse heuristic (and I think this is a potential danger of\n --path-walk, too). We still want to find candidate pairs that were\n copied or renamed, for example, or that substantially share content\n found in different parts of the tree.\n\nSo it would be interesting to be able to see the performance of various\npoints on that line, from full path down to partial paths down to longer\nhashes down to the current hash. The true extreme end of course is no\npath info at all, but I think we know that sucks; that's why we\nimplemented the bitmap name-hash extension in the first place.\n\n> I don't know too much about SimHash or MinHash, but based on what I\n> could gather from some initial reading I'm not sure that they would be\n> effective without increasing the hash length. We'd also get a different\n> kind of locality, such as the appearance of a common word would be more\n> likely to affect the locality than the end of the path.\n\nGood point. This is all heuristics, of course, but I suspect that the\norder of the path is important, and that foo/bar.c and bar/foo.c are\nunlikely to be good matches.\n\n> > I realize that this is taking us back to an idea you've already\n> > presented to the list, but I think (to me, at least) the benefit and\n> > simplicity of that approach has only become clear to me in hindsight\n> > when seeing some alternatives. I would like to apologize for the time\n> > you spent reworking this series back and forth to have the response be\n> > \"maybe we should have just done the first thing you suggested\". Like I\n> > said, I think to me it was really only clear in hindsight.\n> \n> I always assumed that we'd come back to it eventually. There is also the\n> extra bit about making the change to the name-hash compatible with the\n> way name-hashes are stored in the reachability bitmaps. That will need\n> some work before it is ready for prime time.\n\nHaving worked on that feature of bitmaps, I'm not too worried about it.\nI think we'd just need to:\n\n - introduce a new bitmap ext with a flag (HASH_CACHE_V2 or something,\n either with the new hash, or with a \"version\" byte at the start for\n extensibility).\n\n - when bitmaps are not in use, we're free to use whichever hash we\n want internally. If the new hash is consistently better, we'd\n probably just enable it by default.\n\n - when packing using on-disk bitmaps, use internally whichever format\n the on-disk file provided. Technically the format could even provide\n both (in which case we'd prefer the new hash), but I don't see much\n point.\n\n - when writing bitmaps, use whichever hash the command-line options\n asked for. There's an off chance somebody might want to generate a\n .bitmap file whose hashes will be understood by an older version of\n git, in which case they'd use --no-full-name-hash or whatever while\n repacking.\n\nIf we're considering full paths, then that is potentially a bit more\ninvolved, just because we'd want the format to avoid repeating duplicate\npaths for each object (plus they're no longer fixed-size). So probably\nan extension with packed NUL-terminated path strings, plus a set of\nfixed-length offsets into that block, one per object.\n\n> I disagree that all environments will prefer the --full-name-hash. I'm\n> currently repeating the performance tests right now, and I've added one.\n> The issues are:\n> \n> 1. The --full-name-hash approach sometimes leads to a larger pack when\n> using \"git push\" on the client, especially when the name-hash is\n> already effective for compressing across paths.\n\nThat's interesting. I wonder which cases get worse, and if a larger\nwindow size might help. I.e., presumably we are pushing the candidates\nfurther away in the sorted delta list.\n\n> 2. A depth 1 shallow clone cannot use previous versions of a path, so\n> those situations will want to use the normal name hash. This can be\n> accomplished simply by disabling the --full-name-hash option when\n> the --shallow option is present; a more detailed version could be\n> used to check for a large depth before disabling it. This case also\n> disables bitmaps, so that isn't something to worry about.\n\nI'm not sure why a larger hash would be worse in a shallow clone. As you\nnote, with only one version of each path the name-similarity heuristic\nis not likely to buy you much. But I'd have thought that would be true\nfor the existing name hash as well as a longer one. Maybe this is the\n\"over-emphasizing\" case.\n\n-Peff\n"},{"id":"506578","messageId":"20241104233912.2493936-1-jonathantanmy@google.com","threadId":"62429","inReplyTo":"58e2e41e-ff67-4d4b-98f0-48be5eb3df27@gmail.com","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-04T23:39:12Z","receivedAt":"2024-11-04T23:39:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Derrick Stolee writes:\n> You are correct that if the path-walk API emitted multiple batches\n> with the same path name, then we would not detect that via the current\n> testing strategy.\n> \n> The main reason to use the sort is to avoid adding a restriction on\n> the order in which objects appear within the batch.\n> \n> Your recommendation to group a batch into a single line does not\n> strike me as a suitable approach, because long lines become hard to\n> read and difficult to parse diffs. (Also, the order within the batch\n> becomes baked in as a requirement.)\n\nThe hashes in a line can be abbreviated if line length is a concern.\nAlso, note that I am suggesting sorting the OIDs within a line (that is,\na batch), and also sorting the lines (batches) as a whole.\n\n> The biggest question I'd like to ask is this: do you see a risk of\n> a path being repeated? There are cases where it will happen, such as\n> indexed objects that are not reachable anywhere else.\n\nI was thinking that the whole point of this feature is that we group\nobjects by path, so it seems desirable to test that paths are not\nrepeated. (Or repeated as little as possible, if it is not possible\nto avoid repetition e.g. in the case you describe.)\n\n> The way I would consider modifying these tests to reflect the batching\n> would be to associate each batch with a number, causing the order of\n> the paths to become hard-coded in the test. Something like\n> \n> 0:COMMIT::$(git rev-parse ...)\n> 0:COMMIT::$(git rev-parse ...)\n> 1:TREE::$(git rev-parse ...)\n> 1:TREE::$(git rev-parse ...)\n> 2:TREE:right/:$(git rev-parse ...)\n> 3:BLOB:right/a:$(...)\n> 4:TREE:left/:$(git rev-parse ...)\n> 5:BLOB:left/b:$(...)\n> \n> This would imply some amount of order that maybe should become a\n> requirement of the API.\n> \n> Thanks,\n> -Stolee\n\nIf we're willing to declare an order in which we will return paths to\nthe user, that would work too. (I'm not sure that we need to declare an\norder, though.)\n"},{"id":"506581","messageId":"xmqq1pzqwnck.fsf@gitster.g","threadId":"62429","inReplyTo":"20241104172533.GA2985568@coredump.intra.peff.net","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-05T00:11:55Z","receivedAt":"2024-11-05T00:11:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Mon, Nov 04, 2024 at 10:48:49AM -0500, Derrick Stolee wrote:\n>> I disagree that all environments will prefer the --full-name-hash. I'm\n>> currently repeating the performance tests right now, and I've added one.\n>> The issues are:\n>> \n>> 1. The --full-name-hash approach sometimes leads to a larger pack when\n>> using \"git push\" on the client, especially when the name-hash is\n>> already effective for compressing across paths.\n>\n> That's interesting. I wonder which cases get worse, and if a larger\n> window size might help. I.e., presumably we are pushing the candidates\n> further away in the sorted delta list.\n>\n>> 2. A depth 1 shallow clone cannot use previous versions of a path, so\n>> those situations will want to use the normal name hash. This can be\n>> accomplished simply by disabling the --full-name-hash option when\n>> the --shallow option is present; a more detailed version could be\n>> used to check for a large depth before disabling it. This case also\n>> disables bitmaps, so that isn't something to worry about.\n>\n> I'm not sure why a larger hash would be worse in a shallow clone. As you\n> note, with only one version of each path the name-similarity heuristic\n> is not likely to buy you much. But I'd have thought that would be true\n> for the existing name hash as well as a longer one. Maybe this is the\n> \"over-emphasizing\" case.\n\nI too am curious to hear Derrick explain the above points and what\nwas learned from the performance tests. The original hash was\ndesigned to place files that are renamed across directories closer\nto each other in the list sorted by the name hash, so a/Makefile and\nb/Makefile would likely be treated as delta-base candidates while\nfoo/bar.c and bar/foo.c are treated as unrelated things. A push\nof a handful of commits that rename paths would likely place the\nrename source of older commits and rename destination of newer\ncommits into the same delta chain, even with a smaller delta window.\n\nIn such a history, uniformly-distributed-without-regard-to-renames\nhash is likely to make them into two distinct delta chains, leading\nto less optimal delta-base selection.\n\nA whole-repository packing, or a large push or fetch, of the same\nhistory with renamed files are affected a lot less by such negative\neffects of full-name hash. When generating a pack with more commits\nthan the \"--window\", use of the original hash would mean blobs from\npaths that share similar names (e.g., \"Makefile\"s everywhere in the\ndirectory hierarchy) are placed close to each other, but full-name\nhash will likely group the blobs from exactly the same path and\nnothing else together, and the resulting delta-chain for identical\n(and not similar) paths would be sufficiently long. A long delta\nchain has to be broken into multiple chains _anyway_ due to finite\n\"--depth\" setting, so placing blobs from each path into its own\n(initial) delta chain, completely ignoring renamed paths, would\nlikely to give us long enough (initial) delta chain to be split at\nthe depth limit.\n\nIt would lead to a good delta-base selection with smaller window\nsize quite efficiently with full-name hash.\n\nI think a full-name hash forces a single-commit pack of a wide tree\nto give up on deltified blobs, but with the original hash, at least\nsimilar and common files (e.g. Makefile and COPYING) would sit close\ntogether in the delta queue and can be deltified with each other,\nwhich may be where the inefficiency comes from when full-name hash\nis used.\n\n"},{"id":"506719","messageId":"Zyt3aE2yNOgDyw4Y@pks.im","threadId":"62429","inReplyTo":"6f93dff88e7cf81800c8f2c8d39175d962a0576f.1730356023.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-06T14:04:38Z","receivedAt":"2024-11-06T14:04:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 31, 2024 at 06:27:00AM +0000, Derrick Stolee via GitGitGadget wrote:\n[snip]\n> +int cmd__path_walk(int argc, const char **argv)\n> +{\n> +\tint res;\n> +\tstruct rev_info revs = REV_INFO_INIT;\n> +\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n> +\tstruct path_walk_test_data data = { 0 };\n> +\tstruct option options[] = {\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\tinitialize_repository(the_repository);\n> +\tsetup_git_directory();\n> +\trevs.repo = the_repository;\n> +\n> +\targc = parse_options(argc, argv, NULL,\n> +\t\t\t options, path_walk_usage,\n> +\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT | PARSE_OPT_KEEP_ARGV0);\n> +\n> +\tif (argc > 1)\n> +\t\tsetup_revisions(argc, argv, &revs, NULL);\n> +\telse\n> +\t\tusage(path_walk_usage[0]);\n> +\n> +\tinfo.revs = &revs;\n> +\tinfo.path_fn = emit_block;\n> +\tinfo.path_fn_data = &data;\n> +\n> +\tres = walk_objects_by_path(&info);\n> +\n> +\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n> +\t \"blobs:%\" PRIuMAX \"\\n\",\n> +\t data.tree_nr, data.blob_nr);\n> +\n> +\treturn res;\n> +}\n\nThis function is leaking memory. I'd propose to add below patch on top\nto plug them, which makes t6601 pass with the leak sanitizer enabled.\n\nPatrick\n\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 06b103d876..fa3bfe46b5 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -85,7 +85,6 @@ int cmd__path_walk(int argc, const char **argv)\n \t\tOPT_END(),\n \t};\n \n-\tinitialize_repository(the_repository);\n \tsetup_git_directory();\n \trevs.repo = the_repository;\n \n@@ -110,5 +109,6 @@ int cmd__path_walk(int argc, const char **argv)\n \t \"tags:%\" PRIuMAX \"\\n\",\n \t data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);\n \n+\trelease_revisions(&revs);\n \treturn res;\n }\n"},{"id":"506862","messageId":"5085b96d-4757-4ace-9610-baa21c05bf4d@gmail.com","threadId":"62429","inReplyTo":"20241104233912.2493936-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-08T14:53:48Z","receivedAt":"2024-11-08T14:53:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/4/24 6:39 PM, Jonathan Tan wrote:\n> Derrick Stolee writes:\n> \n>> The biggest question I'd like to ask is this: do you see a risk of\n>> a path being repeated? There are cases where it will happen, such as\n>> indexed objects that are not reachable anywhere else.\n> \n> I was thinking that the whole point of this feature is that we group\n> objects by path, so it seems desirable to test that paths are not\n> repeated. (Or repeated as little as possible, if it is not possible\n> to avoid repetition e.g. in the case you describe.)\nIn addition to determining the order of the batches, it can be\nhelpful to demonstrate that we don't call the path_fn with an\nempty batch! I discovered this while making the appropriate\nchanges today and putting the fixes in the right places.\n\nThanks,\n-Stolee\n\n"},{"id":"506863","messageId":"34e00703-1324-494a-9323-ed87718bc235@gmail.com","threadId":"62429","inReplyTo":"Zyt3aE2yNOgDyw4Y@pks.im","subject":"Re: [PATCH 3/6] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-08T14:58:25Z","receivedAt":"2024-11-08T14:58:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/6/24 9:04 AM, Patrick Steinhardt wrote:\n> On Thu, Oct 31, 2024 at 06:27:00AM +0000, Derrick Stolee via GitGitGadget wrote:\n> [snip]\n\n> This function is leaking memory. I'd propose to add below patch on top\n> to plug them, which makes t6601 pass with the leak sanitizer enabled.\nThanks! Applied for the next version.\n\n-Stolee\n\n"},{"id":"506864","messageId":"f02ee8ac-01e4-42e3-b99a-d9616b9ff1bb@gmail.com","threadId":"62429","inReplyTo":"xmqq1pzqwnck.fsf@gitster.g","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-08T15:17:24Z","receivedAt":"2024-11-08T15:17:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/4/24 7:11 PM, Junio C Hamano wrote:\n> Jeff King writes:\n> \n>> On Mon, Nov 04, 2024 at 10:48:49AM -0500, Derrick Stolee wrote:\n>>> I disagree that all environments will prefer the --full-name-hash. I'm\n>>> currently repeating the performance tests right now, and I've added one.\n>>> The issues are:\n>>>\n>>> 1. The --full-name-hash approach sometimes leads to a larger pack when\n>>> using \"git push\" on the client, especially when the name-hash is\n>>> already effective for compressing across paths.\n>>\n>> That's interesting. I wonder which cases get worse, and if a larger\n>> window size might help. I.e., presumably we are pushing the candidates\n>> further away in the sorted delta list.\n\nI think the cases that make things get worse with --full-name-hash are:\n\n 1. The presence of renames, partitioning objects that used to fit into\n the same bucket (in the case of directory renames).\n\n 2. Some standard kinds of files may appear several times across the\n tree but do not change very often and are similar across path.\n\n 3. Common patterns across similar file types, such as similar includes\n in .c and .h files or other kinds of boilerplate in different\n languages.\n\nA larger window size will always expand the range of possible deltas, at\na cost to the time taken to compute the deltas. My first experiment in\nthese repositories was to increase the window size to 250 (from the default\n10). This caused a very slow repack, but the repositories shrunk.\n\nFor example, the big Javascript monorepo that repacked to ~100 GB with\ndefault settings would repack to ~30 GB with --window=250. This was an\nindicator that delta compression would work if we can find the right pairs\nto use for deltas.\n\nThe point of the two strategies (--full-name-hash and --path-walk) is\nabout putting objects close to each other in better ways than the name\nhash sort.\n\n>>> 2. A depth 1 shallow clone cannot use previous versions of a path, so\n>>> those situations will want to use the normal name hash. This can be\n>>> accomplished simply by disabling the --full-name-hash option when\n>>> the --shallow option is present; a more detailed version could be\n>>> used to check for a large depth before disabling it. This case also\n>>> disables bitmaps, so that isn't something to worry about.\n>>\n>> I'm not sure why a larger hash would be worse in a shallow clone. As you\n>> note, with only one version of each path the name-similarity heuristic\n>> is not likely to buy you much. But I'd have thought that would be true\n>> for the existing name hash as well as a longer one. Maybe this is the\n>> \"over-emphasizing\" case.\n\nI'm confused by your wording of \"larger hash\" because the hash size\nis the exact same: 32 bits. It's just that the --full-name-hash option\nhas fewer collisions by abandoning the hope of locality.\n\nIn a depth 1 shallow clone, there are no repeated paths, so any hash\ncollisions are true collisions instead of good candidates for deltas.\nThe full name hash is essentially random, so the delta compression\nalgorithm basically says:\n\n 1. Sort by type.\n 2. Within each type, sort the objects randomly.\n\nWith that sort, the delta compression scan is less effective than the\nstandard name hash.\n\n> I too am curious to hear Derrick explain the above points and what\n> was learned from the performance tests. The original hash was\n> designed to place files that are renamed across directories closer\n> to each other in the list sorted by the name hash, so a/Makefile and\n> b/Makefile would likely be treated as delta-base candidates while\n> foo/bar.c and bar/foo.c are treated as unrelated things. A push\n> of a handful of commits that rename paths would likely place the\n> rename source of older commits and rename destination of newer\n> commits into the same delta chain, even with a smaller delta window.\n\n> In such a history, uniformly-distributed-without-regard-to-renames\n> hash is likely to make them into two distinct delta chains, leading\n> to less optimal delta-base selection.\n\nYes. This is the downside of the --full-name-hash compared to the\nstandard name hash. When repacking an entire repository, the effect\nof these renames is typically not important in the long run as it's\nbasically a single break in the delta chain. The downside comes in\nwhen doing a small fetch or push where the rename has more impact.\n\nThe --path-walk approach does not suffer from this problem because\nit has a second pass that sorts by the name hash and looks for\nbetter deltas than the ones that already exist. Thus, it gets the\nbest of both worlds.\n\nThe performance impact of the two passes of the --path-walk\napproach is interesting, as you'd typically expect this to always\nbe slower. However:\n\n 1. The delta compression within each batch only compares the\n objects within that batch. We do not compare these objects to\n unrelated objects, which can be expensive and wasteful. This\n also means that small batches may even be smaller than the\n delta window, reducing the number of comparisons.\n\n 2. In the second pass, the delta calculation can short-circuit if\n the computed delta would be larger than the current-best delta.\n Thus, the good deltas from the first pass make the second pass\n faster.\n\n> A whole-repository packing, or a large push or fetch, of the same\n> history with renamed files are affected a lot less by such negative\n> effects of full-name hash. When generating a pack with more commits\n> than the \"--window\", use of the original hash would mean blobs from\n> paths that share similar names (e.g., \"Makefile\"s everywhere in the\n> directory hierarchy) are placed close to each other, but full-name\n> hash will likely group the blobs from exactly the same path and\n> nothing else together, and the resulting delta-chain for identical\n> (and not similar) paths would be sufficiently long. A long delta\n> chain has to be broken into multiple chains _anyway_ due to finite\n> \"--depth\" setting, so placing blobs from each path into its own\n> (initial) delta chain, completely ignoring renamed paths, would\n> likely to give us long enough (initial) delta chain to be split at\n> the depth limit.\n> \n> It would lead to a good delta-base selection with smaller window\n> size quite efficiently with full-name hash.>\n> I think a full-name hash forces a single-commit pack of a wide tree\n> to give up on deltified blobs, but with the original hash, at least\n> similar and common files (e.g. Makefile and COPYING) would sit close\n> together in the delta queue and can be deltified with each other,\n> which may be where the inefficiency comes from when full-name hash\n> is used.\n\nYes, this is a good summary of why this works for the data\nefficiency in long histories. Your earlier observations are why\nthe full-name hash has demonstrated issues with smaller time scales.\n\nThese numbers are carefully detailed in the performance tests in\nthe refreshed series [1]. The series also has a way to disable the\nfull-name hash when serving a shallow clone for this reason.\n\n[1] https://lore.kernel.org/git/pull.1823.git.1730775907.gitgitgadget@gmail.com/\n\nThanks,\n-Stolee\n"},{"id":"506916","messageId":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.git.1730356023.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] PATH WALK I: The path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:06Z","receivedAt":"2024-11-09T19:41:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nIntroduction and relation to prior series\n=========================================\n\nThis is a new series that rerolls the initial \"path-walk API\" patches of my\nRFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\npath-walk.h) presents a new way to walk objects such that trees and blobs\nare walked in batches according to their path.\n\nThis also replaces the previous version of ds/path-walk that was being\nreviewed in [2]. The consensus was that the series was too long/dense and\ncould use some reduction in size. This series takes the first few patches,\nbut also makes some updates (which will be described later).\n\n[1]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n[RFC] Path-walk API and applications\n\n[2]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n[PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n\nThis series only introduces the path-walk API, but does so to the full\ncomplexity required to later add the integration with git pack-objects to\nimprove packing compression in both time and space for repositories with\nmany name hash collisions. The compression also at least improves for other\nrepositories, but may not always have an improvement in time.\n\nSome of the changes that are present in this series that differ from the\nprevious version are motivated directly by discoveries made by testing the\nfeature in Git for Windows and microsoft/git forks that shipped these\nfeatures for fast delivery of these improvements to users who needed them.\nThat testing across many environments informed some things that needed to be\nchanged, and in this series those changes are checked by tests in the\nt6601-path-walk.sh test script and the test-tool path-walk test helper.\nThus, the code being introduced in this series is covered by tests even\nthough it is not integrated into the git executable.\n\n\nDiscussion of follow-up applications\n====================================\n\nBy splitting this series out into its own, I was able to reorganize the\npatches such that each application can be build independently off of this\nseries. These are available as pending PRs in gitgitgadget/git:\n\n * Better delta compression with 'git pack-objects' [3]: This application\n allows an option in 'git pack-objects' to change how objects are walked\n in order to group objects with the same path for early delta compression\n before using the name hash sort to look for cross-path deltas. This helps\n significantly in repositories with many name-hash collisions. This\n reduces the size of 'git push' pacifies via a config option and reduces\n the total repo size in 'git repack'.\n\n * The 'git backfill' command [4]: This command downloads missing blobs in a\n bloodless partial clone. In order to save space and network bandwidth, it\n assumes that objects at a common path are likely to delta well with each\n other, so it downloads missing blobs in batches via the path-walk API.\n This presents a way to use blobless clones as a pseudo-resumable clone,\n since the initial clone of commits and trees is a smaller initial\n download and the batch size allows downloading blobs incrementally. When\n pairing this command with the sparse-checkout feature, the path-walk API\n is adjusted to focus on the paths within the sparse-checkout. This allows\n the user to only download the files they are likely to need when\n inspecting history within their scope without downloading the entire\n repository history.\n\n * The 'git survey' command [5]. This application begins the work to mimic\n the behavior of git-sizer, but to use internal data structures for better\n performance and careful understanding of how objects are stored. Using\n the path-walk API, paths with many versions can be considered in a batch\n and sorted into a list to report the paths that contribute most to the\n size of the repository. A version of this command was used to help\n confirm the issues with the name hash collisions. It was also used to\n diagnose why some repacks using the --path-walk option were taking more\n space than without for some repositories. (More on this later.)\n\nQuestion for reviewers: I am prepped to send these three applications to the\nmailing list, but I'll refrain for now to avoid causing too much noise for\nfolks. Would you like to see them on-list while this series is under review?\nOr would you prefer to explore the PRs ([3] [4] and [5])?\n\n[3] https://github.com/gitgitgadget/git/pull/1819\nPATH WALK II: Add --path-walk option to 'git pack-objects'\n\n[4] https://github.com/gitgitgadget/git/pull/1820\nPATH WALK III: Add 'git backfill' command\n\n[5] https://github.com/gitgitgadget/git/pull/1821\nPATH WALK IV: Add 'git survey' command\n\n\nStructure of the Patch Series\n=============================\n\nThis patch series attempts to create the simplest version of the API in\npatch 1, then build functionality incrementally. During the process, each\nchange will introduce an update to:\n\n * The path-walk API itself in path-walk.c and path-walk.h.\n * The documentation of the API in\n Documentation/technical/api-path-walk.txt.\n * The test script t/t6601-path-walk.sh.\n\nThe core of the API relies on using a 'struct rev_info' to define an initial\nset of objects and some form of a commit walk to define what range of\nobjects to visit. Initially, only a subset of 'struct rev_info' options work\nas expected. For example:\n\n * Patch 1 assumes that only commit objects are starting positions, but the\n focus is on exploring trees and blobs.\n * Patch 3 allows users to specify object types, which includes outputting\n the visited commits in a batch.\n * Annotated tags and indexed objects are considered in Patch 4. These are\n grouped because they both exist within the 'pending' object list.\n * UNINTERESTING objects are not considered until Patch 5.\n\n\nChanges in v1 (since previous version)\n======================================\n\nThere are a few hard-won learnings from previous versions of this series due\nto testing this in the wild with many different repositories.\n\n * Initially, the 'git pack-objects --path-walk' feature was not tested with\n the '--shallow' option because it was expected that this option was for\n servers creating a pack containing shallow commits. However, this option\n is also used when pushing from a shallow clone, and this was a critical\n feature that we needed to reduce the size of commits pushed from\n automated environments that were bootstrapped by shallow clones. The crux\n of the change is in Patch 5 and how UNINTERESTING objects are handled. We\n no longer need to push the UNINTERESTING flag around the objects\n ourselves and can use existing logic in list-objects.c to do so. This\n allows using the --objects-edge-aggressive option when necessary to\n reduce the object count when pushing from a shallow clone. (The\n pack-objects series expands on tests to cover this integration point.)\n\n * When looking into cases where 'git repack -adf' outperformed 'git repack\n -adf --path-walk', I discovered that the issue did not reproduce in a\n bare repository. This is due to 'git repack' iterating over all indexed\n objects before walking commits. I had inadvertently put all indexed\n objects in their own category, leading to no good deltas with previous\n versions of those files; I had also not used the 'path' option from the\n pending list, so these objects had invalid name hash values. You will see\n in patch 4 that the pending list is handled quite differently and the\n '--indexed-objects' option is tested directly within t6601.\n\n * I added a new 'test_cmp_sorted' helper because I wanted to simplify some\n repeated sections of t6601.\n\n * Patch 1 has significantly more context than it did before.\n\n * Annotated tags are given a name of \"/tags\" to differentiate them slightly\n from root trees and commits.\n\n\nChanges in v2\n=============\n\n * Updated the test helper to output the batch number, allowing us to\n confirm that OIDs are grouped appropriately. This also signaled a few\n cases where the callback function was being called on an empty set.\n\n * This change has resulted in significant changes to the test data,\n including reordered lines and prepended batch numbers.\n\n * Thanks to Patrick for providing a recommended change to remove memory\n leaks from the test helper.\n\nThanks, -Stolee\n\nDerrick Stolee (6):\n path-walk: introduce an object walk by path\n test-lib-functions: add test_cmp_sorted\n t6601: add helper for testing path-walk API\n path-walk: allow consumer to specify object types\n path-walk: visit tags and cached objects\n path-walk: mark trees and blobs as UNINTERESTING\n\n Documentation/technical/api-path-walk.txt | 63 +++\n Makefile | 2 +\n path-walk.c | 531 ++++++++++++++++++++++\n path-walk.h | 65 +++\n t/helper/test-path-walk.c | 124 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 368 +++++++++++++++\n t/test-lib-functions.sh | 10 +\n 9 files changed, 1165 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\n\nbase-commit: e9356ba3ea2a6754281ff7697b3e5a1697b21e24\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1818%2Fderrickstolee%2Fapi-upstream-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1818/derrickstolee/api-upstream-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1818\n\nRange-diff vs v1:\n\n 1: c71f0a0e361 ! 1: b7e9b81e8b3 path-walk: introduce an object walk by path\n @@ path-walk.c (new)\n +\n +\tlist = strmap_get(&ctx->paths_to_lists, path);\n +\n ++\tif (!list->oids.nr)\n ++\t\treturn 0;\n ++\n +\t/* Evaluate function pointer on this data. */\n +\tret = ctx->info->path_fn(path, &list->oids, list->type,\n +\t\t\t\t ctx->info->path_fn_data);\n 2: 4f9f898fec1 = 2: cf2ed61b324 test-lib-functions: add test_cmp_sorted\n 3: 6f93dff88e7 ! 3: a3c754d93cc t6601: add helper for testing path-walk API\n @@ Commit message\n sets a baseline for the behavior and we can extend it as new options are\n introduced.\n \n + Store and output a 'batch_nr' value so we can demonstrate that the paths are\n + grouped together in a batch and not following some other ordering. This\n + allows us to test the depth-first behavior of the path-walk API. However, we\n + purposefully do not test the order of the objects in the batch, so the\n + output is compared to the expected output through a sort.\n +\n It is important to mention that the behavior of the API will change soon as\n we start to handle UNINTERESTING objects differently, but these tests will\n demonstrate the change in behavior.\n @@ t/helper/test-path-walk.c (new)\n +};\n +\n +struct path_walk_test_data {\n ++\tuintmax_t batch_nr;\n +\tuintmax_t tree_nr;\n +\tuintmax_t blob_nr;\n +};\n @@ t/helper/test-path-walk.c (new)\n +\t}\n +\n +\tfor (size_t i = 0; i < oids->nr; i++)\n -+\t\tprintf(\"%s:%s:%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]));\n ++\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n ++\t\t tdata->batch_nr, typestr, path,\n ++\t\t oid_to_hex(&oids->oid[i]));\n +\n ++\ttdata->batch_nr++;\n +\treturn 0;\n +}\n +\n @@ t/helper/test-path-walk.c (new)\n +\t\tOPT_END(),\n +\t};\n +\n -+\tinitialize_repository(the_repository);\n +\tsetup_git_directory();\n +\trevs.repo = the_repository;\n +\n @@ t/helper/test-path-walk.c (new)\n +\t \"blobs:%\" PRIuMAX \"\\n\",\n +\t data.tree_nr, data.blob_nr);\n +\n ++\trelease_revisions(&revs);\n +\treturn res;\n +}\n \n @@ t/t6601-path-walk.sh (new)\n @@\n +#!/bin/sh\n +\n ++TEST_PASSES_SANITIZE_LEAK=true\n ++\n +test_description='direct path-walk API tests'\n +\n +. ./test-lib.sh\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- --all >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE::$(git rev-parse base^{tree})\n -+\tTREE::$(git rev-parse base~1^{tree})\n -+\tTREE::$(git rev-parse base~2^{tree})\n -+\tTREE:left/:$(git rev-parse base:left)\n -+\tTREE:left/:$(git rev-parse base~2:left)\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\tTREE:right/:$(git rev-parse base~1:right)\n -+\tTREE:right/:$(git rev-parse base~2:right)\n -+\ttrees:9\n -+\tBLOB:a:$(git rev-parse base~2:a)\n -+\tBLOB:left/b:$(git rev-parse base~2:left/b)\n -+\tBLOB:left/b:$(git rev-parse base:left/b)\n -+\tBLOB:right/c:$(git rev-parse base~2:right/c)\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t0:TREE::$(git rev-parse topic^{tree})\n ++\t0:TREE::$(git rev-parse base^{tree})\n ++\t0:TREE::$(git rev-parse base~1^{tree})\n ++\t0:TREE::$(git rev-parse base~2^{tree})\n ++\t1:TREE:right/:$(git rev-parse topic:right)\n ++\t1:TREE:right/:$(git rev-parse base~1:right)\n ++\t1:TREE:right/:$(git rev-parse base~2:right)\n ++\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n ++\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t4:TREE:left/:$(git rev-parse base:left)\n ++\t4:TREE:left/:$(git rev-parse base~2:left)\n ++\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n ++\t5:BLOB:left/b:$(git rev-parse base:left/b)\n ++\t6:BLOB:a:$(git rev-parse base~2:a)\n +\tblobs:6\n ++\ttrees:9\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- topic >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE::$(git rev-parse base~1^{tree})\n -+\tTREE::$(git rev-parse base~2^{tree})\n -+\tTREE:left/:$(git rev-parse base~2:left)\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\tTREE:right/:$(git rev-parse base~1:right)\n -+\tTREE:right/:$(git rev-parse base~2:right)\n -+\ttrees:7\n -+\tBLOB:a:$(git rev-parse base~2:a)\n -+\tBLOB:left/b:$(git rev-parse base~2:left/b)\n -+\tBLOB:right/c:$(git rev-parse base~2:right/c)\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t0:TREE::$(git rev-parse topic^{tree})\n ++\t0:TREE::$(git rev-parse base~1^{tree})\n ++\t0:TREE::$(git rev-parse base~2^{tree})\n ++\t1:TREE:right/:$(git rev-parse topic:right)\n ++\t1:TREE:right/:$(git rev-parse base~1:right)\n ++\t1:TREE:right/:$(git rev-parse base~2:right)\n ++\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n ++\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t4:TREE:left/:$(git rev-parse base~2:left)\n ++\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n ++\t6:BLOB:a:$(git rev-parse base~2:a)\n +\tblobs:5\n ++\ttrees:7\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE:left/:$(git rev-parse topic:left)\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\ttrees:3\n -+\tBLOB:a:$(git rev-parse topic:a)\n -+\tBLOB:left/b:$(git rev-parse topic:left/b)\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse topic:right/d)\n ++\t0:TREE::$(git rev-parse topic^{tree})\n ++\t1:TREE:right/:$(git rev-parse topic:right)\n ++\t2:BLOB:right/d:$(git rev-parse topic:right/d)\n ++\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t4:TREE:left/:$(git rev-parse topic:left)\n ++\t5:BLOB:left/b:$(git rev-parse topic:left/b)\n ++\t6:BLOB:a:$(git rev-parse topic:a)\n +\tblobs:4\n ++\ttrees:3\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- --boundary topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE::$(git rev-parse base~1^{tree})\n -+\tTREE:left/:$(git rev-parse base~1:left)\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\tTREE:right/:$(git rev-parse base~1:right)\n -+\ttrees:5\n -+\tBLOB:a:$(git rev-parse base~1:a)\n -+\tBLOB:left/b:$(git rev-parse base~1:left/b)\n -+\tBLOB:right/c:$(git rev-parse base~1:right/c)\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t0:TREE::$(git rev-parse topic^{tree})\n ++\t0:TREE::$(git rev-parse base~1^{tree})\n ++\t1:TREE:right/:$(git rev-parse topic:right)\n ++\t1:TREE:right/:$(git rev-parse base~1:right)\n ++\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t3:BLOB:right/c:$(git rev-parse base~1:right/c)\n ++\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t4:TREE:left/:$(git rev-parse base~1:left)\n ++\t5:BLOB:left/b:$(git rev-parse base~1:left/b)\n ++\t6:BLOB:a:$(git rev-parse base~1:a)\n +\tblobs:5\n ++\ttrees:5\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n 4: f4bf8be30b5 ! 4: 83b746f569d path-walk: allow consumer to specify object types\n @@ path-walk.c: static int add_children(struct path_walk_context *ctx,\n \t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n \t\t\to = child ? &child->object : NULL;\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n - \n - \tlist = strmap_get(&ctx->paths_to_lists, path);\n + \tif (!list->oids.nr)\n + \t\treturn 0;\n \n -\t/* Evaluate function pointer on this data. */\n -\tret = ctx->info->path_fn(path, &list->oids, list->type,\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n +\t/* Track all commits. */\n -+\tif (info->commits)\n ++\tif (info->commits && commit_list->oids.nr)\n +\t\tret = info->path_fn(\"\", &commit_list->oids, OBJ_COMMIT,\n +\t\t\t\t info->path_fn_data);\n +\toid_array_clear(&commit_list->oids);\n @@ path-walk.h: struct path_walk_info {\n \n ## t/helper/test-path-walk.c ##\n @@ t/helper/test-path-walk.c: static const char * const path_walk_usage[] = {\n - };\n \n struct path_walk_test_data {\n + \tuintmax_t batch_nr;\n ++\n +\tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n -\t data.tree_nr, data.blob_nr);\n +\t data.commit_nr, data.tree_nr, data.blob_nr);\n \n + \trelease_revisions(&revs);\n \treturn res;\n - }\n \n ## t/t6601-path-walk.sh ##\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n \tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n -+\tCOMMIT::$(git rev-parse base)\n -+\tCOMMIT::$(git rev-parse base~1)\n -+\tCOMMIT::$(git rev-parse base~2)\n +-\t0:TREE::$(git rev-parse topic^{tree})\n +-\t0:TREE::$(git rev-parse base^{tree})\n +-\t0:TREE::$(git rev-parse base~1^{tree})\n +-\t0:TREE::$(git rev-parse base~2^{tree})\n +-\t1:TREE:right/:$(git rev-parse topic:right)\n +-\t1:TREE:right/:$(git rev-parse base~1:right)\n +-\t1:TREE:right/:$(git rev-parse base~2:right)\n +-\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n +-\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n +-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t4:TREE:left/:$(git rev-parse base:left)\n +-\t4:TREE:left/:$(git rev-parse base~2:left)\n +-\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n +-\t5:BLOB:left/b:$(git rev-parse base:left/b)\n +-\t6:BLOB:a:$(git rev-parse base~2:a)\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse base)\n ++\t0:COMMIT::$(git rev-parse base~1)\n ++\t0:COMMIT::$(git rev-parse base~2)\n ++\t1:TREE::$(git rev-parse topic^{tree})\n ++\t1:TREE::$(git rev-parse base^{tree})\n ++\t1:TREE::$(git rev-parse base~1^{tree})\n ++\t1:TREE::$(git rev-parse base~2^{tree})\n ++\t2:TREE:right/:$(git rev-parse topic:right)\n ++\t2:TREE:right/:$(git rev-parse base~1:right)\n ++\t2:TREE:right/:$(git rev-parse base~2:right)\n ++\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n ++\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t5:TREE:left/:$(git rev-parse base:left)\n ++\t5:TREE:left/:$(git rev-parse base~2:left)\n ++\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n ++\t6:BLOB:left/b:$(git rev-parse base:left/b)\n ++\t7:BLOB:a:$(git rev-parse base~2:a)\n + \tblobs:6\n +\tcommits:4\n - \tTREE::$(git rev-parse topic^{tree})\n - \tTREE::$(git rev-parse base^{tree})\n - \tTREE::$(git rev-parse base~1^{tree})\n + \ttrees:9\n + \tEOF\n + \n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n \ttest-tool path-walk -- topic >out &&\n \n \tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n -+\tCOMMIT::$(git rev-parse base~1)\n -+\tCOMMIT::$(git rev-parse base~2)\n +-\t0:TREE::$(git rev-parse topic^{tree})\n +-\t0:TREE::$(git rev-parse base~1^{tree})\n +-\t0:TREE::$(git rev-parse base~2^{tree})\n +-\t1:TREE:right/:$(git rev-parse topic:right)\n +-\t1:TREE:right/:$(git rev-parse base~1:right)\n +-\t1:TREE:right/:$(git rev-parse base~2:right)\n +-\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n +-\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n +-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t4:TREE:left/:$(git rev-parse base~2:left)\n +-\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n +-\t6:BLOB:a:$(git rev-parse base~2:a)\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse base~1)\n ++\t0:COMMIT::$(git rev-parse base~2)\n ++\t1:TREE::$(git rev-parse topic^{tree})\n ++\t1:TREE::$(git rev-parse base~1^{tree})\n ++\t1:TREE::$(git rev-parse base~2^{tree})\n ++\t2:TREE:right/:$(git rev-parse topic:right)\n ++\t2:TREE:right/:$(git rev-parse base~1:right)\n ++\t2:TREE:right/:$(git rev-parse base~2:right)\n ++\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n ++\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t5:TREE:left/:$(git rev-parse base~2:left)\n ++\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n ++\t7:BLOB:a:$(git rev-parse base~2:a)\n + \tblobs:5\n +\tcommits:3\n - \tTREE::$(git rev-parse topic^{tree})\n - \tTREE::$(git rev-parse base~1^{tree})\n - \tTREE::$(git rev-parse base~2^{tree})\n + \ttrees:7\n + \tEOF\n + \n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n \ttest-tool path-walk -- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t1:TREE::$(git rev-parse topic^{tree})\n ++\t2:TREE:right/:$(git rev-parse topic:right)\n ++\t3:BLOB:right/d:$(git rev-parse topic:right/d)\n ++\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t5:TREE:left/:$(git rev-parse topic:left)\n ++\t6:BLOB:left/b:$(git rev-parse topic:left/b)\n ++\t7:BLOB:a:$(git rev-parse topic:a)\n ++\tblobs:4\n +\tcommits:1\n - \tTREE::$(git rev-parse topic^{tree})\n - \tTREE:left/:$(git rev-parse topic:left)\n - \tTREE:right/:$(git rev-parse topic:right)\n -@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n - \ttest_cmp_sorted expect out\n - '\n - \n ++\ttrees:3\n ++\tEOF\n ++\n ++\ttest_cmp_sorted expect out\n ++'\n ++\n +test_expect_success 'topic, not base, only blobs' '\n +\ttest-tool path-walk --no-trees --no-commits \\\n +\t\t-- topic --not base >out &&\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\tcat >expect <<-EOF &&\n +\tcommits:0\n +\ttrees:0\n -+\tBLOB:a:$(git rev-parse topic:a)\n -+\tBLOB:left/b:$(git rev-parse topic:left/b)\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse topic:right/d)\n ++\t0:BLOB:right/d:$(git rev-parse topic:right/d)\n ++\t1:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t2:BLOB:left/b:$(git rev-parse topic:left/b)\n ++\t3:BLOB:a:$(git rev-parse topic:a)\n +\tblobs:4\n +\tEOF\n +\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\t\t-- topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse topic)\n +\tcommits:1\n +\ttrees:0\n +\tblobs:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\n +\tcat >expect <<-EOF &&\n +\tcommits:0\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE:left/:$(git rev-parse topic:left)\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\ttrees:3\n + \t0:TREE::$(git rev-parse topic^{tree})\n + \t1:TREE:right/:$(git rev-parse topic:right)\n +-\t2:BLOB:right/d:$(git rev-parse topic:right/d)\n +-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t4:TREE:left/:$(git rev-parse topic:left)\n +-\t5:BLOB:left/b:$(git rev-parse topic:left/b)\n +-\t6:BLOB:a:$(git rev-parse topic:a)\n +-\tblobs:4\n ++\t2:TREE:left/:$(git rev-parse topic:left)\n + \ttrees:3\n +\tblobs:0\n -+\tEOF\n -+\n -+\ttest_cmp_sorted expect out\n -+'\n -+\n - test_expect_success 'topic, not base, boundary' '\n + \tEOF\n + \n + \ttest_cmp_sorted expect out\n +@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \ttest-tool path-walk -- --boundary topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n -+\tCOMMIT::$(git rev-parse base~1)\n +-\t0:TREE::$(git rev-parse topic^{tree})\n +-\t0:TREE::$(git rev-parse base~1^{tree})\n +-\t1:TREE:right/:$(git rev-parse topic:right)\n +-\t1:TREE:right/:$(git rev-parse base~1:right)\n +-\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n +-\t3:BLOB:right/c:$(git rev-parse base~1:right/c)\n +-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t4:TREE:left/:$(git rev-parse base~1:left)\n +-\t5:BLOB:left/b:$(git rev-parse base~1:left/b)\n +-\t6:BLOB:a:$(git rev-parse base~1:a)\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse base~1)\n ++\t1:TREE::$(git rev-parse topic^{tree})\n ++\t1:TREE::$(git rev-parse base~1^{tree})\n ++\t2:TREE:right/:$(git rev-parse topic:right)\n ++\t2:TREE:right/:$(git rev-parse base~1:right)\n ++\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t4:BLOB:right/c:$(git rev-parse base~1:right/c)\n ++\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t5:TREE:left/:$(git rev-parse base~1:left)\n ++\t6:BLOB:left/b:$(git rev-parse base~1:left/b)\n ++\t7:BLOB:a:$(git rev-parse base~1:a)\n + \tblobs:5\n +\tcommits:2\n - \tTREE::$(git rev-parse topic^{tree})\n - \tTREE::$(git rev-parse base~1^{tree})\n - \tTREE:left/:$(git rev-parse base~1:left)\n + \ttrees:5\n + \tEOF\n + \n 5: 3dc27658526 ! 5: 97765aa04c2 path-walk: visit tags and cached objects\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n +\tif (!list)\n +\t\tBUG(\"provided path '%s' that had no associated list\", path);\n +\n + \tif (!list->oids.nr)\n + \t\treturn 0;\n + \n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n -\t (list->type == OBJ_BLOB && ctx->info->blobs))\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n +\t\tstruct hashmap_iter iter;\n +\t\tstruct strmap_entry *entry;\n +\n -+\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry) {\n ++\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry)\n +\t\t\tpush_to_stack(&ctx, entry->key);\n -+\t\t}\n +\n +\t\twhile (!ret && ctx.path_stack.nr) {\n +\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n @@ t/helper/test-path-walk.c: static int emit_block(const char *path, struct oid_ar\n \tdefault:\n \t\tBUG(\"we do not understand this type\");\n \t}\n + \n ++\t/* This should never be output during tests. */\n ++\tif (!oids->nr)\n ++\t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n ++\t\t tdata->batch_nr, typestr, path);\n ++\n + \tfor (size_t i = 0; i < oids->nr; i++)\n + \t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n + \t\t tdata->batch_nr, typestr, path,\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of blob objects\")),\n \t\tOPT_BOOL(0, \"commits\", &info.commits,\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n +\t \"tags:%\" PRIuMAX \"\\n\",\n +\t data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);\n \n + \trelease_revisions(&revs);\n \treturn res;\n - }\n \n ## t/t6601-path-walk.sh ##\n @@ t/t6601-path-walk.sh: test_description='direct path-walk API tests'\n @@ t/t6601-path-walk.sh: test_description='direct path-walk API tests'\n '\n \n test_expect_success 'all' '\n -@@ t/t6601-path-walk.sh: test_expect_success 'all' '\n - \tTREE::$(git rev-parse base^{tree})\n - \tTREE::$(git rev-parse base~1^{tree})\n - \tTREE::$(git rev-parse base~2^{tree})\n -+\tTREE::$(git rev-parse refs/tags/tree-tag^{})\n -+\tTREE::$(git rev-parse refs/tags/tree-tag2^{})\n -+\tTREE:a/:$(git rev-parse base:a)\n - \tTREE:left/:$(git rev-parse base:left)\n - \tTREE:left/:$(git rev-parse base~2:left)\n - \tTREE:right/:$(git rev-parse topic:right)\n - \tTREE:right/:$(git rev-parse base~1:right)\n - \tTREE:right/:$(git rev-parse base~2:right)\n --\ttrees:9\n -+\tTREE:child/:$(git rev-parse refs/tags/tree-tag^{}:child)\n -+\ttrees:13\n - \tBLOB:a:$(git rev-parse base~2:a)\n -+\tBLOB:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n - \tBLOB:left/b:$(git rev-parse base~2:left/b)\n - \tBLOB:left/b:$(git rev-parse base:left/b)\n - \tBLOB:right/c:$(git rev-parse base~2:right/c)\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n - \tBLOB:right/d:$(git rev-parse base~1:right/d)\n --\tblobs:6\n -+\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n -+\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n -+\tBLOB:child/file:$(git rev-parse refs/tags/tree-tag^{}:child/file)\n + \ttest-tool path-walk -- --all >out &&\n + \n ++\tcat >expect <<-EOF &&\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse base)\n ++\t0:COMMIT::$(git rev-parse base~1)\n ++\t0:COMMIT::$(git rev-parse base~2)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/first)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/second.1)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/second.2)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/third)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/fourth)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/tree-tag)\n ++\t1:TAG:/tags:$(git rev-parse refs/tags/blob-tag)\n ++\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n ++\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n ++\t3:TREE::$(git rev-parse topic^{tree})\n ++\t3:TREE::$(git rev-parse base^{tree})\n ++\t3:TREE::$(git rev-parse base~1^{tree})\n ++\t3:TREE::$(git rev-parse base~2^{tree})\n ++\t3:TREE::$(git rev-parse refs/tags/tree-tag^{})\n ++\t3:TREE::$(git rev-parse refs/tags/tree-tag2^{})\n ++\t4:BLOB:a:$(git rev-parse base~2:a)\n ++\t5:TREE:right/:$(git rev-parse topic:right)\n ++\t5:TREE:right/:$(git rev-parse base~1:right)\n ++\t5:TREE:right/:$(git rev-parse base~2:right)\n ++\t6:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t7:BLOB:right/c:$(git rev-parse base~2:right/c)\n ++\t7:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t8:TREE:left/:$(git rev-parse base:left)\n ++\t8:TREE:left/:$(git rev-parse base~2:left)\n ++\t9:BLOB:left/b:$(git rev-parse base~2:left/b)\n ++\t9:BLOB:left/b:$(git rev-parse base:left/b)\n ++\t10:TREE:a/:$(git rev-parse base:a)\n ++\t11:BLOB:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n ++\t12:TREE:child/:$(git rev-parse refs/tags/tree-tag:child)\n ++\t13:BLOB:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n +\tblobs:10\n -+\tTAG:/tags:$(git rev-parse refs/tags/first)\n -+\tTAG:/tags:$(git rev-parse refs/tags/second.1)\n -+\tTAG:/tags:$(git rev-parse refs/tags/second.2)\n -+\tTAG:/tags:$(git rev-parse refs/tags/third)\n -+\tTAG:/tags:$(git rev-parse refs/tags/fourth)\n -+\tTAG:/tags:$(git rev-parse refs/tags/tree-tag)\n -+\tTAG:/tags:$(git rev-parse refs/tags/blob-tag)\n ++\tcommits:4\n +\ttags:7\n ++\ttrees:13\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n +\ttest-tool path-walk -- --indexed-objects >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tcommits:0\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\ttrees:1\n -+\tBLOB:a:$(git rev-parse HEAD:a)\n -+\tBLOB:left/b:$(git rev-parse HEAD:left/b)\n -+\tBLOB:left/c:$(git rev-parse :left/c)\n -+\tBLOB:right/c:$(git rev-parse HEAD:right/c)\n -+\tBLOB:right/d:$(git rev-parse HEAD:right/d)\n ++\t0:BLOB:a:$(git rev-parse HEAD:a)\n ++\t1:BLOB:left/b:$(git rev-parse HEAD:left/b)\n ++\t2:BLOB:left/c:$(git rev-parse :left/c)\n ++\t3:BLOB:right/c:$(git rev-parse HEAD:right/c)\n ++\t4:BLOB:right/d:$(git rev-parse HEAD:right/d)\n ++\t5:TREE:right/:$(git rev-parse topic:right)\n +\tblobs:5\n ++\tcommits:0\n +\ttags:0\n ++\ttrees:1\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n +\n +\ttest-tool path-walk -- --indexed-objects --branches >out &&\n +\n -+\tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n -+\tCOMMIT::$(git rev-parse base)\n -+\tCOMMIT::$(git rev-parse base~1)\n -+\tCOMMIT::$(git rev-parse base~2)\n -+\tcommits:4\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE::$(git rev-parse base^{tree})\n -+\tTREE::$(git rev-parse base~1^{tree})\n -+\tTREE::$(git rev-parse base~2^{tree})\n -+\tTREE:a/:$(git rev-parse base:a)\n -+\tTREE:left/:$(git rev-parse base:left)\n -+\tTREE:left/:$(git rev-parse base~2:left)\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\tTREE:right/:$(git rev-parse base~1:right)\n -+\tTREE:right/:$(git rev-parse base~2:right)\n -+\ttrees:10\n -+\tBLOB:a:$(git rev-parse base~2:a)\n -+\tBLOB:left/b:$(git rev-parse base:left/b)\n -+\tBLOB:left/b:$(git rev-parse base~2:left/b)\n -+\tBLOB:right/c:$(git rev-parse base~2:right/c)\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d)\n -+\tBLOB:right/d:$(git rev-parse :right/d)\n + \tcat >expect <<-EOF &&\n + \t0:COMMIT::$(git rev-parse topic)\n + \t0:COMMIT::$(git rev-parse base)\n +@@ t/t6601-path-walk.sh: test_expect_success 'all' '\n + \t1:TREE::$(git rev-parse base^{tree})\n + \t1:TREE::$(git rev-parse base~1^{tree})\n + \t1:TREE::$(git rev-parse base~2^{tree})\n +-\t2:TREE:right/:$(git rev-parse topic:right)\n +-\t2:TREE:right/:$(git rev-parse base~1:right)\n +-\t2:TREE:right/:$(git rev-parse base~2:right)\n +-\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n +-\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n +-\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t5:TREE:left/:$(git rev-parse base:left)\n +-\t5:TREE:left/:$(git rev-parse base~2:left)\n +-\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n +-\t6:BLOB:left/b:$(git rev-parse base:left/b)\n +-\t7:BLOB:a:$(git rev-parse base~2:a)\n +-\tblobs:6\n ++\t2:BLOB:a:$(git rev-parse base~2:a)\n ++\t3:TREE:right/:$(git rev-parse topic:right)\n ++\t3:TREE:right/:$(git rev-parse base~1:right)\n ++\t3:TREE:right/:$(git rev-parse base~2:right)\n ++\t4:BLOB:right/d:$(git rev-parse base~1:right/d)\n ++\t4:BLOB:right/d:$(git rev-parse :right/d)\n ++\t5:BLOB:right/c:$(git rev-parse base~2:right/c)\n ++\t5:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t6:TREE:left/:$(git rev-parse base:left)\n ++\t6:TREE:left/:$(git rev-parse base~2:left)\n ++\t7:BLOB:left/b:$(git rev-parse base:left/b)\n ++\t7:BLOB:left/b:$(git rev-parse base~2:left/b)\n ++\t8:TREE:a/:$(git rev-parse refs/tags/third:a)\n +\tblobs:7\n + \tcommits:4\n +-\ttrees:9\n +\ttags:0\n ++\ttrees:10\n \tEOF\n \n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n - \tBLOB:right/d:$(git rev-parse base~1:right/d)\n + \t7:BLOB:a:$(git rev-parse base~2:a)\n \tblobs:5\n + \tcommits:3\n +\ttags:0\n + \ttrees:7\n \tEOF\n \n - \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n - \tBLOB:right/d:$(git rev-parse topic:right/d)\n + \t7:BLOB:a:$(git rev-parse topic:a)\n \tblobs:4\n + \tcommits:1\n +\ttags:0\n + \ttrees:3\n \tEOF\n \n - \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n - \tBLOB:right/d:$(git rev-parse topic:right/d)\n + \t\t-- topic --not base >out &&\n + \n + \tcat >expect <<-EOF &&\n +-\tcommits:0\n +-\ttrees:0\n + \t0:BLOB:right/d:$(git rev-parse topic:right/d)\n + \t1:BLOB:right/c:$(git rev-parse topic:right/c)\n + \t2:BLOB:left/b:$(git rev-parse topic:left/b)\n + \t3:BLOB:a:$(git rev-parse topic:a)\n \tblobs:4\n ++\tcommits:0\n +\ttags:0\n ++\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only commits' '\n + \tcat >expect <<-EOF &&\n + \t0:COMMIT::$(git rev-parse topic)\n \tcommits:1\n - \ttrees:0\n +-\ttrees:0\n \tblobs:0\n +\ttags:0\n ++\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n - \tTREE:right/:$(git rev-parse topic:right)\n - \ttrees:3\n + \t\t-- topic --not base >out &&\n + \n + \tcat >expect <<-EOF &&\n +-\tcommits:0\n + \t0:TREE::$(git rev-parse topic^{tree})\n + \t1:TREE:right/:$(git rev-parse topic:right)\n + \t2:TREE:left/:$(git rev-parse topic:left)\n +-\ttrees:3\n ++\tcommits:0\n \tblobs:0\n +\ttags:0\n ++\ttrees:3\n \tEOF\n \n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n - \tBLOB:right/d:$(git rev-parse base~1:right/d)\n + \t7:BLOB:a:$(git rev-parse base~1:a)\n \tblobs:5\n + \tcommits:2\n +\ttags:0\n + \ttrees:5\n \tEOF\n \n \ttest_cmp_sorted expect out\n 6: 0bb607e1fd3 ! 6: a4aaa3b001b path-walk: mark trees and blobs as UNINTERESTING\n @@ path-walk.c: static int add_children(struct path_walk_context *ctx,\n \t}\n \n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n - \tif (!list)\n - \t\tBUG(\"provided path '%s' that had no associated list\", path);\n + \tif (!list->oids.nr)\n + \t\treturn 0;\n \n +\tif (ctx->info->prune_all_uninteresting) {\n +\t\t/*\n @@ path-walk.h: struct path_walk_info {\n \n ## t/helper/test-path-walk.c ##\n @@ t/helper/test-path-walk.c: static int emit_block(const char *path, struct oid_array *oids,\n - \t\tBUG(\"we do not understand this type\");\n - \t}\n + \t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n + \t\t tdata->batch_nr, typestr, path);\n \n -\tfor (size_t i = 0; i < oids->nr; i++)\n --\t\tprintf(\"%s:%s:%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]));\n +-\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n +\tfor (size_t i = 0; i < oids->nr; i++) {\n +\t\tstruct object *o = lookup_unknown_object(the_repository,\n +\t\t\t\t\t\t\t &oids->oid[i]);\n -+\t\tprintf(\"%s:%s:%s%s\\n\", typestr, path, oid_to_hex(&oids->oid[i]),\n ++\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s%s\\n\",\n + \t\t tdata->batch_nr, typestr, path,\n +-\t\t oid_to_hex(&oids->oid[i]));\n ++\t\t oid_to_hex(&oids->oid[i]),\n +\t\t o->flags & UNINTERESTING ? \":UNINTERESTING\" : \"\");\n +\t}\n \n + \ttdata->batch_nr++;\n \treturn 0;\n - }\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n \n ## t/t6601-path-walk.sh ##\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n - \tCOMMIT::$(git rev-parse topic)\n - \tcommits:1\n - \tTREE::$(git rev-parse topic^{tree})\n --\tTREE:left/:$(git rev-parse topic:left)\n -+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n - \tTREE:right/:$(git rev-parse topic:right)\n - \ttrees:3\n --\tBLOB:a:$(git rev-parse topic:a)\n --\tBLOB:left/b:$(git rev-parse topic:left/b)\n -+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n -+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n --\tBLOB:right/d:$(git rev-parse topic:right/d)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n + \t0:COMMIT::$(git rev-parse topic)\n + \t1:TREE::$(git rev-parse topic^{tree})\n + \t2:TREE:right/:$(git rev-parse topic:right)\n +-\t3:BLOB:right/d:$(git rev-parse topic:right/d)\n ++\t3:BLOB:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n + \t4:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t5:TREE:left/:$(git rev-parse topic:left)\n +-\t6:BLOB:left/b:$(git rev-parse topic:left/b)\n +-\t7:BLOB:a:$(git rev-parse topic:a)\n ++\t5:TREE:left/:$(git rev-parse topic:left):UNINTERESTING\n ++\t6:BLOB:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n ++\t7:BLOB:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n + \tcommits:1\n \ttags:0\n - \tEOF\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n \ttest_cmp_sorted expect out\n '\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n -+\tcommits:1\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\ttrees:3\n -+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n -+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n -+\tBLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t1:TAG:/tags:$(git rev-parse fourth)\n ++\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n ++\t3:TREE::$(git rev-parse topic^{tree})\n ++\t4:TREE:right/:$(git rev-parse topic:right)\n ++\t5:BLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n ++\t6:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t7:TREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n ++\t8:BLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n ++\t9:BLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n +\tblobs:5\n -+\tTAG:/tags:$(git rev-parse fourth)\n ++\tcommits:1\n +\ttags:1\n ++\ttrees:3\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n test_expect_success 'topic, not base, only blobs' '\n \ttest-tool path-walk --no-trees --no-commits \\\n \t\t-- topic --not base >out &&\n -@@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n + \n \tcat >expect <<-EOF &&\n - \tcommits:0\n - \ttrees:0\n --\tBLOB:a:$(git rev-parse topic:a)\n --\tBLOB:left/b:$(git rev-parse topic:left/b)\n -+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n -+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n --\tBLOB:right/d:$(git rev-parse topic:right/d)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n +-\t0:BLOB:right/d:$(git rev-parse topic:right/d)\n ++\t0:BLOB:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n + \t1:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t2:BLOB:left/b:$(git rev-parse topic:left/b)\n +-\t3:BLOB:a:$(git rev-parse topic:a)\n ++\t2:BLOB:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n ++\t3:BLOB:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n + \tcommits:0\n \ttags:0\n - \tEOF\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n \tcat >expect <<-EOF &&\n + \t0:TREE::$(git rev-parse topic^{tree})\n + \t1:TREE:right/:$(git rev-parse topic:right)\n +-\t2:TREE:left/:$(git rev-parse topic:left)\n ++\t2:TREE:left/:$(git rev-parse topic:left):UNINTERESTING\n \tcommits:0\n - \tTREE::$(git rev-parse topic^{tree})\n --\tTREE:left/:$(git rev-parse topic:left)\n -+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n - \tTREE:right/:$(git rev-parse topic:right)\n - \ttrees:3\n \tblobs:0\n + \ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \n \tcat >expect <<-EOF &&\n - \tCOMMIT::$(git rev-parse topic)\n --\tCOMMIT::$(git rev-parse base~1)\n -+\tCOMMIT::$(git rev-parse base~1):UNINTERESTING\n - \tcommits:2\n - \tTREE::$(git rev-parse topic^{tree})\n --\tTREE::$(git rev-parse base~1^{tree})\n --\tTREE:left/:$(git rev-parse base~1:left)\n -+\tTREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n -+\tTREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n - \tTREE:right/:$(git rev-parse topic:right)\n --\tTREE:right/:$(git rev-parse base~1:right)\n -+\tTREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n - \ttrees:5\n --\tBLOB:a:$(git rev-parse base~1:a)\n --\tBLOB:left/b:$(git rev-parse base~1:left/b)\n --\tBLOB:right/c:$(git rev-parse base~1:right/c)\n -+\tBLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n -+\tBLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n -+\tBLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n - \tBLOB:right/c:$(git rev-parse topic:right/c)\n --\tBLOB:right/d:$(git rev-parse base~1:right/d)\n -+\tBLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n + \t0:COMMIT::$(git rev-parse topic)\n +-\t0:COMMIT::$(git rev-parse base~1)\n ++\t0:COMMIT::$(git rev-parse base~1):UNINTERESTING\n + \t1:TREE::$(git rev-parse topic^{tree})\n +-\t1:TREE::$(git rev-parse base~1^{tree})\n ++\t1:TREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n + \t2:TREE:right/:$(git rev-parse topic:right)\n +-\t2:TREE:right/:$(git rev-parse base~1:right)\n +-\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n +-\t4:BLOB:right/c:$(git rev-parse base~1:right/c)\n ++\t2:TREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n ++\t3:BLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n ++\t4:BLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n + \t4:BLOB:right/c:$(git rev-parse topic:right/c)\n +-\t5:TREE:left/:$(git rev-parse base~1:left)\n +-\t6:BLOB:left/b:$(git rev-parse base~1:left/b)\n +-\t7:BLOB:a:$(git rev-parse base~1:a)\n ++\t5:TREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n ++\t6:BLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n ++\t7:BLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n \tblobs:5\n + \tcommits:2\n \ttags:0\n - \tEOF\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \ttest_cmp_sorted expect out\n '\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n +\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\tCOMMIT::$(git rev-parse topic)\n -+\tCOMMIT::$(git rev-parse base~1):UNINTERESTING\n -+\tcommits:2\n -+\tTREE::$(git rev-parse topic^{tree})\n -+\tTREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n -+\tTREE:right/:$(git rev-parse topic:right)\n -+\tTREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n -+\ttrees:4\n -+\tBLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n -+\tBLOB:right/c:$(git rev-parse topic:right/c)\n ++\t0:COMMIT::$(git rev-parse topic)\n ++\t0:COMMIT::$(git rev-parse base~1):UNINTERESTING\n ++\t1:TREE::$(git rev-parse topic^{tree})\n ++\t1:TREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n ++\t2:TREE:right/:$(git rev-parse topic:right)\n ++\t2:TREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n ++\t3:BLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n ++\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n +\tblobs:2\n ++\tcommits:2\n +\ttags:0\n ++\ttrees:4\n +\tEOF\n +\n +\ttest_cmp_sorted expect out\n\n-- \ngitgitgadget\n"},{"id":"506917","messageId":"b7e9b81e8b32313f00d38257ba731e73d17224cb.1731181272.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] path-walk: introduce an object walk by path","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:07Z","receivedAt":"2024-11-09T19:41:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a few planned applications, introduce the most basic form\nof a path-walk API. It currently assumes that there are no UNINTERESTING\nobjects, and does not include any complicated filters. It calls a function\npointer on groups of tree and blob objects as grouped by path. This only\nincludes objects the first time they are discovered, so an object that\nappears at multiple paths will not be included in two batches.\n\nThese batches are collected in 'struct type_and_oid_list' objects, which\nstore an object type and an oid_array of objects.\n\nThe data structures are documented in 'struct path_walk_context', but in\nsummary the most important are:\n\n * 'paths_to_lists' is a strmap that connects a path to a\n type_and_oid_list for that path. To avoid conflicts in path names,\n we make sure that tree paths end in \"/\" (except the root path with\n is an empty string) and blob paths do not end in \"/\".\n\n * 'path_stack' is a string list that is added to in an append-only\n way. This stores the stack of our depth-first search on the heap\n instead of using recursion.\n\n * 'path_stack_pushed' is a strmap that stores path names that were\n already added to 'path_stack', to avoid repeating paths in the\n stack. Mostly, this saves us from quadratic lookups from doing\n unsorted checks into the string_list.\n\nThe coupling of 'path_stack' and 'path_stack_pushed' is protected by the\npush_to_stack() method. Call this instead of inserting into these\nstructures directly.\n\nThe walk_objects_by_path() method initializes these structures and\nstarts walking commits from the given rev_info struct. The commits are\nused to find the list of root trees which populate the start of our\ndepth-first search.\n\nThe core of our depth-first search is in a while loop that continues\nwhile we have not indicated an early exit and our 'path_stack' still has\nentries in it. The loop body pops a path off of the stack and \"visits\"\nthe path via the walk_path() method.\n\nThe walk_path() method gets the list of OIDs from the 'path_to_lists'\nstrmap and executes the callback method on that list with the given path\nand type. If the OIDs correspond to tree objects, then iterate over all\ntrees in the list and run add_children() to add the child objects to\ntheir own lists, adding new entries to the stack if necessary.\n\nIn testing, this depth-first search approach was the one that used the\nleast memory while iterating over the object lists. There is still a\nchance that repositories with too-wide path patterns could cause memory\npressure issues. Limiting the stack size could be done in the future by\nlimiting how many objects are being considered in-progress, or by\nvisiting blob paths earlier than trees.\n\nThere are many future adaptations that could be made, but they are left for\nfuture updates when consumers are ready to take advantage of those features.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 45 ++++\n Makefile | 1 +\n path-walk.c | 263 ++++++++++++++++++++++\n path-walk.h | 43 ++++\n 4 files changed, 352 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nnew file mode 100644\nindex 00000000000..c550c77ca30\n--- /dev/null\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -0,0 +1,45 @@\n+Path-Walk API\n+=============\n+\n+The path-walk API is used to walk reachable objects, but to visit objects\n+in batches based on a common path they appear in, or by type.\n+\n+For example, all reachable commits are visited in a group. All tags are\n+visited in a group. Then, all root trees are visited. At some point, all\n+blobs reachable via a path `my/dir/to/A` are visited. When there are\n+multiple paths possible to reach the same object, then only one of those\n+paths is used to visit the object.\n+\n+Basics\n+------\n+\n+To use the path-walk API, include `path-walk.h` and call\n+`walk_objects_by_path()` with a customized `path_walk_info` struct. The\n+struct is used to set all of the options for how the walk should proceed.\n+Let's dig into the different options and their use.\n+\n+`path_fn` and `path_fn_data`::\n+\tThe most important option is the `path_fn` option, which is a\n+\tfunction pointer to the callback that can execute logic on the\n+\tobject IDs for objects grouped by type and path. This function\n+\talso receives a `data` value that corresponds to the\n+\t`path_fn_data` member, for providing custom data structures to\n+\tthis callback function.\n+\n+`revs`::\n+\tTo configure the exact details of the reachable set of objects,\n+\tuse the `revs` member and initialize it using the revision\n+\tmachinery in `revision.h`. Initialize `revs` using calls such as\n+\t`setup_revisions()` or `parse_revision_opt()`. Do not call\n+\t`prepare_revision_walk()`, as that will be called within\n+\t`walk_objects_by_path()`.\n++\n+It is also important that you do not specify the `--objects` flag for the\n+`revs` struct. The revision walk should only be used to walk commits, and\n+the objects will be walked in a separate way based on those starting\n+commits.\n+\n+Examples\n+--------\n+\n+See example usages in future changes.\ndiff --git a/Makefile b/Makefile\nindex 7344a7f7257..d0d8d6888e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\n+LIB_OBJS += path-walk.o\n LIB_OBJS += pathspec.o\n LIB_OBJS += pkt-line.o\n LIB_OBJS += preload-index.o\ndiff --git a/path-walk.c b/path-walk.c\nnew file mode 100644\nindex 00000000000..24cf04c1e7d\n--- /dev/null\n+++ b/path-walk.c\n@@ -0,0 +1,263 @@\n+/*\n+ * path-walk.c: implementation for path-based walks of the object graph.\n+ */\n+#include \"git-compat-util.h\"\n+#include \"path-walk.h\"\n+#include \"blob.h\"\n+#include \"commit.h\"\n+#include \"dir.h\"\n+#include \"hashmap.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"oid-array.h\"\n+#include \"revision.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+#include \"trace2.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+\n+struct type_and_oid_list\n+{\n+\tenum object_type type;\n+\tstruct oid_array oids;\n+};\n+\n+#define TYPE_AND_OID_LIST_INIT { \\\n+\t.type = OBJ_NONE, \t \\\n+\t.oids = OID_ARRAY_INIT\t \\\n+}\n+\n+struct path_walk_context {\n+\t/**\n+\t * Repeats of data in 'struct path_walk_info' for\n+\t * access with fewer characters.\n+\t */\n+\tstruct repository *repo;\n+\tstruct rev_info *revs;\n+\tstruct path_walk_info *info;\n+\n+\t/**\n+\t * Map a path to a 'struct type_and_oid_list'\n+\t * containing the objects discovered at that\n+\t * path.\n+\t */\n+\tstruct strmap paths_to_lists;\n+\n+\t/**\n+\t * Store the current list of paths in a stack, to\n+\t * facilitate depth-first-search without recursion.\n+\t *\n+\t * Use path_stack_pushed to indicate whether a path\n+\t * was previously added to path_stack.\n+\t */\n+\tstruct string_list path_stack;\n+\tstruct strset path_stack_pushed;\n+};\n+\n+static void push_to_stack(struct path_walk_context *ctx,\n+\t\t\t const char *path)\n+{\n+\tif (strset_contains(&ctx->path_stack_pushed, path))\n+\t\treturn;\n+\n+\tstrset_add(&ctx->path_stack_pushed, path);\n+\tstring_list_append(&ctx->path_stack, path);\n+}\n+\n+static int add_children(struct path_walk_context *ctx,\n+\t\t\tconst char *base_path,\n+\t\t\tstruct object_id *oid)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t base_len;\n+\tstruct tree *tree = lookup_tree(ctx->repo, oid);\n+\n+\tif (!tree) {\n+\t\terror(_(\"failed to walk children of tree %s: not found\"),\n+\t\t oid_to_hex(oid));\n+\t\treturn -1;\n+\t} else if (parse_tree_gently(tree, 1)) {\n+\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n+\t}\n+\n+\tstrbuf_addstr(&path, base_path);\n+\tbase_len = path.len;\n+\n+\tparse_tree(tree);\n+\tinit_tree_desc(&desc, &tree->object.oid, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tstruct type_and_oid_list *list;\n+\t\tstruct object *o;\n+\t\t/* Not actually true, but we will ignore submodules later. */\n+\t\tenum object_type type = S_ISDIR(entry.mode) ? OBJ_TREE : OBJ_BLOB;\n+\n+\t\t/* Skip submodules. */\n+\t\tif (S_ISGITLINK(entry.mode))\n+\t\t\tcontinue;\n+\n+\t\tif (type == OBJ_TREE) {\n+\t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else if (type == OBJ_BLOB) {\n+\t\t\tstruct blob *child = lookup_blob(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else {\n+\t\t\t/* Wrong type? */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!o) /* report error?*/\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, base_len);\n+\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n+\n+\t\t/*\n+\t\t * Trees will end with \"/\" for concatenation and distinction\n+\t\t * from blobs at the same path.\n+\t\t */\n+\t\tif (type == OBJ_TREE)\n+\t\t\tstrbuf_addch(&path, '/');\n+\n+\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {\n+\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\tlist->type = type;\n+\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n+\t\t}\n+\t\tpush_to_stack(ctx, path.buf);\n+\n+\t\t/* Skip this object if already seen. */\n+\t\tif (o->flags & SEEN)\n+\t\t\tcontinue;\n+\t\to->flags |= SEEN;\n+\t\toid_array_append(&list->oids, &entry.oid);\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+\tstrbuf_release(&path);\n+\treturn 0;\n+}\n+\n+/*\n+ * For each path in paths_to_explore, walk the trees another level\n+ * and add any found blobs to the batch (but only if they exist and\n+ * haven't been added yet).\n+ */\n+static int walk_path(struct path_walk_context *ctx,\n+\t\t const char *path)\n+{\n+\tstruct type_and_oid_list *list;\n+\tint ret = 0;\n+\n+\tlist = strmap_get(&ctx->paths_to_lists, path);\n+\n+\tif (!list->oids.nr)\n+\t\treturn 0;\n+\n+\t/* Evaluate function pointer on this data. */\n+\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t ctx->info->path_fn_data);\n+\n+\t/* Expand data for children. */\n+\tif (list->type == OBJ_TREE) {\n+\t\tfor (size_t i = 0; i < list->oids.nr; i++) {\n+\t\t\tret |= add_children(ctx,\n+\t\t\t\t\t path,\n+\t\t\t\t\t &list->oids.oid[i]);\n+\t\t}\n+\t}\n+\n+\toid_array_clear(&list->oids);\n+\tstrmap_remove(&ctx->paths_to_lists, path, 1);\n+\treturn ret;\n+}\n+\n+static void clear_strmap(struct strmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\n+\thashmap_for_each_entry(&map->map, &iter, e, ent) {\n+\t\tstruct type_and_oid_list *list = e->value;\n+\t\toid_array_clear(&list->oids);\n+\t}\n+\tstrmap_clear(map, 1);\n+\tstrmap_init(map);\n+}\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info)\n+{\n+\tconst char *root_path = \"\";\n+\tint ret = 0;\n+\tsize_t commits_nr = 0, paths_nr = 0;\n+\tstruct commit *c;\n+\tstruct type_and_oid_list *root_tree_list;\n+\tstruct path_walk_context ctx = {\n+\t\t.repo = info->revs->repo,\n+\t\t.revs = info->revs,\n+\t\t.info = info,\n+\t\t.path_stack = STRING_LIST_INIT_DUP,\n+\t\t.path_stack_pushed = STRSET_INIT,\n+\t\t.paths_to_lists = STRMAP_INIT\n+\t};\n+\n+\ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\t/* Insert a single list for the root tree into the paths. */\n+\tCALLOC_ARRAY(root_tree_list, 1);\n+\troot_tree_list->type = OBJ_TREE;\n+\tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n+\tpush_to_stack(&ctx, root_path);\n+\n+\tif (prepare_revision_walk(info->revs))\n+\t\tdie(_(\"failed to setup revision walk\"));\n+\n+\twhile ((c = get_revision(info->revs))) {\n+\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct tree *t;\n+\t\tcommits_nr++;\n+\n+\t\toid = get_commit_tree_oid(c);\n+\t\tt = lookup_tree(info->revs->repo, oid);\n+\n+\t\tif (!t) {\n+\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (t->object.flags & SEEN)\n+\t\t\tcontinue;\n+\t\tt->object.flags |= SEEN;\n+\t\toid_array_append(&root_tree_list->oids, oid);\n+\t}\n+\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n+\ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n+\twhile (!ret && ctx.path_stack.nr) {\n+\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\tctx.path_stack.nr--;\n+\t\tpaths_nr++;\n+\n+\t\tret = walk_path(&ctx, path);\n+\n+\t\tfree(path);\n+\t}\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n+\ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n+\n+\tclear_strmap(&ctx.paths_to_lists);\n+\tstrset_clear(&ctx.path_stack_pushed);\n+\tstring_list_clear(&ctx.path_stack, 0);\n+\treturn ret;\n+}\ndiff --git a/path-walk.h b/path-walk.h\nnew file mode 100644\nindex 00000000000..c9e94a98bc8\n--- /dev/null\n+++ b/path-walk.h\n@@ -0,0 +1,43 @@\n+/*\n+ * path-walk.h : Methods and structures for walking the object graph in batches\n+ * by the paths that can reach those objects.\n+ */\n+#include \"object.h\" /* Required for 'enum object_type'. */\n+\n+struct rev_info;\n+struct oid_array;\n+\n+/**\n+ * The type of a function pointer for the method that is called on a list of\n+ * objects reachable at a given path.\n+ */\n+typedef int (*path_fn)(const char *path,\n+\t\t struct oid_array *oids,\n+\t\t enum object_type type,\n+\t\t void *data);\n+\n+struct path_walk_info {\n+\t/**\n+\t * revs provides the definitions for the commit walk, including\n+\t * which commits are UNINTERESTING or not.\n+\t */\n+\tstruct rev_info *revs;\n+\n+\t/**\n+\t * The caller wishes to execute custom logic on objects reachable at a\n+\t * given path. Every reachable object will be visited exactly once, and\n+\t * the first path to see an object wins. This may not be a stable choice.\n+\t */\n+\tpath_fn path_fn;\n+\tvoid *path_fn_data;\n+};\n+\n+#define PATH_WALK_INFO_INIT { 0 }\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info);\n-- \ngitgitgadget\n\n"},{"id":"506918","messageId":"cf2ed61b324b019f61afb1b66163b1284e675db2.1731181273.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] test-lib-functions: add test_cmp_sorted","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:08Z","receivedAt":"2024-11-09T19:41:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis test helper will be helpful to reduce repeated logic in\nt6601-path-walk.sh, but may be helpful elsewhere, too.\n\nSigned-off-by: Derrick Stolee \n---\n t/test-lib-functions.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex fde9bf54fc3..16b70aebd60 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1267,6 +1267,16 @@ test_cmp () {\n \teval \"$GIT_TEST_CMP\" '\"$@\"'\n }\n \n+# test_cmp_sorted runs test_cmp on sorted versions of the two\n+# input files. Uses \"$1.sorted\" and \"$2.sorted\" as temp files.\n+\n+test_cmp_sorted () {\n+\tsort <\"$1\" >\"$1.sorted\" &&\n+\tsort <\"$2\" >\"$2.sorted\" &&\n+\ttest_cmp \"$1.sorted\" \"$2.sorted\" &&\n+\trm \"$1.sorted\" \"$2.sorted\"\n+}\n+\n # Check that the given config key has the expected value.\n #\n # test_cmp_config [-C ] \n-- \ngitgitgadget\n\n"},{"id":"506919","messageId":"a3c754d93cc3ecfcb4d2764930873af0b611eda9.1731181273.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:09Z","receivedAt":"2024-11-09T19:41:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests based on the current behavior, doing interesting checks\nfor different sets of branches, ranges, and the --boundary option. This\nsets a baseline for the behavior and we can extend it as new options are\nintroduced.\n\nStore and output a 'batch_nr' value so we can demonstrate that the paths are\ngrouped together in a batch and not following some other ordering. This\nallows us to test the depth-first behavior of the path-walk API. However, we\npurposefully do not test the order of the objects in the batch, so the\noutput is compared to the expected output through a sort.\n\nIt is important to mention that the behavior of the API will change soon as\nwe start to handle UNINTERESTING objects differently, but these tests will\ndemonstrate the change in behavior.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 3 +-\n Makefile | 1 +\n t/helper/test-path-walk.c | 90 ++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 120 ++++++++++++++++++++++\n 6 files changed, 215 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex c550c77ca30..662162ec70b 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -42,4 +42,5 @@ commits.\n Examples\n --------\n \n-See example usages in future changes.\n+See example usages in:\n+\t`t/helper/test-path-walk.c`\ndiff --git a/Makefile b/Makefile\nindex d0d8d6888e3..50413d96492 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -818,6 +818,7 @@ TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n+TEST_BUILTINS_OBJS += test-path-walk.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\n TEST_BUILTINS_OBJS += test-proc-receive.o\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nnew file mode 100644\nindex 00000000000..aa468871079\n--- /dev/null\n+++ b/t/helper/test-path-walk.c\n@@ -0,0 +1,90 @@\n+#define USE_THE_REPOSITORY_VARIABLE\n+\n+#include \"test-tool.h\"\n+#include \"environment.h\"\n+#include \"hex.h\"\n+#include \"object-name.h\"\n+#include \"object.h\"\n+#include \"pretty.h\"\n+#include \"revision.h\"\n+#include \"setup.h\"\n+#include \"parse-options.h\"\n+#include \"path-walk.h\"\n+#include \"oid-array.h\"\n+\n+static const char * const path_walk_usage[] = {\n+\tN_(\"test-tool path-walk -- \"),\n+\tNULL\n+};\n+\n+struct path_walk_test_data {\n+\tuintmax_t batch_nr;\n+\tuintmax_t tree_nr;\n+\tuintmax_t blob_nr;\n+};\n+\n+static int emit_block(const char *path, struct oid_array *oids,\n+\t\t enum object_type type, void *data)\n+{\n+\tstruct path_walk_test_data *tdata = data;\n+\tconst char *typestr;\n+\n+\tswitch (type) {\n+\tcase OBJ_TREE:\n+\t\ttypestr = \"TREE\";\n+\t\ttdata->tree_nr += oids->nr;\n+\t\tbreak;\n+\n+\tcase OBJ_BLOB:\n+\t\ttypestr = \"BLOB\";\n+\t\ttdata->blob_nr += oids->nr;\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tBUG(\"we do not understand this type\");\n+\t}\n+\n+\tfor (size_t i = 0; i < oids->nr; i++)\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n+\t\t tdata->batch_nr, typestr, path,\n+\t\t oid_to_hex(&oids->oid[i]));\n+\n+\ttdata->batch_nr++;\n+\treturn 0;\n+}\n+\n+int cmd__path_walk(int argc, const char **argv)\n+{\n+\tint res;\n+\tstruct rev_info revs = REV_INFO_INIT;\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tstruct path_walk_test_data data = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tsetup_git_directory();\n+\trevs.repo = the_repository;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, path_walk_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT | PARSE_OPT_KEEP_ARGV0);\n+\n+\tif (argc > 1)\n+\t\tsetup_revisions(argc, argv, &revs, NULL);\n+\telse\n+\t\tusage(path_walk_usage[0]);\n+\n+\tinfo.revs = &revs;\n+\tinfo.path_fn = emit_block;\n+\tinfo.path_fn_data = &data;\n+\n+\tres = walk_objects_by_path(&info);\n+\n+\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\t \"blobs:%\" PRIuMAX \"\\n\",\n+\t data.tree_nr, data.blob_nr);\n+\n+\trelease_revisions(&revs);\n+\treturn res;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..43676e7b93a 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -52,6 +52,7 @@ static struct test_cmd cmds[] = {\n \t{ \"parse-subcommand\", cmd__parse_subcommand },\n \t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n+\t{ \"path-walk\", cmd__path_walk },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"proc-receive\", cmd__proc_receive },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..9cfc5da6e57 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__parse_pathspec_file(int argc, const char** argv);\n int cmd__parse_subcommand(int argc, const char **argv);\n int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n+int cmd__path_walk(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\n int cmd__proc_receive(int argc, const char **argv);\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nnew file mode 100755\nindex 00000000000..a3da55933f4\n--- /dev/null\n+++ b/t/t6601-path-walk.sh\n@@ -0,0 +1,120 @@\n+#!/bin/sh\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+\n+test_description='direct path-walk API tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repository' '\n+\tgit checkout -b base &&\n+\n+\tmkdir left &&\n+\tmkdir right &&\n+\techo a >a &&\n+\techo b >left/b &&\n+\techo c >right/c &&\n+\tgit add . &&\n+\tgit commit -m \"first\" &&\n+\n+\techo d >right/d &&\n+\tgit add right &&\n+\tgit commit -m \"second\" &&\n+\n+\techo bb >left/b &&\n+\tgit commit -a -m \"third\" &&\n+\n+\tgit checkout -b topic HEAD~1 &&\n+\techo cc >right/c &&\n+\tgit commit -a -m \"topic\"\n+'\n+\n+test_expect_success 'all' '\n+\ttest-tool path-walk -- --all >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:TREE::$(git rev-parse topic^{tree})\n+\t0:TREE::$(git rev-parse base^{tree})\n+\t0:TREE::$(git rev-parse base~1^{tree})\n+\t0:TREE::$(git rev-parse base~2^{tree})\n+\t1:TREE:right/:$(git rev-parse topic:right)\n+\t1:TREE:right/:$(git rev-parse base~1:right)\n+\t1:TREE:right/:$(git rev-parse base~2:right)\n+\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t4:TREE:left/:$(git rev-parse base:left)\n+\t4:TREE:left/:$(git rev-parse base~2:left)\n+\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n+\t5:BLOB:left/b:$(git rev-parse base:left/b)\n+\t6:BLOB:a:$(git rev-parse base~2:a)\n+\tblobs:6\n+\ttrees:9\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic only' '\n+\ttest-tool path-walk -- topic >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:TREE::$(git rev-parse topic^{tree})\n+\t0:TREE::$(git rev-parse base~1^{tree})\n+\t0:TREE::$(git rev-parse base~2^{tree})\n+\t1:TREE:right/:$(git rev-parse topic:right)\n+\t1:TREE:right/:$(git rev-parse base~1:right)\n+\t1:TREE:right/:$(git rev-parse base~2:right)\n+\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t4:TREE:left/:$(git rev-parse base~2:left)\n+\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n+\t6:BLOB:a:$(git rev-parse base~2:a)\n+\tblobs:5\n+\ttrees:7\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base' '\n+\ttest-tool path-walk -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:TREE::$(git rev-parse topic^{tree})\n+\t1:TREE:right/:$(git rev-parse topic:right)\n+\t2:BLOB:right/d:$(git rev-parse topic:right/d)\n+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t4:TREE:left/:$(git rev-parse topic:left)\n+\t5:BLOB:left/b:$(git rev-parse topic:left/b)\n+\t6:BLOB:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, boundary' '\n+\ttest-tool path-walk -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:TREE::$(git rev-parse topic^{tree})\n+\t0:TREE::$(git rev-parse base~1^{tree})\n+\t1:TREE:right/:$(git rev-parse topic:right)\n+\t1:TREE:right/:$(git rev-parse base~1:right)\n+\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t3:BLOB:right/c:$(git rev-parse base~1:right/c)\n+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t4:TREE:left/:$(git rev-parse base~1:left)\n+\t5:BLOB:left/b:$(git rev-parse base~1:left/b)\n+\t6:BLOB:a:$(git rev-parse base~1:a)\n+\tblobs:5\n+\ttrees:5\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"506920","messageId":"83b746f569df932452d8d102dc29dda0859c4ed2.1731181273.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] path-walk: allow consumer to specify object types","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:10Z","receivedAt":"2024-11-09T19:41:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe add the ability to filter the object types in the path-walk API so\nthe callback function is called fewer times.\n\nThis adds the ability to ask for the commits in a list, as well. We\nre-use the empty string for this set of objects because these are passed\ndirectly to the callback function instead of being part of the\n'path_stack'.\n\nFuture changes will add the ability to visit annotated tags.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 9 ++\n path-walk.c | 33 ++++-\n path-walk.h | 14 +-\n t/helper/test-path-walk.c | 18 ++-\n t/t6601-path-walk.sh | 149 +++++++++++++++-------\n 5 files changed, 173 insertions(+), 50 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 662162ec70b..dce553b6114 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,6 +39,15 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n+`commits`, `blobs`, `trees`::\n+\tBy default, these members are enabled and signal that the path-walk\n+\tAPI should call the `path_fn` on objects of these types. Specialized\n+\tapplications could disable some options to make it simpler to walk\n+\tthe objects or to have fewer calls to `path_fn`.\n++\n+While it is possible to walk only commits in this way, consumers would be\n+better off using the revision walk API instead.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex 24cf04c1e7d..2ca08402367 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -98,6 +98,10 @@ static int add_children(struct path_walk_context *ctx,\n \t\tif (S_ISGITLINK(entry.mode))\n \t\t\tcontinue;\n \n+\t\t/* If the caller doesn't want blobs, then don't bother. */\n+\t\tif (!ctx->info->blobs && type == OBJ_BLOB)\n+\t\t\tcontinue;\n+\n \t\tif (type == OBJ_TREE) {\n \t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n \t\t\to = child ? &child->object : NULL;\n@@ -157,9 +161,11 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n-\t/* Evaluate function pointer on this data. */\n-\tret = ctx->info->path_fn(path, &list->oids, list->type,\n-\t\t\t\t ctx->info->path_fn_data);\n+\t/* Evaluate function pointer on this data, if requested. */\n+\tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n+\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t\tctx->info->path_fn_data);\n \n \t/* Expand data for children. */\n \tif (list->type == OBJ_TREE) {\n@@ -201,6 +207,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n \tstruct type_and_oid_list *root_tree_list;\n+\tstruct type_and_oid_list *commit_list;\n \tstruct path_walk_context ctx = {\n \t\t.repo = info->revs->repo,\n \t\t.revs = info->revs,\n@@ -212,6 +219,9 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\tCALLOC_ARRAY(commit_list, 1);\n+\tcommit_list->type = OBJ_COMMIT;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n@@ -222,10 +232,18 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\tdie(_(\"failed to setup revision walk\"));\n \n \twhile ((c = get_revision(info->revs))) {\n-\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct object_id *oid;\n \t\tstruct tree *t;\n \t\tcommits_nr++;\n \n+\t\tif (info->commits)\n+\t\t\toid_array_append(&commit_list->oids,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t/* If we only care about commits, then skip trees. */\n+\t\tif (!info->trees && !info->blobs)\n+\t\t\tcontinue;\n+\n \t\toid = get_commit_tree_oid(c);\n \t\tt = lookup_tree(info->revs->repo, oid);\n \n@@ -243,6 +261,13 @@ int walk_objects_by_path(struct path_walk_info *info)\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n \ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\t/* Track all commits. */\n+\tif (info->commits && commit_list->oids.nr)\n+\t\tret = info->path_fn(\"\", &commit_list->oids, OBJ_COMMIT,\n+\t\t\t\t info->path_fn_data);\n+\toid_array_clear(&commit_list->oids);\n+\tfree(commit_list);\n+\n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n \twhile (!ret && ctx.path_stack.nr) {\n \t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\ndiff --git a/path-walk.h b/path-walk.h\nindex c9e94a98bc8..2d2afc29b47 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -30,9 +30,21 @@ struct path_walk_info {\n \t */\n \tpath_fn path_fn;\n \tvoid *path_fn_data;\n+\n+\t/**\n+\t * Initialize which object types the path_fn should be called on. This\n+\t * could also limit the walk to skip blobs if not set.\n+\t */\n+\tint commits;\n+\tint trees;\n+\tint blobs;\n };\n \n-#define PATH_WALK_INFO_INIT { 0 }\n+#define PATH_WALK_INFO_INIT { \\\n+\t.blobs = 1,\t\t\\\n+\t.trees = 1,\t\t\\\n+\t.commits = 1,\t\t\\\n+}\n \n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex aa468871079..2b7e6e98d18 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -19,6 +19,8 @@ static const char * const path_walk_usage[] = {\n \n struct path_walk_test_data {\n \tuintmax_t batch_nr;\n+\n+\tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n };\n@@ -30,6 +32,11 @@ static int emit_block(const char *path, struct oid_array *oids,\n \tconst char *typestr;\n \n \tswitch (type) {\n+\tcase OBJ_COMMIT:\n+\t\ttypestr = \"COMMIT\";\n+\t\ttdata->commit_nr += oids->nr;\n+\t\tbreak;\n+\n \tcase OBJ_TREE:\n \t\ttypestr = \"TREE\";\n \t\ttdata->tree_nr += oids->nr;\n@@ -60,6 +67,12 @@ int cmd__path_walk(int argc, const char **argv)\n \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n \tstruct path_walk_test_data data = { 0 };\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"blobs\", &info.blobs,\n+\t\t\t N_(\"toggle inclusion of blob objects\")),\n+\t\tOPT_BOOL(0, \"commits\", &info.commits,\n+\t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"trees\", &info.trees,\n+\t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -81,9 +94,10 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tres = walk_objects_by_path(&info);\n \n-\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\tprintf(\"commits:%\" PRIuMAX \"\\n\"\n+\t \"trees:%\" PRIuMAX \"\\n\"\n \t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.tree_nr, data.blob_nr);\n+\t data.commit_nr, data.tree_nr, data.blob_nr);\n \n \trelease_revisions(&revs);\n \treturn res;\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex a3da55933f4..dcd3c03a2e8 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -33,22 +33,27 @@ test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:TREE::$(git rev-parse topic^{tree})\n-\t0:TREE::$(git rev-parse base^{tree})\n-\t0:TREE::$(git rev-parse base~1^{tree})\n-\t0:TREE::$(git rev-parse base~2^{tree})\n-\t1:TREE:right/:$(git rev-parse topic:right)\n-\t1:TREE:right/:$(git rev-parse base~1:right)\n-\t1:TREE:right/:$(git rev-parse base~2:right)\n-\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n-\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t4:TREE:left/:$(git rev-parse base:left)\n-\t4:TREE:left/:$(git rev-parse base~2:left)\n-\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n-\t5:BLOB:left/b:$(git rev-parse base:left/b)\n-\t6:BLOB:a:$(git rev-parse base~2:a)\n+\t0:COMMIT::$(git rev-parse topic)\n+\t0:COMMIT::$(git rev-parse base)\n+\t0:COMMIT::$(git rev-parse base~1)\n+\t0:COMMIT::$(git rev-parse base~2)\n+\t1:TREE::$(git rev-parse topic^{tree})\n+\t1:TREE::$(git rev-parse base^{tree})\n+\t1:TREE::$(git rev-parse base~1^{tree})\n+\t1:TREE::$(git rev-parse base~2^{tree})\n+\t2:TREE:right/:$(git rev-parse topic:right)\n+\t2:TREE:right/:$(git rev-parse base~1:right)\n+\t2:TREE:right/:$(git rev-parse base~2:right)\n+\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t5:TREE:left/:$(git rev-parse base:left)\n+\t5:TREE:left/:$(git rev-parse base~2:left)\n+\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n+\t6:BLOB:left/b:$(git rev-parse base:left/b)\n+\t7:BLOB:a:$(git rev-parse base~2:a)\n \tblobs:6\n+\tcommits:4\n \ttrees:9\n \tEOF\n \n@@ -59,19 +64,23 @@ test_expect_success 'topic only' '\n \ttest-tool path-walk -- topic >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:TREE::$(git rev-parse topic^{tree})\n-\t0:TREE::$(git rev-parse base~1^{tree})\n-\t0:TREE::$(git rev-parse base~2^{tree})\n-\t1:TREE:right/:$(git rev-parse topic:right)\n-\t1:TREE:right/:$(git rev-parse base~1:right)\n-\t1:TREE:right/:$(git rev-parse base~2:right)\n-\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n-\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t4:TREE:left/:$(git rev-parse base~2:left)\n-\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n-\t6:BLOB:a:$(git rev-parse base~2:a)\n+\t0:COMMIT::$(git rev-parse topic)\n+\t0:COMMIT::$(git rev-parse base~1)\n+\t0:COMMIT::$(git rev-parse base~2)\n+\t1:TREE::$(git rev-parse topic^{tree})\n+\t1:TREE::$(git rev-parse base~1^{tree})\n+\t1:TREE::$(git rev-parse base~2^{tree})\n+\t2:TREE:right/:$(git rev-parse topic:right)\n+\t2:TREE:right/:$(git rev-parse base~1:right)\n+\t2:TREE:right/:$(git rev-parse base~2:right)\n+\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t5:TREE:left/:$(git rev-parse base~2:left)\n+\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n+\t7:BLOB:a:$(git rev-parse base~2:a)\n \tblobs:5\n+\tcommits:3\n \ttrees:7\n \tEOF\n \n@@ -82,15 +91,66 @@ test_expect_success 'topic, not base' '\n \ttest-tool path-walk -- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n+\t0:COMMIT::$(git rev-parse topic)\n+\t1:TREE::$(git rev-parse topic^{tree})\n+\t2:TREE:right/:$(git rev-parse topic:right)\n+\t3:BLOB:right/d:$(git rev-parse topic:right/d)\n+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t5:TREE:left/:$(git rev-parse topic:left)\n+\t6:BLOB:left/b:$(git rev-parse topic:left/b)\n+\t7:BLOB:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\tcommits:1\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only blobs' '\n+\ttest-tool path-walk --no-trees --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n+\ttrees:0\n+\t0:BLOB:right/d:$(git rev-parse topic:right/d)\n+\t1:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t2:BLOB:left/b:$(git rev-parse topic:left/b)\n+\t3:BLOB:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+# No, this doesn't make a lot of sense for the path-walk API,\n+# but it is possible to do.\n+test_expect_success 'topic, not base, only commits' '\n+\ttest-tool path-walk --no-blobs --no-trees \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:COMMIT::$(git rev-parse topic)\n+\tcommits:1\n+\ttrees:0\n+\tblobs:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only trees' '\n+\ttest-tool path-walk --no-blobs --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n \t0:TREE::$(git rev-parse topic^{tree})\n \t1:TREE:right/:$(git rev-parse topic:right)\n-\t2:BLOB:right/d:$(git rev-parse topic:right/d)\n-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t4:TREE:left/:$(git rev-parse topic:left)\n-\t5:BLOB:left/b:$(git rev-parse topic:left/b)\n-\t6:BLOB:a:$(git rev-parse topic:a)\n-\tblobs:4\n+\t2:TREE:left/:$(git rev-parse topic:left)\n \ttrees:3\n+\tblobs:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -100,17 +160,20 @@ test_expect_success 'topic, not base, boundary' '\n \ttest-tool path-walk -- --boundary topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:TREE::$(git rev-parse topic^{tree})\n-\t0:TREE::$(git rev-parse base~1^{tree})\n-\t1:TREE:right/:$(git rev-parse topic:right)\n-\t1:TREE:right/:$(git rev-parse base~1:right)\n-\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n-\t3:BLOB:right/c:$(git rev-parse base~1:right/c)\n-\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t4:TREE:left/:$(git rev-parse base~1:left)\n-\t5:BLOB:left/b:$(git rev-parse base~1:left/b)\n-\t6:BLOB:a:$(git rev-parse base~1:a)\n+\t0:COMMIT::$(git rev-parse topic)\n+\t0:COMMIT::$(git rev-parse base~1)\n+\t1:TREE::$(git rev-parse topic^{tree})\n+\t1:TREE::$(git rev-parse base~1^{tree})\n+\t2:TREE:right/:$(git rev-parse topic:right)\n+\t2:TREE:right/:$(git rev-parse base~1:right)\n+\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t4:BLOB:right/c:$(git rev-parse base~1:right/c)\n+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t5:TREE:left/:$(git rev-parse base~1:left)\n+\t6:BLOB:left/b:$(git rev-parse base~1:left/b)\n+\t7:BLOB:a:$(git rev-parse base~1:a)\n \tblobs:5\n+\tcommits:2\n \ttrees:5\n \tEOF\n \n-- \ngitgitgadget\n\n"},{"id":"506921","messageId":"97765aa04c2fe4a254a93814a6e0de14fa4f9149.1731181273.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] path-walk: visit tags and cached objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:11Z","receivedAt":"2024-11-09T19:41:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe rev_info that is specified for a path-walk traversal may specify\nvisiting tag refs (both lightweight and annotated) and also may specify\nindexed objects (blobs and trees). Update the path-walk API to walk\nthese objects as well.\n\nWhen walking tags, we need to peel the annotated objects until reaching\na non-tag object. If we reach a commit, then we can add it to the\npending objects to make sure we visit in the commit walk portion. If we\nreach a tree, then we will assume that it is a root tree. If we reach a\nblob, then we have no good path name and so add it to a new list of\n\"tagged blobs\".\n\nWhen the rev_info includes the \"--indexed-objects\" flag, then the\npending set includes blobs and trees found in the cache entries and\ncache-tree. The cache entries are usually blobs, though they could be\ntrees in the case of a sparse index. The cache-tree stores\npreviously-hashed tree objects but these are cleared out when staging\nobjects below those paths. We add tests that demonstrate this.\n\nThe indexed objects come with a non-NULL 'path' value in the pending\nitem. This allows us to prepopulate the 'path_to_lists' strmap with\nlists for these paths.\n\nThe tricky thing about this walk is that we will want to combine the\nindexed objects walk with the commit walk, especially in the future case\nof walking objects during a command like 'git repack'.\n\nWhenever possible, we want the objects from the index to be grouped with\nsimilar objects in history. We don't want to miss any paths that appear\nonly in the index and not in the commit history.\n\nThus, we need to be careful to let the path stack be populated initially\nwith only the root tree path (and possibly tags and tagged blobs) and go\nthrough the normal depth-first search. Afterwards, if there are other\npaths that are remaining in the paths_to_lists strmap, we should then\niterate through the stack and visit those objects recursively.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 2 +-\n path-walk.c | 174 +++++++++++++++++++-\n path-walk.h | 2 +\n t/helper/test-path-walk.c | 18 ++-\n t/t6601-path-walk.sh | 186 +++++++++++++++++++---\n 5 files changed, 356 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex dce553b6114..6022c381b7c 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,7 +39,7 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n-`commits`, `blobs`, `trees`::\n+`commits`, `blobs`, `trees`, `tags`::\n \tBy default, these members are enabled and signal that the path-walk\n \tAPI should call the `path_fn` on objects of these types. Specialized\n \tapplications could disable some options to make it simpler to walk\ndiff --git a/path-walk.c b/path-walk.c\nindex 2ca08402367..a1f539dcd46 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -13,10 +13,13 @@\n #include \"revision.h\"\n #include \"string-list.h\"\n #include \"strmap.h\"\n+#include \"tag.h\"\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n \n+static const char *root_path = \"\";\n+\n struct type_and_oid_list\n {\n \tenum object_type type;\n@@ -158,12 +161,16 @@ static int walk_path(struct path_walk_context *ctx,\n \n \tlist = strmap_get(&ctx->paths_to_lists, path);\n \n+\tif (!list)\n+\t\tBUG(\"provided path '%s' that had no associated list\", path);\n+\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n-\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n+\t (list->type == OBJ_TAG && ctx->info->tags))\n \t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n \t\t\t\t\tctx->info->path_fn_data);\n \n@@ -194,6 +201,134 @@ static void clear_strmap(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static void setup_pending_objects(struct path_walk_info *info,\n+\t\t\t\t struct path_walk_context *ctx)\n+{\n+\tstruct type_and_oid_list *tags = NULL;\n+\tstruct type_and_oid_list *tagged_blobs = NULL;\n+\tstruct type_and_oid_list *root_tree_list = NULL;\n+\n+\tif (info->tags)\n+\t\tCALLOC_ARRAY(tags, 1);\n+\tif (info->blobs)\n+\t\tCALLOC_ARRAY(tagged_blobs, 1);\n+\tif (info->trees)\n+\t\troot_tree_list = strmap_get(&ctx->paths_to_lists, root_path);\n+\n+\t/*\n+\t * Pending objects include:\n+\t * * Commits at branch tips.\n+\t * * Annotated tags at tag tips.\n+\t * * Any kind of object at lightweight tag tips.\n+\t * * Trees and blobs in the index (with an associated path).\n+\t */\n+\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n+\t\tstruct object *obj = pending->item;\n+\n+\t\t/* Commits will be picked up by revision walk. */\n+\t\tif (obj->type == OBJ_COMMIT)\n+\t\t\tcontinue;\n+\n+\t\t/* Navigate annotated tag object chains. */\n+\t\twhile (obj->type == OBJ_TAG) {\n+\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n+\t\t\tif (!tag)\n+\t\t\t\tbreak;\n+\t\t\tif (tag->object.flags & SEEN)\n+\t\t\t\tbreak;\n+\t\t\ttag->object.flags |= SEEN;\n+\n+\t\t\tif (tags)\n+\t\t\t\toid_array_append(&tags->oids, &obj->oid);\n+\t\t\tobj = tag->tagged;\n+\t\t}\n+\n+\t\tif (obj->type == OBJ_TAG)\n+\t\t\tcontinue;\n+\n+\t\t/* We are now at a non-tag object. */\n+\t\tif (obj->flags & SEEN)\n+\t\t\tcontinue;\n+\t\tobj->flags |= SEEN;\n+\n+\t\tswitch (obj->type) {\n+\t\tcase OBJ_TREE:\n+\t\t\tif (!info->trees)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = *pending->path ? xstrfmt(\"%s/\", pending->path)\n+\t\t\t\t\t\t\t : xstrdup(\"\");\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_TREE;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t\tfree(path);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&root_tree_list->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (!info->blobs)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = pending->path;\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_BLOB;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&tagged_blobs->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_COMMIT:\n+\t\t\t/* Make sure it is in the object walk */\n+\t\t\tif (obj != pending->item)\n+\t\t\t\tadd_pending_object(info->revs, obj, \"\");\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"should not see any other type here\");\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Add tag objects and tagged blobs if they exist.\n+\t */\n+\tif (tagged_blobs) {\n+\t\tif (tagged_blobs->oids.nr) {\n+\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n+\t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\tpush_to_stack(ctx, tagged_blob_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n+\t\t} else {\n+\t\t\toid_array_clear(&tagged_blobs->oids);\n+\t\t\tfree(tagged_blobs);\n+\t\t}\n+\t}\n+\tif (tags) {\n+\t\tif (tags->oids.nr) {\n+\t\t\tconst char *tag_path = \"/tags\";\n+\t\t\ttags->type = OBJ_TAG;\n+\t\t\tpush_to_stack(ctx, tag_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n+\t\t} else {\n+\t\t\toid_array_clear(&tags->oids);\n+\t\t\tfree(tags);\n+\t\t}\n+\t}\n+}\n+\n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\n * call 'info->path_fn' on each discovered path.\n@@ -202,7 +337,6 @@ static void clear_strmap(struct strmap *map)\n */\n int walk_objects_by_path(struct path_walk_info *info)\n {\n-\tconst char *root_path = \"\";\n \tint ret = 0;\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n@@ -222,15 +356,31 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(commit_list, 1);\n \tcommit_list->type = OBJ_COMMIT;\n \n+\tif (info->tags)\n+\t\tinfo->revs->tag_objects = 1;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n+\t/*\n+\t * Set these values before preparing the walk to catch\n+\t * lightweight tags pointing to non-commits and indexed objects.\n+\t */\n+\tinfo->revs->blob_objects = info->blobs;\n+\tinfo->revs->tree_objects = info->trees;\n+\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n+\n+\ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\tsetup_pending_objects(info, &ctx);\n+\ttrace2_region_leave(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\n \twhile ((c = get_revision(info->revs))) {\n \t\tstruct object_id *oid;\n \t\tstruct tree *t;\n@@ -278,6 +428,26 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \t\tfree(path);\n \t}\n+\n+\t/* Are there paths remaining? Likely they are from indexed objects. */\n+\tif (!strmap_empty(&ctx.paths_to_lists)) {\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *entry;\n+\n+\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry)\n+\t\t\tpush_to_stack(&ctx, entry->key);\n+\n+\t\twhile (!ret && ctx.path_stack.nr) {\n+\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\t\tctx.path_stack.nr--;\n+\t\t\tpaths_nr++;\n+\n+\t\t\tret = walk_path(&ctx, path);\n+\n+\t\t\tfree(path);\n+\t\t}\n+\t}\n+\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n \ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n \ndiff --git a/path-walk.h b/path-walk.h\nindex 2d2afc29b47..ca839f873e4 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -38,12 +38,14 @@ struct path_walk_info {\n \tint commits;\n \tint trees;\n \tint blobs;\n+\tint tags;\n };\n \n #define PATH_WALK_INFO_INIT { \\\n \t.blobs = 1,\t\t\\\n \t.trees = 1,\t\t\\\n \t.commits = 1,\t\t\\\n+\t.tags = 1,\t\t\\\n }\n \n /**\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 2b7e6e98d18..265bd0b443b 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -23,6 +23,7 @@ struct path_walk_test_data {\n \tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n+\tuintmax_t tag_nr;\n };\n \n static int emit_block(const char *path, struct oid_array *oids,\n@@ -47,10 +48,20 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->blob_nr += oids->nr;\n \t\tbreak;\n \n+\tcase OBJ_TAG:\n+\t\ttypestr = \"TAG\";\n+\t\ttdata->tag_nr += oids->nr;\n+\t\tbreak;\n+\n \tdefault:\n \t\tBUG(\"we do not understand this type\");\n \t}\n \n+\t/* This should never be output during tests. */\n+\tif (!oids->nr)\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n+\t\t tdata->batch_nr, typestr, path);\n+\n \tfor (size_t i = 0; i < oids->nr; i++)\n \t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n \t\t tdata->batch_nr, typestr, path,\n@@ -71,6 +82,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of blob objects\")),\n \t\tOPT_BOOL(0, \"commits\", &info.commits,\n \t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"tags\", &info.tags,\n+\t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n@@ -96,8 +109,9 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tprintf(\"commits:%\" PRIuMAX \"\\n\"\n \t \"trees:%\" PRIuMAX \"\\n\"\n-\t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.commit_nr, data.tree_nr, data.blob_nr);\n+\t \"blobs:%\" PRIuMAX \"\\n\"\n+\t \"tags:%\" PRIuMAX \"\\n\",\n+\t data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);\n \n \trelease_revisions(&revs);\n \treturn res;\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex dcd3c03a2e8..bf43ab0e22a 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -9,29 +9,142 @@ test_description='direct path-walk API tests'\n test_expect_success 'setup test repository' '\n \tgit checkout -b base &&\n \n+\t# Make some objects that will only be reachable\n+\t# via non-commit tags.\n+\tmkdir child &&\n+\techo file >child/file &&\n+\tgit add child &&\n+\tgit commit -m \"will abandon\" &&\n+\tgit tag -a -m \"tree\" tree-tag HEAD^{tree} &&\n+\techo file2 >file2 &&\n+\tgit add file2 &&\n+\tgit commit --amend -m \"will abandon\" &&\n+\tgit tag tree-tag2 HEAD^{tree} &&\n+\n+\techo blob >file &&\n+\tblob_oid=$(git hash-object -t blob -w --stdin file2 &&\n+\tblob2_oid=$(git hash-object -t blob -w --stdin a &&\n \techo b >left/b &&\n \techo c >right/c &&\n \tgit add . &&\n-\tgit commit -m \"first\" &&\n+\tgit commit --amend -m \"first\" &&\n+\tgit tag -m \"first\" first HEAD &&\n \n \techo d >right/d &&\n \tgit add right &&\n \tgit commit -m \"second\" &&\n+\tgit tag -a -m \"second (under)\" second.1 HEAD &&\n+\tgit tag -a -m \"second (top)\" second.2 second.1 &&\n \n+\t# Set up file/dir collision in history.\n+\trm a &&\n+\tmkdir a &&\n+\techo a >a/a &&\n \techo bb >left/b &&\n-\tgit commit -a -m \"third\" &&\n+\tgit add a left &&\n+\tgit commit -m \"third\" &&\n+\tgit tag -a -m \"third\" third &&\n \n \tgit checkout -b topic HEAD~1 &&\n \techo cc >right/c &&\n-\tgit commit -a -m \"topic\"\n+\tgit commit -a -m \"topic\" &&\n+\tgit tag -a -m \"fourth\" fourth\n '\n \n test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n+\tcat >expect <<-EOF &&\n+\t0:COMMIT::$(git rev-parse topic)\n+\t0:COMMIT::$(git rev-parse base)\n+\t0:COMMIT::$(git rev-parse base~1)\n+\t0:COMMIT::$(git rev-parse base~2)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/first)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/second.1)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/second.2)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/third)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/fourth)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/tree-tag)\n+\t1:TAG:/tags:$(git rev-parse refs/tags/blob-tag)\n+\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n+\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\t3:TREE::$(git rev-parse topic^{tree})\n+\t3:TREE::$(git rev-parse base^{tree})\n+\t3:TREE::$(git rev-parse base~1^{tree})\n+\t3:TREE::$(git rev-parse base~2^{tree})\n+\t3:TREE::$(git rev-parse refs/tags/tree-tag^{})\n+\t3:TREE::$(git rev-parse refs/tags/tree-tag2^{})\n+\t4:BLOB:a:$(git rev-parse base~2:a)\n+\t5:TREE:right/:$(git rev-parse topic:right)\n+\t5:TREE:right/:$(git rev-parse base~1:right)\n+\t5:TREE:right/:$(git rev-parse base~2:right)\n+\t6:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t7:BLOB:right/c:$(git rev-parse base~2:right/c)\n+\t7:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t8:TREE:left/:$(git rev-parse base:left)\n+\t8:TREE:left/:$(git rev-parse base~2:left)\n+\t9:BLOB:left/b:$(git rev-parse base~2:left/b)\n+\t9:BLOB:left/b:$(git rev-parse base:left/b)\n+\t10:TREE:a/:$(git rev-parse base:a)\n+\t11:BLOB:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n+\t12:TREE:child/:$(git rev-parse refs/tags/tree-tag:child)\n+\t13:BLOB:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\tblobs:10\n+\tcommits:4\n+\ttags:7\n+\ttrees:13\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'indexed objects' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"left\" directory.\n+\techo bogus >left/c &&\n+\tgit add left &&\n+\n+\ttest-tool path-walk -- --indexed-objects >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:BLOB:a:$(git rev-parse HEAD:a)\n+\t1:BLOB:left/b:$(git rev-parse HEAD:left/b)\n+\t2:BLOB:left/c:$(git rev-parse :left/c)\n+\t3:BLOB:right/c:$(git rev-parse HEAD:right/c)\n+\t4:BLOB:right/d:$(git rev-parse HEAD:right/d)\n+\t5:TREE:right/:$(git rev-parse topic:right)\n+\tblobs:5\n+\tcommits:0\n+\ttags:0\n+\ttrees:1\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'branches and indexed objects mix well' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"right\" directory.\n+\techo fake >right/d &&\n+\tgit add right &&\n+\n+\ttest-tool path-walk -- --indexed-objects --branches >out &&\n+\n \tcat >expect <<-EOF &&\n \t0:COMMIT::$(git rev-parse topic)\n \t0:COMMIT::$(git rev-parse base)\n@@ -41,20 +154,23 @@ test_expect_success 'all' '\n \t1:TREE::$(git rev-parse base^{tree})\n \t1:TREE::$(git rev-parse base~1^{tree})\n \t1:TREE::$(git rev-parse base~2^{tree})\n-\t2:TREE:right/:$(git rev-parse topic:right)\n-\t2:TREE:right/:$(git rev-parse base~1:right)\n-\t2:TREE:right/:$(git rev-parse base~2:right)\n-\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n-\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n-\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t5:TREE:left/:$(git rev-parse base:left)\n-\t5:TREE:left/:$(git rev-parse base~2:left)\n-\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n-\t6:BLOB:left/b:$(git rev-parse base:left/b)\n-\t7:BLOB:a:$(git rev-parse base~2:a)\n-\tblobs:6\n+\t2:BLOB:a:$(git rev-parse base~2:a)\n+\t3:TREE:right/:$(git rev-parse topic:right)\n+\t3:TREE:right/:$(git rev-parse base~1:right)\n+\t3:TREE:right/:$(git rev-parse base~2:right)\n+\t4:BLOB:right/d:$(git rev-parse base~1:right/d)\n+\t4:BLOB:right/d:$(git rev-parse :right/d)\n+\t5:BLOB:right/c:$(git rev-parse base~2:right/c)\n+\t5:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t6:TREE:left/:$(git rev-parse base:left)\n+\t6:TREE:left/:$(git rev-parse base~2:left)\n+\t7:BLOB:left/b:$(git rev-parse base:left/b)\n+\t7:BLOB:left/b:$(git rev-parse base~2:left/b)\n+\t8:TREE:a/:$(git rev-parse refs/tags/third:a)\n+\tblobs:7\n \tcommits:4\n-\ttrees:9\n+\ttags:0\n+\ttrees:10\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -81,6 +197,7 @@ test_expect_success 'topic only' '\n \t7:BLOB:a:$(git rev-parse base~2:a)\n \tblobs:5\n \tcommits:3\n+\ttags:0\n \ttrees:7\n \tEOF\n \n@@ -101,6 +218,7 @@ test_expect_success 'topic, not base' '\n \t7:BLOB:a:$(git rev-parse topic:a)\n \tblobs:4\n \tcommits:1\n+\ttags:0\n \ttrees:3\n \tEOF\n \n@@ -112,13 +230,14 @@ test_expect_success 'topic, not base, only blobs' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\tcommits:0\n-\ttrees:0\n \t0:BLOB:right/d:$(git rev-parse topic:right/d)\n \t1:BLOB:right/c:$(git rev-parse topic:right/c)\n \t2:BLOB:left/b:$(git rev-parse topic:left/b)\n \t3:BLOB:a:$(git rev-parse topic:a)\n \tblobs:4\n+\tcommits:0\n+\ttags:0\n+\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -133,8 +252,9 @@ test_expect_success 'topic, not base, only commits' '\n \tcat >expect <<-EOF &&\n \t0:COMMIT::$(git rev-parse topic)\n \tcommits:1\n-\ttrees:0\n \tblobs:0\n+\ttags:0\n+\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -145,12 +265,13 @@ test_expect_success 'topic, not base, only trees' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\tcommits:0\n \t0:TREE::$(git rev-parse topic^{tree})\n \t1:TREE:right/:$(git rev-parse topic:right)\n \t2:TREE:left/:$(git rev-parse topic:left)\n-\ttrees:3\n+\tcommits:0\n \tblobs:0\n+\ttags:0\n+\ttrees:3\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -174,10 +295,33 @@ test_expect_success 'topic, not base, boundary' '\n \t7:BLOB:a:$(git rev-parse base~1:a)\n \tblobs:5\n \tcommits:2\n+\ttags:0\n \ttrees:5\n \tEOF\n \n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'trees are reported exactly once' '\n+\ttest_when_finished \"rm -rf unique-trees\" &&\n+\ttest_create_repo unique-trees &&\n+\t(\n+\t\tcd unique-trees &&\n+\t\tmkdir initial &&\n+\t\ttest_commit initial/file &&\n+\n+\t\tgit switch -c move-to-top &&\n+\t\tgit mv initial/file.t ./ &&\n+\t\ttest_tick &&\n+\t\tgit commit -m moved &&\n+\n+\t\tgit update-ref refs/heads/other HEAD\n+\t) &&\n+\n+\ttest-tool -C unique-trees path-walk -- --all >out &&\n+\ttree=$(git -C unique-trees rev-parse HEAD:) &&\n+\tgrep \"$tree\" out >out-filtered &&\n+\ttest_line_count = 1 out-filtered\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"506922","messageId":"a4aaa3b001b75c19c96130f1c057157a29f9a7f5.1731181273.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] path-walk: mark trees and blobs as UNINTERESTING","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-09T19:41:12Z","receivedAt":"2024-11-09T19:41:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the input rev_info has UNINTERESTING starting points, we want to be\nsure that the UNINTERESTING flag is passed appropriately through the\nobjects. To match how this is done in places such as 'git pack-objects', we\nuse the mark_edges_uninteresting() method.\n\nThis method has an option for using the \"sparse\" walk, which is similar in\nspirit to the path-walk API's walk. To be sure to keep it independent, add a\nnew 'prune_all_uninteresting' option to the path_walk_info struct.\n\nTo check how the UNINTERSTING flag is spread through our objects, extend the\n'test-tool path-walk' command to output whether or not an object has that\nflag. This changes our tests significantly, including the removal of some\nobjects that were previously visited due to the incomplete implementation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 8 +++\n path-walk.c | 73 +++++++++++++++++++++\n path-walk.h | 8 +++\n t/helper/test-path-walk.c | 12 +++-\n t/t6601-path-walk.sh | 79 +++++++++++++++++------\n 5 files changed, 158 insertions(+), 22 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 6022c381b7c..7075d0d5ab5 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -48,6 +48,14 @@ commits.\n While it is possible to walk only commits in this way, consumers would be\n better off using the revision walk API instead.\n \n+`prune_all_uninteresting`::\n+\tBy default, all reachable paths are emitted by the path-walk API.\n+\tThis option allows consumers to declare that they are not\n+\tinterested in paths where all included objects are marked with the\n+\t`UNINTERESTING` flag. This requires using the `boundary` option in\n+\tthe revision walk so that the walk emits commits marked with the\n+\t`UNINTERESTING` flag.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex a1f539dcd46..896ec0c4779 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -8,6 +8,7 @@\n #include \"dir.h\"\n #include \"hashmap.h\"\n #include \"hex.h\"\n+#include \"list-objects.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n #include \"revision.h\"\n@@ -24,6 +25,7 @@ struct type_and_oid_list\n {\n \tenum object_type type;\n \tstruct oid_array oids;\n+\tint maybe_interesting;\n };\n \n #define TYPE_AND_OID_LIST_INIT { \\\n@@ -140,6 +142,9 @@ static int add_children(struct path_walk_context *ctx,\n \t\tif (o->flags & SEEN)\n \t\t\tcontinue;\n \t\to->flags |= SEEN;\n+\n+\t\tif (!(o->flags & UNINTERESTING))\n+\t\t\tlist->maybe_interesting = 1;\n \t\toid_array_append(&list->oids, &entry.oid);\n \t}\n \n@@ -167,6 +172,43 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n+\tif (ctx->info->prune_all_uninteresting) {\n+\t\t/*\n+\t\t * This is true if all objects were UNINTERESTING\n+\t\t * when added to the list.\n+\t\t */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\n+\t\t/*\n+\t\t * But it's still possible that the objects were set\n+\t\t * as UNINTERESTING after being added. Do a quick check.\n+\t\t */\n+\t\tlist->maybe_interesting = 0;\n+\t\tfor (size_t i = 0;\n+\t\t !list->maybe_interesting && i < list->oids.nr;\n+\t\t i++) {\n+\t\t\tif (list->type == OBJ_TREE) {\n+\t\t\t\tstruct tree *t = lookup_tree(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (t && !(t->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else if (list->type == OBJ_BLOB) {\n+\t\t\t\tstruct blob *b = lookup_blob(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (b && !(b->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else {\n+\t\t\t\t/* Tags are always interesting if visited. */\n+\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* We have confirmed that all objects are UNINTERESTING. */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\t}\n+\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n \t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n@@ -201,6 +243,26 @@ static void clear_strmap(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static struct repository *edge_repo;\n+static struct type_and_oid_list *edge_tree_list;\n+\n+static void show_edge(struct commit *commit)\n+{\n+\tstruct tree *t = repo_get_commit_tree(edge_repo, commit);\n+\n+\tif (!t)\n+\t\treturn;\n+\n+\tif (commit->object.flags & UNINTERESTING)\n+\t\tt->object.flags |= UNINTERESTING;\n+\n+\tif (t->object.flags & SEEN)\n+\t\treturn;\n+\tt->object.flags |= SEEN;\n+\n+\toid_array_append(&edge_tree_list->oids, &t->object.oid);\n+}\n+\n static void setup_pending_objects(struct path_walk_info *info,\n \t\t\t\t struct path_walk_context *ctx)\n {\n@@ -309,6 +371,7 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\tif (tagged_blobs->oids.nr) {\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\ttagged_blobs->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n \t\t} else {\n@@ -320,6 +383,7 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\tif (tags->oids.nr) {\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n+\t\t\ttags->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tag_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n \t\t} else {\n@@ -362,6 +426,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n+\troot_tree_list->maybe_interesting = 1;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n@@ -375,6 +440,14 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\t/* Walk trees to mark them as UNINTERESTING. */\n+\tedge_repo = info->revs->repo;\n+\tedge_tree_list = root_tree_list;\n+\tmark_edges_uninteresting(info->revs, show_edge,\n+\t\t\t\t info->prune_all_uninteresting);\n+\tedge_repo = NULL;\n+\tedge_tree_list = NULL;\n+\n \tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n \n \ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\ndiff --git a/path-walk.h b/path-walk.h\nindex ca839f873e4..de0db007dc9 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -39,6 +39,14 @@ struct path_walk_info {\n \tint trees;\n \tint blobs;\n \tint tags;\n+\n+\t/**\n+\t * When 'prune_all_uninteresting' is set and a path has all objects\n+\t * marked as UNINTERESTING, then the path-walk will not visit those\n+\t * objects. It will not call path_fn on those objects and will not\n+\t * walk the children of such trees.\n+\t */\n+\tint prune_all_uninteresting;\n };\n \n #define PATH_WALK_INFO_INIT { \\\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 265bd0b443b..7e791cfaf97 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -62,10 +62,14 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n \t\t tdata->batch_nr, typestr, path);\n \n-\tfor (size_t i = 0; i < oids->nr; i++)\n-\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tstruct object *o = lookup_unknown_object(the_repository,\n+\t\t\t\t\t\t\t &oids->oid[i]);\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s%s\\n\",\n \t\t tdata->batch_nr, typestr, path,\n-\t\t oid_to_hex(&oids->oid[i]));\n+\t\t oid_to_hex(&oids->oid[i]),\n+\t\t o->flags & UNINTERESTING ? \":UNINTERESTING\" : \"\");\n+\t}\n \n \ttdata->batch_nr++;\n \treturn 0;\n@@ -86,6 +90,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n+\t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n+\t\t\t N_(\"toggle pruning of uninteresting paths\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex bf43ab0e22a..d3c0015319a 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -211,11 +211,11 @@ test_expect_success 'topic, not base' '\n \t0:COMMIT::$(git rev-parse topic)\n \t1:TREE::$(git rev-parse topic^{tree})\n \t2:TREE:right/:$(git rev-parse topic:right)\n-\t3:BLOB:right/d:$(git rev-parse topic:right/d)\n+\t3:BLOB:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \t4:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t5:TREE:left/:$(git rev-parse topic:left)\n-\t6:BLOB:left/b:$(git rev-parse topic:left/b)\n-\t7:BLOB:a:$(git rev-parse topic:a)\n+\t5:TREE:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t6:BLOB:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t7:BLOB:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:1\n \ttags:0\n@@ -225,15 +225,38 @@ test_expect_success 'topic, not base' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'fourth, blob-tag2, not base' '\n+\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:COMMIT::$(git rev-parse topic)\n+\t1:TAG:/tags:$(git rev-parse fourth)\n+\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\t3:TREE::$(git rev-parse topic^{tree})\n+\t4:TREE:right/:$(git rev-parse topic:right)\n+\t5:BLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\t6:BLOB:right/c:$(git rev-parse topic:right/c)\n+\t7:TREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t8:BLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t9:BLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tblobs:5\n+\tcommits:1\n+\ttags:1\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'topic, not base, only blobs' '\n \ttest-tool path-walk --no-trees --no-commits \\\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:BLOB:right/d:$(git rev-parse topic:right/d)\n+\t0:BLOB:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \t1:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t2:BLOB:left/b:$(git rev-parse topic:left/b)\n-\t3:BLOB:a:$(git rev-parse topic:a)\n+\t2:BLOB:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t3:BLOB:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:0\n \ttags:0\n@@ -267,7 +290,7 @@ test_expect_success 'topic, not base, only trees' '\n \tcat >expect <<-EOF &&\n \t0:TREE::$(git rev-parse topic^{tree})\n \t1:TREE:right/:$(git rev-parse topic:right)\n-\t2:TREE:left/:$(git rev-parse topic:left)\n+\t2:TREE:left/:$(git rev-parse topic:left):UNINTERESTING\n \tcommits:0\n \tblobs:0\n \ttags:0\n@@ -282,17 +305,17 @@ test_expect_success 'topic, not base, boundary' '\n \n \tcat >expect <<-EOF &&\n \t0:COMMIT::$(git rev-parse topic)\n-\t0:COMMIT::$(git rev-parse base~1)\n+\t0:COMMIT::$(git rev-parse base~1):UNINTERESTING\n \t1:TREE::$(git rev-parse topic^{tree})\n-\t1:TREE::$(git rev-parse base~1^{tree})\n+\t1:TREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n \t2:TREE:right/:$(git rev-parse topic:right)\n-\t2:TREE:right/:$(git rev-parse base~1:right)\n-\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n-\t4:BLOB:right/c:$(git rev-parse base~1:right/c)\n+\t2:TREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t3:BLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\t4:BLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n \t4:BLOB:right/c:$(git rev-parse topic:right/c)\n-\t5:TREE:left/:$(git rev-parse base~1:left)\n-\t6:BLOB:left/b:$(git rev-parse base~1:left/b)\n-\t7:BLOB:a:$(git rev-parse base~1:a)\n+\t5:TREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t6:BLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t7:BLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n \tblobs:5\n \tcommits:2\n \ttags:0\n@@ -302,6 +325,27 @@ test_expect_success 'topic, not base, boundary' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, boundary with pruning' '\n+\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:COMMIT::$(git rev-parse topic)\n+\t0:COMMIT::$(git rev-parse base~1):UNINTERESTING\n+\t1:TREE::$(git rev-parse topic^{tree})\n+\t1:TREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n+\t2:TREE:right/:$(git rev-parse topic:right)\n+\t2:TREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t3:BLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\tcommits:2\n+\ttags:0\n+\ttrees:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n@@ -309,15 +353,12 @@ test_expect_success 'trees are reported exactly once' '\n \t\tcd unique-trees &&\n \t\tmkdir initial &&\n \t\ttest_commit initial/file &&\n-\n \t\tgit switch -c move-to-top &&\n \t\tgit mv initial/file.t ./ &&\n \t\ttest_tick &&\n \t\tgit commit -m moved &&\n-\n \t\tgit update-ref refs/heads/other HEAD\n \t) &&\n-\n \ttest-tool -C unique-trees path-walk -- --all >out &&\n \ttree=$(git -C unique-trees rev-parse HEAD:) &&\n \tgrep \"$tree\" out >out-filtered &&\n-- \ngitgitgadget\n"},{"id":"506959","messageId":"xmqqo72miim6.fsf@gitster.g","threadId":"62429","inReplyTo":"f02ee8ac-01e4-42e3-b99a-d9616b9ff1bb@gmail.com","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-11T02:56:01Z","receivedAt":"2024-11-11T02:56:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> Jeff King writes:\n>> \n>>> That's interesting. I wonder which cases get worse, and if a larger\n>>> window size might help. I.e., presumably we are pushing the candidates\n>>> further away in the sorted delta list.\n>\n> I think the cases that make things get worse with --full-name-hash are:\n>\n> 1. The presence of renames, partitioning objects that used to fit into\n> the same bucket (in the case of directory renames).\n>\n> 2. Some standard kinds of files may appear several times across the\n> tree but do not change very often and are similar across path.\n>\n> 3. Common patterns across similar file types, such as similar includes\n> in .c and .h files or other kinds of boilerplate in different\n> languages.\n> ...\n> In a depth 1 shallow clone, there are no repeated paths, so any hash\n> collisions are true collisions instead of good candidates for deltas.\n\nOr #2 and #3 above, where large boilerplates are shared across\nsimilarly named files.\n\n> Yes. This is the downside of the --full-name-hash compared to the\n> standard name hash. When repacking an entire repository, the effect\n> of these renames is typically not important in the long run as it's\n> basically a single break in the delta chain. The downside comes in\n> when doing a small fetch or push where the rename has more impact.\n\nYes. Due to --depth limit, we need to break delta chains somewhere\nanyway, and a rename boundary is just as good place as any other in\na sufficiently long chain.\n\n> The --path-walk approach does not suffer from this problem because\n> it has a second pass that sorts by the name hash and looks for\n> better deltas than the ones that already exist. Thus, it gets the\n> best of both worlds.\n\nYes, at the cost of being more complex :-)\n\n> The performance impact of the two passes of the --path-walk\n> approach is interesting, as you'd typically expect this to always\n> be slower. However:\n>\n> 1. The delta compression within each batch only compares the\n> objects within that batch. We do not compare these objects to\n> unrelated objects, which can be expensive and wasteful. This\n> also means that small batches may even be smaller than the\n> delta window, reducing the number of comparisons.\n\nInteresting.\n\n> 2. In the second pass, the delta calculation can short-circuit if\n> the computed delta would be larger than the current-best delta.\n> Thus, the good deltas from the first pass make the second pass\n> faster.\n\nYes, the early give-up codepath helps when you already found\nsomething semi-decently good delta base.\n\nThanks for a good summary.\n"},{"id":"507032","messageId":"db91e3ec-9c08-453f-bf76-4aa131f8b798@gmail.com","threadId":"62429","inReplyTo":"xmqqo72miim6.fsf@gitster.g","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-11T13:20:30Z","receivedAt":"2024-11-11T13:20:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/10/24 9:56 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n\n>> The --path-walk approach does not suffer from this problem because\n>> it has a second pass that sorts by the name hash and looks for\n>> better deltas than the ones that already exist. Thus, it gets the\n>> best of both worlds.\n> \n> Yes, at the cost of being more complex :-)\n\nTrue. I hope that the results sufficiently justify the complexity.\nFurther, the later uses of the path-walk API (git backfill and git\nsurvey) build upon this up-front investment.\n\nThanks,\n-Stolee\n\n"},{"id":"507078","messageId":"20241111215502.GC5019@coredump.intra.peff.net","threadId":"62429","inReplyTo":"xmqqo72miim6.fsf@gitster.g","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-11T21:55:02Z","receivedAt":"2024-11-11T21:55:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 11, 2024 at 11:56:01AM +0900, Junio C Hamano wrote:\n\n> > Yes. This is the downside of the --full-name-hash compared to the\n> > standard name hash. When repacking an entire repository, the effect\n> > of these renames is typically not important in the long run as it's\n> > basically a single break in the delta chain. The downside comes in\n> > when doing a small fetch or push where the rename has more impact.\n> \n> Yes. Due to --depth limit, we need to break delta chains somewhere\n> anyway, and a rename boundary is just as good place as any other in\n> a sufficiently long chain.\n\nWe don't necessarily have to break the chains due to depth limits,\nbecause they are not always linear. They can end up as bushy trees,\nlike:\n\n\n A <- B <- C\n \\\n <- D <- E\n \\\n <- F\n\nWe might get that way because it mirrors the shape of history (e.g., if\nD and E happened on a side branch from B and C). But we can also get\nthere from a linear history by choosing a delta that balances size\nversus depth. In the above example, the smallest sequence might be:\n\n A <- B <- C <- D <- E <- F\n\nbut if we have a limit of 3 and A-B-C already exists, then we might\nreject the C-D delta and choose A-D instead (or I guess if it really is\nlinear, probably B-D is more likely). That may be a larger delta by\nitself, but it is still better than storing a full copy of the object.\nAnd we find it by having those candidates close together in the sorted\nlist, reject C-D based on depth, and then moving on to the next\ncandidate.\n\n\nI'm not sure in practice how often we find these kinds of deltas. If you\nlook at, say, all the deltas for \"Makefile\" in git.git like this:\n\n git rev-list --objects --all --reflog --full-history -- Makefile |\n perl -lne 'print $1 if /(.*) Makefile/' |\n git cat-file --batch-check='%(objectsize:disk) %(objectname) %(deltabase)'\n\nmy repo has 33 full copies (you can see non-deltas by grepping for\n\"0{40}$\" in the output) out of 4473 total. So it's not like we never\nbreak chains. But we can use graphviz to visualize it by piping the\nabove through:\n\n perl -alne '\n BEGIN { print \"digraph {\" }\n print \"node_$F[1] [label=$F[0]]\";\n print \"node_$F[1] -> node_$F[2]\" if $F[2] !~ /^0+$/;\n END { print \"}\" }\n '\n\nand then piping it through \"dot\" to make an svg, or using an interactive\nviewer like \"xdot\" (the labels are the on-disk size of each object). I\nsee a lot of wide parts of the graph in the output.\n\nOf course this may all depend on packing patterns, too. I did my\ninvestigations after running \"git repack -adf\" to generate what should\nbe a pretty reasonable pack. You might see something different from\nincremental repacking over time.\n\n\nI'm not sure what any of this means for --path-walk, of course. ;)\nUltimately we care about resulting size and time to compute, so if it\ncan do better on those metrics then it doesn't matter what the graph\nlooks like. But maybe those tools can help us understand where things\nmight go better (or worse) with various approaches.\n\n-Peff\n"},{"id":"507079","messageId":"20241111220447.GD5019@coredump.intra.peff.net","threadId":"62429","inReplyTo":"f02ee8ac-01e4-42e3-b99a-d9616b9ff1bb@gmail.com","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-11T22:04:47Z","receivedAt":"2024-11-11T22:04:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 08, 2024 at 10:17:24AM -0500, Derrick Stolee wrote:\n\n> > > I'm not sure why a larger hash would be worse in a shallow clone. As you\n> > > note, with only one version of each path the name-similarity heuristic\n> > > is not likely to buy you much. But I'd have thought that would be true\n> > > for the existing name hash as well as a longer one. Maybe this is the\n> > > \"over-emphasizing\" case.\n> \n> I'm confused by your wording of \"larger hash\" because the hash size\n> is the exact same: 32 bits. It's just that the --full-name-hash option\n> has fewer collisions by abandoning the hope of locality.\n> \n> In a depth 1 shallow clone, there are no repeated paths, so any hash\n> collisions are true collisions instead of good candidates for deltas.\n> The full name hash is essentially random, so the delta compression\n> algorithm basically says:\n> \n> 1. Sort by type.\n> 2. Within each type, sort the objects randomly.\n> \n> With that sort, the delta compression scan is less effective than the\n> standard name hash.\n\nAh, OK. I'm sorry, I had not really investigated the full-name-hash and\nmisunderstood what it was doing. I thought we were using a larger hash\nin order to give more locality hints. I.e., to let us distinguish\n\"foo/long-path.c\" from \"bar/long-path.c\", but still retain some locality\nbetween them.\n\nBut it is throwing away all locality outside of the exact name. So yes,\nit would never find a rename from \"foo/\" to \"bar/\", as those mean the\nname-hash is effectively random.\n\nSo I guess getting back to what Taylor and I had talked about off-list:\nwe were wondering if there was a way to provide a better \"slider\" for\nlocality as part of the normal delta candidate sorting process. I think\nyou could store the full pathname and then doing a sort based on the\nreverse of the string (so \"foo.c\" and \"bar.c\" would compare \"c\", then\n\".\", etc). And that would let you tell the difference between\n\"foo/long-path.c\" and \"bar/long-path.c\" (preferring the identical\nfilenames over the merely similar ones), but still sort them together\nversus \"some-unrelated-path.c\".\n\nAnd what I wondered (and what I had initially thought full-name-hash was\ndoing) was whether we could store some fixed-size hash that produces a\nsimilar distribution (modulo collisions) to what that reverse-name sort\nwould do.\n\n-Peff\n"},{"id":"507084","messageId":"xmqq1pzhbe19.fsf@gitster.g","threadId":"62429","inReplyTo":"20241111215502.GC5019@coredump.intra.peff.net","subject":"Re: [PATCH 0/6] PATH WALK I: The path-walk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-11T22:29:06Z","receivedAt":"2024-11-11T22:29:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Yes. Due to --depth limit, we need to break delta chains somewhere\n>> anyway, and a rename boundary is just as good place as any other in\n>> a sufficiently long chain.\n>\n> We don't necessarily have to break the chains due to depth limits,\n> because they are not always linear. They can end up as bushy trees,\n\nTrue. And being able to pair blobs before and after a rename will\ngive us more candidates to place in a single bushy tree, so in that\nsense, with a short segment of history, it is understandable that\nthe full-name hash fails to have as many candidates as the original\nhash gives us. But with sufficiently large number of blobs at the\nsame path that are similar (i.e. not a \"pushing a short segment of\nhistory\", but an initial clone), splitting what could be one delta\nfamily into two delta families at the rename boundary is not too\nbad, as long as both halves have enough blobs to deltify against\neach other.\n\n> I'm not sure in practice how often we find these kinds of deltas. If you\n> look at, say, all the deltas for \"Makefile\" in git.git like this:\n>\n> git rev-list --objects --all --reflog --full-history -- Makefile |\n> perl -lne 'print $1 if /(.*) Makefile/' |\n> git cat-file --batch-check='%(objectsize:disk) %(objectname) %(deltabase)'\n>\n> my repo has 33 full copies (you can see non-deltas by grepping for\n> \"0{40}$\" in the output) out of 4473 total. So it's not like we never\n> break chains. But we can use graphviz to visualize it by piping the\n> above through:\n>\n> perl -alne '\n> BEGIN { print \"digraph {\" }\n> print \"node_$F[1] [label=$F[0]]\";\n> print \"node_$F[1] -> node_$F[2]\" if $F[2] !~ /^0+$/;\n> END { print \"}\" }\n> '\n>\n> and then piping it through \"dot\" to make an svg, or using an interactive\n> viewer like \"xdot\" (the labels are the on-disk size of each object). I\n> see a lot of wide parts of the graph in the output.\n>\n> Of course this may all depend on packing patterns, too. I did my\n> investigations after running \"git repack -adf\" to generate what should\n> be a pretty reasonable pack. You might see something different from\n> incremental repacking over time.\n\nThat is very true. I forgot that we do things to encourage bushy\ndelta-base selection. One thing I also am happy to see is the\neffect of our \"clever\" delta-base selection, where the algorithm\ndoes not blindly favor the delta-base that makes the resulting delta\nabsolutely minimal, but takes the depth of the delta-base into\naccount (i.e. a base at a much shallower depth is preferred over a\nbase near the depth limit, even if it results in a slightly larger\ndelta data).\n\n> I'm not sure what any of this means for --path-walk, of course. ;)\n> Ultimately we care about resulting size and time to compute, so if it\n> can do better on those metrics then it doesn't matter what the graph\n> looks like.\n\nTrue, too. Another thing that we care about is the time to access\ndata, and favoring shallow delta chain, even with the help of the\nin-core delta-base cache, has merit.\n\nThanks.\n"},{"id":"507849","messageId":"Zz+2hWlYCo6jlvj2@nand.local","threadId":"62429","inReplyTo":"a3c754d93cc3ecfcb4d2764930873af0b611eda9.1731181273.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/6] t6601: add helper for testing path-walk API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T22:39:01Z","receivedAt":"2024-11-21T22:39:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Nov 09, 2024 at 07:41:09PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> Add some tests based on the current behavior, doing interesting checks\n> for different sets of branches, ranges, and the --boundary option. This\n> sets a baseline for the behavior and we can extend it as new options are\n> introduced.\n>\n> Store and output a 'batch_nr' value so we can demonstrate that the paths are\n> grouped together in a batch and not following some other ordering. This\n> allows us to test the depth-first behavior of the path-walk API. However, we\n> purposefully do not test the order of the objects in the batch, so the\n> output is compared to the expected output through a sort.\n>\n> It is important to mention that the behavior of the API will change soon as\n> we start to handle UNINTERESTING objects differently, but these tests will\n> demonstrate the change in behavior.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n\nNice. I like the approach of implementing the API in a single commit,\nand then demonstrating a trivial \"caller\" by way of a custom test\nhelper. I think that the artifact of having a test helper here is useful\non its own, but it also serves as a good example of how to use the API,\nand provides something to actually test the implementation with.\n\nI'm going to steal this pattern the next time I need to work on\nsomething that necessitates a complex new API ;-).\n\n> +static int emit_block(const char *path, struct oid_array *oids,\n> +\t\t enum object_type type, void *data)\n> +{\n> +\tstruct path_walk_test_data *tdata = data;\n> +\tconst char *typestr;\n> +\n> +\tswitch (type) {\n> +\tcase OBJ_TREE:\n> +\t\ttypestr = \"TREE\";\n> +\t\ttdata->tree_nr += oids->nr;\n> +\t\tbreak;\n> +\n> +\tcase OBJ_BLOB:\n> +\t\ttypestr = \"BLOB\";\n> +\t\ttdata->blob_nr += oids->nr;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tBUG(\"we do not understand this type\");\n> +\t}\n\nI think you could write this as:\n\n if (type == OBJ_TREE)\n tdata->tree_nr += oids->nr;\n else if (type == OBJ_BLOB)\n tdata->blob_nr += oids->nr;\n else\n BUG(\"we do not understand this type\");\n\n typestr = type_name(type);\n\nWhich DRYs things up a bit and uses the type_name() helper. That will\ngive you strings like \"tree\" and \"blob\" instead of \"TREE\" and \"BLOB\",\nbut I'm not sure if the casing is important.\n\nThanks,\nTaylor\n"},{"id":"507850","messageId":"Zz+3whunQkfLX0A1@nand.local","threadId":"62429","inReplyTo":"83b746f569df932452d8d102dc29dda0859c4ed2.1731181273.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/6] path-walk: allow consumer to specify object types","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T22:44:18Z","receivedAt":"2024-11-21T22:44:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Nov 09, 2024 at 07:41:10PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/path-walk.c b/path-walk.c\n> index 24cf04c1e7d..2ca08402367 100644\n> --- a/path-walk.c\n> +++ b/path-walk.c\n> @@ -98,6 +98,10 @@ static int add_children(struct path_walk_context *ctx,\n> \t\tif (S_ISGITLINK(entry.mode))\n> \t\t\tcontinue;\n>\n> +\t\t/* If the caller doesn't want blobs, then don't bother. */\n> +\t\tif (!ctx->info->blobs && type == OBJ_BLOB)\n> +\t\t\tcontinue;\n> +\n\nI was going to ask why we're not reusing the existing property of the\nrev_info struct to specify what object type(s) we do/don't want, but the\nanswer is obvious: we don't want that to change the behavior of the\ncommit-level walk which is used to seed the actual path walk based on\nits results.\n\nHowever, it would be kind of nice to have a single place to specify how\nyou want to traverse objects, and using the rev_info struct seems like a\ngood choice there since you can naively ask Git to parse command-line\narguments like --blobs, --trees, --objects, etc. and set the appropriate\nbits.\n\nI wonder if it might make sense to do something like:\n\n unsigned int tmp_blobs = revs->blob_objects;\n unsigned int tmp_trees = revs->tree_objects;\n unsigned int tmp_tags = revs->tag_objects;\n\n if (prepare_revision_walk(revs))\n die(_(\"failed to setup revision walk\"));\n\n /* commit-level walk */\n\n revs->blob_objects = tmp_blobs;\n revs->tree_objects = tmp_trees;\n revs->tag_objects = tmp_tags;\n\n /* path-level walk */\n\nI don't have strong feelings about it, but it feels like this approach\ncould be cleaner from a caller's perspective. But I can see an argument\nto the contrary that it does introduce some awkwardness with the\ndual-use of those fields.\n\nThanks,\nTaylor\n"},{"id":"507852","messageId":"Zz+61fat+vGgb+xL@nand.local","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/6] PATH WALK I: The path-walk API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T22:57:25Z","receivedAt":"2024-11-21T22:57:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Nov 09, 2024 at 07:41:06PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Derrick Stolee (6):\n> path-walk: introduce an object walk by path\n> test-lib-functions: add test_cmp_sorted\n> t6601: add helper for testing path-walk API\n> path-walk: allow consumer to specify object types\n> path-walk: visit tags and cached objects\n> path-walk: mark trees and blobs as UNINTERESTING\n\nMy apologies for taking so long to review this. Having rad through the\npatches in detail, a couple of thoughts:\n\n - First, I like the structure that you decided on for this series. It\n nicely demonstrates a minimal caller for this new API instead of\n implementing a bunch of untested code. I think that's a great way to\n lay out things up until this point.\n\n - Second, I read through the existing API and only had minor comments.\n I read through the implementation in detail and found it to match my\n expectation of how each step should function.\n\nSo my take-away from spending a few hours with this series is that\neverything seems on track so far, and I think this is in a good spot to\nbuild on for more path-walk features.\n\nThat all said, I am still not totally sold on the idea that we need a\nseparate path-based traversal given the significant benefits of the\nfull-name hash approach that I reviewed earlier today.\n\nTo be clear, I am totally willing to believe that there are some\nbenefits to doing the path-walk approach on top, but I think we should\nconsider those benefits relative to the large amount of highly\nnon-trivial code that we're adding in order to power it.\n\nSo I'm not strongly opposed or in favor of the approach pursued starting\nin this series, I just think that it's worth spending time as a group\n(beyond just you and I) considering the benefits and costs associated\nwith it.\n\nAs for the patches themselves, I think that cooking them for a long time\nin 'next' makes most sense. We will want to land this patch series if\nand only if we decide that the traversal powered by this API is the\nright approach. IOW, I don't think it makes sense to have the path-walk\nstuff in the tree if it has no callers outside of the test helper\nprovided by this series.\n\nOK. I think that's a good stopping point for me on the list today, and I\nlook forward to your responses :-).\n\nThanks,\nTaylor\n"},{"id":"508057","messageId":"Z0Q7oGF6Q5U-f4VX@pks.im","threadId":"62429","inReplyTo":"Zz+61fat+vGgb+xL@nand.local","subject":"Re: [PATCH v2 0/6] PATH WALK I: The path-walk API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T08:56:09Z","receivedAt":"2024-11-25T08:56:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 21, 2024 at 05:57:25PM -0500, Taylor Blau wrote:\n> On Sat, Nov 09, 2024 at 07:41:06PM +0000, Derrick Stolee via GitGitGadget wrote:\n> > Derrick Stolee (6):\n> > path-walk: introduce an object walk by path\n> > test-lib-functions: add test_cmp_sorted\n> > t6601: add helper for testing path-walk API\n> > path-walk: allow consumer to specify object types\n> > path-walk: visit tags and cached objects\n> > path-walk: mark trees and blobs as UNINTERESTING\n> \n> My apologies for taking so long to review this. Having rad through the\n> patches in detail, a couple of thoughts:\n> \n> - First, I like the structure that you decided on for this series. It\n> nicely demonstrates a minimal caller for this new API instead of\n> implementing a bunch of untested code. I think that's a great way to\n> lay out things up until this point.\n> \n> - Second, I read through the existing API and only had minor comments.\n> I read through the implementation in detail and found it to match my\n> expectation of how each step should function.\n> \n> So my take-away from spending a few hours with this series is that\n> everything seems on track so far, and I think this is in a good spot to\n> build on for more path-walk features.\n> \n> That all said, I am still not totally sold on the idea that we need a\n> separate path-based traversal given the significant benefits of the\n> full-name hash approach that I reviewed earlier today.\n\nThe repo size reductions achieved via the path-walk API was only one of\nthe selling points of this series. And from my current understanding we\nwill likely not end up realizing those gains via path-walk, but rather\nvia the much simpler full-name hash algorithm indeed.\n\nBut there were two more selling points:\n\n - git-survey(1) as a native replacement for git-sizer(1). I think it\n is a great idea to have a native tool that allows us to gain deep\n insights into a repository so that we get better signals from our\n users in case they face problems with their repository. I'd love to\n have this tool as a baseline for an extensible format where we can\n eventually also start reporting the health state of refs as well as\n any auxiliary data structures.\n\n - git-backfill(1) as a helper to fetch blobs more efficiently from a\n promisor remote. This is a boon to have as well in our odyssey\n towards a better UI/UX with huge monorepos.\n\nBoth of these tools are quite exciting to me, and there is a need for\nhaving such tools from my point of view.\n\nThe question of course is whether these tools require the path-walk API,\nor whether they could be built on top of existing functionality. But if\nthere are good reasons why the existing functionality is insufficient\nthen I'd be all for having the path-walk API, even if it doesn't help us\nwith repo size reductions as we initially thought.\n\nPatrick\n"},{"id":"508154","messageId":"xmqqa5dmzboa.fsf@gitster.g","threadId":"62429","inReplyTo":"Z0Q7oGF6Q5U-f4VX@pks.im","subject":"Re: [PATCH v2 0/6] PATH WALK I: The path-walk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-26T07:39:17Z","receivedAt":"2024-11-26T07:39:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> The question of course is whether these tools require the path-walk API,\n> or whether they could be built on top of existing functionality. But if\n> there are good reasons why the existing functionality is insufficient\n> then I'd be all for having the path-walk API, even if it doesn't help us\n> with repo size reductions as we initially thought.\n\nIs the implied statement that we didn't quite see sufficient rationale\nto convince ourselves that a new path-walk machinery is needed?\n\nThanks.\n\n"},{"id":"508155","messageId":"Z0V8D4r5YbxjNvPs@pks.im","threadId":"62429","inReplyTo":"xmqqa5dmzboa.fsf@gitster.g","subject":"Re: [PATCH v2 0/6] PATH WALK I: The path-walk API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T07:43:11Z","receivedAt":"2024-11-26T07:43:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 26, 2024 at 04:39:17PM +0900, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > The question of course is whether these tools require the path-walk API,\n> > or whether they could be built on top of existing functionality. But if\n> > there are good reasons why the existing functionality is insufficient\n> > then I'd be all for having the path-walk API, even if it doesn't help us\n> > with repo size reductions as we initially thought.\n> \n> Is the implied statement that we didn't quite see sufficient rationale\n> to convince ourselves that a new path-walk machinery is needed?\n\nNo, it's rather that I didn't find the time yet to have a deeper look at\nthe patch series to figure out for myself whether the path-walk API is\nneeded for them. So I was trying to prompt Derrick with the above to\nfind out whether he thinks that it is needed for both of these features\nand if so why the existing APIs are insufficient.\n\nI'm already sold on the idea of git-survey(1) and git-backfill(1), so if\nthere are two use cases where the API makes sense I'm happy to have the\nadditional complexity even if it's not needed anymore for the repo size\nreduction.\n\nPatrick\n"},{"id":"508158","messageId":"xmqqr06yxvdx.fsf@gitster.g","threadId":"62429","inReplyTo":"Z0V8D4r5YbxjNvPs@pks.im","subject":"Re: [PATCH v2 0/6] PATH WALK I: The path-walk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-26T08:16:26Z","receivedAt":"2024-11-26T08:16:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> > then I'd be all for having the path-walk API, even if it doesn't help us\n>> > with repo size reductions as we initially thought.\n>> \n>> Is the implied statement that we didn't quite see sufficient rationale\n>> to convince ourselves that a new path-walk machinery is needed?\n>\n> No, it's rather that I didn't find the time yet to have a deeper look at\n> the patch series to figure out for myself whether the path-walk API is\n> needed for them. So I was trying to prompt Derrick with the above to\n> find out whether he thinks that it is needed for both of these features\n> and if so why the existing APIs are insufficient.\n>\n> I'm already sold on the idea of git-survey(1) and git-backfill(1), so if\n> there are two use cases where the API makes sense I'm happy to have the\n> additional complexity even if it's not needed anymore for the repo size\n> reduction.\n\nAh, I misspoke and failed to add \"in order to implement these new\nfeatures\" after \"is needed\". I like the idea of \"backfill\" thing,\ntoo (as code paths to deal with promisor remotes irritates me ;-).\n\nThanks.\n"},{"id":"508797","messageId":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v2.git.1731181272.gitgitgadget@gmail.com","subject":"[PATCH v3 0/7] PATH WALK I: The path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:51Z","receivedAt":"2024-12-06T19:46:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nIntroduction and relation to prior series\n=========================================\n\nThis is a new series that rerolls the initial \"path-walk API\" patches of my\nRFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\npath-walk.h) presents a new way to walk objects such that trees and blobs\nare walked in batches according to their path.\n\nThis also replaces the previous version of ds/path-walk that was being\nreviewed in [2]. The consensus was that the series was too long/dense and\ncould use some reduction in size. This series takes the first few patches,\nbut also makes some updates (which will be described later).\n\n[1]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n[RFC] Path-walk API and applications\n\n[2]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n[PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n\nThis series only introduces the path-walk API, but does so to the full\ncomplexity required to later add the integration with git pack-objects to\nimprove packing compression in both time and space for repositories with\nmany name hash collisions. The compression also at least improves for other\nrepositories, but may not always have an improvement in time.\n\nSome of the changes that are present in this series that differ from the\nprevious version are motivated directly by discoveries made by testing the\nfeature in Git for Windows and microsoft/git forks that shipped these\nfeatures for fast delivery of these improvements to users who needed them.\nThat testing across many environments informed some things that needed to be\nchanged, and in this series those changes are checked by tests in the\nt6601-path-walk.sh test script and the test-tool path-walk test helper.\nThus, the code being introduced in this series is covered by tests even\nthough it is not integrated into the git executable.\n\n\nDiscussion of follow-up applications\n====================================\n\nBy splitting this series out into its own, I was able to reorganize the\npatches such that each application can be build independently off of this\nseries. These are available as pending PRs in gitgitgadget/git:\n\n * Better delta compression with 'git pack-objects' [3]: This application\n allows an option in 'git pack-objects' to change how objects are walked\n in order to group objects with the same path for early delta compression\n before using the name hash sort to look for cross-path deltas. This helps\n significantly in repositories with many name-hash collisions. This\n reduces the size of 'git push' pacifies via a config option and reduces\n the total repo size in 'git repack'.\n\n * The 'git backfill' command [4]: This command downloads missing blobs in a\n bloodless partial clone. In order to save space and network bandwidth, it\n assumes that objects at a common path are likely to delta well with each\n other, so it downloads missing blobs in batches via the path-walk API.\n This presents a way to use blobless clones as a pseudo-resumable clone,\n since the initial clone of commits and trees is a smaller initial\n download and the batch size allows downloading blobs incrementally. When\n pairing this command with the sparse-checkout feature, the path-walk API\n is adjusted to focus on the paths within the sparse-checkout. This allows\n the user to only download the files they are likely to need when\n inspecting history within their scope without downloading the entire\n repository history.\n\n * The 'git survey' command [5]. This application begins the work to mimic\n the behavior of git-sizer, but to use internal data structures for better\n performance and careful understanding of how objects are stored. Using\n the path-walk API, paths with many versions can be considered in a batch\n and sorted into a list to report the paths that contribute most to the\n size of the repository. A version of this command was used to help\n confirm the issues with the name hash collisions. It was also used to\n diagnose why some repacks using the --path-walk option were taking more\n space than without for some repositories. (More on this later.)\n\nQuestion for reviewers: I am prepped to send these three applications to the\nmailing list, but I'll refrain for now to avoid causing too much noise for\nfolks. Would you like to see them on-list while this series is under review?\nOr would you prefer to explore the PRs ([3] [4] and [5])?\n\n[3] https://github.com/gitgitgadget/git/pull/1819\nPATH WALK II: Add --path-walk option to 'git pack-objects'\n\n[4] https://github.com/gitgitgadget/git/pull/1820\nPATH WALK III: Add 'git backfill' command\n\n[5] https://github.com/gitgitgadget/git/pull/1821\nPATH WALK IV: Add 'git survey' command\n\n\nStructure of the Patch Series\n=============================\n\nThis patch series attempts to create the simplest version of the API in\npatch 1, then build functionality incrementally. During the process, each\nchange will introduce an update to:\n\n * The path-walk API itself in path-walk.c and path-walk.h.\n * The documentation of the API in\n Documentation/technical/api-path-walk.txt.\n * The test script t/t6601-path-walk.sh.\n\nThe core of the API relies on using a 'struct rev_info' to define an initial\nset of objects and some form of a commit walk to define what range of\nobjects to visit. Initially, only a subset of 'struct rev_info' options work\nas expected. For example:\n\n * Patch 1 assumes that only commit objects are starting positions, but the\n focus is on exploring trees and blobs.\n * Patch 3 allows users to specify object types, which includes outputting\n the visited commits in a batch.\n * Annotated tags and indexed objects are considered in Patch 4. These are\n grouped because they both exist within the 'pending' object list.\n * UNINTERESTING objects are not considered until Patch 5.\n\n\nChanges in v1 (since previous version)\n======================================\n\nThere are a few hard-won learnings from previous versions of this series due\nto testing this in the wild with many different repositories.\n\n * Initially, the 'git pack-objects --path-walk' feature was not tested with\n the '--shallow' option because it was expected that this option was for\n servers creating a pack containing shallow commits. However, this option\n is also used when pushing from a shallow clone, and this was a critical\n feature that we needed to reduce the size of commits pushed from\n automated environments that were bootstrapped by shallow clones. The crux\n of the change is in Patch 5 and how UNINTERESTING objects are handled. We\n no longer need to push the UNINTERESTING flag around the objects\n ourselves and can use existing logic in list-objects.c to do so. This\n allows using the --objects-edge-aggressive option when necessary to\n reduce the object count when pushing from a shallow clone. (The\n pack-objects series expands on tests to cover this integration point.)\n\n * When looking into cases where 'git repack -adf' outperformed 'git repack\n -adf --path-walk', I discovered that the issue did not reproduce in a\n bare repository. This is due to 'git repack' iterating over all indexed\n objects before walking commits. I had inadvertently put all indexed\n objects in their own category, leading to no good deltas with previous\n versions of those files; I had also not used the 'path' option from the\n pending list, so these objects had invalid name hash values. You will see\n in patch 4 that the pending list is handled quite differently and the\n '--indexed-objects' option is tested directly within t6601.\n\n * I added a new 'test_cmp_sorted' helper because I wanted to simplify some\n repeated sections of t6601.\n\n * Patch 1 has significantly more context than it did before.\n\n * Annotated tags are given a name of \"/tags\" to differentiate them slightly\n from root trees and commits.\n\n\nChanges in v2\n=============\n\n * Updated the test helper to output the batch number, allowing us to\n confirm that OIDs are grouped appropriately. This also signaled a few\n cases where the callback function was being called on an empty set.\n\n * This change has resulted in significant changes to the test data,\n including reordered lines and prepended batch numbers.\n\n * Thanks to Patrick for providing a recommended change to remove memory\n leaks from the test helper.\n\n\nChanges in v3\n=============\n\n * Updated test helper to use type_string(), which leads to a change to use\n lowercase strings in the test scripts. That will lead to the range-diff\n looking pretty terrible.\n\n * Added a new patch that changes the visit order of the path-walk API. The\n intention is to reduce memory pressure by emitting blob paths before\n recursing into tree paths. This also has the effect of visiting blobs and\n trees in lexicographic order instead of the reverse.\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n path-walk: introduce an object walk by path\n test-lib-functions: add test_cmp_sorted\n t6601: add helper for testing path-walk API\n path-walk: allow consumer to specify object types\n path-walk: visit tags and cached objects\n path-walk: mark trees and blobs as UNINTERESTING\n path-walk: reorder object visits\n\n Documentation/technical/api-path-walk.txt | 63 +++\n Makefile | 2 +\n path-walk.c | 567 ++++++++++++++++++++++\n path-walk.h | 65 +++\n t/helper/test-path-walk.c | 112 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 368 ++++++++++++++\n t/test-lib-functions.sh | 10 +\n 9 files changed, 1189 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\n\nbase-commit: e9356ba3ea2a6754281ff7697b3e5a1697b21e24\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1818%2Fderrickstolee%2Fapi-upstream-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1818/derrickstolee/api-upstream-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1818\n\nRange-diff vs v2:\n\n 1: b7e9b81e8b3 = 1: b7e9b81e8b3 path-walk: introduce an object walk by path\n 2: cf2ed61b324 = 2: cf2ed61b324 test-lib-functions: add test_cmp_sorted\n 3: a3c754d93cc ! 3: 54886fcb081 t6601: add helper for testing path-walk API\n @@ t/helper/test-path-walk.c (new)\n +\tstruct path_walk_test_data *tdata = data;\n +\tconst char *typestr;\n +\n -+\tswitch (type) {\n -+\tcase OBJ_TREE:\n -+\t\ttypestr = \"TREE\";\n ++\tif (type == OBJ_TREE)\n +\t\ttdata->tree_nr += oids->nr;\n -+\t\tbreak;\n -+\n -+\tcase OBJ_BLOB:\n -+\t\ttypestr = \"BLOB\";\n ++\telse if (type == OBJ_BLOB)\n +\t\ttdata->blob_nr += oids->nr;\n -+\t\tbreak;\n -+\n -+\tdefault:\n ++\telse\n +\t\tBUG(\"we do not understand this type\");\n -+\t}\n ++\n ++\ttypestr = type_name(type);\n +\n +\tfor (size_t i = 0; i < oids->nr; i++)\n +\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- --all >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:TREE::$(git rev-parse topic^{tree})\n -+\t0:TREE::$(git rev-parse base^{tree})\n -+\t0:TREE::$(git rev-parse base~1^{tree})\n -+\t0:TREE::$(git rev-parse base~2^{tree})\n -+\t1:TREE:right/:$(git rev-parse topic:right)\n -+\t1:TREE:right/:$(git rev-parse base~1:right)\n -+\t1:TREE:right/:$(git rev-parse base~2:right)\n -+\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n -+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t4:TREE:left/:$(git rev-parse base:left)\n -+\t4:TREE:left/:$(git rev-parse base~2:left)\n -+\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n -+\t5:BLOB:left/b:$(git rev-parse base:left/b)\n -+\t6:BLOB:a:$(git rev-parse base~2:a)\n ++\t0:tree::$(git rev-parse topic^{tree})\n ++\t0:tree::$(git rev-parse base^{tree})\n ++\t0:tree::$(git rev-parse base~1^{tree})\n ++\t0:tree::$(git rev-parse base~2^{tree})\n ++\t1:tree:right/:$(git rev-parse topic:right)\n ++\t1:tree:right/:$(git rev-parse base~1:right)\n ++\t1:tree:right/:$(git rev-parse base~2:right)\n ++\t2:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t3:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t3:blob:right/c:$(git rev-parse topic:right/c)\n ++\t4:tree:left/:$(git rev-parse base:left)\n ++\t4:tree:left/:$(git rev-parse base~2:left)\n ++\t5:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t5:blob:left/b:$(git rev-parse base:left/b)\n ++\t6:blob:a:$(git rev-parse base~2:a)\n +\tblobs:6\n +\ttrees:9\n +\tEOF\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- topic >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:TREE::$(git rev-parse topic^{tree})\n -+\t0:TREE::$(git rev-parse base~1^{tree})\n -+\t0:TREE::$(git rev-parse base~2^{tree})\n -+\t1:TREE:right/:$(git rev-parse topic:right)\n -+\t1:TREE:right/:$(git rev-parse base~1:right)\n -+\t1:TREE:right/:$(git rev-parse base~2:right)\n -+\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n -+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t4:TREE:left/:$(git rev-parse base~2:left)\n -+\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n -+\t6:BLOB:a:$(git rev-parse base~2:a)\n ++\t0:tree::$(git rev-parse topic^{tree})\n ++\t0:tree::$(git rev-parse base~1^{tree})\n ++\t0:tree::$(git rev-parse base~2^{tree})\n ++\t1:tree:right/:$(git rev-parse topic:right)\n ++\t1:tree:right/:$(git rev-parse base~1:right)\n ++\t1:tree:right/:$(git rev-parse base~2:right)\n ++\t2:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t3:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t3:blob:right/c:$(git rev-parse topic:right/c)\n ++\t4:tree:left/:$(git rev-parse base~2:left)\n ++\t5:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t6:blob:a:$(git rev-parse base~2:a)\n +\tblobs:5\n +\ttrees:7\n +\tEOF\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:TREE::$(git rev-parse topic^{tree})\n -+\t1:TREE:right/:$(git rev-parse topic:right)\n -+\t2:BLOB:right/d:$(git rev-parse topic:right/d)\n -+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t4:TREE:left/:$(git rev-parse topic:left)\n -+\t5:BLOB:left/b:$(git rev-parse topic:left/b)\n -+\t6:BLOB:a:$(git rev-parse topic:a)\n ++\t0:tree::$(git rev-parse topic^{tree})\n ++\t1:tree:right/:$(git rev-parse topic:right)\n ++\t2:blob:right/d:$(git rev-parse topic:right/d)\n ++\t3:blob:right/c:$(git rev-parse topic:right/c)\n ++\t4:tree:left/:$(git rev-parse topic:left)\n ++\t5:blob:left/b:$(git rev-parse topic:left/b)\n ++\t6:blob:a:$(git rev-parse topic:a)\n +\tblobs:4\n +\ttrees:3\n +\tEOF\n @@ t/t6601-path-walk.sh (new)\n +\ttest-tool path-walk -- --boundary topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:TREE::$(git rev-parse topic^{tree})\n -+\t0:TREE::$(git rev-parse base~1^{tree})\n -+\t1:TREE:right/:$(git rev-parse topic:right)\n -+\t1:TREE:right/:$(git rev-parse base~1:right)\n -+\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t3:BLOB:right/c:$(git rev-parse base~1:right/c)\n -+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t4:TREE:left/:$(git rev-parse base~1:left)\n -+\t5:BLOB:left/b:$(git rev-parse base~1:left/b)\n -+\t6:BLOB:a:$(git rev-parse base~1:a)\n ++\t0:tree::$(git rev-parse topic^{tree})\n ++\t0:tree::$(git rev-parse base~1^{tree})\n ++\t1:tree:right/:$(git rev-parse topic:right)\n ++\t1:tree:right/:$(git rev-parse base~1:right)\n ++\t2:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t3:blob:right/c:$(git rev-parse base~1:right/c)\n ++\t3:blob:right/c:$(git rev-parse topic:right/c)\n ++\t4:tree:left/:$(git rev-parse base~1:left)\n ++\t5:blob:left/b:$(git rev-parse base~1:left/b)\n ++\t6:blob:a:$(git rev-parse base~1:a)\n +\tblobs:5\n +\ttrees:5\n +\tEOF\n 4: 83b746f569d ! 4: 42e71e6285f path-walk: allow consumer to specify object types\n @@ t/helper/test-path-walk.c: static const char * const path_walk_usage[] = {\n \tuintmax_t blob_nr;\n };\n @@ t/helper/test-path-walk.c: static int emit_block(const char *path, struct oid_array *oids,\n - \tconst char *typestr;\n - \n - \tswitch (type) {\n -+\tcase OBJ_COMMIT:\n -+\t\ttypestr = \"COMMIT\";\n -+\t\ttdata->commit_nr += oids->nr;\n -+\t\tbreak;\n -+\n - \tcase OBJ_TREE:\n - \t\ttypestr = \"TREE\";\n \t\ttdata->tree_nr += oids->nr;\n + \telse if (type == OBJ_BLOB)\n + \t\ttdata->blob_nr += oids->nr;\n ++\telse if (type == OBJ_COMMIT)\n ++\t\ttdata->commit_nr += oids->nr;\n + \telse\n + \t\tBUG(\"we do not understand this type\");\n + \n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n \tstruct path_walk_test_data data = { 0 };\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n \tcat >expect <<-EOF &&\n --\t0:TREE::$(git rev-parse topic^{tree})\n --\t0:TREE::$(git rev-parse base^{tree})\n --\t0:TREE::$(git rev-parse base~1^{tree})\n --\t0:TREE::$(git rev-parse base~2^{tree})\n --\t1:TREE:right/:$(git rev-parse topic:right)\n --\t1:TREE:right/:$(git rev-parse base~1:right)\n --\t1:TREE:right/:$(git rev-parse base~2:right)\n --\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n --\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n --\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t4:TREE:left/:$(git rev-parse base:left)\n --\t4:TREE:left/:$(git rev-parse base~2:left)\n --\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n --\t5:BLOB:left/b:$(git rev-parse base:left/b)\n --\t6:BLOB:a:$(git rev-parse base~2:a)\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t0:COMMIT::$(git rev-parse base)\n -+\t0:COMMIT::$(git rev-parse base~1)\n -+\t0:COMMIT::$(git rev-parse base~2)\n -+\t1:TREE::$(git rev-parse topic^{tree})\n -+\t1:TREE::$(git rev-parse base^{tree})\n -+\t1:TREE::$(git rev-parse base~1^{tree})\n -+\t1:TREE::$(git rev-parse base~2^{tree})\n -+\t2:TREE:right/:$(git rev-parse topic:right)\n -+\t2:TREE:right/:$(git rev-parse base~1:right)\n -+\t2:TREE:right/:$(git rev-parse base~2:right)\n -+\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n -+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t5:TREE:left/:$(git rev-parse base:left)\n -+\t5:TREE:left/:$(git rev-parse base~2:left)\n -+\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n -+\t6:BLOB:left/b:$(git rev-parse base:left/b)\n -+\t7:BLOB:a:$(git rev-parse base~2:a)\n +-\t0:tree::$(git rev-parse topic^{tree})\n +-\t0:tree::$(git rev-parse base^{tree})\n +-\t0:tree::$(git rev-parse base~1^{tree})\n +-\t0:tree::$(git rev-parse base~2^{tree})\n +-\t1:tree:right/:$(git rev-parse topic:right)\n +-\t1:tree:right/:$(git rev-parse base~1:right)\n +-\t1:tree:right/:$(git rev-parse base~2:right)\n +-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n +-\t3:blob:right/c:$(git rev-parse base~2:right/c)\n +-\t3:blob:right/c:$(git rev-parse topic:right/c)\n +-\t4:tree:left/:$(git rev-parse base:left)\n +-\t4:tree:left/:$(git rev-parse base~2:left)\n +-\t5:blob:left/b:$(git rev-parse base~2:left/b)\n +-\t5:blob:left/b:$(git rev-parse base:left/b)\n +-\t6:blob:a:$(git rev-parse base~2:a)\n ++\t0:commit::$(git rev-parse topic)\n ++\t0:commit::$(git rev-parse base)\n ++\t0:commit::$(git rev-parse base~1)\n ++\t0:commit::$(git rev-parse base~2)\n ++\t1:tree::$(git rev-parse topic^{tree})\n ++\t1:tree::$(git rev-parse base^{tree})\n ++\t1:tree::$(git rev-parse base~1^{tree})\n ++\t1:tree::$(git rev-parse base~2^{tree})\n ++\t2:tree:right/:$(git rev-parse topic:right)\n ++\t2:tree:right/:$(git rev-parse base~1:right)\n ++\t2:tree:right/:$(git rev-parse base~2:right)\n ++\t3:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t4:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t4:blob:right/c:$(git rev-parse topic:right/c)\n ++\t5:tree:left/:$(git rev-parse base:left)\n ++\t5:tree:left/:$(git rev-parse base~2:left)\n ++\t6:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t6:blob:left/b:$(git rev-parse base:left/b)\n ++\t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:6\n +\tcommits:4\n \ttrees:9\n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n \ttest-tool path-walk -- topic >out &&\n \n \tcat >expect <<-EOF &&\n --\t0:TREE::$(git rev-parse topic^{tree})\n --\t0:TREE::$(git rev-parse base~1^{tree})\n --\t0:TREE::$(git rev-parse base~2^{tree})\n --\t1:TREE:right/:$(git rev-parse topic:right)\n --\t1:TREE:right/:$(git rev-parse base~1:right)\n --\t1:TREE:right/:$(git rev-parse base~2:right)\n --\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n --\t3:BLOB:right/c:$(git rev-parse base~2:right/c)\n --\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t4:TREE:left/:$(git rev-parse base~2:left)\n --\t5:BLOB:left/b:$(git rev-parse base~2:left/b)\n --\t6:BLOB:a:$(git rev-parse base~2:a)\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t0:COMMIT::$(git rev-parse base~1)\n -+\t0:COMMIT::$(git rev-parse base~2)\n -+\t1:TREE::$(git rev-parse topic^{tree})\n -+\t1:TREE::$(git rev-parse base~1^{tree})\n -+\t1:TREE::$(git rev-parse base~2^{tree})\n -+\t2:TREE:right/:$(git rev-parse topic:right)\n -+\t2:TREE:right/:$(git rev-parse base~1:right)\n -+\t2:TREE:right/:$(git rev-parse base~2:right)\n -+\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n -+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t5:TREE:left/:$(git rev-parse base~2:left)\n -+\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n -+\t7:BLOB:a:$(git rev-parse base~2:a)\n +-\t0:tree::$(git rev-parse topic^{tree})\n +-\t0:tree::$(git rev-parse base~1^{tree})\n +-\t0:tree::$(git rev-parse base~2^{tree})\n +-\t1:tree:right/:$(git rev-parse topic:right)\n +-\t1:tree:right/:$(git rev-parse base~1:right)\n +-\t1:tree:right/:$(git rev-parse base~2:right)\n +-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n +-\t3:blob:right/c:$(git rev-parse base~2:right/c)\n +-\t3:blob:right/c:$(git rev-parse topic:right/c)\n +-\t4:tree:left/:$(git rev-parse base~2:left)\n +-\t5:blob:left/b:$(git rev-parse base~2:left/b)\n +-\t6:blob:a:$(git rev-parse base~2:a)\n ++\t0:commit::$(git rev-parse topic)\n ++\t0:commit::$(git rev-parse base~1)\n ++\t0:commit::$(git rev-parse base~2)\n ++\t1:tree::$(git rev-parse topic^{tree})\n ++\t1:tree::$(git rev-parse base~1^{tree})\n ++\t1:tree::$(git rev-parse base~2^{tree})\n ++\t2:tree:right/:$(git rev-parse topic:right)\n ++\t2:tree:right/:$(git rev-parse base~1:right)\n ++\t2:tree:right/:$(git rev-parse base~2:right)\n ++\t3:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t4:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t4:blob:right/c:$(git rev-parse topic:right/c)\n ++\t5:tree:left/:$(git rev-parse base~2:left)\n ++\t6:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:5\n +\tcommits:3\n \ttrees:7\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n \ttest-tool path-walk -- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t1:TREE::$(git rev-parse topic^{tree})\n -+\t2:TREE:right/:$(git rev-parse topic:right)\n -+\t3:BLOB:right/d:$(git rev-parse topic:right/d)\n -+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t5:TREE:left/:$(git rev-parse topic:left)\n -+\t6:BLOB:left/b:$(git rev-parse topic:left/b)\n -+\t7:BLOB:a:$(git rev-parse topic:a)\n ++\t0:commit::$(git rev-parse topic)\n ++\t1:tree::$(git rev-parse topic^{tree})\n ++\t2:tree:right/:$(git rev-parse topic:right)\n ++\t3:blob:right/d:$(git rev-parse topic:right/d)\n ++\t4:blob:right/c:$(git rev-parse topic:right/c)\n ++\t5:tree:left/:$(git rev-parse topic:left)\n ++\t6:blob:left/b:$(git rev-parse topic:left/b)\n ++\t7:blob:a:$(git rev-parse topic:a)\n +\tblobs:4\n +\tcommits:1\n +\ttrees:3\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\tcat >expect <<-EOF &&\n +\tcommits:0\n +\ttrees:0\n -+\t0:BLOB:right/d:$(git rev-parse topic:right/d)\n -+\t1:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t2:BLOB:left/b:$(git rev-parse topic:left/b)\n -+\t3:BLOB:a:$(git rev-parse topic:a)\n ++\t0:blob:right/d:$(git rev-parse topic:right/d)\n ++\t1:blob:right/c:$(git rev-parse topic:right/c)\n ++\t2:blob:left/b:$(git rev-parse topic:left/b)\n ++\t3:blob:a:$(git rev-parse topic:a)\n +\tblobs:4\n +\tEOF\n +\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\t\t-- topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:COMMIT::$(git rev-parse topic)\n ++\t0:commit::$(git rev-parse topic)\n +\tcommits:1\n +\ttrees:0\n +\tblobs:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\n +\tcat >expect <<-EOF &&\n +\tcommits:0\n - \t0:TREE::$(git rev-parse topic^{tree})\n - \t1:TREE:right/:$(git rev-parse topic:right)\n --\t2:BLOB:right/d:$(git rev-parse topic:right/d)\n --\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t4:TREE:left/:$(git rev-parse topic:left)\n --\t5:BLOB:left/b:$(git rev-parse topic:left/b)\n --\t6:BLOB:a:$(git rev-parse topic:a)\n + \t0:tree::$(git rev-parse topic^{tree})\n + \t1:tree:right/:$(git rev-parse topic:right)\n +-\t2:blob:right/d:$(git rev-parse topic:right/d)\n +-\t3:blob:right/c:$(git rev-parse topic:right/c)\n +-\t4:tree:left/:$(git rev-parse topic:left)\n +-\t5:blob:left/b:$(git rev-parse topic:left/b)\n +-\t6:blob:a:$(git rev-parse topic:a)\n -\tblobs:4\n -+\t2:TREE:left/:$(git rev-parse topic:left)\n ++\t2:tree:left/:$(git rev-parse topic:left)\n \ttrees:3\n +\tblobs:0\n \tEOF\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \ttest-tool path-walk -- --boundary topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n --\t0:TREE::$(git rev-parse topic^{tree})\n --\t0:TREE::$(git rev-parse base~1^{tree})\n --\t1:TREE:right/:$(git rev-parse topic:right)\n --\t1:TREE:right/:$(git rev-parse base~1:right)\n --\t2:BLOB:right/d:$(git rev-parse base~1:right/d)\n --\t3:BLOB:right/c:$(git rev-parse base~1:right/c)\n --\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t4:TREE:left/:$(git rev-parse base~1:left)\n --\t5:BLOB:left/b:$(git rev-parse base~1:left/b)\n --\t6:BLOB:a:$(git rev-parse base~1:a)\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t0:COMMIT::$(git rev-parse base~1)\n -+\t1:TREE::$(git rev-parse topic^{tree})\n -+\t1:TREE::$(git rev-parse base~1^{tree})\n -+\t2:TREE:right/:$(git rev-parse topic:right)\n -+\t2:TREE:right/:$(git rev-parse base~1:right)\n -+\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t4:BLOB:right/c:$(git rev-parse base~1:right/c)\n -+\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t5:TREE:left/:$(git rev-parse base~1:left)\n -+\t6:BLOB:left/b:$(git rev-parse base~1:left/b)\n -+\t7:BLOB:a:$(git rev-parse base~1:a)\n +-\t0:tree::$(git rev-parse topic^{tree})\n +-\t0:tree::$(git rev-parse base~1^{tree})\n +-\t1:tree:right/:$(git rev-parse topic:right)\n +-\t1:tree:right/:$(git rev-parse base~1:right)\n +-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n +-\t3:blob:right/c:$(git rev-parse base~1:right/c)\n +-\t3:blob:right/c:$(git rev-parse topic:right/c)\n +-\t4:tree:left/:$(git rev-parse base~1:left)\n +-\t5:blob:left/b:$(git rev-parse base~1:left/b)\n +-\t6:blob:a:$(git rev-parse base~1:a)\n ++\t0:commit::$(git rev-parse topic)\n ++\t0:commit::$(git rev-parse base~1)\n ++\t1:tree::$(git rev-parse topic^{tree})\n ++\t1:tree::$(git rev-parse base~1^{tree})\n ++\t2:tree:right/:$(git rev-parse topic:right)\n ++\t2:tree:right/:$(git rev-parse base~1:right)\n ++\t3:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t4:blob:right/c:$(git rev-parse base~1:right/c)\n ++\t4:blob:right/c:$(git rev-parse topic:right/c)\n ++\t5:tree:left/:$(git rev-parse base~1:left)\n ++\t6:blob:left/b:$(git rev-parse base~1:left/b)\n ++\t7:blob:a:$(git rev-parse base~1:a)\n \tblobs:5\n +\tcommits:2\n \ttrees:5\n 5: 97765aa04c2 ! 5: a41f53f7ced path-walk: visit tags and cached objects\n @@ t/helper/test-path-walk.c: struct path_walk_test_data {\n static int emit_block(const char *path, struct oid_array *oids,\n @@ t/helper/test-path-walk.c: static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->blob_nr += oids->nr;\n - \t\tbreak;\n - \n -+\tcase OBJ_TAG:\n -+\t\ttypestr = \"TAG\";\n + \telse if (type == OBJ_COMMIT)\n + \t\ttdata->commit_nr += oids->nr;\n ++\telse if (type == OBJ_TAG)\n +\t\ttdata->tag_nr += oids->nr;\n -+\t\tbreak;\n -+\n - \tdefault:\n + \telse\n \t\tBUG(\"we do not understand this type\");\n - \t}\n + \n + \ttypestr = type_name(type);\n \n +\t/* This should never be output during tests. */\n +\tif (!oids->nr)\n @@ t/t6601-path-walk.sh: test_description='direct path-walk API tests'\n \ttest-tool path-walk -- --all >out &&\n \n +\tcat >expect <<-EOF &&\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t0:COMMIT::$(git rev-parse base)\n -+\t0:COMMIT::$(git rev-parse base~1)\n -+\t0:COMMIT::$(git rev-parse base~2)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/first)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/second.1)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/second.2)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/third)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/fourth)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/tree-tag)\n -+\t1:TAG:/tags:$(git rev-parse refs/tags/blob-tag)\n -+\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n -+\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n -+\t3:TREE::$(git rev-parse topic^{tree})\n -+\t3:TREE::$(git rev-parse base^{tree})\n -+\t3:TREE::$(git rev-parse base~1^{tree})\n -+\t3:TREE::$(git rev-parse base~2^{tree})\n -+\t3:TREE::$(git rev-parse refs/tags/tree-tag^{})\n -+\t3:TREE::$(git rev-parse refs/tags/tree-tag2^{})\n -+\t4:BLOB:a:$(git rev-parse base~2:a)\n -+\t5:TREE:right/:$(git rev-parse topic:right)\n -+\t5:TREE:right/:$(git rev-parse base~1:right)\n -+\t5:TREE:right/:$(git rev-parse base~2:right)\n -+\t6:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t7:BLOB:right/c:$(git rev-parse base~2:right/c)\n -+\t7:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t8:TREE:left/:$(git rev-parse base:left)\n -+\t8:TREE:left/:$(git rev-parse base~2:left)\n -+\t9:BLOB:left/b:$(git rev-parse base~2:left/b)\n -+\t9:BLOB:left/b:$(git rev-parse base:left/b)\n -+\t10:TREE:a/:$(git rev-parse base:a)\n -+\t11:BLOB:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n -+\t12:TREE:child/:$(git rev-parse refs/tags/tree-tag:child)\n -+\t13:BLOB:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n ++\t0:commit::$(git rev-parse topic)\n ++\t0:commit::$(git rev-parse base)\n ++\t0:commit::$(git rev-parse base~1)\n ++\t0:commit::$(git rev-parse base~2)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/first)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/second.1)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/second.2)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/third)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/fourth)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/tree-tag)\n ++\t1:tag:/tags:$(git rev-parse refs/tags/blob-tag)\n ++\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n ++\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n ++\t3:tree::$(git rev-parse topic^{tree})\n ++\t3:tree::$(git rev-parse base^{tree})\n ++\t3:tree::$(git rev-parse base~1^{tree})\n ++\t3:tree::$(git rev-parse base~2^{tree})\n ++\t3:tree::$(git rev-parse refs/tags/tree-tag^{})\n ++\t3:tree::$(git rev-parse refs/tags/tree-tag2^{})\n ++\t4:blob:a:$(git rev-parse base~2:a)\n ++\t5:tree:right/:$(git rev-parse topic:right)\n ++\t5:tree:right/:$(git rev-parse base~1:right)\n ++\t5:tree:right/:$(git rev-parse base~2:right)\n ++\t6:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t7:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t7:blob:right/c:$(git rev-parse topic:right/c)\n ++\t8:tree:left/:$(git rev-parse base:left)\n ++\t8:tree:left/:$(git rev-parse base~2:left)\n ++\t9:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t9:blob:left/b:$(git rev-parse base:left/b)\n ++\t10:tree:a/:$(git rev-parse base:a)\n ++\t11:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n ++\t12:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n ++\t13:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n +\tblobs:10\n +\tcommits:4\n +\ttags:7\n @@ t/t6601-path-walk.sh: test_description='direct path-walk API tests'\n +\ttest-tool path-walk -- --indexed-objects >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:BLOB:a:$(git rev-parse HEAD:a)\n -+\t1:BLOB:left/b:$(git rev-parse HEAD:left/b)\n -+\t2:BLOB:left/c:$(git rev-parse :left/c)\n -+\t3:BLOB:right/c:$(git rev-parse HEAD:right/c)\n -+\t4:BLOB:right/d:$(git rev-parse HEAD:right/d)\n -+\t5:TREE:right/:$(git rev-parse topic:right)\n ++\t0:blob:a:$(git rev-parse HEAD:a)\n ++\t1:blob:left/b:$(git rev-parse HEAD:left/b)\n ++\t2:blob:left/c:$(git rev-parse :left/c)\n ++\t3:blob:right/c:$(git rev-parse HEAD:right/c)\n ++\t4:blob:right/d:$(git rev-parse HEAD:right/d)\n ++\t5:tree:right/:$(git rev-parse topic:right)\n +\tblobs:5\n +\tcommits:0\n +\ttags:0\n @@ t/t6601-path-walk.sh: test_description='direct path-walk API tests'\n +\ttest-tool path-walk -- --indexed-objects --branches >out &&\n +\n \tcat >expect <<-EOF &&\n - \t0:COMMIT::$(git rev-parse topic)\n - \t0:COMMIT::$(git rev-parse base)\n + \t0:commit::$(git rev-parse topic)\n + \t0:commit::$(git rev-parse base)\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n - \t1:TREE::$(git rev-parse base^{tree})\n - \t1:TREE::$(git rev-parse base~1^{tree})\n - \t1:TREE::$(git rev-parse base~2^{tree})\n --\t2:TREE:right/:$(git rev-parse topic:right)\n --\t2:TREE:right/:$(git rev-parse base~1:right)\n --\t2:TREE:right/:$(git rev-parse base~2:right)\n --\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n --\t4:BLOB:right/c:$(git rev-parse base~2:right/c)\n --\t4:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t5:TREE:left/:$(git rev-parse base:left)\n --\t5:TREE:left/:$(git rev-parse base~2:left)\n --\t6:BLOB:left/b:$(git rev-parse base~2:left/b)\n --\t6:BLOB:left/b:$(git rev-parse base:left/b)\n --\t7:BLOB:a:$(git rev-parse base~2:a)\n + \t1:tree::$(git rev-parse base^{tree})\n + \t1:tree::$(git rev-parse base~1^{tree})\n + \t1:tree::$(git rev-parse base~2^{tree})\n +-\t2:tree:right/:$(git rev-parse topic:right)\n +-\t2:tree:right/:$(git rev-parse base~1:right)\n +-\t2:tree:right/:$(git rev-parse base~2:right)\n +-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n +-\t4:blob:right/c:$(git rev-parse base~2:right/c)\n +-\t4:blob:right/c:$(git rev-parse topic:right/c)\n +-\t5:tree:left/:$(git rev-parse base:left)\n +-\t5:tree:left/:$(git rev-parse base~2:left)\n +-\t6:blob:left/b:$(git rev-parse base~2:left/b)\n +-\t6:blob:left/b:$(git rev-parse base:left/b)\n +-\t7:blob:a:$(git rev-parse base~2:a)\n -\tblobs:6\n -+\t2:BLOB:a:$(git rev-parse base~2:a)\n -+\t3:TREE:right/:$(git rev-parse topic:right)\n -+\t3:TREE:right/:$(git rev-parse base~1:right)\n -+\t3:TREE:right/:$(git rev-parse base~2:right)\n -+\t4:BLOB:right/d:$(git rev-parse base~1:right/d)\n -+\t4:BLOB:right/d:$(git rev-parse :right/d)\n -+\t5:BLOB:right/c:$(git rev-parse base~2:right/c)\n -+\t5:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t6:TREE:left/:$(git rev-parse base:left)\n -+\t6:TREE:left/:$(git rev-parse base~2:left)\n -+\t7:BLOB:left/b:$(git rev-parse base:left/b)\n -+\t7:BLOB:left/b:$(git rev-parse base~2:left/b)\n -+\t8:TREE:a/:$(git rev-parse refs/tags/third:a)\n ++\t2:blob:a:$(git rev-parse base~2:a)\n ++\t3:tree:right/:$(git rev-parse topic:right)\n ++\t3:tree:right/:$(git rev-parse base~1:right)\n ++\t3:tree:right/:$(git rev-parse base~2:right)\n ++\t4:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t4:blob:right/d:$(git rev-parse :right/d)\n ++\t5:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t5:blob:right/c:$(git rev-parse topic:right/c)\n ++\t6:tree:left/:$(git rev-parse base:left)\n ++\t6:tree:left/:$(git rev-parse base~2:left)\n ++\t7:blob:left/b:$(git rev-parse base:left/b)\n ++\t7:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t8:tree:a/:$(git rev-parse refs/tags/third:a)\n +\tblobs:7\n \tcommits:4\n -\ttrees:9\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n \n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n - \t7:BLOB:a:$(git rev-parse base~2:a)\n + \t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:5\n \tcommits:3\n +\ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic only' '\n \tEOF\n \n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n - \t7:BLOB:a:$(git rev-parse topic:a)\n + \t7:blob:a:$(git rev-parse topic:a)\n \tblobs:4\n \tcommits:1\n +\ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n \tcat >expect <<-EOF &&\n -\tcommits:0\n -\ttrees:0\n - \t0:BLOB:right/d:$(git rev-parse topic:right/d)\n - \t1:BLOB:right/c:$(git rev-parse topic:right/c)\n - \t2:BLOB:left/b:$(git rev-parse topic:left/b)\n - \t3:BLOB:a:$(git rev-parse topic:a)\n + \t0:blob:right/d:$(git rev-parse topic:right/d)\n + \t1:blob:right/c:$(git rev-parse topic:right/c)\n + \t2:blob:left/b:$(git rev-parse topic:left/b)\n + \t3:blob:a:$(git rev-parse topic:a)\n \tblobs:4\n +\tcommits:0\n +\ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only blobs' '\n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only commits' '\n \tcat >expect <<-EOF &&\n - \t0:COMMIT::$(git rev-parse topic)\n + \t0:commit::$(git rev-parse topic)\n \tcommits:1\n -\ttrees:0\n \tblobs:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n \n \tcat >expect <<-EOF &&\n -\tcommits:0\n - \t0:TREE::$(git rev-parse topic^{tree})\n - \t1:TREE:right/:$(git rev-parse topic:right)\n - \t2:TREE:left/:$(git rev-parse topic:left)\n + \t0:tree::$(git rev-parse topic^{tree})\n + \t1:tree:right/:$(git rev-parse topic:right)\n + \t2:tree:left/:$(git rev-parse topic:left)\n -\ttrees:3\n +\tcommits:0\n \tblobs:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n \n \ttest_cmp_sorted expect out\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n - \t7:BLOB:a:$(git rev-parse base~1:a)\n + \t7:blob:a:$(git rev-parse base~1:a)\n \tblobs:5\n \tcommits:2\n +\ttags:0\n 6: a4aaa3b001b ! 6: 0f1e6c51b2c path-walk: mark trees and blobs as UNINTERESTING\n @@ t/helper/test-path-walk.c: int cmd__path_walk(int argc, const char **argv)\n \n ## t/t6601-path-walk.sh ##\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n - \t0:COMMIT::$(git rev-parse topic)\n - \t1:TREE::$(git rev-parse topic^{tree})\n - \t2:TREE:right/:$(git rev-parse topic:right)\n --\t3:BLOB:right/d:$(git rev-parse topic:right/d)\n -+\t3:BLOB:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n - \t4:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t5:TREE:left/:$(git rev-parse topic:left)\n --\t6:BLOB:left/b:$(git rev-parse topic:left/b)\n --\t7:BLOB:a:$(git rev-parse topic:a)\n -+\t5:TREE:left/:$(git rev-parse topic:left):UNINTERESTING\n -+\t6:BLOB:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n -+\t7:BLOB:a:$(git rev-parse topic:a):UNINTERESTING\n + \t0:commit::$(git rev-parse topic)\n + \t1:tree::$(git rev-parse topic^{tree})\n + \t2:tree:right/:$(git rev-parse topic:right)\n +-\t3:blob:right/d:$(git rev-parse topic:right/d)\n ++\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n + \t4:blob:right/c:$(git rev-parse topic:right/c)\n +-\t5:tree:left/:$(git rev-parse topic:left)\n +-\t6:blob:left/b:$(git rev-parse topic:left/b)\n +-\t7:blob:a:$(git rev-parse topic:a)\n ++\t5:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n ++\t6:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n ++\t7:blob:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:1\n \ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n +\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t1:TAG:/tags:$(git rev-parse fourth)\n -+\t2:BLOB:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n -+\t3:TREE::$(git rev-parse topic^{tree})\n -+\t4:TREE:right/:$(git rev-parse topic:right)\n -+\t5:BLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n -+\t6:BLOB:right/c:$(git rev-parse topic:right/c)\n -+\t7:TREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n -+\t8:BLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n -+\t9:BLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n ++\t0:commit::$(git rev-parse topic)\n ++\t1:tag:/tags:$(git rev-parse fourth)\n ++\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n ++\t3:tree::$(git rev-parse topic^{tree})\n ++\t4:tree:right/:$(git rev-parse topic:right)\n ++\t5:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n ++\t6:blob:right/c:$(git rev-parse topic:right/c)\n ++\t7:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n ++\t8:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n ++\t9:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n +\tblobs:5\n +\tcommits:1\n +\ttags:1\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n --\t0:BLOB:right/d:$(git rev-parse topic:right/d)\n -+\t0:BLOB:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n - \t1:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t2:BLOB:left/b:$(git rev-parse topic:left/b)\n --\t3:BLOB:a:$(git rev-parse topic:a)\n -+\t2:BLOB:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n -+\t3:BLOB:a:$(git rev-parse topic:a):UNINTERESTING\n +-\t0:blob:right/d:$(git rev-parse topic:right/d)\n ++\t0:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n + \t1:blob:right/c:$(git rev-parse topic:right/c)\n +-\t2:blob:left/b:$(git rev-parse topic:left/b)\n +-\t3:blob:a:$(git rev-parse topic:a)\n ++\t2:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n ++\t3:blob:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:0\n \ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, only trees' '\n \tcat >expect <<-EOF &&\n - \t0:TREE::$(git rev-parse topic^{tree})\n - \t1:TREE:right/:$(git rev-parse topic:right)\n --\t2:TREE:left/:$(git rev-parse topic:left)\n -+\t2:TREE:left/:$(git rev-parse topic:left):UNINTERESTING\n + \t0:tree::$(git rev-parse topic^{tree})\n + \t1:tree:right/:$(git rev-parse topic:right)\n +-\t2:tree:left/:$(git rev-parse topic:left)\n ++\t2:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n \tcommits:0\n \tblobs:0\n \ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n \n \tcat >expect <<-EOF &&\n - \t0:COMMIT::$(git rev-parse topic)\n --\t0:COMMIT::$(git rev-parse base~1)\n -+\t0:COMMIT::$(git rev-parse base~1):UNINTERESTING\n - \t1:TREE::$(git rev-parse topic^{tree})\n --\t1:TREE::$(git rev-parse base~1^{tree})\n -+\t1:TREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n - \t2:TREE:right/:$(git rev-parse topic:right)\n --\t2:TREE:right/:$(git rev-parse base~1:right)\n --\t3:BLOB:right/d:$(git rev-parse base~1:right/d)\n --\t4:BLOB:right/c:$(git rev-parse base~1:right/c)\n -+\t2:TREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n -+\t3:BLOB:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n -+\t4:BLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n - \t4:BLOB:right/c:$(git rev-parse topic:right/c)\n --\t5:TREE:left/:$(git rev-parse base~1:left)\n --\t6:BLOB:left/b:$(git rev-parse base~1:left/b)\n --\t7:BLOB:a:$(git rev-parse base~1:a)\n -+\t5:TREE:left/:$(git rev-parse base~1:left):UNINTERESTING\n -+\t6:BLOB:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n -+\t7:BLOB:a:$(git rev-parse base~1:a):UNINTERESTING\n + \t0:commit::$(git rev-parse topic)\n +-\t0:commit::$(git rev-parse base~1)\n ++\t0:commit::$(git rev-parse base~1):UNINTERESTING\n + \t1:tree::$(git rev-parse topic^{tree})\n +-\t1:tree::$(git rev-parse base~1^{tree})\n ++\t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n + \t2:tree:right/:$(git rev-parse topic:right)\n +-\t2:tree:right/:$(git rev-parse base~1:right)\n +-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n +-\t4:blob:right/c:$(git rev-parse base~1:right/c)\n ++\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n ++\t3:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n ++\t4:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n + \t4:blob:right/c:$(git rev-parse topic:right/c)\n +-\t5:tree:left/:$(git rev-parse base~1:left)\n +-\t6:blob:left/b:$(git rev-parse base~1:left/b)\n +-\t7:blob:a:$(git rev-parse base~1:a)\n ++\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n ++\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n ++\t7:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n \tblobs:5\n \tcommits:2\n \ttags:0\n @@ t/t6601-path-walk.sh: test_expect_success 'topic, not base, boundary' '\n +\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n +\n +\tcat >expect <<-EOF &&\n -+\t0:COMMIT::$(git rev-parse topic)\n -+\t0:COMMIT::$(git rev-parse base~1):UNINTERESTING\n -+\t1:TREE::$(git rev-parse topic^{tree})\n -+\t1:TREE::$(git rev-parse base~1^{tree}):UNINTERESTING\n -+\t2:TREE:right/:$(git rev-parse topic:right)\n -+\t2:TREE:right/:$(git rev-parse base~1:right):UNINTERESTING\n -+\t3:BLOB:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n -+\t3:BLOB:right/c:$(git rev-parse topic:right/c)\n ++\t0:commit::$(git rev-parse topic)\n ++\t0:commit::$(git rev-parse base~1):UNINTERESTING\n ++\t1:tree::$(git rev-parse topic^{tree})\n ++\t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n ++\t2:tree:right/:$(git rev-parse topic:right)\n ++\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n ++\t3:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n ++\t3:blob:right/c:$(git rev-parse topic:right/c)\n +\tblobs:2\n +\tcommits:2\n +\ttags:0\n -: ----------- > 7: e716672c041 path-walk: reorder object visits\n\n-- \ngitgitgadget\n"},{"id":"508795","messageId":"b7e9b81e8b32313f00d38257ba731e73d17224cb.1733514358.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 1/7] path-walk: introduce an object walk by path","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:52Z","receivedAt":"2024-12-06T19:46:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a few planned applications, introduce the most basic form\nof a path-walk API. It currently assumes that there are no UNINTERESTING\nobjects, and does not include any complicated filters. It calls a function\npointer on groups of tree and blob objects as grouped by path. This only\nincludes objects the first time they are discovered, so an object that\nappears at multiple paths will not be included in two batches.\n\nThese batches are collected in 'struct type_and_oid_list' objects, which\nstore an object type and an oid_array of objects.\n\nThe data structures are documented in 'struct path_walk_context', but in\nsummary the most important are:\n\n * 'paths_to_lists' is a strmap that connects a path to a\n type_and_oid_list for that path. To avoid conflicts in path names,\n we make sure that tree paths end in \"/\" (except the root path with\n is an empty string) and blob paths do not end in \"/\".\n\n * 'path_stack' is a string list that is added to in an append-only\n way. This stores the stack of our depth-first search on the heap\n instead of using recursion.\n\n * 'path_stack_pushed' is a strmap that stores path names that were\n already added to 'path_stack', to avoid repeating paths in the\n stack. Mostly, this saves us from quadratic lookups from doing\n unsorted checks into the string_list.\n\nThe coupling of 'path_stack' and 'path_stack_pushed' is protected by the\npush_to_stack() method. Call this instead of inserting into these\nstructures directly.\n\nThe walk_objects_by_path() method initializes these structures and\nstarts walking commits from the given rev_info struct. The commits are\nused to find the list of root trees which populate the start of our\ndepth-first search.\n\nThe core of our depth-first search is in a while loop that continues\nwhile we have not indicated an early exit and our 'path_stack' still has\nentries in it. The loop body pops a path off of the stack and \"visits\"\nthe path via the walk_path() method.\n\nThe walk_path() method gets the list of OIDs from the 'path_to_lists'\nstrmap and executes the callback method on that list with the given path\nand type. If the OIDs correspond to tree objects, then iterate over all\ntrees in the list and run add_children() to add the child objects to\ntheir own lists, adding new entries to the stack if necessary.\n\nIn testing, this depth-first search approach was the one that used the\nleast memory while iterating over the object lists. There is still a\nchance that repositories with too-wide path patterns could cause memory\npressure issues. Limiting the stack size could be done in the future by\nlimiting how many objects are being considered in-progress, or by\nvisiting blob paths earlier than trees.\n\nThere are many future adaptations that could be made, but they are left for\nfuture updates when consumers are ready to take advantage of those features.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 45 ++++\n Makefile | 1 +\n path-walk.c | 263 ++++++++++++++++++++++\n path-walk.h | 43 ++++\n 4 files changed, 352 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nnew file mode 100644\nindex 00000000000..c550c77ca30\n--- /dev/null\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -0,0 +1,45 @@\n+Path-Walk API\n+=============\n+\n+The path-walk API is used to walk reachable objects, but to visit objects\n+in batches based on a common path they appear in, or by type.\n+\n+For example, all reachable commits are visited in a group. All tags are\n+visited in a group. Then, all root trees are visited. At some point, all\n+blobs reachable via a path `my/dir/to/A` are visited. When there are\n+multiple paths possible to reach the same object, then only one of those\n+paths is used to visit the object.\n+\n+Basics\n+------\n+\n+To use the path-walk API, include `path-walk.h` and call\n+`walk_objects_by_path()` with a customized `path_walk_info` struct. The\n+struct is used to set all of the options for how the walk should proceed.\n+Let's dig into the different options and their use.\n+\n+`path_fn` and `path_fn_data`::\n+\tThe most important option is the `path_fn` option, which is a\n+\tfunction pointer to the callback that can execute logic on the\n+\tobject IDs for objects grouped by type and path. This function\n+\talso receives a `data` value that corresponds to the\n+\t`path_fn_data` member, for providing custom data structures to\n+\tthis callback function.\n+\n+`revs`::\n+\tTo configure the exact details of the reachable set of objects,\n+\tuse the `revs` member and initialize it using the revision\n+\tmachinery in `revision.h`. Initialize `revs` using calls such as\n+\t`setup_revisions()` or `parse_revision_opt()`. Do not call\n+\t`prepare_revision_walk()`, as that will be called within\n+\t`walk_objects_by_path()`.\n++\n+It is also important that you do not specify the `--objects` flag for the\n+`revs` struct. The revision walk should only be used to walk commits, and\n+the objects will be walked in a separate way based on those starting\n+commits.\n+\n+Examples\n+--------\n+\n+See example usages in future changes.\ndiff --git a/Makefile b/Makefile\nindex 7344a7f7257..d0d8d6888e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\n+LIB_OBJS += path-walk.o\n LIB_OBJS += pathspec.o\n LIB_OBJS += pkt-line.o\n LIB_OBJS += preload-index.o\ndiff --git a/path-walk.c b/path-walk.c\nnew file mode 100644\nindex 00000000000..24cf04c1e7d\n--- /dev/null\n+++ b/path-walk.c\n@@ -0,0 +1,263 @@\n+/*\n+ * path-walk.c: implementation for path-based walks of the object graph.\n+ */\n+#include \"git-compat-util.h\"\n+#include \"path-walk.h\"\n+#include \"blob.h\"\n+#include \"commit.h\"\n+#include \"dir.h\"\n+#include \"hashmap.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"oid-array.h\"\n+#include \"revision.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+#include \"trace2.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+\n+struct type_and_oid_list\n+{\n+\tenum object_type type;\n+\tstruct oid_array oids;\n+};\n+\n+#define TYPE_AND_OID_LIST_INIT { \\\n+\t.type = OBJ_NONE, \t \\\n+\t.oids = OID_ARRAY_INIT\t \\\n+}\n+\n+struct path_walk_context {\n+\t/**\n+\t * Repeats of data in 'struct path_walk_info' for\n+\t * access with fewer characters.\n+\t */\n+\tstruct repository *repo;\n+\tstruct rev_info *revs;\n+\tstruct path_walk_info *info;\n+\n+\t/**\n+\t * Map a path to a 'struct type_and_oid_list'\n+\t * containing the objects discovered at that\n+\t * path.\n+\t */\n+\tstruct strmap paths_to_lists;\n+\n+\t/**\n+\t * Store the current list of paths in a stack, to\n+\t * facilitate depth-first-search without recursion.\n+\t *\n+\t * Use path_stack_pushed to indicate whether a path\n+\t * was previously added to path_stack.\n+\t */\n+\tstruct string_list path_stack;\n+\tstruct strset path_stack_pushed;\n+};\n+\n+static void push_to_stack(struct path_walk_context *ctx,\n+\t\t\t const char *path)\n+{\n+\tif (strset_contains(&ctx->path_stack_pushed, path))\n+\t\treturn;\n+\n+\tstrset_add(&ctx->path_stack_pushed, path);\n+\tstring_list_append(&ctx->path_stack, path);\n+}\n+\n+static int add_children(struct path_walk_context *ctx,\n+\t\t\tconst char *base_path,\n+\t\t\tstruct object_id *oid)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t base_len;\n+\tstruct tree *tree = lookup_tree(ctx->repo, oid);\n+\n+\tif (!tree) {\n+\t\terror(_(\"failed to walk children of tree %s: not found\"),\n+\t\t oid_to_hex(oid));\n+\t\treturn -1;\n+\t} else if (parse_tree_gently(tree, 1)) {\n+\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n+\t}\n+\n+\tstrbuf_addstr(&path, base_path);\n+\tbase_len = path.len;\n+\n+\tparse_tree(tree);\n+\tinit_tree_desc(&desc, &tree->object.oid, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tstruct type_and_oid_list *list;\n+\t\tstruct object *o;\n+\t\t/* Not actually true, but we will ignore submodules later. */\n+\t\tenum object_type type = S_ISDIR(entry.mode) ? OBJ_TREE : OBJ_BLOB;\n+\n+\t\t/* Skip submodules. */\n+\t\tif (S_ISGITLINK(entry.mode))\n+\t\t\tcontinue;\n+\n+\t\tif (type == OBJ_TREE) {\n+\t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else if (type == OBJ_BLOB) {\n+\t\t\tstruct blob *child = lookup_blob(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else {\n+\t\t\t/* Wrong type? */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!o) /* report error?*/\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, base_len);\n+\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n+\n+\t\t/*\n+\t\t * Trees will end with \"/\" for concatenation and distinction\n+\t\t * from blobs at the same path.\n+\t\t */\n+\t\tif (type == OBJ_TREE)\n+\t\t\tstrbuf_addch(&path, '/');\n+\n+\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {\n+\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\tlist->type = type;\n+\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n+\t\t}\n+\t\tpush_to_stack(ctx, path.buf);\n+\n+\t\t/* Skip this object if already seen. */\n+\t\tif (o->flags & SEEN)\n+\t\t\tcontinue;\n+\t\to->flags |= SEEN;\n+\t\toid_array_append(&list->oids, &entry.oid);\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+\tstrbuf_release(&path);\n+\treturn 0;\n+}\n+\n+/*\n+ * For each path in paths_to_explore, walk the trees another level\n+ * and add any found blobs to the batch (but only if they exist and\n+ * haven't been added yet).\n+ */\n+static int walk_path(struct path_walk_context *ctx,\n+\t\t const char *path)\n+{\n+\tstruct type_and_oid_list *list;\n+\tint ret = 0;\n+\n+\tlist = strmap_get(&ctx->paths_to_lists, path);\n+\n+\tif (!list->oids.nr)\n+\t\treturn 0;\n+\n+\t/* Evaluate function pointer on this data. */\n+\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t ctx->info->path_fn_data);\n+\n+\t/* Expand data for children. */\n+\tif (list->type == OBJ_TREE) {\n+\t\tfor (size_t i = 0; i < list->oids.nr; i++) {\n+\t\t\tret |= add_children(ctx,\n+\t\t\t\t\t path,\n+\t\t\t\t\t &list->oids.oid[i]);\n+\t\t}\n+\t}\n+\n+\toid_array_clear(&list->oids);\n+\tstrmap_remove(&ctx->paths_to_lists, path, 1);\n+\treturn ret;\n+}\n+\n+static void clear_strmap(struct strmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\n+\thashmap_for_each_entry(&map->map, &iter, e, ent) {\n+\t\tstruct type_and_oid_list *list = e->value;\n+\t\toid_array_clear(&list->oids);\n+\t}\n+\tstrmap_clear(map, 1);\n+\tstrmap_init(map);\n+}\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info)\n+{\n+\tconst char *root_path = \"\";\n+\tint ret = 0;\n+\tsize_t commits_nr = 0, paths_nr = 0;\n+\tstruct commit *c;\n+\tstruct type_and_oid_list *root_tree_list;\n+\tstruct path_walk_context ctx = {\n+\t\t.repo = info->revs->repo,\n+\t\t.revs = info->revs,\n+\t\t.info = info,\n+\t\t.path_stack = STRING_LIST_INIT_DUP,\n+\t\t.path_stack_pushed = STRSET_INIT,\n+\t\t.paths_to_lists = STRMAP_INIT\n+\t};\n+\n+\ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\t/* Insert a single list for the root tree into the paths. */\n+\tCALLOC_ARRAY(root_tree_list, 1);\n+\troot_tree_list->type = OBJ_TREE;\n+\tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n+\tpush_to_stack(&ctx, root_path);\n+\n+\tif (prepare_revision_walk(info->revs))\n+\t\tdie(_(\"failed to setup revision walk\"));\n+\n+\twhile ((c = get_revision(info->revs))) {\n+\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct tree *t;\n+\t\tcommits_nr++;\n+\n+\t\toid = get_commit_tree_oid(c);\n+\t\tt = lookup_tree(info->revs->repo, oid);\n+\n+\t\tif (!t) {\n+\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (t->object.flags & SEEN)\n+\t\t\tcontinue;\n+\t\tt->object.flags |= SEEN;\n+\t\toid_array_append(&root_tree_list->oids, oid);\n+\t}\n+\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n+\ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n+\twhile (!ret && ctx.path_stack.nr) {\n+\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\tctx.path_stack.nr--;\n+\t\tpaths_nr++;\n+\n+\t\tret = walk_path(&ctx, path);\n+\n+\t\tfree(path);\n+\t}\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n+\ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n+\n+\tclear_strmap(&ctx.paths_to_lists);\n+\tstrset_clear(&ctx.path_stack_pushed);\n+\tstring_list_clear(&ctx.path_stack, 0);\n+\treturn ret;\n+}\ndiff --git a/path-walk.h b/path-walk.h\nnew file mode 100644\nindex 00000000000..c9e94a98bc8\n--- /dev/null\n+++ b/path-walk.h\n@@ -0,0 +1,43 @@\n+/*\n+ * path-walk.h : Methods and structures for walking the object graph in batches\n+ * by the paths that can reach those objects.\n+ */\n+#include \"object.h\" /* Required for 'enum object_type'. */\n+\n+struct rev_info;\n+struct oid_array;\n+\n+/**\n+ * The type of a function pointer for the method that is called on a list of\n+ * objects reachable at a given path.\n+ */\n+typedef int (*path_fn)(const char *path,\n+\t\t struct oid_array *oids,\n+\t\t enum object_type type,\n+\t\t void *data);\n+\n+struct path_walk_info {\n+\t/**\n+\t * revs provides the definitions for the commit walk, including\n+\t * which commits are UNINTERESTING or not.\n+\t */\n+\tstruct rev_info *revs;\n+\n+\t/**\n+\t * The caller wishes to execute custom logic on objects reachable at a\n+\t * given path. Every reachable object will be visited exactly once, and\n+\t * the first path to see an object wins. This may not be a stable choice.\n+\t */\n+\tpath_fn path_fn;\n+\tvoid *path_fn_data;\n+};\n+\n+#define PATH_WALK_INFO_INIT { 0 }\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info);\n-- \ngitgitgadget\n\n"},{"id":"508796","messageId":"cf2ed61b324b019f61afb1b66163b1284e675db2.1733514359.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 2/7] test-lib-functions: add test_cmp_sorted","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:53Z","receivedAt":"2024-12-06T19:46:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis test helper will be helpful to reduce repeated logic in\nt6601-path-walk.sh, but may be helpful elsewhere, too.\n\nSigned-off-by: Derrick Stolee \n---\n t/test-lib-functions.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex fde9bf54fc3..16b70aebd60 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1267,6 +1267,16 @@ test_cmp () {\n \teval \"$GIT_TEST_CMP\" '\"$@\"'\n }\n \n+# test_cmp_sorted runs test_cmp on sorted versions of the two\n+# input files. Uses \"$1.sorted\" and \"$2.sorted\" as temp files.\n+\n+test_cmp_sorted () {\n+\tsort <\"$1\" >\"$1.sorted\" &&\n+\tsort <\"$2\" >\"$2.sorted\" &&\n+\ttest_cmp \"$1.sorted\" \"$2.sorted\" &&\n+\trm \"$1.sorted\" \"$2.sorted\"\n+}\n+\n # Check that the given config key has the expected value.\n #\n # test_cmp_config [-C ] \n-- \ngitgitgadget\n\n"},{"id":"508798","messageId":"54886fcb081601157799d40a6b2e292c0c0a053d.1733514359.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 3/7] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:54Z","receivedAt":"2024-12-06T19:46:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests based on the current behavior, doing interesting checks\nfor different sets of branches, ranges, and the --boundary option. This\nsets a baseline for the behavior and we can extend it as new options are\nintroduced.\n\nStore and output a 'batch_nr' value so we can demonstrate that the paths are\ngrouped together in a batch and not following some other ordering. This\nallows us to test the depth-first behavior of the path-walk API. However, we\npurposefully do not test the order of the objects in the batch, so the\noutput is compared to the expected output through a sort.\n\nIt is important to mention that the behavior of the API will change soon as\nwe start to handle UNINTERESTING objects differently, but these tests will\ndemonstrate the change in behavior.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 3 +-\n Makefile | 1 +\n t/helper/test-path-walk.c | 84 +++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 120 ++++++++++++++++++++++\n 6 files changed, 209 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex c550c77ca30..662162ec70b 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -42,4 +42,5 @@ commits.\n Examples\n --------\n \n-See example usages in future changes.\n+See example usages in:\n+\t`t/helper/test-path-walk.c`\ndiff --git a/Makefile b/Makefile\nindex d0d8d6888e3..50413d96492 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -818,6 +818,7 @@ TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n+TEST_BUILTINS_OBJS += test-path-walk.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\n TEST_BUILTINS_OBJS += test-proc-receive.o\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nnew file mode 100644\nindex 00000000000..def7c81ac4f\n--- /dev/null\n+++ b/t/helper/test-path-walk.c\n@@ -0,0 +1,84 @@\n+#define USE_THE_REPOSITORY_VARIABLE\n+\n+#include \"test-tool.h\"\n+#include \"environment.h\"\n+#include \"hex.h\"\n+#include \"object-name.h\"\n+#include \"object.h\"\n+#include \"pretty.h\"\n+#include \"revision.h\"\n+#include \"setup.h\"\n+#include \"parse-options.h\"\n+#include \"path-walk.h\"\n+#include \"oid-array.h\"\n+\n+static const char * const path_walk_usage[] = {\n+\tN_(\"test-tool path-walk -- \"),\n+\tNULL\n+};\n+\n+struct path_walk_test_data {\n+\tuintmax_t batch_nr;\n+\tuintmax_t tree_nr;\n+\tuintmax_t blob_nr;\n+};\n+\n+static int emit_block(const char *path, struct oid_array *oids,\n+\t\t enum object_type type, void *data)\n+{\n+\tstruct path_walk_test_data *tdata = data;\n+\tconst char *typestr;\n+\n+\tif (type == OBJ_TREE)\n+\t\ttdata->tree_nr += oids->nr;\n+\telse if (type == OBJ_BLOB)\n+\t\ttdata->blob_nr += oids->nr;\n+\telse\n+\t\tBUG(\"we do not understand this type\");\n+\n+\ttypestr = type_name(type);\n+\n+\tfor (size_t i = 0; i < oids->nr; i++)\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n+\t\t tdata->batch_nr, typestr, path,\n+\t\t oid_to_hex(&oids->oid[i]));\n+\n+\ttdata->batch_nr++;\n+\treturn 0;\n+}\n+\n+int cmd__path_walk(int argc, const char **argv)\n+{\n+\tint res;\n+\tstruct rev_info revs = REV_INFO_INIT;\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tstruct path_walk_test_data data = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tsetup_git_directory();\n+\trevs.repo = the_repository;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, path_walk_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT | PARSE_OPT_KEEP_ARGV0);\n+\n+\tif (argc > 1)\n+\t\tsetup_revisions(argc, argv, &revs, NULL);\n+\telse\n+\t\tusage(path_walk_usage[0]);\n+\n+\tinfo.revs = &revs;\n+\tinfo.path_fn = emit_block;\n+\tinfo.path_fn_data = &data;\n+\n+\tres = walk_objects_by_path(&info);\n+\n+\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\t \"blobs:%\" PRIuMAX \"\\n\",\n+\t data.tree_nr, data.blob_nr);\n+\n+\trelease_revisions(&revs);\n+\treturn res;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..43676e7b93a 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -52,6 +52,7 @@ static struct test_cmd cmds[] = {\n \t{ \"parse-subcommand\", cmd__parse_subcommand },\n \t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n+\t{ \"path-walk\", cmd__path_walk },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"proc-receive\", cmd__proc_receive },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..9cfc5da6e57 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__parse_pathspec_file(int argc, const char** argv);\n int cmd__parse_subcommand(int argc, const char **argv);\n int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n+int cmd__path_walk(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\n int cmd__proc_receive(int argc, const char **argv);\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nnew file mode 100755\nindex 00000000000..4e052c09309\n--- /dev/null\n+++ b/t/t6601-path-walk.sh\n@@ -0,0 +1,120 @@\n+#!/bin/sh\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+\n+test_description='direct path-walk API tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repository' '\n+\tgit checkout -b base &&\n+\n+\tmkdir left &&\n+\tmkdir right &&\n+\techo a >a &&\n+\techo b >left/b &&\n+\techo c >right/c &&\n+\tgit add . &&\n+\tgit commit -m \"first\" &&\n+\n+\techo d >right/d &&\n+\tgit add right &&\n+\tgit commit -m \"second\" &&\n+\n+\techo bb >left/b &&\n+\tgit commit -a -m \"third\" &&\n+\n+\tgit checkout -b topic HEAD~1 &&\n+\techo cc >right/c &&\n+\tgit commit -a -m \"topic\"\n+'\n+\n+test_expect_success 'all' '\n+\ttest-tool path-walk -- --all >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t0:tree::$(git rev-parse base^{tree})\n+\t0:tree::$(git rev-parse base~1^{tree})\n+\t0:tree::$(git rev-parse base~2^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t1:tree:right/:$(git rev-parse base~1:right)\n+\t1:tree:right/:$(git rev-parse base~2:right)\n+\t2:blob:right/d:$(git rev-parse base~1:right/d)\n+\t3:blob:right/c:$(git rev-parse base~2:right/c)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse base:left)\n+\t4:tree:left/:$(git rev-parse base~2:left)\n+\t5:blob:left/b:$(git rev-parse base~2:left/b)\n+\t5:blob:left/b:$(git rev-parse base:left/b)\n+\t6:blob:a:$(git rev-parse base~2:a)\n+\tblobs:6\n+\ttrees:9\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic only' '\n+\ttest-tool path-walk -- topic >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t0:tree::$(git rev-parse base~1^{tree})\n+\t0:tree::$(git rev-parse base~2^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t1:tree:right/:$(git rev-parse base~1:right)\n+\t1:tree:right/:$(git rev-parse base~2:right)\n+\t2:blob:right/d:$(git rev-parse base~1:right/d)\n+\t3:blob:right/c:$(git rev-parse base~2:right/c)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse base~2:left)\n+\t5:blob:left/b:$(git rev-parse base~2:left/b)\n+\t6:blob:a:$(git rev-parse base~2:a)\n+\tblobs:5\n+\ttrees:7\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base' '\n+\ttest-tool path-walk -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t2:blob:right/d:$(git rev-parse topic:right/d)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse topic:left)\n+\t5:blob:left/b:$(git rev-parse topic:left/b)\n+\t6:blob:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, boundary' '\n+\ttest-tool path-walk -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t0:tree::$(git rev-parse base~1^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t1:tree:right/:$(git rev-parse base~1:right)\n+\t2:blob:right/d:$(git rev-parse base~1:right/d)\n+\t3:blob:right/c:$(git rev-parse base~1:right/c)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse base~1:left)\n+\t5:blob:left/b:$(git rev-parse base~1:left/b)\n+\t6:blob:a:$(git rev-parse base~1:a)\n+\tblobs:5\n+\ttrees:5\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"508799","messageId":"42e71e6285f6a6976aee4ce04f1285bcb0305223.1733514359.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 4/7] path-walk: allow consumer to specify object types","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:55Z","receivedAt":"2024-12-06T19:46:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe add the ability to filter the object types in the path-walk API so\nthe callback function is called fewer times.\n\nThis adds the ability to ask for the commits in a list, as well. We\nre-use the empty string for this set of objects because these are passed\ndirectly to the callback function instead of being part of the\n'path_stack'.\n\nFuture changes will add the ability to visit annotated tags.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 9 ++\n path-walk.c | 33 ++++-\n path-walk.h | 14 +-\n t/helper/test-path-walk.c | 15 ++-\n t/t6601-path-walk.sh | 149 +++++++++++++++-------\n 5 files changed, 170 insertions(+), 50 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 662162ec70b..dce553b6114 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,6 +39,15 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n+`commits`, `blobs`, `trees`::\n+\tBy default, these members are enabled and signal that the path-walk\n+\tAPI should call the `path_fn` on objects of these types. Specialized\n+\tapplications could disable some options to make it simpler to walk\n+\tthe objects or to have fewer calls to `path_fn`.\n++\n+While it is possible to walk only commits in this way, consumers would be\n+better off using the revision walk API instead.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex 24cf04c1e7d..2ca08402367 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -98,6 +98,10 @@ static int add_children(struct path_walk_context *ctx,\n \t\tif (S_ISGITLINK(entry.mode))\n \t\t\tcontinue;\n \n+\t\t/* If the caller doesn't want blobs, then don't bother. */\n+\t\tif (!ctx->info->blobs && type == OBJ_BLOB)\n+\t\t\tcontinue;\n+\n \t\tif (type == OBJ_TREE) {\n \t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n \t\t\to = child ? &child->object : NULL;\n@@ -157,9 +161,11 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n-\t/* Evaluate function pointer on this data. */\n-\tret = ctx->info->path_fn(path, &list->oids, list->type,\n-\t\t\t\t ctx->info->path_fn_data);\n+\t/* Evaluate function pointer on this data, if requested. */\n+\tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n+\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t\tctx->info->path_fn_data);\n \n \t/* Expand data for children. */\n \tif (list->type == OBJ_TREE) {\n@@ -201,6 +207,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n \tstruct type_and_oid_list *root_tree_list;\n+\tstruct type_and_oid_list *commit_list;\n \tstruct path_walk_context ctx = {\n \t\t.repo = info->revs->repo,\n \t\t.revs = info->revs,\n@@ -212,6 +219,9 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\tCALLOC_ARRAY(commit_list, 1);\n+\tcommit_list->type = OBJ_COMMIT;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n@@ -222,10 +232,18 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\tdie(_(\"failed to setup revision walk\"));\n \n \twhile ((c = get_revision(info->revs))) {\n-\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct object_id *oid;\n \t\tstruct tree *t;\n \t\tcommits_nr++;\n \n+\t\tif (info->commits)\n+\t\t\toid_array_append(&commit_list->oids,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t/* If we only care about commits, then skip trees. */\n+\t\tif (!info->trees && !info->blobs)\n+\t\t\tcontinue;\n+\n \t\toid = get_commit_tree_oid(c);\n \t\tt = lookup_tree(info->revs->repo, oid);\n \n@@ -243,6 +261,13 @@ int walk_objects_by_path(struct path_walk_info *info)\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n \ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\t/* Track all commits. */\n+\tif (info->commits && commit_list->oids.nr)\n+\t\tret = info->path_fn(\"\", &commit_list->oids, OBJ_COMMIT,\n+\t\t\t\t info->path_fn_data);\n+\toid_array_clear(&commit_list->oids);\n+\tfree(commit_list);\n+\n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n \twhile (!ret && ctx.path_stack.nr) {\n \t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\ndiff --git a/path-walk.h b/path-walk.h\nindex c9e94a98bc8..2d2afc29b47 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -30,9 +30,21 @@ struct path_walk_info {\n \t */\n \tpath_fn path_fn;\n \tvoid *path_fn_data;\n+\n+\t/**\n+\t * Initialize which object types the path_fn should be called on. This\n+\t * could also limit the walk to skip blobs if not set.\n+\t */\n+\tint commits;\n+\tint trees;\n+\tint blobs;\n };\n \n-#define PATH_WALK_INFO_INIT { 0 }\n+#define PATH_WALK_INFO_INIT { \\\n+\t.blobs = 1,\t\t\\\n+\t.trees = 1,\t\t\\\n+\t.commits = 1,\t\t\\\n+}\n \n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex def7c81ac4f..a57a05a6391 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -19,6 +19,8 @@ static const char * const path_walk_usage[] = {\n \n struct path_walk_test_data {\n \tuintmax_t batch_nr;\n+\n+\tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n };\n@@ -33,6 +35,8 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->tree_nr += oids->nr;\n \telse if (type == OBJ_BLOB)\n \t\ttdata->blob_nr += oids->nr;\n+\telse if (type == OBJ_COMMIT)\n+\t\ttdata->commit_nr += oids->nr;\n \telse\n \t\tBUG(\"we do not understand this type\");\n \n@@ -54,6 +58,12 @@ int cmd__path_walk(int argc, const char **argv)\n \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n \tstruct path_walk_test_data data = { 0 };\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"blobs\", &info.blobs,\n+\t\t\t N_(\"toggle inclusion of blob objects\")),\n+\t\tOPT_BOOL(0, \"commits\", &info.commits,\n+\t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"trees\", &info.trees,\n+\t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -75,9 +85,10 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tres = walk_objects_by_path(&info);\n \n-\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\tprintf(\"commits:%\" PRIuMAX \"\\n\"\n+\t \"trees:%\" PRIuMAX \"\\n\"\n \t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.tree_nr, data.blob_nr);\n+\t data.commit_nr, data.tree_nr, data.blob_nr);\n \n \trelease_revisions(&revs);\n \treturn res;\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 4e052c09309..4a4939a1b02 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -33,22 +33,27 @@ test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:tree::$(git rev-parse topic^{tree})\n-\t0:tree::$(git rev-parse base^{tree})\n-\t0:tree::$(git rev-parse base~1^{tree})\n-\t0:tree::$(git rev-parse base~2^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t1:tree:right/:$(git rev-parse base~1:right)\n-\t1:tree:right/:$(git rev-parse base~2:right)\n-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n-\t3:blob:right/c:$(git rev-parse base~2:right/c)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse base:left)\n-\t4:tree:left/:$(git rev-parse base~2:left)\n-\t5:blob:left/b:$(git rev-parse base~2:left/b)\n-\t5:blob:left/b:$(git rev-parse base:left/b)\n-\t6:blob:a:$(git rev-parse base~2:a)\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base)\n+\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~2)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree})\n+\t1:tree::$(git rev-parse base~2^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right)\n+\t2:tree:right/:$(git rev-parse base~2:right)\n+\t3:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/c:$(git rev-parse base~2:right/c)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse base:left)\n+\t5:tree:left/:$(git rev-parse base~2:left)\n+\t6:blob:left/b:$(git rev-parse base~2:left/b)\n+\t6:blob:left/b:$(git rev-parse base:left/b)\n+\t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:6\n+\tcommits:4\n \ttrees:9\n \tEOF\n \n@@ -59,19 +64,23 @@ test_expect_success 'topic only' '\n \ttest-tool path-walk -- topic >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:tree::$(git rev-parse topic^{tree})\n-\t0:tree::$(git rev-parse base~1^{tree})\n-\t0:tree::$(git rev-parse base~2^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t1:tree:right/:$(git rev-parse base~1:right)\n-\t1:tree:right/:$(git rev-parse base~2:right)\n-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n-\t3:blob:right/c:$(git rev-parse base~2:right/c)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse base~2:left)\n-\t5:blob:left/b:$(git rev-parse base~2:left/b)\n-\t6:blob:a:$(git rev-parse base~2:a)\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~2)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree})\n+\t1:tree::$(git rev-parse base~2^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right)\n+\t2:tree:right/:$(git rev-parse base~2:right)\n+\t3:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/c:$(git rev-parse base~2:right/c)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse base~2:left)\n+\t6:blob:left/b:$(git rev-parse base~2:left/b)\n+\t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:5\n+\tcommits:3\n \ttrees:7\n \tEOF\n \n@@ -82,15 +91,66 @@ test_expect_success 'topic, not base' '\n \ttest-tool path-walk -- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t3:blob:right/d:$(git rev-parse topic:right/d)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse topic:left)\n+\t6:blob:left/b:$(git rev-parse topic:left/b)\n+\t7:blob:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\tcommits:1\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only blobs' '\n+\ttest-tool path-walk --no-trees --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n+\ttrees:0\n+\t0:blob:right/d:$(git rev-parse topic:right/d)\n+\t1:blob:right/c:$(git rev-parse topic:right/c)\n+\t2:blob:left/b:$(git rev-parse topic:left/b)\n+\t3:blob:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+# No, this doesn't make a lot of sense for the path-walk API,\n+# but it is possible to do.\n+test_expect_success 'topic, not base, only commits' '\n+\ttest-tool path-walk --no-blobs --no-trees \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\tcommits:1\n+\ttrees:0\n+\tblobs:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only trees' '\n+\ttest-tool path-walk --no-blobs --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n \t0:tree::$(git rev-parse topic^{tree})\n \t1:tree:right/:$(git rev-parse topic:right)\n-\t2:blob:right/d:$(git rev-parse topic:right/d)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse topic:left)\n-\t5:blob:left/b:$(git rev-parse topic:left/b)\n-\t6:blob:a:$(git rev-parse topic:a)\n-\tblobs:4\n+\t2:tree:left/:$(git rev-parse topic:left)\n \ttrees:3\n+\tblobs:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -100,17 +160,20 @@ test_expect_success 'topic, not base, boundary' '\n \ttest-tool path-walk -- --boundary topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:tree::$(git rev-parse topic^{tree})\n-\t0:tree::$(git rev-parse base~1^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t1:tree:right/:$(git rev-parse base~1:right)\n-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n-\t3:blob:right/c:$(git rev-parse base~1:right/c)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse base~1:left)\n-\t5:blob:left/b:$(git rev-parse base~1:left/b)\n-\t6:blob:a:$(git rev-parse base~1:a)\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base~1)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right)\n+\t3:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/c:$(git rev-parse base~1:right/c)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse base~1:left)\n+\t6:blob:left/b:$(git rev-parse base~1:left/b)\n+\t7:blob:a:$(git rev-parse base~1:a)\n \tblobs:5\n+\tcommits:2\n \ttrees:5\n \tEOF\n \n-- \ngitgitgadget\n\n"},{"id":"508800","messageId":"a41f53f7ced7a403547ce1c239c5fbb5382cfb0f.1733514359.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 5/7] path-walk: visit tags and cached objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:56Z","receivedAt":"2024-12-06T19:46:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe rev_info that is specified for a path-walk traversal may specify\nvisiting tag refs (both lightweight and annotated) and also may specify\nindexed objects (blobs and trees). Update the path-walk API to walk\nthese objects as well.\n\nWhen walking tags, we need to peel the annotated objects until reaching\na non-tag object. If we reach a commit, then we can add it to the\npending objects to make sure we visit in the commit walk portion. If we\nreach a tree, then we will assume that it is a root tree. If we reach a\nblob, then we have no good path name and so add it to a new list of\n\"tagged blobs\".\n\nWhen the rev_info includes the \"--indexed-objects\" flag, then the\npending set includes blobs and trees found in the cache entries and\ncache-tree. The cache entries are usually blobs, though they could be\ntrees in the case of a sparse index. The cache-tree stores\npreviously-hashed tree objects but these are cleared out when staging\nobjects below those paths. We add tests that demonstrate this.\n\nThe indexed objects come with a non-NULL 'path' value in the pending\nitem. This allows us to prepopulate the 'path_to_lists' strmap with\nlists for these paths.\n\nThe tricky thing about this walk is that we will want to combine the\nindexed objects walk with the commit walk, especially in the future case\nof walking objects during a command like 'git repack'.\n\nWhenever possible, we want the objects from the index to be grouped with\nsimilar objects in history. We don't want to miss any paths that appear\nonly in the index and not in the commit history.\n\nThus, we need to be careful to let the path stack be populated initially\nwith only the root tree path (and possibly tags and tagged blobs) and go\nthrough the normal depth-first search. Afterwards, if there are other\npaths that are remaining in the paths_to_lists strmap, we should then\niterate through the stack and visit those objects recursively.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 2 +-\n path-walk.c | 174 +++++++++++++++++++-\n path-walk.h | 2 +\n t/helper/test-path-walk.c | 15 +-\n t/t6601-path-walk.sh | 186 +++++++++++++++++++---\n 5 files changed, 353 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex dce553b6114..6022c381b7c 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,7 +39,7 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n-`commits`, `blobs`, `trees`::\n+`commits`, `blobs`, `trees`, `tags`::\n \tBy default, these members are enabled and signal that the path-walk\n \tAPI should call the `path_fn` on objects of these types. Specialized\n \tapplications could disable some options to make it simpler to walk\ndiff --git a/path-walk.c b/path-walk.c\nindex 2ca08402367..a1f539dcd46 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -13,10 +13,13 @@\n #include \"revision.h\"\n #include \"string-list.h\"\n #include \"strmap.h\"\n+#include \"tag.h\"\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n \n+static const char *root_path = \"\";\n+\n struct type_and_oid_list\n {\n \tenum object_type type;\n@@ -158,12 +161,16 @@ static int walk_path(struct path_walk_context *ctx,\n \n \tlist = strmap_get(&ctx->paths_to_lists, path);\n \n+\tif (!list)\n+\t\tBUG(\"provided path '%s' that had no associated list\", path);\n+\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n-\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n+\t (list->type == OBJ_TAG && ctx->info->tags))\n \t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n \t\t\t\t\tctx->info->path_fn_data);\n \n@@ -194,6 +201,134 @@ static void clear_strmap(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static void setup_pending_objects(struct path_walk_info *info,\n+\t\t\t\t struct path_walk_context *ctx)\n+{\n+\tstruct type_and_oid_list *tags = NULL;\n+\tstruct type_and_oid_list *tagged_blobs = NULL;\n+\tstruct type_and_oid_list *root_tree_list = NULL;\n+\n+\tif (info->tags)\n+\t\tCALLOC_ARRAY(tags, 1);\n+\tif (info->blobs)\n+\t\tCALLOC_ARRAY(tagged_blobs, 1);\n+\tif (info->trees)\n+\t\troot_tree_list = strmap_get(&ctx->paths_to_lists, root_path);\n+\n+\t/*\n+\t * Pending objects include:\n+\t * * Commits at branch tips.\n+\t * * Annotated tags at tag tips.\n+\t * * Any kind of object at lightweight tag tips.\n+\t * * Trees and blobs in the index (with an associated path).\n+\t */\n+\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n+\t\tstruct object *obj = pending->item;\n+\n+\t\t/* Commits will be picked up by revision walk. */\n+\t\tif (obj->type == OBJ_COMMIT)\n+\t\t\tcontinue;\n+\n+\t\t/* Navigate annotated tag object chains. */\n+\t\twhile (obj->type == OBJ_TAG) {\n+\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n+\t\t\tif (!tag)\n+\t\t\t\tbreak;\n+\t\t\tif (tag->object.flags & SEEN)\n+\t\t\t\tbreak;\n+\t\t\ttag->object.flags |= SEEN;\n+\n+\t\t\tif (tags)\n+\t\t\t\toid_array_append(&tags->oids, &obj->oid);\n+\t\t\tobj = tag->tagged;\n+\t\t}\n+\n+\t\tif (obj->type == OBJ_TAG)\n+\t\t\tcontinue;\n+\n+\t\t/* We are now at a non-tag object. */\n+\t\tif (obj->flags & SEEN)\n+\t\t\tcontinue;\n+\t\tobj->flags |= SEEN;\n+\n+\t\tswitch (obj->type) {\n+\t\tcase OBJ_TREE:\n+\t\t\tif (!info->trees)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = *pending->path ? xstrfmt(\"%s/\", pending->path)\n+\t\t\t\t\t\t\t : xstrdup(\"\");\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_TREE;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t\tfree(path);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&root_tree_list->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (!info->blobs)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = pending->path;\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_BLOB;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&tagged_blobs->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_COMMIT:\n+\t\t\t/* Make sure it is in the object walk */\n+\t\t\tif (obj != pending->item)\n+\t\t\t\tadd_pending_object(info->revs, obj, \"\");\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"should not see any other type here\");\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Add tag objects and tagged blobs if they exist.\n+\t */\n+\tif (tagged_blobs) {\n+\t\tif (tagged_blobs->oids.nr) {\n+\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n+\t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\tpush_to_stack(ctx, tagged_blob_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n+\t\t} else {\n+\t\t\toid_array_clear(&tagged_blobs->oids);\n+\t\t\tfree(tagged_blobs);\n+\t\t}\n+\t}\n+\tif (tags) {\n+\t\tif (tags->oids.nr) {\n+\t\t\tconst char *tag_path = \"/tags\";\n+\t\t\ttags->type = OBJ_TAG;\n+\t\t\tpush_to_stack(ctx, tag_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n+\t\t} else {\n+\t\t\toid_array_clear(&tags->oids);\n+\t\t\tfree(tags);\n+\t\t}\n+\t}\n+}\n+\n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\n * call 'info->path_fn' on each discovered path.\n@@ -202,7 +337,6 @@ static void clear_strmap(struct strmap *map)\n */\n int walk_objects_by_path(struct path_walk_info *info)\n {\n-\tconst char *root_path = \"\";\n \tint ret = 0;\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n@@ -222,15 +356,31 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(commit_list, 1);\n \tcommit_list->type = OBJ_COMMIT;\n \n+\tif (info->tags)\n+\t\tinfo->revs->tag_objects = 1;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n+\t/*\n+\t * Set these values before preparing the walk to catch\n+\t * lightweight tags pointing to non-commits and indexed objects.\n+\t */\n+\tinfo->revs->blob_objects = info->blobs;\n+\tinfo->revs->tree_objects = info->trees;\n+\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n+\n+\ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\tsetup_pending_objects(info, &ctx);\n+\ttrace2_region_leave(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\n \twhile ((c = get_revision(info->revs))) {\n \t\tstruct object_id *oid;\n \t\tstruct tree *t;\n@@ -278,6 +428,26 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \t\tfree(path);\n \t}\n+\n+\t/* Are there paths remaining? Likely they are from indexed objects. */\n+\tif (!strmap_empty(&ctx.paths_to_lists)) {\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *entry;\n+\n+\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry)\n+\t\t\tpush_to_stack(&ctx, entry->key);\n+\n+\t\twhile (!ret && ctx.path_stack.nr) {\n+\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\t\tctx.path_stack.nr--;\n+\t\t\tpaths_nr++;\n+\n+\t\t\tret = walk_path(&ctx, path);\n+\n+\t\t\tfree(path);\n+\t\t}\n+\t}\n+\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n \ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n \ndiff --git a/path-walk.h b/path-walk.h\nindex 2d2afc29b47..ca839f873e4 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -38,12 +38,14 @@ struct path_walk_info {\n \tint commits;\n \tint trees;\n \tint blobs;\n+\tint tags;\n };\n \n #define PATH_WALK_INFO_INIT { \\\n \t.blobs = 1,\t\t\\\n \t.trees = 1,\t\t\\\n \t.commits = 1,\t\t\\\n+\t.tags = 1,\t\t\\\n }\n \n /**\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex a57a05a6391..56289859e69 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -23,6 +23,7 @@ struct path_walk_test_data {\n \tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n+\tuintmax_t tag_nr;\n };\n \n static int emit_block(const char *path, struct oid_array *oids,\n@@ -37,11 +38,18 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->blob_nr += oids->nr;\n \telse if (type == OBJ_COMMIT)\n \t\ttdata->commit_nr += oids->nr;\n+\telse if (type == OBJ_TAG)\n+\t\ttdata->tag_nr += oids->nr;\n \telse\n \t\tBUG(\"we do not understand this type\");\n \n \ttypestr = type_name(type);\n \n+\t/* This should never be output during tests. */\n+\tif (!oids->nr)\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n+\t\t tdata->batch_nr, typestr, path);\n+\n \tfor (size_t i = 0; i < oids->nr; i++)\n \t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n \t\t tdata->batch_nr, typestr, path,\n@@ -62,6 +70,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of blob objects\")),\n \t\tOPT_BOOL(0, \"commits\", &info.commits,\n \t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"tags\", &info.tags,\n+\t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n@@ -87,8 +97,9 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tprintf(\"commits:%\" PRIuMAX \"\\n\"\n \t \"trees:%\" PRIuMAX \"\\n\"\n-\t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.commit_nr, data.tree_nr, data.blob_nr);\n+\t \"blobs:%\" PRIuMAX \"\\n\"\n+\t \"tags:%\" PRIuMAX \"\\n\",\n+\t data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);\n \n \trelease_revisions(&revs);\n \treturn res;\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 4a4939a1b02..1f3d2e0cb76 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -9,29 +9,142 @@ test_description='direct path-walk API tests'\n test_expect_success 'setup test repository' '\n \tgit checkout -b base &&\n \n+\t# Make some objects that will only be reachable\n+\t# via non-commit tags.\n+\tmkdir child &&\n+\techo file >child/file &&\n+\tgit add child &&\n+\tgit commit -m \"will abandon\" &&\n+\tgit tag -a -m \"tree\" tree-tag HEAD^{tree} &&\n+\techo file2 >file2 &&\n+\tgit add file2 &&\n+\tgit commit --amend -m \"will abandon\" &&\n+\tgit tag tree-tag2 HEAD^{tree} &&\n+\n+\techo blob >file &&\n+\tblob_oid=$(git hash-object -t blob -w --stdin file2 &&\n+\tblob2_oid=$(git hash-object -t blob -w --stdin a &&\n \techo b >left/b &&\n \techo c >right/c &&\n \tgit add . &&\n-\tgit commit -m \"first\" &&\n+\tgit commit --amend -m \"first\" &&\n+\tgit tag -m \"first\" first HEAD &&\n \n \techo d >right/d &&\n \tgit add right &&\n \tgit commit -m \"second\" &&\n+\tgit tag -a -m \"second (under)\" second.1 HEAD &&\n+\tgit tag -a -m \"second (top)\" second.2 second.1 &&\n \n+\t# Set up file/dir collision in history.\n+\trm a &&\n+\tmkdir a &&\n+\techo a >a/a &&\n \techo bb >left/b &&\n-\tgit commit -a -m \"third\" &&\n+\tgit add a left &&\n+\tgit commit -m \"third\" &&\n+\tgit tag -a -m \"third\" third &&\n \n \tgit checkout -b topic HEAD~1 &&\n \techo cc >right/c &&\n-\tgit commit -a -m \"topic\"\n+\tgit commit -a -m \"topic\" &&\n+\tgit tag -a -m \"fourth\" fourth\n '\n \n test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base)\n+\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~2)\n+\t1:tag:/tags:$(git rev-parse refs/tags/first)\n+\t1:tag:/tags:$(git rev-parse refs/tags/second.1)\n+\t1:tag:/tags:$(git rev-parse refs/tags/second.2)\n+\t1:tag:/tags:$(git rev-parse refs/tags/third)\n+\t1:tag:/tags:$(git rev-parse refs/tags/fourth)\n+\t1:tag:/tags:$(git rev-parse refs/tags/tree-tag)\n+\t1:tag:/tags:$(git rev-parse refs/tags/blob-tag)\n+\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n+\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\t3:tree::$(git rev-parse topic^{tree})\n+\t3:tree::$(git rev-parse base^{tree})\n+\t3:tree::$(git rev-parse base~1^{tree})\n+\t3:tree::$(git rev-parse base~2^{tree})\n+\t3:tree::$(git rev-parse refs/tags/tree-tag^{})\n+\t3:tree::$(git rev-parse refs/tags/tree-tag2^{})\n+\t4:blob:a:$(git rev-parse base~2:a)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right)\n+\t5:tree:right/:$(git rev-parse base~2:right)\n+\t6:blob:right/d:$(git rev-parse base~1:right/d)\n+\t7:blob:right/c:$(git rev-parse base~2:right/c)\n+\t7:blob:right/c:$(git rev-parse topic:right/c)\n+\t8:tree:left/:$(git rev-parse base:left)\n+\t8:tree:left/:$(git rev-parse base~2:left)\n+\t9:blob:left/b:$(git rev-parse base~2:left/b)\n+\t9:blob:left/b:$(git rev-parse base:left/b)\n+\t10:tree:a/:$(git rev-parse base:a)\n+\t11:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n+\t12:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n+\t13:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\tblobs:10\n+\tcommits:4\n+\ttags:7\n+\ttrees:13\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'indexed objects' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"left\" directory.\n+\techo bogus >left/c &&\n+\tgit add left &&\n+\n+\ttest-tool path-walk -- --indexed-objects >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:blob:a:$(git rev-parse HEAD:a)\n+\t1:blob:left/b:$(git rev-parse HEAD:left/b)\n+\t2:blob:left/c:$(git rev-parse :left/c)\n+\t3:blob:right/c:$(git rev-parse HEAD:right/c)\n+\t4:blob:right/d:$(git rev-parse HEAD:right/d)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\tblobs:5\n+\tcommits:0\n+\ttags:0\n+\ttrees:1\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'branches and indexed objects mix well' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"right\" directory.\n+\techo fake >right/d &&\n+\tgit add right &&\n+\n+\ttest-tool path-walk -- --indexed-objects --branches >out &&\n+\n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n \t0:commit::$(git rev-parse base)\n@@ -41,20 +154,23 @@ test_expect_success 'all' '\n \t1:tree::$(git rev-parse base^{tree})\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right)\n-\t2:tree:right/:$(git rev-parse base~2:right)\n-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/c:$(git rev-parse base~2:right/c)\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base:left)\n-\t5:tree:left/:$(git rev-parse base~2:left)\n-\t6:blob:left/b:$(git rev-parse base~2:left/b)\n-\t6:blob:left/b:$(git rev-parse base:left/b)\n-\t7:blob:a:$(git rev-parse base~2:a)\n-\tblobs:6\n+\t2:blob:a:$(git rev-parse base~2:a)\n+\t3:tree:right/:$(git rev-parse topic:right)\n+\t3:tree:right/:$(git rev-parse base~1:right)\n+\t3:tree:right/:$(git rev-parse base~2:right)\n+\t4:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/d:$(git rev-parse :right/d)\n+\t5:blob:right/c:$(git rev-parse base~2:right/c)\n+\t5:blob:right/c:$(git rev-parse topic:right/c)\n+\t6:tree:left/:$(git rev-parse base:left)\n+\t6:tree:left/:$(git rev-parse base~2:left)\n+\t7:blob:left/b:$(git rev-parse base:left/b)\n+\t7:blob:left/b:$(git rev-parse base~2:left/b)\n+\t8:tree:a/:$(git rev-parse refs/tags/third:a)\n+\tblobs:7\n \tcommits:4\n-\ttrees:9\n+\ttags:0\n+\ttrees:10\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -81,6 +197,7 @@ test_expect_success 'topic only' '\n \t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:5\n \tcommits:3\n+\ttags:0\n \ttrees:7\n \tEOF\n \n@@ -101,6 +218,7 @@ test_expect_success 'topic, not base' '\n \t7:blob:a:$(git rev-parse topic:a)\n \tblobs:4\n \tcommits:1\n+\ttags:0\n \ttrees:3\n \tEOF\n \n@@ -112,13 +230,14 @@ test_expect_success 'topic, not base, only blobs' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\tcommits:0\n-\ttrees:0\n \t0:blob:right/d:$(git rev-parse topic:right/d)\n \t1:blob:right/c:$(git rev-parse topic:right/c)\n \t2:blob:left/b:$(git rev-parse topic:left/b)\n \t3:blob:a:$(git rev-parse topic:a)\n \tblobs:4\n+\tcommits:0\n+\ttags:0\n+\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -133,8 +252,9 @@ test_expect_success 'topic, not base, only commits' '\n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n \tcommits:1\n-\ttrees:0\n \tblobs:0\n+\ttags:0\n+\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -145,12 +265,13 @@ test_expect_success 'topic, not base, only trees' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\tcommits:0\n \t0:tree::$(git rev-parse topic^{tree})\n \t1:tree:right/:$(git rev-parse topic:right)\n \t2:tree:left/:$(git rev-parse topic:left)\n-\ttrees:3\n+\tcommits:0\n \tblobs:0\n+\ttags:0\n+\ttrees:3\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -174,10 +295,33 @@ test_expect_success 'topic, not base, boundary' '\n \t7:blob:a:$(git rev-parse base~1:a)\n \tblobs:5\n \tcommits:2\n+\ttags:0\n \ttrees:5\n \tEOF\n \n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'trees are reported exactly once' '\n+\ttest_when_finished \"rm -rf unique-trees\" &&\n+\ttest_create_repo unique-trees &&\n+\t(\n+\t\tcd unique-trees &&\n+\t\tmkdir initial &&\n+\t\ttest_commit initial/file &&\n+\n+\t\tgit switch -c move-to-top &&\n+\t\tgit mv initial/file.t ./ &&\n+\t\ttest_tick &&\n+\t\tgit commit -m moved &&\n+\n+\t\tgit update-ref refs/heads/other HEAD\n+\t) &&\n+\n+\ttest-tool -C unique-trees path-walk -- --all >out &&\n+\ttree=$(git -C unique-trees rev-parse HEAD:) &&\n+\tgrep \"$tree\" out >out-filtered &&\n+\ttest_line_count = 1 out-filtered\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"508801","messageId":"0f1e6c51b2c2371fdb19dba6af9d0a0e353f0871.1733514359.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 6/7] path-walk: mark trees and blobs as UNINTERESTING","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:57Z","receivedAt":"2024-12-06T19:46:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the input rev_info has UNINTERESTING starting points, we want to be\nsure that the UNINTERESTING flag is passed appropriately through the\nobjects. To match how this is done in places such as 'git pack-objects', we\nuse the mark_edges_uninteresting() method.\n\nThis method has an option for using the \"sparse\" walk, which is similar in\nspirit to the path-walk API's walk. To be sure to keep it independent, add a\nnew 'prune_all_uninteresting' option to the path_walk_info struct.\n\nTo check how the UNINTERSTING flag is spread through our objects, extend the\n'test-tool path-walk' command to output whether or not an object has that\nflag. This changes our tests significantly, including the removal of some\nobjects that were previously visited due to the incomplete implementation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 8 +++\n path-walk.c | 73 +++++++++++++++++++++\n path-walk.h | 8 +++\n t/helper/test-path-walk.c | 12 +++-\n t/t6601-path-walk.sh | 79 +++++++++++++++++------\n 5 files changed, 158 insertions(+), 22 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 6022c381b7c..7075d0d5ab5 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -48,6 +48,14 @@ commits.\n While it is possible to walk only commits in this way, consumers would be\n better off using the revision walk API instead.\n \n+`prune_all_uninteresting`::\n+\tBy default, all reachable paths are emitted by the path-walk API.\n+\tThis option allows consumers to declare that they are not\n+\tinterested in paths where all included objects are marked with the\n+\t`UNINTERESTING` flag. This requires using the `boundary` option in\n+\tthe revision walk so that the walk emits commits marked with the\n+\t`UNINTERESTING` flag.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex a1f539dcd46..896ec0c4779 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -8,6 +8,7 @@\n #include \"dir.h\"\n #include \"hashmap.h\"\n #include \"hex.h\"\n+#include \"list-objects.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n #include \"revision.h\"\n@@ -24,6 +25,7 @@ struct type_and_oid_list\n {\n \tenum object_type type;\n \tstruct oid_array oids;\n+\tint maybe_interesting;\n };\n \n #define TYPE_AND_OID_LIST_INIT { \\\n@@ -140,6 +142,9 @@ static int add_children(struct path_walk_context *ctx,\n \t\tif (o->flags & SEEN)\n \t\t\tcontinue;\n \t\to->flags |= SEEN;\n+\n+\t\tif (!(o->flags & UNINTERESTING))\n+\t\t\tlist->maybe_interesting = 1;\n \t\toid_array_append(&list->oids, &entry.oid);\n \t}\n \n@@ -167,6 +172,43 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n+\tif (ctx->info->prune_all_uninteresting) {\n+\t\t/*\n+\t\t * This is true if all objects were UNINTERESTING\n+\t\t * when added to the list.\n+\t\t */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\n+\t\t/*\n+\t\t * But it's still possible that the objects were set\n+\t\t * as UNINTERESTING after being added. Do a quick check.\n+\t\t */\n+\t\tlist->maybe_interesting = 0;\n+\t\tfor (size_t i = 0;\n+\t\t !list->maybe_interesting && i < list->oids.nr;\n+\t\t i++) {\n+\t\t\tif (list->type == OBJ_TREE) {\n+\t\t\t\tstruct tree *t = lookup_tree(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (t && !(t->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else if (list->type == OBJ_BLOB) {\n+\t\t\t\tstruct blob *b = lookup_blob(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (b && !(b->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else {\n+\t\t\t\t/* Tags are always interesting if visited. */\n+\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* We have confirmed that all objects are UNINTERESTING. */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\t}\n+\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n \t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n@@ -201,6 +243,26 @@ static void clear_strmap(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static struct repository *edge_repo;\n+static struct type_and_oid_list *edge_tree_list;\n+\n+static void show_edge(struct commit *commit)\n+{\n+\tstruct tree *t = repo_get_commit_tree(edge_repo, commit);\n+\n+\tif (!t)\n+\t\treturn;\n+\n+\tif (commit->object.flags & UNINTERESTING)\n+\t\tt->object.flags |= UNINTERESTING;\n+\n+\tif (t->object.flags & SEEN)\n+\t\treturn;\n+\tt->object.flags |= SEEN;\n+\n+\toid_array_append(&edge_tree_list->oids, &t->object.oid);\n+}\n+\n static void setup_pending_objects(struct path_walk_info *info,\n \t\t\t\t struct path_walk_context *ctx)\n {\n@@ -309,6 +371,7 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\tif (tagged_blobs->oids.nr) {\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\ttagged_blobs->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n \t\t} else {\n@@ -320,6 +383,7 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\tif (tags->oids.nr) {\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n+\t\t\ttags->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tag_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n \t\t} else {\n@@ -362,6 +426,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n+\troot_tree_list->maybe_interesting = 1;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n@@ -375,6 +440,14 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\t/* Walk trees to mark them as UNINTERESTING. */\n+\tedge_repo = info->revs->repo;\n+\tedge_tree_list = root_tree_list;\n+\tmark_edges_uninteresting(info->revs, show_edge,\n+\t\t\t\t info->prune_all_uninteresting);\n+\tedge_repo = NULL;\n+\tedge_tree_list = NULL;\n+\n \tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n \n \ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\ndiff --git a/path-walk.h b/path-walk.h\nindex ca839f873e4..de0db007dc9 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -39,6 +39,14 @@ struct path_walk_info {\n \tint trees;\n \tint blobs;\n \tint tags;\n+\n+\t/**\n+\t * When 'prune_all_uninteresting' is set and a path has all objects\n+\t * marked as UNINTERESTING, then the path-walk will not visit those\n+\t * objects. It will not call path_fn on those objects and will not\n+\t * walk the children of such trees.\n+\t */\n+\tint prune_all_uninteresting;\n };\n \n #define PATH_WALK_INFO_INIT { \\\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 56289859e69..7f2d409c5bc 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -50,10 +50,14 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n \t\t tdata->batch_nr, typestr, path);\n \n-\tfor (size_t i = 0; i < oids->nr; i++)\n-\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tstruct object *o = lookup_unknown_object(the_repository,\n+\t\t\t\t\t\t\t &oids->oid[i]);\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s%s\\n\",\n \t\t tdata->batch_nr, typestr, path,\n-\t\t oid_to_hex(&oids->oid[i]));\n+\t\t oid_to_hex(&oids->oid[i]),\n+\t\t o->flags & UNINTERESTING ? \":UNINTERESTING\" : \"\");\n+\t}\n \n \ttdata->batch_nr++;\n \treturn 0;\n@@ -74,6 +78,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n+\t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n+\t\t\t N_(\"toggle pruning of uninteresting paths\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 1f3d2e0cb76..a317cdf289e 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -211,11 +211,11 @@ test_expect_success 'topic, not base' '\n \t0:commit::$(git rev-parse topic)\n \t1:tree::$(git rev-parse topic^{tree})\n \t2:tree:right/:$(git rev-parse topic:right)\n-\t3:blob:right/d:$(git rev-parse topic:right/d)\n+\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse topic:left)\n-\t6:blob:left/b:$(git rev-parse topic:left/b)\n-\t7:blob:a:$(git rev-parse topic:a)\n+\t5:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t6:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t7:blob:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:1\n \ttags:0\n@@ -225,15 +225,38 @@ test_expect_success 'topic, not base' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'fourth, blob-tag2, not base' '\n+\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tag:/tags:$(git rev-parse fourth)\n+\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\t3:tree::$(git rev-parse topic^{tree})\n+\t4:tree:right/:$(git rev-parse topic:right)\n+\t5:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t8:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t9:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tblobs:5\n+\tcommits:1\n+\ttags:1\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'topic, not base, only blobs' '\n \ttest-tool path-walk --no-trees --no-commits \\\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:blob:right/d:$(git rev-parse topic:right/d)\n+\t0:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \t1:blob:right/c:$(git rev-parse topic:right/c)\n-\t2:blob:left/b:$(git rev-parse topic:left/b)\n-\t3:blob:a:$(git rev-parse topic:a)\n+\t2:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t3:blob:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:0\n \ttags:0\n@@ -267,7 +290,7 @@ test_expect_success 'topic, not base, only trees' '\n \tcat >expect <<-EOF &&\n \t0:tree::$(git rev-parse topic^{tree})\n \t1:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:left/:$(git rev-parse topic:left)\n+\t2:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n \tcommits:0\n \tblobs:0\n \ttags:0\n@@ -282,17 +305,17 @@ test_expect_success 'topic, not base, boundary' '\n \n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n-\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~1):UNINTERESTING\n \t1:tree::$(git rev-parse topic^{tree})\n-\t1:tree::$(git rev-parse base~1^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n \t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right)\n-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/c:$(git rev-parse base~1:right/c)\n+\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t3:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\t4:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n \t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base~1:left)\n-\t6:blob:left/b:$(git rev-parse base~1:left/b)\n-\t7:blob:a:$(git rev-parse base~1:a)\n+\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t7:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n \tblobs:5\n \tcommits:2\n \ttags:0\n@@ -302,6 +325,27 @@ test_expect_success 'topic, not base, boundary' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, boundary with pruning' '\n+\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base~1):UNINTERESTING\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\tcommits:2\n+\ttags:0\n+\ttrees:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n@@ -309,15 +353,12 @@ test_expect_success 'trees are reported exactly once' '\n \t\tcd unique-trees &&\n \t\tmkdir initial &&\n \t\ttest_commit initial/file &&\n-\n \t\tgit switch -c move-to-top &&\n \t\tgit mv initial/file.t ./ &&\n \t\ttest_tick &&\n \t\tgit commit -m moved &&\n-\n \t\tgit update-ref refs/heads/other HEAD\n \t) &&\n-\n \ttest-tool -C unique-trees path-walk -- --all >out &&\n \ttree=$(git -C unique-trees rev-parse HEAD:) &&\n \tgrep \"$tree\" out >out-filtered &&\n-- \ngitgitgadget\n\n"},{"id":"508802","messageId":"e716672c041473dd2bf257b7532b86696fef32a0.1733514359.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v3 7/7] path-walk: reorder object visits","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-06T19:45:58Z","receivedAt":"2024-12-06T19:46:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe path-walk API currently uses a stack-based approach to recursing\nthrough the list of paths within the repository. This guarantees that\nafter a tree path is explored, all paths contained within that tree path\nwill be explored before continuing to explore siblings of that tree\npath.\n\nThe initial motivation of this depth-first approach was to minimize\nmemory pressure while exploring the repository. A breadth-first approach\nwould have too many \"active\" paths being stored in the paths_to_lists\nmap.\n\nWe can take this approach one step further by making sure that blob\npaths are visited before tree paths. This allows the API to free the\nmemory for these blob objects before continuing to perform the\ndepth-first search. This modifies the order in which we visit siblings,\nbut does not change the fact that we are performing depth-first search.\n\nTo achieve this goal, use a priority queue with a custom sorting method.\nThe sort needs to handle tags, blobs, and trees (commits are handled\nslightly differently). When objects share a type, we can sort by path\nname. This will keep children of the latest path to leave the stack be\npreferred over the rest of the paths in the stack, since they agree in\nprefix up to and including a directory separator. When the types are\ndifferent, we can prefer tags over other types and blobs over trees.\n\nThis causes significant adjustments to t6601-path-walk.sh to rearrange\nthe order of the visited paths.\n\nSigned-off-by: Derrick Stolee \n---\n path-walk.c | 60 ++++++++++++++++-----\n t/t6601-path-walk.sh | 122 +++++++++++++++++++++----------------------\n 2 files changed, 109 insertions(+), 73 deletions(-)\n\ndiff --git a/path-walk.c b/path-walk.c\nindex 896ec0c4779..b31924df52e 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -11,6 +11,7 @@\n #include \"list-objects.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n+#include \"prio-queue.h\"\n #include \"revision.h\"\n #include \"string-list.h\"\n #include \"strmap.h\"\n@@ -50,16 +51,50 @@ struct path_walk_context {\n \tstruct strmap paths_to_lists;\n \n \t/**\n-\t * Store the current list of paths in a stack, to\n-\t * facilitate depth-first-search without recursion.\n+\t * Store the current list of paths in a priority queue,\n+\t * using object type as a sorting mechanism, mostly to\n+\t * make sure blobs are popped off the stack first. No\n+\t * other sort is made, so within each object type it acts\n+\t * like a stack and performs a DFS within the trees.\n \t *\n \t * Use path_stack_pushed to indicate whether a path\n \t * was previously added to path_stack.\n \t */\n-\tstruct string_list path_stack;\n+\tstruct prio_queue path_stack;\n \tstruct strset path_stack_pushed;\n };\n \n+static int compare_by_type(const void *one, const void *two, void *cb_data)\n+{\n+\tstruct type_and_oid_list *list1, *list2;\n+\tconst char *str1 = one;\n+\tconst char *str2 = two;\n+\tstruct path_walk_context *ctx = cb_data;\n+\n+\tlist1 = strmap_get(&ctx->paths_to_lists, str1);\n+\tlist2 = strmap_get(&ctx->paths_to_lists, str2);\n+\n+\t/*\n+\t * If object types are equal, then use path comparison.\n+\t */\n+\tif (!list1 || !list2 || list1->type == list2->type)\n+\t\treturn strcmp(str1, str2);\n+\n+\t/* Prefer tags to be popped off first. */\n+\tif (list1->type == OBJ_TAG)\n+\t\treturn -1;\n+\tif (list2->type == OBJ_TAG)\n+\t\treturn 1;\n+\n+\t/* Prefer blobs to be popped off second. */\n+\tif (list1->type == OBJ_BLOB)\n+\t\treturn -1;\n+\tif (list2->type == OBJ_BLOB)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n static void push_to_stack(struct path_walk_context *ctx,\n \t\t\t const char *path)\n {\n@@ -67,7 +102,7 @@ static void push_to_stack(struct path_walk_context *ctx,\n \t\treturn;\n \n \tstrset_add(&ctx->path_stack_pushed, path);\n-\tstring_list_append(&ctx->path_stack, path);\n+\tprio_queue_put(&ctx->path_stack, xstrdup(path));\n }\n \n static int add_children(struct path_walk_context *ctx,\n@@ -372,8 +407,8 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n \t\t\ttagged_blobs->maybe_interesting = 1;\n-\t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n+\t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t} else {\n \t\t\toid_array_clear(&tagged_blobs->oids);\n \t\t\tfree(tagged_blobs);\n@@ -384,8 +419,8 @@ static void setup_pending_objects(struct path_walk_info *info,\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n \t\t\ttags->maybe_interesting = 1;\n-\t\t\tpush_to_stack(ctx, tag_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n+\t\t\tpush_to_stack(ctx, tag_path);\n \t\t} else {\n \t\t\toid_array_clear(&tags->oids);\n \t\t\tfree(tags);\n@@ -410,7 +445,10 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\t.repo = info->revs->repo,\n \t\t.revs = info->revs,\n \t\t.info = info,\n-\t\t.path_stack = STRING_LIST_INIT_DUP,\n+\t\t.path_stack = {\n+\t\t\t.compare = compare_by_type,\n+\t\t\t.cb_data = &ctx\n+\t\t},\n \t\t.path_stack_pushed = STRSET_INIT,\n \t\t.paths_to_lists = STRMAP_INIT\n \t};\n@@ -493,8 +531,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n \twhile (!ret && ctx.path_stack.nr) {\n-\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n-\t\tctx.path_stack.nr--;\n+\t\tchar *path = prio_queue_get(&ctx.path_stack);\n \t\tpaths_nr++;\n \n \t\tret = walk_path(&ctx, path);\n@@ -511,8 +548,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\t\tpush_to_stack(&ctx, entry->key);\n \n \t\twhile (!ret && ctx.path_stack.nr) {\n-\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n-\t\t\tctx.path_stack.nr--;\n+\t\t\tchar *path = prio_queue_get(&ctx.path_stack);\n \t\t\tpaths_nr++;\n \n \t\t\tret = walk_path(&ctx, path);\n@@ -526,6 +562,6 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \tclear_strmap(&ctx.paths_to_lists);\n \tstrset_clear(&ctx.path_stack_pushed);\n-\tstring_list_clear(&ctx.path_stack, 0);\n+\tclear_prio_queue(&ctx.path_stack);\n \treturn ret;\n }\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex a317cdf289e..7d765ffe907 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -84,20 +84,20 @@ test_expect_success 'all' '\n \t3:tree::$(git rev-parse refs/tags/tree-tag^{})\n \t3:tree::$(git rev-parse refs/tags/tree-tag2^{})\n \t4:blob:a:$(git rev-parse base~2:a)\n-\t5:tree:right/:$(git rev-parse topic:right)\n-\t5:tree:right/:$(git rev-parse base~1:right)\n-\t5:tree:right/:$(git rev-parse base~2:right)\n-\t6:blob:right/d:$(git rev-parse base~1:right/d)\n-\t7:blob:right/c:$(git rev-parse base~2:right/c)\n-\t7:blob:right/c:$(git rev-parse topic:right/c)\n-\t8:tree:left/:$(git rev-parse base:left)\n-\t8:tree:left/:$(git rev-parse base~2:left)\n-\t9:blob:left/b:$(git rev-parse base~2:left/b)\n-\t9:blob:left/b:$(git rev-parse base:left/b)\n-\t10:tree:a/:$(git rev-parse base:a)\n-\t11:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n-\t12:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n-\t13:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\t5:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n+\t6:tree:a/:$(git rev-parse base:a)\n+\t7:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n+\t8:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\t9:tree:left/:$(git rev-parse base:left)\n+\t9:tree:left/:$(git rev-parse base~2:left)\n+\t10:blob:left/b:$(git rev-parse base~2:left/b)\n+\t10:blob:left/b:$(git rev-parse base:left/b)\n+\t11:tree:right/:$(git rev-parse topic:right)\n+\t11:tree:right/:$(git rev-parse base~1:right)\n+\t11:tree:right/:$(git rev-parse base~2:right)\n+\t12:blob:right/c:$(git rev-parse base~2:right/c)\n+\t12:blob:right/c:$(git rev-parse topic:right/c)\n+\t13:blob:right/d:$(git rev-parse base~1:right/d)\n \tblobs:10\n \tcommits:4\n \ttags:7\n@@ -155,18 +155,18 @@ test_expect_success 'branches and indexed objects mix well' '\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n \t2:blob:a:$(git rev-parse base~2:a)\n-\t3:tree:right/:$(git rev-parse topic:right)\n-\t3:tree:right/:$(git rev-parse base~1:right)\n-\t3:tree:right/:$(git rev-parse base~2:right)\n-\t4:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/d:$(git rev-parse :right/d)\n-\t5:blob:right/c:$(git rev-parse base~2:right/c)\n-\t5:blob:right/c:$(git rev-parse topic:right/c)\n-\t6:tree:left/:$(git rev-parse base:left)\n-\t6:tree:left/:$(git rev-parse base~2:left)\n-\t7:blob:left/b:$(git rev-parse base:left/b)\n-\t7:blob:left/b:$(git rev-parse base~2:left/b)\n-\t8:tree:a/:$(git rev-parse refs/tags/third:a)\n+\t3:tree:a/:$(git rev-parse refs/tags/third:a)\n+\t4:tree:left/:$(git rev-parse base:left)\n+\t4:tree:left/:$(git rev-parse base~2:left)\n+\t5:blob:left/b:$(git rev-parse base:left/b)\n+\t5:blob:left/b:$(git rev-parse base~2:left/b)\n+\t6:tree:right/:$(git rev-parse topic:right)\n+\t6:tree:right/:$(git rev-parse base~1:right)\n+\t6:tree:right/:$(git rev-parse base~2:right)\n+\t7:blob:right/c:$(git rev-parse base~2:right/c)\n+\t7:blob:right/c:$(git rev-parse topic:right/c)\n+\t8:blob:right/d:$(git rev-parse base~1:right/d)\n+\t8:blob:right/d:$(git rev-parse :right/d)\n \tblobs:7\n \tcommits:4\n \ttags:0\n@@ -186,15 +186,15 @@ test_expect_success 'topic only' '\n \t1:tree::$(git rev-parse topic^{tree})\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right)\n-\t2:tree:right/:$(git rev-parse base~2:right)\n-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/c:$(git rev-parse base~2:right/c)\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base~2:left)\n-\t6:blob:left/b:$(git rev-parse base~2:left/b)\n-\t7:blob:a:$(git rev-parse base~2:a)\n+\t2:blob:a:$(git rev-parse base~2:a)\n+\t3:tree:left/:$(git rev-parse base~2:left)\n+\t4:blob:left/b:$(git rev-parse base~2:left/b)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right)\n+\t5:tree:right/:$(git rev-parse base~2:right)\n+\t6:blob:right/c:$(git rev-parse base~2:right/c)\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse base~1:right/d)\n \tblobs:5\n \tcommits:3\n \ttags:0\n@@ -210,12 +210,12 @@ test_expect_success 'topic, not base' '\n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n \t1:tree::$(git rev-parse topic^{tree})\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n-\t6:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n-\t7:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t2:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t3:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t4:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \tblobs:4\n \tcommits:1\n \ttags:0\n@@ -233,12 +233,12 @@ test_expect_success 'fourth, blob-tag2, not base' '\n \t1:tag:/tags:$(git rev-parse fourth)\n \t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n \t3:tree::$(git rev-parse topic^{tree})\n-\t4:tree:right/:$(git rev-parse topic:right)\n-\t5:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n-\t6:blob:right/c:$(git rev-parse topic:right/c)\n-\t7:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n-\t8:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n-\t9:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t4:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t7:tree:right/:$(git rev-parse topic:right)\n+\t8:blob:right/c:$(git rev-parse topic:right/c)\n+\t9:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:5\n \tcommits:1\n \ttags:1\n@@ -253,10 +253,10 @@ test_expect_success 'topic, not base, only blobs' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n-\t1:blob:right/c:$(git rev-parse topic:right/c)\n-\t2:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n-\t3:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t0:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t1:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t2:blob:right/c:$(git rev-parse topic:right/c)\n+\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \tblobs:4\n \tcommits:0\n \ttags:0\n@@ -289,8 +289,8 @@ test_expect_success 'topic, not base, only trees' '\n \n \tcat >expect <<-EOF &&\n \t0:tree::$(git rev-parse topic^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t1:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n \tcommits:0\n \tblobs:0\n \ttags:0\n@@ -308,14 +308,14 @@ test_expect_success 'topic, not base, boundary' '\n \t0:commit::$(git rev-parse base~1):UNINTERESTING\n \t1:tree::$(git rev-parse topic^{tree})\n \t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n-\t3:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n-\t4:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n-\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n-\t7:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t2:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t3:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t4:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t6:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:5\n \tcommits:2\n \ttags:0\n-- \ngitgitgadget\n"},{"id":"509093","messageId":"Z1whbcb71srg3hju@pks.im","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/7] PATH WALK I: The path-walk API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T11:58:37Z","receivedAt":"2024-12-13T11:58:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 06, 2024 at 07:45:51PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n> Introduction and relation to prior series\n> =========================================\n\nSorry for being late to the party -- I wanted to review this series a\nlot earlier, but never really found the time to do so. The patches\nmostly look good to me. I've got a couple of nits on the first patch and\nthink that the error handling could be stricter so that we don't ignore\nanay kind of data corruption during the walk. But other than that I'm\nlooking forward to the usecases this API will enable.\n\nThanks!\n\nPatrick\n"},{"id":"509094","messageId":"Z1whcUJE-MHAhULO@pks.im","threadId":"62429","inReplyTo":"b7e9b81e8b32313f00d38257ba731e73d17224cb.1733514358.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/7] path-walk: introduce an object walk by path","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T11:58:41Z","receivedAt":"2024-12-13T11:59:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 06, 2024 at 07:45:52PM +0000, Derrick Stolee via GitGitGadget wrote:\n> --- /dev/null\n> +++ b/path-walk.c\n> @@ -0,0 +1,263 @@\n> +/*\n> + * path-walk.c: implementation for path-based walks of the object graph.\n> + */\n> +#include \"git-compat-util.h\"\n> +#include \"path-walk.h\"\n> +#include \"blob.h\"\n> +#include \"commit.h\"\n> +#include \"dir.h\"\n> +#include \"hashmap.h\"\n> +#include \"hex.h\"\n> +#include \"object.h\"\n> +#include \"oid-array.h\"\n> +#include \"revision.h\"\n> +#include \"string-list.h\"\n> +#include \"strmap.h\"\n> +#include \"trace2.h\"\n> +#include \"tree.h\"\n> +#include \"tree-walk.h\"\n> +\n> +struct type_and_oid_list\n> +{\n> +\tenum object_type type;\n> +\tstruct oid_array oids;\n> +};\n\nNit: formatting of this struct is off.\n\n> +static void push_to_stack(struct path_walk_context *ctx,\n> +\t\t\t const char *path)\n> +{\n> +\tif (strset_contains(&ctx->path_stack_pushed, path))\n> +\t\treturn;\n> +\n> +\tstrset_add(&ctx->path_stack_pushed, path);\n> +\tstring_list_append(&ctx->path_stack, path);\n> +}\n> +\n> +static int add_children(struct path_walk_context *ctx,\n> +\t\t\tconst char *base_path,\n> +\t\t\tstruct object_id *oid)\n> +{\n\nSo this function assumes that `oid` always refers to a tree? I think it\nwould make sense to clarify this by calling the function accordingly,\nlike e.g. `add_tree_entries()`.\n\n> +\tstruct tree_desc desc;\n> +\tstruct name_entry entry;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tsize_t base_len;\n> +\tstruct tree *tree = lookup_tree(ctx->repo, oid);\n> +\n> +\tif (!tree) {\n> +\t\terror(_(\"failed to walk children of tree %s: not found\"),\n> +\t\t oid_to_hex(oid));\n> +\t\treturn -1;\n> +\t} else if (parse_tree_gently(tree, 1)) {\n> +\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n\nI wonder whether we maybe shouldn't die but instead return an error in\nthe spirit of libification.\n\n> +\t}\n> +\tstrbuf_addstr(&path, base_path);\n> +\tbase_len = path.len;\n> +\n> +\tparse_tree(tree);\n> +\tinit_tree_desc(&desc, &tree->object.oid, tree->buffer, tree->size);\n> +\twhile (tree_entry(&desc, &entry)) {\n> +\t\tstruct type_and_oid_list *list;\n> +\t\tstruct object *o;\n> +\t\t/* Not actually true, but we will ignore submodules later. */\n> +\t\tenum object_type type = S_ISDIR(entry.mode) ? OBJ_TREE : OBJ_BLOB;\n> +\n> +\t\t/* Skip submodules. */\n> +\t\tif (S_ISGITLINK(entry.mode))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (type == OBJ_TREE) {\n> +\t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n> +\t\t\to = child ? &child->object : NULL;\n> +\t\t} else if (type == OBJ_BLOB) {\n> +\t\t\tstruct blob *child = lookup_blob(ctx->repo, &entry.oid);\n> +\t\t\to = child ? &child->object : NULL;\n> +\t\t} else {\n> +\t\t\t/* Wrong type? */\n> +\t\t\tcontinue;\n\nThis code is unreachable, so we could make this a `BUG()`. Might also\nuse a switch instead, but that's more of a stylistic question.\n\n> +\t\t}\n> +\n> +\t\tif (!o) /* report error?*/\n> +\t\t\tcontinue;\n\nSo this can only happen in case `lookup_tree()` or `lookup_blob()`\nrun into an error. I think this error should definitely be bubbled up so\nthat we don't silently skip tree entries in case of repo corruption.\n\n> +\t\tstrbuf_setlen(&path, base_len);\n> +\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n> +\n> +\t\t/*\n> +\t\t * Trees will end with \"/\" for concatenation and distinction\n> +\t\t * from blobs at the same path.\n> +\t\t */\n> +\t\tif (type == OBJ_TREE)\n> +\t\t\tstrbuf_addch(&path, '/');\n> +\n> +\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {\n> +\t\t\tCALLOC_ARRAY(list, 1);\n> +\t\t\tlist->type = type;\n> +\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n> +\t\t}\n> +\t\tpush_to_stack(ctx, path.buf);\n> +\n> +\t\t/* Skip this object if already seen. */\n> +\t\tif (o->flags & SEEN)\n> +\t\t\tcontinue;\n> +\t\to->flags |= SEEN;\n\nThis made me wonder: why do we only skip the object this late? Couldn't\nwe already have done so immediately after we have looked up the object\nto avoid some work? If not, it might be useful to add a comment why it\nhas to come this late.\n\n> +\t\toid_array_append(&list->oids, &entry.oid);\n> +\t}\n> +\n> +\tfree_tree_buffer(tree);\n> +\tstrbuf_release(&path);\n> +\treturn 0;\n> +}\n> +\n> +/*\n> + * For each path in paths_to_explore, walk the trees another level\n> + * and add any found blobs to the batch (but only if they exist and\n> + * haven't been added yet).\n> + */\n> +static int walk_path(struct path_walk_context *ctx,\n> +\t\t const char *path)\n> +{\n> +\tstruct type_and_oid_list *list;\n> +\tint ret = 0;\n\nNit: needless initialization.\n\n> +\n> +\tlist = strmap_get(&ctx->paths_to_lists, path);\n> +\n> +\tif (!list->oids.nr)\n> +\t\treturn 0;\n> +\n> +\t/* Evaluate function pointer on this data. */\n> +\tret = ctx->info->path_fn(path, &list->oids, list->type,\n> +\t\t\t\t ctx->info->path_fn_data);\n> +\n> +\t/* Expand data for children. */\n> +\tif (list->type == OBJ_TREE) {\n> +\t\tfor (size_t i = 0; i < list->oids.nr; i++) {\n> +\t\t\tret |= add_children(ctx,\n> +\t\t\t\t\t path,\n> +\t\t\t\t\t &list->oids.oid[i]);\n> +\t\t}\n> +\t}\n> +\n> +\toid_array_clear(&list->oids);\n> +\tstrmap_remove(&ctx->paths_to_lists, path, 1);\n> +\treturn ret;\n> +}\n> +\n> +static void clear_strmap(struct strmap *map)\n\nNit: this isn't clearing a generic strmap, but rather `paths_to_lists`.\nShould we maybe rename it to `clear_paths_to_lists()`?\n\n> +{\n> +\tstruct hashmap_iter iter;\n> +\tstruct strmap_entry *e;\n> +\n> +\thashmap_for_each_entry(&map->map, &iter, e, ent) {\n> +\t\tstruct type_and_oid_list *list = e->value;\n> +\t\toid_array_clear(&list->oids);\n> +\t}\n> +\tstrmap_clear(map, 1);\n> +\tstrmap_init(map);\n> +}\n> +\n> +/**\n> + * Given the configuration of 'info', walk the commits based on 'info->revs' and\n> + * call 'info->path_fn' on each discovered path.\n> + *\n> + * Returns nonzero on an error.\n> + */\n> +int walk_objects_by_path(struct path_walk_info *info)\n> +{\n> +\tconst char *root_path = \"\";\n> +\tint ret = 0;\n> +\tsize_t commits_nr = 0, paths_nr = 0;\n> +\tstruct commit *c;\n> +\tstruct type_and_oid_list *root_tree_list;\n> +\tstruct path_walk_context ctx = {\n> +\t\t.repo = info->revs->repo,\n> +\t\t.revs = info->revs,\n> +\t\t.info = info,\n> +\t\t.path_stack = STRING_LIST_INIT_DUP,\n> +\t\t.path_stack_pushed = STRSET_INIT,\n> +\t\t.paths_to_lists = STRMAP_INIT\n> +\t};\n> +\n> +\ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n> +\n> +\t/* Insert a single list for the root tree into the paths. */\n> +\tCALLOC_ARRAY(root_tree_list, 1);\n> +\troot_tree_list->type = OBJ_TREE;\n> +\tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n> +\tpush_to_stack(&ctx, root_path);\n> +\n> +\tif (prepare_revision_walk(info->revs))\n> +\t\tdie(_(\"failed to setup revision walk\"));\n> +\n> +\twhile ((c = get_revision(info->revs))) {\n> +\t\tstruct object_id *oid = get_commit_tree_oid(c);\n> +\t\tstruct tree *t;\n> +\t\tcommits_nr++;\n> +\n> +\t\toid = get_commit_tree_oid(c);\n> +\t\tt = lookup_tree(info->revs->repo, oid);\n> +\n> +\t\tif (!t) {\n> +\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n> +\t\t\tcontinue;\n> +\t\t}\n\nIs this error something we should bubble up to the caller? As mentioned\nabove, I'm being cautious about silently accepting potentially-corrupt\ndata. Silent in the spirit of the caller not noticing it, not in the\nsense of the user not noticing it.\n\nPatrick\n"},{"id":"509095","messageId":"Z1whdAJ4RgQlLNl_@pks.im","threadId":"62429","inReplyTo":"a41f53f7ced7a403547ce1c239c5fbb5382cfb0f.1733514359.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/7] path-walk: visit tags and cached objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T11:58:44Z","receivedAt":"2024-12-13T11:59:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 06, 2024 at 07:45:56PM +0000, Derrick Stolee via GitGitGadget wrote:\n> @@ -194,6 +201,134 @@ static void clear_strmap(struct strmap *map)\n> \tstrmap_init(map);\n> }\n> \n> +static void setup_pending_objects(struct path_walk_info *info,\n> +\t\t\t\t struct path_walk_context *ctx)\n> +{\n> +\tstruct type_and_oid_list *tags = NULL;\n> +\tstruct type_and_oid_list *tagged_blobs = NULL;\n> +\tstruct type_and_oid_list *root_tree_list = NULL;\n> +\n> +\tif (info->tags)\n> +\t\tCALLOC_ARRAY(tags, 1);\n> +\tif (info->blobs)\n> +\t\tCALLOC_ARRAY(tagged_blobs, 1);\n> +\tif (info->trees)\n> +\t\troot_tree_list = strmap_get(&ctx->paths_to_lists, root_path);\n> +\n> +\t/*\n> +\t * Pending objects include:\n> +\t * * Commits at branch tips.\n> +\t * * Annotated tags at tag tips.\n> +\t * * Any kind of object at lightweight tag tips.\n> +\t * * Trees and blobs in the index (with an associated path).\n> +\t */\n> +\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n> +\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n> +\t\tstruct object *obj = pending->item;\n> +\n> +\t\t/* Commits will be picked up by revision walk. */\n> +\t\tif (obj->type == OBJ_COMMIT)\n> +\t\t\tcontinue;\n> +\n> +\t\t/* Navigate annotated tag object chains. */\n> +\t\twhile (obj->type == OBJ_TAG) {\n> +\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n> +\t\t\tif (!tag)\n> +\t\t\t\tbreak;\n\nSame here as previous comments, is this an error that we should rather\nreport?\n\n[snip]\n> +\tif (tagged_blobs) {\n> +\t\tif (tagged_blobs->oids.nr) {\n> +\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n> +\t\t\ttagged_blobs->type = OBJ_BLOB;\n> +\t\t\tpush_to_stack(ctx, tagged_blob_path);\n> +\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n> +\t\t} else {\n> +\t\t\toid_array_clear(&tagged_blobs->oids);\n> +\t\t\tfree(tagged_blobs);\n> +\t\t}\n> +\t}\n> +\tif (tags) {\n> +\t\tif (tags->oids.nr) {\n> +\t\t\tconst char *tag_path = \"/tags\";\n> +\t\t\ttags->type = OBJ_TAG;\n> +\t\t\tpush_to_stack(ctx, tag_path);\n> +\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n> +\t\t} else {\n> +\t\t\toid_array_clear(&tags->oids);\n> +\t\t\tfree(tags);\n> +\t\t}\n> +\t}\n> +}\n\nSo this is kind of curious. Does that mean that a file named\n\"tagged-blobs\" would be thrown into the same bag as a tagged blob? Or\nare these special due to the leading \"/\"?\n\nPatrick\n"},{"id":"509294","messageId":"eda7d3b7-8c3b-4486-b292-2507ca7f3bf9@gmail.com","threadId":"62429","inReplyTo":"Z1whcUJE-MHAhULO@pks.im","subject":"Re: [PATCH v3 1/7] path-walk: introduce an object walk by path","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-12-18T14:21:25Z","receivedAt":"2024-12-18T14:21:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/24 6:58 AM, Patrick Steinhardt wrote:\n> On Fri, Dec 06, 2024 at 07:45:52PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n>> +\t} else if (parse_tree_gently(tree, 1)) {\n>> +\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n> \n> I wonder whether we maybe shouldn't die but instead return an error in\n> the spirit of libification.\n\nThis is in fact something that is being tested when 'git pack-objects' has\nthe --path-walk feature. See \"get an error for missing tree object\" in\nt5317 as an example.\n\nIt's not enough to fail, but we need to fail with this error message.\n\nHas there been enough progress in the libification effort to establish a\npattern for returning an error message like \"bad tree object %s\" from an\nAPI like this to the caller?\n\nI will try using a \"error(); return -1;\" and consider that as the best\noption for right now.\n\n>> +\t\t} else {\n>> +\t\t\t/* Wrong type? */\n>> +\t\t\tcontinue;\n> \n> This code is unreachable, so we could make this a `BUG()`. Might also\n> use a switch instead, but that's more of a stylistic question.\n\nI think a BUG() would be good here.\n\n>> +\t\t}\n>> +\n>> +\t\tif (!o) /* report error?*/\n>> +\t\t\tcontinue;\n> \n> So this can only happen in case `lookup_tree()` or `lookup_blob()`\n> run into an error. I think this error should definitely be bubbled up so\n> that we don't silently skip tree entries in case of repo corruption.\n\nLooks like I agreed with you but didn't follow through.\n\n>> +\t\t/* Skip this object if already seen. */\n>> +\t\tif (o->flags & SEEN)\n>> +\t\t\tcontinue;\n>> +\t\to->flags |= SEEN;\n> \n> This made me wonder: why do we only skip the object this late? Couldn't\n> we already have done so immediately after we have looked up the object\n> to avoid some work? If not, it might be useful to add a comment why it\n> has to come this late.\n\nI went to look to see if there was a reason, and at this point there is\nnot a good reason. This should be moved up to avoid some checks and path\nmanipulation.\n\nI think that in a later patch, the use of the UNINTERESTING flag is\nimportant to pass the flag even when the object is already SEEN. This is\nprobably cruft from an earlier version that passed the UNINTERESTING bits\nin this part of the code.\n\n>> +\t\tif (!t) {\n>> +\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n>> +\t\t\tcontinue;\n>> +\t\t}\n> \n> Is this error something we should bubble up to the caller? As mentioned\n> above, I'm being cautious about silently accepting potentially-corrupt\n> data. Silent in the spirit of the caller not noticing it, not in the\n> sense of the user not noticing it.\nCan do.\n\nThanks for the careful review!\n-Stolee\n\n"},{"id":"509295","messageId":"b1ae2658-d576-433c-9180-8adb57479022@gmail.com","threadId":"62429","inReplyTo":"Z1whdAJ4RgQlLNl_@pks.im","subject":"Re: [PATCH v3 5/7] path-walk: visit tags and cached objects","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-12-18T14:23:43Z","receivedAt":"2024-12-18T14:23:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/24 6:58 AM, Patrick Steinhardt wrote:\n> On Fri, Dec 06, 2024 at 07:45:56PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> @@ -194,6 +201,134 @@ static void clear_strmap(struct strmap *map)\n\n>> +\t\t/* Navigate annotated tag object chains. */\n>> +\t\twhile (obj->type == OBJ_TAG) {\n>> +\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n>> +\t\t\tif (!tag)\n>> +\t\t\t\tbreak;\n> \n> Same here as previous comments, is this an error that we should rather\n> report?\n\nCan do.\n\n> [snip]\n>> +\tif (tagged_blobs) {\n>> +\t\tif (tagged_blobs->oids.nr) {\n>> +\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n>> +\t\t\ttagged_blobs->type = OBJ_BLOB;\n>> +\t\t\tpush_to_stack(ctx, tagged_blob_path);\n>> +\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n>> +\t\t} else {\n>> +\t\t\toid_array_clear(&tagged_blobs->oids);\n>> +\t\t\tfree(tagged_blobs);\n>> +\t\t}\n>> +\t}\n>> +\tif (tags) {\n>> +\t\tif (tags->oids.nr) {\n>> +\t\t\tconst char *tag_path = \"/tags\";\n>> +\t\t\ttags->type = OBJ_TAG;\n>> +\t\t\tpush_to_stack(ctx, tag_path);\n>> +\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n>> +\t\t} else {\n>> +\t\t\toid_array_clear(&tags->oids);\n>> +\t\t\tfree(tags);\n>> +\t\t}\n>> +\t}\n>> +}\n> \n> So this is kind of curious. Does that mean that a file named\n> \"tagged-blobs\" would be thrown into the same bag as a tagged blob? Or\n> are these special due to the leading \"/\"?\n\nIndeed, the leading \"/\" differentiates these categories from other paths\nthat are stored in the process.\n\nThanks,\n-Stolee\n\n\n"},{"id":"509409","messageId":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v3.git.1733514358.gitgitgadget@gmail.com","subject":"[PATCH v4 0/7] PATH WALK I: The path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:08Z","receivedAt":"2024-12-20T16:21:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nIntroduction and relation to prior series\n=========================================\n\nThis is a new series that rerolls the initial \"path-walk API\" patches of my\nRFC [1] \"Path-walk API and applications\". This new API (in path-walk.c and\npath-walk.h) presents a new way to walk objects such that trees and blobs\nare walked in batches according to their path.\n\nThis also replaces the previous version of ds/path-walk that was being\nreviewed in [2]. The consensus was that the series was too long/dense and\ncould use some reduction in size. This series takes the first few patches,\nbut also makes some updates (which will be described later).\n\n[1]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n[RFC] Path-walk API and applications\n\n[2]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n[PATCH v2 00/17] pack-objects: add --path-walk option for better deltas\n\nThis series only introduces the path-walk API, but does so to the full\ncomplexity required to later add the integration with git pack-objects to\nimprove packing compression in both time and space for repositories with\nmany name hash collisions. The compression also at least improves for other\nrepositories, but may not always have an improvement in time.\n\nSome of the changes that are present in this series that differ from the\nprevious version are motivated directly by discoveries made by testing the\nfeature in Git for Windows and microsoft/git forks that shipped these\nfeatures for fast delivery of these improvements to users who needed them.\nThat testing across many environments informed some things that needed to be\nchanged, and in this series those changes are checked by tests in the\nt6601-path-walk.sh test script and the test-tool path-walk test helper.\nThus, the code being introduced in this series is covered by tests even\nthough it is not integrated into the git executable.\n\n\nDiscussion of follow-up applications\n====================================\n\nBy splitting this series out into its own, I was able to reorganize the\npatches such that each application can be build independently off of this\nseries. These are available as pending PRs in gitgitgadget/git:\n\n * Better delta compression with 'git pack-objects' [3]: This application\n allows an option in 'git pack-objects' to change how objects are walked\n in order to group objects with the same path for early delta compression\n before using the name hash sort to look for cross-path deltas. This helps\n significantly in repositories with many name-hash collisions. This\n reduces the size of 'git push' pacifies via a config option and reduces\n the total repo size in 'git repack'.\n\n * The 'git backfill' command [4]: This command downloads missing blobs in a\n bloodless partial clone. In order to save space and network bandwidth, it\n assumes that objects at a common path are likely to delta well with each\n other, so it downloads missing blobs in batches via the path-walk API.\n This presents a way to use blobless clones as a pseudo-resumable clone,\n since the initial clone of commits and trees is a smaller initial\n download and the batch size allows downloading blobs incrementally. When\n pairing this command with the sparse-checkout feature, the path-walk API\n is adjusted to focus on the paths within the sparse-checkout. This allows\n the user to only download the files they are likely to need when\n inspecting history within their scope without downloading the entire\n repository history.\n\n * The 'git survey' command [5]. This application begins the work to mimic\n the behavior of git-sizer, but to use internal data structures for better\n performance and careful understanding of how objects are stored. Using\n the path-walk API, paths with many versions can be considered in a batch\n and sorted into a list to report the paths that contribute most to the\n size of the repository. A version of this command was used to help\n confirm the issues with the name hash collisions. It was also used to\n diagnose why some repacks using the --path-walk option were taking more\n space than without for some repositories. (More on this later.)\n\nQuestion for reviewers: I am prepped to send these three applications to the\nmailing list, but I'll refrain for now to avoid causing too much noise for\nfolks. Would you like to see them on-list while this series is under review?\nOr would you prefer to explore the PRs ([3] [4] and [5])?\n\n[3] https://github.com/gitgitgadget/git/pull/1819\nPATH WALK II: Add --path-walk option to 'git pack-objects'\n\n[4] https://github.com/gitgitgadget/git/pull/1820\nPATH WALK III: Add 'git backfill' command\n\n[5] https://github.com/gitgitgadget/git/pull/1821\nPATH WALK IV: Add 'git survey' command\n\n\nStructure of the Patch Series\n=============================\n\nThis patch series attempts to create the simplest version of the API in\npatch 1, then build functionality incrementally. During the process, each\nchange will introduce an update to:\n\n * The path-walk API itself in path-walk.c and path-walk.h.\n * The documentation of the API in\n Documentation/technical/api-path-walk.txt.\n * The test script t/t6601-path-walk.sh.\n\nThe core of the API relies on using a 'struct rev_info' to define an initial\nset of objects and some form of a commit walk to define what range of\nobjects to visit. Initially, only a subset of 'struct rev_info' options work\nas expected. For example:\n\n * Patch 1 assumes that only commit objects are starting positions, but the\n focus is on exploring trees and blobs.\n * Patch 3 allows users to specify object types, which includes outputting\n the visited commits in a batch.\n * Annotated tags and indexed objects are considered in Patch 4. These are\n grouped because they both exist within the 'pending' object list.\n * UNINTERESTING objects are not considered until Patch 5.\n\n\nChanges in v1 (since previous version)\n======================================\n\nThere are a few hard-won learnings from previous versions of this series due\nto testing this in the wild with many different repositories.\n\n * Initially, the 'git pack-objects --path-walk' feature was not tested with\n the '--shallow' option because it was expected that this option was for\n servers creating a pack containing shallow commits. However, this option\n is also used when pushing from a shallow clone, and this was a critical\n feature that we needed to reduce the size of commits pushed from\n automated environments that were bootstrapped by shallow clones. The crux\n of the change is in Patch 5 and how UNINTERESTING objects are handled. We\n no longer need to push the UNINTERESTING flag around the objects\n ourselves and can use existing logic in list-objects.c to do so. This\n allows using the --objects-edge-aggressive option when necessary to\n reduce the object count when pushing from a shallow clone. (The\n pack-objects series expands on tests to cover this integration point.)\n\n * When looking into cases where 'git repack -adf' outperformed 'git repack\n -adf --path-walk', I discovered that the issue did not reproduce in a\n bare repository. This is due to 'git repack' iterating over all indexed\n objects before walking commits. I had inadvertently put all indexed\n objects in their own category, leading to no good deltas with previous\n versions of those files; I had also not used the 'path' option from the\n pending list, so these objects had invalid name hash values. You will see\n in patch 4 that the pending list is handled quite differently and the\n '--indexed-objects' option is tested directly within t6601.\n\n * I added a new 'test_cmp_sorted' helper because I wanted to simplify some\n repeated sections of t6601.\n\n * Patch 1 has significantly more context than it did before.\n\n * Annotated tags are given a name of \"/tags\" to differentiate them slightly\n from root trees and commits.\n\n\nChanges in v2\n=============\n\n * Updated the test helper to output the batch number, allowing us to\n confirm that OIDs are grouped appropriately. This also signaled a few\n cases where the callback function was being called on an empty set.\n\n * This change has resulted in significant changes to the test data,\n including reordered lines and prepended batch numbers.\n\n * Thanks to Patrick for providing a recommended change to remove memory\n leaks from the test helper.\n\n\nChanges in v3\n=============\n\n * Updated test helper to use type_string(), which leads to a change to use\n lowercase strings in the test scripts. That will lead to the range-diff\n looking pretty terrible.\n\n * Added a new patch that changes the visit order of the path-walk API. The\n intention is to reduce memory pressure by emitting blob paths before\n recursing into tree paths. This also has the effect of visiting blobs and\n trees in lexicographic order instead of the reverse.\n\n\nChanges in v4\n=============\n\n * Several style fixes and function renames.\n\n * Better error handling, avoiding some die() statements.\n\n * Additional BUG() statements for \"impossible\" scenarios.\n\n * Optimizations around SEEN objects to avoid extra work. This does have\n some impact on paths that appear in the index but no other versions are\n discovered during the tree walk. This changes a test in t6601 and the\n timing of visiting the blob path \"a\" being delayed to the end.\n\n * The path_walk_info struct now has proper initializers and destructors,\n even though the current destructor is empty.\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n path-walk: introduce an object walk by path\n test-lib-functions: add test_cmp_sorted\n t6601: add helper for testing path-walk API\n path-walk: allow consumer to specify object types\n path-walk: visit tags and cached objects\n path-walk: mark trees and blobs as UNINTERESTING\n path-walk: reorder object visits\n\n Documentation/technical/api-path-walk.txt | 63 +++\n Makefile | 2 +\n path-walk.c | 592 ++++++++++++++++++++++\n path-walk.h | 69 +++\n t/helper/test-path-walk.c | 112 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 368 ++++++++++++++\n t/test-lib-functions.sh | 10 +\n 9 files changed, 1218 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\n\nbase-commit: e9356ba3ea2a6754281ff7697b3e5a1697b21e24\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1818%2Fderrickstolee%2Fapi-upstream-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1818/derrickstolee/api-upstream-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1818\n\nRange-diff vs v3:\n\n 1: b7e9b81e8b3 ! 1: 62f7aae477b path-walk: introduce an object walk by path\n @@ path-walk.c (new)\n +#include \"tree.h\"\n +#include \"tree-walk.h\"\n +\n -+struct type_and_oid_list\n -+{\n ++struct type_and_oid_list {\n +\tenum object_type type;\n +\tstruct oid_array oids;\n +};\n @@ path-walk.c (new)\n +\tstring_list_append(&ctx->path_stack, path);\n +}\n +\n -+static int add_children(struct path_walk_context *ctx,\n -+\t\t\tconst char *base_path,\n -+\t\t\tstruct object_id *oid)\n ++static int add_tree_entries(struct path_walk_context *ctx,\n ++\t\t\t const char *base_path,\n ++\t\t\t struct object_id *oid)\n +{\n +\tstruct tree_desc desc;\n +\tstruct name_entry entry;\n @@ path-walk.c (new)\n +\t\t oid_to_hex(oid));\n +\t\treturn -1;\n +\t} else if (parse_tree_gently(tree, 1)) {\n -+\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n ++\t\terror(\"bad tree object %s\", oid_to_hex(oid));\n ++\t\treturn -1;\n +\t}\n +\n +\tstrbuf_addstr(&path, base_path);\n @@ path-walk.c (new)\n +\t\t\tstruct blob *child = lookup_blob(ctx->repo, &entry.oid);\n +\t\t\to = child ? &child->object : NULL;\n +\t\t} else {\n -+\t\t\t/* Wrong type? */\n -+\t\t\tcontinue;\n ++\t\t\tBUG(\"invalid type for tree entry: %d\", type);\n +\t\t}\n +\n -+\t\tif (!o) /* report error?*/\n ++\t\tif (!o) {\n ++\t\t\terror(_(\"failed to find object %s\"),\n ++\t\t\t oid_to_hex(&o->oid));\n ++\t\t\treturn -1;\n ++\t\t}\n ++\n ++\t\t/* Skip this object if already seen. */\n ++\t\tif (o->flags & SEEN)\n +\t\t\tcontinue;\n ++\t\to->flags |= SEEN;\n +\n +\t\tstrbuf_setlen(&path, base_len);\n +\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n @@ path-walk.c (new)\n +\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n +\t\t}\n +\t\tpush_to_stack(ctx, path.buf);\n -+\n -+\t\t/* Skip this object if already seen. */\n -+\t\tif (o->flags & SEEN)\n -+\t\t\tcontinue;\n -+\t\to->flags |= SEEN;\n +\t\toid_array_append(&list->oids, &entry.oid);\n +\t}\n +\n @@ path-walk.c (new)\n +\t/* Expand data for children. */\n +\tif (list->type == OBJ_TREE) {\n +\t\tfor (size_t i = 0; i < list->oids.nr; i++) {\n -+\t\t\tret |= add_children(ctx,\n ++\t\t\tret |= add_tree_entries(ctx,\n +\t\t\t\t\t path,\n +\t\t\t\t\t &list->oids.oid[i]);\n +\t\t}\n @@ path-walk.c (new)\n +\treturn ret;\n +}\n +\n -+static void clear_strmap(struct strmap *map)\n ++static void clear_paths_to_lists(struct strmap *map)\n +{\n +\tstruct hashmap_iter iter;\n +\tstruct strmap_entry *e;\n @@ path-walk.c (new)\n +\t\tt = lookup_tree(info->revs->repo, oid);\n +\n +\t\tif (!t) {\n -+\t\t\twarning(\"could not find tree %s\", oid_to_hex(oid));\n -+\t\t\tcontinue;\n ++\t\t\terror(\"could not find tree %s\", oid_to_hex(oid));\n ++\t\t\treturn -1;\n +\t\t}\n +\n +\t\tif (t->object.flags & SEEN)\n @@ path-walk.c (new)\n +\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n +\ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n +\n -+\tclear_strmap(&ctx.paths_to_lists);\n ++\tclear_paths_to_lists(&ctx.paths_to_lists);\n +\tstrset_clear(&ctx.path_stack_pushed);\n +\tstring_list_clear(&ctx.path_stack, 0);\n +\treturn ret;\n ++}\n ++\n ++void path_walk_info_init(struct path_walk_info *info)\n ++{\n ++\tstruct path_walk_info empty = PATH_WALK_INFO_INIT;\n ++\tmemcpy(info, &empty, sizeof(empty));\n ++}\n ++\n ++void path_walk_info_clear(struct path_walk_info *info UNUSED)\n ++{\n ++\t/*\n ++\t * This destructor is empty for now, as info->revs\n ++\t * is not owned by 'struct path_walk_info'.\n ++\t */\n +}\n \n ## path-walk.h (new) ##\n @@ path-walk.h (new)\n +struct path_walk_info {\n +\t/**\n +\t * revs provides the definitions for the commit walk, including\n -+\t * which commits are UNINTERESTING or not.\n ++\t * which commits are UNINTERESTING or not. This structure is\n ++\t * expected to be owned by the caller.\n +\t */\n +\tstruct rev_info *revs;\n +\n @@ path-walk.h (new)\n +\n +#define PATH_WALK_INFO_INIT { 0 }\n +\n ++void path_walk_info_init(struct path_walk_info *info);\n ++void path_walk_info_clear(struct path_walk_info *info);\n ++\n +/**\n + * Given the configuration of 'info', walk the commits based on 'info->revs' and\n + * call 'info->path_fn' on each discovered path.\n 2: cf2ed61b324 = 2: 8ad2a5e79a2 test-lib-functions: add test_cmp_sorted\n 3: 54886fcb081 = 3: 2bc0538bce9 t6601: add helper for testing path-walk API\n 4: 42e71e6285f ! 4: db5c8611230 path-walk: allow consumer to specify object types\n @@ Documentation/technical/api-path-walk.txt: It is also important that you do not\n \n \n ## path-walk.c ##\n -@@ path-walk.c: static int add_children(struct path_walk_context *ctx,\n +@@ path-walk.c: static int add_tree_entries(struct path_walk_context *ctx,\n \t\tif (S_ISGITLINK(entry.mode))\n \t\t\tcontinue;\n \n @@ path-walk.h: struct path_walk_info {\n +\t.commits = 1,\t\t\\\n +}\n \n - /**\n - * Given the configuration of 'info', walk the commits based on 'info->revs' and\n + void path_walk_info_init(struct path_walk_info *info);\n + void path_walk_info_clear(struct path_walk_info *info);\n \n ## t/helper/test-path-walk.c ##\n @@ t/helper/test-path-walk.c: static const char * const path_walk_usage[] = {\n 5: a41f53f7ced ! 5: 6df56f465d7 path-walk: visit tags and cached objects\n @@ path-walk.c\n \n +static const char *root_path = \"\";\n +\n - struct type_and_oid_list\n - {\n + struct type_and_oid_list {\n \tenum object_type type;\n + \tstruct oid_array oids;\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n \n \tlist = strmap_get(&ctx->paths_to_lists, path);\n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n \t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n \t\t\t\t\tctx->info->path_fn_data);\n \n -@@ path-walk.c: static void clear_strmap(struct strmap *map)\n +@@ path-walk.c: static void clear_paths_to_lists(struct strmap *map)\n \tstrmap_init(map);\n }\n \n -+static void setup_pending_objects(struct path_walk_info *info,\n -+\t\t\t\t struct path_walk_context *ctx)\n ++static int setup_pending_objects(struct path_walk_info *info,\n ++\t\t\t\t struct path_walk_context *ctx)\n +{\n +\tstruct type_and_oid_list *tags = NULL;\n +\tstruct type_and_oid_list *tagged_blobs = NULL;\n @@ path-walk.c: static void clear_strmap(struct strmap *map)\n +\t\t/* Navigate annotated tag object chains. */\n +\t\twhile (obj->type == OBJ_TAG) {\n +\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n -+\t\t\tif (!tag)\n -+\t\t\t\tbreak;\n ++\t\t\tif (!tag) {\n ++\t\t\t\terror(_(\"failed to find tag %s\"),\n ++\t\t\t\t oid_to_hex(&obj->oid));\n ++\t\t\t\treturn -1;\n ++\t\t\t}\n +\t\t\tif (tag->object.flags & SEEN)\n +\t\t\t\tbreak;\n +\t\t\ttag->object.flags |= SEEN;\n @@ path-walk.c: static void clear_strmap(struct strmap *map)\n +\t\t\tfree(tags);\n +\t\t}\n +\t}\n ++\n ++\treturn 0;\n +}\n +\n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\n * call 'info->path_fn' on each discovered path.\n -@@ path-walk.c: static void clear_strmap(struct strmap *map)\n +@@ path-walk.c: static void clear_paths_to_lists(struct strmap *map)\n */\n int walk_objects_by_path(struct path_walk_info *info)\n {\n -\tconst char *root_path = \"\";\n - \tint ret = 0;\n +-\tint ret = 0;\n ++\tint ret;\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n + \tstruct type_and_oid_list *root_tree_list;\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(commit_list, 1);\n \tcommit_list->type = OBJ_COMMIT;\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n +\tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n +\n +\ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n -+\tsetup_pending_objects(info, &ctx);\n ++\tret = setup_pending_objects(info, &ctx);\n +\ttrace2_region_leave(\"path-walk\", \"pending-walk\", info->revs->repo);\n ++\n ++\tif (ret)\n ++\t\treturn ret;\n +\n \twhile ((c = get_revision(info->revs))) {\n \t\tstruct object_id *oid;\n @@ path-walk.h: struct path_walk_info {\n +\t.tags = 1,\t\t\\\n }\n \n - /**\n + void path_walk_info_init(struct path_walk_info *info);\n \n ## t/helper/test-path-walk.c ##\n @@ t/helper/test-path-walk.c: struct path_walk_test_data {\n 6: 0f1e6c51b2c ! 6: f2ffc32a303 path-walk: mark trees and blobs as UNINTERESTING\n @@ path-walk.c\n #include \"object.h\"\n #include \"oid-array.h\"\n #include \"revision.h\"\n -@@ path-walk.c: struct type_and_oid_list\n - {\n +@@ path-walk.c: static const char *root_path = \"\";\n + struct type_and_oid_list {\n \tenum object_type type;\n \tstruct oid_array oids;\n +\tint maybe_interesting;\n };\n \n #define TYPE_AND_OID_LIST_INIT { \\\n -@@ path-walk.c: static int add_children(struct path_walk_context *ctx,\n - \t\tif (o->flags & SEEN)\n - \t\t\tcontinue;\n - \t\to->flags |= SEEN;\n +@@ path-walk.c: static int add_tree_entries(struct path_walk_context *ctx,\n + \t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n + \t\t}\n + \t\tpush_to_stack(ctx, path.buf);\n +\n +\t\tif (!(o->flags & UNINTERESTING))\n +\t\t\tlist->maybe_interesting = 1;\n ++\n \t\toid_array_append(&list->oids, &entry.oid);\n \t}\n \n @@ path-walk.c: static int walk_path(struct path_walk_context *ctx,\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n \t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n -@@ path-walk.c: static void clear_strmap(struct strmap *map)\n +@@ path-walk.c: static void clear_paths_to_lists(struct strmap *map)\n \tstrmap_init(map);\n }\n \n @@ path-walk.c: static void clear_strmap(struct strmap *map)\n +\toid_array_append(&edge_tree_list->oids, &t->object.oid);\n +}\n +\n - static void setup_pending_objects(struct path_walk_info *info,\n - \t\t\t\t struct path_walk_context *ctx)\n + static int setup_pending_objects(struct path_walk_info *info,\n + \t\t\t\t struct path_walk_context *ctx)\n {\n -@@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n +@@ path-walk.c: static int setup_pending_objects(struct path_walk_info *info,\n \t\tif (tagged_blobs->oids.nr) {\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n @@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n \t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n \t\t} else {\n -@@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n +@@ path-walk.c: static int setup_pending_objects(struct path_walk_info *info,\n \t\tif (tags->oids.nr) {\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n 7: e716672c041 ! 7: ef543429ed9 path-walk: reorder object visits\n @@ path-walk.c: static void push_to_stack(struct path_walk_context *ctx,\n +\tprio_queue_put(&ctx->path_stack, xstrdup(path));\n }\n \n - static int add_children(struct path_walk_context *ctx,\n -@@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n + static int add_tree_entries(struct path_walk_context *ctx,\n +@@ path-walk.c: static int setup_pending_objects(struct path_walk_info *info,\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n \t\t\ttagged_blobs->maybe_interesting = 1;\n @@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n \t\t} else {\n \t\t\toid_array_clear(&tagged_blobs->oids);\n \t\t\tfree(tagged_blobs);\n -@@ path-walk.c: static void setup_pending_objects(struct path_walk_info *info,\n +@@ path-walk.c: static int setup_pending_objects(struct path_walk_info *info,\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n \t\t\ttags->maybe_interesting = 1;\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \t\t\tret = walk_path(&ctx, path);\n @@ path-walk.c: int walk_objects_by_path(struct path_walk_info *info)\n \n - \tclear_strmap(&ctx.paths_to_lists);\n + \tclear_paths_to_lists(&ctx.paths_to_lists);\n \tstrset_clear(&ctx.path_stack_pushed);\n -\tstring_list_clear(&ctx.path_stack, 0);\n +\tclear_prio_queue(&ctx.path_stack);\n \treturn ret;\n }\n + \n \n ## t/t6601-path-walk.sh ##\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n @@ t/t6601-path-walk.sh: test_expect_success 'all' '\n \tcommits:4\n \ttags:7\n @@ t/t6601-path-walk.sh: test_expect_success 'branches and indexed objects mix well' '\n + \t1:tree::$(git rev-parse base^{tree})\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n - \t2:blob:a:$(git rev-parse base~2:a)\n +-\t2:blob:a:$(git rev-parse base~2:a)\n -\t3:tree:right/:$(git rev-parse topic:right)\n -\t3:tree:right/:$(git rev-parse base~1:right)\n -\t3:tree:right/:$(git rev-parse base~2:right)\n @@ t/t6601-path-walk.sh: test_expect_success 'branches and indexed objects mix well\n -\t7:blob:left/b:$(git rev-parse base:left/b)\n -\t7:blob:left/b:$(git rev-parse base~2:left/b)\n -\t8:tree:a/:$(git rev-parse refs/tags/third:a)\n -+\t3:tree:a/:$(git rev-parse refs/tags/third:a)\n -+\t4:tree:left/:$(git rev-parse base:left)\n -+\t4:tree:left/:$(git rev-parse base~2:left)\n -+\t5:blob:left/b:$(git rev-parse base:left/b)\n -+\t5:blob:left/b:$(git rev-parse base~2:left/b)\n -+\t6:tree:right/:$(git rev-parse topic:right)\n -+\t6:tree:right/:$(git rev-parse base~1:right)\n -+\t6:tree:right/:$(git rev-parse base~2:right)\n -+\t7:blob:right/c:$(git rev-parse base~2:right/c)\n -+\t7:blob:right/c:$(git rev-parse topic:right/c)\n -+\t8:blob:right/d:$(git rev-parse base~1:right/d)\n -+\t8:blob:right/d:$(git rev-parse :right/d)\n ++\t2:tree:a/:$(git rev-parse refs/tags/third:a)\n ++\t3:tree:left/:$(git rev-parse base:left)\n ++\t3:tree:left/:$(git rev-parse base~2:left)\n ++\t4:blob:left/b:$(git rev-parse base:left/b)\n ++\t4:blob:left/b:$(git rev-parse base~2:left/b)\n ++\t5:tree:right/:$(git rev-parse topic:right)\n ++\t5:tree:right/:$(git rev-parse base~1:right)\n ++\t5:tree:right/:$(git rev-parse base~2:right)\n ++\t6:blob:right/c:$(git rev-parse base~2:right/c)\n ++\t6:blob:right/c:$(git rev-parse topic:right/c)\n ++\t7:blob:right/d:$(git rev-parse base~1:right/d)\n ++\t7:blob:right/d:$(git rev-parse :right/d)\n ++\t8:blob:a:$(git rev-parse base~2:a)\n \tblobs:7\n \tcommits:4\n \ttags:0\n\n-- \ngitgitgadget\n"},{"id":"509410","messageId":"62f7aae477bc542dce0bb6f8b59438442a77d154.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 1/7] path-walk: introduce an object walk by path","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:09Z","receivedAt":"2024-12-20T16:21:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a few planned applications, introduce the most basic form\nof a path-walk API. It currently assumes that there are no UNINTERESTING\nobjects, and does not include any complicated filters. It calls a function\npointer on groups of tree and blob objects as grouped by path. This only\nincludes objects the first time they are discovered, so an object that\nappears at multiple paths will not be included in two batches.\n\nThese batches are collected in 'struct type_and_oid_list' objects, which\nstore an object type and an oid_array of objects.\n\nThe data structures are documented in 'struct path_walk_context', but in\nsummary the most important are:\n\n * 'paths_to_lists' is a strmap that connects a path to a\n type_and_oid_list for that path. To avoid conflicts in path names,\n we make sure that tree paths end in \"/\" (except the root path with\n is an empty string) and blob paths do not end in \"/\".\n\n * 'path_stack' is a string list that is added to in an append-only\n way. This stores the stack of our depth-first search on the heap\n instead of using recursion.\n\n * 'path_stack_pushed' is a strmap that stores path names that were\n already added to 'path_stack', to avoid repeating paths in the\n stack. Mostly, this saves us from quadratic lookups from doing\n unsorted checks into the string_list.\n\nThe coupling of 'path_stack' and 'path_stack_pushed' is protected by the\npush_to_stack() method. Call this instead of inserting into these\nstructures directly.\n\nThe walk_objects_by_path() method initializes these structures and\nstarts walking commits from the given rev_info struct. The commits are\nused to find the list of root trees which populate the start of our\ndepth-first search.\n\nThe core of our depth-first search is in a while loop that continues\nwhile we have not indicated an early exit and our 'path_stack' still has\nentries in it. The loop body pops a path off of the stack and \"visits\"\nthe path via the walk_path() method.\n\nThe walk_path() method gets the list of OIDs from the 'path_to_lists'\nstrmap and executes the callback method on that list with the given path\nand type. If the OIDs correspond to tree objects, then iterate over all\ntrees in the list and run add_children() to add the child objects to\ntheir own lists, adding new entries to the stack if necessary.\n\nIn testing, this depth-first search approach was the one that used the\nleast memory while iterating over the object lists. There is still a\nchance that repositories with too-wide path patterns could cause memory\npressure issues. Limiting the stack size could be done in the future by\nlimiting how many objects are being considered in-progress, or by\nvisiting blob paths earlier than trees.\n\nThere are many future adaptations that could be made, but they are left for\nfuture updates when consumers are ready to take advantage of those features.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 45 ++++\n Makefile | 1 +\n path-walk.c | 279 ++++++++++++++++++++++\n path-walk.h | 47 ++++\n 4 files changed, 372 insertions(+)\n create mode 100644 Documentation/technical/api-path-walk.txt\n create mode 100644 path-walk.c\n create mode 100644 path-walk.h\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nnew file mode 100644\nindex 00000000000..c550c77ca30\n--- /dev/null\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -0,0 +1,45 @@\n+Path-Walk API\n+=============\n+\n+The path-walk API is used to walk reachable objects, but to visit objects\n+in batches based on a common path they appear in, or by type.\n+\n+For example, all reachable commits are visited in a group. All tags are\n+visited in a group. Then, all root trees are visited. At some point, all\n+blobs reachable via a path `my/dir/to/A` are visited. When there are\n+multiple paths possible to reach the same object, then only one of those\n+paths is used to visit the object.\n+\n+Basics\n+------\n+\n+To use the path-walk API, include `path-walk.h` and call\n+`walk_objects_by_path()` with a customized `path_walk_info` struct. The\n+struct is used to set all of the options for how the walk should proceed.\n+Let's dig into the different options and their use.\n+\n+`path_fn` and `path_fn_data`::\n+\tThe most important option is the `path_fn` option, which is a\n+\tfunction pointer to the callback that can execute logic on the\n+\tobject IDs for objects grouped by type and path. This function\n+\talso receives a `data` value that corresponds to the\n+\t`path_fn_data` member, for providing custom data structures to\n+\tthis callback function.\n+\n+`revs`::\n+\tTo configure the exact details of the reachable set of objects,\n+\tuse the `revs` member and initialize it using the revision\n+\tmachinery in `revision.h`. Initialize `revs` using calls such as\n+\t`setup_revisions()` or `parse_revision_opt()`. Do not call\n+\t`prepare_revision_walk()`, as that will be called within\n+\t`walk_objects_by_path()`.\n++\n+It is also important that you do not specify the `--objects` flag for the\n+`revs` struct. The revision walk should only be used to walk commits, and\n+the objects will be walked in a separate way based on those starting\n+commits.\n+\n+Examples\n+--------\n+\n+See example usages in future changes.\ndiff --git a/Makefile b/Makefile\nindex 7344a7f7257..d0d8d6888e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\n+LIB_OBJS += path-walk.o\n LIB_OBJS += pathspec.o\n LIB_OBJS += pkt-line.o\n LIB_OBJS += preload-index.o\ndiff --git a/path-walk.c b/path-walk.c\nnew file mode 100644\nindex 00000000000..021840ab41b\n--- /dev/null\n+++ b/path-walk.c\n@@ -0,0 +1,279 @@\n+/*\n+ * path-walk.c: implementation for path-based walks of the object graph.\n+ */\n+#include \"git-compat-util.h\"\n+#include \"path-walk.h\"\n+#include \"blob.h\"\n+#include \"commit.h\"\n+#include \"dir.h\"\n+#include \"hashmap.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"oid-array.h\"\n+#include \"revision.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+#include \"trace2.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+\n+struct type_and_oid_list {\n+\tenum object_type type;\n+\tstruct oid_array oids;\n+};\n+\n+#define TYPE_AND_OID_LIST_INIT { \\\n+\t.type = OBJ_NONE, \t \\\n+\t.oids = OID_ARRAY_INIT\t \\\n+}\n+\n+struct path_walk_context {\n+\t/**\n+\t * Repeats of data in 'struct path_walk_info' for\n+\t * access with fewer characters.\n+\t */\n+\tstruct repository *repo;\n+\tstruct rev_info *revs;\n+\tstruct path_walk_info *info;\n+\n+\t/**\n+\t * Map a path to a 'struct type_and_oid_list'\n+\t * containing the objects discovered at that\n+\t * path.\n+\t */\n+\tstruct strmap paths_to_lists;\n+\n+\t/**\n+\t * Store the current list of paths in a stack, to\n+\t * facilitate depth-first-search without recursion.\n+\t *\n+\t * Use path_stack_pushed to indicate whether a path\n+\t * was previously added to path_stack.\n+\t */\n+\tstruct string_list path_stack;\n+\tstruct strset path_stack_pushed;\n+};\n+\n+static void push_to_stack(struct path_walk_context *ctx,\n+\t\t\t const char *path)\n+{\n+\tif (strset_contains(&ctx->path_stack_pushed, path))\n+\t\treturn;\n+\n+\tstrset_add(&ctx->path_stack_pushed, path);\n+\tstring_list_append(&ctx->path_stack, path);\n+}\n+\n+static int add_tree_entries(struct path_walk_context *ctx,\n+\t\t\t const char *base_path,\n+\t\t\t struct object_id *oid)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t base_len;\n+\tstruct tree *tree = lookup_tree(ctx->repo, oid);\n+\n+\tif (!tree) {\n+\t\terror(_(\"failed to walk children of tree %s: not found\"),\n+\t\t oid_to_hex(oid));\n+\t\treturn -1;\n+\t} else if (parse_tree_gently(tree, 1)) {\n+\t\terror(\"bad tree object %s\", oid_to_hex(oid));\n+\t\treturn -1;\n+\t}\n+\n+\tstrbuf_addstr(&path, base_path);\n+\tbase_len = path.len;\n+\n+\tparse_tree(tree);\n+\tinit_tree_desc(&desc, &tree->object.oid, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tstruct type_and_oid_list *list;\n+\t\tstruct object *o;\n+\t\t/* Not actually true, but we will ignore submodules later. */\n+\t\tenum object_type type = S_ISDIR(entry.mode) ? OBJ_TREE : OBJ_BLOB;\n+\n+\t\t/* Skip submodules. */\n+\t\tif (S_ISGITLINK(entry.mode))\n+\t\t\tcontinue;\n+\n+\t\tif (type == OBJ_TREE) {\n+\t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else if (type == OBJ_BLOB) {\n+\t\t\tstruct blob *child = lookup_blob(ctx->repo, &entry.oid);\n+\t\t\to = child ? &child->object : NULL;\n+\t\t} else {\n+\t\t\tBUG(\"invalid type for tree entry: %d\", type);\n+\t\t}\n+\n+\t\tif (!o) {\n+\t\t\terror(_(\"failed to find object %s\"),\n+\t\t\t oid_to_hex(&o->oid));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/* Skip this object if already seen. */\n+\t\tif (o->flags & SEEN)\n+\t\t\tcontinue;\n+\t\to->flags |= SEEN;\n+\n+\t\tstrbuf_setlen(&path, base_len);\n+\t\tstrbuf_add(&path, entry.path, entry.pathlen);\n+\n+\t\t/*\n+\t\t * Trees will end with \"/\" for concatenation and distinction\n+\t\t * from blobs at the same path.\n+\t\t */\n+\t\tif (type == OBJ_TREE)\n+\t\t\tstrbuf_addch(&path, '/');\n+\n+\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {\n+\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\tlist->type = type;\n+\t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n+\t\t}\n+\t\tpush_to_stack(ctx, path.buf);\n+\t\toid_array_append(&list->oids, &entry.oid);\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+\tstrbuf_release(&path);\n+\treturn 0;\n+}\n+\n+/*\n+ * For each path in paths_to_explore, walk the trees another level\n+ * and add any found blobs to the batch (but only if they exist and\n+ * haven't been added yet).\n+ */\n+static int walk_path(struct path_walk_context *ctx,\n+\t\t const char *path)\n+{\n+\tstruct type_and_oid_list *list;\n+\tint ret = 0;\n+\n+\tlist = strmap_get(&ctx->paths_to_lists, path);\n+\n+\tif (!list->oids.nr)\n+\t\treturn 0;\n+\n+\t/* Evaluate function pointer on this data. */\n+\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t ctx->info->path_fn_data);\n+\n+\t/* Expand data for children. */\n+\tif (list->type == OBJ_TREE) {\n+\t\tfor (size_t i = 0; i < list->oids.nr; i++) {\n+\t\t\tret |= add_tree_entries(ctx,\n+\t\t\t\t\t path,\n+\t\t\t\t\t &list->oids.oid[i]);\n+\t\t}\n+\t}\n+\n+\toid_array_clear(&list->oids);\n+\tstrmap_remove(&ctx->paths_to_lists, path, 1);\n+\treturn ret;\n+}\n+\n+static void clear_paths_to_lists(struct strmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\n+\thashmap_for_each_entry(&map->map, &iter, e, ent) {\n+\t\tstruct type_and_oid_list *list = e->value;\n+\t\toid_array_clear(&list->oids);\n+\t}\n+\tstrmap_clear(map, 1);\n+\tstrmap_init(map);\n+}\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info)\n+{\n+\tconst char *root_path = \"\";\n+\tint ret = 0;\n+\tsize_t commits_nr = 0, paths_nr = 0;\n+\tstruct commit *c;\n+\tstruct type_and_oid_list *root_tree_list;\n+\tstruct path_walk_context ctx = {\n+\t\t.repo = info->revs->repo,\n+\t\t.revs = info->revs,\n+\t\t.info = info,\n+\t\t.path_stack = STRING_LIST_INIT_DUP,\n+\t\t.path_stack_pushed = STRSET_INIT,\n+\t\t.paths_to_lists = STRMAP_INIT\n+\t};\n+\n+\ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\t/* Insert a single list for the root tree into the paths. */\n+\tCALLOC_ARRAY(root_tree_list, 1);\n+\troot_tree_list->type = OBJ_TREE;\n+\tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n+\tpush_to_stack(&ctx, root_path);\n+\n+\tif (prepare_revision_walk(info->revs))\n+\t\tdie(_(\"failed to setup revision walk\"));\n+\n+\twhile ((c = get_revision(info->revs))) {\n+\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct tree *t;\n+\t\tcommits_nr++;\n+\n+\t\toid = get_commit_tree_oid(c);\n+\t\tt = lookup_tree(info->revs->repo, oid);\n+\n+\t\tif (!t) {\n+\t\t\terror(\"could not find tree %s\", oid_to_hex(oid));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (t->object.flags & SEEN)\n+\t\t\tcontinue;\n+\t\tt->object.flags |= SEEN;\n+\t\toid_array_append(&root_tree_list->oids, oid);\n+\t}\n+\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n+\ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n+\n+\ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n+\twhile (!ret && ctx.path_stack.nr) {\n+\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\tctx.path_stack.nr--;\n+\t\tpaths_nr++;\n+\n+\t\tret = walk_path(&ctx, path);\n+\n+\t\tfree(path);\n+\t}\n+\ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n+\ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n+\n+\tclear_paths_to_lists(&ctx.paths_to_lists);\n+\tstrset_clear(&ctx.path_stack_pushed);\n+\tstring_list_clear(&ctx.path_stack, 0);\n+\treturn ret;\n+}\n+\n+void path_walk_info_init(struct path_walk_info *info)\n+{\n+\tstruct path_walk_info empty = PATH_WALK_INFO_INIT;\n+\tmemcpy(info, &empty, sizeof(empty));\n+}\n+\n+void path_walk_info_clear(struct path_walk_info *info UNUSED)\n+{\n+\t/*\n+\t * This destructor is empty for now, as info->revs\n+\t * is not owned by 'struct path_walk_info'.\n+\t */\n+}\ndiff --git a/path-walk.h b/path-walk.h\nnew file mode 100644\nindex 00000000000..7cb3538cd8f\n--- /dev/null\n+++ b/path-walk.h\n@@ -0,0 +1,47 @@\n+/*\n+ * path-walk.h : Methods and structures for walking the object graph in batches\n+ * by the paths that can reach those objects.\n+ */\n+#include \"object.h\" /* Required for 'enum object_type'. */\n+\n+struct rev_info;\n+struct oid_array;\n+\n+/**\n+ * The type of a function pointer for the method that is called on a list of\n+ * objects reachable at a given path.\n+ */\n+typedef int (*path_fn)(const char *path,\n+\t\t struct oid_array *oids,\n+\t\t enum object_type type,\n+\t\t void *data);\n+\n+struct path_walk_info {\n+\t/**\n+\t * revs provides the definitions for the commit walk, including\n+\t * which commits are UNINTERESTING or not. This structure is\n+\t * expected to be owned by the caller.\n+\t */\n+\tstruct rev_info *revs;\n+\n+\t/**\n+\t * The caller wishes to execute custom logic on objects reachable at a\n+\t * given path. Every reachable object will be visited exactly once, and\n+\t * the first path to see an object wins. This may not be a stable choice.\n+\t */\n+\tpath_fn path_fn;\n+\tvoid *path_fn_data;\n+};\n+\n+#define PATH_WALK_INFO_INIT { 0 }\n+\n+void path_walk_info_init(struct path_walk_info *info);\n+void path_walk_info_clear(struct path_walk_info *info);\n+\n+/**\n+ * Given the configuration of 'info', walk the commits based on 'info->revs' and\n+ * call 'info->path_fn' on each discovered path.\n+ *\n+ * Returns nonzero on an error.\n+ */\n+int walk_objects_by_path(struct path_walk_info *info);\n-- \ngitgitgadget\n\n"},{"id":"509411","messageId":"8ad2a5e79a25975394e5863f7a35293f1e2e5a44.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 2/7] test-lib-functions: add test_cmp_sorted","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:10Z","receivedAt":"2024-12-20T16:21:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis test helper will be helpful to reduce repeated logic in\nt6601-path-walk.sh, but may be helpful elsewhere, too.\n\nSigned-off-by: Derrick Stolee \n---\n t/test-lib-functions.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex fde9bf54fc3..16b70aebd60 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1267,6 +1267,16 @@ test_cmp () {\n \teval \"$GIT_TEST_CMP\" '\"$@\"'\n }\n \n+# test_cmp_sorted runs test_cmp on sorted versions of the two\n+# input files. Uses \"$1.sorted\" and \"$2.sorted\" as temp files.\n+\n+test_cmp_sorted () {\n+\tsort <\"$1\" >\"$1.sorted\" &&\n+\tsort <\"$2\" >\"$2.sorted\" &&\n+\ttest_cmp \"$1.sorted\" \"$2.sorted\" &&\n+\trm \"$1.sorted\" \"$2.sorted\"\n+}\n+\n # Check that the given config key has the expected value.\n #\n # test_cmp_config [-C ] \n-- \ngitgitgadget\n\n"},{"id":"509412","messageId":"2bc0538bce9ae2f502cd88aedf3b82c66e87c467.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 3/7] t6601: add helper for testing path-walk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:11Z","receivedAt":"2024-12-20T16:21:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests based on the current behavior, doing interesting checks\nfor different sets of branches, ranges, and the --boundary option. This\nsets a baseline for the behavior and we can extend it as new options are\nintroduced.\n\nStore and output a 'batch_nr' value so we can demonstrate that the paths are\ngrouped together in a batch and not following some other ordering. This\nallows us to test the depth-first behavior of the path-walk API. However, we\npurposefully do not test the order of the objects in the batch, so the\noutput is compared to the expected output through a sort.\n\nIt is important to mention that the behavior of the API will change soon as\nwe start to handle UNINTERESTING objects differently, but these tests will\ndemonstrate the change in behavior.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 3 +-\n Makefile | 1 +\n t/helper/test-path-walk.c | 84 +++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t6601-path-walk.sh | 120 ++++++++++++++++++++++\n 6 files changed, 209 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-path-walk.c\n create mode 100755 t/t6601-path-walk.sh\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex c550c77ca30..662162ec70b 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -42,4 +42,5 @@ commits.\n Examples\n --------\n \n-See example usages in future changes.\n+See example usages in:\n+\t`t/helper/test-path-walk.c`\ndiff --git a/Makefile b/Makefile\nindex d0d8d6888e3..50413d96492 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -818,6 +818,7 @@ TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n+TEST_BUILTINS_OBJS += test-path-walk.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\n TEST_BUILTINS_OBJS += test-proc-receive.o\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nnew file mode 100644\nindex 00000000000..def7c81ac4f\n--- /dev/null\n+++ b/t/helper/test-path-walk.c\n@@ -0,0 +1,84 @@\n+#define USE_THE_REPOSITORY_VARIABLE\n+\n+#include \"test-tool.h\"\n+#include \"environment.h\"\n+#include \"hex.h\"\n+#include \"object-name.h\"\n+#include \"object.h\"\n+#include \"pretty.h\"\n+#include \"revision.h\"\n+#include \"setup.h\"\n+#include \"parse-options.h\"\n+#include \"path-walk.h\"\n+#include \"oid-array.h\"\n+\n+static const char * const path_walk_usage[] = {\n+\tN_(\"test-tool path-walk -- \"),\n+\tNULL\n+};\n+\n+struct path_walk_test_data {\n+\tuintmax_t batch_nr;\n+\tuintmax_t tree_nr;\n+\tuintmax_t blob_nr;\n+};\n+\n+static int emit_block(const char *path, struct oid_array *oids,\n+\t\t enum object_type type, void *data)\n+{\n+\tstruct path_walk_test_data *tdata = data;\n+\tconst char *typestr;\n+\n+\tif (type == OBJ_TREE)\n+\t\ttdata->tree_nr += oids->nr;\n+\telse if (type == OBJ_BLOB)\n+\t\ttdata->blob_nr += oids->nr;\n+\telse\n+\t\tBUG(\"we do not understand this type\");\n+\n+\ttypestr = type_name(type);\n+\n+\tfor (size_t i = 0; i < oids->nr; i++)\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n+\t\t tdata->batch_nr, typestr, path,\n+\t\t oid_to_hex(&oids->oid[i]));\n+\n+\ttdata->batch_nr++;\n+\treturn 0;\n+}\n+\n+int cmd__path_walk(int argc, const char **argv)\n+{\n+\tint res;\n+\tstruct rev_info revs = REV_INFO_INIT;\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tstruct path_walk_test_data data = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tsetup_git_directory();\n+\trevs.repo = the_repository;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, path_walk_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT | PARSE_OPT_KEEP_ARGV0);\n+\n+\tif (argc > 1)\n+\t\tsetup_revisions(argc, argv, &revs, NULL);\n+\telse\n+\t\tusage(path_walk_usage[0]);\n+\n+\tinfo.revs = &revs;\n+\tinfo.path_fn = emit_block;\n+\tinfo.path_fn_data = &data;\n+\n+\tres = walk_objects_by_path(&info);\n+\n+\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\t \"blobs:%\" PRIuMAX \"\\n\",\n+\t data.tree_nr, data.blob_nr);\n+\n+\trelease_revisions(&revs);\n+\treturn res;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..43676e7b93a 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -52,6 +52,7 @@ static struct test_cmd cmds[] = {\n \t{ \"parse-subcommand\", cmd__parse_subcommand },\n \t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n+\t{ \"path-walk\", cmd__path_walk },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"proc-receive\", cmd__proc_receive },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..9cfc5da6e57 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__parse_pathspec_file(int argc, const char** argv);\n int cmd__parse_subcommand(int argc, const char **argv);\n int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n+int cmd__path_walk(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\n int cmd__proc_receive(int argc, const char **argv);\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nnew file mode 100755\nindex 00000000000..4e052c09309\n--- /dev/null\n+++ b/t/t6601-path-walk.sh\n@@ -0,0 +1,120 @@\n+#!/bin/sh\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+\n+test_description='direct path-walk API tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repository' '\n+\tgit checkout -b base &&\n+\n+\tmkdir left &&\n+\tmkdir right &&\n+\techo a >a &&\n+\techo b >left/b &&\n+\techo c >right/c &&\n+\tgit add . &&\n+\tgit commit -m \"first\" &&\n+\n+\techo d >right/d &&\n+\tgit add right &&\n+\tgit commit -m \"second\" &&\n+\n+\techo bb >left/b &&\n+\tgit commit -a -m \"third\" &&\n+\n+\tgit checkout -b topic HEAD~1 &&\n+\techo cc >right/c &&\n+\tgit commit -a -m \"topic\"\n+'\n+\n+test_expect_success 'all' '\n+\ttest-tool path-walk -- --all >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t0:tree::$(git rev-parse base^{tree})\n+\t0:tree::$(git rev-parse base~1^{tree})\n+\t0:tree::$(git rev-parse base~2^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t1:tree:right/:$(git rev-parse base~1:right)\n+\t1:tree:right/:$(git rev-parse base~2:right)\n+\t2:blob:right/d:$(git rev-parse base~1:right/d)\n+\t3:blob:right/c:$(git rev-parse base~2:right/c)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse base:left)\n+\t4:tree:left/:$(git rev-parse base~2:left)\n+\t5:blob:left/b:$(git rev-parse base~2:left/b)\n+\t5:blob:left/b:$(git rev-parse base:left/b)\n+\t6:blob:a:$(git rev-parse base~2:a)\n+\tblobs:6\n+\ttrees:9\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic only' '\n+\ttest-tool path-walk -- topic >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t0:tree::$(git rev-parse base~1^{tree})\n+\t0:tree::$(git rev-parse base~2^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t1:tree:right/:$(git rev-parse base~1:right)\n+\t1:tree:right/:$(git rev-parse base~2:right)\n+\t2:blob:right/d:$(git rev-parse base~1:right/d)\n+\t3:blob:right/c:$(git rev-parse base~2:right/c)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse base~2:left)\n+\t5:blob:left/b:$(git rev-parse base~2:left/b)\n+\t6:blob:a:$(git rev-parse base~2:a)\n+\tblobs:5\n+\ttrees:7\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base' '\n+\ttest-tool path-walk -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t2:blob:right/d:$(git rev-parse topic:right/d)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse topic:left)\n+\t5:blob:left/b:$(git rev-parse topic:left/b)\n+\t6:blob:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, boundary' '\n+\ttest-tool path-walk -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:tree::$(git rev-parse topic^{tree})\n+\t0:tree::$(git rev-parse base~1^{tree})\n+\t1:tree:right/:$(git rev-parse topic:right)\n+\t1:tree:right/:$(git rev-parse base~1:right)\n+\t2:blob:right/d:$(git rev-parse base~1:right/d)\n+\t3:blob:right/c:$(git rev-parse base~1:right/c)\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\t4:tree:left/:$(git rev-parse base~1:left)\n+\t5:blob:left/b:$(git rev-parse base~1:left/b)\n+\t6:blob:a:$(git rev-parse base~1:a)\n+\tblobs:5\n+\ttrees:5\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"509413","messageId":"db5c86112300130d9f1a62f67800fc5bc824b268.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 4/7] path-walk: allow consumer to specify object types","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:12Z","receivedAt":"2024-12-20T16:21:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe add the ability to filter the object types in the path-walk API so\nthe callback function is called fewer times.\n\nThis adds the ability to ask for the commits in a list, as well. We\nre-use the empty string for this set of objects because these are passed\ndirectly to the callback function instead of being part of the\n'path_stack'.\n\nFuture changes will add the ability to visit annotated tags.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 9 ++\n path-walk.c | 33 ++++-\n path-walk.h | 14 +-\n t/helper/test-path-walk.c | 15 ++-\n t/t6601-path-walk.sh | 149 +++++++++++++++-------\n 5 files changed, 170 insertions(+), 50 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 662162ec70b..dce553b6114 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,6 +39,15 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n+`commits`, `blobs`, `trees`::\n+\tBy default, these members are enabled and signal that the path-walk\n+\tAPI should call the `path_fn` on objects of these types. Specialized\n+\tapplications could disable some options to make it simpler to walk\n+\tthe objects or to have fewer calls to `path_fn`.\n++\n+While it is possible to walk only commits in this way, consumers would be\n+better off using the revision walk API instead.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex 021840ab41b..05ca7b2442a 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -98,6 +98,10 @@ static int add_tree_entries(struct path_walk_context *ctx,\n \t\tif (S_ISGITLINK(entry.mode))\n \t\t\tcontinue;\n \n+\t\t/* If the caller doesn't want blobs, then don't bother. */\n+\t\tif (!ctx->info->blobs && type == OBJ_BLOB)\n+\t\t\tcontinue;\n+\n \t\tif (type == OBJ_TREE) {\n \t\t\tstruct tree *child = lookup_tree(ctx->repo, &entry.oid);\n \t\t\to = child ? &child->object : NULL;\n@@ -159,9 +163,11 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n-\t/* Evaluate function pointer on this data. */\n-\tret = ctx->info->path_fn(path, &list->oids, list->type,\n-\t\t\t\t ctx->info->path_fn_data);\n+\t/* Evaluate function pointer on this data, if requested. */\n+\tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n+\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n+\t\t\t\t\tctx->info->path_fn_data);\n \n \t/* Expand data for children. */\n \tif (list->type == OBJ_TREE) {\n@@ -203,6 +209,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n \tstruct type_and_oid_list *root_tree_list;\n+\tstruct type_and_oid_list *commit_list;\n \tstruct path_walk_context ctx = {\n \t\t.repo = info->revs->repo,\n \t\t.revs = info->revs,\n@@ -214,6 +221,9 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \ttrace2_region_enter(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\tCALLOC_ARRAY(commit_list, 1);\n+\tcommit_list->type = OBJ_COMMIT;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n@@ -224,10 +234,18 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\tdie(_(\"failed to setup revision walk\"));\n \n \twhile ((c = get_revision(info->revs))) {\n-\t\tstruct object_id *oid = get_commit_tree_oid(c);\n+\t\tstruct object_id *oid;\n \t\tstruct tree *t;\n \t\tcommits_nr++;\n \n+\t\tif (info->commits)\n+\t\t\toid_array_append(&commit_list->oids,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t/* If we only care about commits, then skip trees. */\n+\t\tif (!info->trees && !info->blobs)\n+\t\t\tcontinue;\n+\n \t\toid = get_commit_tree_oid(c);\n \t\tt = lookup_tree(info->revs->repo, oid);\n \n@@ -245,6 +263,13 @@ int walk_objects_by_path(struct path_walk_info *info)\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"commits\", commits_nr);\n \ttrace2_region_leave(\"path-walk\", \"commit-walk\", info->revs->repo);\n \n+\t/* Track all commits. */\n+\tif (info->commits && commit_list->oids.nr)\n+\t\tret = info->path_fn(\"\", &commit_list->oids, OBJ_COMMIT,\n+\t\t\t\t info->path_fn_data);\n+\toid_array_clear(&commit_list->oids);\n+\tfree(commit_list);\n+\n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n \twhile (!ret && ctx.path_stack.nr) {\n \t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\ndiff --git a/path-walk.h b/path-walk.h\nindex 7cb3538cd8f..2cafc71e153 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -31,9 +31,21 @@ struct path_walk_info {\n \t */\n \tpath_fn path_fn;\n \tvoid *path_fn_data;\n+\n+\t/**\n+\t * Initialize which object types the path_fn should be called on. This\n+\t * could also limit the walk to skip blobs if not set.\n+\t */\n+\tint commits;\n+\tint trees;\n+\tint blobs;\n };\n \n-#define PATH_WALK_INFO_INIT { 0 }\n+#define PATH_WALK_INFO_INIT { \\\n+\t.blobs = 1,\t\t\\\n+\t.trees = 1,\t\t\\\n+\t.commits = 1,\t\t\\\n+}\n \n void path_walk_info_init(struct path_walk_info *info);\n void path_walk_info_clear(struct path_walk_info *info);\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex def7c81ac4f..a57a05a6391 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -19,6 +19,8 @@ static const char * const path_walk_usage[] = {\n \n struct path_walk_test_data {\n \tuintmax_t batch_nr;\n+\n+\tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n };\n@@ -33,6 +35,8 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->tree_nr += oids->nr;\n \telse if (type == OBJ_BLOB)\n \t\ttdata->blob_nr += oids->nr;\n+\telse if (type == OBJ_COMMIT)\n+\t\ttdata->commit_nr += oids->nr;\n \telse\n \t\tBUG(\"we do not understand this type\");\n \n@@ -54,6 +58,12 @@ int cmd__path_walk(int argc, const char **argv)\n \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n \tstruct path_walk_test_data data = { 0 };\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"blobs\", &info.blobs,\n+\t\t\t N_(\"toggle inclusion of blob objects\")),\n+\t\tOPT_BOOL(0, \"commits\", &info.commits,\n+\t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"trees\", &info.trees,\n+\t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -75,9 +85,10 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tres = walk_objects_by_path(&info);\n \n-\tprintf(\"trees:%\" PRIuMAX \"\\n\"\n+\tprintf(\"commits:%\" PRIuMAX \"\\n\"\n+\t \"trees:%\" PRIuMAX \"\\n\"\n \t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.tree_nr, data.blob_nr);\n+\t data.commit_nr, data.tree_nr, data.blob_nr);\n \n \trelease_revisions(&revs);\n \treturn res;\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 4e052c09309..4a4939a1b02 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -33,22 +33,27 @@ test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:tree::$(git rev-parse topic^{tree})\n-\t0:tree::$(git rev-parse base^{tree})\n-\t0:tree::$(git rev-parse base~1^{tree})\n-\t0:tree::$(git rev-parse base~2^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t1:tree:right/:$(git rev-parse base~1:right)\n-\t1:tree:right/:$(git rev-parse base~2:right)\n-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n-\t3:blob:right/c:$(git rev-parse base~2:right/c)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse base:left)\n-\t4:tree:left/:$(git rev-parse base~2:left)\n-\t5:blob:left/b:$(git rev-parse base~2:left/b)\n-\t5:blob:left/b:$(git rev-parse base:left/b)\n-\t6:blob:a:$(git rev-parse base~2:a)\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base)\n+\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~2)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree})\n+\t1:tree::$(git rev-parse base~2^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right)\n+\t2:tree:right/:$(git rev-parse base~2:right)\n+\t3:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/c:$(git rev-parse base~2:right/c)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse base:left)\n+\t5:tree:left/:$(git rev-parse base~2:left)\n+\t6:blob:left/b:$(git rev-parse base~2:left/b)\n+\t6:blob:left/b:$(git rev-parse base:left/b)\n+\t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:6\n+\tcommits:4\n \ttrees:9\n \tEOF\n \n@@ -59,19 +64,23 @@ test_expect_success 'topic only' '\n \ttest-tool path-walk -- topic >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:tree::$(git rev-parse topic^{tree})\n-\t0:tree::$(git rev-parse base~1^{tree})\n-\t0:tree::$(git rev-parse base~2^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t1:tree:right/:$(git rev-parse base~1:right)\n-\t1:tree:right/:$(git rev-parse base~2:right)\n-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n-\t3:blob:right/c:$(git rev-parse base~2:right/c)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse base~2:left)\n-\t5:blob:left/b:$(git rev-parse base~2:left/b)\n-\t6:blob:a:$(git rev-parse base~2:a)\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~2)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree})\n+\t1:tree::$(git rev-parse base~2^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right)\n+\t2:tree:right/:$(git rev-parse base~2:right)\n+\t3:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/c:$(git rev-parse base~2:right/c)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse base~2:left)\n+\t6:blob:left/b:$(git rev-parse base~2:left/b)\n+\t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:5\n+\tcommits:3\n \ttrees:7\n \tEOF\n \n@@ -82,15 +91,66 @@ test_expect_success 'topic, not base' '\n \ttest-tool path-walk -- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t3:blob:right/d:$(git rev-parse topic:right/d)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse topic:left)\n+\t6:blob:left/b:$(git rev-parse topic:left/b)\n+\t7:blob:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\tcommits:1\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only blobs' '\n+\ttest-tool path-walk --no-trees --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n+\ttrees:0\n+\t0:blob:right/d:$(git rev-parse topic:right/d)\n+\t1:blob:right/c:$(git rev-parse topic:right/c)\n+\t2:blob:left/b:$(git rev-parse topic:left/b)\n+\t3:blob:a:$(git rev-parse topic:a)\n+\tblobs:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+# No, this doesn't make a lot of sense for the path-walk API,\n+# but it is possible to do.\n+test_expect_success 'topic, not base, only commits' '\n+\ttest-tool path-walk --no-blobs --no-trees \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\tcommits:1\n+\ttrees:0\n+\tblobs:0\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'topic, not base, only trees' '\n+\ttest-tool path-walk --no-blobs --no-commits \\\n+\t\t-- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommits:0\n \t0:tree::$(git rev-parse topic^{tree})\n \t1:tree:right/:$(git rev-parse topic:right)\n-\t2:blob:right/d:$(git rev-parse topic:right/d)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse topic:left)\n-\t5:blob:left/b:$(git rev-parse topic:left/b)\n-\t6:blob:a:$(git rev-parse topic:a)\n-\tblobs:4\n+\t2:tree:left/:$(git rev-parse topic:left)\n \ttrees:3\n+\tblobs:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -100,17 +160,20 @@ test_expect_success 'topic, not base, boundary' '\n \ttest-tool path-walk -- --boundary topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:tree::$(git rev-parse topic^{tree})\n-\t0:tree::$(git rev-parse base~1^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t1:tree:right/:$(git rev-parse base~1:right)\n-\t2:blob:right/d:$(git rev-parse base~1:right/d)\n-\t3:blob:right/c:$(git rev-parse base~1:right/c)\n-\t3:blob:right/c:$(git rev-parse topic:right/c)\n-\t4:tree:left/:$(git rev-parse base~1:left)\n-\t5:blob:left/b:$(git rev-parse base~1:left/b)\n-\t6:blob:a:$(git rev-parse base~1:a)\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base~1)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree})\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right)\n+\t3:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/c:$(git rev-parse base~1:right/c)\n+\t4:blob:right/c:$(git rev-parse topic:right/c)\n+\t5:tree:left/:$(git rev-parse base~1:left)\n+\t6:blob:left/b:$(git rev-parse base~1:left/b)\n+\t7:blob:a:$(git rev-parse base~1:a)\n \tblobs:5\n+\tcommits:2\n \ttrees:5\n \tEOF\n \n-- \ngitgitgadget\n\n"},{"id":"509414","messageId":"6df56f465d7bdb2c2d360752f6530e29b4d3dc3c.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 5/7] path-walk: visit tags and cached objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:13Z","receivedAt":"2024-12-20T16:21:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe rev_info that is specified for a path-walk traversal may specify\nvisiting tag refs (both lightweight and annotated) and also may specify\nindexed objects (blobs and trees). Update the path-walk API to walk\nthese objects as well.\n\nWhen walking tags, we need to peel the annotated objects until reaching\na non-tag object. If we reach a commit, then we can add it to the\npending objects to make sure we visit in the commit walk portion. If we\nreach a tree, then we will assume that it is a root tree. If we reach a\nblob, then we have no good path name and so add it to a new list of\n\"tagged blobs\".\n\nWhen the rev_info includes the \"--indexed-objects\" flag, then the\npending set includes blobs and trees found in the cache entries and\ncache-tree. The cache entries are usually blobs, though they could be\ntrees in the case of a sparse index. The cache-tree stores\npreviously-hashed tree objects but these are cleared out when staging\nobjects below those paths. We add tests that demonstrate this.\n\nThe indexed objects come with a non-NULL 'path' value in the pending\nitem. This allows us to prepopulate the 'path_to_lists' strmap with\nlists for these paths.\n\nThe tricky thing about this walk is that we will want to combine the\nindexed objects walk with the commit walk, especially in the future case\nof walking objects during a command like 'git repack'.\n\nWhenever possible, we want the objects from the index to be grouped with\nsimilar objects in history. We don't want to miss any paths that appear\nonly in the index and not in the commit history.\n\nThus, we need to be careful to let the path stack be populated initially\nwith only the root tree path (and possibly tags and tagged blobs) and go\nthrough the normal depth-first search. Afterwards, if there are other\npaths that are remaining in the paths_to_lists strmap, we should then\niterate through the stack and visit those objects recursively.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 2 +-\n path-walk.c | 184 ++++++++++++++++++++-\n path-walk.h | 2 +\n t/helper/test-path-walk.c | 15 +-\n t/t6601-path-walk.sh | 186 +++++++++++++++++++---\n 5 files changed, 362 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex dce553b6114..6022c381b7c 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -39,7 +39,7 @@ It is also important that you do not specify the `--objects` flag for the\n the objects will be walked in a separate way based on those starting\n commits.\n \n-`commits`, `blobs`, `trees`::\n+`commits`, `blobs`, `trees`, `tags`::\n \tBy default, these members are enabled and signal that the path-walk\n \tAPI should call the `path_fn` on objects of these types. Specialized\n \tapplications could disable some options to make it simpler to walk\ndiff --git a/path-walk.c b/path-walk.c\nindex 05ca7b2442a..f34dbf61de0 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -13,10 +13,13 @@\n #include \"revision.h\"\n #include \"string-list.h\"\n #include \"strmap.h\"\n+#include \"tag.h\"\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n \n+static const char *root_path = \"\";\n+\n struct type_and_oid_list {\n \tenum object_type type;\n \tstruct oid_array oids;\n@@ -160,12 +163,16 @@ static int walk_path(struct path_walk_context *ctx,\n \n \tlist = strmap_get(&ctx->paths_to_lists, path);\n \n+\tif (!list)\n+\t\tBUG(\"provided path '%s' that had no associated list\", path);\n+\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n-\t (list->type == OBJ_BLOB && ctx->info->blobs))\n+\t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n+\t (list->type == OBJ_TAG && ctx->info->tags))\n \t\tret = ctx->info->path_fn(path, &list->oids, list->type,\n \t\t\t\t\tctx->info->path_fn_data);\n \n@@ -196,6 +203,139 @@ static void clear_paths_to_lists(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static int setup_pending_objects(struct path_walk_info *info,\n+\t\t\t\t struct path_walk_context *ctx)\n+{\n+\tstruct type_and_oid_list *tags = NULL;\n+\tstruct type_and_oid_list *tagged_blobs = NULL;\n+\tstruct type_and_oid_list *root_tree_list = NULL;\n+\n+\tif (info->tags)\n+\t\tCALLOC_ARRAY(tags, 1);\n+\tif (info->blobs)\n+\t\tCALLOC_ARRAY(tagged_blobs, 1);\n+\tif (info->trees)\n+\t\troot_tree_list = strmap_get(&ctx->paths_to_lists, root_path);\n+\n+\t/*\n+\t * Pending objects include:\n+\t * * Commits at branch tips.\n+\t * * Annotated tags at tag tips.\n+\t * * Any kind of object at lightweight tag tips.\n+\t * * Trees and blobs in the index (with an associated path).\n+\t */\n+\tfor (size_t i = 0; i < info->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = info->revs->pending.objects + i;\n+\t\tstruct object *obj = pending->item;\n+\n+\t\t/* Commits will be picked up by revision walk. */\n+\t\tif (obj->type == OBJ_COMMIT)\n+\t\t\tcontinue;\n+\n+\t\t/* Navigate annotated tag object chains. */\n+\t\twhile (obj->type == OBJ_TAG) {\n+\t\t\tstruct tag *tag = lookup_tag(info->revs->repo, &obj->oid);\n+\t\t\tif (!tag) {\n+\t\t\t\terror(_(\"failed to find tag %s\"),\n+\t\t\t\t oid_to_hex(&obj->oid));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t\tif (tag->object.flags & SEEN)\n+\t\t\t\tbreak;\n+\t\t\ttag->object.flags |= SEEN;\n+\n+\t\t\tif (tags)\n+\t\t\t\toid_array_append(&tags->oids, &obj->oid);\n+\t\t\tobj = tag->tagged;\n+\t\t}\n+\n+\t\tif (obj->type == OBJ_TAG)\n+\t\t\tcontinue;\n+\n+\t\t/* We are now at a non-tag object. */\n+\t\tif (obj->flags & SEEN)\n+\t\t\tcontinue;\n+\t\tobj->flags |= SEEN;\n+\n+\t\tswitch (obj->type) {\n+\t\tcase OBJ_TREE:\n+\t\t\tif (!info->trees)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = *pending->path ? xstrfmt(\"%s/\", pending->path)\n+\t\t\t\t\t\t\t : xstrdup(\"\");\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_TREE;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t\tfree(path);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&root_tree_list->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (!info->blobs)\n+\t\t\t\tcontinue;\n+\t\t\tif (pending->path) {\n+\t\t\t\tstruct type_and_oid_list *list;\n+\t\t\t\tchar *path = pending->path;\n+\t\t\t\tif (!(list = strmap_get(&ctx->paths_to_lists, path))) {\n+\t\t\t\t\tCALLOC_ARRAY(list, 1);\n+\t\t\t\t\tlist->type = OBJ_BLOB;\n+\t\t\t\t\tstrmap_put(&ctx->paths_to_lists, path, list);\n+\t\t\t\t}\n+\t\t\t\toid_array_append(&list->oids, &obj->oid);\n+\t\t\t} else {\n+\t\t\t\t/* assume a root tree, such as a lightweight tag. */\n+\t\t\t\toid_array_append(&tagged_blobs->oids, &obj->oid);\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase OBJ_COMMIT:\n+\t\t\t/* Make sure it is in the object walk */\n+\t\t\tif (obj != pending->item)\n+\t\t\t\tadd_pending_object(info->revs, obj, \"\");\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"should not see any other type here\");\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Add tag objects and tagged blobs if they exist.\n+\t */\n+\tif (tagged_blobs) {\n+\t\tif (tagged_blobs->oids.nr) {\n+\t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n+\t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\tpush_to_stack(ctx, tagged_blob_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n+\t\t} else {\n+\t\t\toid_array_clear(&tagged_blobs->oids);\n+\t\t\tfree(tagged_blobs);\n+\t\t}\n+\t}\n+\tif (tags) {\n+\t\tif (tags->oids.nr) {\n+\t\t\tconst char *tag_path = \"/tags\";\n+\t\t\ttags->type = OBJ_TAG;\n+\t\t\tpush_to_stack(ctx, tag_path);\n+\t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n+\t\t} else {\n+\t\t\toid_array_clear(&tags->oids);\n+\t\t\tfree(tags);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n /**\n * Given the configuration of 'info', walk the commits based on 'info->revs' and\n * call 'info->path_fn' on each discovered path.\n@@ -204,8 +344,7 @@ static void clear_paths_to_lists(struct strmap *map)\n */\n int walk_objects_by_path(struct path_walk_info *info)\n {\n-\tconst char *root_path = \"\";\n-\tint ret = 0;\n+\tint ret;\n \tsize_t commits_nr = 0, paths_nr = 0;\n \tstruct commit *c;\n \tstruct type_and_oid_list *root_tree_list;\n@@ -224,15 +363,34 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tCALLOC_ARRAY(commit_list, 1);\n \tcommit_list->type = OBJ_COMMIT;\n \n+\tif (info->tags)\n+\t\tinfo->revs->tag_objects = 1;\n+\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n+\t/*\n+\t * Set these values before preparing the walk to catch\n+\t * lightweight tags pointing to non-commits and indexed objects.\n+\t */\n+\tinfo->revs->blob_objects = info->blobs;\n+\tinfo->revs->tree_objects = info->trees;\n+\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n+\n+\ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\tret = setup_pending_objects(info, &ctx);\n+\ttrace2_region_leave(\"path-walk\", \"pending-walk\", info->revs->repo);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n \twhile ((c = get_revision(info->revs))) {\n \t\tstruct object_id *oid;\n \t\tstruct tree *t;\n@@ -280,6 +438,26 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \t\tfree(path);\n \t}\n+\n+\t/* Are there paths remaining? Likely they are from indexed objects. */\n+\tif (!strmap_empty(&ctx.paths_to_lists)) {\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *entry;\n+\n+\t\tstrmap_for_each_entry(&ctx.paths_to_lists, &iter, entry)\n+\t\t\tpush_to_stack(&ctx, entry->key);\n+\n+\t\twhile (!ret && ctx.path_stack.nr) {\n+\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n+\t\t\tctx.path_stack.nr--;\n+\t\t\tpaths_nr++;\n+\n+\t\t\tret = walk_path(&ctx, path);\n+\n+\t\t\tfree(path);\n+\t\t}\n+\t}\n+\n \ttrace2_data_intmax(\"path-walk\", ctx.repo, \"paths\", paths_nr);\n \ttrace2_region_leave(\"path-walk\", \"path-walk\", info->revs->repo);\n \ndiff --git a/path-walk.h b/path-walk.h\nindex 2cafc71e153..3679fa7a859 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -39,12 +39,14 @@ struct path_walk_info {\n \tint commits;\n \tint trees;\n \tint blobs;\n+\tint tags;\n };\n \n #define PATH_WALK_INFO_INIT { \\\n \t.blobs = 1,\t\t\\\n \t.trees = 1,\t\t\\\n \t.commits = 1,\t\t\\\n+\t.tags = 1,\t\t\\\n }\n \n void path_walk_info_init(struct path_walk_info *info);\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex a57a05a6391..56289859e69 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -23,6 +23,7 @@ struct path_walk_test_data {\n \tuintmax_t commit_nr;\n \tuintmax_t tree_nr;\n \tuintmax_t blob_nr;\n+\tuintmax_t tag_nr;\n };\n \n static int emit_block(const char *path, struct oid_array *oids,\n@@ -37,11 +38,18 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\ttdata->blob_nr += oids->nr;\n \telse if (type == OBJ_COMMIT)\n \t\ttdata->commit_nr += oids->nr;\n+\telse if (type == OBJ_TAG)\n+\t\ttdata->tag_nr += oids->nr;\n \telse\n \t\tBUG(\"we do not understand this type\");\n \n \ttypestr = type_name(type);\n \n+\t/* This should never be output during tests. */\n+\tif (!oids->nr)\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n+\t\t tdata->batch_nr, typestr, path);\n+\n \tfor (size_t i = 0; i < oids->nr; i++)\n \t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n \t\t tdata->batch_nr, typestr, path,\n@@ -62,6 +70,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of blob objects\")),\n \t\tOPT_BOOL(0, \"commits\", &info.commits,\n \t\t\t N_(\"toggle inclusion of commit objects\")),\n+\t\tOPT_BOOL(0, \"tags\", &info.tags,\n+\t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_END(),\n@@ -87,8 +97,9 @@ int cmd__path_walk(int argc, const char **argv)\n \n \tprintf(\"commits:%\" PRIuMAX \"\\n\"\n \t \"trees:%\" PRIuMAX \"\\n\"\n-\t \"blobs:%\" PRIuMAX \"\\n\",\n-\t data.commit_nr, data.tree_nr, data.blob_nr);\n+\t \"blobs:%\" PRIuMAX \"\\n\"\n+\t \"tags:%\" PRIuMAX \"\\n\",\n+\t data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);\n \n \trelease_revisions(&revs);\n \treturn res;\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 4a4939a1b02..1f3d2e0cb76 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -9,29 +9,142 @@ test_description='direct path-walk API tests'\n test_expect_success 'setup test repository' '\n \tgit checkout -b base &&\n \n+\t# Make some objects that will only be reachable\n+\t# via non-commit tags.\n+\tmkdir child &&\n+\techo file >child/file &&\n+\tgit add child &&\n+\tgit commit -m \"will abandon\" &&\n+\tgit tag -a -m \"tree\" tree-tag HEAD^{tree} &&\n+\techo file2 >file2 &&\n+\tgit add file2 &&\n+\tgit commit --amend -m \"will abandon\" &&\n+\tgit tag tree-tag2 HEAD^{tree} &&\n+\n+\techo blob >file &&\n+\tblob_oid=$(git hash-object -t blob -w --stdin file2 &&\n+\tblob2_oid=$(git hash-object -t blob -w --stdin a &&\n \techo b >left/b &&\n \techo c >right/c &&\n \tgit add . &&\n-\tgit commit -m \"first\" &&\n+\tgit commit --amend -m \"first\" &&\n+\tgit tag -m \"first\" first HEAD &&\n \n \techo d >right/d &&\n \tgit add right &&\n \tgit commit -m \"second\" &&\n+\tgit tag -a -m \"second (under)\" second.1 HEAD &&\n+\tgit tag -a -m \"second (top)\" second.2 second.1 &&\n \n+\t# Set up file/dir collision in history.\n+\trm a &&\n+\tmkdir a &&\n+\techo a >a/a &&\n \techo bb >left/b &&\n-\tgit commit -a -m \"third\" &&\n+\tgit add a left &&\n+\tgit commit -m \"third\" &&\n+\tgit tag -a -m \"third\" third &&\n \n \tgit checkout -b topic HEAD~1 &&\n \techo cc >right/c &&\n-\tgit commit -a -m \"topic\"\n+\tgit commit -a -m \"topic\" &&\n+\tgit tag -a -m \"fourth\" fourth\n '\n \n test_expect_success 'all' '\n \ttest-tool path-walk -- --all >out &&\n \n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base)\n+\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~2)\n+\t1:tag:/tags:$(git rev-parse refs/tags/first)\n+\t1:tag:/tags:$(git rev-parse refs/tags/second.1)\n+\t1:tag:/tags:$(git rev-parse refs/tags/second.2)\n+\t1:tag:/tags:$(git rev-parse refs/tags/third)\n+\t1:tag:/tags:$(git rev-parse refs/tags/fourth)\n+\t1:tag:/tags:$(git rev-parse refs/tags/tree-tag)\n+\t1:tag:/tags:$(git rev-parse refs/tags/blob-tag)\n+\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})\n+\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\t3:tree::$(git rev-parse topic^{tree})\n+\t3:tree::$(git rev-parse base^{tree})\n+\t3:tree::$(git rev-parse base~1^{tree})\n+\t3:tree::$(git rev-parse base~2^{tree})\n+\t3:tree::$(git rev-parse refs/tags/tree-tag^{})\n+\t3:tree::$(git rev-parse refs/tags/tree-tag2^{})\n+\t4:blob:a:$(git rev-parse base~2:a)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right)\n+\t5:tree:right/:$(git rev-parse base~2:right)\n+\t6:blob:right/d:$(git rev-parse base~1:right/d)\n+\t7:blob:right/c:$(git rev-parse base~2:right/c)\n+\t7:blob:right/c:$(git rev-parse topic:right/c)\n+\t8:tree:left/:$(git rev-parse base:left)\n+\t8:tree:left/:$(git rev-parse base~2:left)\n+\t9:blob:left/b:$(git rev-parse base~2:left/b)\n+\t9:blob:left/b:$(git rev-parse base:left/b)\n+\t10:tree:a/:$(git rev-parse base:a)\n+\t11:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n+\t12:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n+\t13:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\tblobs:10\n+\tcommits:4\n+\ttags:7\n+\ttrees:13\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'indexed objects' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"left\" directory.\n+\techo bogus >left/c &&\n+\tgit add left &&\n+\n+\ttest-tool path-walk -- --indexed-objects >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:blob:a:$(git rev-parse HEAD:a)\n+\t1:blob:left/b:$(git rev-parse HEAD:left/b)\n+\t2:blob:left/c:$(git rev-parse :left/c)\n+\t3:blob:right/c:$(git rev-parse HEAD:right/c)\n+\t4:blob:right/d:$(git rev-parse HEAD:right/d)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\tblobs:5\n+\tcommits:0\n+\ttags:0\n+\ttrees:1\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n+test_expect_success 'branches and indexed objects mix well' '\n+\ttest_when_finished git reset --hard &&\n+\n+\t# stage change into index, adding a blob but\n+\t# also invalidating the cache-tree for the root\n+\t# and the \"right\" directory.\n+\techo fake >right/d &&\n+\tgit add right &&\n+\n+\ttest-tool path-walk -- --indexed-objects --branches >out &&\n+\n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n \t0:commit::$(git rev-parse base)\n@@ -41,20 +154,23 @@ test_expect_success 'all' '\n \t1:tree::$(git rev-parse base^{tree})\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right)\n-\t2:tree:right/:$(git rev-parse base~2:right)\n-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/c:$(git rev-parse base~2:right/c)\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base:left)\n-\t5:tree:left/:$(git rev-parse base~2:left)\n-\t6:blob:left/b:$(git rev-parse base~2:left/b)\n-\t6:blob:left/b:$(git rev-parse base:left/b)\n-\t7:blob:a:$(git rev-parse base~2:a)\n-\tblobs:6\n+\t2:blob:a:$(git rev-parse base~2:a)\n+\t3:tree:right/:$(git rev-parse topic:right)\n+\t3:tree:right/:$(git rev-parse base~1:right)\n+\t3:tree:right/:$(git rev-parse base~2:right)\n+\t4:blob:right/d:$(git rev-parse base~1:right/d)\n+\t4:blob:right/d:$(git rev-parse :right/d)\n+\t5:blob:right/c:$(git rev-parse base~2:right/c)\n+\t5:blob:right/c:$(git rev-parse topic:right/c)\n+\t6:tree:left/:$(git rev-parse base:left)\n+\t6:tree:left/:$(git rev-parse base~2:left)\n+\t7:blob:left/b:$(git rev-parse base:left/b)\n+\t7:blob:left/b:$(git rev-parse base~2:left/b)\n+\t8:tree:a/:$(git rev-parse refs/tags/third:a)\n+\tblobs:7\n \tcommits:4\n-\ttrees:9\n+\ttags:0\n+\ttrees:10\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -81,6 +197,7 @@ test_expect_success 'topic only' '\n \t7:blob:a:$(git rev-parse base~2:a)\n \tblobs:5\n \tcommits:3\n+\ttags:0\n \ttrees:7\n \tEOF\n \n@@ -101,6 +218,7 @@ test_expect_success 'topic, not base' '\n \t7:blob:a:$(git rev-parse topic:a)\n \tblobs:4\n \tcommits:1\n+\ttags:0\n \ttrees:3\n \tEOF\n \n@@ -112,13 +230,14 @@ test_expect_success 'topic, not base, only blobs' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\tcommits:0\n-\ttrees:0\n \t0:blob:right/d:$(git rev-parse topic:right/d)\n \t1:blob:right/c:$(git rev-parse topic:right/c)\n \t2:blob:left/b:$(git rev-parse topic:left/b)\n \t3:blob:a:$(git rev-parse topic:a)\n \tblobs:4\n+\tcommits:0\n+\ttags:0\n+\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -133,8 +252,9 @@ test_expect_success 'topic, not base, only commits' '\n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n \tcommits:1\n-\ttrees:0\n \tblobs:0\n+\ttags:0\n+\ttrees:0\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -145,12 +265,13 @@ test_expect_success 'topic, not base, only trees' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\tcommits:0\n \t0:tree::$(git rev-parse topic^{tree})\n \t1:tree:right/:$(git rev-parse topic:right)\n \t2:tree:left/:$(git rev-parse topic:left)\n-\ttrees:3\n+\tcommits:0\n \tblobs:0\n+\ttags:0\n+\ttrees:3\n \tEOF\n \n \ttest_cmp_sorted expect out\n@@ -174,10 +295,33 @@ test_expect_success 'topic, not base, boundary' '\n \t7:blob:a:$(git rev-parse base~1:a)\n \tblobs:5\n \tcommits:2\n+\ttags:0\n \ttrees:5\n \tEOF\n \n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'trees are reported exactly once' '\n+\ttest_when_finished \"rm -rf unique-trees\" &&\n+\ttest_create_repo unique-trees &&\n+\t(\n+\t\tcd unique-trees &&\n+\t\tmkdir initial &&\n+\t\ttest_commit initial/file &&\n+\n+\t\tgit switch -c move-to-top &&\n+\t\tgit mv initial/file.t ./ &&\n+\t\ttest_tick &&\n+\t\tgit commit -m moved &&\n+\n+\t\tgit update-ref refs/heads/other HEAD\n+\t) &&\n+\n+\ttest-tool -C unique-trees path-walk -- --all >out &&\n+\ttree=$(git -C unique-trees rev-parse HEAD:) &&\n+\tgrep \"$tree\" out >out-filtered &&\n+\ttest_line_count = 1 out-filtered\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"509415","messageId":"f2ffc32a3033549420fb9e3a902cce1cc739440e.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 6/7] path-walk: mark trees and blobs as UNINTERESTING","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:14Z","receivedAt":"2024-12-20T16:21:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the input rev_info has UNINTERESTING starting points, we want to be\nsure that the UNINTERESTING flag is passed appropriately through the\nobjects. To match how this is done in places such as 'git pack-objects', we\nuse the mark_edges_uninteresting() method.\n\nThis method has an option for using the \"sparse\" walk, which is similar in\nspirit to the path-walk API's walk. To be sure to keep it independent, add a\nnew 'prune_all_uninteresting' option to the path_walk_info struct.\n\nTo check how the UNINTERSTING flag is spread through our objects, extend the\n'test-tool path-walk' command to output whether or not an object has that\nflag. This changes our tests significantly, including the removal of some\nobjects that were previously visited due to the incomplete implementation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/api-path-walk.txt | 8 +++\n path-walk.c | 74 +++++++++++++++++++++\n path-walk.h | 8 +++\n t/helper/test-path-walk.c | 12 +++-\n t/t6601-path-walk.sh | 79 +++++++++++++++++------\n 5 files changed, 159 insertions(+), 22 deletions(-)\n\ndiff --git a/Documentation/technical/api-path-walk.txt b/Documentation/technical/api-path-walk.txt\nindex 6022c381b7c..7075d0d5ab5 100644\n--- a/Documentation/technical/api-path-walk.txt\n+++ b/Documentation/technical/api-path-walk.txt\n@@ -48,6 +48,14 @@ commits.\n While it is possible to walk only commits in this way, consumers would be\n better off using the revision walk API instead.\n \n+`prune_all_uninteresting`::\n+\tBy default, all reachable paths are emitted by the path-walk API.\n+\tThis option allows consumers to declare that they are not\n+\tinterested in paths where all included objects are marked with the\n+\t`UNINTERESTING` flag. This requires using the `boundary` option in\n+\tthe revision walk so that the walk emits commits marked with the\n+\t`UNINTERESTING` flag.\n+\n Examples\n --------\n \ndiff --git a/path-walk.c b/path-walk.c\nindex f34dbf61de0..4013569e9e4 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -8,6 +8,7 @@\n #include \"dir.h\"\n #include \"hashmap.h\"\n #include \"hex.h\"\n+#include \"list-objects.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n #include \"revision.h\"\n@@ -23,6 +24,7 @@ static const char *root_path = \"\";\n struct type_and_oid_list {\n \tenum object_type type;\n \tstruct oid_array oids;\n+\tint maybe_interesting;\n };\n \n #define TYPE_AND_OID_LIST_INIT { \\\n@@ -142,6 +144,10 @@ static int add_tree_entries(struct path_walk_context *ctx,\n \t\t\tstrmap_put(&ctx->paths_to_lists, path.buf, list);\n \t\t}\n \t\tpush_to_stack(ctx, path.buf);\n+\n+\t\tif (!(o->flags & UNINTERESTING))\n+\t\t\tlist->maybe_interesting = 1;\n+\n \t\toid_array_append(&list->oids, &entry.oid);\n \t}\n \n@@ -169,6 +175,43 @@ static int walk_path(struct path_walk_context *ctx,\n \tif (!list->oids.nr)\n \t\treturn 0;\n \n+\tif (ctx->info->prune_all_uninteresting) {\n+\t\t/*\n+\t\t * This is true if all objects were UNINTERESTING\n+\t\t * when added to the list.\n+\t\t */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\n+\t\t/*\n+\t\t * But it's still possible that the objects were set\n+\t\t * as UNINTERESTING after being added. Do a quick check.\n+\t\t */\n+\t\tlist->maybe_interesting = 0;\n+\t\tfor (size_t i = 0;\n+\t\t !list->maybe_interesting && i < list->oids.nr;\n+\t\t i++) {\n+\t\t\tif (list->type == OBJ_TREE) {\n+\t\t\t\tstruct tree *t = lookup_tree(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (t && !(t->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else if (list->type == OBJ_BLOB) {\n+\t\t\t\tstruct blob *b = lookup_blob(ctx->repo,\n+\t\t\t\t\t\t\t &list->oids.oid[i]);\n+\t\t\t\tif (b && !(b->object.flags & UNINTERESTING))\n+\t\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t} else {\n+\t\t\t\t/* Tags are always interesting if visited. */\n+\t\t\t\tlist->maybe_interesting = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* We have confirmed that all objects are UNINTERESTING. */\n+\t\tif (!list->maybe_interesting)\n+\t\t\treturn 0;\n+\t}\n+\n \t/* Evaluate function pointer on this data, if requested. */\n \tif ((list->type == OBJ_TREE && ctx->info->trees) ||\n \t (list->type == OBJ_BLOB && ctx->info->blobs) ||\n@@ -203,6 +246,26 @@ static void clear_paths_to_lists(struct strmap *map)\n \tstrmap_init(map);\n }\n \n+static struct repository *edge_repo;\n+static struct type_and_oid_list *edge_tree_list;\n+\n+static void show_edge(struct commit *commit)\n+{\n+\tstruct tree *t = repo_get_commit_tree(edge_repo, commit);\n+\n+\tif (!t)\n+\t\treturn;\n+\n+\tif (commit->object.flags & UNINTERESTING)\n+\t\tt->object.flags |= UNINTERESTING;\n+\n+\tif (t->object.flags & SEEN)\n+\t\treturn;\n+\tt->object.flags |= SEEN;\n+\n+\toid_array_append(&edge_tree_list->oids, &t->object.oid);\n+}\n+\n static int setup_pending_objects(struct path_walk_info *info,\n \t\t\t\t struct path_walk_context *ctx)\n {\n@@ -314,6 +377,7 @@ static int setup_pending_objects(struct path_walk_info *info,\n \t\tif (tagged_blobs->oids.nr) {\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n+\t\t\ttagged_blobs->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n \t\t} else {\n@@ -325,6 +389,7 @@ static int setup_pending_objects(struct path_walk_info *info,\n \t\tif (tags->oids.nr) {\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n+\t\t\ttags->maybe_interesting = 1;\n \t\t\tpush_to_stack(ctx, tag_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n \t\t} else {\n@@ -369,6 +434,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t/* Insert a single list for the root tree into the paths. */\n \tCALLOC_ARRAY(root_tree_list, 1);\n \troot_tree_list->type = OBJ_TREE;\n+\troot_tree_list->maybe_interesting = 1;\n \tstrmap_put(&ctx.paths_to_lists, root_path, root_tree_list);\n \tpush_to_stack(&ctx, root_path);\n \n@@ -382,6 +448,14 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n+\t/* Walk trees to mark them as UNINTERESTING. */\n+\tedge_repo = info->revs->repo;\n+\tedge_tree_list = root_tree_list;\n+\tmark_edges_uninteresting(info->revs, show_edge,\n+\t\t\t\t info->prune_all_uninteresting);\n+\tedge_repo = NULL;\n+\tedge_tree_list = NULL;\n+\n \tinfo->revs->blob_objects = info->revs->tree_objects = 0;\n \n \ttrace2_region_enter(\"path-walk\", \"pending-walk\", info->revs->repo);\ndiff --git a/path-walk.h b/path-walk.h\nindex 3679fa7a859..414d6db23c2 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -40,6 +40,14 @@ struct path_walk_info {\n \tint trees;\n \tint blobs;\n \tint tags;\n+\n+\t/**\n+\t * When 'prune_all_uninteresting' is set and a path has all objects\n+\t * marked as UNINTERESTING, then the path-walk will not visit those\n+\t * objects. It will not call path_fn on those objects and will not\n+\t * walk the children of such trees.\n+\t */\n+\tint prune_all_uninteresting;\n };\n \n #define PATH_WALK_INFO_INIT { \\\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 56289859e69..7f2d409c5bc 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -50,10 +50,14 @@ static int emit_block(const char *path, struct oid_array *oids,\n \t\tprintf(\"%\"PRIuMAX\":%s:%s:EMPTY\\n\",\n \t\t tdata->batch_nr, typestr, path);\n \n-\tfor (size_t i = 0; i < oids->nr; i++)\n-\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s\\n\",\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tstruct object *o = lookup_unknown_object(the_repository,\n+\t\t\t\t\t\t\t &oids->oid[i]);\n+\t\tprintf(\"%\"PRIuMAX\":%s:%s:%s%s\\n\",\n \t\t tdata->batch_nr, typestr, path,\n-\t\t oid_to_hex(&oids->oid[i]));\n+\t\t oid_to_hex(&oids->oid[i]),\n+\t\t o->flags & UNINTERESTING ? \":UNINTERESTING\" : \"\");\n+\t}\n \n \ttdata->batch_nr++;\n \treturn 0;\n@@ -74,6 +78,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tag objects\")),\n \t\tOPT_BOOL(0, \"trees\", &info.trees,\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n+\t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n+\t\t\t N_(\"toggle pruning of uninteresting paths\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex 1f3d2e0cb76..a317cdf289e 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -211,11 +211,11 @@ test_expect_success 'topic, not base' '\n \t0:commit::$(git rev-parse topic)\n \t1:tree::$(git rev-parse topic^{tree})\n \t2:tree:right/:$(git rev-parse topic:right)\n-\t3:blob:right/d:$(git rev-parse topic:right/d)\n+\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse topic:left)\n-\t6:blob:left/b:$(git rev-parse topic:left/b)\n-\t7:blob:a:$(git rev-parse topic:a)\n+\t5:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t6:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t7:blob:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:1\n \ttags:0\n@@ -225,15 +225,38 @@ test_expect_success 'topic, not base' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'fourth, blob-tag2, not base' '\n+\ttest-tool path-walk -- fourth blob-tag2 --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tag:/tags:$(git rev-parse fourth)\n+\t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n+\t3:tree::$(git rev-parse topic^{tree})\n+\t4:tree:right/:$(git rev-parse topic:right)\n+\t5:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t8:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t9:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\tblobs:5\n+\tcommits:1\n+\ttags:1\n+\ttrees:3\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'topic, not base, only blobs' '\n \ttest-tool path-walk --no-trees --no-commits \\\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:blob:right/d:$(git rev-parse topic:right/d)\n+\t0:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \t1:blob:right/c:$(git rev-parse topic:right/c)\n-\t2:blob:left/b:$(git rev-parse topic:left/b)\n-\t3:blob:a:$(git rev-parse topic:a)\n+\t2:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t3:blob:a:$(git rev-parse topic:a):UNINTERESTING\n \tblobs:4\n \tcommits:0\n \ttags:0\n@@ -267,7 +290,7 @@ test_expect_success 'topic, not base, only trees' '\n \tcat >expect <<-EOF &&\n \t0:tree::$(git rev-parse topic^{tree})\n \t1:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:left/:$(git rev-parse topic:left)\n+\t2:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n \tcommits:0\n \tblobs:0\n \ttags:0\n@@ -282,17 +305,17 @@ test_expect_success 'topic, not base, boundary' '\n \n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n-\t0:commit::$(git rev-parse base~1)\n+\t0:commit::$(git rev-parse base~1):UNINTERESTING\n \t1:tree::$(git rev-parse topic^{tree})\n-\t1:tree::$(git rev-parse base~1^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n \t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right)\n-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/c:$(git rev-parse base~1:right/c)\n+\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t3:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n+\t4:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n \t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base~1:left)\n-\t6:blob:left/b:$(git rev-parse base~1:left/b)\n-\t7:blob:a:$(git rev-parse base~1:a)\n+\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t7:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n \tblobs:5\n \tcommits:2\n \ttags:0\n@@ -302,6 +325,27 @@ test_expect_success 'topic, not base, boundary' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, boundary with pruning' '\n+\ttest-tool path-walk --prune -- --boundary topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t0:commit::$(git rev-parse base~1):UNINTERESTING\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\tcommits:2\n+\ttags:0\n+\ttrees:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n@@ -309,15 +353,12 @@ test_expect_success 'trees are reported exactly once' '\n \t\tcd unique-trees &&\n \t\tmkdir initial &&\n \t\ttest_commit initial/file &&\n-\n \t\tgit switch -c move-to-top &&\n \t\tgit mv initial/file.t ./ &&\n \t\ttest_tick &&\n \t\tgit commit -m moved &&\n-\n \t\tgit update-ref refs/heads/other HEAD\n \t) &&\n-\n \ttest-tool -C unique-trees path-walk -- --all >out &&\n \ttree=$(git -C unique-trees rev-parse HEAD:) &&\n \tgrep \"$tree\" out >out-filtered &&\n-- \ngitgitgadget\n\n"},{"id":"509416","messageId":"ef543429ed9cb92e209900e7c7fc4f8e0698a306.1734711676.git.gitgitgadget@gmail.com","threadId":"62429","inReplyTo":"pull.1818.v4.git.1734711675.gitgitgadget@gmail.com","subject":"[PATCH v4 7/7] path-walk: reorder object visits","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T16:21:15Z","receivedAt":"2024-12-20T16:21:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe path-walk API currently uses a stack-based approach to recursing\nthrough the list of paths within the repository. This guarantees that\nafter a tree path is explored, all paths contained within that tree path\nwill be explored before continuing to explore siblings of that tree\npath.\n\nThe initial motivation of this depth-first approach was to minimize\nmemory pressure while exploring the repository. A breadth-first approach\nwould have too many \"active\" paths being stored in the paths_to_lists\nmap.\n\nWe can take this approach one step further by making sure that blob\npaths are visited before tree paths. This allows the API to free the\nmemory for these blob objects before continuing to perform the\ndepth-first search. This modifies the order in which we visit siblings,\nbut does not change the fact that we are performing depth-first search.\n\nTo achieve this goal, use a priority queue with a custom sorting method.\nThe sort needs to handle tags, blobs, and trees (commits are handled\nslightly differently). When objects share a type, we can sort by path\nname. This will keep children of the latest path to leave the stack be\npreferred over the rest of the paths in the stack, since they agree in\nprefix up to and including a directory separator. When the types are\ndifferent, we can prefer tags over other types and blobs over trees.\n\nThis causes significant adjustments to t6601-path-walk.sh to rearrange\nthe order of the visited paths.\n\nSigned-off-by: Derrick Stolee \n---\n path-walk.c | 60 ++++++++++++++++-----\n t/t6601-path-walk.sh | 124 +++++++++++++++++++++----------------------\n 2 files changed, 110 insertions(+), 74 deletions(-)\n\ndiff --git a/path-walk.c b/path-walk.c\nindex 4013569e9e4..136ec08fb0e 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -11,6 +11,7 @@\n #include \"list-objects.h\"\n #include \"object.h\"\n #include \"oid-array.h\"\n+#include \"prio-queue.h\"\n #include \"revision.h\"\n #include \"string-list.h\"\n #include \"strmap.h\"\n@@ -49,16 +50,50 @@ struct path_walk_context {\n \tstruct strmap paths_to_lists;\n \n \t/**\n-\t * Store the current list of paths in a stack, to\n-\t * facilitate depth-first-search without recursion.\n+\t * Store the current list of paths in a priority queue,\n+\t * using object type as a sorting mechanism, mostly to\n+\t * make sure blobs are popped off the stack first. No\n+\t * other sort is made, so within each object type it acts\n+\t * like a stack and performs a DFS within the trees.\n \t *\n \t * Use path_stack_pushed to indicate whether a path\n \t * was previously added to path_stack.\n \t */\n-\tstruct string_list path_stack;\n+\tstruct prio_queue path_stack;\n \tstruct strset path_stack_pushed;\n };\n \n+static int compare_by_type(const void *one, const void *two, void *cb_data)\n+{\n+\tstruct type_and_oid_list *list1, *list2;\n+\tconst char *str1 = one;\n+\tconst char *str2 = two;\n+\tstruct path_walk_context *ctx = cb_data;\n+\n+\tlist1 = strmap_get(&ctx->paths_to_lists, str1);\n+\tlist2 = strmap_get(&ctx->paths_to_lists, str2);\n+\n+\t/*\n+\t * If object types are equal, then use path comparison.\n+\t */\n+\tif (!list1 || !list2 || list1->type == list2->type)\n+\t\treturn strcmp(str1, str2);\n+\n+\t/* Prefer tags to be popped off first. */\n+\tif (list1->type == OBJ_TAG)\n+\t\treturn -1;\n+\tif (list2->type == OBJ_TAG)\n+\t\treturn 1;\n+\n+\t/* Prefer blobs to be popped off second. */\n+\tif (list1->type == OBJ_BLOB)\n+\t\treturn -1;\n+\tif (list2->type == OBJ_BLOB)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n static void push_to_stack(struct path_walk_context *ctx,\n \t\t\t const char *path)\n {\n@@ -66,7 +101,7 @@ static void push_to_stack(struct path_walk_context *ctx,\n \t\treturn;\n \n \tstrset_add(&ctx->path_stack_pushed, path);\n-\tstring_list_append(&ctx->path_stack, path);\n+\tprio_queue_put(&ctx->path_stack, xstrdup(path));\n }\n \n static int add_tree_entries(struct path_walk_context *ctx,\n@@ -378,8 +413,8 @@ static int setup_pending_objects(struct path_walk_info *info,\n \t\t\tconst char *tagged_blob_path = \"/tagged-blobs\";\n \t\t\ttagged_blobs->type = OBJ_BLOB;\n \t\t\ttagged_blobs->maybe_interesting = 1;\n-\t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tagged_blob_path, tagged_blobs);\n+\t\t\tpush_to_stack(ctx, tagged_blob_path);\n \t\t} else {\n \t\t\toid_array_clear(&tagged_blobs->oids);\n \t\t\tfree(tagged_blobs);\n@@ -390,8 +425,8 @@ static int setup_pending_objects(struct path_walk_info *info,\n \t\t\tconst char *tag_path = \"/tags\";\n \t\t\ttags->type = OBJ_TAG;\n \t\t\ttags->maybe_interesting = 1;\n-\t\t\tpush_to_stack(ctx, tag_path);\n \t\t\tstrmap_put(&ctx->paths_to_lists, tag_path, tags);\n+\t\t\tpush_to_stack(ctx, tag_path);\n \t\t} else {\n \t\t\toid_array_clear(&tags->oids);\n \t\t\tfree(tags);\n@@ -418,7 +453,10 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\t.repo = info->revs->repo,\n \t\t.revs = info->revs,\n \t\t.info = info,\n-\t\t.path_stack = STRING_LIST_INIT_DUP,\n+\t\t.path_stack = {\n+\t\t\t.compare = compare_by_type,\n+\t\t\t.cb_data = &ctx\n+\t\t},\n \t\t.path_stack_pushed = STRSET_INIT,\n \t\t.paths_to_lists = STRMAP_INIT\n \t};\n@@ -504,8 +542,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \ttrace2_region_enter(\"path-walk\", \"path-walk\", info->revs->repo);\n \twhile (!ret && ctx.path_stack.nr) {\n-\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n-\t\tctx.path_stack.nr--;\n+\t\tchar *path = prio_queue_get(&ctx.path_stack);\n \t\tpaths_nr++;\n \n \t\tret = walk_path(&ctx, path);\n@@ -522,8 +559,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \t\t\tpush_to_stack(&ctx, entry->key);\n \n \t\twhile (!ret && ctx.path_stack.nr) {\n-\t\t\tchar *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;\n-\t\t\tctx.path_stack.nr--;\n+\t\t\tchar *path = prio_queue_get(&ctx.path_stack);\n \t\t\tpaths_nr++;\n \n \t\t\tret = walk_path(&ctx, path);\n@@ -537,7 +573,7 @@ int walk_objects_by_path(struct path_walk_info *info)\n \n \tclear_paths_to_lists(&ctx.paths_to_lists);\n \tstrset_clear(&ctx.path_stack_pushed);\n-\tstring_list_clear(&ctx.path_stack, 0);\n+\tclear_prio_queue(&ctx.path_stack);\n \treturn ret;\n }\n \ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex a317cdf289e..5f04acb8a2f 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -84,20 +84,20 @@ test_expect_success 'all' '\n \t3:tree::$(git rev-parse refs/tags/tree-tag^{})\n \t3:tree::$(git rev-parse refs/tags/tree-tag2^{})\n \t4:blob:a:$(git rev-parse base~2:a)\n-\t5:tree:right/:$(git rev-parse topic:right)\n-\t5:tree:right/:$(git rev-parse base~1:right)\n-\t5:tree:right/:$(git rev-parse base~2:right)\n-\t6:blob:right/d:$(git rev-parse base~1:right/d)\n-\t7:blob:right/c:$(git rev-parse base~2:right/c)\n-\t7:blob:right/c:$(git rev-parse topic:right/c)\n-\t8:tree:left/:$(git rev-parse base:left)\n-\t8:tree:left/:$(git rev-parse base~2:left)\n-\t9:blob:left/b:$(git rev-parse base~2:left/b)\n-\t9:blob:left/b:$(git rev-parse base:left/b)\n-\t10:tree:a/:$(git rev-parse base:a)\n-\t11:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n-\t12:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n-\t13:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\t5:blob:file2:$(git rev-parse refs/tags/tree-tag2^{}:file2)\n+\t6:tree:a/:$(git rev-parse base:a)\n+\t7:tree:child/:$(git rev-parse refs/tags/tree-tag:child)\n+\t8:blob:child/file:$(git rev-parse refs/tags/tree-tag:child/file)\n+\t9:tree:left/:$(git rev-parse base:left)\n+\t9:tree:left/:$(git rev-parse base~2:left)\n+\t10:blob:left/b:$(git rev-parse base~2:left/b)\n+\t10:blob:left/b:$(git rev-parse base:left/b)\n+\t11:tree:right/:$(git rev-parse topic:right)\n+\t11:tree:right/:$(git rev-parse base~1:right)\n+\t11:tree:right/:$(git rev-parse base~2:right)\n+\t12:blob:right/c:$(git rev-parse base~2:right/c)\n+\t12:blob:right/c:$(git rev-parse topic:right/c)\n+\t13:blob:right/d:$(git rev-parse base~1:right/d)\n \tblobs:10\n \tcommits:4\n \ttags:7\n@@ -154,19 +154,19 @@ test_expect_success 'branches and indexed objects mix well' '\n \t1:tree::$(git rev-parse base^{tree})\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n-\t2:blob:a:$(git rev-parse base~2:a)\n-\t3:tree:right/:$(git rev-parse topic:right)\n-\t3:tree:right/:$(git rev-parse base~1:right)\n-\t3:tree:right/:$(git rev-parse base~2:right)\n-\t4:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/d:$(git rev-parse :right/d)\n-\t5:blob:right/c:$(git rev-parse base~2:right/c)\n-\t5:blob:right/c:$(git rev-parse topic:right/c)\n-\t6:tree:left/:$(git rev-parse base:left)\n-\t6:tree:left/:$(git rev-parse base~2:left)\n-\t7:blob:left/b:$(git rev-parse base:left/b)\n-\t7:blob:left/b:$(git rev-parse base~2:left/b)\n-\t8:tree:a/:$(git rev-parse refs/tags/third:a)\n+\t2:tree:a/:$(git rev-parse refs/tags/third:a)\n+\t3:tree:left/:$(git rev-parse base:left)\n+\t3:tree:left/:$(git rev-parse base~2:left)\n+\t4:blob:left/b:$(git rev-parse base:left/b)\n+\t4:blob:left/b:$(git rev-parse base~2:left/b)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right)\n+\t5:tree:right/:$(git rev-parse base~2:right)\n+\t6:blob:right/c:$(git rev-parse base~2:right/c)\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse base~1:right/d)\n+\t7:blob:right/d:$(git rev-parse :right/d)\n+\t8:blob:a:$(git rev-parse base~2:a)\n \tblobs:7\n \tcommits:4\n \ttags:0\n@@ -186,15 +186,15 @@ test_expect_success 'topic only' '\n \t1:tree::$(git rev-parse topic^{tree})\n \t1:tree::$(git rev-parse base~1^{tree})\n \t1:tree::$(git rev-parse base~2^{tree})\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right)\n-\t2:tree:right/:$(git rev-parse base~2:right)\n-\t3:blob:right/d:$(git rev-parse base~1:right/d)\n-\t4:blob:right/c:$(git rev-parse base~2:right/c)\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base~2:left)\n-\t6:blob:left/b:$(git rev-parse base~2:left/b)\n-\t7:blob:a:$(git rev-parse base~2:a)\n+\t2:blob:a:$(git rev-parse base~2:a)\n+\t3:tree:left/:$(git rev-parse base~2:left)\n+\t4:blob:left/b:$(git rev-parse base~2:left/b)\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right)\n+\t5:tree:right/:$(git rev-parse base~2:right)\n+\t6:blob:right/c:$(git rev-parse base~2:right/c)\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse base~1:right/d)\n \tblobs:5\n \tcommits:3\n \ttags:0\n@@ -210,12 +210,12 @@ test_expect_success 'topic, not base' '\n \tcat >expect <<-EOF &&\n \t0:commit::$(git rev-parse topic)\n \t1:tree::$(git rev-parse topic^{tree})\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n-\t6:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n-\t7:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t2:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t3:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t4:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \tblobs:4\n \tcommits:1\n \ttags:0\n@@ -233,12 +233,12 @@ test_expect_success 'fourth, blob-tag2, not base' '\n \t1:tag:/tags:$(git rev-parse fourth)\n \t2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})\n \t3:tree::$(git rev-parse topic^{tree})\n-\t4:tree:right/:$(git rev-parse topic:right)\n-\t5:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n-\t6:blob:right/c:$(git rev-parse topic:right/c)\n-\t7:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n-\t8:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n-\t9:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t4:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t7:tree:right/:$(git rev-parse topic:right)\n+\t8:blob:right/c:$(git rev-parse topic:right/c)\n+\t9:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:5\n \tcommits:1\n \ttags:1\n@@ -253,10 +253,10 @@ test_expect_success 'topic, not base, only blobs' '\n \t\t-- topic --not base >out &&\n \n \tcat >expect <<-EOF &&\n-\t0:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n-\t1:blob:right/c:$(git rev-parse topic:right/c)\n-\t2:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n-\t3:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t0:blob:a:$(git rev-parse topic:a):UNINTERESTING\n+\t1:blob:left/b:$(git rev-parse topic:left/b):UNINTERESTING\n+\t2:blob:right/c:$(git rev-parse topic:right/c)\n+\t3:blob:right/d:$(git rev-parse topic:right/d):UNINTERESTING\n \tblobs:4\n \tcommits:0\n \ttags:0\n@@ -289,8 +289,8 @@ test_expect_success 'topic, not base, only trees' '\n \n \tcat >expect <<-EOF &&\n \t0:tree::$(git rev-parse topic^{tree})\n-\t1:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t1:tree:left/:$(git rev-parse topic:left):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n \tcommits:0\n \tblobs:0\n \ttags:0\n@@ -308,14 +308,14 @@ test_expect_success 'topic, not base, boundary' '\n \t0:commit::$(git rev-parse base~1):UNINTERESTING\n \t1:tree::$(git rev-parse topic^{tree})\n \t1:tree::$(git rev-parse base~1^{tree}):UNINTERESTING\n-\t2:tree:right/:$(git rev-parse topic:right)\n-\t2:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n-\t3:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n-\t4:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n-\t4:blob:right/c:$(git rev-parse topic:right/c)\n-\t5:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n-\t6:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n-\t7:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t2:blob:a:$(git rev-parse base~1:a):UNINTERESTING\n+\t3:tree:left/:$(git rev-parse base~1:left):UNINTERESTING\n+\t4:blob:left/b:$(git rev-parse base~1:left/b):UNINTERESTING\n+\t5:tree:right/:$(git rev-parse topic:right)\n+\t5:tree:right/:$(git rev-parse base~1:right):UNINTERESTING\n+\t6:blob:right/c:$(git rev-parse base~1:right/c):UNINTERESTING\n+\t6:blob:right/c:$(git rev-parse topic:right/c)\n+\t7:blob:right/d:$(git rev-parse base~1:right/d):UNINTERESTING\n \tblobs:5\n \tcommits:2\n \ttags:0\n-- \ngitgitgadget\n"},{"id":"509630","messageId":"Z263J9O1ybuiN-df@pks.im","threadId":"62429","inReplyTo":"62f7aae477bc542dce0bb6f8b59438442a77d154.1734711676.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/7] path-walk: introduce an object walk by path","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-27T14:18:22Z","receivedAt":"2024-12-27T14:18:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 20, 2024 at 04:21:09PM +0000, Derrick Stolee via GitGitGadget wrote:\n[snip]\n> +static int add_tree_entries(struct path_walk_context *ctx,\n> +\t\t\t const char *base_path,\n> +\t\t\t struct object_id *oid)\n> +{\n> +\tstruct tree_desc desc;\n> +\tstruct name_entry entry;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tsize_t base_len;\n> +\tstruct tree *tree = lookup_tree(ctx->repo, oid);\n> +\n> +\tif (!tree) {\n> +\t\terror(_(\"failed to walk children of tree %s: not found\"),\n> +\t\t oid_to_hex(oid));\n> +\t\treturn -1;\n> +\t} else if (parse_tree_gently(tree, 1)) {\n> +\t\terror(\"bad tree object %s\", oid_to_hex(oid));\n> +\t\treturn -1;\n> +\t}\n\nYou can `return error(_(\"...\"));` directly as it already returns `-1`.\nNot sure whether this by itself warrants a reroll -- probably not. I'll\nleave it up to you.\n\nThe rest of the patch series looks as expected, mostly based on the\nrange diff.\n\nPatrick\n"},{"id":"509631","messageId":"Z263NFdtulSUYRLO@pks.im","threadId":"62429","inReplyTo":"eda7d3b7-8c3b-4486-b292-2507ca7f3bf9@gmail.com","subject":"Re: [PATCH v3 1/7] path-walk: introduce an object walk by path","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-27T14:18:28Z","receivedAt":"2024-12-27T14:18:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Dec 18, 2024 at 09:21:25AM -0500, Derrick Stolee wrote:\n> On 12/13/24 6:58 AM, Patrick Steinhardt wrote:\n> > On Fri, Dec 06, 2024 at 07:45:52PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n> > > +\t} else if (parse_tree_gently(tree, 1)) {\n> > > +\t\tdie(\"bad tree object %s\", oid_to_hex(oid));\n> > \n> > I wonder whether we maybe shouldn't die but instead return an error in\n> > the spirit of libification.\n> \n> This is in fact something that is being tested when 'git pack-objects' has\n> the --path-walk feature. See \"get an error for missing tree object\" in\n> t5317 as an example.\n> \n> It's not enough to fail, but we need to fail with this error message.\n> \n> Has there been enough progress in the libification effort to establish a\n> pattern for returning an error message like \"bad tree object %s\" from an\n> API like this to the caller?\n> \n> I will try using a \"error(); return -1;\" and consider that as the best\n> option for right now.\n\nYeah, I think this is best practice for now where we don't have a\nsuperior mechanism like structured errors.\n\nPatrick\n"}]}