{"thread":{"id":"58386","subject":"[PATCH 0/3] list-object-filter: introduce depth filter","startedAt":"2022-09-01T09:41:34Z","lastAt":"2022-09-11T10:59:41Z","messageCount":12,"participants":["ZheNing Hu via GitGitGadget","Derrick Stolee","Johannes Schindelin","ZheNing Hu"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"462430","messageId":"pull.1343.git.1662025272.gitgitgadget@gmail.com","threadId":"58386","inReplyTo":null,"subject":"[PATCH 0/3] list-object-filter: introduce depth filter","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-01T09:41:09Z","receivedAt":"2022-09-01T09:41:34Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch let partial clone have the similar capabilities of the shallow\nclone git clone --depth=<depth>.\n\nDisadvantages of git clone --depth=<depth> --filter=blob:none: we must call\ngit fetch --unshallow to lift the shallow clone restriction, it will\ndownload all history of current commit.\n\nDisadvantages of git clone --filter=blob:none with git sparse-checkout: The\ngit client needs to send a lot of missing objects' id to the server, this\ncan be very wasteful of network traffic.\n\nNow we can use git clone --filter=\"depth=<depth>\" to omit all commits whose\ndepth is >= <depth>. By this way, we can have the advantages of both shallow\nclone and partial clone: Limiting the depth of commits, get other objects on\ndemand.\n\nUnfinished business for now:\n\n 1. Git fetch has not yet learned the depth filter, if we can solve this\n    problem, we may can have a better \"batch fetch\" for some needed commits\n    (see [1]).\n 2. Sometimes we may want to partial clone to avoid automatic downloads\n    missing objects, e.g. when running git log, we might want to have\n    similar results of shallow clone (without commit graft).\n\n[1]:\nhttps://lore.kernel.org/git/16633d89-6ccd-859d-8533-9861ad831c45@github.com/\n\nZheNing Hu (3):\n  commit-graph: let commit graph respect commit graft\n  list-object-filter: pass traversal_context in filter_init_fn\n  list-object-filter: introduce depth filter\n\n Documentation/rev-list-options.txt  |   6 ++\n builtin/clone.c                     |  10 ++-\n commit-graph.c                      |  36 +++++++--\n list-objects-filter-options.c       |  30 +++++++\n list-objects-filter-options.h       |   6 ++\n list-objects-filter.c               |  78 ++++++++++++++++++-\n list-objects-filter.h               |   2 +\n list-objects.c                      |  10 +--\n list-objects.h                      |   8 ++\n shallow.c                           |  16 ++++\n shallow.h                           |   2 +\n t/t5616-partial-clone.sh            | 116 ++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh |  14 ++++\n upload-pack.c                       |  14 ----\n upload-pack.h                       |  14 ++++\n 15 files changed, 330 insertions(+), 32 deletions(-)\n\n\nbase-commit: d42b38dfb5edf1a7fddd9542d722f91038407819\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1343%2Fadlternative%2Fzh%2Ffilter_depth-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1343/adlternative/zh/filter_depth-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1343\n-- \ngitgitgadget\n"},{"id":"462431","messageId":"19fd72c34dcd1332df638d76b0b028e9d9da3d41.1662025272.git.gitgitgadget@gmail.com","threadId":"58386","inReplyTo":"pull.1343.git.1662025272.gitgitgadget@gmail.com","subject":"[PATCH 1/3] commit-graph: let commit graph respect commit graft","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-01T09:41:10Z","receivedAt":"2022-09-01T09:41:36Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu <adlternative@gmail.com>\n\nIn repo_parse_commit_internal(), if we want to use\ncommit graph, it will call parse_commit_in_graph() to\nparse commit's content from commit graph, otherwise\ncall repo_read_object_file() to parse commit's content\nfrom commit object.\n\nrepo_read_object_file() will respect commit graft,\nwhich can correctly amend commit's parents. But\nparse_commit_in_graph() not. Inconsistencies here may\nresult in incorrect processing of shallow clone.\n\nSo let parse_commit_in_graph() respect commit graft as\nrepo_read_object_file() does, which can solve this problem.\n\nSigned-off-by: ZheNing Hu <adlternative@gmail.com>\n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++------\n 1 file changed, 30 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f2a36032f84..89bb6f87079 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -820,6 +820,7 @@ static int fill_commit_in_graph(struct repository *r,\n \tstruct commit_list **pptr;\n \tconst unsigned char *commit_data;\n \tuint32_t lex_index;\n+\tstruct commit_graft *graft;\n \n \twhile (pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n@@ -833,31 +834,54 @@ static int fill_commit_in_graph(struct repository *r,\n \n \tset_commit_tree(item, NULL);\n \n+\tgraft = lookup_commit_graft(r, &item->object.oid);\n+\tif (graft)\n+\t\tr->parsed_objects->substituted_parent = 1;\n+\n \tpptr = &item->parents;\n \n \tedge_value = get_be32(commit_data + g->hash_len);\n \tif (edge_value == GRAPH_PARENT_NONE)\n \t\treturn 1;\n-\tpptr = insert_parent_or_die(r, g, edge_value, pptr);\n+\tif (!(graft && (graft->nr_parent < 0 || grafts_replace_parents)))\n+\t\tpptr = insert_parent_or_die(r, g, edge_value, pptr);\n \n \tedge_value = get_be32(commit_data + g->hash_len + 4);\n \tif (edge_value == GRAPH_PARENT_NONE)\n \t\treturn 1;\n \tif (!(edge_value & GRAPH_EXTRA_EDGES_NEEDED)) {\n-\t\tpptr = insert_parent_or_die(r, g, edge_value, pptr);\n-\t\treturn 1;\n+\t\tif (!(graft && (graft->nr_parent < 0 || grafts_replace_parents))) {\n+\t\t\tpptr = insert_parent_or_die(r, g, edge_value, pptr);\n+\t\t\treturn 1;\n+\t\t}\n \t}\n \n \tparent_data_ptr = (uint32_t*)(g->chunk_extra_edges +\n \t\t\t  4 * (uint64_t)(edge_value & GRAPH_EDGE_LAST_MASK));\n \tdo {\n \t\tedge_value = get_be32(parent_data_ptr);\n-\t\tpptr = insert_parent_or_die(r, g,\n-\t\t\t\t\t    edge_value & GRAPH_EDGE_LAST_MASK,\n-\t\t\t\t\t    pptr);\n+\t\tif (!(graft && (graft->nr_parent < 0 || grafts_replace_parents))) {\n+\t\t\tpptr = insert_parent_or_die(r, g,\n+\t\t\t\t\t\t    edge_value & GRAPH_EDGE_LAST_MASK,\n+\t\t\t\t\t\t    pptr);\n+\t\t}\n \t\tparent_data_ptr++;\n \t} while (!(edge_value & GRAPH_LAST_EDGE));\n \n+\tif (graft) {\n+\t\tint i;\n+\t\tstruct commit *new_parent;\n+\t\tfor (i = 0; i < graft->nr_parent; i++) {\n+\t\t\tnew_parent = lookup_commit(r,\n+\t\t\t\t\t\t   &graft->parent[i]);\n+\t\t\tif (!new_parent)\n+\t\t\t\tdie(_(\"bad graft parent %s in commit %s\"),\n+\t\t\t\t       oid_to_hex(&graft->parent[i]),\n+\t\t\t\t       oid_to_hex(&item->object.oid));\n+\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n+\t\t}\n+\t}\n+\n \treturn 1;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"462432","messageId":"0df61091c194ed46fda4d70272fcbcfdbedc8770.1662025272.git.gitgitgadget@gmail.com","threadId":"58386","inReplyTo":"pull.1343.git.1662025272.gitgitgadget@gmail.com","subject":"[PATCH 2/3] list-object-filter: pass traversal_context in filter_init_fn","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-01T09:41:11Z","receivedAt":"2022-09-01T09:41:40Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu <adlternative@gmail.com>\n\nPass traversal_context to all filter init functions, so that\nwe can read or modify the rev_info of traversal_context in the\nfilter initialization function.\n\nSigned-off-by: ZheNing Hu <adlternative@gmail.com>\n---\n list-objects-filter.c | 12 ++++++++++--\n list-objects-filter.h |  2 ++\n list-objects.c        | 10 +---------\n list-objects.h        |  8 ++++++++\n 4 files changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 1c1ee3d1bb1..76e8659ea73 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -112,6 +112,7 @@ static enum list_objects_filter_result filter_blobs_none(\n }\n \n static void filter_blobs_none__init(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n@@ -249,6 +250,7 @@ static void filter_trees_free(void *filter_data) {\n }\n \n static void filter_trees_depth__init(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n@@ -336,6 +338,7 @@ include_it:\n }\n \n static void filter_blobs_limit__init(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n@@ -519,6 +522,7 @@ static void filter_sparse_free(void *filter_data)\n }\n \n static void filter_sparse_oid__init(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n@@ -609,6 +613,7 @@ static enum list_objects_filter_result filter_object_type(\n }\n \n static void filter_object_type__init(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n@@ -734,6 +739,7 @@ static void filter_combine__finalize_omits(\n }\n \n static void filter_combine__init(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter* filter)\n {\n@@ -744,7 +750,7 @@ static void filter_combine__init(\n \tCALLOC_ARRAY(d->sub, d->nr);\n \tfor (sub = 0; sub < d->nr; sub++)\n \t\td->sub[sub].filter = list_objects_filter__init(\n-\t\t\tfilter->omits ? &d->sub[sub].omits : NULL,\n+\t\t\tctx, filter->omits ? &d->sub[sub].omits : NULL,\n \t\t\t&filter_options->sub[sub]);\n \n \tfilter->filter_data = d;\n@@ -754,6 +760,7 @@ static void filter_combine__init(\n }\n \n typedef void (*filter_init_fn)(\n+\tstruct traversal_context *ctx,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n@@ -771,6 +778,7 @@ static filter_init_fn s_filters[] = {\n };\n \n struct filter *list_objects_filter__init(\n+\tstruct traversal_context *ctx,\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options)\n {\n@@ -792,7 +800,7 @@ struct filter *list_objects_filter__init(\n \n \tCALLOC_ARRAY(filter, 1);\n \tfilter->omits = omitted;\n-\tinit_fn(filter_options, filter);\n+\tinit_fn(ctx, filter_options, filter);\n \treturn filter;\n }\n \ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 9e98814111c..0a3cb500976 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -5,6 +5,7 @@ struct list_objects_filter_options;\n struct object;\n struct oidset;\n struct repository;\n+struct traversal_context;\n \n /*\n  * During list-object traversal we allow certain objects to be\n@@ -72,6 +73,7 @@ struct filter;\n  * filter *`.\n  */\n struct filter *list_objects_filter__init(\n+\tstruct traversal_context *ctx,\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options);\n \ndiff --git a/list-objects.c b/list-objects.c\nindex 250d9de41cb..698e4dbe8ff 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -13,14 +13,6 @@\n #include \"object-store.h\"\n #include \"trace.h\"\n \n-struct traversal_context {\n-\tstruct rev_info *revs;\n-\tshow_object_fn show_object;\n-\tshow_commit_fn show_commit;\n-\tvoid *show_data;\n-\tstruct filter *filter;\n-};\n-\n static void show_commit(struct traversal_context *ctx,\n \t\t\tstruct commit *commit)\n {\n@@ -448,7 +440,7 @@ void traverse_commit_list_filtered(\n \t};\n \n \tif (revs->filter.choice)\n-\t\tctx.filter = list_objects_filter__init(omitted, &revs->filter);\n+\t\tctx.filter = list_objects_filter__init(&ctx, omitted, &revs->filter);\n \n \tdo_traverse(&ctx);\n \ndiff --git a/list-objects.h b/list-objects.h\nindex 9eaf4de8449..44c598e9ce8 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -16,6 +16,14 @@ void mark_edges_uninteresting(struct rev_info *revs,\n struct oidset;\n struct list_objects_filter_options;\n \n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tstruct filter *filter;\n+};\n+\n void traverse_commit_list_filtered(\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n-- \ngitgitgadget\n\n"},{"id":"462433","messageId":"2f7d1490c43be2a1f363484f1c7031db56b6b673.1662025272.git.gitgitgadget@gmail.com","threadId":"58386","inReplyTo":"pull.1343.git.1662025272.gitgitgadget@gmail.com","subject":"[PATCH 3/3] list-object-filter: introduce depth filter","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-01T09:41:12Z","receivedAt":"2022-09-01T09:41:41Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu <adlternative@gmail.com>\n\n'git clone --depth=<depth>' have a obvious disadvantages:\nWe can't do some git commands that require looking deeper\ncommits. We might use 'git fetch -unshallow' to lift this\nrestriction, but that's not a good idea either: it downloads\ntoo much objects.\n\nRethink this question: why not integrate the functionality\nof shallow clone into parital clone? Partial clone has a\nvery clear advantage: it downloads objects only if the user\nneeds them.\n\nTherefore, add a filter 'depth=<depth>' which can omits all\ncommits whose depth is >= <depth> (<depth> > 0), it just look\nlike  '--depth=<depth>' in git clone, but the git client\ndoesn't treat it as a shallow clone.\n\n'--filter=depth:<depth>' cannot be used with '--depth',\n'--shallow-since', '--shallow-exclude', '--shallow-submodules'.\n\nSigned-off-by: ZheNing Hu <adlternative@gmail.com>\n---\n Documentation/rev-list-options.txt  |   6 ++\n builtin/clone.c                     |  10 ++-\n list-objects-filter-options.c       |  30 +++++++\n list-objects-filter-options.h       |   6 ++\n list-objects-filter.c               |  66 ++++++++++++++++\n shallow.c                           |  16 ++++\n shallow.h                           |   2 +\n t/t5616-partial-clone.sh            | 116 ++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh |  14 ++++\n upload-pack.c                       |  14 ----\n upload-pack.h                       |  14 ++++\n 11 files changed, 279 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 1837509566a..4e2905b9e1e 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -954,6 +954,12 @@ Note that the form '--filter=sparse:path=<path>' that wants to read\n from an arbitrary path on the filesystem has been dropped for security\n reasons.\n +\n+The form '--filter=depth:<depth>' omits all commits whose depth is\n+>= <depth>, it just look like '--depth=<depth>' in git clone, but it\n+will not be treated as shallow-clone, so if you want to see some deeper\n+commits, you can freely do some git commands e.g. git diff to refetch\n+missing git objects without 'git fetch --unshallow'.\n++\n Multiple '--filter=' flags can be specified to combine filters. Only\n objects which are accepted by every filter are included.\n +\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex c4ff4643ecd..0b168ec18ef 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -916,7 +916,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tusage_msg_opt(_(\"You must specify a repository to clone.\"),\n \t\t\tbuiltin_clone_usage, builtin_clone_options);\n \n-\tif (option_depth || option_since || option_not.nr)\n+\tif (option_depth || option_since || option_not.nr ||\n+\t    list_objects_filter_choice_exists(&filter_options, LOFC_DEPTH))\n \t\tdeepen = 1;\n \tif (option_single_branch == -1)\n \t\toption_single_branch = deepen ? 1 : 0;\n@@ -1113,6 +1114,13 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"the option '%s' requires '%s'\"),\n \t\t    \"--also-filter-submodules\", \"--recurse-submodules\");\n \n+\tif ((option_depth || option_since || option_not.nr ||\n+\t     option_shallow_submodules) &&\n+\t     list_objects_filter_choice_exists(&filter_options, LOFC_DEPTH))\n+\t\tdie(_(\"--filter='depth:<depth>' cannot be used with \"\n+\t\t      \"--depth, --shallow-since, --shallow-exclude, \"\n+\t\t      \"--shallow-submodules\"));\n+\n \t/*\n \t * apply the remote name provided by --origin only after this second\n \t * call to git_config, to ensure it overrides all config-based values.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 4b25287886d..687010b84d4 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -31,6 +31,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \t\treturn \"sparse:oid\";\n \tcase LOFC_OBJECT_TYPE:\n \t\treturn \"object:type\";\n+\tcase LOFC_DEPTH:\n+\t\treturn \"depth\";\n \tcase LOFC_COMBINE:\n \t\treturn \"combine\";\n \tcase LOFC__COUNT:\n@@ -40,6 +42,23 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \tBUG(\"list_object_filter_config_name: invalid argument '%d'\", c);\n }\n \n+int list_objects_filter_choice_exists(\n+\tstruct list_objects_filter_options *filter_options,\n+\tenum list_objects_filter_choice choice) {\n+\tint i;\n+\n+\tif (!filter_options)\n+\t\treturn 0;\n+\n+\tif (filter_options->choice == choice)\n+\t\treturn 1;\n+\tif (filter_options->sub_nr)\n+\t\tfor (i = 0; i < filter_options->sub_nr; i++)\n+\t\t\tif (filter_options->sub[i].choice == choice)\n+\t\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n@@ -97,6 +116,17 @@ int gently_parse_list_objects_filter(\n \n \t\treturn 0;\n \n+\t} else if (skip_prefix(arg, \"depth:\", &v0)) {\n+\t\tif (!git_parse_ulong(v0, &filter_options->depth)) {\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'depth:<depth>'\"));\n+\t\t\treturn 1;\n+\t\t} else if (atoi(v0) <= 0) {\n+\t\t\tstrbuf_addf(errbuf, _(\"depth %s is not a positive number\"), v0);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfilter_options->choice = LOFC_DEPTH;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n \t\treturn parse_combine_filter(filter_options, v0, errbuf);\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex ffc02d77e76..a4fd40567d2 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -15,6 +15,7 @@ enum list_objects_filter_choice {\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n \tLOFC_OBJECT_TYPE,\n+\tLOFC_DEPTH,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n@@ -54,6 +55,7 @@ struct list_objects_filter_options {\n \t */\n \n \tchar *sparse_oid_name;\n+\tunsigned long depth;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n \tenum object_type object_type;\n@@ -69,6 +71,10 @@ struct list_objects_filter_options {\n \t */\n };\n \n+int list_objects_filter_choice_exists(\n+\tstruct list_objects_filter_options *filter_options,\n+\tenum list_objects_filter_choice choice);\n+\n /*\n  * Parse value of the argument to the \"filter\" keyword.\n  * On the command line this looks like:\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 76e8659ea73..5b4d8348b54 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -13,6 +13,8 @@\n #include \"oidmap.h\"\n #include \"oidset.h\"\n #include \"object-store.h\"\n+#include \"shallow.h\"\n+#include \"upload-pack.h\"\n \n /* Remember to update object flag allocation in object.h */\n /*\n@@ -69,6 +71,69 @@ struct filter {\n \tstruct oidset *omits;\n };\n \n+static enum list_objects_filter_result filter_noop(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data_)\n+{\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\t/* always include all tree objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFS_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\t/* always include all blob objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t}\n+}\n+\n+static void noop_free(void *filter_data) {\n+\t/* noop */\n+}\n+\n+static void filter_depth__init(\n+\tstruct traversal_context *ctx,\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter *filter)\n+{\n+\tstruct commit_list *result = get_shallow_commits_by_commits(ctx->revs->commits,\n+\t\t\t\t\tfilter_options->depth,\n+\t\t\t\t\tSHALLOW, NOT_SHALLOW);\n+\n+\twhile (result) {\n+\t\tregister_shallow(the_repository, &result->item->object.oid);\n+\t\tresult = result->next;\n+\t}\n+\tfree_commit_list(result);\n+\n+\tfilter->filter_object_fn = filter_noop;\n+\tfilter->free_fn = noop_free;\n+}\n+\n+\n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n@@ -774,6 +839,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n \tfilter_object_type__init,\n+\tfilter_depth__init,\n \tfilter_combine__init,\n };\n \ndiff --git a/shallow.c b/shallow.c\nindex 8cb768ee5f8..9d1d7668ad8 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -122,6 +122,22 @@ static void free_depth_in_slab(int **ptr)\n {\n \tFREE_AND_NULL(*ptr);\n }\n+\n+struct commit_list *get_shallow_commits_by_commits(struct commit_list *commits, int depth,\n+\t\tint shallow_flag, int not_shallow_flag) {\n+\tstruct object_array array = OBJECT_ARRAY_INIT;\n+\tstruct commit_list *result = NULL;\n+\tstruct commit_list *commit = commits;\n+\n+\twhile (commit) {\n+\t\tadd_object_array(&commit->item->object, NULL, &array);\n+\t\tcommit = commit->next;\n+\t}\n+\tresult = get_shallow_commits(&array, depth, shallow_flag, not_shallow_flag);\n+\tobject_array_clear(&array);\n+\treturn result;\n+}\n+\n struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \t\tint shallow_flag, int not_shallow_flag)\n {\ndiff --git a/shallow.h b/shallow.h\nindex aba6ff58294..ed425b72796 100644\n--- a/shallow.h\n+++ b/shallow.h\n@@ -34,6 +34,8 @@ void rollback_shallow_file(struct repository *r, struct shallow_lock *lk);\n \n struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\t\t\t\tint depth, int shallow_flag, int not_shallow_flag);\n+struct commit_list *get_shallow_commits_by_commits(\n+\t\tstruct commit_list *commits, int depth, int shallow_flag, int not_shallow_flag);\n struct commit_list *get_shallow_commits_by_rev_list(\n \t\tint ac, const char **av, int shallow_flag, int not_shallow_flag);\n int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 9aeacc2f6a5..c328f5d76bc 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -458,6 +458,122 @@ test_expect_success 'partial clone with unresolvable sparse filter fails cleanly\n \ttest_i18ngrep \"unable to parse sparse filter data in\" err\n '\n \n+test_expect_success 'setup src repo for depth filter' '\n+\tgit init depth-src &&\n+\tgit -C depth-src config --local uploadpack.allowfilter 1 &&\n+\tgit -C depth-src config --local uploadpack.allowanysha1inwant 1 &&\n+\ttest_commit -C depth-src one &&\n+\ttest_commit -C depth-src two &&\n+\ttest_commit -C depth-src three &&\n+\tgit -C depth-src rm -rf two.t &&\n+\tgit -C depth-src commit -m four\n+'\n+\n+test_expect_success 'partial clone with depth=1 filter succeeds' '\n+\trm -rf dst.git &&\n+\tgit clone --no-local --bare \\\n+\t\t  --filter=depth:1 \\\n+\t\t  depth-src dst.git &&\n+\t(\n+\t\tcd dst.git &&\n+\t\tgit cat-file --batch-check --batch-all-objects >object &&\n+\t\tgrep blob object >blob_count &&\n+\t\ttest_line_count = 2 blob_count &&\n+\t\tgrep tree object >tree_count &&\n+\t\ttest_line_count = 1 tree_count &&\n+\t\tgrep commit object >commit_count &&\n+\t\ttest_line_count = 1 commit_count\n+\t)\n+'\n+\n+test_expect_success 'partial clone with depth=2 filter succeeds' '\n+\trm -rf dst.git &&\n+\tgit clone --no-local --bare \\\n+\t\t  --filter=depth:2 \\\n+\t\t  depth-src dst.git &&\n+\t(\n+\t\tcd dst.git &&\n+\t\tgit cat-file --batch-check --batch-all-objects >object &&\n+\t\tgrep blob object >blob_count &&\n+\t\ttest_line_count = 3 blob_count &&\n+\t\tgrep tree object >tree_count &&\n+\t\ttest_line_count = 2 tree_count &&\n+\t\tgrep commit object >commit_count &&\n+\t\ttest_line_count = 2 commit_count\n+\t)\n+'\n+\n+test_expect_success 'partial clone depth filter combine with blob:none filter succeeds' '\n+\trm -rf dst.git &&\n+\tgit clone --no-local --bare \\\n+\t\t  --filter=\"combine:depth:1+blob:none\" \\\n+\t\t  depth-src dst.git &&\n+\t(\n+\t\tcd dst.git &&\n+\t\tgit cat-file --batch-check --batch-all-objects >object &&\n+\t\t! grep blob object &&\n+\t\tgrep tree object >tree_count &&\n+\t\ttest_line_count = 1 tree_count &&\n+\t\tgrep commit object >commit_count &&\n+\t\ttest_line_count = 1 commit_count\n+\t)\n+'\n+\n+test_expect_success 'refetch other commits after partial clone with depth filter' '\n+\trm -rf dst.git &&\n+\tgit clone --no-local --bare \\\n+\t\t  --filter=depth:1 \\\n+\t\t  depth-src dst.git &&\n+\t(\n+\t\tcd dst.git &&\n+\t\tgit cat-file --batch-check --batch-all-objects >object &&\n+\t\tgrep blob object >blob_count &&\n+\t\ttest_line_count = 2 blob_count &&\n+\t\tgrep tree object >tree_count &&\n+\t\ttest_line_count = 1 tree_count &&\n+\t\tgrep commit object >commit_count &&\n+\t\ttest_line_count = 1 commit_count &&\n+\t\t# git log will trigger refetch commits\n+\t\tgit log &&\n+\t\tgit cat-file --batch-check --batch-all-objects >object &&\n+\t\tgrep blob object >blob_count &&\n+\t\ttest_line_count = 2 blob_count &&\n+\t\tgrep tree object >tree_count &&\n+\t\ttest_line_count = 4 tree_count &&\n+\t\tgrep commit object >commit_count &&\n+\t\ttest_line_count = 4 commit_count &&\n+\t\t# git diff will trigger refetch blobs\n+\t\tgit diff HEAD^ HEAD &&\n+\t\tgit cat-file --batch-check --batch-all-objects >object &&\n+\t\tgrep blob object >blob_count &&\n+\t\ttest_line_count = 3 blob_count &&\n+\t\tgrep tree object >tree_count &&\n+\t\ttest_line_count = 4 tree_count &&\n+\t\tgrep commit object >commit_count &&\n+\t\ttest_line_count = 4 commit_count\n+\t)\n+'\n+\n+test_expect_success 'partial clone with depth filter with shallow clone failed' \"\n+\trm -rf dst.git &&\n+\ttest_must_fail git clone --no-local --bare \\\n+\t\t  --filter=depth:1 --depth=1 \\\n+\t\t  depth-src dst.git 2>err &&\n+\ttest_i18ngrep \\\"fatal: --filter='depth:<depth>' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\\\" err &&\n+\ttest_must_fail git clone --no-local --bare \\\n+\t\t  --filter=depth:1  --shallow-since '300000000 +0700' \\\n+\t\t  depth-src dst.git 2>err &&\n+\ttest_i18ngrep \\\"fatal: --filter='depth:<depth>' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\\\" err &&\n+\ttest_must_fail git clone --no-local --bare \\\n+\t\t  --filter=depth:1 --shallow-exclude one.t \\\n+\t\t  depth-src dst.git 2>err &&\n+\ttest_i18ngrep \\\"fatal: --filter='depth:<depth>' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\\\" err &&\n+\ttest_must_fail git clone --no-local --bare \\\n+\t\t  --filter=depth:1 --shallow-submodules \\\n+\t\t  depth-src dst.git 2>err &&\n+\ttest_i18ngrep \\\"fatal: --filter='depth:<depth>' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\\\" err\n+\"\n+\n setup_triangle () {\n \trm -rf big-blob.txt server client promisor-remote &&\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 8d9d6604f05..a562bdffc80 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -417,6 +417,20 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \ttest_line_count = 10 actual\n '\n \n+test_expect_success 'verify depth:1 includes tip commit expected' '\n+\tgit -C r3 rev-list --objects --filter=depth:1 --no-object-names HEAD >objects &&\n+\tcat objects | git -C r3 cat-file --batch-check=\"%(objecttype)\" >types &&\n+\tgrep commit types >actual &&\n+\ttest_line_count = 1 actual\n+'\n+\n+test_expect_success 'verify depth:2 includes two commits expected' '\n+\tgit -C r3 rev-list --objects --filter=depth:2 --no-object-names HEAD >objects &&\n+\tcat objects | git -C r3 cat-file --batch-check=\"%(objecttype)\" >types &&\n+\tgrep commit types >actual &&\n+\ttest_line_count = 2 actual\n+'\n+\n test_expect_success 'combine:... for a simple combination' '\n \tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n \t\t>actual &&\ndiff --git a/upload-pack.c b/upload-pack.c\nindex b217a1f469e..d4ffebfa6ab 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -28,20 +28,6 @@\n #include \"commit-reach.h\"\n #include \"shallow.h\"\n \n-/* Remember to update object flag allocation in object.h */\n-#define THEY_HAVE\t(1u << 11)\n-#define OUR_REF\t\t(1u << 12)\n-#define WANTED\t\t(1u << 13)\n-#define COMMON_KNOWN\t(1u << 14)\n-\n-#define SHALLOW\t\t(1u << 16)\n-#define NOT_SHALLOW\t(1u << 17)\n-#define CLIENT_SHALLOW\t(1u << 18)\n-#define HIDDEN_REF\t(1u << 19)\n-\n-#define ALL_FLAGS (THEY_HAVE | OUR_REF | WANTED | COMMON_KNOWN | SHALLOW | \\\n-\t\tNOT_SHALLOW | CLIENT_SHALLOW | HIDDEN_REF)\n-\n /* Enum for allowed unadvertised object request (UOR) */\n enum allow_uor {\n \t/* Allow specifying sha1 if it is a ref tip. */\ndiff --git a/upload-pack.h b/upload-pack.h\nindex d6ee25ea98e..36add62f6bc 100644\n--- a/upload-pack.h\n+++ b/upload-pack.h\n@@ -1,6 +1,20 @@\n #ifndef UPLOAD_PACK_H\n #define UPLOAD_PACK_H\n \n+/* Remember to update object flag allocation in object.h */\n+#define THEY_HAVE\t(1u << 11)\n+#define OUR_REF\t\t(1u << 12)\n+#define WANTED\t\t(1u << 13)\n+#define COMMON_KNOWN\t(1u << 14)\n+\n+#define SHALLOW\t\t(1u << 16)\n+#define NOT_SHALLOW\t(1u << 17)\n+#define CLIENT_SHALLOW\t(1u << 18)\n+#define HIDDEN_REF\t(1u << 19)\n+\n+#define ALL_FLAGS (THEY_HAVE | OUR_REF | WANTED | COMMON_KNOWN | SHALLOW | \\\n+\t\tNOT_SHALLOW | CLIENT_SHALLOW | HIDDEN_REF)\n+\n void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\t const int timeout);\n \n-- \ngitgitgadget\n"},{"id":"462483","messageId":"8b9e8c2d-7a64-2d66-83a8-2a7daff9a81c@github.com","threadId":"58386","inReplyTo":"19fd72c34dcd1332df638d76b0b028e9d9da3d41.1662025272.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/3] commit-graph: let commit graph respect commit graft","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-01T19:18:21Z","receivedAt":"2022-09-01T19:18:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/1/2022 5:41 AM, ZheNing Hu via GitGitGadget wrote:\n> From: ZheNing Hu <adlternative@gmail.com>\n> \n> In repo_parse_commit_internal(), if we want to use\n> commit graph, it will call parse_commit_in_graph() to\n> parse commit's content from commit graph, otherwise\n> call repo_read_object_file() to parse commit's content\n> from commit object.\n> \n> repo_read_object_file() will respect commit graft,\n> which can correctly amend commit's parents. But\n> parse_commit_in_graph() not. Inconsistencies here may\n> result in incorrect processing of shallow clone.\n> \n> So let parse_commit_in_graph() respect commit graft as\n> repo_read_object_file() does, which can solve this problem.\n\nIf grafts or replace-objects exist, then the commit-graph\nis disabled and this code will never be called. I would\nexpect a test case demonstrating the change in behavior\nhere, but that is impossible.\n\nThe commit-graph parsing should not be bogged down with\nthis logic.\n\nThanks,\n-Stolee\n\n"},{"id":"462484","messageId":"a14028be-2fd2-258d-94f5-c010669de8a6@github.com","threadId":"58386","inReplyTo":"pull.1343.git.1662025272.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/3] list-object-filter: introduce depth filter","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-01T19:24:18Z","receivedAt":"2022-09-01T19:24:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/1/2022 5:41 AM, ZheNing Hu via GitGitGadget wrote:\n> This patch let partial clone have the similar capabilities of the shallow\n> clone git clone --depth=<depth>.\n...\n> Now we can use git clone --filter=\"depth=<depth>\" to omit all commits whose\n> depth is >= <depth>. By this way, we can have the advantages of both shallow\n> clone and partial clone: Limiting the depth of commits, get other objects on\n> demand.\n\nI have several concerns about this proposal.\n\nThe first is that \"depth=X\" doesn't mean anything after the first\nclone. What will happen when we fetch the remaining objects?\n\nPartial clone is designed to download a subset of objects, but make\nthe remaining reachable objects downloadable on demand. By dropping\nreachable commits, the normal partial clone mechanism would result\nin a 'git rev-list' call asking for a missing commit. Would this\ninherit the \"depth=X\" but result in a huge amount of over-downloading\nthe trees and blobs in that commit range? Would it result in downloading\ncommits one-by-one, and then their root trees (and all reachable objects\nfrom those root trees)?\n\nFinally, computing the set of objects to send is just as expensive as\nif we had a shallow clone (we can't use bitmaps). However, we get the\nadditional problem where fetches do not have a shallow boundary, so\nthe server will send deltas based on objects that are not necessarily\npresent locally, triggering extra requests to resolve those deltas.\n\nThis fallout remains undocumented and unexplored in this series, but I\ndoubt the investigation would result in positive outcomes.\n\n> Disadvantages of git clone --depth=<depth> --filter=blob:none: we must call\n> git fetch --unshallow to lift the shallow clone restriction, it will\n> download all history of current commit.\n\nHow does your proposal fix this? Instead of unshallowing, users will\nstumble across these objects and trigger huge downloads by accident.\n \n> Disadvantages of git clone --filter=blob:none with git sparse-checkout: The\n> git client needs to send a lot of missing objects' id to the server, this\n> can be very wasteful of network traffic.\n\nAsking for a list of blobs (especially limited to a sparse-checkout) is\nmuch more efficient than what will happen when a user tries to do almost\nanything in a repository formed the way you did here.\n\nThinking about this idea, I don't think it is viable. I would need to\nsee a lot of work done to test these scenarios closely to believe that\nthis type of partial clone is a desirable working state.\n\nThanks,\n-Stolee\n"},{"id":"462527","messageId":"o48053s6-5540-1234-5roq-92q6981r2306@tzk.qr","threadId":"58386","inReplyTo":"a14028be-2fd2-258d-94f5-c010669de8a6@github.com","subject":"Re: [PATCH 0/3] list-object-filter: introduce depth filter","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-09-02T13:48:20Z","receivedAt":"2022-09-02T14:23:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi ZheNing,\n\nfirst of all: thank you for working on this. In the past, I thought that\nthis feature would be likely something we would want to have in Git.\n\nBut Stolee's concerns are valid, and made me think about it more. See\nbelow for a more detailed analysis.\n\nOn Thu, 1 Sep 2022, Derrick Stolee wrote:\n\n> On 9/1/2022 5:41 AM, ZheNing Hu via GitGitGadget wrote:\n>\n> > [...]\n> >\n> > Disadvantages of git clone --filter=blob:none with git\n> > sparse-checkout: The git client needs to send a lot of missing\n> > objects' id to the server, this can be very wasteful of network\n> > traffic.\n>\n> Asking for a list of blobs (especially limited to a sparse-checkout) is\n> much more efficient than what will happen when a user tries to do almost\n> anything in a repository formed the way you did here.\n\nI agree. When you have all the commit and tree objects on the local side,\nyou can enumerate all the blob objects you need in one fell swoop, then\nfetch them in a single network round trip.\n\nWhen you lack tree objects, or worse, commit objects, this is not true.\nYou may very well need to fetch _quite_ a bunch of objects, then inspect\nthem to find out that you need to fetch more tree/commit objects, and then\na couple more round trips, before you can enumerate all of the objects you\nneed.\n\nConcrete example: let's assume that you clone git.git with a \"partial\ndepth\" of 50. That is, while cloning, all of the tip commits' graphs will\nbe traversed up until the commits that are removed by 49 edges in the\ncommit graph. For example, v0.99~49 will be present locally after cloning,\nbut not v0.99~50.\n\nNow, the first-parent depth of v0.99 is 955 (verify with `git rev-list\n--count --first-parent v0.99`). None of the commits reachable from v0.99\nother than the tip itself seem to be closer to any other tag, so all\ncommits reachable from v0.99~49 will be missing locally. And since reverts\nare rare, we must assume that the vast majority of the associated root\ntree objects are missing, too.\n\nDigging through history, a contributor might need to investigate where,\nsay, `t/t4100/t-apply-7.expect` was introduced (it was in v0.99~206)\nbecause they found something looking like a bug and they need to read the\ncommit message to see whether it was intentional. They know that this file\nwas already present in v0.99. Naturally, the command-line to investigate\nthat is:\n\n\tgit log --diff-filter=A v0.99 -- t/t4100/t-apply-7.expect\n\nSo what does Git do in that operation? It traverses the commits starting\nfrom v0.99, following the chain along the commit parents. When it\nencounters v0.99~49, it figures out that it has to fetch v0.99~50. To see\nwhether v0.99~49 introduced that file, it then has to inspect that commit\nobject and then fetch the tree object (v0.99~50^{tree}). Then, Git\ninspects that tree to find out the object ID for v0.99~50^{tree}:t/, sees\nthat it is identical to v0.99~49^{tree}:t/ and therefore the pathspec\nfilter skips this commit from the output of the `git log` command. A\ncouple of parent traversals later (always fetching the parent commit\nobject individually, then the associated tree object, then figuring out\nthat `t/` is unchanged) Git will encounter v0.99~55 where `t/` _did_\nchange. So now it also has to fetch _that_ tree object.\n\nIn total, we are looking at 400+ individual network round trips just to\nfetch the required tree/commit objects, i.e. before Git can show you the\noutput of that `git log` command. And that's just for back-filling the\nmissing tree/commit objects.\n\nIf we had done this using a shallow clone, Git would have stopped at the\nshallow boundary, the user would have had a chance to increase the depth\nin bigger chunks (probably first extending the depth by 50, then maybe\n100, then maybe going for 500) and while it would have been a lot of\nmanual labor, the total time would be still a lot shorter than those 400+\nnetwork round trips (which likely would incur some throttling on the\nserver side).\n\n> Thinking about this idea, I don't think it is viable. I would need to\n> see a lot of work done to test these scenarios closely to believe that\n> this type of partial clone is a desirable working state.\n\nIndeed, it is hard to think of a way how the design could result in\nanything but undesirable behavior, both on the client and the server side.\n\nWe also have to consider that our experience with large repositories\ndemonstrates that tree and commit objects delta pretty well and are\nvirtually never a concern when cloning. It is always the sheer amount of\nblob objects that is causing poor user experience when performing\nnon-partial clones of large repositories.\n\nNow, I can be totally wrong in my expectation that there is _no_ scenario\nwhere cloning with a \"partial depth\" would cause anything but poor\nperformance. If I am wrong, then there is value in having this feature,\nbut since it causes undesirable performance in all cases I can think of,\nit definitely should be guarded behind an opt-in flag.\n\nCiao,\nDscho\n"},{"id":"462581","messageId":"CAOLTT8SMQ9nPPs0OnxhH4n_A46WqW8Fv=priFs=NLuBycoBuug@mail.gmail.com","threadId":"58386","inReplyTo":"8b9e8c2d-7a64-2d66-83a8-2a7daff9a81c@github.com","subject":"Re: [PATCH 1/3] commit-graph: let commit graph respect commit graft","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-09-04T05:57:18Z","receivedAt":"2022-09-04T06:05:50Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Derrick Stolee <derrickstolee@github.com> 于2022年9月2日周五 03:18写道：\n>\n> On 9/1/2022 5:41 AM, ZheNing Hu via GitGitGadget wrote:\n> > From: ZheNing Hu <adlternative@gmail.com>\n> >\n> > In repo_parse_commit_internal(), if we want to use\n> > commit graph, it will call parse_commit_in_graph() to\n> > parse commit's content from commit graph, otherwise\n> > call repo_read_object_file() to parse commit's content\n> > from commit object.\n> >\n> > repo_read_object_file() will respect commit graft,\n> > which can correctly amend commit's parents. But\n> > parse_commit_in_graph() not. Inconsistencies here may\n> > result in incorrect processing of shallow clone.\n> >\n> > So let parse_commit_in_graph() respect commit graft as\n> > repo_read_object_file() does, which can solve this problem.\n>\n> If grafts or replace-objects exist, then the commit-graph\n> is disabled and this code will never be called. I would\n> expect a test case demonstrating the change in behavior\n> here, but that is impossible.\n>\n\nThanks for the clarification.\nI don't really know what's the wrong here, but just let do a little test:\n\n1. Revert this commit 19fd72c34dcd1332df638d76b0b028e9d9da3d41\n$ git revert 19fd72\n\n2. Clone the git repo\n$ git clone --bare git@github.com:git/git.git\n\n3. Write commit graph\n$ git commit-graph write\n\n4. Use the depth=<depth> to clone (depth=1)\n$  git clone --no-checkout --no-local --=depth=1 git.git git1\nCloning into 'git1'...\nremote: Enumerating objects: 4306, done.\nremote: Counting objects: 100% (4306/4306), done.\nremote: Compressing objects: 100% (3785/3785), done.\n\n4.  Use the depth=<depth> to clone (depth=2)\n$  git clone --no-checkout --no-local --=depth=2 git.git git2\nCloning into 'git2'...\nremote: Enumerating objects: 4311, done.\nremote: Counting objects: 100% (4311/4311), done.\nremote: Compressing objects: 100% (3788/3788), done.\n\n5. Use the depth filter to clone (depth=1)\n$  git clone --no-checkout --no-local --filter=depth:1 git.git git3\nCloning into 'git3'...\nremote: Enumerating objects: 4306, done.\nremote: Counting objects: 100% (4306/4306), done.\nremote: Compressing objects: 100% (3785/3785), done.\n\n6. Use the depth filter to clone (depth=2)\n$  git clone --no-checkout --no-local --filter=depth:2 git.git git4\nCloning into 'git4'...\nremote: Enumerating objects: 322987, done.\nremote: Counting objects: 100% (322987/322987), done.\nremote: Compressing objects: 100% (77441/77441), done.\n\nAs we can see, when we use --filter=depth:<depth> (depth >= 2),\nit seems like we clone a lot of objects. The result is significantly\ndifferent from git clone --depth=<depth> (depth >= 2).\n\nSo I debug it by reproducing the git pack-objects process:\n\nI find there are different action between --filter=depth:<depth> and\n --depth=<depth> .\n\n--filter=depth:<depth> will be successfully resolved commit parents in\nparse_commit_in_graph(),\n\nCall stack( cmd_pack_objects -> get_object_list -> traverse_commit_list ->\ntraverse_commit_list_filtered -> do_traverse -> get_revision ->\nget_revision_internal -> get_revision_1 -> process_parents ->\nrepo_parse_commit_gently -> repo_parse_commit_internal ->\nparse_commit_in_graph)\n\n--depth=<depth> will failed in parse_commit_in_graph(), and call\nrepo_read_object_file() to resolved commit parents.\n\nCall stack( cmd_pack_objects -> get_object_list -> traverse_commit_list ->\ntraverse_commit_list_filtered -> do_traverse -> get_revision ->\nget_revision_internal -> get_revision_1 -> process_parents ->\nrepo_parse_commit_gently -> repo_parse_commit_internal ->\nrepo_read_object_file)\n\n> The commit-graph parsing should not be bogged down with\n> this logic.\n>\n\nSo I try to fix this problem by let commit-graph respect commit-graft.\nI don't know if I overlook something before...\n\n> Thanks,\n> -Stolee\n>\n\nThanks,\nZheNing Hu\n"},{"id":"462582","messageId":"CAOLTT8T5gxC9F2KS--c8L3=-zzET=eRj_jNZxKeKGcoNZipzWw@mail.gmail.com","threadId":"58386","inReplyTo":"a14028be-2fd2-258d-94f5-c010669de8a6@github.com","subject":"Re: [PATCH 0/3] list-object-filter: introduce depth filter","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-09-04T07:27:47Z","receivedAt":"2022-09-04T07:28:31Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Derrick Stolee <derrickstolee@github.com> 于2022年9月2日周五 03:24写道：\n>\n> On 9/1/2022 5:41 AM, ZheNing Hu via GitGitGadget wrote:\n> > This patch let partial clone have the similar capabilities of the shallow\n> > clone git clone --depth=<depth>.\n> ...\n> > Now we can use git clone --filter=\"depth=<depth>\" to omit all commits whose\n> > depth is >= <depth>. By this way, we can have the advantages of both shallow\n> > clone and partial clone: Limiting the depth of commits, get other objects on\n> > demand.\n>\n> I have several concerns about this proposal.\n>\n> The first is that \"depth=X\" doesn't mean anything after the first\n> clone. What will happen when we fetch the remaining objects?\n>\n\nAccording to the current results, yes, it still downloads a large number\nof commits.\n\nDo a litte test again:\n\n$ git clone --filter=depth:2 git.git git\nCloning into 'git'...\nremote: Enumerating objects: 4311, done.\nremote: Counting objects: 100% (4311/4311), done.\nremote: Compressing objects: 100% (3788/3788), done.\n\nJust see how many objects...\n$ git cat-file --batch-check --batch-all-objects | grep blob | wc -l\nwarning: This repository uses promisor remotes. Some objects may not be loaded.\n    4098\n$ git cat-file --batch-check --batch-all-objects | grep tree | wc -l\nwarning: This repository uses promisor remotes. Some objects may not be loaded.\n     211\n$ git cat-file --batch-check --batch-all-objects | grep commit | wc -l\nwarning: This repository uses promisor remotes. Some objects may not be loaded.\n       2\n\n$ git checkout HEAD~\n\nFetch nothing...because depth=2.\n\n$  git checkout HEAD~\nremote: Enumerating objects: 198514, done.\nremote: Counting objects: 100% (198514/198514), done.\nremote: Compressing objects: 100% (68511/68511), done.\nremote: Total 198514 (delta 128408), reused 198509 (delta 128406), pack-reused 0\nReceiving objects: 100% (198514/198514), 77.07 MiB | 9.58 MiB/s, done.\nResolving deltas: 100% (128408/128408), done.\nremote: Enumerating objects: 1, done.\nremote: Counting objects: 100% (1/1), done.\nremote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0\nReceiving objects: 100% (1/1), 14.35 KiB | 14.35 MiB/s, done.\nremote: Enumerating objects: 198014, done.\nremote: Counting objects: 100% (198014/198014), done.\nremote: Compressing objects: 100% (68362/68362), done.\nremote: Total 198014 (delta 128056), reused 198012 (delta 128055), pack-reused 0\nReceiving objects: 100% (198014/198014), 76.55 MiB | 14.00 MiB/s, done.\nResolving deltas: 100% (128056/128056), done.\nPrevious HEAD position was 624a936234 Merge branch 'en/merge-multi-strategies'\nHEAD is now at 014a9ea207 Merge branch 'en/t4301-more-merge-tree-tests'\n\nFetch a lot of objects... (three times!)\n\n$ git cat-file --batch-check --batch-all-objects | grep blob | wc -l\nwarning: This repository uses promisor remotes. Some objects may not be loaded.\n    4099\n$ git cat-file --batch-check --batch-all-objects | grep tree | wc -l\nwarning: This repository uses promisor remotes. Some objects may not be loaded.\n    130712\n$ git cat-file --batch-check --batch-all-objects | grep commit | wc -l\nwarning: This repository uses promisor remotes. Some objects may not be loaded.\n    67815\n\nIt fetched too many Commits and Trees... But Surprisingly, only one\nmore blob was downloaded.\n\nI admit that this is a very bad action, That's because we\nhave no commits locally...\n\nMaybe one solution: we can also provide a commit-id parameter\ninside the depth filter, like --filter=\"commit:014a9ea207, depth:1\"...\nwe can clone with blob:none filter to download all trees/commits,\nthen fetch blobs with this \"commit-depth\" filter.... even we can\nprovide a more complex filter: --filter=\"commit:014a9ea207, depth:1, type=blob\"\nThis may avoid downloading too many unneeded commits and trees...\n\ngit fetch --filter=\"commit:014a9ea207, depth:1, type=blob\"\n\nIf git fetch have learned this filter, then git checkout or other commands can\n use this filter internally heuristically:\n\ne.g.\n\ngit checkout HEAD~\nif HEAD~ missing | 75% blobs/trees in HEAD~ missing -> use \"commit-depth\" filter\nelse -> use blob:none filter\n\nWe can even make this commit-depth filter support multiple commits later.\n\n> Partial clone is designed to download a subset of objects, but make\n> the remaining reachable objects downloadable on demand. By dropping\n> reachable commits, the normal partial clone mechanism would result\n> in a 'git rev-list' call asking for a missing commit. Would this\n> inherit the \"depth=X\" but result in a huge amount of over-downloading\n> the trees and blobs in that commit range? Would it result in downloading\n> commits one-by-one, and then their root trees (and all reachable objects\n> from those root trees)?\n>\n\nI don't know if it's possible let git rev-list know that commits is missing, and\nstop download them. (just like git cat-file --batch --batch-all-objects does)\n\nSimilarly, you can let git log or other commands to understand this...\n\nProbably a config var: fetch.skipmissingcommits...\n\n> Finally, computing the set of objects to send is just as expensive as\n> if we had a shallow clone (we can't use bitmaps). However, we get the\n> additional problem where fetches do not have a shallow boundary, so\n> the server will send deltas based on objects that are not necessarily\n> present locally, triggering extra requests to resolve those deltas.\n>\n\nAgree, I think this maybe a problem, but there is no good solution for it.\n\n> This fallout remains undocumented and unexplored in this series, but I\n> doubt the investigation would result in positive outcomes.\n>\n> > Disadvantages of git clone --depth=<depth> --filter=blob:none: we must call\n> > git fetch --unshallow to lift the shallow clone restriction, it will\n> > download all history of current commit.\n>\n> How does your proposal fix this? Instead of unshallowing, users will\n> stumble across these objects and trigger huge downloads by accident.\n>\n\nAs mentioned above, I would expect a commit-depth filter to fix this.\n\n> > Disadvantages of git clone --filter=blob:none with git sparse-checkout: The\n> > git client needs to send a lot of missing objects' id to the server, this\n> > can be very wasteful of network traffic.\n>\n> Asking for a list of blobs (especially limited to a sparse-checkout) is\n> much more efficient than what will happen when a user tries to do almost\n> anything in a repository formed the way you did here.\n>\n\nYes. also as mentioned above, enabling this filter in some specific cases:\ne.g. we have the commit but not all trees/blobs in it.\n\n> Thinking about this idea, I don't think it is viable. I would need to\n> see a lot of work done to test these scenarios closely to believe that\n> this type of partial clone is a desirable working state.\n>\n\nAgree.\n\n> Thanks,\n> -Stolee\n\nThanks to these reviews and criticisms, it makes me think more :)\n\nZheNing Hu\n"},{"id":"462583","messageId":"CAOLTT8S2r1gzyF8YAORuGwian+QwSniAPd8br0xn_P5gPyxpgg@mail.gmail.com","threadId":"58386","inReplyTo":"o48053s6-5540-1234-5roq-92q6981r2306@tzk.qr","subject":"Re: [PATCH 0/3] list-object-filter: introduce depth filter","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-09-04T09:14:20Z","receivedAt":"2022-09-04T09:14:37Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> 于2022年9月2日周五 21:48写道：\n>\n> Hi ZheNing,\n>\n> first of all: thank you for working on this. In the past, I thought that\n> this feature would be likely something we would want to have in Git.\n>\n\nOriginally, I just find \"full git checkout\" after partial-clone will\nsend so many blob-ids:\n\n$ git clone --filter=blob:none --no-checkout --sparse\ngit@github.com:derrickstolee/sparse-checkout-example.git\n$ cd sparse-checkout-example\n$ GIT_TRACE_PACKET=$HOME/packet.trace git checkout HEAD\n$ grep want $HOME/packet.trace  | wc -l\n4060\n\nSo I just think about whether this process can be simplified between\nthe client and the server. In git checkout, users only need all the objects\nin a commit. So maybe we can let the git client tell the server about this\ncommit-id, then the server downloads all objects in this commit. Then I\nfind it just looks like git clone|fetch --depth=1, but the shallow-clone doesn't\nseem as easy to extend missing objects as the partial-clone.\n\nhttps://git-scm.com/docs/partial-clone#_non_tasks also said:\n\nEvery time the subject of \"demand loading blobs\" comes up it seems\nthat someone suggests that the server be allowed to \"guess\" and send\nadditional objects that may be related to the requested objects.\n\nSo I guess --filter=depth:<depth> may be a solution, but as you and\nDerrick have said: there are still very many problems with this depth filter.\n\n> But Stolee's concerns are valid, and made me think about it more. See\n> below for a more detailed analysis.\n>\n> On Thu, 1 Sep 2022, Derrick Stolee wrote:\n>\n> > On 9/1/2022 5:41 AM, ZheNing Hu via GitGitGadget wrote:\n> >\n> > > [...]\n> > >\n> > > Disadvantages of git clone --filter=blob:none with git\n> > > sparse-checkout: The git client needs to send a lot of missing\n> > > objects' id to the server, this can be very wasteful of network\n> > > traffic.\n> >\n> > Asking for a list of blobs (especially limited to a sparse-checkout) is\n> > much more efficient than what will happen when a user tries to do almost\n> > anything in a repository formed the way you did here.\n>\n> I agree. When you have all the commit and tree objects on the local side,\n> you can enumerate all the blob objects you need in one fell swoop, then\n> fetch them in a single network round trip.\n>\n> When you lack tree objects, or worse, commit objects, this is not true.\n> You may very well need to fetch _quite_ a bunch of objects, then inspect\n> them to find out that you need to fetch more tree/commit objects, and then\n> a couple more round trips, before you can enumerate all of the objects you\n> need.\n>\n\nI think this is because the previous design was that you had to fetch\nthese missing\ncommits (also trees) and all their ancestors. Maybe we can modify git\nrev-list to\nmake it understand missing commits...\n\n> Concrete example: let's assume that you clone git.git with a \"partial\n> depth\" of 50. That is, while cloning, all of the tip commits' graphs will\n> be traversed up until the commits that are removed by 49 edges in the\n> commit graph. For example, v0.99~49 will be present locally after cloning,\n> but not v0.99~50.\n>\n> Now, the first-parent depth of v0.99 is 955 (verify with `git rev-list\n> --count --first-parent v0.99`). None of the commits reachable from v0.99\n> other than the tip itself seem to be closer to any other tag, so all\n> commits reachable from v0.99~49 will be missing locally. And since reverts\n> are rare, we must assume that the vast majority of the associated root\n> tree objects are missing, too.\n>\n> Digging through history, a contributor might need to investigate where,\n> say, `t/t4100/t-apply-7.expect` was introduced (it was in v0.99~206)\n> because they found something looking like a bug and they need to read the\n> commit message to see whether it was intentional. They know that this file\n> was already present in v0.99. Naturally, the command-line to investigate\n> that is:\n>\n>         git log --diff-filter=A v0.99 -- t/t4100/t-apply-7.expect\n>\n> So what does Git do in that operation? It traverses the commits starting\n> from v0.99, following the chain along the commit parents. When it\n> encounters v0.99~49, it figures out that it has to fetch v0.99~50. To see\n> whether v0.99~49 introduced that file, it then has to inspect that commit\n> object and then fetch the tree object (v0.99~50^{tree}). Then, Git\n> inspects that tree to find out the object ID for v0.99~50^{tree}:t/, sees\n> that it is identical to v0.99~49^{tree}:t/ and therefore the pathspec\n> filter skips this commit from the output of the `git log` command. A\n> couple of parent traversals later (always fetching the parent commit\n> object individually, then the associated tree object, then figuring out\n> that `t/` is unchanged) Git will encounter v0.99~55 where `t/` _did_\n> change. So now it also has to fetch _that_ tree object.\n>\n\nVery convincing example. I think some git commands which may require\nall missing commits history, should fetch all commits in a batch. (so this\ndepth filter is not very useful here)\n\n> In total, we are looking at 400+ individual network round trips just to\n> fetch the required tree/commit objects, i.e. before Git can show you the\n> output of that `git log` command. And that's just for back-filling the\n> missing tree/commit objects.\n>\n> If we had done this using a shallow clone, Git would have stopped at the\n> shallow boundary, the user would have had a chance to increase the depth\n> in bigger chunks (probably first extending the depth by 50, then maybe\n> 100, then maybe going for 500) and while it would have been a lot of\n> manual labor, the total time would be still a lot shorter than those 400+\n> network round trips (which likely would incur some throttling on the\n> server side).\n>\n\nAgree.\n\n> > Thinking about this idea, I don't think it is viable. I would need to\n> > see a lot of work done to test these scenarios closely to believe that\n> > this type of partial clone is a desirable working state.\n>\n> Indeed, it is hard to think of a way how the design could result in\n> anything but undesirable behavior, both on the client and the server side.\n>\n> We also have to consider that our experience with large repositories\n> demonstrates that tree and commit objects delta pretty well and are\n> virtually never a concern when cloning. It is always the sheer amount of\n> blob objects that is causing poor user experience when performing\n> non-partial clones of large repositories.\n>\n\nThanks, I think I understand the problem here. By the way, does it make\nsense to download just some of the commits/trees in some big repository\nwhich have several million commits/trees?\n\n> Now, I can be totally wrong in my expectation that there is _no_ scenario\n> where cloning with a \"partial depth\" would cause anything but poor\n> performance. If I am wrong, then there is value in having this feature,\n> but since it causes undesirable performance in all cases I can think of,\n> it definitely should be guarded behind an opt-in flag.\n>\n\nWell, now I think this depth filter might be a better fit for git fetch.\n\nIf git checkout or other commands which just need to check\nfew commits, and find almost all objects (maybe >= 75%) in a\ncommit are not local, it can use this depth filter to download them.\n\n> Ciao,\n> Dscho\n\nThanks,\nZheNing Hu\n"},{"id":"462700","messageId":"o10o218s-2rq4-9n3p-86np-rn79r7qr2139@tzk.qr","threadId":"58386","inReplyTo":"CAOLTT8S2r1gzyF8YAORuGwian+QwSniAPd8br0xn_P5gPyxpgg@mail.gmail.com","subject":"Re: [PATCH 0/3] list-object-filter: introduce depth filter","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-09-07T10:18:34Z","receivedAt":"2022-09-07T10:21:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi ZheNing,\n\nOn Sun, 4 Sep 2022, ZheNing Hu wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> 于2022年9月2日周五 21:48写道：\n>\n> > [...]\n> > When you have all the commit and tree objects on the local side,\n> > you can enumerate all the blob objects you need in one fell swoop, then\n> > fetch them in a single network round trip.\n> >\n> > When you lack tree objects, or worse, commit objects, this is not true.\n> > You may very well need to fetch _quite_ a bunch of objects, then inspect\n> > them to find out that you need to fetch more tree/commit objects, and then\n> > a couple more round trips, before you can enumerate all of the objects you\n> > need.\n>\n> I think this is because the previous design was that you had to fetch\n> these missing commits (also trees) and all their ancestors. Maybe we can\n> modify git rev-list to make it understand missing commits...\n\nWe do have such a modification, and it is called \"shallow clone\" ;-)\n\nGranted, shallow clones are not a complete solution and turned out to be a\ndead end (i.e. that design cannot be extended into anything more useful).\nBut that approach demonstrates what it would take to implement a logic\nwhereby Git understands that some commit ranges are missing and should not\nbe fetched automatically.\n\n> > [...] it is hard to think of a way how the design could result in\n> > anything but undesirable behavior, both on the client and the server\n> > side.\n> >\n> > We also have to consider that our experience with large repositories\n> > demonstrates that tree and commit objects delta pretty well and are\n> > virtually never a concern when cloning. It is always the sheer amount\n> > of blob objects that is causing poor user experience when performing\n> > non-partial clones of large repositories.\n>\n> Thanks, I think I understand the problem here. By the way, does it make\n> sense to download just some of the commits/trees in some big repository\n> which have several million commits/trees?\n\nIt probably only makes sense if we can come up with a good idea how to\nteach Git the trick to stop downloading so many objects in costly\nroundtrips.\n\nBut I wonder whether your scenarios are so different from the ones I\nencountered, in that commit and tree objects do _not_ delta well on your\nside?\n\nIf they _do_ delta well, i.e. if it is comparatively cheap to just fetch\nthem all in one go, it probably makes more sense to just drop the idea of\nfetching only some commit/tree objects but not others in a partial clone,\nand always fetch all of 'em.\n\n> > Now, I can be totally wrong in my expectation that there is _no_ scenario\n> > where cloning with a \"partial depth\" would cause anything but poor\n> > performance. If I am wrong, then there is value in having this feature,\n> > but since it causes undesirable performance in all cases I can think of,\n> > it definitely should be guarded behind an opt-in flag.\n>\n> Well, now I think this depth filter might be a better fit for git fetch.\n\nI disagree here, because I see all the same challenges as I described for\nclones missing entire commit ranges.\n\n> If git checkout or other commands which just need to check\n> few commits, and find almost all objects (maybe >= 75%) in a\n> commit are not local, it can use this depth filter to download them.\n\nIf you want a clone that does not show any reasonable commit history\nbecause it does not fetch commit objects on-the-fly, then we already have\nsuch a thing with shallow clones.\n\nThe only way to make Git's revision walking logic perform _somewhat_\nreasonably would be to teach it to fetch not just a single commit object\nwhen it was asked for, but to somehow pass a desired depth by which to\n\"unshallow\" automatically.\n\nHowever, such a feature would come with the same undesirable implications\non the server side as shallow clones (fetches into shallow clones are\n_really_ expensive on the server side).\n\nCiao,\nDscho\n"},{"id":"462915","messageId":"CAOLTT8S=4duvizrQQacJ0LjkEKQAL6DY0gwkCuxwDKAFs9XS-w@mail.gmail.com","threadId":"58386","inReplyTo":"o10o218s-2rq4-9n3p-86np-rn79r7qr2139@tzk.qr","subject":"Re: [PATCH 0/3] list-object-filter: introduce depth filter","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-09-11T10:59:24Z","receivedAt":"2022-09-11T10:59:41Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> 于2022年9月7日周三 18:18写道：\n>\n> Hi ZheNing,\n>\n> On Sun, 4 Sep 2022, ZheNing Hu wrote:\n>\n> > Johannes Schindelin <Johannes.Schindelin@gmx.de> 于2022年9月2日周五 21:48写道：\n> >\n> > > [...]\n> > > When you have all the commit and tree objects on the local side,\n> > > you can enumerate all the blob objects you need in one fell swoop, then\n> > > fetch them in a single network round trip.\n> > >\n> > > When you lack tree objects, or worse, commit objects, this is not true.\n> > > You may very well need to fetch _quite_ a bunch of objects, then inspect\n> > > them to find out that you need to fetch more tree/commit objects, and then\n> > > a couple more round trips, before you can enumerate all of the objects you\n> > > need.\n> >\n> > I think this is because the previous design was that you had to fetch\n> > these missing commits (also trees) and all their ancestors. Maybe we can\n> > modify git rev-list to make it understand missing commits...\n>\n> We do have such a modification, and it is called \"shallow clone\" ;-)\n>\n> Granted, shallow clones are not a complete solution and turned out to be a\n> dead end (i.e. that design cannot be extended into anything more useful).\n\nYeah, the depth filter would have been possible to overcome this\nshortcoming, but\nit may require very much network overhead in some special cases.\n\n> But that approach demonstrates what it would take to implement a logic\n> whereby Git understands that some commit ranges are missing and should not\n> be fetched automatically.\n>\n\nAgree. Git uses the commit-graft to do so.\n\n> > > [...] it is hard to think of a way how the design could result in\n> > > anything but undesirable behavior, both on the client and the server\n> > > side.\n> > >\n> > > We also have to consider that our experience with large repositories\n> > > demonstrates that tree and commit objects delta pretty well and are\n> > > virtually never a concern when cloning. It is always the sheer amount\n> > > of blob objects that is causing poor user experience when performing\n> > > non-partial clones of large repositories.\n> >\n> > Thanks, I think I understand the problem here. By the way, does it make\n> > sense to download just some of the commits/trees in some big repository\n> > which have several million commits/trees?\n>\n> It probably only makes sense if we can come up with a good idea how to\n> teach Git the trick to stop downloading so many objects in costly\n> roundtrips.\n>\n\nGood advice. Perhaps we should merge these multiple requests into one.\nMaybe we should use a blob:none filter to download all missing trees/commits\nif we need to iterate through all commits history.\n\n> But I wonder whether your scenarios are so different from the ones I\n> encountered, in that commit and tree objects do _not_ delta well on your\n> side?\n>\n> If they _do_ delta well, i.e. if it is comparatively cheap to just fetch\n> them all in one go, it probably makes more sense to just drop the idea of\n> fetching only some commit/tree objects but not others in a partial clone,\n> and always fetch all of 'em.\n>\n\nDelta is a wonderful thing most of the time (in cases where bulk acquisition\nis required). But sometimes I think users just want to see the message of one\ncommit, so why do they have to download other commits/trees that are not\nrequired?\n\nSometimes users may better understand the working patterns of their git\nobjects than the git server, It may be nice if the user could download the\nspecified object just mapped by its objectid (it is only for blob now, right?)\n\n> > > Now, I can be totally wrong in my expectation that there is _no_ scenario\n> > > where cloning with a \"partial depth\" would cause anything but poor\n> > > performance. If I am wrong, then there is value in having this feature,\n> > > but since it causes undesirable performance in all cases I can think of,\n> > > it definitely should be guarded behind an opt-in flag.\n> >\n> > Well, now I think this depth filter might be a better fit for git fetch.\n>\n> I disagree here, because I see all the same challenges as I described for\n> clones missing entire commit ranges.\n>\n\nOh, a prerequisite is missing here: after we have all commits, trees,\nthen use the\ndepth filter to down missing blobs.\n\n> > If git checkout or other commands which just need to check\n> > few commits, and find almost all objects (maybe >= 75%) in a\n> > commit are not local, it can use this depth filter to download them.\n>\n> If you want a clone that does not show any reasonable commit history\n> because it does not fetch commit objects on-the-fly, then we already have\n> such a thing with shallow clones.\n>\n> The only way to make Git's revision walking logic perform _somewhat_\n> reasonably would be to teach it to fetch not just a single commit object\n> when it was asked for, but to somehow pass a desired depth by which to\n> \"unshallow\" automatically.\n>\n> However, such a feature would come with the same undesirable implications\n> on the server side as shallow clones (fetches into shallow clones are\n> _really_ expensive on the server side).\n>\n\nAgree. letting git shallow clone to be smarter may work, but there are\nbig challenges too.\n\n> Ciao,\n> Dscho\n\nThanks,\nZheNing Hu\n"}]}