{"thread":{"id":"47102","subject":"[PATCH v2 0/6] Partial clone part 1: object filtering","startedAt":"2017-11-02T17:50:38Z","lastAt":"2017-11-08T14:39:42Z","messageCount":20,"participants":["Jeff Hostetler","Jonathan Tan","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":6},"messages":[{"id":"331673","messageId":"20171102175013.3371-1-git@jeffhostetler.com","threadId":"47102","inReplyTo":null,"subject":"[PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:07Z","receivedAt":"2017-11-02T17:50:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nHere is V2 of the list-object filtering. It replaces [1]\nand reflect a refactoring and simplification of the original.\n\nAfter much discussion on the \"list-object-filter-map\" I've replaced\nit with a regular oidset -- the only need for the map was to store\nthe first observed pathname for each blob, but that itself was of\nquestionable value.\n\nI've extended oidmap and oidset to have iterators. These 2 commits\ncould be pulled out and applied on their own, but for now I need\nthem here.\n\nThere were also several comments on the layout of the filtering\nAPI and the layout of the filter source code. I've restructured\nthe filtering routines to put them in the same source file, and\nmade them all static. These are now hidden behind a \"factory-like\"\nfunction with a vtable. This greatly simplifies the code in\ntraverse_commit_list_filtered().\n\nI've added \"--filter-ignore-missing\" parameter to rev-list and\npack-objects to ignore missing objects rather than error out.\nThis allows this patch series to better stand on its own eliminates\nthe need in part 1 for \"patch 9\" from V1.\n\nThis is a brute force ignore all missing objects. Later, in part\n2 or part 3 when --exclude-promisor-objects is introduced, we will\nbe able to ignore EXPECTED missing objects.\n\nFinally, patch 1 in this series is the same [2] which is currently\ncooking in next.\n\n[1] https://public-inbox.org/git/20171024185332.57261-1-git@jeffhostetler.com/\n\n[2] * jh/dir-add-exclude-from-blob (2017-10-27) 1 commit\n - dir: allow exclusions from blob in addition to file\n\n\nJeff Hostetler (6):\n dir: allow exclusions from blob in addition to file\n oidmap: add oidmap iterator methods\n oidset: add iterator methods to oidset\n list-objects: filter objects in traverse_commit_list\n rev-list: add list-objects filtering support\n pack-objects: add list-objects filtering\n\n Documentation/git-pack-objects.txt | 12 +-\n Documentation/git-rev-list.txt | 6 +-\n Documentation/rev-list-options.txt | 34 +++\n Makefile | 2 +\n builtin/pack-objects.c | 28 ++-\n builtin/rev-list.c | 75 +++++-\n dir.c | 132 ++++++++---\n dir.h | 3 +\n list-objects-filter-options.c | 119 ++++++++++\n list-objects-filter-options.h | 55 +++++\n list-objects-filter.c | 408 +++++++++++++++++++++++++++++++++\n list-objects-filter.h | 84 +++++++\n list-objects.c | 95 ++++++--\n list-objects.h | 2 +-\n oidmap.h | 22 ++\n oidset.c | 10 +\n oidset.h | 36 +++\n t/t5317-pack-objects-filter-objects.sh | 369 +++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 225 ++++++++++++++++++\n 19 files changed, 1664 insertions(+), 53 deletions(-)\n create mode 100644 list-objects-filter-options.c\n create mode 100644 list-objects-filter-options.h\n create mode 100644 list-objects-filter.c\n create mode 100644 list-objects-filter.h\n create mode 100755 t/t5317-pack-objects-filter-objects.sh\n create mode 100755 t/t6112-rev-list-filters-objects.sh\n\n-- \n2.9.3\n\n"},{"id":"331674","messageId":"20171102175013.3371-2-git@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"[PATCH v2 1/6] dir: allow exclusions from blob in addition to file","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:08Z","receivedAt":"2017-11-02T17:50:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor add_excludes() to separate the reading of the\nexclude file into a buffer and the parsing of the buffer\ninto exclude_list items.\n\nAdd add_excludes_from_blob_to_list() to allow an exclude\nfile be specified with an OID without assuming a local\nworktree or index exists.\n\nRefactor read_skip_worktree_file_from_index() and add\ndo_read_blob() to eliminate duplication of preliminary\nprocessing of blob contents.\n\nSigned-off-by: Jeff Hostetler \n---\n dir.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++----------------\n dir.h | 3 ++\n 2 files changed, 104 insertions(+), 31 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 1d17b80..1962374 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -220,6 +220,57 @@ int within_depth(const char *name, int namelen,\n \treturn 1;\n }\n \n+/*\n+ * Read the contents of the blob with the given OID into a buffer.\n+ * Append a trailing LF to the end if the last line doesn't have one.\n+ *\n+ * Returns:\n+ * -1 when the OID is invalid or unknown or does not refer to a blob.\n+ * 0 when the blob is empty.\n+ * 1 along with { data, size } of the (possibly augmented) buffer\n+ * when successful.\n+ *\n+ * Optionally updates the given sha1_stat with the given OID (when valid).\n+ */\n+static int do_read_blob(const struct object_id *oid,\n+\t\t\tstruct sha1_stat *sha1_stat,\n+\t\t\tsize_t *size_out,\n+\t\t\tchar **data_out)\n+{\n+\tenum object_type type;\n+\tunsigned long sz;\n+\tchar *data;\n+\n+\t*size_out = 0;\n+\t*data_out = NULL;\n+\n+\tdata = read_sha1_file(oid->hash, &type, &sz);\n+\tif (!data || type != OBJ_BLOB) {\n+\t\tfree(data);\n+\t\treturn -1;\n+\t}\n+\n+\tif (sha1_stat) {\n+\t\tmemset(&sha1_stat->stat, 0, sizeof(sha1_stat->stat));\n+\t\thashcpy(sha1_stat->sha1, oid->hash);\n+\t}\n+\n+\tif (sz == 0) {\n+\t\tfree(data);\n+\t\treturn 0;\n+\t}\n+\n+\tif (data[sz - 1] != '\\n') {\n+\t\tdata = xrealloc(data, st_add(sz, 1));\n+\t\tdata[sz++] = '\\n';\n+\t}\n+\n+\t*size_out = xsize_t(sz);\n+\t*data_out = data;\n+\n+\treturn 1;\n+}\n+\n #define DO_MATCH_EXCLUDE (1<<0)\n #define DO_MATCH_DIRECTORY (1<<1)\n #define DO_MATCH_SUBMODULE (1<<2)\n@@ -600,32 +651,22 @@ void add_exclude(const char *string, const char *base,\n \tx->el = el;\n }\n \n-static void *read_skip_worktree_file_from_index(const struct index_state *istate,\n-\t\t\t\t\t\tconst char *path, size_t *size,\n-\t\t\t\t\t\tstruct sha1_stat *sha1_stat)\n+static int read_skip_worktree_file_from_index(const struct index_state *istate,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t size_t *size_out,\n+\t\t\t\t\t char **data_out,\n+\t\t\t\t\t struct sha1_stat *sha1_stat)\n {\n \tint pos, len;\n-\tunsigned long sz;\n-\tenum object_type type;\n-\tvoid *data;\n \n \tlen = strlen(path);\n \tpos = index_name_pos(istate, path, len);\n \tif (pos < 0)\n-\t\treturn NULL;\n+\t\treturn -1;\n \tif (!ce_skip_worktree(istate->cache[pos]))\n-\t\treturn NULL;\n-\tdata = read_sha1_file(istate->cache[pos]->oid.hash, &type, &sz);\n-\tif (!data || type != OBJ_BLOB) {\n-\t\tfree(data);\n-\t\treturn NULL;\n-\t}\n-\t*size = xsize_t(sz);\n-\tif (sha1_stat) {\n-\t\tmemset(&sha1_stat->stat, 0, sizeof(sha1_stat->stat));\n-\t\thashcpy(sha1_stat->sha1, istate->cache[pos]->oid.hash);\n-\t}\n-\treturn data;\n+\t\treturn -1;\n+\n+\treturn do_read_blob(&istate->cache[pos]->oid, sha1_stat, size_out, data_out);\n }\n \n /*\n@@ -739,6 +780,10 @@ static void invalidate_directory(struct untracked_cache *uc,\n \t\tdir->dirs[i]->recurse = 0;\n }\n \n+static int add_excludes_from_buffer(char *buf, size_t size,\n+\t\t\t\t const char *base, int baselen,\n+\t\t\t\t struct exclude_list *el);\n+\n /*\n * Given a file with name \"fname\", read it (either from disk, or from\n * an index if 'istate' is non-null), parse it and store the\n@@ -754,9 +799,10 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t\tstruct sha1_stat *sha1_stat)\n {\n \tstruct stat st;\n-\tint fd, i, lineno = 1;\n+\tint r;\n+\tint fd;\n \tsize_t size = 0;\n-\tchar *buf, *entry;\n+\tchar *buf;\n \n \tfd = open(fname, O_RDONLY);\n \tif (fd < 0 || fstat(fd, &st) < 0) {\n@@ -764,17 +810,13 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t\twarn_on_fopen_errors(fname);\n \t\telse\n \t\t\tclose(fd);\n-\t\tif (!istate ||\n-\t\t (buf = read_skip_worktree_file_from_index(istate, fname, &size, sha1_stat)) == NULL)\n+\t\tif (!istate)\n \t\t\treturn -1;\n-\t\tif (size == 0) {\n-\t\t\tfree(buf);\n-\t\t\treturn 0;\n-\t\t}\n-\t\tif (buf[size-1] != '\\n') {\n-\t\t\tbuf = xrealloc(buf, st_add(size, 1));\n-\t\t\tbuf[size++] = '\\n';\n-\t\t}\n+\t\tr = read_skip_worktree_file_from_index(istate, fname,\n+\t\t\t\t\t\t &size, &buf,\n+\t\t\t\t\t\t sha1_stat);\n+\t\tif (r != 1)\n+\t\t\treturn r;\n \t} else {\n \t\tsize = xsize_t(st.st_size);\n \t\tif (size == 0) {\n@@ -813,6 +855,17 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t}\n \t}\n \n+\tadd_excludes_from_buffer(buf, size, base, baselen, el);\n+\treturn 0;\n+}\n+\n+static int add_excludes_from_buffer(char *buf, size_t size,\n+\t\t\t\t const char *base, int baselen,\n+\t\t\t\t struct exclude_list *el)\n+{\n+\tint i, lineno = 1;\n+\tchar *entry;\n+\n \tel->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -841,6 +894,23 @@ int add_excludes_from_file_to_list(const char *fname, const char *base,\n \treturn add_excludes(fname, base, baselen, el, istate, NULL);\n }\n \n+int add_excludes_from_blob_to_list(\n+\tstruct object_id *oid,\n+\tconst char *base, int baselen,\n+\tstruct exclude_list *el)\n+{\n+\tchar *buf;\n+\tsize_t size;\n+\tint r;\n+\n+\tr = do_read_blob(oid, NULL, &size, &buf);\n+\tif (r != 1)\n+\t\treturn r;\n+\n+\tadd_excludes_from_buffer(buf, size, base, baselen, el);\n+\treturn 0;\n+}\n+\n struct exclude_list *add_exclude_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src)\n {\ndiff --git a/dir.h b/dir.h\nindex e371705..1bcf391 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -256,6 +256,9 @@ extern struct exclude_list *add_exclude_list(struct dir_struct *dir,\n extern int add_excludes_from_file_to_list(const char *fname, const char *base, int baselen,\n \t\t\t\t\t struct exclude_list *el, struct index_state *istate);\n extern void add_excludes_from_file(struct dir_struct *, const char *fname);\n+extern int add_excludes_from_blob_to_list(struct object_id *oid,\n+\t\t\t\t\t const char *base, int baselen,\n+\t\t\t\t\t struct exclude_list *el);\n extern void parse_exclude_pattern(const char **string, int *patternlen, unsigned *flags, int *nowildcardlen);\n extern void add_exclude(const char *string, const char *base,\n \t\t\tint baselen, struct exclude_list *el, int srcpos);\n-- \n2.9.3\n\n"},{"id":"331675","messageId":"20171102175013.3371-3-git@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"[PATCH v2 2/6] oidmap: add oidmap iterator methods","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:09Z","receivedAt":"2017-11-02T17:50:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the usual map iterator functions to oidmap.\n\nSigned-off-by: Jeff Hostetler \n---\n oidmap.h | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/oidmap.h b/oidmap.h\nindex 18f54cd..d3cd2bb 100644\n--- a/oidmap.h\n+++ b/oidmap.h\n@@ -65,4 +65,26 @@ extern void *oidmap_put(struct oidmap *map, void *entry);\n */\n extern void *oidmap_remove(struct oidmap *map, const struct object_id *key);\n \n+\n+struct oidmap_iter {\n+\tstruct hashmap_iter h_iter;\n+};\n+\n+static inline void oidmap_iter_init(struct oidmap *map, struct oidmap_iter *iter)\n+{\n+\thashmap_iter_init(&map->map, &iter->h_iter);\n+}\n+\n+static inline void *oidmap_iter_next(struct oidmap_iter *iter)\n+{\n+\treturn hashmap_iter_next(&iter->h_iter);\n+}\n+\n+static inline void *oidmap_iter_first(struct oidmap *map,\n+\t\t\t\t struct oidmap_iter *iter)\n+{\n+\toidmap_iter_init(map, iter);\n+\treturn oidmap_iter_next(iter);\n+}\n+\n #endif\n-- \n2.9.3\n\n"},{"id":"331676","messageId":"20171102175013.3371-5-git@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"[PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:11Z","receivedAt":"2017-11-02T17:50:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate traverse_commit_list_filtered() and add filtering\ninterface to allow certain objects to be omitted from the\ntraversal.\n\nUpdate traverse_commit_list() to be a wrapper for the above\nwith a null filter to minimize the number of callers that\nneeded to be changed.\n\nObject filtering will be used in a future commit by rev-list\nand pack-objects for partial clone and fetch to omit unwanted\nobjects from the result.\n\ntraverse_bitmap_commit_list() does not work with filtering.\n\nIf a packfile bitmap is present, it will not be used.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 2 +\n list-objects-filter-options.c | 119 ++++++++++++\n list-objects-filter-options.h | 55 ++++++\n list-objects-filter.c | 408 ++++++++++++++++++++++++++++++++++++++++++\n list-objects-filter.h | 84 +++++++++\n list-objects.c | 95 ++++++++--\n list-objects.h | 2 +-\n 7 files changed, 748 insertions(+), 17 deletions(-)\n create mode 100644 list-objects-filter-options.c\n create mode 100644 list-objects-filter-options.h\n create mode 100644 list-objects-filter.c\n create mode 100644 list-objects-filter.h\n\ndiff --git a/Makefile b/Makefile\nindex cd75985..ca378a4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -807,6 +807,8 @@ LIB_OBJS += levenshtein.o\n LIB_OBJS += line-log.o\n LIB_OBJS += line-range.o\n LIB_OBJS += list-objects.o\n+LIB_OBJS += list-objects-filter.o\n+LIB_OBJS += list-objects-filter-options.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nnew file mode 100644\nindex 0000000..31255e7\n--- /dev/null\n+++ b/list-objects-filter-options.c\n@@ -0,0 +1,119 @@\n+#include \"cache.h\"\n+#include \"commit.h\"\n+#include \"config.h\"\n+#include \"revision.h\"\n+#include \"argv-array.h\"\n+#include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n+\n+/*\n+ * Parse value of the argument to the \"filter\" keword.\n+ * On the command line this looks like:\n+ * --filter=\n+ * and in the pack protocol as:\n+ * \"filter\" SP \n+ *\n+ * ::= blob:none\n+ * blob:limit=[kmg]\n+ * sparse:oid=\n+ * sparse:path=\n+ */\n+int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n+\t\t\t const char *arg)\n+{\n+\tstruct object_context oc;\n+\tstruct object_id sparse_oid;\n+\tconst char *v0;\n+\tconst char *v1;\n+\n+\tif (filter_options->choice)\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\t/*\n+\t * TODO consider rejecting 'arg' if it contains any\n+\t * TODO injection characters (since we might send this\n+\t * TODO to a sub-command or to the server and we don't\n+\t * TODO want to deal with legacy quoting/escaping for\n+\t * TODO a new feature).\n+\t */\n+\n+\tfilter_options->raw_value = strdup(arg);\n+\n+\tif (skip_prefix(arg, \"blob:\", &v0) || skip_prefix(arg, \"blobs:\", &v0)) {\n+\t\tif (!strcmp(v0, \"none\")) {\n+\t\t\tfilter_options->choice = LOFC_BLOB_NONE;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (skip_prefix(v0, \"limit=\", &v1) &&\n+\t\t git_parse_ulong(v1, &filter_options->blob_limit_value)) {\n+\t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\telse if (skip_prefix(arg, \"sparse:\", &v0)) {\n+\t\tif (skip_prefix(v0, \"oid=\", &v1)) {\n+\t\t\tfilter_options->choice = LOFC_SPARSE_OID;\n+\t\t\tif (!get_oid_with_context(v1, GET_OID_BLOB,\n+\t\t\t\t\t\t &sparse_oid, &oc)) {\n+\t\t\t\t/*\n+\t\t\t\t * We successfully converted the \n+\t\t\t\t * into an actual OID. Rewrite the raw_value\n+\t\t\t\t * in canonoical form with just the OID.\n+\t\t\t\t * (If we send this request to the server, we\n+\t\t\t\t * want an absolute expression rather than a\n+\t\t\t\t * local-ref-relative expression.)\n+\t\t\t\t */\n+\t\t\t\tfree((char *)filter_options->raw_value);\n+\t\t\t\tfilter_options->raw_value =\n+\t\t\t\t\txstrfmt(\"sparse:oid=%s\",\n+\t\t\t\t\t\toid_to_hex(&sparse_oid));\n+\t\t\t\tfilter_options->sparse_oid_value =\n+\t\t\t\t\toiddup(&sparse_oid);\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * We could not turn the into an\n+\t\t\t\t * OID. Leave the raw_value as is in case\n+\t\t\t\t * the server can parse it. (It may refer to\n+\t\t\t\t * a branch, commit, or blob we don't have.)\n+\t\t\t\t */\n+\t\t\t}\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (skip_prefix(v0, \"path=\", &v1)) {\n+\t\t\tfilter_options->choice = LOFC_SPARSE_PATH;\n+\t\t\tfilter_options->sparse_path_value = strdup(v1);\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tdie(_(\"invalid filter expression '%s'\"), arg);\n+\treturn 0;\n+}\n+\n+int opt_parse_list_objects_filter(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct list_objects_filter_options *filter_options = opt->value;\n+\n+\tassert(arg);\n+\tassert(!unset);\n+\n+\treturn parse_list_objects_filter(filter_options, arg);\n+}\n+\n+void arg_format_list_objects_filter(\n+\tstruct argv_array *argv_array,\n+\tconst struct list_objects_filter_options *filter_options)\n+{\n+\tif (!filter_options->choice)\n+\t\treturn;\n+\n+\t/*\n+\t * TODO Think about quoting the value.\n+\t */\n+\targv_array_pushf(argv_array, \"--%s=%s\", CL_ARG__FILTER,\n+\t\t\t filter_options->raw_value);\n+}\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nnew file mode 100644\nindex 0000000..c9c5052\n--- /dev/null\n+++ b/list-objects-filter-options.h\n@@ -0,0 +1,55 @@\n+#ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n+#define LIST_OBJECTS_FILTER_OPTIONS_H\n+\n+#include \"parse-options.h\"\n+\n+/*\n+ * The list of defined filters for list-objects.\n+ */\n+enum list_objects_filter_choice {\n+\tLOFC_DISABLED = 0,\n+\tLOFC_BLOB_NONE,\n+\tLOFC_BLOB_LIMIT,\n+\tLOFC_SPARSE_OID,\n+\tLOFC_SPARSE_PATH,\n+\tLOFC__COUNT /* must be last */\n+};\n+\n+struct list_objects_filter_options {\n+\t/*\n+\t * The raw argument value given on the command line or\n+\t * protocol request. (The part after the \"--keyword=\".)\n+\t */\n+\tchar *raw_value;\n+\n+\t/*\n+\t * Parsed values. Only 1 will be set depending on the flags below.\n+\t */\n+\tstruct object_id *sparse_oid_value;\n+\tchar *sparse_path_value;\n+\tunsigned long blob_limit_value;\n+\n+\tenum list_objects_filter_choice choice;\n+};\n+\n+/* Normalized command line arguments */\n+#define CL_ARG__FILTER \"filter\"\n+\n+int parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg);\n+\n+int opt_parse_list_objects_filter(const struct option *opt,\n+\t\t\t\t const char *arg, int unset);\n+\n+#define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n+\t N_(\"object filtering\"), PARSE_OPT_NONEG, \\\n+\t opt_parse_list_objects_filter }\n+\n+struct argv_array;\n+void arg_format_list_objects_filter(\n+\tstruct argv_array *aa,\n+\tconst struct list_objects_filter_options *filter_options);\n+\n+#endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nnew file mode 100644\nindex 0000000..7f28425\n--- /dev/null\n+++ b/list-objects-filter.c\n@@ -0,0 +1,408 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"tag.h\"\n+#include \"commit.h\"\n+#include \"tree.h\"\n+#include \"blob.h\"\n+#include \"diff.h\"\n+#include \"tree-walk.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n+#include \"oidset.h\"\n+\n+/* See object.h and revision.h */\n+#define FILTER_REVISIT (1<<25)\n+\n+/*\n+ * A filter for list-objects to omit ALL blobs from the traversal.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_blobs_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_blobs_none(\n+\tenum list_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_blobs_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\t/* always include all tree objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_SHOW (hard omit) */\n+\t}\n+}\n+\n+static void *filter_blobs_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_blobs_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n+/*\n+ * A filter for list-objects to omit large blobs,\n+ * but always include \".git*\" special files.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_blobs_limit_data {\n+\tstruct oidset *omits;\n+\tunsigned long max_bytes;\n+};\n+\n+static enum list_objects_filter_result filter_blobs_limit(\n+\tenum list_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_blobs_limit_data *filter_data = filter_data_;\n+\tunsigned long object_length;\n+\tenum object_type t;\n+\tint is_special_filename;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\t/* always include all tree objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\tis_special_filename = ((strncmp(filename, \".git\", 4) == 0) &&\n+\t\t\t\t filename[4]);\n+\t\tif (is_special_filename) {\n+\t\t\t/*\n+\t\t\t * Alwayse include \".git*\" special files (regardless\n+\t\t\t * of size).\n+\t\t\t *\n+\t\t\t * (This may cause us to include blobs that we do\n+\t\t\t * not have locally because we are only looking at\n+\t\t\t * the filename and don't actually have to read\n+\t\t\t * them.)\n+\t\t\t */\n+\t\t\tgoto include_it;\n+\t\t}\n+\n+\t\tt = sha1_object_info(obj->oid.hash, &object_length);\n+\t\tif (t != OBJ_BLOB) { /* probably OBJ_NONE */\n+\t\t\t/*\n+\t\t\t * We DO NOT have the blob locally, so we cannot\n+\t\t\t * apply the size filter criteria. Be conservative\n+\t\t\t * and force show it (and let the caller deal with\n+\t\t\t * the ambiguity). (This matches the behavior above\n+\t\t\t * when the special filename matches.)\n+\t\t\t */\n+\t\t\tgoto include_it;\n+\t\t}\n+\n+\t\tif (object_length < filter_data->max_bytes)\n+\t\t\tgoto include_it;\n+\n+\t\t/*\n+\t\t * Provisionally omit it. We've already established\n+\t\t * that this blob is too big and doesn't have a special\n+\t\t * filename, so we *WANT* to omit it. However, there\n+\t\t * may be a special file elsewhere in the tree that\n+\t\t * references this same blob, so we cannot reject it\n+\t\t * just yet. Leave the LOFR_ bits unset so that *IF*\n+\t\t * the blob appears again in the traversal, we will\n+\t\t * be asked again.\n+\t\t *\n+\t\t * If we are keeping a list of the ommitted objects,\n+\t\t * provisionally add it to the list.\n+\t\t */\n+\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_ZERO;\n+\t}\n+\n+include_it:\n+\tif (filter_data->omits)\n+\t\toidset_remove(filter_data->omits, &obj->oid);\n+\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+}\n+\n+static void *filter_blobs_limit__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\td->max_bytes = filter_options->blob_limit_value;\n+\n+\t*filter_fn = filter_blobs_limit;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n+/*\n+ * A filter driven by a sparse-checkout specification to only\n+ * include blobs that a sparse checkout would populate.\n+ *\n+ * The sparse-checkout spec can be loaded from a blob with the\n+ * given OID or from a local pathname. We allow an OID because\n+ * the repo may be bare or we may be doing the filtering on the\n+ * server.\n+ */\n+struct frame {\n+\tint defval;\n+\tint child_prov_omit : 1;\n+};\n+\n+struct filter_sparse_data {\n+\tstruct oidset *omits;\n+\tstruct exclude_list el;\n+\n+\tsize_t nr, alloc;\n+\tstruct frame *array_frame;\n+};\n+\n+static enum list_objects_filter_result filter_sparse(\n+\tenum list_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_sparse_data *filter_data = filter_data_;\n+\tint val, dtype;\n+\tstruct frame *frame;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tdtype = DT_DIR;\n+\t\tval = is_excluded_from_list(pathname, strlen(pathname),\n+\t\t\t\t\t filename, &dtype, &filter_data->el,\n+\t\t\t\t\t &the_index);\n+\t\tif (val < 0)\n+\t\t\tval = filter_data->array_frame[filter_data->nr].defval;\n+\n+\t\tALLOC_GROW(filter_data->array_frame, filter_data->nr + 1,\n+\t\t\t filter_data->alloc);\n+\t\tfilter_data->nr++;\n+\t\tfilter_data->array_frame[filter_data->nr].defval = val;\n+\t\tfilter_data->array_frame[filter_data->nr].child_prov_omit = 0;\n+\n+\t\t/*\n+\t\t * A directory with this tree OID may appear in multiple\n+\t\t * places in the tree. (Think of a directory move, with\n+\t\t * no other changes.) And with a different pathname, the\n+\t\t * is_excluded...() results for this directory and items\n+\t\t * contained within it may be different. So we cannot\n+\t\t * mark it SEEN (yet), since that will prevent process_tree()\n+\t\t * from revisiting this tree object with other pathnames.\n+\t\t *\n+\t\t * Only SHOW the tree object the first time we visit this\n+\t\t * tree object.\n+\t\t *\n+\t\t * We always show all tree objects. A future optimization\n+\t\t * may want to attempt to narrow this.\n+\t\t */\n+\t\tif (obj->flags & FILTER_REVISIT)\n+\t\t\treturn LOFR_ZERO;\n+\t\tobj->flags |= FILTER_REVISIT;\n+\t\treturn LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tassert(filter_data->nr > 0);\n+\n+\t\tframe = &filter_data->array_frame[filter_data->nr];\n+\t\tfilter_data->nr--;\n+\n+\t\t/*\n+\t\t * Tell our parent directory if any of our children were\n+\t\t * provisionally omitted.\n+\t\t */\n+\t\tfilter_data->array_frame[filter_data->nr].child_prov_omit |=\n+\t\t\tframe->child_prov_omit;\n+\n+\t\t/*\n+\t\t * If there are NO provisionally omitted child objects (ALL child\n+\t\t * objects in this folder were INCLUDED), then we can mark the\n+\t\t * folder as SEEN (so we will not have to revisit it again).\n+\t\t */\n+\t\tif (!frame->child_prov_omit)\n+\t\t\treturn LOFR_MARK_SEEN;\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\tframe = &filter_data->array_frame[filter_data->nr];\n+\n+\t\tdtype = DT_REG;\n+\t\tval = is_excluded_from_list(pathname, strlen(pathname),\n+\t\t\t\t\t filename, &dtype, &filter_data->el,\n+\t\t\t\t\t &the_index);\n+\t\tif (val < 0)\n+\t\t\tval = frame->defval;\n+\t\tif (val > 0) {\n+\t\t\tif (filter_data->omits)\n+\t\t\t\toidset_remove(filter_data->omits, &obj->oid);\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Provisionally omit it. We've already established that\n+\t\t * this pathname is not in the sparse-checkout specification\n+\t\t * with the CURRENT pathname, so we *WANT* to omit this blob.\n+\t\t *\n+\t\t * However, a pathname elsewhere in the tree may also\n+\t\t * reference this same blob, so we cannot reject it yet.\n+\t\t * Leave the LOFR_ bits unset so that if the blob appears\n+\t\t * again in the traversal, we will be asked again.\n+\t\t */\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\n+\t\t/*\n+\t\t * Remember that at least 1 blob in this tree was\n+\t\t * provisionally omitted. This prevents us from short\n+\t\t * cutting the tree in future iterations.\n+\t\t */\n+\t\tframe->child_prov_omit = 1;\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+\n+static void filter_sparse_free(void *filter_data)\n+{\n+\tstruct filter_sparse_data *d = filter_data;\n+\t/* TODO free contents of 'd' */\n+\tfree(d);\n+}\n+\n+static void *filter_sparse_oid__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n+\t\t\t\t\t NULL, 0, &d->el) < 0)\n+\t\tdie(\"could not load filter specification\");\n+\n+\tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n+\td->array_frame[d->nr].defval = 0; /* default to include */\n+\td->array_frame[d->nr].child_prov_omit = 0;\n+\n+\t*filter_fn = filter_sparse;\n+\t*filter_free_fn = filter_sparse_free;\n+\treturn d;\n+}\n+\n+static void *filter_sparse_path__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\tif (add_excludes_from_file_to_list(filter_options->sparse_path_value,\n+\t\t\t\t\t NULL, 0, &d->el, NULL) < 0)\n+\t\tdie(\"could not load filter specification\");\n+\n+\tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n+\td->array_frame[d->nr].defval = 0; /* default to include */\n+\td->array_frame[d->nr].child_prov_omit = 0;\n+\n+\t*filter_fn = filter_sparse;\n+\t*filter_free_fn = filter_sparse_free;\n+\treturn d;\n+}\n+\n+typedef void *(*filter_init_fn)(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn);\n+\n+/*\n+ * Must match \"enum list_objects_filter_choice\".\n+ */\n+static filter_init_fn s_filters[] = {\n+\tNULL,\n+\tfilter_blobs_none__init,\n+\tfilter_blobs_limit__init,\n+\tfilter_sparse_oid__init,\n+\tfilter_sparse_path__init,\n+};\n+\n+void *list_objects_filter__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tfilter_init_fn init_fn;\n+\n+\tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n+\n+\tif (filter_options->choice >= LOFC__COUNT)\n+\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\t filter_options->choice);\n+\n+\tinit_fn = s_filters[filter_options->choice];\n+\tif (init_fn)\n+\t\treturn init_fn(omitted, filter_options,\n+\t\t\t filter_fn, filter_free_fn);\n+\t*filter_fn = NULL;\n+\t*filter_free_fn = NULL;\n+\treturn NULL;\n+}\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nnew file mode 100644\nindex 0000000..f30a514\n--- /dev/null\n+++ b/list-objects-filter.h\n@@ -0,0 +1,84 @@\n+#ifndef LIST_OBJECTS_FILTER_H\n+#define LIST_OBJECTS_FILTER_H\n+\n+/*\n+ * During list-object traversal we allow certain objects to be\n+ * filtered (omitted) from the result. The active filter uses\n+ * these result values to guide list-objects.\n+ *\n+ * _ZERO : Do nothing with the object at this time. It may\n+ * be revisited if it appears in another place in\n+ * the tree or in another commit during the overall\n+ * traversal.\n+ *\n+ * _MARK_SEEN : Mark this object as \"SEEN\" in the object flags.\n+ * This will prevent it from being revisited during\n+ * the remainder of the traversal. This DOES NOT\n+ * imply that it will be included in the results.\n+ *\n+ * _SHOW : Show this object in the results (call show() on it).\n+ * In general, objects should only be shown once, but\n+ * this result DOES NOT imply that we mark it SEEN.\n+ *\n+ * Most of the time, you want the combination (_MARK_SEEN | _SHOW)\n+ * but they can be used independently, such as when sparse-checkout\n+ * pattern matching is being applied.\n+ *\n+ * A _MARK_SEEN without _SHOW can be called a hard-omit -- the\n+ * object is not shown and will never be reconsidered (unless a\n+ * previous iteration has already shown it).\n+ *\n+ * A _ZERO is can be called a provisional-omit -- the object is\n+ * not shown, but *may* be revisited (if the object appears again\n+ * in the traversal). Therefore, it will be omitted from the\n+ * results *unless* a later iteration causes it to be shown.\n+ */\n+enum list_objects_filter_result {\n+\tLOFR_ZERO = 0,\n+\tLOFR_MARK_SEEN = 1<<0,\n+\tLOFR_SHOW = 1<<1,\n+};\n+\n+enum list_objects_filter_type {\n+\tLOFT_BEGIN_TREE,\n+\tLOFT_END_TREE,\n+\tLOFT_BLOB\n+};\n+\n+typedef enum list_objects_filter_result (*filter_object_fn)(\n+\tenum list_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data);\n+\n+typedef void (*filter_free_fn)(void *filter_data);\n+\n+struct oidset;\n+struct list_objects_filter_options;\n+\n+void traverse_commit_list_filtered(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\tvoid *show_data,\n+\tstruct oidset *omitted);\n+\n+/*\n+ * Constructor for the set of defined list-objects filters.\n+ * Returns a generic \"void *filter_data\".\n+ *\n+ * The returned \"filter_fn\" will be used by traverse_commit_list()\n+ * to filter the results.\n+ *\n+ * The returned \"filter_free_fn\" is a destructor for the\n+ * filter_data.\n+ */\n+void *list_objects_filter__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn);\n+\n+#endif /* LIST_OBJECTS_FILTER_H */\ndiff --git a/list-objects.c b/list-objects.c\nindex b3931fa..848b040 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -7,16 +7,21 @@\n #include \"tree-walk.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n \n static void process_blob(struct rev_info *revs,\n \t\t\t struct blob *blob,\n \t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t void *cb_data,\n+\t\t\t filter_object_fn filter_fn,\n+\t\t\t void *filter_data)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n+\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_SHOW;\n \n \tif (!revs->blob_objects)\n \t\treturn;\n@@ -24,11 +29,17 @@ static void process_blob(struct rev_info *revs,\n \t\tdie(\"bad blob object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tobj->flags |= SEEN;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tshow(obj, path->buf, cb_data);\n+\tif (filter_fn)\n+\t\tr = filter_fn(LOFT_BLOB, obj,\n+\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t filter_data);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_SHOW)\n+\t\tshow(obj, path->buf, cb_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -69,7 +80,9 @@ static void process_tree(struct rev_info *revs,\n \t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t void *cb_data,\n+\t\t\t filter_object_fn filter_fn,\n+\t\t\t void *filter_data)\n {\n \tstruct object *obj = &tree->object;\n \tstruct tree_desc desc;\n@@ -77,6 +90,7 @@ static void process_tree(struct rev_info *revs,\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n \t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n+\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_SHOW;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -90,9 +104,15 @@ static void process_tree(struct rev_info *revs,\n \t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n-\tobj->flags |= SEEN;\n \tstrbuf_addstr(base, name);\n-\tshow(obj, base->buf, cb_data);\n+\tif (filter_fn)\n+\t\tr = filter_fn(LOFT_BEGIN_TREE, obj,\n+\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t filter_data);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_SHOW)\n+\t\tshow(obj, base->buf, cb_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -112,7 +132,7 @@ static void process_tree(struct rev_info *revs,\n \t\t\tprocess_tree(revs,\n \t\t\t\t lookup_tree(entry.oid),\n \t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data);\n+\t\t\t\t cb_data, filter_fn, filter_data);\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(revs, entry.oid->hash,\n \t\t\t\t\tshow, base, entry.path,\n@@ -121,8 +141,19 @@ static void process_tree(struct rev_info *revs,\n \t\t\tprocess_blob(revs,\n \t\t\t\t lookup_blob(entry.oid),\n \t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data);\n+\t\t\t\t cb_data, filter_fn, filter_data);\n \t}\n+\n+\tif (filter_fn) {\n+\t\tr = filter_fn(LOFT_END_TREE, obj,\n+\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t filter_data);\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tobj->flags |= SEEN;\n+\t\tif (r & LOFR_SHOW)\n+\t\t\tshow(obj, base->buf, cb_data);\n+\t}\n+\n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n@@ -183,10 +214,12 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t show_commit_fn show_commit,\n-\t\t\t show_object_fn show_object,\n-\t\t\t void *data)\n+static void do_traverse(struct rev_info *revs,\n+\t\t\tshow_commit_fn show_commit,\n+\t\t\tshow_object_fn show_object,\n+\t\t\tvoid *show_data,\n+\t\t\tfilter_object_fn filter_fn,\n+\t\t\tvoid *filter_data)\n {\n \tint i;\n \tstruct commit *commit;\n@@ -200,7 +233,7 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t */\n \t\tif (commit->tree)\n \t\t\tadd_pending_tree(revs, commit->tree);\n-\t\tshow_commit(commit, data);\n+\t\tshow_commit(commit, show_data);\n \t}\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = revs->pending.objects + i;\n@@ -211,19 +244,21 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, data);\n+\t\t\tshow_object(obj, name, show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n \t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t &base, path, data);\n+\t\t\t\t &base, path, show_data,\n+\t\t\t\t filter_fn, filter_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n \t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t &base, path, data);\n+\t\t\t\t &base, path, show_data,\n+\t\t\t\t filter_fn, filter_data);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n@@ -232,3 +267,31 @@ void traverse_commit_list(struct rev_info *revs,\n \tobject_array_clear(&revs->pending);\n \tstrbuf_release(&base);\n }\n+\n+void traverse_commit_list(struct rev_info *revs,\n+\t\t\t show_commit_fn show_commit,\n+\t\t\t show_object_fn show_object,\n+\t\t\t void *show_data)\n+{\n+\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+}\n+\n+void traverse_commit_list_filtered(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\tvoid *show_data,\n+\tstruct oidset *omitted)\n+{\n+\tfilter_object_fn filter_fn = NULL;\n+\tfilter_free_fn filter_free_fn = NULL;\n+\tvoid *filter_data = NULL;\n+\n+\tfilter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n+\tdo_traverse(revs, show_commit, show_object, show_data,\n+\t\t filter_fn, filter_data);\n+\tif (filter_data && filter_free_fn)\n+\t\tfilter_free_fn(filter_data);\n+}\ndiff --git a/list-objects.h b/list-objects.h\nindex 0cebf85..33c964c 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -8,4 +8,4 @@ void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, voi\n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n \n-#endif\n+#endif /* LIST_OBJECTS_H */\n-- \n2.9.3\n\n"},{"id":"331677","messageId":"20171102175013.3371-7-git@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"[PATCH v2 6/6] pack-objects: add list-objects filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:13Z","receivedAt":"2017-11-02T17:50:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach pack-objects to use the filtering provided by the\ntraverse_commit_list_filtered() interface to omit unwanted\nobjects from the resulting packfile.\n\nThis feature is intended for partial clone/fetch.\n\nFiltering requires the use of the \"--stdout\" option.\n\nAdd t5317 test.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-pack-objects.txt | 12 +-\n builtin/pack-objects.c | 28 ++-\n t/t5317-pack-objects-filter-objects.sh | 369 +++++++++++++++++++++++++++++++++\n 3 files changed, 407 insertions(+), 2 deletions(-)\n create mode 100755 t/t5317-pack-objects-filter-objects.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 473a161..6786351 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -12,7 +12,8 @@ SYNOPSIS\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n-\t[--revs [--unpacked | --all]] [--stdout | base-name]\n+\t[--revs [--unpacked | --all]]\n+\t[--stdout [--filter=] | base-name]\n \t[--shallow] [--keep-true-parents] < object-list\n \n \n@@ -236,6 +237,15 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tWith this option, parents that are hidden by grafts are packed\n \tnevertheless.\n \n+--filter=::\n+\tRequires `--stdout`. Omits certain objects (usually blobs) from\n+\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\t`` forms.\n+\n+--filter-ignore-missing:\n+\tIgnore missing objects without error. This may be used with\n+\tor without and of the above filtering.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6e77dfd..e16722f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -15,6 +15,8 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n #include \"pack-objects.h\"\n #include \"progress.h\"\n #include \"refs.h\"\n@@ -79,6 +81,9 @@ static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\n \n+static struct list_objects_filter_options filter_options;\n+static int arg_ignore_missing;\n+\n /*\n * stats\n */\n@@ -2547,6 +2552,15 @@ static void show_commit(struct commit *commit, void *data)\n \n static void show_object(struct object *obj, const char *name, void *data)\n {\n+\t/*\n+\t * Quietly ignore missing objects when they are expected. This\n+\t * avoids staging them and getting an odd error later. If we are\n+\t * not expecting them, stage it and let the normal error handling\n+\t * deal with it.\n+\t */\n+\tif (arg_ignore_missing && !has_object_file(&obj->oid))\n+\t\treturn;\n+\n \tadd_preferred_base_object(name);\n \tadd_object_entry(obj->oid.hash, obj->type, name, 0);\n \tobj->flags |= OBJECT_ADDED;\n@@ -2816,7 +2830,10 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(&revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\n+\ttraverse_commit_list_filtered(&filter_options, &revs,\n+\t\t\t\t show_commit, show_object, NULL,\n+\t\t\t\t NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n@@ -2952,6 +2969,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use a bitmap index if available to speed up counting objects\")),\n \t\tOPT_BOOL(0, \"write-bitmap-index\", &write_bitmap_index,\n \t\t\t N_(\"write a bitmap index together with the pack index\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n+\t\tOPT_BOOL(0, \"filter-ignore-missing\", &arg_ignore_missing,\n+\t\t\t N_(\"ignore and omit missing objects from packfile\")),\n \t\tOPT_END(),\n \t};\n \n@@ -3028,6 +3048,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n+\tif (filter_options.choice) {\n+\t\tif (!pack_to_stdout)\n+\t\t\tdie(\"cannot use filtering with an indexable pack.\");\n+\t\tuse_bitmap_index = 0;\n+\t}\n+\n \t/*\n \t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n \t *\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nnew file mode 100755\nindex 0000000..4249557\n--- /dev/null\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -0,0 +1,369 @@\n+#!/bin/sh\n+\n+test_description='git pack-objects with object filtering for partial clone'\n+\n+. ./test-lib.sh\n+\n+# Test blob:none filter.\n+\n+test_expect_success 'setup r1' '\n+\techo \"{print \\$1}\" >print_1.awk &&\n+\techo \"{print \\$2}\" >print_2.awk &&\n+\n+\tgit init r1 &&\n+\tfor n in 1 2 3 4 5\n+\tdo\n+\t\techo \"This is file: $n\" > r1/file.$n\n+\t\tgit -C r1 add file.$n\n+\t\tgit -C r1 commit -m \"$n\"\n+\tdone\n+'\n+\n+test_expect_success 'verify blob count in normal packfile' '\n+\tgit -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r1 pack-objects --rev --stdout >all.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../all.pack &&\n+\tgit -C r1 verify-pack -v ../all.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:none packfile has no blobs' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=blob:none >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../filter.pack &&\n+\tgit -C r1 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\tnr=$(wc -l expected &&\n+\tgit -C r1 verify-pack -v ../filter.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Test blob:limit=[kmg] filter.\n+# We boundary test around the size parameter. The filter is strictly less than\n+# the value, so size 500 and 1000 should have the same results, but 1001 should\n+# filter more.\n+\n+test_expect_success 'setup r2' '\n+\tgit init r2 &&\n+\tfor n in 1000 10000\n+\tdo\n+\t\tprintf \"%\"$n\"s\" X > r2/large.$n\n+\t\tgit -C r2 add large.$n\n+\t\tgit -C r2 commit -m \"$n\"\n+\tdone\n+'\n+\n+test_expect_success 'verify blob count in normal packfile' '\n+\tgit -C r2 ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 pack-objects --rev --stdout >all.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../all.pack &&\n+\tgit -C r2 verify-pack -v ../all.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=500 omits all blobs' '\n+\tgit -C r2 pack-objects --rev --stdout --filter=blob:limit=500 >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../filter.pack &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\tnr=$(wc -l filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../filter.pack &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\tnr=$(wc -l expected &&\n+\tgit -C r2 pack-objects --rev --stdout --filter=blob:limit=1001 >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../filter.pack &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=10001' '\n+\tgit -C r2 ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 pack-objects --rev --stdout --filter=blob:limit=10001 >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../filter.pack &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=1k' '\n+\tgit -C r2 ls-files -s large.1000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 pack-objects --rev --stdout --filter=blob:limit=1k >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../filter.pack &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=1m' '\n+\tgit -C r2 ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 pack-objects --rev --stdout --filter=blob:limit=1m >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r2 index-pack ../filter.pack &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify normal and blob:limit packfiles have same commits/trees' '\n+\tgit -C r2 verify-pack -v ../all.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 verify-pack -v ../filter.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Test sparse:path= filter.\n+# Use a local file containing a sparse-checkout specification to filter\n+# out blobs not required for the corresponding sparse-checkout. We do not\n+# require sparse-checkout to actually be enabled.\n+\n+test_expect_success 'setup r3' '\n+\tgit init r3 &&\n+\tmkdir r3/dir1 &&\n+\tfor n in sparse1 sparse2\n+\tdo\n+\t\techo \"This is file: $n\" > r3/$n\n+\t\tgit -C r3 add $n\n+\t\techo \"This is file: dir1/$n\" > r3/dir1/$n\n+\t\tgit -C r3 add dir1/$n\n+\tdone &&\n+\tgit -C r3 commit -m \"sparse\" &&\n+\techo dir1/ >pattern1 &&\n+\techo sparse1 >pattern2\n+'\n+\n+test_expect_success 'verify blob count in normal packfile' '\n+\tgit -C r3 ls-files -s sparse1 sparse2 dir1/sparse1 dir1/sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 pack-objects --rev --stdout >all.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r3 index-pack ../all.pack &&\n+\tgit -C r3 verify-pack -v ../all.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify sparse:path=pattern1' '\n+\tgit -C r3 ls-files -s dir1/sparse1 dir1/sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 pack-objects --rev --stdout --filter=sparse:path=../pattern1 >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r3 index-pack ../filter.pack &&\n+\tgit -C r3 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify normal and sparse:path=pattern1 packfiles have same commits/trees' '\n+\tgit -C r3 verify-pack -v ../all.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 verify-pack -v ../filter.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify sparse:path=pattern2' '\n+\tgit -C r3 ls-files -s sparse1 dir1/sparse1 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 pack-objects --rev --stdout --filter=sparse:path=../pattern2 >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r3 index-pack ../filter.pack &&\n+\tgit -C r3 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify normal and sparse:path=pattern2 packfiles have same commits/trees' '\n+\tgit -C r3 verify-pack -v ../all.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 verify-pack -v ../filter.pack \\\n+\t\t| grep -E \"commit|tree\" \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Test sparse:oid= filter.\n+# Like sparse:path, but we get the sparse-checkout specification from\n+# a blob rather than a file on disk.\n+\n+test_expect_success 'setup r4' '\n+\tgit init r4 &&\n+\tmkdir r4/dir1 &&\n+\tfor n in sparse1 sparse2\n+\tdo\n+\t\techo \"This is file: $n\" > r4/$n\n+\t\tgit -C r4 add $n\n+\t\techo \"This is file: dir1/$n\" > r4/dir1/$n\n+\t\tgit -C r4 add dir1/$n\n+\tdone &&\n+\techo dir1/ >r4/pattern &&\n+\tgit -C r4 add pattern &&\n+\tgit -C r4 commit -m \"pattern\"\n+'\n+\n+test_expect_success 'verify blob count in normal packfile' '\n+\tgit -C r4 ls-files -s pattern sparse1 sparse2 dir1/sparse1 dir1/sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r4 pack-objects --rev --stdout >all.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r4 index-pack ../all.pack &&\n+\tgit -C r4 verify-pack -v ../all.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify sparse:oid=OID' '\n+\tgit -C r4 ls-files -s dir1/sparse1 dir1/sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\toid=$(git -C r4 ls-files -s pattern | awk -f print_2.awk) &&\n+\tgit -C r4 pack-objects --rev --stdout --filter=sparse:oid=$oid >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r4 index-pack ../filter.pack &&\n+\tgit -C r4 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify sparse:oid=oid-ish' '\n+\tgit -C r4 ls-files -s dir1/sparse1 dir1/sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r4 pack-objects --rev --stdout --filter=sparse:oid=master:pattern >filter.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r4 index-pack ../filter.pack &&\n+\tgit -C r4 verify-pack -v ../filter.pack \\\n+\t\t| grep blob \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Delete some loose objects and use pack-objects, but WITHOUT any filtering.\n+# This models previously omitted objects that we did not receive.\n+\n+test_expect_success 'setup r1 - delete loose blobs' '\n+\tgit -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tfor id in `cat expected | sed \"s|..|&/|\"`\n+\tdo\n+\t\trm r1/.git/objects/$id\n+\tdone\n+'\n+\n+test_expect_success 'verify pack-objects fails w/ missing objects' '\n+\ttest_must_fail git -C r1 pack-objects --rev --stdout >miss.pack <<-EOF\n+\tHEAD\n+\tEOF\n+'\n+\n+test_expect_success 'verify pack-objects w/ ignore-missing' '\n+\tgit -C r1 pack-objects --rev --stdout --filter-ignore-missing >miss.pack <<-EOF\n+\tHEAD\n+\tEOF\n+'\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"331678","messageId":"20171102175013.3371-6-git@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"[PATCH v2 5/6] rev-list: add list-objects filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:12Z","receivedAt":"2017-11-02T17:50:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach rev-list to use the filtering provided by the\ntraverse_commit_list_filtered() interface to omit\nunwanted objects from the result. This feature is\nintended to help with partial clone.\n\nObject filtering is only allowed when one of the \"--objects*\"\noptions are used.\n\nWhen the \"--filter-print-omitted\" option is used, the omitted\nobjects are printed at the end. These are marked with a \"~\".\nThis option can be combined with \"--quiet\" to get a list of\njust the omitted objects.\n\nNormally, rev-list will stop with an error when there are\nmissing objects.\n\nWhen the \"--filter-print-missing\" option is used, rev-list\nwill print a list of any missing objects that should have\nbeen included in the output (rather than stopping).\nThese are marked with a \"?\".\n\nWhen the \"--filter-ignore-missing\" option is used, rev-list\nwill silently ignore any missing objects and continue without\nerror.\n\nAdd t6112 test.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-rev-list.txt | 6 +-\n Documentation/rev-list-options.txt | 34 ++++++\n builtin/rev-list.c | 75 +++++++++++-\n t/t6112-rev-list-filters-objects.sh | 225 ++++++++++++++++++++++++++++++++++++\n 4 files changed, 337 insertions(+), 3 deletions(-)\n create mode 100755 t/t6112-rev-list-filters-objects.sh\n\ndiff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt\nindex ef22f17..b8a3a5b 100644\n--- a/Documentation/git-rev-list.txt\n+++ b/Documentation/git-rev-list.txt\n@@ -47,7 +47,11 @@ SYNOPSIS\n \t [ --fixed-strings | -F ]\n \t [ --date=]\n \t [ [ --objects | --objects-edge | --objects-edge-aggressive ]\n-\t [ --unpacked ] ]\n+\t [ --unpacked ]\n+\t [ --filter= ] ]\n+\t [ --filter-print-missing ]\n+\t [ --filter-print-omitted ]\n+\t [ --filter-ignore-missing ]\n \t [ --pretty | --header ]\n \t [ --bisect ]\n \t [ --bisect-vars ]\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 13501e1..9233134 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -706,6 +706,40 @@ ifdef::git-rev-list[]\n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\n \tin packs.\n+\n+--filter=::\n+\tOnly useful with one of the `--objects*`; omits objects (usually\n+\tblobs) from the list of printed objects. The ''\n+\tmay be one of the following:\n++\n+The form '--filter=blob:none' omits all blobs.\n++\n+The form '--filter=blob:limit=[kmg]' omits blobs larger than n bytes\n+or units. The value may be zero. Special files matching '.git*' are\n+alwayse included, regardless of size.\n++\n+The form '--filter=sparse:oid=' uses a sparse-checkout\n+specification contained in the object (or the object that the expression\n+evaluates to) to omit blobs not required by the corresponding sparse\n+checkout.\n++\n+The form '--filter=sparse:path=' similarly uses a sparse-checkout\n+specification contained in .\n+\n+--filter-print-missing::\n+\tPrints a list of the missing objects for the requested traversal.\n+\tObject IDs are prefixed with a ``?'' character. The object type\n+\tis printed after the ID. This may be used with or without any of\n+\tthe above filtering options.\n+\n+--filter-ignore-missing::\n+\tIgnores missing objects encountered during the requested traversal.\n+\tThis may be used with or without any of the above filtering options.\n+\n+--filter-print-omitted::\n+\tOnly useful with one of the above `--filter*`; prints a list\n+\tof the omitted objects. Object IDs are prefixed with a ``~''\n+\tcharacter.\n endif::git-rev-list[]\n \n --no-walk[=(sorted|unsorted)]::\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex c1c74d4..cc9fa40 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -4,6 +4,8 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"builtin.h\"\n@@ -12,6 +14,7 @@\n #include \"bisect.h\"\n #include \"progress.h\"\n #include \"reflog-walk.h\"\n+#include \"oidset.h\"\n \n static const char rev_list_usage[] =\n \"git rev-list [OPTION] ... [ -- paths... ]\\n\"\n@@ -54,6 +57,15 @@ static const char rev_list_usage[] =\n \n static struct progress *progress;\n static unsigned progress_counter;\n+static struct list_objects_filter_options filter_options;\n+static struct oidset missing_objects;\n+static struct oidset omitted_objects;\n+static int arg_print_missing;\n+static int arg_print_omitted;\n+static int arg_ignore_missing;\n+\n+#define DEFAULT_OIDSET_SIZE (16*1024)\n+\n \n static void finish_commit(struct commit *commit, void *data);\n static void show_commit(struct commit *commit, void *data)\n@@ -181,8 +193,16 @@ static void finish_commit(struct commit *commit, void *data)\n static void finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid))\n+\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\t\tif (arg_print_missing) {\n+\t\t\toidset_insert(&missing_objects, &obj->oid);\n+\t\t\treturn;\n+\t\t}\n+\t\tif (arg_ignore_missing)\n+\t\t\treturn;\n+\n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(&obj->oid);\n }\n@@ -335,6 +355,30 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n+\n+\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\t\tparse_list_objects_filter(&filter_options, arg);\n+\t\t\tif (filter_options.choice && !revs.blob_objects)\n+\t\t\t\tdie(_(\"object filtering requires --objects\"));\n+\t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n+\t\t\t !filter_options.sparse_oid_value)\n+\t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n+\t\t\t\t filter_options.raw_value);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--filter-print-missing\")) {\n+\t\t\targ_print_missing = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n+\t\t\targ_print_omitted = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--filter-ignore-missing\")) {\n+\t\t\targ_ignore_missing = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\t\n \t\tusage(rev_list_usage);\n \n \t}\n@@ -360,6 +404,9 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (revs.show_notes)\n \t\tdie(_(\"rev-list does not support display of notes\"));\n \n+\tif (filter_options.choice && use_bitmap_index)\n+\t\tdie(_(\"cannot combine --use-bitmap-index with object filtering\"));\n+\n \tsave_commit_buffer = (revs.verbose_header ||\n \t\t\t revs.grep_filter.pattern_list ||\n \t\t\t revs.grep_filter.header_list);\n@@ -404,7 +451,31 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n-\ttraverse_commit_list(&revs, show_commit, show_object, &info);\n+\tif (arg_print_missing)\n+\t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n+\tif (arg_print_omitted)\n+\t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n+\n+\ttraverse_commit_list_filtered(\n+\t\t&filter_options, &revs, show_commit, show_object, &info,\n+\t\t(arg_print_omitted ? &omitted_objects : NULL));\n+\n+\tif (arg_print_omitted) {\n+\t\tstruct oidset_iter iter;\n+\t\tstruct object_id *oid;\n+\t\toidset_iter_init(&omitted_objects, &iter);\n+\t\twhile ((oid = oidset_iter_next(&iter)))\n+\t\t\tprintf(\"~%s\\n\", oid_to_hex(oid));\n+\t\toidset_clear(&omitted_objects);\n+\t}\n+\tif (arg_print_missing) {\n+\t\tstruct oidset_iter iter;\n+\t\tstruct object_id *oid;\n+\t\toidset_iter_init(&missing_objects, &iter);\n+\t\twhile ((oid = oidset_iter_next(&iter)))\n+\t\t\tprintf(\"?%s\\n\", oid_to_hex(oid));\n+\t\toidset_clear(&missing_objects);\n+\t}\n \n \tstop_progress(&progress);\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nnew file mode 100755\nindex 0000000..9327974\n--- /dev/null\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -0,0 +1,225 @@\n+#!/bin/sh\n+\n+test_description='git rev-list with object filtering for partial clone'\n+\n+. ./test-lib.sh\n+\n+# Test the blob:none filter.\n+\n+test_expect_success 'setup r1' '\n+\techo \"{print \\$1}\" >print_1.awk &&\n+\techo \"{print \\$2}\" >print_2.awk &&\n+\n+\tgit init r1 &&\n+\tfor n in 1 2 3 4 5\n+\tdo\n+\t\techo \"This is file: $n\" > r1/file.$n\n+\t\tgit -C r1 add file.$n\n+\t\tgit -C r1 commit -m \"$n\"\n+\tdone\n+'\n+\n+test_expect_success 'verify blob:none omits all 5 blobs' '\n+\tgit -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r1 rev-list HEAD --quiet --objects --filter-print-omitted --filter=blob:none \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify emitted+omitted == all' '\n+\tgit -C r1 rev-list HEAD --objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r1 rev-list HEAD --objects --filter-print-omitted --filter=blob:none \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+\n+# Test blob:limit=[kmg] filter.\n+# We boundary test around the size parameter. The filter is strictly less than\n+# the value, so size 500 and 1000 should have the same results, but 1001 should\n+# filter more.\n+\n+test_expect_success 'setup r2' '\n+\tgit init r2 &&\n+\tfor n in 1000 10000\n+\tdo\n+\t\tprintf \"%\"$n\"s\" X > r2/large.$n\n+\t\tgit -C r2 add large.$n\n+\t\tgit -C r2 commit -m \"$n\"\n+\tdone\n+'\n+\n+test_expect_success 'verify blob:limit=500 omits all blobs' '\n+\tgit -C r2 ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 rev-list HEAD --quiet --objects --filter-print-omitted --filter=blob:limit=500 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify emitted+omitted == all' '\n+\tgit -C r2 rev-list HEAD --objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 rev-list HEAD --objects --filter-print-omitted --filter=blob:limit=500 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=1000' '\n+\tgit -C r2 ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 rev-list HEAD --quiet --objects --filter-print-omitted --filter=blob:limit=1000 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=1001' '\n+\tgit -C r2 ls-files -s large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 rev-list HEAD --quiet --objects --filter-print-omitted --filter=blob:limit=1001 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=1k' '\n+\tgit -C r2 ls-files -s large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r2 rev-list HEAD --quiet --objects --filter-print-omitted --filter=blob:limit=1k \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify blob:limit=1m' '\n+\tcat expected &&\n+\tgit -C r2 rev-list HEAD --quiet --objects --filter-print-omitted --filter=blob:limit=1m \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Test sparse:path= filter.\n+# Use a local file containing a sparse-checkout specification to filter\n+# out blobs not required for the corresponding sparse-checkout. We do not\n+# require sparse-checkout to actually be enabled.\n+\n+test_expect_success 'setup r3' '\n+\tgit init r3 &&\n+\tmkdir r3/dir1 &&\n+\tfor n in sparse1 sparse2\n+\tdo\n+\t\techo \"This is file: $n\" > r3/$n\n+\t\tgit -C r3 add $n\n+\t\techo \"This is file: dir1/$n\" > r3/dir1/$n\n+\t\tgit -C r3 add dir1/$n\n+\tdone &&\n+\tgit -C r3 commit -m \"sparse\" &&\n+\techo dir1/ >pattern1 &&\n+\techo sparse1 >pattern2\n+'\n+\n+test_expect_success 'verify sparse:path=pattern1 omits top-level files' '\n+\tgit -C r3 ls-files -s sparse1 sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=sparse:path=../pattern1 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify sparse:path=pattern2 omits both sparse2 files' '\n+\tgit -C r3 ls-files -s sparse2 dir1/sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=sparse:path=../pattern2 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Test sparse:oid= filter.\n+# Like sparse:path, but we get the sparse-checkout specification from\n+# a blob rather than a file on disk.\n+\n+test_expect_success 'setup r3 part 2' '\n+\techo dir1/ >r3/pattern &&\n+\tgit -C r3 add pattern &&\n+\tgit -C r3 commit -m \"pattern\"\n+'\n+\n+test_expect_success 'verify sparse:oid=OID omits top-level files' '\n+\tgit -C r3 ls-files -s pattern sparse1 sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\toid=$(git -C r3 ls-files -s pattern | awk -f print_2.awk) &&\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=sparse:oid=$oid \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n+\tgit -C r3 ls-files -s pattern sparse1 sparse2 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=sparse:oid=master:pattern \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Delete some loose objects and use rev-list, but WITHOUT any filtering.\n+# This models previously omitted objects that we did not receive.\n+\n+test_expect_success 'rev-list W/ print-missing' '\n+\tgit -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tfor id in `cat expected | sed \"s|..|&/|\"`\n+\tdo\n+\t\trm r1/.git/objects/$id\n+\tdone &&\n+\tgit -C r1 rev-list --quiet HEAD --filter-print-missing --objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/?//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'rev-list W/O print-missing fails' '\n+\ttest_must_fail git -C r1 rev-list --quiet --objects HEAD\n+'\n+\n+test_expect_success 'rev-list W/ ignore-missing' '\n+\tgit -C r1 rev-list --quiet --filter-ignore-missing --objects HEAD\n+'\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"331679","messageId":"20171102175013.3371-4-git@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"[PATCH v2 3/6] oidset: add iterator methods to oidset","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T17:50:10Z","receivedAt":"2017-11-02T17:50:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the usual iterator methods to oidset.\nAdd oidset_remove().\n\nSigned-off-by: Jeff Hostetler \n---\n oidset.c | 10 ++++++++++\n oidset.h | 36 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+)\n\ndiff --git a/oidset.c b/oidset.c\nindex f1f874a..454c54f 100644\n--- a/oidset.c\n+++ b/oidset.c\n@@ -24,6 +24,16 @@ int oidset_insert(struct oidset *set, const struct object_id *oid)\n \treturn 0;\n }\n \n+int oidset_remove(struct oidset *set, const struct object_id *oid)\n+{\n+\tstruct oidmap_entry *entry;\n+\n+\tentry = oidmap_remove(&set->map, oid);\n+\tfree(entry);\n+\n+\treturn (entry != NULL);\n+}\n+\n void oidset_clear(struct oidset *set)\n {\n \toidmap_free(&set->map, 1);\ndiff --git a/oidset.h b/oidset.h\nindex f4c9e0f..783abce 100644\n--- a/oidset.h\n+++ b/oidset.h\n@@ -24,6 +24,12 @@ struct oidset {\n \n #define OIDSET_INIT { OIDMAP_INIT }\n \n+\n+static inline void oidset_init(struct oidset *set, size_t initial_size)\n+{\n+\treturn oidmap_init(&set->map, initial_size);\n+}\n+\n /**\n * Returns true iff `set` contains `oid`.\n */\n@@ -39,9 +45,39 @@ int oidset_contains(const struct oidset *set, const struct object_id *oid);\n int oidset_insert(struct oidset *set, const struct object_id *oid);\n \n /**\n+ * Remove the oid from the set.\n+ *\n+ * Returns 1 if the oid was present in the set, 0 otherwise.\n+ */\n+int oidset_remove(struct oidset *set, const struct object_id *oid);\n+\n+/**\n * Remove all entries from the oidset, freeing any resources associated with\n * it.\n */\n void oidset_clear(struct oidset *set);\n \n+struct oidset_iter {\n+\tstruct oidmap_iter m_iter;\n+};\n+\n+static inline void oidset_iter_init(struct oidset *set,\n+\t\t\t\t struct oidset_iter *iter)\n+{\n+\toidmap_iter_init(&set->map, &iter->m_iter);\n+}\n+\n+static inline struct object_id *oidset_iter_next(struct oidset_iter *iter)\n+{\n+\tstruct oidmap_entry *e = oidmap_iter_next(&iter->m_iter);\n+\treturn e ? &e->oid : NULL;\n+}\n+\n+static inline struct object_id *oidset_iter_first(struct oidset *set,\n+\t\t\t\t\t\t struct oidset_iter *iter)\n+{\n+\toidset_iter_init(set, iter);\n+\treturn oidset_iter_next(iter);\n+}\n+\n #endif /* OIDSET_H */\n-- \n2.9.3\n\n"},{"id":"331689","messageId":"20171102123245.0f768968703ec4e35d3d1f81@google.com","threadId":"47102","inReplyTo":"20171102175013.3371-5-git@jeffhostetler.com","subject":"Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-02T19:32:45Z","receivedAt":"2017-11-02T19:32:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 2 Nov 2017 17:50:11 +0000\nJeff Hostetler wrote:\n\n> +int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n> +\t\t\t const char *arg)\n\nReturning void is fine, I think. It seems that all your code paths\neither return 0 or die.\n\n> +{\n> +\tstruct object_context oc;\n> +\tstruct object_id sparse_oid;\n> +\tconst char *v0;\n> +\tconst char *v1;\n> +\n> +\tif (filter_options->choice)\n> +\t\tdie(_(\"multiple object filter types cannot be combined\"));\n> +\n> +\t/*\n> +\t * TODO consider rejecting 'arg' if it contains any\n> +\t * TODO injection characters (since we might send this\n> +\t * TODO to a sub-command or to the server and we don't\n> +\t * TODO want to deal with legacy quoting/escaping for\n> +\t * TODO a new feature).\n> +\t */\n> +\n> +\tfilter_options->raw_value = strdup(arg);\n> +\n> +\tif (skip_prefix(arg, \"blob:\", &v0) || skip_prefix(arg, \"blobs:\", &v0)) {\n> +\t\tif (!strcmp(v0, \"none\")) {\n> +\t\t\tfilter_options->choice = LOFC_BLOB_NONE;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\n> +\t\tif (skip_prefix(v0, \"limit=\", &v1) &&\n> +\t\t git_parse_ulong(v1, &filter_options->blob_limit_value)) {\n> +\t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n> +\telse if (skip_prefix(arg, \"sparse:\", &v0)) {\n\nStyle: join the 2 lines above.\n\n> +\t\tif (skip_prefix(v0, \"oid=\", &v1)) {\n> +\t\t\tfilter_options->choice = LOFC_SPARSE_OID;\n> +\t\t\tif (!get_oid_with_context(v1, GET_OID_BLOB,\n> +\t\t\t\t\t\t &sparse_oid, &oc)) {\n> +\t\t\t\t/*\n> +\t\t\t\t * We successfully converted the \n> +\t\t\t\t * into an actual OID. Rewrite the raw_value\n> +\t\t\t\t * in canonoical form with just the OID.\n> +\t\t\t\t * (If we send this request to the server, we\n> +\t\t\t\t * want an absolute expression rather than a\n> +\t\t\t\t * local-ref-relative expression.)\n> +\t\t\t\t */\n\nI think this would lead to confusing behavior - for example, a fetch\nwith \"--filter=oid=mybranch:sparseconfig\" would have different results\ndepending on whether \"mybranch\" refers to a valid object locally.\n\nThe way I see it, this should either (i) only accept full 40-character\nOIDs or (ii) retain the raw string to be interpreted only when the\nfiltering is done. (i) is simpler and safer, but is not so useful. In\nboth cases, if the user really wants client-side interpretation, they\ncan still use \"$(git rev-parse foo)\" to make it explicit.\n\n> +\t\t\t\tfree((char *)filter_options->raw_value);\n> +\t\t\t\tfilter_options->raw_value =\n> +\t\t\t\t\txstrfmt(\"sparse:oid=%s\",\n> +\t\t\t\t\t\toid_to_hex(&sparse_oid));\n> +\t\t\t\tfilter_options->sparse_oid_value =\n> +\t\t\t\t\toiddup(&sparse_oid);\n> +\t\t\t} else {\n> +\t\t\t\t/*\n> +\t\t\t\t * We could not turn the into an\n> +\t\t\t\t * OID. Leave the raw_value as is in case\n> +\t\t\t\t * the server can parse it. (It may refer to\n> +\t\t\t\t * a branch, commit, or blob we don't have.)\n> +\t\t\t\t */\n> +\t\t\t}\n> +\t\t\treturn 0;\n> +\t\t}\n> +\n> +\t\tif (skip_prefix(v0, \"path=\", &v1)) {\n> +\t\t\tfilter_options->choice = LOFC_SPARSE_PATH;\n> +\t\t\tfilter_options->sparse_path_value = strdup(v1);\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n> +\n> +\tdie(_(\"invalid filter expression '%s'\"), arg);\n> +\treturn 0;\n> +}\n\n[snip]\n\n> +void arg_format_list_objects_filter(\n> +\tstruct argv_array *argv_array,\n> +\tconst struct list_objects_filter_options *filter_options)\n\nIs this function used anywhere (in this patch or subsequent patches)?\n\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> +/* See object.h and revision.h */\n> +#define FILTER_REVISIT (1<<25)\n\nLooking later in the code, this flag indicates that a tree has been\nSHOWN, so it might be better to just call this FILTER_SHOWN.\n\nAlso document this flag in object.h in the table above FLAG_BITS.\n\n> +static enum list_objects_filter_result filter_blobs_limit(\n> +\tenum list_objects_filter_type filter_type,\n> +\tstruct object *obj,\n> +\tconst char *pathname,\n> +\tconst char *filename,\n> +\tvoid *filter_data_)\n> +{\n> +\tstruct filter_blobs_limit_data *filter_data = filter_data_;\n> +\tunsigned long object_length;\n> +\tenum object_type t;\n> +\tint is_special_filename;\n> +\n> +\tswitch (filter_type) {\n> +\tdefault:\n> +\t\tdie(\"unkown filter_type\");\n\nSpelling of \"unknown\" (also elsewhere in this file).\n\n> +\t\treturn LOFR_ZERO;\n> +\n> +\tcase LOFT_BEGIN_TREE:\n> +\t\tassert(obj->type == OBJ_TREE);\n> +\t\t/* always include all tree objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n> +\n> +\tcase LOFT_END_TREE:\n> +\t\tassert(obj->type == OBJ_TREE);\n> +\t\treturn LOFR_ZERO;\n> +\n> +\tcase LOFT_BLOB:\n> +\t\tassert(obj->type == OBJ_BLOB);\n> +\t\tassert((obj->flags & SEEN) == 0);\n> +\n> +\t\tis_special_filename = ((strncmp(filename, \".git\", 4) == 0) &&\n> +\t\t\t\t filename[4]);\n\nYou can just use starts_with(). Also, the filename[4] check is probably\nunnecessary.\n\n> +\t\tif (is_special_filename) {\n> +\t\t\t/*\n> +\t\t\t * Alwayse include \".git*\" special files (regardless\n> +\t\t\t * of size).\n> +\t\t\t *\n> +\t\t\t * (This may cause us to include blobs that we do\n> +\t\t\t * not have locally because we are only looking at\n> +\t\t\t * the filename and don't actually have to read\n> +\t\t\t * them.)\n> +\t\t\t */\n> +\t\t\tgoto include_it;\n> +\t\t}\n> +\n> +\t\tt = sha1_object_info(obj->oid.hash, &object_length);\n> +\t\tif (t != OBJ_BLOB) { /* probably OBJ_NONE */\n> +\t\t\t/*\n> +\t\t\t * We DO NOT have the blob locally, so we cannot\n> +\t\t\t * apply the size filter criteria. Be conservative\n> +\t\t\t * and force show it (and let the caller deal with\n> +\t\t\t * the ambiguity). (This matches the behavior above\n> +\t\t\t * when the special filename matches.)\n> +\t\t\t */\n> +\t\t\tgoto include_it;\n> +\t\t}\n> +\n> +\t\tif (object_length < filter_data->max_bytes)\n> +\t\t\tgoto include_it;\n> +\n> +\t\t/*\n> +\t\t * Provisionally omit it. We've already established\n> +\t\t * that this blob is too big and doesn't have a special\n> +\t\t * filename, so we *WANT* to omit it. However, there\n> +\t\t * may be a special file elsewhere in the tree that\n> +\t\t * references this same blob, so we cannot reject it\n> +\t\t * just yet. Leave the LOFR_ bits unset so that *IF*\n> +\t\t * the blob appears again in the traversal, we will\n> +\t\t * be asked again.\n> +\t\t *\n> +\t\t * If we are keeping a list of the ommitted objects,\n> +\t\t * provisionally add it to the list.\n> +\t\t */\n> +\n> +\t\tif (filter_data->omits)\n> +\t\t\toidset_insert(filter_data->omits, &obj->oid);\n> +\t\treturn LOFR_ZERO;\n> +\t}\n> +\n> +include_it:\n> +\tif (filter_data->omits)\n> +\t\toidset_remove(filter_data->omits, &obj->oid);\n> +\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n> +}\n\n[snip]\n\n> +struct frame {\n> +\tint defval;\n\nDocument this variable?\n\n> +\tint child_prov_omit : 1;\n\nI think it's clearer if we use \"unsigned\" here. Also, document this\n(e.g. \"1 if any descendant of this tree object was provisionally\nomitted\").\n\n> +/*\n> + * During list-object traversal we allow certain objects to be\n> + * filtered (omitted) from the result. The active filter uses\n> + * these result values to guide list-objects.\n> + *\n> + * _ZERO : Do nothing with the object at this time. It may\n> + * be revisited if it appears in another place in\n> + * the tree or in another commit during the overall\n> + * traversal.\n> + *\n> + * _MARK_SEEN : Mark this object as \"SEEN\" in the object flags.\n> + * This will prevent it from being revisited during\n> + * the remainder of the traversal. This DOES NOT\n> + * imply that it will be included in the results.\n> + *\n> + * _SHOW : Show this object in the results (call show() on it).\n> + * In general, objects should only be shown once, but\n> + * this result DOES NOT imply that we mark it SEEN.\n> + *\n> + * Most of the time, you want the combination (_MARK_SEEN | _SHOW)\n> + * but they can be used independently, such as when sparse-checkout\n> + * pattern matching is being applied.\n> + *\n> + * A _MARK_SEEN without _SHOW can be called a hard-omit -- the\n> + * object is not shown and will never be reconsidered (unless a\n> + * previous iteration has already shown it).\n> + *\n> + * A _ZERO is can be called a provisional-omit -- the object is\n> + * not shown, but *may* be revisited (if the object appears again\n> + * in the traversal). Therefore, it will be omitted from the\n> + * results *unless* a later iteration causes it to be shown.\n> + */\n> +enum list_objects_filter_result {\n> +\tLOFR_ZERO = 0,\n> +\tLOFR_MARK_SEEN = 1<<0,\n> +\tLOFR_SHOW = 1<<1,\n> +};\n\nThanks - this looks like a good explanation to me.\n\n> +enum list_objects_filter_type {\n> +\tLOFT_BEGIN_TREE,\n> +\tLOFT_END_TREE,\n> +\tLOFT_BLOB\n> +};\n\nOptional: probably a better name would be list_objects_filter_situation.\n\n> +void traverse_commit_list_filtered(\n> +\tstruct list_objects_filter_options *filter_options,\n> +\tstruct rev_info *revs,\n> +\tshow_commit_fn show_commit,\n> +\tshow_object_fn show_object,\n> +\tvoid *show_data,\n> +\tstruct oidset *omitted)\n> +{\n> +\tfilter_object_fn filter_fn = NULL;\n> +\tfilter_free_fn filter_free_fn = NULL;\n> +\tvoid *filter_data = NULL;\n> +\n> +\tfilter_data = list_objects_filter__init(omitted, filter_options,\n> +\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n> +\tdo_traverse(revs, show_commit, show_object, show_data,\n> +\t\t filter_fn, filter_data);\n> +\tif (filter_data && filter_free_fn)\n> +\t\tfilter_free_fn(filter_data);\n> +}\n\nThis function traverse_commit_list_filtered() is in list-objects.c but\nin list-objects-filter.h, if I'm reading the diff correctly?\n\nOverall, this looks like a good change. Object traversal was upgraded\nwith the behaviors of MARK_SEEN and SHOW independently controllable and\nwith the ability to do things post-tree (in addition to pre-tree and\nblob), and this was used to support a few types of filtering, which\nsubsequent patches will allow the user to invoke through \"--filter=\".\n"},{"id":"331698","messageId":"20171102124445.fbffd43521cd35f6a71e1851@google.com","threadId":"47102","inReplyTo":"20171102175013.3371-1-git@jeffhostetler.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-02T19:44:45Z","receivedAt":"2017-11-02T19:44:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 2 Nov 2017 17:50:07 +0000\nJeff Hostetler wrote:\n\n> From: Jeff Hostetler \n> \n> Here is V2 of the list-object filtering. It replaces [1]\n> and reflect a refactoring and simplification of the original.\n\nThanks, overall this looks quite good. I reviewed patches 2-6 (skipping\n1 since it's already in next), made my comments on 4, and don't have any\nfor the rest (besides what's below).\n\n> I've added \"--filter-ignore-missing\" parameter to rev-list and\n> pack-objects to ignore missing objects rather than error out.\n> This allows this patch series to better stand on its own eliminates\n> the need in part 1 for \"patch 9\" from V1.\n> \n> This is a brute force ignore all missing objects. Later, in part\n> 2 or part 3 when --exclude-promisor-objects is introduced, we will\n> be able to ignore EXPECTED missing objects.\n\n(This is regarding patches 5 and 6.) Is the intention to support both\nflags? (That is, --ignore-missing to ignore without checking whether the\nobject being missing is not unexpected, and --exclude-promisor-objects\nto check and ignore.)\n"},{"id":"331771","messageId":"alpine.DEB.2.21.1.1711031254280.6482@virtualbox","threadId":"47102","inReplyTo":"20171102123245.0f768968703ec4e35d3d1f81@google.com","subject":"Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-11-03T11:54:56Z","receivedAt":"2017-11-03T11:55:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jonathan,\n\nOn Thu, 2 Nov 2017, Jonathan Tan wrote:\n\n> On Thu, 2 Nov 2017 17:50:11 +0000\n> Jeff Hostetler wrote:\n> \n> > +int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n> > +\t\t\t const char *arg)\n> \n> Returning void is fine, I think. It seems that all your code paths\n> either return 0 or die.\n\nCan we please start to encourage libified code, rather than discourage it?\n\nThank you,\nDscho\n"},{"id":"331776","messageId":"0dd48f07-6174-1a4e-c1f0-bae5b18cebd1@jeffhostetler.com","threadId":"47102","inReplyTo":"alpine.DEB.2.21.1.1711031254280.6482@virtualbox","subject":"Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-03T13:37:43Z","receivedAt":"2017-11-03T13:37:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/3/2017 7:54 AM, Johannes Schindelin wrote:\n> Hi Jonathan,\n> \n> On Thu, 2 Nov 2017, Jonathan Tan wrote:\n> \n>> On Thu, 2 Nov 2017 17:50:11 +0000\n>> Jeff Hostetler wrote:\n>>\n>>> +int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n>>> +\t\t\t const char *arg)\n>>\n>> Returning void is fine, I think. It seems that all your code paths\n>> either return 0 or die.\n> \n> Can we please start to encourage libified code, rather than discourage it?\n\n\nI did that so that I could call it from the opt_parse_... version below\nit that is used by the OPT_ macros.\n\nAnd Johannes is right, it bothers me that there doesn't seem to be a hard\nline where one should or should not call die() vs returning an error code.\n\nJeff\n\n"},{"id":"331778","messageId":"bd41699f-19c7-02a0-0af1-8f22b3277a1b@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102124445.fbffd43521cd35f6a71e1851@google.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-03T13:43:21Z","receivedAt":"2017-11-03T13:43:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/2/2017 3:44 PM, Jonathan Tan wrote:\n> On Thu, 2 Nov 2017 17:50:07 +0000\n> Jeff Hostetler wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Here is V2 of the list-object filtering. It replaces [1]\n>> and reflect a refactoring and simplification of the original.\n> \n> Thanks, overall this looks quite good. I reviewed patches 2-6 (skipping\n> 1 since it's already in next), made my comments on 4, and don't have any\n> for the rest (besides what's below).\n> \n>> I've added \"--filter-ignore-missing\" parameter to rev-list and\n>> pack-objects to ignore missing objects rather than error out.\n>> This allows this patch series to better stand on its own eliminates\n>> the need in part 1 for \"patch 9\" from V1.\n>>\n>> This is a brute force ignore all missing objects. Later, in part\n>> 2 or part 3 when --exclude-promisor-objects is introduced, we will\n>> be able to ignore EXPECTED missing objects.\n> \n> (This is regarding patches 5 and 6.) Is the intention to support both\n> flags? (That is, --ignore-missing to ignore without checking whether the\n> object being missing is not unexpected, and --exclude-promisor-objects\n> to check and ignore.)\n> \n\nYes, I thought we should have both (perhaps renamed or combined\ninto 1 parameter with value, such as --exclude=missing vs --exclude=promisor)\nand let the user decide how strict they want to be.\n\nJeff\n\n"},{"id":"331784","messageId":"xmqq7ev7pfln.fsf@gitster.mtv.corp.google.com","threadId":"47102","inReplyTo":"bd41699f-19c7-02a0-0af1-8f22b3277a1b@jeffhostetler.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-03T15:05:40Z","receivedAt":"2017-11-03T15:05:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> Yes, I thought we should have both (perhaps renamed or combined\n> into 1 parameter with value, such as --exclude=missing vs --exclude=promisor)\n> and let the user decide how strict they want to be.\n\nAssuming we eventually get promisor support working, would there be\nany use case where \"any missing is OK\" mode would be useful in a\nsense more reasonable than \"because we could have such a mode\" and\n\"it is not our business to prevent users from playing with fire\"?\n"},{"id":"331796","messageId":"97585434-6d57-dad4-d7aa-e48dacec1b3f@jeffhostetler.com","threadId":"47102","inReplyTo":"xmqq7ev7pfln.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-03T18:34:39Z","receivedAt":"2017-11-03T18:34:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/3/2017 11:05 AM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n> \n>> Yes, I thought we should have both (perhaps renamed or combined\n>> into 1 parameter with value, such as --exclude=missing vs --exclude=promisor)\n>> and let the user decide how strict they want to be.\n> \n> Assuming we eventually get promisor support working, would there be\n> any use case where \"any missing is OK\" mode would be useful in a\n> sense more reasonable than \"because we could have such a mode\" and\n> \"it is not our business to prevent users from playing with fire\"?\n> \n\nFor now, I'd like to keep my \"any missing is OK\" option.\nI do think it has value all by itself.\n\nWe are essentially using something like that now with our GVFS\nusers on the gigantic Windows repo and haven't had any issues.\n\nBut yes, when we get promisor support working, we could revisit\nthe need for this parameter.\n\nHowever, I do have some scaling concerns here. If for example,\nI have 100M missing blobs (because we did an only commits-and-trees\nclone), the cost to compute \"promisor missing\" vs \"just missing\"\nmight be prohibitively expensive. It could be something we want\nfsck/gc to be aware of, but other commands may want to just assume\nany missing objects are expected and continue.\n\nHopefully, we won't have a scale problem, but we just don't know\nyet.\n\nJeff\n"},{"id":"331931","messageId":"68f3c18f-8bee-dbb7-26f3-c1070d9177ac@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102175013.3371-5-git@jeffhostetler.com","subject":"Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-06T17:51:52Z","receivedAt":"2017-11-06T17:51:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/2/2017 1:50 PM, Jeff Hostetler wrote:\n> From: Jeff Hostetler \n> \n> Create traverse_commit_list_filtered() and add filtering\n> interface to allow certain objects to be omitted from the\n> traversal.\n> ...\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> new file mode 100644\n> index 0000000..7f28425\n> --- /dev/null\n> +++ b/list-objects-filter.c\n> ...\n> +/*\n> + * A filter for list-objects to omit large blobs,\n> + * but always include \".git*\" special files.\n> + * And to OPTIONALLY collect a list of the omitted OIDs.\n> + */\n\nJonathan and I were talking off-list about the performance\neffects of inspecting the pathnames to identify the \".git*\"\nspecial files. I added it in my first draft back in the spring,\nthinking that even if you set the blob-limit to a small\nnumber (or zero), you'd probably still always want the\n.gitattribute and .gitignore files. But now with the addition\nof the sparse filter and functional dynamic object fetching,\nI'm not sure I see the need for this.\n\nAlso, if the primary use of the blob-limit is to filter out\ngiant binary assets, it is unlikely anyone is going to have\na 1MB+ .git* file, so it is unlikely that the is_special_file\nwould include anything that wouldn't already be included by\nthe size criteria.\n\nSo, if there's no objections, I think I'll remove this and\nsimplify the blob-limit filter function. (That would let me\nget rid of the provisional omit code here.)\n\nJeff\n"},{"id":"331932","messageId":"20171106100807.f459c0181637d20a7e556158@google.com","threadId":"47102","inReplyTo":"68f3c18f-8bee-dbb7-26f3-c1070d9177ac@jeffhostetler.com","subject":"Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-06T18:08:07Z","receivedAt":"2017-11-06T18:08:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 6 Nov 2017 12:51:52 -0500\nJeff Hostetler wrote:\n\n> Jonathan and I were talking off-list about the performance\n> effects of inspecting the pathnames to identify the \".git*\"\n> special files. I added it in my first draft back in the spring,\n> thinking that even if you set the blob-limit to a small\n> number (or zero), you'd probably still always want the\n> .gitattribute and .gitignore files. But now with the addition\n> of the sparse filter and functional dynamic object fetching,\n> I'm not sure I see the need for this.\n> \n> Also, if the primary use of the blob-limit is to filter out\n> giant binary assets, it is unlikely anyone is going to have\n> a 1MB+ .git* file, so it is unlikely that the is_special_file\n> would include anything that wouldn't already be included by\n> the size criteria.\n> \n> So, if there's no objections, I think I'll remove this and\n> simplify the blob-limit filter function. (That would let me\n> get rid of the provisional omit code here.)\n\nThis sounds like a good idea to me. (For the record, one of the\nperformance impacts of checking the filename is that bitmaps can't be\nused to obtain a whitelist of what is to be packed - instead, a regular\nobject walk must be used.)\n"},{"id":"332021","messageId":"f698d5a8-bf31-cea1-a8da-88b755b0b7af@jeffhostetler.com","threadId":"47102","inReplyTo":"20171102123245.0f768968703ec4e35d3d1f81@google.com","subject":"Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-07T18:54:20Z","receivedAt":"2017-11-07T18:54:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/2/2017 3:32 PM, Jonathan Tan wrote:\n> On Thu, 2 Nov 2017 17:50:11 +0000\n> Jeff Hostetler wrote:\n> \n>> +\t\tif (skip_prefix(v0, \"oid=\", &v1)) {\n>> +\t\t\tfilter_options->choice = LOFC_SPARSE_OID;\n>> +\t\t\tif (!get_oid_with_context(v1, GET_OID_BLOB,\n>> +\t\t\t\t\t\t &sparse_oid, &oc)) {\n>> +\t\t\t\t/*\n>> +\t\t\t\t * We successfully converted the \n>> +\t\t\t\t * into an actual OID. Rewrite the raw_value\n>> +\t\t\t\t * in canonoical form with just the OID.\n>> +\t\t\t\t * (If we send this request to the server, we\n>> +\t\t\t\t * want an absolute expression rather than a\n>> +\t\t\t\t * local-ref-relative expression.)\n>> +\t\t\t\t */\n> \n> I think this would lead to confusing behavior - for example, a fetch\n> with \"--filter=oid=mybranch:sparseconfig\" would have different results\n> depending on whether \"mybranch\" refers to a valid object locally.\n> \n> The way I see it, this should either (i) only accept full 40-character\n> OIDs or (ii) retain the raw string to be interpreted only when the\n> filtering is done. (i) is simpler and safer, but is not so useful. In\n> both cases, if the user really wants client-side interpretation, they\n> can still use \"$(git rev-parse foo)\" to make it explicit.\n\nGood point. I'll change it to always pass the original expression\nso that it is evaluated wherever the filtering is actually performed.\n\n\n> \n>> +\t\t\t\tfree((char *)filter_options->raw_value);\n>> +\t\t\t\tfilter_options->raw_value =\n>> +\t\t\t\t\txstrfmt(\"sparse:oid=%s\",\n>> +\t\t\t\t\t\toid_to_hex(&sparse_oid));\n>> +\t\t\t\tfilter_options->sparse_oid_value =\n>> +\t\t\t\t\toiddup(&sparse_oid);\n>> +\t\t\t} else {\n>> +\t\t\t\t/*\n>> +\t\t\t\t * We could not turn the into an\n>> +\t\t\t\t * OID. Leave the raw_value as is in case\n>> +\t\t\t\t * the server can parse it. (It may refer to\n>> +\t\t\t\t * a branch, commit, or blob we don't have.)\n>> +\t\t\t\t */\n>> +\t\t\t}\n>> +\t\t\treturn 0;\n>> +\t\t}\n>> +\n>> +\t\tif (skip_prefix(v0, \"path=\", &v1)) {\n>> +\t\t\tfilter_options->choice = LOFC_SPARSE_PATH;\n>> +\t\t\tfilter_options->sparse_path_value = strdup(v1);\n>> +\t\t\treturn 0;\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tdie(_(\"invalid filter expression '%s'\"), arg);\n>> +\treturn 0;\n>> +}\n> \n> [snip]\n> \n>> +void arg_format_list_objects_filter(\n>> +\tstruct argv_array *argv_array,\n>> +\tconst struct list_objects_filter_options *filter_options)\n> \n> Is this function used anywhere (in this patch or subsequent patches)?\n\nIt is used in upload-pack.c in part 3. I'll remove it from part 1\nand revisit in part 3.\n \n\n\n>> diff --git a/list-objects-filter.c b/list-objects-filter.c\n>> +/* See object.h and revision.h */\n>> +#define FILTER_REVISIT (1<<25)\n> \n> Looking later in the code, this flag indicates that a tree has been\n> SHOWN, so it might be better to just call this FILTER_SHOWN.\n\nI'll amend this. There are already several SHOWN bits that behave\nslightly differently. I'll update and document this better. Thanks.\n\n\n> \n> [snip]\n> \n>> +struct frame {\n>> +\tint defval;\n> \n> Document this variable?\n> \n>> +\tint child_prov_omit : 1;\n> \n> I think it's clearer if we use \"unsigned\" here. Also, document this\n> (e.g. \"1 if any descendant of this tree object was provisionally\n> omitted\").\n\ngot it. thanks.\n\n\n>> +enum list_objects_filter_type {\n>> +\tLOFT_BEGIN_TREE,\n>> +\tLOFT_END_TREE,\n>> +\tLOFT_BLOB\n>> +};\n> \n> Optional: probably a better name would be list_objects_filter_situation.\n\ngot it. thanks.\n\n \n>> +void traverse_commit_list_filtered(\n>> +\tstruct list_objects_filter_options *filter_options,\n>> +\tstruct rev_info *revs,\n>> +\tshow_commit_fn show_commit,\n>> +\tshow_object_fn show_object,\n>> +\tvoid *show_data,\n>> +\tstruct oidset *omitted)\n>> +{\n>> +\tfilter_object_fn filter_fn = NULL;\n>> +\tfilter_free_fn filter_free_fn = NULL;\n>> +\tvoid *filter_data = NULL;\n>> +\n>> +\tfilter_data = list_objects_filter__init(omitted, filter_options,\n>> +\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n>> +\tdo_traverse(revs, show_commit, show_object, show_data,\n>> +\t\t filter_fn, filter_data);\n>> +\tif (filter_data && filter_free_fn)\n>> +\t\tfilter_free_fn(filter_data);\n>> +}\n> \n> This function traverse_commit_list_filtered() is in list-objects.c but\n> in list-objects-filter.h, if I'm reading the diff correctly?\n\noops. thanks.\n\n\n> \n> Overall, this looks like a good change. Object traversal was upgraded\n> with the behaviors of MARK_SEEN and SHOW independently controllable and\n> with the ability to do things post-tree (in addition to pre-tree and\n> blob), and this was used to support a few types of filtering, which\n> subsequent patches will allow the user to invoke through \"--filter=\".\n> \n\nthanks\nJeff\n"},{"id":"332048","messageId":"20171107164118.97cc65c4030de0922b19ddd6@google.com","threadId":"47102","inReplyTo":"97585434-6d57-dad4-d7aa-e48dacec1b3f@jeffhostetler.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-08T00:41:18Z","receivedAt":"2017-11-08T00:41:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 3 Nov 2017 14:34:39 -0400\nJeff Hostetler wrote:\n\n> > Assuming we eventually get promisor support working, would there be\n> > any use case where \"any missing is OK\" mode would be useful in a\n> > sense more reasonable than \"because we could have such a mode\" and\n> > \"it is not our business to prevent users from playing with fire\"?\n> > \n> \n> For now, I'd like to keep my \"any missing is OK\" option.\n> I do think it has value all by itself.\n> \n> We are essentially using something like that now with our GVFS\n> users on the gigantic Windows repo and haven't had any issues.\n> \n> But yes, when we get promisor support working, we could revisit\n> the need for this parameter.\n\nWell, it's probably not a good idea to include this parameter, and then\nsubsequently remove it.\n\n> However, I do have some scaling concerns here. If for example,\n> I have 100M missing blobs (because we did an only commits-and-trees\n> clone), the cost to compute \"promisor missing\" vs \"just missing\"\n> might be prohibitively expensive. It could be something we want\n> fsck/gc to be aware of, but other commands may want to just assume\n> any missing objects are expected and continue.\n> \n> Hopefully, we won't have a scale problem, but we just don't know\n> yet.\n\nI can see some use for this parameter - for example, when doing a report\nfor statistical purposes (percentage of objects missing, for example) or\nfor a background task that downloads missing objects into a cache. Also,\npower users who know what they're doing (or normal users in an\nemergency) can use this option when they have no network connection if\nthey really need to find something out from the local repo.\n\nIn these cases, the promisor check (after detecting that the object is\nmissing) is indeed not so useful, I think. (Or we can do the\n--exclude=missing and --exclude=promisor idea that Jeff mentioned -\n--exclude=missing now, and --exclude=promisor after we add promisor\nsupport.)\n\nThis is conceptually different from gc's use of\n--exclude-promisor-objects (in my patch set), which does not intend to\ntouch promisor objects (objects that are known to be in the promisor\nremote), whether they are present or not.\n\nHaving said that, I would be OK if we didn't have tolerance (and/or\nreporting) of missing objects right now. As far as I know, for the\ninitial implementation of partial clone, only the server performs any\nfiltering, and we assume that the server possesses all objects (so it\ndoes not need to filter out any missing objects).\n"},{"id":"332053","messageId":"xmqq8tfhoai3.fsf@gitster.mtv.corp.google.com","threadId":"47102","inReplyTo":"20171107164118.97cc65c4030de0922b19ddd6@google.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-08T00:54:44Z","receivedAt":"2017-11-08T00:54:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> I can see some use for this parameter - for example, when doing a report\n> for statistical purposes (percentage of objects missing, for example) or\n> for a background task that downloads missing objects into a cache. Also,\n> power users who know what they're doing (or normal users in an\n> emergency) can use this option when they have no network connection if\n> they really need to find something out from the local repo.\n>\n> In these cases, the promisor check (after detecting that the object is\n> missing) is indeed not so useful, I think. (Or we can do the\n> --exclude=missing and --exclude=promisor idea that Jeff mentioned -\n> --exclude=missing now, and --exclude=promisor after we add promisor\n> support.)\n\nThis sounds like a reasonable thing to have in the endgame state to\nme.\n\n> Having said that, I would be OK if we didn't have tolerance (and/or\n> reporting) of missing objects right now. As far as I know, for the\n> initial implementation of partial clone, only the server performs any\n> filtering, and we assume that the server possesses all objects (so it\n> does not need to filter out any missing objects).\n\nTrue. It does not have to exist in an early part, but I do not\nthink we would terribly mind if it does, if only to help debugging\nand development.\n\nThanks for thinking it through.\n"},{"id":"332076","messageId":"de392bf3-bd53-1c17-3a43-c2e1604cbd59@jeffhostetler.com","threadId":"47102","inReplyTo":"xmqq8tfhoai3.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 0/6] Partial clone part 1: object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-08T14:39:35Z","receivedAt":"2017-11-08T14:39:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/7/2017 7:54 PM, Junio C Hamano wrote:\n> Jonathan Tan writes:\n> \n>> I can see some use for this parameter - for example, when doing a report\n>> for statistical purposes (percentage of objects missing, for example) or\n>> for a background task that downloads missing objects into a cache. Also,\n>> power users who know what they're doing (or normal users in an\n>> emergency) can use this option when they have no network connection if\n>> they really need to find something out from the local repo.\n>>\n>> In these cases, the promisor check (after detecting that the object is\n>> missing) is indeed not so useful, I think. (Or we can do the\n>> --exclude=missing and --exclude=promisor idea that Jeff mentioned -\n>> --exclude=missing now, and --exclude=promisor after we add promisor\n>> support.)\n> \n> This sounds like a reasonable thing to have in the endgame state to\n> me.\n\nOK thanks, I'll change it to --exclude=missing in my next version.\n \n> \n>> Having said that, I would be OK if we didn't have tolerance (and/or\n>> reporting) of missing objects right now. As far as I know, for the\n>> initial implementation of partial clone, only the server performs any\n>> filtering, and we assume that the server possesses all objects (so it\n>> does not need to filter out any missing objects).\n> \n> True. It does not have to exist in an early part, but I do not\n> think we would terribly mind if it does, if only to help debugging\n> and development.\n> \n> Thanks for thinking it through.\n> \n\nRight, it could come later, but having it here in part 1 as part\nof the initial series completes the pre-promisor portion of these\ncommands. Having a print-missing option lets rev-list -- as is --\nbe used in a bulk-fetch-object pre-checkout hook or as part of a\n\"give me what I need before I go offline\" command. This is useful\nby itself. It does augment the dynamic fetch-object code added in\npart 2 and the unpack-trees changes in part 3 to call fetch-object.\n\nJeff\n\n\n"}]}