{"thread":{"id":"47036","subject":"[PATCH] dir: allow exclusions from blob in addition to file","startedAt":"2017-10-26T18:02:39Z","lastAt":"2017-10-27T15:02:47Z","messageCount":4,"participants":["Jeff Hostetler","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"331103","messageId":"20171026180222.39421-1-git@jeffhostetler.com","threadId":"47036","inReplyTo":null,"subject":"[PATCH] dir: allow exclusions from blob in addition to file","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-10-26T18:02:21Z","receivedAt":"2017-10-26T18:02:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nI pulled commit 01/13 from Tuesday's partial clone part 1 patch series [1]\nand refactored the changes in dir.c to try to address Junio's comments in [2]\nWRT sharing more code with the existing read_skip_worktree_file_from_index().\n\nThis patch can be discussed independently of the partial clone series.\n\n[1] https://public-inbox.org/git/20171024185332.57261-2-git@jeffhostetler.com/\n[2] https://public-inbox.org/git/xmqqpo9afu3s.fsf@gitster.mtv.corp.google.com/\n\nJeff Hostetler (1):\n  dir: allow exclusions from blob in addition to file\n\n dir.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++----------------\n dir.h |   3 ++\n 2 files changed, 104 insertions(+), 31 deletions(-)\n\n-- \n2.9.3\n\n"},{"id":"331104","messageId":"20171026180222.39421-2-git@jeffhostetler.com","threadId":"47036","inReplyTo":"20171026180222.39421-1-git@jeffhostetler.com","subject":"[PATCH] dir: allow exclusions from blob in addition to file","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-10-26T18:02:22Z","receivedAt":"2017-10-26T18:02:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nRefactor add_excludes() to separate the reading of the\nexclude file into a buffer and the parsing of the buffer\ninto exclude_list items.\n\nAdd add_excludes_from_blob_to_list() to allow an exclude\nfile be specified with an OID without assuming a local\nworktree or index exists.\n\nRefactor read_skip_worktree_file_from_index() and add\ndo_read_blob() to eliminate duplication of preliminary\nprocessing of blob contents.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n dir.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++----------------\n dir.h |   3 ++\n 2 files changed, 104 insertions(+), 31 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 1d17b80..1962374 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -220,6 +220,57 @@ int within_depth(const char *name, int namelen,\n \treturn 1;\n }\n \n+/*\n+ * Read the contents of the blob with the given OID into a buffer.\n+ * Append a trailing LF to the end if the last line doesn't have one.\n+ *\n+ * Returns:\n+ *    -1 when the OID is invalid or unknown or does not refer to a blob.\n+ *     0 when the blob is empty.\n+ *     1 along with { data, size } of the (possibly augmented) buffer\n+ *       when successful.\n+ *\n+ * Optionally updates the given sha1_stat with the given OID (when valid).\n+ */\n+static int do_read_blob(const struct object_id *oid,\n+\t\t\tstruct sha1_stat *sha1_stat,\n+\t\t\tsize_t *size_out,\n+\t\t\tchar **data_out)\n+{\n+\tenum object_type type;\n+\tunsigned long sz;\n+\tchar *data;\n+\n+\t*size_out = 0;\n+\t*data_out = NULL;\n+\n+\tdata = read_sha1_file(oid->hash, &type, &sz);\n+\tif (!data || type != OBJ_BLOB) {\n+\t\tfree(data);\n+\t\treturn -1;\n+\t}\n+\n+\tif (sha1_stat) {\n+\t\tmemset(&sha1_stat->stat, 0, sizeof(sha1_stat->stat));\n+\t\thashcpy(sha1_stat->sha1, oid->hash);\n+\t}\n+\n+\tif (sz == 0) {\n+\t\tfree(data);\n+\t\treturn 0;\n+\t}\n+\n+\tif (data[sz - 1] != '\\n') {\n+\t\tdata = xrealloc(data, st_add(sz, 1));\n+\t\tdata[sz++] = '\\n';\n+\t}\n+\n+\t*size_out = xsize_t(sz);\n+\t*data_out = data;\n+\n+\treturn 1;\n+}\n+\n #define DO_MATCH_EXCLUDE   (1<<0)\n #define DO_MATCH_DIRECTORY (1<<1)\n #define DO_MATCH_SUBMODULE (1<<2)\n@@ -600,32 +651,22 @@ void add_exclude(const char *string, const char *base,\n \tx->el = el;\n }\n \n-static void *read_skip_worktree_file_from_index(const struct index_state *istate,\n-\t\t\t\t\t\tconst char *path, size_t *size,\n-\t\t\t\t\t\tstruct sha1_stat *sha1_stat)\n+static int read_skip_worktree_file_from_index(const struct index_state *istate,\n+\t\t\t\t\t      const char *path,\n+\t\t\t\t\t      size_t *size_out,\n+\t\t\t\t\t      char **data_out,\n+\t\t\t\t\t      struct sha1_stat *sha1_stat)\n {\n \tint pos, len;\n-\tunsigned long sz;\n-\tenum object_type type;\n-\tvoid *data;\n \n \tlen = strlen(path);\n \tpos = index_name_pos(istate, path, len);\n \tif (pos < 0)\n-\t\treturn NULL;\n+\t\treturn -1;\n \tif (!ce_skip_worktree(istate->cache[pos]))\n-\t\treturn NULL;\n-\tdata = read_sha1_file(istate->cache[pos]->oid.hash, &type, &sz);\n-\tif (!data || type != OBJ_BLOB) {\n-\t\tfree(data);\n-\t\treturn NULL;\n-\t}\n-\t*size = xsize_t(sz);\n-\tif (sha1_stat) {\n-\t\tmemset(&sha1_stat->stat, 0, sizeof(sha1_stat->stat));\n-\t\thashcpy(sha1_stat->sha1, istate->cache[pos]->oid.hash);\n-\t}\n-\treturn data;\n+\t\treturn -1;\n+\n+\treturn do_read_blob(&istate->cache[pos]->oid, sha1_stat, size_out, data_out);\n }\n \n /*\n@@ -739,6 +780,10 @@ static void invalidate_directory(struct untracked_cache *uc,\n \t\tdir->dirs[i]->recurse = 0;\n }\n \n+static int add_excludes_from_buffer(char *buf, size_t size,\n+\t\t\t\t    const char *base, int baselen,\n+\t\t\t\t    struct exclude_list *el);\n+\n /*\n  * Given a file with name \"fname\", read it (either from disk, or from\n  * an index if 'istate' is non-null), parse it and store the\n@@ -754,9 +799,10 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t\tstruct sha1_stat *sha1_stat)\n {\n \tstruct stat st;\n-\tint fd, i, lineno = 1;\n+\tint r;\n+\tint fd;\n \tsize_t size = 0;\n-\tchar *buf, *entry;\n+\tchar *buf;\n \n \tfd = open(fname, O_RDONLY);\n \tif (fd < 0 || fstat(fd, &st) < 0) {\n@@ -764,17 +810,13 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t\twarn_on_fopen_errors(fname);\n \t\telse\n \t\t\tclose(fd);\n-\t\tif (!istate ||\n-\t\t    (buf = read_skip_worktree_file_from_index(istate, fname, &size, sha1_stat)) == NULL)\n+\t\tif (!istate)\n \t\t\treturn -1;\n-\t\tif (size == 0) {\n-\t\t\tfree(buf);\n-\t\t\treturn 0;\n-\t\t}\n-\t\tif (buf[size-1] != '\\n') {\n-\t\t\tbuf = xrealloc(buf, st_add(size, 1));\n-\t\t\tbuf[size++] = '\\n';\n-\t\t}\n+\t\tr = read_skip_worktree_file_from_index(istate, fname,\n+\t\t\t\t\t\t       &size, &buf,\n+\t\t\t\t\t\t       sha1_stat);\n+\t\tif (r != 1)\n+\t\t\treturn r;\n \t} else {\n \t\tsize = xsize_t(st.st_size);\n \t\tif (size == 0) {\n@@ -813,6 +855,17 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t}\n \t}\n \n+\tadd_excludes_from_buffer(buf, size, base, baselen, el);\n+\treturn 0;\n+}\n+\n+static int add_excludes_from_buffer(char *buf, size_t size,\n+\t\t\t\t    const char *base, int baselen,\n+\t\t\t\t    struct exclude_list *el)\n+{\n+\tint i, lineno = 1;\n+\tchar *entry;\n+\n \tel->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -841,6 +894,23 @@ int add_excludes_from_file_to_list(const char *fname, const char *base,\n \treturn add_excludes(fname, base, baselen, el, istate, NULL);\n }\n \n+int add_excludes_from_blob_to_list(\n+\tstruct object_id *oid,\n+\tconst char *base, int baselen,\n+\tstruct exclude_list *el)\n+{\n+\tchar *buf;\n+\tsize_t size;\n+\tint r;\n+\n+\tr = do_read_blob(oid, NULL, &size, &buf);\n+\tif (r != 1)\n+\t\treturn r;\n+\n+\tadd_excludes_from_buffer(buf, size, base, baselen, el);\n+\treturn 0;\n+}\n+\n struct exclude_list *add_exclude_list(struct dir_struct *dir,\n \t\t\t\t      int group_type, const char *src)\n {\ndiff --git a/dir.h b/dir.h\nindex e371705..1bcf391 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -256,6 +256,9 @@ extern struct exclude_list *add_exclude_list(struct dir_struct *dir,\n extern int add_excludes_from_file_to_list(const char *fname, const char *base, int baselen,\n \t\t\t\t\t  struct exclude_list *el, struct  index_state *istate);\n extern void add_excludes_from_file(struct dir_struct *, const char *fname);\n+extern int add_excludes_from_blob_to_list(struct object_id *oid,\n+\t\t\t\t\t  const char *base, int baselen,\n+\t\t\t\t\t  struct exclude_list *el);\n extern void parse_exclude_pattern(const char **string, int *patternlen, unsigned *flags, int *nowildcardlen);\n extern void add_exclude(const char *string, const char *base,\n \t\t\tint baselen, struct exclude_list *el, int srcpos);\n-- \n2.9.3\n\n"},{"id":"331132","messageId":"xmqq8tfxe693.fsf@gitster.mtv.corp.google.com","threadId":"47036","inReplyTo":"20171026180222.39421-2-git@jeffhostetler.com","subject":"Re: [PATCH] dir: allow exclusions from blob in addition to file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-27T01:20:24Z","receivedAt":"2017-10-27T01:20:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Refactor add_excludes() to separate the reading of the\n> exclude file into a buffer and the parsing of the buffer\n> into exclude_list items.\n>\n> Add add_excludes_from_blob_to_list() to allow an exclude\n> file be specified with an OID without assuming a local\n> worktree or index exists.\n>\n> Refactor read_skip_worktree_file_from_index() and add\n> do_read_blob() to eliminate duplication of preliminary\n> processing of blob contents.\n>\n> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n> ---\n\nYeah, with a separate do_read_blob() helper, this one looks a lot\neasier to follow, at least to me---as the author, you might find the\nearlier one just as easy, I suspect, though ;-)\n\nThanks.  Will queue.\n"},{"id":"331155","messageId":"937c3ec0-fb9b-0f11-b00d-f015d8e4bf4b@jeffhostetler.com","threadId":"47036","inReplyTo":"xmqq8tfxe693.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] dir: allow exclusions from blob in addition to file","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-10-27T15:02:40Z","receivedAt":"2017-10-27T15:02:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/26/2017 9:20 PM, Junio C Hamano wrote:\n> Jeff Hostetler <git@jeffhostetler.com> writes:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Refactor add_excludes() to separate the reading of the\n>> exclude file into a buffer and the parsing of the buffer\n>> into exclude_list items.\n>>\n>> Add add_excludes_from_blob_to_list() to allow an exclude\n>> file be specified with an OID without assuming a local\n>> worktree or index exists.\n>>\n>> Refactor read_skip_worktree_file_from_index() and add\n>> do_read_blob() to eliminate duplication of preliminary\n>> processing of blob contents.\n>>\n>> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n>> ---\n> \n> Yeah, with a separate do_read_blob() helper, this one looks a lot\n> easier to follow, at least to me---as the author, you might find the\n> earlier one just as easy, I suspect, though ;-)\n> \n> Thanks.  Will queue.\n> \n\nYeah, I think the net result is better and easier to follow.\nThanks,\nJeff\n"}]}