{"thread":{"id":"27422","subject":"[PATCH 0/4] streaming conversion API","startedAt":"2011-05-21T06:58:43Z","lastAt":"2011-05-25T02:23:04Z","messageCount":12,"participants":["Junio C Hamano","René Scharfe"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"168363","messageId":"1305961127-26540-1-git-send-email-gitster@pobox.com","threadId":"27422","inReplyTo":null,"subject":"[PATCH 0/4] streaming conversion API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:58:43Z","receivedAt":"2011-05-21T06:58:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This builds on top of jc/streaming topic, in which I earlier hinted that a\nfiltering mechanism that does not need to slurp everything in core can\nlater be plugged into the git_istream API. This topioc does exactly that.\n\nThe first patch moves declarations related to conversion from cache.h to\nits own convert.h header file, so that new streaming conversion API can be\nadded to it without bloating cache.h even more.\n\nThe second patch defines the streaming filter API, adds a pass-through\n\"null\" filter, plugs the streaming filter API to the git_istream API and\nupdates the write out codepath to use it.  The design of this API is an\nobject-oriented one that is similar to the istream API. The user of the\ngit_istream API first grabs \"stream_filter\" object based on the path, and\npasses that filter object when creating a git_istream. The data read from\nthe underlying object is fed to the filter, and the output from the filter\nis returned to the caller of the read_istream().\n\nThe third patch adds a non-guessing LF-to-CRLF streaming filter.\n\nThe fourth patch implements the ident filter. As both LF-to-CRLF and ident\nfilters can be active at the same time, it also introduces the \"cascade\"\nmechanism that takes two filters, plugs one's output to the other's input,\nand feeds the former and returns what is read from the latter.\n\nI suspect that \"smudge\" filter should be relatively easy to add and\ncombine with the existing filters using the same cascade mechanism,\nbut I'll leave it as an exercise to interested readers.\n\nJunio C Hamano (4):\n  convert.h: move declarations for conversion from cache.h\n  Add streaming filter API\n  Add LF-to-CRLF streaming conversion\n  streaming filter: ident filter and filter cascading\n\n cache.h     |   38 +------\n convert.c   |  363 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n convert.h   |   65 +++++++++++\n entry.c     |   16 ++-\n streaming.c |  100 ++++++++++++++++-\n streaming.h |    2 +-\n 6 files changed, 530 insertions(+), 54 deletions(-)\n create mode 100644 convert.h\n\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168364","messageId":"1305961127-26540-2-git-send-email-gitster@pobox.com","threadId":"27422","inReplyTo":"1305961127-26540-1-git-send-email-gitster@pobox.com","subject":"[PATCH] convert.h: move declarations for conversion from cache.h","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:58:44Z","receivedAt":"2011-05-21T06:58:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Before adding the streaming filter API to the conversion layer,\nmove the existing declarations related to the conversion to its\nown header file.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n cache.h   |   38 +-------------------------------------\n convert.h |   44 ++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 45 insertions(+), 37 deletions(-)\n create mode 100644 convert.h\n\ndiff --git a/cache.h b/cache.h\nindex a5067ba..c781c11 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -6,6 +6,7 @@\n #include \"hash.h\"\n #include \"advice.h\"\n #include \"gettext.h\"\n+#include \"convert.h\"\n \n #include SHA1_HEADER\n #ifndef git_SHA_CTX\n@@ -582,35 +583,6 @@ extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n \n-enum safe_crlf {\n-\tSAFE_CRLF_FALSE = 0,\n-\tSAFE_CRLF_FAIL = 1,\n-\tSAFE_CRLF_WARN = 2\n-};\n-\n-extern enum safe_crlf safe_crlf;\n-\n-enum auto_crlf {\n-\tAUTO_CRLF_FALSE = 0,\n-\tAUTO_CRLF_TRUE = 1,\n-\tAUTO_CRLF_INPUT = -1\n-};\n-\n-extern enum auto_crlf auto_crlf;\n-\n-enum eol {\n-\tEOL_UNSET,\n-\tEOL_CRLF,\n-\tEOL_LF,\n-#ifdef NATIVE_CRLF\n-\tEOL_NATIVE = EOL_CRLF\n-#else\n-\tEOL_NATIVE = EOL_LF\n-#endif\n-};\n-\n-extern enum eol core_eol;\n-\n enum branch_track {\n \tBRANCH_TRACK_UNSPECIFIED = -1,\n \tBRANCH_TRACK_NEVER = 0,\n@@ -1153,14 +1125,6 @@ extern void trace_strbuf(const char *key, const struct strbuf *buf);\n \n void packet_trace_identity(const char *prog);\n \n-/* convert.c */\n-/* returns 1 if *dst was used */\n-extern int convert_to_git(const char *path, const char *src, size_t len,\n-                          struct strbuf *dst, enum safe_crlf checksafe);\n-extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);\n-extern int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst);\n-extern int can_bypass_conversion(const char *path);\n-\n /* add */\n /*\n  * return 0 if success, 1 - if addition of a file failed and\ndiff --git a/convert.h b/convert.h\nnew file mode 100644\nindex 0000000..b1b4a38\n--- /dev/null\n+++ b/convert.h\n@@ -0,0 +1,44 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#ifndef CONVERT_H\n+#define CONVERT_H\n+\n+enum safe_crlf {\n+\tSAFE_CRLF_FALSE = 0,\n+\tSAFE_CRLF_FAIL = 1,\n+\tSAFE_CRLF_WARN = 2\n+};\n+\n+extern enum safe_crlf safe_crlf;\n+\n+enum auto_crlf {\n+\tAUTO_CRLF_FALSE = 0,\n+\tAUTO_CRLF_TRUE = 1,\n+\tAUTO_CRLF_INPUT = -1\n+};\n+\n+extern enum auto_crlf auto_crlf;\n+\n+enum eol {\n+\tEOL_UNSET,\n+\tEOL_CRLF,\n+\tEOL_LF,\n+#ifdef NATIVE_CRLF\n+\tEOL_NATIVE = EOL_CRLF\n+#else\n+\tEOL_NATIVE = EOL_LF\n+#endif\n+};\n+\n+extern enum eol core_eol;\n+\n+/* returns 1 if *dst was used */\n+extern int convert_to_git(const char *path, const char *src, size_t len,\n+\t\t\t  struct strbuf *dst, enum safe_crlf checksafe);\n+extern int convert_to_working_tree(const char *path, const char *src,\n+\t\t\t\t   size_t len, struct strbuf *dst);\n+extern int renormalize_buffer(const char *path, const char *src, size_t len,\n+\t\t\t      struct strbuf *dst);\n+extern int can_bypass_conversion(const char *path);\n+#endif /* CONVERT_H */\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168365","messageId":"1305961127-26540-3-git-send-email-gitster@pobox.com","threadId":"27422","inReplyTo":"1305961127-26540-1-git-send-email-gitster@pobox.com","subject":"[PATCH 2/4] Add streaming filter API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:58:45Z","receivedAt":"2011-05-21T06:58:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This introduces an API to plug custom filters to an input stream.\n\nThe caller gets get_stream_filter(\"path\") to obtain an appropriate\nfilter for the path, and then uses it when opening an input stream\nvia open_istream().  After that, the caller can read from the stream\nwith read_istream(), and close it with close_istream(), just like an\nunfiltered stream.\n\nThis only adds a \"null\" filter that is a pass-thru filter, but later\nchanges can add LF-to-CRLF and other filters, and the callers of the\nstreaming API do not have to change.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n convert.c   |   84 +++++++++++++++++++++++++++++++++++++++++++++----\n convert.h   |   23 +++++++++++++-\n entry.c     |   16 ++++++---\n streaming.c |  100 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n streaming.h |    2 +-\n 5 files changed, 209 insertions(+), 16 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 264af1d..1ec91a3 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -814,12 +814,69 @@ int renormalize_buffer(const char *path, const char *src, size_t len, struct str\n \treturn ret | convert_to_git(path, src, len, dst, 0);\n }\n \n+/*****************************************************************\n+ *\n+ * Streaming converison support\n+ *\n+ *****************************************************************/\n+\n+typedef int (*filter_fn)(struct stream_filter *,\n+\t\t\t const char *input, size_t *isize_p,\n+\t\t\t char *output, size_t *osize_p);\n+typedef void (*free_fn)(struct stream_filter *);\n+\n+struct stream_filter_vtbl {\n+\tfilter_fn filter;\n+\tfree_fn free;\n+};\n+\n+struct stream_filter {\n+\tstruct stream_filter_vtbl *vtbl;\n+};\n+\n+static int null_filter_fn(struct stream_filter *filter,\n+\t\t\t  const char *input, size_t *isize_p,\n+\t\t\t  char *output, size_t *osize_p)\n+{\n+\tsize_t count = *isize_p;\n+\tif (*osize_p < count)\n+\t\tcount = *osize_p;\n+\tif (count) {\n+\t\tmemmove(output, input, count);\n+\t\t*isize_p -= count;\n+\t\t*osize_p -= count;\n+\t}\n+\treturn 0;\n+}\n+\n+static void null_free_fn(struct stream_filter *filter)\n+{\n+\t; /* nothing -- null instances are shared */\n+}\n+\n+static struct stream_filter_vtbl null_vtbl = {\n+\tnull_filter_fn,\n+\tnull_free_fn,\n+};\n+\n+static struct stream_filter null_filter_singleton = {\n+\t&null_vtbl,\n+};\n+\n+int is_null_stream_filter(struct stream_filter *filter)\n+{\n+\treturn filter == &null_filter_singleton;\n+}\n+\n /*\n- * You would be crazy to set CRLF, smuge/clean or ident to\n- * a large binary blob you would want us not to slurp into\n- * the memory!\n+ * Return an appropriately constructed filter for the path, or NULL if\n+ * the contents cannot be filtered without reading the whole thing\n+ * in-core.\n+ *\n+ * Note that you would be crazy to set CRLF, smuge/clean or ident to a\n+ * large binary blob you would want us not to slurp into the memory!\n  */\n-int can_bypass_conversion(const char *path)\n+struct stream_filter *get_stream_filter(const char *path, const unsigned char *sha1)\n {\n \tstruct conv_attrs ca;\n \tenum crlf_action crlf_action;\n@@ -828,11 +885,24 @@ int can_bypass_conversion(const char *path)\n \n \tif (ca.ident ||\n \t    (ca.drv && (ca.drv->smudge || ca.drv->clean)))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n \tif ((crlf_action == CRLF_BINARY) || (crlf_action == CRLF_INPUT) ||\n \t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n-\t\treturn 1;\n-\treturn 0;\n+\t\treturn &null_filter_singleton;\n+\n+\treturn NULL;\n+}\n+\n+void free_stream_filter(struct stream_filter *filter)\n+{\n+\tfilter->vtbl->free(filter);\n+}\n+\n+int stream_filter(struct stream_filter *filter,\n+\t\t  const char *input, size_t *isize_p,\n+\t\t  char *output, size_t *osize_p)\n+{\n+\treturn filter->vtbl->filter(filter, input, isize_p, output, osize_p);\n }\ndiff --git a/convert.h b/convert.h\nindex d61c111..c9906f9 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -37,5 +37,26 @@ extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t   size_t len, struct strbuf *dst);\n extern int renormalize_buffer(const char *path, const char *src, size_t len,\n \t\t\t      struct strbuf *dst);\n-extern int can_bypass_conversion(const char *path);\n+\n+/*****************************************************************\n+ *\n+ * Streaming converison support\n+ *\n+ *****************************************************************/\n+\n+struct stream_filter; /* opaque */\n+\n+extern struct stream_filter *get_stream_filter(const char *path, const unsigned char *);\n+extern void free_stream_filter(struct stream_filter *);\n+extern int is_null_stream_filter(struct stream_filter *);\n+\n+/*\n+ * Use as much input up to *isize_p and fill output up to *osize_p;\n+ * update isize_p and osize_p to indicate how much buffer space was\n+ * consumed and filled. Return 0 on success, non-zero on error.\n+ */\n+extern int stream_filter(struct stream_filter *,\n+\t\t\t const char *input, size_t *isize_p,\n+\t\t\t char *output, size_t *osize_p);\n+\n #endif /* CONVERT_H */\ndiff --git a/entry.c b/entry.c\nindex e2dc16c..852fea1 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -116,6 +116,7 @@ static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n }\n \n static int streaming_write_entry(struct cache_entry *ce, char *path,\n+\t\t\t\t struct stream_filter *filter,\n \t\t\t\t const struct checkout *state, int to_tempfile,\n \t\t\t\t int *fstat_done, struct stat *statbuf)\n {\n@@ -126,7 +127,7 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \tssize_t kept = 0;\n \tint fd = -1;\n \n-\tst = open_istream(ce->sha1, &type, &sz);\n+\tst = open_istream(ce->sha1, &type, &sz, filter);\n \tif (!st)\n \t\treturn -1;\n \tif (type != OBJ_BLOB)\n@@ -186,11 +187,14 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \tsize_t wrote, newsize = 0;\n \tstruct stat st;\n \n-\tif ((ce_mode_s_ifmt == S_IFREG) &&\n-\t    can_bypass_conversion(path) &&\n-\t    !streaming_write_entry(ce, path, state, to_tempfile,\n-\t\t\t\t   &fstat_done, &st))\n-\t\tgoto finish;\n+\tif (ce_mode_s_ifmt == S_IFREG) {\n+\t\tstruct stream_filter *filter = get_stream_filter(path, ce->sha1);\n+\t\tif (filter &&\n+\t\t    !streaming_write_entry(ce, path, filter,\n+\t\t\t\t\t   state, to_tempfile,\n+\t\t\t\t\t   &fstat_done, &st))\n+\t\t\tgoto finish;\n+\t}\n \n \tswitch (ce_mode_s_ifmt) {\n \tcase S_IFREG:\ndiff --git a/streaming.c b/streaming.c\nindex 0602926..565f000 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -41,6 +41,9 @@ struct stream_vtbl {\n static open_method_decl(incore);\n static open_method_decl(loose);\n static open_method_decl(pack_non_delta);\n+static struct git_istream *attach_stream_filter(struct git_istream *st,\n+\t\t\t\t\t\tstruct stream_filter *filter);\n+\n \n static open_istream_fn open_istream_tbl[] = {\n \topen_istream_incore,\n@@ -48,6 +51,17 @@ static open_istream_fn open_istream_tbl[] = {\n \topen_istream_pack_non_delta,\n };\n \n+#define FILTER_BUFFER (1024*16)\n+\n+struct filtered_istream {\n+\tstruct git_istream *upstream;\n+\tstruct stream_filter *filter;\n+\tchar ibuf[FILTER_BUFFER];\n+\tchar obuf[FILTER_BUFFER];\n+\tint i_end, i_ptr;\n+\tint o_end, o_ptr;\n+};\n+\n struct git_istream {\n \tconst struct stream_vtbl *vtbl;\n \tunsigned long size; /* inflated size of full object */\n@@ -72,6 +86,8 @@ struct git_istream {\n \t\t\tstruct packed_git *pack;\n \t\t\toff_t pos;\n \t\t} in_pack;\n+\n+\t\tstruct filtered_istream filtered;\n \t} u;\n };\n \n@@ -112,7 +128,8 @@ static enum input_source istream_source(const unsigned char *sha1,\n \n struct git_istream *open_istream(const unsigned char *sha1,\n \t\t\t\t enum object_type *type,\n-\t\t\t\t unsigned long *size)\n+\t\t\t\t unsigned long *size,\n+\t\t\t\t struct stream_filter *filter)\n {\n \tstruct git_istream *st;\n \tstruct object_info oi;\n@@ -129,6 +146,14 @@ struct git_istream *open_istream(const unsigned char *sha1,\n \t\t\treturn NULL;\n \t\t}\n \t}\n+\tif (st && filter) {\n+\t\t/* Add \"&& !is_null_stream_filter(filter)\" for performance */\n+\t\tstruct git_istream *nst = attach_stream_filter(st, filter);\n+\t\tif (!nst)\n+\t\t\tclose_istream(st);\n+\t\tst = nst;\n+\t}\n+\n \t*size = st->size;\n \treturn st;\n }\n@@ -149,6 +174,79 @@ static void close_deflated_stream(struct git_istream *st)\n \n /*****************************************************************\n  *\n+ * Filtered stream\n+ *\n+ *****************************************************************/\n+\n+static close_method_decl(filtered)\n+{\n+\tfree_stream_filter(st->u.filtered.filter);\n+\treturn close_istream(st->u.filtered.upstream);\n+}\n+\n+static read_method_decl(filtered)\n+{\n+\tstruct filtered_istream *fs = &(st->u.filtered);\n+\tsize_t filled = 0;\n+\n+\twhile (sz) {\n+\t\t/* do we already have filtered output? */\n+\t\tif (fs->o_ptr < fs->o_end) {\n+\t\t\tsize_t to_move = fs->o_end - fs->o_ptr;\n+\t\t\tif (sz < to_move)\n+\t\t\t\tto_move = sz;\n+\t\t\tmemcpy(buf + filled, fs->obuf + fs->o_ptr, to_move);\n+\t\t\tfs->o_ptr += to_move;\n+\t\t\tsz -= to_move;\n+\t\t\tfilled += to_move;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfs->o_end = fs->o_ptr = 0;\n+\n+\t\t/* do we have anything to feed the filter with? */\n+\t\tif (fs->i_ptr < fs->i_end) {\n+\t\t\tsize_t to_feed = fs->i_end - fs->i_ptr;\n+\t\t\tsize_t to_receive = FILTER_BUFFER;\n+\t\t\tif (stream_filter(fs->filter,\n+\t\t\t\t\t  fs->ibuf + fs->i_ptr, &to_feed,\n+\t\t\t\t\t  fs->obuf, &to_receive))\n+\t\t\t\treturn -1;\n+\t\t\tfs->i_ptr = fs->i_end - to_feed;\n+\t\t\tfs->o_end = FILTER_BUFFER - to_receive;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfs->i_end = fs->i_ptr = 0;\n+\n+\t\t/* refill the input from the upstream */\n+\t\tfs->i_end = read_istream(fs->upstream, fs->ibuf, FILTER_BUFFER);\n+\t\tif (fs->i_end <= 0)\n+\t\t\tbreak;\n+\t}\n+\treturn filled;\n+}\n+\n+static struct stream_vtbl filtered_vtbl = {\n+\tclose_istream_filtered,\n+\tread_istream_filtered,\n+};\n+\n+static struct git_istream *attach_stream_filter(struct git_istream *st,\n+\t\t\t\t\t\tstruct stream_filter *filter)\n+{\n+\tstruct git_istream *ifs = xmalloc(sizeof(*ifs));\n+\tstruct filtered_istream *fs = &(ifs->u.filtered);\n+\n+\tifs->vtbl = &filtered_vtbl;\n+\tfs->upstream = st;\n+\tfs->filter = filter;\n+\tfs->i_end = fs->i_ptr = 0;\n+\tfs->o_end = fs->o_ptr = 0;\n+\tifs->size = -1; /* unknown */\n+\treturn ifs;\n+}\n+\n+/*****************************************************************\n+ *\n  * Loose object stream\n  *\n  *****************************************************************/\ndiff --git a/streaming.h b/streaming.h\nindex 18cbe68..589e857 100644\n--- a/streaming.h\n+++ b/streaming.h\n@@ -8,7 +8,7 @@\n /* opaque */\n struct git_istream;\n \n-extern struct git_istream *open_istream(const unsigned char *, enum object_type *, unsigned long *);\n+extern struct git_istream *open_istream(const unsigned char *, enum object_type *, unsigned long *, struct stream_filter *);\n extern int close_istream(struct git_istream *);\n extern ssize_t read_istream(struct git_istream *, char *, size_t);\n \n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168366","messageId":"1305961127-26540-4-git-send-email-gitster@pobox.com","threadId":"27422","inReplyTo":"1305961127-26540-1-git-send-email-gitster@pobox.com","subject":"[PATCH 3/4] Add LF-to-CRLF streaming conversion","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:58:46Z","receivedAt":"2011-05-21T06:58:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"If we do not have to guess or validate by scanning the input, we can\njust stream this through.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n convert.c |   37 +++++++++++++++++++++++++++++++++++++\n 1 files changed, 37 insertions(+), 0 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 1ec91a3..82fb75b 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -868,6 +868,39 @@ int is_null_stream_filter(struct stream_filter *filter)\n \treturn filter == &null_filter_singleton;\n }\n \n+static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n+\t\t\t\tconst char *input, size_t *isize_p,\n+\t\t\t\tchar *output, size_t *osize_p)\n+{\n+\tsize_t count = *isize_p;\n+\tif (count) {\n+\t\tsize_t i, o;\n+\t\tfor (i = o = 0; o < *osize_p && i < count; i++) {\n+\t\t\tchar ch = input[i];\n+\t\t\tif (ch == '\\n') {\n+\t\t\t\tif (o + 1 < *osize_p)\n+\t\t\t\t\toutput[o++] = '\\r';\n+\t\t\t\telse\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\toutput[o++] = ch;\n+\t\t}\n+\n+\t\t*osize_p -= o;\n+\t\t*isize_p -= i;\n+\t}\n+\treturn 0;\n+}\n+\n+static struct stream_filter_vtbl lf_to_crlf_vtbl = {\n+\tlf_to_crlf_filter_fn,\n+\tnull_free_fn,\n+};\n+\n+static struct stream_filter lf_to_crlf_filter_singleton = {\n+\t&lf_to_crlf_vtbl,\n+};\n+\n /*\n  * Return an appropriately constructed filter for the path, or NULL if\n  * the contents cannot be filtered without reading the whole thing\n@@ -892,6 +925,10 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n \t\treturn &null_filter_singleton;\n \n+\tif (output_eol(crlf_action) == EOL_CRLF &&\n+\t    !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))\n+\t\treturn &lf_to_crlf_filter_singleton;\n+\n \treturn NULL;\n }\n \n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168367","messageId":"1305961127-26540-5-git-send-email-gitster@pobox.com","threadId":"27422","inReplyTo":"1305961127-26540-1-git-send-email-gitster@pobox.com","subject":"[PATCH 4/4] streaming filter: ident filter and filter cascading","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:58:47Z","receivedAt":"2011-05-21T06:58:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Add support for \"ident\" filter on the output codepath. Because we now have\nmore than one filter active, also add support to cascade them together.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n\n * There is one known issue in this \"ident\" filter implementation. If the\n   input data ends while the filter is still in the \"SKIPPING\" state, it\n   should go to the \"DRAINING\" codepath using the same logic that triggers\n   when it gets to the end of line, copying out what was already queued.\n   However, the API does not offer an explicit way for the input side to\n   say \"that's all\". We probably need to make a zero-byte input to the\n   filter function as an explicit EOF signal or something---currently I\n   think it will loop forever, but it is getting late, so I will not be\n   further touching this code tonight.\n\n convert.c |  254 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 246 insertions(+), 8 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 82fb75b..6b90c4d 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -868,6 +868,10 @@ int is_null_stream_filter(struct stream_filter *filter)\n \treturn filter == &null_filter_singleton;\n }\n \n+\n+/*\n+ * LF-to-CRLF filter\n+ */\n static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n \t\t\t\tconst char *input, size_t *isize_p,\n \t\t\t\tchar *output, size_t *osize_p)\n@@ -901,6 +905,237 @@ static struct stream_filter lf_to_crlf_filter_singleton = {\n \t&lf_to_crlf_vtbl,\n };\n \n+\n+/*\n+ * Cascade filter\n+ */\n+#define FILTER_BUFFER 1024\n+struct cascade_filter {\n+\tstruct stream_filter filter;\n+\tstruct stream_filter *one;\n+\tstruct stream_filter *two;\n+\tchar ibuf[FILTER_BUFFER];\n+\tchar obuf[FILTER_BUFFER];\n+\tint i_end, i_ptr;\n+\tint o_end, o_ptr;\n+};\n+\n+static int cascade_filter_fn(struct stream_filter *filter,\n+\t\t\t     const char *input, size_t *isize_p,\n+\t\t\t     char *output, size_t *osize_p)\n+{\n+\tstruct cascade_filter *cas = (struct cascade_filter *) filter;\n+\tsize_t filled = 0;\n+\tsize_t sz = *osize_p;\n+\tsize_t to_feed, remaining;\n+\n+\t/*\n+\t * input --> ibuf --(one)--> obuf --(two)--> output\n+\t */\n+\twhile (filled < sz) {\n+\t\t/* do we already have output to feed two? */\n+\t\tif (cas->o_ptr < cas->o_end) {\n+\t\t\tto_feed = cas->o_end - cas->o_ptr;\n+\t\t\tremaining = sz - filled;\n+\t\t\tif (stream_filter(cas->two,\n+\t\t\t\t\t  cas->obuf + cas->o_ptr, &to_feed,\n+\t\t\t\t\t  output + filled, &remaining))\n+\t\t\t\treturn -1;\n+\t\t\tcas->o_ptr += (cas->o_end - cas->o_ptr) - to_feed;\n+\t\t\tfilled = sz - remaining;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tcas->o_end = cas->o_ptr = 0;\n+\n+\t\t/* do we have anything to feed one? */\n+\t\tif (cas->i_ptr < cas->i_end) {\n+\t\t\tto_feed = cas->i_end - cas->i_ptr;\n+\t\t\tremaining = sizeof(cas->obuf);\n+\t\t\tif (stream_filter(cas->one,\n+\t\t\t\t\t  cas->ibuf + cas->i_ptr, &to_feed,\n+\t\t\t\t\t  cas->obuf, &remaining))\n+\t\t\t\treturn -1;\n+\t\t\tcas->i_ptr += (cas->i_end - cas->i_ptr) - to_feed;\n+\t\t\tcas->o_end = sizeof(cas->obuf) - remaining;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tcas->i_end = cas->i_ptr = 0;\n+\n+\t\t/* read from our upstream */\n+\t\tto_feed = *isize_p;\n+\t\tif (sizeof(cas->ibuf) < to_feed)\n+\t\t\tto_feed = sizeof(cas->ibuf);\n+\t\tif (!to_feed)\n+\t\t\tbreak;\n+\t\tmemcpy(cas->ibuf, input, to_feed);\n+\t\tinput += to_feed;\n+\t\t*isize_p -= to_feed;\n+\t\tcas->i_end = to_feed;\n+\t}\n+\t*osize_p -= filled;\n+\treturn 0;\n+}\n+\n+static void cascade_free_fn(struct stream_filter *filter)\n+{\n+\tstruct cascade_filter *cas = (struct cascade_filter *)filter;\n+\tfree_stream_filter(cas->one);\n+\tfree_stream_filter(cas->two);\n+\tfree(filter);\n+}\n+\n+static struct stream_filter_vtbl cascade_vtbl = {\n+\tcascade_filter_fn,\n+\tcascade_free_fn,\n+};\n+\n+static struct stream_filter *cascade_filter(struct stream_filter *one,\n+\t\t\t\t\t    struct stream_filter *two)\n+{\n+\tstruct cascade_filter *cascade;\n+\n+\tif (!one || is_null_stream_filter(one))\n+\t\treturn two;\n+\tif (!two || is_null_stream_filter(two))\n+\t\treturn one;\n+\n+\tcascade = xmalloc(sizeof(*cascade));\n+\tcascade->one = one;\n+\tcascade->two = two;\n+\tcascade->i_end = cascade->i_ptr = 0;\n+\tcascade->o_end = cascade->o_ptr = 0;\n+\tcascade->filter.vtbl = &cascade_vtbl;\n+\treturn (struct stream_filter *)cascade;\n+}\n+\n+/*\n+ * ident filter\n+ */\n+#define IDENT_DRAINING (-1)\n+#define IDENT_SKIPPING (-2)\n+struct ident_filter {\n+\tstruct stream_filter filter;\n+\tstruct strbuf left;\n+\tint state;\n+\tchar ident[45]; /* \": x40 $\" */\n+};\n+\n+static int is_foreign_ident(const char *str)\n+{\n+\tint i;\n+\n+\tif (prefixcmp(str, \"$Id: \"))\n+\t\treturn 0;\n+\tfor (i = 5; str[i]; i++) {\n+\t\tif (isspace(str[i]) && str[i+1] != '$')\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ident_filter_fn(struct stream_filter *filter,\n+\t\t\t   const char *input, size_t *isize_p,\n+\t\t\t   char *output, size_t *osize_p)\n+{\n+\tstruct ident_filter *ident = (struct ident_filter *)filter;\n+\tstatic const char head[] = \"$Id\";\n+\n+\twhile (*isize_p || (ident->state == IDENT_DRAINING)) {\n+\t\tint ch;\n+\n+\t\tif (ident->state == IDENT_DRAINING) {\n+\t\t\t/* Draining what is in left */\n+\t\t\tsize_t to_drain = ident->left.len;\n+\n+\t\t\tif (!to_drain) {\n+\t\t\t\tident->state = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!*osize_p)\n+\t\t\t\tbreak;\n+\t\t\tif (*osize_p < to_drain)\n+\t\t\t\tto_drain = *osize_p;\n+\t\t\tmemcpy(output, ident->left.buf, to_drain);\n+\t\t\tstrbuf_remove(&ident->left, 0, to_drain);\n+\t\t\toutput += to_drain;\n+\t\t\t(*osize_p) -= to_drain;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tch = *(input++);\n+\t\t(*isize_p)--;\n+\n+\t\tif (ident->state == IDENT_SKIPPING) {\n+\t\t\t/*\n+\t\t\t * Skipping until '$' or LF, but keeping them\n+\t\t\t * in case it is a foreign ident.\n+\t\t\t */\n+\t\t\tstrbuf_addch(&ident->left, ch);\n+\t\t\tif (ch != '\\n' && ch != '$')\n+\t\t\t\tcontinue;\n+\t\t\tif (ch == '$' && !is_foreign_ident(ident->left.buf)) {\n+\t\t\t\tstrbuf_reset(&ident->left);\n+\t\t\t\tstrbuf_addstr(&ident->left, head);\n+\t\t\t\tstrbuf_addstr(&ident->left, ident->ident);\n+\t\t\t}\n+\t\t\tident->state = IDENT_DRAINING;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (ident->state < sizeof(head) &&\n+\t\t    head[ident->state] == ch) {\n+\t\t\tident->state++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (ident->state)\n+\t\t\tstrbuf_add(&ident->left, head, ident->state);\n+\t\tif (ident->state == sizeof(head) - 1) {\n+\t\t\tif (ch != ':' && ch != '$') {\n+\t\t\t\tstrbuf_addch(&ident->left, ch);\n+\t\t\t\tident->state = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (ch == ':') {\n+\t\t\t\tstrbuf_addch(&ident->left, ch);\n+\t\t\t\tident->state = IDENT_SKIPPING;\n+\t\t\t} else {\n+\t\t\t\tstrbuf_addstr(&ident->left, ident->ident);\n+\t\t\t\tident->state = IDENT_DRAINING;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tstrbuf_addch(&ident->left, ch);\n+\t\tident->state = IDENT_DRAINING;\n+\t}\n+\treturn 0;\n+}\n+\n+static void ident_free_fn(struct stream_filter *filter)\n+{\n+\tstruct ident_filter *ident = (struct ident_filter *)filter;\n+\tstrbuf_release(&ident->left);\n+\tfree(filter);\n+}\n+\n+static struct stream_filter_vtbl ident_vtbl = {\n+\tident_filter_fn,\n+\tident_free_fn,\n+};\n+\n+static struct stream_filter *ident_filter(const unsigned char *sha1)\n+{\n+\tstruct ident_filter *ident = xmalloc(sizeof(*ident));\n+\n+\tsprintf(ident->ident, \": %s $\", sha1_to_hex(sha1));\n+\tstrbuf_init(&ident->left, 0);\n+\tident->filter.vtbl = &ident_vtbl;\n+\tident->state = 0;\n+\treturn (struct stream_filter *)ident;\n+}\n+\n /*\n  * Return an appropriately constructed filter for the path, or NULL if\n  * the contents cannot be filtered without reading the whole thing\n@@ -913,23 +1148,26 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n {\n \tstruct conv_attrs ca;\n \tenum crlf_action crlf_action;\n+\tstruct stream_filter *filter = NULL;\n \n \tconvert_attrs(&ca, path);\n \n-\tif (ca.ident ||\n-\t    (ca.drv && (ca.drv->smudge || ca.drv->clean)))\n-\t\treturn NULL;\n+\tif (ca.drv && (ca.drv->smudge || ca.drv->clean))\n+\t\treturn filter;\n+\n+\tif (ca.ident)\n+\t\tfilter = ident_filter(sha1);\n \n \tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n \tif ((crlf_action == CRLF_BINARY) || (crlf_action == CRLF_INPUT) ||\n \t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n-\t\treturn &null_filter_singleton;\n+\t\tfilter = cascade_filter(filter, &null_filter_singleton);\n \n-\tif (output_eol(crlf_action) == EOL_CRLF &&\n-\t    !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))\n-\t\treturn &lf_to_crlf_filter_singleton;\n+\telse if (output_eol(crlf_action) == EOL_CRLF &&\n+\t\t !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))\n+\t\tfilter = cascade_filter(filter, &lf_to_crlf_filter_singleton);\n \n-\treturn NULL;\n+\treturn filter;\n }\n \n void free_stream_filter(struct stream_filter *filter)\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168431","messageId":"4DD82931.6000101@lsrfire.ath.cx","threadId":"27422","inReplyTo":"1305961127-26540-5-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 4/4] streaming filter: ident filter and filter cascading","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-05-21T21:05:53Z","receivedAt":"2011-05-21T21:05:53Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 21.05.2011 08:58, schrieb Junio C Hamano:\n> Add support for \"ident\" filter on the output codepath. Because we now have\n> more than one filter active, also add support to cascade them together.\n> \n> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> ---\n> \n>  * There is one known issue in this \"ident\" filter implementation. If the\n>    input data ends while the filter is still in the \"SKIPPING\" state, it\n>    should go to the \"DRAINING\" codepath using the same logic that triggers\n>    when it gets to the end of line, copying out what was already queued.\n>    However, the API does not offer an explicit way for the input side to\n>    say \"that's all\". We probably need to make a zero-byte input to the\n>    filter function as an explicit EOF signal or something---currently I\n>    think it will loop forever, but it is getting late, so I will not be\n>    further touching this code tonight.\n\nAFAIU it will leave skipping mode once there is no input data anymore\ninstead of looping forever.  The accumulated stuff will be discarded.\n\nThe test suite is passed, by the way, even though t0021 seems to contain\na test for this issue (NoTerminatingSymbolAtEOF).  It uses echo to\ncreate the files so there will be a terminating newline before the end\nof the file, though, so this subtest probably needs to be changed.\n\n> \n>  convert.c |  254 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n>  1 files changed, 246 insertions(+), 8 deletions(-)\n> \n\nPerhaps split off the cascading part into its own patch?\n\n> diff --git a/convert.c b/convert.c\n> index 82fb75b..6b90c4d 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -868,6 +868,10 @@ int is_null_stream_filter(struct stream_filter *filter)\n>  \treturn filter == &null_filter_singleton;\n>  }\n>  \n> +\n> +/*\n> + * LF-to-CRLF filter\n> + */\n>  static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n>  \t\t\t\tconst char *input, size_t *isize_p,\n>  \t\t\t\tchar *output, size_t *osize_p)\n> @@ -901,6 +905,237 @@ static struct stream_filter lf_to_crlf_filter_singleton = {\n>  \t&lf_to_crlf_vtbl,\n>  };\n>  \n> +\n> +/*\n> + * Cascade filter\n> + */\n> +#define FILTER_BUFFER 1024\n> +struct cascade_filter {\n> +\tstruct stream_filter filter;\n> +\tstruct stream_filter *one;\n> +\tstruct stream_filter *two;\n> +\tchar ibuf[FILTER_BUFFER];\n> +\tchar obuf[FILTER_BUFFER];\n> +\tint i_end, i_ptr;\n> +\tint o_end, o_ptr;\n> +};\n> +\n> +static int cascade_filter_fn(struct stream_filter *filter,\n> +\t\t\t     const char *input, size_t *isize_p,\n> +\t\t\t     char *output, size_t *osize_p)\n> +{\n> +\tstruct cascade_filter *cas = (struct cascade_filter *) filter;\n> +\tsize_t filled = 0;\n> +\tsize_t sz = *osize_p;\n> +\tsize_t to_feed, remaining;\n> +\n> +\t/*\n> +\t * input --> ibuf --(one)--> obuf --(two)--> output\n> +\t */\n> +\twhile (filled < sz) {\n> +\t\t/* do we already have output to feed two? */\n> +\t\tif (cas->o_ptr < cas->o_end) {\n> +\t\t\tto_feed = cas->o_end - cas->o_ptr;\n> +\t\t\tremaining = sz - filled;\n> +\t\t\tif (stream_filter(cas->two,\n> +\t\t\t\t\t  cas->obuf + cas->o_ptr, &to_feed,\n> +\t\t\t\t\t  output + filled, &remaining))\n> +\t\t\t\treturn -1;\n> +\t\t\tcas->o_ptr += (cas->o_end - cas->o_ptr) - to_feed;\n> +\t\t\tfilled = sz - remaining;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tcas->o_end = cas->o_ptr = 0;\n> +\n> +\t\t/* do we have anything to feed one? */\n> +\t\tif (cas->i_ptr < cas->i_end) {\n> +\t\t\tto_feed = cas->i_end - cas->i_ptr;\n> +\t\t\tremaining = sizeof(cas->obuf);\n> +\t\t\tif (stream_filter(cas->one,\n> +\t\t\t\t\t  cas->ibuf + cas->i_ptr, &to_feed,\n> +\t\t\t\t\t  cas->obuf, &remaining))\n> +\t\t\t\treturn -1;\n> +\t\t\tcas->i_ptr += (cas->i_end - cas->i_ptr) - to_feed;\n> +\t\t\tcas->o_end = sizeof(cas->obuf) - remaining;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tcas->i_end = cas->i_ptr = 0;\n> +\n> +\t\t/* read from our upstream */\n> +\t\tto_feed = *isize_p;\n> +\t\tif (sizeof(cas->ibuf) < to_feed)\n> +\t\t\tto_feed = sizeof(cas->ibuf);\n> +\t\tif (!to_feed)\n> +\t\t\tbreak;\n> +\t\tmemcpy(cas->ibuf, input, to_feed);\n> +\t\tinput += to_feed;\n> +\t\t*isize_p -= to_feed;\n> +\t\tcas->i_end = to_feed;\n> +\t}\n> +\t*osize_p -= filled;\n> +\treturn 0;\n> +}\n> +\n> +static void cascade_free_fn(struct stream_filter *filter)\n> +{\n> +\tstruct cascade_filter *cas = (struct cascade_filter *)filter;\n> +\tfree_stream_filter(cas->one);\n> +\tfree_stream_filter(cas->two);\n> +\tfree(filter);\n> +}\n> +\n> +static struct stream_filter_vtbl cascade_vtbl = {\n> +\tcascade_filter_fn,\n> +\tcascade_free_fn,\n> +};\n> +\n> +static struct stream_filter *cascade_filter(struct stream_filter *one,\n> +\t\t\t\t\t    struct stream_filter *two)\n> +{\n> +\tstruct cascade_filter *cascade;\n> +\n> +\tif (!one || is_null_stream_filter(one))\n> +\t\treturn two;\n> +\tif (!two || is_null_stream_filter(two))\n> +\t\treturn one;\n\nThe is_null_stream_filter() check could be left out initially.  This\nwould allow for testing the cascading part with null filters.\n\n> +\n> +\tcascade = xmalloc(sizeof(*cascade));\n> +\tcascade->one = one;\n> +\tcascade->two = two;\n> +\tcascade->i_end = cascade->i_ptr = 0;\n> +\tcascade->o_end = cascade->o_ptr = 0;\n> +\tcascade->filter.vtbl = &cascade_vtbl;\n> +\treturn (struct stream_filter *)cascade;\n> +}\n> +\n> +/*\n> + * ident filter\n> + */\n> +#define IDENT_DRAINING (-1)\n> +#define IDENT_SKIPPING (-2)\n> +struct ident_filter {\n> +\tstruct stream_filter filter;\n> +\tstruct strbuf left;\n> +\tint state;\n> +\tchar ident[45]; /* \": x40 $\" */\n> +};\n> +\n> +static int is_foreign_ident(const char *str)\n> +{\n> +\tint i;\n> +\n> +\tif (prefixcmp(str, \"$Id: \"))\n> +\t\treturn 0;\n> +\tfor (i = 5; str[i]; i++) {\n> +\t\tif (isspace(str[i]) && str[i+1] != '$')\n> +\t\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static int ident_filter_fn(struct stream_filter *filter,\n> +\t\t\t   const char *input, size_t *isize_p,\n> +\t\t\t   char *output, size_t *osize_p)\n> +{\n> +\tstruct ident_filter *ident = (struct ident_filter *)filter;\n> +\tstatic const char head[] = \"$Id\";\n> +\n> +\twhile (*isize_p || (ident->state == IDENT_DRAINING)) {\n> +\t\tint ch;\n> +\n> +\t\tif (ident->state == IDENT_DRAINING) {\n> +\t\t\t/* Draining what is in left */\n> +\t\t\tsize_t to_drain = ident->left.len;\n> +\n> +\t\t\tif (!to_drain) {\n> +\t\t\t\tident->state = 0;\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t\tif (!*osize_p)\n> +\t\t\t\tbreak;\n> +\t\t\tif (*osize_p < to_drain)\n> +\t\t\t\tto_drain = *osize_p;\n> +\t\t\tmemcpy(output, ident->left.buf, to_drain);\n> +\t\t\tstrbuf_remove(&ident->left, 0, to_drain);\n> +\t\t\toutput += to_drain;\n> +\t\t\t(*osize_p) -= to_drain;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tch = *(input++);\n> +\t\t(*isize_p)--;\n> +\n> +\t\tif (ident->state == IDENT_SKIPPING) {\n> +\t\t\t/*\n> +\t\t\t * Skipping until '$' or LF, but keeping them\n> +\t\t\t * in case it is a foreign ident.\n> +\t\t\t */\n> +\t\t\tstrbuf_addch(&ident->left, ch);\n> +\t\t\tif (ch != '\\n' && ch != '$')\n> +\t\t\t\tcontinue;\n> +\t\t\tif (ch == '$' && !is_foreign_ident(ident->left.buf)) {\n> +\t\t\t\tstrbuf_reset(&ident->left);\n> +\t\t\t\tstrbuf_addstr(&ident->left, head);\n> +\t\t\t\tstrbuf_addstr(&ident->left, ident->ident);\n> +\t\t\t}\n> +\t\t\tident->state = IDENT_DRAINING;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (ident->state < sizeof(head) &&\n\n\t\t// minus one because otherwise we'd compare the\n\t\t// terminating NUL as well even though we're not\n\t\t// actually looking for a NUL\n\t\tif (ident->state < sizeof(head) - 1 &&\n\n> +\t\t    head[ident->state] == ch) {\n> +\t\t\tident->state++;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (ident->state)\n> +\t\t\tstrbuf_add(&ident->left, head, ident->state);\n> +\t\tif (ident->state == sizeof(head) - 1) {\n> +\t\t\tif (ch != ':' && ch != '$') {\n> +\t\t\t\tstrbuf_addch(&ident->left, ch);\n> +\t\t\t\tident->state = 0;\n\n\t\t\t\t// clean the plate right here and now;\n\t\t\t\t// can be combined with the case at the\n\t\t\t\t// end of the loop\n\t\t\t\tident->state = IDENT_DRAINING;\n\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (ch == ':') {\n> +\t\t\t\tstrbuf_addch(&ident->left, ch);\n> +\t\t\t\tident->state = IDENT_SKIPPING;\n> +\t\t\t} else {\n> +\t\t\t\tstrbuf_addstr(&ident->left, ident->ident);\n> +\t\t\t\tident->state = IDENT_DRAINING;\n> +\t\t\t}\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tstrbuf_addch(&ident->left, ch);\n> +\t\tident->state = IDENT_DRAINING;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static void ident_free_fn(struct stream_filter *filter)\n> +{\n> +\tstruct ident_filter *ident = (struct ident_filter *)filter;\n> +\tstrbuf_release(&ident->left);\n> +\tfree(filter);\n> +}\n> +\n> +static struct stream_filter_vtbl ident_vtbl = {\n> +\tident_filter_fn,\n> +\tident_free_fn,\n> +};\n> +\n> +static struct stream_filter *ident_filter(const unsigned char *sha1)\n> +{\n> +\tstruct ident_filter *ident = xmalloc(sizeof(*ident));\n> +\n> +\tsprintf(ident->ident, \": %s $\", sha1_to_hex(sha1));\n> +\tstrbuf_init(&ident->left, 0);\n> +\tident->filter.vtbl = &ident_vtbl;\n> +\tident->state = 0;\n> +\treturn (struct stream_filter *)ident;\n> +}\n> +\n>  /*\n>   * Return an appropriately constructed filter for the path, or NULL if\n>   * the contents cannot be filtered without reading the whole thing\n> @@ -913,23 +1148,26 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n>  {\n>  \tstruct conv_attrs ca;\n>  \tenum crlf_action crlf_action;\n> +\tstruct stream_filter *filter = NULL;\n>  \n>  \tconvert_attrs(&ca, path);\n>  \n> -\tif (ca.ident ||\n> -\t    (ca.drv && (ca.drv->smudge || ca.drv->clean)))\n> -\t\treturn NULL;\n> +\tif (ca.drv && (ca.drv->smudge || ca.drv->clean))\n> +\t\treturn filter;\n> +\n> +\tif (ca.ident)\n> +\t\tfilter = ident_filter(sha1);\n>  \n>  \tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n>  \tif ((crlf_action == CRLF_BINARY) || (crlf_action == CRLF_INPUT) ||\n>  \t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n> -\t\treturn &null_filter_singleton;\n> +\t\tfilter = cascade_filter(filter, &null_filter_singleton);\n>  \n> -\tif (output_eol(crlf_action) == EOL_CRLF &&\n> -\t    !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))\n> -\t\treturn &lf_to_crlf_filter_singleton;\n> +\telse if (output_eol(crlf_action) == EOL_CRLF &&\n> +\t\t !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))\n> +\t\tfilter = cascade_filter(filter, &lf_to_crlf_filter_singleton);\n>  \n> -\treturn NULL;\n> +\treturn filter;\n>  }\n>  \n>  void free_stream_filter(struct stream_filter *filter)\n"},{"id":"168432","messageId":"4DD82DB2.8000101@lsrfire.ath.cx","threadId":"27422","inReplyTo":"4DD82931.6000101@lsrfire.ath.cx","subject":"Re: [PATCH 4/4] streaming filter: ident filter and filter cascading","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-05-21T21:25:06Z","receivedAt":"2011-05-21T21:25:06Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 21.05.2011 23:05, schrieb René Scharfe:\n> Am 21.05.2011 08:58, schrieb Junio C Hamano:\n>> Add support for \"ident\" filter on the output codepath. Because we now have\n>> more than one filter active, also add support to cascade them together.\n>>\n>> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n>> ---\n>>\n>>  * There is one known issue in this \"ident\" filter implementation. If the\n>>    input data ends while the filter is still in the \"SKIPPING\" state, it\n>>    should go to the \"DRAINING\" codepath using the same logic that triggers\n>>    when it gets to the end of line, copying out what was already queued.\n>>    However, the API does not offer an explicit way for the input side to\n>>    say \"that's all\". We probably need to make a zero-byte input to the\n>>    filter function as an explicit EOF signal or something---currently I\n>>    think it will loop forever, but it is getting late, so I will not be\n>>    further touching this code tonight.\n> \n> AFAIU it will leave skipping mode once there is no input data anymore\n> instead of looping forever.  The accumulated stuff will be discarded.\n> \n> The test suite is passed, by the way, even though t0021 seems to contain\n> a test for this issue (NoTerminatingSymbolAtEOF).  It uses echo to\n> create the files so there will be a terminating newline before the end\n> of the file, though, so this subtest probably needs to be changed.\n\n-- >8 --\nSubject: t0021-conversion.sh: fix NoTerminatingSymbolAtEOF test\n\nThe last line of the test file \"expanded-keywords\" ended in a newline,\nwhich is a valid terminator for ident.  Use printf instead of echo to omit\nit and thus really test if a file that ends unexpectedly in the middle of\nan ident tag is handled properly.\n\nAlso take the oppertunity to calculate the expected ID dynamically\ninstead of hardcoding it into the test script.  This should make future\nchanges easier.\n\nSigned-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>\n---\n t/t0021-conversion.sh |   23 ++++++++++++-----------\n 1 files changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 9078b84..275421e 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -66,25 +66,26 @@ test_expect_success expanded_in_repo '\n \t\techo \"\\$Id:NoSpaceAtEitherEnd\\$\"\n \t\techo \"\\$Id: NoTerminatingSymbol\"\n \t\techo \"\\$Id: Foreign Commit With Spaces \\$\"\n-\t\techo \"\\$Id: NoTerminatingSymbolAtEOF\"\n+\t\tprintf \"\\$Id: NoTerminatingSymbolAtEOF\"\n \t} > expanded-keywords &&\n \n+\tgit add expanded-keywords &&\n+\tgit commit -m \"File with keywords expanded\" &&\n+\tid=$(git rev-parse --verify :expanded-keywords) &&\n+\n \t{\n \t\techo \"File with expanded keywords\"\n-\t\techo \"\\$Id: fd0478f5f1486f3d5177d4c3f6eb2765e8fc56b9 \\$\"\n-\t\techo \"\\$Id: fd0478f5f1486f3d5177d4c3f6eb2765e8fc56b9 \\$\"\n-\t\techo \"\\$Id: fd0478f5f1486f3d5177d4c3f6eb2765e8fc56b9 \\$\"\n-\t\techo \"\\$Id: fd0478f5f1486f3d5177d4c3f6eb2765e8fc56b9 \\$\"\n-\t\techo \"\\$Id: fd0478f5f1486f3d5177d4c3f6eb2765e8fc56b9 \\$\"\n-\t\techo \"\\$Id: fd0478f5f1486f3d5177d4c3f6eb2765e8fc56b9 \\$\"\n+\t\techo \"\\$Id: $id \\$\"\n+\t\techo \"\\$Id: $id \\$\"\n+\t\techo \"\\$Id: $id \\$\"\n+\t\techo \"\\$Id: $id \\$\"\n+\t\techo \"\\$Id: $id \\$\"\n+\t\techo \"\\$Id: $id \\$\"\n \t\techo \"\\$Id: NoTerminatingSymbol\"\n \t\techo \"\\$Id: Foreign Commit With Spaces \\$\"\n-\t\techo \"\\$Id: NoTerminatingSymbolAtEOF\"\n+\t\tprintf \"\\$Id: NoTerminatingSymbolAtEOF\"\n \t} > expected-output &&\n \n-\tgit add expanded-keywords &&\n-\tgit commit -m \"File with keywords expanded\" &&\n-\n \techo \"expanded-keywords ident\" >> .gitattributes &&\n \n \trm -f expanded-keywords &&\n-- \n1.7.5.2\n"},{"id":"168433","messageId":"4DD837EE.3010608@lsrfire.ath.cx","threadId":"27422","inReplyTo":"4DD82931.6000101@lsrfire.ath.cx","subject":"Re: [PATCH 4/4] streaming filter: ident filter and filter cascading","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-05-21T22:08:46Z","receivedAt":"2011-05-21T22:08:46Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 21.05.2011 23:05, schrieb René Scharfe:\n>> +\t\tif (ident->state < sizeof(head) &&\n> \n> \t\t// minus one because otherwise we'd compare the\n> \t\t// terminating NUL as well even though we're not\n> \t\t// actually looking for a NUL\n> \t\tif (ident->state < sizeof(head) - 1 &&\n\nPossibly, but that doesn't matter, as the right number of characters is\nremembered and the second test below is not passed if we sailed past the\nNUL.  Sorry for the noise.\n\n> \n>> +\t\t    head[ident->state] == ch) {\n>> +\t\t\tident->state++;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tif (ident->state)\n>> +\t\t\tstrbuf_add(&ident->left, head, ident->state);\n>> +\t\tif (ident->state == sizeof(head) - 1) {\n"},{"id":"168435","messageId":"7v1uzrzh04.fsf@alter.siamese.dyndns.org","threadId":"27422","inReplyTo":"4DD837EE.3010608@lsrfire.ath.cx","subject":"Re: [PATCH 4/4] streaming filter: ident filter and filter cascading","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-22T01:00:27Z","receivedAt":"2011-05-22T01:00:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:\n\n> Am 21.05.2011 23:05, schrieb René Scharfe:\n>>> +\t\tif (ident->state < sizeof(head) &&\n>> \n>> \t\t// minus one because otherwise we'd compare the\n>> \t\t// terminating NUL as well even though we're not\n>> \t\t// actually looking for a NUL\n>> \t\tif (ident->state < sizeof(head) - 1 &&\n>\n> Possibly, but that doesn't matter, as the right number of characters is\n> remembered and the second test below is not passed if we sailed past the\n> NUL.  Sorry for the noise.\n\nThanks for a careful reading.\n\nIn the second re-roll, I'll split the cascade and ident into two patches\n(ident comes first, with a fake cascade that returns NULL to indicate\nthere is no cascading supported when two \"real\" filters are given). Also I\ndo not need two buffers in the cascade. The input to cascade can be fed\ndirectly to the input of cascade->one to fill the cascade's buffer, and we\ncan feed cascade->two with what is in the cascade's buffer to directly\ndrain to the output of the cascade.\n\nBut that won't happen today.\n"},{"id":"168619","messageId":"7vy61v612b.fsf@alter.siamese.dyndns.org","threadId":"27422","inReplyTo":"4DD82DB2.8000101@lsrfire.ath.cx","subject":"Re: [PATCH 4/4] streaming filter: ident filter and filter cascading","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-25T01:06:36Z","receivedAt":"2011-05-25T01:06:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:\n\n> Subject: t0021-conversion.sh: fix NoTerminatingSymbolAtEOF test\n\nThanks.\n"},{"id":"168622","messageId":"7vlixv5xqj.fsf_-_@alter.siamese.dyndns.org","threadId":"27422","inReplyTo":"7vy61v612b.fsf@alter.siamese.dyndns.org","subject":"[PATCH] t0021: test application of both crlf and ident","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-25T02:18:28Z","receivedAt":"2011-05-25T02:18:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n\n * This comes on top of René's updates to the test. \n\n t/t0021-conversion.sh |   32 ++++++++++++++++++++++++--------\n 1 files changed, 24 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 275421e..f19e651 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -66,10 +66,14 @@ test_expect_success expanded_in_repo '\n \t\techo \"\\$Id:NoSpaceAtEitherEnd\\$\"\n \t\techo \"\\$Id: NoTerminatingSymbol\"\n \t\techo \"\\$Id: Foreign Commit With Spaces \\$\"\n-\t\tprintf \"\\$Id: NoTerminatingSymbolAtEOF\"\n-\t} > expanded-keywords &&\n+\t} >expanded-keywords.0 &&\n \n-\tgit add expanded-keywords &&\n+\t{\n+\t\tcat expanded-keywords.0 &&\n+\t\tprintf \"\\$Id: NoTerminatingSymbolAtEOF\"\n+\t} >expanded-keywords &&\n+\tcat expanded-keywords >expanded-keywords-crlf &&\n+\tgit add expanded-keywords expanded-keywords-crlf &&\n \tgit commit -m \"File with keywords expanded\" &&\n \tid=$(git rev-parse --verify :expanded-keywords) &&\n \n@@ -83,15 +87,27 @@ test_expect_success expanded_in_repo '\n \t\techo \"\\$Id: $id \\$\"\n \t\techo \"\\$Id: NoTerminatingSymbol\"\n \t\techo \"\\$Id: Foreign Commit With Spaces \\$\"\n+\t} >expected-output.0 &&\n+\t{\n+\t\tcat expected-output.0 &&\n \t\tprintf \"\\$Id: NoTerminatingSymbolAtEOF\"\n-\t} > expected-output &&\n+\t} >expected-output &&\n+\t{\n+\t\tappend_cr <expected-output.0 &&\n+\t\tprintf \"\\$Id: NoTerminatingSymbolAtEOF\"\n+\t} >expected-output-crlf &&\n+\t{\n+\t\techo \"expanded-keywords ident\"\n+\t\techo \"expanded-keywords-crlf ident text eol=crlf\"\n+\t} >>.gitattributes &&\n \n-\techo \"expanded-keywords ident\" >> .gitattributes &&\n+\trm -f expanded-keywords expanded-keywords-crlf &&\n \n-\trm -f expanded-keywords &&\n \tgit checkout -- expanded-keywords &&\n-\tcat expanded-keywords &&\n-\tcmp expanded-keywords expected-output\n+\ttest_cmp expanded-keywords expected-output &&\n+\n+\tgit checkout -- expanded-keywords-crlf &&\n+\ttest_cmp expanded-keywords-crlf expected-output-crlf\n '\n \n # The use of %f in a filter definition is expanded to the path to\n-- \n1.7.5.2.483.gc61ca\n"},{"id":"168623","messageId":"7vfwo35xiv.fsf_-_@alter.siamese.dyndns.org","threadId":"27422","inReplyTo":"7vy61v612b.fsf@alter.siamese.dyndns.org","subject":"[PATCH v2?] streaming: filter cascading","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-25T02:23:04Z","receivedAt":"2011-05-25T02:23:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This implements an internal \"cascade\" filter mechanism that plugs\ntwo filters in series.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n\n * I said that I suspect cascade was not working when I sent out \"What's\n   cooking\", and I was right X-<.\n\n   This replaces what has been queued for a few days.\n\n convert.c |  126 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 files changed, 112 insertions(+), 14 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 0c42bcb..85939c2 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -915,6 +915,112 @@ static struct stream_filter lf_to_crlf_filter_singleton = {\n \n \n /*\n+ * Cascade filter\n+ */\n+#define FILTER_BUFFER 1024\n+struct cascade_filter {\n+\tstruct stream_filter filter;\n+\tstruct stream_filter *one;\n+\tstruct stream_filter *two;\n+\tchar buf[FILTER_BUFFER];\n+\tint end, ptr;\n+};\n+\n+static int cascade_filter_fn(struct stream_filter *filter,\n+\t\t\t     const char *input, size_t *isize_p,\n+\t\t\t     char *output, size_t *osize_p)\n+{\n+\tstruct cascade_filter *cas = (struct cascade_filter *) filter;\n+\tsize_t filled = 0;\n+\tsize_t sz = *osize_p;\n+\tsize_t to_feed, remaining;\n+\n+\t/*\n+\t * input -- (one) --> buf -- (two) --> output\n+\t */\n+\twhile (filled < sz) {\n+\t\tremaining = sz - filled;\n+\n+\t\t/* do we already have something to feed two with? */\n+\t\tif (cas->ptr < cas->end) {\n+\t\t\tto_feed = cas->end - cas->ptr;\n+\t\t\tif (stream_filter(cas->two,\n+\t\t\t\t\t  cas->buf + cas->ptr, &to_feed,\n+\t\t\t\t\t  output + filled, &remaining))\n+\t\t\t\treturn -1;\n+\t\t\tcas->ptr += (cas->end - cas->ptr) - to_feed;\n+\t\t\tfilled = sz - remaining;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* feed one from upstream and have it emit into our buffer */\n+\t\tto_feed = input ? *isize_p : 0;\n+\t\tif (input && !to_feed)\n+\t\t\tbreak;\n+\t\tremaining = sizeof(cas->buf);\n+\t\tif (stream_filter(cas->one,\n+\t\t\t\t  input, &to_feed,\n+\t\t\t\t  cas->buf, &remaining))\n+\t\t\treturn -1;\n+\t\tcas->end = sizeof(cas->buf) - remaining;\n+\t\tcas->ptr = 0;\n+\t\tif (input) {\n+\t\t\tsize_t fed = *isize_p - to_feed;\n+\t\t\t*isize_p -= fed;\n+\t\t\tinput += fed;\n+\t\t}\n+\n+\t\t/* do we know that we drained one completely? */\n+\t\tif (input || cas->end)\n+\t\t\tcontinue;\n+\n+\t\t/* tell two to drain; we have nothing more to give it */\n+\t\tto_feed = 0;\n+\t\tremaining = sz - filled;\n+\t\tif (stream_filter(cas->two,\n+\t\t\t\t  NULL, &to_feed,\n+\t\t\t\t  output + filled, &remaining))\n+\t\t\treturn -1;\n+\t\tif (remaining == (sz - filled))\n+\t\t\tbreak; /* completely drained two */\n+\t\tfilled = sz - remaining;\n+\t}\n+\t*osize_p -= filled;\n+\treturn 0;\n+}\n+\n+static void cascade_free_fn(struct stream_filter *filter)\n+{\n+\tstruct cascade_filter *cas = (struct cascade_filter *)filter;\n+\tfree_stream_filter(cas->one);\n+\tfree_stream_filter(cas->two);\n+\tfree(filter);\n+}\n+\n+static struct stream_filter_vtbl cascade_vtbl = {\n+\tcascade_filter_fn,\n+\tcascade_free_fn,\n+};\n+\n+static struct stream_filter *cascade_filter(struct stream_filter *one,\n+\t\t\t\t\t    struct stream_filter *two)\n+{\n+\tstruct cascade_filter *cascade;\n+\n+\tif (!one || is_null_stream_filter(one))\n+\t\treturn two;\n+\tif (!two || is_null_stream_filter(two))\n+\t\treturn one;\n+\n+\tcascade = xmalloc(sizeof(*cascade));\n+\tcascade->one = one;\n+\tcascade->two = two;\n+\tcascade->end = cascade->ptr = 0;\n+\tcascade->filter.vtbl = &cascade_vtbl;\n+\treturn (struct stream_filter *)cascade;\n+}\n+\n+/*\n  * ident filter\n  */\n #define IDENT_DRAINING (-1)\n@@ -1083,20 +1189,12 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n \n \tif ((crlf_action == CRLF_BINARY) || (crlf_action == CRLF_INPUT) ||\n-\t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE)) {\n-\t\tif (filter) {\n-\t\t\tfree_stream_filter(filter);\n-\t\t\treturn NULL;\n-\t\t}\n-\t\treturn &null_filter_singleton;\n-\t} else if (output_eol(crlf_action) == EOL_CRLF &&\n-\t\t   !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS)) {\n-\t\tif (filter) {\n-\t\t\tfree_stream_filter(filter);\n-\t\t\treturn NULL;\n-\t\t}\n-\t\treturn &lf_to_crlf_filter_singleton;\n-\t}\n+\t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n+\t\tfilter = cascade_filter(filter, &null_filter_singleton);\n+\n+\telse if (output_eol(crlf_action) == EOL_CRLF &&\n+\t\t !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))\n+\t\tfilter = cascade_filter(filter, &lf_to_crlf_filter_singleton);\n \n \treturn filter;\n }\n-- \n1.7.5.2.483.gc61ca\n"}]}