{"thread":{"id":"48059","subject":"[PATCH 2/2] json-writer: unit test","startedAt":"2018-03-16T20:00:43Z","lastAt":"2018-03-20T16:44:29Z","messageCount":11,"participants":["git@jeffhostetler.com","Jeff King","Ævar Arnfjörð Bjarmason","Jacob Keller","Jeff Hostetler"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"341914","messageId":"20180316194057.77513-3-git@jeffhostetler.com","threadId":"48059","inReplyTo":"20180316194057.77513-1-git@jeffhostetler.com","subject":"[PATCH 2/2] json-writer: unit test","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2018-03-16T19:40:57Z","receivedAt":"2018-03-16T20:00:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Makefile                    |   1 +\n t/helper/test-json-writer.c | 146 ++++++++++++++++++++++++++++++++++++++++++++\n t/t0019-json-writer.sh      |  10 +++\n 3 files changed, 157 insertions(+)\n create mode 100644 t/helper/test-json-writer.c\n create mode 100755 t/t0019-json-writer.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9000369..57f58e6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh\n TEST_PROGRAMS_NEED_X += test-genrandom\n TEST_PROGRAMS_NEED_X += test-hashmap\n TEST_PROGRAMS_NEED_X += test-index-version\n+TEST_PROGRAMS_NEED_X += test-json-writer\n TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash\n TEST_PROGRAMS_NEED_X += test-line-buffer\n TEST_PROGRAMS_NEED_X += test-match-trees\ndiff --git a/t/helper/test-json-writer.c b/t/helper/test-json-writer.c\nnew file mode 100644\nindex 0000000..bb43efb\n--- /dev/null\n+++ b/t/helper/test-json-writer.c\n@@ -0,0 +1,146 @@\n+#include \"cache.h\"\n+#include \"json-writer.h\"\n+\n+const char *expect_obj1 = \"{\\\"a\\\":\\\"abc\\\",\\\"b\\\":42,\\\"c\\\":true}\";\n+const char *expect_obj2 = \"{\\\"a\\\":-1,\\\"b\\\":2147483647,\\\"c\\\":0}\";\n+const char *expect_obj3 = \"{\\\"a\\\":0,\\\"b\\\":4294967295,\\\"c\\\":18446744073709551615}\";\n+const char *expect_obj4 = \"{\\\"t\\\":true,\\\"f\\\":false,\\\"n\\\":null}\";\n+const char *expect_obj5 = \"{\\\"abc\\\\tdef\\\":\\\"abc\\\\\\\\def\\\"}\";\n+\n+struct strbuf obj1 = STRBUF_INIT;\n+struct strbuf obj2 = STRBUF_INIT;\n+struct strbuf obj3 = STRBUF_INIT;\n+struct strbuf obj4 = STRBUF_INIT;\n+struct strbuf obj5 = STRBUF_INIT;\n+\n+void make_obj1(void)\n+{\n+\tjw_object_begin(&obj1);\n+\tjw_object_append_string(&obj1, \"a\", \"abc\");\n+\tjw_object_append_int(&obj1, \"b\", 42);\n+\tjw_object_append_true(&obj1, \"c\");\n+\tjw_object_end(&obj1);\n+}\n+\n+void make_obj2(void)\n+{\n+\tjw_object_begin(&obj2);\n+\tjw_object_append_int(&obj2, \"a\", -1);\n+\tjw_object_append_int(&obj2, \"b\", 0x7fffffff);\n+\tjw_object_append_int(&obj2, \"c\", 0);\n+\tjw_object_end(&obj2);\n+}\n+\n+void make_obj3(void)\n+{\n+\tjw_object_begin(&obj3);\n+\tjw_object_append_uint64(&obj3, \"a\", 0);\n+\tjw_object_append_uint64(&obj3, \"b\", 0xffffffff);\n+\tjw_object_append_uint64(&obj3, \"c\", 0xffffffffffffffff);\n+\tjw_object_end(&obj3);\n+}\n+\n+void make_obj4(void)\n+{\n+\tjw_object_begin(&obj4);\n+\tjw_object_append_true(&obj4, \"t\");\n+\tjw_object_append_false(&obj4, \"f\");\n+\tjw_object_append_null(&obj4, \"n\");\n+\tjw_object_end(&obj4);\n+}\n+\n+void make_obj5(void)\n+{\n+\tjw_object_begin(&obj5);\n+\tjw_object_append_string(&obj5, \"abc\" \"\\x09\" \"def\", \"abc\" \"\\\\\" \"def\");\n+\tjw_object_end(&obj5);\n+}\n+\n+const char *expect_arr1 = \"[\\\"abc\\\",42,true]\";\n+const char *expect_arr2 = \"[-1,2147483647,0]\";\n+const char *expect_arr3 = \"[0,4294967295,18446744073709551615]\";\n+const char *expect_arr4 = \"[true,false,null]\";\n+\n+struct strbuf arr1 = STRBUF_INIT;\n+struct strbuf arr2 = STRBUF_INIT;\n+struct strbuf arr3 = STRBUF_INIT;\n+struct strbuf arr4 = STRBUF_INIT;\n+\n+void make_arr1(void)\n+{\n+\tjw_array_begin(&arr1);\n+\tjw_array_append_string(&arr1, \"abc\");\n+\tjw_array_append_int(&arr1, 42);\n+\tjw_array_append_true(&arr1);\n+\tjw_array_end(&arr1);\n+}\n+\n+void make_arr2(void)\n+{\n+\tjw_array_begin(&arr2);\n+\tjw_array_append_int(&arr2, -1);\n+\tjw_array_append_int(&arr2, 0x7fffffff);\n+\tjw_array_append_int(&arr2, 0);\n+\tjw_array_end(&arr2);\n+}\n+\n+void make_arr3(void)\n+{\n+\tjw_array_begin(&arr3);\n+\tjw_array_append_uint64(&arr3, 0);\n+\tjw_array_append_uint64(&arr3, 0xffffffff);\n+\tjw_array_append_uint64(&arr3, 0xffffffffffffffff);\n+\tjw_array_end(&arr3);\n+}\n+\n+void make_arr4(void)\n+{\n+\tjw_array_begin(&arr4);\n+\tjw_array_append_true(&arr4);\n+\tjw_array_append_false(&arr4);\n+\tjw_array_append_null(&arr4);\n+\tjw_array_end(&arr4);\n+}\n+\n+char *expect_nest1 =\n+\t\"{\\\"obj1\\\":{\\\"a\\\":\\\"abc\\\",\\\"b\\\":42,\\\"c\\\":true},\\\"arr1\\\":[\\\"abc\\\",42,true]}\";\n+\n+struct strbuf nest1 = STRBUF_INIT;\n+\n+void make_nest1(void)\n+{\n+\tjw_object_begin(&nest1);\n+\tjw_object_append_object(&nest1, \"obj1\", obj1.buf);\n+\tjw_object_append_array(&nest1, \"arr1\", arr1.buf);\n+\tjw_object_end(&nest1);\n+}\n+\n+void cmp(const char *test, const struct strbuf *buf, const char *exp)\n+{\n+\tif (!strcmp(buf->buf, exp))\n+\t\treturn;\n+\n+\tprintf(\"error[%s]: observed '%s' expected '%s'\\n\",\n+\t       test, buf->buf, exp);\n+\texit(1);\n+}\n+\n+#define t(v) do { make_##v(); cmp(#v, &v, expect_##v); } while (0)\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tt(obj1);\n+\tt(obj2);\n+\tt(obj3);\n+\tt(obj4);\n+\tt(obj5);\n+\n+\tt(arr1);\n+\tt(arr2);\n+\tt(arr3);\n+\tt(arr4);\n+\n+\tt(nest1);\n+\n+\treturn 0;\n+}\ndiff --git a/t/t0019-json-writer.sh b/t/t0019-json-writer.sh\nnew file mode 100755\nindex 0000000..7b86087\n--- /dev/null\n+++ b/t/t0019-json-writer.sh\n@@ -0,0 +1,10 @@\n+#!/bin/sh\n+\n+test_description='test json-writer JSON generation'\n+. ./test-lib.sh\n+\n+test_expect_success 'unit test of json-writer routines' '\n+\ttest-json-writer\n+'\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"341915","messageId":"20180316194057.77513-2-git@jeffhostetler.com","threadId":"48059","inReplyTo":"20180316194057.77513-1-git@jeffhostetler.com","subject":"[PATCH 1/2] json_writer: new routines to create data in JSON format","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2018-03-16T19:40:56Z","receivedAt":"2018-03-16T20:00:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd basic routines to generate data in JSON format.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Makefile      |   1 +\n json-writer.c | 224 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n json-writer.h | 120 +++++++++++++++++++++++++++++++\n 3 files changed, 345 insertions(+)\n create mode 100644 json-writer.c\n create mode 100644 json-writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 1a9b23b..9000369 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -815,6 +815,7 @@ LIB_OBJS += hashmap.o\n LIB_OBJS += help.o\n LIB_OBJS += hex.o\n LIB_OBJS += ident.o\n+LIB_OBJS += json-writer.o\n LIB_OBJS += kwset.o\n LIB_OBJS += levenshtein.o\n LIB_OBJS += line-log.o\ndiff --git a/json-writer.c b/json-writer.c\nnew file mode 100644\nindex 0000000..755ff80\n--- /dev/null\n+++ b/json-writer.c\n@@ -0,0 +1,224 @@\n+#include \"cache.h\"\n+#include \"json-writer.h\"\n+\n+/*\n+ * Append JSON-quoted version of the given string to 'out'.\n+ */\n+static void jw_append_quoted_string(struct strbuf *out, const char *in)\n+{\n+\tstrbuf_addch(out, '\"');\n+\tfor (/**/; *in; in++) {\n+\t\tunsigned char c = (unsigned char)*in;\n+\t\tif (c == '\"')\n+\t\t\tstrbuf_add(out, \"\\\\\\\"\", 2);\n+\t\telse if (c == '\\\\')\n+\t\t\tstrbuf_add(out, \"\\\\\\\\\", 2);\n+\t\telse if (c == '\\n')\n+\t\t\tstrbuf_add(out, \"\\\\n\", 2);\n+\t\telse if (c == '\\r')\n+\t\t\tstrbuf_add(out, \"\\\\r\", 2);\n+\t\telse if (c == '\\t')\n+\t\t\tstrbuf_add(out, \"\\\\t\", 2);\n+\t\telse if (c == '\\f')\n+\t\t\tstrbuf_add(out, \"\\\\f\", 2);\n+\t\telse if (c == '\\b')\n+\t\t\tstrbuf_add(out, \"\\\\b\", 2);\n+\t\telse if (c < 0x20)\n+\t\t\tstrbuf_addf(out, \"\\\\u%04x\", c);\n+\t\telse\n+\t\t\tstrbuf_addch(out, c);\n+\t}\n+\tstrbuf_addch(out, '\"');\n+}\n+\n+void jw_object_begin(struct strbuf *out)\n+{\n+\tstrbuf_reset(out);\n+\tstrbuf_addch(out, '{');\n+}\n+\n+void jw_object_append_string(struct strbuf *out, const char *key,\n+\t\t\t     const char *value)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addch(out, ':');\n+\tjw_append_quoted_string(out, value);\n+}\n+\n+void jw_object_append_int(struct strbuf *out, const char *key, int value)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addf(out, \":%d\", value);\n+}\n+\n+void jw_object_append_uint64(struct strbuf *out, const char *key, uint64_t value)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addf(out, \":%\"PRIuMAX, value);\n+}\n+\n+void jw_object_append_double(struct strbuf *out, const char *key, double value)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addf(out, \":%f\", value);\n+}\n+\n+void jw_object_append_true(struct strbuf *out, const char *key)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addstr(out, \":true\");\n+}\n+\n+void jw_object_append_false(struct strbuf *out, const char *key)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addstr(out, \":false\");\n+}\n+\n+void jw_object_append_null(struct strbuf *out, const char *key)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addstr(out, \":null\");\n+}\n+\n+void jw_object_append_object(struct strbuf *out, const char *key,\n+\t\t\t     const char *value)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addch(out, ':');\n+\tstrbuf_addstr(out, value);\n+}\n+\n+void jw_object_append_array(struct strbuf *out, const char *key,\n+\t\t\t    const char *value)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\n+\tjw_append_quoted_string(out, key);\n+\tstrbuf_addch(out, ':');\n+\tstrbuf_addstr(out, value);\n+}\n+\n+void jw_object_end(struct strbuf *out)\n+{\n+\tstrbuf_addch(out, '}');\n+}\n+\n+void jw_array_begin(struct strbuf *out)\n+{\n+\tstrbuf_reset(out);\n+\tstrbuf_addch(out, '[');\n+}\n+\n+void jw_array_append_string(struct strbuf *out, const char *elem)\n+{\n+\tstruct strbuf qe = STRBUF_INIT;\n+\n+\tjw_append_quoted_string(&qe, elem);\n+\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addstr(out, qe.buf);\n+\n+\tstrbuf_release(&qe);\n+}\n+\n+void jw_array_append_int(struct strbuf *out, int elem)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addf(out, \"%d\", elem);\n+}\n+\n+void jw_array_append_uint64(struct strbuf *out, uint64_t elem)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addf(out, \"%\"PRIuMAX, elem);\n+}\n+\n+void jw_array_append_double(struct strbuf *out, double elem)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addf(out, \"%f\", elem);\n+}\n+\n+void jw_array_append_true(struct strbuf *out)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addstr(out, \"true\");\n+}\n+\n+void jw_array_append_false(struct strbuf *out)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addstr(out, \"false\");\n+}\n+\n+void jw_array_append_null(struct strbuf *out)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addstr(out, \"null\");\n+}\n+\n+void jw_array_append_object(struct strbuf *out, const char *obj)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addstr(out, obj);\n+}\n+\n+void jw_array_append_array(struct strbuf *out, const char *array)\n+{\n+\tif (out->len > 1)\n+\t\tstrbuf_addch(out, ',');\n+\tstrbuf_addstr(out, array);\n+}\n+\n+void jw_array_append_argc_argv(struct strbuf *out, int argc, const char **argv)\n+{\n+\tint k;\n+\n+\tfor (k = 0; k < argc; k++)\n+\t\tjw_array_append_string(out, argv[k]);\n+}\n+\n+void jw_array_append_argv(struct strbuf *out, const char **argv)\n+{\n+\twhile (*argv)\n+\t\tjw_array_append_string(out, *argv++);\n+}\n+\n+void jw_array_end(struct strbuf *out)\n+{\n+\tstrbuf_addch(out, ']');\n+}\ndiff --git a/json-writer.h b/json-writer.h\nnew file mode 100644\nindex 0000000..0a60ab3\n--- /dev/null\n+++ b/json-writer.h\n@@ -0,0 +1,120 @@\n+#ifndef JSON_WRITER_H\n+#define JSON_WRITER_H\n+\n+/*\n+ * JSON data structures are defined at:\n+ *      http://json.org/\n+ *      http://www.ietf.org/rfc/rfc7159.txt\n+ *\n+ * The JSON-writer API allows one to build JSON data structures using a\n+ * \"struct strbuf\" buffer.  This is intended as a simple API to build\n+ * output strings; it is not intended to be a general object model for\n+ * JSON data.\n+ *\n+ * All string values (both keys and string r-values) are properly quoted\n+ * and escaped if they contain special characters.\n+ *\n+ * These routines create compact JSON data (with no unnecessary whitespace,\n+ * newlines, or indenting).\n+ *\n+ * Both \"JSON objects\" (aka sets of k/v pairs) and \"JSON array\" can be\n+ * constructed using a 'begin append* end' model.\n+ *\n+ * Example JSON Object Usage:\n+ *\n+ *      struct strbuf obj1 = STRBUF_INIT;\n+ *\n+ *      jw_object_begin(&obj1);\n+ *          jw_object_append_string(&obj1, \"a\", \"abc\");\n+ *          jw_object_append_int(&obj1, \"b\", 42);\n+ *          jw_object_append_true(&obj1, \"c\");\n+ *      jw_object_end(&obj1);\n+ *\n+ *      printf(\"%s\\n\", obj1.buf);\n+ *\n+ * Should yield:   \n+ *\n+ *      {\"a\":\"abc\",\"b\":42,\"c\":true}\n+ *\n+ * Example JSON Array Usage:\n+ *\n+ *      struct strbuf arr1 = STRBUF_INIT;\n+ *\n+ *      jw_array_begin(&arr1);\n+ *          jw_array_append_string(&arr1, \"abc\");\n+ *          jw_array_append_int(&arr1, 42);\n+ *          jw_array_append_true(&arr1);\n+ *      jw_array_end(&arr1);\n+ *\n+ *      printf(\"%s\\n\", arr1.buf);\n+ *\n+ * Should yield:\n+ *\n+ *      [\"abc\",42,true]\n+ *\n+ * Nested JSON structures are also supported.  These should be composed bottom\n+ * up using multiple strbuf variables.\n+ *\n+ * Example Nested Usage (using the above \"obj1\" and \"arr1\" variables):\n+ *\n+ *       struct strbuf obj2 = STRBUF_INIT;\n+ *\n+ *       jw_object_begin(&obj2);\n+ *           jw_object_append_object(&obj2, \"obj1\", obj1.buf);\n+ *           jw_object_append_array(&obj2, \"arr1\", arr1.buf);\n+ *       jw_object_end(&obj2);\n+ *\n+ *       printf(\"%s\\n\", obj2.buf);\n+ *\n+ * Should yield:\n+ *\n+ *       {\"obj1\":{\"a\":\"abc\",\"b\":42,\"c\":true},\"arr1\":[\"abc\",42,true]}\n+ *\n+ * And:\n+ *\n+ *       struct strbuf arr2 = STRBUF_INIT;\n+ *\n+ *       jw_array_begin(&arr2);\n+ *           jw_array_append_object(&arr2, obj1.buf);\n+ *           jw_array_append_array(&arr2, arr1.buf);\n+ *       jw_array_end(&arr2);\n+ *\n+ *       printf(\"%s\\n\", arr2.buf);\n+ *\n+ * Should yield:\n+ *\n+ *       [{\"a\":\"abc\",\"b\":42,\"c\":true},[\"abc\",42,true]]\n+ *\n+ */\n+\n+void jw_object_begin(struct strbuf *out);\n+void jw_object_append_string(struct strbuf *out, const char *key,\n+\t\t\t     const char *value);\n+void jw_object_append_int(struct strbuf *out, const char *key, int value);\n+void jw_object_append_uint64(struct strbuf *out, const char *key,\n+\t\t\t     uint64_t value);\n+void jw_object_append_double(struct strbuf *out, const char *key, double value);\n+void jw_object_append_true(struct strbuf *out, const char *key);\n+void jw_object_append_false(struct strbuf *out, const char *key);\n+void jw_object_append_null(struct strbuf *out, const char *key);\n+void jw_object_append_object(struct strbuf *out, const char *key,\n+\t\t\t     const char *value);\n+void jw_object_append_array(struct strbuf *out, const char *key,\n+\t\t\t    const char *value);\n+void jw_object_end(struct strbuf *out);\n+\n+void jw_array_begin(struct strbuf *out);\n+void jw_array_append_string(struct strbuf *out, const char *elem);\n+void jw_array_append_int(struct strbuf *out, int elem);\n+void jw_array_append_uint64(struct strbuf *out, uint64_t elem);\n+void jw_array_append_double(struct strbuf *out, double elem);\n+void jw_array_append_true(struct strbuf *out);\n+void jw_array_append_false(struct strbuf *out);\n+void jw_array_append_null(struct strbuf *out);\n+void jw_array_append_object(struct strbuf *out, const char *obj);\n+void jw_array_append_array(struct strbuf *out, const char *array);\n+void jw_array_append_argc_argv(struct strbuf *out, int argc, const char **argv);\n+void jw_array_append_argv(struct strbuf *out, const char **argv);\n+void jw_array_end(struct strbuf *out);\n+\n+#endif /* JSON_WRITER_H */\n-- \n2.9.3\n\n"},{"id":"341916","messageId":"20180316194057.77513-1-git@jeffhostetler.com","threadId":"48059","inReplyTo":null,"subject":"[PATCH 0/2] routines to generate JSON data","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2018-03-16T19:40:55Z","receivedAt":"2018-03-16T20:00:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nThis patch series adds a set of utility routines to compose data in JSON\nformat into a \"struct strbuf\".  The resulting string can then be output\nby commands wanting to support a JSON output format.\n\nThis is a stand alone patch.  Nothing currently uses these routines.  I'm\ncurrently working on a series to log \"telemetry\" data (as we discussed\nbriefly during Ævar's \"Performance Misc\" session [1] in Barcelona last\nweek).  And I want emit the data in JSON rather than a fixed column/field\nformat.  The JSON routines here are independent of that, so it made sense\nto submit the JSON part by itself.\n\nBack when we added porcelain=v2 format to status, we talked about adding a\nJSON format.  I think the routines in this patch would let us easily do\nthat, if someone were interested.  (Extending status is not on my radar\nright now, however.)\n\nDocumentation for the new API is given in json-writer.h at the bottom of\nthe first patch.\n\nI wasn't sure how to unit test the API from a shell script, so I added a\nhelper command that does most of the work in the second patch.\n\n[1] https://public-inbox.org/git/20180313004940.GG61720@google.com/T/\n\n\nJeff Hostetler (2):\n  json_writer: new routines to create data in JSON format\n  json-writer: unit test\n\n Makefile                    |   2 +\n json-writer.c               | 224 ++++++++++++++++++++++++++++++++++++++++++++\n json-writer.h               | 120 ++++++++++++++++++++++++\n t/helper/test-json-writer.c | 146 +++++++++++++++++++++++++++++\n t/t0019-json-writer.sh      |  10 ++\n 5 files changed, 502 insertions(+)\n create mode 100644 json-writer.c\n create mode 100644 json-writer.h\n create mode 100644 t/helper/test-json-writer.c\n create mode 100755 t/t0019-json-writer.sh\n\n-- \n2.9.3\n\n"},{"id":"341943","messageId":"20180316211837.GB12333@sigill.intra.peff.net","threadId":"48059","inReplyTo":"20180316194057.77513-1-git@jeffhostetler.com","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-16T21:18:37Z","receivedAt":"2018-03-16T21:18:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 16, 2018 at 07:40:55PM +0000, git@jeffhostetler.com wrote:\n\n> This patch series adds a set of utility routines to compose data in JSON\n> format into a \"struct strbuf\".  The resulting string can then be output\n> by commands wanting to support a JSON output format.\n> \n> This is a stand alone patch.  Nothing currently uses these routines.  I'm\n> currently working on a series to log \"telemetry\" data (as we discussed\n> briefly during Ævar's \"Performance Misc\" session [1] in Barcelona last\n> week).  And I want emit the data in JSON rather than a fixed column/field\n> format.  The JSON routines here are independent of that, so it made sense\n> to submit the JSON part by itself.\n> \n> Back when we added porcelain=v2 format to status, we talked about adding a\n> JSON format.  I think the routines in this patch would let us easily do\n> that, if someone were interested.  (Extending status is not on my radar\n> right now, however.)\n\nI really like the idea of being able to send our machine-readable output\nin some \"standard\" syntax for which people may already have parsers. But\none big hangup with JSON is that it assumes all strings are UTF-8. That\nmay be OK for telemetry data, but it would probably lead to problems for\nsomething like status porcelain, since Git's view of paths is just a\nstring of bytes (not to mention possible uses elsewhere like author\nnames, subject lines, etc).\n\nBefore we commit to a standardized format, I think we need to work out\na solution there (because I'd much rather not go down this road for\ntelemetry data only to find that we cannot use the same standardized\nformat in other parts of Git).\n\nSome possible solutions I can think of:\n\n  1. Ignore the UTF-8 requirement, making a JSON-like output (which I\n     think is what your patches do). I'm not sure what problems this\n     might cause on the parsing side.\n\n  2. Specially encode non-UTF-8 bits. I'm not familiar enough with JSON\n     to know the options here, but my understanding is that numeric\n     escapes are just for inserting unicode code points. _Can_ you\n     actually transport arbitrary binary data across JSON without\n     base64-encoding it (yech)?\n\n  3. Some other similar format. YAML comes to mind. Last time I looked\n     (quite a while ago), it seemed insanely complex, but I think you\n     could implement only a reasonable subset. OTOH, I think the tools\n     ecosystem for parsing JSON (e.g., jq) is much better.\n\n> Documentation for the new API is given in json-writer.h at the bottom of\n> the first patch.\n\nThe API generally looks pleasant, but the nesting surprised me.  E.g.,\nI'd have expected:\n\n  jw_array_begin(out);\n  jw_array_begin(out);\n  jw_array_append_int(out, 42);\n  jw_array_end(out);\n  jw_array_end(out);\n\nto result in an array containing an array containing an integer. But\narray_begin() actually resets the strbuf, so you can't build up nested\nitems like this internally. Ditto for objects within objects. You have\nto use two separate strbufs and copy the data an extra time.\n\nTo make the above work, I think you'd have to store a little more state.\nE.g., the \"array_append\" functions check \"out->len\" to see if they need\nto add a separating comma. That wouldn't work if we might be part of a\nnested array. So I think you'd need a context struct like:\n\n  struct json_writer {\n    int first_item;\n    struct strbuf out;\n  };\n  #define JSON_WRITER_INIT { 1, STRBUF_INIT }\n\nto store the state and the output. As a bonus, you could also use it to\nstore some other sanity checks (e.g., keep a \"depth\" counter and BUG()\nwhen somebody tries to access the finished strbuf with a hanging-open\nobject or array).\n\n> I wasn't sure how to unit test the API from a shell script, so I added a\n> helper command that does most of the work in the second patch.\n\nIn general I'd really prefer to keep the shell script as the driver for\nthe tests, and have t/helper programs just be conduits. E.g., something\nlike:\n\n  cat >expect <<-\\EOF &&\n  {\"key\": \"value\", \"foo\": 42}\n  EOF\n  test-json-writer >actual \\\n    object_begin \\\n    object_append_string key value \\\n    object_append_int foo 42 \\\n    object_end &&\n  test_cmp expect actual\n\nIt's a bit tedious (though fairly mechanical) to expose the API in this\nway, but it makes it much easier to debug, modify, or add tests later\non (for example, I had to modify the C program to find out that my\nappend example above wouldn't work).\n\n-Peff\n"},{"id":"341964","messageId":"87tvtfd3sl.fsf@evledraar.gmail.com","threadId":"48059","inReplyTo":"20180316211837.GB12333@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-16T23:00:26Z","receivedAt":"2018-03-16T23:00:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 16 2018, Jeff King jotted:\n\n> I really like the idea of being able to send our machine-readable output\n> in some \"standard\" syntax for which people may already have parsers. But\n> one big hangup with JSON is that it assumes all strings are UTF-8.\n\nFWIW It's not UTF-8 but \"Unicode characters\", i.e. any Unicode encoding\nis valid, not that it changes anything you're pointing out, but people\non Win32 could use UTF-16 as-is if their filenames were in that format.\n\nI'm just going to use UTF-8 synonymously with \"Unicode encoding\" for the\nrest of this mail...\n\n> Some possible solutions I can think of:\n>\n>   1. Ignore the UTF-8 requirement, making a JSON-like output (which I\n>      think is what your patches do). I'm not sure what problems this\n>      might cause on the parsing side.\n\nMaybe some JSON parsers are more permissive, but they'll commonly just\ndie on non-Unicode (usually UTF-8) input, e.g.:\n\n    $ (echo -n '{\"str \": \"'; head -c 3 /dev/urandom ; echo -n '\"}') | perl -0666 -MJSON::XS -wE 'say decode_json(<>)->{str}'\n    malformed UTF-8 character in JSON string, at character offset 10 (before \"\\x{fffd}e\\x{fffd}\"}\") at -e line 1, <> chunk 1.\n\n>   2. Specially encode non-UTF-8 bits. I'm not familiar enough with JSON\n>      to know the options here, but my understanding is that numeric\n>      escapes are just for inserting unicode code points. _Can_ you\n>      actually transport arbitrary binary data across JSON without\n>      base64-encoding it (yech)?\n\nThere's no way to transfer binary data in JSON without it being shoved\ninto a UTF-8 encoding, so you'd need to know on the other side that\nsuch-and-such a field has binary in it, i.e. you'll need to invent your\nown schema.\n\nE.g.:\n\n    head -c 10 /dev/urandom | perl -MDevel::Peek -MJSON::XS -wE 'my $in = <STDIN>; my $roundtrip = decode_json(encode_json({str => $in}))->{str}; utf8::decode($roundtrip) if $ARGV[0]; say Dump [$in, $roundtrip]' 0\n\nYou can tweak that trailing \"0\" to \"1\" to toggle the ad-hoc schema,\ni.e. after we decode the JSON we go and manually UTF-8 decode it to get\nback at the same binary data, otherwise we end up with an UTF-8 escaped\nversion of what we put in.\n\n>   3. Some other similar format. YAML comes to mind. Last time I looked\n>      (quite a while ago), it seemed insanely complex, but I think you\n>      could implement only a reasonable subset. OTOH, I think the tools\n>      ecosystem for parsing JSON (e.g., jq) is much better.\n\nThe lack of fast schema-less formats that supported arrays, hashes\netc. and didn't suck when it came to mixed binary/UTF-8 led us to\nimplementing our own at work: https://github.com/Sereal/Sereal\n\nI think for git's use-case we're probably best off with JSON. It's going\nto work almost all of the time, and when it doesn't it's going to be on\nsomeone's weird non-UTF-8 repo, and those people are probably used to\ndealing with crap because of that anyway and can just manually decode\ntheir thing after it gets double-encoded.\n\nThat sucks, but given that we'll be using this either for just ASCII\n(telemetry) or UTF-8 most of the time, and that realistically other\nformats either suck more or aren't nearly as ubiquitous...\n"},{"id":"341972","messageId":"CA+P7+xpGWZaFMWhN72beUVaV_HuECAAEYhf+UT-CPZUV+Gew4Q@mail.gmail.com","threadId":"48059","inReplyTo":"20180316211837.GB12333@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2018-03-17T07:38:32Z","receivedAt":"2018-03-17T07:38:59Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Mar 16, 2018 at 2:18 PM, Jeff King <peff@peff.net> wrote:\n>   3. Some other similar format. YAML comes to mind. Last time I looked\n>      (quite a while ago), it seemed insanely complex, but I think you\n>      could implement only a reasonable subset. OTOH, I think the tools\n>      ecosystem for parsing JSON (e.g., jq) is much better.\n>\n\nI would personally avoid YAML. It's \"easier\" for humans to read/parse,\nbut honestly JSON is already simple enough and anyone who writes C or\njavascript can likely parse and hand-write JSON anyways. YAML lacks\nbuilt-in parsers for most languages, where as many scripting languages\nalready have JSON parsing built in, or have more easily attainable\nlibraries available. In contrast, the YAML libraries are much more\ncomplex and less likely to be available.\n\nThat's just my own experience at $dayjob though.\n\nThanks,\nJake\n"},{"id":"342133","messageId":"e2a9d2a9-f2e2-f285-52c3-be11668da543@jeffhostetler.com","threadId":"48059","inReplyTo":"20180316211837.GB12333@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-03-19T10:19:26Z","receivedAt":"2018-03-19T10:19:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/16/2018 5:18 PM, Jeff King wrote:\n> On Fri, Mar 16, 2018 at 07:40:55PM +0000, git@jeffhostetler.com wrote:\n> \n[...]\n> I really like the idea of being able to send our machine-readable output\n> in some \"standard\" syntax for which people may already have parsers. But\n> one big hangup with JSON is that it assumes all strings are UTF-8. That\n> may be OK for telemetry data, but it would probably lead to problems for\n> something like status porcelain, since Git's view of paths is just a\n> string of bytes (not to mention possible uses elsewhere like author\n> names, subject lines, etc).\n[...]\n\nI'll come back to the UTF-8/YAML questions in a separate response.\n\n\n>> Documentation for the new API is given in json-writer.h at the bottom of\n>> the first patch.\n> \n> The API generally looks pleasant, but the nesting surprised me.  E.g.,\n> I'd have expected:\n> \n>    jw_array_begin(out);\n>    jw_array_begin(out);\n>    jw_array_append_int(out, 42);\n>    jw_array_end(out);\n>    jw_array_end(out);\n> \n> to result in an array containing an array containing an integer. But\n> array_begin() actually resets the strbuf, so you can't build up nested\n> items like this internally. Ditto for objects within objects. You have\n> to use two separate strbufs and copy the data an extra time.\n> \n> To make the above work, I think you'd have to store a little more state.\n> E.g., the \"array_append\" functions check \"out->len\" to see if they need\n> to add a separating comma. That wouldn't work if we might be part of a\n> nested array. So I think you'd need a context struct like:\n> \n>    struct json_writer {\n>      int first_item;\n>      struct strbuf out;\n>    };\n>    #define JSON_WRITER_INIT { 1, STRBUF_INIT }\n> \n> to store the state and the output. As a bonus, you could also use it to\n> store some other sanity checks (e.g., keep a \"depth\" counter and BUG()\n> when somebody tries to access the finished strbuf with a hanging-open\n> object or array).\n\nYeah, I thought about that, but I think it gets more complex than that.\nI'd need a stack of \"first_item\" values.  Or maybe the _begin() needs to\nincrement a depth and set first_item and the _end() needs to always\nunset first_item.  I'll look at this gain.\n\nThe thing I liked about the bottom-up construction is that it is easier\nto collect multiple sets in parallel and combine them during the final\nroll-up.  With the in-line nesting, you're tempted to try to construct\nthe resulting JSON in a single series and that may not fit what the code\nis trying to do.  For example, if I wanted to collect an array of error\nmessages as they are generated and an array of argv arrays and any alias\nexpansions, then put together a final JSON string containing them and\nthe final exit code, I'd need to build it in parts.  I can build these\nparts in pieces of JSON and combine them at the end -- or build up other\nsimilar data structures (string arrays, lists, or whatever) and then\nhave a JSON conversion step.  But we can make it work both ways, I just\nwanted to keep it simpler.\n\n\n>> I wasn't sure how to unit test the API from a shell script, so I added a\n>> helper command that does most of the work in the second patch.\n> \n> In general I'd really prefer to keep the shell script as the driver for\n> the tests, and have t/helper programs just be conduits. E.g., something\n> like:\n> \n>    cat >expect <<-\\EOF &&\n>    {\"key\": \"value\", \"foo\": 42}\n>    EOF\n>    test-json-writer >actual \\\n>      object_begin \\\n>      object_append_string key value \\\n>      object_append_int foo 42 \\\n>      object_end &&\n>    test_cmp expect actual\n> \n> It's a bit tedious (though fairly mechanical) to expose the API in this\n> way, but it makes it much easier to debug, modify, or add tests later\n> on (for example, I had to modify the C program to find out that my\n> append example above wouldn't work).\n\nYeah, I wasn't sure if such a simple api required exposing all that\nmachinery to the shell or not.  And the api is fairly self-contained\nand not depending on a lot of disk/repo setup or anything, so my tests\nwould be essentially static WRT everything else.\n\nWith my t0019 script you should have been able use -x -v to see what\nwas failing.\n\n> \n> -Peff\n> \n\nthanks for the quick review\nJeff\n"},{"id":"342213","messageId":"d814130f-dce7-4513-02a5-469c97464715@jeffhostetler.com","threadId":"48059","inReplyTo":"CA+P7+xpGWZaFMWhN72beUVaV_HuECAAEYhf+UT-CPZUV+Gew4Q@mail.gmail.com","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-03-19T17:31:50Z","receivedAt":"2018-03-19T17:32:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/17/2018 3:38 AM, Jacob Keller wrote:\n> On Fri, Mar 16, 2018 at 2:18 PM, Jeff King <peff@peff.net> wrote:\n>>    3. Some other similar format. YAML comes to mind. Last time I looked\n>>       (quite a while ago), it seemed insanely complex, but I think you\n>>       could implement only a reasonable subset. OTOH, I think the tools\n>>       ecosystem for parsing JSON (e.g., jq) is much better.\n>>\n> \n> I would personally avoid YAML. It's \"easier\" for humans to read/parse,\n> but honestly JSON is already simple enough and anyone who writes C or\n> javascript can likely parse and hand-write JSON anyways. YAML lacks\n> built-in parsers for most languages, where as many scripting languages\n> already have JSON parsing built in, or have more easily attainable\n> libraries available. In contrast, the YAML libraries are much more\n> complex and less likely to be available.\n> \n> That's just my own experience at $dayjob though.\n\nAgreed.  I just looked at the spec for it and I think it would be\nharder for us to be assured we are generating valid output with\nleading whitespace being significant (without a lot more inspection\nof the strings being passed down to us).\n\nJeff\n\n"},{"id":"342291","messageId":"20180320054223.GC15813@sigill.intra.peff.net","threadId":"48059","inReplyTo":"e2a9d2a9-f2e2-f285-52c3-be11668da543@jeffhostetler.com","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-20T05:42:23Z","receivedAt":"2018-03-20T05:42:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 19, 2018 at 06:19:26AM -0400, Jeff Hostetler wrote:\n\n> > To make the above work, I think you'd have to store a little more state.\n> > E.g., the \"array_append\" functions check \"out->len\" to see if they need\n> > to add a separating comma. That wouldn't work if we might be part of a\n> > nested array. So I think you'd need a context struct like:\n> > \n> >    struct json_writer {\n> >      int first_item;\n> >      struct strbuf out;\n> >    };\n> >    #define JSON_WRITER_INIT { 1, STRBUF_INIT }\n> > \n> > to store the state and the output. As a bonus, you could also use it to\n> > store some other sanity checks (e.g., keep a \"depth\" counter and BUG()\n> > when somebody tries to access the finished strbuf with a hanging-open\n> > object or array).\n> \n> Yeah, I thought about that, but I think it gets more complex than that.\n> I'd need a stack of \"first_item\" values.  Or maybe the _begin() needs to\n> increment a depth and set first_item and the _end() needs to always\n> unset first_item.  I'll look at this gain.\n\nI think you may be able to get by with just unsetting first_item for any\n\"end\". Because as you \"pop\" to whatever data structure is holding\nwhatever has ended, you know it's no longer the first item (whatever\njust ended was there before it).\n\nI admit I haven't thought too hard on it, though, so maybe I'm missing\nsomething.\n\n> The thing I liked about the bottom-up construction is that it is easier\n> to collect multiple sets in parallel and combine them during the final\n> roll-up.  With the in-line nesting, you're tempted to try to construct\n> the resulting JSON in a single series and that may not fit what the code\n> is trying to do.  For example, if I wanted to collect an array of error\n> messages as they are generated and an array of argv arrays and any alias\n> expansions, then put together a final JSON string containing them and\n> the final exit code, I'd need to build it in parts.  I can build these\n> parts in pieces of JSON and combine them at the end -- or build up other\n> similar data structures (string arrays, lists, or whatever) and then\n> have a JSON conversion step.  But we can make it work both ways, I just\n> wanted to keep it simpler.\n\nYeah, I agree that kind of bottom-up construction would be nice for some\ncases. I'm mostly worried about inefficiency copying the strings over\nand over as we build up the final output.  Maybe that's premature\nworrying, though.\n\nIf the first_item thing isn't too painful, then it might be nice to have\nboth approaches available.\n\n> > In general I'd really prefer to keep the shell script as the driver for\n> > the tests, and have t/helper programs just be conduits. E.g., something\n> > like:\n> > \n> >    cat >expect <<-\\EOF &&\n> >    {\"key\": \"value\", \"foo\": 42}\n> >    EOF\n> >    test-json-writer >actual \\\n> >      object_begin \\\n> >      object_append_string key value \\\n> >      object_append_int foo 42 \\\n> >      object_end &&\n> >    test_cmp expect actual\n> > \n> > It's a bit tedious (though fairly mechanical) to expose the API in this\n> > way, but it makes it much easier to debug, modify, or add tests later\n> > on (for example, I had to modify the C program to find out that my\n> > append example above wouldn't work).\n> \n> Yeah, I wasn't sure if such a simple api required exposing all that\n> machinery to the shell or not.  And the api is fairly self-contained\n> and not depending on a lot of disk/repo setup or anything, so my tests\n> would be essentially static WRT everything else.\n> \n> With my t0019 script you should have been able use -x -v to see what\n> was failing.\n\nI was able to run the test-helper directly. The tricky thing is that I\nhad to write new C code to test my theory about how the API worked.\nAdmittedly that's not something most people would do regularly, but I\noften seem to end up doing that kind of probing and debugging. Many\ntimes I've found the more generic t/helper programs useful.\n\nI also wonder if various parts of the system embrace JSON, if we'd want\nto have a tool for generating it as part of other tests (e.g., to create\n\"expect\" files).\n\n-Peff\n"},{"id":"342292","messageId":"20180320055200.GD15813@sigill.intra.peff.net","threadId":"48059","inReplyTo":"87tvtfd3sl.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-20T05:52:00Z","receivedAt":"2018-03-20T05:52:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 17, 2018 at 12:00:26AM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Fri, Mar 16 2018, Jeff King jotted:\n> \n> > I really like the idea of being able to send our machine-readable output\n> > in some \"standard\" syntax for which people may already have parsers. But\n> > one big hangup with JSON is that it assumes all strings are UTF-8.\n> \n> FWIW It's not UTF-8 but \"Unicode characters\", i.e. any Unicode encoding\n> is valid, not that it changes anything you're pointing out, but people\n> on Win32 could use UTF-16 as-is if their filenames were in that format.\n\nBut AIUI, non-UTF8 has to come as \"\\u\" escapes, right? That at least\ngives us an \"out\" for exotic characters, but I don't think we can just\nblindly dump pathnames into quoted strings, can we?\n\n> > Some possible solutions I can think of:\n> >\n> >   1. Ignore the UTF-8 requirement, making a JSON-like output (which I\n> >      think is what your patches do). I'm not sure what problems this\n> >      might cause on the parsing side.\n> \n> Maybe some JSON parsers are more permissive, but they'll commonly just\n> die on non-Unicode (usually UTF-8) input, e.g.:\n> \n>     $ (echo -n '{\"str \": \"'; head -c 3 /dev/urandom ; echo -n '\"}') | perl -0666 -MJSON::XS -wE 'say decode_json(<>)->{str}'\n>     malformed UTF-8 character in JSON string, at character offset 10 (before \"\\x{fffd}e\\x{fffd}\"}\") at -e line 1, <> chunk 1.\n\nOK, that's about what I expected.\n\n> >   2. Specially encode non-UTF-8 bits. I'm not familiar enough with JSON\n> >      to know the options here, but my understanding is that numeric\n> >      escapes are just for inserting unicode code points. _Can_ you\n> >      actually transport arbitrary binary data across JSON without\n> >      base64-encoding it (yech)?\n> \n> There's no way to transfer binary data in JSON without it being shoved\n> into a UTF-8 encoding, so you'd need to know on the other side that\n> such-and-such a field has binary in it, i.e. you'll need to invent your\n> own schema.\n\nYuck. That's what I was afraid of. Is there any kind of standard scheme\nhere? It seems like we lose all of the benefits of JSON if the receiver\nhas to know whether and when to de-base64 (or whatever) our data.\n\n> I think for git's use-case we're probably best off with JSON. It's going\n> to work almost all of the time, and when it doesn't it's going to be on\n> someone's weird non-UTF-8 repo, and those people are probably used to\n> dealing with crap because of that anyway and can just manually decode\n> their thing after it gets double-encoded.\n\nThat sounds a bit hand-wavy. While I agree that anybody using non-utf8\nat this point is slightly insane, Git _does_ actually work with\narbitrary encodings in things like pathnames. It just seems kind of lame\nto settle on a new universal encoding format for output that's actually\nless capable than the current output.\n\n> That sucks, but given that we'll be using this either for just ASCII\n> (telemetry) or UTF-8 most of the time, and that realistically other\n> formats either suck more or aren't nearly as ubiquitous...\n\nI'd hoped to be able to output something like \"git status\" in JSON,\nwhich is inherently going to deal with user paths.\n\n-Peff\n"},{"id":"342340","messageId":"09dca55b-6c8b-2793-901d-d9f2cf5dd873@jeffhostetler.com","threadId":"48059","inReplyTo":"20180320054223.GC15813@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] routines to generate JSON data","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-03-20T16:44:00Z","receivedAt":"2018-03-20T16:44:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/20/2018 1:42 AM, Jeff King wrote:\n> On Mon, Mar 19, 2018 at 06:19:26AM -0400, Jeff Hostetler wrote:\n> \n>>> To make the above work, I think you'd have to store a little more state.\n>>> E.g., the \"array_append\" functions check \"out->len\" to see if they need\n>>> to add a separating comma. That wouldn't work if we might be part of a\n>>> nested array. So I think you'd need a context struct like:\n>>>\n>>>     struct json_writer {\n>>>       int first_item;\n>>>       struct strbuf out;\n>>>     };\n>>>     #define JSON_WRITER_INIT { 1, STRBUF_INIT }\n>>>\n>>> to store the state and the output. As a bonus, you could also use it to\n>>> store some other sanity checks (e.g., keep a \"depth\" counter and BUG()\n>>> when somebody tries to access the finished strbuf with a hanging-open\n>>> object or array).\n>>\n>> Yeah, I thought about that, but I think it gets more complex than that.\n>> I'd need a stack of \"first_item\" values.  Or maybe the _begin() needs to\n>> increment a depth and set first_item and the _end() needs to always\n>> unset first_item.  I'll look at this gain.\n> \n> I think you may be able to get by with just unsetting first_item for any\n> \"end\". Because as you \"pop\" to whatever data structure is holding\n> whatever has ended, you know it's no longer the first item (whatever\n> just ended was there before it).\n> \n> I admit I haven't thought too hard on it, though, so maybe I'm missing\n> something.\n\nI'll take a look.  Thanks.\n\n  \n>> The thing I liked about the bottom-up construction is that it is easier\n>> to collect multiple sets in parallel and combine them during the final\n>> roll-up.  With the in-line nesting, you're tempted to try to construct\n>> the resulting JSON in a single series and that may not fit what the code\n>> is trying to do.  For example, if I wanted to collect an array of error\n>> messages as they are generated and an array of argv arrays and any alias\n>> expansions, then put together a final JSON string containing them and\n>> the final exit code, I'd need to build it in parts.  I can build these\n>> parts in pieces of JSON and combine them at the end -- or build up other\n>> similar data structures (string arrays, lists, or whatever) and then\n>> have a JSON conversion step.  But we can make it work both ways, I just\n>> wanted to keep it simpler.\n> \n> Yeah, I agree that kind of bottom-up construction would be nice for some\n> cases. I'm mostly worried about inefficiency copying the strings over\n> and over as we build up the final output.  Maybe that's premature\n> worrying, though.\n> \n> If the first_item thing isn't too painful, then it might be nice to have\n> both approaches available.\n\nTrue.\n\n  \n>>> In general I'd really prefer to keep the shell script as the driver for\n>>> the tests, and have t/helper programs just be conduits. E.g., something\n>>> like:\n>>>\n>>>     cat >expect <<-\\EOF &&\n>>>     {\"key\": \"value\", \"foo\": 42}\n>>>     EOF\n>>>     test-json-writer >actual \\\n>>>       object_begin \\\n>>>       object_append_string key value \\\n>>>       object_append_int foo 42 \\\n>>>       object_end &&\n>>>     test_cmp expect actual\n>>>\n>>> It's a bit tedious (though fairly mechanical) to expose the API in this\n>>> way, but it makes it much easier to debug, modify, or add tests later\n>>> on (for example, I had to modify the C program to find out that my\n>>> append example above wouldn't work).\n>>\n>> Yeah, I wasn't sure if such a simple api required exposing all that\n>> machinery to the shell or not.  And the api is fairly self-contained\n>> and not depending on a lot of disk/repo setup or anything, so my tests\n>> would be essentially static WRT everything else.\n>>\n>> With my t0019 script you should have been able use -x -v to see what\n>> was failing.\n> \n> I was able to run the test-helper directly. The tricky thing is that I\n> had to write new C code to test my theory about how the API worked.\n> Admittedly that's not something most people would do regularly, but I\n> often seem to end up doing that kind of probing and debugging. Many\n> times I've found the more generic t/helper programs useful.\n> \n> I also wonder if various parts of the system embrace JSON, if we'd want\n> to have a tool for generating it as part of other tests (e.g., to create\n> \"expect\" files).\n\nOk, let me see what I can come up with.\n\nThanks\nJeff\n\n"}]}