{"thread":{"id":"51337","subject":"[PATCH 0/8] Add 'ls-files --json' to dump the index in json","startedAt":"2019-06-19T09:59:08Z","lastAt":"2019-06-24T12:50:45Z","messageCount":36,"participants":["Nguyễn Thái Ngọc Duy","Ævar Arnfjörð Bjarmason","Derrick Stolee","Duy Nguyen","Jeff King","Junio C Hamano","Jeff Hostetler","Johannes Schindelin","brian m. carlson"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"377492","messageId":"20190619095858.30124-1-pclouds@gmail.com","threadId":"51337","inReplyTo":null,"subject":"[PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:50Z","receivedAt":"2019-06-19T09:59:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is probably just my itch. Every time I have to do something with\nthe index, I need to add a little bit code here, a little bit there to\nget a better \"view\" of the index.\n\nThis solves it for me. It allows me to see pretty much everything in the\nindex (except really low detail stuff like pathname compression). It's\nreadable by human, but also easy to parse if you need to do statistics\nand stuff. You could even do a \"diff\" between two indexes.\n\nI'm not really sure if anybody else finds this useful. Because if not,\nI guess there's not much point trying to merge it to git.git just for a\nsingle user. Maintaining off tree is still a pain for me, but I think\nI can manage it.\n\nNguyễn Thái Ngọc Duy (8):\n ls-files: add --json to dump the index\n split-index.c: dump \"link\" extension as json\n fsmonitor.c: dump \"FSMN\" extension as json\n resolve-undo.c: dump \"REUC\" extension as json\n read-cache.c: dump \"EOIE\" extension as json\n read-cache.c: dump \"IEOT\" extension as json\n cache-tree.c: dump \"TREE\" extension as json\n dir.c: dump \"UNTR\" extension as json\n\n Documentation/git-ls-files.txt | 5 ++\n builtin/ls-files.c | 30 +++++--\n cache-tree.c | 41 ++++++++--\n cache-tree.h | 5 +-\n cache.h | 2 +\n dir.c | 56 ++++++++++++-\n dir.h | 4 +-\n fsmonitor.c | 9 +++\n json-writer.c | 30 +++++++\n json-writer.h | 29 +++++++\n read-cache.c | 139 ++++++++++++++++++++++++++++++---\n resolve-undo.c | 36 ++++++++-\n resolve-undo.h | 4 +-\n split-index.c | 13 ++-\n 14 files changed, 376 insertions(+), 27 deletions(-)\n\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377493","messageId":"20190619095858.30124-2-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 1/8] ls-files: add --json to dump the index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:51Z","receivedAt":"2019-06-19T09:59:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"So far we don't have a command to basically dump the index file out,\nwith all its glory details. Checking some info, for example, stat\ntime, usually involves either writing new code or firing up \"xxd\" and\ndecoding values by yourself.\n\nThis --json is supposed to help that. It dumps the index in a human\nreadable format but also easy to be processed with tools. And it will\nprint almost enough info to reconstruct the index later.\n\nIn this patch we only dump the main part, not extensions. But at the\nend of the series, the entire index is dumped. The end result could be\nvery verbose even on a small repository such as git.git.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-ls-files.txt | 5 +++\n builtin/ls-files.c | 30 +++++++++++---\n cache.h | 2 +\n json-writer.c | 16 ++++++++\n json-writer.h | 21 ++++++++++\n read-cache.c | 73 +++++++++++++++++++++++++++++++++-\n 6 files changed, 140 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex 8461c0e83e..54011c8f65 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -60,6 +60,11 @@ OPTIONS\n --stage::\n \tShow staged contents' mode bits, object name and stage number in the output.\n \n+--json::\n+\tDump the entire index content in JSON format. This is for\n+\tdebugging purposes and the JSON structure may change from time\n+\tto time.\n+\n --directory::\n \tIf a whole directory is classified as \"other\", show just its\n \tname (with a trailing slash) and not its whole contents.\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 7f83c9a6f2..d00f6d3074 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -8,6 +8,7 @@\n #include \"cache.h\"\n #include \"repository.h\"\n #include \"config.h\"\n+#include \"json-writer.h\"\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"builtin.h\"\n@@ -31,6 +32,7 @@ static int show_modified;\n static int show_killed;\n static int show_valid_bit;\n static int show_fsmonitor_bit;\n+static int show_json;\n static int line_terminator = '\\n';\n static int debug_mode;\n static int show_eol;\n@@ -543,6 +545,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tN_(\"show staged contents' object name in the output\")),\n \t\tOPT_BOOL('k', \"killed\", &show_killed,\n \t\t\tN_(\"show files on the filesystem that need to be removed\")),\n+\t\tOPT_BOOL(0, \"json\", &show_json,\n+\t\t\tN_(\"dump index content in json format\")),\n \t\tOPT_BIT(0, \"directory\", &dir.flags,\n \t\t\tN_(\"show 'other' directories' names only\"),\n \t\t\tDIR_SHOW_OTHER_DIRECTORIES),\n@@ -660,8 +664,12 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \n \t/* With no flags, we default to showing the cached files */\n \tif (!(show_stage || show_deleted || show_others || show_unmerged ||\n-\t show_killed || show_modified || show_resolve_undo))\n+\t show_killed || show_modified || show_resolve_undo || show_json))\n \t\tshow_cached = 1;\n+\tif (show_json && (show_stage || show_deleted || show_others ||\n+\t\t\t show_unmerged || show_killed || show_modified ||\n+\t\t\t show_cached || show_resolve_undo || with_tree))\n+\t\tdie(_(\"--show-json cannot be used with other --show- options, or --with-tree\"));\n \n \tif (with_tree) {\n \t\t/*\n@@ -673,10 +681,22 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\toverlay_tree_on_index(the_repository->index, with_tree, max_prefix);\n \t}\n \n-\tshow_files(the_repository, &dir);\n-\n-\tif (show_resolve_undo)\n-\t\tshow_ru_info(the_repository->index);\n+\tif (!show_json) {\n+\t\tshow_files(the_repository, &dir);\n+\n+\t\tif (show_resolve_undo)\n+\t\t\tshow_ru_info(the_repository->index);\n+\t} else {\n+\t\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\t\tdiscard_index(the_repository->index);\n+\t\tthe_repository->index->jw = &jw;\n+\t\tif (repo_read_index(the_repository) < 0)\n+\t\t\tdie(\"index file corrupt\");\n+\t\tputs(jw.json.buf);\n+\t\tthe_repository->index->jw = NULL;\n+\t\tjw_release(&jw);\n+\t}\n \n \tif (ps_matched) {\n \t\tint bad;\ndiff --git a/cache.h b/cache.h\nindex bf20337ef4..84d0aeed20 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -326,6 +326,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define UNTRACKED_CHANGED\t(1 << 7)\n #define FSMONITOR_CHANGED\t(1 << 8)\n \n+struct json_writer;\n struct split_index;\n struct untracked_cache;\n \n@@ -350,6 +351,7 @@ struct index_state {\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n \tstruct mem_pool *ce_mem_pool;\n+\tstruct json_writer *jw;\n };\n \n /* Name hashing */\ndiff --git a/json-writer.c b/json-writer.c\nindex aadb9dbddc..281bc50b39 100644\n--- a/json-writer.c\n+++ b/json-writer.c\n@@ -202,6 +202,22 @@ void jw_object_null(struct json_writer *jw, const char *key)\n \tstrbuf_addstr(&jw->json, \"null\");\n }\n \n+void jw_object_stat_data(struct json_writer *jw, const char *name,\n+\t\t\t const struct stat_data *sd)\n+{\n+\tjw_object_inline_begin_object(jw, name);\n+\tjw_object_intmax(jw, \"st_ctime.sec\", sd->sd_ctime.sec);\n+\tjw_object_intmax(jw, \"st_ctime.nsec\", sd->sd_ctime.nsec);\n+\tjw_object_intmax(jw, \"st_mtime.sec\", sd->sd_mtime.sec);\n+\tjw_object_intmax(jw, \"st_mtime.nsec\", sd->sd_mtime.nsec);\n+\tjw_object_intmax(jw, \"st_dev\", sd->sd_dev);\n+\tjw_object_intmax(jw, \"st_ino\", sd->sd_ino);\n+\tjw_object_intmax(jw, \"st_uid\", sd->sd_uid);\n+\tjw_object_intmax(jw, \"st_gid\", sd->sd_gid);\n+\tjw_object_intmax(jw, \"st_size\", sd->sd_size);\n+\tjw_end(jw);\n+}\n+\n static void increase_indent(struct strbuf *sb,\n \t\t\t const struct json_writer *jw,\n \t\t\t int indent)\ndiff --git a/json-writer.h b/json-writer.h\nindex 83906b09c1..38f9c9bf68 100644\n--- a/json-writer.h\n+++ b/json-writer.h\n@@ -44,6 +44,8 @@\n \n #include \"strbuf.h\"\n \n+struct stat_data;\n+\n struct json_writer\n {\n \t/*\n@@ -81,6 +83,8 @@ void jw_object_true(struct json_writer *jw, const char *key);\n void jw_object_false(struct json_writer *jw, const char *key);\n void jw_object_bool(struct json_writer *jw, const char *key, int value);\n void jw_object_null(struct json_writer *jw, const char *key);\n+void jw_object_stat_data(struct json_writer *jw, const char *key,\n+\t\t\t const struct stat_data *sd);\n void jw_object_sub_jw(struct json_writer *jw, const char *key,\n \t\t const struct json_writer *value);\n \n@@ -104,4 +108,21 @@ void jw_array_inline_begin_array(struct json_writer *jw);\n int jw_is_terminated(const struct json_writer *jw);\n void jw_end(struct json_writer *jw);\n \n+/*\n+ * These _gently versions accept NULL json_writer to reduce too much\n+ * branching at the call site.\n+ */\n+static inline void jw_object_inline_begin_array_gently(struct json_writer *jw,\n+\t\t\t\t\t\t const char *name)\n+{\n+\tif (jw)\n+\t\tjw_object_inline_begin_array(jw, name);\n+}\n+\n+static inline void jw_end_gently(struct json_writer *jw)\n+{\n+\tif (jw)\n+\t\tjw_end(jw);\n+}\n+\n #endif /* JSON_WRITER_H */\ndiff --git a/read-cache.c b/read-cache.c\nindex 4dd22f4f6e..eec030b3bb 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -25,6 +25,7 @@\n #include \"fsmonitor.h\"\n #include \"thread-utils.h\"\n #include \"progress.h\"\n+#include \"json-writer.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -1952,6 +1953,50 @@ static void *load_index_extensions(void *_data)\n \treturn NULL;\n }\n \n+static void dump_cache_entry(struct index_state *istate,\n+\t\t\t int index,\n+\t\t\t unsigned long offset,\n+\t\t\t const struct cache_entry *ce)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct json_writer *jw = istate->jw;\n+\n+\tjw_array_inline_begin_object(jw);\n+\n+\t/*\n+\t * this is technically redundant, but it's for easier\n+\t * navigation when there hundreds of entries\n+\t */\n+\tjw_object_intmax(jw, \"id\", index);\n+\n+\tjw_object_string(jw, \"name\", ce->name);\n+\n+\tstrbuf_addf(&sb, \"%06o\", ce->ce_mode);\n+\tjw_object_string(jw, \"mode\", sb.buf);\n+\tstrbuf_release(&sb);\n+\n+\tjw_object_intmax(jw, \"flags\", ce->ce_flags);\n+\t/*\n+\t * again redundant info, just so you don't have to decode\n+\t * flags values manually\n+\t */\n+\tif (ce->ce_flags & CE_VALID)\n+\t\tjw_object_true(jw, \"assume-unchanged\");\n+\tif (ce->ce_flags & CE_INTENT_TO_ADD)\n+\t\tjw_object_true(jw, \"intent-to-add\");\n+\tif (ce->ce_flags & CE_SKIP_WORKTREE)\n+\t\tjw_object_true(jw, \"skip-worktree\");\n+\tif (ce_stage(ce))\n+\t\tjw_object_intmax(jw, \"stage\", ce_stage(ce));\n+\n+\tjw_object_string(jw, \"oid\", oid_to_hex(&ce->oid));\n+\n+\tjw_object_stat_data(jw, \"stat\", &ce->ce_stat_data);\n+\tjw_object_intmax(jw, \"file-offset\", offset);\n+\n+\tjw_end(jw);\n+}\n+\n /*\n * A helper function that will load the specified range of cache entries\n * from the memory mapped file and add them to the given index.\n@@ -1972,6 +2017,9 @@ static unsigned long load_cache_entry_block(struct index_state *istate,\n \t\tce = create_from_disk(ce_mem_pool, istate->version, disk_ce, &consumed, previous_ce);\n \t\tset_index_entry(istate, i, ce);\n \n+\t\tif (istate->jw)\n+\t\t\tdump_cache_entry(istate, i, src_offset, ce);\n+\n \t\tsrc_offset += consumed;\n \t\tprevious_ce = ce;\n \t}\n@@ -1983,6 +2031,8 @@ static unsigned long load_all_cache_entries(struct index_state *istate,\n {\n \tunsigned long consumed;\n \n+\tjw_object_inline_begin_array_gently(istate->jw, \"entries\");\n+\n \tif (istate->version == 4) {\n \t\tmem_pool_init(&istate->ce_mem_pool,\n \t\t\t\testimate_cache_size_from_compressed(istate->cache_nr));\n@@ -1993,6 +2043,8 @@ static unsigned long load_all_cache_entries(struct index_state *istate,\n \n \tconsumed = load_cache_entry_block(istate, istate->ce_mem_pool,\n \t\t\t\t\t0, istate->cache_nr, mmap, src_offset, NULL);\n+\n+\tjw_end_gently(istate->jw);\n \treturn consumed;\n }\n \n@@ -2120,6 +2172,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tsize_t extension_offset = 0;\n \tint nr_threads, cpus;\n \tstruct index_entry_offset_table *ieot = NULL;\n+\tint jw_pretty = 1;\n \n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n@@ -2154,6 +2207,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache_nr = ntohl(hdr->hdr_entries);\n \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n+\tistate->timestamp.sec = st.st_mtime;\n+\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \tistate->initialized = 1;\n \n \tp.istate = istate;\n@@ -2176,6 +2231,20 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tif (!HAVE_THREADS)\n \t\tnr_threads = 1;\n \n+\tif (istate->jw) {\n+\t\tjw_object_begin(istate->jw, jw_pretty);\n+\t\tjw_object_intmax(istate->jw, \"version\", istate->version);\n+\t\tjw_object_string(istate->jw, \"oid\", oid_to_hex(&istate->oid));\n+\t\tjw_object_intmax(istate->jw, \"st_mtime.sec\", istate->timestamp.sec);\n+\t\tjw_object_intmax(istate->jw, \"st_mtime.nsec\", istate->timestamp.nsec);\n+\n+\t\t/*\n+\t\t * Threading may mess up json writing. This is for\n+\t\t * debugging only, so performance is not a concern.\n+\t\t */\n+\t\tnr_threads = 1;\n+\t}\n+\n \tif (nr_threads > 1) {\n \t\textension_offset = read_eoie_extension(mmap, mmap_size);\n \t\tif (extension_offset) {\n@@ -2204,8 +2273,6 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += load_all_cache_entries(istate, mmap, mmap_size, src_offset);\n \t}\n \n-\tistate->timestamp.sec = st.st_mtime;\n-\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \n \t/* if we created a thread, join it otherwise load the extensions on the primary thread */\n \tif (extension_offset) {\n@@ -2216,6 +2283,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tp.src_offset = src_offset;\n \t\tload_index_extensions(&p);\n \t}\n+\tjw_end_gently(istate->jw);\n+\n \tmunmap((void *)mmap, mmap_size);\n \n \t/*\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377494","messageId":"20190619095858.30124-3-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 2/8] split-index.c: dump \"link\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:52Z","receivedAt":"2019-06-19T09:59:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n json-writer.c | 14 ++++++++++++++\n json-writer.h | 2 ++\n split-index.c | 13 ++++++++++++-\n 3 files changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/json-writer.c b/json-writer.c\nindex 281bc50b39..70403580ca 100644\n--- a/json-writer.c\n+++ b/json-writer.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"ewah/ewok.h\"\n #include \"json-writer.h\"\n \n void jw_init(struct json_writer *jw)\n@@ -218,6 +219,19 @@ void jw_object_stat_data(struct json_writer *jw, const char *name,\n \tjw_end(jw);\n }\n \n+static void dump_ewah_one(size_t pos, void *jw)\n+{\n+\tjw_array_intmax(jw, pos);\n+}\n+\n+void jw_object_ewah(struct json_writer *jw, const char *key,\n+\t\t struct ewah_bitmap *ewah)\n+{\n+\tjw_object_inline_begin_array(jw, key);\n+\tewah_each_bit(ewah, dump_ewah_one, jw);\n+\tjw_end(jw);\n+}\n+\n static void increase_indent(struct strbuf *sb,\n \t\t\t const struct json_writer *jw,\n \t\t\t int indent)\ndiff --git a/json-writer.h b/json-writer.h\nindex 38f9c9bf68..3c173647d3 100644\n--- a/json-writer.h\n+++ b/json-writer.h\n@@ -85,6 +85,8 @@ void jw_object_bool(struct json_writer *jw, const char *key, int value);\n void jw_object_null(struct json_writer *jw, const char *key);\n void jw_object_stat_data(struct json_writer *jw, const char *key,\n \t\t\t const struct stat_data *sd);\n+void jw_object_ewah(struct json_writer *jw, const char *key,\n+\t\t struct ewah_bitmap *ewah);\n void jw_object_sub_jw(struct json_writer *jw, const char *key,\n \t\t const struct json_writer *value);\n \ndiff --git a/split-index.c b/split-index.c\nindex e6154e4ea9..d7b4420c92 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"json-writer.h\"\n #include \"split-index.h\"\n #include \"ewah/ewok.h\"\n \n@@ -16,6 +17,7 @@ int read_link_extension(struct index_state *istate,\n {\n \tconst unsigned char *data = data_;\n \tstruct split_index *si;\n+\tunsigned long original_sz = sz;\n \tint ret;\n \n \tif (sz < the_hash_algo->rawsz)\n@@ -25,7 +27,7 @@ int read_link_extension(struct index_state *istate,\n \tdata += the_hash_algo->rawsz;\n \tsz -= the_hash_algo->rawsz;\n \tif (!sz)\n-\t\treturn 0;\n+\t\tgoto done;\n \tsi->delete_bitmap = ewah_new();\n \tret = ewah_read_mmap(si->delete_bitmap, data, sz);\n \tif (ret < 0)\n@@ -38,6 +40,15 @@ int read_link_extension(struct index_state *istate,\n \t\treturn error(\"corrupt replace bitmap in link extension\");\n \tif (ret != sz)\n \t\treturn error(\"garbage at the end of link extension\");\n+done:\n+\tif (istate->jw) {\n+\t\tjw_object_inline_begin_object(istate->jw, \"split-index\");\n+\t\tjw_object_string(istate->jw, \"oid\", oid_to_hex(&si->base_oid));\n+\t\tjw_object_ewah(istate->jw, \"delete-bitmap\", si->delete_bitmap);\n+\t\tjw_object_ewah(istate->jw, \"replace-bitmap\", si->replace_bitmap);\n+\t\tjw_object_intmax(istate->jw, \"ext-size\", original_sz);\n+\t\tjw_end(istate->jw);\n+\t}\n \treturn 0;\n }\n \n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377495","messageId":"20190619095858.30124-4-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 3/8] fsmonitor.c: dump \"FSMN\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:53Z","receivedAt":"2019-06-19T09:59:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n fsmonitor.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 1dee0aded1..f6ba437255 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"json-writer.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -50,6 +51,14 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,\n \t}\n \tistate->fsmonitor_dirty = fsmonitor_dirty;\n \n+\tif (istate->jw) {\n+\t\tjw_object_inline_begin_object(istate->jw, \"fsmonitor\");\n+\t\tjw_object_intmax(istate->jw, \"version\", hdr_version);\n+\t\tjw_object_intmax(istate->jw, \"last-update\", istate->fsmonitor_last_update);\n+\t\tjw_object_ewah(istate->jw, \"dirty\", fsmonitor_dirty);\n+\t\tjw_object_intmax(istate->jw, \"ext-size\", sz);\n+\t\tjw_end(istate->jw);\n+\t}\n \ttrace_printf_key(&trace_fsmonitor, \"read fsmonitor extension successful\");\n \treturn 0;\n }\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377496","messageId":"20190619095858.30124-5-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 4/8] resolve-undo.c: dump \"REUC\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:54Z","receivedAt":"2019-06-19T09:59:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 2 +-\n resolve-undo.c | 36 +++++++++++++++++++++++++++++++++++-\n resolve-undo.h | 4 +++-\n 3 files changed, 39 insertions(+), 3 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex eec030b3bb..3b5c63f53a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1701,7 +1701,7 @@ static int read_index_extension(struct index_state *istate,\n \t\tistate->cache_tree = cache_tree_read(data, sz);\n \t\tbreak;\n \tcase CACHE_EXT_RESOLVE_UNDO:\n-\t\tistate->resolve_undo = resolve_undo_read(data, sz);\n+\t\tistate->resolve_undo = resolve_undo_read(data, sz, istate->jw);\n \t\tbreak;\n \tcase CACHE_EXT_LINK:\n \t\tif (read_link_extension(istate, data, sz))\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 236320f179..999020bc40 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"json-writer.h\"\n #include \"resolve-undo.h\"\n #include \"string-list.h\"\n \n@@ -49,7 +50,8 @@ void resolve_undo_write(struct strbuf *sb, struct string_list *resolve_undo)\n \t}\n }\n \n-struct string_list *resolve_undo_read(const char *data, unsigned long size)\n+struct string_list *resolve_undo_read(const char *data, unsigned long size,\n+\t\t\t\t struct json_writer *jw)\n {\n \tstruct string_list *resolve_undo;\n \tsize_t len;\n@@ -59,6 +61,11 @@ struct string_list *resolve_undo_read(const char *data, unsigned long size)\n \n \tresolve_undo = xcalloc(1, sizeof(*resolve_undo));\n \tresolve_undo->strdup_strings = 1;\n+\tif (jw) {\n+\t\tjw_object_inline_begin_object(jw, \"resolve-undo\");\n+\t\tjw_object_intmax(jw, \"ext-size\", size);\n+\t\tjw_object_inline_begin_array(jw, \"entries\");\n+\t}\n \n \twhile (size) {\n \t\tstruct string_list_item *lost;\n@@ -94,6 +101,33 @@ struct string_list *resolve_undo_read(const char *data, unsigned long size)\n \t\t\tsize -= rawsz;\n \t\t\tdata += rawsz;\n \t\t}\n+\n+\t\tif (jw) {\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t\tjw_array_inline_begin_object(jw);\n+\t\t\tjw_object_string(jw, \"path\", lost->string);\n+\n+\t\t\tjw_object_inline_begin_array(jw, \"mode\");\n+\t\t\tfor (i = 0; i < 3; i++) {\n+\t\t\t\tstrbuf_addf(&sb, \"%06o\", ui->mode[i]);\n+\t\t\t\tjw_array_string(jw, sb.buf);\n+\t\t\t\tstrbuf_reset(&sb);\n+\t\t\t}\n+\t\t\tjw_end(jw);\n+\n+\t\t\tjw_object_inline_begin_array(jw, \"oid\");\n+\t\t\tfor (i = 0; i < 3; i++)\n+\t\t\t\tjw_array_string(jw, oid_to_hex(&ui->oid[i]));\n+\t\t\tjw_end(jw);\n+\n+\t\t\tjw_end(jw);\n+\t\t\tstrbuf_release(&sb);\n+\t\t}\n+\t}\n+\tif (jw) {\n+\t\tjw_end(jw);\t/* entries */\n+\t\tjw_end(jw);\t/* resolve-undo */\n \t}\n \treturn resolve_undo;\n \ndiff --git a/resolve-undo.h b/resolve-undo.h\nindex 2b3f0f901e..46b4e93a7e 100644\n--- a/resolve-undo.h\n+++ b/resolve-undo.h\n@@ -3,6 +3,8 @@\n \n #include \"cache.h\"\n \n+struct json_writer;\n+\n struct resolve_undo_info {\n \tunsigned int mode[3];\n \tstruct object_id oid[3];\n@@ -10,7 +12,7 @@ struct resolve_undo_info {\n \n void record_resolve_undo(struct index_state *, struct cache_entry *);\n void resolve_undo_write(struct strbuf *, struct string_list *);\n-struct string_list *resolve_undo_read(const char *, unsigned long);\n+struct string_list *resolve_undo_read(const char *, unsigned long, struct json_writer *);\n void resolve_undo_clear_index(struct index_state *);\n int unmerge_index_entry_at(struct index_state *, int);\n void unmerge_index(struct index_state *, const struct pathspec *);\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377497","messageId":"20190619095858.30124-6-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 5/8] read-cache.c: dump \"EOIE\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:55Z","receivedAt":"2019-06-19T09:59:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 30 +++++++++++++++++++++++++++---\n 1 file changed, 27 insertions(+), 3 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3b5c63f53a..04863c3853 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1914,7 +1914,7 @@ struct index_entry_offset_table\n static struct index_entry_offset_table *read_ieot_extension(const char *mmap, size_t mmap_size, size_t offset);\n static void write_ieot_extension(struct strbuf *sb, struct index_entry_offset_table *ieot);\n \n-static size_t read_eoie_extension(const char *mmap, size_t mmap_size);\n+static size_t read_eoie_extension(const char *mmap, size_t mmap_size, struct json_writer *jw);\n static void write_eoie_extension(struct strbuf *sb, git_hash_ctx *eoie_context, size_t offset);\n \n struct load_index_extensions\n@@ -2243,10 +2243,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\t * debugging only, so performance is not a concern.\n \t\t */\n \t\tnr_threads = 1;\n+\t\t/* and dump EOIE extension even with threading off */\n+\t\tread_eoie_extension(mmap, mmap_size, istate->jw);\n \t}\n \n \tif (nr_threads > 1) {\n-\t\textension_offset = read_eoie_extension(mmap, mmap_size);\n+\t\textension_offset = read_eoie_extension(mmap, mmap_size, NULL);\n \t\tif (extension_offset) {\n \t\t\tint err;\n \n@@ -3504,7 +3506,8 @@ int should_validate_cache_entries(void)\n #define EOIE_SIZE (4 + GIT_SHA1_RAWSZ) /* <4-byte offset> + <20-byte hash> */\n #define EOIE_SIZE_WITH_HEADER (4 + 4 + EOIE_SIZE) /* <4-byte signature> + <4-byte length> + EOIE_SIZE */\n \n-static size_t read_eoie_extension(const char *mmap, size_t mmap_size)\n+static size_t read_eoie_extension(const char *mmap, size_t mmap_size,\n+\t\t\t\t struct json_writer *jw)\n {\n \t/*\n \t * The end of index entries (EOIE) extension is guaranteed to be last\n@@ -3548,6 +3551,12 @@ static size_t read_eoie_extension(const char *mmap, size_t mmap_size)\n \t\treturn 0;\n \tindex += sizeof(uint32_t);\n \n+\tif (jw) {\n+\t\tjw_object_inline_begin_object(jw, \"end-of-index\");\n+\t\tjw_object_intmax(jw, \"offset\", offset);\n+\t\tjw_object_intmax(jw, \"ext-size\", extsize);\n+\t\tjw_object_inline_begin_array(jw, \"extensions\");\n+\t}\n \t/*\n \t * The hash is computed over extension types and their sizes (but not\n \t * their contents). E.g. if we have \"TREE\" extension that is N-bytes\n@@ -3576,9 +3585,24 @@ static size_t read_eoie_extension(const char *mmap, size_t mmap_size)\n \n \t\tthe_hash_algo->update_fn(&c, mmap + src_offset, 8);\n \n+\t\tif (jw) {\n+\t\t\tchar name[5];\n+\n+\t\t\tjw_array_inline_begin_object(jw);\n+\t\t\tmemcpy(name, mmap + src_offset, 4);\n+\t\t\tname[4] = '\\0';\n+\t\t\tjw_object_string(jw, \"name\", name);\n+\t\t\tjw_object_intmax(jw, \"size\", extsize);\n+\t\t\tjw_end(jw);\n+\t\t}\n+\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n+\tif (jw) {\n+\t\tjw_end(jw);\t/* extensions */\n+\t\tjw_end(jw);\t/* end-of-index */\n+\t}\n \tthe_hash_algo->final_fn(hash, &c);\n \tif (!hasheq(hash, (const unsigned char *)index))\n \t\treturn 0;\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377498","messageId":"20190619095858.30124-7-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 6/8] read-cache.c: dump \"IEOT\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:56Z","receivedAt":"2019-06-19T09:59:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 34 +++++++++++++++++++++++++++++-----\n 1 file changed, 29 insertions(+), 5 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 04863c3853..200834e77e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1911,7 +1911,7 @@ struct index_entry_offset_table\n \tstruct index_entry_offset entries[FLEX_ARRAY];\n };\n \n-static struct index_entry_offset_table *read_ieot_extension(const char *mmap, size_t mmap_size, size_t offset);\n+static struct index_entry_offset_table *read_ieot_extension(const char *mmap, size_t mmap_size, size_t offset, struct json_writer *jw);\n static void write_ieot_extension(struct strbuf *sb, struct index_entry_offset_table *ieot);\n \n static size_t read_eoie_extension(const char *mmap, size_t mmap_size, struct json_writer *jw);\n@@ -2232,6 +2232,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tnr_threads = 1;\n \n \tif (istate->jw) {\n+\t\tsize_t off;\n+\n \t\tjw_object_begin(istate->jw, jw_pretty);\n \t\tjw_object_intmax(istate->jw, \"version\", istate->version);\n \t\tjw_object_string(istate->jw, \"oid\", oid_to_hex(&istate->oid));\n@@ -2243,8 +2245,11 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\t * debugging only, so performance is not a concern.\n \t\t */\n \t\tnr_threads = 1;\n-\t\t/* and dump EOIE extension even with threading off */\n-\t\tread_eoie_extension(mmap, mmap_size, istate->jw);\n+\t\t/* and dump EOIE/IOET extensions even with threading off */\n+\t\toff = read_eoie_extension(mmap, mmap_size, istate->jw);\n+\t\tif (off)\n+\t\t\tfree(read_ieot_extension(mmap, mmap_size,\n+\t\t\t\t\t\t off, istate->jw));\n \t}\n \n \tif (nr_threads > 1) {\n@@ -2266,7 +2271,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t * to multi-thread the reading of the cache entries.\n \t */\n \tif (extension_offset && nr_threads > 1)\n-\t\tieot = read_ieot_extension(mmap, mmap_size, extension_offset);\n+\t\tieot = read_ieot_extension(mmap, mmap_size, extension_offset, NULL);\n \n \tif (ieot) {\n \t\tsrc_offset += load_cache_entries_threaded(istate, mmap, mmap_size, nr_threads, ieot);\n@@ -3630,7 +3635,9 @@ static void write_eoie_extension(struct strbuf *sb, git_hash_ctx *eoie_context,\n \n #define IEOT_VERSION\t(1)\n \n-static struct index_entry_offset_table *read_ieot_extension(const char *mmap, size_t mmap_size, size_t offset)\n+static struct index_entry_offset_table *read_ieot_extension(\n+\tconst char *mmap, size_t mmap_size,\n+\tsize_t offset, struct json_writer *jw)\n {\n \tconst char *index = NULL;\n \tuint32_t extsize, ext_version;\n@@ -3666,6 +3673,12 @@ static struct index_entry_offset_table *read_ieot_extension(const char *mmap, si\n \t\terror(\"invalid number of IEOT entries %d\", nr);\n \t\treturn NULL;\n \t}\n+\tif (jw) {\n+\t\tjw_object_inline_begin_object(jw, \"index-entry-offsets\");\n+\t\tjw_object_intmax(jw, \"version\", ext_version);\n+\t\tjw_object_intmax(jw, \"ext-size\", extsize);\n+\t\tjw_object_inline_begin_array(jw, \"entries\");\n+\t}\n \tieot = xmalloc(sizeof(struct index_entry_offset_table)\n \t\t + (nr * sizeof(struct index_entry_offset)));\n \tieot->nr = nr;\n@@ -3674,6 +3687,17 @@ static struct index_entry_offset_table *read_ieot_extension(const char *mmap, si\n \t\tindex += sizeof(uint32_t);\n \t\tieot->entries[i].nr = get_be32(index);\n \t\tindex += sizeof(uint32_t);\n+\n+\t\tif (jw) {\n+\t\t\tjw_array_inline_begin_object(jw);\n+\t\t\tjw_object_intmax(jw, \"offset\", ieot->entries[i].offset);\n+\t\t\tjw_object_intmax(jw, \"count\", ieot->entries[i].nr);\n+\t\t\tjw_end(jw);\n+\t\t}\n+\t}\n+\tif (jw) {\n+\t\tjw_end(jw);\t/* entries */\n+\t\tjw_end(jw);\t/* index-entry-offsets */\n \t}\n \n \treturn ieot;\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377499","messageId":"20190619095858.30124-8-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 7/8] cache-tree.c: dump \"TREE\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:57Z","receivedAt":"2019-06-19T09:59:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n cache-tree.c | 41 ++++++++++++++++++++++++++++++++++++-----\n cache-tree.h | 5 ++++-\n read-cache.c | 2 +-\n 3 files changed, 41 insertions(+), 7 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex b13bfaf71e..fc44016fe8 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -3,6 +3,7 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"cache-tree.h\"\n+#include \"json-writer.h\"\n #include \"object-store.h\"\n #include \"replace-object.h\"\n \n@@ -492,7 +493,8 @@ void cache_tree_write(struct strbuf *sb, struct cache_tree *root)\n \twrite_one(sb, root, \"\", 0);\n }\n \n-static struct cache_tree *read_one(const char **buffer, unsigned long *size_p)\n+static struct cache_tree *read_one(const char **buffer, unsigned long *size_p,\n+\t\t\t\t struct json_writer *jw)\n {\n \tconst char *buf = *buffer;\n \tunsigned long size = *size_p;\n@@ -546,6 +548,15 @@ static struct cache_tree *read_one(const char **buffer, unsigned long *size_p)\n \t\t\t*buffer, subtree_nr);\n #endif\n \n+\tif (jw) {\n+\t\tif (it->entry_count >= 0) {\n+\t\t\tjw_object_string(jw, \"oid\", oid_to_hex(&it->oid));\n+\t\t\tjw_object_intmax(jw, \"entry_count\", it->entry_count);\n+\t\t} else {\n+\t\t\tjw_object_null(jw, \"oid\");\n+\t\t}\n+\t\tjw_object_inline_begin_array(jw, \"subdirs\");\n+\t}\n \t/*\n \t * Just a heuristic -- we do not add directories that often but\n \t * we do not want to have to extend it immediately when we do,\n@@ -559,12 +570,18 @@ static struct cache_tree *read_one(const char **buffer, unsigned long *size_p)\n \t\tstruct cache_tree_sub *subtree;\n \t\tconst char *name = buf;\n \n-\t\tsub = read_one(&buf, &size);\n+\t\tif (jw) {\n+\t\t\tjw_array_inline_begin_object(jw);\n+\t\t\tjw_object_string(jw, \"name\", name);\n+\t\t}\n+\t\tsub = read_one(&buf, &size, jw);\n+\t\tjw_end_gently(jw);\n \t\tif (!sub)\n \t\t\tgoto free_return;\n \t\tsubtree = cache_tree_sub(it, name);\n \t\tsubtree->cache_tree = sub;\n \t}\n+\tjw_end_gently(jw);\n \tif (subtree_nr != it->subtree_nr)\n \t\tdie(\"cache-tree: internal error\");\n \t*buffer = buf;\n@@ -576,11 +593,25 @@ static struct cache_tree *read_one(const char **buffer, unsigned long *size_p)\n \treturn NULL;\n }\n \n-struct cache_tree *cache_tree_read(const char *buffer, unsigned long size)\n+struct cache_tree *cache_tree_read(const char *buffer, unsigned long size,\n+\t\t\t\t struct json_writer *jw)\n {\n+\tstruct cache_tree *ret;\n+\n+\tif (jw) {\n+\t\tjw_object_inline_begin_object(jw, \"cache-tree\");\n+\t\tjw_object_intmax(jw, \"ext-size\", size);\n+\t\tjw_object_inline_begin_object(jw, \"root\");\n+\t}\n \tif (buffer[0])\n-\t\treturn NULL; /* not the whole tree */\n-\treturn read_one(&buffer, &size);\n+\t\tret = NULL; /* not the whole tree */\n+\telse\n+\t\tret = read_one(&buffer, &size, jw);\n+\tif (jw) {\n+\t\tjw_end(jw);\t/* root */\n+\t\tjw_end(jw);\t/* cache-tree */\n+\t}\n+\treturn ret;\n }\n \n static struct cache_tree *cache_tree_find(struct cache_tree *it, const char *path)\ndiff --git a/cache-tree.h b/cache-tree.h\nindex 757bbc48bc..fc3c73284b 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -6,6 +6,8 @@\n #include \"tree-walk.h\"\n \n struct cache_tree;\n+struct json_writer;\n+\n struct cache_tree_sub {\n \tstruct cache_tree *cache_tree;\n \tint count;\t\t/* internally used by update_one() */\n@@ -28,7 +30,8 @@ void cache_tree_invalidate_path(struct index_state *, const char *);\n struct cache_tree_sub *cache_tree_sub(struct cache_tree *, const char *);\n \n void cache_tree_write(struct strbuf *, struct cache_tree *root);\n-struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n+struct cache_tree *cache_tree_read(const char *buffer, unsigned long size,\n+\t\t\t\t struct json_writer *jw);\n \n int cache_tree_fully_valid(struct cache_tree *);\n int cache_tree_update(struct index_state *, int);\ndiff --git a/read-cache.c b/read-cache.c\nindex 200834e77e..289705b816 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1698,7 +1698,7 @@ static int read_index_extension(struct index_state *istate,\n {\n \tswitch (CACHE_EXT(ext)) {\n \tcase CACHE_EXT_TREE:\n-\t\tistate->cache_tree = cache_tree_read(data, sz);\n+\t\tistate->cache_tree = cache_tree_read(data, sz, istate->jw);\n \t\tbreak;\n \tcase CACHE_EXT_RESOLVE_UNDO:\n \t\tistate->resolve_undo = resolve_undo_read(data, sz, istate->jw);\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377500","messageId":"20190619095858.30124-9-pclouds@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"[PATCH 8/8] dir.c: dump \"UNTR\" extension as json","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T09:58:58Z","receivedAt":"2019-06-19T09:59:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The big part of UNTR extension is dumped at the end instead of dumping\nas soon as we read it, because we actually \"patch\" some fields in\nuntracked_cache_dir with EWAH bitmaps at the end.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n dir.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++-\n dir.h | 4 +++-\n json-writer.h | 6 ++++++\n read-cache.c | 2 +-\n 4 files changed, 65 insertions(+), 3 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex ba4a51c296..f389eee24a 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -19,6 +19,7 @@\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"json-writer.h\"\n #include \"submodule-config.h\"\n \n /*\n@@ -2826,7 +2827,42 @@ static void load_oid_stat(struct oid_stat *oid_stat, const unsigned char *data,\n \toid_stat->valid = 1;\n }\n \n-struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz)\n+static void jw_object_oid_stat(struct json_writer *jw, const char *key,\n+\t\t\t const struct oid_stat *oid_stat)\n+{\n+\tjw_object_inline_begin_object(jw, key);\n+\tjw_object_bool(jw, \"valid\", oid_stat->valid);\n+\tjw_object_string(jw, \"oid\", oid_to_hex(&oid_stat->oid));\n+\tjw_object_stat_data(jw, \"stat\", &oid_stat->stat);\n+\tjw_end(jw);\n+}\n+\n+static void jw_object_untracked_cache_dir(struct json_writer *jw,\n+\t\t\t\t\t const struct untracked_cache_dir *ucd)\n+{\n+\tint i;\n+\n+\tjw_object_bool(jw, \"valid\", ucd->valid);\n+\tjw_object_bool(jw, \"check-only\", ucd->check_only);\n+\tjw_object_stat_data(jw, \"stat\", &ucd->stat_data);\n+\tjw_object_string(jw, \"exclude-oid\", oid_to_hex(&ucd->exclude_oid));\n+\tjw_object_inline_begin_array(jw, \"untracked\");\n+\tfor (i = 0; i < ucd->untracked_nr; i++)\n+\t\tjw_array_string(jw, ucd->untracked[i]);\n+\tjw_end(jw);\n+\n+\tjw_object_inline_begin_object(jw, \"dirs\");\n+\tfor (i = 0; i < ucd->dirs_nr; i++) {\n+\t\tjw_object_inline_begin_object(jw, ucd->dirs[i]->name);\n+\t\tjw_object_untracked_cache_dir(jw, ucd->dirs[i]);\n+\t\tjw_end(jw);\n+\t}\n+\tjw_end(jw);\n+}\n+\n+struct untracked_cache *read_untracked_extension(const void *data,\n+\t\t\t\t\t\t unsigned long sz,\n+\t\t\t\t\t\t struct json_writer *jw)\n {\n \tstruct untracked_cache *uc;\n \tstruct read_data rd;\n@@ -2864,6 +2900,17 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \tuc->dir_flags = get_be32(next + ouc_offset(dir_flags));\n \texclude_per_dir = (const char *)next + exclude_per_dir_offset;\n \tuc->exclude_per_dir = xstrdup(exclude_per_dir);\n+\n+\tif (jw) {\n+\t\tjw_object_inline_begin_object(jw, \"untracked-cache\");\n+\t\tjw_object_intmax(jw, \"ext-size\", sz);\n+\t\tjw_object_string(jw, \"ident\", ident);\n+\t\tjw_object_oid_stat(jw, \"info/exclude\", &uc->ss_info_exclude);\n+\t\tjw_object_oid_stat(jw, \"excludes-file\", &uc->ss_excludes_file);\n+\t\tjw_object_intmax(jw, \"flags\", uc->dir_flags);\n+\t\tjw_object_string(jw, \"excludes-per-dir\", uc->exclude_per_dir);\n+\t}\n+\n \t/* NUL after exclude_per_dir is covered by sizeof(*ouc) */\n \tnext += exclude_per_dir_offset + strlen(exclude_per_dir) + 1;\n \tif (next >= end)\n@@ -2905,6 +2952,12 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \tewah_each_bit(rd.sha1_valid, read_oid, &rd);\n \tnext = rd.data;\n \n+\tif (jw) {\n+\t\tjw_object_inline_begin_object(jw, \"root\");\n+\t\tjw_object_untracked_cache_dir(jw, uc->root);\n+\t\tjw_end(jw);\n+\t}\n+\n done:\n \tfree(rd.ucd);\n \tewah_free(rd.valid);\n@@ -2915,6 +2968,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \t\tfree_untracked_cache(uc);\n \t\tuc = NULL;\n \t}\n+\tjw_end_gently(jw);\n \treturn uc;\n }\n \ndiff --git a/dir.h b/dir.h\nindex 680079bbe3..80efdd05c4 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -6,6 +6,8 @@\n #include \"cache.h\"\n #include \"strbuf.h\"\n \n+struct json_writer;\n+\n struct dir_entry {\n \tunsigned int len;\n \tchar name[FLEX_ARRAY]; /* more */\n@@ -362,7 +364,7 @@ void untracked_cache_remove_from_index(struct index_state *, const char *);\n void untracked_cache_add_to_index(struct index_state *, const char *);\n \n void free_untracked_cache(struct untracked_cache *);\n-struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz);\n+struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz, struct json_writer *jw);\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\n void add_untracked_cache(struct index_state *istate);\n void remove_untracked_cache(struct index_state *istate);\ndiff --git a/json-writer.h b/json-writer.h\nindex 3c173647d3..f778e019a2 100644\n--- a/json-writer.h\n+++ b/json-writer.h\n@@ -121,6 +121,12 @@ static inline void jw_object_inline_begin_array_gently(struct json_writer *jw,\n \t\tjw_object_inline_begin_array(jw, name);\n }\n \n+static inline void jw_array_inline_begin_object_gently(struct json_writer *jw)\n+{\n+\tif (jw)\n+\t\tjw_array_inline_begin_object(jw);\n+}\n+\n static inline void jw_end_gently(struct json_writer *jw)\n {\n \tif (jw)\ndiff --git a/read-cache.c b/read-cache.c\nindex 289705b816..d7d9ce7260 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1708,7 +1708,7 @@ static int read_index_extension(struct index_state *istate,\n \t\t\treturn -1;\n \t\tbreak;\n \tcase CACHE_EXT_UNTRACKED:\n-\t\tistate->untracked = read_untracked_extension(data, sz);\n+\t\tistate->untracked = read_untracked_extension(data, sz, istate->jw);\n \t\tbreak;\n \tcase CACHE_EXT_FSMONITOR:\n \t\tread_fsmonitor_extension(istate, data, sz);\n-- \n2.22.0.rc0.322.g2b0371e29a\n\n"},{"id":"377502","messageId":"87lfxx98a9.fsf@evledraar.gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-2-pclouds@gmail.com","subject":"Re: [PATCH 1/8] ls-files: add --json to dump the index","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-19T10:30:38Z","receivedAt":"2019-06-19T10:30:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 19 2019, Nguyễn Thái Ngọc Duy wrote:\n\n> +\t\tdie(_(\"--show-json cannot be used with other --show- options, or --with-tree\"));\n\nShould be --json, not --show-json, right? I assume --show-json is left\nover from an earlier version.\n"},{"id":"377507","messageId":"a5d2d2e9-a22f-9ab8-f88f-9c8c8520735d@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-19T11:58:17Z","receivedAt":"2019-06-19T11:58:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n> This is probably just my itch. Every time I have to do something with\n> the index, I need to add a little bit code here, a little bit there to\n> get a better \"view\" of the index.\n> \n> This solves it for me. It allows me to see pretty much everything in the\n> index (except really low detail stuff like pathname compression). It's\n> readable by human, but also easy to parse if you need to do statistics\n> and stuff. You could even do a \"diff\" between two indexes.\n> \n> I'm not really sure if anybody else finds this useful. Because if not,\n> I guess there's not much point trying to merge it to git.git just for a\n> single user. Maintaining off tree is still a pain for me, but I think\n> I can manage it.\n\nI think we (Microsoft/VFS for Git engineers) would use this tool, as we\nfrequently need to diagnose something that went wrong in a user's index.\nKevin Willford built a tool to search the index and figure out what's\ngoing on, but I'm not sure it parses all of the new extensions or was\nupdated to parse the v5 index.\n\nHaving a translation from the internal index format to an easier-to-parse\nformat is valuable.\n\nThanks,\n-Stolee\n"},{"id":"377510","messageId":"CACsJy8BkAPRQNgMQnCkOy3Wrdaj25uK8g-fQ3okpc3BZ33kzrQ@mail.gmail.com","threadId":"51337","inReplyTo":"a5d2d2e9-a22f-9ab8-f88f-9c8c8520735d@gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T12:42:29Z","receivedAt":"2019-06-19T12:42:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jun 19, 2019 at 6:58 PM Derrick Stolee wrote:\n>\n> On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n> > This is probably just my itch. Every time I have to do something with\n> > the index, I need to add a little bit code here, a little bit there to\n> > get a better \"view\" of the index.\n> >\n> > This solves it for me. It allows me to see pretty much everything in the\n> > index (except really low detail stuff like pathname compression). It's\n> > readable by human, but also easy to parse if you need to do statistics\n> > and stuff. You could even do a \"diff\" between two indexes.\n> >\n> > I'm not really sure if anybody else finds this useful. Because if not,\n> > I guess there's not much point trying to merge it to git.git just for a\n> > single user. Maintaining off tree is still a pain for me, but I think\n> > I can manage it.\n>\n> I think we (Microsoft/VFS for Git engineers) would use this tool, as we\n> frequently need to diagnose something that went wrong in a user's index.\n> Kevin Willford built a tool to search the index and figure out what's\n> going on, but I'm not sure it parses all of the new extensions or was\n> updated to parse the v5 index.\n\nOK I suggest you try it out and see if it really fits your internal\ntools. I wanted to balance between manual inspection and automation so\nthe output may not be the best for tools. I also try not to freeze the\nformat for more wiggle room, which would be fine for one-time scripts,\nbut if you want to have real tools depend on it, we may have to look\nharder at the output format and make sure it's good enough for some\ntime, and have some documentation.\n\nAlso, I don't suppose it matters, but just for the record I don't care\nat all about --json performance. I suppose Jeff's json writer does not\ncache the entire json output in memory, so dumping giant index files\nis fine. But some other things, like reading the index with multiple\nthreads, are also disabled.\n-- \nDuy\n"},{"id":"377511","messageId":"45e49624-be8e-deff-bf9d-aee052991189@gmail.com","threadId":"51337","inReplyTo":"CACsJy8BkAPRQNgMQnCkOy3Wrdaj25uK8g-fQ3okpc3BZ33kzrQ@mail.gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-19T12:48:08Z","receivedAt":"2019-06-19T12:48:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2019 8:42 AM, Duy Nguyen wrote:\n> On Wed, Jun 19, 2019 at 6:58 PM Derrick Stolee wrote:\n>>\n>> On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n>>> This is probably just my itch. Every time I have to do something with\n>>> the index, I need to add a little bit code here, a little bit there to\n>>> get a better \"view\" of the index.\n>>>\n>>> This solves it for me. It allows me to see pretty much everything in the\n>>> index (except really low detail stuff like pathname compression). It's\n>>> readable by human, but also easy to parse if you need to do statistics\n>>> and stuff. You could even do a \"diff\" between two indexes.\n>>>\n>>> I'm not really sure if anybody else finds this useful. Because if not,\n>>> I guess there's not much point trying to merge it to git.git just for a\n>>> single user. Maintaining off tree is still a pain for me, but I think\n>>> I can manage it.\n>>\n>> I think we (Microsoft/VFS for Git engineers) would use this tool, as we\n>> frequently need to diagnose something that went wrong in a user's index.\n>> Kevin Willford built a tool to search the index and figure out what's\n>> going on, but I'm not sure it parses all of the new extensions or was\n>> updated to parse the v5 index.\n> \n> OK I suggest you try it out and see if it really fits your internal\n> tools. I wanted to balance between manual inspection and automation so\n> the output may not be the best for tools. I also try not to freeze the\n> format for more wiggle room, which would be fine for one-time scripts,\n> but if you want to have real tools depend on it, we may have to look\n> harder at the output format and make sure it's good enough for some\n> time, and have some documentation.\n> \n> Also, I don't suppose it matters, but just for the record I don't care\n> at all about --json performance. I suppose Jeff's json writer does not\n> cache the entire json output in memory, so dumping giant index files\n> is fine. But some other things, like reading the index with multiple\n> threads, are also disabled.\n\nPerformance is not critical here, and in fact would become slower for\nsure because of the extra parsing details. However, I think using JSON\nas a translation layer will make any tools that consume the JSON be\nmore resilient to future index format updates. That stability is\nvaluable. Even though the JSON format is not guaranteed to stay the\nsame, it is easier to update an object model to the JSON format than\na new binary parser.\n\nThanks,\n-Stolee\n"},{"id":"377514","messageId":"e3b8d7bf-486d-d52f-9a33-6a7ff837552e@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-2-pclouds@gmail.com","subject":"Re: [PATCH 1/8] ls-files: add --json to dump the index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-19T13:03:27Z","receivedAt":"2019-06-19T13:03:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n> So far we don't have a command to basically dump the index file out,\n> with all its glory details. Checking some info, for example, stat\n> time, usually involves either writing new code or firing up \"xxd\" and\n> decoding values by yourself.\n> \n> This --json is supposed to help that. It dumps the index in a human\n> readable format but also easy to be processed with tools. And it will\n> print almost enough info to reconstruct the index later.\n\nIn an earlier message, I stated that I like the idea of this feature.\nI know that you wanted to get that feedback before working too hard on\nthe patch series, so now that interest is declared, please add some tests\nthat verify this output looks as you expect.\n\n> In this patch we only dump the main part, not extensions. But at the\n> end of the series, the entire index is dumped. The end result could be\n> very verbose even on a small repository such as git.git.\n\nI would expect this commit to include a complete \"output matches expected\"\ntest, and the later patches can update the index to include these\nextensions then verify that their sections appear in the output using\na grep.\n\n> +--json::\n> +\tDump the entire index content in JSON format. This is for\n> +\tdebugging purposes and the JSON structure may change from time\n> +\tto time.\n> +\n\n\"...purposes and the JSON structure may change from time to time\" may be better\nas \"...purposes. The JSON structure is subject to change.\"\n\n> +\t\tOPT_BOOL(0, \"json\", &show_json,\n> +\t\t\tN_(\"dump index content in json format\")),\n\nWe should probably use \"JSON\" here in the help text.\n\n> -\tshow_files(the_repository, &dir);\n> -\n> -\tif (show_resolve_undo)\n> -\t\tshow_ru_info(the_repository->index);\n> +\tif (!show_json) {\n> +\t\tshow_files(the_repository, &dir);\n> +\n> +\t\tif (show_resolve_undo)\n> +\t\t\tshow_ru_info(the_repository->index);\n> +\t} else {\n> +\t\tstruct json_writer jw = JSON_WRITER_INIT;\n> +\n> +\t\tdiscard_index(the_repository->index);\n> +\t\tthe_repository->index->jw = &jw;\n> +\t\tif (repo_read_index(the_repository) < 0)\n> +\t\t\tdie(\"index file corrupt\");\n> +\t\tputs(jw.json.buf);\n> +\t\tthe_repository->index->jw = NULL;\n> +\t\tjw_release(&jw);\n> +\t}\n> \n> \tif (ps_matched) {\n> \t\tint bad;\n\nI see this 'ps_matched' condition at the end, which is related to\nthe '--error-unmatch' option. I added \"--error-unmatch foo\" to my\ncommand and got the appropriate error message:\n\n error: pathspec 'foo' did not match any file(s) known to git\n Did you forget to 'git add'?\n\nThis was sent to stderr while the JSON was in stdout, so this should\nbe appropriate to allow both options. Just pointing it out to make\nsure this is intended.\n\n> +void jw_object_stat_data(struct json_writer *jw, const char *name,\n> +\t\t\t const struct stat_data *sd)\n> +{\n> +\tjw_object_inline_begin_object(jw, name);\n> +\tjw_object_intmax(jw, \"st_ctime.sec\", sd->sd_ctime.sec);\n> +\tjw_object_intmax(jw, \"st_ctime.nsec\", sd->sd_ctime.nsec);\n> +\tjw_object_intmax(jw, \"st_mtime.sec\", sd->sd_mtime.sec);\n> +\tjw_object_intmax(jw, \"st_mtime.nsec\", sd->sd_mtime.nsec);\n> +\tjw_object_intmax(jw, \"st_dev\", sd->sd_dev);\n> +\tjw_object_intmax(jw, \"st_ino\", sd->sd_ino);\n> +\tjw_object_intmax(jw, \"st_uid\", sd->sd_uid);\n> +\tjw_object_intmax(jw, \"st_gid\", sd->sd_gid);\n> +\tjw_object_intmax(jw, \"st_size\", sd->sd_size);\n> +\tjw_end(jw);\n> +}\n\nIf these are all part of the same object, are the \"st_\" prefixes\nnecessary for every member?\n\n> +\t/*\n> +\t * again redundant info, just so you don't have to decode\n> +\t * flags values manually\n> +\t */\n> +\tif (ce->ce_flags & CE_VALID)\n> +\t\tjw_object_true(jw, \"assume-unchanged\");\n> +\tif (ce->ce_flags & CE_INTENT_TO_ADD)\n> +\t\tjw_object_true(jw, \"intent-to-add\");\n> +\tif (ce->ce_flags & CE_SKIP_WORKTREE)\n> +\t\tjw_object_true(jw, \"skip-worktree\");\n> +\tif (ce_stage(ce))\n> +\t\tjw_object_intmax(jw, \"stage\", ce_stage(ce));\n\nI'm really glad these flags are getting expanded! Much easier to\nunderstand what's going on this way.\n\n> +\tif (istate->jw) {\n> +\t\tjw_object_begin(istate->jw, jw_pretty);\n> +\t\tjw_object_intmax(istate->jw, \"version\", istate->version);\n> +\t\tjw_object_string(istate->jw, \"oid\", oid_to_hex(&istate->oid));\n> +\t\tjw_object_intmax(istate->jw, \"st_mtime.sec\", istate->timestamp.sec);\n> +\t\tjw_object_intmax(istate->jw, \"st_mtime.nsec\", istate->timestamp.nsec);\n\nHere, the \"st_\" prefixes are not on every member, but would it\nbe confusing if they were not there? Also, including a \".\" in\na member name may be troublesome for JSON, as that typically\nmeans we are accessing a member of an object. Perhaps use _sec\nand _nsec here and in the earlier stat_data block.\n\nThanks,\n-Stolee\n"},{"id":"377515","messageId":"3dba92fa-68ce-8204-fdfe-a0435528a9c2@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-5-pclouds@gmail.com","subject":"Re: [PATCH 4/8] resolve-undo.c: dump \"REUC\" extension as json","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-19T13:16:30Z","receivedAt":"2019-06-19T13:16:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n\n> +\tif (jw) {\n> +\t\tjw_object_inline_begin_object(jw, \"resolve-undo\");\n> +\t\tjw_object_intmax(jw, \"ext-size\", size);\n> +\t\tjw_object_inline_begin_array(jw, \"entries\");\n> +\t}\n\nWhile reading this block, I noticed the use of hyphens in the\nmember names could cause some problems when translating into\nobject models in some languages. While this is valid JSON, I\nfound helpful recommendations in the Google JSON Style Guide [1]\nthat could apply here and elsewhere. Specifically, this\nrecommendation:\n\n \"Property names must be camel-cased, ascii strings.\"\n\nTreating JSON members as camel-cased variable names would\npromote consumption by third-party tools.\n\nThanks,\n-Stolee\n\n[1] https://google.github.io/styleguide/jsoncstyleguide.xml?showone=Property_Name_Format#Property_Name_Format\n"},{"id":"377516","messageId":"8a56ac88-f3b9-c80d-441e-8e74eb64921d@gmail.com","threadId":"51337","inReplyTo":"20190619095858.30124-7-pclouds@gmail.com","subject":"Re: [PATCH 6/8] read-cache.c: dump \"IEOT\" extension as json","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-19T13:18:38Z","receivedAt":"2019-06-19T13:18:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:> @@ -2266,7 +2271,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n> \t * to multi-thread the reading of the cache entries.\n> \t */\n> \tif (extension_offset && nr_threads > 1)\n> -\t\tieot = read_ieot_extension(mmap, mmap_size, extension_offset);\n> +\t\tieot = read_ieot_extension(mmap, mmap_size, extension_offset, NULL);\n\nI tried applying this series on top of v2.22.0 and ran into an issue\non this patch, and the message seemed to imply the problem was at this\nblock. I couldn't figure out what was wrong, but maybe the series is\nbased on a different commit?\n\nThat said, I applied the previous patches, compiled, and manually\ntested those features. Seemed to be working as advertised.\n\nThanks,\n-Stolee\n"},{"id":"377517","messageId":"CACsJy8A1Zkkf-QxZSioD61UUj05Xap=f5NJLhOzabVc=dLz25A@mail.gmail.com","threadId":"51337","inReplyTo":"8a56ac88-f3b9-c80d-441e-8e74eb64921d@gmail.com","subject":"Re: [PATCH 6/8] read-cache.c: dump \"IEOT\" extension as json","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-19T13:24:01Z","receivedAt":"2019-06-19T13:24:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jun 19, 2019 at 8:18 PM Derrick Stolee wrote:\n>\n> On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:> @@ -2266,7 +2271,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n> > * to multi-thread the reading of the cache entries.\n> > */\n> > if (extension_offset && nr_threads > 1)\n> > - ieot = read_ieot_extension(mmap, mmap_size, extension_offset);\n> > + ieot = read_ieot_extension(mmap, mmap_size, extension_offset, NULL);\n>\n> I tried applying this series on top of v2.22.0 and ran into an issue\n> on this patch, and the message seemed to imply the problem was at this\n> block. I couldn't figure out what was wrong, but maybe the series is\n> based on a different commit?\n\nit's on 'master', a6a95cd1b4 (The second batch, 2019-06-17). There are\na couple patches since v2.22.0 that touch read-cache.c, but they don't\ntouch these lines explictly...\n-- \nDuy\n"},{"id":"377522","messageId":"fd33bf49-3708-63fc-71e3-fe0d599245c2@gmail.com","threadId":"51337","inReplyTo":"CACsJy8A1Zkkf-QxZSioD61UUj05Xap=f5NJLhOzabVc=dLz25A@mail.gmail.com","subject":"Re: [PATCH 6/8] read-cache.c: dump \"IEOT\" extension as json","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-19T14:26:18Z","receivedAt":"2019-06-19T14:26:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2019 9:24 AM, Duy Nguyen wrote:\n> On Wed, Jun 19, 2019 at 8:18 PM Derrick Stolee wrote:\n>>\n>> On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:> @@ -2266,7 +2271,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n>>> * to multi-thread the reading of the cache entries.\n>>> */\n>>> if (extension_offset && nr_threads > 1)\n>>> - ieot = read_ieot_extension(mmap, mmap_size, extension_offset);\n>>> + ieot = read_ieot_extension(mmap, mmap_size, extension_offset, NULL);\n>>\n>> I tried applying this series on top of v2.22.0 and ran into an issue\n>> on this patch, and the message seemed to imply the problem was at this\n>> block. I couldn't figure out what was wrong, but maybe the series is\n>> based on a different commit?\n> \n> it's on 'master', a6a95cd1b4 (The second batch, 2019-06-17). There are\n> a couple patches since v2.22.0 that touch read-cache.c, but they don't\n> touch these lines explictly...\n\nThanks, I should have tried from master myself. Starting there worked.\n"},{"id":"377552","messageId":"20190619191709.GF28145@sigill.intra.peff.net","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-19T19:17:09Z","receivedAt":"2019-06-19T19:17:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 19, 2019 at 04:58:50PM +0700, Nguyễn Thái Ngọc Duy wrote:\n\n> This is probably just my itch. Every time I have to do something with\n> the index, I need to add a little bit code here, a little bit there to\n> get a better \"view\" of the index.\n> \n> This solves it for me. It allows me to see pretty much everything in the\n> index (except really low detail stuff like pathname compression). It's\n> readable by human, but also easy to parse if you need to do statistics\n> and stuff. You could even do a \"diff\" between two indexes.\n> \n> I'm not really sure if anybody else finds this useful. Because if not,\n> I guess there's not much point trying to merge it to git.git just for a\n> single user. Maintaining off tree is still a pain for me, but I think\n> I can manage it.\n\nI don't have any particular use for this, but I am all in favor of tools\nthat make it easier to access and analyze information kept in our\non-disk formats (some of this is available via --debug, I think, but\nAFAIK most of the extension bits are not).\n\nAnd I'd rather see something like JSON than inventing yet another ad-hoc\noutput format.\n\nI think your warning in the manpage that this is for debugging is fine,\nas it does not put us on the hook for maintaining the feature nor its\nformat forever. We might want to call it \"--debug=json\" or something,\nthough, in case we do want real stable json support later (though of\ncourse we would be free to steal the option then, since we're making no\npromises).\n\n-Peff\n"},{"id":"377607","messageId":"xmqqk1dgq525.fsf@gitster-ct.c.googlers.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-20T04:00:34Z","receivedAt":"2019-06-20T04:00:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This is probably just my itch. Every time I have to do something with\n> the index, I need to add a little bit code here, a little bit there to\n> get a better \"view\" of the index.\n\n;-) \n\nJSON is not particularly my cup-of-tea but it is better than many\nother things exactly for one reason (everybody and their dog have\nheard of it), and certainly is much superiour than inventing our own\nad-hoc format. \n\nThanks for working on this (I do not expect I would see an immediate\nneed for this myself, though).\n\n\n"},{"id":"377689","messageId":"1d24e390-5269-2a9a-4301-d4c462052e25@jeffhostetler.com","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2019-06-20T19:12:23Z","receivedAt":"2019-06-20T19:12:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n> This is probably just my itch. Every time I have to do something with\n> the index, I need to add a little bit code here, a little bit there to\n> get a better \"view\" of the index.\n> \n> This solves it for me. It allows me to see pretty much everything in the\n> index (except really low detail stuff like pathname compression). It's\n> readable by human, but also easy to parse if you need to do statistics\n> and stuff. You could even do a \"diff\" between two indexes.\n> \n> I'm not really sure if anybody else finds this useful. Because if not,\n> I guess there's not much point trying to merge it to git.git just for a\n> single user. Maintaining off tree is still a pain for me, but I think\n> I can manage it.\n> \n> Nguyễn Thái Ngọc Duy (8):\n> ls-files: add --json to dump the index\n> split-index.c: dump \"link\" extension as json\n> fsmonitor.c: dump \"FSMN\" extension as json\n> resolve-undo.c: dump \"REUC\" extension as json\n> read-cache.c: dump \"EOIE\" extension as json\n> read-cache.c: dump \"IEOT\" extension as json\n> cache-tree.c: dump \"TREE\" extension as json\n> dir.c: dump \"UNTR\" extension as json\n> \n> Documentation/git-ls-files.txt | 5 ++\n> builtin/ls-files.c | 30 +++++--\n> cache-tree.c | 41 ++++++++--\n> cache-tree.h | 5 +-\n> cache.h | 2 +\n> dir.c | 56 ++++++++++++-\n> dir.h | 4 +-\n> fsmonitor.c | 9 +++\n> json-writer.c | 30 +++++++\n> json-writer.h | 29 +++++++\n> read-cache.c | 139 ++++++++++++++++++++++++++++++---\n> resolve-undo.c | 36 ++++++++-\n> resolve-undo.h | 4 +-\n> split-index.c | 13 ++-\n> 14 files changed, 376 insertions(+), 27 deletions(-)\n> \n\nThanks for working on this! I've been wanting to do something\nlike this for a while. I too am tired of digging thru hex dumps\nor \"od\" output whenever I have an odd problem to investigate.\nThis will certainly help.\n\nJeff\n"},{"id":"377720","messageId":"CACsJy8AYw1QLjWALUNr7hSZemWyTSG0U8Fc_TH=9HNR-LcygTg@mail.gmail.com","threadId":"51337","inReplyTo":"20190619191709.GF28145@sigill.intra.peff.net","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-21T08:37:45Z","receivedAt":"2019-06-21T08:38:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 20, 2019 at 2:17 AM Jeff King wrote:\n> I think your warning in the manpage that this is for debugging is fine,\n> as it does not put us on the hook for maintaining the feature nor its\n> format forever. We might want to call it \"--debug=json\" or something,\n\nHmm.. does it mean we make --debug PARSE_OPT_OPTARG? In other words,\n\"--debug\" still means \"text\", --debug=json is obvious, but \"--debug\njson\" means \"text\" debug with pathspec \"json\". Which is really\nhorrible in my opinion.\n\nOr is it ok to just make the argument mandatory? That would be a\nbehavior change, but I suppose --debug is a thing only we use and\ncould still be a safe thing to do...\n\n> though, in case we do want real stable json support later (though of\n> course we would be free to steal the option then, since we're making no\n> promises).\n>\n> -Peff\n\n\n\n-- \nDuy\n"},{"id":"377740","messageId":"nycvar.QRO.7.76.6.1906211501350.44@tvgsbejvaqbjf.bet","threadId":"51337","inReplyTo":"e3b8d7bf-486d-d52f-9a33-6a7ff837552e@gmail.com","subject":"Re: [PATCH 1/8] ls-files: add --json to dump the index","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-21T13:04:16Z","receivedAt":"2019-06-21T13:04:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 19 Jun 2019, Derrick Stolee wrote:\n\n> On 6/19/2019 5:58 AM, Nguyễn Thái Ngọc Duy wrote:\n>\n> > +--json::\n> > +\tDump the entire index content in JSON format. This is for\n> > +\tdebugging purposes and the JSON structure may change from time\n> > +\tto time.\n> > +\n>\n> \"...purposes and the JSON structure may change from time to time\" may be\n> better as \"...purposes. The JSON structure is subject to change.\"\n\nIt would probably make even more sense to mark this as an experimental\nfeature for now (i.e. prefix the description with \"(EXPERIMENTAL) \", so\nthat users will have a harder time to miss that vague statement at the\nend.\n\nOnce the feature stabilized enough, it would probably make sense to start\nversioning the JSON format (`--json[=]`, defaulting to the\nlatest).\n\nThat would make this a pretty useful feature not only for debugging, I\nwould imagine, but really would set a precedent for a better \"API\" for\n3rd-party applications to use than the current one.\n\nThanks,\nDscho\n"},{"id":"377741","messageId":"nycvar.QRO.7.76.6.1906211515160.44@tvgsbejvaqbjf.bet","threadId":"51337","inReplyTo":"20190619191709.GF28145@sigill.intra.peff.net","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-21T13:16:52Z","receivedAt":"2019-06-21T13:16:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Wed, 19 Jun 2019, Jeff King wrote:\n\n> On Wed, Jun 19, 2019 at 04:58:50PM +0700, Nguyễn Thái Ngọc Duy wrote:\n>\n> > This is probably just my itch. Every time I have to do something with\n> > the index, I need to add a little bit code here, a little bit there to\n> > get a better \"view\" of the index.\n> >\n> > This solves it for me. It allows me to see pretty much everything in the\n> > index (except really low detail stuff like pathname compression). It's\n> > readable by human, but also easy to parse if you need to do statistics\n> > and stuff. You could even do a \"diff\" between two indexes.\n> >\n> > I'm not really sure if anybody else finds this useful. Because if not,\n> > I guess there's not much point trying to merge it to git.git just for a\n> > single user. Maintaining off tree is still a pain for me, but I think\n> > I can manage it.\n>\n> I don't have any particular use for this, but I am all in favor of tools\n> that make it easier to access and analyze information kept in our\n> on-disk formats (some of this is available via --debug, I think, but\n> AFAIK most of the extension bits are not).\n>\n> And I'd rather see something like JSON than inventing yet another ad-hoc\n> output format.\n>\n> I think your warning in the manpage that this is for debugging is fine,\n> as it does not put us on the hook for maintaining the feature nor its\n> format forever. We might want to call it \"--debug=json\" or something,\n> though, in case we do want real stable json support later (though of\n> course we would be free to steal the option then, since we're making no\n> promises).\n\nTraditionally, we have not catered well to 3rd-party applications in Git,\nand this JSON format would provide a way out of that problem.\n\nSo I would like *not* to lock the door on letting this feature stabilize\norganically.\n\nI'd be much more in favor of `--json[=]`, with an initial version\nof 0 to indicate that it really is unstable for now.\n\nCiao,\nDscho\n"},{"id":"377744","messageId":"CACsJy8A=Vhwgv8cvs8s2f_aJ5W335s8CdqRQCDSFrdJu2j9fjw@mail.gmail.com","threadId":"51337","inReplyTo":"nycvar.QRO.7.76.6.1906211515160.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-21T13:49:50Z","receivedAt":"2019-06-21T13:50:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jun 21, 2019 at 8:16 PM Johannes Schindelin\n wrote:\n\n> > I think your warning in the manpage that this is for debugging is fine,\n> > as it does not put us on the hook for maintaining the feature nor its\n> > format forever. We might want to call it \"--debug=json\" or something,\n> > though, in case we do want real stable json support later (though of\n> > course we would be free to steal the option then, since we're making no\n> > promises).\n>\n> Traditionally, we have not catered well to 3rd-party applications in Git,\n> and this JSON format would provide a way out of that problem.\n>\n> So I would like *not* to lock the door on letting this feature stabilize\n> organically.\n>\n> I'd be much more in favor of `--json[=]`, with an initial version\n> of 0 to indicate that it really is unstable for now.\n\nConsidering the amount of code to output these, supporting multiple\nformats would be a nightmare. I may be ok with versioning the output\nso the tool know what format they need to deal with, but I'd rather\nsupport just one version. For third parties wanting to dig deep, I\nthink libgit2 would be a much better fit.\n-- \nDuy\n"},{"id":"377748","messageId":"xmqqd0j7km7x.fsf@gitster-ct.c.googlers.com","threadId":"51337","inReplyTo":"CACsJy8A=Vhwgv8cvs8s2f_aJ5W335s8CdqRQCDSFrdJu2j9fjw@mail.gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-21T15:10:58Z","receivedAt":"2019-06-21T15:11:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Considering the amount of code to output these, supporting multiple\n> formats would be a nightmare. I may be ok with versioning the output\n> so the tool know what format they need to deal with, but I'd rather\n> support just one version. For third parties wanting to dig deep, I\n> think libgit2 would be a much better fit.\n\nYeah, I think starting with --debug=json (or --debug-json) until we\nsee some stability in the output and got comfortable to the idea of\n\"version X\" to mean what we output at that point, and then renaming\nit to \"--json\" with \"version: 1\" in the output stream so that third\nparty can use it (and interpret it according to version 1 rules) is\nthe way to go. Third-party tools are welcome to read --debug-json\noutput as an early-adoption practice waiting for the real thing, but\nwe do not want to be locked into a schema too eary before we are\nready.\n\nThanks.\n"},{"id":"377773","messageId":"20190621204814.GB25874@sigill.intra.peff.net","threadId":"51337","inReplyTo":"CACsJy8AYw1QLjWALUNr7hSZemWyTSG0U8Fc_TH=9HNR-LcygTg@mail.gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-21T20:48:14Z","receivedAt":"2019-06-21T20:48:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2019 at 03:37:45PM +0700, Duy Nguyen wrote:\n\n> On Thu, Jun 20, 2019 at 2:17 AM Jeff King wrote:\n> > I think your warning in the manpage that this is for debugging is fine,\n> > as it does not put us on the hook for maintaining the feature nor its\n> > format forever. We might want to call it \"--debug=json\" or something,\n> \n> Hmm.. does it mean we make --debug PARSE_OPT_OPTARG? In other words,\n> \"--debug\" still means \"text\", --debug=json is obvious, but \"--debug\n> json\" means \"text\" debug with pathspec \"json\". Which is really\n> horrible in my opinion.\n\nYeah, that's the nature of OPTARG. ;)\n\n> Or is it ok to just make the argument mandatory? That would be a\n> behavior change, but I suppose --debug is a thing only we use and\n> could still be a safe thing to do...\n\nYeah, I think that would be perfectly fine (or you could just call it\n--debug-json as a new option, if you didn't want to make people do\n--debug=text for the existing behavior).\n\n-Peff\n"},{"id":"377774","messageId":"20190621205150.GC25874@sigill.intra.peff.net","threadId":"51337","inReplyTo":"nycvar.QRO.7.76.6.1906211515160.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-21T20:51:50Z","receivedAt":"2019-06-21T20:51:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2019 at 03:16:52PM +0200, Johannes Schindelin wrote:\n\n> > I think your warning in the manpage that this is for debugging is fine,\n> > as it does not put us on the hook for maintaining the feature nor its\n> > format forever. We might want to call it \"--debug=json\" or something,\n> > though, in case we do want real stable json support later (though of\n> > course we would be free to steal the option then, since we're making no\n> > promises).\n> \n> Traditionally, we have not catered well to 3rd-party applications in Git,\n> and this JSON format would provide a way out of that problem.\n> \n> So I would like *not* to lock the door on letting this feature stabilize\n> organically.\n\nI'd like it to stabilize organically, too, but my thinking was that we'd\nwait a while and then promote it to a stable name eventually.\n\n> I'd be much more in favor of `--json[=]`, with an initial version\n> of 0 to indicate that it really is unstable for now.\n\nThat's OK with me, too, if you think \"0\" indicates that sufficiently\n(we've used \"v0\" in a lot of other places to refer to stable protocols,\nlike the git:// one). Maybe it's OK with some documentation making it\nclear.\n\nI'm not sure whether we want to be locked into supporting this v0\nforever or not (though maybe it would not be such a burden).\n\nI think JSON-based output also has the potential to need fewer bumps.\nIt's syntactically stable, so it's really just about our schema. And\nit's easy to say \"newer versions of Git may produce new keys; you can\nignore them\", as long as we do not change the meaning of existing keys.\nThat might be an easier promise to make.\n\n-Peff\n"},{"id":"377775","messageId":"20190621205238.GD25874@sigill.intra.peff.net","threadId":"51337","inReplyTo":"xmqqd0j7km7x.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-21T20:52:39Z","receivedAt":"2019-06-21T20:52:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2019 at 08:10:58AM -0700, Junio C Hamano wrote:\n\n> Duy Nguyen writes:\n> \n> > Considering the amount of code to output these, supporting multiple\n> > formats would be a nightmare. I may be ok with versioning the output\n> > so the tool know what format they need to deal with, but I'd rather\n> > support just one version. For third parties wanting to dig deep, I\n> > think libgit2 would be a much better fit.\n> \n> Yeah, I think starting with --debug=json (or --debug-json) until we\n> see some stability in the output and got comfortable to the idea of\n> \"version X\" to mean what we output at that point, and then renaming\n> it to \"--json\" with \"version: 1\" in the output stream so that third\n> party can use it (and interpret it according to version 1 rules) is\n> the way to go. Third-party tools are welcome to read --debug-json\n> output as an early-adoption practice waiting for the real thing, but\n> we do not want to be locked into a schema too eary before we are\n> ready.\n\nI should have read the whole thread before responding. I made a similar\ncomment to Dscho, so I guess that is now two of us. :)\n\n-Peff\n"},{"id":"377795","messageId":"20190621233055.GB965782@genre.crustytoothpaste.net","threadId":"51337","inReplyTo":"20190619095858.30124-1-pclouds@gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-06-21T23:30:55Z","receivedAt":"2019-06-21T23:31:04Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2019-06-19 at 09:58:50, Nguyễn Thái Ngọc Duy wrote:\n> This is probably just my itch. Every time I have to do something with\n> the index, I need to add a little bit code here, a little bit there to\n> get a better \"view\" of the index.\n> \n> This solves it for me. It allows me to see pretty much everything in the\n> index (except really low detail stuff like pathname compression). It's\n> readable by human, but also easy to parse if you need to do statistics\n> and stuff. You could even do a \"diff\" between two indexes.\n> \n> I'm not really sure if anybody else finds this useful. Because if not,\n> I guess there's not much point trying to merge it to git.git just for a\n> single user. Maintaining off tree is still a pain for me, but I think\n> I can manage it.\n\nI'm generally in favor of this, but we need to document what this does\nwhen it encounters paths that are not valid UTF-8. (Ideally, the answer\nis, \"die()\", but I suspect the answer will be \"silently produce invalid\noutput\".) Those can of course occur on Unix systems, but also on\nWindows, where unpaired surrogates can occur.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"377799","messageId":"CACsJy8DX9QtG+3=0_H4sVOdNArXWcTapb7dfY53kGtLABRYOYQ@mail.gmail.com","threadId":"51337","inReplyTo":"20190621233055.GB965782@genre.crustytoothpaste.net","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-22T02:54:45Z","receivedAt":"2019-06-22T02:55:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jun 22, 2019 at 6:31 AM brian m. carlson\n wrote:\n>\n> On 2019-06-19 at 09:58:50, Nguyễn Thái Ngọc Duy wrote:\n> > This is probably just my itch. Every time I have to do something with\n> > the index, I need to add a little bit code here, a little bit there to\n> > get a better \"view\" of the index.\n> >\n> > This solves it for me. It allows me to see pretty much everything in the\n> > index (except really low detail stuff like pathname compression). It's\n> > readable by human, but also easy to parse if you need to do statistics\n> > and stuff. You could even do a \"diff\" between two indexes.\n> >\n> > I'm not really sure if anybody else finds this useful. Because if not,\n> > I guess there's not much point trying to merge it to git.git just for a\n> > single user. Maintaining off tree is still a pain for me, but I think\n> > I can manage it.\n>\n> I'm generally in favor of this, but we need to document what this does\n> when it encounters paths that are not valid UTF-8. (Ideally, the answer\n> is, \"die()\", but I suspect the answer will be \"silently produce invalid\n> output\".)\n\nI think you're right, we don't assume anything when writing json\nstrings, so it's not going to be utf-8 (or die) if the path is also\nnot valid utf-8. The good thing is all this could be done in just one\nplace, append_quoted_string(), if someone needs too. I'll just go\ndocument the fact that we may produce invalid UTF-8.\n-- \nDuy\n"},{"id":"377846","messageId":"nycvar.QRO.7.76.6.1906241129490.44@tvgsbejvaqbjf.bet","threadId":"51337","inReplyTo":"CACsJy8A=Vhwgv8cvs8s2f_aJ5W335s8CdqRQCDSFrdJu2j9fjw@mail.gmail.com","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-24T09:33:01Z","receivedAt":"2019-06-24T09:32:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Fri, 21 Jun 2019, Duy Nguyen wrote:\n\n> On Fri, Jun 21, 2019 at 8:16 PM Johannes Schindelin\n> wrote:\n>\n> > > I think your warning in the manpage that this is for debugging is fine,\n> > > as it does not put us on the hook for maintaining the feature nor its\n> > > format forever. We might want to call it \"--debug=json\" or something,\n> > > though, in case we do want real stable json support later (though of\n> > > course we would be free to steal the option then, since we're making no\n> > > promises).\n> >\n> > Traditionally, we have not catered well to 3rd-party applications in Git,\n> > and this JSON format would provide a way out of that problem.\n> >\n> > So I would like *not* to lock the door on letting this feature stabilize\n> > organically.\n> >\n> > I'd be much more in favor of `--json[=]`, with an initial version\n> > of 0 to indicate that it really is unstable for now.\n>\n> Considering the amount of code to output these, supporting multiple\n> formats would be a nightmare. I may be ok with versioning the output\n> so the tool know what format they need to deal with, but I'd rather\n> support just one version.\n\nOnce the format stabilized, I don't think it would be a huge burden to\nsupport multiple formats, if we ever had to update.\n\nIt would, however, be a huge burden on third-party applications. In\neffect, we could be lazy, but we would put a lot more burden on others\nthan we saved ourselves, so that would be a bit... selfish.\n\n> For third parties wanting to dig deep, I think libgit2 would be a much\n> better fit.\n\nIf we (i.e. the core Git contributors) were contributing new features/bug\nfixes to libgit2, that would be a good recommendation.\n\nBut we don't. We essentially ignore libgit2 (and all of their learnings)\nall the time.\n\nEven worse, for years, even decades, we recommended the command-line as\n\"the API\". If you want to reverse that recommendation, I think it merits a\nbigger discussion than a flimsical comment buried in a thread about an\nexperimental feature.\n\nCiao,\nDscho\n"},{"id":"377847","messageId":"nycvar.QRO.7.76.6.1906241133080.44@tvgsbejvaqbjf.bet","threadId":"51337","inReplyTo":"20190621205238.GD25874@sigill.intra.peff.net","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-24T09:35:08Z","receivedAt":"2019-06-24T09:35:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff & Junio,\n\nOn Fri, 21 Jun 2019, Jeff King wrote:\n\n> On Fri, Jun 21, 2019 at 08:10:58AM -0700, Junio C Hamano wrote:\n>\n> > Duy Nguyen writes:\n> >\n> > > Considering the amount of code to output these, supporting multiple\n> > > formats would be a nightmare. I may be ok with versioning the output\n> > > so the tool know what format they need to deal with, but I'd rather\n> > > support just one version. For third parties wanting to dig deep, I\n> > > think libgit2 would be a much better fit.\n> >\n> > Yeah, I think starting with --debug=json (or --debug-json) until we\n> > see some stability in the output and got comfortable to the idea of\n> > \"version X\" to mean what we output at that point, and then renaming\n> > it to \"--json\" with \"version: 1\" in the output stream so that third\n> > party can use it (and interpret it according to version 1 rules) is\n> > the way to go. Third-party tools are welcome to read --debug-json\n> > output as an early-adoption practice waiting for the real thing, but\n> > we do not want to be locked into a schema too eary before we are\n> > ready.\n>\n> I should have read the whole thread before responding. I made a similar\n> comment to Dscho, so I guess that is now two of us. :)\n\nIt is a bit of a chicken-and-egg problem. You want the format to\nstabilize. But you also don't want to commit to one final format. And you\nchoose as option name a deliberately discouraging one, deterring the\n(third-party application) developers who could most help you evolve the\nformat to a sensible and useful stable version.\n\nCiao,\nDscho\n"},{"id":"377848","messageId":"CACsJy8BtqiUx92=U8kTe_0LFv2PFkUq3BaBqhpLfx5tgBHENuw@mail.gmail.com","threadId":"51337","inReplyTo":"nycvar.QRO.7.76.6.1906241129490.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-24T09:35:36Z","receivedAt":"2019-06-24T09:36:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 24, 2019 at 4:32 PM Johannes Schindelin\n wrote:\n>\n> Hi Duy,\n>\n> On Fri, 21 Jun 2019, Duy Nguyen wrote:\n>\n> > On Fri, Jun 21, 2019 at 8:16 PM Johannes Schindelin\n> > wrote:\n> >\n> > > > I think your warning in the manpage that this is for debugging is fine,\n> > > > as it does not put us on the hook for maintaining the feature nor its\n> > > > format forever. We might want to call it \"--debug=json\" or something,\n> > > > though, in case we do want real stable json support later (though of\n> > > > course we would be free to steal the option then, since we're making no\n> > > > promises).\n> > >\n> > > Traditionally, we have not catered well to 3rd-party applications in Git,\n> > > and this JSON format would provide a way out of that problem.\n> > >\n> > > So I would like *not* to lock the door on letting this feature stabilize\n> > > organically.\n> > >\n> > > I'd be much more in favor of `--json[=]`, with an initial version\n> > > of 0 to indicate that it really is unstable for now.\n> >\n> > Considering the amount of code to output these, supporting multiple\n> > formats would be a nightmare. I may be ok with versioning the output\n> > so the tool know what format they need to deal with, but I'd rather\n> > support just one version.\n>\n> Once the format stabilized, I don't think it would be a huge burden to\n> support multiple formats, if we ever had to update.\n>\n> It would, however, be a huge burden on third-party applications. In\n> effect, we could be lazy, but we would put a lot more burden on others\n> than we saved ourselves, so that would be a bit... selfish.\n\nJSON is the land of high level languages. They can adapt to new format\nquite easily, compared to restructuring C to support multiple\ndifferent formats. Yes I'm quite OK with being selfish in this case.\n-- \nDuy\n"},{"id":"377849","messageId":"nycvar.QRO.7.76.6.1906241136040.44@tvgsbejvaqbjf.bet","threadId":"51337","inReplyTo":"20190621205150.GC25874@sigill.intra.peff.net","subject":"Re: [PATCH 0/8] Add 'ls-files --json' to dump the index in json","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-24T09:52:45Z","receivedAt":"2019-06-24T09:52:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 21 Jun 2019, Jeff King wrote:\n\n> On Fri, Jun 21, 2019 at 03:16:52PM +0200, Johannes Schindelin wrote:\n>\n> > > I think your warning in the manpage that this is for debugging is fine,\n> > > as it does not put us on the hook for maintaining the feature nor its\n> > > format forever. We might want to call it \"--debug=json\" or something,\n> > > though, in case we do want real stable json support later (though of\n> > > course we would be free to steal the option then, since we're making no\n> > > promises).\n> >\n> > Traditionally, we have not catered well to 3rd-party applications in Git,\n> > and this JSON format would provide a way out of that problem.\n> >\n> > So I would like *not* to lock the door on letting this feature stabilize\n> > organically.\n>\n> I'd like it to stabilize organically, too, but my thinking was that we'd\n> wait a while and then promote it to a stable name eventually.\n\nGit's command-line options have stabilized organically.\n\nExample: to include untracked files in `git stash`, use `-u` or\n`--include-untracked`, to include them in `git add`, use `-A` or `--all`,\nto include them in `git grep`, use `--untracked` (no short option), to\ninclude them in `git ls-files`, use `-o` or `--others`. The command `git\ncommit` does not even have an option to include untracked files.\n\nYou know of more examples of organically grown designs in Git, I am sure.\nGiven those examples, I am not sure that I want the JSON format to\nstabilize organically.\n\n> > I'd be much more in favor of `--json[=]`, with an initial\n> > version of 0 to indicate that it really is unstable for now.\n>\n> That's OK with me, too, if you think \"0\" indicates that sufficiently\n> (we've used \"v0\" in a lot of other places to refer to stable protocols,\n> like the git:// one). Maybe it's OK with some documentation making it\n> clear.\n\nI did think that the `0` would be clear, but you are probably right.\n\n> I'm not sure whether we want to be locked into supporting this v0\n> forever or not (though maybe it would not be such a burden).\n>\n> I think JSON-based output also has the potential to need fewer bumps.\n> It's syntactically stable, so it's really just about our schema. And\n> it's easy to say \"newer versions of Git may produce new keys; you can\n> ignore them\", as long as we do not change the meaning of existing keys.\n> That might be an easier promise to make.\n\nRight.\n\nThanks,\nDscho\n"},{"id":"377870","messageId":"CACsJy8B7j=G2VdYPU8W4eoGJhmJ=7r5gK0FV2QQ8twjzNQQzuQ@mail.gmail.com","threadId":"51337","inReplyTo":"e3b8d7bf-486d-d52f-9a33-6a7ff837552e@gmail.com","subject":"Re: [PATCH 1/8] ls-files: add --json to dump the index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-24T12:50:16Z","receivedAt":"2019-06-24T12:50:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jun 19, 2019 at 8:03 PM Derrick Stolee wrote:\n> > - show_files(the_repository, &dir);\n> > -\n> > - if (show_resolve_undo)\n> > - show_ru_info(the_repository->index);\n> > + if (!show_json) {\n> > + show_files(the_repository, &dir);\n> > +\n> > + if (show_resolve_undo)\n> > + show_ru_info(the_repository->index);\n> > + } else {\n> > + struct json_writer jw = JSON_WRITER_INIT;\n> > +\n> > + discard_index(the_repository->index);\n> > + the_repository->index->jw = &jw;\n> > + if (repo_read_index(the_repository) < 0)\n> > + die(\"index file corrupt\");\n> > + puts(jw.json.buf);\n> > + the_repository->index->jw = NULL;\n> > + jw_release(&jw);\n> > + }\n> >\n> > if (ps_matched) {\n> > int bad;\n>\n> I see this 'ps_matched' condition at the end, which is related to\n> the '--error-unmatch' option. I added \"--error-unmatch foo\" to my\n> command and got the appropriate error message:\n>\n> error: pathspec 'foo' did not match any file(s) known to git\n> Did you forget to 'git add'?\n>\n> This was sent to stderr while the JSON was in stdout, so this should\n> be appropriate to allow both options. Just pointing it out to make\n> sure this is intended.\n\n--error-unmatch only makes sense when you specify pathspec (like\n\"foo\") but that does not work well with --json at all because we don't\ndo filtering (how do we even filter in extensions?). I'll just make\nsure that \"ls-files --json \" is rejected. That'll cover\n--error-unmatch.\n-- \nDuy\n"}]}