{"thread":{"id":"48185","subject":"[PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","startedAt":"2018-03-29T15:18:49Z","lastAt":"2018-05-08T14:00:59Z","messageCount":103,"participants":["Johannes Schindelin","Stefan Beller","Jeff King","Junio C Hamano","Ævar Arnfjörð Bjarmason","Duy Nguyen","Eric Sunshine"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"343334","messageId":"cover.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":null,"subject":"[PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:30Z","receivedAt":"2018-03-29T15:18:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This patch series started out as a single patch trying to figure out what it\ntakes to fix that annoying bug that has been reported several times over the\nyears, where `git config --unset` would leave empty sections behind, and `git\nconfig --add` would not reuse them.\n\nLittle did I know that this would turn not only into a full patch to fix this\nissue, but into a full-blown series of nine patches.\n\nThe first patch is somewhat of a \"while at it\" bug fix that I first thought\nwould be a lot more critical than it actually is: It really only affects config\nfiles that start with a section followed immediately (i.e. without a newline)\nby a one-letter boolean setting (i.e. without a `= ` part). So while it\nis a real bug fix, I doubt anybody ever got bitten by it.\n\nNonetheless, I would be confortable with this patch going into v2.17.0, even at\nthis late stage. The final verdict is Junio's, of course.\n\nThe next swath of patches add some tests, and adjust one test about which I\nalready complained at length yesterday, so I will spare you the same ordeal\ntoday. These fixes are pretty straight-forward, and I always try to keep my\nadded tests as concise as possible, so please tell me if you find a way to make\nthem smaller (without giving up readability and debuggability).\n\nFinally, the interesting part, where I do two things, essentially (with\npreparatory steps for each thing):\n\n1. I add the ability for `git config --unset/--unset-all` to detect that it\n can remove a section that has just become empty (see below for some more\n discussion of what I consider \"become empty\"), and\n\n2. I add the ability for `git config [--add] key value` to re-use empty\n sections.\n\nNote that the --unset/--unset-all part is the hairy one, and I would love it if\npeople could concentrate on wrapping their heads around that function, and\nobviously tell me how I can change it to make it more readable (or even point\nout incorrect behavior).\n\nNow, to the really important part: why does this patch series not conflict with\nmy very early statements that we cannot simply remove empty sections because we\nmay end up with stale comments?\n\nWell, the patch in question takes pains to determine *iff* there are any\ncomments surrounding, or included in, the section. If any are found: previous\nbehavior. Under the assumption that the user edited the file, we keep it as\nintact as possible (see below for some argument against this). If no comments\nare found, and let's face it, this is probably *the* common case, as few people\nedit their config files by hand these days (neither should they because it is\ntoo easy to end up with an unparseable one), the now-empty section *is*\nremoved.\n\nSo what is the argument against this extra care to detect comments? Well, if\nyou have something like this:\n\n\t[section]\n\t\t; Here we comment about the variable called snarf\n\t\tsnarf = froop\n\nand we run `git config --unset section.snarf`, we end up with this config:\n\n\t[section]\n\t\t; Here we comment about the variable called snarf\n\nwhich obviously does not make sense. However, that is already established\nbehavior for quite a few years, and I do not even try to think of a way how\nthis could be solved.\n\n\nJohannes Schindelin (9):\n git_config_set: fix off-by-two\n t1300: rename it to reflect that `repo-config` was deprecated\n t1300: avoid relying on a bug\n t1300: remove unreasonable expectation from TODO\n t1300: `--unset-all` can leave an empty section behind (bug)\n git_config_set: simplify the way the section name is remembered\n git config --unset: remove empty sections (in normal situations)\n git_config_set: use do_config_from_file() directly\n git_config_set: reuse empty sections\n\n config.c | 234 +++++++++++++++++++++++++---\n t/{t1300-repo-config.sh => t1300-config.sh} | 36 ++++-\n 2 files changed, 250 insertions(+), 20 deletions(-)\n rename t/{t1300-repo-config.sh => t1300-config.sh} (98%)\n\n\nbase-commit: 03df4959472e7d4b5117bb72ac86e1e2bcf21723\nPublished-As: https://github.com/dscho/git/releases/tag/empty-config-section-v1\nFetch-It-Via: git fetch https://github.com/dscho/git empty-config-section-v1\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n"},{"id":"343335","messageId":"899ea23951627426ccd0aac79f824af386c5590c.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 1/9] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:35Z","receivedAt":"2018-03-29T15:18:52Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Currently, we are slightly overzealous When removing an entry from a\nconfig file of this form:\n\n\t[abc]a\n\t[xyz]\n\t\tkey = value\n\nWhen calling `git config --unset abc.a` on this file, it leaves this\n(invalid) config behind:\n\n\t[\n\t[xyz]\n\t\tkey = value\n\nThe reason is that we try to search for the beginning of the line (or\nfor the end of the preceding section header on the same line) that\ndefines abc.a, but as an optimization, we subtract 2 from the offset\npointing just after the definition before we call\nfind_beginning_of_line(). That function, however, *also* performs that\noptimization and promptly fails to find the section header correctly.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/config.c b/config.c\nindex b0c20e6cb8a..5cc049aaef0 100644\n--- a/config.c\n+++ b/config.c\n@@ -2632,7 +2632,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n \t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.offset[i]-2, &new_line);\n+\t\t\t\t\tstore.offset[i], &new_line);\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343336","messageId":"6023d70cff199424893d116339f860952c6a6b4b.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 2/9] t1300: rename it to reflect that `repo-config` was deprecated","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:40Z","receivedAt":"2018-03-29T15:18:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Signed-off-by: Johannes Schindelin \n---\n t/{t1300-repo-config.sh => t1300-config.sh} | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n rename t/{t1300-repo-config.sh => t1300-config.sh} (100%)\n\ndiff --git a/t/t1300-repo-config.sh b/t/t1300-config.sh\nsimilarity index 100%\nrename from t/t1300-repo-config.sh\nrename to t/t1300-config.sh\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343337","messageId":"385674162d2853f6aea153d9d30482034d9fa3df.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 3/9] t1300: avoid relying on a bug","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:45Z","receivedAt":"2018-03-29T15:19:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The test case 'unset with cont. lines' relied on a bug that is about to\nbe fixed: it tests *explicitly* that removing the last entry from a\nconfig section leaves an *empty* section behind.\n\nLet's fix this test case not to rely on that behavior, simply by\npreventing the section from becoming empty.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 4f8e6f5fde3..1ece7bad05f 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -108,6 +108,7 @@ bar = foo\n [beta]\n baz = multiple \\\n lines\n+foo = bar\n EOF\n \n test_expect_success 'unset with cont. lines' '\n@@ -118,6 +119,7 @@ cat > expect <<\\EOF\n [alpha]\n bar = foo\n [beta]\n+foo = bar\n EOF\n \n test_expect_success 'unset with cont. lines is correct' 'test_cmp expect .git/config'\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343338","messageId":"484133f267ec2ebb6d6c27ccb40410579c72032a.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 5/9] t1300: `--unset-all` can leave an empty section behind (bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:53Z","receivedAt":"2018-03-29T15:19:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"We already have a test demonstrating that removing the last entry from a\nconfig section fails to remove the section header of the now-empty\nsection.\n\nThe same can happen, of course, if we remove the last entries in one fell\nswoop. This is *also* a bug, and should be fixed at the same time.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 3ad3df0c83e..ff79a213567 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1452,6 +1452,17 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \ttest_cmp expect .git/config\n '\n \n+test_expect_failure '--unset-all removes section if empty & uncommented' '\n+\tcat >.git/config <<-\\EOF &&\n+\t[section]\n+\tkey = value1\n+\tkey = value2\n+\tEOF\n+\n+\tgit config --unset-all section.key &&\n+\ttest_line_count = 0 .git/config\n+'\n+\n test_expect_failure 'adding a key into an empty section reuses header' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343339","messageId":"5801f7529465b7dbdfbe3c105e2a8212c253ba0d.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 4/9] t1300: remove unreasonable expectation from TODO","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:50Z","receivedAt":"2018-03-29T15:19:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"In https://public-inbox.org/git/7vvc8alzat.fsf@alter.siamese.dyndns.org/\na reasonable patch was made quite a bit less so by changing a test case\ndemonstrating a bug to a test case that demonstrates that we ask for too\nmuch: the test case 'unsetting the last key in a section removes header'\nnow expects a future bug fix to be able to determine whether a free-form\ncomment above a section header refers to said section or not.\n\nRather than shooting for the stars (and not even getting off the\nground), let's start shooting for something obtainable and be reasonably\nconfident that we *can* get it.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 1ece7bad05f..3ad3df0c83e 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1413,7 +1413,7 @@ test_expect_success 'urlmatch with wildcard' '\n '\n \n # good section hygiene\n-test_expect_failure 'unsetting the last key in a section removes header' '\n+test_expect_failure '--unset last key removes section (except if commented)' '\n \tcat >.git/config <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n \t# a comment specific to this \"section\" section.\n@@ -1427,6 +1427,25 @@ test_expect_failure 'unsetting the last key in a section removes header' '\n \n \tcat >expect <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n+\t# a comment specific to this \"section\" section.\n+\t[section]\n+\t# some intervening lines\n+\t# that should also be dropped\n+\n+\t# please be careful when you update the above variable\n+\tEOF\n+\n+\tgit config --unset section.key &&\n+\ttest_cmp expect .git/config &&\n+\n+\tcat >.git/config <<-\\EOF &&\n+\t[section]\n+\tkey = value\n+\t[next-section]\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[next-section]\n \tEOF\n \n \tgit config --unset section.key &&\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343340","messageId":"01ae8d501a9c605d16f1ab83212b8fc3bda49f48.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 6/9] git_config_set: simplify the way the section name is remembered","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:18:57Z","receivedAt":"2018-03-29T15:19:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This not only reduces the number of lines, but also opens the door for\nreusing the section name later (which the upcoming patch to remove\nnow-empty sections will do).\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 10 ++++------\n 1 file changed, 4 insertions(+), 6 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 5cc049aaef0..d35dffa50de 100644\n--- a/config.c\n+++ b/config.c\n@@ -2486,12 +2486,14 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tstruct lock_file lock = LOCK_INIT;\n \tchar *filename_buf = NULL;\n \tchar *contents = NULL;\n+\tchar *section_name = NULL;\n \tsize_t contents_sz;\n \n \t/* parse-key returns negative; flip the sign to feed exit(3) */\n-\tret = 0 - git_config_parse_key(key, &store.key, &store.baselen);\n+\tret = 0 - git_config_parse_key(key, §ion_name, &store.baselen);\n \tif (ret)\n \t\tgoto out_free;\n+\tstore.key = section_name;\n \n \tstore.multi_replace = multi_replace;\n \n@@ -2505,7 +2507,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tfd = hold_lock_file_for_update(&lock, config_filename, 0);\n \tif (fd < 0) {\n \t\terror_errno(\"could not lock config file %s\", config_filename);\n-\t\tfree(store.key);\n \t\tret = CONFIG_NO_LOCK;\n \t\tgoto out_free;\n \t}\n@@ -2515,8 +2516,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t */\n \tin_fd = open(config_filename, O_RDONLY);\n \tif ( in_fd < 0 ) {\n-\t\tfree(store.key);\n-\n \t\tif ( ENOENT != errno ) {\n \t\t\terror_errno(\"opening %s\", config_filename);\n \t\t\tret = CONFIG_INVALID_FILE; /* same as \"invalid config file\" */\n@@ -2571,7 +2570,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t */\n \t\tif (git_config_from_file(store_aux, config_filename, NULL)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n-\t\t\tfree(store.key);\n \t\t\tif (store.value_regex != NULL &&\n \t\t\t store.value_regex != CONFIG_REGEX_NONE) {\n \t\t\t\tregfree(store.value_regex);\n@@ -2581,7 +2579,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tgoto out_free;\n \t\t}\n \n-\t\tfree(store.key);\n \t\tif (store.value_regex != NULL &&\n \t\t store.value_regex != CONFIG_REGEX_NONE) {\n \t\t\tregfree(store.value_regex);\n@@ -2682,6 +2679,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n out_free:\n \trollback_lock_file(&lock);\n+\tfree(section_name);\n \tfree(filename_buf);\n \tif (contents)\n \t\tmunmap(contents, contents_sz);\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343341","messageId":"c246c8bc2fb1cd6fe6307463d299cf56fbe4dc5b.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 7/9] git config --unset: remove empty sections (in normal situations)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:19:00Z","receivedAt":"2018-03-29T15:19:15Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The original reasoning for not removing section headers upon removal of\nthe last entry went like this: the user could have added comments about\nthe section, or about the entries therein, and if there were other\ncomments there, we would not know whether we should remove them.\n\nIn particular, a concocted example was presented that looked like this\n(and was added to t1300):\n\n\t# some generic comment on the configuration file itself\n\t# a comment specific to this \"section\" section.\n\t[section]\n\t# some intervening lines\n\t# that should also be dropped\n\n\tkey = value\n\t# please be careful when you update the above variable\n\nThe ideal thing for `git config --unset section.key` in this case would\nbe to leave only the first line behind, because all the other comments\nare now obsolete.\n\nHowever, this is unfeasible, short of adding a complete Natural Language\nProcessing module to Git, which seems not only a lot of work, but a\ntotally unreasonable feature (for little benefit to most users).\n\nNow, the real kicker about this problem is: most users do not edit their\nconfig files at all! In their use case, the config looks like this\ninstead:\n\n\t[section]\n\t\tkey = value\n\n... and it is totally obvious what should happen if the entry is\nremoved.\n\nLet's generalize this observation to this conservative strategy: if we\nare removing the last entry from a section, and there are no comments\ninside that section nor surrounding it, then remove the entire section.\nOtherwise behave as before: leave the now-empty section (including those\ncomments, even the one about the now-deleted entry).\n\nWe have to be careful, though, to handle also the case where there are\n*multiple* entries that are removed: any subset of them might be the last\nentries of their respective sections.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 181 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n t/t1300-config.sh | 4 +-\n 2 files changed, 182 insertions(+), 3 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex d35dffa50de..503aef4b318 100644\n--- a/config.c\n+++ b/config.c\n@@ -2429,6 +2429,177 @@ static ssize_t find_beginning_of_line(const char *contents, size_t size,\n \treturn offset;\n }\n \n+/*\n+ * This function determines whether the offset is in a line that starts with a\n+ * comment character.\n+ *\n+ * Note: it does *not* report when a regular line (section header, config\n+ * setting) *ends* in a comment.\n+ */\n+static int is_in_comment_line(const char *contents, size_t offset)\n+{\n+\tint comment = 0;\n+\n+\twhile (offset > 0)\n+\t\tswitch (contents[--offset]) {\n+\t\tcase ';':\n+\t\tcase '#':\n+\t\t\tcomment = 1;\n+\t\t\tbreak;\n+\t\tcase '\\n':\n+\t\t\tbreak;\n+\t\tcase ' ':\n+\t\tcase '\\t':\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tcomment = 0;\n+\t\t}\n+\n+\treturn comment;\n+}\n+\n+/*\n+ * If we are about to unset the last key(s) in a section, and if there are\n+ * no comments surrounding (or included in) the section, we will want to\n+ * extend begin/end to remove the entire section.\n+ *\n+ * Note: the parameter `i_ptr` points to the index into the store.offset\n+ * array, reflecting the end offset of the respective entry to be deleted.\n+ * This index may be incremented if a section has more than one entry (which\n+ * all are to be removed).\n+ */\n+static void maybe_remove_section(const char *contents, size_t size,\n+\t\t\t\t const char *section_name,\n+\t\t\t\t size_t section_name_len,\n+\t\t\t\t size_t *begin, int *i_ptr, int *new_line)\n+{\n+\tsize_t begin2, end2;\n+\tint seen_section = 0, dummy, i = *i_ptr;\n+\n+\t/*\n+\t * First, make sure that this is the last key in the section, and that\n+\t * there are no comments that are possibly about the current section.\n+\t */\n+next_entry:\n+\tfor (end2 = store.offset[i]; end2 < size; end2++) {\n+\t\tswitch (contents[end2]) {\n+\t\tcase ' ':\n+\t\tcase '\\t':\n+\t\tcase '\\n':\n+\t\t\tcontinue;\n+\t\tcase '\\r':\n+\t\t\tif (++end2 < size && contents[end2] == '\\n')\n+\t\t\t\tcontinue;\n+\t\t\tbreak;\n+\t\tcase '[':\n+\t\t\t/* If the section name is repeated, continue */\n+\t\t\tif (end2 + 1 + section_name_len < size &&\n+\t\t\t contents[end2 + section_name_len] == ']' &&\n+\t\t\t !memcmp(contents + end2 + 1, section_name,\n+\t\t\t\t section_name_len)) {\n+\t\t\t\tend2 += section_name_len;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tgoto look_before;\n+\t\tcase ';':\n+\t\tcase '#':\n+\t\t\t/* There is a comment, cannot remove this section */\n+\t\t\treturn;\n+\t\tdefault:\n+\t\t\t/* There are other keys in that section */\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Uh oh... we found something else in this section. But do\n+\t\t * we want to remove this, too?\n+\t\t */\n+\t\tif (++i >= store.seen)\n+\t\t\treturn;\n+\n+\t\tbegin2 = find_beginning_of_line(contents, size, store.offset[i],\n+\t\t\t\t\t\t&dummy);\n+\t\tif (begin2 > end2)\n+\t\t\treturn;\n+\n+\t\t/* Looks like we want to remove the next one, too... */\n+\t\tgoto next_entry;\n+\t}\n+\n+look_before:\n+\t/*\n+\t * Now, ensure that this is the first key, and that there are no\n+\t * comments before the entry nor before the section header.\n+\t */\n+\tfor (begin2 = *begin; begin2 > 0; )\n+\t\tswitch (contents[begin2 - 1]) {\n+\t\tcase ' ':\n+\t\tcase '\\t':\n+\t\t\tbegin2--;\n+\t\t\tcontinue;\n+\t\tcase '\\n':\n+\t\t\tif (--begin2 > 0 && contents[begin2 - 1] == '\\r')\n+\t\t\t\tbegin2--;\n+\t\t\tcontinue;\n+\t\tcase ']':\n+\t\t\tif (begin2 > section_name_len + 1 &&\n+\t\t\t contents[begin2 - section_name_len - 2] == '[' &&\n+\t\t\t !memcmp(contents + begin2 - section_name_len - 1,\n+\t\t\t\t section_name, section_name_len)) {\n+\t\t\t\tbegin2 -= section_name_len + 2;\n+\t\t\t\tseen_section = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * It looks like a section header, but it could be a\n+\t\t\t * comment instead...\n+\t\t\t */\n+\t\t\tif (is_in_comment_line(contents, begin2))\n+\t\t\t\treturn;\n+\n+\t\t\t/*\n+\t\t\t * We encountered the previous section header: This\n+\t\t\t * really was the only entry, so remove the entire\n+\t\t\t * section.\n+\t\t\t */\n+\t\t\tif (contents[begin2] != '\\n') {\n+\t\t\t\tbegin2--;\n+\t\t\t\t*new_line = 1;\n+\t\t\t}\n+\n+\t\t\tstore.offset[i] = end2;\n+\t\t\t*begin = begin2;\n+\t\t\t*i_ptr = i;\n+\t\t\treturn;\n+\t\tdefault:\n+\t\t\t/*\n+\t\t\t * Any other character means it is either a comment or\n+\t\t\t * a config setting; if it is a comment, we do not want\n+\t\t\t * to remove this section. If it is a config setting,\n+\t\t\t * we only want to remove this section if this is\n+\t\t\t * already the next section.\n+\t\t\t */\n+\t\t\tif (seen_section &&\n+\t\t\t !is_in_comment_line(contents, begin2)) {\n+\t\t\t\tif (contents[begin2] != '\\n') {\n+\t\t\t\t\tbegin2--;\n+\t\t\t\t\t*new_line = 1;\n+\t\t\t\t}\n+\n+\t\t\t\tstore.offset[i] = end2;\n+\t\t\t\t*begin = begin2;\n+\t\t\t\t*i_ptr = i;\n+\t\t\t}\n+\t\t\treturn;\n+\t\t}\n+\n+\t/* This section extends to the beginning of the file. */\n+\tstore.offset[i] = end2;\n+\t*begin = begin2;\n+\t*i_ptr = i;\n+}\n+\n int git_config_set_in_file_gently(const char *config_filename,\n \t\t\t\t const char *key, const char *value)\n {\n@@ -2626,10 +2797,18 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\tstore.offset[i] = copy_end = contents_sz;\n \t\t\t} else if (store.state != KEY_SEEN) {\n \t\t\t\tcopy_end = store.offset[i];\n-\t\t\t} else\n+\t\t\t} else {\n \t\t\t\tcopy_end = find_beginning_of_line(\n \t\t\t\t\tcontents, contents_sz,\n \t\t\t\t\tstore.offset[i], &new_line);\n+\t\t\t\tif (!value)\n+\t\t\t\t\tmaybe_remove_section(contents,\n+\t\t\t\t\t\t\t contents_sz,\n+\t\t\t\t\t\t\t section_name,\n+\t\t\t\t\t\t\t store.baselen,\n+\t\t\t\t\t\t\t ©_end, &i,\n+\t\t\t\t\t\t\t &new_line);\n+\t\t\t}\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex ff79a213567..ecbcc9cf3d0 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1413,7 +1413,7 @@ test_expect_success 'urlmatch with wildcard' '\n '\n \n # good section hygiene\n-test_expect_failure '--unset last key removes section (except if commented)' '\n+test_expect_success '--unset last key removes section (except if commented)' '\n \tcat >.git/config <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n \t# a comment specific to this \"section\" section.\n@@ -1452,7 +1452,7 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \ttest_cmp expect .git/config\n '\n \n-test_expect_failure '--unset-all removes section if empty & uncommented' '\n+test_expect_success '--unset-all removes section if empty & uncommented' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n \tkey = value1\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343342","messageId":"e3032300946eb5962878341f7796f5872c4d138d.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:19:04Z","receivedAt":"2018-03-29T15:19:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Technically, it is the git_config_set_multivar_in_file_gently()\nfunction that we modify here (but the oneline would get too long if we\nwere that precise).\n\nThis change prepares the git_config_set machinery to allow reusing empty\nsections, by using the file-local function do_config_from_file()\ndirectly (whose signature can then be changed without any effect outside\nof config.c).\n\nAn incidental benefit is that we avoid a level of indirection, and we\nalso avoid calling flockfile()/funlockfile() when we already know that\nwe are not operating on stdin/stdout here.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 11 +++++++++--\n 1 file changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 503aef4b318..eb1e0d335fc 100644\n--- a/config.c\n+++ b/config.c\n@@ -2706,6 +2706,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\tstruct stat st;\n \t\tsize_t copy_begin, copy_end;\n \t\tint i, new_line = 0;\n+\t\tFILE *f;\n \n \t\tif (value_regex == NULL)\n \t\t\tstore.value_regex = NULL;\n@@ -2739,7 +2740,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t * As a side effect, we make sure to transform only a valid\n \t\t * existing config file.\n \t\t */\n-\t\tif (git_config_from_file(store_aux, config_filename, NULL)) {\n+\t\tf = fopen_or_warn(config_filename, \"r\");\n+\t\tif (!f || do_config_from_file(store_aux, CONFIG_ORIGIN_FILE,\n+\t\t\t\t\t config_filename, config_filename,\n+\t\t\t\t\t f, NULL)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n \t\t\tif (store.value_regex != NULL &&\n \t\t\t store.value_regex != CONFIG_REGEX_NONE) {\n@@ -2747,8 +2751,11 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\tfree(store.value_regex);\n \t\t\t}\n \t\t\tret = CONFIG_INVALID_FILE;\n+\t\t\tif (f)\n+\t\t\t\tfclose(f);\n \t\t\tgoto out_free;\n-\t\t}\n+\t\t} else\n+\t\t\tfclose(f);\n \n \t\tif (store.value_regex != NULL &&\n \t\t store.value_regex != CONFIG_REGEX_NONE) {\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343343","messageId":"29868fec2606f5f6aa1c5de2928580f2ca24e6e2.1522336130.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH 9/9] git_config_set: reuse empty sections","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-29T15:19:09Z","receivedAt":"2018-03-29T15:19:27Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"It can happen quite easily that the last setting in a config section is\nremoved, and to avoid confusion when there are comments in the config\nabout that section, we keep a lone section header, i.e. an empty\nsection.\n\nThe code to add new entries in the config tries to be cute by reusing\nthe parsing code that is used to retrieve config settings, but that\nposes the problem that the latter use case does *not* care about empty\nsections, therefore even the former user case won't see them.\n\nFix this by introducing a mode where the parser reports also empty\nsections (with a trailing '.' as tell-tale), and then using that when\nadding new config entries.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 32 +++++++++++++++++++++++---------\n t/t1300-config.sh | 2 +-\n 2 files changed, 24 insertions(+), 10 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex eb1e0d335fc..b04c40f76bc 100644\n--- a/config.c\n+++ b/config.c\n@@ -653,13 +653,15 @@ static int get_base_var(struct strbuf *name)\n \t}\n }\n \n-static int git_parse_source(config_fn_t fn, void *data)\n+static int git_parse_source(config_fn_t fn, void *data,\n+\t\t\t int include_section_headers)\n {\n \tint comment = 0;\n \tint baselen = 0;\n \tstruct strbuf *var = &cf->var;\n \tint error_return = 0;\n \tchar *error_msg = NULL;\n+\tint saw_section_header = 0;\n \n \t/* U+FEFF Byte Order Mark in UTF8 */\n \tconst char *bomptr = utf8_bom;\n@@ -685,6 +687,16 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t\tif (cf->eof)\n \t\t\t\treturn 0;\n \t\t\tcomment = 0;\n+\t\t\tif (saw_section_header) {\n+\t\t\t\tif (include_section_headers) {\n+\t\t\t\t\tcf->linenr--;\n+\t\t\t\t\terror_return = fn(var->buf, NULL, data);\n+\t\t\t\t\tif (error_return < 0)\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\tcf->linenr++;\n+\t\t\t\t}\n+\t\t\t\tsaw_section_header = 0;\n+\t\t\t}\n \t\t\tcontinue;\n \t\t}\n \t\tif (comment || isspace(c))\n@@ -700,6 +712,7 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t\t\tbreak;\n \t\t\tstrbuf_addch(var, '.');\n \t\t\tbaselen = var->len;\n+\t\t\tsaw_section_header = 1;\n \t\t\tcontinue;\n \t\t}\n \t\tif (!isalpha(c))\n@@ -1398,7 +1411,8 @@ int git_default_config(const char *var, const char *value, void *dummy)\n * fgetc, ungetc, ftell of top need to be initialized before calling\n * this function.\n */\n-static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n+static int do_config_from(struct config_source *top, config_fn_t fn, void *data,\n+\t\t\t int include_section_headers)\n {\n \tint ret;\n \n@@ -1410,7 +1424,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n \tstrbuf_init(&top->var, 1024);\n \tcf = top;\n \n-\tret = git_parse_source(fn, data);\n+\tret = git_parse_source(fn, data, include_section_headers);\n \n \t/* pop config-file parsing state stack */\n \tstrbuf_release(&top->value);\n@@ -1423,7 +1437,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n static int do_config_from_file(config_fn_t fn,\n \t\tconst enum config_origin_type origin_type,\n \t\tconst char *name, const char *path, FILE *f,\n-\t\tvoid *data)\n+\t\tvoid *data, int include_section_headers)\n {\n \tstruct config_source top;\n \n@@ -1436,12 +1450,12 @@ static int do_config_from_file(config_fn_t fn,\n \ttop.do_ungetc = config_file_ungetc;\n \ttop.do_ftell = config_file_ftell;\n \n-\treturn do_config_from(&top, fn, data);\n+\treturn do_config_from(&top, fn, data, include_section_headers);\n }\n \n static int git_config_from_stdin(config_fn_t fn, void *data)\n {\n-\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data);\n+\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data, 0);\n }\n \n int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n@@ -1452,7 +1466,7 @@ int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n \tf = fopen_or_warn(filename, \"r\");\n \tif (f) {\n \t\tflockfile(f);\n-\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data);\n+\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data, 0);\n \t\tfunlockfile(f);\n \t\tfclose(f);\n \t}\n@@ -1475,7 +1489,7 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \ttop.do_ungetc = config_buf_ungetc;\n \ttop.do_ftell = config_buf_ftell;\n \n-\treturn do_config_from(&top, fn, data);\n+\treturn do_config_from(&top, fn, data, 0);\n }\n \n int git_config_from_blob_oid(config_fn_t fn,\n@@ -2743,7 +2757,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\tf = fopen_or_warn(config_filename, \"r\");\n \t\tif (!f || do_config_from_file(store_aux, CONFIG_ORIGIN_FILE,\n \t\t\t\t\t config_filename, config_filename,\n-\t\t\t\t\t f, NULL)) {\n+\t\t\t\t\t f, NULL, 1)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n \t\t\tif (store.value_regex != NULL &&\n \t\t\t store.value_regex != CONFIG_REGEX_NONE) {\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex ecbcc9cf3d0..867397ae930 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1463,7 +1463,7 @@ test_expect_success '--unset-all removes section if empty & uncommented' '\n \ttest_line_count = 0 .git/config\n '\n \n-test_expect_failure 'adding a key into an empty section reuses header' '\n+test_expect_success 'adding a key into an empty section reuses header' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n \tEOF\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n"},{"id":"343351","messageId":"CAGZ79ka41uD7A_64kz5oyrhCBci+oXmDGEVc6mHWyQd7d9avAQ@mail.gmail.com","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-03-29T17:58:43Z","receivedAt":"2018-03-29T17:58:49Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Mar 29, 2018 at 8:18 AM, Johannes Schindelin\n wrote:\n\n> So what is the argument against this extra care to detect comments? Well, if\n> you have something like this:\n>\n> [section]\n> ; Here we comment about the variable called snarf\n> snarf = froop\n>\n> and we run `git config --unset section.snarf`, we end up with this config:\n>\n> [section]\n> ; Here we comment about the variable called snarf\n>\n> which obviously does not make sense. However, that is already established\n> behavior for quite a few years, and I do not even try to think of a way how\n> this could be solved.\n\nBy commenting out the key/value pair instead of deleting it.\nIt's called --unset, not --delete ;)\n\nNow onto reviewing the patches.\n\nStefan\n"},{"id":"343355","messageId":"CAGZ79kapTWGsYznt7rr0QTNX+uH85TPY8AOA1jtDJ6_q8edX1Q@mail.gmail.com","threadId":"48185","inReplyTo":"899ea23951627426ccd0aac79f824af386c5590c.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-03-29T18:15:33Z","receivedAt":"2018-03-29T18:15:39Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Mar 29, 2018 at 8:18 AM, Johannes Schindelin\n wrote:\n> Currently, we are slightly overzealous When removing an entry from a\n> config file of this form:\n>\n> [abc]a\n> [xyz]\n> key = value\n>\n> When calling `git config --unset abc.a` on this file, it leaves this\n> (invalid) config behind:\n>\n> [\n> [xyz]\n> key = value\n>\n> The reason is that we try to search for the beginning of the line (or\n> for the end of the preceding section header on the same line) that\n> defines abc.a, but as an optimization, we subtract 2 from the offset\n> pointing just after the definition before we call\n> find_beginning_of_line(). That function, however, *also* performs that\n> optimization and promptly fails to find the section header correctly.\n\nThis commit message would be more convincing if we had it in test form.\n\n [abc]a\n\nis not written by Git, but would be written from an outside tool or person\nand we barely cope with it?\n\nThanks,\nStefan\n\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> config.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/config.c b/config.c\n> index b0c20e6cb8a..5cc049aaef0 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -2632,7 +2632,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n> } else\n> copy_end = find_beginning_of_line(\n> contents, contents_sz,\n> - store.offset[i]-2, &new_line);\n> + store.offset[i], &new_line);\n>\n> if (copy_end > 0 && contents[copy_end-1] != '\\n')\n> new_line = 1;\n> --\n> 2.16.2.windows.1.26.g2cc3565eb4b\n>\n>\n"},{"id":"343360","messageId":"20180329193932.GA2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T19:39:33Z","receivedAt":"2018-03-29T19:39:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:18:30PM +0200, Johannes Schindelin wrote:\n\n> Little did I know that this would turn not only into a full patch to fix this\n> issue, but into a full-blown series of nine patches.\n\nIt's amazing how often that happens. :)\n\n> The first patch is somewhat of a \"while at it\" bug fix that I first thought\n> would be a lot more critical than it actually is: It really only affects config\n> files that start with a section followed immediately (i.e. without a newline)\n> by a one-letter boolean setting (i.e. without a `= ` part). So while it\n> is a real bug fix, I doubt anybody ever got bitten by it.\n\nThat makes me wonder if somebody could craft a malicious config to do\nsomething bad. But I don't think so. Config is trusted already, and it\nlooks like this bug is both hard to trigger and doesn't result in any\nkind of memory funniness, just a bogus output.\n\n> Now, to the really important part: why does this patch series not conflict with\n> my very early statements that we cannot simply remove empty sections because we\n> may end up with stale comments?\n> \n> Well, the patch in question takes pains to determine *iff* there are any\n> comments surrounding, or included in, the section. If any are found: previous\n> behavior. Under the assumption that the user edited the file, we keep it as\n> intact as possible (see below for some argument against this). If no comments\n> are found, and let's face it, this is probably *the* common case, as few people\n> edit their config files by hand these days (neither should they because it is\n> too easy to end up with an unparseable one), the now-empty section *is*\n> removed.\n\nI'm not against people editing their config files by hand. But I think\nwhat you propose here makes a lot of sense, because it works as long as\nyou don't intermingle hand- and auto-editing in the same section (and it\neven works if you do intermingle, as long as you don't use comments,\nwhich are probably even more rare).\n\nSo it seems like quite a sensible compromise, and I think should make\nmost people happy.\n\n-Peff\n"},{"id":"343361","messageId":"20180329194159.GB2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"CAGZ79kapTWGsYznt7rr0QTNX+uH85TPY8AOA1jtDJ6_q8edX1Q@mail.gmail.com","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T19:41:59Z","receivedAt":"2018-03-29T19:42:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 11:15:33AM -0700, Stefan Beller wrote:\n\n> > When calling `git config --unset abc.a` on this file, it leaves this\n> > (invalid) config behind:\n> >\n> > [\n> > [xyz]\n> > key = value\n> >\n> > The reason is that we try to search for the beginning of the line (or\n> > for the end of the preceding section header on the same line) that\n> > defines abc.a, but as an optimization, we subtract 2 from the offset\n> > pointing just after the definition before we call\n> > find_beginning_of_line(). That function, however, *also* performs that\n> > optimization and promptly fails to find the section header correctly.\n> \n> This commit message would be more convincing if we had it in test form.\n\nI agree a test might be nice. But I don't find the commit message\nunconvincing at all. It explains pretty clearly why the bug occurs, and\nyou can verify it by looking at find_beginning_of_line.\n\n> [abc]a\n> \n> is not written by Git, but would be written from an outside tool or person\n> and we barely cope with it?\n\nYes, I don't think git would ever write onto the same line. But clearly\nwe should handle anything that's syntactically valid.\n\n-Peff\n"},{"id":"343362","messageId":"20180329194221.GC2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"6023d70cff199424893d116339f860952c6a6b4b.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 2/9] t1300: rename it to reflect that `repo-config` was deprecated","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T19:42:21Z","receivedAt":"2018-03-29T19:42:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:18:40PM +0200, Johannes Schindelin wrote:\n\n> Signed-off-by: Johannes Schindelin \n> ---\n> t/{t1300-repo-config.sh => t1300-config.sh} | 0\n> 1 file changed, 0 insertions(+), 0 deletions(-)\n> rename t/{t1300-repo-config.sh => t1300-config.sh} (100%)\n\nThis has only been bugging me for oh, about 10 years. Thanks.\n\n-Peff\n"},{"id":"343363","messageId":"20180329194313.GD2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"385674162d2853f6aea153d9d30482034d9fa3df.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 3/9] t1300: avoid relying on a bug","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T19:43:14Z","receivedAt":"2018-03-29T19:43:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:18:45PM +0200, Johannes Schindelin wrote:\n\n> The test case 'unset with cont. lines' relied on a bug that is about to\n> be fixed: it tests *explicitly* that removing the last entry from a\n> config section leaves an *empty* section behind.\n> \n> Let's fix this test case not to rely on that behavior, simply by\n> preventing the section from becoming empty.\n\nSeems like a good solution. I don't think we care in particular about\ntesting a multi-line value at the end of the file.\n\n-Peff\n"},{"id":"343364","messageId":"20180329195218.GE2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"5801f7529465b7dbdfbe3c105e2a8212c253ba0d.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 4/9] t1300: remove unreasonable expectation from TODO","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T19:52:18Z","receivedAt":"2018-03-29T19:52:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:18:50PM +0200, Johannes Schindelin wrote:\n\n> In https://public-inbox.org/git/7vvc8alzat.fsf@alter.siamese.dyndns.org/\n> a reasonable patch was made quite a bit less so by changing a test case\n> demonstrating a bug to a test case that demonstrates that we ask for too\n> much: the test case 'unsetting the last key in a section removes header'\n> now expects a future bug fix to be able to determine whether a free-form\n> comment above a section header refers to said section or not.\n> \n> Rather than shooting for the stars (and not even getting off the\n> ground), let's start shooting for something obtainable and be reasonably\n> confident that we *can* get it.\n\nAs I said before, I'm fine with turning this test into something more\nrealistic.\n\nAn obvious question is whether we should preserve the original\nunrealistic parts by splitting it: the realistic parts into one\nexpect_failure (that we'd switch to expect_success by the end of this\nseries), and then an unrealistic one to serve as a documentation of the\nideal, with a comment explaining why it's unrealistic.\n\nI doubt the \"unrealistic\" half would be serving much purpose though, so\nI'm OK to see it get eliminated here.\n\n-Peff\n"},{"id":"343365","messageId":"20180329195445.GF2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"484133f267ec2ebb6d6c27ccb40410579c72032a.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 5/9] t1300: `--unset-all` can leave an empty section behind (bug)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T19:54:46Z","receivedAt":"2018-03-29T19:54:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:18:53PM +0200, Johannes Schindelin wrote:\n\n> We already have a test demonstrating that removing the last entry from a\n> config section fails to remove the section header of the now-empty\n> section.\n> \n> The same can happen, of course, if we remove the last entries in one fell\n> swoop. This is *also* a bug, and should be fixed at the same time.\n\nYep, makes sense, and the diff is obviously correct.\n\n-Peff\n"},{"id":"343369","messageId":"xmqq8taa626o.fsf@gitster-ct.c.googlers.com","threadId":"48185","inReplyTo":"20180329195218.GE2939@sigill.intra.peff.net","subject":"Re: [PATCH 4/9] t1300: remove unreasonable expectation from TODO","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-29T20:45:35Z","receivedAt":"2018-03-29T20:45:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> An obvious question is whether we should preserve the original\n> unrealistic parts by splitting it: the realistic parts into one\n> expect_failure (that we'd switch to expect_success by the end of this\n> series), and then an unrealistic one to serve as a documentation of the\n> ideal, with a comment explaining why it's unrealistic.\n>\n> I doubt the \"unrealistic\" half would be serving much purpose though, so\n> I'm OK to see it get eliminated here.\n\nLikewise. The series looks good so far.\n"},{"id":"343371","messageId":"20180329213229.GG2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"c246c8bc2fb1cd6fe6307463d299cf56fbe4dc5b.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 7/9] git config --unset: remove empty sections (in normal situations)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T21:32:30Z","receivedAt":"2018-03-29T21:32:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:19:00PM +0200, Johannes Schindelin wrote:\n\n> Let's generalize this observation to this conservative strategy: if we\n> are removing the last entry from a section, and there are no comments\n> inside that section nor surrounding it, then remove the entire section.\n> Otherwise behave as before: leave the now-empty section (including those\n> comments, even the one about the now-deleted entry).\n\nYep, as I said earlier, this makes a ton of sense to me.\n\n> +/*\n> + * This function determines whether the offset is in a line that starts with a\n> + * comment character.\n> + *\n> + * Note: it does *not* report when a regular line (section header, config\n> + * setting) *ends* in a comment.\n> + */\n> +static int is_in_comment_line(const char *contents, size_t offset)\n> +{\n> +\tint comment = 0;\n> +\n> +\twhile (offset > 0)\n> +\t\tswitch (contents[--offset]) {\n> +\t\tcase ';':\n> +\t\tcase '#':\n> +\t\t\tcomment = 1;\n> +\t\t\tbreak;\n> +\t\tcase '\\n':\n> +\t\t\tbreak;\n> +\t\tcase ' ':\n> +\t\tcase '\\t':\n> +\t\t\tcontinue;\n> +\t\tdefault:\n> +\t\t\tcomment = 0;\n> +\t\t}\n> +\n> +\treturn comment;\n> +}\n\nThis doesn't pay any attention to quoting, so I wondered if it would get\nfooled by a line like:\n\n key = \"this content has a # comment in it\"\n\nor even:\n\n [section \"this section has a # comment in it\"]\n\nbut those don't count because the line doesn't _start_ with the comment\ncharacter. Could we design one that does? This isn't valid:\n\n [section]\n key = multiline \\\n # with comment\n\nBut I think this is:\n\n [section]\n key = \"multiline \\\n # with comment\"\n\nSo let's see if we can fool it:\n\n-- >8 --\n\ncat >file <<-\\EOF\n[one]\nkey = \"multiline \\\n # with comment\"\n[two]\nkey = true\nEOF\n\n# should produce \"multiline # with comment\"\n./git config --file=file one.key\n\n# this should ideally remove the section\n./git config --file=file --unset two.key\ncat file\n\n-- 8< --\n\nThat seems to work as expected. I'm not 100% sure why, though, since I\nthought we'd hit the \"seen_section && !is_in_comment_line\" bit of the\nlook_before loop. Running it through gdb, I'm not convinced that\nis_in_comment_line is working correctly, though. Shouldn't it stop when\nit sees the newline, and return \"comment\"? There's a \"break\" there, but\nit doesn't break out of the loop due to the switch statement.\n\nSo we'll _always_ walk back to the beginning of file. So I suspect your\ntest passes because it does:\n\n # this is the start of the file\n [section]\n key = true\n\nbut:\n\n [anotherSection]\n key = true\n # a comment not at the start\n [section]\n key = true\n\ndoes the wrong thing, and removes [section]. If we fix that bug like\nthis:\n\ndiff --git a/config.c b/config.c\nindex b04c40f76b..3b2c7e9387 100644\n--- a/config.c\n+++ b/config.c\n@@ -2461,7 +2461,7 @@ static int is_in_comment_line(const char *contents, size_t offset)\n \t\t\tcomment = 1;\n \t\t\tbreak;\n \t\tcase '\\n':\n-\t\t\tbreak;\n+\t\t\treturn comment;\n \t\tcase ' ':\n \t\tcase '\\t':\n \t\t\tcontinue;\n\nthen it keeps \"[section]\" correctly. But now if we go back to our funny\nmultiline example, it does the wrong thing (it keeps [two], even though\nthat's not _really_ a comment).\n\nTo be honest, I could live with that as an open bug. It's a pretty\nridiculous situation, and the worst case is that we err on the side of\ncaution and don't remove the section. And I think it would be hard to\nfix. We could look for the continuation backslash when we find the\nnewline, but that gets fooled by:\n\n # a comment \\\n # with a pointless backslash\n\nYou can't just notice the quote and say \"oh, I'm in a quoted section\"\nbecause that gets fooled by:\n\n # a pointless \"quote\n\nTo know whether that quote is valid or not, you have to find the other\nquote. But doing that backwards is hard (if not impossible).\n\n> +static void maybe_remove_section(const char *contents, size_t size,\n> +\t\t\t\t const char *section_name,\n> +\t\t\t\t size_t section_name_len,\n> +\t\t\t\t size_t *begin, int *i_ptr, int *new_line)\n> +{\n> +\tsize_t begin2, end2;\n> +\tint seen_section = 0, dummy, i = *i_ptr;\n> +\n> +\t/*\n> +\t * First, make sure that this is the last key in the section, and that\n> +\t * there are no comments that are possibly about the current section.\n> +\t */\n> +next_entry:\n> +\tfor (end2 = store.offset[i]; end2 < size; end2++) {\n> +\t\tswitch (contents[end2]) {\n> +\t\tcase ' ':\n> +\t\tcase '\\t':\n> +\t\tcase '\\n':\n> +\t\t\tcontinue;\n> +\t\tcase '\\r':\n> +\t\t\tif (++end2 < size && contents[end2] == '\\n')\n> +\t\t\t\tcontinue;\n> +\t\t\tbreak;\n> +\t\tcase '[':\n> +\t\t\t/* If the section name is repeated, continue */\n> +\t\t\tif (end2 + 1 + section_name_len < size &&\n> +\t\t\t contents[end2 + section_name_len] == ']' &&\n> +\t\t\t !memcmp(contents + end2 + 1, section_name,\n> +\t\t\t\t section_name_len)) {\n> +\t\t\t\tend2 += section_name_len;\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t\tgoto look_before;\n> +\t\tcase ';':\n> +\t\tcase '#':\n> +\t\t\t/* There is a comment, cannot remove this section */\n> +\t\t\treturn;\n> +\t\tdefault:\n> +\t\t\t/* There are other keys in that section */\n> +\t\t\tbreak;\n> +\t\t}\n\nOK, this all makes sense. We're scanning forward to find the next '[',\nwithout finding any keys or comments. We don't have to worry about\nquoting because we'd quit as soon as we see a key anyway. I like the\nspecial-case for finding our same section name, since that would help\nclean up cruft from existing versions of Git.\n\nIt looks like there may be an off-by-one, though. Should it be checking:\n\n contents[end2 + 1 + section_name_len] == ']'\n\nto skip over the opening '['? In a simple example:\n\n [foo]\n bar = true\n [foo]\n\nwe don't seem to remove the second section header. It works with the\npatch below:\n\ndiff --git a/config.c b/config.c\nindex b04c40f76b..48dcb52840 100644\n--- a/config.c\n+++ b/config.c\n@@ -2508,10 +2508,10 @@ static void maybe_remove_section(const char *contents, size_t size,\n \t\tcase '[':\n \t\t\t/* If the section name is repeated, continue */\n \t\t\tif (end2 + 1 + section_name_len < size &&\n-\t\t\t contents[end2 + section_name_len] == ']' &&\n+\t\t\t contents[end2 + 1 + section_name_len] == ']' &&\n \t\t\t !memcmp(contents + end2 + 1, section_name,\n \t\t\t\t section_name_len)) {\n-\t\t\t\tend2 += section_name_len;\n+\t\t\t\tend2 += section_name_len + 1;\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tgoto look_before;\n\nUnfortunately I think this whole thing breaks down with subsections. If\nwe try this:\n\n [foo \"subsection\"]\n bar = true\n [foo \"subsection\"]\n\nthen the section_name variable contains \"foo.subsection\", which we can't\ntextually match. And we end up failing to remove either section (the\nlatter one because of this loop, and the former because of the same\nproblem in the look_before loop).\n\n> +\t\t/*\n> +\t\t * Uh oh... we found something else in this section. But do\n> +\t\t * we want to remove this, too?\n> +\t\t */\n> +\t\tif (++i >= store.seen)\n> +\t\t\treturn;\n> +\n> +\t\tbegin2 = find_beginning_of_line(contents, size, store.offset[i],\n> +\t\t\t\t\t\t&dummy);\n> +\t\tif (begin2 > end2)\n> +\t\t\treturn;\n> +\n> +\t\t/* Looks like we want to remove the next one, too... */\n> +\t\tgoto next_entry;\n> +\t}\n\nOK, makes sense.\n\n> +look_before:\n> +\t/*\n> +\t * Now, ensure that this is the first key, and that there are no\n> +\t * comments before the entry nor before the section header.\n> +\t */\n> +\tfor (begin2 = *begin; begin2 > 0; )\n> +\t\tswitch (contents[begin2 - 1]) {\n> +\t\tcase ' ':\n> +\t\tcase '\\t':\n> +\t\t\tbegin2--;\n> +\t\t\tcontinue;\n> +\t\tcase '\\n':\n> +\t\t\tif (--begin2 > 0 && contents[begin2 - 1] == '\\r')\n> +\t\t\t\tbegin2--;\n> +\t\t\tcontinue;\n> +\t\tcase ']':\n> +\t\t\tif (begin2 > section_name_len + 1 &&\n> +\t\t\t contents[begin2 - section_name_len - 2] == '[' &&\n> +\t\t\t !memcmp(contents + begin2 - section_name_len - 1,\n> +\t\t\t\t section_name, section_name_len)) {\n> +\t\t\t\tbegin2 -= section_name_len + 2;\n> +\t\t\t\tseen_section = 1;\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n\nOK, this is the backwards mirror image of the earlier part. Which makes\nsense. And this handles the reverse case for the doubled section name:\n\n [foo]\n [foo]\n bar = true\n\nbecause we'd hit this section-name check twice, and just set\n\"seen_section = 1\" both times. So that works (modulo the subsection\nparsing thing).\n\nAs far as quoting goes, now we're coming from the back of each line now.\nAnd I don't think we strictly require double-quotes around string\nvalues. So imagine this:\n\n [one]\n foo = this has [brackets]\n bar = this does not\n\nWhen deleting one.bar, we'd erroneously think that closing bracket is\nthe prior section header. I _think_ it behaves correctly, though,\nbecause we then say \"well, delete everything back to that bracket\ncharacter\". Which happens to be the correct thing to do anyway.\n\nBut let's get more devious. What about this:\n\n [one]\n foo = fake section [one]\n bar = whatever\n\nIf I unset foo.bar with your patch, I end up with the truncated:\n\n [one]\n foo = fake sectio\n\nYikes. This is obviously a ridiculous example, but the failure case is\npretty nasty.\n\nAgain, the tricky thing here is that we're parsing backwards. We don't\nknow what's syntactically relevant and what isn't.\n\n> +\n> +\t\t\t/*\n> +\t\t\t * It looks like a section header, but it could be a\n> +\t\t\t * comment instead...\n> +\t\t\t */\n> +\t\t\tif (is_in_comment_line(contents, begin2))\n> +\t\t\t\treturn;\n\nThis would get fooled if we allowed line continuation in subsection\nnames, like:\n\n [one \"subsection\\\n # with newline\"]\n key = true\n\nbut it looks like our parser doesn't allow that (aside from it being\nslightly insane, of course). Good.\n\n> +\t\t\t/*\n> +\t\t\t * We encountered the previous section header: This\n> +\t\t\t * really was the only entry, so remove the entire\n> +\t\t\t * section.\n> +\t\t\t */\n> +\t\t\tif (contents[begin2] != '\\n') {\n> +\t\t\t\tbegin2--;\n> +\t\t\t\t*new_line = 1;\n> +\t\t\t}\n> +\n> +\t\t\tstore.offset[i] = end2;\n> +\t\t\t*begin = begin2;\n> +\t\t\t*i_ptr = i;\n> +\t\t\treturn;\n\nOK, makes sense.\n\n> +\t\tdefault:\n> +\t\t\t/*\n> +\t\t\t * Any other character means it is either a comment or\n> +\t\t\t * a config setting; if it is a comment, we do not want\n> +\t\t\t * to remove this section. If it is a config setting,\n> +\t\t\t * we only want to remove this section if this is\n> +\t\t\t * already the next section.\n> +\t\t\t */\n> +\t\t\tif (seen_section &&\n> +\t\t\t !is_in_comment_line(contents, begin2)) {\n> +\t\t\t\tif (contents[begin2] != '\\n') {\n> +\t\t\t\t\tbegin2--;\n> +\t\t\t\t\t*new_line = 1;\n> +\t\t\t\t}\n> +\n> +\t\t\t\tstore.offset[i] = end2;\n> +\t\t\t\t*begin = begin2;\n> +\t\t\t\t*i_ptr = i;\n> +\t\t\t}\n> +\t\t\treturn;\n> +\t\t}\n\nHere's where we get fooled by is_in_comment_line() that I showed at the\nbeginning. We don't have to worry about other quoting, because any key\n(quoted or not) would cause us to abort, since it's in the section.\n\n> +\t/* This section extends to the beginning of the file. */\n> +\tstore.offset[i] = end2;\n> +\t*begin = begin2;\n> +\t*i_ptr = i;\n> +}\n\nRight, makes sense.\n\n\nOk, phew. That was a tough read. So here's what I see:\n\n 1. Minor bug in is_in_comment_line(), patch above.\n\n 2. Minor bug in matching section names, patch above.\n\n 3. Matching subsection names doesn't work. I think this should be\n fixable with a helper function which can match '[one \"two\"]' when\n given \"one.two\".\n\n 4. Backwards parsing causes is_in_comment_line to trigger more than it\n should. I can live with that because the trigger is arcane, and the\n error behavior is pretty harmless.\n\n 5. Backwards parsing can find a bogus section. Also arcane, but the\n error behavior is pretty scary.\n\n(4) and (5) are the ones that I don't see a way to fix, given the\ncurrent way in which we do the config-writing (i.e., running it through\nthe regular read-parser and then trying to \"patch up\" the found\nlocations). I think that's also what's contributing to the code being\nhard to read, since you end up doing quite a lot of manual re-parsing.\n\nI think the sane way to do this would be to parse the whole thing into\na tree (that includes things like comments and whitespace), and then we\ncould much more easily manipulate that tree, without dealing with the\nparsing (forwards _and_ backwards). But that's a pretty big change from\nthe current code.\n\nIt also potentially means duplicating the parsing logic, unless we teach\nthe regular reader to do the tree-parse, and then pick out the config\nfrom that. That's likely much slower than the existing parser (since\nwe'd allocate a bunch of tree nodes instead of just dumping strings to\nthe callbacks). But these days we cache the parsed config anyway, so I'm\nnot sure if a slight slowdown would actually matter that much.\n\nI guess the holy grail would be a parser which reports _all_ syntactic\nevents (section names, keys, comments, whitespace, etc) as a stream\nwithout storing anything. And then the normal reader could just discard\nthe non-key events, and the writer here could build the tree from those\nevents.\n\n-Peff\n"},{"id":"343372","messageId":"20180329213823.GH2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"e3032300946eb5962878341f7796f5872c4d138d.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T21:38:23Z","receivedAt":"2018-03-29T21:38:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:19:04PM +0200, Johannes Schindelin wrote:\n\n> Technically, it is the git_config_set_multivar_in_file_gently()\n> function that we modify here (but the oneline would get too long if we\n> were that precise).\n> \n> This change prepares the git_config_set machinery to allow reusing empty\n> sections, by using the file-local function do_config_from_file()\n> directly (whose signature can then be changed without any effect outside\n> of config.c).\n> \n> An incidental benefit is that we avoid a level of indirection, and we\n> also avoid calling flockfile()/funlockfile() when we already know that\n> we are not operating on stdin/stdout here.\n\nI'm not sure I understand that last paragraph. What does flockfile() have\nto do with stdin/stdout?\n\nThe point of those calls is that we're locking the FILE handle, so that\nit's safe for the lower-level config code to run getc_unlocked(), which\nis faster.\n\nSo without those, we're calling getc_unlocked() without holding the\nlock. I think it probably works in practice because we know that we're\nsingle-threaded, but it seems a bit sketchy.\n\n-Peff\n"},{"id":"343373","messageId":"20180329215038.GI2939@sigill.intra.peff.net","threadId":"48185","inReplyTo":"29868fec2606f5f6aa1c5de2928580f2ca24e6e2.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 9/9] git_config_set: reuse empty sections","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-29T21:50:38Z","receivedAt":"2018-03-29T21:50:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 29, 2018 at 05:19:09PM +0200, Johannes Schindelin wrote:\n\n> It can happen quite easily that the last setting in a config section is\n> removed, and to avoid confusion when there are comments in the config\n> about that section, we keep a lone section header, i.e. an empty\n> section.\n> \n> The code to add new entries in the config tries to be cute by reusing\n> the parsing code that is used to retrieve config settings, but that\n> poses the problem that the latter use case does *not* care about empty\n> sections, therefore even the former user case won't see them.\n> \n> Fix this by introducing a mode where the parser reports also empty\n> sections (with a trailing '.' as tell-tale), and then using that when\n> adding new config entries.\n\nHeh, so it seems we are partway to the \"event-stream\" suggestion I made\nearlier. I agree this is the right way to approach this problem.\n\nI wondered if we allow keys to end in \".\", but it seems that we don't.\n\n> diff --git a/config.c b/config.c\n> index eb1e0d335fc..b04c40f76bc 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -653,13 +653,15 @@ static int get_base_var(struct strbuf *name)\n> \t}\n> }\n> \n> -static int git_parse_source(config_fn_t fn, void *data)\n> +static int git_parse_source(config_fn_t fn, void *data,\n> +\t\t\t int include_section_headers)\n\nWe already have a \"struct config_options\", but we do a terrible job of\npassing it around (since it only impacts the include stuff right now,\nand that all gets handled at a very outer level).\n\nRather than plumb this one int through everywhere, should we add it to\nthat struct and plumb the struct through?\n\n-Peff\n"},{"id":"343405","messageId":"nycvar.QRO.7.76.6.1803301411480.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"CAGZ79ka41uD7A_64kz5oyrhCBci+oXmDGEVc6mHWyQd7d9avAQ@mail.gmail.com","subject":"Re: [PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T12:14:19Z","receivedAt":"2018-03-30T12:14:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stefan,\n\nOn Thu, 29 Mar 2018, Stefan Beller wrote:\n\n> On Thu, Mar 29, 2018 at 8:18 AM, Johannes Schindelin\n> wrote:\n> \n> > So what is the argument against this extra care to detect comments? Well, if\n> > you have something like this:\n> >\n> > [section]\n> > ; Here we comment about the variable called snarf\n> > snarf = froop\n> >\n> > and we run `git config --unset section.snarf`, we end up with this config:\n> >\n> > [section]\n> > ; Here we comment about the variable called snarf\n> >\n> > which obviously does not make sense. However, that is already established\n> > behavior for quite a few years, and I do not even try to think of a way how\n> > this could be solved.\n> \n> By commenting out the key/value pair instead of deleting it.\n> It's called --unset, not --delete ;)\n\nThat would open the door to new bug reports when a user starts with this\nconcocted config:\n\n\t[section]\n\t\t# This is a comment about the `key` setting\n\t\tkey = value\n\nand then does this:\n\n\tgit config --unset section.key\n\tgit config section.key value\n\tgit config --unset section.key\n\tgit config section.key value\n\tgit config --unset section.key\n\tgit config section.key value\n\nand then ends up with a config like this:\n\n\t[section]\n\t\t# This is a comment about the `key` setting\n\t\t;key = value\n\t\t;key = value\n\t\t;key = value\n\t\tkey = value\n\nAnd note that the comment might be about `value` instead, so reusing a\ncommented-out `key` setting won't fly, either.\n\nI *did* give this problem a couple of minutes of thought before writing my\nassessment that is quoted above ;-)\n\nCiao,\nDscho\n"},{"id":"343406","messageId":"nycvar.QRO.7.76.6.1803301415240.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329194159.GB2939@sigill.intra.peff.net","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T12:32:47Z","receivedAt":"2018-03-30T12:33:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 11:15:33AM -0700, Stefan Beller wrote:\n> \n> > > When calling `git config --unset abc.a` on this file, it leaves this\n> > > (invalid) config behind:\n> > >\n> > > [\n> > > [xyz]\n> > > key = value\n> > >\n> > > The reason is that we try to search for the beginning of the line (or\n> > > for the end of the preceding section header on the same line) that\n> > > defines abc.a, but as an optimization, we subtract 2 from the offset\n> > > pointing just after the definition before we call\n> > > find_beginning_of_line(). That function, however, *also* performs that\n> > > optimization and promptly fails to find the section header correctly.\n> > \n> > This commit message would be more convincing if we had it in test form.\n> \n> I agree a test might be nice. But I don't find the commit message\n> unconvincing at all. It explains pretty clearly why the bug occurs, and\n> you can verify it by looking at find_beginning_of_line.\n> \n> > [abc]a\n> > \n> > is not written by Git, but would be written from an outside tool or person\n> > and we barely cope with it?\n> \n> Yes, I don't think git would ever write onto the same line. But clearly\n> we should handle anything that's syntactically valid.\n\nI was tempted to add the test case, because it is easy to test it.\n\nBut I then decided *not* to add it. Why? Testing is a balance between \"can\ndo\" and \"need to do\".\n\nCan you imagine that I did *not* run the entire test suite before\nsubmitting this patch series, because it takes an incredible *90 minutes*\nto run *on a fast Windows machine*?\n\nSeriously, this is hurting me. I do not complain about this due to some\nmental illness forcing me to do it. I complain about this so often\n*because it slows me down*, you gentle people. And you don't seem to care,\nat least the test suite gets noticably worse by the month. I frankly do\nnot know what to do about this, as you keep adding and adding and it gets\nless and less feasible for me to run the full test suite. I seem to be\ntotally unable to get through to you with the message that this is a real\nproblem with a real need to get fixed.\n\nSo with this in mind, I do not want to add a test case for a concocted\nexample that won't affect anybody except users who *want* to trigger this\nbug.\n\nI hope you agree,\nDscho\n\nP.S.: Of course I ran the entire test suite. Not on Windows, but in a\nLinux VM, because Linux is what Git is fine-tuned for, most obviously so.\nAn alien digging up ancient Earth history in the far future might be\ntempted to assume that Git was developed to develop Linux which was\ndeveloped to develop Git, and then ask herself why humans bothered at all.\n\nI actually ran the entire test suite on Linux on every single patch, via\n`git rebase -x \"make -j15 DEVELOPER=1 test\" @{u}`, as I usually do before\nsubmitting a patch series.\n\nAnd it *did* find an obscure bug in an earlier iteration, where\nt5512-ls-remote.sh demonstrated that looking at only one entry at a time\nis not enough: `git config --unset-all uploadpack.hiderefs` *also* needs\nto remove the now-empty section, because we might end up with the empty\nsections in the wrong order, and the order of [transfer] and [uploadpack]\n*matters* if the transfer.hiderefs setting is negative and the\nuploadpack.hiderefs setting is positive, as is the case in 'overrides work\nbetween mixed transfer/upload-pack hideRefs'. (Side-note: this looks like\na pretty obvious design bug to me, as there is *no tooling* to switch\naround the order of these settings. Even worse: if somebody gets\ninstructions to add those settings, and there is already a [transfer]\nsection in the config: you're out of luck! You will have to *know* that\nthe order matters, *and add a second [transfer] section manually*!)\n"},{"id":"343407","messageId":"nycvar.QRO.7.76.6.1803301433020.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329193932.GA2939@sigill.intra.peff.net","subject":"Re: [PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T12:35:18Z","receivedAt":"2018-03-30T12:35:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:18:30PM +0200, Johannes Schindelin wrote:\n> \n> > The first patch is somewhat of a \"while at it\" bug fix that I first\n> > thought would be a lot more critical than it actually is: It really\n> > only affects config files that start with a section followed\n> > immediately (i.e. without a newline) by a one-letter boolean setting\n> > (i.e. without a `= ` part). So while it is a real bug fix, I\n> > doubt anybody ever got bitten by it.\n> \n> That makes me wonder if somebody could craft a malicious config to do\n> something bad.\n\nI thought about that, and could not think of anything other than social\nengineering vectors. Even in that case, the error message is instructive\nenough that the user should be able to fix the config without consulting\nStackOverflow.\n\n> > Now, to the really important part: why does this patch series not\n> > conflict with my very early statements that we cannot simply remove\n> > empty sections because we may end up with stale comments?\n> > \n> > Well, the patch in question takes pains to determine *iff* there are\n> > any comments surrounding, or included in, the section. If any are\n> > found: previous behavior. Under the assumption that the user edited\n> > the file, we keep it as intact as possible (see below for some\n> > argument against this). If no comments are found, and let's face it,\n> > this is probably *the* common case, as few people edit their config\n> > files by hand these days (neither should they because it is too easy\n> > to end up with an unparseable one), the now-empty section *is*\n> > removed.\n> \n> I'm not against people editing their config files by hand. But I think\n> what you propose here makes a lot of sense, because it works as long as\n> you don't intermingle hand- and auto-editing in the same section (and it\n> even works if you do intermingle, as long as you don't use comments,\n> which are probably even more rare).\n> \n> So it seems like quite a sensible compromise, and I think should make\n> most people happy.\n\nThanks for confirming my line of thinking,\nDscho\n"},{"id":"343409","messageId":"nycvar.QRO.7.76.6.1803301435590.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329194221.GC2939@sigill.intra.peff.net","subject":"Re: [PATCH 2/9] t1300: rename it to reflect that `repo-config` was deprecated","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T12:37:37Z","receivedAt":"2018-03-30T12:37:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:18:40PM +0200, Johannes Schindelin wrote:\n> \n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > t/{t1300-repo-config.sh => t1300-config.sh} | 0\n> > 1 file changed, 0 insertions(+), 0 deletions(-)\n> > rename t/{t1300-repo-config.sh => t1300-config.sh} (100%)\n> \n> This has only been bugging me for oh, about 10 years.\n\nYep.\n\nWe should have done that right after moving the builtins' code to\nbuiltins/.\n\nWhich reminds me that we *still* do not have a lib/ where all the source\ncode for libgit.a lives. And then maybe standalone/ for the source code of\nthe non-builtin tools. And... this would make for a fine micro-project\nnext year, I guess. Or in ten.\n\nCiao,\nDscho\n"},{"id":"343410","messageId":"nycvar.QRO.7.76.6.1803301438120.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329194313.GD2939@sigill.intra.peff.net","subject":"Re: [PATCH 3/9] t1300: avoid relying on a bug","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T12:38:33Z","receivedAt":"2018-03-30T12:38:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:18:45PM +0200, Johannes Schindelin wrote:\n> \n> > The test case 'unset with cont. lines' relied on a bug that is about to\n> > be fixed: it tests *explicitly* that removing the last entry from a\n> > config section leaves an *empty* section behind.\n> > \n> > Let's fix this test case not to rely on that behavior, simply by\n> > preventing the section from becoming empty.\n> \n> Seems like a good solution. I don't think we care in particular about\n> testing a multi-line value at the end of the file.\n\n... and if we did, we should have documented that.\n\nCiao,\nDscho\n"},{"id":"343411","messageId":"nycvar.QRO.7.76.6.1803301439000.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329195218.GE2939@sigill.intra.peff.net","subject":"Re: [PATCH 4/9] t1300: remove unreasonable expectation from TODO","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T12:42:37Z","receivedAt":"2018-03-30T12:42:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:18:50PM +0200, Johannes Schindelin wrote:\n> \n> > In https://public-inbox.org/git/7vvc8alzat.fsf@alter.siamese.dyndns.org/\n> > a reasonable patch was made quite a bit less so by changing a test case\n> > demonstrating a bug to a test case that demonstrates that we ask for too\n> > much: the test case 'unsetting the last key in a section removes header'\n> > now expects a future bug fix to be able to determine whether a free-form\n> > comment above a section header refers to said section or not.\n> > \n> > Rather than shooting for the stars (and not even getting off the\n> > ground), let's start shooting for something obtainable and be reasonably\n> > confident that we *can* get it.\n> \n> As I said before, I'm fine with turning this test into something more\n> realistic.\n\nGood.\n\nOf course, I worked hard to come up with a patch series, i.e. I put in\nsome effort to placate anybody who would be offended by my accompanying\nrant.\n\n> An obvious question is whether we should preserve the original\n> unrealistic parts by splitting it: the realistic parts into one\n> expect_failure (that we'd switch to expect_success by the end of this\n> series), and then an unrealistic one to serve as a documentation of the\n> ideal, with a comment explaining why it's unrealistic.\n\nAs stated before, I think it would be a mistake to mark up this\nunrealistic example with `test_expect_failure`. We do, after all, suggest\noccasionally to grep for that when somebody asks what they could work on.\nAnd you do not want to set somebody like that up for failure by pointing\nthem to such a \"bug\".\n\nHowever, I did keep the example to demonstrate the expectation that\nsections with surrounding comments are kept. That was very much intended.\n\nAnd the reason I did not change the unrealistic example? So that it is\neasier to review in our patch-based review process, where I try to avoid\nhunks that might distract from the intent of the change.\n\nCiao,\nDscho\n"},{"id":"343412","messageId":"nycvar.QRO.7.76.6.1803301454400.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329213229.GG2939@sigill.intra.peff.net","subject":"Re: [PATCH 7/9] git config --unset: remove empty sections (in normal situations)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T13:00:06Z","receivedAt":"2018-03-30T13:00:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:19:00PM +0200, Johannes Schindelin wrote:\n> \n> > Let's generalize this observation to this conservative strategy: if we\n> > are removing the last entry from a section, and there are no comments\n> > inside that section nor surrounding it, then remove the entire section.\n> > Otherwise behave as before: leave the now-empty section (including those\n> > comments, even the one about the now-deleted entry).\n> \n> Yep, as I said earlier, this makes a ton of sense to me.\n> \n> [... thorough review ...]\n\nThank you for taking the time (and figuring out my off-by-ones, am I not\nthe king of those?).\n\nYour in-depth analysis of the backtracking approach also makes sense, in\nparticular the awful bug that looks very, very similar to what 1/9 fixes\nelsewhere.\n\nI'll take some time to go over your comments in detail, but there is one\nsuggestion that I think I'll want to pursue first:\n\n> I guess the holy grail would be a parser which reports _all_ syntactic\n> events (section names, keys, comments, whitespace, etc) as a stream\n> without storing anything. And then the normal reader could just discard\n> the non-key events, and the writer here could build the tree from those\n> events.\n\nI already changed the do_config_from_file()/do_config_from() code path to\nallow for handing back section headers. And I *think* that approach should\nbe easily extended to allow for an optional callback for these syntactic\nevents (and we do not need more than that, as the parsed \"tree\" really is\na list: there is nothing nested about ini files, so we really only have a\nlinear list of blocks (event type, offset range)).\n\nI'll think about this a little bit, and hopefully come back with v2 in a\nwhile that uses that approach.\n\nThank you so much for that suggestion,\nDscho\n"},{"id":"343413","messageId":"nycvar.QRO.7.76.6.1803301500560.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329213823.GH2939@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T13:02:00Z","receivedAt":"2018-03-30T13:02:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:19:04PM +0200, Johannes Schindelin wrote:\n> \n> > Technically, it is the git_config_set_multivar_in_file_gently()\n> > function that we modify here (but the oneline would get too long if we\n> > were that precise).\n> > \n> > This change prepares the git_config_set machinery to allow reusing empty\n> > sections, by using the file-local function do_config_from_file()\n> > directly (whose signature can then be changed without any effect outside\n> > of config.c).\n> > \n> > An incidental benefit is that we avoid a level of indirection, and we\n> > also avoid calling flockfile()/funlockfile() when we already know that\n> > we are not operating on stdin/stdout here.\n> \n> I'm not sure I understand that last paragraph. What does flockfile() have\n> to do with stdin/stdout?\n> \n> The point of those calls is that we're locking the FILE handle, so that\n> it's safe for the lower-level config code to run getc_unlocked(), which\n> is faster.\n> \n> So without those, we're calling getc_unlocked() without holding the\n> lock. I think it probably works in practice because we know that we're\n> single-threaded, but it seems a bit sketchy.\n\nOops. I misunderstood the purpose of flockfile(), then. I thought it was\nonly about multiple users of stdin/stdout.\n\nWill have a look whether flockfile()/funlockfile() can be moved into\ndo_config_from_file() instead.\n\nCiao,\nDscho\n"},{"id":"343414","messageId":"20180330130911.GA29568@sigill.intra.peff.net","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803301454400.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 7/9] git config --unset: remove empty sections (in normal situations)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T13:09:11Z","receivedAt":"2018-03-30T13:09:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 30, 2018 at 03:00:06PM +0200, Johannes Schindelin wrote:\n\n> > I guess the holy grail would be a parser which reports _all_ syntactic\n> > events (section names, keys, comments, whitespace, etc) as a stream\n> > without storing anything. And then the normal reader could just discard\n> > the non-key events, and the writer here could build the tree from those\n> > events.\n> \n> I already changed the do_config_from_file()/do_config_from() code path to\n> allow for handing back section headers. And I *think* that approach should\n> be easily extended to allow for an optional callback for these syntactic\n> events (and we do not need more than that, as the parsed \"tree\" really is\n> a list: there is nothing nested about ini files, so we really only have a\n> linear list of blocks (event type, offset range)).\n\nTrue. I was thinking we'd want sections with keys, whitespace, and\ncomments under them. But even that does not really make sense. As this\npatch series shows, comments do not \"belong\" to a section, and the file\nreally needs to be considered as a stream.\n\nSo yeah, if we can parse it into a sequence of events in one\nforward-pass and then manipulate that sequence, I think it should be\nsufficient (and _way_ more readable than the current code, even before\nthe bits you are trying to fix here).\n\n> I'll think about this a little bit, and hopefully come back with v2 in a\n> while that uses that approach.\n> \n> Thank you so much for that suggestion,\n\nGreat. Thanks for working on this.\n\n-Peff\n"},{"id":"343415","messageId":"20180330131456.GB29568@sigill.intra.peff.net","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803301500560.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T13:14:56Z","receivedAt":"2018-03-30T13:15:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 30, 2018 at 03:02:00PM +0200, Johannes Schindelin wrote:\n\n> > I'm not sure I understand that last paragraph. What does flockfile() have\n> > to do with stdin/stdout?\n> > \n> > The point of those calls is that we're locking the FILE handle, so that\n> > it's safe for the lower-level config code to run getc_unlocked(), which\n> > is faster.\n> > \n> > So without those, we're calling getc_unlocked() without holding the\n> > lock. I think it probably works in practice because we know that we're\n> > single-threaded, but it seems a bit sketchy.\n> \n> Oops. I misunderstood the purpose of flockfile(), then. I thought it was\n> only about multiple users of stdin/stdout.\n> \n> Will have a look whether flockfile()/funlockfile() can be moved into\n> do_config_from_file() instead.\n\nIn a sense stdin/stdout are much more susceptible to this because\nthey're global variables, and any thread may touch them. For the config\ncode, we open our own handle that we don't expose elsewhere. So probably\nit would be fine just to use the unlocked variants even without locking.\n\nBut IMHO it's good practice to always flockfile() before using the\nunlocked variants. My reading of POSIX is that it's OK to use the\nunlocked variants without holding the lock (if you know there won't be\ncontention), but if it's not hard to err on the side of safety, I'd\nprefer it.\n\n-Peff\n"},{"id":"343416","messageId":"nycvar.QRO.7.76.6.1803301515110.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180329215038.GI2939@sigill.intra.peff.net","subject":"Re: [PATCH 9/9] git_config_set: reuse empty sections","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T13:15:47Z","receivedAt":"2018-03-30T13:16:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Thu, 29 Mar 2018, Jeff King wrote:\n\n> On Thu, Mar 29, 2018 at 05:19:09PM +0200, Johannes Schindelin wrote:\n> \n> > It can happen quite easily that the last setting in a config section is\n> > removed, and to avoid confusion when there are comments in the config\n> > about that section, we keep a lone section header, i.e. an empty\n> > section.\n> > \n> > The code to add new entries in the config tries to be cute by reusing\n> > the parsing code that is used to retrieve config settings, but that\n> > poses the problem that the latter use case does *not* care about empty\n> > sections, therefore even the former user case won't see them.\n> > \n> > Fix this by introducing a mode where the parser reports also empty\n> > sections (with a trailing '.' as tell-tale), and then using that when\n> > adding new config entries.\n> \n> Heh, so it seems we are partway to the \"event-stream\" suggestion I made\n> earlier. I agree this is the right way to approach this problem.\n> \n> I wondered if we allow keys to end in \".\", but it seems that we don't.\n> \n> > diff --git a/config.c b/config.c\n> > index eb1e0d335fc..b04c40f76bc 100644\n> > --- a/config.c\n> > +++ b/config.c\n> > @@ -653,13 +653,15 @@ static int get_base_var(struct strbuf *name)\n> > \t}\n> > }\n> > \n> > -static int git_parse_source(config_fn_t fn, void *data)\n> > +static int git_parse_source(config_fn_t fn, void *data,\n> > +\t\t\t int include_section_headers)\n> \n> We already have a \"struct config_options\", but we do a terrible job of\n> passing it around (since it only impacts the include stuff right now,\n> and that all gets handled at a very outer level).\n> \n> Rather than plumb this one int through everywhere, should we add it to\n> that struct and plumb the struct through?\n\nYesss!\n\nAgain, thank you so much for this really valuable review. This is even\nbetter than what I hoped for.\n\nCiao,\nDscho\n"},{"id":"343426","messageId":"nycvar.QRO.7.76.6.1803301558360.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180330131456.GB29568@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T14:01:56Z","receivedAt":"2018-03-30T14:02:15Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 30 Mar 2018, Jeff King wrote:\n\n> On Fri, Mar 30, 2018 at 03:02:00PM +0200, Johannes Schindelin wrote:\n> \n> > > I'm not sure I understand that last paragraph. What does flockfile() have\n> > > to do with stdin/stdout?\n> > > \n> > > The point of those calls is that we're locking the FILE handle, so that\n> > > it's safe for the lower-level config code to run getc_unlocked(), which\n> > > is faster.\n> > > \n> > > So without those, we're calling getc_unlocked() without holding the\n> > > lock. I think it probably works in practice because we know that we're\n> > > single-threaded, but it seems a bit sketchy.\n> > \n> > Oops. I misunderstood the purpose of flockfile(), then. I thought it was\n> > only about multiple users of stdin/stdout.\n> > \n> > Will have a look whether flockfile()/funlockfile() can be moved into\n> > do_config_from_file() instead.\n> \n> In a sense stdin/stdout are much more susceptible to this because\n> they're global variables, and any thread may touch them. For the config\n> code, we open our own handle that we don't expose elsewhere. So probably\n> it would be fine just to use the unlocked variants even without locking.\n> \n> But IMHO it's good practice to always flockfile() before using the\n> unlocked variants. My reading of POSIX is that it's OK to use the\n> unlocked variants without holding the lock (if you know there won't be\n> contention), but if it's not hard to err on the side of safety, I'd\n> prefer it.\n\nYou know what is *really* funny?\n\n-- snip --\nstatic int git_config_from_stdin(config_fn_t fn, void *data)\n{\n return do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data, 0);\n}\n\nint git_config_from_file(config_fn_t fn, const char *filename, void *data)\n{\n int ret = -1;\n FILE *f;\n\n f = fopen_or_warn(filename, \"r\");\n if (f) {\n flockfile(f);\n ret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data, 0);\n funlockfile(f);\n fclose(f);\n }\n return ret;\n}\n-- snap --\n\nSo the _stdin variant *goes out of its way not to flockfile()*...\n\nBut I guess all this will become moot when I start handing down the config\noptions. It does mean that I have to change the signatures in header\nfiles, oh well ;-)\n\nBut then I can drop this here patch and we can stop musing about\nflockfile() ;-)\n\nCiao,\nDscho\n"},{"id":"343428","messageId":"20180330140840.GA5869@sigill.intra.peff.net","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803301558360.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T14:08:41Z","receivedAt":"2018-03-30T14:08:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 30, 2018 at 04:01:56PM +0200, Johannes Schindelin wrote:\n\n> You know what is *really* funny?\n> \n> -- snip --\n> static int git_config_from_stdin(config_fn_t fn, void *data)\n> {\n> return do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data, 0);\n> }\n> \n> int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n> {\n> int ret = -1;\n> FILE *f;\n> \n> f = fopen_or_warn(filename, \"r\");\n> if (f) {\n> flockfile(f);\n> ret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data, 0);\n> funlockfile(f);\n> fclose(f);\n> }\n> return ret;\n> }\n> -- snap --\n> \n> So the _stdin variant *goes out of its way not to flockfile()*...\n\n*facepalm* That's probably my fault, since git_config_from_stdin()\nexisted already when I did the flockfile stuff.\n\nProbably the flockfile should go into do_config_from_file(), where we\nspecify to use the unlocked variants.\n\n> But I guess all this will become moot when I start handing down the config\n> options. It does mean that I have to change the signatures in header\n> files, oh well ;-)\n> \n> But then I can drop this here patch and we can stop musing about\n> flockfile() ;-)\n\nYeah, I'll wait to see how your refactor turns out.\n\n-Peff\n"},{"id":"343429","messageId":"87in9dwsxl.fsf@evledraar.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803301415240.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-30T14:15:34Z","receivedAt":"2018-03-30T14:15:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 30 2018, Johannes Schindelin wrote:\n\n> On Thu, 29 Mar 2018, Jeff King wrote:\n>\n>> On Thu, Mar 29, 2018 at 11:15:33AM -0700, Stefan Beller wrote:\n>>\n>> > > When calling `git config --unset abc.a` on this file, it leaves this\n>> > > (invalid) config behind:\n>> > >\n>> > > [\n>> > > [xyz]\n>> > > key = value\n>> > >\n>> > > The reason is that we try to search for the beginning of the line (or\n>> > > for the end of the preceding section header on the same line) that\n>> > > defines abc.a, but as an optimization, we subtract 2 from the offset\n>> > > pointing just after the definition before we call\n>> > > find_beginning_of_line(). That function, however, *also* performs that\n>> > > optimization and promptly fails to find the section header correctly.\n>> >\n>> > This commit message would be more convincing if we had it in test form.\n>>\n>> I agree a test might be nice. But I don't find the commit message\n>> unconvincing at all. It explains pretty clearly why the bug occurs, and\n>> you can verify it by looking at find_beginning_of_line.\n>>\n>> > [abc]a\n>> >\n>> > is not written by Git, but would be written from an outside tool or person\n>> > and we barely cope with it?\n>>\n>> Yes, I don't think git would ever write onto the same line. But clearly\n>> we should handle anything that's syntactically valid.\n>\n> I was tempted to add the test case, because it is easy to test it.\n>\n> But I then decided *not* to add it. Why? Testing is a balance between \"can\n> do\" and \"need to do\".\n>\n> Can you imagine that I did *not* run the entire test suite before\n> submitting this patch series, because it takes an incredible *90 minutes*\n> to run *on a fast Windows machine*?\n\nI think if it's worth fixing it's worth testing for, a future change to\nthe config code could easily introduce a regression for this, and\nparticularly in this type of code obscure edge cases like this can point\nto bugs elsewhere.\n\nWe have the EXPENSIVE_ON_WINDOWS prerequisite already in master from an\nearlier series of mine, maybe we could use that here, or add some other\nprereq like OVERLY_EXHAUSTIVE which by default could depend on\nEXPENSIVE_ON_WINDOWS, i.e. we'd have a set of overly pedantic tests that\nwe skip on Windows by default, as there's no reason to suspect they're\nplatform-dependent, but we'd like to know if they regress.\n"},{"id":"343430","messageId":"87h8oxwsui.fsf@evledraar.gmail.com","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-30T14:17:25Z","receivedAt":"2018-03-30T14:17:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 29 2018, Johannes Schindelin wrote:\n\n> Nonetheless, I would be confortable with this patch going into v2.17.0, even at\n> this late stage. The final verdict is Junio's, of course.\n\nThanks a lot for working on this. I'm keen to stress test this, but\nwon't have time in the next few days, and in any case think that the\nparts that change functionality should wait until after 2.17 (but\ne.g. the test renaming would be fine for a cherry-pick).\n"},{"id":"343435","messageId":"xmqqefk14jlb.fsf@gitster-ct.c.googlers.com","threadId":"48185","inReplyTo":"87in9dwsxl.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-30T16:24:48Z","receivedAt":"2018-03-30T16:24:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I think if it's worth fixing it's worth testing for, a future change to\n> the config code could easily introduce a regression for this, and\n> particularly in this type of code obscure edge cases like this can point\n> to bugs elsewhere.\n\nYup. \"The port to my favourite platform is too slow, and everybody\nshould learn to live with thin test coverage\" would not be a good\nstrategy in the longer run.\n\n"},{"id":"343437","messageId":"CACsJy8DPof1jFe_cOS5+5dRHehU+_Y9y86P+5f3s_C0u85BXBw@mail.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803301415240.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-30T16:36:48Z","receivedAt":"2018-03-30T16:37:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 2:32 PM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Thu, 29 Mar 2018, Jeff King wrote:\n>\n>> On Thu, Mar 29, 2018 at 11:15:33AM -0700, Stefan Beller wrote:\n>>\n>> > > When calling `git config --unset abc.a` on this file, it leaves this\n>> > > (invalid) config behind:\n>> > >\n>> > > [\n>> > > [xyz]\n>> > > key = value\n>> > >\n>> > > The reason is that we try to search for the beginning of the line (or\n>> > > for the end of the preceding section header on the same line) that\n>> > > defines abc.a, but as an optimization, we subtract 2 from the offset\n>> > > pointing just after the definition before we call\n>> > > find_beginning_of_line(). That function, however, *also* performs that\n>> > > optimization and promptly fails to find the section header correctly.\n>> >\n>> > This commit message would be more convincing if we had it in test form.\n>>\n>> I agree a test might be nice. But I don't find the commit message\n>> unconvincing at all. It explains pretty clearly why the bug occurs, and\n>> you can verify it by looking at find_beginning_of_line.\n>>\n>> > [abc]a\n>> >\n>> > is not written by Git, but would be written from an outside tool or person\n>> > and we barely cope with it?\n>>\n>> Yes, I don't think git would ever write onto the same line. But clearly\n>> we should handle anything that's syntactically valid.\n>\n> I was tempted to add the test case, because it is easy to test it.\n>\n> But I then decided *not* to add it. Why? Testing is a balance between \"can\n> do\" and \"need to do\".\n>\n> Can you imagine that I did *not* run the entire test suite before\n> submitting this patch series, because it takes an incredible *90 minutes*\n> to run *on a fast Windows machine*?\n\nWhat's wrong with firing up a new worktree, run the test suite there\nand go back to do something else so you won't waste time just waiting\nfor test results and submit? Sure there is a mental overhead for\nswitching tasks, but at 90 minutes, I think it's worth doing.\n-- \nDuy\n"},{"id":"343453","messageId":"nycvar.QRO.7.76.6.1803302042570.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"xmqqefk14jlb.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T18:44:46Z","receivedAt":"2018-03-30T18:45:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 30 Mar 2018, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n> \n> > I think if it's worth fixing it's worth testing for, a future change to\n> > the config code could easily introduce a regression for this, and\n> > particularly in this type of code obscure edge cases like this can point\n> > to bugs elsewhere.\n> \n> Yup. \"The port to my favourite platform is too slow, and everybody\n> should learn to live with thin test coverage\" would not be a good\n> strategy in the longer run.\n\nWhat would be a *really* good strategy is: \"Oh, there is a problem! Let's\nacknowledge it and try to come up with a solution rather than a\nwork-around\".\n\nEXPENSIVE_ON_WINDOWS is a symptom. Not a solution.\n\nAnd you are actively hurting my ability to contribute, I hope you are\naware of that.\n\nCiao,\nDscho"},{"id":"343454","messageId":"87fu4hwgfa.fsf@evledraar.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803301415240.5026@qfpub.tvgsbejvaqbjf.bet","subject":"A potential approach to making tests faster on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-30T18:45:45Z","receivedAt":"2018-03-30T18:45:52Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 30 2018, Johannes Schindelin wrote [expressing frustrations\nabout Windows test suite slowness]:\n\nI've wondered for a while whether it wouldn't be a viable approach to\nmake something like an interpreter for our test suite to get around this\nproblem, i.e. much of it's very repetitive and just using a few shell\nfunctions we've defined, what if we had C equivalents of those?\n\nDuy had a WIP patch set a while ago to add C test suite support, but I\nthought what if we turn that inside-out, and instead have a shell\ninterpreter that knows about the likes of test_cmp, and executes them\ndirectly?\n\nHere's proof of concept as a patch to the dash shell:\n\n u dash (debian/master=) $ git diff\n diff --git a/src/builtins.def.in b/src/builtins.def.in\n index 4441fe4..b214a17 100644\n --- a/src/builtins.def.in\n +++ b/src/builtins.def.in\n @@ -92,3 +92,4 @@ ulimitcmd ulimit\n #endif\n testcmd test [\n killcmd -u kill\n +testcmpcmd test_cmp\n diff --git a/src/jobs.c b/src/jobs.c\n index c2c2332..905563f 100644\n --- a/src/jobs.c\n +++ b/src/jobs.c\n @@ -1502,3 +1502,12 @@ getstatus(struct job *job) {\n jobno(job), job->nprocs, status, retval));\n return retval;\n }\n +\n +#include \n +int\n +testcmpcmd(argc, argv)\n + int argc;\n + char **argv;\n +{\n + fprintf(stderr, \"Got %d arguments\\n\", argc);\n +}\n\nI just added that to jobs.c because it was easiest, then test_cmp\nbecomes a builtin:\n\n u dash (debian/master=) $ src/dash -c 'type test_cmp'\n test_cmp is a shell builtin\n u dash (debian/master=) $ src/dash -c 'echo foo && test_cmp 1 2 3'\n foo\n Got 4 arguments\n\nI.e. it's really easy to add new built in commands to the dash shell\n(and probably other shells, but dash is really small & fast).\n\nWe could carry some patch like that to dash, and also patch it so\ntest-lib.sh could know that that was our own custom shell, and we'd then\nskip defining functions like test_cmp, and instead use that new builtin.\n\nSimilarly, it could then be linked to our own binaries, and the\ntest-tool would be a builtin that would appropriately dispatch, and we\ncould even eventually make \"git\" a shell builtin.\n\nI don't have time or interest to work on this now, but thought it was\ninteresting to share. This assumes that something in shellscript like:\n\n while echo foo; do echo bar; done\n\nIs no slower on Windows than *nix, since it's purely using built-ins, as\nopposed to something that would shell out.\n"},{"id":"343455","messageId":"nycvar.QRO.7.76.6.1803302045190.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"87h8oxwsui.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/9] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T18:46:25Z","receivedAt":"2018-03-30T18:46:46Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 30 Mar 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Thu, Mar 29 2018, Johannes Schindelin wrote:\n> \n> > Nonetheless, I would be confortable with this patch going into\n> > v2.17.0, even at this late stage. The final verdict is Junio's, of\n> > course.\n> \n> Thanks a lot for working on this. I'm keen to stress test this, but\n> won't have time in the next few days, and in any case think that the\n> parts that change functionality should wait until after 2.17 (but e.g.\n> the test renaming would be fine for a cherry-pick).\n\nObviously this was never meant to get into v2.17.0 (apart maybe from 1/9,\nwhich however is so contested over that addition of the test case under\nthe assumption that anybody but me would dare to touch those parts of the\ncode).\n\nCiao,\nDscho"},{"id":"343456","messageId":"nycvar.QRO.7.76.6.1803302046550.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"CACsJy8DPof1jFe_cOS5+5dRHehU+_Y9y86P+5f3s_C0u85BXBw@mail.gmail.com","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T18:53:41Z","receivedAt":"2018-03-30T18:54:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Fri, 30 Mar 2018, Duy Nguyen wrote:\n\n> On Fri, Mar 30, 2018 at 2:32 PM, Johannes Schindelin\n> wrote:\n> >\n> > On Thu, 29 Mar 2018, Jeff King wrote:\n> >\n> >> On Thu, Mar 29, 2018 at 11:15:33AM -0700, Stefan Beller wrote:\n> >>\n> >> > > When calling `git config --unset abc.a` on this file, it leaves this\n> >> > > (invalid) config behind:\n> >> > >\n> >> > > [\n> >> > > [xyz]\n> >> > > key = value\n> >> > >\n> >> > > The reason is that we try to search for the beginning of the line (or\n> >> > > for the end of the preceding section header on the same line) that\n> >> > > defines abc.a, but as an optimization, we subtract 2 from the offset\n> >> > > pointing just after the definition before we call\n> >> > > find_beginning_of_line(). That function, however, *also* performs that\n> >> > > optimization and promptly fails to find the section header correctly.\n> >> >\n> >> > This commit message would be more convincing if we had it in test form.\n> >>\n> >> I agree a test might be nice. But I don't find the commit message\n> >> unconvincing at all. It explains pretty clearly why the bug occurs, and\n> >> you can verify it by looking at find_beginning_of_line.\n> >>\n> >> > [abc]a\n> >> >\n> >> > is not written by Git, but would be written from an outside tool or person\n> >> > and we barely cope with it?\n> >>\n> >> Yes, I don't think git would ever write onto the same line. But clearly\n> >> we should handle anything that's syntactically valid.\n> >\n> > I was tempted to add the test case, because it is easy to test it.\n> >\n> > But I then decided *not* to add it. Why? Testing is a balance between \"can\n> > do\" and \"need to do\".\n> >\n> > Can you imagine that I did *not* run the entire test suite before\n> > submitting this patch series, because it takes an incredible *90 minutes*\n> > to run *on a fast Windows machine*?\n> \n> What's wrong with firing up a new worktree, run the test suite there\n> and go back to do something else so you won't waste time just waiting\n> for test results and submit? Sure there is a mental overhead for\n> switching tasks, but at 90 minutes, I think it's worth doing.\n\nOf course it is worth doing. That's why I often test the end result on\nWindows (waiting those 90 minutes, but I do not fire up a new worktree, I\nuse my cloud privilege and let Azure/Visual Studio Team Services do the\nwork for me, without slowing down my laptop).\n\nWhat I would love to do, however, would be to test all intermediate\npatches, too, as that often shows a problem with my frequent reorderings\nvia interactive rebases. And 90 minutes times 9 is... 13 hours and 30\nminutes. That's a really long time.\n\nI think the best course of action would be to incrementally do away with\nthe shell scripted test framework, in the way you outlined earlier this\nyear. This would *also* buy us a wealth of other benefits, such as better\ncontrol over the parallelization, resource usage, etc.\n\nIt would also finally make it easier to introduce something like \"smart\ntesting\" where code coverage could be computed (this works only for C\ncode, of course, not for the many scripted parts of core Git), and a diff\ncould be inspected to discover which tests *really* need to be run,\nskipping the tests that would only touch unchanged code.\n\nCiao,\nDscho\n"},{"id":"343458","messageId":"xmqqk1tt2xww.fsf@gitster-ct.c.googlers.com","threadId":"48185","inReplyTo":"87fu4hwgfa.fsf@evledraar.gmail.com","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-30T18:58:23Z","receivedAt":"2018-03-30T18:58:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Fri, Mar 30 2018, Johannes Schindelin wrote [expressing frustrations\n> about Windows test suite slowness]:\n>\n> I've wondered for a while whether it wouldn't be a viable approach to\n> make something like an interpreter for our test suite to get around this\n> problem, i.e. much of it's very repetitive and just using a few shell\n> functions we've defined, what if we had C equivalents of those?\n> ...\n>\n> I don't have time or interest to work on this now, but thought it was\n> interesting to share. This assumes that something in shellscript like:\n>\n> while echo foo; do echo bar; done\n>\n> Is no slower on Windows than *nix, since it's purely using built-ins, as\n> opposed to something that would shell out.\n\nThat's interesting; it certainly is appreciated to be constructive\nto find a usable solution.\n\n"},{"id":"343460","messageId":"xmqqfu4h2xsl.fsf@gitster-ct.c.googlers.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803302042570.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-30T19:00:58Z","receivedAt":"2018-03-30T19:01:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> What would be a *really* good strategy is: \"Oh, there is a problem! Let's\n> acknowledge it and try to come up with a solution rather than a\n> work-around\".\n>\n> EXPENSIVE_ON_WINDOWS is a symptom. Not a solution.\n\nYes, it is a workaround. Making shell faster on windows would of\ncourse be one possible solution to make t/t*.sh scripts go faster\n;-) Or update parts of t/t*.sh so that the equivalent test coverage\ncan be kept while running making them go faster on Windows.\n\n\n\n\n"},{"id":"343462","messageId":"nycvar.QRO.7.76.6.1803302102480.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180330140840.GA5869@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] git_config_set: use do_config_from_file() directly","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-30T19:04:13Z","receivedAt":"2018-03-30T19:04:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 30 Mar 2018, Jeff King wrote:\n\n> On Fri, Mar 30, 2018 at 04:01:56PM +0200, Johannes Schindelin wrote:\n> \n> > You know what is *really* funny?\n> > \n> > -- snip --\n> > static int git_config_from_stdin(config_fn_t fn, void *data)\n> > {\n> > return do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data, 0);\n> > }\n> > \n> > int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n> > {\n> > int ret = -1;\n> > FILE *f;\n> > \n> > f = fopen_or_warn(filename, \"r\");\n> > if (f) {\n> > flockfile(f);\n> > ret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data, 0);\n> > funlockfile(f);\n> > fclose(f);\n> > }\n> > return ret;\n> > }\n> > -- snap --\n> > \n> > So the _stdin variant *goes out of its way not to flockfile()*...\n> \n> *facepalm* That's probably my fault, since git_config_from_stdin()\n> existed already when I did the flockfile stuff.\n> \n> Probably the flockfile should go into do_config_from_file(), where we\n> specify to use the unlocked variants.\n\nAh, that makes sense now! I am glad I could also help ;-)\n\n> > But I guess all this will become moot when I start handing down the config\n> > options. It does mean that I have to change the signatures in header\n> > files, oh well ;-)\n> > \n> > But then I can drop this here patch and we can stop musing about\n> > flockfile() ;-)\n> \n> Yeah, I'll wait to see how your refactor turns out.\n\nI don't think I'll touch too much in that part of the code. My changes\nshould not cause merge conflicts with a patch moving the\nflockfile()/funlockfile() calls to do_config_from_file().\n\nCiao,\nDscho\n"},{"id":"343465","messageId":"20180330191620.GA32338@sigill.intra.peff.net","threadId":"48185","inReplyTo":"87fu4hwgfa.fsf@evledraar.gmail.com","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T19:16:20Z","receivedAt":"2018-03-30T19:16:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 30, 2018 at 08:45:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> I've wondered for a while whether it wouldn't be a viable approach to\n> make something like an interpreter for our test suite to get around this\n> problem, i.e. much of it's very repetitive and just using a few shell\n> functions we've defined, what if we had C equivalents of those?\n\nI've had a similar thought, though I wonder how far we could get with\njust shell. I even tried it out with test_cmp:\n\n https://public-inbox.org/git/20161020215647.5no7effvutwep2xt@sigill.intra.peff.net/\n\nBut Johannes Sixt pointed out that they already do this (see\nmingw_test_cmp in test-lib-functions).\n\nI also tried to explore a few numbers about process invocations to see\nif running shell commands is the problem:\n\n https://public-inbox.org/git/20161020123111.qnbsainul2g54z4z@sigill.intra.peff.net/\n\nThere was some discussion there about whether the problem is programs\nbeing exec'd, or if it's forks due to subshells. And if it is programs\nbeing exec'd, whether it's shell programs or if it is simply that we\nexec Git a huge number of times.\n\n-Peff\n"},{"id":"343466","messageId":"CACsJy8DFva4y_Am-ZBei=VWXi4OdA0Y9Hc4xx7BnOHzVZP5gTQ@mail.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1803302046550.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-30T19:16:02Z","receivedAt":"2018-03-30T19:16:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 8:53 PM, Johannes Schindelin\n wrote:\n> I think the best course of action would be to incrementally do away with\n> the shell scripted test framework, in the way you outlined earlier this\n> year. This would *also* buy us a wealth of other benefits, such as better\n> control over the parallelization, resource usage, etc.\n\nIf you have not noticed, I'm a bit busy with all sorts of stuff and\nprobably won't continue that work. And since it affects you the most,\nyou probably have the best motive to tackle it ;-) I don't think\ncomplaining about slow test suite helps. And avoiding adding more\ntests because of that definitely does not help.\n\n> It would also finally make it easier to introduce something like \"smart\n> testing\" where code coverage could be computed (this works only for C\n> code, of course, not for the many scripted parts of core Git), and a diff\n> could be inspected to discover which tests *really* need to be run,\n> skipping the tests that would only touch unchanged code.\n-- \nDuy\n"},{"id":"343670","messageId":"nycvar.QRO.7.76.6.1804031127400.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"xmqqfu4h2xsl.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T09:31:30Z","receivedAt":"2018-04-03T09:31:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 30 Mar 2018, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > What would be a *really* good strategy is: \"Oh, there is a problem! Let's\n> > acknowledge it and try to come up with a solution rather than a\n> > work-around\".\n> >\n> > EXPENSIVE_ON_WINDOWS is a symptom. Not a solution.\n> \n> Yes, it is a workaround. Making shell faster on windows would of\n> course be one possible solution to make t/t*.sh scripts go faster\n> ;-) Or update parts of t/t*.sh so that the equivalent test coverage\n> can be kept while running making them go faster on Windows.\n\nWhat makes you think that I did not try my hardest for around 812 hours in\ntotal so far to make the shell faster?\n\nCiao,\nDscho\n\nP.S.: I do not have the actual number of hours I spent on both MSYS2's\nruntime and BusyBox and Git to find *some* way to make it faster, as my\ntime-keeping is organized in a different way that makes it hard to query\nthe overall number. But I can state with confidence that it is easily in\nthe 200-300 hour range, if not beyond that.\n\nIt is very frustrating to spend that much time with only little gains here\nand there (and BusyBox-w32 is simply not robust enough yet, apart from\nalso not showing a significant improvement in performance). Please do not\nmake this experience even more frustrating. Thanks.\n"},{"id":"343671","messageId":"nycvar.QRO.7.76.6.1804031133270.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180330191620.GA32338@sigill.intra.peff.net","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T09:49:38Z","receivedAt":"2018-04-03T09:50:02Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 30 Mar 2018, Jeff King wrote:\n\n> On Fri, Mar 30, 2018 at 08:45:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> > I've wondered for a while whether it wouldn't be a viable approach to\n> > make something like an interpreter for our test suite to get around\n> > this problem, i.e. much of it's very repetitive and just using a few\n> > shell functions we've defined, what if we had C equivalents of those?\n> \n> I've had a similar thought, though I wonder how far we could get with\n> just shell. I even tried it out with test_cmp:\n> \n> https://public-inbox.org/git/20161020215647.5no7effvutwep2xt@sigill.intra.peff.net/\n> \n> But Johannes Sixt pointed out that they already do this (see\n> mingw_test_cmp in test-lib-functions).\n\nRight.\n\nAdditionally, I noticed that that simple loop in shell is *also* very slow on\nWindows (at least in the MSYS2 Bash we use in Git for Windows).\n\nUnder the assumption that it is the Bash with the loop that uses too much\nPOSIX emulation to make it fast, I re-implemented mingw_test_cmp in pure\nC:\nhttps://github.com/git-for-windows/git/commit/8a96ef63a0083ba02305dfeef6ff92c31b4fd7c3\n\nUnfortunately, it did not produce any noticeable speed improvement, so I\ndid not even finish the conversion (when the cmp fails, it does not show\nyou any helpful diff yet).\n\n> I also tried to explore a few numbers about process invocations to see\n> if running shell commands is the problem:\n> \n> https://public-inbox.org/git/20161020123111.qnbsainul2g54z4z@sigill.intra.peff.net/\n\nThis mail was still in my inbox, in want of me saying something about\nthis.\n\nMy main evidence that shell scripts on macOS are slower than on Linux was\nthe difference of the improvement incurred by moving more things from\ngit-rebase--interactive.sh into sequencer.c: Linux saw an improvement only\nof about 3x, while macOS saw an improvement of 4x, IIRC. If I don't\nremember the absolute numbers correctly, at least I vividly remember the\nqualitative difference: It was noticeable.\n\n> There was some discussion there about whether the problem is programs\n> being exec'd, or if it's forks due to subshells. And if it is programs\n> being exec'd, whether it's shell programs or if it is simply that we\n> exec Git a huge number of times.\n\nOne large problem there is that it is really hard to analyze performance\nover such a heterogenous code base: part C, part Perl, part Unix shell\n(and of course, when you say Unix shell, you imply dozens of separate\ntools that *also* need to be performance-profiled). I have very good\nprofiling tools for C, I saw some built-in performance profiling for Perl,\nbut there is no good performance profiling for Unix shell scripting: I\ndoubt that the inventors of shell scripting had speed-critical production\ncode in mind when they came up with the idea.\n\nI did invest dozens of hours earlier this year trying to obtain debug\nsymbols in .pdb format (ready for Visual Studio's really envy-inducing\nperformance profiler) also for the MSYS2 runtime and Bash, so that I could\nanalyze what makes things so awfully slow in Git's test suite.\n\nThe only problem is that I also have to do other things in my day-job, so\nthat project waits patiently until I have some time to come back to that\nproject.\n\nCiao,\nDscho"},{"id":"343675","messageId":"87sh8cv8a5.fsf@evledraar.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1804031133270.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-04-03T11:28:18Z","receivedAt":"2018-04-03T11:28:26Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Apr 03 2018, Johannes Schindelin wrote:\n\n> Hi Peff,\n>\n> On Fri, 30 Mar 2018, Jeff King wrote:\n>\n>> On Fri, Mar 30, 2018 at 08:45:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> > I've wondered for a while whether it wouldn't be a viable approach to\n>> > make something like an interpreter for our test suite to get around\n>> > this problem, i.e. much of it's very repetitive and just using a few\n>> > shell functions we've defined, what if we had C equivalents of those?\n>>\n>> I've had a similar thought, though I wonder how far we could get with\n>> just shell. I even tried it out with test_cmp:\n>>\n>> https://public-inbox.org/git/20161020215647.5no7effvutwep2xt@sigill.intra.peff.net/\n>>\n>> But Johannes Sixt pointed out that they already do this (see\n>> mingw_test_cmp in test-lib-functions).\n>\n> Right.\n>\n> Additionally, I noticed that that simple loop in shell is *also* very slow on\n> Windows (at least in the MSYS2 Bash we use in Git for Windows).\n>\n> Under the assumption that it is the Bash with the loop that uses too much\n> POSIX emulation to make it fast, I re-implemented mingw_test_cmp in pure\n> C:\n> https://github.com/git-for-windows/git/commit/8a96ef63a0083ba02305dfeef6ff92c31b4fd7c3\n>\n> Unfortunately, it did not produce any noticeable speed improvement, so I\n> did not even finish the conversion (when the cmp fails, it does not show\n> you any helpful diff yet).\n\nI don't know the details of Windows, but it sounds like you're trying to\nperformance test two things that are going to suck for different\nreasons.\n\nOn one hand the pure-*.sh comparison would be slower than just diff on\n*nix, because it's not C, so you'll get that slowness, but gain in not\nhaving to fork another process.\n\nOn the other hand the C implementation is going to be really fast, but\nit's going to take you a long time to get it started on Windows.\n\nWhich is why I think it would be really interesting to see the third\napproach I suggested, i.e. hack the shell to make the test_cmp a builtin\nand test that. Then you won't fork, but will get the advantage of your\nfast C codepath.\n\nAlso, even if test_cmp is much faster, Peff's results over at\nhttps://public-inbox.org/git/20161020123111.qnbsainul2g54z4z@sigill.intra.peff.net/\nsuggest that you may not notice anyway. Aside from the points raised\nthere about the bin wrappers it seems the easiest wins are having a\nbuiltin version of \"rm\" and \"cat\".\n\nAre you able to compile dash on Windows with some modification of the\npatch I sent upthread? If not it doesn't seem too hard to do the same\ntrick for bash, see:\n\n git grep '\\balias\\b' -- builtins\n\nOnce you have bash.git checked out. I.e. you add a bit of Makefile\nboilerplate and you should be able to get a new builtin.\n\n>> I also tried to explore a few numbers about process invocations to see\n>> if running shell commands is the problem:\n>>\n>> https://public-inbox.org/git/20161020123111.qnbsainul2g54z4z@sigill.intra.peff.net/\n>\n> This mail was still in my inbox, in want of me saying something about\n> this.\n>\n> My main evidence that shell scripts on macOS are slower than on Linux was\n> the difference of the improvement incurred by moving more things from\n> git-rebase--interactive.sh into sequencer.c: Linux saw an improvement only\n> of about 3x, while macOS saw an improvement of 4x, IIRC. If I don't\n> remember the absolute numbers correctly, at least I vividly remember the\n> qualitative difference: It was noticeable.\n>\n>> There was some discussion there about whether the problem is programs\n>> being exec'd, or if it's forks due to subshells. And if it is programs\n>> being exec'd, whether it's shell programs or if it is simply that we\n>> exec Git a huge number of times.\n>\n> One large problem there is that it is really hard to analyze performance\n> over such a heterogenous code base: part C, part Perl, part Unix shell\n> (and of course, when you say Unix shell, you imply dozens of separate\n> tools that *also* need to be performance-profiled). I have very good\n> profiling tools for C, I saw some built-in performance profiling for Perl,\n> but there is no good performance profiling for Unix shell scripting: I\n> doubt that the inventors of shell scripting had speed-critical production\n> code in mind when they came up with the idea.\n>\n> I did invest dozens of hours earlier this year trying to obtain debug\n> symbols in .pdb format (ready for Visual Studio's really envy-inducing\n> performance profiler) also for the MSYS2 runtime and Bash, so that I could\n> analyze what makes things so awfully slow in Git's test suite.\n>\n> The only problem is that I also have to do other things in my day-job, so\n> that project waits patiently until I have some time to come back to that\n> project.\n>\n> Ciao,\n> Dscho\n"},{"id":"343677","messageId":"nycvar.QRO.7.76.6.1804031150030.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"87fu4hwgfa.fsf@evledraar.gmail.com","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T11:43:10Z","receivedAt":"2018-04-03T11:43:40Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 30 Mar 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Mar 30 2018, Johannes Schindelin wrote [expressing frustrations\n> about Windows test suite slowness]:\n\nTo be precise (and I think it is important to be precise here): it is not\nthe Windows test suite about which I talked, it is Git's test suite, as\nrun on Windows. It might sound like a small difference, but it is not: the\nfault really lies with Git because it wants to be a portable software.\n\n> I've wondered for a while whether it wouldn't be a viable approach to\n> make something like an interpreter for our test suite to get around this\n> problem, i.e. much of it's very repetitive and just using a few shell\n> functions we've defined, what if we had C equivalents of those?\n\nThere has even been an attempt to do this by Linus Torvalds himself:\n\nhttps://public-inbox.org/git/Pine.LNX.4.64.0602232229340.3771@g5.osdl.org/\n\nIt has not really gone anywhere...\n\nTo be honest, I had a different idea (because I do not really want to\nmaintain yet another piece of software): BusyBox. The source code is clean\nenough, and it should, in theory, allow us to go really fast.\n\n> Duy had a WIP patch set a while ago to add C test suite support, but I\n> thought what if we turn that inside-out, and instead have a shell\n> interpreter that knows about the likes of test_cmp, and executes them\n> directly?\n\nThe problem, of course, is: if you add Git-test-suite-specific stuff to\nany Unix shell, you are going to have to maintain this fork, and all of a\nsudden it has become a lot harder to develop Git, and to port it.\n\nQuite frankly, I would rather go with Duy's original approach, or a\nvariation thereof, as snuck into the wildmatch discussion here:\n\n\thttps://public-inbox.org/git/20180110090724.GA2893@ash/\n\n> Here's proof of concept as a patch to the dash shell:\n> \n> u dash (debian/master=) $ git diff\n> diff --git a/src/builtins.def.in b/src/builtins.def.in\n> index 4441fe4..b214a17 100644\n> --- a/src/builtins.def.in\n> +++ b/src/builtins.def.in\n> @@ -92,3 +92,4 @@ ulimitcmd ulimit\n> #endif\n> testcmd test [\n> killcmd -u kill\n> +testcmpcmd test_cmp\n> diff --git a/src/jobs.c b/src/jobs.c\n> index c2c2332..905563f 100644\n> --- a/src/jobs.c\n> +++ b/src/jobs.c\n> @@ -1502,3 +1502,12 @@ getstatus(struct job *job) {\n> jobno(job), job->nprocs, status, retval));\n> return retval;\n> }\n> +\n> +#include \n> +int\n> +testcmpcmd(argc, argv)\n> + int argc;\n> + char **argv;\n> +{\n> + fprintf(stderr, \"Got %d arguments\\n\", argc);\n> +}\n> \n> I just added that to jobs.c because it was easiest, then test_cmp\n> becomes a builtin:\n> \n> u dash (debian/master=) $ src/dash -c 'type test_cmp'\n> test_cmp is a shell builtin\n> u dash (debian/master=) $ src/dash -c 'echo foo && test_cmp 1 2 3'\n> foo\n> Got 4 arguments\n> \n> I.e. it's really easy to add new built in commands to the dash shell\n> (and probably other shells, but dash is really small & fast).\n> \n> We could carry some patch like that to dash, and also patch it so\n> test-lib.sh could know that that was our own custom shell, and we'd then\n> skip defining functions like test_cmp, and instead use that new builtin.\n\nOr even use the output of `type test_cmp` as a tell-tale.\n\n> Similarly, it could then be linked to our own binaries, and the\n> test-tool would be a builtin that would appropriately dispatch, and we\n> could even eventually make \"git\" a shell builtin.\n> \n> I don't have time or interest to work on this now, but thought it was\n> interesting to share. This assumes that something in shellscript like:\n> \n> while echo foo; do echo bar; done\n> \n> Is no slower on Windows than *nix, since it's purely using built-ins, as\n> opposed to something that would shell out.\n\nIt is still interpreting stuff. And it still goes through the POSIX\nemulation layer.\n\nI did see reports on the Git for Windows bug tracker that gave me the\nimpression that such loops in Unix shell scripts may not, in fact, be as\nperformant in MSYS2's Bash as you would like to believe:\n\nhttps://github.com/git-for-windows/git/issues/1533#issuecomment-372025449\n\nCiao,\nDscho"},{"id":"343691","messageId":"20180403132717.GC18824@sigill.intra.peff.net","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1804031150030.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-03T13:27:17Z","receivedAt":"2018-04-03T13:27:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 01:43:10PM +0200, Johannes Schindelin wrote:\n\n> > I don't have time or interest to work on this now, but thought it was\n> > interesting to share. This assumes that something in shellscript like:\n> > \n> > while echo foo; do echo bar; done\n> > \n> > Is no slower on Windows than *nix, since it's purely using built-ins, as\n> > opposed to something that would shell out.\n> \n> It is still interpreting stuff. And it still goes through the POSIX\n> emulation layer.\n> \n> I did see reports on the Git for Windows bug tracker that gave me the\n> impression that such loops in Unix shell scripts may not, in fact, be as\n> performant in MSYS2's Bash as you would like to believe:\n> \n> https://github.com/git-for-windows/git/issues/1533#issuecomment-372025449\n\nThe main problem with `read` loops in shell is that the shell makes one\nread() syscall per character. It has to, because doing otherwise is\nuser-visible in cases where the descriptor may get passed to a different\nprocess.\n\nThere's unfortunately no portable way to say \"please just read this\nquickly, I promise nobody else is going to read the descriptor\". And nor\ndo I know of any shell which is smart enough to know that it's going to\nconsume to EOF anyway (as you would for something like \"cmd | while\nread\").\n\nIf you know you have bash, you can use \"-N\" to get a more efficient\nread:\n\n $ echo foo | strace -e read bash -c 'read foo'\n [...]\n read(0, \"f\", 1) = 1\n read(0, \"o\", 1) = 1\n read(0, \"o\", 1) = 1\n read(0, \"\\n\", 1) = 1\n\n $ echo foo | strace -e read bash -c 'read -N 10 foo'\n [...]\n read(0, \"foo\\n\", 10) = 4\n read(0, \"\", 6) = 0\n\nbut then you have another problem: how to split the resulting buffer\ninto lines in shell. ;)\n\nBut if we're at the point of creating custom C builtins for\nbusybox/dash/etc, you should be able to create a primitive for \"read\nthis using buffered stdio, other processes be damned, and return one\nline at a time\".\n\n-Peff\n"},{"id":"343701","messageId":"CACsJy8Bt3AEZmjudL+scXqOC_oW1WtU0_QZVD9PM+drGbEBeVw@mail.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1804031127400.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-03T15:29:23Z","receivedAt":"2018-04-03T15:29:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Apr 3, 2018 at 11:31 AM, Johannes Schindelin\n wrote:\n> It is very frustrating to spend that much time with only little gains here\n> and there (and BusyBox-w32 is simply not robust enough yet, apart from\n> also not showing a significant improvement in performance).\n\nYou still use busybox-w32? It's amazing that people still use it after\nthe linux subsystem comes. busybox has a lot of commands built in\n(i.e. no new processes) and unless rmyorston did something more, the\n\"fork\" in ash shell should be as cheap as it could be: it simply\nserializes data and sends to the new process.\n\nIf performance does not improve, I guess the process creation cost\ndominates. There's not much we could do except moving away from the\nzillion processes test framework: either something C-based or another\nscripting language (ok I don't want to bring this up again)\n-- \nDuy\n"},{"id":"343702","messageId":"nycvar.QRO.7.76.6.1804031744220.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"CACsJy8Bt3AEZmjudL+scXqOC_oW1WtU0_QZVD9PM+drGbEBeVw@mail.gmail.com","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T15:47:44Z","receivedAt":"2018-04-03T15:48:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Tue, 3 Apr 2018, Duy Nguyen wrote:\n\n> On Tue, Apr 3, 2018 at 11:31 AM, Johannes Schindelin\n> wrote:\n> > It is very frustrating to spend that much time with only little gains\n> > here and there (and BusyBox-w32 is simply not robust enough yet, apart\n> > from also not showing a significant improvement in performance).\n> \n> You still use busybox-w32?\n\nYes.\n\n> It's amazing that people still use it after the linux subsystem comes.\n\nI use WSL myself. But you need to realize that WSL is only available on\nWindows 10 (many users still use Windows 7), and it is a little tricky to\nget to work in Docker containers, I heard, so I did not even try.\n\nAlso, many Windows users are unfamiliar with Linux, and forcing them to\nlearn and install a Linux distribution on their machine when all they want\nis to use Git is a bit... much.\n\n> busybox has a lot of commands built in (i.e. no new processes) and\n> unless rmyorston did something more, the \"fork\" in ash shell should be\n> as cheap as it could be: it simply serializes data and sends to the new\n> process.\n\nYes, I had the pleasure of reading that code.\n\nIt might surprise you, but I had to come up with quite a bit of patches to\nmake the test suite pass. And it does not really pass, as I randomly get\nhangs...\n\n> If performance does not improve, I guess the process creation cost\n> dominates. There's not much we could do except moving away from the\n> zillion processes test framework: either something C-based or another\n> scripting language (ok I don't want to bring this up again)\n\nThere is no need to guess. I now have .pdb files, and once I have a good\nexample of a shell script construct that is particularly slow, and once I\nfind some time to work on it, I will dig into the bottlenecks.\n\nCiao,\nDscho\n"},{"id":"343703","messageId":"nycvar.QRO.7.76.6.1804031749080.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"87sh8cv8a5.fsf@evledraar.gmail.com","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T15:55:57Z","receivedAt":"2018-04-03T15:56:19Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 3 Apr 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> [...] I think it would be really interesting to see the third\n> approach I suggested, i.e. hack the shell to make the test_cmp a builtin\n> and test that. Then you won't fork, but will get the advantage of your\n> fast C codepath.\n\nThat should be relatively equivalent to running in BusyBox-w32's ash.\nBusyBox-w32 is a pure-Win32 version of BusyBox (i.e. it does not use any\nPOSIX emulation layer, not Cygwin nor MSYS2).\n\nI did not notice any Earth-shaking performance improvement when running a\ntest with BusyBox-w32's ash. It was a couple of percent, maybe even 20%\nfaster, but nowhere near the orders of magnitude I had been expecting.\n\n> Also, even if test_cmp is much faster, Peff's results over at\n> https://public-inbox.org/git/20161020123111.qnbsainul2g54z4z@sigill.intra.peff.net/\n> suggest that you may not notice anyway. Aside from the points raised\n> there about the bin wrappers it seems the easiest wins are having a\n> builtin version of \"rm\" and \"cat\".\n\nIn BusyBox-w32, `rm` and `cat` *are* built-ins.\n\n> Are you able to compile dash on Windows with some modification of the\n> patch I sent upthread?\n\nIn theory, yes. In practice, I lack the time (and I do not expect this to\nhave any performance benefit over using BusyBox-w32 to run the test suite).\n\nCiao,\nDscho"},{"id":"343704","messageId":"nycvar.QRO.7.76.6.1804031758160.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"20180403132717.GC18824@sigill.intra.peff.net","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:00:05Z","receivedAt":"2018-04-03T16:00:26Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Tue, 3 Apr 2018, Jeff King wrote:\n\n> On Tue, Apr 03, 2018 at 01:43:10PM +0200, Johannes Schindelin wrote:\n> \n> > > I don't have time or interest to work on this now, but thought it was\n> > > interesting to share. This assumes that something in shellscript like:\n> > > \n> > > while echo foo; do echo bar; done\n> > > \n> > > Is no slower on Windows than *nix, since it's purely using built-ins, as\n> > > opposed to something that would shell out.\n> > \n> > It is still interpreting stuff. And it still goes through the POSIX\n> > emulation layer.\n> > \n> > I did see reports on the Git for Windows bug tracker that gave me the\n> > impression that such loops in Unix shell scripts may not, in fact, be as\n> > performant in MSYS2's Bash as you would like to believe:\n> > \n> > https://github.com/git-for-windows/git/issues/1533#issuecomment-372025449\n> \n> The main problem with `read` loops in shell is that the shell makes one\n> read() syscall per character. It has to, because doing otherwise is\n> user-visible in cases where the descriptor may get passed to a different\n> process.\n\nThank you for the explanation. Makes tons of sense now.\n\n> There's unfortunately no portable way to say \"please just read this\n> quickly, I promise nobody else is going to read the descriptor\". And nor\n> do I know of any shell which is smart enough to know that it's going to\n> consume to EOF anyway (as you would for something like \"cmd | while\n> read\").\n> \n> If you know you have bash, you can use \"-N\" to get a more efficient\n> read:\n> \n> $ echo foo | strace -e read bash -c 'read foo'\n> [...]\n> read(0, \"f\", 1) = 1\n> read(0, \"o\", 1) = 1\n> read(0, \"o\", 1) = 1\n> read(0, \"\\n\", 1) = 1\n> \n> $ echo foo | strace -e read bash -c 'read -N 10 foo'\n> [...]\n> read(0, \"foo\\n\", 10) = 4\n> read(0, \"\", 6) = 0\n> \n> but then you have another problem: how to split the resulting buffer\n> into lines in shell. ;)\n\nTrue.\n\n> But if we're at the point of creating custom C builtins for\n> busybox/dash/etc, you should be able to create a primitive for \"read\n> this using buffered stdio, other processes be damned, and return one\n> line at a time\".\n\nWell, you know, I do not think that papering over the root cause will make\nanything better. And the root cause is that we use a test framework\nwritten in Unix shell.\n\nI will have to set aside some time to dig into the bottlenecks there and\nfigure out what parts I can safely convert into \"test builtins\", i.e. into\nthe test-tool Duy introduced, to avoid having shell scripts do the\nheavy-lifting.\n\nCiao,\nDscho\n"},{"id":"343705","messageId":"518729d5abdd93980f388821a96a69c8e4cde82e.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 01/15] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:00Z","receivedAt":"2018-04-03T16:28:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Currently, we are slightly overzealous When removing an entry from a\nconfig file of this form:\n\n\t[abc]a\n\t[xyz]\n\t\tkey = value\n\nWhen calling `git config --unset abc.a` on this file, it leaves this\n(invalid) config behind:\n\n\t[\n\t[xyz]\n\t\tkey = value\n\nThe reason is that we try to search for the beginning of the line (or\nfor the end of the preceding section header on the same line) that\ndefines abc.a, but as an optimization, we subtract 2 from the offset\npointing just after the definition before we call\nfind_beginning_of_line(). That function, however, *also* performs that\noptimization and promptly fails to find the section header correctly.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/config.c b/config.c\nindex b0c20e6cb8a..5cc049aaef0 100644\n--- a/config.c\n+++ b/config.c\n@@ -2632,7 +2632,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n \t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.offset[i]-2, &new_line);\n+\t\t\t\t\tstore.offset[i], &new_line);\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343706","messageId":"cover.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522336130.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 00/15] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:27:55Z","receivedAt":"2018-04-03T16:28:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This patch series originally only tried to help fixing that annoying bug that\nhas been reported several times over the years, where `git config --unset`\nwould leave empty sections behind, and `git config --add` would not reuse them.\n\nThe first patch is somewhat of a \"while at it\" bug fix that I first thought\nwould be a lot more critical than it actually is: It really only affects config\nfiles that start with a section followed immediately (i.e. without a newline)\nby a one-letter boolean setting (i.e. without a `= ` part). So while it\nis a real bug fix, I doubt anybody ever got bitten by it.\n\nThe next swath of patches add and fix some tests, while also fixing the bug\nwhere --replace-all would sometimes insert extra line breaks.\n\nThese fixes are pretty straight-forward, and I always try to keep my added\ntests as concise as possible, so please tell me if you find a way to make them\nsmaller (without giving up readability and debuggability).\n\nThen, I introduce a couple of building blocks: a \"config parser event stream\",\ni.e. an optional callback that can be used to report events such as \"comment\", \"white-space\", etc together with the corresponding extents in the config file.\n\nFinally, the interesting part, where I do two things, essentially (with\npreparatory steps for each thing):\n\n1. I add the ability for `git config --unset/--unset-all` to detect that it\n can remove a section that has just become empty (see below for some more\n discussion of what I consider \"become empty\"), and\n\n2. I add the ability for `git config [--add] key value` to re-use empty\n sections.\n\nI am very, very grateful for the time Peff spent on reviewing the previous\niteration, and hope that he realizes just how much the elegance of the\nevent-stream-based version is due to his excellent review.\n\nTo reiterate why does this patch series not conflict with my very early\nstatements that we cannot simply remove empty sections because we may end up\nwith stale comments?\n\nWell, the patch in question takes pains to determine *iff* there are any\ncomments surrounding, or included in, the section. If any are found: previous\nbehavior. Under the assumption that the user edited the file, we keep it as\nintact as possible (see below for some argument against this). If no comments\nare found, and let's face it, this is probably *the* common case, as few people\nedit their config files by hand these days (neither should they because it is\ntoo easy to end up with an unparseable one), the now-empty section *is*\nremoved.\n\nSo what is the argument against this extra care to detect comments? Well, if\nyou have something like this:\n\n\t[section]\n\t\t; Here we comment about the variable called snarf\n\t\tsnarf = froop\n\nand we run `git config --unset section.snarf`, we end up with this config:\n\n\t[section]\n\t\t; Here we comment about the variable called snarf\n\nwhich obviously does not make sense. However, that is already established\nbehavior for quite a few years, and I do not even try to think of a way how\nthis could be solved.\n\nChanges since v1:\n\n- a new feature was introduced where the config parser can be asked to report\n all \"events\" (like, section header or comment) via a callback function.\n\n- the patches to reuse empty sections, and to remove sections that just became\n empty (without any surrounding comments) were rewritten to make use of that\n config parser event stream (incidentally fixing a couple of problems with\n the backtracking version which were pointed out by Peff).\n\n- to make those changes easier to review, they have been split up into several\n tiny logical steps: the file-local `store` was replaced with callback data,\n some fields were renamed for consistency, the state machine when parsing the\n config was replaced by easier-to-understand flags, etc.\n\n- while pouring over the code, I managed to find another obscure bug: under\n certain circumstances, --replace-all could produce extra new-lines. This is\n now fixed as part of the preparatory patches.\n\n\nJohannes Schindelin (15):\n git_config_set: fix off-by-two\n t1300: rename it to reflect that `repo-config` was deprecated\n t1300: demonstrate that --replace-all can \"invent\" newlines\n config --replace-all: avoid extra line breaks\n t1300: avoid relying on a bug\n t1300: remove unreasonable expectation from TODO\n t1300: `--unset-all` can leave an empty section behind (bug)\n config: introduce an optional event stream while parsing\n config: avoid using the global variable `store`\n config_set_store: rename some fields for consistency\n git_config_set: do not use a state machine\n git_config_set: make use of the config parser's event stream\n git config --unset: remove empty sections (in the common case)\n git_config_set: reuse empty sections\n TODOs\n\n config.c | 449 ++++++++++++++++++++--------\n config.h | 25 ++\n t/{t1300-repo-config.sh => t1300-config.sh} | 57 +++-\n 3 files changed, 396 insertions(+), 135 deletions(-)\n rename t/{t1300-repo-config.sh => t1300-config.sh} (97%)\n\n\nbase-commit: 468165c1d8a442994a825f3684528361727cd8c0\nPublished-As: https://github.com/dscho/git/releases/tag/empty-config-section-v2\nFetch-It-Via: git fetch https://github.com/dscho/git empty-config-section-v2\n\nInterdiff vs v1 (sorry for the size, it is essentially a rewrite of more\nthan half of the previous iteration):\n diff --git a/config.c b/config.c\n index b04c40f76bc..ee7ea24123d 100644\n --- a/config.c\n +++ b/config.c\n @@ -653,21 +653,65 @@ static int get_base_var(struct strbuf *name)\n \t}\n }\n \n +struct parse_event_data {\n +\tenum config_event_t previous_type;\n +\tsize_t previous_offset;\n +\tconst struct config_options *opts;\n +};\n +\n +static inline int do_event(enum config_event_t type,\n +\t\t\t struct parse_event_data *data)\n +{\n +\tsize_t offset;\n +\n +\tif (!data->opts || !data->opts->event_fn)\n +\t\treturn 0;\n +\n +\tif (type == CONFIG_EVENT_WHITESPACE &&\n +\t data->previous_type == type)\n +\t\treturn 0;\n +\n +\toffset = cf->do_ftell(cf);\n +\t/*\n +\t * At EOF, the parser always \"inserts\" an extra '\\n', therefore\n +\t * the end offset of the event is the current file position, otherwise\n +\t * we will already have advanced to the next event.\n +\t */\n +\tif (type != CONFIG_EVENT_EOF)\n +\t\toffset--;\n +\n +\tif (data->previous_type != CONFIG_EVENT_EOF &&\n +\t data->opts->event_fn(data->previous_type, data->previous_offset,\n +\t\t\t\t offset, data->opts->event_fn_data) < 0)\n +\t\treturn -1;\n +\n +\tdata->previous_type = type;\n +\tdata->previous_offset = offset;\n +\n +\treturn 0;\n +}\n +\n static int git_parse_source(config_fn_t fn, void *data,\n -\t\t\t int include_section_headers)\n +\t\t\t const struct config_options *opts)\n {\n \tint comment = 0;\n \tint baselen = 0;\n \tstruct strbuf *var = &cf->var;\n \tint error_return = 0;\n \tchar *error_msg = NULL;\n -\tint saw_section_header = 0;\n \n \t/* U+FEFF Byte Order Mark in UTF8 */\n \tconst char *bomptr = utf8_bom;\n \n +\t/* For the parser event callback */\n +\tstruct parse_event_data event_data = {\n +\t\tCONFIG_EVENT_EOF, 0, opts\n +\t};\n +\n \tfor (;;) {\n -\t\tint c = get_next_char();\n +\t\tint c;\n +\n +\t\tc = get_next_char();\n \t\tif (bomptr && *bomptr) {\n \t\t\t/* We are at the file beginning; skip UTF8-encoded BOM\n \t\t\t * if present. Sane editors won't put this in on their\n @@ -684,39 +728,47 @@ static int git_parse_source(config_fn_t fn, void *data,\n \t\t\t}\n \t\t}\n \t\tif (c == '\\n') {\n -\t\t\tif (cf->eof)\n +\t\t\tif (cf->eof) {\n +\t\t\t\tif (do_event(CONFIG_EVENT_EOF, &event_data) < 0)\n +\t\t\t\t\treturn -1;\n \t\t\t\treturn 0;\n -\t\t\tcomment = 0;\n -\t\t\tif (saw_section_header) {\n -\t\t\t\tif (include_section_headers) {\n -\t\t\t\t\tcf->linenr--;\n -\t\t\t\t\terror_return = fn(var->buf, NULL, data);\n -\t\t\t\t\tif (error_return < 0)\n -\t\t\t\t\t\tbreak;\n -\t\t\t\t\tcf->linenr++;\n -\t\t\t\t}\n -\t\t\t\tsaw_section_header = 0;\n \t\t\t}\n +\t\t\tif (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)\n +\t\t\t\treturn -1;\n +\t\t\tcomment = 0;\n \t\t\tcontinue;\n \t\t}\n -\t\tif (comment || isspace(c))\n +\t\tif (comment)\n \t\t\tcontinue;\n +\t\tif (isspace(c)) {\n +\t\t\tif (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)\n +\t\t\t\t\treturn -1;\n +\t\t\tcontinue;\n +\t\t}\n \t\tif (c == '#' || c == ';') {\n +\t\t\tif (do_event(CONFIG_EVENT_COMMENT, &event_data) < 0)\n +\t\t\t\t\treturn -1;\n \t\t\tcomment = 1;\n \t\t\tcontinue;\n \t\t}\n \t\tif (c == '[') {\n +\t\t\tif (do_event(CONFIG_EVENT_SECTION, &event_data) < 0)\n +\t\t\t\t\treturn -1;\n +\n \t\t\t/* Reset prior to determining a new stem */\n \t\t\tstrbuf_reset(var);\n \t\t\tif (get_base_var(var) < 0 || var->len < 1)\n \t\t\t\tbreak;\n \t\t\tstrbuf_addch(var, '.');\n \t\t\tbaselen = var->len;\n -\t\t\tsaw_section_header = 1;\n \t\t\tcontinue;\n \t\t}\n \t\tif (!isalpha(c))\n \t\t\tbreak;\n +\n +\t\tif (do_event(CONFIG_EVENT_ENTRY, &event_data) < 0)\n +\t\t\treturn -1;\n +\n \t\t/*\n \t\t * Truncate the var name back to the section header\n \t\t * stem prior to grabbing the suffix part of the name\n @@ -728,6 +780,9 @@ static int git_parse_source(config_fn_t fn, void *data,\n \t\t\tbreak;\n \t}\n \n +\tif (do_event(CONFIG_EVENT_ERROR, &event_data) < 0)\n +\t\treturn -1;\n +\n \tswitch (cf->origin_type) {\n \tcase CONFIG_ORIGIN_BLOB:\n \t\terror_msg = xstrfmt(_(\"bad config line %d in blob %s\"),\n @@ -1412,7 +1467,7 @@ int git_default_config(const char *var, const char *value, void *dummy)\n * this function.\n */\n static int do_config_from(struct config_source *top, config_fn_t fn, void *data,\n -\t\t\t int include_section_headers)\n +\t\t\t const struct config_options *opts)\n {\n \tint ret;\n \n @@ -1424,7 +1479,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data,\n \tstrbuf_init(&top->var, 1024);\n \tcf = top;\n \n -\tret = git_parse_source(fn, data, include_section_headers);\n +\tret = git_parse_source(fn, data, opts);\n \n \t/* pop config-file parsing state stack */\n \tstrbuf_release(&top->value);\n @@ -1437,7 +1492,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data,\n static int do_config_from_file(config_fn_t fn,\n \t\tconst enum config_origin_type origin_type,\n \t\tconst char *name, const char *path, FILE *f,\n -\t\tvoid *data, int include_section_headers)\n +\t\tvoid *data, const struct config_options *opts)\n {\n \tstruct config_source top;\n \n @@ -1450,15 +1505,18 @@ static int do_config_from_file(config_fn_t fn,\n \ttop.do_ungetc = config_file_ungetc;\n \ttop.do_ftell = config_file_ftell;\n \n -\treturn do_config_from(&top, fn, data, include_section_headers);\n +\treturn do_config_from(&top, fn, data, opts);\n }\n \n static int git_config_from_stdin(config_fn_t fn, void *data)\n {\n -\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data, 0);\n +\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin,\n +\t\t\t\t data, NULL);\n }\n \n -int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n +int git_config_from_file_with_options(config_fn_t fn, const char *filename,\n +\t\t\t\t void *data,\n +\t\t\t\t const struct config_options *opts)\n {\n \tint ret = -1;\n \tFILE *f;\n @@ -1466,13 +1524,19 @@ int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n \tf = fopen_or_warn(filename, \"r\");\n \tif (f) {\n \t\tflockfile(f);\n -\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data, 0);\n +\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename,\n +\t\t\t\t\t filename, f, data, opts);\n \t\tfunlockfile(f);\n \t\tfclose(f);\n \t}\n \treturn ret;\n }\n \n +int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n +{\n +\treturn git_config_from_file_with_options(fn, filename, data, NULL);\n +}\n +\n int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_type,\n \t\t\tconst char *name, const char *buf, size_t len, void *data)\n {\n @@ -1489,7 +1553,7 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \ttop.do_ungetc = config_buf_ungetc;\n \ttop.do_ftell = config_buf_ftell;\n \n -\treturn do_config_from(&top, fn, data, 0);\n +\treturn do_config_from(&top, fn, data, NULL);\n }\n \n int git_config_from_blob_oid(config_fn_t fn,\n @@ -2233,96 +2297,98 @@ void git_die_config(const char *key, const char *err, ...)\n * Find all the stuff for git_config_set() below.\n */\n \n -static struct {\n +struct config_set_store {\n \tint baselen;\n \tchar *key;\n \tint do_not_match;\n \tregex_t *value_regex;\n \tint multi_replace;\n -\tsize_t *offset;\n -\tunsigned int offset_alloc;\n -\tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n -\tunsigned int seen;\n -} store;\n +\tstruct {\n +\t\tsize_t begin, end;\n +\t\tenum config_event_t type;\n +\t\tint is_keys_section;\n +\t} *parsed;\n +\tunsigned int parsed_nr, parsed_alloc, *seen, seen_nr, seen_alloc;\n +\tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n +};\n \n -static int matches(const char *key, const char *value)\n +static int matches(const char *key, const char *value,\n +\t\t const struct config_set_store *store)\n {\n -\tif (strcmp(key, store.key))\n +\tif (strcmp(key, store->key))\n \t\treturn 0; /* not ours */\n -\tif (!store.value_regex)\n +\tif (!store->value_regex)\n \t\treturn 1; /* always matches */\n -\tif (store.value_regex == CONFIG_REGEX_NONE)\n +\tif (store->value_regex == CONFIG_REGEX_NONE)\n \t\treturn 0; /* never matches */\n \n -\treturn store.do_not_match ^\n -\t\t(value && !regexec(store.value_regex, value, 0, NULL, 0));\n +\treturn store->do_not_match ^\n +\t\t(value && !regexec(store->value_regex, value, 0, NULL, 0));\n +}\n +\n +static int store_aux_event(enum config_event_t type,\n +\t\t\t size_t begin, size_t end, void *data)\n +{\n +\tstruct config_set_store *store = data;\n +\n +\tALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);\n +\tstore->parsed[store->parsed_nr].begin = begin;\n +\tstore->parsed[store->parsed_nr].end = end;\n +\tstore->parsed[store->parsed_nr].type = type;\n +\n +\tif (type == CONFIG_EVENT_SECTION) {\n +\t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n +\t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n +\n +\t\t/* Is this the section we were looking for? */\n +\t\tstore->is_keys_section =\n +\t\t\tstore->parsed[store->parsed_nr].is_keys_section =\n +\t\t\tcf->var.len - 1 == store->baselen &&\n +\t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n +\t\tif (store->is_keys_section) {\n +\t\t\tstore->section_seen = 1;\n +\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n +\t\t\t\t store->seen_alloc);\n +\t\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n +\t\t}\n +\t}\n +\n +\tstore->parsed_nr++;\n +\n +\treturn 0;\n }\n \n static int store_aux(const char *key, const char *value, void *cb)\n {\n -\tconst char *ep;\n -\tsize_t section_len;\n +\tstruct config_set_store *store = cb;\n \n -\tswitch (store.state) {\n -\tcase KEY_SEEN:\n -\t\tif (matches(key, value)) {\n -\t\t\tif (store.seen == 1 && store.multi_replace == 0) {\n +\tif (store->key_seen) {\n +\t\tif (matches(key, value, store)) {\n +\t\t\tif (store->seen_nr == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n \t\t\t}\n \n -\t\t\tALLOC_GROW(store.offset, store.seen + 1,\n -\t\t\t\t store.offset_alloc);\n +\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n +\t\t\t\t store->seen_alloc);\n \n -\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n -\t\t\tstore.seen++;\n +\t\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n +\t\t\tstore->seen_nr++;\n \t\t}\n -\t\tbreak;\n -\tcase SECTION_SEEN:\n +\t} else if (store->is_keys_section) {\n \t\t/*\n -\t\t * What we are looking for is in store.key (both\n -\t\t * section and var), and its section part is baselen\n -\t\t * long. We found key (again, both section and var).\n -\t\t * We would want to know if this key is in the same\n -\t\t * section as what we are looking for. We already\n -\t\t * know we are in the same section as what should\n -\t\t * hold store.key.\n +\t\t * Do not increment matches yet: this may not be a match, but we\n +\t\t * are in the desired section.\n \t\t */\n -\t\tep = strrchr(key, '.');\n -\t\tsection_len = ep - key;\n -\n -\t\tif ((section_len != store.baselen) ||\n -\t\t memcmp(key, store.key, section_len+1)) {\n -\t\t\tstore.state = SECTION_END_SEEN;\n -\t\t\tbreak;\n -\t\t}\n +\t\tALLOC_GROW(store->seen, store->seen_nr + 1, store->seen_alloc);\n +\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n +\t\tstore->section_seen = 1;\n \n -\t\t/*\n -\t\t * Do not increment matches: this is no match, but we\n -\t\t * just made sure we are in the desired section.\n -\t\t */\n -\t\tALLOC_GROW(store.offset, store.seen + 1,\n -\t\t\t store.offset_alloc);\n -\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n -\t\t/* fallthru */\n -\tcase SECTION_END_SEEN:\n -\tcase START:\n -\t\tif (matches(key, value)) {\n -\t\t\tALLOC_GROW(store.offset, store.seen + 1,\n -\t\t\t\t store.offset_alloc);\n -\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n -\t\t\tstore.state = KEY_SEEN;\n -\t\t\tstore.seen++;\n -\t\t} else {\n -\t\t\tif (strrchr(key, '.') - key == store.baselen &&\n -\t\t\t !strncmp(key, store.key, store.baselen)) {\n -\t\t\t\t\tstore.state = SECTION_SEEN;\n -\t\t\t\t\tALLOC_GROW(store.offset,\n -\t\t\t\t\t\t store.seen + 1,\n -\t\t\t\t\t\t store.offset_alloc);\n -\t\t\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n -\t\t\t}\n +\t\tif (matches(key, value, store)) {\n +\t\t\tstore->seen_nr++;\n +\t\t\tstore->key_seen = 1;\n \t\t}\n \t}\n +\n \treturn 0;\n }\n \n @@ -2334,31 +2400,33 @@ static int write_error(const char *filename)\n \treturn 4;\n }\n \n -static struct strbuf store_create_section(const char *key)\n +static struct strbuf store_create_section(const char *key,\n +\t\t\t\t\t const struct config_set_store *store)\n {\n \tconst char *dot;\n \tint i;\n \tstruct strbuf sb = STRBUF_INIT;\n \n -\tdot = memchr(key, '.', store.baselen);\n +\tdot = memchr(key, '.', store->baselen);\n \tif (dot) {\n \t\tstrbuf_addf(&sb, \"[%.*s \\\"\", (int)(dot - key), key);\n -\t\tfor (i = dot - key + 1; i < store.baselen; i++) {\n +\t\tfor (i = dot - key + 1; i < store->baselen; i++) {\n \t\t\tif (key[i] == '\"' || key[i] == '\\\\')\n \t\t\t\tstrbuf_addch(&sb, '\\\\');\n \t\t\tstrbuf_addch(&sb, key[i]);\n \t\t}\n \t\tstrbuf_addstr(&sb, \"\\\"]\\n\");\n \t} else {\n -\t\tstrbuf_addf(&sb, \"[%.*s]\\n\", store.baselen, key);\n +\t\tstrbuf_addf(&sb, \"[%.*s]\\n\", store->baselen, key);\n \t}\n \n \treturn sb;\n }\n \n -static ssize_t write_section(int fd, const char *key)\n +static ssize_t write_section(int fd, const char *key,\n +\t\t\t const struct config_set_store *store)\n {\n -\tstruct strbuf sb = store_create_section(key);\n +\tstruct strbuf sb = store_create_section(key, store);\n \tssize_t ret;\n \n \tret = write_in_full(fd, sb.buf, sb.len);\n @@ -2367,11 +2435,12 @@ static ssize_t write_section(int fd, const char *key)\n \treturn ret;\n }\n \n -static ssize_t write_pair(int fd, const char *key, const char *value)\n +static ssize_t write_pair(int fd, const char *key, const char *value,\n +\t\t\t const struct config_set_store *store)\n {\n \tint i;\n \tssize_t ret;\n -\tint length = strlen(key + store.baselen + 1);\n +\tint length = strlen(key + store->baselen + 1);\n \tconst char *quote = \"\";\n \tstruct strbuf sb = STRBUF_INIT;\n \n @@ -2391,7 +2460,7 @@ static ssize_t write_pair(int fd, const char *key, const char *value)\n \t\tquote = \"\\\"\";\n \n \tstrbuf_addf(&sb, \"\\t%.*s = %s\",\n -\t\t length, key + store.baselen + 1, quote);\n +\t\t length, key + store->baselen + 1, quote);\n \n \tfor (i = 0; value[i]; i++)\n \t\tswitch (value[i]) {\n @@ -2417,201 +2486,85 @@ static ssize_t write_pair(int fd, const char *key, const char *value)\n \treturn ret;\n }\n \n -static ssize_t find_beginning_of_line(const char *contents, size_t size,\n -\tsize_t offset_, int *found_bracket)\n -{\n -\tsize_t equal_offset = size, bracket_offset = size;\n -\tssize_t offset;\n -\n -contline:\n -\tfor (offset = offset_-2; offset > 0\n -\t\t\t&& contents[offset] != '\\n'; offset--)\n -\t\tswitch (contents[offset]) {\n -\t\t\tcase '=': equal_offset = offset; break;\n -\t\t\tcase ']': bracket_offset = offset; break;\n -\t\t}\n -\tif (offset > 0 && contents[offset-1] == '\\\\') {\n -\t\toffset_ = offset;\n -\t\tgoto contline;\n -\t}\n -\tif (bracket_offset < equal_offset) {\n -\t\t*found_bracket = 1;\n -\t\toffset = bracket_offset+1;\n -\t} else\n -\t\toffset++;\n -\n -\treturn offset;\n -}\n -\n -/*\n - * This function determines whether the offset is in a line that starts with a\n - * comment character.\n - *\n - * Note: it does *not* report when a regular line (section header, config\n - * setting) *ends* in a comment.\n - */\n -static int is_in_comment_line(const char *contents, size_t offset)\n -{\n -\tint comment = 0;\n -\n -\twhile (offset > 0)\n -\t\tswitch (contents[--offset]) {\n -\t\tcase ';':\n -\t\tcase '#':\n -\t\t\tcomment = 1;\n -\t\t\tbreak;\n -\t\tcase '\\n':\n -\t\t\tbreak;\n -\t\tcase ' ':\n -\t\tcase '\\t':\n -\t\t\tcontinue;\n -\t\tdefault:\n -\t\t\tcomment = 0;\n -\t\t}\n -\n -\treturn comment;\n -}\n -\n /*\n * If we are about to unset the last key(s) in a section, and if there are\n * no comments surrounding (or included in) the section, we will want to\n * extend begin/end to remove the entire section.\n *\n - * Note: the parameter `i_ptr` points to the index into the store.offset\n - * array, reflecting the end offset of the respective entry to be deleted.\n - * This index may be incremented if a section has more than one entry (which\n - * all are to be removed).\n + * Note: the parameter `seen_ptr` points to the index into the store.seen\n + * array. * This index may be incremented if a section has more than one\n + * entry (which all are to be removed).\n */\n -static void maybe_remove_section(const char *contents, size_t size,\n -\t\t\t\t const char *section_name,\n -\t\t\t\t size_t section_name_len,\n -\t\t\t\t size_t *begin, int *i_ptr, int *new_line)\n +static void maybe_remove_section(struct config_set_store *store,\n +\t\t\t\t const char *contents,\n +\t\t\t\t size_t *begin_offset, size_t *end_offset,\n +\t\t\t\t int *seen_ptr)\n {\n -\tsize_t begin2, end2;\n -\tint seen_section = 0, dummy, i = *i_ptr;\n +\tsize_t begin;\n +\tint i, seen, section_seen = 0;\n \n \t/*\n -\t * First, make sure that this is the last key in the section, and that\n -\t * there are no comments that are possibly about the current section.\n +\t * First, ensure that this is the first key, and that there are no\n +\t * comments before the entry nor before the section header.\n \t */\n -next_entry:\n -\tfor (end2 = store.offset[i]; end2 < size; end2++) {\n -\t\tswitch (contents[end2]) {\n -\t\tcase ' ':\n -\t\tcase '\\t':\n -\t\tcase '\\n':\n -\t\t\tcontinue;\n -\t\tcase '\\r':\n -\t\t\tif (++end2 < size && contents[end2] == '\\n')\n -\t\t\t\tcontinue;\n -\t\t\tbreak;\n -\t\tcase '[':\n -\t\t\t/* If the section name is repeated, continue */\n -\t\t\tif (end2 + 1 + section_name_len < size &&\n -\t\t\t contents[end2 + section_name_len] == ']' &&\n -\t\t\t !memcmp(contents + end2 + 1, section_name,\n -\t\t\t\t section_name_len)) {\n -\t\t\t\tend2 += section_name_len;\n -\t\t\t\tcontinue;\n -\t\t\t}\n -\t\t\tgoto look_before;\n -\t\tcase ';':\n -\t\tcase '#':\n -\t\t\t/* There is a comment, cannot remove this section */\n +\tseen = *seen_ptr;\n +\tfor (i = store->seen[seen]; i > 0; i--) {\n +\t\tenum config_event_t type = store->parsed[i - 1].type;\n +\n +\t\tif (type == CONFIG_EVENT_COMMENT)\n +\t\t\t/* There is a comment before this entry or section */\n \t\t\treturn;\n -\t\tdefault:\n -\t\t\t/* There are other keys in that section */\n +\t\tif (type == CONFIG_EVENT_ENTRY) {\n +\t\t\tif (!section_seen)\n +\t\t\t\t/* This is not the section's first entry. */\n +\t\t\t\treturn;\n +\t\t\t/* We encountered no comment before the section. */\n \t\t\tbreak;\n \t\t}\n -\n -\t\t/*\n -\t\t * Uh oh... we found something else in this section. But do\n -\t\t * we want to remove this, too?\n -\t\t */\n -\t\tif (++i >= store.seen)\n -\t\t\treturn;\n -\n -\t\tbegin2 = find_beginning_of_line(contents, size, store.offset[i],\n -\t\t\t\t\t\t&dummy);\n -\t\tif (begin2 > end2)\n -\t\t\treturn;\n -\n -\t\t/* Looks like we want to remove the next one, too... */\n -\t\tgoto next_entry;\n +\t\tif (type == CONFIG_EVENT_SECTION) {\n +\t\t\tif (!store->parsed[i - 1].is_keys_section)\n +\t\t\t\tbreak;\n +\t\t\tsection_seen = 1;\n +\t\t}\n \t}\n +\tbegin = store->parsed[i].begin;\n \n -look_before:\n \t/*\n -\t * Now, ensure that this is the first key, and that there are no\n -\t * comments before the entry nor before the section header.\n +\t * Next, make sure that we are removing he last key(s) in the section,\n +\t * and that there are no comments that are possibly about the current\n +\t * section.\n \t */\n -\tfor (begin2 = *begin; begin2 > 0; )\n -\t\tswitch (contents[begin2 - 1]) {\n -\t\tcase ' ':\n -\t\tcase '\\t':\n -\t\t\tbegin2--;\n -\t\t\tcontinue;\n -\t\tcase '\\n':\n -\t\t\tif (--begin2 > 0 && contents[begin2 - 1] == '\\r')\n -\t\t\t\tbegin2--;\n -\t\t\tcontinue;\n -\t\tcase ']':\n -\t\t\tif (begin2 > section_name_len + 1 &&\n -\t\t\t contents[begin2 - section_name_len - 2] == '[' &&\n -\t\t\t !memcmp(contents + begin2 - section_name_len - 1,\n -\t\t\t\t section_name, section_name_len)) {\n -\t\t\t\tbegin2 -= section_name_len + 2;\n -\t\t\t\tseen_section = 1;\n -\t\t\t\tcontinue;\n -\t\t\t}\n -\n -\t\t\t/*\n -\t\t\t * It looks like a section header, but it could be a\n -\t\t\t * comment instead...\n -\t\t\t */\n -\t\t\tif (is_in_comment_line(contents, begin2))\n -\t\t\t\treturn;\n -\n -\t\t\t/*\n -\t\t\t * We encountered the previous section header: This\n -\t\t\t * really was the only entry, so remove the entire\n -\t\t\t * section.\n -\t\t\t */\n -\t\t\tif (contents[begin2] != '\\n') {\n -\t\t\t\tbegin2--;\n -\t\t\t\t*new_line = 1;\n -\t\t\t}\n +\tfor (i = store->seen[seen] + 1; i < store->parsed_nr; i++) {\n +\t\tenum config_event_t type = store->parsed[i].type;\n \n -\t\t\tstore.offset[i] = end2;\n -\t\t\t*begin = begin2;\n -\t\t\t*i_ptr = i;\n +\t\tif (type == CONFIG_EVENT_COMMENT)\n \t\t\treturn;\n -\t\tdefault:\n -\t\t\t/*\n -\t\t\t * Any other character means it is either a comment or\n -\t\t\t * a config setting; if it is a comment, we do not want\n -\t\t\t * to remove this section. If it is a config setting,\n -\t\t\t * we only want to remove this section if this is\n -\t\t\t * already the next section.\n -\t\t\t */\n -\t\t\tif (seen_section &&\n -\t\t\t !is_in_comment_line(contents, begin2)) {\n -\t\t\t\tif (contents[begin2] != '\\n') {\n -\t\t\t\t\tbegin2--;\n -\t\t\t\t\t*new_line = 1;\n -\t\t\t\t}\n -\n -\t\t\t\tstore.offset[i] = end2;\n -\t\t\t\t*begin = begin2;\n -\t\t\t\t*i_ptr = i;\n -\t\t\t}\n +\t\tif (type == CONFIG_EVENT_SECTION) {\n +\t\t\tif (store->parsed[i].is_keys_section)\n +\t\t\t\tcontinue;\n +\t\t\tbreak;\n +\t\t}\n +\t\tif (type == CONFIG_EVENT_ENTRY) {\n +\t\t\tif (++seen < store->seen_nr &&\n +\t\t\t i == store->seen[seen])\n +\t\t\t\t/* We want to remove this entry, too */\n +\t\t\t\tcontinue;\n +\t\t\t/* There is another entry in this section. */\n \t\t\treturn;\n \t\t}\n +\t}\n \n -\t/* This section extends to the beginning of the file. */\n -\tstore.offset[i] = end2;\n -\t*begin = begin2;\n -\t*i_ptr = i;\n +\t/*\n +\t * We are really removing the last entry/entries from this section, and\n +\t * there are no enclosed or surrounding comments. Remove the entire,\n +\t * now-empty section.\n +\t */\n +\t*seen_ptr = seen;\n +\t*begin_offset = begin;\n +\tif (i < store->parsed_nr)\n +\t\t*end_offset = store->parsed[i].begin;\n +\telse\n +\t\t*end_offset = store->parsed[store->parsed_nr - 1].end;\n }\n \n int git_config_set_in_file_gently(const char *config_filename,\n @@ -2671,14 +2624,15 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tstruct lock_file lock = LOCK_INIT;\n \tchar *filename_buf = NULL;\n \tchar *contents = NULL;\n -\tchar *section_name = NULL;\n \tsize_t contents_sz;\n +\tstruct config_set_store store;\n +\n +\tmemset(&store, 0, sizeof(store));\n \n \t/* parse-key returns negative; flip the sign to feed exit(3) */\n -\tret = 0 - git_config_parse_key(key, §ion_name, &store.baselen);\n +\tret = 0 - git_config_parse_key(key, &store.key, &store.baselen);\n \tif (ret)\n \t\tgoto out_free;\n -\tstore.key = section_name;\n \n \tstore.multi_replace = multi_replace;\n \n @@ -2692,6 +2646,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tfd = hold_lock_file_for_update(&lock, config_filename, 0);\n \tif (fd < 0) {\n \t\terror_errno(\"could not lock config file %s\", config_filename);\n +\t\tfree(store.key);\n \t\tret = CONFIG_NO_LOCK;\n \t\tgoto out_free;\n \t}\n @@ -2701,6 +2656,8 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t */\n \tin_fd = open(config_filename, O_RDONLY);\n \tif ( in_fd < 0 ) {\n +\t\tfree(store.key);\n +\n \t\tif ( ENOENT != errno ) {\n \t\t\terror_errno(\"opening %s\", config_filename);\n \t\t\tret = CONFIG_INVALID_FILE; /* same as \"invalid config file\" */\n @@ -2713,14 +2670,14 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t}\n \n \t\tstore.key = (char *)key;\n -\t\tif (write_section(fd, key) < 0 ||\n -\t\t write_pair(fd, key, value) < 0)\n +\t\tif (write_section(fd, key, &store) < 0 ||\n +\t\t write_pair(fd, key, value, &store) < 0)\n \t\t\tgoto write_err_out;\n \t} else {\n \t\tstruct stat st;\n \t\tsize_t copy_begin, copy_end;\n \t\tint i, new_line = 0;\n -\t\tFILE *f;\n +\t\tstruct config_options opts;\n \n \t\tif (value_regex == NULL)\n \t\t\tstore.value_regex = NULL;\n @@ -2743,34 +2700,36 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t}\n \t\t}\n \n -\t\tALLOC_GROW(store.offset, 1, store.offset_alloc);\n -\t\tstore.offset[0] = 0;\n -\t\tstore.state = START;\n -\t\tstore.seen = 0;\n +\t\tALLOC_GROW(store.parsed, 1, store.parsed_alloc);\n +\t\tstore.parsed[0].end = 0;\n +\n +\t\tmemset(&opts, 0, sizeof(opts));\n +\t\topts.event_fn = store_aux_event;\n +\t\topts.event_fn_data = &store;\n \n \t\t/*\n -\t\t * After this, store.offset will contain the *end* offset\n -\t\t * of the last match, or remain at 0 if no match was found.\n +\t\t * After this, store.parsed will contain offsets of all the\n +\t\t * parsed elements, and store.seen will contain a list of\n +\t\t * matches, as indices into store.parsed.\n +\t\t *\n \t\t * As a side effect, we make sure to transform only a valid\n \t\t * existing config file.\n \t\t */\n -\t\tf = fopen_or_warn(config_filename, \"r\");\n -\t\tif (!f || do_config_from_file(store_aux, CONFIG_ORIGIN_FILE,\n -\t\t\t\t\t config_filename, config_filename,\n -\t\t\t\t\t f, NULL, 1)) {\n +\t\tif (git_config_from_file_with_options(store_aux,\n +\t\t\t\t\t\t config_filename,\n +\t\t\t\t\t\t &store, &opts)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n +\t\t\tfree(store.key);\n \t\t\tif (store.value_regex != NULL &&\n \t\t\t store.value_regex != CONFIG_REGEX_NONE) {\n \t\t\t\tregfree(store.value_regex);\n \t\t\t\tfree(store.value_regex);\n \t\t\t}\n \t\t\tret = CONFIG_INVALID_FILE;\n -\t\t\tif (f)\n -\t\t\t\tfclose(f);\n \t\t\tgoto out_free;\n -\t\t} else\n -\t\t\tfclose(f);\n +\t\t}\n \n +\t\tfree(store.key);\n \t\tif (store.value_regex != NULL &&\n \t\t store.value_regex != CONFIG_REGEX_NONE) {\n \t\t\tregfree(store.value_regex);\n @@ -2778,8 +2737,8 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t}\n \n \t\t/* if nothing to unset, or too many matches, error out */\n -\t\tif ((store.seen == 0 && value == NULL) ||\n -\t\t\t\t(store.seen > 1 && multi_replace == 0)) {\n +\t\tif ((store.seen_nr == 0 && value == NULL) ||\n +\t\t (store.seen_nr > 1 && multi_replace == 0)) {\n \t\t\tret = CONFIG_NOTHING_SET;\n \t\t\tgoto out_free;\n \t\t}\n @@ -2810,25 +2769,48 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tgoto out_free;\n \t\t}\n \n -\t\tif (store.seen == 0)\n -\t\t\tstore.seen = 1;\n +\t\tif (store.seen_nr == 0) {\n +\t\t\tif (!store.seen_alloc) {\n +\t\t\t\t/* Did not see key nor section */\n +\t\t\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n +\t\t\t\tstore.seen[0] = store.parsed_nr\n +\t\t\t\t\t- !!store.parsed_nr;\n +\t\t\t}\n +\t\t\tstore.seen_nr = 1;\n +\t\t}\n \n -\t\tfor (i = 0, copy_begin = 0; i < store.seen; i++) {\n -\t\t\tif (store.offset[i] == 0) {\n -\t\t\t\tstore.offset[i] = copy_end = contents_sz;\n -\t\t\t} else if (store.state != KEY_SEEN) {\n -\t\t\t\tcopy_end = store.offset[i];\n +\t\tfor (i = 0, copy_begin = 0; i < store.seen_nr; i++) {\n +\t\t\tsize_t replace_end;\n +\t\t\tint j = store.seen[i];\n +\n +\t\t\tnew_line = 0;\n +\t\t\tif (!store.key_seen) {\n +\t\t\t\tcopy_end = store.parsed[j].end;\n +\t\t\t\t/* include '\\n' when copying section header */\n +\t\t\t\tif (copy_end > 0 && copy_end < contents_sz &&\n +\t\t\t\t contents[copy_end - 1] != '\\n' &&\n +\t\t\t\t contents[copy_end] == '\\n')\n +\t\t\t\t\tcopy_end++;\n +\t\t\t\treplace_end = copy_end;\n \t\t\t} else {\n -\t\t\t\tcopy_end = find_beginning_of_line(\n -\t\t\t\t\tcontents, contents_sz,\n -\t\t\t\t\tstore.offset[i], &new_line);\n +\t\t\t\treplace_end = store.parsed[j].end;\n +\t\t\t\tcopy_end = store.parsed[j].begin;\n \t\t\t\tif (!value)\n -\t\t\t\t\tmaybe_remove_section(contents,\n -\t\t\t\t\t\t\t contents_sz,\n -\t\t\t\t\t\t\t section_name,\n -\t\t\t\t\t\t\t store.baselen,\n -\t\t\t\t\t\t\t ©_end, &i,\n -\t\t\t\t\t\t\t &new_line);\n +\t\t\t\t\tmaybe_remove_section(&store, contents,\n +\t\t\t\t\t\t\t ©_end,\n +\t\t\t\t\t\t\t &replace_end, &i);\n +\t\t\t\t/*\n +\t\t\t\t * Swallow preceding white-space on the same\n +\t\t\t\t * line.\n +\t\t\t\t */\n +\t\t\t\twhile (copy_end > 0 ) {\n +\t\t\t\t\tchar c = contents[copy_end - 1];\n +\n +\t\t\t\t\tif (isspace(c) && c != '\\n')\n +\t\t\t\t\t\tcopy_end--;\n +\t\t\t\t\telse\n +\t\t\t\t\t\tbreak;\n +\t\t\t\t}\n \t\t\t}\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n @@ -2843,16 +2825,16 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\t write_str_in_full(fd, \"\\n\") < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n -\t\t\tcopy_begin = store.offset[i];\n +\t\t\tcopy_begin = replace_end;\n \t\t}\n \n \t\t/* write the pair (value == NULL means unset) */\n \t\tif (value != NULL) {\n -\t\t\tif (store.state == START) {\n -\t\t\t\tif (write_section(fd, key) < 0)\n +\t\t\tif (!store.section_seen) {\n +\t\t\t\tif (write_section(fd, key, &store) < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n -\t\t\tif (write_pair(fd, key, value) < 0)\n +\t\t\tif (write_pair(fd, key, value, &store) < 0)\n \t\t\t\tgoto write_err_out;\n \t\t}\n \n @@ -2879,7 +2861,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n out_free:\n \trollback_lock_file(&lock);\n -\tfree(section_name);\n \tfree(filename_buf);\n \tif (contents)\n \t\tmunmap(contents, contents_sz);\n @@ -2977,7 +2958,8 @@ static int section_name_is_ok(const char *name)\n \n /* if new_name == NULL, the section is removed instead */\n static int git_config_copy_or_rename_section_in_file(const char *config_filename,\n -\t\t\t\t const char *old_name, const char *new_name, int copy)\n +\t\t\t\t const char *old_name,\n +\t\t\t\t const char *new_name, int copy)\n {\n \tint ret = 0, remove = 0;\n \tchar *filename_buf = NULL;\n @@ -2987,6 +2969,9 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \tFILE *config_file = NULL;\n \tstruct stat st;\n \tstruct strbuf copystr = STRBUF_INIT;\n +\tstruct config_set_store store;\n +\n +\tmemset(&store, 0, sizeof(store));\n \n \tif (new_name && !section_name_is_ok(new_name)) {\n \t\tret = error(\"invalid section name: %s\", new_name);\n @@ -3056,7 +3041,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \t\t\t\t}\n \t\t\t\tstore.baselen = strlen(new_name);\n \t\t\t\tif (!copy) {\n -\t\t\t\t\tif (write_section(out_fd, new_name) < 0) {\n +\t\t\t\t\tif (write_section(out_fd, new_name, &store) < 0) {\n \t\t\t\t\t\tret = write_error(get_lock_file_path(&lock));\n \t\t\t\t\t\tgoto out;\n \t\t\t\t\t}\n @@ -3077,7 +3062,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \t\t\t\t\t\toutput[0] = '\\t';\n \t\t\t\t\t}\n \t\t\t\t} else {\n -\t\t\t\t\tcopystr = store_create_section(new_name);\n +\t\t\t\t\tcopystr = store_create_section(new_name, &store);\n \t\t\t\t}\n \t\t\t}\n \t\t\tremove = 0;\n diff --git a/config.h b/config.h\n index ef70a9cac1e..5a2394daae2 100644\n --- a/config.h\n +++ b/config.h\n @@ -28,15 +28,40 @@ enum config_origin_type {\n \tCONFIG_ORIGIN_CMDLINE\n };\n \n +enum config_event_t {\n +\tCONFIG_EVENT_SECTION,\n +\tCONFIG_EVENT_ENTRY,\n +\tCONFIG_EVENT_WHITESPACE,\n +\tCONFIG_EVENT_COMMENT,\n +\tCONFIG_EVENT_EOF,\n +\tCONFIG_EVENT_ERROR\n +};\n +\n +/*\n + * The parser event function (if not NULL) is called with the event type and\n + * the begin/end offsets of the parsed elements.\n + *\n + * Note: for CONFIG_EVENT_ENTRY (i.e. config variables), the trailing newline\n + * character is considered part of the element.\n + */\n +typedef int (*config_parser_event_fn_t)(enum config_event_t type,\n +\t\t\t\t\tsize_t begin_offset, size_t end_offset,\n +\t\t\t\t\tvoid *event_fn_data);\n +\n struct config_options {\n \tunsigned int respect_includes : 1;\n \tconst char *commondir;\n \tconst char *git_dir;\n +\tconfig_parser_event_fn_t event_fn;\n +\tvoid *event_fn_data;\n };\n \n typedef int (*config_fn_t)(const char *, const char *, void *);\n extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n +extern int git_config_from_file_with_options(config_fn_t fn, const char *,\n +\t\t\t\t\t void *,\n +\t\t\t\t\t const struct config_options *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n extern int git_config_from_blob_oid(config_fn_t fn, const char *name,\n diff --git a/t/t1300-config.sh b/t/t1300-config.sh\n index 867397ae930..6d0e13020d1 100755\n --- a/t/t1300-config.sh\n +++ b/t/t1300-config.sh\n @@ -1643,4 +1643,25 @@ test_expect_success '--local requires a repo' '\n \ttest_expect_code 128 nongit git config --local foo.bar\n '\n \n +test_expect_success '--replace-all does not invent newlines' '\n +\tq_to_tab >.git/config <<-\\EOF &&\n +\t[abc]key\n +\tQkeepSection\n +\t[xyz]\n +\tQkey = 1\n +\t[abc]\n +\tQkey = a\n +\tEOF\n +\tq_to_tab >expect <<-\\EOF &&\n +\t[abc]\n +\tQkeepSection\n +\t[xyz]\n +\tQkey = 1\n +\t[abc]\n +\tQkey = b\n +\tEOF\n +\tgit config --replace-all abc.key b &&\n +\ttest_cmp .git/config expect\n +'\n +\n test_done\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n"},{"id":"343707","messageId":"083953bc0d5c7f497722a1b42b0e0be884fd1a4c.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 02/15] t1300: rename it to reflect that `repo-config` was deprecated","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:06Z","receivedAt":"2018-04-03T16:28:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Signed-off-by: Johannes Schindelin \n---\n t/{t1300-repo-config.sh => t1300-config.sh} | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n rename t/{t1300-repo-config.sh => t1300-config.sh} (100%)\n\ndiff --git a/t/t1300-repo-config.sh b/t/t1300-config.sh\nsimilarity index 100%\nrename from t/t1300-repo-config.sh\nrename to t/t1300-config.sh\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343708","messageId":"532243287ff81725502f09251d9f749ce1d34417.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 03/15] t1300: demonstrate that --replace-all can \"invent\" newlines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:10Z","receivedAt":"2018-04-03T16:28:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Signed-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 4f8e6f5fde3..cc417687e8d 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1611,4 +1611,25 @@ test_expect_success '--local requires a repo' '\n \ttest_expect_code 128 nongit git config --local foo.bar\n '\n \n+test_expect_failure '--replace-all does not invent newlines' '\n+\tq_to_tab >.git/config <<-\\EOF &&\n+\t[abc]key\n+\tQkeepSection\n+\t[xyz]\n+\tQkey = 1\n+\t[abc]\n+\tQkey = a\n+\tEOF\n+\tq_to_tab >expect <<-\\EOF &&\n+\t[abc]\n+\tQkeepSection\n+\t[xyz]\n+\tQkey = 1\n+\t[abc]\n+\tQkey = b\n+\tEOF\n+\tgit config --replace-all abc.key b &&\n+\ttest_cmp .git/config expect\n+'\n+\n test_done\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343709","messageId":"8264d7aa630a23167f8cffcb23462371f7511be8.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 04/15] config --replace-all: avoid extra line breaks","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:14Z","receivedAt":"2018-04-03T16:28:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"When replacing multiple config entries at once, we did not re-set the\nflag that indicates whether we need to insert a new-line before the new\nentry. As a consequence, an extra new-line was inserted under certain\ncircumstances.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 1 +\n t/t1300-config.sh | 2 +-\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/config.c b/config.c\nindex 5cc049aaef0..f10f8c6f52f 100644\n--- a/config.c\n+++ b/config.c\n@@ -2625,6 +2625,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tstore.seen = 1;\n \n \t\tfor (i = 0, copy_begin = 0; i < store.seen; i++) {\n+\t\t\tnew_line = 0;\n \t\t\tif (store.offset[i] == 0) {\n \t\t\t\tstore.offset[i] = copy_end = contents_sz;\n \t\t\t} else if (store.state != KEY_SEEN) {\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex cc417687e8d..aed12be492f 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1611,7 +1611,7 @@ test_expect_success '--local requires a repo' '\n \ttest_expect_code 128 nongit git config --local foo.bar\n '\n \n-test_expect_failure '--replace-all does not invent newlines' '\n+test_expect_success '--replace-all does not invent newlines' '\n \tq_to_tab >.git/config <<-\\EOF &&\n \t[abc]key\n \tQkeepSection\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343710","messageId":"4c8159bd9d77f7e21052e9c9b453f47b83847885.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 05/15] t1300: avoid relying on a bug","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:18Z","receivedAt":"2018-04-03T16:28:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The test case 'unset with cont. lines' relied on a bug that is about to\nbe fixed: it tests *explicitly* that removing the last entry from a\nconfig section leaves an *empty* section behind.\n\nLet's fix this test case not to rely on that behavior, simply by\npreventing the section from becoming empty.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex aed12be492f..7c0ee208dea 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -108,6 +108,7 @@ bar = foo\n [beta]\n baz = multiple \\\n lines\n+foo = bar\n EOF\n \n test_expect_success 'unset with cont. lines' '\n@@ -118,6 +119,7 @@ cat > expect <<\\EOF\n [alpha]\n bar = foo\n [beta]\n+foo = bar\n EOF\n \n test_expect_success 'unset with cont. lines is correct' 'test_cmp expect .git/config'\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343711","messageId":"5a4791e582116d83425918f9bb13201a44c91aa6.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 06/15] t1300: remove unreasonable expectation from TODO","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:22Z","receivedAt":"2018-04-03T16:28:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"In https://public-inbox.org/git/7vvc8alzat.fsf@alter.siamese.dyndns.org/\na reasonable patch was made quite a bit less so by changing a test case\ndemonstrating a bug to a test case that demonstrates that we ask for too\nmuch: the test case 'unsetting the last key in a section removes header'\nnow expects a future bug fix to be able to determine whether a free-form\ncomment above a section header refers to said section or not.\n\nRather than shooting for the stars (and not even getting off the\nground), let's start shooting for something obtainable and be reasonably\nconfident that we *can* get it.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 7c0ee208dea..187fc5b195f 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1413,7 +1413,7 @@ test_expect_success 'urlmatch with wildcard' '\n '\n \n # good section hygiene\n-test_expect_failure 'unsetting the last key in a section removes header' '\n+test_expect_failure '--unset last key removes section (except if commented)' '\n \tcat >.git/config <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n \t# a comment specific to this \"section\" section.\n@@ -1427,6 +1427,25 @@ test_expect_failure 'unsetting the last key in a section removes header' '\n \n \tcat >expect <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n+\t# a comment specific to this \"section\" section.\n+\t[section]\n+\t# some intervening lines\n+\t# that should also be dropped\n+\n+\t# please be careful when you update the above variable\n+\tEOF\n+\n+\tgit config --unset section.key &&\n+\ttest_cmp expect .git/config &&\n+\n+\tcat >.git/config <<-\\EOF &&\n+\t[section]\n+\tkey = value\n+\t[next-section]\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[next-section]\n \tEOF\n \n \tgit config --unset section.key &&\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343712","messageId":"331fbdb33d900c349b211dd763627a2d244eda4a.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 07/15] t1300: `--unset-all` can leave an empty section behind (bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:25Z","receivedAt":"2018-04-03T16:28:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"We already have a test demonstrating that removing the last entry from a\nconfig section fails to remove the section header of the now-empty\nsection.\n\nThe same can happen, of course, if we remove the last entries in one fell\nswoop. This is *also* a bug, and should be fixed at the same time.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 187fc5b195f..10b9bf4b088 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1452,6 +1452,17 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \ttest_cmp expect .git/config\n '\n \n+test_expect_failure '--unset-all removes section if empty & uncommented' '\n+\tcat >.git/config <<-\\EOF &&\n+\t[section]\n+\tkey = value1\n+\tkey = value2\n+\tEOF\n+\n+\tgit config --unset-all section.key &&\n+\ttest_line_count = 0 .git/config\n+'\n+\n test_expect_failure 'adding a key into an empty section reuses header' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343713","messageId":"a38025bcbc07b83b578ff2f1cde786bdca0c918c.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 08/15] config: introduce an optional event stream while parsing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:29Z","receivedAt":"2018-04-03T16:28:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This extends our config parser so that it can optionally produce an event\nstream via callback function, where it reports e.g. when a comment was\nparsed, or a section header, etc.\n\nThis parser will be used subsequently to handle the scenarios better where\nremoving config entries would make sections empty, or where a new entry\ncould be added to an already-existing, empty section.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 102 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--------\n config.h | 25 ++++++++++++++++\n 2 files changed, 115 insertions(+), 12 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex f10f8c6f52f..4cd745f6628 100644\n--- a/config.c\n+++ b/config.c\n@@ -653,7 +653,46 @@ static int get_base_var(struct strbuf *name)\n \t}\n }\n \n-static int git_parse_source(config_fn_t fn, void *data)\n+struct parse_event_data {\n+\tenum config_event_t previous_type;\n+\tsize_t previous_offset;\n+\tconst struct config_options *opts;\n+};\n+\n+static inline int do_event(enum config_event_t type,\n+\t\t\t struct parse_event_data *data)\n+{\n+\tsize_t offset;\n+\n+\tif (!data->opts || !data->opts->event_fn)\n+\t\treturn 0;\n+\n+\tif (type == CONFIG_EVENT_WHITESPACE &&\n+\t data->previous_type == type)\n+\t\treturn 0;\n+\n+\toffset = cf->do_ftell(cf);\n+\t/*\n+\t * At EOF, the parser always \"inserts\" an extra '\\n', therefore\n+\t * the end offset of the event is the current file position, otherwise\n+\t * we will already have advanced to the next event.\n+\t */\n+\tif (type != CONFIG_EVENT_EOF)\n+\t\toffset--;\n+\n+\tif (data->previous_type != CONFIG_EVENT_EOF &&\n+\t data->opts->event_fn(data->previous_type, data->previous_offset,\n+\t\t\t\t offset, data->opts->event_fn_data) < 0)\n+\t\treturn -1;\n+\n+\tdata->previous_type = type;\n+\tdata->previous_offset = offset;\n+\n+\treturn 0;\n+}\n+\n+static int git_parse_source(config_fn_t fn, void *data,\n+\t\t\t const struct config_options *opts)\n {\n \tint comment = 0;\n \tint baselen = 0;\n@@ -664,8 +703,15 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t/* U+FEFF Byte Order Mark in UTF8 */\n \tconst char *bomptr = utf8_bom;\n \n+\t/* For the parser event callback */\n+\tstruct parse_event_data event_data = {\n+\t\tCONFIG_EVENT_EOF, 0, opts\n+\t};\n+\n \tfor (;;) {\n-\t\tint c = get_next_char();\n+\t\tint c;\n+\n+\t\tc = get_next_char();\n \t\tif (bomptr && *bomptr) {\n \t\t\t/* We are at the file beginning; skip UTF8-encoded BOM\n \t\t\t * if present. Sane editors won't put this in on their\n@@ -682,18 +728,33 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t\t}\n \t\t}\n \t\tif (c == '\\n') {\n-\t\t\tif (cf->eof)\n+\t\t\tif (cf->eof) {\n+\t\t\t\tif (do_event(CONFIG_EVENT_EOF, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n \t\t\t\treturn 0;\n+\t\t\t}\n+\t\t\tif (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)\n+\t\t\t\treturn -1;\n \t\t\tcomment = 0;\n \t\t\tcontinue;\n \t\t}\n-\t\tif (comment || isspace(c))\n+\t\tif (comment)\n \t\t\tcontinue;\n+\t\tif (isspace(c)) {\n+\t\t\tif (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (c == '#' || c == ';') {\n+\t\t\tif (do_event(CONFIG_EVENT_COMMENT, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n \t\t\tcomment = 1;\n \t\t\tcontinue;\n \t\t}\n \t\tif (c == '[') {\n+\t\t\tif (do_event(CONFIG_EVENT_SECTION, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n+\n \t\t\t/* Reset prior to determining a new stem */\n \t\t\tstrbuf_reset(var);\n \t\t\tif (get_base_var(var) < 0 || var->len < 1)\n@@ -704,6 +765,10 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t}\n \t\tif (!isalpha(c))\n \t\t\tbreak;\n+\n+\t\tif (do_event(CONFIG_EVENT_ENTRY, &event_data) < 0)\n+\t\t\treturn -1;\n+\n \t\t/*\n \t\t * Truncate the var name back to the section header\n \t\t * stem prior to grabbing the suffix part of the name\n@@ -715,6 +780,9 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t\tbreak;\n \t}\n \n+\tif (do_event(CONFIG_EVENT_ERROR, &event_data) < 0)\n+\t\treturn -1;\n+\n \tswitch (cf->origin_type) {\n \tcase CONFIG_ORIGIN_BLOB:\n \t\terror_msg = xstrfmt(_(\"bad config line %d in blob %s\"),\n@@ -1398,7 +1466,8 @@ int git_default_config(const char *var, const char *value, void *dummy)\n * fgetc, ungetc, ftell of top need to be initialized before calling\n * this function.\n */\n-static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n+static int do_config_from(struct config_source *top, config_fn_t fn, void *data,\n+\t\t\t const struct config_options *opts)\n {\n \tint ret;\n \n@@ -1410,7 +1479,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n \tstrbuf_init(&top->var, 1024);\n \tcf = top;\n \n-\tret = git_parse_source(fn, data);\n+\tret = git_parse_source(fn, data, opts);\n \n \t/* pop config-file parsing state stack */\n \tstrbuf_release(&top->value);\n@@ -1423,7 +1492,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n static int do_config_from_file(config_fn_t fn,\n \t\tconst enum config_origin_type origin_type,\n \t\tconst char *name, const char *path, FILE *f,\n-\t\tvoid *data)\n+\t\tvoid *data, const struct config_options *opts)\n {\n \tstruct config_source top;\n \n@@ -1436,15 +1505,18 @@ static int do_config_from_file(config_fn_t fn,\n \ttop.do_ungetc = config_file_ungetc;\n \ttop.do_ftell = config_file_ftell;\n \n-\treturn do_config_from(&top, fn, data);\n+\treturn do_config_from(&top, fn, data, opts);\n }\n \n static int git_config_from_stdin(config_fn_t fn, void *data)\n {\n-\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data);\n+\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin,\n+\t\t\t\t data, NULL);\n }\n \n-int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n+int git_config_from_file_with_options(config_fn_t fn, const char *filename,\n+\t\t\t\t void *data,\n+\t\t\t\t const struct config_options *opts)\n {\n \tint ret = -1;\n \tFILE *f;\n@@ -1452,13 +1524,19 @@ int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n \tf = fopen_or_warn(filename, \"r\");\n \tif (f) {\n \t\tflockfile(f);\n-\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data);\n+\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename,\n+\t\t\t\t\t filename, f, data, opts);\n \t\tfunlockfile(f);\n \t\tfclose(f);\n \t}\n \treturn ret;\n }\n \n+int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n+{\n+\treturn git_config_from_file_with_options(fn, filename, data, NULL);\n+}\n+\n int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_type,\n \t\t\tconst char *name, const char *buf, size_t len, void *data)\n {\n@@ -1475,7 +1553,7 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \ttop.do_ungetc = config_buf_ungetc;\n \ttop.do_ftell = config_buf_ftell;\n \n-\treturn do_config_from(&top, fn, data);\n+\treturn do_config_from(&top, fn, data, NULL);\n }\n \n int git_config_from_blob_oid(config_fn_t fn,\ndiff --git a/config.h b/config.h\nindex ef70a9cac1e..5a2394daae2 100644\n--- a/config.h\n+++ b/config.h\n@@ -28,15 +28,40 @@ enum config_origin_type {\n \tCONFIG_ORIGIN_CMDLINE\n };\n \n+enum config_event_t {\n+\tCONFIG_EVENT_SECTION,\n+\tCONFIG_EVENT_ENTRY,\n+\tCONFIG_EVENT_WHITESPACE,\n+\tCONFIG_EVENT_COMMENT,\n+\tCONFIG_EVENT_EOF,\n+\tCONFIG_EVENT_ERROR\n+};\n+\n+/*\n+ * The parser event function (if not NULL) is called with the event type and\n+ * the begin/end offsets of the parsed elements.\n+ *\n+ * Note: for CONFIG_EVENT_ENTRY (i.e. config variables), the trailing newline\n+ * character is considered part of the element.\n+ */\n+typedef int (*config_parser_event_fn_t)(enum config_event_t type,\n+\t\t\t\t\tsize_t begin_offset, size_t end_offset,\n+\t\t\t\t\tvoid *event_fn_data);\n+\n struct config_options {\n \tunsigned int respect_includes : 1;\n \tconst char *commondir;\n \tconst char *git_dir;\n+\tconfig_parser_event_fn_t event_fn;\n+\tvoid *event_fn_data;\n };\n \n typedef int (*config_fn_t)(const char *, const char *, void *);\n extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n+extern int git_config_from_file_with_options(config_fn_t fn, const char *,\n+\t\t\t\t\t void *,\n+\t\t\t\t\t const struct config_options *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n extern int git_config_from_blob_oid(config_fn_t fn, const char *name,\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343714","messageId":"a9689c880f9226c236aec58f8036740ff39efbb8.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 09/15] config: avoid using the global variable `store`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:34Z","receivedAt":"2018-04-03T16:28:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"It is much easier to reason about, when the config code to set/unset\nvariables or to remove/rename sections does not rely on a global (or\nfile-local) variable.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 119 +++++++++++++++++++++++++++++++++++----------------------------\n 1 file changed, 66 insertions(+), 53 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 4cd745f6628..90ae71cb905 100644\n--- a/config.c\n+++ b/config.c\n@@ -2297,7 +2297,7 @@ void git_die_config(const char *key, const char *err, ...)\n * Find all the stuff for git_config_set() below.\n */\n \n-static struct {\n+struct config_set_store {\n \tint baselen;\n \tchar *key;\n \tint do_not_match;\n@@ -2307,56 +2307,58 @@ static struct {\n \tunsigned int offset_alloc;\n \tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n \tunsigned int seen;\n-} store;\n+};\n \n-static int matches(const char *key, const char *value)\n+static int matches(const char *key, const char *value,\n+\t\t const struct config_set_store *store)\n {\n-\tif (strcmp(key, store.key))\n+\tif (strcmp(key, store->key))\n \t\treturn 0; /* not ours */\n-\tif (!store.value_regex)\n+\tif (!store->value_regex)\n \t\treturn 1; /* always matches */\n-\tif (store.value_regex == CONFIG_REGEX_NONE)\n+\tif (store->value_regex == CONFIG_REGEX_NONE)\n \t\treturn 0; /* never matches */\n \n-\treturn store.do_not_match ^\n-\t\t(value && !regexec(store.value_regex, value, 0, NULL, 0));\n+\treturn store->do_not_match ^\n+\t\t(value && !regexec(store->value_regex, value, 0, NULL, 0));\n }\n \n static int store_aux(const char *key, const char *value, void *cb)\n {\n \tconst char *ep;\n \tsize_t section_len;\n+\tstruct config_set_store *store = cb;\n \n-\tswitch (store.state) {\n+\tswitch (store->state) {\n \tcase KEY_SEEN:\n-\t\tif (matches(key, value)) {\n-\t\t\tif (store.seen == 1 && store.multi_replace == 0) {\n+\t\tif (matches(key, value, store)) {\n+\t\t\tif (store->seen == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n \t\t\t}\n \n-\t\t\tALLOC_GROW(store.offset, store.seen + 1,\n-\t\t\t\t store.offset_alloc);\n+\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n+\t\t\t\t store->offset_alloc);\n \n-\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n-\t\t\tstore.seen++;\n+\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\t\tstore->seen++;\n \t\t}\n \t\tbreak;\n \tcase SECTION_SEEN:\n \t\t/*\n-\t\t * What we are looking for is in store.key (both\n+\t\t * What we are looking for is in store->key (both\n \t\t * section and var), and its section part is baselen\n \t\t * long. We found key (again, both section and var).\n \t\t * We would want to know if this key is in the same\n \t\t * section as what we are looking for. We already\n \t\t * know we are in the same section as what should\n-\t\t * hold store.key.\n+\t\t * hold store->key.\n \t\t */\n \t\tep = strrchr(key, '.');\n \t\tsection_len = ep - key;\n \n-\t\tif ((section_len != store.baselen) ||\n-\t\t memcmp(key, store.key, section_len+1)) {\n-\t\t\tstore.state = SECTION_END_SEEN;\n+\t\tif ((section_len != store->baselen) ||\n+\t\t memcmp(key, store->key, section_len+1)) {\n+\t\t\tstore->state = SECTION_END_SEEN;\n \t\t\tbreak;\n \t\t}\n \n@@ -2364,26 +2366,27 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t * Do not increment matches: this is no match, but we\n \t\t * just made sure we are in the desired section.\n \t\t */\n-\t\tALLOC_GROW(store.offset, store.seen + 1,\n-\t\t\t store.offset_alloc);\n-\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n+\t\tALLOC_GROW(store->offset, store->seen + 1,\n+\t\t\t store->offset_alloc);\n+\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n \t\t/* fallthru */\n \tcase SECTION_END_SEEN:\n \tcase START:\n-\t\tif (matches(key, value)) {\n-\t\t\tALLOC_GROW(store.offset, store.seen + 1,\n-\t\t\t\t store.offset_alloc);\n-\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n-\t\t\tstore.state = KEY_SEEN;\n-\t\t\tstore.seen++;\n+\t\tif (matches(key, value, store)) {\n+\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n+\t\t\t\t store->offset_alloc);\n+\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\t\tstore->state = KEY_SEEN;\n+\t\t\tstore->seen++;\n \t\t} else {\n-\t\t\tif (strrchr(key, '.') - key == store.baselen &&\n-\t\t\t !strncmp(key, store.key, store.baselen)) {\n-\t\t\t\t\tstore.state = SECTION_SEEN;\n-\t\t\t\t\tALLOC_GROW(store.offset,\n-\t\t\t\t\t\t store.seen + 1,\n-\t\t\t\t\t\t store.offset_alloc);\n-\t\t\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n+\t\t\tif (strrchr(key, '.') - key == store->baselen &&\n+\t\t\t !strncmp(key, store->key, store->baselen)) {\n+\t\t\t\t\tstore->state = SECTION_SEEN;\n+\t\t\t\t\tALLOC_GROW(store->offset,\n+\t\t\t\t\t\t store->seen + 1,\n+\t\t\t\t\t\t store->offset_alloc);\n+\t\t\t\t\tstore->offset[store->seen] =\n+\t\t\t\t\t\tcf->do_ftell(cf);\n \t\t\t}\n \t\t}\n \t}\n@@ -2398,31 +2401,33 @@ static int write_error(const char *filename)\n \treturn 4;\n }\n \n-static struct strbuf store_create_section(const char *key)\n+static struct strbuf store_create_section(const char *key,\n+\t\t\t\t\t const struct config_set_store *store)\n {\n \tconst char *dot;\n \tint i;\n \tstruct strbuf sb = STRBUF_INIT;\n \n-\tdot = memchr(key, '.', store.baselen);\n+\tdot = memchr(key, '.', store->baselen);\n \tif (dot) {\n \t\tstrbuf_addf(&sb, \"[%.*s \\\"\", (int)(dot - key), key);\n-\t\tfor (i = dot - key + 1; i < store.baselen; i++) {\n+\t\tfor (i = dot - key + 1; i < store->baselen; i++) {\n \t\t\tif (key[i] == '\"' || key[i] == '\\\\')\n \t\t\t\tstrbuf_addch(&sb, '\\\\');\n \t\t\tstrbuf_addch(&sb, key[i]);\n \t\t}\n \t\tstrbuf_addstr(&sb, \"\\\"]\\n\");\n \t} else {\n-\t\tstrbuf_addf(&sb, \"[%.*s]\\n\", store.baselen, key);\n+\t\tstrbuf_addf(&sb, \"[%.*s]\\n\", store->baselen, key);\n \t}\n \n \treturn sb;\n }\n \n-static ssize_t write_section(int fd, const char *key)\n+static ssize_t write_section(int fd, const char *key,\n+\t\t\t const struct config_set_store *store)\n {\n-\tstruct strbuf sb = store_create_section(key);\n+\tstruct strbuf sb = store_create_section(key, store);\n \tssize_t ret;\n \n \tret = write_in_full(fd, sb.buf, sb.len);\n@@ -2431,11 +2436,12 @@ static ssize_t write_section(int fd, const char *key)\n \treturn ret;\n }\n \n-static ssize_t write_pair(int fd, const char *key, const char *value)\n+static ssize_t write_pair(int fd, const char *key, const char *value,\n+\t\t\t const struct config_set_store *store)\n {\n \tint i;\n \tssize_t ret;\n-\tint length = strlen(key + store.baselen + 1);\n+\tint length = strlen(key + store->baselen + 1);\n \tconst char *quote = \"\";\n \tstruct strbuf sb = STRBUF_INIT;\n \n@@ -2455,7 +2461,7 @@ static ssize_t write_pair(int fd, const char *key, const char *value)\n \t\tquote = \"\\\"\";\n \n \tstrbuf_addf(&sb, \"\\t%.*s = %s\",\n-\t\t length, key + store.baselen + 1, quote);\n+\t\t length, key + store->baselen + 1, quote);\n \n \tfor (i = 0; value[i]; i++)\n \t\tswitch (value[i]) {\n@@ -2565,6 +2571,9 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tchar *filename_buf = NULL;\n \tchar *contents = NULL;\n \tsize_t contents_sz;\n+\tstruct config_set_store store;\n+\n+\tmemset(&store, 0, sizeof(store));\n \n \t/* parse-key returns negative; flip the sign to feed exit(3) */\n \tret = 0 - git_config_parse_key(key, &store.key, &store.baselen);\n@@ -2607,8 +2616,8 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t}\n \n \t\tstore.key = (char *)key;\n-\t\tif (write_section(fd, key) < 0 ||\n-\t\t write_pair(fd, key, value) < 0)\n+\t\tif (write_section(fd, key, &store) < 0 ||\n+\t\t write_pair(fd, key, value, &store) < 0)\n \t\t\tgoto write_err_out;\n \t} else {\n \t\tstruct stat st;\n@@ -2647,7 +2656,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t * As a side effect, we make sure to transform only a valid\n \t\t * existing config file.\n \t\t */\n-\t\tif (git_config_from_file(store_aux, config_filename, NULL)) {\n+\t\tif (git_config_from_file(store_aux, config_filename, &store)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n \t\t\tfree(store.key);\n \t\t\tif (store.value_regex != NULL &&\n@@ -2731,10 +2740,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t/* write the pair (value == NULL means unset) */\n \t\tif (value != NULL) {\n \t\t\tif (store.state == START) {\n-\t\t\t\tif (write_section(fd, key) < 0)\n+\t\t\t\tif (write_section(fd, key, &store) < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-\t\t\tif (write_pair(fd, key, value) < 0)\n+\t\t\tif (write_pair(fd, key, value, &store) < 0)\n \t\t\t\tgoto write_err_out;\n \t\t}\n \n@@ -2858,7 +2867,8 @@ static int section_name_is_ok(const char *name)\n \n /* if new_name == NULL, the section is removed instead */\n static int git_config_copy_or_rename_section_in_file(const char *config_filename,\n-\t\t\t\t const char *old_name, const char *new_name, int copy)\n+\t\t\t\t const char *old_name,\n+\t\t\t\t const char *new_name, int copy)\n {\n \tint ret = 0, remove = 0;\n \tchar *filename_buf = NULL;\n@@ -2868,6 +2878,9 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \tFILE *config_file = NULL;\n \tstruct stat st;\n \tstruct strbuf copystr = STRBUF_INIT;\n+\tstruct config_set_store store;\n+\n+\tmemset(&store, 0, sizeof(store));\n \n \tif (new_name && !section_name_is_ok(new_name)) {\n \t\tret = error(\"invalid section name: %s\", new_name);\n@@ -2937,7 +2950,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \t\t\t\t}\n \t\t\t\tstore.baselen = strlen(new_name);\n \t\t\t\tif (!copy) {\n-\t\t\t\t\tif (write_section(out_fd, new_name) < 0) {\n+\t\t\t\t\tif (write_section(out_fd, new_name, &store) < 0) {\n \t\t\t\t\t\tret = write_error(get_lock_file_path(&lock));\n \t\t\t\t\t\tgoto out;\n \t\t\t\t\t}\n@@ -2958,7 +2971,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \t\t\t\t\t\toutput[0] = '\\t';\n \t\t\t\t\t}\n \t\t\t\t} else {\n-\t\t\t\t\tcopystr = store_create_section(new_name);\n+\t\t\t\t\tcopystr = store_create_section(new_name, &store);\n \t\t\t\t}\n \t\t\t}\n \t\t\tremove = 0;\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343715","messageId":"d58727931e42cd7c09a4643ca48046582088731d.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 10/15] config_set_store: rename some fields for consistency","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:38Z","receivedAt":"2018-04-03T16:28:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The `seen` field is the actual length of the `offset` array, and the\n`offset_alloc` field records what was allocated (to avoid resizing\nwherever `seen` has to be incremented).\n\nElsewhere, we use the convention `name` for the array, where `name` is\ndescriptive enough to guess its purpose, `name_nr` for the actual length\nand `name_alloc` to record the maximum length without needing to resize.\n\nLet's make the names of the fields in question consistent with that\nconvention.\n\nThis will also help with the next steps where we will let the\ngit_config_set() machinery use the config event stream that we just\nintroduced.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 63 +++++++++++++++++++++++++++++++--------------------------------\n 1 file changed, 31 insertions(+), 32 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 90ae71cb905..b73b48b5650 100644\n--- a/config.c\n+++ b/config.c\n@@ -2303,10 +2303,9 @@ struct config_set_store {\n \tint do_not_match;\n \tregex_t *value_regex;\n \tint multi_replace;\n-\tsize_t *offset;\n-\tunsigned int offset_alloc;\n+\tsize_t *seen;\n+\tunsigned int seen_nr, seen_alloc;\n \tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n-\tunsigned int seen;\n };\n \n static int matches(const char *key, const char *value,\n@@ -2332,15 +2331,15 @@ static int store_aux(const char *key, const char *value, void *cb)\n \tswitch (store->state) {\n \tcase KEY_SEEN:\n \t\tif (matches(key, value, store)) {\n-\t\t\tif (store->seen == 1 && store->multi_replace == 0) {\n+\t\t\tif (store->seen_nr == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n \t\t\t}\n \n-\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n-\t\t\t\t store->offset_alloc);\n+\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t\t store->seen_alloc);\n \n-\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n-\t\t\tstore->seen++;\n+\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n+\t\t\tstore->seen_nr++;\n \t\t}\n \t\tbreak;\n \tcase SECTION_SEEN:\n@@ -2366,26 +2365,26 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t * Do not increment matches: this is no match, but we\n \t\t * just made sure we are in the desired section.\n \t\t */\n-\t\tALLOC_GROW(store->offset, store->seen + 1,\n-\t\t\t store->offset_alloc);\n-\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t store->seen_alloc);\n+\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n \t\t/* fallthru */\n \tcase SECTION_END_SEEN:\n \tcase START:\n \t\tif (matches(key, value, store)) {\n-\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n-\t\t\t\t store->offset_alloc);\n-\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t\t store->seen_alloc);\n+\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n \t\t\tstore->state = KEY_SEEN;\n-\t\t\tstore->seen++;\n+\t\t\tstore->seen_nr++;\n \t\t} else {\n \t\t\tif (strrchr(key, '.') - key == store->baselen &&\n \t\t\t !strncmp(key, store->key, store->baselen)) {\n \t\t\t\t\tstore->state = SECTION_SEEN;\n-\t\t\t\t\tALLOC_GROW(store->offset,\n-\t\t\t\t\t\t store->seen + 1,\n-\t\t\t\t\t\t store->offset_alloc);\n-\t\t\t\t\tstore->offset[store->seen] =\n+\t\t\t\t\tALLOC_GROW(store->seen,\n+\t\t\t\t\t\t store->seen_nr + 1,\n+\t\t\t\t\t\t store->seen_alloc);\n+\t\t\t\t\tstore->seen[store->seen_nr] =\n \t\t\t\t\t\tcf->do_ftell(cf);\n \t\t\t}\n \t\t}\n@@ -2645,10 +2644,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t}\n \t\t}\n \n-\t\tALLOC_GROW(store.offset, 1, store.offset_alloc);\n-\t\tstore.offset[0] = 0;\n+\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n+\t\tstore.seen[0] = 0;\n \t\tstore.state = START;\n-\t\tstore.seen = 0;\n+\t\tstore.seen_nr = 0;\n \n \t\t/*\n \t\t * After this, store.offset will contain the *end* offset\n@@ -2676,8 +2675,8 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t}\n \n \t\t/* if nothing to unset, or too many matches, error out */\n-\t\tif ((store.seen == 0 && value == NULL) ||\n-\t\t\t\t(store.seen > 1 && multi_replace == 0)) {\n+\t\tif ((store.seen_nr == 0 && value == NULL) ||\n+\t\t (store.seen_nr > 1 && multi_replace == 0)) {\n \t\t\tret = CONFIG_NOTHING_SET;\n \t\t\tgoto out_free;\n \t\t}\n@@ -2708,19 +2707,19 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tgoto out_free;\n \t\t}\n \n-\t\tif (store.seen == 0)\n-\t\t\tstore.seen = 1;\n+\t\tif (store.seen_nr == 0)\n+\t\t\tstore.seen_nr = 1;\n \n-\t\tfor (i = 0, copy_begin = 0; i < store.seen; i++) {\n+\t\tfor (i = 0, copy_begin = 0; i < store.seen_nr; i++) {\n \t\t\tnew_line = 0;\n-\t\t\tif (store.offset[i] == 0) {\n-\t\t\t\tstore.offset[i] = copy_end = contents_sz;\n+\t\t\tif (store.seen[i] == 0) {\n+\t\t\t\tstore.seen[i] = copy_end = contents_sz;\n \t\t\t} else if (store.state != KEY_SEEN) {\n-\t\t\t\tcopy_end = store.offset[i];\n+\t\t\t\tcopy_end = store.seen[i];\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n \t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.offset[i], &new_line);\n+\t\t\t\t\tstore.seen[i], &new_line);\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n@@ -2734,7 +2733,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\t write_str_in_full(fd, \"\\n\") < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-\t\t\tcopy_begin = store.offset[i];\n+\t\t\tcopy_begin = store.seen[i];\n \t\t}\n \n \t\t/* write the pair (value == NULL means unset) */\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343716","messageId":"4b3eba873df4355d7a77dfbd74963c636b1ba08e.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 11/15] git_config_set: do not use a state machine","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:42Z","receivedAt":"2018-04-03T16:28:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"While a neat theoretical construct, state machines are hard to read. In\nthis instance, it does not even make a whole lot of sense because we are\nmore interested in flags, anyway: has the section been seen? Has the key\nbeen seen? Does the current section match the key we are looking for?\n\nBesides, the state `SECTION_SEEN` was named in a misleading way: it did\nnot indicate that we saw the section matching the key we are looking\nfor, but it instead indicated that we are *currently* in that section.\n\nLet's just replace the state machine logic by clear and obvious flags.\n\nThis will also make it easier to review the upcoming patches to use the\nnewly-introduced `event_fn` callback of the config parser.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 59 +++++++++++++++++++++++++++++------------------------------\n 1 file changed, 29 insertions(+), 30 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex b73b48b5650..84e8f7ffeb8 100644\n--- a/config.c\n+++ b/config.c\n@@ -2305,7 +2305,7 @@ struct config_set_store {\n \tint multi_replace;\n \tsize_t *seen;\n \tunsigned int seen_nr, seen_alloc;\n-\tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n+\tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n };\n \n static int matches(const char *key, const char *value,\n@@ -2328,8 +2328,7 @@ static int store_aux(const char *key, const char *value, void *cb)\n \tsize_t section_len;\n \tstruct config_set_store *store = cb;\n \n-\tswitch (store->state) {\n-\tcase KEY_SEEN:\n+\tif (store->key_seen) {\n \t\tif (matches(key, value, store)) {\n \t\t\tif (store->seen_nr == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n@@ -2341,8 +2340,8 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n \t\t\tstore->seen_nr++;\n \t\t}\n-\t\tbreak;\n-\tcase SECTION_SEEN:\n+\t\treturn 0;\n+\t} else if (store->is_keys_section) {\n \t\t/*\n \t\t * What we are looking for is in store->key (both\n \t\t * section and var), and its section part is baselen\n@@ -2357,10 +2356,9 @@ static int store_aux(const char *key, const char *value, void *cb)\n \n \t\tif ((section_len != store->baselen) ||\n \t\t memcmp(key, store->key, section_len+1)) {\n-\t\t\tstore->state = SECTION_END_SEEN;\n-\t\t\tbreak;\n+\t\t\tstore->is_keys_section = 0;\n+\t\t\treturn 0;\n \t\t}\n-\n \t\t/*\n \t\t * Do not increment matches: this is no match, but we\n \t\t * just made sure we are in the desired section.\n@@ -2368,27 +2366,29 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n \t\t\t store->seen_alloc);\n \t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t\t/* fallthru */\n-\tcase SECTION_END_SEEN:\n-\tcase START:\n-\t\tif (matches(key, value, store)) {\n-\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n-\t\t\t\t store->seen_alloc);\n-\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t\t\tstore->state = KEY_SEEN;\n-\t\t\tstore->seen_nr++;\n-\t\t} else {\n-\t\t\tif (strrchr(key, '.') - key == store->baselen &&\n-\t\t\t !strncmp(key, store->key, store->baselen)) {\n-\t\t\t\t\tstore->state = SECTION_SEEN;\n-\t\t\t\t\tALLOC_GROW(store->seen,\n-\t\t\t\t\t\t store->seen_nr + 1,\n-\t\t\t\t\t\t store->seen_alloc);\n-\t\t\t\t\tstore->seen[store->seen_nr] =\n-\t\t\t\t\t\tcf->do_ftell(cf);\n-\t\t\t}\n+\t}\n+\n+\tif (matches(key, value, store)) {\n+\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t store->seen_alloc);\n+\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n+\t\tstore->seen_nr++;\n+\t\tstore->key_seen = 1;\n+\t\tstore->section_seen = 1;\n+\t\tstore->is_keys_section = 1;\n+\t} else {\n+\t\tif (strrchr(key, '.') - key == store->baselen &&\n+\t\t !strncmp(key, store->key, store->baselen)) {\n+\t\t\t\tstore->section_seen = 1;\n+\t\t\t\tstore->is_keys_section = 1;\n+\t\t\t\tALLOC_GROW(store->seen,\n+\t\t\t\t\t store->seen_nr + 1,\n+\t\t\t\t\t store->seen_alloc);\n+\t\t\t\tstore->seen[store->seen_nr] =\n+\t\t\t\t\tcf->do_ftell(cf);\n \t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -2646,7 +2646,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n \t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n \t\tstore.seen[0] = 0;\n-\t\tstore.state = START;\n \t\tstore.seen_nr = 0;\n \n \t\t/*\n@@ -2714,7 +2713,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tnew_line = 0;\n \t\t\tif (store.seen[i] == 0) {\n \t\t\t\tstore.seen[i] = copy_end = contents_sz;\n-\t\t\t} else if (store.state != KEY_SEEN) {\n+\t\t\t} else if (!store.key_seen) {\n \t\t\t\tcopy_end = store.seen[i];\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n@@ -2738,7 +2737,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n \t\t/* write the pair (value == NULL means unset) */\n \t\tif (value != NULL) {\n-\t\t\tif (store.state == START) {\n+\t\t\tif (!store.section_seen) {\n \t\t\t\tif (write_section(fd, key, &store) < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343717","messageId":"f65d03849854bb7e46041225e9f7bb4ba8f043f7.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 12/15] git_config_set: make use of the config parser's event stream","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:46Z","receivedAt":"2018-04-03T16:29:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"In the recent commit with the title \"config: introduce an optional event\nstream while parsing\", we introduced an optional callback to keep track\nof the config parser's events \"comment\", \"white-space\", \"section header\"\nand \"entry\".\n\nOne motivation for this feature was to make use of it in the code that\nedits the config. And this commit makes it so.\n\nNote: this patch changes the meaning of the `seen` array that records\nwhether we saw the config entry that is to be edited: previously, it\ncontained the end offset of the found entry. Now, we introduce a new\narray `parsed` that keeps a record of *all* config parser events (with\nbegin/end offsets), and the items in the `seen` array now point into the\n`parsed` array.\n\nThere are two reasons why we do it this way:\n\n1. To keep the implementation simple, the config parser's event stream\n reports the event only after the config callback was called, so we\n would not receive the begin offset otherwise.\n\n2. In the following patches, we will re-use the `parsed` array to fix two\n long-standing bugs related to empty sections.\n\nNote that this also makes the code more robust with respect to finding the\nbegin offset of the part(s) of the config file to be edited, as we no\nlonger back-track to find the beginning of the line.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 170 ++++++++++++++++++++++++++++++---------------------------------\n 1 file changed, 81 insertions(+), 89 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 84e8f7ffeb8..345b1d2f140 100644\n--- a/config.c\n+++ b/config.c\n@@ -2303,8 +2303,11 @@ struct config_set_store {\n \tint do_not_match;\n \tregex_t *value_regex;\n \tint multi_replace;\n-\tsize_t *seen;\n-\tunsigned int seen_nr, seen_alloc;\n+\tstruct {\n+\t\tsize_t begin, end;\n+\t\tenum config_event_t type;\n+\t} *parsed;\n+\tunsigned int parsed_nr, parsed_alloc, *seen, seen_nr, seen_alloc;\n \tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n };\n \n@@ -2322,10 +2325,31 @@ static int matches(const char *key, const char *value,\n \t\t(value && !regexec(store->value_regex, value, 0, NULL, 0));\n }\n \n+static int store_aux_event(enum config_event_t type,\n+\t\t\t size_t begin, size_t end, void *data)\n+{\n+\tstruct config_set_store *store = data;\n+\n+\tALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);\n+\tstore->parsed[store->parsed_nr].begin = begin;\n+\tstore->parsed[store->parsed_nr].end = end;\n+\tstore->parsed[store->parsed_nr].type = type;\n+\tstore->parsed_nr++;\n+\n+\tif (type == CONFIG_EVENT_SECTION) {\n+\t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n+\t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n+\n+\t\t/* Is this the section we were looking for? */\n+\t\tstore->is_keys_section = cf->var.len - 1 == store->baselen &&\n+\t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int store_aux(const char *key, const char *value, void *cb)\n {\n-\tconst char *ep;\n-\tsize_t section_len;\n \tstruct config_set_store *store = cb;\n \n \tif (store->key_seen) {\n@@ -2337,55 +2361,21 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n \t\t\t\t store->seen_alloc);\n \n-\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n+\t\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n \t\t\tstore->seen_nr++;\n \t\t}\n-\t\treturn 0;\n \t} else if (store->is_keys_section) {\n \t\t/*\n-\t\t * What we are looking for is in store->key (both\n-\t\t * section and var), and its section part is baselen\n-\t\t * long. We found key (again, both section and var).\n-\t\t * We would want to know if this key is in the same\n-\t\t * section as what we are looking for. We already\n-\t\t * know we are in the same section as what should\n-\t\t * hold store->key.\n+\t\t * Do not increment matches yet: this may not be a match, but we\n+\t\t * are in the desired section.\n \t\t */\n-\t\tep = strrchr(key, '.');\n-\t\tsection_len = ep - key;\n-\n-\t\tif ((section_len != store->baselen) ||\n-\t\t memcmp(key, store->key, section_len+1)) {\n-\t\t\tstore->is_keys_section = 0;\n-\t\t\treturn 0;\n-\t\t}\n-\t\t/*\n-\t\t * Do not increment matches: this is no match, but we\n-\t\t * just made sure we are in the desired section.\n-\t\t */\n-\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n-\t\t\t store->seen_alloc);\n-\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t}\n-\n-\tif (matches(key, value, store)) {\n-\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n-\t\t\t store->seen_alloc);\n-\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t\tstore->seen_nr++;\n-\t\tstore->key_seen = 1;\n+\t\tALLOC_GROW(store->seen, store->seen_nr + 1, store->seen_alloc);\n+\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n \t\tstore->section_seen = 1;\n-\t\tstore->is_keys_section = 1;\n-\t} else {\n-\t\tif (strrchr(key, '.') - key == store->baselen &&\n-\t\t !strncmp(key, store->key, store->baselen)) {\n-\t\t\t\tstore->section_seen = 1;\n-\t\t\t\tstore->is_keys_section = 1;\n-\t\t\t\tALLOC_GROW(store->seen,\n-\t\t\t\t\t store->seen_nr + 1,\n-\t\t\t\t\t store->seen_alloc);\n-\t\t\t\tstore->seen[store->seen_nr] =\n-\t\t\t\t\tcf->do_ftell(cf);\n+\n+\t\tif (matches(key, value, store)) {\n+\t\t\tstore->seen_nr++;\n+\t\t\tstore->key_seen = 1;\n \t\t}\n \t}\n \n@@ -2486,32 +2476,6 @@ static ssize_t write_pair(int fd, const char *key, const char *value,\n \treturn ret;\n }\n \n-static ssize_t find_beginning_of_line(const char *contents, size_t size,\n-\tsize_t offset_, int *found_bracket)\n-{\n-\tsize_t equal_offset = size, bracket_offset = size;\n-\tssize_t offset;\n-\n-contline:\n-\tfor (offset = offset_-2; offset > 0\n-\t\t\t&& contents[offset] != '\\n'; offset--)\n-\t\tswitch (contents[offset]) {\n-\t\t\tcase '=': equal_offset = offset; break;\n-\t\t\tcase ']': bracket_offset = offset; break;\n-\t\t}\n-\tif (offset > 0 && contents[offset-1] == '\\\\') {\n-\t\toffset_ = offset;\n-\t\tgoto contline;\n-\t}\n-\tif (bracket_offset < equal_offset) {\n-\t\t*found_bracket = 1;\n-\t\toffset = bracket_offset+1;\n-\t} else\n-\t\toffset++;\n-\n-\treturn offset;\n-}\n-\n int git_config_set_in_file_gently(const char *config_filename,\n \t\t\t\t const char *key, const char *value)\n {\n@@ -2622,6 +2586,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\tstruct stat st;\n \t\tsize_t copy_begin, copy_end;\n \t\tint i, new_line = 0;\n+\t\tstruct config_options opts;\n \n \t\tif (value_regex == NULL)\n \t\t\tstore.value_regex = NULL;\n@@ -2644,17 +2609,24 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t}\n \t\t}\n \n-\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n-\t\tstore.seen[0] = 0;\n-\t\tstore.seen_nr = 0;\n+\t\tALLOC_GROW(store.parsed, 1, store.parsed_alloc);\n+\t\tstore.parsed[0].end = 0;\n+\n+\t\tmemset(&opts, 0, sizeof(opts));\n+\t\topts.event_fn = store_aux_event;\n+\t\topts.event_fn_data = &store;\n \n \t\t/*\n-\t\t * After this, store.offset will contain the *end* offset\n-\t\t * of the last match, or remain at 0 if no match was found.\n+\t\t * After this, store.parsed will contain offsets of all the\n+\t\t * parsed elements, and store.seen will contain a list of\n+\t\t * matches, as indices into store.parsed.\n+\t\t *\n \t\t * As a side effect, we make sure to transform only a valid\n \t\t * existing config file.\n \t\t */\n-\t\tif (git_config_from_file(store_aux, config_filename, &store)) {\n+\t\tif (git_config_from_file_with_options(store_aux,\n+\t\t\t\t\t\t config_filename,\n+\t\t\t\t\t\t &store, &opts)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n \t\t\tfree(store.key);\n \t\t\tif (store.value_regex != NULL &&\n@@ -2706,19 +2678,39 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tgoto out_free;\n \t\t}\n \n-\t\tif (store.seen_nr == 0)\n+\t\tif (store.seen_nr == 0) {\n+\t\t\tif (!store.seen_alloc) {\n+\t\t\t\t/* Did not see key nor section */\n+\t\t\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n+\t\t\t\tstore.seen[0] = store.parsed_nr\n+\t\t\t\t\t- !!store.parsed_nr;\n+\t\t\t}\n \t\t\tstore.seen_nr = 1;\n+\t\t}\n \n \t\tfor (i = 0, copy_begin = 0; i < store.seen_nr; i++) {\n+\t\t\tsize_t replace_end;\n+\t\t\tint j = store.seen[i];\n+\n \t\t\tnew_line = 0;\n-\t\t\tif (store.seen[i] == 0) {\n-\t\t\t\tstore.seen[i] = copy_end = contents_sz;\n-\t\t\t} else if (!store.key_seen) {\n-\t\t\t\tcopy_end = store.seen[i];\n-\t\t\t} else\n-\t\t\t\tcopy_end = find_beginning_of_line(\n-\t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.seen[i], &new_line);\n+\t\t\tif (!store.key_seen) {\n+\t\t\t\treplace_end = copy_end = store.parsed[j].end;\n+\t\t\t} else {\n+\t\t\t\treplace_end = store.parsed[j].end;\n+\t\t\t\tcopy_end = store.parsed[j].begin;\n+\t\t\t\t/*\n+\t\t\t\t * Swallow preceding white-space on the same\n+\t\t\t\t * line.\n+\t\t\t\t */\n+\t\t\t\twhile (copy_end > 0 ) {\n+\t\t\t\t\tchar c = contents[copy_end - 1];\n+\n+\t\t\t\t\tif (isspace(c) && c != '\\n')\n+\t\t\t\t\t\tcopy_end--;\n+\t\t\t\t\telse\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n@@ -2732,7 +2724,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\t write_str_in_full(fd, \"\\n\") < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-\t\t\tcopy_begin = store.seen[i];\n+\t\t\tcopy_begin = replace_end;\n \t\t}\n \n \t\t/* write the pair (value == NULL means unset) */\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343718","messageId":"8046fb9c3d0af0f442ff65ce703f0898a95d8976.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 13/15] git config --unset: remove empty sections (in the common case)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:28:50Z","receivedAt":"2018-04-03T16:29:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The original reasoning for not removing section headers upon removal of\nthe last entry went like this: the user could have added comments about\nthe section, or about the entries therein, and if there were other\ncomments there, we would not know whether we should remove them.\n\nIn particular, a concocted example was presented that looked like this\n(and was added to t1300):\n\n\t# some generic comment on the configuration file itself\n\t# a comment specific to this \"section\" section.\n\t[section]\n\t# some intervening lines\n\t# that should also be dropped\n\n\tkey = value\n\t# please be careful when you update the above variable\n\nThe ideal thing for `git config --unset section.key` in this case would\nbe to leave only the first line behind, because all the other comments\nare now obsolete.\n\nHowever, this is unfeasible, short of adding a complete Natural Language\nProcessing module to Git, which seems not only a lot of work, but a\ntotally unreasonable feature (for little benefit to most users).\n\nNow, the real kicker about this problem is: most users do not edit their\nconfig files at all! In their use case, the config looks like this\ninstead:\n\n\t[section]\n\t\tkey = value\n\n... and it is totally obvious what should happen if the entry is\nremoved: the entire section should vanish.\n\nLet's generalize this observation to this conservative strategy: if we\nare removing the last entry from a section, and there are no comments\ninside that section nor surrounding it, then remove the entire section.\nOtherwise behave as before: leave the now-empty section (including those\ncomments, even ones about the now-deleted entry).\n\nWe have to be extra careful to handle the case where more than one entry\nis removed: any subset of them might be the last entries of their\nrespective sections (and if there are no comments in or around that\nsection, the section should be removed, too).\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++--\n t/t1300-config.sh | 4 +--\n 2 files changed, 93 insertions(+), 4 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 345b1d2f140..271e9605ec1 100644\n--- a/config.c\n+++ b/config.c\n@@ -2306,6 +2306,7 @@ struct config_set_store {\n \tstruct {\n \t\tsize_t begin, end;\n \t\tenum config_event_t type;\n+\t\tint is_keys_section;\n \t} *parsed;\n \tunsigned int parsed_nr, parsed_alloc, *seen, seen_nr, seen_alloc;\n \tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n@@ -2334,17 +2335,20 @@ static int store_aux_event(enum config_event_t type,\n \tstore->parsed[store->parsed_nr].begin = begin;\n \tstore->parsed[store->parsed_nr].end = end;\n \tstore->parsed[store->parsed_nr].type = type;\n-\tstore->parsed_nr++;\n \n \tif (type == CONFIG_EVENT_SECTION) {\n \t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n \t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n \n \t\t/* Is this the section we were looking for? */\n-\t\tstore->is_keys_section = cf->var.len - 1 == store->baselen &&\n+\t\tstore->is_keys_section =\n+\t\t\tstore->parsed[store->parsed_nr].is_keys_section =\n+\t\t\tcf->var.len - 1 == store->baselen &&\n \t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n \t}\n \n+\tstore->parsed_nr++;\n+\n \treturn 0;\n }\n \n@@ -2476,6 +2480,87 @@ static ssize_t write_pair(int fd, const char *key, const char *value,\n \treturn ret;\n }\n \n+/*\n+ * If we are about to unset the last key(s) in a section, and if there are\n+ * no comments surrounding (or included in) the section, we will want to\n+ * extend begin/end to remove the entire section.\n+ *\n+ * Note: the parameter `seen_ptr` points to the index into the store.seen\n+ * array. * This index may be incremented if a section has more than one\n+ * entry (which all are to be removed).\n+ */\n+static void maybe_remove_section(struct config_set_store *store,\n+\t\t\t\t const char *contents,\n+\t\t\t\t size_t *begin_offset, size_t *end_offset,\n+\t\t\t\t int *seen_ptr)\n+{\n+\tsize_t begin;\n+\tint i, seen, section_seen = 0;\n+\n+\t/*\n+\t * First, ensure that this is the first key, and that there are no\n+\t * comments before the entry nor before the section header.\n+\t */\n+\tseen = *seen_ptr;\n+\tfor (i = store->seen[seen]; i > 0; i--) {\n+\t\tenum config_event_t type = store->parsed[i - 1].type;\n+\n+\t\tif (type == CONFIG_EVENT_COMMENT)\n+\t\t\t/* There is a comment before this entry or section */\n+\t\t\treturn;\n+\t\tif (type == CONFIG_EVENT_ENTRY) {\n+\t\t\tif (!section_seen)\n+\t\t\t\t/* This is not the section's first entry. */\n+\t\t\t\treturn;\n+\t\t\t/* We encountered no comment before the section. */\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (type == CONFIG_EVENT_SECTION) {\n+\t\t\tif (!store->parsed[i - 1].is_keys_section)\n+\t\t\t\tbreak;\n+\t\t\tsection_seen = 1;\n+\t\t}\n+\t}\n+\tbegin = store->parsed[i].begin;\n+\n+\t/*\n+\t * Next, make sure that we are removing he last key(s) in the section,\n+\t * and that there are no comments that are possibly about the current\n+\t * section.\n+\t */\n+\tfor (i = store->seen[seen] + 1; i < store->parsed_nr; i++) {\n+\t\tenum config_event_t type = store->parsed[i].type;\n+\n+\t\tif (type == CONFIG_EVENT_COMMENT)\n+\t\t\treturn;\n+\t\tif (type == CONFIG_EVENT_SECTION) {\n+\t\t\tif (store->parsed[i].is_keys_section)\n+\t\t\t\tcontinue;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (type == CONFIG_EVENT_ENTRY) {\n+\t\t\tif (++seen < store->seen_nr &&\n+\t\t\t i == store->seen[seen])\n+\t\t\t\t/* We want to remove this entry, too */\n+\t\t\t\tcontinue;\n+\t\t\t/* There is another entry in this section. */\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We are really removing the last entry/entries from this section, and\n+\t * there are no enclosed or surrounding comments. Remove the entire,\n+\t * now-empty section.\n+\t */\n+\t*seen_ptr = seen;\n+\t*begin_offset = begin;\n+\tif (i < store->parsed_nr)\n+\t\t*end_offset = store->parsed[i].begin;\n+\telse\n+\t\t*end_offset = store->parsed[store->parsed_nr - 1].end;\n+}\n+\n int git_config_set_in_file_gently(const char *config_filename,\n \t\t\t\t const char *key, const char *value)\n {\n@@ -2698,6 +2783,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t} else {\n \t\t\t\treplace_end = store.parsed[j].end;\n \t\t\t\tcopy_end = store.parsed[j].begin;\n+\t\t\t\tif (!value)\n+\t\t\t\t\tmaybe_remove_section(&store, contents,\n+\t\t\t\t\t\t\t ©_end,\n+\t\t\t\t\t\t\t &replace_end, &i);\n \t\t\t\t/*\n \t\t\t\t * Swallow preceding white-space on the same\n \t\t\t\t * line.\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 10b9bf4b088..6d34513eedd 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1413,7 +1413,7 @@ test_expect_success 'urlmatch with wildcard' '\n '\n \n # good section hygiene\n-test_expect_failure '--unset last key removes section (except if commented)' '\n+test_expect_success '--unset last key removes section (except if commented)' '\n \tcat >.git/config <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n \t# a comment specific to this \"section\" section.\n@@ -1452,7 +1452,7 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \ttest_cmp expect .git/config\n '\n \n-test_expect_failure '--unset-all removes section if empty & uncommented' '\n+test_expect_success '--unset-all removes section if empty & uncommented' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n \tkey = value1\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n\n\n"},{"id":"343719","messageId":"000be096b9a3ece6374ca5d0bf5eb3c2a872bf2f.1522772789.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 14/15] git_config_set: reuse empty sections","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:29:04Z","receivedAt":"2018-04-03T16:29:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"It can happen quite easily that the last setting in a config section is\nremoved, and to avoid confusion when there are comments in the config\nabout that section, we keep a lone section header, i.e. an empty\nsection.\n\nNow that we use the `event_fn` callback, it is easy to add support for\nre-using empty sections, so let's do that.\n\nNote: t5512-ls-remote requires that this change is applied *after* the\npatch \"git config --unset: remove empty sections (in the common case)\":\nwithout that patch, there would be empty `transfer` and `uploadpack`\nsections ready for reuse, but in the *wrong* order (and sconsequently,\nt5512's \"overrides work between mixed transfer/upload-pack hideRefs\"\nwould fail).\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 14 +++++++++++++-\n t/t1300-config.sh | 2 +-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 271e9605ec1..ee7ea24123d 100644\n--- a/config.c\n+++ b/config.c\n@@ -2345,6 +2345,12 @@ static int store_aux_event(enum config_event_t type,\n \t\t\tstore->parsed[store->parsed_nr].is_keys_section =\n \t\t\tcf->var.len - 1 == store->baselen &&\n \t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n+\t\tif (store->is_keys_section) {\n+\t\t\tstore->section_seen = 1;\n+\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t\t store->seen_alloc);\n+\t\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n+\t\t}\n \t}\n \n \tstore->parsed_nr++;\n@@ -2779,7 +2785,13 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n \t\t\tnew_line = 0;\n \t\t\tif (!store.key_seen) {\n-\t\t\t\treplace_end = copy_end = store.parsed[j].end;\n+\t\t\t\tcopy_end = store.parsed[j].end;\n+\t\t\t\t/* include '\\n' when copying section header */\n+\t\t\t\tif (copy_end > 0 && copy_end < contents_sz &&\n+\t\t\t\t contents[copy_end - 1] != '\\n' &&\n+\t\t\t\t contents[copy_end] == '\\n')\n+\t\t\t\t\tcopy_end++;\n+\t\t\t\treplace_end = copy_end;\n \t\t\t} else {\n \t\t\t\treplace_end = store.parsed[j].end;\n \t\t\t\tcopy_end = store.parsed[j].begin;\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 6d34513eedd..6d0e13020d1 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1463,7 +1463,7 @@ test_expect_success '--unset-all removes section if empty & uncommented' '\n \ttest_line_count = 0 .git/config\n '\n \n-test_expect_failure 'adding a key into an empty section reuses header' '\n+test_expect_success 'adding a key into an empty section reuses header' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n \tEOF\n-- \n2.16.2.windows.1.26.g2cc3565eb4b\n"},{"id":"343720","messageId":"nycvar.QRO.7.76.6.1804031829400.5026@qfpub.tvgsbejvaqbjf.bet","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v2 00/15] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-03T16:30:40Z","receivedAt":"2018-04-03T16:30:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi team,\n\nOn Tue, 3 Apr 2018, Johannes Schindelin wrote:\n\n> Johannes Schindelin (15):\n> git_config_set: fix off-by-two\n> t1300: rename it to reflect that `repo-config` was deprecated\n> t1300: demonstrate that --replace-all can \"invent\" newlines\n> config --replace-all: avoid extra line breaks\n> t1300: avoid relying on a bug\n> t1300: remove unreasonable expectation from TODO\n> t1300: `--unset-all` can leave an empty section behind (bug)\n> config: introduce an optional event stream while parsing\n> config: avoid using the global variable `store`\n> config_set_store: rename some fields for consistency\n> git_config_set: do not use a state machine\n> git_config_set: make use of the config parser's event stream\n> git config --unset: remove empty sections (in the common case)\n> git_config_set: reuse empty sections\n> TODOs\n\nPlease note that the `TODOs` commit is a left-over of my internal\nbook-keeping, and its diff is actually empty. Hence `format-patch` does\nnot even generate a mail for it, so there is no [PATCH v2 15/15].\n\nThanks,\nDscho\n"},{"id":"343769","messageId":"CAPig+cSEmHnmfhE1_1j0aVr0cfh4um=MxDr_n3jQq0skHesXGw@mail.gmail.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1804031133270.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-04-03T21:36:50Z","receivedAt":"2018-04-03T21:36:59Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Apr 3, 2018 at 5:49 AM, Johannes Schindelin\n wrote:\n> My main evidence that shell scripts on macOS are slower than on Linux was\n> the difference of the improvement incurred by moving more things from\n> git-rebase--interactive.sh into sequencer.c: Linux saw an improvement only\n> of about 3x, while macOS saw an improvement of 4x, IIRC. If I don't\n> remember the absolute numbers correctly, at least I vividly remember the\n> qualitative difference: It was noticeable.\n\nMacOS is _slow_, much, much slower than, say, Linux.\n\nSeveral years ago, when I had this machine configured for multi-boot,\nI ran MacOS and Linux on bare metal. Back then, using ram disk for the\n\"trash\" directories, and disabling Spotlight indexing on MacOS to\navoid it eating CPU and causing I/O contention, the Git test suite\nwould run to completion on Linux in slightly over 1 minute. On MacOS,\nit would take over 10 minutes; 10 times slower.\n\nThese days, the Git test suite takes 15 minutes to run on the same\nhardware (with same conditions: ram disk and Spotlight disabled),\nwhich is painfully slow, thus I rarely do it. Unfortunately, I don't\nhave Linux installed on bare metal anymore, so I can't make a proper\ncomparison, but I do run Linux in a virtual machine under MacOS and,\neven though its running within a virtualized environment, Linux is\nstill much faster than MacOS, taking 4:25 (slow, but not to the point\nof outright pain).\n\nThat the test suite runs so much faster on Linux (bare metal or\nvirtualized) than MacOS on this machine, I have attributed (or\nunderstood as being due) to poor HFS+ filesystem performance. It's\neven worse when Spotlight interferes. Presumably, the new, recently\nreleased, Mac filesystem has improved performance, but it's restricted\nto SSD's, whereas this machine has a physical drive, thus I can't test\nit.\n"},{"id":"344009","messageId":"20180406212227.GA7870@sigill.intra.peff.net","threadId":"48185","inReplyTo":"a38025bcbc07b83b578ff2f1cde786bdca0c918c.1522772789.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v2 08/15] config: introduce an optional event stream while parsing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-06T21:22:28Z","receivedAt":"2018-04-06T21:22:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 06:28:29PM +0200, Johannes Schindelin wrote:\n\n> This extends our config parser so that it can optionally produce an event\n> stream via callback function, where it reports e.g. when a comment was\n> parsed, or a section header, etc.\n> \n> This parser will be used subsequently to handle the scenarios better where\n> removing config entries would make sections empty, or where a new entry\n> could be added to an already-existing, empty section.\n\nNice, it looks like this didn't end up being too bad to go in this\ndirection. It seems like this is an optional \"also emit the events here\"\nfunction you can set. I think in the long run we could actually just\nalways emit the events to this function. And then we could wrap that to\nprovide an interface that matches the existing callbacks (just an\nevent-stream callback that sees EVENT_ENTRY and calls the sub-callback).\n\nBut that might end up quite a pain, since we have a zillion entry points\ninto the config parser, making wrapping tough. So I'm perfectly happy to\nstop here for now.\n\n> +static inline int do_event(enum config_event_t type,\n> +\t\t\t struct parse_event_data *data)\n\nI'm not sure if \"inline\" here is a good idea, as it seems to get called\nquite a few times. If we're trying to make things fast, bloating the\ninstruction cache may have the opposite effect.\n\n-Peff\n"},{"id":"344010","messageId":"20180406212351.GB7870@sigill.intra.peff.net","threadId":"48185","inReplyTo":"a9689c880f9226c236aec58f8036740ff39efbb8.1522772789.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v2 09/15] config: avoid using the global variable `store`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-06T21:23:51Z","receivedAt":"2018-04-06T21:23:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 06:28:34PM +0200, Johannes Schindelin wrote:\n\n> It is much easier to reason about, when the config code to set/unset\n> variables or to remove/rename sections does not rely on a global (or\n> file-local) variable.\n\nAgreed.\n\n> -static struct {\n> +struct config_set_store {\n\nThis made me think of the existing \"configset\", which is quite a\ndifferent thing. Maybe just \"config_store_data\" or something would clash\nless.\n\n-Peff\n"},{"id":"344012","messageId":"20180406212840.GC7870@sigill.intra.peff.net","threadId":"48185","inReplyTo":"4b3eba873df4355d7a77dfbd74963c636b1ba08e.1522772789.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v2 11/15] git_config_set: do not use a state machine","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-06T21:28:41Z","receivedAt":"2018-04-06T21:28:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 06:28:42PM +0200, Johannes Schindelin wrote:\n\n> While a neat theoretical construct, state machines are hard to read. In\n> this instance, it does not even make a whole lot of sense because we are\n> more interested in flags, anyway: has the section been seen? Has the key\n> been seen? Does the current section match the key we are looking for?\n> \n> Besides, the state `SECTION_SEEN` was named in a misleading way: it did\n> not indicate that we saw the section matching the key we are looking\n> for, but it instead indicated that we are *currently* in that section.\n> \n> Let's just replace the state machine logic by clear and obvious flags.\n> \n> This will also make it easier to review the upcoming patches to use the\n> newly-introduced `event_fn` callback of the config parser.\n\nI think this is probably a good direction. But one thing state machines\ncan help with is keeping the state to a manageable size. With 3 bits of\nflags, we now have 8 possible states, up from the previous 4.\n\nClearly some of those are nonsensical (can you be in key_seen without\nsection_seen? I'd think not), but it's up to the code to interpret and\nreset those manually.\n\nI'll defer to your judgement, though, on this making things for the\nfuture patches more readable. You spend a lot more time poking at it\nthan I have.\n\n-Peff\n"},{"id":"344013","messageId":"20180406213358.GD7870@sigill.intra.peff.net","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v2 00/15] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-06T21:33:58Z","receivedAt":"2018-04-06T21:34:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 06:27:55PM +0200, Johannes Schindelin wrote:\n\n> I am very, very grateful for the time Peff spent on reviewing the previous\n> iteration, and hope that he realizes just how much the elegance of the\n> event-stream-based version is due to his excellent review.\n\nUnfortunately I ran out of time this week to give this version an\nequally careful review, and I'm about to go on vacation for a few weeks.\n\nI did give a cursory look over it, and the new maybe_remove_section() is\nmuch more pleasant. So aside from a few minor nits I pointed out, this\ngenerally looks good.\n\nOne thing I'd like to have seen is a few more tests covering exotic\ncases that I turned up in my earlier review. Some of the weird multiline\ncases I care less about, but we should probably cover at least:\n\n 1. Comment behavior when removing a section that isn't at the\n beginning of the file.\n\n 2. Removing the final key from a section with a subsection.\n\nThose should both be natural fallouts of the new method, but it would be\ngood to have test coverage.\n\nThanks for reworking this, and if it's still not merged when I get back,\nI promise to review it more carefully then. :)\n\n-Peff\n"},{"id":"344014","messageId":"20180406214006.GE7870@sigill.intra.peff.net","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1804031758160.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-06T21:40:07Z","receivedAt":"2018-04-06T21:40:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 06:00:05PM +0200, Johannes Schindelin wrote:\n\n> > But if we're at the point of creating custom C builtins for\n> > busybox/dash/etc, you should be able to create a primitive for \"read\n> > this using buffered stdio, other processes be damned, and return one\n> > line at a time\".\n> \n> Well, you know, I do not think that papering over the root cause will make\n> anything better. And the root cause is that we use a test framework\n> written in Unix shell.\n\nI'm not entirely convinced of this. My earlier numbers show that we\nspend a lot of time actually running Git. But that's not because we're\nwritten in shell, but because the stable interface to Git is running\nindividual processes.\n\nSo we can unit-test wildmatch or similar in a single C program, but I\nthink we inherently need to run \"git init\" a lot of times.\n\nNow I think there's reason to doubt some of my numbers. I was counting\nexec's, and non-exec forks due to subshells, etc, may be important. So I\nclaim only that I remain unconvinced that we are certain of the root\ncause.\n\nAt any rate, I would be happy to see more study into this. If we can\ncreate a measurable speedup for an existing script, that might give us a\nblueprint for speeding up the whole suite.\n\n-Peff\n"},{"id":"344018","messageId":"CAGZ79kaCSyz94_-a_Xq0LWT-tGqr+46me3gHy8d55HpxdU6f-A@mail.gmail.com","threadId":"48185","inReplyTo":"20180406214006.GE7870@sigill.intra.peff.net","subject":"Re: A potential approach to making tests faster on Windows","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-06T21:57:54Z","receivedAt":"2018-04-06T21:57:59Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Apr 6, 2018 at 2:40 PM, Jeff King wrote:\n> On Tue, Apr 03, 2018 at 06:00:05PM +0200, Johannes Schindelin wrote:\n>\n>> > But if we're at the point of creating custom C builtins for\n>> > busybox/dash/etc, you should be able to create a primitive for \"read\n>> > this using buffered stdio, other processes be damned, and return one\n>> > line at a time\".\n>>\n>> Well, you know, I do not think that papering over the root cause will make\n>> anything better. And the root cause is that we use a test framework\n>> written in Unix shell.\n>\n> I'm not entirely convinced of this. My earlier numbers show that we\n> spend a lot of time actually running Git. But that's not because we're\n> written in shell, but because the stable interface to Git is running\n> individual processes.\n>\n> So we can unit-test wildmatch or similar in a single C program, but I\n> think we inherently need to run \"git init\" a lot of times.\n>\n> Now I think there's reason to doubt some of my numbers. I was counting\n> exec's, and non-exec forks due to subshells, etc, may be important. So I\n> claim only that I remain unconvinced that we are certain of the root\n> cause.\n>\n> At any rate, I would be happy to see more study into this. If we can\n> create a measurable speedup for an existing script, that might give us a\n> blueprint for speeding up the whole suite.\n\nThe setup of each test is finicky, as we'd do different setups for each test\nas we'd test different things. I once wondered if we'd want to have a\n\"ready made\" directory that contains repositories in various states\nthat we can copy for each test and only need minimal adjustments\ninstead of writing the setup from scratch in each script.\n"},{"id":"344122","messageId":"xmqqk1th8f8i.fsf@gitster-ct.c.googlers.com","threadId":"48185","inReplyTo":"nycvar.QRO.7.76.6.1804031127400.5026@qfpub.tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/9] git_config_set: fix off-by-two","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-08T23:12:29Z","receivedAt":"2018-04-08T23:12:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> Yes, it is a workaround. Making shell faster on windows would of\n>> course be one possible solution to make t/t*.sh scripts go faster\n>> ;-) Or update parts of t/t*.sh so that the equivalent test coverage\n>> can be kept while running making them go faster on Windows.\n>\n> What makes you think that I did not try my hardest for around 812 hours in\n> total so far to make the shell faster?\n\nNowhere in these four lines I ever said that I think you did not\nwork hard to solve the performance issues you have.\n\n"},{"id":"344162","messageId":"nycvar.QRO.7.76.6.1804090924060.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48185","inReplyTo":"20180406212227.GA7870@sigill.intra.peff.net","subject":"Re: [PATCH v2 08/15] config: introduce an optional event stream while parsing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T07:35:07Z","receivedAt":"2018-04-09T07:35:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 6 Apr 2018, Jeff King wrote:\n\n> On Tue, Apr 03, 2018 at 06:28:29PM +0200, Johannes Schindelin wrote:\n> \n> > This extends our config parser so that it can optionally produce an\n> > event stream via callback function, where it reports e.g. when a\n> > comment was parsed, or a section header, etc.\n> > \n> > This parser will be used subsequently to handle the scenarios better\n> > where removing config entries would make sections empty, or where a\n> > new entry could be added to an already-existing, empty section.\n> \n> Nice, it looks like this didn't end up being too bad to go in this\n> direction. It seems like this is an optional \"also emit the events here\"\n> function you can set.\n\nYes.\n\n> I think in the long run we could actually just always emit the events to\n> this function. And then we could wrap that to provide an interface that\n> matches the existing callbacks (just an event-stream callback that sees\n> EVENT_ENTRY and calls the sub-callback).\n\nWell, not precisely. The event stream was implemented in a minimal\nfashion, in particular *not* emitting enough information in the event\nstream for that. To keep things as little intrusive as possible, the\nCONFIG_EVENT_ENTRY event is only emitted *after* the config_fn is called,\nand at that point we do not even know the key and the value any more.\n\nI fear that it would make the code quite a bit more complicated to change\nit in the way you suggested.\n\nSide note: a slightly ugly aspect of my patch series is that the\nCONFIG_EVENT_SECTION event *also* does not provide the interesting\ninformation (in this case, the section name), but that it has to be\ninferred from the cf->var field (which is file-local to config.c, and\nwhich has been set to the section name followed by a single '.' at that\npoint). Again, this keeps the diff simpler to review, and that's why I did\nit that way.\n\n> But that might end up quite a pain, since we have a zillion entry points\n> into the config parser, making wrapping tough. So I'm perfectly happy to\n> stop here for now.\n\nRight.\n\n> > +static inline int do_event(enum config_event_t type,\n> > +\t\t\t struct parse_event_data *data)\n> \n> I'm not sure if \"inline\" here is a good idea, as it seems to get called\n> quite a few times. If we're trying to make things fast, bloating the\n> instruction cache may have the opposite effect.\n\nGood point.\n\nThe reason I declared this as inline function was that I test whether\neither data->opts or data->opts->event_fn are NULL, and whether we are\ncontinuing to look at whitespace, for early returns from that function.\nWhich I wanted to avoid doing in a hot function (I'd rather skip calling\nthe function if it is pointless to call it).\n\nHowever, the config code is hardly performance-critical, as we do not\nexpect to parse hundreds of kilobytes, right? So that \"inline\" was a\npremature optimization.\n\nThanks,\nDscho\n"},{"id":"344163","messageId":"nycvar.QRO.7.76.6.1804090936260.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48185","inReplyTo":"20180406212351.GB7870@sigill.intra.peff.net","subject":"Re: [PATCH v2 09/15] config: avoid using the global variable `store`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T07:36:41Z","receivedAt":"2018-04-09T07:36:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 6 Apr 2018, Jeff King wrote:\n\n> On Tue, Apr 03, 2018 at 06:28:34PM +0200, Johannes Schindelin wrote:\n> \n> > -static struct {\n> > +struct config_set_store {\n> \n> This made me think of the existing \"configset\", which is quite a\n> different thing. Maybe just \"config_store_data\" or something would clash\n> less.\n\nSure,\nDscho\n"},{"id":"344164","messageId":"nycvar.QRO.7.76.6.1804090945460.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48185","inReplyTo":"20180406212840.GC7870@sigill.intra.peff.net","subject":"Re: [PATCH v2 11/15] git_config_set: do not use a state machine","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T07:50:27Z","receivedAt":"2018-04-09T07:50:52Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 6 Apr 2018, Jeff King wrote:\n\n> On Tue, Apr 03, 2018 at 06:28:42PM +0200, Johannes Schindelin wrote:\n> \n> > While a neat theoretical construct, state machines are hard to read. In\n> > this instance, it does not even make a whole lot of sense because we are\n> > more interested in flags, anyway: has the section been seen? Has the key\n> > been seen? Does the current section match the key we are looking for?\n> > \n> > Besides, the state `SECTION_SEEN` was named in a misleading way: it did\n> > not indicate that we saw the section matching the key we are looking\n> > for, but it instead indicated that we are *currently* in that section.\n> > \n> > Let's just replace the state machine logic by clear and obvious flags.\n> > \n> > This will also make it easier to review the upcoming patches to use the\n> > newly-introduced `event_fn` callback of the config parser.\n> \n> I think this is probably a good direction. But one thing state machines\n> can help with is keeping the state to a manageable size. With 3 bits of\n> flags, we now have 8 possible states, up from the previous 4.\n> \n> Clearly some of those are nonsensical (can you be in key_seen without\n> section_seen? I'd think not), but it's up to the code to interpret and\n> reset those manually.\n\nThat is true. On the other hand, it is easy to miss incorrect state\ntransitions in state machines (or to miss unused states).\n\n> I'll defer to your judgement, though, on this making things for the\n> future patches more readable. You spend a lot more time poking at it\n> than I have.\n\nThe original reason to get rid of the state machine was: I did not need\nthe states any more in the end. Since the section name is set via the\nevent stream we now know in the config_fn whether we are in the correct\nsection or not.\n\nI also liked the fact that it was much easier to reason about correct\ncode: \"Did I catch all the states that apply here?\" is a hairier question\nthan \"Is this flag true?\"\n\nThanks,\nDscho\n"},{"id":"344165","messageId":"nycvar.QRO.7.76.6.1804091017480.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48185","inReplyTo":"20180406213358.GD7870@sigill.intra.peff.net","subject":"Re: [PATCH v2 00/15] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:19:56Z","receivedAt":"2018-04-09T08:20:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 6 Apr 2018, Jeff King wrote:\n\n> On Tue, Apr 03, 2018 at 06:27:55PM +0200, Johannes Schindelin wrote:\n> \n> > I am very, very grateful for the time Peff spent on reviewing the\n> > previous iteration, and hope that he realizes just how much the\n> > elegance of the event-stream-based version is due to his excellent\n> > review.\n> \n> Unfortunately I ran out of time this week to give this version an\n> equally careful review, and I'm about to go on vacation for a few weeks.\n\nNo worries, and thank you for your review. I know I am adding more stuff\nto review these days than I review other stuff, but I promise that I will\ntry to get more reviews in once I am done with this patch series (and with\nthe --rebase-merges one).\n\n> I did give a cursory look over it, and the new maybe_remove_section() is\n> much more pleasant. So aside from a few minor nits I pointed out, this\n> generally looks good.\n\nThanks!\n\n> One thing I'd like to have seen is a few more tests covering exotic\n> cases that I turned up in my earlier review. Some of the weird multiline\n> cases I care less about, but we should probably cover at least:\n> \n> 1. Comment behavior when removing a section that isn't at the\n> beginning of the file.\n> \n> 2. Removing the final key from a section with a subsection.\n> \n> Those should both be natural fallouts of the new method, but it would be\n> good to have test coverage.\n\nI added this, in a new commit I call \"t1300: add a few more hairy examples\nof sections becoming empty\".\n\n> Thanks for reworking this, and if it's still not merged when I get back,\n> I promise to review it more carefully then. :)\n\n:-)\n\nHave a good vacation!\nDscho\n"},{"id":"344166","messageId":"518729d5abdd93980f388821a96a69c8e4cde82e.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 01/15] git_config_set: fix off-by-two","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:25Z","receivedAt":"2018-04-09T08:31:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Currently, we are slightly overzealous When removing an entry from a\nconfig file of this form:\n\n\t[abc]a\n\t[xyz]\n\t\tkey = value\n\nWhen calling `git config --unset abc.a` on this file, it leaves this\n(invalid) config behind:\n\n\t[\n\t[xyz]\n\t\tkey = value\n\nThe reason is that we try to search for the beginning of the line (or\nfor the end of the preceding section header on the same line) that\ndefines abc.a, but as an optimization, we subtract 2 from the offset\npointing just after the definition before we call\nfind_beginning_of_line(). That function, however, *also* performs that\noptimization and promptly fails to find the section header correctly.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/config.c b/config.c\nindex b0c20e6cb8a..5cc049aaef0 100644\n--- a/config.c\n+++ b/config.c\n@@ -2632,7 +2632,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n \t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.offset[i]-2, &new_line);\n+\t\t\t\t\tstore.offset[i], &new_line);\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344167","messageId":"cover.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1522772789.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 00/15] Assorted fixes for `git config` (including the \"empty sections\" bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:19Z","receivedAt":"2018-04-09T08:31:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This patch series originally only tried to help fixing that annoying bug that\nhas been reported several times over the years, where `git config --unset`\nwould leave empty sections behind, and `git config --add` would not reuse them.\n\nThe first patch is somewhat of a \"while at it\" bug fix that I first thought\nwould be a lot more critical than it actually is: It really only affects config\nfiles that start with a section followed immediately (i.e. without a newline)\nby a one-letter boolean setting (i.e. without a `= ` part). So while it\nis a real bug fix, I doubt anybody ever got bitten by it.\n\nThe next swath of patches add and fix some tests, while also fixing the bug\nwhere --replace-all would sometimes insert extra line breaks.\n\nThen, I introduce a couple of building blocks: a \"config parser event stream\",\ni.e. an optional callback that can be used to report events such as \"comment\",\n\"white-space\", etc together with the corresponding extents in the config file.\n\nFinally, the interesting part, where I do two things, essentially (with\npreparatory steps for each thing):\n\n1. I add the ability for `git config --unset/--unset-all` to detect that it\n can remove a section that has just become empty (see below for some more\n discussion of what I consider \"become empty\"), and\n\n2. I add the ability for `git config [--add] key value` to re-use empty\n sections.\n\nTo reiterate why does this patch series not conflict with my very early\nstatements that we cannot simply remove empty sections because we may end up\nwith stale comments?\n\nWell, the patch in question takes pains to determine *iff* there are any\ncomments surrounding, or included in, the section. If any are found: previous\nbehavior. Under the assumption that the user edited the file, we keep it as\nintact as possible (see below for some argument against this). If no comments\nare found, and let's face it, this is probably *the* common case, as few people\nedit their config files by hand these days (neither should they because it is\ntoo easy to end up with an unparseable one), the now-empty section *is*\nremoved.\n\nSo what is the argument against this extra care to detect comments? Well, if\nyou have something like this:\n\n\t[section]\n\t\t; Here we comment about the variable called snarf\n\t\tsnarf = froop\n\nand we run `git config --unset section.snarf`, we end up with this config:\n\n\t[section]\n\t\t; Here we comment about the variable called snarf\n\nwhich obviously does not make sense. However, that is already established\nbehavior for quite a few years, and I do not even try to think of a way how\nthis could be solved.\n\nChanges since v2:\n\n- removed the `inline` attribute from the `do_event()` function.\n\n- renamed `struct config_set_store` to `struct config_store_data`, to make its\n roled more obvious.\n\n- a whole slew of concocted test cases were added to the test to verify that\n a section that becomes empty is removed, based on Peff's analysis at\n https://public-inbox.org/git/20180329213229.GG2939@sigill.intra.peff.net/\n\n\nJohannes Schindelin (15):\n git_config_set: fix off-by-two\n t1300: rename it to reflect that `repo-config` was deprecated\n t1300: demonstrate that --replace-all can \"invent\" newlines\n config --replace-all: avoid extra line breaks\n t1300: avoid relying on a bug\n t1300: remove unreasonable expectation from TODO\n t1300: add a few more hairy examples of sections becoming empty\n t1300: `--unset-all` can leave an empty section behind (bug)\n config: introduce an optional event stream while parsing\n config: avoid using the global variable `store`\n config_set_store: rename some fields for consistency\n git_config_set: do not use a state machine\n git_config_set: make use of the config parser's event stream\n git config --unset: remove empty sections (in the common case)\n git_config_set: reuse empty sections\n\n config.c | 448 ++++++++++++++------\n config.h | 25 ++\n t/{t1300-repo-config.sh => t1300-config.sh} | 102 ++++-\n 3 files changed, 439 insertions(+), 136 deletions(-)\n rename t/{t1300-repo-config.sh => t1300-config.sh} (95%)\n\n\nbase-commit: 468165c1d8a442994a825f3684528361727cd8c0\nPublished-As: https://github.com/dscho/git/releases/tag/empty-config-section-v3\nFetch-It-Via: git fetch https://github.com/dscho/git empty-config-section-v3\n\nInterdiff vs v2:\n diff --git a/config.c b/config.c\n index ee7ea24123d..6155d0651bd 100644\n --- a/config.c\n +++ b/config.c\n @@ -659,8 +659,7 @@ struct parse_event_data {\n \tconst struct config_options *opts;\n };\n \n -static inline int do_event(enum config_event_t type,\n -\t\t\t struct parse_event_data *data)\n +static int do_event(enum config_event_t type, struct parse_event_data *data)\n {\n \tsize_t offset;\n \n @@ -2297,7 +2296,7 @@ void git_die_config(const char *key, const char *err, ...)\n * Find all the stuff for git_config_set() below.\n */\n \n -struct config_set_store {\n +struct config_store_data {\n \tint baselen;\n \tchar *key;\n \tint do_not_match;\n @@ -2313,7 +2312,7 @@ struct config_set_store {\n };\n \n static int matches(const char *key, const char *value,\n -\t\t const struct config_set_store *store)\n +\t\t const struct config_store_data *store)\n {\n \tif (strcmp(key, store->key))\n \t\treturn 0; /* not ours */\n @@ -2329,7 +2328,7 @@ static int matches(const char *key, const char *value,\n static int store_aux_event(enum config_event_t type,\n \t\t\t size_t begin, size_t end, void *data)\n {\n -\tstruct config_set_store *store = data;\n +\tstruct config_store_data *store = data;\n \n \tALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);\n \tstore->parsed[store->parsed_nr].begin = begin;\n @@ -2360,7 +2359,7 @@ static int store_aux_event(enum config_event_t type,\n \n static int store_aux(const char *key, const char *value, void *cb)\n {\n -\tstruct config_set_store *store = cb;\n +\tstruct config_store_data *store = cb;\n \n \tif (store->key_seen) {\n \t\tif (matches(key, value, store)) {\n @@ -2401,7 +2400,7 @@ static int write_error(const char *filename)\n }\n \n static struct strbuf store_create_section(const char *key,\n -\t\t\t\t\t const struct config_set_store *store)\n +\t\t\t\t\t const struct config_store_data *store)\n {\n \tconst char *dot;\n \tint i;\n @@ -2424,7 +2423,7 @@ static struct strbuf store_create_section(const char *key,\n }\n \n static ssize_t write_section(int fd, const char *key,\n -\t\t\t const struct config_set_store *store)\n +\t\t\t const struct config_store_data *store)\n {\n \tstruct strbuf sb = store_create_section(key, store);\n \tssize_t ret;\n @@ -2436,7 +2435,7 @@ static ssize_t write_section(int fd, const char *key,\n }\n \n static ssize_t write_pair(int fd, const char *key, const char *value,\n -\t\t\t const struct config_set_store *store)\n +\t\t\t const struct config_store_data *store)\n {\n \tint i;\n \tssize_t ret;\n @@ -2495,7 +2494,7 @@ static ssize_t write_pair(int fd, const char *key, const char *value,\n * array. * This index may be incremented if a section has more than one\n * entry (which all are to be removed).\n */\n -static void maybe_remove_section(struct config_set_store *store,\n +static void maybe_remove_section(struct config_store_data *store,\n \t\t\t\t const char *contents,\n \t\t\t\t size_t *begin_offset, size_t *end_offset,\n \t\t\t\t int *seen_ptr)\n @@ -2625,7 +2624,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tchar *filename_buf = NULL;\n \tchar *contents = NULL;\n \tsize_t contents_sz;\n -\tstruct config_set_store store;\n +\tstruct config_store_data store;\n \n \tmemset(&store, 0, sizeof(store));\n \n @@ -2969,7 +2968,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \tFILE *config_file = NULL;\n \tstruct stat st;\n \tstruct strbuf copystr = STRBUF_INIT;\n -\tstruct config_set_store store;\n +\tstruct config_store_data store;\n \n \tmemset(&store, 0, sizeof(store));\n \n diff --git a/t/t1300-config.sh b/t/t1300-config.sh\n index 6d0e13020d1..eef0bbe4f9f 100755\n --- a/t/t1300-config.sh\n +++ b/t/t1300-config.sh\n @@ -1449,7 +1449,50 @@ test_expect_success '--unset last key removes section (except if commented)' '\n \tEOF\n \n \tgit config --unset section.key &&\n -\ttest_cmp expect .git/config\n +\ttest_cmp expect .git/config &&\n +\n +\tq_to_tab >.git/config <<-\\EOF &&\n +\t[one]\n +\tQkey = \"multiline \\\n +\tQQ# with comment\"\n +\t[two]\n +\tkey = true\n +\tEOF\n +\tgit config --unset two.key &&\n +\t! grep two .git/config &&\n +\n +\tq_to_tab >.git/config <<-\\EOF &&\n +\t[one]\n +\tQkey = \"multiline \\\n +\tQQ# with comment\"\n +\t[one]\n +\tkey = true\n +\tEOF\n +\tgit config --unset-all one.key &&\n +\ttest_line_count = 0 .git/config &&\n +\n +\tq_to_tab >.git/config <<-\\EOF &&\n +\t[one]\n +\tQkey = true\n +\tQ# a comment not at the start\n +\t[two]\n +\tQkey = true\n +\tEOF\n +\tgit config --unset two.key &&\n +\tgrep two .git/config &&\n +\n +\tq_to_tab >.git/config <<-\\EOF &&\n +\t[one]\n +\tQkey = not [two \"subsection\"]\n +\t[two \"subsection\"]\n +\t[two \"subsection\"]\n +\tQkey = true\n +\t[TWO \"subsection\"]\n +\t[one]\n +\tEOF\n +\tgit config --unset two.subsection.key &&\n +\ttest \"not [two subsection]\" = \"$(git config one.key)\" &&\n +\ttest_line_count = 3 .git/config\n '\n \n test_expect_success '--unset-all removes section if empty & uncommented' '\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n"},{"id":"344168","messageId":"532243287ff81725502f09251d9f749ce1d34417.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 03/15] t1300: demonstrate that --replace-all can \"invent\" newlines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:33Z","receivedAt":"2018-04-09T08:31:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Signed-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 4f8e6f5fde3..cc417687e8d 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1611,4 +1611,25 @@ test_expect_success '--local requires a repo' '\n \ttest_expect_code 128 nongit git config --local foo.bar\n '\n \n+test_expect_failure '--replace-all does not invent newlines' '\n+\tq_to_tab >.git/config <<-\\EOF &&\n+\t[abc]key\n+\tQkeepSection\n+\t[xyz]\n+\tQkey = 1\n+\t[abc]\n+\tQkey = a\n+\tEOF\n+\tq_to_tab >expect <<-\\EOF &&\n+\t[abc]\n+\tQkeepSection\n+\t[xyz]\n+\tQkey = 1\n+\t[abc]\n+\tQkey = b\n+\tEOF\n+\tgit config --replace-all abc.key b &&\n+\ttest_cmp .git/config expect\n+'\n+\n test_done\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344169","messageId":"8264d7aa630a23167f8cffcb23462371f7511be8.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 04/15] config --replace-all: avoid extra line breaks","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:36Z","receivedAt":"2018-04-09T08:31:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"When replacing multiple config entries at once, we did not re-set the\nflag that indicates whether we need to insert a new-line before the new\nentry. As a consequence, an extra new-line was inserted under certain\ncircumstances.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 1 +\n t/t1300-config.sh | 2 +-\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/config.c b/config.c\nindex 5cc049aaef0..f10f8c6f52f 100644\n--- a/config.c\n+++ b/config.c\n@@ -2625,6 +2625,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tstore.seen = 1;\n \n \t\tfor (i = 0, copy_begin = 0; i < store.seen; i++) {\n+\t\t\tnew_line = 0;\n \t\t\tif (store.offset[i] == 0) {\n \t\t\t\tstore.offset[i] = copy_end = contents_sz;\n \t\t\t} else if (store.state != KEY_SEEN) {\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex cc417687e8d..aed12be492f 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1611,7 +1611,7 @@ test_expect_success '--local requires a repo' '\n \ttest_expect_code 128 nongit git config --local foo.bar\n '\n \n-test_expect_failure '--replace-all does not invent newlines' '\n+test_expect_success '--replace-all does not invent newlines' '\n \tq_to_tab >.git/config <<-\\EOF &&\n \t[abc]key\n \tQkeepSection\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344170","messageId":"083953bc0d5c7f497722a1b42b0e0be884fd1a4c.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 02/15] t1300: rename it to reflect that `repo-config` was deprecated","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:29Z","receivedAt":"2018-04-09T08:31:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Signed-off-by: Johannes Schindelin \n---\n t/{t1300-repo-config.sh => t1300-config.sh} | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n rename t/{t1300-repo-config.sh => t1300-config.sh} (100%)\n\ndiff --git a/t/t1300-repo-config.sh b/t/t1300-config.sh\nsimilarity index 100%\nrename from t/t1300-repo-config.sh\nrename to t/t1300-config.sh\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344171","messageId":"4c8159bd9d77f7e21052e9c9b453f47b83847885.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 05/15] t1300: avoid relying on a bug","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:39Z","receivedAt":"2018-04-09T08:31:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The test case 'unset with cont. lines' relied on a bug that is about to\nbe fixed: it tests *explicitly* that removing the last entry from a\nconfig section leaves an *empty* section behind.\n\nLet's fix this test case not to rely on that behavior, simply by\npreventing the section from becoming empty.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex aed12be492f..7c0ee208dea 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -108,6 +108,7 @@ bar = foo\n [beta]\n baz = multiple \\\n lines\n+foo = bar\n EOF\n \n test_expect_success 'unset with cont. lines' '\n@@ -118,6 +119,7 @@ cat > expect <<\\EOF\n [alpha]\n bar = foo\n [beta]\n+foo = bar\n EOF\n \n test_expect_success 'unset with cont. lines is correct' 'test_cmp expect .git/config'\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344172","messageId":"5a4791e582116d83425918f9bb13201a44c91aa6.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 06/15] t1300: remove unreasonable expectation from TODO","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:42Z","receivedAt":"2018-04-09T08:31:59Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"In https://public-inbox.org/git/7vvc8alzat.fsf@alter.siamese.dyndns.org/\na reasonable patch was made quite a bit less so by changing a test case\ndemonstrating a bug to a test case that demonstrates that we ask for too\nmuch: the test case 'unsetting the last key in a section removes header'\nnow expects a future bug fix to be able to determine whether a free-form\ncomment above a section header refers to said section or not.\n\nRather than shooting for the stars (and not even getting off the\nground), let's start shooting for something obtainable and be reasonably\nconfident that we *can* get it.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 7c0ee208dea..187fc5b195f 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1413,7 +1413,7 @@ test_expect_success 'urlmatch with wildcard' '\n '\n \n # good section hygiene\n-test_expect_failure 'unsetting the last key in a section removes header' '\n+test_expect_failure '--unset last key removes section (except if commented)' '\n \tcat >.git/config <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n \t# a comment specific to this \"section\" section.\n@@ -1427,6 +1427,25 @@ test_expect_failure 'unsetting the last key in a section removes header' '\n \n \tcat >expect <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n+\t# a comment specific to this \"section\" section.\n+\t[section]\n+\t# some intervening lines\n+\t# that should also be dropped\n+\n+\t# please be careful when you update the above variable\n+\tEOF\n+\n+\tgit config --unset section.key &&\n+\ttest_cmp expect .git/config &&\n+\n+\tcat >.git/config <<-\\EOF &&\n+\t[section]\n+\tkey = value\n+\t[next-section]\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[next-section]\n \tEOF\n \n \tgit config --unset section.key &&\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344173","messageId":"880bd424742c2bc562d567c6fad3409ce99aa825.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 07/15] t1300: add a few more hairy examples of sections becoming empty","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:31:57Z","receivedAt":"2018-04-09T08:32:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"During the review of the first iteration of the patch series to remove\nsections that become empty upon --unset or --unset-all, Jeff King\nidentified a couple of problematic cases with the backtracking approach\nthat was still used then to \"look backwards for the section header\":\nhttps://public-inbox.org/git/20180329213229.GG2939@sigill.intra.peff.net/\n\nThis patch adds a couple of concocted examples designed to fool a\nbacktracking parser.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 45 ++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 44 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 187fc5b195f..bc30cfb3468 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1449,7 +1449,50 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \tEOF\n \n \tgit config --unset section.key &&\n-\ttest_cmp expect .git/config\n+\ttest_cmp expect .git/config &&\n+\n+\tq_to_tab >.git/config <<-\\EOF &&\n+\t[one]\n+\tQkey = \"multiline \\\n+\tQQ# with comment\"\n+\t[two]\n+\tkey = true\n+\tEOF\n+\tgit config --unset two.key &&\n+\t! grep two .git/config &&\n+\n+\tq_to_tab >.git/config <<-\\EOF &&\n+\t[one]\n+\tQkey = \"multiline \\\n+\tQQ# with comment\"\n+\t[one]\n+\tkey = true\n+\tEOF\n+\tgit config --unset-all one.key &&\n+\ttest_line_count = 0 .git/config &&\n+\n+\tq_to_tab >.git/config <<-\\EOF &&\n+\t[one]\n+\tQkey = true\n+\tQ# a comment not at the start\n+\t[two]\n+\tQkey = true\n+\tEOF\n+\tgit config --unset two.key &&\n+\tgrep two .git/config &&\n+\n+\tq_to_tab >.git/config <<-\\EOF &&\n+\t[one]\n+\tQkey = not [two \"subsection\"]\n+\t[two \"subsection\"]\n+\t[two \"subsection\"]\n+\tQkey = true\n+\t[TWO \"subsection\"]\n+\t[one]\n+\tEOF\n+\tgit config --unset two.subsection.key &&\n+\ttest \"not [two subsection]\" = \"$(git config one.key)\" &&\n+\ttest_line_count = 3 .git/config\n '\n \n test_expect_failure 'adding a key into an empty section reuses header' '\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344174","messageId":"c3cf926667ef4a071016507ae902aaa9e7518ea4.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 08/15] t1300: `--unset-all` can leave an empty section behind (bug)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:02Z","receivedAt":"2018-04-09T08:32:15Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"We already have a test demonstrating that removing the last entry from a\nconfig section fails to remove the section header of the now-empty\nsection.\n\nThe same can happen, of course, if we remove the last entries in one fell\nswoop. This is *also* a bug, and should be fixed at the same time.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t1300-config.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex bc30cfb3468..9d23a8ca972 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1495,6 +1495,17 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \ttest_line_count = 3 .git/config\n '\n \n+test_expect_failure '--unset-all removes section if empty & uncommented' '\n+\tcat >.git/config <<-\\EOF &&\n+\t[section]\n+\tkey = value1\n+\tkey = value2\n+\tEOF\n+\n+\tgit config --unset-all section.key &&\n+\ttest_line_count = 0 .git/config\n+'\n+\n test_expect_failure 'adding a key into an empty section reuses header' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344175","messageId":"f8a9b92526e1b577bfdee078c02a6d4f1cc80c34.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 09/15] config: introduce an optional event stream while parsing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:05Z","receivedAt":"2018-04-09T08:32:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This extends our config parser so that it can optionally produce an event\nstream via callback function, where it reports e.g. when a comment was\nparsed, or a section header, etc.\n\nThis parser will be used subsequently to handle the scenarios better where\nremoving config entries would make sections empty, or where a new entry\ncould be added to an already-existing, empty section.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++-------\n config.h | 25 ++++++++++++++\n 2 files changed, 114 insertions(+), 12 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex f10f8c6f52f..03d8e7709fe 100644\n--- a/config.c\n+++ b/config.c\n@@ -653,7 +653,45 @@ static int get_base_var(struct strbuf *name)\n \t}\n }\n \n-static int git_parse_source(config_fn_t fn, void *data)\n+struct parse_event_data {\n+\tenum config_event_t previous_type;\n+\tsize_t previous_offset;\n+\tconst struct config_options *opts;\n+};\n+\n+static int do_event(enum config_event_t type, struct parse_event_data *data)\n+{\n+\tsize_t offset;\n+\n+\tif (!data->opts || !data->opts->event_fn)\n+\t\treturn 0;\n+\n+\tif (type == CONFIG_EVENT_WHITESPACE &&\n+\t data->previous_type == type)\n+\t\treturn 0;\n+\n+\toffset = cf->do_ftell(cf);\n+\t/*\n+\t * At EOF, the parser always \"inserts\" an extra '\\n', therefore\n+\t * the end offset of the event is the current file position, otherwise\n+\t * we will already have advanced to the next event.\n+\t */\n+\tif (type != CONFIG_EVENT_EOF)\n+\t\toffset--;\n+\n+\tif (data->previous_type != CONFIG_EVENT_EOF &&\n+\t data->opts->event_fn(data->previous_type, data->previous_offset,\n+\t\t\t\t offset, data->opts->event_fn_data) < 0)\n+\t\treturn -1;\n+\n+\tdata->previous_type = type;\n+\tdata->previous_offset = offset;\n+\n+\treturn 0;\n+}\n+\n+static int git_parse_source(config_fn_t fn, void *data,\n+\t\t\t const struct config_options *opts)\n {\n \tint comment = 0;\n \tint baselen = 0;\n@@ -664,8 +702,15 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t/* U+FEFF Byte Order Mark in UTF8 */\n \tconst char *bomptr = utf8_bom;\n \n+\t/* For the parser event callback */\n+\tstruct parse_event_data event_data = {\n+\t\tCONFIG_EVENT_EOF, 0, opts\n+\t};\n+\n \tfor (;;) {\n-\t\tint c = get_next_char();\n+\t\tint c;\n+\n+\t\tc = get_next_char();\n \t\tif (bomptr && *bomptr) {\n \t\t\t/* We are at the file beginning; skip UTF8-encoded BOM\n \t\t\t * if present. Sane editors won't put this in on their\n@@ -682,18 +727,33 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t\t}\n \t\t}\n \t\tif (c == '\\n') {\n-\t\t\tif (cf->eof)\n+\t\t\tif (cf->eof) {\n+\t\t\t\tif (do_event(CONFIG_EVENT_EOF, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n \t\t\t\treturn 0;\n+\t\t\t}\n+\t\t\tif (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)\n+\t\t\t\treturn -1;\n \t\t\tcomment = 0;\n \t\t\tcontinue;\n \t\t}\n-\t\tif (comment || isspace(c))\n+\t\tif (comment)\n \t\t\tcontinue;\n+\t\tif (isspace(c)) {\n+\t\t\tif (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (c == '#' || c == ';') {\n+\t\t\tif (do_event(CONFIG_EVENT_COMMENT, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n \t\t\tcomment = 1;\n \t\t\tcontinue;\n \t\t}\n \t\tif (c == '[') {\n+\t\t\tif (do_event(CONFIG_EVENT_SECTION, &event_data) < 0)\n+\t\t\t\t\treturn -1;\n+\n \t\t\t/* Reset prior to determining a new stem */\n \t\t\tstrbuf_reset(var);\n \t\t\tif (get_base_var(var) < 0 || var->len < 1)\n@@ -704,6 +764,10 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t}\n \t\tif (!isalpha(c))\n \t\t\tbreak;\n+\n+\t\tif (do_event(CONFIG_EVENT_ENTRY, &event_data) < 0)\n+\t\t\treturn -1;\n+\n \t\t/*\n \t\t * Truncate the var name back to the section header\n \t\t * stem prior to grabbing the suffix part of the name\n@@ -715,6 +779,9 @@ static int git_parse_source(config_fn_t fn, void *data)\n \t\t\tbreak;\n \t}\n \n+\tif (do_event(CONFIG_EVENT_ERROR, &event_data) < 0)\n+\t\treturn -1;\n+\n \tswitch (cf->origin_type) {\n \tcase CONFIG_ORIGIN_BLOB:\n \t\terror_msg = xstrfmt(_(\"bad config line %d in blob %s\"),\n@@ -1398,7 +1465,8 @@ int git_default_config(const char *var, const char *value, void *dummy)\n * fgetc, ungetc, ftell of top need to be initialized before calling\n * this function.\n */\n-static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n+static int do_config_from(struct config_source *top, config_fn_t fn, void *data,\n+\t\t\t const struct config_options *opts)\n {\n \tint ret;\n \n@@ -1410,7 +1478,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n \tstrbuf_init(&top->var, 1024);\n \tcf = top;\n \n-\tret = git_parse_source(fn, data);\n+\tret = git_parse_source(fn, data, opts);\n \n \t/* pop config-file parsing state stack */\n \tstrbuf_release(&top->value);\n@@ -1423,7 +1491,7 @@ static int do_config_from(struct config_source *top, config_fn_t fn, void *data)\n static int do_config_from_file(config_fn_t fn,\n \t\tconst enum config_origin_type origin_type,\n \t\tconst char *name, const char *path, FILE *f,\n-\t\tvoid *data)\n+\t\tvoid *data, const struct config_options *opts)\n {\n \tstruct config_source top;\n \n@@ -1436,15 +1504,18 @@ static int do_config_from_file(config_fn_t fn,\n \ttop.do_ungetc = config_file_ungetc;\n \ttop.do_ftell = config_file_ftell;\n \n-\treturn do_config_from(&top, fn, data);\n+\treturn do_config_from(&top, fn, data, opts);\n }\n \n static int git_config_from_stdin(config_fn_t fn, void *data)\n {\n-\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin, data);\n+\treturn do_config_from_file(fn, CONFIG_ORIGIN_STDIN, \"\", NULL, stdin,\n+\t\t\t\t data, NULL);\n }\n \n-int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n+int git_config_from_file_with_options(config_fn_t fn, const char *filename,\n+\t\t\t\t void *data,\n+\t\t\t\t const struct config_options *opts)\n {\n \tint ret = -1;\n \tFILE *f;\n@@ -1452,13 +1523,19 @@ int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n \tf = fopen_or_warn(filename, \"r\");\n \tif (f) {\n \t\tflockfile(f);\n-\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data);\n+\t\tret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename,\n+\t\t\t\t\t filename, f, data, opts);\n \t\tfunlockfile(f);\n \t\tfclose(f);\n \t}\n \treturn ret;\n }\n \n+int git_config_from_file(config_fn_t fn, const char *filename, void *data)\n+{\n+\treturn git_config_from_file_with_options(fn, filename, data, NULL);\n+}\n+\n int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_type,\n \t\t\tconst char *name, const char *buf, size_t len, void *data)\n {\n@@ -1475,7 +1552,7 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \ttop.do_ungetc = config_buf_ungetc;\n \ttop.do_ftell = config_buf_ftell;\n \n-\treturn do_config_from(&top, fn, data);\n+\treturn do_config_from(&top, fn, data, NULL);\n }\n \n int git_config_from_blob_oid(config_fn_t fn,\ndiff --git a/config.h b/config.h\nindex ef70a9cac1e..5a2394daae2 100644\n--- a/config.h\n+++ b/config.h\n@@ -28,15 +28,40 @@ enum config_origin_type {\n \tCONFIG_ORIGIN_CMDLINE\n };\n \n+enum config_event_t {\n+\tCONFIG_EVENT_SECTION,\n+\tCONFIG_EVENT_ENTRY,\n+\tCONFIG_EVENT_WHITESPACE,\n+\tCONFIG_EVENT_COMMENT,\n+\tCONFIG_EVENT_EOF,\n+\tCONFIG_EVENT_ERROR\n+};\n+\n+/*\n+ * The parser event function (if not NULL) is called with the event type and\n+ * the begin/end offsets of the parsed elements.\n+ *\n+ * Note: for CONFIG_EVENT_ENTRY (i.e. config variables), the trailing newline\n+ * character is considered part of the element.\n+ */\n+typedef int (*config_parser_event_fn_t)(enum config_event_t type,\n+\t\t\t\t\tsize_t begin_offset, size_t end_offset,\n+\t\t\t\t\tvoid *event_fn_data);\n+\n struct config_options {\n \tunsigned int respect_includes : 1;\n \tconst char *commondir;\n \tconst char *git_dir;\n+\tconfig_parser_event_fn_t event_fn;\n+\tvoid *event_fn_data;\n };\n \n typedef int (*config_fn_t)(const char *, const char *, void *);\n extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n+extern int git_config_from_file_with_options(config_fn_t fn, const char *,\n+\t\t\t\t\t void *,\n+\t\t\t\t\t const struct config_options *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n extern int git_config_from_blob_oid(config_fn_t fn, const char *name,\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344176","messageId":"8ba9842d8eee51b6dac772292eefef2c4c36952e.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 10/15] config: avoid using the global variable `store`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:09Z","receivedAt":"2018-04-09T08:32:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"It is much easier to reason about, when the config code to set/unset\nvariables or to remove/rename sections does not rely on a global (or\nfile-local) variable.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 119 ++++++++++++++++++++++++++++++-------------------------\n 1 file changed, 66 insertions(+), 53 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 03d8e7709fe..0c0a965267d 100644\n--- a/config.c\n+++ b/config.c\n@@ -2296,7 +2296,7 @@ void git_die_config(const char *key, const char *err, ...)\n * Find all the stuff for git_config_set() below.\n */\n \n-static struct {\n+struct config_store_data {\n \tint baselen;\n \tchar *key;\n \tint do_not_match;\n@@ -2306,56 +2306,58 @@ static struct {\n \tunsigned int offset_alloc;\n \tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n \tunsigned int seen;\n-} store;\n+};\n \n-static int matches(const char *key, const char *value)\n+static int matches(const char *key, const char *value,\n+\t\t const struct config_store_data *store)\n {\n-\tif (strcmp(key, store.key))\n+\tif (strcmp(key, store->key))\n \t\treturn 0; /* not ours */\n-\tif (!store.value_regex)\n+\tif (!store->value_regex)\n \t\treturn 1; /* always matches */\n-\tif (store.value_regex == CONFIG_REGEX_NONE)\n+\tif (store->value_regex == CONFIG_REGEX_NONE)\n \t\treturn 0; /* never matches */\n \n-\treturn store.do_not_match ^\n-\t\t(value && !regexec(store.value_regex, value, 0, NULL, 0));\n+\treturn store->do_not_match ^\n+\t\t(value && !regexec(store->value_regex, value, 0, NULL, 0));\n }\n \n static int store_aux(const char *key, const char *value, void *cb)\n {\n \tconst char *ep;\n \tsize_t section_len;\n+\tstruct config_store_data *store = cb;\n \n-\tswitch (store.state) {\n+\tswitch (store->state) {\n \tcase KEY_SEEN:\n-\t\tif (matches(key, value)) {\n-\t\t\tif (store.seen == 1 && store.multi_replace == 0) {\n+\t\tif (matches(key, value, store)) {\n+\t\t\tif (store->seen == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n \t\t\t}\n \n-\t\t\tALLOC_GROW(store.offset, store.seen + 1,\n-\t\t\t\t store.offset_alloc);\n+\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n+\t\t\t\t store->offset_alloc);\n \n-\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n-\t\t\tstore.seen++;\n+\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\t\tstore->seen++;\n \t\t}\n \t\tbreak;\n \tcase SECTION_SEEN:\n \t\t/*\n-\t\t * What we are looking for is in store.key (both\n+\t\t * What we are looking for is in store->key (both\n \t\t * section and var), and its section part is baselen\n \t\t * long. We found key (again, both section and var).\n \t\t * We would want to know if this key is in the same\n \t\t * section as what we are looking for. We already\n \t\t * know we are in the same section as what should\n-\t\t * hold store.key.\n+\t\t * hold store->key.\n \t\t */\n \t\tep = strrchr(key, '.');\n \t\tsection_len = ep - key;\n \n-\t\tif ((section_len != store.baselen) ||\n-\t\t memcmp(key, store.key, section_len+1)) {\n-\t\t\tstore.state = SECTION_END_SEEN;\n+\t\tif ((section_len != store->baselen) ||\n+\t\t memcmp(key, store->key, section_len+1)) {\n+\t\t\tstore->state = SECTION_END_SEEN;\n \t\t\tbreak;\n \t\t}\n \n@@ -2363,26 +2365,27 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t * Do not increment matches: this is no match, but we\n \t\t * just made sure we are in the desired section.\n \t\t */\n-\t\tALLOC_GROW(store.offset, store.seen + 1,\n-\t\t\t store.offset_alloc);\n-\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n+\t\tALLOC_GROW(store->offset, store->seen + 1,\n+\t\t\t store->offset_alloc);\n+\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n \t\t/* fallthru */\n \tcase SECTION_END_SEEN:\n \tcase START:\n-\t\tif (matches(key, value)) {\n-\t\t\tALLOC_GROW(store.offset, store.seen + 1,\n-\t\t\t\t store.offset_alloc);\n-\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n-\t\t\tstore.state = KEY_SEEN;\n-\t\t\tstore.seen++;\n+\t\tif (matches(key, value, store)) {\n+\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n+\t\t\t\t store->offset_alloc);\n+\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\t\tstore->state = KEY_SEEN;\n+\t\t\tstore->seen++;\n \t\t} else {\n-\t\t\tif (strrchr(key, '.') - key == store.baselen &&\n-\t\t\t !strncmp(key, store.key, store.baselen)) {\n-\t\t\t\t\tstore.state = SECTION_SEEN;\n-\t\t\t\t\tALLOC_GROW(store.offset,\n-\t\t\t\t\t\t store.seen + 1,\n-\t\t\t\t\t\t store.offset_alloc);\n-\t\t\t\t\tstore.offset[store.seen] = cf->do_ftell(cf);\n+\t\t\tif (strrchr(key, '.') - key == store->baselen &&\n+\t\t\t !strncmp(key, store->key, store->baselen)) {\n+\t\t\t\t\tstore->state = SECTION_SEEN;\n+\t\t\t\t\tALLOC_GROW(store->offset,\n+\t\t\t\t\t\t store->seen + 1,\n+\t\t\t\t\t\t store->offset_alloc);\n+\t\t\t\t\tstore->offset[store->seen] =\n+\t\t\t\t\t\tcf->do_ftell(cf);\n \t\t\t}\n \t\t}\n \t}\n@@ -2397,31 +2400,33 @@ static int write_error(const char *filename)\n \treturn 4;\n }\n \n-static struct strbuf store_create_section(const char *key)\n+static struct strbuf store_create_section(const char *key,\n+\t\t\t\t\t const struct config_store_data *store)\n {\n \tconst char *dot;\n \tint i;\n \tstruct strbuf sb = STRBUF_INIT;\n \n-\tdot = memchr(key, '.', store.baselen);\n+\tdot = memchr(key, '.', store->baselen);\n \tif (dot) {\n \t\tstrbuf_addf(&sb, \"[%.*s \\\"\", (int)(dot - key), key);\n-\t\tfor (i = dot - key + 1; i < store.baselen; i++) {\n+\t\tfor (i = dot - key + 1; i < store->baselen; i++) {\n \t\t\tif (key[i] == '\"' || key[i] == '\\\\')\n \t\t\t\tstrbuf_addch(&sb, '\\\\');\n \t\t\tstrbuf_addch(&sb, key[i]);\n \t\t}\n \t\tstrbuf_addstr(&sb, \"\\\"]\\n\");\n \t} else {\n-\t\tstrbuf_addf(&sb, \"[%.*s]\\n\", store.baselen, key);\n+\t\tstrbuf_addf(&sb, \"[%.*s]\\n\", store->baselen, key);\n \t}\n \n \treturn sb;\n }\n \n-static ssize_t write_section(int fd, const char *key)\n+static ssize_t write_section(int fd, const char *key,\n+\t\t\t const struct config_store_data *store)\n {\n-\tstruct strbuf sb = store_create_section(key);\n+\tstruct strbuf sb = store_create_section(key, store);\n \tssize_t ret;\n \n \tret = write_in_full(fd, sb.buf, sb.len);\n@@ -2430,11 +2435,12 @@ static ssize_t write_section(int fd, const char *key)\n \treturn ret;\n }\n \n-static ssize_t write_pair(int fd, const char *key, const char *value)\n+static ssize_t write_pair(int fd, const char *key, const char *value,\n+\t\t\t const struct config_store_data *store)\n {\n \tint i;\n \tssize_t ret;\n-\tint length = strlen(key + store.baselen + 1);\n+\tint length = strlen(key + store->baselen + 1);\n \tconst char *quote = \"\";\n \tstruct strbuf sb = STRBUF_INIT;\n \n@@ -2454,7 +2460,7 @@ static ssize_t write_pair(int fd, const char *key, const char *value)\n \t\tquote = \"\\\"\";\n \n \tstrbuf_addf(&sb, \"\\t%.*s = %s\",\n-\t\t length, key + store.baselen + 1, quote);\n+\t\t length, key + store->baselen + 1, quote);\n \n \tfor (i = 0; value[i]; i++)\n \t\tswitch (value[i]) {\n@@ -2564,6 +2570,9 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \tchar *filename_buf = NULL;\n \tchar *contents = NULL;\n \tsize_t contents_sz;\n+\tstruct config_store_data store;\n+\n+\tmemset(&store, 0, sizeof(store));\n \n \t/* parse-key returns negative; flip the sign to feed exit(3) */\n \tret = 0 - git_config_parse_key(key, &store.key, &store.baselen);\n@@ -2606,8 +2615,8 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t}\n \n \t\tstore.key = (char *)key;\n-\t\tif (write_section(fd, key) < 0 ||\n-\t\t write_pair(fd, key, value) < 0)\n+\t\tif (write_section(fd, key, &store) < 0 ||\n+\t\t write_pair(fd, key, value, &store) < 0)\n \t\t\tgoto write_err_out;\n \t} else {\n \t\tstruct stat st;\n@@ -2646,7 +2655,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t * As a side effect, we make sure to transform only a valid\n \t\t * existing config file.\n \t\t */\n-\t\tif (git_config_from_file(store_aux, config_filename, NULL)) {\n+\t\tif (git_config_from_file(store_aux, config_filename, &store)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n \t\t\tfree(store.key);\n \t\t\tif (store.value_regex != NULL &&\n@@ -2730,10 +2739,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t/* write the pair (value == NULL means unset) */\n \t\tif (value != NULL) {\n \t\t\tif (store.state == START) {\n-\t\t\t\tif (write_section(fd, key) < 0)\n+\t\t\t\tif (write_section(fd, key, &store) < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-\t\t\tif (write_pair(fd, key, value) < 0)\n+\t\t\tif (write_pair(fd, key, value, &store) < 0)\n \t\t\t\tgoto write_err_out;\n \t\t}\n \n@@ -2857,7 +2866,8 @@ static int section_name_is_ok(const char *name)\n \n /* if new_name == NULL, the section is removed instead */\n static int git_config_copy_or_rename_section_in_file(const char *config_filename,\n-\t\t\t\t const char *old_name, const char *new_name, int copy)\n+\t\t\t\t const char *old_name,\n+\t\t\t\t const char *new_name, int copy)\n {\n \tint ret = 0, remove = 0;\n \tchar *filename_buf = NULL;\n@@ -2867,6 +2877,9 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \tFILE *config_file = NULL;\n \tstruct stat st;\n \tstruct strbuf copystr = STRBUF_INIT;\n+\tstruct config_store_data store;\n+\n+\tmemset(&store, 0, sizeof(store));\n \n \tif (new_name && !section_name_is_ok(new_name)) {\n \t\tret = error(\"invalid section name: %s\", new_name);\n@@ -2936,7 +2949,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \t\t\t\t}\n \t\t\t\tstore.baselen = strlen(new_name);\n \t\t\t\tif (!copy) {\n-\t\t\t\t\tif (write_section(out_fd, new_name) < 0) {\n+\t\t\t\t\tif (write_section(out_fd, new_name, &store) < 0) {\n \t\t\t\t\t\tret = write_error(get_lock_file_path(&lock));\n \t\t\t\t\t\tgoto out;\n \t\t\t\t\t}\n@@ -2957,7 +2970,7 @@ static int git_config_copy_or_rename_section_in_file(const char *config_filename\n \t\t\t\t\t\toutput[0] = '\\t';\n \t\t\t\t\t}\n \t\t\t\t} else {\n-\t\t\t\t\tcopystr = store_create_section(new_name);\n+\t\t\t\t\tcopystr = store_create_section(new_name, &store);\n \t\t\t\t}\n \t\t\t}\n \t\t\tremove = 0;\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344177","messageId":"6f7ed0721749001fa73dadfd271bac255ea8651d.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 11/15] config_set_store: rename some fields for consistency","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:13Z","receivedAt":"2018-04-09T08:32:27Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The `seen` field is the actual length of the `offset` array, and the\n`offset_alloc` field records what was allocated (to avoid resizing\nwherever `seen` has to be incremented).\n\nElsewhere, we use the convention `name` for the array, where `name` is\ndescriptive enough to guess its purpose, `name_nr` for the actual length\nand `name_alloc` to record the maximum length without needing to resize.\n\nLet's make the names of the fields in question consistent with that\nconvention.\n\nThis will also help with the next steps where we will let the\ngit_config_set() machinery use the config event stream that we just\nintroduced.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 63 ++++++++++++++++++++++++++++----------------------------\n 1 file changed, 31 insertions(+), 32 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 0c0a965267d..2341620c11a 100644\n--- a/config.c\n+++ b/config.c\n@@ -2302,10 +2302,9 @@ struct config_store_data {\n \tint do_not_match;\n \tregex_t *value_regex;\n \tint multi_replace;\n-\tsize_t *offset;\n-\tunsigned int offset_alloc;\n+\tsize_t *seen;\n+\tunsigned int seen_nr, seen_alloc;\n \tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n-\tunsigned int seen;\n };\n \n static int matches(const char *key, const char *value,\n@@ -2331,15 +2330,15 @@ static int store_aux(const char *key, const char *value, void *cb)\n \tswitch (store->state) {\n \tcase KEY_SEEN:\n \t\tif (matches(key, value, store)) {\n-\t\t\tif (store->seen == 1 && store->multi_replace == 0) {\n+\t\t\tif (store->seen_nr == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n \t\t\t}\n \n-\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n-\t\t\t\t store->offset_alloc);\n+\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t\t store->seen_alloc);\n \n-\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n-\t\t\tstore->seen++;\n+\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n+\t\t\tstore->seen_nr++;\n \t\t}\n \t\tbreak;\n \tcase SECTION_SEEN:\n@@ -2365,26 +2364,26 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t * Do not increment matches: this is no match, but we\n \t\t * just made sure we are in the desired section.\n \t\t */\n-\t\tALLOC_GROW(store->offset, store->seen + 1,\n-\t\t\t store->offset_alloc);\n-\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t store->seen_alloc);\n+\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n \t\t/* fallthru */\n \tcase SECTION_END_SEEN:\n \tcase START:\n \t\tif (matches(key, value, store)) {\n-\t\t\tALLOC_GROW(store->offset, store->seen + 1,\n-\t\t\t\t store->offset_alloc);\n-\t\t\tstore->offset[store->seen] = cf->do_ftell(cf);\n+\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t\t store->seen_alloc);\n+\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n \t\t\tstore->state = KEY_SEEN;\n-\t\t\tstore->seen++;\n+\t\t\tstore->seen_nr++;\n \t\t} else {\n \t\t\tif (strrchr(key, '.') - key == store->baselen &&\n \t\t\t !strncmp(key, store->key, store->baselen)) {\n \t\t\t\t\tstore->state = SECTION_SEEN;\n-\t\t\t\t\tALLOC_GROW(store->offset,\n-\t\t\t\t\t\t store->seen + 1,\n-\t\t\t\t\t\t store->offset_alloc);\n-\t\t\t\t\tstore->offset[store->seen] =\n+\t\t\t\t\tALLOC_GROW(store->seen,\n+\t\t\t\t\t\t store->seen_nr + 1,\n+\t\t\t\t\t\t store->seen_alloc);\n+\t\t\t\t\tstore->seen[store->seen_nr] =\n \t\t\t\t\t\tcf->do_ftell(cf);\n \t\t\t}\n \t\t}\n@@ -2644,10 +2643,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t}\n \t\t}\n \n-\t\tALLOC_GROW(store.offset, 1, store.offset_alloc);\n-\t\tstore.offset[0] = 0;\n+\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n+\t\tstore.seen[0] = 0;\n \t\tstore.state = START;\n-\t\tstore.seen = 0;\n+\t\tstore.seen_nr = 0;\n \n \t\t/*\n \t\t * After this, store.offset will contain the *end* offset\n@@ -2675,8 +2674,8 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t}\n \n \t\t/* if nothing to unset, or too many matches, error out */\n-\t\tif ((store.seen == 0 && value == NULL) ||\n-\t\t\t\t(store.seen > 1 && multi_replace == 0)) {\n+\t\tif ((store.seen_nr == 0 && value == NULL) ||\n+\t\t (store.seen_nr > 1 && multi_replace == 0)) {\n \t\t\tret = CONFIG_NOTHING_SET;\n \t\t\tgoto out_free;\n \t\t}\n@@ -2707,19 +2706,19 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tgoto out_free;\n \t\t}\n \n-\t\tif (store.seen == 0)\n-\t\t\tstore.seen = 1;\n+\t\tif (store.seen_nr == 0)\n+\t\t\tstore.seen_nr = 1;\n \n-\t\tfor (i = 0, copy_begin = 0; i < store.seen; i++) {\n+\t\tfor (i = 0, copy_begin = 0; i < store.seen_nr; i++) {\n \t\t\tnew_line = 0;\n-\t\t\tif (store.offset[i] == 0) {\n-\t\t\t\tstore.offset[i] = copy_end = contents_sz;\n+\t\t\tif (store.seen[i] == 0) {\n+\t\t\t\tstore.seen[i] = copy_end = contents_sz;\n \t\t\t} else if (store.state != KEY_SEEN) {\n-\t\t\t\tcopy_end = store.offset[i];\n+\t\t\t\tcopy_end = store.seen[i];\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n \t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.offset[i], &new_line);\n+\t\t\t\t\tstore.seen[i], &new_line);\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n@@ -2733,7 +2732,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\t write_str_in_full(fd, \"\\n\") < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-\t\t\tcopy_begin = store.offset[i];\n+\t\t\tcopy_begin = store.seen[i];\n \t\t}\n \n \t\t/* write the pair (value == NULL means unset) */\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344178","messageId":"3e792cdb73935da823175ce745715fa77c376402.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 12/15] git_config_set: do not use a state machine","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:17Z","receivedAt":"2018-04-09T08:32:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"While a neat theoretical construct, state machines are hard to read. In\nthis instance, it does not even make a whole lot of sense because we are\nmore interested in flags, anyway: has the section been seen? Has the key\nbeen seen? Does the current section match the key we are looking for?\n\nBesides, the state `SECTION_SEEN` was named in a misleading way: it did\nnot indicate that we saw the section matching the key we are looking\nfor, but it instead indicated that we are *currently* in that section.\n\nLet's just replace the state machine logic by clear and obvious flags.\n\nThis will also make it easier to review the upcoming patches to use the\nnewly-introduced `event_fn` callback of the config parser.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 59 ++++++++++++++++++++++++++++----------------------------\n 1 file changed, 29 insertions(+), 30 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 2341620c11a..3f1cbfa181e 100644\n--- a/config.c\n+++ b/config.c\n@@ -2304,7 +2304,7 @@ struct config_store_data {\n \tint multi_replace;\n \tsize_t *seen;\n \tunsigned int seen_nr, seen_alloc;\n-\tenum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;\n+\tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n };\n \n static int matches(const char *key, const char *value,\n@@ -2327,8 +2327,7 @@ static int store_aux(const char *key, const char *value, void *cb)\n \tsize_t section_len;\n \tstruct config_store_data *store = cb;\n \n-\tswitch (store->state) {\n-\tcase KEY_SEEN:\n+\tif (store->key_seen) {\n \t\tif (matches(key, value, store)) {\n \t\t\tif (store->seen_nr == 1 && store->multi_replace == 0) {\n \t\t\t\twarning(_(\"%s has multiple values\"), key);\n@@ -2340,8 +2339,8 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n \t\t\tstore->seen_nr++;\n \t\t}\n-\t\tbreak;\n-\tcase SECTION_SEEN:\n+\t\treturn 0;\n+\t} else if (store->is_keys_section) {\n \t\t/*\n \t\t * What we are looking for is in store->key (both\n \t\t * section and var), and its section part is baselen\n@@ -2356,10 +2355,9 @@ static int store_aux(const char *key, const char *value, void *cb)\n \n \t\tif ((section_len != store->baselen) ||\n \t\t memcmp(key, store->key, section_len+1)) {\n-\t\t\tstore->state = SECTION_END_SEEN;\n-\t\t\tbreak;\n+\t\t\tstore->is_keys_section = 0;\n+\t\t\treturn 0;\n \t\t}\n-\n \t\t/*\n \t\t * Do not increment matches: this is no match, but we\n \t\t * just made sure we are in the desired section.\n@@ -2367,27 +2365,29 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n \t\t\t store->seen_alloc);\n \t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t\t/* fallthru */\n-\tcase SECTION_END_SEEN:\n-\tcase START:\n-\t\tif (matches(key, value, store)) {\n-\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n-\t\t\t\t store->seen_alloc);\n-\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t\t\tstore->state = KEY_SEEN;\n-\t\t\tstore->seen_nr++;\n-\t\t} else {\n-\t\t\tif (strrchr(key, '.') - key == store->baselen &&\n-\t\t\t !strncmp(key, store->key, store->baselen)) {\n-\t\t\t\t\tstore->state = SECTION_SEEN;\n-\t\t\t\t\tALLOC_GROW(store->seen,\n-\t\t\t\t\t\t store->seen_nr + 1,\n-\t\t\t\t\t\t store->seen_alloc);\n-\t\t\t\t\tstore->seen[store->seen_nr] =\n-\t\t\t\t\t\tcf->do_ftell(cf);\n-\t\t\t}\n+\t}\n+\n+\tif (matches(key, value, store)) {\n+\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t store->seen_alloc);\n+\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n+\t\tstore->seen_nr++;\n+\t\tstore->key_seen = 1;\n+\t\tstore->section_seen = 1;\n+\t\tstore->is_keys_section = 1;\n+\t} else {\n+\t\tif (strrchr(key, '.') - key == store->baselen &&\n+\t\t !strncmp(key, store->key, store->baselen)) {\n+\t\t\t\tstore->section_seen = 1;\n+\t\t\t\tstore->is_keys_section = 1;\n+\t\t\t\tALLOC_GROW(store->seen,\n+\t\t\t\t\t store->seen_nr + 1,\n+\t\t\t\t\t store->seen_alloc);\n+\t\t\t\tstore->seen[store->seen_nr] =\n+\t\t\t\t\tcf->do_ftell(cf);\n \t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -2645,7 +2645,6 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n \t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n \t\tstore.seen[0] = 0;\n-\t\tstore.state = START;\n \t\tstore.seen_nr = 0;\n \n \t\t/*\n@@ -2713,7 +2712,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tnew_line = 0;\n \t\t\tif (store.seen[i] == 0) {\n \t\t\t\tstore.seen[i] = copy_end = contents_sz;\n-\t\t\t} else if (store.state != KEY_SEEN) {\n+\t\t\t} else if (!store.key_seen) {\n \t\t\t\tcopy_end = store.seen[i];\n \t\t\t} else\n \t\t\t\tcopy_end = find_beginning_of_line(\n@@ -2737,7 +2736,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n \t\t/* write the pair (value == NULL means unset) */\n \t\tif (value != NULL) {\n-\t\t\tif (store.state == START) {\n+\t\t\tif (!store.section_seen) {\n \t\t\t\tif (write_section(fd, key, &store) < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344179","messageId":"a187a526f46b008c2e81dad6ef90c7d2ddc56044.1523262449.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 13/15] git_config_set: make use of the config parser's event stream","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:20Z","receivedAt":"2018-04-09T08:32:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"In the recent commit with the title \"config: introduce an optional event\nstream while parsing\", we introduced an optional callback to keep track\nof the config parser's events \"comment\", \"white-space\", \"section header\"\nand \"entry\".\n\nOne motivation for this feature was to make use of it in the code that\nedits the config. And this commit makes it so.\n\nNote: this patch changes the meaning of the `seen` array that records\nwhether we saw the config entry that is to be edited: previously, it\ncontained the end offset of the found entry. Now, we introduce a new\narray `parsed` that keeps a record of *all* config parser events (with\nbegin/end offsets), and the items in the `seen` array now point into the\n`parsed` array.\n\nThere are two reasons why we do it this way:\n\n1. To keep the implementation simple, the config parser's event stream\n reports the event only after the config callback was called, so we\n would not receive the begin offset otherwise.\n\n2. In the following patches, we will re-use the `parsed` array to fix two\n long-standing bugs related to empty sections.\n\nNote that this also makes the code more robust with respect to finding the\nbegin offset of the part(s) of the config file to be edited, as we no\nlonger back-track to find the beginning of the line.\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 170 ++++++++++++++++++++++++++-----------------------------\n 1 file changed, 81 insertions(+), 89 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 3f1cbfa181e..72d71fc9a4e 100644\n--- a/config.c\n+++ b/config.c\n@@ -2302,8 +2302,11 @@ struct config_store_data {\n \tint do_not_match;\n \tregex_t *value_regex;\n \tint multi_replace;\n-\tsize_t *seen;\n-\tunsigned int seen_nr, seen_alloc;\n+\tstruct {\n+\t\tsize_t begin, end;\n+\t\tenum config_event_t type;\n+\t} *parsed;\n+\tunsigned int parsed_nr, parsed_alloc, *seen, seen_nr, seen_alloc;\n \tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n };\n \n@@ -2321,10 +2324,31 @@ static int matches(const char *key, const char *value,\n \t\t(value && !regexec(store->value_regex, value, 0, NULL, 0));\n }\n \n+static int store_aux_event(enum config_event_t type,\n+\t\t\t size_t begin, size_t end, void *data)\n+{\n+\tstruct config_store_data *store = data;\n+\n+\tALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);\n+\tstore->parsed[store->parsed_nr].begin = begin;\n+\tstore->parsed[store->parsed_nr].end = end;\n+\tstore->parsed[store->parsed_nr].type = type;\n+\tstore->parsed_nr++;\n+\n+\tif (type == CONFIG_EVENT_SECTION) {\n+\t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n+\t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n+\n+\t\t/* Is this the section we were looking for? */\n+\t\tstore->is_keys_section = cf->var.len - 1 == store->baselen &&\n+\t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int store_aux(const char *key, const char *value, void *cb)\n {\n-\tconst char *ep;\n-\tsize_t section_len;\n \tstruct config_store_data *store = cb;\n \n \tif (store->key_seen) {\n@@ -2336,55 +2360,21 @@ static int store_aux(const char *key, const char *value, void *cb)\n \t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n \t\t\t\t store->seen_alloc);\n \n-\t\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n+\t\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n \t\t\tstore->seen_nr++;\n \t\t}\n-\t\treturn 0;\n \t} else if (store->is_keys_section) {\n \t\t/*\n-\t\t * What we are looking for is in store->key (both\n-\t\t * section and var), and its section part is baselen\n-\t\t * long. We found key (again, both section and var).\n-\t\t * We would want to know if this key is in the same\n-\t\t * section as what we are looking for. We already\n-\t\t * know we are in the same section as what should\n-\t\t * hold store->key.\n+\t\t * Do not increment matches yet: this may not be a match, but we\n+\t\t * are in the desired section.\n \t\t */\n-\t\tep = strrchr(key, '.');\n-\t\tsection_len = ep - key;\n-\n-\t\tif ((section_len != store->baselen) ||\n-\t\t memcmp(key, store->key, section_len+1)) {\n-\t\t\tstore->is_keys_section = 0;\n-\t\t\treturn 0;\n-\t\t}\n-\t\t/*\n-\t\t * Do not increment matches: this is no match, but we\n-\t\t * just made sure we are in the desired section.\n-\t\t */\n-\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n-\t\t\t store->seen_alloc);\n-\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t}\n-\n-\tif (matches(key, value, store)) {\n-\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n-\t\t\t store->seen_alloc);\n-\t\tstore->seen[store->seen_nr] = cf->do_ftell(cf);\n-\t\tstore->seen_nr++;\n-\t\tstore->key_seen = 1;\n+\t\tALLOC_GROW(store->seen, store->seen_nr + 1, store->seen_alloc);\n+\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n \t\tstore->section_seen = 1;\n-\t\tstore->is_keys_section = 1;\n-\t} else {\n-\t\tif (strrchr(key, '.') - key == store->baselen &&\n-\t\t !strncmp(key, store->key, store->baselen)) {\n-\t\t\t\tstore->section_seen = 1;\n-\t\t\t\tstore->is_keys_section = 1;\n-\t\t\t\tALLOC_GROW(store->seen,\n-\t\t\t\t\t store->seen_nr + 1,\n-\t\t\t\t\t store->seen_alloc);\n-\t\t\t\tstore->seen[store->seen_nr] =\n-\t\t\t\t\tcf->do_ftell(cf);\n+\n+\t\tif (matches(key, value, store)) {\n+\t\t\tstore->seen_nr++;\n+\t\t\tstore->key_seen = 1;\n \t\t}\n \t}\n \n@@ -2485,32 +2475,6 @@ static ssize_t write_pair(int fd, const char *key, const char *value,\n \treturn ret;\n }\n \n-static ssize_t find_beginning_of_line(const char *contents, size_t size,\n-\tsize_t offset_, int *found_bracket)\n-{\n-\tsize_t equal_offset = size, bracket_offset = size;\n-\tssize_t offset;\n-\n-contline:\n-\tfor (offset = offset_-2; offset > 0\n-\t\t\t&& contents[offset] != '\\n'; offset--)\n-\t\tswitch (contents[offset]) {\n-\t\t\tcase '=': equal_offset = offset; break;\n-\t\t\tcase ']': bracket_offset = offset; break;\n-\t\t}\n-\tif (offset > 0 && contents[offset-1] == '\\\\') {\n-\t\toffset_ = offset;\n-\t\tgoto contline;\n-\t}\n-\tif (bracket_offset < equal_offset) {\n-\t\t*found_bracket = 1;\n-\t\toffset = bracket_offset+1;\n-\t} else\n-\t\toffset++;\n-\n-\treturn offset;\n-}\n-\n int git_config_set_in_file_gently(const char *config_filename,\n \t\t\t\t const char *key, const char *value)\n {\n@@ -2621,6 +2585,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\tstruct stat st;\n \t\tsize_t copy_begin, copy_end;\n \t\tint i, new_line = 0;\n+\t\tstruct config_options opts;\n \n \t\tif (value_regex == NULL)\n \t\t\tstore.value_regex = NULL;\n@@ -2643,17 +2608,24 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t}\n \t\t}\n \n-\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n-\t\tstore.seen[0] = 0;\n-\t\tstore.seen_nr = 0;\n+\t\tALLOC_GROW(store.parsed, 1, store.parsed_alloc);\n+\t\tstore.parsed[0].end = 0;\n+\n+\t\tmemset(&opts, 0, sizeof(opts));\n+\t\topts.event_fn = store_aux_event;\n+\t\topts.event_fn_data = &store;\n \n \t\t/*\n-\t\t * After this, store.offset will contain the *end* offset\n-\t\t * of the last match, or remain at 0 if no match was found.\n+\t\t * After this, store.parsed will contain offsets of all the\n+\t\t * parsed elements, and store.seen will contain a list of\n+\t\t * matches, as indices into store.parsed.\n+\t\t *\n \t\t * As a side effect, we make sure to transform only a valid\n \t\t * existing config file.\n \t\t */\n-\t\tif (git_config_from_file(store_aux, config_filename, &store)) {\n+\t\tif (git_config_from_file_with_options(store_aux,\n+\t\t\t\t\t\t config_filename,\n+\t\t\t\t\t\t &store, &opts)) {\n \t\t\terror(\"invalid config file %s\", config_filename);\n \t\t\tfree(store.key);\n \t\t\tif (store.value_regex != NULL &&\n@@ -2705,19 +2677,39 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\tgoto out_free;\n \t\t}\n \n-\t\tif (store.seen_nr == 0)\n+\t\tif (store.seen_nr == 0) {\n+\t\t\tif (!store.seen_alloc) {\n+\t\t\t\t/* Did not see key nor section */\n+\t\t\t\tALLOC_GROW(store.seen, 1, store.seen_alloc);\n+\t\t\t\tstore.seen[0] = store.parsed_nr\n+\t\t\t\t\t- !!store.parsed_nr;\n+\t\t\t}\n \t\t\tstore.seen_nr = 1;\n+\t\t}\n \n \t\tfor (i = 0, copy_begin = 0; i < store.seen_nr; i++) {\n+\t\t\tsize_t replace_end;\n+\t\t\tint j = store.seen[i];\n+\n \t\t\tnew_line = 0;\n-\t\t\tif (store.seen[i] == 0) {\n-\t\t\t\tstore.seen[i] = copy_end = contents_sz;\n-\t\t\t} else if (!store.key_seen) {\n-\t\t\t\tcopy_end = store.seen[i];\n-\t\t\t} else\n-\t\t\t\tcopy_end = find_beginning_of_line(\n-\t\t\t\t\tcontents, contents_sz,\n-\t\t\t\t\tstore.seen[i], &new_line);\n+\t\t\tif (!store.key_seen) {\n+\t\t\t\treplace_end = copy_end = store.parsed[j].end;\n+\t\t\t} else {\n+\t\t\t\treplace_end = store.parsed[j].end;\n+\t\t\t\tcopy_end = store.parsed[j].begin;\n+\t\t\t\t/*\n+\t\t\t\t * Swallow preceding white-space on the same\n+\t\t\t\t * line.\n+\t\t\t\t */\n+\t\t\t\twhile (copy_end > 0 ) {\n+\t\t\t\t\tchar c = contents[copy_end - 1];\n+\n+\t\t\t\t\tif (isspace(c) && c != '\\n')\n+\t\t\t\t\t\tcopy_end--;\n+\t\t\t\t\telse\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n \n \t\t\tif (copy_end > 0 && contents[copy_end-1] != '\\n')\n \t\t\t\tnew_line = 1;\n@@ -2731,7 +2723,7 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t\t write_str_in_full(fd, \"\\n\") < 0)\n \t\t\t\t\tgoto write_err_out;\n \t\t\t}\n-\t\t\tcopy_begin = store.seen[i];\n+\t\t\tcopy_begin = replace_end;\n \t\t}\n \n \t\t/* write the pair (value == NULL means unset) */\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344180","messageId":"7f3d1b7520622801ccdf1bc9898eb7e9b4e78c8e.1523262450.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 14/15] git config --unset: remove empty sections (in the common case)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:24Z","receivedAt":"2018-04-09T08:32:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The original reasoning for not removing section headers upon removal of\nthe last entry went like this: the user could have added comments about\nthe section, or about the entries therein, and if there were other\ncomments there, we would not know whether we should remove them.\n\nIn particular, a concocted example was presented that looked like this\n(and was added to t1300):\n\n\t# some generic comment on the configuration file itself\n\t# a comment specific to this \"section\" section.\n\t[section]\n\t# some intervening lines\n\t# that should also be dropped\n\n\tkey = value\n\t# please be careful when you update the above variable\n\nThe ideal thing for `git config --unset section.key` in this case would\nbe to leave only the first line behind, because all the other comments\nare now obsolete.\n\nHowever, this is unfeasible, short of adding a complete Natural Language\nProcessing module to Git, which seems not only a lot of work, but a\ntotally unreasonable feature (for little benefit to most users).\n\nNow, the real kicker about this problem is: most users do not edit their\nconfig files at all! In their use case, the config looks like this\ninstead:\n\n\t[section]\n\t\tkey = value\n\n... and it is totally obvious what should happen if the entry is\nremoved: the entire section should vanish.\n\nLet's generalize this observation to this conservative strategy: if we\nare removing the last entry from a section, and there are no comments\ninside that section nor surrounding it, then remove the entire section.\nOtherwise behave as before: leave the now-empty section (including those\ncomments, even ones about the now-deleted entry).\n\nWe have to be extra careful to handle the case where more than one entry\nis removed: any subset of them might be the last entries of their\nrespective sections (and if there are no comments in or around that\nsection, the section should be removed, too).\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++-\n t/t1300-config.sh | 4 +-\n 2 files changed, 93 insertions(+), 4 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 72d71fc9a4e..2c7a10acdaa 100644\n--- a/config.c\n+++ b/config.c\n@@ -2305,6 +2305,7 @@ struct config_store_data {\n \tstruct {\n \t\tsize_t begin, end;\n \t\tenum config_event_t type;\n+\t\tint is_keys_section;\n \t} *parsed;\n \tunsigned int parsed_nr, parsed_alloc, *seen, seen_nr, seen_alloc;\n \tunsigned int key_seen:1, section_seen:1, is_keys_section:1;\n@@ -2333,17 +2334,20 @@ static int store_aux_event(enum config_event_t type,\n \tstore->parsed[store->parsed_nr].begin = begin;\n \tstore->parsed[store->parsed_nr].end = end;\n \tstore->parsed[store->parsed_nr].type = type;\n-\tstore->parsed_nr++;\n \n \tif (type == CONFIG_EVENT_SECTION) {\n \t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n \t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n \n \t\t/* Is this the section we were looking for? */\n-\t\tstore->is_keys_section = cf->var.len - 1 == store->baselen &&\n+\t\tstore->is_keys_section =\n+\t\t\tstore->parsed[store->parsed_nr].is_keys_section =\n+\t\t\tcf->var.len - 1 == store->baselen &&\n \t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n \t}\n \n+\tstore->parsed_nr++;\n+\n \treturn 0;\n }\n \n@@ -2475,6 +2479,87 @@ static ssize_t write_pair(int fd, const char *key, const char *value,\n \treturn ret;\n }\n \n+/*\n+ * If we are about to unset the last key(s) in a section, and if there are\n+ * no comments surrounding (or included in) the section, we will want to\n+ * extend begin/end to remove the entire section.\n+ *\n+ * Note: the parameter `seen_ptr` points to the index into the store.seen\n+ * array. * This index may be incremented if a section has more than one\n+ * entry (which all are to be removed).\n+ */\n+static void maybe_remove_section(struct config_store_data *store,\n+\t\t\t\t const char *contents,\n+\t\t\t\t size_t *begin_offset, size_t *end_offset,\n+\t\t\t\t int *seen_ptr)\n+{\n+\tsize_t begin;\n+\tint i, seen, section_seen = 0;\n+\n+\t/*\n+\t * First, ensure that this is the first key, and that there are no\n+\t * comments before the entry nor before the section header.\n+\t */\n+\tseen = *seen_ptr;\n+\tfor (i = store->seen[seen]; i > 0; i--) {\n+\t\tenum config_event_t type = store->parsed[i - 1].type;\n+\n+\t\tif (type == CONFIG_EVENT_COMMENT)\n+\t\t\t/* There is a comment before this entry or section */\n+\t\t\treturn;\n+\t\tif (type == CONFIG_EVENT_ENTRY) {\n+\t\t\tif (!section_seen)\n+\t\t\t\t/* This is not the section's first entry. */\n+\t\t\t\treturn;\n+\t\t\t/* We encountered no comment before the section. */\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (type == CONFIG_EVENT_SECTION) {\n+\t\t\tif (!store->parsed[i - 1].is_keys_section)\n+\t\t\t\tbreak;\n+\t\t\tsection_seen = 1;\n+\t\t}\n+\t}\n+\tbegin = store->parsed[i].begin;\n+\n+\t/*\n+\t * Next, make sure that we are removing he last key(s) in the section,\n+\t * and that there are no comments that are possibly about the current\n+\t * section.\n+\t */\n+\tfor (i = store->seen[seen] + 1; i < store->parsed_nr; i++) {\n+\t\tenum config_event_t type = store->parsed[i].type;\n+\n+\t\tif (type == CONFIG_EVENT_COMMENT)\n+\t\t\treturn;\n+\t\tif (type == CONFIG_EVENT_SECTION) {\n+\t\t\tif (store->parsed[i].is_keys_section)\n+\t\t\t\tcontinue;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (type == CONFIG_EVENT_ENTRY) {\n+\t\t\tif (++seen < store->seen_nr &&\n+\t\t\t i == store->seen[seen])\n+\t\t\t\t/* We want to remove this entry, too */\n+\t\t\t\tcontinue;\n+\t\t\t/* There is another entry in this section. */\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We are really removing the last entry/entries from this section, and\n+\t * there are no enclosed or surrounding comments. Remove the entire,\n+\t * now-empty section.\n+\t */\n+\t*seen_ptr = seen;\n+\t*begin_offset = begin;\n+\tif (i < store->parsed_nr)\n+\t\t*end_offset = store->parsed[i].begin;\n+\telse\n+\t\t*end_offset = store->parsed[store->parsed_nr - 1].end;\n+}\n+\n int git_config_set_in_file_gently(const char *config_filename,\n \t\t\t\t const char *key, const char *value)\n {\n@@ -2697,6 +2782,10 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \t\t\t} else {\n \t\t\t\treplace_end = store.parsed[j].end;\n \t\t\t\tcopy_end = store.parsed[j].begin;\n+\t\t\t\tif (!value)\n+\t\t\t\t\tmaybe_remove_section(&store, contents,\n+\t\t\t\t\t\t\t ©_end,\n+\t\t\t\t\t\t\t &replace_end, &i);\n \t\t\t\t/*\n \t\t\t\t * Swallow preceding white-space on the same\n \t\t\t\t * line.\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 9d23a8ca972..d973fd53398 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1413,7 +1413,7 @@ test_expect_success 'urlmatch with wildcard' '\n '\n \n # good section hygiene\n-test_expect_failure '--unset last key removes section (except if commented)' '\n+test_expect_success '--unset last key removes section (except if commented)' '\n \tcat >.git/config <<-\\EOF &&\n \t# some generic comment on the configuration file itself\n \t# a comment specific to this \"section\" section.\n@@ -1495,7 +1495,7 @@ test_expect_failure '--unset last key removes section (except if commented)' '\n \ttest_line_count = 3 .git/config\n '\n \n-test_expect_failure '--unset-all removes section if empty & uncommented' '\n+test_expect_success '--unset-all removes section if empty & uncommented' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n \tkey = value1\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n\n\n"},{"id":"344181","messageId":"f0d3f88b7c1bf34cd880650efb11bc2a34509ab4.1523262450.git.johannes.schindelin@gmx.de","threadId":"48185","inReplyTo":"cover.1523262449.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 15/15] git_config_set: reuse empty sections","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-09T08:32:29Z","receivedAt":"2018-04-09T08:32:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"It can happen quite easily that the last setting in a config section is\nremoved, and to avoid confusion when there are comments in the config\nabout that section, we keep a lone section header, i.e. an empty\nsection.\n\nNow that we use the `event_fn` callback, it is easy to add support for\nre-using empty sections, so let's do that.\n\nNote: t5512-ls-remote requires that this change is applied *after* the\npatch \"git config --unset: remove empty sections (in the common case)\":\nwithout that patch, there would be empty `transfer` and `uploadpack`\nsections ready for reuse, but in the *wrong* order (and sconsequently,\nt5512's \"overrides work between mixed transfer/upload-pack hideRefs\"\nwould fail).\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 14 +++++++++++++-\n t/t1300-config.sh | 2 +-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 2c7a10acdaa..6155d0651bd 100644\n--- a/config.c\n+++ b/config.c\n@@ -2344,6 +2344,12 @@ static int store_aux_event(enum config_event_t type,\n \t\t\tstore->parsed[store->parsed_nr].is_keys_section =\n \t\t\tcf->var.len - 1 == store->baselen &&\n \t\t\t!strncasecmp(cf->var.buf, store->key, store->baselen);\n+\t\tif (store->is_keys_section) {\n+\t\t\tstore->section_seen = 1;\n+\t\t\tALLOC_GROW(store->seen, store->seen_nr + 1,\n+\t\t\t\t store->seen_alloc);\n+\t\t\tstore->seen[store->seen_nr] = store->parsed_nr;\n+\t\t}\n \t}\n \n \tstore->parsed_nr++;\n@@ -2778,7 +2784,13 @@ int git_config_set_multivar_in_file_gently(const char *config_filename,\n \n \t\t\tnew_line = 0;\n \t\t\tif (!store.key_seen) {\n-\t\t\t\treplace_end = copy_end = store.parsed[j].end;\n+\t\t\t\tcopy_end = store.parsed[j].end;\n+\t\t\t\t/* include '\\n' when copying section header */\n+\t\t\t\tif (copy_end > 0 && copy_end < contents_sz &&\n+\t\t\t\t contents[copy_end - 1] != '\\n' &&\n+\t\t\t\t contents[copy_end] == '\\n')\n+\t\t\t\t\tcopy_end++;\n+\t\t\t\treplace_end = copy_end;\n \t\t\t} else {\n \t\t\t\treplace_end = store.parsed[j].end;\n \t\t\t\tcopy_end = store.parsed[j].begin;\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex d973fd53398..eef0bbe4f9f 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1506,7 +1506,7 @@ test_expect_success '--unset-all removes section if empty & uncommented' '\n \ttest_line_count = 0 .git/config\n '\n \n-test_expect_failure 'adding a key into an empty section reuses header' '\n+test_expect_success 'adding a key into an empty section reuses header' '\n \tcat >.git/config <<-\\EOF &&\n \t[section]\n \tEOF\n-- \n2.17.0.windows.1.4.g7e4058d72e3\n"},{"id":"346983","messageId":"20180508134248.GA25513@sigill.intra.peff.net","threadId":"48185","inReplyTo":"a187a526f46b008c2e81dad6ef90c7d2ddc56044.1523262449.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v3 13/15] git_config_set: make use of the config parser's event stream","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-05-08T13:42:48Z","receivedAt":"2018-05-08T13:42:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 09, 2018 at 10:32:20AM +0200, Johannes Schindelin wrote:\n\n> +static int store_aux_event(enum config_event_t type,\n> +\t\t\t size_t begin, size_t end, void *data)\n> +{\n> +\tstruct config_store_data *store = data;\n> +\n> +\tALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);\n> +\tstore->parsed[store->parsed_nr].begin = begin;\n> +\tstore->parsed[store->parsed_nr].end = end;\n> +\tstore->parsed[store->parsed_nr].type = type;\n> +\tstore->parsed_nr++;\n> +\n> +\tif (type == CONFIG_EVENT_SECTION) {\n> +\t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n> +\t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n\nI triggered this BUG today while playing around. Here's a minimal\nreproduction:\n\n echo '[broken' >config\n git config --file=config a.b c\n\nI'm not sure if it should simply be a die() and not a BUG(), since\nit depends on the input. Or if it is a BUG and we expected an earlier\npart of the code (like the event generator) to catch this broken case\nbefore we get to this function.\n\n-Peff\n"},{"id":"346984","messageId":"20180508140054.GA27712@sigill.intra.peff.net","threadId":"48185","inReplyTo":"20180508134248.GA25513@sigill.intra.peff.net","subject":"Re: [PATCH v3 13/15] git_config_set: make use of the config parser's event stream","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-05-08T14:00:54Z","receivedAt":"2018-05-08T14:00:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 08, 2018 at 09:42:48AM -0400, Jeff King wrote:\n\n> On Mon, Apr 09, 2018 at 10:32:20AM +0200, Johannes Schindelin wrote:\n> \n> > +static int store_aux_event(enum config_event_t type,\n> > +\t\t\t size_t begin, size_t end, void *data)\n> > +{\n> > +\tstruct config_store_data *store = data;\n> > +\n> > +\tALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);\n> > +\tstore->parsed[store->parsed_nr].begin = begin;\n> > +\tstore->parsed[store->parsed_nr].end = end;\n> > +\tstore->parsed[store->parsed_nr].type = type;\n> > +\tstore->parsed_nr++;\n> > +\n> > +\tif (type == CONFIG_EVENT_SECTION) {\n> > +\t\tif (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')\n> > +\t\t\tBUG(\"Invalid section name '%s'\", cf->var.buf);\n> \n> I triggered this BUG today while playing around. Here's a minimal\n> reproduction:\n> \n> echo '[broken' >config\n> git config --file=config a.b c\n> \n> I'm not sure if it should simply be a die() and not a BUG(), since\n> it depends on the input. Or if it is a BUG and we expected an earlier\n> part of the code (like the event generator) to catch this broken case\n> before we get to this function.\n\nBy the way, one side effect of BUG() here is that we call abort(), which\nmeans that our atexit handlers don't run. And a crufty \"config.lock\"\nfile is left that prevents running the command again.\n\nIn our discussion elsewhere of having BUG() just call exit(), I'm not\nsure if we'd want it to skip those cleanups or not (it's helpful to\nnot run them if you're trying to debug, but otherwise is annoying).\n\n-Peff\n"}]}