# [PATCH 0/5] Replacing strbuf_getline_lf() by strbuf_getline() on trimmed input 14 messages from 2016-01-30 to 2016-02-22. Participants: Moritz Neeb, Junio C Hamano. Thread: https://gitlist.dev/t/41291 ## Moritz Neeb, 2016-01-30 17:51 Subject: [PATCH 0/5] Replacing strbuf_getline_lf() by strbuf_getline() on trimmed input Message-ID: <56ACF82B.2030005@moritzneeb.de> URL: https://gitlist.dev/e/56ACF82B.2030005%40moritzneeb.de ``` This series deals with strbuf_getline_lf() in certain codepaths: Those, where the input that is read, is trimmed before doing anything that could possibly expect a CR character. As the codepath after trimming can not in any way expect a CR at the end of the line, because trim dropped it before the change, this can be safely done. The series is an idea out of [1], where Junio proposed to replace the calls to strbuf_getline_lf() because it 'would [be] a good way to document them as dealing with "text"'. Apart from a general review, I would be happy about comments how I can improve this (and future) patches, as this is my first one. Especially I am not sure about how many arguments from the discussion on the mailing list should be put into the commits. -Moritz [1] http://thread.gmane.org/gmane.comp.version-control.git/284104 Moritz Neeb (5): bisect: read bisect paths with strbuf_getline() clean: read user input with strbuf_getline() notes: read copied notes with strbuf_getline() remote: read $GIT_DIR/branches/* with strbuf_getline() wt-status: read rebase todolist with strbuf_getline() bisect.c | 2 +- builtin/clean.c | 6 +++--- builtin/notes.c | 2 +- remote.c | 2 +- wt-status.c | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) -- 2.4.3 ``` ## Moritz Neeb, 2016-01-30 18:03 Subject: [PATCH 1/5] bisect: read bisect paths with strbuf_getline() Message-ID: <56ACFB03.3020000@moritzneeb.de> URL: https://gitlist.dev/e/56ACFB03.3020000%40moritzneeb.de In-Reply-To: <56ACF82B.2030005@moritzneeb.de> ``` The lines read from BISECT_NAMES are trimmed with strbuf_trim() immediately. There is thus no logic expecting CR, so strbuf_getline_lf() can be replaced by its CRLF counterpart. Signed-off-by: Moritz Neeb --- bisect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bisect.c b/bisect.c index 06ec54e..bf7c885 100644 --- a/bisect.c +++ b/bisect.c @@ -440,7 +440,7 @@ static void read_bisect_paths(struct argv_array *array) if (!fp) die_errno("Could not open file '%s'", filename); - while (strbuf_getline_lf(&str, fp) != EOF) { + while (strbuf_getline(&str, fp) != EOF) { strbuf_trim(&str); if (sq_dequote_to_argv_array(str.buf, array)) die("Badly quoted content in file '%s': %s", -- 2.4.3 ``` ## Moritz Neeb, 2016-01-30 18:04 Subject: [PATCH 2/5] clean: read user input with strbuf_getline() Message-ID: <56ACFB3D.6010309@moritzneeb.de> URL: https://gitlist.dev/e/56ACFB3D.6010309%40moritzneeb.de In-Reply-To: <56ACF82B.2030005@moritzneeb.de> ``` The input that is read is trimmed with strbuf_trim() immediately. There is thus no logic expecting CR, so strbuf_getline_lf() can be replaced by its CRLF counterpart. Signed-off-by: Moritz Neeb --- builtin/clean.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/builtin/clean.c b/builtin/clean.c index 7b08237..956283d 100644 --- a/builtin/clean.c +++ b/builtin/clean.c @@ -570,7 +570,7 @@ static int *list_and_choose(struct menu_opts *opts, struct menu_stuff *stuff) clean_get_color(CLEAN_COLOR_RESET)); } - if (strbuf_getline_lf(&choice, stdin) != EOF) { + if (strbuf_getline(&choice, stdin) != EOF) { strbuf_trim(&choice); } else { eof = 1; @@ -652,7 +652,7 @@ static int filter_by_patterns_cmd(void) clean_print_color(CLEAN_COLOR_PROMPT); printf(_("Input ignore patterns>> ")); clean_print_color(CLEAN_COLOR_RESET); - if (strbuf_getline_lf(&confirm, stdin) != EOF) + if (strbuf_getline(&confirm, stdin) != EOF) strbuf_trim(&confirm); else putchar('\n'); @@ -750,7 +750,7 @@ static int ask_each_cmd(void) qname = quote_path_relative(item->string, NULL, &buf); /* TRANSLATORS: Make sure to keep [y/N] as is */ printf(_("Remove %s [y/N]? "), qname); - if (strbuf_getline_lf(&confirm, stdin) != EOF) { + if (strbuf_getline(&confirm, stdin) != EOF) { strbuf_trim(&confirm); } else { putchar('\n'); -- 2.4.3 ``` ## Moritz Neeb, 2016-01-30 18:05 Subject: [PATCH 3/5] notes: read copied notes with strbuf_getline() Message-ID: <56ACFB53.7070400@moritzneeb.de> URL: https://gitlist.dev/e/56ACFB53.7070400%40moritzneeb.de In-Reply-To: <56ACF82B.2030005@moritzneeb.de> ``` The notes that are copied from stdin are trimmed with strbuf_rtrim() after splitting by ' '. There is thus no logic expecting CR, so strbuf_getline_lf() can be replaced by its CRLF counterpart. Signed-off-by: Moritz Neeb --- builtin/notes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builtin/notes.c b/builtin/notes.c index ed6f222..bbd7544 100644 --- a/builtin/notes.c +++ b/builtin/notes.c @@ -290,7 +290,7 @@ static int notes_copy_from_stdin(int force, const char *rewrite_cmd) t = &default_notes_tree; } - while (strbuf_getline_lf(&buf, stdin) != EOF) { + while (strbuf_getline(&buf, stdin) != EOF) { unsigned char from_obj[20], to_obj[20]; struct strbuf **split; int err; -- 2.4.3 ``` ## Moritz Neeb, 2016-01-30 18:05 Subject: [PATCH 4/5] remote: read $GIT_DIR/branches/* with strbuf_getline() Message-ID: <56ACFB6D.4040804@moritzneeb.de> URL: https://gitlist.dev/e/56ACFB6D.4040804%40moritzneeb.de In-Reply-To: <56ACF82B.2030005@moritzneeb.de> ``` The line read from the branch file is directly trimmed after reading with strbuf_trim(). There is thus no logic expecting CR, so strbuf_getline_lf() can be replaced by its CRLF counterpart. Signed-off-by: Moritz Neeb --- remote.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/remote.c b/remote.c index 7d61dab..78f97f0 100644 --- a/remote.c +++ b/remote.c @@ -281,7 +281,7 @@ static void read_branches_file(struct remote *remote) if (!f) return; - strbuf_getline_lf(&buf, f); + strbuf_getline(&buf, f); fclose(f); strbuf_trim(&buf); if (!buf.len) { -- 2.4.3 ``` ## Moritz Neeb, 2016-01-30 18:05 Subject: [PATCH 5/5] wt-status: read rebase todolist with strbuf_getline() Message-ID: <56ACFB7B.6050800@moritzneeb.de> URL: https://gitlist.dev/e/56ACFB7B.6050800%40moritzneeb.de In-Reply-To: <56ACF82B.2030005@moritzneeb.de> ``` In read_rebase_todolist() every line is, after reading, checked for a comment_line_char. After that it is trimmed via strbuf_trim(). Assuming we do never expect a CR as comment_line_char, strbuf_getline_lf() can be safely replaced by its CRLF counterpart. Signed-off-by: Moritz Neeb --- Notes: Looking at the code in read_rebase_todolist(), the lines are read, checked for a comment_line_char and then trimmed. I wonder why the input is not trimmed before checking for this character? Is it safe to replace strbuf_getline_lf() by strbuf_getline() anyway? The only case I can imagine that could lead to unexpected behaviour then would be when someone sets the comment_line_char to CR. How likely is that? Why is the trim _after_ checking for the comment char anyway? Should something like " # foobar" not be considered as comment? I decided to roll out the patch because I considered it not as a risk to be taken seriously, that the comment line char will be '\r'. Meta-question: Should something of this discussion be put into the commit? wt-status.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/wt-status.c b/wt-status.c index ab4f80d..f053b2f 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1076,7 +1076,7 @@ static void read_rebase_todolist(const char *fname, struct string_list *lines) if (!f) die_errno("Could not open file %s for reading", git_path("%s", fname)); - while (!strbuf_getline_lf(&line, f)) { + while (!strbuf_getline(&line, f)) { if (line.len && line.buf[0] == comment_line_char) continue; strbuf_trim(&line); -- 2.4.3 ``` ## Junio C Hamano, 2016-02-01 21:30 Subject: Re: [PATCH 1/5] bisect: read bisect paths with strbuf_getline() Message-ID: URL: https://gitlist.dev/e/xmqqlh74unja.fsf%40gitster.mtv.corp.google.com In-Reply-To: <56ACFB03.3020000@moritzneeb.de> ``` Moritz Neeb writes: > The lines read from BISECT_NAMES are trimmed with strbuf_trim() > immediately. There is thus no logic expecting CR, so > strbuf_getline_lf() can be replaced by its CRLF counterpart. We do not indent the whole log message. You would also want to think about the necessity of strbuf_trim() here. Now strbuf_getline() would trim the trailing CR, would we still need to call strbuf_trim() here? The code will break if you just remove the call, but on the other hand, you will realize that the trimming done by calling it is excessive and unnecessary, once you inspect the code and learn who writes the file being read here and how. > Signed-off-by: Moritz Neeb > --- > bisect.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/bisect.c b/bisect.c > index 06ec54e..bf7c885 100644 > --- a/bisect.c > +++ b/bisect.c > @@ -440,7 +440,7 @@ static void read_bisect_paths(struct argv_array *array) > if (!fp) > die_errno("Could not open file '%s'", filename); > - while (strbuf_getline_lf(&str, fp) != EOF) { > + while (strbuf_getline(&str, fp) != EOF) { > strbuf_trim(&str); > if (sq_dequote_to_argv_array(str.buf, array)) > die("Badly quoted content in file '%s': %s", ``` ## Junio C Hamano, 2016-02-01 21:30 Subject: Re: [PATCH 2/5] clean: read user input with strbuf_getline() Message-ID: URL: https://gitlist.dev/e/xmqqh9hsunhw.fsf%40gitster.mtv.corp.google.com In-Reply-To: <56ACFB3D.6010309@moritzneeb.de> ``` The same comment (including "think if this trim is still necessary") as 1/5 applies here. ``` ## Junio C Hamano, 2016-02-01 21:34 Subject: Re: [PATCH 3/5] notes: read copied notes with strbuf_getline() Message-ID: URL: https://gitlist.dev/e/xmqqd1sgunci.fsf%40gitster.mtv.corp.google.com In-Reply-To: <56ACFB53.7070400@moritzneeb.de> ``` Same comment as 1/5 and 2/5 applies. You need to think if strbuf_rtim(split[1]) is necessary here. ``` ## Junio C Hamano, 2016-02-01 21:39 Subject: Re: [PATCH 5/5] wt-status: read rebase todolist with strbuf_getline() Message-ID: URL: https://gitlist.dev/e/xmqq8u34un3d.fsf%40gitster.mtv.corp.google.com In-Reply-To: <56ACFB7B.6050800@moritzneeb.de> ``` Moritz Neeb writes: > In read_rebase_todolist() every line is, after reading, checked > for a comment_line_char. After that it is trimmed via strbuf_trim(). > Assuming we do never expect a CR as comment_line_char, > strbuf_getline_lf() can be safely replaced by its CRLF counterpart. > > Signed-off-by: Moritz Neeb > --- > > Notes: > Looking at the code in read_rebase_todolist(), the > lines are read, checked for a comment_line_char and then trimmed. I > wonder why the input is not trimmed before checking for this character? > Is it safe to replace strbuf_getline_lf() by strbuf_getline() anyway? > The only case I can imagine that could lead to unexpected behaviour then > would be when someone sets the comment_line_char to CR. How likely is that? > Why is the trim _after_ checking for the comment char anyway? Should > something like " # foobar" not be considered as comment? > I decided to roll out the patch because I considered it not as a risk to be > taken seriously, that the comment line char will be '\r'. > Meta-question: Should something of this discussion be put into the commit? Yes to the meta question. Add something like this as the second paragraph of the log message, but do *not* change the patch text. The current code checks if the line begins with a comment line char (typically '#') before trimming, which is consistent with the way how commands like 'git commit' prepares commented lines in that this does not treat a line that begins with whitespaces and '#' as commented out. We however made a mistake in the parser of "git rebase -i" long time ago to allow such a line to be treated as a comment, and made an exception with 1db168ee (rebase-i: loosen over-eager check_bad_cmd check, 2015-10-01). It probably is a good idea to match that exception by swapping the order of comment check and trimming in this codepath as well. > > wt-status.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/wt-status.c b/wt-status.c > index ab4f80d..f053b2f 100644 > --- a/wt-status.c > +++ b/wt-status.c > @@ -1076,7 +1076,7 @@ static void read_rebase_todolist(const char *fname, struct string_list *lines) > if (!f) > die_errno("Could not open file %s for reading", > git_path("%s", fname)); > - while (!strbuf_getline_lf(&line, f)) { > + while (!strbuf_getline(&line, f)) { > if (line.len && line.buf[0] == comment_line_char) > continue; > strbuf_trim(&line); ``` ## Moritz Neeb, 2016-02-14 21:01 Subject: Re: [PATCH 1/5] bisect: read bisect paths with strbuf_getline() Message-ID: <56C0EB3F.90202@moritzneeb.de> URL: https://gitlist.dev/e/56C0EB3F.90202%40moritzneeb.de In-Reply-To: ``` On 02/01/2016 10:30 PM, Junio C Hamano wrote: > Moritz Neeb writes: > >> The lines read from BISECT_NAMES are trimmed with strbuf_trim() >> immediately. There is thus no logic expecting CR, so >> strbuf_getline_lf() can be replaced by its CRLF counterpart. > > We do not indent the whole log message. Thanks for spotting, I will change that in the next iteration. > You would also want to think about the necessity of strbuf_trim() > here. Now strbuf_getline() would trim the trailing CR, would we > still need to call strbuf_trim() here? The code will break if you > just remove the call, but on the other hand, you will realize that > the trimming done by calling it is excessive and unnecessary, once > you inspect the code and learn who writes the file being read here > and how. I am not sure what you mean by excessive: How much can I assume that the input is like expected? The files we are talking about are supposed to be read and written by git only. But could be modified in theory with an editor, right? Then things could break, right? This question maybe holds true for the other patches as well, I still have to look into them. Assuming that things are just like expected, what has to be done in read_bisect_paths before dequoting is to remove the leading space, which is added by 'git rev-parse --sq-quote "$@"' in git-bisect.sh. I am just not sure, where to remove it: * in sq_quote_argv? - see some analysis below. * in read_bisect_paths via strbuf_ltrim instead of strbuf_trim, is that what you meant? * in sq_dequote_argv_array? - see below as well. I tried to understand, why the space is added by sq_quote_argv() in quotes.c in the first place. It is there since the beginning in 7cf6720. I have the feeling that the space is redundant for bisect, because the only other codepath where BISECT_NAMES is read is in bisect_visualize(), where the necessary space to separate the arguments is added before '$(...)': eval '"$@"' --bisect -- $(cat "$GIT_DIR/BISECT_NAMES") Other places where sq_quote_argv() and sq_dequote_to_argv_array() are called as well are builtin/am.c and trace.c. In builtin/am.c when reading 'apply-opt', the data is trimmed in read_state_file, so the space in the beginning is superfluous. Looking at trace.c it gets clear that this code is dependent on this space, otherwise the trace output will look like "trace: built-in: git'status'" which is obviously bogus. It would maybe make more sense to add this space directly there than to trim it in all the other places. I realized that this code came up in 7cf6720 as well which explains why things are like they are (and why I'm CC'ing Christian Couder). Another option, at the moment my favourite because I think it's the least invasive, would be to "trim" this one space directly in sq_dequote_step(). See the patch below. This would not affect the tracing functionality, because there is no dequoting used. -- >8 -- Subject: [PATCH] quote: remove leading space in sq_dequote_step Because sq_quote_argv adds a leading space (which is expected in trace.c), sq_dequote_step should remove this space again, such that the operations of quoting and dequoting are inverse of each other. Signed-off-by: Moritz Neeb --- quote.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/quote.c b/quote.c index fe884d2..2714f27 100644 --- a/quote.c +++ b/quote.c @@ -63,6 +63,8 @@ static char *sq_dequote_step(char *arg, char **next) char *src = arg; char c; + if (*src == ' ') + src++; if (*src != '\'') return NULL; for (;;) { -- 2.4.3 ``` ## Junio C Hamano, 2016-02-15 05:05 Subject: Re: [PATCH 1/5] bisect: read bisect paths with strbuf_getline() Message-ID: URL: https://gitlist.dev/e/xmqqtwlar2bd.fsf%40gitster.mtv.corp.google.com In-Reply-To: <56C0EB3F.90202@moritzneeb.de> ``` Moritz Neeb writes: >> You would also want to think about the necessity of strbuf_trim() >> here. Now strbuf_getline() would trim the trailing CR, would we >> still need to call strbuf_trim() here? The code will break if you >> just remove the call, but on the other hand, you will realize that >> the trimming done by calling it is excessive and unnecessary, once >> you inspect the code and learn who writes the file being read here >> and how. > > I am not sure what you mean by excessive: How much can I assume that > the input is like expected? The files we are talking about are supposed > to be read and written by git only. But could be modified in theory with > an editor, right? Then things could break, right? This question maybe holds > true for the other patches as well, I still have to look into them. These are all good questions you as a Git contributor to be asking yourself, and I really like the fact that you are thinking aloud here. As to bisect pathspec, while I do not think it is sensible to change it in the middle of bisection, I do not think it would cause the bisect command to produce an incorrect result if you did so. You would be changing the definition of what is the "middle point" of the history on the fly by changing the pathspec, hence you may end up making the bisection suboptimal, but it shouldn't affect the correctness of the bisection. So I tend to agree with you that it is a good idea to be prepared to parse what the end user may have futzed with the editor, as long as the user did not break the syntax of the quoted string. Your answer to the question quoted at the top of this message could be a summary of the above reasoning, with a conclusion: "we would want to tolerate trailing (or leading) whitespaces the user might add with his editor; even though we use strbuf_getline() to remove the CR at the end of line, we still need to strbuf_trim() the line we read". Side note: of course, you can instead say "while it is technically possible, we have never advertised this file to be editable by user, or encouraged them to do so" and tighten the parsing to assume only what we write out, erroring out when we see any attempt to edit it. Either way, the most important thing is to record the reasoning behind the design choice you made clearly in the proposed commit log message. Thanks. ``` ## Moritz Neeb, 2016-02-21 23:48 Subject: Re: [PATCH 1/5] bisect: read bisect paths with strbuf_getline() Message-ID: <56CA4CE6.9070802@moritzneeb.de> URL: https://gitlist.dev/e/56CA4CE6.9070802%40moritzneeb.de In-Reply-To: ``` On 02/15/2016 06:05 AM, Junio C Hamano wrote: > Moritz Neeb writes: > >>> You would also want to think about the necessity of strbuf_trim() >>> here. Now strbuf_getline() would trim the trailing CR, would we >>> still need to call strbuf_trim() here? The code will break if you >>> just remove the call, but on the other hand, you will realize that >>> the trimming done by calling it is excessive and unnecessary, once >>> you inspect the code and learn who writes the file being read here >>> and how. >> >> I am not sure what you mean by excessive: How much can I assume that >> the input is like expected? The files we are talking about are supposed >> to be read and written by git only. But could be modified in theory with >> an editor, right? Then things could break, right? This question maybe holds >> true for the other patches as well, I still have to look into them. > > These are all good questions you as a Git contributor to be asking > yourself, and I really like the fact that you are thinking aloud > here. Thanks for this feedback I will try to continue this thinking aloud whenever appropriate. Sorry by the way for only answering every week - I currently only have time to work on git during the weekend. What is (assuming familiarity with the codebase) the expected "cooking"-time of a smaller patch? I'm sure the answer is "depends", but maybe you can give away some experiences. ``` ## Moritz Neeb, 2016-02-22 00:07 Subject: Re: [PATCH 1/5] bisect: read bisect paths with strbuf_getline() Message-ID: <56CA5146.7050008@moritzneeb.de> URL: https://gitlist.dev/e/56CA5146.7050008%40moritzneeb.de In-Reply-To: ``` On 02/15/2016 06:05 AM, Junio C Hamano wrote: > As to bisect pathspec, while I do not think it is sensible to change > it in the middle of bisection, I do not think it would cause the > bisect command to produce an incorrect result if you did so. You > would be changing the definition of what is the "middle point" of > the history on the fly by changing the pathspec, hence you may end > up making the bisection suboptimal, but it shouldn't affect the > correctness of the bisection. Yep I also think it would not affect the correctness. To reformulate your explanation: changing the pathspec does not change the good and bad "frontiers". And those should be still correct under the assumption you're still looking for the same bug/regression. As to the question if it is sensible, I also don't really think it could be helpful. The only way I would expect it to be useful, it so "refine" the search by adding specific files to the filepath. But then it's possible to end up in a dead-end if the commits left are not touching the specified files with: No testable commit found. Maybe you started with bad path parameters? I have also seen other messed up messages, like "-1 commits left" while playing around. > So I tend to agree with you that it is a good idea to be prepared to > parse what the end user may have futzed with the editor, as long as > the user did not break the syntax of the quoted string. Now I am not convinced anymore, because I think if it potentially leads to troubles, it should not be made easier to get oneself into them. > Side note: of course, you can instead say "while it is > technically possible, we have never advertised this file to > be editable by user, or encouraged them to do so" and > tighten the parsing to assume only what we write out, > erroring out when we see any attempt to edit it. I will include something like that in the patch v2 that is to follow soon. ```