threads / discuss / 62287

Verbose Commit Ignore Line Fails via CRLF Line Endings

Subject: Verbose Commit Ignore Line Fails via CRLF Line Endings

## tl;dr

9 messages between Oct 8, 2024 and Oct 11, 2024.

replies: 8people: 4as markdown or json

Spencer Fretwell· Oct 8, 2024, 18:30 UTC · lore
gc -v produces CRLF file which does not ignore verbose description.
OSX + git-annex (pre-commit hook disabled just in case)
using subl -w (sublime text 4) as editor, which indicated the change
in line endings
- committing as Unix fixes the issue (workaround, tedious)
- leaving as Windows is the issue
see attached bugreport

Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)
- gc -v
What did you expect to happen? (Expected behavior)
- line endings respect OS
- gc without -v uses Unix line endings
- line ending is visible in sublime (editor = subl -w)
What happened instead? (Actual behavior)
- gc -v uses CRLF (Windows) line endings
- as seen in sublime (editor = subl -w)
What's different between what you expected and what actually happened?
- use of CRLF is leaving verbose output in commit message
Anything else you want to add:

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

[System Info] git version: git version 2.46.2 cpu: arm64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh feature: fsmonitor--daemon libcurl: 8.4.0 zlib: 1.2.12 uname: Darwin 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:13:04 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6020 arm64 compiler info: clang: 15.0.0 (clang-1500.3.9.4) libc info: no libc information available $SHELL (typically, interactive shell): /bin/zsh

[Enabled Hooks] post-checkout post-merge post-receive

Spencer Fretwell· Oct 8, 2024, 18:37 UTC · re: Spencer Fretwell · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

On second glance it might involve having a file being committed that has CRLF endings in it. Committing files with LF only does not produce issue.

I suspect git either needs to add support for a magic ignore line ending with a CRLF, or has to sanitize verbose output when that output has CRLF (less ideal)

Bug originally seen as early as [2018](https://superuser.com/questions/1367811/sometimes-git-includes-the-diff-to-commit-message-when-using-verbose)

On Tue, Oct 8, 2024 at 12:30 PM Spencer Fretwell <spencer.fretwell@gmail.com> wrote:

Show 10 quoted lines
>
> gc -v produces CRLF file which does not ignore verbose description.
>
> OSX + git-annex (pre-commit hook disabled just in case)
> using subl -w (sublime text 4) as editor, which indicated the change
> in line endings
> - committing as Unix fixes the issue (workaround, tedious)
> - leaving as Windows is the issue
>
> see attached bugreport
brian m. carlson· Oct 8, 2024, 19:09 UTC · re: Spencer Fretwell · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

On 2024-10-08 at 18:37:47, Spencer Fretwell wrote:
Show 10 quoted lines
> On second glance it might involve having a file being committed that
> has CRLF endings in it.
> Committing files with LF only does not produce issue.
> 
> I suspect git either needs to add support for a magic ignore line
> ending with a CRLF,
> or has to sanitize verbose output when that output has CRLF (less ideal)
> 
> Bug originally seen as early as
> [2018](https://superuser.com/questions/1367811/sometimes-git-includes-the-diff-to-commit-message-when-using-verbose)

Ah, yes, that would probably explain it. Your editor is seeing some lines that are CRLF and assuming that the entire file is CRLF, which it should not. In Vim and Neovim, you can do `setl ff=unix` when loading a buffer for `git commit` that will force it to always use Unix line endings, and there's probably some functionality in Sublime Text that lets you force a particular line ending.

What might also help is that if these are intended to be text files, you can set a `.gitattributes` file as outlined in the Git FAQ[0]:

  # By default, guess.
  *	text=auto
  # Mark all C files as text.
  *.c	text
  # Ensure all shell files have LF endings and all batch files have CRLF
  # endings in the working tree and both have LF in the repo.
  *.sh text eol=lf
  *.bat text eol=crlf
  # Mark all JPEG files as binary.
  *.jpg	binary

You can customize the file patterns as you see fit. Once you do that and run `git add --renormalize .` (which you should do in the same commit as adding your `.gitattributes` file) and then commit, your text files will be stored in the repository as LF even though they may have CRLF endings in the working tree, and your diff output won't contain CRLF.

[0] https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt#recommended-storage-settings
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
Spencer Fretwell· Oct 8, 2024, 19:34 UTC · re: brian m. carlson · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

Thanks Brian,

It appears sublime auto-normalizes endings to "whatever occurs most frequently in the first 32kB". So, I guess it was witnessing the CRLF from the verbose output and replacing all lines with CRLF. Thanks for the reminder about --renormalize.

Is there any chance for git to support a CRLF magic ignore line, particularly considering the variation in standard line ending across different platforms? I tried autocrlf=input as well and it sadly doesn't normalize the commit message file itself. Either way (magic ignore with CRLF or normalizing line endings in the commit message), would be appreciated for mixed line ending workflows (especially considering WSL)

On Tue, Oct 8, 2024 at 1:09 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 45 quoted lines
>
> On 2024-10-08 at 18:37:47, Spencer Fretwell wrote:
> > On second glance it might involve having a file being committed that
> > has CRLF endings in it.
> > Committing files with LF only does not produce issue.
> >
> > I suspect git either needs to add support for a magic ignore line
> > ending with a CRLF,
> > or has to sanitize verbose output when that output has CRLF (less ideal)
> >
> > Bug originally seen as early as
> > [2018](https://superuser.com/questions/1367811/sometimes-git-includes-the-diff-to-commit-message-when-using-verbose)
>
> Ah, yes, that would probably explain it.  Your editor is seeing some
> lines that are CRLF and assuming that the entire file is CRLF, which it
> should not.  In Vim and Neovim, you can do `setl ff=unix` when loading a
> buffer for `git commit` that will force it to always use Unix line
> endings, and there's probably some functionality in Sublime Text that
> lets you force a particular line ending.
>
> What might also help is that if these are intended to be text files, you
> can set a `.gitattributes` file as outlined in the Git FAQ[0]:
>
>   # By default, guess.
>   *     text=auto
>   # Mark all C files as text.
>   *.c   text
>   # Ensure all shell files have LF endings and all batch files have CRLF
>   # endings in the working tree and both have LF in the repo.
>   *.sh text eol=lf
>   *.bat text eol=crlf
>   # Mark all JPEG files as binary.
>   *.jpg binary
>
> You can customize the file patterns as you see fit.  Once you do that
> and run `git add --renormalize .` (which you should do in the same
> commit as adding your `.gitattributes` file) and then commit, your text
> files will be stored in the repository as LF even though they may have
> CRLF endings in the working tree, and your diff output won't contain
> CRLF.
>
> [0] https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt#recommended-storage-settings
> --
> brian m. carlson (they/them or he/him)
> Toronto, Ontario, CA
brian m. carlson· Oct 8, 2024, 19:52 UTC · re: Spencer Fretwell · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

On 2024-10-08 at 19:34:46, Spencer Fretwell wrote:
Show 14 quoted lines
> Thanks Brian,
> 
> It appears sublime auto-normalizes endings to "whatever occurs most
> frequently in the first 32kB". So, I guess it was witnessing the CRLF
> from the verbose output and replacing all lines with CRLF. Thanks for
> the reminder about --renormalize.
> 
> Is there any chance for git to support a CRLF magic ignore line,
> particularly considering the variation in standard line ending across
> different platforms? I tried autocrlf=input as well and it sadly
> doesn't normalize the commit message file itself. Either way (magic
> ignore with CRLF or normalizing line endings in the commit message),
> would be appreciated for mixed line ending workflows (especially
> considering WSL)

The answer is essentially that I don't know. We typically make decisions on whether we'll accept features when we see the patch. My guess is that, assuming someone (maybe you) sends a patch, it will probably be accepted, since I wouldn't expect it would be very difficult to do or have major impacts on the code. It might, as with any patch, take a couple of rounds, though.

I use Linux or rarely other Unix systems and always use LF endings, so I don't plan to send a patch since this doesn't affect me, but assuming the patch looked reasonable, I don't see myself having an objection to it.

-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
Torsten Bögershausen· Oct 11, 2024, 04:45 UTC · re: brian m. carlson · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

On Tue, Oct 08, 2024 at 07:52:33PM +0000, brian m. carlson wrote:
Show 27 quoted lines
> On 2024-10-08 at 19:34:46, Spencer Fretwell wrote:
> > Thanks Brian,
> >
> > It appears sublime auto-normalizes endings to "whatever occurs most
> > frequently in the first 32kB". So, I guess it was witnessing the CRLF
> > from the verbose output and replacing all lines with CRLF. Thanks for
> > the reminder about --renormalize.
> >
> > Is there any chance for git to support a CRLF magic ignore line,
> > particularly considering the variation in standard line ending across
> > different platforms? I tried autocrlf=input as well and it sadly
> > doesn't normalize the commit message file itself. Either way (magic
> > ignore with CRLF or normalizing line endings in the commit message),
> > would be appreciated for mixed line ending workflows (especially
> > considering WSL)
>
> The answer is essentially that I don't know.  We typically make
> decisions on whether we'll accept features when we see the patch.  My
> guess is that, assuming someone (maybe you) sends a patch, it will
> probably be accepted, since I wouldn't expect it would be very difficult
> to do or have major impacts on the code.  It might, as with any patch,
> take a couple of rounds, though.
>
> I use Linux or rarely other Unix systems and always use LF endings, so I
> don't plan to send a patch since this doesn't affect me, but assuming
> the patch looked reasonable, I don't see myself having an objection to
> it.
Hej Spencer,
you are probably the first one reporting this, thanks for doing so.

I have the suspicion, that your repo has files commit with CRLF, and that leads to a CRLF entering the diff, and that leads your editor to produce CRLF in the commit-message.

In order to debug this a little bit, can you run a git ls-files --eol | grep i/crlf in your repo ?

In general, when working with cross-platform, or only with windows, it is a good practice to use a .gitattributes file to specify the line endings. Do you have such a file in your repo, and is it commited ?

Junio C Hamano· Oct 11, 2024, 17:59 UTC · re: brian m. carlson · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
Show 19 quoted lines
>> Is there any chance for git to support a CRLF magic ignore line,
>> particularly considering the variation in standard line ending across
>> different platforms? I tried autocrlf=input as well and it sadly
>> doesn't normalize the commit message file itself. Either way (magic
>> ignore with CRLF or normalizing line endings in the commit message),
>> would be appreciated for mixed line ending workflows (especially
>> considering WSL)
>
> The answer is essentially that I don't know.  We typically make
> decisions on whether we'll accept features when we see the patch.  My
> guess is that, assuming someone (maybe you) sends a patch, it will
> probably be accepted, since I wouldn't expect it would be very difficult
> to do or have major impacts on the code.  It might, as with any patch,
> take a couple of rounds, though.
>
> I use Linux or rarely other Unix systems and always use LF endings, so I
> don't plan to send a patch since this doesn't affect me, but assuming
> the patch looked reasonable, I don't see myself having an objection to
> it.

I do not offhand see a downside if we loosened the detection of the cut line a bit. What is happening on the Git end (I do not care how Editor screws up and munges what we wrote before "Do not modify or remove the line above"---to us, it looks like the user did not honor that instruction) is:

wt-status.c defines cut_line[] to be
        static const char cut_line[] =
        "------------------------ >8 ------------------------\n";
IOW, we insist that after the dashes we have LF.
Then wt-status.c:wt_status_locate_end() does this:
        size_t wt_status_locate_end(const char *s, size_t len)
        {
                const char *p;
                struct strbuf pattern = STRBUF_INIT;
                strbuf_addf(&pattern, "\n%s %s", comment_line_str, cut_line);
                if (starts_with(s, pattern.buf + 1))
                        len = 0;
                else if ((p = strstr(s, pattern.buf))) {
			size_t newlen = p - s + 1;
			...

That is, if the buffer begins with the cut line (including the LF at the end of the line, without any CR before it), we found it at the beginning of the buffer at len==0, otherwise if we find a region of memory that begins with LF, followed by the scissors, followed by LF (again, without allowing CR before it), we have a match there.

That is reasonable as long as the user does not muck with the line that we told them not to touch. In this case, the editor is corrupting the line without being instructed by the user so the user cannot do much about it.

We could loosen the rule by doing something along the following line:
 - define the cut_line constant without LF at the end of the line.
 - teach wt_status_append_cut_line() to compensate for the lack of
   LF in cut_line.
 - teach wt_status_locate_end() to notice and ignore CR if it
   appears at the end of the cut_line.

I'll go offline at the end of this week, so I will stop here at illustrating the above approach in the form of an untested patch below. I am sure I'd have many off by one errors in it, but it should be sufficient to outline the idea.

 wt-status.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)
diff --git c/wt-status.c w/wt-status.c
index 6a8c05d1cf..b54bc5de81 100644
--- c/wt-status.c
+++ w/wt-status.c
@@ -39,7 +39,7 @@
 #define UF_DELAY_WARNING_IN_MS (2 * 1000)
 
 static const char cut_line[] =
-"------------------------ >8 ------------------------\n";
+"------------------------ >8 ------------------------";
 
 static char default_wt_status_colors[][COLOR_MAXLEN] = {
 	GIT_COLOR_NORMAL, /* WT_STATUS_HEADER */
@@ -1095,15 +1095,28 @@ static void wt_longstatus_print_other(struct wt_status *s,
 	status_printf_ln(s, GIT_COLOR_NORMAL, "%s", "");
 }
 
+static int at_eol(const char *s, size_t len, size_t loc)
+{
+	if (len <= loc)
+		return 0;
+	if (s[loc] == '\r')
+		loc++;
+	if (len <= loc)
+		return 0;
+	return s[loc] == '\n';
+}
+
 size_t wt_status_locate_end(const char *s, size_t len)
 {
 	const char *p;
 	struct strbuf pattern = STRBUF_INIT;
 
-	strbuf_addf(&pattern, "\n%s %s", comment_line_str, cut_line);
-	if (starts_with(s, pattern.buf + 1))
+	strbuf_addf(&pattern, "%s %s", comment_line_str, cut_line);
+	if (starts_with(s, pattern.buf) && at_eol(s, len, pattern.len))
 		len = 0;
-	else if ((p = strstr(s, pattern.buf))) {
+	else if ((p = strstr(s, pattern.buf)) &&
+		 (s < p && p[-1] == '\n') &&
+		 at_eol(p, p - s, pattern.len)) {
 		size_t newlen = p - s + 1;
 		if (newlen < len)
 			len = newlen;
@@ -1116,7 +1129,7 @@ void wt_status_append_cut_line(struct strbuf *buf)
 {
 	const char *explanation = _("Do not modify or remove the line above.\nEverything below it will be ignored.");
 
-	strbuf_commented_addf(buf, comment_line_str, "%s", cut_line);
+	strbuf_commented_addf(buf, comment_line_str, "%s\n", cut_line);
 	strbuf_add_commented_lines(buf, explanation, strlen(explanation), comment_line_str);
 }
 
brian m. carlson· Oct 8, 2024, 19:02 UTC · re: Spencer Fretwell · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

On 2024-10-08 at 18:30:53, Spencer Fretwell wrote:
> gc -v produces CRLF file which does not ignore verbose description.

I'm not sure what software you're suggesting here, but we don't distribute any software called "gc". I can't imagine `git gc` is affecting anything here. Is this maybe an alias for `git commit` or something else? (I'm going to assume so below.) If so, do you see the problem with a plain `git commit -v`?

Show 7 quoted lines
> OSX + git-annex (pre-commit hook disabled just in case)
> using subl -w (sublime text 4) as editor, which indicated the change
> in line endings
> - committing as Unix fixes the issue (workaround, tedious)
> - leaving as Windows is the issue
> 
> see attached bugreport
Show 12 quoted lines
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
> 
> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> - gc -v
> 
> What did you expect to happen? (Expected behavior)
> 
> - line endings respect OS
> - gc without -v uses Unix line endings
> - line ending is visible in sublime (editor = subl -w)

Does this happen if you use a different editor as well, such as vi or nano? I don't see this on a Linux system with `git commit -v` using Neovim.

Also, if you do see this problem when you open the editor, can you leave the editor as it is (don't save or modify the file), and change into another terminal window and use something like `xxd -g` or `od -tx1` to see if `.git/COMMIT_EDITMSG` has CRLF pairs (that should show up as `0d 0a`) or just LF (plain `0a`)?

The reason I ask this is that I'm wondering if maybe there's a configuration of your editor that's causing it to switch into a special diff mode with `git commit -v` that it thinks should have CRLF endings.

-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
Torsten Bögershausen· Oct 11, 2024, 16:10 UTC · lore

Re: Verbose Commit Ignore Line Fails via CRLF Line Endings

On Fri, Oct 11, 2024 at 02:33:46PM +0000, Spencer Fretwell wrote:
(Please try to avoid top-posting here on this list)
Show 54 quoted lines
> On Fri Oct 11, 2024, 04:45 AM GMT, Torsten Bögershausen <mailto:tboegi@web.de> wrote:
> > On Tue, Oct 08, 2024 at 07:52:33PM +0000, brian m. carlson wrote:
> >> On 2024-10-08 at 19:34:46, Spencer Fretwell wrote:
> >> > Thanks Brian,
> >> >
> >> > It appears sublime auto-normalizes endings to "whatever occurs most
> >> > frequently in the first 32kB". So, I guess it was witnessing the CRLF
> >> > from the verbose output and replacing all lines with CRLF. Thanks for
> >> > the reminder about --renormalize.
> >> >
> >> > Is there any chance for git to support a CRLF magic ignore line,
> >> > particularly considering the variation in standard line ending across
> >> > different platforms? I tried autocrlf=input as well and it sadly
> >> > doesn't normalize the commit message file itself. Either way (magic
> >> > ignore with CRLF or normalizing line endings in the commit message),
> >> > would be appreciated for mixed line ending workflows (especially
> >> > considering WSL)
> >>
> >> The answer is essentially that I don't know. We typically make
> >> decisions on whether we'll accept features when we see the patch. My
> >> guess is that, assuming someone (maybe you) sends a patch, it will
> >> probably be accepted, since I wouldn't expect it would be very difficult
> >> to do or have major impacts on the code. It might, as with any patch,
> >> take a couple of rounds, though.
> >>
> >> I use Linux or rarely other Unix systems and always use LF endings, so I
> >> don't plan to send a patch since this doesn't affect me, but assuming
> >> the patch looked reasonable, I don't see myself having an objection to
> >> it.
> >
> > Hej Spencer,
> >
> > you are probably the first one reporting this, thanks for doing so.
> >
> > I have the suspicion, that your repo has files commit with CRLF,
> > and that leads to a CRLF entering the diff, and that leads your
> > editor to produce CRLF in the commit-message.
> >
> > In order to debug this a little bit, can you run a
> > git ls-files --eol | grep i/crlf
> > in your repo ?
> >
> > In general, when working with cross-platform, or only with windows,
> > it is a good practice to use a .gitattributes file to specify the
> > line endings.
> > Do you have such a file in your repo, and is it commited ?
> >
> >
> >
> Hi Torsten,
>
> To catch you up, yes the problem involved commiting a long CRLF file whose line endings predominated the commit message, so my editor assumed the endings should be CRLF for the message and normalized all lines including the magic ‘comment below’ line. This in turn appears to nullify the magic line so the patch itself saves to the commit message
>
> Would an attribute file LF the patch contents? I guess it should given the patch is what is comitted. The other solution would be for git to handle the CRLF magic line better, either sanitizing to LF during comment removal in the commit message or just by interpreting the magic line with CRLF as still a magic line. One is a process update the other a syntax update, but other than knowing that I am unfamiliar with the got codebase and this wary about attempting a patch…
I am not sure, if I understand your question fully correct.
As far as I know, Git does not manipulate the line endings
in the commit message. That is something that can be done.
However, it seems that 2 things come together at you side:
a) Files are commited with CRLF.
  That is not recommendet, I would say.
  Could you do me a favorite and run
  git ls-files --eol | grep i/crlf
b) Your editor makes the decision to convert all LF into CRLF
  That is, hm, fixable by either using a different editor
  or setting
  git config --global core.editor notepad (under windows).
  The "notepad" executable under Windows is a shell script,
  which converts the commit message into CRLF, invokes notepad.exe,
  and converts the commit message back into LF, before handing it
  to Git.
  That will do the trick as well.

If someone wants to send a patch to normalize the line endings from the commit-message, I am happy to review it.

← back to recent threads