patchcredential.c: fix credential reading with regards to CR/LF

From: Nikita Leonov <nykyta.leonov@gmail.com>

This fix makes using Git credentials more friendly to Windows users. In previous version it was unable to finish input correctly without configuration changes (tested in PowerShell, CMD, Cygwin).

We know credential filling should be finished by empty input, but the current implementation does not take into account CR/LF ending, and hence instead of the empty string we get '\r', which is interpreted as an incorrect string.

So this commit changes default reading function to a more Windows compatible reading function.

Signed-off-by: Nikita Leonov <nykyta.leonov@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
    Prepare git credential to read input with DOS line endings
    
    This just came in via Git for Windows
    [https://github.com/git-for-windows/git/pull/2516].

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-710%2Fdscho%2Fcrlf-aware-git-credential-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-710/dscho/crlf-aware-git-credential-v1
Pull-Request: https://github.com/git/git/pull/710

 credential.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> writes:

In the older days, strbuf_getline() used to be what we have as strbuf_getdelim() today, i.e. explicitly request what record separator to use to grab a line.

The use of strbuf_getline_lf() we see in the pre-image of this patch came from 8f309aeb ("strbuf: introduce strbuf_getline_{lf,nul}()", 2016-01-13), which mechanically replaced the use of strbuf_getline(... '\n'), in anticipation for later effort to identify ones that are better to accept CRLF and turn them into _crlf variant. Later all the callers of (old) strbuf_getline() went away, and strbuf_getline_crlf() that allowed both LF and CRLF termination (which is most friendly to human-readable line of text) took over the shortest and sweetest name, strbuf_getline().

So, the "later" effort just happend to this code. It was a bit overdue, but it's better late than never.

There are many more conversions of strbuf_getline(..., '\n') to strbuf_getline_lf(...) made by 8f309aeb to other parts of credential stuff. Has anybody from the Windows side made sure these other ones are also better to accept CRLF, too?

I'd wait for a bit to hear either "oh, we found two more and here is an updated patch" or "we looked at them and this is the only one", before touching this patch.

Thanks.

On Fri, Feb 14, 2020 at 01:49:56PM +0000, Johannes Schindelin via GitGitGadget wrote:

This does make it impossible to have a CR at the end of a data value. I think that should be OK (we already disallow LF with no mechanism for quoting, because who the hell puts a LF in their password?).

But we should perhaps update the section in git-credential(1) that describes the rules. I had trouble coming up with a wording that wasn't totally awkward, though:

This contribution came in via Git for Windows [https://github.com/git-for-windows/git/pull/2516].

Sadly, I did not find the time to go through all the changes of 8f309aeb ("strbuf: introduce strbuf_getline_{lf,nul}()", 2016-01-13) (as Junio asked [https://public-inbox.org/git/xmqqmu9lnjdh.fsf@gitster-ct.c.googlers.com]). Rather than delaying this patch indefinitely, I admit defeat on that angle.

Changes since v1:

 * Added a commit to adjust credential-daemon and credential-store in the
   same manner.
 * Adjusted the documentation accordingly.

Nikita Leonov (3):
  credential.c: fix credential reading with regards to CR/LF
  credentials: make line reading Windows compatible
  docs: make notes regarding credential line reading

 Documentation/git-credential.txt   |  4 +++-
 Documentation/gitcredentials.txt   |  2 ++
 builtin/credential-cache--daemon.c |  4 ++--
 builtin/credential-store.c         |  2 +-
 credential.c                       |  2 +-
 t/t0302-credential-store.sh        | 16 ++++++----------
 6 files changed, 15 insertions(+), 15 deletions(-)

base-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-710%2Fdscho%2Fcrlf-aware-git-credential-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-710/dscho/crlf-aware-git-credential-v2
Pull-Request: https://github.com/git/git/pull/710

Range-diff vs v1:

 1:  2a9cc710bb = 1:  27f6400a21 credential.c: fix credential reading with regards to CR/LF
 -:  ---------- > 2:  f69076036f credentials: make line reading Windows compatible
 -:  ---------- > 3:  61baea1061 docs: make notes regarding credential line reading

From: Nikita Leonov <nykyta.leonov@gmail.com>

This fix makes using Git credentials more friendly to Windows users. In previous version it was unable to finish input correctly without configuration changes (tested in PowerShell, CMD, Cygwin).

We know credential filling should be finished by empty input, but the current implementation does not take into account CR/LF ending, and hence instead of the empty string we get '\r', which is interpreted as an incorrect string.

So this commit changes default reading function to a more Windows compatible reading function.

Signed-off-by: Nikita Leonov <nykyta.leonov@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 credential.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

On Mon, Sep 28, 2020 at 11:40:22AM +0000, Nikita Leonov via GitGitGadget wrote:

Unlike the credential-store file case, where we expect the data to be URL-encoded anyway (and so any true "\r" in the data would not be found in raw form), this means that the credential protocol can no longer represent "\r" at the end of a value.

And we'd match "example.com\r" and "example.com" as the same (unlikely, since carriage returns aren't allowed in hostnames, and curl will complain about this). We'd also match "cert://some/path\r" and "cert://some/path". Or "https://example.com/path\r" and its match, if you have credential.useHTTPPath set.

That may be acceptable if it makes things more convenient. Those are all pretty obscure cases, and I find it hard to believe an attacker could hijack credentials using this (it implies that the only difference between their malicious url and a known-good one is a trailing CR).

This part of the commit message confused me a little:

If all we care about is the empty line, and not data lines, then we could do this:

Hi Peff,

On Mon, 28 Sep 2020, Jeff King wrote:

Indeed.

True. It's a problem with all of those URLs that end in Carriage Returns ;-)

Indeed.

I checked again with github.com/git-for-windows/git/pull/2516, where the patch originally entered the public eye, but could not find any background information.

But I would highly doubt that the empty lines were the biggest problem: Sure, we would fail to recognize an empty line with CR/LF line endings when reading with `strbuf_getline_lf()`, but we would totally misunderstand the entire rest of the lines, too. For example, we would mistake `quit\r` for an unknown command, and hence simply ignore it.

I do agree, however, that your confusion validly points out a flaw in the commit message: the "empty line" comment is a red herring.

Therefore, I spent some time pouring over the commit message. This is my current version:

    credential: treat CR/LF as line endings in the credential protocol

    This fix makes using Git credentials more friendly to Windows users: it
    allows a credential helper to communicate using CR/LF line endings ("DOS
    line endings" commonly found on Windows) instead of LF-only line endings
    ("Unix line endings").

    Note that this changes the behavior a bit: if a credential helper
    produces, say, a password with a trailing Carriage Return character,
    that will now be culled even when the rest of the lines end only in Line
    Feed characters, indicating that the Carriage Return was not meant to be
    part of the line ending.

    In practice, it seems _very_ unlikely that something like this happens.
    Passwords usually need to consist of non-control characters, URLs need
    to have special characters URL-encoded, and user names, well, are names.

    So let's change the credential machinery to accept both CR/LF and LF
    line endings.

    While we do this for the credential helper protocol, we do _not_ do
    adjust `git credential-cache--daemon` (which won't work on Windows,
    anyway, because it requires Unix sockets) nor `git credential-store`
    (which writes the file `~/.git-credentials` which we consider an
    implementation detail that should be opaque to the user, read: we do
    expect users _not_ to edit this file manually).

What do you think?

Ciao, Dscho

On Fri, Oct 02, 2020 at 01:37:23PM +0200, Johannes Schindelin wrote:

I think we are on the same page, and this revision does a good job of fixing my complaint about the commit message. Thanks. One minor typo:

s/do$//

-Peff

Hi Peff,

On Fri, 2 Oct 2020, Jeff King wrote:

Thanks, fixed!

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

I am not Peff, but I was also drawn into the same confusion by the "we never see an empty line" red herring.

There are some micronits, but the above made a lot easier to understand (I think you could even add "quit\r" bit to make it even easier to understand) than the original description.

Thanks.

Hi Junio,

On Fri, 2 Oct 2020, Junio C Hamano wrote:

:-)

Okay, I incorporated a comment talking about `quit\r` and will submit a new iteration right now.

Thanks, Dscho

From: Nikita Leonov <nykyta.leonov@gmail.com>

This commit adds notes to git-credential.txt and to gitcredentials.txt specifying that 'LF' and 'CR/LF' endings are treated the same.

Signed-off-by: Nikita Leonov <nykyta.leonov@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Documentation/git-credential.txt | 4 +++-
 Documentation/gitcredentials.txt | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

From: Nikita Leonov <nykyta.leonov@gmail.com>

This commit makes reading process regarding credentials compatible with 'CR/LF' line ending. It makes using git more convenient on systems like Windows.

Signed-off-by: Nikita Leonov <nykyta.leonov@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/credential-cache--daemon.c |  4 ++--
 builtin/credential-store.c         |  2 +-
 t/t0302-credential-store.sh        | 16 ++++++----------
 3 files changed, 9 insertions(+), 13 deletions(-)

"Nikita Leonov via GitGitGadget" <gitgitgadget@gmail.com> writes:

I can see why this is a good thing for "store" and the two updated pieces of the test script demonstrate it very well.

But it is unclear why and how cache-daemon benefits from this change. That needs to be justified.

As to the log message (this is 70% style, but there are consequences), we tend not to say "This commit does X"---because such a statement is often insufficient to illustrate if the commit indeed does X, and explain why it is a good thing to do X in the first place.

Instead, we first explain that the current system does not do X (in present tense, so we do NOT say "previously we did not do X"), then explain why doing X would be a good thing, and finally give an order to the codebase to start doing X. For this change, it might look like this:

    The credential subsystem has assumed that lines always end with
    LF.  On a system whose text file ends with CRLF (e.g. Windows),
    this assumption causes the CR at the end of the line as an extra
    byte appended to the data, and worse, there is no way to send an
    "empty line" to signal an end of a group of lines, because such
    a line would be taken as a line with a lone CR on it.

    Because credential-store writes to and reads from a text file on
    disk, which users may manually edit with their editor and turn
    the line-ending to CRLF.

    Accept lines that end with CR/LF, as well as those that end with
    LF.

    Note that the credential-cache--daemon is not touched, because
    it is only about interacting with in-core cache, and there is
    nowhere CRLF comes into the picture.

Note: I didn't know why we need to touch the daemon, and that is why
I wrote the paragraph on it as such, but I expect that the real log
message to justify the change would be quite different and explain
why the daemon code needs the same change well---such a description
would not be "Note that", but would come before the "Accept lines
that end with..." and after the paragraph about credential-store.

Thanks.

On Mon, Sep 28, 2020 at 01:58:03PM -0700, Junio C Hamano wrote:

I suspect it doesn't need touched, because it is internal to git-daemon. But it does handle CRLF for some lines, because the first patch modified credential_read(), which the daemon builds on (and which _is_ user-facing via git-credential). So there's perhaps an argument that these calls should just be made consistent, even though the only one who would write them is our matching client. If that is the argument to be made, I think it would make sense to do so in a separate patch, since there's no functional change.

(I'm also slightly puzzled that anybody on Windows would care about credential-cache, since it require unix sockets. But maybe in a world of WSL people are actually able to mix the two. I confess I haven't kept up with the state of things in Windows).

-Peff

Jeff King <peff@peff.net> writes:

True, so some, if not all, parts of these changes start to look more and more like "I change LF to CRLF purely for political correctness, even I know nobody sends CRLF in these cases (or "even I do not know if anybody sends CRLF in these cases", which essentially amounts to the same thing but may be worse)", needless code churns.

Sigh.

Hi Junio & Peff,

On Wed, 30 Sep 2020, Junio C Hamano wrote:

Indeed. I did not make the connection that `credential-cache--daemon` uses Unix sockets internally (and hence if you try to use it on Windows, you are greeted with the message "credential-cache--daemon unavailable; no unix socket support").

Will drop this part from the next iteration.

Ciao, Dscho

On Mon, Sep 28, 2020 at 4:41 AM Nikita Leonov via GitGitGadget <gitgitgadget@gmail.com> wrote:

note that this test was put in place to protect users from regressions like the one we got after the release of 2.26.1 where users that had '\r' as part of their credentials were getting an error[1]

while I am sympathetic to the change (indeed I proposed something similar, but was reminded by Peff that while it looks like a text file it was designed to be considered opaque and therefore should use UNIX LF as record terminator by specification), I am concerned this could result in a similar regression since we know they are still users out there that had modified this file manually (something that was not recommended) and are currently relying on the fact that these lines are invalid and therefore silently ignored.

Carlo

[1] https://lore.kernel.org/git/ad80aa0d-3a35-6d7e-7958-b3520e16c855@ed4u.de/

Carlo Arenas <carenas@gmail.com> writes:

I think you meant to remind us and this thread of the earlier review and discussion thread, which begins at

  https://lore.kernel.org/git/20200426234750.40418-1-carenas@gmail.com/

And thanks for doing so---I totally forgot about it.

On Mon, Sep 28, 2020 at 04:26:38PM -0700, Carlo Arenas wrote:

Going over that old thread, I'm not sure that anybody raised a real use case where somebody expected a CR at the end of a line. The discussion was mostly about whether proposed changes would or would not be compatible with existing behavior.

I think that:

  - we'd never write a raw CR ourselves, as we'd urlencode the character

  - if somebody did put in a raw CR manually like:

      https://example.com\r\n

    then we'd currently fail to match "example.com". Which is probably
    not what they wanted. I suspect that \r in a hostname is bogus
    anyway (certainly curl will complain about it).

  - you could do the same in a path:

      https://example.com/foo\r\n

    which _is_ legal, but I think we'd generally ignore it completely
    unless credential.usehttppath is set (for network-accessible
    requests, that is; you could probably point your local cert file at
    something bogus, but I think the main areas of interest here are
    people manipulating the credential protocol via malicious urls).

So I'm OK loosening the format in the name of convenience, as long as we're confident that it's not opening up any security problems. I can't think of any such problems, though.

It sounds from your email like you may have found me arguing the opposite. That's entirely possible. ;) But I couldn't find it in the thread Junio linked.

-Peff

Jeff King <peff@peff.net> writes:

I may be misremembering, but an argument I recall against the kind of change we are dicussing now was that we ignore such an entry right now, and the user may have added an entry for the host anew, possibly with a more recent password. Changing the parsing to ignore CR would silently resurrect such a stale entry that the user has written off as unused, and depending on the order of entries in the file, a site that used to work may start failing suddenly.

I don't think I'd care too heavily either way. As long as other people will deal with possible backward-incompatibility fallout, I do not mind seeing the change ;-).

I still don't see why we need to touch the cache-daemon, though.

Thanks.

On Mon, Sep 28, 2020 at 05:41:14PM -0700, Junio C Hamano wrote:

Yeah, that is probably what would happen. I have to admit that it's such an obscure case that I'm not sure I really care. It's unlikely in practice, and if somebody did report such a case, I think my first response would be "well, why did you have a broken entry stuck in your file?".

Yeah, I touched on that more in another response.

-Peff

Jeff King <peff@peff.net> writes:

I think we know the likely answer. "I once used Windows to edit the file manually".

After which the file looks broken, so the user may have re-added via the credential API (with LF line ending) a new entry for the host and have been happily using the result.

On Mon, Sep 28, 2020 at 05:54:37PM -0700, Junio C Hamano wrote:

I wrote something less charitable towards the user at first, and then toned it down. But I think I toned it down too much. Maybe my response would be "garbage in, garbage out; it was lucky that it worked before".

-Peff

Jeff King <peff@peff.net> writes:

OK, so what's the final verdict from us? This is good enough to move forward as-is?

On Wed, Sep 30, 2020 at 03:25:09PM -0700, Junio C Hamano wrote:

I'd prefer to see the credential-cache--daemon changes either dropped, or split out into a separate patch with the justification of: this probably doesn't matter in practice, but it makes the whole protocol between client and server treat CRLF consistently.

I had also raised a question in:

  https://lore.kernel.org/git/20200929004220.GC898702@coredump.intra.peff.net/

about whether they just care about the empty line, or about CRLF on data lines. If the former (which is what the commit message claims), then I think we can do something much simpler. But I suspect it is the latter, and it is simply the commit message that is a bit misleading.

Other than those nits, I think the series is OK.

-Peff

Jeff King <peff@peff.net> writes:

Sure. But credential-store side is also iffy; it is not like they want CRLF on data lines (if they want CR in data, that needs to be encoded). The only reason I can think of that the change to "-store" makes any difference is when people edit it, but the file is not designed to be manually edited, so even that part of the series needs a better justification. It's not like "We want to be compatible" without "why it is better to be compatible" is a good rationale, when we define the file format not to be manually edited in the first place.

Thanks.

On Wed, Sep 30, 2020 at 03:56:27PM -0700, Junio C Hamano wrote:

Yeah, I agree that just teaching git-credential's stdin to handle CR would be an OK stopping point. I don't have a strong opinion on credential-store's on disk format. At least allowing CRLF there is _plausibly_ useful, unlike credential-cache--daemon's pipe. And I doubt that making the change would hurt anybody.

-Peff

Jeff King <peff@peff.net> writes:

I agree 100% with all of the above, including the part that says it is also OK to let credential-store read from its files with line ending converted to CRLF.

Thanks.

Hi,

On Thu, 1 Oct 2020, Junio C Hamano wrote:

After mulling over this, I am more inclined to drop the credential-store changes, too. That file _is_ an implementation detail, and not intended for interactive editing. And as Carlo pointed out so correctly, the regression tests were not introduced "just for fun", they really want to make sure that we keep handling CR/LF the way we currently do.

So I will drop both credential-cache--daemon and credential-store changes from the next iteration.

Ciao, Dscho

"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> writes:

Oh, that's OK. Thanks for resurrecting the stalled patch. I didn't mean the suggested action while I was waiting to be an exhaustive audit---I merely wanted to avoid having to queue a few more separate ones soon after applying a patch for just a single place.

I am kind of surprised to see that these need to be touched at all.

The change to credential-store is sort-of understandable, as the file it uses could be hand-modified in an editor and its line ending could have been changed to CRLF, hence it needs to be prepared.

But does inter-process communication with the daemon need to care about CRLF line endings? Do parts of the protocol within the credential system use platform specific line ending?

From: Nikita Leonov <nykyta.leonov@gmail.com>

This fix makes using Git credentials more friendly to Windows users: it allows a credential helper to communicate using CR/LF line endings ("DOS line endings" commonly found on Windows) instead of LF-only line endings ("Unix line endings").

Note that this changes the behavior a bit: if a credential helper produces, say, a password with a trailing Carriage Return character, that will now be culled even when the rest of the lines end only in Line Feed characters, indicating that the Carriage Return was not meant to be part of the line ending.

In practice, it seems _very_ unlikely that something like this happens. Passwords usually need to consist of non-control characters, URLs need to have special characters URL-encoded, and user names, well, are names.

However, it _does_ help on Windows, where CR/LF line endings are common: as unrecognized commands are simply ignored by the credential machinery, even a command like `quit\r` (which is clearly intended to abort) would simply be ignored (silently) by Git.

So let's change the credential machinery to accept both CR/LF and LF line endings.

While we do this for the credential helper protocol, we do _not_ adjust `git credential-cache--daemon` (which won't work on Windows, anyway, because it requires Unix sockets) nor `git credential-store` (which writes the file `~/.git-credentials` which we consider an implementation detail that should be opaque to the user, read: we do expect users _not_ to edit this file manually).

Signed-off-by: Nikita Leonov <nykyta.leonov@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
    Prepare git credential to read input with DOS line endings
    
    This contribution came in via Git for Windows
    [https://github.com/git-for-windows/git/pull/2516].
    
    Sadly, I did not find the time to go through all the changes of 8f309aeb
    ("strbuf: introduce strbuf_getline_{lf,nul}()", 2016-01-13) (as Junio
    asked
    [https://public-inbox.org/git/xmqqmu9lnjdh.fsf@gitster-ct.c.googlers.com]
    ). Rather than delaying this patch indefinitely, I admit defeat on that
    angle.
    
    Changes since v2:
    
     * Dropped the credential-cache--daemon and credential-store changes
       again.
     * Enhanced the commit message (also explaining why we don't touch the
       daemon and the store).
    
    Changes since v1:
    
     * Added a commit to adjust credential-daemon and credential-store in
       the same manner.
     * Adjusted the documentation accordingly.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-710%2Fdscho%2Fcrlf-aware-git-credential-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-710/dscho/crlf-aware-git-credential-v3
Pull-Request: https://github.com/git/git/pull/710

Range-diff vs v2:

 1:  27f6400a21 ! 1:  f6eeb18d3a credential.c: fix credential reading with regards to CR/LF
     @@ Metadata
      Author: Nikita Leonov <nykyta.leonov@gmail.com>
      
       ## Commit message ##
     -    credential.c: fix credential reading with regards to CR/LF
     +    credential: treat CR/LF as line endings in the credential protocol
      
     -    This fix makes using Git credentials more friendly to Windows users. In
     -    previous version it was unable to finish input correctly without
     -    configuration changes (tested in PowerShell, CMD, Cygwin).
     +    This fix makes using Git credentials more friendly to Windows users: it
     +    allows a credential helper to communicate using CR/LF line endings ("DOS
     +    line endings" commonly found on Windows) instead of LF-only line endings
     +    ("Unix line endings").
      
     -    We know credential filling should be finished by empty input, but the
     -    current implementation does not take into account CR/LF ending, and
     -    hence instead of the empty string we get '\r', which is interpreted as
     -    an incorrect string.
     +    Note that this changes the behavior a bit: if a credential helper
     +    produces, say, a password with a trailing Carriage Return character,
     +    that will now be culled even when the rest of the lines end only in Line
     +    Feed characters, indicating that the Carriage Return was not meant to be
     +    part of the line ending.
      
     -    So this commit changes default reading function to a more Windows
     -    compatible reading function.
     +    In practice, it seems _very_ unlikely that something like this happens.
     +    Passwords usually need to consist of non-control characters, URLs need
     +    to have special characters URL-encoded, and user names, well, are names.
     +
     +    However, it _does_ help on Windows, where CR/LF line endings are common:
     +    as unrecognized commands are simply ignored by the credential machinery,
     +    even a command like `quit\r` (which is clearly intended to abort) would
     +    simply be ignored (silently) by Git.
     +
     +    So let's change the credential machinery to accept both CR/LF and LF
     +    line endings.
     +
     +    While we do this for the credential helper protocol, we do _not_ adjust
     +    `git credential-cache--daemon` (which won't work on Windows, anyway,
     +    because it requires Unix sockets) nor `git credential-store` (which
     +    writes the file `~/.git-credentials` which we consider an implementation
     +    detail that should be opaque to the user, read: we do expect users _not_
     +    to edit this file manually).
      
          Signed-off-by: Nikita Leonov <nykyta.leonov@gmail.com>
          Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
 2:  f69076036f < -:  ---------- credentials: make line reading Windows compatible
 3:  61baea1061 < -:  ---------- docs: make notes regarding credential line reading

 credential.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)