v3, 4 partsgitfaq: give advice on using eol attribute in gitattributes

This series introduces some additional Git FAQ entries on various topics. They are all things I've seen in my professional life or on Stack Overflow, so I've written documentation.

There were some suggestions in the past that the text "modify, tamper with, or buffer" might be somewhat redundant, but I've chosen to keep the text as it is to avoid arguments like, "Well, buffering the entire request or response isn't really modifying it, so Git should just work in that situation," when we already know that doesn't work.

Changes from v2 (partial):
* Add documentation on proxies to the configuration documentation as
  well.
* Mention some security problems that are known to occur with TLS MITM
  proxies.  This mirrors the similar Git LFS documentation.
* Provide a documentation example about how to use proxies with SSH.
* Recommend running a `git fsck` after syncing with rsync.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.

brian m. carlson (4):
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees
  doc: mention that proxies must be completely transparent

 Documentation/config/http.txt |   5 ++
 Documentation/gitfaq.txt      | 105 ++++++++++++++++++++++++++++++++--
 2 files changed, 104 insertions(+), 6 deletions(-)

In the FAQ, we tell people how to use the text attribute, but we fail to explain what to do with the eol attribute. As we ourselves have noticed, most shell implementations do not care for carriage returns, and as such, people will practically always want them to use LF endings. Similar things can be said for batch files on Windows, except with CRLF endings.

Since these are common things to have in a repository, let's help users make a good decision by recommending that they use the gitattributes file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which originally referred to "the following entry", even though a new entry has been inserted in between. The cross-reference notation should prevent this from occurring and provide a link in formats, such as HTML, which support that.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Sounds good.

Thanks for being forward-looking and extra careful.

"this" being ... Not what gets stored in the object database but what is done to the working tree.

What is being "overridden" is that the earlier two mentioned here are configuration variables that apply to _all_ text files in general, and the attribute mechanism is a way to give settings that are more tailored for each path. I think the reason I found the above a bit hard to understand when I read it for the first time was because it didn't "click" that this paragraph was about configuration giving the general default and attributes overriding it. Perhaps...

    ... are used in the repository.  The `core.autocrlf` and
    `core.eol` configuration variables specify what line-ending
    convention is followed when any text file is checked out.  You
    can also use the `eol` attribute (e.g., "eol=crlf") to override
    which files get what line-ending treatment.

or something?

I'd strike "(e.g., for functionality reasons)" out, as the next paragraph makes it sufficiently clear.

On 2024-07-04 at 05:22:13, Junio C Hamano wrote:

Sure, that sounds like a nice improvement.

Sure, I can do that.

Users very commonly want to sync their working tree with uncommitted changes across machines, often to carry across in-progress work or stashes. Despite this not being a recommended approach, users want to do it and are not dissuaded by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing service, which is a bad idea. Users have reported problems where they end up with duplicate files that won't go away (with names like "file.c 2"), broken references, oddly named references that have date stamps appended to them, missing objects, and general corruption and data loss. That's because almost all of these tools sync file by file, which is a great technique if your project is a single word processing document or spreadsheet, but is utterly abysmal for Git repositories because they don't necessarily snapshot the entire repository correctly. They also tend to sync the files immediately instead of when the repository is quiescent, so writing multiple files, as occurs during a commit or a gc, can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that the repository is quiescent, so let's suggest that and dissuade people from using cloud syncing tools. Let's tell people about common things they should be aware of before doing this and that this is still potentially risky. Additionally, let's tell people that Git's security model does not permit sharing working trees across users in case they planned to do that. While we'd still prefer users didn't try to do this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 48 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 46 insertions(+), 2 deletions(-)

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

I can see (and appreciate) that you struggled to find a good section to piggyback on, instead of giving this topic its own section. But do these two make a good mix? They seem to be totally different topics.

A naïve reader may say "but isn't it the point of these cloud syncing service that they will eventually catch up???" and we may want to have a good story why it does not work.

    You create many objects in one repository in loose form, cloud
    syncing service kicks in to transfer them to the second
    repository, and then in the original repository an auto-gc kicks
    in so some of the loose objects fail to propagate.  The packfile
    that is the result of auto-gc will eventually propagate to the
    second repository, but before it completes, the second
    repository would be in an inconsistent state, and especially if
    the ref updates are propagated before objects, then the second
    repository will be in a corrupt state.  It would be a disaster
    if another auto-gc kicked in there.

is one scenario I came up with.

On 2024-07-04 at 05:21:55, Junio C Hamano wrote:

I can try again.

The most common situation we see is that refs tend to be renamed to things like "refs/heads/main 2", which is obviously not a valid refname and doesn't work, or the ref gets rolled back to an older version. Working trees also get stuck into weird states where files keep coming back or getting deleted, or the index gets two differently named copies, neither of which is "index".

It is _less_ likely that objects are renamed, but it could be that the tool thinks they've been legitimately deleted if the loose objects get packed and then they do get deleted elsewhere without another source of those objects existing. I'm not sure how object loss happens in the real world with these services, but there have been users reporting it on StackOverflow, so I'm confident it does occur.

If we have users who ask about this, I'm happy to answer them on the list. I don't want to explain the various and sundry scenarios in the FAQ entry in order to keep it short, but I can find several examples of problems if need be.

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Yeah, any time two repositories that are "cloud synched" are accessed simultaneously, all h*ll can easily break loose. You may move your 'master' branch to a commit while the other one may move their 'master' branch to a different commit. You may end up having "master" that points at one of these commits but one of you may have already lost the only reference to the commit you wanted to have at the tip of your 'master' branch. One of you may even trigger auto-gc to spread the damage.

OK, that approach would work as long as you are still involved in the project, but having even one concrete example would help in the longer term to (1) reduce the bus factor and (2) save time you do not have to spend responding to every such question.

Thanks.

Many corporate environments and local systems have proxies in use. Note the situations in which proxies can be used and how to configure them. At the same time, note what standards a proxy must follow to work with Git. Explicitly call out certain classes that are known to routinely have problems reported various places online, including in the Git for Windows issue tracker and on Stack Overflow, and recommend against the use of such software, noting that they are associated with myriad security problems (including, for example, breaking sandboxing and image integrity[0], and, for TLS middleboxes, the use of insecure protocols and ciphers and lack of certificate verification[1]). Don't mention the specific nature of these security problems in the FAQ entry because they are extremely numerous and varied and we wish to keep the FAQ entry relatively brief.

[0] https://issues.chromium.org/issues/40285192 [1] https://faculty.cc.gatech.edu/~mbailey/publications/ndss17_interception.pdf

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

We already document in the FAQ that proxies must be completely transparent and not modify the request or response in any way, but add similar documentation to the http.proxy entry. We know that while the FAQ is very useful, users sometimes are less likely to read in favor of the documentation specific to an option or command, so adding it in both places will help users be adequately informed.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/config/http.txt | 5 +++++
 1 file changed, 5 insertions(+)

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Just to help other readers:

 v1
 https://lore.kernel.org/git/20211020010624.675562-1-sandals@crustytoothpaste.net/

 v2
 https://lore.kernel.org/git/20211107225525.431138-1-sandals@crustytoothpaste.net/

are where the previous discussions are found.

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Buffering the entire thing will break because ...? Deadlock? Or is there anything more subtle going on?

Are we affected by any frame boundary (do we even notice?) that happens at layer lower than our own pkt-line layer at all (i.e. we sent two chunks and we fail to work on them correctly if the network collapses them into one chunk, without changing a single byte, just changing the number of read() system calls that reads them?)?

On 2024-07-04 at 05:22:27, Junio C Hamano wrote:

When we use the smart HTTP protocol, the server sends keep-alive and status messages as one of the data streams, which is important because (a) the user is usually impatient and wants to know what's going on and (b) it may take a long time to pack the data, especially for large repositories, and sending no data may result in the connection being dropped or the client being served a 500 by an intermediate layer. We know this does happen and I've seen reports of it.

We've also seen some cases where proxies refuse to accept
Transfer-Encoding: chunked (let's party like it's 1999) and send a 411
back since there's no Content-Length header.  That's presumably because
they want to scan the contents for "bad" data all in one chunk, but Git
has to stream the contents unless the data fits in the buffer size.
(This is the one case where http.postBuffer actually makes a
difference.)  I very much doubt that the appliance actually wants to get
a 2 GiB payload to scan, since it probably doesn't have tons of memory
in the first place, but that is what it's asking for.

No, that's not a problem. We read four bytes for the pkt-line header, and then we read the entire body based on that length until we get all of it. This is also the way OpenSSL works for TLS packets and is known to work well. If the underlying TCP connection provides a partial or incomplete packet (which can happen due to MTU), we'll just block until the rest comes in, which is fine.

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

And this is an example of "a proxy that buffers the data, without modifying or tampering with, would still break transport"?

This is "a proxy that wanted to buffer the data but failed to do so" that ended up modifying the data Gits sitting at both ends of the connection can observe, so it is a bit different issue. It clearly falls into "modify or tampering with" category.

I forgot to say this clearly when I wrote the message you are responding to, but I am trying to see if we can clarify the "or buffer" part in "modify, tamper with, or buffer", as offhand I did not think of a reason why a proxy would break the Git communication if it receives a segment that was 2MB originally from upload-pack, and forwards the contents of the segment in two 1MB segments without tampering or modifying the payload bytes at all to fetch-pack.

Thanks.

On 2024-07-06 at 05:59:57, Junio C Hamano wrote:

Yes. The connection usually ends up dropped from the view of the client, which is hard to debug (because it also looks like a network problem, except often without any output from the remote side).

On Thu, Jul 04, 2024 at 09:23:28PM +0000, brian m. carlson wrote:

Additionally, I think for non-HTTP transports (think proxying ssh through socat or similar), buffering the v0 protocol is likely a total disaster. The fetch protocol assumes both sides spewing at each other in real time.

HTTP, even v0, follows a request/response model, so we're safer there. I do think some amount of buffering is often going to be OK in practice. You'd get delayed keep-alives and progress reports, which may range from "annoying" to "something in the middle decided to time out". So I'm OK with just telling people "make sure your proxies aren't buffering" as a general rule, rather than trying to get into the nitty gritty of what is going to break and how.

-Peff

Jeff King <peff@peff.net> writes:

Yeah, beyond one "window" that a series of "have"s are allowed to be in flight, no further "have"s are sent before seeing an "ack/nack" response, so if you buffer too much, they can deadlock fairly easily.

Sounds fair.  Thanks.

This series introduces some additional Git FAQ entries on various topics. They are all things I've seen in my professional life or on Stack Overflow, so I've written documentation.

Changes from v3:
* Improve text for eol options in .gitattributes.
* Split working tree syncing into its own category.
* Add some more explanation about what might go wrong in such a case.
* Rephrase some text to avoid repetition.

Changes from v2 (partial):
* Add documentation on proxies to the configuration documentation as
  well.
* Mention some security problems that are known to occur with TLS MITM
  proxies.  This mirrors the similar Git LFS documentation.
* Provide a documentation example about how to use proxies with SSH.
* Recommend running a `git fsck` after syncing with rsync.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.

brian m. carlson (4):
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees
  doc: mention that proxies must be completely transparent

 Documentation/config/http.txt |   5 ++
 Documentation/gitfaq.txt      | 109 ++++++++++++++++++++++++++++++++--
 2 files changed, 110 insertions(+), 4 deletions(-)

Many corporate environments and local systems have proxies in use. Note the situations in which proxies can be used and how to configure them. At the same time, note what standards a proxy must follow to work with Git. Explicitly call out certain classes that are known to routinely have problems reported various places online, including in the Git for Windows issue tracker and on Stack Overflow, and recommend against the use of such software, noting that they are associated with myriad security problems (including, for example, breaking sandboxing and image integrity[0], and, for TLS middleboxes, the use of insecure protocols and ciphers and lack of certificate verification[1]). Don't mention the specific nature of these security problems in the FAQ entry because they are extremely numerous and varied and we wish to keep the FAQ entry relatively brief.

[0] https://issues.chromium.org/issues/40285192 [1] https://faculty.cc.gatech.edu/~mbailey/publications/ndss17_interception.pdf

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

Users very commonly want to sync their working tree with uncommitted changes across machines, often to carry across in-progress work or stashes. Despite this not being a recommended approach, users want to do it and are not dissuaded by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing service, which is a bad idea. Users have reported problems where they end up with duplicate files that won't go away (with names like "file.c 2"), broken references, oddly named references that have date stamps appended to them, missing objects, and general corruption and data loss. That's because almost all of these tools sync file by file, which is a great technique if your project is a single word processing document or spreadsheet, but is utterly abysmal for Git repositories because they don't necessarily snapshot the entire repository correctly. They also tend to sync the files immediately instead of when the repository is quiescent, so writing multiple files, as occurs during a commit or a gc, can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that the repository is quiescent, so let's suggest that and dissuade people from using cloud syncing tools. Let's tell people about common things they should be aware of before doing this and that this is still potentially risky. Additionally, let's tell people that Git's security model does not permit sharing working trees across users in case they planned to do that. While we'd still prefer users didn't try to do this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 52 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

We already document in the FAQ that proxies must be completely transparent and not modify the request or response in any way, but add similar documentation to the http.proxy entry. We know that while the FAQ is very useful, users sometimes are less likely to read in favor of the documentation specific to an option or command, so adding it in both places will help users be adequately informed.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/config/http.txt | 5 +++++
 1 file changed, 5 insertions(+)

In the FAQ, we tell people how to use the text attribute, but we fail to explain what to do with the eol attribute. As we ourselves have noticed, most shell implementations do not care for carriage returns, and as such, people will practically always want them to use LF endings. Similar things can be said for batch files on Windows, except with CRLF endings.

Since these are common things to have in a repository, let's help users make a good decision by recommending that they use the gitattributes file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which originally referred to "the following entry", even though a new entry has been inserted in between. The cross-reference notation should prevent this from occurring and provide a link in formats, such as HTML, which support that.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)