threads / patch / 56857

v2, 3 partsgitfaq: add documentation on proxies

Subject: [PATCH v2 1/3] gitfaq: add documentation on proxies

## tl;dr

19 messages between Nov 7, 2021 and Jan 6, 2022. Diffs are folded; open one to read it.

replies: 18people: 4as markdown or json

brian m. carlson· Nov 7, 2021, 22:55 UTC · lore

[PATCH v2 0/3] Additional FAQ entries

This series introduces some additional Git FAQ entries on various topics. They are all things I've seen in my professional life or on Stack Overflow, so I've written documentation.

I've opted not to include backing up repositories in the syncing patch because I think they're separate topics. We could well end up with an additional FAQ entry on that topic, which is left as an exercise for the reader.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.
brian m. carlson (3):
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees
 Documentation/gitfaq.txt | 90 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 84 insertions(+), 6 deletions(-)
brian m. carlson· Nov 7, 2021, 22:55 UTC · re: brian m. carlson · lore

Many corporate environments and local systems have proxies in use. Note the situations in which proxies can be used and how to configure them. At the same time, note what standards a proxy must follow to work with Git. Explicitly call out certain classes that are known to routinely have problems reported various places online, including in the Git for Windows issue tracker and on Stack Overflow, and recommend against the use of such software.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
Show changes to Documentation/gitfaq.txt +23 −0
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 946691c153..5c21951f7b 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -241,6 +241,29 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+[[proxy]]
+Can I use a proxy with Git?::
+	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
+	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
+	it also can be configured with `http.proxy` and similar options for HTTPS (see
+	linkgit:git-config[1]).  The `http.proxy` and related options can be
+	customized on a per-URL pattern basis.  In addition, Git can in theory
+	function normally with transparent proxies that exist on the network.
++
+For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
+include `netcat` and `socat`.  However, they must be configured not to exit when
+seeing EOF on standard input, which usually means that `netcat` will require
+`-q` and `socat` will require a timeout with something like `-t 10`.
++
+Note that in all cases, for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that
+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.
+
 Design
 ------
Eric Sunshine· Nov 7, 2021, 23:27 UTC · re: brian m. carlson · lore

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 7 quoted lines
> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.
A couple minor comments; may not be worth a reroll or discussion...
Show 16 quoted lines
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> +[[proxy]]
> +Can I use a proxy with Git?::
> +       Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
> +       `https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
> +       it also can be configured with `http.proxy` and similar options for HTTPS (see
> +       linkgit:git-config[1]).  The `http.proxy` and related options can be
> +       customized on a per-URL pattern basis.  In addition, Git can in theory
> +       function normally with transparent proxies that exist on the network.
> ++
> +For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
> +include `netcat` and `socat`.  However, they must be configured not to exit when
> +seeing EOF on standard input, which usually means that `netcat` will require
> +`-q` and `socat` will require a timeout with something like `-t 10`.

I've seen this come up on the mailing list a couple times recently, though I haven't really followed along and don't use Git through an SSH proxy, thus I did have to go do some reading to understand what this is talking about. Perhaps people searching out this FAQ entry will already have sufficient context to understand what this is saying, so maybe no additional context is needed here. However, I was wondering if it might make sense for this to give a bit of reason explaining _why_ these tools need to be configured to not exit immediately upon EOF. As it stands now, this solution is a black box; it will work but people won't understand why. Perhaps that doesn't matter since most people consulting a FAQ like this probably just want to get the thing working and don't care about the underlying details. Then again, if the underlying reason is made more readily apparent, maybe this knowledge can become more widespread.

> +Note that in all cases, for Git to work properly, the proxy must be completely
> +transparent.  The proxy cannot modify, tamper with, change, or buffer the
> +connection in any way, or Git will almost certainly fail to work.  Note that

"modify, tamper with, change" sounds like it came from the Department of Redundancy Department. I like the sound of "tamper with" since the image it conveys feels quite suitable here. Perhaps this could be simplified to:

   The proxy cannot tamper with or buffer the...
Show 5 quoted lines
> +many proxies, including many TLS middleboxes, Windows antivirus and firewall
> +programs other than Windows Defender and Windows Firewall, and filtering proxies
> +fail to meet this standard, and as a result end up breaking Git.  Because of the
> +many reports of problems, we recommend against the use of these classes of
> +software and devices.
brian m. carlson· Nov 8, 2021, 01:53 UTC · re: Eric Sunshine · lore

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

On 2021-11-07 at 23:27:24, Eric Sunshine wrote:
Show 14 quoted lines
> I've seen this come up on the mailing list a couple times recently,
> though I haven't really followed along and don't use Git through an
> SSH proxy, thus I did have to go do some reading to understand what
> this is talking about. Perhaps people searching out this FAQ entry
> will already have sufficient context to understand what this is
> saying, so maybe no additional context is needed here. However, I was
> wondering if it might make sense for this to give a bit of reason
> explaining _why_ these tools need to be configured to not exit
> immediately upon EOF. As it stands now, this solution is a black box;
> it will work but people won't understand why. Perhaps that doesn't
> matter since most people consulting a FAQ like this probably just want
> to get the thing working and don't care about the underlying details.
> Then again, if the underlying reason is made more readily apparent,
> maybe this knowledge can become more widespread.

I'll try to see if I can stuff in a sentence there about why that's necessary. I think I understand it sufficiently well to summarize it.

Show 6 quoted lines
> "modify, tamper with, change" sounds like it came from the Department
> of Redundancy Department. I like the sound of "tamper with" since the
> image it conveys feels quite suitable here. Perhaps this could be
> simplified to:
> 
>    The proxy cannot tamper with or buffer the...

I realize this sounds redundant, but I'm trying to avoid the situation where people say, "I'm not _tampering_ with it, since I'm authorized to do this by the company. I'm just modifying it to remove this inappropriate content/malware/data leak." My goal here is to make it crystal clear that if you do this, you'll break things, and provide ammunition for people to go to their IT departments and say, "Look, your proxy prevents me from doing my job. The Git developers say so. Fix it."

I can drop one of "change" and "modify", though, since I think they're synonyms.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Junio C Hamano· Nov 8, 2021, 21:24 UTC · re: brian m. carlson · lore

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> I realize this sounds redundant, but I'm trying to avoid the situation
> where people say, "I'm not _tampering_ with it, since I'm authorized to
> do this by the company.  I'm just modifying it to remove this
> inappropriate content/malware/data leak."

It might be a good strategy to educate users by leaving a deliberate hole to let them quibble with such an excuse. When it comes to on-the-wire protocol, their management do not have authority to grant them right to tamper with the data and that is not limited to Git.

Johannes Schindelin· Jan 4, 2022, 13:40 UTC · re: brian m. carlson · lore

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

Hi brian,
On Sun, 7 Nov 2021, brian m. carlson wrote:
Show 5 quoted lines
> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for

Not a big issue, but I think there is an "at" or "to" missing before "various places online".

Show 27 quoted lines
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/gitfaq.txt | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
>
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 946691c153..5c21951f7b 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,29 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>
> +[[proxy]]
> +Can I use a proxy with Git?::
> +	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
> +	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
> +	it also can be configured with `http.proxy` and similar options for HTTPS (see
> +	linkgit:git-config[1]).  The `http.proxy` and related options can be
> +	customized on a per-URL pattern basis.  In addition, Git can in theory
> +	function normally with transparent proxies that exist on the network.
> ++
> +For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
> +include `netcat` and `socat`.

The first idea I had after reading this is: where are these examples documented? Certainly not where I expected them, namely at https://git-scm.com/docs/git-config#Documentation/git-config.txt-coregitProxy

And then I got puzzled. Why would `gitproxy` be used for the _SSH_ protocol? And I don't think it is used. I think it is only used for connections using the `git://` protocol. I might very easily be wrong, of course.

> However, they must be configured not to exit when +seeing EOF on
> standard input, which usually means that `netcat` will require +`-q` and
> `socat` will require a timeout with something like `-t 10`.

IMHO it would make sense to add a concrete example, or maybe even two concrete examples, one for `netcat` and one for `socat`.

Show 10 quoted lines
> ++
> +Note that in all cases, for Git to work properly, the proxy must be completely
> +transparent.  The proxy cannot modify, tamper with, change, or buffer the
> +connection in any way, or Git will almost certainly fail to work.  Note that
> +many proxies, including many TLS middleboxes, Windows antivirus and firewall
> +programs other than Windows Defender and Windows Firewall, and filtering proxies
> +fail to meet this standard, and as a result end up breaking Git.  Because of the
> +many reports of problems, we recommend against the use of these classes of
> +software and devices.
> +
This is good advice.

Ciao, Dscho

brian m. carlson· Nov 7, 2021, 22:55 UTC · re: brian m. carlson · lore

[PATCH v2 3/3] gitfaq: add entry about syncing working trees

Users very commonly want to sync their working tree with uncommitted changes across machines, often to carry across in-progress work or stashes. Despite this not being a recommended approach, users want to do it and are not dissuaded by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing service, which is a bad idea. Users have reported problems where they end up with duplicate files that won't go away (with names like "file.c 2"), broken references, oddly named references that have date stamps appended to them, missing objects, and general corruption and data loss. That's because almost all of these tools sync file by file, which is a great technique if your project is a single word processing document or spreadsheet, but is utterly abysmal for Git repositories because they don't necessarily snapshot the entire repository correctly. They also tend to sync the files immediately instead of when the repository is quiescent, so writing multiple files, as occurs during a commit or a gc, can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that the repository is quiescent, so let's suggest that and dissuade people from using cloud syncing tools. Let's tell people about common things they should be aware of before doing this and that this is still potentially risky. Additionally, let's tell people that Git's security model does not permit sharing working trees across users in case they planned to do that. While we'd still prefer users didn't try to do this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 47 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 45 insertions(+), 2 deletions(-)
Show changes to Documentation/gitfaq.txt +45 −1
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index ae1b526565..e5dab89d6c 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::
@@ -185,6 +185,49 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  This is
+	potentially risky and in some cases can cause repository corruption or data
+	loss.
++
+Usually, doing so will cause `git status` to need to re-read every file in the
+working tree.  Additionally, Git's security model does not permit sharing a
+working tree across untrusted users, so it is only safe to sync a working tree
+if it will only be used by a single user across all machines.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+If you do this, the recommended approach is to use `rsync -a --delete-after`
+(ideally with an encrypted connection such as with `ssh`) on the root of
+repository.  You should ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc` and operations invoked by your editor).
++
+Be aware that even with these recommendations, syncing in this way has some risk
+since it bypasses Git's normal integrity checking for repositories, so having
+backups is advised.  You may also with to do a `git fsck` to verify the
+integrity of your data on the destination system after syncing.
+
 Common Issues
 -------------
Eric Sunshine· Nov 8, 2021, 00:12 UTC · re: brian m. carlson · lore

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 17 quoted lines
> Users very commonly want to sync their working tree with uncommitted
> changes across machines, often to carry across in-progress work or
> stashes.  Despite this not being a recommended approach, users want to
> do it and are not dissuaded by suggestions not to, so let's recommend a
> sensible technique.
> [...]
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -185,6 +185,49 @@ Then, you can adjust your push URL to use `git@example_author` or
> +[[sync-working-tree]]
> +How do I sync a working tree across systems?::
> +       First, decide whether you want to do this at all.  Git usually works better
> +       when you push or pull your work using the typical `git push` and `git fetch`
> +       commands and isn't designed to share a working tree across systems.  This is
> +       potentially risky and in some cases can cause repository corruption or data
> +       loss.

The phrase "usually works better" makes this statement feel weak, thus it may not convey the potential severity of the issue. I wonder if rewording it something like this would make the statement more forceful:

    Git works best when you `git push` and `git pull` your work
    between machines; it is not designed to share a working tree
    across systems. [...]
Show 12 quoted lines
> +Usually, doing so will cause `git status` to need to re-read every file in the
> +working tree.  Additionally, Git's security model does not permit sharing a
> +working tree across untrusted users, so it is only safe to sync a working tree
> +if it will only be used by a single user across all machines.
> ++
> +It is important not to use a cloud syncing service to sync any portion of a Git
> +repository, since this can cause corruption, such as missing objects, changed
> +or added files, broken refs, and a wide variety of other corruption.  These
> +services tend to sync file by file and don't understand the structure of a Git
> +repository.  This is especially bad if they sync the repository in the middle of
> +it being updated, since that is very likely to cause incomplete or partial
> +updates and therefore data loss.

Taking into consideration that people who are experiencing such corruption will likely include the name of the syncing service in their search query, would it make sense to mention some well-known services here in order to make it more likely that people will actually find this entry? Something like this, perhaps:

    It is important not to use a cloud syncing service (such as DropBox,
    FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
    repository...
Show 10 quoted lines
> +Therefore, it's better to push your work to either the other system or a central
> +server using the normal push and pull mechanism.  However, this doesn't always
> +preserve important data, like stashes, so some people prefer to share a working
> +tree across systems.
> ++
> +If you do this, the recommended approach is to use `rsync -a --delete-after`
> +(ideally with an encrypted connection such as with `ssh`) on the root of
> +repository.  You should ensure several things when you do this:
> ++
> +* There are no additional worktrees enabled for your repository.

I don't fully understand this restriction. Can you explain it (at least here in the email discussion)?

> +* You are not using a separate Git directory outside of your repository root.
Same question about this restriction.
Show 10 quoted lines
> +* You are comfortable with the destination directory being an exact copy of the
> +       source directory, _deleting any data that is already there_.
> +* The repository is in a quiescent state for the duration of the transfer (that
> +       is, no operations of any sort are taking place on it, including background
> +       operations like `git gc` and operations invoked by your editor).
> ++
> +Be aware that even with these recommendations, syncing in this way has some risk
> +since it bypasses Git's normal integrity checking for repositories, so having
> +backups is advised.  You may also with to do a `git fsck` to verify the
> +integrity of your data on the destination system after syncing.
s/with/wish/

In fact, as with "usually" above, "wish" may be too weak. Perhaps say instead that it is "recommended" that you use `git fsck` to verify the integrity.

Junio C Hamano· Nov 8, 2021, 22:09 UTC · re: Eric Sunshine · lore

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

Eric Sunshine <sunshine@sunshineco.com> writes:
Show 9 quoted lines
> Taking into consideration that people who are experiencing such
> corruption will likely include the name of the syncing service in
> their search query, would it make sense to mention some well-known
> services here in order to make it more likely that people will
> actually find this entry? Something like this, perhaps:
>
>     It is important not to use a cloud syncing service (such as DropBox,
>     FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
>     repository...

I do agree that in a repository being actively modified, any backup/sync solution that works per-file fashion would not work well. But is "cloud" a good word to characterise and group these per-file backup/sync solution?

Doesn't rsync work the same per-file fashion, and the only reason why it is a better fit is because it is not continuous, not attempting to "sync" while the repository is in use, until the user explicitly says "OK, I am ready to go home, so let's stop working here and send everything over to home with rsync"?

Show 8 quoted lines
>> +* There are no additional worktrees enabled for your repository.
>
> I don't fully understand this restriction. Can you explain it (at
> least here in the email discussion)?
>
>> +* You are not using a separate Git directory outside of your repository root.
>
> Same question about this restriction.

As long as you know what you are doing and catch everything in quiescent state, you should be fine, I would think.

Junio C Hamano· Nov 9, 2021, 00:10 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

Junio C Hamano <gitster@pobox.com> writes:
Show 5 quoted lines
> Doesn't rsync work the same per-file fashion, and the only reason
> why it is a better fit is because it is not continuous, not
> attempting to "sync" while the repository is in use, until the user
> explicitly says "OK, I am ready to go home, so let's stop working
> here and send everything over to home with rsync"?

OK, so not "per-file" but "continuous" is the root problem, and "cloud" would be a good word because all the popular ones share that "continuous" trait.

This part of the proposed patch text may need rethinking a bit.
> +or added files, broken refs, and a wide variety of other corruption.  These
> +services tend to sync file by file and don't understand the structure of a Git
> +repository.  This is especially bad if they sync the repository in the middle of

That is, "file by file" is not a problem per-se, "don't understand the structure" is a huge problem, and "continuous" may contribute to the problem further.

I wonder if you let the "cloud" services to continuously sync your repository, then go quiescent for a while and then start touching the destination, it would be sufficient, though. The refs with funny "2" suffixes and the like are the symptom of two sides simultanously mucking with the same file (e.g ".git/refs/main") and the "cloud sync" could not decide what the right final outcome is, right?

I also wonder if we add a way to transfer reflog entries, that imply the object reachability, say "git push --with-reflog", over the wire, it would be sufficient to do everything with Git.

Before you go home, you'd do
    git stash save --untracked && git stash apply
    git push --mirror --with-reflog --with-stash

to save away modified and untracked files to a stash entry [*], and push all the refs with their reflog entries (including refs/stash which normally gets refused because it has only two levels).

    Side note. If there were a variant of "git stash save" that only
    saves away without modifying the working tree and the index, I'd
    use that single command instead of "save and immediately restore
    by applying" kludge.

Then at the destination, you'd figure out what the current branch was (the stash message should record tha name of the branch), check that branch out, and running "git stash pop" will give you pretty much the same environment.

brian m. carlson· Nov 14, 2021, 23:40 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

On 2021-11-09 at 00:10:05, Junio C Hamano wrote:
Show 21 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Doesn't rsync work the same per-file fashion, and the only reason
> > why it is a better fit is because it is not continuous, not
> > attempting to "sync" while the repository is in use, until the user
> > explicitly says "OK, I am ready to go home, so let's stop working
> > here and send everything over to home with rsync"?
> 
> OK, so not "per-file" but "continuous" is the root problem, and
> "cloud" would be a good word because all the popular ones share that
> "continuous" trait.
> 
> This part of the proposed patch text may need rethinking a bit.
> 
> > +or added files, broken refs, and a wide variety of other corruption.  These
> > +services tend to sync file by file and don't understand the structure of a Git
> > +repository.  This is especially bad if they sync the repository in the middle of
> 
> That is, "file by file" is not a problem per-se, "don't understand
> the structure" is a huge problem, and "continuous" may contribute to
> the problem further.

"File by file" is not a problem per se, but it is a problem if you don't snapshot the repository or it's in use. For example, if you used an LVM snapshot on an in-use repository and then synced that, we'd guarantee that the repository was fully consistent. Practically nobody does that, though.

But to clarify: the goal of this type of software is to sync single, independent documents. As such, these tools don't necessarily consider that it's important to sync other files in the same directory at the same time. You can end up with cases where some parts of your repository get synced, then some other files in other directories, then other parts of the repository, resulting in different states over different times. There's just no guarantee about how they behave with all of the files in a given repository because they aren't considered a relevant unit. That's why I mention that they sync file by file. If they did syncing directory by directory, we'd likely have a lot fewer problems.

Show 7 quoted lines
> I wonder if you let the "cloud" services to continuously sync your
> repository, then go quiescent for a while and then start touching
> the destination, it would be sufficient, though.  The refs with
> funny "2" suffixes and the like are the symptom of two sides
> simultanously mucking with the same file (e.g ".git/refs/main") and
> the "cloud sync" could not decide what the right final outcome is,
> right?

I think this is very risky. Yes, usually the duplicate files are caused by competing changes on both sides, but having seen users with this problem before, I'm not sure if we can safely guarantee this is the only case.

> I also wonder if we add a way to transfer reflog entries, that imply
> the object reachability, say "git push --with-reflog", over the
> wire, it would be sufficient to do everything with Git.
Yes, please.  I'd love to see this.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
brian m. carlson· Nov 7, 2021, 22:55 UTC · re: brian m. carlson · lore

[PATCH v2 2/3] gitfaq: give advice on using eol attribute in gitattributes

In the FAQ, we tell people how to use the text attribute, but we fail to explain what to do with the eol attribute. As we ourselves have noticed, most shell implementations do not care for carriage returns, and as such, people will practically always want them to use LF endings. Similar things can be said for batch files on Windows, except with CRLF endings.

Since these are common things to have in a repository, let's help users make a good decision by recommending that they use the gitattributes file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which originally referred to "the following entry", even though a new entry has been inserted in between. The cross-reference notation should prevent this from occurring and provide a link in formats, such as HTML, which support that.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
Show changes to Documentation/gitfaq.txt +16 −4
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 5c21951f7b..ae1b526565 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -406,8 +406,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::
 +
 You can store the files in the repository with Unix line endings and convert
 them automatically to your platform's line endings.  To do that, set the
-configuration option `core.eol` to `native` and see the following entry for
-information about how to configure files as text or binary.
+configuration option `core.eol` to `native` and see
+<<recommended-storage-settings,the question on recommended storage settings>>
+for information about how to configure files as text or binary.
 +
 You can also control this behavior with the `core.whitespace` setting if you
 don't wish to remove the carriage returns from your line endings.
@@ -469,14 +470,25 @@ references, URLs, and hashes stored in the repository.
 +
 We also recommend setting a linkgit:gitattributes[5] file to explicitly mark
 which files are text and which are binary.  If you want Git to guess, you can
-set the attribute `text=auto`.  For example, the following might be appropriate
-in some projects:
+set the attribute `text=auto`.
++
+With text files, Git will generally ensure that LF endings are used in the
+repository, and will honor `core.autocrlf` and `core.eol` to decide what options
+to use when checking files out.  You can also override this by specifying a
+particular line ending such as `eol=lf` or `eol=crlf` if those files must always
+have that ending in the working tree (e.g., for functionality reasons).
++
+For example, the following might be appropriate in some projects:
 +
 ----
 # By default, guess.
 *	text=auto
 # Mark all C files as text.
 *.c	text
+# Ensure all shell files end up with LF endings and all batch files end up
+# with CRLF endings in the working tree and both end up with LF in the repo.
+*.sh text eol=lf
+*.bat text eol=crlf
 # Mark all JPEG files as binary.
 *.jpg	binary
 ----
Eric Sunshine· Nov 7, 2021, 23:48 UTC · re: brian m. carlson · lore

Re: [PATCH v2 2/3] gitfaq: give advice on using eol attribute in gitattributes

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 35 quoted lines
> In the FAQ, we tell people how to use the text attribute, but we fail to
> explain what to do with the eol attribute.  As we ourselves have
> noticed, most shell implementations do not care for carriage returns,
> and as such, people will practically always want them to use LF endings.
> Similar things can be said for batch files on Windows, except with CRLF
> endings.
>
> Since these are common things to have in a repository, let's help users
> make a good decision by recommending that they use the gitattributes
> file to correctly check out the endings.
> [...]
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -469,14 +470,25 @@ references, URLs, and hashes stored in the repository.
> +With text files, Git will generally ensure that LF endings are used in the
> +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> +to use when checking files out.  You can also override this by specifying a
> +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> +have that ending in the working tree (e.g., for functionality reasons).
> ++
> +For example, the following might be appropriate in some projects:
>  +
>  ----
>  # By default, guess.
>  *      text=auto
>  # Mark all C files as text.
>  *.c    text
> +# Ensure all shell files end up with LF endings and all batch files end up
> +# with CRLF endings in the working tree and both end up with LF in the repo.
> +*.sh text eol=lf
> +*.bat text eol=crlf
>  # Mark all JPEG files as binary.
>  *.jpg  binary
>  ----

I like the concrete explanation in the commit message of why `.sh` and `.bat` files need to be configured specially, and was expecting that quite useful information to be repeated here in the body. That does seem exactly like what someone would come to the FAQ searching for, and even though the above content hints at it, the hint may be just vague enough to be overlooked. Consequently, I'm wondering if it ought to be spelled out more explicitly here. In particular, if we give examples of actual error messages people might encounter when attempting to run a CRLF shell script or an LF batch file, that would really give a FAQ searcher/reader something to latch onto when trying to solve a problem. This could be done as a lead-in paragraph immediately before the "For example, the following...".

Also, following v1 review, I think you had intended[1]: s/end up with/have/
[1]: https://lore.kernel.org/git/YW9wgbN%2Fb8NkVp4z@camp.crustytoothpaste.net/
brian m. carlson· Nov 8, 2021, 02:09 UTC · re: Eric Sunshine · lore

Re: [PATCH v2 2/3] gitfaq: give advice on using eol attribute in gitattributes

On 2021-11-07 at 23:48:00, Eric Sunshine wrote:
Show 12 quoted lines
> I like the concrete explanation in the commit message of why `.sh` and
> `.bat` files need to be configured specially, and was expecting that
> quite useful information to be repeated here in the body. That does
> seem exactly like what someone would come to the FAQ searching for,
> and even though the above content hints at it, the hint may be just
> vague enough to be overlooked. Consequently, I'm wondering if it ought
> to be spelled out more explicitly here. In particular, if we give
> examples of actual error messages people might encounter when
> attempting to run a CRLF shell script or an LF batch file, that would
> really give a FAQ searcher/reader something to latch onto when trying
> to solve a problem. This could be done as a lead-in paragraph
> immediately before the "For example, the following...".

I think that we could definitely add some text here to explain why one would want to use this, providing an example.

> Also, following v1 review, I think you had intended[1]: s/end up with/have/
I will fix that in v3.  Thanks for the reminder.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Eric Sunshine· Nov 8, 2021, 00:16 UTC · re: brian m. carlson · lore

Re: [PATCH v2 0/3] Additional FAQ entries

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 12 quoted lines
> This series introduces some additional Git FAQ entries on various
> topics.  They are all things I've seen in my professional life or on
> Stack Overflow, so I've written documentation.
>
> Changes from v1:
> * Drop the monorepo patch for now; I want to revise it further.
> * Reorder the working tree patch to place more warnings up front.
> * Mention core.gitproxy and socat.
> * Rephrase text in the EOL entry to read correctly and be easier to
>   understand.
> * Improve the commit message for the working tree FAQ entry to make it
>   clearer that users wish to transfer uncommitted changes.

Thanks for re-rolling. FAQ entries are, of course, bikeshedding fodder, so I left a number of bikeshedding comments and pointed out one minor typo. Whether any of the comments warrant a re-roll or are even worth discussing is a matter of opinion, and they could certainly all be addressed later (by someone) atop the current series if desired; I don't think any of the comments were blockers.

Johannes Schindelin· Jan 4, 2022, 13:53 UTC · re: Eric Sunshine · lore

Re: [PATCH v2 0/3] Additional FAQ entries

Hi Eric,
On Sun, 7 Nov 2021, Eric Sunshine wrote:
Show 17 quoted lines
> On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > This series introduces some additional Git FAQ entries on various
> > topics.  They are all things I've seen in my professional life or on
> > Stack Overflow, so I've written documentation.
> >
> > Changes from v1:
> > * Drop the monorepo patch for now; I want to revise it further.
> > * Reorder the working tree patch to place more warnings up front.
> > * Mention core.gitproxy and socat.
> > * Rephrase text in the EOL entry to read correctly and be easier to
> >   understand.
> > * Improve the commit message for the working tree FAQ entry to make it
> >   clearer that users wish to transfer uncommitted changes.
>
> Thanks for re-rolling. FAQ entries are, of course, bikeshedding
> fodder,

That is one possible perspective. Another possible perspective is that FAQ entries are the first thing users in distress might turn to, so the more helpful they are, the better. Maybe it would make sense to focus on that a bit more. As in your suggestion to name cloud sync providers by name, to make the FAQ entry eminently more "findable". To me, that did not at all feel like pointless bike-shedding, but more like a really valuable improvement that users will appreciate (even if they will be oblivious of the fact that they appreciate it, of course).

Ciao, Dscho

Johannes Schindelin· Jan 4, 2022, 13:54 UTC · re: brian m. carlson · lore

Re: [PATCH v2 0/3] Additional FAQ entries

Hi brian,
On Sun, 7 Nov 2021, brian m. carlson wrote:
Show 8 quoted lines
> This series introduces some additional Git FAQ entries on various
> topics.  They are all things I've seen in my professional life or on
> Stack Overflow, so I've written documentation.
>
> I've opted not to include backing up repositories in the syncing patch
> because I think they're separate topics.  We could well end up with an
> additional FAQ entry on that topic, which is left as an exercise for the
> reader.
Makes sense.

You hinted in the thread that you were planning on submitting a v3. I hope that my feedback regarding `core.gitproxy` is still in time for that.

Thank you, Dscho

brian m. carlson· Jan 6, 2022, 01:58 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 0/3] Additional FAQ entries

On 2022-01-04 at 13:54:52, Johannes Schindelin wrote:
Show 17 quoted lines
> Hi brian,
> 
> On Sun, 7 Nov 2021, brian m. carlson wrote:
> 
> > This series introduces some additional Git FAQ entries on various
> > topics.  They are all things I've seen in my professional life or on
> > Stack Overflow, so I've written documentation.
> >
> > I've opted not to include backing up repositories in the syncing patch
> > because I think they're separate topics.  We could well end up with an
> > additional FAQ entry on that topic, which is left as an exercise for the
> > reader.
> 
> Makes sense.
> 
> You hinted in the thread that you were planning on submitting a v3. I hope
> that my feedback regarding `core.gitproxy` is still in time for that.

I can definitely include that in v3. I'm hoping I can get to it later this week or during the weekend before my schedule becomes busy again.

I appreciate the thoughtful comments.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Eric Sunshine· Nov 9, 2021, 00:02 UTC · lore

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

On Mon, Nov 8, 2021 at 5:34 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 17 quoted lines
> On 2021-11-08 at 00:12:14, Eric Sunshine wrote:
> > Taking into consideration that people who are experiencing such
> > corruption will likely include the name of the syncing service in
> > their search query, would it make sense to mention some well-known
> > services here in order to make it more likely that people will
> > actually find this entry? Something like this, perhaps:
> >
> >     It is important not to use a cloud syncing service (such as DropBox,
> >     FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
> >     repository...
>
> There are a lot of these services.  My preference is to not name
> specific ones here, just like we don't name specific ones when we state
> that you shouldn't use an antivirus or firewall other than the Windows
> default, mostly because I'm not interested in angering corporate
> lawyers.  My advice on this topic is always general: XYZ is a cloud
> syncing service, and you should not use any cloud syncing services for Git.

My "would it make sense" question arose from taking into consideration how people are likely to use a search engine for a particular problem:

    BuzzingBee syncing corrupted git repository

Without naming specific services or tools, it seems much less likely that people will find this FAQ entry, thus will end up posting to StackOverflow (or wherever) anyhow -- despite your intention and effort behind the FAQ in the first place. I'm thinking about discoverability -- which is the same reason I suggested in my review of the other patch that it might be a good idea to add actual error messages a person might encounter with a CRLF shell script or an LF batch file.

> Additionally, the popular options differ by region.  For example, there
> are some services which are probably popular in China which are not
> popular elsewhere, and I don't think it's a good idea to try to guess
> which ones happen to be most popular around the world.

The other way to look at it is that listing many popular services (wherever they happen to be) makes it more likely that search engines will lead them to this FAQ entry and alleviate the need to post a question somewhere.

Anyhow, it was just a thought. I think I've said all I have to say on this subject. As I mentioned in response to the cover letter, all of my comments were of the bikeshedding variety, and I didn't see any show-stoppers in the series as posted.

Show 8 quoted lines
> > > +* There are no additional worktrees enabled for your repository.
> >
> > I don't fully understand this restriction. Can you explain it (at
> > least here in the email discussion)?
>
> If you sync the main repository and working tree, but not the worktrees
> themselves, then you end up with incorrect data in the .git directory,
> which contains information about the worktree.

I still don't follow. What incorrect data will be in the .git directory? Do you mean the absolute path pointing from the .git directory to the worktree?

> More importantly, it can
> contain absolute path information, which would be undesirable even if
> you did sync the worktrees, say, if you used two different usernames
> (and hence two different home directories) on the two systems.

Okay, I figured that that was probably one of your concerns, but it was difficult to tell from the raw "no additional worktrees". On the other hand, I can easily see people syncing between machines in which they have the same username and same directory structure, so this bullet point seems overly restrictive.

Show 7 quoted lines
> > > +* You are not using a separate Git directory outside of your repository root.
> >
> > Same question about this restriction.
>
> Again, if you sync the root of the working tree but don't sync the
> separate Git directory, you won't have copied the index or any of the
> other data.

Okay, again, this is indeed what I thought you might have in mind, but it was difficult to be sure. And, again, this seems overly restrictive since there may be plenty of scenarios in which this works well.

I suppose both of these points would feel more reasonable if they didn't sound like hard requirements, perhaps by explaining that the simple case of no worktrees and no separate repository has the least restrictions and is easiest to get up and running; more complex cases can work too with the caveat that worktree and separate repository directories ought to be synced too, and that pathnames need to be identical on all machines.

← back to recent threads