threads / patch / 56743

patch, 4 partsAdditional FAQ entries

Subject: [PATCH 0/4] Additional FAQ entries

## tl;dr

30 messages between Oct 20, 2021 and Oct 25, 2021. Diffs are folded; open one to read it.

replies: 29people: 7as markdown or json

brian m. carlson· Oct 20, 2021, 01:06 UTC · lore

This series introduces some additional Git FAQ entries on various topics. They are all things I've seen in my professional life or on Stack Overflow, and I tire of repeating myself, so I've written documentation.

The first covers monorepos. It tries to strike a neutral tone and indicate that we're improving support for them, but it does note that many large users still run into scale problems, especially at a point where it's inconvenient to change[0]. I do frequently see these problems on some scale, so my hope is that we can make users consider this case carefully. I have mentioned only scalability features which are released and reasonably mature, but if I've missed any that I should have mentioned, please let me know, since I don't always follow the list as closely as I should.

The second covers proxies, in the generic sense of that word. It instructs users how to use an HTTP proxy intentionally, and it also notes that any proxy which buffers or tampers with the data will break Git. I frequently see users with TLS middleboxes or antivirus software other than the Windows default with breakage on Stack Overflow[1], and we see them occasionally on the list as well. Because these tools are known to cause both functionality and security problems, both for Git and in general[2][3][4], we explicitly recommend against them here. I am happy to provide additional citations to Project Zero, various bug reports, security experts, and the cryptographic literature if folks have doubts about whether this is the right approach.

The third should be relatively uncontroversial. We discuss using eol attributes in gitattributes, which we had neglected to cover before, and illustrate their utility. In addition, we fix a reference.

The fourth and final patch covers syncing working trees. An earlier version of this patch has come up on the list before and I had never sent a rerolled version. I realize that we don't love that people want to do this, but users are currently using cloud syncing services and experiencing data loss, so I figured we'd better tell them how to do it as safely as they can so they stop losing data.

I realize that folks are busy with the Contributor Summit and other things, and so of course reviews may be delayed. I've tried to pick people who I think will have relevant interest in each of these areas, but of course feedback from anyone is welcome.

[0] For instance, the Windows repo.
[1] In addition to users for whom SSH "magically" works when HTTPS does
    not, many of which are likely the same issue.
[2] https://robert.ocallahan.org/2017/01/disable-your-antivirus-software-except.html
[3] https://twitter.com/parityzero/status/826489070310207489
[4] https://groups.google.com/g/mozilla.dev.platform/c/Bh8U0DLHrCc
brian m. carlson (4):
  gitfaq: add advice on monorepos
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees
 Documentation/gitfaq.txt | 107 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 101 insertions(+), 6 deletions(-)
brian m. carlson· Oct 20, 2021, 01:06 UTC · re: brian m. carlson · lore

[PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

In the FAQ, we tell people how to use the text attribute, but we fail to explain what to do with the eol attribute. As we ourselves have noticed, most shell implementations do not care for carriage returns, and as such, people will practically always want them to use LF endings. Similar things can be said for batch files on Windows, except with CRLF endings.

Since these are common things to have in a repository, let's help users make a good decision by recommending that they use the gitattributes file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which originally referred to "the following entry", even though a new entry has been inserted in between. The cross-reference notation should prevent this from occurring and provide a link in formats, such as HTML, which support that.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
Show changes to Documentation/gitfaq.txt +16 −4
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index abc0f62e6c..85ac99c7b2 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -401,8 +401,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::
 +
 You can store the files in the repository with Unix line endings and convert
 them automatically to your platform's line endings.  To do that, set the
-configuration option `core.eol` to `native` and see the following entry for
-information about how to configure files as text or binary.
+configuration option `core.eol` to `native` and see
+<<recommended-storage-settings,the question on recommended storage settings>>
+for information about how to configure files as text or binary.
 +
 You can also control this behavior with the `core.whitespace` setting if you
 don't wish to remove the carriage returns from your line endings.
@@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
 +
 We also recommend setting a linkgit:gitattributes[5] file to explicitly mark
 which files are text and which are binary.  If you want Git to guess, you can
-set the attribute `text=auto`.  For example, the following might be appropriate
-in some projects:
+set the attribute `text=auto`.
++
+With text files, Git will generally the repository contains LF endings in the
+repository, and will honor `core.autocrlf` and `core.eol` to decide what options
+to use when checking files out.  You can also override this by specifying a
+particular line ending such as `eol=lf` or `eol=crlf` if those files must always
+have that ending (e.g., for functionality reasons).
++
+For example, the following might be appropriate in some projects:
 +
 ----
 # By default, guess.
 *	text=auto
 # Mark all C files as text.
 *.c	text
+# Ensure all shell files end up with LF endings and all batch files end up
+# with CRLF endings in the working tree and both end up with LF in the repo.
+*.sh text eol=lf
+*.bat text eol=crlf
 # Mark all JPEG files as binary.
 *.jpg	binary
 ----
Eric Sunshine· Oct 20, 2021, 01:21 UTC · re: brian m. carlson · lore

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 26 quoted lines
> In the FAQ, we tell people how to use the text attribute, but we fail to
> explain what to do with the eol attribute.  As we ourselves have
> noticed, most shell implementations do not care for carriage returns,
> and as such, people will practically always want them to use LF endings.
> Similar things can be said for batch files on Windows, except with CRLF
> endings.
>
> Since these are common things to have in a repository, let's help users
> make a good decision by recommending that they use the gitattributes
> file to correctly check out the endings.
>
> In addition, let's correct the cross-reference to this question, which
> originally referred to "the following entry", even though a new entry
> has been inserted in between.  The cross-reference notation should
> prevent this from occurring and provide a link in formats, such as HTML,
> which support that.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> +With text files, Git will generally the repository contains LF endings in the
> +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> +to use when checking files out.  You can also override this by specifying a
> +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> +have that ending (e.g., for functionality reasons).
The first sentence in the paragraph is unparseable.
> +# Ensure all shell files end up with LF endings and all batch files end up
> +# with CRLF endings in the working tree and both end up with LF in the repo.
> +*.sh text eol=lf
> +*.bat text eol=crlf
Maybe: s/end up with/have/g
brian m. carlson· Oct 20, 2021, 01:27 UTC · re: Eric Sunshine · lore

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
Show 30 quoted lines
> On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > In the FAQ, we tell people how to use the text attribute, but we fail to
> > explain what to do with the eol attribute.  As we ourselves have
> > noticed, most shell implementations do not care for carriage returns,
> > and as such, people will practically always want them to use LF endings.
> > Similar things can be said for batch files on Windows, except with CRLF
> > endings.
> >
> > Since these are common things to have in a repository, let's help users
> > make a good decision by recommending that they use the gitattributes
> > file to correctly check out the endings.
> >
> > In addition, let's correct the cross-reference to this question, which
> > originally referred to "the following entry", even though a new entry
> > has been inserted in between.  The cross-reference notation should
> > prevent this from occurring and provide a link in formats, such as HTML,
> > which support that.
> >
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > +With text files, Git will generally the repository contains LF endings in the
> > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > +to use when checking files out.  You can also override this by specifying a
> > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > +have that ending (e.g., for functionality reasons).
> 
> The first sentence in the paragraph is unparseable.
Yes, I think perhaps I omitted the word "ensure".

And I should reflect that they should have that ending in the working tree, which I neglected to mention.

Show 6 quoted lines
> > +# Ensure all shell files end up with LF endings and all batch files end up
> > +# with CRLF endings in the working tree and both end up with LF in the repo.
> > +*.sh text eol=lf
> > +*.bat text eol=crlf
> 
> Maybe: s/end up with/have/g
Great.  That does read more clearly and succinctly.
I'll include those fixes in a reroll.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Johannes Schindelin· Oct 20, 2021, 12:02 UTC · re: brian m. carlson · lore

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

Hi brian,
On Wed, 20 Oct 2021, brian m. carlson wrote:
Show 18 quoted lines
> On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> > On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> > <sandals@crustytoothpaste.net> wrote:
> >
> > > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > > +With text files, Git will generally the repository contains LF endings in the
> > > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > > +to use when checking files out.  You can also override this by specifying a
> > > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > > +have that ending (e.g., for functionality reasons).
> >
> > The first sentence in the paragraph is unparseable.
>
> Yes, I think perhaps I omitted the word "ensure".
>
> And I should reflect that they should have that ending in the working
> tree, which I neglected to mention.

Please note that Git for Windows defaults to `core.autoCRLF=true`, therefore this sentence is not completely correct. Maybe something as short as "(except in Git for Windows, which defaults to CRLF endings)" would suffice?

Ciao, Dscho

brian m. carlson· Oct 20, 2021, 22:25 UTC · re: Johannes Schindelin · lore

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

On 2021-10-20 at 12:02:02, Johannes Schindelin wrote:
Show 27 quoted lines
> Hi brian,
> 
> On Wed, 20 Oct 2021, brian m. carlson wrote:
> 
> > On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> > > On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> > > <sandals@crustytoothpaste.net> wrote:
> > >
> > > > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > > > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > > > +With text files, Git will generally the repository contains LF endings in the
> > > > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > > > +to use when checking files out.  You can also override this by specifying a
> > > > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > > > +have that ending (e.g., for functionality reasons).
> > >
> > > The first sentence in the paragraph is unparseable.
> >
> > Yes, I think perhaps I omitted the word "ensure".
> >
> > And I should reflect that they should have that ending in the working
> > tree, which I neglected to mention.
> 
> Please note that Git for Windows defaults to `core.autoCRLF=true`,
> therefore this sentence is not completely correct. Maybe something as
> short as "(except in Git for Windows, which defaults to CRLF endings)"
> would suffice?

What I meant by that sentence was that I should add, "in the working tree" to the sentence ending "if those files must always have that ending". I believe that is still the case for Git for Windows, since otherwise our shell files in the repository would be broken there, and I'm fairly confident they are not.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Johannes Schindelin· Oct 21, 2021, 12:02 UTC · re: brian m. carlson · lore

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

Hi brian,
On Wed, 20 Oct 2021, brian m. carlson wrote:
Show 34 quoted lines
> On 2021-10-20 at 12:02:02, Johannes Schindelin wrote:
> > Hi brian,
> >
> > On Wed, 20 Oct 2021, brian m. carlson wrote:
> >
> > > On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> > > > On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> > > > <sandals@crustytoothpaste.net> wrote:
> > > >
> > > > > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > > > > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > > > > +With text files, Git will generally the repository contains LF endings in the
> > > > > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > > > > +to use when checking files out.  You can also override this by specifying a
> > > > > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > > > > +have that ending (e.g., for functionality reasons).
> > > >
> > > > The first sentence in the paragraph is unparseable.
> > >
> > > Yes, I think perhaps I omitted the word "ensure".
> > >
> > > And I should reflect that they should have that ending in the working
> > > tree, which I neglected to mention.
> >
> > Please note that Git for Windows defaults to `core.autoCRLF=true`,
> > therefore this sentence is not completely correct. Maybe something as
> > short as "(except in Git for Windows, which defaults to CRLF endings)"
> > would suffice?
>
> What I meant by that sentence was that I should add, "in the working
> tree" to the sentence ending "if those files must always have that
> ending".  I believe that is still the case for Git for Windows, since
> otherwise our shell files in the repository would be broken there, and
> I'm fairly confident they are not.

I swear I read this sentence multiple times yesterday, and still managed to miss that you were talking about LF endings _in the repository_.

FWIW I think that `core.autoCRLF` defaults to `off` everywhere but on Windows, so _technically_ Git would not ensure LF endings e.g. on Linux. In practice, it is highly unlikely that a Linux user would generate text files with CR/LF endings, so it would not matter in practice.

I feel a bit bad for arguing subtleties that realistically won't matter, so please feel free to leave the sentence as you have intended it.

Ciao, Dscho

brian m. carlson· Oct 20, 2021, 01:06 UTC · re: brian m. carlson · lore

[PATCH 4/4] doc: add a FAQ entry about syncing working trees

Users very commonly want to sync their working tree across machines, often to carry across in-progress work or stashes. Despite this not being a recommended approach, users want to do it and are not dissuaded by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing service, which is a bad idea. Users have reported problems where they end up with duplicate files that won't go away (with names like "file.c 2"), broken references, oddly named references that have date stamps appended to them, missing objects, and general corruption and data loss. That's because almost all of these tools sync file by file, which is a great technique if your project is a single word processing document or spreadsheet, but is utterly abysmal for Git repositories because they don't necessarily snapshot the entire repository correctly. They also tend to sync the files immediately instead of when the repository is quiescent, so writing multiple files, as occurs during a commit or a gc, can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that the repository is quiescent, so let's suggest that and dissuade people from using cloud syncing tools. Let's tell people about common things they should be aware of before doing this and that this is still potentially risky. Additionally, let's tell people that Git's security model does not permit sharing working trees across users in case they planned to do that. While we'd still prefer users didn't try to do this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 43 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)
Show changes to Documentation/gitfaq.txt +41 −1
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 85ac99c7b2..4a8a46f980 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::
@@ -185,6 +185,45 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  Doing so
+	can cause `git status` to need to re-read every file in the working tree.
+	Additionally, Git's security model does not permit sharing a working tree
+	across untrusted users, so it is only safe to sync a working tree if it will
+	only be used by a single user across all machines.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+The recommended approach is to use `rsync -a --delete-after` (ideally with an
+encrypted connection such as with `ssh`) on the root of repository.  You should
+ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc`).
++
+Be aware that even with these recommendations, syncing in this way is
+potentially risky since it bypasses Git's normal integrity checking for
+repositories, so having backups is advised.
+
 Common Issues
 -------------
Bagas Sanjaya· Oct 20, 2021, 04:58 UTC · re: brian m. carlson · lore

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

On 20/10/21 08.06, brian m. carlson wrote:
> +Be aware that even with these recommendations, syncing in this way is
> +potentially risky since it bypasses Git's normal integrity checking for
> +repositories, so having backups is advised.
> +

This raises question: how users can backup their repos? The answer is same as this FAQ for syncing working trees: have your repo to be pushed to the spare server at disposal, or use rsync (as long as the preconditions are met).

-- 
An old man doll... just what I always wanted! - Clara
Philip Oakley· Oct 20, 2021, 14:05 UTC · re: Bagas Sanjaya · lore

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

On 20/10/2021 05:58, Bagas Sanjaya wrote:
Show 11 quoted lines
> On 20/10/21 08.06, brian m. carlson wrote:
>> +Be aware that even with these recommendations, syncing in this way is
>> +potentially risky since it bypasses Git's normal integrity checking for
>> +repositories, so having backups is advised.
>> +
>
> This raises question: how users can backup their repos? The answer is
> same as this FAQ for syncing working trees: have your repo to be
> pushed to the spare server at disposal, or use rsync (as long as the
> preconditions are met).
>

Perhaps, for backing up a repo, users might be pointed toward `git bundle` (with caveats about `--all` being over complete..)

-- 
Philip
Ævar Arnfjörð Bjarmason· Oct 20, 2021, 23:35 UTC · re: brian m. carlson · lore

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

On Wed, Oct 20 2021, brian m. carlson wrote:
> +The recommended approach is to use `rsync -a --delete-after` (ideally with an
> +encrypted connection such as with `ssh`) on the root of repository.  You should
> +ensure several things when you do this:

What's the reason to recommend --delete-after in particular? I realize that e.g. in the .git directory not using *A* delete option *will* cause corruption, e.g. if you can leave behind stale loose refs with an up-to-date pack-refs file.

But isn't that equally covered by --delete and --delete-before? I'm not very well worsed in rsync, but aren't the two equivalent as far as the end-state goes?

If the intention with --delete-after over --delete or --delete-before is to somehow make the repository useful during the transfer, doesn't that go against the later advice of:

> +* The repository is in a quiescent state for the duration of the transfer (that
> +	is, no operations of any sort are taking place on it, including background
> +	operations like `git gc`).
Also for this:
> +Be aware that even with these recommendations, syncing in this way is
> +potentially risky since it bypasses Git's normal integrity checking for
> +repositories, so having backups is advised.

Perhaps we should recommend running a "git gc" or other integrity check after (or "git fsck"), although those don't cover some cases, e.g. the pack-refs v.s. loose refs problem in the case of a missing --delete-whatever.

brian m. carlson· Oct 21, 2021, 00:03 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

On 2021-10-20 at 23:35:43, Ævar Arnfjörð Bjarmason wrote:
Show 15 quoted lines
> 
> On Wed, Oct 20 2021, brian m. carlson wrote:
> 
> > +The recommended approach is to use `rsync -a --delete-after` (ideally with an
> > +encrypted connection such as with `ssh`) on the root of repository.  You should
> > +ensure several things when you do this:
> 
> What's the reason to recommend --delete-after in particular? I realize
> that e.g. in the .git directory not using *A* delete option *will* cause
> corruption, e.g. if you can leave behind stale loose refs with an
> up-to-date pack-refs file.
> 
> But isn't that equally covered by --delete and --delete-before? I'm not
> very well worsed in rsync, but aren't the two equivalent as far as the
> end-state goes?

Yes. The goal is that if something goes wrong, you have all the objects you did before, even if you have some potentially invalid refs. The goal is to make it a little less risky if you interrupt it with a Ctrl-C because you realize the destination contained data you wanted. I always prefer --delete-after for that reason, assuming the destination has sufficient disk space.

It shouldn't make a difference in a successful end state, however.
Show 9 quoted lines
> 
> > +Be aware that even with these recommendations, syncing in this way is
> > +potentially risky since it bypasses Git's normal integrity checking for
> > +repositories, so having backups is advised.
> 
> Perhaps we should recommend running a "git gc" or other integrity check
> after (or "git fsck"), although those don't cover some cases, e.g. the
> pack-refs v.s. loose refs problem in the case of a missing
> --delete-whatever.
I can recommend something like that.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Ævar Arnfjörð Bjarmason· Oct 21, 2021, 00:33 UTC · re: brian m. carlson · lore

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

On Thu, Oct 21 2021, brian m. carlson wrote:
Show 24 quoted lines
> [[PGP Signed Part:Undecided]]
> On 2021-10-20 at 23:35:43, Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Wed, Oct 20 2021, brian m. carlson wrote:
>> 
>> > +The recommended approach is to use `rsync -a --delete-after` (ideally with an
>> > +encrypted connection such as with `ssh`) on the root of repository.  You should
>> > +ensure several things when you do this:
>> 
>> What's the reason to recommend --delete-after in particular? I realize
>> that e.g. in the .git directory not using *A* delete option *will* cause
>> corruption, e.g. if you can leave behind stale loose refs with an
>> up-to-date pack-refs file.
>> 
>> But isn't that equally covered by --delete and --delete-before? I'm not
>> very well worsed in rsync, but aren't the two equivalent as far as the
>> end-state goes?
>
> Yes.  The goal is that if something goes wrong, you have all the objects
> you did before, even if you have some potentially invalid refs.  The
> goal is to make it a little less risky if you interrupt it with a Ctrl-C
> because you realize the destination contained data you wanted.  I always
> prefer --delete-after for that reason, assuming the destination has
> sufficient disk space.

Isn't it preferable to recommend --delete-before for that reason? I.e. --delete-after will produce subtle corruption of e.g. refs potentially pointing to the wrong thing.

But if you recommend --delete-before I think (but maybe I'm missing some cases) that it will be more likely to produce obvious corruption, e.g. git dying due to missing objects.

Anyway, I'm also happy to just leave this as-is, it just stood out to me as od..

Show 12 quoted lines
> It shouldn't make a difference in a successful end state, however.
>> 
>> > +Be aware that even with these recommendations, syncing in this way is
>> > +potentially risky since it bypasses Git's normal integrity checking for
>> > +repositories, so having backups is advised.
>> 
>> Perhaps we should recommend running a "git gc" or other integrity check
>> after (or "git fsck"), although those don't cover some cases, e.g. the
>> pack-refs v.s. loose refs problem in the case of a missing
>> --delete-whatever.
>
> I can recommend something like that.

...or just leave it as-is is also fine with me, whatever you think is best.

brian m. carlson· Oct 20, 2021, 01:06 UTC · re: brian m. carlson · lore

[PATCH 4/4] gitfaq: add entry about syncing working trees

Users very commonly want to sync their working tree across machines, often to carry across in-progress work or stashes. Despite this not being a recommended approach, users want to do it and are not dissuaded by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing service, which is a bad idea. Users have reported problems where they end up with duplicate files that won't go away (with names like "file.c 2"), broken references, oddly named references that have date stamps appended to them, missing objects, and general corruption and data loss. That's because almost all of these tools sync file by file, which is a great technique if your project is a single word processing document or spreadsheet, but is utterly abysmal for Git repositories because they don't necessarily snapshot the entire repository correctly. They also tend to sync the files immediately instead of when the repository is quiescent, so writing multiple files, as occurs during a commit or a gc, can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that the repository is quiescent, so let's suggest that and dissuade people from using cloud syncing tools. Let's tell people about common things they should be aware of before doing this and that this is still potentially risky. Additionally, let's tell people that Git's security model does not permit sharing working trees across users in case they planned to do that. While we'd still prefer users didn't try to do this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 43 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)
Show changes to Documentation/gitfaq.txt +41 −1
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 85ac99c7b2..4a8a46f980 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::
@@ -185,6 +185,45 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  Doing so
+	can cause `git status` to need to re-read every file in the working tree.
+	Additionally, Git's security model does not permit sharing a working tree
+	across untrusted users, so it is only safe to sync a working tree if it will
+	only be used by a single user across all machines.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+The recommended approach is to use `rsync -a --delete-after` (ideally with an
+encrypted connection such as with `ssh`) on the root of repository.  You should
+ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc`).
++
+Be aware that even with these recommendations, syncing in this way is
+potentially risky since it bypasses Git's normal integrity checking for
+repositories, so having backups is advised.
+
 Common Issues
 -------------
Eric Sunshine· Oct 20, 2021, 01:38 UTC · re: brian m. carlson · lore

Re: [PATCH 4/4] gitfaq: add entry about syncing working trees

On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

> gitfaq: add entry about syncing working trees

You sent two [4/4] patches. I'm guessing the one prefixed by "gitfaq:" is the correct one.

Show 39 quoted lines
> Users very commonly want to sync their working tree across machines,
> often to carry across in-progress work or stashes.  Despite this not
> being a recommended approach, users want to do it and are not dissuaded
> by suggestions not to, so let's recommend a sensible technique.
> [...]
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -185,6 +185,45 @@ Then, you can adjust your push URL to use `git@example_author` or
> +[[sync-working-tree]]
> +How do I sync a working tree across systems?::
> +       First, decide whether you want to do this at all.  Git usually works better
> +       when you push or pull your work using the typical `git push` and `git fetch`
> +       commands and isn't designed to share a working tree across systems.  Doing so
> +       can cause `git status` to need to re-read every file in the working tree.
> +       Additionally, Git's security model does not permit sharing a working tree
> +       across untrusted users, so it is only safe to sync a working tree if it will
> +       only be used by a single user across all machines.
> ++
> +Therefore, it's better to push your work to either the other system or a central
> +server using the normal push and pull mechanism.  However, this doesn't always
> +preserve important data, like stashes, so some people prefer to share a working
> +tree across systems.
> ++
> +It is important not to use a cloud syncing service to sync any portion of a Git
> +repository, since this can cause corruption, such as missing objects, changed
> +or added files, broken refs, and a wide variety of other corruption.  These
> +services tend to sync file by file and don't understand the structure of a Git
> +repository.  This is especially bad if they sync the repository in the middle of
> +it being updated, since that is very likely to cause incomplete or partial
> +updates and therefore data loss.
> ++
> +The recommended approach is to use `rsync -a --delete-after` (ideally with an
> +encrypted connection such as with `ssh`) on the root of repository.  You should
> +ensure several things when you do this:
> [...]
> +Be aware that even with these recommendations, syncing in this way is
> +potentially risky since it bypasses Git's normal integrity checking for
> +repositories, so having backups is advised.

Considering the potential damage which can result from this sort of synching, this entire section seems too gentle. My knee-jerk reaction is that it would be better to strongly dissuade upfront rather than saying that it's okay to do this if you really want to. As such, I'm wondering if organizing the section like this would be better:

(1) Make a strong statement against doing this: "<strong>Don't do it.</strong>"

(2) Explain why users shouldn't do it; in particular, the final paragraph above which talks about integrity checks and whatnot should be right up near the top along with discussion of corruption.

(3) Say that cloud-synching services must _not_ be used and explain why.

(4) Relent a tiny bit and explain that the only slightly acceptable mechanism is `rsync` when used in a very strict fashion (quiescent repository, etc.)

brian m. carlson· Oct 20, 2021, 21:36 UTC · re: Eric Sunshine · lore

Re: [PATCH 4/4] gitfaq: add entry about syncing working trees

On 2021-10-20 at 01:38:47, Eric Sunshine wrote:
Show 6 quoted lines
> On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > gitfaq: add entry about syncing working trees
> 
> You sent two [4/4] patches. I'm guessing the one prefixed by "gitfaq:"
> is the correct one.
Yes, I appear to have done a bad cherry-pick.  Will fix in v2.
Show 7 quoted lines
> Considering the potential damage which can result from this sort of
> synching, this entire section seems too gentle. My knee-jerk reaction
> is that it would be better to strongly dissuade upfront rather than
> saying that it's okay to do this if you really want to. As such, I'm
> wondering if organizing the section like this would be better:
> 
> (1) Make a strong statement against doing this: "<strong>Don't do it.</strong>"

I agree this is dangerous. The reason this is so painful is why I long ago stopped having a desktop: I needed to sync in-progress work frequently, and having multiple machines is too much of a hassle for that case. The laptop is more portable and can be used everywhere, even if less powerful.

However, some people do legitimately need to work on the same project across machines, and the current tooling for syncing stashes and other in-progress work is insufficient. Therefore, if we just tell people, "Don't do this," they're going to stop reading and ignore us, because we've neglected their needs and they have a job to do. That would be worse, because instead of using something like rsync, they might use a cloud syncing service, and then they'll be really in a bad place.

However, I'm happy to try the reorganization you proposed, even if I don't necessarily adopt the strength of the proposal, and see how it works.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Johannes Schindelin· Oct 20, 2021, 12:09 UTC · re: brian m. carlson · lore

Re: [PATCH 4/4] gitfaq: add entry about syncing working trees

Hi brian,
On Wed, 20 Oct 2021, brian m. carlson wrote:
> Users very commonly want to sync their working tree across machines,

I was confused at first, because I do sync my working trees frequently. I do this via `git push` and `git pull`, though.

Maybe clarify that you mean rsync, or sync services like DropBox and OneDrive? I see that you mention "cloud syncing service" below, but I believe that it might be better to lead with the examples.

Show 25 quoted lines
> often to carry across in-progress work or stashes.  Despite this not
> being a recommended approach, users want to do it and are not dissuaded
> by suggestions not to, so let's recommend a sensible technique.
>
> The technique that many users are using is their preferred cloud syncing
> service, which is a bad idea.  Users have reported problems where they
> end up with duplicate files that won't go away (with names like "file.c
> 2"), broken references, oddly named references that have date stamps
> appended to them, missing objects, and general corruption and data loss.
> That's because almost all of these tools sync file by file, which is a
> great technique if your project is a single word processing document or
> spreadsheet, but is utterly abysmal for Git repositories because they
> don't necessarily snapshot the entire repository correctly.  They also
> tend to sync the files immediately instead of when the repository is
> quiescent, so writing multiple files, as occurs during a commit or a gc,
> can confuse the tools and lead to corruption.
>
> We know that the old standby, rsync, is up to the task, provided that
> the repository is quiescent, so let's suggest that and dissuade people
> from using cloud syncing tools.  Let's tell people about common things
> they should be aware of before doing this and that this is still
> potentially risky.  Additionally, let's tell people that Git's security
> model does not permit sharing working trees across users in case they
> planned to do that.  While we'd still prefer users didn't try to do
> this, hopefully this will lead them in a safer direction.
The remainder of the commit message is very clear.

Thank you, Dscho

brian m. carlson· Oct 20, 2021, 01:06 UTC · re: brian m. carlson · lore

[PATCH 1/4] gitfaq: add advice on monorepos

Many projects around the world have chosen monorepos, and active development on Git is ongoing to support them better. However, as projects using monorepos grow, they often find various performance and scalability problems that are unpleasant to deal with.

Add a FAQ entry to note that while Git is attempting improvements in this area, it is not uncommon to see performance problems that necessitate the use of partial or shallow clone, sparse checkout, or the like, and that if users wish to avoid these problems, avoiding a monorepo may be best.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
Show changes to Documentation/gitfaq.txt +26 −0
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 8c1f2d5675..946691c153 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -241,6 +241,32 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+Design
+------
+
+[[monorepos]]
+Should we use a monorepo or many individual repos?::
+	This is a decision that is typically made based on an organization's needs and
+	desires for their projects.  Git has several features, such as shallow clone,
+	partial clone, and sparse checkout to make working with large repositories
+	easier, and there is active development on making the monorepo experience
+	better.
++
+However, at a certain size, the performance of a monorepo will likely become
+unacceptable _unless_ you use these features.  If you choose to start with a
+monorepo and continue to grow, you may end up unhappy with the performance
+characteristics at a point where making a change is difficult.  The performance
+of using many smaller repositories will almost always be much better and will
+generally not necessitate the use of these more advanced features.  If you are
+concerned about future performance of your repository and related tools, you may
+wish to avoid a monorepo.
++
+Ultimately, you should make a decision fully informed about the potential
+benefits and downsides, including the capabilities, performance, and future
+requirements for your repository and related tools, including your hosting
+platform, build tools, and other programs you typically use as part of your
+workflow.
+
 Merging and Rebasing
 --------------------
Bagas Sanjaya· Oct 20, 2021, 04:45 UTC · re: brian m. carlson · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

On 20/10/21 08.06, brian m. carlson wrote:
Show 27 quoted lines
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.
> +
>   Merging and Rebasing
>   --------------------
>   
> 
It seems like recommending to split repo, right?

Ultimately, if people choose split repo instead of monorepo, it will only delay the necessity to use advanced features (partial/shallow clones, sparse checkout, etc.) when the repos become large.

For balanced view, we should describe benefits and drawbacks of both monorepo and split repos.

-- 
An old man doll... just what I always wanted! - Clara
Ævar Arnfjörð Bjarmason· Oct 20, 2021, 10:54 UTC · re: brian m. carlson · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

On Wed, Oct 20 2021, brian m. carlson wrote:
Show 22 quoted lines
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.

In the context of git development we're typically talking about really big repos when we're talking about monorepos, saying "monorepo" communicates among other things that the user of that pattern is unwilling to use splitting up as a way to address any scalability issues they may have.

But a monorepo doesn't really say anything about size per-se, and it would be confusing to conflate the two in a FAQ. I may be wrong, perhaps the term has really come to exclusively refer to colossal size, but I haven't seen or heard it exclusively (or even mainly) used like that

My understanding of what a monorepo in the context of software development is a collection of all your "main" code and all its dependencies in one repository, such that a person working on it rarely or never has to worry about N=N dependencies between different pieces of that collection, they'll all move in unison. You are able to atomically change a function and all its users.

I think people have a different understanding of "all its dependencies". Some monorepo users really mean it and try to e.g. fold their system configuration system that might manage files in /etc in with their monorepo, others might have "another monorepo" for that software, etc.

I bet that the vast majority of monorepo users are never going to experience scaling problems, e.g. having your laptop dotfiles and automation of /etc in one repo is a "monorepo", and most companies/teams that use monorepos I'd bet are in the long tail of size distribution. They're not going to grow to the size of a MS's, FB's etc. monorepo, but they might benefit (or not) from the monorepo /workflow/.

Anyway, all of the above can be read as a suggestion that we should split any discussion of "large repo [that runs into scaling issues]" from "monorepo", the latter should of course make a passing reference to scaling (as the pattern will lead to that sooner than not), but IMO not conflate the two.

brian m. carlson· Oct 20, 2021, 21:19 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

On 2021-10-20 at 10:54:47, Ævar Arnfjörð Bjarmason wrote:
Show 36 quoted lines
> 
> On Wed, Oct 20 2021, brian m. carlson wrote:
> 
> > +[[monorepos]]
> > +Should we use a monorepo or many individual repos?::
> > +	This is a decision that is typically made based on an organization's needs and
> > +	desires for their projects.  Git has several features, such as shallow clone,
> > +	partial clone, and sparse checkout to make working with large repositories
> > +	easier, and there is active development on making the monorepo experience
> > +	better.
> > ++
> > +However, at a certain size, the performance of a monorepo will likely become
> > +unacceptable _unless_ you use these features.  If you choose to start with a
> > +monorepo and continue to grow, you may end up unhappy with the performance
> > +characteristics at a point where making a change is difficult.  The performance
> > +of using many smaller repositories will almost always be much better and will
> > +generally not necessitate the use of these more advanced features.  If you are
> > +concerned about future performance of your repository and related tools, you may
> > +wish to avoid a monorepo.
> > ++
> > +Ultimately, you should make a decision fully informed about the potential
> > +benefits and downsides, including the capabilities, performance, and future
> > +requirements for your repository and related tools, including your hosting
> > +platform, build tools, and other programs you typically use as part of your
> > +workflow.
> 
> In the context of git development we're typically talking about really
> big repos when we're talking about monorepos, saying "monorepo"
> communicates among other things that the user of that pattern is
> unwilling to use splitting up as a way to address any scalability issues
> they may have.
> 
> But a monorepo doesn't really say anything about size per-se, and it
> would be confusing to conflate the two in a FAQ. I may be wrong, perhaps
> the term has really come to exclusively refer to colossal size, but I
> haven't seen or heard it exclusively (or even mainly) used like that

I routinely hear "monorepo" used to imply repositories of specifically large size. However, I'm happy to rephrase to make it clearer.

Show 7 quoted lines
> I bet that the vast majority of monorepo users are never going to
> experience scaling problems, e.g. having your laptop dotfiles and
> automation of /etc in one repo is a "monorepo", and most companies/teams
> that use monorepos I'd bet are in the long tail of size
> distribution. They're not going to grow to the size of a MS's, FB's
> etc. monorepo, but they might benefit (or not) from the monorepo
> /workflow/.

I almost never hear individuals refer to such a configuration as a monorepo. Technically, it is one, yes, but I almost always hear it in the context of an organization's repository covering all of their services or the entirety of one major project.

I will point out that I personally would run into scaling issues if I put all of my projects in the same repository. I have many projects, and that would quickly become unsustainable, since the resources I have at my disposal are more limited than most organizations.

Show 5 quoted lines
> Anyway, all of the above can be read as a suggestion that we should
> split any discussion of "large repo [that runs into scaling issues]"
> from "monorepo", the latter should of course make a passing reference to
> scaling (as the pattern will lead to that sooner than not), but IMO not
> conflate the two.

I'm happy to clarify, but I think we need to mention the word "monorepo" specifically because (a) that's the term that's commonly used for this approach and (b) that approach is one that tends to lead to significantly greater growth in a single repository leading to scale problems.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Johannes Schindelin· Oct 20, 2021, 11:55 UTC · re: brian m. carlson · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

Hi brian,
thank you for this excellent idea to talk about monorepos.
On Wed, 20 Oct 2021, brian m. carlson wrote:
Show 16 quoted lines
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 8c1f2d5675..946691c153 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,32 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>
> +Design
> +------
> +
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories

May I request taking out shallow clones? A user new to Git might think that shallow clones are a sane way to clone a large repository. In practice, this only makes sense for "throw-away" clones, though. As soon as you fetch in such clones, performance will be so horrible that it is frequently a better idea to start with a partial clone instead.

At the same time, I would like to swap in "sparse index" for "shallow clone" because it _does_ have the best potential of all currently discussed new features to improve performance with monorepos.

Show 5 quoted lines
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a

I would like to add a plug for Scalar here. Maybe we can link to this "opinionated tool based on Git" here? I wouldn't ask if I didn't _know_ that it helps monorepo users out there.

Show 12 quoted lines
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.

I wish we had a good article to link to, here. Yes, it is a decision that should be fully informed, and yes, this FAQ entry is not the place for a thorough discussion of monorepos and how Git can be asked to handle them more efficiently.

Do you know of any good resource that we could use here?

Thanks, Dscho

Show 5 quoted lines
> +
>  Merging and Rebasing
>  --------------------
>
>
Philip Oakley· Oct 20, 2021, 14:11 UTC · re: brian m. carlson · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

On 20/10/2021 02:06, brian m. carlson wrote:
Show 50 quoted lines
> Many projects around the world have chosen monorepos, and active
> development on Git is ongoing to support them better.  However, as
> projects using monorepos grow, they often find various performance
> and scalability problems that are unpleasant to deal with.
>
> Add a FAQ entry to note that while Git is attempting improvements in
> this area, it is not uncommon to see performance problems that
> necessitate the use of partial or shallow clone, sparse checkout, or the
> like, and that if users wish to avoid these problems, avoiding a
> monorepo may be best.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/gitfaq.txt | 26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
>
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 8c1f2d5675..946691c153 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,32 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>  
> +Design
> +------
> +
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.
> +

Does this need some comparison, or link, with sub-module methods and issues? Such as the nested sub-module problem, the distinction between active sub-modules and quiescent sub-modules (e.g. libraries Vx.y.z)?

As an aside, I don't think we provide any background to the Git philosophy that frames some of these issues.

-- 
Philip
brian m. carlson· Oct 20, 2021, 22:22 UTC · re: Philip Oakley · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

On 2021-10-20 at 14:11:09, Philip Oakley wrote:
> Does this need some comparison, or link, with sub-module methods and
> issues? Such as the nested sub-module problem, the distinction between
> active sub-modules and quiescent sub-modules (e.g. libraries Vx.y.z)?

I don't think it does. Some projects choose to use many repositories with submodules, and some use many repositories without submodules. At work, we do the latter, and it tends to work just fine.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Philip Oakley· Oct 25, 2021, 10:44 UTC · re: brian m. carlson · lore

Re: [PATCH 1/4] gitfaq: add advice on monorepos

Hi Brian, On 20/10/2021 23:22, brian m. carlson wrote:

Show 7 quoted lines
> On 2021-10-20 at 14:11:09, Philip Oakley wrote:
>> Does this need some comparison, or link, with sub-module methods and
>> issues? Such as the nested sub-module problem, the distinction between
>> active sub-modules and quiescent sub-modules (e.g. libraries Vx.y.z)?
> I don't think it does.  Some projects choose to use many repositories
> with submodules, and some use many repositories without submodules.  At
> work, we do the latter, and it tends to work just fine.

To clarify, my comment was with regard to the complementary discussions about _choice_ of repo types, rather than mono-repo and other post-choice issues. Possibly, part of such a discussion on choice of repo-type, could include the potential slippery slope between the different uses of sub-modules.

I feel a lot of the difficult sub-module discussion are because we don't have a common terminology for the different (mental) models of sub-module use, their benefits and problems. -- Philip

brian m. carlson· Oct 20, 2021, 01:06 UTC · re: brian m. carlson · lore

[PATCH 2/4] gitfaq: add documentation on proxies

Many corporate environments and local systems have proxies in use. Note the situations in which proxies can be used and how to configure them. At the same time, note what standards a proxy must follow to work with Git. Explicitly call out certain classes that are known to routinely have problems reported various places online, including in the Git for Windows issue tracker and on Stack Overflow, and recommend against the use of such software.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
Show changes to Documentation/gitfaq.txt +18 −0
diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 946691c153..abc0f62e6c 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -241,6 +241,24 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+[[proxy]]
+Can I use a proxy with Git?::
+	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
+	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
+	it also can be configured with `http.proxy` and similar options for HTTPS (see
+	linkgit:git-config[1]).  The `http.proxy` and related options can be
+	customized on a per-URL pattern basis.  In addition, Git can in theory
+	function normally with transparent proxies that exist on the network.
++
+However, note that for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that
+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.
+
 Design
 ------
Johannes Schindelin· Oct 20, 2021, 11:57 UTC · re: brian m. carlson · lore

Re: [PATCH 2/4] gitfaq: add documentation on proxies

Hi brian,
On Wed, 20 Oct 2021, brian m. carlson wrote:
Show 7 quoted lines
> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.
I fully agree with this patch, therefore: ACK!

Ciao, Dscho

brian m. carlson· Oct 20, 2021, 22:17 UTC · re: Johannes Schindelin · lore

Re: [PATCH 2/4] gitfaq: add documentation on proxies

On 2021-10-20 at 11:57:42, Johannes Schindelin wrote:
Show 13 quoted lines
> Hi brian,
> 
> On Wed, 20 Oct 2021, brian m. carlson wrote:
> 
> > Many corporate environments and local systems have proxies in use.  Note
> > the situations in which proxies can be used and how to configure them.
> > At the same time, note what standards a proxy must follow to work with
> > Git.  Explicitly call out certain classes that are known to routinely
> > have problems reported various places online, including in the Git for
> > Windows issue tracker and on Stack Overflow, and recommend against the
> > use of such software.
> 
> I fully agree with this patch, therefore: ACK!

I CC'd you specifically because of this patch, and I'm glad to hear you agree, even if I'm sorry you have to deal with these reports.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Junio C Hamano· Oct 20, 2021, 14:48 UTC · re: brian m. carlson · lore

Re: [PATCH 2/4] gitfaq: add documentation on proxies

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
Show 7 quoted lines
> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.

The new section only talks about http(s), but I recall Peff recently helped folks with broken invocation of socat, a relay that does not handle severing connection only in one direction very well.

I wonder if it fits in this section.
Show 34 quoted lines
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/gitfaq.txt | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 946691c153..abc0f62e6c 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,24 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>  
> +[[proxy]]
> +Can I use a proxy with Git?::
> +	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
> +	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
> +	it also can be configured with `http.proxy` and similar options for HTTPS (see
> +	linkgit:git-config[1]).  The `http.proxy` and related options can be
> +	customized on a per-URL pattern basis.  In addition, Git can in theory
> +	function normally with transparent proxies that exist on the network.
> ++
> +However, note that for Git to work properly, the proxy must be completely
> +transparent.  The proxy cannot modify, tamper with, change, or buffer the
> +connection in any way, or Git will almost certainly fail to work.  Note that
> +many proxies, including many TLS middleboxes, Windows antivirus and firewall
> +programs other than Windows Defender and Windows Firewall, and filtering proxies
> +fail to meet this standard, and as a result end up breaking Git.  Because of the
> +many reports of problems, we recommend against the use of these classes of
> +software and devices.
> +
>  Design
>  ------
>
brian m. carlson· Oct 20, 2021, 22:19 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/4] gitfaq: add documentation on proxies

On 2021-10-20 at 14:48:04, Junio C Hamano wrote:
Show 15 quoted lines
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > Many corporate environments and local systems have proxies in use.  Note
> > the situations in which proxies can be used and how to configure them.
> > At the same time, note what standards a proxy must follow to work with
> > Git.  Explicitly call out certain classes that are known to routinely
> > have problems reported various places online, including in the Git for
> > Windows issue tracker and on Stack Overflow, and recommend against the
> > use of such software.
> 
> The new section only talks about http(s), but I recall Peff recently
> helped folks with broken invocation of socat, a relay that does not
> handle severing connection only in one direction very well.
> 
> I wonder if it fits in this section.
I think it does, and I'll add it in a reroll.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

← back to recent threads