{"thread":{"id":"55229","subject":"[PATCH 3/4] docs: add a FAQ section on push and fetch problems","startedAt":"2021-02-27T19:21:44Z","lastAt":"2021-03-15T20:41:33Z","messageCount":16,"participants":["brian m. carlson","Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"417946","messageId":"20210227191813.96148-4-sandals@crustytoothpaste.net","threadId":"55229","inReplyTo":"20210227191813.96148-1-sandals@crustytoothpaste.net","subject":"[PATCH 3/4] docs: add a FAQ section on push and fetch problems","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-27T19:18:12Z","receivedAt":"2021-02-27T19:21:44Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There are a lot of questions on the Internet about common problems with\nfetching and pushing. Roughly, the vast majority of these problems are\nwhen using HTTP and involve HTTP/2 streams, certain HTTP errors, or\nconnections which are interrupted. This latter case is especially\ncommon on Windows where non-default antivirus and firewall software\nfrequently tampers with connections in undesirable ways.\n\nLet's add some FAQ entries explaining what is happening and how to\ntroubleshoot and solve these problems. When discussing network\nconnection issues, explicitly call out TLS man-in-the-middle devices,\nproxies, antivirus programs, and firewall applications, which are the\ncause of most of these problems, and encourage users to report these\nprograms as broken. Since many sites offer both HTTPS and SSH, suggest\nusing SSH, which is often not intercepted, as a good way to work around\nthese problems.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 100 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 100 insertions(+)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex a132f66032..fde54d2664 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -185,6 +185,106 @@ Then, you can adjust your push URL to use `git@example_author` or\n `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url\n git@example_author:org1/project1.git`).\n \n+Problems Fetching and Pushing\n+-----------------------------\n+\n+[[remote-connection-http-2-stream-error]]\n+Why do I get an error about an HTTP/2 stream not being closed cleanly?::\n+\tSometimes when pushing or fetching over HTTP, users see a message such as \"RPC\n+\tfailed; curl 92 HTTP/2 stream 0 was not closed cleanly\". This message\n+\tindicates that Git is using HTTP/2, a recent version of the HTTP protocol, and\n+\tthat the remote server, or a middlebox, such as a proxy, TLS middlebox,\n+\tantivirus, or firewall, failed to speak the protocol correctly and thus the\n+\tconnection was interrupted.\n++\n+In such a case, the software causing the problem is buggy and will likely be\n+broken with a wide variety of web browsers and other HTTP-using applications.\n+The best thing to do is contact the responsible party to get the software fixed.\n++\n+If that isn't possible, you can set the `http.version` option to `HTTP/1.1`,\n+which will force the use of an older version of HTTP. This should allow Git to\n+function with this broken software or device. If the remote server supports\n+SSH, you may wish to try switching to SSH instead.\n+\n+[[remote-connection-http-411]]\n+Why do I get an error about an HTTP 411 status?::\n+\tSometimes users see error messages when pushing that refer to HTTP status 411,\n+\tsuch as \"RPC failed; result=22, HTTP code = 411.\" This status means that the\n+\tserver or a machine in the middle, such as a proxy, TLS middlebox, antivirus,\n+\tfirewall, or other middlebox, refuses to accept a streaming data connection.\n++\n+When pushing or fetching over HTTP, Git normally uses a small buffer and, if the\n+data is large, uses HTTP 1.1 chunked transfer encoding or HTTP 2 streaming to\n+send the data without a defined size. This is useful because it allows a push\n+or fetch to start much faster and therefore complete much faster. This type of\n+streaming has been standardized since 1999 and is well understood, and all\n+modern software should be capable of supporting it.\n++\n+However, in this case, the remote server or middlebox is misconfigured and does\n+not correctly support this. The best thing to do is contact the responsible\n+party and ask them to fix the server or middlebox, since this misconfiguration\n+can affect many pieces of software, some of which will simply not function at\n+all in this environment.\n++\n+If the remote server supports SSH, you may wish to try using SSH instead. If\n+that is not possible, you can set `http.postBuffer` to a larger value as a\n+workaround. This is one of the few times when that option is useful, but note\n+<> that doing so will increase\n+the memory usage for every push, no matter how small, and will not be able to\n+handle pushes of arbitrary sizes, so fixing the broken server or device or\n+switching to SSH is preferable in almost all cases.\n+\n+[[remote-connection-reset]]\n+Why do I get errors that the connection was reset?::\n+\tWhen pushing or fetching, sometimes users see problems where the connection\n+\twas reset. Common symptoms of this problem include (but are not limited to)\n+\tmessages like the following:\n++\n+* RPC failed; curl 56 OpenSSL SSL_read: Connection was reset, errno 10054\n+* RPC failed; curl 55 SSL_write() returned SYSCALL, errno = 10053\n+* RPC failed; curl 56 LibreSSL SSL_read: SSL_ERROR_SYSCALL, errno 60\n+* RPC failed; result=56, HTTP code = 200\n+* RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.\n++\n+These messages, and almost every message with a libcurl error code of 55 or 56,\n+essentially mean that the network connection between Git and the remote server\n+was terminated unexpectedly. This can be caused by any sort of generic network\n+problem, such as packet loss or an unstable connection. Sometimes users also\n+see it when connected to a VPN if the connection over the VPN is unstable. In\n+such a case, disabling the VPN or switching to a different connection may help\n+the problem, or sending or receiving less data may work around the problem.\n++\n+This may also be caused by devices or software in the middle of the connection\n+which attempt to inspect the data. For example, if you're on a network which\n+uses a TLS middlebox or a proxy, these devices may attempt to inspect the data\n+and terminate the connection if the data is too large for them to handle or if\n+they mistakenly think it is malicious, offensive, inappropriate, or otherwise\n+unacceptable. To test if this is the problem, try using a different network\n+where these devices are not enabled, or contact your network administrator and\n+report the problem to them.\n++\n+On Windows, and to a lesser extent on other platforms, antivirus, firewall, or\n+network monitoring software that is not the default (on Windows, something other\n+than Windows Defender and Windows Firewall) can intercept network connections\n+and cause the same problems as the devices mentioned above. This may also\n+happen when using Git under the Windows Subsystem for Linux with such software.\n+To test if this is the problem, remove the non-default software completely and\n+restart your computer. Some such software does not disable the broken\n+functionality properly when it is set to disabled and so removing the software\n+is the only way to perform the test. If this is the problem, use the default\n+software instead, report the problem to the software vendor, or contact your\n+network administrator and report the problem to them.\n++\n+If you are using HTTPS and the remote server supports SSH, you may wish to try\n+using SSH instead.\n++\n+Note that in all these cases, this is not a problem in Git, but a problem with\n+the network or the devices and software managing it. Some parties mistakenly\n+recommend adjusting the `http.postBuffer` setting to work around this, but\n+<> for why that usually doesn't work, and\n+even when it does work, indicates a defect in the network or software such as\n+one mentioned above in this answer.\n+\n Common Issues\n -------------\n \n"},{"id":"417947","messageId":"20210227191813.96148-2-sandals@crustytoothpaste.net","threadId":"55229","inReplyTo":"20210227191813.96148-1-sandals@crustytoothpaste.net","subject":"[PATCH 1/4] docs: add a question on syncing repositories to the FAQ","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-27T19:18:10Z","receivedAt":"2021-02-27T19:21:59Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"It is very common that users want to transport repositories with working\ntrees across machines. While this is not recommended, many users do it\nanyway and moreover, do it using cloud syncing services, which often\ncorrupt their data. The results of such are often seen in tales of woe\non common user question fora.\n\nLet's tell users what we recommend they do in this circumstance and how\nto do it safely. Warn them about the dangers of untrusted working trees\nand the downsides of index refreshes, as well as the problems with cloud\nsyncing services.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex afdaeab850..042b11e88a 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -241,6 +241,45 @@ How do I know if I want to do a fetch or a pull?::\n \tignore the upstream changes. A pull consists of a fetch followed\n \timmediately by either a merge or rebase. See linkgit:git-pull[1].\n \n+[[syncing-across-computers]]\n+How do I sync a Git repository across multiple computers, VMs, or operating systems?::\n+\tThe best way to sync a repository across computers is by pushing and fetching.\n+\tThis uses the native Git mechanisms to transport data efficiently and is the\n+\teasiest and best way to move data across machines. If the machines aren't\n+\tconnected by a network, you can use `git bundle` to create a file with your\n+\tchanges and then fetch or pull them from the file on the remote machine.\n+\tPushing and fetching are also the only secure ways to interact with a\n+\trepository you don't own or trust.\n++\n+However, sometimes people want to sync a repository with a working tree across\n+machines. While this isn't recommended, it can be done with `rsync` (usually\n+over an SSH connection), but only when the repository is completely idle (that\n+is, no processes, including `git gc`, are modifying it at all). If `rsync`\n+isn't available, you can use `tar` to create a tar archive of the repository and\n+copy it to another machine. Zip files shouldn't be used due to their poor\n+support for permissions and symbolic links.\n++\n+You may also use a shared file system between the two machines that is POSIX\n+compliant, such as SSHFS (SFTP) or NFSv4. If you are using SFTP for this\n+purpose, the server should support fsync and POSIX renames (OpenSSH does). File\n+systems that don't provide POSIX semantics, such as DAV mounts, shouldn't be\n+used.\n++\n+Note that you must not work with untrusted working trees, since it's trivial\n+for an attacker to set configuration options that will cause arbitrary code to\n+be executed on your machine. Also, in almost all cases when sharing a working\n+tree across machines, Git will need to re-read all files the next time you run\n+`git status` or otherwise refresh the index, which can be slow. This generally\n+can't be avoided and is part of the reason why sharing a working tree isn't\n+recommended.\n++\n+In no circumstances should you share a working tree or bare repository using a\n+cloud syncing service or store it in a directory managed by such a service.\n+Such services sync file by file and don't maintain the invariants required for\n+repository integrity; in addition, they can cause files to be added, removed, or\n+duplicated unexpectedly. If you must use one of these services, use it to store\n+the repository in a tar archive instead.\n+\n Merging and Rebasing\n --------------------\n \n"},{"id":"417948","messageId":"20210227191813.96148-1-sandals@crustytoothpaste.net","threadId":"55229","inReplyTo":null,"subject":"[PATCH 0/4] Documentation updates to FAQ and git-archive","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-27T19:18:09Z","receivedAt":"2021-02-27T19:22:49Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series introduces several new FAQ items and an update to the\ngit-archive documentation.\n\nThe first three patches introduce FAQ entries for questions I've seen\nextremely frequently on Stack Overflow. Since clearly users are seeing\nthese problems, we should update our documentation to address them and\nhelp users find clear and accurate solutions.\n\nI realize that suggesting people share a working tree across systems is\ncontroversial, but people are doing it, so let's tell them how to do it\nsafely. Users frequently use things like Dropbox, OneDrive, iCloud, and\nsimilar cloud syncing services to do this and then wonder why things are\nbroken or their repository is corrupted. We tell them that\nPOSIX-compliant file systems should be used and give examples of what we\nknow does and doesn't work, and we tell them about the security pitfalls\nof untrusted working trees.\n\nThe third patch addresses several common situations with HTTP pushes and\nfetches. The majority of these problems are going to be with TLS MITM\ndevices, intercepting and filtering proxies of various sorts, and\nnon-default antivirus and firewalls, all of which security experts\nsteadfastly recommend against. We don't do so here (yet), but we do\nexplicitly call them out as potential sources of problems and we\nencourage users to report these problems to vendors and network\nadministrators so that they can be addressed.\n\nThe fourth patch states a fact which we've been explicit about on the\nlist but have never documented: that the output of git archive is not\nstable. I do recall that I sent a patch breaking kernel.org's\ninfrastructure in the past due to a change in archive output and I'd\nlike to avoid other folks relying on bit-for-bit identical output.\n\nbrian m. carlson (4):\n docs: add a question on syncing repositories to the FAQ\n docs: add line ending configuration article to FAQ\n docs: add a FAQ section on push and fetch problems\n docs: note that archives are not stable\n\n Documentation/git-archive.txt | 3 +\n Documentation/gitfaq.txt | 176 +++++++++++++++++++++++++++++++++-\n 2 files changed, 178 insertions(+), 1 deletion(-)\n\n"},{"id":"417949","messageId":"20210227191813.96148-3-sandals@crustytoothpaste.net","threadId":"55229","inReplyTo":"20210227191813.96148-1-sandals@crustytoothpaste.net","subject":"[PATCH 2/4] docs: add line ending configuration article to FAQ","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-27T19:18:11Z","receivedAt":"2021-02-27T19:22:55Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"A common source of problems when working across projects is getting line\nendings to work in a consistent way. Let's explain to users how to\nconfigure their line endings such that they're automatically converted\nusing the .gitattributes file. Update a reference to an incorrect FAQ\nentry by referring to the previous entry instead of the following one.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 37 ++++++++++++++++++++++++++++++++++++-\n 1 file changed, 36 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex 042b11e88a..a132f66032 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -387,6 +387,41 @@ repository will apply to all users of the repository.\n See the following entry for information about normalizing line endings as well,\n and see linkgit:gitattributes[5] for more information about attribute files.\n \n+[[line-ending-gitattributes]]\n+How do I fix my line endings to work well across platforms?::\n+\tThe best way to do this is to ask Git to perform automatic line ending\n+\tconversion in your repository such that it always stores LF (Unix) line\n+\tendings in the repository and checks them out to the user's preferred endings.\n+\tThis is done using the `text` attribute in the `.gitattributes` file in the\n+\troot of your repository. If you want to use the built-in heuristic to\n+\tdetermine text files, you can write this:\n++\n+----\n+* text=auto\n+----\n++\n+If you have certain files that must always use specific line endings when\n+checked out, such as shell scripts, or PowerShell files, you can specifically\n+specify the line endings to be used, and you can also specifically mark some\n+files as not wanting line-ending conversion (`-text`):\n++\n+----\n+*.sh text eol=lf\n+*.ps1 text eol=crlf\n+*.jpg -text\n+----\n++\n+When you're done making these changes to the `.gitattributes` file, run `git add\n+--renormalize .` and then commit. This will make sure that the files in the\n+repository are properly stored with LF endings.\n++\n+Using this approach means that each developer can choose the line endings that\n+are best for their environment while keeping the repository consistent, avoiding\n+needless changes in the repository based on differing line endings, and allowing\n+tools like `git diff` to not display spurious whitespace errors.\n++\n+See linkgit:gitattributes[5] for more information about attribute files.\n+\n [[windows-diff-control-m]]\n I'm on Windows and git diff shows my files as having a `^M` at the end.::\n \tBy default, Git expects files to be stored with Unix line endings. As such,\n@@ -396,7 +431,7 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::\n +\n You can store the files in the repository with Unix line endings and convert\n them automatically to your platform's line endings. To do that, set the\n-configuration option `core.eol` to `native` and see the following entry for\n+configuration option `core.eol` to `native` and see the previous entry for\n information about how to configure files as text or binary.\n +\n You can also control this behavior with the `core.whitespace` setting if you\n"},{"id":"417950","messageId":"20210227191813.96148-5-sandals@crustytoothpaste.net","threadId":"55229","inReplyTo":"20210227191813.96148-1-sandals@crustytoothpaste.net","subject":"[PATCH 4/4] docs: note that archives are not stable","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-27T19:18:13Z","receivedAt":"2021-02-27T19:23:29Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We have in the past told users on the list that git archive does not\nnecessarily produce stable archives, but we've never explicitly\ndocumented this. Unfortunately, we've had people in the past who have\nrelied on the relative stability of our archives to their detriment and\nthen had breakage occur.\n\nLet's tell people that we don't guarantee stable archives so that they\ncan make good choices about how they structure their tooling and don't\nend up with problems if we need to change archives later.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-archive.txt | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex 9f8172828d..1f126cbdcc 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -30,6 +30,9 @@ extended pax header if the tar format is used; it can be extracted\n using 'git get-tar-commit-id'. In ZIP files it is stored as a file\n comment.\n \n+The output of 'git archive' is not guaranteed to be stable and may change\n+between versions.\n+\n OPTIONS\n -------\n \n"},{"id":"417973","messageId":"87k0qsl65o.fsf@evledraar.gmail.com","threadId":"55229","inReplyTo":"20210227191813.96148-4-sandals@crustytoothpaste.net","subject":"Re: [PATCH 3/4] docs: add a FAQ section on push and fetch problems","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-28T12:37:39Z","receivedAt":"2021-02-28T12:38:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 27 2021, brian m. carlson wrote:\n\n> There are a lot of questions on the Internet about common problems with\n> fetching and pushing. Roughly, the vast majority of these problems are\n> when using HTTP and involve HTTP/2 streams, certain HTTP errors, or\n> connections which are interrupted. This latter case is especially\n> common on Windows where non-default antivirus and firewall software\n> frequently tampers with connections in undesirable ways.\n>\n> Let's add some FAQ entries explaining what is happening and how to\n> troubleshoot and solve these problems. When discussing network\n> connection issues, explicitly call out TLS man-in-the-middle devices,\n> proxies, antivirus programs, and firewall applications, which are the\n> cause of most of these problems, and encourage users to report these\n> programs as broken. Since many sites offer both HTTPS and SSH, suggest\n> using SSH, which is often not intercepted, as a good way to work around\n> these problems.\n>\n> Signed-off-by: brian m. carlson \n> ---\n> Documentation/gitfaq.txt | 100 +++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 100 insertions(+)\n>\n> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\n> index a132f66032..fde54d2664 100644\n> --- a/Documentation/gitfaq.txt\n> +++ b/Documentation/gitfaq.txt\n> @@ -185,6 +185,106 @@ Then, you can adjust your push URL to use `git@example_author` or\n> `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url\n> git@example_author:org1/project1.git`).\n> \n> +Problems Fetching and Pushing\n> +-----------------------------\n> +\n> +[[remote-connection-http-2-stream-error]]\n> +Why do I get an error about an HTTP/2 stream not being closed cleanly?::\n> +\tSometimes when pushing or fetching over HTTP, users see a message such as \"RPC\n> +\tfailed; curl 92 HTTP/2 stream 0 was not closed cleanly\". This message\n> +\tindicates that Git is using HTTP/2, a recent version of the HTTP protocol, and\n> +\tthat the remote server, or a middlebox, such as a proxy, TLS middlebox,\n> +\tantivirus, or firewall, failed to speak the protocol correctly and thus the\n> +\tconnection was interrupted.\n> ++\n> +In such a case, the software causing the problem is buggy and will likely be\n> +broken with a wide variety of web browsers and other HTTP-using applications.\n> +The best thing to do is contact the responsible party to get the software fixed.\n> ++\n> +If that isn't possible, you can set the `http.version` option to `HTTP/1.1`,\n> +which will force the use of an older version of HTTP. This should allow Git to\n> +function with this broken software or device. If the remote server supports\n> +SSH, you may wish to try switching to SSH instead.\n> +\n> +[[remote-connection-http-411]]\n> +Why do I get an error about an HTTP 411 status?::\n> +\tSometimes users see error messages when pushing that refer to HTTP status 411,\n> +\tsuch as \"RPC failed; result=22, HTTP code = 411.\" This status means that the\n> +\tserver or a machine in the middle, such as a proxy, TLS middlebox, antivirus,\n> +\tfirewall, or other middlebox, refuses to accept a streaming data connection.\n> ++\n> +When pushing or fetching over HTTP, Git normally uses a small buffer and, if the\n> +data is large, uses HTTP 1.1 chunked transfer encoding or HTTP 2 streaming to\n> +send the data without a defined size. This is useful because it allows a push\n> +or fetch to start much faster and therefore complete much faster. This type of\n> +streaming has been standardized since 1999 and is well understood, and all\n> +modern software should be capable of supporting it.\n> ++\n> +However, in this case, the remote server or middlebox is misconfigured and does\n> +not correctly support this. The best thing to do is contact the responsible\n> +party and ask them to fix the server or middlebox, since this misconfiguration\n> +can affect many pieces of software, some of which will simply not function at\n> +all in this environment.\n> ++\n> +If the remote server supports SSH, you may wish to try using SSH instead. If\n> +that is not possible, you can set `http.postBuffer` to a larger value as a\n> +workaround. This is one of the few times when that option is useful, but note\n> +<> that doing so will increase\n> +the memory usage for every push, no matter how small, and will not be able to\n> +handle pushes of arbitrary sizes, so fixing the broken server or device or\n> +switching to SSH is preferable in almost all cases.\n> +\n> +[[remote-connection-reset]]\n> +Why do I get errors that the connection was reset?::\n> +\tWhen pushing or fetching, sometimes users see problems where the connection\n> +\twas reset. Common symptoms of this problem include (but are not limited to)\n> +\tmessages like the following:\n> ++\n> +* RPC failed; curl 56 OpenSSL SSL_read: Connection was reset, errno 10054\n> +* RPC failed; curl 55 SSL_write() returned SYSCALL, errno = 10053\n> +* RPC failed; curl 56 LibreSSL SSL_read: SSL_ERROR_SYSCALL, errno 60\n> +* RPC failed; result=56, HTTP code = 200\n> +* RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.\n\nI haven't looked in details at the content of the FAQ itself being added\nhere (as far as proposed solutions etc. go), but I wonder if this\nwouldn't be 10x more useful to users if we cross-linked these errors\nwith the docs, e.g.:\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0290b04891..ffb1001703 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -829,7 +829,7 @@ static int run_slot(struct active_request_slot *slot,\n strbuf_addstr(&msg, curl_errorstr);\n }\n }\n- error(_(\"RPC failed; %s\"), msg.buf);\n+ error(_(\"RPC failed (see 'git help faq'); %s\"), msg.buf);\n strbuf_release(&msg);\n }\n\n\n> +These messages, and almost every message with a libcurl error code of 55 or 56,\n> +essentially mean that the network connection between Git and the remote server\n> +was terminated unexpectedly. This can be caused by any sort of generic network\n> +problem, such as packet loss or an unstable connection. Sometimes users also\n> +see it when connected to a VPN if the connection over the VPN is unstable. In\n> +such a case, disabling the VPN or switching to a different connection may help\n> +the problem, or sending or receiving less data may work around the problem.\n> ++\n> +This may also be caused by devices or software in the middle of the connection\n> +which attempt to inspect the data. For example, if you're on a network which\n> +uses a TLS middlebox or a proxy, these devices may attempt to inspect the data\n> +and terminate the connection if the data is too large for them to handle or if\n> +they mistakenly think it is malicious, offensive, inappropriate, or otherwise\n> +unacceptable. To test if this is the problem, try using a different network\n> +where these devices are not enabled, or contact your network administrator and\n> +report the problem to them.\n> ++\n> +On Windows, and to a lesser extent on other platforms, antivirus, firewall, or\n> +network monitoring software that is not the default (on Windows, something other\n> +than Windows Defender and Windows Firewall) can intercept network connections\n> +and cause the same problems as the devices mentioned above. This may also\n> +happen when using Git under the Windows Subsystem for Linux with such software.\n> +To test if this is the problem, remove the non-default software completely and\n> +restart your computer. Some such software does not disable the broken\n> +functionality properly when it is set to disabled and so removing the software\n> +is the only way to perform the test. If this is the problem, use the default\n> +software instead, report the problem to the software vendor, or contact your\n> +network administrator and report the problem to them.\n> ++\n> +If you are using HTTPS and the remote server supports SSH, you may wish to try\n> +using SSH instead.\n> ++\n> +Note that in all these cases, this is not a problem in Git, but a problem with\n> +the network or the devices and software managing it. Some parties mistakenly\n> +recommend adjusting the `http.postBuffer` setting to work around this, but\n> +<> for why that usually doesn't work, and\n> +even when it does work, indicates a defect in the network or software such as\n> +one mentioned above in this answer.\n> +\n> Common Issues\n> -------------\n> \n\n"},{"id":"417974","messageId":"87h7lwl5mv.fsf@evledraar.gmail.com","threadId":"55229","inReplyTo":"20210227191813.96148-5-sandals@crustytoothpaste.net","subject":"Re: [PATCH 4/4] docs: note that archives are not stable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-28T12:48:56Z","receivedAt":"2021-02-28T12:49:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 27 2021, brian m. carlson wrote:\n\n> We have in the past told users on the list that git archive does not\n> necessarily produce stable archives, but we've never explicitly\n> documented this. Unfortunately, we've had people in the past who have\n> relied on the relative stability of our archives to their detriment and\n> then had breakage occur.\n>\n> Let's tell people that we don't guarantee stable archives so that they\n> can make good choices about how they structure their tooling and don't\n> end up with problems if we need to change archives later.\n>\n> Signed-off-by: brian m. carlson \n> ---\n> Documentation/git-archive.txt | 3 +++\n> 1 file changed, 3 insertions(+)\n>\n> diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\n> index 9f8172828d..1f126cbdcc 100644\n> --- a/Documentation/git-archive.txt\n> +++ b/Documentation/git-archive.txt\n> @@ -30,6 +30,9 @@ extended pax header if the tar format is used; it can be extracted\n> using 'git get-tar-commit-id'. In ZIP files it is stored as a file\n> comment.\n> \n> +The output of 'git archive' is not guaranteed to be stable and may change\n> +between versions.\n\nIs \"stable archive\" a well-known term people would understand, or is\nsomeone going to read this thinking they might extract different content\ntoday than tomorrow ? :) I wonder how much if anything this means to\nsomeone not privy to the recent thread[1] that prompted this patch.\n\nPerhaps something like this instead:\n\n The output of 'git archive' is guaranteed to be the same across\n versions of git, but the archive itself is not guaranteed to be\n bit-for-bit identical.\n\n In practice the output of 'git archive' is relatively stable across\n git versions, but has changed in the past, and most likely will in\n the future.\n\n Since the tar format provides multiple ways to encode the same\n output (ordering, headers, padding etc.) you should not rely on\n output being bit-for-bit identical across versions of git for\n e.g. GPG signing a SHA-256 hash of an archive generated with one\n version of git, and then expecting to be able to validate that GPG\n signature with a freshly generated archive made with same arguments\n on another version of git.\n\n1. https://lore.kernel.org/git/20210122213954.7dlnnpngjoay3oia@chatter.i7.local/\n"},{"id":"417975","messageId":"87eeh0l52n.fsf@evledraar.gmail.com","threadId":"55229","inReplyTo":"20210227191813.96148-2-sandals@crustytoothpaste.net","subject":"Re: [PATCH 1/4] docs: add a question on syncing repositories to the FAQ","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-28T13:01:04Z","receivedAt":"2021-02-28T13:06:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 27 2021, brian m. carlson wrote:\n\n> It is very common that users want to transport repositories with working\n> trees across machines. While this is not recommended, many users do it\n> anyway and moreover, do it using cloud syncing services, which often\n> corrupt their data. The results of such are often seen in tales of woe\n> on common user question fora.\n>\n> Let's tell users what we recommend they do in this circumstance and how\n> to do it safely. Warn them about the dangers of untrusted working trees\n> and the downsides of index refreshes, as well as the problems with cloud\n> syncing services.\n>\n> Signed-off-by: brian m. carlson \n> ---\n> Documentation/gitfaq.txt | 39 +++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 39 insertions(+)\n>\n> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\n> index afdaeab850..042b11e88a 100644\n> --- a/Documentation/gitfaq.txt\n> +++ b/Documentation/gitfaq.txt\n> @@ -241,6 +241,45 @@ How do I know if I want to do a fetch or a pull?::\n> \tignore the upstream changes. A pull consists of a fetch followed\n> \timmediately by either a merge or rebase. See linkgit:git-pull[1].\n> \n> +[[syncing-across-computers]]\n> +How do I sync a Git repository across multiple computers, VMs, or operating systems?::\n> +\tThe best way to sync a repository across computers is by pushing and fetching.\n> +\tThis uses the native Git mechanisms to transport data efficiently and is the\n> +\teasiest and best way to move data across machines. If the machines aren't\n> +\tconnected by a network, you can use `git bundle` to create a file with your\n> +\tchanges and then fetch or pull them from the file on the remote machine.\n> +\tPushing and fetching are also the only secure ways to interact with a\n> +\trepository you don't own or trust.\n> ++\n> +However, sometimes people want to sync a repository with a working tree across\n> +machines. While this isn't recommended, it can be done with `rsync` (usually\n> +over an SSH connection), but only when the repository is completely idle (that\n> +is, no processes, including `git gc`, are modifying it at all). If `rsync`\n> +isn't available, you can use `tar` to create a tar archive of the repository and\n> +copy it to another machine. Zip files shouldn't be used due to their poor\n> +support for permissions and symbolic links.\n> ++\n> +You may also use a shared file system between the two machines that is POSIX\n> +compliant, such as SSHFS (SFTP) or NFSv4. If you are using SFTP for this\n> +purpose, the server should support fsync and POSIX renames (OpenSSH does). File\n> +systems that don't provide POSIX semantics, such as DAV mounts, shouldn't be\n> +used.\n> ++\n> +Note that you must not work with untrusted working trees, since it's trivial\n> +for an attacker to set configuration options that will cause arbitrary code to\n> +be executed on your machine. Also, in almost all cases when sharing a working\n> +tree across machines, Git will need to re-read all files the next time you run\n> +`git status` or otherwise refresh the index, which can be slow. This generally\n> +can't be avoided and is part of the reason why sharing a working tree isn't\n> +recommended.\n> ++\n> +In no circumstances should you share a working tree or bare repository using a\n> +cloud syncing service or store it in a directory managed by such a service.\n> +Such services sync file by file and don't maintain the invariants required for\n> +repository integrity; in addition, they can cause files to be added, removed, or\n> +duplicated unexpectedly. If you must use one of these services, use it to store\n> +the repository in a tar archive instead.\n\nI think documentation on this topic is needed, but wonder if we couldn't\nmake this more understandable by going to the heart of the matter, i.e.:\n\n * We prefer push/pull/bundle to copy/replicate .git content\n\n * Regardless, a .git directory can be copied across systems just fine\n if you recursively guarantee snapshot integrity, e.g. it doesn't\n depend on the endian-ness of the OS, or has anything like symlinks in\n there made by git itself.\n\n * Anything which copies .git data on a concurrently updated repo can\n lead to corruption, whether that's cp -R, rsync with any combination\n of flags, some cloud syncing service that expects to present that\n tree to two computers without guaranteeing POSIX fs semantics between\n the two etc.\n\n * A common pitfall with such copying of a .git directory is that file\n deletions are also critical, e.g. rsync without --delete is almost\n guaranteed to produce a corrupt .git if repeated enough times\n (e.g. git might prefer stale loose refs over now-packed ones).\n\n * It's OK to copy .git between system that differ in their support of\n symbolic links, but the work tree may be in an inconsistent state and\n need some manner of \"git reset\" to repair it.\n\nAnd, not sure if this is correct:\n\n * It may be OK to edit a .git directory on a non-POSIX conforming fs\n (but perhaps validate the result with \"git fsck\"). But it's not OK to\n have two writing git processes work on such a repository at the same\n time. Keep in mind that certain operations and default settings (such\n as background gc, see `gc.autoDetach` in linkgit:git-config[1]) might\n result in two processes working on the directory even if you're\n changing it only in one terminal window at a time.\n\nI.e. to go a bit beyond the docs you have of basically saying \"there be\ndragons in non-POSIX\" and describe the particular scenarios where it can\ngo wrong. Something like the above still leaves the door open to users\nusing cloud syncing services, which they can then judge for themselves\nas being OK or not. I'm sure there's some that are far from POSIX\ncompliance that are OK in practice if the above warnings are observed.\n\n"},{"id":"417982","messageId":"YDvbxSfulnzY6vGp@camp.crustytoothpaste.net","threadId":"55229","inReplyTo":"87k0qsl65o.fsf@evledraar.gmail.com","subject":"Re: [PATCH 3/4] docs: add a FAQ section on push and fetch problems","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-28T18:07:01Z","receivedAt":"2021-02-28T18:08:05Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-02-28 at 12:37:39, Ævar Arnfjörð Bjarmason wrote:\n> I haven't looked in details at the content of the FAQ itself being added\n> here (as far as proposed solutions etc. go), but I wonder if this\n> wouldn't be 10x more useful to users if we cross-linked these errors\n> with the docs, e.g.:\n> \n> diff --git a/remote-curl.c b/remote-curl.c\n> index 0290b04891..ffb1001703 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -829,7 +829,7 @@ static int run_slot(struct active_request_slot *slot,\n> strbuf_addstr(&msg, curl_errorstr);\n> }\n> }\n> - error(_(\"RPC failed; %s\"), msg.buf);\n> + error(_(\"RPC failed (see 'git help faq'); %s\"), msg.buf);\n> strbuf_release(&msg);\n> }\n\nSure, I can send a patch to do that. That's a great idea.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"417983","messageId":"YDvexO2NFM0KZ1Jo@camp.crustytoothpaste.net","threadId":"55229","inReplyTo":"87h7lwl5mv.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/4] docs: note that archives are not stable","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-28T18:19:48Z","receivedAt":"2021-02-28T18:21:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-02-28 at 12:48:56, Ævar Arnfjörð Bjarmason wrote:\n> Perhaps something like this instead:\n> \n> The output of 'git archive' is guaranteed to be the same across\n> versions of git, but the archive itself is not guaranteed to be\n> bit-for-bit identical.\n> \n> In practice the output of 'git archive' is relatively stable across\n> git versions, but has changed in the past, and most likely will in\n> the future.\n> \n> Since the tar format provides multiple ways to encode the same\n> output (ordering, headers, padding etc.) you should not rely on\n> output being bit-for-bit identical across versions of git for\n> e.g. GPG signing a SHA-256 hash of an archive generated with one\n> version of git, and then expecting to be able to validate that GPG\n> signature with a freshly generated archive made with same arguments\n> on another version of git.\n\nI think something like this is good. I'm a bit nervous about telling\npeople that the output is relatively stable because that will likely\npush people in the direction that we don't want to encourage. I might\nrephrase the first two paragraphs as so:\n\n The output of 'git archive' is guaranteed to be the same across\n versions of git, but the archive itself is not guaranteed to be\n bit-for-bit identical. The output of 'git archive' has changed\n in the past, and most likely will in the future.\n\nI'm not very familiar with the zip format, but I assume that it also has\nfeatures that allow equivalent but not bit-for-bit equal archives.\nLooking at Wikipedia leads me to believe that one could indeed create\ndifferent archives just by either writing a Zip64 record or not, and if\nwe store the SHA-1 revision ID in a comment, then we would also produce\na different archive when using an equivalent SHA-256 repo. And of\ncourse there's compression, which allows many different but equivalent\nserializations. So we'd probably need to say the same thing about zip\nfiles as well.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"417984","messageId":"87zgzojaj4.fsf@evledraar.gmail.com","threadId":"55229","inReplyTo":"YDvexO2NFM0KZ1Jo@camp.crustytoothpaste.net","subject":"Re: [PATCH 4/4] docs: note that archives are not stable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-28T18:46:07Z","receivedAt":"2021-02-28T18:47:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Feb 28 2021, brian m. carlson wrote:\n\n> On 2021-02-28 at 12:48:56, Ævar Arnfjörð Bjarmason wrote:\n>> Perhaps something like this instead:\n>> \n>> The output of 'git archive' is guaranteed to be the same across\n>> versions of git, but the archive itself is not guaranteed to be\n>> bit-for-bit identical.\n>> \n>> In practice the output of 'git archive' is relatively stable across\n>> git versions, but has changed in the past, and most likely will in\n>> the future.\n>> \n>> Since the tar format provides multiple ways to encode the same\n>> output (ordering, headers, padding etc.) you should not rely on\n>> output being bit-for-bit identical across versions of git for\n>> e.g. GPG signing a SHA-256 hash of an archive generated with one\n>> version of git, and then expecting to be able to validate that GPG\n>> signature with a freshly generated archive made with same arguments\n>> on another version of git.\n>\n> I think something like this is good. I'm a bit nervous about telling\n> people that the output is relatively stable because that will likely\n> push people in the direction that we don't want to encourage. I might\n> rephrase the first two paragraphs as so:\n>\n> The output of 'git archive' is guaranteed to be the same across\n> versions of git, but the archive itself is not guaranteed to be\n> bit-for-bit identical. The output of 'git archive' has changed\n> in the past, and most likely will in the future.\n>\n> I'm not very familiar with the zip format, but I assume that it also has\n> features that allow equivalent but not bit-for-bit equal archives.\n> Looking at Wikipedia leads me to believe that one could indeed create\n> different archives just by either writing a Zip64 record or not, and if\n> we store the SHA-1 revision ID in a comment, then we would also produce\n> a different archive when using an equivalent SHA-256 repo. And of\n> course there's compression, which allows many different but equivalent\n> serializations. So we'd probably need to say the same thing about zip\n> files as well.\n\nYes, I think your version is better, and we should have some wording so\nit generalizes to the various output formats we support, perhaps further\nnoting that the \"relatively stable\" (if you want to keep a note of that)\nonly refers to our own output, not when we invoke gzip or zip.\n\nI thought that \"relatively stable\" and \"[when you extract it you get the\nsame thing]\" were good to note, to say that e.g. GPG signing across\nversions = bad, but if you e.g. offer downloadable archives with the\ncontents of tags, there's no reason to make your git version a part of a\ncache key for the purposes of saving yourself CPU time when\n(re-)generating them.\n"},{"id":"418050","messageId":"xmqqtupurbvr.fsf@gitster.c.googlers.com","threadId":"55229","inReplyTo":"20210227191813.96148-4-sandals@crustytoothpaste.net","subject":"Re: [PATCH 3/4] docs: add a FAQ section on push and fetch problems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-01T18:02:00Z","receivedAt":"2021-03-01T18:06:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> +[[remote-connection-http-411]]\n> +Why do I get an error about an HTTP 411 status?::\n> +\tSometimes users see error messages when pushing that refer to HTTP status 411,\n> +\tsuch as \"RPC failed; result=22, HTTP code = 411.\" This status means that the\n> +\tserver or a machine in the middle, such as a proxy, TLS middlebox, antivirus,\n> +\tfirewall, or other middlebox, refuses to accept a streaming data connection.\n> ++\n> +When pushing or fetching over HTTP, Git normally uses a small buffer and, if the\n> +data is large, uses HTTP 1.1 chunked transfer encoding or HTTP 2 streaming to\n> +send the data without a defined size. This is useful because it allows a push\n> +or fetch to start much faster and therefore complete much faster. This type of\n> +streaming has been standardized since 1999 and is well understood, and all\n> +modern software should be capable of supporting it.\n> ++\n> +However, in this case, the remote server or middlebox is misconfigured and does\n> +not correctly support this. The best thing to do is contact the responsible\n> +party and ask them to fix the server or middlebox, since this misconfiguration\n> +can affect many pieces of software, some of which will simply not function at\n> +all in this environment.\n> ++\n> +If the remote server supports SSH, you may wish to try using SSH instead. If\n> +that is not possible, you can set `http.postBuffer` to a larger value as a\n> +workaround. This is one of the few times when that option is useful, but note\n> +<> that doing so will increase\n> +the memory usage for every push, no matter how small, and will not be able to\n> +handle pushes of arbitrary sizes, so fixing the broken server or device or\n> +switching to SSH is preferable in almost all cases.\n\nDon't we rather want to merge this with [[http-postbuffer] part of\nthe faq? If we can have two header lines for the same description\n(i.e. the FAQ list may have\n\n \"What does `http.postBuffer` do?\" aka \"I got HTTP 411--what now?\"\n\nas either a single link or a two separate but clearly related\nentries), that might be ideal.\n\nThanks.\n\n"},{"id":"418052","messageId":"xmqqpn0irb9a.fsf@gitster.c.googlers.com","threadId":"55229","inReplyTo":"YDvexO2NFM0KZ1Jo@camp.crustytoothpaste.net","subject":"Re: [PATCH 4/4] docs: note that archives are not stable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-01T18:15:29Z","receivedAt":"2021-03-01T18:19:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> The output of 'git archive' is guaranteed to be the same across\n> versions of git, but the archive itself is not guaranteed to be\n> bit-for-bit identical.\n\nI do not quite get this; your original was clearer. What does it\nmean to \"be the same across versions of git but not identical\" at\nthe same time? If output from Git version 1.0 and 2.0 are guranteed\nto be the same across versions, what more is there for the readers\nto worry about the format stability?\n\nPerhaps you meant\n\n\t... is guaranteed to be the same for any given version of\n\tGit across ports.\n\nor something? It would allow kernel.org's use of \"Konstantin tells\nkernel.org users to use Git version X to run 'git archive' and\ncreate detached signature on the output, and upload only the\nsignature. The site uses the same Git version X to run 'git\narchive' to create a tarball and the detached signature magically\nmatches, as the output on two places are bit-for-bit identical\".\n\n> The output of 'git archive' has changed\n> in the past, and most likely will in the future.\n\nThat is correct as a statement of fact. I feel that saying it is\neither redundant and insufficient at the same time. If we want to\ntell them \"do not depend on the output being bit-for-bit identical\",\nwe should say it more explicitly after this sentence, I would think.\n\n\n"},{"id":"418123","messageId":"YD7aDwX/aiRN0GZs@camp.crustytoothpaste.net","threadId":"55229","inReplyTo":"xmqqpn0irb9a.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 4/4] docs: note that archives are not stable","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-03-03T00:36:31Z","receivedAt":"2021-03-03T06:42:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-03-01 at 18:15:29, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > The output of 'git archive' is guaranteed to be the same across\n> > versions of git, but the archive itself is not guaranteed to be\n> > bit-for-bit identical.\n> \n> I do not quite get this; your original was clearer. What does it\n> mean to \"be the same across versions of git but not identical\" at\n> the same time? If output from Git version 1.0 and 2.0 are guranteed\n> to be the same across versions, what more is there for the readers\n> to worry about the format stability?\n> \n> Perhaps you meant\n> \n> \t... is guaranteed to be the same for any given version of\n> \tGit across ports.\n> \n> or something? It would allow kernel.org's use of \"Konstantin tells\n> kernel.org users to use Git version X to run 'git archive' and\n> create detached signature on the output, and upload only the\n> signature. The site uses the same Git version X to run 'git\n> archive' to create a tarball and the detached signature magically\n> matches, as the output on two places are bit-for-bit identical\".\n\nI think what I had intended was that Git produces deterministic output,\nbut I don't actually think that's true across ports. If someone uses a\ndifferent version of zlib on a different OS, the output may differ.\n\nI'll rephrase to avoid giving a misleading impression.\n\n> > The output of 'git archive' has changed\n> > in the past, and most likely will in the future.\n> \n> That is correct as a statement of fact. I feel that saying it is\n> either redundant and insufficient at the same time. If we want to\n> tell them \"do not depend on the output being bit-for-bit identical\",\n> we should say it more explicitly after this sentence, I would think.\n\nI agree we should explicitly say that.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"418132","messageId":"xmqqeegwloah.fsf@gitster.c.googlers.com","threadId":"55229","inReplyTo":"YD7aDwX/aiRN0GZs@camp.crustytoothpaste.net","subject":"Re: [PATCH 4/4] docs: note that archives are not stable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-03T06:55:02Z","receivedAt":"2021-03-03T07:07:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> I think what I had intended was that Git produces deterministic output,\n> but I don't actually think that's true across ports. If someone uses a\n> different version of zlib on a different OS, the output may differ.\n\nI agree. When I wrote my response, I had the \"tar\" format in mind,\nwhich we write everything ourselves, but zip and also the compressed\noutput is a different story---we do rely on third-party libraries.\n\nThanks.\n"},{"id":"419183","messageId":"YE/GNoqt1Ja2xZU3@camp.crustytoothpaste.net","threadId":"55229","inReplyTo":"87eeh0l52n.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/4] docs: add a question on syncing repositories to the FAQ","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-03-15T20:40:22Z","receivedAt":"2021-03-15T20:41:33Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-02-28 at 13:01:04, Ævar Arnfjörð Bjarmason wrote:\n> make this more understandable by going to the heart of the matter, i.e.:\n> \n> * We prefer push/pull/bundle to copy/replicate .git content\n> \n> * Regardless, a .git directory can be copied across systems just fine\n> if you recursively guarantee snapshot integrity, e.g. it doesn't\n> depend on the endian-ness of the OS, or has anything like symlinks in\n> there made by git itself.\n\nI'll revise to make this clearer.\n\n> * Anything which copies .git data on a concurrently updated repo can\n> lead to corruption, whether that's cp -R, rsync with any combination\n> of flags, some cloud syncing service that expects to present that\n> tree to two computers without guaranteeing POSIX fs semantics between\n> the two etc.\n> \n> * A common pitfall with such copying of a .git directory is that file\n> deletions are also critical, e.g. rsync without --delete is almost\n> guaranteed to produce a corrupt .git if repeated enough times\n> (e.g. git might prefer stale loose refs over now-packed ones).\n\nI'll include that as well.\n\n> * It's OK to copy .git between system that differ in their support of\n> symbolic links, but the work tree may be in an inconsistent state and\n> need some manner of \"git reset\" to repair it.\n\nAnd this.\n\n> And, not sure if this is correct:\n> \n> * It may be OK to edit a .git directory on a non-POSIX conforming fs\n> (but perhaps validate the result with \"git fsck\"). But it's not OK to\n> have two writing git processes work on such a repository at the same\n> time. Keep in mind that certain operations and default settings (such\n> as background gc, see `gc.autoDetach` in linkgit:git-config[1]) might\n> result in two processes working on the directory even if you're\n> changing it only in one terminal window at a time.\n\nI'll reflect the fact that only one process may modify the repository at\na time.\n\n> I.e. to go a bit beyond the docs you have of basically saying \"there be\n> dragons in non-POSIX\" and describe the particular scenarios where it can\n> go wrong. Something like the above still leaves the door open to users\n> using cloud syncing services, which they can then judge for themselves\n> as being OK or not. I'm sure there's some that are far from POSIX\n> compliance that are OK in practice if the above warnings are observed.\n\nUnfortunately, it's a bit hard to make a concise entry that explains all\nthe different ways that using a non-POSIX compliant file system can\nbreak things. We've seen NFS where open(2) is broken, both with\npermissions and O_EXCL; cloud syncing services that restore files that\nhave been deleted; DAV mounts that don't support the necessary\nsemantics; systems where O_APPEND doesn't have POSIX semantics; and a\nwhole host of other sadness. I don't therefore think that we want to\ntell people that we think that using a file system that doesn't support\nPOSIX semantics is okay, because in most cases, they are not.\n\nUsers frequently try to judge for themselves what works and then they\ntry one of the above things which clearly does not, so saying, \"Try it\nand see if it breaks,\" just tends to result in users complaining about\nrepository corruption later on. I'm really tired of answering these\nsame questions again and again and telling users that their repositories\nare hosed and that they've lost data, so I want to be definitive that we\ndon't recommend or support these various broken environments and that\nusers should not use them. It may be in rare cases that users have\nextensive knowledge about the behavior of their file systems and Git's\nrequirements and can make a sound judgment to use a non-POSIX file\nsystem, but the people on the planet who can do that effectively are\nalmost all Git developers. I will state that \"File systems that don't\nprovide POSIX semantics, such as DAV mounts, shouldn't be used without\nfully understanding the situation and requirements,\" which I think is\nthe most generous recommendation we can safely give.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"}]}