{"thread":{"id":"61732","subject":"[PATCH v3 2/4] gitfaq: give advice on using eol attribute in gitattributes","startedAt":"2024-07-04T00:40:32Z","lastAt":"2024-07-09T23:38:54Z","messageCount":22,"participants":["brian m. carlson","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":3,"patchTotal":4},"messages":[{"id":"498067","messageId":"20240704003818.750223-3-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 2/4] gitfaq: give advice on using eol attribute in gitattributes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T00:38:16Z","receivedAt":"2024-07-04T00:40:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In the FAQ, we tell people how to use the text attribute, but we fail to\nexplain what to do with the eol attribute. As we ourselves have\nnoticed, most shell implementations do not care for carriage returns,\nand as such, people will practically always want them to use LF endings.\nSimilar things can be said for batch files on Windows, except with CRLF\nendings.\n\nSince these are common things to have in a repository, let's help users\nmake a good decision by recommending that they use the gitattributes\nfile to correctly check out the endings.\n\nIn addition, let's correct the cross-reference to this question, which\noriginally referred to \"the following entry\", even though a new entry\nhas been inserted in between. The cross-reference notation should\nprevent this from occurring and provide a link in formats, such as HTML,\nwhich support that.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex e4125b1178..cdc5f5f4f8 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -393,8 +393,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::\n +\n You can store the files in the repository with Unix line endings and convert\n them automatically to your platform's line endings. To do that, set the\n-configuration option `core.eol` to `native` and see the following entry for\n-information about how to configure files as text or binary.\n+configuration option `core.eol` to `native` and see\n+<>\n+for information about how to configure files as text or binary.\n +\n You can also control this behavior with the `core.whitespace` setting if you\n don't wish to remove the carriage returns from your line endings.\n@@ -456,14 +457,26 @@ references, URLs, and hashes stored in the repository.\n +\n We also recommend setting a linkgit:gitattributes[5] file to explicitly mark\n which files are text and which are binary. If you want Git to guess, you can\n-set the attribute `text=auto`. For example, the following might be appropriate\n-in some projects:\n+set the attribute `text=auto`.\n++\n+With text files, Git will generally ensure that LF endings are used in the\n+repository, and will honor `core.autocrlf` and `core.eol` to decide what options\n+to use when checking files out. You can also override this by specifying a\n+particular line ending such as `eol=lf` or `eol=crlf` if those files must always\n+have that ending in the working tree (e.g., for functionality reasons).\n++\n+For example, generally shell files must have LF endings and batch files must\n+have CRLF endings, so the following might be appropriate in some projects:\n +\n ----\n # By default, guess.\n *\ttext=auto\n # Mark all C files as text.\n *.c\ttext\n+# Ensure all shell files have LF endings and all batch files have CRLF\n+# endings in the working tree and both have LF in the repo.\n+*.sh text eol=lf\n+*.bat text eol=crlf\n # Mark all JPEG files as binary.\n *.jpg\tbinary\n ----\n"},{"id":"498068","messageId":"20240704003818.750223-4-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 3/4] gitfaq: add entry about syncing working trees","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T00:38:17Z","receivedAt":"2024-07-04T00:40:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Users very commonly want to sync their working tree with uncommitted\nchanges across machines, often to carry across in-progress work or\nstashes. Despite this not being a recommended approach, users want to\ndo it and are not dissuaded by suggestions not to, so let's recommend a\nsensible technique.\n\nThe technique that many users are using is their preferred cloud syncing\nservice, which is a bad idea. Users have reported problems where they\nend up with duplicate files that won't go away (with names like \"file.c\n2\"), broken references, oddly named references that have date stamps\nappended to them, missing objects, and general corruption and data loss.\nThat's because almost all of these tools sync file by file, which is a\ngreat technique if your project is a single word processing document or\nspreadsheet, but is utterly abysmal for Git repositories because they\ndon't necessarily snapshot the entire repository correctly. They also\ntend to sync the files immediately instead of when the repository is\nquiescent, so writing multiple files, as occurs during a commit or a gc,\ncan confuse the tools and lead to corruption.\n\nWe know that the old standby, rsync, is up to the task, provided that\nthe repository is quiescent, so let's suggest that and dissuade people\nfrom using cloud syncing tools. Let's tell people about common things\nthey should be aware of before doing this and that this is still\npotentially risky. Additionally, let's tell people that Git's security\nmodel does not permit sharing working trees across users in case they\nplanned to do that. While we'd still prefer users didn't try to do\nthis, hopefully this will lead them in a safer direction.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 48 ++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 46 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex cdc5f5f4f8..0e7f1c680d 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -83,8 +83,8 @@ Windows would be the configuration `\"C:\\Program Files\\Vim\\gvim.exe\" --nofork`,\n which quotes the filename with spaces and specifies the `--nofork` option to\n avoid backgrounding the process.\n \n-Credentials\n------------\n+Credentials and Transfers\n+-------------------------\n \n [[http-credentials]]\n How do I specify my credentials when pushing over HTTP?::\n@@ -185,6 +185,50 @@ Then, you can adjust your push URL to use `git@example_author` or\n `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url\n git@example_author:org1/project1.git`).\n \n+[[sync-working-tree]]\n+How do I sync a working tree across systems?::\n+\tFirst, decide whether you want to do this at all. Git works best when you\n+\tpush or pull your work using the typical `git push` and `git fetch` commands\n+\tand isn't designed to share a working tree across systems. This is\n+\tpotentially risky and in some cases can cause repository corruption or data\n+\tloss.\n++\n+Usually, doing so will cause `git status` to need to re-read every file in the\n+working tree. Additionally, Git's security model does not permit sharing a\n+working tree across untrusted users, so it is only safe to sync a working tree\n+if it will only be used by a single user across all machines.\n++\n+It is important not to use a cloud syncing service to sync any portion of a Git\n+repository, since this can cause corruption, such as missing objects, changed\n+or added files, broken refs, and a wide variety of other corruption. These\n+services tend to sync file by file on a continuous basis and don't understand\n+the structure of a Git repository. This is especially bad if they sync the\n+repository in the middle of it being updated, since that is very likely to\n+cause incomplete or partial updates and therefore data loss.\n++\n+Therefore, it's better to push your work to either the other system or a central\n+server using the normal push and pull mechanism. However, this doesn't always\n+preserve important data, like stashes, so some people prefer to share a working\n+tree across systems.\n++\n+If you do this, the recommended approach is to use `rsync -a --delete-after`\n+(ideally with an encrypted connection such as with `ssh`) on the root of\n+repository. You should ensure several things when you do this:\n++\n+* If you have additional worktrees or a separate Git directory, they must be\n+ synced at the same time as the main working tree and repository.\n+* You are comfortable with the destination directory being an exact copy of the\n+ source directory, _deleting any data that is already there_.\n+* The repository (including all worktrees and the Git directory) is in a\n+ quiescent state for the duration of the transfer (that is, no operations of\n+ any sort are taking place on it, including background operations like `git\n+ gc` and operations invoked by your editor).\n++\n+Be aware that even with these recommendations, syncing in this way has some risk\n+since it bypasses Git's normal integrity checking for repositories, so having\n+backups is advised. You may also wish to do a `git fsck` to verify the\n+integrity of your data on the destination system after syncing.\n+\n Common Issues\n -------------\n \n"},{"id":"498069","messageId":"20240704003818.750223-2-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 1/4] gitfaq: add documentation on proxies","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T00:38:15Z","receivedAt":"2024-07-04T00:40:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Many corporate environments and local systems have proxies in use. Note\nthe situations in which proxies can be used and how to configure them.\nAt the same time, note what standards a proxy must follow to work with\nGit. Explicitly call out certain classes that are known to routinely\nhave problems reported various places online, including in the Git for\nWindows issue tracker and on Stack Overflow, and recommend against the\nuse of such software, noting that they are associated with myriad\nsecurity problems (including, for example, breaking sandboxing and image\nintegrity[0], and, for TLS middleboxes, the use of insecure protocols\nand ciphers and lack of certificate verification[1]). Don't mention the\nspecific nature of these security problems in the FAQ entry because they\nare extremely numerous and varied and we wish to keep the FAQ entry\nrelatively brief.\n\n[0] https://issues.chromium.org/issues/40285192\n[1] https://faculty.cc.gatech.edu/~mbailey/publications/ndss17_interception.pdf\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex 8c1f2d5675..e4125b1178 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -241,6 +241,42 @@ How do I know if I want to do a fetch or a pull?::\n \tignore the upstream changes. A pull consists of a fetch followed\n \timmediately by either a merge or rebase. See linkgit:git-pull[1].\n \n+[[proxy]]\n+Can I use a proxy with Git?::\n+\tYes, Git supports the use of proxies. Git honors the standard `http_proxy`,\n+\t`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and\n+\tit also can be configured with `http.proxy` and similar options for HTTPS (see\n+\tlinkgit:git-config[1]). The `http.proxy` and related options can be\n+\tcustomized on a per-URL pattern basis. In addition, Git can in theory\n+\tfunction normally with transparent proxies that exist on the network.\n++\n+For SSH, Git can support a proxy using OpenSSH's `ProxyCommand`. Commonly used\n+tools include `netcat` and `socat`. However, they must be configured not to\n+exit when seeing EOF on standard input, which usually means that `netcat` will\n+require `-q` and `socat` will require a timeout with something like `-t 10`.\n+This is required because the way the Git SSH server knows that no more requests\n+will be made is an EOF on standard input, but when that happens, the server may\n+not have yet processed the final request, so dropping the connection at that\n+point would interrupt that request.\n++\n+An example configuration entry in `~/.ssh/config` with an HTTP proxy might look\n+like this:\n++\n+----\n+Host git.example.org\n+ User git\n+ ProxyCommand socat -t 10 - PROXY:proxy.example.org:%h:%p,proxyport=8080\n+----\n++\n+Note that in all cases, for Git to work properly, the proxy must be completely\n+transparent. The proxy cannot modify, tamper with, or buffer the connection in\n+any way, or Git will almost certainly fail to work. Note that many proxies,\n+including many TLS middleboxes, Windows antivirus and firewall programs other\n+than Windows Defender and Windows Firewall, and filtering proxies fail to meet\n+this standard, and as a result end up breaking Git. Because of the many\n+reports of problems and their poor security history, we recommend against the\n+use of these classes of software and devices.\n+\n Merging and Rebasing\n --------------------\n \n"},{"id":"498070","messageId":"20240704003818.750223-5-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 4/4] doc: mention that proxies must be completely transparent","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T00:38:18Z","receivedAt":"2024-07-04T00:40:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We already document in the FAQ that proxies must be completely\ntransparent and not modify the request or response in any way, but add\nsimilar documentation to the http.proxy entry. We know that while the\nFAQ is very useful, users sometimes are less likely to read in favor of\nthe documentation specific to an option or command, so adding it in both\nplaces will help users be adequately informed.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/config/http.txt | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/Documentation/config/http.txt b/Documentation/config/http.txt\nindex 2d4e0c9b86..a9c7480f6a 100644\n--- a/Documentation/config/http.txt\n+++ b/Documentation/config/http.txt\n@@ -7,6 +7,11 @@ http.proxy::\n \tlinkgit:gitcredentials[7] for more information. The syntax thus is\n \t'[protocol://][user[:password]@]proxyhost[:port]'. This can be overridden\n \ton a per-remote basis; see remote..proxy\n++\n+Any proxy, however configured, must be completely transparent and must not\n+modify, transform, or buffer the request or response in any way. Proxies which\n+are not completely transparent are known to cause various forms of breakage\n+with Git.\n \n http.proxyAuthMethod::\n \tSet the method with which to authenticate against the HTTP proxy. This\n"},{"id":"498071","messageId":"20240704003818.750223-1-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":null,"subject":"[PATCH v3 0/4] Additional FAQ entries","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T00:38:14Z","receivedAt":"2024-07-04T00:40:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series introduces some additional Git FAQ entries on various\ntopics. They are all things I've seen in my professional life or on\nStack Overflow, so I've written documentation.\n\nThere were some suggestions in the past that the text \"modify, tamper\nwith, or buffer\" might be somewhat redundant, but I've chosen to keep\nthe text as it is to avoid arguments like, \"Well, buffering the entire\nrequest or response isn't really modifying it, so Git should just work\nin that situation,\" when we already know that doesn't work.\n\nChanges from v2 (partial):\n* Add documentation on proxies to the configuration documentation as\n well.\n* Mention some security problems that are known to occur with TLS MITM\n proxies. This mirrors the similar Git LFS documentation.\n* Provide a documentation example about how to use proxies with SSH.\n* Recommend running a `git fsck` after syncing with rsync.\n\nChanges from v1:\n* Drop the monorepo patch for now; I want to revise it further.\n* Reorder the working tree patch to place more warnings up front.\n* Mention core.gitproxy and socat.\n* Rephrase text in the EOL entry to read correctly and be easier to\n understand.\n* Improve the commit message for the working tree FAQ entry to make it\n clearer that users wish to transfer uncommitted changes.\n\nbrian m. carlson (4):\n gitfaq: add documentation on proxies\n gitfaq: give advice on using eol attribute in gitattributes\n gitfaq: add entry about syncing working trees\n doc: mention that proxies must be completely transparent\n\n Documentation/config/http.txt | 5 ++\n Documentation/gitfaq.txt | 105 ++++++++++++++++++++++++++++++++--\n 2 files changed, 104 insertions(+), 6 deletions(-)\n\n"},{"id":"498072","messageId":"xmqqh6d69cdk.fsf@gitster.g","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-04T01:25:43Z","receivedAt":"2024-07-04T01:25:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> This series introduces some additional Git FAQ entries on various\n> topics. They are all things I've seen in my professional life or on\n> Stack Overflow, so I've written documentation.\n\nJust to help other readers:\n\n v1\n https://lore.kernel.org/git/20211020010624.675562-1-sandals@crustytoothpaste.net/\n\n v2\n https://lore.kernel.org/git/20211107225525.431138-1-sandals@crustytoothpaste.net/\n\nare where the previous discussions are found.\n\n> There were some suggestions in the past that the text \"modify, tamper\n> with, or buffer\" might be somewhat redundant, but I've chosen to keep\n> the text as it is to avoid arguments like, \"Well, buffering the entire\n> request or response isn't really modifying it, so Git should just work\n> in that situation,\" when we already know that doesn't work.\n>\n> Changes from v2 (partial):\n> * Add documentation on proxies to the configuration documentation as\n> well.\n> * Mention some security problems that are known to occur with TLS MITM\n> proxies. This mirrors the similar Git LFS documentation.\n> * Provide a documentation example about how to use proxies with SSH.\n> * Recommend running a `git fsck` after syncing with rsync.\n>\n> Changes from v1:\n> * Drop the monorepo patch for now; I want to revise it further.\n> * Reorder the working tree patch to place more warnings up front.\n> * Mention core.gitproxy and socat.\n> * Rephrase text in the EOL entry to read correctly and be easier to\n> understand.\n> * Improve the commit message for the working tree FAQ entry to make it\n> clearer that users wish to transfer uncommitted changes.\n>\n> brian m. carlson (4):\n> gitfaq: add documentation on proxies\n> gitfaq: give advice on using eol attribute in gitattributes\n> gitfaq: add entry about syncing working trees\n> doc: mention that proxies must be completely transparent\n>\n> Documentation/config/http.txt | 5 ++\n> Documentation/gitfaq.txt | 105 ++++++++++++++++++++++++++++++++--\n> 2 files changed, 104 insertions(+), 6 deletions(-)\n"},{"id":"498073","messageId":"xmqqjzi191fw.fsf@gitster.g","threadId":"61732","inReplyTo":"20240704003818.750223-4-sandals@crustytoothpaste.net","subject":"Re: [PATCH v3 3/4] gitfaq: add entry about syncing working trees","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-04T05:21:55Z","receivedAt":"2024-07-04T05:21:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> -Credentials\n> ------------\n> +Credentials and Transfers\n> +-------------------------\n\nI can see (and appreciate) that you struggled to find a good section\nto piggyback on, instead of giving this topic its own section. But\ndo these two make a good mix? They seem to be totally different\ntopics.\n\n> +It is important not to use a cloud syncing service to sync any portion of a Git\n> +repository, since this can cause corruption, such as missing objects, changed\n> +or added files, broken refs, and a wide variety of other corruption. These\n> +services tend to sync file by file on a continuous basis and don't understand\n> +the structure of a Git repository. This is especially bad if they sync the\n> +repository in the middle of it being updated, since that is very likely to\n> +cause incomplete or partial updates and therefore data loss.\n\nA naïve reader may say \"but isn't it the point of these cloud\nsyncing service that they will eventually catch up???\" and we may\nwant to have a good story why it does not work.\n\n You create many objects in one repository in loose form, cloud\n syncing service kicks in to transfer them to the second\n repository, and then in the original repository an auto-gc kicks\n in so some of the loose objects fail to propagate. The packfile\n that is the result of auto-gc will eventually propagate to the\n second repository, but before it completes, the second\n repository would be in an inconsistent state, and especially if\n the ref updates are propagated before objects, then the second\n repository will be in a corrupt state. It would be a disaster\n if another auto-gc kicked in there.\n\nis one scenario I came up with.\n\n\n"},{"id":"498074","messageId":"xmqqcynt91fe.fsf@gitster.g","threadId":"61732","inReplyTo":"20240704003818.750223-3-sandals@crustytoothpaste.net","subject":"Re: [PATCH v3 2/4] gitfaq: give advice on using eol attribute in gitattributes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-04T05:22:13Z","receivedAt":"2024-07-04T05:22:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> In the FAQ, we tell people how to use the text attribute, but we fail to\n> explain what to do with the eol attribute. As we ourselves have\n> noticed, most shell implementations do not care for carriage returns,\n> and as such, people will practically always want them to use LF endings.\n> Similar things can be said for batch files on Windows, except with CRLF\n> endings.\n\nSounds good.\n\n> Since these are common things to have in a repository, let's help users\n> make a good decision by recommending that they use the gitattributes\n> file to correctly check out the endings.\n>\n> In addition, let's correct the cross-reference to this question, which\n> originally referred to \"the following entry\", even though a new entry\n> has been inserted in between. The cross-reference notation should\n> prevent this from occurring and provide a link in formats, such as HTML,\n> which support that.\n\nThanks for being forward-looking and extra careful.\n\n> +With text files, Git will generally ensure that LF endings are used in the\n> +repository, and will honor `core.autocrlf` and `core.eol` to decide what options\n> +to use when checking files out. You can also override this by specifying a\n> +particular line ending such as `eol=lf` or `eol=crlf` if those files must always\n\n\"this\" being ... Not what gets stored in the object database but\nwhat is done to the working tree.\n\nWhat is being \"overridden\" is that the earlier two mentioned here\nare configuration variables that apply to _all_ text files in\ngeneral, and the attribute mechanism is a way to give settings that\nare more tailored for each path. I think the reason I found the\nabove a bit hard to understand when I read it for the first time was\nbecause it didn't \"click\" that this paragraph was about configuration\ngiving the general default and attributes overriding it. Perhaps...\n\n ... are used in the repository. The `core.autocrlf` and\n `core.eol` configuration variables specify what line-ending\n convention is followed when any text file is checked out. You\n can also use the `eol` attribute (e.g., \"eol=crlf\") to override\n which files get what line-ending treatment.\n\nor something?\n\n> +have that ending in the working tree (e.g., for functionality reasons).\n\nI'd strike \"(e.g., for functionality reasons)\" out, as the next\nparagraph makes it sufficiently clear.\n\n> +For example, generally shell files must have LF endings and batch files must\n> +have CRLF endings, so the following might be appropriate in some projects:\n> +\n> ----\n> # By default, guess.\n> *\ttext=auto\n> # Mark all C files as text.\n> *.c\ttext\n> +# Ensure all shell files have LF endings and all batch files have CRLF\n> +# endings in the working tree and both have LF in the repo.\n> +*.sh text eol=lf\n> +*.bat text eol=crlf\n> # Mark all JPEG files as binary.\n> *.jpg\tbinary\n> ----\n"},{"id":"498075","messageId":"xmqqzfqx7muk.fsf@gitster.g","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-04T05:22:27Z","receivedAt":"2024-07-04T05:22:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> This series introduces some additional Git FAQ entries on various\n> topics. They are all things I've seen in my professional life or on\n> Stack Overflow, so I've written documentation.\n>\n> There were some suggestions in the past that the text \"modify, tamper\n> with, or buffer\" might be somewhat redundant, but I've chosen to keep\n> the text as it is to avoid arguments like, \"Well, buffering the entire\n> request or response isn't really modifying it, so Git should just work\n> in that situation,\" when we already know that doesn't work.\n\nBuffering the entire thing will break because ...? Deadlock? Or is\nthere anything more subtle going on?\n\nAre we affected by any frame boundary (do we even notice?) that\nhappens at layer lower than our own pkt-line layer at all (i.e. we\nsent two chunks and we fail to work on them correctly if the network\ncollapses them into one chunk, without changing a single byte, just\nchanging the number of read() system calls that reads them?)?\n"},{"id":"498096","messageId":"ZocPYKyVzSDIekTK@tapette.crustytoothpaste.net","threadId":"61732","inReplyTo":"xmqqjzi191fw.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] gitfaq: add entry about syncing working trees","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T21:08:48Z","receivedAt":"2024-07-04T21:08:51Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-07-04 at 05:21:55, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > -Credentials\n> > ------------\n> > +Credentials and Transfers\n> > +-------------------------\n> \n> I can see (and appreciate) that you struggled to find a good section\n> to piggyback on, instead of giving this topic its own section. But\n> do these two make a good mix? They seem to be totally different\n> topics.\n\nI can try again.\n\n> > +It is important not to use a cloud syncing service to sync any portion of a Git\n> > +repository, since this can cause corruption, such as missing objects, changed\n> > +or added files, broken refs, and a wide variety of other corruption. These\n> > +services tend to sync file by file on a continuous basis and don't understand\n> > +the structure of a Git repository. This is especially bad if they sync the\n> > +repository in the middle of it being updated, since that is very likely to\n> > +cause incomplete or partial updates and therefore data loss.\n> \n> A naïve reader may say \"but isn't it the point of these cloud\n> syncing service that they will eventually catch up???\" and we may\n> want to have a good story why it does not work.\n> \n> You create many objects in one repository in loose form, cloud\n> syncing service kicks in to transfer them to the second\n> repository, and then in the original repository an auto-gc kicks\n> in so some of the loose objects fail to propagate. The packfile\n> that is the result of auto-gc will eventually propagate to the\n> second repository, but before it completes, the second\n> repository would be in an inconsistent state, and especially if\n> the ref updates are propagated before objects, then the second\n> repository will be in a corrupt state. It would be a disaster\n> if another auto-gc kicked in there.\n> \n> is one scenario I came up with.\n\nThe most common situation we see is that refs tend to be renamed to\nthings like \"refs/heads/main 2\", which is obviously not a valid refname\nand doesn't work, or the ref gets rolled back to an older version.\nWorking trees also get stuck into weird states where files keep coming\nback or getting deleted, or the index gets two differently named copies,\nneither of which is \"index\".\n\nIt is _less_ likely that objects are renamed, but it could be that the\ntool thinks they've been legitimately deleted if the loose objects get\npacked and then they do get deleted elsewhere without another source of\nthose objects existing. I'm not sure how object loss happens in the\nreal world with these services, but there have been users reporting it\non StackOverflow, so I'm confident it does occur.\n\nIf we have users who ask about this, I'm happy to answer them on the\nlist. I don't want to explain the various and sundry scenarios in the\nFAQ entry in order to keep it short, but I can find several examples of\nproblems if need be.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"498097","messageId":"ZocPrtrOVTqSeUTY@tapette.crustytoothpaste.net","threadId":"61732","inReplyTo":"xmqqcynt91fe.fsf@gitster.g","subject":"Re: [PATCH v3 2/4] gitfaq: give advice on using eol attribute in gitattributes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T21:10:06Z","receivedAt":"2024-07-04T21:10:09Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-07-04 at 05:22:13, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> > +With text files, Git will generally ensure that LF endings are used in the\n> > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options\n> > +to use when checking files out. You can also override this by specifying a\n> > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always\n> \n> \"this\" being ... Not what gets stored in the object database but\n> what is done to the working tree.\n> \n> What is being \"overridden\" is that the earlier two mentioned here\n> are configuration variables that apply to _all_ text files in\n> general, and the attribute mechanism is a way to give settings that\n> are more tailored for each path. I think the reason I found the\n> above a bit hard to understand when I read it for the first time was\n> because it didn't \"click\" that this paragraph was about configuration\n> giving the general default and attributes overriding it. Perhaps...\n> \n> ... are used in the repository. The `core.autocrlf` and\n> `core.eol` configuration variables specify what line-ending\n> convention is followed when any text file is checked out. You\n> can also use the `eol` attribute (e.g., \"eol=crlf\") to override\n> which files get what line-ending treatment.\n> \n> or something?\n\nSure, that sounds like a nice improvement.\n\n> > +have that ending in the working tree (e.g., for functionality reasons).\n> \n> I'd strike \"(e.g., for functionality reasons)\" out, as the next\n> paragraph makes it sufficiently clear.\n\nSure, I can do that.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"498098","messageId":"ZocS0NgiAbg5Mnzp@tapette.crustytoothpaste.net","threadId":"61732","inReplyTo":"xmqqzfqx7muk.fsf@gitster.g","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-04T21:23:28Z","receivedAt":"2024-07-04T21:23:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-07-04 at 05:22:27, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > This series introduces some additional Git FAQ entries on various\n> > topics. They are all things I've seen in my professional life or on\n> > Stack Overflow, so I've written documentation.\n> >\n> > There were some suggestions in the past that the text \"modify, tamper\n> > with, or buffer\" might be somewhat redundant, but I've chosen to keep\n> > the text as it is to avoid arguments like, \"Well, buffering the entire\n> > request or response isn't really modifying it, so Git should just work\n> > in that situation,\" when we already know that doesn't work.\n> \n> Buffering the entire thing will break because ...? Deadlock? Or is\n> there anything more subtle going on?\n\nWhen we use the smart HTTP protocol, the server sends keep-alive and\nstatus messages as one of the data streams, which is important because\n(a) the user is usually impatient and wants to know what's going on and\n(b) it may take a long time to pack the data, especially for large\nrepositories, and sending no data may result in the connection being\ndropped or the client being served a 500 by an intermediate layer. We\nknow this does happen and I've seen reports of it.\n\nWe've also seen some cases where proxies refuse to accept\nTransfer-Encoding: chunked (let's party like it's 1999) and send a 411\nback since there's no Content-Length header. That's presumably because\nthey want to scan the contents for \"bad\" data all in one chunk, but Git\nhas to stream the contents unless the data fits in the buffer size.\n(This is the one case where http.postBuffer actually makes a\ndifference.) I very much doubt that the appliance actually wants to get\na 2 GiB payload to scan, since it probably doesn't have tons of memory\nin the first place, but that is what it's asking for.\n\n> Are we affected by any frame boundary (do we even notice?) that\n> happens at layer lower than our own pkt-line layer at all (i.e. we\n> sent two chunks and we fail to work on them correctly if the network\n> collapses them into one chunk, without changing a single byte, just\n> changing the number of read() system calls that reads them?)?\n\nNo, that's not a problem. We read four bytes for the pkt-line header,\nand then we read the entire body based on that length until we get all\nof it. This is also the way OpenSSL works for TLS packets and is known\nto work well. If the underlying TCP connection provides a partial or\nincomplete packet (which can happen due to MTU), we'll just block until\nthe rest comes in, which is fine.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"498137","messageId":"xmqqv81j3w8i.fsf@gitster.g","threadId":"61732","inReplyTo":"ZocPYKyVzSDIekTK@tapette.crustytoothpaste.net","subject":"Re: [PATCH v3 3/4] gitfaq: add entry about syncing working trees","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-06T05:50:05Z","receivedAt":"2024-07-06T05:50:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> The most common situation we see is that refs tend to be renamed to\n> things like \"refs/heads/main 2\", which is obviously not a valid refname\n> and doesn't work, or the ref gets rolled back to an older version.\n> Working trees also get stuck into weird states where files keep coming\n> back or getting deleted, or the index gets two differently named copies,\n> neither of which is \"index\".\n>\n> It is _less_ likely that objects are renamed, but it could be that the\n> tool thinks they've been legitimately deleted if the loose objects get\n> packed and then they do get deleted elsewhere without another source of\n> those objects existing.\n\nYeah, any time two repositories that are \"cloud synched\" are\naccessed simultaneously, all h*ll can easily break loose. You may\nmove your 'master' branch to a commit while the other one may move\ntheir 'master' branch to a different commit. You may end up having\n\"master\" that points at one of these commits but one of you may have\nalready lost the only reference to the commit you wanted to have at\nthe tip of your 'master' branch. One of you may even trigger auto-gc\nto spread the damage.\n\n> If we have users who ask about this, I'm happy to answer them on the\n> list. I don't want to explain the various and sundry scenarios in the\n> FAQ entry in order to keep it short, but I can find several examples of\n> problems if need be.\n\nOK, that approach would work as long as you are still involved in\nthe project, but having even one concrete example would help in the\nlonger term to (1) reduce the bus factor and (2) save time you do\nnot have to spend responding to every such question.\n\nThanks.\n"},{"id":"498138","messageId":"xmqqo77b3vs2.fsf@gitster.g","threadId":"61732","inReplyTo":"ZocS0NgiAbg5Mnzp@tapette.crustytoothpaste.net","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-06T05:59:57Z","receivedAt":"2024-07-06T05:59:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n>> Buffering the entire thing will break because ...? Deadlock? Or is\n>> there anything more subtle going on?\n>\n> When we use the smart HTTP protocol, the server sends keep-alive and\n> status messages as one of the data streams, which is important because\n> (a) the user is usually impatient and wants to know what's going on and\n> (b) it may take a long time to pack the data, especially for large\n> repositories, and sending no data may result in the connection being\n> dropped or the client being served a 500 by an intermediate layer. We\n> know this does happen and I've seen reports of it.\n\nAnd this is an example of \"a proxy that buffers the data, without\nmodifying or tampering with, would still break transport\"?\n\n> We've also seen some cases where proxies refuse to accept\n> Transfer-Encoding: chunked (let's party like it's 1999) and send a 411\n> back since there's no Content-Length header.\n\nThis is \"a proxy that wanted to buffer the data but failed to do so\"\nthat ended up modifying the data Gits sitting at both ends of the\nconnection can observe, so it is a bit different issue. It clearly\nfalls into \"modify or tampering with\" category.\n\nI forgot to say this clearly when I wrote the message you are\nresponding to, but I am trying to see if we can clarify the \"or\nbuffer\" part in \"modify, tamper with, or buffer\", as offhand I did\nnot think of a reason why a proxy would break the Git communication\nif it receives a segment that was 2MB originally from upload-pack,\nand forwards the contents of the segment in two 1MB segments without\ntampering or modifying the payload bytes at all to fetch-pack.\n\nThanks.\n"},{"id":"498153","messageId":"20240706064758.GG700645@coredump.intra.peff.net","threadId":"61732","inReplyTo":"ZocS0NgiAbg5Mnzp@tapette.crustytoothpaste.net","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-07-06T06:47:58Z","receivedAt":"2024-07-06T06:48:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 04, 2024 at 09:23:28PM +0000, brian m. carlson wrote:\n\n> > Buffering the entire thing will break because ...? Deadlock? Or is\n> > there anything more subtle going on?\n> \n> When we use the smart HTTP protocol, the server sends keep-alive and\n> status messages as one of the data streams, which is important because\n> (a) the user is usually impatient and wants to know what's going on and\n> (b) it may take a long time to pack the data, especially for large\n> repositories, and sending no data may result in the connection being\n> dropped or the client being served a 500 by an intermediate layer. We\n> know this does happen and I've seen reports of it.\n\nAdditionally, I think for non-HTTP transports (think proxying ssh\nthrough socat or similar), buffering the v0 protocol is likely a total\ndisaster. The fetch protocol assumes both sides spewing at each other in\nreal time.\n\nHTTP, even v0, follows a request/response model, so we're safer there. I\ndo think some amount of buffering is often going to be OK in practice.\nYou'd get delayed keep-alives and progress reports, which may range from\n\"annoying\" to \"something in the middle decided to time out\". So I'm OK\nwith just telling people \"make sure your proxies aren't buffering\" as a\ngeneral rule, rather than trying to get into the nitty gritty of what is\ngoing to break and how.\n\n-Peff\n"},{"id":"498169","messageId":"xmqqjzhyqw17.fsf@gitster.g","threadId":"61732","inReplyTo":"20240706064758.GG700645@coredump.intra.peff.net","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-06T17:18:12Z","receivedAt":"2024-07-06T17:18:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Jul 04, 2024 at 09:23:28PM +0000, brian m. carlson wrote:\n>\n>> > Buffering the entire thing will break because ...? Deadlock? Or is\n>> > there anything more subtle going on?\n>> \n>> When we use the smart HTTP protocol, the server sends keep-alive and\n>> status messages as one of the data streams, which is important because\n>> (a) the user is usually impatient and wants to know what's going on and\n>> (b) it may take a long time to pack the data, especially for large\n>> repositories, and sending no data may result in the connection being\n>> dropped or the client being served a 500 by an intermediate layer. We\n>> know this does happen and I've seen reports of it.\n>\n> Additionally, I think for non-HTTP transports (think proxying ssh\n> through socat or similar), buffering the v0 protocol is likely a total\n> disaster. The fetch protocol assumes both sides spewing at each other in\n> real time.\n\nYeah, beyond one \"window\" that a series of \"have\"s are allowed to be\nin flight, no further \"have\"s are sent before seeing an \"ack/nack\"\nresponse, so if you buffer too much, they can deadlock fairly easily.\n\n> ... So I'm OK\n> with just telling people \"make sure your proxies aren't buffering\" as a\n> general rule, rather than trying to get into the nitty gritty of what is\n> going to break and how.\n\nSounds fair. Thanks.\n"},{"id":"498205","messageId":"Zos4Ta6ddzoBuX-1@tapette.crustytoothpaste.net","threadId":"61732","inReplyTo":"xmqqo77b3vs2.fsf@gitster.g","subject":"Re: [PATCH v3 0/4] Additional FAQ entries","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-08T00:52:29Z","receivedAt":"2024-07-08T00:52:31Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-07-06 at 05:59:57, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> >> Buffering the entire thing will break because ...? Deadlock? Or is\n> >> there anything more subtle going on?\n> >\n> > When we use the smart HTTP protocol, the server sends keep-alive and\n> > status messages as one of the data streams, which is important because\n> > (a) the user is usually impatient and wants to know what's going on and\n> > (b) it may take a long time to pack the data, especially for large\n> > repositories, and sending no data may result in the connection being\n> > dropped or the client being served a 500 by an intermediate layer. We\n> > know this does happen and I've seen reports of it.\n> \n> And this is an example of \"a proxy that buffers the data, without\n> modifying or tampering with, would still break transport\"?\n\nYes. The connection usually ends up dropped from the view of the\nclient, which is hard to debug (because it also looks like a network\nproblem, except often without any output from the remote side).\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"498392","messageId":"20240709233746.445860-1-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240704003818.750223-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 0/4] Additional FAQ entries","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-09T23:37:42Z","receivedAt":"2024-07-09T23:38:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series introduces some additional Git FAQ entries on various\ntopics. They are all things I've seen in my professional life or on\nStack Overflow, so I've written documentation.\n\nChanges from v3:\n* Improve text for eol options in .gitattributes.\n* Split working tree syncing into its own category.\n* Add some more explanation about what might go wrong in such a case.\n* Rephrase some text to avoid repetition.\n\nChanges from v2 (partial):\n* Add documentation on proxies to the configuration documentation as\n well.\n* Mention some security problems that are known to occur with TLS MITM\n proxies. This mirrors the similar Git LFS documentation.\n* Provide a documentation example about how to use proxies with SSH.\n* Recommend running a `git fsck` after syncing with rsync.\n\nChanges from v1:\n* Drop the monorepo patch for now; I want to revise it further.\n* Reorder the working tree patch to place more warnings up front.\n* Mention core.gitproxy and socat.\n* Rephrase text in the EOL entry to read correctly and be easier to\n understand.\n* Improve the commit message for the working tree FAQ entry to make it\n clearer that users wish to transfer uncommitted changes.\n\nbrian m. carlson (4):\n gitfaq: add documentation on proxies\n gitfaq: give advice on using eol attribute in gitattributes\n gitfaq: add entry about syncing working trees\n doc: mention that proxies must be completely transparent\n\n Documentation/config/http.txt | 5 ++\n Documentation/gitfaq.txt | 109 ++++++++++++++++++++++++++++++++--\n 2 files changed, 110 insertions(+), 4 deletions(-)\n\n"},{"id":"498393","messageId":"20240709233746.445860-2-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240709233746.445860-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 1/4] gitfaq: add documentation on proxies","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-09T23:37:43Z","receivedAt":"2024-07-09T23:38:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Many corporate environments and local systems have proxies in use. Note\nthe situations in which proxies can be used and how to configure them.\nAt the same time, note what standards a proxy must follow to work with\nGit. Explicitly call out certain classes that are known to routinely\nhave problems reported various places online, including in the Git for\nWindows issue tracker and on Stack Overflow, and recommend against the\nuse of such software, noting that they are associated with myriad\nsecurity problems (including, for example, breaking sandboxing and image\nintegrity[0], and, for TLS middleboxes, the use of insecure protocols\nand ciphers and lack of certificate verification[1]). Don't mention the\nspecific nature of these security problems in the FAQ entry because they\nare extremely numerous and varied and we wish to keep the FAQ entry\nrelatively brief.\n\n[0] https://issues.chromium.org/issues/40285192\n[1] https://faculty.cc.gatech.edu/~mbailey/publications/ndss17_interception.pdf\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex 8c1f2d5675..e4125b1178 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -241,6 +241,42 @@ How do I know if I want to do a fetch or a pull?::\n \tignore the upstream changes. A pull consists of a fetch followed\n \timmediately by either a merge or rebase. See linkgit:git-pull[1].\n \n+[[proxy]]\n+Can I use a proxy with Git?::\n+\tYes, Git supports the use of proxies. Git honors the standard `http_proxy`,\n+\t`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and\n+\tit also can be configured with `http.proxy` and similar options for HTTPS (see\n+\tlinkgit:git-config[1]). The `http.proxy` and related options can be\n+\tcustomized on a per-URL pattern basis. In addition, Git can in theory\n+\tfunction normally with transparent proxies that exist on the network.\n++\n+For SSH, Git can support a proxy using OpenSSH's `ProxyCommand`. Commonly used\n+tools include `netcat` and `socat`. However, they must be configured not to\n+exit when seeing EOF on standard input, which usually means that `netcat` will\n+require `-q` and `socat` will require a timeout with something like `-t 10`.\n+This is required because the way the Git SSH server knows that no more requests\n+will be made is an EOF on standard input, but when that happens, the server may\n+not have yet processed the final request, so dropping the connection at that\n+point would interrupt that request.\n++\n+An example configuration entry in `~/.ssh/config` with an HTTP proxy might look\n+like this:\n++\n+----\n+Host git.example.org\n+ User git\n+ ProxyCommand socat -t 10 - PROXY:proxy.example.org:%h:%p,proxyport=8080\n+----\n++\n+Note that in all cases, for Git to work properly, the proxy must be completely\n+transparent. The proxy cannot modify, tamper with, or buffer the connection in\n+any way, or Git will almost certainly fail to work. Note that many proxies,\n+including many TLS middleboxes, Windows antivirus and firewall programs other\n+than Windows Defender and Windows Firewall, and filtering proxies fail to meet\n+this standard, and as a result end up breaking Git. Because of the many\n+reports of problems and their poor security history, we recommend against the\n+use of these classes of software and devices.\n+\n Merging and Rebasing\n --------------------\n \n"},{"id":"498394","messageId":"20240709233746.445860-4-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240709233746.445860-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 3/4] gitfaq: add entry about syncing working trees","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-09T23:37:45Z","receivedAt":"2024-07-09T23:38:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Users very commonly want to sync their working tree with uncommitted\nchanges across machines, often to carry across in-progress work or\nstashes. Despite this not being a recommended approach, users want to\ndo it and are not dissuaded by suggestions not to, so let's recommend a\nsensible technique.\n\nThe technique that many users are using is their preferred cloud syncing\nservice, which is a bad idea. Users have reported problems where they\nend up with duplicate files that won't go away (with names like \"file.c\n2\"), broken references, oddly named references that have date stamps\nappended to them, missing objects, and general corruption and data loss.\nThat's because almost all of these tools sync file by file, which is a\ngreat technique if your project is a single word processing document or\nspreadsheet, but is utterly abysmal for Git repositories because they\ndon't necessarily snapshot the entire repository correctly. They also\ntend to sync the files immediately instead of when the repository is\nquiescent, so writing multiple files, as occurs during a commit or a gc,\ncan confuse the tools and lead to corruption.\n\nWe know that the old standby, rsync, is up to the task, provided that\nthe repository is quiescent, so let's suggest that and dissuade people\nfrom using cloud syncing tools. Let's tell people about common things\nthey should be aware of before doing this and that this is still\npotentially risky. Additionally, let's tell people that Git's security\nmodel does not permit sharing working trees across users in case they\nplanned to do that. While we'd still prefer users didn't try to do\nthis, hopefully this will lead them in a safer direction.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 52 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex 058ef32a97..f2917d142c 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -185,6 +185,58 @@ Then, you can adjust your push URL to use `git@example_author` or\n `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url\n git@example_author:org1/project1.git`).\n \n+Transfers\n+---------\n+\n+[[sync-working-tree]]\n+How do I sync a working tree across systems?::\n+\tFirst, decide whether you want to do this at all. Git works best when you\n+\tpush or pull your work using the typical `git push` and `git fetch` commands\n+\tand isn't designed to share a working tree across systems. This is\n+\tpotentially risky and in some cases can cause repository corruption or data\n+\tloss.\n++\n+Usually, doing so will cause `git status` to need to re-read every file in the\n+working tree. Additionally, Git's security model does not permit sharing a\n+working tree across untrusted users, so it is only safe to sync a working tree\n+if it will only be used by a single user across all machines.\n++\n+It is important not to use a cloud syncing service to sync any portion of a Git\n+repository, since this can cause corruption, such as missing objects, changed\n+or added files, broken refs, and a wide variety of other problems. These\n+services tend to sync file by file on a continuous basis and don't understand\n+the structure of a Git repository. This is especially bad if they sync the\n+repository in the middle of it being updated, since that is very likely to\n+cause incomplete or partial updates and therefore data loss.\n++\n+An example of the kind of corruption that can occur is conflicts over the state\n+of refs, such that both sides end up with different commits on a branch that\n+the other doesn't have. This can result in important objects becoming\n+unreferenced and possibly pruned by `git gc`, causing data loss.\n++\n+Therefore, it's better to push your work to either the other system or a central\n+server using the normal push and pull mechanism. However, this doesn't always\n+preserve important data, like stashes, so some people prefer to share a working\n+tree across systems.\n++\n+If you do this, the recommended approach is to use `rsync -a --delete-after`\n+(ideally with an encrypted connection such as with `ssh`) on the root of\n+repository. You should ensure several things when you do this:\n++\n+* If you have additional worktrees or a separate Git directory, they must be\n+ synced at the same time as the main working tree and repository.\n+* You are comfortable with the destination directory being an exact copy of the\n+ source directory, _deleting any data that is already there_.\n+* The repository (including all worktrees and the Git directory) is in a\n+ quiescent state for the duration of the transfer (that is, no operations of\n+ any sort are taking place on it, including background operations like `git\n+ gc` and operations invoked by your editor).\n++\n+Be aware that even with these recommendations, syncing in this way has some risk\n+since it bypasses Git's normal integrity checking for repositories, so having\n+backups is advised. You may also wish to do a `git fsck` to verify the\n+integrity of your data on the destination system after syncing.\n+\n Common Issues\n -------------\n \n"},{"id":"498395","messageId":"20240709233746.445860-5-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240709233746.445860-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 4/4] doc: mention that proxies must be completely transparent","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-09T23:37:46Z","receivedAt":"2024-07-09T23:38:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We already document in the FAQ that proxies must be completely\ntransparent and not modify the request or response in any way, but add\nsimilar documentation to the http.proxy entry. We know that while the\nFAQ is very useful, users sometimes are less likely to read in favor of\nthe documentation specific to an option or command, so adding it in both\nplaces will help users be adequately informed.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/config/http.txt | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/Documentation/config/http.txt b/Documentation/config/http.txt\nindex 2d4e0c9b86..a9c7480f6a 100644\n--- a/Documentation/config/http.txt\n+++ b/Documentation/config/http.txt\n@@ -7,6 +7,11 @@ http.proxy::\n \tlinkgit:gitcredentials[7] for more information. The syntax thus is\n \t'[protocol://][user[:password]@]proxyhost[:port]'. This can be overridden\n \ton a per-remote basis; see remote..proxy\n++\n+Any proxy, however configured, must be completely transparent and must not\n+modify, transform, or buffer the request or response in any way. Proxies which\n+are not completely transparent are known to cause various forms of breakage\n+with Git.\n \n http.proxyAuthMethod::\n \tSet the method with which to authenticate against the HTTP proxy. This\n"},{"id":"498396","messageId":"20240709233746.445860-3-sandals@crustytoothpaste.net","threadId":"61732","inReplyTo":"20240709233746.445860-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 2/4] gitfaq: give advice on using eol attribute in gitattributes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-07-09T23:37:44Z","receivedAt":"2024-07-09T23:38:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In the FAQ, we tell people how to use the text attribute, but we fail to\nexplain what to do with the eol attribute. As we ourselves have\nnoticed, most shell implementations do not care for carriage returns,\nand as such, people will practically always want them to use LF endings.\nSimilar things can be said for batch files on Windows, except with CRLF\nendings.\n\nSince these are common things to have in a repository, let's help users\nmake a good decision by recommending that they use the gitattributes\nfile to correctly check out the endings.\n\nIn addition, let's correct the cross-reference to this question, which\noriginally referred to \"the following entry\", even though a new entry\nhas been inserted in between. The cross-reference notation should\nprevent this from occurring and provide a link in formats, such as HTML,\nwhich support that.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitfaq.txt | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt\nindex e4125b1178..058ef32a97 100644\n--- a/Documentation/gitfaq.txt\n+++ b/Documentation/gitfaq.txt\n@@ -393,8 +393,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::\n +\n You can store the files in the repository with Unix line endings and convert\n them automatically to your platform's line endings. To do that, set the\n-configuration option `core.eol` to `native` and see the following entry for\n-information about how to configure files as text or binary.\n+configuration option `core.eol` to `native` and see\n+<>\n+for information about how to configure files as text or binary.\n +\n You can also control this behavior with the `core.whitespace` setting if you\n don't wish to remove the carriage returns from your line endings.\n@@ -456,14 +457,26 @@ references, URLs, and hashes stored in the repository.\n +\n We also recommend setting a linkgit:gitattributes[5] file to explicitly mark\n which files are text and which are binary. If you want Git to guess, you can\n-set the attribute `text=auto`. For example, the following might be appropriate\n-in some projects:\n+set the attribute `text=auto`.\n++\n+With text files, Git will generally ensure that LF endings are used in the\n+repository. The `core.autocrlf` and `core.eol` configuration variables specify\n+what line-ending convention is followed when any text file is checked out. You\n+can also use the `eol` attribute (e.g., `eol=crlf`) to override which files get\n+what line-ending treatment.\n++\n+For example, generally shell files must have LF endings and batch files must\n+have CRLF endings, so the following might be appropriate in some projects:\n +\n ----\n # By default, guess.\n *\ttext=auto\n # Mark all C files as text.\n *.c\ttext\n+# Ensure all shell files have LF endings and all batch files have CRLF\n+# endings in the working tree and both have LF in the repo.\n+*.sh text eol=lf\n+*.bat text eol=crlf\n # Mark all JPEG files as binary.\n *.jpg\tbinary\n ----\n"}]}