{"thread":{"id":"55487","subject":"Git via MITM transparent proxy with HTTPS Interception","startedAt":"2021-04-13T12:08:15Z","lastAt":"2021-04-14T15:41:28Z","messageCount":7,"participants":["Vitaly VS","Jason Pyeron","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"421880","messageId":"CAEaE=iyUGiPK-HX850mEgC=X6atEhbjJ0dCK0dci0nOCahPhgQ@mail.gmail.com","threadId":"55487","inReplyTo":null,"subject":"Git via MITM transparent proxy with HTTPS Interception","fromName":"Vitaly VS","fromEmail":"strikervitaly@gmail.com","sentAt":"2021-04-13T12:07:58Z","receivedAt":"2021-04-13T12:08:15Z","isPatch":false,"sender":{"key":"strikervitaly@gmail.com","avatar":"https://gravatar.com/avatar/435f0699054eca8896c8de6c1ea94deabd74d91cfe106e226441c39e9beb5ea0?d=mp&s=160"},"body":"Hello! Can a Git client work properly through a MITM transparent proxy\nwith HTTPS interception?\n\nIs there any documentation or recommendations on how to configure a\nMITM proxy with HTTPS interception for the Git work?\n\nGetting a bunch of errors when trying to \"git clone https://SOME_REPO.git\"\nOn small REPOs (about 1-5 MB) there is a chance that the clone will be\nsuccessful, but mostly I get these errors:\n\ngit clone https://github.com/aaptel/wireshark.git\nCloning into 'wireshark'...\nremote: Enumerating objects: 524729, done.\nfatal: protocol error: bad line length character: ??:s00 KiB/s\nerror: inflate: data stream error (invalid literal/lengths set)\nfatal: pack has bad object at offset 2093488: inflate returned -3\nfatal: index-pack failed\n\ngit clone https://github.com/aaptel/wireshark.git\nCloning into 'wireshark'...\nremote: Enumerating objects: 524729, done.\nfatal: protocol error: bad line length character: ????06 MiB/s\nerror: inflate: data stream error (incorrect data check)\nfatal: pack has bad object at offset 17119052: inflate returned -3\nfatal: index-pack failed\n\n\ngit clone https://github.com/aaptel/wireshark.git\nCloning into 'wireshark'...\nremote: Enumerating objects: 524729, done.\nerror: RPC failed; curl 56 Malformed encoding found in chunked-encoding\nfatal: the remote end hung up unexpectedly\nfatal: early EOF\nfatal: index-pack failed\n\ngit clone https://github.com/Homebrew/brew.git\nCloning into 'brew'...\nremote: Enumerating objects: 148, done.\nremote: Counting objects: 100% (148/148), done.\nremote: Compressing objects: 100% (80/80), done.\nReceiving objects:   3% (6247/180213), 2.64 MiB | 1005.00 KiB/s\nReceiving objects:   4% (8247/180213), 3.75 MiB | 1.00 MiB/s\nReceiving objects:   5% (9011/180213), 4.47 MiB | 1.05 MiB/s\nfatal: protocol error: bad line length character: ?V?V7 MiB/s\nerror: inflate: data stream error (incorrect data check)\nfatal: pack has bad object at offset 6558416: inflate returned -3\nfatal: index-pack failed\nerror: RPC failed; curl 56 Malformed encoding found in chunked-encoding\n\ngit clone https://github.com/Homebrew/brew.git\nCloning into 'brew'...\nremote: Enumerating objects: 148, done.\nremote: Counting objects: 100% (148/148), done.\nremote: Compressing objects: 100% (80/80), done.\nReceiving objects:   0% (1/180213)\nReceiving objects:   0% (687/180213), 436.01 KiB | 397.00 KiB/s\nReceiving objects:   0% (1029/180213), 548.01 KiB | 338.00 KiB/s\nReceiving objects:   1% (1803/180213), 972.01 KiB | 309.00 KiB/s\nReceiving objects:   1% (2091/180213), 1.11 MiB | 309.00 KiB/s\nReceiving objects:   2% (3605/180213), 1.82 MiB | 214.00 KiB/s\nfatal: protocol error: bad line length character: O20000 KiB/s\nfatal: pack has bad object at offset 2776352: inflate returned -5\nfatal: index-pack failed\nerror: RPC failed; curl 56 Malformed encoding found in chunked-encoding\n\nP.S. We trust proxy root certificate in the system, also tried to add\nin config but no luck\n"},{"id":"421889","messageId":"01d301d7305f$e4acdf80$ae069e80$@pdinc.us","threadId":"55487","inReplyTo":"CAEaE=iyUGiPK-HX850mEgC=X6atEhbjJ0dCK0dci0nOCahPhgQ@mail.gmail.com","subject":"RE: Git via MITM transparent proxy with HTTPS Interception","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2021-04-13T12:24:04Z","receivedAt":"2021-04-13T12:58:17Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> From: Vitaly VS\n> Sent: Tuesday, April 13, 2021 8:08 AM\n> \n> Hello! Can a Git client work properly through a MITM transparent proxy\n> with HTTPS interception?\n\nYes, we do it all the time.\n\n> \n> Is there any documentation or recommendations on how to configure a\n> MITM proxy with HTTPS interception for the Git work?\n> \n\nNot that I am aware of. It is not a Git issue per se. The WAF or Proxy should not (appear) to alter any of the contents of the stream (when allowed).\n\n> Getting a bunch of errors when trying to \"git clone https://SOME_REPO.git\"\n> On small REPOs (about 1-5 MB) there is a chance that the clone will be\n> successful, but mostly I get these errors:\n> \n\nIt is likely off-topic, but what is your proxy configuration? I have personally used Git through Apache and F5 MITM proxies.\n\n> git clone https://github.com/aaptel/wireshark.git\n> Cloning into 'wireshark'...\n> remote: Enumerating objects: 524729, done.\n> fatal: protocol error: bad line length character: ??:s00 KiB/s\n> error: inflate: data stream error (invalid literal/lengths set)\n> fatal: pack has bad object at offset 2093488: inflate returned -3\n> fatal: index-pack failed\n\nEnable git and curl tracing, contact your proxy team and ask for packet capture with decryption.\n\n> \n> git clone https://github.com/aaptel/wireshark.git\n> Cloning into 'wireshark'...\n> remote: Enumerating objects: 524729, done.\n> fatal: protocol error: bad line length character: ????06 MiB/s\n> error: inflate: data stream error (incorrect data check)\n> fatal: pack has bad object at offset 17119052: inflate returned -3\n> fatal: index-pack failed\n> \n> \n> git clone https://github.com/aaptel/wireshark.git\n> Cloning into 'wireshark'...\n> remote: Enumerating objects: 524729, done.\n> error: RPC failed; curl 56 Malformed encoding found in chunked-encoding\n> fatal: the remote end hung up unexpectedly\n> fatal: early EOF\n> fatal: index-pack failed\n> \n> git clone https://github.com/Homebrew/brew.git\n> Cloning into 'brew'...\n> remote: Enumerating objects: 148, done.\n> remote: Counting objects: 100% (148/148), done.\n> remote: Compressing objects: 100% (80/80), done.\n> Receiving objects:   3% (6247/180213), 2.64 MiB | 1005.00 KiB/s\n> Receiving objects:   4% (8247/180213), 3.75 MiB | 1.00 MiB/s\n> Receiving objects:   5% (9011/180213), 4.47 MiB | 1.05 MiB/s\n> fatal: protocol error: bad line length character: ?V?V7 MiB/s\n> error: inflate: data stream error (incorrect data check)\n> fatal: pack has bad object at offset 6558416: inflate returned -3\n> fatal: index-pack failed\n> error: RPC failed; curl 56 Malformed encoding found in chunked-encoding\n> \n> git clone https://github.com/Homebrew/brew.git\n> Cloning into 'brew'...\n> remote: Enumerating objects: 148, done.\n> remote: Counting objects: 100% (148/148), done.\n> remote: Compressing objects: 100% (80/80), done.\n> Receiving objects:   0% (1/180213)\n> Receiving objects:   0% (687/180213), 436.01 KiB | 397.00 KiB/s\n> Receiving objects:   0% (1029/180213), 548.01 KiB | 338.00 KiB/s\n> Receiving objects:   1% (1803/180213), 972.01 KiB | 309.00 KiB/s\n> Receiving objects:   1% (2091/180213), 1.11 MiB | 309.00 KiB/s\n> Receiving objects:   2% (3605/180213), 1.82 MiB | 214.00 KiB/s\n> fatal: protocol error: bad line length character: O20000 KiB/s\n> fatal: pack has bad object at offset 2776352: inflate returned -5\n> fatal: index-pack failed\n> error: RPC failed; curl 56 Malformed encoding found in chunked-encoding\n> \n> P.S. We trust proxy root certificate in the system, also tried to add\n> in config but no luck\n\nThat is assumed, otherwise you would not have started transferring any data.\n\n[I set the reply to header, don’t email me directly I am on the list]\n\n--\nJason Pyeron  | Architect\nContractor    |\nPD Inc        |\n10 w 24th St  |\nBaltimore, MD |\n\n.mil: jason.j.pyeron.ctr@mail.mil\n.com: jpyeron@pdinc.us\ntel : 202-741-9397\n\n\n\n"},{"id":"421940","messageId":"YHYxtvKgKz+Uv2xO@camp.crustytoothpaste.net","threadId":"55487","inReplyTo":"CAEaE=iyUGiPK-HX850mEgC=X6atEhbjJ0dCK0dci0nOCahPhgQ@mail.gmail.com","subject":"Re: Git via MITM transparent proxy with HTTPS Interception","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-04-14T00:05:10Z","receivedAt":"2021-04-14T00:05:49Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-04-13 at 12:07:58, Vitaly VS wrote:\n> Hello! Can a Git client work properly through a MITM transparent proxy\n> with HTTPS interception?\n\nYes, with some important caveats.  The proxy must be completely\ntransparent.  It must not modify or impede the data in any way, it must\nspeak both HTTP 1.1 and HTTP 2 correctly and fully, and the proxy must\nspeak TLS completely correctly, including terminating the connection in\naccordance with the protocol.  It must be completely impossible to tell\nthat a proxy is being used.\n\nI do want to point out that TLS interception is by definition a security\nvulnerability and almost always significantly weakens security, often by\nusing weaker protocols, breaking or disabling certificate verification,\nand impeding the upgrading and interoperability of the protocol[0].  You\nshould definitely read and understand the literature about TLS\nintercepting proxies and have personally verified that your\nimplementation is free of vulnerabilities before deploying.  You\nshouldn't rely on your implementer for this information, because they\nusually aren't aware that their implementation has vulnerabilities.\n\nAlso, my experience is that many, many proxies of this nature are\ncompletely broken and don't work correctly, so if you are not fully\naware of what's going on and haven't fully tested your implementation,\nyou shouldn't deploy this technology.  I frequently answer questions\nfrom users in scenarios such as this who are having problems due to a\nbroken proxy and often have to tell them to contact their network\nadministrator.  I therefore do not in any sense recommend deploying such\ninfrastructure.\n\nGit really does need a properly functioning HTTP and TLS implementation\nand things tend to break in a variety of ways when encountering broken\nproxies.  I would say \"exciting ways\", but I recognize them all now and\nthey're not exciting anymore.\n\n> Getting a bunch of errors when trying to \"git clone https://SOME_REPO.git\"\n> On small REPOs (about 1-5 MB) there is a chance that the clone will be\n> successful, but mostly I get these errors:\n\nYour proxy is broken and doesn't speak the protocol correctly.  It isn't\na transparent proxy.  You should either remove it or contact your\nnetwork administrator to have it removed.\n\n[0] https://www.ftc.gov/system/files/documents/public_comments/2016/09/00019-129028.pdf\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"421958","messageId":"CAEaE=izSNyxRcvMd5bArHnmi0F2G83nouge9e_qxiQmA0AsWog@mail.gmail.com","threadId":"55487","inReplyTo":"YHYxtvKgKz+Uv2xO@camp.crustytoothpaste.net","subject":"Re: Git via MITM transparent proxy with HTTPS Interception","fromName":"Vitaly VS","fromEmail":"strikervitaly@gmail.com","sentAt":"2021-04-14T09:35:48Z","receivedAt":"2021-04-14T09:36:02Z","isPatch":false,"sender":{"key":"strikervitaly@gmail.com","avatar":"https://gravatar.com/avatar/435f0699054eca8896c8de6c1ea94deabd74d91cfe106e226441c39e9beb5ea0?d=mp&s=160"},"body":"Thank you for the fast response.\n\nAbout our network environment\nWe are Cisco WSA(Servers SW, ASA, ISR) that proxies http/https\ntraffic. Client requests a website, network device redirects traffic\nto WSA using WCCPv2, then WSA proxies the request to Cisco ASA\nFirewall and internet.\n\nYes, that our transparent proxy is not completely transparent because\nHTTPS Interception.\nIf network guys turning off HTTPS Interception for github.com \"git\nclone\" work well through the transparent proxy...\n\nDisabled https interception for github is a security issue for\nus(corporate risks, code leak, etc). That's why I asked about can the\ngit client working with https interception.\n\nProxy didn't alter any of the contents of the stream(that says to me\nour SecOps), but I've not received decrypted traffic yet to be sure.\nHTTPS traffic caching but we are also disabled this feature for github.\n\nCommon downloads with curl or browser from the same sources from\ngithub or gitlab working well.\n\nBrian, really thank you for pdf but we haven't Client-end TLS\ninterception on our clients.\n\nср, 14 апр. 2021 г. в 03:05, brian m. carlson <sandals@crustytoothpaste.net>:\n\n>\n> On 2021-04-13 at 12:07:58, Vitaly VS wrote:\n> > Hello! Can a Git client work properly through a MITM transparent proxy\n> > with HTTPS interception?\n>\n> Yes, with some important caveats.  The proxy must be completely\n> transparent.  It must not modify or impede the data in any way, it must\n> speak both HTTP 1.1 and HTTP 2 correctly and fully, and the proxy must\n> speak TLS completely correctly, including terminating the connection in\n> accordance with the protocol.  It must be completely impossible to tell\n> that a proxy is being used.\n>\n> I do want to point out that TLS interception is by definition a security\n> vulnerability and almost always significantly weakens security, often by\n> using weaker protocols, breaking or disabling certificate verification,\n> and impeding the upgrading and interoperability of the protocol[0].  You\n> should definitely read and understand the literature about TLS\n> intercepting proxies and have personally verified that your\n> implementation is free of vulnerabilities before deploying.  You\n> shouldn't rely on your implementer for this information, because they\n> usually aren't aware that their implementation has vulnerabilities.\n>\n> Also, my experience is that many, many proxies of this nature are\n> completely broken and don't work correctly, so if you are not fully\n> aware of what's going on and haven't fully tested your implementation,\n> you shouldn't deploy this technology.  I frequently answer questions\n> from users in scenarios such as this who are having problems due to a\n> broken proxy and often have to tell them to contact their network\n> administrator.  I therefore do not in any sense recommend deploying such\n> infrastructure.\n>\n> Git really does need a properly functioning HTTP and TLS implementation\n> and things tend to break in a variety of ways when encountering broken\n> proxies.  I would say \"exciting ways\", but I recognize them all now and\n> they're not exciting anymore.\n>\n> > Getting a bunch of errors when trying to \"git clone https://SOME_REPO.git\"\n> > On small REPOs (about 1-5 MB) there is a chance that the clone will be\n> > successful, but mostly I get these errors:\n>\n> Your proxy is broken and doesn't speak the protocol correctly.  It isn't\n> a transparent proxy.  You should either remove it or contact your\n> network administrator to have it removed.\n>\n> [0] https://www.ftc.gov/system/files/documents/public_comments/2016/09/00019-129028.pdf\n> --\n> brian m. carlson (he/him or they/them)\n> Houston, Texas, US\n"},{"id":"421965","messageId":"YHbWtlJDSyuAO+vf@camp.crustytoothpaste.net","threadId":"55487","inReplyTo":"CAEaE=izSNyxRcvMd5bArHnmi0F2G83nouge9e_qxiQmA0AsWog@mail.gmail.com","subject":"Re: Git via MITM transparent proxy with HTTPS Interception","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-04-14T11:49:10Z","receivedAt":"2021-04-14T11:49:24Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-04-14 at 09:35:48, Vitaly VS wrote:\n> Thank you for the fast response.\n> \n> About our network environment\n> We are Cisco WSA(Servers SW, ASA, ISR) that proxies http/https\n> traffic. Client requests a website, network device redirects traffic\n> to WSA using WCCPv2, then WSA proxies the request to Cisco ASA\n> Firewall and internet.\n> \n> Yes, that our transparent proxy is not completely transparent because\n> HTTPS Interception.\n> If network guys turning off HTTPS Interception for github.com \"git\n> clone\" work well through the transparent proxy...\n\nYes, that's because you're tampering with the data.  The output you're\ngetting clearly indicates something is modifying the data.  TLS normally\nprotects the data from accidental as well as intentional errors, so\nthere's no situation in which this could be an accident but for your\nproxy.\n\nGit will work in this case if and only if your proxy does the things I\ntold you, which your proxy doesn't.  It isn't a transparent proxy, since\nthat by definition requires that requests and responses are not\nmodified.\n\nI do appreciate you mentioning the proxy you're using so we can include\nit as a known broken proxy in future versions of the Git FAQ.\n\n> Disabled https interception for github is a security issue for\n> us(corporate risks, code leak, etc). That's why I asked about can the\n> git client working with https interception.\n\nMany major companies manage to avoid these risks without introducing\nsecurity holes into their network and breaking common applications that\nspeak standard protocols by avoiding using TLS-intercepting proxies.  In\nfact, I've worked at a company which was very diligent about these\nmatters and had strict policies on them, and in no situation did we\nintercept TLS traffic.\n\n> Proxy didn't alter any of the contents of the stream(that says to me\n> our SecOps), but I've not received decrypted traffic yet to be sure.\n> HTTPS traffic caching but we are also disabled this feature for github.\n>\n> Common downloads with curl or browser from the same sources from\n> github or gitlab working well.\n\nGit sends data that is compressed but not using a normal compressed\narchive format.  Thus, if you do anything that inspects the data to see\nif it is \"malicious\" or \"inappropriate,\" your technology will likely\nflag data that just happens to have a byte sequence that collides with\nsomething that you think is bad.  For example, if you flag the text\n\"sex\" because you think it is inappropriate, then the probably is about\n1 in 2^24 that sequence will appear in the stream and you will break the\nprotocol, since compressed data often appears random.\n\nThis is, I suspect, why Git tends to break in situations where other\nprograms do not.\n\nIf you want Git to work reliably, you can never modify the data of the\nstream, no matter what.  You also can't buffer the stream (for\nexample, try to turn chunked encoding to non-chunked).  This is\nsomething you're going to have to accept; bargaining isn't going to work\nhere, no matter how much you want it to.\n\nI can't force you to listen to me here, but I strongly recommend that if\nyou don't, you clearly communicate to your users using Git what you're\ndoing and that you know this will break Git so other parties don't have\nto.  I'm sure that the support teams for GitHub and GitLab will tell you\nthat it's your proxy and that you have to remove or disable it just as I\nam here.\n\n> Brian, really thank you for pdf but we haven't Client-end TLS\n> interception on our clients.\n\nHere are some articles covering hardware middleboxes as well:\n\nhttps://blog.cloudflare.com/monsters-in-the-middleboxes/\nhttps://jhalderm.com/pub/papers/interception-ndss17.pdf\n\nI should point out that this problem is so pervasive that TLS 1.3\nincludes intentional countermeasures against some of the worst practices\nof TLS middleboxes.  I'm certain that most of the TLS working group\nwould prevent TLS middleboxes from working at all if they could find a\nway to do so, and many of those people are at the vanguard of Internet\nsecurity.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"421968","messageId":"2a8d01d73129$cb798d40$626ca7c0$@pdinc.us","threadId":"55487","inReplyTo":"YHbWtlJDSyuAO+vf@camp.crustytoothpaste.net","subject":"RE: Git via MITM transparent proxy with HTTPS Interception","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2021-04-14T12:29:20Z","receivedAt":"2021-04-14T12:29:20Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> From: brian m. Carlson\n> Sent: Wednesday, April 14, 2021 7:49 AM\n> \n> On 2021-04-14 at 09:35:48, Vitaly VS wrote:\n> > Thank you for the fast response.\n> >\n> > About our network environment\n> > We are Cisco WSA(Servers SW, ASA, ISR) that proxies http/https\n> > traffic. Client requests a website, network device redirects traffic\n> > to WSA using WCCPv2, then WSA proxies the request to Cisco ASA\n> > Firewall and internet.\n> >\n> > Yes, that our transparent proxy is not completely transparent because\n> > HTTPS Interception.\n> > If network guys turning off HTTPS Interception for github.com \"git\n> > clone\" work well through the transparent proxy...\n> \n> Yes, that's because you're tampering with the data.  The output you're\n> getting clearly indicates something is modifying the data.  TLS normally\n> protects the data from accidental as well as intentional errors, so\n> there's no situation in which this could be an accident but for your\n> proxy.\n> \n> Git will work in this case if and only if your proxy does the things I\n> told you, which your proxy doesn't.  It isn't a transparent proxy, since\n> that by definition requires that requests and responses are not\n> modified.\n> \n> I do appreciate you mentioning the proxy you're using so we can include\n> it as a known broken proxy in future versions of the Git FAQ.\n> \n\nThis is a non-supported use case for Git. That being said - you are going to have to trace out the communications. This is going to start with GIT_TRACE=true GIT_CURL_VERBOSE=1 on your command.\n\nYou are going to need to invest in the time and knowledge to identify why your Cisco WSA is modifying data.\n\n> > Disabled https interception for github is a security issue for\n> > us(corporate risks, code leak, etc). That's why I asked about can the\n> > git client working with https interception.\n> \n> Many major companies manage to avoid these risks without introducing\n> security holes into their network and breaking common applications that\n> speak standard protocols by avoiding using TLS-intercepting proxies.  In\n> fact, I've worked at a company which was very diligent about these\n> matters and had strict policies on them, and in no situation did we\n> intercept TLS traffic.\n> \n> > Proxy didn't alter any of the contents of the stream(that says to me\n> > our SecOps), but I've not received decrypted traffic yet to be sure.\n> > HTTPS traffic caching but we are also disabled this feature for github.\n> >\n> > Common downloads with curl or browser from the same sources from\n> > github or gitlab working well.\n> \n> Git sends data that is compressed but not using a normal compressed\n> archive format.  Thus, if you do anything that inspects the data to see\n> if it is \"malicious\" or \"inappropriate,\" your technology will likely\n> flag data that just happens to have a byte sequence that collides with\n> something that you think is bad.  For example, if you flag the text\n> \"sex\" because you think it is inappropriate, then the probably is about\n> 1 in 2^24 that sequence will appear in the stream and you will break the\n> protocol, since compressed data often appears random.\n> \n> This is, I suspect, why Git tends to break in situations where other\n> programs do not.\n> \n> If you want Git to work reliably, you can never modify the data of the\n> stream, no matter what.  You also can't buffer the stream (for\n> example, try to turn chunked encoding to non-chunked).  This is\n> something you're going to have to accept; bargaining isn't going to work\n> here, no matter how much you want it to.\n> \n> I can't force you to listen to me here, but I strongly recommend that if\n> you don't, you clearly communicate to your users using Git what you're\n> doing and that you know this will break Git so other parties don't have\n> to.  I'm sure that the support teams for GitHub and GitLab will tell you\n> that it's your proxy and that you have to remove or disable it just as I\n> am here.\n> \n> > Brian, really thank you for pdf but we haven't Client-end TLS\n> > interception on our clients.\n> \n> Here are some articles covering hardware middleboxes as well:\n> \n> https://blog.cloudflare.com/monsters-in-the-middleboxes/\n> https://jhalderm.com/pub/papers/interception-ndss17.pdf\n> \n> I should point out that this problem is so pervasive that TLS 1.3\n> includes intentional countermeasures against some of the worst practices\n> of TLS middleboxes.  I'm certain that most of the TLS working group\n> would prevent TLS middleboxes from working at all if they could find a\n> way to do so, and many of those people are at the vanguard of Internet\n> security.\n\nWhile I agree, I work in environments where I do not have the luxury to change their minds. As a result I sometimes have to develop (very expensive use of time) test suites to tease out the secret sauce of security appliances. Sometimes we are fortunate that there is enough money to get the vendor to care and share.\n\nYou are more likely to be able to setup a local git mirror, within your security perimeter and enforce a review (think peer review software) of outbound pushes on a non-MITM connection between those two points only. Security management are much more likely to approve this bypassing your security appliance.\n\n[I set the reply to header, don’t email me directly I am on the list]\n\n--\nJason Pyeron  | Architect\nPD Inc        |\n10 w 24th St  |\nBaltimore, MD |\n \n.mil: jason.j.pyeron.ctr@mail.mil\n.com: jpyeron@pdinc.us\ntel : 202-741-9397\n\n\n\n\n"},{"id":"421970","messageId":"CAEaE=iyxs-vgSq2x6Pb0ASCRVoSENE6fwiARmEXObs2wOTt-wA@mail.gmail.com","threadId":"55487","inReplyTo":"2a8d01d73129$cb798d40$626ca7c0$@pdinc.us","subject":"Re: Git via MITM transparent proxy with HTTPS Interception","fromName":"Vitaly VS","fromEmail":"strikervitaly@gmail.com","sentAt":"2021-04-14T15:41:12Z","receivedAt":"2021-04-14T15:41:28Z","isPatch":false,"sender":{"key":"strikervitaly@gmail.com","avatar":"https://gravatar.com/avatar/435f0699054eca8896c8de6c1ea94deabd74d91cfe106e226441c39e9beb5ea0?d=mp&s=160"},"body":"Thanks for your answers and recommendations!\n\nI totally agree that https interception is a bad idea, but we can't\nget away from it.\n\nWe have been living with this problem for about 1.5 years. It gets\nworse every month. We have reached the boiling point and want to solve\nthe problem without turning off https interception.\n\n\n> You are more likely to be able to setup a local git mirror, within your security perimeter and enforce a review (think peer review software) of outbound pushes on a non-MITM connection between those two points only. Security management are much more likely to approve this bypassing your security appliance.\n\nWe have a local mirror(artifactory) inside the security perimeter, but\nit's impossible for us to download tons of terabytes from GitHub...\n\n\n> Yes, that's because you're tampering with the data. The output you're\n> getting clearly indicates something is modifying the data. TLS normally\n> protects the data from accidental as well as intentional errors, so\n> there's no situation in which this could be an accident but for your\n> proxy.\n\n> Git will work in this case if and only if your proxy does the things I\n> told you, which your proxy doesn't. It isn't a transparent proxy, since\n> that by definition requires that requests and responses are not\n> modified.\n\n> This is a non-supported use case for Git. That being said - you are going to have to trace out the communications. This is going to start with GIT_TRACE = true GIT_CURL_VERBOSE = 1 on your command.\n\n> You are going to need to invest in the time and knowledge to identify why your Cisco WSA is modifying data.\n\nThink it is. We think this is a bug on Cisco WSA that in some cases\ntraffic may change. Let's look at the decrypted traffic.\n\n> I do appreciate you mentioning the proxy you're using so we can include\n> it as a known broken proxy in future versions of the Git FAQ.\n\nHmmm, is there already such a list in the Git FAQ?\n\n\n> Git sends data that is compressed but not using a normal compressed\n> archive format. Thus, if you do anything that inspects the data to see\n> if it is \"malicious\" or \"inappropriate,\" your technology will likely\n> flag data that just happens to have a byte sequence that collides with\n> something that you think is bad. For example, if you flag the text\n> \"sex\" because you think it is inappropriate, then the probably is about\n> 1 in 2 ^ 24 that sequence will appear in the stream and you will break the\n> protocol, since compressed data often appears random.\n\nGood idea, it might be. Thank you, we will look in this direction.\n\nср, 14 апр. 2021 г. в 15:29, Jason Pyeron <jpyeron@pdinc.us>:\n\n>\n> > From: brian m. Carlson\n> > Sent: Wednesday, April 14, 2021 7:49 AM\n> >\n> > On 2021-04-14 at 09:35:48, Vitaly VS wrote:\n> > > Thank you for the fast response.\n> > >\n> > > About our network environment\n> > > We are Cisco WSA(Servers SW, ASA, ISR) that proxies http/https\n> > > traffic. Client requests a website, network device redirects traffic\n> > > to WSA using WCCPv2, then WSA proxies the request to Cisco ASA\n> > > Firewall and internet.\n> > >\n> > > Yes, that our transparent proxy is not completely transparent because\n> > > HTTPS Interception.\n> > > If network guys turning off HTTPS Interception for github.com \"git\n> > > clone\" work well through the transparent proxy...\n> >\n> > Yes, that's because you're tampering with the data.  The output you're\n> > getting clearly indicates something is modifying the data.  TLS normally\n> > protects the data from accidental as well as intentional errors, so\n> > there's no situation in which this could be an accident but for your\n> > proxy.\n> >\n> > Git will work in this case if and only if your proxy does the things I\n> > told you, which your proxy doesn't.  It isn't a transparent proxy, since\n> > that by definition requires that requests and responses are not\n> > modified.\n> >\n> > I do appreciate you mentioning the proxy you're using so we can include\n> > it as a known broken proxy in future versions of the Git FAQ.\n> >\n>\n> This is a non-supported use case for Git. That being said - you are going to have to trace out the communications. This is going to start with GIT_TRACE=true GIT_CURL_VERBOSE=1 on your command.\n>\n> You are going to need to invest in the time and knowledge to identify why your Cisco WSA is modifying data.\n>\n> > > Disabled https interception for github is a security issue for\n> > > us(corporate risks, code leak, etc). That's why I asked about can the\n> > > git client working with https interception.\n> >\n> > Many major companies manage to avoid these risks without introducing\n> > security holes into their network and breaking common applications that\n> > speak standard protocols by avoiding using TLS-intercepting proxies.  In\n> > fact, I've worked at a company which was very diligent about these\n> > matters and had strict policies on them, and in no situation did we\n> > intercept TLS traffic.\n> >\n> > > Proxy didn't alter any of the contents of the stream(that says to me\n> > > our SecOps), but I've not received decrypted traffic yet to be sure.\n> > > HTTPS traffic caching but we are also disabled this feature for github.\n> > >\n> > > Common downloads with curl or browser from the same sources from\n> > > github or gitlab working well.\n> >\n> > Git sends data that is compressed but not using a normal compressed\n> > archive format.  Thus, if you do anything that inspects the data to see\n> > if it is \"malicious\" or \"inappropriate,\" your technology will likely\n> > flag data that just happens to have a byte sequence that collides with\n> > something that you think is bad.  For example, if you flag the text\n> > \"sex\" because you think it is inappropriate, then the probably is about\n> > 1 in 2^24 that sequence will appear in the stream and you will break the\n> > protocol, since compressed data often appears random.\n> >\n> > This is, I suspect, why Git tends to break in situations where other\n> > programs do not.\n> >\n> > If you want Git to work reliably, you can never modify the data of the\n> > stream, no matter what.  You also can't buffer the stream (for\n> > example, try to turn chunked encoding to non-chunked).  This is\n> > something you're going to have to accept; bargaining isn't going to work\n> > here, no matter how much you want it to.\n> >\n> > I can't force you to listen to me here, but I strongly recommend that if\n> > you don't, you clearly communicate to your users using Git what you're\n> > doing and that you know this will break Git so other parties don't have\n> > to.  I'm sure that the support teams for GitHub and GitLab will tell you\n> > that it's your proxy and that you have to remove or disable it just as I\n> > am here.\n> >\n> > > Brian, really thank you for pdf but we haven't Client-end TLS\n> > > interception on our clients.\n> >\n> > Here are some articles covering hardware middleboxes as well:\n> >\n> > https://blog.cloudflare.com/monsters-in-the-middleboxes/\n> > https://jhalderm.com/pub/papers/interception-ndss17.pdf\n> >\n> > I should point out that this problem is so pervasive that TLS 1.3\n> > includes intentional countermeasures against some of the worst practices\n> > of TLS middleboxes.  I'm certain that most of the TLS working group\n> > would prevent TLS middleboxes from working at all if they could find a\n> > way to do so, and many of those people are at the vanguard of Internet\n> > security.\n>\n> While I agree, I work in environments where I do not have the luxury to change their minds. As a result I sometimes have to develop (very expensive use of time) test suites to tease out the secret sauce of security appliances. Sometimes we are fortunate that there is enough money to get the vendor to care and share.\n>\n> You are more likely to be able to setup a local git mirror, within your security perimeter and enforce a review (think peer review software) of outbound pushes on a non-MITM connection between those two points only. Security management are much more likely to approve this bypassing your security appliance.\n>\n> [I set the reply to header, don’t email me directly I am on the list]\n>\n> --\n> Jason Pyeron  | Architect\n> PD Inc        |\n> 10 w 24th St  |\n> Baltimore, MD |\n>\n> .mil: jason.j.pyeron.ctr@mail.mil\n> .com: jpyeron@pdinc.us\n> tel : 202-741-9397\n>\n>\n>\n>\n"}]}