{"thread":{"id":"58015","subject":"Windows long file paths bug(s) with \"-c core.longpaths=true\" whilst cloning","startedAt":"2022-06-15T12:45:33Z","lastAt":"2022-06-18T21:57:19Z","messageCount":2,"participants":["Lukas Buricin","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"457274","messageId":"CA+c+RB=ud_==QYJMgcwQ=X4imQhxsFvMuKp_bP0H_MBY1BbUgQ@mail.gmail.com","threadId":"58015","inReplyTo":null,"subject":"Windows long file paths bug(s) with \"-c core.longpaths=true\" whilst cloning","fromName":"Lukas Buricin","fromEmail":"lukas.buricin@cubicmotion.com","sentAt":"2022-06-15T12:45:16Z","receivedAt":"2022-06-15T12:45:33Z","isPatch":false,"sender":{"key":"lukas.buricin@cubicmotion.com","avatar":null},"body":"Hi\n\nI am seeing multiple problems when cloning. I use \"-c\ncore.longpaths=true\" and I of course have the long paths enabled in\nWindows.\n\nCloning long file names appears to work only when the file names\nthemselves breach the limit MAX_PATH 260 characters, but not in other\ncases.\n\n1) When cloning in a directory path that is on its own longer than\nMAX_PATH. For example, when cloning to\n\nC:\\my_very_long_named_directory_..._reaching_over_MAX_PATH_260_long\n\nthen the content of '.git' folder in\n\nC:\\my_very_long_named_directory_..._reaching_over_MAX_PATH_260_long\\.git\n\nfails to get created. Whilst this may seem like an unlikely scenario\nfor a single repository being in such a long-named directory, it can\nhappen with a more complex hierarchy of submodules, where the problem\nbecomes more obvious (\"external\" represents directories containing\nsubmodules):\n\nc:\\my_projects\\my_project\\external\\app_components\\external\\app_api_layer\\external\\framework\\external\\video_components\\external\\core_components\\external\\base\\external\\windows-third-parties\\gtest\\...\n\nThe first problem seems to be in compat\\mingw.c, inside char\n*mingw_mktemp(char *template) where we still use wchar_t\nwtemplate[MAX_PATH];\n\nAfter replacing it with MAX_LONG_PATH the problem moves further to\nusages of xutftowcs_path() that internally assumes everything to be\njust MAX_PATH.\n\nWhen replaced all the calls to xutftowcs_path() with\nxutftowcs_path_ex() providing MAX_LONG_PATH and passing\n\"core_long_paths\", the initial parent git process moves on. However\n...\n\n2) The git process spawns another child git process, which doesn't\nreflect the parent core_long_paths (being 1), because in the child\nprocess it's 0, hence all the functions that should be allowed to\nprepend the paths with \\\\?\\ are told not to extend anything. This\nobviously leads to more failures.\n\n3) Even when the directory paths and the files do fit in the 260 limit\n(I have shrunk all the paths by renaming 'external' to 'ext' in order\nto fit in the MAX_PATH), git falls over in its internal .git folder\nbecause it stores some information in significantly longer paths than\nwhat is present in the repository itself. Using the example of\n\nc:\\my_projects\\my_project\\external\\app_components\\external\\app_api_layer\\external\\framework\\external\\video_components\\external\\core_components\\external\\base\\external\\windows-third-parties\\gtest\\...\n\nthere will be folders 'modules' added at each level in the .git\ninternal folder for all submodules, so will end up somewhat like:\n\nc:\\my_projects\\my_project\\.git\\modules\\external\\app_components\\modules\\external\\app_api_layer\\modules\\external\\framework\\modules\\external\\video_components\\modules\\external\\core_components\\modules\\external\\base\\modules\\external\\windows-third-parties\\modules\\gtest\\...\n\nSo the content in .git is much more likely to breach the MAX_PATH\nlimit than the repository \"user\" files themselves.\n\nIn general, aften seeing the code (admittedly first time ever), I also\nhave a few questions, because I might be missing some context ...\n\n1) Why do we use MAX_PATH for buffers at all? This might be my\nmisunderstanding, but I would have thought that providing long file\nname support would essentially mean not using that contant other than\nfor determining whether to prepend given wchar_t* output by \\\\?\\ when\nthe input const char* is longer than MAX_PATH, before passing the\nwchar_t* in Windows API calls? This conversion and eventual prepending\ncould be done in one place and should be no burden for the CPU.\n\n2) Why not having long paths by default? Again, this might be me\nmissing some historical context of regression, but it just seems\nlogical to simply prepend by \\\\?\\ whenever needed rather than fail.\n\n3) Why to have a \"-c core.longpaths=true\" command line argument at\nall? Given that we can detect whether the long file names are enabled\nin the OS, we could easily drop that argument and have the full\nsupport for free? (Losing also the problem of passing the flag to\nchild git processes). In fact we might even not need the OS support\ndetection, we may simply rely on the Windows API return values and\nGetLastError() whilst always proving MAX_LONG_PATH and eventually\nprepending paths with \\\\?\\.\n\nThank you very much in advance for consideration and for eventual explanations.\n\nHave a nice day.\n\nLukas\n"},{"id":"457504","messageId":"nycvar.QRO.7.76.6.2206182308590.349@tvgsbejvaqbjf.bet","threadId":"58015","inReplyTo":"CA+c+RB=ud_==QYJMgcwQ=X4imQhxsFvMuKp_bP0H_MBY1BbUgQ@mail.gmail.com","subject":"Re: Windows long file paths bug(s) with \"-c core.longpaths=true\" whilst cloning","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-18T21:57:07Z","receivedAt":"2022-06-18T21:57:19Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Lukas,\n\nOn Wed, 15 Jun 2022, Lukas Buricin wrote:\n\n> I am seeing multiple problems when cloning. I use \"-c\n> core.longpaths=true\" and I of course have the long paths enabled in\n> Windows.\n>\n> Cloning long file names appears to work only when the file names\n> themselves breach the limit MAX_PATH 260 characters, but not in other\n> cases.\n>\n> 1) When cloning in a directory path that is on its own longer than\n> MAX_PATH. For example, when cloning to\n>\n> C:\\my_very_long_named_directory_..._reaching_over_MAX_PATH_260_long\n>\n> then the content of '.git' folder in\n>\n> C:\\my_very_long_named_directory_..._reaching_over_MAX_PATH_260_long\\.git\n>\n> fails to get created. Whilst this may seem like an unlikely scenario\n> for a single repository being in such a long-named directory, it can\n> happen with a more complex hierarchy of submodules, where the problem\n> becomes more obvious (\"external\" represents directories containing\n> submodules):\n>\n> c:\\my_projects\\my_project\\external\\app_components\\external\\app_api_layer\\external\\framework\\external\\video_components\\external\\core_components\\external\\base\\external\\windows-third-parties\\gtest\\...\n>\n> The first problem seems to be in compat\\mingw.c, inside char\n> *mingw_mktemp(char *template) where we still use wchar_t\n> wtemplate[MAX_PATH];\n>\n> After replacing it with MAX_LONG_PATH the problem moves further to\n> usages of xutftowcs_path() that internally assumes everything to be\n> just MAX_PATH.\n\nIndeed. You will see much of the same journey, in the form of testable\npatches, at https://github.com/git-for-windows/git/pull/3877.\n\nOne major problem, of course, is the assumption of `mktemp()` that the\nbuffer passed as a parameter has the size `MAX_PATH`. So if we start\nassuming that the size is actually much larger, we risk buffer overruns\nunless we litter Git's code even outside of `compat/mingw.c` with\n`MAX_LONG_PATH` instances, a proposal I have no doubt would be shot down\nimmediately on the Git mailing list.\n\n> When replaced all the calls to xutftowcs_path() with\n> xutftowcs_path_ex() providing MAX_LONG_PATH and passing\n> \"core_long_paths\", the initial parent git process moves on. However\n> ...\n>\n> 2) The git process spawns another child git process, which doesn't\n> reflect the parent core_long_paths (being 1), because in the child\n> process it's 0,\n\nWhy would it be 0, though? The `-c core.longPaths=true` option is parsed\ninto the environment variable `GIT_CONFIG_PARAMETERS`, which is passed\ndown to the child process.\n\n> hence all the functions that should be allowed to prepend the paths with\n> \\\\?\\ are told not to extend anything. This obviously leads to more\n> failures.\n>\n> 3) Even when the directory paths and the files do fit in the 260 limit\n> (I have shrunk all the paths by renaming 'external' to 'ext' in order\n> to fit in the MAX_PATH), git falls over in its internal .git folder\n> because it stores some information in significantly longer paths than\n> what is present in the repository itself. Using the example of\n>\n> c:\\my_projects\\my_project\\external\\app_components\\external\\app_api_layer\\external\\framework\\external\\video_components\\external\\core_components\\external\\base\\external\\windows-third-parties\\gtest\\...\n>\n> there will be folders 'modules' added at each level in the .git\n> internal folder for all submodules, so will end up somewhat like:\n>\n> c:\\my_projects\\my_project\\.git\\modules\\external\\app_components\\modules\\external\\app_api_layer\\modules\\external\\framework\\modules\\external\\video_components\\modules\\external\\core_components\\modules\\external\\base\\modules\\external\\windows-third-parties\\modules\\gtest\\...\n>\n> So the content in .git is much more likely to breach the MAX_PATH\n> limit than the repository \"user\" files themselves.\n\nRight, and there is the additional problem that the current working\ndirectory cannot have more than 260 characters, either, when spawning new\nprocesses.\n\nAnd we _also_ have to deal with the reality that by far not all\nexecutables handle long paths gracefully, even as we teach `git.exe` more\ntricks.\n\nFor example, the MSYS2 Bash used by Git for Windows does not handle long\npaths. Which is a problem because we still have some core functionality in\nGit being implemented as Unix shell script: The `submodule`, `bisect`, and\n`mergetool` commands are commonly-used examples.\n\nNow, one approach we discussed over at\nhttps://github.com/git-for-windows/git/pull/3877 is to configure that\nregistry setting that allows longer paths for executables whose manifest\nindicate that they are \"long-paths aware\". The mechanics of rolling\nsomething like that out would get tricky, though, as it would mean that\nthe MSYS2 runtime would have to be modified, too, and for obvious reasons\nit would then require _all_ of the programs that link to it to be made\nlong paths-aware, too, otherwise we would be bogged down in buffer\noverrun-related problems.\n\n> In general, aften seeing the code (admittedly first time ever), I also\n> have a few questions, because I might be missing some context ...\n>\n> 1) Why do we use MAX_PATH for buffers at all?\n\nThat is a good question. Historical reasons, I suppose. You mentioned\n`mktemp()` above, which is a fine example how the C library functions were\noriginally designed with a hard-coded assumption as to buffers of size\n`MAX_PATH`. On Windows, there is at least `_mktemp_s()`, which takes a\npointer and a size as parameters, but on Linux/Unix, I am unaware of any\nsuch function, they all seem to expect buffers of size `MAX_PATH`, which\nessentially limits what Git can use. I guess a MAX_PATH of 1024 \"ought to\nbe enough for anybody\".\n\n> This might be my misunderstanding, but I would have thought that\n> providing long file name support would essentially mean not using that\n> contant other than for determining whether to prepend given wchar_t*\n> output by \\\\?\\ when the input const char* is longer than MAX_PATH,\n> before passing the wchar_t* in Windows API calls? This conversion and\n> eventual prepending could be done in one place and should be no burden\n> for the CPU.\n\nEven functions like `GetFullPathNameW()` require you to provide some\nreasonably-sized buffer as parameter. That's the reson why we still stick\nwith MAX_LONG_PATH, which we chose as 4096. Because I (and IIRC Karsten\nBlees) guessed at the time that 4096 out to be enough for anybody.\n\n> 2) Why not having long paths by default? Again, this might be me\n> missing some historical context of regression, but it just seems\n> logical to simply prepend by \\\\?\\ whenever needed rather than fail.\n\nThere are a few reasons why they should not be on by default, most\nimportantly that many 3rd-party programs cannot handle them. For example,\nfor a long time CMD itself could not. And of course the MSYS2 version of\n`rm.exe`, which we tried to use in the test suite, cannot handle them.\n\nRather than succeeding to clone repositories into long paths by default\nonly to let the user run into trouble not only using their favorite tools\non the checked out files, but then also be unable to `rm -rf` the thing,\nwe chose to disable long paths support by default instead.\n\nThere are also performance implications. Accessing files via their\nrelative paths is much quicker than first constructing that `\\\\?\\` form\n(which must be an absolute path).\n\n> 3) Why to have a \"-c core.longpaths=true\" command line argument at\n> all? Given that we can detect whether the long file names are enabled\n> in the OS, we could easily drop that argument and have the full\n> support for free? (Losing also the problem of passing the flag to\n> child git processes). In fact we might even not need the OS support\n> detection, we may simply rely on the Windows API return values and\n> GetLastError() whilst always proving MAX_LONG_PATH and eventually\n> prepending paths with \\\\?\\.\n\nEven just the `mktemp()` example you found _already_ puts a stick into the\nwheels of your idea.\n\nI also sense that there's confusion between the two forms long paths\nsupport can take:\n\n- Convert paths that would otherwise be too long to the `\\\\?\\` form and\n  use the `*W()` functions of the Windows API. This is supported by all\n  Windows versions that are supported by Git for Windows.\n\n- Require users to configure a registry setting, use the `*W()` family of\n  Win32 API functions exclusively, add a manifest entry that your\n  executable is long paths-aware, and skip that `\\\\?\\` prefixing. See\n  https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation\n  for more details. This requires Windows 10 Build 1607 or later.\n\nThe more elegant solution would be the latter, of course, but its Windows\n10 Build 1607 requirement is incompatible with Git for Windows' promise\nthat it still supports Windows versions all the way back to Vista\n(although you can expect Vista support to be dropped after Git for Windows\nv2.37.0 is released, with Windows 7 and Windows 8 support to follow around\nthe end of this year).\n\nCiao,\nDscho\n"}]}