{"thread":{"id":"50029","subject":"[wishlist] support of cloning recursively from non-bare submodule hierarchies?","startedAt":"2018-12-13T17:19:26Z","lastAt":"2018-12-13T21:59:34Z","messageCount":2,"participants":["Yaroslav Halchenko","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"365276","messageId":"20181213171917.GC4633@hopa.kiewit.dartmouth.edu","threadId":"50029","inReplyTo":null,"subject":"[wishlist] support of cloning recursively from non-bare submodule hierarchies?","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2018-12-13T17:19:17Z","receivedAt":"2018-12-13T17:19:26Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"Example - on http://datasets.datalad.org we have a few hundred datasets\norganized into a hierarchy as git submodules.  Each  git submodules carries its\nown .git/ directory so they are \"self sufficient\" and we could readily assess\ntheir sizes, and \"cut the tree\" at any level without looking for the\nsupermodule somewhere high up in the tree.\n\n.gitmodules typically has relative paths for the url and path for the\nsubmodules there, the form which I think we chose because it used to work (I\ncould be utterly wrong! but I think it was done in an informed fashion)\nfor git clone --recursive:\n\n\t$> curl http://datasets.datalad.org/labs/gobbini/famface/.gitmodules\n\t[submodule \"data\"]\n\t\tpath = data\n\t\turl = ./data\n\nand possibly outside:\n\n\t$> curl http://datasets.datalad.org/labs/gobbini/famface/data/.gitmodules \n\t[submodule \"scripts/mridefacer\"]\n\t\tpath = scripts/mridefacer\n\t\turl = https://github.com/yarikoptic/mridefacer\n\nBut unfortunately git doesn't even consider such (valid AFAIK) situation\nwhile cloning where url has to have .git suffix but repository is not bare and\na relative \"data\" path (or \"./data\" url) is referring to the worktree.\n\n\t$> git clone --recursive http://datasets.datalad.org/labs/gobbini/famface/.git \n\tCloning into 'famface'...\n\tremote: Counting objects: 61, done.\n\tremote: Compressing objects: 100% (54/54), done.\n\tremote: Total 61 (delta 14), reused 0 (delta 0)\n\tUnpacking objects: 100% (61/61), done.\n\tSubmodule 'data' (http://datasets.datalad.org/labs/gobbini/famface/.git/data) registered for path 'data'\n\tCloning into '/tmp/famface/data'...\n\tfatal: repository 'http://datasets.datalad.org/labs/gobbini/famface/.git/data/' not found\n\tfatal: clone of 'http://datasets.datalad.org/labs/gobbini/famface/.git/data' into submodule path '/tmp/famface/data' failed\n\tFailed to clone 'data'. Retry scheduled\n\tCloning into '/tmp/famface/data'...\n\tfatal: repository 'http://datasets.datalad.org/labs/gobbini/famface/.git/data/' not found\n\tfatal: clone of 'http://datasets.datalad.org/labs/gobbini/famface/.git/data' into submodule path '/tmp/famface/data' failed\n\tFailed to clone 'data' a second time, aborting\n\non the server I use the \"smart HTTP\" git backend, but not sure if that is the one to blame, since\nI do not see in the logs any attempt to get the /data from not under .git/:\n\n\t10.31.188.88 - - [13/Dec/2018:12:18:38 -0500] \"GET /labs/gobbini/famface/.git/info/refs?service=git-upload-pack HTTP/1.1\" 200 681 \"-\" \"git/2.20.0.rc2.403.gdbc3b29805\"\n\t10.31.188.88 - - [13/Dec/2018:12:18:38 -0500] \"POST /labs/gobbini/famface/.git/git-upload-pack HTTP/1.1\" 200 69276 \"-\" \"git/2.20.0.rc2.403.gdbc3b29805\"\n\n\t==> datasets.datalad.org-error.log <==\n\t[Thu Dec 13 12:18:38.673447 2018] [core:info] [pid 7570:tid 140683541153536] [client 10.31.188.88:32794] AH00128: File does not exist: /srv/datasets.datalad.org/www/labs/gobbini/famface/.git/data/info/refs\n\n\t==> datasets.datalad.org-access-comb.log <==\n\t10.31.188.88 - - [13/Dec/2018:12:18:38 -0500] \"GET /labs/gobbini/famface/.git/data/info/refs?service=git-upload-pack HTTP/1.1\" 404 485 \"-\" \"git/2.20.0.rc2.403.gdbc3b29805\"\n\n\t==> datasets.datalad.org-error.log <==\n\t[Thu Dec 13 12:18:38.689277 2018] [core:info] [pid 7572:tid 140683574724352] [client 10.31.188.88:32796] AH00128: File does not exist: /srv/datasets.datalad.org/www/labs/gobbini/famface/.git/data/info/refs\n\n\t==> datasets.datalad.org-access-comb.log <==\n\t10.31.188.88 - - [13/Dec/2018:12:18:38 -0500] \"GET /labs/gobbini/famface/.git/data/info/refs?service=git-upload-pack HTTP/1.1\" 404 485 \"-\" \"git/2.20.0.rc2.403.gdbc3b29805\"\n\n\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"},{"id":"365309","messageId":"CAGZ79kYMULBBDBt6Sx___m37TNkruYmpKto5ZGG7u7X1jHuKig@mail.gmail.com","threadId":"50029","inReplyTo":"20181213171917.GC4633@hopa.kiewit.dartmouth.edu","subject":"Re: [wishlist] support of cloning recursively from non-bare submodule hierarchies?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-12-13T21:59:18Z","receivedAt":"2018-12-13T21:59:34Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Dec 13, 2018 at 9:19 AM Yaroslav Halchenko <yoh@onerussian.com> wrote:\n>\n> Example - on http://datasets.datalad.org we have a few hundred datasets\n> organized into a hierarchy as git submodules.  Each  git submodules carries its\n> own .git/ directory so they are \"self sufficient\" and we could readily assess\n> their sizes, and \"cut the tree\" at any level without looking for the\n> supermodule somewhere high up in the tree.\n>\n> .gitmodules typically has relative paths for the url and path for the\n> submodules there, the form which I think we chose because it used to work (I\n> could be utterly wrong! but I think it was done in an informed fashion)\n> for git clone --recursive:\n>\n>         $> curl http://datasets.datalad.org/labs/gobbini/famface/.gitmodules\n>         [submodule \"data\"]\n>                 path = data\n>                 url = ./data\n>\n> and possibly outside:\n>\n>         $> curl http://datasets.datalad.org/labs/gobbini/famface/data/.gitmodules\n>         [submodule \"scripts/mridefacer\"]\n>                 path = scripts/mridefacer\n>                 url = https://github.com/yarikoptic/mridefacer\n\nSo far so good.\n\n> But unfortunately git doesn't even consider such (valid AFAIK) situation\n> while cloning where url has to have .git suffix but repository is not bare and\n> a relative \"data\" path (or \"./data\" url) is referring to the worktree.\n>\n>         $> git clone --recursive http://datasets.datalad.org/labs/gobbini/famface/.git\n[..]\n>         Submodule 'data' (http://datasets.datalad.org/labs/gobbini/famface/.git/data) registered for path 'data'\n\nand here it goes wrong, and you would have expected to see\n.../gobbini/famface/data, eliding the .git ?\n\nI just checked and this did not work neither in v2.18.0 nor v2.0.0 of\nGit, so it is\neither a real old regression in submodules, or something else.\nIs it possible that the clone worked once without the additional .git\nin the superproject URL?\n\n> on the server I use the \"smart HTTP\" git backend, but not sure if that is the one to blame, since\n> I do not see in the logs any attempt to get the /data from not under .git/:\n\nIf we want to strip off \"/.git\" of urls to make submodules work,\nwe'd want to look at builtin/submodule--helper.c::compute_submodule_clone_url\nthat was recently introduced.\n\nI wonder if we'd just want to cut off the \"/.git\" and assume the submodule\nis there in the worktree. Or if we need to see if the submodule was\nabsorbed into .git/modules/<name> on the remote side. (But if the\nsubmodule is checked out both would work)\n"}]}