{"thread":{"id":"54072","subject":"Cloning subfolder as new root; subfolder as worktree","startedAt":"2020-08-20T08:03:18Z","lastAt":"2020-08-20T08:03:18Z","messageCount":1,"participants":["Mickey Endito"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"404082","messageId":"fO2Zef1hJ8xCWRCeWPtEHSr1YtlbqeXFqR09Frz0CmXU_Z8awBLgw_F8wdBiUsxuJrTWxvMAM1A6mUfqtWLvLIfBWNdDwDHSWS-SHNpjUz0=@protonmail.com","threadId":"54072","inReplyTo":null,"subject":"Cloning subfolder as new root; subfolder as worktree","fromName":"Mickey Endito","fromEmail":"mickey.endito.2323@protonmail.com","sentAt":"2020-08-20T08:03:02Z","receivedAt":"2020-08-20T08:03:18Z","isPatch":false,"sender":{"key":"mickey.endito.2323@protonmail.com","avatar":null},"body":"Dear all,\n\nI'm currently missing a feature in git to be able to clone a subfolder as the\nroot or phrased differently use a subfolder as a worktree.\n\nThis mail has become rather long, so here is an overview: First, I state the\nproblem and give some use cases. Then, I list a couple of partial workarounds\nand other methods for the problem, which are currently achievable with git.\nLastly, I provide an idea how this feature could be implemented within git\nobject model.\n\n\nThe Problem\n-----------\n\nAssume you have a git repository A with the following file structure:\n\nfoo/a.txt\nbar/b.txt\nbar/baz/c.txt\n\nWhat I want to achieve is creating a git repo where bar/ equals the root /,\ni.e. a repo B with the contents\n\nb.txt\nbaz/c.txt\n\nWe can describe that as a lens or zoom of the repo A.\nI believe svn had that capability but I'm not sure.\n\n\nApplications for that feature\n-----------------------------\n\nNotation: I use \"repo:path\" to indicate that path should be seen relative to\nthe repository repo.\n\n* I have a repo which mimics (parts of) my entire file system (think\n  configuration files).  I'd like to be able to check out the subfolder\n  repo:/etc/foobar in the actual filesystem folder /etc. While not checking out\n  the rest of repo:/etc as that would lead in a disaster.\n\n* I have a website project where the html files (which are not generated by\n  a build script) are in repo:www/ and I'd like to check them out to /srv/www/\n  for deployment.\n\n* Think of a big project with different components possibly stacked deep in\n  a directory structure. We want to work on a single component somewhere down\n  that structure, e.g. repo:client/new/x11/gtk/daemon/test/testapp. We could\n  use git-sparse-checkout for that but that would leave us with a lot of\n  quasi-empty directories client/new/x11/gtk/daemon/test/ where in this case\n  only the testapp directory is relevant.\n\n\nThings/Workarounds I am aware of\n--------------------------------\n\n1. git submodules\n\nYou probably think use git-submodules. However, bar/ is not a dependency or\nlibrary where that would make sense but rather a part of of repo A.  So the\nlogical dependency is reversed: it's not A that depends on B but rather\nB depends on A.  In some use cases changes in bar/ require additional changes\nin foo (in that use cases B is like a read-only view).\n\n2. git clone + filter-branch\n\nWe can clone the repo followed by a git-filter-branch (or its alternative\ngit-filter-repo)\n\ngit clone /path/to/repo/A /path/to/repo/B cd /path/to/repo/B git filter-branch\n--subdirectory-filter bar -- --all\n\nThis creates a sort of read-only clone. But has massive drawbacks:\n* We cannot do a simple git pull to update repo B to the new state of repo A.\n  To do that we have to clone and filer-branch it again.\n* It changes commit-IDs.\n* We cannot push changes done in B back to A.\n\n3. git-subtree\n\nWe can use git-subtree to filter the subdirectory and then clone the generated\nbranch as repo B, like so:\n\n# in repo A\ncd /path/to/repo/A\ngit subtree split --prefix=bar --annotate 'bar: ' --branch branch-bar\ngit clone -b branch-bar /path/to/repo/A /path/to/repo/B\n\nHere we have:\n* It requires support from repo A which must generate the branch-bar.\n* Repo A now must contain two commit-histories (the main branch and the\n  branch-bar) of the same logical-history.  In particular the commit ids are\n  different for the same logical commit in the main branch and the branch-bar.\n* branch-bar must be regenerated every time. I have not (yet) investigated\n  whether git-subtree is capable of continuing a split from the last commit. So\n  far I only managed that it recreates all commits in branch-bar (but with the\n  same commit-ids as before)\n* Because the commit-ids of branch-bar do not change (at least when called with\n  the same arguments), we can use git pull to update repo B\n* We can push changes in repo B back to repo A in the branch-bar. But\n  I currently do not see a simple method how to incorporate this changes into\n  the main branch.\n\n4. git-sparse-checkout\n\nWe can use git-sparse-checkout like so\n\ngit clone /path/to/repo/A --no-checkout /path/to/repo/B\ncd /path/to/repo/B\ngit sparse-checkout init\ngit spares-checkout set bar\n\nThis is kind a close to what I want in the sense that we can push and pull and\nthe commit-ids are unaltered. However, this totally gets the directory\nstructure wrong, which is a no-go in some of the above use cases.\n\n\nAn idea for a solution\n----------------------\n\nThe following is an idea how the above feature could be implemented. This is\njust a rough sketch and I have not thought how this approach would interact\nwith other git tooling.\n\nWe add a (for example; names are up to debate) --subfolder argument to git\nclone:\n\ngit clone --subfolder bar /path/to/repo/A /path/to/repo/B\n\nThis clones the complete repo A but checks out contents of path bar in the root\ndirectory. The HEAD points to the (full) commit. Additionally somewhere(tm) we\nstore that we have zoomed in to only see paths in bar (maybe git-worktree can\nbe expanded for that?).\n\nThat is stuff under bar is treated like a checkedout repo while all other stuff\nis treated like being in a bare repo. (This at least should be the guide line\nwhen thinking about the behaviour git should provide)\n\nDoing a git push, git pull does the normal update of the repo but when checking\nout files to the working directory only those files under bar/ are considered.\n\nWhen editing and committing files in repo B, the following would be a sane thing\nto do: The (old) tree of the current HEAD is taken and then the subtree\ncorresponding to bar is replaced with the tree in the index. That way we\ngenerate a full valid commit which can be pushed back to repo A.\n\nIf we switch/checkout to a branch/commit that has not bar/ directory, then the\ncheckout copy should be empty. If we add something and commit it, then the\nparent tree-objects of the new commit should be altered to contain the path bar.\nAs git does not track directories this should work out as expected.\n\nMerge conflicts happen. If these happen for files inside the bar directory, the\nwe can do our usual stuff. Due to the flexibility of git we can arrange that the\ncommits/trees to be merged have a conflict outside of the bar directory. In\nthat case we cannot produce a working copy of the commit. Thus, it seems\nappropriate to abort whatever we do and inform the user to use a full clone for\ndoing the merge.\n\nWhen cloning repo B to repo C, there are not restrictions whatsoever as B has\na full copy of the repository (just not checked out). So when looking from\n\"outside\" repo A and repo B are indistinguishable. Thus the following works:\n\n\ngit clone --subfolder bar /path/to/repo/A /path/to/repo/B\ngit clone /path/to/repo/B /path/to/repo/C\n\nRepo A and C are the same (both without a zoom).\n\ngit clone --subfolder bar /path/to/repo/A /path/to/repo/B\ngit clone --subfolder foo /path/to/repo/B /path/to/repo/C1\n\ngit clone --subfolder foo /path/to/repo/A /path/to/repo/C2\n\nRepo C1 and C2 are the same (both zoomed to foo/).\n\nNon-goals:\n\nThe following (weird?) thing is outside of the scope of this idea.\nzooming in into two (or more) directories simultaneously, e.g.\n\nrepo A:\nfoo1/bar/...\nfoo2/foo3/baz/...\n\nand with the hypothetical git clone --subfolder foo1 --subfolder foo2/foo3\n\nwe get\n\nrepo B:\nbar/...\nbaz/...\n\nAlso converting a zoomed repo B into a full (non-bare) repo A is not part of it.\nAlthough I think, this could easily be achieved by some usage of deleting the\nreference to the subfolder and doing a `git reset --hard` on the working copy.\n\n\n\nSummary\n-------\n\nThat the problem is not about the size of the checkout (which sparse-checkout\ntackles), nor the size of the repo or the mount of data which needs to be\ndownloaded (both of which clone --depth tackles), its about getting the\ndirectory structure in repo B right while also keeping a strong link to repo\nA as upstream to pull (and maybe push) changes.\n\nIf I have missed any approach for a solution I'd like to hear about it.\n\nBest\nMickey\n\n"}]}