{"thread":{"id":"63434","subject":"sparse-checkout and symlinks?","startedAt":"2025-05-10T16:05:42Z","lastAt":"2025-05-10T17:29:57Z","messageCount":2,"participants":["Gabriel Scherer","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"517740","messageId":"2281c960-7750-4319-8042-dade0ddfff67@irif.fr","threadId":"63434","inReplyTo":null,"subject":"sparse-checkout and symlinks?","fromName":"Gabriel Scherer","fromEmail":"gabriel.scherer@irif.fr","sentAt":"2025-05-10T15:12:06Z","receivedAt":"2025-05-10T16:05:42Z","isPatch":false,"sender":{"key":"gabriel.scherer@irif.fr","avatar":null},"body":"Dear git list,\n\nsparse-checkout interacts badly with symlinks within a git repository: \nif b/file is a symlink to a/file, and the user asks for a \nsparse-checkout with only b/, they get a dead link (b/file points to \nnothing).\n\nI initially assumed that replacing a file by a symlink to another file \nwith the same content would not be observable by other users of the \nrepository. This assumption is incorrect in presence of sparse checkouts.\n\nI would find it natural to have sparse-checkout \"follow symlinks\". When \nchecking b/file as the user requests, git would notice that it is a \nsymlink and do one of the following:\n\n1. if the link target a/file is not in the specified sparse checkout \nset, copy its content instead of creating a dead symlink\n    (Downside: this could lead to duplication if several in-checkout \nfiles point to a/file.)\n2. or add a/file to the sparse checkout set\n    (Note: simply checking it out silently is not enough as 'reapply' \nwould then drop it)\n\nDoes this sound reasonable to you? Would you have recommendations on \nwhat the interface for such a feature should look like?\n- which of the alternatives above would you recommend?\n- should this be enabled only by a new configuration or command-line \noption (to which subcommand?), how would you name it?\n\nThanks in advance\n\n\n## More details on the use-case\n\nI'm trying to reduce the working directory size of a gigabyte-large git \nrepository ( https://github.com/typst/packages \n<https://github.com/typst/packages> ) which contains a substantial \namount of duplicated files, by replacing duplicates by symlinks. The \nrepository uses a continuous integration script to run automated tests \non each proposed change, which uses sparse-checkout on only the \ndirectories listed as containing modified files.(The directories \ncorrespond to independent \"packages\" so it makes sense to check them \nseparately.) This breaks when the modified directories contain symlinks \nto other, non-modified directories.\n\n\n"},{"id":"517743","messageId":"CABPp-BEM4Ex6AF1PNs4V2MA5x0sWM0vs1e_G7EL-CGpihXnTJA@mail.gmail.com","threadId":"63434","inReplyTo":"2281c960-7750-4319-8042-dade0ddfff67@irif.fr","subject":"Re: sparse-checkout and symlinks?","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-05-10T17:29:45Z","receivedAt":"2025-05-10T17:29:57Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, May 10, 2025 at 9:05 AM Gabriel Scherer <Gabriel.Scherer@irif.fr> wrote:\n>\n> Dear git list,\n>\n> sparse-checkout interacts badly with symlinks within a git repository:\n> if b/file is a symlink to a/file, and the user asks for a\n> sparse-checkout with only b/, they get a dead link (b/file points to\n> nothing).\n\nThat's what I'd expect.\n\n> I initially assumed that replacing a file by a symlink to another file\n> with the same content would not be observable by other users of the\n> repository. This assumption is incorrect in presence of sparse checkouts.\n>\n> I would find it natural to have sparse-checkout \"follow symlinks\". When\n> checking b/file as the user requests, git would notice that it is a\n> symlink and do one of the following:\n>\n> 1. if the link target a/file is not in the specified sparse checkout\n> set, copy its content instead of creating a dead symlink\n>     (Downside: this could lead to duplication if several in-checkout\n> files point to a/file.)\n\nAnd the file would immediately show as modified in status, which seems\nlike a rather negative surprise.  If someone does a `git add -u` or\nsimilar, they'd convert the symlink to a regular file, which could be\nanother form of gotcha.\n\n> 2. or add a/file to the sparse checkout set\n>     (Note: simply checking it out silently is not enough as 'reapply'\n> would then drop it)\n\nThis is a solution that does not work with the default cone mode.  It\nmay also surprise users who expected the sparse checkout rules to be\nsomething entirely under their control.\n\n\nBoth solutions would also interact rather poorly with sparse indexes;\neither looks to me like a bit of a foot-gun for them.\n\n> Does this sound reasonable to you? Would you have recommendations on\n> what the interface for such a feature should look like?\n> - which of the alternatives above would you recommend?\n\nHonestly, neither.  The problem isn't limited to symlinks; some examples:\n  * you could have a script in one part of the checkout that tries to\ninvoke a script in the other part\n  * you could have a source code file in the non-sparse part that has\na directive to include/import/require source code in the\nnon-sparse-checkout\n...and there are many other ways files could depend on others.\n\nSymlinks are only special in that they require no programming or other\nknowledge to determine that there is a dependency between files.\n\nI'd rather continue to follow the expectation that users of sparse\ncheckouts need to determine the relevant set of dependencies and\ndetermine which sparsity rules make sense in their repo.  I suspect\nthat each repo might be somewhat special here, and thus each might\nhave their own tool for creating sparse-checkouts using repo-specific\nknowledge (e.g. \"I want moduleA plus whatever it depends upon\") which\ntheir repo-specific tool then translates into the appropriate set of\npaths or patterns to use.  symlinks would be just one of many kinds of\ndependencies that such a tool would consider.  I understand that some\nrepos might be big enough that users want to use sparse-checkouts, but\nnot big enough that one of the developers wants to write such a tool.\nStill, I'd rather not attempt dependency analysis in git[*], and\ninstead require the users to do the dependency analysis.\n\n> - should this be enabled only by a new configuration or command-line\n> option (to which subcommand?), how would you name it?\n>\n> Thanks in advance\n>\n>\n> ## More details on the use-case\n>\n> I'm trying to reduce the working directory size of a gigabyte-large git\n> repository ( https://github.com/typst/packages\n> <https://github.com/typst/packages> ) which contains a substantial\n> amount of duplicated files, by replacing duplicates by symlinks. The\n> repository uses a continuous integration script to run automated tests\n> on each proposed change, which uses sparse-checkout on only the\n> directories listed as containing modified files.(The directories\n> correspond to independent \"packages\" so it makes sense to check them\n> separately.) This breaks when the modified directories contain symlinks\n> to other, non-modified directories.\n\nI know it's not quite what you want to hear, but I believe a better\nsolution here is to have your script check for the dependencies it\nneeds (via symlinks, in this case) and include those dependencies in\nthe sparse-checkout it creates.\n\nHope that helps,\nElijah\n\n\n[*] I'll add a slight carve-out to this statement if there was a\ngit-specific way to declare dependencies that we can then parse.  Such\na thing has been proposed before; see\nhttps://lore.kernel.org/git/pull.627.git.1588857462.gitgitgadget@gmail.com/\n.  However, multiple gotchas were identified that derailed that\nproposal, so those would need some solutions.  Even if we were to do\nthat, though, you'd still have to specify the dependency explicitly in\nsome additional file rather than just depending upon the symlink.\nFurther, that particular proposal would have only worked with cone\nmode which goes against your specific request here.\n"}]}