{"thread":{"id":"29376","subject":"Simulating an empty git repository without having said repository on disk","startedAt":"2012-01-16T18:34:04Z","lastAt":"2012-01-16T23:09:56Z","messageCount":3,"participants":["Richard Hartmann","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"182617","messageId":"CAD77+gR=txp8sKrA57ztQX0a1-QZM7wwR6ThBq77c=c+AbsS0w@mail.gmail.com","threadId":"29376","inReplyTo":null,"subject":"Simulating an empty git repository without having said repository on disk","fromName":"Richard Hartmann","fromEmail":"richih.mailinglist@gmail.com","sentAt":"2012-01-16T18:34:04Z","receivedAt":"2012-01-16T18:34:04Z","isPatch":false,"sender":{"key":"richih.mailinglist@gmail.com","avatar":"https://avatars.githubusercontent.com/u/754723?v=4"},"body":"Hi all,\n\nfor vcsh[1], I need a rather hackish feature: List all files untracked\nby vcsh. The plan to achieve this is:\n\nGet lists of all files by all repos which' GIT_WORK_TREE is in one\ndirectory ($HOME, by default), merge all lists into one and use that\nas a .gitignore or exclude. Then run `git status` with $GIT_WORK_TREE\npointing to $HOME while using said ignore/exclude. That will give me a\nlist of all files & directories which are not tracked by any of the\ngit repos managed by vcsh.\n\nI could create an empty git repo to run this operation in, but that\nseems wasteful. Thus, I would prefer to run this command against a\nnon-existing, empty git repo. Problem is, I could not find a way to do\nthis.\n\n\nI know the empty tree's SHA is hard-coded into git so I was hoping\nthere would be a way to trick git using this, but I couldn't find a\nway.\n\n\nAny and all help appreciated, even if it's just a \"no, this is not possible\"\n\n\nThanks,\nRichard\n\n\n[1] https://github.com/RichiH/vcsh\n"},{"id":"182630","messageId":"20120116204131.GC18699@sigill.intra.peff.net","threadId":"29376","inReplyTo":"CAD77+gR=txp8sKrA57ztQX0a1-QZM7wwR6ThBq77c=c+AbsS0w@mail.gmail.com","subject":"Re: Simulating an empty git repository without having said repository on disk","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-01-16T20:41:31Z","receivedAt":"2012-01-16T20:41:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 16, 2012 at 07:34:04PM +0100, Richard Hartmann wrote:\n\n> for vcsh[1], I need a rather hackish feature: List all files untracked\n> by vcsh. The plan to achieve this is:\n> \n> Get lists of all files by all repos which' GIT_WORK_TREE is in one\n> directory ($HOME, by default), merge all lists into one and use that\n> as a .gitignore or exclude. Then run `git status` with $GIT_WORK_TREE\n> pointing to $HOME while using said ignore/exclude. That will give me a\n> list of all files & directories which are not tracked by any of the\n> git repos managed by vcsh.\n\nI don't use vcsh, but I seem to recall that it works by overlaying the\nworking trees of different repositories on each other, right? So you\ncan't just say \"oh, files in foo/ belong to repository 'bar'\". You must\ntake the union of the set of tracked files from all repos, then find the\ndifference of that from the set of all files.\n\nCan individual repos mark things as excluded, too? Or do you have a\nmaster exclusion list? I.e., if I decide that I don't want \"foo\" tracked\nat all, how do I tell vcsh?\n\n> I could create an empty git repo to run this operation in, but that\n> seems wasteful. Thus, I would prefer to run this command against a\n> non-existing, empty git repo. Problem is, I could not find a way to do\n> this.\n> \n> I know the empty tree's SHA is hard-coded into git so I was hoping\n> there would be a way to trick git using this, but I couldn't find a\n> way.\n\nI'm not sure why you care about the empty tree if you are only looking\nat untracked files. Or perhaps the problem is that you are using \"git\nstatus\", which fundamentally cares about looking at differences between\nHEAD and the index, even though you don't care in this case. In that case,\nmaybe \"git ls-files -o\" would be more appropriate?\n\nThe most straightforward way in git would be to generate a temporary\nindex that mentions all of the tracked files, like this:\n\n  tmp=/some/tmp/index\n  rm -f $tmp\n  for i in repo; do\n    git --git-dir=$repo ls-files -z |\n      GIT_INDEX_FILE=$tmp xargs -0 git update-index --add\n  done\n  GIT_INDEX_FILE=$tmp git ls-files -o\n\nbut that is very close to your \"create an empty git repo\" (in fact, you\nmight even need to in order for update-index to be happy). But it would\ngive you a place to put a master exclusion list (you would use it as\n--exclude=... in the final ls-files).\n\nIf you have per-repo exclusion lists, then you could take a different\napproach, and simply run \"git ls-files -o\" for each repository. Any\nfiles which appeared in _every_ output would be untracked (since tracked\nfiles or individually-excluded files would be omitted from at least one\nrepo). Like:\n\n  # get the list of untracked files from each repo's perspective\n  count=0\n  for i in repo; do\n    count=$(($count + 1))\n    git --git-dir=$repo ls-files -o\n  done >output\n\n  # now count how many times each entry appears. Truly untracked things\n  # appear $count times.\n  sort <output |\n  uniq -c |\n  perl -lne \"/^\\s*$count (.*)/ and print \\$1\"\n\nThe downside is that you are doing $count traversals of the untracked\ndirectories. On an OS with a reasonable lstat and a directory structure\nthat fits into cache, that is probably not too big a deal, though.\n\n> Any and all help appreciated, even if it's just a \"no, this is not possible\"\n\nI took a lot of guesses at exactly what you want. It might be more clear\nif you gave us an example situation along with the output you expect.\n\n-Peff\n"},{"id":"182646","messageId":"CAD77+gSB+aYjrMk+zV7BVstGME7Mmxtxg0jkG9KvY3vxwrvvaw@mail.gmail.com","threadId":"29376","inReplyTo":"20120116204131.GC18699@sigill.intra.peff.net","subject":"Re: Simulating an empty git repository without having said repository on disk","fromName":"Richard Hartmann","fromEmail":"richih.mailinglist@gmail.com","sentAt":"2012-01-16T23:09:56Z","receivedAt":"2012-01-16T23:09:56Z","isPatch":false,"sender":{"key":"richih.mailinglist@gmail.com","avatar":"https://avatars.githubusercontent.com/u/754723?v=4"},"body":"On Mon, Jan 16, 2012 at 21:41, Jeff King <peff@peff.net> wrote:\n\n> I don't use vcsh, but I seem to recall that it works by overlaying the\n> working trees of different repositories on each other, right?\n\nIn parallel, but yes.\n\n\n> So you\n> can't just say \"oh, files in foo/ belong to repository 'bar'\". You must\n> take the union of the set of tracked files from all repos, then find the\n> difference of that from the set of all files.\n\nCorrect.\n\n\n> Can individual repos mark things as excluded, too? Or do you have a\n> master exclusion list? I.e., if I decide that I don't want \"foo\" tracked\n> at all, how do I tell vcsh?\n\nThat's something I am still contemplating as there are several ways:\n\n* excludes\n* pre-/appends to the gitignore of every repo\n* runtime magic\n\nFeedback welcome :)\n\n\n> I'm not sure why you care about the empty tree if you are only looking\n> at untracked files. Or perhaps the problem is that you are using \"git\n> status\", which fundamentally cares about looking at differences between\n> HEAD and the index, even though you don't care in this case. In that case,\n> maybe \"git ls-files -o\" would be more appropriate?\n\n--others does not work as I need to look at several repos. I tried to\nget the union of --others, but that creates 'argument too large' very\nquickly.\n\nInitially, I tried with find, but as that is depth-first, it takes\nages when compared to git's early stopping at directories.\n\n\n> The most straightforward way in git would be to generate a temporary\n> index that mentions all of the tracked files, like this:\n>\n>  tmp=/some/tmp/index\n>  rm -f $tmp\n>  for i in repo; do\n>    git --git-dir=$repo ls-files -z |\n>      GIT_INDEX_FILE=$tmp xargs -0 git update-index --add\n>  done\n>  GIT_INDEX_FILE=$tmp git ls-files -o\n>\n> but that is very close to your \"create an empty git repo\" (in fact, you\n> might even need to in order for update-index to be happy). But it would\n> give you a place to put a master exclusion list (you would use it as\n> --exclude=... in the final ls-files).\n>\n> If you have per-repo exclusion lists, then you could take a different\n> approach, and simply run \"git ls-files -o\" for each repository. Any\n> files which appeared in _every_ output would be untracked (since tracked\n> files or individually-excluded files would be omitted from at least one\n> repo). Like:\n\nSee above, but I will try yours as well.\n\n\n>  perl -lne \"/^\\s*$count (.*)/ and print \\$1\"\n\nI know I sound picky, but I would also like to avoid any third-party\ndependencies if possible. Perl is common, but not installed\neverywhere.\n\n\n> The downside is that you are doing $count traversals of the untracked\n> directories. On an OS with a reasonable lstat and a directory structure\n> that fits into cache, that is probably not too big a deal, though.\n\nWith cold cache, it can take ages. Especially once you have a few\ngit-annex repos in $HOME.\n\n\n> I took a lot of guesses at exactly what you want. It might be more clear\n> if you gave us an example situation along with the output you expect.\n\nrepo foo tracks .foo and .foo.d, bar .bar, etc\n\n% vcsh list #lists repos\nfoo\nbar\nbaz\n% ls -aR\n.foo\n.foo.d/\n.bar\n.baz.d/\n.quux.d/\n.quux.d/foo\n.quux.d/quux.d/quux\n.quux.d/quux.d/quuux\n.quux.d/quux.d/quuuux\n.quux.d/quuuux.d/quuuux\npants\nshirts\n% vcsh run foo git ls-files # run command in context of repo foo\n.foo\n.foo.d\n.quux.d/foo\n% vcsh list-untracked # with the code I want in it\n.quux.d/quux.d/\n.quux.d/quuuux.d/\npants\nshirts\n%\n\nI hope that makes sense.\n\nThe only part that does not already work today is list-untracked.\n\nFor failed attempts look at\nhttps://github.com/RichiH/vcsh/tree/list-untracked\nhttps://github.com/RichiH/vcsh/tree/list-untracked-2\n\n\nThanks,\nRichard\n"}]}