{"thread":{"id":"46219","subject":"Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs","startedAt":"2017-06-21T00:21:26Z","lastAt":"2017-06-21T16:27:07Z","messageCount":2,"participants":["SZEDER Gábor","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"322792","messageId":"CAM0VKjnfVuVnUfnfCceD-QfUdj=6sWTdBDXkj9jZujqMJqdAQA@mail.gmail.com","threadId":"46219","inReplyTo":null,"subject":"Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2017-06-21T00:21:19Z","receivedAt":"2017-06-21T00:21:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sat, Jun 17, 2017 at 2:33 PM, Jeff King <peff@peff.net> wrote:\n> On Sat, Jun 17, 2017 at 02:25:39PM +0200, SZEDER Gábor wrote:\n>> Indeed.  Even in my repos with close to 10k remotes the amount of\n>> memory wasted by the duplicated refspecs is not an problem, there are\n>> more pressing issues there.\n>\n> This is sort of a side note, but I'm curious why you need 10k remotes.\n\nFor fun! :o)\n\nI have a couple of repos containing all branches from all forks (and\nforks of forks, recursively) of a few interesting repos on GitHub,\nwhere each fork is a remote.\n\n  (Why on earth would I want to do that?\n  Sometimes it's good to see what others are up to.  Apparently a lot\n  of people fork projects, push their changes, but then never upstream\n  them.\n  Such an all-forks-in-one repo allows me to run e.g. 'git log --all\n  -p master.. relevant.c' and then search its output for changes in\n  interesting functions (thankfully function names are included in\n  hunk headers; alas line log doesn't work with --all).  Occasionally\n  this unearths some treasures: other people's commits and branches\n  scratching the same itch that I was about to scratch, or at least\n  solving part of my problem and I can build on top of them.)\n\n> We used to do this at GitHub as part of our fork storage (the shared\n> repository had each fork as a remote). We fixed the quadratic issues\n> like d0da003d5 (use a hashmap to make remotes faster, 2014-07-29). But\n> even the linear work to read the config is noticeable (and hits you on\n> every command, even ones that don't care about remotes).\n\nIndeed, that's one of those \"more pressing issues\".\nI worked it around by storing the remotes in a separate config file\nwhich is not included from .git/config, because they're not needed for\nlocal operations.  And when I occasionally do want to fetch, then I\nrun 'git -c include.path=..../.git/config.forks fetch ...'.\n\n> Now instead we\n> pass the refspecs directly to fetch whenever move objects between the\n> storage repos. They were the same for every remote anyway (and I'd guess\n> that is true, too, of your 10k remotes).\n\nI do have different fetch refspecs for every remote, i.e. the repo\n'github.com/user/repo' has '+refs/heads/*:refs/forks/user/repo/*'.\n\nIncidentally, this was what triggered\nsg/clone-refspec-from-command-line-config back then, because 'git\nclone' didn't grab 'refs/forks/*', and the means advertised in the\ndocs to do so didn't work.\n\n\nGábor\n"},{"id":"322815","messageId":"20170621160018.dysakibspeiv3gqr@sigill.intra.peff.net","threadId":"46219","inReplyTo":"CAM0VKjnfVuVnUfnfCceD-QfUdj=6sWTdBDXkj9jZujqMJqdAQA@mail.gmail.com","subject":"Re: Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-21T16:00:18Z","receivedAt":"2017-06-21T16:27:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 21, 2017 at 02:21:19AM +0200, SZEDER Gábor wrote:\n\n>   Such an all-forks-in-one repo allows me to run e.g. 'git log --all\n>   -p master.. relevant.c' and then search its output for changes in\n>   interesting functions (thankfully function names are included in\n>   hunk headers; alas line log doesn't work with --all).  Occasionally\n>   this unearths some treasures: other people's commits and branches\n>   scratching the same itch that I was about to scratch, or at least\n>   solving part of my problem and I can build on top of them.)\n\nOK, that's indeed an interesting use case.\n\n> > Now instead we\n> > pass the refspecs directly to fetch whenever move objects between the\n> > storage repos. They were the same for every remote anyway (and I'd guess\n> > that is true, too, of your 10k remotes).\n> \n> I do have different fetch refspecs for every remote, i.e. the repo\n> 'github.com/user/repo' has '+refs/heads/*:refs/forks/user/repo/*'.\n\nRight, ours have different refspecs, too. But they can be generated\nprogramatically. So we just do:\n\n  git fetch ../$id +refs/*:refs/forks/$id/*\n\nwithout bothering to have a configured remote.\n\n-Peff\n"}]}