{"thread":{"id":"39605","subject":"Looking for feedback and help with a git-mirror for local usage","startedAt":"2015-06-11T20:44:08Z","lastAt":"2015-06-14T08:17:56Z","messageCount":3,"participants":["Bernd Naumann","David Aguilar"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"263640","messageId":"5579F318.7050503@kr217.de","threadId":"39605","inReplyTo":null,"subject":"Looking for feedback and help with a git-mirror for local usage","fromName":"Bernd Naumann","fromEmail":"bernd@kr217.de","sentAt":"2015-06-11T20:44:08Z","receivedAt":"2015-06-11T20:44:08Z","isPatch":false,"sender":{"key":"bernd@kr217.de","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nHello,\n\nI have came up with an idea\n# Yep I know, exactly that kind of e-mail everyone wants to read ;)\nand I'm working currently on a shell-prototype to face the following\nsituation and problem and need some feedback/advise:\n\n\nI often build in example 'openwrt' with various build-scripts which\ndepends heavily on a fresh or clean environment and they omit many\nsources via `git clone`, which results sometimes in over 100 MB of\ntraffic just for one build. /* Later needed .tar.gz source archives\nare stored in a symlinked download directory which is supported by\n'openwrt/.config' since a few months... to reduce network traffic. */\n\nMy connection to the internet is not the fastest in world and\nsometimes unstable, so I wanted to have some kind of local bare\nrepository mirror, which is possible with `git clone --mirror`.\n\n- From these repositories I can later clone from, by calling `git clone\n- --reference /path/to.git <url>`, but I do not wish to edit all the\nbuild-scripts and Makefiles.\n\n\nSo I wrote a git wrapper script (`$HOME/bin/git`), which checks if\n`git` was called with 'clone', and if so, then it will first clones\nthe repository as a mirror and then clones from that local mirror. If\nthe mirror already exists, then it will only be updated (`git remote\nupdate`). This works for now.\n\n/*\n  To be able to have multiple identical named repositories,\n  the script builds paths like:\n\n  ~/var/cache/gitmirror $ find . -name \"*.git\"\n\n  ./github.com/openwrt-management/packages.git\n  ./github.com/openwrt/packages.git\n  ./github.com/openwrt-routing/packages.git\n  ./nbd.name/packages.git\n  ./git.openwrt.org/packages.git\n  ./git.openwrt.org/openwrt.git\n\nIt strips the schema from the url and replaces \":\" with \"/\" in case a\nport is specified or a svn link is provided. The remaining should be a\nvalid linux file and directory structure, if I guess correctly!?\n*/\n\nOk, so far, so good, but the implementation of the current\nshell-prototype looks way too hacky [0] and I have found some edge\ncases on which my script will fail:\n  The script depends on the fact that the last, or at least the second\nlast argument is a valid git-url, but the following is a valid call, too\n:\n\n  `git --no-pager \\\n   clone git@github.com:openwrt/packages.git openwrt-packages --depth 1`\n\nBut this is not valid:\n\n`git clone https://github.com/openwrt/packages.git --reference\npackages.git packages-2`\nor\n`git clone --verbose https://github.com/openwrt/packages.git\npackages-2 --reference packages.git`\n\n\nI found out that git-clone actually also can only make a guess what is\nthe url and what not.\n\n\n\nHowever, now I'm looking for a way to write something like a submodul\nfor git which will check for a *new* git-config value like\n\"user.mirror\" (or something...) which points to a directory, and will\nbe used to clone from, and in case of 'fetch', 'pull' or 'remote\nupdate' update the mirror first, and then the update of the current\nworking directory is gotten from that mirror. (And in case of 'push'\nthe mirror would be updated from the working dir, of course.)\n\n\nI would like to hear some toughs on that, and how I could start to\nbuild this submodul, or if someone more talented, then I am, is willed\nto spent some time on that. If requested/wished I could send a link to\nthe shell-prototype.\n\n\n[0] For a reason I have to do ugly things like\n`$( eval exec /usr/bin/git clone --mirror $REPO_URL ) 2>&1 >/dev/null`\ncause otherwise in case of just `eval exec` the script stops after\nexecution, and without `eval exec` arguments with spaces will\ninterpreted as seperated arguments, which is no good, because of failing\n.\n\n\nThanks for your time!\nYours faithfully, Bernd\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v2.0.19 (GNU/Linux)\n\niQIcBAEBAgAGBQJVefMYAAoJEEYW3OihUKBPJkAP/iiFBoHnJXloX0SRQHjEUBDf\nC5PQ/42IZTB5ghM959IBA0QZ4p4BEFcwu8q7xKKE2FtiUzAAb1hRiXZOV7S7DZ1s\niPDCOk8hTp9eqSLgfDL6WX7ztGByFoT9GodwpTFBLU31RvooWO1BYc/jrd3lMA4k\n4lk+8SM1dOffJm0g+A4YCsE59P7Rn/t0iJYepaN7cXVMdgKvuZ0iVi9CvHAipPUG\nxuCwNYCM6tcOvnjZH/Nqa57+l5LfQ7qIA6YBlG77wOwDHgX+GPkYAqq+xOq28aP0\n+W7duf32SgkQBwSTnfntYd4G+QZqIktP30Ik0e8hCcU37ECcEP2s28CebY2825/n\nFaZEutK3sE+lk47j89ndvPdtpHybchUi/0zNftPY0ngU6Yc/0YMMq9KeYM6kt6+s\n8coSvt5AQLhgR+NMQhXF4nKtfcvt9B+xZtag6Re/zA8AwIrBFFvu7dGkvG9aydDe\nZwvt4/ddYxEouEPhwr4+KmmM2ll8tHoBcJJYr+xoqQlE/nSPfF/gvsQVqciijp4b\nafyStwFYGHPo68pMvEZx+xXYaAhkKSAvaN5vupy1e5765E0F5DWOV5P026L45D7V\nyKFVa/eYZc/iJjQcjzpch9mq/Jiblht6XXR1YDlHg5PoKE3Chs8EjYp0wyPWqGS0\nlrCPzhwrMLVmmksF0wcN\n=P4eQ\n-----END PGP SIGNATURE-----\n"},{"id":"263655","messageId":"557AB9FC.30102@kr217.de","threadId":"39605","inReplyTo":"5579F318.7050503@kr217.de","subject":"Re: Need some help on patching buildin-files // was: Looking for feedback and help with a git-mirror for local usage","fromName":"Bernd Naumann","fromEmail":"bernd@kr217.de","sentAt":"2015-06-12T10:52:44Z","receivedAt":"2015-06-12T10:52:44Z","isPatch":false,"sender":{"key":"bernd@kr217.de","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nHello again,\n\nAfter digging the code I may have got a clue where to start but I\nwould  still appreciate some help from a developer, cause I have never\nlearned to write C. (Some basics at school which happened over a\ndecade ago.)\n\nCurrently I have questions on:\n\n* How to patch clone: would cmd_clone() a good place? Or are there\nother calls which might be better. I think about to insert the check\nif a mirror will be setup or just updated, right after dest_exists.\n\n* Is it correct that a new config key just get specified via a config\nfile or by cmd_init_db()? So later, a check on that value is enough?\nWould be the section 'user' a good place for this key or is it\nsomething that would get a own/new section?\n\n* Have I missed a relevant file?\n\ngit/git.c\ngit/builtin/clone.c\ngit/builtin/fetch.c\ngit/builtin/push.c\ngit/buildin/remote.c\nalong with the translation and Documentation, of course.\n\n\nIf you have some comments on that, please share these with me, and if\nyou are interested in helping me to got this implemented, I would\nappreciate that :)\n\nSincere regards,\nBernd\n\n\nOn 06/11/2015 10:44 PM, Bernd Naumann wrote:\n> Hello,\n> \n> I have came up with an idea # Yep I know, exactly that kind of \n> e-mail everyone wants to read ;) and I'm working currently on a \n> shell-prototype to face the following situation and problem and \n> need some feedback/advise:\n> \n> \n> I often build in example 'openwrt' with various build-scripts which\n> depends heavily on a fresh or clean environment and they omit many\n> sources via `git clone`, which results sometimes in over 100 MB of\n> traffic just for one build. /* Later needed .tar.gz source archives\n> are stored in a symlinked download directory which is supported by\n> 'openwrt/.config' since a few months... to reduce network traffic.\n> */\n> \n> My connection to the internet is not the fastest in world and \n> sometimes unstable, so I wanted to have some kind of local bare \n> repository mirror, which is possible with `git clone --mirror`.\n> \n> From these repositories I can later clone from, by calling `git \n> clone --reference /path/to.git <url>`, but I do not wish to edit \n> all the build-scripts and Makefiles.\n> \n> \n> So I wrote a git wrapper script (`$HOME/bin/git`), which checks if\n>  `git` was called with 'clone', and if so, then it will first \n> clones the repository as a mirror and then clones from that local \n> mirror. If the mirror already exists, then it will only be updated \n> (`git remote update`). This works for now.\n> \n> /* To be able to have multiple identical named repositories, the \n> script builds paths like:\n> \n> ~/var/cache/gitmirror $ find . -name \"*.git\"\n> \n> ./github.com/openwrt-management/packages.git \n> ./github.com/openwrt/packages.git \n> ./github.com/openwrt-routing/packages.git ./nbd.name/packages.git \n> ./git.openwrt.org/packages.git ./git.openwrt.org/openwrt.git\n> \n> It strips the schema from the url and replaces \":\" with \"/\" in\n> case a port is specified or a svn link is provided. The remaining\n> should be a valid linux file and directory structure, if I guess \n> correctly!? */\n> \n> Ok, so far, so good, but the implementation of the current \n> shell-prototype looks way too hacky [0] and I have found some edge\n>  cases on which my script will fail: The script depends on the\n> fact that the last, or at least the second last argument is a\n> valid git-url, but the following is a valid call, too :\n> \n> `git --no-pager \\ clone git@github.com:openwrt/packages.git \n> openwrt-packages --depth 1`\n> \n> But this is not valid:\n> \n> `git clone https://github.com/openwrt/packages.git --reference \n> packages.git packages-2` or `git clone --verbose \n> https://github.com/openwrt/packages.git packages-2 --reference \n> packages.git`\n> \n> \n> I found out that git-clone actually also can only make a guess\n> what is the url and what not.\n> \n> \n> \n> However, now I'm looking for a way to write something like a \n> submodul for git which will check for a *new* git-config value like\n> \"user.mirror\" (or something...) which points to a directory, and\n> will be used to clone from, and in case of 'fetch', 'pull' or \n> 'remote update' update the mirror first, and then the update of\n> the current working directory is gotten from that mirror. (And in\n> case of 'push' the mirror would be updated from the working dir,\n> of course.)\n> \n> \n> I would like to hear some toughs on that, and how I could start to\n>  build this submodul, or if someone more talented, then I am, is \n> willed to spent some time on that. If requested/wished I could\n> send a link to the shell-prototype.\n> \n> \n> [0] For a reason I have to do ugly things like `$( eval exec \n> /usr/bin/git clone --mirror $REPO_URL ) 2>&1 >/dev/null` cause \n> otherwise in case of just `eval exec` the script stops after \n> execution, and without `eval exec` arguments with spaces will \n> interpreted as seperated arguments, which is no good, because of \n> failing .\n> \n> \n> Thanks for your time! Yours faithfully, Bernd -- To unsubscribe \n> from this list: send the line \"unsubscribe git\" in the body of a \n> message to majordomo@vger.kernel.org More majordomo info at \n> http://vger.kernel.org/majordomo-info.html\n> \n\n- -- \nBernd Naumann <bernd@kr217.de>\n\nPGP:   0xA150A04F via pool.sks-keyservers.net\nXMPP:  bn@weimarnetz.de\n\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v2.0.19 (GNU/Linux)\n\niQIcBAEBAgAGBQJVern8AAoJEEYW3OihUKBPLPYP/2RYrqH7qtH9ZVCc+dN6kEMb\nKzEFiSF7Vg7EIcSYIeyw7SS8M/3QyBHmdClq6Gcgby7yAYuSsXcY4V0xja12cI2g\nglH+5kXfZg11shJdi530GGLNVyTaLhhNUxqmrB56FHP31nOeFGEYzLhDs16mh4z8\n2YiN4wT62O8R/yjRReaeRBe2cTniga1ZeDVFgYGE2atWmGOb2DLEfDyxAWIUeu0r\nRfLF7NPb5ZLAlSBfifmMeRJ7Fu8Ewf0m3ESZoBlD3+CW68k7vefTo+iVvmRJnkEl\n0p89IJMCdPEwTsXwXUMnI0xcofM9tLthGQ+x482rQTxUYvkzQnjT2vBc/DTe08Ok\n+xS4JaZl+22IlyRt8KFJOLHhZfuZgYOlGqoHqxbIPZyNvR+AtuGRSGdGEJoc2ACb\naij+smTlN3k8X3DZVGPNlsNaFCRVgGin2Yad4pOIk/mlkR6xx3LfFB6qv8mSoj0z\nkmFDPAdYWlGps+hPeM76Ql9UN+wgcD+1y2TpMJGyUy4YoGmOQq4TWO1JMUVjWWie\nie5Yf/JD8fopmrV3BM2sT7gLmi75zpsc3Em/i6S5zUIhZ0v+UfNE5dSim0KcAgyS\nI0apoXZZw3UjaTilmd3h/ecuTa1lBygNucnGnDfm3NAGYDWjy0LR8Mu7+2rBogvk\nJCF1hf3qka80GEwsZqu+\n=3nO1\n-----END PGP SIGNATURE-----\n"},{"id":"263787","messageId":"20150614081754.GA15245@gmail.com","threadId":"39605","inReplyTo":"557AB9FC.30102@kr217.de","subject":"Re: Need some help on patching buildin-files // was: Looking for feedback and help with a git-mirror for local usage","fromName":"David Aguilar","fromEmail":"davvid@gmail.com","sentAt":"2015-06-14T08:17:56Z","receivedAt":"2015-06-14T08:17:56Z","isPatch":false,"sender":{"key":"davvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/13196?v=4"},"body":"On Fri, Jun 12, 2015 at 12:52:44PM +0200, Bernd Naumann wrote:\n> Hello again,\n> \n> After digging the code I may have got a clue where to start but I\n> would  still appreciate some help from a developer, cause I have never\n> learned to write C. (Some basics at school which happened over a\n> decade ago.)\n> \n> Currently I have questions on:\n> \n> * How to patch clone: would cmd_clone() a good place? Or are there\n> other calls which might be better. I think about to insert the check\n> if a mirror will be setup or just updated, right after dest_exists.\n\nIf you'd still like to modify \"git clone\" itself, then the\n\"cmd_clone\" entry point is certainly the place to start.\nI would suggest exploring other alternatives, though.\n\n\nIs it possible to use a caching HTTP proxy, so that \"git clone\"\ngoes through a local caching proxy?  I haven't tried this myself,\nso maybe it's not even possible, but that seems like a natural\nhttp-ish solution.\n\n\nAnother idea is to use Git's URL rewriting feature.  If your\nclone URLs all follow a similar pattern then they can\nautomatically be rewritten to point to some other URL.\n\ne.g. in ~/.gitconfig:\n\n[url \"file:///home/git/mirror/github.com/\"]\n\tinsteadOf = \"https://github.com/\"\n\nThis will make git clone from /home/git/mirror/github.com/\nwhenever it sees https://github.com/ URLs.\n\nThis is not perfect because it ends up cloning from your local\ncopies rather than setting up the references via --mirror, but\nat least it avoids hitting the network.  You'll need to\nperiodically update your local mirrors, though.\n\nIf you prefer to keep ~/.gitconfig pristine then you could do it\nin a wrapper script by injecting e.g. the \"-c\" config flags,\n\n\tgit \\\n\t-c url.file://foo/bar/.insteadOf=https://github.com/ \\\n\tclone ...\n\n> [...snip...]\n> > \n> > I often build in example 'openwrt' with various build-scripts which\n> > depends heavily on a fresh or clean environment and they omit many\n> > sources via `git clone`, which results sometimes in over 100 MB of\n> > traffic just for one build. /* Later needed .tar.gz source archives\n> > are stored in a symlinked download directory which is supported by\n> > 'openwrt/.config' since a few months... to reduce network traffic.\n> > */\n\nWhy does a rebuild delete existing Git repositories?\nThat seems like a bad practice, and shouldn't be needed.\nIf possible, it would be worth improving the build scripts.\n\nFor example, a clone can be made pristine by doing\n\"git reset --hard && git clean -fdx\".  Deleting a repository\njust so that it can be re-cloned is very wasteful.\n\n> > My connection to the internet is not the fastest in world and \n> > sometimes unstable, so I wanted to have some kind of local bare \n> > repository mirror, which is possible with `git clone --mirror`.\n> > \n> > From these repositories I can later clone from, by calling `git \n> > clone --reference /path/to.git <url>`, but I do not wish to edit \n> > all the build-scripts and Makefiles.\n\nMaybe it'd be possible to make just the \"git clone\" part of the\nbuild scripts configurable?\n\nThat'd make it really easy to inject a wrapper script that scans\nthe arguments and injects the needed --mirror arguments, in the\ncase that the above options won't work.\n\n\n> > So I wrote a git wrapper script (`$HOME/bin/git`), which checks if\n> >  `git` was called with 'clone', and if so, then it will first \n> > clones the repository as a mirror and then clones from that local \n> > mirror. If the mirror already exists, then it will only be updated \n> > (`git remote update`). This works for now.\n> > \n> > [...snip...]\n> > \n> > Ok, so far, so good, but the implementation of the current \n> > shell-prototype looks way too hacky [0] and I have found some edge\n> >  cases on which my script will fail: The script depends on the\n> > fact that the last, or at least the second last argument is a\n> > valid git-url, but the following is a valid call, too :\n> > \n> > `git --no-pager \\ clone git@github.com:openwrt/packages.git \n> > openwrt-packages --depth 1`\n> > \n> > But this is not valid:\n> > \n> > `git clone https://github.com/openwrt/packages.git --reference \n> > packages.git packages-2` or `git clone --verbose \n> > https://github.com/openwrt/packages.git packages-2 --reference \n> > packages.git`\n> > \n> > \n> > I found out that git-clone actually also can only make a guess\n> > what is the url and what not.\n\nAnother option is to rewrite the wrapper script in a better language.\nFor example, Python's argparse module can handle the above cases\nwith minimal fuss.\n\nAnyways, as I said before, the root problem is really the build\nscripts.  I bet modifying the build scripts to reuse existing\ngit repositories is easier than modifying \"git clone\".\n\ncheers,\n-- \nDavid\n"}]}