Re: Need some help on patching buildin-files // was: Looking for feedback and help with a git-mirror for local usage

From: David Aguilar <davvid@gmail.com>
Date: Jun 14, 2015, 08:17 UTC
Message-ID: <20150614081754.GA15245@gmail.com>
In-Reply-To: <557AB9FC.30102@kr217.de>

On Fri, Jun 12, 2015 at 12:52:44PM +0200, Bernd Naumann wrote:

Show 12 quoted lines

> Hello again,
> 
> After digging the code I may have got a clue where to start but I
> would  still appreciate some help from a developer, cause I have never
> learned to write C. (Some basics at school which happened over a
> decade ago.)
> 
> Currently I have questions on:
> 
> * How to patch clone: would cmd_clone() a good place? Or are there
> other calls which might be better. I think about to insert the check
> if a mirror will be setup or just updated, right after dest_exists.

If you'd still like to modify "git clone" itself, then the "cmd_clone" entry point is certainly the place to start. I would suggest exploring other alternatives, though.

Is it possible to use a caching HTTP proxy, so that "git clone" goes through a local caching proxy? I haven't tried this myself, so maybe it's not even possible, but that seems like a natural http-ish solution.

Another idea is to use Git's URL rewriting feature. If your clone URLs all follow a similar pattern then they can automatically be rewritten to point to some other URL.

e.g. in ~/.gitconfig:

[url "file:///home/git/mirror/github.com/"]
	insteadOf = "https://github.com/"

This will make git clone from /home/git/mirror/github.com/ whenever it sees https://github.com/ URLs.

This is not perfect because it ends up cloning from your local copies rather than setting up the references via --mirror, but at least it avoids hitting the network. You'll need to periodically update your local mirrors, though.

If you prefer to keep ~/.gitconfig pristine then you could do it in a wrapper script by injecting e.g. the "-c" config flags,

	git \
	-c url.file://foo/bar/.insteadOf=https://github.com/ \
	clone ...

Show 9 quoted lines

> [...snip...]
> > 
> > I often build in example 'openwrt' with various build-scripts which
> > depends heavily on a fresh or clean environment and they omit many
> > sources via `git clone`, which results sometimes in over 100 MB of
> > traffic just for one build. /* Later needed .tar.gz source archives
> > are stored in a symlinked download directory which is supported by
> > 'openwrt/.config' since a few months... to reduce network traffic.
> > */

Why does a rebuild delete existing Git repositories? That seems like a bad practice, and shouldn't be needed. If possible, it would be worth improving the build scripts.

For example, a clone can be made pristine by doing "git reset --hard && git clean -fdx". Deleting a repository just so that it can be re-cloned is very wasteful.

Show 7 quoted lines

> > My connection to the internet is not the fastest in world and 
> > sometimes unstable, so I wanted to have some kind of local bare 
> > repository mirror, which is possible with `git clone --mirror`.
> > 
> > From these repositories I can later clone from, by calling `git 
> > clone --reference /path/to.git <url>`, but I do not wish to edit 
> > all the build-scripts and Makefiles.

Maybe it'd be possible to make just the "git clone" part of the build scripts configurable?

That'd make it really easy to inject a wrapper script that scans the arguments and injects the needed --mirror arguments, in the case that the above options won't work.

Show 27 quoted lines

> > So I wrote a git wrapper script (`$HOME/bin/git`), which checks if
> >  `git` was called with 'clone', and if so, then it will first 
> > clones the repository as a mirror and then clones from that local 
> > mirror. If the mirror already exists, then it will only be updated 
> > (`git remote update`). This works for now.
> > 
> > [...snip...]
> > 
> > Ok, so far, so good, but the implementation of the current 
> > shell-prototype looks way too hacky [0] and I have found some edge
> >  cases on which my script will fail: The script depends on the
> > fact that the last, or at least the second last argument is a
> > valid git-url, but the following is a valid call, too :
> > 
> > `git --no-pager \ clone git@github.com:openwrt/packages.git 
> > openwrt-packages --depth 1`
> > 
> > But this is not valid:
> > 
> > `git clone https://github.com/openwrt/packages.git --reference 
> > packages.git packages-2` or `git clone --verbose 
> > https://github.com/openwrt/packages.git packages-2 --reference 
> > packages.git`
> > 
> > 
> > I found out that git-clone actually also can only make a guess
> > what is the url and what not.

Another option is to rewrite the wrapper script in a better language. For example, Python's argparse module can handle the above cases with minimal fuss.

Anyways, as I said before, the root problem is really the build scripts. I bet modifying the build scripts to reuse existing git repositories is easier than modifying "git clone".

cheers,

-- 
David