Re: [RFC] How to pass Git config command line instructions to Submodule commands?
- From
Stefan Beller <sbeller@google.com>
- Date
- Apr 25, 2016, 17:02 UTC
- Message-ID
- <CAGZ79kYmAr-O6_Jw2KO9eZEfZQ+_WBiERF=nhOYLJCZpUjSSyA@mail.gmail.com>
- In-Reply-To
- <60724588-B06E-47E8-9302-8709C4601826@gmail.com>
On Mon, Apr 25, 2016 at 3:39 AM, Lars Schneider <larsxschneider@gmail.com> wrote:
Show 23 quoted lines
> Hi, > > a few folks from the Git LFS project and I try to make cloning of repositories > with a lot of LFS files faster. > > The core problem is that Git LFS uses a Git smudge filter to replace LFS > pointers with the actual file content. Right now, a smudge filter can only > be executed on an individual file which makes the operation slow for many > files [1]. > > We solved this issue by temporarily disabling the smudge filter for the clone > command via Git config (optimized in 1a8630 [2]): > > git -c filter.lfs.smudge= -c filter.lfs.required=false clone <url> <path> > > Afterwards Git LFS runs a special command to download and replace all LFS > content in bulk [3]. This works great for LFS repositories. > > However, I noticed that git config command line instructions such as > "-c filter.lfs.smudge=" are not passed to Git submodule operations. Thus > this does not work as expected: > > git -c filter.lfs.smudge= -c filter.lfs.required=false clone --recursive <url> <path>
I have cc'd Jacob Keller, who authored origin/jk/submodule-c-credential, which does work in that area (deciding which config option to pass down into the submodule commands).
Show 22 quoted lines
>
> I tried to work around that by copying the relevant pieced from the Git
> Submodule command [4] and applying the command line Git config
> manually (look closely at the modified checkout command):
>
> git -c filter.lfs.smudge= -c filter.lfs.required=false clone $@
> if [[ -z $2 ]]; then
> CLONE_PATH=$(basename ${1%.git});
> else
> CLONE_PATH=$2;
> fi
> pushd "$CLONE_PATH"
> git submodule init
> wt_prefix=$(git rev-parse --show-prefix)
> git submodule--helper list --prefix "$wt_prefix" | {
> while read mode sha1 stage sm_path
> do
> name=$(git submodule--helper name "$sm_path") || exit
> url=$(git config submodule."$name".url)
> if ! test -d "$sm_path"/.git && ! test -f "$sm_path"/.git
> then
> git submodule--helper clone --prefix "$wt_prefix" --path "$sm_path" --name "$name" --url "$url"The init and then clone should be covered by "git submodule--helper update-clone", which may be better named as "list-and-clone-if-necessary", then you get parallel cloning for free as well. ;)
Show 12 quoted lines
> pushd "$sm_path" > git -c filter.lfs.smudge= -c filter.lfs.required=false checkout -q $sha1 || exit > git-lfs pull || exit > popd > fi > done > } > popd > > Do you see an easier way to pass command line Git config instructions to the > underlaying Git Submodule commands? If not, do you think a patch adding this > would be worth working on?
I would build on top of origin/jk/submodule-c-credential at least, and using "git submodule--helper update-clone" (origin/sb/submodule-parallel-update)
> > I also started a discussion about that on the Git LFS issue page [5].
Unrelated to this, but about LFS: Currently it is only possible to store the big blobs at a $VENDOR hosting site. How would you backup you whole repository including all binariers?
I think we should add an option to store one blob in a dedicated ref (refs/lfs/$blobsha1 or such). That way you can make a backup of the repository including all large files using "git clone --mirror" and you don't have to rely on the $VENDOR hosting your files.
Thanks, Stefan
Show 11 quoted lines
> > Thanks, > Lars > > > [1] https://github.com/github/git-lfs/issues/931 > [2] https://github.com/git/git/commit/1a8630dc3b1cc6f1361a4e5d94630133c24c97d9 > [3] https://developer.atlassian.com/blog/2016/04/git-lfs-12-clone-faster/ > [4] https://github.com/git/git/blob/6a6636270fbaf74609cd3e1bd207dd2c420d640a/git-submodule.sh#L686-L731 > [5] https://github.com/github/git-lfs/issues/1172 >