Shallow submodule efficiency

Hi!

I have the feeling that “git submodule update --depth 1” is less clever than it could be. Here is one example I observed with git 2.0.0:

  git init foo
  cd foo
  git clone --single-branch \
            -b v0.99 https://github.com/git/git.git git-scm
  git submodule add https://github.com/git/git.git git-scm
  git commit -m Submod
  git clone --dissociate . ../bar
  cd ../bar
  git submodule update --init --depth 1 git-scm

This will download quite a bit of history, then result in an error message:

  error: no such remote ref a3eb250f996bf5e12376ec88622c4ccaabf20ea8
  Fetched in submodule path 'git-scm', but it did not contain
  a3eb250f996bf5e12376ec88622c4ccaabf20ea8. Direct fetching of that
  commit failed.

That seems so avoidable, since the commit in question is a tag, so it would be perfectly possible to fetch that specific commit from the server directly. Something like the following commands would do the trick:

  git fetch $url $(git ls-remote $url | \
                   awk /$sha1/'{print $2}' | sed 's/\^{}//')

If the commit in question is NOT a ref, then whether asking for it by unlisted SHA1 is supported will probably depend on the server's uploadpack.allowReachableSHA1InWant setting. I guess this is a reason why fb43e31 made the fetch for a specific SHA1 a fallback after the fetch for the default branch. Nevertheless, in case of “--depth 1” I think it would make sense to abort early: if none of the listed refs matches the requested one, and asking by SHA1 isn't supported by the server, then there is no point in fetching anything, since we won't be able to satisfy the submodule requirement either way.

For the case of “--depth n” with n > 1, I was wondering whether it would make sense to prefer the branch listed in submodule.‹name›.branch over the default branch.

I think shallow submodules would be very useful to embed libraries into projects, without too much care for history (and without the download times getting it entails), but with efficient updates to affected files only in case of a change in library version. But not being able to get a specific tag as a shallow submodule is a major showstopper here, I think.

Greetings,
 Martin von Gagern

On Mon, Jun 27, 2016 at 10:39 PM, Martin von Gagern <Martin.vGagern@gmx.net> wrote:

2.9.0 (as "Direct fetching of " is not part of 2.0.0 IIRC) ?

Yeah there are a few things going on, which try to cover up an error in design IMO.

* The depth is measured from the tip of a branch in the submodule,
   not from the sha1 that the superproject points to.
* Shallowness is treated separately in the superproject and submodules as they
  have a strong notion of being independent. It would be cool to have a thing
  `git clone --recurse-submodules --depth=15
--submodule-depth-as-reachable-from-superproject`
  which would obtain the submodules as shallow as possible, but it
includes all versions that
  the 15 commits in the superproject points to. (may be 1 up to 15
  different non-sequential versions)

* `git submodule update --init --depth 1` is using clone instead of fetch
  currently when the submodule doesn't exist yet. The clone is buried in
  the `submodule--helper update-clone` that is a mixture of listing
the submodules
  and cloning multiple submodules in parallel if possible. So I would
assume it is
  easier to teach git clone to behave correctly and then stop retrying
in git-submodule.sh
  if `just_cloned` is set in the `cmd_update()`.

Makes sense! I think the easiest way forward to implement this will be:

* `git clone` learns a (maybe undocumented internal) option `--get-sha1`
  `--branch` looks similar to what we want, but doesn't quite fit as we do not
  know, whether we're on a tag or not. The submodule tells us just the
  recorded sha1, not the branch/tag. So maybe we'd end up calling it
  `--detach-at=<sha1>`, that will
  -> inspect the ls-remote for the sha1 being there
  -> if the sha1 is there (at least once) clone as if --branch <tag> was given
  -> if not found and the server advertised  allowReachableSHA1InWant,
try again inside the clone

* `submodule--helper update-clone` passes the  `--get-sha1` to the
clones of the submodules

* cmd_update() in git-submodule.sh will only checkout submodules and
not try again
  to fetch them if `just_cloned` is set as the cloning did the best it could.

Makes sense to me.

Thanks for taking your time to point this out and start this discussion!

Thanks, Stefan

Hi Stefan,

On 28.06.2016 19:20, Stefan Beller wrote:

Yes, sorry. I had this tested with 2.8.3 at first, then waited for my update to 2.9.0 to reproduce, and garbled the text while adjusting it.

That name makes a lot of sense to me.

Clone but detach, to be consistent. Yes.

All of this has to pass through transport and get-pack, right?

Sounds like a very reasonable roadmap to me.

Do you think there will be someone volunteering to tackle this?

Greetings,
  Martin

On Tue, Jun 28, 2016 at 12:08 PM, Martin von Gagern <Martin.vGagern@gmx.net> wrote:

Oh, right.

Yeah we have to go through the transport layer, i.e. from builtin/clone.c we manipulate the transport object as defined in transport.h and code in transport.c What do you mean by get-pack, though?

I have it on my (low priority) TODO list, so if you want it sooner than later, go for it yourself. Otherwise just wait. Maybe Duy has some interest as well.

Thanks, Stefan