reducing object store size with remote alternates or shallow clone?

I'm trying to package a linux kernel source tree and would like to just tar up a tree with .git/.  However this is a bit problematic as the size of tgz is about 500M which exceeds the limit of the image I'm trying to produce.

I was wondering if anyone had a means to reduce the size drastically.  I'm ok if a post processing step is required to get full git support.  One idea I had was doing a shallow clone and if there was some means to "reconnect" it to a full work state after the fact.

thanks

- k

Kumar Gala <galak@kernel.crashing.org> writes:

Although your message body does not state your design constraints clearly, your subject line hints that you are fine with a solution that involves use of remote alternates by the recipient of your tarball.

I am _guessing_ that you have a fork from some well known tree and want to sneakernet it to your recipients (iow, they do not "git pull" from your repository).

How about doing

    $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    $ git fetch $LINUS
    $ git bundle create myfork.bundle HEAD..master

and sending the thing over?  The recipient would then do something like:

    $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    $ git clone $LINUS linux
    $ cd linux
    $ git pull /tmp/myfork.bundle master

On 08/24/2010 11:45 AM, Junio C Hamano wrote:

I think you mean

      $ git fetch $LINUS master
      $ git bundle create myfork.bundle FETCH_HEAD..master

-Brandon

Brandon Casey <brandon.casey.ctr@nrlssc.navy.mil> writes:

Thanks, of course you are right.

Strictly speaking, as I know there is only one branch in the repository of Linus, there is no need to say "master" when fetching, but it would be a good discipline to explicitly specify what you mean on the command line.

On 08/24/2010 01:59 PM, Junio C Hamano wrote:

Hmm. It appears that if the current checked-out branch has a configured merge ref, then a fetch that supplies a repository url (not a remote name) and no fetch refspec, will not fall back to fetch HEAD from the remote repository.

i.e. the following fetch does not retrieve any objects nor update FETCH_HEAD

   $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux
   $ cd linux
   $ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

but, if you create a new branch, that has no merge ref configuration, then git behaves as expected:

   $ git branch foo
   $ git checkout foo
   $ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

Namely we retrieve new objects and update FETCH_HEAD:

   From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
   * branch            HEAD       -> FETCH_HEAD

I think the problem is in builtin/fetch.c: get_ref_map().

When fetch is called as above, with a repository url but no refspec, we get this call sequence:

   cmd_fetch -> fetch_one
     fetch_one -> do_fetch(argc = 0)
       do_fetch -> get_ref_map(ref_count = 0)
         line 148: has_merge is assigned 1 since the current checked out
                   branch has a merge ref configured
         The 'if' branch is entered, the 'for' loop is not entered,
         ref_map retains its NULL initialization value and
         get_ref_map() returns NULL
       do_fetch -> fetch_refs(ref_map = NULL)
         and the transports do nothing since no refs have been requested

Perhaps the fix should look something like this (warning copy/paste):

From: Brandon Casey <drafnel@gmail.com>

When 'git fetch' is supplied just a repository URL (not a remote name), and without a fetch refspec, it should fetch from the remote HEAD branch and update FETCH_HEAD with the fetched ref. Currently, when 'git fetch' is called like this, it fails to retrieve anything, and does not update FETCH_HEAD, if the current checked-out branch has a configured merge ref.

i.e. this fetch fails to retrieve anything nor update FETCH_HEAD:

   git checkout master
   git config branch.master.merge refs/heads/master
   git fetch git://git.kernel.org/pub/scm/git/git.git

but this one does:

   git config --unset branch.master.merge
   git fetch git://git.kernel.org/pub/scm/git/git.git

Add a test to demonstrate this flaw.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 t/t5510-fetch.sh |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

Brandon Casey <casey@nrlssc.navy.mil> writes:

Hmph, we can call it a regression, since we certainly won't see this failure with versions of git that is unaware of branch.*.merge.

But what should we be expecting?

Just as a datapoint, an ancient git (e.g. v1.4.0), the above command line would have fetched the HEAD from the remote side, no matter what that symref is pointing at. Your [2/2] patch replicates this behaviour, which is fine by me [*1*].

Your test only checks if we leave _anything_ in FETCH_HEAD, and does not check if we only fetch one, if we fetch all the refs, or if we fetch what the configuration branch.*.merge asks for (but without the corresponding branch.*.remote configuration, doing so is pointless).

I think it would be better to have two tests. One arranges the current branch to track the same branch the HEAD at the remote points at, and the other arranges the current branch to track a branch different from the HEAD at the remote points at. In both cases, as "fetch" should ignore the configuration, we should get the object pointed by the HEAD on the remote side.

Thanks.

[Footnote]

*1* A plausible alternative is to match the given URL against list of existing remote.<name>.url (make sure there is only one), and behave as if that the remote name is given. I can be persuaded either way, but not looking at the configuration feels a lot simpler to explain and understand (i.e. "with name, we use the set of configuration variable given to that name; without name, there is no configuration for us to look up").

From: Brandon Casey <drafnel@gmail.com>

When 'git fetch' is supplied a single argument, it tries to match it against a configured remote and then fetch the refs specified by the named remote's fetchspec. Additionally, or alternatively, if the current branch has a merge ref configured, and if the name of the remote supplied to fetch matches the one in the branch's configuration, then git also adds the merge ref to the list of refs to update.

If the argument to fetch does not specify a named remote, or if the name supplied does not match the remote configured for the current branch, then the current branch's merge configuration should not be considered.

git currently mishandles the case when the argument to fetch specifies a GIT URL(i.e. not a named remote) and the current branch has a configured merge ref. In this case, fetch should ignore the branch's merge ref and attempt to fetch from the remote repository's HEAD branch. But, since fetch only checks _whether_ the current branch has a merge ref configured, and does _not_ check whether the branch's configured remote matches the command line argument (until later), it will mistakenly enter the wrong branch of an 'if' statement and will not fall back to fetch the HEAD branch. The fetch ends up doing nothing and returns with a successful zero status.

Fix this by comparing the remote repository's name to the branch's remote name, in addition to whether it has a configured merge ref, sooner, so that fetch can correctly decide whether the branch's configuration is interesting or not, and fall back to fetching from the remote's HEAD branch when appropriate.

This fixes the test in t5510.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 builtin/fetch.c  |    3 ++-
 t/t5510-fetch.sh |    2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

Brandon Casey wrote:

Thanks for a fix.

What will happen with this (invalid) branch?

	[branch "tmp"]
		merge = refs/heads/tmp

On 08/25/2010 04:16 PM, Jonathan Nieder wrote:

The same thing that would have happened before, since a few lines further down there is this:

   if (has_merge &&
       !strcmp(branch->remote_name, remote->name))
           add_merge_config(&ref_map, remote_refs, branch, &tail);

I didn't trace branch_get() to check whether it returns an object with remote_name initialized in all cases. I relied on the form of the existing code. Perhaps it's worth investigating. If something needs to be fixed, then it was already broken and deserves a separate patch anyway.

-Brandon

Brandon Casey <casey@nrlssc.navy.mil> writes:

Couldn't branch->remote_name or remote->name be NULL here?

I think remote->name would be the same as the URL given on the command line (i.e. no risk of being NULL), and has_merge would be false even when branch.<name>.merge is specified if branch.<name>.remote is missing (i.e. no risk of running this strcmp() to begin with), but the latter safety guarantee is a bit too subtle for my taste to go without a in-code comment.

Thanks.

From: Brandon Casey <drafnel@gmail.com>

Save future readers the trouble of tracing code to determine that the two uses of branch->remote_name are safe when has_merge is set, by adding a comment explaining that it is so.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 builtin/fetch.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

From: Brandon Casey <drafnel@gmail.com>

Commit 6106ce46 introduced a test to demonstrate fetch's failure to retrieve any objects or update FETCH_HEAD when it was supplied a repository URL and the current branch had a configured merge spec. This commit expands the original test based on comments from Junio Hamano. In addition to actually verifying that the fetch updates FETCH_HEAD correctly, and does not update the current branch, two more tests are added to ensure that the merge configuration is ignored even when the supplied URL matches the URL of the remote configured for the branch.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 t/t5510-fetch.sh |   30 ++++++++++++++++++++++++++++--
 1 files changed, 28 insertions(+), 2 deletions(-)