threads / discuss / 41145

how to fetch entire heirarchy of refs from a remote?

Subject: how to fetch entire heirarchy of refs from a remote?

## tl;dr

8 messages between Jan 7, 2016 and Jan 8, 2016.

replies: 7people: 2as markdown or json

Jacob Keller· Jan 7, 2016, 23:06 UTC · lore

Hi,

Say I have a repository which has a series of refs in a heirarchy such as a gerrit repository with changes specified something like:

refs/changes/96/55596/3 refs/changes/96/55596/4 refs/changes/96/57496/1 refs/changes/96/57496/2 refs/changes/96/69796/1 refs/changes/96/69796/2 refs/changes/96/71696/1 refs/changes/96/71696/2 refs/changes/97/40197/1 refs/changes/97/40197/2 refs/changes/97/40197/3 refs/changes/97/40197/4 ... refs/changes/97/71697/2 refs/changes/97/71697/3 refs/changes/98/47298/1 refs/changes/98/47298/2 refs/changes/98/47298/3 refs/changes/98/57298/1 refs/changes/98/57298/2 refs/changes/98/73598/1 refs/changes/99/44099/1 refs/changes/99/69299/1

Is it possible to specify a refspec such that all of these will be fetched? I tried doing

+refs/changes/*:refs/changes/*

but this doesn't work since the * can only be one portion of a refspec

The problem is that I would like to be able to pull all of these changes in a git mirror so that I can reduce network traffic I use to the gerrit server, by updating my local copy once and using the local copy on the other tasks I need. This is necessary since the gerrit server is (a) far away and (b) closes my connection when I have too many requests.

But, git clone --mirror and git fetch have failed to pull every ref, and only end up with heads and tags.

Regards, Jake

Junio C Hamano· Jan 7, 2016, 23:15 UTC · re: Jacob Keller · lore

Re: how to fetch entire heirarchy of refs from a remote?

Jacob Keller <jacob.keller@gmail.com> writes:

Show 14 quoted lines

> refs/changes/98/47298/3
> refs/changes/98/57298/1
> refs/changes/98/57298/2
> refs/changes/98/73598/1
> refs/changes/99/44099/1
> refs/changes/99/69299/1
>
>
> Is it possible to specify a refspec such that all of these will be
> fetched? I tried doing
>
> +refs/changes/*:refs/changes/*
>
> but this doesn't work since the * can only be one portion of a refspec

I am not sure if I follow. With refs/heads/*:refs/remotes/origin/* (e.g. following git://github.com/gitster/git for example), I am reasonably sure you would get my refs/heads/jk/notes-dwim-doc and other topics you authored without the slash between jk and notes-dwim-doc getting in the way.

Jacob Keller· Jan 7, 2016, 23:23 UTC · re: Junio C Hamano · lore

Re: how to fetch entire heirarchy of refs from a remote?

On Thu, Jan 7, 2016 at 3:15 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 24 quoted lines

> Jacob Keller <jacob.keller@gmail.com> writes:
>
>> refs/changes/98/47298/3
>> refs/changes/98/57298/1
>> refs/changes/98/57298/2
>> refs/changes/98/73598/1
>> refs/changes/99/44099/1
>> refs/changes/99/69299/1
>>
>>
>> Is it possible to specify a refspec such that all of these will be
>> fetched? I tried doing
>>
>> +refs/changes/*:refs/changes/*
>>
>> but this doesn't work since the * can only be one portion of a refspec
>
> I am not sure if I follow.  With refs/heads/*:refs/remotes/origin/*
> (e.g. following git://github.com/gitster/git for example), I am
> reasonably sure you would get my refs/heads/jk/notes-dwim-doc and
> other topics you authored without the slash between jk and
> notes-dwim-doc getting in the way.
>
>

That appears to be correct, but.. why doesn't:

git fetch origin +refs/changes/*:refs/changes/*

seem to anything at all?

Ideally this owuld fetch all the remote refs in refs/changes and stick them into my refs, but it appears to do nothing at all.

Regards, Jake

Jacob Keller· Jan 7, 2016, 23:25 UTC · re: Junio C Hamano · lore

Re: how to fetch entire heirarchy of refs from a remote?

On Thu, Jan 7, 2016 at 3:15 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 24 quoted lines

> Jacob Keller <jacob.keller@gmail.com> writes:
>
>> refs/changes/98/47298/3
>> refs/changes/98/57298/1
>> refs/changes/98/57298/2
>> refs/changes/98/73598/1
>> refs/changes/99/44099/1
>> refs/changes/99/69299/1
>>
>>
>> Is it possible to specify a refspec such that all of these will be
>> fetched? I tried doing
>>
>> +refs/changes/*:refs/changes/*
>>
>> but this doesn't work since the * can only be one portion of a refspec
>
> I am not sure if I follow.  With refs/heads/*:refs/remotes/origin/*
> (e.g. following git://github.com/gitster/git for example), I am
> reasonably sure you would get my refs/heads/jk/notes-dwim-doc and
> other topics you authored without the slash between jk and
> notes-dwim-doc getting in the way.
>
>

Infact, even running

git fetch origin +refs/changes/98/73598/1:refs/changes/98/73598/1

does not end up placing anything into refs/changes?

Is it because it happens to not be known by git? That seems rather short sighted...

Regards, Jake

Jacob Keller· Jan 7, 2016, 23:36 UTC · re: Jacob Keller · lore

Re: how to fetch entire heirarchy of refs from a remote?

On Thu, Jan 7, 2016 at 3:06 PM, Jacob Keller <jacob.keller@gmail.com> wrote:

Show 6 quoted lines

>
> But, git clone --mirror and git fetch have failed to pull every ref,
> and only end up with heads and tags.
>
> Regards,
> Jake

It turns out that it *does* pull the refs, but they don't appear to show up inside the "refs" folder.. is there a reason for this?

ie:

ls .git/refs/changes

shows nothing,but

git ls-remote file:///path/to/repo

does show them? Any particular reason for why this is? I am confused why refs aren't showing up inside the refs folder...

Regards, Jake

Junio C Hamano· Jan 7, 2016, 23:45 UTC · re: Jacob Keller · lore

Re: how to fetch entire heirarchy of refs from a remote?

Jacob Keller <jacob.keller@gmail.com> writes:

Show 21 quoted lines

> On Thu, Jan 7, 2016 at 3:06 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
>>
>> But, git clone --mirror and git fetch have failed to pull every ref,
>> and only end up with heads and tags.
>>
>> Regards,
>> Jake
>
> It turns out that it *does* pull the refs, but they don't appear to
> show up inside the "refs" folder.. is there a reason for this?
>
> ie:
>
> ls .git/refs/changes
>
> shows nothing,but
>
> git ls-remote file:///path/to/repo
>
> does show them? Any particular reason for why this is? I am confused
> why refs aren't showing up inside the refs folder...

Does

	git for-each-ref refs/changes/

in your local resulting repository show them?

There is nothing that says "ls -R .git/refs/" is the way to enumerate available refs (in fact it is a _wrong_ way that is guaranteed to give you wrong results).

Hint: .git/packed-refs

Jacob Keller· Jan 8, 2016, 16:30 UTC · re: Junio C Hamano · lore

Re: how to fetch entire heirarchy of refs from a remote?

On Thu, Jan 7, 2016 at 3:45 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 15 quoted lines

>> does show them? Any particular reason for why this is? I am confused
>> why refs aren't showing up inside the refs folder...
>
> Does
>
>         git for-each-ref refs/changes/
>
> in your local resulting repository show them?
>
> There is nothing that says "ls -R .git/refs/" is the way to
> enumerate available refs (in fact it is a _wrong_ way that is
> guaranteed to give you wrong results).
>
> Hint: .git/packed-refs
>

Yep I figured that out :) Now I know better!

Regards, Jake

Jacob Keller· Jan 7, 2016, 23:39 UTC · re: Jacob Keller · lore

Re: how to fetch entire heirarchy of refs from a remote?

On Thu, Jan 7, 2016 at 3:06 PM, Jacob Keller <jacob.keller@gmail.com> wrote:

Show 49 quoted lines

> Hi,
>
> Say I have a repository which has a series of refs in a heirarchy such
> as a gerrit repository with changes specified something like:
>
> refs/changes/96/55596/3
> refs/changes/96/55596/4
> refs/changes/96/57496/1
> refs/changes/96/57496/2
> refs/changes/96/69796/1
> refs/changes/96/69796/2
> refs/changes/96/71696/1
> refs/changes/96/71696/2
> refs/changes/97/40197/1
> refs/changes/97/40197/2
> refs/changes/97/40197/3
> refs/changes/97/40197/4
> ...
> refs/changes/97/71697/2
> refs/changes/97/71697/3
> refs/changes/98/47298/1
> refs/changes/98/47298/2
> refs/changes/98/47298/3
> refs/changes/98/57298/1
> refs/changes/98/57298/2
> refs/changes/98/73598/1
> refs/changes/99/44099/1
> refs/changes/99/69299/1
>
>
> Is it possible to specify a refspec such that all of these will be
> fetched? I tried doing
>
> +refs/changes/*:refs/changes/*
>
> but this doesn't work since the * can only be one portion of a refspec
>
> The problem is that I would like to be able to pull all of these
> changes in a git mirror so that I can reduce network traffic I use to
> the gerrit server, by updating my local copy once and using the local
> copy on the other tasks I need. This is necessary since the gerrit
> server is (a) far away and (b) closes my connection when I have too
> many requests.
>
> But, git clone --mirror and git fetch have failed to pull every ref,
> and only end up with heads and tags.
>
> Regards,
> Jake

So, sorry for the list spam. Things work correctly, it just is using packed-refs, which is a feature I wasn't aware of :) Git already did what I expected but in a way I did not, so I was very confused.

Regards, Jake