How to check new commit availability without full fetch?

Hi List, I am trying to find a way to check availability of new commits *before* doing fetch or pull. Unfortunately, neither fetch nor pull take "--dry-run" option (unlike push) I am sure I am not the only one with such an itch.

Any help and advice are greatly appreciated.

--Leo--

On Sun, 10 Jan 2010, Leo Razoumov wrote:

But... _Why_ do you want/need to do that?

You could use ls-remote to see what the remote branch is pointing to, e.g.:

	git ls-remote origin master

and compare with the local view of that remote branch:

	git show-ref origin/master

And if both SHA1 strings match then there is nothing new to fetch.

Maybe you are. There is very little point knowing that the remote repo has new commits if you're not going to fetch them, so I don't understand why you need this.

Nicolas

Nicolas Pitre <nico@fluxnic.net> writes:

A feel good factor is in play? IOW, "I am short of time, so I won't be able to really afford to 'git pull' and test the result of re-integrating my changes to what happened on the other end. If I can learn that there is nothing happening over there, then I won't have to do anything and know that I am up to date."

On Sun, 10 Jan 2010, Junio C Hamano wrote:

Just do a fetch then. If the fetch progress display looks like if it is going to take a while then just interrupt it and go home. If the fetch looks trivial then just merge it. In any case, the "feel good" factor can't be that great by only knowing if the remote has changed or not.

Well maybe if it hasn't changed then you know right away how to feel about it (equally with a fetch in that case), and if the remote is indeed different then you can't tell whether the changes are trivial or not without actually fetching them.

Nicolas

On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:

Forced interruption is not such a good idea. I would favor a non-destructive way to monitor availability of remote commits.

BTW, pull and push are in a way symmetric operations. Is there any deep reason why push supports --dry-run but pull/fetch does not??

--Leo--

Hi,

On Mon, Jan 11, 2010 at 9:36 AM, Leo Razoumov <slonik.az@gmail.com> wrote:

By default, when you add a remote (with git remote add), git sets up the fetch refspec in your config that looks like

  [remote "foo"]
    url = git://foo.com/git/foo.git
    fetch = refs/heads/*:refs/remotes/foo/*

That is to say, branches on the remote repo will be fetched into a "safe" area, refs/remotes/foo/, away from the branches that you normally work with in refs/heads/.

However, if you have a different config and you're fetching directly into refs/heads/, then I can see why you would want to "peek" first with --dry-run before fetching. Are you doing this?

It's more accurate to say that push and fetch are symmetric, because pull is fetch with merge or rebase tacked on.

Even then, push and fetch are not _that_ symmetric...

Tay Ray Chuan <rctay89@gmail.com> writes:

I don't. Until all the objects are safely transferred, none of the refs are updated, whether they are directly slurped into local branch namespace or remote tracking branch namespace. So no matter what the configuration is, interrupted transfer, forced or otherwise, is safe.

On Sun, Jan 10, 2010 at 8:08 PM, Junio C Hamano <gitster@pobox.com> wrote:

I suggest adding this kind of information to the fetch/pull (and I suppose push) documentation.

Michael Witten <mfwitten@gmail.com> writes:

Sounds good.  Pleaes make it so.

On Sun, 10 Jan 2010, Leo Razoumov wrote:

You still don't answer my question though. Again, _why_ do you need to know about remote commit availability without fetching them?

Pushing involves resource usage on your own machine while pulling/fetching involves the remote machine. Your choice to "waste" CPU cycles on your own machine is not the same as having anybody do the same on a central server.

Nicolas

On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:

I use git to track almost all my data (code and otherwise) and spread it between several computers. I end up with several local repos having the same local branches. It happens once in a while that I fetch into a given remote/foo from several local foo branches from different machines and the operation fails. It happens because the commits have not been yet consistently distributed among the repos. To do the forensics and figure out who should update whom first I need a quick and non-destructive way to fetch dry-run.

--Leo--

On Mon, 11 Jan 2010, Leo Razoumov wrote:

There is probably something awkward about your setup then.

Normally you should have a remote description for any of the remote repositories you fetch from. So if you have, say, remote machine_a with repo foo, machine_b with repo bar, and machine_c with repo baz, then fetching any of those will _only_ mirror locally the state of those remote repositories. There is no ordering required as there can't be any conflicts in the mere fact of mirroring what the other guys have. That's what remote tracking branches are for: they follow the state of a remote repository and are never altered by local changes. And you can have as many of those as you wish and they will never conflict with each other as each remote description is independent. And this is true whether or not the remote repository lives on the same machine (that would be a remote directory in that case).

And even if most if not all those remotes are actually copies of each others, then the first fetch to occur will transfer the new objects while fetching the other ones will simply notice that the required objects are already available locally and only the ref will be updated.

The ordering comes into play when it is time to _merge_ those remote branches into, say, the local master branch. That's why it is probably a good thing to use fetch+merge instead of pull in this case. But this is then a local matter and nothing that depends on the fetch ordering.

Nicolas

On 2010-01-11, Nicolas Pitre <nico@fluxnic.net> wrote:

Setup might be, indeed, awkward but it handles very diverse tasks. As I said in my earlier emails different repos fetch into the *same* remote/foo. So there could be conflicts and using fetch -f could cause loss of data.

Before switching to git I used mercurial for the same purpose and it has command that are equivalent to fetch --dry-run.

--Leo--

On Sun, Jan 10, 2010 at 08:36:44PM -0500, Leo Razoumov wrote:

Not really... 'pull' = 'fetch' + 'merge', while 'push' only propagates changes without any merging. You can say 'fetch' and 'push' are in a way symmetric operations, but this symmetry is limited due to difference in usage between local and remote branches.

I guess it is because no one needs it. 'push' has --dry-run, because it updates local references in a remote repository. So, you may want to be sure that you are pushing the right thing. On the other hand, I see no reason to have --dry-run for 'fetch', because it updates only remote references, making them to point to the current state of the corresponding branches. 'fetch' does not change any local branch, so I see no reason for --dry-run.

What use case do you have in mind that needs --dry-run for 'fetch'?

Dmitry

söndagen den 10 januari 2010 12.12.09 skrev  Leo Razoumov:

Fetch has --dry-run. It's a fairly new option. The drawback is that it still does the fetch, but it does not update the refs. If you re.run it again it'll be quicker.

A faster option is to use ls-remote, but you'll have to parse the data yourself and compare with your remote refs to see what refs has changed, and that will not tell you /what/ the changes are.

-- robin

Robin Rosenberg <robin.rosenberg@dewire.com> writes:

Doesn't that worry us if it really is quicker?

If --dry-run doesn't update the refs, why do the objects that were transferred by them not get asked the next time? There must be a bug somewhere, but it is getting late already, so I'll leave it to experts in the transfer area to figure it out...

On Mon, 11 Jan 2010, Junio C Hamano wrote:

What about builtin-fetch.c:quickfetch() ?

Nicolas

Nicolas Pitre <nico@fluxnic.net> writes:

Ahh, you are right. It walks from objects the remote side told us are at the tip, and stops at what we know are complete (i.e. reachable from our tip of objects); immediately after --dry-run slurped objects, the next fetch will prove everything is locally available and complete before going over the network.

But either I am very confused or the use of fields from "struct ref" is unintuitive in this codepath.

Why does it feed ref->old_sha1?  We are feeding _their_ tip commits to:

    rev-list --objects --stdin --not --all

and expecting it to report failure when some of their tip commits lead to what we don't have yet. The reason why we have old_sha1[] vs new_sha1[] is because we want to report what changed from what, and also to protect us from simultaneous updates by doing compare-and-swap using the value we read from our refs when we started in old_sha1[], so I would have expected that ref_map elements would have _their_ commits on the new_sha1[] side, but apparently that is not what is happening, and it has been this way for a long time. The use of old_sha1[] came from 4191c35 (git-fetch: avoid local fetching from alternate (again), 2007-11-11), so it is a lot more likely that I am confused than the code is wrong and nobody noticed so far.

What am I missing?

On Mon, 11 Jan 2010, Junio C Hamano wrote:

Very confusing indeed. I first discovered about quickfetch() myself when I fixed shallow clone leading to commit 86386829.

If old_sha1[] was our refs then quickfetch() would always succeed and we'd never fetch anything.

Digging a bit, it looks like get_remote_heads() is storing the remote's heads into old_sha1. And so is performed in get_refs_from_bundle(), and in insert_packed_refs() from get_refs_via_rsync(), etc.

Looking at the struct ref definition, we can see:

struct ref {
        struct ref *next;
        unsigned char old_sha1[20];
        unsigned char new_sha1[20];
        [...]
        struct ref *peer_ref; /* when renaming */
        [...]
};

And apparently store_updated_refs() ends up using that peer_ref like this:

        for (rm = ref_map; rm; rm = rm->next) {
                struct ref *ref = NULL;

                if (rm->peer_ref) {
                        ref = xcalloc(1, sizeof(*ref) + strlen(rm->peer_ref->name) + 1);
                        strcpy(ref->name, rm->peer_ref->name);
                        hashcpy(ref->old_sha1, rm->peer_ref->old_sha1);
                        hashcpy(ref->new_sha1, rm->old_sha1);
                        ref->force = rm->peer_ref->force;
                }

So.... Doesn't this all look like a total mess of needless (and even leaked in this case) allocations and duplications, besides being completely unintuitive? Both hashcpy() above certainly throw my sense of logic aside...

Nicolas

Leo Razoumov <slonik.az@gmail.com> writes:

Try git remote show origin.

Andreas.