Fetching commit instead of ref

Hi

I am involved in the git-subrepo project (https://github.com/ingydotnet/git-subrepo/). It's an attempt to simplify the inclusion of repos into other repos.

In a certain situation I would really need to fetch all commits related to a specific commit (SHA). I have read the git fetch documentation and found nothing regarding this. It only seems to support fetching references.

I found some traces on stack overflow: https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository

Following that recommendation it feels like it almost works: $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit remote: Counting objects: 2311, done. remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311 Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done. Resolving deltas: 100% (1174/1174), done. ----> So far so good, but then an error message appear: error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec ----> And nothing seems to be fetched.

Is there a way to fetch a commit and any ancestors to that commit based on a SHA?

Why do I need this?
In git-subrepo we try to recreate another repo within our main repo. Creating the necessary parent references when they appear. In some cases we need to make sure that we have access to the correct commits from the subrepo, but we don't have any references except a SHA.
 
-- Magnus
 
 
MAGNUS CARLSSON
 Staff Software Engineer
 ARRIS
 
 o: +46 13 36 75 92
 e: magnus.carlsson@arris.com
 w: www.arris.com
 
 ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583
 
 This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any "legal  advice privilege".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).
      

On Mon, 18 Dec 2017 12:30:23 +0000 "Carlsson, Magnus" <Magnus.Carlsson@arris.com> wrote:

The documentation has been updated in version 2.15.0 to describe this. But as the commit message of commit 83558a412afa ("fetch doc: src side of refspec could be full SHA-1", 2017-10-18) says, this functionality was available earlier.

You'll need to set uploadpack.allowTipSHA1InWant, uploadpack.allowReachableSHA1InWant, or uploadpack.allowAnySHA1InWant on the server. (This might need to be better documented - I see this documented for fetch-pack, but not fetch.)

On Mon, Dec 18, 2017 at 4:30 AM, Carlsson, Magnus <Magnus.Carlsson@arris.com> wrote:

Ask the server operator to configure the server to allow fetching commits, specifically "git config uploadpack.allowReachableSHA1InWant 1"

A very similar issue happens with submodules, which tries to fetch the branch(es) first and then by SHA1 as a fallback, but this fallback may fail as well due to the miss-configured server, at that point submodules just error out.

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

Yes, that is what the error message is telling you.  

You'd need to coordinate with the server operator so that the server allows such an request; uploadpack.allowAnySHA1InWant may help.

First: Thanks everyone for your answers.

I understand that there is a fetch pack, problem is that I can't force every git server provider to turn it on... Tested with github and they don't seem to have it on by default.

I understand that you don't want to allow people fetching single commits all the time, but is there a reason that you don't allow SHA instead of references when you fetch an entire tree?

Is there a workaround? Can I somehow ask a remote on a valid reference that includes a SHA? So I can later fetch that reference. In my case I would like to avoid to fetch more then necessary as it pollutes the main repository.

-- Magnus

MAGNUS CARLSSON Staff Software Engineer ARRIS

o: +46 13 36 75 92 e: magnus.carlsson@arris.com w: www.arris.com

ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583

This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any "legal advice privilege".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).

________________________________________
From: Junio C Hamano <gitster@pobox.com>
Sent: Monday, December 18, 2017 19:44
To: Carlsson, Magnus
Cc: git@vger.kernel.org
Subject: Re: Fetching commit instead of ref

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

Yes, that is what the error message is telling you.

You'd need to coordinate with the server operator so that the server allows such an request; uploadpack.allowAnySHA1InWant may help.

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

I do not think we don't want to allow "single commit" fetch. We do not allow fetching histories from an arbitrary point in the graph, unless we can prove that what gets fetched is what the serving end intended to expose to the fetcher---you should view it as a security issue.

The default way to determine that a fetch request is getting only the things the serving end wanted to publish is to see the requested tips of the histories to be fetched are all pointed by refs. Which means that a client of a hosting site can rewind and repoint a ref after pushing a wrong branch that contained undesirable matter by accident. Moving the ref to make the undesirable object unreachable is all that is needed to "hide" it from the public view, even when the client does not have a way to trigger "gc" immediately on the hosting site.

There are variants of this security measure. If the hosting site is willing to spend extra cycles, we could loosen the "is the request fetching only what is published?" check to see if the requested tips are reachable from the tips of refs, instead of exactly at refs. It preserves "a mistake can be hidden with a forced push" property the same way as above, but it is costly and is prone to abuse.

If you are confident about your pushes all the time, you can take a stance "anything in the repository is meant to be read". That is what the "allowAnySHA1InWant" does. Not everybody however wants to do this for obvious reasons, so it is not a default.

Thanks Junio for your answer!

From my simple tests it seems that github doesn't have this on by default so that seems a little dull.

Do you know if there is a way to actually find a ref that contains the SHA from a remote?

Finally, you say that its a security feature, but from the log it feels like you are actually downloading things first and then you determine if you are allowed to fetch it or?

$ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit remote: Counting objects: 2311, done. remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311 -- >>> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done. Resolving deltas: 100% (1174/1174), done. error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec

-- Magnus

MAGNUS CARLSSON Staff Software Engineer ARRIS

o: +46 13 36 75 92 e: magnus.carlsson@arris.com w: www.arris.com

ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583

This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any "legal advice privilege".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).

________________________________________
From: Junio C Hamano <gitster@pobox.com>
Sent: Tuesday, December 19, 2017 17:07
To: Carlsson, Magnus
Cc: git@vger.kernel.org
Subject: Re: Fetching commit instead of ref

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

I do not think we don't want to allow "single commit" fetch. We do not allow fetching histories from an arbitrary point in the graph, unless we can prove that what gets fetched is what the serving end intended to expose to the fetcher---you should view it as a security issue.

The default way to determine that a fetch request is getting only the things the serving end wanted to publish is to see the requested tips of the histories to be fetched are all pointed by refs. Which means that a client of a hosting site can rewind and repoint a ref after pushing a wrong branch that contained undesirable matter by accident. Moving the ref to make the undesirable object unreachable is all that is needed to "hide" it from the public view, even when the client does not have a way to trigger "gc" immediately on the hosting site.

There are variants of this security measure. If the hosting site is willing to spend extra cycles, we could loosen the "is the request fetching only what is published?" check to see if the requested tips are reachable from the tips of refs, instead of exactly at refs. It preserves "a mistake can be hidden with a forced push" property the same way as above, but it is costly and is prone to abuse.

If you are confident about your pushes all the time, you can take a stance "anything in the repository is meant to be read". That is what the "allowAnySHA1InWant" does. Not everybody however wants to do this for obvious reasons, so it is not a default.

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

Unless that "fetch" is doing a recursive fetch from another repository (which causes the counting to be shown) or something silly, the above makes it look like that the server is broken.

A quick test locally does not seem to show such a behaviour, but I do not enable pack bitmaps and I know github does at least for some repositories, so it is possible there is a problem somewhere in that codepath.