{"thread":{"id":"47457","subject":"Fetching commit instead of ref","startedAt":"2017-12-18T12:37:37Z","lastAt":"2017-12-22T18:47:20Z","messageCount":8,"participants":["Carlsson, Magnus","Jonathan Tan","Stefan Beller","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"334963","messageId":"1513600223353.28799@arris.com","threadId":"47457","inReplyTo":null,"subject":"Fetching commit instead of ref","fromName":"Carlsson, Magnus","fromEmail":"magnus.carlsson@arris.com","sentAt":"2017-12-18T12:30:23Z","receivedAt":"2017-12-18T12:37:37Z","isPatch":false,"sender":{"key":"magnus.carlsson@arris.com","avatar":null},"body":"Hi\n\nI am involved in the git-subrepo project (https://github.com/ingydotnet/git-subrepo/). It's an attempt to simplify the inclusion of repos into other repos.\n\nIn a certain situation I would really need to fetch all commits related to a specific commit (SHA). I have read the git fetch documentation and found nothing regarding this. It only seems to support fetching references.\n\nI found some traces on stack overflow:\nhttps://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository\n\nFollowing that recommendation it feels like it almost works:\n$ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit\nremote: Counting objects: 2311, done.\nremote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311\nReceiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.\nResolving deltas: 100% (1174/1174), done.\n----> So far so good, but then an error message appear:\nerror: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n----> And nothing seems to be fetched.\n\nIs there a way to fetch a commit and any ancestors to that commit based on a SHA?\n\nWhy do I need this?\nIn git-subrepo we try to recreate another repo within our main repo. Creating the necessary parent references when they appear. In some cases we need to make sure that we have access to the correct commits from the subrepo, but we don't have any references except a SHA.\n \n-- Magnus\n \n \nMAGNUS CARLSSON\n Staff Software Engineer\n ARRIS\n \n o: +46 13 36 75 92\n e: magnus.carlsson@arris.com\n w: www.arris.com\n \n ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583\n \n This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any \"legal advice privilege\".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).\n "},{"id":"334976","messageId":"20171218101655.16105728beb82d38732e48a0@google.com","threadId":"47457","inReplyTo":"1513600223353.28799@arris.com","subject":"Re: Fetching commit instead of ref","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-12-18T18:16:55Z","receivedAt":"2017-12-18T18:19:30Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 18 Dec 2017 12:30:23 +0000\n\"Carlsson, Magnus\" wrote:\n\n> In a certain situation I would really need to fetch all commits\n> related to a specific commit (SHA). I have read the git fetch\n> documentation and found nothing regarding this. It only seems to\n> support fetching references.\n\nThe documentation has been updated in version 2.15.0 to describe this.\nBut as the commit message of commit 83558a412afa (\"fetch doc: src side\nof refspec could be full SHA-1\", 2017-10-18) says, this functionality\nwas available earlier.\n\n> I found some traces on stack overflow:\n> https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository\n> \n> Following that recommendation it feels like it almost works:\n> $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit\n> remote: Counting objects: 2311, done.\n> remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311\n> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.\n> Resolving deltas: 100% (1174/1174), done.\n> ----> So far so good, but then an error message appear:\n> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n> ----> And nothing seems to be fetched.\n> \n> Is there a way to fetch a commit and any ancestors to that commit based on a SHA?\n\nYou'll need to set uploadpack.allowTipSHA1InWant,\nuploadpack.allowReachableSHA1InWant, or uploadpack.allowAnySHA1InWant on\nthe server. (This might need to be better documented - I see this\ndocumented for fetch-pack, but not fetch.)\n"},{"id":"334977","messageId":"CAGZ79kaePjuN3fEOAj=0-qn=_3B_3AQXDMnNPPEbNLOVSWS0mQ@mail.gmail.com","threadId":"47457","inReplyTo":"1513600223353.28799@arris.com","subject":"Re: Fetching commit instead of ref","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-12-18T18:22:15Z","receivedAt":"2017-12-18T18:22:25Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Dec 18, 2017 at 4:30 AM, Carlsson, Magnus\n wrote:\n> Hi\n>\n> I am involved in the git-subrepo project (https://github.com/ingydotnet/git-subrepo/). It's an attempt to simplify the inclusion of repos into other repos.\n>\n> In a certain situation I would really need to fetch all commits related to a specific commit (SHA). I have read the git fetch documentation and found nothing regarding this. It only seems to support fetching references.\n>\n> I found some traces on stack overflow:\n> https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository\n>\n> Following that recommendation it feels like it almost works:\n> $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit\n> remote: Counting objects: 2311, done.\n> remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311\n> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.\n> Resolving deltas: 100% (1174/1174), done.\n> ----> So far so good, but then an error message appear:\n> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n> ----> And nothing seems to be fetched.\n>\n> Is there a way to fetch a commit and any ancestors to that commit based on a SHA?\n\nAsk the server operator to configure the server to allow fetching commits,\nspecifically \"git config uploadpack.allowReachableSHA1InWant 1\"\n\n\n>\n> Why do I need this?\n> In git-subrepo we try to recreate another repo within our main repo. Creating the necessary parent references when they appear. In some cases we need to make sure that we have access to the correct commits from the subrepo, but we don't have any references except a SHA.\n\nA very similar issue happens with submodules, which tries to fetch the\nbranch(es) first and then by SHA1 as a fallback, but this fallback may\nfail as well due to the miss-configured server, at that point\nsubmodules just error out.\n"},{"id":"334982","messageId":"xmqqh8snyiz0.fsf@gitster.mtv.corp.google.com","threadId":"47457","inReplyTo":"1513600223353.28799@arris.com","subject":"Re: Fetching commit instead of ref","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-18T18:44:03Z","receivedAt":"2017-12-18T18:44:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Carlsson, Magnus\" writes:\n\n> ----> So far so good, but then an error message appear:\n> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n> ----> And nothing seems to be fetched.\n\nYes, that is what the error message is telling you. \n\nYou'd need to coordinate with the server operator so that the server\nallows such an request; uploadpack.allowAnySHA1InWant may help.\n"},{"id":"335014","messageId":"1513672915856.50628@arris.com","threadId":"47457","inReplyTo":"xmqqh8snyiz0.fsf@gitster.mtv.corp.google.com","subject":"Re: Fetching commit instead of ref","fromName":"Carlsson, Magnus","fromEmail":"magnus.carlsson@arris.com","sentAt":"2017-12-19T08:41:56Z","receivedAt":"2017-12-19T08:43:15Z","isPatch":false,"sender":{"key":"magnus.carlsson@arris.com","avatar":null},"body":"First: Thanks everyone for your answers.\n\nI understand that there is a fetch pack, problem is that I can't force every git server provider to turn it on... Tested with github and they don't seem to have it on by default.\n\nI understand that you don't want to allow people fetching single commits all the time, but is there a reason that you don't allow SHA instead of references when you fetch an entire tree?\n\nIs there a workaround? Can I somehow ask a remote on a valid reference that includes a SHA? So I can later fetch that reference. In my case I would like to avoid to fetch more then necessary as it pollutes the main repository.\n\n-- Magnus\n\nMAGNUS CARLSSON\nStaff Software Engineer\nARRIS\n\no: +46 13 36 75 92\ne: magnus.carlsson@arris.com\nw: www.arris.com\n\nARRIS: Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583\n\nThis electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed. It is confidential and may be attorney-client privileged. In any event the Sender reserves, to the fullest extent, any \"legal advice privilege\". Any further distribution or copying of this message is strictly prohibited. If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).\n\n________________________________________\nFrom: Junio C Hamano \nSent: Monday, December 18, 2017 19:44\nTo: Carlsson, Magnus\nCc: git@vger.kernel.org\nSubject: Re: Fetching commit instead of ref\n\n\"Carlsson, Magnus\" writes:\n\n> ----> So far so good, but then an error message appear:\n> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n> ----> And nothing seems to be fetched.\n\nYes, that is what the error message is telling you.\n\nYou'd need to coordinate with the server operator so that the server\nallows such an request; uploadpack.allowAnySHA1InWant may help.\n"},{"id":"335022","messageId":"xmqqtvwmwvkp.fsf@gitster.mtv.corp.google.com","threadId":"47457","inReplyTo":"1513672915856.50628@arris.com","subject":"Re: Fetching commit instead of ref","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-19T16:07:02Z","receivedAt":"2017-12-19T16:07:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Carlsson, Magnus\" writes:\n\n> I understand that you don't want to allow people fetching single\n> commits all the time, but is there a reason that you don't allow\n> SHA instead of references when you fetch an entire tree?\n\nI do not think we don't want to allow \"single commit\" fetch. We do\nnot allow fetching histories from an arbitrary point in the graph,\nunless we can prove that what gets fetched is what the serving end\nintended to expose to the fetcher---you should view it as a security\nissue.\n\nThe default way to determine that a fetch request is getting only\nthe things the serving end wanted to publish is to see the requested\ntips of the histories to be fetched are all pointed by refs. Which\nmeans that a client of a hosting site can rewind and repoint a ref\nafter pushing a wrong branch that contained undesirable matter by\naccident. Moving the ref to make the undesirable object unreachable\nis all that is needed to \"hide\" it from the public view, even when\nthe client does not have a way to trigger \"gc\" immediately on the\nhosting site.\n\nThere are variants of this security measure. If the hosting site is\nwilling to spend extra cycles, we could loosen the \"is the request\nfetching only what is published?\" check to see if the requested tips\nare reachable from the tips of refs, instead of exactly at refs. It\npreserves \"a mistake can be hidden with a forced push\" property the\nsame way as above, but it is costly and is prone to abuse.\n\nIf you are confident about your pushes all the time, you can take\na stance \"anything in the repository is meant to be read\". That is\nwhat the \"allowAnySHA1InWant\" does. Not everybody however wants to\ndo this for obvious reasons, so it is not a default.\n\n\n"},{"id":"335065","messageId":"1513756114533.56362@arris.com","threadId":"47457","inReplyTo":"xmqqtvwmwvkp.fsf@gitster.mtv.corp.google.com","subject":"Re: Fetching commit instead of ref","fromName":"Carlsson, Magnus","fromEmail":"magnus.carlsson@arris.com","sentAt":"2017-12-20T07:48:34Z","receivedAt":"2017-12-20T07:48:54Z","isPatch":false,"sender":{"key":"magnus.carlsson@arris.com","avatar":null},"body":"Thanks Junio for your answer!\n\nFrom my simple tests it seems that github doesn't have this on by default so that seems a little dull.\n\nDo you know if there is a way to actually find a ref that contains the SHA from a remote?\n\nFinally, you say that its a security feature, but from the log it feels like you are actually downloading things first and then you determine if you are allowed to fetch it or?\n\n$ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit\nremote: Counting objects: 2311, done.\nremote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311\n-- >>> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.\nResolving deltas: 100% (1174/1174), done.\nerror: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n\n-- Magnus\n\n\nMAGNUS CARLSSON\nStaff Software Engineer\nARRIS\n\no: +46 13 36 75 92\ne: magnus.carlsson@arris.com\nw: www.arris.com\n\nARRIS: Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583\n\nThis electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed. It is confidential and may be attorney-client privileged. In any event the Sender reserves, to the fullest extent, any \"legal advice privilege\". Any further distribution or copying of this message is strictly prohibited. If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).\n\n________________________________________\nFrom: Junio C Hamano \nSent: Tuesday, December 19, 2017 17:07\nTo: Carlsson, Magnus\nCc: git@vger.kernel.org\nSubject: Re: Fetching commit instead of ref\n\n\"Carlsson, Magnus\" writes:\n\n> I understand that you don't want to allow people fetching single\n> commits all the time, but is there a reason that you don't allow\n> SHA instead of references when you fetch an entire tree?\n\nI do not think we don't want to allow \"single commit\" fetch. We do\nnot allow fetching histories from an arbitrary point in the graph,\nunless we can prove that what gets fetched is what the serving end\nintended to expose to the fetcher---you should view it as a security\nissue.\n\nThe default way to determine that a fetch request is getting only\nthe things the serving end wanted to publish is to see the requested\ntips of the histories to be fetched are all pointed by refs. Which\nmeans that a client of a hosting site can rewind and repoint a ref\nafter pushing a wrong branch that contained undesirable matter by\naccident. Moving the ref to make the undesirable object unreachable\nis all that is needed to \"hide\" it from the public view, even when\nthe client does not have a way to trigger \"gc\" immediately on the\nhosting site.\n\nThere are variants of this security measure. If the hosting site is\nwilling to spend extra cycles, we could loosen the \"is the request\nfetching only what is published?\" check to see if the requested tips\nare reachable from the tips of refs, instead of exactly at refs. It\npreserves \"a mistake can be hidden with a forced push\" property the\nsame way as above, but it is costly and is prone to abuse.\n\nIf you are confident about your pushes all the time, you can take\na stance \"anything in the repository is meant to be read\". That is\nwhat the \"allowAnySHA1InWant\" does. Not everybody however wants to\ndo this for obvious reasons, so it is not a default.\n\n\n"},{"id":"335181","messageId":"xmqqlghusiq7.fsf@gitster.mtv.corp.google.com","threadId":"47457","inReplyTo":"1513756114533.56362@arris.com","subject":"Re: Fetching commit instead of ref","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-22T18:47:12Z","receivedAt":"2017-12-22T18:47:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Carlsson, Magnus\" writes:\n\n> $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit\n> remote: Counting objects: 2311, done.\n> remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311\n> -- >>> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.\n> Resolving deltas: 100% (1174/1174), done.\n> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec\n\nUnless that \"fetch\" is doing a recursive fetch from another\nrepository (which causes the counting to be shown) or something\nsilly, the above makes it look like that the server is broken.\n\nA quick test locally does not seem to show such a behaviour, but I\ndo not enable pack bitmaps and I know github does at least for some\nrepositories, so it is possible there is a problem somewhere in that\ncodepath.\n"}]}