{"thread":{"id":"59884","subject":"Determining whether you have a commit locally, in a partial clone?","startedAt":"2023-06-20T11:26:46Z","lastAt":"2023-06-27T08:10:10Z","messageCount":9,"participants":["Tao Klerks","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"478539","messageId":"CAPMMpohiTzANyhzL-mS-gg2kzbOEOiDktNbdwEXBKy9uL0-JgA@mail.gmail.com","threadId":"59884","inReplyTo":null,"subject":"Determining whether you have a commit locally, in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2023-06-20T11:26:29Z","receivedAt":"2023-06-20T11:26:46Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nI \"discovered\" today that when you're in a partial clone, naive tests\nto check for whether you have a commit locally no longer work - they\nfetch the commit on-demand:\n\ngit cat-file -t SOME_HASH_NOT_IN_REFSPEC\ngit rev-list SOME_HASH_NOT_IN_REFSPEC\n\nI didn't realize this until today: even commits can be \"filtered out\"\nby partial clone, so any reference to a commit that is not found\nlocally must be resolved transparently via jit-fetch.\n\nI'm optimizing some stuff for users, so I need to know whether a given\ncommit exists locally or not... but I can't seem to figure out how!\n\nI tried using \"git rev-list\"'s \"--exclude-promisor-objects\" option,\nbut I guess I don't understand what that's supposed to do. In my case\nit just made a simple check like \"git rev-list\n--exclude-promisor-objects SOME_HASH_NOT_IN_REFSPEC\" take forever (10\nmins and counting).\n\nI confirmed that removing (commenting out) the\n\"remote.origin.promisor\" and \"remote.origin.partialclonefilter\" config\nkeys achieves my objective, but I can't figure out how to do it\nsafely; \"-c remote.origin.promisor=false -c\nremote.origin.partialclonefilter=\" does *not* seem to work. The\nexistence of a \"remote.origin.partialclonefilter\" value, even if it is\nempty, appears to override the \"remote.origin.promisor=false\" setting.\n\nAs far as I can tell, config values cannot be unset with \"-c\" - in\nfact I see that credential.helper was granted special support for\nempty string as a way of signalling \"no credential helper\" by Jeff\nKing in 2016.\n\nSo I guess I have two questions:\n* Is there any way to run a single git command in a \"don't use\npromisors\" context?\n* Is the fact that \"-c remote.origin.partialclonefilter=\" doesn't work\nfor temporarily unsetting the filter a bug/issue to be resolved?\n\nThanks,\nTao\n"},{"id":"478541","messageId":"CAPMMpog8Hv_KcjNxbh_wzjwrFYt7TuTvrVy1XEtJMm6RWSKzRg@mail.gmail.com","threadId":"59884","inReplyTo":"CAPMMpohiTzANyhzL-mS-gg2kzbOEOiDktNbdwEXBKy9uL0-JgA@mail.gmail.com","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2023-06-20T12:04:06Z","receivedAt":"2023-06-20T12:04:23Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Tue, Jun 20, 2023 at 1:26 PM Tao Klerks <tao@klerks.biz> wrote:\n>\n> * Is there any way to run a single git command in a \"don't use\n> promisors\" context?\n\nMy apologies for the self-reply, I did find a workaround here:\n\ngit -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC\n\nI don't understand what's happening here at all, because setting\n\"remote.origin.url\" to \"True\" in this way works to prevent the\nnormally configured URL from kicking in, but setting a different value\n(or empty value) does not. The following will end up fetching from the\noriginally-configured URL, completely ignoring the passed-in config:\n\ngit -c remote.origin.url=BADURL log SOME_HASH_NOT_IN_REFSPEC\n\nI'm still interested to know whether there's a less-hacky way of\ndetermining \"do I have this commit locally?\", of course, but fwiw I am\nunblocked.\n"},{"id":"478571","messageId":"xmqqo7lam1ei.fsf@gitster.g","threadId":"59884","inReplyTo":"CAPMMpog8Hv_KcjNxbh_wzjwrFYt7TuTvrVy1XEtJMm6RWSKzRg@mail.gmail.com","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-20T18:31:01Z","receivedAt":"2023-06-20T18:31:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tao Klerks <tao@klerks.biz> writes:\n\n> My apologies for the self-reply, I did find a workaround here:\n>\n> git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC\n>\n> I don't understand what's happening here at all, because setting\n> \"remote.origin.url\" to \"True\" in this way works to prevent the\n> normally configured URL from kicking in, ...\n\nInteresting.  This happens inside remote.c:handle_config() where\ngit_config_string() is used to reject a non-string value given to\nthe \"remote.<name>.url\" variable and abort the process to read from\nthe configuration file by returning -1 and causes whoever uses that\nconfiguration value to die.  As the command line configuration is\nread first, aborting the configuration reader early would mean that\nthe configured values would not even be read.\n\nI am not sure why this does not cause the entire thing to die,\nthough.  It requires further digging, for which I do not have time\nfor right now..\n\n> ... but setting a different value\n> (or empty value) does not.\n\nMost likely, this is because you are giving a syntactically correct\nvalue so the usual \"last one wins\" logic kicks in.\n\nIn hindsight, I think (1) the first one should probably fail the\n\"git log\" process (not just the lazy fetch subprocess), and (2)\nthere should be an explicit way, e.g. giving an empty string, to\n\"clear\" the list of .url accumulated so far.\n\n(2) may look something silly like this:\n\n remote.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git c/remote.c w/remote.c\nindex 0764fca0db..ecc146856a 100644\n--- c/remote.c\n+++ w/remote.c\n@@ -64,12 +64,22 @@ static const char *alias_url(const char *url, struct rewrites *r)\n \n static void add_url(struct remote *remote, const char *url)\n {\n+\tif (!*url) {\n+\t\tremote->url_nr = 0;\n+\t\treturn;\n+\t}\n+\n \tALLOC_GROW(remote->url, remote->url_nr + 1, remote->url_alloc);\n \tremote->url[remote->url_nr++] = url;\n }\n \n static void add_pushurl(struct remote *remote, const char *pushurl)\n {\n+\tif (!*pushurl) {\n+\t\tremote->pushurl_nr = 0;\n+\t\treturn;\n+\t}\n+\n \tALLOC_GROW(remote->pushurl, remote->pushurl_nr + 1, remote->pushurl_alloc);\n \tremote->pushurl[remote->pushurl_nr++] = pushurl;\n }\n\n"},{"id":"478576","messageId":"CAPMMpojUpJD21x2i_hshTB96TBFVd-_WRV54KHT2-4R8DUh8=Q@mail.gmail.com","threadId":"59884","inReplyTo":"CAPMMpog8Hv_KcjNxbh_wzjwrFYt7TuTvrVy1XEtJMm6RWSKzRg@mail.gmail.com","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2023-06-20T19:12:24Z","receivedAt":"2023-06-20T19:12:41Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Tue, Jun 20, 2023 at 2:04 PM Tao Klerks <tao@klerks.biz> wrote:\n>\n> On Tue, Jun 20, 2023 at 1:26 PM Tao Klerks <tao@klerks.biz> wrote:\n> >\n> > * Is there any way to run a single git command in a \"don't use\n> > promisors\" context?\n>\n> My apologies for the self-reply, I did find a workaround here:\n>\n> git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC\n>\n\nAnother self-reply unfortunately: This workaround stops working in git 2.39 :(\n\nI haven't understood exactly what's going on, but I guess the fetch\nfailure was explicitly curtailing *something* extremely expensive that\ngets to go forward anyway as of 2.39.\n\nI imagine it's related to the changelog entry \"Remove error detection\nfrom a function that fetches from promisor remotes\":\nhttps://github.com/gitgitgadget/git/blob/79bdd48716a4c455bdc8ffd91d57a18d5cd55baa/Documentation/RelNotes/2.39.0.txt\n\nI'm back to begging for any hints here: Any idea how I can determine\nwhether a given commit object exists locally, *without causing it to\nbe fetched by the act of checking for it?*\n\nThanks,\nTao\n"},{"id":"478577","messageId":"CAPMMpoj+5XT9LwL==ih-cmdYyz6RN6Ysr=g661keTOTH-QrK-g@mail.gmail.com","threadId":"59884","inReplyTo":"xmqqo7lam1ei.fsf@gitster.g","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2023-06-20T19:41:27Z","receivedAt":"2023-06-20T19:41:42Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Tue, Jun 20, 2023 at 8:31 PM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Tao Klerks <tao@klerks.biz> writes:\n>\n> > My apologies for the self-reply, I did find a workaround here:\n> >\n> > git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC\n> >\n> > I don't understand what's happening here at all, because setting\n> > \"remote.origin.url\" to \"True\" in this way works to prevent the\n> > normally configured URL from kicking in, ...\n>\n> Interesting.  This happens inside remote.c:handle_config() where\n> git_config_string() is used to reject a non-string value given to\n> the \"remote.<name>.url\" variable and abort the process to read from\n> the configuration file by returning -1 and causes whoever uses that\n> configuration value to die.  As the command line configuration is\n> read first, aborting the configuration reader early would mean that\n> the configured values would not even be read.\n>\n> I am not sure why this does not cause the entire thing to die,\n> though.  It requires further digging, for which I do not have time\n> for right now..\n\nI'm reasonably sure this is because the process that reads this config\nvalue is a subprocess, and that subprocess dying was actually\n*helping* me in this case.\n\n>\n> > ... but setting a different value\n> > (or empty value) does not.\n>\n> Most likely, this is because you are giving a syntactically correct\n> value so the usual \"last one wins\" logic kicks in.\n>\n\nI don't understand - surely what's provided \"-c\" should beat what's in\nthe repo's .config? \"Last one wins\" is exactly what I would expect,\nbut don't seem to be seeing...\n\n> In hindsight, I think (1) the first one should probably fail the\n> \"git log\" process (not just the lazy fetch subprocess), and (2)\n> there should be an explicit way, e.g. giving an empty string, to\n> \"clear\" the list of .url accumulated so far.\n>\n> (2) may look something silly like this:\n>\n>  remote.c | 10 ++++++++++\n>  1 file changed, 10 insertions(+)\n>\n> diff --git c/remote.c w/remote.c\n> index 0764fca0db..ecc146856a 100644\n> --- c/remote.c\n> +++ w/remote.c\n> @@ -64,12 +64,22 @@ static const char *alias_url(const char *url, struct rewrites *r)\n>\n>  static void add_url(struct remote *remote, const char *url)\n>  {\n> +       if (!*url) {\n> +               remote->url_nr = 0;\n> +               return;\n> +       }\n> +\n>         ALLOC_GROW(remote->url, remote->url_nr + 1, remote->url_alloc);\n>         remote->url[remote->url_nr++] = url;\n>  }\n>\n>  static void add_pushurl(struct remote *remote, const char *pushurl)\n>  {\n> +       if (!*pushurl) {\n> +               remote->pushurl_nr = 0;\n> +               return;\n> +       }\n> +\n>         ALLOC_GROW(remote->pushurl, remote->pushurl_nr + 1, remote->pushurl_alloc);\n>         remote->pushurl[remote->pushurl_nr++] = pushurl;\n>  }\n>\n\n\nThank you, I'll play with this when I get the chance.\n"},{"id":"478612","messageId":"20230621064459.GA607974@coredump.intra.peff.net","threadId":"59884","inReplyTo":"CAPMMpojUpJD21x2i_hshTB96TBFVd-_WRV54KHT2-4R8DUh8=Q@mail.gmail.com","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-21T06:44:59Z","receivedAt":"2023-06-21T06:47:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 09:12:24PM +0200, Tao Klerks wrote:\n\n> I'm back to begging for any hints here: Any idea how I can determine\n> whether a given commit object exists locally, *without causing it to\n> be fetched by the act of checking for it?*\n\nThis is not very efficient, but:\n\n  git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered |\n  grep $some_sha1\n\nwill tell you whether we have the object locally.\n\nI don't work with partial clones often, but it feels like being able to\nsay:\n\n  git --no-partial-fetch cat-file ...\n\nwould be a useful primitive to have. The implementation might start\nsomething like this:\n\ndiff --git a/object-file.c b/object-file.c\nindex 7c1af5c8db..494cdd7706 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1555,6 +1555,14 @@ void disable_obj_read_lock(void)\n \n int fetch_if_missing = 1;\n \n+static int allow_lazy_fetch(void)\n+{\n+\tstatic int ret = -1;\n+\tif (ret < 0)\n+\t\tret = git_env_bool(\"GIT_PARTIAL_FETCH\", 1);\n+\treturn ret;\n+}\n+\n static int do_oid_object_info_extended(struct repository *r,\n \t\t\t\t       const struct object_id *oid,\n \t\t\t\t       struct object_info *oi, unsigned flags)\n@@ -1622,6 +1630,7 @@ static int do_oid_object_info_extended(struct repository *r,\n \n \t\t/* Check if it is a missing object */\n \t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n+\t\t    allow_lazy_fetch() &&\n \t\t    !already_retried &&\n \t\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n \t\t\tpromisor_remote_get_direct(r, real, 1);\n\nand then have git.c populate the environment variable, similar to how we\nhandle --literal-pathspecs, etc.\n\nThat fetch_if_missing kind of does the same thing, but it's mostly\ncontrolled by programs themselves which try to handle missing remote\nobjects specially. It does seem like you might be able to bend it to\nyour will here, though. I think without any patches that:\n\n  git rev-list --objects --exclude-promisor-objects $oid\n\nwill tell you whether we have the object or not (since it turns off\nfetch_if_missing, and thus will either succeed, printing nothing, or\nbail if the object can't be found). It feels like --missing=error should\nfunction similarly, but it seems to still lazy-fetch (I guess since it's\nthe default, the point is to just find truly unavailable objects). Using\n--missing=print disables the lazy-fetch, but it seems to bail\nimmediately if you ask it about a missing object (I didn't dig, but my\nguess is that --missing is mostly about objects we traverse, not the\ninitial tips).\n\n-Peff\n"},{"id":"478613","messageId":"20230621065446.GB607974@coredump.intra.peff.net","threadId":"59884","inReplyTo":"xmqqo7lam1ei.fsf@gitster.g","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-21T06:54:46Z","receivedAt":"2023-06-21T06:55:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 11:31:01AM -0700, Junio C Hamano wrote:\n\n> In hindsight, I think (1) the first one should probably fail the\n> \"git log\" process (not just the lazy fetch subprocess), and (2)\n> there should be an explicit way, e.g. giving an empty string, to\n> \"clear\" the list of .url accumulated so far.\n\nAgreed. I think in general that any \"list\" config that is not doing\nlast-one-wins should have let a blank entry reset the list.\n\n> (2) may look something silly like this:\n\nI know you said \"silly\", but in case anybody wants to follow up on\nthis...\n\n> diff --git c/remote.c w/remote.c\n> index 0764fca0db..ecc146856a 100644\n> --- c/remote.c\n> +++ w/remote.c\n> @@ -64,12 +64,22 @@ static const char *alias_url(const char *url, struct rewrites *r)\n>  \n>  static void add_url(struct remote *remote, const char *url)\n>  {\n> +\tif (!*url) {\n> +\t\tremote->url_nr = 0;\n> +\t\treturn;\n> +\t}\n\n...it probably needs to free() the entries from 0..url_nr, and then also\nfree the unused empty-string \"url\".\n\nBut I think there's some questionable memory-ownership stuff going on\nhere. The main config caller passes in a newly allocated string, handing\nover ownership to this list (despite the \"const\" parameter!). But in\nadd_url_alias(), we take a single allocated \"url\" string and put it in\nboth the \"url\" and \"pushurl\" lists. The former goes through alias_url()\nwhich sometimes allocates a new string and sometimes does not.\n\nProbably both of these would be better off as strvecs or string_lists.\n\n-Peff\n"},{"id":"478619","messageId":"CAPMMpoha6rBA-T-7cn3DQT_nbNfknigLTky55x0TEmt4Ay2GRA@mail.gmail.com","threadId":"59884","inReplyTo":"20230621064459.GA607974@coredump.intra.peff.net","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2023-06-21T10:10:33Z","receivedAt":"2023-06-21T10:11:58Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Wed, Jun 21, 2023 at 8:45 AM Jeff King <peff@peff.net> wrote:\n>\n> On Tue, Jun 20, 2023 at 09:12:24PM +0200, Tao Klerks wrote:\n>\n> > I'm back to begging for any hints here: Any idea how I can determine\n> > whether a given commit object exists locally, *without causing it to\n> > be fetched by the act of checking for it?*\n>\n> This is not very efficient, but:\n>\n>   git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered |\n>   grep $some_sha1\n>\n> will tell you whether we have the object locally.\n>\n\nThanks so much for your help!\n\nin Windows (msys or git bash) this is still very slow in my repo with\n6,500,000 local objects - around 60s - but in linux on the same repo\nit's quite a lot faster, at 5s. A large proportion of my users are on\nWindows though, so I don't think this will be \"good enough\" for my\npurposes, when I often need to check for the existence of dozens or\neven hundreds of commits.\n\n> I don't work with partial clones often, but it feels like being able to\n> say:\n>\n>   git --no-partial-fetch cat-file ...\n>\n> would be a useful primitive to have.\n\nIt feels that way to me, yes!\n\nOn the other hand, I find very little demand for it when I search \"the\ninternet\" - or I don't know how to search for it.\n\n\n> The implementation might start\n> something like this:\n>\n> diff --git a/object-file.c b/object-file.c\n> index 7c1af5c8db..494cdd7706 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1555,6 +1555,14 @@ void disable_obj_read_lock(void)\n>\n>  int fetch_if_missing = 1;\n>\n> +static int allow_lazy_fetch(void)\n> +{\n> +       static int ret = -1;\n> +       if (ret < 0)\n> +               ret = git_env_bool(\"GIT_PARTIAL_FETCH\", 1);\n> +       return ret;\n> +}\n> +\n>  static int do_oid_object_info_extended(struct repository *r,\n>                                        const struct object_id *oid,\n>                                        struct object_info *oi, unsigned flags)\n> @@ -1622,6 +1630,7 @@ static int do_oid_object_info_extended(struct repository *r,\n>\n>                 /* Check if it is a missing object */\n>                 if (fetch_if_missing && repo_has_promisor_remote(r) &&\n> +                   allow_lazy_fetch() &&\n>                     !already_retried &&\n>                     !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n>                         promisor_remote_get_direct(r, real, 1);\n>\n> and then have git.c populate the environment variable, similar to how we\n> handle --literal-pathspecs, etc.\n>\n> That fetch_if_missing kind of does the same thing, but it's mostly\n> controlled by programs themselves which try to handle missing remote\n> objects specially.\n\nThanks, I will play with this if I get the chance. That said, I don't\ncontrol my users' distributions of Git, so on a purely practical basis\nI'm looking for something that will work in git 2.39 to whatever\nfuture version would introduce such a capability. (before 2.39, the\n\"set remote to False\" hack works)\n\n> It does seem like you might be able to bend it to\n> your will here, though. I think without any patches that:\n>\n>   git rev-list --objects --exclude-promisor-objects $oid\n>\n> will tell you whether we have the object or not (since it turns off\n> fetch_if_missing, and thus will either succeed, printing nothing, or\n> bail if the object can't be found).\n\nThis behaves in a way that I don't understand:\n\nIn the repo that I'm working in, this command runs successfully\n*without fetching*, but it takes a *very* long time - 300+ seconds -\nmuch longer than even the \"inefficient\" 'cat-file'-based printing of\nall (6.5M) local object ids that you proposed above. I haven't\nattempted to understand what's going on in there (besides running with\nGIT_TRACE2_PERF, which showed nothing interesting), but the idea that\ngit would have to work super-hard to find an object by its ID seems\ncounter to everything I know about it. Would there be value in my\ntrying to understand & reproduce this in a shareable repo, or is there\nalready an explanation as to why this command could/should ever do\nnon-trivial work, even in the largest partial repos?\n\n> It feels like --missing=error should\n> function similarly, but it seems to still lazy-fetch (I guess since it's\n> the default, the point is to just find truly unavailable objects). Using\n> --missing=print disables the lazy-fetch, but it seems to bail\n> immediately if you ask it about a missing object (I didn't dig, but my\n> guess is that --missing is mostly about objects we traverse, not the\n> initial tips).\n\nWoah, \"--missing=print\" seems to work!!!\n\nThe following gives me the commit hash if I have it locally, and an\nerror otherwise - consistently across linux and windows, git versions\n2.41, 2.39, 2.38, and 2.36 - without fetching, and without crazy\nCPU-churning:\n\ngit rev-list --missing=print -1 $oid\n\nThank you thank you thank you!\n\nI feel like I should try to work something into the doc about this,\nbut I'm not sure how to express this: \"--missing=error is the default,\nbut it doesn't actually error out when you're explicitly asking about\na missing commit, it fetches it instead - but --missing=print actually\n*does* error out if you explicitly ask about a missing commit\" seems\nlike a strange thing to be saying.\n\nThanks again for finding me an efficient working strategy here!\n"},{"id":"478870","messageId":"20230627080954.GF1226768@coredump.intra.peff.net","threadId":"59884","inReplyTo":"CAPMMpoha6rBA-T-7cn3DQT_nbNfknigLTky55x0TEmt4Ay2GRA@mail.gmail.com","subject":"Re: Determining whether you have a commit locally, in a partial clone?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-27T08:09:54Z","receivedAt":"2023-06-27T08:10:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 21, 2023 at 12:10:33PM +0200, Tao Klerks wrote:\n\n> > This is not very efficient, but:\n> >\n> >   git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered |\n> >   grep $some_sha1\n> >\n> > will tell you whether we have the object locally.\n> >\n> \n> Thanks so much for your help!\n> \n> in Windows (msys or git bash) this is still very slow in my repo with\n> 6,500,000 local objects - around 60s - but in linux on the same repo\n> it's quite a lot faster, at 5s. A large proportion of my users are on\n> Windows though, so I don't think this will be \"good enough\" for my\n> purposes, when I often need to check for the existence of dozens or\n> even hundreds of commits.\n\nYeah, it's just a lot of object names to print, most of which you don't\ncare about. :)\n\nThe more efficient thing would be to open the actual pack .idx files and\nlook for the names via binary search. I don't think you can convince git\nto do that, though I suspect you could write a trivial libgit2 program\nthat does.\n\n> > I don't work with partial clones often, but it feels like being able to\n> > say:\n> >\n> >   git --no-partial-fetch cat-file ...\n> >\n> > would be a useful primitive to have.\n> \n> It feels that way to me, yes!\n> \n> On the other hand, I find very little demand for it when I search \"the\n> internet\" - or I don't know how to search for it.\n\nI think partial clones are still new enough that not many people are\nusing them heavily. And when they do, not managing the partial state at\na very advanced level; I think tools for pruning locally cached objects\n(which you could refetch) is only just being worked on now.\n\n> > It does seem like you might be able to bend it to\n> > your will here, though. I think without any patches that:\n> >\n> >   git rev-list --objects --exclude-promisor-objects $oid\n> >\n> > will tell you whether we have the object or not (since it turns off\n> > fetch_if_missing, and thus will either succeed, printing nothing, or\n> > bail if the object can't be found).\n> \n> This behaves in a way that I don't understand:\n> \n> In the repo that I'm working in, this command runs successfully\n> *without fetching*, but it takes a *very* long time - 300+ seconds -\n> much longer than even the \"inefficient\" 'cat-file'-based printing of\n> all (6.5M) local object ids that you proposed above. I haven't\n> attempted to understand what's going on in there (besides running with\n> GIT_TRACE2_PERF, which showed nothing interesting), but the idea that\n> git would have to work super-hard to find an object by its ID seems\n> counter to everything I know about it. Would there be value in my\n> trying to understand & reproduce this in a shareable repo, or is there\n> already an explanation as to why this command could/should ever do\n> non-trivial work, even in the largest partial repos?\n\nI think it's actually doing the gigantic traversal (and just limiting it\nwhen it sees objects that are not available). You probably want\n\"--no-walk\" at least, but really you don't even want to walk the trees\nof any commits you specify (so you'd want to omit \"--objects\" if you are\nasking about a commit, and otherwise include it, which is slightly\nawkward).\n\n> > It feels like --missing=error should\n> > function similarly, but it seems to still lazy-fetch (I guess since it's\n> > the default, the point is to just find truly unavailable objects). Using\n> > --missing=print disables the lazy-fetch, but it seems to bail\n> > immediately if you ask it about a missing object (I didn't dig, but my\n> > guess is that --missing is mostly about objects we traverse, not the\n> > initial tips).\n> \n> Woah, \"--missing=print\" seems to work!!!\n> \n> The following gives me the commit hash if I have it locally, and an\n> error otherwise - consistently across linux and windows, git versions\n> 2.41, 2.39, 2.38, and 2.36 - without fetching, and without crazy\n> CPU-churning:\n> \n> git rev-list --missing=print -1 $oid\n> \n> Thank you thank you thank you!\n\nHmph, I thought I tried that before and it didn't work, but it seems to\nwork for me now. I guess I was hoping to have it print the missing\nobject rather than exiting with an error, but if you do one object at a\ntime then the error is sufficient signal. :)\n\nYou might want \"--objects\" if you're going to ask about non-commits.\nThough it might not be necessary. I suspect Git would bail trying to\nlook up the object in the first place if we don't have it, and if we do\nhave it then it just becomes a silent noop.\n\n> I feel like I should try to work something into the doc about this,\n> but I'm not sure how to express this: \"--missing=error is the default,\n> but it doesn't actually error out when you're explicitly asking about\n> a missing commit, it fetches it instead - but --missing=print actually\n> *does* error out if you explicitly ask about a missing commit\" seems\n> like a strange thing to be saying.\n\nI think we are relying on the side effect that everything except\n--missing=error will turn off auto-fetching. I don't know if that's\nsomething we'd want to document. It seems reasonable to me that we might\nlater change the implementation so that we kick in the --missing\nbehavior only after parsing the initial list of traversal tips (I mean,\nI don't know why we would do that in particular, but it seems like the\nkind of thing we'd want to reserve as an implementation detail subject\nto change).\n\nI do think in the long run that a big \"--do-not-lazy-fetch\" flag would\nbe the right solution to let the user tell us what they want.\n\n> Thanks again for finding me an efficient working strategy here!\n\nI'm glad it worked. I was mostly just thinking out loud. ;)\n\n-Peff\n"}]}