{"thread":{"id":"59005","subject":"git fetch --dry-run changes the local repo and transfers data","startedAt":"2022-12-26T17:21:42Z","lastAt":"2023-01-03T11:07:24Z","messageCount":4,"participants":["Tim Hockin","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"469540","messageId":"CAO_RewZeUBVvqT+UgXL5V5EeTOt14zZ-5vJYA4YvNr-jNyupXg@mail.gmail.com","threadId":"59005","inReplyTo":null,"subject":"git fetch --dry-run changes the local repo and transfers data","fromName":"Tim Hockin","fromEmail":"thockin@google.com","sentAt":"2022-12-26T17:21:26Z","receivedAt":"2022-12-26T17:21:42Z","isPatch":false,"sender":{"key":"thockin@google.com","avatar":null},"body":"I'm trying to test if a remote repo has a given SHA. `ls-remote` does\nnot work for this unless it is a tag or a HEAD (which is not\nguaranteed for this case).\n\n`git fetch --dry-run $remote $rev` SEEMS to fit the bill, except it\nchanges local state. For example:\n\n```\n$ git cat-file -t f80f1b23b4cab2a295a091c623bb4746d188bd4a\nfatal: git cat-file: could not get object info\n\n$ git rev-parse FETCH_HEAD\n42d21cf12aab73ac8bc6245cc74ac9850bdf6989\n\n$ git fetch --dry-run file:///tmp/e2e.2527526915/repo\nf80f1b23b4cab2a295a091c623bb4746d188bd4a\nremote: Enumerating objects: 20, done.\nremote: Counting objects: 100% (20/20), done.\nremote: Compressing objects: 100% (9/9), done.\nremote: Total 18 (delta 6), reused 0 (delta 0), pack-reused 0\nUnpacking objects: 100% (18/18), 415.91 KiB | 84.00 KiB/s, done.\nFrom file:///tmp/e2e.2527526915/repo\n * branch            f80f1b23b4cab2a295a091c623bb4746d188bd4a -> FETCH_HEAD\n\n$ git cat-file -t f80f1b23b4cab2a295a091c623bb4746d188bd4a\ncommit\n\n$ git rev-parse FETCH_HEAD\n42d21cf12aab73ac8bc6245cc74ac9850bdf6989\n```\n\nFETCH_HEAD was not updated (good) but the rev in question is now\npresent locally (bad), further tested by making a very large commit\nand watching the fetch be very slow.  Also the same on non file://\nrepo (I tried https)\n\nAm I using --dry-run wrong (or misunderstanding its purpose)?\n\nTim\n"},{"id":"469548","messageId":"xmqqcz85t3dr.fsf@gitster.g","threadId":"59005","inReplyTo":"CAO_RewZeUBVvqT+UgXL5V5EeTOt14zZ-5vJYA4YvNr-jNyupXg@mail.gmail.com","subject":"Re: git fetch --dry-run changes the local repo and transfers data","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-12-27T12:52:48Z","receivedAt":"2022-12-27T12:52:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tim Hockin <thockin@google.com> writes:\n\n> I'm trying to test if a remote repo has a given SHA. `ls-remote` does\n> not work for this unless it is a tag or a HEAD (which is not\n> guaranteed for this case).\n>\n> `git fetch --dry-run $remote $rev` SEEMS to fit the bill, except it\n> changes local state. For example:\n\nWell, without changing the \"local state\", you cannot see if that\nreport repository has or does not have the commit.\n\n> FETCH_HEAD was not updated (good)\n\nNone of your refs are updated, either.\n\nThere are things that are not reachable from your refs (or other\nanchoring points like the index and the reflog).  As far as Git is\nconcerned, they don't exist [*], and that is why Git will happily\nremove them with \"git gc\", for example.  They are crufts and it is\neasier to think of them as not a part of \"local data\".\n\nSo I suspect that ...\n\n> but the rev in question is now\n> present locally (bad),\n\n... this complaint is a bit misguided.  After all, the procdure you\ngave above is exactly how you would ask the \"I'm trying to test if a\nremote repo has a given SHA\" question about commit f80f1b23b4ca.  If\nthe operation did not transfer data, you would not be able to get\nthe answer to that question.\n\n> Am I using --dry-run wrong (or misunderstanding its purpose)?\n\nMaybe (mis)understanding on Git's data model (see above [*]) is the\nroot cause of this confusion.\n"},{"id":"469562","messageId":"CAO_RewbVicTznpDeCDG0Uqng-MdQ_GKtp-Vgz8kmtaXoczQGOg@mail.gmail.com","threadId":"59005","inReplyTo":"xmqqcz85t3dr.fsf@gitster.g","subject":"Re: git fetch --dry-run changes the local repo and transfers data","fromName":"Tim Hockin","fromEmail":"thockin@google.com","sentAt":"2022-12-27T18:42:25Z","receivedAt":"2022-12-27T18:42:42Z","isPatch":false,"sender":{"key":"thockin@google.com","avatar":null},"body":"Thanks.  What threw me is that I expected it to be \"very fast\" and\n\"very cheap\" . If I commit a multi-gig file on the server, I see the\ndry-run fetch takes several seconds (clearly indicating some work\nproportional to the server repo size).  I don't want to transfer that\nfile on a dry-run, I hoped the server and client were both\ndry-running, andb the server could simply say \"here's metadata for\nwhat I _would have_ returned if this was real\".  Not possible?\n\nTim\n\nOn Tue, Dec 27, 2022 at 4:52 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Tim Hockin <thockin@google.com> writes:\n>\n> > I'm trying to test if a remote repo has a given SHA. `ls-remote` does\n> > not work for this unless it is a tag or a HEAD (which is not\n> > guaranteed for this case).\n> >\n> > `git fetch --dry-run $remote $rev` SEEMS to fit the bill, except it\n> > changes local state. For example:\n>\n> Well, without changing the \"local state\", you cannot see if that\n> report repository has or does not have the commit.\n>\n> > FETCH_HEAD was not updated (good)\n>\n> None of your refs are updated, either.\n>\n> There are things that are not reachable from your refs (or other\n> anchoring points like the index and the reflog).  As far as Git is\n> concerned, they don't exist [*], and that is why Git will happily\n> remove them with \"git gc\", for example.  They are crufts and it is\n> easier to think of them as not a part of \"local data\".\n>\n> So I suspect that ...\n>\n> > but the rev in question is now\n> > present locally (bad),\n>\n> ... this complaint is a bit misguided.  After all, the procdure you\n> gave above is exactly how you would ask the \"I'm trying to test if a\n> remote repo has a given SHA\" question about commit f80f1b23b4ca.  If\n> the operation did not transfer data, you would not be able to get\n> the answer to that question.\n>\n> > Am I using --dry-run wrong (or misunderstanding its purpose)?\n>\n> Maybe (mis)understanding on Git's data model (see above [*]) is the\n> root cause of this confusion.\n"},{"id":"469764","messageId":"Y7QMWN86dDLXc4dZ@coredump.intra.peff.net","threadId":"59005","inReplyTo":"CAO_RewbVicTznpDeCDG0Uqng-MdQ_GKtp-Vgz8kmtaXoczQGOg@mail.gmail.com","subject":"Re: git fetch --dry-run changes the local repo and transfers data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-03T11:07:04Z","receivedAt":"2023-01-03T11:07:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 27, 2022 at 10:42:25AM -0800, Tim Hockin wrote:\n\n> Thanks.  What threw me is that I expected it to be \"very fast\" and\n> \"very cheap\" . If I commit a multi-gig file on the server, I see the\n> dry-run fetch takes several seconds (clearly indicating some work\n> proportional to the server repo size).  I don't want to transfer that\n> file on a dry-run, I hoped the server and client were both\n> dry-running, andb the server could simply say \"here's metadata for\n> what I _would have_ returned if this was real\".  Not possible?\n\nNo, the server has no notion of a dry run.\n\nI think the best you could do with fetch is to ask for a smaller set of\nobjects. For example:\n\n  git fetch --depth=1 --filter=tree:0 \\\n    https://github.com/git/git \\\n    2e71cbbddd64695d43383c25c7a054ac4ff86882\n\nwill grab a single object. You can even \"git show -s 2e71cbbd\" on the\nresult to see it (the \"-s\" is important to avoid it fetching the trees\nto do a diff!). Two things to be aware of:\n\n  - this may have some lingering effects in your repository, as the\n    shallow and partial features store some metadata locally to make\n    sense of the situation. You're probably best off doing it in a\n    temporary repository.\n\n  - not all servers will support --filter; it has to be enabled in the\n    config.\n\nThere is potentially a more direct option, though. A while back, commit\na2ba162cda (object-info: support for retrieving object info, 2021-04-20)\nadded an extension that lets you get the size of an object on the\nserver. Unfortunately I don't think anybody ever wrote client-side\nsupport. So you'd have to rig up something yourself like:\n\n  # write git's packet format: 4-hex length followed by data\n  pkt() {\n    printf '%04x%s' \"$((4+${#1}))\" \"$1\"\n  }\n\n  # a sample input; you should be able to query multiple objects if you\n  # want by adding more \"oid\" lines\n  {\n    pkt \"command=object-info\"\n    printf \"0001\"\n    pkt \"size\"\n    pkt \"oid 2e71cbbddd64695d43383c25c7a054ac4ff86882\"\n    printf \"0000\"\n  } >input\n\n  # this makes a local request; it's important we're in v2 mode, since\n  # the extension only applies there. For http, I think you'd want\n  # something like:\n  #\n  #  curl -H 'Git-Protocol: version=2' https://example.com/repo.git/git-upload-pack\n  #\n  # but I didn't test it.\n  GIT_PROTOCOL=version=2 git-upload-pack /path/to/repo.git <input >output\n\nI've left parsing the output as an exercise for the reader. But you\nshould be able to notice whether the object is present or not based on\nthe result.\n\nNot all servers may support the extension. For example, I think GitHub's\nservers have disabled it.\n\n-Peff\n"}]}