{"thread":{"id":"49537","subject":"Git wire protocol v2 fails fetching shallow changes with `pack has XXX unresolved deltas` on large repos with lots of tags Inbox x","startedAt":"2018-10-11T09:20:51Z","lastAt":"2018-10-19T13:59:03Z","messageCount":4,"participants":["Arturas Moskvinas","Jonathan Tan"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"360154","messageId":"CAGY-PBgsG-T3JY=awszwGMpFx+jDx-a1FcV0s6vR067bSqgZaA@mail.gmail.com","threadId":"49537","inReplyTo":null,"subject":"Git wire protocol v2 fails fetching shallow changes with `pack has XXX unresolved deltas` on large repos with lots of tags Inbox x","fromName":"Arturas Moskvinas","fromEmail":"arturas@uber.com","sentAt":"2018-10-11T09:20:36Z","receivedAt":"2018-10-11T09:20:51Z","isPatch":false,"sender":{"key":"arturas@uber.com","avatar":null},"body":"Hello,\n\nOn large repositories with lots of tags - git wire protocol v2 fails\nto fetch shallow changes, it fails with error `pack has XXX unresolved\ndeltas`. Unfortunately I didn't find easy way to reproduce it except\ncloning+fetching chromium repository, the way jenkins does.\nReproduction steps:\n```\n$ git clone --mirror https://chromium.googlesource.com/chromium/src chromium\nCloning into bare repository 'chromium'...\nremote: Sending approximately 12.66 GiB ...\nremote: Counting objects: 8523271, done\nremote: Finding sources: 100% (8399287/8399287)\nremote: Total 19829104 (delta 11897257), reused 19828780 (delta 11897257)\nReceiving objects: 100% (19829104/19829104), 20.45 GiB | 29.17 MiB/s, done.\nResolving deltas: 100% (11897257/11897257), done\n$ mkdir chromium-test\n$ cd chromium-test\n$ git init\nInitialized empty Git repository in /home/arturas/.git/\n$ git -c protocol.version=2 fetch --tags file://$(pwd)/../chromium/\n+refs/heads/*:refs/remotes/origin/* --depth=1\nremote: Enumerating objects: 4614683, done.\nremote: Counting objects: 100% (4614683/4614683), done.\nremote: Compressing objects: 100% (1556691/1556691), done.\nReceiving objects: 100% (4614683/4614683), 8.29 GiB | 24.71 MiB/s, done.\nremote: Total 4614683 (delta 3470249), reused 3989191 (delta 2923800)\nResolving deltas: 100% (3470249/3470249), done.\nFrom file:///home/arturas/chromium-test/../chromium\n * [new branch] ignore/bar -> origin/ignore/bar\n * [new branch] ignore/foo -> origin/ignore/foo\n * [new branch] infra/config -> origin/infra/config\n * [new branch] lkgr -> origin/lkgr\n * [new branch] master -> origin/master\n * [new tag] 10.0.601.0 -> 10.0.601.0\n...\n$ git -c protocol.version=2 fetch --tags file://$(pwd)/../chromium/\n+refs/heads/*:refs/remotes/origin/* --depth=1\nremote: Enumerating objects: 642969, done.\nremote: Counting objects: 100% (588314/588314), done.\nremote: Compressing objects: 100% (240564/240564), done.\nReceiving objects: 100% (571353/571353), 967.83 MiB | 10.52 MiB/s, done.\nremote: Total 571353 (delta 372206), reused 497895 (delta 310113)\nResolving deltas: 99% (371726/372206), completed with 5582 local objects.\nfatal: pack has 480 unresolved deltas\nfatal: index-pack failed\n$ git --version\ngit version 2.19.1\n```\nNo changes are needed in mirrored repository. Crash happens both with\n2.18.0 and 2.19.1 git versions. Having repository locally is not\nrequired but reduces test runtime, you can quite reliably reproduce\nissue when fetching over net directly from chromium.orgbypassing\nmirroring step.\n\n--\nArturas Moskvinas\n"},{"id":"360370","messageId":"20181013004356.257709-1-jonathantanmy@google.com","threadId":"49537","inReplyTo":"CAGY-PBgsG-T3JY=awszwGMpFx+jDx-a1FcV0s6vR067bSqgZaA@mail.gmail.com","subject":"Re: Git wire protocol v2 fails fetching shallow changes with `pack has XXX unresolved deltas` on large repos with lots of tags Inbox x","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-13T00:43:56Z","receivedAt":"2018-10-13T00:44:04Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On large repositories with lots of tags - git wire protocol v2 fails\n> to fetch shallow changes, it fails with error `pack has XXX unresolved\n> deltas`. Unfortunately I didn't find easy way to reproduce it except\n> cloning+fetching chromium repository, the way jenkins does.\n> Reproduction steps:\n\n[snip]\n\nThanks for your bug report and reproduction steps. I managed to\nreproduce your issue and took a look.\n\nThe main issue seems to be that with v2, upload-pack doesn't pass\n\"--shallow X\" to pack-objects (the write_one_shallow() callback is never\ncalled, even if I change the \"if (shallow_nr)\" to \"if (1)\"), so\npack-objects probably doesn't know that some objects cannot be used as\ndelta bases. (With v0, write_one_shallow() is indeed called.) The issue\nprobably lies in how v0 and v2 handle client-provided shallows\ndifferently.\n\nThere also seems to be another issue in that negotiation occurs\ndifferently in these 2 protocols - I see the full list of \"have\" lines\nbeing sent in the final request to the server in v0, but a very limited\nlist in v2. This might be because of the ref prefix limiting in v2, but\nI haven't fully investigated it.\n\nI'll look some more into this.\n"},{"id":"360784","messageId":"20181017205340.190036-1-jonathantanmy@google.com","threadId":"49537","inReplyTo":"CAGY-PBgsG-T3JY=awszwGMpFx+jDx-a1FcV0s6vR067bSqgZaA@mail.gmail.com","subject":"Re: Git wire protocol v2 fails fetching shallow changes with `pack has XXX unresolved deltas` on large repos with lots of tags Inbox x","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-17T20:53:40Z","receivedAt":"2018-10-17T20:53:46Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> No changes are needed in mirrored repository. Crash happens both with\n> 2.18.0 and 2.19.1 git versions. Having repository locally is not\n> required but reduces test runtime, you can quite reliably reproduce\n> issue when fetching over net directly from chromium.orgbypassing\n> mirroring step.\n\nDo you have the reproduction steps for this? If I run\n\n git -c protocol.version=2 fetch --tags \\\n https://chromium.googlesource.com/chromium/src \\\n +refs/heads/*:refs/remotes/origin/* --depth=1\n\nrepeatedly in the same repository, it succeeds. (And I've checked with\nGIT_TRACE_PACKET that it uses protocol v2.)\n"},{"id":"360915","messageId":"CAGY-PBiKaKwhjyHCDF5Crq6z-D17WMW5iqUjfSNiQqaHcZP0dw@mail.gmail.com","threadId":"49537","inReplyTo":"20181017205340.190036-1-jonathantanmy@google.com","subject":"Re: Git wire protocol v2 fails fetching shallow changes with `pack has XXX unresolved deltas` on large repos with lots of tags Inbox x","fromName":"Arturas Moskvinas","fromEmail":"arturas@uber.com","sentAt":"2018-10-19T13:58:49Z","receivedAt":"2018-10-19T13:59:03Z","isPatch":false,"sender":{"key":"arturas@uber.com","avatar":null},"body":"Hello,\n\nUnfortunately I do not have for such step. On one of our internal\nrepositories it is failing but unreliably when fetching from remote.\n\n--\nArturas Moskvinas\n\nOn Wed, Oct 17, 2018 at 11:53 PM Jonathan Tan wrote:\n>\n> > No changes are needed in mirrored repository. Crash happens both with\n> > 2.18.0 and 2.19.1 git versions. Having repository locally is not\n> > required but reduces test runtime, you can quite reliably reproduce\n> > issue when fetching over net directly from chromium.orgbypassing\n> > mirroring step.\n>\n> Do you have the reproduction steps for this? If I run\n>\n> git -c protocol.version=2 fetch --tags \\\n> https://chromium.googlesource.com/chromium/src \\\n> +refs/heads/*:refs/remotes/origin/* --depth=1\n>\n> repeatedly in the same repository, it succeeds. (And I've checked with\n> GIT_TRACE_PACKET that it uses protocol v2.)\n"}]}