{"thread":{"id":"52219","subject":"[BUG/FEATURE] Git pushing and fetching many more objects than strictly required","startedAt":"2019-11-08T14:11:22Z","lastAt":"2019-11-12T13:39:39Z","messageCount":5,"participants":["Paul van Loon","Jonathan Tan","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"385785","messageId":"bc43c70e-861d-425d-d7c4-73a3e25b7383@cheerful.com","threadId":"52219","inReplyTo":null,"subject":"[BUG/FEATURE] Git pushing and fetching many more objects than strictly required","fromName":"Paul van Loon","fromEmail":"nospam@cheerful.com","sentAt":"2019-11-08T14:06:00Z","receivedAt":"2019-11-08T14:11:22Z","isPatch":false,"sender":{"key":"nospam@cheerful.com","avatar":null},"body":"$ git --version\ngit version 2.21.0\n\nWhen fetching/pushing to a forked repo on Github, I've noticed several times that much more objects were being fetched or pushed than were strictly necessary.\nI'm not sure if it's a bug, or just a opportunity for performance improvement.\n\nI got these traces:\n\n$ git fetch --all\nFetching origin\nremote: Enumerating objects: 29507, done.\nremote: Counting objects: 100% (29507/29507), done.\nremote: Compressing objects: 100% (33/33), done.\nremote: Total 53914 (delta 29478), reused 29500 (delta 29471), pack-reused 24407\nReceiving objects: 100% (53914/53914), 31.90 MiB | 111.00 KiB/s, done.\nResolving deltas: 100% (42462/42462), completed with 7382 local objects.\n--\n\n$ git push -v origin 'refs/replace/*:refs/replace/*'\nPushing to XXXX\nEnumerating objects: 2681, done.\nCounting objects: 100% (2681/2681), done.\nDelta compression using up to 8 threads\nCompressing objects: 100% (1965/1965), done.\nWriting objects: 100% (2582/2582), 1.96 MiB | 1024 bytes/s, done.\nTotal 2582 (delta 95), reused 1446 (delta 58)\nremote: Resolving deltas: 100% (95/95), completed with 33 local objects.\nTo XXXX\n * [new branch]            refs/replace/XXXX -> refs/replace/XXXX\n--\n\nEspecially the pushing of a single replace commit involved 2582 objects to be written. This was after first a fetch was done.\n\nThis especially hurts on flaky and slow connections, especially the more objects need to be written/read, the bigger the chance of the connection failing.\nIn combination with the inability to restart fetches/pushes without down/uploading ALL objects again, this can become quite a frustrating experience.\n\nAny thoughts?\n\nRegards,\nPaul\n\n\n"},{"id":"385794","messageId":"20191108184723.246596-1-jonathantanmy@google.com","threadId":"52219","inReplyTo":"bc43c70e-861d-425d-d7c4-73a3e25b7383@cheerful.com","subject":"Re: [BUG/FEATURE] Git pushing and fetching many more objects than strictly required","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-11-08T18:47:23Z","receivedAt":"2019-11-08T18:47:29Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> $ git fetch --all\n> Fetching origin\n> remote: Enumerating objects: 29507, done.\n> remote: Counting objects: 100% (29507/29507), done.\n> remote: Compressing objects: 100% (33/33), done.\n> remote: Total 53914 (delta 29478), reused 29500 (delta 29471), pack-reused 24407\n> Receiving objects: 100% (53914/53914), 31.90 MiB | 111.00 KiB/s, done.\n> Resolving deltas: 100% (42462/42462), completed with 7382 local objects.\n> --\n> \n> $ git push -v origin 'refs/replace/*:refs/replace/*'\n> Pushing to XXXX\n> Enumerating objects: 2681, done.\n> Counting objects: 100% (2681/2681), done.\n> Delta compression using up to 8 threads\n> Compressing objects: 100% (1965/1965), done.\n> Writing objects: 100% (2582/2582), 1.96 MiB | 1024 bytes/s, done.\n> Total 2582 (delta 95), reused 1446 (delta 58)\n> remote: Resolving deltas: 100% (95/95), completed with 33 local objects.\n> To XXXX\n>  * [new branch]            refs/replace/XXXX -> refs/replace/XXXX\n\nCould you verify that refs/replace/XXXX (or one of its close ancestors)\nwas fetched by the \"git fetch --all\" command? \"--all\" fetches all\nremotes, not all refs.\n"},{"id":"385811","messageId":"e537d298-6431-c36a-2fc3-e41baa10d0a3@cheerful.com","threadId":"52219","inReplyTo":"20191108184723.246596-1-jonathantanmy@google.com","subject":"Re: [BUG/FEATURE] Git pushing and fetching many more objects than strictly required","fromName":"Paul van Loon","fromEmail":"nospam@cheerful.com","sentAt":"2019-11-08T20:54:02Z","receivedAt":"2019-11-08T20:54:32Z","isPatch":false,"sender":{"key":"nospam@cheerful.com","avatar":null},"body":"On 2019-11-08 19:47, Jonathan Tan wrote:\n>> $ git fetch --all\n>> Fetching origin\n>> remote: Enumerating objects: 29507, done.\n>> remote: Counting objects: 100% (29507/29507), done.\n>> remote: Compressing objects: 100% (33/33), done.\n>> remote: Total 53914 (delta 29478), reused 29500 (delta 29471), pack-reused 24407\n>> Receiving objects: 100% (53914/53914), 31.90 MiB | 111.00 KiB/s, done.\n>> Resolving deltas: 100% (42462/42462), completed with 7382 local objects.\n>> --\n>>\n>> $ git push -v origin 'refs/replace/*:refs/replace/*'\n>> Pushing to XXXX\n>> Enumerating objects: 2681, done.\n>> Counting objects: 100% (2681/2681), done.\n>> Delta compression using up to 8 threads\n>> Compressing objects: 100% (1965/1965), done.\n>> Writing objects: 100% (2582/2582), 1.96 MiB | 1024 bytes/s, done.\n>> Total 2582 (delta 95), reused 1446 (delta 58)\n>> remote: Resolving deltas: 100% (95/95), completed with 33 local objects.\n>> To XXXX\n>>  * [new branch]            refs/replace/XXXX -> refs/replace/XXXX\n>\n> Could you verify that refs/replace/XXXX (or one of its close ancestors)\n> was fetched by the \"git fetch --all\" command? \"--all\" fetches all\n> remotes, not all refs.\n\nNo, it was not fetched. HOWEVER, the ONLY thing the replace commit (1 single object) does is point to an existing parent object. No other new objects are referenced.\nThose 'ancestor' objects were all fetched.\n\nPaul\n\n"},{"id":"385813","messageId":"20191108212156.GA15365@sigill.intra.peff.net","threadId":"52219","inReplyTo":"e537d298-6431-c36a-2fc3-e41baa10d0a3@cheerful.com","subject":"Re: [BUG/FEATURE] Git pushing and fetching many more objects than strictly required","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-08T21:21:56Z","receivedAt":"2019-11-08T21:21:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 08, 2019 at 09:54:02PM +0100, Paul van Loon wrote:\n\n> >> $ git push -v origin 'refs/replace/*:refs/replace/*'\n> >> Pushing to XXXX\n> >> Enumerating objects: 2681, done.\n> >> Counting objects: 100% (2681/2681), done.\n> >> Delta compression using up to 8 threads\n> >> Compressing objects: 100% (1965/1965), done.\n> >> Writing objects: 100% (2582/2582), 1.96 MiB | 1024 bytes/s, done.\n> >> Total 2582 (delta 95), reused 1446 (delta 58)\n> >> remote: Resolving deltas: 100% (95/95), completed with 33 local objects.\n> >> To XXXX\n> >>  * [new branch]            refs/replace/XXXX -> refs/replace/XXXX\n> >\n> > Could you verify that refs/replace/XXXX (or one of its close ancestors)\n> > was fetched by the \"git fetch --all\" command? \"--all\" fetches all\n> > remotes, not all refs.\n> \n> No, it was not fetched. HOWEVER, the ONLY thing the replace commit (1 single object) does is point to an existing parent object. No other new objects are referenced.\n> Those 'ancestor' objects were all fetched.\n\nWas it a parent object at the tip of a ref?\n\nThe push protocol, unlike the fetch protocol, doesn't expend any effort\nto negotiate to find a common base. It just feeds the ref tips of the\nreceiver to pack-objects (which then does traverse down to a merge base,\nbut it can't always do so if the sender doesn't have all of the\nobjects).\n\nIt's hard to say more without having a reproducible case to look at.\n\nSome possible things to poke at:\n\n  - record the stdin from the local push to the local pack-objects,\n    which shows which objects we're planning to send and which we're\n    claiming the other side has. That would help determine if the push\n    isn't feeding enough information to pack-objects, or if pack-objects\n    isn't trying hard enough to find the minimal set of objects\n\n    There's not really an easy way to do this, but something like strace\n    might help.\n\n  - try building reachability bitmaps (e.g., \"git repack -adb\") in the\n    local clone. When those are present, pack-objects will compute the\n    object set more thoroughly (because it can do so efficiently).\n\nI don't _think_ the fact that it's in refs/replace should matter to push\n(in terms of what it feeds to pack-objects). But obviously another thing\nto try is whether pushing to or from a different ref has any impact.\n\n-Peff\n"},{"id":"386012","messageId":"61aa8bc7-997d-a7d7-e37e-92b30e04d005@cheerful.com","threadId":"52219","inReplyTo":"20191108212156.GA15365@sigill.intra.peff.net","subject":"Re: [BUG/FEATURE] Git pushing and fetching many more objects than strictly required","fromName":"Paul van Loon","fromEmail":"nospam@cheerful.com","sentAt":"2019-11-12T13:39:10Z","receivedAt":"2019-11-12T13:39:39Z","isPatch":false,"sender":{"key":"nospam@cheerful.com","avatar":null},"body":"On 2019-11-08 22:21, Jeff King wrote:\n> On Fri, Nov 08, 2019 at 09:54:02PM +0100, Paul van Loon wrote:\n>\n>>>> $ git push -v origin 'refs/replace/*:refs/replace/*'\n>>>> Pushing to XXXX\n>>>> Enumerating objects: 2681, done.\n>>>> Counting objects: 100% (2681/2681), done.\n>>>> Delta compression using up to 8 threads\n>>>> Compressing objects: 100% (1965/1965), done.\n>>>> Writing objects: 100% (2582/2582), 1.96 MiB | 1024 bytes/s, done.\n>>>> Total 2582 (delta 95), reused 1446 (delta 58)\n>>>> remote: Resolving deltas: 100% (95/95), completed with 33 local objects.\n>>>> To XXXX\n>>>>  * [new branch]            refs/replace/XXXX -> refs/replace/XXXX\n>>>\n>>> Could you verify that refs/replace/XXXX (or one of its close ancestors)\n>>> was fetched by the \"git fetch --all\" command? \"--all\" fetches all\n>>> remotes, not all refs.\n>>\n>> No, it was not fetched. HOWEVER, the ONLY thing the replace commit (1 single object) does is point to an existing parent object. No other new objects are referenced.\n>> Those 'ancestor' objects were all fetched.\n>\n> Was it a parent object at the tip of a ref?\n\nNo, it was a newly created replace object (created with git replace --edit)\n\n>\n> The push protocol, unlike the fetch protocol, doesn't expend any effort\n> to negotiate to find a common base. It just feeds the ref tips of the\n> receiver to pack-objects (which then does traverse down to a merge base,\n> but it can't always do so if the sender doesn't have all of the\n> objects).\n\nSo this would be the opportunity for performance improvement I guess.\n\n>\n> It's hard to say more without having a reproducible case to look at.\n>\n> Some possible things to poke at:\n>\n>   - record the stdin from the local push to the local pack-objects,\n>     which shows which objects we're planning to send and which we're\n>     claiming the other side has. That would help determine if the push\n>     isn't feeding enough information to pack-objects, or if pack-objects\n>     isn't trying hard enough to find the minimal set of objects\n>\n>     There's not really an easy way to do this, but something like strace\n>     might help.\n\nThat's way above my Git expertise.\n\n>   - try building reachability bitmaps (e.g., \"git repack -adb\") in the\n>     local clone. When those are present, pack-objects will compute the\n>     object set more thoroughly (because it can do so efficiently).\n>\n> I don't _think_ the fact that it's in refs/replace should matter to push\n> (in terms of what it feeds to pack-objects). But obviously another thing\n> to try is whether pushing to or from a different ref has any impact.\n\nI'll do some additional experiments\n\n> -Peff\n>\n\n"}]}