{"thread":{"id":"55237","subject":"Argument list too long when fetching many missing objects from partial clone","startedAt":"2021-03-01T13:19:54Z","lastAt":"2021-03-03T06:42:28Z","messageCount":5,"participants":["Bagas Sanjaya","Jeff Hostetler","Bryan Turner","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"418034","messageId":"fd1ba192-b90c-ef20-0843-c0e47925c76a@gmail.com","threadId":"55237","inReplyTo":null,"subject":"Argument list too long when fetching many missing objects from partial clone","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-03-01T13:18:29Z","receivedAt":"2021-03-01T13:19:54Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nI'm poking around with partial clone. On my local server on my computer\nI have full clone of Git (git.git) repo.\n\nI do partial clone from that server as remote (which is actually on the same\ncomputer):\n\n    git clone https://my-local-server.git/myself/git.git --filter=blob:none\n\nInside the partial clone, I tried to convert it to full clone. First, I gather\nlist of missing objects:\n\n    git rev-list --objects --all --missing=print | grep -oP '^\\?\\K\\w+' > .git/missing.list\n\nThen I fetched those:\n\n    git fetch origin $(cat .git/missing.list)\n\nWhat did you expect to happen? (Expected behavior)\n\nAll missing objects fetched successfully\n\nWhat happened instead? (Actual behavior)\n\nGit returned:\n\n     -bash: /opt/git/bin/git: Argument list too long\n\nWhat's different between what you expected and what actually happened?\n\n(nothing)\n\nAnything else you want to add:\n\nI think the problem lies on how to feed list of 110K+ objects to `git fetch`.\n\nNote: Both the local server and the computer use Git 2.30.1\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n[System Info]\ngit version:\ngit version 2.30.1\ncpu: x86_64\nbuilt from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64\ncompiler info: gnuc: 9.3\nlibc info: glibc: 2.31\n$SHELL (typically, interactive shell): /bin/bash\n\n[Enabled Hooks]\n(none)\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"418058","messageId":"be2a9647-3e3c-b88f-0f86-729dba5488c3@jeffhostetler.com","threadId":"55237","inReplyTo":"fd1ba192-b90c-ef20-0843-c0e47925c76a@gmail.com","subject":"Re: Argument list too long when fetching many missing objects from partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-03-01T19:21:04Z","receivedAt":"2021-03-01T19:24:14Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/1/21 8:18 AM, Bagas Sanjaya wrote:\n> Thank you for filling out a Git bug report!\n> Please answer the following questions to help us understand your issue.\n> \n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n> I'm poking around with partial clone. On my local server on my computer\n> I have full clone of Git (git.git) repo.\n> \n> I do partial clone from that server as remote (which is actually on the \n> same\n> computer):\n> \n>     git clone https://my-local-server.git/myself/git.git --filter=blob:none\n> \n> Inside the partial clone, I tried to convert it to full clone. First, I \n> gather\n> list of missing objects:\n> \n>     git rev-list --objects --all --missing=print | grep -oP '^\\?\\K\\w+' > \n> .git/missing.list\n> \n> Then I fetched those:\n> \n>     git fetch origin $(cat .git/missing.list)\n> \n\nThat command will build a very very long command line and cause the\nshell to complain.  Instead, try passing the list of objects via stdin.\n\n     $ cat .git/missing.list | git fetch --stdin origin\n\n\n\n> What did you expect to happen? (Expected behavior)\n> \n> All missing objects fetched successfully\n> \n> What happened instead? (Actual behavior)\n> \n> Git returned:\n> \n>      -bash: /opt/git/bin/git: Argument list too long\n> \n> What's different between what you expected and what actually happened?\n> \n> (nothing)\n> \n> Anything else you want to add:\n> \n> I think the problem lies on how to feed list of 110K+ objects to `git \n> fetch`.\n> \n> Note: Both the local server and the computer use Git 2.30.1\n> \n> Please review the rest of the bug report below.\n> You can delete any lines you don't wish to share.\n> \n> [System Info]\n> git version:\n> git version 2.30.1\n> cpu: x86_64\n> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad\n> sizeof-long: 8\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 \n> 12:56:19 WIB 2021 x86_64\n> compiler info: gnuc: 9.3\n> libc info: glibc: 2.31\n> $SHELL (typically, interactive shell): /bin/bash\n> \n> [Enabled Hooks]\n> (none)\n> \n"},{"id":"418060","messageId":"CAGyf7-GJQU1s4H-pgVMEc_WB3C4ehppLieT2oiyxa5B8=yyjxA@mail.gmail.com","threadId":"55237","inReplyTo":"fd1ba192-b90c-ef20-0843-c0e47925c76a@gmail.com","subject":"Re: Argument list too long when fetching many missing objects from partial clone","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2021-03-01T19:22:57Z","receivedAt":"2021-03-01T19:26:41Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:\n>\n> Thank you for filling out a Git bug report!\n> Please answer the following questions to help us understand your issue.\n>\n> What did you do before the bug happened? (Steps to reproduce your issue)\n>\n> I'm poking around with partial clone. On my local server on my computer\n> I have full clone of Git (git.git) repo.\n>\n> I do partial clone from that server as remote (which is actually on the same\n> computer):\n>\n>     git clone https://my-local-server.git/myself/git.git --filter=blob:none\n>\n> Inside the partial clone, I tried to convert it to full clone. First, I gather\n> list of missing objects:\n>\n>     git rev-list --objects --all --missing=print | grep -oP '^\\?\\K\\w+' > .git/missing.list\n>\n> Then I fetched those:\n>\n>     git fetch origin $(cat .git/missing.list)\n\nSince you're on Git 2.30, instead of trying to place all the object\nIDs on the `git fetch` command line, have you tried adding `--stdin`\nand having it read from your `missing.list` directly? Something like\n`git fetch --stdin origin <.git/missing.list` might do what you need.\n\n>\n> What did you expect to happen? (Expected behavior)\n>\n> All missing objects fetched successfully\n>\n> What happened instead? (Actual behavior)\n>\n> Git returned:\n>\n>      -bash: /opt/git/bin/git: Argument list too long\n>\n> What's different between what you expected and what actually happened?\n>\n> (nothing)\n>\n> Anything else you want to add:\n>\n> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.\n>\n> Note: Both the local server and the computer use Git 2.30.1\n>\n> Please review the rest of the bug report below.\n> You can delete any lines you don't wish to share.\n>\n> [System Info]\n> git version:\n> git version 2.30.1\n> cpu: x86_64\n> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad\n> sizeof-long: 8\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64\n> compiler info: gnuc: 9.3\n> libc info: glibc: 2.31\n> $SHELL (typically, interactive shell): /bin/bash\n>\n> [Enabled Hooks]\n> (none)\n>\n> --\n> An old man doll... just what I always wanted! - Clara\n"},{"id":"418101","messageId":"f185e87e-d8d2-3ac4-9ca1-7b96ffb30562@gmail.com","threadId":"55237","inReplyTo":"CAGyf7-GJQU1s4H-pgVMEc_WB3C4ehppLieT2oiyxa5B8=yyjxA@mail.gmail.com","subject":"Re: Argument list too long when fetching many missing objects from partial clone","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-03-02T06:55:27Z","receivedAt":"2021-03-02T15:28:36Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"After trying your suggestion, I now get:\n\nerror: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413\nfatal: unable to write request to remote: Broken pipe\n\nOn 02/03/21 02.22, Bryan Turner wrote:\n> On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:\n>>\n>> Thank you for filling out a Git bug report!\n>> Please answer the following questions to help us understand your issue.\n>>\n>> What did you do before the bug happened? (Steps to reproduce your issue)\n>>\n>> I'm poking around with partial clone. On my local server on my computer\n>> I have full clone of Git (git.git) repo.\n>>\n>> I do partial clone from that server as remote (which is actually on the same\n>> computer):\n>>\n>>      git clone https://my-local-server.git/myself/git.git --filter=blob:none\n>>\n>> Inside the partial clone, I tried to convert it to full clone. First, I gather\n>> list of missing objects:\n>>\n>>      git rev-list --objects --all --missing=print | grep -oP '^\\?\\K\\w+' > .git/missing.list\n>>\n>> Then I fetched those:\n>>\n>>      git fetch origin $(cat .git/missing.list)\n> \n> Since you're on Git 2.30, instead of trying to place all the object\n> IDs on the `git fetch` command line, have you tried adding `--stdin`\n> and having it read from your `missing.list` directly? Something like\n> `git fetch --stdin origin <.git/missing.list` might do what you need.\n> \n>>\n>> What did you expect to happen? (Expected behavior)\n>>\n>> All missing objects fetched successfully\n>>\n>> What happened instead? (Actual behavior)\n>>\n>> Git returned:\n>>\n>>       -bash: /opt/git/bin/git: Argument list too long\n>>\n>> What's different between what you expected and what actually happened?\n>>\n>> (nothing)\n>>\n>> Anything else you want to add:\n>>\n>> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.\n>>\n>> Note: Both the local server and the computer use Git 2.30.1\n>>\n>> Please review the rest of the bug report below.\n>> You can delete any lines you don't wish to share.\n>>\n>> [System Info]\n>> git version:\n>> git version 2.30.1\n>> cpu: x86_64\n>> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad\n>> sizeof-long: 8\n>> sizeof-size_t: 8\n>> shell-path: /bin/sh\n>> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64\n>> compiler info: gnuc: 9.3\n>> libc info: glibc: 2.31\n>> $SHELL (typically, interactive shell): /bin/bash\n>>\n>> [Enabled Hooks]\n>> (none)\n>>\n>> --\n>> An old man doll... just what I always wanted! - Clara\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"418124","messageId":"YD7bczBsIR5rkqfc@camp.crustytoothpaste.net","threadId":"55237","inReplyTo":"f185e87e-d8d2-3ac4-9ca1-7b96ffb30562@gmail.com","subject":"Re: Argument list too long when fetching many missing objects from partial clone","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-03-03T00:42:27Z","receivedAt":"2021-03-03T06:42:28Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"[Note: we generally prefer inline responses.]\n\nOn 2021-03-02 at 06:55:27, Bagas Sanjaya wrote:\n> After trying your suggestion, I now get:\n> \n> error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413\n> fatal: unable to write request to remote: Broken pipe\n\nHmmm, this means that your local Git server is unhappy with the size of\nthe request.  That means your request is probably very large, and it's\nlikely that it is kindly asking you not to DoS it.\n\n> On 02/03/21 02.22, Bryan Turner wrote:\n> > On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:\n> > > Inside the partial clone, I tried to convert it to full clone. First, I gather\n> > > list of missing objects:\n> > > \n> > >      git rev-list --objects --all --missing=print | grep -oP '^\\?\\K\\w+' > .git/missing.list\n> > > \n> > > Then I fetched those:\n> > > \n> > >      git fetch origin $(cat .git/missing.list)\n> > \n> > Since you're on Git 2.30, instead of trying to place all the object\n> > IDs on the `git fetch` command line, have you tried adding `--stdin`\n> > and having it read from your `missing.list` directly? Something like\n> > `git fetch --stdin origin <.git/missing.list` might do what you need.\n\nYou may want to try this:\n\n  xargs git fetch origin < .git/missing.list\n\nThat will batch the requests to fit on the command line, which will\nprobably also have the pleasant side effect that it makes each\nindividual request smaller to avoid the 413.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"}]}