{"thread":{"id":"59282","subject":"[BUGREPORT] Why is git-push fetching content?","startedAt":"2023-02-21T22:02:23Z","lastAt":"2023-07-08T08:51:53Z","messageCount":7,"participants":["Sean Allred","brian m. carlson","Tao Klerks"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"472411","messageId":"m0ttze4qzl.fsf@epic96565.epic.com","threadId":"59282","inReplyTo":null,"subject":"[BUGREPORT] Why is git-push fetching content?","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-02-21T22:01:04Z","receivedAt":"2023-02-21T22:02:23Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n\n    # in a new directory,\n    cd $(mktemp -d)\n\n    # initialize a new repository\n    git init\n\n    # fetch a single commit from a remote\n    git fetch --filter=tree:0 --depth=1 $REMOTE $COMMIT_OID\n\n    # create a ref on that remote\n    git push --no-verify $REMOTE $COMMIT_OID:$REFNAME\n\nWhat did you expect to happen? (Expected behavior)\n\n    I expected this process to complete very, very quickly. We believe\n    the version where it had been doing so was ~2.37.\n\nWhat happened instead? (Actual behavior)\n\n    The fetch completes nearly instantly as expected. We receive ~200B\n    from the remote for the commit object itself. What's truly bizarre\n    is what happens during the push. It starts receiving objects from\n    the remote! By the end of this process, the local repository is a\n    whopping ~700MB -- though interestingly only about a tenth of the\n    full repository size.\n\n    This result in particular is strange in context. I would expect to\n    either see 'almost all' the repository content, 'about half' (we\n    have two trunks and fetching a single commit would at most fetch one\n    of them), or 'virtual none at all'. There isn't a straightforward\n    explanation for why 'one tenth' would make sense.\n\nWhat's different between what you expected and what actually happened?\n\n    Why should git-push ever be fetching objects? This doesn't map well\n    to my mental model of the relationship between push/fetch.\n\n    I would expect the local repository to stay in that 'git init'+200B\n    range.\n\nAnything else you want to add:\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n    I've truncated the system information normally included by\n    git-bugreport as I am sending this email from a different machine.\n\n    Versions of Git that can reproduce:\n\n      - 2.39.2.windows.1     (Windows 10)\n\n        git version:\n        git version 2.39.2.windows.1\n        cpu: x86_64\n        built from commit: a82fa99b36ddfd643e61ed45e52abe314687df67\n        sizeof-long: 4\n        sizeof-size_t: 8\n        shell-path: /bin/sh\n        feature: fsmonitor--daemon\n        uname: Windows 10.0 19044\n        compiler info: gnuc: 12.2\n        libc info: no libc information available\n        $SHELL (typically, interactive shell): C:\\Program Files\\Git\\usr\\bin\\bash.exe\n\n      - 2.31.1               (AIX UNIX 7.2)\n\n        git version:\n        git version 2.31.1\n        cpu: 00F905E64C00\n        no commit associated with this build\n        sizeof-long: 8\n        sizeof-size_t: 8\n        shell-path: /opt/freeware/bin/bash\n        uname: AIX 2 7 00FBC37A4C00\n        compiler info: gnuc: 8.3\n        libc info: no libc information available\n        $SHELL (typically, interactive shell): /usr/bin/ksh\n\n--\nSean Allred\n"},{"id":"472417","messageId":"Y/VNiuI7OZ2YiXx8@tapette.crustytoothpaste.net","threadId":"59282","inReplyTo":"m0ttze4qzl.fsf@epic96565.epic.com","subject":"Re: [BUGREPORT] Why is git-push fetching content?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-21T23:02:34Z","receivedAt":"2023-02-21T23:03:07Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-02-21 at 22:01:04, Sean Allred wrote:\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n>     # in a new directory,\n>     cd $(mktemp -d)\n> \n>     # initialize a new repository\n>     git init\n> \n>     # fetch a single commit from a remote\n>     git fetch --filter=tree:0 --depth=1 $REMOTE $COMMIT_OID\n> \n>     # create a ref on that remote\n>     git push --no-verify $REMOTE $COMMIT_OID:$REFNAME\n> \n> What did you expect to happen? (Expected behavior)\n> \n>     I expected this process to complete very, very quickly. We believe\n>     the version where it had been doing so was ~2.37.\n> \n> What happened instead? (Actual behavior)\n> \n>     The fetch completes nearly instantly as expected. We receive ~200B\n>     from the remote for the commit object itself. What's truly bizarre\n>     is what happens during the push. It starts receiving objects from\n>     the remote! By the end of this process, the local repository is a\n>     whopping ~700MB -- though interestingly only about a tenth of the\n>     full repository size.\n> \n>     This result in particular is strange in context. I would expect to\n>     either see 'almost all' the repository content, 'about half' (we\n>     have two trunks and fetching a single commit would at most fetch one\n>     of them), or 'virtual none at all'. There isn't a straightforward\n>     explanation for why 'one tenth' would make sense.\n\nIt's hard to know for certain what's going on here, but it depends on\nyour history.  You did a partial clone with no trees, so you've likely\nreceived a single commit object and no trees or blobs.\n\nHowever, when you push a commit, that necessitates pushing the trees and\nblobs as well, and you don't have those.  If the remote said that it\nalready had the commit, then it might push no objects at all (which I've\nseen before) and thus just update the references.  However, if it pushes\neven one commit, it may need to walk the history and find common\ncommits, which will necessitate fetching objects, and it will have to\npush any trees and blobs as well, which also will require objects to be\nfetched.\n\nMy guess is that this is probably made worse by the fact that this is\nshallow, and that necessitates certain additional computations, which\nmeans more objects are fetched. However, I'm not super sure how that\ncode works, so I think it may be helpful for someone else to chime in\nwho's more familiar with this.\n\nIf you want to see what's going on, you can run with\n`GIT_TRACE=1 GIT_TRACE_PACKET=1`, which may show interesting information\nabout the negotiation.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"472446","messageId":"m0pma14sbx.fsf@epic96565.epic.com","threadId":"59282","inReplyTo":"Y/VNiuI7OZ2YiXx8@tapette.crustytoothpaste.net","subject":"Re: [BUGREPORT] Why is git-push fetching content?","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-02-22T15:04:23Z","receivedAt":"2023-02-22T15:45:29Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"\n\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> It's hard to know for certain what's going on here, but it depends on\n> your history.  You did a partial clone with no trees, so you've likely\n> received a single commit object and no trees or blobs.\n\nYup, this was the intention behind `--depth=1 --filter=tree:0`. The\nserver doing this ref update needs to be faster than having the full\nhistory would allow.\n\n> However, when you push a commit, that necessitates pushing the trees and\n> blobs as well, and you don't have those.  If the remote said that it\n> already had the commit, then it might push no objects at all (which I've\n> seen before) and thus just update the references.  However, if it pushes\n> even one commit, it may need to walk the history and find common\n> commits, which will necessitate fetching objects, and it will have to\n> push any trees and blobs as well, which also will require objects to be\n> fetched.\n\nAbsolutely. The commit in question was fetched from the same remote to\nwhich we're pushing, so it would seem by definition that git-push should\nnot need to push *any* object content whatsoever.\n\n> My guess is that this is probably made worse by the fact that this is\n> shallow, and that necessitates certain additional computations, which\n> means more objects are fetched. However, I'm not super sure how that\n> code works, so I think it may be helpful for someone else to chime in\n> who's more familiar with this.\n\nI'm certain this is just an unforeseen interaction between all these\npieces. I wouldn't be too surprised if we're among only a handful of\nfolks using git in this way.\n\n> If you want to see what's going on, you can run with\n> `GIT_TRACE=1 GIT_TRACE_PACKET=1`, which may show interesting information\n> about the negotiation.\n\nI'm not sure of the best way to include this information, so I'm just\ngoing to inline it. I've edited this log file to remove several tens of\nthousands of lines of object hashes and operational refnames. I've\nannotated it with some guesses of what things might mean. I'm still\n*relatively* new to reading such log files for serious debugging.\n\n    08:30:47.655623 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin\n    08:30:47.655623 git.c:460               trace: built-in: git push --no-verify -o emc2.enable-logging git@$REPO.git FETCH_HEAD:refs/tags/hswebrec/app/stage1/latest\n\nThis is the command actually run in the foreground. As you might\nsurmise, we're uing Git4Win. It's worth noting that FETCH_HEAD here is\n0962f6cd9b1f2b5a012581823c12f0f0619bd3f5.\n\n    08:30:47.655623 run-command.c:655       trace: run_command: unset GIT_PREFIX; ssh git@$REPO_HOST 'git-receive-pack '\\''$REPO_PROJECT.git'\\'''\n    08:30:48.100544 pkt-line.c:80           packet:         push< 4edfa5e150857e21c686826e1e430f6b014ed173 refs/archive/app/devnull\\0report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta push-options object-format=sha1 agent=git/2.38.4.gl1\n    08:30:48.100544 pkt-line.c:80           packet:         push< 5d27e7331f08365c9bb3d342ae020807a386f42a refs/heads/app/10.1/stage1\n\n    ---8<--- literally thousands of refs removed...\n\n    08:30:49.121310 pkt-line.c:80           packet:         push< 90c12d8c0ad0559047b3b9de78d948901fcffac3 refs/tags/zrb/I10121236/726711\n    08:30:49.121310 pkt-line.c:80           packet:         push< 90c12d8c0ad0559047b3b9de78d948901fcffac3 refs/tags/zrb/I10121236/726712\n    08:30:49.121310 pkt-line.c:80           packet:         push< 0000\n\nLooks like the above was the remote telling the client what objects it\nhas by virtue of what refs it is tracking.\n\n    08:30:49.193113 pkt-line.c:80           packet:         push> shallow 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5\n    08:30:49.193113 pkt-line.c:80           packet:         push> 0000000000000000000000000000000000000000 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5 refs/tags/hswebrec/app/stage1/latest\\0 report-status-v2 side-band-64k quiet push-options object-format=sha1 agent=git/2.39.2.windows.1\n    08:30:49.193113 pkt-line.c:80           packet:         push> 0000\n\nAnd this is the client telling the remote what objects it has ('shallow\n0962f6...'?) and what changes it would like to make.\n\n    08:30:49.193113 pkt-line.c:80           packet:         push> emc2.enable-logging\n    08:30:49.193113 pkt-line.c:80           packet:         push> 0000\n\n...as well as our push-option that enabled more logging for us (nothing\nrelevant to Git communication -- mostly just internal web services and\nprint-statement debugging).\n\n    08:30:49.193113 run-command.c:655       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset -q --shallow\n    08:30:49.224949 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:30:49.224949 git.c:460               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset -q --shallow\n    08:30:49.224949 run-command.c:655       trace: run_command: git -c fetch.negotiationAlgorithm=noop fetch git@$REPO.git --no-tags --no-write-fetch-head --recurse-submodules=no --filter=blob:none --stdin\n    08:30:49.246673 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:30:49.256718 git.c:460               trace: built-in: git fetch git@$REPO.git --no-tags --no-write-fetch-head --recurse-submodules=no --filter=blob:none --stdin\n    08:30:49.256718 run-command.c:655       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL git@tracklab.epic.com 'git-upload-pack '\\''epic/test/trackdev/mono-23/app.git'\\'''\n\nIt looks like this is the client initiating a fetch.\n\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< version 2\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< agent=git/2.38.4.gl1\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< ls-refs=unborn\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< fetch=shallow wait-for-done filter\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< server-option\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< object-format=sha1\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< object-info\n\nThe remote tells the client what version it is so the client can send a\nrequest the remote understands.\n\n    08:30:49.699720 pkt-line.c:80           packet:        fetch< 0000\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> command=fetch\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> agent=git/2.39.2.windows.1\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> object-format=sha1\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> 0001\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> thin-pack\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> no-progress\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> ofs-delta\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> shallow 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> filter blob:none\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> want 8da2fa849db733188b1820865deb800d8e6abfc6\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> done\n    08:30:49.699720 pkt-line.c:80           packet:        fetch> 0000\n\nThe client asks the remote for content. Looks like the filter here got\nchanged from tree:0 to blob:none. This could be the bug -- and could\nexplain the 'weird' amount of content that was actually downloaded. A\nfully-fleshed-out clone would be about 8GB, but I could certainly see a\nblobless history being ~700MB. Interesting to note here that\n0962f6^{tree} is 8da2fa.\n\n    08:30:49.715316 pkt-line.c:80           packet:        fetch< shallow-info\n    08:30:49.715316 pkt-line.c:80           packet:        fetch< 0001\n    08:30:49.715316 pkt-line.c:80           packet:        fetch< packfile\n\nNot sure what this bit is, to be honest.\n\n    08:30:50.527739 pkt-line.c:80           packet:     sideband< PACK ...\n    08:30:50.542281 run-command.c:655       trace: run_command: git index-pack --stdin --fix-thin '--keep=fetch-pack 6448 on win-pool7447' --promisor --pack_header=2,71814\n    08:30:50.574053 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:30:50.590271 git.c:460               trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 6448 on win-pool7447' --promisor --pack_header=2,71814\n    08:30:53.027721 pkt-line.c:80           packet:     sideband< 0000\n    08:30:53.200395 run-command.c:655       trace: run_command: git maintenance run --auto --no-quiet\n    08:30:53.246231 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:30:53.248264 git.c:460               trace: built-in: git maintenance run --auto --no-quiet\n    08:30:59.362381 run-command.c:655       trace: run_command: git -c fetch.negotiationAlgorithm=noop fetch git@$REPO.git --no-tags --no-write-fetch-head --recurse-submodules=no --filter=blob:none --stdin\n    08:30:59.378458 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:30:59.394133 git.c:460               trace: built-in: git fetch git@$REPO.git --no-tags --no-write-fetch-head --recurse-submodules=no --filter=blob:none --stdin\n    08:33:28.966748 run-command.c:655       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL git@tracklab.epic.com 'git-upload-pack '\\''epic/test/trackdev/mono-23/app.git'\\'''\n\nNor why we'd go through a separate round of fetching.\n\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< version 2\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< agent=git/2.38.4.gl1\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< ls-refs=unborn\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< fetch=shallow wait-for-done filter\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< server-option\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< object-format=sha1\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< object-info\n    08:33:29.530124 pkt-line.c:80           packet:        fetch< 0000\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> command=fetch\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> agent=git/2.39.2.windows.1\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> object-format=sha1\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> 0001\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> thin-pack\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> no-progress\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> ofs-delta\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> shallow 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> filter blob:none\n\nBut we can see this blob:none 'mistake' again...\n\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> want 0000063ae70e4385b0527df060daf0a81b306c8d\n    08:33:51.764098 pkt-line.c:80           packet:        fetch> want 00003efd5bd3b3795950588f7d051e8d0b42def3\n\n    ---8<--- many, many thousands of objects removed\n\n    08:33:54.154726 pkt-line.c:80           packet:        fetch> want ffffd8aad00ab11c0672096203f57564b286da08\n    08:33:54.154726 pkt-line.c:80           packet:        fetch> want ffffd967eaed43bf87d40a84f2f9e12c59575abe\n    08:33:54.154726 pkt-line.c:80           packet:        fetch> done\n    08:33:54.154726 pkt-line.c:80           packet:        fetch> 0000\n\n... with all of those trees\n\n    08:33:56.058314 pkt-line.c:80           packet:        fetch< shallow-info\n    08:33:56.058314 pkt-line.c:80           packet:        fetch< 0001\n    08:33:56.058314 pkt-line.c:80           packet:        fetch< packfile\n    08:33:57.783631 pkt-line.c:80           packet:     sideband< PACK ...\n    08:33:57.799321 run-command.c:655       trace: run_command: git index-pack --stdin --fix-thin '--keep=fetch-pack 6540 on win-pool7447' --promisor --pack_header=2,344628\n    08:33:57.830596 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:33:57.830596 git.c:460               trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 6540 on win-pool7447' --promisor --pack_header=2,344628\n    08:34:35.033275 pkt-line.c:80           packet:     sideband< 0000\n    08:34:46.298777 run-command.c:655       trace: run_command: git maintenance run --auto --no-quiet\n    08:34:46.330052 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core\n    08:34:46.345653 git.c:460               trace: built-in: git maintenance run --auto --no-quiet\n    08:45:14.094158 pkt-line.c:80           packet:     sideband< \\1\n    08:45:18.281129 pkt-line.c:80           packet:     sideband< \\2pre-receive started at 1677077118283\n    remote: pre-receive started at 1677077118283\n    08:45:18.297266 pkt-line.c:80           packet:     sideband< \\2Received line '0000000000000000000000000000000000000000 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5 refs/tags/hswebrec/app/stage1/l\n    08:45:18.297266 pkt-line.c:80           packet:     sideband< \\2atest'\n    remote: Received line '0000000000000000000000000000000000000000 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5 refs/tags/hswebrec/app/stage1/latest'\n\n    ---8<--- clipped pre-receive output\n\n    08:45:18.550847 pkt-line.c:80           packet:     sideband< \\2pre-receive finished at 1677077118556 (273 ms)\n    remote: pre-receive finished at 1677077118556 (273 ms)\n    08:45:21.448647 pkt-line.c:80           packet:     sideband< \\1000eunpack ok002cok refs/tags/hswebrec/app/stage1/latest0000\n    08:45:21.448647 pkt-line.c:80           packet:         push< unpack ok\n    08:45:21.448647 pkt-line.c:80           packet:         push< ok refs/tags/hswebrec/app/stage1/latest\n    08:45:21.448647 pkt-line.c:80           packet:         push< 0000\n    08:45:21.798337 pkt-line.c:80           packet:     sideband< \\2post-receive started at 1677077121804\n    remote: post-receive started at 1677077121804\n    08:45:21.814043 pkt-line.c:80           packet:     sideband< \\2Received line '0000000000000000000000000000000000000000 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5 refs/tags/hswebrec/app/stage1/l\n    08:45:21.814043 pkt-line.c:80           packet:     sideband< \\2atest'\n    remote: Received line '0000000000000000000000000000000000000000 0962f6cd9b1f2b5a012581823c12f0f0619bd3f5 refs/tags/hswebrec/app/stage1/latest'\n\n    ---8<--- clipped post-receive output\n\n    08:45:21.972423 pkt-line.c:80           packet:     sideband< \\23.0000; path=/; Httponly; Secure\"]}}post-receive finished at 1677077121979 (175 ms)\n    remote: post-receive finished at 1677077121979 (175 ms)\n    08:45:22.051793 pkt-line.c:80           packet:     sideband< 0000\n    To $REPO.git\n     * [new tag]         FETCH_HEAD -> hswebrec/app/stage1/latest\n\n--\nSean Allred\n"},{"id":"472447","messageId":"m0lekp4rjv.fsf@epic96565.epic.com","threadId":"59282","inReplyTo":"7bfb7ecd4a4c78668f97b00d5f06af0c9b2878269476e89c3311eeb8071b1ab3@mu.id","subject":"Re: [BUGREPORT] Why is git-push fetching content?","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-02-22T15:48:19Z","receivedAt":"2023-02-22T16:02:19Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"\nApologies for the double-email; in switching between desktops, I\nprematurely sent my last message. Luckily I was very nearly done.\n\nSean Allred <allred.sean@gmail.com> writes:\n> But we can see this blob:none 'mistake' again...\n>\n>     08:33:51.764098 pkt-line.c:80           packet:        fetch> want 0000063ae70e4385b0527df060daf0a81b306c8d\n>     08:33:51.764098 pkt-line.c:80           packet:        fetch> want 00003efd5bd3b3795950588f7d051e8d0b42def3\n>\n>     ---8<--- many, many thousands of objects removed\n>\n>     08:33:54.154726 pkt-line.c:80           packet:        fetch> want ffffd8aad00ab11c0672096203f57564b286da08\n>     08:33:54.154726 pkt-line.c:80           packet:        fetch> want ffffd967eaed43bf87d40a84f2f9e12c59575abe\n>     08:33:54.154726 pkt-line.c:80           packet:        fetch> done\n>     08:33:54.154726 pkt-line.c:80           packet:        fetch> 0000\n>\n> ... with all of those trees\n\nI was verifying my suspicion in the other desktop -- but my suspicion\nwas incorrect. These aren't all trees; in fact, the objects listed above\nare *just* blobs -- no other object types. One could assume that these\nare all the blobs in the 8da2fa tree, but I would expect that we'd get\nall the subtrees in 8da2fa as well in that case.\n\nI'm not sure how much more information I can extract from this list of\nblobs, but I'm open to suggestions if we think there's a pattern here to\nbe discovered.\n\n--\nSean Allred\n"},{"id":"478538","messageId":"CAPMMpoiC8oca0AVNy1f+zy26L_b-ADyNopY4zO3r+v6v-KEH=A@mail.gmail.com","threadId":"59282","inReplyTo":"m0pma14sbx.fsf@epic96565.epic.com","subject":"Re: [BUGREPORT] Why is git-push fetching content?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2023-06-20T11:26:14Z","receivedAt":"2023-06-20T11:26:31Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Wed, Feb 22, 2023 at 4:45 PM Sean Allred <allred.sean@gmail.com> wrote:\n>\n>\n> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> > It's hard to know for certain what's going on here, but it depends on\n> > your history.  You did a partial clone with no trees, so you've likely\n> > received a single commit object and no trees or blobs.\n>\n> Yup, this was the intention behind `--depth=1 --filter=tree:0`. The\n> server doing this ref update needs to be faster than having the full\n> history would allow.\n>\n\nFWIW, you're not alone - we do exactly the same thing, for the same\nreasons, and get the same outcome: We want to create a tag in a CI\njob, that particular CI job has no reason to check out the code, all\nwe know is we want ref XXXXX to point to commit YYYYY.\n\nThe most logical way to achieve that seems to be to do a shallow\npartial no-checkout clone of commit YYYYY, and then push to remote ref\nXXXXX, but the push ends up doing extra seemingly-unnecessary\njit-fetching work.\n\nIn our case it's still better than any alternative we've found, but\nwastes a few seconds that we'd love to see optimized away.\n"},{"id":"479323","messageId":"m0zg46eueb.fsf@epic96565.epic.com","threadId":"59282","inReplyTo":"CAPMMpoiC8oca0AVNy1f+zy26L_b-ADyNopY4zO3r+v6v-KEH=A@mail.gmail.com","subject":"Re: [BUGREPORT] Why is git-push fetching content?","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-07-08T06:27:54Z","receivedAt":"2023-07-08T07:26:26Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"\nThanks for the replies. I'd like to bump this up again. This has come up\nin a new context and I don't see a viable workaround for us that doesn't\ninvolve a rewrite of the process and an excessive amount of new\ninfrastructure.\n\nI have a feeling this is somehow a general issue with promisor remotes,\nthough I don't know enough about how they work to know where to start\ninvestigation. I've got what I believe to be minimal reproduction steps\nbelow.\n\nTao Klerks <tao@klerks.biz> writes:\n> On Wed, Feb 22, 2023 at 4:45 PM Sean Allred <allred.sean@gmail.com> wrote:\n>> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n>> > It's hard to know for certain what's going on here, but it depends on\n>> > your history.  You did a partial clone with no trees, so you've likely\n>> > received a single commit object and no trees or blobs.\n>>\n>> Yup, this was the intention behind `--depth=1 --filter=tree:0`. The\n>> server doing this ref update needs to be faster than having the full\n>> history would allow.\n>>\n>\n> FWIW, you're not alone - we do exactly the same thing, for the same\n> reasons, and get the same outcome: We want to create a tag in a CI\n> job, that particular CI job has no reason to check out the code, all\n> we know is we want ref XXXXX to point to commit YYYYY.\n>\n> [...]\n>\n> In our case it's still better than any alternative we've found, but\n> wastes a few seconds that we'd love to see optimized away.\n\nUnfortunately in our case, 'a few seconds' is tens of minutes (I'm\nworking with a repository of several million commits) and is timing out\nthe remote host.\n\n----\n\nI devised some minimal steps to reproduce what I believe to be a related\nissue: rev-list fetching content. I've prepared a public repository on\ngithub.com to demonstrate, but you should be able to recreate this\nrepository if needed by just making a handful of commits to a couple\narbitrary files.\n\n    (cwd:tmp)\n    $ git clone --no-checkout --depth=1 --no-tags --filter=tree:0 https://github.com/vermiculus/testibus.git\n    Cloning into 'testibus'...\n    remote: Enumerating objects: 1, done.\n    remote: Counting objects: 100% (1/1), done.\n    remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0\n    Receiving objects: 100% (1/1), done.\n\nSweet, I've only received one object from the remote. This makes sense\nper what I want: a treeless, blobless, fetch of a single commit. Let's\ndouble-check.\n\n    (cwd:testibus)\n    $ git fsck\n    Checking object directories: 100% (256/256), done.\n    Checking objects: 100% (2/2), done.\n\nI have two objects? How'd that second one get in there? What is it?\nLet's try to find out...\n\n    (cwd:testibus)\n    $ git rev-list --objects --all\n    d86642e7ae089b69e8a0b20a3e39337435833f92\n\nAlright, I've got the commit object. That makes sense.\n\n    c0fa909c5f67047abc027d9b06e1352954ee33f7\n\nWeird, I also got the tree on the commit, even though I specified that\nthis should be a treeless clone.\n\n    remote: Enumerating objects: 1, done.\n    remote: Counting objects: 100% (1/1), done.\n    remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0\n    Receiving objects: 100% (1/1), 54 bytes | 54.00 KiB/s, done.\n    94b334d80405218e281a6f5b48d31f73cd3af4be file\n\nWoah woah! All I did was rev-list; why are we fetching content?\n\nThis is why I believe this is related to the push issue I'm ultimately\nfacing -- I'm not familiar with the specifics, but it stands to reason\nthat git-push needs to (somehow) iterate through objects in order to\nnegotiate a packfile with the remote. I suspect these two issues have\nthe same root cause.\n\nI believe the following can be used with git-bisect to determine if this\ntruly ever worked or is a regression:\n\n    setup:\n        #!/bin/bash\n\n        repo=\"https://github.com/vermiculus/testibus.git\"\n        repo_dir=\"~/path/to/repo\"\n\n        git clone --no-checkout --depth=1 --no-tags --filter=tree:0 \"$repo\" \"$repo_dir\"\n        git -C \"$repo_dir\" remote set-url origin unreachable\n\n    bisect script:\n        git -C \"$repo_dir\" rev-list --objects --all\n\n        (obviously using the just-built git)\n\nI'm going to start running this bisect, but I suspect it will take a\nwhile, so I wanted to get this out there.\n\n--\nSean Allred\n"},{"id":"479324","messageId":"m0pm52eqg5.fsf@epic96565.epic.com","threadId":"59282","inReplyTo":"m0zg46eueb.fsf@epic96565.epic.com","subject":"Re: [BUGREPORT] Why is git-push fetching content?","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-07-08T08:39:19Z","receivedAt":"2023-07-08T08:51:53Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"\nFollowing up with the results of my bisect (more discussion below). I'm\nforced to conclude this may somehow have never worked as I'm expecting\n(even though I do recall it working well in a long-gone environment),\nbut I'm very much hoping I just did the bisect incorrectly. (It's not a\nfeature I need to use much.)\n\nSo, is this a bug or is this working as intended for a good reason?\n\nSean Allred <allred.sean@gmail.com> writes:\n> Thanks for the replies. I'd like to bump this up again. This has come up\n> in a new context and I don't see a viable workaround for us that doesn't\n> involve a rewrite of the process and an excessive amount of new\n> infrastructure.\n>\n> I have a feeling this is somehow a general issue with promisor remotes,\n> though I don't know enough about how they work to know where to start\n> investigation. I've got what I believe to be minimal reproduction steps\n> below.\n>\n> [...]\n>\n> I believe the following can be used with git-bisect to determine if this\n> truly ever worked or is a regression:\n>\n>     setup:\n>         #!/bin/bash\n>\n>         repo=\"https://github.com/vermiculus/testibus.git\"\n>         repo_dir=\"~/path/to/repo\"\n>\n>         git clone --no-checkout --depth=1 --no-tags --filter=tree:0 \"$repo\" \"$repo_dir\"\n>         git -C \"$repo_dir\" remote set-url origin unreachable\n>\n>     bisect script:\n>         git -C \"$repo_dir\" rev-list --objects --all\n>\n>         (obviously using the just-built git)\n>\n> I'm going to start running this bisect, but I suspect it will take a\n> while, so I wanted to get this out there.\n\nI ended up using a bisect script that looks like this\n\n    #!/bin/bash\n    make clean\n    NO_GETTEXT=1 make -j8 || exit 125\n    ./bin-wrappers/git -C \"$1\" rev-list --objects --all || exit 1\n    git rev-parse HEAD >> ../good-commits\n\nand running\n\n    git bisect start main 637fc4467e57872008171958eda0428818a7ee03\n    git bisect run ../bisect-script.sh ~/tmp/testibus/\n\nIt took less time than I thought, but unfortunately I was never able to\nactually find a 'good' commit. I arbitrarily chose \"partial-clone:\ndesign doc\" (Jeff Hostetler, Dec 14 2017) as the first commit to the\npartial-clone design document (under the assumption that it worked at\nsome point). If potentially lying to git-bisect in this way is\nespecially liable to bust it, I can start the exponentially-more-\nexpensive process of testing every commit along --first-parent, but I\nsuspect this may have never worked as I'm expecting.\n\n--\nSean Allred\n"}]}