{"thread":{"id":"58141","subject":"'git sparse-checkout add' hangs if local repo is using alternative objects store","startedAt":"2022-07-12T14:00:07Z","lastAt":"2022-07-18T19:31:36Z","messageCount":3,"participants":["Dian Xu","Derrick Stolee"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"458887","messageId":"CAKSRnEzYYQ1EOmbUvafQvzJUNW8kojLuD+BbakxU09JSm=niHA@mail.gmail.com","threadId":"58141","inReplyTo":null,"subject":"'git sparse-checkout add' hangs if local repo is using alternative objects store","fromName":"Dian Xu","fromEmail":"dianxudev@gmail.com","sentAt":"2022-07-12T13:58:51Z","receivedAt":"2022-07-12T14:00:07Z","isPatch":false,"sender":{"key":"dianxudev@gmail.com","avatar":null},"body":"Dear Git developers,\n\nReporting Issue:\n    'git sparse-checkout add' hangs if local repo is using alternative\nobjects store\n\nFound in:\n    Git 2.37.0\n\nReproduction steps:\n    1. Local repo setup:\n        git init\n        git remote add origin <repo_url>\n        echo <alternative_objects_store> >! .git/objects/info/alternates\n        git config --unset-all remote.origin.fetch\n        git config --unset-all remote.origin.tagopt\n        git config --add remote.origin.tagopt --no-tags\n        git config --add remote.origin.fetch\n+refs/heads/main:refs/remotes/origin/main\n        git config core.sparsecheckout true\n        git config core.sparsecheckoutcone true\n        git fetch\n\n    2. Sparse checkout <dir1>:\n        git sparse-checkout set <dir1>\n        git checkout -b tmp1 origin/main\n\n    3. Add another <dir2> via sparse-checkout, which hangs:\n        git sparse-checkout add <dir2>\n            09:39:11.169116 read-cache.c:2451       performance:\n6.326754679 s:  read cache .git/index\n            09:39:11.731989 unpack-trees.c:496      performance:\n0.197235658 s:   check_updates\n            09:39:11.732031 unpack-trees.c:2062     performance:\n0.512473791 s:  update_sparsity\n            09:39:17.162018 read-cache.c:3135       performance:\n5.429975480 s:  write index, changed mask = 2\n            09:39:17.637510 cache-tree.c:483        performance:\n0.154317183 s:  cache_tree_update\n            09:39:17.857204 cache-tree.c:483        performance:\n0.019983088 s:  cache_tree_update\n\nRepo and dir info:\n    <repo_url> is a large repo, containing large number of directories\nand 2 million files;\n    Local repo is using <alternative_objects_store>, which is working\nby verifying: git count-objects -vH;\n    <dir1> used here is one level deep, <dir2> used here is 5 level deep\n\nObservations:\n    1. Even 'sparse-checkout add' hangs, scm code from <dir2> was\nactually downloaded pretty fast\n    2. If waited out 'sparse-checkout add', git seems start to\ndownload large number of objects from alternative objects store into\nlocal .git/objects\n\nPlease help us take a look at this issue and let us know if you need\nmore information.\n\nThanks,\n\nDian Xu\nMathworks, Inc\n1 Lakeside Campus Drive, Natick, MA 01760\n508-647-3583\n"},{"id":"459117","messageId":"89310a11-b8cd-b643-5d46-c6d273f66b49@github.com","threadId":"58141","inReplyTo":"CAKSRnEzYYQ1EOmbUvafQvzJUNW8kojLuD+BbakxU09JSm=niHA@mail.gmail.com","subject":"Re: 'git sparse-checkout add' hangs if local repo is using alternative objects store","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-15T03:06:41Z","receivedAt":"2022-07-15T03:07:02Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/12/2022 9:58 AM, Dian Xu wrote:\n\n> Reproduction steps:\n>     1. Local repo setup:\n>         git init\n>         git remote add origin <repo_url>\n>         echo <alternative_objects_store> >! .git/objects/info/alternates\n\nIs this alternative object store on the local disk? Or is it\nacross a network connection?\n\n>         git config --unset-all remote.origin.fetch\n>         git config --unset-all remote.origin.tagopt\n>         git config --add remote.origin.tagopt --no-tags\n>         git config --add remote.origin.fetch\n> +refs/heads/main:refs/remotes/origin/main\n>         git config core.sparsecheckout true\n>         git config core.sparsecheckoutcone true\n>         git fetch\n\nThis is interesting that you are fetching without having\nany existing refs, only the alternate store. Are you using\nsomething like the core.alternateRefsCommand config option?\n\nOtherwise, I would expect this \"git fetch\" to act as if it\nwas a normal clone (unless of course the alternate already\nhas the current tip of 'main'). Maybe fetch negotiation helps\nfind some common commits that exist in the alternate, but the\nclient is useless in that discovery without refs.\n\n>     2. Sparse checkout <dir1>:\n>         git sparse-checkout set <dir1>\n>         git checkout -b tmp1 origin/main\n> \n>     3. Add another <dir2> via sparse-checkout, which hangs:\n>         git sparse-checkout add <dir2>\n>             09:39:11.169116 read-cache.c:2451       performance:\n> 6.326754679 s:  read cache .git/index\n\nCould you re-run this with GIT_TRACE2_PERF=1 so we can see\neven more details about how this works?\n\n> Repo and dir info:\n>     <repo_url> is a large repo, containing large number of directories\n> and 2 million files;\n>     Local repo is using <alternative_objects_store>, which is working\n> by verifying: git count-objects -vH;\n>     <dir1> used here is one level deep, <dir2> used here is 5 level deep\n> \n> Observations:\n>     1. Even 'sparse-checkout add' hangs, scm code from <dir2> was\n> actually downloaded pretty fast\n\nI'm interested to learn what you mean by \"downloaded\" here.\n\n>     2. If waited out 'sparse-checkout add', git seems start to\n> download large number of objects from alternative objects store into\n> local .git/objects\n\nHm. Adding the objects into .git/objects is strange. It should\nrecognize that the objects are in the alternate and not write\nthem to the local repo (this is independent of the sparse-checkout\nbuiltin).\n\nThanks,\n-Stolee\n"},{"id":"459316","messageId":"CAKSRnEyrc1ZzeShYNOHumMjnkB0MWHaZTRd6n7SrRACT=eAiYA@mail.gmail.com","threadId":"58141","inReplyTo":"89310a11-b8cd-b643-5d46-c6d273f66b49@github.com","subject":"Re: 'git sparse-checkout add' hangs if local repo is using alternative objects store","fromName":"Dian Xu","fromEmail":"dianxudev@gmail.com","sentAt":"2022-07-18T19:29:32Z","receivedAt":"2022-07-18T19:31:36Z","isPatch":false,"sender":{"key":"dianxudev@gmail.com","avatar":null},"body":"Hi Derrick,\n\nThank you for your reply. Please see my reply inline:\n\nOn Thu, Jul 14, 2022 at 11:06 PM Derrick Stolee\n<derrickstolee@github.com> wrote:\n>\n> On 7/12/2022 9:58 AM, Dian Xu wrote:\n>\n> > Reproduction steps:\n> >     1. Local repo setup:\n> >         git init\n> >         git remote add origin <repo_url>\n> >         echo <alternative_objects_store> >! .git/objects/info/alternates\n>\n> Is this alternative object store on the local disk? Or is it\n> across a network connection?\n\n\nThe alternate object store I used is across network connection.\n\n\n>\n> >         git config --unset-all remote.origin.fetch\n> >         git config --unset-all remote.origin.tagopt\n> >         git config --add remote.origin.tagopt --no-tags\n> >         git config --add remote.origin.fetch\n> > +refs/heads/main:refs/remotes/origin/main\n> >         git config core.sparsecheckout true\n> >         git config core.sparsecheckoutcone true\n> >         git fetch\n>\n> This is interesting that you are fetching without having\n> any existing refs, only the alternate store. Are you using\n> something like the core.alternateRefsCommand config option?\n>\n> Otherwise, I would expect this \"git fetch\" to act as if it\n> was a normal clone (unless of course the alternate already\n> has the current tip of 'main'). Maybe fetch negotiation helps\n> find some common commits that exist in the alternate, but the\n> client is useless in that discovery without refs.\n\n\nI am not sure I understand your statement fully here:\n    - All these config setup here is to let 'git fetch' only fetches\ne.g. 'refs/heads/main', nothing else\n    - The HEAD of the AOS (alternate object store) is indeed at refs/heads/main\n    - I can remove all these 4 specific refs config settings (now\nfetching all refs instead), and the issue can still be reproduced\n\n\n>\n> >     2. Sparse checkout <dir1>:\n> >         git sparse-checkout set <dir1>\n> >         git checkout -b tmp1 origin/main\n> >\n> >     3. Add another <dir2> via sparse-checkout, which hangs:\n> >         git sparse-checkout add <dir2>\n> >             09:39:11.169116 read-cache.c:2451       performance:\n> > 6.326754679 s:  read cache .git/index\n>\n> Could you re-run this with GIT_TRACE2_PERF=1 so we can see\n> even more details about how this works?\n\n\nPlease see the output with GIT_TRACE2_PERF at the end of the email,\nwhich hung at:\n    15:17:28.216829 cache-tree.c:477             | d0 | main\n          | region_enter | r1  | 85.183554 |           | cache_tree\n| ..label:update\n\n\n>\n> > Repo and dir info:\n> >     <repo_url> is a large repo, containing large number of directories\n> > and 2 million files;\n> >     Local repo is using <alternative_objects_store>, which is working\n> > by verifying: git count-objects -vH;\n> >     <dir1> used here is one level deep, <dir2> used here is 5 level deep\n> >\n> > Observations:\n> >     1. Even 'sparse-checkout add' hangs, scm code from <dir2> was\n> > actually downloaded pretty fast\n>\n> I'm interested to learn what you mean by \"downloaded\" here.\n\n\nAfter the 'git sparse-checkout add' hung, I ctr-C then cd to inspect\nthe source code, they are on disk\n\n\n>\n> >     2. If waited out 'sparse-checkout add', git seems start to\n> > download large number of objects from alternative objects store into\n> > local .git/objects\n>\n> Hm. Adding the objects into .git/objects is strange. It should\n> recognize that the objects are in the alternate and not write\n> them to the local repo (this is independent of the sparse-checkout\n> builtin).\n\n\nYep, seems like git is downloading objects from AOS to local, which is\na main concern. We choose to use AOS to save space/time in the first\nplace.\n\n\n>\n> Thanks,\n> -Stolee\n\nPlease let me know if you need more details.\n\nThanks,\n\nDian Xu\nMathworks, Inc\n1 Lakeside Campus Drive, Natick, MA 01760\n508-647-3583\n\n----------GIT_TRACE2_PERF----------\n\n% git sparse-checkout add matlab/config\n15:16:03.038943 common-main.c:49             | d0 | main\n      | version      |     |           |           |              |\n2.37.0\n15:16:03.038997 common-main.c:50             | d0 | main\n      | start        |     |  0.005727 |           |              |\ngit sparse-checkout add matlab/config\n15:16:03.039144 compat/linux/procinfo.c:170  | d0 | main\n      | cmd_ancestry |     |           |           |              |\nancestry:[tcsh mate-terminal mate-panel mate-session lightdm lightdm\nsystemd]\n15:16:03.044327 repository.c:136             | d0 | main\n      | def_repo     | r1  |           |           |              |\nworktree:/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone\n15:16:03.045148 git.c:461                    | d0 | main\n      | cmd_name     |     |           |           |              |\nsparse-checkout (sparse-checkout)\n15:16:03.045589 read-cache.c:2447            | d0 | main\n      | region_enter | r1  |  0.012321 |           | index        |\nlabel:do_read_index .git/index\n15:16:07.665184 cache-tree.c:628             | d0 | main\n      | region_enter | r1  |  4.631908 |           | cache_tree   |\n..label:read\n15:16:07.821302 cache-tree.c:630             | d0 | main\n      | region_leave | r1  |  4.788028 |  0.156120 | cache_tree   |\n..label:read\n15:16:07.834318 read-cache.c:2393            | d0 | main\n      | data         | r1  |  4.801047 |  4.788726 | index        |\n..read/version:3\n15:16:07.834364 read-cache.c:2395            | d0 | main\n      | data         | r1  |  4.801097 |  4.788776 | index        |\n..read/cache_nr:2314721\n15:16:07.834375 read-cache.c:2451       performance: 4.788756364 s:\nread cache .git/index\n15:16:07.834379 read-cache.c:2452            | d0 | main\n      | region_leave | r1  |  4.801112 |  4.788791 | index        |\nlabel:do_read_index .git/index\n15:16:07.950067 progress.c:268               | d0 | main\n      | region_enter | r1  |  4.916794 |           | progress     |\nlabel:Updating index flags\n15:16:07.950130 unpack-trees.c:1671          | d0 | main\n      | region_enter | r1  |  4.916863 |           | unpack_trees |\n..label:clear_ce_flags(0x00000000,0x02000000)\n15:16:08.083835 unpack-trees.c:1678          | d0 | main\n      | region_leave | r1  |  5.050562 |  0.133699 | unpack_trees |\n..label:clear_ce_flags(0x00000000,0x02000000)\n15:16:08.083902 progress.c:339               | d0 | main\n      | data         | r1  |  5.050634 |  0.133840 | progress     |\n..total_objects:2314721\n15:16:08.083910 progress.c:346               | d0 | main\n      | region_leave | r1  |  5.050643 |  0.133849 | progress     |\nlabel:Updating index flags\n15:16:08.161023 progress.c:268               | d0 | main\n      | region_enter | r1  |  5.127750 |           | progress     |\nlabel:Updating files\nUpdating files: 100% (21877/21877), done.\n15:17:11.071782 progress.c:339               | d0 | main\n      | data         | r1  | 68.038510 | 62.910760 | progress     |\n..total_objects:21877\n15:17:11.071825 progress.c:346               | d0 | main\n      | region_leave | r1  | 68.038557 | 62.910807 | progress     |\nlabel:Updating files\n15:17:11.071859 progress.c:268               | d0 | main\n      | region_enter | r1  | 68.038591 |           | progress     |\nlabel:Filtering content\n15:17:11.071911 progress.c:339               | d0 | main\n      | data         | r1  | 68.038643 |  0.000052 | progress     |\n..total_objects:0\n15:17:11.071930 progress.c:346               | d0 | main\n      | region_leave | r1  | 68.038662 |  0.000071 | progress     |\nlabel:Filtering content\n15:17:11.071960 unpack-trees.c:496      performance: 62.934445487 s:\ncheck_updates\n15:17:11.071974 unpack-trees.c:2062     performance: 63.187061820 s:\nupdate_sparsity\n15:17:11.071998 read-cache.c:3179            | d0 | main\n      | region_enter | r1  | 68.038728 |           | index        |\nlabel:do_write_index\n/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone/.git/index.lock\n15:17:12.387925 cache-tree.c:532             | d0 | main\n      | region_enter | r1  | 69.354652 |           | cache_tree   |\n..label:write\n15:17:12.426139 cache-tree.c:534             | d0 | main\n      | region_leave | r1  | 69.392868 |  0.038216 | cache_tree   |\n..label:write\n15:17:16.696737 read-cache.c:3135       performance: 5.624723676 s:\nwrite index, changed mask = 2\n15:17:16.696828 read-cache.c:3141            | d0 | main\n      | data         | r1  | 73.663556 |  5.624828 | index        |\n..write/version:3\n15:17:16.696897 read-cache.c:3143            | d0 | main\n      | data         | r1  | 73.663629 |  5.624901 | index        |\n..write/cache_nr:2314721\n15:17:16.696953 read-cache.c:3182            | d0 | main\n      | region_leave | r1  | 73.663682 |  5.624954 | index        |\nlabel:do_write_index\n/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone/.git/index.lock\n15:17:17.089520 sparse-index.c:209           | d0 | main\n      | region_enter | r1  | 74.056246 |           | index        |\nlabel:convert_to_sparse\n15:17:17.146799 cache-tree.c:477             | d0 | main\n      | region_enter | r1  | 74.113525 |           | cache_tree   |\n..label:update\n15:17:24.864900 cache-tree.c:482             | d0 | main\n      | region_leave | r1  | 81.831626 |  7.718101 | cache_tree   |\n..label:update\n15:17:24.864975 cache-tree.c:483        performance: 7.718183242 s:\ncache_tree_update\n15:17:24.864989 sparse-index.c:223           | d0 | main\n      | region_leave | r1  | 81.831721 |  7.775475 | index        |\nlabel:convert_to_sparse\n15:17:24.911366 cache-tree.c:477             | d0 | main\n      | region_enter | r1  | 81.878090 |           | cache_tree   |\nlabel:update\n15:17:25.890481 cache-tree.c:482             | d0 | main\n      | region_leave | r1  | 82.857209 |  0.979119 | cache_tree   |\nlabel:update\n15:17:25.890528 cache-tree.c:483        performance: 0.979171895 s:\ncache_tree_update\n15:17:25.890536 sparse-index.c:340           | d0 | main\n      | region_enter | r1  | 82.857269 |           | index        |\nlabel:ensure_full_index\n15:17:28.216829 cache-tree.c:477             | d0 | main\n      | region_enter | r1  | 85.183554 |           | cache_tree   |\n..label:update\n"}]}