git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: 'git sparse-checkout add' hangs if local repo is using alternative objects store

From
DXDian Xu <dianxudev@gmail.com>
Date
Jul 18, 2022, 19:29 UTC
Message-ID
<CAKSRnEyrc1ZzeShYNOHumMjnkB0MWHaZTRd6n7SrRACT=eAiYA@mail.gmail.com>
In-Reply-To
<89310a11-b8cd-b643-5d46-c6d273f66b49@github.com>
Hi Derrick,
Thank you for your reply. Please see my reply inline:

On Thu, Jul 14, 2022 at 11:06 PM Derrick Stolee <derrickstolee@github.com> wrote:

Show 11 quoted lines
>
> On 7/12/2022 9:58 AM, Dian Xu wrote:
>
> > Reproduction steps:
> >     1. Local repo setup:
> >         git init
> >         git remote add origin <repo_url>
> >         echo <alternative_objects_store> >! .git/objects/info/alternates
>
> Is this alternative object store on the local disk? Or is it
> across a network connection?
The alternate object store I used is across network connection.
Show 19 quoted lines
>
> >         git config --unset-all remote.origin.fetch
> >         git config --unset-all remote.origin.tagopt
> >         git config --add remote.origin.tagopt --no-tags
> >         git config --add remote.origin.fetch
> > +refs/heads/main:refs/remotes/origin/main
> >         git config core.sparsecheckout true
> >         git config core.sparsecheckoutcone true
> >         git fetch
>
> This is interesting that you are fetching without having
> any existing refs, only the alternate store. Are you using
> something like the core.alternateRefsCommand config option?
>
> Otherwise, I would expect this "git fetch" to act as if it
> was a normal clone (unless of course the alternate already
> has the current tip of 'main'). Maybe fetch negotiation helps
> find some common commits that exist in the alternate, but the
> client is useless in that discovery without refs.
I am not sure I understand your statement fully here:
    - All these config setup here is to let 'git fetch' only fetches
e.g. 'refs/heads/main', nothing else
    - The HEAD of the AOS (alternate object store) is indeed at refs/heads/main
    - I can remove all these 4 specific refs config settings (now
fetching all refs instead), and the issue can still be reproduced
Show 12 quoted lines
>
> >     2. Sparse checkout <dir1>:
> >         git sparse-checkout set <dir1>
> >         git checkout -b tmp1 origin/main
> >
> >     3. Add another <dir2> via sparse-checkout, which hangs:
> >         git sparse-checkout add <dir2>
> >             09:39:11.169116 read-cache.c:2451       performance:
> > 6.326754679 s:  read cache .git/index
>
> Could you re-run this with GIT_TRACE2_PERF=1 so we can see
> even more details about how this works?
Please see the output with GIT_TRACE2_PERF at the end of the email,
which hung at:
    15:17:28.216829 cache-tree.c:477             | d0 | main
          | region_enter | r1  | 85.183554 |           | cache_tree
| ..label:update
Show 13 quoted lines
>
> > Repo and dir info:
> >     <repo_url> is a large repo, containing large number of directories
> > and 2 million files;
> >     Local repo is using <alternative_objects_store>, which is working
> > by verifying: git count-objects -vH;
> >     <dir1> used here is one level deep, <dir2> used here is 5 level deep
> >
> > Observations:
> >     1. Even 'sparse-checkout add' hangs, scm code from <dir2> was
> > actually downloaded pretty fast
>
> I'm interested to learn what you mean by "downloaded" here.

After the 'git sparse-checkout add' hung, I ctr-C then cd to inspect the source code, they are on disk

Show 9 quoted lines
>
> >     2. If waited out 'sparse-checkout add', git seems start to
> > download large number of objects from alternative objects store into
> > local .git/objects
>
> Hm. Adding the objects into .git/objects is strange. It should
> recognize that the objects are in the alternate and not write
> them to the local repo (this is independent of the sparse-checkout
> builtin).

Yep, seems like git is downloading objects from AOS to local, which is a main concern. We choose to use AOS to save space/time in the first place.

>
> Thanks,
> -Stolee
Please let me know if you need more details.
Thanks,

Dian Xu Mathworks, Inc 1 Lakeside Campus Drive, Natick, MA 01760 508-647-3583

----------GIT_TRACE2_PERF----------
% git sparse-checkout add matlab/config
15:16:03.038943 common-main.c:49             | d0 | main
      | version      |     |           |           |              |
2.37.0
15:16:03.038997 common-main.c:50             | d0 | main
      | start        |     |  0.005727 |           |              |
git sparse-checkout add matlab/config
15:16:03.039144 compat/linux/procinfo.c:170  | d0 | main
      | cmd_ancestry |     |           |           |              |
ancestry:[tcsh mate-terminal mate-panel mate-session lightdm lightdm
systemd]
15:16:03.044327 repository.c:136             | d0 | main
      | def_repo     | r1  |           |           |              |
worktree:/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone
15:16:03.045148 git.c:461                    | d0 | main
      | cmd_name     |     |           |           |              |
sparse-checkout (sparse-checkout)
15:16:03.045589 read-cache.c:2447            | d0 | main
      | region_enter | r1  |  0.012321 |           | index        |
label:do_read_index .git/index
15:16:07.665184 cache-tree.c:628             | d0 | main
      | region_enter | r1  |  4.631908 |           | cache_tree   |
..label:read
15:16:07.821302 cache-tree.c:630             | d0 | main
      | region_leave | r1  |  4.788028 |  0.156120 | cache_tree   |
..label:read
15:16:07.834318 read-cache.c:2393            | d0 | main
      | data         | r1  |  4.801047 |  4.788726 | index        |
..read/version:3
15:16:07.834364 read-cache.c:2395            | d0 | main
      | data         | r1  |  4.801097 |  4.788776 | index        |
..read/cache_nr:2314721
15:16:07.834375 read-cache.c:2451       performance: 4.788756364 s:
read cache .git/index
15:16:07.834379 read-cache.c:2452            | d0 | main
      | region_leave | r1  |  4.801112 |  4.788791 | index        |
label:do_read_index .git/index
15:16:07.950067 progress.c:268               | d0 | main
      | region_enter | r1  |  4.916794 |           | progress     |
label:Updating index flags
15:16:07.950130 unpack-trees.c:1671          | d0 | main
      | region_enter | r1  |  4.916863 |           | unpack_trees |
..label:clear_ce_flags(0x00000000,0x02000000)
15:16:08.083835 unpack-trees.c:1678          | d0 | main
      | region_leave | r1  |  5.050562 |  0.133699 | unpack_trees |
..label:clear_ce_flags(0x00000000,0x02000000)
15:16:08.083902 progress.c:339               | d0 | main
      | data         | r1  |  5.050634 |  0.133840 | progress     |
..total_objects:2314721
15:16:08.083910 progress.c:346               | d0 | main
      | region_leave | r1  |  5.050643 |  0.133849 | progress     |
label:Updating index flags
15:16:08.161023 progress.c:268               | d0 | main
      | region_enter | r1  |  5.127750 |           | progress     |
label:Updating files
Updating files: 100% (21877/21877), done.
15:17:11.071782 progress.c:339               | d0 | main
      | data         | r1  | 68.038510 | 62.910760 | progress     |
..total_objects:21877
15:17:11.071825 progress.c:346               | d0 | main
      | region_leave | r1  | 68.038557 | 62.910807 | progress     |
label:Updating files
15:17:11.071859 progress.c:268               | d0 | main
      | region_enter | r1  | 68.038591 |           | progress     |
label:Filtering content
15:17:11.071911 progress.c:339               | d0 | main
      | data         | r1  | 68.038643 |  0.000052 | progress     |
..total_objects:0
15:17:11.071930 progress.c:346               | d0 | main
      | region_leave | r1  | 68.038662 |  0.000071 | progress     |
label:Filtering content
15:17:11.071960 unpack-trees.c:496      performance: 62.934445487 s:
check_updates
15:17:11.071974 unpack-trees.c:2062     performance: 63.187061820 s:
update_sparsity
15:17:11.071998 read-cache.c:3179            | d0 | main
      | region_enter | r1  | 68.038728 |           | index        |
label:do_write_index
/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone/.git/index.lock
15:17:12.387925 cache-tree.c:532             | d0 | main
      | region_enter | r1  | 69.354652 |           | cache_tree   |
..label:write
15:17:12.426139 cache-tree.c:534             | d0 | main
      | region_leave | r1  | 69.392868 |  0.038216 | cache_tree   |
..label:write
15:17:16.696737 read-cache.c:3135       performance: 5.624723676 s:
write index, changed mask = 2
15:17:16.696828 read-cache.c:3141            | d0 | main
      | data         | r1  | 73.663556 |  5.624828 | index        |
..write/version:3
15:17:16.696897 read-cache.c:3143            | d0 | main
      | data         | r1  | 73.663629 |  5.624901 | index        |
..write/cache_nr:2314721
15:17:16.696953 read-cache.c:3182            | d0 | main
      | region_leave | r1  | 73.663682 |  5.624954 | index        |
label:do_write_index
/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone/.git/index.lock
15:17:17.089520 sparse-index.c:209           | d0 | main
      | region_enter | r1  | 74.056246 |           | index        |
label:convert_to_sparse
15:17:17.146799 cache-tree.c:477             | d0 | main
      | region_enter | r1  | 74.113525 |           | cache_tree   |
..label:update
15:17:24.864900 cache-tree.c:482             | d0 | main
      | region_leave | r1  | 81.831626 |  7.718101 | cache_tree   |
..label:update
15:17:24.864975 cache-tree.c:483        performance: 7.718183242 s:
cache_tree_update
15:17:24.864989 sparse-index.c:223           | d0 | main
      | region_leave | r1  | 81.831721 |  7.775475 | index        |
label:convert_to_sparse
15:17:24.911366 cache-tree.c:477             | d0 | main
      | region_enter | r1  | 81.878090 |           | cache_tree   |
label:update
15:17:25.890481 cache-tree.c:482             | d0 | main
      | region_leave | r1  | 82.857209 |  0.979119 | cache_tree   |
label:update
15:17:25.890528 cache-tree.c:483        performance: 0.979171895 s:
cache_tree_update
15:17:25.890536 sparse-index.c:340           | d0 | main
      | region_enter | r1  | 82.857269 |           | index        |
label:ensure_full_index
15:17:28.216829 cache-tree.c:477             | d0 | main
      | region_enter | r1  | 85.183554 |           | cache_tree   |
..label:update
Previous: Derrick Stolee
Message 3 of 3 in “'git sparse-checkout add' hangs if local repo is using alternative objects store”
  1. Dian XuJul 12, 2022
  2. Derrick StoleeJul 15, 2022
  3. Dian XuJul 18, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.