threads / discuss / 57686

Git 2.36, ls-tree submodule regression?

Subject: Git 2.36, ls-tree submodule regression?

## tl;dr

6 messages between Apr 7, 2022 and Apr 7, 2022.

replies: 5people: 4as markdown or json

Bryan Turner· Apr 7, 2022, 00:34 UTC · lore

In the test suite for Bitbucket Server, we have a test repository[1] which has a commit that has a .gitmodules file that lists 3 submodules, each of which has an entry in the tree with "Subproject commit <SHA>".

The submodules aren't fetched (at least one of them doesn't actually exist anywhere to fetch), but running git ls-tree in Git 2.35.1 and prior shows the expected tree data:

$ git ls-tree -r -l ecf42d8adc4d7eb4386e37882d053704067712d4: -- 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189 346 .gitmodules 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39 20 add_file/add_file.txt 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949 25 basic_branching/file.txt 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0 167 binary/B.zip 100644 blob 72555d02a139700be930a6734b57c5fab84512b9 167 binary/C.zip 100644 blob 72555d02a139700be930a6734b57c5fab84512b9 167 binary/D.zip 100644 blob 668076c308990933c2df66efb87b946704252486 46 branch_mod_merge/file.txt 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b 33 modification/mod_file.txt 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2 18 moved_dir/file.txt 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963 24 mv_file/moved_file.txt 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a 66 out_of_order/file.txt 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87 - submodule/atlassian-refapp 160000 commit 92b18c97d9d92d2bd02cfd1d268f22fddddafb67 - submodule/bamboo-git-plugin 160000 commit 996b4f5a7e242a29fe573582d98f5174142bb97c - submodule/stash 120000 blob 4cbb553f3f4ac2ee7b01ff6c951d6bf583c39c15 10 symlink/link.txt 100644 blob 57a3592bb2cf8e76e2faef7660945a2fe0e24360 27 symlink/target.txt

The same ls-tree command in 2.36.0-rc0 fails:

$ /opt/git/2.36.0-rc0/bin/git ls-tree -r -l ecf42d8adc4d7eb4386e37882d053704067712d4: -- 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189 346 .gitmodules 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39 20 add_file/add_file.txt 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949 25 basic_branching/file.txt 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0 167 binary/B.zip 100644 blob 72555d02a139700be930a6734b57c5fab84512b9 167 binary/C.zip 100644 blob 72555d02a139700be930a6734b57c5fab84512b9 167 binary/D.zip 100644 blob 668076c308990933c2df66efb87b946704252486 46 branch_mod_merge/file.txt 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b 33 modification/mod_file.txt 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2 18 moved_dir/file.txt 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963 24 mv_file/moved_file.txt 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a 66 out_of_order/file.txt 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87 - submodule/atlassian-refapp error: Could not read 0ac826865ce7e8bc76022d49424f52e09b16dc87 fatal: Invalid commit 0ac826865ce7e8bc76022d49424f52e09b16dc87 in submodule path submodule/atlassian-refapp

Bisecting the error selects:
$ git bisect good
9c4d58ff2c385f49585197c8650356955e1fa02e is the first bad commit
commit 9c4d58ff2c385f49585197c8650356955e1fa02e
Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date:   Wed Mar 23 17:13:15 2022 +0800
    ls-tree: split up "fast path" callbacks
    Make the various if/else in the callbacks for the "fast path" a lot
    easier to read by just using common functions for the parts that are
    common, and have per-format callbacks for those parts that are
    different.
    Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
    Signed-off-by: Teng Long <dyroneteng@gmail.com>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>
 builtin/ls-tree.c | 199 ++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 125 insertions(+), 74 deletions(-)
(I've CC'd Ævar and Teng on this message.)

Looking at the changes in that commit I see differences in the handling for long listings, so I tried removing the -l. At that point, the command works again:

$ /opt/git/2.36.0-rc0/bin/git ls-tree -r ecf42d8adc4d7eb4386e37882d053704067712d4: -- 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189 .gitmodules 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39 add_file/add_file.txt 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949 basic_branching/file.txt 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0 binary/B.zip 100644 blob 72555d02a139700be930a6734b57c5fab84512b9 binary/C.zip 100644 blob 72555d02a139700be930a6734b57c5fab84512b9 binary/D.zip 100644 blob 668076c308990933c2df66efb87b946704252486 branch_mod_merge/file.txt 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b modification/mod_file.txt 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2 moved_dir/file.txt 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963 mv_file/moved_file.txt 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a out_of_order/file.txt 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87 submodule/atlassian-refapp 160000 commit 92b18c97d9d92d2bd02cfd1d268f22fddddafb67 submodule/bamboo-git-plugin 160000 commit 996b4f5a7e242a29fe573582d98f5174142bb97c submodule/stash 120000 blob 4cbb553f3f4ac2ee7b01ff6c951d6bf583c39c15 symlink/link.txt 100644 blob 57a3592bb2cf8e76e2faef7660945a2fe0e24360 symlink/target.txt

Given the command works without -l, and all -l is supposed to do is add sizes to blobs, it seems like something has regressed in the logic that decides whether to continue recursing or move on to the next sibling?

Can someone else whose C is a little less rusty check this and see if it's a bug? It almost seems like it _has_ to be, though. Bitbucket Server manages bare repositories which often include submodules, but none of those submodules (even assuming they're valid, which is not always the case for repositories in the wild) are ever _fetched_ into that bare repository. That implies this would always fail in any repository that contains submodules referencing other repositories.

Best regards, Bryan Turner

[1] https://packages.atlassian.com/maven/com/atlassian/stash/git/subrepositories/1.0/subrepositories-1.0.zip is a zipped bare repository that contains the ecf42d8 commit used above

Bryan Turner· Apr 7, 2022, 01:29 UTC · re: Bryan Turner · lore

Re: Git 2.36, ls-tree submodule regression?

On Wed, Apr 6, 2022 at 5:34 PM Bryan Turner <bturner@atlassian.com> wrote:
Show 86 quoted lines
>
> In the test suite for Bitbucket Server, we have a test repository[1]
> which has a commit that has a .gitmodules file that lists 3
> submodules, each of which has an entry in the tree with "Subproject
> commit <SHA>".
>
> The submodules aren't fetched (at least one of them doesn't actually
> exist anywhere to fetch), but running git ls-tree in Git 2.35.1 and
> prior shows the expected tree data:
>
> $ git ls-tree -r -l ecf42d8adc4d7eb4386e37882d053704067712d4: --
> 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189     346 .gitmodules
> 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39      20
> add_file/add_file.txt
> 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949      25
> basic_branching/file.txt
> 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0     167 binary/B.zip
> 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/C.zip
> 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/D.zip
> 100644 blob 668076c308990933c2df66efb87b946704252486      46
> branch_mod_merge/file.txt
> 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b      33
> modification/mod_file.txt
> 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2      18 moved_dir/file.txt
> 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963      24
> mv_file/moved_file.txt
> 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a      66
> out_of_order/file.txt
> 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87       -
> submodule/atlassian-refapp
> 160000 commit 92b18c97d9d92d2bd02cfd1d268f22fddddafb67       -
> submodule/bamboo-git-plugin
> 160000 commit 996b4f5a7e242a29fe573582d98f5174142bb97c       - submodule/stash
> 120000 blob 4cbb553f3f4ac2ee7b01ff6c951d6bf583c39c15      10 symlink/link.txt
> 100644 blob 57a3592bb2cf8e76e2faef7660945a2fe0e24360      27 symlink/target.txt
>
> The same ls-tree command in 2.36.0-rc0 fails:
>
> $ /opt/git/2.36.0-rc0/bin/git ls-tree -r -l
> ecf42d8adc4d7eb4386e37882d053704067712d4: --
> 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189     346 .gitmodules
> 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39      20
> add_file/add_file.txt
> 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949      25
> basic_branching/file.txt
> 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0     167 binary/B.zip
> 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/C.zip
> 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/D.zip
> 100644 blob 668076c308990933c2df66efb87b946704252486      46
> branch_mod_merge/file.txt
> 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b      33
> modification/mod_file.txt
> 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2      18 moved_dir/file.txt
> 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963      24
> mv_file/moved_file.txt
> 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a      66
> out_of_order/file.txt
> 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87       -
> submodule/atlassian-refapp
> error: Could not read 0ac826865ce7e8bc76022d49424f52e09b16dc87
> fatal: Invalid commit 0ac826865ce7e8bc76022d49424f52e09b16dc87 in
> submodule path submodule/atlassian-refapp
>
> Bisecting the error selects:
>
> $ git bisect good
> 9c4d58ff2c385f49585197c8650356955e1fa02e is the first bad commit
> commit 9c4d58ff2c385f49585197c8650356955e1fa02e
> Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Date:   Wed Mar 23 17:13:15 2022 +0800
>
>     ls-tree: split up "fast path" callbacks
>
>     Make the various if/else in the callbacks for the "fast path" a lot
>     easier to read by just using common functions for the parts that are
>     common, and have per-format callbacks for those parts that are
>     different.
>
>     Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>     Signed-off-by: Teng Long <dyroneteng@gmail.com>
>     Signed-off-by: Junio C Hamano <gitster@pobox.com>
>
>  builtin/ls-tree.c | 199 ++++++++++++++++++++++++++++++++++--------------------
>  1 file changed, 125 insertions(+), 74 deletions(-)
>
> (I've CC'd Ævar and Teng on this message.)

Actually, it looks like Ævar might already have a fix up for it in [2]. I'll see if I can apply that patch locally and verify it.

Best regards, Bryan Turner

[2] https://lore.kernel.org/git/patch-v2-1.1-ed83b3b74ab-20220404T234507Z-avarab@gmail.com/
Bryan Turner· Apr 7, 2022, 01:41 UTC · re: Bryan Turner · lore

Re: Git 2.36, ls-tree submodule regression?

On Wed, Apr 6, 2022 at 6:29 PM Bryan Turner <bturner@atlassian.com> wrote:
Show 91 quoted lines
>
> On Wed, Apr 6, 2022 at 5:34 PM Bryan Turner <bturner@atlassian.com> wrote:
> >
> > In the test suite for Bitbucket Server, we have a test repository[1]
> > which has a commit that has a .gitmodules file that lists 3
> > submodules, each of which has an entry in the tree with "Subproject
> > commit <SHA>".
> >
> > The submodules aren't fetched (at least one of them doesn't actually
> > exist anywhere to fetch), but running git ls-tree in Git 2.35.1 and
> > prior shows the expected tree data:
> >
> > $ git ls-tree -r -l ecf42d8adc4d7eb4386e37882d053704067712d4: --
> > 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189     346 .gitmodules
> > 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39      20
> > add_file/add_file.txt
> > 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949      25
> > basic_branching/file.txt
> > 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0     167 binary/B.zip
> > 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/C.zip
> > 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/D.zip
> > 100644 blob 668076c308990933c2df66efb87b946704252486      46
> > branch_mod_merge/file.txt
> > 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b      33
> > modification/mod_file.txt
> > 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2      18 moved_dir/file.txt
> > 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963      24
> > mv_file/moved_file.txt
> > 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a      66
> > out_of_order/file.txt
> > 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87       -
> > submodule/atlassian-refapp
> > 160000 commit 92b18c97d9d92d2bd02cfd1d268f22fddddafb67       -
> > submodule/bamboo-git-plugin
> > 160000 commit 996b4f5a7e242a29fe573582d98f5174142bb97c       - submodule/stash
> > 120000 blob 4cbb553f3f4ac2ee7b01ff6c951d6bf583c39c15      10 symlink/link.txt
> > 100644 blob 57a3592bb2cf8e76e2faef7660945a2fe0e24360      27 symlink/target.txt
> >
> > The same ls-tree command in 2.36.0-rc0 fails:
> >
> > $ /opt/git/2.36.0-rc0/bin/git ls-tree -r -l
> > ecf42d8adc4d7eb4386e37882d053704067712d4: --
> > 100644 blob c0de405f50ea72f9d2b973dd504671ba63ea7189     346 .gitmodules
> > 100644 blob 493021b1c9e0ecfebcd5ebce646a44138bc60b39      20
> > add_file/add_file.txt
> > 100644 blob d8c00a7fe6eb6a00378e103a46e3c472c084a949      25
> > basic_branching/file.txt
> > 100644 blob 927b01c060cf0dc5036e19abeab8e332d9075ea0     167 binary/B.zip
> > 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/C.zip
> > 100644 blob 72555d02a139700be930a6734b57c5fab84512b9     167 binary/D.zip
> > 100644 blob 668076c308990933c2df66efb87b946704252486      46
> > branch_mod_merge/file.txt
> > 100644 blob b1d99e581f0bb8138b8997f9268294e41faee05b      33
> > modification/mod_file.txt
> > 100644 blob 46fa903c8af8a92c2714a47ae5a341a99c2ee3b2      18 moved_dir/file.txt
> > 100644 blob f49abb7d6c366e6f83470e36439bc162234d7963      24
> > mv_file/moved_file.txt
> > 100644 blob 2e6b03cbbe90d3c9afcfc98967a5a71b4c9cff1a      66
> > out_of_order/file.txt
> > 160000 commit 0ac826865ce7e8bc76022d49424f52e09b16dc87       -
> > submodule/atlassian-refapp
> > error: Could not read 0ac826865ce7e8bc76022d49424f52e09b16dc87
> > fatal: Invalid commit 0ac826865ce7e8bc76022d49424f52e09b16dc87 in
> > submodule path submodule/atlassian-refapp
> >
> > Bisecting the error selects:
> >
> > $ git bisect good
> > 9c4d58ff2c385f49585197c8650356955e1fa02e is the first bad commit
> > commit 9c4d58ff2c385f49585197c8650356955e1fa02e
> > Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> > Date:   Wed Mar 23 17:13:15 2022 +0800
> >
> >     ls-tree: split up "fast path" callbacks
> >
> >     Make the various if/else in the callbacks for the "fast path" a lot
> >     easier to read by just using common functions for the parts that are
> >     common, and have per-format callbacks for those parts that are
> >     different.
> >
> >     Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> >     Signed-off-by: Teng Long <dyroneteng@gmail.com>
> >     Signed-off-by: Junio C Hamano <gitster@pobox.com>
> >
> >  builtin/ls-tree.c | 199 ++++++++++++++++++++++++++++++++++--------------------
> >  1 file changed, 125 insertions(+), 74 deletions(-)
> >
> > (I've CC'd Ævar and Teng on this message.)
>
> Actually, it looks like Ævar might already have a fix up for it in
> [2]. I'll see if I can apply that patch locally and verify it.

Confirmed that what's currently in "next" (which I believe will become 2.36.0-rc1 at some point?) does not manifest this issue.

Thanks for the fix--posted before I could even ask for it--and sorry for the noise.

Best regards, Bryan Turner

Teng Long· Apr 7, 2022, 02:22 UTC · re: Bryan Turner · lore

Re: Git 2.36, ls-tree submodule regression?

Show 27 quoted lines
> Bisecting the error selects:
> 
> $ git bisect good
> 9c4d58ff2c385f49585197c8650356955e1fa02e is the first bad commit
> commit 9c4d58ff2c385f49585197c8650356955e1fa02e
> Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Date:   Wed Mar 23 17:13:15 2022 +0800
> 
>     ls-tree: split up "fast path" callbacks
> 
>     Make the various if/else in the callbacks for the "fast path" a lot
>     easier to read by just using common functions for the parts that are
>     common, and have per-format callbacks for those parts that are
>     different.
> 
>     Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>     Signed-off-by: Teng Long <dyroneteng@gmail.com>
>     Signed-off-by: Junio C Hamano <gitster@pobox.com>
> 
>  builtin/ls-tree.c | 199 ++++++++++++++++++++++++++++++++++--------------------
>  1 file changed, 125 insertions(+), 74 deletions(-)
> 
> (I've CC'd Ævar and Teng on this message.)
> 
> Looking at the changes in that commit I see differences in the
> handling for long listings, so I tried removing the -l. At that point,
> the command works again:
Yes, it's a bug but already found and fixed by Ævar Arnfjörð Bjarmason.
> Given the command works without -l, and all -l is supposed to do is
> add sizes to blobs, it seems like something has regressed in the logic
> that decides whether to continue recursing or move on to the next
> sibling?
As you described blow, I almost comfirmed it's the problem that
found in [1] and Junio C Hamano already replied it in [2]
 
Show 7 quoted lines
> Can someone else whose C is a little less rusty check this and see if
> it's a bug? It almost seems like it _has_ to be, though. Bitbucket
> Server manages bare repositories which often include submodules, but
> none of those submodules (even assuming they're valid, which is not
> always the case for repositories in the wild) are ever _fetched_ into
> that bare repository. That implies this would always fail in any
> repository that contains submodules referencing other repositories.
Sorry for the trouble.
Thank you for debugging this and reporting this problem. 
Links:
  1. https://public-inbox.org/git/YktP0LltNKq7JZZs@google.com/
  2. https://public-inbox.org/git/xmqq8rsi1ap4.fsf@gitster.g/
Johannes Schindelin· Apr 7, 2022, 12:37 UTC · re: Teng Long · lore

Re: Git 2.36, ls-tree submodule regression?

Hi,
On Thu, 7 Apr 2022, Teng Long wrote:
Show 29 quoted lines
> > Bisecting the error selects:
> >
> > $ git bisect good
> > 9c4d58ff2c385f49585197c8650356955e1fa02e is the first bad commit
> > commit 9c4d58ff2c385f49585197c8650356955e1fa02e
> > Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> > Date:   Wed Mar 23 17:13:15 2022 +0800
> >
> >     ls-tree: split up "fast path" callbacks
> >
> >     Make the various if/else in the callbacks for the "fast path" a lot
> >     easier to read by just using common functions for the parts that are
> >     common, and have per-format callbacks for those parts that are
> >     different.
> >
> >     Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> >     Signed-off-by: Teng Long <dyroneteng@gmail.com>
> >     Signed-off-by: Junio C Hamano <gitster@pobox.com>
> >
> >  builtin/ls-tree.c | 199 ++++++++++++++++++++++++++++++++++--------------------
> >  1 file changed, 125 insertions(+), 74 deletions(-)
> >
> > (I've CC'd Ævar and Teng on this message.)
> >
> > Looking at the changes in that commit I see differences in the
> > handling for long listings, so I tried removing the -l. At that point,
> > the command works again:
>
> Yes, it's a bug but already found and fixed by Ævar Arnfjörð Bjarmason.

Correction, it was fixed by Josh Steadmon, and Ævar proposed an alternative, but Josh's version made it into the `main` branch:

https://github.com/git/git/commit/350296cc78912c245847ec65e55143053450cce1

Ciao, Johannes

Junio C Hamano· Apr 7, 2022, 18:20 UTC · re: Johannes Schindelin · lore

Re: Git 2.36, ls-tree submodule regression?

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> Correction, it was fixed by Josh Steadmon, and Ævar proposed an
> alternative, but Josh's version made it into the `main` branch:

Correct, although the actual fixes were identical one-liner and it didn't really matter in the end result where the fix was taken from.

← back to recent threads