git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse

From
Derrick Stolee <derrickstolee@github.com>
Date
Sep 8, 2022, 20:46 UTC
Message-ID
<093827ae-41ef-5f7c-7829-647536ce1305@github.com>
In-Reply-To
<xmqqczc5rblr.fsf@gitster.g>
On 9/8/2022 1:59 PM, Junio C Hamano wrote:
Show 29 quoted lines
> Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> writes:
> 
>> +
>> +	/*
>> +	 * NEEDSWORK: when reading a submodule, the sparsity settings in the
>> +	 * superproject are incorrectly forgotten or misused. For example:
>> +	 *
>> +	 * 1. "command_requires_full_index"
>> +	 * 	When this setting is turned on for `grep`, only the superproject
>> +	 *	knows it. All the submodules are read with their own configs
>> +	 *	and get prepare_repo_settings()'d. Therefore, these submodules
>> +	 *	"forget" the sparse-index feature switch. As a result, the index
>> +	 *	of these submodules are expanded unexpectedly.
> 
> Is this fundamental, or is it just this version of the patch is
> incomplete in that it still does not propagate the bit from
> the_repository->settings to submodule's settings?  Should a change
> to propagate the bit be included for this topic to be complete?
> 
> To put it another way, when grep with this version of the patch
> recurses into a submodule, does it work correctly even without
> flipping command_requires_full_index on in the "struct repository"
> instance for the submodule?  If so, then the NEEDSWORK above may be
> just performance issue.  If it behaves incorrectly, then it means
> we cannot safely make "git grep" aware of sparse index yet.  It is
> hard to tell which one you meant in the above.
> 
> I think the same question needs to be asked for other points
> (omitted from quoting) in this list.

I think this comment is misplaced. It should either be contained in the commit message or placed closer to this diff hunk:

Show 19 quoted lines
>> @@ -537,8 +561,20 @@ static int grep_cache(struct grep_opt *opt,
>>  
>>  		strbuf_setlen(&name, name_base_len);
>>  		strbuf_addstr(&name, ce->name);
>> +		if (S_ISSPARSEDIR(ce->ce_mode)) {
>> +			enum object_type type;
>> +			struct tree_desc tree;
>> +			void *data;
>> +			unsigned long size;
>> +
>> +			data = read_object_file(&ce->oid, &type, &size);
>> +			init_tree_desc(&tree, data, size);
>>  
>> -		if (S_ISREG(ce->ce_mode) &&
>> +			hit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);
>> +			strbuf_setlen(&name, name_base_len);
>> +			strbuf_addstr(&name, ce->name);
>> +			free(data);
>> +		} else if (S_ISREG(ce->ce_mode) &&

The conclusion we were trying to reach is that you (Junio) correctly identified a bug in how we were calling grep_tree() in this hunk in its v4 form.

HOWEVER: it "doesn't matter" because the sparse index doesn't work
at all within a submodule. Specifically, if a super-repo does not
enable sparse-checkout, but the submodule _does_, then we don't
know how Git will behave currently. His reasonings go on to explain
why the situation is fraught:
* command_requires_full_index is set in a builtin only for the
  top-level project, so when we traverse into a submodule, we don't
  re-check if the current builtin has integrated with sparse index
  and expand a sparse index to a full one.
* core_apply_sparse_checkout is a global not even associated with
  a repository struct. What happens when a super project is not
  sparse but a submodule is? Or vice-versa? I honestly don't know,
  and it will require testing to find out.

Shaoxuan's comment is attempting to list the reasons why submodules do not currently work with sparse-index, and specifically that we can add tests that _should_ exercise this code in a meaningful way, but because of the current limitations of the codebase, the code isn't actually exercised in that scenario.

In order to actually create a test that demonstrates how submodules and sparse-checkout work with this logic, we need to do some serious refactoring of the sparse-checkout logic to care about the repository struct, along with some other concerns specifically around the sparse index. This doesn't seem appropriate for the GSoC timeline or even for just this topic.

Victoria and I have noted this issue down and will try to find time to investigate further, with a target of being able to actually exercise this grep_tree() call within a sparse index in a submodule, giving us full confidence that name_base_len is the correct value to put in that parameter.

Thanks, -Stolee

Previous: Junio C HamanoNext: Junio C Hamano
Message 56 of 69 in “grep: integrate with sparse index”
  1. 0/2 grep: integrate with sparse indexShaoxuan Yuan, Aug 17, 2022
  2. 1/2 builtin/grep.c: add --sparse optionShaoxuan Yuan, Aug 17, 2022
  3. Derrick StoleeAug 17, 2022
  4. Junio C HamanoAug 17, 2022
  5. Victoria DyeAug 17, 2022
  6. Derrick StoleeAug 17, 2022
  7. Junio C HamanoAug 17, 2022
  8. Elijah NewrenAug 17, 2022
  9. Shaoxuan YuanAug 24, 2022
  10. Derrick StoleeAug 24, 2022
  11. 2/2 builtin/grep.c: integrate with sparse indexShaoxuan Yuan, Aug 17, 2022
  12. Derrick StoleeAug 17, 2022
  13. Shaoxuan YuanAug 24, 2022
  14. Derrick StoleeAug 25, 2022
  15. Derrick StoleeAug 17, 2022
  16. 0/2 grep: integrate with sparse indexShaoxuan Yuan, Aug 29, 2022
  17. 2/2 builtin/grep.c: integrate with sparse indexShaoxuan Yuan, Aug 29, 2022
  18. Derrick StoleeAug 30, 2022
  19. 1/2 builtin/grep.c: add --sparse optionShaoxuan Yuan, Aug 29, 2022
  20. 0/3 grep: integrate with sparse indexShaoxuan Yuan, Sep 1, 2022
  21. 1/3 builtin/grep.c: add --sparse optionShaoxuan Yuan, Sep 1, 2022
  22. 2/3 builtin/grep.c: integrate with sparse indexShaoxuan Yuan, Sep 1, 2022
  23. 3/3 builtin/grep.c: walking tree instead of expanding index with --sparseShaoxuan Yuan, Sep 1, 2022
  24. Derrick StoleeSep 1, 2022
  25. Shaoxuan YuanSep 1, 2022
  26. Junio C HamanoSep 1, 2022
  27. Junio C HamanoSep 1, 2022
  28. Shaoxuan YuanSep 1, 2022
  29. Shaoxuan YuanSep 1, 2022
  30. Victoria DyeSep 2, 2022
  31. Shaoxuan YuanSep 2, 2022
  32. 0/3 grep: integrate with sparse indexShaoxuan Yuan, Sep 3, 2022
  33. 1/3 builtin/grep.c: add --sparse optionShaoxuan Yuan, Sep 3, 2022
  34. 2/3 builtin/grep.c: integrate with sparse indexShaoxuan Yuan, Sep 3, 2022
  35. 3/3 builtin/grep.c: walking tree instead of expanding index with --sparseShaoxuan Yuan, Sep 3, 2022
  36. Junio C HamanoSep 3, 2022
  37. Shaoxuan YuanSep 8, 2022
  38. 0/3 grep: integrate with sparse indexShaoxuan Yuan, Sep 8, 2022
  39. 1/3 builtin/grep.c: add --sparse optionShaoxuan Yuan, Sep 8, 2022
  40. Victoria DyeSep 10, 2022
  41. Elijah NewrenSep 14, 2022
  42. Junio C HamanoSep 15, 2022
  43. Elijah NewrenSep 18, 2022
  44. Victoria DyeSep 18, 2022
  45. Junio C HamanoSep 19, 2022
  46. Shaoxuan YuanSep 19, 2022
  47. Ævar Arnfjörð BjarmasonSep 19, 2022
  48. Elijah NewrenSep 20, 2022
  49. Shaoxuan YuanSep 17, 2022
  50. Elijah NewrenSep 18, 2022
  51. Shaoxuan YuanSep 19, 2022
  52. Shaoxuan YuanSep 17, 2022
  53. 2/3 builtin/grep.c: integrate with sparse indexShaoxuan Yuan, Sep 8, 2022
  54. 3/3 builtin/grep.c: walking tree instead of expanding index with --sparseShaoxuan Yuan, Sep 8, 2022
  55. Junio C HamanoSep 8, 2022
  56. Derrick StoleeSep 8, 2022
  57. Junio C HamanoSep 8, 2022
  58. Shaoxuan YuanSep 8, 2022
  59. Derrick StoleeSep 9, 2022
  60. Junio C HamanoSep 13, 2022
  61. Victoria DyeSep 10, 2022
  62. 0/1 grep: integrate with sparse indexShaoxuan Yuan, Sep 23, 2022
  63. 1/1 builtin/grep.c: integrate with sparse indexShaoxuan Yuan, Sep 23, 2022
  64. Junio C HamanoSep 23, 2022
  65. Junio C HamanoSep 23, 2022
  66. Junio C HamanoSep 26, 2022
  67. Derrick StoleeSep 23, 2022
  68. Victoria DyeSep 23, 2022
  69. Junio C HamanoSep 23, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.