git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 6/9] sha1-file: use an object_directory for the main object dir

From
Stefan Beller <sbeller@google.com>
Date
Nov 12, 2018, 19:04 UTC
Message-ID
<CAGZ79kaH6WiSYeW2DCyZxYfg+qYuwTnF=2ZBO8rERJS_-8GeGw@mail.gmail.com>
In-Reply-To
<20181112160910.GA7612@sigill.intra.peff.net>
On Mon, Nov 12, 2018 at 8:09 AM Jeff King <peff@peff.net> wrote:
Show 25 quoted lines
>
> On Mon, Nov 12, 2018 at 10:48:36AM -0500, Derrick Stolee wrote:
>
> > > If the "the first one is the main store, the rest are alternates" bit is
> > > too subtle, we could mark each "struct object_directory" with a bit for
> > > "is_local".
> >
> > This is probably a good thing to do proactively. We have the equivalent in
> > the packed_git struct, but that's also because they get out of order. At the
> > moment, I can't think of a read-only action that needs to treat the local
> > object directory more carefully. The closest I know about is 'git
> > pack-objects --local', but that also writes a pack-file.
> >
> > I assume that when we write a pack-file to the "default location" we use
> > get_object_directory() instead of referring to the default object_directory?
>
> Generally, yes, though that should eventually be going away in favor of
> accessing it via a "struct repository". And after my series,
> get_object_directory() is just returning the_repository->objects->odb->path
> (i.e., using the "first one is main" rule).
>
> One thing that makes me nervous about a "local" flag in each struct is
> that it implies that it's the source of truth for where to write to. So
> what does git_object_directory() look like after that? Do we leave it
> with the "first one is main" rule? Or does it become:

s/git/get/ ;-) get_object_directory is very old and was introduced in e1b10391ea (Use git config file for committer name and email info, 2005-10-11) by Linus.

I would argue that we might want to get rid of that function now, actually as it doesn't seem to add value to the code (assuming the BUG never triggers), and using a_repo->objects->objectdir or after this series a_repo->objects->odb->path; is just as short.

    $ git grep get_object_directory |wc -l
    30
    $ git grep -- "->objects->objectdir"  |wc -l
    10
Ah well, we're not there yet.
Show 7 quoted lines
>   for (odb = the_repository->objects->odb; odb; odb = odb->next) {
>         if (odb->local)
>                 return odb->path;
>   }
>   return NULL; /* yikes? */
>
> ? That feels like it's making things more complicated, not less.
It depends if the caller cares about the local flag.

I'd think we can have more than one local, eventually? Just think of the partial clone stuff that may have a local set of promised stuff and another set of actual objects, which may be stored in different local odbs.

If the caller cares about the distinction, they would need to write out this loop as above themselves. If they don't care, we could migrate them to not use this function, so we can get rid of it?

Show 48 quoted lines
> > > -   for (odb = r->objects->alt_odb_list; odb; odb = odb->next) {
> > > -           prepare_multi_pack_index_one(r, odb->path, 0);
> > > -           prepare_packed_git_one(r, odb->path, 0);
> > > +   for (odb = r->objects->odb; odb; odb = odb->next) {
> > > +           int local = (odb == r->objects->odb);
> >
> > Here seems to be a place where `odb->is_local` would help.
>
> Yes, though I don't mind this spot in particular, as the check is pretty
> straight-forward.
>
> I think an example that would benefit more is the check_and_freshen()
> stuff. There we have two almost-the-same wrappers, one of which operates
> on just the first element of the list, and the other of which operates
> on all of the elements after the first.
>
> It could become:
>
>   static int check_and_freshen_odb(struct object_directory *odb_list,
>                                    const struct object_id *oid,
>                                    int freshen,
>                                    int local)
>   {
>         struct object_directory *odb;
>
>         for (odb = odb_list; odb; odb = odb->next) {
>                 static struct strbuf path = STRBUF_INIT;
>
>                 if (odb->local != local)
>                         continue;
>
>                 odb_loose_path(odb, &path, oid->hash);
>                 return check_and_freshen_file(path.buf, freshen);
>         }
>   }
>
>   int check_and_freshen_local(const struct object_id *oid, int freshen)
>   {
>         return check_and_freshen_odb(the_repository->objects->odb, oid,
>                                      freshen, 1);
>   }
>
>   int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)
>   {
>         return check_and_freshen_odb(the_repository->objects->odb, oid,
>                                      freshen, 0);
>   }
>

I am fine with (a maybe better documented) "first is local" rule, but the code above looks intriguing, except a little wasteful (we need two full loops in check_and_freshen, but ideally we can do by just one loop).

What does the local flag mean anyway in a world where we have many odbs in a repository, that are not distinguishable except by their order? AFAICT it is actually to be used for differentiating how much we care in fsck/cat-file/packing, as it may be borrowed from an alternate, so maybe the flag is rather to be named after ownership and not so much about it locality? (I think "borrowed" or "owned" or even just "important" or "external" or "alternate" may work)

Stefan
Previous: Jeff KingNext: Jeff King
Message 51 of 99 in “index-pack: improve performance on NFS”
  1. index-pack: improve performance on NFSJansen, Geert, Oct 25, 2018
  2. Junio C HamanoOct 26, 2018
  3. Ævar Arnfjörð BjarmasonOct 26, 2018
  4. Junio C HamanoOct 27, 2018
  5. Jeff KingOct 27, 2018
  6. Ævar Arnfjörð BjarmasonOct 27, 2018
  7. 0/4 index-pack: optionally turn off SHA-1 collision checkingÆvar Arnfjörð Bjarmason, Oct 28, 2018
  8. Geert JansenOct 30, 2018
  9. Junio C HamanoOct 30, 2018
  10. 0/3 index-pack: test updatesÆvar Arnfjörð Bjarmason, Oct 30, 2018
  11. index-pack: add ability to disable SHA-1 collision checkÆvar Arnfjörð Bjarmason, Nov 13, 2018
  12. Junio C HamanoNov 14, 2018
  13. Ævar Arnfjörð BjarmasonNov 14, 2018
  14. 1/3 pack-objects test: modernize styleÆvar Arnfjörð Bjarmason, Oct 30, 2018
  15. 2/3 pack-objects tests: don't leave test .git corrupt at endÆvar Arnfjörð Bjarmason, Oct 30, 2018
  16. 3/3 index-pack tests: don't leave test repo dirty at endÆvar Arnfjörð Bjarmason, Oct 30, 2018
  17. 1/4 pack-objects test: modernize styleÆvar Arnfjörð Bjarmason, Oct 28, 2018
  18. 2/4 pack-objects tests: don't leave test .git corrupt at endÆvar Arnfjörð Bjarmason, Oct 28, 2018
  19. 3/4 index-pack tests: don't leave test repo dirty at endÆvar Arnfjörð Bjarmason, Oct 28, 2018
  20. 4/4 index-pack: add ability to disable SHA-1 collision checkÆvar Arnfjörð Bjarmason, Oct 28, 2018
  21. Jeff KingOct 29, 2018
  22. Jeff KingOct 29, 2018
  23. Ævar Arnfjörð BjarmasonOct 29, 2018
  24. Jeff KingOct 29, 2018
  25. Geert JansenNov 7, 2018
  26. Jeff KingNov 8, 2018
  27. Geert JansenNov 8, 2018
  28. Jeff KingNov 8, 2018
  29. Geert JansenNov 8, 2018
  30. Ævar Arnfjörð BjarmasonNov 8, 2018
  31. Ævar Arnfjörð BjarmasonNov 9, 2018
  32. Jeff KingNov 12, 2018
  33. 0/9 caching loose objectsJeff King, Nov 12, 2018
  34. 1/9 fsck: do not reuse child_process structsJeff King, Nov 12, 2018
  35. Derrick StoleeNov 12, 2018
  36. 2/9 submodule--helper: prefer strip_suffix() to ends_with()Jeff King, Nov 12, 2018
  37. Stefan BellerNov 12, 2018
  38. 3/9 rename "alternate_object_database" to "object_directory"Jeff King, Nov 12, 2018
  39. Derrick StoleeNov 12, 2018
  40. Jeff KingNov 12, 2018
  41. Ramsay JonesNov 12, 2018
  42. 4/9 sha1_file_name(): overwrite buffer instead of appendingJeff King, Nov 12, 2018
  43. Derrick StoleeNov 12, 2018
  44. 5/9 handle alternates paths the same as the main object dirJeff King, Nov 12, 2018
  45. Derrick StoleeNov 12, 2018
  46. Jeff KingNov 12, 2018
  47. Derrick StoleeNov 12, 2018
  48. 6/9 sha1-file: use an object_directory for the main object dirJeff King, Nov 12, 2018
  49. Derrick StoleeNov 12, 2018
  50. Jeff KingNov 12, 2018
  51. Stefan BellerNov 12, 2018
  52. Jeff KingNov 22, 2018
  53. Stefan BellerNov 12, 2018
  54. 7/9 object-store: provide helpers for loose_objects_cacheJeff King, Nov 12, 2018
  55. René ScharfeNov 12, 2018
  56. Jeff KingNov 12, 2018
  57. 8/9 sha1-file: use loose object cache for quick existence checkJeff King, Nov 12, 2018
  58. Derrick StoleeNov 12, 2018
  59. Ævar Arnfjörð BjarmasonNov 12, 2018
  60. Jeff KingNov 12, 2018
  61. Ævar Arnfjörð BjarmasonNov 12, 2018
  62. Ævar Arnfjörð BjarmasonNov 12, 2018
  63. Ævar Arnfjörð BjarmasonNov 13, 2018
  64. René ScharfeNov 14, 2018
  65. René ScharfeDec 2, 2018
  66. Jeff KingDec 3, 2018
  67. René ScharfeDec 4, 2018
  68. Jeff KingDec 5, 2018
  69. René ScharfeDec 5, 2018
  70. Jeff KingDec 5, 2018
  71. Jeff KingDec 5, 2018
  72. René ScharfeDec 5, 2018
  73. Jeff KingDec 5, 2018
  74. Geert JansenNov 12, 2018
  75. René ScharfeNov 27, 2018
  76. Jeff KingDec 1, 2018
  77. 9/9 fetch-pack: drop custom loose object cacheJeff King, Nov 12, 2018
  78. René ScharfeNov 12, 2018
  79. Ævar Arnfjörð BjarmasonNov 12, 2018
  80. Jeff KingNov 12, 2018
  81. René ScharfeNov 12, 2018
  82. Derrick StoleeNov 12, 2018
  83. Stefan BellerNov 12, 2018
  84. Ævar Arnfjörð BjarmasonNov 9, 2018
  85. Duy NguyenNov 9, 2018
  86. Ævar Arnfjörð BjarmasonNov 10, 2018
  87. Jeff KingNov 12, 2018
  88. Geert JansenNov 12, 2018
  89. Duy NguyenOct 27, 2018
  90. Jeff KingOct 29, 2018
  91. Junio C HamanoOct 29, 2018
  92. Jeff KingOct 29, 2018
  93. Ævar Arnfjörð BjarmasonOct 29, 2018
  94. Geert JansenOct 29, 2018
  95. Jeff KingOct 29, 2018
  96. Geert JansenOct 29, 2018
  97. Jeff KingOct 29, 2018
  98. Stefan BellerOct 29, 2018
  99. Jeff KingOct 29, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.