git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check

From
René Scharfe <l.s.r@web.de>
Date
Dec 5, 2018, 18:41 UTC
Message-ID
<48ab861b-46c8-2bd5-0508-fa106efd8e56@web.de>
In-Reply-To
<20181205081544.GA9271@sigill.intra.peff.net>
Am 05.12.2018 um 09:15 schrieb Jeff King:
Show 12 quoted lines
> On Wed, Dec 05, 2018 at 01:51:36AM -0500, Jeff King wrote:
> 
>>> This
>>> function is easily converted to struct object_id, though, as its single
>>> caller can pass one on -- this makes the copy unnecessary.
>>
>> If you mean modifying sha1_loose_object_info() to take an oid, then
>> sure, I agree that is a good step forward (and that is exactly the "punt
>> until" moment I meant).
> 
> So the simple thing is to do that, and then have it pass oid->hash to
> the other functions it uses.
Yes.
> If we start to convert those, there's a
> little bit of a rabbit hole, but it's actually not too bad.

You don't need to crawl in just for quick_has_loose(), but eventually everything has to be converted. It seems a bit much for one patch, but perhaps that's just my ever-decreasing attention span speaking.

Converting one function prototype or struct member at a time seems about the right amount of change per patch to me. That's not always possible due to entanglement, of course.

Show 5 quoted lines
> Most of the spill-over is into the dumb-http code. Note that it actually
> uses sha1 itself! That probably needs to be the_hash_algo (though I'm
> not even sure how we'd negotiate the algorithm across a dumb fetch). At
> any rate, I don't think this patch makes anything _worse_ in that
> respect.
Right.
Show 10 quoted lines
> diff --git a/sha1-file.c b/sha1-file.c
> index 3ddf4c9426..0705709036 100644
> --- a/sha1-file.c
> +++ b/sha1-file.c
> @@ -333,12 +333,12 @@ int raceproof_create_file(const char *path, create_file_fn fn, void *cb)
>  	return ret;
>  }
>  
> -static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
> +static void fill_loose_path(struct strbuf *buf, const struct object_id *oid)
The new name fits.
Show 8 quoted lines
> @@ -879,15 +879,15 @@ int git_open_cloexec(const char *name, int flags)
>   * Note that it may point to static storage and is only valid until another
>   * call to stat_sha1_file().
>   */
> -static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
> -			  struct stat *st, const char **path)
> +static int stat_loose_object(struct repository *r, const struct object_id *oid,
> +			     struct stat *st, const char **path)

Hmm, read_sha1_file() was renamed to read_object_file() earlier this year, and I'd have expected this to become stat_object_file(). Names..

Anyway, the comment above and one a few lines below should be updated as well.

Show 17 quoted lines
>  {
>  	struct object_directory *odb;
>  	static struct strbuf buf = STRBUF_INIT;
>  
>  	prepare_alt_odb(r);
>  	for (odb = r->objects->odb; odb; odb = odb->next) {
> -		*path = odb_loose_path(odb, &buf, sha1);
> +		*path = odb_loose_path(odb, &buf, oid);
>  		if (!lstat(*path, st))
>  			return 0;
>  	}
> @@ -900,7 +900,7 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
>   * descriptor. See the caveats on the "path" parameter above.
>   */
>  static int open_sha1_file(struct repository *r,
> -			  const unsigned char *sha1, const char **path)
> +			  const struct object_id *oid, const char **path)
That function should lose the "sha1" in its name as well.
> -static void *map_sha1_file_1(struct repository *r, const char *path,
> -			     const unsigned char *sha1, unsigned long *size)
> +static void *map_loose_object_1(struct repository *r, const char *path,
> +			     const struct object_id *oid, unsigned long *size)
Similarly, map_object_file_1()?
Show 5 quoted lines
> -void *map_sha1_file(struct repository *r,
> -		    const unsigned char *sha1, unsigned long *size)
> +void *map_loose_object(struct repository *r,
> +		       const struct object_id *oid,
> +		       unsigned long *size)
Similar.
Show 8 quoted lines
> @@ -1045,7 +1043,9 @@ static int unpack_sha1_header_to_strbuf(git_zstream *stream, unsigned char *map,
>  	return -1;
>  }
>  
> -static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)
> +static void *unpack_loose_rest(git_zstream *stream,
> +			       void *buffer, unsigned long size,
> +			       const struct object_id *oid)
Hmm, both old and new name here look weird to me at this point.
Show 6 quoted lines
> -static int sha1_loose_object_info(struct repository *r,
> -				  const unsigned char *sha1,
> -				  struct object_info *oi, int flags)
> +static int loose_object_info(struct repository *r,
> +			     const struct object_id *oid,
> +			     struct object_info *oi, int flags)
And nothing of value was lost. :)
René
Previous: Jeff KingNext: Jeff King
Message 72 of 99 in “index-pack: improve performance on NFS”
  1. index-pack: improve performance on NFSJansen, Geert, Oct 25, 2018
  2. Junio C HamanoOct 26, 2018
  3. Ævar Arnfjörð BjarmasonOct 26, 2018
  4. Junio C HamanoOct 27, 2018
  5. Jeff KingOct 27, 2018
  6. Ævar Arnfjörð BjarmasonOct 27, 2018
  7. 0/4 index-pack: optionally turn off SHA-1 collision checkingÆvar Arnfjörð Bjarmason, Oct 28, 2018
  8. Geert JansenOct 30, 2018
  9. Junio C HamanoOct 30, 2018
  10. 0/3 index-pack: test updatesÆvar Arnfjörð Bjarmason, Oct 30, 2018
  11. index-pack: add ability to disable SHA-1 collision checkÆvar Arnfjörð Bjarmason, Nov 13, 2018
  12. Junio C HamanoNov 14, 2018
  13. Ævar Arnfjörð BjarmasonNov 14, 2018
  14. 1/3 pack-objects test: modernize styleÆvar Arnfjörð Bjarmason, Oct 30, 2018
  15. 2/3 pack-objects tests: don't leave test .git corrupt at endÆvar Arnfjörð Bjarmason, Oct 30, 2018
  16. 3/3 index-pack tests: don't leave test repo dirty at endÆvar Arnfjörð Bjarmason, Oct 30, 2018
  17. 1/4 pack-objects test: modernize styleÆvar Arnfjörð Bjarmason, Oct 28, 2018
  18. 2/4 pack-objects tests: don't leave test .git corrupt at endÆvar Arnfjörð Bjarmason, Oct 28, 2018
  19. 3/4 index-pack tests: don't leave test repo dirty at endÆvar Arnfjörð Bjarmason, Oct 28, 2018
  20. 4/4 index-pack: add ability to disable SHA-1 collision checkÆvar Arnfjörð Bjarmason, Oct 28, 2018
  21. Jeff KingOct 29, 2018
  22. Jeff KingOct 29, 2018
  23. Ævar Arnfjörð BjarmasonOct 29, 2018
  24. Jeff KingOct 29, 2018
  25. Geert JansenNov 7, 2018
  26. Jeff KingNov 8, 2018
  27. Geert JansenNov 8, 2018
  28. Jeff KingNov 8, 2018
  29. Geert JansenNov 8, 2018
  30. Ævar Arnfjörð BjarmasonNov 8, 2018
  31. Ævar Arnfjörð BjarmasonNov 9, 2018
  32. Jeff KingNov 12, 2018
  33. 0/9 caching loose objectsJeff King, Nov 12, 2018
  34. 1/9 fsck: do not reuse child_process structsJeff King, Nov 12, 2018
  35. Derrick StoleeNov 12, 2018
  36. 2/9 submodule--helper: prefer strip_suffix() to ends_with()Jeff King, Nov 12, 2018
  37. Stefan BellerNov 12, 2018
  38. 3/9 rename "alternate_object_database" to "object_directory"Jeff King, Nov 12, 2018
  39. Derrick StoleeNov 12, 2018
  40. Jeff KingNov 12, 2018
  41. Ramsay JonesNov 12, 2018
  42. 4/9 sha1_file_name(): overwrite buffer instead of appendingJeff King, Nov 12, 2018
  43. Derrick StoleeNov 12, 2018
  44. 5/9 handle alternates paths the same as the main object dirJeff King, Nov 12, 2018
  45. Derrick StoleeNov 12, 2018
  46. Jeff KingNov 12, 2018
  47. Derrick StoleeNov 12, 2018
  48. 6/9 sha1-file: use an object_directory for the main object dirJeff King, Nov 12, 2018
  49. Derrick StoleeNov 12, 2018
  50. Jeff KingNov 12, 2018
  51. Stefan BellerNov 12, 2018
  52. Jeff KingNov 22, 2018
  53. Stefan BellerNov 12, 2018
  54. 7/9 object-store: provide helpers for loose_objects_cacheJeff King, Nov 12, 2018
  55. René ScharfeNov 12, 2018
  56. Jeff KingNov 12, 2018
  57. 8/9 sha1-file: use loose object cache for quick existence checkJeff King, Nov 12, 2018
  58. Derrick StoleeNov 12, 2018
  59. Ævar Arnfjörð BjarmasonNov 12, 2018
  60. Jeff KingNov 12, 2018
  61. Ævar Arnfjörð BjarmasonNov 12, 2018
  62. Ævar Arnfjörð BjarmasonNov 12, 2018
  63. Ævar Arnfjörð BjarmasonNov 13, 2018
  64. René ScharfeNov 14, 2018
  65. René ScharfeDec 2, 2018
  66. Jeff KingDec 3, 2018
  67. René ScharfeDec 4, 2018
  68. Jeff KingDec 5, 2018
  69. René ScharfeDec 5, 2018
  70. Jeff KingDec 5, 2018
  71. Jeff KingDec 5, 2018
  72. René ScharfeDec 5, 2018
  73. Jeff KingDec 5, 2018
  74. Geert JansenNov 12, 2018
  75. René ScharfeNov 27, 2018
  76. Jeff KingDec 1, 2018
  77. 9/9 fetch-pack: drop custom loose object cacheJeff King, Nov 12, 2018
  78. René ScharfeNov 12, 2018
  79. Ævar Arnfjörð BjarmasonNov 12, 2018
  80. Jeff KingNov 12, 2018
  81. René ScharfeNov 12, 2018
  82. Derrick StoleeNov 12, 2018
  83. Stefan BellerNov 12, 2018
  84. Ævar Arnfjörð BjarmasonNov 9, 2018
  85. Duy NguyenNov 9, 2018
  86. Ævar Arnfjörð BjarmasonNov 10, 2018
  87. Jeff KingNov 12, 2018
  88. Geert JansenNov 12, 2018
  89. Duy NguyenOct 27, 2018
  90. Jeff KingOct 29, 2018
  91. Junio C HamanoOct 29, 2018
  92. Jeff KingOct 29, 2018
  93. Ævar Arnfjörð BjarmasonOct 29, 2018
  94. Geert JansenOct 29, 2018
  95. Jeff KingOct 29, 2018
  96. Geert JansenOct 29, 2018
  97. Jeff KingOct 29, 2018
  98. Stefan BellerOct 29, 2018
  99. Jeff KingOct 29, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.