git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 24/44] packfile: compute and use the index CRC offset

From
brian m. carlson <sandals@crustytoothpaste.net>
Date
May 25, 2020, 19:59 UTC
Message-ID
<20200525195930.309665-25-sandals@crustytoothpaste.net>
In-Reply-To
<20200525195930.309665-1-sandals@crustytoothpaste.net>

Both v2 pack index files and the v3 format specified as part of the NewHash work have similar data starting at the CRC table. Much of the existing code wants to read either this table or the offset entries following it, and in doing so computes the offset each time.

In order to share as much code between v2 and v3, compute the offset of the CRC table and store it when the pack is opened. Use this value to compute offsets to not only the CRC table, but to the offset entries beyond it.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 builtin/index-pack.c | 6 +-----
 object-store.h       | 1 +
 packfile.c           | 1 +
 3 files changed, 3 insertions(+), 5 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index f176dd28c8..7bea1fba52 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1555,13 +1555,9 @@ static void read_v2_anomalous_offsets(struct packed_git *p,
 {
 	const uint32_t *idx1, *idx2;
 	uint32_t i;
-	const uint32_t hashwords = the_hash_algo->rawsz / sizeof(uint32_t);
 
 	/* The address of the 4-byte offset table */
-	idx1 = (((const uint32_t *)p->index_data)
-		+ 2 /* 8-byte header */
-		+ 256 /* fan out */
-		+ hashwords * p->num_objects /* object ID table */
+	idx1 = (((const uint32_t *)((const uint8_t *)p->index_data + p->crc_offset))
 		+ p->num_objects /* CRC32 table */
 		);
 
diff --git a/object-store.h b/object-store.h
index d1e490f203..f439d47af8 100644
--- a/object-store.h
+++ b/object-store.h
@@ -70,6 +70,7 @@ struct packed_git {
 	size_t index_size;
 	uint32_t num_objects;
 	uint32_t num_bad_objects;
+	uint32_t crc_offset;
 	unsigned char *bad_object_sha1;
 	int index_version;
 	time_t mtime;
diff --git a/packfile.c b/packfile.c
index f4e752996d..6ab5233613 100644
--- a/packfile.c
+++ b/packfile.c
@@ -178,6 +178,7 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
 		     */
 		    (sizeof(off_t) <= 4))
 			return error("pack too large for current definition of off_t in %s", path);
+		p->crc_offset = 8 + 4 * 256 + nr * hashsz;
 	}
 
 	p->index_version = version;
Previous: brian m. carlsonNext: brian m. carlson
Message 124 of 175 in “SHA-256 part 2/3: protocol functionality”
  1. 00/44 SHA-256 part 2/3: protocol functionalitybrian m. carlson, May 13, 2020
  2. 01/44 t1050: match object ID paths in a hash-insensitive waybrian m. carlson, May 13, 2020
  3. 02/44 Documentation: document v1 protocol object-format capabilitybrian m. carlson, May 13, 2020
  4. Martin ÅgrenMay 13, 2020
  5. Junio C HamanoMay 14, 2020
  6. brian m. carlsonMay 15, 2020
  7. Junio C HamanoMay 16, 2020
  8. 04/44 wrapper: add function to compare strings with different NUL terminationbrian m. carlson, May 13, 2020
  9. Martin ÅgrenMay 13, 2020
  10. 06/44 connect: add function to parse multiple v1 capability valuesbrian m. carlson, May 13, 2020
  11. 07/44 connect: add function to fetch value of a v2 server capabilitybrian m. carlson, May 13, 2020
  12. Martin ÅgrenMay 13, 2020
  13. 08/44 pkt-line: add a member for hash algorithmbrian m. carlson, May 13, 2020
  14. 10/44 connect: add function to detect supported v1 hash functionsbrian m. carlson, May 13, 2020
  15. Martin ÅgrenMay 13, 2020
  16. brian m. carlsonMay 13, 2020
  17. 15/44 builtin/receive-pack: detect when the server doesn't support our hashbrian m. carlson, May 13, 2020
  18. Martin ÅgrenMay 16, 2020
  19. 12/44 connect: make parse_feature_value externbrian m. carlson, May 13, 2020
  20. Martin ÅgrenMay 13, 2020
  21. 13/44 fetch-pack: detect when the server doesn't support our hashbrian m. carlson, May 13, 2020
  22. 25/44 packfile: compute and use the index CRC offsetbrian m. carlson, May 13, 2020
  23. Martin ÅgrenMay 16, 2020
  24. 24/44 t3200: mark assertion with SHA1 prerequisitebrian m. carlson, May 13, 2020
  25. Martin ÅgrenMay 16, 2020
  26. 22/44 fetch-pack: parse and advertise the object-format capabilitybrian m. carlson, May 13, 2020
  27. Martin ÅgrenMay 16, 2020
  28. 23/44 setup: set the_repository's hash algo when checking formatbrian m. carlson, May 13, 2020
  29. Martin ÅgrenMay 16, 2020
  30. brian m. carlsonMay 16, 2020
  31. 21/44 t5704: send object-format capability with SHA-256brian m. carlson, May 13, 2020
  32. Martin ÅgrenMay 16, 2020
  33. brian m. carlsonMay 16, 2020
  34. 18/44 remote-curl: implement object-format extensionsbrian m. carlson, May 13, 2020
  35. 30/44 connect: pass full packet reader when parsing v2 refsbrian m. carlson, May 13, 2020
  36. Martin ÅgrenMay 16, 2020
  37. 36/44 builtin/index-pack: add option to specify hash algorithmbrian m. carlson, May 13, 2020
  38. Martin ÅgrenMay 16, 2020
  39. brian m. carlsonMay 16, 2020
  40. Martin ÅgrenMay 17, 2020
  41. brian m. carlsonMay 17, 2020
  42. 32/44 serve: advertise object-format capability for protocol v2brian m. carlson, May 13, 2020
  43. Martin ÅgrenMay 16, 2020
  44. 44/44 remote-testgit: adapt for object-formatbrian m. carlson, May 13, 2020
  45. 20/44 t5562: pass object-format in synthesized test databrian m. carlson, May 13, 2020
  46. Martin ÅgrenMay 16, 2020
  47. brian m. carlsonMay 16, 2020
  48. 43/44 bundle: detect hash algorithm when reading refsbrian m. carlson, May 13, 2020
  49. 41/44 t5703: use object-format serve optionbrian m. carlson, May 13, 2020
  50. 42/44 t5300: pass --object-format to git index-packbrian m. carlson, May 13, 2020
  51. 40/44 t5702: offer an object-format capability in the testbrian m. carlson, May 13, 2020
  52. 39/44 t/helper: initialize the repository for test-sha1-arraybrian m. carlson, May 13, 2020
  53. 03/44 connect: have ref processing code take struct packet_readerbrian m. carlson, May 13, 2020
  54. Martin ÅgrenMay 13, 2020
  55. 09/44 transport: add a hash algorithm memberbrian m. carlson, May 13, 2020
  56. 16/44 docs: update remote helper docs for object-format extensionsbrian m. carlson, May 13, 2020
  57. 19/44 builtin/clone: initialize hash algorithm properlybrian m. carlson, May 13, 2020
  58. Martin ÅgrenMay 16, 2020
  59. 14/44 connect: detect algorithm when fetching refsbrian m. carlson, May 13, 2020
  60. Martin ÅgrenMay 16, 2020
  61. brian m. carlsonMay 16, 2020
  62. 05/44 remote: advertise the object-format capability on the server sidebrian m. carlson, May 13, 2020
  63. 27/44 builtin/show-index: provide options to determine hash algobrian m. carlson, May 13, 2020
  64. Junio C HamanoMay 18, 2020
  65. brian m. carlsonMay 19, 2020
  66. 33/44 t5500: make hash independentbrian m. carlson, May 13, 2020
  67. 38/44 remote-curl: avoid truncating refs with ls-remotebrian m. carlson, May 13, 2020
  68. 35/44 remote-curl: detect algorithm for dumb HTTP by sizebrian m. carlson, May 13, 2020
  69. Martin ÅgrenMay 16, 2020
  70. 37/44 t1050: pass algorithm to index-pack when outside repobrian m. carlson, May 13, 2020
  71. 34/44 builtin/ls-remote: initialize repository based on fetchbrian m. carlson, May 13, 2020
  72. Martin ÅgrenMay 16, 2020
  73. brian m. carlsonMay 16, 2020
  74. 31/44 connect: parse v2 refs with correct hash algorithmbrian m. carlson, May 13, 2020
  75. Martin ÅgrenMay 16, 2020
  76. brian m. carlsonMay 17, 2020
  77. 11/44 send-pack: detect when the server doesn't support our hashbrian m. carlson, May 13, 2020
  78. Martin ÅgrenMay 13, 2020
  79. brian m. carlsonMay 13, 2020
  80. 28/44 t1302: expect repo format version 1 for SHA-256brian m. carlson, May 13, 2020
  81. 26/44 t5302: modernize test formattingbrian m. carlson, May 13, 2020
  82. 29/44 Documentation/technical: document object-format for protocol v2brian m. carlson, May 13, 2020
  83. 17/44 transport-helper: implement object-format extensionsbrian m. carlson, May 13, 2020
  84. 00/44 SHA-256 part 2/3: protocol functionalitybrian m. carlson, May 25, 2020
  85. 01/44 t1050: match object ID paths in a hash-insensitive waybrian m. carlson, May 25, 2020
  86. 02/44 Documentation: document v1 protocol object-format capabilitybrian m. carlson, May 25, 2020
  87. 04/44 wrapper: add function to compare strings with different NUL terminationbrian m. carlson, May 25, 2020
  88. 03/44 connect: have ref processing code take struct packet_readerbrian m. carlson, May 25, 2020
  89. 14/44 connect: detect algorithm when fetching refsbrian m. carlson, May 25, 2020
  90. 13/44 fetch-pack: detect when the server doesn't support our hashbrian m. carlson, May 25, 2020
  91. 12/44 connect: make parse_feature_value externbrian m. carlson, May 25, 2020
  92. 16/44 docs: update remote helper docs for object-format extensionsbrian m. carlson, May 25, 2020
  93. 11/44 send-pack: detect when the server doesn't support our hashbrian m. carlson, May 25, 2020
  94. 27/44 t1302: expect repo format version 1 for SHA-256brian m. carlson, May 25, 2020
  95. 25/44 t5302: modernize test formattingbrian m. carlson, May 25, 2020
  96. 32/44 t5500: make hash independentbrian m. carlson, May 25, 2020
  97. 34/44 remote-curl: detect algorithm for dumb HTTP by sizebrian m. carlson, May 25, 2020
  98. 35/44 builtin/index-pack: add option to specify hash algorithmbrian m. carlson, May 25, 2020
  99. 43/44 bundle: detect hash algorithm when reading refsbrian m. carlson, May 25, 2020
  100. 39/44 t5702: offer an object-format capability in the testbrian m. carlson, May 25, 2020
  101. 44/44 remote-testgit: adapt for object-formatbrian m. carlson, May 25, 2020
  102. 38/44 t/helper: initialize the repository for test-sha1-arraybrian m. carlson, May 25, 2020
  103. 37/44 remote-curl: avoid truncating refs with ls-remotebrian m. carlson, May 25, 2020
  104. 36/44 t1050: pass algorithm to index-pack when outside repobrian m. carlson, May 25, 2020
  105. 33/44 builtin/ls-remote: initialize repository based on fetchbrian m. carlson, May 25, 2020
  106. 30/44 connect: parse v2 refs with correct hash algorithmbrian m. carlson, May 25, 2020
  107. 05/44 remote: advertise the object-format capability on the server sidebrian m. carlson, May 25, 2020
  108. 23/44 t3200: mark assertion with SHA1 prerequisitebrian m. carlson, May 25, 2020
  109. 29/44 connect: pass full packet reader when parsing v2 refsbrian m. carlson, May 25, 2020
  110. 09/44 transport: add a hash algorithm memberbrian m. carlson, May 25, 2020
  111. 10/44 connect: add function to detect supported v1 hash functionsbrian m. carlson, May 25, 2020
  112. 07/44 connect: add function to fetch value of a v2 server capabilitybrian m. carlson, May 25, 2020
  113. 08/44 pkt-line: add a member for hash algorithmbrian m. carlson, May 25, 2020
  114. 06/44 connect: add function to parse multiple v1 capability valuesbrian m. carlson, May 25, 2020
  115. 21/44 fetch-pack: parse and advertise the object-format capabilitybrian m. carlson, May 25, 2020
  116. 19/44 builtin/clone: initialize hash algorithm properlybrian m. carlson, May 25, 2020
  117. 20/44 t5562: pass object-format in synthesized test databrian m. carlson, May 25, 2020
  118. 40/44 t5703: use object-format serve optionbrian m. carlson, May 25, 2020
  119. 41/44 t5704: send object-format capability with SHA-256brian m. carlson, May 25, 2020
  120. 42/44 t5300: pass --object-format to git index-packbrian m. carlson, May 25, 2020
  121. 31/44 serve: advertise object-format capability for protocol v2brian m. carlson, May 25, 2020
  122. 28/44 Documentation/technical: document object-format for protocol v2brian m. carlson, May 25, 2020
  123. 26/44 builtin/show-index: provide options to determine hash algobrian m. carlson, May 25, 2020
  124. 24/44 packfile: compute and use the index CRC offsetbrian m. carlson, May 25, 2020
  125. 22/44 setup: set the_repository's hash algo when checking formatbrian m. carlson, May 25, 2020
  126. 15/44 builtin/receive-pack: detect when the server doesn't support our hashbrian m. carlson, May 25, 2020
  127. 18/44 remote-curl: implement object-format extensionsbrian m. carlson, May 25, 2020
  128. 17/44 transport-helper: implement object-format extensionsbrian m. carlson, May 25, 2020
  129. 00/44 SHA-256 part 2/3: protocol functionalitybrian m. carlson, Jun 19, 2020
  130. 02/44 Documentation: document v1 protocol object-format capabilitybrian m. carlson, Jun 19, 2020
  131. 03/44 connect: have ref processing code take struct packet_readerbrian m. carlson, Jun 19, 2020
  132. 10/44 connect: add function to detect supported v1 hash functionsbrian m. carlson, Jun 19, 2020
  133. 17/44 transport-helper: implement object-format extensionsbrian m. carlson, Jun 19, 2020
  134. 22/44 setup: set the_repository's hash algo when checking formatbrian m. carlson, Jun 19, 2020
  135. 21/44 fetch-pack: parse and advertise the object-format capabilitybrian m. carlson, Jun 19, 2020
  136. 26/44 builtin/show-index: provide options to determine hash algobrian m. carlson, Jun 19, 2020
  137. 25/44 t5302: modernize test formattingbrian m. carlson, Jun 19, 2020
  138. 30/44 connect: parse v2 refs with correct hash algorithmbrian m. carlson, Jun 19, 2020
  139. 34/44 remote-curl: detect algorithm for dumb HTTP by sizebrian m. carlson, Jun 19, 2020
  140. 29/44 connect: pass full packet reader when parsing v2 refsbrian m. carlson, Jun 19, 2020
  141. 31/44 serve: advertise object-format capability for protocol v2brian m. carlson, Jun 19, 2020
  142. 37/44 remote-curl: avoid truncating refs with ls-remotebrian m. carlson, Jun 19, 2020
  143. 33/44 builtin/ls-remote: initialize repository based on fetchbrian m. carlson, Jun 19, 2020
  144. 28/44 Documentation/technical: document object-format for protocol v2brian m. carlson, Jun 19, 2020
  145. 39/44 t5702: offer an object-format capability in the testbrian m. carlson, Jun 19, 2020
  146. 41/44 t5704: send object-format capability with SHA-256brian m. carlson, Jun 19, 2020
  147. 01/44 t1050: match object ID paths in a hash-insensitive waybrian m. carlson, Jun 19, 2020
  148. 32/44 t5500: make hash independentbrian m. carlson, Jun 19, 2020
  149. 16/44 docs: update remote helper docs for object-format extensionsbrian m. carlson, Jun 19, 2020
  150. 09/44 transport: add a hash algorithm memberbrian m. carlson, Jun 19, 2020
  151. 14/44 connect: detect algorithm when fetching refsbrian m. carlson, Jun 19, 2020
  152. 07/44 connect: add function to fetch value of a v2 server capabilitybrian m. carlson, Jun 19, 2020
  153. 05/44 remote: advertise the object-format capability on the server sidebrian m. carlson, Jun 19, 2020
  154. 06/44 connect: add function to parse multiple v1 capability valuesbrian m. carlson, Jun 19, 2020
  155. 44/44 remote-testgit: adapt for object-formatbrian m. carlson, Jun 19, 2020
  156. 43/44 bundle: detect hash algorithm when reading refsbrian m. carlson, Jun 19, 2020
  157. 04/44 wrapper: add function to compare strings with different NUL terminationbrian m. carlson, Jun 19, 2020
  158. 35/44 builtin/index-pack: add option to specify hash algorithmbrian m. carlson, Jun 19, 2020
  159. 27/44 t1302: expect repo format version 1 for SHA-256brian m. carlson, Jun 19, 2020
  160. 40/44 t5703: use object-format serve optionbrian m. carlson, Jun 19, 2020
  161. 38/44 t/helper: initialize the repository for test-sha1-arraybrian m. carlson, Jun 19, 2020
  162. 36/44 t1050: pass algorithm to index-pack when outside repobrian m. carlson, Jun 19, 2020
  163. 42/44 t5300: pass --object-format to git index-packbrian m. carlson, Jun 19, 2020
  164. 15/44 builtin/receive-pack: detect when the server doesn't support our hashbrian m. carlson, Jun 19, 2020
  165. 18/44 remote-curl: implement object-format extensionsbrian m. carlson, Jun 19, 2020
  166. 23/44 t3200: mark assertion with SHA1 prerequisitebrian m. carlson, Jun 19, 2020
  167. 24/44 packfile: compute and use the index CRC offsetbrian m. carlson, Jun 19, 2020
  168. 11/44 send-pack: detect when the server doesn't support our hashbrian m. carlson, Jun 19, 2020
  169. 20/44 t5562: pass object-format in synthesized test databrian m. carlson, Jun 19, 2020
  170. 19/44 builtin/clone: initialize hash algorithm properlybrian m. carlson, Jun 19, 2020
  171. 13/44 fetch-pack: detect when the server doesn't support our hashbrian m. carlson, Jun 19, 2020
  172. 08/44 pkt-line: add a member for hash algorithmbrian m. carlson, Jun 19, 2020
  173. 12/44 connect: make parse_feature_value externbrian m. carlson, Jun 19, 2020
  174. Junio C HamanoJun 19, 2020
  175. brian m. carlsonJun 20, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.