git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 2/5] use size_t to store pack .idx byte offsets

From
Jeff King <peff@peff.net>
Date
Nov 13, 2020, 05:07 UTC
Message-ID
<20201113050701.GB744691@coredump.intra.peff.net>
In-Reply-To
<20201113050631.GA744608@coredump.intra.peff.net>

We sometimes store the offset into a pack .idx file as an "unsigned long", but the mmap'd size of a pack .idx file can exceed 4GB. This is sufficient on LP64 systems like Linux, but will be too small on LLP64 systems like Windows, where "unsigned long" is still only 32 bits. Let's use size_t, which is a better type for an offset into a memory buffer.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/pack-redundant.c | 6 +++---
 packfile.c               | 4 ++--
 2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index 178e3409b7..3e70f2a4c1 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -236,7 +236,7 @@ static struct pack_list * pack_list_difference(const struct pack_list *A,
 
 static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)
 {
-	unsigned long p1_off = 0, p2_off = 0, p1_step, p2_step;
+	size_t p1_off = 0, p2_off = 0, p1_step, p2_step;
 	const unsigned char *p1_base, *p2_base;
 	struct llist_item *p1_hint = NULL, *p2_hint = NULL;
 	const unsigned int hashsz = the_hash_algo->rawsz;
@@ -280,7 +280,7 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)
 static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)
 {
 	size_t ret = 0;
-	unsigned long p1_off = 0, p2_off = 0, p1_step, p2_step;
+	size_t p1_off = 0, p2_off = 0, p1_step, p2_step;
 	const unsigned char *p1_base, *p2_base;
 	const unsigned int hashsz = the_hash_algo->rawsz;
 
@@ -499,7 +499,7 @@ static void scan_alt_odb_packs(void)
 static struct pack_list * add_pack(struct packed_git *p)
 {
 	struct pack_list l;
-	unsigned long off = 0, step;
+	size_t off = 0, step;
 	const unsigned char *base;
 
 	if (!p->pack_local && !(alt_odb || verbose))
diff --git a/packfile.c b/packfile.c
index a72c2a261f..63fe9ee8be 100644
--- a/packfile.c
+++ b/packfile.c
@@ -164,8 +164,8 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
 		 * variable sized table containing 8-byte entries
 		 * for offsets larger than 2^31.
 		 */
-		unsigned long min_size = 8 + 4*256 + (size_t)nr*(hashsz + 4 + 4) + hashsz + hashsz;
-		unsigned long max_size = min_size;
+		size_t min_size = 8 + 4*256 + (size_t)nr*(hashsz + 4 + 4) + hashsz + hashsz;
+		size_t max_size = min_size;
 		if (nr)
 			max_size += ((size_t)nr - 1)*8;
 		if (idx_size < min_size || idx_size > max_size)
-- 
2.29.2.705.g306f91dc4e
Previous: Jeff KingNext: Jeff King
Message 3 of 19 in “handling 4GB .idx files”
  1. 0/5 handling 4GB .idx filesJeff King, Nov 13, 2020
  2. 1/5 compute pack .idx byte offsets using size_tJeff King, Nov 13, 2020
  3. 2/5 use size_t to store pack .idx byte offsetsJeff King, Nov 13, 2020
  4. 3/5 fsck: correctly compute checksums on idx files larger than 4GBJeff King, Nov 13, 2020
  5. 4/5 block-sha1: take a size_t length parameterJeff King, Nov 13, 2020
  6. 5/5 packfile: detect overflow in .idx file size checksJeff King, Nov 13, 2020
  7. Johannes SchindelinNov 13, 2020
  8. Thomas BraunNov 15, 2020
  9. Jeff KingNov 16, 2020
  10. Derrick StoleeNov 16, 2020
  11. Jeff KingNov 16, 2020
  12. Thomas BraunNov 30, 2020
  13. Jeff KingDec 1, 2020
  14. t7900's new expensive testJeff King, Dec 1, 2020
  15. Derrick StoleeDec 1, 2020
  16. t7900: speed up expensive testJeff King, Dec 2, 2020
  17. Derrick StoleeDec 3, 2020
  18. Taylor BlauDec 1, 2020
  19. Jeff KingDec 2, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.