git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 2/2] Find size of SHA1 object without inflating everything.

From
Junio C Hamano <junkio@cox.net>
Date
Jun 2, 2005, 22:20 UTC
Message-ID
<7voeao1juh.fsf_-_@assigned-by-dhcp.cox.net>
In-Reply-To
<Pine.LNX.4.63.0506021733520.17354@localhost.localdomain>

This adds sha1_file_size() helper function and uses it in the rename/copy similarity estimator. The helper function handles deltified object as well.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

*** Thanks and credits goes to Nico for suggesting not to *** use unpack_sha1_rest().

 cache.h     |    1 +
 diff.c      |   11 ++++++-----
 sha1_file.c |   60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+), 5 deletions(-)
diff --git a/cache.h b/cache.h
--- a/cache.h
+++ b/cache.h
@@ -154,6 +154,7 @@ extern void * map_sha1_file(const unsign
 extern int unpack_sha1_header(z_stream *stream, void *map, unsigned long mapsize, void *buffer, unsigned long size);
 extern int parse_sha1_header(char *hdr, char *type, unsigned long *sizep);
 extern int sha1_delta_base(const unsigned char *, unsigned char *);
+extern int sha1_file_size(const unsigned char *, unsigned long *);
 extern void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size);
 extern void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size);
 extern int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *return_sha1);
diff --git a/diff.c b/diff.c
--- a/diff.c
+++ b/diff.c
@@ -333,7 +333,6 @@ int diff_populate_filespec(struct diff_f
 		close(fd);
 	}
 	else {
-		/* We cannot do size only for SHA1 blobs */
 		char type[20];
 		struct sha1_size_cache *e;
 
@@ -343,11 +342,13 @@ int diff_populate_filespec(struct diff_f
 				s->size = e->size;
 				return 0;
 			}
+			if (!sha1_file_size(s->sha1, &s->size))
+				locate_size_cache(s->sha1, s->size);
+		}
+		else {
+			s->data = read_sha1_file(s->sha1, type, &s->size);
+			s->should_free = 1;
 		}
-		s->data = read_sha1_file(s->sha1, type, &s->size);
-		s->should_free = 1;
-		if (s->data && size_only)
-			locate_size_cache(s->sha1, s->size);
 	}
 	return 0;
 }
diff --git a/sha1_file.c b/sha1_file.c
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -432,6 +432,66 @@ int sha1_delta_base(const unsigned char 
 	return ret;
 }
 
+int sha1_file_size(const unsigned char *sha1, unsigned long *sizep)
+{
+	int ret, status;
+	unsigned long mapsize, size;
+	void *map;
+	z_stream stream;
+	char hdr[64], type[20];
+	const unsigned char *data;
+	unsigned char cmd;
+	int i;
+
+	map = map_sha1_file(sha1, &mapsize);
+	if (!map)
+		return -1;
+	ret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));
+	status = -1;
+	if (ret < Z_OK || parse_sha1_header(hdr, type, &size) < 0)
+		goto out;
+	if (strcmp(type, "delta")) {
+		*sizep = size;
+		status = 0;
+		goto out;
+	}
+
+	/* We are dealing with a delta object.  Inflated, the first
+	 * 20 bytes hold the base object SHA1, and delta data follows
+	 * immediately after it.
+	 *
+	 * The initial part of the delta starts at delta_data_head +
+	 * 20.  Borrow code from patch-delta to read the result size.
+	 */
+	data = hdr + strlen(hdr) + 1 + 20;
+
+	/* Skip over the source size; we are not interested in
+	 * it and we cannot verify it because we do not want
+	 * to read the base object.
+	 */
+	cmd = *data++;
+	while (cmd) {
+		if (cmd & 1)
+			data++;
+		cmd >>= 1;
+	}
+	/* Read the result size */
+	size = i = 0;
+	cmd = *data++;
+	while (cmd) {
+		if (cmd & 1)
+			size |= *data++ << i;
+		i += 8;
+		cmd >>= 1;
+	}
+	*sizep = size;
+	status = 0;
+ out:
+	inflateEnd(&stream);
+	munmap(map, mapsize);
+	return status;
+}
+
 void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size)
 {
 	unsigned long mapsize;
------------
Previous: Linus TorvaldsNext: Junio C Hamano
Message 9 of 20 in “diff: mode bits fixes”
  1. diff: mode bits fixesJunio C Hamano, Jun 1, 2005
  2. Handle deltified object correctly in git-*-pull family.Junio C Hamano, Jun 2, 2005
  3. Linus TorvaldsJun 2, 2005
  4. Handle deltified object correctly in git-*-pull family.Junio C Hamano, Jun 2, 2005
  5. Nicolas PitreJun 2, 2005
  6. Nicolas PitreJun 2, 2005
  7. 1/2 Handle deltified object correctly in git-*-pull family.Junio C Hamano, Jun 2, 2005
  8. Linus TorvaldsJun 2, 2005
  9. 2/2 Find size of SHA1 object without inflating everything.Junio C Hamano, Jun 2, 2005
  10. Find size of SHA1 object without inflating everything.Junio C Hamano, Jun 2, 2005
  11. Linus TorvaldsJun 2, 2005
  12. Junio C HamanoJun 2, 2005
  13. McMullan, JasonJun 2, 2005
  14. Junio C HamanoJun 2, 2005
  15. Use correct U*MAX.Junio C Hamano, Jun 2, 2005
  16. Petr BaudisJun 3, 2005
  17. Junio C HamanoJun 3, 2005
  18. Petr BaudisJun 4, 2005
  19. Junio C HamanoJun 4, 2005
  20. Find size of SHA1 object without inflating everything.Junio C Hamano, Jun 2, 2005

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.