git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 3/3] Enable the new binary header format for unpacked objects

From
Linus Torvalds <torvalds@osdl.org>
Date
Jul 11, 2006, 17:16 UTC
Message-ID
<Pine.LNX.4.64.0607111012110.5623@g5.osdl.org>
In-Reply-To
<Pine.LNX.4.64.0607111004360.5623@g5.osdl.org>
Enable the new binary header format for unpacked objects

This makes unpacked objects use the same header encoding as the packed objects do, which should eventually allow us to be able to re-use the object data directly when creating pack-files (rather than having to decode and re-encode the data when inserting it in a pack).

It's enabled by default, but can be disabled with
	[core]
		BinaryHeaders = false
in the config file.
We can read both formats, of course, so you can have a mixed archive.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---

This should _not_ be applied to the main git sources, at least not with the default being "use_binary_headers = 1".

But if you change that initial assignment to 0, this should be reasonably good.

Not extensively tested, of course. It fails t9102-git-svn-deep-rmdir.sh for me for some reason, I didn't really look at it yet, since this whole thing is more for Carl Baldwin to play with right now.

 Documentation/config.txt |    5 ++++
 cache.h                  |    1 +
 config.c                 |    5 ++++
 environment.c            |    1 +
 sha1_file.c              |   65 ++++++++++++++++++++++++++++++++++++++++------
 5 files changed, 69 insertions(+), 8 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 0b434c1..bc95416 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -97,6 +97,11 @@ core.compression::
 	compression, and 1..9 are various speed/size tradeoffs, 9 being
 	slowest.
 
+core.binaryheaders::
+	A boolean, that if set to false, disables the use of the
+	new-style binary objects headers that share the same format with
+	the headers in a pack file.
+
 alias.*::
 	Command aliases for the gitlink:git[1] command wrapper - e.g.
 	after defining "alias.last = cat-file commit HEAD", the invocation
diff --git a/cache.h b/cache.h
index d433d46..756d89f 100644
--- a/cache.h
+++ b/cache.h
@@ -176,6 +176,7 @@ extern int commit_lock_file(struct lock_
 extern void rollback_lock_file(struct lock_file *);
 
 /* Environment bits from configuration mechanism */
+extern int use_binary_headers;
 extern int trust_executable_bit;
 extern int assume_unchanged;
 extern int prefer_symlink_refs;
diff --git a/config.c b/config.c
index 8445f7d..2497447 100644
--- a/config.c
+++ b/config.c
@@ -289,6 +289,11 @@ int git_default_config(const char *var, 
 		return 0;
 	}
 
+	if (!strcmp(var, "core.binaryheaders")) {
+		use_binary_headers = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "user.name")) {
 		strlcpy(git_default_name, value, sizeof(git_default_name));
 		return 0;
diff --git a/environment.c b/environment.c
index 43823ff..340214d 100644
--- a/environment.c
+++ b/environment.c
@@ -11,6 +11,7 @@ #include "cache.h"
 
 char git_default_email[MAX_GITNAME];
 char git_default_name[MAX_GITNAME];
+int use_binary_headers = 1;
 int trust_executable_bit = 1;
 int assume_unchanged = 0;
 int prefer_symlink_refs = 0;
diff --git a/sha1_file.c b/sha1_file.c
index ca5f0c0..700f455 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -699,11 +699,14 @@ static int unpack_sha1_header(z_stream *
 
 static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size, unsigned int hdrlen)
 {
-	int bytes = hdrlen;
+	unsigned long bytes = hdrlen, n;
 	unsigned char *buf = xmalloc(1+size);
 
-	memcpy(buf, (char *) buffer + bytes, stream->total_out - bytes);
-	bytes = stream->total_out - bytes;
+	n = stream->total_out - bytes;
+	if (n > size)
+		n = size;
+	memcpy(buf, (char *) buffer + bytes, n);
+	bytes = n;
 	if (bytes < size) {
 		stream->next_out = buf + bytes;
 		stream->avail_out = size - bytes;
@@ -1240,7 +1243,7 @@ struct packed_git *find_sha1_pack(const 
 
 int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep)
 {
-	int status, hdrlen;
+	int status;
 	unsigned long mapsize, size;
 	void *map;
 	z_stream stream;
@@ -1349,24 +1352,70 @@ void *read_object_with_reference(const u
 	}
 }
 
+static int write_binary_header(unsigned char *hdr, enum object_type type, unsigned long len)
+{
+	int hdr_len;
+	unsigned char c;
+
+	c = (type << 4) | (len & 15);
+	len >>= 4;
+	hdr_len = 1;
+	while (len) {
+		*hdr++ = c;
+		hdr_len++;
+		c = (len & 0x7f);
+		len >>= 7;
+	}
+	*hdr = c | 0x80;
+	return hdr_len;
+}
+
+static int generate_binary_header(unsigned char *hdr, const char *type, unsigned long len)
+{
+	int obj_type;
+
+	if (!strcmp(type, blob_type))
+		obj_type = OBJ_BLOB;
+	else if (!strcmp(type, tree_type))
+		obj_type = OBJ_TREE;
+	else if (!strcmp(type, commit_type))
+		obj_type = OBJ_COMMIT;
+	else if (!strcmp(type, tag_type))
+		obj_type = OBJ_TAG;
+	else
+		die("trying to generate bogus object of type '%s'", type);
+	return write_binary_header(hdr, obj_type, len);
+}
+
 char *write_sha1_file_prepare(void *buf,
 			      unsigned long len,
 			      const char *type,
 			      unsigned char *sha1,
 			      unsigned char *hdr,
-			      int *hdrlen)
+			      int *hdrlenp)
 {
 	SHA_CTX c;
+	int hdr_len;
 
-	/* Generate the header */
-	*hdrlen = sprintf((char *)hdr, "%s %lu", type, len)+1;
+	/*
+	 * Generate the header.
+	 *
+	 * NOTE! Regardless of whether we end up using the ASCII
+	 * or binary header, we always generate the SHA1 of the
+	 * file as if we had the ASCII header.
+	 */
+	hdr_len = sprintf((char *)hdr, "%s %lu", type, len)+1;
 
 	/* Sha1.. */
 	SHA1_Init(&c);
-	SHA1_Update(&c, hdr, *hdrlen);
+	SHA1_Update(&c, hdr, hdr_len);
 	SHA1_Update(&c, buf, len);
 	SHA1_Final(sha1, &c);
 
+	if (use_binary_headers)
+		hdr_len = generate_binary_header(hdr, type, len);
+	*hdrlenp = hdr_len;
+
 	return sha1_file_name(sha1);
 }
Previous: Junio C Hamano
Message 37 of 37 in “Revisiting large binary files issue.”
  1. Carl BaldwinJul 10, 2006
  2. Junio C HamanoJul 10, 2006
  3. Peter BaumannJul 11, 2006
  4. Linus TorvaldsJul 10, 2006
  5. [RFC]: Pack-file object format for individual objects (Was: Revisiting large binary files issue.)sf, Jul 11, 2006
  6. Linus TorvaldsJul 11, 2006
  7. sfJul 11, 2006
  8. Linus TorvaldsJul 11, 2006
  9. Linus TorvaldsJul 11, 2006
  10. Carl BaldwinJul 11, 2006
  11. Linus TorvaldsJul 11, 2006
  12. 1/3 Make the unpacked object header functions static to sha1_file.cLinus Torvalds, Jul 11, 2006
  13. 2/3 sha1_file: add the ability to parse objects in "pack file format"Linus Torvalds, Jul 11, 2006
  14. Johannes SchindelinJul 11, 2006
  15. Linus TorvaldsJul 11, 2006
  16. Johannes SchindelinJul 11, 2006
  17. Linus TorvaldsJul 11, 2006
  18. Johannes SchindelinJul 11, 2006
  19. Junio C HamanoJul 11, 2006
  20. sfJul 11, 2006
  21. Linus TorvaldsJul 11, 2006
  22. sfJul 11, 2006
  23. Junio C HamanoJul 11, 2006
  24. Linus TorvaldsJul 12, 2006
  25. Johannes SchindelinJul 12, 2006
  26. Linus TorvaldsJul 12, 2006
  27. Linus TorvaldsJul 12, 2006
  28. Junio C HamanoJul 12, 2006
  29. Linus TorvaldsJul 12, 2006
  30. Junio C HamanoJul 12, 2006
  31. Linus TorvaldsJul 12, 2006
  32. Peter BaumannJul 12, 2006
  33. Junio C HamanoJul 12, 2006
  34. Peter BaumannJul 12, 2006
  35. Linus TorvaldsJul 12, 2006
  36. Junio C HamanoJul 12, 2006
  37. 3/3 Enable the new binary header format for unpacked objectsLinus Torvalds, Jul 11, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.