git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 09/12] streaming_write_entry(): support files with holes

From
Junio C Hamano <gitster@pobox.com>
Date
May 21, 2011, 06:56 UTC
Message-ID
<1305960995-25738-10-git-send-email-gitster@pobox.com>
In-Reply-To
<1305960995-25738-1-git-send-email-gitster@pobox.com>

One typical use of a large binary file is to hold a sparse on-disk hash table with a lot of holes. Help preserving the holes with lseek().

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 entry.c |   21 +++++++++++++++++++--
 1 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/entry.c b/entry.c
index da37d01..e2dc16c 100644
--- a/entry.c
+++ b/entry.c
@@ -123,6 +123,7 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,
 	enum object_type type;
 	unsigned long sz;
 	int result = -1;
+	ssize_t kept = 0;
 	int fd = -1;
 
 	st = open_istream(ce->sha1, &type, &sz);
@@ -136,18 +137,34 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,
 		goto close_and_exit;
 
 	for (;;) {
-		char buf[10240];
-		ssize_t wrote;
+		char buf[1024 * 16];
+		ssize_t wrote, holeto;
 		ssize_t readlen = read_istream(st, buf, sizeof(buf));
 
 		if (!readlen)
 			break;
+		if (sizeof(buf) == readlen) {
+			for (holeto = 0; holeto < readlen; holeto++)
+				if (buf[holeto])
+					break;
+			if (readlen == holeto) {
+				kept += holeto;
+				continue;
+			}
+		}
 
+		if (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)
+			goto close_and_exit;
+		else
+			kept = 0;
 		wrote = write_in_full(fd, buf, readlen);
 
 		if (wrote != readlen)
 			goto close_and_exit;
 	}
+	if (kept && (lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1 ||
+		     write(fd, "", 1) != 1))
+		goto close_and_exit;
 	*fstat_done = fstat_output(fd, state, statbuf);
 
 close_and_exit:
-- 
1.7.5.2.369.g8fc017
Previous: Junio C HamanoNext: Junio C Hamano
Message 10 of 13 in “writing out a huge blob to working tree”
  1. 00/12 writing out a huge blob to working treeJunio C Hamano, May 21, 2011
  2. 01/12 packed_object_info_detail(): do not return a stringJunio C Hamano, May 21, 2011
  3. 02/12 sha1_object_info_extended(): expose a bit more infoJunio C Hamano, May 21, 2011
  4. 03/12 sha1_object_info_extended(): hint about objects in delta-base cacheJunio C Hamano, May 21, 2011
  5. 04/12 unpack_object_header(): make it publicJunio C Hamano, May 21, 2011
  6. 05/12 write_entry(): separate two helper functions outJunio C Hamano, May 21, 2011
  7. 06/12 streaming: a new API to read from the object storeJunio C Hamano, May 21, 2011
  8. 07/12 streaming_write_entry(): use streaming API in write_entry()Junio C Hamano, May 21, 2011
  9. 08/12 convert: CRLF_INPUT is a no-op in the output codepathJunio C Hamano, May 21, 2011
  10. 09/12 streaming_write_entry(): support files with holesJunio C Hamano, May 21, 2011
  11. 10/12 streaming: read non-delta incrementally from a packJunio C Hamano, May 21, 2011
  12. 11/12 sha1_file.c: expose helpers to read loose objectsJunio C Hamano, May 21, 2011
  13. 12/12 streaming: read loose objects incrementallyJunio C Hamano, May 21, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.