git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFH] zlib gurus out there?

From
Junio C Hamano <junkio@cox.net>
Date
Mar 7, 2006, 23:48 UTC
Message-ID
<7vzmk1izpa.fsf_-_@assigned-by-dhcp.cox.net>
In-Reply-To
<Pine.LNX.4.64.0602250012230.31162@localhost.localdomain>

I've been staring at reusing existing data while packing, and this occurred to me...

During packing, suppose that we chose to store an object in base form, undeltified. And also suppose we have that object loose in .git/objects/??/ directory. We already have it in deflated form, but with its own header. I started wondering if we can somehow reuse this.

A short object format brush-up lesson is in order here.  
* An undeltified object in a pack is represented like this:
 (1) the header is a dense variable size binary data, that
     encodes type and inflated length;
 (2) deflated data immediately follows the header.
* On the other hand, a loose object is represented like this:
 (1) the header looks like sprintf("%s %lu%c", type, len, 0);
 (2) concatenate the data to the header;
 (3) SHA1 checksum of the above becomes the object name.
 (4) deflate the header and data using the same z_stream, in two
     steps, like this (sha1_file.c::write_sha1_file):
	/* Compress it */
	stream.next_out = compressed;
	stream.avail_out = size;
	/* First header.. */
	stream.next_in = hdr;
	stream.avail_in = hdrlen;
	while (deflate(&stream, 0) == Z_OK)
		/* nothing */;
	/* Then the data itself.. */
	stream.next_in = buf;
	stream.avail_in = len;
	while (deflate(&stream, Z_FINISH) == Z_OK)
		/* nothing */;
	deflateEnd(&stream);
	size = stream.total_out;

So I thought... if we cause a full flush after the header part, I can find the flush boundaries from a loose object file and copy the rest into a packfile I am generating, after placing the binary encoded header. If this works, we do not have to inflate loose object to read it and deflate it to store that in the pack. We will get a better packing as well, since we deflate loose objects with Z_BEST_COMPRESSION, while packs are done with Z_DEFAULT_COMPRESSION. While pack-objects read from a loose object, if we can detect that there is no full flush after the header, we would do the traditional inflate-deflate cycle, so this would be backward compatible.

However, I am stuck with the first step, which is to do a full flush after the header. An obvious change to the code quoted above writes out a corrupt object:

	/* First header.. */
	stream.next_in = hdr;
	stream.avail_in = hdrlen;
-	while (deflate(&stream, 0) == Z_OK)
+	while (deflate(&stream, Z_FULL_FLUSH) == Z_OK)
		/* nothing */;

git-fsck-objects complains that sha1 does not match. It appears that the sha1_file.c::unpack_sha1_rest() somehow barfs upon seeing the full flush, but I haven't dug into it yet.

Would anybody with more experience with zlib want to help?
Previous: Nicolas PitreNext: Linus Torvalds
Message 25 of 35 in “diff-delta: produce optimal pack data”
  1. diff-delta: produce optimal pack dataNicolas Pitre, Feb 22, 2006
  2. Junio C HamanoFeb 24, 2006
  3. Nicolas PitreFeb 24, 2006
  4. Junio C HamanoFeb 24, 2006
  5. Carl BaldwinFeb 24, 2006
  6. Nicolas PitreFeb 24, 2006
  7. Carl BaldwinFeb 24, 2006
  8. Nicolas PitreFeb 24, 2006
  9. Carl BaldwinFeb 24, 2006
  10. Nicolas PitreFeb 24, 2006
  11. Carl BaldwinFeb 24, 2006
  12. Nicolas PitreFeb 24, 2006
  13. Carl BaldwinFeb 24, 2006
  14. Nicolas PitreFeb 25, 2006
  15. Linus TorvaldsFeb 24, 2006
  16. Nicolas PitreFeb 24, 2006
  17. Junio C HamanoFeb 24, 2006
  18. Nicolas PitreFeb 24, 2006
  19. Nicolas PitreFeb 24, 2006
  20. Linus TorvaldsFeb 25, 2006
  21. Nicolas PitreFeb 25, 2006
  22. Linus TorvaldsFeb 25, 2006
  23. Nicolas PitreFeb 25, 2006
  24. Nicolas PitreFeb 25, 2006
  25. [RFH] zlib gurus out there?Junio C Hamano, Mar 7, 2006
  26. Linus TorvaldsMar 8, 2006
  27. Junio C HamanoMar 8, 2006
  28. Linus TorvaldsMar 8, 2006
  29. Johannes SchindelinMar 8, 2006
  30. write_sha1_file(): Perform Z_FULL_FLUSH between header and dataSergey Vlasov, Mar 8, 2006
  31. Junio C HamanoMar 8, 2006
  32. Sergey VlasovMar 8, 2006
  33. Linus TorvaldsFeb 25, 2006
  34. Carl BaldwinFeb 24, 2006
  35. Nicolas PitreFeb 24, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.