{"thread":{"id":"1113","subject":"zlib experts, please..","startedAt":"2005-07-05T23:57:55Z","lastAt":"2005-07-06T00:04:28Z","messageCount":2,"participants":["Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"5681","messageId":"Pine.LNX.4.58.0507051644100.3570@g5.osdl.org","threadId":"1113","inReplyTo":null,"subject":"zlib experts, please..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-05T23:57:55Z","receivedAt":"2005-07-05T23:57:55Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nI just hit an interesting pack failure because of how git (mis-)uses zlib, \nand I'm wondering what to do about it.\n\nIn particular, the \"git-unpack-objects\" code gets a data stream, and only\nknows the _unpacked_ size of each object, because writing packed size is\nextremely inconvenient in many ways (let me count the ways.. At pack time,\nwe want to fill in the size field before we've even packed things, and at\nunpack time, we really don't care about the packed size, but we _do_ care\nabout the unpacked size in order to be able to allocate the right sized\nallocation for the result).\n\nHowever, it turns out that there's a silly special case: a zero-sized \n\"blob\" object will ne encoded as a single byte \"0x30\" followed by the \n\"packed representation of empty\".\n\nNow, you'd expect the packed representation of empty to be empty, but \nthat's not apparently what zlib does. It actually seems to pack zero bytes \nas 8 bytes of \"78 9c 03 00 00 00 00 01\". Which is fine, I don't care, with \ngit this will literally happen for only one single object, so it's not \nlike I care about the expansion.\n\nBut what I care about is that when git-unpack-objects sees that it wants a \nzero-byte object, and asks zlib to unpack it, zlib will not actually use \nthe bytes it wrote - it will just say \"oh, you wanted zero bytes, here's \nzero bytes\". Which means that the stream handling gets upset.\n\nNow, I can easily fix this by just teaching the packing code that it \nshould pack the zero-byte object as zero bytes, and not let zlib mess it \nup. In fact, I've done exactly that. However, now I worry that there's \nsome other case where zlib uncompression doesn't eat everything that zlib \ncompression generated. I've not found it, and I think a zero sized case is \nspecial (it's kind of like a \"break\" event), but this is yet another cry \nfor zlib expertise in case somebody knows...\n\n\t\tLinus\n"},{"id":"5682","messageId":"Pine.LNX.4.58.0507051701101.3570@g5.osdl.org","threadId":"1113","inReplyTo":"Pine.LNX.4.58.0507051644100.3570@g5.osdl.org","subject":"Re: zlib experts, please..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-06T00:04:28Z","receivedAt":"2005-07-06T00:04:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 5 Jul 2005, Linus Torvalds wrote:\n> \n> But what I care about is that when git-unpack-objects sees that it wants a \n> zero-byte object, and asks zlib to unpack it, zlib will not actually use \n> the bytes it wrote - it will just say \"oh, you wanted zero bytes, here's \n> zero bytes\". Which means that the stream handling gets upset.\n\nActually, never mind. zlib does exactly the right thing, and eats the 8 \nbytes of header that it wrote for \"empty\". The real culprit was that \ngit-unpack-objects incorrectly made that zero a special case on reading, \nbut not writing.\n\nSo just ignore me. zlib actually does what I expected it to do, even for \nthe zero case.\n\n\t\tLinus\n"}]}