{"thread":{"id":"60773","subject":"How OBJ_REF_DELTA pack file size calculated","startedAt":"2024-01-20T04:28:26Z","lastAt":"2024-01-20T18:26:32Z","messageCount":2,"participants":["Farhan Khan","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"487131","messageId":"afea6dc9-e557-4730-abe6-00947f77be06@app.fastmail.com","threadId":"60773","inReplyTo":null,"subject":"How OBJ_REF_DELTA pack file size calculated","fromName":"Farhan Khan","fromEmail":"farhan@farhan.codes","sentAt":"2024-01-20T04:28:05Z","receivedAt":"2024-01-20T04:28:26Z","isPatch":false,"sender":{"key":"farhan@farhan.codes","avatar":"https://gravatar.com/avatar/64c1dc837f48efeff1b51db4131f6ebe84b1d0c1caa43f73d8d45f2dee8e67ce?d=mp&s=160"},"body":"Hi,\n\nI am trying to implement REF_OBJ_DELTA, but having some trouble with offsets (sizes in the pack file) that do not align, specifically the size of the whole object does not seem to add up.\n\nFor example, consider this run:\n\ngit verify-pack -v .git/objects/pack/pack-48269bdfe1d28d20f603c6b23eed5717b7474e76.pack  | grep 82daab01f43e34b9f7c8e0db81a9951933b04f1b\n\n82daab01f43e34b9f7c8e0db81a9951933b04f1b commit 94 101 82749 1 ecd0e8c88ed8891da372f5630d542150b0a0531e\n\nThe size of the object is 94 bytes\nThe size of the entry is 101 bytes.\n\nMy patching/reconstruction of the object works, the compressed size is 97 bytes. However, I cannot figure out where the 101 comes from. The size of the object header is 2 bytes, the OBJ_REF_DELTA is 20 bytes (the SHA1), but that does not add up to 101 bytes.\n\nI am trying to understand where the 101 bytes comes from.\n\nIf not, can you please point me to where in the code the offset size for OBJ_REF_DELTA is calculated. I tried myself from buildin/verify-pack.c, but there seems to be some multi-threating/processing going on and I was not able to determine where the calculation happens.\n\nThanks!\n--\nFarhan Khan\nPGP Fingerprint: 1312 89CE 663E 1EB2 179C 1C83 C41D 2281 F8DA C0DE\n\n"},{"id":"487145","messageId":"xmqq1qabg8tm.fsf@gitster.g","threadId":"60773","inReplyTo":"afea6dc9-e557-4730-abe6-00947f77be06@app.fastmail.com","subject":"Re: How OBJ_REF_DELTA pack file size calculated","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-20T18:26:29Z","receivedAt":"2024-01-20T18:26:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Farhan Khan\" <farhan@farhan.codes> writes:\n\n> 82daab01f43e34b9f7c8e0db81a9951933b04f1b commit 94 101 82749 1 ecd0e8c88ed8891da372f5630d542150b0a0531e\n>\n> The size of the object is 94 bytes\n> The size of the entry is 101 bytes.\n\n> My patching/reconstruction of the object works, the compressed\n> size is 97 bytes.\n\nWhat do you mean by this?\n\nThe dense object header expresses the inflated size of the object\n(which should be 94 in your case).  By expressing it as a delta\nagainst some other object in the pack and then deflating the delta,\nwe should get the data that is much smaller than 94+20 if we choose\nto express it in the OBJ_REF_DELTA representation, as with such a\nsuboptimal delta base, we would be better off expressing it as a\nbase object that is merely deflated.  We do not need 20-byte offset\noverhead, and when reconstructing the object, they do not need to\ndeflate the base object and apply the delta.\n\nSo I am not sure what you mean by \"the compressed size is 97 bytes\".\n\n> However, I cannot figure out where the 101 comes\n> from. The size of the object header is 2 bytes, the OBJ_REF_DELTA\n> is 20 bytes (the SHA1), but that does not add up to 101 bytes.\n\n$ git help format-pack\n\n   - The header is followed by a number of object entries, each of\n     which looks like this:\n\n     (undeltified representation)\n     n-byte type and length (3-bit type, (n-1)*7+4-bit length)\n     compressed data\n\n     (deltified representation)\n     n-byte type and length (3-bit type, (n-1)*7+4-bit length)\n     base object name if OBJ_REF_DELTA or a negative relative\n\t offset from the delta object's position in the pack if this\n\t is an OBJ_OFS_DELTA object\n     compressed delta data\n\n     Observation: the length of each object is encoded in a variable\n     length format and is not constrained to 32-bit or anything.\n\nSo, if the object header for this object in this pack is 2 bytes\nlong as you observed above, then 101 bytes should be 2 bytes of \nheader, 20 bytes of base object name, and the remainder would be a\ndeflated delta data that is 101 - 22 = 79 bytes.  Reading the base\nobject and applying that delta (which deflates to 79 bytes) would\nreconstruct the original 94 bytes of the object.\n\n"}]}