{"thread":{"id":"50832","subject":"Calculating pack file SHA value","startedAt":"2019-03-28T01:06:24Z","lastAt":"2019-03-28T02:02:31Z","messageCount":2,"participants":["Farhan Khan","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"372598","messageId":"a48b86698802006045ed0af060b4e822@farhan.codes","threadId":"50832","inReplyTo":null,"subject":"Calculating pack file SHA value","fromName":"Farhan Khan","fromEmail":"farhan@farhan.codes","sentAt":"2019-03-28T01:06:20Z","receivedAt":"2019-03-28T01:06:24Z","isPatch":false,"sender":{"key":"farhan@farhan.codes","avatar":"https://gravatar.com/avatar/64c1dc837f48efeff1b51db4131f6ebe84b1d0c1caa43f73d8d45f2dee8e67ce?d=mp&s=160"},"body":"Hi all,\n\nI am trying to figure out how to calculate the SHA value of a pack file when you\nrun `git index-pack file.pack`. I am close, but having a bit of trouble at the\nend. Here's my understanding so far.\n\nGit buffers data to be processed and when its exhausted, updates the SHA\nchecksum with the previously read data. This is from builtin/index-pack.c,\nspecifically fill() which calls flush() to update the SHA value. My question is,\nhow does git determine how many bytes at a time to process?\n\nThe size of the buffer is the file-scope variable input_len. This size seems to\nbe 4096 several times until the very end where it reduces to less-than 4096\n(obviously this depends on the pack file, but in my case its 1074 bytes).\nOrdinarily I would think its a result of the read() call not receiving the full\n4096 bytes, but there still are left over bytes in the file but my manual\nverification shows there are still remaining bytes in the file which are not run\nthrough the SHA checksum.\n\nHow does git calculate a pack file's SHA verification? How does it know what\nsize (number of bytes) to read when running flush() to update the buffer?\n(typically 4096). How does it know when in the file to stop updating the SHA1\nvalue?\n\nI hope my questions are clear. Thanks!\n\n---\nFarhan Khan\nPGP Fingerprint: 1312 89CE 663E 1EB2 179C 1C83 C41D 2281 F8DA C0DE\n"},{"id":"372600","messageId":"20190328020227.GB7887@sigill.intra.peff.net","threadId":"50832","inReplyTo":"a48b86698802006045ed0af060b4e822@farhan.codes","subject":"Re: Calculating pack file SHA value","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-03-28T02:02:27Z","receivedAt":"2019-03-28T02:02:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 27, 2019 at 09:06:20PM -0400, Farhan Khan wrote:\n\n> I am trying to figure out how to calculate the SHA value of a pack file when you\n> run `git index-pack file.pack`. I am close, but having a bit of trouble at the\n> end. Here's my understanding so far.\n\nIt's all but the last 20 bytes. You should be able to reproduce it with:\n\n  # the computed sha1\n  size=$(stat --format=%s $pack)\n  head -c $((size-20)) $pack | sha1sum\n\n  # the sha1 stored in the file, which should match\n  tail -c 20 $pack | xxd\n\n> Git buffers data to be processed and when its exhausted, updates the SHA\n> checksum with the previously read data. This is from builtin/index-pack.c,\n> specifically fill() which calls flush() to update the SHA value. My question is,\n> how does git determine how many bytes at a time to process?\n> \n> The size of the buffer is the file-scope variable input_len. This size seems to\n> be 4096 several times until the very end where it reduces to less-than 4096\n> (obviously this depends on the pack file, but in my case its 1074 bytes).\n> Ordinarily I would think its a result of the read() call not receiving the full\n> 4096 bytes, but there still are left over bytes in the file but my manual\n> verification shows there are still remaining bytes in the file which are not run\n> through the SHA checksum.\n\nOn the fill() side, we may over-read bytes into our buffer. But it's on\nthe use() side that we actually decide bytes have been used. Note that\nit increments input_offset, and then flush() only hashes bytes up to\nthat offset.\n\nSo index-pack is not just blindly hashing N-20 bytes. It's actually\nparsing the packfile as it goes, and putting any data it has parsed\ncorrectly into the hash. At the end, we _should_ be left with exactly 20\nbytes, and they should match exactly the hash we've computed up to that\npoint. And in --verify mode (and maybe even other modes) it should be\nconfirming that.\n\n> How does git calculate a pack file's SHA verification? How does it know what\n> size (number of bytes) to read when running flush() to update the buffer?\n> (typically 4096). How does it know when in the file to stop updating the SHA1\n> value?\n\nThe key observation is that flush() isn't actually reading into the\nbuffer. It's throwing away bytes that have already been marked as used\nby use() and shifting the rest to the front of the buffer. And then\nfill() is free to read more data into the rest of it.\n\n-Peff\n"}]}