{"thread":{"id":"17647","subject":"checking sha1's of files","startedAt":"2009-02-08T09:39:19Z","lastAt":"2009-02-08T12:57:52Z","messageCount":7,"participants":["Caleb Cushing","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"103703","messageId":"81bfc67a0902080139j1331d967g34bfc18f4068e443@mail.gmail.com","threadId":"17647","inReplyTo":null,"subject":"checking sha1's of files","fromName":"Caleb Cushing","fromEmail":"xenoterracide@gmail.com","sentAt":"2009-02-08T09:39:19Z","receivedAt":"2009-02-08T09:39:19Z","isPatch":false,"sender":{"key":"xenoterracide@gmail.com","avatar":"https://gravatar.com/avatar/af3f0745dfa0ea9c4ee551d7d0a3cfe7ba8d229754c11678ab2ed23c3fa57065?d=mp&s=160"},"body":"I need to check the hashes of specific files in the repo in an\nautomated fashion, in another tool.\n\nto be less vague currently gentoo's portage tree has manifests for\neach file in the tree, on funtoo and regen2 (forks) we've imported the\ntree into git. Git has all the manifesting that's needed (most of it,\nstill doesn't help with files outside the tree) in it. I'd like to be\nable to remove manifests from the tree, however I still want to check\nthat the ebuilds (package format) are consistent at run time. Checking\nthe entire tree is not sane.\n\nI figure the best way to do this is to first check stat against the\nindex, then, if that passes check the sha1, if that passes continue to\nthe next step.\n\nI don't want to do anything like determine the output of a git command\nin my code, I'd rather check to see if the check passed or failed\nusing return codes or some such. If it is capable of checking these\nbut would require me to parse output I'd still like to know, as it may\nlet me get the fix in faster, and I can do better later.\n\nI know git may not be currently capable of this behavior, which means\nI should extend it, or even write a new program to deal with it. If\nthis is the case, is there any documentation on how git does this?\naside from the source? could someone point me in the general direction\nof source files I should be looking at? maybe even specific functions?\n\nany help with this endeavor of any kind would be appreciated as the\nmanifests 'cause the repo to balloon, not to mention are just a pain\nto manage as they can't actually be merged.\n-- \nCaleb Cushing\n\nhttp://xenoterracide.blogspot.com\n"},{"id":"103711","messageId":"7veiy9z1v8.fsf@gitster.siamese.dyndns.org","threadId":"17647","inReplyTo":"81bfc67a0902080139j1331d967g34bfc18f4068e443@mail.gmail.com","subject":"Re: checking sha1's of files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-08T09:58:35Z","receivedAt":"2009-02-08T09:58:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Caleb Cushing writes:\n\n> I need to check the hashes of specific files in the repo in an\n> automated fashion, in another tool.\n\nWhat \"hash\" are you talking about? sha1? md5? crc?\n\nI *think* you are trying to say that gentoo has a tool to compute some\nsort of hash for regular files in their source tree by:\n\n\t\"gentoo's portage tree has manifests for each file\"\n\nbut without knowing what kind of hash they use, I cannot tell you if you\ncan reuse some part of git to compute their hash without using their tools\n(it also is unclear why you are not using their tool to compute their hash\nand instead are expecting git to know about the specific hash function\nused by them).\n\nFor exammple,\n\n\t\"sha1sum Makefile\"\n\nwould give you the SHA-1 checksum of the contents of the Makefile. Is\nthat what gentoo's tools expect? If that is the case, that is different\nfrom the blob object name git will give to the contents of that Makefile,\nso you cannot reuse much of git.\n"},{"id":"103721","messageId":"81bfc67a0902080253x5082201co7da3319b8237f5a0@mail.gmail.com","threadId":"17647","inReplyTo":"7veiy9z1v8.fsf@gitster.siamese.dyndns.org","subject":"Re: checking sha1's of files","fromName":"Caleb Cushing","fromEmail":"xenoterracide@gmail.com","sentAt":"2009-02-08T10:53:31Z","receivedAt":"2009-02-08T10:53:31Z","isPatch":false,"sender":{"key":"xenoterracide@gmail.com","avatar":"https://gravatar.com/avatar/af3f0745dfa0ea9c4ee551d7d0a3cfe7ba8d229754c11678ab2ed23c3fa57065?d=mp&s=160"},"body":"On Sun, Feb 8, 2009 at 4:58 AM, Junio C Hamano wrote:\n> Caleb Cushing writes:\n>\n>> I need to check the hashes of specific files in the repo in an\n>> automated fashion, in another tool.\n>\n> What \"hash\" are you talking about? sha1? md5? crc?\n\njust at at a vague glance currently sha1, sha256, and rmd160 if I'm\nreading the manifest file correct.\n\n> I *think* you are trying to say that gentoo has a tool to compute some\n> sort of hash for regular files in their source tree by:\n>\n> \"gentoo's portage tree has manifests for each file\"\n>\n> but without knowing what kind of hash they use, I cannot tell you if you\n> can reuse some part of git to compute their hash without using their tools\n> (it also is unclear why you are not using their tool to compute their hash\n> and instead are expecting git to know about the specific hash function\n> used by them).\n\nI think you misunderstand. I'm not trying to use git to compute their\nhash, I'm trying to replace their hash with git. Once I've figured out\nhow to validate the git hash with 'emerge' (said tool) I will be\nremoving their hash's.\n\n> For exammple,\n>\n> \"sha1sum Makefile\"\n>\n> would give you the SHA-1 checksum of the contents of the Makefile. Is\n> that what gentoo's tools expect?\n\nI'm honestly not sure how all it's calculated, only that the gentoo\ntool will have to be modified, only part of the Manifest file that\nwill be left is the check that is used to validate files not in the\ntree (e.g. our distfiles, package tarballs)\n\n> If that is the case, that is different\n> from the blob object name git will give to the contents of that Makefile,\n> so you cannot reuse much of git.\n>\n\ngit has it's own internal integrity check right? on the blob's.\n\nI don't intend to make git fit gentoo's system, I intend to make\ngentoo (or regen2 rather) use git's integrity system. but integrity\nchecks are only a small part of the larger tool. I have the feeling\nthat I will have to write code on both ends to make it work. Gentoo\nuses rsync and no git repository, so to validate integrity they use\nfiles with recorded hashes in them. as I understand it git hashes the\nfiles (blobs) internally, so now that we've imported the rsync tree\nand started using the 'git' protocol we're really doing the same thing\ntwice, and recording it more than that.\n\nhas this explanation helped?\n\n-- \nCaleb Cushing\n\nhttp://xenoterracide.blogspot.com\n"},{"id":"103730","messageId":"20090208111343.GC14359@coredump.intra.peff.net","threadId":"17647","inReplyTo":"81bfc67a0902080253x5082201co7da3319b8237f5a0@mail.gmail.com","subject":"Re: checking sha1's of files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-02-08T11:13:43Z","receivedAt":"2009-02-08T11:13:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 08, 2009 at 05:53:31AM -0500, Caleb Cushing wrote:\n\n> > but without knowing what kind of hash they use, I cannot tell you if you\n> > can reuse some part of git to compute their hash without using their tools\n> > (it also is unclear why you are not using their tool to compute their hash\n> > and instead are expecting git to know about the specific hash function\n> > used by them).\n> \n> I think you misunderstand. I'm not trying to use git to compute their\n> hash, I'm trying to replace their hash with git. Once I've figured out\n> how to validate the git hash with 'emerge' (said tool) I will be\n> removing their hash's.\n\nI'm still not sure I entirely understand what you are trying to do, but\nthese building blocks may help.\n\nYou can see git's idea of the hash of a file in history (or in the\nindex) by asking rev-parse:\n\n # hash of 'Makefile' in the most recent commit on the current branch\n $ git rev-parse HEAD:Makefile\n 27b9569746179e68c635bdaab8e57395f63faf01\n\n # hash of 'Makefile' in the index\n $ git rev-parse :Makefile\n 27b9569746179e68c635bdaab8e57395f63faf01\n\n # hash of 'Makefile' in some arbitrary revision\n $ git rev-parse v1.5.1:Makefile\n b159ffd0ae49c28725de6549132e0ad3a3b69d20\n\nAnd you can compute the git blob hash of any file use git hash-object:\n\n $ git hash-object --stdin < Makefile\n 27b9569746179e68c635bdaab8e57395f63faf01\n\nSo if I understand you correctly, you would use the former when\ngenerating your manifests from a revision, and the latter when verifying\nthe contents of the filesystem against those manifests.\n\n-Peff\n"},{"id":"103735","messageId":"81bfc67a0902080422s7f12d86ft5196c184c06385f9@mail.gmail.com","threadId":"17647","inReplyTo":"20090208111343.GC14359@coredump.intra.peff.net","subject":"Re: checking sha1's of files","fromName":"Caleb Cushing","fromEmail":"xenoterracide@gmail.com","sentAt":"2009-02-08T12:22:31Z","receivedAt":"2009-02-08T12:22:31Z","isPatch":false,"sender":{"key":"xenoterracide@gmail.com","avatar":"https://gravatar.com/avatar/af3f0745dfa0ea9c4ee551d7d0a3cfe7ba8d229754c11678ab2ed23c3fa57065?d=mp&s=160"},"body":"> So if I understand you correctly, you would use the former when\n> generating your manifests from a revision, and the latter when verifying\n> the contents of the filesystem against those manifests.\n\n\nthat sounds about right actually. except there won't be manifests e.g.\nI may actually run\n\nfind . -name Manifest -exec rm '{}' +\n\nfrom the root of the tree. (depends on whether I can get rid of\nmanifesting distfiles, but they don't change so often, so aren't\nreally a problem).\n\n> $ git hash-object --stdin < Makefile\n> 27b9569746179e68c635bdaab8e57395f63faf01\n\nis there anyway built in way to do that and check to see if the hash\nmatches HEAD? (before I go and write a string comparison so that it\ndoes)\n-- \nCaleb Cushing\n\nhttp://xenoterracide.blogspot.com\n"},{"id":"103737","messageId":"20090208122731.GA24442@coredump.intra.peff.net","threadId":"17647","inReplyTo":"81bfc67a0902080422s7f12d86ft5196c184c06385f9@mail.gmail.com","subject":"Re: checking sha1's of files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-02-08T12:27:31Z","receivedAt":"2009-02-08T12:27:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 08, 2009 at 07:22:31AM -0500, Caleb Cushing wrote:\n\n> > $ git hash-object --stdin < Makefile\n> > 27b9569746179e68c635bdaab8e57395f63faf01\n> \n> is there anyway built in way to do that and check to see if the hash\n> matches HEAD? (before I go and write a string comparison so that it\n> does)\n\nIf you want to know whether a file matches HEAD, just do:\n\n git diff --quiet HEAD -- $LIST_OF_FILES\n\nwhich will return '0' for no changes or '1' if there are changes.\n\nWhich really has nothing to do with hashes at all (though git will use\nthem internally to avoid actually running a textual diff at all). I was\nassuming that you didn't necessarily _have_ the git repository at\nverification time. So the hash becomes an easy way of saying \"this is\nwhat the file _should_ look like\".\n\nBut again, I'm not sure I really understand the workflow you're trying\nto perform.\n\n-Peff\n"},{"id":"103741","messageId":"81bfc67a0902080457k65988e5bw8a33ea8aff2c0c2f@mail.gmail.com","threadId":"17647","inReplyTo":"20090208122731.GA24442@coredump.intra.peff.net","subject":"Re: checking sha1's of files","fromName":"Caleb Cushing","fromEmail":"xenoterracide@gmail.com","sentAt":"2009-02-08T12:57:52Z","receivedAt":"2009-02-08T12:57:52Z","isPatch":false,"sender":{"key":"xenoterracide@gmail.com","avatar":"https://gravatar.com/avatar/af3f0745dfa0ea9c4ee551d7d0a3cfe7ba8d229754c11678ab2ed23c3fa57065?d=mp&s=160"},"body":"> If you want to know whether a file matches HEAD, just do:\n>\n> git diff --quiet HEAD -- $LIST_OF_FILES\n>\n> which will return '0' for no changes or '1' if there are changes.\n\n*headdesks* and somehow I think this is exactly what I need.\n\n> Which really has nothing to do with hashes at all (though git will use\n> them internally to avoid actually running a textual diff at all). I was\n> assuming that you didn't necessarily _have_ the git repository at\n> verification time. So the hash becomes an easy way of saying \"this is\n> what the file _should_ look like\".\n\nI love git because it's so powerful, just like the rest of *nix. No\nmatter how long I use it something always makes me feel like a n00b.\n\nthe problem is that I only half understand what git is doing, just\nenough to attempt a communication of what I want to do. It doesn't\nhelp that I'm not the expert on the other side of this problem either.\n\nthanks for the help guys, even if I do come off like a moron.\n-- \nCaleb Cushing\n\nhttp://xenoterracide.blogspot.com\n"}]}