{"thread":{"id":"49492","subject":"Is there some script to find un-delta-able objects?","startedAt":"2018-10-05T14:20:33Z","lastAt":"2018-10-05T16:56:48Z","messageCount":5,"participants":["Ævar Arnfjörð Bjarmason","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"359689","messageId":"87d0soh3v8.fsf@evledraar.gmail.com","threadId":"49492","inReplyTo":null,"subject":"Is there some script to find un-delta-able objects?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-05T14:20:27Z","receivedAt":"2018-10-05T14:20:33Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I.e. something to generate the .gitattributes file using this format:\n\nhttps://git-scm.com/docs/gitattributes#_packing_objects\n\nSome stuff is obvious, like \"*.gpg binary -delta\", but I'm wondering if\nthere's some repo scanner utility to spew this out for a given repo.\n"},{"id":"359694","messageId":"20181005161943.GA8816@sigill.intra.peff.net","threadId":"49492","inReplyTo":"87d0soh3v8.fsf@evledraar.gmail.com","subject":"Re: Is there some script to find un-delta-able objects?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-05T16:19:43Z","receivedAt":"2018-10-05T16:19:47Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 05, 2018 at 04:20:27PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> I.e. something to generate the .gitattributes file using this format:\n> \n> https://git-scm.com/docs/gitattributes#_packing_objects\n> \n> Some stuff is obvious, like \"*.gpg binary -delta\", but I'm wondering if\n> there's some repo scanner utility to spew this out for a given repo.\n\nI'm not sure what you mean by \"un-delta-able\" objects. Do you mean ones\nwhere we're not likely to find a delta? Or ones where Git will not try\nto look for a delta?\n\nIf the latter, I think the only rules are the \"-delta\" attribute and the\nobject size. You should be able to use git-check-attr and \"git-cat-file\"\nto get that info.\n\nIf the former, I don't know how you would know. We can only report on\nwhat isn't a delta _yet_.\n\n-Peff\n"},{"id":"359697","messageId":"87bm88gx7a.fsf@evledraar.gmail.com","threadId":"49492","inReplyTo":"20181005161943.GA8816@sigill.intra.peff.net","subject":"Re: Is there some script to find un-delta-able objects?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-05T16:44:25Z","receivedAt":"2018-10-05T16:44:30Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 05 2018, Jeff King wrote:\n\n> On Fri, Oct 05, 2018 at 04:20:27PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> I.e. something to generate the .gitattributes file using this format:\n>>\n>> https://git-scm.com/docs/gitattributes#_packing_objects\n>>\n>> Some stuff is obvious, like \"*.gpg binary -delta\", but I'm wondering if\n>> there's some repo scanner utility to spew this out for a given repo.\n>\n> I'm not sure what you mean by \"un-delta-able\" objects. Do you mean ones\n> where we're not likely to find a delta? Or ones where Git will not try\n> to look for a delta?\n>\n> If the latter, I think the only rules are the \"-delta\" attribute and the\n> object size. You should be able to use git-check-attr and \"git-cat-file\"\n> to get that info.\n>\n> If the former, I don't know how you would know. We can only report on\n> what isn't a delta _yet_.\n\nSome version of the former. Ones where we haven't found any (or much of)\nuseful deltas yet. E.g. say I had a repository with a lot of files\ngenerated by this command at various points in the history:\n\n dd if=/dev/urandom of=file.binary count=1024 bs=1024\n\nSome script similar to git-sizer which could report that the\npacked+compressed+delta'd version of the 10 *.binary files I had in my\nhistory had a 1:1 ratio of how large they were in .git, v.s. how large\nthe sum of each file retrieved by \"git show\" was (i.e. uncompressed,\nun-delta'd).\n\nThat doesn't mean that tomorrow I won't commit 10 new objects which\nwould have a really good delta ratio to those 10 existing files,\nbringing the ratio to ~1:2, but if I had some report like:\n\n \n\nFor a given repo that could be fed into .gitattributes to say we\nshouldn't bother to delta files of certain extensions.\n"},{"id":"359700","messageId":"xmqqk1mwwdak.fsf@gitster-ct.c.googlers.com","threadId":"49492","inReplyTo":"20181005161943.GA8816@sigill.intra.peff.net","subject":"Re: Is there some script to find un-delta-able objects?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-05T16:47:47Z","receivedAt":"2018-10-05T16:47:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Fri, Oct 05, 2018 at 04:20:27PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> I.e. something to generate the .gitattributes file using this format:\n>> \n>> https://git-scm.com/docs/gitattributes#_packing_objects\n>> \n>> Some stuff is obvious, like \"*.gpg binary -delta\", but I'm wondering if\n>> there's some repo scanner utility to spew this out for a given repo.\n>\n> I'm not sure what you mean by \"un-delta-able\" objects. Do you mean ones\n> where we're not likely to find a delta? Or ones where Git will not try\n> to look for a delta?\n>\n> If the latter, I think the only rules are the \"-delta\" attribute and the\n> object size. You should be able to use git-check-attr and \"git-cat-file\"\n> to get that info.\n>\n> If the former, I don't know how you would know. We can only report on\n> what isn't a delta _yet_.\n\nI am reasonably sure that the question is about solving the former\nso that \"-delta\" attribute is set appropriately.\n\nIniitially, I thought that it is likely an undeltifiable object has\nhigher randomness than deltifiable ones and that can be exploited,\nbut if you have such a highly random blob A (and no other object\nlike it) in the repository and then later acquire another blob B\nthat happens to share most of the data with A, then A and B by\nthemselves will pass the \"highly random\" test but still yet each can\nbe expressed as a delta derived from the other. So your \"what isn't\na delta yet\" is a reasonable assessment of what mechanically can be\nknown. \n\nKnowledge/heuristic like \"No two '*.gpg' files are expected to be\nalike\" needs something more than the randomness of individual files,\nI guess.\n\n"},{"id":"359705","messageId":"20181005165644.GD11254@sigill.intra.peff.net","threadId":"49492","inReplyTo":"87bm88gx7a.fsf@evledraar.gmail.com","subject":"Re: Is there some script to find un-delta-able objects?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-05T16:56:45Z","receivedAt":"2018-10-05T16:56:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 05, 2018 at 06:44:25PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Some version of the former. Ones where we haven't found any (or much of)\n> useful deltas yet. E.g. say I had a repository with a lot of files\n> generated by this command at various points in the history:\n> \n> dd if=/dev/urandom of=file.binary count=1024 bs=1024\n> \n> Some script similar to git-sizer which could report that the\n> packed+compressed+delta'd version of the 10 *.binary files I had in my\n> history had a 1:1 ratio of how large they were in .git, v.s. how large\n> the sum of each file retrieved by \"git show\" was (i.e. uncompressed,\n> un-delta'd).\n\nYou can get the uncompressed and on-disk sizes with:\n\n git cat-file --batch-all-objects \\\n --batch-check='%(objectname) %(objectsize) %(objectsize:disk)'\n\nand then compare the sizes/ratios however you like. If you want just a\nsubset of the blobs, drop the \"--batch-all-objects\" and just feed the\nobject names or even \"HEAD:filename\" on stdin).\n\n> That doesn't mean that tomorrow I won't commit 10 new objects which\n> would have a really good delta ratio to those 10 existing files,\n> bringing the ratio to ~1:2, but if I had some report like:\n> \n> \n> \n> For a given repo that could be fed into .gitattributes to say we\n> shouldn't bother to delta files of certain extensions.\n\nI don't know of a tool that does that, but I think a modest application\nof perl to the cat-file output would produce it.\n\n-Peff\n"}]}