{"thread":{"id":"41641","subject":"Is there a --stat or --numstat like option that'll allow me to have my cake and eat it too?","startedAt":"2016-03-08T15:08:21Z","lastAt":"2016-03-08T20:58:59Z","messageCount":4,"participants":["Ævar Arnfjörð Bjarmason","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"280372","messageId":"CACBZZX7o+VA1RVvja3xtBQf+rr2bWoByas4D5GKZ_VfQr7H19w@mail.gmail.com","threadId":"41641","inReplyTo":null,"subject":"Is there a --stat or --numstat like option that'll allow me to have my cake and eat it too?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-03-08T15:08:21Z","receivedAt":"2016-03-08T15:08:21Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I maintain a hook for Git that allows you to block binary pushes[1],\nfrom other implementations I've seen it's the least stupid thing out\nthere that does that.\n\nBasically on-push it parses this:\n\n    git log --pretty=format:%H -M100% --stat=9000,9001 <old_ref>..<new_ref>\n\nThe --stat=9000,9001 is there to make sure we still get the filename\nif it's long[2].\n\nIt's important that this is something like \"git-log\" instead of\n\"git-show for each\" for performance (think a push with hundreds of\ncommits). It's also important that it's not \"git diff\" (think a push\nthat adds/removes a huge binary file within one push). I also don't\nwant to manually parse \"git log --numstat -p\" or whatever for\nperformance reasons since every push hangs on this.\n\nIt's somewhat of a pain to parse that  --stat output, because I have\nto look for /\\|\\s+Bin / in the output to detect binary changes.\n\nYou might be thinking \"why don't you use --numstat?\". Because while\nthat option does most of what I want it doesn't show the old/new size\nof the binary file, so I can't have a policy to allow e.g. <=1KB files\nwithout doing a second pass with --stat or \"git show\".\n\nBoth formats also have various parsing edge cases, e.g. with -M100% I\nhave to parse out renames like \"foo.png => bar.png\", but you can also\ncreate a file with \" => \" in the filename and there's no way to\ndisambiguate it.\n\nBoth formats also only show lines added/deleted, but --numstat doesn't\nshow the size before/after for binary files, so if I want to also\nprohibit huge non-binary files I can't without running both --stat and\n--numstat.\n\nWhat I really want is something for git-log more like\ngit-for-each-ref, so I could emit the following info for each file\nbeing modified delimited by some binary marker:\n\n    - file name before\n    - file name after\n    - is rename?\n    - is binary?\n    - size in bytes before\n    - size it bytes after\n    - removed lines\n    - added lines\n\nI think no combination of git-log options or any built-in machinery\ncomes close to giving me all of that without having to do multiple\npasses with some combination of git-log and git-show, but I'd love to\nbe proven wrong.\n\n1. https://github.com/avar/pre-receive-reject-binaries\n2. OVER NINE THOUSAND should be enough for everyone, right?\n"},{"id":"280376","messageId":"xmqqoaaolvfl.fsf@gitster.mtv.corp.google.com","threadId":"41641","inReplyTo":"CACBZZX7o+VA1RVvja3xtBQf+rr2bWoByas4D5GKZ_VfQr7H19w@mail.gmail.com","subject":"Re: Is there a --stat or --numstat like option that'll allow me to have my cake and eat it too?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-08T17:40:46Z","receivedAt":"2016-03-08T17:40:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> What I really want is something for git-log more like\n> git-for-each-ref, so I could emit the following info for each file\n> being modified delimited by some binary marker:\n>\n>     - file name before\n>     - file name after\n>     - is rename?\n>     - is binary?\n>     - size in bytes before\n>     - size it bytes after\n>     - removed lines\n>     - added lines\n>\n> I think no combination of git-log options or any built-in machinery\n> comes close to giving me all of that without having to do multiple\n> passes with some combination of git-log and git-show, but I'd love to\n> be proven wrong.\n\nI do not think such a thing exists.  From the look of the above\nlist, if I were implementing it, I'd think it would be the easiest\nif it is built as a new output format of \"diff\" that sits next to\nexisting --stat, --patch, and --numstat formats, i.e. you would be\nwriting a new aevars_stat_consume() callback function and calling\nxdi_diff_outf() like everybody else.  A possible output format may\nlook as if we are showing \"log --patch\" output with a bit more\nextended diff header lines (e.g. in addition to \"rename from\", etc.,\nyou would have \"bytes before\" and other new types of headers), but\nwithout the actual patch text.\n"},{"id":"280386","messageId":"20160308205124.GA25849@sigill.intra.peff.net","threadId":"41641","inReplyTo":"CACBZZX7o+VA1RVvja3xtBQf+rr2bWoByas4D5GKZ_VfQr7H19w@mail.gmail.com","subject":"Re: Is there a --stat or --numstat like option that'll allow me to have my cake and eat it too?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-03-08T20:51:24Z","receivedAt":"2016-03-08T20:51:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 08, 2016 at 04:08:21PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> What I really want is something for git-log more like\n> git-for-each-ref, so I could emit the following info for each file\n> being modified delimited by some binary marker:\n> \n>     - file name before\n>     - file name after\n>     - is rename?\n>     - is binary?\n>     - size in bytes before\n>     - size it bytes after\n>     - removed lines\n>     - added lines\n\nIf you get the full sha1s of each object (e.g., by adding --raw), then\nyou can dump them all to a single cat-file invocation to efficiently get\nthe sizes.\n\nI'm not quite sure I understand why you want to know about renames and\nadded/removed lines if you are just blocking binary files. If I were\nimplementing this[1], I'd probably just block based on blob size, which\nyou can do with:\n\n  git rev-list --objects $old..$new |\n  git cat-file --batch-check='%(objectsize) %(objectname) %(rest)' |\n  perl -alne 'print if $F[0] > 1_000_000; # or whatever' |\n  while read size sha1 file; do\n\techo \"Whoops, $file ($sha1) is too big\"\n\texit 1\n  done\n\nYou can also use %(objectsize:disk) to get the on-disk size (which can\ntell you about things that don't compress well, which tend to be the\nsorts of things you are trying to keep out).\n\nYou can't ask about binary-ness, but I don't think it would unreasonable\nfor cat-file to have a \"would git consider this content binary?\"\nplaceholder for --batch-check.\n\nThe other things are properties of the comparison, not of individual\nobjects, so you'll have to get them from \"git log\". But with some clever\nscripting, I think you could feed those sha1s (or $commit:$path\nspecifiers) into a single cat-file invocation to get the before/after\nsizes.\n\n-Peff\n\n[1] GitHub has hard and soft limits for various blob sizes, and at one\n    point the implementation looked very similar to what I showed here.\n    The downside is that for a large push, the rev-list can actually\n    take a fair bit of time (e.g., consider pushing up all of the kernel\n    history to a brand new repo), and this is on top of the similar work\n    already done by index-pack and check_everything_connected().\n\n    These days I have a hacky patch to notice the too-big size directly\n    in index-pack, which is essentially free. It doesn't know about the\n    file path, so we pull that out later in the pre-receive hook. But we\n    only have to do so in the uncommon case that there _is_ actually a\n    too-big file, so normal pushes incur no penalty.\n"},{"id":"280387","messageId":"CACBZZX4xU3qzRB=H3R3PZwZdBNc_pUm2rff3KG965J_J5WgFVQ@mail.gmail.com","threadId":"41641","inReplyTo":"20160308205124.GA25849@sigill.intra.peff.net","subject":"Re: Is there a --stat or --numstat like option that'll allow me to have my cake and eat it too?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-03-08T20:58:59Z","receivedAt":"2016-03-08T20:58:59Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Tue, Mar 8, 2016 at 9:51 PM, Jeff King <peff@peff.net> wrote:\n> On Tue, Mar 08, 2016 at 04:08:21PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n>> What I really want is something for git-log more like\n>> git-for-each-ref, so I could emit the following info for each file\n>> being modified delimited by some binary marker:\n>>\n>>     - file name before\n>>     - file name after\n>>     - is rename?\n>>     - is binary?\n>>     - size in bytes before\n>>     - size it bytes after\n>>     - removed lines\n>>     - added lines\n>\n> If you get the full sha1s of each object (e.g., by adding --raw), then\n> you can dump them all to a single cat-file invocation to efficiently get\n> the sizes.\n>\n> I'm not quite sure I understand why you want to know about renames and\n> added/removed lines if you are just blocking binary files. If I were\n> implementing this[1], I'd probably just block based on blob size, which\n> you can do with:\n\nI want to know about renames because if you're just moving an existing\nbinary file around that's fine, it's not adding a new big blob to the\nrepo.\n\nThe hook also has a facility to commit binary stuff if you add \"yes I\nknow what I'm doing and want to commit N bytes to the repo\" to the\ncommit message. Mostly when people do this it's an accident.\n\nI wanted to know about added/removed lines because I was looking into\nextending this non-binary files. Today at work someone committed 300MB\nof text files to a branch, we could delete it in that case, but it\nwould also be nice to have limits on that sort of thing too.\n\n>   git rev-list --objects $old..$new |\n>   git cat-file --batch-check='%(objectsize) %(objectname) %(rest)' |\n>   perl -alne 'print if $F[0] > 1_000_000; # or whatever' |\n>   while read size sha1 file; do\n>         echo \"Whoops, $file ($sha1) is too big\"\n>         exit 1\n>   done\n>\n> You can also use %(objectsize:disk) to get the on-disk size (which can\n> tell you about things that don't compress well, which tend to be the\n> sorts of things you are trying to keep out).\n>\n> You can't ask about binary-ness, but I don't think it would unreasonable\n> for cat-file to have a \"would git consider this content binary?\"\n> placeholder for --batch-check.\n>\n> The other things are properties of the comparison, not of individual\n> objects, so you'll have to get them from \"git log\". But with some clever\n> scripting, I think you could feed those sha1s (or $commit:$path\n> specifiers) into a single cat-file invocation to get the before/after\n> sizes.\n>\n> -Peff\n>\n> [1] GitHub has hard and soft limits for various blob sizes, and at one\n>     point the implementation looked very similar to what I showed here.\n>     The downside is that for a large push, the rev-list can actually\n>     take a fair bit of time (e.g., consider pushing up all of the kernel\n>     history to a brand new repo), and this is on top of the similar work\n>     already done by index-pack and check_everything_connected().\n>\n>     These days I have a hacky patch to notice the too-big size directly\n>     in index-pack, which is essentially free. It doesn't know about the\n>     file path, so we pull that out later in the pre-receive hook. But we\n>     only have to do so in the uncommon case that there _is_ actually a\n>     too-big file, so normal pushes incur no penalty.\n\nAll good tips / insights. I'll definitely check some of this out.\n"}]}