# [PATCH] Show binary file size change in diff --stat 12 messages from 2007-04-04 to 2007-04-04. Participants: Andy Parkins, Rogan Dawes, Geert Bosch, Johannes Schindelin, Linus Torvalds, Junio C Hamano. Thread: https://gitlist.dev/t/7510 ## Andy Parkins, 2007-04-04 13:14 Subject: [PATCH] Show binary file size change in diff --stat Message-ID: <200704041414.14797.andyparkins@gmail.com> URL: https://gitlist.dev/e/200704041414.14797.andyparkins%40gmail.com ``` Previously, a binary file in the diffstat would show as: some-binary-file.bin | Bin The space after the "Bin" was never used. This patch changes binary lines in the diffstat to be: some-binary-file.bin | Bin 12345 -> 123456 bytes The very nice "->" notation was suggested by Johannes Schindelin, and shows the before and after sizes more clearly than "+" and "-" would. If a size is 0 it's not shown (although it would probably be better to treat no-file differently from zero-byte-file). The user can see what changed in the binary file, and how big the new file is. This is in keeping with the information in the rest of the diffstat. The diffstat_t members "added" and "deleted" were unused when the file was binary, so this patch loads them with the file sizes in builtin_diffstat(). These figures are then read in show_stats() when the file is marked binary. Signed-off-by: Andy Parkins --- This is a resend; I still wish for it every time I see a --stat with binary files changed. Is there any objection to it that I can address? diff.c | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-) diff --git a/diff.c b/diff.c index d8f9242..1a527e9 100644 --- a/diff.c +++ b/diff.c @@ -811,7 +811,12 @@ static void show_stats(struct diffstat_t* data, struct diff_options *options) if (data->files[i]->is_binary) { show_name(prefix, name, len, reset, set); - printf(" Bin\n"); + printf(" Bin "); + printf("%s%i%s", del_c, deleted, reset); + printf(" -> "); + printf("%s%i%s", add_c, added, reset); + printf(" bytes"); + printf("\n"); goto free_diffstat_file; } else if (data->files[i]->is_unmerged) { @@ -1185,9 +1190,11 @@ static void builtin_diffstat(const char *name_a, const char *name_b, if (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0) die("unable to read files to diff"); - if (mmfile_is_binary(&mf1) || mmfile_is_binary(&mf2)) + if (mmfile_is_binary(&mf1) || mmfile_is_binary(&mf2)) { data->is_binary = 1; - else { + data->added = mf2.size; + data->deleted = mf1.size; + } else { /* Crazy xdl interfaces.. */ xpparam_t xpp; xdemitconf_t xecfg; -- 1.5.1.rc3.703.g5709-dirty ``` ## Rogan Dawes, 2007-04-04 13:34 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: <4613A974.60808@dawes.za.net> URL: https://gitlist.dev/e/4613A974.60808%40dawes.za.net In-Reply-To: <200704041414.14797.andyparkins@gmail.com> ``` Andy Parkins wrote: > Previously, a binary file in the diffstat would show as: > > some-binary-file.bin | Bin > > The space after the "Bin" was never used. This patch changes binary > lines in the diffstat to be: > > some-binary-file.bin | Bin 12345 -> 123456 bytes > > The very nice "->" notation was suggested by Johannes Schindelin, and > shows the before and after sizes more clearly than "+" and "-" would. > If a size is 0 it's not shown (although it would probably be better to > treat no-file differently from zero-byte-file). > > The user can see what changed in the binary file, and how big the new > file is. This is in keeping with the information in the rest of the > diffstat. > > The diffstat_t members "added" and "deleted" were unused when the file > was binary, so this patch loads them with the file sizes in > builtin_diffstat(). These figures are then read in show_stats() when > the file is marked binary. > > Signed-off-by: Andy Parkins > --- > This is a resend; I still wish for it every time I see a --stat with > binary files changed. > > Is there any objection to it that I can address? Well, how about my comments in <45E67978.9030805@dawes.za.net>, suggesting that the edit difference (number of steps required to transform one to the other) would be a better indication? I think it is better because it is consistent with what we currently do for text files: show the number of lines added/deleted. For binary files, it would be consistent to show the number of bytes added/deleted. I have not investigated the output format for the libxdiff binary patch format, but hopefully it would not be too difficult to calculate the deletions and additions. Rogan ``` ## Geert Bosch, 2007-04-04 14:40 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: URL: https://gitlist.dev/e/BAFDAA7B-60EC-44FD-8DAA-EB2F9C835F51%40adacore.com In-Reply-To: <4613A974.60808@dawes.za.net> ``` On Apr 4, 2007, at 09:34, Rogan Dawes wrote: > For binary files, it would be consistent to show the number of > bytes added/deleted. I have not investigated the output format for > the libxdiff binary patch format, but hopefully it would not be too > difficult to calculate the deletions and additions. For binary files it is impractical to do insert/delete type of differences. For text files, treating lines as indivisible entities to insert/delete make some sense. For binary files, you'd have to use some arbitrary context-defined breakpoints and then go from there. The result would be some very complicated and unclear algorithm that would have no use in the real world. Many binary files, such as an images, waveforms or virtually any compressed stream, can change in a way that changes all bytes in the file, while the changes in the displayed image or the uncompressed stream are imperceptible or absent. Guessing semantic differences between binary blobs is hopeless and subjective, while differences in size are fact. -Geert ``` ## Andy Parkins, 2007-04-04 14:40 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: <200704041540.59977.andyparkins@gmail.com> URL: https://gitlist.dev/e/200704041540.59977.andyparkins%40gmail.com In-Reply-To: <4613A974.60808@dawes.za.net> ``` On Wednesday 2007 April 04 14:34, Rogan Dawes wrote: > Well, how about my comments in <45E67978.9030805@dawes.za.net>, > suggesting that the edit difference (number of steps required to > transform one to the other) would be a better indication? Perhaps. There is certainly a difference between: somefile.bin | 1000 -> 1000 bytes and somefile.bin | 500 bytes removed, 500 bytes added > I think it is better because it is consistent with what we currently do > for text files: show the number of lines added/deleted. The thing is, "lines" is an understandable unit for a text file, so it's useful to show. I'm not sure the same is true of "bytes" for a binary file. Those bytes could represent anything; the true unit of a binary file is dependent on its type. > For binary files, it would be consistent to show the number of bytes > added/deleted. I have not investigated the output format for the > libxdiff binary patch format, but hopefully it would not be too > difficult to calculate the deletions and additions. I'm inclined to agree with Johannes, while it's certainly something that /could/ be shown - is it more useful? There is no guarantee that a small change in the underlying content is represented by a small change in the binary diff. As an example: compress a file, change a byte, compress it again, perform a binary diff; what is that diff telling you about the change? (My answer is: not much). Andy -- Dr Andy Parkins, M Eng (hons), MIET andyparkins@gmail.com ``` ## Rogan Dawes, 2007-04-04 15:51 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: <4613C97C.9050600@dawes.za.net> URL: https://gitlist.dev/e/4613C97C.9050600%40dawes.za.net In-Reply-To: <200704041540.59977.andyparkins@gmail.com> ``` Andy Parkins wrote: > On Wednesday 2007 April 04 14:34, Rogan Dawes wrote: > >> Well, how about my comments in <45E67978.9030805@dawes.za.net>, >> suggesting that the edit difference (number of steps required to >> transform one to the other) would be a better indication? > > Perhaps. There is certainly a difference between: > > somefile.bin | 1000 -> 1000 bytes > > and > > somefile.bin | 500 bytes removed, 500 bytes added > >> I think it is better because it is consistent with what we currently do >> for text files: show the number of lines added/deleted. > > The thing is, "lines" is an understandable unit for a text file, so it's > useful to show. I'm not sure the same is true of "bytes" for a binary file. > Those bytes could represent anything; the true unit of a binary file is > dependent on its type. I think bytes are the only reasonable unit for a binary file, since we have no idea what a meaningful divisor may be. So, defaulting to the smallest possible unit (other than going to the bit-level) makes perfect sense. >> For binary files, it would be consistent to show the number of bytes >> added/deleted. I have not investigated the output format for the >> libxdiff binary patch format, but hopefully it would not be too >> difficult to calculate the deletions and additions. > > I'm inclined to agree with Johannes, while it's certainly something > that /could/ be shown - is it more useful? There is no guarantee that a > small change in the underlying content is represented by a small change in > the binary diff. > > As an example: compress a file, change a byte, compress it again, perform a > binary diff; what is that diff telling you about the change? (My answer is: > not much). Well, at least as much as the resulting sizes tell you, if not more. Here is a counter example for a text file, where lines changed do not actually reflect the real changes in the file: the contents of an XML file being wrapped in an additional tag. Semantically, all that has changed is an opening and closing tag. But, we still show that on a line by line basis, the entire file has changed (because the indentation changes). So you'd have n lines deleted, and n+2 lines added (for the additional opening and closing tag). > Andy I still maintain that showing bytes changed is the only consistent thing to do, unless we have additional logic that allows us to do "per file-type" diff statistics. Maybe .gitattributes will allow/enable this? Regards, Rogan P.S. I'm not volunteering to inflict my novice C-skills on the git community, so this is really "just my 2c" ``` ## Rogan Dawes, 2007-04-04 16:00 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: <4613CB87.2090306@dawes.za.net> URL: https://gitlist.dev/e/4613CB87.2090306%40dawes.za.net In-Reply-To: ``` Geert Bosch wrote: > > On Apr 4, 2007, at 09:34, Rogan Dawes wrote: > >> For binary files, it would be consistent to show the number of bytes >> added/deleted. I have not investigated the output format for the >> libxdiff binary patch format, but hopefully it would not be too >> difficult to calculate the deletions and additions. > > For binary files it is impractical to do insert/delete type of differences. > For text files, treating lines as indivisible entities to insert/delete > make some sense. For binary files, you'd have to use some arbitrary > context-defined breakpoints and then go from there. The result would > be some very complicated and unclear algorithm that would have no use > in the real world. > > Many binary files, such as an images, waveforms or virtually any compressed > stream, can change in a way that changes all bytes in the file, while > the changes in the displayed image or the uncompressed stream are > imperceptible or absent. Guessing semantic differences between binary > blobs is hopeless and subjective, while differences in size are fact. > > -Geert As per my mail to Andy, we *already* do this for text files. e.g. wrap an XML document in an additional tag, and update the indentation to match. The semantic change is minimal (perhaps 2 new lines), but the reported change reflects n lines deleted, and n+2 added. Exactly because we *don't* do any semantic analysis (for text or binary files), we should simply report the number of bytes changed, exactly as we do for text files (reporting number of lines changed). This is _consistent_ with what we do currently for text files. Note that Andy's apparent preference (to know how the sizes have changed) can still largely be satisfied by this approach. somefile.bin | 1000 -> 1000 bytes and somefile.bin | 500 bytes removed, 500 bytes added You can still see that the overall size of the file has not changed, but you get the additional information about how many bytes were actually changed at the same time, which you don't get just showing the sizes. Rogan ``` ## Johannes Schindelin, 2007-04-04 16:22 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0704041819340.4045%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <4613C97C.9050600@dawes.za.net> ``` Hi, On Wed, 4 Apr 2007, Rogan Dawes wrote: > Andy Parkins wrote: > > > As an example: compress a file, change a byte, compress it again, > > perform a binary diff; what is that diff telling you about the change? > > (My answer is: not much). > > Well, at least as much as the resulting sizes tell you, if not more. The subtle difference: your approach is _expensive_ in terms of CPU time, while the byte change approach is _dirt cheap_. Since it seems that there are gazillions of examples where one or the other (or both) do not make sense, I'd rather have the fast one. HOWEVER, there is something I really would like to have, namely a clear distinction between new file / deleted file, and 0->n / n->0 changes. Ciao, Dscho ``` ## Rogan Dawes, 2007-04-04 16:26 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: <4613D1C1.6010002@dawes.za.net> URL: https://gitlist.dev/e/4613D1C1.6010002%40dawes.za.net In-Reply-To: ``` Johannes Schindelin wrote: > Hi, > > On Wed, 4 Apr 2007, Rogan Dawes wrote: > >> Andy Parkins wrote: >> >>> As an example: compress a file, change a byte, compress it again, >>> perform a binary diff; what is that diff telling you about the change? >>> (My answer is: not much). >> Well, at least as much as the resulting sizes tell you, if not more. > > The subtle difference: your approach is _expensive_ in terms of CPU time, > while the byte change approach is _dirt cheap_. I don't dispute that for a second. > Since it seems that there are gazillions of examples where one or the > other (or both) do not make sense, I'd rather have the fast one. Fair enough. Rogan ``` ## Linus Torvalds, 2007-04-04 16:40 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0704040935350.6730%40woody.linux-foundation.org In-Reply-To: ``` On Wed, 4 Apr 2007, Johannes Schindelin wrote: > > The subtle difference: your approach is _expensive_ in terms of CPU time, > while the byte change approach is _dirt cheap_. Well, you could do a combination (still dirt cheap): - show the size before/after (and yes, new/delete should be separate from "zero size before/after") - show the size of the binary patch. No "X added bytes" vs "Y bytes deleted", just "size of binary patch". It could be really small, even if 10k was deleted, or the file was totally re-organized by moving chunks around. It would still be a meaningful thing to know - if only because it tells you how much space the delta takes. So even if it's a .jpg, and the actual *picture* didn't change a lot (ie you did some new version with color correction or something: it looks similar to the old one, but the *diff* is basically "rewrite it all"), knowing the size of the delta at least has the meaning of "this is how basically much space it will take when you send the binary diff in an email". That's fairly close to what "5 new lines, 1 deleted line" message means. It's another way to give you an approximate idea of how big the changes were. Linus ``` ## Johannes Schindelin, 2007-04-04 16:59 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0704041856210.4045%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: ``` Hi, On Wed, 4 Apr 2007, Linus Torvalds wrote: > On Wed, 4 Apr 2007, Johannes Schindelin wrote: > > > > The subtle difference: your approach is _expensive_ in terms of CPU time, > > while the byte change approach is _dirt cheap_. > > Well, you could do a combination (still dirt cheap): > - show the size before/after (and yes, new/delete should be separate from > "zero size before/after") > - show the size of the binary patch. ... and by this (size of binary patch) you mean the deltified object? In general, we do not have the binary patch. And the generation of that binary patch is what I was referring to being expensive. Remember, most binary files are way larger than the average source code files we have, since it is much, much easier to generate binary data than to write meaningful code. Therefore, the binary patch generation has to look at much larger pieces to begin with, which translates into CPU time. Having said that, I have to admit that I don't have numbers backing this reasoning up. So, if somebody comes up with numbers contradicting my theory, I will gladly change my mind. Ciao, Dscho ``` ## Linus Torvalds, 2007-04-04 17:12 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0704041011170.6730%40woody.linux-foundation.org In-Reply-To: ``` On Wed, 4 Apr 2007, Johannes Schindelin wrote: > > ... and by this (size of binary patch) you mean the deltified object? yes. Just the size of the delta. Although I guess you're right - we may not have generated that. Linus ``` ## Junio C Hamano, 2007-04-04 17:47 Subject: Re: [PATCH] Show binary file size change in diff --stat Message-ID: <7vbqi458vn.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vbqi458vn.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > On Wed, 4 Apr 2007, Johannes Schindelin wrote: >> >> The subtle difference: your approach is _expensive_ in terms of CPU time, >> while the byte change approach is _dirt cheap_. > > Well, you could do a combination (still dirt cheap): > - show the size before/after (and yes, new/delete should be separate from > "zero size before/after") > - show the size of the binary patch. > > No "X added bytes" vs "Y bytes deleted", just "size of binary patch". It > could be really small, even if 10k was deleted, or the file was totally > re-organized by moving chunks around. > > It would still be a meaningful thing to know - if only because it tells > you how much space the delta takes. I agree wrt the kind of information to give, except that I am not so sure about new/delete vs zero before/after. We do not do that for a text file, and when people do care about the distinction, they would use --summary. I often have wished that we could make --stat imply --summary; the only reason we did not do that is because the --stat option started its life as an imitation of "diffstat". I've seen our diff-delta change its output size once. It was a nice improvement to make the delta much smaller than before, but I had to rewrite the rename similarity in diffcore not to depend on the diff-delta algorithm change, to keep it stable across diff-delta improvements (the alternative was to futz with the default threshold). I suspect we might see similar confusion if we show the delta size, depending on people's expectations. This is a very minor issue, but I thought I should mention it. There is a machine readable output format for the same --stat information called --numstat. It currently signals the binary-ness by showing '-' instead of line count. We could extend it by showing '-' + number of bytes. So here are some more suggestions: (1) --stat for binary files to show preimage and postimage sizes like this (if we were to do delta size -- otherwise drop " (.*" at the end): penguin.jpg | Bin 745245 -> 660689 (delta: 4434) (2) --numstat for binary files to show preimage and postimage sizes like this: penguin.jpg -745245 -660689 (3) independent from all of the above, make --stat imply --summary and perhaps introduce --no-summary if people do not want --summary given when they say --stat; ```