{"thread":{"id":"53848","subject":"Help understanding unexpected diff output","startedAt":"2020-07-13T10:40:44Z","lastAt":"2020-10-23T14:24:09Z","messageCount":5,"participants":["Ulrich Windl","Junio C Hamano","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"401484","messageId":"5F0C38BB020000A100039FF0@gwsmtp.uni-regensburg.de","threadId":"53848","inReplyTo":null,"subject":"Help understanding unexpected diff output","fromName":"Ulrich Windl","fromEmail":"ulrich.windl@rz.uni-regensburg.de","sentAt":"2020-07-13T10:34:35Z","receivedAt":"2020-07-13T10:40:44Z","isPatch":false,"sender":{"key":"ulrich.windl@rz.uni-regensburg.de","avatar":null},"body":"Hi!\n\nI hope we all agree that diff should make our lives easier, so here is something where I think it does not apply:\n\nI inspected some changes using \"git diff\", and it was hard to me to find the actual differences (see git-diff-png):\nI had to look a long time at those olcAccess lines to find any difference, so finally I pasted both blocks into separate Emacs files a and b (via PuTTY) and used \"ediff-files\" (see diff-emacs.png).\nWithin there, the was only a part of one line changed!\n\nI don't understand this. I had even used \"git diff -w\" to ignore space changes (which aren't expected anyway).\n\nAny ideas what might be wrong and how to get better diff output?\n\n(git-2.26.2-27.36.1.x86_64 of SLES12 being used)\n\nRegards,\nUlrich\n\n\n\n\n\n"},{"id":"401488","messageId":"xmqqsgdvo3ec.fsf@gitster.c.googlers.com","threadId":"53848","inReplyTo":"5F0C38BB020000A100039FF0@gwsmtp.uni-regensburg.de","subject":"Re: Help understanding unexpected diff output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-13T15:12:11Z","receivedAt":"2020-07-13T15:12:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Ulrich Windl\" writes:\n\n> I inspected some changes using \"git diff\", and it was hard to me\n> to find the actual differences (see git-diff-png): I had to look a\n> long time at those olcAccess lines to find any difference, so\n> finally I pasted both blocks into separate Emacs files a and b\n> (via PuTTY) and used \"ediff-files\" (see diff-emacs.png). Within\n> there, the was only a part of one line changed!\n\nWith redactions and making them into two images, you did a good job\nkilling interested parties' willingness to help, as they made using\ntools almost impossible.\n\nSo here is only based on eyeballing the images, and I probably have\nmissed what you wanted readers to find, but don't blame me ;-)\n\nIt seems that the file got 6 lines removed, that begins with\n\"olcAccess: {$n}\" (0 <= n <= 6), at around 8th line of the file, and\nthen very similar 6 lines that are not identical to what got removed\nat the end of the file. The ediff you showed tells me that the 6\nlines removed from very early part of the file and 6 lines added to\nthe end of the file are similar but not identical. These two 6-line\nblocks differ only by the 5th line (i.e. \"olcAccess: {4}\") having\n\",pwdGraceUseTime,pwdChangedTime\" in the newer version, but not in\nthe older version.\n\nAm I reading what you sent us correctly so far?\n\nNow, if this change to the \"olcAccess: {4}\" line in the new file\nweren't there, i.e. if the 6-line block that disappeared from the\nearly part of the file and the 6-line block that was added near the\nend of the file were identical (in such a case, your ediff would\nshow no changes, I presume), then what would your desired output\nfrom \"git diff\"?\n\nI would imagine that the answer would still be a very similar looking\n\n --- a/0/...\n +++ b/0/...\n @@ -5,12 +5,6 @@\n olcDatabase: {1}hdb\n olcDbDirectory: /var/lib/ldap\n olcSuffix: dc=... \n -olcAccess: {0} ...\n -olcAccess: {1} ...\n -olcAccess: {2} ...\n -olcAccess: {3} ...\n -olcAccess: {4} ...\n -olcAccess: {5} ...\n olcLimits: {0}dn.exact=....\n olcRootDn: cn=Admin,dc=...\n olcRootPW: {SSHA}yZ...\n @@ -52,6 +46,12 @@\n entryUUID: ...\n creatorsName: cn=config\n createTimestamp: ...\n -entryCSN: ...\n +olcAccess: {0} ...\n +olcAccess: {1} ...\n +olcAccess: {2} ...\n +olcAccess: {3} ...\n +olcAccess: {4} ...\n +olcAccess: {5} ...\n +entryCSN: ...\n modifiersName: cn=config\n -modifyTimestamp: ...\n +modifyTimestamp: ...\n\nwith identical \"olcAccess: {$n}\" lines removed from earlier part the\nfile and added to the end of the file.\n\nSo I am not sure what you are asking. It would not be correct to show\nsomething like\n\n --- a/0/...\n +++ b/0/...\n @@ -52,6 +46,6 @@\n entryUUID: ...\n creatorsName: cn=config\n createTimestamp: ...\n -entryCSN: ...\n +entryCSN: ...\n modifiersName: cn=config\n -modifyTimestamp: ...\n +modifyTimestamp: ...\n\nand pretend that the original \"olcAccess: {$n}\" lines did not get\nmoved from the earlier part of the file to the end of the file.\n\nPerhaps you may be looking for \"git diff --color-moved\"? I dunno.\n"},{"id":"401542","messageId":"5F0D7252020000A10003A03E@gwsmtp.uni-regensburg.de","threadId":"53848","inReplyTo":"xmqqsgdvo3ec.fsf@gitster.c.googlers.com","subject":"Antw: [EXT] Re: Help understanding unexpected diff output","fromName":"Ulrich Windl","fromEmail":"ulrich.windl@rz.uni-regensburg.de","sentAt":"2020-07-14T08:52:34Z","receivedAt":"2020-07-14T08:52:41Z","isPatch":false,"sender":{"key":"ulrich.windl@rz.uni-regensburg.de","avatar":null},"body":">>> Junio C Hamano schrieb am 13.07.2020 um 17:12 in\nNachricht\n:\n> \"Ulrich Windl\" writes:\n> \n>> I inspected some changes using \"git diff\", and it was hard to me\n>> to find the actual differences (see git‑diff‑png): I had to look a\n>> long time at those olcAccess lines to find any difference, so\n>> finally I pasted both blocks into separate Emacs files a and b\n>> (via PuTTY) and used \"ediff‑files\" (see diff‑emacs.png). Within\n>> there, the was only a part of one line changed!\n> \n> With redactions and making them into two images, you did a good job\n> killing interested parties' willingness to help, as they made using\n> tools almost impossible.\n\nJunio,\n\nwe had some discussions in the past, and you should know that I'm willing to\nprovide information if needed.\nPart of the problem is that I can't (to my knowledge) copy and paste colored\ntext into plain text emails. Thus screenshots.\nMy intention was to bring up the issue, making it (more or less obvious) that\nthe differences are much harder to see in git output than in Emacs output.\nThe redaction was due to a password that could easily be cracked if presented\nand some resource name I wanted to avoid to get a clearance to publish first.\nIt still could be a \"stupid me\", but can't we talk like adults?\n\n> \n> So here is only based on eyeballing the images, and I probably have\n> missed what you wanted readers to find, but don't blame me ;‑)\n> \n> It seems that the file got 6 lines removed, that begins with\n> \"olcAccess: {$n}\" (0 <= n <= 6), at around 8th line of the file, and\n> then very similar 6 lines that are not identical to what got removed\n> at the end of the file. The ediff you showed tells me that the 6\n> lines removed from very early part of the file and 6 lines added to\n> the end of the file are similar but not identical. These two 6‑line\n> blocks differ only by the 5th line (i.e. \"olcAccess: {4}\") having\n> \",pwdGraceUseTime,pwdChangedTime\" in the newer version, but not in\n> the older version.\n> \n> Am I reading what you sent us correctly so far?\n\nYes, if you look at the highlighted part (difference in cyan, \"chink\" in\nyellow) in the Emacs screenshot, it should point out the \"real\" differences,\nwhile git output made it harder to see.\n\n> \n> Now, if this change to the \"olcAccess: {4}\" line in the new file\n> weren't there, i.e. if the 6‑line block that disappeared from the\n> early part of the file and the 6‑line block that was added near the\n> end of the file were identical (in such a case, your ediff would\n> show no changes, I presume), then what would your desired output\n> from \"git diff\"?\n> \n> I would imagine that the answer would still be a very similar looking\n> \n> ‑‑‑ a/0/...\n> +++ b/0/...\n> @@ ‑5,12 +5,6 @@\n> olcDatabase: {1}hdb\n> olcDbDirectory: /var/lib/ldap\n> olcSuffix: dc=... \n> ‑olcAccess: {0} ...\n> ‑olcAccess: {1} ...\n> ‑olcAccess: {2} ...\n> ‑olcAccess: {3} ...\n> ‑olcAccess: {4} ...\n> ‑olcAccess: {5} ...\n> olcLimits: {0}dn.exact=....\n> olcRootDn: cn=Admin,dc=...\n> olcRootPW: {SSHA}yZ...\n> @@ ‑52,6 +46,12 @@\n> entryUUID: ...\n> creatorsName: cn=config\n> createTimestamp: ...\n> ‑entryCSN: ...\n> +olcAccess: {0} ...\n> +olcAccess: {1} ...\n> +olcAccess: {2} ...\n> +olcAccess: {3} ...\n> +olcAccess: {4} ...\n> +olcAccess: {5} ...\n> +entryCSN: ...\n> modifiersName: cn=config\n> ‑modifyTimestamp: ...\n> +modifyTimestamp: ...\n> \n> with identical \"olcAccess: {$n}\" lines removed from earlier part the\n> file and added to the end of the file.\n> \n> So I am not sure what you are asking. It would not be correct to show\n> something like\n\nAccording to my eyes (just using Emacs to make sure) those olcAccess lines\n\"numbered\" 0 to 3, and 5 should be \"context lines\" as they are not changed.\nOnly line numbered {4} has a difference, and as the order 0 to 5 is unchanged I\ndon't see why the diff-block is that large. I see that some lines that are not\nchanged were reorderd, but that should not affect the olcAccess block. Maybe I\njust need different diff options; I don't know.\n\n\n> \n> ‑‑‑ a/0/...\n> +++ b/0/...\n> @@ ‑52,6 +46,6 @@\n> entryUUID: ...\n> creatorsName: cn=config\n> createTimestamp: ...\n> ‑entryCSN: ...\n> +entryCSN: ...\n> modifiersName: cn=config\n> ‑modifyTimestamp: ...\n> +modifyTimestamp: ...\n> \n> and pretend that the original \"olcAccess: {$n}\" lines did not get\n> moved from the earlier part of the file to the end of the file.\n> \n> Perhaps you may be looking for \"git diff ‑‑color‑moved\"? I dunno.\n\nI kind of understand: The 5-line block of olcAccess was moved within a larger\nblock, preventing diff from seeing the small change within.\nI'm attaching the \"git diff --color-moved\" output as another screenshot (this\ntime with very wide terminal to make one line a line). Still I redacted\n(\"blued-out\") some information I may not publish, but believe me that the parts\n\"blued out\" are identical.\n\nIn addition I made a test-case that includes only the first 18 columns (cut\n-c1-18) of the file, both before (b0) and after (b1) the commit. That should be\ngood enough for the reordering effect. A plain \"diff -u shows\":\n# diff -u b0 b1\n--- b0 2020-07-14 10:43:05.000000000 +0200\n+++ b1 2020-07-14 10:41:37.000000000 +0200\n@@ -5,12 +5,6 @@\n olcDatabase: {1}hd\n olcDbDirectory: /v\n olcSuffix: dc=sap,\n-olcAccess: {0}to *\n-olcAccess: {1}to d\n-olcAccess: {2}to a\n-olcAccess: {3}to a\n-olcAccess: {4}to a\n-olcAccess: {5}to *\n olcLimits: {0}dn.e\n olcRootDN: cn=Admi\n olcRootPW: {SSHA}y\n@@ -52,6 +46,12 @@\n entryUUID: db3ffe4\n creatorsName: cn=c\n createTimestamp: 2\n-entryCSN: 20170706\n+olcAccess: {0}to *\n+olcAccess: {1}to d\n+olcAccess: {2}to a\n+olcAccess: {3}to a\n+olcAccess: {4}to a\n+olcAccess: {5}to *\n+entryCSN: 20200114\n modifiersName: cn=\n modifyTimestamp: 2\n\nSo it seems the actual problem was that the block had moved, and I did not\nnotice that when looking at the diffs. An \"sdiff\" illustrates:\n # sdiff b0 b1\n\ndn: olcDatabase={1 dn:\nolcDatabase={1\nobjectClass: olcDa objectClass:\nolcDa\nobjectClass: olcHd objectClass:\nolcHd\nolcDatabase: {1}hd olcDatabase:\n{1}hd\nolcDbDirectory: /v \nolcDbDirectory: /v\nolcSuffix: dc=sap, olcSuffix:\ndc=sap,\nolcAccess: {0}to * <\nolcAccess: {1}to d <\nolcAccess: {2}to a <\nolcAccess: {3}to a <\nolcAccess: {4}to a <\nolcAccess: {5}to * <\nolcLimits: {0}dn.e olcLimits:\n{0}dn.e\nolcRootDN: cn=Admi olcRootDN:\ncn=Admi\nolcRootPW: {SSHA}y olcRootPW:\n{SSHA}y\nolcSecurity: ssf=1 olcSecurity:\nssf=1\nolcSyncrepl: {0}ri olcSyncrepl:\n{0}ri\nolcSyncrepl: {1}ri olcSyncrepl:\n{1}ri\nolcSyncrepl: {2}ri olcSyncrepl:\n{2}ri\nolcUpdateRef: ldap olcUpdateRef:\nldap\nolcUpdateRef: ldap olcUpdateRef:\nldap\nolcUpdateRef: ldap olcUpdateRef:\nldap\nolcMirrorMode: TRU olcMirrorMode:\nTRU\nolcDbCacheSize: 10 \nolcDbCacheSize: 10\nolcDbCheckpoint: 1 \nolcDbCheckpoint: 1\nolcDbConfig: {0}se olcDbConfig:\n{0}se\nolcDbConfig: {1}se olcDbConfig:\n{1}se\nolcDbConfig: {2}se olcDbConfig:\n{2}se\nolcDbConfig: {3}se olcDbConfig:\n{3}se\nolcDbConfig: {4}se olcDbConfig:\n{4}se\nolcDbConfig: {5}se olcDbConfig:\n{5}se\nolcDbIDLcacheSize: \nolcDbIDLcacheSize:\nolcDbIndex: object olcDbIndex:\nobject\nolcDbIndex: uidNum olcDbIndex:\nuidNum\nolcDbIndex: gidNum olcDbIndex:\ngidNum\nolcDbIndex: member olcDbIndex:\nmember\nolcDbIndex: member olcDbIndex:\nmember\nolcDbIndex: cn eq, olcDbIndex: cn\neq,\nolcDbIndex: uid eq olcDbIndex:\nuid eq\nolcDbIndex: sn eq, olcDbIndex: sn\neq,\nolcDbIndex: givenN olcDbIndex:\ngivenN\nolcDbIndex: entryU olcDbIndex:\nentryU\nolcDbIndex: entryC olcDbIndex:\nentryC\nolcDbIndex: ipServ olcDbIndex:\nipServ\nolcDbIndex: ipServ olcDbIndex:\nipServ\nolcDbIndex: roleOc olcDbIndex:\nroleOc\nolcDbIndex: mail e olcDbIndex:\nmail e\nolcDbIndex: displa olcDbIndex:\ndispla\nolcDbIndex: modify olcDbIndex:\nmodify\nstructuralObjectCl \nstructuralObjectCl\nentryUUID: db3ffe4 entryUUID:\ndb3ffe4\ncreatorsName: cn=c creatorsName:\ncn=c\ncreateTimestamp: 2 \ncreateTimestamp: 2\nentryCSN: 20170706 | olcAccess:\n{0}to *\n > olcAccess:\n{1}to d\n > olcAccess:\n{2}to a\n > olcAccess:\n{3}to a\n > olcAccess:\n{4}to a\n > olcAccess:\n{5}to *\n > entryCSN:\n20200114\nmodifiersName: cn= modifiersName:\ncn=\nmodifyTimestamp: 2 \nmodifyTimestamp: 2\n\nRegards,\nUlrich\n\n\n\n\n"},{"id":"401557","messageId":"xmqqr1tekjwo.fsf@gitster.c.googlers.com","threadId":"53848","inReplyTo":"5F0D7252020000A10003A03E@gwsmtp.uni-regensburg.de","subject":"Re: Antw: [EXT] Re: Help understanding unexpected diff output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-14T18:53:43Z","receivedAt":"2020-07-14T19:05:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Ulrich Windl\" writes:\n\n>> I would imagine that the answer would still be a very similar looking\n>> \n>> ‑‑‑ a/0/...\n>> +++ b/0/...\n>> @@ ‑5,12 +5,6 @@\n>> olcDatabase: {1}hdb\n>> olcDbDirectory: /var/lib/ldap\n>> olcSuffix: dc=... \n>> ‑olcAccess: {0} ...\n>> ‑olcAccess: {1} ...\n>> ‑olcAccess: {2} ...\n>> ‑olcAccess: {3} ...\n>> ‑olcAccess: {4} ...\n>> ‑olcAccess: {5} ...\n>> olcLimits: {0}dn.exact=....\n>> olcRootDn: cn=Admin,dc=...\n>> olcRootPW: {SSHA}yZ...\n>> @@ ‑52,6 +46,12 @@\n>> entryUUID: ...\n>> creatorsName: cn=config\n>> createTimestamp: ...\n>> ‑entryCSN: ...\n>> +olcAccess: {0} ...\n>> +olcAccess: {1} ...\n>> +olcAccess: {2} ...\n>> +olcAccess: {3} ...\n>> +olcAccess: {4} ...\n>> +olcAccess: {5} ...\n>> +entryCSN: ...\n>> modifiersName: cn=config\n>> ‑modifyTimestamp: ...\n>> +modifyTimestamp: ...\n>> \n>> with identical \"olcAccess: {$n}\" lines removed from earlier part the\n>> file and added to the end of the file.\n>> \n>> So I am not sure what you are asking. It would not be correct to show\n>> something like\n>\n> According to my eyes (just using Emacs to make sure) those olcAccess lines\n> \"numbered\" 0 to 3, and 5 should be \"context lines\" as they are not changed.\n\nAnd the \"slightly different\" example I asked you about has {4} in\nboth blocks identical, you are arguing that these 6 lines all should\nappear as context?\n\nIt certainly is possible but then the patch would look quite\ndifferent. As \"olcAccess: {0}\" line in the preimage (i.e. a/)\nappears at line 8, but that same line appears in the postimage\n(i.e. b/) at line 49, such a patch that match these two olcAccess\nblocks as unchanged MUST delete many lines that come after\n\"olcAccess: {5}\" in the preimage (i.e. starting at line 14 with\nolcLimits, line ending at entryCSN: at line 55 must be removed,\nbecause they used to appear immediately after \"olcAccess: {5}\" in\nthe preimage a/, but in the postimage b/, none of tme appear after\nthe \"olcAccess: {5}\" that you claim to be common and unchanged (in\nthe postimage, instead you have only four lines that has\"entryCSN:\",\n\"modiferName\", etc. before the end of the file).\n\nOf course, these lines in the line range 14-55 actually are the ones\nthat did not change, as we can see above, so if you insist that you\nmust keep the 6-line \"olcAccess\" block as common and unchanged,\nbecause your desired patch is deleting them from the pre-image after\n\"olcAccess: {5}\" line, your desired patch must be adding them back\nto the postimage somewhere (namely, before \"olcAccess: {0}\" line).\n\nSuch a patch is also a valid one in that it expresses the difference\nbetween a/ and b/ in terms of a sequence of \"delete these lines from\nhere\" and \"insert these lines to here\", but it would be far less\ninteresting than what we see above. Instead of \"we deleted 6 lines\nnear the beginning of the file\" plus \"then added 6 lines near the\nend--ah, these 6 lines by the way are identical\", you would instead\nsay \"we deleted the block of lines 14-55 that appear immediately\nafter olcAccess:{5}\" plus \"we inserted the identical block of lines\nimmediately before olcAccess{0}\". And you'd complain in the\nopposite way: \"These lines are identical but appear in different\nlocations in the preimage and the postimage -- why aren't they shown\nas context lines?\"\n"},{"id":"408235","messageId":"1be0d453-692a-faf7-c72d-225ee7aa12d6@iee.email","threadId":"53848","inReplyTo":"5F0D7252020000A10003A03E@gwsmtp.uni-regensburg.de","subject":"Re: Antw: [EXT] Re: Help understanding unexpected diff output","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2020-10-23T14:24:05Z","receivedAt":"2020-10-23T14:24:09Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Ulrich,\n(only just catching up with intriguing queries)\n\nOn 14/07/2020 09:52, Ulrich Windl wrote:\n> some lines that are not\n> changed were reorderd\n\nIt may be me, but I think your comment above is at the core of the\nunexpected diff output.\n\nIf text is moved, then normally it will (obviously?) require something\nwithin the diff report to say where it was removed from, and then where\nit was inserted. It seems like your expectation in your use-case was\nthat the moved line changes wouldn't be shown.\n\nI don't believe there is an `--ignore-moved-lines` option, but if\nthere's a good rationale / use-case for it, then maybe it's something to\nlook at coding.\n\n--\nPhilip\n"}]}