Re: Tackling Git Limitations with Singular Large Line-seperated Plaintext files

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Jun 27, 2014, 19:47 UTC
Message-ID: <CA+55aFwFne6gj6P_Vm+uGffbF--vd-yke0899k==iVgHkb+gWQ@mail.gmail.com>
In-Reply-To: <CA+55aFx6vFyZvpyQot_3Ym7wsCZ06abjNx_hEKkza-N856jMnw@mail.gmail.com>

On Fri, Jun 27, 2014 at 12:38 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:

>
> I think it might be possible to just specify a special diff algorithm
> (git already supports that, obviously), and just introduce a new "use
> binary diffs with a textual representation" model.

Another model would be to just insert newlines in the data, and use the regular textual diff on that "preprocessed" format.

The problem of *where* to insert the newlines is somewhat interesting, since the stupid approaches ("chunk it up in 64-byte lines") don't work with data insertion/deletion (all the lines will now be different just because the data is offset), but there are algorithms that handle that reasonably well, like breaking lines at certain well-defined patterns (the patterns can then be defined either explicitly or algorithmically - like calculating a hash/crc over the last rolling N characters and breaking if the result matches some modulo calculation).

                Linus