RFC: Native clean/smudge filter for UTF-16 files

Hi,

I am working with a team that owns a repository with lots of UTF-16 files. Converting these files to UTF-8 is no good option as downstream applications require the UTF-16 encoding. Keeping the files in UTF-16 is no good option either as Git and Git related tools (e.g. GitHub) consider the files binary and consequently do not render diffs.

The obvious solution is to setup a clean/smudge filter like this [1]:
    [filter "winutf16"]
        clean = iconv -f utf-16 -t utf-8
        smudge = iconv -f utf-8 -t utf-16

In general this works well but the "per-file" clean/smudge process invocation can be slow for many files. I could apply the same trick that we used for Git LFS and write a iconv that processes all files with a single invocation (see "edcc85814c convert: add filter.<driver>.process option" [2]).

Alternatively, I could add a native attribute to Git that translates UTF-16 to UTF-8 and back. A conversion function is already available in "mingw.h" [3] on Windows. Limiting this feature to Windows wouldn't be a problem from my point of view as UTF-16 is only relevant on Windows anyways. The attribute could look like this:

    *.txt        text encoding=utf-16

There was a previous discussion on the topic and Jonathan already suggested a "native" clean/smudge filter in 2010 [4]. Also the "encoding" attribute is already present but, as far as I can tell, is only used by the git gui for viewing [5].

Do you think a patch that converts UTF-16 files to UTF-8 via an attribute "encoding=utf-16" on Windows would have a chance to get accepted?

Thanks, Lars

[1] https://github.com/msysgit/msysgit/issues/113#issue-13142846 [2] https://github.com/git/git/commit/edcc85814c87ebd7f3b1b7d3979fac3dfb84d308 [3] https://github.com/git/git/blob/14c63a9dc093d6738454f6369a4f5663ca732cf7/compat/mingw.h#L501-L533 [4] https://public-inbox.org/git/20101022195331.GA12014@burratino/ [5] https://github.com/git/git/commit/1ffca60f0b0395e1e593e64d66e7ed3c47d8517e