Use of new .gitattributes working-tree-encoding attribute across different platform types

> Hi, 
> 
> Sorry for incomplete post earlier. Here is the full post:
> 
> 
> In the latest version of git a new attribute has been added, working-tree-encoding. The release notes states: 
> 
> 'The new "working-tree-encoding" attribute can ask Git to convert the
>    contents to the specified encoding when checking out to the working
>    tree (and the other way around when checking in).'
>  We have been using this attribute on our z/OS systems using a version of git from Rocket software to convert files to EBCDIC for quite a while now. On other platforms (Linux, AIX etc) git ignored this attribute and therefore left the files in ASCII.
> 
> We have common code that is supposed to be usable across different platforms and hence different file encodings. With the full support of the working-tree-encoding in the latest version of git on all platforms, how do we have files converted to different encodings on different platforms?
> I could not find anything that would allow us to say 'if platform = z/OS then encoding=EBCDIC else encoding=ASCII'.   Is there a way this can be done?
>  
>  
>   
>  
> Thanks
>  Steve Groeger

>  Java Runtimes Development
>  IBM Hursley
>  IBM United Kingdom Ltd
>  Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
>  Fax (44) 1962 816800
>  Lotus Notes: Steve Groeger/UK/IBM
>  Internet: groeges@uk.ibm.com  
>    
>  
> Unless stated otherwise above:
>  IBM United Kingdom Limited - Registered in England and Wales with number 741598.
>  Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU      
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> 

> On Wed, Jun 27, 2018 at 07:54:52AM +0000, Steve Groeger wrote:
> > We have common code that is supposed to be usable across different platforms and hence different file encodings. With the full support of the working-tree-encoding in the latest version of git on all platforms, how do we have files converted to different encodings on different platforms?
> > I could not find anything that would allow us to say 'if platform = z/OS then encoding=EBCDIC else encoding=ASCII'.   Is there a way this can be done?
> 
> I don't believe there is such functionality.  Git doesn't have
> attributes that are conditional on the platform in that sort of way.
> You could use a smudge/clean filter and adjust the filter for the
> platform you're on, which might meet your needs.

> On Jun 28, 2018, at 4:34 PM, Jeff King <peff@peff.net> wrote:
> 
> On Thu, Jun 28, 2018 at 02:44:47AM +0000, brian m. carlson wrote:
> 
>> On Wed, Jun 27, 2018 at 07:54:52AM +0000, Steve Groeger wrote:
>>> We have common code that is supposed to be usable across different platforms and hence different file encodings. With the full support of the working-tree-encoding in the latest version of git on all platforms, how do we have files converted to different encodings on different platforms?
>>> I could not find anything that would allow us to say 'if platform = z/OS then encoding=EBCDIC else encoding=ASCII'.   Is there a way this can be done?
>> 
>> I don't believe there is such functionality.  Git doesn't have
>> attributes that are conditional on the platform in that sort of way.
>> You could use a smudge/clean filter and adjust the filter for the
>> platform you're on, which might meet your needs.
> 
> We do have prior art in the line-ending code, though. There the
> attributes say either that a file needs a specific line-ending type
> (which is relatively rare), or that it should follow the system type,
> which is then set separately in the config.
> 
> I have the impression that the working-tree-encoding stuff was made to
> handle the first case, but not the second. It doesn't seem like an
> outrageous thing to eventually add.
> 
> (Though I agree that clean/smudge filters would work, and can even
> implement the existing working-tree-encoding feature, albeit less
> efficiently and conveniently).

> How about this:
> 
> 1) We allow users to set the encoding "auto". Example:
> 
> 	*.txt working-tree-encoding=auto
> 
> 2) We define a new variable `core.autoencoding`. By default the value is 
> UTF-8 (== no re-encoding) but user can set to any value in their Git config. 
> Example:
> 
>     git config --global core.autoencoding UTF-16
> 
> All files marked with the value "auto" will use the encoding defined in
> `core.autoencoding`.
> 
> Would that work?

> Yeah, that was along the lines that I was thinking. I wonder if anybody
> would ever need two such auto-encodings, though. Probably not. But
> another way to think about it would be to allow something like:
> 
>   working-tree-encoding=foo
> 
> and then in your config "foo" to map to some encoding.
> 
> But that may be over-engineering, I dunno. utf8 has always been enough
> for me. :)

> On Thu, Jun 28, 2018 at 01:27:07PM -0400, Jeff King wrote:
> > Yeah, that was along the lines that I was thinking. I wonder if anybody
> > would ever need two such auto-encodings, though. Probably not. But
> > another way to think about it would be to allow something like:
> > 
> >   working-tree-encoding=foo
> > 
> > and then in your config "foo" to map to some encoding.
> > 
> > But that may be over-engineering, I dunno. utf8 has always been enough
> > for me. :)
> 
> I had a thought the other day about why this solution might be valuable.
> Different platforms encode different values for iconv character sets.
> So, for example, one may have platforms supporting some disjoint sets of
> the following:
> 
> * LATIN-1
> * LATIN1
> * ISO8859-1
> * ISO-8859-1
> * ISO_8859-1
> * ISO_8859-1:1987
> * some lowercase variants of these
> 
> Therefore, specifying a working-tree-encoding value that works across a
> wide variety of system may be non-trivial.  This is less of a problem
> with UTF-8, but having the ability to pick an encoding and remap it to a
> supported value may be useful nevertheless.

> One thing I almost did in the example I gave above was to literally call
> the encoding name by a "real" one. I.e.:
>
>   echo '*.txt working-tree-encoding=iso-8859-1' >.gitattributes
>   git config encoding.iso-8859-1.replace latin1
>
> or something. But I wondered if it was a little crazy as a practice,
> since mapping "iso-8859-1" to "utf-8" is probably going to lead to
> headaches.
>
> But your example above of semantically equivalent variants with
> different spellings would be a good use of that trick.

> On Jun 28, 2018, at 4:34 PM, Jeff King <peff@peff.net> wrote:
> 
> On Thu, Jun 28, 2018 at 02:44:47AM +0000, brian m. carlson wrote:
> 
>> On Wed, Jun 27, 2018 at 07:54:52AM +0000, Steve Groeger wrote:
>>> We have common code that is supposed to be usable across different platforms and hence different file encodings. With the full support of the working-tree-encoding in the latest version of git on all platforms, how do we have files converted to different encodings on different platforms?
>>> I could not find anything that would allow us to say 'if platform = z/OS then encoding=EBCDIC else encoding=ASCII'.   Is there a way this can be done?
>> 
>> I don't believe there is such functionality.  Git doesn't have
>> attributes that are conditional on the platform in that sort of way.
>> You could use a smudge/clean filter and adjust the filter for the
>> platform you're on, which might meet your needs.
> 
> We do have prior art in the line-ending code, though. There the
> attributes say either that a file needs a specific line-ending type
> (which is relatively rare), or that it should follow the system type,
> which is then set separately in the config.
> 
> I have the impression that the working-tree-encoding stuff was made to
> handle the first case, but not the second. It doesn't seem like an
> outrageous thing to eventually add.
> 
> (Though I agree that clean/smudge filters would work, and can even
> implement the existing working-tree-encoding feature, albeit less
> efficiently and conveniently).

> -----Lars Schneider <larsxschneider@gmail.com> wrote: -----
> To: Jeff King <peff@peff.net>
> From: Lars Schneider <larsxschneider@gmail.com>
> Date: 06/28/2018 18:21
> Cc: "brian m. carlson" <sandals@crustytoothpaste.net>, Steve Groeger <GROEGES@uk.ibm.com>, git@vger.kernel.org
> Subject: Re: Use of new .gitattributes working-tree-encoding attribute across different platform types
> 
> 
>> On Jun 28, 2018, at 4:34 PM, Jeff King <peff@peff.net> wrote:
>> 
>> On Thu, Jun 28, 2018 at 02:44:47AM +0000, brian m. carlson wrote:
>> 
>>> On Wed, Jun 27, 2018 at 07:54:52AM +0000, Steve Groeger wrote:
>>>> We have common code that is supposed to be usable across different platforms and hence different file encodings. With the full support of the working-tree-encoding in the latest version of git on all platforms, how do we have files converted to different encodings on different platforms?
>>>> I could not find anything that would allow us to say 'if platform = z/OS then encoding=EBCDIC else encoding=ASCII'.   Is there a way this can be done?
>>> 
>>> I don't believe there is such functionality.  Git doesn't have
>>> attributes that are conditional on the platform in that sort of way.
>>> You could use a smudge/clean filter and adjust the filter for the
>>> platform you're on, which might meet your needs.
>> 
>> We do have prior art in the line-ending code, though. There the
>> attributes say either that a file needs a specific line-ending type
>> (which is relatively rare), or that it should follow the system type,
>> which is then set separately in the config.
>> 
>> I have the impression that the working-tree-encoding stuff was made to
>> handle the first case, but not the second. It doesn't seem like an
>> outrageous thing to eventually add.
>> 
>> (Though I agree that clean/smudge filters would work, and can even
>> implement the existing working-tree-encoding feature, albeit less
>> efficiently and conveniently).
> 
> Thanks for the suggestion Peff! 
> How about this:
> 
> 1) We allow users to set the encoding "auto". Example:
> 
> 	*.txt working-tree-encoding=auto
> 
> 2) We define a new variable `core.autoencoding`. By default the value is 
> UTF-8 (== no re-encoding) but user can set to any value in their Git config. 
> Example:
> 
>    git config --global core.autoencoding UTF-16
> 
> All files marked with the value "auto" will use the encoding defined in
> `core.autoencoding`.
> 
> Would that work?
> 
> @steve: Would that fix your problem?

> Brian had a good argument [1] for an even more flexible system
> proposed by Peff:
> 
> 
> 1) We allow users to define custom encoding mappings in their Git config. 
> Example:
> 
>     git config --global core.encoding.myenc UTF-16