Re: mingw, windows, crlf/lf, and git

On Sun, 11 Feb 2007, Theodore Tso wrote:

I agree that a file-type approch would work, but I personally think it's too inflexible (just cr/lf vs lf? There are tons of other interesting issues that are valid). I also think it falls down on another (and in some ways much more fundamental problem): these things exist EVEN WHEN THE FILE ITSELF DOES NOT EXIST!

In other words, a policy about cr/lf is *not* a policy about actual content. It's something much more: it's a policy about representation in general, which includes *potential* content. It should obviously take effect on "git add" even with content that didn't exist before, and to work well, it should do so without the user having to think about it.

Equally importantly, this happens with content that was added by people who simply DO NOT CARE. In other words, I think a "file type" thing fundamentally cannot work, because under UNIX, it would be stupid and pointless, so any project that is maintained under UNIX might _add_ the file types, but since they won't matter, they'll inevitably be wrong (ie people forgot to mark a binary thing binary, or a text thing as text).

So: file types or attributes are broken. They cannot work well.

But enough on the negative rambling, I do have a positive and constructive suggestion, because I actually think I have a great model for it. But I've never cared enough (and since the main target would be some windows issue, I suspect I never really _will_ care enough) to really worry about it.

Anyway, if somebody really wants to look at this, and wants to create something that is actually _usable_, my suggestion is to simply extend on the ".gitignore" file approach. The great thing about .gitignore is that

 (a) you can track it like you track any other file

     This makes merges a *lot* easier. You see it as conflicts, you can 
     fix it up, and in general, you can use all the same tools with it as 
     you use with anything else. In contrast, explicit per-file filetypes 
     are _horrible_ for maintenance.

 (b) you can add to it with *patterns*, which is exactly what you want for 
     file types.

     You can do things like

	*.bin: binary
	*: text

     to say "everythgn that matches *.bin is binary, the rest is text", 
     and solves the maintenance issue trivially. Everybody will like it. 
     For the kernel, for example, we'd have a really easy

	Documentation/logo.gif: binary
	*: text

     and that would probably take care of it.

     You can also have a few default file patterns built in, which would 
     take care of it for 99% of all projects without anybody ever having 
     to even think about it - even under DOS.

 (c) it doesn't actually affect database representation, it only changes 
     behaviour for programs, which is also  exactly what you want (if you 
     have per-file "file types", you end up having serious problems at 
     merge time: when I say "affect database representation", I don't mean 
     that I think git cannot change its database, I literally mean at a 
     "higher" level: represening per-file attributes is a DISASTER from a 
     merge situation)

     So not only is it backwards-compatible with traditional git usage, 
     it's much more fundamentally simple: it doesn't add any new core data 
     structures or rules. All the core stays exactly as it is, and it just 
     affects higher-level behaviour. And that's important: one reason git 
     has been so stable is that the really core data structures are really 
     really stable and simple.

     Even when we did *really* core changes like the whole packfile thing, 
     the fundamental data structures didn't change at all *conceptually*.

 (d) it's actually a lot more flexible than file types.

     Merge stategies, anybody? We can easily have the default merge 
     strategy be the normal three-way merge (which is obviously the right 
     thing for almost anything), but how about something like

	*.doc: binary,merge=doc-merge

     which tells git that it should use a separate "doc-merge" program to 
     merge those kinds of files when it needs to do a nontrivial merge..

 (e) exactly like ".gitignore", you should also be able to have a 
     ".git/info/exclude" file that is your _private_ rules, and 
     per-directory ".gitignore" files that are the _hierarchical_ rules.

     This just makes maintenance much simpler. Not one big file that has 
     everything, and that clashes. Make the top-level one contain all the 
     generic default rules, and then lower down we can have more specific 
     rules for very specific things, exactly like the kernel .gitignore 
     files do. The top-level file should *not* have to know all the 
     details of some architecture- or sub-project specific file behaviour.

     Similarly, having an untracked file (.git/info/exclude) allows people 
     to have rules that make sense for *them*, but that might not make 
     sense for the upstream developers (say, somebody crazy enough to 
     develop Linux under Windows). So people can have their purely local 
     rules without forcing them on others.

Anyway, that would be my suggestion. Call it ".gitattributes" or something. Make it a nice ASCII format, exactly like .gitignore, and make all the rules exactly the same, except it has a ": <attributelist>" at the end for each line.

Start off supporting just "binary" and "text", but keep in mind that people may want other things. Individualized merge strategies etc.

Also, keep in mind that a *lot* of git operations will work purely on a SHA1 level, and those operations fundamentally *will*not*care* about file types. So when you merge a file, for example, the initial merge will be done purely on SHA1's, and git would do all the normal "if it didn't change in branch 1, take the branch 2 version directly" without ever even *looking* at any file rules.

This is important, because this is what makes git efficient for large projects, and which would allow git to _remain_ efficient even in the face of having to read all those comples .gitattributes files. When we merge two repositories with 20,000+ files, we usually really only "merge" a couple of the files.

Same goes for "text" mode. The "text" thing would only affect things like "git add" etc that use "git-update-index" to calculate the new SHA1. We'd never use it "normally". "git diff" would still be instantaneous, because the git index shows the file still matches, and that is all done on a SHA1 only level. So only when you do a "git add" or when it needs to refresh the index because the file changed, and it reads in the file, will it actually care about whether it's a text or a binary file.

This is actually *exactly* what you want. Not just for performance, but simply because this is also how you can take something like the Linux archive, and "just use it" under Windows, even if your editor adds (or wants) CR/LF.

Btw, how would I implement this? If I really were energetic enough to implement it, I would do:

 (a) Add a flag to "git-ls-files" logic to add "type information" in 
     front.

     Not only do you want this *anyway* for other reasons, but for
     binary/text, the thing you actually care most about is "git add", and 
     it already basically just does "take this file pattern, feed it 
     through git-ls-files, and add those files". So you'd get it basically 
     for free.

     It is also fairly easy to add at this stage, because you can simply 
     look for all the places that work with "info/exclude" and 
     ".gitignore", and you know that "Ahh, I need to teach these exact 
     places to understand about attributes". So you'd add an 
     "add_attributes_from_file()" function etc etc.

     Quite straightforward. In fact, you might be able to use the 
     gitignore parsing *as*is*, and just teach it about more flags that 
     just "ignore": both in "struct dir_entry" and in "struct exclude".

 (b) Teach the git-update-index logic about hashing text blobs.

 (c) Profit!

It really should be fairly straightforward. I'm sure it wouldn't be *entirely* trivial, but I'm also fairly sure that somebody reasonably competent could do it in a couple of days (with testing) if they were just sufficiently motivated to get started.

Anybody?

		Linus