.gitignore, .gitattributes, .gitmodules, .gitprecious?, .gitacls? etc.

Git is using more and more .git* files to store metadata.
I have two comments on this:
1. It may be better to combine all these files into one (.gitmeta) with different sections
2. Storing metadata in regular source-controlled files feels wrong to me. I cannot be very specific why (call it intuition), but 
something inside me screams "bad design" :-). We've already seen some chicken-and-the-egg problems with crlf and .gitattributes. So, 
may be it would be better to keep this metadata only in repository (and index). One option would be to associate such a META object 
with every TREE (an make it optional). Then allow editing this object with something like 'git meta'.

- Dmitry 

Dmitry Kakurin <dmitry.kakurin@gmail.com> writes:

Merging what has traditionally been known as .gitignore's capability to attributes has been discussed, and I think it would make sense in longer term, as 'this path pattern is to be ignored' is just a special case of a more general attribute. And "precious" handling would naturally fit there. However, as the .gitignore and .git/info/exclude has been as old as git itself (I think it was introduced around early May 2005), I do not see us even start talking about deprecating .gitignore.

I do not think .gitmodules fits the model of what .gitattributes solves. .gitattributes is about the attribute of paths, while .gitmodules is about attribute of subprojects, and one attribute of a subproject is where in the superproject directory hierarchy it sits at.

I do not know what you are talking about with .gitacls. Personally I am not interested in turning git into a back-up program at all, so if you are talking beyond what has already been suggested as "owner", "group" and "perms" attributes that could be stored in .gitattributes, I do not think it belongs to git.

You are free to _feel_ whatever you want without thinking, but please keep that _feeling_ to yourself, and speak it out after making it into an _opinion_, which would take a bit of thinking about it first. For example, think about what you could do without confusing a total newbie after the initial clone. You cannot avoid chicken-and-egg problem. I think reading from index as a fallback measure when work tree file is missing is a very good compromise we came up recently. The wish of the user (i.e. the owner of the work tree) overrides what is in the index, and the index is how the repository contents are initially propagated back to the work tree.

----- Original Message ----- 
From: "Junio C Hamano" <gitster@pobox.com>
 >> 2. Storing metadata in regular source-controlled files feels wrong to

I did quite a bit of thinking before posting it. Not sure what made you think otherwise.

Can you specify _exactly_ how it works now? And I'll show you a bunch of corner cases where it's broken.

- Dmitry

Dmitry Kakurin <dmitry.kakurin@gmail.com> writes:

As I made it clear, it is a compromise. I am not interested in discussing corner cases with you -- I am sure there are.

If you are offering improvements, I am all ears.

----- Original Message ----- 
From: "Junio C Hamano" <gitster@pobox.com>

Usually many corner cases == design flaw.

I thought I did: I've observed that these problem are caused by storing metadata in regular files (that exist both in repo/index and in workplace). My knowledge of Git internals is quite limited, but if *I* were to do it right now, I'd introduce a META entry in every TREE object that would point to a BLOB that contains combined content of .gitattributes, .gitignore etc. Then command like 'git meta' would open vi for this BLOB, let you edit it, and then would upload it back to index.

- Dmitry

Dmitry Kakurin <dmitry.kakurin@gmail.com> writes:

And that observation solves the initial checkout issue how?

A tree that has .gitattributes (and I am assuming in the longer term you can use "ignore" and "precious" in .gitattributes instead of using .gitignore) POINTS TO A BLOB already, so what you are saying does not add anything to what we already have, other than that you are renaming .gitattributes to "META ENTRY".

When you do "git checkout -- this-path", you are checking things out from the index and at that point you may not have _any_ tree yet (think "before initial commit"). A "META ENTRY" that exists only in a tree does not work -- it has to come to index somehow for it to work with how git works.

----- Original Message ----- 
From: "Junio C Hamano" <gitster@pobox.com>

There is always only one place to check for metadata - the index.

Almost true! The difference is: META BLOBS are not created as files in the workspace (not during checkout, not ever). In order to edit it you'd have to use 'git meta' command. So once again, there is only one place to check for metadata - the index.

I don't fully understand the difficulty. Here is how I see the initial checkin to work:
Let's say you just did 'git init' and want to add file.txt with unusual crlf. So you don't want automatic translation.
What you'll do is:
* git meta
  In vi you put 'file.txt -crlf'
  git creates a TREE object in the index with only entry META that point to BLOB with this file.
  so it's kind of an empty tree from user's perspective.
* git add file.txt
  while adding the file git consults META BLOB from the TREE (stored in index) and does not do any crlf translation
  the TREE now contains 2 entries: META, and BLOB for file.txt
* git commit
   nothing special here

Now someone else doing initial checkout:
* git checkout
  Before writing file.txt to disk git consults META BLOB (in the index) and does not do any crlf translation.

- Dmitry

On Sun, Aug 26, 2007 at 10:02:50AM CEST, Dmitry Kakurin wrote:

That sounds so incredibly ugly, I really would hate to see that.

It's still not clear to me how this would help anything, though I didn't watch late Git development. Can you explain some particular scenario where this would improve the current situation?

Dmitry Kakurin wrote:

Can I just chime in here and express my distaste for this idea, on several grounds, but the summary is that svn does it this way, so it must be wrong.

These files which store metadata would be stored in a way that is "in another dimension" to the project files, despite being a part of the history. That means that all tools built to deal with regular files and directories will not be able to merge the changes to the attributes without special support. I think this is broken.

This is something I frequently run up against with people coming from Subversion, which supports unversioned revision properties which can change randomly and without trace, and per-file/directory properties which are simply files which you can't refer to in the regular way, and are interpreted in an application-specific way.

My question to these people, and my question to you is: why do these files need to be served from another dimension, what value does it add?

You see, either way, their contents need to be processed in an application-specific way. Same thing with git's "commit properties" - basically just RFC822.. headers used in the commit message. People I have talked to have described this as "more arbitrary" than conventions for attributes which are structured. However, when pressed I have yet to hear a clear argument why this is the case.

As far as file properties goes, I still like Linus' idea of making these files which are accessed by treating the file as a directory (eg filename.txt/ACL, filename.txt/mime-type), and that approach could be represented in git well.

Sam.

Sam Vilain <sam@vilain.net> writes:

That presumes that a good way to merge attributes is to use a text file merge algorithm, complete with finding diff context lines in a basically unchanged order.

And I don't see that this is a sensible merge strategy at all. No matter where the attributes are stored, whether in a file or somewhere else, any useful merge strategy would require an algorithm quite different from the currently used one.

Now this might be a case for pluggable merge strategies: after all, there might be non-git related files with similar unordered per-line merge semantics, or files expressing some information about files.

Well, at least _some_ interesting Reiser4 idea resurfaces.

David Kastrup wrote:

Yes. Is that not a reasonable assumption, in the absence of anything more enlightened?

That was in there too? Man that Reiser4 manifesto read like the Naked Lunch.

Sam.

Sam Vilain <sam@vilain.net> writes:

Is that a trick question? My comment was exactly about not throwing away the information ("This is not arbitrary text but talks about the files in our tree.") that would make for more enlightened use.

Hi,

On Mon, 27 Aug 2007, Sam Vilain wrote:

Umm.

It is not about _text_ file merge algorithms, but algorithms _outside of git_!

If you tuck the stuff away in some obscure database where it is hard to access, you make it more complicated and time consuming to access the data than it needs to be to _begin with_.

It is funny. No, I mean really funny. People criticise Reiser4, and only preciously few actually have ideas as good as in Reiser4. Yes, Reiser4 was not developed as openly as it should have been. Yes, Hans was not the most diplomatic poster ever, on lkml. No, even the stupid ideas in Reiser4 are not half as stupid as mistaking the working tree for a place where only _text_ files reside.

Ciao, Dscho

Hi,

On Sat, 25 Aug 2007, Dmitry Kakurin wrote:

Well, it certainly appears to me that your proposal to move metadata from the working tree (where it is visible, and easily editable with the editor of _your_ choice) to the index (where it is hidden, and could only be edited with _yet another_ git command) is not well thought through.

It certainly would make some common operation much more complicated, for no gain at all.

Should you still not be convinced, please find some convincing mails by Linus (which are much longer than I would have the patience to write) where he goes into detail _why_ it is _wrong_ to _hide_ things away from the working tree.

(Just a small hint: git is much more powerful _because_ it keeps metadata visibly in the filesystem.)

As for your proposal to munge the different metadata files into sections of _one_ file: I doubt that this is "cleaner" or "more elegant" than what we have now. For one, if a script fscks up one file, it does not fsck up the others.

For another, scripts do not have to jump through hoops to edit the metadata files, as long as they do not have sections (containing completely independent informations).

Ciao, Dscho

also sprach Junio C Hamano <gitster@pobox.com> [2007.08.26.0637 +0200]:

I tried to find related threads in the archives but failed. If someone has a pointer handy, I'd appreciate it. Alternatively, please feel free to just bounce related messages from your own archives to me.

I am interested in this mainly because of a somewhat related idea of honouring .gitignore/* in case it's a directory [0].

0. http://marc.info/?l=git&m=118725982332041&w=2

Junio C Hamano <gitster <at> pobox.com> writes:

different sections

Sorry about entering this discussion so late. I am just wondering about one thing.

Couldn't all this directory/ownership/permission tracing be easily done by using hooks? E.g. Having a pre-status and pre-commit hook one could fire up a program/script to collect all the extra info he wants to trace and store it somewhere (typically in some traced file). The other way round one could have a post-checkout hook and he could arrange it to fire up some program to look into the extra-info file to set up all the meta-data he wants.

This would be very flexible and would permit to manage absolutely /any/ kind of the metadata leaving absolute freedom about how to do so.

Am I missing something here?

Sergio

Sergio Callegari <scallegari@arces.unibo.it> writes:

Merging.

On Mon, Aug 27, 2007 at 05:48:34PM CEST, David Kastrup wrote:

Fetching.

Petr Baudis <pasky <at> suse.cz> writes:

Even here, I must be missing something, as I cannot see the issue.

If I need to fetch from someone who is tracing metadata, then there are 2 alternatives:

1) I am fetching only for myself and I am not interested in metadata at all.
All I need to do is to fetch. With this I will fetch a repository with
one/some extra file/files (e.g. .helper-metadata).

2) I am interested in the metadata tracing (e.g. to interact with
my origin). Then it is sufficient to first install the same set of
metadata tracing helpers as my origin and after that to do the fetch.
With this I will fetch a repository including the traced metadata files
just as above, yet these would be immediately be used by the helpers
through the hooks. For instance, as soon as anything gets checked out the
proper metadata can be applied.

Obviously, before installing any hooks, I should trust their origin. But I believe that if hooks get this kind of usage, rapidly we will see the growth of trustable "standard" hooks-bundles for many tasks.

David Kastrup <dak <at> gnu.org> writes:

Sorry, maybe I am really missing something, since merging does not look to me as an issue.

Why cannot git simply do the merging in the working tree as it normally does, including merging of the traced metadata file generated by the metadata helpers invoked via the hooks? Only, again more hooks are needed and likely a post-merge hook, so that at the end of the merge, the metadata can be applied.

Only, to have things going on smoothly, one should be so wise to assure that the metadata helpers save metadata as nice, sorted text files in order to minimize the burden of manual intervention if there are conflicts in metadata merging.

BTW.  Having a post-checkout hook could also help getting rid of unwanted
empty directories, couldn't it?
 
Sergio

On Mon, Aug 27, 2007 at 17:07:34 +0000, Sergio Callegari wrote:

The post-checkout (no need for post-merge -- after in-index merge is done, the files are checked out to worktree, so post-checkout would run anyway) could actually apply any custom merge strategy required to avoid/clean up spurious conflicts in the metadata file (eg. adding two files that go after each other would be a textual conflict). The relevant versions are stored in index stages at that point.

Probably not. I would imagine it would actually only run for the files being checked out -- and there is nothing checked out in empty directories. (Well, it would run once or once per directory with list of checked out files on standard input).

----- Original Message -----
From: "Petr Baudis" <pasky@suse.cz>

Here is the problem: we need to apply crlf attributes to a file. We
could have .gitattributes both in the index and in the worktree.
Which one do we use?
In general .gitattributes file could be (U)nchanged, (C)hanged, (NP)
NotPresent in each place.
This gives us 3x3 matrix = 9 special cases to handle. When you think
about this a little more you realize that even that
is not enough and we need to take into account direction of data
movement (index -> filesystem or vice versa).
This doubles the matrix to 18 cases. Even if we come up with a very
good choice for every case (which is doubtful)
there is no way this could be *easily* explained to end user, or even memorized.
This leads to a simple idea that mostly works:
1. when files are moved from index to filesystem, then only
.gitattributes in the index is used, if it's not there == no special
attributes.
2. when files are moved from filesystem to index, then only
.gitattributes in filesystem is used, again if it's not there == no
special attributes.

Then, in any operation, only one .gitattributes is taken into account. We could stop here, but to me this redundancy still has some room for confusion and looks unnecessary. Plus there is still room for unintentional abuse (by mistake).

That's why I think it's a good idea to always have only one .gitattributes.

- Dmitry

On Tue, Aug 28, 2007 at 09:49:47 +0100, Johannes Schindelin wrote:

I do.

IMNSHO it should be the other way around:
 .gitattributes in index, than index version is used.
 .gitattributes not in index, but in worktree, than that tree version is used.

Why? Because when you check out another version, the .gitattributes commited in that version need to be applied, since it might be different from whatever is currently in the tree.

In the other direction, the tree version seems to make more sense, but in reality it does not. If you do a partial commit of a single file, than the index version gets into the commit, so that should better be the version used to store the file. On the other hand if you change .gitattributes, the normal commit rules are that you need to add it for commit, so needing to add it to make it effective goes well together with it. update-index would need to always handle .git* before other entries to make add . and commit -a work correctly.

The case for worktree only is for cases when you for some reason want to have local .gitattributes. Though I am not sure that should actually work, because you couldn't have local .gitattributes if there is versioned version. .git/info/attributes would be better for that. Which gets us back to "always use .gitattributes from *index*" (and read it from tree before other files when adding it).

On 9/4/07, Jan Hudec <bulb@ucw.cz> wrote:

Consider scenario when my commit #1 has .gitattributes:
    a.txt -nocrlf
and file a.txt
You pull it.
Now I make some changes to a.txt and realize that a.txt *is* a text
file now. I remove the entry from .gitattributes and notice that it
becomes empty. So I just remove .gitattributes file all together. It
becomes commit#2.
Now you pull it again. There *is* .gitattributes in local directory,
but index does not have it (because I've removed it on purpose). What
should happen?
I assert that since index does not have .gitattributes the one from
local directory should not be used.

Think about dedicated build machine scenario: I have a machine that always does sync + build. After every sync the local directory should always be identical to what-was-committed. With every commit 3 things could happen: .gitattributes could appear, disappear or change. In every case "build machine" must produce the exact copy of what-was-checked-in. The only way I see this happening is by using *only* index version of .gitattributes when files are moved index -> workspace.

A similar reasoning works for other direction (workplace -> index).

"Dmitry Kakurin" <dmitry.kakurin@gmail.com> writes:

Thinking about the reason _why_ .gitattributes may be updated, one would notice that it is because somebody did this command sequence:

	git checkout		;# now work tree is clean
	edit .gitattributes	;# modify the attributes of a file
	edit file		;# edit the file attributes talks about
	git add file		;# this can be affected by .gitattributes
	git add .gitattributes	;# this is changed in the same commit
	git commit

Now, should we always take .gitattributes from the index?

On 9/5/07, Junio C Hamano <gitster@pobox.com> wrote:

No, from couple of my emails back:

This is a must-do change (IMHO).

To go one step further (again IMHO) is to eliminate workspace version of .gitattributes all together:

On Wed, Sep 05, 2007 at 01:14:44 -0700, Junio C Hamano wrote:

Yes, they should:

$ git checkout $ edit .gitattributes $ edit file $ git add file $ git commit ;# this does NOT have the changes to .gitattributes

the above case is a user error that can (at some cost) be detected:

$ git checkout
$ edit .gitattributes
$ edit file
$ git add file
$ git add .gitattributes
Warning! Changes to gitattributes affects handling of files scheduled for
commit. Please add following files again before commit:
  file
$

It would be possible to special-case .gitattributes in add to:
 - do diff between the old and new value of .gitattributes in index,
 - list files changed in index compared to HEAD,
 - match each of them to all patterns in the diff,
 - if any matches, print the warning and list of matches.
It might be even possible to actually inspect the changes and apply those
that can be automatically (and not ask user to re-add), but some filters
loose information, so user interaction is needed to add good version.

On Wed, Aug 29, 2007 at 00:07:55 +0100, Johannes Schindelin wrote:

Whether it's insane depends on whether you want to keep git purely for source control -- in which case they are definitely insane -- or want to allow git to grow to a tool useful for other cases, like tracking content of /etc, whole filesystem images and similar stuff -- in which case most of those setups are not insane at all.

Personally I would vote for a middle ground. To keep core git simple, but to provide enough hooks to build the other tools on top of it. Enough hooks in this case would mean (at least the way I can imagine as workable) hooks to run when transfering files between worktree and index and place for the hooks to store the extra tracked information. This place could be some of special named file (.gitxattrs or anything) which would be updated in index by the hook, special index entries somehow related to individual filenames and/or headers prepended to the blobs and trees. Any of those methods can be used for storing permissions, acl, extended attributes and other random stuff and different ones could be used by different hooks at the same time.

Each of those methods has it's advantages and disadvantages. The special files are easiest to store if the hook is not available or can't represent the data in given worktree -- simply make the special files regular. The entries attached to files (which are equivalent of extended attributes) would work with standard merge on them (because they are handled individually) and the headers to files would probably require least changes to git (I believe they are implementable for files with git as is using the hook provided for keyword expansion (or is it not there?)).