Re: Tracking file metadata in git -- fix metastore or enhance git?

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Apr 8, 2011, 19:45 UTC
Message-ID: <20110408194548.GA26094@elie>
In-Reply-To: <Pine.BSM.4.64L.1104081903550.22999@herc.mirbsd.org>

Thorsten Glaser wrote:

> Jonathan Nieder dixit:

Show 7 quoted lines

>> I think the most native-looking way to store metadata associated to
>> paths is .gitattributes.  It also has the nice feature of allowing a
>> single attribute to apply to multiple files.
>
> Eh, no. Think of extended attributes like, say, NTFS Resource Forks.
> They’re just different “lines” into the “plane” a file can be, if
> you excuse the metapher. (All parallel, of course.)

Do you mean no, it doesn't have that feature? ;-)

Each git commit (try it with "git cat-file commit HEAD) looks like so:

	tree <tree name>
	parent <commit name for first parent>
	parent <commit name for second parent>
	...
	author <author identity and time of authorship>
	committer <committer identity and time committed>
	encoding <encoding of log message (optional)>

	<free-form change description>

Where could one sneak in some per-path metadata?

 - as new header fields after "encoder" (teaching git fsck, git commit
   --amend, and so on about it)?  That can work but it would slow down
   operations not interested in this metadata.  It is best not to have
   O(number of paths) header fields.

 - in the change description?  Yes, that can work, too, and it doesn't
   even require changing the commit format.

 - a new header field pointing to another object?  That is possible as
   a last resort.

Anyway, filenames and associated content are not what commits are about; commits are just nodes in a revision graph, with trees representing the tracked trees.

Okay, so what about the trees?

	<mode> SP <filename> NUL <object name>
	...

Where can we sneak something in?

 - use a currently invalid <mode>?  No, tracking metadata is probably
   not worth breaking old git clients.
 - use an invalid object name?  No (for the same reason).
 - use a special filename?  Then old git clients will treat the file
   as a regular file, so they still get access to the data.

So you see, using ordinary files (whether called .gitattributes or foo.c.ntfs-resource-fork) to track this extra data makes a lot of sense.

Now Michael mentioned an alternative, which is to store this information in separate objects. That way, you could push your history without the extra metadata, you could edit the metadata without changing the commit names of the history, separately garbage-collect metadata you're not interested in, etc. If that is your goal, then "git notes" is exactly the right solution.

> They are just
> another facet of each file.

Sure, like the atime, the inode number, the uid of the user who wrote them, and the model number of the disk used to store it.

Oh, you mean they're _relevant_ facets? Yes, that's believable, though I suspect that's only going to sometimes be the case. So the operator should say "yes, I'm interested in tracking this extra information". To summarize the above, some ways this could work behind the scenes:

 * dotfiles with metadata;

 * a Makefile to install files with metadata (i.e., the "source"
   consists of plain files, while the "build product" has the
   specified metadata);

 * something else.  Hopefully the above explains the relevant
   constraints so you can surprise us.

Hope that helps. Jonathan