Storing permissions

Hi Linus et al.,

I'm trying to use git, but I frequenty run into problems with file permissions -- some archives (including the master git archive) contain group-writable files, but when I check them out, the permissions get trimmed by my umask (quite sensibly) and update-cache complains that they need update.

Does it really make sense to store full permissions in the trees? I think that remembering the x-bit should be good enough for almost all purposes and the other permissions should be left to the local environment.

Another possibility is to keep the permissions in the trees, but just make update-cache ignore differences in write permissions.

				Have a nice fortnight

Martin wrote:

That matches my experience - store 1 bit of mode state - executable or not.

Let local environment determine read, write and umask permissions.

PJ> That matches my experience - store 1 bit of mode state - executable or not.

Sounds like svn ;-).

Junio wrote:

I have no idea what svn is.

Paul Jackson wrote:

svn = common abbreviation for "Subversion", a widely-used centralized SCM tool intentionally similar to CVS.

--- David A. Wheeler

It makes some sense in principle, but without storing what they mean (i.e., group==?) it certainly makes no sense. It's a bit like unpacking a tar file.

I suspect a non-readable file would cause a bit of a problem in the low-level commands.

Morten

Morten wrote:

There's no "they" there.

I think Martin's proposal, to which I agreed, was to store a _single_ bit. If any of the execute permissions of the incoming file are set, then the bit is stored ON, else it is stored OFF. On 'checkout', if the bit is ON, then the file permission is set mode 0777 (modulo umask), else it is set mode 0666 (modulo umask).

You might disagree that this is a good idea, but it certainly does 'make sense' (as in 'is sensibly well defined').

Probably so. If someone sets their umask 0333 or less, then they are either fools or QA (software quality assurance, or test) engineers.

On Sat, 16 Apr 2005, Paul Jackson wrote:

I think I agree.

Anybody willing to send me a patch? One issue is that if done the obvious
way it's an incompatible change, and old tree objects won't be valid any
more. It might be ok to just change the "compare cache" check to only care
about a few bits, though: S_IXUSR and S_IFDIR. And then always write new 
"tree" objects out with mode set to one of
 - 040000: we already do this for directories
 - 100644: normal files without S_IXUSR set
 - 100755: normal files _with_ S_IXUSR set

Then, at compare time, we only look at S_IXUSR matching for files (we never compare directory modes anyway). And at file create time, we create them with 0666 and 0777 respectively, and let the users umask sort it out (and if the user has 0100 set in his umask, he can damn well blame himself).

This would pretty much match the existing kernel tree, for example. We'd end up with some new trees there (and in git), but not a lot of incompatibility. And old trees would still work fine, they'd just get written out differently.

Anybody want to send a patch to do this?

		Linus

Linus wrote:

I think I agree. But since I am reluctant to take enough time to understand the code well enough to write this patch, I'll shut up now ;).

On Sat, 16 Apr 2005, Linus Torvalds wrote:

Actually, I just did it. Seems to work for the only test-case I tried, namely I just committed it, and checked that the permissions all ended up being recorded as 0644 in the tree (if it has the -x bit set, they get recorded as 0755).

When checking out, we always check out with 0666 or 0777, and just let umask do its thing. We only test bit 0100 when checking for differences.

Maybe I missed some case, but this does indeed seem saner than the "try to restore all bits" case. If somebody sees any problems, please holler.

(Btw, you may or may not need to blow away your "index" file by just re-creating it with a "read-tree" after you've updated to this. I _tried_ to make sure that the compare just ignored the ce_mode bits, but the fact is, your index file may be "corrupt" in the sense that it has permission sets that sparse expects to never generate in an index file any more..)

		Linus

Linus Torvalds wrote:

There's a minor reason to write out ALL the perm bit data, but
only care about a few bits coming back in: Some people use
SCM systems as a generalized backup system, so you can back up
your system to an arbitrary known state in the past
(e.g., "Change my /etc files to the state I was at
just before I installed that &*#@ program!").
For more on this, see:
  http://www.onlamp.com/pub/a/onlamp/2005/01/06/svn_homedir.html

If you store all the bits, then you CAN restore things more exactly the way they were. This is imperfect, since it doesn't cover more exotic permission values from SELinux, xattrs, whatever. For some, that's enough.

Yeah, I know, not the main purpose of git. But what the heck, I _like_ flexible infrastructures.

--- David A. Wheeler

David wrote:

There's always the 'configurable option' approach.

Someone, I doubt Linus will have any interest in it, could volunteer to make the masks of st_mode, used when storing and recovering file permissions, be configurable by some environment variable settings, which default to whatever Linus provided.

But, in general, if you want a generalized backup system, git is not it.

Git skips all files whose name begins with the dot '.' character, and anything that is not a regular file or directory. Git makes no concessions to working adequately on file systems lacking normal inode numbers (such as smb, fat, vfat). Git obscures the archive format a modest amount, for pure speed and to encourage use only via appropriate wrappers. Git is tuned for blazing speed at the operations that Linus needs, not for trivial recovery, using the most basic tools, under harsh circumstances.

The basic idea of using such an 'object database' (though I dislike that term -- too high falutin vague) of files stored by their hash is a good one. But a different core implementation is needed for backups.

I have one that I use for my own backups, but it is written in Python, and uses MD5, one or the other of which likely disqualifies it from further consideration by half the readers of this list.

On Sun, 17 Apr 2005, David A. Wheeler wrote:

I think this should be possible with a different tag than "tree". All the bits aren't sufficient, anyway; the unincluded values include the user and group, which are likely to matter for some things in /etc. But there's no reason that the core can't support both a system-local complete representation of the dentry and a user-relative representation of a source distribution with different tags.

For that matter, it could accept "dir" objects in commits as well, and use version-control-type logic on history while refusing to do non-sensical things with them.

	-Daniel
*This .sig left intentionally blank*

On Sun, 17 Apr 2005, David A. Wheeler wrote:

Yes. I was actually thinking about having system config files in a git repository when I started it, since I noticed how nicely it would do exactly that.

However, since the mode bits also end up being part of the name of the tree object (ie they are most certainly part of the hash), it's really basically impossible to only care about one bit but writing out many bits: it's the same issue of having multiple "identical" blocks with different names.

It's ok if it happens occasionally (it _will_ happen at the point of a tree conversion to the new format, for example), but it's not ok if it happens all the time - which it would, since some people have umask 002 (and individual groups) and others have umask 022 (and shared groups), and I can imagine that some anal people have umask 0077 ("I don't want to play with others").

The trees would constantly bounce between a million different combinations (since _some_ files would be checked out with the "other" mode).

At least if you always honor umask or always totally ignore umask, you get a nice repetable thing. We tried the "always ignore" umask thing, and the problem with that is that while _git_ ended up always doing a "fchmod()" to reset the whole permission mask, anybody who created files any other way and then checked them in would end up using umask.

One solution is to tell git with a command line flag and/or config file entry that "for this repo, I want you to honor all bits". That should be easy enough to add at some point, and then you really get what you want.

That said, git won't be really good at doing system backup. I actually _do_ save a full 32-bit of "mode" (hey, you could have "immutable" bits etc set), but anybody who does anything fancy at all with mtime would be screwed, for example.

Also, right now we don't actually save any other type of file than regular/directory, so you'd have to come up with a good save-format for symlinks (easy, I guess - just make a "link" blob) and device nodes (that one probably should be saved in the "cache_entry" itself, possibly encoded where the sha1 hash normally is).

Also, I made a design decision that git only cares about non-dotfiles. Git literally never sees or looks at _anything_ that starts with a ".". I think that's absolutely the right thing to do for an SCM (if you hide your files, I really don't think you should expect the SCM to see it), but it's obviously not the right thing for a backup thing.

(It _might_ be the right thing for a system config file, though, eg tracking something like "/etc" with git might be ok, modulo the other issues).

			Linus

Linus Torvalds wrote:

...

Yes, I thought of that too.  And I agree, that should do the job.

My real concern is I'm looking at the early design of the storage format so that it's POSSIBLE to extend git in obvious ways. As long as it's possible later, then that's a great thing.

...

Again, a command line flag or config file entry could change that in the future, if desired. So this is a decision that could be changed later... the best kind of decision :-).

--- David A. Wheeler

There's one more mode bit we might actually care about: the symlink bit. (One would store the target as the blob, presumably, but chmod isn't going to create symlinks out of regular files.)

Morten

Hi,

What's the status of dotfile support? I can only find one thread that really discusses the issue:

On Sun, Apr 17, 2005 at 12:21:47PM -0400, David A. Wheeler wrote:

Personally I like dotfile support, and I compare it to the language encoding issue for filenames. Linus says we should treat filenames as binary data, and thus it won't matter what characters someone uses. I agree completely, but by not supporting dotfiles, we're creating a big exception, in that if a filename begins with a dot, we treat it in a much different way, in fact we ignore it completely.

In the above quote, Linus says "if you hide your files, I really don't think you should expect the SCM to see it". But what about the case where the user is not the one choosing to create dotfiles? If I want to put a bunch of config files for various apps into a git repository, I don't get to pick their names in many cases, at least not without changing the way I invoke my apps (or the scripts that do it for me). But it's still worthwhile to have those config files in version control. It makes it much easier to experiment with new tools, recover from my mistakes, and share my successes with others.

So that's my pitch: Leaving out dotfile support seems like it creates an unnecessary limitation that eliminates some valid uses of git.

Be well, Zack

Zack Brown wrote:

What sort of "dotfile support" are you hinting at? git being able to handle them, or git being able to ignore them? Both are implemented. The former by default and the latter through .gitignore.

Files you want to version-control ofcourse has to be added with "git add", but that's not just dotfiles and it's really the only sane behaviour.

OK, I see my mistake.

I should have tested better. I started off with a non-versioned directory containing dotfiles and regular files. I did a cg-init, only to discover that the dotfiles were not included in the repository at that time. So I just assumed they couldn't be added either.

But I just tested, and yes indeed, it is possible to cg-add a dotfile.

So my question is, why does cg-init ignore dotfiles within the directory when it first initializes the repository?

Be well, Zack

On Wed, Dec 07, 2005 at 04:28:48PM +0100, Andreas Ericsson wrote:

Dear diary, on Wed, Dec 07, 2005 at 05:11:30PM CET, I got a letter where Zack Brown <zbrown@tumblerings.org> said that...

Precisely for Linus' reasons - by default, I believe your VCS shouldn't care about your hidden files, because they are hidden, and probably not content but kind of meta-content - the exception is .gitignore, but you can add more even on cg-init time. I have to admit that cg-init documentation wasn't sufficiently clear about this autoignoring stuff, I've tried to improve that now, and cg-init will warn you (post hoc) that it autoignored some files.

Thanks,

Hi,

On Wed, 7 Dec 2005, Zack Brown wrote:

In the current git repository, ".gitignore" is a versioned file.

Hth, Dscho

Zack Brown <zbrown@tumblerings.org> writes:

Valid argument, and resolved thusly quite some time ago, with commit 320d3a1b1aa04d75f0aaff3cc7cf582e144a84c6 on May 24th 2005.

You probably have missed it because unfortunately this commit happened after the latest issue of git traffic ;-).

On Wed, Dec 07, 2005 at 09:43:29AM -0800, Junio C Hamano wrote:

Actually I've started including summaries of git list threads in Kernel Traffic as well, for the past few issues. :)

Be well, Zack

Zack Brown <zbrown@tumblerings.org> writes:

I was just trying to be funny.  My apologies.

I do read kernel-traffic and appreciate your helping us keep current very much.

Zack Brown wrote:

Works fine for me; I now maintain my shell config files in git.

	-hpa