confused about preserved permissions

Hello,

I am a bit confused about how git stores and restores permissions. I went to the web but found little other than this post from Linus in 2005:

  http://www.gelato.unsw.edu.au/archives/git/0504/0870.html

where he basically says that git will *not* save permissions and that it also does not save symlinks. I thus discarded the information as being outdated. I have also looked into the source, but the following question is more about git design than implementation.

Let me illustrate the source of my confusion:

  $ git init
  Initialized empty Git repository in .git/
  $ umask
  002
  $ date > file1; ls -l file1
  -rw-rw-r-- 1 madduck madduck 30 2007-08-20 18:15 file1
  $ git add file1; git commit -m.
  Created initial commit 5762460: .
  1 files changed, 1 insertions(+), 0 deletions(-)
  create mode 100644 file1

file1 is mode 664, but gut says "create mode 100644 file1", independently of whether I use --shared when initialising, or what value I set it to. Why does it override the group write bit?

But it gets worse:

  $ git checkout -b t
  Switched to a new branch "t"
  $ git rm file1; git commit -m.
  rm 'file1'
  Created commit a813b4e: .
  1 files changed, 0 insertions(+), 1 deletions(-)
  delete mode 100644 file1
  $ umask 007
  $ git checkout master
  Switched to branch "master"
  $ ls -l file1
  -rw-rw----  1 madduck madduck    30 2007-08-20 18:15 file1

So git doesn't even restore 644 when I switch branches, nor on initial clone, or on re-checkout after removing the local copy.

Am I doing something wrong?

Why does git bother saving a mode when later it never seems to use it again?

And is there a way to tell git not to mess with the permissions in the worktree when switching branches? I am not sure what's causing it since the inode of files does not seem to change, so git apparently explicitly fchmod()'s files.

I'd be grateful for any documentation to read in addition to the code. I saw core.filemode, but that's only about the x bit, it seems.

Thanks,

On Mon, Aug 20, 2007 at 04:44:11PM +0000, martin f krafft wrote:

  You may want to read git-config around core.sharedRepository.

also sprach Pierre Habouzit <madcoder@debian.org> [2007.08.20.1854 +0200]:

Hello Pierre, thanks for taking the time to respond.

core.sharedRepository (or init --shared for that matter) seems to only deal with permissions of $GIT_DIR. I am worrying about the permissions of the worktree. I also stated in the OP that I had used --shared but none of the settings made any difference...

Cheers,

On Mon, Aug 20, 2007 at 06:44:11PM +0200, martin f krafft <madduck@madduck.net> wrote:

Same applies to git-archive, which generates files with mode 666 and directories with 777, while it could follow the modes in the repository... or at least, that's what the manpage claims, but facts seem to be quite different...

mh@namakemono:~/git/git$ git-archive --format=tar HEAD | tar -tvf - -rw-rw-r-- root/root 2365 2007-08-19 20:45 .gitignore -rw-rw-r-- root/root 1973 2007-08-19 20:45 .mailmap -rw-rw-r-- root/root 18787 2007-08-19 20:45 COPYING drwxrwxr-x root/root 0 2007-08-19 20:45 Documentation/ -rw-rw-r-- root/root 63 2007-08-19 20:45 Documentation/.gitignore -rw-rw-r-- root/root 4553 2007-08-19 20:45 Documentation/Makefile (...) mh@namakemono:~/git/git$ git-ls-tree HEAD 100644 blob 63c918c667fa005ff12ad89437f2fdc80926e21c .gitignore 100644 blob 5529b198e8d14decbe4ad99db3f7fb632de0439d .mailmap 100644 blob 6ff87c4664981e4397625791c8ea3bbb5f2279a3 COPYING 040000 tree 865a32f4b417cbb601524ac2b78ca64ff232302c Documentation 100755 blob 3c0032cec592a765692234f1cba47dfdcc3a9200 GIT-VERSION-GEN (...)

mh@namakemono:~/git/git$ umask 0022

And no value is set for umask in git config...

I also never understood why there were no permissions set on directories in trees... nor why, while the sha1 for child objects are "packed", the modes aren't...

Mike

Mike Hommey <mh@glandium.org> writes:

Git had permissions at one point of time. It makes colloboration with other people with different umasks a nuisance. However, not all uses are colloborative. So it might be nice to be able to specify permission/uid/gid policies that do a configurable level of munging for stuff passed into and out of the index and/or the repositories.

Because directories are not actually tracked. They are created and deleted as-needed.

In my proposal for allowing directories to get tracked, permissions of 000 would indicate a tree without a corresponding tracked directory. Other permissions would correspond to a tracked directory. I am still stuck over the representation in the index.

One idea is to unconditionally have an entry "dirname" without permissions, and optionally "dirname/" with permissions iff the directory is supposed to be tracked, both to be sorted in alphabetically. The idea of the first entry is being able to detect merge conflicts without extra passes.

But I have not worked on the stuff for a while.

Because a change of the mode of a file will then not cause different sha1 sums at the file level.

On Mon, Aug 20, 2007 at 07:58:43PM +0200, David Kastrup <dak@gnu.org> wrote:

I don't see why it would prevent to have a permission set to it... the permission technically can be recorded in the parent tree, along its sha1. Filesystems are also like this.

I don't see why you would need an additional entry for the directory permission.

I think i wasn't clear enough... I just wondered why the format for tree entries is something like (if you'd write it in perl): sprintf "%06o %s\0%s", $mode, $file, pack("H[40]", $sha1)

Mike

Mike Hommey <mh@glandium.org> writes:

No, they aren't. Filesystems don't create and delete directories on the fly. If we record any information about a directory, deleting it automagically would not be appropriate since we would lose information that has not been explicitly deleted.

One doesn't. It would be just there to facilitate single-pass detection of merge conflicts. No non-redundant information content to it at all, but it delivers the "we'll need a directory here" information earlier in the sort order.

But one can probably do without that by just stashing away non-directory entries namex* possibly colliding with a tree until the opportunity for namex/* has passed by, and then processing them. That makes for a cleaner index format.

There is still the question when one stores just a single entry whether to sort this under "namex" or "namex/" and whether to also have it appear when it corresponds to a non-tracked directory.

It does not really make sense settling on one format before one implements the various merge/update/checkout utilities since then one sees what is most convenient.

Now I am sure I don't get your point.

Mike Hommey schrieb:

In that case the default value for tar.umask applies, which is 002. The manpage needs an update.. Thanks for spotting this.

You can set it to 022 explicitly or to "user" to restrict permissions:

   $ git config tar.umask 022		# -rw-r--r--
   $ git config tar.umask user		# your Unix umask applies

Originally file permissions were simply copied into the archive, but this was inconvenient for users that needed more permission bits set, e.g. to share an extracted archive with with others by default.

Also please note that tar applies your umask when extracting files (exception: GNU tar doesn't do that by default if run as root), so the permissions of extracted files may look different from the output of "tar -t".

René

As noted by Mike Hommey, the documentation for the config setting tar.umask is not up-to-date. Commit f08b3b0e2e9ad87767d80ff03b013c686e08ba4b changed the default from 0 to 2; this patch finally documents it.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---

On Tue, Aug 21, 2007 at 08:01:16PM +0200, René Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:

Actually, I'd say it's not enough ; users may be confused by the sentence just above:

By default, file and directories modes are set to 0666 or 0777 in tar archives.

Which is true, but if you are not very familiar with umask, you won't actually gather from the rest that the modes are not going to be that by default.

Mike

Mike Hommey schrieb:

Hmm.  And I forgot to update two other places.  How about this?  Too short?

 Documentation/config.txt       |   14 +++++---------
 Documentation/git-archive.txt  |   15 ++++++---------
 Documentation/git-tar-tree.txt |   15 ++++++---------
 3 files changed, 17 insertions(+), 27 deletions(-)

martin f krafft, Mon, Aug 20, 2007 18:44:11 +0200:

Git tracks just the x-bit. Everything else is controlled by your umask.

Symlinks are supported since long time.

On Aug 20, 2007, at 6:44 PM, martin f krafft wrote:

[...]

Hi, this thread drifted away and although the discussion is interesting, the original question hasn't been answered so far and I am in a situation where I'd like to know the answer (because my working tree needs to be group-readable for a given project and my umask 066 keeps annoying me although I don't want to change it).

Someone on IRC pointed me to http://git.or.cz/gitwiki/ ContentLimitations which says:

"By design, git cannot track other aspects of the filesystem, including:
   * File modes (except for the "executable" bit, and being symbolic  
link)"

That's weird since the file mode is saved in the tree, isn't there a way to ask Git to restore this file mode?

Cheers,

Benoit SIGOURE wrote:

At this time, there is not. git tracks only the executable bit. Even though it stores (and reports) the complete mode, it only ever stores one of these mode values: 100644, 100755, 120000. Yes, that's an enormous waste of bits, but that's how it is.

-- Hannes

Benoit SIGOURE <tsuna@lrde.epita.fr> writes:

The wording you quoted is wrong. By design, we "chose not to" track other aspects, even though the underlying data structure has enough space to use other bit patterns.

We deliberately chose not to use that space, and this default is very unlikely to change.

In very early days of git, we allowed the work tree modes 0644 vs 0664 propagated back to the index modes and regular file modes recorded in the tree entries. This caused unnecessary pain for poeple merging real projects for no real gain whatsoever, and the behaviour was fixed to minimally track, hence we do not track anything but executable bits.

I do not oppose to a new per-project configuration option to make use of the existing space to record differences vs 0644 vs 0664 vs 0600. However, I have already seen the downsides, so:

 (1) I am not interested in implementing that myself;

 (2) the places that canonicalize the mode bits obtained from
     the filesystem to 0644 is fairly centralized so it would
     not be too hard to implement (and that is one good reason
     why I do _not_ have to be the person to do so);

 (3) however, (2) means that everybody calls that
     canonicalization logic, and the unintended side effects
     need to be audited for codepaths of all the callers, which
     means a large test suite is probably needed; and

 (4) to me, reviewing such a patch will be much lower priority
     than other patches for the above reasons.

also sprach Junio C Hamano <gitster@pobox.com> [2007.08.23.0009 +0200]:

The downsides included change in SHA hash on mode change, as far as I can remember. Anything else?

martin f krafft <madduck@madduck.net> writes:

The modes are recorded in patches and push/pull, so if you have several people working with different permissions/umasks, you get wagonloads of unnecessary patches and get your local permissions messed up by other contributors.

For colloborative work, you _really_ don't want to have _personal_ preferences distributed.

Any patch offering to optionally track permissions must make very sure that it retains the possibility to have permissions in the directory policed to the values that are actually a property of the source files rather than the personal work environment.

also sprach David Kastrup <dak@gnu.org> [2007.08.23.0812 +0200]:

How about a patch that

  (a) ensures that modes of existing files are preserved
      across checkouts and merges?
  (b) for files it creates (e.g. checkout of a branch introducing
      a file), it consults the default for this file tracked in the
      archive and uses that.
  (c) in order to change the mode recorded for a file, a user has to
      tell git-add to specifically record the mode change in the
      index.

There should be a tri-state policy: mode-from-archive vs. umask, and umask could be according to core.umask or if that's set to user, then the user's umask.

On Aug 23, 2007, at 8:12 AM, David Kastrup wrote:

Hmm yes, that's right.

Let's look at the problem from another point of view then: I want my *working tree* to be group readable even though my umask is 066. Would it be possible to add a local config option in the .git repo to tell git that it should create new file this way (exactly like core.sharedRepository but core.sharedWorkingCopy or whatever).

WDYT?

Benoit SIGOURE <tsuna@lrde.epita.fr> writes:

I have to wonder if there is any sane development tool that supports that kind of thing. E.g. vi, emacs, gcc,...? If you allowed a tool to do that, what's the point of having a umask?

Junio C Hamano <gitster@pobox.com> writes:

Hm? Are we talking about the same Emacs here? The thing that has a directory editor built into it with which one can do chmod and so on?

Here are a few functions:

    executable-make-buffer-file-executable-if-script-p is an autoloaded Lisp function in `executable'.
    (executable-make-buffer-file-executable-if-script-p)

    Make file executable according to umask if not already executable.
    If file already has any execute bits set at all, do not change existing
    file modes.

----------------
dired-do-chmod
  Command: Change the mode of the marked (or next ARG) files.
This calls chmod, thus symbolic modes like `g+w' are allowed.

(fn &optional ARG)

----------------
executable-set-magic
  Command: Set this buffer's interpreter to INTERPRETER with optional ARGUMENT.
The variables `executable-magicless-file-regexp', `executable-prefix',
`executable-insert', `executable-query' and `executable-chmod' control
when and how magic numbers are inserted or replaced and scripts made
executable.

(fn INTERPRETER &optional ARGUMENT NO-QUERY-FLAG INSERT-FLAG)

However, Emacs does not provide a hook into the system call as far as I can see: it calls chmod explicitly.

On Thu, Aug 23, 2007 at 09:48:23 +0200, Benoit SIGOURE wrote:

You need to change umask not just for git, but for your editor and compiler when you are working in the shared work tree as well, no? So what about teaching your *shell* to change it as appropriate?

In zsh if you define a function chpwd, it will be called whenever you change current working directory. If you define it as:

chpwd() {
    # The / at the end is to make foo/* match path foo
    case `pwd`/ in
	/path/to/your/worktree/*) umask 002;;
	*) umask 066;;
    esac
}

would make any command (ie. git, editor and compiler/make/...) ran from the shared worktree run with umask 002 (or whatever depending on permissions you want there) and anything ran from anywhere else use your normal umask 066.

I don't use bash, but I am almost sure you can get a suitable hook there as well. If nothing else I recall there is a way to run a function from prompt expansion, which would do the trick.

martin f krafft <madduck@madduck.net> writes:

More serious practical problem was that sane developers tend to have either 022 or 002 umask. You may check out my project (whose regular files are either 0664 or 0775, according to my umask settings), but the checkout honors your umask and you would get 0644 or 0755 on your copies, and you would check in with differemtn modes. The merge I would do later with you would get needless conflicts on modes.

What makes the mode conflict "needless" is that we are primarily talking about source code control, by the way. It is very reasonable if you want to retarget git, probably borrowing most of its infrastructure, to build a backup system to keep track of files in /etc hierarchy. It would be crucial for _your_ system to keep track of owner/group and mode bits. If the modes or ownership conflict during a merge, that is _not_ needless conflict at all --- the user needs to decide if /etc/shadow should or should not be world readable, and send a nasty message to whoever asked you to pull such a change to make the file world readable after you detect such a mistake while you are merging. It becomes essential to your "backup" system.

But in the context of git project, and source code management, which is its primary application, it is not worth to keep track of such mode differences that come from umask differences of contributors. You could add such a full-mode-bits and ownership support to git, but that would stay secondary at best. It is entirely possible that the code to support such an extension would be too intrusive to clutter git codebase, at which point the "backup" application may have to become a fork of git, not part of it.

On Mon, Aug 20, 2007 at 08:40:13PM +0200, David Kastrup <dak@gnu.org> wrote:

git doesn't magically create directories either. It actually stores something about them: the hash of the tree object that represents them. And it has permissions associated with these hashes.

What it doesn't have, though, is tracking of the directory's history.

(...)

See what a raw tree object looks like: git-cat-file tree 708453d64796eb617cb8a1602959e00356693315

Mike

On Mon, Aug 20, 2007 at 20:48:29 +0200, Mike Hommey wrote:

Git does not have tracking of file's history either.

There are actually no permissions to record with a directory, since only executable bit is tracked and that always has to be turned on.

Git actually does store directories in tree objects, but never in the index. There was a long thread about tracking existence of directories. IMNSHO the index can start to *always* contain directories, with reader being able to detect index written without them and automatically fill them in, to keep compatibility. Than a config option could switch between removing empty directories implicitly when there are no versioned files in them, or explicitely.

On Mon, Aug 20, 2007 at 09:43:28PM +0200, Jan Hudec <bulb@ucw.cz> wrote:

Well, it has ways to track file's history, with blame and log, for example. There is nothing similar for directories, though it could be possible to do.

Mike

Mike Hommey, Mon, Aug 20, 2007 21:50:37 +0200:

It would be not exactly "tracking". You can present the history of changes which involved the said directory.

And you could do it from day one: git log -- sub/dir

On Mon, Aug 20, 2007 at 10:07:19PM +0200, Alex Riesen <raa.lkml@gmail.com> wrote:

It could be possible to really do tracking, too, like it is possible for files (including renames, moves, etc.)

Mike

On Mon, Aug 20, 2007 at 22:10:42 +0200, Mike Hommey wrote:

It depends on what you imagine under tracking, but git does not track files in a sense of knowing about renames, moves etc. Git can't tell you -- and by design never will -- that file foo was moved to bar in revision abcdef01. All it can tell you is "looking at the trees, I would guess that file foo was moved to bar in revision abcdef01". That is not tracking in a sense keeping track of anything. It is just digging out interesting similarities between the individual trees.

Mike Hommey <mh@glandium.org> writes:

You are confused. Git does not track renames or moves for files, either. It just records snapshots. It reverse engineers renames, moves, copying when cranking out diffs and logs and blames.

On Mon, Aug 20, 2007 at 10:42:34PM +0200, David Kastrup <dak@gnu.org> wrote:

Yes, I know. And that could be done for directories, too.

Mike

On Mon, Aug 20, 2007 at 21:50:37 +0200, Mike Hommey wrote:

Neither of them is file's history. They dig in the global history for records, that are, by some heuristic, related to something. Blame really looks for a bunch of lines and log looks for given path (can be whole subtree).

Mike Hommey <mh@glandium.org> writes:

The tree object is not a representation of the directory in the file system. It is a hierarchical hash in the repository, stored for efficiency reasons.

No, with the files. Think again: the link between file system and repository is the index, and the current index format has no representation for trees or directories.

So git _can't_ store any information about a _directory_, since no information about directories passes through the index.

Well, a tree is a container for files (which in turn consist of their permissions, file names, and blobs).

On Mon, Aug 20, 2007 at 10:39:57PM +0200, David Kastrup <dak@gnu.org> wrote:

If, as Jan suggested in a previous message, you introduce directories in the index, I still don't see why you would need two entries in the tree object...

The question here was why the permissions are encoded with "%06o" while the hash is packed. Anyways, it's just a boring detail.

Mike

Mike Hommey <mh@glandium.org> writes:

As I already said: you don't. It may just be _convenient_ in the _index_ for doing single pass merge collision detection (when a directory is replaced by a file or vice versa). But there would be no additional information stored in the index, and it definitely would not be worth to do double entries in the repository: the repository is already capable of representing all the desired information and it would be wasteful to introduce any complications there when the current format already is fine.

Mike Hommey <mh@glandium.org> wrote:

Because text format is simple and pretty much everyone understands it, especially when you are talking about UNIX mode/permission bits in octal, the name is "text", and then oh, wait, those 40 bytes of hex is a lot of data - we'll just make that 20 bytes of binary instead. :-)

OK, I don't know if that's really true. Probably only Linus can speak to why the tree format is what it is. Its just how I have rationalized it myself.

Pack v4 changes the storage of the mode to be in binary, at least within the packfile, but it does still have to convert it back to the "%06o" format when computing the SHA-1 of a tree object. Believe it or not, pack v4 is being worked on again... so maybe in a future version of Git the tree modes will take up a tiny bit less space.

I haven't specifically looked at this, but I would also suspect that within any given tree's zlib dictionary the mode string portion is given a reasonably short bit sequence. So its doubtful that the mode takes up 6 bytes when compressed by zlib, its probably more likely around 1 byte. Pack v4's big win here isn't due to the mode, its due to the filenames being compressed better across trees.

"Shawn O. Pearce" <spearce@spearce.org> writes:

That is almost true, but there is one factual error I need to correct.

In-tree representation of the mode is not actually "%06o" but just "%o". In very early days of git, we used to have extra leading "0" in trees (e.g. "040000"), but that is something modern fsck even warns about. IOW, it is not the norm.

It is represented as text because we _can_ add any number of bits to the data later if we wanted to. Basic tree objects that contain only the kind of data we traditionally used will continue to work, while trees that contain (yet to be invented) new types that are represented with longer mode bits may of course not be read by older tools.

On the other hand, by definition, SHA-1 is 20-bytes. If we wanted to be able to replace hash function, we _could_ have done hashtype + length + data format (and length is represented with either text or "7-bit-per-byte plus stop bit" format as in the pack format), but there was no reason to.

The same logic applies to the loose object header -- length is not "network byte order 4-byte integer" (or 8-byte), but just a textual representation of an unsigned integer of unspecified length. The current code happen to use "%lu" with ulong only because that is the largest integral type that can be used portably and is not cumbersome to use. On future architectures with larger word size, we do not have to update the data structure definition nor even the code to read and write loose objects. Using a blob that is longer than 2^64 bytes may or may not be possible depending on how long your ulong is, of course.

On Mon, Aug 20, 2007 at 07:06:27PM -0700, Junio C Hamano <gitster@pobox.com> wrote:

(...)

That makes sense. (BTW, I must have misremembered the %06o thing, because my script to hash random trees does use %o).

Anyways, that still doesn't say why all these permission bits are kept while only the executable bit matters, though it must be related to backwards compatibility of hashes.

I'd also like to head about why git-archive doesn't set permissions like the manpage says (see my first post in the thread)

Mike

Mike Hommey <mh@glandium.org> writes:

It is not much that will in practice be retained. Keeping the whole kaboodle (including uid and gid) makes git suitable for archival purposes on Posixy platforms. The current code offers just a single policy (suitable for distributed development) with regard to what it maintains and stores (well, actually two: you can tell it to ignore the execute bit from the file system), but it would be reasonably straightforward to extend this with further policies.