Re: [RFC PATCH] Re: Empty directories...

From: David Kastrup <dak@gnu.org>
Date: Jul 22, 2007, 13:53 UTC
Message-ID: <857iosmto0.fsf@lola.goethe.zz>
In-Reply-To: <200707221406.25541.jnareb@gmail.com>

Jakub, this mail is too long already, and it does not make sense to tack a changed proposal to its end since then the readers will be exhausted at the time they come there. So I'll instead tack a followup to the "big picture" mail instead where I outline a modified approach which is presumably easier to understand and completely backwards-compatible, incorporating your feedback.

There is probably little sense in wasting your time on a detailed response: feel free to point out where you don't see myself making sense. I have no problem with people coming to different conclusions that I do, but I would prefer it if it is not because they consider myself a raving lunatic, but because they have different opinions regarding the details.

"I can follow you, but I disagree with your conclusion" is perfectly fine for now since I am going to propose something else, anyway.

Thanks for the feedback.  It gave me some good ideas.

Jakub Narebski <jnareb@gmail.com> writes:

Show 8 quoted lines

> On Sun, 22 July 2007, David Kastrup wrote:
>> Jakub Narebski <jnareb@gmail.com> writes:
>>> David Kastrup wrote:
>>>
>>>> I must be really bad at explaining things, or I am losing a fight
>>>> against preconceptions fixed beyond my imagination.
>
> Or you are wrong...

Well, there is little reason for you to take my word on it, but I happen to have a history of designing and implementing systems where I have been responsible for every single byte, bootloader, firmware, applications, target compiler, assembler, whatever. I have been exposed to Unix and working with it several years before Linux even existed. I also have a track record of being not exactly stupid.

So I pretty much can rule out that I am wrong on the factual side.

But where I may be wrong is in estimating the how obvious the design can appear to others, and how useful and maintainable for others it may be in the long run. Linus says "code talks", but that's actually not half the story. If my code says that it works and the evidence is there, but nobody is able to understand _why_ it works, it has no place in a project where I am not permanently around.

If smart people don't get what I am talking about, it does not matter that the patch is surprisingly well-contained: it will be a maintenance nightmare because people will never figure out why something stopped working after some particular change.

Show 7 quoted lines

>> I disagree here.  The object database _can_ represent an _empty_
>> directory that has been added explicitly, because up to now no
>> operations existed that actually left an empty tree.  But it can't
>> distinguish a _non_-empty directory that has been added explicitly
>> from non-empty directory that has not been added explicitly.
>
> True. I forgot about that.

Thanks. It is almost a revelation that anybody can agree on any point with me at the moment.

Show 5 quoted lines

> IMHO it would be best to first provide plumbing infrastructure (as
> e.g.  it was the case of submodule support), then add option to
> git-update-index to change the "stickiness"/"autoremoval" status of
> a directory (of a tree), and _last_ think about how to change the
> porcelain (git-add and git-rm).

Sure. It does no harm to think about reducing the amount of breaking porcelain, though.

Show 12 quoted lines

> [...]
>
>> And a perfectly consistent way is to make those trees with an
>> explicitly added directory _non-empty_, by virtue of putting a file
>> "." in them.  This file, of course, exists in every physical
>> directory, but we may or may not decide to let it be tracked by
>> git, using the gitignore mechanism on the pattern ".".  Perfectly
>> expedient.
>
> Here we disagree. I think putting "." in a tree as marker of having
> it not be automatically deleted when empty, as opposed to marking
> tree using filemode in the parent, is not a good idea.

Well, "not a good idea" is a far step forward from "stupid idiot babbling nonsense", so we may make progress towards actually being able to _weigh_ different options. I can actually associate with "not a good idea", not least because nobody else seems to get the idea, and that makes it infeasible for maintenance.

So I'll address some points and then propose a different way of implementing what will in the end amount to rather similar semantics, but with a different view of looking at those semantics, one that corresponds well with the implementation.

Show 8 quoted lines

> The only advantage to the "." idea is that it can use gitignore
> mechanism (both in-tree .gitignore, tracked or not, and info/exclude
> file). But I also think that the fact that gitignore mechanism is
> recursive is more of disadvantage than advantage.
>
> First, it is _not_ consistent. Working directory trees _always_ have
> '.'  in them, while trees would have or would have not it, depending
> if they would be "sticky" or "autoremoved".

Let me point out again that this inconsistency is already present in the difference of tracked and untracked _files_: they are always in the working directory, while trees have or not have them, depending on whether they are "registered" or "not".

There is no inconsistency involved here, but it seems to make people _very_ uncomfortable to factor out the "stays around even if empty" functionality and call it "dir/." from the "can hold content" functionality which is in effect called "dir/", and basically associate tracked physical existence just with the former.

The recursiveness of the gitignore mechanism has the advantage that when maintaining a large repository with actual or logical subprojects, one does not need to pick a single policy for all subprojects. I think that is quite important. It could possibly be achieved with some other method of having per-subproject configuration, but I see little wrong in using what is there and documented already.

Show 7 quoted lines

> Second, the "easy implementation" is anything but easy. "git add ."
> as a way to mark directory as "sticky" is not backward compatibile:
> currently it mean to add _all contents_ of current directory.
> Implementation is tricky: as we have seen trying to unlink '.' or
> create '.' can unfortunately succeed on [some Sun OS, and UFS
> filesystem] (which follows POSIX stupidly to the letter) f**king up
> the filesystem.

I was not suggesting actually leaving any such calls in place: after all, they would presumably lead to error messages. But I agree that this could lead to nasty surprises when somebody with a legacy version of git worked with a repository containing "." as explicit entries of some file type.

> The alternative proposal of adding "magic mode" to mark directory as
> "not remove when empty" is largely tested; it is very similar to the
> subproject support.

Good.  Because it is what I converged to last night.

Show 8 quoted lines

> Third, is contrary to the git philosophy of tracking contents.
> "Stickiness" is an attribute; the fact that directory is explicitely
> tracked or not does not change contents of a directory. Compare to
> 'blob' which contains only contents of a file: not a filename, not a
> pathname, not [subset of] filemode.
>
> Fourth, is very artificial. What would you put for filemode for '.'?
> 040000 (i.e. directory)?

Taken already. By something very artificial, namely a tree... Yes, this was a wart in my proposal.

> What would you put for sha1?  Sha1 of an empty directory?

Some fixed value.  Everywhere the same.  Not really relevant.

Show 9 quoted lines

>> That basically implies that no information about directories could
>> be tracked in the repository.  And yes, we need appropriate
>> information in the index.  Again, the information whether a
>> directory was added explicitly.
>
> Whether directory is automatically managed by git (automatically
> removed or untracked). But we need directory entry in index for
> git-diff, for example to recognize if there is or there is not empty
> directory, or if a directory is automanaged or not.

One conclusion that I have come to (and I think I am in agreement with Linus here) is that the information "empty or not" is actually useless separately: when I add files below a directory to the repository, the directory _can't_ be empty. And git has no way of knowing whether it is non-empty because I wanted the directory to be there, or whether it is non-empty because I could not have checked in the files into the tree below it otherwise.

Show 9 quoted lines

>> And the repository is a versioned and hierarchically hashed version
>> of the index, but its trees contain _no_ information that is not
>> already inherently represented by the files alone. [...]
>
> The above sentence is nonsensical. Index is helper for repository,
> and can be derived from repository. Not vice versa.
>
> Trees do contain information which is not inherently present by the 
> blobs.

Could you give examples for such information? As long as we are not talking about _history_, I am at a loss at what else you mean. File names and permissions?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum