[ANNOUNCE] GIT 0.99.9g

GIT 0.99.9g is found at usual places. There are a couple of important changes, as the slow march towards 1.0 continues.

 - The RPM package has been split into a few packages by Jim
   Radford.  Unfortunately I am not equipped sufficiently to
   test the resulting RPMs, so please feed me updates and
   corrections as needed.  I think archimport part needs to be
   split out just like its svn/cvs cousins, and perhaps
   documentation into another separate package.

 - Fredrik Kuivinen's merge-recursive strategy is now the
   default merge strategy for two-head merge that happens after
   git-pull.  I do not expect this to cause major disruptions,
   but if this breaks things there is a workaround to override
   this [*1*].

Although I did not hear anybody jumping up-and-down to merge svnimport updates from Yaacov Akiba Slama, I did not hear it broke things either, so it graduated to the master branch and included in this release. It obviously improved things for Yaacov, and I am hoping this would not cause disruptions for people's existing setup.

Also included are unexciting bits of fixes here and there.

On the "proposed updates" front, things finally seem to be calming down.

 - One important newcomer is git-pack-redundant.  It is still in
   "pu" not because I doubt what it does is useful, but simply
   because I have not had a chance to study how it does its
   thing.  I expect to fully merge it into "master" before 1.0
   happens.

 - Among my own toys in the "pu" branch:

   - Determination of merge base for Octopus merge was quite
     pessimistic, and a proposed fix is in there; since I will
     be regularly and frequently doing Octopus merges, I'll soon
     know if this change breaks things; otherwise it will
     graduate to "master" shortly.

   - merge-base computation done by show-branch was a bit loose
     compared to the real merge-base, as pointed out by Linus on
     the list, although it does not seem to matter too much in
     practice.  Also I plan to look into merge-base to see if I
     can fix the horizon effect cheaply but that work has not
     started yet (it is triggered by fairly pathological case).

   - I got tired of not being able to get the committer date
     (except the raw format which is unreadable) out of git-log,
     and added --pretty=fuller format.  This should not break
     people's existing setup, so I expect it to move to "master"
     soon, maybe with a name change if somebody can suggest a
     better name for it.

   - Change merge-one-file to handle the case where two sides
     add the same path differently.  Instead of punting, try to
     do two-file merge from both sides.  This _might_ turn out
     to be useful, but I do not know yet, so it won't graduate
     to "master" unless somebody convinces me (and the
     community) that it is useful in some use-case scenario.

   - Add git-lost+found.  Currently the implementation stores
     found refs under .git/lost+found/{commit,other}
     directories, but writing out their object names to the
     standard output and let the users decide what to do with
     them was suggested on the list by Daniel, which makes sense
     as well.  There are pros and cons so until we know if it is
     useful and if so in what form, it will not come out of "pu"
     branch.

   - I do not consider either git-shallow-pack and git-changes
     "master" material.  The former is a hack to create
     deliberately broken repository.  The latter is supporting a
     wrong workflow, as Linus described the other day.  You can
     temporarily fetch what you want to compare into local
     repository and run git-log or git-whatchanged normally.

Oh, and we will not be moving things out of /usr/bin/ during 1.0 timeframe.

[Footnote]

*1* If for whatever reason you would prefer to keep using the 'resolve' strategy as before when you run 'git-pull', you can either do 'git-pull -s resolve <remote> <refspec>...' on the command line, or add the following in your .git/config file:

        [pull]
                twohead = resolve

On the other hand, if you like to try resolve and then recursive, you can have this instead (the order does matter, the first one is tried first):

        [pull]
                twohead = resolve
                twohead = recursive

Junio C Hamano wrote:

IMHO git-prune-packed should prune redundant pack files...

:( bummer. I do like the elegance of having /usr/bin/git executing stuff out of /usr/libexec/git.

/usr/libexec/git also makes it IMO cleaner when integrating git plugins from third parties (rpm -Uvh git-newfeature), because you don't have to worry about the /usr/bin namespace.

	Jeff

Jeff Garzik wrote:

It's nice in concept, but I think there are a lot of reasons why this is a bad idea:

- "man" doesn't handle it.  It would be another thing if "man" could be 
taught to understand commands like "man cvs checkout" or "man git fetch".

- There is no general way to teach shells etc about it, for tab 
completion etc.

- Makes it harder (but not impossible) to run git from a build directory 
without installing it first.

In comparison, the issue of clutter in /usr/bin is actually a pretty small issue, especially with htree. Most vendors have gone back to putting everything into /usr/bin since all variants that involve splitting it up seem to be more of a loss than a gain.

	-hpa

H. Peter Anvin wrote:

This is moot. man-pages can still be named git-fetch.

Add the lib directory to the path (for git-<tab><tab>) or have it auto-evaluate the result of a git command-listing.

Provided adding --lib=. is considered difficult, yes. Btw, this problem still applies as some of the programs run other programs that are expected to be in the path.

I've just posted a patch (used my submit-patch script, which was stupid since I should have posted it here) that doesn't have any of these problems.

Fair enough. With the patch I've just sent (C implementation of the 'git' program) this option is certainly available.

Andreas Ericsson wrote:

Yes, of course, but that requires the user to be aware of yet another program-specific convention. I do believe that supporting hierarchial man pages would be a good thing, but one has to start that in the proper point.

... which means the end user has to do something specific to their environment.

All in all, I think the negatives outweigh the positives.

	-hpa

H. Peter Anvin wrote:

Someone sent in a (broken) patch that pulls up the proper man-page for
	git help <command>

It's a rather good idea, so I'll be working it into the C implementation of git as soon as the core of it is implemented.

Perhaps, but allowing the possibility of splitting them can't be wrong. When that's in place we only have to decide if we're going to or not.

Jeff Garzik <jgarzik@pobox.com> writes:

Maybe not in git-prune-packed, but you are right. We should at least do that in git-prune. Maybe with something like the patch at the end?

Bummer here as well. This is not my first preference, but more or less "all things considered...". I can go over cogito and stgit with Pasky and Catalin and coordinate the transition, but at the same time, everybody's existing scripts need to be adjusted. As Linus said, we broke kernel.org snapshot scripts number of times.

Also places we execute git-upload-pack and git-receive-pack over an SSH connection need to be updated to execute 'git' with the first parameter 'upload-pack' and 'receive-pack' to make sure it would keep working with older or newer git on the other end.

After all that happens, we can start installing things in /usr/lib/git/. During the transition, the C rewrite of git wrapper posted by Andreas Ericsson might help, so I am planning to merge it before 1.0, after deciding what the right word for the "path to the rest of git executables" should be.

So let's say 1.0 will ship with all things in /usr/bin/, with updated docs that explain the situation: (1) the dash form 'git-frotz' is being deprecated, and you are encouraged to spell it as 'git frotz'; (2) if you want to use the dash form in your scripts for performance reasons, you need to have something like PATH="$(git --exec-path):$PATH" at the beginning of your script.

And after some time (say 2 months) we can switch.

I just do not want to wait that long before 1.0.

-- >8 -- cut here -- >8 -- [PATCH] git-prune: prune redundant packs.

---

Junio C Hamano wrote:

I've cooked up a patch that takes care of this if;
	git daemon
is executed (rather than git-daemon). Otherwise we could just mention in 
the docs that git-daemon must be run with the --libdir parameter (or 
whatever we decide to call it). If it prepends the libdir to the path 
everything will work same as always in the rest of the code so it'll be 
a very small change.

All I really need to finalize it is that name, so It's up to you how fast you want it. Perhaps we could take a poll?

The suggestions so far come from the threads "git binary directory?" and "[PATCH] C implementation of the 'git' program" and some I just thought up. And here are the nominees;

	libdir
	path
	exec-path

Kay Sievers pointed out that libexec is not "LSB conformant" so it might be going away and is thus not listed.

exec-path was the last suggested name I got from Junio.

The form will be
	exec_path=$(prefix)/lib/git-@@VERSION@@
	GIT_EXEC_PATH
	--exec-path

for Makefile, environment and 'git', respectively. Substitute the obvious part with whatever you prefer.

Sounds sensible, although I'm implementing Linus' idea of prepending the GIT_EXEC_PATH to $PATH so the porcelainish scripts in git-core shouldn't have to do it. If someone executes git-<script> from command-line I think it's safe to assume that they've added the exec-path to $PATH themselves. Someone *might* run

	/usr/lib/git-$GIT_VERSION/git-<script>

which should be cautioned against in the documentation.

Andreas Ericsson <ae@op5.se> writes:

Actually I was more worried about these git native protocols going over ssh, which is not helped by git-daemon. I think teaching the libdir to git-shell would make sense for "git restricted shell" users, but most users coming from ssh to run git native protocols would need to have some way of running the executable on the other end.

My current thinking about this problem is that the handful programs that need to run "on the other end" should stay in /usr/bin, even after we move most things out of /usr/bin, if only to avoid configuration hassles. They are:

	receive-pack, upload-pack
        ssh-fetch, ssh-pull, ssh-push, ssh-upload

BTW, does anybody actually use these commit walkers over SSH? Cogito switched out of it before 0.16r1 if I understand correctly, and git barebone never used it. It might not be a bad idea to deprecate these altogether, now packed transfer seems to be much nicer.

BTW^2, git-octopus should be deprecated as well; I am a bit reluctant to see git-resolve go but it probably should too.

Somehow libdir reminds me of where libraries are installed by the Makefile, which usually does not mean executables, and that was the reason I mentioned --exec-path. Although I do not have strong preference myself either way, I do not think the list cares too much either, so in order not to waste time by indecision, let's just say we use this one:

This sounds good to me; let's go with it.  Thanks.

Junio C Hamano wrote:

I liked your suggestion of deprecating the /usr/bin use a month or two before it's effected better. We could then provide symlinks for the necessary programs that point to their real locations in GIT_EXEC_PATH and (someday) drop those links when they're no longer needed.

I'll get busy then.

Andreas Ericsson <ae@op5.se> writes:

Yes, but the problem is when that "someday" comes. Unlike a single machine installation where we can tell "git" to look into somewhere different at the same time we move the subcommands out of /usr/bin, "the other end" can lag behind and sometimes not under control of the end user.

I think it's simpler to manage and can be made configuration free if we keep receive-pack and upload-pack in /usr/bin and always call these programs in dash-form (i.e. not "git upload-pack") from the other end. .bash_profile is not read for incoming ssh connections to execute a single command, but many people set their PATH in there, without setting PATH in .bashrc.

I personally think that having to set PATH in .bashrc it is actually a bug in what bash does, but that is OT.

Dear diary, on Mon, Nov 14, 2005 at 08:46:37AM CET, I got a letter where Junio C Hamano <junkio@cox.net> said that...

Yes, Cogito switched out of it before 0.16rc1, but there is still plenty of users of older Cogito versions. Well, when 0.16 is out, those should upgrade anyway, so this could be a nice gentle kick... ;-)

Junio C Hamano wrote:

Thanks for the merge. IMHO, the commit labelled "Bundle file copies from multiple branches into a merge" (109fc2b97b73090a4a0a6550cdf9b2446fd12389) needs more attention/discussion.

In svn, there is no concept of branches or tag, but because the copy is cheap, directories are used to simulate branches and tags. The repository will be like :

/trunk/path/to/file /branches/branch_1/path/to/file /branches/branch_n/path/to/file /tags/tag_1/path/to/file /tags/tag_m/path/to/file

Now, someone can copy directory or files from the trunk or any branch/tag into any other directory. For instance one can commit the following tree as a new revision :

/trunk/path/to/file /trunk/new/path/to/file (this is a copy of /branches/branch_1/path/to/file) /branches/branch_1/path/to/file /branches/branch_n/path/to/file /tags/tag_1/path/to/file /tags/tag_m/path/to/file

Now the commit 109fc2b97b73090a4a0a6550cdf9b2446fd12389 creates a new 
commit with two parents:
1) HEAD
2) the git branch called "branch_1"

 From what I read about the definition of commit in git's documentation, 
that seems to be ok, but can this marking of  "branch_1" as a parent of 
this commit be dangerous for merges done later in pure git ?

--yas

Yaacov Akiba Slama <ya@slamail.org> writes:

I think it is reasonable to record both as parents, to make the development history in branch_1 accessible from the trunk branch after they are merged, and I do not think it is dangerous at all. It is just a regular merge which, when viewed from trunk side of the history, creates a directory called 'new' at the top level, and adds bunch of files there, and if you are viewing it with rename/copy detection you may even notice that those changes are mostly copy edits.

But the above example brings up an interesting question. Subversion lets you copy freely and does not require the developer to express machine-readably what that copy is about. Also it lets copy partial trees. So it is entirely plausible to run your project like this:

	1. Repo has /trunk/i386/blah.c; i.e. 'ls' at the
           toplevel of the working tree shows 'i386' directory.

               /trunk/i386/blah.h

        2. Somebody wants to do x86-64 equivalent of existing
           thing, and starts preparing it by copying existing
           i386 thing, into his branch, and do development
           there.

		/trunk/i386/blah.c
                /branches/wip-x86-64/blah.c (copy from /trunk/i386)

        3. Later, that x86-64 equivalent matures, and gets
           merged into trunk:

		/trunk/i386/blah.c
		/trunk/x86-64/blah.c (merge back from /branches/wip-x86-64)
                /branches/wip-x86-64/blah.c (development ceased)

But it is also plausible to do this instead:

	2'. Instead of the above, you copy the whole thing

		/trunk/i386/blah.c
                /branches/wip/i386/blah.c (copy from /trunk)
                /branches/wip/x86-64/blah.c (then copy from /branches/wip/i386)

        3'. Instead of the above:

		/trunk/i386/blah.c (merge back from /branches/wip)
		/trunk/x86-64/blah.c (merge back from /branches/wip)
                /branches/wip/i386/blah.c (development ceased)
                /branches/wip/x86-64/blah.c (development ceased)

Do you need to handle the history resulting from these cases differently when importing from subversion? I have a feeling that the user needs to tell what really happend for you to handle this sensibly (tree root level is different), but I am not offhand sure what the issues are.

Junio C Hamano wrote:

May I *STRONGLY* urge you to name that something different. "lost+found" is a name with special properties in Unix; for example, many backup solutions will ignore a directory with that name.

	-hpa

"H. Peter Anvin" <hpa@zytor.com> writes:

Yeah, the original proposal (in TODO list) explicitly stated why I chose lost-found instead of lost+found back then, and somebody on the list (could have been Pasky but I may be mistaken) said not to worry. In any case, if we go the route Daniel suggests, we would not be storing anything on the filesystem ourselves so this would be a non-issue.

Dear diary, on Thu, Nov 10, 2005 at 06:44:43PM CET, I got a letter where Junio C Hamano <junkio@cox.net> said that...

It was the Large Angry SCM. I share your concern.

I like Daniel's route as well, for the separate command. But it would be nice to also have a way to tell git-fsck-cache to save the lost+found refs as it goes, much like the filesystem fsck. So if it reports some unreachable refs, you will not need to tell it to do the same job _another_ time to find out the refs and pass them to gitk. Then again, if we do this, the utility of a separate command will be questionable.

On Thu, 10 Nov 2005, Petr Baudis wrote:

Maybe git-fsck-objects should have an option to make it note dangling objects of certain types, and then count these as reachable? (That is, you want the head of an unreachable chain listed for recovery, but not other things reachable from it; you also may want the list of blobs and trees not reachable either from a ref or from something listed for recovery, but not omitting a blob reachable only from an unreachable tree)

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

I thought that was what those 'dangling blah' was about... That is, if you make commit A and then on top of that commit B, and lose both, you will see dnagling for B but not A (which is reachable from B).

On Thu, 10 Nov 2005, Junio C Hamano wrote:

Right; what I was pointing out is that you want the dangling commits, but the unreachable blobs and trees, and want reachability to count things listed as dangling, which is a somewhat novel combination of desires, I think.

	-Daniel
*This .sig left intentionally blank*

On Thu, 10 Nov 2005, Junio C Hamano wrote:

I don't know how many people do this, but with the current kernel sources, "git-fsck-cache --full" takes about a minute on a reasonable fast machine with everything in cache (ie no real disk activity to speak of)

I personally think that's fine, since I repack my trees every once in a while, and almost never run a "--full" check, I only do incrementals (which are basically free). And I suspect that I run fsck a lot more than anybody else does.

But the point is, that if you actually run fsck every time you want to visualize your pending commits, you're going to feel the pain.

I think having some kind of lost+found so that you don't have to re-run fsck just because you decided to look at them some other way (use "git log" instead of "gitk" or whatever) makes a lot of sense. But yes, it shouldn't really be called "lost+found" due to some rather serious confusion that can cause.

		Linus

On Thu, 10 Nov 2005, Linus Torvalds wrote:

                 ^^^^^^^

That should be "dangling", of course. 

		Linus

Junio C Hamano wrote:

Just realized one more issue with this... a lot of non-Unix filesystems can't deal with files with a + sign.

	-hpa

Hi,

On Thu, 10 Nov 2005, H. Peter Anvin wrote:

Two reasons against renaming:

- we call it fsck-objects for a reason. We are working on a file system, 
  which just so happens to be implemented in user space, not kernel space.
  If lost+found has to find a new name, so does fsck-objects.

- lost+found has a special meaning, granted. So, a backup would not be 
  made of it. So what? I *don't* want it backup'ed. I want to repair what
  was wrong with it. When I repaired it, the result is stored somewhere
  else. To backup lost+found would make as much sense as to backup /tmp.

Ciao, Dscho

Johannes Schindelin wrote:

I'm sorry, but that is bull. The problem here isn't the conventional naming, it's that you're implementing your filesystem on top of another filesystem, and you're running into a layering conflict.

The default should ALWAYS be no data loss.

	-hpa

On Thu, Nov 10, 2005 at 12:14:29AM -0800, Junio C Hamano wrote:

I don't agree. The chance of running git-archimport and not having arch installed is significantly less likely than the chance of not noticing that the git-archimport program exists because it was moved into a separate package that you didn't know you needed to install in the first place.

The main reason I see for splitting cvs and email import out is the non-standard dependencies, cvsps and perl(Email::Valid). While for svn import it's to keep from requiring subversion-perl of everone who installs git-core. This dependency is added automatically, so you cannot easily just ignore it like you can in the arch/tla case.

There is no need for a separate documentation RPM, since the documentation is marked as such and rpm has a standard way to avoid installing them (--excludedocs).

-Jim

Jim Radford wrote:

How is this different for when svnimport and cvsimport was moved out? I don't think anyone expected people to run those commands by accident without noticing that they fail without the svn || cvs installed underneath.

Define "non-standard". String::ShellQuote isn't installed by default on Fedora Core 3 but is required by git-archimport.

It's fairly simple to provide a custom find-requires script.

Andreas Ericsson <ae@op5.se> writes:

I am with Andreas here.

If you are using git to manage development in a tightly knitted group (e.g. company internal project) and are unlikely to be exchanging email patches, not having to pull Email::Valid into your system is a good thing, because you would not be using git-mail. If you are not dealing with development history stored in svn or tla, not having to install git-svn nor git-tla is a good thing.

Technical reasons like package availablity and required package size count, but I do not think we should be doing the split purely for technical reasons. The split should aim primarily to give convenience for the users.

That reminds me, Andreas, you said something about feeding me spec updates last week but Jim beated you. I am open to improvements from both of you. Obviously the invitation is not limited to two of you ;-).

On Thu, Nov 10, 2005 at 12:48:15PM -0800, Junio C Hamano wrote:

You are right. I had missed that it required String::ShellQuote. It should go in it's own RPM (and it already has).

-Jim