[ANNOUNCE] GIT 1.5.5-rc2

GIT 1.5.5-rc2 was tagged tonight, and it is available from the usual places.

  http://www.kernel.org/pub/software/scm/git/

  git-1.5.5.rc2.tar.{gz,bz2}			(tarball)
  git-htmldocs-1.5.5.rc2.tar.{gz,bz2}		(preformatted docs)
  git-manpages-1.5.5.rc2.tar.{gz,bz2}		(preformatted docs)
  testing/git-*-1.5.5.rc2-1.$arch.rpm           (RPM)

The draft release notes as of tonight follows.

GIT v1.5.5 Release Notes ========================

Updates since v1.5.4 --------------------

(subsystems)

 * Comes with git-gui 0.9.3.

(portability)

 * We shouldn't ask for BSD group ownership semantics by setting g+s bit
   on directories on older BSD systems that refuses chmod() by non root
   users.  BSD semantics is the default there anyway.

 * Bunch of portability improvement patches coming from an effort to port
   to Solaris has been applied.

(performance)

 * On platforms with suboptimal qsort(3) implementation, there
   is an option to use more reasonable substitute we ship with
   our software.

 * New configuration variable "pack.packsizelimit" can be used
   in place of command line option --max-pack-size.

 * "git fetch" over the native git protocol used to make a
   connection to find out the set of current remote refs and
   another to actually download the pack data.  We now use only
   one connection for these tasks.

 * "git commit" does not run lstat(2) more than necessary
   anymore.

(usability, bells and whistles)

 * Bash completion script (in contrib) are aware of more commands and
   options.

 * You can be warned when core.autocrlf conversion is applied in
   such a way that results in an irreversible conversion.

 * A catch-all "color.ui" configuration variable can be used to
   enable coloring of all color-capable commands, instead of
   individual ones such as "color.status" and "color.branch".

 * The commands refused to take absolute pathnames where they
   require pathnames relative to the work tree or the current
   subdirectory.  They now can take absolute pathnames in such a
   case as long as the pathnames do not refer outside of the
   work tree.  E.g. "git add $(pwd)/foo" now works.

 * Error messages used to be sent to stderr, only to get hidden,
   when $PAGER was in use.  They now are sent to stdout along
   with the command output to be shown in the $PAGER.

 * A pattern "foo/" in .gitignore file now matches a directory
   "foo".  Pattern "foo" also matches as before.

 * bash completion's prompt helper function can talk about
   operation in-progress (e.g. merge, rebase, etc.).

 * Configuration variables "url.<usethis>.insteadof = <otherurl>" can be
   used to tell "git-fetch" and "git-push" to use different URL than what
   is given from the command line.

 * "git add -i" behaves better even before you make an initial commit.

 * "git am" refused to run from a subdirectory without a good reason.

 * After "git apply --whitespace=fix" fixes whitespace errors in a patch,
   a line before the fix can appear as a context or preimage line in a
   later patch, causing the patch not to apply.  The command now knows to
   see through whitespace fixes done to context lines to successfully
   apply such a patch series.

 * "git branch" (and "git checkout -b") to branch from a local branch can
   optionally set "branch.<name>.merge" to mark the new branch to build on
   the other local branch, when "branch.autosetupmerge" is set to
   "always", or when passing the command line option "--track" (this option
   was ignored when branching from local branches).  By default, this does
   not happen when branching from a local branch.

 * "git checkout" to switch to a branch that has "branch.<name>.merge" set
   (i.e. marked to build on another branch) reports how much the branch
   and the other branch diverged.

 * When "git checkout" has to update a lot of paths, it used to be silent
   for 4 seconds before it showed any progress report.  It is now a bit
   more impatient and starts showing progress report early.

 * "git commit" learned a new hook "prepare-commit-msg" that can
   inspect what is going to be committed and prepare the commit
   log message template to be edited.

 * "git cvsimport" can now take more than one -M options.

 * "git describe" learned to limit the tags to be used for
   naming with --match option.

 * "git describe --contains" now barfs when the named commit
   cannot be described.

 * "git describe --exact-match" describes only commits that are tagged.

 * "git describe --long" describes a tagged commit as $tag-0-$sha1,
   instead of just showing the exact tagname.

 * "git describe" warns when using a tag whose name and path contradict
   with each other.

 * "git diff" learned "--relative" option to limit and output paths
   relative to the current directory when working in a subdirectory.

 * "git diff" learned "--dirstat" option to show birds-eye-summary of
   changes more concisely than "--diffstat".

 * "git format-patch" learned --cover-letter option to generate a cover
   letter template.

 * "git gc" learned --quiet option.

 * "git gc" now automatically prunes unreachable objects that are two
   weeks old or older.

 * "git gc --auto" can be disabled more easily by just setting gc.auto
   to zero.  It also tolerates more packfiles by default.

 * "git grep" now knows "--name-only" is a synonym for the "-l" option.

 * "git help <alias>" now reports "'git <alias>' is alias to <what>",
   instead of saying "No manual entry for git-<alias>".

 * "git help" can use different backends to show manual pages and this can
   be configured using "man.viewer" configuration.

 * "gitk" does not restore window position from $HOME/.gitk anymore (it
   still restores the size).

 * "git log --grep=<what>" learned "--fixed-strings" option to look for
   <what> without treating it as a regular expression.

 * "git gui" learned an auto-spell checking.

 * "git push <somewhere> HEAD" and "git push <somewhere> +HEAD" works as
   expected; they push the current branch (and only the current branch).
   In addition, HEAD can be written as the value of "remote.<there>.push"
   configuration variable.

 * When the configuration variable "pack.threads" is set to 0, "git
   repack" auto detects the number of CPUs and uses that many threads.

 * "git send-email" learned to prompt for passwords
   interactively.

 * "git send-email" learned an easier way to suppress CC
   recipients.

 * "git stash" learned "pop" command, that applies the latest stash and
   removes it from the stash, and "drop" command to discard the named
   stash entry.

 * "git submodule" learned a new subcommand "summary" to show the
   symmetric difference between the HEAD version and the work tree version
   of the submodule commits.

 * Various "git cvsimport", "git cvsexportcommit", "git svn" and
   "git p4" improvements.

(internal)

 * Duplicated code between git-help and git-instaweb that
   launches user's preferred browser has been refactored.

 * It is now easier to write test scripts that records known
   breakages.

 * "git checkout" is rewritten in C.

 * "git remote" is rewritten in C.

 * Two conflict hunks that are separated by a very short span of common
   lines are now coalesced into one larger hunk, to make the result easier
   to read.

 * Run-command API's use of file descriptors is documented clearer and
   is more consistent now.

 * diff output can be sent to FILE * that is different from stdout.  This
   will help reimplementing more things in C.

Fixes since v1.5.4 ------------------

All of the fixes in v1.5.4 maintenance series are included in this release, unless otherwise noted.

 * "git-http-push" did not allow deletion of remote ref with the usual
   "push <remote> :<branch>" syntax.

 * "git-rebase --abort" did not go back to the right location if
   "git-reset" was run during the "git-rebase" session.

 * "git imap-send" without setting imap.host did not error out but
   segfaulted.

On Thu, Mar 27, 2008 at 11:30:27PM -0700, Junio C Hamano wrote:

I never got a response to my patches to fix encoding issues in "send-email --compose". It _is_ a bugfix, but I don't know if it is 1.5.5-worthy. Forgotten (and I should resubmit now), or should I wait until after the release?

-Peff

Jeff King <peff@peff.net> writes:

I was getting the impression that it was still in "ah, but this is better", "you are right, but how about doing this", stage and was hoping that "ok, based on the discussion here is the final one" will come soon.

On Fri, Mar 28, 2008 at 02:05:09PM -0700, Junio C Hamano wrote:

Ah. I think the current status is "here are two patches that work, but will always assume utf-8 encoding" which I think is not unreasonable as a bugfix. A nice feature would be to allow setting the encoding, but:

  - I think that is a feature, and one that nobody has expressed an
    interest in. In fact, the little rfc2047 encoding already being done
    in send-email blindly assumed utf-8.
  - if that feature is going to be done, I think some thought would have
    to go into how encodings should be specified so we don't end up with
    too many (or too few) places where you have to specify the encoding
    (IOW, I think that send-email.compose-encoding is probably too
    specific, but reusing an existing encoding variable is not quite
    right).
  - As a non-user of send-email, a rare user of encodings at all, and an
    always user of utf-8, I'm not too interested in such a feature, nor
    would I feel comfortable speaking on behalf of users who _would_ use
    such a feature.

So I think it is worth taking the patches for 1.5.5 as they are a strict improvement over the old behavior (the only reason they would not be is if somebody used a mail pipeline that assumed non-MIME stuff was in some random charset instead of us-ascii, and by setting utf-8 we are now confusing that pipeline; it seems unlikely to me, and it violates the standards).

On top of which I think they are a fine stepping stone to selecting the encoding (IOW, if I _were_ going to do such a feature, I think I would still submit those two patches as-is, and add configurability as a third patch anyway).

My only real concern is that they break something unrelated, as we are late in the -rc cycle.

-Peff

On Fri, Mar 28, 2008 at 05:23:40PM -0400, Jeff King wrote:

After sending my lengthy response, I realized that you might have actually been talking about the minor fixups that happened, and missed the "correct" 2/2 which was sent later in the thread. So following this are the most up-to-date versions. 1/1 is the same as the original, 2/2 does Teemu's "let user set subject in editor" suggestion, plus the follow-on syntax fixup.

-Peff

If the compose message contains non-ascii characters, then we assume it is in utf-8 and include the appropriate MIME headers. If the user has already included a MIME-Version header, then we assume they know what they are doing and don't add any headers.

Signed-off-by: Jeff King <peff@peff.net>
---
 git-send-email.perl   |   24 ++++++++++++++++++++++++
 t/t9001-send-email.sh |   44 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+), 0 deletions(-)

We always use 'utf-8' as the encoding, since we currently have no way of getting the information from the user.

This also refactors the quoting of recipient names, since both processes can share the rfc2047 quoting code.

Signed-off-by: Jeff King <peff@peff.net>
---
 git-send-email.perl   |   19 +++++++++++++++++--
 t/t9001-send-email.sh |   15 +++++++++++++++
 2 files changed, 32 insertions(+), 2 deletions(-)

Den Friday 28 March 2008 22.29.01 skrev Jeff King:

Don't set encoding to UTF-8 unless it actually looks like UTF-8.

-- robin

On Sat, Mar 29, 2008 at 08:19:07AM +0100, Robin Rosenberg wrote:

OK. Do you have an example function that guesses with high probability whether a string is utf-8? If there are non-ascii characters but we _don't_ guess utf-8, what should we do?

-Peff

Den Saturday 29 March 2008 08.22.03 skrev Jeff King:

Any test for valid UTF-8 will do that with a very high probability. The perl UTF-8 "api" is a mess. I couldn't find such a routine!?. Calling decode/encode and see if you get the original string works, but that is too clumsy, IMHO.

-- robin

On Sat, Mar 29, 2008 at 09:41:53AM +0100, Robin Rosenberg wrote:

Does that work? I would think you would have to compare the normalized versions of each string, since decode(encode($x)) is not, AIUI, guaranteed to produce $x.

-Peff

Den Saturday 29 March 2008 09.49.48 skrev Jeff King:

I don't claim to understand it either. Hopefully some perl guru will step forward and just explain how to do this in perl.

My proof is entirely empirical. What happens is that attempting to decode a non-UTF-8 string will put a unicode surrogate pair into the (now Unicode) string and encoding will just encode the surrogate pair into UTF-8 and not the original. As a result, the encode(decode($x)) eq $x *only* if $x is a valid UTF-8 octet sequence. Why would you not get the original back if you start with valid UTF-8?

-- robin

On Sat, Mar 29, 2008 at 10:02:43AM +0100, Robin Rosenberg wrote:

Because some UTF-8 sequences have multiple representations, and that information may be lost by whatever intermediate form is the result of decode($x). In practice, I don't know if this happens or not.

Though it looks like there is an Encode::is_utf8 function (which is also utf8::is_utf8, but only in perl >= 5.8.1). So we could use that, but it needs the utf-8 flag turned on for the string. Maybe utf8::valid is actually what we want.

But there is still a larger question. You have some binary bytes that will go in a subject header. There are non-ascii bytes. There are non-utf8 sequences. What do you do?

-Peff

Den Saturday 29 March 2008 10.11.45 skrev Jeff King:

Care to give an example?

-- robon

On Sat, Mar 29, 2008 at 10:39:43AM +0100, Robin Rosenberg wrote:

There were several given in the "OS X normalize your UTF-8 filenames" thread a while back. They generally boil down to "a<UMLAUT MODIFIER>" versus "<A WITH UMLAUT>" both of which are valid UTF-8.

-Peff

Den Saturday 29 March 2008 10.43.22 skrev Jeff King:

That is what /OS X/ does with file names. It changes one unicode code point to a sequence of other "equivalent" code points. I'm pretty sure perl does not do that.

-- robin

On Sat, Mar 29, 2008 at 01:54:47PM +0100, Robin Rosenberg wrote:

My point is that we don't _know_ what is happening in between the decode and encode. Does that intermediate form have the information required to convert back to the exact same bytes as the original form? I don't think you've provided any evidence that it does or does not.

But here is some evidence that it does work:

$ cat test.pl
sub is_valid {
  my $orig = shift;
  my $test = $orig;
  utf8::decode($test);
  utf8::encode($test);
  return $orig eq $test ? "yes" : "no";
}
print "utf-8: ", is_valid("\xc3\xb6"), "\n";
print "latin-1: ", is_valid("\xc3"), "\n";
print "utf-8 w/ combining: ", is_valid("o\xcc\x88"), "\n";

$ perl test.pl utf-8: yes latin-1: no utf-8 w/ combining: yes

But it still feels a little wrong to test by converting. There must be some way to ask "is this valid utf-8" (there are several candidate functions, but I don't think either of us quite knows the right way to invoke them).

-Peff

Jeff King wrote:

No, it doesn't. If you want that, save a copy of the string (it's a lazy copy anyway).

The module that will let you see into the strings to see what it happening is Devel::Peek. Using that, you will see the state of the UTF8 scalar flag. For example;

 maia:~$ perl -Mutf8 -MDevel::Peek -le 'Dump "Güt"'
 SV = PV(0x605d08) at 0x62f230
   REFCNT = 1
   FLAGS = (PADBUSY,PADTMP,POK,READONLY,pPOK,UTF8)
   PV = 0x60cd20 "G\303\274t"\0 [UTF8 "G\x{fc}t"]
   CUR = 4
   LEN = 8

By default, all strings that are read from files will NOT have this flag set, unless the filehandle that was read from was marked as being utf-8 (in order to preserve C semantics by default);

 maia:~$ echo "Güt" | perl -MDevel::Peek -nle 'Dump $_'
 SV = PV(0x6052d0) at 0x604220
   REFCNT = 1
   FLAGS = (POK,pPOK)
   PV = 0x62f0e0 "G\303\274t"\0
   CUR = 4
   LEN = 80
 maia:~$ echo "Güt" | perl -MDevel::Peek -nle 'BEGIN { binmode STDIN,
":utf8" } Dump $_'
 SV = PV(0x6052d0) at 0x604220
   REFCNT = 1
   FLAGS = (POK,pPOK,UTF8)
   PV = 0x62f100 "G\303\274t"\0 [UTF8 "G\x{fc}t"]
   CUR = 4
   LEN = 80

utf8::decode works in-place; it is essentially checking that the string is valid, and if so, marking it as UTF8.

   my ($encoding);
   if (utf8::decode($string)) {
       if (utf8::is_utf($string)) {
           $encoding = "UTF-8";
       }
       else {
           $encoding = "US-ASCII";
       }
   }
   else {
       $encoding = "ISO8859-1"
   }

For US-ASCII, you'll only have to encode if the string contains special characters (those below \037) or any "=" characters.

You could try using langinfo CODESET instead of hardcoding ISO8859-1 like that, but at least on my system can return bizarre values like ANSI_X3.4-1968, which may be in some contexts a "correct" description of the encoding, but is unlikely to be understood by mail clients.

I think you were just reading the note on the utf8::valid function a little too strongly.

You could use this block;

   if ($string =~ m/[\200-\377]/) {
       Encode::_utf8_on($string);
       if (!utf8::valid($string)) {
           Encode::_utf8_off($string);
       }
   }

Anyway, I guess all this rubbish is why people use CPAN modules, so that they don't have to continually rediscover every single protocol quirk and reinvent the wheel.

ie, it would be much, much simpler to use MIME::Entity->build for all of this, and remove the duplication of code.

Sam.

On Sun, Mar 30, 2008 at 04:40:53PM +1300, Sam Vilain wrote:

We do already save a copy. The question is that Robin is proposing decode/encode to check for validity. It was not clear to me that such a process would always return the exact same bytes even for valid utf-8.

But it seems like you are saying below that it is really just the "decode" part of that which is interesting:

OK, that was the magic invocation we were looking for. Thank you.

Ah, yeah. I think our tests are lacking in that they check for only [^[:ascii:]].

Yes, I actually made a similar comment recently. send-email could probably be shorter, easier to read, and have fewer bugs if it used one of the many mail-handling CPAN modules. I think it would pretty much involve scrapping the current send-email and starting fresh, though.

Thanks for your input.

-Peff

Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:

The sequence to decode followed by encode will test if you have a valid one and if it is canonically encoded, which is testing too much. You only want to check if it is valid, and do not care about normalization.

I see this in perluniintro.pod:

    =item *

    How Do I Detect Data That's Not Valid In a Particular Encoding?

    Use the C<Encode> package to try converting it.
    For example,

        use Encode 'decode_utf8';
        if (decode_utf8($string_of_bytes_that_I_think_is_utf8)) {
            # valid
        } else {
            # invalid
        }

For commit log messages, we traditionally use similar idea to guess by checking if it looks like an UTF-8 encoded string and otherwise assume Latin-1 (and I think we still do if the user does not tell us).

If this issue is only about the --compose part of send-email, perhaps you can interactively ask instead of "otherwise assume Latin-1"?

Den Saturday 29 March 2008 08.22.03 skrev Jeff King:

I guess the best bet is to assume the locale. Btw, is the encoding header from the commit (when present) completely lost? (not that it can be trusted anyway).

-- robin

On Sat, Mar 29, 2008 at 09:44:55AM +0100, Robin Rosenberg wrote:

What do you mean by "assume the locale"? Is there a portable way to say "this is the encoding of the locale the user has chosen?" On my system I set LANG=en_US, and behind-the-scenes magic chooses utf-8 versus iso8859-1.

And there is no encoding header for the commit; the point of this patch is to handle the "cover letter" message created by "send-email --compose" (we should already be doing the right thing for the patch emails, since the commit encoding is output by format-patch in a content-type header before we even get to send-email).

-Peff

Den Saturday 29 March 2008 09.53.04 skrev Jeff King:

The environment variables are only part of the story. There is a langinfo API for this. See I18N::Langinfo(3pm) that knows about those and something else.

# perl -e 'require I18N::Langinfo; I18N::Langinfo->import(qw(langinfo CODESET)); $codeset = langinfo(CODESET()); print "My codeset=". $codeset."\n";' My codeset=ISO-8859-15

-- robin

On Sat, Mar 29, 2008 at 10:38:48AM +0100, Robin Rosenberg wrote:

Hmm, neat. So perhaps it would make sense to just use this value instead of utf-8, and not worry about examining the actual text (since any such examination is at best a guess, anyway)?

Any idea what version of perl started shipping I18N::Langinfo? I couldn't see anything useful from grepping the Changes files.

-Peff

PS Your 'require' is more simply written as 'use I18N::Langinfo qw(langinfo CODESET)', or perhaps even simpler:

  perl -MI18N::Langinfo=langinfo,CODESET ...

Den Saturday 29 March 2008 10.52.38 skrev Jeff King:

I think you really should try the UTF-8 guess, since a file may well be UTF-8 even if the user locale is something else. Especially for XML files, UTF-8 is common, but there are many more cases. Look into git-gui/po for more examples. The probability of a UTF-8 test being wrong is just so unimaginable low.

See the man page, from which I stole it. It suggests you wrap it all inside eval {}, just in case your perl does not have langinfo.

As for the is_utf8() i'm not sure what it does, but I can't make it work.

-- robin

On Sat, Mar 29, 2008 at 01:54:10PM +0100, Robin Rosenberg wrote:

Thinking about this more, I think it is only half the solution. If something is not valid utf-8, then we know it must be something else. But if something is valid utf-8, is it necessarily utf-8? I think we are going to have a much higher probability of guessing wrong there.

For example, consider the bytes { 0xc3, 0xb6 }. In utf-8, they are 'ö'. But in iso8859-1, they also have meaning (paragraph symbol followed by Ã). Now that is an unlikely combination to come up. And maybe for Latin-1, having two non-ascii characters next to each other is unlikely. But over all commonly used encodings, what is the probability in an average text of that encoding that it contains valid UTF-8? For example, I have no idea what patterns can be found in EUCJP.

Yes, that does make sense for a script (I just couldn't see it because the entire toy example would be inside the eval).

There is some magic with how Perl marks strings as "binary" versus "utf-8" that I don't quite understand. And I think is_utf8 is really about asking "is the utf-8 flag set".

I think this discussion would benefit greatly from somebody who has more of a clue how perl i18n stuff works. Why don't you work up a patch that makes sense for you, and then hopefully that will get some attention?

-Peff

Den Saturday 29 March 2008 22.18.49 skrev Jeff King:

First that is even by random an unlikely sequence. For any "real" is string it simply won't happen, even in this context. Try scanning everything you can think of and see if you find such a sequence that is not actually UTF-8.

See here http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf

Note that a random string is a randomly generated string. Not a random string from the set of actually existing strings.

The only real question as I see it is whether perl has a builtin metod that works better than the decode/encode. Anyone?

-- robin

On Sat, Mar 29, 2008 at 10:43:40PM +0100, Robin Rosenberg wrote:

That's the problem I was mentioning: "everything I can think of" is basically just us-ascii with a few accented characters. I don't know how, e.g., Japanese texts will fare with such a test.

Thanks, that is an interesting read. And he seems to indicate that you can guess with a reasonable degree of success. But a few points on that work:

  - he has a specific methodology for guessing, which is more elaborate
    than what you proposed. So to get his results, you would need to
    implement his method. Hopefully if perl does have a "guess if this
    looks like utf8" method, it uses a similar scheme.

  - he does admit that some encodings have difficult to assess
    probabilities, and it will vary from language to language. See page
    22:

      If a specific language does not use all three letters (a single
      letter on the left and the corresponding two letters on the
      right), then this combination presents no danger. Further checks
      can then be made with a dictionary, although there is the problem
      that a dictionary never contains all possible words, and that of
      course resource names don't necessarily have to be words.

  - he mentions Latin, Cyrillic, and Hebrew encodings. I note the
    conspicuous absence of any Asian languages.

Sure. But looking at random strings isn't terribly useful; there is a non-uniform distribution over the set of strings, dependent on the _actual_ encoding. So there are going to be "good" encodings that will guess well, and there will be "bad" encodings that might not (and by "will", I mean "there may be"; that is the very thing I am saying we don't have good evidence for).

-Peff

Jeff King wrote:

Module::CoreList knows.  See the man page for that.

Sam.

On Sun, Mar 30, 2008 at 03:12:46PM +1300, Sam Vilain wrote:

Thanks, I didn't know about that (I foolishly assumed that such information would be, well, along with the core of perl).

The answer is: I18N::Langinfo started shipping with 5.007003. I think we have pretty much given up on perl < 5.6 (at least from my experience with 5.005 on Solaris), so it is probably safe to use.

-Peff

Last night I was going through old mail-logs and found this and another one that this is a follow-up to, which I think are still needed. Does anybody see anything wrong with them?

Jeff King <peff@peff.net> writes:

On Wed, May 21, 2008 at 12:39:44PM -0700, Junio C Hamano wrote:

Ah, thanks for bringing this up. I noticed a few weeks ago that it hadn't been applied and meant to bring it up, but somehow I failed to do so.

Obviously I'm in support of this one, but I also think Horst's patch looks correct.

-Peff