Git on Windows, CRLF issues

Hi!

I have began moving old repositories for Windows-based software to Git. Since the tool I am moving from stores everything with CRLF line endings and have RCS-like keyword expansion, I'm treating it all as binary data when exporting to Git, i.e I have CRLF in the checked-in data (and I do want that, since this is Windows-only sources).

Now the latests msysgit comes along and (finally!) sets core.autocrlf to true by default.

How do I handle this without having everyone breaking check-ins? I can't require everyone to do unset core.autocrlf globally. Can I do that with gitattributes?

Hi,

On Mon, 21 Apr 2008, Peter Karlsson wrote:

It is actually nice to hear at least _somebody_ not insulting us for this decision. Thank you!

I think that the only solution to this is (sorry!) to have one single big checkin which converts all CR/LF to LF line endings...

Desole, Dscho

On 4/21/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

If it were me (and I hope it will be, soon, if we can entirely shut down svn internally), I would prefer to use git-filter-branch to go through *all* my checkins and fix up the CRLFs in all of them. That way the history will be clean and diffs/annotates/merges will go more smoothly.

Does anyone know the most efficient way to do this with git-filter-branch, when there are already thousands of files in the repo with CRLF in them? Running dos2unix on all the files for every single revision could take a *very* long time.

Have fun,

Avery

On Mon, Apr 21, 2008 at 05:53:34PM -0400, Avery Pennarun wrote:

Yes, a tree filter would probably be quite slow due to checking out, and then munging all of the files.

You could maybe do an index filter that gets the blob SHA1 of each file that is new, and just munges those. But I think it is even simpler to just keep a cache of original blob hashes mapping to munged blob hashes.

Something like:

  git filter-branch --index-filter '
    git ls-files --stage |
    perl /path/to/caching-munger |
    git update-index --index-info
  '

where your caching munger looks something like:

-- >8 -- #!/usr/bin/perl

use strict;
use DB_File;
use Fcntl;
tie my %cache, 'DB_File', "$ENV{HOME}/filter-cache", O_RDWR|O_CREAT, 0666
  or die "unable to open db: $!";

while(<>) {
  my ($mode, $hash, $path) = /^(\d+) ([0-9a-f]{40}) \d\t(.*)/
    or die "bad ls-files line: $_";
  $cache{$hash} = munge($hash)
    unless exists $cache{$hash};
  print "$mode $cache{$hash}\t$path\n";
}

sub munge {
  my $h = shift;
  my $r = scalar `git show $h | sed 's/\$/\\r/' | git hash-object -w --stdin`;
  chomp $r;
  return $r;
}
-- 8< --

so we keep a dbm of the hash mapping, and do no work if we have already seen this blob. If we don't, then we actually do the expensive 'show | munge | hash-object'. And here our munge adds a CR, but you should be able to do an arbitrary transformation.

-Peff

On 4/21/08, Jeff King <peff@peff.net> wrote:

Thanks, this is really cool. I'll try it next time I'm messing with our repositories (this week is unfortunately a bit too busy).

Do you think git would benefit from having a generalized version of this script? Basically, the user provides a "munge" script on the command line, and there's a git-filter-branch mode for auto-munging (with a cache) every file in every checkin. Even if it's *only* ever used for CRLF, I can imagine this being useful to a lot of people.

Thanks,

Avery

Avery Pennarun:

Definitely. Also, something that would work with a) several branches (i.e traverse all the branches; keeping the points at which they diverge), and b) submodules (i.e apply the same changes to the submodules and updating the submodule index accordingly).

I ended up doing CRLF conversion for most of the repositories I had converted. Fortunately, most of them had a single branch, so after having created a small script that did CRLF->LF for the text files, I could do a

  git filter-branch --tree-filter 'c:/temp/crlf2lf.sh' \
                    --tag-name-filter 'cat' HEAD

on each repository and get everything converted during my lunch break.

What I couldn't figure out is why, after converting everything, removing all references to the repositories I cloned from, and removing references to the old objects in the reflogs, why

 git fsck --unreachable

did not report any unreachable objects? I would have guessed the entire old history and its objects would now be invalidated and could be killed off.

On Wed, Apr 23, 2008 at 08:11:49AM +0100, Peter Karlsson wrote:

Sure, but that is quite slow on a larger tree, since it has to do a full checkout for each commit. The idea of the specialized filter was to avoid that. But if your project was small enough to do it that way, that certainly works.

Did you remove refs/original/ ?

-Peff

Jeff King:

Indeed. That's why I would welcome a script such as the one you mentioned :-) Fortunately, the repositories I worked on were small enough to not suffer too much (even when using Git on Windows, which is a bit slower than on Linux).

[Not seeing any unreachable objects]

That, and cloned the repository to a new location after the conversion, and removing the references to "origin" there. It does seem that the objects are still there, but I can't see them with "gitk --all".

On Wednesday 23 April 2008, Peter Karlsson wrote:

Maybe they are kept alive by reflogs?

Have fun! :)

...Johan

Peter Karlsson schrieb:

Did you clone locally? Then you must use the file:// protocol, otherwise everything is hard-linked from the origin.

-- Hannes

On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:

It was easy enough to work up the patch below, which allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible, but running a tight loop like this is just painful. I suspect filter-branch in something like perl would be a lot faster and just as flexible (you could even do it in C, but you'd probably have to invent a little domain-specific scripting language).

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Lots of system time in both. I'm sure we spend a fair bit of time hitting our very large map and blob-cache directories, which would be much more nicely implemented as associative arrays in memory (if we were using a more featureful language).

Anyway, here is the patch. I don't know if it is even worth applying, since it is still painfully slow.

---
 git-filter-branch.sh |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

Hi,

On Wed, 23 Apr 2008, Jeff King wrote:

You create a quite huge blob-cache, so you are pretty heavy on disk-I/O. Have you tried (as suggested in the man page) to run this on a huge RAM disk? That should blow you away.

I hoped that the rewrite-commits attempt was more than just that: an attempt. So there is a point you could start from, doing things in C.

But I doubt that you get any joy: either your language is too limited, or you will get the same problems (fork() overhead) again.

I like your patch:

Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Ciao, Dscho

On Wed, Apr 23, 2008 at 11:13:27AM +0100, Johannes Schindelin wrote:

No, I didn't. But the disk I/O is pretty minimal. The blob cache is only a few megabytes, and it stays entirely in Linux's disk cache. My disk light only blinks every 5-10 seconds to flush dirty pages to disk.

Ah, right. I totally forgot about that effort. I will take a peek next time I need to do some filtering.

I think it could use some documentation updates. Avery, do you want to try adding a CRLF example to the manpage?

-Peff

Jeff King schrieb:

...

In practice, this is not sufficient. The blob filter must have an opportunity to decide what it wants to do, not just blindly munge every blob. The minimum is a path name, e.g. in $1:

	new=$(git cat-file blob $sha1 |
		$SHELL_PATH -c "$filter_blob" ignored "$path" |
		git hash-object -w --stdin)

-- Hannes

On Wed, Apr 23, 2008 at 12:58:57PM +0200, Johannes Sixt wrote:

I intentionally left that out, because:

  - I assumed if you were going to do trickery with pathnames, you
    should just be doing an index filter

  - it violates the cache assumption, which is that blob $X is always
    transformed the same way

I assume you are wanting to do something like:

  git filter-branch --blob-filter '
    case "$1" in
      *.jpg) cat ;;
          *) tr a-z A-Z ;;
    esac
  '

Obviously it is unlikely to get the same blob sha1 as "foo.jpg" and "foo.txt", but it just feels a little wrong.

-Peff

Jeff King schrieb:

Yes, that's how I intended it to work. What's wrong here? The fact that a user might name a JPEG foo.txt instead of foo.jpg? Or that the same blob might appear with entirely different names, including different suffixes? Well, tough luck. Use an index filter. But without any sort of hint what the blob is about, your original --blob-filter is useless except for the most simplistic repositories.

-- Hannes

On Wed, Apr 23, 2008 at 01:46:20PM +0200, Johannes Sixt wrote:

Yes, the script produces incorrect results if you have the same blob with different names. IOW, if I accidentally add a JPEG as 'foo', and then later rename it to 'foo.jpg', it will munge the blob the first time it sees it, and then use the munged value for 'foo.jpg', since we never even run the case statement. Yes, this is not terribly likely, but it does seem like an awful (and hard to diagnose!) bug to have hiding in the script.

The correct fix is either:

  - the blob cache needs to take into account sha1 _and_ path

  - the cache lookup needs to be _inside_ the path filter. In that case
    you would either have to support it in the script (e.g.,
    --blob-ignore jpg), or you could make the caching an optional part
    of the blob filter (the way you can call 'map' explicitly from your
    filters).

-Peff

Jeff King <peff@peff.net> writes:

But once you start saying "even originally the same blob (i.e. identified by one object name) can be rewritten into different result, depending on where in the tree it appears", would it make sense to have blob filters to begin with?

Shouldn't that kind of of context sensitive (in the space dimension -- you can introduce the context sensitivity in the time dimension by saying there may even be cases where you would want to filter differently depending on the path and which commit the blob appears, which is even worse) filtering be best left to the tree or index filter?

On 4/23/08, Junio C Hamano <gitster@pobox.com> wrote:

What I really want is the equivalent of "dos2unix --recursive *.c *.txt etc" for all commits.

Theoretically, a .txt file might be renamed to a .jpg file, in which case funny things would happen with such a filter, depending which commit was seen first.

I'm pretty confident that this will never happen to me, but it's a valid concern.

Have fun,

Avery

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

I start to wonder if "git fast-export --all | my-intelligent-perl-script | git fast-import" would not be a better solution here.

All you would have to do is to detect when a blob begins, and how long it is, and work with that. If your trees do not contain any binary files, it should be trivial.

Ciao, Dscho

On 4/24/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Err, yes... as long as there are no binary files. I'm not so lucky, and life is a little more complex in that case. It also gives no easy way of selectively applying the blob filter based on filename, which is pretty important when you do have some binary files and you're trying to decide whether to run dos2unix.

(In contrast, the *other* objection, which is that the same blob might have multiple filenames, doesn't bother me at all, since I'm sure I don't have any .txt files that were accidentally named .jpg at some point.)

I agree that a working solution based on git-fast-export/git-fast-import should run faster than any of the other proposed solutions, but my version of Jeff's patch is quite fast and it's easy to compose simple command lines that "make simple things simple and hard things possible":

  git-filter-branch --blob-filter dos2unix HEAD
  git-filter-branch --blob-filter 'case "$path" in *.c) expand -8;; *)
cat;; esac' HEAD

It sure beats writing a perl script every time you want to do something.

Jeff wrote:

The problem is that doing an optimization on an index filter is kind of hard for a user to express, and each user will have to implement the caching logic by hand every time. Using --index-filter at all requires extremely high levels of shell and git knowledge.

The fact that the blob transformation might "slightly depend on" the path is not actually very important; fundamentally we're still transforming blobs, not paths. We're just using the filename as a *hint* about what kind of transformation we need to do on that particular blob.

I think the measure of a good idea here is how straightforward it is to express what you want on the command line, and --blob-filter makes it easy to express a certain class of filters.

Have fun,

Avery

On Wed, Apr 23, 2008 at 04:01:10PM -0700, Junio C Hamano wrote:

Yes, that was my original reasoning. But I think the problem then is that the blob filter isn't terribly useful. IOW, it is not really a separate filter, but rather an optimizing pattern for an index filter, so maybe calling it a blob filter is the wrong approach, and it would be better as a short perl script in contrib/filter-branch. Then you could call:

  git filter-branch --index-filter '
    /path/to/git/contrib/filter-branch/dos2unix \
      "*.txt" "*.c"
  '

-Peff

On 4/23/08, Johannes Sixt <j.sixt@viscovery.net> wrote:

Actually, it may not have been intentional, but because of the way 'eval' works, the munge script will find that $path already contains the path of the file being munged. Works for me.

Have fun,

Avery

Avery Pennarun schrieb:

Yes, of course! So I stand corrected, and Jeff's patch makes sense.

For consistency's sake, the path should be made available in, say, GIT_BLOB_PATH just like the commit is available in GIT_COMMIT.

-- Hannes

Avery Pennarun schrieb:

I chose to write a custom script. Otherwise, a file that stays the same throughout the history would still have been converted on each commit. My script converted each unique file only once, then reconstructed the tree objects and then changed the commits.

In the end I don't think it payed off. It took me a week or so to convert the repo; I just could have let filter-branch run for a week, too. But I also have to mention that I did the CVS->git conversion a few times to get a suitable history, and I also repeated the CRLF conversion sometimes, and back then git-filter-branch did not exist in its current shape.

-- Hannes

Peter Karlsson <peter@softwolves.pp.se> writes:

I think you can, by unsetting `crlf` attribute, i.e. putting the following in .gitattributes:

   * -crlf

See gitattributes(5):

  `crlf`
  ^^^^^^

  This attribute controls the line-ending convention.

  [...]

  Unset::

        Unsetting the `crlf` attribute on a path is meant to
        mark the path as a "binary" file.  The path never goes
        through line endings conversion upon checkin/checkout.

Not tested!

Jakub Narebski:

Yeah, that does indeed seem to work, no matter how core.autocrlf is configured globally. I think this is the best way to go for the repositories I am working on (as they are very much DOS/Windows-only).

Does anyone know how to hack an existing repository so that I can add such a .gitattributes file to all commits? I've tried reading the git-filter-branch manual page a few times, but I am still confused by it.

I guess I need some combination of "git filter-branch --tree-filter" and "git update-index --add".

It doesn't matter much that the all commits are re-written, as I am still the only one to have cloned them.

Peter Karlsson schrieb:

Something like (untested, using bash):

X=$(echo "* -crlf" | git hash-object -w --stdin)
git filter-branch \
	--index-filter $'git-update-index --index-info <<< \
				"100644 $X\t.gitattributes"' \
	-- --all

-- Hannes

Peter Karlsson schrieb:

I see 2 other options:

1. Create a custom setup of msysgit that has core.autocrlf set to false.

2. You are still converting repositories? Convert the files in your
repository to LF. I did it like this, but it was a week -or more- worth of
labor to get the scripts in a shape that I could reproduce the conversion
(and it all happened before core.autocrlf even existed).

-- Hannes

Johannes Sixt:

Or, perhaps, this is the way to go.

Got to figure out how to get CRLF->LF conversion working without having RCS keyword expansion going haywire. I'm using RCS format as a middle-man between the old repositories (PVCS) I'm converting and parsecvs which imports them into Git. The old repository has expanded keywords, and I must avoid having RCS/CVS expand them as they would expand in a different manner... :-/