renormalize histroy with smudge/clean-filter

Hello all,

I have set up clean/smudge filters to normalzize files from an application to reduce the pain when those files are tracked by git.

The clean/smudge filter work well on new commit and the result of smudge+smudge+clean is the same as the result of a simple clean, so the filter should be fine IMHO.

But whenever I do any operations which introduce not yet normalized commits, I keep getting errors.

So to get rod of those errors, I'd like to also renormalize the history:

  $ git rebase --root --strategy renormalize
  error: Your local changes to the following files would be overwritten by
  merge:
        foo/bar/baz
  Please commit your changes or stash them before you merge.
  Aborting
  $ git add foo/bar/baz
  $ git commit -m renormalize foo/bar/baz
  $ git rebase --continue
  git: 'merge-renormalize' is not a git command. See 'git --help'.
  error: could not apply abcdef... Foo Bar Baz
  [ ... ]

Huh? I never entered a command "merge-renormalize"

BTW: It does not make any difference whether I add "-c merge.renormalze=true"

What would be the proper way to renormalize history?

Any help?

On 2025-02-05 at 21:47:26, Josef Wolf wrote:

Yes, this is known to occur. It notably happens with Git LFS, which uses smudge and clean filters, and suffers from this same problem. Renormalizing is indeed the right solution.

When you use command like `--strategy foo` with a custom strategy, Git calls a binary called `git merge-foo` to implement that strategy. So while you didn't explicitly invoke that, when you used the nonstandard strategy `renormalize` (which, by the way, does not exist), Git invoked it when you rebased, since rebases by default use merges under the hood.

That option also does not exist. Can you tell us where you found such a recommendation? If we've been misleading people in our documentation, I'd like to fix.

The command that needs to be done is `git add --renormalize .` I think you probably want to do is something like this: `git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'`.

You might also be able to use `git filter-repo` to do this in a nicer way, but I'm not aware of how to do that. I've CCed the maintainer, however, in case he or anyone else can provide an answer.

Thanks for your help, Brian!

On Wed, Feb 05, 2025 at 10:55:24PM +0000, brian m. carlson wrote:

Uh, You're right: renormalize is not a merge-strategy on its own but an option to the ort strategy.

   $ git rebase --root --strategy ort -X renormalize
   Updating files: 100% (372/372), done.
   error: Your local changes to the following files would be overwritten by merge:
       gt8/P-0113/G
       gt8/P-0113/P-0113-0_A-2
       gt8/P-0113/U
   Please commit your changes or stash them before you merge.
   Aborting
                        
Those are (some) of the files which are subject to the filtering. I can go
further with:

  $ git add --renormalize . && git commit --amend --no-edit && git rebase --continue

So this approach works. Although it needs some manual intervention.

Well, this is described in git(1) manpage:

   [ ... ]
   SYNOPSIS
       git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]
   [ ... ]                                            ^^^^^^^^^^^^^^^^^^^

Unfortunately, this runs the command on every commit and gives a warning when a cmmit don't touch a filtered file:

  $ git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'
  [ ... ]
  No changes
  You asked to amend the most recent commit, but doing so would make
  it empty. You can repeat your command with --allow-empty, or you can
  remove the commit entirely with "git reset HEAD^".

Is there a way to run the command only when rebase halts?

On 2025-02-05 at 23:59:31, Josef Wolf wrote:

The -c option does exist, and apparently the merge.renormalize option does as well, so I apologize. It looks like it's only used in merge-recursive and not merge-ort.c, so I'm not sure if it's still effective. Elijah would know for certain, since he's the author of merge-ort as well.

Yeah, that's a problem with a rebase in general here. You could try `git rebase --root -X renormalize` here, which will use the `renormalize` option, but you may run into the same problem. I _think_ with the default merge strategy in rebase that it will keep the empty commits, so your linear parts of history won't be changed, although you'll probably drop the merge commits (and any conflict resolutions) unless you use `--rebase-merges`.

If this is a small project, that may not be a problem, but I would recommend `git filter-repo` here if that's an option because it will preserve your history in a nicer way.

On Wed, Feb 5, 2025 at 4:29 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

init_*merge_options() are defined in merge-recursive.c, and these call merge_recursive_config() which is also in merge-recursive.c, but the parsed options are shared between the two backends; you'll note that merge-ort.h includes merge-recursive.h to get all these. And merge-ort does have the necessary code to use and understand the merge.renormalize option. (Of course, the fact that renormalization *requires* an index made it a bit nasty, because merge-ort was written to avoid the index as a data structure, so I had to do some ugly shenanigans in order to support that option -- https://lore.kernel.org/git/CABPp-BE1TvFJ1eOa8Ci5JTMET+dzZh3m3NxppqqWPyEp1UeAVg@mail.gmail.com/. But that's beside the point here.)

Thanks for all the insights and explanations!

I have to admit that I have a hard time to understand why the merges (and even conflicts) happen.

I have a totally linear history here. Thus, I'd expect the rebase to do something like (in pseudo-code)

   foreach $commit original-branch-commits
       git cherry-pick $commit

So I tried this and I see that cherry-pick seems to ignore the clear-filter setting and commits the smudge'd version?

My expectation would have been that every operation would run the clear filter before storing it in the repo.

Why is not everything going into the repo cleared?

On Thu, Feb 06, 2025 at 12:07:00AM -0800, Elijah Newren wrote:

On Thu, Feb 06, 2025 at 02:40:06PM +0100, Josef Wolf wrote:

I've done a lot of try and error with this approach and have come to the conclusion, that cherry-pick totally mis-behaves in the presence of clean/smudge filters.

IMHO, content should never have any chance to bypass clean filter on its way to the repository. git-cherry-pick violates this and commits the smudged content, leading to problems which can be resolved only by using git add --renormalize . && git commit --amend --no-edit

But even when git-cherry-pick starts on a normaalized commit, it tries to apply the picked commit without cleaning it before, so again conflits will be thrown.

[First]

[then]

I suspect, actually, that the biggest problem here is that cherry-pick defaults to working by using merge. Given that you want to create a new linear set of "cleaned" commits, you don't want to use `git cherry-pick` at all. Just restore the files from the original commit, then add and commit.

Chris

On Thu, Feb 06, 2025 at 10:10:26PM -0800, Chris Torek wrote:

Ummm... That's far beyond my git expertise...

I completely fail to understand why git insists to operate on smudged files in many situations.

IIUC, once clean/smudge are installed, all internal operations should be done on clean files. So why do I need this "git add --renormalize ." at all and (in the case of cherry-pick) there is not even any way to renormalize before picking.

But maybe my understanding is too simplicistic here...

On Fri, Feb 07, 2025 at 11:45:10AM +0100, Josef Wolf wrote:

[]

Now, well, there is a lot of history here. Why things work, and what is working. The short version: The '--renormalize' functionality came into Git much later then all other commands, if I simplify things.

There had been different answers here in this thread, and I try to be helpful.

In general, this could work, fully untested:

Take the first commit from your svn import. Check out a branch. Add a proper (!) .gitattributes file. run 'git add --renornormalize .' 'git commit'

Now the fun starts. From what I understand, the following could work:

  foreach $commit original-branch-commits
       git merge  -X renormalize $commit

However, I don't have such a repo to test things.

On Fri, Feb 7, 2025 at 2:46 AM Josef Wolf <jw@raven.inka.de> wrote:

It doesn't, really, and that's not the basis of the problem with rebase using merge. However:

To simplify (perhaps oversimplify, but I'll hope not), you're running afoul of an optimization trick.

Git is famously *fast* (as compared to most of the systems that came before or at the same time anyway). In the old days when I used CVS and Subversion and the like, we'd run a commit or update, and then go out for coffee or lunch or whatever, because we knew we were not going to be able to do anything for another ten minutes or perhaps even an hour or more. Then Git came along and we'd run "git checkout" or "git commit" and it would say it was done, often without even a noticeable pause, and we'd wonder if it actually did anything at all.

Git gets this speed through a lot of clever tricks, and one of them interacts poorly with clean and smudge filters *if you ever change the filter*. If the filter says constant, the tricks still work -- but what you are doing (in effect anyway) here is to change to a new filter with each commit.

Running with an explicit `--renormalize` turns off the efficiency trick. This is documented (indirectly) where

That's mostly correct. The problem here is that while `git merge` (both recursive and the new ort) has a renormalize option internally, it's not exposed to cherry-pick. Oddly, checkout obeys it. Perhaps builtin/revert.c should as well?

Chris

Oops, I left something unfinished:

On Fri, Feb 7, 2025 at 3:12 AM Chris Torek <chris.torek@gmail.com> wrote:

I forgot to fill in the "where" part.

It seems to be in both the FAQ and in `gitattributes`:

  Documentation/gitfaq.txt:You will need to run `git add
--renormalize` to have this take effect.  Note
  Documentation/gitattributes.txt:Note: Whenever the clean filter is
changed, the
repo should be renormalized:

Chris

On Fri, Feb 7, 2025 at 3:13 AM Chris Torek <chris.torek@gmail.com> wrote:

Perhaps not as a config option, but it can be selected via -Xrenormalize .

However, whether it is exposed or used doesn't matter. Renormalization in the merge machinery (this is the same for both the recursive and ort backends) is something passed to xdiff[1], for doing 3-way content merges of individual files. If a merge/rebase/cherry-pick/revert doesn't need to do a 3-way content merge for some file, then no normalization will be done for it. This could happen, for example, if one side of history being merged modified a file and the other side of history being merged didn't touch that file. And as a special case, that includes when one side of history adds the file and the other side of history doesn't have the file.

In particular, for the cherry-picks or rebasing that Josef is doing going back to the root of history, that is simply doing merges against a side of history that hasn't modified any of his files, so there isn't going to be any automatic renormalization.

The rest of what you write about optimizations is spot on, though. This isn't a bug in cherry-pick (or merge or rebase); renormalizing all files proactively in the merge machinery whenever a merge or cherry-pick is done would be orders of magnitude slower for any decently sized repository; it's simply out of the question. I think Phillip's suggestion elsewhere in this thread (git rebase --root -x 'git add --renormalize . && { git diff --quiet --cached || git commit --amend --no-edit; }') would be what Josef needs to run, ASSUMING the history Josef is operating on is linear.

Hope that helps, Elijah

[1] Okay, technically renormalization is also used to turn modify/delete conflicts into simple deletes, when the only modification was a normalization of the file contents. I don't think that's relevant to Josef's case, though, so I elided it in the explanation.

On Fri, Feb 07, 2025 at 06:01:43AM -0800, Elijah Newren wrote:

Sounds like trade of time against correctness?

See, I am sitting here trying to get this repo into a sane state for about two weeks now, and I keep getting conflicts and/or errors thrown onto me on every single attempt I try. I'd be happy to drink a whole can of coffee while some hypothetical "git renormalize-this-repo --force" is running.

On Fri, Feb 7, 2025 at 12:34 PM Josef Wolf <jw@raven.inka.de> wrote:

I may have misunderstood what folks were saying in my reading & skimming of this thread. I thought some folks were suggesting

   git rebase --root -X renormalize

as a way to renormalize the history, assuming you have linear history. I was arguing against that; it's not going to work and isn't meant to[1]. I also see I didn't look closely enough at Phillip's suggestion, which was:

   git rebase --root -x 'git add --renormalize . && { git diff --quiet
--cached || git commit --amend --no-edit; }'

which will work if you do a lot of manual work to resolve line ending difference conflicts. Since the git add at each step will modify the files on which the next commit is based, that causes the application of the subsequent commit to conflict, and you probably will have difficulty seeing those conflicts since they tend to just be line ending differences. But, mixing that with Brian's suggestion, you get:

  git rebase --root -X renormalize -x 'git add --renormalize . && {
git diff --quiet --cached || git commit --amend --no-edit; }'

which should probably work if you have a linear history (though I've never tried it myself; I've never actually used the renormalization stuff beyond making sure that merge-ort matched merge-recursive). The `git add --renormalize .` does the work of changing files, and the `-X renormalize` to git allows it to handle merging subsequent commits with the munged line ending differences as it does its work.

Were you trying one of these three?  Or something else?

Elijah

[1] The renormalize option to the merge machinery ensures that new blobs produced by the merge have normalized content, and avoid conflicts when the only differences between files are normalization ones. This option does not ensure that new trees only reference new content nor that they only reference normalized content; _any_ pre-existing blobs in the repository are fair game for new trees to reference. As per the manual: "renormalize...This runs a virtual check-out and check-in of all three stages of a file when resolving a three-way merge..." So, the existing behavior of the renormalize option to rebase/cherry-pick/merge is correct. It may not be what you want, but I don't think cherry-picking/rebasing/merging with the renormalize option is the right tool for this job.

Hi Elijah and Josef

On 08/02/2025 00:23, Elijah Newren wrote:

Indeed, I'd missed that (like you I've not actually used any smudge/clean filters)

I've tried that out with a small modification in the script below which seems to work. The modification is to add "--attr-source=$(git rev-parse HEAD)" between "git" and "rebase" so that git always has a .gitattributes file to read when rebasing commits that were made before that file was added. I wonder if we should add something about renormalizing a repository to the FAQ based on your footnote

 > [1] The renormalize option to the merge machinery ensures that new
 > blobs produced by the merge have normalized content, and avoid
 > conflicts when the only differences between files are normalization
 > ones.  This option does not ensure that new trees only reference new
 > content nor that they only reference normalized content; _any_
 > pre-existing blobs in the repository are fair game for new trees to
 > reference.  As per the manual: "renormalize...This runs a virtual
 > check-out and check-in of all three stages of a file when resolving a
 > three-way merge..."  So, the existing behavior of the renormalize
 > option to rebase/cherry-pick/merge is correct.  It may not be what you
 > want, but I don't think cherry-picking/rebasing/merging with the
 > renormalize option is the right tool for this job.
 >

Best Wishes

Phillip

--- >8 ---
#!/bin/sh
set -e
d="$(mktemp -d)"
cd "$d"
git init
echo "The   quick  brown" >file
git add file
git commit -m line-1
echo "fox  jumps    over" >>file
git commit -a -m line-2
echo "the      lazy   dog" >>file
git commit -a -m line-3
echo "file filter=space" >.gitattributes
git config filter.space.clean "sed -e 's/  */ /g'"
git config filter.space.smudge cat
git add .gitattributes
git commit -a -m 'add .gitattributes'
git reset --hard HEAD
git --attr-source=$(git rev-parse HEAD) rebase --root -X renormalize \
     -x 'git add --renormalize . && { git diff --cached --quiet || git 
commit --amend --no-edit; }'
git log -p

On Sat, Feb 08, 2025 at 11:14:57AM +0000, Phillip Wood wrote:

Uh!

My expectation would have been that this is the default?

Why on earth would one want a changing filter setting during a rebase? Can anybody outline a use-case for changing filter during operaion?

If I define a filter, I'd rather want it to be in effect on every commit of every branch.

Hi Phillip,

On Sat, Feb 8, 2025 at 3:15 AM Phillip Wood <phillip.wood123@gmail.com> wrote:

Ooh, nice catch. If folks had an appropriate .gitattributes file in place in older versions of history, they probably wouldn't have gotten into the mess.

and perhaps your helpful example?  (although it does assume linear history)  :-)

So, I'm slightly surprised here. Does the --attr-source specified to the outer git become an environment variable or something for the inner git-add invocation? How does the git add subprocess know about it?

...<does some searches ending with>...

$ git grep -5 GIT_ATTR_SOURCE -- git.c git.c- } else if (!strcmp(cmd, "--attr-source")) { git.c- if (*argc < 2) { git.c- fprintf(stderr, _("no attribute source given for --attr-source\n" )); git.c- usage(git_usage_string); git.c- } git.c: setenv(GIT_ATTR_SOURCE_ENVIRONMENT, (*argv)[1], 1); git.c- if (envchanged) git.c- *envchanged = 1; git.c- (*argv)++; git.c- (*argc)--; git.c- } else if (skip_prefix(cmd, "--attr-source=", &cmd)) { git.c- set_git_attr_source(cmd); git.c: setenv(GIT_ATTR_SOURCE_ENVIRONMENT, cmd, 1); git.c- if (envchanged) git.c- *envchanged = 1; git.c- } else if (!strcmp(cmd, "--no-advice")) { git.c- setenv(GIT_ADVICE_ENVIRONMENT, "0", 1); git.c- if (envchanged)

ahah, so it is passed via environment variable to the subprocess.

Anyway, nice catch.

Hi Elijah,

On Sat, Feb 08, 2025 at 01:43:05PM -0800, Elijah Newren wrote:

Well, you can't assume that paople get it right from the very start. An important use case of git is fixing errors made in the past, right?

In my case, I had no choice. I HAD to commit those propritary data files as-is, because I had no clue how they are structured and how those hashes are calculated. As time passed, I learned what I need to do to smudge+clean those files. But at that time a whole bunch of commits were already done.

On this roadtrip, I had to modify those .gitattributes files in various ways.

The only variant of those .gitattributes file which will work properly is the newest one. And this is also the variant wich will work for all the olter commits.

So no, I don't see why using any of the older variants of this .gitattributes would make any sense.

I find this to be confusing: the primary call should not need this parameter, since it is invoked from HEAD anyway. Everything else gets it via env-vars. I'd assume this variable will also be passed to the commands which are invoked by the -x switch?

On Sat, Feb 8, 2025 at 6:28 PM Josef Wolf <jw@raven.inka.de> wrote:

The original question said

But I'll answer this one—general operations on older history can't use a newer gitattributes declaration without explicit instruction because they'd have to know from which future to pull. Remember Git can branch, so (even assuming we had a fast way to calculate this, which AIUI we don't) from a single commit there can be multiple valid future commits with different gitattributes.

For the starting point of an operation that eventually invokes other operations, where the start clearly uses one gitattributes, it _might_ be reasonable to assume that would propagate down to the other operations.

But when subsequent operations logically operate on older history, it also seems reasonable (and unsurprising) to "do what the repository intended at that specific commit." Git assumes the latter and provides a way for you to indicate the former. Perhaps it's worth an explainer somewhere?

On Sat, Feb 08, 2025 at 09:33:05PM -0500, D. Ben Knoble wrote:

Yes, this is the way .gitattributes work. And to be honest, I always found it strange that this setting travels along the history.

But there is also ~/.giconfig, which has the drawback that in won't travel with the repo.

IMHO, it would make more sense to have some sort of global storage which travels along with the repo.

On Sat, Feb 8, 2025 at 3:28 PM Josef Wolf <jw@raven.inka.de> wrote:

[...]

Sorry if it sounded like that was passing judgement; that was not what I intended. I've been in a lot of messes too. I mean, I wrote git-filter-repo because of how many things there were to clean up. I get it, life is messy. Hindsight is 20/20. You can't let perfect be the enemy of the good. You can't prioritize "everything", you have to pick your battles. Iterative improvement, etc.

No, the primary call I think would need the parameter too; it changes HEAD immediately when it starts rebasing, and continues changing it with each commit it rebases; since it's operating on older versions, by default it'd likely pick the .gitattributes from those older versions as it goes.

Yes, I was surprised Phillip's command with --attr-source on the outer-level git invocation worked until I discovered that the code indeed sets the environment variable (which subprocesses, like those created by the --exec/-x switch, will inherit). So, yes, the -x switch stuff seems to inherit that environment variable that the primary call sets in response to that parameter.

On Sat, Feb 08, 2025 at 11:21:12PM -0800, Elijah Newren wrote:

OK. I see...

Umm... OK... This means that specifying --attr-source to the commands for the -x switch is wrong, since they have a different HEAD?

On Sun, Feb 9, 2025 at 3:58 AM Josef Wolf <jw@raven.inka.de> wrote:

Not quite: the command actually did

So the subprocesses will see the attributes source as a full-length commit hash, not "HEAD"

On Fri, Feb 07, 2025 at 04:23:45PM -0800, Elijah Newren wrote:

Yes. And this did not work.

Then there was Brian's suggenstion, so I tried:

   git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'

which won't work because not every commit touches a filtered file, so I also tried:

   git rebase --root -x 'git add --renormalize . && git status --quiet -uno | git commit --amend --no-edit'

which also did not work. Looks like git-status always exits with success. Why?

This did not work also: generated LOTS of conflicts.

Oh, have I mentioned that I am not only about line endings? Yes, I mentioned it in the very first mail. In addition to line endings, I am also about XML files from a proprietary application which reorders the XML-elements into a random order every time it ist run. So the clean-filter needs to sort the XML elements into some "canonical" order.

Yes, this finally works, IF

   git add --renormalize . && git commit --amend --no-edit

is run before starting the rebase process.

BTW: why won't

    git rebase --root -X renormalize \
     -x 'git add --renormalize .' \
     -x 'git diff --quiet --cached || git commit --amend --no-edit'

work?

Yes. And even more...

Oh, the application I am talking about also tracks changes in those XML files in corresponding hash files. I added those hash files into .gitignore and re-create them in the smudge-filter. This works fine so far, but it also generates lots of conflicts during renormalization. So I created a helper for the -x parameter of the renormalize-process to also remove those hash files:

   #! /bin/sh -e
   
   find gt8/ETS/Projekte/* -maxdepth 1 \
      -name "[BDGIUP].ets5hash" -o \
      -name "P-*.ets5hash" \
      -print0 \
     | xargs -r0 git status --short -uno \
     | sed -n "s/^...\(.*\.ets5hash\)$/\1/p" \
     | xargs -r git rm -f git --attr-source=$(git rev-parse HEAD) diff --quiet --cached || \
           git --attr-source=$(git rev-parse HEAD) commit --amend --no-edit
   
   git --attr-source=$(git rev-parse HEAD) add --renormalize .
   git --attr-source=$(git rev-parse HEAD) diff --quiet --cached || \
       git --attr-source=$(git rev-parse HEAD) commit --amend --no-edit

But no matter how I construt this, the renormalize keeps conflicting on these files. Whehn I do

    git rm -f gt8/ETS/Projekte/XXX/U.ets5hash
    git --attr-source=$(git rev-parse HEAD) commit --amend --no-edit
    git rebase --continue

manually, it works fine. Why won't the git-rm work when called from git-rebase directly?

OK.

But then, non-normalized content should go through the clean-filter before it is handed over to diff/merge when filtering is active. At least when --renormalize is in effect. Using smudged content for diff/merge operations is a sure recipe for failure.

A virtual check-out and check-in should result in smudge+clean. Running this on smudged content results in smudge+smudge+clean. Which by definition is equivalent to a simple clean. No conflicts shoud happen, then.

So the _description_ looks correct. But where do the conflicts coming from?

I don't see how the description matches actual behaviour

On Sat, Feb 8, 2025 at 12:58 PM Josef Wolf <jw@raven.inka.de> wrote:

[...]

Not quite; if the diff/merge doesn't need to look at the content of the file to resolve the merge (i.e. the merge can simply use the file's already known hash as the resolution), then, since that content isn't read it shouldn't go through any filters.

Whenever you merge two trees, only the files modified on both sides need to be inspected; the rest can be resolved without looking at their content.

The description says "This runs a virtual check-out and check-in of all three stages of a file when resolving a three-way merge..."

So, when a file needs a three-way merge to be resolved, then the virtual check-out and check-in is done. When no three-way merge is needed for a file, no virtual check-out and check-in is done.

Perhaps the documentation would be clearer if it read:

           renormalize
               This runs a virtual check-out and check-in of all three stages
               of any file which needs a three-way merge.

?

On Sat, Feb 08, 2025 at 09:57:09PM +0100, Josef Wolf wrote:

I just stumbled over another wirdeness:

This application stores the bulk of the data as text files and XML files with CRLF. But there are also some binary files. So I set gitattributes like this:

   # Catch bulk as text=crlf, rely on git to detect binary
   */*     text=auto eol=crlf
   #
   # those are known to be text=crlf
   */B     text eol=crlf
   */P-*   text eol=crlf
   #
   # smudge-clean filter
   */B     filter=etsfile
   */P-*   filter=etsfile
   #
   # files I dont't want to touch (mostly binaries)
   */*.dll       -filter -text
   */*.ver       -filter -text
   */*.lang      -filter -text
   */*.store     -filter -text
   */*.ets5hash  -filter -text

But "git ls-files --eol" gives me this:

     i/lf    w/lf    attr/text eol=crlf      gt8/ETS/Projekte/P-0113/B

Why is git ignoring my explicit CRLF setting?

This is on linux and on Windows+MSYS2. I don't have $GIT_DIR/info/attributes and ~/.gitconfig also doesn't specify any line ending things

On Sun, Feb 09, 2025 at 10:25:14AM +0100, Josef Wolf wrote:

This looks a little bit strange to me.
What happens if you replace "*/*" with "*" like this.
*     text=auto eol=crlf

Same here. What is B ? Is it a directory ?

*.dll -filter -text (and the same for everything else)

Hello Torsten,

On Sun, Feb 09, 2025 at 12:14:06PM +0100, Torsten Bögershausen wrote:

This should match all files in directories one level deeper than the directory where .gitattributes live:

   If there is a separator at the beginning or middle (or both) of the pattern,
   then the pattern is relative to the directory level of the particular
   .gitignore file itself.

Same result, but when I commit .gitattributes, I get a warning that git will do lf->crl conversion. But even after commit, no conversion is done and git-ls-files still shows:

   i/lf    w/lf    attr/text=auto eol=crlf gt8/ETS/Projekte/.gitignore

Only after removal followed by "git reset --hard", I get:

   i/lf    w/lfcr  attr/text=auto eol=crlf gt8/ETS/Projekte/.gitignore

No. It is one of the XML files I want to smudge+clean

Uh! It starts getting real wired.

After one more change to .gitattributes, one of the files marked as binary checks out as an EMPTY file and I can't find any git command to fix the situation:

  $ cat .gitattributes
  # Most files in ETS ProjectStore are XML with CRLF
  #
  * text=auto eol=crlf
  
  .gitignore     text
  .gitattributes text
  
  # Binary files
  #
  *.dat       -filter -text
  *.dll       -filter -text
  *.ver       -filter -text
  *.lang      -filter -text
  *.store     -filter -text  # <--- this is the problematic file
  *.ets5hash  -filter -text
  
  # Smudge/clean filter
  #
  */B     filter=etsfile
  */D     filter=etsfile
  */G     filter=etsfile
  */I     filter=etsfile
  */P     filter=etsfile
  */U     filter=etsfile
  */P-*   filter=etsfile

  $ git diff
  diff --git a/gt8/ETS/Projekte/P-0113/P-0113.store b/gt8/ETS/Projekte/P-0113/P-0113.store
  index c33a5239..e69de29b 100755
  --- a/gt8/ETS/Projekte/P-0113/P-0113.store
  +++ b/gt8/ETS/Projekte/P-0113/P-0113.store
  @@ -1 +0,0 @@
  -4TamRjepVNV8F+bC4nBcBwXIymvb2IQdu0qEuMSB0o0=
  \ No newline at end of file
  $ git reset --hard
  HEAD is now at 6fba03d9 Fix .gitattributes again
  $ git diff
  diff --git a/gt8/ETS/Projekte/P-0113/P-0113.store b/gt8/ETS/Projekte/P-0113/P-0113.store
  index c33a5239..e69de29b 100755
  --- a/gt8/ETS/Projekte/P-0113/P-0113.store
  +++ b/gt8/ETS/Projekte/P-0113/P-0113.store
  @@ -1 +0,0 @@
  -4TamRjepVNV8F+bC4nBcBwXIymvb2IQdu0qEuMSB0o0=
  \ No newline at end of file
  $ rm -rf P-0113/ ; git checkout P-0113/
  Updated 382 paths from the index
  $ git diff
  diff --git a/gt8/ETS/Projekte/P-0113/P-0113.store b/gt8/ETS/Projekte/P-0113/P-0113.store
  index c33a5239..e69de29b 100755
  --- a/gt8/ETS/Projekte/P-0113/P-0113.store
  +++ b/gt8/ETS/Projekte/P-0113/P-0113.store
  @@ -1 +0,0 @@
  -4TamRjepVNV8F+bC4nBcBwXIymvb2IQdu0qEuMSB0o0=
  \ No newline at end of file
  $ git ls-files --eol |grep P-0113.store
  i/none  w/none  attr/-text              P-0113/P-0113.store
  i/lf    w/crlf  attr/text=auto eol=crlf P-0113/storeVersion
  $

On Sun, Feb 09, 2025 at 06:54:50PM +0100, Josef Wolf wrote:

Upps. That's probably an ordering problem:

Later line overrides first line.

Please ignore my last mail.

On Fri, Feb 07, 2025 at 03:12:44AM -0800, Chris Torek wrote:

Well, I know the days of CVS. I even know the days of RCS. And yeah, bak in those days you used to cross your fingers hoping that all will go well while drinking the coffee.

I think there is more into git than speed.

Josef Wolf <jw@raven.inka.de> writes:

Nah, I suspect that the reason is much simpler.  

Many tools in Git toolset (like cherry-pick) were written long before clean-smudge got popular, and they were written by those who did not need clean-smudge. Those capable of updating them still have not felt the need for clean-smudge for themselves. Motivate them and we may see responses ;-)

Hi Josef

On 05/02/2025 23:59, Josef Wolf wrote:

You could try using "git diff --cached --quiet" to avoid running "git commit" if there are no changes.

     git rebase --root -x 'git add --renormalize . && { git diff --quiet 
--cached || git commit --amend --no-edit; }'

Best Wishes

Phillip

On Wed, Feb 5, 2025 at 2:55 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

`git add --renormalize .` requires a full checkout and an index. filter-repo was written to not require checkouts or an index; it should be able to operate in a bare repository as well. So, these simply don't go that well together. If we had a way to ask git "how would renormalization modify this buffer if it were at this path" we might be able to provide something (though that might require having a whole bunch of .gitattributes contents available, which might also make it tricky). Folks have requested it (https://github.com/newren/git-filter-repo/issues/375), and the final commenter provided a workaround that might be good enough for you, but I kind of think we need a way to ask git "how would renormalization modify this buffer if it were at this path" short of creating a full index and checkout.

Elijah Newren <newren@gmail.com> writes:

That makes it sound as if you are asking for "diff/patch" between pre- and post- renormalization operation, but wouldn't the question be more like "pretend this buffer content were at this path in a checkout of this tree-ish. Now compute what 'git add --renormalize' would give us for that path".

What would it take?  

 - An equivalent of the in-core index (but you need to specify from
   which tree-ish it should be taken from) so that you can learn
   what attributes are attached to the path in question.  You may
   want to grab`filter`, `ident`, `working-tree-encoding`, etc. out
   of the attribute subsystem.

 - Access to the "config" data, to learn what exact commands to
   spawn to filter the buffer for, and what encoding and line
   terminating conventions are used for given path.  You may want to
   grab values of "filter.<name>.{clean,smudge}", "core.eol", etc.,
   for example.

 - A sandbox to safely run these external commands needed for
   smudge/clean filters.

It does not sound entirely trivial, but it does not look too much of recket science, either.

Still struggling with my filter problem.

Here is what I do:

- Set up a clean filter which enforces CRLF (yes, for this specific use
  case I want CRLF even on linux)

- Smudge filter does not modify the file at all

- Set up git to fail when filter fails, so I can double-check that the
  filter is actually runnning:

   $ grep -A3 filter..etsfile ~/.gitconfig
   [filter "etsfile"]
      required = true
      clean = ets-utils -c
      smudge = ets-utils -s %f

- Specify file as non-text and install the filter:

    $ grep etsfile .gitattributes
    */P -text filter=etsfile
    $ git commit .gitattributes

- Check that git gets attributes as I want them:

    $ git --attr-source=$(git rev-parse HEAD) check-attr -a P-0113/P
    P-0113/P: text: unset
    P-0113/P: filter: etsfile
    $ git ls-files --eol P-0113/P
    i/lf    w/      attr/-text              P-0113/P

- Create helper for renormalization

    $ cat renormalization-helper
    #! /bin/sh -e
    git add --renormalize .
    git diff --quiet --cached || \
        git commit --amend --no-edit
    
- Run the renormalization for the linear history:

    $ git --attr-source=$(git rev-parse HEAD) \
         rebase --root -X renormalize \
         -x $(dirname $0)/renormalize-helper

So at this point, I'd expect the falie to have CRLF line endings. But it doesn't, so I do:

    $ rm -rf P-0113
    git checkout  --attr-source=$(git rev-parse HEAD) P-0113

Still no CRLF, so I look at what is stored by git:

    $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U

Again, no CRLF.

So I check all revisions in the history. Resut: no revision has CRLF.

So the renormalization process does not work for me at all.

Any ideas?

On Wed, Feb 12, 2025 at 12:57:07AM +0100, Josef Wolf wrote:

In general, clean filters do their work when 'git add' or 'git commit file' is run. Does the filter do the CRLF conversion ? Or is it done in .gitattributes ?

That will change the index, the repo, but not the working tree on disk, right ?

Just to make sure: You want to see the CRLF in the files on disk ? Do you have a valid .gitattributes file on disk now ? If yes, what does 'git ls-files --eol P-0113' say ? What does 'git status' say ?

In general, renormalization is about the content inside the repo. If a filter is applied, or .gitattributes are changed, the files on disk are not updated automatically. 'mv -f P-0113 /tmp && git checkout P-0113' may be needed.

Yes. The best thing to do (tm) would be to create a dummy repo, do all all the operations from scratch and post the stuff here. In other words, write a shell script that creates an empty repo, fills it with content, and does all the operations. That would enable people to reproduce it and look what is going on. Hope that make sense.

Hi Torsten,

On Wed, Feb 12, 2025 at 07:12:36AM +0100, Torsten Bögershausen wrote:

Yes. This is done in the renormalise-helper shell script, which I included into my description below:

As I wrote above: yes, the clean filter enforces CRLF

No. .gitattributes states that git should not modify the file since I have set it -text, as I wrote:

"git reset --hard" or even "rm -rf P-0113; git checkout P-0113", also do not bring the CRLF into the file, see below.

In the first place I want to see them in the repo. And a fresh checkout should bring them into the files on disk, since -text is in effect.

git recognizes my setting -text and filter=etsfile, as I wrote:

As I wrote above:

Nothing, since

  git add --renormalize . && git commit --amend --no-edit

have been done by the helper script on every commit of the history

This is why I checkd the contents which are stored in the repo:

Well, I did this instead:

Well, if I _knew_ what triggers the problem, I could create such a script.

As long as I can not figure what triggers the problem, I have to dig into internals of this old repo with long-running history.

Hi folks,

while investigating/recovering my problems with renormalizing with clean/smudge filtering, I stumbled on collisions while creating a fresh clone of the repo from the server:

   $ LANG= git clone ssh://gitrepos@my.server/repo
   smart-home-ets5hashes-removed
   Cloning into 'smart-home-ets5hashes-removed'...
   remote: Enumerating objects: 7499, done.
   remote: Counting objects: 100% (7499/7499), done.
   remote: Compressing objects: 100% (3263/3263), done.
   remote: Total 7499 (delta 3955), reused 7109 (delta 3594), pack-reused 0
   Receiving objects: 100% (7499/7499), 140.12 MiB | 10.54 MiB/s, done.
   Resolving deltas: 100% (3955/3955), done.
   Updating files: 100% (1423/1423), done.
   warning: the following paths have collided (e.g. case-sensitive paths
   on a case-insensitive filesystem) and only one from the same
   colliding group is in the working tree:
   
  'Projects/P-0113/B.ets5hash'
  [more files deleted]

This is on linux, so the FS is _not_ case-insensitive.

The list of files given here is almost identical to the list of files which always give me collisions during renormalization process.

Here is an explanation of how and why those files ended up in the repo and a hypothesis of why they might be in conflicting state.

Those files contain hash values of the real data files for a proprietary application and are re-calculated on every invocation of the application. The application won't even start up if those hashes don't match. And it won't tell why it won't start, it just says "Corrupt data".

At the time this repository started, I had no knowledge how the hashes of those files are calculated, so I had to commit them along with the associated data files to keep the application happy. This results in conflicts with many git operastions, of course.

Then I learned how those files can be re-calculated and wrote a smudge-filter to keep them in sync with the data files.

Since I was now able to recreate those files, I put them into .gitignore and installed the smudge-filter to recalculate them. But I left the files in the repo as a fallback, just to be sure. And I kept committing them every now and then whenever git showed differences, although they already were in .gitignore.

So I guess those collisions might come from committing the ignored files. Unfortunately, I could not reproduce this effect on a fresh repo, yet.

And the next question is: why do those conflicts cause the renormalization process to completely fail, even when the conflicts are resolved during the renormalization rebase? This, I also could not reproduced on a fresh repo.

On Wed, Feb 12, 2025 at 12:57:07AM +0100, Josef Wolf wrote:

On Thu, Feb 13, 2025 at 12:36:14PM +0100, Josef Wolf wrote:

That sounds fishy (tm) Does 'git ls-files' give any hints ?

Since none of the methods using plain git worked, my next try was to reach out to git-filter-repo:

Again, using my renormalize-helper script:

  $ cat renormalize-helper
  #! /bin/sh -e
  
  git add --renormalize .
  git diff --quiet --cached || \
      git commit --amend --no-edit

So I go with git-filter-repo:
 
   $ git clone ssh://gitrepos@my.server/repo fresh-clone
   $ cd fresh-clone

   $ git-filter-repo \
       --prune-empty always \
       --invert-paths --use-base-name \
       --path-regex '\.ets5hash$'

   $ for branch in branch-1 branch-2 branch-3 ; do
        git checkout -b $branch-renormalized $branch
   
        git add --renormalize .
        git diff --quiet --cached || \
             git commit -m"Renormalize HEAD"
   
        git rebase \
             --root -X renormalize \
             -x $renormalize_helper
     done

This went without problem and contents looked fine, so I really thought I got it finally.

But then I tried to move .gitattributes to the very beginnig of history:

   $ git rebase -i --root

AGAIN conflicts due to line ending errors. Adding '--attr-source=$(git rev-parse HEAD)' and '-x renormalize-helper' did not help beside moving the conflicts to another location.

Thus, although the renormalization process finished successfully, there are _still_ commits with unclean content in the repository.

I REALLY REALLY REALLY think there should be an option

--always-apply-clean-filter-to-all-content-before-feeding-to-merge-or-diff

or something!

On Wed, Feb 12, 2025 at 12:57:07AM +0100, Josef Wolf wrote:

On 2025-02-11 at 23:57:07, Josef Wolf wrote:

Is there a reason you can't use `eol=crlf` instead of a smudge/clean filter? That looks like this in the Git repo:

  *.bat text eol=crlf

That might be an easier way to accomplish what you want and it will always result in CRLF in the working tree, regardless of operating system, even though in the repository it will still use LF[0].

Note that if you need a specific encoding, there's also `working-tree-encoding` as well.

[0] Okay, technically someone can override it with `.git/info/attributes`, but if they do that and it doesn't work, that's their own fault. We don't worry about that case in this project.

On Fri, Feb 14, 2025 at 08:21:28PM +0000, brian m. carlson wrote:

Yes. Most of the data files of this (proprietary) application are XML files using mostly CRLF, but there is also LF ancoded content. Like this:

[ ... ]
 <foo>^M
  <bar>
  fonly LF in contents of bar
  </bar>^M
 </foo>^M

In addition, it randomly shuffles the XML elements at every startup, even if no changes are done. To prevent conflocts from this, I need to sort the XML elements into a canonical ordering in the clean filter.