{"thread":{"id":"62907","subject":"renormalize histroy with smudge/clean-filter","startedAt":"2025-02-05T21:48:25Z","lastAt":"2025-02-14T20:56:08Z","messageCount":44,"participants":["Josef Wolf","brian m. carlson","Elijah Newren","Phillip Wood","Junio C Hamano","Chris Torek","Torsten Bögershausen","D. Ben Knoble"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"511910","messageId":"20250205214726.GA30202@raven.inka.de","threadId":"62907","inReplyTo":null,"subject":"renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-05T21:47:26Z","receivedAt":"2025-02-05T21:48:25Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hello all,\n\nI have set up clean/smudge filters to normalzize files from an application to\nreduce the pain when those files are tracked by git.\n\nThe clean/smudge filter work well on new commit and the result of\nsmudge+smudge+clean is the same as the result of a simple clean, so the filter\nshould be fine IMHO.\n\nBut whenever I do any operations which introduce not yet normalized commits, I\nkeep getting errors.\n\nSo to get rod of those errors, I'd like to also renormalize the history:\n\n  $ git rebase --root --strategy renormalize\n  error: Your local changes to the following files would be overwritten by\n  merge:\n        foo/bar/baz\n  Please commit your changes or stash them before you merge.\n  Aborting\n  $ git add foo/bar/baz\n  $ git commit -m renormalize foo/bar/baz\n  $ git rebase --continue\n  git: 'merge-renormalize' is not a git command. See 'git --help'.\n  error: could not apply abcdef... Foo Bar Baz\n  [ ... ]\n\nHuh? I never entered a command \"merge-renormalize\"\n\nBTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n\nWhat would be the proper way to renormalize history?\n\nAny help?\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"511912","messageId":"Z6PsXGnxM3UBR3nM@tapette.crustytoothpaste.net","threadId":"62907","inReplyTo":"20250205214726.GA30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-02-05T22:55:24Z","receivedAt":"2025-02-05T22:55:33Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-02-05 at 21:47:26, Josef Wolf wrote:\n> Hello all,\n> \n> I have set up clean/smudge filters to normalzize files from an application to\n> reduce the pain when those files are tracked by git.\n> \n> The clean/smudge filter work well on new commit and the result of\n> smudge+smudge+clean is the same as the result of a simple clean, so the filter\n> should be fine IMHO.\n> \n> But whenever I do any operations which introduce not yet normalized commits, I\n> keep getting errors.\n\nYes, this is known to occur.  It notably happens with Git LFS, which\nuses smudge and clean filters, and suffers from this same problem.\nRenormalizing is indeed the right solution.\n\n> So to get rod of those errors, I'd like to also renormalize the history:\n> \n>   $ git rebase --root --strategy renormalize\n>   error: Your local changes to the following files would be overwritten by\n>   merge:\n>         foo/bar/baz\n>   Please commit your changes or stash them before you merge.\n>   Aborting\n>   $ git add foo/bar/baz\n>   $ git commit -m renormalize foo/bar/baz\n>   $ git rebase --continue\n>   git: 'merge-renormalize' is not a git command. See 'git --help'.\n>   error: could not apply abcdef... Foo Bar Baz\n>   [ ... ]\n> \n> Huh? I never entered a command \"merge-renormalize\"\n\nWhen you use command like `--strategy foo` with a custom strategy, Git\ncalls a binary called `git merge-foo` to implement that strategy.  So\nwhile you didn't explicitly invoke that, when you used the nonstandard\nstrategy `renormalize` (which, by the way, does not exist), Git invoked\nit when you rebased, since rebases by default use merges under the hood.\n\n> BTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n\nThat option also does not exist.  Can you tell us where you found such a\nrecommendation?  If we've been misleading people in our documentation,\nI'd like to fix.\n\n> What would be the proper way to renormalize history?\n\nThe command that needs to be done is `git add --renormalize .`  I think\nyou probably want to do is something like this: `git rebase --root -x\n'git add --renormalize . && git commit --amend --no-edit'`.\n\nYou might also be able to use `git filter-repo` to do this in a nicer\nway, but I'm not aware of how to do that.  I've CCed the maintainer,\nhowever, in case he or anyone else can provide an answer.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"511914","messageId":"20250205235931.GB30202@raven.inka.de","threadId":"62907","inReplyTo":"Z6PsXGnxM3UBR3nM@tapette.crustytoothpaste.net","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-05T23:59:31Z","receivedAt":"2025-02-06T00:00:19Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Thanks for your help, Brian!\n\nOn Wed, Feb 05, 2025 at 10:55:24PM +0000, brian m. carlson wrote:\n> On 2025-02-05 at 21:47:26, Josef Wolf wrote:\n> > \n> > Huh? I never entered a command \"merge-renormalize\"\n> \n> When you use command like `--strategy foo` with a custom strategy, Git\n> calls a binary called `git merge-foo` to implement that strategy.  So\n> while you didn't explicitly invoke that, when you used the nonstandard\n> strategy `renormalize` (which, by the way, does not exist), Git invoked\n> it when you rebased, since rebases by default use merges under the hood.\n\nUh, You're right: renormalize is not a merge-strategy on its own but an option\nto the ort strategy.\n\n   $ git rebase --root --strategy ort -X renormalize\n   Updating files: 100% (372/372), done.\n   error: Your local changes to the following files would be overwritten by merge:\n       gt8/P-0113/G\n       gt8/P-0113/P-0113-0_A-2\n       gt8/P-0113/U\n   Please commit your changes or stash them before you merge.\n   Aborting\n                        \nThose are (some) of the files which are subject to the filtering. I can go\nfurther with:\n\n  $ git add --renormalize . && git commit --amend --no-edit && git rebase --continue\n\nSo this approach works. Although it needs some manual intervention.\n\n> > BTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n> \n> That option also does not exist.\n\nWell, this is described in git(1) manpage:\n\n   [ ... ]\n   SYNOPSIS\n       git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n   [ ... ]                                            ^^^^^^^^^^^^^^^^^^^\n\n\n> git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n\nUnfortunately, this runs the command on every commit and gives a warning when\na cmmit don't touch a filtered file:\n\n  $ git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n  [ ... ]\n  No changes\n  You asked to amend the most recent commit, but doing so would make\n  it empty. You can repeat your command with --allow-empty, or you can\n  remove the commit entirely with \"git reset HEAD^\".\n\nIs there a way to run the command only when rebase halts?\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"511916","messageId":"Z6QCX1QZxxwC7RVQ@tapette.crustytoothpaste.net","threadId":"62907","inReplyTo":"20250205235931.GB30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-02-06T00:29:19Z","receivedAt":"2025-02-06T00:29:23Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-02-05 at 23:59:31, Josef Wolf wrote:\n> > > BTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n> > \n> > That option also does not exist.\n> \n> Well, this is described in git(1) manpage:\n> \n>    [ ... ]\n>    SYNOPSIS\n>        git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n>    [ ... ]                                            ^^^^^^^^^^^^^^^^^^^\n> \n\nThe -c option does exist, and apparently the merge.renormalize option\ndoes as well, so I apologize.  It looks like it's only used in\nmerge-recursive and not merge-ort.c, so I'm not sure if it's still\neffective.  Elijah would know for certain, since he's the author of\nmerge-ort as well.\n\n> > git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n> \n> Unfortunately, this runs the command on every commit and gives a warning when\n> a cmmit don't touch a filtered file:\n> \n>   $ git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n>   [ ... ]\n>   No changes\n>   You asked to amend the most recent commit, but doing so would make\n>   it empty. You can repeat your command with --allow-empty, or you can\n>   remove the commit entirely with \"git reset HEAD^\".\n\nYeah, that's a problem with a rebase in general here.  You could try\n`git rebase --root -X renormalize` here, which will use the\n`renormalize` option, but you may run into the same problem.  I _think_\nwith the default merge strategy in rebase that it will keep the empty\ncommits, so your linear parts of history won't be changed, although\nyou'll probably drop the merge commits (and any conflict resolutions)\nunless you use `--rebase-merges`.\n\nIf this is a small project, that may not be a problem, but I would\nrecommend `git filter-repo` here if that's an option because it will\npreserve your history in a nicer way.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"511966","messageId":"CABPp-BHkUW7NVatPOVeBPybwSq9s-HjJ1FTgwU0eZRStatfXMA@mail.gmail.com","threadId":"62907","inReplyTo":"Z6PsXGnxM3UBR3nM@tapette.crustytoothpaste.net","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-06T07:55:34Z","receivedAt":"2025-02-06T07:55:46Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 5, 2025 at 2:55 PM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n>\n> On 2025-02-05 at 21:47:26, Josef Wolf wrote:\n> > Hello all,\n> >\n> > I have set up clean/smudge filters to normalzize files from an application to\n> > reduce the pain when those files are tracked by git.\n> >\n> > The clean/smudge filter work well on new commit and the result of\n> > smudge+smudge+clean is the same as the result of a simple clean, so the filter\n> > should be fine IMHO.\n> >\n> > But whenever I do any operations which introduce not yet normalized commits, I\n> > keep getting errors.\n>\n> Yes, this is known to occur.  It notably happens with Git LFS, which\n> uses smudge and clean filters, and suffers from this same problem.\n> Renormalizing is indeed the right solution.\n>\n> > So to get rod of those errors, I'd like to also renormalize the history:\n> >\n> >   $ git rebase --root --strategy renormalize\n> >   error: Your local changes to the following files would be overwritten by\n> >   merge:\n> >         foo/bar/baz\n> >   Please commit your changes or stash them before you merge.\n> >   Aborting\n> >   $ git add foo/bar/baz\n> >   $ git commit -m renormalize foo/bar/baz\n> >   $ git rebase --continue\n> >   git: 'merge-renormalize' is not a git command. See 'git --help'.\n> >   error: could not apply abcdef... Foo Bar Baz\n> >   [ ... ]\n> >\n> > Huh? I never entered a command \"merge-renormalize\"\n>\n> When you use command like `--strategy foo` with a custom strategy, Git\n> calls a binary called `git merge-foo` to implement that strategy.  So\n> while you didn't explicitly invoke that, when you used the nonstandard\n> strategy `renormalize` (which, by the way, does not exist), Git invoked\n> it when you rebased, since rebases by default use merges under the hood.\n>\n> > BTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n>\n> That option also does not exist.  Can you tell us where you found such a\n> recommendation?  If we've been misleading people in our documentation,\n> I'd like to fix.\n>\n> > What would be the proper way to renormalize history?\n>\n> The command that needs to be done is `git add --renormalize .`  I think\n> you probably want to do is something like this: `git rebase --root -x\n> 'git add --renormalize . && git commit --amend --no-edit'`.\n>\n> You might also be able to use `git filter-repo` to do this in a nicer\n> way, but I'm not aware of how to do that.  I've CCed the maintainer,\n> however, in case he or anyone else can provide an answer.\n\n`git add --renormalize .` requires a full checkout and an index.\nfilter-repo was written to not require checkouts or an index; it\nshould be able to operate in a bare repository as well.  So, these\nsimply don't go that well together.  If we had a way to ask git \"how\nwould renormalization modify this buffer if it were at this path\" we\nmight be able to provide something (though that might require having a\nwhole bunch of .gitattributes contents available, which might also\nmake it tricky).  Folks have requested it\n(https://github.com/newren/git-filter-repo/issues/375), and the final\ncommenter provided a workaround that might be good enough for you, but\nI kind of think we need a way to ask git \"how would renormalization\nmodify this buffer if it were at this path\" short of creating a full\nindex and checkout.\n"},{"id":"511984","messageId":"CABPp-BFZ3oyKiryKMPph+nfokC=sFa7wn1wdas863273bzy7pA@mail.gmail.com","threadId":"62907","inReplyTo":"Z6QCX1QZxxwC7RVQ@tapette.crustytoothpaste.net","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-06T08:07:00Z","receivedAt":"2025-02-06T08:07:11Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 5, 2025 at 4:29 PM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n>\n> On 2025-02-05 at 23:59:31, Josef Wolf wrote:\n> > > > BTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n> > >\n> > > That option also does not exist.\n> >\n> > Well, this is described in git(1) manpage:\n> >\n> >    [ ... ]\n> >    SYNOPSIS\n> >        git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n> >    [ ... ]                                            ^^^^^^^^^^^^^^^^^^^\n> >\n>\n> The -c option does exist, and apparently the merge.renormalize option\n> does as well, so I apologize.  It looks like it's only used in\n> merge-recursive and not merge-ort.c, so I'm not sure if it's still\n> effective.  Elijah would know for certain, since he's the author of\n> merge-ort as well.\n\ninit_*merge_options() are defined in merge-recursive.c, and these call\nmerge_recursive_config() which is also in merge-recursive.c, but the\nparsed options are shared between the two backends; you'll note that\nmerge-ort.h includes merge-recursive.h to get all these.  And\nmerge-ort does have the necessary code to use and understand the\nmerge.renormalize option.  (Of course, the fact that renormalization\n*requires* an index made it a bit nasty, because merge-ort was written\nto avoid the index as a data structure, so I had to do some ugly\nshenanigans in order to support that option --\nhttps://lore.kernel.org/git/CABPp-BE1TvFJ1eOa8Ci5JTMET+dzZh3m3NxppqqWPyEp1UeAVg@mail.gmail.com/.\nBut that's beside the point here.)\n"},{"id":"511990","messageId":"c88b7b54-d032-4d91-95f8-2f139ec45b78@gmail.com","threadId":"62907","inReplyTo":"20250205235931.GB30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-02-06T10:13:43Z","receivedAt":"2025-02-06T10:13:47Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Josef\n\nOn 05/02/2025 23:59, Josef Wolf wrote:\n> \n>> git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n> \n> Unfortunately, this runs the command on every commit and gives a warning when\n> a cmmit don't touch a filtered file:\n> \n>    $ git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n>    [ ... ]\n>    No changes\n>    You asked to amend the most recent commit, but doing so would make\n>    it empty. You can repeat your command with --allow-empty, or you can\n>    remove the commit entirely with \"git reset HEAD^\".\n> \n> Is there a way to run the command only when rebase halts?\n\nYou could try using \"git diff --cached --quiet\" to avoid running \"git \ncommit\" if there are no changes.\n\n     git rebase --root -x 'git add --renormalize . && { git diff --quiet \n--cached || git commit --amend --no-edit; }'\n\nBest Wishes\n\nPhillip\n\n"},{"id":"511998","messageId":"20250206134006.GC30202@raven.inka.de","threadId":"62907","inReplyTo":"CABPp-BFZ3oyKiryKMPph+nfokC=sFa7wn1wdas863273bzy7pA@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-06T13:40:06Z","receivedAt":"2025-02-06T13:40:20Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Thanks for all the insights and explanations!\n\nI have to admit that I have a hard time to understand why the merges (and even\nconflicts) happen.\n\nI have a totally linear history here. Thus, I'd expect the rebase to do\nsomething like (in pseudo-code)\n\n   foreach $commit original-branch-commits\n       git cherry-pick $commit\n\nSo I tried this and I see that cherry-pick seems to ignore the clear-filter\nsetting and commits the smudge'd version?\n\nMy expectation would have been that every operation would run the clear filter\nbefore storing it in the repo.\n\nWhy is not everything going into the repo cleared?\n\nOn Thu, Feb 06, 2025 at 12:07:00AM -0800, Elijah Newren wrote:\n> On Wed, Feb 5, 2025 at 4:29 PM brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n> >\n> > On 2025-02-05 at 23:59:31, Josef Wolf wrote:\n> > > > > BTW: It does not make any difference whether I add \"-c merge.renormalze=true\"\n> > > >\n> > > > That option also does not exist.\n> > >\n> > > Well, this is described in git(1) manpage:\n> > >\n> > >    [ ... ]\n> > >    SYNOPSIS\n> > >        git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n> > >    [ ... ]                                            ^^^^^^^^^^^^^^^^^^^\n> > >\n> >\n> > The -c option does exist, and apparently the merge.renormalize option\n> > does as well, so I apologize.  It looks like it's only used in\n> > merge-recursive and not merge-ort.c, so I'm not sure if it's still\n> > effective.  Elijah would know for certain, since he's the author of\n> > merge-ort as well.\n> \n> init_*merge_options() are defined in merge-recursive.c, and these call\n> merge_recursive_config() which is also in merge-recursive.c, but the\n> parsed options are shared between the two backends; you'll note that\n> merge-ort.h includes merge-recursive.h to get all these.  And\n> merge-ort does have the necessary code to use and understand the\n> merge.renormalize option.  (Of course, the fact that renormalization\n> *requires* an index made it a bit nasty, because merge-ort was written\n> to avoid the index as a data structure, so I had to do some ugly\n> shenanigans in order to support that option --\n> https://lore.kernel.org/git/CABPp-BE1TvFJ1eOa8Ci5JTMET+dzZh3m3NxppqqWPyEp1UeAVg@mail.gmail.com/.\n> But that's beside the point here.)\n> \n> \n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512010","messageId":"xmqqcyfukiel.fsf@gitster.g","threadId":"62907","inReplyTo":"CABPp-BHkUW7NVatPOVeBPybwSq9s-HjJ1FTgwU0eZRStatfXMA@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-06T19:00:34Z","receivedAt":"2025-02-06T19:00:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren <newren@gmail.com> writes:\n\n> I kind of think we need a way to ask git \"how would renormalization\n> modify this buffer if it were at this path\" short of creating a full\n> index and checkout.\n\nThat makes it sound as if you are asking for \"diff/patch\" between\npre- and post- renormalization operation, but wouldn't the question\nbe more like \"pretend this buffer content were at this path in a\ncheckout of this tree-ish.  Now compute what 'git add --renormalize'\nwould give us for that path\".\n\n\nWhat would it take?  \n\n - An equivalent of the in-core index (but you need to specify from\n   which tree-ish it should be taken from) so that you can learn\n   what attributes are attached to the path in question.  You may\n   want to grab`filter`, `ident`, `working-tree-encoding`, etc. out\n   of the attribute subsystem.\n\n - Access to the \"config\" data, to learn what exact commands to\n   spawn to filter the buffer for, and what encoding and line\n   terminating conventions are used for given path.  You may want to\n   grab values of \"filter.<name>.{clean,smudge}\", \"core.eol\", etc.,\n   for example.\n\n - A sandbox to safely run these external commands needed for\n   smudge/clean filters.\n\nIt does not sound entirely trivial, but it does not look too much\nof recket science, either.\n"},{"id":"512013","messageId":"20250206200418.GD30202@raven.inka.de","threadId":"62907","inReplyTo":"20250206134006.GC30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-06T20:04:18Z","receivedAt":"2025-02-06T20:06:09Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Thu, Feb 06, 2025 at 02:40:06PM +0100, Josef Wolf wrote:\n\n>    foreach $commit original-branch-commits\n>        git cherry-pick $commit\n\nI've done a lot of try and error with this approach and have come to the\nconclusion, that cherry-pick totally mis-behaves in the presence of\nclean/smudge filters.\n\nIMHO, content should never have any chance to bypass clean filter on its way\nto the repository. git-cherry-pick violates this and commits the smudged\ncontent, leading to problems which can be resolved only by using\ngit add --renormalize . && git commit --amend --no-edit\n\nBut even when git-cherry-pick starts on a normaalized commit, it tries to\napply the picked commit without cleaning it before, so again conflits will\nbe thrown.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512025","messageId":"CAPx1Gvc2piLT=p+dvzcJPTMDQAAjQfz__O4KiRWs-fOMg8dpTw@mail.gmail.com","threadId":"62907","inReplyTo":"20250206200418.GD30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2025-02-07T06:10:26Z","receivedAt":"2025-02-07T06:10:40Z","isPatch":false,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"[First]\n\n> On Thu, Feb 06, 2025 at 02:40:06PM +0100, Josef Wolf wrote:\n>\n> >    foreach $commit original-branch-commits\n> >        git cherry-pick $commit\n\n[then]\n\n>On Thu, Feb 6, 2025 at 12:07 PM Josef Wolf <jw@raven.inka.de> wrote:\n> I've done a lot of try and error with this approach and have come to the\n> conclusion, that cherry-pick totally mis-behaves in the presence of\n> clean/smudge filters.\n\nI suspect, actually, that the biggest problem here is that cherry-pick\ndefaults to working by using merge. Given that you want to create\na new linear set of \"cleaned\" commits, you don't want to use\n`git cherry-pick` at all. Just restore the files from the original\ncommit, then add and commit.\n\nChris\n"},{"id":"512050","messageId":"20250207104510.GE30202@raven.inka.de","threadId":"62907","inReplyTo":"CAPx1Gvc2piLT=p+dvzcJPTMDQAAjQfz__O4KiRWs-fOMg8dpTw@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-07T10:45:10Z","receivedAt":"2025-02-07T10:46:08Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Thu, Feb 06, 2025 at 10:10:26PM -0800, Chris Torek wrote:\n> [First]\n> \n> > On Thu, Feb 06, 2025 at 02:40:06PM +0100, Josef Wolf wrote:\n> >\n> > >    foreach $commit original-branch-commits\n> > >        git cherry-pick $commit\n> \n> [then]\n> \n> >On Thu, Feb 6, 2025 at 12:07 PM Josef Wolf <jw@raven.inka.de> wrote:\n> > I've done a lot of try and error with this approach and have come to the\n> > conclusion, that cherry-pick totally mis-behaves in the presence of\n> > clean/smudge filters.\n> \n> I suspect, actually, that the biggest problem here is that cherry-pick\n> defaults to working by using merge. Given that you want to create\n> a new linear set of \"cleaned\" commits, you don't want to use\n> `git cherry-pick` at all. Just restore the files from the original\n> commit, then add and commit.\n\nUmmm... That's far beyond my git expertise...\n\nI completely fail to understand why git insists to operate on smudged files in\nmany situations.\n\nIIUC, once clean/smudge are installed, all internal operations should be done\non clean files. So why do I need this \"git add --renormalize .\" at all and (in\nthe case of cherry-pick) there is not even any way to renormalize before\npicking.\n\nBut maybe my understanding is too simplicistic here...\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512070","messageId":"20250207110625.GA28576@tb-raspi4","threadId":"62907","inReplyTo":"20250207104510.GE30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2025-02-07T11:06:26Z","receivedAt":"2025-02-07T11:06:34Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Feb 07, 2025 at 11:45:10AM +0100, Josef Wolf wrote:\n> On Thu, Feb 06, 2025 at 10:10:26PM -0800, Chris Torek wrote:\n> > [First]\n> >\n> > > On Thu, Feb 06, 2025 at 02:40:06PM +0100, Josef Wolf wrote:\n[]\n> Ummm... That's far beyond my git expertise...\n>\n> I completely fail to understand why git insists to operate on smudged files in\n> many situations.\n>\n> IIUC, once clean/smudge are installed, all internal operations should be done\n> on clean files. So why do I need this \"git add --renormalize .\" at all and (in\n> the case of cherry-pick) there is not even any way to renormalize before\n> picking.\n>\n> But maybe my understanding is too simplicistic here...\n\nNow, well, there is a lot of history here.\nWhy things work, and what is working.\nThe short version:\nThe '--renormalize' functionality came into Git much later then\nall other commands, if I simplify things.\n\nThere had been different answers here in this thread, and I try to\nbe helpful.\n\nIn general, this could work, fully untested:\n\nTake the first commit from your svn import.\nCheck out a branch.\nAdd a proper (!) .gitattributes file.\nrun 'git add --renornormalize .'\n'git commit'\n\nNow the fun starts. From what I understand, the following could work:\n\n  foreach $commit original-branch-commits\n       git merge  -X renormalize $commit\n\nHowever, I don't have such a repo to test things.\n"},{"id":"512071","messageId":"CAPx1GvcyaZqYK+SvgtfsajqtkMty1jOcVAtwfmam-LpOjyd0jw@mail.gmail.com","threadId":"62907","inReplyTo":"20250207104510.GE30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2025-02-07T11:12:44Z","receivedAt":"2025-02-07T11:12:58Z","isPatch":false,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Fri, Feb 7, 2025 at 2:46 AM Josef Wolf <jw@raven.inka.de> wrote:\n> I completely fail to understand why git insists to operate on smudged files in\n> many situations.\n\nIt doesn't, really, and that's not the basis of the problem with rebase using\nmerge.  However:\n\n> IIUC, once clean/smudge are installed, all internal operations should be done\n> on clean files. So why do I need this \"git add --renormalize .\" at all ...\n\nTo simplify (perhaps oversimplify, but I'll hope not), you're running afoul\nof an optimization trick.\n\nGit is famously *fast* (as compared to most of the systems that came\nbefore or at the same time anyway). In the old days when I used CVS\nand Subversion and the like, we'd run a commit or update, and then go\nout for coffee or lunch or whatever, because we knew we were not\ngoing to be able to do anything for another ten minutes or perhaps\neven an hour or more. Then Git came along and we'd run \"git checkout\"\nor \"git commit\" and it would say it was done, often without even a\nnoticeable pause, and we'd wonder if it actually did anything at all.\n\nGit gets this speed through a lot of clever tricks, and one of them\ninteracts poorly with clean and smudge filters *if you ever change\nthe filter*. If the filter says constant, the tricks still work -- but what\nyou are doing (in effect anyway) here is to change to a new filter\nwith each commit.\n\nRunning with an explicit `--renormalize` turns off the efficiency trick.\nThis is documented (indirectly) where\n\n> and (in the case of cherry-pick) there is not even any way to\n> renormalize before picking.\n\nThat's mostly correct. The problem here is that while `git merge`\n(both recursive and the new ort) has a renormalize option\ninternally, it's not exposed to cherry-pick. Oddly, checkout\nobeys it. Perhaps builtin/revert.c should as well?\n\nChris\n"},{"id":"512072","messageId":"CAPx1GvfYDRheWChrisUkZsLr4ucWO_o_k9Dh8wS3xcz2P-Pxig@mail.gmail.com","threadId":"62907","inReplyTo":"CAPx1GvcyaZqYK+SvgtfsajqtkMty1jOcVAtwfmam-LpOjyd0jw@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2025-02-07T11:17:17Z","receivedAt":"2025-02-07T11:17:30Z","isPatch":false,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"Oops, I left something unfinished:\n\nOn Fri, Feb 7, 2025 at 3:12 AM Chris Torek <chris.torek@gmail.com> wrote:\n> Running with an explicit `--renormalize` turns off the efficiency trick.\n> This is documented (indirectly) where\n\nI forgot to fill in the \"where\" part.\n\nIt seems to be in both the FAQ and in `gitattributes`:\n\n  Documentation/gitfaq.txt:You will need to run `git add\n--renormalize` to have this take effect.  Note\n  Documentation/gitattributes.txt:Note: Whenever the clean filter is\nchanged, the\nrepo should be renormalized:\n\nChris\n"},{"id":"512097","messageId":"CABPp-BFnx2m75jsa3_kTPet97HY+xwb_6JmPiKM5+OARPy=mGA@mail.gmail.com","threadId":"62907","inReplyTo":"CAPx1GvcyaZqYK+SvgtfsajqtkMty1jOcVAtwfmam-LpOjyd0jw@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-07T14:01:43Z","receivedAt":"2025-02-07T14:01:55Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 7, 2025 at 3:13 AM Chris Torek <chris.torek@gmail.com> wrote:\n>\n> > and (in the case of cherry-pick) there is not even any way to\n> > renormalize before picking.\n>\n> That's mostly correct. The problem here is that while `git merge`\n> (both recursive and the new ort) has a renormalize option\n> internally, it's not exposed to cherry-pick.\n\nPerhaps not as a config option, but it can be selected via -Xrenormalize .\n\nHowever, whether it is exposed or used doesn't matter.\nRenormalization in the merge machinery (this is the same for both the\nrecursive and ort backends) is something passed to xdiff[1], for doing\n3-way content merges of individual files. If a\nmerge/rebase/cherry-pick/revert doesn't need to do a 3-way content\nmerge for some file, then no normalization will be done for it.  This\ncould happen, for example, if one side of history being merged\nmodified a file and the other side of history being merged didn't\ntouch that file.  And as a special case, that includes when one side\nof history adds the file and the other side of history doesn't have\nthe file.\n\nIn particular, for the cherry-picks or rebasing that Josef is doing\ngoing back to the root of history, that is simply doing merges against\na side of history that hasn't modified any of his files, so there\nisn't going to be any automatic renormalization.\n\nThe rest of what you write about optimizations is spot on, though.\nThis isn't a bug in cherry-pick (or merge or rebase); renormalizing\nall files proactively in the merge machinery whenever a merge or\ncherry-pick is done would be orders of magnitude slower for any\ndecently sized repository; it's simply out of the question.  I think\nPhillip's suggestion elsewhere in this thread (git rebase --root -x\n'git add --renormalize . && { git diff --quiet --cached || git commit\n--amend --no-edit; }') would be what Josef needs to run, ASSUMING the\nhistory Josef is operating on is linear.\n\nHope that helps,\nElijah\n\n[1] Okay, technically renormalization is also used to turn\nmodify/delete conflicts into simple deletes, when the only\nmodification was a normalization of the file contents.  I don't think\nthat's relevant to Josef's case, though, so I elided it in the\nexplanation.\n"},{"id":"512101","messageId":"xmqqcyfthih8.fsf@gitster.g","threadId":"62907","inReplyTo":"20250207104510.GE30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-07T15:39:31Z","receivedAt":"2025-02-07T15:39:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josef Wolf <jw@raven.inka.de> writes:\n\n> I completely fail to understand why git insists to operate on smudged files in\n> many situations.\n>\n> IIUC, once clean/smudge are installed, all internal operations should be done\n> on clean files. So why do I need this \"git add --renormalize .\" at all and (in\n> the case of cherry-pick) there is not even any way to renormalize before\n> picking.\n>\n> But maybe my understanding is too simplicistic here...\n\nNah, I suspect that the reason is much simpler.  \n\nMany tools in Git toolset (like cherry-pick) were written long\nbefore clean-smudge got popular, and they were written by those who\ndid not need clean-smudge.  Those capable of updating them still\nhave not felt the need for clean-smudge for themselves.  Motivate\nthem and we may see responses ;-)\n"},{"id":"512116","messageId":"20250207202104.GF30202@raven.inka.de","threadId":"62907","inReplyTo":"CAPx1GvcyaZqYK+SvgtfsajqtkMty1jOcVAtwfmam-LpOjyd0jw@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-07T20:21:04Z","receivedAt":"2025-02-07T20:22:09Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Fri, Feb 07, 2025 at 03:12:44AM -0800, Chris Torek wrote:\n> On Fri, Feb 7, 2025 at 2:46 AM Josef Wolf <jw@raven.inka.de> wrote:\n\n> Git is famously *fast* (as compared to most of the systems that came\n> before or at the same time anyway). In the old days when I used CVS\n> and Subversion and the like, we'd run a commit or update, and then go\n> out for coffee or lunch or whatever, because we knew we were not\n> going to be able to do anything for another ten minutes or perhaps\n> even an hour or more. Then Git came along and we'd run \"git checkout\"\n> or \"git commit\" and it would say it was done, often without even a\n> noticeable pause, and we'd wonder if it actually did anything at all.\n\nWell, I know the days of CVS. I even know the days of RCS. And yeah, bak in\nthose days you used to cross your fingers hoping that all will go well\nwhile drinking the coffee.\n\nI think there is more into git than speed.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512117","messageId":"20250207203248.GG30202@raven.inka.de","threadId":"62907","inReplyTo":"CABPp-BFnx2m75jsa3_kTPet97HY+xwb_6JmPiKM5+OARPy=mGA@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-07T20:32:48Z","receivedAt":"2025-02-07T20:34:18Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Fri, Feb 07, 2025 at 06:01:43AM -0800, Elijah Newren wrote:\n> On Fri, Feb 7, 2025 at 3:13 AM Chris Torek <chris.torek@gmail.com> wrote:\n\n> renormalizing\n> all files proactively in the merge machinery whenever a merge or\n> cherry-pick is done would be orders of magnitude slower for any\n> decently sized repository; it's simply out of the question.\n\nSounds like trade of time against correctness?\n\nSee, I am sitting here trying to get this repo into a sane state for about\ntwo weeks now, and I keep getting conflicts and/or errors thrown onto me on\nevery single attempt I try. I'd be happy to drink a whole can of coffee while\nsome hypothetical \"git renormalize-this-repo --force\" is running.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512125","messageId":"CABPp-BFGUa_DRBe1WLVfCOKh53+F15KxW_c_OZAMwZCxuAQCiw@mail.gmail.com","threadId":"62907","inReplyTo":"20250207203248.GG30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-08T00:23:45Z","receivedAt":"2025-02-08T00:23:57Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 7, 2025 at 12:34 PM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> On Fri, Feb 07, 2025 at 06:01:43AM -0800, Elijah Newren wrote:\n> > On Fri, Feb 7, 2025 at 3:13 AM Chris Torek <chris.torek@gmail.com> wrote:\n>\n> > renormalizing\n> > all files proactively in the merge machinery whenever a merge or\n> > cherry-pick is done would be orders of magnitude slower for any\n> > decently sized repository; it's simply out of the question.\n>\n> Sounds like trade of time against correctness?\n\nI may have misunderstood what folks were saying in my reading &\nskimming of this thread.  I thought some folks were suggesting\n\n   git rebase --root -X renormalize\n\nas a way to renormalize the history, assuming you have linear history.\nI was arguing against that; it's not going to work and isn't meant\nto[1].  I also see I didn't look closely enough at Phillip's\nsuggestion, which was:\n\n   git rebase --root -x 'git add --renormalize . && { git diff --quiet\n--cached || git commit --amend --no-edit; }'\n\nwhich will work if you do a lot of manual work to resolve line ending\ndifference conflicts.  Since the git add at each step will modify the\nfiles on which the next commit is based, that causes the application\nof the subsequent commit to conflict, and you probably will have\ndifficulty seeing those conflicts since they tend to just be line\nending differences.  But, mixing that with Brian's suggestion, you\nget:\n\n  git rebase --root -X renormalize -x 'git add --renormalize . && {\ngit diff --quiet --cached || git commit --amend --no-edit; }'\n\nwhich should probably work if you have a linear history (though I've\nnever tried it myself; I've never actually used the renormalization\nstuff beyond making sure that merge-ort matched merge-recursive).  The\n`git add --renormalize .` does the work of changing files, and the `-X\nrenormalize` to git allows it to handle merging subsequent commits\nwith the munged line ending differences as it does its work.\n\nWere you trying one of these three?  Or something else?\n\nElijah\n\n\n[1] The renormalize option to the merge machinery ensures that new\nblobs produced by the merge have normalized content, and avoid\nconflicts when the only differences between files are normalization\nones.  This option does not ensure that new trees only reference new\ncontent nor that they only reference normalized content; _any_\npre-existing blobs in the repository are fair game for new trees to\nreference.  As per the manual: \"renormalize...This runs a virtual\ncheck-out and check-in of all three stages of a file when resolving a\nthree-way merge...\"  So, the existing behavior of the renormalize\noption to rebase/cherry-pick/merge is correct.  It may not be what you\nwant, but I don't think cherry-picking/rebasing/merging with the\nrenormalize option is the right tool for this job.\n"},{"id":"512132","messageId":"ba65ce17-8768-4d60-aec6-badd12930b81@gmail.com","threadId":"62907","inReplyTo":"CABPp-BFGUa_DRBe1WLVfCOKh53+F15KxW_c_OZAMwZCxuAQCiw@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-02-08T11:14:57Z","receivedAt":"2025-02-08T11:15:00Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah and Josef\n\nOn 08/02/2025 00:23, Elijah Newren wrote:\n> On Fri, Feb 7, 2025 at 12:34 PM Josef Wolf <jw@raven.inka.de> wrote:\n>> On Fri, Feb 07, 2025 at 06:01:43AM -0800, Elijah Newren wrote:\n>>> On Fri, Feb 7, 2025 at 3:13 AM Chris Torek <chris.torek@gmail.com> wrote:\n>>\n> I also see I didn't look closely enough at Phillip's\n> suggestion, which was:\n> \n>     git rebase --root -x 'git add --renormalize . && { git diff --quiet\n> --cached || git commit --amend --no-edit; }'\n> \n> which will work if you do a lot of manual work to resolve line ending\n> difference conflicts.  Since the git add at each step will modify the\n> files on which the next commit is based, that causes the application\n> of the subsequent commit to conflict,\n\nIndeed, I'd missed that (like you I've not actually used any \nsmudge/clean filters)\n\n> and you probably will have\n> difficulty seeing those conflicts since they tend to just be line\n> ending differences.  But, mixing that with Brian's suggestion, you\n> get:\n> \n>    git rebase --root -X renormalize -x 'git add --renormalize . && {\n> git diff --quiet --cached || git commit --amend --no-edit; }'\n> \n> which should probably work if you have a linear history\n\nI've tried that out with a small modification in the script below which \nseems to work. The modification is to add \"--attr-source=$(git rev-parse \nHEAD)\" between \"git\" and \"rebase\" so that git always has a \n.gitattributes file to read when rebasing commits that were made before \nthat file was added. I wonder if we should add something about \nrenormalizing a repository to the FAQ based on your footnote\n\n > [1] The renormalize option to the merge machinery ensures that new\n > blobs produced by the merge have normalized content, and avoid\n > conflicts when the only differences between files are normalization\n > ones.  This option does not ensure that new trees only reference new\n > content nor that they only reference normalized content; _any_\n > pre-existing blobs in the repository are fair game for new trees to\n > reference.  As per the manual: \"renormalize...This runs a virtual\n > check-out and check-in of all three stages of a file when resolving a\n > three-way merge...\"  So, the existing behavior of the renormalize\n > option to rebase/cherry-pick/merge is correct.  It may not be what you\n > want, but I don't think cherry-picking/rebasing/merging with the\n > renormalize option is the right tool for this job.\n >\n\nBest Wishes\n\nPhillip\n\n--- >8 ---\n#!/bin/sh\nset -e\nd=\"$(mktemp -d)\"\ncd \"$d\"\ngit init\necho \"The   quick  brown\" >file\ngit add file\ngit commit -m line-1\necho \"fox  jumps    over\" >>file\ngit commit -a -m line-2\necho \"the      lazy   dog\" >>file\ngit commit -a -m line-3\necho \"file filter=space\" >.gitattributes\ngit config filter.space.clean \"sed -e 's/  */ /g'\"\ngit config filter.space.smudge cat\ngit add .gitattributes\ngit commit -a -m 'add .gitattributes'\ngit reset --hard HEAD\ngit --attr-source=$(git rev-parse HEAD) rebase --root -X renormalize \\\n     -x 'git add --renormalize . && { git diff --cached --quiet || git \ncommit --amend --no-edit; }'\ngit log -p\n\n"},{"id":"512139","messageId":"20250208205709.GH30202@raven.inka.de","threadId":"62907","inReplyTo":"CABPp-BFGUa_DRBe1WLVfCOKh53+F15KxW_c_OZAMwZCxuAQCiw@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-08T20:57:09Z","receivedAt":"2025-02-08T20:58:24Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Fri, Feb 07, 2025 at 04:23:45PM -0800, Elijah Newren wrote:\n> I may have misunderstood what folks were saying in my reading &\n> skimming of this thread.  I thought some folks were suggesting\n> \n>    git rebase --root -X renormalize\n>\n> as a way to renormalize the history, assuming you have linear history.\n\nYes. And this did not work.\n\nThen there was Brian's suggenstion, so I tried:\n\n   git rebase --root -x 'git add --renormalize . && git commit --amend --no-edit'\n\nwhich won't work because not every commit touches a filtered file, so I also\ntried:\n\n   git rebase --root -x 'git add --renormalize . && git status --quiet -uno | git commit --amend --no-edit'\n\nwhich also did not work. Looks like git-status always exits with success. Why?\n\n> I was arguing against that; it's not going to work and isn't meant\n> to[1].  I also see I didn't look closely enough at Phillip's\n> suggestion, which was:\n> \n>    git rebase --root -x 'git add --renormalize . && { git diff --quiet\n> --cached || git commit --amend --no-edit; }'\n> \n> which will work if you do a lot of manual work to resolve line ending\n> difference conflicts.  Since the git add at each step will modify the\n> files on which the next commit is based, that causes the application\n> of the subsequent commit to conflict, and you probably will have\n> difficulty seeing those conflicts since they tend to just be line\n> ending differences.\n\nThis did not work also: generated LOTS of conflicts.\n\nOh, have I mentioned that I am not only about line endings? Yes, I mentioned\nit in the very first mail. In addition to line endings, I am also about XML\nfiles from a proprietary application which reorders the XML-elements into a\nrandom order every time it ist run. So the clean-filter needs to sort the\nXML elements into some \"canonical\" order.\n\n> But, mixing that with Brian's suggestion, you get:\n> \n>   git rebase --root -X renormalize -x 'git add --renormalize . && { git diff --quiet --cached || git commit --amend --no-edit; }'\n\nYes, this finally works, IF\n\n   git add --renormalize . && git commit --amend --no-edit\n\nis run before starting the rebase process.\n\nBTW: why won't\n\n    git rebase --root -X renormalize \\\n     -x 'git add --renormalize .' \\\n     -x 'git diff --quiet --cached || git commit --amend --no-edit'\n\nwork?\n\n> Were you trying one of these three?  Or something else?\n\nYes. And even more...\n\nOh, the application I am talking about also tracks changes in those XML files\nin corresponding hash files. I added those hash files into .gitignore and\nre-create them in the smudge-filter. This works fine so far, but it also\ngenerates lots of conflicts during renormalization. So I created a helper for\nthe -x parameter of the renormalize-process to also remove those hash files:\n\n   #! /bin/sh -e\n   \n   find gt8/ETS/Projekte/* -maxdepth 1 \\\n      -name \"[BDGIUP].ets5hash\" -o \\\n      -name \"P-*.ets5hash\" \\\n      -print0 \\\n     | xargs -r0 git status --short -uno \\\n     | sed -n \"s/^...\\(.*\\.ets5hash\\)$/\\1/p\" \\\n     | xargs -r git rm -f git --attr-source=$(git rev-parse HEAD) diff --quiet --cached || \\\n           git --attr-source=$(git rev-parse HEAD) commit --amend --no-edit\n   \n   git --attr-source=$(git rev-parse HEAD) add --renormalize .\n   git --attr-source=$(git rev-parse HEAD) diff --quiet --cached || \\\n       git --attr-source=$(git rev-parse HEAD) commit --amend --no-edit\n\nBut no matter how I construt this, the renormalize keeps conflicting on these\nfiles. Whehn I do\n\n    git rm -f gt8/ETS/Projekte/XXX/U.ets5hash\n    git --attr-source=$(git rev-parse HEAD) commit --amend --no-edit\n    git rebase --continue\n\nmanually, it works fine. Why won't the git-rm work when called from git-rebase directly?\n\n> [1] The renormalize option to the merge machinery ensures that new\n> blobs produced by the merge have normalized content, and avoid\n> conflicts when the only differences between files are normalization\n> ones.  This option does not ensure that new trees only reference new\n> content nor that they only reference normalized content; _any_\n> pre-existing blobs in the repository are fair game for new trees to\n> reference.\n\nOK.\n\nBut then, non-normalized content should go through the clean-filter before it\nis handed over to diff/merge when filtering is active. At least when --renormalize\nis in effect. Using smudged content for diff/merge operations is a sure recipe\nfor failure.\n\n> As per the manual: \"renormalize...This runs a virtual\n> check-out and check-in of all three stages of a file when resolving a\n> three-way merge...\"  So, the existing behavior of the renormalize\n> option to rebase/cherry-pick/merge is correct.\n\nA virtual check-out and check-in should result in smudge+clean. Running this\non smudged content results in smudge+smudge+clean. Which by definition is\nequivalent to a simple clean. No conflicts shoud happen, then.\n\nSo the _description_ looks correct. But where do the conflicts coming from?\n\n> It may not be what you want\n\nI don't see how the description matches actual behaviour\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512140","messageId":"20250208210824.GI30202@raven.inka.de","threadId":"62907","inReplyTo":"ba65ce17-8768-4d60-aec6-badd12930b81@gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-08T21:08:24Z","receivedAt":"2025-02-08T21:10:06Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Sat, Feb 08, 2025 at 11:14:57AM +0000, Phillip Wood wrote:\n> The modification is to add \"--attr-source=$(git rev-parse HEAD)\"\n\nUh!\n\nMy expectation would have been that this is the default?\n\nWhy on earth would one want a changing filter setting during a rebase?\nCan anybody outline a use-case for changing filter during operaion?\n\nIf I define a filter, I'd rather want it to be in effect on every commit of\nevery branch.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512141","messageId":"CABPp-BGwZ029Y8Kfr2kkGiUDZ613kxS81JXzk36V85=77KcYfA@mail.gmail.com","threadId":"62907","inReplyTo":"ba65ce17-8768-4d60-aec6-badd12930b81@gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-08T21:43:05Z","receivedAt":"2025-02-08T21:43:17Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Phillip,\n\nOn Sat, Feb 8, 2025 at 3:15 AM Phillip Wood <phillip.wood123@gmail.com> wrote:\n>\n> Hi Elijah and Josef\n>\n> On 08/02/2025 00:23, Elijah Newren wrote:\n> > On Fri, Feb 7, 2025 at 12:34 PM Josef Wolf <jw@raven.inka.de> wrote:\n> >> On Fri, Feb 07, 2025 at 06:01:43AM -0800, Elijah Newren wrote:\n> >>> On Fri, Feb 7, 2025 at 3:13 AM Chris Torek <chris.torek@gmail.com> wrote:\n> >>\n> > I also see I didn't look closely enough at Phillip's\n> > suggestion, which was:\n> >\n> >     git rebase --root -x 'git add --renormalize . && { git diff --quiet\n> > --cached || git commit --amend --no-edit; }'\n> >\n> > which will work if you do a lot of manual work to resolve line ending\n> > difference conflicts.  Since the git add at each step will modify the\n> > files on which the next commit is based, that causes the application\n> > of the subsequent commit to conflict,\n>\n> Indeed, I'd missed that (like you I've not actually used any\n> smudge/clean filters)\n>\n> > and you probably will have\n> > difficulty seeing those conflicts since they tend to just be line\n> > ending differences.  But, mixing that with Brian's suggestion, you\n> > get:\n> >\n> >    git rebase --root -X renormalize -x 'git add --renormalize . && {\n> > git diff --quiet --cached || git commit --amend --no-edit; }'\n> >\n> > which should probably work if you have a linear history\n>\n> I've tried that out with a small modification in the script below which\n> seems to work. The modification is to add \"--attr-source=$(git rev-parse\n> HEAD)\" between \"git\" and \"rebase\" so that git always has a\n> .gitattributes file to read when rebasing commits that were made before\n> that file was added.\n\nOoh, nice catch.  If folks had an appropriate .gitattributes file in\nplace in older versions of history, they probably wouldn't have gotten\ninto the mess.\n\n> I wonder if we should add something about\n> renormalizing a repository to the FAQ based on your footnote.\n\nand perhaps your helpful example?  (although it does assume linear history)  :-)\n\n>  > [1] The renormalize option to the merge machinery ensures that new\n>  > blobs produced by the merge have normalized content, and avoid\n>  > conflicts when the only differences between files are normalization\n>  > ones.  This option does not ensure that new trees only reference new\n>  > content nor that they only reference normalized content; _any_\n>  > pre-existing blobs in the repository are fair game for new trees to\n>  > reference.  As per the manual: \"renormalize...This runs a virtual\n>  > check-out and check-in of all three stages of a file when resolving a\n>  > three-way merge...\"  So, the existing behavior of the renormalize\n>  > option to rebase/cherry-pick/merge is correct.  It may not be what you\n>  > want, but I don't think cherry-picking/rebasing/merging with the\n>  > renormalize option is the right tool for this job.\n>  >\n>\n> Best Wishes\n>\n> Phillip\n>\n> --- >8 ---\n> #!/bin/sh\n> set -e\n> d=\"$(mktemp -d)\"\n> cd \"$d\"\n> git init\n> echo \"The   quick  brown\" >file\n> git add file\n> git commit -m line-1\n> echo \"fox  jumps    over\" >>file\n> git commit -a -m line-2\n> echo \"the      lazy   dog\" >>file\n> git commit -a -m line-3\n> echo \"file filter=space\" >.gitattributes\n> git config filter.space.clean \"sed -e 's/  */ /g'\"\n> git config filter.space.smudge cat\n> git add .gitattributes\n> git commit -a -m 'add .gitattributes'\n> git reset --hard HEAD\n> git --attr-source=$(git rev-parse HEAD) rebase --root -X renormalize \\\n>      -x 'git add --renormalize . && { git diff --cached --quiet || git\n> commit --amend --no-edit; }'\n\nSo, I'm slightly surprised here.  Does the --attr-source specified to\nthe outer git become an environment variable or something for the\ninner git-add invocation?  How does the git add subprocess know about\nit?\n\n...<does some searches ending with>...\n\n$ git grep -5 GIT_ATTR_SOURCE -- git.c\ngit.c-          } else if (!strcmp(cmd, \"--attr-source\")) {\ngit.c-                  if (*argc < 2) {\ngit.c-                          fprintf(stderr, _(\"no attribute source\ngiven for --attr-source\\n\" ));\ngit.c-                          usage(git_usage_string);\ngit.c-                  }\ngit.c:                  setenv(GIT_ATTR_SOURCE_ENVIRONMENT, (*argv)[1], 1);\ngit.c-                  if (envchanged)\ngit.c-                          *envchanged = 1;\ngit.c-                  (*argv)++;\ngit.c-                  (*argc)--;\ngit.c-          } else if (skip_prefix(cmd, \"--attr-source=\", &cmd)) {\ngit.c-                  set_git_attr_source(cmd);\ngit.c:                  setenv(GIT_ATTR_SOURCE_ENVIRONMENT, cmd, 1);\ngit.c-                  if (envchanged)\ngit.c-                          *envchanged = 1;\ngit.c-          } else if (!strcmp(cmd, \"--no-advice\")) {\ngit.c-                  setenv(GIT_ADVICE_ENVIRONMENT, \"0\", 1);\ngit.c-                  if (envchanged)\n\nahah, so it is passed via environment variable to the subprocess.\n\nAnyway, nice catch.\n"},{"id":"512144","messageId":"CABPp-BGQ0pc=AZ0fdXcqDbhMLbm2xBvi71g0mXAVDagz19NkEg@mail.gmail.com","threadId":"62907","inReplyTo":"20250208205709.GH30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-08T21:56:40Z","receivedAt":"2025-02-08T21:56:52Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 8, 2025 at 12:58 PM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> On Fri, Feb 07, 2025 at 04:23:45PM -0800, Elijah Newren wrote:\n[...]\n> > [1] The renormalize option to the merge machinery ensures that new\n> > blobs produced by the merge have normalized content, and avoid\n> > conflicts when the only differences between files are normalization\n> > ones.  This option does not ensure that new trees only reference new\n> > content nor that they only reference normalized content; _any_\n> > pre-existing blobs in the repository are fair game for new trees to\n> > reference.\n>\n> OK.\n>\n> But then, non-normalized content should go through the clean-filter before it\n> is handed over to diff/merge when filtering is active.\n\nNot quite; if the diff/merge doesn't need to look at the content of\nthe file to resolve the merge (i.e. the merge can simply use the\nfile's already known hash as the resolution), then, since that content\nisn't read it shouldn't go through any filters.\n\nWhenever you merge two trees, only the files modified on both sides\nneed to be inspected; the rest can be resolved without looking at\ntheir content.\n\n> > As per the manual: \"renormalize...This runs a virtual\n> > check-out and check-in of all three stages of a file when resolving a\n> > three-way merge...\"  So, the existing behavior of the renormalize\n> > option to rebase/cherry-pick/merge is correct.\n>\n> A virtual check-out and check-in should result in smudge+clean. Running this\n> on smudged content results in smudge+smudge+clean. Which by definition is\n> equivalent to a simple clean. No conflicts shoud happen, then.\n>\n> So the _description_ looks correct. But where do the conflicts coming from?\n>\n> > It may not be what you want\n>\n> I don't see how the description matches actual behaviour\n\nThe description says \"This runs a virtual check-out and check-in of\nall three stages of a file when resolving a three-way merge...\"\n\nSo, when a file needs a three-way merge to be resolved, then the\nvirtual check-out and check-in is done.  When no three-way merge is\nneeded for a file, no virtual check-out and check-in is done.\n\nPerhaps the documentation would be clearer if it read:\n\n           renormalize\n               This runs a virtual check-out and check-in of all three stages\n               of any file which needs a three-way merge.\n\n?\n"},{"id":"512146","messageId":"20250208232651.GJ30202@raven.inka.de","threadId":"62907","inReplyTo":"CABPp-BGwZ029Y8Kfr2kkGiUDZ613kxS81JXzk36V85=77KcYfA@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-08T23:26:51Z","receivedAt":"2025-02-08T23:28:19Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hi Elijah,\n\nOn Sat, Feb 08, 2025 at 01:43:05PM -0800, Elijah Newren wrote:\n\n> Ooh, nice catch.  If folks had an appropriate .gitattributes file in\n> place in older versions of history, they probably wouldn't have gotten\n> into the mess.\n\nWell, you can't assume that paople get it right from the very start. An\nimportant use case of git is fixing errors made in the past, right?\n\nIn my case, I had no choice. I HAD to commit those propritary data files\nas-is, because I had no clue how they are structured and how those hashes are\ncalculated. As time passed, I learned what I need to do to smudge+clean those\nfiles. But at that time a whole bunch of commits were already done.\n\nOn this roadtrip, I had to modify those .gitattributes files in various ways.\n\nThe only variant of those .gitattributes file which will work properly is the\nnewest one. And this is also the variant wich will work for all the olter\ncommits.\n\nSo no, I don't see why using any of the older variants of this .gitattributes\nwould make any sense.\n\n> ahah, so it is passed via environment variable to the subprocess.\n\nI find this to be confusing: the primary call should not need this parameter,\nsince it is invoked from HEAD anyway. Everything else gets it via env-vars.\nI'd assume this variable will also be passed to the commands which are invoked\nby the -x switch?\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512147","messageId":"CALnO6CCUYSM69V4CRiFV=EvQLCC7LCdzuY2gqryj_G_nAWqj-w@mail.gmail.com","threadId":"62907","inReplyTo":"20250208232651.GJ30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-02-09T02:33:05Z","receivedAt":"2025-02-09T02:33:19Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Sat, Feb 8, 2025 at 6:28 PM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> Hi Elijah,\n>\n> On Sat, Feb 08, 2025 at 01:43:05PM -0800, Elijah Newren wrote:\n>\n> > Ooh, nice catch.  If folks had an appropriate .gitattributes file in\n> > place in older versions of history, they probably wouldn't have gotten\n> > into the mess.\n>\n> Well, you can't assume that paople get it right from the very start. An\n> important use case of git is fixing errors made in the past, right?\n>\n> In my case, I had no choice. I HAD to commit those propritary data files\n> as-is, because I had no clue how they are structured and how those hashes are\n> calculated. As time passed, I learned what I need to do to smudge+clean those\n> files. But at that time a whole bunch of commits were already done.\n>\n> On this roadtrip, I had to modify those .gitattributes files in various ways.\n>\n> The only variant of those .gitattributes file which will work properly is the\n> newest one. And this is also the variant wich will work for all the olter\n> commits.\n>\n> So no, I don't see why using any of the older variants of this .gitattributes\n> would make any sense.\n>\n\nThe original question said\n\n> Why on earth would one want a changing filter setting during a rebase?\n> Can anybody outline a use-case for changing filter during operaion? [sic]\n\nBut I'll answer this one—general operations on older history can't use\na newer gitattributes declaration without explicit instruction because\nthey'd have to know from which future to pull. Remember Git can\nbranch, so (even assuming we had a fast way to calculate this, which\nAIUI we don't) from a single commit there can be multiple valid future\ncommits with different gitattributes.\n\nFor the starting point of an operation that eventually invokes other\noperations, where the start clearly uses one gitattributes, it _might_\nbe reasonable to assume that would propagate down to the other\noperations.\n\nBut when subsequent operations logically operate on older history, it\nalso seems reasonable (and unsurprising) to \"do what the repository\nintended at that specific commit.\" Git assumes the latter and provides\na way for you to indicate the former. Perhaps it's worth an explainer\nsomewhere?\n\n-- \nD. Ben Knoble\n"},{"id":"512148","messageId":"CABPp-BEzOWVa5zqOMuUSH5xCJ+CUk6sJnLhE5OdnDiNR0U9jfA@mail.gmail.com","threadId":"62907","inReplyTo":"20250208232651.GJ30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-02-09T07:21:12Z","receivedAt":"2025-02-09T07:21:24Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 8, 2025 at 3:28 PM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> Hi Elijah,\n>\n> On Sat, Feb 08, 2025 at 01:43:05PM -0800, Elijah Newren wrote:\n>\n> > Ooh, nice catch.  If folks had an appropriate .gitattributes file in\n> > place in older versions of history, they probably wouldn't have gotten\n> > into the mess.\n>\n> Well, you can't assume that paople get it right from the very start. An\n> important use case of git is fixing errors made in the past, right?\n[...]\n\nSorry if it sounded like that was passing judgement; that was not what\nI intended.  I've been in a lot of messes too.  I mean, I wrote\ngit-filter-repo because of how many things there were to clean up.  I\nget it, life is messy.  Hindsight is 20/20.  You can't let perfect be\nthe enemy of the good.  You can't prioritize \"everything\", you have to\npick your battles.  Iterative improvement, etc.\n\n> > ahah, so it is passed via environment variable to the subprocess.\n>\n> I find this to be confusing: the primary call should not need this parameter,\n> since it is invoked from HEAD anyway.\n\nNo, the primary call I think would need the parameter too; it changes\nHEAD immediately when it starts rebasing, and continues changing it\nwith each commit it rebases; since it's operating on older versions,\nby default it'd likely pick the .gitattributes from those older\nversions as it goes.\n\n> Everything else gets it via env-vars.\n> I'd assume this variable will also be passed to the commands which are invoked\n> by the -x switch?\n\nYes, I was surprised Phillip's command with --attr-source on the\nouter-level git invocation worked until I discovered that the code\nindeed sets the environment variable (which subprocesses, like those\ncreated by the --exec/-x switch, will inherit).  So, yes, the -x\nswitch stuff seems to inherit that environment variable that the\nprimary call sets in response to that parameter.\n"},{"id":"512156","messageId":"20250209085332.GK30202@raven.inka.de","threadId":"62907","inReplyTo":"CALnO6CCUYSM69V4CRiFV=EvQLCC7LCdzuY2gqryj_G_nAWqj-w@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-09T08:53:32Z","receivedAt":"2025-02-09T08:54:07Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Sat, Feb 08, 2025 at 09:33:05PM -0500, D. Ben Knoble wrote:\n\n> > So no, I don't see why using any of the older variants of this .gitattributes\n> > would make any sense.\n> \n> The original question said\n> \n> > Why on earth would one want a changing filter setting during a rebase?\n> > Can anybody outline a use-case for changing filter during operaion? [sic]\n> \n> But I'll answer this one—general operations on older history can't use\n> a newer gitattributes declaration without explicit instruction because\n> they'd have to know from which future to pull. Remember Git can\n> branch, so (even assuming we had a fast way to calculate this, which\n> AIUI we don't) from a single commit there can be multiple valid future\n> commits with different gitattributes.\n\nYes, this is the way .gitattributes work. And to be honest, I always found it\nstrange that this setting travels along the history.\n\nBut there is also ~/.giconfig, which has the drawback that in won't travel\nwith the repo.\n\nIMHO, it would make more sense to have some sort of global storage which\ntravels along with the repo.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512157","messageId":"20250209085756.GL30202@raven.inka.de","threadId":"62907","inReplyTo":"CABPp-BEzOWVa5zqOMuUSH5xCJ+CUk6sJnLhE5OdnDiNR0U9jfA@mail.gmail.com","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-09T08:57:56Z","receivedAt":"2025-02-09T08:58:08Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Sat, Feb 08, 2025 at 11:21:12PM -0800, Elijah Newren wrote:\n> On Sat, Feb 8, 2025 at 3:28 PM Josef Wolf <jw@raven.inka.de> wrote:\n\n> > > ahah, so it is passed via environment variable to the subprocess.\n> >\n> > I find this to be confusing: the primary call should not need this parameter,\n> > since it is invoked from HEAD anyway.\n> \n> No, the primary call I think would need the parameter too; it changes\n> HEAD immediately when it starts rebasing, and continues changing it\n> with each commit it rebases; since it's operating on older versions,\n> by default it'd likely pick the .gitattributes from those older\n> versions as it goes.\n\nOK. I see...\n\n> > Everything else gets it via env-vars.\n> > I'd assume this variable will also be passed to the commands which are invoked\n> > by the -x switch?\n> \n> Yes, I was surprised Phillip's command with --attr-source on the\n> outer-level git invocation worked until I discovered that the code\n> indeed sets the environment variable (which subprocesses, like those\n> created by the --exec/-x switch, will inherit).  So, yes, the -x\n> switch stuff seems to inherit that environment variable that the\n> primary call sets in response to that parameter.\n\nUmm... OK... This means that specifying --attr-source to the commands for the\n-x switch is wrong, since they have a different HEAD?\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512158","messageId":"20250209092514.GM30202@raven.inka.de","threadId":"62907","inReplyTo":"20250208205709.GH30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-09T09:25:14Z","receivedAt":"2025-02-09T09:26:08Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Sat, Feb 08, 2025 at 09:57:09PM +0100, Josef Wolf wrote:\n\nI just stumbled over another wirdeness:\n\n> Oh, have I mentioned that I am not only about line endings? Yes, I mentioned\n> it in the very first mail. In addition to line endings, I am also about XML\n> files from a proprietary application which reorders the XML-elements into a\n> random order every time it ist run. So the clean-filter needs to sort the\n> XML elements into some \"canonical\" order.\n\nThis application stores the bulk of the data as text files and XML files with\nCRLF. But there are also some binary files. So I set gitattributes like this:\n\n   # Catch bulk as text=crlf, rely on git to detect binary\n   */*     text=auto eol=crlf\n   #\n   # those are known to be text=crlf\n   */B     text eol=crlf\n   */P-*   text eol=crlf\n   #\n   # smudge-clean filter\n   */B     filter=etsfile\n   */P-*   filter=etsfile\n   #\n   # files I dont't want to touch (mostly binaries)\n   */*.dll       -filter -text\n   */*.ver       -filter -text\n   */*.lang      -filter -text\n   */*.store     -filter -text\n   */*.ets5hash  -filter -text\n\nBut \"git ls-files --eol\" gives me this:\n\n     i/lf    w/lf    attr/text eol=crlf      gt8/ETS/Projekte/P-0113/B\n\nWhy is git ignoring my explicit CRLF setting?\n\nThis is on linux and on Windows+MSYS2. I don't have $GIT_DIR/info/attributes\nand ~/.gitconfig also doesn't specify any line ending things\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512160","messageId":"20250209111406.GA12069@tb-raspi4","threadId":"62907","inReplyTo":"20250209092514.GM30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2025-02-09T11:14:06Z","receivedAt":"2025-02-09T11:14:09Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Sun, Feb 09, 2025 at 10:25:14AM +0100, Josef Wolf wrote:\n> On Sat, Feb 08, 2025 at 09:57:09PM +0100, Josef Wolf wrote:\n>\n> I just stumbled over another wirdeness:\n>\n> > Oh, have I mentioned that I am not only about line endings? Yes, I mentioned\n> > it in the very first mail. In addition to line endings, I am also about XML\n> > files from a proprietary application which reorders the XML-elements into a\n> > random order every time it ist run. So the clean-filter needs to sort the\n> > XML elements into some \"canonical\" order.\n>\n> This application stores the bulk of the data as text files and XML files with\n> CRLF. But there are also some binary files. So I set gitattributes like this:\n>\n>    # Catch bulk as text=crlf, rely on git to detect binary\n>    */*     text=auto eol=crlf\n\nThis looks a little bit strange to me.\nWhat happens if you replace \"*/*\" with \"*\" like this.\n*     text=auto eol=crlf\n\n\n>    #\n>    # those are known to be text=crlf\n>    */B     text eol=crlf\n>    */P-*   text eol=crlf\nSame here. What is B ? Is it a directory ?\n\n>    #\n>    # smudge-clean filter\n>    */B     filter=etsfile\n>    */P-*   filter=etsfile\n>    #\n>    # files I dont't want to touch (mostly binaries)\n>    */*.dll       -filter -text\n>    */*.ver       -filter -text\n>    */*.lang      -filter -text\n>    */*.store     -filter -text\n>    */*.ets5hash  -filter -text\n*.dll       -filter -text\n(and the same for everything else)\n\n>\n> But \"git ls-files --eol\" gives me this:\n>\n>      i/lf    w/lf    attr/text eol=crlf      gt8/ETS/Projekte/P-0113/B\n>\n> Why is git ignoring my explicit CRLF setting?\n>\n> This is on linux and on Windows+MSYS2. I don't have $GIT_DIR/info/attributes\n> and ~/.gitconfig also doesn't specify any line ending things\n>\n> --\n> Josef Wolf\n> jw@raven.inka.de\n>\n"},{"id":"512163","messageId":"20250209150924.GN30202@raven.inka.de","threadId":"62907","inReplyTo":"20250209111406.GA12069@tb-raspi4","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-09T15:09:24Z","receivedAt":"2025-02-09T15:10:07Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hello Torsten,\n\nOn Sun, Feb 09, 2025 at 12:14:06PM +0100, Torsten Bögershausen wrote:\n> On Sun, Feb 09, 2025 at 10:25:14AM +0100, Josef Wolf wrote:\n\n> > This application stores the bulk of the data as text files and XML files with\n> > CRLF. But there are also some binary files. So I set gitattributes like this:\n> >\n> >    # Catch bulk as text=crlf, rely on git to detect binary\n> >    */*     text=auto eol=crlf\n> > \n> This looks a little bit strange to me.\n\nThis should match all files in directories one level deeper than the directory\nwhere .gitattributes live:\n\n   If there is a separator at the beginning or middle (or both) of the pattern,\n   then the pattern is relative to the directory level of the particular\n   .gitignore file itself.\n\n> What happens if you replace \"*/*\" with \"*\" like this.\n> *     text=auto eol=crlf\n\nSame result, but when I commit .gitattributes, I get a warning that git will\ndo lf->crl conversion. But even after commit, no conversion is done and\ngit-ls-files still shows:\n\n   i/lf    w/lf    attr/text=auto eol=crlf gt8/ETS/Projekte/.gitignore\n\nOnly after removal followed by \"git reset --hard\", I get:\n\n   i/lf    w/lfcr  attr/text=auto eol=crlf gt8/ETS/Projekte/.gitignore\n\n> >    #\n> >    # those are known to be text=crlf\n> >    */B     text eol=crlf\n> >    */P-*   text eol=crlf\n> Same here. What is B ? Is it a directory ?\n\nNo. It is one of the XML files I want to smudge+clean\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512164","messageId":"20250209175450.GO30202@raven.inka.de","threadId":"62907","inReplyTo":"20250209150924.GN30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-09T17:54:50Z","receivedAt":"2025-02-09T17:56:09Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Uh! It starts getting real wired.\n\nAfter one more change to .gitattributes, one of the files marked as binary\nchecks out as an EMPTY file and I can't find any git command to fix the\nsituation:\n\n  $ cat .gitattributes\n  # Most files in ETS ProjectStore are XML with CRLF\n  #\n  * text=auto eol=crlf\n  \n  .gitignore     text\n  .gitattributes text\n  \n  # Binary files\n  #\n  *.dat       -filter -text\n  *.dll       -filter -text\n  *.ver       -filter -text\n  *.lang      -filter -text\n  *.store     -filter -text  # <--- this is the problematic file\n  *.ets5hash  -filter -text\n  \n  # Smudge/clean filter\n  #\n  */B     filter=etsfile\n  */D     filter=etsfile\n  */G     filter=etsfile\n  */I     filter=etsfile\n  */P     filter=etsfile\n  */U     filter=etsfile\n  */P-*   filter=etsfile\n\n  $ git diff\n  diff --git a/gt8/ETS/Projekte/P-0113/P-0113.store b/gt8/ETS/Projekte/P-0113/P-0113.store\n  index c33a5239..e69de29b 100755\n  --- a/gt8/ETS/Projekte/P-0113/P-0113.store\n  +++ b/gt8/ETS/Projekte/P-0113/P-0113.store\n  @@ -1 +0,0 @@\n  -4TamRjepVNV8F+bC4nBcBwXIymvb2IQdu0qEuMSB0o0=\n  \\ No newline at end of file\n  $ git reset --hard\n  HEAD is now at 6fba03d9 Fix .gitattributes again\n  $ git diff\n  diff --git a/gt8/ETS/Projekte/P-0113/P-0113.store b/gt8/ETS/Projekte/P-0113/P-0113.store\n  index c33a5239..e69de29b 100755\n  --- a/gt8/ETS/Projekte/P-0113/P-0113.store\n  +++ b/gt8/ETS/Projekte/P-0113/P-0113.store\n  @@ -1 +0,0 @@\n  -4TamRjepVNV8F+bC4nBcBwXIymvb2IQdu0qEuMSB0o0=\n  \\ No newline at end of file\n  $ rm -rf P-0113/ ; git checkout P-0113/\n  Updated 382 paths from the index\n  $ git diff\n  diff --git a/gt8/ETS/Projekte/P-0113/P-0113.store b/gt8/ETS/Projekte/P-0113/P-0113.store\n  index c33a5239..e69de29b 100755\n  --- a/gt8/ETS/Projekte/P-0113/P-0113.store\n  +++ b/gt8/ETS/Projekte/P-0113/P-0113.store\n  @@ -1 +0,0 @@\n  -4TamRjepVNV8F+bC4nBcBwXIymvb2IQdu0qEuMSB0o0=\n  \\ No newline at end of file\n  $ git ls-files --eol |grep P-0113.store\n  i/none  w/none  attr/-text              P-0113/P-0113.store\n  i/lf    w/crlf  attr/text=auto eol=crlf P-0113/storeVersion\n  $\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512165","messageId":"20250209180150.GP30202@raven.inka.de","threadId":"62907","inReplyTo":"20250209175450.GO30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-09T18:01:50Z","receivedAt":"2025-02-09T18:02:18Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Sun, Feb 09, 2025 at 06:54:50PM +0100, Josef Wolf wrote:\n\nUpps. That's probably an ordering problem:\n\n>   *.store     -filter -text  # <--- this is the problematic file\n> [ ... ]\n>   */P-*   filter=etsfile\n\nLater line overrides first line.\n\nPlease ignore my last mail.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512187","messageId":"CALnO6CBh0UDSeR4Q1VfU7vdSvHFYuO=j_rijVpAE-YH9V=Cqew@mail.gmail.com","threadId":"62907","inReplyTo":"20250209085756.GL30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-02-10T17:51:23Z","receivedAt":"2025-02-10T17:51:37Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Sun, Feb 9, 2025 at 3:58 AM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> On Sat, Feb 08, 2025 at 11:21:12PM -0800, Elijah Newren wrote:\n> > Yes, I was surprised Phillip's command with --attr-source on the\n> > outer-level git invocation worked until I discovered that the code\n> > indeed sets the environment variable (which subprocesses, like those\n> > created by the --exec/-x switch, will inherit).  So, yes, the -x\n> > switch stuff seems to inherit that environment variable that the\n> > primary call sets in response to that parameter.\n>\n> Umm... OK... This means that specifying --attr-source to the commands for the\n> -x switch is wrong, since they have a different HEAD?\n\nNot quite: the command actually did\n\n> git --attr-source=$(git rev-parse HEAD) […]\n\nSo the subprocesses will see the attributes source as a full-length\ncommit hash, not \"HEAD\"\n\n-- \nD. Ben Knoble\n"},{"id":"512268","messageId":"20250211235707.GQ30202@raven.inka.de","threadId":"62907","inReplyTo":"20250205214726.GA30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter, again","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-11T23:57:07Z","receivedAt":"2025-02-11T23:58:24Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Still struggling with my filter problem.\n\nHere is what I do:\n\n- Set up a clean filter which enforces CRLF (yes, for this specific use\n  case I want CRLF even on linux)\n\n- Smudge filter does not modify the file at all\n\n- Set up git to fail when filter fails, so I can double-check that the\n  filter is actually runnning:\n\n   $ grep -A3 filter..etsfile ~/.gitconfig\n   [filter \"etsfile\"]\n      required = true\n      clean = ets-utils -c\n      smudge = ets-utils -s %f\n\n- Specify file as non-text and install the filter:\n\n    $ grep etsfile .gitattributes\n    */P -text filter=etsfile\n    $ git commit .gitattributes\n\n- Check that git gets attributes as I want them:\n\n    $ git --attr-source=$(git rev-parse HEAD) check-attr -a P-0113/P\n    P-0113/P: text: unset\n    P-0113/P: filter: etsfile\n    $ git ls-files --eol P-0113/P\n    i/lf    w/      attr/-text              P-0113/P\n\n- Create helper for renormalization\n\n    $ cat renormalization-helper\n    #! /bin/sh -e\n    git add --renormalize .\n    git diff --quiet --cached || \\\n        git commit --amend --no-edit\n    \n- Run the renormalization for the linear history:\n\n    $ git --attr-source=$(git rev-parse HEAD) \\\n         rebase --root -X renormalize \\\n         -x $(dirname $0)/renormalize-helper\n\nSo at this point, I'd expect the falie to have CRLF line endings. But it\ndoesn't, so I do:\n\n    $ rm -rf P-0113\n    git checkout  --attr-source=$(git rev-parse HEAD) P-0113\n\nStill no CRLF, so I look at what is stored by git:\n\n    $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U\n\nAgain, no CRLF.\n\nSo I check all revisions in the history. Resut: no revision has CRLF.\n\nSo the renormalization process does not work for me at all.\n\nAny ideas?\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512293","messageId":"20250212061236.GA990@tb-raspi4","threadId":"62907","inReplyTo":"20250211235707.GQ30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter, again","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2025-02-12T06:12:36Z","receivedAt":"2025-02-12T06:17:50Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Feb 12, 2025 at 12:57:07AM +0100, Josef Wolf wrote:\n> Still struggling with my filter problem.\n>\n> Here is what I do:\n>\n> - Set up a clean filter which enforces CRLF (yes, for this specific use\n>   case I want CRLF even on linux)\n\nIn general, clean filters do their work when 'git add' or 'git commit file'\nis run.\nDoes the filter do the CRLF conversion ?\nOr is it done in .gitattributes ?\n\n>\n> - Smudge filter does not modify the file at all\n>\n> - Set up git to fail when filter fails, so I can double-check that the\n>   filter is actually runnning:\n>\n>    $ grep -A3 filter..etsfile ~/.gitconfig\n>    [filter \"etsfile\"]\n>       required = true\n>       clean = ets-utils -c\n>       smudge = ets-utils -s %f\n>\n> - Specify file as non-text and install the filter:\n>\n>     $ grep etsfile .gitattributes\n>     */P -text filter=etsfile\n>     $ git commit .gitattributes\n>\n> - Check that git gets attributes as I want them:\n>\n>     $ git --attr-source=$(git rev-parse HEAD) check-attr -a P-0113/P\n>     P-0113/P: text: unset\n>     P-0113/P: filter: etsfile\n>     $ git ls-files --eol P-0113/P\n>     i/lf    w/      attr/-text              P-0113/P\n>\n> - Create helper for renormalization\n>\n>     $ cat renormalization-helper\n>     #! /bin/sh -e\n>     git add --renormalize .\n>     git diff --quiet --cached || \\\n>         git commit --amend --no-edit\n>\n> - Run the renormalization for the linear history:\n>\n>     $ git --attr-source=$(git rev-parse HEAD) \\\n>          rebase --root -X renormalize \\\n>          -x $(dirname $0)/renormalize-helper\n\nThat will change the index, the repo, but not the working tree on disk,\nright ?\n\n>\n> So at this point, I'd expect the falie to have CRLF line endings. But it\n> doesn't, so I do:\n>\n>     $ rm -rf P-0113\n>     git checkout  --attr-source=$(git rev-parse HEAD) P-0113\n>\n> Still no CRLF, so I look at what is stored by git:\n>\n>     $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U\n>\n> Again, no CRLF.\n\nJust to make sure:\nYou want to see the CRLF in the files on disk ?\nDo you have a valid .gitattributes file on disk now ?\nIf yes, what does 'git ls-files --eol P-0113' say ?\nWhat does 'git status' say ?\n\n>\n> So I check all revisions in the history. Resut: no revision has CRLF.\n> So the renormalization process does not work for me at all.\n\nIn general, renormalization is about the content inside the repo.\nIf a filter is applied, or .gitattributes are changed, the files\non disk are not updated automatically.\n'mv -f P-0113 /tmp && git checkout P-0113' may be needed.\n\n>\n> Any ideas?\n\nYes. The best thing to do (tm) would be to create a dummy repo,\ndo all all the operations from scratch and post the stuff here.\nIn other words, write a shell script that creates an empty repo,\nfills it with content, and does all the operations.\nThat would enable people to reproduce it and look what is going on.\nHope that make sense.\n\n>\n> --\n> Josef Wolf\n> jw@raven.inka.de\n>\n"},{"id":"512296","messageId":"20250212081842.GR30202@raven.inka.de","threadId":"62907","inReplyTo":"20250212061236.GA990@tb-raspi4","subject":"Re: renormalize histroy with smudge/clean-filter, again","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-12T08:18:42Z","receivedAt":"2025-02-12T08:20:06Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hi Torsten,\n\nOn Wed, Feb 12, 2025 at 07:12:36AM +0100, Torsten Bögershausen wrote:\n\n> > - Set up a clean filter which enforces CRLF (yes, for this specific use\n> >   case I want CRLF even on linux)\n> \n> In general, clean filters do their work when 'git add' or 'git commit file'\n> is run.\n\nYes. This is done in the renormalise-helper shell script, which I included\ninto my description below:\n\n> >     $ cat renormalization-helper\n> >     #! /bin/sh -e\n> >     git add --renormalize .\n> >     git diff --quiet --cached || \\\n> >         git commit --amend --no-edit\n\n> Does the filter do the CRLF conversion ?\n\nAs I wrote above: yes, the clean filter enforces CRLF\n\n> Or is it done in .gitattributes ?\n\nNo. .gitattributes states that git should not modify the file since I have set\nit -text, as I wrote:\n\n> >     */P -text filter=etsfile\n\n\n> > - Run the renormalization for the linear history:\n> >\n> >     $ git --attr-source=$(git rev-parse HEAD) \\\n> >          rebase --root -X renormalize \\\n> >          -x $(dirname $0)/renormalize-helper\n> \n> That will change the index, the repo, but not the working tree on disk,\n> right ?\n\n\"git reset --hard\" or even \"rm -rf P-0113; git checkout P-0113\", also do not\nbring the CRLF into the file, see below.\n\n> > So at this point, I'd expect the falie to have CRLF line endings. But it\n> > doesn't, so I do:\n> >\n> >     $ rm -rf P-0113\n> >     git checkout  --attr-source=$(git rev-parse HEAD) P-0113\n> >\n> > Still no CRLF, so I look at what is stored by git:\n> >\n> >     $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U\n> >\n> > Again, no CRLF.\n> \n> Just to make sure:\n> You want to see the CRLF in the files on disk ?\n\nIn the first place I want to see them in the repo. And a fresh checkout should\nbring them into the files on disk, since -text is in effect.\n\n> Do you have a valid .gitattributes file on disk now ?\n\ngit recognizes my setting -text and filter=etsfile, as I wrote:\n\n> >     $ git --attr-source=$(git rev-parse HEAD) check-attr -a P-0113/P\n> >     P-0113/P: text: unset\n> >     P-0113/P: filter: etsfile\n\n> If yes, what does 'git ls-files --eol P-0113' say ?\n\nAs I wrote above:\n\n> >     $ git ls-files --eol P-0113/P\n> >     i/lf    w/      attr/-text              P-0113/P\n\n> What does 'git status' say ?\n\nNothing, since\n\n  git add --renormalize . && git commit --amend --no-edit\n\nhave been done by the helper script on every commit of the history\n\n> > So I check all revisions in the history. Resut: no revision has CRLF.\n> > So the renormalization process does not work for me at all.\n> \n> In general, renormalization is about the content inside the repo.\n> If a filter is applied, or .gitattributes are changed, the files\n> on disk are not updated automatically.\n\nThis is why I checkd the contents which are stored in the repo:\n\n> >     $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U\n\n> 'mv -f P-0113 /tmp && git checkout P-0113' may be needed.\n\nWell, I did this instead:\n\n> >     $ rm -rf P-0113\n> >     git checkout  --attr-source=$(git rev-parse HEAD) P-0113\n\n> Yes. The best thing to do (tm) would be to create a dummy repo,\n> do all all the operations from scratch and post the stuff here.\n> In other words, write a shell script that creates an empty repo,\n> fills it with content, and does all the operations.\n> That would enable people to reproduce it and look what is going on.\n> Hope that make sense.\n\nWell, if I _knew_ what triggers the problem, I could create such a script.\n\nAs long as I can not figure what triggers the problem, I have to dig into\ninternals of this old repo with long-running history.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512361","messageId":"20250213113614.GS30202@raven.inka.de","threadId":"62907","inReplyTo":"20250211235707.GQ30202@raven.inka.de","subject":"Collisions while cloning (was: Re: renormalize histroy with smudge/clean-filter, again)","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-13T11:36:14Z","receivedAt":"2025-02-13T11:38:24Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hi folks,\n\nwhile investigating/recovering my problems with renormalizing with\nclean/smudge filtering, I stumbled on collisions while creating a fresh clone\nof the repo from the server:\n\n   $ LANG= git clone ssh://gitrepos@my.server/repo\n   smart-home-ets5hashes-removed\n   Cloning into 'smart-home-ets5hashes-removed'...\n   remote: Enumerating objects: 7499, done.\n   remote: Counting objects: 100% (7499/7499), done.\n   remote: Compressing objects: 100% (3263/3263), done.\n   remote: Total 7499 (delta 3955), reused 7109 (delta 3594), pack-reused 0\n   Receiving objects: 100% (7499/7499), 140.12 MiB | 10.54 MiB/s, done.\n   Resolving deltas: 100% (3955/3955), done.\n   Updating files: 100% (1423/1423), done.\n   warning: the following paths have collided (e.g. case-sensitive paths\n   on a case-insensitive filesystem) and only one from the same\n   colliding group is in the working tree:\n   \n  'Projects/P-0113/B.ets5hash'\n  [more files deleted]\n\nThis is on linux, so the FS is _not_ case-insensitive.\n\nThe list of files given here is almost identical to the list of files which\nalways give me collisions during renormalization process.\n\nHere is an explanation of how and why those files ended up in the repo and a\nhypothesis of why they might be in conflicting state.\n\nThose files contain hash values of the real data files for a proprietary\napplication and are re-calculated on every invocation of the application. The\napplication won't even start up if those hashes don't match. And it won't tell\nwhy it won't start, it just says \"Corrupt data\".\n\nAt the time this repository started, I had no knowledge how the hashes of\nthose files are calculated, so I had to commit them along with the associated\ndata files to keep the application happy. This results in conflicts with many\ngit operastions, of course.\n\nThen I learned how those files can be re-calculated and wrote a smudge-filter\nto keep them in sync with the data files.\n\nSince I was now able to recreate those files, I put them into .gitignore and\ninstalled the smudge-filter to recalculate them. But I left the files in the\nrepo as a fallback, just to be sure. And I kept committing them every now and\nthen whenever git showed differences, although they already were in\n.gitignore.\n\nSo I guess those collisions might come from committing the ignored\nfiles. Unfortunately, I could not reproduce this effect on a fresh repo, yet.\n\nAnd the next question is: why do those conflicts cause the renormalization\nprocess to completely fail, even when the conflicts are resolved during the\nrenormalization rebase? This, I also could not reproduced on a fresh repo.\n\n\nOn Wed, Feb 12, 2025 at 12:57:07AM +0100, Josef Wolf wrote:\n> Still struggling with my filter problem.\n> \n> Here is what I do:\n> \n> - Set up a clean filter which enforces CRLF (yes, for this specific use\n>   case I want CRLF even on linux)\n> \n> - Smudge filter does not modify the file at all\n> \n> - Set up git to fail when filter fails, so I can double-check that the\n>   filter is actually runnning:\n> \n>    $ grep -A3 filter..etsfile ~/.gitconfig\n>    [filter \"etsfile\"]\n>       required = true\n>       clean = ets-utils -c\n>       smudge = ets-utils -s %f\n> \n> - Specify file as non-text and install the filter:\n> \n>     $ grep etsfile .gitattributes\n>     */P -text filter=etsfile\n>     $ git commit .gitattributes\n> \n> - Check that git gets attributes as I want them:\n> \n>     $ git --attr-source=$(git rev-parse HEAD) check-attr -a P-0113/P\n>     P-0113/P: text: unset\n>     P-0113/P: filter: etsfile\n>     $ git ls-files --eol P-0113/P\n>     i/lf    w/      attr/-text              P-0113/P\n> \n> - Create helper for renormalization\n> \n>     $ cat renormalization-helper\n>     #! /bin/sh -e\n>     git add --renormalize .\n>     git diff --quiet --cached || \\\n>         git commit --amend --no-edit\n>     \n> - Run the renormalization for the linear history:\n> \n>     $ git --attr-source=$(git rev-parse HEAD) \\\n>          rebase --root -X renormalize \\\n>          -x $(dirname $0)/renormalize-helper\n> \n> So at this point, I'd expect the falie to have CRLF line endings. But it\n> doesn't, so I do:\n> \n>     $ rm -rf P-0113\n>     git checkout  --attr-source=$(git rev-parse HEAD) P-0113\n> \n> Still no CRLF, so I look at what is stored by git:\n> \n>     $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U\n> \n> Again, no CRLF.\n> \n> So I check all revisions in the history. Resut: no revision has CRLF.\n> \n> So the renormalization process does not work for me at all.\n> \n> Any ideas?\n> \n> -- \n> Josef Wolf\n> jw@raven.inka.de\n> \n> \n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512366","messageId":"20250213164035.GA24612@tb-raspi4","threadId":"62907","inReplyTo":"20250213113614.GS30202@raven.inka.de","subject":"Re: Collisions while cloning (was: Re: renormalize histroy with smudge/clean-filter, again)","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2025-02-13T16:40:35Z","receivedAt":"2025-02-13T16:40:37Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Thu, Feb 13, 2025 at 12:36:14PM +0100, Josef Wolf wrote:\n> Hi folks,\n>\n> while investigating/recovering my problems with renormalizing with\n> clean/smudge filtering, I stumbled on collisions while creating a fresh clone\n> of the repo from the server:\n>\n>    $ LANG= git clone ssh://gitrepos@my.server/repo\n>    smart-home-ets5hashes-removed\n>    Cloning into 'smart-home-ets5hashes-removed'...\n>    remote: Enumerating objects: 7499, done.\n>    remote: Counting objects: 100% (7499/7499), done.\n>    remote: Compressing objects: 100% (3263/3263), done.\n>    remote: Total 7499 (delta 3955), reused 7109 (delta 3594), pack-reused 0\n>    Receiving objects: 100% (7499/7499), 140.12 MiB | 10.54 MiB/s, done.\n>    Resolving deltas: 100% (3955/3955), done.\n>    Updating files: 100% (1423/1423), done.\n>    warning: the following paths have collided (e.g. case-sensitive paths\n>    on a case-insensitive filesystem) and only one from the same\n>    colliding group is in the working tree:\n>\n>   'Projects/P-0113/B.ets5hash'\n>   [more files deleted]\n>\n> This is on linux, so the FS is _not_ case-insensitive.\n>\n\nThat sounds fishy (tm)\nDoes 'git ls-files' give any hints ?\n"},{"id":"512435","messageId":"20250214200313.GT30202@raven.inka.de","threadId":"62907","inReplyTo":"20250211235707.GQ30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter, again","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-14T20:03:13Z","receivedAt":"2025-02-14T20:04:14Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Since none of the methods using plain git worked, my next try was to reach out\nto git-filter-repo:\n\nAgain, using my renormalize-helper script:\n\n  $ cat renormalize-helper\n  #! /bin/sh -e\n  \n  git add --renormalize .\n  git diff --quiet --cached || \\\n      git commit --amend --no-edit\n\nSo I go with git-filter-repo:\n \n   $ git clone ssh://gitrepos@my.server/repo fresh-clone\n   $ cd fresh-clone\n\n   $ git-filter-repo \\\n       --prune-empty always \\\n       --invert-paths --use-base-name \\\n       --path-regex '\\.ets5hash$'\n\n   $ for branch in branch-1 branch-2 branch-3 ; do\n        git checkout -b $branch-renormalized $branch\n   \n        git add --renormalize .\n        git diff --quiet --cached || \\\n             git commit -m\"Renormalize HEAD\"\n   \n        git rebase \\\n             --root -X renormalize \\\n             -x $renormalize_helper\n     done\n\nThis went without problem and contents looked fine, so I really thought I got\nit finally.\n\nBut then I tried to move .gitattributes to the very beginnig of history:\n\n   $ git rebase -i --root\n\nAGAIN conflicts due to line ending errors. Adding\n'--attr-source=$(git rev-parse HEAD)' and '-x renormalize-helper' did not\nhelp beside moving the conflicts to another location.\n\nThus, although the renormalization process finished successfully, there are\n_still_ commits with unclean content in the repository.\n\nI REALLY REALLY REALLY think there should be an option\n\n--always-apply-clean-filter-to-all-content-before-feeding-to-merge-or-diff\n\nor something!\n\n\n\nOn Wed, Feb 12, 2025 at 12:57:07AM +0100, Josef Wolf wrote:\n> Still struggling with my filter problem.\n> \n> Here is what I do:\n> \n> - Set up a clean filter which enforces CRLF (yes, for this specific use\n>   case I want CRLF even on linux)\n> \n> - Smudge filter does not modify the file at all\n> \n> - Set up git to fail when filter fails, so I can double-check that the\n>   filter is actually runnning:\n> \n>    $ grep -A3 filter..etsfile ~/.gitconfig\n>    [filter \"etsfile\"]\n>       required = true\n>       clean = ets-utils -c\n>       smudge = ets-utils -s %f\n> \n> - Specify file as non-text and install the filter:\n> \n>     $ grep etsfile .gitattributes\n>     */P -text filter=etsfile\n>     $ git commit .gitattributes\n> \n> - Check that git gets attributes as I want them:\n> \n>     $ git --attr-source=$(git rev-parse HEAD) check-attr -a P-0113/P\n>     P-0113/P: text: unset\n>     P-0113/P: filter: etsfile\n>     $ git ls-files --eol P-0113/P\n>     i/lf    w/      attr/-text              P-0113/P\n> \n> - Create helper for renormalization\n> \n>     $ cat renormalization-helper\n>     #! /bin/sh -e\n>     git add --renormalize .\n>     git diff --quiet --cached || \\\n>         git commit --amend --no-edit\n>     \n> - Run the renormalization for the linear history:\n> \n>     $ git --attr-source=$(git rev-parse HEAD) \\\n>          rebase --root -X renormalize \\\n>          -x $(dirname $0)/renormalize-helper\n> \n> So at this point, I'd expect the falie to have CRLF line endings. But it\n> doesn't, so I do:\n> \n>     $ rm -rf P-0113\n>     git checkout  --attr-source=$(git rev-parse HEAD) P-0113\n> \n> Still no CRLF, so I look at what is stored by git:\n> \n>     $ git --attr-source=$(git rev-parse HEAD) show 873a9b:P-0113/P |less -U\n> \n> Again, no CRLF.\n> \n> So I check all revisions in the history. Resut: no revision has CRLF.\n> \n> So the renormalization process does not work for me at all.\n> \n> Any ideas?\n> \n> -- \n> Josef Wolf\n> jw@raven.inka.de\n> \n> \n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"512436","messageId":"Z6-lyJNvXHhrVXhg@tapette.crustytoothpaste.net","threadId":"62907","inReplyTo":"20250211235707.GQ30202@raven.inka.de","subject":"Re: renormalize histroy with smudge/clean-filter, again","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-02-14T20:21:28Z","receivedAt":"2025-02-14T20:21:37Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-02-11 at 23:57:07, Josef Wolf wrote:\n> Still struggling with my filter problem.\n> \n> Here is what I do:\n> \n> - Set up a clean filter which enforces CRLF (yes, for this specific use\n>   case I want CRLF even on linux)\n\nIs there a reason you can't use `eol=crlf` instead of a smudge/clean\nfilter?  That looks like this in the Git repo:\n\n  *.bat text eol=crlf\n\nThat might be an easier way to accomplish what you want and it will\nalways result in CRLF in the working tree, regardless of operating\nsystem, even though in the repository it will still use LF[0].\n\nNote that if you need a specific encoding, there's also\n`working-tree-encoding` as well.\n\n[0] Okay, technically someone can override it with\n`.git/info/attributes`, but if they do that and it doesn't work, that's\ntheir own fault.  We don't worry about that case in this project.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"512438","messageId":"20250214205513.GU30202@raven.inka.de","threadId":"62907","inReplyTo":"Z6-lyJNvXHhrVXhg@tapette.crustytoothpaste.net","subject":"Re: renormalize histroy with smudge/clean-filter, again","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2025-02-14T20:55:13Z","receivedAt":"2025-02-14T20:56:08Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"On Fri, Feb 14, 2025 at 08:21:28PM +0000, brian m. carlson wrote:\n> On 2025-02-11 at 23:57:07, Josef Wolf wrote:\n> > Still struggling with my filter problem.\n> > \n> > Here is what I do:\n> > \n> > - Set up a clean filter which enforces CRLF (yes, for this specific use\n> >   case I want CRLF even on linux)\n> \n> Is there a reason you can't use `eol=crlf` instead of a smudge/clean\n> filter?  That looks like this in the Git repo:\n\nYes. Most of the data files of this (proprietary) application are XML files\nusing mostly CRLF, but there is also LF ancoded content. Like this:\n\n[ ... ]\n <foo>^M\n  <bar>\n  fonly LF in contents of bar\n  </bar>^M\n </foo>^M\n\nIn addition, it randomly shuffles the XML elements at every startup, even if\nno changes are done. To prevent conflocts from this, I need to sort the XML\nelements into a canonical ordering in the clean filter.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"}]}