threads / discuss / 53260

git-filter-repo For Windows (possibly a request for a bash for Windows primer)

Subject: git-filter-repo For Windows (possibly a request for a bash for Windows primer)

## tl;dr

6 messages between Apr 20, 2020 and Apr 25, 2020.

replies: 5people: 4as markdown or json

Kerry, Richard· Apr 20, 2020, 14:03 UTC · lore

This follows my earlier query for which the answer was "use git-filter-repo". I've cloned git-filter-repo from GitHub, and I've installed the latest Python (version 3.8). I'm working on Windows.  Windows 10 if it makes any difference, so git is Git For Windows. The first instruction for git-filter-repo is to copy it into the "--exec-path" of git.  That fails, saying Access is Denied.  So leaving aside the option to change the permissions I've followed the second half of the sentence and added my git-filter-repo location to my Path (or PATH if it's in bash).  Now running "get filter-repo" gives: /usr/bin/env: 'python3.exe': No such file or directory

My python executable is called python3.exe.  There is a note in the git-filter-repo project about certain Windows installations where it isn't called python3, but that now raises in my mind the question of whether I need to tweak git-filter-repo to look for "python3.exe" rather than just "python3".  Or can I take it that the Windows version of bash understands that it needs to add ".exe" automatically?
Assuming that is handled automatically, what do I now need to do to satisfy the error message?  Do I need to put something in bash-for-Windows' /usr/bin/env directory?  Or is there something else entirely that I should be doing?
And is it OK to ask about this on this list, or should I be going to a git-filter-branch resource?  Or for Git For Windows?

Regards, Richard.

Richard Kerry
BNCS Engineer, SI SOL Telco & Media Vertical Practice
M: +44 (0)7812 325518
2nd Floor, MidCity Place, 71 High Holborn, London, WC1V 6EA
https://webmail.siemens-it-solutions.com/owa/redir.aspx?C=9fb20d019e3e4cb99344d708709a3177&URL=mailto%3arichard.kerry%40atos.net
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Elijah Newren· Apr 20, 2020, 16:53 UTC · re: Kerry, Richard · lore

Re: git-filter-repo For Windows (possibly a request for a bash for Windows primer)

On Mon, Apr 20, 2020 at 8:11 AM Kerry, Richard <richard.kerry@atos.net> wrote:
Show 15 quoted lines
>
>
> This follows my earlier query for which the answer was "use git-filter-repo".
> I've cloned git-filter-repo from GitHub, and I've installed the latest Python (version 3.8).  I'm working on Windows.  Windows 10 if it makes any difference, so git is Git For Windows.
> The first instruction for git-filter-repo is to copy it into the "--exec-path" of git.  That fails, saying Access is Denied.  So leaving aside the option to change the permissions I've followed the second half of the sentence and added my git-filter-repo location to my Path (or PATH if it's in bash).  Now running "get filter-repo" gives:
> /usr/bin/env: 'python3.exe': No such file or directory
>
> My python executable is called python3.exe.  There is a note in the git-filter-repo project about certain Windows installations where it isn't called python3, but that now raises in my mind the question of whether I need to tweak git-filter-repo to look for "python3.exe" rather than just "python3".  Or can I take it that the Windows version of bash understands that it needs to add ".exe" automatically?
>
> Assuming that is handled automatically, what do I now need to do to satisfy the error message?  Do I need to put something in bash-for-Windows' /usr/bin/env directory?  Or is there something else entirely that I should be doing?
>
> And is it OK to ask about this on this list, or should I be going to a git-filter-branch resource?  Or for Git For Windows?
>
> Regards,
> Richard.

Emailing this list is fine. You can also open an issue on GitHub for the project if you prefer; I respond in both places. I may have limited ability to help with Windows issues since I don't have a Windows system to use or test on, though I have succeeded in fixing a few Windows bugs based on reporter feedback. (So maybe this list is better because more people can chime in with Windows experience?) I've got at least a dozen or so reports of people using it successfully on Windows so someone has figured it out. I'll try to provide a few pointers that might help.

You don't have to use "#!/usr/bin/env python3" or "#!/usr/bin/env python". I could have put "#!/usr/bin/python3" there, but that might make it specific to my system. I figured other systems installed python3 somewhere other than /usr/bin/, but that /usr/bin/env should be somewhat common and should find the python installation. If /usr/bin/env doesn't happen to exist on your system, though, it is perfectly fine to have "#!/System/Commands/python3" or whatever is needed.

If installation is difficult, you could consider using a package manager. In particular, the scoop package manager, built for Windows, was one of the first out there to package git-filter-repo -- it was even ahead of most the linux package managers. (And, yes, scoop can also be used to install python3 as far as I understand.) I haven't used it myself (because I don't have a Windows system as I mentioned before), but since they were so interested in packaging git-filter-repo and making "scoop install git-filter-repo" part of the instructions (https://github.com/newren/git-filter-repo/issues/20), they look like a friendly bunch that are willing to help.

I also looked through the issues and PRs related to Windows, and
thought that the following links particularly had interesting comments
that might be useful to someone running on Windows:
  * https://github.com/newren/git-filter-repo/issues/36
  * https://github.com/newren/git-filter-repo/pull/10
  * https://github.com/newren/git-filter-repo/issues/48
I don't know if any of them will solve your particular problem, but
they seemed more likely to be of interest or use than most.  Also,
I'll repeat the link for search for reports of Windows issues with
git-filter-repo in case I missed one that might be useful to you:
  * https://github.com/newren/git-filter-repo/issues?q=is%3Aissue+windows
Anyway, I hope at least one of those pointers is useful to you.
Elijah
Elijah Newren· Apr 20, 2020, 17:24 UTC · re: Elijah Newren · lore

Re: git-filter-repo For Windows (possibly a request for a bash for Windows primer)

On Mon, Apr 20, 2020 at 9:53 AM Elijah Newren <newren@gmail.com> wrote:
Show 59 quoted lines
>
> On Mon, Apr 20, 2020 at 8:11 AM Kerry, Richard <richard.kerry@atos.net> wrote:
> >
> >
> > This follows my earlier query for which the answer was "use git-filter-repo".
> > I've cloned git-filter-repo from GitHub, and I've installed the latest Python (version 3.8).  I'm working on Windows.  Windows 10 if it makes any difference, so git is Git For Windows.
> > The first instruction for git-filter-repo is to copy it into the "--exec-path" of git.  That fails, saying Access is Denied.  So leaving aside the option to change the permissions I've followed the second half of the sentence and added my git-filter-repo location to my Path (or PATH if it's in bash).  Now running "get filter-repo" gives:
> > /usr/bin/env: 'python3.exe': No such file or directory
> >
> > My python executable is called python3.exe.  There is a note in the git-filter-repo project about certain Windows installations where it isn't called python3, but that now raises in my mind the question of whether I need to tweak git-filter-repo to look for "python3.exe" rather than just "python3".  Or can I take it that the Windows version of bash understands that it needs to add ".exe" automatically?
> >
> > Assuming that is handled automatically, what do I now need to do to satisfy the error message?  Do I need to put something in bash-for-Windows' /usr/bin/env directory?  Or is there something else entirely that I should be doing?
> >
> > And is it OK to ask about this on this list, or should I be going to a git-filter-branch resource?  Or for Git For Windows?
> >
> > Regards,
> > Richard.
>
> Emailing this list is fine.  You can also open an issue on GitHub for
> the project if you prefer; I respond in both places.  I may have
> limited ability to help with Windows issues since I don't have a
> Windows system to use or test on, though I have succeeded in fixing a
> few Windows bugs based on reporter feedback. (So maybe this list is
> better because more people can chime in with Windows experience?)
> I've got at least a dozen or so reports of people using it
> successfully on Windows so someone has figured it out.  I'll try to
> provide a few pointers that might help.
>
> You don't have to use "#!/usr/bin/env python3" or "#!/usr/bin/env
> python".  I could have put "#!/usr/bin/python3" there, but that might
> make it specific to my system.  I figured other systems installed
> python3 somewhere other than /usr/bin/, but that /usr/bin/env should
> be somewhat common and should find the python installation.  If
> /usr/bin/env doesn't happen to exist on your system, though, it is
> perfectly fine to have "#!/System/Commands/python3" or whatever is
> needed.
>
> If installation is difficult, you could consider using a package
> manager.  In particular, the scoop package manager, built for Windows,
> was one of the first out there to package git-filter-repo -- it was
> even ahead of most the linux package managers.  (And, yes, scoop can
> also be used to install python3 as far as I understand.)  I haven't
> used it myself (because I don't have a Windows system as I mentioned
> before), but since they were so interested in packaging
> git-filter-repo and making "scoop install git-filter-repo" part of the
> instructions (https://github.com/newren/git-filter-repo/issues/20),
> they look like a friendly bunch that are willing to help.
>
> I also looked through the issues and PRs related to Windows, and
> thought that the following links particularly had interesting comments
> that might be useful to someone running on Windows:
>   * https://github.com/newren/git-filter-repo/issues/36
>   * https://github.com/newren/git-filter-repo/pull/10
>   * https://github.com/newren/git-filter-repo/issues/48
> I don't know if any of them will solve your particular problem, but
> they seemed more likely to be of interest or use than most.  Also,
> I'll repeat the link for search for reports of Windows issues with
> git-filter-repo in case I missed one that might be useful to you:
>   * https://github.com/newren/git-filter-repo/issues?q=is%3Aissue+windows

Oh, also https://github.com/newren/git-filter-repo/issues/56 which suggests PowerShell may mess things up (and in a different way than issue 36 showed).

Philip Oakley· Apr 20, 2020, 18:46 UTC · re: Elijah Newren · lore

Re: git-filter-repo For Windows (possibly a request for a bash for Windows primer)

On 20/04/2020 18:24, Elijah Newren wrote:
Show 60 quoted lines
> On Mon, Apr 20, 2020 at 9:53 AM Elijah Newren <newren@gmail.com> wrote:
>> On Mon, Apr 20, 2020 at 8:11 AM Kerry, Richard <richard.kerry@atos.net> wrote:
>>>
>>> This follows my earlier query for which the answer was "use git-filter-repo".
>>> I've cloned git-filter-repo from GitHub, and I've installed the latest Python (version 3.8).  I'm working on Windows.  Windows 10 if it makes any difference, so git is Git For Windows.
>>> The first instruction for git-filter-repo is to copy it into the "--exec-path" of git.  That fails, saying Access is Denied.  So leaving aside the option to change the permissions I've followed the second half of the sentence and added my git-filter-repo location to my Path (or PATH if it's in bash).  Now running "get filter-repo" gives:
>>> /usr/bin/env: 'python3.exe': No such file or directory
>>>
>>> My python executable is called python3.exe.  There is a note in the git-filter-repo project about certain Windows installations where it isn't called python3, but that now raises in my mind the question of whether I need to tweak git-filter-repo to look for "python3.exe" rather than just "python3".  Or can I take it that the Windows version of bash understands that it needs to add ".exe" automatically?
>>>
>>> Assuming that is handled automatically, what do I now need to do to satisfy the error message?  Do I need to put something in bash-for-Windows' /usr/bin/env directory?  Or is there something else entirely that I should be doing?
>>>
>>> And is it OK to ask about this on this list, or should I be going to a git-filter-branch resource?  Or for Git For Windows?
>>>
>>> Regards,
>>> Richard.
>> Emailing this list is fine.  You can also open an issue on GitHub for
>> the project if you prefer; I respond in both places.  I may have
>> limited ability to help with Windows issues since I don't have a
>> Windows system to use or test on, though I have succeeded in fixing a
>> few Windows bugs based on reporter feedback. (So maybe this list is
>> better because more people can chime in with Windows experience?)
>> I've got at least a dozen or so reports of people using it
>> successfully on Windows so someone has figured it out.  I'll try to
>> provide a few pointers that might help.
>>
>> You don't have to use "#!/usr/bin/env python3" or "#!/usr/bin/env
>> python".  I could have put "#!/usr/bin/python3" there, but that might
>> make it specific to my system.  I figured other systems installed
>> python3 somewhere other than /usr/bin/, but that /usr/bin/env should
>> be somewhat common and should find the python installation.  If
>> /usr/bin/env doesn't happen to exist on your system, though, it is
>> perfectly fine to have "#!/System/Commands/python3" or whatever is
>> needed.
>>
>> If installation is difficult, you could consider using a package
>> manager.  In particular, the scoop package manager, built for Windows,
>> was one of the first out there to package git-filter-repo -- it was
>> even ahead of most the linux package managers.  (And, yes, scoop can
>> also be used to install python3 as far as I understand.)  I haven't
>> used it myself (because I don't have a Windows system as I mentioned
>> before), but since they were so interested in packaging
>> git-filter-repo and making "scoop install git-filter-repo" part of the
>> instructions (https://github.com/newren/git-filter-repo/issues/20),
>> they look like a friendly bunch that are willing to help.
>>
>> I also looked through the issues and PRs related to Windows, and
>> thought that the following links particularly had interesting comments
>> that might be useful to someone running on Windows:
>>   * https://github.com/newren/git-filter-repo/issues/36
>>   * https://github.com/newren/git-filter-repo/pull/10
>>   * https://github.com/newren/git-filter-repo/issues/48
>> I don't know if any of them will solve your particular problem, but
>> they seemed more likely to be of interest or use than most.  Also,
>> I'll repeat the link for search for reports of Windows issues with
>> git-filter-repo in case I missed one that might be useful to you:
>>   * https://github.com/newren/git-filter-repo/issues?q=is%3Aissue+windows
> Oh, also https://github.com/newren/git-filter-repo/issues/56 which
> suggests PowerShell may mess things up (and in a different way than
> issue 36 showed).

A similar Python issue occurs on Windows, as reported at https://github.com/git-for-windows/git/issues/2594

with a reply by dscho reminding that:
This issue is listed specifically in the Known Issues in our Release Notes:
        Some console programs, most notably non-MSYS2 Python, PHP, Node
and OpenSSL, interact correctly with MinTTY only when called through
winpty (e.g. the Python console needs to be started as winpty python
instead of just python).

The installer even suggests (on the page titled "Configuring the terminal emulator to use with Git Bash") to

    Use Windows' default console window
    [...] which works well with Win32 console programs such as
interactive Python or node.js [...]

Hope that helps. -- Philip

Kerry, Richard· Apr 21, 2020, 14:52 UTC · re: Philip Oakley · lore

Re: git-filter-repo For Windows (possibly a request for a bash for Windows primer)

Thanks for everyone's advice and assistance. I had a look at all the links provided and wasn't convinced any of them quite covered my requirement, but I have now made some progress using Scoop.

I had hoped that all I would need to do would be:
1. Install latest Python for Windows (3.8). Install for All Users means it's in C:\Program Files\Python38, and that's in the Path.
2. Clone git-filter-repo.
3. Point Path at git-filter-repo.  (Alternative was to copy it into "exec-path" folder under Git, but that was protected.)
4. Run it.

That didn't work: /usr/bin/env: 'python3': No such file or directory. That string pointed me at the shebang line of git-filter-repo, so I set about trying anything that I thought might point at my Python installation.  Nothing I tried worked.  Some things, like "/bin/sh", did indicate that the right file was being found and executed, but gave errors that it was being executed as a shell script, which of course it isn't. Any attempt to point at the python exe seemed to say things along the lines of "git can find it but not understand it" (sorry I can't now copy in the real error message as I had to close the Bash shell in order to update GfW.)

So I gave up on getting it to run using raw Windows - well not entirely raw Windows as it's going to be run from Bash-for-git-for-windows (or so I thought). Installed Scoop.  Got Scoop to install git-filter-repo and python.  At last that lets me run it and have it do something useful. Though actually as Scoop gives its instructions for being run from PowerShell I'm not in the end running from bash at all.

First attempt at running it was using:

git filter-repo --analyze --source=core_system_archive which responded "not a git repository" even though core_system_archive does contain a ".git". But then  git filter-repo --analyze --source=core_system_archive/.git or git filter-repo --analyze --source=./core_system_archive/.git didn't work either, same error, although it looks to me like core_system_archive/.git is a git repo.  So would I need some different syntax to make that work? (And I've since noticed that examples use space rather than '=' to separate keyword from param, so I've tried that, with the same failures) So next option was to "cd .\core_system_archive", then just git filter-repo --analyze  Which has worked. Though it isn't clear what was wrong with the parameter I used for --source. How am I supposed to reference a source repo without cd-ing into it?

So thanks everyone for a lot of hints, but for me, under Windows, only the Scoop package manager managed to get all the right settings for filter-repo to work. And only now, having been rather rushing to get this to work due to other priorities, have I seen the comment about using pip to install it, so maybe that would also have worked.

Regards, Richard.

Oh, and having also seen Philip's reply (below) I don't think I would have recognized it as relating to my issue. What is winpty? Should that have gone in the shebang line of git-filter-repo in some form?

From: Philip Oakley <philipoakley@iee.email>
with a reply by dscho reminding that:
This issue is listed specifically in the Known Issues in our Release Notes:
        Some console programs, most notably non-MSYS2 Python, PHP, Node
and OpenSSL, interact correctly with MinTTY only when called through
winpty (e.g. the Python console needs to be started as winpty python
instead of just python).
The installer even suggests (on the page titled "Configuring the
terminal emulator to use with Git Bash") to
    Use Windows' default console window
    [...] which works well with Win32 console programs such as
interactive Python or node.js [...]

Regards Richard Kerry

BNCS Engineer, SI SOL Telco & Media Vertical Practice
M: +44 (0)7812 325518
2nd Floor, MidCity Place, 71 High Holborn, London, WC1V 6EA
richard.kerry@atos.net
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy
 it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free
 and will not be liable for any damages resulting from any virus transmitted.
Johannes Schindelin· Apr 25, 2020, 14:15 UTC · re: Kerry, Richard · lore

Re: git-filter-repo For Windows (possibly a request for a bash for Windows primer)

Hi Richard,
On Tue, 21 Apr 2020, Kerry, Richard wrote:
Show 7 quoted lines
> Thanks for everyone's advice and assistance.
> I had a look at all the links provided and wasn't convinced any of them
> quite covered my requirement, but I have now made some progress using
> Scoop.
>
> I had hoped that all I would need to do would be:
> 1. Install latest Python for Windows (3.8). Install for All Users means it's in C:\Program Files\Python38, and that's in the Path.

I installed it via the Microsoft Store. This put it into my user-specific AppData, and it added two items to the PATH.

Unfortunately, even calling `python3.exe` from the PATH does not work: it is apparently some placeholder that MSYS2 (and therefore Git Bash and `env.exe`) have no idea what to do with.

What did work for me was `env python`. Of course, I don't have Python 2.x installed, so that might not be an option for you.

> 2. Clone git-filter-repo.
> 3. Point Path at git-filter-repo.  (Alternative was to copy it into "exec-path" folder under Git, but that was protected.)

For the record, Git for Windows also adds the `bin` folder in your home directory to the PATH. That is where you are supposed to put additional things.

Git for Windows considers C:\Program Files\Git its own home, and it might remove your files without prior warning during an upgrade.

It will leave `~/bin/` alone, of course.
Hopefully these two hints prove useful to you.

Ciao, Johannes

Show 65 quoted lines
> 4. Run it.
>
> That didn't work: /usr/bin/env: 'python3': No such file or directory.
> That string pointed me at the shebang line of git-filter-repo, so I set about trying anything that I thought might point at my Python installation.  Nothing I tried worked.  Some things, like "/bin/sh", did indicate that the right file was being found and executed, but gave errors that it was being executed as a shell script, which of course it isn't.  Any attempt to point at the python exe seemed to say things along the lines of "git can find it but not understand it" (sorry I can't now copy in the real error message as I had to close the Bash shell in order to update GfW.)
>
> So I gave up on getting it to run using raw Windows - well not entirely raw Windows as it's going to be run from Bash-for-git-for-windows (or so I thought).
> Installed Scoop.  Got Scoop to install git-filter-repo and python.  At last that lets me run it and have it do something useful.
> Though actually as Scoop gives its instructions for being run from PowerShell I'm not in the end running from bash at all.
>
> First attempt at running it was using:
>
> git filter-repo --analyze --source=core_system_archive
> which responded "not a git repository" even though core_system_archive does contain a ".git".
> But then 
> git filter-repo --analyze --source=core_system_archive/.git
> or
> git filter-repo --analyze --source=./core_system_archive/.git
> didn't work either, same error, although it looks to me like core_system_archive/.git is a git repo.  So would I need some different syntax to make that work?  (And I've since noticed that examples use space rather than '=' to separate keyword from param, so I've tried that, with the same failures)
> So next option was to "cd .\core_system_archive", then just
> git filter-repo --analyze 
> Which has worked.
> Though it isn't clear what was wrong with the parameter I used for --source.  How am I supposed to reference a source repo without cd-ing into it?
>
> So thanks everyone for a lot of hints, but for me, under Windows, only the Scoop package manager managed to get all the right settings for filter-repo to work.
> And only now, having been rather rushing to get this to work due to other priorities, have I seen the comment about using pip to install it, so maybe that would also have worked.
>
> Regards,
> Richard.
>
> Oh, and having also seen Philip's reply (below) I don't think I would have recognized it as relating to my issue.
> What is winpty?  Should that have gone in the shebang line of git-filter-repo in some form?
>
>
> From: Philip Oakley <philipoakley@iee.email>
>
> with a reply by dscho reminding that:
>
> This issue is listed specifically in the Known Issues in our Release Notes:
>
>         Some console programs, most notably non-MSYS2 Python, PHP, Node
> and OpenSSL, interact correctly with MinTTY only when called through
> winpty (e.g. the Python console needs to be started as winpty python
> instead of just python).
>
> The installer even suggests (on the page titled "Configuring the
> terminal emulator to use with Git Bash") to
>     Use Windows' default console window
>     [...] which works well with Win32 console programs such as
> interactive Python or node.js [...]
>
>
> Regards
> Richard Kerry
>
> BNCS Engineer, SI SOL Telco & Media Vertical Practice
> M: +44 (0)7812 325518
> 2nd Floor, MidCity Place, 71 High Holborn, London, WC1V 6EA
> richard.kerry@atos.net
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy
>  it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free
>  and will not be liable for any damages resulting from any virus transmitted.
>
>
>

← back to recent threads