[Bug] git-credential-netrc.perl is not built and is not available in `exec-path`

which is located in `contrib/credential/netrc`.

If I'm not mistaken, it's because it sets the `SCRIPT_PERL` environment variable here: https://github.com/git/git/blob/88dd4282d949cdafff516650c1be8aaf4d67983f/contrib/credential/netrc/Makefile#L13

But the Git Makefile un-sets environment variables here: https://github.com/git/git/blob/88dd4282d949cdafff516650c1be8aaf4d67983f/Makefile#L587

Also credential helpers in `contrib/credential` are not present in the output of `git help -a`, is that a bug?

I'm using Git v2.31.1.

Hi,

Siavash wrote:

I'm missing some basic information:
- what platform are you running on?
- what commands did you use to install Git?
- what are you trying to do?  That can help us with understanding the
  problem or giving better advice tailored to the use case.  The netrc
  credential helper is mostly meant as a demo, since libcurl supports
  ~/.netrc on its own already.

Thanks and hope that helps, Jonathan

Hi,

On 2021-05-21 01:11:26 +0430, Jonathan Nieder <jrnieder@gmail.com> wrote:

I'm using Arch Linux with Linux v5.12.5-arch1-1.

`sudo pacman -S git`

I wanted to use `git-credential-netrc.perl` to use `.authinfo.gpg` as git credential. The `gitcredentials` manual said use: `git config --global credential-helper foo` for using credential helpers. It also mentioned that `git credential-` gets prepended to the external helpers. So I expected: `git config --global credential-helper 'netrc'` to work.

I can use it as a helper if I use an absolute path: '/usr/share/git/credential/netrc/git-credential-netrc.perl'

After looking at the `PKGBUILD` file in Arch Linux's git repository, I found out that they are not making "netrc". https://github.com/archlinux/svntogit-packages/blob/4864e39da0bc99e373f3cb728272a93d66b58cd6/trunk/PKGBUILD#L63-L64

So this is not a bug. As Jeff King mentioned as well. I should have checked the `PKGBUILD`.

The output of `git --exec-path` is `/usr/lib/git-core`. And there are other helpers there(made by Arch Linux). This is the output of `lsd /usr/lib/git-core/ | rg credential`: git-credential git-credential-cache git-credential-cache--daemon git-credential-gnome-keyring git-credential-libsecret git-credential-store

I expected the third-party helpers to be in the output of `git help -a` as mentioned in the manual. They are not in the `PATH`, but they are in git's `exec-path`. So shouldn't they be printed in the help output?

On 2021-05-20 at 09:51:25, Siavash wrote:

Things in contrib are not built by default because they don't necessarily work everywhere. For example, the osxkeychain credential helper won't compile on Linux because the requisite shared libraries are specific to macOS. You'll need to compile them manually and install them in a suitable location.

Note that that location can be someplace like ~/bin, if that's in your PATH. For example, since the Debian packages don't yet ship the libsecret credential helper, I've built it and placed it there. Now that I've done that, git help -a shows git credential-libsecret as an option.

On Fri, May 21, 2021 at 01:55:32AM +0000, brian m. carlson wrote:

I agree with this, but just following up with a bit of a devil's advocate: why not put osxkeychain into a regular "make install", but make it conditional via a Makefile knob, like we do for other platform-specific features?

The big reason most helpers are in contrib/ is because I wanted them to be true third-party contributions that do not share any code with Git. That avoids licensing questions when linking with libraries, and puts them on an equal footing for people who want to implement a helper for their favorite obscure tool.

It has lead to a somewhat funny situation, though. I don't really consider osxkeychain well maintained. It's not built nor tested as part our regular build. I wrote it long ago, but I never actually _used_ it day-to-day, as I've never had a mac. It doesn't seem to have gotten any commits since 2013.

And yet, my impression is that basically every Git user on macOS is using it every day, because both Apple Git and homebrew build it by default (and I think at least in the case of Apple Git, it's hard-coded into the config). A little scary, but nobody seems to have complained. :)

I wonder if we could build it and run it through t0303 as part of the mac CI process (though I recall at the time that it was really finicky for automated testing; it wouldn't even run over an ssh session).

Likewise, we probably could be building and testing the libsecret ones via the Linux CI job (I don't use those either myself, but presumably they pass t0303).

One curiosity is that:

  cd contrib/credential/netrc
  make install

builds and installs into .../libexec/git-core. And we don't seem to include that in "git help -a" (it's not listed in our generated command list, but nor is it an "external command" found in the $PATH).

-Peff

On 2021-05-21 at 10:04:53, Jeff King wrote:

Sure, let's do it.

For osxkeychain, it's probably pretty simple to always build it, since macOS will always have the appropriate libraries if the compiler is installed. I would be in favor of also building by default on Linux and having a Makefile knob to disable that, since the requisite libraries are a part of nearly every distribution and doing so will spur distros to ship it, which many do not.

Running the tests will be harder. macOS, I believe, requires an interactive session to have the keychain unlocked, and on Linux, you require gnome-keyring or an equivalent daemon running, which practically means that you need a desktop session.

On Fri, May 21, 2021 at 10:27:08PM +0000, brian m. carlson wrote:

Hmm. So I tried just building oxskeychain in our CI via contrib, and it fails. :(

It looks like it's OK with clang, but not gcc:

  https://github.com/peff/git/runs/2647748209?check_suite_focus=true

Maybe nobody cares about gcc for this use, but I'm inclined to leave it to somebody who actually runs macOS to poke at further.

I assume you mean contrib/credential/libsecret here. I'd worry that flipping it on by default is annoying for people who build from source. I don't have dev libraries for things like libsecret or glib on my development system[1], and I'd be surprised if most server-oriented machines have even the non-dev versions.

Which isn't an argument against making it easier to build them from the main Makefile, but it seems like having them on by default (with a NO_LIBSECRET) would cause more surprises than the other way around (USE_LIBSECRET or similar).

I suspect that just opening a bug report against distro packages might get some traction (especially if it comes with a patch to create the extra package). I do wonder if packagers are hesitant to reach into contrib/, not knowing how well maintained the contents are (to be honest, I am not confident in how well maintained they are, either; it might be helpful if somebody who routinely used each helper stepped up to say that they would maintain it).

-Peff

[1] I was actually surprised I had libsecret installed at all.
    Apparently pinentry-gtk2 links against it, which I have for use with
    gpg. But my system might not be considered typical anyway, as I
    don't use any kind of "desktop environment".

Jeff King wrote:

I have tried that; do doesn't work. If git developers have $x in "contrib" it's for a reason.

Of course they are! The word "contrib" has an obvious meaning.

This is precisely the reason why I tried to graduate "contrib/completion" out of "contrib" to no avail [1].

If you want to install anything from "contrib" by default, it should be "contrib/completion" first, which is already very well tested, and shipped by many distributions. I would move it outside "contrib", to be semantically consistent.

Then maybe "contrib/credential", sure, following "contrib/completion".

But to do a mish-mash between "/" and "/contrib". Does not make sense.

You can't eat your cake and have it too.

Either they are contributed, or they aren't.

Cheers.

[1] https://lore.kernel.org/git/1399607587-14127-1-git-send-email-felipe.contreras@gmail.com/

Unless you have a backend (like gnome-keyring) libsecret is useless.

Can you do "secret-tool lookup *" and fetch any of your passwords?

Cheers.

On Sun, May 23 2021, Felipe Contreras wrote:

[Minor edit to the quoted text to inline the link]:

Seems like that patch just got no replies at the time. FWIW I'd very much be for it and would encourage you to re-submit it.

I'm not sure s/shared/contrib/g is the best naming though, but maybe I'm contributing to needless bikeshedding by mentioning that.

You apparently named it like that to match where distros usually install it (/usr/share), but we also have docs there, locale, and the perl/ directory usually (well, at least on my distro) ends up there.

I wonder if just a top-level completion/* wouldn't be best, or if we want to group them all together something like optional/{completion,credential}/ or other name suggesting that these are meant to interact with not-always-present 3rd party software. Maybe integrations/* ?

For some of these names a general re-arrangement of contrib/* would be a logical thing to follow, e.g. I think it would make sense to carve out various ci/, contrib/coccinelle, Documentation/doc-diff, check-builtins.sh etc. and other "only for supporting git.git development" or "only called by our own Makefile(s)" into some consistently named pattern.

I'm also very much in favor of building and testing all of this software by default, to the best of our ability. We've had some avoidable bitrot e.g. in subtree and mw-to-git in the past, some of that is a pain to test (e.g. requiring an installed MediaWiki), but we can usually build/test some part of it (e.g. in that case, does it even compile as Perl code?). In other cases we could compile/test things by default on certain platforms if they're platform-specific.

Ævar Arnfjörð Bjarmason wrote:

It is the best location because that's where completions go.

You can check the location bash-completion suggests to install completions to:

  % pkg-config --variable=completionsdir bash-completion
  /usr/share/bash-completion/completions

In the case of zsh it's /usr/share/zsh/site-functions.

Additionally, if you install them in your home directory, it should be $XDG_DATA_HOME/bash-completion/completions.

$XDG_DATA_HOME is $HOME/.local/share (analogous to /usr/share).

Distributions install them there, because that's where they are expected (by bash-completion and zsh).

extra/ is a better name.

However, there's already many things that are optional, like gitk and git-gui, do they belong there too? For that matter locales are optional too.

I think if such a decison to have an extra/ directory is made, it should be orthogonal to the completion graduation.

Me too.

Yeah, some spring cleaning would be good.

I'll put sending the patch in my unending to-do list.

Cheers.

On Mon, May 24 2021, Felipe Contreras wrote:

*Nod* I mean just because it ends up there in the FHS doesn't mean it's
 best for us to mirror that structure in git.git.

The line I was attempting to draw was components that optionally interact with optional 3rd party software.

The i18n framework isn't like that because we build it and interact with ourselves, ditto for say PCRE. Optional, but /usr/bin/git is using it.

As opposed to bash/zsh completions, git will run just fine without either of those shells installed.

The git-gui and gitk programs are also first-party software, just like git-send-email or whatever. We just have knobs not to build them because of the dependencies. It looks like we might be spinning them away from git.git entirely in slow-motion, but so far they're first-class commands.

Thanks.

Ævar Arnfjörð Bjarmason wrote:

It's not just that it ends there, it's how it ends there.

Right now the Arch Linux's git package does this:

  find contrib/ -name '.gitignore' -delete
  cp -a ./contrib/* "$pkgdir"/usr/share/git/

I would rather have an install-shared target to populate /usr/share/git. Having a standar location for distributions would allow scripts to simplify instructions, like:

  source /usr/share/git/completion/prompt.sh

Sure, how install-shared populates /usr/share/git is kind of orthogonal, but it would make sense for install-shared to install stuff from shared/.

git will also run fine without git-send-email, git-instaweb, and git-p4.

I know what is the status quo, but when talking about suggestions for improvement the status quo does not matter. Either the status quo makes sense, or it doesn't.

I know gitk is "first-class", but *should* it? If so, why?

And I know git-completion.bash isn't "first-class", but shouldn't it? If so, why not?

We even run git-completion.bash tests by default, gitk doesn't have tests. We don't even track its history; all the commits are squashed into a single "merge". And surely bash is a much more likely dependency to be present in the user's system than Tcl/Tk.

So I don't really see what gitk has, that git-completion.bash hasn't.

Either both belong in extra/, or none of them do (and if one does an the other doesn't, to me it's clear which is which).

Cheers.

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

Add-ons?

I think dispelling the conception by distro people must begin with an update to contrib/README where it clearly says these are "contributed software" that are not part of "Git". They are not something we stand behind like what we have in the rest of the source tree.

Also, the description there is rather stale. For example, the last paragraph must go as its entirety. The ecosystem has expanded thousand-fold since the document was written, and we no longer encourage people to add new things to contrib/ directory at all. We instead encourage them to write and sell their ware on their own merit alone, without planning to "borrow" authority and gain undue advantage over its competitors by being in the contrib/ area in our tree. .

On 25/05/21 13.51, Junio C Hamano wrote:

Archlinux packagers did remove contrib/ because they don't interest on it [1]:

So basically we should deprecate contrib/, and then after next few releases we proceed to remove it, right?

[1]: https://lore.kernel.org/git/60ac51a7ecf74_1f8f720843@natae.notmuch/

Bagas Sanjaya wrote:

Huh? The whole content of contrib/ is installed in /usr/share/git/ [1]

Some stuff like git-completion.bash is installed *twice*.

Once in `/usr/share/bash-completion/completions/git` and another in `/usr/share/git/completion/git-completion.bash`.

I tried that many years ago[2]. Didn't work.

[1] https://archlinux.org/packages/extra/x86_64/git/#filelink [2] https://lore.kernel.org/git/1399662703-355-1-git-send-email-felipe.contreras@gmail.com/

Junio C Hamano wrote:

An add-on extends the functionality of the core. These are not add-ons.

They know what the README says.

But they also know what the README says is not true. git-completion.bash is completely fine for wide-spread use, and in fact, it's already used widely. They are not going to remove `/usr/share/bash-completion/completions/git` just because upstream developers write something innacurate in contrib/README.

Their first responsibility is to provide the most usable system they can to their users. git completions for bash is essential towards that goal.

As long as the README contains innacurate information they will be forced to ignore it, and keep installing valuable stuff from contrib/ at their own discretion.

They why did you reject my git-related tool[1], and yet accepted a direct competitor git-contacts[2]? It certainly looks like you have undue advantage to one tool over the other.

There's is also hg-to-git, a competition of git-remote-hg.

And for that matter git-completion.zsh, a tool I created, and is now in competition with git-completion[3].

It would be nice if what was stated above was true, but alas it's not.

Cheers.

[1] https://lore.kernel.org/git/CAMP44s2+rNG+VPVJ82UsfrmHugnUwBVJfYF5QRVn0=JC061RFg@mail.gmail.com/ [2] https://lore.kernel.org/git/7vobamgqqh.fsf@alter.siamese.dyndns.org/ [3] https://github.com/felipec/git-completion

On Thu, May 20, 2021 at 02:21:25PM +0430, Siavash wrote:

That part is OK. The netrc build runs "make SCRIPT_PERL=...", which will override any instances where it is set in the top-level Makefile.

So just doing:

  cd contrib/credential/netrc
  make install

should be enough to make it work.

-Peff