The imporantance of including http credential caching in 1.7.7

Earlier today, Scott Chacon alerted me to this email thread: http://www.spinics.net/lists/git/msg164192.html (many apologies to the list, I am not sure how to properly reference this email or reply to it since I have not been subscribed until today).

This is very disheartening to hear. Of all the minor changes, bugs, and potential features, I believe that http credential caching is the absolute most important thing that git core can do to promote adoption. I've believed this for more than a year now and I'm incredibly disappointed that it's being shelved for yet another release.

Over the past two years as a designer for GitHub, I've answered ~thousands of support requests and talked face to face with ~thousands of developers of varying git skill levels. Once a month our company does online git training for newbies - https://github.com/training/online and I've had many discussions about newcomer's struggles with Matthew McCullough and Scott Chacon. Previously, I worked at ENTP where I helped polish the very popular "Git for Designers" guide http://hoth.entp.com/output/git_for_designers.html based on feedback. I was also lead designer for GitHub for Mac - an OSX GUI aimed at people less familiar with the command line.

I bring all of this up because I'd like to think I have a good handle on common problems that git newcomers or people resisting git adoption run into. I've been deeply involved in spreading git adoption full time for nearly 3 years - it's something that's incredibly important to me professionally and personally. And the number one problem that *always* comes up is SSH key complexity.

A lot of these problems surface themselves in people saying it's hard to setup git on Windows. When I push these people to explain further, it turns out that the problem is always setting up SSH key authentication on Windows, not necessarily git.

It's incredibly frustrating that git's biggest roadblock has nothing to do with version control at all, but rather network authentication. Just as I believe only *you* can own your availability, I believe git should do it's best to own authentication.

HTTP credential caching combined with Smart HTTP make git one hell of an amazing tool for newcomers and experts alike. I like to envision a world in which git with both these features shipped with the latest OSX install. Developers, designers, or anyone with an inkling for code would have exactly two steps to get started with any given git host:

  1. Set your git config email / username
  2. git clone https://example.com/repository

Contrast this to our current help page for OSX: http://help.github.com/mac-set-up-git/ or worse yet, our Windows setup page with all of it's yelling about what kind of SSH keys to setup: http://help.github.com/win-set-up-git/

Please note that I am not arguing against the merits of SSH keys - for those developers who understand them, they're fantastic. But the reality is the great majority of people who interact with version control do not understand them at all. This results in passwordless SSH keys, confusion, and frustration.

If another git version comes and goes without http credential caching, I fear we as a community have purposefully ignored the largest potential for git adoption currently available. This is important enough for me that I believe it would be in git's best interest to delay the release of 1.7.7 until this feature has been patched to the core team's standards.

I'll summarize with a graph of my opinion of where git's potential for adoption lies:

------------------------------------------------------------------------------
            OPPORTUNITY FOR GIT ADOPTION ACCORDING TO KYLE NEATH

                      Caching of http credentials
                                  |
[=====================================================================||=====]
                                                                          |
                                               Everything else in the universe

------------------------------------------------------------------------------

With hopes and butterflies,

Kyle Neath Director of Design at GitHub

Heya,

On Wed, Sep 7, 2011 at 07:33, Kyle Neath <kneath@gmail.com> wrote:

I, personally, find the graph very convincing :).

Junio mentioned in WC that he wants to see some feedback on it's usage, perhaps that you can help provide this by providing a git patched with this functionality to some of your users and see how they respond?

On Wed, Sep 7, 2011 at 12:46 AM, Sverre Rabbelier <srabbelier@gmail.com> wrote:

Perhaps this is where things get complicated. Due to the nature of the people I'm discussing (people afraid of SSH keys), getting them to run a custom version of git is pretty far out of the question. Peff alluded to this in his reply.

I think the best evidence I can provide is in context of GitHub for Mac. GitHub for Mac defaults to Smart HTTP, with the credentials cached in OSX's Keychain. Effectively, it functions as a patched git with credential caching.

Two months after shipping, we have around 20,000 people that use the application regularly. In that time, we've gotten a ton of positive feedback. So there is a great number of people interested in a git client that uses Smart HTTP. One of the bigger complaints we get is that people constantly have to enter their username & password if they choose to drop to the command line.

Of course, this is all fuzzy data since it's not really comparable with git core. But it's the only data I have.

Then again, perhaps the fact that we spent development time hacking around git's limitation is a data point in itself. Git GUI developers are spending development time to fill a hole in git core.

Kyle

Kyle Neath <kneath@gmail.com> writes:

You are misreading the message utterly wrong that it is not even funny.

If this were a new, insignificant, and obscure feature in a piece of software with mere 20k users, it may be OK to release a new version with the feature in an uncooked shape. The feature may turn out to work only in a very limited scope but not general enough to accomodate other needs you did not anticipate, and your next version may have to fix it in a backward incompatible way, forcing existing users to unlearn what the initial uncooked design and implementation gave them, but by definition such an insignificant and obscure new feature in a system with a small userbase wouldn't have been used by too many people by the time you have to fix it, and the extent of the damage is limited.

Git is not that piece of software with only 20k users, and a lot more importantly, credential caching API is not a feature in an insignificant and obscure corner of the user experience. We have to have at least a warm fuzzy feeling that the API is right from the start so the plug-ins people write for our initial version and the data its users will accumulate with it will not go to waste by hastily releasing a version with an uncooked design and implementation.

"Not urgent" is not "Not important". Quite the contrary, it is important enough that we cannot afford to be hasty.

See http://thread.gmane.org/gmane.comp.version-control.git/180245 for an example of how a discussion can be done to help us get closer to the warm fuzzy feeling of getting the API right in a more mature manner. Don't feel too bad if you become ashamed of the rant you wrote contrasting with that exchange. It is a sign that you are learning.

Welcome to open source. Do not expect others to scratch every itch of your niche. Think of it as an opportunity granted to you to make a difference, not as a roadblock somebody else threw at you because they hate you.

Be thankful, not whiny.

Be good.

Have fun.

Kyle Neath venit, vidit, dixit 07.09.2011 07:33:

So, it's been a year or more that you've been aware of the importance of this issue (from your/github's perspective), and we hear about it now, at the end of the rc phase. I don't know whether jk/http-auth-keyring has been done on github payroll or during spare time. But as you can see, all that is missing is real-world success reports, along with the single-user-multiple-passwords issue which is probably answered best from the real-world perspective (how common is it, do we even need to address it now). So, given the importance this has for you and the company, you sure can help out with what is still left to do, can you?

But note that the credential api is only as good a solution as the helpers. Do we really want all users to employ credential-store because it is "much simpler than ssh"? Deploying what we have now and advocating it as a solution for the ssh-challenged (which will happen whether we, you or someone else advocates it) could easily mean telling people to store their credentials unencrypted. The slash-dot story will be "git security hole", "git stores passwords unencrypted" and so on. And I don't care as much about the PR as about the users whom we'd be serving badly.

So, for the ssh-challenged, we really should make sure that secure helpers are in their hand when the credential api hits a released version (or revert the insecure store helper). There's a KDE attempt on the list. Gnome, Win, Mac helpers anyone? Win version of the cache helper?

Michael

Junio C Hamano <gitster@pobox.com> wrote:

For the sake of my paycheck, I should certainly hope not! I'm not at all suggesting we merge what we have in. However, I do think this feature is important enough to delay the release. I trust in the judgement of the core members to know when something is ready for inclusion in master.

Michael J Gruber <git@drmicha.warpmail.net> wrote:

I apologize if it sounds like that. I've been discussing this situation with many people (including Jeff King) for a very long time now, and it was my understanding that the credential caching was done and simply waiting for a new release. This is the first I've heard that it will not be included in 1.7.7, so I'm voicing my opinion now. Admittedly, late in the game - and I apologize for that.

I'd be happy to help in any capacity I can. Unfortunately I'm no C hacker, and I've accepted that as a character flaw (it's something I'm working on). I'm afraid I can't be of much help with the actual code. What I can provide is an alternate viewpoint to the core team. A viewpoint of someone who's spent 3 years trying to make git easier for newcomers.

I urge the core team to think about what kind of opportunity we have here. Credential caching isn't some minor feature. It's the last piece of the puzzle that promotes Smart HTTP to a viable alternative over the git & ssh protocols. Smart HTTP solves an huge collection of problems people have with using git day to day. One URL to push privately and pull anonymously. No firewall restrictions at universities or workplaces. Username & password authentication. I get kind of turned on just thinking about it.

Kyle

Kyle Neath <kneath@gmail.com> writes:

We do not delay a release for any new feature, as we shoot for 8 to 10 week release cycle these days. For an important feature that we must avoid painting ourselves into a corner that we cannot get out of if we screw up the initial design due to possible backward compatibility issues, it is even more true than other minor features.

This cycle is being delayed longer than anybody wishes for an external reason, and unfortunately it does not mean we have longer period to eyeball the topics in cooking due to that same external reason.

From: "Kyle Neath" <kneath@gmail.com>

Help in any capacity : Would it not be possible for GitHub to provide for those key users such a trial version that includes the patches identified to obtain the "real-world success reports" that are needed, as mentioned in the "Re: What's cooking in git.git (Aug 2011, #07; Wed, 24)"

This should help satisfy the needs from both sides, even if you can only push it to a few clients.

"Philip Oakley" <philipoakley@iee.org> writes:

That would not help very much, as (1) we know what Jeff included as sample keystores are more or less cooked and good, but (2) nobody in your style of trial will come up with different keystore to exercise the API to make sure that will not paint us into a corner we cannot upgrade without major pain in the backward compatibility area. The "real-world success reports" you can generate would only for (1) but at this point we are not worried about that. We are more (much more) worried about (2).

Kyle Neath venit, vidit, dixit 07.09.2011 22:14:

OK, I've calmed down :)

It would be interesting to know what we can rely on in the user group you're thinking about (which I called ssh-challenged). Setting up ssh keys is too complicated. Can we require a working gpg setup? They do want to check sigs, don't they?

What I have in mind is a very simple, but secure version of Jeff's credential-store, respectively his example, somewhat like:

---%<--- STORAGE=$HOME/.credentials

for i in "$@"; do
	case "$i" in
	--unique=*)
		unique=${i#--unique=} ;;
	esac
done

key=$(git config get credential.gpgkey) # or error out

if ! test -e "$STORAGE/$unique"; then
	mkdir -m 0700 "$STORAGE"
	git credential-getpass "$@" | gpg -ear $key >"$STORAGE/$unique"
fi

gpg <"$STORAGE/$unique" ---%<---

Or that in C, probably using Junio's gpg-lib. That would be secure and useful *if* we can rely on people having a convenient gpg setup (gpg-agent or such).

So: What credential store/password wallet/etc. can we rely on for this
group? Is gpg fair game?

Michael

On Thu, Sep 8, 2011 at 9:17 AM, Michael J Gruber <git@drmicha.warpmail.net> wrote: [snip]

I don't think you can require a working gpg setup (at least for not addressing the ssh-challenged group).

[snip]

I think there probably need to be providers for using Keychain under the Mac, gnome-keyring and kwallet under Linux, and probably something using the wincrypt API under Windows. I don't think there's a one-store-fits-all solution here, unfortunately. :-(

I'm actually tempted try and work on a couple of those myself.

-John

On Thu, Sep 08, 2011 at 11:02:11AM -0400, John Szakmeister wrote:

Agreed. Anything harder than ssh keys is right out the window, because they're always the alternative these people could be using (but can't or don't want to).

We could make our own gpg-based password wallet system, but I think it's a really bad idea, for two reasons:

  1. It's reinventing the wheel. Which is bad enough as it is, but is
     doubly bad with security-related code, because it's very easy to
     screw something up when you're writing a lot of new code.

  2. It's inconvenient for users. Nobody wants a separate wallet system
     with its own master password. They want to integrate with the
     wallet system they're already using. Which is generally going to be
     way nicer _anyway_, because it's going to be part of the OS and do
     helpful things like unlock the secret store using their login
     credentials.

Exactly. That's why the helpers communicate via pipes. They don't have to be included with core git at all; you should be able to just drop a third-party git-credential-foo into your PATH.

Please do! I mentioned a few people working on helpers elsewhere in this thread, so you may want to see what they've done and/or coordinate to avoid duplicate effort. Let me know if you have trouble finding the appropriate threads in the list archive.

-Peff

Jeff King venit, vidit, dixit 08.09.2011 21:18:

Sue, the question was: What is easy enough? I hoped that people would be using gpg to check signed tags, and that there might be a simple, convenient gnupg installer for Win and Mac which ties into the respective wallet systems or provides one they use already.

So please let's not deploy credential-store...

On 1.+2.: The idea/hope was to use an existing wallet system which people use for gnupg already to store their passphrase. If that is not used then my suggestion does not help much (the issue of widespread deployment), though it still is a secure version of credential-store for those who want a desktop-independent secure credential store.

It seemed appropriate to leverage GitHub for this:

https://github.com/gitigit/git/wiki/Git-Credentials-Hub

Feel free to add!

Cheers, Michael

Michael J Gruber <git@drmicha.warpmail.net> writes:

I wouldn't be surprised if many people just don't check signed tags at all -- if the repositories they're using even have them in the first place -- particularly amongst the audience in question.

-miles

On Fri, Sep 09, 2011 at 10:05:48AM +0200, Michael J Gruber wrote:

I suspect most people aren't checking signed tags. And even if they did have gpg installed, most people aren't going to want a new password wallet. They're going to want integration with what they're already using.

Which isn't to say that a gpg-based wallet is wrong, it's just that I don't think it's filling the role that really needs filled. If you want to make such a wallet helper, you're welcome to. But it doesn't necessarily need to be a part of git core, and if it's not, then maybe it's worth looking at the zillion other password wallet programs that exist.

FWIW, I keep my passwords in a gpg-encrypted file and wrote a 10-line shell script helper to do lookups for git. :)

I'm tempted to agree. But I also think it represents a nice lowest common denominator. No hassle, no setup, but no security either. And there are situations where that's appropriate (e.g., for unattended cron operation, it's not much different than an unencrypted ssh key on disk). My compromise was to put a big warning at the top of the documentation. Maybe that's not enough, though.

And as far as reinventing the wheel with security code, I don't think git-credential-store counts. It's not secure at all, so there's very little to screw up. :)

Yeah, if there is an existing wallet system based around gpg, then absolutely there should be a helper for it. But I don't know that there is such a widely deployed system. And the helper for it doesn't need to ship with git-core; anybody who uses their wallet system is free to write and distribute the helper.

-Peff

On Wed, Sep 07, 2011 at 02:56:03PM +0200, Michael J Gruber wrote:

To be absolutely clear here, this feature was 100% paid for by GitHub (which isn't to say that I don't think it's a good idea. On the contrary, I think it's awesome; but GitHub money is what provided the time for me to work on it).

When I started at GitHub in January, I was given a giant list of things that GitHub felt would make core git better, but that they hadn't the personnel to improve. And I was told to use my own judgement in adding or removing items from the list based on what I thought git needed, and to prioritize as I saw fit. The fact that it took six months for me to come up with credential patches is because that's how long it took me to figure out what I wanted to write, and to clear my backlog of other git tasks.

So I think the wheels have been turning on this for quite a while from GitHub's perspective.

At the same time, I agree very much with Junio; releasing something with a bad API and then having to fix it later is much worse than delaying the release of a feature by a little bit. And we have very little data on whether the API is "right" at this point. Initially I was concerned that there wasn't going to be enough interest while the patches were in 'next', and that we would have to make a release in order to get people interested enough in writing helpers. But right after I said that, Lukas Sandström showed up with a kdewallet helper. And Ted Zlatanov is working on something for the freedesktop secrets API.

And already there's been some discussion that perhaps the current interface isn't quite what we want and is going to need tweaking. So we are moving forward, and I still hope that we can target the next release of "master" in 8-10 weeks. But this time with more confidence that what's being released is actually right.

In the meantime, the best thing we can do to push it forward is to write helpers. I implemented some basic ones that should work anywhere, but aren't as nice as integration with existing keychains. Some people are working on Linux ones. The single best thing GitHub can do to push this forward right now is to provide a well-written OS X Keychain helper, and to provide feedback on whether git's end of the API is good enough.

-Peff

Jeff King venit, vidit, dixit 08.09.2011 21:10:

Thanks for clarifying. While it should make no difference for the acceptance of patches, it's great to see GitHub invest into scratching their Git itches, and thus contribute back. That's how open source works as a business model :)

...

... and one for Git on Windows? It seems we're lacking both Win and OS X developers here.

... continuing in the other subthread...

On Fri, 09 Sep 2011 10:06:08 +0200 Michael J Gruber <git@drmicha.warpmail.net> wrote: 

MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X MJG> developers here.

Windows doesn't have a standard keychain service, does it?

The OS X Keychain helper should be pretty easy in terms of the system calls (he says after a quick Google search), the hard part IMHO is figuring out the right way to store credentials in it. There are several ways to structure the schema.

For modern Linux systems it's best to target the Secrets API, which is DBUS and XML-based and works with both the KDE and GNOME keychains. I only know about it what I have learned from Michael Albinus' interface in the Emacs source tree, but it certainly seems capable enough. This is what Jeff King was alluding to, I think, about what I'm working on. I have not been able to work on it so far, not for lack of trying.

My #1 target is to implement a GPG-based credential helper using a netrc-style file. I believe that would be the most useful one, though not the easiest one to set up for inexperienced users.

Ted

[Added back some of the CC's]

Ted: we don't usually cull the CC list on the git mailing list.

2011/9/9 Ted Zlatanov <tzz@lifelogs.com>: [snip]

No, it doesn't, but you can use the wincrypt API which allows you to at least encrypt the password from the user's login credentials. In particular, CryptProtectData() and CryptUnprotectData(). That way you can at least have the password stored encrypted on disk.

-John

On Fri, Sep 9, 2011 at 12:32 PM, John Szakmeister <john@szakmeister.net> wrote:

Actually, it seems recent Windows versions does have a credential manager, including an API:

http://www.yanzzee.com/2009/09/windows-keychain.html http://msdn.microsoft.com/en-us/library/aa374731(v=VS.85).aspx#credentials_management_functions

On Fri, Sep 9, 2011 at 6:48 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote: [snip]

Yay!  It's about time they grew that feature. :-)

-John

On Fri, 9 Sep 2011 06:54:15 -0400 John Szakmeister <john@szakmeister.net> wrote: 

JS> On Fri, Sep 9, 2011 at 6:48 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote: JS> [snip]

JS> Yay!  It's about time they grew that feature. :-)

That could work. I hope a Windows developer can take a look. That leaves VMS credential support... Come on guys!

Ted

On Fri, 9 Sep 2011 06:32:25 -0400 John Szakmeister <john@szakmeister.net> wrote: 

JS> [Added back some of the CC's] JS> Ted: we don't usually cull the CC list on the git mailing list.

Sorry, I followed up via GMane (a NNTP interface to the mailing list). I'll use `r'eply instead of `f'ollowup.

I actually set

Mail-Copies-To: never
Gmane-Reply-To-List: yes

because I hate getting CC'd on discussions I already follow via GMane. But that's just my preference :)

JS> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>: JS> [snip] MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X MJG> developers here.

JS> No, it doesn't, but you can use the wincrypt API which allows you to JS> at least encrypt the password from the user's login credentials. In JS> particular, CryptProtectData() and CryptUnprotectData(). That way you JS> can at least have the password stored encrypted on disk.

I don't think that's sufficient but could be wrong.

Ted

On Fri, Sep 09, 2011 at 10:06:08AM +0200, Michael J Gruber wrote:

Yes. I don't often enough mention how awesome GitHub is for funding me and giving me a free hand to improve git. They're doing everything right. So let me mention it here one more time. :)

I mentioned OS X because of Kyle's mention of the GitHub for Mac client. But yes, I do think in the long term we want something similar on Windows. GitHub recently hired a developer with some Windows experience; I'll try to see if he's interested in writing a credential helper.

-Peff