Setting up Password protected repositories?

So at the risk of asking a dead-stupid question for which I should likely be soundly beaten for even asking...

How does one set up a git repository on a public site but password/ssh protected it so that only a few trusted people can access and update it?

Clearly, we should be using ssh+git: here somehow, right? git-daemon has to know where to server out the files from, but how do we get it to enforce a ssh-based access? It's easy to prevent http: from serving out the repository -- just place it outside of the webroot directory structure.

I can already enforce pushes to be password protected, but the protection on pulls and clones eludes me. I'd also like to NOT grant login shell access on the repository server machine, so I'm envisioning placing pubic keys from trusted people somewhere too...

I feel like I am being blindly stupid here.

Any help for the weary?

Thanks, jdl