threads / discuss / 59085

ctrl-z ignored by git; creates blobs from non-existent repos

Subject: ctrl-z ignored by git; creates blobs from non-existent repos

## tl;dr

7 messages between Jan 13, 2023 and Jan 17, 2023.

replies: 6people: 4as markdown or json

Crls· Jan 13, 2023, 22:01 UTC · lore

Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent repos

Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts for a username

3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty *git* automatically creates blobs with directories and disregards

^Z altogether It goes ahead anyway and creates pleases. It looks as if ^Z is not enough.

Expected: The same issue does not happen with other non-existent repos 
e.g., git clone git.zx2c4/ it returns the message of fatal repo not found
   According to console_codes (4) CAN (0x18, ^X) and SUB (0x1A, ^Z) 
abort escape sequences should be clear enough
git version 2.36.3
Theodore Ts'o· Jan 15, 2023, 15:45 UTC · re: Crls · lore

Re: ctrl-z ignored by git; creates blobs from non-existent repos

On Fri, Jan 13, 2023 at 05:01:01PM -0500, Crls wrote:
Show 6 quoted lines
> Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent
> repos
> 
> Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo
> or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts
> for a username

% git clone github whateverrepo/whatevernonexistentrepo fatal: repository 'github' does not exist

I think what you meant was:

% git clone https://github.com/whateverrepo/whatevernonexistentrepo Cloning into 'whatevernonexistentrepo'... Username for 'https://github.com':

> 3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty
> *git* automatically creates blobs with directories and disregards

So it's not that Control-Z is being ignored. It's that by the time you see the prompt for "Username for 'https://github.com': ", the directories already exist. Try looking at whatevernonexistentrepo/.git as soon as the prompt shows up. You'll see that the .git directory has been greated.

Now, when you type ^Z, the git processes are stopped --- but the objects are created already.

Username for 'https://github.com': ^Z [1]+ Stopped git clone https://github.com/whateverrepo/whatevernonexistentrepo % ps aux | grep git tytso 5097 0.0 0.0 9736 4480 pts/0 T 10:41 0:00 git clone https://github.com/wha tytso 5098 0.0 0.0 9736 3992 pts/0 T 10:41 0:00 /usr/lib/git-core/git remote-htt tytso 5099 0.0 0.1 102332 16104 pts/0 T 10:41 0:00 /usr/lib/git-core/git-remote-htt tytso 5140 0.0 0.0 6332 2072 pts/0 S+ 10:43 0:00 grep git

The 'T' means that the processes are stopped.
> Expected: The same issue does not happen with other non-existent repos e.g.,
> git clone git.zx2c4/ it returns the message of fatal repo not found

So what's going on is that github.com is not returning a non-existent repo error; it's prompting for a username/password, as _if_ the repository exists. That's presumably to prevent disclosing information as to whether or not a private repository exists or not.

Once the authentication fails, git will remove the partially created repro, so it's really not a problem in practice.

Cheers,
						- Ted
Jeff King· Jan 15, 2023, 18:36 UTC · re: Theodore Ts'o · lore

Re: ctrl-z ignored by git; creates blobs from non-existent repos

On Sun, Jan 15, 2023 at 10:45:19AM -0500, Theodore Ts'o wrote:
Show 7 quoted lines
> > Expected: The same issue does not happen with other non-existent repos e.g.,
> > git clone git.zx2c4/ it returns the message of fatal repo not found
> 
> So what's going on is that github.com is not returning a non-existent
> repo error; it's prompting for a username/password, as _if_ the
> repository exists.  That's presumably to prevent disclosing
> information as to whether or not a private repository exists or not.
I can confirm that this is exactly the reason.
-Peff
Crls· Jan 15, 2023, 22:47 UTC · re: Theodore Ts'o · lore

Re: ctrl-z ignored by git; creates blobs from non-existent repos

On Sun, Jan 15, 2023 at 10:45:19AM -0500, Theodore Ts'o wrote:
Show 17 quoted lines
> On Fri, Jan 13, 2023 at 05:01:01PM -0500, Crls wrote:
> > Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent
> > repos
> > 
> > Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo
> > or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts
> > for a username
> 
> % git clone github whateverrepo/whatevernonexistentrepo
> fatal: repository 'github' does not exist
> 
> I think what you meant was:
> 
> % git clone https://github.com/whateverrepo/whatevernonexistentrepo
> Cloning into 'whatevernonexistentrepo'...
> Username for 'https://github.com': 
> 
Yes. That's what I meant… thank you. I was having a problem with the formatting while sending the message to the mailing list
Content-Policy reject msg: The message contains HTML subpart, therefore we consider it SPAM or Outlook Virus. TEXT/PLAIN is accepted.! BF:; S229631AbjAMVJQ (in reply to end of DATA command)
Show 11 quoted lines
> > 3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty
> > *git* automatically creates blobs with directories and disregards
> 
> So it's not that Control-Z is being ignored.  It's that by the time
> you see the prompt for "Username for 'https://github.com': ", the
> directories already exist.  Try looking at
> whatevernonexistentrepo/.git as soon as the prompt shows up.  You'll
> see that the .git directory has been greated.
> 
> Now, when you type ^Z, the git processes are stopped --- but the
> objects are created already.
The directories exist not because ^Z is used, but because by the time git-clone prompts for a username, git is already set on what to do next. Correct? in other words, the process is shoved down my throat. Or the user's throat in this case. Or going by another analogy, it certainly sounds as if I meant: «Git, please git-clone such and such repo, but let me fix just a typo on the repo name before submitting it, pretty please» and then Git replies: «too late for that chick-a-doodle» and there it goes. It injects blobs all over (well, not all over but on the dir specified).
Show 19 quoted lines
> 
> Username for 'https://github.com': ^Z
> [1]+  Stopped                 git clone https://github.com/whateverrepo/whatevernonexistentrepo
> % ps aux | grep git
> tytso       5097  0.0  0.0   9736  4480 pts/0    T    10:41   0:00 git clone https://github.com/wha
> tytso       5098  0.0  0.0   9736  3992 pts/0    T    10:41   0:00 /usr/lib/git-core/git remote-htt
> tytso       5099  0.0  0.1 102332 16104 pts/0    T    10:41   0:00 /usr/lib/git-core/git-remote-htt
> tytso       5140  0.0  0.0   6332  2072 pts/0    S+   10:43   0:00 grep git
> 
> 
> The 'T' means that the processes are stopped.
> 
> > Expected: The same issue does not happen with other non-existent repos e.g.,
> > git clone git.zx2c4/ it returns the message of fatal repo not found
> 
> So what's going on is that github.com is not returning a non-existent
> repo error; it's prompting for a username/password, as _if_ the
> repository exists.  That's presumably to prevent disclosing
> information as to whether or not a private repository exists or not.
I agree with you there. Coincidentally speaking, why does a username warrants a prompt from git, is simply beyond me. I mean, that is certainly the more far-fetched reasoning of implementation I've read in a long long time.
Can you git-clone a user? What about the user's settings? What about the remainder its gpg tokens and so forth? In other words, if a user's repo is not found, why even prompting for a username? The latter, that is, the user's repo, is redundant,  when the prompt is clearly asking for a username, and not a repo.
Show 7 quoted lines
> 
> Once the authentication fails, git will remove the partially created
> repro, so it's really not a problem in practice.
> 
> Cheers,
> 
> 						- Ted
Not true. Preventing the disclosure of information has nothing to do with the issue here. If anything seems clear to me, is that prompting for a username, does indeed disclose usernames, private, public and whatnot from either github or gitlab.
And Technically speaking, yes, I agree with you in that if ^Z main operation is to suspend the process, there's not much further ado. Right?
p.s
I consulted this issue with Thomas Dickey, before sending an email through this interface, and he acknowledges that the operation from git occurs before. Thus confirming some of your statements, and also… that by the time git exits and it's done with,  in whatever dir of my choosing, ^Z couldn't do anything else about it. But I beg to differ. While ^Z may not do anything about it, then git should instead. Or so I think.
running stty -a returns:
swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;
take care Ted
Carlos
-- 
Seems a computer engineer, a systems analyst, and a programmer were
driving down a mountain when the brakes gave out.  They screamed down the
mountain, gaining speed, but finally managed to grind to a halt, more by
luck than anything else, just inches from a thousand foot drop to jagged
rocks.  They all got out of the car:
        The computer engineer said, "I think I can fix it."
        The systems analyst said, "No, no, I think we should take it
into town and have a specialist look at it."
        The programmer said, "OK, but first I think we should get back
in and see if it does it again."
Jeff King· Jan 15, 2023, 23:33 UTC · re: Crls · lore

Re: ctrl-z ignored by git; creates blobs from non-existent repos

On Sun, Jan 15, 2023 at 05:47:02PM -0500, Crls wrote:
Show 12 quoted lines
> > Now, when you type ^Z, the git processes are stopped --- but the
> > objects are created already.
> 
> The directories exist not because ^Z is used, but because by the time
> git-clone prompts for a username, git is already set on what to do
> next. Correct? in other words, the process is shoved down my throat.
> Or the user's throat in this case. Or going by another analogy, it
> certainly sounds as if I meant: «Git, please git-clone such and such
> repo, but let me fix just a typo on the repo name before submitting
> it, pretty please» and then Git replies: «too late for that
> chick-a-doodle» and there it goes. It injects blobs all over (well,
> not all over but on the dir specified).

I don't know what you mean by "blobs" here, since we fail to download any Git objects at all, let alone blobs. But yes, Git creates the local repository, and then tries to fetch into it. So the directory is created before it even contacts the remote server at all, and you will see the same behavior whether the repository exists or not. And then if an error occurs, it will rolls back by deleting the newly-created repository.

It _could_ be possible to contact the server first, and only when things looked successful-ish, to create the local repository. But:

  1. The clone command is simply not written that way, and converting it
     now would be tricky. The clone command's view of the world is that
     it makes a new repository, sets up config, etc, then fetches into
     it.
  2. It would not fix all cases anyway. At some point we have to say
     "this looks close enough to success to create the directory", but
     things can still go wrong after that.

Now if you have a problem with the rollback, there might be a bug there. But it sounds like you are simply stopping the process and not letting it finish. It should roll back even if it receives a signal death, but ^Z is stopping the job and putting it in limbo. If it hurts, don't do it (or use "fg" or "bg" to let it finish).

Show 15 quoted lines
> > So what's going on is that github.com is not returning a non-existent
> > repo error; it's prompting for a username/password, as _if_ the
> > repository exists.  That's presumably to prevent disclosing
> > information as to whether or not a private repository exists or not.
> 
> I agree with you there. Coincidentally speaking, why does a username
> warrants a prompt from git, is simply beyond me. I mean, that is
> certainly the more far-fetched reasoning of implementation I've read
> in a long long time.
> 
> Can you git-clone a user? What about the user's settings? What about
> the remainder its gpg tokens and so forth? In other words, if a user's
> repo is not found, why even prompting for a username? The latter, that
> is, the user's repo, is redundant,  when the prompt is clearly asking
> for a username, and not a repo.

Huh? GitHub cannot tell you whether you have access to a repository (or, for privacy reasons of the owner of the hypothetical repository, whether a repository of that name might exist) unless you authenticate. So it returns an HTTP 401. Your authentication in this case requires a username and password. Git asks for the username first, then the password.

Is there something else you think GitHub should be returning there? Or something else you think Git should do with an HTTP 401?

-Peff
Crls· Jan 17, 2023, 22:40 UTC · re: Jeff King · lore

Re: ctrl-z ignored by git; creates blobs from non-existent repos

On Sun, Jan 15, 2023 at 06:33:09PM -0500, Jeff King wrote:
Show 34 quoted lines
> On Sun, Jan 15, 2023 at 05:47:02PM -0500, Crls wrote:
> 
> > > Now, when you type ^Z, the git processes are stopped --- but the
> > > objects are created already.
> > 
> > The directories exist not because ^Z is used, but because by the time
> > git-clone prompts for a username, git is already set on what to do
> > next. Correct? in other words, the process is shoved down my throat.
> > Or the user's throat in this case. Or going by another analogy, it
> > certainly sounds as if I meant: «Git, please git-clone such and such
> > repo, but let me fix just a typo on the repo name before submitting
> > it, pretty please» and then Git replies: «too late for that
> > chick-a-doodle» and there it goes. It injects blobs all over (well,
> > not all over but on the dir specified).
> 
> I don't know what you mean by "blobs" here, since we fail to download
> any Git objects at all, let alone blobs. But yes, Git creates the local
> repository, and then tries to fetch into it. So the directory is created
> before it even contacts the remote server at all, and you will see the
> same behavior whether the repository exists or not. And then if an error
> occurs, it will rolls back by deleting the newly-created repository.
> 
> It _could_ be possible to contact the server first, and only when things
> looked successful-ish, to create the local repository. But:
> 
>   1. The clone command is simply not written that way, and converting it
>      now would be tricky. The clone command's view of the world is that
>      it makes a new repository, sets up config, etc, then fetches into
>      it.
> 
>   2. It would not fix all cases anyway. At some point we have to say
>      "this looks close enough to success to create the directory", but
>      things can still go wrong after that.
> 
What I meant by blobs is in the following:
Ending up with a list of directories and subdirectories, is not unusual, is actually very common really with every git-clone while fetching a non-existent repo with a git clone operation either through github or gitlab. First it comes with two (2) git, or one git instance first, then another  and finally one (1) git-remote-helper instance.
ps -a | grep 'git' can confirm it
You'd be able to see 

19159 pts/9 00:00:00 git 19160 pts/9 00:00:00 git 19161 pts/9 00:00:00 git-remote-http <defunct> 19188 pts/9 00:00:00 git 19189 pts/9 00:00:00 git 19190 pts/9 00:00:00 git-remote-http <defunct>

Obviously, its <defunct processs > part was only made possible just right after and invoking a `kill -9 process id`
And if you wanted to find out the leftovers behind through a `du -h <reponame>` by either by git/git-remote-helper (it doesn't really matter at this point which one out of the two may have been the culprit here):

8.0K /.git/info 4.0K /.git/objects/info 4.0K /.git/objects/pack 12K /.git/objects 64K /.git/hooks 4.0K /.git/refs/tags 4.0K /.git/refs/heads 12K /.git/refs 4.0K /.git/branches 116K /.git 120K /

These  are the default values. Any curl operation through git with a non-existent repo will get those values.  
Now. We can all proceed trying to sugar coat these leftovers behind an explanation of whatever git-remote-http is actually doing or for that matter, as I said before and without being redundant here, the git-clone processs itself, but it doesn't realyl matter, the end results are the same. Are there leftovers? You can bet on it. It may not happen — and borowwing someone else's phrasing, wording, — «instantaneous», for it's actually quite clever really, to the detriment of an end-user. 
I've read tibbits from gitremote-helpers(7) quote: When Git encounters a URL of the form <transport>://<address>, where <transport> is a protocol that it cannot handle natively, it automatically invokes git remote-<transport> with the full URL as thesecond argument. End quote.  But it's unsettling. The whole issue. The gitremote-helper is not helping, no pun intended. Luckily for these use cases ^C and ^\ seem to be functioning. 
Show 33 quoted lines
> Now if you have a problem with the rollback, there might be a bug there.
> But it sounds like you are simply stopping the process and not letting
> it finish. It should roll back even if it receives a signal death, but
> ^Z is stopping the job and putting it in limbo. If it hurts, don't do it
> (or use "fg" or "bg" to let it finish).
> 
> > > So what's going on is that github.com is not returning a non-existent
> > > repo error; it's prompting for a username/password, as _if_ the
> > > repository exists.  That's presumably to prevent disclosing
> > > information as to whether or not a private repository exists or not.
> > 
> > I agree with you there. Coincidentally speaking, why does a username
> > warrants a prompt from git, is simply beyond me. I mean, that is
> > certainly the more far-fetched reasoning of implementation I've read
> > in a long long time.
> > 
> > Can you git-clone a user? What about the user's settings? What about
> > the remainder its gpg tokens and so forth? In other words, if a user's
> > repo is not found, why even prompting for a username? The latter, that
> > is, the user's repo, is redundant,  when the prompt is clearly asking
> > for a username, and not a repo.
> 
> Huh? GitHub cannot tell you whether you have access to a repository (or,
> for privacy reasons of the owner of the hypothetical repository, whether
> a repository of that name might exist) unless you authenticate. So it
> returns an HTTP 401. Your authentication in this case requires a
> username and password. Git asks for the username first, then the
> password.
> 
> Is there something else you think GitHub should be returning there? Or
> something else you think Git should do with an HTTP 401?
> 
> -Peff
-- 
Don't hit the keys so hard, it hurts.
Junio C Hamano· Jan 16, 2023, 02:07 UTC · re: Crls · lore

Re: ctrl-z ignored by git; creates blobs from non-existent repos

Crls <kaploceh@gmail.com> writes:
Show 10 quoted lines
> ... Coincidentally speaking, why does a username warrants a prompt
> from git, is simply beyond me. I mean, that is certainly the more
> far-fetched reasoning of implementation I've read in a long long
> time.
>
> Can you git-clone a user? What about the user's settings? What
> about the remainder its gpg tokens and so forth? In other words,
> if a user's repo is not found, why even prompting for a username?
> The latter, that is, the user's repo, is redundant, when the
> prompt is clearly asking for a username, and not a repo.

When you "git clone", you'd give a repository path to the server. If the repository is not open to the general anonymous public, then the server needs to check who you are (by asking username) and verify that you are who you claim to be (by asking password).

Here two things you need to pay attention to.
 - A user can be the
   owner of more than one repositories, and
 - a repository can be accessed by users other than its owner.

So even after the repository is known by the server, the server still needs to ask you who you are.

Imagine that there are many projects hosted at the same site, the repository path is named after the codename of the project, and the project codename is need-to-know secret.

If the server side reacted differently between an attempt to clone existing repositories and missing ones, an attacker can try

	git clone https://site.example.com/projects/$X    

with many X's and observe the behaviour of the server. If the server is known to respond with "no such repository" for a missing one, while responding with "please identify you" for an existing one, you can easily tell if a word $X is a project codename, that is supposed to be secret.

> Preventing the disclosure of information has nothing to do with
> the issue here. If anything seems clear to me, is that prompting
> for a username, does indeed disclose usernames, private, public
> and whatnot from either github or gitlab.

When you need to identify yourself to GitHub or GitLab, you'd give your username and password. You know that GitHub or GitLab have the username so it is not secret to them. Otherwise they wouldn't be able to even recognise you.

So I am not sure how it "seems clear" that asking for the username is a problem. The observed behaviour to ask for the username even for a missing repository is all about avoiding to disclose one bit of information: whether a repository exists at the given URL.

← back to recent threads