{"thread":{"id":"59085","subject":"ctrl-z ignored by git; creates blobs from non-existent repos","startedAt":"2023-01-13T22:01:10Z","lastAt":"2023-01-17T23:01:42Z","messageCount":7,"participants":["Crls","Theodore Ts'o","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"470311","messageId":"632d051b-d81b-b35d-0641-c2488a124810@gmail.com","threadId":"59085","inReplyTo":null,"subject":"ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Crls","fromEmail":"kaploceh@gmail.com","sentAt":"2023-01-13T22:01:01Z","receivedAt":"2023-01-13T22:01:10Z","isPatch":false,"sender":{"key":"kaploceh@gmail.com","avatar":null},"body":"Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent \nrepos\n\nSteps to reproduce 1- git clone github \nwhateverrepo/whatevernonexistentrepo or 1- git clone gitlab \nwhateverrepo/whatevernonexistentrepo 2= Git prompts for a username\n\n3- Press Ctrl-Z to stop *git* from running either on the virtual \nconsole/tty *git* automatically creates blobs with directories and \ndisregards\n\n^Z altogether It goes ahead anyway and creates pleases. It looks as if \n^Z is not enough.\n\nExpected: The same issue does not happen with other non-existent repos \ne.g., git clone git.zx2c4/ it returns the message of fatal repo not found\n\n   According to console_codes (4) CAN (0x18, ^X) and SUB (0x1A, ^Z) \nabort escape sequences should be clear enough\n\n\ngit version 2.36.3\n\n"},{"id":"470386","messageId":"Y8Qfj32h89hq5UD6@mit.edu","threadId":"59085","inReplyTo":"632d051b-d81b-b35d-0641-c2488a124810@gmail.com","subject":"Re: ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2023-01-15T15:45:19Z","receivedAt":"2023-01-15T15:46:24Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Fri, Jan 13, 2023 at 05:01:01PM -0500, Crls wrote:\n> Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent\n> repos\n> \n> Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo\n> or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts\n> for a username\n\n% git clone github whateverrepo/whatevernonexistentrepo\nfatal: repository 'github' does not exist\n\nI think what you meant was:\n\n% git clone https://github.com/whateverrepo/whatevernonexistentrepo\nCloning into 'whatevernonexistentrepo'...\nUsername for 'https://github.com': \n\n> 3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty\n> *git* automatically creates blobs with directories and disregards\n\nSo it's not that Control-Z is being ignored.  It's that by the time\nyou see the prompt for \"Username for 'https://github.com': \", the\ndirectories already exist.  Try looking at\nwhatevernonexistentrepo/.git as soon as the prompt shows up.  You'll\nsee that the .git directory has been greated.\n\nNow, when you type ^Z, the git processes are stopped --- but the\nobjects are created already.\n\nUsername for 'https://github.com': ^Z\n[1]+  Stopped                 git clone https://github.com/whateverrepo/whatevernonexistentrepo\n% ps aux | grep git\ntytso       5097  0.0  0.0   9736  4480 pts/0    T    10:41   0:00 git clone https://github.com/wha\ntytso       5098  0.0  0.0   9736  3992 pts/0    T    10:41   0:00 /usr/lib/git-core/git remote-htt\ntytso       5099  0.0  0.1 102332 16104 pts/0    T    10:41   0:00 /usr/lib/git-core/git-remote-htt\ntytso       5140  0.0  0.0   6332  2072 pts/0    S+   10:43   0:00 grep git\n\n\nThe 'T' means that the processes are stopped.\n\n> Expected: The same issue does not happen with other non-existent repos e.g.,\n> git clone git.zx2c4/ it returns the message of fatal repo not found\n\nSo what's going on is that github.com is not returning a non-existent\nrepo error; it's prompting for a username/password, as _if_ the\nrepository exists.  That's presumably to prevent disclosing\ninformation as to whether or not a private repository exists or not.\n\nOnce the authentication fails, git will remove the partially created\nrepro, so it's really not a problem in practice.\n\nCheers,\n\n\t\t\t\t\t\t- Ted\n"},{"id":"470388","messageId":"Y8RHlukE5vtpq8Pj@coredump.intra.peff.net","threadId":"59085","inReplyTo":"Y8Qfj32h89hq5UD6@mit.edu","subject":"Re: ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-15T18:36:06Z","receivedAt":"2023-01-15T18:36:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 15, 2023 at 10:45:19AM -0500, Theodore Ts'o wrote:\n\n> > Expected: The same issue does not happen with other non-existent repos e.g.,\n> > git clone git.zx2c4/ it returns the message of fatal repo not found\n> \n> So what's going on is that github.com is not returning a non-existent\n> repo error; it's prompting for a username/password, as _if_ the\n> repository exists.  That's presumably to prevent disclosing\n> information as to whether or not a private repository exists or not.\n\nI can confirm that this is exactly the reason.\n\n-Peff\n"},{"id":"470405","messageId":"Y8SCZvMu7DZZH1Pl@localhost.my.domain","threadId":"59085","inReplyTo":"Y8Qfj32h89hq5UD6@mit.edu","subject":"Re: ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Crls","fromEmail":"kaploceh@gmail.com","sentAt":"2023-01-15T22:47:02Z","receivedAt":"2023-01-15T22:47:40Z","isPatch":false,"sender":{"key":"kaploceh@gmail.com","avatar":null},"body":"On Sun, Jan 15, 2023 at 10:45:19AM -0500, Theodore Ts'o wrote:\n> On Fri, Jan 13, 2023 at 05:01:01PM -0500, Crls wrote:\n> > Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent\n> > repos\n> > \n> > Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo\n> > or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts\n> > for a username\n> \n> % git clone github whateverrepo/whatevernonexistentrepo\n> fatal: repository 'github' does not exist\n> \n> I think what you meant was:\n> \n> % git clone https://github.com/whateverrepo/whatevernonexistentrepo\n> Cloning into 'whatevernonexistentrepo'...\n> Username for 'https://github.com': \n> \nYes. That's what I meant… thank you. I was having a problem with the formatting while sending the message to the mailing list\n\nContent-Policy reject msg: The message contains HTML subpart, therefore we consider it SPAM or Outlook Virus. TEXT/PLAIN is accepted.! BF:; S229631AbjAMVJQ (in reply to end of DATA command)\n\n\n> > 3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty\n> > *git* automatically creates blobs with directories and disregards\n> \n> So it's not that Control-Z is being ignored.  It's that by the time\n> you see the prompt for \"Username for 'https://github.com': \", the\n> directories already exist.  Try looking at\n> whatevernonexistentrepo/.git as soon as the prompt shows up.  You'll\n> see that the .git directory has been greated.\n> \n> Now, when you type ^Z, the git processes are stopped --- but the\n> objects are created already.\n\nThe directories exist not because ^Z is used, but because by the time git-clone prompts for a username, git is already set on what to do next. Correct? in other words, the process is shoved down my throat. Or the user's throat in this case. Or going by another analogy, it certainly sounds as if I meant: «Git, please git-clone such and such repo, but let me fix just a typo on the repo name before submitting it, pretty please» and then Git replies: «too late for that chick-a-doodle» and there it goes. It injects blobs all over (well, not all over but on the dir specified).\n\n\n> \n> Username for 'https://github.com': ^Z\n> [1]+  Stopped                 git clone https://github.com/whateverrepo/whatevernonexistentrepo\n> % ps aux | grep git\n> tytso       5097  0.0  0.0   9736  4480 pts/0    T    10:41   0:00 git clone https://github.com/wha\n> tytso       5098  0.0  0.0   9736  3992 pts/0    T    10:41   0:00 /usr/lib/git-core/git remote-htt\n> tytso       5099  0.0  0.1 102332 16104 pts/0    T    10:41   0:00 /usr/lib/git-core/git-remote-htt\n> tytso       5140  0.0  0.0   6332  2072 pts/0    S+   10:43   0:00 grep git\n> \n> \n> The 'T' means that the processes are stopped.\n> \n> > Expected: The same issue does not happen with other non-existent repos e.g.,\n> > git clone git.zx2c4/ it returns the message of fatal repo not found\n> \n> So what's going on is that github.com is not returning a non-existent\n> repo error; it's prompting for a username/password, as _if_ the\n> repository exists.  That's presumably to prevent disclosing\n> information as to whether or not a private repository exists or not.\n\nI agree with you there. Coincidentally speaking, why does a username warrants a prompt from git, is simply beyond me. I mean, that is certainly the more far-fetched reasoning of implementation I've read in a long long time.\n\nCan you git-clone a user? What about the user's settings? What about the remainder its gpg tokens and so forth? In other words, if a user's repo is not found, why even prompting for a username? The latter, that is, the user's repo, is redundant,  when the prompt is clearly asking for a username, and not a repo.\n\n\n> \n> Once the authentication fails, git will remove the partially created\n> repro, so it's really not a problem in practice.\n> \n> Cheers,\n> \n> \t\t\t\t\t\t- Ted\n\nNot true. Preventing the disclosure of information has nothing to do with the issue here. If anything seems clear to me, is that prompting for a username, does indeed disclose usernames, private, public and whatnot from either github or gitlab.\n\nAnd Technically speaking, yes, I agree with you in that if ^Z main operation is to suspend the process, there's not much further ado. Right?\n\np.s\n\nI consulted this issue with Thomas Dickey, before sending an email through this interface, and he acknowledges that the operation from git occurs before. Thus confirming some of your statements, and also… that by the time git exits and it's done with,  in whatever dir of my choosing, ^Z couldn't do anything else about it. But I beg to differ. While ^Z may not do anything about it, then git should instead. Or so I think.\n\nrunning stty -a returns:\n\nswtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;\n\ntake care Ted\n\nCarlos\n\n\n-- \nSeems a computer engineer, a systems analyst, and a programmer were\ndriving down a mountain when the brakes gave out.  They screamed down the\nmountain, gaining speed, but finally managed to grind to a halt, more by\nluck than anything else, just inches from a thousand foot drop to jagged\nrocks.  They all got out of the car:\n        The computer engineer said, \"I think I can fix it.\"\n        The systems analyst said, \"No, no, I think we should take it\ninto town and have a specialist look at it.\"\n        The programmer said, \"OK, but first I think we should get back\nin and see if it does it again.\"\n"},{"id":"470409","messageId":"Y8SNNeQ3fQdrf5Fi@coredump.intra.peff.net","threadId":"59085","inReplyTo":"Y8SCZvMu7DZZH1Pl@localhost.my.domain","subject":"Re: ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-15T23:33:09Z","receivedAt":"2023-01-15T23:34:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 15, 2023 at 05:47:02PM -0500, Crls wrote:\n\n> > Now, when you type ^Z, the git processes are stopped --- but the\n> > objects are created already.\n> \n> The directories exist not because ^Z is used, but because by the time\n> git-clone prompts for a username, git is already set on what to do\n> next. Correct? in other words, the process is shoved down my throat.\n> Or the user's throat in this case. Or going by another analogy, it\n> certainly sounds as if I meant: «Git, please git-clone such and such\n> repo, but let me fix just a typo on the repo name before submitting\n> it, pretty please» and then Git replies: «too late for that\n> chick-a-doodle» and there it goes. It injects blobs all over (well,\n> not all over but on the dir specified).\n\nI don't know what you mean by \"blobs\" here, since we fail to download\nany Git objects at all, let alone blobs. But yes, Git creates the local\nrepository, and then tries to fetch into it. So the directory is created\nbefore it even contacts the remote server at all, and you will see the\nsame behavior whether the repository exists or not. And then if an error\noccurs, it will rolls back by deleting the newly-created repository.\n\nIt _could_ be possible to contact the server first, and only when things\nlooked successful-ish, to create the local repository. But:\n\n  1. The clone command is simply not written that way, and converting it\n     now would be tricky. The clone command's view of the world is that\n     it makes a new repository, sets up config, etc, then fetches into\n     it.\n\n  2. It would not fix all cases anyway. At some point we have to say\n     \"this looks close enough to success to create the directory\", but\n     things can still go wrong after that.\n\nNow if you have a problem with the rollback, there might be a bug there.\nBut it sounds like you are simply stopping the process and not letting\nit finish. It should roll back even if it receives a signal death, but\n^Z is stopping the job and putting it in limbo. If it hurts, don't do it\n(or use \"fg\" or \"bg\" to let it finish).\n\n> > So what's going on is that github.com is not returning a non-existent\n> > repo error; it's prompting for a username/password, as _if_ the\n> > repository exists.  That's presumably to prevent disclosing\n> > information as to whether or not a private repository exists or not.\n> \n> I agree with you there. Coincidentally speaking, why does a username\n> warrants a prompt from git, is simply beyond me. I mean, that is\n> certainly the more far-fetched reasoning of implementation I've read\n> in a long long time.\n> \n> Can you git-clone a user? What about the user's settings? What about\n> the remainder its gpg tokens and so forth? In other words, if a user's\n> repo is not found, why even prompting for a username? The latter, that\n> is, the user's repo, is redundant,  when the prompt is clearly asking\n> for a username, and not a repo.\n\nHuh? GitHub cannot tell you whether you have access to a repository (or,\nfor privacy reasons of the owner of the hypothetical repository, whether\na repository of that name might exist) unless you authenticate. So it\nreturns an HTTP 401. Your authentication in this case requires a\nusername and password. Git asks for the username first, then the\npassword.\n\nIs there something else you think GitHub should be returning there? Or\nsomething else you think Git should do with an HTTP 401?\n\n-Peff\n"},{"id":"470419","messageId":"xmqqmt6jdyf6.fsf@gitster.g","threadId":"59085","inReplyTo":"Y8SCZvMu7DZZH1Pl@localhost.my.domain","subject":"Re: ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-16T02:07:57Z","receivedAt":"2023-01-16T02:08:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Crls <kaploceh@gmail.com> writes:\n\n> ... Coincidentally speaking, why does a username warrants a prompt\n> from git, is simply beyond me. I mean, that is certainly the more\n> far-fetched reasoning of implementation I've read in a long long\n> time.\n>\n> Can you git-clone a user? What about the user's settings? What\n> about the remainder its gpg tokens and so forth? In other words,\n> if a user's repo is not found, why even prompting for a username?\n> The latter, that is, the user's repo, is redundant, when the\n> prompt is clearly asking for a username, and not a repo.\n\nWhen you \"git clone\", you'd give a repository path to the server.\nIf the repository is not open to the general anonymous public, then\nthe server needs to check who you are (by asking username) and\nverify that you are who you claim to be (by asking password).\n\nHere two things you need to pay attention to.\n\n - A user can be the\n   owner of more than one repositories, and\n\n - a repository can be accessed by users other than its owner.\n\nSo even after the repository is known by the server, the server\nstill needs to ask you who you are.\n\nImagine that there are many projects hosted at the same site, the\nrepository path is named after the codename of the project, and the\nproject codename is need-to-know secret.\n\nIf the server side reacted differently between an attempt to clone\nexisting repositories and missing ones, an attacker can try\n\n\tgit clone https://site.example.com/projects/$X    \n\nwith many X's and observe the behaviour of the server.  If the\nserver is known to respond with \"no such repository\" for a missing\none, while responding with \"please identify you\" for an existing\none, you can easily tell if a word $X is a project codename, that is\nsupposed to be secret.\n\n> Preventing the disclosure of information has nothing to do with\n> the issue here. If anything seems clear to me, is that prompting\n> for a username, does indeed disclose usernames, private, public\n> and whatnot from either github or gitlab.\n\nWhen you need to identify yourself to GitHub or GitLab, you'd give\nyour username and password.  You know that GitHub or GitLab have the\nusername so it is not secret to them.  Otherwise they wouldn't be\nable to even recognise you.\n\nSo I am not sure how it \"seems clear\" that asking for the username\nis a problem.  The observed behaviour to ask for the username even\nfor a missing repository is all about avoiding to disclose one bit\nof information: whether a repository exists at the given URL.\n"},{"id":"470554","messageId":"Y8cjxv0CkJPTN+NA@localhost.my.domain","threadId":"59085","inReplyTo":"Y8SNNeQ3fQdrf5Fi@coredump.intra.peff.net","subject":"Re: ctrl-z ignored by git; creates blobs from non-existent repos","fromName":"Crls","fromEmail":"kaploceh@gmail.com","sentAt":"2023-01-17T22:40:06Z","receivedAt":"2023-01-17T23:01:42Z","isPatch":false,"sender":{"key":"kaploceh@gmail.com","avatar":null},"body":"On Sun, Jan 15, 2023 at 06:33:09PM -0500, Jeff King wrote:\n> On Sun, Jan 15, 2023 at 05:47:02PM -0500, Crls wrote:\n> \n> > > Now, when you type ^Z, the git processes are stopped --- but the\n> > > objects are created already.\n> > \n> > The directories exist not because ^Z is used, but because by the time\n> > git-clone prompts for a username, git is already set on what to do\n> > next. Correct? in other words, the process is shoved down my throat.\n> > Or the user's throat in this case. Or going by another analogy, it\n> > certainly sounds as if I meant: «Git, please git-clone such and such\n> > repo, but let me fix just a typo on the repo name before submitting\n> > it, pretty please» and then Git replies: «too late for that\n> > chick-a-doodle» and there it goes. It injects blobs all over (well,\n> > not all over but on the dir specified).\n> \n> I don't know what you mean by \"blobs\" here, since we fail to download\n> any Git objects at all, let alone blobs. But yes, Git creates the local\n> repository, and then tries to fetch into it. So the directory is created\n> before it even contacts the remote server at all, and you will see the\n> same behavior whether the repository exists or not. And then if an error\n> occurs, it will rolls back by deleting the newly-created repository.\n> \n> It _could_ be possible to contact the server first, and only when things\n> looked successful-ish, to create the local repository. But:\n> \n>   1. The clone command is simply not written that way, and converting it\n>      now would be tricky. The clone command's view of the world is that\n>      it makes a new repository, sets up config, etc, then fetches into\n>      it.\n> \n>   2. It would not fix all cases anyway. At some point we have to say\n>      \"this looks close enough to success to create the directory\", but\n>      things can still go wrong after that.\n> \n\n\nWhat I meant by blobs is in the following:\n\nEnding up with a list of directories and subdirectories, is not unusual, is actually very common really with every git-clone while fetching a non-existent repo with a git clone operation either through github or gitlab. First it comes with two (2) git, or one git instance first, then another  and finally one (1) git-remote-helper instance.\n\nps -a | grep 'git' can confirm it\n\nYou'd be able to see \n\n19159 pts/9    00:00:00 git\n19160 pts/9    00:00:00 git\n19161 pts/9    00:00:00 git-remote-http <defunct>\n19188 pts/9    00:00:00 git\n19189 pts/9    00:00:00 git\n19190 pts/9    00:00:00 git-remote-http <defunct>\n\n\nObviously, its <defunct processs > part was only made possible just right after and invoking a `kill -9 process id`\n\nAnd if you wanted to find out the leftovers behind through a `du -h <reponame>` by either by git/git-remote-helper (it doesn't really matter at this point which one out of the two may have been the culprit here):\n\n8.0K    /.git/info\n4.0K    /.git/objects/info\n4.0K    /.git/objects/pack\n12K     /.git/objects\n64K     /.git/hooks\n4.0K    /.git/refs/tags\n4.0K    /.git/refs/heads\n12K     /.git/refs\n4.0K    /.git/branches\n116K    /.git\n120K    /\n\nThese  are the default values. Any curl operation through git with a non-existent repo will get those values.  \n\nNow. We can all proceed trying to sugar coat these leftovers behind an explanation of whatever git-remote-http is actually doing or for that matter, as I said before and without being redundant here, the git-clone processs itself, but it doesn't realyl matter, the end results are the same. Are there leftovers? You can bet on it. It may not happen — and borowwing someone else's phrasing, wording, — «instantaneous», for it's actually quite clever really, to the detriment of an end-user. \n\nI've read tibbits from gitremote-helpers(7) quote: When Git encounters a URL of the form <transport>://<address>, where <transport> is a protocol that it cannot handle natively, it automatically invokes git remote-<transport> with the full URL as thesecond argument. End quote.  But it's unsettling. The whole issue. The gitremote-helper is not helping, no pun intended. Luckily for these use cases ^C and ^\\ seem to be functioning. \n\n\n> Now if you have a problem with the rollback, there might be a bug there.\n> But it sounds like you are simply stopping the process and not letting\n> it finish. It should roll back even if it receives a signal death, but\n> ^Z is stopping the job and putting it in limbo. If it hurts, don't do it\n> (or use \"fg\" or \"bg\" to let it finish).\n> \n> > > So what's going on is that github.com is not returning a non-existent\n> > > repo error; it's prompting for a username/password, as _if_ the\n> > > repository exists.  That's presumably to prevent disclosing\n> > > information as to whether or not a private repository exists or not.\n> > \n> > I agree with you there. Coincidentally speaking, why does a username\n> > warrants a prompt from git, is simply beyond me. I mean, that is\n> > certainly the more far-fetched reasoning of implementation I've read\n> > in a long long time.\n> > \n> > Can you git-clone a user? What about the user's settings? What about\n> > the remainder its gpg tokens and so forth? In other words, if a user's\n> > repo is not found, why even prompting for a username? The latter, that\n> > is, the user's repo, is redundant,  when the prompt is clearly asking\n> > for a username, and not a repo.\n> \n> Huh? GitHub cannot tell you whether you have access to a repository (or,\n> for privacy reasons of the owner of the hypothetical repository, whether\n> a repository of that name might exist) unless you authenticate. So it\n> returns an HTTP 401. Your authentication in this case requires a\n> username and password. Git asks for the username first, then the\n> password.\n> \n> Is there something else you think GitHub should be returning there? Or\n> something else you think Git should do with an HTTP 401?\n> \n> -Peff\n\n-- \nDon't hit the keys so hard, it hurts.\n"}]}