{"thread":{"id":"56504","subject":"data loss when doing ls-remote and piped to command","startedAt":"2021-09-15T12:52:15Z","lastAt":"2021-09-18T06:33:48Z","messageCount":13,"participants":["Rolf Eike Beer","Junio C Hamano","Tobias Ulmer","Mike Galbraith","Linus Torvalds","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"435979","messageId":"6786526.72e2EbofS7@devpool47","threadId":"56504","inReplyTo":null,"subject":"data loss when doing ls-remote and piped to command","fromName":"Rolf Eike Beer","fromEmail":"eb@emlix.com","sentAt":"2021-09-15T12:43:10Z","receivedAt":"2021-09-15T12:52:15Z","isPatch":false,"sender":{"key":"eb@emlix.com","avatar":null},"body":"The given repository is a clone of the vanilla kernel.\n\n/usr/bin/git --git-dir=/home/ebeer/repos/upstream/linux/.git ls-remote origin 2>&1 | less\n\nAnd I then see things like this:\n\n6f38b5d6cfd43dde3058a10c68baae9cf17af912        refs/tags/v5.0-rc2\n1c7fc5cbc33980acd13ae83d0b416db002fe95601e7f97f64b59514d936     refs/tags/v5.7-rc2^{}\nd0709bb6da2ab6d49b11643e98abdf79b1a2817f        refs/tags/v5.7-rc3\n\nThis also happens when I cd into the repository and just run \"git ls-remote\" \non some of my repositories, but much less often.\n\nThe remainder of the overly long line is the correct id for that tag. The \nerror does not happen on every run, and on some of my repositories it also \ndiffers from run to run on which tag that happens. Here it seems that it is \nquite stable to happening on this tag. However, a different user on the same \nmachine running the very same command had it happen on v5.7-rc3.\n\nI have the same on my laptop, both run on Opensuse Tumbleweed, updated at the \nsame time this morning. This seems to be quite fragile regarding latency or \nsuch: I can reproduce it with our internal git server, but not with \nkernel.org. This is not bound to less, we originally observed the error on a \nentirely different tool that tried to parse the output of ls-remote.\n\nGiven that there are quite a lot of tags missing I suspect it may be that the \npipe handling is somewhere broken, i.e. too much data is written to a pipe \nthat is already full. I have not been able to provoke that using pv by rate \nlimiting the output so far.\n\n[System Info]\ngit Version:\ngit version 2.33.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 5.14.1-1-default #1 SMP Sat Sep 4 08:22:51 UTC 2021 (67af907) x86_64\nCompiler Info: gnuc: 11.2\nlibc Info: glibc: 2.33\n$SHELL (typically, interactive shell): /bin/bash\n\n[no hooks]\n\n-- \nRolf Eike Beer, emlix GmbH, http://www.emlix.com\nFon +49 551 30664-0, Fax +49 551 30664-11\nGothaer Platz 3, 37083 Göttingen, Germany\nSitz der Gesellschaft: Göttingen, Amtsgericht Göttingen HR B 3160\nGeschäftsführung: Heike Jordan, Dr. Uwe Kracke – Ust-IdNr.: DE 205 198 055\n\nemlix - smart embedded open source\n"},{"id":"436012","messageId":"xmqqwnnhwvnd.fsf@gitster.g","threadId":"56504","inReplyTo":"6786526.72e2EbofS7@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-15T18:17:42Z","receivedAt":"2021-09-15T18:17:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Rolf Eike Beer <eb@emlix.com> writes:\n\n> The given repository is a clone of the vanilla kernel.\n>\n> /usr/bin/git --git-dir=/home/ebeer/repos/upstream/linux/.git ls-remote origin 2>&1 | less\n>\n> And I then see things like this:\n>\n> 6f38b5d6cfd43dde3058a10c68baae9cf17af912        refs/tags/v5.0-rc2\n> 1c7fc5cbc33980acd13ae83d0b416db002fe95601e7f97f64b59514d936     refs/tags/v5.7-rc2^{}\n> d0709bb6da2ab6d49b11643e98abdf79b1a2817f        refs/tags/v5.7-rc3\n\nNot offering any solution, just an observation of the problem and\nannotating the report.\n\nWhat we see on the second line is the beginning of peeled\nv5.0-rc2^{} up to the \"acd13\" (that is, the first 19 bytes of the\nline), followed by the full line for peeled v5.7-rc2^{} (which\nbegins with \"ae83d\").  12407 bytes in between are missing, which\nis even more puzzling as it is not a nice round number.\n\nI wonder if this is \"less\" misconfigured and misbehaving.  Did the\nuser after seeing v5.7-* tags scroll back with 'b' or something?\n\nIf the output (including the 2>&1 redirection) is sent to a file and\nthen \"cat <that-file\" is invoked, does the same thing happen?  How\nabout \"cat <that-file | less\"?\n"},{"id":"436087","messageId":"2279155.Qy0YqsFniq@devpool47","threadId":"56504","inReplyTo":"xmqqwnnhwvnd.fsf@gitster.g","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Rolf Eike Beer","fromEmail":"eb@emlix.com","sentAt":"2021-09-16T06:38:13Z","receivedAt":"2021-09-16T06:38:23Z","isPatch":false,"sender":{"key":"eb@emlix.com","avatar":null},"body":"Am Mittwoch, 15. September 2021, 20:17:42 CEST schrieb Junio C Hamano:\n> Rolf Eike Beer <eb@emlix.com> writes:\n> > The given repository is a clone of the vanilla kernel.\n> > \n> > /usr/bin/git --git-dir=/home/ebeer/repos/upstream/linux/.git ls-remote\n> > origin 2>&1 | less\n> > \n> > And I then see things like this:\n> > \n> > 6f38b5d6cfd43dde3058a10c68baae9cf17af912        refs/tags/v5.0-rc2\n> > 1c7fc5cbc33980acd13ae83d0b416db002fe95601e7f97f64b59514d936    \n> > refs/tags/v5.7-rc2^{} d0709bb6da2ab6d49b11643e98abdf79b1a2817f       \n> > refs/tags/v5.7-rc3\n> Not offering any solution, just an observation of the problem and\n> annotating the report.\n> \n> What we see on the second line is the beginning of peeled\n> v5.0-rc2^{} up to the \"acd13\" (that is, the first 19 bytes of the\n> line), followed by the full line for peeled v5.7-rc2^{} (which\n> begins with \"ae83d\").  12407 bytes in between are missing, which\n> is even more puzzling as it is not a nice round number.\n> \n> I wonder if this is \"less\" misconfigured and misbehaving.  Did the\n> user after seeing v5.7-* tags scroll back with 'b' or something?\n\nTo quote myself:\n\n>> This is not bound to less, we originally observed the error on a \n>> entirely different tool that tried to parse the output of ls-remote.\n\nIn fact when less opened I just started to scroll down until I visually \nnoticed an error.\n\n> If the output (including the 2>&1 redirection) is sent to a file and\n> then \"cat <that-file\" is invoked, does the same thing happen?  How\n> about \"cat <that-file | less\"?\n\nThe redirection seems to be an important part of it. I now did:\n\ngit ... 2>&1 | sha256sum\n\nThis gives different results basically on every run. I also noticed that \nhaving more tags makes it easier to reproduce, so a stable kernel in contrast \nto vanilla is a better trigger. Doing that without the stderr redirection gave \nthe same result every time I tried.\n\nRegards,\n\nEike\n-- \nRolf Eike Beer, emlix GmbH, http://www.emlix.com\nFon +49 551 30664-0, Fax +49 551 30664-11\nGothaer Platz 3, 37083 Göttingen, Germany\nSitz der Gesellschaft: Göttingen, Amtsgericht Göttingen HR B 3160\nGeschäftsführung: Heike Jordan, Dr. Uwe Kracke – Ust-IdNr.: DE 205 198 055\n\nemlix - smart embedded open source\n"},{"id":"436096","messageId":"85a103f6-8b3c-2f21-cc0f-04f517c0c9a1@emlix.com","threadId":"56504","inReplyTo":"2279155.Qy0YqsFniq@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Tobias Ulmer","fromEmail":"tu@emlix.com","sentAt":"2021-09-16T10:12:48Z","receivedAt":"2021-09-16T10:22:23Z","isPatch":false,"sender":{"key":"tu@emlix.com","avatar":null},"body":"On 16/09/2021 08:38, Rolf Eike Beer wrote:\n...\n> The redirection seems to be an important part of it. I now did:\n> \n> git ... 2>&1 | sha256sum\n\nI've tried to reproduce this since yesterday, but couldn't until now:\n\n2>&1 made all the difference, took less than a minute.\n\nDifferent repo, different machine, but also running Tumbleweed \n5.14.1-1-default, git 2.33.0\n\nwhile [ \"`git --git-dir=$PWD/in/linux/.git ls-remote origin 2>&1 | tee \nfailed.out | sha1sum`\" = \"7fa299e589bacdc908395730beff542b0fc684eb  -\" \n]; do echo -n .; done\n..........\n\nfailed.out has multiple lines like this:\n\n--8<--\n4e77f7f1261f65cff06918bc5e66d02a418fc842        refs/tags/v3.10.18^{}\nf7b8df0cc81cf82a4ac6834225bddbe46a340455a4a5d52f29d08d923ce8d232b0b497da674dd2c \nrefs/tags/v3.18\nb2776bf7149bddd1f4161f14f79520f17fc1d71d        refs/tags/v3.18^{}\n--8<--\n\n\nRunning the same on Archlinux (5.13.13-arch1-1, 2.33.0) doesn't show the \nproblem.\nThis may well turn out not to be git, but a kernel issue.\n\n@Eike: I think at this point you should try to downgrade and see whether \nthat makes any difference\n"},{"id":"436116","messageId":"2677927.DK6gFqPMyL@devpool47","threadId":"56504","inReplyTo":"85a103f6-8b3c-2f21-cc0f-04f517c0c9a1@emlix.com","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Rolf Eike Beer","fromEmail":"eb@emlix.com","sentAt":"2021-09-16T12:17:34Z","receivedAt":"2021-09-16T12:17:42Z","isPatch":false,"sender":{"key":"eb@emlix.com","avatar":null},"body":"Am Donnerstag, 16. September 2021, 12:12:48 CEST schrieb Tobias Ulmer:\n> On 16/09/2021 08:38, Rolf Eike Beer wrote:\n> ...\n> \n> > The redirection seems to be an important part of it. I now did:\n> > \n> > git ... 2>&1 | sha256sum\n> \n> I've tried to reproduce this since yesterday, but couldn't until now:\n> \n> 2>&1 made all the difference, took less than a minute.\n> \n> Different repo, different machine, but also running Tumbleweed\n> 5.14.1-1-default, git 2.33.0\n> \n> while [ \"`git --git-dir=$PWD/in/linux/.git ls-remote origin 2>&1 | tee\n> failed.out | sha1sum`\" = \"7fa299e589bacdc908395730beff542b0fc684eb  -\"\n> ]; do echo -n .; done\n> ..........\n> \n> failed.out has multiple lines like this:\n> \n> --8<--\n> 4e77f7f1261f65cff06918bc5e66d02a418fc842        refs/tags/v3.10.18^{}\n> f7b8df0cc81cf82a4ac6834225bddbe46a340455a4a5d52f29d08d923ce8d232b0b497da674d\n> d2c refs/tags/v3.18\n> b2776bf7149bddd1f4161f14f79520f17fc1d71d        refs/tags/v3.18^{}\n> --8<--\n> \n> \n> Running the same on Archlinux (5.13.13-arch1-1, 2.33.0) doesn't show the\n> problem.\n> This may well turn out not to be git, but a kernel issue.\n\nLinus,\n\nsince you have been hacking around in pipe.c recently, I fear this isn't \nentirely impossible. Have you any idea?\n\nFor easier reference, the complete thread is at:\n\nhttps://public-inbox.org/git/85a103f6-8b3c-2f21-cc0f-04f517c0c9a1@emlix.com/T/\n\nEike\n-- \nRolf Eike Beer, emlix GmbH, http://www.emlix.com\nFon +49 551 30664-0, Fax +49 551 30664-11\nGothaer Platz 3, 37083 Göttingen, Germany\nSitz der Gesellschaft: Göttingen, Amtsgericht Göttingen HR B 3160\nGeschäftsführung: Heike Jordan, Dr. Uwe Kracke – Ust-IdNr.: DE 205 198 055\n\nemlix - smart embedded open source\n"},{"id":"436129","messageId":"b14d79e49e3abe3fdf00cf18bb8c992b4575c5cc.camel@gmx.de","threadId":"56504","inReplyTo":"2677927.DK6gFqPMyL@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Mike Galbraith","fromEmail":"efault@gmx.de","sentAt":"2021-09-16T15:49:59Z","receivedAt":"2021-09-16T15:50:48Z","isPatch":false,"sender":{"key":"efault@gmx.de","avatar":null},"body":"On Thu, 2021-09-16 at 14:17 +0200, Rolf Eike Beer wrote:\n> Am Donnerstag, 16. September 2021, 12:12:48 CEST schrieb Tobias Ulmer:\n> > On 16/09/2021 08:38, Rolf Eike Beer wrote:\n> > ...\n> >\n> > > The redirection seems to be an important part of it. I now did:\n> > >\n> > > git ... 2>&1 | sha256sum\n> >\n> > I've tried to reproduce this since yesterday, but couldn't until now:\n> >\n> > 2>&1 made all the difference, took less than a minute.\n> >\n> > Different repo, different machine, but also running Tumbleweed\n> > 5.14.1-1-default, git 2.33.0\n> >\n> > while [ \"`git --git-dir=$PWD/in/linux/.git ls-remote origin 2>&1 | tee\n> > failed.out | sha1sum`\" = \"7fa299e589bacdc908395730beff542b0fc684eb  -\"\n> > ]; do echo -n .; done\n> > ..........\n> >\n> > failed.out has multiple lines like this:\n> >\n> > --8<--\n> > 4e77f7f1261f65cff06918bc5e66d02a418fc842        refs/tags/v3.10.18^{}\n> > f7b8df0cc81cf82a4ac6834225bddbe46a340455a4a5d52f29d08d923ce8d232b0b497da674d\n> > d2c refs/tags/v3.18\n> > b2776bf7149bddd1f4161f14f79520f17fc1d71d        refs/tags/v3.18^{}\n> > --8<--\n> >\n> >\n> > Running the same on Archlinux (5.13.13-arch1-1, 2.33.0) doesn't show the\n> > problem.\n> > This may well turn out not to be git, but a kernel issue.\n>\n> Linus,\n>\n> since you have been hacking around in pipe.c recently, I fear this isn't\n> entirely impossible. Have you any idea?\n>\n> For easier reference, the complete thread is at:\n>\n> https://public-inbox.org/git/85a103f6-8b3c-2f21-cc0f-04f517c0c9a1@emlix.com/T/\n>\n\nI use git-daemon (2.33) and reference clones for my local pile of\nkernel trees (74), so out of curiosity, modified the above ls-remote\nloop to fit one of them, and tried to reproduce with both master.today\n(ff1ffd71) and SUSE's stable branch (where Tumbleweed gets source,\ncurrently at 5.14.4).  Both kernels failed to reproduce given a few\nminutes each (zzzz) to do so.  I'm running Leap-15.3 vs Tumbleweed, but\nthat shouldn't matter.\n\n\t-Mike\n"},{"id":"436133","messageId":"CAHk-=wgyk0mwYcMRC8HakzoAKL2Y3gwzD433tqKYYhV+r1PLnA@mail.gmail.com","threadId":"56504","inReplyTo":"2677927.DK6gFqPMyL@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2021-09-16T17:11:22Z","receivedAt":"2021-09-16T18:42:13Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, Sep 16, 2021 at 5:17 AM Rolf Eike Beer <eb@emlix.com> wrote:\n>\n> Am Donnerstag, 16. September 2021, 12:12:48 CEST schrieb Tobias Ulmer:\n> > > The redirection seems to be an important part of it. I now did:\n> > >\n> > > git ... 2>&1 | sha256sum\n> >\n> > I've tried to reproduce this since yesterday, but couldn't until now:\n> >\n> > 2>&1 made all the difference, took less than a minute.\n\nSo if that redirection is what matters, and what causes problems, I\ncan almost guarantee that the reason is very simple:\n\nYour git repository (or more likely your upstream) has some problem,\nit's getting reported on stderr, and because you mix stdout and stderr\nwith that '2>&1', you get randomly mixed output.\n\nThen it depends on timing where the mixing happens.\n\nOr rather, it depends on various different factors, like the buffering\ndone internally by stdio (where stdout generally will be\nblock-buffered, while stderr is usually line-buffered, which is why\nyou get odd mixing of the two).\n\nBut timing can be an effect particularly with \"git ls-remote\" and\nfriends, because you may get errors from the transport asynchronously.\n\nSo the different buffering ends up causing the effect of mixing things\nin the middle of lines, while the timing differences due to the\nasynchronous nature of the remote access pipeline will likely then\ncause that odd mixing to be different.\n\nEnd result: corrupted lines, and different sha256sum every time.\n\n> > Running the same on Archlinux (5.13.13-arch1-1, 2.33.0) doesn't show the\n> > problem.\n> > This may well turn out not to be git, but a kernel issue.\n\nMuch more likely that the other box just doesn't have the error situation.\n\n> since you have been hacking around in pipe.c recently, I fear this isn't\n> entirely impossible. Have you any idea?\n\nAlmost certainly not the kernel. Kernel - and other - differences\ncould affect timing, of course, but the whole \"2>&1\" really is\nfundamentally bogus.\n\nIf you don't have any errors, then the \"2>&1\" doesn't matter.\n\nAnd if you *do* have errors, then by definition the \"2>&1\" will mix in\nthe errors with the output randomly and piping them together is\nsenseless.\n\nEither way, it's wrong.\n\nSo what I'd suggest Tobias should do is\n\n    git ... 2> err | sha256sum\n\nwhich will send the errors to the \"err\" file. Take a look at that file\nafterwards and see what is in it.\n\nBasically, '2&>1\" is almost never the right thing to do, unless you\nexplicitly don't care about the output and just want to suppress it.\n\nSo \"2&>1 > /dev/null\" is common and natural.\n\nOf course, people also use it when they just want to eyeball the\nerrors mixed in, so doing that\n\n   ... 2&>1 | less\n\nthing isn't necessarily *wrong*, but it's somewhat dangerous and\nconfusing. Because when you do it you do need to be very aware of the\nfact that the errors and output will be *mixed*. And the mixing will\nnot necessarily be at all sensible.\n\nFinally: pipes on a low level guarantee certain atomicity constraints,\nso if you do low-level \"write()\" calls of size PIPE_BUF or less, the\ncontents will not be interleaved randomly.  HOWEVER. That's only true\nat that \"write()\" level. The moment you use <stdio> for your IO, you\nhave buffering inside of the standard IO libraries, and if your code\nisn't explicitly very careful about it, using setbuf() and fflush()\nand friends, you'll get that random mixing.\n\nAnyway. That was a long email just to tell people it's almost\ncertainly user error, not the kernel.\n\n            Linus\n"},{"id":"436161","messageId":"xmqq7dfgtfpt.fsf@gitster.g","threadId":"56504","inReplyTo":"CAHk-=wgyk0mwYcMRC8HakzoAKL2Y3gwzD433tqKYYhV+r1PLnA@mail.gmail.com","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-16T20:42:22Z","receivedAt":"2021-09-16T20:42:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> writes:\n\n> On Thu, Sep 16, 2021 at 5:17 AM Rolf Eike Beer <eb@emlix.com> wrote:\n>>\n>> Am Donnerstag, 16. September 2021, 12:12:48 CEST schrieb Tobias Ulmer:\n>> > > The redirection seems to be an important part of it. I now did:\n>> > >\n>> > > git ... 2>&1 | sha256sum\n>> >\n>> > I've tried to reproduce this since yesterday, but couldn't until now:\n>> >\n>> > 2>&1 made all the difference, took less than a minute.\n> \n> So if that redirection is what matters, and what causes problems, I\n> can almost guarantee that the reason is very simple:\n> ...\n> Anyway. That was a long email just to tell people it's almost\n> certainly user error, not the kernel.\n\nYes, 2>&1 will mix messages from the standard error stream at random\nplaces in the output, which explains the checksum quite well.\n\nI am not sure if it explains the initial report where\n\n\tls-remote 2>&1 | less\n\nproduced\n\n    > 6f38b5d6cfd43dde3058a10c68baae9cf17af912        refs/tags/v5.0-rc2\n    > 1c7fc5cbc33980acd13ae83d0b416db002fe95601e7f97f64b59514d936     refs/tags/v5.7-rc2^{}\n    > d0709bb6da2ab6d49b11643e98abdf79b1a2817f        refs/tags/v5.7-rc3\n\n    What we see on the second line is the beginning of peeled\n    v5.0-rc2^{} up to the \"acd13\" (that is, the first 19 bytes of the\n    line), followed by the full line for peeled v5.7-rc2^{} (which\n    begins with \"ae83d\").  12407 bytes in between are missing, which\n    is even more puzzling as it is not a nice round number.\n\nI can sort of guess that the progress display during transfer, which\ncomes out on the standard error stream and uses terminal control\nsequences like \"go back to the end of the line without feeding a new\nline\", \"erase to the end of the line\", etc., would be contributing,\nbut because it is piped to \"less\", which would make it \"visible\"\n(i.e. you do not get the raw escape but see three capital letters\nESC in reverse), it does not quite explain how the display was\nbroken.\n\nIn any case, I do not think the kernel is involved, or more\ngenerally I do not think any \"loss of output bytes\" is happening\nhere.  It's just \"| less\" that failed to show a range about 12k\nbytes long is mystery to me ;-).\n\n\n"},{"id":"436210","messageId":"d90b59b8b0d049d4afd72faf04ff680ae5f91b85.camel@gmx.de","threadId":"56504","inReplyTo":"b14d79e49e3abe3fdf00cf18bb8c992b4575c5cc.camel@gmx.de","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Mike Galbraith","fromEmail":"efault@gmx.de","sentAt":"2021-09-17T06:38:34Z","receivedAt":"2021-09-17T06:38:43Z","isPatch":false,"sender":{"key":"efault@gmx.de","avatar":null},"body":"On Thu, 2021-09-16 at 17:49 +0200, Mike Galbraith wrote:\n> Both kernels failed to reproduce...\n\nNor did the TW kernel (now 5.14.2-1-default) reproduce, neither in my\nLeap-15.3 box, nor in a TW KVM set up to play server.  'course that\ndoesn't mean there's no kernel bug lurking, means with certainty only\nthat if there is one, the posted reproducer ain't all that wonderful.\n\n\t-Mike\n"},{"id":"436212","messageId":"2722184.bRktqFsmb4@devpool47","threadId":"56504","inReplyTo":"xmqq7dfgtfpt.fsf@gitster.g","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Rolf Eike Beer","fromEmail":"eb@emlix.com","sentAt":"2021-09-17T06:59:07Z","receivedAt":"2021-09-17T06:59:15Z","isPatch":false,"sender":{"key":"eb@emlix.com","avatar":null},"body":"Am Donnerstag, 16. September 2021, 22:42:22 CEST schrieb Junio C Hamano:\n> Linus Torvalds <torvalds@linux-foundation.org> writes:\n> > On Thu, Sep 16, 2021 at 5:17 AM Rolf Eike Beer <eb@emlix.com> wrote:\n> >> Am Donnerstag, 16. September 2021, 12:12:48 CEST schrieb Tobias Ulmer:\n> >> > > The redirection seems to be an important part of it. I now did:\n> >> > > \n> >> > > git ... 2>&1 | sha256sum\n> >> > \n> >> > I've tried to reproduce this since yesterday, but couldn't until now:\n> >> > \n> >> > 2>&1 made all the difference, took less than a minute.\n> > \n> > So if that redirection is what matters, and what causes problems, I\n> > can almost guarantee that the reason is very simple:\n> > ...\n> > Anyway. That was a long email just to tell people it's almost\n> > certainly user error, not the kernel.\n> \n> Yes, 2>&1 will mix messages from the standard error stream at random\n> places in the output, which explains the checksum quite well.\n\nIf there would be any errors. The point is: if I run the command with \">/dev/\nnull\" just to the terminals a hundred times there is never any output on \nstderr at all. If I pipe stderr into a file it's empty after all of this (yes, \nI did append, not overwrite).\n\nThat the particular construct in this case is sort of nonsense is granted, I \njust hit it because some tool here used some very similar construct and \nsuddenly started failing. \"less\" isn't the original reproducer, it was just \nsomething I started testing with to be able to easily visually inspect the \noutput.\n\nWhat you need is a _fast_ git server. kernel.org or github.com seem to be too \nslow for this if you don't sit somewhere in their datacenter. Use something in \nyour local network, a Xeon E5 with lot's of RAM and connected with 1GBit/s \nEthernet in my case.\n\nAnd the reader must be \"somewhat\" slow. Using sha256sum works reliably for me. \nUsing \"wc -l\" does not, also md5sum and sha1sum are too fast as it seems.\n\nWhen I run the whole thing with strace I can't see the effect, which isn't \nreally surprising. But there is a difference between the cases where I run \nwith redirection \"2>&1\":\n\nioctl(2, TCGETS, 0x7ffd6f119b10)        = -1 ENOTTY (Inappropriate ioctl for \ndevice)\n\nand without:\n\nioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0\n\nAFAICT this is the only place where fd 2 is used at all during the whole time.\n\nRegards,\n\nEike\n-- \nRolf Eike Beer, emlix GmbH, http://www.emlix.com\nFon +49 551 30664-0, Fax +49 551 30664-11\nGothaer Platz 3, 37083 Göttingen, Germany\nSitz der Gesellschaft: Göttingen, Amtsgericht Göttingen HR B 3160\nGeschäftsführung: Heike Jordan, Dr. Uwe Kracke – Ust-IdNr.: DE 205 198 055\n\nemlix - smart embedded open source\n"},{"id":"436245","messageId":"YUTo1BTp7BXOw6K9@coredump.intra.peff.net","threadId":"56504","inReplyTo":"2722184.bRktqFsmb4@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-17T19:13:24Z","receivedAt":"2021-09-17T19:13:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 17, 2021 at 08:59:07AM +0200, Rolf Eike Beer wrote:\n\n> What you need is a _fast_ git server. kernel.org or github.com seem to be too \n> slow for this if you don't sit somewhere in their datacenter. Use something in \n> your local network, a Xeon E5 with lot's of RAM and connected with 1GBit/s \n> Ethernet in my case.\n\nOne thing that puzzled me here: is the bad output between the server and\nls-remote, or between ls-remote and its output pipe?\n\nI'd guess it has to be the latter, since otherwise ls-remote itself\nwould barf with an error message.\n\nIn that case, I'd think \"git ls-remote .\" would give you the fastest\noutcome, because it's talking to upload-pack on the local box. But I'm\nalso confused how the speed could matter, as ls-remote reads the entire\ninput into an in-memory array, and then formats it.\n\nWe do the write using printf(). Is it possible your libc's stdio may\ndrop bytes when the pipe is full, rather than blocking? In general, I'd\nexpect write() to block, so libc doesn't have to care at all. But might\nthere be something in your environment putting the pipe into\nnon-blocking mode, and we get EAGAIN or something? If so, I'd expect\nstdio to return the error.\n\nMaybe patching Git like this would help:\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex f4fd823af8..5936b2b42c 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -146,7 +146,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\tconst struct ref_array_item *ref = ref_array.items[i];\n \t\tif (show_symref_target && ref->symref)\n \t\t\tprintf(\"ref: %s\\t%s\\n\", ref->symref, ref->refname);\n-\t\tprintf(\"%s\\t%s\\n\", oid_to_hex(&ref->objectname), ref->refname);\n+\t\tif (printf(\"%s\\t%s\\n\", oid_to_hex(&ref->objectname), ref->refname) < 0)\n+\t\t\tdie_errno(\"printf failed\");\n \t\tstatus = 0; /* we found something */\n \t}\n \n\n> And the reader must be \"somewhat\" slow. Using sha256sum works reliably for me. \n> Using \"wc -l\" does not, also md5sum and sha1sum are too fast as it seems.\n\nIf a slow pipe is involved, maybe:\n\n  git ls-remote . | (sleep 5; cat) | sha256sum\n\nwould help reproduce. Assuming ls-remote's output is bigger than your\nsystem pipe buffer (which is another interesting thing to check), then\nit should block for 5 seconds on write() midway through the output,\nwhich you can verify with strace.\n\n-Peff\n"},{"id":"436251","messageId":"CAHk-=wgv42Wm3uHPntZNEYFu-dDVYW7yRen1fUBi6keZaKb+_g@mail.gmail.com","threadId":"56504","inReplyTo":"2722184.bRktqFsmb4@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2021-09-17T19:28:31Z","receivedAt":"2021-09-17T19:28:52Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, Sep 16, 2021 at 11:59 PM Rolf Eike Beer <eb@emlix.com> wrote:\n>\n> When I run the whole thing with strace I can't see the effect, which isn't\n> really surprising. But there is a difference between the cases where I run\n> with redirection \"2>&1\":\n>\n> ioctl(2, TCGETS, 0x7ffd6f119b10)        = -1 ENOTTY (Inappropriate ioctl for device)\n\nEhh. That format of strace implies that you didn't use \"strace -f\"\n(which would have the PID in it).\n\nAlthough maybe you edited it out.\n\nI think the error output would come from the other process (ssh, or\nwhatever process you use to run \"git-upload-pack\" on the other end).\n\nI still strongly doubt it's about pipes - we've had changes to them,\nbut if they are broken we'd see a lot more breakage than some very\nincidental use by git.\n\nBut I can easily see it being timing-dependent. And yes, sadly\n'strace' can often end up hiding any timing issues because it\nobviously slows down the target quite a bit.\n\nDoing \"strace -o tracefile -f\" in a loop would be interesting if you\ncan reproduce it (and then stop when you reproduce it, so that the\nfinal 'tracefile' is the one for the case that reproduced it).\n\n            Linus\n"},{"id":"436294","messageId":"9bd95c0ad91bd490adf2b6e57495411a0f72fe50.camel@gmx.de","threadId":"56504","inReplyTo":"2722184.bRktqFsmb4@devpool47","subject":"Re: data loss when doing ls-remote and piped to command","fromName":"Mike Galbraith","fromEmail":"efault@gmx.de","sentAt":"2021-09-18T06:33:38Z","receivedAt":"2021-09-18T06:33:48Z","isPatch":false,"sender":{"key":"efault@gmx.de","avatar":null},"body":"On Fri, 2021-09-17 at 08:59 +0200, Rolf Eike Beer wrote:\n>\n> What you need is a _fast_ git server. kernel.org or github.com seem to be too\n> slow for this if you don't sit somewhere in their datacenter. Use something in\n> your local network, a Xeon E5 with lot's of RAM and connected with 1GBit/s\n> Ethernet in my case.\n\nEven faster: what's coming across that wire should be a constant (is?),\nvariable is only delivery/consumption jitter.  If there's really really\na pipe problem lurking, you should also be able to trigger by saving\nthe data once, and just catting it, letting interrupts etc provide\njitter.  Which stdout is left of '|' in a script shouldn't matter one\nwhit to the interpreter/kernel conversation, they're all the same.\n\nThat said, if I had a reproducer I was confident pointed to the kernel,\nI'd try to bisect.. boring as hell, but highly effective.\n\n\t-Mike\n"}]}