Re: data loss when doing ls-remote and piped to command

From: Rolf Eike Beer <eb@emlix.com>
Date: Sep 16, 2021, 06:38 UTC
Message-ID: <2279155.Qy0YqsFniq@devpool47>
In-Reply-To: <xmqqwnnhwvnd.fsf@gitster.g>

Am Mittwoch, 15. September 2021, 20:17:42 CEST schrieb Junio C Hamano:

Show 23 quoted lines

> Rolf Eike Beer <eb@emlix.com> writes:
> > The given repository is a clone of the vanilla kernel.
> > 
> > /usr/bin/git --git-dir=/home/ebeer/repos/upstream/linux/.git ls-remote
> > origin 2>&1 | less
> > 
> > And I then see things like this:
> > 
> > 6f38b5d6cfd43dde3058a10c68baae9cf17af912        refs/tags/v5.0-rc2
> > 1c7fc5cbc33980acd13ae83d0b416db002fe95601e7f97f64b59514d936    
> > refs/tags/v5.7-rc2^{} d0709bb6da2ab6d49b11643e98abdf79b1a2817f       
> > refs/tags/v5.7-rc3
> Not offering any solution, just an observation of the problem and
> annotating the report.
> 
> What we see on the second line is the beginning of peeled
> v5.0-rc2^{} up to the "acd13" (that is, the first 19 bytes of the
> line), followed by the full line for peeled v5.7-rc2^{} (which
> begins with "ae83d").  12407 bytes in between are missing, which
> is even more puzzling as it is not a nice round number.
> 
> I wonder if this is "less" misconfigured and misbehaving.  Did the
> user after seeing v5.7-* tags scroll back with 'b' or something?

To quote myself:

>> This is not bound to less, we originally observed the error on a 
>> entirely different tool that tried to parse the output of ls-remote.

In fact when less opened I just started to scroll down until I visually noticed an error.

> If the output (including the 2>&1 redirection) is sent to a file and
> then "cat <that-file" is invoked, does the same thing happen?  How
> about "cat <that-file | less"?

The redirection seems to be an important part of it. I now did:

git ... 2>&1 | sha256sum

This gives different results basically on every run. I also noticed that having more tags makes it easier to reproduce, so a stable kernel in contrast to vanilla is a better trigger. Doing that without the stderr redirection gave the same result every time I tried.

Regards,

Eike

-- 
Rolf Eike Beer, emlix GmbH, http://www.emlix.com
Fon +49 551 30664-0, Fax +49 551 30664-11
Gothaer Platz 3, 37083 Göttingen, Germany
Sitz der Gesellschaft: Göttingen, Amtsgericht Göttingen HR B 3160
Geschäftsführung: Heike Jordan, Dr. Uwe Kracke – Ust-IdNr.: DE 205 198 055

emlix - smart embedded open source