From: Felipe Contreras <felipe.contreras@gmail.com>
Date: Thu, 27 May 2021 19:19:00 GMT
Subject: Re: Runaway sed memory use in test on older sed+glibc (was "Re: [PATCH v6 1/3] test: add helper functions for git-bundle")
Message-ID: <60aff0a416f40_265302082c@natae.notmuch>
In-Reply-To: <87tumol4tg.fsf@evledraar.gmail.com>

Ævar Arnfjörð Bjarmason wrote:
> 
> On Thu, May 27 2021, Jiang Xin wrote:
> 
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> 于2021年5月27日周四
> > 上午2:51写道：
> >>
> >>
> >> On Mon, Jan 11 2021, Jiang Xin wrote:
> >>
> >> > From: Jiang Xin <zhiyou.jx@alibaba-inc.com>
> >> >
> >> > Move git-bundle related functions from t5510 to a library, and this
> >> > lib
> >> > will be shared with a new testcase t6020 which finds a known
> >> > breakage of
> >> > "git-bundle".
> >> > [...]
> >> > +
> >> > +# Format the output of git commands to make a user-friendly and
> >> > stable
> >> > +# text.  We can easily prepare the expect text without having to
> >> > worry
> >> > +# about future changes of the commit ID and spaces of the output.
> >> > +make_user_friendly_and_stable_output () {
> >> > +     sed \
> >> > +             -e "s/${A%${A#???????}}[0-9a-f]*/<COMMIT-A>/g" \
> >> > +             -e "s/${B%${B#???????}}[0-9a-f]*/<COMMIT-B>/g" \
> >> > +             -e "s/${C%${C#???????}}[0-9a-f]*/<COMMIT-C>/g" \
> >> > +             -e "s/${D%${D#???????}}[0-9a-f]*/<COMMIT-D>/g" \
> >> > +             -e "s/${E%${E#???????}}[0-9a-f]*/<COMMIT-E>/g" \
> >> > +             -e "s/${F%${F#???????}}[0-9a-f]*/<COMMIT-F>/g" \
> >> > +             -e "s/${G%${G#???????}}[0-9a-f]*/<COMMIT-G>/g" \
> >> > +             -e "s/${H%${H#???????}}[0-9a-f]*/<COMMIT-H>/g" \
> >> > +             -e "s/${I%${I#???????}}[0-9a-f]*/<COMMIT-I>/g" \
> >> > +             -e "s/${J%${J#???????}}[0-9a-f]*/<COMMIT-J>/g" \
> >> > +             -e "s/${K%${K#???????}}[0-9a-f]*/<COMMIT-K>/g" \
> >> > +             -e "s/${L%${L#???????}}[0-9a-f]*/<COMMIT-L>/g" \
> >> > +             -e "s/${M%${M#???????}}[0-9a-f]*/<COMMIT-M>/g" \
> >> > +             -e "s/${N%${N#???????}}[0-9a-f]*/<COMMIT-N>/g" \
> >> > +             -e "s/${O%${O#???????}}[0-9a-f]*/<COMMIT-O>/g" \
> >> > +             -e "s/${P%${P#???????}}[0-9a-f]*/<COMMIT-P>/g" \
> >> > +             -e "s/${TAG1%${TAG1#???????}}[0-9a-f]*/<TAG-1>/g" \
> >> > +             -e "s/${TAG2%${TAG2#???????}}[0-9a-f]*/<TAG-2>/g" \
> >> > +             -e "s/${TAG3%${TAG3#???????}}[0-9a-f]*/<TAG-3>/g" \
> >> > +             -e "s/ *\$//"
> >> > +}
> >>
> >> On one of the gcc farm boxes, a i386 box (gcc45) this fails because
> >> sed
> >> gets killed after >500MB of memory use (I was just eyeballing it in
> >> htop) on the "reate bundle from special rev: main^!" test. This with
> >> GNU
> >> sed 4.2.2.
> >>
> >> I suspect this regex pattern creates some runaway behavior in sed
> >> that's
> >> since been fixed (or maybe it's the glibc regex engine?). The glibc is
> >> 2.19-18+deb8u10:
> >>
> >>     + git bundle list-heads special-rev.bdl
> >>     + make_user_friendly_and_stable_output
> >>     + sed -e s/[0-9a-f]*/<COMMIT-A>/g -e s/[0-9a-f]*/<COMMIT-B>/g -e
> >> s/[0-9a-f]*/<COMMIT-C>/g -e s/[0-9a-f]*/<COMMIT-D>/g -e
> >> s/[0-9a-f]*/<COMMIT-E>/g -e s/[0-9a-f]*/<COMMIT-F>/g -e
> >> s/[0-9a-f]*/<COMMIT-G>/g -e s/[0-9a-f]*/<COMMIT-H>/g -e
> >> s/[0-9a-f]*/<COMMIT-I>/g -e s/[0-9a-f]*/<COMMIT-J>/g -e
> >> s/[0-9a-f]*/<COMMIT-K>/g -e s/[0-9a-f]*/<COMMIT-L>/g -e
> >> s/[0-9a-f]*/<COMMIT-M>/g -e s/[0-9a-f]*/<COMMIT-N>/g -e
> >> s/[0-9a-f]*/<COMMIT-O>/g -e s/[0-9a-f]*/<COMMIT-P>/g -e
> >> s/[0-9a-f]*/<TAG-1>/g -e s/[0-9a-f]*/<TAG-2>/g -e
> >> s/[0-9a-f]*/<TAG-3>/g -e s/ *$//
> >>     sed: couldn't re-allocate memory
> >
> > I wrote a program on macOS to check memory footprint for sed and perl.
> > See:
> >
> >     https://github.com/jiangxin/compare-sed-perl
> 
> Interesting use of Go for as a /usr/bin/time -v replacement :)

Here's a Ruby version:
https://dpaste.com/FYT2QKHJE

I'm not sure if will be useful in this particular case, but Ruby code
always ends up simpler ;)

-- 
Felipe Contreras