Re: [PATCH] pkt-line: re-'static'-ify buffer in packet_write_fmt_1()

From: Lars Schneider <larsxschneider@gmail.com>
Date: Aug 29, 2017, 17:51 UTC
Message-ID: <B1E291F2-86FF-4982-A092-92FAED65385C@gmail.com>
In-Reply-To: <CAN0heSoUqcOqVspZkbPahWQdtVpSdtSZoCFWu0ZQJfN3F0mD2g@mail.gmail.com>

Show 41 quoted lines

> On 28 Aug 2017, at 06:11, Martin Ågren <martin.agren@gmail.com> wrote:
> 
> On 28 August 2017 at 01:23, Jeff King <peff@peff.net> wrote:
>> On Sun, Aug 27, 2017 at 10:04:55PM +0200, Lars Schneider wrote:
>> 
>>> I did run all tests under valgrind using "make valgrind" and I found
>>> the following files with potential issues:
>>> 
>>> cat valgrind.out | perl -nE 'say /^==.+\((.+)\:.+\)$/' | sort | uniq -c
>>> 7102
>>>  2 clone.c
>>> 33 common-main.c
>>>  6 connect.c
>>> 64 git.c
>>>  4 ls-remote.c
>>> 126 run-command.c
>>> 12 transport.c
>>>  7 worktree.c
>> 
>> I'm not sure where valgrind.out comes from. The individual
>> test-results/*.out files may have valgrind output, but I don't think
>> they usually contain leak output.
>> 
>> Doing "valgrind ./git-upload-pack . </dev/null >/dev/null" mentions
>> leaked memory but not the locations. Adding --leak-check=full shows that
>> most of it comes from format_packet().
>> 
>> And applying Martin's patch drops the "definitely lost" category down to
>> 0 bytes (there's still 550k in "still reachable", but those are in the
>> "exit will free them for us" category).
>> 
>>> No mention of "pkt-line.c". Did you run Git with valgrind on one of
>>> your repositories to find it?
>> 
>> I'm curious, too. I don't think the valgrind setup in our test suite is
>> great for finding leaks right now.
> 
> Sorry for being brief. I've patched t/valgrind/valgrind.sh to say
> "--leak-check=yes". Then I run "./t0000 --valgrind", simply because
> running the complete suite gives more reports than I could possibly
> process.

I set $TOOL_OPTIONS in valgrind.sh: to '--leak-check=full --errors-for-leak-kinds=definite'

... but I also had to adjust t/test-lib-functions.sh:test_create_repo as I ran into the error "cannot run git init -- have you built things yet?".

With these changes I was able to see the leak running valgrind with t5110.

---

What if we run a few selected tests with valgrind and count all files that valgrind mentions (a single leak has multiple file mentions because of the stack trace and other leak indicators). We record these counts and let TravisCI scream if one of the numbers increases.

I wonder how stable/fragile such a metric would be as a simple refactoring could easily change these numbers. Below I ran valgrind on t5510 before and after Martin's patch. The diff below clearly shows the pkt-line leak.

Would it make sense to pursue something like this in TravisCI to avoid "pkt-line" kind of leaks in the future?

## Valgrind run with leak

$ ./t5510-fetch.sh --valgrind | perl -nE 'say /^==.+\((.+)\:.+\)$/' | sort | uniq -c
15529
  39 abspath.c
  30 add.c
  34 branch.c
  10 bundle.c
 268 clone.c
  13 commit.c
 471 common-main.c
  52 config.c
  50 connect.c
   4 connected.c
   1 diff-lib.c
 102 dir.c
 120 environment.c
   4 fetch-pack.c
  47 fetch.c
  14 files-backend.c
   1 git-compat-util.h
 871 git.c
  96 init-db.c
  26 iterator.c
   1 list-objects.c
   4 log-tree.c
   2 object.c
   1 pack-objects.c
   6 parse-options.c
  10 pathspec.c
  83 pkt-line.c
   6 precompose_utf8.c
   2 push.c
  67 refs.c
  96 remote.c
  90 repository.c
   4 rev-list.c
  10 revision.c
 165 run-command.c
  42 setup.c
 288 strbuf.c
  51 strbuf.h
   2 tag.c
 107 transport.c
  10 tree-diff.c
  85 upload-pack.c
   1 usage.c
 455 wrapper.c

## Valgrind run with Matrin's patch to fix the leak
$ ./t5510-fetch.sh --valgrind | perl -nE 'say /^==.+\((.+)\:.+\)$/' | sort | uniq -c
14931
  39 abspath.c
  30 add.c
  34 branch.c
  10 bundle.c
 268 clone.c
  13 commit.c
 433 common-main.c
  52 config.c
  50 connect.c
   6 connected.c
   1 diff-lib.c
 102 dir.c
 120 environment.c
   4 fetch-pack.c
  53 fetch.c
  14 files-backend.c
   1 git-compat-util.h
 879 git.c
  96 init-db.c
   1 iterator.c
   1 list-objects.c
   4 log-tree.c
   2 object.c
   1 pack-objects.c
   6 parse-options.c
  10 pathspec.c
   6 precompose_utf8.c
   2 push.c
  26 refs.c
  96 remote.c
  90 repository.c
   6 rev-list.c
  14 revision.c
 171 run-command.c
  42 setup.c
 246 strbuf.c
  50 strbuf.h
   2 tag.c
 107 transport.c
  10 tree-diff.c
   2 upload-pack.c
   1 usage.c
 415 wrapper.c

## Diff

15529                    | 14931 
  39 abspath.c               39 abspath.c
  30 add.c                   30 add.c
  34 branch.c                34 branch.c
  10 bundle.c                10 bundle.c
 268 clone.c                268 clone.c
  13 commit.c                13 commit.c
 471 common-main.c       |  433 common-main.c
  52 config.c                52 config.c
  50 connect.c               50 connect.c
   4 connected.c         |    6 connected.c
   1 diff-lib.c               1 diff-lib.c
 102 dir.c                  102 dir.c
 120 environment.c          120 environment.c
   4 fetch-pack.c             4 fetch-pack.c
  47 fetch.c             |   53 fetch.c
  14 files-backend.c         14 files-backend.c
   1 git-compat-util.h        1 git-compat-util.h
 871 git.c               |  879 git.c
  96 init-db.c               96 init-db.c
  26 iterator.c          |    1 iterator.c
   1 list-objects.c           1 list-objects.c
   4 log-tree.c               4 log-tree.c
   2 object.c                 2 object.c
   1 pack-objects.c           1 pack-objects.c
   6 parse-options.c          6 parse-options.c
  10 pathspec.c              10 pathspec.c
  83 pkt-line.c          <
   6 precompose_utf8.c        6 precompose_utf8.c
   2 push.c                   2 push.c
  67 refs.c              |   26 refs.c
  96 remote.c                96 remote.c
  90 repository.c            90 repository.c
   4 rev-list.c          |    6 rev-list.c
  10 revision.c          |   14 revision.c
 165 run-command.c       |  171 run-command.c
  42 setup.c                 42 setup.c
 288 strbuf.c            |  246 strbuf.c
  51 strbuf.h            |   50 strbuf.h
   2 tag.c                    2 tag.c
 107 transport.c            107 transport.c
  10 tree-diff.c             10 tree-diff.c
  85 upload-pack.c       |    2 upload-pack.c
   1 usage.c                  1 usage.c
 455 wrapper.c           |  415 wrapp

Thanks, Lars