{"thread":{"id":"55888","subject":"UNLEAK(), leak checking in the default tests etc.","startedAt":"2021-06-09T15:00:57Z","lastAt":"2022-02-16T17:45:14Z","messageCount":125,"participants":["Ævar Arnfjörð Bjarmason","Andrzej Hunt","Felipe Contreras","Jeff King","SZEDER Gábor","Eric Sunshine","Đoàn Trần Công Danh","Bruno Albuquerque","Junio C Hamano","Emily Shaffer","Carlo Marcelo Arenas Belón"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"426891","messageId":"87czsv2idy.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":null,"subject":"UNLEAK(), leak checking in the default tests etc.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-09T14:38:52Z","receivedAt":"2021-06-09T15:00:57Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\n[In-Reply-To\n,\nbut intentionally breaking threading for a new topic]\n\nOn Sun, Mar 14 2021, Andrzej Hunt via GitGitGadget wrote:\n\n> Most of these pointers can safely be freed when cmd_clone() completes,\n> therefore we make sure to free them. The one exception is that we\n> have to UNLEAK(repo) because it can point either to argv[0], or a\n> malloc'd string returned by absolute_pathdup().\n\nI ran into this when manually checking with valgrind and discovered that\nyou need SANITIZERS for -DSUPPRESS_ANNOTATED_LEAKS to squash it.\n\nI wonder if that shouldn't be in DEVOPTS (or even a default under\nDEVELOPER=1). I.e. you don't need any other special compile flags, just\na compiled git that you then run under valgrind to spot this.\n\n> builtin/clone.c | 14 ++++++++++----\n> 1 file changed, 10 insertions(+), 4 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 51e844a2de0a..952fe3d8fc88 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -964,10 +964,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> {\n> \tint is_bundle = 0, is_local;\n> \tconst char *repo_name, *repo, *work_tree, *git_dir;\n> -\tchar *path, *dir, *display_repo = NULL;\n> +\tchar *path = NULL, *dir, *display_repo = NULL;\n> \tint dest_exists, real_dest_exists = 0;\n> \tconst struct ref *refs, *remote_head;\n> -\tconst struct ref *remote_head_points_at;\n> +\tstruct ref *remote_head_points_at = NULL;\n> \tconst struct ref *our_head_points_at;\n> \tstruct ref *mapped_refs;\n> \tconst struct ref *ref;\n> @@ -1017,9 +1017,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \trepo_name = argv[0];\n> \n> \tpath = get_repo_path(repo_name, &is_bundle);\n> -\tif (path)\n> +\tif (path) {\n> +\t\tFREE_AND_NULL(path);\n> \t\trepo = absolute_pathdup(repo_name);\n> -\telse if (strchr(repo_name, ':')) {\n> +\t} else if (strchr(repo_name, ':')) {\n> \t\trepo = repo_name;\n> \t\tdisplay_repo = transport_anonymize_url(repo);\n> \t} else\n\nIn this case it seems better to just have a :\n\n int repo_heap = 0;\n\n Then set \"repo_heap = 1\" in that absolute_pathdup(repo_name) branch,\n and...\n\n> @@ -1393,6 +1394,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \tstrbuf_release(&reflog_msg);\n> \tstrbuf_release(&branch_top);\n> \tstrbuf_release(&key);\n> +\tfree_refs(mapped_refs);\n> +\tfree_refs(remote_head_points_at);\n> +\tfree(dir);\n> +\tfree(path);\n> +\tUNLEAK(repo);\n\nHere do:\n\n if (repo_heap)\n free(repo);\n\nBut maybe there's some other out of the box way to make leak checking\nJust Work without special flags in this case. I'm just noting this one\nbecause it ended up being the only one that leaked unless I compiled\nwith -DSUPPRESS_ANNOTATED_LEAKS. I was fixing some leaks in the bundle\ncode.\n\nAnyway, getting to the \"default tests\" point. I fixed a memory leak, and\nwanted to it tested that the specific command doesn't leak in git's\ndefault tests.\n\nDo we have such a thing, if not why not?\n\nThe closest I got to getting this was:\n\n GIT_VALGRIND_MODE=memcheck GIT_VALGRIND_OPTIONS=\"--leak-check=full --errors-for-leak-kinds=definite --error-exitcode=123\" --valgrind\n\nBut as t/README notes it implies --verbose so we can't currently run it\nunder the test harness (although I have out-of-tree patches to fix that\nin general).\n\nIt seems pretty straightforward to turn that specific thing into a test\nwith a prereq to detect if valgrind works in that mode at all, and then\ndo (in some dedicated test file):\n\n\t# Exit/skip if we can't setup valgrind, then setup relevant\n # valgrind options (maybe needing to re-source test-lib.sh, ew!)\n\ttest_expect_successs 'ls-heads should not leak' '\n\t\tgit bundle ls-heads a.bdl\n\t'\n\nBut from what I've found so far no such thing exists, and it seems to\nthe extent that this is checked it's run manually as a one-off (see git\nlog --grep=valgrind), but we don't explicitly test for this\nanywhere. Have I missed something?\n"},{"id":"426911","messageId":"fcb0eaee-6ae1-f2cc-51d5-103eea64532a@ahunt.org","threadId":"55888","inReplyTo":"87czsv2idy.fsf@evledraar.gmail.com","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-06-09T17:44:12Z","receivedAt":"2021-06-09T17:44:24Z","isPatch":false,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 09/06/2021 16:38, Ævar Arnfjörð Bjarmason wrote:\n> \n> [In-Reply-To\n> ,\n> but intentionally breaking threading for a new topic]\n> \n> On Sun, Mar 14 2021, Andrzej Hunt via GitGitGadget wrote:\n> \n>> Most of these pointers can safely be freed when cmd_clone() completes,\n>> therefore we make sure to free them. The one exception is that we\n>> have to UNLEAK(repo) because it can point either to argv[0], or a\n>> malloc'd string returned by absolute_pathdup().\n> \n> I ran into this when manually checking with valgrind and discovered that\n> you need SANITIZERS for -DSUPPRESS_ANNOTATED_LEAKS to squash it.\n> \n> I wonder if that shouldn't be in DEVOPTS (or even a default under\n> DEVELOPER=1). I.e. you don't need any other special compile flags, just\n> a compiled git that you then run under valgrind to spot this.\n\nI'm not familiar with git's development conventions/philosophy, but my \n2c is that it's better not to enable it by default in order to minimise \ndivergence from the code that users are running. OTOH it's not a major \ndifference in behaviour so perhaps that's not a concern here.\n\nMore significantly: I get the impression it's easier to do leak checking \nusing LSAN, which requires recompiling git anyway - at which point you \nget the flag for free - so how often will people actually perform leak \nchecking with Valgrind in the first place?\n\n> \n>> builtin/clone.c | 14 ++++++++++----\n>> 1 file changed, 10 insertions(+), 4 deletions(-)\n>>\n>> diff --git a/builtin/clone.c b/builtin/clone.c\n>> index 51e844a2de0a..952fe3d8fc88 100644\n>> --- a/builtin/clone.c\n>> +++ b/builtin/clone.c\n>> @@ -964,10 +964,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>> {\n>> \tint is_bundle = 0, is_local;\n>> \tconst char *repo_name, *repo, *work_tree, *git_dir;\n>> -\tchar *path, *dir, *display_repo = NULL;\n>> +\tchar *path = NULL, *dir, *display_repo = NULL;\n>> \tint dest_exists, real_dest_exists = 0;\n>> \tconst struct ref *refs, *remote_head;\n>> -\tconst struct ref *remote_head_points_at;\n>> +\tstruct ref *remote_head_points_at = NULL;\n>> \tconst struct ref *our_head_points_at;\n>> \tstruct ref *mapped_refs;\n>> \tconst struct ref *ref;\n>> @@ -1017,9 +1017,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>> \trepo_name = argv[0];\n>> \n>> \tpath = get_repo_path(repo_name, &is_bundle);\n>> -\tif (path)\n>> +\tif (path) {\n>> +\t\tFREE_AND_NULL(path);\n>> \t\trepo = absolute_pathdup(repo_name);\n>> -\telse if (strchr(repo_name, ':')) {\n>> +\t} else if (strchr(repo_name, ':')) {\n>> \t\trepo = repo_name;\n>> \t\tdisplay_repo = transport_anonymize_url(repo);\n>> \t} else\n> \n> In this case it seems better to just have a :\n> \n> int repo_heap = 0;\n> \n> Then set \"repo_heap = 1\" in that absolute_pathdup(repo_name) branch,\n> and...\n> \n>> @@ -1393,6 +1394,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>> \tstrbuf_release(&reflog_msg);\n>> \tstrbuf_release(&branch_top);\n>> \tstrbuf_release(&key);\n>> +\tfree_refs(mapped_refs);\n>> +\tfree_refs(remote_head_points_at);\n>> +\tfree(dir);\n>> +\tfree(path);\n>> +\tUNLEAK(repo);\n> \n> Here do:\n> \n> if (repo_heap)\n> free(repo);\n> \n\nAlthough this is possible, I don't think it's worth it: if UNLEAK \nalready exists, we might as well use it here to make the code simpler. \nAnd UNLEAK is unlikely to go away anytime soon because... (continued below)\n\n> But maybe there's some other out of the box way to make leak checking\n> Just Work without special flags in this case. I'm just noting this one\n> because it ended up being the only one that leaked unless I compiled\n> with -DSUPPRESS_ANNOTATED_LEAKS. I was fixing some leaks in the bundle\n> code.\n\nThere are trickier examples where a cmd_* function has a complex struct \non the stack, and correctly clearing all allocated memory pointed to by \nits members (or in turn further children with potentially multiple \nlevels of indirection) is a lot of work - and that work doesn't actually \nbenefit the user in any way. In other words, we either need to be able \nto use UNLEAK to suppress certain classes of uninteresting memory leaks \n- which allows us to focus on the interesting/real leaks - or someone \nhas to spend a lot of time doing cleanup by hand (and/or someone has to \nimplement a bunch of new cleanup functions)).\n\nIn your example above, the UNLEAK can be avoided at the cost of one \nadditional tracking variable - but in many other cases avoiding an \nUNLEAK is much more expensive. It's certainly valid to debate the merits \nof the UNLEAK here, but that won't remove the need for UNLEAK's \nexistence in general.\n\n(The most common example that I remember is where cmd_* has a rev_info, \nand AFAICT there's no one-liner to clean that up. Using UNLEAK is \nhonestly the best approach there. I don't think I've actually submitted \nany patches doing this, but I have a few in my local backlog.)\n> Anyway, getting to the \"default tests\" point. I fixed a memory leak, and\n> wanted to it tested that the specific command doesn't leak in git's\n> default tests.\n> \n> Do we have such a thing, if not why not?\n> \n> The closest I got to getting this was:\n> \n> GIT_VALGRIND_MODE=memcheck GIT_VALGRIND_OPTIONS=\"--leak-check=full --errors-for-leak-kinds=definite --error-exitcode=123\" --valgrind\n\nIt's easy to perform leak-checking runs *if* you're OK recompiling with \nLSAN, instead of using valgrind. My usual recipe for running against a \nrange of tests is something like:\n\n make SANITIZE=address,leak \nASAN_OPTIONS=\"detect_leaks=1:abort_on_error=1\" CFLAGS=\"-Og -g\" \nT=\"\\$(wildcard t00[0-9][0-9]-*.sh)\" test\n\nAdditionally: I usually specify CC=clang, although gcc+LSAN has mostly \nbeen stable enough in my experience so you might be able to skip that.\n(I've found ASAN+LSAN to be more stable than LSAN by itself, which is \nwhy I specify address+leak, but adding ASAN in turn requires overriding \nASAN_OPTIONS to reenable leak checking.)\n\nI don't know whether or not Valgrind is more/less effective at finding \nleaks, so being able to run the test suite under valgrind would be nice \nfor comparison purposes though.\n\nATB,\n\n Andrzej\n"},{"id":"426934","messageId":"60c126456351d_aab462081@natae.notmuch","threadId":"55888","inReplyTo":"fcb0eaee-6ae1-f2cc-51d5-103eea64532a@ahunt.org","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-06-09T20:36:21Z","receivedAt":"2021-06-09T20:36:40Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Andrzej Hunt wrote:\n> On 09/06/2021 16:38, Ævar Arnfjörð Bjarmason wrote:\n\n> > I wonder if that shouldn't be in DEVOPTS (or even a default under\n> > DEVELOPER=1). I.e. you don't need any other special compile flags, just\n> > a compiled git that you then run under valgrind to spot this.\n> \n> I'm not familiar with git's development conventions/philosophy, but my \n> 2c is that it's better not to enable it by default in order to minimise \n> divergence from the code that users are running.\n\nIt woudln't be on by default, you would need to turn it on with\n`make DEVEOPER=1`.\n\n-- \nFelipe Contreras"},{"id":"426978","messageId":"YMHtdYxNCf6DwUZG@coredump.intra.peff.net","threadId":"55888","inReplyTo":"fcb0eaee-6ae1-f2cc-51d5-103eea64532a@ahunt.org","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-10T10:46:13Z","receivedAt":"2021-06-10T10:46:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 09, 2021 at 07:44:12PM +0200, Andrzej Hunt wrote:\n\n> > I ran into this when manually checking with valgrind and discovered that\n> > you need SANITIZERS for -DSUPPRESS_ANNOTATED_LEAKS to squash it.\n> > \n> > I wonder if that shouldn't be in DEVOPTS (or even a default under\n> > DEVELOPER=1). I.e. you don't need any other special compile flags, just\n> > a compiled git that you then run under valgrind to spot this.\n> \n> I'm not familiar with git's development conventions/philosophy, but my 2c is\n> that it's better not to enable it by default in order to minimise divergence\n> from the code that users are running. OTOH it's not a major difference in\n> behaviour so perhaps that's not a concern here.\n\nYeah, I'd rather not enable the option during normal builds. It carries\na run-time penalty (it is actually building a pointless data structure\nthat _does_ effectively leak the pointers, but backed by a global so\nthey're \"findable\" by leak checkers). So it changes speed and possibly\ncorrectness of the final binary in a way that is different from what\npeople would actually run in practice.\n\nThat might be worth it if there was some advantage to just turning it\non (i.e., if by running with it all the time we might detect some bug).\nBut by itself it does nothing useful.\n\nIf you really want to leak-check more thoroughly the normal binary, then\nIMHO you'd be better off to convert UNLEAK() sites to actual free calls.\n\n> More significantly: I get the impression it's easier to do leak checking\n> using LSAN, which requires recompiling git anyway - at which point you get\n> the flag for free - so how often will people actually perform leak checking\n> with Valgrind in the first place?\n\nAnd yeah, I'd very much agree here. It's definitely not wrong to run\nwith Valgrind. But it's slower and much less thorough than ASan (probably not for\nleak detection, but definitely for bug-finding, since it can't look at\nstack variables).\n\nIf you do use it, and want to build with -DSUPPRESS_ANNOTATED_LEAKS all\nthe time, that's OK, but I don't think it makes sense for it to the\ndefault even under DEVELOPER=1. I'm not opposed to a patch to make it\neasier to flip the switch, though (but I also find sticking a line in\nyour config.mak to be pretty easy already).\n\n-Peff\n"},{"id":"426984","messageId":"87y2bi0vvl.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"fcb0eaee-6ae1-f2cc-51d5-103eea64532a@ahunt.org","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-10T10:56:55Z","receivedAt":"2021-06-10T12:04:46Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 09 2021, Andrzej Hunt wrote:\n\n> On 09/06/2021 16:38, Ævar Arnfjörð Bjarmason wrote:\n>> [In-Reply-To\n>> ,\n>> but intentionally breaking threading for a new topic]\n>> On Sun, Mar 14 2021, Andrzej Hunt via GitGitGadget wrote:\n>> \n>>> Most of these pointers can safely be freed when cmd_clone() completes,\n>>> therefore we make sure to free them. The one exception is that we\n>>> have to UNLEAK(repo) because it can point either to argv[0], or a\n>>> malloc'd string returned by absolute_pathdup().\n>> I ran into this when manually checking with valgrind and discovered\n>> that\n>> you need SANITIZERS for -DSUPPRESS_ANNOTATED_LEAKS to squash it.\n>> I wonder if that shouldn't be in DEVOPTS (or even a default under\n>> DEVELOPER=1). I.e. you don't need any other special compile flags, just\n>> a compiled git that you then run under valgrind to spot this.\n>\n> I'm not familiar with git's development conventions/philosophy, but my\n> 2c is that it's better not to enable it by default in order to\n> minimise divergence from the code that users are running. OTOH it's\n> not a major difference in behaviour so perhaps that's not a concern\n> here.\n>\n> More significantly: I get the impression it's easier to do leak\n> checking using LSAN, which requires recompiling git anyway - at which\n> point you get the flag for free - so how often will people actually\n> perform leak checking with Valgrind in the first place?\n\n*Nod*, I didn't investigate the runtime penalty you and Jeff point\nout. In any case, it seems that can also be done with valgrind exclusion\nrules and/or manually ignoring these cases in the test wrapper.\n\n>> \n>>> builtin/clone.c | 14 ++++++++++----\n>>> 1 file changed, 10 insertions(+), 4 deletions(-)\n>>>\n>>> diff --git a/builtin/clone.c b/builtin/clone.c\n>>> index 51e844a2de0a..952fe3d8fc88 100644\n>>> --- a/builtin/clone.c\n>>> +++ b/builtin/clone.c\n>>> @@ -964,10 +964,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>>> {\n>>> \tint is_bundle = 0, is_local;\n>>> \tconst char *repo_name, *repo, *work_tree, *git_dir;\n>>> -\tchar *path, *dir, *display_repo = NULL;\n>>> +\tchar *path = NULL, *dir, *display_repo = NULL;\n>>> \tint dest_exists, real_dest_exists = 0;\n>>> \tconst struct ref *refs, *remote_head;\n>>> -\tconst struct ref *remote_head_points_at;\n>>> +\tstruct ref *remote_head_points_at = NULL;\n>>> \tconst struct ref *our_head_points_at;\n>>> \tstruct ref *mapped_refs;\n>>> \tconst struct ref *ref;\n>>> @@ -1017,9 +1017,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>>> \trepo_name = argv[0];\n>>> \tpath = get_repo_path(repo_name, &is_bundle);\n>>> -\tif (path)\n>>> +\tif (path) {\n>>> +\t\tFREE_AND_NULL(path);\n>>> \t\trepo = absolute_pathdup(repo_name);\n>>> -\telse if (strchr(repo_name, ':')) {\n>>> +\t} else if (strchr(repo_name, ':')) {\n>>> \t\trepo = repo_name;\n>>> \t\tdisplay_repo = transport_anonymize_url(repo);\n>>> \t} else\n>> In this case it seems better to just have a :\n>> int repo_heap = 0;\n>> Then set \"repo_heap = 1\" in that absolute_pathdup(repo_name)\n>> branch,\n>> and...\n>> \n>>> @@ -1393,6 +1394,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>>> \tstrbuf_release(&reflog_msg);\n>>> \tstrbuf_release(&branch_top);\n>>> \tstrbuf_release(&key);\n>>> +\tfree_refs(mapped_refs);\n>>> +\tfree_refs(remote_head_points_at);\n>>> +\tfree(dir);\n>>> +\tfree(path);\n>>> +\tUNLEAK(repo);\n>> Here do:\n>> if (repo_heap)\n>> free(repo);\n>> \n>\n> Although this is possible, I don't think it's worth it: if UNLEAK\n> already exists, we might as well use it here to make the code\n> simpler. And UNLEAK is unlikely to go away anytime soon\n> because... (continued below)\n>\n>> But maybe there's some other out of the box way to make leak checking\n>> Just Work without special flags in this case. I'm just noting this one\n>> because it ended up being the only one that leaked unless I compiled\n>> with -DSUPPRESS_ANNOTATED_LEAKS. I was fixing some leaks in the bundle\n>> code.\n>\n> There are trickier examples where a cmd_* function has a complex\n> struct on the stack, and correctly clearing all allocated memory\n> pointed to by its members (or in turn further children with\n> potentially multiple levels of indirection) is a lot of work - and\n> that work doesn't actually benefit the user in any way. In other\n> words, we either need to be able to use UNLEAK to suppress certain\n> classes of uninteresting memory leaks - which allows us to focus on\n> the interesting/real leaks - or someone has to spend a lot of time\n> doing cleanup by hand (and/or someone has to implement a bunch of new\n> cleanup functions)).\n>\n> In your example above, the UNLEAK can be avoided at the cost of one\n> additional tracking variable - but in many other cases avoiding an \n> UNLEAK is much more expensive. It's certainly valid to debate the\n> merits of the UNLEAK here, but that won't remove the need for UNLEAK's \n> existence in general.\n>\n> (The most common example that I remember is where cmd_* has a\n> rev_info, and AFAICT there's no one-liner to clean that up. Using\n> UNLEAK is honestly the best approach there. I don't think I've\n> actually submitted any patches doing this, but I have a few in my\n> local backlog.)\n\nThe thing I was patching happened to be making rev_info * not leak. I\nprobably didn't cover some more complex cases, but some simple cases\nseem relatively easy.\n\nI.e. it just doesn't have a release() function, and at least the things\nI was looking at (bundle.c code) were relatively easy cases where we\nwere just missing a loop to free() data from some struct.\n\nBut yes, I agree that free()-ing just before we exit() is rather useless\nin itself, the reason I wanted it is because it's a useful (although not\nperfect) proxy for checking if the APIs the command uses as a one-off\nleak when used as libraries, where we may be processing N items, later\ndoing other work etc.\n\nWe should probably eventually have a s/free/end_free()/g and imitate\nperl(1)'s PERL_DESTRUCT_LEVEL option. I.e. you can globally configure\nperl to run in a mode that assumes a one-off command, in that case\nyou'll just let the OS handle the cleanup, or one where you care about\nmemory leaks because you're using it e.g. as an embedded library.\n\nBut maybe it's not even worth it. In Perl the main benefit is that it's\na programming language with DESTROY handlers etc., so destruction can\noften be expensive; turning it off entirely can also be buggy, imagine\nrelying on destructors to free temporary files etc.\n\nWe have that issue in theory with the interaction of atexit() handlers\nand e.g. things that would behave differently at a distance if certain\nthing were free()'d already, but in practice we probably don't.\n\nBut maybe it's not even worth pursuing. Have you (or anyone else) tried\ne.g. benchmarking git's tests or t/perf tests where free() is defined to\nbe some noop stub? I'd expect it not to matter, but maybe I'm wrong...\n\n>> Anyway, getting to the \"default tests\" point. I fixed a memory leak, and\n>> wanted to it tested that the specific command doesn't leak in git's\n>> default tests.\n>> Do we have such a thing, if not why not?\n>> The closest I got to getting this was:\n>> GIT_VALGRIND_MODE=memcheck\n>> GIT_VALGRIND_OPTIONS=\"--leak-check=full\n>> --errors-for-leak-kinds=definite --error-exitcode=123\" \n>> --valgrind\n>\n> It's easy to perform leak-checking runs *if* you're OK recompiling\n> with LSAN, instead of using valgrind. My usual recipe for running\n> against a range of tests is something like:\n\nI thought valgrind would be a better approach since we might rely on it\njust being there, so we could run some known-good commands that don't\nleak even in a \"normal\" test run, but...\n\n> make SANITIZE=address,leak\n> ASAN_OPTIONS=\"detect_leaks=1:abort_on_error=1\" CFLAGS=\"-Og -g\" \n> T=\"\\$(wildcard t00[0-9][0-9]-*.sh)\" test\n>\n> Additionally: I usually specify CC=clang, although gcc+LSAN has mostly\n> been stable enough in my experience so you might be able to skip that.\n> (I've found ASAN+LSAN to be more stable than LSAN by itself, which is\n> why I specify address+leak, but adding ASAN in turn requires\n> overriding ASAN_OPTIONS to reenable leak checking.)\n>\n> I don't know whether or not Valgrind is more/less effective at finding\n> leaks, so being able to run the test suite under valgrind would be\n> nice for comparison purposes though.\n\nI didn't know how to set that up, that seems easy enough.\n\nThis works for me:\n\n make CC=clang SANITIZE=address,leak CFLAGS=\"-00 -g\"\n (cd t && make ASAN_OPTIONS=\"\" [...])\n\nI.e. it's just SANITIZE & flags that's important at compile-time. You\ndoubtless knew that, mainly for my own notes & others following along.\n\nI ran it, noted the failing tests, produced a giant GIT_SKIP_TESTS list\nand hacked ci/ to run that as a new linux-clang-SANITIZE job. That messy\nWIP code is currently running at:\nhttps://github.com/avar/git/runs/2793150092\n\nWouldn't it be a good idea to have such a job and slowly work on the\nexclusion list?\n\nE.g. I saw that t0004 failed, which was trivially fixed with a single\nstrbuf_release(), and we could guard against regressions.\n\nAnyway, I can submit some cleaned-up patches for that. I was just\nfishing for whether there was some good reason not to do it, since there\nseemed to have been interest in leak fixes, but it hadn't made it into\nCI / some \"blessed\" GIT_TEST_* mode or whatever. I.e. maybe the reports\nwere unstable or unreliable...\n\n"},{"id":"427005","messageId":"YMIVzYgNddsR4FSd@coredump.intra.peff.net","threadId":"55888","inReplyTo":"87y2bi0vvl.fsf@evledraar.gmail.com","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-10T13:38:21Z","receivedAt":"2021-06-10T13:38:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 10, 2021 at 12:56:55PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > More significantly: I get the impression it's easier to do leak\n> > checking using LSAN, which requires recompiling git anyway - at which\n> > point you get the flag for free - so how often will people actually\n> > perform leak checking with Valgrind in the first place?\n> \n> *Nod*, I didn't investigate the runtime penalty you and Jeff point\n> out. In any case, it seems that can also be done with valgrind exclusion\n> rules and/or manually ignoring these cases in the test wrapper.\n\nI had trouble using valgrind's exclusions; there's more discussion in\n0e5bba53af (add UNLEAK annotation for reducing leak false positives,\n2017-09-08), but the gist of it is that it's awkward to annotate the\npoint of leak, rather than the point of allocation (so you have to\nprovide the complete callstack to the allocation, which is a maintenance\nheadache).\n\nOf course, if you find ways to make useful annotations with valgrind,\nI'm all for it. We have a few in t/valgrind already.\n\n> But maybe it's not even worth pursuing. Have you (or anyone else) tried\n> e.g. benchmarking git's tests or t/perf tests where free() is defined to\n> be some noop stub? I'd expect it not to matter, but maybe I'm wrong...\n\nI haven't. Even though I originated UNLEAK(), I'm not really all that\nconcerned about the cost of free() in general. My motivation for\nintroducing it (versus adding free() calls) was mostly about convenience\n(complex data structures that don't have an easy free/release function,\nbut also the fact that you can still access data after marking it with\nunleak).\n\nThe fact that it also preempts any arguments about the performance of\ncalling free() was just a bonus. ;)\n\nTo be clear, I could easily be convinced by real numbers that the cost\nof free() at program end matters. I am just saying I am not one of the\npeople who is going to argue that position in the meantime.\n\n> I didn't know how to set that up, that seems easy enough.\n> \n> This works for me:\n> \n> make CC=clang SANITIZE=address,leak CFLAGS=\"-00 -g\"\n> (cd t && make ASAN_OPTIONS=\"\" [...])\n> \n> I.e. it's just SANITIZE & flags that's important at compile-time. You\n> doubtless knew that, mainly for my own notes & others following along.\n\nIt should Just Work with:\n\n make SANITIZE=leak test\n\nfor both gcc and clang. You do need ASAN_OPTIONS if you're asking ASan\nto do leak-checking (since we usually suppress that for the obvious\nreason that almost every test fails). I'm not sure if using both ASan\nand LSan together confuses LSan there (if so, it may be reasonable for\ntest-lib.sh to modify its ASAN_OPTIONS setting if LSan is enabled).\n\n> I ran it, noted the failing tests, produced a giant GIT_SKIP_TESTS list\n> and hacked ci/ to run that as a new linux-clang-SANITIZE job. That messy\n> WIP code is currently running at:\n> https://github.com/avar/git/runs/2793150092\n> \n> Wouldn't it be a good idea to have such a job and slowly work on the\n> exclusion list?\n> \n> E.g. I saw that t0004 failed, which was trivially fixed with a single\n> strbuf_release(), and we could guard against regressions.\n\nI don't mind that. My intent was to get the whole suite clean\neventually, and then start worrying about regressions. But that may take\na while.\n\nI do think it would be worth splitting out ASan from leak-checking. The\nwhole suite should run clean with regular ASan already, and we'd want to\nfind regressions there even in the tests that aren't leak-clean. I do\nperiodic ASan runs already; the main argument against doing it for every\nCI run is just that's a lot more CPU. But maybe not enough to be\nprohibitive? It's probably still way cheaper than running the test suite\non Windows.\n\n-Peff\n"},{"id":"427028","messageId":"bd212451-d8f0-e041-3460-bbbff57542d8@ahunt.org","threadId":"55888","inReplyTo":"YMIVzYgNddsR4FSd@coredump.intra.peff.net","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-06-10T15:32:41Z","receivedAt":"2021-06-10T15:32:57Z","isPatch":false,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 10/06/2021 15:38, Jeff King wrote:\n>> I ran it, noted the failing tests, produced a giant GIT_SKIP_TESTS list\n>> and hacked ci/ to run that as a new linux-clang-SANITIZE job. That messy\n>> WIP code is currently running at:\n>> https://github.com/avar/git/runs/2793150092\n>>\n>> Wouldn't it be a good idea to have such a job and slowly work on the\n>> exclusion list?\n>>\n>> E.g. I saw that t0004 failed, which was trivially fixed with a single\n>> strbuf_release(), and we could guard against regressions.\n> \n> I don't mind that. My intent was to get the whole suite clean\n> eventually, and then start worrying about regressions. But that may take\n> a while.\n> \n> I do think it would be worth splitting out ASan from leak-checking. The\n> whole suite should run clean with regular ASan already, and we'd want to\n> find regressions there even in the tests that aren't leak-clean. I do\n> periodic ASan runs already; the main argument against doing it for every\n> CI run is just that's a lot more CPU. But maybe not enough to be\n> prohibitive? It's probably still way cheaper than running the test suite\n> on Windows.\n\nI've been running tests with ASAN in the Github Actions environment, and \na single run takes just over 30 minutes [1] - which I believe is similar \nto the normal test jobs (they do run the test suite twice in that time I \nthink).\n\nI've been doing the same with UBSAN, and that's even faster at 15-20 \nminutes [2]. However I get the impression that ASAN issues are both more \ncommon (at least on seen), and more impactful - so I would argue that \nASAN should be prioritised if there's spare capacity. (I have no idea if \nASAN+UBSAN can be combined, but I suspect that doing so would make the \ntests slower?)\n\nI'm also running LSAN tests in CI to try and catch regressions, but I've \nonly enabled a handful of tests so far. My much simpler approach was to \nspecify the range of tests to run as 0-X, and as we make progress on \nfixing leaks, X will slowly approach 9999 (currently we're at something \nlike X~=5, although I'm not too far off sending out some patches to \nboost that to 99). The skip-tests approach seems much more useful!\n\nATB,\n\n Andrzej\n\n[1] https://github.com/ahunt/git/runs/2789921851?check_suite_focus=true\n[2] https://github.com/ahunt/git/runs/2760632000?check_suite_focus=true\n\n"},{"id":"427038","messageId":"YMI/g1sHxJgb8/YD@coredump.intra.peff.net","threadId":"55888","inReplyTo":"bd212451-d8f0-e041-3460-bbbff57542d8@ahunt.org","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-10T16:36:19Z","receivedAt":"2021-06-10T16:36:23Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 10, 2021 at 05:32:41PM +0200, Andrzej Hunt wrote:\n\n> > I do think it would be worth splitting out ASan from leak-checking. The\n> > whole suite should run clean with regular ASan already, and we'd want to\n> > find regressions there even in the tests that aren't leak-clean. I do\n> > periodic ASan runs already; the main argument against doing it for every\n> > CI run is just that's a lot more CPU. But maybe not enough to be\n> > prohibitive? It's probably still way cheaper than running the test suite\n> > on Windows.\n> \n> I've been running tests with ASAN in the Github Actions environment, and a\n> single run takes just over 30 minutes [1] - which I believe is similar to\n> the normal test jobs (they do run the test suite twice in that time I\n> think).\n> \n> I've been doing the same with UBSAN, and that's even faster at 15-20 minutes\n> [2]. However I get the impression that ASAN issues are both more common (at\n> least on seen), and more impactful - so I would argue that ASAN should be\n> prioritised if there's spare capacity. (I have no idea if ASAN+UBSAN can be\n> combined, but I suspect that doing so would make the tests slower?)\n\nI routinely do SANITIZE=address,undefined since they are both useful\n(and we do not trigger either in the current test suite). I never\nmeasured the time of their combined use versus just one, but surely it's\nfaster the two-at-once approach is faster than running the test suite\ntwice.\n\n> I'm also running LSAN tests in CI to try and catch regressions, but I've\n> only enabled a handful of tests so far. My much simpler approach was to\n> specify the range of tests to run as 0-X, and as we make progress on fixing\n> leaks, X will slowly approach 9999 (currently we're at something like X~=5,\n> although I'm not too far off sending out some patches to boost that to 99).\n> The skip-tests approach seems much more useful!\n\nDepending how fine-grained you get with skip-tests, it can create a\nhassle as individual tests are removed or reordered (and now somebody\nhas to maintain the skip list). Doing it with whole scripts (whether\nsaying \"these ones are OK\" or \"these ones are known bad\") seems like\nless maintenance overall. The results aren't as fine-grained, but for\nsomething that is meant to be a transitional step, I'm not sure it's\nworth the trouble to get more specific.\n\n-Peff\n"},{"id":"427055","messageId":"20210610190121.GD6312@szeder.dev","threadId":"55888","inReplyTo":"87czsv2idy.fsf@evledraar.gmail.com","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-10T19:01:21Z","receivedAt":"2021-06-10T19:01:32Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 09, 2021 at 04:38:52PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> GIT_VALGRIND_MODE=memcheck GIT_VALGRIND_OPTIONS=\"--leak-check=full --errors-for-leak-kinds=definite --error-exitcode=123\" --valgrind\n> \n> But as t/README notes it implies --verbose so we can't currently run it\n> under the test harness (although I have out-of-tree patches to fix that\n> in general).\n\n'--valgrind' doesn't imply '--verbose' if '--verbose-log' was given,\nand that works with the test harness just fine; see 88c6e9d31c\n(test-lib: --valgrind should not override --verbose-log, 2017-09-05).\n\n"},{"id":"427118","messageId":"18238547-0fbb-a9c4-a769-7e6d865b171a@ahunt.org","threadId":"55888","inReplyTo":"YMI/g1sHxJgb8/YD@coredump.intra.peff.net","subject":"Re: UNLEAK(), leak checking in the default tests etc.","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-06-11T15:44:09Z","receivedAt":"2021-06-11T15:44:19Z","isPatch":false,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 10/06/2021 18:36, Jeff King wrote:\n> On Thu, Jun 10, 2021 at 05:32:41PM +0200, Andrzej Hunt wrote:\n> \n>>> I do think it would be worth splitting out ASan from leak-checking. The\n>>> whole suite should run clean with regular ASan already, and we'd want to\n>>> find regressions there even in the tests that aren't leak-clean. I do\n>>> periodic ASan runs already; the main argument against doing it for every\n>>> CI run is just that's a lot more CPU. But maybe not enough to be\n>>> prohibitive? It's probably still way cheaper than running the test suite\n>>> on Windows.\n>>\n>> I've been running tests with ASAN in the Github Actions environment, and a\n>> single run takes just over 30 minutes [1] - which I believe is similar to\n>> the normal test jobs (they do run the test suite twice in that time I\n>> think).\n>>\n>> I've been doing the same with UBSAN, and that's even faster at 15-20 minutes\n>> [2]. However I get the impression that ASAN issues are both more common (at\n>> least on seen), and more impactful - so I would argue that ASAN should be\n>> prioritised if there's spare capacity. (I have no idea if ASAN+UBSAN can be\n>> combined, but I suspect that doing so would make the tests slower?)\n> \n> I routinely do SANITIZE=address,undefined since they are both useful\n> (and we do not trigger either in the current test suite). I never\n> measured the time of their combined use versus just one, but surely it's\n> faster the two-at-once approach is faster than running the test suite\n> twice.\n\nI'm seeing 33 minutes for SANITIZE=address,undefined - which is no \nslower than SANITIZE=address by itself (disclaimer: it's only one \nmeasurement):\nhttps://github.com/ahunt/git/runs/2795642716?check_suite_focus=true\n(The job's name is wrong but if you look in the logs you can confirm \nthat it's using address+undefined.)\n\nThe usual linux and mac test-jobs are actually from 24 to 30 minutes \n(the numbers seem a bit variable) - with the exception of one faster 10 \nminute job:\nhttps://github.com/git/git/actions/runs/925771097\nvs\nhttps://github.com/git/git/actions/runs/927729395\n\nSo to summarise: adding an ASAN+UBSAN job would make things a bit \nslower, but not a huge amount slower.\n"},{"id":"429957","messageId":"cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com","threadId":"55888","inReplyTo":"87czsv2idy.fsf@evledraar.gmail.com","subject":"[PATCH 0/4] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T00:11:45Z","receivedAt":"2021-07-14T00:11:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As a follow-up to my recent thread asking if we had some test mode or\nCI to test for memory leak regression (we don't), add such a test\nmode, and run it in CI.\n\nCurrently the two new CI targets take ~2-3 minutes to run in GitHub\nCI, whereas the normal test targets take 20-30 minutes. The tests run\nslower, but we have a small whitelist of test scripts that are OK.\n\n1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n\nÆvar Arnfjörð Bjarmason (4):\n tests: add a test mode for SANITIZE=leak, run it in CI\n SANITIZE tests: fix memory leaks in t13*config*, add to whitelist\n SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n SANITIZE tests: fix leak in mailmap.c\n\n .github/workflows/main.yml | 6 ++++\n Makefile | 5 +++\n ci/install-dependencies.sh | 4 +--\n ci/lib.sh | 18 ++++++++---\n ci/run-build-and-tests.sh | 4 +--\n config.c | 17 ++++++++---\n mailmap.c | 2 ++\n protocol-caps.c | 5 +--\n t/README | 16 ++++++++++\n t/t0500-progress-display.sh | 3 +-\n t/t1300-config.sh | 16 ++++++----\n t/t4203-mailmap.sh | 6 ++++\n t/t5701-git-serve.sh | 3 +-\n t/test-lib.sh | 61 +++++++++++++++++++++++++++++++++++++\n 14 files changed, 142 insertions(+), 24 deletions(-)\n\n-- \n2.32.0-dev\n\n"},{"id":"429958","messageId":"patch-1.4-a61a294132-20210714T001007Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com","subject":"[PATCH 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T00:11:46Z","receivedAt":"2021-07-14T00:11:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak there has been no\ncorresponding GIT_TEST_* mode for it, i.e. memory leaks have been\nfixed as one-offs without structured regression testing.\n\nThis change add such a mode, we now have new\nlinux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\ntests as linux-{clang,gcc}, except that almost all of them are\nskipped.\n\nThere is a whitelist of some tests that are OK in test-lib.sh, and\nindividual tests can be opted-in by setting\nGIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\nindividual test can be skipped with the \"!SANITIZE_LEAK\"\nprerequisite. See the updated t/README for more details.\n\nI'm using the GIT_TEST_SANITIZE_LEAK=true and !SANITIZE_LEAK pattern\nin a couple of tests whose memory leaks I'll fix in subsequent\ncommits.\n\nI'm not being aggressive about opting in tests, it's not all tests\nthat currently pass under SANITIZE=leak, just a small number of\nknown-good tests. We can add more later as we fix leaks and grow more\nconfident in this test mode.\n\nSee the recent discussion at [1] about the lack of this sort of test\nmode, and 0e5bba53af (add UNLEAK annotation for reducing leak false\npositives, 2017-09-08) for the initial addition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nWhen calling maybe_skip_all_sanitize_leak matching against\n\"$TEST_NAME\" instead of \"$this_test\" as other \"match_pattern_list()\"\nusers do is intentional. I'd like to match things like \"t13*config*\"\nin subsequent commits. This part of the API isn't public, so we can\nfreely change it in the future.\n\n1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 6 ++++\n Makefile | 5 ++++\n ci/install-dependencies.sh | 4 +--\n ci/lib.sh | 18 +++++++----\n ci/run-build-and-tests.sh | 4 +--\n t/README | 16 ++++++++++\n t/t0500-progress-display.sh | 3 +-\n t/t5701-git-serve.sh | 2 +-\n t/test-lib.sh | 60 +++++++++++++++++++++++++++++++++++++\n 9 files changed, 107 insertions(+), 11 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 73856bafc9..b81ec34959 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -297,6 +297,12 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-clang-sanitize-leak\n+ cc: clang\n+ pool: ubuntu-latest\n+ - jobname: linux-gcc-sanitize-leak\n+ cc: clang\n+ pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/Makefile b/Makefile\nindex 502e0c9a81..d4cad5136f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1216,6 +1216,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1260,6 +1263,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2793,6 +2797,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 67852d0d37..31e519cde9 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang*|linux-gcc*)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc*)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f..34fd914438 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,14 +183,16 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n-\tif [ \"$jobname\" = linux-gcc ]\n-\tthen\n+linux-clang*|linux-gcc*)\n+\tcase \"$jobname\" in\n+\tlinux-gcc*)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \texport GIT_TEST_HTTPD=true\n \n@@ -233,4 +235,10 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-*-sanitize-leak)\n+\texport SANITIZE=leak\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..07b9c09f45 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -12,7 +12,7 @@ esac\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc*)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n@@ -29,7 +29,7 @@ linux-gcc)\n \texport GIT_TEST_CHECKOUT_WORKERS=2\n \tmake test\n \t;;\n-linux-clang)\n+linux-clang*)\n \texport GIT_TEST_DEFAULT_HASH=sha1\n \tmake test\n \texport GIT_TEST_DEFAULT_HASH=sha256\ndiff --git a/t/README b/t/README\nindex 1a2072b2c8..303d0be817 100644\n--- a/t/README\n+++ b/t/README\n@@ -448,6 +448,22 @@ GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n to and 'checkout.thresholdForParallelism' to 0, forcing the\n execution of the parallel-checkout code.\n \n+GIT_TEST_SANITIZE_LEAK= will force the tests to run when git\n+is compiled with SANITIZE=leak (we pick it up via\n+../GIT-BUILD-OPTIONS).\n+\n+By default all tests are skipped when compiled with SANITIZE=leak, and\n+individual test scripts opt themselves in to leak testing by setting\n+GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n+tests use the SANITIZE_LEAK prerequisite to skip individiual tests\n+(i.e. test_expect_success !SANITIZE_LEAK [...]).\n+\n+So the GIT_TEST_SANITIZE_LEAK setting is different in behavior from\n+both other GIT_TEST_*=[true|false] settings, but more useful given how\n+SANITIZE=leak works & the state of the test suite. Manually setting\n+GIT_TEST_SANITIZE_LEAK=true is only useful during development when\n+finding and fixing memory leaks.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503a..7afb9abb1f 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -2,6 +2,7 @@\n \n test_description='progress display'\n \n+GIT_TEST_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n show_cr () {\n@@ -283,7 +284,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n-test_expect_success 'progress generates traces' '\n+test_expect_success !SANITIZE_LEAK 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tthroughput 102400 1000\n \tupdate\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 930721f053..d58efb0aa9 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -243,7 +243,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n \n # Test the basics of object-info\n #\n-test_expect_success 'basics of object-info' '\n+test_expect_success !SANITIZE_LEAK 'basics of object-info' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=object-info\n \tobject-format=$(test_oid algo)\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 7036f83b33..9201510e16 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1353,6 +1353,40 @@ then\n \texit 1\n fi\n \n+# SANITIZE=leak test mode\n+sanitize_leak_true=\n+add_sanitize_leak_true () {\n+\tsanitize_leak_true=\"$sanitize_leak_true$1 \"\n+}\n+\n+sanitize_leak_false=\n+add_sanitize_leak_false () {\n+\tsanitize_leak_false=\"$sanitize_leak_false$1 \"\n+}\n+\n+sanitize_leak_opt_in_msg=\"opt-in with GIT_TEST_SANITIZE_LEAK=true\"\n+maybe_skip_all_sanitize_leak () {\n+\t# Whitelist patterns\n+\tadd_sanitize_leak_true 't000*'\n+\tadd_sanitize_leak_true 't001*'\n+\tadd_sanitize_leak_true 't006*'\n+\n+\t# Blacklist patterns (overrides whitelist)\n+\tadd_sanitize_leak_false 't000[469]*'\n+\tadd_sanitize_leak_false 't001[2459]*'\n+\tadd_sanitize_leak_false 't006[0248]*'\n+\n+\tif match_pattern_list \"$1\" \"$sanitize_leak_false\"\n+\tthen\n+\t\tskip_all=\"test $this_test on SANITIZE=leak blacklist, $sanitize_leak_opt_in_msg\"\n+\t\ttest_done\n+\telif match_pattern_list \"$1\" \"$sanitize_leak_true\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\treturn 1\n+}\n+\n # Are we running this test at all?\n remove_trash=\n this_test=${0##*/}\n@@ -1364,6 +1398,31 @@ then\n \ttest_done\n fi\n \n+# Aggressively skip non-whitelisted tests when compiled with\n+# SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test -z \"$GIT_TEST_SANITIZE_LEAK\" &&\n+\t\tmaybe_skip_all_sanitize_leak \"$TEST_NAME\"\n+\tthen\n+\t\tsay_color info >&3 \"test $this_test on SANITIZE=leak whitelist\"\n+\t\tGIT_TEST_SANITIZE_LEAK=true\n+\tfi\n+\n+\t# We need to see it in \"git env--helper\" (via\n+\t# test_bool_env)\n+\texport GIT_TEST_SANITIZE_LEAK\n+\n+\tif ! test_bool_env GIT_TEST_SANITIZE_LEAK false\n+\tthen\n+\t\tskip_all=\"skip all tests in $this_test under SANITIZE=leak, $sanitize_leak_opt_in_msg\"\n+\t\ttest_done\n+\tfi\n+elif test_bool_env GIT_TEST_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n@@ -1516,6 +1575,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.32.0-dev\n\n"},{"id":"429960","messageId":"patch-2.4-f3a5f26366-20210714T001007Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com","subject":"[PATCH 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T00:11:47Z","receivedAt":"2021-07-14T00:11:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a couple of trivial memory leaks introduced in 3efd0bedc6 (config:\nadd conditional include, 2017-03-01) and my own 867ad08a26 (hooks:\nallow customizing where the hook directory is, 2016-05-04).\n\nIn the latter case the \"fix\" is UNLEAK() on the global variable. This\nallows us to run all t13*config* tests under SANITIZE=leak.\n\nWith this change we can now run almost the whole set of config.c\ntests (t13*config) under SANITIZE=leak, so let's do so, with a few\nexceptions:\n\n * The test added in ce81b1da23 (config: add new way to pass config\n via `--config-env`, 2021-01-12), it fails in GitHub CI, but passes\n for me locally. Let's just skip it for now.\n\n * Ditto the split_cmdline and \"aliases of builtins\" tests, the former\n required splitting up an existing test, there an issue with the test\n that would have also been revealed by skipping it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n config.c | 17 ++++++++++++-----\n t/t1300-config.sh | 16 ++++++++++------\n t/test-lib.sh | 1 +\n 3 files changed, 23 insertions(+), 11 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex f9c400ad30..38e132c0e2 100644\n--- a/config.c\n+++ b/config.c\n@@ -138,8 +138,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n \t\treturn config_error_nonbool(\"include.path\");\n \n \texpanded = expand_user_path(path, 0);\n-\tif (!expanded)\n-\t\treturn error(_(\"could not expand include path '%s'\"), path);\n+\tif (!expanded) {\n+\t\tret = error(_(\"could not expand include path '%s'\"), path);\n+\t\tgoto cleanup;\n+\t}\n \tpath = expanded;\n \n \t/*\n@@ -149,8 +151,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n \tif (!is_absolute_path(path)) {\n \t\tchar *slash;\n \n-\t\tif (!cf || !cf->path)\n-\t\t\treturn error(_(\"relative config includes must come from files\"));\n+\t\tif (!cf || !cf->path) {\n+\t\t\tret = error(_(\"relative config includes must come from files\"));\n+\t\t\tgoto cleanup;\n+\t\t}\n \n \t\tslash = find_last_dir_sep(cf->path);\n \t\tif (slash)\n@@ -168,6 +172,7 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n \t\tret = git_config_from_file(git_config_include, path, inc);\n \t\tinc->depth--;\n \t}\n+cleanup:\n \tstrbuf_release(&buf);\n \tfree(expanded);\n \treturn ret;\n@@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(var, \"core.attributesfile\"))\n \t\treturn git_config_pathname(&git_attributes_file, var, value);\n \n-\tif (!strcmp(var, \"core.hookspath\"))\n+\tif (!strcmp(var, \"core.hookspath\")) {\n+\t\tUNLEAK(git_hooks_path);\n \t\treturn git_config_pathname(&git_hooks_path, var, value);\n+\t}\n \n \tif (!strcmp(var, \"core.bare\")) {\n \t\tis_bare_repository_cfg = git_config_bool(var, value);\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 9ff46f3b04..93ad0f4887 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1050,12 +1050,16 @@ test_expect_success SYMLINKS 'symlink to nonexistent configuration' '\n \ttest_must_fail git config --file=linktolinktonada --list\n '\n \n-test_expect_success 'check split_cmdline return' \"\n-\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n-\ttest_must_fail git split-cmdline-fix &&\n+test_expect_success 'setup check split_cmdline return' \"\n \techo foo > foo &&\n \tgit add foo &&\n-\tgit commit -m 'initial commit' &&\n+\tgit commit -m 'initial commit'\n+\"\n+\n+test_expect_success !SANITIZE_LEAK 'check split_cmdline return' \"\n+\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n+\ttest_must_fail git split-cmdline-fix &&\n+\n \tgit config branch.main.mergeoptions 'echo \\\"' &&\n \ttest_must_fail git merge main\n \"\n@@ -1101,7 +1105,7 @@ test_expect_success 'key sanity-checking' '\n \tgit config foo.\"ba =z\".bar false\n '\n \n-test_expect_success 'git -c works with aliases of builtins' '\n+test_expect_success !SANITIZE_LEAK 'git -c works with aliases of builtins' '\n \tgit config alias.checkconfig \"-c foo.check=bar config foo.check\" &&\n \techo bar >expect &&\n \tgit checkconfig >actual &&\n@@ -1397,7 +1401,7 @@ test_expect_success 'git --config-env with missing value' '\n \tgrep \"invalid config format: config\" error\n '\n \n-test_expect_success 'git --config-env fails with invalid parameters' '\n+test_expect_success !SANITIZE_LEAK 'git --config-env fails with invalid parameters' '\n \ttest_must_fail git --config-env=foo.flag config --bool foo.flag 2>error &&\n \ttest_i18ngrep \"invalid config format: foo.flag\" error &&\n \ttest_must_fail git --config-env=foo.flag= config --bool foo.flag 2>error &&\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9201510e16..98e20950c3 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1370,6 +1370,7 @@ maybe_skip_all_sanitize_leak () {\n \tadd_sanitize_leak_true 't000*'\n \tadd_sanitize_leak_true 't001*'\n \tadd_sanitize_leak_true 't006*'\n+\tadd_sanitize_leak_true 't13*config*'\n \n \t# Blacklist patterns (overrides whitelist)\n \tadd_sanitize_leak_false 't000[469]*'\n-- \n2.32.0-dev\n\n"},{"id":"429959","messageId":"patch-3.4-2aeb3e8038-20210714T001007Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com","subject":"[PATCH 3/4] SANITIZE tests: fix memory leaks in t5701*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T00:11:48Z","receivedAt":"2021-07-14T00:11:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in a2ba162cda (object-info: support for retrieving\nobject info, 2021-04-20) which appears to have been based on a\nmisunderstanding of how the pkt-line.c API works, there is no need to\nstrdup() input to, it's just a printf()-like format function.\n\nThis fixes a potentially large memory leak, since the number of OID\nlines the \"object-info\" call can be arbitrarily large (or a small one\nif the request is small).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n protocol-caps.c | 5 +++--\n t/t5701-git-serve.sh | 1 +\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex 13a9e63a04..901b6795e4 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -69,9 +69,10 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpacket_writer_write(writer, \"%s\",\n-\t\t\t\t strbuf_detach(&send_buffer, NULL));\n+\t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n+\t\tstrbuf_reset(&send_buffer);\n \t}\n+\tstrbuf_release(&send_buffer);\n }\n \n int cap_object_info(struct repository *r, struct strvec *keys,\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex d58efb0aa9..e2f4832adf 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -5,6 +5,7 @@ test_description='test protocol v2 server commands'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'test capability advertisement' '\n-- \n2.32.0-dev\n\n"},{"id":"429961","messageId":"patch-4.4-b8062a09f9-20210714T001007Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com","subject":"[PATCH 4/4] SANITIZE tests: fix leak in mailmap.c","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T00:11:49Z","receivedAt":"2021-07-14T00:12:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Get closer to being able to run t4203-mailmap.sh by fixing a couple of\nmemory leak in mailmap.c.\n\nIn the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\nand clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\nclear the \"me\" structure, but while we freed parts of the\nmailmap_entry structure, we didn't free the structure itself. The same\ngoes for the \"mailmap_info\" structure.\n---\n mailmap.c | 2 ++\n t/t4203-mailmap.sh | 6 ++++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/mailmap.c b/mailmap.c\nindex d1f7c0d272..e1c8736093 100644\n--- a/mailmap.c\n+++ b/mailmap.c\n@@ -36,6 +36,7 @@ static void free_mailmap_info(void *p, const char *s)\n \t\t s, debug_str(mi->name), debug_str(mi->email));\n \tfree(mi->name);\n \tfree(mi->email);\n+\tfree(mi);\n }\n \n static void free_mailmap_entry(void *p, const char *s)\n@@ -51,6 +52,7 @@ static void free_mailmap_entry(void *p, const char *s)\n \n \tme->namemap.strdup_strings = 1;\n \tstring_list_clear_func(&me->namemap, free_mailmap_info);\n+\tfree(me);\n }\n \n /*\ndiff --git a/t/t4203-mailmap.sh b/t/t4203-mailmap.sh\nindex 0b2d21ec55..c7de4299cf 100755\n--- a/t/t4203-mailmap.sh\n+++ b/t/t4203-mailmap.sh\n@@ -79,6 +79,12 @@ test_expect_success 'check-mailmap bogus contact --stdin' '\n \ttest_must_fail git check-mailmap --stdin bogus expect <<-EOF &&\n \t$GIT_AUTHOR_NAME (1):\n-- \n2.32.0-dev\n\n"},{"id":"429992","messageId":"CAPig+cTnRiZaHYw9LBVADoRY695D=ckRbyPzE1qG7o3qCsNeQQ@mail.gmail.com","threadId":"55888","inReplyTo":"patch-4.4-b8062a09f9-20210714T001007Z-avarab@gmail.com","subject":"Re: [PATCH 4/4] SANITIZE tests: fix leak in mailmap.c","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-07-14T02:19:25Z","receivedAt":"2021-07-14T02:19:40Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jul 13, 2021 at 8:12 PM Ævar Arnfjörð Bjarmason\n wrote:\n> Get closer to being able to run t4203-mailmap.sh by fixing a couple of\n> memory leak in mailmap.c.\n>\n> In the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\n> and clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\n> clear the \"me\" structure, but while we freed parts of the\n> mailmap_entry structure, we didn't free the structure itself. The same\n> goes for the \"mailmap_info\" structure.\n> ---\n\nMissing sign-off.\n"},{"id":"429994","messageId":"YO5YsWV5dIW3XbiV@danh.dev","threadId":"55888","inReplyTo":"patch-1.4-a61a294132-20210714T001007Z-avarab@gmail.com","subject":"Re: [PATCH 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2021-07-14T03:23:29Z","receivedAt":"2021-07-14T03:23:35Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2021-07-14 02:11:46+0200, Ævar Arnfjörð Bjarmason wrote:\n> While git can be compiled with SANITIZE=leak there has been no\n> corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n> fixed as one-offs without structured regression testing.\n> \n> This change add such a mode, we now have new\n> linux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\n> tests as linux-{clang,gcc}, except that almost all of them are\n> skipped.\n> \n> There is a whitelist of some tests that are OK in test-lib.sh, and\n> individual tests can be opted-in by setting\n> GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n> individual test can be skipped with the \"!SANITIZE_LEAK\"\n> prerequisite. See the updated t/README for more details.\n> \n> I'm using the GIT_TEST_SANITIZE_LEAK=true and !SANITIZE_LEAK pattern\n> in a couple of tests whose memory leaks I'll fix in subsequent\n> commits.\n> \n> I'm not being aggressive about opting in tests, it's not all tests\n> that currently pass under SANITIZE=leak, just a small number of\n> known-good tests. We can add more later as we fix leaks and grow more\n> confident in this test mode.\n> \n> See the recent discussion at [1] about the lack of this sort of test\n> mode, and 0e5bba53af (add UNLEAK annotation for reducing leak false\n> positives, 2017-09-08) for the initial addition of SANITIZE=leak.\n> \n> See also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n> 7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n> 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n> past history of \"one-off\" SANITIZE=leak (and more) fixes.\n> \n> When calling maybe_skip_all_sanitize_leak matching against\n> \"$TEST_NAME\" instead of \"$this_test\" as other \"match_pattern_list()\"\n> users do is intentional. I'd like to match things like \"t13*config*\"\n> in subsequent commits. This part of the API isn't public, so we can\n> freely change it in the future.\n> \n> 1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> .github/workflows/main.yml | 6 ++++\n> Makefile | 5 ++++\n> ci/install-dependencies.sh | 4 +--\n> ci/lib.sh | 18 +++++++----\n> ci/run-build-and-tests.sh | 4 +--\n> t/README | 16 ++++++++++\n> t/t0500-progress-display.sh | 3 +-\n> t/t5701-git-serve.sh | 2 +-\n> t/test-lib.sh | 60 +++++++++++++++++++++++++++++++++++++\n> 9 files changed, 107 insertions(+), 11 deletions(-)\n> \n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index 73856bafc9..b81ec34959 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -297,6 +297,12 @@ jobs:\n> - jobname: linux-gcc-default\n> cc: gcc\n> pool: ubuntu-latest\n> + - jobname: linux-clang-sanitize-leak\n> + cc: clang\n> + pool: ubuntu-latest\n> + - jobname: linux-gcc-sanitize-leak\n> + cc: clang\n\nI think you meant:\n\n\tcc: gcc\n\n?\n> + pool: ubuntu-latest\n> env:\n> CC: ${{matrix.vector.cc}}\n> jobname: ${{matrix.vector.jobname}}\n> diff --git a/Makefile b/Makefile\n> index 502e0c9a81..d4cad5136f 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1216,6 +1216,9 @@ PTHREAD_CFLAGS =\n> SPARSE_FLAGS ?=\n> SP_EXTRA_FLAGS = -Wno-universal-initializer\n> \n> +# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n> +SANITIZE_LEAK =\n> +\n> # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n> # usually result in less CPU usage at the cost of higher peak memory.\n> # Setting it to 0 will feed all files in a single spatch invocation.\n> @@ -1260,6 +1263,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n> endif\n> ifneq ($(filter leak,$(SANITIZERS)),)\n> BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n> +SANITIZE_LEAK = YesCompiledWithIt\n> endif\n> ifneq ($(filter address,$(SANITIZERS)),)\n> NO_REGEX = NeededForASAN\n> @@ -2793,6 +2797,7 @@ GIT-BUILD-OPTIONS: FORCE\n> \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n> \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n> \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n> +\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n> \t@echo X=\\'$(X)\\' >>$@+\n> ifdef TEST_OUTPUT_DIRECTORY\n> \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index 67852d0d37..31e519cde9 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n> libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n> \n> case \"$jobname\" in\n> -linux-clang|linux-gcc)\n> +linux-clang*|linux-gcc*)\n\nThis also affects linux-gcc-default, is it intended?\nI think no? So, a case for linux-gcc-default is needed here.\n\n> \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n> \tsudo apt-get -q update\n> \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n> \t\t$UBUNTU_COMMON_PKGS\n> \tcase \"$jobname\" in\n> -\tlinux-gcc)\n> +\tlinux-gcc*)\n> \t\tsudo apt-get -q -y install gcc-8\n> \t\t;;\n> \tesac\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 476c3f369f..34fd914438 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -183,14 +183,16 @@ export GIT_TEST_CLONE_2GB=true\n> export SKIP_DASHED_BUILT_INS=YesPlease\n> \n> case \"$jobname\" in\n> -linux-clang|linux-gcc)\n> -\tif [ \"$jobname\" = linux-gcc ]\n> -\tthen\n> +linux-clang*|linux-gcc*)\n\nDitto.\n\n> +\tcase \"$jobname\" in\n> +\tlinux-gcc*)\n> \t\texport CC=gcc-8\n> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n> -\telse\n> +\t\t;;\n> +\t*)\n> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n> -\tfi\n> +\t\t;;\n> +\tesac\n> \n> \texport GIT_TEST_HTTPD=true\n> \n> @@ -233,4 +235,10 @@ linux-musl)\n> \t;;\n> esac\n> \n> +case \"$jobname\" in\n> +linux-*-sanitize-leak)\n> +\texport SANITIZE=leak\n> +\t;;\n> +esac\n> +\n> MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> index 3ce81ffee9..07b9c09f45 100755\n> --- a/ci/run-build-and-tests.sh\n> +++ b/ci/run-build-and-tests.sh\n> @@ -12,7 +12,7 @@ esac\n> \n> make\n> case \"$jobname\" in\n> -linux-gcc)\n> +linux-gcc*)\n\nBut, I'm not sure about this one, though.\nlinux-gcc-default falls into '*' leg, as of it's now.\nDo we want to run it in this leg or the original one?\n\n> \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> \tmake test\n> \texport GIT_TEST_SPLIT_INDEX=yes\n> @@ -29,7 +29,7 @@ linux-gcc)\n> \texport GIT_TEST_CHECKOUT_WORKERS=2\n> \tmake test\n> \t;;\n> -linux-clang)\n> +linux-clang*)\n> \texport GIT_TEST_DEFAULT_HASH=sha1\n> \tmake test\n> \texport GIT_TEST_DEFAULT_HASH=sha256\n> diff --git a/t/README b/t/README\n> index 1a2072b2c8..303d0be817 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -448,6 +448,22 @@ GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n> to and 'checkout.thresholdForParallelism' to 0, forcing the\n> execution of the parallel-checkout code.\n> \n> +GIT_TEST_SANITIZE_LEAK= will force the tests to run when git\n> +is compiled with SANITIZE=leak (we pick it up via\n> +../GIT-BUILD-OPTIONS).\n> +\n> +By default all tests are skipped when compiled with SANITIZE=leak, and\n> +individual test scripts opt themselves in to leak testing by setting\n> +GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n> +tests use the SANITIZE_LEAK prerequisite to skip individiual tests\n> +(i.e. test_expect_success !SANITIZE_LEAK [...]).\n> +\n> +So the GIT_TEST_SANITIZE_LEAK setting is different in behavior from\n> +both other GIT_TEST_*=[true|false] settings, but more useful given how\n> +SANITIZE=leak works & the state of the test suite. Manually setting\n> +GIT_TEST_SANITIZE_LEAK=true is only useful during development when\n> +finding and fixing memory leaks.\n> +\n> Naming Tests\n> ------------\n> \n> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n> index 22058b503a..7afb9abb1f 100755\n> --- a/t/t0500-progress-display.sh\n> +++ b/t/t0500-progress-display.sh\n> @@ -2,6 +2,7 @@\n> \n> test_description='progress display'\n> \n> +GIT_TEST_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> \n> show_cr () {\n> @@ -283,7 +284,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n> \ttest_cmp expect out\n> '\n> \n> -test_expect_success 'progress generates traces' '\n> +test_expect_success !SANITIZE_LEAK 'progress generates traces' '\n> \tcat >in <<-\\EOF &&\n> \tthroughput 102400 1000\n> \tupdate\n> diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\n> index 930721f053..d58efb0aa9 100755\n> --- a/t/t5701-git-serve.sh\n> +++ b/t/t5701-git-serve.sh\n> @@ -243,7 +243,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n> \n> # Test the basics of object-info\n> #\n> -test_expect_success 'basics of object-info' '\n> +test_expect_success !SANITIZE_LEAK 'basics of object-info' '\n> \ttest-tool pkt-line pack >in <<-EOF &&\n> \tcommand=object-info\n> \tobject-format=$(test_oid algo)\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 7036f83b33..9201510e16 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1353,6 +1353,40 @@ then\n> \texit 1\n> fi\n> \n> +# SANITIZE=leak test mode\n> +sanitize_leak_true=\n> +add_sanitize_leak_true () {\n> +\tsanitize_leak_true=\"$sanitize_leak_true$1 \"\n> +}\n> +\n> +sanitize_leak_false=\n> +add_sanitize_leak_false () {\n> +\tsanitize_leak_false=\"$sanitize_leak_false$1 \"\n> +}\n> +\n> +sanitize_leak_opt_in_msg=\"opt-in with GIT_TEST_SANITIZE_LEAK=true\"\n> +maybe_skip_all_sanitize_leak () {\n> +\t# Whitelist patterns\n> +\tadd_sanitize_leak_true 't000*'\n> +\tadd_sanitize_leak_true 't001*'\n> +\tadd_sanitize_leak_true 't006*'\n> +\n> +\t# Blacklist patterns (overrides whitelist)\n> +\tadd_sanitize_leak_false 't000[469]*'\n> +\tadd_sanitize_leak_false 't001[2459]*'\n> +\tadd_sanitize_leak_false 't006[0248]*'\n> +\n> +\tif match_pattern_list \"$1\" \"$sanitize_leak_false\"\n> +\tthen\n> +\t\tskip_all=\"test $this_test on SANITIZE=leak blacklist, $sanitize_leak_opt_in_msg\"\n> +\t\ttest_done\n> +\telif match_pattern_list \"$1\" \"$sanitize_leak_true\"\n> +\tthen\n> +\t\treturn 0\n> +\tfi\n> +\treturn 1\n> +}\n> +\n> # Are we running this test at all?\n> remove_trash=\n> this_test=${0##*/}\n> @@ -1364,6 +1398,31 @@ then\n> \ttest_done\n> fi\n> \n> +# Aggressively skip non-whitelisted tests when compiled with\n> +# SANITIZE=leak\n> +if test -n \"$SANITIZE_LEAK\"\n> +then\n> +\tif test -z \"$GIT_TEST_SANITIZE_LEAK\" &&\n> +\t\tmaybe_skip_all_sanitize_leak \"$TEST_NAME\"\n> +\tthen\n> +\t\tsay_color info >&3 \"test $this_test on SANITIZE=leak whitelist\"\n> +\t\tGIT_TEST_SANITIZE_LEAK=true\n> +\tfi\n> +\n> +\t# We need to see it in \"git env--helper\" (via\n> +\t# test_bool_env)\n> +\texport GIT_TEST_SANITIZE_LEAK\n> +\n> +\tif ! test_bool_env GIT_TEST_SANITIZE_LEAK false\n> +\tthen\n> +\t\tskip_all=\"skip all tests in $this_test under SANITIZE=leak, $sanitize_leak_opt_in_msg\"\n> +\t\ttest_done\n> +\tfi\n> +elif test_bool_env GIT_TEST_SANITIZE_LEAK false\n> +then\n> +\terror \"GIT_TEST_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n> +fi\n> +\n> # Last-minute variable setup\n> HOME=\"$TRASH_DIRECTORY\"\n> GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n> @@ -1516,6 +1575,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n> test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n> test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n> test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n> +test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n> \n> if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n> then\n> -- \n> 2.32.0-dev\n> \n\n-- \nDanh\n"},{"id":"430082","messageId":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com","subject":"[PATCH v2 0/4] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T17:23:50Z","receivedAt":"2021-07-14T17:24:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As a follow-up to my recent thread asking if we had some test mode or\nCI to test for memory leak regression (we don't), add such a test\nmode, and run it in CI.\n\nCurrently the two new CI targets take ~2-3 minutes to run in GitHub\nCI, whereas the normal test targets take 20-30 minutes. The tests run\nslower, but we have a small whitelist of test scripts that are OK.\n\nv2:\n\n * Fixes issues spotted by Đoàn Trần Công Danh and Eric Sunshine,\n thanks both!\n\n * I got rid of the change to t0500, I saw it being flaky in GitHub\n CI, and looks like there'll be other concurrent edits to that file,\n so leaving it be.\n\nv1: http://lore.kernel.org/git/cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com\n\nÆvar Arnfjörð Bjarmason (4):\n tests: add a test mode for SANITIZE=leak, run it in CI\n SANITIZE tests: fix memory leaks in t13*config*, add to whitelist\n SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n SANITIZE tests: fix leak in mailmap.c\n\n .github/workflows/main.yml | 6 ++++\n Makefile | 5 ++++\n ci/install-dependencies.sh | 4 +--\n ci/lib.sh | 18 +++++++----\n ci/run-build-and-tests.sh | 4 +--\n config.c | 17 +++++++----\n mailmap.c | 2 ++\n protocol-caps.c | 5 ++--\n t/README | 16 ++++++++++\n t/t1300-config.sh | 16 ++++++----\n t/t4203-mailmap.sh | 6 ++++\n t/t5701-git-serve.sh | 3 +-\n t/test-lib.sh | 61 ++++++++++++++++++++++++++++++++++++++\n 13 files changed, 140 insertions(+), 23 deletions(-)\n\nRange-diff against v1:\n1: b7948c408d ! 1: 0795436a24 tests: add a test mode for SANITIZE=leak, run it in CI\n @@ .github/workflows/main.yml: jobs:\n + cc: clang\n + pool: ubuntu-latest\n + - jobname: linux-gcc-sanitize-leak\n -+ cc: clang\n ++ cc: gcc\n + pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n @@ ci/install-dependencies.sh: UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl\n \n case \"$jobname\" in\n -linux-clang|linux-gcc)\n -+linux-clang*|linux-gcc*)\n ++linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n -\tlinux-gcc)\n -+\tlinux-gcc*)\n ++\tlinux-gcc|linux-gcc-sanitize-leak)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n -linux-clang|linux-gcc)\n -\tif [ \"$jobname\" = linux-gcc ]\n -\tthen\n -+linux-clang*|linux-gcc*)\n ++linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n +\tcase \"$jobname\" in\n -+\tlinux-gcc*)\n ++\tlinux-gcc|linux-gcc-sanitize-leak)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n -\telse\n @@ ci/lib.sh: linux-musl)\n esac\n \n +case \"$jobname\" in\n -+linux-*-sanitize-leak)\n ++linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n +\texport SANITIZE=leak\n +\t;;\n +esac\n @@ ci/run-build-and-tests.sh: esac\n make\n case \"$jobname\" in\n -linux-gcc)\n -+linux-gcc*)\n ++linux-gcc|linux-gcc-sanitize-leak)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n @@ ci/run-build-and-tests.sh: linux-gcc)\n \tmake test\n \t;;\n -linux-clang)\n -+linux-clang*)\n ++linux-clang|linux-clang-sanitize-leak)\n \texport GIT_TEST_DEFAULT_HASH=sha1\n \tmake test\n \texport GIT_TEST_DEFAULT_HASH=sha256\n @@ t/README: GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n ------------\n \n \n - ## t/t0500-progress-display.sh ##\n -@@\n - \n - test_description='progress display'\n - \n -+GIT_TEST_SANITIZE_LEAK=true\n - . ./test-lib.sh\n - \n - show_cr () {\n -@@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shortens a lot' '\n - \ttest_cmp expect out\n - '\n - \n --test_expect_success 'progress generates traces' '\n -+test_expect_success !SANITIZE_LEAK 'progress generates traces' '\n - \tcat >in <<-\\EOF &&\n - \tthroughput 102400 1000\n - \tupdate\n -\n ## t/t5701-git-serve.sh ##\n @@ t/t5701-git-serve.sh: test_expect_success 'unexpected lines are not allowed in fetch request' '\n \n2: babcb1c289 = 2: 867e8e9a6c SANITIZE tests: fix memory leaks in t13*config*, add to whitelist\n3: 11aa2f3bb5 = 3: b7fb5d5a56 SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n4: 7f4e433559 ! 4: ad8680f529 SANITIZE tests: fix leak in mailmap.c\n @@ Commit message\n mailmap_entry structure, we didn't free the structure itself. The same\n goes for the \"mailmap_info\" structure.\n \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n +\n ## mailmap.c ##\n @@ mailmap.c: static void free_mailmap_info(void *p, const char *s)\n \t\t s, debug_str(mi->name), debug_str(mi->email));\n-- \n2.32.0.853.g5a570c9bf9\n\n"},{"id":"430085","messageId":"patch-1.4-0795436a24-20210714T172251Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","subject":"[PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T17:23:51Z","receivedAt":"2021-07-14T17:24:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak there has been no\ncorresponding GIT_TEST_* mode for it, i.e. memory leaks have been\nfixed as one-offs without structured regression testing.\n\nThis change add such a mode, we now have new\nlinux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\ntests as linux-{clang,gcc}, except that almost all of them are\nskipped.\n\nThere is a whitelist of some tests that are OK in test-lib.sh, and\nindividual tests can be opted-in by setting\nGIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\nindividual test can be skipped with the \"!SANITIZE_LEAK\"\nprerequisite. See the updated t/README for more details.\n\nI'm using the GIT_TEST_SANITIZE_LEAK=true and !SANITIZE_LEAK pattern\nin a couple of tests whose memory leaks I'll fix in subsequent\ncommits.\n\nI'm not being aggressive about opting in tests, it's not all tests\nthat currently pass under SANITIZE=leak, just a small number of\nknown-good tests. We can add more later as we fix leaks and grow more\nconfident in this test mode.\n\nSee the recent discussion at [1] about the lack of this sort of test\nmode, and 0e5bba53af (add UNLEAK annotation for reducing leak false\npositives, 2017-09-08) for the initial addition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nWhen calling maybe_skip_all_sanitize_leak matching against\n\"$TEST_NAME\" instead of \"$this_test\" as other \"match_pattern_list()\"\nusers do is intentional. I'd like to match things like \"t13*config*\"\nin subsequent commits. This part of the API isn't public, so we can\nfreely change it in the future.\n\n1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 6 ++++\n Makefile | 5 ++++\n ci/install-dependencies.sh | 4 +--\n ci/lib.sh | 18 ++++++++----\n ci/run-build-and-tests.sh | 4 +--\n t/README | 16 ++++++++++\n t/t5701-git-serve.sh | 2 +-\n t/test-lib.sh | 60 ++++++++++++++++++++++++++++++++++++++\n 8 files changed, 105 insertions(+), 10 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 73856bafc9..752fe187f9 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -297,6 +297,12 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-clang-sanitize-leak\n+ cc: clang\n+ pool: ubuntu-latest\n+ - jobname: linux-gcc-sanitize-leak\n+ cc: gcc\n+ pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/Makefile b/Makefile\nindex 502e0c9a81..d4cad5136f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1216,6 +1216,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1260,6 +1263,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2793,6 +2797,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 67852d0d37..8ac72d7246 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-gcc-sanitize-leak)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f..bb02b5abf4 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,14 +183,16 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n-\tif [ \"$jobname\" = linux-gcc ]\n-\tthen\n+linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n+\tcase \"$jobname\" in\n+\tlinux-gcc|linux-gcc-sanitize-leak)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \texport GIT_TEST_HTTPD=true\n \n@@ -233,4 +235,10 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n+\texport SANITIZE=leak\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..5fe047b5c6 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -12,7 +12,7 @@ esac\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc|linux-gcc-sanitize-leak)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n@@ -29,7 +29,7 @@ linux-gcc)\n \texport GIT_TEST_CHECKOUT_WORKERS=2\n \tmake test\n \t;;\n-linux-clang)\n+linux-clang|linux-clang-sanitize-leak)\n \texport GIT_TEST_DEFAULT_HASH=sha1\n \tmake test\n \texport GIT_TEST_DEFAULT_HASH=sha256\ndiff --git a/t/README b/t/README\nindex 1a2072b2c8..303d0be817 100644\n--- a/t/README\n+++ b/t/README\n@@ -448,6 +448,22 @@ GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n to and 'checkout.thresholdForParallelism' to 0, forcing the\n execution of the parallel-checkout code.\n \n+GIT_TEST_SANITIZE_LEAK= will force the tests to run when git\n+is compiled with SANITIZE=leak (we pick it up via\n+../GIT-BUILD-OPTIONS).\n+\n+By default all tests are skipped when compiled with SANITIZE=leak, and\n+individual test scripts opt themselves in to leak testing by setting\n+GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n+tests use the SANITIZE_LEAK prerequisite to skip individiual tests\n+(i.e. test_expect_success !SANITIZE_LEAK [...]).\n+\n+So the GIT_TEST_SANITIZE_LEAK setting is different in behavior from\n+both other GIT_TEST_*=[true|false] settings, but more useful given how\n+SANITIZE=leak works & the state of the test suite. Manually setting\n+GIT_TEST_SANITIZE_LEAK=true is only useful during development when\n+finding and fixing memory leaks.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 930721f053..d58efb0aa9 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -243,7 +243,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n \n # Test the basics of object-info\n #\n-test_expect_success 'basics of object-info' '\n+test_expect_success !SANITIZE_LEAK 'basics of object-info' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=object-info\n \tobject-format=$(test_oid algo)\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 7036f83b33..9201510e16 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1353,6 +1353,40 @@ then\n \texit 1\n fi\n \n+# SANITIZE=leak test mode\n+sanitize_leak_true=\n+add_sanitize_leak_true () {\n+\tsanitize_leak_true=\"$sanitize_leak_true$1 \"\n+}\n+\n+sanitize_leak_false=\n+add_sanitize_leak_false () {\n+\tsanitize_leak_false=\"$sanitize_leak_false$1 \"\n+}\n+\n+sanitize_leak_opt_in_msg=\"opt-in with GIT_TEST_SANITIZE_LEAK=true\"\n+maybe_skip_all_sanitize_leak () {\n+\t# Whitelist patterns\n+\tadd_sanitize_leak_true 't000*'\n+\tadd_sanitize_leak_true 't001*'\n+\tadd_sanitize_leak_true 't006*'\n+\n+\t# Blacklist patterns (overrides whitelist)\n+\tadd_sanitize_leak_false 't000[469]*'\n+\tadd_sanitize_leak_false 't001[2459]*'\n+\tadd_sanitize_leak_false 't006[0248]*'\n+\n+\tif match_pattern_list \"$1\" \"$sanitize_leak_false\"\n+\tthen\n+\t\tskip_all=\"test $this_test on SANITIZE=leak blacklist, $sanitize_leak_opt_in_msg\"\n+\t\ttest_done\n+\telif match_pattern_list \"$1\" \"$sanitize_leak_true\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\treturn 1\n+}\n+\n # Are we running this test at all?\n remove_trash=\n this_test=${0##*/}\n@@ -1364,6 +1398,31 @@ then\n \ttest_done\n fi\n \n+# Aggressively skip non-whitelisted tests when compiled with\n+# SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test -z \"$GIT_TEST_SANITIZE_LEAK\" &&\n+\t\tmaybe_skip_all_sanitize_leak \"$TEST_NAME\"\n+\tthen\n+\t\tsay_color info >&3 \"test $this_test on SANITIZE=leak whitelist\"\n+\t\tGIT_TEST_SANITIZE_LEAK=true\n+\tfi\n+\n+\t# We need to see it in \"git env--helper\" (via\n+\t# test_bool_env)\n+\texport GIT_TEST_SANITIZE_LEAK\n+\n+\tif ! test_bool_env GIT_TEST_SANITIZE_LEAK false\n+\tthen\n+\t\tskip_all=\"skip all tests in $this_test under SANITIZE=leak, $sanitize_leak_opt_in_msg\"\n+\t\ttest_done\n+\tfi\n+elif test_bool_env GIT_TEST_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n@@ -1516,6 +1575,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.32.0.853.g5a570c9bf9\n\n"},{"id":"430084","messageId":"patch-2.4-867e8e9a6c-20210714T172251Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","subject":"[PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T17:23:52Z","receivedAt":"2021-07-14T17:24:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a couple of trivial memory leaks introduced in 3efd0bedc6 (config:\nadd conditional include, 2017-03-01) and my own 867ad08a26 (hooks:\nallow customizing where the hook directory is, 2016-05-04).\n\nIn the latter case the \"fix\" is UNLEAK() on the global variable. This\nallows us to run all t13*config* tests under SANITIZE=leak.\n\nWith this change we can now run almost the whole set of config.c\ntests (t13*config) under SANITIZE=leak, so let's do so, with a few\nexceptions:\n\n * The test added in ce81b1da23 (config: add new way to pass config\n via `--config-env`, 2021-01-12), it fails in GitHub CI, but passes\n for me locally. Let's just skip it for now.\n\n * Ditto the split_cmdline and \"aliases of builtins\" tests, the former\n required splitting up an existing test, there an issue with the test\n that would have also been revealed by skipping it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n config.c | 17 ++++++++++++-----\n t/t1300-config.sh | 16 ++++++++++------\n t/test-lib.sh | 1 +\n 3 files changed, 23 insertions(+), 11 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex f9c400ad30..38e132c0e2 100644\n--- a/config.c\n+++ b/config.c\n@@ -138,8 +138,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n \t\treturn config_error_nonbool(\"include.path\");\n \n \texpanded = expand_user_path(path, 0);\n-\tif (!expanded)\n-\t\treturn error(_(\"could not expand include path '%s'\"), path);\n+\tif (!expanded) {\n+\t\tret = error(_(\"could not expand include path '%s'\"), path);\n+\t\tgoto cleanup;\n+\t}\n \tpath = expanded;\n \n \t/*\n@@ -149,8 +151,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n \tif (!is_absolute_path(path)) {\n \t\tchar *slash;\n \n-\t\tif (!cf || !cf->path)\n-\t\t\treturn error(_(\"relative config includes must come from files\"));\n+\t\tif (!cf || !cf->path) {\n+\t\t\tret = error(_(\"relative config includes must come from files\"));\n+\t\t\tgoto cleanup;\n+\t\t}\n \n \t\tslash = find_last_dir_sep(cf->path);\n \t\tif (slash)\n@@ -168,6 +172,7 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n \t\tret = git_config_from_file(git_config_include, path, inc);\n \t\tinc->depth--;\n \t}\n+cleanup:\n \tstrbuf_release(&buf);\n \tfree(expanded);\n \treturn ret;\n@@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(var, \"core.attributesfile\"))\n \t\treturn git_config_pathname(&git_attributes_file, var, value);\n \n-\tif (!strcmp(var, \"core.hookspath\"))\n+\tif (!strcmp(var, \"core.hookspath\")) {\n+\t\tUNLEAK(git_hooks_path);\n \t\treturn git_config_pathname(&git_hooks_path, var, value);\n+\t}\n \n \tif (!strcmp(var, \"core.bare\")) {\n \t\tis_bare_repository_cfg = git_config_bool(var, value);\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 9ff46f3b04..93ad0f4887 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -1050,12 +1050,16 @@ test_expect_success SYMLINKS 'symlink to nonexistent configuration' '\n \ttest_must_fail git config --file=linktolinktonada --list\n '\n \n-test_expect_success 'check split_cmdline return' \"\n-\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n-\ttest_must_fail git split-cmdline-fix &&\n+test_expect_success 'setup check split_cmdline return' \"\n \techo foo > foo &&\n \tgit add foo &&\n-\tgit commit -m 'initial commit' &&\n+\tgit commit -m 'initial commit'\n+\"\n+\n+test_expect_success !SANITIZE_LEAK 'check split_cmdline return' \"\n+\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n+\ttest_must_fail git split-cmdline-fix &&\n+\n \tgit config branch.main.mergeoptions 'echo \\\"' &&\n \ttest_must_fail git merge main\n \"\n@@ -1101,7 +1105,7 @@ test_expect_success 'key sanity-checking' '\n \tgit config foo.\"ba =z\".bar false\n '\n \n-test_expect_success 'git -c works with aliases of builtins' '\n+test_expect_success !SANITIZE_LEAK 'git -c works with aliases of builtins' '\n \tgit config alias.checkconfig \"-c foo.check=bar config foo.check\" &&\n \techo bar >expect &&\n \tgit checkconfig >actual &&\n@@ -1397,7 +1401,7 @@ test_expect_success 'git --config-env with missing value' '\n \tgrep \"invalid config format: config\" error\n '\n \n-test_expect_success 'git --config-env fails with invalid parameters' '\n+test_expect_success !SANITIZE_LEAK 'git --config-env fails with invalid parameters' '\n \ttest_must_fail git --config-env=foo.flag config --bool foo.flag 2>error &&\n \ttest_i18ngrep \"invalid config format: foo.flag\" error &&\n \ttest_must_fail git --config-env=foo.flag= config --bool foo.flag 2>error &&\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9201510e16..98e20950c3 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1370,6 +1370,7 @@ maybe_skip_all_sanitize_leak () {\n \tadd_sanitize_leak_true 't000*'\n \tadd_sanitize_leak_true 't001*'\n \tadd_sanitize_leak_true 't006*'\n+\tadd_sanitize_leak_true 't13*config*'\n \n \t# Blacklist patterns (overrides whitelist)\n \tadd_sanitize_leak_false 't000[469]*'\n-- \n2.32.0.853.g5a570c9bf9\n\n"},{"id":"430083","messageId":"patch-3.4-b7fb5d5a56-20210714T172251Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","subject":"[PATCH v2 3/4] SANITIZE tests: fix memory leaks in t5701*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T17:23:53Z","receivedAt":"2021-07-14T17:24:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in a2ba162cda (object-info: support for retrieving\nobject info, 2021-04-20) which appears to have been based on a\nmisunderstanding of how the pkt-line.c API works, there is no need to\nstrdup() input to, it's just a printf()-like format function.\n\nThis fixes a potentially large memory leak, since the number of OID\nlines the \"object-info\" call can be arbitrarily large (or a small one\nif the request is small).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n protocol-caps.c | 5 +++--\n t/t5701-git-serve.sh | 1 +\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex 13a9e63a04..901b6795e4 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -69,9 +69,10 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpacket_writer_write(writer, \"%s\",\n-\t\t\t\t strbuf_detach(&send_buffer, NULL));\n+\t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n+\t\tstrbuf_reset(&send_buffer);\n \t}\n+\tstrbuf_release(&send_buffer);\n }\n \n int cap_object_info(struct repository *r, struct strvec *keys,\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex d58efb0aa9..e2f4832adf 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -5,6 +5,7 @@ test_description='test protocol v2 server commands'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'test capability advertisement' '\n-- \n2.32.0.853.g5a570c9bf9\n\n"},{"id":"430086","messageId":"patch-4.4-ad8680f529-20210714T172251Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","subject":"[PATCH v2 4/4] SANITIZE tests: fix leak in mailmap.c","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T17:23:54Z","receivedAt":"2021-07-14T17:24:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Get closer to being able to run t4203-mailmap.sh by fixing a couple of\nmemory leak in mailmap.c.\n\nIn the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\nand clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\nclear the \"me\" structure, but while we freed parts of the\nmailmap_entry structure, we didn't free the structure itself. The same\ngoes for the \"mailmap_info\" structure.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n mailmap.c | 2 ++\n t/t4203-mailmap.sh | 6 ++++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/mailmap.c b/mailmap.c\nindex d1f7c0d272..e1c8736093 100644\n--- a/mailmap.c\n+++ b/mailmap.c\n@@ -36,6 +36,7 @@ static void free_mailmap_info(void *p, const char *s)\n \t\t s, debug_str(mi->name), debug_str(mi->email));\n \tfree(mi->name);\n \tfree(mi->email);\n+\tfree(mi);\n }\n \n static void free_mailmap_entry(void *p, const char *s)\n@@ -51,6 +52,7 @@ static void free_mailmap_entry(void *p, const char *s)\n \n \tme->namemap.strdup_strings = 1;\n \tstring_list_clear_func(&me->namemap, free_mailmap_info);\n+\tfree(me);\n }\n \n /*\ndiff --git a/t/t4203-mailmap.sh b/t/t4203-mailmap.sh\nindex 0b2d21ec55..c7de4299cf 100755\n--- a/t/t4203-mailmap.sh\n+++ b/t/t4203-mailmap.sh\n@@ -79,6 +79,12 @@ test_expect_success 'check-mailmap bogus contact --stdin' '\n \ttest_must_fail git check-mailmap --stdin bogus expect <<-EOF &&\n \t$GIT_AUTHOR_NAME (1):\n-- \n2.32.0.853.g5a570c9bf9\n\n"},{"id":"430100","messageId":"eebb4f74-b5e3-6c11-3b84-fcee1b876992@ahunt.org","threadId":"55888","inReplyTo":"patch-1.4-0795436a24-20210714T172251Z-avarab@gmail.com","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-07-14T18:42:27Z","receivedAt":"2021-07-14T18:42:39Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 14/07/2021 19:23, Ævar Arnfjörð Bjarmason wrote:\n> While git can be compiled with SANITIZE=leak there has been no\n> corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n> fixed as one-offs without structured regression testing.\n> \n> This change add such a mode, we now have new\n> linux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\n> tests as linux-{clang,gcc}, except that almost all of them are\n> skipped.\n> \n> There is a whitelist of some tests that are OK in test-lib.sh, and\n> individual tests can be opted-in by setting\n> GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n> individual test can be skipped with the \"!SANITIZE_LEAK\"\n> prerequisite. See the updated t/README for more details.\n> \n> I'm using the GIT_TEST_SANITIZE_LEAK=true and !SANITIZE_LEAK pattern\n> in a couple of tests whose memory leaks I'll fix in subsequent\n> commits.\n> \n> I'm not being aggressive about opting in tests, it's not all tests\n> that currently pass under SANITIZE=leak, just a small number of\n> known-good tests. We can add more later as we fix leaks and grow more\n> confident in this test mode.\n> \n> See the recent discussion at [1] about the lack of this sort of test\n> mode, and 0e5bba53af (add UNLEAK annotation for reducing leak false\n> positives, 2017-09-08) for the initial addition of SANITIZE=leak.\n> \n> See also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n> 7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n> 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n> past history of \"one-off\" SANITIZE=leak (and more) fixes.\n> \n> When calling maybe_skip_all_sanitize_leak matching against\n> \"$TEST_NAME\" instead of \"$this_test\" as other \"match_pattern_list()\"\n> users do is intentional. I'd like to match things like \"t13*config*\"\n> in subsequent commits. This part of the API isn't public, so we can\n> freely change it in the future.\n> \n> 1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> .github/workflows/main.yml | 6 ++++\n> Makefile | 5 ++++\n> ci/install-dependencies.sh | 4 +--\n> ci/lib.sh | 18 ++++++++----\n> ci/run-build-and-tests.sh | 4 +--\n> t/README | 16 ++++++++++\n> t/t5701-git-serve.sh | 2 +-\n> t/test-lib.sh | 60 ++++++++++++++++++++++++++++++++++++++\n> 8 files changed, 105 insertions(+), 10 deletions(-)\n> \n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index 73856bafc9..752fe187f9 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -297,6 +297,12 @@ jobs:\n> - jobname: linux-gcc-default\n> cc: gcc\n> pool: ubuntu-latest\n> + - jobname: linux-clang-sanitize-leak\n> + cc: clang\n> + pool: ubuntu-latest\n> + - jobname: linux-gcc-sanitize-leak\n> + cc: gcc\n> + pool: ubuntu-latest\n\nIs there any advantage to running leak checking with both gcc and clang? \nMy understanding is that you end up using the same sanitiser \nimplementation under the hood - I can't remember if using a different \ncompiler actually helps find different leaks though.\n\nMy other question is: if we are adding a new job - should it really be \njust a leak checking job? Leak checking is just a subset of ASAN \n(Address Sanitizer). And as discussed at [1] it's possible to run ASAN \nand UBSAN (Undefined Behaviour Sanitizer) in the same build. I feel like \nit's much more useful to first add a combined ASAN+UBSAN job, followed \nby enabling leak-checking as part of ASAN in those jobs for known \nleak-free tests - as opposed to only adding leak checking. We currently \ndisable Leak checking for ASAN here [2], but that could be made \nconditional on the test ID (i.e. check an allowlist to enable leak \nchecking for some tests)?\n\nI think it's worth focusing on ASAN+UBSAN first because they tend to \nfind more impactful issues (e.g. buffer overflows, and other real bugs) \n- whereas leaks... are ugly, but leaks in git don't actually have much \nuser impact?\n\n[1] \nhttps://lore.kernel.org/git/YMI%2Fg1sHxJgb8%2FYD@coredump.intra.peff.net/\n\n[2] https://git.kernel.org/pub/scm/git/git.git/tree/t/test-lib.sh#n44\n\n> env:\n> CC: ${{matrix.vector.cc}}\n> jobname: ${{matrix.vector.jobname}}\n> diff --git a/Makefile b/Makefile\n> index 502e0c9a81..d4cad5136f 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1216,6 +1216,9 @@ PTHREAD_CFLAGS =\n> SPARSE_FLAGS ?=\n> SP_EXTRA_FLAGS = -Wno-universal-initializer\n> \n> +# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n> +SANITIZE_LEAK =\n> +\n> # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n> # usually result in less CPU usage at the cost of higher peak memory.\n> # Setting it to 0 will feed all files in a single spatch invocation.\n> @@ -1260,6 +1263,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n> endif\n> ifneq ($(filter leak,$(SANITIZERS)),)\n> BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n> +SANITIZE_LEAK = YesCompiledWithIt > endif\n> ifneq ($(filter address,$(SANITIZERS)),)\n> NO_REGEX = NeededForASAN\n> @@ -2793,6 +2797,7 @@ GIT-BUILD-OPTIONS: FORCE\n> \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n> \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n> \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n> +\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n> \t@echo X=\\'$(X)\\' >>$@+\n> ifdef TEST_OUTPUT_DIRECTORY\n> \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index 67852d0d37..8ac72d7246 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n> libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n> \n> case \"$jobname\" in\n> -linux-clang|linux-gcc)\n> +linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n\nHow about `linux-clang*|linux-gcc*)` here and below?\n\n> \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n> \tsudo apt-get -q update\n> \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n> \t\t$UBUNTU_COMMON_PKGS\n> \tcase \"$jobname\" in\n> -\tlinux-gcc)\n> +\tlinux-gcc|linux-gcc-sanitize-leak)\n> \t\tsudo apt-get -q -y install gcc-8\n> \t\t;;\n> \tesac\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 476c3f369f..bb02b5abf4 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -183,14 +183,16 @@ export GIT_TEST_CLONE_2GB=true\n> export SKIP_DASHED_BUILT_INS=YesPlease\n> \n> case \"$jobname\" in\n> -linux-clang|linux-gcc)\n> -\tif [ \"$jobname\" = linux-gcc ]\n> -\tthen\n> +linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n> +\tcase \"$jobname\" in\n> +\tlinux-gcc|linux-gcc-sanitize-leak)\n> \t\texport CC=gcc-8\n> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n> -\telse\n> +\t\t;;\n> +\t*)\n> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n> -\tfi\n> +\t\t;;\n> +\tesac\n> \n> \texport GIT_TEST_HTTPD=true\n> \n> @@ -233,4 +235,10 @@ linux-musl)\n> \t;;\n> esac\n> \n> +case \"$jobname\" in\n> +linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n> +\texport SANITIZE=leak\n> +\t;;\n> +esac\n> +\n\nHave you considered doing this in the yaml job configuration instead? \nIt's possible to set env-vars in yaml, although it will require some \ncareful tweaking - here's an example where I'm setting different values \nfor SANITIZE depending on job (you'd probably just have to set it to \nempty for the non leak-checking jobs):\n\nhttps://github.com/ahunt/git/blob/master/.github/workflows/ahunt-sync-next2.yml#L51-L69\n\nThat does make the yaml more complex, but I think it's worth it to \nreduce the amount of special-casing elsewhere (and is also worth it if \nwe ever add other sanitisers)?\n\n> MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> index 3ce81ffee9..5fe047b5c6 100755\n> --- a/ci/run-build-and-tests.sh\n> +++ b/ci/run-build-and-tests.sh\n> @@ -12,7 +12,7 @@ esac\n> \n> make\n> case \"$jobname\" in\n> -linux-gcc)\n> +linux-gcc|linux-gcc-sanitize-leak)\n> \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> \tmake test\n> \texport GIT_TEST_SPLIT_INDEX=yes\n> @@ -29,7 +29,7 @@ linux-gcc)\n> \texport GIT_TEST_CHECKOUT_WORKERS=2\n> \tmake test\n> \t;;\n> -linux-clang)\n> +linux-clang|linux-clang-sanitize-leak)\n> \texport GIT_TEST_DEFAULT_HASH=sha1\n> \tmake test\n> \texport GIT_TEST_DEFAULT_HASH=sha256\n> diff --git a/t/README b/t/README\n> index 1a2072b2c8..303d0be817 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -448,6 +448,22 @@ GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n> to and 'checkout.thresholdForParallelism' to 0, forcing the\n> execution of the parallel-checkout code.\n> \n> +GIT_TEST_SANITIZE_LEAK= will force the tests to run when git\n> +is compiled with SANITIZE=leak (we pick it up via\n> +../GIT-BUILD-OPTIONS).\n> +\n> +By default all tests are skipped when compiled with SANITIZE=leak, and\n> +individual test scripts opt themselves in to leak testing by setting\n> +GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n> +tests use the SANITIZE_LEAK prerequisite to skip individiual tests\n> +(i.e. test_expect_success !SANITIZE_LEAK [...]).\n> +\n> +So the GIT_TEST_SANITIZE_LEAK setting is different in behavior from\n> +both other GIT_TEST_*=[true|false] settings, but more useful given how\n> +SANITIZE=leak works & the state of the test suite. Manually setting\n> +GIT_TEST_SANITIZE_LEAK=true is only useful during development when\n> +finding and fixing memory leaks.\n> +\n> Naming Tests\n> ------------\n> \n> diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\n> index 930721f053..d58efb0aa9 100755\n> --- a/t/t5701-git-serve.sh\n> +++ b/t/t5701-git-serve.sh\n> @@ -243,7 +243,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n> \n> # Test the basics of object-info\n> #\n> -test_expect_success 'basics of object-info' '\n> +test_expect_success !SANITIZE_LEAK 'basics of object-info' '\n> \ttest-tool pkt-line pack >in <<-EOF &&\n> \tcommand=object-info\n> \tobject-format=$(test_oid algo)\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 7036f83b33..9201510e16 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1353,6 +1353,40 @@ then\n> \texit 1\n> fi\n> \n> +# SANITIZE=leak test mode\n> +sanitize_leak_true=\n> +add_sanitize_leak_true () {\n> +\tsanitize_leak_true=\"$sanitize_leak_true$1 \"\n> +}\n> +\n> +sanitize_leak_false=\n> +add_sanitize_leak_false () {\n> +\tsanitize_leak_false=\"$sanitize_leak_false$1 \"\n> +}\n> +\n> +sanitize_leak_opt_in_msg=\"opt-in with GIT_TEST_SANITIZE_LEAK=true\"\n> +maybe_skip_all_sanitize_leak () {\n> +\t# Whitelist patterns\n> +\tadd_sanitize_leak_true 't000*'\n> +\tadd_sanitize_leak_true 't001*'\n> +\tadd_sanitize_leak_true 't006*'\n> +\n> +\t# Blacklist patterns (overrides whitelist)\n> +\tadd_sanitize_leak_false 't000[469]*'\n> +\tadd_sanitize_leak_false 't001[2459]*'\n> +\tadd_sanitize_leak_false 't006[0248]*'\n> +\n> +\tif match_pattern_list \"$1\" \"$sanitize_leak_false\"\n> +\tthen\n> +\t\tskip_all=\"test $this_test on SANITIZE=leak blacklist, $sanitize_leak_opt_in_msg\"\n> +\t\ttest_done\n> +\telif match_pattern_list \"$1\" \"$sanitize_leak_true\"\n> +\tthen\n> +\t\treturn 0\n> +\tfi\n> +\treturn 1\n> +}\n> +\n> # Are we running this test at all?\n> remove_trash=\n> this_test=${0##*/}\n> @@ -1364,6 +1398,31 @@ then\n> \ttest_done\n> fi\n> \n> +# Aggressively skip non-whitelisted tests when compiled with\n> +# SANITIZE=leak\n> +if test -n \"$SANITIZE_LEAK\"\n> +then\n> +\tif test -z \"$GIT_TEST_SANITIZE_LEAK\" &&\n> +\t\tmaybe_skip_all_sanitize_leak \"$TEST_NAME\"\n> +\tthen\n> +\t\tsay_color info >&3 \"test $this_test on SANITIZE=leak whitelist\"\n> +\t\tGIT_TEST_SANITIZE_LEAK=true\n> +\tfi\n> +\n> +\t# We need to see it in \"git env--helper\" (via\n> +\t# test_bool_env)\n> +\texport GIT_TEST_SANITIZE_LEAK\n> +\n> +\tif ! test_bool_env GIT_TEST_SANITIZE_LEAK false\n> +\tthen\n> +\t\tskip_all=\"skip all tests in $this_test under SANITIZE=leak, $sanitize_leak_opt_in_msg\"\n> +\t\ttest_done\n> +\tfi\n> +elif test_bool_env GIT_TEST_SANITIZE_LEAK false\n> +then\n> +\terror \"GIT_TEST_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n> +fi\n> +\n> # Last-minute variable setup\n> HOME=\"$TRASH_DIRECTORY\"\n> GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n> @@ -1516,6 +1575,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n> test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n> test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n> test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n> +test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n> \n> if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n> then\n> \n"},{"id":"430102","messageId":"871ea493-e108-e748-0234-f929690ad2fd@ahunt.org","threadId":"55888","inReplyTo":"patch-2.4-867e8e9a6c-20210714T172251Z-avarab@gmail.com","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-07-14T18:57:37Z","receivedAt":"2021-07-14T18:57:53Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 14/07/2021 19:23, Ævar Arnfjörð Bjarmason wrote:\n> Fix a couple of trivial memory leaks introduced in 3efd0bedc6 (config:\n> add conditional include, 2017-03-01) and my own 867ad08a26 (hooks:\n> allow customizing where the hook directory is, 2016-05-04).\n> \n> In the latter case the \"fix\" is UNLEAK() on the global variable. This\n> allows us to run all t13*config* tests under SANITIZE=leak.\n> \n> With this change we can now run almost the whole set of config.c\n> tests (t13*config) under SANITIZE=leak, so let's do so, with a few\n> exceptions:\n> \n> * The test added in ce81b1da23 (config: add new way to pass config\n> via `--config-env`, 2021-01-12), it fails in GitHub CI, but passes\n> for me locally. Let's just skip it for now.\n> \n> * Ditto the split_cmdline and \"aliases of builtins\" tests, the former\n> required splitting up an existing test, there an issue with the test\n> that would have also been revealed by skipping it.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> config.c | 17 ++++++++++++-----\n> t/t1300-config.sh | 16 ++++++++++------\n> t/test-lib.sh | 1 +\n> 3 files changed, 23 insertions(+), 11 deletions(-)\n> \n> diff --git a/config.c b/config.c\n> index f9c400ad30..38e132c0e2 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -138,8 +138,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n> \t\treturn config_error_nonbool(\"include.path\");\n> \n> \texpanded = expand_user_path(path, 0);\n> -\tif (!expanded)\n> -\t\treturn error(_(\"could not expand include path '%s'\"), path);\n> +\tif (!expanded) {\n> +\t\tret = error(_(\"could not expand include path '%s'\"), path);\n> +\t\tgoto cleanup;\n> +\t}\n> \tpath = expanded;\n> \n> \t/*\n> @@ -149,8 +151,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n> \tif (!is_absolute_path(path)) {\n> \t\tchar *slash;\n> \n> -\t\tif (!cf || !cf->path)\n> -\t\t\treturn error(_(\"relative config includes must come from files\"));\n> +\t\tif (!cf || !cf->path) {\n> +\t\t\tret = error(_(\"relative config includes must come from files\"));\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> \n> \t\tslash = find_last_dir_sep(cf->path);\n> \t\tif (slash)\n> @@ -168,6 +172,7 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n> \t\tret = git_config_from_file(git_config_include, path, inc);\n> \t\tinc->depth--;\n> \t}\n> +cleanup:\n> \tstrbuf_release(&buf);\n> \tfree(expanded);\n> \treturn ret;\n> @@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> \tif (!strcmp(var, \"core.attributesfile\"))\n> \t\treturn git_config_pathname(&git_attributes_file, var, value);\n> \n> -\tif (!strcmp(var, \"core.hookspath\"))\n> +\tif (!strcmp(var, \"core.hookspath\")) {\n> +\t\tUNLEAK(git_hooks_path);\n> \t\treturn git_config_pathname(&git_hooks_path, var, value);\n> +\t}\n\nWhy is the UNLEAK necessary here? We generally want to limit use of \nUNLEAK to cmd_* functions or direct helpers. git_default_core_config() \nseems generic enough that it could be called from anywhere, and using \nUNLEAK here means we're potentially masking a real leak?\n\nIIUC the leak here happens because:\n- git_hooks_path is a global variable - hence it's unlikely we'd ever\n bother cleaning it up.\n- git_default_core_config() gets called a first time with\n core.hookspath, and we end up allocating new memory into\n git_hooks_path.\n- git_default_core_config() gets called again with core.hookspath,\n and we overwrite git_hooks_path with a new string which leaks\n the string that git_hooks_path used to point to.\n\nSo I think the real fix is to free(git_hooks_path) instead of an UNLEAK? \n(Looking at the surrounding code, it looks like the same pattern of leak \nmight be repeated for other similar globals - is it worth auditing those \nwhile we're here?)\n\n> \n> \tif (!strcmp(var, \"core.bare\")) {\n> \t\tis_bare_repository_cfg = git_config_bool(var, value);\n> diff --git a/t/t1300-config.sh b/t/t1300-config.sh\n> index 9ff46f3b04..93ad0f4887 100755\n> --- a/t/t1300-config.sh\n> +++ b/t/t1300-config.sh\n> @@ -1050,12 +1050,16 @@ test_expect_success SYMLINKS 'symlink to nonexistent configuration' '\n> \ttest_must_fail git config --file=linktolinktonada --list\n> '\n> \n> -test_expect_success 'check split_cmdline return' \"\n> -\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n> -\ttest_must_fail git split-cmdline-fix &&\n> +test_expect_success 'setup check split_cmdline return' \"\n> \techo foo > foo &&\n> \tgit add foo &&\n> -\tgit commit -m 'initial commit' &&\n> +\tgit commit -m 'initial commit'\n> +\"\n> +\n> +test_expect_success !SANITIZE_LEAK 'check split_cmdline return' \"\n> +\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n> +\ttest_must_fail git split-cmdline-fix &&\n> +\n> \tgit config branch.main.mergeoptions 'echo \\\"' &&\n> \ttest_must_fail git merge main\n> \"\n> @@ -1101,7 +1105,7 @@ test_expect_success 'key sanity-checking' '\n> \tgit config foo.\"ba =z\".bar false\n> '\n> \n> -test_expect_success 'git -c works with aliases of builtins' '\n> +test_expect_success !SANITIZE_LEAK 'git -c works with aliases of builtins' '\n> \tgit config alias.checkconfig \"-c foo.check=bar config foo.check\" &&\n> \techo bar >expect &&\n> \tgit checkconfig >actual &&\n> @@ -1397,7 +1401,7 @@ test_expect_success 'git --config-env with missing value' '\n> \tgrep \"invalid config format: config\" error\n> '\n> \n> -test_expect_success 'git --config-env fails with invalid parameters' '\n> +test_expect_success !SANITIZE_LEAK 'git --config-env fails with invalid parameters' '\n> \ttest_must_fail git --config-env=foo.flag config --bool foo.flag 2>error &&\n> \ttest_i18ngrep \"invalid config format: foo.flag\" error &&\n> \ttest_must_fail git --config-env=foo.flag= config --bool foo.flag 2>error &&\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 9201510e16..98e20950c3 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1370,6 +1370,7 @@ maybe_skip_all_sanitize_leak () {\n> \tadd_sanitize_leak_true 't000*'\n> \tadd_sanitize_leak_true 't001*'\n> \tadd_sanitize_leak_true 't006*'\n> +\tadd_sanitize_leak_true 't13*config*'\n> \n> \t# Blacklist patterns (overrides whitelist)\n> \tadd_sanitize_leak_false 't000[469]*'\n> \n"},{"id":"430152","messageId":"87k0lszere.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"eebb4f74-b5e3-6c11-3b84-fcee1b876992@ahunt.org","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T22:39:12Z","receivedAt":"2021-07-14T22:56:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 14 2021, Andrzej Hunt wrote:\n\n> On 14/07/2021 19:23, Ævar Arnfjörð Bjarmason wrote:\n>> While git can be compiled with SANITIZE=leak there has been no\n>> corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n>> fixed as one-offs without structured regression testing.\n>> This change add such a mode, we now have new\n>> linux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\n>> tests as linux-{clang,gcc}, except that almost all of them are\n>> skipped.\n>> There is a whitelist of some tests that are OK in test-lib.sh, and\n>> individual tests can be opted-in by setting\n>> GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n>> individual test can be skipped with the \"!SANITIZE_LEAK\"\n>> prerequisite. See the updated t/README for more details.\n>> I'm using the GIT_TEST_SANITIZE_LEAK=true and !SANITIZE_LEAK pattern\n>> in a couple of tests whose memory leaks I'll fix in subsequent\n>> commits.\n>> I'm not being aggressive about opting in tests, it's not all tests\n>> that currently pass under SANITIZE=leak, just a small number of\n>> known-good tests. We can add more later as we fix leaks and grow more\n>> confident in this test mode.\n>> See the recent discussion at [1] about the lack of this sort of test\n>> mode, and 0e5bba53af (add UNLEAK annotation for reducing leak false\n>> positives, 2017-09-08) for the initial addition of SANITIZE=leak.\n>> See also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n>> 7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n>> 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n>> past history of \"one-off\" SANITIZE=leak (and more) fixes.\n>> When calling maybe_skip_all_sanitize_leak matching against\n>> \"$TEST_NAME\" instead of \"$this_test\" as other \"match_pattern_list()\"\n>> users do is intentional. I'd like to match things like \"t13*config*\"\n>> in subsequent commits. This part of the API isn't public, so we can\n>> freely change it in the future.\n>> 1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> .github/workflows/main.yml | 6 ++++\n>> Makefile | 5 ++++\n>> ci/install-dependencies.sh | 4 +--\n>> ci/lib.sh | 18 ++++++++----\n>> ci/run-build-and-tests.sh | 4 +--\n>> t/README | 16 ++++++++++\n>> t/t5701-git-serve.sh | 2 +-\n>> t/test-lib.sh | 60 ++++++++++++++++++++++++++++++++++++++\n>> 8 files changed, 105 insertions(+), 10 deletions(-)\n>> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n>> index 73856bafc9..752fe187f9 100644\n>> --- a/.github/workflows/main.yml\n>> +++ b/.github/workflows/main.yml\n>> @@ -297,6 +297,12 @@ jobs:\n>> - jobname: linux-gcc-default\n>> cc: gcc\n>> pool: ubuntu-latest\n>> + - jobname: linux-clang-sanitize-leak\n>> + cc: clang\n>> + pool: ubuntu-latest\n>> + - jobname: linux-gcc-sanitize-leak\n>> + cc: gcc\n>> + pool: ubuntu-latest\n>\n> Is there any advantage to running leak checking with both gcc and\n> clang? My understanding is that you end up using the same sanitiser \n> implementation under the hood - I can't remember if using a different\n> compiler actually helps find different leaks though.\n\nI didn't know that, makes sense. I'll make it one job and have it use\nwhatever CC is.\n\n> My other question is: if we are adding a new job - should it really be\n> just a leak checking job? Leak checking is just a subset of ASAN \n> (Address Sanitizer). And as discussed at [1] it's possible to run ASAN\n> and UBSAN (Undefined Behaviour Sanitizer) in the same build. I feel\n> like it's much more useful to first add a combined ASAN+UBSAN job,\n> followed by enabling leak-checking as part of ASAN in those jobs for\n> known leak-free tests - as opposed to only adding leak checking. We\n> currently disable Leak checking for ASAN here [2], but that could be\n> made conditional on the test ID (i.e. check an allowlist to enable\n> leak checking for some tests)?\n\nIt sounds good to support that, but at least right now I've got the itch\nof finding leaks during development, and I think in any case being able\nto do a full run with just sanitizing, leak checking (or combined) makes\nsense, i.e. to make GIT_TEST_SANITIZE_LEAK=* the top-level interface.\n\nI haven't checked how noisy ASAN is, is it like the leak checking where\nwe fail almost all tests now?\n\nAnyway, once we have some test mode like this it'll be trivial to extend\nit. I mainly want us to get this into CI so we can have an expanding\nline in the sand with regressions.\n\n> I think it's worth focusing on ASAN+UBSAN first because they tend to\n> find more impactful issues (e.g. buffer overflows, and other real\n> bugs) - whereas leaks... are ugly, but leaks in git don't actually\n> have much user impact?\n\nWe have one-off commands, but also long-lived things like \"git cat-file\n--batch\", it's useful if we don't leak in those.\n\nThe entry point to those tends to be one-off commands in tests, so\nchecking leaks for all commands in a test (if you can get there) is a\nuseful indicator for how the underlying API performs.\n\nI think in the git.git codebase we don't have much of an issue with\nbuffer overflows etc, because we tend to consistently use APIs like\nstrbuf that avoid those issues, but then again I haven't run the tests\nwith that, maybe I'll be unpleasantly surprised.\n\nI also find leak checking to be useful during development to spot faulty\nassumptions, i.e. the leak itself may not be a big deal, but it's\nusually an early sign that I'm structuring something incorrectly.\n\n> [1]\n> https://lore.kernel.org/git/YMI%2Fg1sHxJgb8%2FYD@coredump.intra.peff.net/\n>\n> [2] https://git.kernel.org/pub/scm/git/git.git/tree/t/test-lib.sh#n44\n>\n>> env:\n>> CC: ${{matrix.vector.cc}}\n>> jobname: ${{matrix.vector.jobname}}\n>> diff --git a/Makefile b/Makefile\n>> index 502e0c9a81..d4cad5136f 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -1216,6 +1216,9 @@ PTHREAD_CFLAGS =\n>> SPARSE_FLAGS ?=\n>> SP_EXTRA_FLAGS = -Wno-universal-initializer\n>> +# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n>> +SANITIZE_LEAK =\n>> +\n>> # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n>> # usually result in less CPU usage at the cost of higher peak memory.\n>> # Setting it to 0 will feed all files in a single spatch invocation.\n>> @@ -1260,6 +1263,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n>> endif\n>> ifneq ($(filter leak,$(SANITIZERS)),)\n>> BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n>> +SANITIZE_LEAK = YesCompiledWithIt > endif\n>> ifneq ($(filter address,$(SANITIZERS)),)\n>> NO_REGEX = NeededForASAN\n>> @@ -2793,6 +2797,7 @@ GIT-BUILD-OPTIONS: FORCE\n>> \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n>> \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n>> \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n>> +\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n>> \t@echo X=\\'$(X)\\' >>$@+\n>> ifdef TEST_OUTPUT_DIRECTORY\n>> \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n>> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n>> index 67852d0d37..8ac72d7246 100755\n>> --- a/ci/install-dependencies.sh\n>> +++ b/ci/install-dependencies.sh\n>> @@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n>> libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n>> case \"$jobname\" in\n>> -linux-clang|linux-gcc)\n>> +linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n>\n> How about `linux-clang*|linux-gcc*)` here and below?\n\nI did that in v1, as Đoàn Trần Công Danh pointed out we have other jobs\nthat would match that.\n\n>> \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n>> \tsudo apt-get -q update\n>> \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n>> \t\t$UBUNTU_COMMON_PKGS\n>> \tcase \"$jobname\" in\n>> -\tlinux-gcc)\n>> +\tlinux-gcc|linux-gcc-sanitize-leak)\n>> \t\tsudo apt-get -q -y install gcc-8\n>> \t\t;;\n>> \tesac\n>> diff --git a/ci/lib.sh b/ci/lib.sh\n>> index 476c3f369f..bb02b5abf4 100755\n>> --- a/ci/lib.sh\n>> +++ b/ci/lib.sh\n>> @@ -183,14 +183,16 @@ export GIT_TEST_CLONE_2GB=true\n>> export SKIP_DASHED_BUILT_INS=YesPlease\n>> case \"$jobname\" in\n>> -linux-clang|linux-gcc)\n>> -\tif [ \"$jobname\" = linux-gcc ]\n>> -\tthen\n>> +linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n>> +\tcase \"$jobname\" in\n>> +\tlinux-gcc|linux-gcc-sanitize-leak)\n>> \t\texport CC=gcc-8\n>> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n>> -\telse\n>> +\t\t;;\n>> +\t*)\n>> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n>> -\tfi\n>> +\t\t;;\n>> +\tesac\n>> \texport GIT_TEST_HTTPD=true\n>> @@ -233,4 +235,10 @@ linux-musl)\n>> \t;;\n>> esac\n>> +case \"$jobname\" in\n>> +linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n>> +\texport SANITIZE=leak\n>> +\t;;\n>> +esac\n>> +\n>\n> Have you considered doing this in the yaml job configuration instead?\n> It's possible to set env-vars in yaml, although it will require some \n> careful tweaking - here's an example where I'm setting different\n> values for SANITIZE depending on job (you'd probably just have to set\n> it to empty for the non leak-checking jobs):\n>\n> https://github.com/ahunt/git/blob/master/.github/workflows/ahunt-sync-next2.yml#L51-L69\n>\n> That does make the yaml more complex, but I think it's worth it to\n> reduce the amount of special-casing elsewhere (and is also worth it if \n> we ever add other sanitisers)?\n\nI'm not too familiar with git.git's ci/* dir, but I think it's like that\nin general because we don't just run in GitHub CI, but want to support\nAzure, Travis etc.\n\nSo almost all of the logic is in those shellscripts, right now this\ntarget just runs in the GitHub CI, but it would be useful to make it\ndrop-in enabled elsewhere.\n\nI think given that that doing anything overly clever in the YAML would\nprobably be counter-productive.\n"},{"id":"430154","messageId":"87h7gwzeps.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"871ea493-e108-e748-0234-f929690ad2fd@ahunt.org","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T22:56:21Z","receivedAt":"2021-07-14T22:57:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 14 2021, Andrzej Hunt wrote:\n\n> On 14/07/2021 19:23, Ævar Arnfjörð Bjarmason wrote:\n>> Fix a couple of trivial memory leaks introduced in 3efd0bedc6 (config:\n>> add conditional include, 2017-03-01) and my own 867ad08a26 (hooks:\n>> allow customizing where the hook directory is, 2016-05-04).\n>> In the latter case the \"fix\" is UNLEAK() on the global\n>> variable. This\n>> allows us to run all t13*config* tests under SANITIZE=leak.\n>> With this change we can now run almost the whole set of config.c\n>> tests (t13*config) under SANITIZE=leak, so let's do so, with a few\n>> exceptions:\n>> * The test added in ce81b1da23 (config: add new way to pass config\n>> via `--config-env`, 2021-01-12), it fails in GitHub CI, but passes\n>> for me locally. Let's just skip it for now.\n>> * Ditto the split_cmdline and \"aliases of builtins\" tests, the\n>> former\n>> required splitting up an existing test, there an issue with the test\n>> that would have also been revealed by skipping it.\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> config.c | 17 ++++++++++++-----\n>> t/t1300-config.sh | 16 ++++++++++------\n>> t/test-lib.sh | 1 +\n>> 3 files changed, 23 insertions(+), 11 deletions(-)\n>> diff --git a/config.c b/config.c\n>> index f9c400ad30..38e132c0e2 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> @@ -138,8 +138,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n>> \t\treturn config_error_nonbool(\"include.path\");\n>> \texpanded = expand_user_path(path, 0);\n>> -\tif (!expanded)\n>> -\t\treturn error(_(\"could not expand include path '%s'\"), path);\n>> +\tif (!expanded) {\n>> +\t\tret = error(_(\"could not expand include path '%s'\"), path);\n>> +\t\tgoto cleanup;\n>> +\t}\n>> \tpath = expanded;\n>> \t/*\n>> @@ -149,8 +151,10 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n>> \tif (!is_absolute_path(path)) {\n>> \t\tchar *slash;\n>> -\t\tif (!cf || !cf->path)\n>> -\t\t\treturn error(_(\"relative config includes must come from files\"));\n>> +\t\tif (!cf || !cf->path) {\n>> +\t\t\tret = error(_(\"relative config includes must come from files\"));\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> \t\tslash = find_last_dir_sep(cf->path);\n>> \t\tif (slash)\n>> @@ -168,6 +172,7 @@ static int handle_path_include(const char *path, struct config_include_data *inc\n>> \t\tret = git_config_from_file(git_config_include, path, inc);\n>> \t\tinc->depth--;\n>> \t}\n>> +cleanup:\n>> \tstrbuf_release(&buf);\n>> \tfree(expanded);\n>> \treturn ret;\n>> @@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>> \tif (!strcmp(var, \"core.attributesfile\"))\n>> \t\treturn git_config_pathname(&git_attributes_file, var, value);\n>> -\tif (!strcmp(var, \"core.hookspath\"))\n>> +\tif (!strcmp(var, \"core.hookspath\")) {\n>> +\t\tUNLEAK(git_hooks_path);\n>> \t\treturn git_config_pathname(&git_hooks_path, var, value);\n>> +\t}\n>\n> Why is the UNLEAK necessary here? We generally want to limit use of\n> UNLEAK to cmd_* functions or direct helpers. git_default_core_config() \n> seems generic enough that it could be called from anywhere, and using\n> UNLEAK here means we're potentially masking a real leak?\n>\n> IIUC the leak here happens because:\n> - git_hooks_path is a global variable - hence it's unlikely we'd ever\n> bother cleaning it up.\n> - git_default_core_config() gets called a first time with\n> core.hookspath, and we end up allocating new memory into\n> git_hooks_path.\n> - git_default_core_config() gets called again with core.hookspath,\n> and we overwrite git_hooks_path with a new string which leaks\n> the string that git_hooks_path used to point to.\n>\n> So I think the real fix is to free(git_hooks_path) instead of an\n> UNLEAK? (Looking at the surrounding code, it looks like the same\n> pattern of leak might be repeated for other similar globals - is it\n> worth auditing those while we're here?)\n\nGood point, I'll fix that.\n\nI was doing this rather blindly to see if I could get this larg batch of\ntests to pass with some a minimal fixes/whitelisting of some \"known\nbad\".\n\n>> \tif (!strcmp(var, \"core.bare\")) {\n>> \t\tis_bare_repository_cfg = git_config_bool(var, value);\n>> diff --git a/t/t1300-config.sh b/t/t1300-config.sh\n>> index 9ff46f3b04..93ad0f4887 100755\n>> --- a/t/t1300-config.sh\n>> +++ b/t/t1300-config.sh\n>> @@ -1050,12 +1050,16 @@ test_expect_success SYMLINKS 'symlink to nonexistent configuration' '\n>> \ttest_must_fail git config --file=linktolinktonada --list\n>> '\n>> -test_expect_success 'check split_cmdline return' \"\n>> -\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n>> -\ttest_must_fail git split-cmdline-fix &&\n>> +test_expect_success 'setup check split_cmdline return' \"\n>> \techo foo > foo &&\n>> \tgit add foo &&\n>> -\tgit commit -m 'initial commit' &&\n>> +\tgit commit -m 'initial commit'\n>> +\"\n>> +\n>> +test_expect_success !SANITIZE_LEAK 'check split_cmdline return' \"\n>> +\tgit config alias.split-cmdline-fix 'echo \\\"' &&\n>> +\ttest_must_fail git split-cmdline-fix &&\n>> +\n>> \tgit config branch.main.mergeoptions 'echo \\\"' &&\n>> \ttest_must_fail git merge main\n>> \"\n>> @@ -1101,7 +1105,7 @@ test_expect_success 'key sanity-checking' '\n>> \tgit config foo.\"ba =z\".bar false\n>> '\n>> -test_expect_success 'git -c works with aliases of builtins' '\n>> +test_expect_success !SANITIZE_LEAK 'git -c works with aliases of builtins' '\n>> \tgit config alias.checkconfig \"-c foo.check=bar config foo.check\" &&\n>> \techo bar >expect &&\n>> \tgit checkconfig >actual &&\n>> @@ -1397,7 +1401,7 @@ test_expect_success 'git --config-env with missing value' '\n>> \tgrep \"invalid config format: config\" error\n>> '\n>> -test_expect_success 'git --config-env fails with invalid\n>> parameters' '\n>> +test_expect_success !SANITIZE_LEAK 'git --config-env fails with invalid parameters' '\n>> \ttest_must_fail git --config-env=foo.flag config --bool foo.flag 2>error &&\n>> \ttest_i18ngrep \"invalid config format: foo.flag\" error &&\n>> \ttest_must_fail git --config-env=foo.flag= config --bool foo.flag 2>error &&\n>> diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> index 9201510e16..98e20950c3 100644\n>> --- a/t/test-lib.sh\n>> +++ b/t/test-lib.sh\n>> @@ -1370,6 +1370,7 @@ maybe_skip_all_sanitize_leak () {\n>> \tadd_sanitize_leak_true 't000*'\n>> \tadd_sanitize_leak_true 't001*'\n>> \tadd_sanitize_leak_true 't006*'\n>> +\tadd_sanitize_leak_true 't13*config*'\n>> \t# Blacklist patterns (overrides whitelist)\n>> \tadd_sanitize_leak_false 't000[469]*'\n>> \n\n"},{"id":"430250","messageId":"c6c7cfea-8292-2034-f9d6-de350e6e2692@ahunt.org","threadId":"55888","inReplyTo":"patch-3.4-b7fb5d5a56-20210714T172251Z-avarab@gmail.com","subject":"Re: [PATCH v2 3/4] SANITIZE tests: fix memory leaks in t5701*, add to whitelist","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-07-15T17:37:02Z","receivedAt":"2021-07-15T17:37:13Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 14/07/2021 19:23, Ævar Arnfjörð Bjarmason wrote:\n> Fix a memory leak in a2ba162cda (object-info: support for retrieving\n> object info, 2021-04-20) which appears to have been based on a\n> misunderstanding of how the pkt-line.c API works, there is no need to\n> strdup() input to, it's just a printf()-like format function.\n> \n> This fixes a potentially large memory leak, since the number of OID\n> lines the \"object-info\" call can be arbitrarily large (or a small one\n> if the request is small).\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> protocol-caps.c | 5 +++--\n> t/t5701-git-serve.sh | 1 +\n> 2 files changed, 4 insertions(+), 2 deletions(-)\n> \n> diff --git a/protocol-caps.c b/protocol-caps.c\n> index 13a9e63a04..901b6795e4 100644\n> --- a/protocol-caps.c\n> +++ b/protocol-caps.c\n> @@ -69,9 +69,10 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n> \t\t\t}\n> \t\t}\n> \n> -\t\tpacket_writer_write(writer, \"%s\",\n> -\t\t\t\t strbuf_detach(&send_buffer, NULL));\n> +\t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n> +\t\tstrbuf_reset(&send_buffer);\n> \t}\n> +\tstrbuf_release(&send_buffer);\n> }\n\nGood catch! strbuf's seem to be a common source of leak, where either \nthe release is forgotten or detach is used incorrectly - and I'm tempted \nto try and implement some automated checks to catch those (I wonder if \ncoccicheck is powerful enough for this?).\n\n> ...\n"},{"id":"430251","messageId":"35b37777-a79b-6dce-eb45-f7cd9d569ddb@ahunt.org","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","subject":"Re: [PATCH v2 0/4] add a test mode for SANITIZE=leak, run it in CI","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-07-15T17:37:35Z","receivedAt":"2021-07-15T17:37:47Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"On 14/07/2021 19:23, Ævar Arnfjörð Bjarmason wrote:\n> As a follow-up to my recent thread asking if we had some test mode or\n> CI to test for memory leak regression (we don't), add such a test\n> mode, and run it in CI.\n> \n> Currently the two new CI targets take ~2-3 minutes to run in GitHub\n> CI, whereas the normal test targets take 20-30 minutes. The tests run\n> slower, but we have a small whitelist of test scripts that are OK.\n> \n> v2:\n> \n> * Fixes issues spotted by Đoàn Trần Công Danh and Eric Sunshine,\n> thanks both!\n> \n> * I got rid of the change to t0500, I saw it being flaky in GitHub\n> CI, and looks like there'll be other concurrent edits to that file,\n> so leaving it be.\n> \n> v1: http://lore.kernel.org/git/cover-0.4-0000000000-20210714T001007Z-avarab@gmail.com\n\n> \n> Ævar Arnfjörð Bjarmason (4):\n> tests: add a test mode for SANITIZE=leak, run it in CI\n> SANITIZE tests: fix memory leaks in t13*config*, add to whitelist\n> SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n> SANITIZE tests: fix leak in mailmap.c\n> \n\nThe leak fixes look good to me, modulo the UNLEAK as already commented \non in patch 2/4 - thank you!\n\nI don't feel qualified to review the test and CI related scripting, \nhopefully someone else will be able to look at those changes :).\n\nATB,\n\nAndrzej\n\n[...snip...]\n"},{"id":"430268","messageId":"YPCjTpumyh1P/DQj@coredump.intra.peff.net","threadId":"55888","inReplyTo":"patch-1.4-0795436a24-20210714T172251Z-avarab@gmail.com","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-15T21:06:22Z","receivedAt":"2021-07-15T21:06:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 14, 2021 at 07:23:51PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> While git can be compiled with SANITIZE=leak there has been no\n> corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n> fixed as one-offs without structured regression testing.\n\nThis opening puzzled me. I'm not sure I understand why we need a special\nGIT_TEST_* mode for it. If you do \"make SANITIZE=leak test\", then your\nbinaries will leak-check while running the tests.\n\nI.e., there is nothing that test-lib.sh itself needs to do differently\nto enable it.\n\nWhat we _do_ need is some mechanism of annotating to tests to say \"this\nis known to leak\", so that we can skip them for normal integration runs.\n\nAnd that is part of what's going on in this patch, but I'm not sure it\nis the simplest way to do it. The first question is: how do we want to\nannotate the tests. By marking individual scripts or tests in the\ntest-files themselves? Or by using a separate list of \"these scripts or\ntests are known to pass\"?\n\nIMHO the latter is preferable. It keeps the annotations out of the way\nof normal work (they are a temporary thing until we eventually pass the\nwhole suite leak free, but I expect they'll be with us for a while). The\ndownside is that the annotations may get out of sync with test numbers.\nBut if we are primarily annotating whole scripts (and not individual\ntests), then that is generally pretty stable.\n\nAnd with that in mind, can we just use an existing mechanism for picking\nwhich tests to run, and drive it externally from the CI job?\n\nWe already have GIT_SKIP_TESTS and --run. Those are perhaps a bit\nawkward for feeding huge lists to, and there is no environment\nequivalent for --run (so you can't trigger it easily from \"make test\").\nBut what if we could do something like:\n\n GIT_TEST_RUN_FROM=t/leak-free make SANITIZE=leak test\n\nand then t/leak-free contained the usual patterns like:\n\n t000*\n t1234.5\n\nand so on. That requires two new features in test-lib.sh:\n\n - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP\n (instead of just the command-line --run).\n\n - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a\n file rather than the environment (I suppose the caller could just\n stuff the contents into the variable, but I expect that test-lib.sh\n may want to pare down the entries that do not even apply to the\n current script for the sake of efficiency in checking each test).\n\nThat infrastructure would then be applicable to other cases, too. Or\neven just useful for using another list (or no list at all) when you\nare looking at whether other tests are leak-free or not.\n\n> This change add such a mode, we now have new\n> linux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\n> tests as linux-{clang,gcc}, except that almost all of them are\n> skipped.\n\nI'm not clear on what we expect to get out of running it with both clang\nand gcc. They should be producing identical results.\n\n-Peff\n"},{"id":"430270","messageId":"YPClS0fj2HOJE5nH@coredump.intra.peff.net","threadId":"55888","inReplyTo":"eebb4f74-b5e3-6c11-3b84-fcee1b876992@ahunt.org","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-15T21:14:51Z","receivedAt":"2021-07-15T21:14:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 14, 2021 at 08:42:27PM +0200, Andrzej Hunt wrote:\n\n> My other question is: if we are adding a new job - should it really be just\n> a leak checking job? Leak checking is just a subset of ASAN (Address\n> Sanitizer). And as discussed at [1] it's possible to run ASAN and UBSAN\n> (Undefined Behaviour Sanitizer) in the same build. I feel like it's much\n> more useful to first add a combined ASAN+UBSAN job, followed by enabling\n> leak-checking as part of ASAN in those jobs for known leak-free tests - as\n> opposed to only adding leak checking. We currently disable Leak checking for\n> ASAN here [2], but that could be made conditional on the test ID (i.e. check\n> an allowlist to enable leak checking for some tests)?\n\nI do think it's worth having an ASan+UBSan job. In the CI we use for our\ncustom fork of Git at GitHub, we run it for every pull request (and I do\nbring upstream any applicable fixes). It's kind of expensive compared to\na regular \"make test\", but probably not nearly as bad as just running\nthe regular test suite on Windows.\n\nAnd it's true that ASan can do leak-checking, too. In the long run, when\nwe are leak-free, I think it may make sense to combine the jobs. But in\nthe interim state where we can run the whole suite with ASan/UBSan, but\nnot with LSan, I think it's simpler to just keep them separate. That\nlets us just entirely skip tests or scripts in the leak-checking run. I\nhaven't measured, but I also expect that LSan is not much more expensive\nthan a regular run, so combining the two isn't that big a win).\n\nSo I do like your suggestion, but I think it just be orthogonal further\nto leak-checking.\n\n-Peff\n"},{"id":"430271","messageId":"YPCrvOce5qRWk6Rq@coredump.intra.peff.net","threadId":"55888","inReplyTo":"871ea493-e108-e748-0234-f929690ad2fd@ahunt.org","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-15T21:42:20Z","receivedAt":"2021-07-15T21:42:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 14, 2021 at 08:57:37PM +0200, Andrzej Hunt wrote:\n\n> > @@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> > \tif (!strcmp(var, \"core.attributesfile\"))\n> > \t\treturn git_config_pathname(&git_attributes_file, var, value);\n> > -\tif (!strcmp(var, \"core.hookspath\"))\n> > +\tif (!strcmp(var, \"core.hookspath\")) {\n> > +\t\tUNLEAK(git_hooks_path);\n> > \t\treturn git_config_pathname(&git_hooks_path, var, value);\n> > +\t}\n> \n> Why is the UNLEAK necessary here? We generally want to limit use of UNLEAK\n> to cmd_* functions or direct helpers. git_default_core_config() seems\n> generic enough that it could be called from anywhere, and using UNLEAK here\n> means we're potentially masking a real leak?\n> \n> IIUC the leak here happens because:\n> - git_hooks_path is a global variable - hence it's unlikely we'd ever\n> bother cleaning it up.\n> - git_default_core_config() gets called a first time with\n> core.hookspath, and we end up allocating new memory into\n> git_hooks_path.\n> - git_default_core_config() gets called again with core.hookspath,\n> and we overwrite git_hooks_path with a new string which leaks\n> the string that git_hooks_path used to point to.\n> \n> So I think the real fix is to free(git_hooks_path) instead of an UNLEAK?\n> (Looking at the surrounding code, it looks like the same pattern of leak\n> might be repeated for other similar globals - is it worth auditing those\n> while we're here?)\n\nThis is a common leak pattern in Git. We do something like:\n\n static const char *foo = \"default\";\n ...\n int config_cb(const char *var, const char *value, void *)\n {\n if (!strcmp(var, \"core.foo\"))\n\t foo = xstrdup(value);\n }\n\nSo we leak if the variable appears twice. But we can't just call\n\"free(foo)\" here. In the first call, it's pointing to a string literal!\n\nIn the case of git_hooks_path, it defaults to NULL, so this works out\nOK. But it's setting up a trap for somebody later on, who assigns it a\ndefault value (and the compiler won't help; it's a \"const char *\", so\nthe assignment is fine, and the free() would already be casting away the\nconstness).\n\nI see a few possible solutions:\n\n - instead of strdup'ing long-lived config values, strintern() them.\n This is really leaking them, but in a way that we hold on to the old\n values. This is actually more or less what UNLEAK() is doing under\n the hood (saving a reference to the old buffer, even the variable is\n overwritten).\n\n - find a way to tell when a string comes from the heap versus a\n literal. I don't think you can do this portably without keeping your\n own separate flag. We could abstract away some of the pain with a\n struct like:\n\n struct def_string {\n /* might point to heap memory; const because you must\n * check flag before modifying */\n const char *value;\n int from_heap;\n }\n\n /* regular static initialization is OK if you don't want a default */\n #define DEF_STRING_INIT(str) { .value = str }\n\n static void def_string_set(struct def_string *ds, const char *value)\n {\n if (ds->from_heap)\n free(ds->value);\n ds->value = xstrdup(value);\n ds->from_heap = 1;\n }\n\n The annoying thing is all of the users need to refer to\n git_hook_path.value instead of just git_hook_path. If you don't mind\n a little macro hackery, we could get around that by declaring pairs\n of variables. Like:\n\n #define DEF_STRING_DECLARE(name, value) \\\n const char *name = value; \\\n int name##_from_heap\n\n #define DEF_STRING_SET(name, value) do { \\\n if (name##_from_heap) \\\n free(name); \\\n name = xstrdup(value); \\\n name##_from_heap = 1; \\\n } while(0)\n\nI can't say I _love_ any of that, but I think it would work (and\nprobably we'd adapt our helpers like git_config_pathname() to take a\ndef_string. Or I guess just have a def_string_free() which can be called\nbefore writing into them).\n\nBut maybe there's a better solution I'm missing.\n\n-Peff\n"},{"id":"430272","messageId":"YPCsDLoiiAG/C/ft@coredump.intra.peff.net","threadId":"55888","inReplyTo":"patch-3.4-b7fb5d5a56-20210714T172251Z-avarab@gmail.com","subject":"Re: [PATCH v2 3/4] SANITIZE tests: fix memory leaks in t5701*, add to whitelist","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-15T21:43:40Z","receivedAt":"2021-07-15T21:43:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 14, 2021 at 07:23:53PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Fix a memory leak in a2ba162cda (object-info: support for retrieving\n> object info, 2021-04-20) which appears to have been based on a\n> misunderstanding of how the pkt-line.c API works, there is no need to\n> strdup() input to, it's just a printf()-like format function.\n> \n> This fixes a potentially large memory leak, since the number of OID\n> lines the \"object-info\" call can be arbitrarily large (or a small one\n> if the request is small).\n\nVery nice. This will also be much more efficient, since we get to reuse\nthe same buffer in each run through the loop.\n\n-Peff\n"},{"id":"430293","messageId":"07624f34-2c0d-d577-e4bc-ab8ebe146ffe@ahunt.org","threadId":"55888","inReplyTo":"YPCrvOce5qRWk6Rq@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-07-16T05:18:59Z","receivedAt":"2021-07-16T05:19:15Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 15/07/2021 23:42, Jeff King wrote:\n> On Wed, Jul 14, 2021 at 08:57:37PM +0200, Andrzej Hunt wrote:\n> \n>>> @@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>>> \tif (!strcmp(var, \"core.attributesfile\"))\n>>> \t\treturn git_config_pathname(&git_attributes_file, var, value);\n>>> -\tif (!strcmp(var, \"core.hookspath\"))\n>>> +\tif (!strcmp(var, \"core.hookspath\")) {\n>>> +\t\tUNLEAK(git_hooks_path);\n>>> \t\treturn git_config_pathname(&git_hooks_path, var, value);\n>>> +\t}\n>>\n>> Why is the UNLEAK necessary here? We generally want to limit use of UNLEAK\n>> to cmd_* functions or direct helpers. git_default_core_config() seems\n>> generic enough that it could be called from anywhere, and using UNLEAK here\n>> means we're potentially masking a real leak?\n>>\n>> IIUC the leak here happens because:\n>> - git_hooks_path is a global variable - hence it's unlikely we'd ever\n>> bother cleaning it up.\n>> - git_default_core_config() gets called a first time with\n>> core.hookspath, and we end up allocating new memory into\n>> git_hooks_path.\n>> - git_default_core_config() gets called again with core.hookspath,\n>> and we overwrite git_hooks_path with a new string which leaks\n>> the string that git_hooks_path used to point to.\n>>\n>> So I think the real fix is to free(git_hooks_path) instead of an UNLEAK?\n>> (Looking at the surrounding code, it looks like the same pattern of leak\n>> might be repeated for other similar globals - is it worth auditing those\n>> while we're here?)\n> \n> This is a common leak pattern in Git. We do something like:\n> \n> static const char *foo = \"default\";\n> ...\n> int config_cb(const char *var, const char *value, void *)\n> {\n> if (!strcmp(var, \"core.foo\"))\n> \t foo = xstrdup(value);\n> }\n> \n> So we leak if the variable appears twice. But we can't just call\n> \"free(foo)\" here. In the first call, it's pointing to a string literal!\n> \n> In the case of git_hooks_path, it defaults to NULL, so this works out\n> OK. But it's setting up a trap for somebody later on, who assigns it a\n> default value (and the compiler won't help; it's a \"const char *\", so\n> the assignment is fine, and the free() would already be casting away the\n> constness).\n\nAh, right. I didn't think about the risk of future breakages.\n\n> \n> I see a few possible solutions:\n> [...]\n> I can't say I _love_ any of that, but I think it would work (and\n> probably we'd adapt our helpers like git_config_pathname() to take a\n> def_string. Or I guess just have a def_string_free() which can be called\n> before writing into them).\n\nIs it worth sidestepping the whole globals issue by migrating \ncore.hookspath (and other string config values) to be fetched via \ngit_config_get_pathname() and equivalents at the point of use instead?\n\nI looked at the commit below which introduced git_config_get* which \nsuggests that these methods were indeed intended to be an improvement \nover the callback based API, and IIUC switching over should have a bunch \nof advantages:\n - Removes some potential bugs that can happen if git_config() was never\n called with the right callback.\n - Potentially reduces the number of times we have to iterate over the\n config in the first place (assuming we migrate *all* config access\n and not just strings).\n - Fewer globals - which reduces potential for such leaks (and probably\n makes it easier to read the code in the first place).\nOTOH I'm not familiar enough with this code to know what the \ndisadvantages of such a migration might be (it's definitely going to be \na lot of work... but that's going to apply to any of the approaches we \ncan choose to fix these leaks).\n\ngit_config_get* were introduced in:\n 3c8687a73e (add `config_set` API for caching config-like files, \n2014-07-28)\n"},{"id":"430307","messageId":"87wnpqy8zd.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"YPCrvOce5qRWk6Rq@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-16T07:46:33Z","receivedAt":"2021-07-16T08:10:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 15 2021, Jeff King wrote:\n\n> On Wed, Jul 14, 2021 at 08:57:37PM +0200, Andrzej Hunt wrote:\n>\n>> > @@ -1331,8 +1336,10 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>> > \tif (!strcmp(var, \"core.attributesfile\"))\n>> > \t\treturn git_config_pathname(&git_attributes_file, var, value);\n>> > -\tif (!strcmp(var, \"core.hookspath\"))\n>> > +\tif (!strcmp(var, \"core.hookspath\")) {\n>> > +\t\tUNLEAK(git_hooks_path);\n>> > \t\treturn git_config_pathname(&git_hooks_path, var, value);\n>> > +\t}\n>> \n>> Why is the UNLEAK necessary here? We generally want to limit use of UNLEAK\n>> to cmd_* functions or direct helpers. git_default_core_config() seems\n>> generic enough that it could be called from anywhere, and using UNLEAK here\n>> means we're potentially masking a real leak?\n>> \n>> IIUC the leak here happens because:\n>> - git_hooks_path is a global variable - hence it's unlikely we'd ever\n>> bother cleaning it up.\n>> - git_default_core_config() gets called a first time with\n>> core.hookspath, and we end up allocating new memory into\n>> git_hooks_path.\n>> - git_default_core_config() gets called again with core.hookspath,\n>> and we overwrite git_hooks_path with a new string which leaks\n>> the string that git_hooks_path used to point to.\n>> \n>> So I think the real fix is to free(git_hooks_path) instead of an UNLEAK?\n>> (Looking at the surrounding code, it looks like the same pattern of leak\n>> might be repeated for other similar globals - is it worth auditing those\n>> while we're here?)\n>\n> This is a common leak pattern in Git. We do something like:\n>\n> static const char *foo = \"default\";\n> ...\n> int config_cb(const char *var, const char *value, void *)\n> {\n> if (!strcmp(var, \"core.foo\"))\n> \t foo = xstrdup(value);\n> }\n>\n> So we leak if the variable appears twice. But we can't just call\n> \"free(foo)\" here. In the first call, it's pointing to a string literal!\n>\n> In the case of git_hooks_path, it defaults to NULL, so this works out\n> OK. But it's setting up a trap for somebody later on, who assigns it a\n> default value (and the compiler won't help; it's a \"const char *\", so\n> the assignment is fine, and the free() would already be casting away the\n> constness).\n>\n> I see a few possible solutions:\n>\n> - instead of strdup'ing long-lived config values, strintern() them.\n> This is really leaking them, but in a way that we hold on to the old\n> values. This is actually more or less what UNLEAK() is doing under\n> the hood (saving a reference to the old buffer, even the variable is\n> overwritten).\n>\n> - find a way to tell when a string comes from the heap versus a\n> literal. I don't think you can do this portably without keeping your\n> own separate flag. We could abstract away some of the pain with a\n> struct like:\n>\n> struct def_string {\n> /* might point to heap memory; const because you must\n> * check flag before modifying */\n> const char *value;\n> int from_heap;\n> }\n>\n> /* regular static initialization is OK if you don't want a default */\n> #define DEF_STRING_INIT(str) { .value = str }\n>\n> static void def_string_set(struct def_string *ds, const char *value)\n> {\n> if (ds->from_heap)\n> free(ds->value);\n> ds->value = xstrdup(value);\n> ds->from_heap = 1;\n> }\n>\n> The annoying thing is all of the users need to refer to\n> git_hook_path.value instead of just git_hook_path. If you don't mind\n> a little macro hackery, we could get around that by declaring pairs\n> of variables. Like:\n>\n> #define DEF_STRING_DECLARE(name, value) \\\n> const char *name = value; \\\n> int name##_from_heap\n>\n> #define DEF_STRING_SET(name, value) do { \\\n> if (name##_from_heap) \\\n> free(name); \\\n> name = xstrdup(value); \\\n> name##_from_heap = 1; \\\n> } while(0)\n>\n> I can't say I _love_ any of that, but I think it would work (and\n> probably we'd adapt our helpers like git_config_pathname() to take a\n> def_string. Or I guess just have a def_string_free() which can be called\n> before writing into them).\n>\n> But maybe there's a better solution I'm missing.\n\nInstead of: \"int from_heap\" in your \"def_string\" I think we should just\nuse \"struct string_list_item\". I.e. you want a void* here. Why?\n\n\n\nI have an unsent series for handling some more common cases in the\nstring-list API. I started writing it due to a very related problem,\ni.e. that we conflate \"string init dup/nodup\" with \"do we want to\nfree?\".\n\nWe (ab)use the \"strdup_strings\" in a few places to free that sort of\nthing at the end if we have heap-allocated strings, but ones we did not\nstrdup ourselves, e.g. this in merge-ort.c (not picking on Elijah (CC'd)\nhere, it's common in lots of places, and this one was pretty much lifted\nfrom merge-recursive).\n\n opti->paths_to_free.strdup_strings = 1;\n string_list_clear(&opti->paths_to_free, 0);\n opti->paths_to_free.strdup_strings = 0;\n\nSo I improved the string-list and strmap free functions so you can\ninstead do:\n\n string_list_clear_strings((&opti->paths_to_free, 0);\n\nAnd that along with some other changes allows you to clear (or not) any\ncombination of the string, util, or have a callback function of your own\nrun (but be ensured to run all of those before we get to any of the\nother freeing).\n\n\n\nYou must be thinking what any of this has to do with heap strings in C,\nwell one common case you've not discussed is that we sometimes do the\nequivalent of, with string-list.h or not (somewhat pseudocode);\n\n\tvoid add_to_list(struct string_list *list, char *on_heap_now_we_own_it)\n\t{\n\t\tchar *ptr = on_heap_now_we_own_it;\n\t\tchar *mydup = xstrdup(\"foo\");\n\n\t ptr++; /* skip first byte */\n\t\tstring_list_append(list, ptr);\n\t\tstring_list_append(list, mydup);\n\t}\n\nAnd:\n\n struct string_list list = STRING_LIST_INIT_NODUP;\n /* other stuff here, we get strings from somewhere etc. */\n add_to_list(list, some_string);\n\nSo now you're left with needing to free both at the end, but we since we\ndid ptr++ there we can't free() that (we'd need to free(ptr - 1), but\nhow to keep track of that?).\n\nWell, tying this back to my clear() improvements for string-list.h I\nthought a really neat solution to this was:\n\n string_list_append(list, ptr)->util = on_heap_now_we_own_it;\n string_list_append(list, mydup)->util = mydup;\n\nI.e. by convention we store the pointer we need to free (if any) in the\n\"util\" field.\n\nAnd then if you get a string not from the heap you just leave the \"util\"\nas NULL, and at the end you just free() all your \"util\" fields, and it\njust so happens that some of them are the same as the \"string\" field.\n\nWe're not in the habit of passing loose \"string_list_item\" around now,\nbut I don't see why we wouldn't (possibly with a change to extract that\nbit out, so we could use it in other places).\n\nThe neat thing about doing this is also that you're not left with every\nAPI boundary needing to deal with your new \"def_string\", a lot of them\nuse string_list already, and hardly need to change anything, to the\nextent that we do need to change anything having a \"void *util\" is a lot\nmore generally usable. You end up getting memory management for free as\nyou gain a feature to pass arbitrary data along with your items.\n"},{"id":"430341","messageId":"877dhqxqbt.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"YPCjTpumyh1P/DQj@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-16T14:46:12Z","receivedAt":"2021-07-16T14:53:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 15 2021, Jeff King wrote:\n\n> On Wed, Jul 14, 2021 at 07:23:51PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> While git can be compiled with SANITIZE=leak there has been no\n>> corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n>> fixed as one-offs without structured regression testing.\n>\n> This opening puzzled me. I'm not sure I understand why we need a special\n> GIT_TEST_* mode for it. If you do \"make SANITIZE=leak test\", then your\n> binaries will leak-check while running the tests.\n>\n> I.e., there is nothing that test-lib.sh itself needs to do differently\n> to enable it.\n>\n> What we _do_ need is some mechanism of annotating to tests to say \"this\n> is known to leak\", so that we can skip them for normal integration runs.\n>\n> And that is part of what's going on in this patch, but I'm not sure it\n> is the simplest way to do it. The first question is: how do we want to\n> annotate the tests. By marking individual scripts or tests in the\n> test-files themselves? Or by using a separate list of \"these scripts or\n> tests are known to pass\"?\n>\n> IMHO the latter is preferable. It keeps the annotations out of the way\n> of normal work (they are a temporary thing until we eventually pass the\n> whole suite leak free, but I expect they'll be with us for a while). The\n> downside is that the annotations may get out of sync with test numbers.\n> But if we are primarily annotating whole scripts (and not individual\n> tests), then that is generally pretty stable.\n>\n> And with that in mind, can we just use an existing mechanism for picking\n> which tests to run, and drive it externally from the CI job?\n>\n> We already have GIT_SKIP_TESTS and --run. Those are perhaps a bit\n> awkward for feeding huge lists to, and there is no environment\n> equivalent for --run (so you can't trigger it easily from \"make test\").\n> But what if we could do something like:\n>\n> GIT_TEST_RUN_FROM=t/leak-free make SANITIZE=leak test\n>\n> and then t/leak-free contained the usual patterns like:\n>\n> t000*\n> t1234.5\n>\n> and so on. That requires two new features in test-lib.sh:\n>\n> - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP\n> (instead of just the command-line --run).\n>\n> - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a\n> file rather than the environment (I suppose the caller could just\n> stuff the contents into the variable, but I expect that test-lib.sh\n> may want to pare down the entries that do not even apply to the\n> current script for the sake of efficiency in checking each test).\n>\n> That infrastructure would then be applicable to other cases, too. Or\n> even just useful for using another list (or no list at all) when you\n> are looking at whether other tests are leak-free or not.\n\nI've included a mechanism for whitelisting specific globs, the idea was\nnot to have that be too detailed, but we'd e.g. get to the point of t00*\nor whatever passing.\n\nAnything that's a lot more granular than that is doing to suck,\ne.g. exposing teh GIT_TEST_SKIP and --run features. of specific test\nnumbers, now you need to count your tests if you add one in the middle\nof one of those, and more likely you won't test under the mode and just\nsee it in CI.\n\nThe marking at a distance I've done also has that problem in theory, but\nI think in practice we'll use it carefully for globs of tests unlikely\nto break.\n\nThis whole thing is much more with the GIT_TEST_SANITIZE_LEAK mode, it's\na really common case that we e.g. leak in some revision.c API user, we\nshould fix that, but holding up marking the rest of at test whose entire\ntests otherwise pass is bad, it means you can't do any testing of a\ngiven API or subsystem without getting the entire file to pass.\n\nWhereas while we're fixing very common leaks in the codabase it's likely\nthat any given test file will have a few such tests.\n\nIt also means everything works by default, you get an appropriate notice\nfrom prove(1), and even if you run one test manually it'll skip, but\nemit a message saying you can set the env var to force its run.\n\n>> This change add such a mode, we now have new\n>> linux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\n>> tests as linux-{clang,gcc}, except that almost all of them are\n>> skipped.\n>\n> I'm not clear on what we expect to get out of running it with both clang\n> and gcc. They should be producing identical results.\n\nIndeed, addressed elsewhere, i.e. it's just a thinko of mine.\n"},{"id":"430366","messageId":"YPHLUsXsWO+JYS5X@coredump.intra.peff.net","threadId":"55888","inReplyTo":"877dhqxqbt.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-16T18:09:22Z","receivedAt":"2021-07-16T18:09:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 16, 2021 at 04:46:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > and so on. That requires two new features in test-lib.sh:\n> >\n> > - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP\n> > (instead of just the command-line --run).\n> >\n> > - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a\n> > file rather than the environment (I suppose the caller could just\n> > stuff the contents into the variable, but I expect that test-lib.sh\n> > may want to pare down the entries that do not even apply to the\n> > current script for the sake of efficiency in checking each test).\n> >\n> > That infrastructure would then be applicable to other cases, too. Or\n> > even just useful for using another list (or no list at all) when you\n> > are looking at whether other tests are leak-free or not.\n> \n> I've included a mechanism for whitelisting specific globs, the idea was\n> not to have that be too detailed, but we'd e.g. get to the point of t00*\n> or whatever passing.\n> \n> Anything that's a lot more granular than that is doing to suck,\n> e.g. exposing teh GIT_TEST_SKIP and --run features. of specific test\n> numbers, now you need to count your tests if you add one in the middle\n> of one of those, and more likely you won't test under the mode and just\n> see it in CI.\n\nI think you can do the same level of skipping with GIT_TEST_SKIP,\nthough. My argument was just that adding a new mechanism does not make\nsense when we already have one. I.e., running:\n\n GIT_SKIP_TESTS='\n t[123456789]*\n t0[^0]*\n t00[^016]*\n t000[469]\n t001[2459]\n t006[0248]\n ' make SANITIZE=leak test\n\nworks already to do the same thing. The only thing we might want is a\nnicer syntax (e.g., to allow positive and negative patterns, or to read\nfrom a file). But that would benefit all users of GIT_SKIP_TESTS, not\njust people interested in leaks.\n\n> It also means everything works by default, you get an appropriate notice\n> from prove(1), and even if you run one test manually it'll skip, but\n> emit a message saying you can set the env var to force its run.\n\nWith GIT_SKIP_TESTS you obviously don't get a message saying \"try\nskipping this test\" when it fails. :) But IMHO that is not that big a\ndeal. You'll get a test failure with good LSan output. If you are\nworking on expanding leak-checker coverage, you already know about your\noptions for skipping. If you're adding a new test that leaks, you might\nconsider fixing the leak (though not always, if it's far from code\nyou're touching).\n\n-Peff\n"},{"id":"430373","messageId":"YPHT5N8ManGNTfMh@coredump.intra.peff.net","threadId":"55888","inReplyTo":"YPHLUsXsWO+JYS5X@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-16T18:45:56Z","receivedAt":"2021-07-16T18:45:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 16, 2021 at 02:09:22PM -0400, Jeff King wrote:\n\n> > Anything that's a lot more granular than that is doing to suck,\n> > e.g. exposing teh GIT_TEST_SKIP and --run features. of specific test\n> > numbers, now you need to count your tests if you add one in the middle\n> > of one of those, and more likely you won't test under the mode and just\n> > see it in CI.\n> \n> I think you can do the same level of skipping with GIT_TEST_SKIP,\n> though. My argument was just that adding a new mechanism does not make\n> sense when we already have one. I.e., running:\n> \n> GIT_SKIP_TESTS='\n> t[123456789]*\n> t0[^0]*\n> t00[^016]*\n> t000[469]\n> t001[2459]\n> t006[0248]\n> ' make SANITIZE=leak test\n> \n> works already to do the same thing. The only thing we might want is a\n> nicer syntax (e.g., to allow positive and negative patterns, or to read\n> from a file). But that would benefit all users of GIT_SKIP_TESTS, not\n> just people interested in leaks.\n\nI cheated a little here; an unrelated bug does cause a failure in t0000\nwith this pattern. I've just sent:\n\n https://lore.kernel.org/git/YPHTY5G9JaQFKlX5@coredump.intra.peff.net/\n\nto fix it.\n\n-Peff\n"},{"id":"430375","messageId":"87v95aw0a6.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"YPHLUsXsWO+JYS5X@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-16T18:56:04Z","receivedAt":"2021-07-16T19:01:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 16 2021, Jeff King wrote:\n\n> On Fri, Jul 16, 2021 at 04:46:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > and so on. That requires two new features in test-lib.sh:\n>> >\n>> > - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP\n>> > (instead of just the command-line --run).\n>> >\n>> > - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a\n>> > file rather than the environment (I suppose the caller could just\n>> > stuff the contents into the variable, but I expect that test-lib.sh\n>> > may want to pare down the entries that do not even apply to the\n>> > current script for the sake of efficiency in checking each test).\n>> >\n>> > That infrastructure would then be applicable to other cases, too. Or\n>> > even just useful for using another list (or no list at all) when you\n>> > are looking at whether other tests are leak-free or not.\n>> \n>> I've included a mechanism for whitelisting specific globs, the idea was\n>> not to have that be too detailed, but we'd e.g. get to the point of t00*\n>> or whatever passing.\n>> \n>> Anything that's a lot more granular than that is doing to suck,\n>> e.g. exposing teh GIT_TEST_SKIP and --run features. of specific test\n>> numbers, now you need to count your tests if you add one in the middle\n>> of one of those, and more likely you won't test under the mode and just\n>> see it in CI.\n>\n> I think you can do the same level of skipping with GIT_TEST_SKIP,\n> though. My argument was just that adding a new mechanism does not make\n> sense when we already have one. I.e., running:\n>\n> GIT_SKIP_TESTS='\n> t[123456789]*\n> t0[^0]*\n> t00[^016]*\n> t000[469]\n> t001[2459]\n> t006[0248]\n> ' make SANITIZE=leak test\n>\n> works already to do the same thing. The only thing we might want is a\n> nicer syntax (e.g., to allow positive and negative patterns, or to read\n> from a file). But that would benefit all users of GIT_SKIP_TESTS, not\n> just people interested in leaks.\n\nA glob in this series is t13*config*, you can't do that with\nGIT_SKIP_TESTS because it only includes the numeric part of the test,\ni.e. t1300, not t1300-config, or t1306-xdg-files.\n\nBut sure, it could happen via some other mechanism than the exact one I\npicked, or we could add GIT_SKIP_TESTS2 or whatever.\n\nI would like to be able to compile with it and run \"make test\" without a\nwall of failures by default, i.e. we should be able to tell regressions\nfrom known-OK to get anywhere with it, but that's orthagonal to the\nexact mechanism.\n\n>> It also means everything works by default, you get an appropriate notice\n>> from prove(1), and even if you run one test manually it'll skip, but\n>> emit a message saying you can set the env var to force its run.\n>\n> With GIT_SKIP_TESTS you obviously don't get a message saying \"try\n> skipping this test\" when it fails. :) But IMHO that is not that big a\n> deal. You'll get a test failure with good LSan output. If you are\n> working on expanding leak-checker coverage, you already know about your\n> options for skipping. If you're adding a new test that leaks, you might\n> consider fixing the leak (though not always, if it's far from code\n> you're touching).\n\nI do think it makes sense as a test mode test-lib.sh is aware of,\ne.g. on obvious next step is to not fail everything right away, but just\nlet the test run and log all failures to a file, then e.g. fail one test\nat the end, or if we're running in that mode collate all the callstacks\nand emit a summary for the whole test run.\n\nBut yes, the message it emits now isn't such a big deal.\n"},{"id":"430380","messageId":"YPHcagVJRlN7p/8S@coredump.intra.peff.net","threadId":"55888","inReplyTo":"87v95aw0a6.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-16T19:22:18Z","receivedAt":"2021-07-16T19:22:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 16, 2021 at 08:56:04PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > I think you can do the same level of skipping with GIT_TEST_SKIP,\n> > though. My argument was just that adding a new mechanism does not make\n> > sense when we already have one. I.e., running:\n> >\n> > GIT_SKIP_TESTS='\n> > t[123456789]*\n> > t0[^0]*\n> > t00[^016]*\n> > t000[469]\n> > t001[2459]\n> > t006[0248]\n> > ' make SANITIZE=leak test\n> >\n> > works already to do the same thing. The only thing we might want is a\n> > nicer syntax (e.g., to allow positive and negative patterns, or to read\n> > from a file). But that would benefit all users of GIT_SKIP_TESTS, not\n> > just people interested in leaks.\n> \n> A glob in this series is t13*config*, you can't do that with\n> GIT_SKIP_TESTS because it only includes the numeric part of the test,\n> i.e. t1300, not t1300-config, or t1306-xdg-files.\n\nThat seems like a feature that GIT_SKIP_TESTS could learn (though IMHO\njust using the test number in your patterns is sufficient).\n\n> I would like to be able to compile with it and run \"make test\" without a\n> wall of failures by default, i.e. we should be able to tell regressions\n> from known-OK to get anywhere with it, but that's orthagonal to the\n> exact mechanism.\n\nRight, I definitely agree on the goal. I just don't see the need to add\na new, very-specific mechanism. The skip-list above is gross and\nobviously not something you'd want to type. Driving it from a ci script\nis not too bad, but I agree people who want to leak-check locally would\nwant an easy way to use it, too. That's why I suggested extending it to\na file that could be easily specified (and possibly even auto-triggered\nin the Makefile by SANITIZE=leak).\n\n> > With GIT_SKIP_TESTS you obviously don't get a message saying \"try\n> > skipping this test\" when it fails. :) But IMHO that is not that big a\n> > deal. You'll get a test failure with good LSan output. If you are\n> > working on expanding leak-checker coverage, you already know about your\n> > options for skipping. If you're adding a new test that leaks, you might\n> > consider fixing the leak (though not always, if it's far from code\n> > you're touching).\n> \n> I do think it makes sense as a test mode test-lib.sh is aware of,\n> e.g. on obvious next step is to not fail everything right away, but just\n> let the test run and log all failures to a file, then e.g. fail one test\n> at the end, or if we're running in that mode collate all the callstacks\n> and emit a summary for the whole test run.\n\nThat's a more compelling reason, if we did implement that feature. My\nhope was that all of this would be a temporary state, though, and we'd\nget to a point where you can simply run \"make SANITIZE=leak test\" and\nactually run all of the tests. And then such a feature would not be that\ninteresting, because failures would be rare and cause for immediate\nhuman attention.\n\n-Peff\n"},{"id":"430393","messageId":"YPH3OOOfK9RVebqZ@coredump.intra.peff.net","threadId":"55888","inReplyTo":"87wnpqy8zd.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-16T21:16:40Z","receivedAt":"2021-07-16T21:16:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 16, 2021 at 09:46:33AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > I can't say I _love_ any of that, but I think it would work (and\n> > probably we'd adapt our helpers like git_config_pathname() to take a\n> > def_string. Or I guess just have a def_string_free() which can be called\n> > before writing into them).\n> >\n> > But maybe there's a better solution I'm missing.\n> \n> Instead of: \"int from_heap\" in your \"def_string\" I think we should just\n> use \"struct string_list_item\". I.e. you want a void* here. Why?\n\nYes, an equivalent way to write it is with a separate to_free buffer.\nBut why would we want it to be void? And why would we want to use a\nstring_list_item, which is otherwise unrelated?\n\n> Well, tying this back to my clear() improvements for string-list.h I\n> thought a really neat solution to this was:\n> \n> string_list_append(list, ptr)->util = on_heap_now_we_own_it;\n> string_list_append(list, mydup)->util = mydup;\n> \n> I.e. by convention we store the pointer we need to free (if any) in the\n> \"util\" field.\n\nThat works, but now \"util\" is not available for all the _other_ uses for\nwhich it was intended. And if we're not using it for those other uses,\nthen why does it need to exist at all? If we are only using it to hold\nthe allocated string pointer, then shouldn't it be \"char *to_free\"?\n\n> We're not in the habit of passing loose \"string_list_item\" around now,\n> but I don't see why we wouldn't (possibly with a change to extract that\n> bit out, so we could use it in other places).\n\nIt seems unnecessarily confusing to me. It sounds like you have a struct\nwhich just _happens_ to have a \"void *\" in it you can re-use, so you\nstart using it in lots of other places that are not in fact string lists\nat all. That is confusing to me on the face, but what happens when\nstring_list needs a feature which requires adding more fields to it?\n\nIf the point is to have a maybe-allocated string, why not make that a\ntype itself? And then if we want string_list to use it, it can.\n\n> The neat thing about doing this is also that you're not left with every\n> API boundary needing to deal with your new \"def_string\", a lot of them\n> use string_list already, and hardly need to change anything, to the\n> extent that we do need to change anything having a \"void *util\" is a lot\n> more generally usable. You end up getting memory management for free as\n> you gain a feature to pass arbitrary data along with your items.\n\nI don't think most interfaces take a string_list_item now, so wouldn't\nthey similarly need to be changed? Though the point is that all of these\ndegrade to a regular C-string, so when you are just passing the value\n(and not ownership), you would just dereference at that point.\n\n-Peff\n"},{"id":"430395","messageId":"YPH4Cf0OPni6sqql@coredump.intra.peff.net","threadId":"55888","inReplyTo":"07624f34-2c0d-d577-e4bc-ab8ebe146ffe@ahunt.org","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-16T21:20:09Z","receivedAt":"2021-07-16T21:20:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 16, 2021 at 07:18:59AM +0200, Andrzej Hunt wrote:\n\n> > I see a few possible solutions:\n> > [...]\n> > I can't say I _love_ any of that, but I think it would work (and\n> > probably we'd adapt our helpers like git_config_pathname() to take a\n> > def_string. Or I guess just have a def_string_free() which can be called\n> > before writing into them).\n> \n> Is it worth sidestepping the whole globals issue by migrating core.hookspath\n> (and other string config values) to be fetched via git_config_get_pathname()\n> and equivalents at the point of use instead?\n\nYeah, I almost suggested that. It probably does work OK in this\ninstance, but it's a much bigger change to convert all cases.\n\nI'm also not sure if you'd run into tricky details. For instance,\ncalling git_config_* is doing an expensive-ish lookup in the cached\nconfig (well, expensive to accessing a single pointer). So in cases\nwhere we're going to access the string many times (say, in a loop), we'd\nwant our own variable. That might end up being easy by calling it once\noutside the loop. Or it might not be, for cases where the variable is\nused under the hood by a helper function.\n\n-Peff\n"},{"id":"434261","messageId":"87y28hwylq.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"YPH3OOOfK9RVebqZ@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T12:47:01Z","receivedAt":"2021-08-31T13:08:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 16 2021, Jeff King wrote:\n\n[Very late reply, just getting back to this thread]\n\n> On Fri, Jul 16, 2021 at 09:46:33AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > I can't say I _love_ any of that, but I think it would work (and\n>> > probably we'd adapt our helpers like git_config_pathname() to take a\n>> > def_string. Or I guess just have a def_string_free() which can be called\n>> > before writing into them).\n>> >\n>> > But maybe there's a better solution I'm missing.\n>> \n>> Instead of: \"int from_heap\" in your \"def_string\" I think we should just\n>> use \"struct string_list_item\". I.e. you want a void* here. Why?\n>\n> Yes, an equivalent way to write it is with a separate to_free buffer.\n> But why would we want it to be void? And why would we want to use a\n> string_list_item, which is otherwise unrelated?\n\nWe could factor \"string_list_item\" out into a \"string_pair\" or\nwhatever.\n\nSorry, I didn't mean to get into the naming/code location aspect of the\ndiscussion, just the sensibility of using a \"char */void * pair\" for\nthese common memory management cases, v.s. your suggestion of having a\n\"char *, int from_heap\" pair.\n\n>> Well, tying this back to my clear() improvements for string-list.h I\n>> thought a really neat solution to this was:\n>> \n>> string_list_append(list, ptr)->util = on_heap_now_we_own_it;\n>> string_list_append(list, mydup)->util = mydup;\n>> \n>> I.e. by convention we store the pointer we need to free (if any) in the\n>> \"util\" field.\n>\n> That works, but now \"util\" is not available for all the _other_ uses for\n> which it was intended. And if we're not using it for those other uses,\n> then why does it need to exist at all? If we are only using it to hold\n> the allocated string pointer, then shouldn't it be \"char *to_free\"?\n\nBecause having it be \"char *\" doesn't cover the common case of\ne.g. getting an already allocated \"struct something *\" which contains\nyour string, setting the \"string\" in \"struct string_list_item\" to some\nstring in that struct, and the \"util\" to the struct itself, as we now\nown it and want to free() it later in its entirety.\n\nThat and the even more common case I mentioned upthread of wanting to\nferry around the truncated version of some char *, but still wanting to\naccount for the original for an eventual free().\n\nBut yes, if you want to account for freeing that data *and* have util\nset to something else you'll need to have e.g. your own wrapper struct\nand your own string_list_clear_func() callback.\n\nI'm not suggesting that this handles every possible scenario, just that\nhaving look at a lot of the code involved recently this seemed like a\nneat solution for the common cases.\n\n>> We're not in the habit of passing loose \"string_list_item\" around now,\n>> but I don't see why we wouldn't (possibly with a change to extract that\n>> bit out, so we could use it in other places).\n>\n> It seems unnecessarily confusing to me. It sounds like you have a struct\n> which just _happens_ to have a \"void *\" in it you can re-use, so you\n> start using it in lots of other places that are not in fact string lists\n> at all. That is confusing to me on the face, but what happens when\n> string_list needs a feature which requires adding more fields to it?\n>\n> If the point is to have a maybe-allocated string, why not make that a\n> type itself? And then if we want string_list to use it, it can.\n\n*nod*, covered above. My examples were unnecessarily confusing...\n\n>> The neat thing about doing this is also that you're not left with every\n>> API boundary needing to deal with your new \"def_string\", a lot of them\n>> use string_list already, and hardly need to change anything, to the\n>> extent that we do need to change anything having a \"void *util\" is a lot\n>> more generally usable. You end up getting memory management for free as\n>> you gain a feature to pass arbitrary data along with your items.\n>\n> I don't think most interfaces take a string_list_item now, so wouldn't\n> they similarly need to be changed? Though the point is that all of these\n> degrade to a regular C-string, so when you are just passing the value\n> (and not ownership), you would just dereference at that point.\n\nSure, just like things would need to be changed to handle your proposed\n\"struct def_string\".\n\nBy piggy-backing on an already used struct in our codebase we can get a\nlot of that memory management pretty much for free without much\nchurn.\n\nIf you squint and pretend that \"struct string_list_item\" isn't called\nsomething to do with that particular collections API (but it would make\nuse of it) then we've already set up most of the scaffolding and\nmanagement for this.\n"},{"id":"434268","messageId":"cover-v3-0.8-00000000000-20210831T132546Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.4-0000000000-20210714T172251Z-avarab@gmail.com","subject":"[PATCH v3 0/8] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:34Z","receivedAt":"2021-08-31T13:35:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We can compile git with SANITIZE=leak, and have had various efforts in\nthe past such as 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04)\nto plug memory leaks, but have had no CI testing of it to ensure that\nwe don't get regressions. This series adds a GIT_TEST_* mode for\nchecking those regressions, and runs it in CI.\n\nSince I submitted v2 the delta between origin/master..origin/seen\nbroke even t0001-init.sh when run under SANITIZE=leak, so this series\nwill cause test smoke on \"seen\". It's a prblom with another topic[1]\nthough, which I and Emily will fix.\n\nChanges since v2:\n\n * In v2 compiling with SANITIZE=leak would change things so only\n known-good passing tests were run by default, everything else would\n pass as a dummy. Now the default running of tests is unchanged, but\n if we run with GIT_TEST_PASSING_SANITIZE_LEAK=true only those tests\n are run which set and export TEST_PASSES_SANITIZE_LEAK=true.\n\n * The facility for declaring known-good tests in test-lib.sh based on\n wildcards is gone, instead individual tests need to declare if\n they're OK under SANITIZE=leak. This is done via \"export\n TEST_PASSES_SANITIZE_LEAK=true\", there's a handy import of\n \"./test-pragma-SANITIZE=leak-ok.sh\" before sourcing \"./test-lib.sh\"\n itself to set this.\n\n * The various leak fixes are gone entirely. I'll submit some of those\n independently or as follow-ups.\n\n * We now mark 57 tests in the test suite as OK under\n SANITIZE=leak. This is far from all of them, but gives us a decent\n set to start out with. The largest chunk of these is in t0*.sh.\n\n * The CI job is not run under both GCC and Clang, but just whatever\n with one default compiler (which happens to be GCC). I'd missed\n that running under both was pointless.\n\n It would be meaningful to run this under e.g. OSX & Windows too, as\n we take different codepaths there, but that can be left to a\n follow-up series.\n\n1. https://lore.kernel.org/git/8735qvyw0p.fsf@evledraar.gmail.com/\n\nÆvar Arnfjörð Bjarmason (8):\n Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n CI: refactor \"if\" to \"case\" statement\n tests: add a test mode for SANITIZE=leak, run it in CI\n tests: annotate t000*.sh with TEST_PASSES_SANITIZE_LEAK=true\n tests: annotate t001*.sh with TEST_PASSES_SANITIZE_LEAK=true\n tests: annotate t002*.sh with TEST_PASSES_SANITIZE_LEAK=true\n tests: annotate select t0*.sh with TEST_PASSES_SANITIZE_LEAK=true\n tests: annotate select t*.sh with TEST_PASSES_SANITIZE_LEAK=true\n\n .github/workflows/main.yml | 2 ++\n Makefile | 5 +++++\n ci/install-dependencies.sh | 4 ++--\n ci/lib.sh | 29 +++++++++++++++++--------\n ci/run-build-and-tests.sh | 4 ++--\n t/README | 7 ++++++\n t/t0000-basic.sh | 1 +\n t/t0001-init.sh | 1 +\n t/t0002-gitfile.sh | 1 +\n t/t0003-attributes.sh | 1 +\n t/t0004-unwritable.sh | 3 ++-\n t/t0005-signals.sh | 2 ++\n t/t0007-git-var.sh | 2 ++\n t/t0008-ignores.sh | 1 +\n t/t0010-racy-git.sh | 1 +\n t/t0011-hashmap.sh | 1 +\n t/t0013-sha1dc.sh | 1 +\n t/t0016-oidmap.sh | 1 +\n t/t0017-env-helper.sh | 1 +\n t/t0018-advice.sh | 1 +\n t/t0022-crlf-rename.sh | 1 +\n t/t0024-crlf-archive.sh | 1 +\n t/t0025-crlf-renormalize.sh | 1 +\n t/t0026-eol-config.sh | 1 +\n t/t0029-core-unsetenvvars.sh | 1 +\n t/t0030-stripspace.sh | 1 +\n t/t0052-simple-ipc.sh | 1 +\n t/t0061-run-command.sh | 1 +\n t/t0063-string-list.sh | 1 +\n t/t0066-dir-iterator.sh | 1 +\n t/t0067-parse_pathspec_file.sh | 1 +\n t/t0091-bugreport.sh | 1 +\n t/t1010-mktree.sh | 1 +\n t/t1100-commit-tree-options.sh | 1 +\n t/t1308-config-set.sh | 1 +\n t/t1309-early-config.sh | 1 +\n t/t1420-lost-found.sh | 1 +\n t/t1430-bad-ref-name.sh | 1 +\n t/t1509-root-work-tree.sh | 1 +\n t/t2002-checkout-cache-u.sh | 1 +\n t/t2050-git-dir-relative.sh | 1 +\n t/t2081-parallel-checkout-collisions.sh | 1 +\n t/t2100-update-cache-badpath.sh | 1 +\n t/t2200-add-update.sh | 1 +\n t/t2201-add-update-typechange.sh | 1 +\n t/t2202-add-addremove.sh | 1 +\n t/t2204-add-ignored.sh | 1 +\n t/t2300-cd-to-toplevel.sh | 1 +\n t/t3000-ls-files-others.sh | 1 +\n t/t3004-ls-files-basic.sh | 1 +\n t/t3006-ls-files-long.sh | 1 +\n t/t3008-ls-files-lazy-init-name-hash.sh | 1 +\n t/t3100-ls-tree-restrict.sh | 1 +\n t/t3101-ls-tree-dirname.sh | 1 +\n t/t3102-ls-tree-wildcards.sh | 1 +\n t/t3103-ls-tree-misc.sh | 1 +\n t/t3205-branch-color.sh | 1 +\n t/t3211-peel-ref.sh | 1 +\n t/t3300-funny-names.sh | 1 +\n t/t3902-quoted.sh | 1 +\n t/t4002-diff-basic.sh | 1 +\n t/t4026-color.sh | 1 +\n t/t4300-merge-tree.sh | 1 +\n t/test-lib.sh | 22 +++++++++++++++++++\n t/test-pragma-SANITIZE=leak-ok.sh | 8 +++++++\n 65 files changed, 128 insertions(+), 14 deletions(-)\n create mode 100644 t/test-pragma-SANITIZE=leak-ok.sh\n\nRange-diff against v2:\n-: ----------- > 1: 85619728d41 Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n-: ----------- > 2: 91c36b94eaa CI: refactor \"if\" to \"case\" statement\n1: df5a44e70b5 ! 3: 7e3577e4e3c tests: add a test mode for SANITIZE=leak, run it in CI\n @@ Commit message\n corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n fixed as one-offs without structured regression testing.\n \n - This change add such a mode, we now have new\n - linux-{clang,gcc}-sanitize-leak CI targets, these targets run the same\n - tests as linux-{clang,gcc}, except that almost all of them are\n - skipped.\n + This change add such a mode, and a new linux-SANITIZE=leak CI\n + target. The test mode and CI target only runs a whitelist of\n + known-good tests using a mechanism discussed below, to ensure that we\n + won't add regressions to code that's had its memory leaks fixed.\n \n - There is a whitelist of some tests that are OK in test-lib.sh, and\n - individual tests can be opted-in by setting\n - GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n - individual test can be skipped with the \"!SANITIZE_LEAK\"\n - prerequisite. See the updated t/README for more details.\n + The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n + mode. When running in that mode all tests except those that have opted\n + themselves in to running by setting and exporting\n + TEST_PASSES_SANITIZE_LEAK=true before sourcing test-lib.sh.\n \n - I'm using the GIT_TEST_SANITIZE_LEAK=true and !SANITIZE_LEAK pattern\n - in a couple of tests whose memory leaks I'll fix in subsequent\n - commits.\n + I'm adding a \"test-pragma-SANITIZE=leak-ok.sh\" wrapper for setting and\n + exporting that variable, as the assignment/export boilerplate would\n + otherwise get quite verbose and repetitive in subsequent commits.\n \n - I'm not being aggressive about opting in tests, it's not all tests\n - that currently pass under SANITIZE=leak, just a small number of\n - known-good tests. We can add more later as we fix leaks and grow more\n - confident in this test mode.\n + The tests using the \"test-pragma-SANITIZE=leak-ok.sh\" pragma can in\n + turn make use of the \"SANITIZE_LEAK\" prerequisite added in a preceding\n + commit, should they wish to selectively skip tests even under\n + \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n \n - See the recent discussion at [1] about the lack of this sort of test\n - mode, and 0e5bba53af (add UNLEAK annotation for reducing leak false\n - positives, 2017-09-08) for the initial addition of SANITIZE=leak.\n + Now tests that don't set the \"test-pragma-SANITIZE=leak-ok.sh\" pragma\n + will be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n +\n + $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n + 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n +\n + In subsequents commit we'll conservatively add more\n + TEST_PASSES_SANITIZE_LEAK=true annotations. The idea is that as memory\n + leaks are fixed we can add more known-good tests to this CI target, to\n + ensure that we won't have regressions.\n +\n + As of writing this we've got major regressions between master..seen,\n + i.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n + 'ah/plugleaks', 2021-08-04) have regressed recently.\n +\n + See the discussion at <87czsv2idy.fsf@evledraar.gmail.com> about the\n + lack of this sort of test mode, and 0e5bba53af (add UNLEAK annotation\n + for reducing leak false positives, 2017-09-08) for the initial\n + addition of SANITIZE=leak.\n \n See also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n 7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n past history of \"one-off\" SANITIZE=leak (and more) fixes.\n \n - When calling maybe_skip_all_sanitize_leak matching against\n - \"$TEST_NAME\" instead of \"$this_test\" as other \"match_pattern_list()\"\n - users do is intentional. I'd like to match things like \"t13*config*\"\n - in subsequent commits. This part of the API isn't public, so we can\n - freely change it in the future.\n -\n - 1. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n -\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## .github/workflows/main.yml ##\n @@ .github/workflows/main.yml: jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n -+ - jobname: linux-clang-sanitize-leak\n -+ cc: clang\n -+ pool: ubuntu-latest\n -+ - jobname: linux-gcc-sanitize-leak\n -+ cc: gcc\n ++ - jobname: linux-SANITIZE=leak\n + pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\n \n - ## Makefile ##\n -@@ Makefile: PTHREAD_CFLAGS =\n - SPARSE_FLAGS ?=\n - SP_EXTRA_FLAGS = -Wno-universal-initializer\n - \n -+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n -+SANITIZE_LEAK =\n -+\n - # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n - # usually result in less CPU usage at the cost of higher peak memory.\n - # Setting it to 0 will feed all files in a single spatch invocation.\n -@@ Makefile: BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n - endif\n - ifneq ($(filter leak,$(SANITIZERS)),)\n - BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n -+SANITIZE_LEAK = YesCompiledWithIt\n - endif\n - ifneq ($(filter address,$(SANITIZERS)),)\n - NO_REGEX = NeededForASAN\n -@@ Makefile: GIT-BUILD-OPTIONS: FORCE\n - \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n - \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n - \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n -+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n - \t@echo X=\\'$(X)\\' >>$@+\n - ifdef TEST_OUTPUT_DIRECTORY\n - \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n -\n ## ci/install-dependencies.sh ##\n @@ ci/install-dependencies.sh: UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n -linux-clang|linux-gcc)\n -+linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n ++linux-clang|linux-gcc|linux-SANITIZE=leak)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n -\tlinux-gcc)\n -+\tlinux-gcc|linux-gcc-sanitize-leak)\n ++\tlinux-gcc|linux-SANITIZE=leak)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n \n case \"$jobname\" in\n -linux-clang|linux-gcc)\n --\tif [ \"$jobname\" = linux-gcc ]\n --\tthen\n -+linux-clang|linux-gcc|linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n -+\tcase \"$jobname\" in\n -+\tlinux-gcc|linux-gcc-sanitize-leak)\n ++linux-clang|linux-gcc|linux-SANITIZE=leak)\n + \tcase \"$jobname\" in\n +-\tlinux-gcc)\n ++\tlinux-gcc|linux-SANITIZE=leak)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n --\telse\n -+\t\t;;\n -+\t*)\n - \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n --\tfi\n -+\t\t;;\n -+\tesac\n - \n - \texport GIT_TEST_HTTPD=true\n - \n + \t\t;;\n @@ ci/lib.sh: linux-musl)\n \t;;\n esac\n \n +case \"$jobname\" in\n -+linux-clang-sanitize-leak|linux-gcc-sanitize-leak)\n ++linux-SANITIZE=leak)\n +\texport SANITIZE=leak\n ++\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n +\t;;\n +esac\n +\n @@ ci/run-build-and-tests.sh: esac\n make\n case \"$jobname\" in\n -linux-gcc)\n -+linux-gcc|linux-gcc-sanitize-leak)\n ++linux-gcc|linux-SANITIZE=leak)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n @@ ci/run-build-and-tests.sh: linux-gcc)\n \tmake test\n \t;;\n -linux-clang)\n -+linux-clang|linux-clang-sanitize-leak)\n ++linux-clang|linux-SANITIZE=leak)\n \texport GIT_TEST_DEFAULT_HASH=sha1\n \tmake test\n \texport GIT_TEST_DEFAULT_HASH=sha256\n @@ t/README: GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n to and 'checkout.thresholdForParallelism' to 0, forcing the\n execution of the parallel-checkout code.\n \n -+GIT_TEST_SANITIZE_LEAK= will force the tests to run when git\n -+is compiled with SANITIZE=leak (we pick it up via\n -+../GIT-BUILD-OPTIONS).\n -+\n -+By default all tests are skipped when compiled with SANITIZE=leak, and\n -+individual test scripts opt themselves in to leak testing by setting\n -+GIT_TEST_SANITIZE_LEAK=true before sourcing test-lib.sh. Within those\n -+tests use the SANITIZE_LEAK prerequisite to skip individiual tests\n -+(i.e. test_expect_success !SANITIZE_LEAK [...]).\n -+\n -+So the GIT_TEST_SANITIZE_LEAK setting is different in behavior from\n -+both other GIT_TEST_*=[true|false] settings, but more useful given how\n -+SANITIZE=leak works & the state of the test suite. Manually setting\n -+GIT_TEST_SANITIZE_LEAK=true is only useful during development when\n -+finding and fixing memory leaks.\n ++GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n ++SANITIZE=leak will run only those tests that have whitelisted\n ++themselves as passing with no memory leaks. Do this by sourcing\n ++\"test-pragma-SANITIZE=leak-ok.sh\" before sourcing \"test-lib.sh\" itself\n ++at the top of the test script. This test mode is used by the\n ++\"linux-SANITIZE=leak\" CI target.\n +\n Naming Tests\n ------------\n \n \n - ## t/t5701-git-serve.sh ##\n -@@ t/t5701-git-serve.sh: test_expect_success 'unexpected lines are not allowed in fetch request' '\n + ## t/t0000-basic.sh ##\n +@@ t/t0000-basic.sh: swapping compression and hashing order, the person who is making the\n + modification *should* take notice and update the test vectors here.\n + '\n + \n ++. ./test-pragma-SANITIZE=leak-ok.sh\n + . ./test-lib.sh\n \n - # Test the basics of object-info\n - #\n --test_expect_success 'basics of object-info' '\n -+test_expect_success !SANITIZE_LEAK 'basics of object-info' '\n - \ttest-tool pkt-line pack >in <<-EOF &&\n - \tcommand=object-info\n - \tobject-format=$(test_oid algo)\n + try_local_xy () {\n \n ## t/test-lib.sh ##\n -@@ t/test-lib.sh: then\n - \texit 1\n - fi\n - \n -+# SANITIZE=leak test mode\n -+sanitize_leak_true=\n -+add_sanitize_leak_true () {\n -+\tsanitize_leak_true=\"$sanitize_leak_true$1 \"\n -+}\n -+\n -+sanitize_leak_false=\n -+add_sanitize_leak_false () {\n -+\tsanitize_leak_false=\"$sanitize_leak_false$1 \"\n -+}\n -+\n -+sanitize_leak_opt_in_msg=\"opt-in with GIT_TEST_SANITIZE_LEAK=true\"\n -+maybe_skip_all_sanitize_leak () {\n -+\t# Whitelist patterns\n -+\tadd_sanitize_leak_true 't000*'\n -+\tadd_sanitize_leak_true 't001*'\n -+\tadd_sanitize_leak_true 't006*'\n -+\n -+\t# Blacklist patterns (overrides whitelist)\n -+\tadd_sanitize_leak_false 't000[469]*'\n -+\tadd_sanitize_leak_false 't001[2459]*'\n -+\tadd_sanitize_leak_false 't006[0248]*'\n -+\n -+\tif match_pattern_list \"$1\" \"$sanitize_leak_false\"\n -+\tthen\n -+\t\tskip_all=\"test $this_test on SANITIZE=leak blacklist, $sanitize_leak_opt_in_msg\"\n -+\t\ttest_done\n -+\telif match_pattern_list \"$1\" \"$sanitize_leak_true\"\n -+\tthen\n -+\t\treturn 0\n -+\tfi\n -+\treturn 1\n -+}\n -+\n - # Are we running this test at all?\n - remove_trash=\n - this_test=${0##*/}\n @@ t/test-lib.sh: then\n \ttest_done\n fi\n @@ t/test-lib.sh: then\n +# SANITIZE=leak\n +if test -n \"$SANITIZE_LEAK\"\n +then\n -+\tif test -z \"$GIT_TEST_SANITIZE_LEAK\" &&\n -+\t\tmaybe_skip_all_sanitize_leak \"$TEST_NAME\"\n ++\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n +\tthen\n -+\t\tsay_color info >&3 \"test $this_test on SANITIZE=leak whitelist\"\n -+\t\tGIT_TEST_SANITIZE_LEAK=true\n -+\tfi\n ++\t\t# We need to see it in \"git env--helper\" (via\n ++\t\t# test_bool_env)\n ++\t\texport TEST_PASSES_SANITIZE_LEAK\n +\n -+\t# We need to see it in \"git env--helper\" (via\n -+\t# test_bool_env)\n -+\texport GIT_TEST_SANITIZE_LEAK\n -+\n -+\tif ! test_bool_env GIT_TEST_SANITIZE_LEAK false\n -+\tthen\n -+\t\tskip_all=\"skip all tests in $this_test under SANITIZE=leak, $sanitize_leak_opt_in_msg\"\n -+\t\ttest_done\n ++\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n ++\t\tthen\n ++\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n ++\t\t\ttest_done\n ++\t\tfi\n +\tfi\n -+elif test_bool_env GIT_TEST_SANITIZE_LEAK false\n ++elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n +then\n -+\terror \"GIT_TEST_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n ++\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n +fi\n +\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n -@@ t/test-lib.sh: test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n - test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n - test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n - test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n -+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n - \n - if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n - then\n +\n + ## t/test-pragma-SANITIZE=leak-ok.sh (new) ##\n +@@\n ++#!/bin/sh\n ++\n ++## This \"pragma\" (as in \"perldoc perlpragma\") declares that the test\n ++## will pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. Source this\n ++## before sourcing test-lib.sh\n ++\n ++TEST_PASSES_SANITIZE_LEAK=true\n ++export TEST_PASSES_SANITIZE_LEAK\n2: 51fd1c400da < -: ----------- SANITIZE tests: fix memory leaks in t13*config*, add to whitelist\n3: 720852eee0b < -: ----------- SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n4: 80edda308c9 < -: ----------- SANITIZE tests: fix leak in mailmap.c\n-: ----------- > 4: 0cd14d64165 tests: annotate t000*.sh with TEST_PASSES_SANITIZE_LEAK=true\n-: ----------- > 5: ed5f5705755 tests: annotate t001*.sh with TEST_PASSES_SANITIZE_LEAK=true\n-: ----------- > 6: 2599016c4e7 tests: annotate t002*.sh with TEST_PASSES_SANITIZE_LEAK=true\n-: ----------- > 7: ddc4d6d2cf1 tests: annotate select t0*.sh with TEST_PASSES_SANITIZE_LEAK=true\n-: ----------- > 8: e611d2c23d9 tests: annotate select t*.sh with TEST_PASSES_SANITIZE_LEAK=true\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434267","messageId":"patch-v3-1.8-85619728d41-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 1/8] Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:35Z","receivedAt":"2021-08-31T13:35:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When SANITIZE=leak is specified we'll now add a SANITIZE_LEAK flag to\nGIT-BUILD-OPTIONS, this can then be picked up by the test-lib.sh,\nwhich sets a SANITIZE_LEAK prerequisite.\n\nWe can then skip specific tests that are known to fail under\nSANITIZE=leak, add one such annotation to t0004-unwritable.sh, which\nnow passes under SANITIZE=leak.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++++\n t/t0004-unwritable.sh | 2 +-\n t/test-lib.sh | 1 +\n 3 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex d1feab008fc..005d647d46e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1221,6 +1221,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1265,6 +1268,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2812,6 +2816,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex e3137d638ee..fbdcb926b3a 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -21,7 +21,7 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi\n \ttest_must_fail git write-tree\n '\n \n-test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository' '\n+test_expect_success POSIXPERM,SANITY,!SANITIZE_LEAK 'commit should notice unwritable repository' '\n \ttest_when_finished \"chmod 775 .git/objects .git/objects/??\" &&\n \tchmod a-w .git/objects .git/objects/?? &&\n \ttest_must_fail git commit -m second\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d61..4ab18914a3d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1533,6 +1533,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434269","messageId":"patch-v3-2.8-91c36b94eaa-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 2/8] CI: refactor \"if\" to \"case\" statement","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:36Z","receivedAt":"2021-08-31T13:35:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor an \"if\" statement for \"linux-gcc\" to a \"case\" statement in\npreparation for another case being added to it, and do the same for\nthe \"osx-gcc\" just below it for consistency.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..33b9777ab7e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -184,13 +184,15 @@ export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n linux-clang|linux-gcc)\n-\tif [ \"$jobname\" = linux-gcc ]\n-\tthen\n+\tcase \"$jobname\" in\n+\tlinux-gcc)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \texport GIT_TEST_HTTPD=true\n \n@@ -207,13 +209,15 @@ linux-clang|linux-gcc)\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n osx-clang|osx-gcc)\n-\tif [ \"$jobname\" = osx-gcc ]\n-\tthen\n+\tcase \"$jobname\" in\n+\tosx-gcc)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \t# t9810 occasionally fails on Travis CI OS X\n \t# t9816 occasionally fails with \"TAP out of sequence errors\" on\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434270","messageId":"patch-v3-3.8-7e3577e4e3c-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 3/8] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:37Z","receivedAt":"2021-08-31T13:35:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak there has been no\ncorresponding GIT_TEST_* mode for it, i.e. memory leaks have been\nfixed as one-offs without structured regression testing.\n\nThis change add such a mode, and a new linux-SANITIZE=leak CI\ntarget. The test mode and CI target only runs a whitelist of\nknown-good tests using a mechanism discussed below, to ensure that we\nwon't add regressions to code that's had its memory leaks fixed.\n\nThe CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\nmode. When running in that mode all tests except those that have opted\nthemselves in to running by setting and exporting\nTEST_PASSES_SANITIZE_LEAK=true before sourcing test-lib.sh.\n\nI'm adding a \"test-pragma-SANITIZE=leak-ok.sh\" wrapper for setting and\nexporting that variable, as the assignment/export boilerplate would\notherwise get quite verbose and repetitive in subsequent commits.\n\nThe tests using the \"test-pragma-SANITIZE=leak-ok.sh\" pragma can in\nturn make use of the \"SANITIZE_LEAK\" prerequisite added in a preceding\ncommit, should they wish to selectively skip tests even under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nNow tests that don't set the \"test-pragma-SANITIZE=leak-ok.sh\" pragma\nwill be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n\nIn subsequents commit we'll conservatively add more\nTEST_PASSES_SANITIZE_LEAK=true annotations. The idea is that as memory\nleaks are fixed we can add more known-good tests to this CI target, to\nensure that we won't have regressions.\n\nAs of writing this we've got major regressions between master..seen,\ni.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n'ah/plugleaks', 2021-08-04) have regressed recently.\n\nSee the discussion at <87czsv2idy.fsf@evledraar.gmail.com> about the\nlack of this sort of test mode, and 0e5bba53af (add UNLEAK annotation\nfor reducing leak false positives, 2017-09-08) for the initial\naddition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 2 ++\n ci/install-dependencies.sh | 4 ++--\n ci/lib.sh | 11 +++++++++--\n ci/run-build-and-tests.sh | 4 ++--\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/test-lib.sh | 21 +++++++++++++++++++++\n t/test-pragma-SANITIZE=leak-ok.sh | 8 ++++++++\n 8 files changed, 52 insertions(+), 6 deletions(-)\n create mode 100644 t/test-pragma-SANITIZE=leak-ok.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 47876a4f02e..d11b971f970 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -232,6 +232,8 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-SANITIZE=leak\n+ pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 5772081b6e5..30276ae1e00 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-SANITIZE=leak)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-SANITIZE=leak)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 33b9777ab7e..d86b83ed203 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,9 +183,9 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-SANITIZE=leak)\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-SANITIZE=leak)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n \t\t;;\n@@ -237,4 +237,11 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-SANITIZE=leak)\n+\texport SANITIZE=leak\n+\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee94..f0b9775b6c7 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -12,7 +12,7 @@ esac\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc|linux-SANITIZE=leak)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n@@ -29,7 +29,7 @@ linux-gcc)\n \texport GIT_TEST_CHECKOUT_WORKERS=2\n \tmake test\n \t;;\n-linux-clang)\n+linux-clang|linux-SANITIZE=leak)\n \texport GIT_TEST_DEFAULT_HASH=sha1\n \tmake test\n \texport GIT_TEST_DEFAULT_HASH=sha256\ndiff --git a/t/README b/t/README\nindex 9e701223020..f5dfac568d1 100644\n--- a/t/README\n+++ b/t/README\n@@ -448,6 +448,13 @@ GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n to and 'checkout.thresholdForParallelism' to 0, forcing the\n execution of the parallel-checkout code.\n \n+GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n+SANITIZE=leak will run only those tests that have whitelisted\n+themselves as passing with no memory leaks. Do this by sourcing\n+\"test-pragma-SANITIZE=leak-ok.sh\" before sourcing \"test-lib.sh\" itself\n+at the top of the test script. This test mode is used by the\n+\"linux-SANITIZE=leak\" CI target.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex cb87768513c..14836c97cc6 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -18,6 +18,7 @@ swapping compression and hashing order, the person who is making the\n modification *should* take notice and update the test vectors here.\n '\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n try_local_xy () {\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 4ab18914a3d..332dd59257d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1379,6 +1379,27 @@ then\n \ttest_done\n fi\n \n+# Aggressively skip non-whitelisted tests when compiled with\n+# SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+\tthen\n+\t\t# We need to see it in \"git env--helper\" (via\n+\t\t# test_bool_env)\n+\t\texport TEST_PASSES_SANITIZE_LEAK\n+\n+\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n+\t\tthen\n+\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n+\t\t\ttest_done\n+\t\tfi\n+\tfi\n+elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\ndiff --git a/t/test-pragma-SANITIZE=leak-ok.sh b/t/test-pragma-SANITIZE=leak-ok.sh\nnew file mode 100644\nindex 00000000000..5f03397075d\n--- /dev/null\n+++ b/t/test-pragma-SANITIZE=leak-ok.sh\n@@ -0,0 +1,8 @@\n+#!/bin/sh\n+\n+## This \"pragma\" (as in \"perldoc perlpragma\") declares that the test\n+## will pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. Source this\n+## before sourcing test-lib.sh\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+export TEST_PASSES_SANITIZE_LEAK\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434271","messageId":"patch-v3-4.8-0cd14d64165-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 4/8] tests: annotate t000*.sh with TEST_PASSES_SANITIZE_LEAK=true","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:38Z","receivedAt":"2021-08-31T13:35:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Annotate the t000*.sh tests that pass under SANITIZE=leak, these tests\nnow pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. We skip\nt0006-date.sh and t0009-prio-queue.sh due to outstanding memory leaks.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0001-init.sh | 1 +\n t/t0002-gitfile.sh | 1 +\n t/t0003-attributes.sh | 1 +\n t/t0004-unwritable.sh | 1 +\n t/t0005-signals.sh | 2 ++\n t/t0007-git-var.sh | 2 ++\n t/t0008-ignores.sh | 1 +\n 7 files changed, 9 insertions(+)\n\ndiff --git a/t/t0001-init.sh b/t/t0001-init.sh\nindex df544bb321f..8ce04bcabd2 100755\n--- a/t/t0001-init.sh\n+++ b/t/t0001-init.sh\n@@ -2,6 +2,7 @@\n \n test_description='git init'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n check_config () {\ndiff --git a/t/t0002-gitfile.sh b/t/t0002-gitfile.sh\nindex 8440e6add12..3dcb3d16944 100755\n--- a/t/t0002-gitfile.sh\n+++ b/t/t0002-gitfile.sh\n@@ -7,6 +7,7 @@ Verify that plumbing commands work when .git is a file\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n objpath() {\ndiff --git a/t/t0003-attributes.sh b/t/t0003-attributes.sh\nindex 1e4c672b84a..4ef24a35ab5 100755\n--- a/t/t0003-attributes.sh\n+++ b/t/t0003-attributes.sh\n@@ -2,6 +2,7 @@\n \n test_description=gitattributes\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n attr_check_basic () {\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex fbdcb926b3a..35571947ec5 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -2,6 +2,7 @@\n \n test_description='detect unwritable repository and fail correctly'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t0005-signals.sh b/t/t0005-signals.sh\nindex 4c214bd11c4..cd3ecf403e0 100755\n--- a/t/t0005-signals.sh\n+++ b/t/t0005-signals.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='signals work as we expect'\n+\n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n cat >expect <\n---\n t/t0010-racy-git.sh | 1 +\n t/t0011-hashmap.sh | 1 +\n t/t0013-sha1dc.sh | 1 +\n t/t0016-oidmap.sh | 1 +\n t/t0017-env-helper.sh | 1 +\n t/t0018-advice.sh | 1 +\n 6 files changed, 6 insertions(+)\n\ndiff --git a/t/t0010-racy-git.sh b/t/t0010-racy-git.sh\nindex 5657c5a87b6..9a627077be4 100755\n--- a/t/t0010-racy-git.sh\n+++ b/t/t0010-racy-git.sh\n@@ -2,6 +2,7 @@\n \n test_description='racy GIT'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n # This test can give false success if your machine is sufficiently\ndiff --git a/t/t0011-hashmap.sh b/t/t0011-hashmap.sh\nindex 5343ffd3f92..02b07ffa75c 100755\n--- a/t/t0011-hashmap.sh\n+++ b/t/t0011-hashmap.sh\n@@ -1,6 +1,7 @@\n #!/bin/sh\n \n test_description='test hashmap and string hash functions'\n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_hashmap() {\ndiff --git a/t/t0013-sha1dc.sh b/t/t0013-sha1dc.sh\nindex 419f31a8f7d..812b5fcaff3 100755\n--- a/t/t0013-sha1dc.sh\n+++ b/t/t0013-sha1dc.sh\n@@ -1,6 +1,7 @@\n #!/bin/sh\n \n test_description='test sha1 collision detection'\n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n TEST_DATA=\"$TEST_DIRECTORY/t0013\"\n \ndiff --git a/t/t0016-oidmap.sh b/t/t0016-oidmap.sh\nindex 31f8276ba82..a9e135d859b 100755\n--- a/t/t0016-oidmap.sh\n+++ b/t/t0016-oidmap.sh\n@@ -1,6 +1,7 @@\n #!/bin/sh\n \n test_description='test oidmap'\n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n # This purposefully is very similar to t0011-hashmap.sh\ndiff --git a/t/t0017-env-helper.sh b/t/t0017-env-helper.sh\nindex 4a159f99e44..14bb6797b30 100755\n--- a/t/t0017-env-helper.sh\n+++ b/t/t0017-env-helper.sh\n@@ -2,6 +2,7 @@\n \n test_description='test env--helper'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n \ndiff --git a/t/t0018-advice.sh b/t/t0018-advice.sh\nindex 39e5e4b34f8..326752a9711 100755\n--- a/t/t0018-advice.sh\n+++ b/t/t0018-advice.sh\n@@ -2,6 +2,7 @@\n \n test_description='Test advise_if_enabled functionality'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success 'advice should be printed when config variable is unset' '\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434273","messageId":"patch-v3-6.8-2599016c4e7-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 6/8] tests: annotate t002*.sh with TEST_PASSES_SANITIZE_LEAK=true","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:40Z","receivedAt":"2021-08-31T13:36:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Annotate the t002*.sh tests that pass under SANITIZE=leak, these tests\nnow pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. We skip\nt0020-crlf.sh, t0021-conversion.sh, t0023-crlf-am.sh and\nt0028-working-tree-encoding.sh due to outstanding memory leaks.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0022-crlf-rename.sh | 1 +\n t/t0024-crlf-archive.sh | 1 +\n t/t0025-crlf-renormalize.sh | 1 +\n t/t0026-eol-config.sh | 1 +\n t/t0029-core-unsetenvvars.sh | 1 +\n 5 files changed, 5 insertions(+)\n\ndiff --git a/t/t0022-crlf-rename.sh b/t/t0022-crlf-rename.sh\nindex 7af3fbcc7b9..d8ae0879bdb 100755\n--- a/t/t0022-crlf-rename.sh\n+++ b/t/t0022-crlf-rename.sh\n@@ -2,6 +2,7 @@\n \n test_description='ignore CR in CRLF sequence while computing similiarity'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t0024-crlf-archive.sh b/t/t0024-crlf-archive.sh\nindex 4e9fa3cd684..95913032524 100755\n--- a/t/t0024-crlf-archive.sh\n+++ b/t/t0024-crlf-archive.sh\n@@ -2,6 +2,7 @@\n \n test_description='respect crlf in git archive'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\nindex e13363ade5c..88cbdc5ed3a 100755\n--- a/t/t0025-crlf-renormalize.sh\n+++ b/t/t0025-crlf-renormalize.sh\n@@ -2,6 +2,7 @@\n \n test_description='CRLF renormalization'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t0026-eol-config.sh b/t/t0026-eol-config.sh\nindex c5203e232c8..3be010e2f12 100755\n--- a/t/t0026-eol-config.sh\n+++ b/t/t0026-eol-config.sh\n@@ -2,6 +2,7 @@\n \n test_description='CRLF conversion'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n has_cr() {\ndiff --git a/t/t0029-core-unsetenvvars.sh b/t/t0029-core-unsetenvvars.sh\nindex 24ce46a6ea1..87566900c2b 100755\n--- a/t/t0029-core-unsetenvvars.sh\n+++ b/t/t0029-core-unsetenvvars.sh\n@@ -2,6 +2,7 @@\n \n test_description='test the Windows-only core.unsetenvvars setting'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n if ! test_have_prereq MINGW\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434274","messageId":"patch-v3-7.8-ddc4d6d2cf1-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 7/8] tests: annotate select t0*.sh with TEST_PASSES_SANITIZE_LEAK=true","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:41Z","receivedAt":"2021-08-31T13:36:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Annotate a few t0*.sh tests that pass with SANITIZE=leak, these tests\nnow pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. These aren't all\nof the ones in t0*.sh that pass, I'm selecting a few ones that test\nsome core APIs, and the simple \"git bugreport\" built-in.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0030-stripspace.sh | 1 +\n t/t0052-simple-ipc.sh | 1 +\n t/t0061-run-command.sh | 1 +\n t/t0063-string-list.sh | 1 +\n t/t0066-dir-iterator.sh | 1 +\n t/t0067-parse_pathspec_file.sh | 1 +\n t/t0091-bugreport.sh | 1 +\n 7 files changed, 7 insertions(+)\n\ndiff --git a/t/t0030-stripspace.sh b/t/t0030-stripspace.sh\nindex 0c24a0f9a37..d00f7dd01e8 100755\n--- a/t/t0030-stripspace.sh\n+++ b/t/t0030-stripspace.sh\n@@ -5,6 +5,7 @@\n \n test_description='git stripspace'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n t40='A quick brown fox jumps over the lazy do'\ndiff --git a/t/t0052-simple-ipc.sh b/t/t0052-simple-ipc.sh\nindex ff98be31a51..f76a1f5e249 100755\n--- a/t/t0052-simple-ipc.sh\n+++ b/t/t0052-simple-ipc.sh\n@@ -2,6 +2,7 @@\n \n test_description='simple command server'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test-tool simple-ipc SUPPORTS_SIMPLE_IPC || {\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 7d599675e35..89fd3b18e52 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -5,6 +5,7 @@\n \n test_description='Test run command'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n cat >hello-script <<-EOF\ndiff --git a/t/t0063-string-list.sh b/t/t0063-string-list.sh\nindex c6ee9f66b11..0bd69de4f75 100755\n--- a/t/t0063-string-list.sh\n+++ b/t/t0063-string-list.sh\n@@ -5,6 +5,7 @@\n \n test_description='Test string list functionality'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_split () {\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex 92910e4e6c1..edafdbbe7dc 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -2,6 +2,7 @@\n \n test_description='Test the dir-iterator functionality'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success 'setup' '\ndiff --git a/t/t0067-parse_pathspec_file.sh b/t/t0067-parse_pathspec_file.sh\nindex 7bab49f361a..cc2540db9f9 100755\n--- a/t/t0067-parse_pathspec_file.sh\n+++ b/t/t0067-parse_pathspec_file.sh\n@@ -2,6 +2,7 @@\n \n test_description='Test parse_pathspec_file()'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success 'one item from stdin' '\ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 526304ff95b..946909dbfde 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -2,6 +2,7 @@\n \n test_description='git bugreport'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n # Headers \"[System Info]\" will be followed by a non-empty line if we put some\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434275","messageId":"patch-v3-8.8-e611d2c23d9-20210831T132607Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132607Z-avarab@gmail.com","subject":"[PATCH v3 8/8] tests: annotate select t*.sh with TEST_PASSES_SANITIZE_LEAK=true","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:35:42Z","receivedAt":"2021-08-31T13:36:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Annotate a few t*.sh tests that pass with SANITIZE=leak, these tests\nnow pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. These aren't all\nof the ones in t*.sh that pass, I'm selecting a few arbitrary passing\nones that stress a few common commands and APIs.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1010-mktree.sh | 1 +\n t/t1100-commit-tree-options.sh | 1 +\n t/t1308-config-set.sh | 1 +\n t/t1309-early-config.sh | 1 +\n t/t1420-lost-found.sh | 1 +\n t/t1430-bad-ref-name.sh | 1 +\n t/t1509-root-work-tree.sh | 1 +\n t/t2002-checkout-cache-u.sh | 1 +\n t/t2050-git-dir-relative.sh | 1 +\n t/t2081-parallel-checkout-collisions.sh | 1 +\n t/t2100-update-cache-badpath.sh | 1 +\n t/t2200-add-update.sh | 1 +\n t/t2201-add-update-typechange.sh | 1 +\n t/t2202-add-addremove.sh | 1 +\n t/t2204-add-ignored.sh | 1 +\n t/t2300-cd-to-toplevel.sh | 1 +\n t/t3000-ls-files-others.sh | 1 +\n t/t3004-ls-files-basic.sh | 1 +\n t/t3006-ls-files-long.sh | 1 +\n t/t3008-ls-files-lazy-init-name-hash.sh | 1 +\n t/t3100-ls-tree-restrict.sh | 1 +\n t/t3101-ls-tree-dirname.sh | 1 +\n t/t3102-ls-tree-wildcards.sh | 1 +\n t/t3103-ls-tree-misc.sh | 1 +\n t/t3205-branch-color.sh | 1 +\n t/t3211-peel-ref.sh | 1 +\n t/t3300-funny-names.sh | 1 +\n t/t3902-quoted.sh | 1 +\n t/t4002-diff-basic.sh | 1 +\n t/t4026-color.sh | 1 +\n t/t4300-merge-tree.sh | 1 +\n 31 files changed, 31 insertions(+)\n\ndiff --git a/t/t1010-mktree.sh b/t/t1010-mktree.sh\nindex b946f876864..3912625cfad 100755\n--- a/t/t1010-mktree.sh\n+++ b/t/t1010-mktree.sh\n@@ -2,6 +2,7 @@\n \n test_description='git mktree'\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t1100-commit-tree-options.sh b/t/t1100-commit-tree-options.sh\nindex ae66ba5babf..d603d9a8544 100755\n--- a/t/t1100-commit-tree-options.sh\n+++ b/t/t1100-commit-tree-options.sh\n@@ -12,6 +12,7 @@ Also make sure that command line parser understands the normal\n \"flags first and then non flag arguments\" command line.\n '\n \n+. ./test-pragma-SANITIZE=leak-ok.sh\n . ./test-lib.sh\n \n cat >expected <\n---\n\nThis was originally submitted as part of the SANITIZE=leak series as\nhttps://lore.kernel.org/git/patch-4.4-ad8680f529-20210714T172251Z-avarab@gmail.com/\n\nIn its v3 I stopped doing these leak fixes & test changes, let's just\nconsider this separately. We'll eventually want to add SANITIZE=leak\nwhitelisting to the relevant test if and when my SANITIZE=leak series\ngoes in, but we can just do that then along with adding various other\ntests.\n\nRange-diff:\n1: 80edda308c9 ! 1: f11eb44e4c5 SANITIZE tests: fix leak in mailmap.c\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - SANITIZE tests: fix leak in mailmap.c\n -\n - Get closer to being able to run t4203-mailmap.sh by fixing a couple of\n - memory leak in mailmap.c.\n + mailmap.c: fix a memory leak in free_mailap_{info,entry}()\n \n In the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\n and clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\n @@ Commit message\n mailmap_entry structure, we didn't free the structure itself. The same\n goes for the \"mailmap_info\" structure.\n \n + This brings us from 50 failures when running t4203-mailmap.sh to\n + 49. Not really progress as far as the number of failures is concerned,\n + but as far as I can tell this fixes all leaks in mailmap.c\n + itself. There's still users of it such as builtin/log.c that call\n + read_mailmap() without a clear_mailmap(), but that's on them.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## mailmap.c ##\n @@ mailmap.c: static void free_mailmap_entry(void *p, const char *s)\n }\n \n /*\n -\n - ## t/t4203-mailmap.sh ##\n -@@ t/t4203-mailmap.sh: test_expect_success 'check-mailmap bogus contact --stdin' '\n - \ttest_must_fail git check-mailmap --stdin bogus expect <<-EOF &&\n - \t$GIT_AUTHOR_NAME (1):\n\n mailmap.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/mailmap.c b/mailmap.c\nindex 462b3956340..40ce152024d 100644\n--- a/mailmap.c\n+++ b/mailmap.c\n@@ -37,6 +37,7 @@ static void free_mailmap_info(void *p, const char *s)\n \t\t s, debug_str(mi->name), debug_str(mi->email));\n \tfree(mi->name);\n \tfree(mi->email);\n+\tfree(mi);\n }\n \n static void free_mailmap_entry(void *p, const char *s)\n@@ -52,6 +53,7 @@ static void free_mailmap_entry(void *p, const char *s)\n \n \tme->namemap.strdup_strings = 1;\n \tstring_list_clear_func(&me->namemap, free_mailmap_info);\n+\tfree(me);\n }\n \n /*\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434278","messageId":"patch-1.1-9acbc21cdd3-20210831T134632Z-avarab@gmail.com","threadId":"55888","inReplyTo":"patch-3.4-b7fb5d5a56-20210714T172251Z-avarab@gmail.com","subject":"[PATCH] protocol-caps.c: fix memory leak in send_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T13:46:42Z","receivedAt":"2021-08-31T13:46:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in a2ba162cda (object-info: support for retrieving\nobject info, 2021-04-20) which appears to have been based on a\nmisunderstanding of how the pkt-line.c API works. There is no need to\nstrdup() input to packet_writer_write(), it's just a printf()-like\nformat function.\n\nThis fixes a potentially large memory leak, since the number of OID\nlines the \"object-info\" call can be arbitrarily large (or a small one\nif the request is small).\n\nThis makes t5701-git-serve.sh pass again under SANITIZE=leak, as it\ndid before a2ba162cda2.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nThis was originally submitted as part of the SANITIZE=leak series as\nhttps://lore.kernel.org/git/patch-3.4-b7fb5d5a56-20210714T172251Z-avarab@gmail.com/\n\nIn its v3 I stopped doing these leak fixes & test changes, let's just\nconsider this separately. We'll eventually want to add SANITIZE=leak\nwhitelisting to the relevant test if and when my SANITIZE=leak series\ngoes in, but we can just do that then along with adding various other\ntests.\n\nRange-diff:\n1: 720852eee0b ! 1: 9acbc21cdd3 SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - SANITIZE tests: fix memory leaks in t5701*, add to whitelist\n + protocol-caps.c: fix memory leak in send_info()\n \n Fix a memory leak in a2ba162cda (object-info: support for retrieving\n object info, 2021-04-20) which appears to have been based on a\n - misunderstanding of how the pkt-line.c API works, there is no need to\n - strdup() input to, it's just a printf()-like format function.\n + misunderstanding of how the pkt-line.c API works. There is no need to\n + strdup() input to packet_writer_write(), it's just a printf()-like\n + format function.\n \n This fixes a potentially large memory leak, since the number of OID\n lines the \"object-info\" call can be arbitrarily large (or a small one\n if the request is small).\n \n + This makes t5701-git-serve.sh pass again under SANITIZE=leak, as it\n + did before a2ba162cda2.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## protocol-caps.c ##\n @@ protocol-caps.c: static void send_info(struct repository *r, struct packet_write\n }\n \n int cap_object_info(struct repository *r, struct strvec *keys,\n -\n - ## t/t5701-git-serve.sh ##\n -@@ t/t5701-git-serve.sh: test_description='test protocol v2 server commands'\n - GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n - export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n - \n -+GIT_TEST_SANITIZE_LEAK=true\n - . ./test-lib.sh\n - \n - test_expect_success 'test capability advertisement' '\n\n protocol-caps.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex 13a9e63a04a..901b6795e42 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -69,9 +69,10 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpacket_writer_write(writer, \"%s\",\n-\t\t\t\t strbuf_detach(&send_buffer, NULL));\n+\t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n+\t\tstrbuf_reset(&send_buffer);\n \t}\n+\tstrbuf_release(&send_buffer);\n }\n \n int cap_object_info(struct repository *r, struct strvec *keys,\n-- \n2.33.0.805.g739b16c2189\n\n"},{"id":"434291","messageId":"CAPeR6H6g_VM3SUyjfYfc+mQa27af7AJE9wbKN_TUdG6m5rAUow@mail.gmail.com","threadId":"55888","inReplyTo":"patch-1.1-9acbc21cdd3-20210831T134632Z-avarab@gmail.com","subject":"Re: [PATCH] protocol-caps.c: fix memory leak in send_info()","fromName":"Bruno Albuquerque","fromEmail":"bga@google.com","sentAt":"2021-08-31T15:32:36Z","receivedAt":"2021-08-31T15:33:00Z","isPatch":true,"sender":{"key":"bga@google.com","avatar":"https://avatars.githubusercontent.com/u/80971974?v=4"},"body":"On Tue, Aug 31, 2021 at 6:46 AM Ævar Arnfjörð Bjarmason\n wrote:\n\n[Replying again as I used HTML mail by mistake. Sorry.]\n\n> Fix a memory leak in a2ba162cda (object-info: support for retrieving\n> object info, 2021-04-20) which appears to have been based on a\n> misunderstanding of how the pkt-line.c API works. There is no need to\n> strdup() input to packet_writer_write(), it's just a printf()-like\n> format function.\n>\n> This fixes a potentially large memory leak, since the number of OID\n> lines the \"object-info\" call can be arbitrarily large (or a small one\n> if the request is small).\n>\n> This makes t5701-git-serve.sh pass again under SANITIZE=leak, as it\n> did before a2ba162cda2.\n\n\nThanks for cleaning up after me. Yes, this was my lack of knowledge on\nhow the internals of Git works. I was also not aware of SANITIZE=leak\nso thanks for the heads up. This looks good to me.\n"},{"id":"434297","messageId":"CAPig+cTzJJA5z51QUwrzCOPmzn_Xzvc6JgyQT36RENMwpt7gJw@mail.gmail.com","threadId":"55888","inReplyTo":"patch-1.1-f11eb44e4c5-20210831T134023Z-avarab@gmail.com","subject":"Re: [PATCH] mailmap.c: fix a memory leak in free_mailap_{info,entry}()","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-08-31T16:22:55Z","receivedAt":"2021-08-31T16:23:09Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Aug 31, 2021 at 9:43 AM Ævar Arnfjörð Bjarmason\n wrote:\n> In the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\n> and clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\n> clear the \"me\" structure, but while we freed parts of the\n> mailmap_entry structure, we didn't free the structure itself. The same\n> goes for the \"mailmap_info\" structure.\n>\n> This brings us from 50 failures when running t4203-mailmap.sh to\n> 49. Not really progress as far as the number of failures is concerned,\n> but as far as I can tell this fixes all leaks in mailmap.c\n> itself. There's still users of it such as builtin/log.c that call\n> read_mailmap() without a clear_mailmap(), but that's on them.\n\nAs a standalone patch, the \"50 failures\" is confusing and sounds quite\nalarming. Adding even a tiny bit of context:\n\n s/50 failure/50 SANITIZE failures/\n\nwould help reduce the confusion. Alternatively, just dropping the\nsecond paragraph altogether would clear up any misunderstanding since\nthe first paragraph and the patch body stand well on their own without\nany additional explanation.\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n"},{"id":"434316","messageId":"xmqqtuj5tr96.fsf@gitster.g","threadId":"55888","inReplyTo":"CAPeR6H6g_VM3SUyjfYfc+mQa27af7AJE9wbKN_TUdG6m5rAUow@mail.gmail.com","subject":"Re: [PATCH] protocol-caps.c: fix memory leak in send_info()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-31T18:15:33Z","receivedAt":"2021-08-31T18:15:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Bruno Albuquerque writes:\n\n> On Tue, Aug 31, 2021 at 6:46 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>\n> [Replying again as I used HTML mail by mistake. Sorry.]\n>\n>> Fix a memory leak in a2ba162cda (object-info: support for retrieving\n>> object info, 2021-04-20) which appears to have been based on a\n>> misunderstanding of how the pkt-line.c API works. There is no need to\n>> strdup() input to packet_writer_write(), it's just a printf()-like\n>> format function.\n>>\n>> This fixes a potentially large memory leak, since the number of OID\n>> lines the \"object-info\" call can be arbitrarily large (or a small one\n>> if the request is small).\n>>\n>> This makes t5701-git-serve.sh pass again under SANITIZE=leak, as it\n>> did before a2ba162cda2.\n>\n>\n> Thanks for cleaning up after me. Yes, this was my lack of knowledge on\n> how the internals of Git works. I was also not aware of SANITIZE=leak\n> so thanks for the heads up. This looks good to me.\n\nThanks, both.\n\nWill apply.\n"},{"id":"434325","messageId":"YS6FKEApva30sKgl@coredump.intra.peff.net","threadId":"55888","inReplyTo":"patch-1.1-f11eb44e4c5-20210831T134023Z-avarab@gmail.com","subject":"Re: [PATCH] mailmap.c: fix a memory leak in free_mailap_{info,entry}()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-31T19:38:16Z","receivedAt":"2021-08-31T19:38:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 03:42:52PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> In the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\n> and clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\n> clear the \"me\" structure, but while we freed parts of the\n> mailmap_entry structure, we didn't free the structure itself. The same\n> goes for the \"mailmap_info\" structure.\n> \n> This brings us from 50 failures when running t4203-mailmap.sh to\n> 49. Not really progress as far as the number of failures is concerned,\n> but as far as I can tell this fixes all leaks in mailmap.c\n> itself. There's still users of it such as builtin/log.c that call\n> read_mailmap() without a clear_mailmap(), but that's on them.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n\nThanks, the patch looks good to me. I agree with Eric that mentioning\n\"leak failures\" in the second paragraph would make it less confusing. :)\n\n-Peff\n"},{"id":"434327","messageId":"xmqq8s0htn25.fsf@gitster.g","threadId":"55888","inReplyTo":"YS6FKEApva30sKgl@coredump.intra.peff.net","subject":"Re: [PATCH] mailmap.c: fix a memory leak in free_mailap_{info,entry}()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-31T19:46:10Z","receivedAt":"2021-08-31T19:46:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Tue, Aug 31, 2021 at 03:42:52PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> In the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\n>> and clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\n>> clear the \"me\" structure, but while we freed parts of the\n>> mailmap_entry structure, we didn't free the structure itself. The same\n>> goes for the \"mailmap_info\" structure.\n>> \n>> This brings us from 50 failures when running t4203-mailmap.sh to\n>> 49. Not really progress as far as the number of failures is concerned,\n>> but as far as I can tell this fixes all leaks in mailmap.c\n>> itself. There's still users of it such as builtin/log.c that call\n>> read_mailmap() without a clear_mailmap(), but that's on them.\n>> \n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>\n> Thanks, the patch looks good to me. I agree with Eric that mentioning\n> \"leak failures\" in the second paragraph would make it less confusing. :)\n\nHere is what I queued.\n\nThanks, all.\n\nFrom ccdd5d1eb14a6735c34428e856c0de33f1055520 Mon Sep 17 00:00:00 2001\nFrom: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \nDate: Tue, 31 Aug 2021 15:42:52 +0200\nSubject: [PATCH] mailmap.c: fix a memory leak in free_mailap_{info,entry}()\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nIn the free_mailmap_entry() code added in 0925ce4d49 (Add map_user()\nand clear_mailmap() to mailmap, 2009-02-08) the intent was clearly to\nclear the \"me\" structure, but while we freed parts of the\nmailmap_entry structure, we didn't free the structure itself. The same\ngoes for the \"mailmap_info\" structure.\n\nThis brings the number of SANITIZE=leak failures in t4203-mailmap.sh\ndown from 50 to 49. Not really progress as far as the number of\nfailures is concerned, but as far as I can tell this fixes all leaks\nin mailmap.c itself. There's still users of it such as builtin/log.c\nthat call read_mailmap() without a clear_mailmap(), but that's on\nthem.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Junio C Hamano \n---\n mailmap.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/mailmap.c b/mailmap.c\nindex d1f7c0d272..e1c8736093 100644\n--- a/mailmap.c\n+++ b/mailmap.c\n@@ -36,6 +36,7 @@ static void free_mailmap_info(void *p, const char *s)\n \t\t s, debug_str(mi->name), debug_str(mi->email));\n \tfree(mi->name);\n \tfree(mi->email);\n+\tfree(mi);\n }\n \n static void free_mailmap_entry(void *p, const char *s)\n@@ -51,6 +52,7 @@ static void free_mailmap_entry(void *p, const char *s)\n \n \tme->namemap.strdup_strings = 1;\n \tstring_list_clear_func(&me->namemap, free_mailmap_info);\n+\tfree(me);\n }\n \n /*\n-- \n2.33.0-323-g897a01baa9\n\n"},{"id":"434330","messageId":"87k0k1wf48.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"CAPeR6H69a_HMwWnpHzssaCm_ow=ic7AnzMdZVQJQ2ECRDaWzaA@mail.gmail.com","subject":"Re: [PATCH] protocol-caps.c: fix memory leak in send_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T20:08:43Z","receivedAt":"2021-08-31T20:09:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 31 2021, Bruno Albuquerque wrote:\n\n> On Tue, Aug 31, 2021 at 6:46 AM Ævar Arnfjörð Bjarmason wrote:\n>\n> Fix a memory leak in a2ba162cda (object-info: support for retrieving\n> object info, 2021-04-20) which appears to have been based on a\n> misunderstanding of how the pkt-line.c API works. There is no need to\n> strdup() input to packet_writer_write(), it's just a printf()-like\n> format function.\n>\n> This fixes a potentially large memory leak, since the number of OID\n> lines the \"object-info\" call can be arbitrarily large (or a small one\n> if the request is small).\n>\n> This makes t5701-git-serve.sh pass again under SANITIZE=leak, as it\n> did before a2ba162cda2.\n>\n> Thanks for cleaning up after me. Yes, this was my lack of knowledge on how the internals of Git works. I was also not aware of SANITIZE=leak so thanks for\n> the heads up. This looks good to me.\n\nThanks, for what it's worth the series I submitted in parallel to this\nto add a SANITIZE=leak CI mode at\nhttps://lore.kernel.org/git/cover-v3-0.8-00000000000-20210831T132546Z-avarab@gmail.com\ncould use reviewers :)\n\nI.e. having some real tests for this sort of thing and running them in\nCI will help to catch any such issues earlier.\n"},{"id":"434398","messageId":"YS8xj9XtKqEEy/Bb@coredump.intra.peff.net","threadId":"55888","inReplyTo":"87y28hwylq.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T07:53:51Z","receivedAt":"2021-09-01T07:53:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 02:47:01PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > That works, but now \"util\" is not available for all the _other_ uses for\n> > which it was intended. And if we're not using it for those other uses,\n> > then why does it need to exist at all? If we are only using it to hold\n> > the allocated string pointer, then shouldn't it be \"char *to_free\"?\n> \n> Because having it be \"char *\" doesn't cover the common case of\n> e.g. getting an already allocated \"struct something *\" which contains\n> your string, setting the \"string\" in \"struct string_list_item\" to some\n> string in that struct, and the \"util\" to the struct itself, as we now\n> own it and want to free() it later in its entirety.\n\nOK. I buy that storing a void pointer makes it more flexible. I'm not\naltogether convinced this pattern is especially common, but it's not any\nharder to work with than a \"need_to_free\" flag, so there's no reason not\nto do that (and to be fair, I didn't look around for possible uses of\nthe pattern; it's just not one I think of as common off the top of my\nhead).\n\n> That and the even more common case I mentioned upthread of wanting to\n> ferry around the truncated version of some char *, but still wanting to\n> account for the original for an eventual free().\n> \n> But yes, if you want to account for freeing that data *and* have util\n> set to something else you'll need to have e.g. your own wrapper struct\n> and your own string_list_clear_func() callback.\n\nBut stuffing it into the util field of string_list really feels like a\nstretch, and something that would make existing string_list use painful.\nThere are tons of cases where util points to some totally unrelated (in\nterms of memory ownership) item. I'd venture to say most cases where\nstring_list_clear() is called without free_util would count here.\n\n> > I don't think most interfaces take a string_list_item now, so wouldn't\n> > they similarly need to be changed? Though the point is that all of these\n> > degrade to a regular C-string, so when you are just passing the value\n> > (and not ownership), you would just dereference at that point.\n> \n> Sure, just like things would need to be changed to handle your proposed\n> \"struct def_string\".\n> \n> By piggy-backing on an already used struct in our codebase we can get a\n> lot of that memory management pretty much for free without much\n> churn.\n> \n> If you squint and pretend that \"struct string_list_item\" isn't called\n> something to do with that particular collections API (but it would make\n> use of it) then we've already set up most of the scaffolding and\n> management for this.\n\nIt's that squinting that bothers me. Sure, it's _kinda_ similar. And I\ndon't have any problem with some kind of struct that says \"this is a\nstring, and when you are done with it, this is how you free it\". And I\ndon't have any problem with building the \"dup\" version of string_list\nwith that struct as a primitive. But it seems to me to be orthogonal\nfrom the \"util\" pointer of a string_list, which is about creating a\nmapping from the string to some other thing (which may or may not\ncontain the string, and may or may not be owned).\n\nTBH, I have always found the \"util\" field of string_list a bit ugly (and\nreally most of string_list). I think most cases would be better off with\na different data structure (a set or a hash table), but we didn't have\nconvenient versions of those for a long time. I don't mind seeing\nconversions of string_list to other data structures. But that seems to\nbe working against using string_list's string struct in more places.\n\n-Peff\n"},{"id":"434405","messageId":"YS9OT/pn5rRK9cGB@coredump.intra.peff.net","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132546Z-avarab@gmail.com","subject":"Re: [PATCH v3 0/8] add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T09:56:31Z","receivedAt":"2021-09-01T09:56:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 03:35:34PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> * In v2 compiling with SANITIZE=leak would change things so only\n> known-good passing tests were run by default, everything else would\n> pass as a dummy. Now the default running of tests is unchanged, but\n> if we run with GIT_TEST_PASSING_SANITIZE_LEAK=true only those tests\n> are run which set and export TEST_PASSES_SANITIZE_LEAK=true.\n> \n> * The facility for declaring known-good tests in test-lib.sh based on\n> wildcards is gone, instead individual tests need to declare if\n> they're OK under SANITIZE=leak.[...]\n\nHmm. This still seems more complicated than we need. If we just want a\nflag in each script, then test-lib.sh can use that flag to tweak\nLSAN_OPTIONS. See the patch below.\n\nThat has two drawbacks:\n\n - it doesn't have any way to switch the flag per-test. But IMHO it is\n a mistake to go in that direction. This is all temporary scaffolding\n while we have leaks, and the script-level of granularity is fine.\n\n - it runs the tests not marked as LSAN-OK, just without leak checking,\n which is redundant in CI where we're already running them. But we\n could still be collecting leak stats (and just not failing the\n tests). See the patch below.\n\n If we do care about not running them, then I think it makes more\n sense to extend the run/skip mechanisms and build on that.\n\n (I also think I prefer the central list of \"mark these scripts as OK\n for leak-checking\", rather than annotating individuals. Because\n again, this is temporary, and it's nice to keep it in a sandbox that\n only people working on leak-checking would look at or touch).\n\nI realize this is kind-of bikeshedding, and I'm not vehemently opposed\nto what you have here. It just seems like fewer moving parts would be\nless likely to confuse folks who want to poke at it.\n\n> This is done via \"export\n> TEST_PASSES_SANITIZE_LEAK=true\", there's a handy import of\n> \"./test-pragma-SANITIZE=leak-ok.sh\" before sourcing \"./test-lib.sh\"\n> itself to set this.\n\nI found the extra level of indirection added by this pragma confusing.\nWe just need to set a variable, which is also a one-liner, and one that\nis more obvious about what it's doing. In your code you also export it,\nbut that's not necessary for something that test-lib.sh is going to look\nat. Or if it's really necessary at some point, then test-lib.sh can do\nthe export itself.\n\n> Ævar Arnfjörð Bjarmason (8):\n> Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n> CI: refactor \"if\" to \"case\" statement\n> tests: add a test mode for SANITIZE=leak, run it in CI\n> tests: annotate t000*.sh with TEST_PASSES_SANITIZE_LEAK=true\n> tests: annotate t001*.sh with TEST_PASSES_SANITIZE_LEAK=true\n> tests: annotate t002*.sh with TEST_PASSES_SANITIZE_LEAK=true\n> tests: annotate select t0*.sh with TEST_PASSES_SANITIZE_LEAK=true\n> tests: annotate select t*.sh with TEST_PASSES_SANITIZE_LEAK=true\n\nSort of a meta-question, but what's the plan for folks who add a new\ntest to say t0000, and it reveals a leak in code they didn't touch?\n\nThey'll get a CI failure (as will Junio if he picks up the patch), so\nsomebody is going to have to deal with it. Do they fix it? Do they unset\nthe \"this script is OK\" flag? Do they mark the individual test as\nnon-lsan-ok?\n\nI do like the idea of finding real regressions. But while the state of\nleak-checking is still so immature, I'm worried about this adding extra\nfriction for developers. Especially if they get some spooky action at a\ndistance caused by a leak in far-away code.\n\nAnyway, here's LSAN_OPTIONS thing I was thinking of.\n\n---\ndiff --git a/t/t0001-init.sh b/t/t0001-init.sh\nindex df544bb321..b1da18955d 100755\n--- a/t/t0001-init.sh\n+++ b/t/t0001-init.sh\n@@ -2,6 +2,7 @@\n \n test_description='git init'\n \n+TEST_LSAN_OK=1\n . ./test-lib.sh\n \n check_config () {\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d6..62627afeaf 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -44,9 +44,30 @@ GIT_BUILD_DIR=\"$TEST_DIRECTORY\"/..\n : ${ASAN_OPTIONS=detect_leaks=0:abort_on_error=1}\n export ASAN_OPTIONS\n \n-# If LSAN is in effect we _do_ want leak checking, but we still\n-# want to abort so that we notice the problems.\n-: ${LSAN_OPTIONS=abort_on_error=1}\n+if test -n \"$LSAN_OPTIONS\"\n+then\n+\t# Leave user-provided options alone.\n+\t:\n+elif test -n \"$TEST_LSAN_OK\"\n+then\n+\t# The test script has declared itself as LSAN-clean; turn on full leak\n+\t# checking.\n+\tLSAN_OPTIONS=abort_on_error=1\n+else\n+\t# The test script has possible LSAN failures. Just disable\n+\t# leak-checking entirely. Another option would be to log the failures\n+\t# with:\n+\t#\n+\t# LSAN_OPTIONS=exitcode=0:log_path=$TEST_DIRECTORY/lsan/out\n+\t#\n+\t# The results are rather confusing, though, as the logs are\n+\t# per-process; you have no idea which one came from which test script.\n+\t# Ideally we'd send them to descriptor 4 along with the rest of the\n+\t# script log, but there's no LSAN_OPTION for that (recent versions of\n+\t# libsanitizer do have a public function to do so, so we could hook it\n+\t# ourselves via common-main).\n+\tLSAN_OPTIONS=detect_leaks=0\n+fi\n export LSAN_OPTIONS\n \n if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n"},{"id":"434408","messageId":"YS9ZIDpANfsh7N+S@coredump.intra.peff.net","threadId":"55888","inReplyTo":"YS9OT/pn5rRK9cGB@coredump.intra.peff.net","subject":"Re: [PATCH v3 0/8] add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T10:42:40Z","receivedAt":"2021-09-01T10:42:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 05:56:31AM -0400, Jeff King wrote:\n\n> +else\n> +\t# The test script has possible LSAN failures. Just disable\n> +\t# leak-checking entirely. Another option would be to log the failures\n> +\t# with:\n> +\t#\n> +\t# LSAN_OPTIONS=exitcode=0:log_path=$TEST_DIRECTORY/lsan/out\n> +\t#\n> +\t# The results are rather confusing, though, as the logs are\n> +\t# per-process; you have no idea which one came from which test script.\n> +\t# Ideally we'd send them to descriptor 4 along with the rest of the\n> +\t# script log, but there's no LSAN_OPTION for that (recent versions of\n> +\t# libsanitizer do have a public function to do so, so we could hook it\n> +\t# ourselves via common-main).\n> +\tLSAN_OPTIONS=detect_leaks=0\n> +fi\n\nI was curious about the fd thing. The patch below implements it, and\nlets t0203 (for example) pass while including its sanitizer output in a\n--verbose-log.\n\nBut this is exactly the kind of gross complexity I was suggesting to\navoid. :)\n\ndiff --git a/Makefile b/Makefile\nindex d1feab008f..ba2174fb79 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1260,6 +1260,7 @@ ifdef SANITIZE\n SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag))\n BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)\n BASIC_CFLAGS += -fno-omit-frame-pointer\n+BASIC_CFLAGS += -DENABLE_CUSTOM_SANITIZER_OPTIONS\n ifneq ($(filter undefined,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\ndiff --git a/common-main.c b/common-main.c\nindex 71e21dd20a..bff594ac04 100644\n--- a/common-main.c\n+++ b/common-main.c\n@@ -2,6 +2,10 @@\n #include \"exec-cmd.h\"\n #include \"attr.h\"\n \n+#ifdef ENABLE_CUSTOM_SANITIZER_OPTIONS\n+#include \n+#endif\n+\n /*\n * Many parts of Git have subprograms communicate via pipe, expect the\n * upstream of a pipe to die with SIGPIPE when the downstream of a\n@@ -23,6 +27,18 @@ static void restore_sigpipe_to_default(void)\n \tsignal(SIGPIPE, SIG_DFL);\n }\n \n+static void handle_custom_sanitizer_options(void)\n+{\n+#ifdef ENABLE_CUSTOM_SANITIZER_OPTIONS\n+\tconst char *v;\n+\tv = getenv(\"GIT_SANITIZER_FD\");\n+\tif (v) {\n+\t\t/* weird int-as-void interface from libsanitizer */\n+\t\t__sanitizer_set_report_fd((void *)(intptr_t)atoi(v));\n+\t}\n+#endif\n+}\n+\n int main(int argc, const char **argv)\n {\n \tint result;\n@@ -37,6 +53,8 @@ int main(int argc, const char **argv)\n \tsanitize_stdfds();\n \trestore_sigpipe_to_default();\n \n+\thandle_custom_sanitizer_options();\n+\n \tgit_resolve_executable_dir(argv[0]);\n \n \tgit_setup_gettext();\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 62627afeaf..674bd30c44 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -54,22 +54,16 @@ then\n \t# checking.\n \tLSAN_OPTIONS=abort_on_error=1\n else\n-\t# The test script has possible LSAN failures. Just disable\n-\t# leak-checking entirely. Another option would be to log the failures\n-\t# with:\n-\t#\n-\t# LSAN_OPTIONS=exitcode=0:log_path=$TEST_DIRECTORY/lsan/out\n-\t#\n-\t# The results are rather confusing, though, as the logs are\n-\t# per-process; you have no idea which one came from which test script.\n-\t# Ideally we'd send them to descriptor 4 along with the rest of the\n-\t# script log, but there's no LSAN_OPTION for that (recent versions of\n-\t# libsanitizer do have a public function to do so, so we could hook it\n-\t# ourselves via common-main).\n-\tLSAN_OPTIONS=detect_leaks=0\n+\t# The test script has possible LSAN failures. Just log them but don't\n+\t# touch the exit code.\n+\tLSAN_OPTIONS=exitcode=0\n fi\n export LSAN_OPTIONS\n \n+# If we do generate output, try to avoid it getting tangled up with stderr.\n+GIT_SANITIZER_FD=4\n+export GIT_SANITIZER_FD\n+\n if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n then\n \techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n@@ -463,6 +457,7 @@ unset VISUAL EMAIL LANGUAGE $(\"$PERL_PATH\" -e '\n \t\tPERF_\n \t\tCURL_VERBOSE\n \t\tTRACE_CURL\n+\t\tSANITIZER_.*\n \t));\n \tmy @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env);\n \tprint join(\"\\n\", @vars);\n"},{"id":"434424","messageId":"875yvkwllw.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"YS8xj9XtKqEEy/Bb@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/4] SANITIZE tests: fix memory leaks in t13*config*, add to whitelist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-01T11:45:15Z","receivedAt":"2021-09-01T12:01:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 01 2021, Jeff King wrote:\n\n> On Tue, Aug 31, 2021 at 02:47:01PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > That works, but now \"util\" is not available for all the _other_ uses for\n>> > which it was intended. And if we're not using it for those other uses,\n>> > then why does it need to exist at all? If we are only using it to hold\n>> > the allocated string pointer, then shouldn't it be \"char *to_free\"?\n>> \n>> Because having it be \"char *\" doesn't cover the common case of\n>> e.g. getting an already allocated \"struct something *\" which contains\n>> your string, setting the \"string\" in \"struct string_list_item\" to some\n>> string in that struct, and the \"util\" to the struct itself, as we now\n>> own it and want to free() it later in its entirety.\n>\n> OK. I buy that storing a void pointer makes it more flexible. I'm not\n> altogether convinced this pattern is especially common, but it's not any\n> harder to work with than a \"need_to_free\" flag, so there's no reason not\n> to do that (and to be fair, I didn't look around for possible uses of\n> the pattern; it's just not one I think of as common off the top of my\n> head).\n>\n>> That and the even more common case I mentioned upthread of wanting to\n>> ferry around the truncated version of some char *, but still wanting to\n>> account for the original for an eventual free().\n>> \n>> But yes, if you want to account for freeing that data *and* have util\n>> set to something else you'll need to have e.g. your own wrapper struct\n>> and your own string_list_clear_func() callback.\n>\n> But stuffing it into the util field of string_list really feels like a\n> stretch, and something that would make existing string_list use painful.\n> There are tons of cases where util points to some totally unrelated (in\n> terms of memory ownership) item. I'd venture to say most cases where\n> string_list_clear() is called without free_util would count here.\n\nFor what it's worth I've got some WIP code that's part of my daily build\nwhere I did end up going through all those callers, as part of general\nstring_list_clear() improvements mentioned offhand in\nhttps://lore.kernel.org/git/87bl6kq631.fsf@evledraar.gmail.com/\n\nThis is just from fuzzy memory & I can't recall the specifics (and\nhaven't combed through that WIP code now), but it's something like that\nin the ~100 uses of string_list in our codebase 60-70% are the simple\ncase where the \"strdup_strings\" and string_list_clear() is enough, maybe\nanother 10-20% have a \"util\" field they manage or not, 5%-ish have a\nsimple string_list_clear_func().\n\nIt was just 2-3 cases that leaked memory due to skipping a prefix and\nsticking it in the list, and maybe another 1-2 where the void* to a\nstruct containing the string stuck into the string slot was something we\ncould use.\n\nSo it's not \"common\" in the sense of absolute numbers, but I did run\ninto a handful of them, and having them handled by having the\nstring_list take an arbitrary \"util\" was something I found neat.\n\nI should probably have said \"well known\" (as in \"well known technique\"),\n\"idiomatic\" or something...\n\n>> > I don't think most interfaces take a string_list_item now, so wouldn't\n>> > they similarly need to be changed? Though the point is that all of these\n>> > degrade to a regular C-string, so when you are just passing the value\n>> > (and not ownership), you would just dereference at that point.\n>> \n>> Sure, just like things would need to be changed to handle your proposed\n>> \"struct def_string\".\n>> \n>> By piggy-backing on an already used struct in our codebase we can get a\n>> lot of that memory management pretty much for free without much\n>> churn.\n>> \n>> If you squint and pretend that \"struct string_list_item\" isn't called\n>> something to do with that particular collections API (but it would make\n>> use of it) then we've already set up most of the scaffolding and\n>> management for this.\n>\n> It's that squinting that bothers me. Sure, it's _kinda_ similar. And I\n> don't have any problem with some kind of struct that says \"this is a\n> string, and when you are done with it, this is how you free it\". And I\n> don't have any problem with building the \"dup\" version of string_list\n> with that struct as a primitive. But it seems to me to be orthogonal\n> from the \"util\" pointer of a string_list, which is about creating a\n> mapping from the string to some other thing (which may or may not\n> contain the string, and may or may not be owned).\n\nThe \"util\" is whatever the user makes it. We could add a\n\"pointer_to_free\" to every container type to solve this more\ncleanly/generally at the API level, but just handing the problem to the\nuser seems better to me. I.e. an API like string_list has convenience\nfunctions for freeing all the \"util\", if you only need it for memory\ntracking use it as-is, if you need a \"real util\" *and* such tracking\njust create a 2-member wrapper struct yourself & use that.\n\n> TBH, I have always found the \"util\" field of string_list a bit ugly (and\n> really most of string_list). I think most cases would be better off with\n> a different data structure (a set or a hash table), but we didn't have\n> convenient versions of those for a long time. I don't mind seeing\n> conversions of string_list to other data structures. But that seems to\n> be working against using string_list's string struct in more places.\n\nIf we followed my idle musings we'd be using string_list_item in more\nplaces, not necessarily string_list, and would rename\ns/string_list_item/string_and_util/ or something.\n\nOne way to look at this problem is that we're pretty close to just\nre-inventing the sort of generalized refcounted container type that some\nprogramming languages carry around. E.g. Perl has a \"struct SV*\" that a\n$string maps to, but also hash and array values etc.\n\nThose languages usually have a \"refcount\" or whatever, but since we're\nusing this in native C and it's usually (or at least should be) clear\nwho owns the memory just having something to point free() at will do.\n\nI'm just saying that if we're going halfway there it would be\nunfortunate if we'd end up with a \"struct def_string\" which wouldn't\nhandle this \"borrowing a string from a struct\" case.\n\nOr maybe we should just use \"struct strbuf\" and do copying in even more\nplaces...\n"},{"id":"434520","messageId":"877dfzb0tw.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"YS9OT/pn5rRK9cGB@coredump.intra.peff.net","subject":"Re: [PATCH v3 0/8] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-02T12:25:33Z","receivedAt":"2021-09-02T12:48:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 01 2021, Jeff King wrote:\n\n> On Tue, Aug 31, 2021 at 03:35:34PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> * In v2 compiling with SANITIZE=leak would change things so only\n>> known-good passing tests were run by default, everything else would\n>> pass as a dummy. Now the default running of tests is unchanged, but\n>> if we run with GIT_TEST_PASSING_SANITIZE_LEAK=true only those tests\n>> are run which set and export TEST_PASSES_SANITIZE_LEAK=true.\n>> \n>> * The facility for declaring known-good tests in test-lib.sh based on\n>> wildcards is gone, instead individual tests need to declare if\n>> they're OK under SANITIZE=leak.[...]\n>\n> Hmm. This still seems more complicated than we need. If we just want a\n> flag in each script, then test-lib.sh can use that flag to tweak\n> LSAN_OPTIONS. See the patch below.\n\nOn the \"pragma\" include v.s. env var + export: I figured this would be\neasier to read as I thought the export was required (I don't think it is\nin most cases, but e.g. for t0000*.sh I think it is, but that's from\nmemory...).\n\n> That has two drawbacks:\n>\n> - it doesn't have any way to switch the flag per-test. But IMHO it is\n> a mistake to go in that direction. This is all temporary scaffolding\n> while we have leaks, and the script-level of granularity is fine.\n\nWe have a lot of tests that do simple checking of the tool itself, and\nlater in the script might be stressing trace2, or common sources of\nleaks like \"git log\" in combination with the tool (e.g. the commit-graph\ntests).\n\nSo being able to tweak this inside the script is useful, but that can of\ncourse also be done with this proposed TEST_LSAN_OK + prereq.\n\n> - it runs the tests not marked as LSAN-OK, just without leak checking,\n> which is redundant in CI where we're already running them. But we\n> could still be collecting leak stats (and just not failing the\n> tests). See the patch below.\n\nSure, I'd prefer \n\n> If we do care about not running them, then I think it makes more\n> sense to extend the run/skip mechanisms and build on that.\n\nThe patch I have here is already nicely integrated with the skip\nmechanism. I.e. we use skip_all which shows a summary in any TAP\nconsumer, and we can skip individual tests with prerequisites.\n\n> (I also think I prefer the central list of \"mark these scripts as OK\n> for leak-checking\", rather than annotating individuals. Because\n> again, this is temporary, and it's nice to keep it in a sandbox that\n> only people working on leak-checking would look at or touch).\n>\n> I realize this is kind-of bikeshedding, and I'm not vehemently opposed\n> to what you have here. It just seems like fewer moving parts would be\n> less likely to confuse folks who want to poke at it.\n\nI can see that for the proposed v2 mechanism, but in this v3 nothing\nchanges unless you opt-in to things via new GIT_TEST_* setting. So the\nchance for confusion seems minimal to nonexisting.\n\nI was interested in doing some summaries of existing leaks\neventually. It seems even with LSAN_OPTIONS=detect_leaks=0 compiling\nwith SANITIZE=leak make things a bit slower, but not by much (but actual\nleak checking is much slower).\n\nBut I'd prefer to leave any \"write out leak logs and summarize\" step for\nsome later change.\n\n>> This is done via \"export\n>> TEST_PASSES_SANITIZE_LEAK=true\", there's a handy import of\n>> \"./test-pragma-SANITIZE=leak-ok.sh\" before sourcing \"./test-lib.sh\"\n>> itself to set this.\n>\n> I found the extra level of indirection added by this pragma confusing.\n> We just need to set a variable, which is also a one-liner, and one that\n> is more obvious about what it's doing. In your code you also export it,\n> but that's not necessary for something that test-lib.sh is going to look\n> at. Or if it's really necessary at some point, then test-lib.sh can do\n> the export itself.\n\n*nod*, will remove it per discussion above.\n\n>> Ævar Arnfjörð Bjarmason (8):\n>> Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n>> CI: refactor \"if\" to \"case\" statement\n>> tests: add a test mode for SANITIZE=leak, run it in CI\n>> tests: annotate t000*.sh with TEST_PASSES_SANITIZE_LEAK=true\n>> tests: annotate t001*.sh with TEST_PASSES_SANITIZE_LEAK=true\n>> tests: annotate t002*.sh with TEST_PASSES_SANITIZE_LEAK=true\n>> tests: annotate select t0*.sh with TEST_PASSES_SANITIZE_LEAK=true\n>> tests: annotate select t*.sh with TEST_PASSES_SANITIZE_LEAK=true\n>\n> Sort of a meta-question, but what's the plan for folks who add a new\n> test to say t0000, and it reveals a leak in code they didn't touch?\n\nThen CI will fail on this job. We'd have those same failures now\n(e.g. the mentioned current delta between master..seen), we just don't\nsee them. Having visibility on them seems like an improvement.\n\n> They'll get a CI failure (as will Junio if he picks up the patch), so\n> somebody is going to have to deal with it. Do they fix it? Do they unset\n> the \"this script is OK\" flag? Do they mark the individual test as\n> non-lsan-ok?\n\nI'd think they'd fix it, or make marking the regression as OK part of\ntheir re-roll, just like failures on master..seen now.\n\nIf you're getting at that we should start out this job as an FYI job\nthat doesn't impact the CI run's overall status if it fails I think that\nwould be OK as a start.\n\n> I do like the idea of finding real regressions. But while the state of\n> leak-checking is still so immature, I'm worried about this adding extra\n> friction for developers. Especially if they get some spooky action at a\n> distance caused by a leak in far-away code.\n\nYeah, ultimately this series is an implicit endorsement of us caring\nmore than we do now.\n\nI think this friction point is going to be mitigated a lot by the\nability I've added to not just skip entire test scripts, but allow\nprereq skipping of some tests, early bailing out etc.\n\nIt allows you to say add a \"git log\" test at the end of some test that\notherwise just uses some core API or a test tool and not have to throw\nthe baby out with the bathwater in terms of disabling all existing leak\nchecks there to make forward progress (or split up the entire test\nscript).\n\n> Anyway, here's LSAN_OPTIONS thing I was thinking of.\n\nThanks, that & your follow-up is very interesting. Can I assume this has\nyour SOB? I'd like to add that redirect to fd 4 change to this series.\n\n> diff --git a/t/t0001-init.sh b/t/t0001-init.sh\n> index df544bb321..b1da18955d 100755\n> --- a/t/t0001-init.sh\n> +++ b/t/t0001-init.sh\n> @@ -2,6 +2,7 @@\n> \n> test_description='git init'\n> \n> +TEST_LSAN_OK=1\n> . ./test-lib.sh\n> \n> check_config () {\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index abcfbed6d6..62627afeaf 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -44,9 +44,30 @@ GIT_BUILD_DIR=\"$TEST_DIRECTORY\"/..\n> : ${ASAN_OPTIONS=detect_leaks=0:abort_on_error=1}\n> export ASAN_OPTIONS\n> \n> -# If LSAN is in effect we _do_ want leak checking, but we still\n> -# want to abort so that we notice the problems.\n> -: ${LSAN_OPTIONS=abort_on_error=1}\n> +if test -n \"$LSAN_OPTIONS\"\n> +then\n> +\t# Leave user-provided options alone.\n> +\t:\n> +elif test -n \"$TEST_LSAN_OK\"\n> +then\n> +\t# The test script has declared itself as LSAN-clean; turn on full leak\n> +\t# checking.\n> +\tLSAN_OPTIONS=abort_on_error=1\n> +else\n> +\t# The test script has possible LSAN failures. Just disable\n> +\t# leak-checking entirely. Another option would be to log the failures\n> +\t# with:\n> +\t#\n> +\t# LSAN_OPTIONS=exitcode=0:log_path=$TEST_DIRECTORY/lsan/out\n> +\t#\n> +\t# The results are rather confusing, though, as the logs are\n> +\t# per-process; you have no idea which one came from which test script.\n> +\t# Ideally we'd send them to descriptor 4 along with the rest of the\n> +\t# script log, but there's no LSAN_OPTION for that (recent versions of\n> +\t# libsanitizer do have a public function to do so, so we could hook it\n> +\t# ourselves via common-main).\n> +\tLSAN_OPTIONS=detect_leaks=0\n> +fi\n> export LSAN_OPTIONS\n> \n> if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n\n"},{"id":"434629","messageId":"YTIDXHx/IMtcaQR5@coredump.intra.peff.net","threadId":"55888","inReplyTo":"877dfzb0tw.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 0/8] add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-03T11:13:32Z","receivedAt":"2021-09-03T11:13:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 02, 2021 at 02:25:33PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > Hmm. This still seems more complicated than we need. If we just want a\n> > flag in each script, then test-lib.sh can use that flag to tweak\n> > LSAN_OPTIONS. See the patch below.\n> \n> On the \"pragma\" include v.s. env var + export: I figured this would be\n> easier to read as I thought the export was required (I don't think it is\n> in most cases, but e.g. for t0000*.sh I think it is, but that's from\n> memory...).\n\nI admit that half of my complaint with the pragma is the weird filename\nwith an \"=\" in it. :) But I do think just assigning the variable is the\nmost readable thing. If t0000 needs to export for whatever reason, it\ncan do so (preferably with a comment explaining why).\n\n> > That has two drawbacks:\n> >\n> > - it doesn't have any way to switch the flag per-test. But IMHO it is\n> > a mistake to go in that direction. This is all temporary scaffolding\n> > while we have leaks, and the script-level of granularity is fine.\n> \n> We have a lot of tests that do simple checking of the tool itself, and\n> later in the script might be stressing trace2, or common sources of\n> leaks like \"git log\" in combination with the tool (e.g. the commit-graph\n> tests).\n> \n> So being able to tweak this inside the script is useful, but that can of\n> course also be done with this proposed TEST_LSAN_OK + prereq.\n\nGetting rid of the \"let's tell the tests that we were built with LSAN\"\nwas part of the simplicity I was going for (and obviously does preclude\na prerequisite). I had hoped we wouldn't need to do per-test stuff,\nbecause this was all a temporary state. But maybe that's naive.\n\n> > If we do care about not running them, then I think it makes more\n> > sense to extend the run/skip mechanisms and build on that.\n> \n> The patch I have here is already nicely integrated with the skip\n> mechanism. I.e. we use skip_all which shows a summary in any TAP\n> consumer, and we can skip individual tests with prerequisites.\n\nI meant here that we'd be driving the selection externally from the\ntests using the skip/run mechanisms (something along the lines of what I\nsketched out before).\n\nBut I admit that there isn't really a big difference between the two\napproaches. Since you've coded this one up already, let's go in that\ndirection (i.e., this series).\n\n> I was interested in doing some summaries of existing leaks\n> eventually. It seems even with LSAN_OPTIONS=detect_leaks=0 compiling\n> with SANITIZE=leak make things a bit slower, but not by much (but actual\n> leak checking is much slower).\n> \n> But I'd prefer to leave any \"write out leak logs and summarize\" step for\n> some later change.\n\nOK, I can live with that (especially given how apparently difficult it\nis to convince LSAN to do it).\n\n> > Sort of a meta-question, but what's the plan for folks who add a new\n> > test to say t0000, and it reveals a leak in code they didn't touch?\n> \n> Then CI will fail on this job. We'd have those same failures now\n> (e.g. the mentioned current delta between master..seen), we just don't\n> see them. Having visibility on them seems like an improvement.\n> \n> > They'll get a CI failure (as will Junio if he picks up the patch), so\n> > somebody is going to have to deal with it. Do they fix it? Do they unset\n> > the \"this script is OK\" flag? Do they mark the individual test as\n> > non-lsan-ok?\n> \n> I'd think they'd fix it, or make marking the regression as OK part of\n> their re-roll, just like failures on master..seen now.\n> \n> If you're getting at that we should start out this job as an FYI job\n> that doesn't impact the CI run's overall status if it fails I think that\n> would be OK as a start.\n\nI think that would be OK, but I'm not quite sure of the best way to do\nit. Why don't we start it as a regular required job, and then we can see\nhow often it is causing a headache. If once every few months somebody\nfixes a leak, I'd be happy. If new developers are getting tangled up\nconstantly in unrelated leaks, then that's something we'd need to\nrevisit.\n\n> > I do like the idea of finding real regressions. But while the state of\n> > leak-checking is still so immature, I'm worried about this adding extra\n> > friction for developers. Especially if they get some spooky action at a\n> > distance caused by a leak in far-away code.\n> \n> Yeah, ultimately this series is an implicit endorsement of us caring\n> more than we do now.\n> \n> I think this friction point is going to be mitigated a lot by the\n> ability I've added to not just skip entire test scripts, but allow\n> prereq skipping of some tests, early bailing out etc.\n\nI half-agree with your final paragraph. The biggest friction point I\nthink will be for new folks when CI starts failing, and they don't\nunderstand why (or where the problem is, or how to debug it, etc). But\nlike I said, let's see what happens.\n\n> > Anyway, here's LSAN_OPTIONS thing I was thinking of.\n> \n> Thanks, that & your follow-up is very interesting. Can I assume this has\n> your SOB? I'd like to add that redirect to fd 4 change to this series.\n\nYes, go for it.\n\n-Peff\n"},{"id":"434888","messageId":"cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.8-00000000000-20210831T132546Z-avarab@gmail.com","subject":"[PATCH v4 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T15:33:28Z","receivedAt":"2021-09-07T15:33:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We can compile git with SANITIZE=leak, and have had various efforts in\nthe past such as 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04)\nto plug memory leaks, but have had no CI testing of it to ensure that\nwe don't get regressions. This series adds a GIT_TEST_* mode for\nchecking those regressions, and runs it in CI.\n\nSince I submitted v2 the delta between origin/master..origin/seen\nbroke even t0001-init.sh when run under SANITIZE=leak, so this series\nwill cause test smoke on \"seen\".\n\nThat failure is due to a bug in es/config-based-hooks [1] and the\nhn/reftable topic, i.e. these patches are legitimately catching\nregressions in \"seen\" from day 1.\n\nChanges since v3:\n\n * Much updated commit message\n\n * Re-arranged the t/README change to avoid a conflict with \"seen\".\n\n * Now testing OSX as well as Linux. Full CI passes on top of \"master\"\n on both: https://github.com/avar/git/runs/3535331215\n\n * I ejected the previous 4-8/8 patches of adding SANITIZE=leak\n annotations to various tests, let's focus on the test mode itself\n here and not overly distracting ourselves with whatever other\n regressions on \"seen\" those annotations might cause, I can submit\n those annotations later.\n\n * As noted in the updated commit message I didn't end up going with\n Jeff King's suggestion of supporting LSAN_OPTIONS directly, and\n fixing the \"fd\" the tests write to. All of those things can be\n extended or fixed later.\n\n1. https://lore.kernel.org/git/8735qvyw0p.fsf@evledraar.gmail.com/ [1]\n\nÆvar Arnfjörð Bjarmason (3):\n Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n CI: refactor \"if\" to \"case\" statement\n tests: add a test mode for SANITIZE=leak, run it in CI\n\n .github/workflows/main.yml | 6 ++++++\n Makefile | 5 +++++\n ci/install-dependencies.sh | 6 +++---\n ci/lib.sh | 31 +++++++++++++++++++++----------\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/t0004-unwritable.sh | 3 ++-\n t/test-lib.sh | 21 +++++++++++++++++++++\n 9 files changed, 67 insertions(+), 15 deletions(-)\n\nRange-diff against v3:\n1: 85619728d41 = 1: bdfe2279271 Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n2: 91c36b94eaa ! 2: 6aaa60e3759 CI: refactor \"if\" to \"case\" statement\n @@ Metadata\n ## Commit message ##\n CI: refactor \"if\" to \"case\" statement\n \n - Refactor an \"if\" statement for \"linux-gcc\" to a \"case\" statement in\n - preparation for another case being added to it, and do the same for\n - the \"osx-gcc\" just below it for consistency.\n + Refactor an \"if\" statement for \"linux-gcc\" and \"osx-gcc\" to a \"case\"\n + statement in preparation for another case being added to them.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n3: 7e3577e4e3c ! 3: fffbfc35c00 tests: add a test mode for SANITIZE=leak, run it in CI\n @@ Metadata\n ## Commit message ##\n tests: add a test mode for SANITIZE=leak, run it in CI\n \n - While git can be compiled with SANITIZE=leak there has been no\n - corresponding GIT_TEST_* mode for it, i.e. memory leaks have been\n - fixed as one-offs without structured regression testing.\n + While git can be compiled with SANITIZE=leak we have not run\n + regression tests under that mode, memory leaks have only been fixed as\n + one-offs without structured regression testing.\n \n - This change add such a mode, and a new linux-SANITIZE=leak CI\n - target. The test mode and CI target only runs a whitelist of\n - known-good tests using a mechanism discussed below, to ensure that we\n - won't add regressions to code that's had its memory leaks fixed.\n + This change add CI testing for it. We'll now build with GCC under\n + Linux and test t000[04]*.sh with SANITIZE=leak, and likewise with GCC\n + on OSX. The new jobs are called \"linux-SANITIZE=leak\" and\n + \"osx-SANITIZE=leak\".\n \n The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n - mode. When running in that mode all tests except those that have opted\n - themselves in to running by setting and exporting\n - TEST_PASSES_SANITIZE_LEAK=true before sourcing test-lib.sh.\n + mode. When running in that mode, we'll assert that we were compiled\n + with SANITIZE=leak, and then skip all tests except those that we've\n + opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n + test-lib.sh (see discussion in t/README).\n \n - I'm adding a \"test-pragma-SANITIZE=leak-ok.sh\" wrapper for setting and\n - exporting that variable, as the assignment/export boilerplate would\n - otherwise get quite verbose and repetitive in subsequent commits.\n + The tests using the \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\n + turn make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\n + selectively skip tests even under\n + \"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In a preceding commit we\n + started doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\n + it'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n \n - The tests using the \"test-pragma-SANITIZE=leak-ok.sh\" pragma can in\n - turn make use of the \"SANITIZE_LEAK\" prerequisite added in a preceding\n - commit, should they wish to selectively skip tests even under\n - \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n -\n - Now tests that don't set the \"test-pragma-SANITIZE=leak-ok.sh\" pragma\n - will be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n + Now tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will be\n + skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n \n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n \n - In subsequents commit we'll conservatively add more\n - TEST_PASSES_SANITIZE_LEAK=true annotations. The idea is that as memory\n - leaks are fixed we can add more known-good tests to this CI target, to\n - ensure that we won't have regressions.\n + The intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\n + as follow-up change, but let's start small to begin with.\n +\n + It would also be possible to implement a more lightweight version of\n + this by only relying on setting \"LSAN_OPTIONS\". See\n + [1] and\n + [2] for a discussion of\n + that. I've opted for this approach of adding a GIT_TEST_* mode instead\n + because it's consistent with how we handle other special test modes.\n +\n + Being able to add a \"!SANITIZE_LEAK\" prerequisite and calling\n + \"test_done\" early if it isn't satisfied also means that we can more\n + incrementally add regression tests without being forced to fix\n + widespread and hard-to-fix leaks at the same time.\n +\n + We have tests that do simple checking of some tool we're interested\n + in, but later on in the script might be stressing trace2, or common\n + sources of leaks like \"git log\" in combination with the tool (e.g. the\n + commit-graph tests). To be clear having a prerequisite could also be\n + accomplished by using \"LSAN_OPTIONS\" directly.\n +\n + On the topi of \"LSAN_OPTIONS\": It would be nice to have a mode to\n + aggregate all failures in our various scripts, see [2] for a start at\n + doing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\n + that for now, it can be added later, and that proposed patch is also\n + hindered by us wanting to test e.g. test-tool leaks (and by proxy, any\n + API leaks they uncover), not just the \"common-main.c\" entry point.\n \n As of writing this we've got major regressions between master..seen,\n i.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n @@ Commit message\n 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n past history of \"one-off\" SANITIZE=leak (and more) fixes.\n \n + The reason for using gcc on OSX over the clang default is because\n + it'll currently fail to build with:\n +\n + clang: error: unsupported option '-fsanitize=leak' for target 'x86_64-apple-darwin19.6.0'\n +\n + If that's sorted out in the future we might want to run that job with\n + \"clang\" merely to make use of the default, and also to add some\n + compiler variance into the mix. Both use the\n + \"AddressSanitizerLeakSanitizer\" library[3], so in they shouldn't be\n + have differently under GCC or clang.\n +\n + 1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n + 2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n + 3. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## .github/workflows/main.yml ##\n @@ .github/workflows/main.yml: jobs:\n cc: gcc\n pool: ubuntu-latest\n + - jobname: linux-SANITIZE=leak\n ++ cc: gcc\n + pool: ubuntu-latest\n ++ - jobname: osx-SANITIZE=leak\n ++ cc: gcc\n ++ pool: macos-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\n @@ ci/install-dependencies.sh: UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n +@@ ci/install-dependencies.sh: linux-clang|linux-gcc)\n + \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n + \tpopd\n + \t;;\n +-osx-clang|osx-gcc)\n ++osx-clang|osx-gcc|osx-SANITIZE=leak)\n + \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n + \t# Uncomment this if you want to run perf tests:\n + \t# brew install gnu-time\n \n ## ci/lib.sh ##\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n \t\t;;\n +@@ ci/lib.sh: linux-clang|linux-gcc)\n + \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n + \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n + \t;;\n +-osx-clang|osx-gcc)\n ++osx-clang|osx-gcc|osx-SANITIZE=leak)\n + \tcase \"$jobname\" in\n +-\tosx-gcc)\n ++\tosx-gcc|osx-SANITIZE=leak)\n + \t\texport CC=gcc-9\n + \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n + \t\t;;\n @@ ci/lib.sh: linux-musl)\n \t;;\n esac\n \n +case \"$jobname\" in\n -+linux-SANITIZE=leak)\n ++linux-SANITIZE=leak|osx-SANITIZE=leak)\n +\texport SANITIZE=leak\n +\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n +\t;;\n @@ ci/run-build-and-tests.sh: esac\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n -@@ ci/run-build-and-tests.sh: linux-gcc)\n - \texport GIT_TEST_CHECKOUT_WORKERS=2\n - \tmake test\n - \t;;\n --linux-clang)\n -+linux-clang|linux-SANITIZE=leak)\n - \texport GIT_TEST_DEFAULT_HASH=sha1\n - \tmake test\n - \texport GIT_TEST_DEFAULT_HASH=sha256\n \n ## t/README ##\n -@@ t/README: GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n - to and 'checkout.thresholdForParallelism' to 0, forcing the\n - execution of the parallel-checkout code.\n +@@ t/README: excluded as so much relies on it, but this might change in the future.\n + GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n + test suite. Accept any boolean values that are accepted by git-config.\n \n +GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n +SANITIZE=leak will run only those tests that have whitelisted\n -+themselves as passing with no memory leaks. Do this by sourcing\n -+\"test-pragma-SANITIZE=leak-ok.sh\" before sourcing \"test-lib.sh\" itself\n -+at the top of the test script. This test mode is used by the\n -+\"linux-SANITIZE=leak\" CI target.\n ++themselves as passing with no memory leaks. Tests can be whitelisted\n ++by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n ++\"test-lib.sh\" itself at the top of the test script. This test mode is\n ++used by the \"linux-SANITIZE=leak\" CI target.\n +\n - Naming Tests\n - ------------\n + GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n + default to n.\n \n \n ## t/t0000-basic.sh ##\n @@ t/t0000-basic.sh: swapping compression and hashing order, the person who is maki\n modification *should* take notice and update the test vectors here.\n '\n \n -+. ./test-pragma-SANITIZE=leak-ok.sh\n ++TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n try_local_xy () {\n \n + ## t/t0004-unwritable.sh ##\n +@@\n + \n + test_description='detect unwritable repository and fail correctly'\n + \n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + test_expect_success setup '\n +\n ## t/test-lib.sh ##\n @@ t/test-lib.sh: then\n \ttest_done\n fi\n \n -+# Aggressively skip non-whitelisted tests when compiled with\n -+# SANITIZE=leak\n ++# skip non-whitelisted tests when compiled with SANITIZE=leak\n +if test -n \"$SANITIZE_LEAK\"\n +then\n +\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n @@ t/test-lib.sh: then\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n -\n - ## t/test-pragma-SANITIZE=leak-ok.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+## This \"pragma\" (as in \"perldoc perlpragma\") declares that the test\n -+## will pass under GIT_TEST_PASSING_SANITIZE_LEAK=true. Source this\n -+## before sourcing test-lib.sh\n -+\n -+TEST_PASSES_SANITIZE_LEAK=true\n -+export TEST_PASSES_SANITIZE_LEAK\n4: 0cd14d64165 < -: ----------- tests: annotate t000*.sh with TEST_PASSES_SANITIZE_LEAK=true\n5: ed5f5705755 < -: ----------- tests: annotate t001*.sh with TEST_PASSES_SANITIZE_LEAK=true\n6: 2599016c4e7 < -: ----------- tests: annotate t002*.sh with TEST_PASSES_SANITIZE_LEAK=true\n7: ddc4d6d2cf1 < -: ----------- tests: annotate select t0*.sh with TEST_PASSES_SANITIZE_LEAK=true\n8: e611d2c23d9 < -: ----------- tests: annotate select t*.sh with TEST_PASSES_SANITIZE_LEAK=true\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434887","messageId":"patch-v4-1.3-bdfe2279271-20210907T151855Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com","subject":"[PATCH v4 1/3] Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T15:33:29Z","receivedAt":"2021-09-07T15:33:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When SANITIZE=leak is specified we'll now add a SANITIZE_LEAK flag to\nGIT-BUILD-OPTIONS, this can then be picked up by the test-lib.sh,\nwhich sets a SANITIZE_LEAK prerequisite.\n\nWe can then skip specific tests that are known to fail under\nSANITIZE=leak, add one such annotation to t0004-unwritable.sh, which\nnow passes under SANITIZE=leak.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++++\n t/t0004-unwritable.sh | 2 +-\n t/test-lib.sh | 1 +\n 3 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 429c276058d..34c12ea6e6f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1221,6 +1221,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1265,6 +1268,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2812,6 +2816,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex e3137d638ee..fbdcb926b3a 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -21,7 +21,7 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi\n \ttest_must_fail git write-tree\n '\n \n-test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository' '\n+test_expect_success POSIXPERM,SANITY,!SANITIZE_LEAK 'commit should notice unwritable repository' '\n \ttest_when_finished \"chmod 775 .git/objects .git/objects/??\" &&\n \tchmod a-w .git/objects .git/objects/?? &&\n \ttest_must_fail git commit -m second\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d61..4ab18914a3d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1533,6 +1533,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434889","messageId":"patch-v4-2.3-6aaa60e3759-20210907T151855Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com","subject":"[PATCH v4 2/3] CI: refactor \"if\" to \"case\" statement","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T15:33:30Z","receivedAt":"2021-09-07T15:33:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor an \"if\" statement for \"linux-gcc\" and \"osx-gcc\" to a \"case\"\nstatement in preparation for another case being added to them.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..33b9777ab7e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -184,13 +184,15 @@ export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n linux-clang|linux-gcc)\n-\tif [ \"$jobname\" = linux-gcc ]\n-\tthen\n+\tcase \"$jobname\" in\n+\tlinux-gcc)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \texport GIT_TEST_HTTPD=true\n \n@@ -207,13 +209,15 @@ linux-clang|linux-gcc)\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n osx-clang|osx-gcc)\n-\tif [ \"$jobname\" = osx-gcc ]\n-\tthen\n+\tcase \"$jobname\" in\n+\tosx-gcc)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \t# t9810 occasionally fails on Travis CI OS X\n \t# t9816 occasionally fails with \"TAP out of sequence errors\" on\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434890","messageId":"patch-v4-3.3-fffbfc35c00-20210907T151855Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com","subject":"[PATCH v4 3/3] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T15:33:31Z","receivedAt":"2021-09-07T15:33:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak we have not run\nregression tests under that mode, memory leaks have only been fixed as\none-offs without structured regression testing.\n\nThis change add CI testing for it. We'll now build with GCC under\nLinux and test t000[04]*.sh with SANITIZE=leak, and likewise with GCC\non OSX. The new jobs are called \"linux-SANITIZE=leak\" and\n\"osx-SANITIZE=leak\".\n\nThe CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\nmode. When running in that mode, we'll assert that we were compiled\nwith SANITIZE=leak, and then skip all tests except those that we've\nopted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\ntest-lib.sh (see discussion in t/README).\n\nThe tests using the \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\nturn make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\nselectively skip tests even under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In a preceding commit we\nstarted doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\nit'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nNow tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will be\nskipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n\nThe intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\nas follow-up change, but let's start small to begin with.\n\nIt would also be possible to implement a more lightweight version of\nthis by only relying on setting \"LSAN_OPTIONS\". See\n[1] and\n[2] for a discussion of\nthat. I've opted for this approach of adding a GIT_TEST_* mode instead\nbecause it's consistent with how we handle other special test modes.\n\nBeing able to add a \"!SANITIZE_LEAK\" prerequisite and calling\n\"test_done\" early if it isn't satisfied also means that we can more\nincrementally add regression tests without being forced to fix\nwidespread and hard-to-fix leaks at the same time.\n\nWe have tests that do simple checking of some tool we're interested\nin, but later on in the script might be stressing trace2, or common\nsources of leaks like \"git log\" in combination with the tool (e.g. the\ncommit-graph tests). To be clear having a prerequisite could also be\naccomplished by using \"LSAN_OPTIONS\" directly.\n\nOn the topi of \"LSAN_OPTIONS\": It would be nice to have a mode to\naggregate all failures in our various scripts, see [2] for a start at\ndoing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\nthat for now, it can be added later, and that proposed patch is also\nhindered by us wanting to test e.g. test-tool leaks (and by proxy, any\nAPI leaks they uncover), not just the \"common-main.c\" entry point.\n\nAs of writing this we've got major regressions between master..seen,\ni.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n'ah/plugleaks', 2021-08-04) have regressed recently.\n\nSee the discussion at <87czsv2idy.fsf@evledraar.gmail.com> about the\nlack of this sort of test mode, and 0e5bba53af (add UNLEAK annotation\nfor reducing leak false positives, 2017-09-08) for the initial\naddition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nThe reason for using gcc on OSX over the clang default is because\nit'll currently fail to build with:\n\n clang: error: unsupported option '-fsanitize=leak' for target 'x86_64-apple-darwin19.6.0'\n\nIf that's sorted out in the future we might want to run that job with\n\"clang\" merely to make use of the default, and also to add some\ncompiler variance into the mix. Both use the\n\"AddressSanitizerLeakSanitizer\" library[3], so in they shouldn't be\nhave differently under GCC or clang.\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n3. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 6 ++++++\n ci/install-dependencies.sh | 6 +++---\n ci/lib.sh | 15 +++++++++++----\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/t0004-unwritable.sh | 1 +\n t/test-lib.sh | 20 ++++++++++++++++++++\n 8 files changed, 50 insertions(+), 8 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 68596f25927..b41572293c9 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -232,6 +232,12 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-SANITIZE=leak\n+ cc: gcc\n+ pool: ubuntu-latest\n+ - jobname: osx-SANITIZE=leak\n+ cc: gcc\n+ pool: macos-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 5772081b6e5..a89e72c1438 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-SANITIZE=leak)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-SANITIZE=leak)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n@@ -37,7 +37,7 @@ linux-clang|linux-gcc)\n \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n \tpopd\n \t;;\n-osx-clang|osx-gcc)\n+osx-clang|osx-gcc|osx-SANITIZE=leak)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 33b9777ab7e..36b7c0d3020 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,9 +183,9 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-SANITIZE=leak)\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-SANITIZE=leak)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n \t\t;;\n@@ -208,9 +208,9 @@ linux-clang|linux-gcc)\n \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n-osx-clang|osx-gcc)\n+osx-clang|osx-gcc|osx-SANITIZE=leak)\n \tcase \"$jobname\" in\n-\tosx-gcc)\n+\tosx-gcc|osx-SANITIZE=leak)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n \t\t;;\n@@ -237,4 +237,11 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-SANITIZE=leak|osx-SANITIZE=leak)\n+\texport SANITIZE=leak\n+\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee94..4133239fc36 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -12,7 +12,7 @@ esac\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc|linux-SANITIZE=leak)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\ndiff --git a/t/README b/t/README\nindex 9e701223020..4864f208c8a 100644\n--- a/t/README\n+++ b/t/README\n@@ -366,6 +366,13 @@ excluded as so much relies on it, but this might change in the future.\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n test suite. Accept any boolean values that are accepted by git-config.\n \n+GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n+SANITIZE=leak will run only those tests that have whitelisted\n+themselves as passing with no memory leaks. Tests can be whitelisted\n+by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n+\"test-lib.sh\" itself at the top of the test script. This test mode is\n+used by the \"linux-SANITIZE=leak\" CI target.\n+\n GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n default to n.\n \ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex cb87768513c..54318af3861 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -18,6 +18,7 @@ swapping compression and hashing order, the person who is making the\n modification *should* take notice and update the test vectors here.\n '\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n try_local_xy () {\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex fbdcb926b3a..37d68ef03be 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -2,6 +2,7 @@\n \n test_description='detect unwritable repository and fail correctly'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 4ab18914a3d..3b7acfec23b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1379,6 +1379,26 @@ then\n \ttest_done\n fi\n \n+# skip non-whitelisted tests when compiled with SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+\tthen\n+\t\t# We need to see it in \"git env--helper\" (via\n+\t\t# test_bool_env)\n+\t\texport TEST_PASSES_SANITIZE_LEAK\n+\n+\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n+\t\tthen\n+\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n+\t\t\ttest_done\n+\t\tfi\n+\tfi\n+elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434893","messageId":"CAPig+cSPwd4XviefDUieoKPXQKnONhb5dqPWZ-+NQtS74ottMw@mail.gmail.com","threadId":"55888","inReplyTo":"patch-v4-3.3-fffbfc35c00-20210907T151855Z-avarab@gmail.com","subject":"Re: [PATCH v4 3/3] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-09-07T16:29:57Z","receivedAt":"2021-09-07T16:30:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Sep 7, 2021 at 11:33 AM Ævar Arnfjörð Bjarmason\n wrote:\n> [...]\n> On the topi of \"LSAN_OPTIONS\": It would be nice to have a mode to\n> aggregate all failures in our various scripts, see [2] for a start at\n> doing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\n> that for now, it can be added later, and that proposed patch is also\n> hindered by us wanting to test e.g. test-tool leaks (and by proxy, any\n> API leaks they uncover), not just the \"common-main.c\" entry point.\n\ns/topi/topic/\n"},{"id":"434895","messageId":"YTeW+MpIVNCcd2nF@coredump.intra.peff.net","threadId":"55888","inReplyTo":"cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com","subject":"Re: [PATCH v4 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-07T16:44:40Z","receivedAt":"2021-09-07T16:44:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 07, 2021 at 05:33:28PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Changes since v3:\n> \n> * Much updated commit message\n> \n> * Re-arranged the t/README change to avoid a conflict with \"seen\".\n> \n> * Now testing OSX as well as Linux. Full CI passes on top of \"master\"\n> on both: https://github.com/avar/git/runs/3535331215\n> \n> * I ejected the previous 4-8/8 patches of adding SANITIZE=leak\n> annotations to various tests, let's focus on the test mode itself\n> here and not overly distracting ourselves with whatever other\n> regressions on \"seen\" those annotations might cause, I can submit\n> those annotations later.\n> \n> * As noted in the updated commit message I didn't end up going with\n> Jeff King's suggestion of supporting LSAN_OPTIONS directly, and\n> fixing the \"fd\" the tests write to. All of those things can be\n> extended or fixed later.\n\nOK, I think we should proceed with this series/approach, then. The\nquestion of friction when CI fails is an open one, but we won't know\nuntil we have more data. So let's see what happens. :)\n\nThe patches themselves look fine to me, though I had a few nits on the\nthird commit message.\n\n-Peff\n"},{"id":"434897","messageId":"YTeYrq1/BlFobAGk@coredump.intra.peff.net","threadId":"55888","inReplyTo":"patch-v4-3.3-fffbfc35c00-20210907T151855Z-avarab@gmail.com","subject":"Re: [PATCH v4 3/3] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-07T16:51:58Z","receivedAt":"2021-09-07T16:52:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 07, 2021 at 05:33:31PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Subject: [PATCH v4 3/3] tests: add a test mode for SANITIZE=leak, run it in CI\n\nThe patch looks OK to me. There are a bunch of typos/nits in the commit\nmessage which made it a little harder to read. I don't care _that_ much,\nbut there's one inaccuracy I wanted to point out, and the others are\nalong for the ride. :)\n\n> While git can be compiled with SANITIZE=leak we have not run\n> regression tests under that mode, memory leaks have only been fixed as\n> one-offs without structured regression testing.\n\nFunky comma placement. Maybe:\n\n While git can be compiled with SANITIZE=leak, we have not run\n regression tests under that mode. Memory leaks have only been fixed as\n one-offs without structured regression testing.\n\n> This change add CI testing for it. We'll now build with GCC under\n> Linux and test t000[04]*.sh with SANITIZE=leak, and likewise with GCC\n> on OSX. The new jobs are called \"linux-SANITIZE=leak\" and\n> \"osx-SANITIZE=leak\".\n\ns/add/adds/\n\nA matter of taste, but I find the \"linux-SANITIZE=leak\" a little funny\nto read because of the mixed-caps and punctuation. Just linux-leaks or\nsomething is descriptive enough. Pure bikeshedding, of course.\n\n> On the topi of \"LSAN_OPTIONS\": It would be nice to have a mode to\n> aggregate all failures in our various scripts, see [2] for a start at\n> doing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\n> that for now, it can be added later, and that proposed patch is also\n> hindered by us wanting to test e.g. test-tool leaks (and by proxy, any\n> API leaks they uncover), not just the \"common-main.c\" entry point.\n\nI think test-tool does actually use common-main.c, so we'd be covered\nthere, too. That said, I'm perfectly fine to leave this for now (or\nperhaps never; if we can get the whole suite passing with leak-checking\non, then aggregating the many leak reports without having test failures\nwill become a moot point).\n\n> +# skip non-whitelisted tests when compiled with SANITIZE=leak\n> +if test -n \"$SANITIZE_LEAK\"\n> +then\n> +\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n> +\tthen\n> +\t\t# We need to see it in \"git env--helper\" (via\n> +\t\t# test_bool_env)\n> +\t\texport TEST_PASSES_SANITIZE_LEAK\n> +\n> +\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n> +\t\tthen\n> +\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n> +\t\t\ttest_done\n> +\t\tfi\n> +\tfi\n> +elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n> +then\n> +\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n> +fi\n\nI wondered if it would be helpful for this to be more forgiving. But\nthere's not much point in setting GIT_TEST_PASSING_SANITIZE_LEAK all the\ntime (say, in your config.mak), since it will just skip a bunch of\ntests. So it probably does make sense to alert the user that \"oops, you\ndid not actually build things correctly\".\n\n-Peff\n"},{"id":"434907","messageId":"xmqqtuiwb5z9.fsf@gitster.g","threadId":"55888","inReplyTo":"YTeW+MpIVNCcd2nF@coredump.intra.peff.net","subject":"Re: [PATCH v4 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-07T18:22:50Z","receivedAt":"2021-09-07T18:22:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Tue, Sep 07, 2021 at 05:33:28PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> OK, I think we should proceed with this series/approach, then. The\n> question of friction when CI fails is an open one, but we won't know\n> until we have more data. So let's see what happens. :)\n>\n> The patches themselves look fine to me, though I had a few nits on the\n> third commit message.\n\nThank you all.\n\nLet's see a copy-edited v5 and we can go from there.\n\n\n"},{"id":"434956","messageId":"cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com","subject":"[PATCH v5 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T21:30:36Z","receivedAt":"2021-09-07T21:30:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We can compile git with SANITIZE=leak, and have had various efforts in\nthe past such as 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04)\nto plug memory leaks, but have had no CI testing of it to ensure that\nwe don't get regressions. This series adds a GIT_TEST_* mode for\nchecking those regressions, and runs it in CI.\n\nSince I submitted v2 the delta between origin/master..origin/seen\nbroke even t0001-init.sh when run under SANITIZE=leak, so this series\nwill cause test smoke on \"seen\".\n\nThat failure is due to a bug in es/config-based-hooks [1] and the\nhn/reftable topic, i.e. these patches are legitimately catching\nregressions in \"seen\" from day 1.\n\nChanges since v4 (see\nhttps://lore.kernel.org/git/cover-v4-0.3-00000000000-20210907T151855Z-avarab@gmail.com/):\n\n * Renamed the jobs to linux-leaks and osx-leaks, per Jeff King's\n suggestion.\n\n * Took all the suggestions from Jeff King for commit message\n improvements, and tried to make some of my own fixing overly\n verbose wording/grammar errors etc.\n\n * Ditto the small typo fix Eric Sunshine pointed out. Thanks both!\n\nSee https://github.com/avar/git/runs/3538356269 for the CI run for\nthis version.\n\nÆvar Arnfjörð Bjarmason (3):\n Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n CI: refactor \"if\" to \"case\" statement\n tests: add a test mode for SANITIZE=leak, run it in CI\n\n .github/workflows/main.yml | 6 ++++++\n Makefile | 5 +++++\n ci/install-dependencies.sh | 6 +++---\n ci/lib.sh | 31 +++++++++++++++++++++----------\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/t0004-unwritable.sh | 3 ++-\n t/test-lib.sh | 21 +++++++++++++++++++++\n 9 files changed, 67 insertions(+), 15 deletions(-)\n\nRange-diff against v4:\n1: bdfe2279271 = 1: bdfe2279271 Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n2: 6aaa60e3759 = 2: 6aaa60e3759 CI: refactor \"if\" to \"case\" statement\n3: fffbfc35c00 ! 3: f3cd04b16d1 tests: add a test mode for SANITIZE=leak, run it in CI\n @@ Metadata\n ## Commit message ##\n tests: add a test mode for SANITIZE=leak, run it in CI\n \n - While git can be compiled with SANITIZE=leak we have not run\n - regression tests under that mode, memory leaks have only been fixed as\n + While git can be compiled with SANITIZE=leak, we have not run\n + regression tests under that mode. Memory leaks have only been fixed as\n one-offs without structured regression testing.\n \n - This change add CI testing for it. We'll now build with GCC under\n - Linux and test t000[04]*.sh with SANITIZE=leak, and likewise with GCC\n - on OSX. The new jobs are called \"linux-SANITIZE=leak\" and\n - \"osx-SANITIZE=leak\".\n + This change adds CI testing for it. We'll now build and test\n + t000[04]*.sh under both Linux and OSX. The new jobs are called\n + \"linux-leaks\" and \"osx-leaks\".\n \n The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n mode. When running in that mode, we'll assert that we were compiled\n - with SANITIZE=leak, and then skip all tests except those that we've\n - opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n - test-lib.sh (see discussion in t/README).\n + with SANITIZE=leak. We'll then skip all tests, except those that we've\n + opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n \n - The tests using the \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\n + A test tests setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\n turn make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\n selectively skip tests even under\n \"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In a preceding commit we\n started doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\n it'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n \n - Now tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will be\n - skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n + This is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\n + be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n \n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n @@ Commit message\n commit-graph tests). To be clear having a prerequisite could also be\n accomplished by using \"LSAN_OPTIONS\" directly.\n \n - On the topi of \"LSAN_OPTIONS\": It would be nice to have a mode to\n + On the topic of \"LSAN_OPTIONS\": It would be nice to have a mode to\n aggregate all failures in our various scripts, see [2] for a start at\n doing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\n - that for now, it can be added later, and that proposed patch is also\n - hindered by us wanting to test e.g. test-tool leaks (and by proxy, any\n - API leaks they uncover), not just the \"common-main.c\" entry point.\n + that for now, it can be added later.\n \n As of writing this we've got major regressions between master..seen,\n i.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n 'ah/plugleaks', 2021-08-04) have regressed recently.\n \n - See the discussion at <87czsv2idy.fsf@evledraar.gmail.com> about the\n - lack of this sort of test mode, and 0e5bba53af (add UNLEAK annotation\n - for reducing leak false positives, 2017-09-08) for the initial\n - addition of SANITIZE=leak.\n + See the discussion at <87czsv2idy.fsf@evledraar.gmail.com>[3] about\n + the lack of this sort of test mode, and 0e5bba53af (add UNLEAK\n + annotation for reducing leak false positives, 2017-09-08) for the\n + initial addition of SANITIZE=leak.\n \n See also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n 7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n past history of \"one-off\" SANITIZE=leak (and more) fixes.\n \n - The reason for using gcc on OSX over the clang default is because\n - it'll currently fail to build with:\n + The reason for using gcc on OSX over the clang default is because when\n + used with clang on \"macos-latest\" it'll currently fail to build with:\n \n clang: error: unsupported option '-fsanitize=leak' for target 'x86_64-apple-darwin19.6.0'\n \n If that's sorted out in the future we might want to run that job with\n \"clang\" merely to make use of the default, and also to add some\n compiler variance into the mix. Both use the\n - \"AddressSanitizerLeakSanitizer\" library[3], so in they shouldn't be\n - have differently under GCC or clang.\n + \"AddressSanitizerLeakSanitizer\" library[4], so in they shouldn't\n + behave differently under GCC or clang.\n \n 1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n 2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n - 3. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n + 3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n + 4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ .github/workflows/main.yml: jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n -+ - jobname: linux-SANITIZE=leak\n ++ - jobname: linux-leaks\n + cc: gcc\n + pool: ubuntu-latest\n -+ - jobname: osx-SANITIZE=leak\n ++ - jobname: osx-leaks\n + cc: gcc\n + pool: macos-latest\n env:\n @@ ci/install-dependencies.sh: UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl\n \n case \"$jobname\" in\n -linux-clang|linux-gcc)\n -+linux-clang|linux-gcc|linux-SANITIZE=leak)\n ++linux-clang|linux-gcc|linux-leaks)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n -\tlinux-gcc)\n -+\tlinux-gcc|linux-SANITIZE=leak)\n ++\tlinux-gcc|linux-leaks)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n @@ ci/install-dependencies.sh: linux-clang|linux-gcc)\n \tpopd\n \t;;\n -osx-clang|osx-gcc)\n -+osx-clang|osx-gcc|osx-SANITIZE=leak)\n ++osx-clang|osx-gcc|osx-leaks)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n \n case \"$jobname\" in\n -linux-clang|linux-gcc)\n -+linux-clang|linux-gcc|linux-SANITIZE=leak)\n ++linux-clang|linux-gcc|linux-leaks)\n \tcase \"$jobname\" in\n -\tlinux-gcc)\n -+\tlinux-gcc|linux-SANITIZE=leak)\n ++\tlinux-gcc|linux-leaks)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n \t\t;;\n @@ ci/lib.sh: linux-clang|linux-gcc)\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n -osx-clang|osx-gcc)\n -+osx-clang|osx-gcc|osx-SANITIZE=leak)\n ++osx-clang|osx-gcc|osx-leaks)\n \tcase \"$jobname\" in\n -\tosx-gcc)\n -+\tosx-gcc|osx-SANITIZE=leak)\n ++\tosx-gcc|osx-leaks)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n \t\t;;\n @@ ci/lib.sh: linux-musl)\n esac\n \n +case \"$jobname\" in\n -+linux-SANITIZE=leak|osx-SANITIZE=leak)\n ++linux-leaks|osx-leaks)\n +\texport SANITIZE=leak\n +\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n +\t;;\n @@ ci/run-build-and-tests.sh: esac\n make\n case \"$jobname\" in\n -linux-gcc)\n -+linux-gcc|linux-SANITIZE=leak)\n ++linux-gcc|linux-leaks)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\n @@ t/README: excluded as so much relies on it, but this might change in the future.\n +themselves as passing with no memory leaks. Tests can be whitelisted\n +by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n +\"test-lib.sh\" itself at the top of the test script. This test mode is\n -+used by the \"linux-SANITIZE=leak\" CI target.\n ++used by the \"linux-leaks\" CI target.\n +\n GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n default to n.\n-- \n2.33.0.819.g59feb45f5e0\n\n"},{"id":"434955","messageId":"patch-v5-1.3-bdfe2279271-20210907T212626Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com","subject":"[PATCH v5 1/3] Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T21:30:37Z","receivedAt":"2021-09-07T21:30:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When SANITIZE=leak is specified we'll now add a SANITIZE_LEAK flag to\nGIT-BUILD-OPTIONS, this can then be picked up by the test-lib.sh,\nwhich sets a SANITIZE_LEAK prerequisite.\n\nWe can then skip specific tests that are known to fail under\nSANITIZE=leak, add one such annotation to t0004-unwritable.sh, which\nnow passes under SANITIZE=leak.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++++\n t/t0004-unwritable.sh | 2 +-\n t/test-lib.sh | 1 +\n 3 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 429c276058d..34c12ea6e6f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1221,6 +1221,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1265,6 +1268,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2812,6 +2816,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex e3137d638ee..fbdcb926b3a 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -21,7 +21,7 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi\n \ttest_must_fail git write-tree\n '\n \n-test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository' '\n+test_expect_success POSIXPERM,SANITY,!SANITIZE_LEAK 'commit should notice unwritable repository' '\n \ttest_when_finished \"chmod 775 .git/objects .git/objects/??\" &&\n \tchmod a-w .git/objects .git/objects/?? &&\n \ttest_must_fail git commit -m second\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d61..4ab18914a3d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1533,6 +1533,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.33.0.819.g59feb45f5e0\n\n"},{"id":"434957","messageId":"patch-v5-2.3-6aaa60e3759-20210907T212626Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com","subject":"[PATCH v5 2/3] CI: refactor \"if\" to \"case\" statement","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T21:30:38Z","receivedAt":"2021-09-07T21:30:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor an \"if\" statement for \"linux-gcc\" and \"osx-gcc\" to a \"case\"\nstatement in preparation for another case being added to them.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..33b9777ab7e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -184,13 +184,15 @@ export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n linux-clang|linux-gcc)\n-\tif [ \"$jobname\" = linux-gcc ]\n-\tthen\n+\tcase \"$jobname\" in\n+\tlinux-gcc)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \texport GIT_TEST_HTTPD=true\n \n@@ -207,13 +209,15 @@ linux-clang|linux-gcc)\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n osx-clang|osx-gcc)\n-\tif [ \"$jobname\" = osx-gcc ]\n-\tthen\n+\tcase \"$jobname\" in\n+\tosx-gcc)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n-\telse\n+\t\t;;\n+\t*)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n-\tfi\n+\t\t;;\n+\tesac\n \n \t# t9810 occasionally fails on Travis CI OS X\n \t# t9816 occasionally fails with \"TAP out of sequence errors\" on\n-- \n2.33.0.819.g59feb45f5e0\n\n"},{"id":"434958","messageId":"patch-v5-3.3-f3cd04b16d1-20210907T212626Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com","subject":"[PATCH v5 3/3] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T21:30:39Z","receivedAt":"2021-09-07T21:30:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak, we have not run\nregression tests under that mode. Memory leaks have only been fixed as\none-offs without structured regression testing.\n\nThis change adds CI testing for it. We'll now build and test\nt000[04]*.sh under both Linux and OSX. The new jobs are called\n\"linux-leaks\" and \"osx-leaks\".\n\nThe CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\nmode. When running in that mode, we'll assert that we were compiled\nwith SANITIZE=leak. We'll then skip all tests, except those that we've\nopted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n\nA test tests setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\nturn make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\nselectively skip tests even under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In a preceding commit we\nstarted doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\nit'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nThis is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\nbe skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n\nThe intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\nas follow-up change, but let's start small to begin with.\n\nIt would also be possible to implement a more lightweight version of\nthis by only relying on setting \"LSAN_OPTIONS\". See\n[1] and\n[2] for a discussion of\nthat. I've opted for this approach of adding a GIT_TEST_* mode instead\nbecause it's consistent with how we handle other special test modes.\n\nBeing able to add a \"!SANITIZE_LEAK\" prerequisite and calling\n\"test_done\" early if it isn't satisfied also means that we can more\nincrementally add regression tests without being forced to fix\nwidespread and hard-to-fix leaks at the same time.\n\nWe have tests that do simple checking of some tool we're interested\nin, but later on in the script might be stressing trace2, or common\nsources of leaks like \"git log\" in combination with the tool (e.g. the\ncommit-graph tests). To be clear having a prerequisite could also be\naccomplished by using \"LSAN_OPTIONS\" directly.\n\nOn the topic of \"LSAN_OPTIONS\": It would be nice to have a mode to\naggregate all failures in our various scripts, see [2] for a start at\ndoing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\nthat for now, it can be added later.\n\nAs of writing this we've got major regressions between master..seen,\ni.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n'ah/plugleaks', 2021-08-04) have regressed recently.\n\nSee the discussion at <87czsv2idy.fsf@evledraar.gmail.com>[3] about\nthe lack of this sort of test mode, and 0e5bba53af (add UNLEAK\nannotation for reducing leak false positives, 2017-09-08) for the\ninitial addition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nThe reason for using gcc on OSX over the clang default is because when\nused with clang on \"macos-latest\" it'll currently fail to build with:\n\n clang: error: unsupported option '-fsanitize=leak' for target 'x86_64-apple-darwin19.6.0'\n\nIf that's sorted out in the future we might want to run that job with\n\"clang\" merely to make use of the default, and also to add some\ncompiler variance into the mix. Both use the\n\"AddressSanitizerLeakSanitizer\" library[4], so in they shouldn't\nbehave differently under GCC or clang.\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 6 ++++++\n ci/install-dependencies.sh | 6 +++---\n ci/lib.sh | 15 +++++++++++----\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/t0004-unwritable.sh | 1 +\n t/test-lib.sh | 20 ++++++++++++++++++++\n 8 files changed, 50 insertions(+), 8 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 68596f25927..a2d345fb00e 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -232,6 +232,12 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n+ - jobname: osx-leaks\n+ cc: gcc\n+ pool: macos-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 5772081b6e5..bb88afd3699 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,13 +12,13 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-leaks)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n@@ -37,7 +37,7 @@ linux-clang|linux-gcc)\n \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n \tpopd\n \t;;\n-osx-clang|osx-gcc)\n+osx-clang|osx-gcc|osx-leaks)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 33b9777ab7e..043c99d31cb 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,9 +183,9 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-leaks)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n \t\t;;\n@@ -208,9 +208,9 @@ linux-clang|linux-gcc)\n \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n-osx-clang|osx-gcc)\n+osx-clang|osx-gcc|osx-leaks)\n \tcase \"$jobname\" in\n-\tosx-gcc)\n+\tosx-gcc|osx-leaks)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n \t\t;;\n@@ -237,4 +237,11 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-leaks|osx-leaks)\n+\texport SANITIZE=leak\n+\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee94..23d2fa5565a 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -12,7 +12,7 @@ esac\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc|linux-leaks)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\ndiff --git a/t/README b/t/README\nindex 9e701223020..8b5f86a46f3 100644\n--- a/t/README\n+++ b/t/README\n@@ -366,6 +366,13 @@ excluded as so much relies on it, but this might change in the future.\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n test suite. Accept any boolean values that are accepted by git-config.\n \n+GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n+SANITIZE=leak will run only those tests that have whitelisted\n+themselves as passing with no memory leaks. Tests can be whitelisted\n+by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n+\"test-lib.sh\" itself at the top of the test script. This test mode is\n+used by the \"linux-leaks\" CI target.\n+\n GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n default to n.\n \ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex cb87768513c..54318af3861 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -18,6 +18,7 @@ swapping compression and hashing order, the person who is making the\n modification *should* take notice and update the test vectors here.\n '\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n try_local_xy () {\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex fbdcb926b3a..37d68ef03be 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -2,6 +2,7 @@\n \n test_description='detect unwritable repository and fail correctly'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 4ab18914a3d..3b7acfec23b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1379,6 +1379,26 @@ then\n \ttest_done\n fi\n \n+# skip non-whitelisted tests when compiled with SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+\tthen\n+\t\t# We need to see it in \"git env--helper\" (via\n+\t\t# test_bool_env)\n+\t\texport TEST_PASSES_SANITIZE_LEAK\n+\n+\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n+\t\tthen\n+\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n+\t\t\ttest_done\n+\t\tfi\n+\tfi\n+elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n HOME=\"$TRASH_DIRECTORY\"\n GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n-- \n2.33.0.819.g59feb45f5e0\n\n"},{"id":"435018","messageId":"CAPig+cTHFpE-GZvCpwVAmkXkVybHsU=CB_UWrU7TwYcW3KLuaQ@mail.gmail.com","threadId":"55888","inReplyTo":"patch-v5-3.3-f3cd04b16d1-20210907T212626Z-avarab@gmail.com","subject":"Re: [PATCH v5 3/3] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-09-08T04:46:54Z","receivedAt":"2021-09-08T04:47:07Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Sep 7, 2021 at 5:30 PM Ævar Arnfjörð Bjarmason wrote:\n> [...]\n> A test tests setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\n> turn make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\n> selectively skip tests even under\n> \"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In a preceding commit we\n> started doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\n> it'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nIs the wording \"A test tests setting ... setting\" intentional?\n"},{"id":"435069","messageId":"xmqq35qf72jp.fsf@gitster.g","threadId":"55888","inReplyTo":"cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com","subject":"Re: [PATCH v5 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-08T11:02:50Z","receivedAt":"2021-09-08T11:02:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> We can compile git with SANITIZE=leak, and have had various efforts in\n> the past such as 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04)\n> to plug memory leaks, but have had no CI testing of it to ensure that\n> we don't get regressions. This series adds a GIT_TEST_* mode for\n> checking those regressions, and runs it in CI.\n>\n> Since I submitted v2 the delta between origin/master..origin/seen\n> broke even t0001-init.sh when run under SANITIZE=leak, so this series\n> will cause test smoke on \"seen\".\n>\n> That failure is due to a bug in es/config-based-hooks [1] and the\n> hn/reftable topic, i.e. these patches are legitimately catching\n> regressions in \"seen\" from day 1.\n\nSo is there a point in sending this out to the list, before sending\nfixes to these broken topic and making sure they get corrected?\n\nBecause the CI does not \"bisect\" to tell us \"ok, up to this point in\n'seen', all the topics merged play well together\", the overall\neffect in the bigger picture is that 'seen' with this series would\ncause CI to stay in failed state.\n\nFor now, I'll keep this near the tip of 'seen'.\n\nThanks.\n\n"},{"id":"435079","messageId":"87sfyfgtfh.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"xmqq35qf72jp.fsf@gitster.g","subject":"Re: [PATCH v5 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T12:03:30Z","receivedAt":"2021-09-08T12:09:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> We can compile git with SANITIZE=leak, and have had various efforts in\n>> the past such as 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04)\n>> to plug memory leaks, but have had no CI testing of it to ensure that\n>> we don't get regressions. This series adds a GIT_TEST_* mode for\n>> checking those regressions, and runs it in CI.\n>>\n>> Since I submitted v2 the delta between origin/master..origin/seen\n>> broke even t0001-init.sh when run under SANITIZE=leak, so this series\n>> will cause test smoke on \"seen\".\n>>\n>> That failure is due to a bug in es/config-based-hooks [1] and the\n>> hn/reftable topic, i.e. these patches are legitimately catching\n>> regressions in \"seen\" from day 1.\n>\n> So is there a point in sending this out to the list, before sending\n> fixes to these broken topic and making sure they get corrected?\n>\n> Because the CI does not \"bisect\" to tell us \"ok, up to this point in\n> 'seen', all the topics merged play well together\", the overall\n> effect in the bigger picture is that 'seen' with this series would\n> cause CI to stay in failed state.\n>\n> For now, I'll keep this near the tip of 'seen'.\n\nThe breakages with it are in combination with:\n\n ab/config-based-hooks-base\n es/config-based-hooks\n hn/reftable\n\nYou've got v4 of ab/config-based-hooks-base, the v5 is at [1], but we've\nbeen waiting on emily to re-roll hers on top. As noted in that E-Mail\nI've got a working re-roll of it as\navar-nasamuffin/config-based-hooks-restart-3 in my repo.\n\nThat'll leave hn/reftable, which given [2] I thought you were planning\nto eject, and wiht the number of fixups for it / the planned re-doing of\nit by Han-Wen[3] maybe it's better to do that now?\n\nWhat do you think about that plan?\n\nI.e. ejecting hn/reftable while waiting on a re-roll, and either\nejecting es/config-based-hooks while waiting, or I can submit the\navar-nasamuffin/config-based-hooks-restart-3 I've got pending Emily's\nown re-roll (which may or may not be different from that).\n\nThat along with picking up the v5 of my ab/config-based-hooks-base\nshould make \"seen\" pass with SANITIZE=leak on these tests, unless\nthere's other just-introduced regressions. I tried re-building it a few\ndays ago, I haven't done that just now.\n\n1. https://lore.kernel.org/git/cover-v5-00.36-00000000000-20210902T125110Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/xmqq4kaxe5dt.fsf@gitster.g/\n3. https://lore.kernel.org/git/CAFQ2z_N8pUsp3cdBpybHBD-V9_1sARCZvSxr0UkMfcwCoQfCbw@mail.gmail.com/\n"},{"id":"435343","messageId":"YTqUgOcNYGl8Nljp@google.com","threadId":"55888","inReplyTo":"87sfyfgtfh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 0/3] add a test mode for SANITIZE=leak, run it in CI","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-09-09T23:10:56Z","receivedAt":"2021-09-09T23:11:05Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Sep 08, 2021 at 02:03:30PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> \n> On Wed, Sep 08 2021, Junio C Hamano wrote:\n> \n> > Ævar Arnfjörð Bjarmason writes:\n> >\n> >> We can compile git with SANITIZE=leak, and have had various efforts in\n> >> the past such as 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04)\n> >> to plug memory leaks, but have had no CI testing of it to ensure that\n> >> we don't get regressions. This series adds a GIT_TEST_* mode for\n> >> checking those regressions, and runs it in CI.\n> >>\n> >> Since I submitted v2 the delta between origin/master..origin/seen\n> >> broke even t0001-init.sh when run under SANITIZE=leak, so this series\n> >> will cause test smoke on \"seen\".\n> >>\n> >> That failure is due to a bug in es/config-based-hooks [1] and the\n> >> hn/reftable topic, i.e. these patches are legitimately catching\n> >> regressions in \"seen\" from day 1.\n> >\n> > So is there a point in sending this out to the list, before sending\n> > fixes to these broken topic and making sure they get corrected?\n> >\n> > Because the CI does not \"bisect\" to tell us \"ok, up to this point in\n> > 'seen', all the topics merged play well together\", the overall\n> > effect in the bigger picture is that 'seen' with this series would\n> > cause CI to stay in failed state.\n> >\n> > For now, I'll keep this near the tip of 'seen'.\n> \n> The breakages with it are in combination with:\n> \n> ab/config-based-hooks-base\n> es/config-based-hooks\n> hn/reftable\n> \n> You've got v4 of ab/config-based-hooks-base, the v5 is at [1], but we've\n> been waiting on emily to re-roll hers on top. As noted in that E-Mail\n> I've got a working re-roll of it as\n> avar-nasamuffin/config-based-hooks-restart-3 in my repo.\n> \n> That'll leave hn/reftable, which given [2] I thought you were planning\n> to eject, and wiht the number of fixups for it / the planned re-doing of\n> it by Han-Wen[3] maybe it's better to do that now?\n> \n> What do you think about that plan?\n> \n> I.e. ejecting hn/reftable while waiting on a re-roll, and either\n> ejecting es/config-based-hooks while waiting, or I can submit the\n> avar-nasamuffin/config-based-hooks-restart-3 I've got pending Emily's\n> own re-roll (which may or may not be different from that).\n\nMy own reroll is waiting on some feedback internally and probably won't\nshow up this week at all, so I suggest to kick mine out and prioritize\nthe reftable stuff for now.\n\n - Emily\n\n> \n> That along with picking up the v5 of my ab/config-based-hooks-base\n> should make \"seen\" pass with SANITIZE=leak on these tests, unless\n> there's other just-introduced regressions. I tried re-building it a few\n> days ago, I haven't done that just now.\n> \n> 1. https://lore.kernel.org/git/cover-v5-00.36-00000000000-20210902T125110Z-avarab@gmail.com/\n> 2. https://lore.kernel.org/git/xmqq4kaxe5dt.fsf@gitster.g/\n> 3. https://lore.kernel.org/git/CAFQ2z_N8pUsp3cdBpybHBD-V9_1sARCZvSxr0UkMfcwCoQfCbw@mail.gmail.com/\n"},{"id":"436064","messageId":"20210916035603.76369-1-carenas@gmail.com","threadId":"55888","inReplyTo":"patch-v5-3.3-f3cd04b16d1-20210907T212626Z-avarab@gmail.com","subject":"[PATCH] fixup! tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-16T03:56:03Z","receivedAt":"2021-09-16T03:56:34Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"Use the standard gcc in Linux, instead of the older version\n\nRemove the osx-leaks job; neither clang or gcc support it and won't\nuntil clang 14 is released.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n .github/workflows/main.yml | 3 ---\n ci/install-dependencies.sh | 4 ++--\n ci/lib.sh | 8 ++++----\n 3 files changed, 6 insertions(+), 9 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 7c273147a0..59acc35d37 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -234,9 +234,6 @@ jobs:\n - jobname: linux-leaks\n cc: gcc\n pool: ubuntu-latest\n- - jobname: osx-leaks\n- cc: gcc\n- pool: macos-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex bb88afd369..1d0e48f451 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -18,7 +18,7 @@ linux-clang|linux-gcc|linux-leaks)\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS\n \tcase \"$jobname\" in\n-\tlinux-gcc|linux-leaks)\n+\tlinux-gcc)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\n@@ -37,7 +37,7 @@ linux-clang|linux-gcc|linux-leaks)\n \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n \tpopd\n \t;;\n-osx-clang|osx-gcc|osx-leaks)\n+osx-clang|osx-gcc)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex cf62f786a3..36f594751d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -185,7 +185,7 @@ export SKIP_DASHED_BUILT_INS=YesPlease\n case \"$jobname\" in\n linux-clang|linux-gcc|linux-leaks)\n \tcase \"$jobname\" in\n-\tlinux-gcc|linux-leaks)\n+\tlinux-gcc)\n \t\texport CC=gcc-8\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n \t\t;;\n@@ -208,9 +208,9 @@ linux-clang|linux-gcc|linux-leaks)\n \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n-osx-clang|osx-gcc|osx-leaks)\n+osx-clang|osx-gcc)\n \tcase \"$jobname\" in\n-\tosx-gcc|osx-leaks)\n+\tosx-gcc)\n \t\texport CC=gcc-9\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n \t\t;;\n@@ -239,7 +239,7 @@ linux-musl)\n esac\n \n case \"$jobname\" in\n-linux-leaks|osx-leaks)\n+*-leaks)\n \texport SANITIZE=leak\n \texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n \t;;\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"436088","messageId":"87mtodgddu.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"20210916035603.76369-1-carenas@gmail.com","subject":"Re: [PATCH] fixup! tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-16T06:14:25Z","receivedAt":"2021-09-16T07:58:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 15 2021, Carlo Marcelo Arenas Belón wrote:\n\n> Use the standard gcc in Linux, instead of the older version\n>\n> Remove the osx-leaks job; neither clang or gcc support it and won't\n> until clang 14 is released.\n\nThanks, that's well spotted. I'll fix this in a re-roll. I just tested\nand the osx job does nothing.\n\nFWIW I'd tested and it errored on clang, but didn't check the gcc case\nof silently ignoring it.\n\n> Signed-off-by: Carlo Marcelo Arenas Belón \n> ---\n> .github/workflows/main.yml | 3 ---\n> ci/install-dependencies.sh | 4 ++--\n> ci/lib.sh | 8 ++++----\n> 3 files changed, 6 insertions(+), 9 deletions(-)\n>\n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index 7c273147a0..59acc35d37 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -234,9 +234,6 @@ jobs:\n> - jobname: linux-leaks\n> cc: gcc\n> pool: ubuntu-latest\n> - - jobname: osx-leaks\n> - cc: gcc\n> - pool: macos-latest\n> env:\n> CC: ${{matrix.vector.cc}}\n> jobname: ${{matrix.vector.jobname}}\n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index bb88afd369..1d0e48f451 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -18,7 +18,7 @@ linux-clang|linux-gcc|linux-leaks)\n> \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n> \t\t$UBUNTU_COMMON_PKGS\n> \tcase \"$jobname\" in\n> -\tlinux-gcc|linux-leaks)\n> +\tlinux-gcc)\n> \t\tsudo apt-get -q -y install gcc-8\n> \t\t;;\n> \tesac\n\n\n> @@ -37,7 +37,7 @@ linux-clang|linux-gcc|linux-leaks)\n> \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n> \tpopd\n> \t;;\n> -osx-clang|osx-gcc|osx-leaks)\n> +osx-clang|osx-gcc)\n> \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n> \t# Uncomment this if you want to run perf tests:\n> \t# brew install gnu-time\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index cf62f786a3..36f594751d 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -185,7 +185,7 @@ export SKIP_DASHED_BUILT_INS=YesPlease\n> case \"$jobname\" in\n> linux-clang|linux-gcc|linux-leaks)\n> \tcase \"$jobname\" in\n> -\tlinux-gcc|linux-leaks)\n> +\tlinux-gcc)\n> \t\texport CC=gcc-8\n> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n> \t\t;;\n> @@ -208,9 +208,9 @@ linux-clang|linux-gcc|linux-leaks)\n> \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n> \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n> \t;;\n> -osx-clang|osx-gcc|osx-leaks)\n> +osx-clang|osx-gcc)\n> \tcase \"$jobname\" in\n> -\tosx-gcc|osx-leaks)\n> +\tosx-gcc)\n> \t\texport CC=gcc-9\n> \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n> \t\t;;\n> @@ -239,7 +239,7 @@ linux-musl)\n> esac\n> \n> case \"$jobname\" in\n> -linux-leaks|osx-leaks)\n> +*-leaks)\n> \texport SANITIZE=leak\n> \texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n> \t;;\n\nI'll leave this stray cleanup out, yes it's functionally equivalent, but\nit really helps to be able to see an identifier in main.yml and grep for\nit across the untyped-language boundary of that YAML going into\nshellscript.\n"},{"id":"436107","messageId":"cover-v6-0.2-00000000000-20210916T085311Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com","subject":"[PATCH v6 0/2] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-16T10:48:51Z","receivedAt":"2021-09-16T10:49:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This v6 incorporates a suggested fixup from Carlo Marcelo Arenas\nBelón, we weren't running at all under OSX as it turns out :\nhttps://lore.kernel.org/git/20210916035603.76369-1-carenas@gmail.com/\n\nSo the osx-leaks job has been dropped, and we're not using an older\ncompiler anymore (I'd just copy/pasted that setting). Since we don't\nneed it, we can drop the 2nd patch of v5. For v5 see:\nhttps://lore.kernel.org/git/cover-v5-0.3-00000000000-20210907T212626Z-avarab@gmail.com/\n\nThis also incorporates a wording fix from Eric Sunshine.\n\nThe rest of this CL is just a message for Eric Sunshine, included here\nfor what should be in case he'll see this on-list:\n\nEric: If you're reading this I dropped you from CC because since\naround September 2nd mailer-daemon@googlemail.com has been failing to\ndeliver all mail to you. It appears the dynadot.com MTA you use has\nbanned delivery from GMail's public IP's due to spam complaints:\n\n The recipient server did not accept our requests to connect. Learn\n more at https://support.google.com/mail/answer/7720\n [parkmail.dynadot.com. 68.68.98.83: 421 parkmail.dynadot.com your ip\n address has been banned due to spam complaints 209.85.167.53 ]\n [parkmail.dynadot.com. 68.68.98.74: 421 parkmail.dynadot.com your ip\n address has been banned due to spam complaints 209.85.167.48 ]\n [parkmail.dynadot.com. 68.68.98.84: 421 parkmail.dynadot.com your ip\n address has been banned due to spam complaints 209.85.167.48 ]\n\nÆvar Arnfjörð Bjarmason (2):\n Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n tests: add a test mode for SANITIZE=leak, run it in CI\n\n .github/workflows/main.yml | 3 +++\n Makefile | 5 +++++\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 9 ++++++++-\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/t0004-unwritable.sh | 3 ++-\n t/test-lib.sh | 21 +++++++++++++++++++++\n 9 files changed, 49 insertions(+), 4 deletions(-)\n\nRange-diff against v5:\n1: bdfe2279271 = 1: fc7ba4cb1c3 Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n2: 6aaa60e3759 < -: ----------- CI: refactor \"if\" to \"case\" statement\n3: f3cd04b16d1 ! 2: 8dcb1269881 tests: add a test mode for SANITIZE=leak, run it in CI\n @@ Commit message\n one-offs without structured regression testing.\n \n This change adds CI testing for it. We'll now build and test\n - t000[04]*.sh under both Linux and OSX. The new jobs are called\n - \"linux-leaks\" and \"osx-leaks\".\n + t000[04]*.sh under Linux with a new job called \"linux-leaks\".\n \n The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n mode. When running in that mode, we'll assert that we were compiled\n with SANITIZE=leak. We'll then skip all tests, except those that we've\n opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n \n - A test tests setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in\n - turn make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\n + A test setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in turn\n + make use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\n selectively skip tests even under\n - \"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In a preceding commit we\n + \"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In the preceding commit we\n started doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\n it'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n \n @@ Commit message\n 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\n past history of \"one-off\" SANITIZE=leak (and more) fixes.\n \n - The reason for using gcc on OSX over the clang default is because when\n - used with clang on \"macos-latest\" it'll currently fail to build with:\n -\n - clang: error: unsupported option '-fsanitize=leak' for target 'x86_64-apple-darwin19.6.0'\n -\n - If that's sorted out in the future we might want to run that job with\n - \"clang\" merely to make use of the default, and also to add some\n - compiler variance into the mix. Both use the\n - \"AddressSanitizerLeakSanitizer\" library[4], so in they shouldn't\n - behave differently under GCC or clang.\n + As noted in [5] we can't support this on OSX yet until Clang 14 is\n + released, at that point we'll probably want to resurrect that\n + \"osx-leaks\" job.\n \n 1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n 2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n 3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n 4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n + 5. https://lore.kernel.org/git/20210916035603.76369-1-carenas@gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n + Signed-off-by: Carlo Marcelo Arenas Belón \n \n ## .github/workflows/main.yml ##\n @@ .github/workflows/main.yml: jobs:\n @@ .github/workflows/main.yml: jobs:\n + - jobname: linux-leaks\n + cc: gcc\n + pool: ubuntu-latest\n -+ - jobname: osx-leaks\n -+ cc: gcc\n -+ pool: macos-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\n @@ ci/install-dependencies.sh: UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n - \t\t$UBUNTU_COMMON_PKGS\n - \tcase \"$jobname\" in\n --\tlinux-gcc)\n -+\tlinux-gcc|linux-leaks)\n - \t\tsudo apt-get -q -y install gcc-8\n - \t\t;;\n - \tesac\n -@@ ci/install-dependencies.sh: linux-clang|linux-gcc)\n - \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n - \tpopd\n - \t;;\n --osx-clang|osx-gcc)\n -+osx-clang|osx-gcc|osx-leaks)\n - \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n - \t# Uncomment this if you want to run perf tests:\n - \t# brew install gnu-time\n \n ## ci/lib.sh ##\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n @@ ci/lib.sh: export GIT_TEST_CLONE_2GB=true\n case \"$jobname\" in\n -linux-clang|linux-gcc)\n +linux-clang|linux-gcc|linux-leaks)\n - \tcase \"$jobname\" in\n --\tlinux-gcc)\n -+\tlinux-gcc|linux-leaks)\n + \tif [ \"$jobname\" = linux-gcc ]\n + \tthen\n \t\texport CC=gcc-8\n - \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python3\"\n - \t\t;;\n -@@ ci/lib.sh: linux-clang|linux-gcc)\n - \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n - \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n - \t;;\n --osx-clang|osx-gcc)\n -+osx-clang|osx-gcc|osx-leaks)\n - \tcase \"$jobname\" in\n --\tosx-gcc)\n -+\tosx-gcc|osx-leaks)\n - \t\texport CC=gcc-9\n - \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n - \t\t;;\n @@ ci/lib.sh: linux-musl)\n \t;;\n esac\n \n +case \"$jobname\" in\n -+linux-leaks|osx-leaks)\n ++linux-leaks)\n +\texport SANITIZE=leak\n +\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n +\t;;\n @@ ci/lib.sh: linux-musl)\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n ## ci/run-build-and-tests.sh ##\n -@@ ci/run-build-and-tests.sh: esac\n +@@ ci/run-build-and-tests.sh: fi\n \n make\n case \"$jobname\" in\n @@ t/test-lib.sh: then\n +fi\n +\n # Last-minute variable setup\n + USER_HOME=\"$HOME\"\n HOME=\"$TRASH_DIRECTORY\"\n - GNUPGHOME=\"$HOME/gnupg-home-not-used\"\n-- \n2.33.0.1056.gb2c8c79e36d\n\n"},{"id":"436108","messageId":"patch-v6-1.2-fc7ba4cb1c3-20210916T085311Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v6-0.2-00000000000-20210916T085311Z-avarab@gmail.com","subject":"[PATCH v6 1/2] Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-16T10:48:52Z","receivedAt":"2021-09-16T10:49:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When SANITIZE=leak is specified we'll now add a SANITIZE_LEAK flag to\nGIT-BUILD-OPTIONS, this can then be picked up by the test-lib.sh,\nwhich sets a SANITIZE_LEAK prerequisite.\n\nWe can then skip specific tests that are known to fail under\nSANITIZE=leak, add one such annotation to t0004-unwritable.sh, which\nnow passes under SANITIZE=leak.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++++\n t/t0004-unwritable.sh | 2 +-\n t/test-lib.sh | 1 +\n 3 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex b90af71a7a2..b4ad91743b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1224,6 +1224,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1268,6 +1271,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2815,6 +2819,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex e3137d638ee..fbdcb926b3a 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -21,7 +21,7 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi\n \ttest_must_fail git write-tree\n '\n \n-test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository' '\n+test_expect_success POSIXPERM,SANITY,!SANITIZE_LEAK 'commit should notice unwritable repository' '\n \ttest_when_finished \"chmod 775 .git/objects .git/objects/??\" &&\n \tchmod a-w .git/objects .git/objects/?? &&\n \ttest_must_fail git commit -m second\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d5ee9642548..06831086060 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1536,6 +1536,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.33.0.1056.gb2c8c79e36d\n\n"},{"id":"436109","messageId":"patch-v6-2.2-8dcb1269881-20210916T085312Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v6-0.2-00000000000-20210916T085311Z-avarab@gmail.com","subject":"[PATCH v6 2/2] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-16T10:48:53Z","receivedAt":"2021-09-16T10:49:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak, we have not run\nregression tests under that mode. Memory leaks have only been fixed as\none-offs without structured regression testing.\n\nThis change adds CI testing for it. We'll now build and test\nt000[04]*.sh under Linux with a new job called \"linux-leaks\".\n\nThe CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\nmode. When running in that mode, we'll assert that we were compiled\nwith SANITIZE=leak. We'll then skip all tests, except those that we've\nopted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n\nA test setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in turn\nmake use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\nselectively skip tests even under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In the preceding commit we\nstarted doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\nit'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nThis is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\nbe skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n\nThe intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\nas follow-up change, but let's start small to begin with.\n\nIt would also be possible to implement a more lightweight version of\nthis by only relying on setting \"LSAN_OPTIONS\". See\n[1] and\n[2] for a discussion of\nthat. I've opted for this approach of adding a GIT_TEST_* mode instead\nbecause it's consistent with how we handle other special test modes.\n\nBeing able to add a \"!SANITIZE_LEAK\" prerequisite and calling\n\"test_done\" early if it isn't satisfied also means that we can more\nincrementally add regression tests without being forced to fix\nwidespread and hard-to-fix leaks at the same time.\n\nWe have tests that do simple checking of some tool we're interested\nin, but later on in the script might be stressing trace2, or common\nsources of leaks like \"git log\" in combination with the tool (e.g. the\ncommit-graph tests). To be clear having a prerequisite could also be\naccomplished by using \"LSAN_OPTIONS\" directly.\n\nOn the topic of \"LSAN_OPTIONS\": It would be nice to have a mode to\naggregate all failures in our various scripts, see [2] for a start at\ndoing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\nthat for now, it can be added later.\n\nAs of writing this we've got major regressions between master..seen,\ni.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n'ah/plugleaks', 2021-08-04) have regressed recently.\n\nSee the discussion at <87czsv2idy.fsf@evledraar.gmail.com>[3] about\nthe lack of this sort of test mode, and 0e5bba53af (add UNLEAK\nannotation for reducing leak false positives, 2017-09-08) for the\ninitial addition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nAs noted in [5] we can't support this on OSX yet until Clang 14 is\nreleased, at that point we'll probably want to resurrect that\n\"osx-leaks\" job.\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n5. https://lore.kernel.org/git/20210916035603.76369-1-carenas@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n .github/workflows/main.yml | 3 +++\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 9 ++++++++-\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0000-basic.sh | 1 +\n t/t0004-unwritable.sh | 1 +\n t/test-lib.sh | 20 ++++++++++++++++++++\n 8 files changed, 42 insertions(+), 3 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex b053b01c66e..47281684782 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -232,6 +232,9 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 5772081b6e5..1d0e48f4515 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,7 +12,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..82cb17f8eea 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,7 +183,7 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tif [ \"$jobname\" = linux-gcc ]\n \tthen\n \t\texport CC=gcc-8\n@@ -233,4 +233,11 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-leaks)\n+\texport SANITIZE=leak\n+\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex f3aba5d6cbb..ba29a93d84b 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -17,7 +17,7 @@ fi\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc|linux-leaks)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\ndiff --git a/t/README b/t/README\nindex e924bd81e2d..ab84278b7eb 100644\n--- a/t/README\n+++ b/t/README\n@@ -366,6 +366,13 @@ excluded as so much relies on it, but this might change in the future.\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n test suite. Accept any boolean values that are accepted by git-config.\n \n+GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n+SANITIZE=leak will run only those tests that have whitelisted\n+themselves as passing with no memory leaks. Tests can be whitelisted\n+by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n+\"test-lib.sh\" itself at the top of the test script. This test mode is\n+used by the \"linux-leaks\" CI target.\n+\n GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n default to n.\n \ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex cb87768513c..54318af3861 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -18,6 +18,7 @@ swapping compression and hashing order, the person who is making the\n modification *should* take notice and update the test vectors here.\n '\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n try_local_xy () {\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex fbdcb926b3a..37d68ef03be 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -2,6 +2,7 @@\n \n test_description='detect unwritable repository and fail correctly'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 06831086060..9310d9d900a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1381,6 +1381,26 @@ then\n \ttest_done\n fi\n \n+# skip non-whitelisted tests when compiled with SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+\tthen\n+\t\t# We need to see it in \"git env--helper\" (via\n+\t\t# test_bool_env)\n+\t\texport TEST_PASSES_SANITIZE_LEAK\n+\n+\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n+\t\tthen\n+\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n+\t\t\ttest_done\n+\t\tfi\n+\tfi\n+elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n USER_HOME=\"$HOME\"\n HOME=\"$TRASH_DIRECTORY\"\n-- \n2.33.0.1056.gb2c8c79e36d\n\n"},{"id":"436355","messageId":"cover-v7-0.2-00000000000-20210919T075619Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v6-0.2-00000000000-20210916T085311Z-avarab@gmail.com","subject":"[PATCH v7 0/2] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-19T08:03:52Z","receivedAt":"2021-09-19T08:04:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series adds a small beachhead of tests we run in CI that we\nassert to be memory-leak free with the SANITIZE=leak test mode. Once\nit lands the intent is to expand the parts of the test suite we\nwhitelist as memory-leak free.\n\nThis v7 of the \"test with SANITIZE=leak in CI\" topic should be ready\nfor merging down. The v6 got marked as \"Will merge to 'next'?\", but as\nCarlo points out[1] there were concurrent regresisons in\nt0000-basic.sh that caused the tests to faile. There's proposed fixes\nto those[2] as well as Carlo's own series to fix other issues with\nit[3].\n\nAll of those are worth doing, but the reason I picked t0000-basic.sh\nwas that it would hopefully stay leak free through the\nseen->next->master cycle.\n\nLet's not pick that one, but instead a few of the very small and basic\ntests in t00*.sh.\n\nThese all run cleanly on top of master, and also when merged with next\nand seen (except for the semantic \"seen\" failure due to merging with\nv6 of this topic, and therefore t0000-basic.sh being run in the test\nmode).\n\nFor v6 of this topic see:\nhttps://lore.kernel.org/git/cover-v6-0.2-00000000000-20210916T085311Z-avarab@gmail.com\n\n1. https://lore.kernel.org/git/CAPUEsphMUNYRACmK-nksotP1RrMn09mNGFdEHLLuNEWH4AcU7Q@mail.gmail.com/\n2. https://lore.kernel.org/git/pull.1092.git.git.1631972978.gitgitgadget@gmail.com/\n3. https://lore.kernel.org/git/20210916023706.55760-1-carenas@gmail.com/\n\nÆvar Arnfjörð Bjarmason (2):\n Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n tests: add a test mode for SANITIZE=leak, run it in CI\n\n .github/workflows/main.yml | 3 +++\n Makefile | 5 +++++\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 9 ++++++++-\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0004-unwritable.sh | 3 ++-\n t/t0011-hashmap.sh | 2 ++\n t/t0016-oidmap.sh | 2 ++\n t/t0017-env-helper.sh | 1 +\n t/t0018-advice.sh | 1 +\n t/t0030-stripspace.sh | 1 +\n t/t0063-string-list.sh | 1 +\n t/t0091-bugreport.sh | 1 +\n t/test-lib.sh | 21 +++++++++++++++++++++\n 15 files changed, 57 insertions(+), 4 deletions(-)\n\nRange-diff against v6:\n1: fc7ba4cb1c3 = 1: fc7ba4cb1c3 Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n2: 8dcb1269881 ! 2: 56592952db5 tests: add a test mode for SANITIZE=leak, run it in CI\n @@ Commit message\n regression tests under that mode. Memory leaks have only been fixed as\n one-offs without structured regression testing.\n \n - This change adds CI testing for it. We'll now build and test\n - t000[04]*.sh under Linux with a new job called \"linux-leaks\".\n + This change adds CI testing for it. We'll now build and small set of\n + whitelisted t00*.sh tests under Linux with a new job called\n + \"linux-leaks\".\n \n The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n mode. When running in that mode, we'll assert that we were compiled\n @@ t/README: excluded as so much relies on it, but this might change in the future.\n default to n.\n \n \n - ## t/t0000-basic.sh ##\n -@@ t/t0000-basic.sh: swapping compression and hashing order, the person who is making the\n - modification *should* take notice and update the test vectors here.\n - '\n + ## t/t0004-unwritable.sh ##\n +@@\n + \n + test_description='detect unwritable repository and fail correctly'\n \n +TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n - try_local_xy () {\n + test_expect_success setup '\n \n - ## t/t0004-unwritable.sh ##\n + ## t/t0011-hashmap.sh ##\n @@\n + #!/bin/sh\n \n - test_description='detect unwritable repository and fail correctly'\n + test_description='test hashmap and string hash functions'\n ++\n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + test_hashmap() {\n +\n + ## t/t0016-oidmap.sh ##\n +@@\n + #!/bin/sh\n \n + test_description='test oidmap'\n ++\n +TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n - test_expect_success setup '\n + # This purposefully is very similar to t0011-hashmap.sh\n +\n + ## t/t0017-env-helper.sh ##\n +@@\n + \n + test_description='test env--helper'\n + \n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + \n +\n + ## t/t0018-advice.sh ##\n +@@\n + \n + test_description='Test advise_if_enabled functionality'\n + \n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + test_expect_success 'advice should be printed when config variable is unset' '\n +\n + ## t/t0030-stripspace.sh ##\n +@@\n + \n + test_description='git stripspace'\n + \n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + t40='A quick brown fox jumps over the lazy do'\n +\n + ## t/t0063-string-list.sh ##\n +@@\n + \n + test_description='Test string list functionality'\n + \n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + test_split () {\n +\n + ## t/t0091-bugreport.sh ##\n +@@\n + \n + test_description='git bugreport'\n + \n ++TEST_PASSES_SANITIZE_LEAK=true\n + . ./test-lib.sh\n + \n + # Headers \"[System Info]\" will be followed by a non-empty line if we put some\n \n ## t/test-lib.sh ##\n @@ t/test-lib.sh: then\n-- \n2.33.0.1092.g44c994ea1be\n\n"},{"id":"436354","messageId":"patch-v7-2.2-56592952db5-20210919T075619Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v7-0.2-00000000000-20210919T075619Z-avarab@gmail.com","subject":"[PATCH v7 2/2] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-19T08:03:54Z","receivedAt":"2021-09-19T08:04:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak, we have not run\nregression tests under that mode. Memory leaks have only been fixed as\none-offs without structured regression testing.\n\nThis change adds CI testing for it. We'll now build and small set of\nwhitelisted t00*.sh tests under Linux with a new job called\n\"linux-leaks\".\n\nThe CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\nmode. When running in that mode, we'll assert that we were compiled\nwith SANITIZE=leak. We'll then skip all tests, except those that we've\nopted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n\nA test setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in turn\nmake use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\nselectively skip tests even under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In the preceding commit we\nstarted doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\nit'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nThis is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\nbe skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n\nThe intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\nas follow-up change, but let's start small to begin with.\n\nIt would also be possible to implement a more lightweight version of\nthis by only relying on setting \"LSAN_OPTIONS\". See\n[1] and\n[2] for a discussion of\nthat. I've opted for this approach of adding a GIT_TEST_* mode instead\nbecause it's consistent with how we handle other special test modes.\n\nBeing able to add a \"!SANITIZE_LEAK\" prerequisite and calling\n\"test_done\" early if it isn't satisfied also means that we can more\nincrementally add regression tests without being forced to fix\nwidespread and hard-to-fix leaks at the same time.\n\nWe have tests that do simple checking of some tool we're interested\nin, but later on in the script might be stressing trace2, or common\nsources of leaks like \"git log\" in combination with the tool (e.g. the\ncommit-graph tests). To be clear having a prerequisite could also be\naccomplished by using \"LSAN_OPTIONS\" directly.\n\nOn the topic of \"LSAN_OPTIONS\": It would be nice to have a mode to\naggregate all failures in our various scripts, see [2] for a start at\ndoing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\nthat for now, it can be added later.\n\nAs of writing this we've got major regressions between master..seen,\ni.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n'ah/plugleaks', 2021-08-04) have regressed recently.\n\nSee the discussion at <87czsv2idy.fsf@evledraar.gmail.com>[3] about\nthe lack of this sort of test mode, and 0e5bba53af (add UNLEAK\nannotation for reducing leak false positives, 2017-09-08) for the\ninitial addition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nAs noted in [5] we can't support this on OSX yet until Clang 14 is\nreleased, at that point we'll probably want to resurrect that\n\"osx-leaks\" job.\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n5. https://lore.kernel.org/git/20210916035603.76369-1-carenas@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n .github/workflows/main.yml | 3 +++\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 9 ++++++++-\n ci/run-build-and-tests.sh | 2 +-\n t/README | 7 +++++++\n t/t0004-unwritable.sh | 1 +\n t/t0011-hashmap.sh | 2 ++\n t/t0016-oidmap.sh | 2 ++\n t/t0017-env-helper.sh | 1 +\n t/t0018-advice.sh | 1 +\n t/t0030-stripspace.sh | 1 +\n t/t0063-string-list.sh | 1 +\n t/t0091-bugreport.sh | 1 +\n t/test-lib.sh | 20 ++++++++++++++++++++\n 14 files changed, 50 insertions(+), 3 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex b053b01c66e..47281684782 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -232,6 +232,9 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 5772081b6e5..1d0e48f4515 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,7 +12,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..82cb17f8eea 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,7 +183,7 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tif [ \"$jobname\" = linux-gcc ]\n \tthen\n \t\texport CC=gcc-8\n@@ -233,4 +233,11 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-leaks)\n+\texport SANITIZE=leak\n+\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex f3aba5d6cbb..ba29a93d84b 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -17,7 +17,7 @@ fi\n \n make\n case \"$jobname\" in\n-linux-gcc)\n+linux-gcc|linux-leaks)\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n \tmake test\n \texport GIT_TEST_SPLIT_INDEX=yes\ndiff --git a/t/README b/t/README\nindex e924bd81e2d..ab84278b7eb 100644\n--- a/t/README\n+++ b/t/README\n@@ -366,6 +366,13 @@ excluded as so much relies on it, but this might change in the future.\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n test suite. Accept any boolean values that are accepted by git-config.\n \n+GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n+SANITIZE=leak will run only those tests that have whitelisted\n+themselves as passing with no memory leaks. Tests can be whitelisted\n+by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n+\"test-lib.sh\" itself at the top of the test script. This test mode is\n+used by the \"linux-leaks\" CI target.\n+\n GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n default to n.\n \ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex fbdcb926b3a..37d68ef03be 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -2,6 +2,7 @@\n \n test_description='detect unwritable repository and fail correctly'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t0011-hashmap.sh b/t/t0011-hashmap.sh\nindex 5343ffd3f92..e094975b13b 100755\n--- a/t/t0011-hashmap.sh\n+++ b/t/t0011-hashmap.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test hashmap and string hash functions'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_hashmap() {\ndiff --git a/t/t0016-oidmap.sh b/t/t0016-oidmap.sh\nindex 31f8276ba82..0faef1f4f11 100755\n--- a/t/t0016-oidmap.sh\n+++ b/t/t0016-oidmap.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test oidmap'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # This purposefully is very similar to t0011-hashmap.sh\ndiff --git a/t/t0017-env-helper.sh b/t/t0017-env-helper.sh\nindex 4a159f99e44..2e42fba9567 100755\n--- a/t/t0017-env-helper.sh\n+++ b/t/t0017-env-helper.sh\n@@ -2,6 +2,7 @@\n \n test_description='test env--helper'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n \ndiff --git a/t/t0018-advice.sh b/t/t0018-advice.sh\nindex 39e5e4b34f8..c13057a4ca3 100755\n--- a/t/t0018-advice.sh\n+++ b/t/t0018-advice.sh\n@@ -2,6 +2,7 @@\n \n test_description='Test advise_if_enabled functionality'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'advice should be printed when config variable is unset' '\ndiff --git a/t/t0030-stripspace.sh b/t/t0030-stripspace.sh\nindex 0c24a0f9a37..ae1ca380c1a 100755\n--- a/t/t0030-stripspace.sh\n+++ b/t/t0030-stripspace.sh\n@@ -5,6 +5,7 @@\n \n test_description='git stripspace'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n t40='A quick brown fox jumps over the lazy do'\ndiff --git a/t/t0063-string-list.sh b/t/t0063-string-list.sh\nindex c6ee9f66b11..46d4839194b 100755\n--- a/t/t0063-string-list.sh\n+++ b/t/t0063-string-list.sh\n@@ -5,6 +5,7 @@\n \n test_description='Test string list functionality'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_split () {\ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 526304ff95b..eeedbfa9193 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -2,6 +2,7 @@\n \n test_description='git bugreport'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # Headers \"[System Info]\" will be followed by a non-empty line if we put some\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 06831086060..9310d9d900a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1381,6 +1381,26 @@ then\n \ttest_done\n fi\n \n+# skip non-whitelisted tests when compiled with SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+\tthen\n+\t\t# We need to see it in \"git env--helper\" (via\n+\t\t# test_bool_env)\n+\t\texport TEST_PASSES_SANITIZE_LEAK\n+\n+\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n+\t\tthen\n+\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n+\t\t\ttest_done\n+\t\tfi\n+\tfi\n+elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n USER_HOME=\"$HOME\"\n HOME=\"$TRASH_DIRECTORY\"\n-- \n2.33.0.1092.g44c994ea1be\n\n"},{"id":"436356","messageId":"patch-v7-1.2-fc7ba4cb1c3-20210919T075619Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v7-0.2-00000000000-20210919T075619Z-avarab@gmail.com","subject":"[PATCH v7 1/2] Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-19T08:03:53Z","receivedAt":"2021-09-19T08:04:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When SANITIZE=leak is specified we'll now add a SANITIZE_LEAK flag to\nGIT-BUILD-OPTIONS, this can then be picked up by the test-lib.sh,\nwhich sets a SANITIZE_LEAK prerequisite.\n\nWe can then skip specific tests that are known to fail under\nSANITIZE=leak, add one such annotation to t0004-unwritable.sh, which\nnow passes under SANITIZE=leak.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++++\n t/t0004-unwritable.sh | 2 +-\n t/test-lib.sh | 1 +\n 3 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex b90af71a7a2..b4ad91743b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1224,6 +1224,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1268,6 +1271,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2815,6 +2819,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex e3137d638ee..fbdcb926b3a 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -21,7 +21,7 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi\n \ttest_must_fail git write-tree\n '\n \n-test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository' '\n+test_expect_success POSIXPERM,SANITY,!SANITIZE_LEAK 'commit should notice unwritable repository' '\n \ttest_when_finished \"chmod 775 .git/objects .git/objects/??\" &&\n \tchmod a-w .git/objects .git/objects/?? &&\n \ttest_must_fail git commit -m second\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d5ee9642548..06831086060 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1536,6 +1536,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.33.0.1092.g44c994ea1be\n\n"},{"id":"436714","messageId":"20210922111741.82142-1-carenas@gmail.com","threadId":"55888","inReplyTo":"patch-v7-2.2-56592952db5-20210919T075619Z-avarab@gmail.com","subject":"[PATCH] fixup! tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-22T11:17:41Z","receivedAt":"2021-09-22T11:17:55Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"runs cleanly in seen as shown by :\n\n https://github.com/carenas/git/runs/3673976105\n\npreviously failing in the extended checks as shown at at least by :\n\n https://github.com/git/git/runs/3657308323\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n t/t0016-oidmap.sh | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/t/t0016-oidmap.sh b/t/t0016-oidmap.sh\nindex 0faef1f4f1..f81aa9ea03 100755\n--- a/t/t0016-oidmap.sh\n+++ b/t/t0016-oidmap.sh\n@@ -2,7 +2,6 @@\n \n test_description='test oidmap'\n \n-TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # This purposefully is very similar to t0011-hashmap.sh\n-- \n2.33.0.911.gbe391d4e11\n\n"},{"id":"436816","messageId":"87h7ec59m7.fsf@evledraar.gmail.com","threadId":"55888","inReplyTo":"20210922111741.82142-1-carenas@gmail.com","subject":"Re: [PATCH] fixup! tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-23T01:50:19Z","receivedAt":"2021-09-23T02:06:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 22 2021, Carlo Marcelo Arenas Belón wrote:\n\n> runs cleanly in seen as shown by :\n>\n> https://github.com/carenas/git/runs/3673976105\n>\n> previously failing in the extended checks as shown at at least by :\n>\n> https://github.com/git/git/runs/3657308323\n\nThanks, it broke because it combined with sg/test-split-index-fix,\nrunning the test with GIT_TEST_SPLIT_INDEX=true reveals a memory leak\nthat we weren't testing until then.\n\nJunio: I think just applying this fixup is the right thing for now, are\nyou willing to do that or should I submit a re-roll with it?\n\n> Signed-off-by: Carlo Marcelo Arenas Belón \n> ---\n> t/t0016-oidmap.sh | 1 -\n> 1 file changed, 1 deletion(-)\n>\n> diff --git a/t/t0016-oidmap.sh b/t/t0016-oidmap.sh\n> index 0faef1f4f1..f81aa9ea03 100755\n> --- a/t/t0016-oidmap.sh\n> +++ b/t/t0016-oidmap.sh\n> @@ -2,7 +2,6 @@\n> \n> test_description='test oidmap'\n> \n> -TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> \n> # This purposefully is very similar to t0011-hashmap.sh\n\n"},{"id":"436838","messageId":"cover-v8-0.2-00000000000-20210923T091819Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v7-0.2-00000000000-20210919T075619Z-avarab@gmail.com","subject":"[PATCH v8 0/2] add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-23T09:20:44Z","receivedAt":"2021-09-23T09:20:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series adds a small beachhead of tests we run in CI that we\nassert to be memory-leak free with the SANITIZE=leak test mode. Once\nit lands the intent is to expand the parts of the test suite we\nwhitelist as memory-leak free.\n\nFor the v7 see:\nhttps://lore.kernel.org/git/cover-v7-0.2-00000000000-20210919T075619Z-avarab@gmail.com/\n\nThis v8 fixes a test failure that happened in combination with the\nsg/test-split-index-fix topic, which just unearthed an old\nGIT_TEST_SPLIT_INDEX=true memory leak.\n\nCarlo Marcelo Arenas Belón had a fixup for it (that's currently\napplied to the v7) here:\nhttps://lore.kernel.org/git/20210922111741.82142-1-carenas@gmail.com/\n\nI acked it in\nhttps://lore.kernel.org/git/87h7ec59m7.fsf@evledraar.gmail.com/; but\non second thought I think this is a better solution for the reasons\nnoted in the updated commit message.\n\nÆvar Arnfjörð Bjarmason (2):\n Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n tests: add a test mode for SANITIZE=leak, run it in CI\n\n .github/workflows/main.yml | 3 +++\n Makefile | 5 +++++\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 9 ++++++++-\n t/README | 7 +++++++\n t/t0004-unwritable.sh | 3 ++-\n t/t0011-hashmap.sh | 2 ++\n t/t0016-oidmap.sh | 2 ++\n t/t0017-env-helper.sh | 1 +\n t/t0018-advice.sh | 1 +\n t/t0030-stripspace.sh | 1 +\n t/t0063-string-list.sh | 1 +\n t/t0091-bugreport.sh | 1 +\n t/test-lib.sh | 21 +++++++++++++++++++++\n 14 files changed, 56 insertions(+), 3 deletions(-)\n\nRange-diff against v7:\n1: fc7ba4cb1c3 = 1: c68a7108dc4 Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n2: 56592952db5 ! 2: 90ecd49c910 tests: add a test mode for SANITIZE=leak, run it in CI\n @@ Commit message\n The intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\n as follow-up change, but let's start small to begin with.\n \n + In ci/run-build-and-tests.sh we make use of the default \"*\" case to\n + run \"make test\" without any GIT_TEST_* modes. SANITIZE=leak is known\n + to fail in combination with GIT_TEST_SPLIT_INDEX=true in\n + t0016-oidmap.sh, and we're likely to have other such failures in\n + various GIT_TEST_* modes. Let's focus on getting the base tests\n + passing, we can expand coverage to GIT_TEST_* modes later.\n +\n It would also be possible to implement a more lightweight version of\n this by only relying on setting \"LSAN_OPTIONS\". See\n [1] and\n @@ ci/lib.sh: linux-musl)\n +\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n - ## ci/run-build-and-tests.sh ##\n -@@ ci/run-build-and-tests.sh: fi\n - \n - make\n - case \"$jobname\" in\n --linux-gcc)\n -+linux-gcc|linux-leaks)\n - \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n - \tmake test\n - \texport GIT_TEST_SPLIT_INDEX=yes\n -\n ## t/README ##\n @@ t/README: excluded as so much relies on it, but this might change in the future.\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n-- \n2.33.0.1228.gdc65525c655\n\n"},{"id":"436837","messageId":"patch-v8-1.2-c68a7108dc4-20210923T091819Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v8-0.2-00000000000-20210923T091819Z-avarab@gmail.com","subject":"[PATCH v8 1/2] Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-23T09:20:45Z","receivedAt":"2021-09-23T09:20:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When SANITIZE=leak is specified we'll now add a SANITIZE_LEAK flag to\nGIT-BUILD-OPTIONS, this can then be picked up by the test-lib.sh,\nwhich sets a SANITIZE_LEAK prerequisite.\n\nWe can then skip specific tests that are known to fail under\nSANITIZE=leak, add one such annotation to t0004-unwritable.sh, which\nnow passes under SANITIZE=leak.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++++\n t/t0004-unwritable.sh | 2 +-\n t/test-lib.sh | 1 +\n 3 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 9df565f27bb..d7390e6b2b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1214,6 +1214,9 @@ PTHREAD_CFLAGS =\n SPARSE_FLAGS ?=\n SP_EXTRA_FLAGS = -Wno-universal-initializer\n \n+# For informing GIT-BUILD-OPTIONS of the SANITIZE=leak target\n+SANITIZE_LEAK =\n+\n # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1258,6 +1261,7 @@ BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS\n endif\n ifneq ($(filter leak,$(SANITIZERS)),)\n BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS\n+SANITIZE_LEAK = YesCompiledWithIt\n endif\n ifneq ($(filter address,$(SANITIZERS)),)\n NO_REGEX = NeededForASAN\n@@ -2803,6 +2807,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n+\t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex e3137d638ee..fbdcb926b3a 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -21,7 +21,7 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi\n \ttest_must_fail git write-tree\n '\n \n-test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository' '\n+test_expect_success POSIXPERM,SANITY,!SANITIZE_LEAK 'commit should notice unwritable repository' '\n \ttest_when_finished \"chmod 775 .git/objects .git/objects/??\" &&\n \tchmod a-w .git/objects .git/objects/?? &&\n \ttest_must_fail git commit -m second\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d5ee9642548..06831086060 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1536,6 +1536,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$SANITIZE_LEAK\" && test_set_prereq SANITIZE_LEAK\n \n if test -z \"$GIT_TEST_CHECK_CACHE_TREE\"\n then\n-- \n2.33.0.1228.gdc65525c655\n\n"},{"id":"436839","messageId":"patch-v8-2.2-90ecd49c910-20210923T091819Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v8-0.2-00000000000-20210923T091819Z-avarab@gmail.com","subject":"[PATCH v8 2/2] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-23T09:20:46Z","receivedAt":"2021-09-23T09:21:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"While git can be compiled with SANITIZE=leak, we have not run\nregression tests under that mode. Memory leaks have only been fixed as\none-offs without structured regression testing.\n\nThis change adds CI testing for it. We'll now build and small set of\nwhitelisted t00*.sh tests under Linux with a new job called\n\"linux-leaks\".\n\nThe CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\nmode. When running in that mode, we'll assert that we were compiled\nwith SANITIZE=leak. We'll then skip all tests, except those that we've\nopted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n\nA test setting \"TEST_PASSES_SANITIZE_LEAK=true\" setting can in turn\nmake use of the \"SANITIZE_LEAK\" prerequisite, should they wish to\nselectively skip tests even under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\". In the preceding commit we\nstarted doing this in \"t0004-unwritable.sh\" under SANITIZE=leak, now\nit'll combine nicely with \"GIT_TEST_PASSING_SANITIZE_LEAK=true\".\n\nThis is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\nbe skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\n $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh\n 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set\n\nThe intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations\nas follow-up change, but let's start small to begin with.\n\nIn ci/run-build-and-tests.sh we make use of the default \"*\" case to\nrun \"make test\" without any GIT_TEST_* modes. SANITIZE=leak is known\nto fail in combination with GIT_TEST_SPLIT_INDEX=true in\nt0016-oidmap.sh, and we're likely to have other such failures in\nvarious GIT_TEST_* modes. Let's focus on getting the base tests\npassing, we can expand coverage to GIT_TEST_* modes later.\n\nIt would also be possible to implement a more lightweight version of\nthis by only relying on setting \"LSAN_OPTIONS\". See\n[1] and\n[2] for a discussion of\nthat. I've opted for this approach of adding a GIT_TEST_* mode instead\nbecause it's consistent with how we handle other special test modes.\n\nBeing able to add a \"!SANITIZE_LEAK\" prerequisite and calling\n\"test_done\" early if it isn't satisfied also means that we can more\nincrementally add regression tests without being forced to fix\nwidespread and hard-to-fix leaks at the same time.\n\nWe have tests that do simple checking of some tool we're interested\nin, but later on in the script might be stressing trace2, or common\nsources of leaks like \"git log\" in combination with the tool (e.g. the\ncommit-graph tests). To be clear having a prerequisite could also be\naccomplished by using \"LSAN_OPTIONS\" directly.\n\nOn the topic of \"LSAN_OPTIONS\": It would be nice to have a mode to\naggregate all failures in our various scripts, see [2] for a start at\ndoing that which sets \"log_path\" in \"LSAN_OPTIONS\". I've punted on\nthat for now, it can be added later.\n\nAs of writing this we've got major regressions between master..seen,\ni.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch\n'ah/plugleaks', 2021-08-04) have regressed recently.\n\nSee the discussion at <87czsv2idy.fsf@evledraar.gmail.com>[3] about\nthe lack of this sort of test mode, and 0e5bba53af (add UNLEAK\nannotation for reducing leak false positives, 2017-09-08) for the\ninitial addition of SANITIZE=leak.\n\nSee also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19),\n7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent\n936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the\npast history of \"one-off\" SANITIZE=leak (and more) fixes.\n\nAs noted in [5] we can't support this on OSX yet until Clang 14 is\nreleased, at that point we'll probably want to resurrect that\n\"osx-leaks\" job.\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/\n3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/\n4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/\n5. https://lore.kernel.org/git/20210916035603.76369-1-carenas@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n .github/workflows/main.yml | 3 +++\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 9 ++++++++-\n t/README | 7 +++++++\n t/t0004-unwritable.sh | 1 +\n t/t0011-hashmap.sh | 2 ++\n t/t0016-oidmap.sh | 2 ++\n t/t0017-env-helper.sh | 1 +\n t/t0018-advice.sh | 1 +\n t/t0030-stripspace.sh | 1 +\n t/t0063-string-list.sh | 1 +\n t/t0091-bugreport.sh | 1 +\n t/test-lib.sh | 20 ++++++++++++++++++++\n 13 files changed, 49 insertions(+), 2 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex b053b01c66e..47281684782 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -232,6 +232,9 @@ jobs:\n - jobname: linux-gcc-default\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n env:\n CC: ${{matrix.vector.cc}}\n jobname: ${{matrix.vector.jobname}}\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 5772081b6e5..1d0e48f4515 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -12,7 +12,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..82cb17f8eea 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -183,7 +183,7 @@ export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-leaks)\n \tif [ \"$jobname\" = linux-gcc ]\n \tthen\n \t\texport CC=gcc-8\n@@ -233,4 +233,11 @@ linux-musl)\n \t;;\n esac\n \n+case \"$jobname\" in\n+linux-leaks)\n+\texport SANITIZE=leak\n+\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\t;;\n+esac\n+\n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/t/README b/t/README\nindex 51065d08006..b92155a822e 100644\n--- a/t/README\n+++ b/t/README\n@@ -366,6 +366,13 @@ excluded as so much relies on it, but this might change in the future.\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n test suite. Accept any boolean values that are accepted by git-config.\n \n+GIT_TEST_PASSING_SANITIZE_LEAK= when compiled with\n+SANITIZE=leak will run only those tests that have whitelisted\n+themselves as passing with no memory leaks. Tests can be whitelisted\n+by setting \"TEST_PASSES_SANITIZE_LEAK=true\" before sourcing\n+\"test-lib.sh\" itself at the top of the test script. This test mode is\n+used by the \"linux-leaks\" CI target.\n+\n GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version'\n default to n.\n \ndiff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh\nindex fbdcb926b3a..37d68ef03be 100755\n--- a/t/t0004-unwritable.sh\n+++ b/t/t0004-unwritable.sh\n@@ -2,6 +2,7 @@\n \n test_description='detect unwritable repository and fail correctly'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t0011-hashmap.sh b/t/t0011-hashmap.sh\nindex 5343ffd3f92..e094975b13b 100755\n--- a/t/t0011-hashmap.sh\n+++ b/t/t0011-hashmap.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test hashmap and string hash functions'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_hashmap() {\ndiff --git a/t/t0016-oidmap.sh b/t/t0016-oidmap.sh\nindex 31f8276ba82..0faef1f4f11 100755\n--- a/t/t0016-oidmap.sh\n+++ b/t/t0016-oidmap.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test oidmap'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # This purposefully is very similar to t0011-hashmap.sh\ndiff --git a/t/t0017-env-helper.sh b/t/t0017-env-helper.sh\nindex 4a159f99e44..2e42fba9567 100755\n--- a/t/t0017-env-helper.sh\n+++ b/t/t0017-env-helper.sh\n@@ -2,6 +2,7 @@\n \n test_description='test env--helper'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n \ndiff --git a/t/t0018-advice.sh b/t/t0018-advice.sh\nindex 39e5e4b34f8..c13057a4ca3 100755\n--- a/t/t0018-advice.sh\n+++ b/t/t0018-advice.sh\n@@ -2,6 +2,7 @@\n \n test_description='Test advise_if_enabled functionality'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'advice should be printed when config variable is unset' '\ndiff --git a/t/t0030-stripspace.sh b/t/t0030-stripspace.sh\nindex 0c24a0f9a37..ae1ca380c1a 100755\n--- a/t/t0030-stripspace.sh\n+++ b/t/t0030-stripspace.sh\n@@ -5,6 +5,7 @@\n \n test_description='git stripspace'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n t40='A quick brown fox jumps over the lazy do'\ndiff --git a/t/t0063-string-list.sh b/t/t0063-string-list.sh\nindex c6ee9f66b11..46d4839194b 100755\n--- a/t/t0063-string-list.sh\n+++ b/t/t0063-string-list.sh\n@@ -5,6 +5,7 @@\n \n test_description='Test string list functionality'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_split () {\ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 526304ff95b..eeedbfa9193 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -2,6 +2,7 @@\n \n test_description='git bugreport'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # Headers \"[System Info]\" will be followed by a non-empty line if we put some\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 06831086060..9310d9d900a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1381,6 +1381,26 @@ then\n \ttest_done\n fi\n \n+# skip non-whitelisted tests when compiled with SANITIZE=leak\n+if test -n \"$SANITIZE_LEAK\"\n+then\n+\tif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+\tthen\n+\t\t# We need to see it in \"git env--helper\" (via\n+\t\t# test_bool_env)\n+\t\texport TEST_PASSES_SANITIZE_LEAK\n+\n+\t\tif ! test_bool_env TEST_PASSES_SANITIZE_LEAK false\n+\t\tthen\n+\t\t\tskip_all=\"skipping $this_test under GIT_TEST_PASSING_SANITIZE_LEAK=true\"\n+\t\t\ttest_done\n+\t\tfi\n+\tfi\n+elif test_bool_env GIT_TEST_PASSING_SANITIZE_LEAK false\n+then\n+\terror \"GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak\"\n+fi\n+\n # Last-minute variable setup\n USER_HOME=\"$HOME\"\n HOME=\"$TRASH_DIRECTORY\"\n-- \n2.33.0.1228.gdc65525c655\n\n"},{"id":"440404","messageId":"xmqq4k8s6eri.fsf_-_@gitster.g","threadId":"55888","inReplyTo":"patch-v8-2.2-90ecd49c910-20210923T091819Z-avarab@gmail.com","subject":"Re* [PATCH v8 2/2] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-03T22:44:17Z","receivedAt":"2021-11-03T22:44:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n> mode. When running in that mode, we'll assert that we were compiled\n> with SANITIZE=leak. We'll then skip all tests, except those that we've\n> opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n> ...\n> This is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\n> be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n\nI've been playing with this locally, but cannot shake the nagging\nfeeling that GIT_TEST_PASSING_SANITIZE_LEAK must default to true.\nOtherwise, it is one more thing they need to find out and set when\nthey do\n\n make SANITYZE=leak test\n\nbecause they want to be a good developer and to ensure that they did\nnot introduce new leaks.\n\nIf we want to encourage folks to locally run the leak checks before\ndeclaring their own work \"done\", that is.\n\nThose who are hunting for and cleaning up existing leaks can and\nshould set it to false, no?\n\n\nIn any case, here is a small fallout out of my adventure into this\ncorner.\n\n----- >8 --------- >8 --------- >8 --------- >8 -----\nSubject: t0006: date_mode can leak .strftime_fmt member\n\nAs there is no date_mode_release() API function, and given the\nset of current callers it probably is not worth adding one, let's\nrelease the .strftime_fmt member that is obtained from strdup()\nbefore the caller of show_date() is done with it.\n\nThis allows us to mark t0006 as passing under the leak sanitizer.\n\nSigned-off-by: Junio C Hamano \n---\n t/helper/test-date.c | 2 ++\n t/t0006-date.sh | 2 ++\n 2 files changed, 4 insertions(+)\n\ndiff --git c/t/helper/test-date.c w/t/helper/test-date.c\nindex 099eff4f0f..e15ea02626 100644\n--- c/t/helper/test-date.c\n+++ w/t/helper/test-date.c\n@@ -53,6 +53,8 @@ static void show_dates(const char **argv, const char *format)\n \n \t\tprintf(\"%s -> %s\\n\", *argv, show_date(t, tz, &mode));\n \t}\n+\n+\tfree((void *)mode.strftime_fmt);\n }\n \n static void parse_dates(const char **argv)\ndiff --git c/t/t0006-date.sh w/t/t0006-date.sh\nindex 6b757d7169..5d01f57b27 100755\n--- c/t/t0006-date.sh\n+++ w/t/t0006-date.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test date parsing and printing'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # arbitrary reference time: 2009-08-30 19:20:00\n"},{"id":"440411","messageId":"xmqqee7w4wsq.fsf@gitster.g","threadId":"55888","inReplyTo":"xmqq4k8s6eri.fsf_-_@gitster.g","subject":"Re: Re* [PATCH v8 2/2] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-03T23:57:41Z","receivedAt":"2021-11-03T23:57:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n>> mode. When running in that mode, we'll assert that we were compiled\n>> with SANITIZE=leak. We'll then skip all tests, except those that we've\n>> opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n>> ...\n>> This is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\n>> be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n>\n> I've been playing with this locally, but cannot shake the nagging\n> feeling that GIT_TEST_PASSING_SANITIZE_LEAK must default to true.\n> Otherwise, it is one more thing they need to find out and set when\n> they do\n>\n> make SANITYZE=leak test\n>\n> because they want to be a good developer and to ensure that they did\n> not introduce new leaks.\n>\n> If we want to encourage folks to locally run the leak checks before\n> declaring their own work \"done\", that is.\n>\n> Those who are hunting for and cleaning up existing leaks can and\n> should set it to false, no?\n\nAnother thing while I am at it, I have a feeling that the polarity\nof the TEST_PASSES_SANITIZE_LEAK declaration is the other way\naround.\n\nMarking the tests that do not yet pass the leak check with a special\nannotation will make it easier to find not-yet-clean tests for those\nwho have too much time on their hands ;-) to find ones that are\naffected by the leaky tests.\n"},{"id":"440445","messageId":"211104.86mtmki5ol.gmgdl@evledraar.gmail.com","threadId":"55888","inReplyTo":"xmqq4k8s6eri.fsf_-_@gitster.g","subject":"Re: Re* [PATCH v8 2/2] tests: add a test mode for SANITIZE=leak, run it in CI","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-04T10:06:15Z","receivedAt":"2021-11-04T10:19:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 03 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test\n>> mode. When running in that mode, we'll assert that we were compiled\n>> with SANITIZE=leak. We'll then skip all tests, except those that we've\n>> opted-in by setting \"TEST_PASSES_SANITIZE_LEAK=true\".\n>> ...\n>> This is how tests that don't set \"TEST_PASSES_SANITIZE_LEAK=true\" will\n>> be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true:\n>\n> I've been playing with this locally, but cannot shake the nagging\n> feeling that GIT_TEST_PASSING_SANITIZE_LEAK must default to true.\n> Otherwise, it is one more thing they need to find out and set when\n> they do\n>\n> make SANITYZE=leak test\n>\n> because they want to be a good developer and to ensure that they did\n> not introduce new leaks.\n>\n> If we want to encourage folks to locally run the leak checks before\n> declaring their own work \"done\", that is.\n>\n> Those who are hunting for and cleaning up existing leaks can and\n> should set it to false, no?\n\nI agree that that would make a lot more sense and be more useful :)\n\nThat was the behavior of the patch I originally suggested for\nintegrating this SANITIZE=leak[1], but due to feedback on it I ended up\nkeeping the pre-image behavior of how SANITIZE=leak worked, unless there\nwere any opt-in test modes etc. in play:\nhttps://lore.kernel.org/git/patch-1.4-a61a294132-20210714T001007Z-avarab@gmail.com/\n\nI think at this point it's probably better to just keep it as it is...\n\n> in any case, here is a small fallout out of my adventure into this\n> corner.\n>\n> ----- >8 --------- >8 --------- >8 --------- >8 -----\n> Subject: t0006: date_mode can leak .strftime_fmt member\n>\n> As there is no date_mode_release() API function, and given the\n> set of current callers it probably is not worth adding one, let's\n> release the .strftime_fmt member that is obtained from strdup()\n> before the caller of show_date() is done with it.\n>\n> This allows us to mark t0006 as passing under the leak sanitizer.\n>\n> Signed-off-by: Junio C Hamano \n> ---\n> t/helper/test-date.c | 2 ++\n> t/t0006-date.sh | 2 ++\n> 2 files changed, 4 insertions(+)\n>\n> diff --git c/t/helper/test-date.c w/t/helper/test-date.c\n> index 099eff4f0f..e15ea02626 100644\n> --- c/t/helper/test-date.c\n> +++ w/t/helper/test-date.c\n> @@ -53,6 +53,8 @@ static void show_dates(const char **argv, const char *format)\n> \n> \t\tprintf(\"%s -> %s\\n\", *argv, show_date(t, tz, &mode));\n> \t}\n> +\n> +\tfree((void *)mode.strftime_fmt);\n> }\n\nI'd notice that failure before, but hadn't looked into it. That was\neasier to fix than I thought.\n\nThis fix looks good to me, except that you also need to change this at\nthe top:\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex e15ea026267..9defeb57360 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -34,7 +34,7 @@ static void show_human_dates(const char **argv)\n \n static void show_dates(const char **argv, const char *format)\n {\n- struct date_mode mode;\n+ struct date_mode mode = { 0 };\n \n parse_date_format(format, &mode);\n for (; *argv; argv++) {\n\nI.e. this makes this specific thing pass, but in other tests we'd end up\nfreeing a non-NULL and randomly initialized pointer unless we init it to\nzero.\n\n> \n> static void parse_dates(const char **argv)\n> diff --git c/t/t0006-date.sh w/t/t0006-date.sh\n> index 6b757d7169..5d01f57b27 100755\n> --- c/t/t0006-date.sh\n> +++ w/t/t0006-date.sh\n> @@ -1,6 +1,8 @@\n> #!/bin/sh\n> \n> test_description='test date parsing and printing'\n> +\n> +TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> \n> # arbitrary reference time: 2009-08-30 19:20:00\n\nAnd yeah, that's all that's needed in the test file then.\n"},{"id":"441344","messageId":"patch-1.1-15f5bd3e4f4-20211116T183025Z-avarab@gmail.com","threadId":"55888","inReplyTo":"211104.86mtmki5ol.gmgdl@evledraar.gmail.com","subject":"[PATCH] t0006: date_mode can leak .strftime_fmt member","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-16T18:31:12Z","receivedAt":"2021-11-16T18:31:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Junio C Hamano \n\nAs there is no date_mode_release() API function, and given the\nset of current callers it probably is not worth adding one, let's\nrelease the .strftime_fmt member that is obtained from strdup()\nbefore the caller of show_date() is done with it.\n\nThis allows us to mark t0006 as passing under the leak sanitizer.\n\nSigned-off-by: Junio C Hamano \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nA trivial leak test from Junio that fell between the cracks. Submitted\nwith my suggested fix-up in\nhttps://lore.kernel.org/git/211104.86mtmki5ol.gmgdl@evledraar.gmail.com/\n\n t/helper/test-date.c | 4 +++-\n t/t0006-date.sh | 2 ++\n 2 files changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 099eff4f0fc..27a36a5c5fe 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -34,7 +34,7 @@ static void show_human_dates(const char **argv)\n \n static void show_dates(const char **argv, const char *format)\n {\n-\tstruct date_mode mode;\n+\tstruct date_mode mode = { 0 };\n \n \tparse_date_format(format, &mode);\n \tfor (; *argv; argv++) {\n@@ -53,6 +53,8 @@ static void show_dates(const char **argv, const char *format)\n \n \t\tprintf(\"%s -> %s\\n\", *argv, show_date(t, tz, &mode));\n \t}\n+\n+\tfree((void *)mode.strftime_fmt);\n }\n \n static void parse_dates(const char **argv)\ndiff --git a/t/t0006-date.sh b/t/t0006-date.sh\nindex 6b757d71692..5d01f57b270 100755\n--- a/t/t0006-date.sh\n+++ b/t/t0006-date.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test date parsing and printing'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # arbitrary reference time: 2009-08-30 19:20:00\n-- \n2.34.0.795.g1e9501ab396\n\n"},{"id":"441351","messageId":"xmqqlf1napn4.fsf@gitster.g","threadId":"55888","inReplyTo":"patch-1.1-15f5bd3e4f4-20211116T183025Z-avarab@gmail.com","subject":"Re: [PATCH] t0006: date_mode can leak .strftime_fmt member","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-16T19:04:15Z","receivedAt":"2021-11-16T19:04:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> As there is no date_mode_release() API function, and given the\n> set of current callers it probably is not worth adding one, let's\n> release the .strftime_fmt member that is obtained from strdup()\n> before the caller of show_date() is done with it.\n\nI do not know what the last line exactly wants to say. Perhaps the\noriginal author meant \"after\", not \"before\"? ;-)\n"},{"id":"441353","messageId":"YZQHEiFnOdyxYX5t@coredump.intra.peff.net","threadId":"55888","inReplyTo":"patch-1.1-15f5bd3e4f4-20211116T183025Z-avarab@gmail.com","subject":"Re: [PATCH] t0006: date_mode can leak .strftime_fmt member","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-11-16T19:31:30Z","receivedAt":"2021-11-16T19:31:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 16, 2021 at 07:31:12PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> As there is no date_mode_release() API function, and given the\n> set of current callers it probably is not worth adding one, let's\n> release the .strftime_fmt member that is obtained from strdup()\n> before the caller of show_date() is done with it.\n\nIt does feel a bit ugly to assume that we can touch strftime_fmt here,\nespecially since we don't even confirm that we parsed DATE_STRFTIME.\nYou initialize it as NULL and the current code doesn't touch it\notherwise, so there's no bug. But it would be reasonable for other date\nformats to store ancillary data as a union with strftime_fmt, which\nwould invalidate this.\n\nIt also seems like other callers will need to do similar cleanup. E.g.,\n\"git -c log.date=format:foo log\" has the same leak. So maybe it is worth\nadding an actual cleanup function.\n\n-Peff\n"},{"id":"447556","messageId":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","threadId":"55888","inReplyTo":"YZQHEiFnOdyxYX5t@coredump.intra.peff.net","subject":"[PATCH 0/5] date.[ch] API: split from cache.h, add API docs, stop leaking memory","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:03:48Z","receivedAt":"2022-02-02T21:04:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This is a follow-up to a much smaller patch[1] discussed in November\nto make t0006-date.sh pass with SANITIZE=leak.\n\nIn reply Jeff King pointed out that reaching into its private guts in\nthe test helper felt ugly[2].\n\nSo this series pursues a more thorough approach, creating a date.h,\nmoving our date functions there out of cache.h, documenting the core\nfunctions, and finally adding and using a date_mode_release()\nfunction.\n\nIt's definitely taking the long way around, but I think that the end\nresult is worth it. I then have a follow-up series to plug memory\nleaks in revision.h, which will make use of this new API.\n\n1. https://lore.kernel.org/git/patch-1.1-15f5bd3e4f4-20211116T183025Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/YZQHEiFnOdyxYX5t@coredump.intra.peff.net/\n\nÆvar Arnfjörð Bjarmason (5):\n cache.h: remove always unused show_date_human() declaration\n date API: create a date.h, split from cache.h\n date API: provide and use a DATE_MODE_INIT\n date API: add basic API docs\n date API: add and use a date_mode_release()\n\n archive-zip.c | 1 +\n builtin/am.c | 1 +\n builtin/commit.c | 1 +\n builtin/fast-import.c | 1 +\n builtin/show-branch.c | 1 +\n builtin/tag.c | 1 +\n cache.h | 50 -----------------------------\n config.c | 1 +\n date.c | 9 ++++--\n date.h | 73 +++++++++++++++++++++++++++++++++++++++++++\n http-backend.c | 1 +\n ident.c | 1 +\n object-name.c | 1 +\n pretty.h | 10 ++++++\n ref-filter.c | 3 +-\n refs.c | 1 +\n strbuf.c | 1 +\n t/helper/test-date.c | 5 ++-\n t/t0006-date.sh | 2 ++\n 19 files changed, 110 insertions(+), 54 deletions(-)\n create mode 100644 date.h\n\n-- \n2.35.0.913.g12b4baa2536\n\n"},{"id":"447557","messageId":"patch-1.5-fb21bd7b2c5-20220202T195651Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","subject":"[PATCH 1/5] cache.h: remove always unused show_date_human() declaration","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:03:49Z","receivedAt":"2022-02-02T21:04:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There has never been a show_date_human() function on the \"master\"\nbranch in git.git. This declaration was added in b841d4ff438 (Add\n`human` format to test-tool, 2019-01-28).\n\nA look at the ML history reveals that it was leftover cruft from an\nearlier version of that commit[1].\n\n1. https://lore.kernel.org/git/20190118061805.19086-5-ischis2@cox.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b1..49b46244c74 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1586,8 +1586,6 @@ struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n-void show_date_human(timestamp_t time, int tz, const struct timeval *now,\n-\t\t\tstruct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n int parse_expiry_date(const char *date, timestamp_t *timestamp);\n-- \n2.35.0.913.g12b4baa2536\n\n"},{"id":"447558","messageId":"patch-2.5-7de62956db4-20220202T195651Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","subject":"[PATCH 2/5] date API: create a date.h, split from cache.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:03:50Z","receivedAt":"2022-02-02T21:04:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of the date.c functions from cache.h, and adjust\nthe relevant users to include the new date.h header.\n\nThe show_ident_date() function belonged in pretty.h (it's defined in\npretty.c), its two users outside of pretty.c didn't strictly need to\ninclude pretty.h, as they get it indirectly, but let's add it to them\nanyway.\n\nSimilarly, the change to \"builtin/{fast-import,show-branch,tag}.c\"\nisn't needed as far as the compiler is concerned, but since they all\nuse the \"DATE_MODE()\" macro we now define in date.h, let's have them\ninclude it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n archive-zip.c | 1 +\n builtin/am.c | 1 +\n builtin/commit.c | 1 +\n builtin/fast-import.c | 1 +\n builtin/show-branch.c | 1 +\n builtin/tag.c | 1 +\n cache.h | 48 -------------------------------------------\n config.c | 1 +\n date.c | 1 +\n date.h | 43 ++++++++++++++++++++++++++++++++++++++\n http-backend.c | 1 +\n ident.c | 1 +\n object-name.c | 1 +\n pretty.h | 10 +++++++++\n refs.c | 1 +\n strbuf.c | 1 +\n t/helper/test-date.c | 1 +\n 17 files changed, 67 insertions(+), 48 deletions(-)\n create mode 100644 date.h\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 2961e01c754..8ea9d1a5dae 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -9,6 +9,7 @@\n #include \"object-store.h\"\n #include \"userdiff.h\"\n #include \"xdiff-interface.h\"\n+#include \"date.h\"\n \n static int zip_date;\n static int zip_time;\ndiff --git a/builtin/am.c b/builtin/am.c\nindex b6be1f1cb11..cc8cd6d6e4b 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -34,6 +34,7 @@\n #include \"string-list.h\"\n #include \"packfile.h\"\n #include \"repository.h\"\n+#include \"pretty.h\"\n \n /**\n * Returns the length of the first line of msg.\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ed0374e30..6b99ac276d8 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -37,6 +37,7 @@\n #include \"help.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"pretty.h\"\n \n static const char * const builtin_commit_usage[] = {\n \tN_(\"git commit [] [--] ...\"),\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2b2e28bad79..28f2b9cc91f 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -19,6 +19,7 @@\n #include \"mem-pool.h\"\n #include \"commit-reach.h\"\n #include \"khash.h\"\n+#include \"date.h\"\n \n #define PACK_ID_BITS 16\n #define MAX_PACK_ID ((1<] [-f] [-m | -F ]\\n\"\ndiff --git a/cache.h b/cache.h\nindex 49b46244c74..6add78fd701 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1557,46 +1557,6 @@ struct object *repo_peel_to_type(struct repository *r,\n #define peel_to_type(name, namelen, obj, type) \\\n \trepo_peel_to_type(the_repository, name, namelen, obj, type)\n \n-enum date_mode_type {\n-\tDATE_NORMAL = 0,\n-\tDATE_HUMAN,\n-\tDATE_RELATIVE,\n-\tDATE_SHORT,\n-\tDATE_ISO8601,\n-\tDATE_ISO8601_STRICT,\n-\tDATE_RFC2822,\n-\tDATE_STRFTIME,\n-\tDATE_RAW,\n-\tDATE_UNIX\n-};\n-\n-struct date_mode {\n-\tenum date_mode_type type;\n-\tconst char *strftime_fmt;\n-\tint local;\n-};\n-\n-/*\n- * Convenience helper for passing a constant type, like:\n- *\n- * show_date(t, tz, DATE_MODE(NORMAL));\n- */\n-#define DATE_MODE(t) date_mode_from_type(DATE_##t)\n-struct date_mode *date_mode_from_type(enum date_mode_type type);\n-\n-const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n-void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n-int parse_date(const char *date, struct strbuf *out);\n-int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n-int parse_expiry_date(const char *date, timestamp_t *timestamp);\n-void datestamp(struct strbuf *out);\n-#define approxidate(s) approxidate_careful((s), NULL)\n-timestamp_t approxidate_careful(const char *, int *);\n-timestamp_t approxidate_relative(const char *date);\n-void parse_date_format(const char *format, struct date_mode *mode);\n-int date_overflows(timestamp_t date);\n-time_t tm_to_time_t(const struct tm *tm);\n-\n #define IDENT_STRICT\t 1\n #define IDENT_NO_DATE\t 2\n #define IDENT_NO_NAME\t 4\n@@ -1642,14 +1602,6 @@ struct ident_split {\n */\n int split_ident_line(struct ident_split *, const char *, int);\n \n-/*\n- * Like show_date, but pull the timestamp and tz parameters from\n- * the ident_split. It will also sanity-check the values and produce\n- * a well-known sentinel date if they appear bogus.\n- */\n-const char *show_ident_date(const struct ident_split *id,\n-\t\t\t const struct date_mode *mode);\n-\n /*\n * Compare split idents for equality or strict ordering. Note that we\n * compare only the ident part of the line, ignoring any timestamp.\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a01..9c9dc8a6f62 100644\n--- a/config.c\n+++ b/config.c\n@@ -6,6 +6,7 @@\n *\n */\n #include \"cache.h\"\n+#include \"date.h\"\n #include \"branch.h\"\n #include \"config.h\"\n #include \"environment.h\"\ndiff --git a/date.c b/date.c\nindex 84bb4451c1a..863b07e9e63 100644\n--- a/date.c\n+++ b/date.c\n@@ -5,6 +5,7 @@\n */\n \n #include \"cache.h\"\n+#include \"date.h\"\n \n /*\n * This is like mktime, but without normalization of tm_wday and tm_yday.\ndiff --git a/date.h b/date.h\nnew file mode 100644\nindex 00000000000..5db9ec8dd29\n--- /dev/null\n+++ b/date.h\n@@ -0,0 +1,43 @@\n+#ifndef DATE_H\n+#define DATE_H\n+\n+enum date_mode_type {\n+\tDATE_NORMAL = 0,\n+\tDATE_HUMAN,\n+\tDATE_RELATIVE,\n+\tDATE_SHORT,\n+\tDATE_ISO8601,\n+\tDATE_ISO8601_STRICT,\n+\tDATE_RFC2822,\n+\tDATE_STRFTIME,\n+\tDATE_RAW,\n+\tDATE_UNIX\n+};\n+\n+struct date_mode {\n+\tenum date_mode_type type;\n+\tconst char *strftime_fmt;\n+\tint local;\n+};\n+\n+/*\n+ * Convenience helper for passing a constant type, like:\n+ *\n+ * show_date(t, tz, DATE_MODE(NORMAL));\n+ */\n+#define DATE_MODE(t) date_mode_from_type(DATE_##t)\n+struct date_mode *date_mode_from_type(enum date_mode_type type);\n+\n+const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n+void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n+int parse_date(const char *date, struct strbuf *out);\n+int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n+int parse_expiry_date(const char *date, timestamp_t *timestamp);\n+void datestamp(struct strbuf *out);\n+#define approxidate(s) approxidate_careful((s), NULL)\n+timestamp_t approxidate_careful(const char *, int *);\n+timestamp_t approxidate_relative(const char *date);\n+void parse_date_format(const char *format, struct date_mode *mode);\n+int date_overflows(timestamp_t date);\n+time_t tm_to_time_t(const struct tm *tm);\n+#endif\ndiff --git a/http-backend.c b/http-backend.c\nindex 807fb8839e7..81a7229ece0 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -13,6 +13,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"protocol.h\"\n+#include \"date.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\ndiff --git a/ident.c b/ident.c\nindex 6aba4b5cb6f..89ca5b47008 100644\n--- a/ident.c\n+++ b/ident.c\n@@ -7,6 +7,7 @@\n */\n #include \"cache.h\"\n #include \"config.h\"\n+#include \"date.h\"\n \n static struct strbuf git_default_name = STRBUF_INIT;\n static struct strbuf git_default_email = STRBUF_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex fdff4601b2c..f9527817b64 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -15,6 +15,7 @@\n #include \"submodule.h\"\n #include \"midx.h\"\n #include \"commit-reach.h\"\n+#include \"date.h\"\n \n static int get_oid_oneline(struct repository *r, const char *, struct object_id *, struct commit_list *);\n \ndiff --git a/pretty.h b/pretty.h\nindex 2f16acd213d..f34e24c53a4 100644\n--- a/pretty.h\n+++ b/pretty.h\n@@ -2,6 +2,7 @@\n #define PRETTY_H\n \n #include \"cache.h\"\n+#include \"date.h\"\n #include \"string-list.h\"\n \n struct commit;\n@@ -163,4 +164,13 @@ int format_set_trailers_options(struct process_trailer_options *opts,\n \t\t\tconst char **arg,\n \t\t\tchar **invalid_arg);\n \n+/*\n+ * Like show_date, but pull the timestamp and tz parameters from\n+ * the ident_split. It will also sanity-check the values and produce\n+ * a well-known sentinel date if they appear bogus.\n+ */\n+const char *show_ident_date(const struct ident_split *id,\n+\t\t\t const struct date_mode *mode);\n+\n+\n #endif /* PRETTY_H */\ndiff --git a/refs.c b/refs.c\nindex addb26293b4..33ed3732d1b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,6 +19,7 @@\n #include \"strvec.h\"\n #include \"repository.h\"\n #include \"sigchain.h\"\n+#include \"date.h\"\n \n /*\n * List of all available backends\ndiff --git a/strbuf.c b/strbuf.c\nindex 613fee8c82e..00abeb55afd 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -2,6 +2,7 @@\n #include \"refs.h\"\n #include \"string-list.h\"\n #include \"utf8.h\"\n+#include \"date.h\"\n \n int starts_with(const char *str, const char *prefix)\n {\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 099eff4f0fc..ded3d059f56 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -1,5 +1,6 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n+#include \"date.h\"\n \n static const char *usage_msg = \"\\n\"\n \" test-tool date relative [time_t]...\\n\"\n-- \n2.35.0.913.g12b4baa2536\n\n"},{"id":"447559","messageId":"patch-3.5-2d5210f9421-20220202T195651Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","subject":"[PATCH 3/5] date API: provide and use a DATE_MODE_INIT","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:03:51Z","receivedAt":"2022-02-02T21:04:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Provide and use a DATE_MODE_INIT macro. Most of the users of struct\ndate_mode\" use it via pretty.h's \"struct pretty_print_context\" which\ndoesn't have an initialization macro, so we're still bound to being\ninitialized to \"{ 0 }\" by default.\n\nBut we can change the couple of callers that directly declared a\nvariable on the stack to instead use the initializer, and thus do away\nwith the \"mode.local = 0\" added in add00ba2de9 (date: make \"local\"\northogonal to date format, 2015-09-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.c | 3 +--\n date.h | 4 ++++\n ref-filter.c | 2 +-\n t/helper/test-date.c | 2 +-\n 4 files changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/date.c b/date.c\nindex 863b07e9e63..54c709e4a08 100644\n--- a/date.c\n+++ b/date.c\n@@ -206,11 +206,10 @@ void show_date_relative(timestamp_t time, struct strbuf *timebuf)\n \n struct date_mode *date_mode_from_type(enum date_mode_type type)\n {\n-\tstatic struct date_mode mode;\n+\tstatic struct date_mode mode = DATE_MODE_INIT;\n \tif (type == DATE_STRFTIME)\n \t\tBUG(\"cannot create anonymous strftime date_mode struct\");\n \tmode.type = type;\n-\tmode.local = 0;\n \treturn &mode;\n }\n \ndiff --git a/date.h b/date.h\nindex 5db9ec8dd29..c3a00d08ed6 100644\n--- a/date.h\n+++ b/date.h\n@@ -20,6 +20,10 @@ struct date_mode {\n \tint local;\n };\n \n+#define DATE_MODE_INIT { \\\n+\t.type = DATE_NORMAL, \\\n+}\n+\n /*\n * Convenience helper for passing a constant type, like:\n *\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f7a2f17bfd9..3399bde932f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1251,7 +1251,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam\n \tchar *zone;\n \ttimestamp_t timestamp;\n \tlong tz;\n-\tstruct date_mode date_mode = { DATE_NORMAL };\n+\tstruct date_mode date_mode = DATE_MODE_INIT;\n \tconst char *formatp;\n \n \t/*\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex ded3d059f56..111071e1dd1 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -35,7 +35,7 @@ static void show_human_dates(const char **argv)\n \n static void show_dates(const char **argv, const char *format)\n {\n-\tstruct date_mode mode;\n+\tstruct date_mode mode = DATE_MODE_INIT;\n \n \tparse_date_format(format, &mode);\n \tfor (; *argv; argv++) {\n-- \n2.35.0.913.g12b4baa2536\n\n"},{"id":"447560","messageId":"patch-4.5-aab2ae9cc72-20220202T195651Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","subject":"[PATCH 4/5] date API: add basic API docs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:03:52Z","receivedAt":"2022-02-02T21:04:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add basic API doc comments to date.h, and while doing so move the the\nparse_date_format() function adjacent to show_date(). This way all the\n\"struct date_mode\" functions are grouped together. Documenting the\nrest is one of our #leftoverbits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.h | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/date.h b/date.h\nindex c3a00d08ed6..4ed83506de9 100644\n--- a/date.h\n+++ b/date.h\n@@ -1,6 +1,12 @@\n #ifndef DATE_H\n #define DATE_H\n \n+/**\n+ * The date mode type. This has DATE_NORMAL at an explicit \"= 0\" to\n+ * accommodate a memset([...], 0, [...]) initialization when \"struct\n+ * date_mode\" is used as an embedded struct member, as in the case of\n+ * e.g. \"struct pretty_print_context\" and \"struct rev_info\".\n+ */\n enum date_mode_type {\n \tDATE_NORMAL = 0,\n \tDATE_HUMAN,\n@@ -24,7 +30,7 @@ struct date_mode {\n \t.type = DATE_NORMAL, \\\n }\n \n-/*\n+/**\n * Convenience helper for passing a constant type, like:\n *\n * show_date(t, tz, DATE_MODE(NORMAL));\n@@ -32,7 +38,21 @@ struct date_mode {\n #define DATE_MODE(t) date_mode_from_type(DATE_##t)\n struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n+/**\n+ * Show the date given an initialized \"struct date_mode\" (usually from\n+ * the DATE_MODE() macro).\n+ */\n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n+\n+/**\n+ * Parse a date format for later use with show_date().\n+ *\n+ * When the \"date_mode_type\" is DATE_STRFTIME the \"strftime_fmt\"\n+ * member of \"struct date_mode\" will be a malloc()'d format string to\n+ * be used with strbuf_addftime().\n+ */\n+void parse_date_format(const char *format, struct date_mode *mode);\n+\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n@@ -41,7 +61,6 @@ void datestamp(struct strbuf *out);\n #define approxidate(s) approxidate_careful((s), NULL)\n timestamp_t approxidate_careful(const char *, int *);\n timestamp_t approxidate_relative(const char *date);\n-void parse_date_format(const char *format, struct date_mode *mode);\n int date_overflows(timestamp_t date);\n time_t tm_to_time_t(const struct tm *tm);\n #endif\n-- \n2.35.0.913.g12b4baa2536\n\n"},{"id":"447561","messageId":"patch-5.5-b67e23549ed-20220202T195651Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","subject":"[PATCH 5/5] date API: add and use a date_mode_release()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:03:53Z","receivedAt":"2022-02-02T21:04:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the parse_date_format() function by providing a\nnew date_mode_release() companion function.\n\nBy using this in \"t/helper/test-date.c\" we can mark the\n\"t0006-date.sh\" test as passing when git is compiled with\nSANITIZE=leak, and whitelist it to run under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\" by adding\n\"TEST_PASSES_SANITIZE_LEAK=true\" to the test itself.\n\nThe other tests that expose this memory leak (i.e. take the\n\"mode->type == DATE_STRFTIME\" branch in parse_date_format()) are\n\"t6300-for-each-ref.sh\" and \"t7004-tag.sh\". The former is due to an\neasily fixed leak in \"ref-filter.c\", and brings the failures in\n\"t6300-for-each-ref.sh\" down from 51 to 48.\n\nFixing the remaining leaks will have to wait until there's a\nrelease_revisions() in \"revision.c\", as they have to do with leaks via\n\"struct rev_info\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.c | 5 +++++\n date.h | 9 ++++++++-\n ref-filter.c | 1 +\n t/helper/test-date.c | 2 ++\n t/t0006-date.sh | 2 ++\n 5 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/date.c b/date.c\nindex 54c709e4a08..68a260c214d 100644\n--- a/date.c\n+++ b/date.c\n@@ -993,6 +993,11 @@ void parse_date_format(const char *format, struct date_mode *mode)\n \t\tdie(\"unknown date format %s\", format);\n }\n \n+void date_mode_release(struct date_mode *mode)\n+{\n+\tfree((char *)mode->strftime_fmt);\n+}\n+\n void datestamp(struct strbuf *out)\n {\n \ttime_t now;\ndiff --git a/date.h b/date.h\nindex 4ed83506de9..bfcd4eb458c 100644\n--- a/date.h\n+++ b/date.h\n@@ -49,10 +49,17 @@ const char *show_date(timestamp_t time, int timezone, const struct date_mode *mo\n *\n * When the \"date_mode_type\" is DATE_STRFTIME the \"strftime_fmt\"\n * member of \"struct date_mode\" will be a malloc()'d format string to\n- * be used with strbuf_addftime().\n+ * be used with strbuf_addftime(), in which case you'll need to call\n+ * date_mode_release() later.\n */\n void parse_date_format(const char *format, struct date_mode *mode);\n \n+/**\n+ * Release a \"struct date_mode\", currently only required if\n+ * parse_date_format() has parsed a \"DATE_STRFTIME\" format.\n+ */\n+void date_mode_release(struct date_mode *mode);\n+\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 3399bde932f..7838bd22b8d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1276,6 +1276,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam\n \t\tgoto bad;\n \tv->s = xstrdup(show_date(timestamp, tz, &date_mode));\n \tv->value = timestamp;\n+\tdate_mode_release(&date_mode);\n \treturn;\n bad:\n \tv->s = xstrdup(\"\");\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 111071e1dd1..45951b1df87 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -54,6 +54,8 @@ static void show_dates(const char **argv, const char *format)\n \n \t\tprintf(\"%s -> %s\\n\", *argv, show_date(t, tz, &mode));\n \t}\n+\n+\tdate_mode_release(&mode);\n }\n \n static void parse_dates(const char **argv)\ndiff --git a/t/t0006-date.sh b/t/t0006-date.sh\nindex 794186961ee..2490162071e 100755\n--- a/t/t0006-date.sh\n+++ b/t/t0006-date.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test date parsing and printing'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # arbitrary reference time: 2009-08-30 19:20:00\n-- \n2.35.0.913.g12b4baa2536\n\n"},{"id":"447563","messageId":"220202.86v8xx6k16.gmgdl@evledraar.gmail.com","threadId":"55888","inReplyTo":"patch-2.5-7de62956db4-20220202T195651Z-avarab@gmail.com","subject":"Re: [PATCH 2/5] date API: create a date.h, split from cache.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T21:19:17Z","receivedAt":"2022-02-02T21:22:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 02 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Move the declaration of the date.c functions from cache.h, and adjust\n> the relevant users to include the new date.h header.\n>\n> The show_ident_date() function belonged in pretty.h (it's defined in\n> pretty.c), its two users outside of pretty.c didn't strictly need to\n> include pretty.h, as they get it indirectly, but let's add it to them\n> anyway.\n>\n> Similarly, the change to \"builtin/{fast-import,show-branch,tag}.c\"\n> isn't needed as far as the compiler is concerned, but since they all\n> use the \"DATE_MODE()\" macro we now define in date.h, let's have them\n> include it.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> archive-zip.c | 1 +\n> builtin/am.c | 1 +\n> builtin/commit.c | 1 +\n> builtin/fast-import.c | 1 +\n> builtin/show-branch.c | 1 +\n> builtin/tag.c | 1 +\n> cache.h | 48 -------------------------------------------\n> config.c | 1 +\n> date.c | 1 +\n> date.h | 43 ++++++++++++++++++++++++++++++++++++++\n> http-backend.c | 1 +\n> ident.c | 1 +\n> object-name.c | 1 +\n> pretty.h | 10 +++++++++\n> refs.c | 1 +\n> strbuf.c | 1 +\n> t/helper/test-date.c | 1 +\n> 17 files changed, 67 insertions(+), 48 deletions(-)\n> create mode 100644 date.h\n\nI managed to notice just after hitting \"send\" that I'd forgotten to\n\"make hdr-check\". This commit will need the below fix-up. I'll hold off\non a v2 for now for any further comments though:\n\ndiff --git a/reflog-walk.h b/reflog-walk.h\nindex f26408f6cc1..e9e00ffd479 100644\n--- a/reflog-walk.h\n+++ b/reflog-walk.h\n@@ -5,6 +5,7 @@\n \n struct commit;\n struct reflog_walk_info;\n+struct date_mode;\n \n void init_reflog_walk(struct reflog_walk_info **info);\n int add_reflog_for_walk(struct reflog_walk_info *info,\n"},{"id":"447800","messageId":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com","subject":"[PATCH v2 0/5] date.[ch] API: split from cache.h, add API docs, stop leaking memory","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T23:53:01Z","receivedAt":"2022-02-04T23:53:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix memory leaks in the date.[ch] API, in preparation for larger\nchanges to make the revision walking API stop leaking memory.\n\nThis is a trivial re-roll to v1, to fix an issue that \"make hdr-check\"\nspotted. For v1 see:\nhttps://lore.kernel.org/git/cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (5):\n cache.h: remove always unused show_date_human() declaration\n date API: create a date.h, split from cache.h\n date API: provide and use a DATE_MODE_INIT\n date API: add basic API docs\n date API: add and use a date_mode_release()\n\n archive-zip.c | 1 +\n builtin/am.c | 1 +\n builtin/commit.c | 1 +\n builtin/fast-import.c | 1 +\n builtin/show-branch.c | 1 +\n builtin/tag.c | 1 +\n cache.h | 50 -----------------------------\n config.c | 1 +\n date.c | 9 ++++--\n date.h | 73 +++++++++++++++++++++++++++++++++++++++++++\n http-backend.c | 1 +\n ident.c | 1 +\n object-name.c | 1 +\n pretty.h | 10 ++++++\n ref-filter.c | 3 +-\n reflog-walk.h | 1 +\n refs.c | 1 +\n strbuf.c | 1 +\n t/helper/test-date.c | 5 ++-\n t/t0006-date.sh | 2 ++\n 20 files changed, 111 insertions(+), 54 deletions(-)\n create mode 100644 date.h\n\nRange-diff against v1:\n1: fb21bd7b2c5 = 1: fb21bd7b2c5 cache.h: remove always unused show_date_human() declaration\n2: 7de62956db4 ! 2: 96c904d0b9a date API: create a date.h, split from cache.h\n @@ pretty.h: int format_set_trailers_options(struct process_trailer_options *opts,\n +\n #endif /* PRETTY_H */\n \n + ## reflog-walk.h ##\n +@@\n + \n + struct commit;\n + struct reflog_walk_info;\n ++struct date_mode;\n + \n + void init_reflog_walk(struct reflog_walk_info **info);\n + int add_reflog_for_walk(struct reflog_walk_info *info,\n +\n ## refs.c ##\n @@\n #include \"strvec.h\"\n3: 2d5210f9421 = 3: 9ef003a83bd date API: provide and use a DATE_MODE_INIT\n4: aab2ae9cc72 = 4: 3f70b1aa4c5 date API: add basic API docs\n5: b67e23549ed = 5: 60dbadacb16 date API: add and use a date_mode_release()\n-- \n2.35.1.940.ge7a5b4b05f2\n\n"},{"id":"447801","messageId":"patch-v2-1.5-fb21bd7b2c5-20220204T235143Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"[PATCH v2 1/5] cache.h: remove always unused show_date_human() declaration","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T23:53:02Z","receivedAt":"2022-02-04T23:53:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There has never been a show_date_human() function on the \"master\"\nbranch in git.git. This declaration was added in b841d4ff438 (Add\n`human` format to test-tool, 2019-01-28).\n\nA look at the ML history reveals that it was leftover cruft from an\nearlier version of that commit[1].\n\n1. https://lore.kernel.org/git/20190118061805.19086-5-ischis2@cox.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b1..49b46244c74 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1586,8 +1586,6 @@ struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n-void show_date_human(timestamp_t time, int tz, const struct timeval *now,\n-\t\t\tstruct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n int parse_expiry_date(const char *date, timestamp_t *timestamp);\n-- \n2.35.1.940.ge7a5b4b05f2\n\n"},{"id":"447802","messageId":"patch-v2-2.5-96c904d0b9a-20220204T235143Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"[PATCH v2 2/5] date API: create a date.h, split from cache.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T23:53:03Z","receivedAt":"2022-02-04T23:53:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of the date.c functions from cache.h, and adjust\nthe relevant users to include the new date.h header.\n\nThe show_ident_date() function belonged in pretty.h (it's defined in\npretty.c), its two users outside of pretty.c didn't strictly need to\ninclude pretty.h, as they get it indirectly, but let's add it to them\nanyway.\n\nSimilarly, the change to \"builtin/{fast-import,show-branch,tag}.c\"\nisn't needed as far as the compiler is concerned, but since they all\nuse the \"DATE_MODE()\" macro we now define in date.h, let's have them\ninclude it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n archive-zip.c | 1 +\n builtin/am.c | 1 +\n builtin/commit.c | 1 +\n builtin/fast-import.c | 1 +\n builtin/show-branch.c | 1 +\n builtin/tag.c | 1 +\n cache.h | 48 -------------------------------------------\n config.c | 1 +\n date.c | 1 +\n date.h | 43 ++++++++++++++++++++++++++++++++++++++\n http-backend.c | 1 +\n ident.c | 1 +\n object-name.c | 1 +\n pretty.h | 10 +++++++++\n reflog-walk.h | 1 +\n refs.c | 1 +\n strbuf.c | 1 +\n t/helper/test-date.c | 1 +\n 18 files changed, 68 insertions(+), 48 deletions(-)\n create mode 100644 date.h\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 2961e01c754..8ea9d1a5dae 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -9,6 +9,7 @@\n #include \"object-store.h\"\n #include \"userdiff.h\"\n #include \"xdiff-interface.h\"\n+#include \"date.h\"\n \n static int zip_date;\n static int zip_time;\ndiff --git a/builtin/am.c b/builtin/am.c\nindex b6be1f1cb11..cc8cd6d6e4b 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -34,6 +34,7 @@\n #include \"string-list.h\"\n #include \"packfile.h\"\n #include \"repository.h\"\n+#include \"pretty.h\"\n \n /**\n * Returns the length of the first line of msg.\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ed0374e30..6b99ac276d8 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -37,6 +37,7 @@\n #include \"help.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"pretty.h\"\n \n static const char * const builtin_commit_usage[] = {\n \tN_(\"git commit [] [--] ...\"),\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2b2e28bad79..28f2b9cc91f 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -19,6 +19,7 @@\n #include \"mem-pool.h\"\n #include \"commit-reach.h\"\n #include \"khash.h\"\n+#include \"date.h\"\n \n #define PACK_ID_BITS 16\n #define MAX_PACK_ID ((1<] [-f] [-m | -F ]\\n\"\ndiff --git a/cache.h b/cache.h\nindex 49b46244c74..6add78fd701 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1557,46 +1557,6 @@ struct object *repo_peel_to_type(struct repository *r,\n #define peel_to_type(name, namelen, obj, type) \\\n \trepo_peel_to_type(the_repository, name, namelen, obj, type)\n \n-enum date_mode_type {\n-\tDATE_NORMAL = 0,\n-\tDATE_HUMAN,\n-\tDATE_RELATIVE,\n-\tDATE_SHORT,\n-\tDATE_ISO8601,\n-\tDATE_ISO8601_STRICT,\n-\tDATE_RFC2822,\n-\tDATE_STRFTIME,\n-\tDATE_RAW,\n-\tDATE_UNIX\n-};\n-\n-struct date_mode {\n-\tenum date_mode_type type;\n-\tconst char *strftime_fmt;\n-\tint local;\n-};\n-\n-/*\n- * Convenience helper for passing a constant type, like:\n- *\n- * show_date(t, tz, DATE_MODE(NORMAL));\n- */\n-#define DATE_MODE(t) date_mode_from_type(DATE_##t)\n-struct date_mode *date_mode_from_type(enum date_mode_type type);\n-\n-const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n-void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n-int parse_date(const char *date, struct strbuf *out);\n-int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n-int parse_expiry_date(const char *date, timestamp_t *timestamp);\n-void datestamp(struct strbuf *out);\n-#define approxidate(s) approxidate_careful((s), NULL)\n-timestamp_t approxidate_careful(const char *, int *);\n-timestamp_t approxidate_relative(const char *date);\n-void parse_date_format(const char *format, struct date_mode *mode);\n-int date_overflows(timestamp_t date);\n-time_t tm_to_time_t(const struct tm *tm);\n-\n #define IDENT_STRICT\t 1\n #define IDENT_NO_DATE\t 2\n #define IDENT_NO_NAME\t 4\n@@ -1642,14 +1602,6 @@ struct ident_split {\n */\n int split_ident_line(struct ident_split *, const char *, int);\n \n-/*\n- * Like show_date, but pull the timestamp and tz parameters from\n- * the ident_split. It will also sanity-check the values and produce\n- * a well-known sentinel date if they appear bogus.\n- */\n-const char *show_ident_date(const struct ident_split *id,\n-\t\t\t const struct date_mode *mode);\n-\n /*\n * Compare split idents for equality or strict ordering. Note that we\n * compare only the ident part of the line, ignoring any timestamp.\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a01..9c9dc8a6f62 100644\n--- a/config.c\n+++ b/config.c\n@@ -6,6 +6,7 @@\n *\n */\n #include \"cache.h\"\n+#include \"date.h\"\n #include \"branch.h\"\n #include \"config.h\"\n #include \"environment.h\"\ndiff --git a/date.c b/date.c\nindex 84bb4451c1a..863b07e9e63 100644\n--- a/date.c\n+++ b/date.c\n@@ -5,6 +5,7 @@\n */\n \n #include \"cache.h\"\n+#include \"date.h\"\n \n /*\n * This is like mktime, but without normalization of tm_wday and tm_yday.\ndiff --git a/date.h b/date.h\nnew file mode 100644\nindex 00000000000..5db9ec8dd29\n--- /dev/null\n+++ b/date.h\n@@ -0,0 +1,43 @@\n+#ifndef DATE_H\n+#define DATE_H\n+\n+enum date_mode_type {\n+\tDATE_NORMAL = 0,\n+\tDATE_HUMAN,\n+\tDATE_RELATIVE,\n+\tDATE_SHORT,\n+\tDATE_ISO8601,\n+\tDATE_ISO8601_STRICT,\n+\tDATE_RFC2822,\n+\tDATE_STRFTIME,\n+\tDATE_RAW,\n+\tDATE_UNIX\n+};\n+\n+struct date_mode {\n+\tenum date_mode_type type;\n+\tconst char *strftime_fmt;\n+\tint local;\n+};\n+\n+/*\n+ * Convenience helper for passing a constant type, like:\n+ *\n+ * show_date(t, tz, DATE_MODE(NORMAL));\n+ */\n+#define DATE_MODE(t) date_mode_from_type(DATE_##t)\n+struct date_mode *date_mode_from_type(enum date_mode_type type);\n+\n+const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n+void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n+int parse_date(const char *date, struct strbuf *out);\n+int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n+int parse_expiry_date(const char *date, timestamp_t *timestamp);\n+void datestamp(struct strbuf *out);\n+#define approxidate(s) approxidate_careful((s), NULL)\n+timestamp_t approxidate_careful(const char *, int *);\n+timestamp_t approxidate_relative(const char *date);\n+void parse_date_format(const char *format, struct date_mode *mode);\n+int date_overflows(timestamp_t date);\n+time_t tm_to_time_t(const struct tm *tm);\n+#endif\ndiff --git a/http-backend.c b/http-backend.c\nindex 807fb8839e7..81a7229ece0 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -13,6 +13,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"protocol.h\"\n+#include \"date.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\ndiff --git a/ident.c b/ident.c\nindex 6aba4b5cb6f..89ca5b47008 100644\n--- a/ident.c\n+++ b/ident.c\n@@ -7,6 +7,7 @@\n */\n #include \"cache.h\"\n #include \"config.h\"\n+#include \"date.h\"\n \n static struct strbuf git_default_name = STRBUF_INIT;\n static struct strbuf git_default_email = STRBUF_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex fdff4601b2c..f9527817b64 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -15,6 +15,7 @@\n #include \"submodule.h\"\n #include \"midx.h\"\n #include \"commit-reach.h\"\n+#include \"date.h\"\n \n static int get_oid_oneline(struct repository *r, const char *, struct object_id *, struct commit_list *);\n \ndiff --git a/pretty.h b/pretty.h\nindex 2f16acd213d..f34e24c53a4 100644\n--- a/pretty.h\n+++ b/pretty.h\n@@ -2,6 +2,7 @@\n #define PRETTY_H\n \n #include \"cache.h\"\n+#include \"date.h\"\n #include \"string-list.h\"\n \n struct commit;\n@@ -163,4 +164,13 @@ int format_set_trailers_options(struct process_trailer_options *opts,\n \t\t\tconst char **arg,\n \t\t\tchar **invalid_arg);\n \n+/*\n+ * Like show_date, but pull the timestamp and tz parameters from\n+ * the ident_split. It will also sanity-check the values and produce\n+ * a well-known sentinel date if they appear bogus.\n+ */\n+const char *show_ident_date(const struct ident_split *id,\n+\t\t\t const struct date_mode *mode);\n+\n+\n #endif /* PRETTY_H */\ndiff --git a/reflog-walk.h b/reflog-walk.h\nindex f26408f6cc1..e9e00ffd479 100644\n--- a/reflog-walk.h\n+++ b/reflog-walk.h\n@@ -5,6 +5,7 @@\n \n struct commit;\n struct reflog_walk_info;\n+struct date_mode;\n \n void init_reflog_walk(struct reflog_walk_info **info);\n int add_reflog_for_walk(struct reflog_walk_info *info,\ndiff --git a/refs.c b/refs.c\nindex addb26293b4..33ed3732d1b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,6 +19,7 @@\n #include \"strvec.h\"\n #include \"repository.h\"\n #include \"sigchain.h\"\n+#include \"date.h\"\n \n /*\n * List of all available backends\ndiff --git a/strbuf.c b/strbuf.c\nindex 613fee8c82e..00abeb55afd 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -2,6 +2,7 @@\n #include \"refs.h\"\n #include \"string-list.h\"\n #include \"utf8.h\"\n+#include \"date.h\"\n \n int starts_with(const char *str, const char *prefix)\n {\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 099eff4f0fc..ded3d059f56 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -1,5 +1,6 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n+#include \"date.h\"\n \n static const char *usage_msg = \"\\n\"\n \" test-tool date relative [time_t]...\\n\"\n-- \n2.35.1.940.ge7a5b4b05f2\n\n"},{"id":"447803","messageId":"patch-v2-3.5-9ef003a83bd-20220204T235143Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"[PATCH v2 3/5] date API: provide and use a DATE_MODE_INIT","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T23:53:04Z","receivedAt":"2022-02-04T23:53:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Provide and use a DATE_MODE_INIT macro. Most of the users of struct\ndate_mode\" use it via pretty.h's \"struct pretty_print_context\" which\ndoesn't have an initialization macro, so we're still bound to being\ninitialized to \"{ 0 }\" by default.\n\nBut we can change the couple of callers that directly declared a\nvariable on the stack to instead use the initializer, and thus do away\nwith the \"mode.local = 0\" added in add00ba2de9 (date: make \"local\"\northogonal to date format, 2015-09-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.c | 3 +--\n date.h | 4 ++++\n ref-filter.c | 2 +-\n t/helper/test-date.c | 2 +-\n 4 files changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/date.c b/date.c\nindex 863b07e9e63..54c709e4a08 100644\n--- a/date.c\n+++ b/date.c\n@@ -206,11 +206,10 @@ void show_date_relative(timestamp_t time, struct strbuf *timebuf)\n \n struct date_mode *date_mode_from_type(enum date_mode_type type)\n {\n-\tstatic struct date_mode mode;\n+\tstatic struct date_mode mode = DATE_MODE_INIT;\n \tif (type == DATE_STRFTIME)\n \t\tBUG(\"cannot create anonymous strftime date_mode struct\");\n \tmode.type = type;\n-\tmode.local = 0;\n \treturn &mode;\n }\n \ndiff --git a/date.h b/date.h\nindex 5db9ec8dd29..c3a00d08ed6 100644\n--- a/date.h\n+++ b/date.h\n@@ -20,6 +20,10 @@ struct date_mode {\n \tint local;\n };\n \n+#define DATE_MODE_INIT { \\\n+\t.type = DATE_NORMAL, \\\n+}\n+\n /*\n * Convenience helper for passing a constant type, like:\n *\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f7a2f17bfd9..3399bde932f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1251,7 +1251,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam\n \tchar *zone;\n \ttimestamp_t timestamp;\n \tlong tz;\n-\tstruct date_mode date_mode = { DATE_NORMAL };\n+\tstruct date_mode date_mode = DATE_MODE_INIT;\n \tconst char *formatp;\n \n \t/*\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex ded3d059f56..111071e1dd1 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -35,7 +35,7 @@ static void show_human_dates(const char **argv)\n \n static void show_dates(const char **argv, const char *format)\n {\n-\tstruct date_mode mode;\n+\tstruct date_mode mode = DATE_MODE_INIT;\n \n \tparse_date_format(format, &mode);\n \tfor (; *argv; argv++) {\n-- \n2.35.1.940.ge7a5b4b05f2\n\n"},{"id":"447804","messageId":"patch-v2-4.5-3f70b1aa4c5-20220204T235143Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"[PATCH v2 4/5] date API: add basic API docs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T23:53:05Z","receivedAt":"2022-02-04T23:53:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add basic API doc comments to date.h, and while doing so move the the\nparse_date_format() function adjacent to show_date(). This way all the\n\"struct date_mode\" functions are grouped together. Documenting the\nrest is one of our #leftoverbits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.h | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/date.h b/date.h\nindex c3a00d08ed6..4ed83506de9 100644\n--- a/date.h\n+++ b/date.h\n@@ -1,6 +1,12 @@\n #ifndef DATE_H\n #define DATE_H\n \n+/**\n+ * The date mode type. This has DATE_NORMAL at an explicit \"= 0\" to\n+ * accommodate a memset([...], 0, [...]) initialization when \"struct\n+ * date_mode\" is used as an embedded struct member, as in the case of\n+ * e.g. \"struct pretty_print_context\" and \"struct rev_info\".\n+ */\n enum date_mode_type {\n \tDATE_NORMAL = 0,\n \tDATE_HUMAN,\n@@ -24,7 +30,7 @@ struct date_mode {\n \t.type = DATE_NORMAL, \\\n }\n \n-/*\n+/**\n * Convenience helper for passing a constant type, like:\n *\n * show_date(t, tz, DATE_MODE(NORMAL));\n@@ -32,7 +38,21 @@ struct date_mode {\n #define DATE_MODE(t) date_mode_from_type(DATE_##t)\n struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n+/**\n+ * Show the date given an initialized \"struct date_mode\" (usually from\n+ * the DATE_MODE() macro).\n+ */\n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n+\n+/**\n+ * Parse a date format for later use with show_date().\n+ *\n+ * When the \"date_mode_type\" is DATE_STRFTIME the \"strftime_fmt\"\n+ * member of \"struct date_mode\" will be a malloc()'d format string to\n+ * be used with strbuf_addftime().\n+ */\n+void parse_date_format(const char *format, struct date_mode *mode);\n+\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n@@ -41,7 +61,6 @@ void datestamp(struct strbuf *out);\n #define approxidate(s) approxidate_careful((s), NULL)\n timestamp_t approxidate_careful(const char *, int *);\n timestamp_t approxidate_relative(const char *date);\n-void parse_date_format(const char *format, struct date_mode *mode);\n int date_overflows(timestamp_t date);\n time_t tm_to_time_t(const struct tm *tm);\n #endif\n-- \n2.35.1.940.ge7a5b4b05f2\n\n"},{"id":"447805","messageId":"patch-v2-5.5-60dbadacb16-20220204T235143Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"[PATCH v2 5/5] date API: add and use a date_mode_release()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T23:53:06Z","receivedAt":"2022-02-04T23:53:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the parse_date_format() function by providing a\nnew date_mode_release() companion function.\n\nBy using this in \"t/helper/test-date.c\" we can mark the\n\"t0006-date.sh\" test as passing when git is compiled with\nSANITIZE=leak, and whitelist it to run under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\" by adding\n\"TEST_PASSES_SANITIZE_LEAK=true\" to the test itself.\n\nThe other tests that expose this memory leak (i.e. take the\n\"mode->type == DATE_STRFTIME\" branch in parse_date_format()) are\n\"t6300-for-each-ref.sh\" and \"t7004-tag.sh\". The former is due to an\neasily fixed leak in \"ref-filter.c\", and brings the failures in\n\"t6300-for-each-ref.sh\" down from 51 to 48.\n\nFixing the remaining leaks will have to wait until there's a\nrelease_revisions() in \"revision.c\", as they have to do with leaks via\n\"struct rev_info\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.c | 5 +++++\n date.h | 9 ++++++++-\n ref-filter.c | 1 +\n t/helper/test-date.c | 2 ++\n t/t0006-date.sh | 2 ++\n 5 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/date.c b/date.c\nindex 54c709e4a08..68a260c214d 100644\n--- a/date.c\n+++ b/date.c\n@@ -993,6 +993,11 @@ void parse_date_format(const char *format, struct date_mode *mode)\n \t\tdie(\"unknown date format %s\", format);\n }\n \n+void date_mode_release(struct date_mode *mode)\n+{\n+\tfree((char *)mode->strftime_fmt);\n+}\n+\n void datestamp(struct strbuf *out)\n {\n \ttime_t now;\ndiff --git a/date.h b/date.h\nindex 4ed83506de9..bfcd4eb458c 100644\n--- a/date.h\n+++ b/date.h\n@@ -49,10 +49,17 @@ const char *show_date(timestamp_t time, int timezone, const struct date_mode *mo\n *\n * When the \"date_mode_type\" is DATE_STRFTIME the \"strftime_fmt\"\n * member of \"struct date_mode\" will be a malloc()'d format string to\n- * be used with strbuf_addftime().\n+ * be used with strbuf_addftime(), in which case you'll need to call\n+ * date_mode_release() later.\n */\n void parse_date_format(const char *format, struct date_mode *mode);\n \n+/**\n+ * Release a \"struct date_mode\", currently only required if\n+ * parse_date_format() has parsed a \"DATE_STRFTIME\" format.\n+ */\n+void date_mode_release(struct date_mode *mode);\n+\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 3399bde932f..7838bd22b8d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1276,6 +1276,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam\n \t\tgoto bad;\n \tv->s = xstrdup(show_date(timestamp, tz, &date_mode));\n \tv->value = timestamp;\n+\tdate_mode_release(&date_mode);\n \treturn;\n bad:\n \tv->s = xstrdup(\"\");\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 111071e1dd1..45951b1df87 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -54,6 +54,8 @@ static void show_dates(const char **argv, const char *format)\n \n \t\tprintf(\"%s -> %s\\n\", *argv, show_date(t, tz, &mode));\n \t}\n+\n+\tdate_mode_release(&mode);\n }\n \n static void parse_dates(const char **argv)\ndiff --git a/t/t0006-date.sh b/t/t0006-date.sh\nindex 794186961ee..2490162071e 100755\n--- a/t/t0006-date.sh\n+++ b/t/t0006-date.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test date parsing and printing'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # arbitrary reference time: 2009-08-30 19:20:00\n-- \n2.35.1.940.ge7a5b4b05f2\n\n"},{"id":"448371","messageId":"220214.86czjpxst6.gmgdl@evledraar.gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"Re: [PATCH v2 0/5] date.[ch] API: split from cache.h, add API docs, stop leaking memory","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-14T17:25:58Z","receivedAt":"2022-02-14T17:27:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 05 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Fix memory leaks in the date.[ch] API, in preparation for larger\n> changes to make the revision walking API stop leaking memory.\n>\n> This is a trivial re-roll to v1, to fix an issue that \"make hdr-check\"\n> spotted. For v1 see:\n> https://lore.kernel.org/git/cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com/\n\nJunio: I think this series may have fallen between the cracks. Any\nchance you're willing to pick this up? I'm keen to submit the larger\nrevision.[ch] leak fixes in this cycle, and this is one of the few\nremaining dependencies for that.\n\n> Ævar Arnfjörð Bjarmason (5):\n> cache.h: remove always unused show_date_human() declaration\n> date API: create a date.h, split from cache.h\n> date API: provide and use a DATE_MODE_INIT\n> date API: add basic API docs\n> date API: add and use a date_mode_release()\n>\n> archive-zip.c | 1 +\n> builtin/am.c | 1 +\n> builtin/commit.c | 1 +\n> builtin/fast-import.c | 1 +\n> builtin/show-branch.c | 1 +\n> builtin/tag.c | 1 +\n> cache.h | 50 -----------------------------\n> config.c | 1 +\n> date.c | 9 ++++--\n> date.h | 73 +++++++++++++++++++++++++++++++++++++++++++\n> http-backend.c | 1 +\n> ident.c | 1 +\n> object-name.c | 1 +\n> pretty.h | 10 ++++++\n> ref-filter.c | 3 +-\n> reflog-walk.h | 1 +\n> refs.c | 1 +\n> strbuf.c | 1 +\n> t/helper/test-date.c | 5 ++-\n> t/t0006-date.sh | 2 ++\n> 20 files changed, 111 insertions(+), 54 deletions(-)\n> create mode 100644 date.h\n>\n> Range-diff against v1:\n> 1: fb21bd7b2c5 = 1: fb21bd7b2c5 cache.h: remove always unused show_date_human() declaration\n> 2: 7de62956db4 ! 2: 96c904d0b9a date API: create a date.h, split from cache.h\n> @@ pretty.h: int format_set_trailers_options(struct process_trailer_options *opts,\n> +\n> #endif /* PRETTY_H */\n> \n> + ## reflog-walk.h ##\n> +@@\n> + \n> + struct commit;\n> + struct reflog_walk_info;\n> ++struct date_mode;\n> + \n> + void init_reflog_walk(struct reflog_walk_info **info);\n> + int add_reflog_for_walk(struct reflog_walk_info *info,\n> +\n> ## refs.c ##\n> @@\n> #include \"strvec.h\"\n> 3: 2d5210f9421 = 3: 9ef003a83bd date API: provide and use a DATE_MODE_INIT\n> 4: aab2ae9cc72 = 4: 3f70b1aa4c5 date API: add basic API docs\n> 5: b67e23549ed = 5: 60dbadacb16 date API: add and use a date_mode_release()\n\n"},{"id":"448391","messageId":"xmqq5yph2pkq.fsf@gitster.g","threadId":"55888","inReplyTo":"220214.86czjpxst6.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 0/5] date.[ch] API: split from cache.h, add API docs, stop leaking memory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-14T19:52:37Z","receivedAt":"2022-02-14T21:14:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Sat, Feb 05 2022, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Fix memory leaks in the date.[ch] API, in preparation for larger\n>> changes to make the revision walking API stop leaking memory.\n>>\n>> This is a trivial re-roll to v1, to fix an issue that \"make hdr-check\"\n>> spotted. For v1 see:\n>> https://lore.kernel.org/git/cover-0.5-00000000000-20220202T195651Z-avarab@gmail.com/\n>\n> Junio: I think this series may have fallen between the cracks. Any\n> chance you're willing to pick this up? I'm keen to submit the larger\n> revision.[ch] leak fixes in this cycle, and this is one of the few\n> remaining dependencies for that.\n\nI haven't seen the topic reviewed, and I haven't even had a chance\nto give a cursory look, so until then, it will remain on the list\narchive.\n\nThanks for reminding.\n"},{"id":"448396","messageId":"xmqqzgmtynvx.fsf@gitster.g","threadId":"55888","inReplyTo":"patch-5.5-b67e23549ed-20220202T195651Z-avarab@gmail.com","subject":"Re: [PATCH 5/5] date API: add and use a date_mode_release()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-15T00:28:02Z","receivedAt":"2022-02-15T00:28:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Fix a memory leak in the parse_date_format() function by providing a\n> new date_mode_release() companion function.\n>\n> By using this in \"t/helper/test-date.c\" we can mark the\n> \"t0006-date.sh\" test as passing when git is compiled with\n> SANITIZE=leak, and whitelist it to run under\n> \"GIT_TEST_PASSING_SANITIZE_LEAK=true\" by adding\n> \"TEST_PASSES_SANITIZE_LEAK=true\" to the test itself.\n>\n> The other tests that expose this memory leak (i.e. take the\n> \"mode->type == DATE_STRFTIME\" branch in parse_date_format()) are\n> \"t6300-for-each-ref.sh\" and \"t7004-tag.sh\". The former is due to an\n> easily fixed leak in \"ref-filter.c\", and brings the failures in\n> \"t6300-for-each-ref.sh\" down from 51 to 48.\n>\n> Fixing the remaining leaks will have to wait until there's a\n> release_revisions() in \"revision.c\", as they have to do with leaks via\n> \"struct rev_info\".\n\nHere are hits from \"git grep -e parse_date_format -e date_mode_release\":\n\nbuiltin/blame.c:701:\t\tparse_date_format(value, &blame_date_mode);\nbuiltin/log.c:162:\t\tparse_date_format(default_date_mode, &rev->date_mode);\ndate.c:966:void parse_date_format(const char *format, struct date_mode *mode)\ndate.c:996:void date_mode_release(struct date_mode *mode)\ndate.h:53: * date_mode_release() later.\ndate.h:55:void parse_date_format(const char *format, struct date_mode *mode);\ndate.h:59: * parse_date_format() has parsed a \"DATE_STRFTIME\" format.\ndate.h:61:void date_mode_release(struct date_mode *mode);\nref-filter.c:1266:\t\tparse_date_format(formatp, &date_mode);\nref-filter.c:1279:\tdate_mode_release(&date_mode);\nrevision.c:2478:\t\tparse_date_format(optarg, &revs->date_mode);\nt/helper/test-date.c:40:\tparse_date_format(format, &mode);\nt/helper/test-date.c:58:\tdate_mode_release(&mode);\n\nUnlike builtin/log.c which uses the date_mode member that is\nembedded in a rev_info, the one used by format_time() in\nbuiltin/blame.c should be releasable without waiting for updating\nrevision.c, right?\n\n"},{"id":"448398","messageId":"xmqqtud0zxj2.fsf@gitster.g","threadId":"55888","inReplyTo":"patch-4.5-aab2ae9cc72-20220202T195651Z-avarab@gmail.com","subject":"Re: [PATCH 4/5] date API: add basic API docs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-15T02:14:25Z","receivedAt":"2022-02-15T02:14:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> +/**\n> + * Show the date given an initialized \"struct date_mode\" (usually from\n> + * the DATE_MODE() macro).\n> + */\n> const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n\nIt's a bit of wasted bytes to explain \"show_date()\" as \"show\". In\nthe context of this function, the verb \"show\" in its name does not\nmean emitting to any output stream, but return a short-lived memory\nstuffed with a date string formatted according to the date mode\nthat the caller needs to either immediately consume or strdup() away\nif it wants to use it later, which is a lot more helpful thing to\ntell to the readers.\n\n /**\n * Format <'time', 'timezone'> into static memory according to\n * 'mode' and return it.\n */\n\nor something along that line?\n"},{"id":"448401","messageId":"xmqqo838zv7q.fsf@gitster.g","threadId":"55888","inReplyTo":"patch-2.5-7de62956db4-20220202T195651Z-avarab@gmail.com","subject":"Re: [PATCH 2/5] date API: create a date.h, split from cache.h","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-15T03:04:25Z","receivedAt":"2022-02-15T03:04:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Move the declaration of the date.c functions from cache.h, and adjust\n> the relevant users to include the new date.h header.\n\nIt makes the patch larger than it could be to split off part of\ncache.h into a new header and force users to include the new date.h\nin the same commit, rather than first including date.h in cache.h\nso that users do not have to change, and then update the inclusion\nin a separate follow-up commit. The end result looks OK, though.\n\n"},{"id":"448553","messageId":"patch-v3-1.5-97746d97810-20220216T081203Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","subject":"[PATCH v3 1/5] cache.h: remove always unused show_date_human() declaration","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:14:01Z","receivedAt":"2022-02-16T08:14:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There has never been a show_date_human() function on the \"master\"\nbranch in git.git. This declaration was added in b841d4ff438 (Add\n`human` format to test-tool, 2019-01-28).\n\nA look at the ML history reveals that it was leftover cruft from an\nearlier version of that commit[1].\n\n1. https://lore.kernel.org/git/20190118061805.19086-5-ischis2@cox.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4148b6322d5..703a474e5a7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1588,8 +1588,6 @@ struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n-void show_date_human(timestamp_t time, int tz, const struct timeval *now,\n-\t\t\tstruct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n int parse_expiry_date(const char *date, timestamp_t *timestamp);\n-- \n2.35.1.1028.g2d2d4be19de\n\n"},{"id":"448554","messageId":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com","subject":"[PATCH v3 0/5] date.[ch] API: split from cache.h, add API docs, stop leaking memory","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:14:00Z","receivedAt":"2022-02-16T08:14:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix memory leaks in the date.[ch] API, in preparation for larger\nchanges to make the revision walking API stop leaking memory.\n\nThis is small re-roll of v2 to address Junio's feedback on that\nversion. For v2 see:\nhttps://lore.kernel.org/git/cover-v2-0.5-00000000000-20220204T235143Z-avarab@gmail.com/\n\nThis is a documentation and commit-message only update. As explained\nbelow I think it makes sense to punt on the \"builtin/blame.c\" leak,\nand to keep 2/5 as-is with date.h not included in cache.h, but those\nthings are now all rationalized in the commit message. Thanks for the\nreview Junio!\n\nÆvar Arnfjörð Bjarmason (5):\n cache.h: remove always unused show_date_human() declaration\n date API: create a date.h, split from cache.h\n date API: provide and use a DATE_MODE_INIT\n date API: add basic API docs\n date API: add and use a date_mode_release()\n\n archive-zip.c | 1 +\n builtin/am.c | 1 +\n builtin/commit.c | 1 +\n builtin/fast-import.c | 1 +\n builtin/show-branch.c | 1 +\n builtin/tag.c | 1 +\n cache.h | 50 -----------------------------\n config.c | 1 +\n date.c | 9 ++++--\n date.h | 74 +++++++++++++++++++++++++++++++++++++++++++\n http-backend.c | 1 +\n ident.c | 1 +\n object-name.c | 1 +\n pretty.h | 10 ++++++\n ref-filter.c | 3 +-\n reflog-walk.h | 1 +\n refs.c | 1 +\n strbuf.c | 1 +\n t/helper/test-date.c | 5 ++-\n t/t0006-date.sh | 2 ++\n 20 files changed, 112 insertions(+), 54 deletions(-)\n create mode 100644 date.h\n\nRange-diff against v2:\n1: fb21bd7b2c5 = 1: 97746d97810 cache.h: remove always unused show_date_human() declaration\n2: 96c904d0b9a ! 2: f73aa601e95 date API: create a date.h, split from cache.h\n @@ Commit message\n use the \"DATE_MODE()\" macro we now define in date.h, let's have them\n include it.\n \n + We could simply include this new header in \"cache.h\", but as this\n + change shows these functions weren't common enough to warrant\n + including in it in the first place. By moving them out of cache.h\n + changes to this API will no longer cause a (mostly) full re-build of\n + the project when \"make\" is run.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## archive-zip.c ##\n3: 9ef003a83bd = 3: 764147e90e1 date API: provide and use a DATE_MODE_INIT\n4: 3f70b1aa4c5 ! 4: 5c244960133 date API: add basic API docs\n @@ date.h: struct date_mode {\n struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n +/**\n -+ * Show the date given an initialized \"struct date_mode\" (usually from\n -+ * the DATE_MODE() macro).\n ++ * Format <'time', 'timezone'> into static memory according to 'mode'\n ++ * and return it. The mode is an initialized \"struct date_mode\"\n ++ * (usually from the DATE_MODE() macro).\n + */\n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n +\n5: 60dbadacb16 ! 5: b1ee9a30913 date API: add and use a date_mode_release()\n @@ Commit message\n release_revisions() in \"revision.c\", as they have to do with leaks via\n \"struct rev_info\".\n \n + There is also a leak in \"builtin/blame.c\" due to its call to\n + parse_date_format() to parse the \"blame.date\" configuration. However\n + as it declares a file-level \"static struct date_mode blame_date_mode\"\n + to track the data, LSAN will not report it as a leak. It's possible to\n + get valgrind(1) to complain about it with e.g.:\n +\n + valgrind --leak-check=full --show-leak-kinds=all ./git -P -c blame.date=format:%Y blame README.md\n +\n + But let's focus on things LSAN complains about, and are thus\n + observable with \"TEST_PASSES_SANITIZE_LEAK=true\". We should get to\n + fixing memory leaks in \"builtin/blame.c\", but as doing so would\n + require some re-arrangement of cmd_blame() let's leave it for some\n + other time.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## date.c ##\n-- \n2.35.1.1028.g2d2d4be19de\n\n"},{"id":"448555","messageId":"patch-v3-2.5-f73aa601e95-20220216T081203Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","subject":"[PATCH v3 2/5] date API: create a date.h, split from cache.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:14:02Z","receivedAt":"2022-02-16T08:14:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of the date.c functions from cache.h, and adjust\nthe relevant users to include the new date.h header.\n\nThe show_ident_date() function belonged in pretty.h (it's defined in\npretty.c), its two users outside of pretty.c didn't strictly need to\ninclude pretty.h, as they get it indirectly, but let's add it to them\nanyway.\n\nSimilarly, the change to \"builtin/{fast-import,show-branch,tag}.c\"\nisn't needed as far as the compiler is concerned, but since they all\nuse the \"DATE_MODE()\" macro we now define in date.h, let's have them\ninclude it.\n\nWe could simply include this new header in \"cache.h\", but as this\nchange shows these functions weren't common enough to warrant\nincluding in it in the first place. By moving them out of cache.h\nchanges to this API will no longer cause a (mostly) full re-build of\nthe project when \"make\" is run.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n archive-zip.c | 1 +\n builtin/am.c | 1 +\n builtin/commit.c | 1 +\n builtin/fast-import.c | 1 +\n builtin/show-branch.c | 1 +\n builtin/tag.c | 1 +\n cache.h | 48 -------------------------------------------\n config.c | 1 +\n date.c | 1 +\n date.h | 43 ++++++++++++++++++++++++++++++++++++++\n http-backend.c | 1 +\n ident.c | 1 +\n object-name.c | 1 +\n pretty.h | 10 +++++++++\n reflog-walk.h | 1 +\n refs.c | 1 +\n strbuf.c | 1 +\n t/helper/test-date.c | 1 +\n 18 files changed, 68 insertions(+), 48 deletions(-)\n create mode 100644 date.h\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 2961e01c754..8ea9d1a5dae 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -9,6 +9,7 @@\n #include \"object-store.h\"\n #include \"userdiff.h\"\n #include \"xdiff-interface.h\"\n+#include \"date.h\"\n \n static int zip_date;\n static int zip_time;\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 7de2c89ef22..eb24bc89bb5 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -34,6 +34,7 @@\n #include \"string-list.h\"\n #include \"packfile.h\"\n #include \"repository.h\"\n+#include \"pretty.h\"\n \n /**\n * Returns the length of the first line of msg.\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ed0374e30..6b99ac276d8 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -37,6 +37,7 @@\n #include \"help.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"pretty.h\"\n \n static const char * const builtin_commit_usage[] = {\n \tN_(\"git commit [] [--] ...\"),\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2b2e28bad79..28f2b9cc91f 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -19,6 +19,7 @@\n #include \"mem-pool.h\"\n #include \"commit-reach.h\"\n #include \"khash.h\"\n+#include \"date.h\"\n \n #define PACK_ID_BITS 16\n #define MAX_PACK_ID ((1<] [-f] [-m | -F ]\\n\"\ndiff --git a/cache.h b/cache.h\nindex 703a474e5a7..48e77aa0697 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1559,46 +1559,6 @@ struct object *repo_peel_to_type(struct repository *r,\n #define peel_to_type(name, namelen, obj, type) \\\n \trepo_peel_to_type(the_repository, name, namelen, obj, type)\n \n-enum date_mode_type {\n-\tDATE_NORMAL = 0,\n-\tDATE_HUMAN,\n-\tDATE_RELATIVE,\n-\tDATE_SHORT,\n-\tDATE_ISO8601,\n-\tDATE_ISO8601_STRICT,\n-\tDATE_RFC2822,\n-\tDATE_STRFTIME,\n-\tDATE_RAW,\n-\tDATE_UNIX\n-};\n-\n-struct date_mode {\n-\tenum date_mode_type type;\n-\tconst char *strftime_fmt;\n-\tint local;\n-};\n-\n-/*\n- * Convenience helper for passing a constant type, like:\n- *\n- * show_date(t, tz, DATE_MODE(NORMAL));\n- */\n-#define DATE_MODE(t) date_mode_from_type(DATE_##t)\n-struct date_mode *date_mode_from_type(enum date_mode_type type);\n-\n-const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n-void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n-int parse_date(const char *date, struct strbuf *out);\n-int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n-int parse_expiry_date(const char *date, timestamp_t *timestamp);\n-void datestamp(struct strbuf *out);\n-#define approxidate(s) approxidate_careful((s), NULL)\n-timestamp_t approxidate_careful(const char *, int *);\n-timestamp_t approxidate_relative(const char *date);\n-void parse_date_format(const char *format, struct date_mode *mode);\n-int date_overflows(timestamp_t date);\n-time_t tm_to_time_t(const struct tm *tm);\n-\n #define IDENT_STRICT\t 1\n #define IDENT_NO_DATE\t 2\n #define IDENT_NO_NAME\t 4\n@@ -1644,14 +1604,6 @@ struct ident_split {\n */\n int split_ident_line(struct ident_split *, const char *, int);\n \n-/*\n- * Like show_date, but pull the timestamp and tz parameters from\n- * the ident_split. It will also sanity-check the values and produce\n- * a well-known sentinel date if they appear bogus.\n- */\n-const char *show_ident_date(const struct ident_split *id,\n-\t\t\t const struct date_mode *mode);\n-\n /*\n * Compare split idents for equality or strict ordering. Note that we\n * compare only the ident part of the line, ignoring any timestamp.\ndiff --git a/config.c b/config.c\nindex e0c03d154c9..430868f1ec0 100644\n--- a/config.c\n+++ b/config.c\n@@ -6,6 +6,7 @@\n *\n */\n #include \"cache.h\"\n+#include \"date.h\"\n #include \"branch.h\"\n #include \"config.h\"\n #include \"environment.h\"\ndiff --git a/date.c b/date.c\nindex 84bb4451c1a..863b07e9e63 100644\n--- a/date.c\n+++ b/date.c\n@@ -5,6 +5,7 @@\n */\n \n #include \"cache.h\"\n+#include \"date.h\"\n \n /*\n * This is like mktime, but without normalization of tm_wday and tm_yday.\ndiff --git a/date.h b/date.h\nnew file mode 100644\nindex 00000000000..5db9ec8dd29\n--- /dev/null\n+++ b/date.h\n@@ -0,0 +1,43 @@\n+#ifndef DATE_H\n+#define DATE_H\n+\n+enum date_mode_type {\n+\tDATE_NORMAL = 0,\n+\tDATE_HUMAN,\n+\tDATE_RELATIVE,\n+\tDATE_SHORT,\n+\tDATE_ISO8601,\n+\tDATE_ISO8601_STRICT,\n+\tDATE_RFC2822,\n+\tDATE_STRFTIME,\n+\tDATE_RAW,\n+\tDATE_UNIX\n+};\n+\n+struct date_mode {\n+\tenum date_mode_type type;\n+\tconst char *strftime_fmt;\n+\tint local;\n+};\n+\n+/*\n+ * Convenience helper for passing a constant type, like:\n+ *\n+ * show_date(t, tz, DATE_MODE(NORMAL));\n+ */\n+#define DATE_MODE(t) date_mode_from_type(DATE_##t)\n+struct date_mode *date_mode_from_type(enum date_mode_type type);\n+\n+const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n+void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n+int parse_date(const char *date, struct strbuf *out);\n+int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n+int parse_expiry_date(const char *date, timestamp_t *timestamp);\n+void datestamp(struct strbuf *out);\n+#define approxidate(s) approxidate_careful((s), NULL)\n+timestamp_t approxidate_careful(const char *, int *);\n+timestamp_t approxidate_relative(const char *date);\n+void parse_date_format(const char *format, struct date_mode *mode);\n+int date_overflows(timestamp_t date);\n+time_t tm_to_time_t(const struct tm *tm);\n+#endif\ndiff --git a/http-backend.c b/http-backend.c\nindex 807fb8839e7..81a7229ece0 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -13,6 +13,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"protocol.h\"\n+#include \"date.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\ndiff --git a/ident.c b/ident.c\nindex 6aba4b5cb6f..89ca5b47008 100644\n--- a/ident.c\n+++ b/ident.c\n@@ -7,6 +7,7 @@\n */\n #include \"cache.h\"\n #include \"config.h\"\n+#include \"date.h\"\n \n static struct strbuf git_default_name = STRBUF_INIT;\n static struct strbuf git_default_email = STRBUF_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 92862eeb1ac..060d892a97f 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -15,6 +15,7 @@\n #include \"submodule.h\"\n #include \"midx.h\"\n #include \"commit-reach.h\"\n+#include \"date.h\"\n \n static int get_oid_oneline(struct repository *r, const char *, struct object_id *, struct commit_list *);\n \ndiff --git a/pretty.h b/pretty.h\nindex 2f16acd213d..f34e24c53a4 100644\n--- a/pretty.h\n+++ b/pretty.h\n@@ -2,6 +2,7 @@\n #define PRETTY_H\n \n #include \"cache.h\"\n+#include \"date.h\"\n #include \"string-list.h\"\n \n struct commit;\n@@ -163,4 +164,13 @@ int format_set_trailers_options(struct process_trailer_options *opts,\n \t\t\tconst char **arg,\n \t\t\tchar **invalid_arg);\n \n+/*\n+ * Like show_date, but pull the timestamp and tz parameters from\n+ * the ident_split. It will also sanity-check the values and produce\n+ * a well-known sentinel date if they appear bogus.\n+ */\n+const char *show_ident_date(const struct ident_split *id,\n+\t\t\t const struct date_mode *mode);\n+\n+\n #endif /* PRETTY_H */\ndiff --git a/reflog-walk.h b/reflog-walk.h\nindex f26408f6cc1..e9e00ffd479 100644\n--- a/reflog-walk.h\n+++ b/reflog-walk.h\n@@ -5,6 +5,7 @@\n \n struct commit;\n struct reflog_walk_info;\n+struct date_mode;\n \n void init_reflog_walk(struct reflog_walk_info **info);\n int add_reflog_for_walk(struct reflog_walk_info *info,\ndiff --git a/refs.c b/refs.c\nindex 7017ae59804..b74f3815a52 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,6 +19,7 @@\n #include \"strvec.h\"\n #include \"repository.h\"\n #include \"sigchain.h\"\n+#include \"date.h\"\n \n /*\n * List of all available backends\ndiff --git a/strbuf.c b/strbuf.c\nindex 613fee8c82e..00abeb55afd 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -2,6 +2,7 @@\n #include \"refs.h\"\n #include \"string-list.h\"\n #include \"utf8.h\"\n+#include \"date.h\"\n \n int starts_with(const char *str, const char *prefix)\n {\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 099eff4f0fc..ded3d059f56 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -1,5 +1,6 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n+#include \"date.h\"\n \n static const char *usage_msg = \"\\n\"\n \" test-tool date relative [time_t]...\\n\"\n-- \n2.35.1.1028.g2d2d4be19de\n\n"},{"id":"448556","messageId":"patch-v3-3.5-764147e90e1-20220216T081203Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","subject":"[PATCH v3 3/5] date API: provide and use a DATE_MODE_INIT","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:14:03Z","receivedAt":"2022-02-16T08:14:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Provide and use a DATE_MODE_INIT macro. Most of the users of struct\ndate_mode\" use it via pretty.h's \"struct pretty_print_context\" which\ndoesn't have an initialization macro, so we're still bound to being\ninitialized to \"{ 0 }\" by default.\n\nBut we can change the couple of callers that directly declared a\nvariable on the stack to instead use the initializer, and thus do away\nwith the \"mode.local = 0\" added in add00ba2de9 (date: make \"local\"\northogonal to date format, 2015-09-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.c | 3 +--\n date.h | 4 ++++\n ref-filter.c | 2 +-\n t/helper/test-date.c | 2 +-\n 4 files changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/date.c b/date.c\nindex 863b07e9e63..54c709e4a08 100644\n--- a/date.c\n+++ b/date.c\n@@ -206,11 +206,10 @@ void show_date_relative(timestamp_t time, struct strbuf *timebuf)\n \n struct date_mode *date_mode_from_type(enum date_mode_type type)\n {\n-\tstatic struct date_mode mode;\n+\tstatic struct date_mode mode = DATE_MODE_INIT;\n \tif (type == DATE_STRFTIME)\n \t\tBUG(\"cannot create anonymous strftime date_mode struct\");\n \tmode.type = type;\n-\tmode.local = 0;\n \treturn &mode;\n }\n \ndiff --git a/date.h b/date.h\nindex 5db9ec8dd29..c3a00d08ed6 100644\n--- a/date.h\n+++ b/date.h\n@@ -20,6 +20,10 @@ struct date_mode {\n \tint local;\n };\n \n+#define DATE_MODE_INIT { \\\n+\t.type = DATE_NORMAL, \\\n+}\n+\n /*\n * Convenience helper for passing a constant type, like:\n *\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f7a2f17bfd9..3399bde932f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1251,7 +1251,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam\n \tchar *zone;\n \ttimestamp_t timestamp;\n \tlong tz;\n-\tstruct date_mode date_mode = { DATE_NORMAL };\n+\tstruct date_mode date_mode = DATE_MODE_INIT;\n \tconst char *formatp;\n \n \t/*\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex ded3d059f56..111071e1dd1 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -35,7 +35,7 @@ static void show_human_dates(const char **argv)\n \n static void show_dates(const char **argv, const char *format)\n {\n-\tstruct date_mode mode;\n+\tstruct date_mode mode = DATE_MODE_INIT;\n \n \tparse_date_format(format, &mode);\n \tfor (; *argv; argv++) {\n-- \n2.35.1.1028.g2d2d4be19de\n\n"},{"id":"448557","messageId":"patch-v3-4.5-5c244960133-20220216T081203Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","subject":"[PATCH v3 4/5] date API: add basic API docs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:14:04Z","receivedAt":"2022-02-16T08:14:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add basic API doc comments to date.h, and while doing so move the the\nparse_date_format() function adjacent to show_date(). This way all the\n\"struct date_mode\" functions are grouped together. Documenting the\nrest is one of our #leftoverbits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.h | 24 ++++++++++++++++++++++--\n 1 file changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/date.h b/date.h\nindex c3a00d08ed6..bbd6a6477b5 100644\n--- a/date.h\n+++ b/date.h\n@@ -1,6 +1,12 @@\n #ifndef DATE_H\n #define DATE_H\n \n+/**\n+ * The date mode type. This has DATE_NORMAL at an explicit \"= 0\" to\n+ * accommodate a memset([...], 0, [...]) initialization when \"struct\n+ * date_mode\" is used as an embedded struct member, as in the case of\n+ * e.g. \"struct pretty_print_context\" and \"struct rev_info\".\n+ */\n enum date_mode_type {\n \tDATE_NORMAL = 0,\n \tDATE_HUMAN,\n@@ -24,7 +30,7 @@ struct date_mode {\n \t.type = DATE_NORMAL, \\\n }\n \n-/*\n+/**\n * Convenience helper for passing a constant type, like:\n *\n * show_date(t, tz, DATE_MODE(NORMAL));\n@@ -32,7 +38,22 @@ struct date_mode {\n #define DATE_MODE(t) date_mode_from_type(DATE_##t)\n struct date_mode *date_mode_from_type(enum date_mode_type type);\n \n+/**\n+ * Format <'time', 'timezone'> into static memory according to 'mode'\n+ * and return it. The mode is an initialized \"struct date_mode\"\n+ * (usually from the DATE_MODE() macro).\n+ */\n const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n+\n+/**\n+ * Parse a date format for later use with show_date().\n+ *\n+ * When the \"date_mode_type\" is DATE_STRFTIME the \"strftime_fmt\"\n+ * member of \"struct date_mode\" will be a malloc()'d format string to\n+ * be used with strbuf_addftime().\n+ */\n+void parse_date_format(const char *format, struct date_mode *mode);\n+\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\n@@ -41,7 +62,6 @@ void datestamp(struct strbuf *out);\n #define approxidate(s) approxidate_careful((s), NULL)\n timestamp_t approxidate_careful(const char *, int *);\n timestamp_t approxidate_relative(const char *date);\n-void parse_date_format(const char *format, struct date_mode *mode);\n int date_overflows(timestamp_t date);\n time_t tm_to_time_t(const struct tm *tm);\n #endif\n-- \n2.35.1.1028.g2d2d4be19de\n\n"},{"id":"448558","messageId":"patch-v3-5.5-b1ee9a30913-20220216T081203Z-avarab@gmail.com","threadId":"55888","inReplyTo":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","subject":"[PATCH v3 5/5] date API: add and use a date_mode_release()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:14:05Z","receivedAt":"2022-02-16T08:14:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the parse_date_format() function by providing a\nnew date_mode_release() companion function.\n\nBy using this in \"t/helper/test-date.c\" we can mark the\n\"t0006-date.sh\" test as passing when git is compiled with\nSANITIZE=leak, and whitelist it to run under\n\"GIT_TEST_PASSING_SANITIZE_LEAK=true\" by adding\n\"TEST_PASSES_SANITIZE_LEAK=true\" to the test itself.\n\nThe other tests that expose this memory leak (i.e. take the\n\"mode->type == DATE_STRFTIME\" branch in parse_date_format()) are\n\"t6300-for-each-ref.sh\" and \"t7004-tag.sh\". The former is due to an\neasily fixed leak in \"ref-filter.c\", and brings the failures in\n\"t6300-for-each-ref.sh\" down from 51 to 48.\n\nFixing the remaining leaks will have to wait until there's a\nrelease_revisions() in \"revision.c\", as they have to do with leaks via\n\"struct rev_info\".\n\nThere is also a leak in \"builtin/blame.c\" due to its call to\nparse_date_format() to parse the \"blame.date\" configuration. However\nas it declares a file-level \"static struct date_mode blame_date_mode\"\nto track the data, LSAN will not report it as a leak. It's possible to\nget valgrind(1) to complain about it with e.g.:\n\n valgrind --leak-check=full --show-leak-kinds=all ./git -P -c blame.date=format:%Y blame README.md\n\nBut let's focus on things LSAN complains about, and are thus\nobservable with \"TEST_PASSES_SANITIZE_LEAK=true\". We should get to\nfixing memory leaks in \"builtin/blame.c\", but as doing so would\nrequire some re-arrangement of cmd_blame() let's leave it for some\nother time.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n date.c | 5 +++++\n date.h | 9 ++++++++-\n ref-filter.c | 1 +\n t/helper/test-date.c | 2 ++\n t/t0006-date.sh | 2 ++\n 5 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/date.c b/date.c\nindex 54c709e4a08..68a260c214d 100644\n--- a/date.c\n+++ b/date.c\n@@ -993,6 +993,11 @@ void parse_date_format(const char *format, struct date_mode *mode)\n \t\tdie(\"unknown date format %s\", format);\n }\n \n+void date_mode_release(struct date_mode *mode)\n+{\n+\tfree((char *)mode->strftime_fmt);\n+}\n+\n void datestamp(struct strbuf *out)\n {\n \ttime_t now;\ndiff --git a/date.h b/date.h\nindex bbd6a6477b5..5d4eaba0a90 100644\n--- a/date.h\n+++ b/date.h\n@@ -50,10 +50,17 @@ const char *show_date(timestamp_t time, int timezone, const struct date_mode *mo\n *\n * When the \"date_mode_type\" is DATE_STRFTIME the \"strftime_fmt\"\n * member of \"struct date_mode\" will be a malloc()'d format string to\n- * be used with strbuf_addftime().\n+ * be used with strbuf_addftime(), in which case you'll need to call\n+ * date_mode_release() later.\n */\n void parse_date_format(const char *format, struct date_mode *mode);\n \n+/**\n+ * Release a \"struct date_mode\", currently only required if\n+ * parse_date_format() has parsed a \"DATE_STRFTIME\" format.\n+ */\n+void date_mode_release(struct date_mode *mode);\n+\n void show_date_relative(timestamp_t time, struct strbuf *timebuf);\n int parse_date(const char *date, struct strbuf *out);\n int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 3399bde932f..7838bd22b8d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1276,6 +1276,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam\n \t\tgoto bad;\n \tv->s = xstrdup(show_date(timestamp, tz, &date_mode));\n \tv->value = timestamp;\n+\tdate_mode_release(&date_mode);\n \treturn;\n bad:\n \tv->s = xstrdup(\"\");\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex 111071e1dd1..45951b1df87 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -54,6 +54,8 @@ static void show_dates(const char **argv, const char *format)\n \n \t\tprintf(\"%s -> %s\\n\", *argv, show_date(t, tz, &mode));\n \t}\n+\n+\tdate_mode_release(&mode);\n }\n \n static void parse_dates(const char **argv)\ndiff --git a/t/t0006-date.sh b/t/t0006-date.sh\nindex 794186961ee..2490162071e 100755\n--- a/t/t0006-date.sh\n+++ b/t/t0006-date.sh\n@@ -1,6 +1,8 @@\n #!/bin/sh\n \n test_description='test date parsing and printing'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n # arbitrary reference time: 2009-08-30 19:20:00\n-- \n2.35.1.1028.g2d2d4be19de\n\n"},{"id":"448623","messageId":"xmqqr182u2n0.fsf@gitster.g","threadId":"55888","inReplyTo":"cover-v3-0.5-00000000000-20220216T081203Z-avarab@gmail.com","subject":"Re: [PATCH v3 0/5] date.[ch] API: split from cache.h, add API docs, stop leaking memory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-16T17:45:07Z","receivedAt":"2022-02-16T17:45:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> 2: 96c904d0b9a ! 2: f73aa601e95 date API: create a date.h, split from cache.h\n> @@ Commit message\n> use the \"DATE_MODE()\" macro we now define in date.h, let's have them\n> include it.\n> \n> + We could simply include this new header in \"cache.h\", but as this\n> + change shows these functions weren't common enough to warrant\n> + including in it in the first place. By moving them out of cache.h\n> + changes to this API will no longer cause a (mostly) full re-build of\n> + the project when \"make\" is run.\n> +\n\nIf this step were to include the new header in \"cache.h\" to reduce\nthe patch noise, and there were a follow-up step to update the *.c\nfiles to include the new header while removing the inclusion of the\nheader from \"cache.h\", then the above would make a fine draft for\nthe log message that justifies that follow-up step.\n\nBut if we are doing these two things in a single step, the paragraph\nwould not make a very useful comment to help readers of \"git log\".\n\n> 4: 3f70b1aa4c5 ! 4: 5c244960133 date API: add basic API docs\n> @@ date.h: struct date_mode {\n> struct date_mode *date_mode_from_type(enum date_mode_type type);\n> \n> +/**\n> -+ * Show the date given an initialized \"struct date_mode\" (usually from\n> -+ * the DATE_MODE() macro).\n> ++ * Format <'time', 'timezone'> into static memory according to 'mode'\n> ++ * and return it. The mode is an initialized \"struct date_mode\"\n> ++ * (usually from the DATE_MODE() macro).\n> + */\n> const char *show_date(timestamp_t time, int timezone, const struct date_mode *mode);\n\nOK.\n\n> 5: 60dbadacb16 ! 5: b1ee9a30913 date API: add and use a date_mode_release()\n> @@ Commit message\n> release_revisions() in \"revision.c\", as they have to do with leaks via\n> \"struct rev_info\".\n> \n> + There is also a leak in \"builtin/blame.c\" due to its call to\n> + parse_date_format() to parse the \"blame.date\" configuration. However\n> + as it declares a file-level \"static struct date_mode blame_date_mode\"\n> + to track the data, LSAN will not report it as a leak.\n\nAh, it is not even a leak, then. Is blame the only thing that uses\nparse_date_format() outside the revision walkers?\n\nThanks.\n"}]}