Re: [Outreachy] Move existing tests to a unit testing framework

From: Achu Luma <ach.lumap@gmail.com>
Date: Oct 24, 2023, 14:25 UTC
Message-ID: <CAFR+8DzdFbwaiHtZSdLMqWYWh=fK0WA4c48+eBug-ZeAgddhcQ@mail.gmail.com>
In-Reply-To: <CAP8UFD0A_vWCZ5cVAZqdTBebdhZNye_FmNNJF+vA7epUx2JWHQ@mail.gmail.com>

On Mon, Oct 23, 2023 at 2:41 PM Christian Couder <christian.couder@gmail.com> wrote:

Show 19 quoted lines

>
> On Fri, Oct 20, 2023 at 3:16 PM Achu Luma <ach.lumap@gmail.com> wrote:
> >
> > Dear Git Community and Mentors,
> >
> > I hope this email finds you well. As a follow-up to my previous application, I'd like to provide additional details on the process of migrating existing unit tests from the t/helper/ directory to the new Git unit test framework.
>
> Thanks for these details!
>
> > -- Identify Target Unit Tests: Start by identifying the specific unit tests in the t/helper/ directory that we want to port to the new Git unit test framework. Ensure that the tests are suitable for migration and that the benefits of doing so outweigh the effort(By avoiding integration tests). The following points have been developed with on going work on the unit-tests framework visible here:
> >
> > 1- https://lore.kernel.org/git/0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com/
> > 2- https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc
>
> Maybe if you have time you could add some descriptions or comments
> related to the above emails and documents. For example you could tell
> what the new unit test framework will be like, how the unit tests will
> look like, etc. Maybe a short overview would be nice.
>

sure,
1- https://lore.kernel.org/git/0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com/
:
    The emails highlight the significant milestones achieved in
defining and testing the custom TAP
     framework for writing git unit tests. It also contains some
examples of implementation such as
     that of STRBUF_INIT with output:
      ok 1 - static initialization works
     1..1

2-  https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc:
     From this technical doc, the new unit test framework in the Git
project represents a significant
     enhancement, introducing a systematic and efficient approach to
unit testing. The custom git
     TAP implementation was selected from several alternatives based
on strict criteria as the most
     suitable test framework for porting the unit tests.
     The unit tests are  written in pure C, eliminating the need for
the previous shell/test-tool helper
     setup, simplifying test configuration, data handling, and
reducing testing runtime.
     Each unit test is encapsulated as a function and employs a range
of predefined check functions
     for validation. These checks can evaluate conditions, compare
integers or characters, validate
     strings, and more, providing comprehensive coverage for test scenarios.

    When a test is run using the TEST() macro, it undergoes a series
of checks, and if any check fails,
    a diagnostic message is printed to aid in debugging. This
diagnostic output includes information
    about the specific check that failed, the file and line number
where it occurred, and a clear comparison
    of the expected and actual values. Such detailed reporting
simplifies the identification and resolution
    of issues, contributing to codebase stability.

    Additionally, the framework supports features like skipping tests
with explanations, sending custom
    diagnostic messages using test_msg(), and marking known-to-fail
checks using TEST_TODO().
    This flexibility allows developers to tailor their tests to
specific scenarios while ensuring a
    comprehensive testing suite.

Show 5 quoted lines

> You could also try to apply the patches in the series that adds the
> test framework, or alternatively use the 'seen' branch where the
> series has been merged, and start playing with it by writing, or
> porting, a small example test.
>

ok, I think I can push a patch for one.

Show 5 quoted lines

> > -- Create a New C Test File: For each unit test I plan to migrate, create a new C source file (.c) in the Git project's test suite directory(t/unit-tests). Name it appropriately to reflect the purpose of the test.
>
> Could you provide an example of what the new name would be for an
> existing test that is worth porting?
>

Sure... let's consider an existing unit test in t/helper directory
such as  t/helper/test-date.c or
its shell named t0006-date.sh, which is part of the current
shell-based test suite. In the context
of the new unit testing framework, this test could be reimagined and renamed as
"t-date.c". The "t-" prefix is typically used for test program files
in Git, and "date" is retained to
 reflect the nature of the tests within this suite.

Show 13 quoted lines

> > --  Include Necessary Headers:In the new C test file, include the necessary Git unit test framework headers. Typically, this includes headers like "test-lib.h" and others relevant to the specific test.
> > #include "test-lib.h"
>
> Maybe you could continue the above example and tell which headers
> would be needed for it?
>
> > -- Convert Test Logic: Refactor the test logic from the original Shell script into the new C-based test format. Use the testing macros provided by the Git unit test framework, such as test_expect_success, test_expect_failure, etc., to define the tests.
> > test_expect_success("simple progress display", "{
> >     // Test logic here...
> > }");
>
> Ok, a simple example would be nice too.
>

we can continue with the example used for naming: test-date.c. a typical t-date.c unit test would look like the following: -- #include "test-lib.h" #include "date.h" -- date.h here is a necessary header file. Now refactoring the test logic from the original shell script: -- #include "test-lib.h" #include "date.h"

static void test_parse_dates(void)
{
    const char *dates[] = { "invalid_date", "2023-10-17 10:00:00 +0200", NULL };

    for (const char **argv = dates; *argv; argv++) {
        check_int(parse_dates((const char *[]){ *argv, NULL }), 0);
    }
}
--

Show 13 quoted lines

> > -- Add Test Descriptions: Provide clear and informative descriptions for each test using the testing macros. These descriptions will help in identifying the purpose of each test when the test suite is run.
>
> This would seem to be part of the previous step, as you would have to
> provide a description when using the testing macro. But Ok.
>
> > -- Define a Test Entry Point: Create a cmd_main function as the entry point for the C-based tests. Inside this function, include the test functions using the testing macros.
> > int cmd_main(int argc, const char **argv) {
> >     // Test functions...
> >     return test_done();
> > }
>
> Yeah, continuing an example would be nice.
>

Continuing, we can add a test entrance as follows: -- #include "test-lib.h" #include "date.h"

static void test_parse_dates(void)
{
    const char *dates[] = { "invalid_date", "2023-10-17 10:00:00 +0200", NULL };

    for (const char **argv = dates; *argv; argv++) {
        check_int(parse_dates((const char *[]){ *argv, NULL }), 0);
    }
}

int main(int argc UNUSED, const char **argv UNUSED)
{
    TEST(test_parse_dates, "Test date parsing");

    return test_done();
}
--

A typical unit tests with the custom TAP framework would look something like above. This might run in theory but I have not yet run it as I used it here just for demonstration. The unit tests can be built using "make unit-tests." Additionally, Makefile can be modified to add the file to the build: -- UNIT_TEST_PROGRAMS += t-date --

Show 20 quoted lines

> > -- Ensure TAP Format Output: Ensure that the C-based tests produce output in the Test Anything Protocol (TAP) format. This format includes the test name, status (ok or not ok), and any diagnostic information.
>
> That means using TEST* macros in the cmd_main() function, as they
> should do the right thing or is there more to be done here?
>
> > -- Test Interaction: Ensure that the migrated tests interact correctly with the new Git unit test framework and any other tests that may be relevant. Consider dependencies and interactions with other parts of the Git project.
>
> I am not sure what work would be needed here. Is there more to do than
> compiling the test files? Having an example would be nice.
>
> > -- Test Execution: Run the migrated tests to verify that they produce the expected results when executed as part of the Git project's test suite. Use the Git testing framework's test runners to execute the tests.
>
> Ok.
>
> > -- Documentation Update: Update the Git project's documentation to reflect the changes made during the migration. Include a reference to the original unit tests in the t/helper/ directory and indicate that these tests have been ported to the new Git unit test framework.
>
> I am not sure that we would want that. I think we might instead want
> to document things in t/helper/ that we don't want to port to the new
> unit test framework and why.
>

Ok noted.

Show 13 quoted lines

> > By following these points, I think I can successfully port existing unit tests from the t/helper/ directory to use the new Git unit test framework. This migration helps standardize and streamline the testing process within the Git project, improving code quality and maintainability.
>
> Yeah!
>
> > Next Steps:
> >
> > I am eager to discuss these suggestions and collaborate with the Git community to ensure the success of this project. I will continue to engage with the community, seek guidance, and refine my proposal as per your suggestions.
> >  I look forward to the opportunity to contribute to the Git project and help make it even more robust and reliable.
>
> Thanks for this application and sorry for the late answer!
>
> Best,
> Christian.

I look forward to feedback and better understanding of the new testing framework.

BR, Achu Luma.