{"thread":{"id":"61232","subject":"RFC: Move existing tests to a unit testing framework - Seeking Review","startedAt":"2024-03-30T14:38:55Z","lastAt":"2024-04-02T06:56:53Z","messageCount":2,"participants":["Aishwarya Narayanan","Kaartic Sivaraam"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"491888","messageId":"CAHCXyj0fn5_sS8FM+US-6trM+_QbCcUnLtx8=4eJdFe=rXxkQQ@mail.gmail.com","threadId":"61232","inReplyTo":null,"subject":"RFC: Move existing tests to a unit testing framework - Seeking Review","fromName":"Aishwarya Narayanan","fromEmail":"aishnana.03@gmail.com","sentAt":"2024-03-30T14:38:43Z","receivedAt":"2024-03-30T14:38:55Z","isPatch":false,"sender":{"key":"aishnana.03@gmail.com","avatar":null},"body":"Dear Git Maintainers,\nThis email aims to gather feedback and initiate a discussion on the\nfeasibility and implementation of this idea.\n\n---\nGoogle Summer of Code 2024\n\nName: Aishwarya N\n\nUniversity: SASTRA University\n\nProgram: B.Tech Electrical and Electronics Engineering\n\nOrganization: Git\n\nProject Idea: Move existing tests to a unit testing framework\n\nGitHub: https://github.com/Aishwarya-Narayanan03\n\nEmail: aishnana.03@gmail.com\n\nTimezone: IST (GMT+5:30)\n\n\nAbout Me\n\nMy name is Aishwarya N., and I am soon to graduate from SASTRA Deemed\nUniversity in Thanjavur, India, with a B.Tech. in Electrical and\nElectronics Engineering. With graduation only two months away, I'm\nready to plunge into the exciting world of open source and offer my\nexpertise to a project that enables developers. The power of building\nsomething from nothing through programming is a truly special and\nempowering feeling. It's no wonder it resonates with me! The Git\nproject particularly appeals to me. Contributing to the tool that\nsupports innumerable software developments provides an opportunity to\nhave a genuine, global impact.\n\n\nWhy Me?\n\nAdaptable and Open to Challenges: I thrive in lively surroundings and\nsee new challenges as opportunities to learn and progress. I\nacknowledge that mistakes are unavoidable on the route to mastery, and\nI am not afraid to learn from them.\n\nAn avid learner: My thirst for knowledge is boundless. I'm always\nlooking for new methods to improve myself and push the boundaries of\ntechnology.\n\nCollaborative Spirit: While I'm quite satisfied working independently,\nI believe that collaborating is crucial for accomplishing ambitious\ngoals. I am an adept communicator who is eager to work with the\nbrilliant Git community.\n\nWhy Git?\n\nThe fast-paced world of software piques my curiosity, and Git, a basic\ntool for collaboration and version control, is a necessary skill set\nfor success in this area of expertise. But, beyond its functionality,\nGit is fascinating.I'm excited to contribute to the project that\nunderpins numerous developer workflows.\n\n\n\nProject Summary\n\nThe Git project recently introduced a new unit testing framework aimed\nat improving the quality and maintainability of its source code.\nHowever, some legacy unit tests, such as the \"reftable\" tests in\n't0032-reftable-unittest.sh', predate this framework and continue to\nuse the previous testing method.\n\nThis project's goal is to adapt the existing \"reftable\" unit tests to\nuse the new unit testing framework. This conversion will assure\ncodebase continuity, increase test readability and maintainability,\nand allow for the use of the new framework's additional features and\ncapabilities.\n\n\nBefore GSoC\n\nMicroproject:Avoiding suppressing git's exit code in test scripts\n\nLearning Git Codebase\n\n1.Understanding Test Scripts: Exploring the test scripts in the Git\nproject's t- directory provided personal experience with real-world\nShell scripting used for integration testing. This helpd me understand\nhow Git's features are tested and confirmed.\n\n2.Reading Commits: The reference to commit c6f44e1da5 demonstrated how\nto handle Git commands in test scripts. Analysing this commit allowed\nme to examine the adjustments done to address the issue of suppressing\nGit's exit code, as well as get insight into the codebase's structure\nand convention.\n\n3.Navigating the Git Repository: Searching for instances of\nproblematic code and finding solutions within the Git repository\nenhanced my ability to successfully navigate and search a big\ncodebase.\n\nBest Practices\n\n1.Avoiding Pitfalls:Avoiding typical mistakes in test scripts, such as\nsuppressing exit codes, can lead to more dependable testing\ninfrastructure.\n\n2.Understanding Pipe Usage: Knowing when and how to use pipes in Shell\nscripting is critical. This microproject describes circumstances in\nwhich pipes may mistakenly suppress exit codes and provides\nsuggestions for proper use.\n\nConnecting With Mentors:\n\n1.Engaging with Git contributors: This microproject allowed me to\nconnect with mentors and Git project contributors. Seeking clarity on\nspecific code practices or debating changes to test scripts encourages\ninvolvement in the open-source community.\n\n2.Receiving comments: Submitting updates or queries about the\nmicroproject helped in receving useful comments from experienced\ndevelopers.This feedback helped me learn the Git codebase and improve\nmy scripting skills.\n\n3.Community Involvement: Participating in discussions about the\nmicroproject on platforms such as Git email lists or forums allowed me\nto communicate with a diverse community of developers while sharing\ninformation and experiences.\n\n\nRelated Works\n\nhttps://lore.kernel.org/git/CAHCXyj3U69qyhYewOLY9hN2rvi_5ZuSxQEBJbDxrKefm9MzVWg@mail.gmail.com/\n- Microproject Approach\n\nhttps://lore.kernel.org/git/ZgQffea0krKmZUEt@tanuki/ - [RFC PATCH]\nDescription: While printf might be unlikely to fail here,I felt the\nchange aligns with the project's goal of ensuring Git command exit\ncodes are captured. This approach is more robust and avoids potential\nissues in the future. Status: Declined\n\nhttps://public-inbox.org/git/xmqqttkqwfwe.fsf@gitster.g/\n\nDescription: This patch increases the robustness of Git's testing\nframework by guaranteeing that Git commands' exit codes are\nappropriately examined during test execution. Previously, suppressed\nexit codes allowed tests to pass despite Git command failures. This\nfix stores the output of Git commands in variables and examines their\nexit codes to detect errors.The discussion focuses on improvements to\nthe patch submission process, such as adhering to coding principles\nand referencing appropriate documentation for proper formatting and\ntest script adjustments.\nStatus: Declined\n\nhttps://public-inbox.org/git/CAHCXyj1hUVNNuCOgsNv4GJUi79_o9iWZDvV8Ocz3DodreYoL7g@mail.gmail.com/\n\nDescription: This patch resolves an issue in Git test scripts in which\nthe exit code of git ls-files -t may be disregarded, resulting in\ntests passing despite Git command failures. The new version guarantees\nthat Git commands used in pipelines produce captured output and\nperform adequate exit code checks. It also enhances code style by\nadhering to established rules. This adds to more robust Git testing by\nensuring the identification and reporting of Git command errors within\ntest scripts.\n\nStatus: Under Review\n\n\n\nPorting Unit Tests:\n\n[1]https://lore.kernel.org/git/cover.1692297001.git.steadmon@google.com/\n\nThis patch series provides unit test functionality for the Git project.\nThe series includes the following patches:\nA project plan document that outlines the goals for introducing unit\ntests, as well as a review of potential frameworks and the features\nused to assess them.\nAn implementation of the TAP unit test framework with a sample unit\ntest and Makefile integration.\nChanges were made to the Git build system so that the unit tests could\nbe run in CI.\nThe TAP framework was selected because of the following advantages:\nSimple to use and comprehend.\nWidely used and integrates seamlessly with existing Git tools.\n\nThe project plan document mentions a number of outstanding TODOs,\nincluding how to ensure tests run on various OS platforms and how to\ngather unit test statistics.\nThis is a well-written patch series with a straightforward and logical\napproach. The documentation is comprehensive and instructive, and the\ncode is well-organized and simple to grasp.\nHere are some extra factors to consider:\nThe patch series discusses collecting unit test statistics. We could\nyou clarify on the type of statistics we want to collect?\nHow do we intend to deal with flaky tests?\n\n[2]https://lore.kernel.org/git/20240112102743.1440-1-ach.lumap@gmail.com/\n\nThis is an email thread about a patch series submitted by Achu Luma to\nport the unit tests for C character classification functions from the\nlegacy helper tool to the new unit testing framework\n([Outreachy][PATCH v5] Port helper/test-ctype.c to\nunit-tests/t-ctype.c).\n\nHere are the major changes implemented in the patch series:\nThe unit tests for C character classification routines (isdigit(),\nisspace(), etc.) are moved from the historical approach, which used\nthe test-tool command test-tool ctype in t/helper/test-ctype.c, to the\nnew unit testing framework (t/unit-tests/test-lib.h).\nRefactors the tests to use the framework's testing macros (TEST() and\ncheck_*()).\nAdded tests for handling EOF (end-of-file).\nOverall, the patch series was well received and thought to be a\npositive development. The reviewers made various suggestions for\nimprovement, such as expanding on the types of statistics the author\nis interested in gathering for the unit tests and how to deal with\nflaky tests.\n\n[3] https://lore.kernel.org/git/20240112102122.1422-1-ach.lumap@gmail.com/\n\nThis patch migrates unit tests for the advise_if_enabled functionality\nfrom the legacy approach to a new unit testing framework. Here's a\nbreakdown of the changes:\n\nLegacy approach:\n\nUsed the test-tool advise command in t/helper/test-advise.c.\n\nRelied on helper functions from test-tool.h.\n\nTests were written in shell script (t/t0018-advice.sh).\n\nNew approach:\n\nUses the new unit testing framework with TEST() macros\n(t/unit-tests/t-advise.c).\n\nIncludes necessary headers for advise.h, config.h, and setup.h.\n\nTests are written in C code.\n\nThe patch removes the following files from the legacy approach:\n\nt/helper/test-advise.c\n\nt/t0018-advice.sh\n\nThe new unit test file t/unit-tests/t-advise.c includes three tests\ncovering different scenarios:\n\nAdvice is printed when the configuration variable is unset.\n\nAdvice is printed when the configuration variable is set to true.\n\nAdvice is not printed when the configuration variable is set to false.\n\nOverall, this patch improves the codebase by using a more standardized\nand maintainable approach for unit testing.\n\n[4]https://lore.kernel.org/git/20240205162506.1835-1-ach.lumap@gmail.com/\n\nRefactoring 64-bit Pre-requisite Checks in date.h\n\nThis is a patch for the Linux kernel which refactors pre-requisite\nchecks for 64-bit systems into reusable functions in the date.h header\nfile.\n\nContext:\n\nThe codebase recently added a unit testing framework.\n\nSome shell tests use a helper tool that checks for 64-bit\nprerequisites (TIME_IS_64BIT and TIME_T_IS_64BIT).\n\nThese checks are also needed in the new unit tests.\n\nTo avoid duplication, the patch moves these checks to date.h as inline\nfunctions.\n\nChanges:\n\nTwo new inline functions are added to date.h:\n\ncheck_prereq_TIME_IS_64BIT: Checks if the size of timestamp_t is 8\nbytes (indicating a 64-bit system).\n\ncheck_prereq_TIME_T_IS_64BIT: Checks if the size of time_t is 8 bytes\n(indicating a 64-bit system).\n\nThe shell test functions in t/helper/test-date.c are modified to use\nthe new functions instead of directly checking the sizes.\n\nBenefits:\n\nReduces code duplication.\n\nImproves code maintainability.\n\nSimplifies porting shell test code to the new unit test framework.\n\nNext Steps:\n\nA following commit will port the relevant code from\nt/helper/test-date.c to the new unit test file t/unit-tests/t-date.c,\nusing the newly introduced functions for pre-requisite checks.\n\n[3] Unit Testing in Git:\n\nhttps://github.com/git/git/blob/master/Documentation/technical/unit-tests.txt\ndiscusses unit testing for the Git project. This gives an idea of the\nbackground and importance of unit testing in the project applying to.\n\n\n\nDuring GSoc\nThe main goals of this project are:\n1. Understand the existing \"reftable\" unit tests: Examine the present\nimplementation of the \"reftable\" unit tests in\n't0032-reftable-unittest.sh' to ensure that you understand their\nfunctionality and purpose.\n2. Learn the new unit testing framework: Get a thorough grasp of Git's\nnew unit testing framework, including setup, usage, and recommended\npractices.\n3.Ensuring test coverage and reliability: Check that the converted\ntests have sufficient code coverage and accurately simulate the\nexpected behaviour of the \"reftable\" capability. Test and validate the\nnew tests thoroughly to ensure their reliability.\n4. Documentation and code comments: Document the conversion process,\nincluding any issues encountered and the reasoning behind design\ndecisions. Update the code comments and documentation to reflect the\nchanges to the tests.\n\nDeliverables\n\nü Migrated reftable unit tests created in C with the unit testing framework.\n\nü  Unit tests thoroughly performed and validated to ensure accurate\ncoverage of reftable functionality.\n\nü Documentation explaining the migration procedure, as well as\nrevisions to existing documentation.\n\nü The final report summarises the project's accomplishments, problems,\nand future recommendations.\n\nExpected timeline\nCommunity bonding period (May 1 - May 26)\n\nFamiliarise myself with the Git project's code and contribution standards.\nDiscuss the project plan and milestones with mentors.\nEstablish the development environment and tools required for the project.\n\nPhase 1: Analysis and Planning (May 27 - June 15)\nExamine the current reftable unit tests and the unit testing framework\ndocumentation.\nPlan the migration process by identifying the test cases and\ncapabilities that will be covered.\nBreak down tasks and establish a precise project timeline.\n\nPhase 2: Transition to C-based Unit Tests (June 16–July 15)\nBegin rebuilding the existing shell scripts as C-based unit tests.\nEnsure that the migration process is accurate and comprehensive, while\nretaining test coverage.\nCommunicate with mentors on a regular basis to receive feedback and guidance.\n\nPhase 3: Testing and Validation (July 16–August 5)\nEvaluate the migrated unit tests against different scenarios and edge\nsituations.\nDebug and resolve any errors or anomalies that arise during testing.\nBefore finalising, ensure that the unit tests are proper and reliable.\n\nPhase 4: Documentation and Reporting (August 6–August 26)\nDocument the migration process, including the obstacles encountered\nand solutions implemented.\nUpdate the relevant documentation to reflect the modifications to the\nreftable unit tests.\nPrepare a detailed report summarising the project's results and contributions.\n\nAvailability\n\nMy university holidays begin on June 30. However, considering this is\nmy final year with only three classes and major project, I believe it\nwill not be a problem. I intent to work 6 hours a day, 5 days a week\nthis summer.I have no scheduled trips except for GIM-SRIP from May 6th\nto 31st. I will inform Mentors advance notice of any changes in\navailability.\n\nAfter GSoC\n\nI realise that it would be much more beneficial if our GSoC\nparticipants could stay after the event. And that is exactly what I\nwant to do. I think doing open-source, especially with a community\nthat fuels a popular development tool, is simply fascinating. And, in\naddition to being refreshing, such a community allows me to learn and\ngrow, with the possibility of making more significant contributions if\nI continue to participate.While this is my first foray into\nopen-source development and GSoC, I am excited by the prospect of\ncontributing to the ongoing success of Git. I have been actively\nlearning and gaining a strong foundation in Git commands, programming\nlanguages, and version control concepts.\n\nClosing Remarks\n\nFinally, I'd like to express my gratitude to the community and, in\nparticular, my mentors, Patrick Steinhardt,Christian Couder,Karthik\nNayak,Junio C Hamano. They did and continue to do an exceptional task\nof maintaining and empowering the Git open-source community, as well\nas offering much-needed and kind assistance to a new contributor like\nme.\n---\nRegards\nAishwarya Narayanana\n"},{"id":"492025","messageId":"CA+ARAtpqD0um9bVrjRKG0DmrxVR-46uSKfDKnO+H1rUp0i+4Ww@mail.gmail.com","threadId":"61232","inReplyTo":"CAHCXyj0fn5_sS8FM+US-6trM+_QbCcUnLtx8=4eJdFe=rXxkQQ@mail.gmail.com","subject":"Re: RFC: Move existing tests to a unit testing framework - Seeking Review","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2024-04-02T06:56:42Z","receivedAt":"2024-04-02T06:56:53Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"Hi Aishwarya Narayanan,\n\nThank you for your proposal.\n\nOn Sat, Mar 30, 2024 at 8:08 PM Aishwarya Narayanan\n<aishnana.03@gmail.com> wrote:\n>\n> This project's goal is to adapt the existing \"reftable\" unit tests to\n> use the new unit testing framework. This conversion will assure\n> codebase continuity, increase test readability and maintainability,\n> and allow for the use of the new framework's additional features and\n> capabilities.\n>\n\nYour project summary and the deliverables below seem to mention that you're\ngoing to work on migration of the reftable unit tests but the project\ntitle seems\nto be \"Move existing tests to a unit testing framework'.\n\nJust to be clear, the \"Move existing tests to a unit testing framework\" and the\n\"Convert reftable unit tests to use the unit testing framework\" are\ntwo different\nprojects as clarified in the ideas page for Git [1].\n\nIf you wish to work on the reftable unit test migration, I request you\nto adjust the\nproposal to clearly clarify the same. Make sure to understand that the reftable\nunit tests differ from the other unit tests in their coding style. You\nmight also want\nto consider them while drafting the proposal. Check the mail from\nPatrick[2] for more\ninformation on the same.\n\n[1]: https://git.github.io/SoC-2024-Ideas/\n[2]: https://public-inbox.org/git/ZfwnrL6Zl_lcV09y@tanuki/\n\n--\nSivaraam\n"}]}