{"thread":{"id":"63262","subject":"[GSoC v2] Project Proposal: Machine-Readable Repository Information Query Tool","startedAt":"2025-04-07T03:44:17Z","lastAt":"2025-04-07T03:44:17Z","messageCount":1,"participants":["Lucas Seiki Oshiro"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"515741","messageId":"985FB467-6B19-4387-9B7B-BE3B185B087A@gmail.com","threadId":"63262","inReplyTo":null,"subject":"[GSoC v2] Project Proposal: Machine-Readable Repository Information Query Tool","fromName":"Lucas Seiki Oshiro","fromEmail":"lucasseikioshiro@gmail.com","sentAt":"2025-04-07T03:44:01Z","receivedAt":"2025-04-07T03:44:17Z","isPatch":false,"sender":{"key":"lucasseikioshiro@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701580?v=4"},"body":"Hello again!\n\nThis is the second version of my GSoC proposal. I made some changes based on\nPatrick review of v1 by Patrick Steinhardt:\n\n- Detailing the status of my patches;\n- Focusing in `rev-parse` features, listing all that Patrick suggested and\n  searching for a few more that may be useful;\n- Given it is more focused on rev-parse, the parts about surveying users\n  and projects for features was reduced but not entirely suppressed, as I'll\n  need to find __where__ and __why__ this new command will be important. But\n  now this will not be as important I as thought it would be;\n- Changed the expected output, which becomes very smaller compared to what I\n  thought it would be;\n- Detailing a little bit more some examples of functions that I expect to use\n  for retrieving data to populate the JSON that will be outputted;\n- Breaking the schedule into 6 steps, bringing part the development of the\n  JSON serializer to the beginning.\n\nThanks again for the review. I plan to submit the final version 20 hours from\nnow.\n\n---\n\n# Machine-Readable Repository Information Query Tool\n\n## Contact info\n\n- Name: Lucas Seiki Oshiro\n- Timezone: GMT-3\n- IRC:\n- GitHub: https://github.com/lucasoshiro\n- LinkedIn: https://www.linkedin.com/in/lucasseikioshiro/\n\n## About me\n\nMy name is Lucas Oshiro, I'm a developer and CS bachelor from São Paulo,\nBrazil. Currently I'm pursuing a master degree in CS at University of São\nPaulo. My interest in Git dates from years ago and I even submitted a\npatch to its codebase in the past, though I couldn't complete it due to\nscheduling conflicts with my capstone project.\n\nHaving experience in the academia, industry and FLOSS, I highly value\ncode quality, code legibility, well-maintained Git histories, unit tests\nand documentation.\n\n### Previous experience with Git\n\nBefore this year, I haven't been involved directly with Git community,\nhowever, I kept my interest in Git alive by:\n\n- Translating the \"Git Internals\" chapter of Pro Git to Brazilian\n  Portuguese: https://github.com/progit/progit2-pt-br/pull/81;\n\n- Writing some blog posts about Git, for example:\n  - one explaining how Git can be used as a debugging tool:\n    https://lucasoshiro.github.io/posts-en/2023-02-13-git-debug/;\n\n  - other explaining how Git merge submodules:\n  https://lucasoshiro.github.io/posts-en/2022-03-12-merge-submodule/;\n\n- Writing a compatible subset of Git in Haskell from scratch:\n https://github.com/lucasoshiro/oshit;\n\n- Helping organizing a Git Introductory Workshop at my University:\n  https://flusp.ime.usp.br/events/git-introductory-workshop/;\n\n- Presenting some lectures about Git in a company that I worked some\n  years ago, covering the Git internals (objects, references, packfile)\n  and debugging and archaeology related Git tools (blame, bisect,\n  pickaxe, ls-files, etc).\n\n### Previous experience with C and open-source\n\nI also have experience with C and some C++. During my CS\ncourse, C was one of the primary languages that I used. I also\nworked with C/C++, for example, in:\n\n- Writing an AMQP message broker from scratch:\n  https://github.com/lucasoshiro/amqp_broker;\n\n- Contributing with simple patches to the IIO subsystem of the Linux\n  kernel: https://lucasoshiro.github.io/floss-en/2020-06-06-kernel_linux/;\n\n- Contributing to the Marlin firmware for 3D printers:\n  https://lucasoshiro.github.io/floss-en/2020-06-30-Marlin/;\n\n- Writing a module for the ns-3 network simulator, dealing with both C\n  and C++ codebases (currently under development, I plan to write a\n  paper and make the code available soon);\n\nDuring my CS course I also was member of FLUSP\n(https://flusp.ime.usp.br), a group in my university focused on FLOSS\ncontributions and from Hardware Livre USP\n(https://hardwarelivreusp.org), another group that was focused on\nworking with open-source hardware.\n\nAs a master's student, I'm one of the Open Science Ambassadors of my\nUniversity (https://cienciaaberta.usp.br/sobre-o-projeto/, in\nPortuguese), promoting the Open Science principles, which include\nopen-source software, in the unit where I study.\n\nI also contributed to some other free/open-source software, which I list\nhere: https://lucasoshiro.github.io/floss-en/\n\n### Activity in the Git community in 2025\n\nSince when I decided to submit a proposal for GSoC, I sent some patches\nto the Git codebase and git.github.io:\n\n- My microproject, replacing some `test -f` by `test_path_is_file`:\n  https://lore.kernel.org/git/20250208165731.78804-1-lucasseikioshiro@gmail.com/,\n  merged to master;\n\n- Adding a paragraph to the merge-strategies documentation describing how\n  Git merges submodules (based on the blog post that I mentioned\n  before):\n  https://lore.kernel.org/git/20250227014406.20527-1-lucasseikioshiro@gmail.com/,\n  merge to master;\n\n- A patchset adding a new `--subject-extra-prefix` flag for `git format-patch`,\n  allowing the user to quickly prepend tags like [GSoC], [Newbie] or [Outreachy]\n  to the beginning of the subject:\n  https://lore.kernel.org/git/20250303220029.10716-1-lucasseikioshiro@gmail.com/.\n  This patchset was rejected in favor of just using `--subject-prefix='GSoC\n  PATCH'` or similar;\n\n- Given the feedback on the previous rejected patchset, I opened a Pull\n  Request on git.github.io replacing the occurrences of `[GSoC][PATCH]`\n  by `[GSoC PATCH]`, merged to master;\n\n- Adding a new userdiff driver for INI files, initially target for\n  gitconfig files:\n  https://lore.kernel.org/git/20250331031309.94682-1-lucasseikioshiro@gmail.com/.\n  Currently it is still under revision.\n\nBeyond contributions, I also helped people on the mailing list that\nneeded assistance on Git features and documentation.\n\n## Project Proposal\n\nBased on the information provided in\nhttps://git.github.io/SoC-2025-Ideas/, the goal of this project is to\ncreate a new Git command for querying information from a repository and\nreturning it as a semi-structured data format as a JSON output.\n\nIn the scope of this project, the JSON output will only include data\nthat can currently be retrieved through existing Git commands. The main\nidea is to centralize data from `git rev-parse`, which currently is\noverloaded with features that doesn't fit its main purpose.\n\nSome of the data that we expect to retrieve and centralize are:\n\n- The hashing algorithm (i.e. `sha1` or `sha256`), which currently can\n  be retrieved using `git rev-parse --show-object-format`;\n\n- The Git directory of the repository, currently retrieved by running\n  `git rev-parse --git-dir`;\n\n- The common Git directory, currently retrieved by running\n  `--git-common-dir`;\n\n- The top level directory of the repository, currently retrieved by using\n  `git rev-parse --show-toplevel`;\n\n- The reference database format (i.e. currently, `files` or `reftable`),\n  currently retrieved by running `git rev-parse --show-ref-format`;\n\n- The absolute path of the superproject, currently retrieved by running\n  `git rev-parse --show-superproject-working-tree`;\n\n- Whether this is a bare repository, currently retrieved by running\n  `git --is-bare-repository`;\n\n- Whether this is a shallow repository, currently retrieved by running\n  `git --is-shallow-repository`.\n\nGiven that the information that we want to compile are currently\naccessible with different sets of flags, the user that wants to read\nthem needs to have an advanced knowledge on Git. Once having the\nrepository details consolidated in a single command, the user will be\nable to quickly retrieve what it desires without navigating a complex\ncombination of commands and flags.\n\n### Use cases\n\nSome use cases that will be benefited of this feature will be:\n\n- CLI tools that display formatted information about a Git repository,\n  for example, OneFetch (https://github.com/o2sh/onefetch);\n\n- Text editors, IDEs and plugins that have front-ends for Git, such as\n  Magit (https://magit.vc) or GitLens\n  (https://www.gitkraken.com/gitlens);\n\n- FLOSS repository tracking software, for example,\n  kworkflow (https://github.com/kworkflow),\n  ctracker (https://github.com/quic/contribution-tracker);\n\n- Any other tool that integrates with Git and currently relies on\n  `rev-parse` to get information about the repository;\n\n### Planned features\n\nThis project consists of one big feature: produce the JSON with\nrepository metadata. At first, this should be populated with the data\nthat currently can only be retrieved through `git rev-parse`, like the\nones listed before.\n\nOther data may be added depending on the demands of the Git community\nand the community.\n\nBy now, it's not possible to decide precisely how this command would\nwork without it being more discussed. But a first draft on how it would\nbe invoked and the output that it will produce is:\n\n~~~\n$ git metadata\n\n{\n    \"object-format\": \"sha1\",\n    \"git-dir\": \".git\",\n    \"common-dir\": \".\",\n    \"toplevel\": \"/home/user/my_repo\",\n    \"ref-format\": \"files\",\n    \"superproject-working-tree\": \"/home/user/my_super_repo\",\n    \"bare-repository\": true,\n    \"shallow-repository\": false\n}\n~~~\n\nThis first draft will be sent to the mailing list in order to get\nfeedback from the Git community.\n\n### Development plan\n\nSince this is a new command that is not directly related to any specific\nexistent command, it will probably be placed in a new file inside the\n`builtin` directory.\n\nThe functionality of this command can be divided into two categories:\n\n1. **Data gathering**: retrieving data from different sources, calling\n   existent functions and reading data structures declared in other\n   files;\n\n2. **Data serialization**: formatting the gathered data in a JSON\n   format. This represents two challenges: generating the JSON itself\n   and designing the schema for how the desired data will be presented.\n\nSince the exported data is already provided by other Git commands, it\nprobably won't be difficult to implement this side of the\nfunctionality. The main task would be inspecting the existing codebase\nand find the functions and data structures that will feed our output.\nFor example:\n\n- Reference database format: `git rev-parse` retrieves it from\n  `ref_storage_format_to_name`;\n- Whether the repository is bare: `git rev-parse` retrieves it from the\n  function `is_bare_repository`;\n- Whether the repository is shallow: `git rev-parse` retrieves it from\n  the function `is_repository_shallow`.\n\nDesigning the schema, however, requires special planning, as the\nflexibility of semi-structured data like JSON may lead to early\nbad decisions. A solution may emerge by analysing other software that\nexport JSON as metadata.\n\n### Schedule\n\n1. **Now -- May 5th**: Requirements gathering\n   - Present to the Git community the proposal, asking what are the\n     features that it demands;\n   - Inspect codebases that uses Git (specially `rev-parse`) as data\n     source;\n   - Studying more deeply the codebase, specially the `rev-parse` source\n     code;\n\n2. **May 6th -- June 1st**: Community bonding\n   - Getting in touch with the mentors;\n   - Present to the community a first proposal of the JSON schema;\n   - Receive feedback from the community about the schema;\n   - Present a first proposal on the command line interface;\n   - Receive feedback from the community about the command line\n     interface;\n\n3. **June 2nd -- June 31th**: First coding round\n   - Decide how the CLI should behave, that is, what should be exported\n     by default, what should be outputted only by using flags and what\n     should not be outputted by using disabling flags;\n   - Write a minimal JSON serializer, focusing on export only the\n     top-level object with only string values;\n   - Define a simple data structure that holds only one field of the\n     data that we want to export;\n   - Introduce a first version of the command, outputting the first data\n     structure using the first JSON serializer;\n   - Write test cases in different scenarios comparing the output of the\n     new command with the outputs of the existing commands;\n   - Write a minimal documentation file for the new command, making it\n     explicitly that the command is experimental and it's still under\n     development;\n\n4. **July 1st -- July 14th**: Second coding round\n   - Fill the data structure with all other string fields that should be\n     exported;\n   - Add flags for filtering the data output;\n   - Write tests for each one of the new fields and flags;\n   - Improve the documentation file, explaining what the command does,\n     its output format and what the flags that were implemented so far\n     do;\n\n5. **July 15th -- August 15th**: Third coding round\n   - Improve the JSON serializer, allowing other data types: array,\n     number, boolean, null and nested objects;\n   - Fill the data structure with the remaining non-string fields;\n   - Write flags, tests and documentation for the types that were\n     implemented in this round;\n\n6. **August 16th -- August 25th**: Fourth coding round\n   - Polish the documentation, providing examples on how to use it and\n     notes on why this command was created;\n   - Finish remaining work from the previous rounds that still need to\n     be concluded;\n\n### Availability\n\n2025 is my last year in my master's degree. Currently, I'm not attending\nany classes and I am more focused on developing the software of my\nresearch, performing experiments and writing scientific articles and my\nthesis. Since my advisor is aware that I'm proposing a GSoC project, it\nwill be possible to work on Git while working on my master's tasks.\n\n"}]}