{"thread":{"id":"63244","subject":"[GSoC] Project Proposal: Machine-Readable Repository Information Query Tool","startedAt":"2025-04-02T18:22:27Z","lastAt":"2025-04-04T15:01:33Z","messageCount":5,"participants":["Lucas Seiki Oshiro","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"515526","messageId":"7EB151DA-0BDB-4D54-BBB8-CEE69F51F13A@gmail.com","threadId":"63244","inReplyTo":null,"subject":"[GSoC] Project Proposal: Machine-Readable Repository Information Query Tool","fromName":"Lucas Seiki Oshiro","fromEmail":"lucasseikioshiro@gmail.com","sentAt":"2025-04-02T18:22:11Z","receivedAt":"2025-04-02T18:22:27Z","isPatch":false,"sender":{"key":"lucasseikioshiro@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701580?v=4"},"body":"Hi!\n\nAs you may noticed by the my interactions here, I'm going to send a \nproposal for GSoC 2025!\n\nI'm interested in the project idea currently entitled \"Project Proposal:\nMachine-Readable Repository Information Query Tool\". My main motivations\non why I have chosen this idea is because I think it will be useful for\ninfrastructure teams and FLOSS researchers.\n\nI'm sending here first version of my proposal. I'll be grateful if you\nsend me feedback on it! In this proposal I'm presenting myself again,\nthe possible use cases of this feature, a first idea on how it would\nwork and a activity schedule.\n\nThanks!\n\n---\n\n\n# Machine-Readable Repository Information Query Tool\n\n## Contact info\n\n- Name: Lucas Seiki Oshiro\n- Timezone: GMT-3\n- IRC:\n- GitHub: https://github.com/lucasoshiro\n- LinkedIn: https://www.linkedin.com/in/lucasseikioshiro/\n\n## About me\n\nMy name is Lucas Oshiro, I'm a developer and CS bachelor from São Paulo,\nBrazil. Currently I'm pursuing a master degree in CS at University of São\nPaulo. My interest in Git dates from years ago and I even submitted a\npatch to its codebase in the past, though I couldn't complete it due to\nscheduling conflicts with my capstone project.\n\nHaving experience in the academia, industry and FLOSS, I highly value\ncode quality, code legibility, well-maintained Git histories, unit tests\nand documentation.\n\n### Previous experience with Git\n\nBefore this year, I haven't been involved directly with Git community,\nhowever, I kept my interest in Git alive by:\n\n- Translating the \"Git Internals\" chapter of Pro Git to Brazilian\n  Portuguese: https://github.com/progit/progit2-pt-br/pull/81;\n\n- Writing some blog posts about Git, for example:\n  - one explaining how Git can be used as a debugging tool:\n    https://lucasoshiro.github.io/posts-en/2023-02-13-git-debug/;\n\n  - other explaining how Git merge submodules:\n  https://lucasoshiro.github.io/posts-en/2022-03-12-merge-submodule/;\n\n- Writing a compatible subset of Git in Haskell from scratch:\n https://github.com/lucasoshiro/oshit;\n\n- Helping organizing a Git Introductory Workshop at my University:\n  https://flusp.ime.usp.br/events/git-introductory-workshop/;\n\n- Presenting some lectures about Git in a company that I worked some\n  years ago, covering the Git internals (objects, references, packfile)\n  and debugging and archaeology related Git tools (blame, bisect,\n  pickaxe, ls-files, etc).\n\n### Previous experience with C and open-source\n\nI also have experience with C and some C++. During my CS\ncourse, C was one of the primary languages that I used. I also\nworked with C/C++, for example, in:\n\n- Writing an AMQP message broker from scratch: \n  https://github.com/lucasoshiro/amqp_broker;\n\n- Contributing with simple patches to the IIO subsystem of the Linux\n  kernel: https://lucasoshiro.github.io/floss-en/2020-06-06-kernel_linux/;\n\n- Contributing to the Marlin firmware for 3D printers:\n  https://lucasoshiro.github.io/floss-en/2020-06-30-Marlin/;\n\n- Writing a module for the ns-3 network simulator, dealing with both C\n  and C++ codebases (currently under development, I plan to write a\n  paper and make the code available soon);\n\nDuring my CS course I also was member of FLUSP\n(https://flusp.ime.usp.br), a group in my university focused on FLOSS\ncontributions and from Hardware Livre USP\n(https://hardwarelivreusp.org), another group that was focused on\nworking with open-source hardware.\n\nAs a master's student, I'm one of the Open Science Ambassadors of my\nUniversity (https://cienciaaberta.usp.br/sobre-o-projeto/, in\nPortuguese), promoting the Open Science principles, which include\nopen-source software, in the unit where I study.\n\nI also contributed to some other free/open-source software, which I list\nhere: https://lucasoshiro.github.io/floss-en/\n\n### Activity in the Git community in 2025\n\nSince when I decided to submit a proposal for GSoC, I sent some patches\nto the Git codebase and git.github.io:\n\n- My microproject, replacing some `test -f` by `test_path_is_file`:\n  https://lore.kernel.org/git/20250208165731.78804-1-lucasseikioshiro@gmail.com/;\n\n- Adding a paragraph to the merge-strategies documentation describing how\n  Git merges submodules (based on the blog post that I mentioned\n  before):\n  https://lore.kernel.org/git/20250227014406.20527-1-lucasseikioshiro@gmail.com/;\n  \n- A patchset adding a new `--subject-extra-prefix` flag for `git\n  format-patch`, allowing the user to quickly prepend tags like [GSoC],\n  [Newbie] or [Outreachy] to the beginning of the subject. This patchset\n  was rejected in favor of just using `--subject-prefix='GSoC PATCH'` or\n  similar. It can be seen here:\n  https://lore.kernel.org/git/20250303220029.10716-1-lucasseikioshiro@gmail.com/;\n\n- Given the feedback on the previous rejected patchset, I opened a Pull\n  Request on git.github.io replacing the occurrences of `[GSoC][PATCH]`\n  by `[GSoC PATCH]`;\n  \n- Adding a new userdiff driver for INI files, initially target for\n  gitconfig files. Currently it is still under revision:\n  https://lore.kernel.org/git/20250331031309.94682-1-lucasseikioshiro@gmail.com/.\n\nBeyond contributions, I also helped people on the mailing list that\nneeded assistance on Git documentation.\n\n## Project Proposal\n\nBased on the information provided in\nhttps://git.github.io/SoC-2025-Ideas/, the goal of this project is to\ncreate a new Git command for querying information from a repository and\nreturning it as a semi-structured data format as a JSON output.\n\nIn the scope of this project, the JSON output will only include data\nthat can currently be retrieved through existing Git commands, for\nexample:\n\n- `git branch`: information about branches, such as the commit that each\n  branch currently references and their upstreams;\n\n- `git tag`: information about the tags, such as the author or commit\n  date and the messages they hold (in the case of annotated tags);\n\n- `git remote`: the URL of each remote;\n\n- `git log`: statistics about the commit history, such of the\n  distribution of commits over time and by author, the distribution of\n  lines changed by each author;\n\n- `git submodule`: information about the submodules, mainly the commits\n  that they are referencing and their remote URLs;\n\n- `git rev-parse`: the current branch name, the current commit, the path\n  of the repository top level directory, if the repository is a bare\n  repository or if the repository is under bisection.\n\nGiven that the information that we want to compile are currently\naccessible only through different commands with different sets of flags,\nthe user that wants to read them needs to have an advanced knowledge on\nGit. Once having the repository details consolidated in a single\ncommand, the user will be able to quickly retrieve what it desires\nwithout navigating a complex combination of commands and flags.\n\n### Use cases\n\nSome use cases that will be benefited of this feature will be:\n\n- CLI tools that display formatted information about a Git repository,\n  for example, OneFetch (https://github.com/o2sh/onefetch);\n\n- Text editors, IDEs and plugins that have front-ends for Git, such as\n  Magit (https://magit.vc) or GitLens (https://www.gitkraken.com/gitlens);\n\n- FLOSS repository tracking software, for example,\n  kworkflow (https://github.com/kworkflow),\n  ctracker (https://github.com/quic/contribution-tracker);\n\n- Academic researchers on FLOSS projects that need statistics on the\n  repositories that they are querying;\n\n- Continuous integration workflows that perform checks on the\n  repository before allowing a branch to be merged into another or\n  before a deploy;\n\n- Code quality tools that will be able to inspect the health of the\n  commit history.\n\n### Planned features\n\nSince the features haven't been defined yet, this will need to be\nplanned after surveying people and projects that potentially will use\nthat:\n\n- Searching on code hosting tools (e.g. GitHub, GitLab) for open-source\n  software that retrieve data from Git and what they do with them;\n  \n- Contacting people in academia that use Git repositories as data\n  sources for their researches and find out what valuable information\n  this command can provide them;\n  \n- Contacting people from the industry, specially in infrastructure teams\n  to understand the challenges they face when retrieving data from Git.\n  \nGiven that I have worked in a infrastructure team and that I have\ncolleagues and professors at the university that currently research\nFLOSS software and communities, I have contacts that can provide input\non what should be considered when developing this new command.\n\nBy now, it's not possible to decide how exactly this command would work,\nbut a first draft is this (supposing that `metadata` is the name of the\ncommand and `--submodule` is a flag that enable the submodule metadata):\n\n~~~\n$ git metadata --submodule\n\n{\n  \"symbolic_refs\": {\n    \"HEAD\": \"main\"\n  },\n  \"branches\": [\n    {\n      \"name\": \"main\",\n      \"commit_id\": \"ac72c22f3c8a9280c81171ccc6cedff3171344cf\",\n      \"remote\": \"origin/main\"\n    },\n    {\n      \"name\": \"feature\",\n      \"commit_id\": \"1e373e02767337bd6b996da6598eed822a805878\",\n      \"remote\": \"fork/feature\"\n    }\n  ],\n  \"tags\": [\n    {\n      \"name\": \"v1.0\",\n      \"message\": \"First version\",\n      \"author_timestamp\": \"1743554265\",\n      \"commiter_timestamp\": \"1743554265\"\n    }\n  ],\n  \"remotes\": [\n    {\n      \"name\": \"origin\",\n      \"url\": \"https://example.com/foo\"\n    },\n    {\n      \"name\": \"fork\",\n      \"url\": \"user@example.com/foo\"\n    }\n  ],\n  \"submodules\": [\n    {\n      \"path\": \"my_dir/my_submodule_dir\",\n      \"url\": \"https://example.com/bar\",\n      \"commit_id\": \"94436069f106c0014897b1c93e8fc3e49c8fc156\"\n    }\n  ]\n}\n~~~\n\n### Development plan\n\nSince this is a new command that is not directly related to any specific\nexistent command, it will probably be placed in a new file inside the\n`builtin` directory.\n\nThe functionality of this command can be divided into two categories:\n\n1. **Data gathering**: retrieving data from different sources, calling\n   existent functions and reading data structures declared in other\n   files;\n\n2. **Data serialization**: formatting the gathered data in a JSON\n   format. This represents two challenges: generating the JSON itself\n   and designing the schema for how the desired data will be presented.\n   \nSince the exported data is already provided by other Git commands, it\nprobably won't be difficult to implement this side of the\nfunctionality. The main task would be inspecting the existing codebase\nand find the functions and data structures that will feed our output.\n\nDesigning the schema, however, requires special planning, as the\nflexibility of semi-structured data like JSON may lead to early\nbad decisions. A solution may emerge by analysing other software that\nexport JSON as metadata.\n\n### Schedule\n\n1. **Now -- May 5th**: Requirements gathering\n   - Inspect codebases that uses Git as data sources; \n   - Contacting academic researchers on FLOSS;\n   - Contacting industry infrastructure professionals;\n\n2. **May 6th -- June 1st**: Community bonding\n   - Getting in touch with the mentors;\n   - Present to the community a first proposal of the JSON schema;\n   - Receive feedback from the community about the schema;\n   - Present a first proposal on the command line interface;\n   - Receive feedback from the community about the command line\n     interface;\n\n3. **June 2nd -- July 14th**: First coding round\n   - Write data structures that correspond to the presented JSON schema;\n   - Fill the data structures with data obtained from routines of the\n     existing codebase;\n\n4. **July 15th -- August 25th**: Second coding round\n   - Implementing the command line interface option handlers;\n   - Write the JSON serializer.\n\n### Availability\n\n2025 is my last year in my master's degree. Currently, I'm not attending\nany classes and I am more focused on developing the software of my\nresearch, performing experiments and writing scientific articles and my\nthesis. Since my advisor is aware that I'm proposing a GSoC project, it\nwill be possible to work on Git while working on my master's tasks.\n\n"},{"id":"515574","messageId":"Z-5famP3CgaSfDc2@pks.im","threadId":"63244","inReplyTo":"7EB151DA-0BDB-4D54-BBB8-CEE69F51F13A@gmail.com","subject":"Re: [GSoC] Project Proposal: Machine-Readable Repository Information Query Tool","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-03T10:14:02Z","receivedAt":"2025-04-03T10:14:10Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 02, 2025 at 03:22:11PM -0300, Lucas Seiki Oshiro wrote:\n> ### Activity in the Git community in 2025\n> \n> Since when I decided to submit a proposal for GSoC, I sent some patches\n> to the Git codebase and git.github.io:\n> \n> - My microproject, replacing some `test -f` by `test_path_is_file`:\n>   https://lore.kernel.org/git/20250208165731.78804-1-lucasseikioshiro@gmail.com/;\n> \n> - Adding a paragraph to the merge-strategies documentation describing how\n>   Git merges submodules (based on the blog post that I mentioned\n>   before):\n>   https://lore.kernel.org/git/20250227014406.20527-1-lucasseikioshiro@gmail.com/;\n>   \n> - A patchset adding a new `--subject-extra-prefix` flag for `git\n>   format-patch`, allowing the user to quickly prepend tags like [GSoC],\n>   [Newbie] or [Outreachy] to the beginning of the subject. This patchset\n>   was rejected in favor of just using `--subject-prefix='GSoC PATCH'` or\n>   similar. It can be seen here:\n>   https://lore.kernel.org/git/20250303220029.10716-1-lucasseikioshiro@gmail.com/;\n> \n> - Given the feedback on the previous rejected patchset, I opened a Pull\n>   Request on git.github.io replacing the occurrences of `[GSoC][PATCH]`\n>   by `[GSoC PATCH]`;\n>   \n> - Adding a new userdiff driver for INI files, initially target for\n>   gitconfig files. Currently it is still under revision:\n>   https://lore.kernel.org/git/20250331031309.94682-1-lucasseikioshiro@gmail.com/.\n> \n> Beyond contributions, I also helped people on the mailing list that\n> needed assistance on Git documentation.\n\nCould you please also amend the status (merged to master, merged to\nnext, under discussion) for each of these items?\n\n> ## Project Proposal\n> \n> Based on the information provided in\n> https://git.github.io/SoC-2025-Ideas/, the goal of this project is to\n> create a new Git command for querying information from a repository and\n> returning it as a semi-structured data format as a JSON output.\n> \n> In the scope of this project, the JSON output will only include data\n> that can currently be retrieved through existing Git commands, for\n> example:\n> \n> - `git branch`: information about branches, such as the commit that each\n>   branch currently references and their upstreams;\n> \n> - `git tag`: information about the tags, such as the author or commit\n>   date and the messages they hold (in the case of annotated tags);\n> \n> - `git remote`: the URL of each remote;\n> \n> - `git log`: statistics about the commit history, such of the\n>   distribution of commits over time and by author, the distribution of\n>   lines changed by each author;\n> \n> - `git submodule`: information about the submodules, mainly the commits\n>   that they are referencing and their remote URLs;\n> \n> - `git rev-parse`: the current branch name, the current commit, the path\n>   of the repository top level directory, if the repository is a bare\n>   repository or if the repository is under bisection.\n> \n> Given that the information that we want to compile are currently\n> accessible only through different commands with different sets of flags,\n> the user that wants to read them needs to have an advanced knowledge on\n> Git. Once having the repository details consolidated in a single\n> command, the user will be able to quickly retrieve what it desires\n> without navigating a complex combination of commands and flags.\n\nI already noticed in another proposal, but it seems a bit like the idea\nis underspecced. The idea isn't to make _all_ information about the\nrepository accessible. It's rather that we want to give a better home to\ninformation about the underlying repository itself. To clarify further,\nI'm talking about information like:\n\n  - Which object hash does the repository use?\n  - What is the ref database format?\n  - Where is the Git directory?\n  - Where is the common directory?\n  - What is the top-level directory?\n\nThis kind of information is exposed via git-rev-parse(1) already, see\nthe section \"Options for Files\". But git-rev-parse(1) is not really a\ngood match at all given that its main intent is to parse revisions. Over\ntime though it developed into a kind of grab-bag of different unrelated\nfunctionality that we didn't really have a nice home for elsewhere.\n\n> ### Development plan\n> \n> Since this is a new command that is not directly related to any specific\n> existent command, it will probably be placed in a new file inside the\n> `builtin` directory.\n> \n> The functionality of this command can be divided into two categories:\n> \n> 1. **Data gathering**: retrieving data from different sources, calling\n>    existent functions and reading data structures declared in other\n>    files;\n> \n> 2. **Data serialization**: formatting the gathered data in a JSON\n>    format. This represents two challenges: generating the JSON itself\n>    and designing the schema for how the desired data will be presented.\n>    \n> Since the exported data is already provided by other Git commands, it\n> probably won't be difficult to implement this side of the\n> functionality. The main task would be inspecting the existing codebase\n> and find the functions and data structures that will feed our output.\n> \n> Designing the schema, however, requires special planning, as the\n> flexibility of semi-structured data like JSON may lead to early\n> bad decisions. A solution may emerge by analysing other software that\n> export JSON as metadata.\n> \n> ### Schedule\n> \n> 1. **Now -- May 5th**: Requirements gathering\n>    - Inspect codebases that uses Git as data sources; \n>    - Contacting academic researchers on FLOSS;\n>    - Contacting industry infrastructure professionals;\n> \n> 2. **May 6th -- June 1st**: Community bonding\n>    - Getting in touch with the mentors;\n>    - Present to the community a first proposal of the JSON schema;\n>    - Receive feedback from the community about the schema;\n>    - Present a first proposal on the command line interface;\n>    - Receive feedback from the community about the command line\n>      interface;\n> \n> 3. **June 2nd -- July 14th**: First coding round\n>    - Write data structures that correspond to the presented JSON schema;\n>    - Fill the data structures with data obtained from routines of the\n>      existing codebase;\n> \n> 4. **July 15th -- August 25th**: Second coding round\n>    - Implementing the command line interface option handlers;\n>    - Write the JSON serializer.\n\nI generally recommend students to take on smaller batches of work that\ncan be submitted individually. The way it is structured now means that\nyou will end up with a single deliverable at the end of your project.\nBut structuring the project like that introduces a high risk that you\nwon't be able to land anything until the end of your project in case\nthere is a bigger discussion around parts of these patches.\n\nInstead, it would make sense to identify smaller batches of work that\nare self-contained enough to be submitted upstream. This ensures that\nyou get early feedback and that you can iterate on your design as early\nas possible in the project.\n\nPatrick\n"},{"id":"515601","messageId":"BC393559-6A33-447D-9AE8-BB2D9D0BC0D6@gmail.com","threadId":"63244","inReplyTo":"Z-5famP3CgaSfDc2@pks.im","subject":"Re: [GSoC] Project Proposal: Machine-Readable Repository Information Query Tool","fromName":"Lucas Seiki Oshiro","fromEmail":"lucasseikioshiro@gmail.com","sentAt":"2025-04-03T18:02:51Z","receivedAt":"2025-04-03T18:03:06Z","isPatch":false,"sender":{"key":"lucasseikioshiro@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701580?v=4"},"body":"\n> Could you please also amend the status (merged to master, merged to\n> next, under discussion) for each of these items?\n\nOk! This may change until the GSoC submission deadline, but by now:\n\n- Microproject: merged to master\n- Merge Documentation: merged to master\n- Extra prefix flag: closed\n- Fix on git.github.io <http://git.github.io/>: merged to master\n- Driver for INI files: under discussion\n\n> This kind of information is exposed via git-rev-parse(1) already, see\n> the section \"Options for Files\".\n\nThanks for your clarification! But still, I was discussing with people\nhere at my university who deals directly with research on FLOSS\nrepositories, is it worth to find other uses cases for this new command?\n\n> I generally recommend students to take on smaller batches of work that\n> can be submitted individually. The way it is structured now means that\n> you will end up with a single deliverable at the end of your project.\n> But structuring the project like that introduces a high risk that you\n> won't be able to land anything until the end of your project in case\n> there is a bigger discussion around parts of these patches.\n\nI see... After sending this first version I was thinking about it and\nit would be hard to test the functionalities without having the JSON\nserializer minimally working. I'll send another version, proposing\nsmaller batches of work and writing a minimal serializer at the\nbeginning of the project, improving with the new features.\n\n> Patrick\n\nThanks for your time!\n\n"},{"id":"515644","messageId":"Z--h3rhY5XAERVHm@pks.im","threadId":"63244","inReplyTo":"BC393559-6A33-447D-9AE8-BB2D9D0BC0D6@gmail.com","subject":"Re: [GSoC] Project Proposal: Machine-Readable Repository Information Query Tool","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-04T09:09:50Z","receivedAt":"2025-04-04T09:09:55Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 03, 2025 at 03:02:51PM -0300, Lucas Seiki Oshiro wrote:\n> > This kind of information is exposed via git-rev-parse(1) already, see\n> > the section \"Options for Files\".\n> \n> Thanks for your clarification! But still, I was discussing with people\n> here at my university who deals directly with research on FLOSS\n> repositories, is it worth to find other uses cases for this new command?\n\nIt doesn't hurt to think about additional usecases, sure. But one of the\nthings that I want to caution against is that we now create the next\n\"grab bag\" of unrelated features under a common name. After all, we are\ntrying to fix exactly this state in git-rev-parse(1). So the intent of\nthe command should be clearly defined to avoid this.\n\nPatrick\n"},{"id":"515673","messageId":"A951AB6E-2CFD-4849-BF93-3DC34EC3637A@gmail.com","threadId":"63244","inReplyTo":"Z--h3rhY5XAERVHm@pks.im","subject":"Re: [GSoC] Project Proposal: Machine-Readable Repository Information Query Tool","fromName":"Lucas Seiki Oshiro","fromEmail":"lucasseikioshiro@gmail.com","sentAt":"2025-04-04T15:01:18Z","receivedAt":"2025-04-04T15:01:33Z","isPatch":false,"sender":{"key":"lucasseikioshiro@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701580?v=4"},"body":"\n> So the intent of the command should be clearly defined to avoid this.\n\nThanks, Patrick! I'll send a more focused v2 soon.\n\n"}]}