{"thread":{"id":"66786","subject":"[Outreachy] Proposal draft v1: Improve how command arguments and options are scanned and parsed","startedAt":"2026-11-05T19:57:30Z","lastAt":"2026-11-09T15:09:10Z","messageCount":3,"participants":["Tanga Joris"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"557613","messageId":"20261105195730.1518433-1-tangajoris@gmail.com","threadId":"66786","inReplyTo":null,"subject":"[Outreachy] Proposal draft v1: Improve how command arguments and options are scanned and parsed","fromName":"Tanga Joris","fromEmail":"tangajoris@gmail.com","sentAt":"2026-11-05T19:57:30Z","receivedAt":"2026-11-05T19:57:30Z","isPatch":false,"sender":{"key":"tangajoris@gmail.com","avatar":null},"body":"Hi everyone,\n\nThis is the first draft of my proposal for the project \"Improve how\ncommand arguments and options are scanned and parsed\" for the\nDecember 2026 Outreachy round. I would be grateful for feedback,\nespecially on sections 4 and 5.\n\nIn short: before a command does any real work, it has to answer\nrequests like \"-h\", \"--help-all\" and \"--git-completion-helper\". Today\nthat happens in three places that do not agree. On master, all 120\nbuiltins that use parse-options answer \"-h\" outside a repository,\nbut only 31 of them answer \"--git-completion-helper\" there. I would\nlike to make these entry points behave the same for every command,\nguard that with tests, and then move the early scanning of options\ntowards parse-options itself.\n\n\n1. About me\n-----------\nName: Tanga Joris\nPronouns: he/him\nEmail: tangajoris@gmail.com\nGitHub: https://github.com/tmjoris\nBlog: https://tmjoris.github.io\nLinkedIn: https://www.linkedin.com/in/joris-tanga\nLocation: Nairobi, Kenya (EAT, UTC+3)\n\nI also meet all the Outreachy eligibility rules.\n\nI am a curious developer who loves playing around with systems.\nIdentifying and solving technical problems is my jam, because it\ngives me the sense that I am making an impact somewhere. I am about\nto graduate from the United States International University - Africa,\nwhere I study Applied Computer Technology. I've had the luxury of\nplaying a lot with C during my time learning at ALX, where it was my\nintroduction to programming.\n\nI first met this kind of parsing problem in my own project, kiln\n(https://github.com/tmjoris/kiln), which is a Go CLI and daemon that\nruns code in Firecracker or gVisor sandboxes. Its main() parses\nthe global flags with flag.Parse(), which stops at the subcommand,\nand each subcommand parses the rest with its own flag.FlagSet. So\n\"kiln -addr host:7070 run\" works, but \"kiln run -addr host:7070\"\nfails with \"flag provided but not defined: -addr\". Git has the same\nquestion of who looks at an argument first, and what they do with it.\n\nAvailability: 40 hours a week from 7 December 2026 to 8 March 2027.\nI have no classes, exams or other work during the internship.\nNairobi is UTC+3, so my working day overlaps with European hours.\n\n\n2. My Git work so far\n---------------------\nMicroproject: checkout: use Q_() for messages with counts\n  https://lore.kernel.org/git/20261022004834.246892-1-tangajoris@gmail.com/\n  Topic tj/checkout-orphan-more-plural, merged to master in\n  a04194e134. v2 reworded the log message as Junio suggested and\n  also converted \"only one reference expected, %d given.\" after\n  Tuomas's comment.\n\nSeries: drop NO_PARSEOPT from builtins that use parse-options\n  v1: https://lore.kernel.org/git/20261024052449.95869-1-tangajoris@gmail.com/\n  v4: https://lore.kernel.org/git/20261105003914.708836-1-tangajoris@gmail.com/\n  Topic tj/drop-no-parseopt.\n\n  patch-id, diff-pairs, fast-import and var use parse-options but\n  were still marked NO_PARSEOPT, so bash completion never asked them\n  for their options. René noticed that once fast-import was\n  unmarked, \"git fast-import --<TAB>\" would wait for input, because\n  fast-import reads its \"feature\" lines before it parses options. My\n  v2 special-cased that in fast-import. Junio found it \"way too ugly\n  to target this so specifically\" and suggested teaching\n  show_usage_with_options_if_asked() to answer the completion helper,\n  which v3 does. In v4, Kaartic's review added a pointer to the next\n  commit, a comment in parse-options.h, and a t0012 test that catches\n  builtins that use parse-options but are still marked NO_PARSEOPT.\n  Cynthia showed that my test did not cover the \"-all\" form. When I\n  tried her test I found it still passed with the fix reverted, so I\n  added the leftover tmp_pack_* check to it.\n\n  The lesson I took from Junio's comment is the one this proposal is\n  built on: when a command has to answer something before it starts\n  working, the answer belongs in the shared place every command\n  already goes through, not in one command.\n\nPatch: annotate: pass the arguments to blame unchanged\n  https://lore.kernel.org/git/20261105073101.1449292-1-tangajoris@gmail.com/\n  cmd_annotate() put \"-c\" in front of the user's arguments, so\n  \"--git-completion-helper\" was no longer the lone argument and the\n  diff option parser answered it instead. \"git annotate --<TAB>\"\n  offered the options of \"git diff\", without --porcelain. I found it\n  while trying to make the completion helper work outside a\n  repository (section 4.A): with only that git.c change,\n  \"git annotate --git-completion-helper\" segfaults there.\n  Junio: \"Nice simplification.\" ... \"Looking good. Thanks.\"\n\nBug triage: \"Git crashed on `git apply --check`\"\n  https://lore.kernel.org/git/CALdwsC1cMKZgRPZA+pBaJe4PPs9q9W0K9CQLmM2bgUCyysTB-w@mail.gmail.com/\n  I reproduced the assertion failure and found the trigger in the\n  reporter's file: a blank line after the \"diff --git\" line of a\n  new-file section. An indented \"new file mode\" line and a\n  \"--- /dev/null\" before \"new file mode\" crash it too. All three still\n  crash on master; the fix is my next patch.\n\nReviews:\n  rev-list: skip --grep argument in early scan (Azuama), v3\n  https://lore.kernel.org/git/CAKrRMNrnd2+HxTxadKaha+K9ASWi_oN4h7gdjVYAGmyODwmi=w@mail.gmail.com/\n  The tests fail without the fix, but --author, --committer and\n  --grep-reflog still have the same problem, and the NEEDSWORK\n  example above the loop becomes stale.\n\n  promisor-remote: honor promisor.fetchOrder when trying remotes (Nelly)\n  https://lore.kernel.org/git/CAKrRMNpAW+f2XXMFqpB8ffZsTdyPsaFtHe8dku=JVXACp1M3Hw@mail.gmail.com/\n  Setting the new variable makes \"git show\" in a partial clone die\n  with a BUG() in string-list.c, because an in-place split is used on\n  a list that duplicates its strings.\n\n  hash: handle invalid algorithms when formatting object IDs\n  (shrimech, Rust)\n  https://lore.kernel.org/git/CAKrRMNrLvPck5psPpKX1eWe26To5srfFSLFObZ4zTX09RE551w@mail.gmail.com/\n  \"cargo test\" passes and the new test panics without the fix. I\n  asked whether algo 0 should be printed the way the C code does it.\n\n  config: move pack size limit into repo config (Joy), v3\n  https://lore.kernel.org/git/CAKrRMNqzOX=vUUMrMf5P_Atar8_qCSbGdtBktfT+iL4hJfFX-Q@mail.gmail.com/\n  2/2 adds a member to struct repo_config_values, which Junio's\n  jc/freeze-repo-config-values asks not to do for now, and v3\n  conflicts with cc/pack-index-v3 in 'seen'.\n\nBlog: https://tmjoris.github.io (\"From git config to git\nsend-email\", about my first weeks on the list).\n\n\n3. Where commands look at their arguments today\n-----------------------------------------------\nBefore a builtin's real option parsing runs, its arguments can be\nlooked at in three places:\n\n  1. git.c, run_builtin(): decides whether to set up the repository.\n     A lone \"-h\" or \"--help-all\" turns RUN_SETUP into\n     RUN_SETUP_GENTLY, so \"git cmd -h\" works outside a repository.\n  2. show_usage_with_options_if_asked(): called by 15 builtins that\n     must answer before doing any work (fast-import reads its input,\n     others need a repository). With my series it answers -h,\n     --help-all and both completion helpers.\n  3. parse_options() itself answers them for everyone else, but only\n     when the request is the lone argument.\n\nThen there are the early scans: commands that read a few options\nbefore the real parse, such as fast-import's\n--allow-unsafe-features, column's --command, and the hand-written\nloop in rev-list (the one Azuama's patch touches, whose NEEDSWORK\ncomment calls it \"utterly broken\"). Christian's cc/early-scan-options\ngives them early_scan_options(), which reads the same struct option\narray as parse_options(). Junio's view on its first round is the\ndirection I want to follow: \"in the long term we would rather see a\nproperly refactored parse-options machinery perform the prescan\n(perhaps with some kind of 'dry-run' option given to the machinery)\nthan yet another ad-hoc parser\".\n\nJunio was also clear, on the diff-files conversion, that converting\na parser is not worth it on its own for plumbing \"meant for use by\nscripts\". So in this proposal every change has to fix something a\nuser can see, or remove a place where two parsers can disagree.\n\n\n4. Plan\n-------\nA. Make the entry points agree (weeks 1-4)\n   - Teach git.c to set up gently for --git-completion-helper and\n     --git-completion-helper-all, as 129b3632f3 did for --help-all,\n     so that completion works outside a repository. This needs my\n     NO_PARSEOPT series and the annotate fix first: with the git.c\n     change alone, annotate, am, ls-files and checkout-index\n     segfault outside a repository, and branch, commit, merge,\n     rebase, status and update-index hit a BUG(). With all three,\n     all 124 parse-options builtins answer there. Add the case to\n     the per-builtin loop in t1517-outside-repo.sh.\n   - Look for other commands that rewrite their argv before handing\n     it to another cmd_*(), the way annotate did.\n   - Turn the checks I ran by hand into tests: t1517 for \"outside a\n     repository\", t0012 for \"parse-options users are not marked\n     NO_PARSEOPT\", and a check that no helper reads stdin.\n\nB. Conversions only where there is a concrete reason (weeks 4-8)\n   - merge-index: -o and -q are only recognised in that order. On\n     master:\n       git merge-index -o -q cat -a   -> works\n       git merge-index -q -o cat -a   -> fatal: git merge-index: cat\n                                         not in the cache\n     Parse the leading options with parse-options and\n     PARSE_OPT_STOP_AT_NON_OPTION, and drop its NO_PARSEOPT mark.\n     Tests in t6060.\n   - Early scans that disagree with the real parse, like the\n     rev-list loop above, but only with a reproduced user-facing bug.\n     column, bisect and check-ref-format are in Cynthia's plan, so I\n     would not touch them.\n\nC. Towards parse-options doing the prescan (weeks 8-13, RFC first)\n   Christian's series already lets the scan and the real parser look\n   options up the same way. The scan still stops at short options and\n   at options with low-level callbacks, because it cannot run them.\n   I would like to study what a \"dry run\" mode of parse_options()\n   would need (no writes through opt->value, no callbacks, no die()),\n   send that to the list as an RFC, and implement the first step only\n   if the design is agreed. Otherwise it becomes the final report's\n   list of open problems.\n\nEvery change is tested first: the test must fail on master, every new\nbranch must make a test fail when I break it on purpose, and output\nand exit codes are compared with master. That is how I found the\nannotate bug.\n\n\n5. Timeline\n-----------\nNow to 6 Dec   Land tj/drop-no-parseopt and the annotate fix. Send\n               the git apply fix. Keep reviewing parse-options patches.\nWeeks 1-2      git.c gentle setup for the completion helper, with the\n               t1517 test. First blog post.\nWeeks 3-4      Turn the outside-repository checks into tests; look\n               for other argv-rewriting wrappers like annotate.\nWeeks 5-8      merge-index; early-scan fixes with a reproduced bug.\nWeeks 8-11     \"dry run\" study and RFC on the list.\nWeeks 11-13    First step of the RFC if agreed, otherwise the open\n               problems report. Rerolls, final blog post.\nEvery week     A blog post and a written update to the mentors.\n\nIf I fall behind, I drop C's implementation first (keeping the RFC),\nthen B's early-scan items.\n\n\n6. Communication\n----------------\nDesign questions go to the list before code. I answer every review\ncomment on the list before sending another version, as Kaartic\nreminded me to. I will blog every week at https://tmjoris.github.io,\nwhich anyone can read without an account, and link each post on the\nlist.\n\n\n7. After the internship\n-----------------------\nI want to keep working on whatever is left in the final report,\nespecially the parse-options dry run, keep the entry-point tests up\nto date as commands change, and keep reviewing parse-options patches.\nI would also like to help the next group of applicants the way\nKaartic, René, Junio and Cynthia helped me.\n\n\n8. References\n-------------\n[1] Microproject: https://lore.kernel.org/git/20261022004834.246892-1-tangajoris@gmail.com/\n[2] tj/drop-no-parseopt v1 and René's review:\n    https://lore.kernel.org/git/20261024052449.95869-1-tangajoris@gmail.com/\n    https://lore.kernel.org/git/8b60c7f6-4a1c-495a-a594-aaf322279539@web.de/\n[3] v2 1/2 and Junio's suggestion:\n    https://lore.kernel.org/git/20261029012616.139783-1-tangajoris@gmail.com/\n    https://lore.kernel.org/git/xmqqy0bg2xj5.fsf@gitster.g/\n[4] v3, Kaartic's and Cynthia's reviews, and v4:\n    https://lore.kernel.org/git/20261029231621.48958-1-tangajoris@gmail.com/\n    https://lore.kernel.org/git/123de4e4-3387-4eef-8a05-9ed1ad1fce30@gmail.com/\n    https://lore.kernel.org/git/d026d6a0-5b1b-4bbf-bd1f-5f41c9c2a4e4@gmail.com/\n    https://lore.kernel.org/git/20261103092503.14872-1-herroyalpriestess@gmail.com/\n    https://lore.kernel.org/git/20261105003914.708836-1-tangajoris@gmail.com/\n[5] Junio on the early-scan v1 cover (long-term direction):\n    https://lore.kernel.org/git/xmqqpkyviizc.fsf@gitster.g/\n[6] Junio on diff-files v6 (value of converting plumbing):\n    https://lore.kernel.org/git/xmqq8q3blfh1.fsf@gitster.g/\n[7] 129b3632f3 (builtin: also setup gently for --help-all, 2025-08-03)\n[8] cc/early-scan-options v4:\n    https://lore.kernel.org/git/20261102151447.734680-6-christian.couder@gmail.com/\n\nThanks,\nTanga\n\n"},{"id":"557846","messageId":"20261108175038.946313-1-tangajoris@gmail.com","threadId":"66786","inReplyTo":"20261105195730.1518433-1-tangajoris@gmail.com","subject":"[Outreachy] Proposal draft v2: Improve how command arguments and options are scanned and parsed","fromName":"Tanga Joris","fromEmail":"tangajoris@gmail.com","sentAt":"2026-11-08T17:50:38Z","receivedAt":"2026-11-08T17:50:38Z","isPatch":false,"sender":{"key":"tangajoris@gmail.com","avatar":null},"body":"Hi everyone,\n\nThis is the second draft of my proposal for \"Improve how command\narguments and options are scanned and parsed\" [1].\n\nChanges since v1:\n - v1 was mostly about making the entry points agree. The project\n   page asks for converting hand-written parsing and improving the\n   API where needed, so I rebuilt the plan around that.\n - Added the commands I plan to work on, with what I saw when I ran\n   them on master.\n - Moved the completion helper work to before the internship.\n - The plan has three items: stash show, the number fixes, and \"git\n   diff\". For \"git diff\" (3B), I ask the mentors which way to go.\n - Added section 4 with other places I found, like git daemon and\n   the revision options.\n - Shortened everything else.\n\nI would be grateful for feedback on section 3, and on the questions\nin section 7. I plan to start with stash show. If someone is already\nworking on one of these commands, please tell me and I will pick\nanother one.\n\n\n1. About me\n-----------\nName: Tanga Joris\nPronouns: he/him\nEmail: tangajoris@gmail.com\nGitHub: https://github.com/tmjoris\nBlog: https://tmjoris.github.io\nLocation: Nairobi, Kenya (EAT, UTC+3)\n\nI meet all the Outreachy eligibility rules.\n\nI am about to graduate from the United States International\nUniversity - Africa, where I study Applied Computer Technology. I\nlearned C at ALX, where it was my introduction to programming.\n\nI had done some small parsing work before, in my own project, kiln,\nand it made me curious about how much goes on behind something as\nseemingly simple as parsing command-line arguments. I enjoy that kind\nof work: starting with something that looks straightforward,\nfollowing the implementation, and gradually understanding the\ndecisions and assumptions behind it.\n\nkiln (https://github.com/tmjoris/kiln) is a Go CLI that runs code in\nsandboxes. \"kiln -addr host:7070 run\" works, but \"kiln run -addr\nhost:7070\" fails. This is because main() and each subcommand parse\ntheir flags separately. Git has the same question of who looks at an\nargument first, and what they do with it.\n\nAvailability: 40 hours a week from 7 December 2026 to 8 March 2027.\nI have no classes, exams or other work during the internship.\nNairobi is UTC+3, so my working day overlaps with European hours.\n\n\n2. My Git work so far\n---------------------\n- Microproject: checkout: use Q_() for messages with counts [2].\n  Merged to master in a04194e134.\n- tj/drop-no-parseopt, v5 [3]. Four builtins used parse-options but\n  were still marked NO_PARSEOPT, so completion never asked them for\n  their options. The topic removes the flag from them, and adds a\n  t0012 test that catches such builtins. Junio: \"OK. Let's mark the\n  topic as ready for 'next'.\"\n- tj/annotate-blame-args [4]. I found it while trying to make\n  --git-completion-helper work outside a repository. cmd_annotate()\n  put \"-c\" in front of the user's arguments, so the helper option was\n  no longer the only argument. The diff option parser answered it\n  instead, and \"git annotate --<TAB>\" offered the options of\n  \"git diff\". The \"-c\" was not needed, so the fix deletes code.\n  Junio: \"Nice simplification.\"\n  Both topics are marked \"Will merge to 'next'?\" in What's cooking\n  [5].\n- apply: a fix for the \"git apply --check\" crash reported on the\n  list [6]. While looking into it, I found two more ways to trigger\n  the same assertion.\n- Reviews:\n  - Azuama's rev-list early scan fix [7]. The tests failed without\n    the fix, but --author, --committer and --grep-reflog had the\n    same problem. Azuama's v4 skips their values too, and its\n    NEEDSWORK comment now uses my \"--author --missing-only\" example\n    [8]. Junio then asked for a different approach than a list of\n    options in rev-list.c.\n  - Joy's pack size limit patch [9]. 2/2 added a member to struct\n    repo_config_values, which Junio had asked people not to do for\n    now. So I asked whether 1/2 could be sent on its own, and Joy\n    sent it as a separate patch [10].\n  - Nelly's promisor.fetchOrder patch [11]. Setting the new variable\n    made \"git show\" in a partial clone die with a BUG() in\n    string-list.c, because an in-place split was used on a list that\n    duplicates its strings.\n  - shrimech's Rust patch for formatting object IDs [12]. \"cargo\n    test\" passed and the new test failed without the fix. I asked\n    whether algo 0 should be printed the way the C code does it.\n    brian explained that algo 0 coming from C is meant to panic.\n\n\n3. Plan\n-------\nBefore the internship, I plan to land my two topics. If there is\ntime, I would also send one small git.c patch so that\n--git-completion-helper works outside a repository, as was done for\n--help-all last year. That patch needs both of my topics first.\n\nTo pick the commands, I started from the 29 builtins marked\nNO_PARSEOPT in git.c. I also read builtins that loop over argv by\nhand before or after their real parse. That is how I found stash\nshow and the --no-index loop in \"git diff\". I ran each\ncandidate on master (fb255012ac), and I was able to reproduce\neverything below. Other places I found are in section 4.\n\nA. Convert hand-written parsing to parse-options\n\n  1. stash show. git stash show has a problem when somebody uses\n     -G <word>, which keeps only the changes that mention <word>.\n     Running \"git stash show -p -Gbar\" shows the diff of the patch,\n     but running \"git stash show -p -G bar\" fails with \"error: bar is\n     not a valid reference\". -S has the same problem.\n\n     It fails because, after parse_options(), stash show runs a loop\n     over each argument that is left over. If that argument starts\n     with \"-\", it goes to the diff options. If it does not, it is\n     treated as a stash name. That is also what happens to \"bar\".\n     Since it is treated as a stash name, and it is not a valid one,\n     the command fails.\n\n     blame and shortlog do not have this problem. They parse their\n     own options and the diff options together with\n     parse_revision_opt(), so -G gets its value. I would do the same\n     in stash show. What is left at the end would still be the stash\n     name, so \"git stash show 1\" keeps working. Some options, like\n     --all, are ignored today but would start to fail, so the commit\n     message has to say so.\n\n  2. Numbers in index-pack and fetch-pack. index-pack reads\n     --max-input-size with strtoumax(). The issue is that it only\n     reads the digits at the start, and silently ignores the rest of\n     the characters:\n       git index-pack --stdin --max-input-size=1k   limit of 1 byte\n       git index-pack --stdin --max-input-size=abc  no limit\n     So \"1k\" means 1 byte, and \"abc\" means 0, which means no limit.\n     fetch-pack reads --depth with strtol() in the same way, so a bad\n     depth fetches the whole history:\n       git fetch-pack --depth=abc ../src HEAD       no depth limit\n     The receive.maxInputSize config rejects \"abc\" and reads \"1k\" as\n     1024, so --max-input-size and the config do not agree.\n\n     These are plumbing, and Junio asked on the diff-files\n     conversion whether converting is worth it for plumbing \"meant\n     for use by scripts\" [13]. So I would send small fixes for the\n     numbers, and convert the rest only if the list wants it.\n     unpack-objects has the same --max-input-size code, so I would\n     ask on the list whether to fix it in the same series.\n\n  For each command I will first add tests for what works today, like\n  \"git stash show 1\". The test for the bug must fail before the fix.\n  I will also break each new branch on purpose, to check that a test\n  catches it.\n\nB. Improve the early scan where \"git diff\" needs it\n\n  Before parsing anything, \"git diff\" runs a small loop whose job is\n  to look for --no-index. The loop's answer determines which parser\n  reads the rest of the command line: either the no-index one or the\n  normal one. An issue arises because the loop stops at the first\n  argument that doesn't start with \"-\". So in \"git diff -G x\n  --no-index a b\", it stops at \"x\", the value of -G, and never sees\n  --no-index. Since it never saw --no-index, the loop picks the\n  normal parser, which then gets --no-index as an option and fails\n  with \"invalid option\".\n\n  The opposite also happens. In \"git diff -S --no-index HEAD~1 HEAD\",\n  \"--no-index\" is the value of -S, a search for that text. But the\n  loop takes it for the option, so Git switches to no-index mode and\n  fails with \"Could not access 'HEAD~1'\". Inside a repository, with\n  a and b in the work tree:\n       git diff -G x --no-index a b         error: invalid option:\n                                            --no-index\n       git diff -S --no-index HEAD~1 HEAD   error: Could not access\n                                            'HEAD~1'\n  With the value stuck (-Gx, -S--no-index), both work.\n\n  Christian's v1 cover letter lists \"git diff\" among the commands\n  that could use the early scan, but were left out because it \"has a\n  number of short options like `-S`, `-G`, `-O` taking separate\n  values\" [14].\n\n  Junio asked in Azuama's rev-list thread whether a separate early\n  scan is a good direction at all [15]. For \"git diff\" I think one is\n  still needed, because --no-index has to be found before either\n  parser runs.\n\n  But early_scan_options() cannot be used here yet. It needs a list\n  of the options, and \"git diff\" does not have one in a single place.\n  It also stops at every short option, like -G, so it would first\n  have to learn to step over them. For long options, patch 2/4 of\n  Christian's series split lookup_long_opt() out of parse_long_opt(),\n  so the scan can find them. Short options have nothing like that yet.\n\n  Where the loop stops also decides when two paths outside the\n  repository are treated as --no-index. So \"git diff -G x ../a ../b\"\n  fails too, while \"git diff ../a ../b\" works. t4053, the test script\n  for --no-index, has no case with a short option before --no-index.\n  So I will add tests for these three cases first.\n\n  Either way, the loop should not keep its own list of the options\n  that take a value. In his reply to Azuama's v4 [16], Junio called\n  that approach \"inherently fragile\".\n\n  Question: should I work towards using the early scan here, or would\n  you rather see the loop in \"git diff\" fixed first?\n\n\n4. Other places I found\n-----------------------\nThese are not in the timeline, but I would pick one of them with the\nmentors if there is time.\n\ngit daemon. It reads all its options by hand. --port is read with\nstrtoul(), and nothing checks that the number is a valid port:\n       git daemon --port=70000 ...    listens on port 4464, no error\n       git daemon --port=0x2329 ...   listens on port 9001\nA small fix would be a range check. A full conversion to\nparse-options has to keep every argument form working, because\nexisting inetd and start-up scripts call it with those arguments.\n\nRevision options. handle_revision_opt() in revision.c parses the\noptions of log, rev-list, show and others by hand. Christian's v1\ncover letter suggests converting them to struct option [14]. The\ndiff options were moved to parse-options in 2019, but the revision\noptions were not. This is core code, so I would only start it if the\nmentors want it.\n\n\n5. Timeline\n-----------\nNow to 6 Dec   Land tj/drop-no-parseopt and tj/annotate-blame-args.\nWeeks 1-2      stash show: tests first, then the fix.\nWeeks 3-4      The number fixes in index-pack and fetch-pack, and the\n               \"git diff\" tests from 3B. This is the holidays, so\n               small patches.\nWeeks 5-10     \"git diff\" --no-index, in the way the list prefers.\n               RFC first if it changes the scan.\nWeeks 11-12    Rerolls, or the git daemon range check if there is\n               time.\nWeek 13        Spare week for late reviews, and the final blog post.\n\nReview on the list takes days, so the weeks overlap. While one series\nwaits, I work on the next one.\n\nAt minimum: stash show and the number fixes in 'next', and the \"git\ndiff\" tests and a fix sent and discussed. If the list does not want\nthe early scan there, the fix is the loop in \"git diff\" alone. If I\nfall behind, I drop the section 4 item first.\n\n\n6. Communication\n----------------\nA blog post every week for Christian, and a short written update\nevery week for Siddharth, by email. The update says what I sent,\nwhat is in review, what is blocked, and what I will do next week.\nDesign questions go to the list before code. I answer every review\ncomment on the list before sending another version, as Kaartic\nreminded me to.\n\n\n7. Questions for mentors\n------------------------\n1. Should I start with stash show, or would you pick another command?\n2. Which item from section 4, if any, would you like to see after\n   that?\nThe question about \"git diff\" is at the end of 3B.\n\n\n8. After the internship\n-----------------------\nI want to keep working on the early scan and the revision options,\nand keep reviewing parse-options patches. I would also like to help\nthe next group of applicants the way reviewers on the list have\nhelped me.\n\n\n9. References\n-------------\n[1] Proposal v1:\n     https://lore.kernel.org/git/20261105195730.1518433-1-tangajoris@gmail.com/\n[2] Microproject, checkout: use Q_():\n     https://lore.kernel.org/git/20261022004834.246892-1-tangajoris@gmail.com/\n[3] tj/drop-no-parseopt v5:\n     https://lore.kernel.org/git/20261105223458.1534785-1-tangajoris@gmail.com/\n[4] annotate: pass the arguments to blame unchanged:\n     https://lore.kernel.org/git/20261105073101.1449292-1-tangajoris@gmail.com/\n[5] What's cooking in git.git (Nov 2026, #03):\n     https://lore.kernel.org/git/xmqqmrrlfzcd.fsf@gitster.g/\n[6] apply fix for \"git apply --check\":\n     https://lore.kernel.org/git/20261107195429.96842-1-tangajoris@gmail.com/\n[7] My review of Azuama's rev-list v3:\n     https://lore.kernel.org/git/CAKrRMNrnd2+HxTxadKaha+K9ASWi_oN4h7gdjVYAGmyODwmi=w@mail.gmail.com/\n[8] Azuama's rev-list v4:\n     https://lore.kernel.org/git/20261105120937.10269-1-azuama69@gmail.com/\n[9] My review of Joy's pack size limit v3:\n     https://lore.kernel.org/git/CAKrRMNqzOX=vUUMrMf5P_Atar8_qCSbGdtBktfT+iL4hJfFX-Q@mail.gmail.com/\n[10] Joy's separate patch for 1/2:\n     https://lore.kernel.org/git/pull.2298.git.1793966368860.gitgitgadget@gmail.com/\n[11] My review of Nelly's promisor.fetchOrder patch:\n     https://lore.kernel.org/git/CAKrRMNpAW+f2XXMFqpB8ffZsTdyPsaFtHe8dku=JVXACp1M3Hw@mail.gmail.com/\n[12] My review of shrimech's Rust patch:\n     https://lore.kernel.org/git/CAKrRMNrLvPck5psPpKX1eWe26To5srfFSLFObZ4zTX09RE551w@mail.gmail.com/\n[13] Junio on the diff-files conversion:\n     https://lore.kernel.org/git/xmqq8q3blfh1.fsf@gitster.g/\n[14] Christian's early-scan v1 cover letter:\n     https://lore.kernel.org/git/20260902161047.476753-1-christian.couder@gmail.com/\n[15] Junio on whether rev-list needs an early scan:\n     https://lore.kernel.org/git/xmqqa4nlkelo.fsf@gitster.g/\n[16] Junio on Azuama's rev-list v4:\n     https://lore.kernel.org/git/xmqqwlqqokxr.fsf@gitster.g/\n\nThanks,\nTanga\n\n"},{"id":"557949","messageId":"20261109150910.169348-1-tangajoris@gmail.com","threadId":"66786","inReplyTo":"20261108175038.946313-1-tangajoris@gmail.com","subject":"[Outreachy] Proposal draft v3: Improve how command arguments and options are scanned and parsed","fromName":"Tanga Joris","fromEmail":"tangajoris@gmail.com","sentAt":"2026-11-09T15:09:10Z","receivedAt":"2026-11-09T15:09:10Z","isPatch":false,"sender":{"key":"tangajoris@gmail.com","avatar":null},"body":"Hi everyone,\n\nThis is the third draft of my proposal for \"Improve how command\narguments and options are scanned and parsed\" [1].\n\nChanges since v2:\n - Moved the git daemon conversion into the plan, so the plan does\n   not depend on the early-scan work.\n - Tried early_scan_options() in \"git diff\", and added what happened\n   to 3B.\n - Removed the fallback of fixing the loop alone from the minimum\n   outcome.\n - Updated the timeline.\n\nI would be grateful for feedback on section 3, and on the questions\nin section 7. I plan to start with stash show. If someone is already\nworking on one of these commands, please tell me and I will pick\nanother one.\n\n\n1. About me\n-----------\nName: Tanga Joris\nPronouns: he/him\nEmail: tangajoris@gmail.com\nGitHub: https://github.com/tmjoris\nBlog: https://tmjoris.github.io\nLocation: Nairobi, Kenya (EAT, UTC+3)\n\nI meet all the Outreachy eligibility rules.\n\nI am about to graduate from the United States International\nUniversity - Africa, where I study Applied Computer Technology. I\nlearned C at ALX, where it was my introduction to programming.\n\nI had done some small parsing work before, in my own project, kiln\n(https://github.com/tmjoris/kiln), a Go CLI that runs code in\nsandboxes. \"kiln -addr host:7070 run\" works, but \"kiln run -addr\nhost:7070\" fails. This is because main() and each subcommand parse\ntheir flags separately. Git has the same question of who looks at an\nargument first, and what they do with it.\n\nThat work made me curious about how much goes on behind something as\nseemingly simple as parsing command-line arguments. I enjoy that kind\nof work: starting with something that looks straightforward,\nfollowing the implementation, and gradually understanding the\ndecisions and assumptions behind it.\n\nAvailability: 40 hours a week from 7 December 2026 to 8 March 2027.\nI have no classes, exams or other work during the internship.\nNairobi is UTC+3, so my working day overlaps with European hours.\n\n\n2. My Git work so far\n---------------------\n- Microproject: checkout: use Q_() for messages with counts [2].\n  Merged to master in a04194e134.\n- tj/drop-no-parseopt, v5 [3]. Four builtins used parse-options but\n  were still marked NO_PARSEOPT, so completion never asked them for\n  their options. The topic removes the flag from them, and adds a\n  t0012 test that catches such builtins. Junio: \"OK. Let's mark the\n  topic as ready for 'next'.\"\n- tj/annotate-blame-args [4]. I found it while trying to make\n  --git-completion-helper work outside a repository. cmd_annotate()\n  put \"-c\" in front of the user's arguments, so the helper option was\n  no longer the only argument. The diff option parser answered it\n  instead, and \"git annotate --<TAB>\" offered the options of\n  \"git diff\". The \"-c\" was not needed, so the fix deletes code.\n  Junio: \"Nice simplification.\"\n  Both topics are marked \"Will merge to 'next'?\" in What's cooking\n  [5].\n  Both topics merge cleanly on top of 'next', which has Christian's\n  early-scan series, and t9300, t0040, t0012 and t9902 pass.\n- apply: a fix for the \"git apply --check\" crash reported on the\n  list [6]. While looking into it, I found two more ways to trigger\n  the same assertion. Phillip asked for an error instead of accepting\n  these broken patches, and I agreed. v2 will follow once we settle\n  the details.\n- A fix for the splitIndex bug that Andy Koppe reported on the list\n  [7]. With core.splitIndex and index.skipHash both set, index\n  entries are lost when a new shared index is written.\n- Reviews:\n  - Edson's credential conversion [8]. I suggested a helper for the\n    code that was repeated in four functions, and a test for the\n    check on extra arguments, which no test covered.\n  - Azuama's rev-list early scan fix [9]. The tests failed without\n    the fix, but --author, --committer and --grep-reflog had the\n    same problem. Azuama's v4 skips their values too, and its\n    NEEDSWORK comment now uses my \"--author --missing-only\" example\n    [10]. Junio then asked for a different approach than a list of\n    options in rev-list.c.\n  - Joy's pack size limit patch [11]. 2/2 added a member to struct\n    repo_config_values, which Junio had asked people not to do for\n    now. So I asked whether 1/2 could be sent on its own, and Joy\n    sent it as a separate patch [12].\n  - Nelly's promisor.fetchOrder patch [13]. Setting the new variable\n    made \"git show\" in a partial clone die with a BUG() in\n    string-list.c, because an in-place split was used on a list that\n    duplicates its strings.\n  - shrimech's Rust patch for formatting object IDs [14]. \"cargo\n    test\" passed and the new test failed without the fix. I asked\n    whether algo 0 should be printed the way the C code does it.\n    brian explained that algo 0 coming from C is meant to panic.\n\n\n3. Plan\n-------\nBefore the internship, I plan to land my two topics. If there is\ntime, I would also send one small git.c patch so that\n--git-completion-helper works outside a repository, as was done for\n--help-all last year. That patch needs both of my topics first.\n\nTo pick the commands, I started from the 29 builtins marked\nNO_PARSEOPT in git.c. I also read builtins that loop over argv by\nhand before or after their real parse. That is how I found stash\nshow and the --no-index loop in \"git diff\". I ran each\ncandidate on master (fb255012ac), and I was able to reproduce\neverything below. Other places I found are in section 4.\n\nA. Convert hand-written parsing to parse-options\n\n  1. stash show. git stash show has a problem when somebody uses\n     -G <word>, which keeps only the changes that mention <word>.\n     Running \"git stash show -p -Gbar\" shows the diff of the patch,\n     but running \"git stash show -p -G bar\" fails with \"error: bar is\n     not a valid reference\". -S has the same problem.\n\n     It fails because, after parse_options(), stash show runs a loop\n     over each argument that is left over. If that argument starts\n     with \"-\", it goes to the diff options. If it does not, it is\n     treated as a stash name. That is also what happens to \"bar\".\n     Since it is treated as a stash name, and it is not a valid one,\n     the command fails.\n\n     blame and shortlog do not have this problem. They parse their\n     own options and the diff options together with\n     parse_revision_opt(), so -G gets its value. I would do the same\n     in stash show. What is left at the end would still be the stash\n     name, so \"git stash show 1\" keeps working. Some options, like\n     --all, are ignored today but would start to fail, so the commit\n     message has to say so.\n\n  2. Numbers in index-pack and fetch-pack. index-pack reads\n     --max-input-size with strtoumax(). The issue is that it only\n     reads the digits at the start, and silently ignores the rest of\n     the characters:\n       git index-pack --stdin --max-input-size=1k   limit of 1 byte\n       git index-pack --stdin --max-input-size=abc  no limit\n     So \"1k\" means 1 byte, and \"abc\" means 0, which means no limit.\n     fetch-pack reads --depth with strtol() in the same way, so a bad\n     depth fetches the whole history:\n       git fetch-pack --depth=abc ../src HEAD       no depth limit\n     The receive.maxInputSize config rejects \"abc\" and reads \"1k\" as\n     1024, so --max-input-size and the config do not agree.\n\n     These are plumbing, and Junio asked on the diff-files\n     conversion whether converting is worth it for plumbing \"meant\n     for use by scripts\" [15]. So I would send small fixes for the\n     numbers, and convert the rest only if the list wants it.\n     unpack-objects has the same --max-input-size code, so I would\n     ask on the list whether to fix it in the same series.\n\n  3. git daemon. It reads all its options by hand, in one loop of\n     about 150 lines. --port is read with strtoul(), and nothing\n     checks that the number is a valid port:\n       git daemon --port=70000 ...    listens on port 4464, no error\n     I would convert the loop to parse-options, with\n     PARSE_OPT_STOP_AT_NON_OPTION for the list of directories. Other\n     programs outside builtin/, like imap-send and scalar, already\n     use parse_options(). Existing inetd and start-up scripts must\n     keep working, so every argument form stays, and --no- forms that\n     do not exist today stay rejected. The daemon re-runs itself with\n     its original arguments and --serve, and parse_options() changes\n     argv, so I have to keep a copy. Tests in t5570.\n\n     Junio's question about plumbing [15] may apply here too. No\n     other Git program runs git daemon, which is why I picked it over\n     index-pack and fetch-pack. I would ask on the list first. If the\n     list does not want it, I would send a range check for --port and\n     pick another command with the mentors.\n\n  For each command I will first add tests for what works today, like\n  \"git stash show 1\". The test for the bug must fail before the fix.\n  I will also break each new branch on purpose, to check that a test\n  catches it.\n\nB. Improve the early scan where \"git diff\" needs it\n\n  Before parsing anything, \"git diff\" runs a small loop whose job is\n  to look for --no-index. The loop's answer determines which parser\n  reads the rest of the command line: either the no-index one or the\n  normal one. An issue arises because the loop stops at the first\n  argument that doesn't start with \"-\". So in \"git diff -G x\n  --no-index a b\", it stops at \"x\", the value of -G, and never sees\n  --no-index. Since it never saw --no-index, the loop picks the\n  normal parser, which then gets --no-index as an option and fails\n  with \"invalid option\".\n\n  The opposite also happens. In \"git diff -S --no-index HEAD~1 HEAD\",\n  \"--no-index\" is the value of -S, a search for that text. But the\n  loop takes it for the option, so Git switches to no-index mode and\n  fails with \"Could not access 'HEAD~1'\". Inside a repository, with\n  a and b in the work tree:\n       git diff -G x --no-index a b         error: invalid option:\n                                            --no-index\n       git diff -S --no-index HEAD~1 HEAD   error: Could not access\n                                            'HEAD~1'\n  With the value stuck (-Gx, -S--no-index), both work.\n\n  Christian's v1 cover letter lists \"git diff\" among the commands\n  that could use the early scan, but were left out because it \"has a\n  number of short options like `-S`, `-G`, `-O` taking separate\n  values\" [16]. Few people type --no-index after -G. But \"git diff -G\n  x ../a ../b\" does not need --no-index at all, and it fails too.\n\n  Junio asked in Azuama's rev-list thread whether a separate early\n  scan is a good direction at all [17]. For \"git diff\" I think one\n  is still needed, because --no-index decides which parser reads the\n  rest of the command line. So it has to be found before either\n  parser runs.\n\n  The early scan has one rule that I find useful. It may miss an\n  option that parse_options() would accept, but it must never report\n  one that parse_options() would not. So when it is unsure, it stops.\n  Missing an option is the safe failure.\n\n  To see how far the scan gets today, I replaced the loop with\n  early_scan_options() on top of 'next', with add_diff_options() and\n  --no-index as its table. \"git diff -S --no-index HEAD~1 HEAD\" then\n  works. But \"git diff -G x --no-index a b\" still fails, and these\n  stop working:\n       git diff -p --no-index a b\n       git diff -U1 --no-index a b\n       git diff -Gx --no-index a b\n       git diff -w ../a ../b\n  The scan stops at every short option, so it never reaches\n  --no-index. t4053 still passes with this change, because it has no\n  case with a short option before --no-index. So the scan has to\n  step over short options first. For long options, patch 2/4 of\n  Christian's series split lookup_long_opt() out of parse_long_opt(),\n  so the scan can find them. Short options have nothing like that\n  yet.\n\n  Where the loop stops also decides when two paths outside the\n  repository are treated as --no-index. So \"git diff -G x ../a ../b\"\n  fails too, while \"git diff ../a ../b\" works. I will add tests for\n  these cases first.\n\n  Question: should I teach the scan to step over short options? Or\n  would a smaller fix be better, where the loop looks options up in\n  the add_diff_options() table? Junio called a hand-kept list of\n  options \"inherently fragile\" [18], so I don't plan on adding one.\n\n\n4. Other places I found\n-----------------------\nThis is not in the timeline, but I would start it with the mentors\nif there is time.\n\nRevision options. handle_revision_opt() in revision.c parses the\noptions of log, rev-list, show and others by hand. Christian's v1\ncover letter suggests converting them to struct option [16]. The\ndiff options were moved to parse-options in 2019, but the revision\noptions were not. This is core code, so I would only start it if the\nmentors want it.\n\n\n5. Timeline\n-----------\nNow to 6 Dec   Land tj/drop-no-parseopt and tj/annotate-blame-args.\nWeeks 1-2      stash show: tests first, then the fix.\nWeeks 3-4      The number fixes in index-pack and fetch-pack, and the\n               \"git diff\" tests from 3B. This is the holidays, so\n               small patches.\nWeeks 5-7      Convert git daemon to parse-options.\nWeeks 8-12     \"git diff\" --no-index, in the way the list prefers.\n               RFC first if it changes the scan.\nWeek 13        Spare week for late reviews, and the final blog post.\n\nReview on the list takes days, so the weeks overlap. While one series\nwaits, I work on the next one.\n\nAt minimum: stash show, the number fixes and the git daemon\nconversion sent and reviewed, and the \"git diff\" tests and an RFC\nsent. If I fall behind, the \"git diff\" work stops at the tests and\nthe RFC.\n\n\n6. Communication\n----------------\nA blog post every week for Christian, and a short written update\nevery week for Siddharth, by email. The update says what I sent,\nwhat is in review, what is blocked, and what I will do next week.\nDesign questions go to the list before code. I answer every review\ncomment on the list before sending another version, as Kaartic\nreminded me to.\n\n\n7. Questions for mentors\n------------------------\n1. Should I start with stash show, or would you pick another command?\n2. Would you like to see the revision options work from section 4?\nThe question about \"git diff\" is at the end of 3B.\n\n\n8. After the internship\n-----------------------\nI want to keep working on the early scan and the revision options,\nand keep reviewing parse-options patches. I would also like to help\nthe next group of applicants the way reviewers on the list have\nhelped me.\n\n\n9. References\n-------------\n[1] Proposal v2:\n     https://lore.kernel.org/git/20261108175038.946313-1-tangajoris@gmail.com/\n[2] Microproject, checkout: use Q_():\n     https://lore.kernel.org/git/20261022004834.246892-1-tangajoris@gmail.com/\n[3] tj/drop-no-parseopt v5:\n     https://lore.kernel.org/git/20261105223458.1534785-1-tangajoris@gmail.com/\n[4] annotate: pass the arguments to blame unchanged:\n     https://lore.kernel.org/git/20261105073101.1449292-1-tangajoris@gmail.com/\n[5] What's cooking in git.git (Nov 2026, #03):\n     https://lore.kernel.org/git/xmqqmrrlfzcd.fsf@gitster.g/\n[6] apply fix for \"git apply --check\":\n     https://lore.kernel.org/git/20261107195429.96842-1-tangajoris@gmail.com/\n[7] splitIndex fix:\n     https://lore.kernel.org/git/20261108234235.1594316-1-tangajoris@gmail.com/\n[8] My review of Edson's credential patch:\n     https://lore.kernel.org/git/CAKrRMNqsh=d+_C8u+q1EBeQVwDZaywGQGZ89oYLyyvmB=8u1jw@mail.gmail.com/\n[9] My review of Azuama's rev-list v3:\n     https://lore.kernel.org/git/CAKrRMNrnd2+HxTxadKaha+K9ASWi_oN4h7gdjVYAGmyODwmi=w@mail.gmail.com/\n[10] Azuama's rev-list v4:\n     https://lore.kernel.org/git/20261105120937.10269-1-azuama69@gmail.com/\n[11] My review of Joy's pack size limit v3:\n     https://lore.kernel.org/git/CAKrRMNqzOX=vUUMrMf5P_Atar8_qCSbGdtBktfT+iL4hJfFX-Q@mail.gmail.com/\n[12] Joy's separate patch for 1/2:\n     https://lore.kernel.org/git/pull.2298.git.1793966368860.gitgitgadget@gmail.com/\n[13] My review of Nelly's promisor.fetchOrder patch:\n     https://lore.kernel.org/git/CAKrRMNpAW+f2XXMFqpB8ffZsTdyPsaFtHe8dku=JVXACp1M3Hw@mail.gmail.com/\n[14] My review of shrimech's Rust patch:\n     https://lore.kernel.org/git/CAKrRMNrLvPck5psPpKX1eWe26To5srfFSLFObZ4zTX09RE551w@mail.gmail.com/\n[15] Junio on the diff-files conversion:\n     https://lore.kernel.org/git/xmqq8q3blfh1.fsf@gitster.g/\n[16] Christian's early-scan v1 cover letter:\n     https://lore.kernel.org/git/20260902161047.476753-1-christian.couder@gmail.com/\n[17] Junio on whether rev-list needs an early scan:\n     https://lore.kernel.org/git/xmqqa4nlkelo.fsf@gitster.g/\n[18] Junio on Azuama's rev-list v4:\n     https://lore.kernel.org/git/xmqqwlqqokxr.fsf@gitster.g/\n\nThanks,\nTanga\n\n"}]}