{"thread":{"id":"65247","subject":"[GSoC Proposal] Improve Disk Space Recovery for Partial Clones","startedAt":"2026-03-14T20:35:22Z","lastAt":"2026-03-25T11:47:27Z","messageCount":4,"participants":["Siddharth Shrimali","Karthik Nayak"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"538992","messageId":"CAGWgyh-NASOa+6NPhQHCte2_A7OkNm1r2qAov3Kn1+r1d25hYw@mail.gmail.com","threadId":"65247","inReplyTo":null,"subject":"[GSoC Proposal] Improve Disk Space Recovery for Partial Clones","fromName":"Siddharth Shrimali","fromEmail":"r.siddharth.shrimali@gmail.com","sentAt":"2026-03-14T20:34:44Z","receivedAt":"2026-03-14T20:35:22Z","isPatch":false,"sender":{"key":"r.siddharth.shrimali@gmail.com","avatar":null},"body":"Hello Git Community,\n\nI am Siddharth Shrimali, a contributor currently focused on the\npartial clone subsystem and test modernization (most recently the\nt0410 series). I would like to submit my proposal for GSoC 2026:\n'Improve Disk Space Recovery for Partial Clones.'\n\nI've had a great experience collaborating with the community so far\nand would appreciate any feedback or suggestions on the technical\napproach outlined below.\n\n---\n\n1. PERSONAL INFORMATION\n\nName:       Siddharth Shrimali\nEmail:        r.siddharth.shrimali@gmail.com\nGitHub:      https://github.com/siddharthshrimali\nLinkedIn:    https://www.linkedin.com/in/siddharthshrimali/\nUniversity:  Walchand College of Engineering, India\nDegree:      B.Tech, Computer Science and Engineering\n             (2nd year, 4th semester)\nTimezone:  IST (UTC+5:30)\n\n---\n\n2. PROJECT ABSTRACT\n\nGit's partial clone feature lets clients work with large repositories\nwithout downloading all objects upfront. Objects are fetched lazily from\npromisor remotes as needed, and git backfill can proactively fill them\nin. But the reverse does not exist: once blobs have been fetched, there\nis no way to drop them back to a promised-but-absent state. Over time,\nclients accumulate large locally-held blobs they no longer need, with no\npath to reclaim that space short of a full re-clone.\n\nThis project implements that missing direction: a mechanism to safely\nidentify blobs that are available on a promisor remote, remove them from\nlocal storage, and re-enter them into the promisor contract so that the\nexisting lazy-fetch infrastructure re-fetches them transparently on next\naccess. The result is a reversible partial clone lifecycle.\n\nProject size: 175 hours. Difficulty: Medium to Hard.\nLanguages: C, shell (bash/POSIX sh).\nMentors: Christian Couder, Karthik Nayak, Justin Tobler,\n         Siddharth Asthana, Ayush Chandekar.\n\n---\n\n3. PROBLEM STATEMENT\n\nWhen a user clones with --filter=blob:none, Git records the remote as a\npromisor remote and fetches blobs lazily on demand via\npromisor_remote_get_direct(). Over months of work, through lazy fetches,\nexplicit backfills, and day-to-day operations, the local repository\naccumulates blobs that may no longer be needed. There is currently no\nmechanism to drop these blobs back to a promised-but-absent state. The\nonly option is a fresh re-clone, which is disruptive and expensive.\n\nThe gap is the reverse of git backfill: backfill moves objects from\npromisor-promised to locally-present and this project moves them back.\n\n---\n\n4. GETTING STARTED PROGRESS\n\nThe project ideas page listed specific steps to get started. I have\ncompleted all of them:\n\n  - Built Git from source and ran the full test suite.\n\n  - Set up a partial clone experiment: created a local promisor remote\n    via file://, cloned with --filter=blob:none, triggered lazy fetches,\n    and inspected the resulting .promisor sidecar files to understand\n    how the promisor contract works at the file level.\n\n  - Studied builtin/backfill.c end to end, along with\n    promisor-remote.c, list-objects-filter.c, and\n    Documentation/technical/partial-clone.txt.\n\n  - Submitted a micro-patch series (see Section 6) that was accepted\n    and merged to master.\n\n  - Read the partial clone and Protocol v2 documentation thoroughly.\n\n---\n\n5. TECHNICAL APPROACH\n\nThe full flow, at a high level:\n\n  1. Enumerate locally-held blobs above a size threshold.\n  2. Verify each candidate is available on the promisor remote.\n  3. Write a .promisor sidecar registering dropped OIDs as promised.\n     (This must happen before removal. See Section 5.3.)\n  4. Remove confirmed blobs from local storage.\n  5. Enforce safety guards.\n\n---\n\n5.1 Blob Enumeration\n\nWalk the object graph from all local refs using traverse_commit_list()\nin list-objects.c, with a size-based filter consistent with the existing\nblob:limit=N syntax from list-objects-filter.c. The goal is to collect\nblob OIDs that exceed the threshold and are currently locally present,\nmeaning they resolve via find_pack_entry() or as a loose object, and are\nnot already in a promised/missing state.\n\n---\n\n5.2 Remote Availability Verification\n\nBefore removing anything, we confirm the promisor remote holds each\ncandidate blob. The cleanest way to do this is the Protocol v2\n\"object-info\" capability, which lets us query the remote for object\nmetadata without downloading the object bodies, in a single batched\nround trip.\n\nFor remotes that do not advertise object-info, a conservative fallback\napplies: if a blob is reachable from a commit the remote has, and the\nremote's advertised filter covers that object, its presence on the\nremote is implied.\n\n---\n\n5.3 Re-entering the Promisor Contract (write before delete)\n\nAfter removal, any access to a dropped blob must trigger a lazy fetch\nvia promisor_remote_get_direct() rather than a hard \"object missing\"\nerror. To achieve this, we write a new (possibly empty) packfile with a\n.promisor sidecar that lists the dropped OIDs.\n\nThe write-before-delete ordering is the most critical part of the\nimplementation. The .promisor sidecar must be written and fsynced before\nany object is physically removed. If a crash or interrupt happens in a\nwindow where an object is neither locally present nor promised, the\nrepository ends up in an unrecoverable state.\n\n---\n\n5.4 Object Removal\n\nTwo cases must be handled:\n\n  a) Loose objects: compute the path via oid_to_path() (correct for\n     both SHA-1 and SHA-256) and unlink it.\n\n  b) Packed objects: a single object cannot be surgically removed from\n     a packfile. Instead, we drive the repack machinery in\n     builtin/repack.c to produce a new pack that excludes the target\n     OIDs. The --filter mechanism on git pack-objects already supports\n     object exclusion and can be driven with an explicit OID list.\n\n---\n\n5.5 Safety Guards\n\n  - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,\n    REVERT_HEAD, or rebase-merge/ is present under .git/. Removing\n    objects mid-operation could corrupt an otherwise-recoverable state.\n\n  - Index safety: by default, refuse to drop blobs referenced by the\n    current index (read via read_cache()). An explicit flag can override\n    this for users who understand the implications.\n\n  - Remote reachability: refuse if the promisor remote is not reachable\n    at invocation time. Objects should not be dropped if they cannot be\n    immediately re-fetched when needed.\n\n  - Dry-run (--dry-run): enumerate and verify candidates, print what\n    would be removed and the estimated space reclaimed, without touching\n    any objects.\n\n---\n\n5.6 Command Placement\n\nThe project description explicitly leaves placement as an open design\nquestion. I will send a design email to the list during the bonding\nperiod and defer to community feedback before writing any code. My\ncurrent thinking is:\n\n  - A new option on git backfill (e.g., --drop-blobs): backfill fills\n    objects in; this flag drains them back out. The symmetry is clean\n    and the command is already familiar to users working with partial\n    clones.\n\n  - An opt-in git maintenance task for scheduled periodic space\n    reclamation, wrapping the same core logic.\n\n---\n\n6. MICRO-PROJECT\n\nAs the required GSoC qualification task, I submitted a two-patch series\nto modernize t/t3700-add.sh:\n\n  [PATCH v3 1/2] t3700: avoid suppressing git's exit code\n  [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics\n\n  Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>\n  Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/\n\nPatch 1 breaks pipelines of the form \"git foo | grep bar\" that silently\nswallows git's exit code, by redirecting git output to a temp file and\ngrepping that. Patch 2 replaces all plain grep / ! grep calls with the\ntest_grep helper for better failure diagnostics. The series was iterated\nto v3 incorporating structural feedback from Junio C Hamano.\n\nStatus: Merged to master.\n\nI have also contributed several other patches to Git since February\n2026, including a C code fix in builtin/submodule--helper.c, test\nmodernizations in t3200 and t9123, and notably in\nt/t0410-partial-clone.sh which is directly relevant to this project.\nFull details are in Section 9.\n\n---\n\n7. DELIVERABLES\n\nThe deliverables section lists the concrete outputs I commit to\nproducing by the end of the project. This is what both the mentors and\nI can use to track progress and evaluate success.\n\n  1. A design email to git@vger.kernel.org during the bonding period\n     covering interface, command placement, safety model, and\n     write-before-delete ordering, with community sign-off before any\n     implementation begins.\n\n  2. Core logic: blob enumeration, remote availability verification\n     (Protocol v2 object-info + fallback), .promisor sidecar write, and\n     safe removal of loose and packed objects.\n\n  3. Command integration: wired into the agreed-upon command surface\n     (git backfill option and/or git maintenance task).\n\n  4. A comprehensive test suite covering: basic drop and re-fetch,\n     dry-run, size filtering, safety guard rejections, SHA-256\n     compatibility, loose and packed object cases, and\n     write-before-delete crash safety.\n\n  5. Documentation: man page updates for the modified command(s) and\n     updates to Documentation/technical/partial-clone.txt describing the\n     new object lifecycle (fetch -> hold -> drop -> re-fetch).\n\n---\n\n8. TIMELINE (175 hours)\n\nIn Progress (Now - May 1)\n\n  - Continue submitting patches to the list while selections are pending.\n  - Read builtin/repack.c and the Protocol v2 object-info code so the\n    bonding period can focus on design rather than catching up on reading.\n  - Experiment with .promisor sidecar files hands-on and draft the design\n    email early so it is ready to send on day one of bonding.\n\nCommunity Bonding Period (May 1 - May 26)\n\n  - Go deeper into promisor-remote.c, builtin/backfill.c,\n    builtin/repack.c, and list-objects-filter.c with the specific goal\n    of understanding the exact call chains relevant to this project.\n  - Trace the full lazy-fetch path in a debugger end to end.\n  - Send the design proposal to the list and incorporate feedback before\n    writing any implementation code.\n\nWeek 1-2 (May 27 - June 9) | Blob enumeration\n\n  - Implement blob enumeration via traverse_commit_list() with a size\n    filter, collecting locally-present blob OIDs above the threshold.\n  - Tests: correct OIDs collected, size filter respected, already-missing\n    blobs excluded.\n\nWeek 3-4 (June 10 - June 23) | Remote verification\n\n  - Implement remote availability check via Protocol v2 object-info,\n    batched for efficiency.\n  - Implement the conservative reachability-implies-presence fallback.\n  - Tests: mock promisor remote, batch query correctness, fallback path.\n\nWeek 5-6 (June 24 - July 7) | Removal and re-promisor\n\n  - Implement .promisor sidecar write for candidate OIDs (before removal).\n  - Implement loose object removal (oid_to_path + unlink).\n  - Implement packed object removal by driving repack with an OID\n    exclusion list.\n  - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch\n    triggers correctly after drop, write-before-delete ordering safety.\n  - Send an interim patch series to the list for early review.\n\nWeek 7-8 (July 8 - July 21) | Safety checks and command wiring\n\n  - Implement all safety guards: in-progress operation detection, index\n    blob check, remote reachability check.\n  - Implement --dry-run.\n  - Wire core logic into the agreed command surface with flags:\n    --filter, --dry-run, --verbose.\n  - Midterm evaluation: working prototype, tests passing, docs draft.\n\nWeek 9-10 (July 22 - August 4) | Edge cases and review\n\n  - Handle edge cases: multiple worktrees, alternates, submodules with\n    their own promisor remotes, SHA-256 repositories.\n  - Address mailing list review feedback from the interim series.\n  - Performance testing on a repository with many locally-held blobs.\n\nWeek 11-12 (August 5 - August 18) | Documentation and cleanup\n\n  - Write and finalize man page documentation.\n  - Update Documentation/technical/partial-clone.txt.\n  - Final patch series submission incorporating all review feedback.\n\nBuffer (August 19 - August 25)\n\n  - Address any remaining review comments.\n  - Ensure all tests pass on CI for both SHA-1 and SHA-256.\n  - Final evaluation submission.\n\n---\n\n9. MY CONTRIBUTIONS TO GIT\n\nI have been contributing to Git since early March 2026. All patches\nwere submitted to git@vger.kernel.org and iterated through review with\nJunio C Hamano, Jeff King, and Patrick Steinhardt.\n\nMerged to master:\n\n  1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)\n     <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>\n     (See Section 6 for full description. This is my micro-project.)\n     https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/\n     Status: Merged to master.\n\n  2. [PATCH] t9123: use test_when_finished for cleanup\n     <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>\n     Moved bare setup code in t9123 that ran outside any test block into\n     a proper test_expect_success block. Used test_when_finished to\n     register cleanup of the 'import' directory, ensuring it runs even\n     if the test fails, replacing a manual rm -rf that could be skipped\n     on failure.\n     https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/\n     Status: Merged to master.\n\n  3. [PATCH v2] t: fix \"that that\" typo in lib-unicode-nfc-nfd.sh\n     <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>\n     Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.\n     v1 over-reached by also fixing \"that that\" in two other files where\n     the phrasing was grammatically intentional; v2 correctly scoped the\n     change after re-reading the context.\n     https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/\n     Status: Merged to master.\n\nQueued for master / next:\n\n  4. [PATCH v2] submodule--helper: replace malloc with xmalloc\n     <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>\n     Replaced a raw malloc() in submodule_summary_callback() with\n     xmalloc(), which calls die() on allocation failure rather than\n     returning NULL and risking a NULL dereference. Also used sizeof(*temp)\n     instead of the explicit struct name, and dropped the redundant C cast\n     on the xmalloc() return. Improved the commit message in v2 to explain\n     the reasoning behind removing the cast, as requested by Junio.\n     https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/\n     Status: Will merge to next.\n\n  5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID\n     <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>\n     Replaced a hardcoded 40-zero string in t/t3200-branch.sh with\n     $ZERO_OID to make the test hash-algorithm independent. The 40-char\n     string caused premature failure under SHA-256 because Git reports\n     \"malformed object name\" (40 vs 64 chars) before reaching the actual\n     object-type check. Using $ZERO_OID with test_grep for the \"must\n     point to a commit\" error ensures the test validates the right failure\n     mode. Suggested-by Patrick Steinhardt.\n     https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/\n     Status: Will merge to master.\n\n  6. [PATCH v3] t0410: modernize delete_object helper\n     <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>\n     Modernized the delete_object helper in t/t0410-partial-clone.sh,\n     the primary test file for partial clones and directly relevant to\n     this project, by replacing a fragile manual sed-based object path\n     calculation with test_oid_to_path, making the helper correct for\n     both SHA-1 and SHA-256. Added 'local' variable declarations with\n     proper quoting for shell portability. Suggested-by Jeff King, who\n     also caught that unquoted 'local' assignments are flagged by\n     check-non-portable-shell. This patch prompted Junio to add a note\n     to Documentation/SubmittingPatches reminding contributors to run\n     'make test' from the top level.\n     Helped-by: Pushkar Singh.\n     https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/\n     Status: Good to go to next.\n\nCollaboration:\n\n  7. Helped with [PATCH v5] help: cleanup the construction of keys_uniq\n     <20260311192453.62213-1-amishhhaaaa@gmail.com>\n     (patch by Amisha Chhajed). Suggested moving strbuf initialization\n     and release out of the inner loop to reuse the same buffer across\n     iterations, avoiding repeated alloc/free overhead. Listed as\n     Suggested-by in the patch.\n     https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/\n     Status: Will merge to next.\n\nPatch #6 is worth highlighting separately: working on\nt0410-partial-clone.sh required reading the entire file carefully,\nunderstanding how promisor remote tests are structured, and reasoning\nabout hash-algorithm independence in the partial clone context. That\nwork gives me a direct head start on the test infrastructure for this\nproject.\n\n---\n\n10. RELEVANT EXPERIENCE\n\nC and shell (bash/POSIX sh) are my primary languages for this project.\nMy submodule--helper patch (patch #4 above) demonstrates comfort with C\nmemory management patterns in Git's codebase (xmalloc/die() convention,\nsizeof(*ptr) idiom, implicit void* promotion). My test patches\ndemonstrate hands-on familiarity with Git's shell test framework:\ntest_grep, test_oid_to_path, test_must_fail, test_when_finished, and the\nportability rules enforced by check-non-portable-shell. I am also\nfamiliar with Go, which gives me a general systems programming background.\n\nI have submitted patches through multiple review cycles on the mailing\nlist, incorporated feedback from senior maintainers, and am comfortable\nwith the in-reply-to threading conventions and interdiff-based review\nprocess.\n\nYou can find all my patches at:\nhttps://lore.kernel.org/git/?q=Siddharth+Shrimali\n\n---\n\n11. AVAILABILITY\n\nI can commit 25-30 hours per week to GSoC during the coding period\n(May-August). My 4th semester exams conclude by mid-May, so I will be\npartly available at the start and fully available from mid-May onward\nwith no academic conflicts through the end of summer. I have no\ninternships, part-time work, or other commitments planned for this\nperiod.\n\nI will send a brief weekly status update to my mentors and post\nquestions or design discussions to the mailing list as they come up, to\nstay unblocked and keep the community informed of progress.\n\n---\n\n12. WHY THIS PROJECT\n\nI have been using Git for a while, but it was only when I started\ncontributing patches that I actually began reading its internals. Partial\nclone was one of the first areas I looked at carefully, partly because of\nthe t0410 patch and partly out of genuine curiosity. As I read\npromisor-remote.c and backfill.c, the missing reverse direction became\nclear. Backfill exists, but its counterpart does not. That felt like an\nunfinished thought.\n\nI find the write-before-delete ordering problem genuinely interesting. It\nis one of those situations where the obvious implementation is wrong, and\ngetting it right requires careful reasoning about crash recovery. The same\nis true for packed object removal, where you cannot simply delete one\nobject and call it done. These are exactly the kinds of problems I enjoy\nthinking through.\n\nI also want to be candid that I do not yet know every corner of this\ncodebase. There will be parts of the implementation where I will need\nguidance. That is exactly why I want to use the bonding period to read,\nexperiment, and deepen my understanding before writing any implementation\ncode, and why the design discussion on the list matters to me. I would\nmuch rather understand the problem thoroughly than rush into code that\nlater has to be thrown away.\n\n---\n\nThank you for your time and for considering my proposal. I have genuinely\nenjoyed contributing to Git over the past few weeks and I am looking\nforward to doing more of it, with or without GSoC.\nIf you have any questions or if there is anything missing from this\nproposal, I am happy to follow up on the list.\n\n---\n\nSiddharth Shrimali\nr.siddharth.shrimali@gmail.com\n"},{"id":"539172","messageId":"CAOLa=ZSqQ=w8EVFMUwn5EUfMmNBgzGzG458Ex1ixiBSTiWnWYg@mail.gmail.com","threadId":"65247","inReplyTo":"CAGWgyh-NASOa+6NPhQHCte2_A7OkNm1r2qAov3Kn1+r1d25hYw@mail.gmail.com","subject":"Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-16T23:05:25Z","receivedAt":"2026-03-16T23:05:27Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":null},"body":"Siddharth Shrimali <r.siddharth.shrimali@gmail.com> writes:\n\nHello Siddharth,\n\n> Hello Git Community,\n>\n> I am Siddharth Shrimali, a contributor currently focused on the\n> partial clone subsystem and test modernization (most recently the\n> t0410 series). I would like to submit my proposal for GSoC 2026:\n> 'Improve Disk Space Recovery for Partial Clones.'\n>\n> I've had a great experience collaborating with the community so far\n> and would appreciate any feedback or suggestions on the technical\n> approach outlined below.\n>\n> ---\n>\n> 1. PERSONAL INFORMATION\n>\n> Name:       Siddharth Shrimali\n> Email:        r.siddharth.shrimali@gmail.com\n> GitHub:      https://github.com/siddharthshrimali\n> LinkedIn:    https://www.linkedin.com/in/siddharthshrimali/\n> University:  Walchand College of Engineering, India\n> Degree:      B.Tech, Computer Science and Engineering\n>              (2nd year, 4th semester)\n> Timezone:  IST (UTC+5:30)\n>\n> ---\n>\n> 2. PROJECT ABSTRACT\n>\n> Git's partial clone feature lets clients work with large repositories\n> without downloading all objects upfront. Objects are fetched lazily from\n> promisor remotes as needed, and git backfill can proactively fill them\n> in. But the reverse does not exist: once blobs have been fetched, there\n> is no way to drop them back to a promised-but-absent state. Over time,\n> clients accumulate large locally-held blobs they no longer need, with no\n> path to reclaim that space short of a full re-clone.\n>\n> This project implements that missing direction: a mechanism to safely\n> identify blobs that are available on a promisor remote, remove them from\n> local storage, and re-enter them into the promisor contract so that the\n> existing lazy-fetch infrastructure re-fetches them transparently on next\n> access. The result is a reversible partial clone lifecycle.\n>\n> Project size: 175 hours. Difficulty: Medium to Hard.\n> Languages: C, shell (bash/POSIX sh).\n> Mentors: Christian Couder, Karthik Nayak, Justin Tobler,\n>          Siddharth Asthana, Ayush Chandekar.\n>\n> ---\n>\n> 3. PROBLEM STATEMENT\n>\n> When a user clones with --filter=blob:none, Git records the remote as a\n> promisor remote and fetches blobs lazily on demand via\n> promisor_remote_get_direct(). Over months of work, through lazy fetches,\n> explicit backfills, and day-to-day operations, the local repository\n> accumulates blobs that may no longer be needed. There is currently no\n> mechanism to drop these blobs back to a promised-but-absent state. The\n> only option is a fresh re-clone, which is disruptive and expensive.\n>\n> The gap is the reverse of git backfill: backfill moves objects from\n> promisor-promised to locally-present and this project moves them back.\n>\n> ---\n>\n> 4. GETTING STARTED PROGRESS\n>\n> The project ideas page listed specific steps to get started. I have\n> completed all of them:\n>\n>   - Built Git from source and ran the full test suite.\n>\n>   - Set up a partial clone experiment: created a local promisor remote\n>     via file://, cloned with --filter=blob:none, triggered lazy fetches,\n>     and inspected the resulting .promisor sidecar files to understand\n>     how the promisor contract works at the file level.\n>\n>   - Studied builtin/backfill.c end to end, along with\n>     promisor-remote.c, list-objects-filter.c, and\n>     Documentation/technical/partial-clone.txt.\n>\n>   - Submitted a micro-patch series (see Section 6) that was accepted\n>     and merged to master.\n>\n>   - Read the partial clone and Protocol v2 documentation thoroughly.\n>\n> ---\n>\n> 5. TECHNICAL APPROACH\n>\n> The full flow, at a high level:\n>\n>   1. Enumerate locally-held blobs above a size threshold.\n>   2. Verify each candidate is available on the promisor remote.\n>   3. Write a .promisor sidecar registering dropped OIDs as promised.\n>      (This must happen before removal. See Section 5.3.)\n>   4. Remove confirmed blobs from local storage.\n>   5. Enforce safety guards.\n>\n> ---\n>\n> 5.1 Blob Enumeration\n>\n> Walk the object graph from all local refs using traverse_commit_list()\n> in list-objects.c, with a size-based filter consistent with the existing\n> blob:limit=N syntax from list-objects-filter.c. The goal is to collect\n> blob OIDs that exceed the threshold and are currently locally present,\n> meaning they resolve via find_pack_entry() or as a loose object, and are\n> not already in a promised/missing state.\n\nHow does this work when used on a partial clone with 'blob:none'?\n\n> ---\n>\n> 5.2 Remote Availability Verification\n>\n> Before removing anything, we confirm the promisor remote holds each\n> candidate blob. The cleanest way to do this is the Protocol v2\n> \"object-info\" capability, which lets us query the remote for object\n> metadata without downloading the object bodies, in a single batched\n> round trip\n\nMakes sense.\n\n>\n> For remotes that do not advertise object-info, a conservative fallback\n> applies: if a blob is reachable from a commit the remote has, and the\n> remote's advertised filter covers that object, its presence on the\n> remote is implied.\n\nHow do we know a blob is reachable from a commit that the remote\nadvertises?\n\n>\n> ---\n>\n> 5.3 Re-entering the Promisor Contract (write before delete)\n>\n> After removal, any access to a dropped blob must trigger a lazy fetch\n> via promisor_remote_get_direct() rather than a hard \"object missing\"\n> error. To achieve this, we write a new (possibly empty) packfile with a\n> .promisor sidecar that lists the dropped OIDs.\n>\n> The write-before-delete ordering is the most critical part of the\n> implementation. The .promisor sidecar must be written and fsynced before\n> any object is physically removed. If a crash or interrupt happens in a\n> window where an object is neither locally present nor promised, the\n> repository ends up in an unrecoverable state.\n>\n\nYeah, this makes sense too.\n\n> ---\n>\n> 5.4 Object Removal\n>\n> Two cases must be handled:\n>\n>   a) Loose objects: compute the path via oid_to_path() (correct for\n>      both SHA-1 and SHA-256) and unlink it.\n>\n>   b) Packed objects: a single object cannot be surgically removed from\n>      a packfile. Instead, we drive the repack machinery in\n>      builtin/repack.c to produce a new pack that excludes the target\n>      OIDs. The --filter mechanism on git pack-objects already supports\n>      object exclusion and can be driven with an explicit OID list.\n>\n> ---\n>\n> 5.5 Safety Guards\n>\n>   - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,\n>     REVERT_HEAD, or rebase-merge/ is present under .git/. Removing\n>     objects mid-operation could corrupt an otherwise-recoverable state.\n>\n\nBut couldn't one of these operations start after we start the process of\ncleanup?\n\n>   - Index safety: by default, refuse to drop blobs referenced by the\n>     current index (read via read_cache()). An explicit flag can override\n>     this for users who understand the implications.\n>\n>   - Remote reachability: refuse if the promisor remote is not reachable\n>     at invocation time. Objects should not be dropped if they cannot be\n>     immediately re-fetched when needed.\n>\n>   - Dry-run (--dry-run): enumerate and verify candidates, print what\n>     would be removed and the estimated space reclaimed, without touching\n>     any objects.\n>\n\nNice.\n\n> ---\n>\n> 5.6 Command Placement\n>\n> The project description explicitly leaves placement as an open design\n> question. I will send a design email to the list during the bonding\n> period and defer to community feedback before writing any code. My\n> current thinking is:\n>\n>   - A new option on git backfill (e.g., --drop-blobs): backfill fills\n>     objects in; this flag drains them back out. The symmetry is clean\n>     and the command is already familiar to users working with partial\n>     clones.\n>\n\nI'm not sure if 'git backfill --drop-blobs' makes sense, since the\ncommand talks about filling in data and the flag talks about cleaning up\ndata. But this seems to be the closest relative.\n\n>   - An opt-in git maintenance task for scheduled periodic space\n>     reclamation, wrapping the same core logic.\n>\n\nThis would be my top pick, seems like data cleanup is definitely a\nmaintenance task.\n\n> ---\n>\n> 6. MICRO-PROJECT\n>\n> As the required GSoC qualification task, I submitted a two-patch series\n> to modernize t/t3700-add.sh:\n>\n>   [PATCH v3 1/2] t3700: avoid suppressing git's exit code\n>   [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics\n>\n>   Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>\n>   Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/\n>\n> Patch 1 breaks pipelines of the form \"git foo | grep bar\" that silently\n> swallows git's exit code, by redirecting git output to a temp file and\n> grepping that. Patch 2 replaces all plain grep / ! grep calls with the\n> test_grep helper for better failure diagnostics. The series was iterated\n> to v3 incorporating structural feedback from Junio C Hamano.\n>\n> Status: Merged to master.\n>\n> I have also contributed several other patches to Git since February\n> 2026, including a C code fix in builtin/submodule--helper.c, test\n> modernizations in t3200 and t9123, and notably in\n> t/t0410-partial-clone.sh which is directly relevant to this project.\n> Full details are in Section 9.\n>\n> ---\n>\n> 7. DELIVERABLES\n>\n> The deliverables section lists the concrete outputs I commit to\n> producing by the end of the project. This is what both the mentors and\n> I can use to track progress and evaluate success.\n>\n>   1. A design email to git@vger.kernel.org during the bonding period\n>      covering interface, command placement, safety model, and\n>      write-before-delete ordering, with community sign-off before any\n>      implementation begins.\n>\n>   2. Core logic: blob enumeration, remote availability verification\n>      (Protocol v2 object-info + fallback), .promisor sidecar write, and\n>      safe removal of loose and packed objects.\n>\n>   3. Command integration: wired into the agreed-upon command surface\n>      (git backfill option and/or git maintenance task).\n>\n>   4. A comprehensive test suite covering: basic drop and re-fetch,\n>      dry-run, size filtering, safety guard rejections, SHA-256\n>      compatibility, loose and packed object cases, and\n>      write-before-delete crash safety.\n>\n>   5. Documentation: man page updates for the modified command(s) and\n>      updates to Documentation/technical/partial-clone.txt describing the\n>      new object lifecycle (fetch -> hold -> drop -> re-fetch).\n>\n> ---\n>\n> 8. TIMELINE (175 hours)\n>\n> In Progress (Now - May 1)\n>\n>   - Continue submitting patches to the list while selections are pending.\n>   - Read builtin/repack.c and the Protocol v2 object-info code so the\n>     bonding period can focus on design rather than catching up on reading.\n>   - Experiment with .promisor sidecar files hands-on and draft the design\n>     email early so it is ready to send on day one of bonding.\n>\n> Community Bonding Period (May 1 - May 26)\n>\n>   - Go deeper into promisor-remote.c, builtin/backfill.c,\n>     builtin/repack.c, and list-objects-filter.c with the specific goal\n>     of understanding the exact call chains relevant to this project.\n>   - Trace the full lazy-fetch path in a debugger end to end.\n>   - Send the design proposal to the list and incorporate feedback before\n>     writing any implementation code.\n>\n> Week 1-2 (May 27 - June 9) | Blob enumeration\n>\n>   - Implement blob enumeration via traverse_commit_list() with a size\n>     filter, collecting locally-present blob OIDs above the threshold.\n>   - Tests: correct OIDs collected, size filter respected, already-missing\n>     blobs excluded.\n>\n> Week 3-4 (June 10 - June 23) | Remote verification\n>\n>   - Implement remote availability check via Protocol v2 object-info,\n>     batched for efficiency.\n>   - Implement the conservative reachability-implies-presence fallback.\n>   - Tests: mock promisor remote, batch query correctness, fallback path.\n>\n> Week 5-6 (June 24 - July 7) | Removal and re-promisor\n>\n>   - Implement .promisor sidecar write for candidate OIDs (before removal).\n>   - Implement loose object removal (oid_to_path + unlink).\n>   - Implement packed object removal by driving repack with an OID\n>     exclusion list.\n>   - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch\n>     triggers correctly after drop, write-before-delete ordering safety.\n>   - Send an interim patch series to the list for early review.\n>\n> Week 7-8 (July 8 - July 21) | Safety checks and command wiring\n>\n>   - Implement all safety guards: in-progress operation detection, index\n>     blob check, remote reachability check.\n>   - Implement --dry-run.\n>   - Wire core logic into the agreed command surface with flags:\n>     --filter, --dry-run, --verbose.\n>   - Midterm evaluation: working prototype, tests passing, docs draft.\n>\n> Week 9-10 (July 22 - August 4) | Edge cases and review\n>\n>   - Handle edge cases: multiple worktrees, alternates, submodules with\n>     their own promisor remotes, SHA-256 repositories.\n>   - Address mailing list review feedback from the interim series.\n>   - Performance testing on a repository with many locally-held blobs.\n>\n> Week 11-12 (August 5 - August 18) | Documentation and cleanup\n>\n>   - Write and finalize man page documentation.\n>   - Update Documentation/technical/partial-clone.txt.\n>   - Final patch series submission incorporating all review feedback.\n>\n> Buffer (August 19 - August 25)\n>\n>   - Address any remaining review comments.\n>   - Ensure all tests pass on CI for both SHA-1 and SHA-256.\n>   - Final evaluation submission.\n>\n> ---\n>\n> 9. MY CONTRIBUTIONS TO GIT\n>\n> I have been contributing to Git since early March 2026. All patches\n> were submitted to git@vger.kernel.org and iterated through review with\n> Junio C Hamano, Jeff King, and Patrick Steinhardt.\n>\n> Merged to master:\n>\n>   1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)\n>      <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>\n>      (See Section 6 for full description. This is my micro-project.)\n>      https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/\n>      Status: Merged to master.\n>\n>   2. [PATCH] t9123: use test_when_finished for cleanup\n>      <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>\n>      Moved bare setup code in t9123 that ran outside any test block into\n>      a proper test_expect_success block. Used test_when_finished to\n>      register cleanup of the 'import' directory, ensuring it runs even\n>      if the test fails, replacing a manual rm -rf that could be skipped\n>      on failure.\n>      https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/\n>      Status: Merged to master.\n>\n>   3. [PATCH v2] t: fix \"that that\" typo in lib-unicode-nfc-nfd.sh\n>      <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>\n>      Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.\n>      v1 over-reached by also fixing \"that that\" in two other files where\n>      the phrasing was grammatically intentional; v2 correctly scoped the\n>      change after re-reading the context.\n>      https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/\n>      Status: Merged to master.\n>\n> Queued for master / next:\n>\n>   4. [PATCH v2] submodule--helper: replace malloc with xmalloc\n>      <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>\n>      Replaced a raw malloc() in submodule_summary_callback() with\n>      xmalloc(), which calls die() on allocation failure rather than\n>      returning NULL and risking a NULL dereference. Also used sizeof(*temp)\n>      instead of the explicit struct name, and dropped the redundant C cast\n>      on the xmalloc() return. Improved the commit message in v2 to explain\n>      the reasoning behind removing the cast, as requested by Junio.\n>      https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/\n>      Status: Will merge to next.\n>\n>   5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID\n>      <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>\n>      Replaced a hardcoded 40-zero string in t/t3200-branch.sh with\n>      $ZERO_OID to make the test hash-algorithm independent. The 40-char\n>      string caused premature failure under SHA-256 because Git reports\n>      \"malformed object name\" (40 vs 64 chars) before reaching the actual\n>      object-type check. Using $ZERO_OID with test_grep for the \"must\n>      point to a commit\" error ensures the test validates the right failure\n>      mode. Suggested-by Patrick Steinhardt.\n>      https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/\n>      Status: Will merge to master.\n>\n>   6. [PATCH v3] t0410: modernize delete_object helper\n>      <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>\n>      Modernized the delete_object helper in t/t0410-partial-clone.sh,\n>      the primary test file for partial clones and directly relevant to\n>      this project, by replacing a fragile manual sed-based object path\n>      calculation with test_oid_to_path, making the helper correct for\n>      both SHA-1 and SHA-256. Added 'local' variable declarations with\n>      proper quoting for shell portability. Suggested-by Jeff King, who\n>      also caught that unquoted 'local' assignments are flagged by\n>      check-non-portable-shell. This patch prompted Junio to add a note\n>      to Documentation/SubmittingPatches reminding contributors to run\n>      'make test' from the top level.\n>      Helped-by: Pushkar Singh.\n>      https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/\n>      Status: Good to go to next.\n>\n> Collaboration:\n>\n>   7. Helped with [PATCH v5] help: cleanup the construction of keys_uniq\n>      <20260311192453.62213-1-amishhhaaaa@gmail.com>\n>      (patch by Amisha Chhajed). Suggested moving strbuf initialization\n>      and release out of the inner loop to reuse the same buffer across\n>      iterations, avoiding repeated alloc/free overhead. Listed as\n>      Suggested-by in the patch.\n>      https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/\n>      Status: Will merge to next.\n>\n> Patch #6 is worth highlighting separately: working on\n> t0410-partial-clone.sh required reading the entire file carefully,\n> understanding how promisor remote tests are structured, and reasoning\n> about hash-algorithm independence in the partial clone context. That\n> work gives me a direct head start on the test infrastructure for this\n> project.\n>\n> ---\n>\n> 10. RELEVANT EXPERIENCE\n>\n> C and shell (bash/POSIX sh) are my primary languages for this project.\n> My submodule--helper patch (patch #4 above) demonstrates comfort with C\n> memory management patterns in Git's codebase (xmalloc/die() convention,\n> sizeof(*ptr) idiom, implicit void* promotion). My test patches\n> demonstrate hands-on familiarity with Git's shell test framework:\n> test_grep, test_oid_to_path, test_must_fail, test_when_finished, and the\n> portability rules enforced by check-non-portable-shell. I am also\n> familiar with Go, which gives me a general systems programming background.\n>\n> I have submitted patches through multiple review cycles on the mailing\n> list, incorporated feedback from senior maintainers, and am comfortable\n> with the in-reply-to threading conventions and interdiff-based review\n> process.\n>\n> You can find all my patches at:\n> https://lore.kernel.org/git/?q=Siddharth+Shrimali\n>\n> ---\n>\n> 11. AVAILABILITY\n>\n> I can commit 25-30 hours per week to GSoC during the coding period\n> (May-August). My 4th semester exams conclude by mid-May, so I will be\n> partly available at the start and fully available from mid-May onward\n> with no academic conflicts through the end of summer. I have no\n> internships, part-time work, or other commitments planned for this\n> period.\n>\n> I will send a brief weekly status update to my mentors and post\n> questions or design discussions to the mailing list as they come up, to\n> stay unblocked and keep the community informed of progress.\n>\n> ---\n>\n> 12. WHY THIS PROJECT\n>\n> I have been using Git for a while, but it was only when I started\n> contributing patches that I actually began reading its internals. Partial\n> clone was one of the first areas I looked at carefully, partly because of\n> the t0410 patch and partly out of genuine curiosity. As I read\n> promisor-remote.c and backfill.c, the missing reverse direction became\n> clear. Backfill exists, but its counterpart does not. That felt like an\n> unfinished thought.\n>\n> I find the write-before-delete ordering problem genuinely interesting. It\n> is one of those situations where the obvious implementation is wrong, and\n> getting it right requires careful reasoning about crash recovery. The same\n> is true for packed object removal, where you cannot simply delete one\n> object and call it done. These are exactly the kinds of problems I enjoy\n> thinking through.\n>\n> I also want to be candid that I do not yet know every corner of this\n> codebase. There will be parts of the implementation where I will need\n> guidance. That is exactly why I want to use the bonding period to read,\n> experiment, and deepen my understanding before writing any implementation\n> code, and why the design discussion on the list matters to me. I would\n> much rather understand the problem thoroughly than rush into code that\n> later has to be thrown away.\n>\n> ---\n>\n> Thank you for your time and for considering my proposal. I have genuinely\n> enjoyed contributing to Git over the past few weeks and I am looking\n> forward to doing more of it, with or without GSoC.\n> If you have any questions or if there is anything missing from this\n> proposal, I am happy to follow up on the list.\n>\n> ---\n>\n\nThe rest of this proposal looks good to me!\n\n\n\n> Siddharth Shrimali\n> r.siddharth.shrimali@gmail.com\n\nRegards,\nKarthik\n"},{"id":"539219","messageId":"CAGWgyh_VmcNuay3AjsZ9zj9mP4g-c3Y9uDfuO5=kP9ix7Kdi3g@mail.gmail.com","threadId":"65247","inReplyTo":"CAOLa=ZSqQ=w8EVFMUwn5EUfMmNBgzGzG458Ex1ixiBSTiWnWYg@mail.gmail.com","subject":"Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones","fromName":"Siddharth Shrimali","fromEmail":"r.siddharth.shrimali@gmail.com","sentAt":"2026-03-17T13:28:35Z","receivedAt":"2026-03-17T13:29:13Z","isPatch":false,"sender":{"key":"r.siddharth.shrimali@gmail.com","avatar":null},"body":"Hi Karthik, thank you for the review!\n\nKarthik Nayak <karthik.188@gmail.com> wrote:\n\nLet me address each point.\n\n> > 5.1 Blob Enumeration\n> >\n> > Walk the object graph from all local refs using traverse_commit_list()\n> > in list-objects.c, with a size-based filter consistent with the existing\n> > blob:limit=N syntax from list-objects-filter.c. The goal is to collect\n> > blob OIDs that exceed the threshold and are currently locally present,\n> > meaning they resolve via find_pack_entry() or as a loose object, and are\n> > not already in a promised/missing state.\n>\n> How does this work when used on a partial clone with 'blob:none'?\n>\n\nRight, an object walk using traverse_commit_list() would\nhit missing trees and blobs and end up triggering a lazy-fetch storm,\nwhich is the opposite of what we want.\nThe enumeration needs to stay strictly local. We can do this by\npassing OBJECT_INFO_SKIP_FETCH_OBJECT to oid_object_info_extended(),\nsimilar to how remove_fetched_oids() works. That way we only look at\nblobs that are physically on disk and never touch the promisor\nmachinery.\n\n\n> > For remotes that do not advertise object-info, a conservative fallback\n> > applies: if a blob is reachable from a commit the remote has, and the\n> > remote's advertised filter covers that object, its presence on the\n> > remote is implied.\n>\n> How do we know a blob is reachable from a commit that the remote\n> advertises?\n>\n\nFair point,\nI think the cleaner approach is to just drop the local fallback\nentirely and make Protocol v2 object-info a hard requirement for v1.\nIt is stricter but much safer and we only drop what we can explicitly\nconfirm the remote has.\n\n> > 5.5 Safety Guards\n> >\n> >   - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,\n> >     REVERT_HEAD, or rebase-merge/ is present under .git/. Removing\n> >     objects mid-operation could corrupt an otherwise-recoverable state.\n> >\n>\n> But couldn't one of these operations start after we start the process of\n> cleanup?\n>\n\nYes, you are right. A merge or rebase could still start halfway through\nthe cleanup and lead to a race condition.\nSo for that, write-before-delete already handles this case. If a merge\nstarts mid-cleanup and needs a blob we just deleted, it will not hit a\nfatal \"object missing\" error, the OID is already in the .promisor file,\nso the lazy-fetch mechanism is called and fetches it back transparently.\nThe repository stays in a consistent state regardless of when the\nconcurrent operation starts.\nFor packfile removal, we can rely on the existing .keep and lock mechanisms\nnatively present in the repack machinery to prevent GC collisions.\n\n> >   - A new option on git backfill (e.g., --drop-blobs): backfill fills\n> >     objects in; this flag drains them back out. The symmetry is clean\n> >     and the command is already familiar to users working with partial\n> >     clones.\n> >\n>\n> I'm not sure if 'git backfill --drop-blobs' makes sense, since the\n> command talks about filling in data and the flag talks about cleaning up\n> data. But this seems to be the closest relative.\n>\n> >   - An opt-in git maintenance task for scheduled periodic space\n> >     reclamation, wrapping the same core logic.\n> >\n>\n> This would be my top pick, seems like data cleanup is definitely a\n> maintenance task.\n>\n\nAgreed on both, a flag that does the opposite of what the command\nname says is confusing, and git maintenance is a much more natural\nhome for something like this. I will focus the design discussion\naround making it a maintenance task.\n\n\nThanks again for the thorough review.\n\nSiddharth Shrimali\nr.siddharth.shrimali@gmail.com\n"},{"id":"539930","messageId":"CAGWgyh8bpiC2vmRS2X_v1KKxd1g8_qAnbWd+rLRoBxpfmmZtPg@mail.gmail.com","threadId":"65247","inReplyTo":"CAGWgyh_VmcNuay3AjsZ9zj9mP4g-c3Y9uDfuO5=kP9ix7Kdi3g@mail.gmail.com","subject":"Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones","fromName":"Siddharth Shrimali","fromEmail":"r.siddharth.shrimali@gmail.com","sentAt":"2026-03-25T11:46:48Z","receivedAt":"2026-03-25T11:47:27Z","isPatch":false,"sender":{"key":"r.siddharth.shrimali@gmail.com","avatar":null},"body":"Hello Git Community,\n\nI am Siddharth Shrimali, a contributor currently focused on the\npartial clone subsystem and test modernization (most recently the\nt0410 series). I would like to submit my updated proposal for GSoC\n2026: 'Improve Disk Space Recovery for Partial Clones.'\n\nThis is a revised version incorporating feedback from Karthik Nayak.\nI would appreciate any further feedback or suggestions on the\ntechnical approach outlined below.\n\n---\n\n1. PERSONAL INFORMATION\n\nName:       Siddharth Shrimali\nEmail:      r.siddharth.shrimali@gmail.com\nGitHub:     https://github.com/siddharthshrimali\nLinkedIn:   https://www.linkedin.com/in/siddharthshrimali/\nUniversity: Walchand College of Engineering, India\nDegree:     B.Tech, Computer Science and Engineering\n            (2nd year, 4th semester)\nTimezone:   IST (UTC+5:30)\n\n---\n\n2. PROJECT ABSTRACT\n\nGit's partial clone feature lets clients work with large repositories\nwithout downloading all objects upfront. Objects are fetched lazily from\npromisor remotes as needed, and git backfill can proactively fill them\nin. But the reverse does not exist: once blobs have been fetched, there\nis no way to drop them back to a promised-but-absent state. Over time,\nclients accumulate large locally-held blobs they no longer need, with no\npath to reclaim that space short of a full re-clone.\n\nThis project implements that missing direction: a mechanism to safely\nidentify blobs that are available on a promisor remote, remove them from\nlocal storage, and re-enter them into the promisor contract so that the\nexisting lazy-fetch infrastructure re-fetches them transparently on next\naccess. The result is a reversible partial clone lifecycle.\n\nProject size: 175 hours. Difficulty: Medium to Hard.\nLanguages: C, Shell (bash/POSIX sh).\nMentors: Christian Couder, Karthik Nayak, Justin Tobler,\n         Siddharth Asthana, Ayush Chandekar.\n\n---\n\n3. PROBLEM STATEMENT\n\nWhen a user clones with --filter=blob:none, Git records the remote as a\npromisor remote and fetches blobs lazily on demand via\npromisor_remote_get_direct(). Over months of work, through lazy fetches,\nexplicit backfills, and day-to-day operations, the local repository\naccumulates blobs that may no longer be needed. There is currently no\nmechanism to drop these blobs back to a promised-but-absent state. The\nonly option is a fresh re-clone, which is disruptive and expensive.\n\nThe gap is the reverse of git backfill: backfill moves objects from\npromisor-promised to locally-present and this project moves them back.\n\n---\n\n4. GETTING STARTED PROGRESS\n\nThe project ideas page listed specific steps to get started. I have\ncompleted all of them:\n\n  - Built Git from source and ran the full test suite.\n\n  - Set up a partial clone experiment: created a local promisor remote\n    via file://, cloned with --filter=blob:none, triggered lazy fetches,\n    and inspected the resulting .promisor sidecar files to understand\n    how the promisor contract works at the file level.\n\n  - Studied builtin/backfill.c end to end, along with\n    promisor-remote.c, list-objects-filter.c, and\n    Documentation/technical/partial-clone.txt.\n\n  - Submitted a micro-patch series (see Section 6) that was accepted\n    and merged to master.\n\n  - Read the partial clone and Protocol v2 documentation thoroughly.\n\n---\n\n5. TECHNICAL APPROACH\n\nThe full flow, at a high level:\n\n  1. Enumerate locally-held blobs above a size threshold.\n  2. Verify each candidate is available on the promisor remote.\n  3. Write a .promisor sidecar registering dropped OIDs as promised.\n     (This must happen before removal. See Section 5.3.)\n  4. Remove confirmed blobs from local storage.\n  5. Enforce safety guards.\n\n---\n\n5.1 Blob Enumeration\n\nA naive object walk using traverse_commit_list() on a partial clone\nwould hit missing trees and blobs and trigger a lazy-fetch storm, which\nis the opposite of what we want. The enumeration must stay strictly\nlocal.\n\nWe achieve this by passing OBJECT_INFO_SKIP_FETCH_OBJECT to\noid_object_info_extended(), the same approach used by\nremove_fetched_oids() in promisor-remote.c. This ensures we only\nevaluate blobs that are physically present on disk and never touch the\npromisor machinery during enumeration.\n\nThe goal is to collect blob OIDs that exceed a configurable size\nthreshold and are currently locally present, meaning they are not\nalready in a promised/missing state.\n\n---\n\n5.2 Remote Availability Verification\n\nBefore removing anything, we confirm the promisor remote holds each\ncandidate blob. We use the Protocol v2 \"object-info\" capability, which\nlets us query the remote for object metadata without downloading the\nobject bodies, in a single batched round trip.\n\nFor v1, Protocol v2 object-info will be a hard requirement. The\nconservative local-graph fallback considered in earlier drafts is\nbrittle in a partial clone context -- walking trees we do not have\nlocally would either fail or trigger fetches -- and the safety\nguarantee it provides is weaker. Making object-info mandatory ensures\nwe only drop what we can explicitly confirm the remote has.\n\n---\n\n5.3 Re-entering the Promisor Contract (write before delete)\n\nAfter removal, any access to a dropped blob must trigger a lazy fetch\nvia promisor_remote_get_direct() rather than a hard \"object missing\"\nerror. To achieve this, we write a new (possibly empty) packfile with a\n.promisor sidecar that lists the dropped OIDs.\n\nThe write-before-delete ordering is the most critical part of the\nimplementation. The .promisor sidecar must be written and fsynced before\nany object is physically removed. If a crash or interrupt happens in a\nwindow where an object is neither locally present nor promised, the\nrepository ends up in an unrecoverable state.\n\nThis ordering also handles the race condition where a concurrent\noperation (such as a merge or rebase) starts mid-cleanup and suddenly\nneeds a blob we just deleted. Because the OID is already in the\n.promisor file before deletion, the concurrent operation will not hit a\nfatal \"object missing\" error -- it will simply trigger the lazy-fetch\nmachinery and re-fetch the blob transparently. The repository stays in\na consistent state regardless of when the concurrent operation starts.\n\nFor packfile removal, the existing .keep and lock mechanisms in the\nrepack machinery provide protection against GC collisions.\n\n---\n\n5.4 Object Removal\n\nTwo cases must be handled:\n\n  a) Loose objects: compute the path via oid_to_path() (correct for\n     both SHA-1 and SHA-256) and unlink it.\n\n  b) Packed objects: a single object cannot be surgically removed from\n     a packfile. Instead, we drive the repack machinery in\n     builtin/repack.c to produce a new pack that excludes the target\n     OIDs. The --filter mechanism on git pack-objects already supports\n     object exclusion and can be driven with an explicit OID list.\n\n---\n\n5.5 Safety Guards\n\n  - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,\n    REVERT_HEAD, or rebase-merge/ is present under .git/. Removing\n    objects mid-operation could corrupt an otherwise-recoverable state.\n    Note that the write-before-delete ordering (Section 5.3) handles\n    the case where such an operation starts after cleanup has begun.\n\n  - Index safety: by default, refuse to drop blobs referenced by the\n    current index (read via read_cache()). An explicit flag can override\n    this for users who understand the implications.\n\n  - Remote reachability: refuse if the promisor remote is not reachable\n    at invocation time. Objects should not be dropped if they cannot be\n    immediately re-fetched when needed.\n\n  - Dry-run (--dry-run): enumerate and verify candidates, print what\n    would be removed and the estimated space reclaimed, without touching\n    any objects.\n\n---\n\n5.6 Command Placement\n\nThe project description explicitly leaves placement as an open design\nquestion. I will send a design email to the list during the bonding\nperiod and defer to community feedback before writing any code.\n\nBased on the discussion with Karthik, git maintenance is the\nprimary candidate. Space reclamation is fundamentally a housekeeping\noperation that fits naturally alongside existing maintenance tasks such\nas loose-objects and incremental-repack. It can be registered as an\nopt-in task (e.g., 'git maintenance run --task=drop-promisor-blobs')\nand scheduled for periodic execution. The design discussion will\nfinalize the exact interface.\n\n---\n\n6. MICRO-PROJECT\n\nAs the required GSoC qualification task, I submitted a two-patch series\nto modernize t/t3700-add.sh:\n\n  [PATCH v3 1/2] t3700: avoid suppressing git's exit code\n  [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics\n\n  Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>\n  Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/\n\nPatch 1 breaks pipelines of the form \"git foo | grep bar\" that silently\nswallow git's exit code, by redirecting git output to a temp file and\ngrepping that. Patch 2 replaces all plain grep / ! grep calls with the\ntest_grep helper for better failure diagnostics. The series was iterated\nto v3 incorporating structural feedback from Junio C Hamano.\n\nStatus: Merged to master.\n\nI have also contributed several other patches to Git since March 2026,\nincluding a C code fix in builtin/submodule--helper.c, test\nmodernizations in t3200 and t9123, work on builtin/backfill.c, and\nnotably in t/t0410-partial-clone.sh which is directly relevant to this\nproject. Full details are in Section 9.\n\n---\n\n7. DELIVERABLES\n\nThe deliverables section lists the concrete outputs I commit to\nproducing by the end of the project. This is what both the mentors and\nI can use to track progress and evaluate success.\n\n  1. A design email to git@vger.kernel.org during the bonding period\n     covering interface, command placement, safety model, and\n     write-before-delete ordering, with community sign-off before any\n     implementation begins.\n\n  2. Core logic: blob enumeration using OBJECT_INFO_SKIP_FETCH_OBJECT,\n     remote availability verification via Protocol v2 object-info,\n     .promisor sidecar write, and safe removal of loose and packed\n     objects.\n\n  3. Command integration: wired into the agreed-upon command surface\n     (git maintenance task as primary candidate).\n\n  4. A comprehensive test suite covering: basic drop and re-fetch,\n     dry-run, size filtering, safety guard rejections, SHA-256\n     compatibility, loose and packed object cases, and\n     write-before-delete crash safety.\n\n  5. Documentation: man page updates for the modified command(s) and\n     updates to Documentation/technical/partial-clone.txt describing the\n     new object lifecycle (fetch -> hold -> drop -> re-fetch).\n\n---\n\n8. TIMELINE (175 hours)\n\nIn Progress (Now - May 1)\n\n  - Continue submitting patches to the list while selections are pending.\n  - Read builtin/repack.c and the Protocol v2 object-info code so the\n    bonding period can focus on design rather than catching up on reading.\n  - Experiment with .promisor sidecar files hands-on and draft the design\n    email early so it is ready to send on day one of bonding.\n\nCommunity Bonding Period (May 1 - May 26)\n\n  - Go deeper into promisor-remote.c, builtin/backfill.c,\n    builtin/repack.c, and list-objects-filter.c with the specific goal\n    of understanding the exact call chains relevant to this project.\n  - Trace the full lazy-fetch path in a debugger end to end.\n  - Send the design proposal to the list and incorporate feedback before\n    writing any implementation code.\n\nWeek 1-2 (May 27 - June 9) | Blob enumeration\n\n  - Implement blob enumeration using oid_object_info_extended() with\n    OBJECT_INFO_SKIP_FETCH_OBJECT, collecting locally-present blob OIDs\n    above the size threshold without triggering lazy fetches.\n  - Tests: correct OIDs collected, size filter respected, already-missing\n    blobs excluded, no lazy fetches triggered during enumeration.\n\nWeek 3-4 (June 10 - June 23) | Remote verification\n\n  - Implement remote availability check via Protocol v2 object-info,\n    batched for efficiency.\n  - Tests: mock promisor remote, batch query correctness, rejection of\n    candidates the remote does not have.\n\nWeek 5-6 (June 24 - July 7) | Removal and re-promisor\n\n  - Implement .promisor sidecar write for candidate OIDs (before removal).\n  - Implement loose object removal (oid_to_path + unlink).\n  - Implement packed object removal by driving repack with an OID\n    exclusion list.\n  - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch\n    triggers correctly after drop, write-before-delete ordering safety.\n  - Send an interim patch series to the list for early review.\n\nWeek 7-8 (July 8 - July 21) | Safety checks and command wiring\n\n  - Implement all safety guards: in-progress operation detection, index\n    blob check, remote reachability check.\n  - Implement --dry-run.\n  - Wire core logic into the agreed command surface with flags:\n    --filter, --dry-run, --verbose.\n  - Midterm evaluation: working prototype, tests passing, docs draft.\n\nWeek 9-10 (July 22 - August 4) | Edge cases and review\n\n  - Handle edge cases: multiple worktrees, alternates, submodules with\n    their own promisor remotes, SHA-256 repositories.\n  - Address mailing list review feedback from the interim series.\n  - Performance testing on a repository with many locally-held blobs.\n\nWeek 11-12 (August 5 - August 18) | Documentation and cleanup\n\n  - Write and finalize man page documentation.\n  - Update Documentation/technical/partial-clone.txt.\n  - Final patch series submission incorporating all review feedback.\n\nBuffer (August 19 - August 25)\n\n  - Address any remaining review comments.\n  - Ensure all tests pass on CI for both SHA-1 and SHA-256.\n  - Final evaluation submission.\n\n---\n\n9. MY CONTRIBUTIONS TO GIT\n\nI have been contributing to Git since early March 2026. All patches\nwere submitted to git@vger.kernel.org and iterated through review with\nJunio C Hamano, Jeff King, Patrick Steinhardt, Derrick Stolee, and\nPhillip Wood.\n\nMerged to master:\n\n  1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)\n     Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/\n     (See Section 6 for full description. This is my micro-project.)\n     Status: Merged to master.\n\n  2. [PATCH] t9123: use test_when_finished for cleanup\n     Message-ID: <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/\n     Moved bare setup code in t9123 that ran outside any test block into\n     a proper test_expect_success block. Used test_when_finished to\n     register cleanup of the 'import' directory, ensuring it runs even\n     if the test fails, replacing a manual rm -rf that could be skipped\n     on failure.\n     Status: Merged to master.\n\n  3. [PATCH v2] t: fix \"that that\" typo in lib-unicode-nfc-nfd.sh\n     Message-ID: <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/\n     Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.\n     v1 over-reached by also fixing \"that that\" in two other files where\n     the phrasing was grammatically intentional; v2 correctly scoped the\n     change after re-reading the context.\n     Status: Merged to master.\n\n  4. [PATCH v2] submodule--helper: replace malloc with xmalloc\n     Message-ID: <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/\n     Replaced a raw malloc() in submodule_summary_callback() with\n     xmalloc(), which calls die() on allocation failure rather than\n     returning NULL and risking a NULL dereference. Also used sizeof(*temp)\n     instead of the explicit struct name, and dropped the redundant C cast\n     on the xmalloc() return. Improved the commit message in v2 to explain\n     the reasoning behind removing the cast, as requested by Junio.\n     Status: Merged to master.\n\n  5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID\n     Message-ID: <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/\n     Replaced a hardcoded 40-zero string in t/t3200-branch.sh with\n     $ZERO_OID to make the test hash-algorithm independent. The 40-char\n     string caused premature failure under SHA-256 because Git reports\n     \"malformed object name\" (40 vs 64 chars) before reaching the actual\n     object-type check. Using $ZERO_OID with test_grep for the \"must\n     point to a commit\" error ensures the test validates the right failure\n     mode. Suggested-by Patrick Steinhardt.\n     Status: Merged to master.\n\n  6. [PATCH v3] t0410: modernize delete_object helper\n     Message-ID: <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/\n     Modernized the delete_object helper in t/t0410-partial-clone.sh,\n     the primary test file for partial clones and directly relevant to\n     this project, by replacing a fragile manual sed-based object path\n     calculation with test_oid_to_path, making the helper correct for\n     both SHA-1 and SHA-256. Added 'local' variable declarations with\n     proper quoting for shell portability. Suggested-by Jeff King, who\n     also caught that unquoted 'local' assignments are flagged by\n     check-non-portable-shell. This patch prompted Junio to add a note\n     to Documentation/SubmittingPatches reminding contributors to run\n     'make test' from the top level.\n     Helped-by: Pushkar Singh.\n     Status: Merged to master.\n\n  7. [PATCH v3] backfill: handle unexpected arguments\n     Message-ID: <20260321031643.5185-1-r.siddharth.shrimali@gmail.com>\n     Link: https://lore.kernel.org/git/20260321031643.5185-1-r.siddharth.shrimali@gmail.com/\n     Added a check in cmd_backfill() to report an error and print the\n     short usage string if any unexpected non-option arguments are passed.\n     Iterated to v3 incorporating feedback from Junio C Hamano (error\n     message style, usage() vs usage_with_options()) and Phillip Wood\n     (test_grep usage in tests). The patch was dropped after Derrick\n     Stolee pointed out an in-flight series adding revision arguments to\n     git backfill, making a hard rejection of positional arguments wrong.\n     However, the report directly motivated Derrick to add a test for\n     this behavior in his series, where I am credited as Reported-by.\n     Status: Dropped, but credited as Reported-by in Derrick Stolee's\n     series at:\n     https://lore.kernel.org/git/9699650aa7dc04cf1cdc26803caa8304b29c1662.1774266019.git.gitgitgadget@gmail.com/\n\nCollaboration:\n\n  8. Helped with [PATCH v5] help: cleanup the construction of keys_uniq\n     Message-ID: <20260311192453.62213-1-amishhhaaaa@gmail.com>\n     Link: https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/\n     (patch by Amisha Chhajed). Suggested moving strbuf initialization\n     and release out of the inner loop to reuse the same buffer across\n     iterations, avoiding repeated alloc/free overhead. Listed as\n     Suggested-by in the patch.\n     Status: Merged to master.\n\nPatch #6 is worth highlighting separately: working on\nt0410-partial-clone.sh required reading the entire file carefully,\nunderstanding how promisor remote tests are structured, and reasoning\nabout hash-algorithm independence in the partial clone context. That\nwork gives me a direct head start on the test infrastructure for this\nproject.\n\n---\n\n10. RELEVANT EXPERIENCE\n\nC and shell (bash/POSIX sh) are my primary languages for this project.\nMy submodule--helper patch (patch #4 above) demonstrates comfort with C\nmemory management patterns in Git's codebase (xmalloc/die() convention,\nsizeof(*ptr) idiom, implicit void* promotion). My test patches\ndemonstrate hands-on familiarity with Git's shell test framework:\ntest_grep, test_oid_to_path, test_must_fail, test_when_finished, and the\nportability rules enforced by check-non-portable-shell. I am also\nfamiliar with Go, which gives me a general systems programming background.\n\nI have submitted patches through multiple review cycles on the mailing\nlist, incorporated feedback from senior maintainers, and am comfortable\nwith the in-reply-to threading conventions and interdiff-based review\nprocess.\n\nYou can find all my patches at:\nhttps://lore.kernel.org/git/?q=Siddharth+Shrimali\n\n---\n\n11. AVAILABILITY\n\nI can commit 25-30 hours per week to GSoC during the coding period\n(May-August). My 4th semester exams conclude by mid-May, so I will be\npartly available at the start and fully available from mid-May onward\nwith no academic conflicts through the end of summer. I have no\ninternships, part-time work, or other commitments planned for this\nperiod.\n\nI will send a brief weekly status update to my mentors and post\nquestions or design discussions to the mailing list as they come up, to\nstay unblocked and keep the community informed of progress.\n\n---\n\n12. WHY THIS PROJECT\n\nI have been using Git for a while, but it was only when I started\ncontributing patches that I actually began reading its internals. Partial\nclone was one of the first areas I looked at carefully, partly because of\nthe t0410 patch and partly out of genuine curiosity. As I read\npromisor-remote.c and backfill.c, the missing reverse direction became\nclear. Backfill exists, but its counterpart does not. That felt like an\nunfinished thought.\n\nI find the write-before-delete ordering problem genuinely interesting. It\nis one of those situations where the obvious implementation is wrong, and\ngetting it right requires careful reasoning about crash recovery. The same\nis true for packed object removal, where you cannot simply delete one\nobject and call it done. These are exactly the kinds of problems I enjoy\nthinking through.\n\nI also want to be candid that I do not yet know every corner of this\ncodebase. There will be parts of the implementation where I will need\nguidance. That is exactly why I want to use the bonding period to read,\nexperiment, and deepen my understanding before writing any implementation\ncode, and why the design discussion on the list matters to me. I would\nmuch rather understand the problem thoroughly than rush into code that\nlater has to be thrown away.\n\n---\n\nThank you for your time and for considering my proposal. I have genuinely\nenjoyed contributing to Git over the past few weeks and I am looking\nforward to doing more of it, with or without GSoC.\nIf you have any questions or if there is anything missing from this proposal,\nI am happy to follow up on the list.\n\n---\n\nSiddharth Shrimali\nr.siddharth.shrimali@gmail.com\n"}]}