Volume XXII, number 280Wednesday, October 7, 2026Latest message 56 minutes ago

The Git List

News and archive of git@vger.kernel.org, since April 2005

[GSoC Proposal] Improve Disk Space Recovery for Partial Clones

4 messages between Mar 14, 2026 and Mar 25, 2026, from Siddharth Shrimali, Karthik Nayak.

Plain Markdown or JSON for tools and agents.

Siddharth ShrimaliMar 14, 2026, 20:34 UTC on lore
Hello Git Community,

I am Siddharth Shrimali, a contributor currently focused on the partial clone subsystem and test modernization (most recently the t0410 series). I would like to submit my proposal for GSoC 2026: 'Improve Disk Space Recovery for Partial Clones.'

I've had a great experience collaborating with the community so far and would appreciate any feedback or suggestions on the technical approach outlined below.

---
1. PERSONAL INFORMATION
Name:       Siddharth Shrimali
Email:        r.siddharth.shrimali@gmail.com
GitHub:      https://github.com/siddharthshrimali
LinkedIn:    https://www.linkedin.com/in/siddharthshrimali/
University:  Walchand College of Engineering, India
Degree:      B.Tech, Computer Science and Engineering
             (2nd year, 4th semester)
Timezone:  IST (UTC+5:30)
---
2. PROJECT ABSTRACT

Git's partial clone feature lets clients work with large repositories without downloading all objects upfront. Objects are fetched lazily from promisor remotes as needed, and git backfill can proactively fill them in. But the reverse does not exist: once blobs have been fetched, there is no way to drop them back to a promised-but-absent state. Over time, clients accumulate large locally-held blobs they no longer need, with no path to reclaim that space short of a full re-clone.

This project implements that missing direction: a mechanism to safely identify blobs that are available on a promisor remote, remove them from local storage, and re-enter them into the promisor contract so that the existing lazy-fetch infrastructure re-fetches them transparently on next access. The result is a reversible partial clone lifecycle.

Project size: 175 hours. Difficulty: Medium to Hard.
Languages: C, shell (bash/POSIX sh).
Mentors: Christian Couder, Karthik Nayak, Justin Tobler,
         Siddharth Asthana, Ayush Chandekar.
---
3. PROBLEM STATEMENT

When a user clones with --filter=blob:none, Git records the remote as a promisor remote and fetches blobs lazily on demand via promisor_remote_get_direct(). Over months of work, through lazy fetches, explicit backfills, and day-to-day operations, the local repository accumulates blobs that may no longer be needed. There is currently no mechanism to drop these blobs back to a promised-but-absent state. The only option is a fresh re-clone, which is disruptive and expensive.

The gap is the reverse of git backfill: backfill moves objects from promisor-promised to locally-present and this project moves them back.

---
4. GETTING STARTED PROGRESS

The project ideas page listed specific steps to get started. I have completed all of them:

  - Built Git from source and ran the full test suite.
  - Set up a partial clone experiment: created a local promisor remote
    via file://, cloned with --filter=blob:none, triggered lazy fetches,
    and inspected the resulting .promisor sidecar files to understand
    how the promisor contract works at the file level.
  - Studied builtin/backfill.c end to end, along with
    promisor-remote.c, list-objects-filter.c, and
    Documentation/technical/partial-clone.txt.
  - Submitted a micro-patch series (see Section 6) that was accepted
    and merged to master.
  - Read the partial clone and Protocol v2 documentation thoroughly.
---
5. TECHNICAL APPROACH
The full flow, at a high level:
  1. Enumerate locally-held blobs above a size threshold.
  2. Verify each candidate is available on the promisor remote.
  3. Write a .promisor sidecar registering dropped OIDs as promised.
     (This must happen before removal. See Section 5.3.)
  4. Remove confirmed blobs from local storage.
  5. Enforce safety guards.
---
5.1 Blob Enumeration

Walk the object graph from all local refs using traverse_commit_list() in list-objects.c, with a size-based filter consistent with the existing blob:limit=N syntax from list-objects-filter.c. The goal is to collect blob OIDs that exceed the threshold and are currently locally present, meaning they resolve via find_pack_entry() or as a loose object, and are not already in a promised/missing state.

---
5.2 Remote Availability Verification

Before removing anything, we confirm the promisor remote holds each candidate blob. The cleanest way to do this is the Protocol v2 "object-info" capability, which lets us query the remote for object metadata without downloading the object bodies, in a single batched round trip.

For remotes that do not advertise object-info, a conservative fallback applies: if a blob is reachable from a commit the remote has, and the remote's advertised filter covers that object, its presence on the remote is implied.

---
5.3 Re-entering the Promisor Contract (write before delete)

After removal, any access to a dropped blob must trigger a lazy fetch via promisor_remote_get_direct() rather than a hard "object missing" error. To achieve this, we write a new (possibly empty) packfile with a .promisor sidecar that lists the dropped OIDs.

The write-before-delete ordering is the most critical part of the implementation. The .promisor sidecar must be written and fsynced before any object is physically removed. If a crash or interrupt happens in a window where an object is neither locally present nor promised, the repository ends up in an unrecoverable state.

---
5.4 Object Removal
Two cases must be handled:
  a) Loose objects: compute the path via oid_to_path() (correct for
     both SHA-1 and SHA-256) and unlink it.
  b) Packed objects: a single object cannot be surgically removed from
     a packfile. Instead, we drive the repack machinery in
     builtin/repack.c to produce a new pack that excludes the target
     OIDs. The --filter mechanism on git pack-objects already supports
     object exclusion and can be driven with an explicit OID list.
---
5.5 Safety Guards
  - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,
    REVERT_HEAD, or rebase-merge/ is present under .git/. Removing
    objects mid-operation could corrupt an otherwise-recoverable state.
  - Index safety: by default, refuse to drop blobs referenced by the
    current index (read via read_cache()). An explicit flag can override
    this for users who understand the implications.
  - Remote reachability: refuse if the promisor remote is not reachable
    at invocation time. Objects should not be dropped if they cannot be
    immediately re-fetched when needed.
  - Dry-run (--dry-run): enumerate and verify candidates, print what
    would be removed and the estimated space reclaimed, without touching
    any objects.
---
5.6 Command Placement

The project description explicitly leaves placement as an open design question. I will send a design email to the list during the bonding period and defer to community feedback before writing any code. My current thinking is:

  - A new option on git backfill (e.g., --drop-blobs): backfill fills
    objects in; this flag drains them back out. The symmetry is clean
    and the command is already familiar to users working with partial
    clones.
  - An opt-in git maintenance task for scheduled periodic space
    reclamation, wrapping the same core logic.
---
6. MICRO-PROJECT

As the required GSoC qualification task, I submitted a two-patch series to modernize t/t3700-add.sh:

  [PATCH v3 1/2] t3700: avoid suppressing git's exit code
  [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics
  Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
  Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/

Patch 1 breaks pipelines of the form "git foo | grep bar" that silently swallows git's exit code, by redirecting git output to a temp file and grepping that. Patch 2 replaces all plain grep / ! grep calls with the test_grep helper for better failure diagnostics. The series was iterated to v3 incorporating structural feedback from Junio C Hamano.

Status: Merged to master.

I have also contributed several other patches to Git since February 2026, including a C code fix in builtin/submodule--helper.c, test modernizations in t3200 and t9123, and notably in t/t0410-partial-clone.sh which is directly relevant to this project. Full details are in Section 9.

---
7. DELIVERABLES

The deliverables section lists the concrete outputs I commit to producing by the end of the project. This is what both the mentors and I can use to track progress and evaluate success.

  1. A design email to git@vger.kernel.org during the bonding period
     covering interface, command placement, safety model, and
     write-before-delete ordering, with community sign-off before any
     implementation begins.
  2. Core logic: blob enumeration, remote availability verification
     (Protocol v2 object-info + fallback), .promisor sidecar write, and
     safe removal of loose and packed objects.
  3. Command integration: wired into the agreed-upon command surface
     (git backfill option and/or git maintenance task).
  4. A comprehensive test suite covering: basic drop and re-fetch,
     dry-run, size filtering, safety guard rejections, SHA-256
     compatibility, loose and packed object cases, and
     write-before-delete crash safety.
  5. Documentation: man page updates for the modified command(s) and
     updates to Documentation/technical/partial-clone.txt describing the
     new object lifecycle (fetch -> hold -> drop -> re-fetch).
---
8. TIMELINE (175 hours)
In Progress (Now - May 1)
  - Continue submitting patches to the list while selections are pending.
  - Read builtin/repack.c and the Protocol v2 object-info code so the
    bonding period can focus on design rather than catching up on reading.
  - Experiment with .promisor sidecar files hands-on and draft the design
    email early so it is ready to send on day one of bonding.
Community Bonding Period (May 1 - May 26)
  - Go deeper into promisor-remote.c, builtin/backfill.c,
    builtin/repack.c, and list-objects-filter.c with the specific goal
    of understanding the exact call chains relevant to this project.
  - Trace the full lazy-fetch path in a debugger end to end.
  - Send the design proposal to the list and incorporate feedback before
    writing any implementation code.
Week 1-2 (May 27 - June 9) | Blob enumeration
  - Implement blob enumeration via traverse_commit_list() with a size
    filter, collecting locally-present blob OIDs above the threshold.
  - Tests: correct OIDs collected, size filter respected, already-missing
    blobs excluded.
Week 3-4 (June 10 - June 23) | Remote verification
  - Implement remote availability check via Protocol v2 object-info,
    batched for efficiency.
  - Implement the conservative reachability-implies-presence fallback.
  - Tests: mock promisor remote, batch query correctness, fallback path.
Week 5-6 (June 24 - July 7) | Removal and re-promisor
  - Implement .promisor sidecar write for candidate OIDs (before removal).
  - Implement loose object removal (oid_to_path + unlink).
  - Implement packed object removal by driving repack with an OID
    exclusion list.
  - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch
    triggers correctly after drop, write-before-delete ordering safety.
  - Send an interim patch series to the list for early review.
Week 7-8 (July 8 - July 21) | Safety checks and command wiring
  - Implement all safety guards: in-progress operation detection, index
    blob check, remote reachability check.
  - Implement --dry-run.
  - Wire core logic into the agreed command surface with flags:
    --filter, --dry-run, --verbose.
  - Midterm evaluation: working prototype, tests passing, docs draft.
Week 9-10 (July 22 - August 4) | Edge cases and review
  - Handle edge cases: multiple worktrees, alternates, submodules with
    their own promisor remotes, SHA-256 repositories.
  - Address mailing list review feedback from the interim series.
  - Performance testing on a repository with many locally-held blobs.
Week 11-12 (August 5 - August 18) | Documentation and cleanup
  - Write and finalize man page documentation.
  - Update Documentation/technical/partial-clone.txt.
  - Final patch series submission incorporating all review feedback.
Buffer (August 19 - August 25)
  - Address any remaining review comments.
  - Ensure all tests pass on CI for both SHA-1 and SHA-256.
  - Final evaluation submission.
---
9. MY CONTRIBUTIONS TO GIT

I have been contributing to Git since early March 2026. All patches were submitted to git@vger.kernel.org and iterated through review with Junio C Hamano, Jeff King, and Patrick Steinhardt.

Merged to master:
  1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)
     <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
     (See Section 6 for full description. This is my micro-project.)
     https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/
     Status: Merged to master.
  2. [PATCH] t9123: use test_when_finished for cleanup
     <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>
     Moved bare setup code in t9123 that ran outside any test block into
     a proper test_expect_success block. Used test_when_finished to
     register cleanup of the 'import' directory, ensuring it runs even
     if the test fails, replacing a manual rm -rf that could be skipped
     on failure.
     https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/
     Status: Merged to master.
  3. [PATCH v2] t: fix "that that" typo in lib-unicode-nfc-nfd.sh
     <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>
     Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.
     v1 over-reached by also fixing "that that" in two other files where
     the phrasing was grammatically intentional; v2 correctly scoped the
     change after re-reading the context.
     https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/
     Status: Merged to master.
Queued for master / next:
  4. [PATCH v2] submodule--helper: replace malloc with xmalloc
     <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>
     Replaced a raw malloc() in submodule_summary_callback() with
     xmalloc(), which calls die() on allocation failure rather than
     returning NULL and risking a NULL dereference. Also used sizeof(*temp)
     instead of the explicit struct name, and dropped the redundant C cast
     on the xmalloc() return. Improved the commit message in v2 to explain
     the reasoning behind removing the cast, as requested by Junio.
     https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/
     Status: Will merge to next.
  5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID
     <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>
     Replaced a hardcoded 40-zero string in t/t3200-branch.sh with
     $ZERO_OID to make the test hash-algorithm independent. The 40-char
     string caused premature failure under SHA-256 because Git reports
     "malformed object name" (40 vs 64 chars) before reaching the actual
     object-type check. Using $ZERO_OID with test_grep for the "must
     point to a commit" error ensures the test validates the right failure
     mode. Suggested-by Patrick Steinhardt.
     https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/
     Status: Will merge to master.
  6. [PATCH v3] t0410: modernize delete_object helper
     <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>
     Modernized the delete_object helper in t/t0410-partial-clone.sh,
     the primary test file for partial clones and directly relevant to
     this project, by replacing a fragile manual sed-based object path
     calculation with test_oid_to_path, making the helper correct for
     both SHA-1 and SHA-256. Added 'local' variable declarations with
     proper quoting for shell portability. Suggested-by Jeff King, who
     also caught that unquoted 'local' assignments are flagged by
     check-non-portable-shell. This patch prompted Junio to add a note
     to Documentation/SubmittingPatches reminding contributors to run
     'make test' from the top level.
     Helped-by: Pushkar Singh.
     https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/
     Status: Good to go to next.
Collaboration:
  7. Helped with [PATCH v5] help: cleanup the construction of keys_uniq
     <20260311192453.62213-1-amishhhaaaa@gmail.com>
     (patch by Amisha Chhajed). Suggested moving strbuf initialization
     and release out of the inner loop to reuse the same buffer across
     iterations, avoiding repeated alloc/free overhead. Listed as
     Suggested-by in the patch.
     https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/
     Status: Will merge to next.

Patch #6 is worth highlighting separately: working on t0410-partial-clone.sh required reading the entire file carefully, understanding how promisor remote tests are structured, and reasoning about hash-algorithm independence in the partial clone context. That work gives me a direct head start on the test infrastructure for this project.

---
10. RELEVANT EXPERIENCE

C and shell (bash/POSIX sh) are my primary languages for this project. My submodule--helper patch (patch #4 above) demonstrates comfort with C memory management patterns in Git's codebase (xmalloc/die() convention, sizeof(*ptr) idiom, implicit void* promotion). My test patches demonstrate hands-on familiarity with Git's shell test framework: test_grep, test_oid_to_path, test_must_fail, test_when_finished, and the portability rules enforced by check-non-portable-shell. I am also familiar with Go, which gives me a general systems programming background.

I have submitted patches through multiple review cycles on the mailing list, incorporated feedback from senior maintainers, and am comfortable with the in-reply-to threading conventions and interdiff-based review process.

You can find all my patches at: https://lore.kernel.org/git/?q=Siddharth+Shrimali

---
11. AVAILABILITY

I can commit 25-30 hours per week to GSoC during the coding period (May-August). My 4th semester exams conclude by mid-May, so I will be partly available at the start and fully available from mid-May onward with no academic conflicts through the end of summer. I have no internships, part-time work, or other commitments planned for this period.

I will send a brief weekly status update to my mentors and post questions or design discussions to the mailing list as they come up, to stay unblocked and keep the community informed of progress.

---
12. WHY THIS PROJECT

I have been using Git for a while, but it was only when I started contributing patches that I actually began reading its internals. Partial clone was one of the first areas I looked at carefully, partly because of the t0410 patch and partly out of genuine curiosity. As I read promisor-remote.c and backfill.c, the missing reverse direction became clear. Backfill exists, but its counterpart does not. That felt like an unfinished thought.

I find the write-before-delete ordering problem genuinely interesting. It is one of those situations where the obvious implementation is wrong, and getting it right requires careful reasoning about crash recovery. The same is true for packed object removal, where you cannot simply delete one object and call it done. These are exactly the kinds of problems I enjoy thinking through.

I also want to be candid that I do not yet know every corner of this codebase. There will be parts of the implementation where I will need guidance. That is exactly why I want to use the bonding period to read, experiment, and deepen my understanding before writing any implementation code, and why the design discussion on the list matters to me. I would much rather understand the problem thoroughly than rush into code that later has to be thrown away.

---

Thank you for your time and for considering my proposal. I have genuinely enjoyed contributing to Git over the past few weeks and I am looking forward to doing more of it, with or without GSoC. If you have any questions or if there is anything missing from this proposal, I am happy to follow up on the list.

---

Siddharth Shrimali r.siddharth.shrimali@gmail.com

Karthik NayakMar 16, 2026, 23:05 UTC in reply to Siddharth Shrimali on lore

Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones

Siddharth Shrimali <r.siddharth.shrimali@gmail.com> writes:
Hello Siddharth,
Show 108 quoted lines
> Hello Git Community,
>
> I am Siddharth Shrimali, a contributor currently focused on the
> partial clone subsystem and test modernization (most recently the
> t0410 series). I would like to submit my proposal for GSoC 2026:
> 'Improve Disk Space Recovery for Partial Clones.'
>
> I've had a great experience collaborating with the community so far
> and would appreciate any feedback or suggestions on the technical
> approach outlined below.
>
> ---
>
> 1. PERSONAL INFORMATION
>
> Name:       Siddharth Shrimali
> Email:        r.siddharth.shrimali@gmail.com
> GitHub:      https://github.com/siddharthshrimali
> LinkedIn:    https://www.linkedin.com/in/siddharthshrimali/
> University:  Walchand College of Engineering, India
> Degree:      B.Tech, Computer Science and Engineering
>              (2nd year, 4th semester)
> Timezone:  IST (UTC+5:30)
>
> ---
>
> 2. PROJECT ABSTRACT
>
> Git's partial clone feature lets clients work with large repositories
> without downloading all objects upfront. Objects are fetched lazily from
> promisor remotes as needed, and git backfill can proactively fill them
> in. But the reverse does not exist: once blobs have been fetched, there
> is no way to drop them back to a promised-but-absent state. Over time,
> clients accumulate large locally-held blobs they no longer need, with no
> path to reclaim that space short of a full re-clone.
>
> This project implements that missing direction: a mechanism to safely
> identify blobs that are available on a promisor remote, remove them from
> local storage, and re-enter them into the promisor contract so that the
> existing lazy-fetch infrastructure re-fetches them transparently on next
> access. The result is a reversible partial clone lifecycle.
>
> Project size: 175 hours. Difficulty: Medium to Hard.
> Languages: C, shell (bash/POSIX sh).
> Mentors: Christian Couder, Karthik Nayak, Justin Tobler,
>          Siddharth Asthana, Ayush Chandekar.
>
> ---
>
> 3. PROBLEM STATEMENT
>
> When a user clones with --filter=blob:none, Git records the remote as a
> promisor remote and fetches blobs lazily on demand via
> promisor_remote_get_direct(). Over months of work, through lazy fetches,
> explicit backfills, and day-to-day operations, the local repository
> accumulates blobs that may no longer be needed. There is currently no
> mechanism to drop these blobs back to a promised-but-absent state. The
> only option is a fresh re-clone, which is disruptive and expensive.
>
> The gap is the reverse of git backfill: backfill moves objects from
> promisor-promised to locally-present and this project moves them back.
>
> ---
>
> 4. GETTING STARTED PROGRESS
>
> The project ideas page listed specific steps to get started. I have
> completed all of them:
>
>   - Built Git from source and ran the full test suite.
>
>   - Set up a partial clone experiment: created a local promisor remote
>     via file://, cloned with --filter=blob:none, triggered lazy fetches,
>     and inspected the resulting .promisor sidecar files to understand
>     how the promisor contract works at the file level.
>
>   - Studied builtin/backfill.c end to end, along with
>     promisor-remote.c, list-objects-filter.c, and
>     Documentation/technical/partial-clone.txt.
>
>   - Submitted a micro-patch series (see Section 6) that was accepted
>     and merged to master.
>
>   - Read the partial clone and Protocol v2 documentation thoroughly.
>
> ---
>
> 5. TECHNICAL APPROACH
>
> The full flow, at a high level:
>
>   1. Enumerate locally-held blobs above a size threshold.
>   2. Verify each candidate is available on the promisor remote.
>   3. Write a .promisor sidecar registering dropped OIDs as promised.
>      (This must happen before removal. See Section 5.3.)
>   4. Remove confirmed blobs from local storage.
>   5. Enforce safety guards.
>
> ---
>
> 5.1 Blob Enumeration
>
> Walk the object graph from all local refs using traverse_commit_list()
> in list-objects.c, with a size-based filter consistent with the existing
> blob:limit=N syntax from list-objects-filter.c. The goal is to collect
> blob OIDs that exceed the threshold and are currently locally present,
> meaning they resolve via find_pack_entry() or as a loose object, and are
> not already in a promised/missing state.
How does this work when used on a partial clone with 'blob:none'?
Show 9 quoted lines
> ---
>
> 5.2 Remote Availability Verification
>
> Before removing anything, we confirm the promisor remote holds each
> candidate blob. The cleanest way to do this is the Protocol v2
> "object-info" capability, which lets us query the remote for object
> metadata without downloading the object bodies, in a single batched
> round trip
Makes sense.
Show 5 quoted lines
>
> For remotes that do not advertise object-info, a conservative fallback
> applies: if a blob is reachable from a commit the remote has, and the
> remote's advertised filter covers that object, its presence on the
> remote is implied.

How do we know a blob is reachable from a commit that the remote advertises?

Show 16 quoted lines
>
> ---
>
> 5.3 Re-entering the Promisor Contract (write before delete)
>
> After removal, any access to a dropped blob must trigger a lazy fetch
> via promisor_remote_get_direct() rather than a hard "object missing"
> error. To achieve this, we write a new (possibly empty) packfile with a
> .promisor sidecar that lists the dropped OIDs.
>
> The write-before-delete ordering is the most critical part of the
> implementation. The .promisor sidecar must be written and fsynced before
> any object is physically removed. If a crash or interrupt happens in a
> window where an object is neither locally present nor promised, the
> repository ends up in an unrecoverable state.
>
Yeah, this makes sense too.
Show 23 quoted lines
> ---
>
> 5.4 Object Removal
>
> Two cases must be handled:
>
>   a) Loose objects: compute the path via oid_to_path() (correct for
>      both SHA-1 and SHA-256) and unlink it.
>
>   b) Packed objects: a single object cannot be surgically removed from
>      a packfile. Instead, we drive the repack machinery in
>      builtin/repack.c to produce a new pack that excludes the target
>      OIDs. The --filter mechanism on git pack-objects already supports
>      object exclusion and can be driven with an explicit OID list.
>
> ---
>
> 5.5 Safety Guards
>
>   - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,
>     REVERT_HEAD, or rebase-merge/ is present under .git/. Removing
>     objects mid-operation could corrupt an otherwise-recoverable state.
>

But couldn't one of these operations start after we start the process of cleanup?

Show 12 quoted lines
>   - Index safety: by default, refuse to drop blobs referenced by the
>     current index (read via read_cache()). An explicit flag can override
>     this for users who understand the implications.
>
>   - Remote reachability: refuse if the promisor remote is not reachable
>     at invocation time. Objects should not be dropped if they cannot be
>     immediately re-fetched when needed.
>
>   - Dry-run (--dry-run): enumerate and verify candidates, print what
>     would be removed and the estimated space reclaimed, without touching
>     any objects.
>
Nice.
Show 14 quoted lines
> ---
>
> 5.6 Command Placement
>
> The project description explicitly leaves placement as an open design
> question. I will send a design email to the list during the bonding
> period and defer to community feedback before writing any code. My
> current thinking is:
>
>   - A new option on git backfill (e.g., --drop-blobs): backfill fills
>     objects in; this flag drains them back out. The symmetry is clean
>     and the command is already familiar to users working with partial
>     clones.
>

I'm not sure if 'git backfill --drop-blobs' makes sense, since the command talks about filling in data and the flag talks about cleaning up data. But this seems to be the closest relative.

>   - An opt-in git maintenance task for scheduled periodic space
>     reclamation, wrapping the same core logic.
>

This would be my top pick, seems like data cleanup is definitely a maintenance task.

Show 296 quoted lines
> ---
>
> 6. MICRO-PROJECT
>
> As the required GSoC qualification task, I submitted a two-patch series
> to modernize t/t3700-add.sh:
>
>   [PATCH v3 1/2] t3700: avoid suppressing git's exit code
>   [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics
>
>   Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
>   Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/
>
> Patch 1 breaks pipelines of the form "git foo | grep bar" that silently
> swallows git's exit code, by redirecting git output to a temp file and
> grepping that. Patch 2 replaces all plain grep / ! grep calls with the
> test_grep helper for better failure diagnostics. The series was iterated
> to v3 incorporating structural feedback from Junio C Hamano.
>
> Status: Merged to master.
>
> I have also contributed several other patches to Git since February
> 2026, including a C code fix in builtin/submodule--helper.c, test
> modernizations in t3200 and t9123, and notably in
> t/t0410-partial-clone.sh which is directly relevant to this project.
> Full details are in Section 9.
>
> ---
>
> 7. DELIVERABLES
>
> The deliverables section lists the concrete outputs I commit to
> producing by the end of the project. This is what both the mentors and
> I can use to track progress and evaluate success.
>
>   1. A design email to git@vger.kernel.org during the bonding period
>      covering interface, command placement, safety model, and
>      write-before-delete ordering, with community sign-off before any
>      implementation begins.
>
>   2. Core logic: blob enumeration, remote availability verification
>      (Protocol v2 object-info + fallback), .promisor sidecar write, and
>      safe removal of loose and packed objects.
>
>   3. Command integration: wired into the agreed-upon command surface
>      (git backfill option and/or git maintenance task).
>
>   4. A comprehensive test suite covering: basic drop and re-fetch,
>      dry-run, size filtering, safety guard rejections, SHA-256
>      compatibility, loose and packed object cases, and
>      write-before-delete crash safety.
>
>   5. Documentation: man page updates for the modified command(s) and
>      updates to Documentation/technical/partial-clone.txt describing the
>      new object lifecycle (fetch -> hold -> drop -> re-fetch).
>
> ---
>
> 8. TIMELINE (175 hours)
>
> In Progress (Now - May 1)
>
>   - Continue submitting patches to the list while selections are pending.
>   - Read builtin/repack.c and the Protocol v2 object-info code so the
>     bonding period can focus on design rather than catching up on reading.
>   - Experiment with .promisor sidecar files hands-on and draft the design
>     email early so it is ready to send on day one of bonding.
>
> Community Bonding Period (May 1 - May 26)
>
>   - Go deeper into promisor-remote.c, builtin/backfill.c,
>     builtin/repack.c, and list-objects-filter.c with the specific goal
>     of understanding the exact call chains relevant to this project.
>   - Trace the full lazy-fetch path in a debugger end to end.
>   - Send the design proposal to the list and incorporate feedback before
>     writing any implementation code.
>
> Week 1-2 (May 27 - June 9) | Blob enumeration
>
>   - Implement blob enumeration via traverse_commit_list() with a size
>     filter, collecting locally-present blob OIDs above the threshold.
>   - Tests: correct OIDs collected, size filter respected, already-missing
>     blobs excluded.
>
> Week 3-4 (June 10 - June 23) | Remote verification
>
>   - Implement remote availability check via Protocol v2 object-info,
>     batched for efficiency.
>   - Implement the conservative reachability-implies-presence fallback.
>   - Tests: mock promisor remote, batch query correctness, fallback path.
>
> Week 5-6 (June 24 - July 7) | Removal and re-promisor
>
>   - Implement .promisor sidecar write for candidate OIDs (before removal).
>   - Implement loose object removal (oid_to_path + unlink).
>   - Implement packed object removal by driving repack with an OID
>     exclusion list.
>   - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch
>     triggers correctly after drop, write-before-delete ordering safety.
>   - Send an interim patch series to the list for early review.
>
> Week 7-8 (July 8 - July 21) | Safety checks and command wiring
>
>   - Implement all safety guards: in-progress operation detection, index
>     blob check, remote reachability check.
>   - Implement --dry-run.
>   - Wire core logic into the agreed command surface with flags:
>     --filter, --dry-run, --verbose.
>   - Midterm evaluation: working prototype, tests passing, docs draft.
>
> Week 9-10 (July 22 - August 4) | Edge cases and review
>
>   - Handle edge cases: multiple worktrees, alternates, submodules with
>     their own promisor remotes, SHA-256 repositories.
>   - Address mailing list review feedback from the interim series.
>   - Performance testing on a repository with many locally-held blobs.
>
> Week 11-12 (August 5 - August 18) | Documentation and cleanup
>
>   - Write and finalize man page documentation.
>   - Update Documentation/technical/partial-clone.txt.
>   - Final patch series submission incorporating all review feedback.
>
> Buffer (August 19 - August 25)
>
>   - Address any remaining review comments.
>   - Ensure all tests pass on CI for both SHA-1 and SHA-256.
>   - Final evaluation submission.
>
> ---
>
> 9. MY CONTRIBUTIONS TO GIT
>
> I have been contributing to Git since early March 2026. All patches
> were submitted to git@vger.kernel.org and iterated through review with
> Junio C Hamano, Jeff King, and Patrick Steinhardt.
>
> Merged to master:
>
>   1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)
>      <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
>      (See Section 6 for full description. This is my micro-project.)
>      https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/
>      Status: Merged to master.
>
>   2. [PATCH] t9123: use test_when_finished for cleanup
>      <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>
>      Moved bare setup code in t9123 that ran outside any test block into
>      a proper test_expect_success block. Used test_when_finished to
>      register cleanup of the 'import' directory, ensuring it runs even
>      if the test fails, replacing a manual rm -rf that could be skipped
>      on failure.
>      https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/
>      Status: Merged to master.
>
>   3. [PATCH v2] t: fix "that that" typo in lib-unicode-nfc-nfd.sh
>      <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>
>      Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.
>      v1 over-reached by also fixing "that that" in two other files where
>      the phrasing was grammatically intentional; v2 correctly scoped the
>      change after re-reading the context.
>      https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/
>      Status: Merged to master.
>
> Queued for master / next:
>
>   4. [PATCH v2] submodule--helper: replace malloc with xmalloc
>      <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>
>      Replaced a raw malloc() in submodule_summary_callback() with
>      xmalloc(), which calls die() on allocation failure rather than
>      returning NULL and risking a NULL dereference. Also used sizeof(*temp)
>      instead of the explicit struct name, and dropped the redundant C cast
>      on the xmalloc() return. Improved the commit message in v2 to explain
>      the reasoning behind removing the cast, as requested by Junio.
>      https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/
>      Status: Will merge to next.
>
>   5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID
>      <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>
>      Replaced a hardcoded 40-zero string in t/t3200-branch.sh with
>      $ZERO_OID to make the test hash-algorithm independent. The 40-char
>      string caused premature failure under SHA-256 because Git reports
>      "malformed object name" (40 vs 64 chars) before reaching the actual
>      object-type check. Using $ZERO_OID with test_grep for the "must
>      point to a commit" error ensures the test validates the right failure
>      mode. Suggested-by Patrick Steinhardt.
>      https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/
>      Status: Will merge to master.
>
>   6. [PATCH v3] t0410: modernize delete_object helper
>      <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>
>      Modernized the delete_object helper in t/t0410-partial-clone.sh,
>      the primary test file for partial clones and directly relevant to
>      this project, by replacing a fragile manual sed-based object path
>      calculation with test_oid_to_path, making the helper correct for
>      both SHA-1 and SHA-256. Added 'local' variable declarations with
>      proper quoting for shell portability. Suggested-by Jeff King, who
>      also caught that unquoted 'local' assignments are flagged by
>      check-non-portable-shell. This patch prompted Junio to add a note
>      to Documentation/SubmittingPatches reminding contributors to run
>      'make test' from the top level.
>      Helped-by: Pushkar Singh.
>      https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/
>      Status: Good to go to next.
>
> Collaboration:
>
>   7. Helped with [PATCH v5] help: cleanup the construction of keys_uniq
>      <20260311192453.62213-1-amishhhaaaa@gmail.com>
>      (patch by Amisha Chhajed). Suggested moving strbuf initialization
>      and release out of the inner loop to reuse the same buffer across
>      iterations, avoiding repeated alloc/free overhead. Listed as
>      Suggested-by in the patch.
>      https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/
>      Status: Will merge to next.
>
> Patch #6 is worth highlighting separately: working on
> t0410-partial-clone.sh required reading the entire file carefully,
> understanding how promisor remote tests are structured, and reasoning
> about hash-algorithm independence in the partial clone context. That
> work gives me a direct head start on the test infrastructure for this
> project.
>
> ---
>
> 10. RELEVANT EXPERIENCE
>
> C and shell (bash/POSIX sh) are my primary languages for this project.
> My submodule--helper patch (patch #4 above) demonstrates comfort with C
> memory management patterns in Git's codebase (xmalloc/die() convention,
> sizeof(*ptr) idiom, implicit void* promotion). My test patches
> demonstrate hands-on familiarity with Git's shell test framework:
> test_grep, test_oid_to_path, test_must_fail, test_when_finished, and the
> portability rules enforced by check-non-portable-shell. I am also
> familiar with Go, which gives me a general systems programming background.
>
> I have submitted patches through multiple review cycles on the mailing
> list, incorporated feedback from senior maintainers, and am comfortable
> with the in-reply-to threading conventions and interdiff-based review
> process.
>
> You can find all my patches at:
> https://lore.kernel.org/git/?q=Siddharth+Shrimali
>
> ---
>
> 11. AVAILABILITY
>
> I can commit 25-30 hours per week to GSoC during the coding period
> (May-August). My 4th semester exams conclude by mid-May, so I will be
> partly available at the start and fully available from mid-May onward
> with no academic conflicts through the end of summer. I have no
> internships, part-time work, or other commitments planned for this
> period.
>
> I will send a brief weekly status update to my mentors and post
> questions or design discussions to the mailing list as they come up, to
> stay unblocked and keep the community informed of progress.
>
> ---
>
> 12. WHY THIS PROJECT
>
> I have been using Git for a while, but it was only when I started
> contributing patches that I actually began reading its internals. Partial
> clone was one of the first areas I looked at carefully, partly because of
> the t0410 patch and partly out of genuine curiosity. As I read
> promisor-remote.c and backfill.c, the missing reverse direction became
> clear. Backfill exists, but its counterpart does not. That felt like an
> unfinished thought.
>
> I find the write-before-delete ordering problem genuinely interesting. It
> is one of those situations where the obvious implementation is wrong, and
> getting it right requires careful reasoning about crash recovery. The same
> is true for packed object removal, where you cannot simply delete one
> object and call it done. These are exactly the kinds of problems I enjoy
> thinking through.
>
> I also want to be candid that I do not yet know every corner of this
> codebase. There will be parts of the implementation where I will need
> guidance. That is exactly why I want to use the bonding period to read,
> experiment, and deepen my understanding before writing any implementation
> code, and why the design discussion on the list matters to me. I would
> much rather understand the problem thoroughly than rush into code that
> later has to be thrown away.
>
> ---
>
> Thank you for your time and for considering my proposal. I have genuinely
> enjoyed contributing to Git over the past few weeks and I am looking
> forward to doing more of it, with or without GSoC.
> If you have any questions or if there is anything missing from this
> proposal, I am happy to follow up on the list.
>
> ---
>
The rest of this proposal looks good to me!
> Siddharth Shrimali
> r.siddharth.shrimali@gmail.com

Regards, Karthik

Siddharth ShrimaliMar 17, 2026, 13:28 UTC in reply to Karthik Nayak on lore

Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones

Hi Karthik, thank you for the review!
Karthik Nayak <karthik.188@gmail.com> wrote:
Let me address each point.
Show 11 quoted lines
> > 5.1 Blob Enumeration
> >
> > Walk the object graph from all local refs using traverse_commit_list()
> > in list-objects.c, with a size-based filter consistent with the existing
> > blob:limit=N syntax from list-objects-filter.c. The goal is to collect
> > blob OIDs that exceed the threshold and are currently locally present,
> > meaning they resolve via find_pack_entry() or as a loose object, and are
> > not already in a promised/missing state.
>
> How does this work when used on a partial clone with 'blob:none'?
>

Right, an object walk using traverse_commit_list() would hit missing trees and blobs and end up triggering a lazy-fetch storm, which is the opposite of what we want. The enumeration needs to stay strictly local. We can do this by passing OBJECT_INFO_SKIP_FETCH_OBJECT to oid_object_info_extended(), similar to how remove_fetched_oids() works. That way we only look at blobs that are physically on disk and never touch the promisor machinery.

Show 8 quoted lines
> > For remotes that do not advertise object-info, a conservative fallback
> > applies: if a blob is reachable from a commit the remote has, and the
> > remote's advertised filter covers that object, its presence on the
> > remote is implied.
>
> How do we know a blob is reachable from a commit that the remote
> advertises?
>

Fair point, I think the cleaner approach is to just drop the local fallback entirely and make Protocol v2 object-info a hard requirement for v1. It is stricter but much safer and we only drop what we can explicitly confirm the remote has.

Show 10 quoted lines
> > 5.5 Safety Guards
> >
> >   - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,
> >     REVERT_HEAD, or rebase-merge/ is present under .git/. Removing
> >     objects mid-operation could corrupt an otherwise-recoverable state.
> >
>
> But couldn't one of these operations start after we start the process of
> cleanup?
>

Yes, you are right. A merge or rebase could still start halfway through the cleanup and lead to a race condition. So for that, write-before-delete already handles this case. If a merge starts mid-cleanup and needs a blob we just deleted, it will not hit a fatal "object missing" error, the OID is already in the .promisor file, so the lazy-fetch mechanism is called and fetches it back transparently. The repository stays in a consistent state regardless of when the concurrent operation starts. For packfile removal, we can rely on the existing .keep and lock mechanisms natively present in the repack machinery to prevent GC collisions.

Show 17 quoted lines
> >   - A new option on git backfill (e.g., --drop-blobs): backfill fills
> >     objects in; this flag drains them back out. The symmetry is clean
> >     and the command is already familiar to users working with partial
> >     clones.
> >
>
> I'm not sure if 'git backfill --drop-blobs' makes sense, since the
> command talks about filling in data and the flag talks about cleaning up
> data. But this seems to be the closest relative.
>
> >   - An opt-in git maintenance task for scheduled periodic space
> >     reclamation, wrapping the same core logic.
> >
>
> This would be my top pick, seems like data cleanup is definitely a
> maintenance task.
>

Agreed on both, a flag that does the opposite of what the command name says is confusing, and git maintenance is a much more natural home for something like this. I will focus the design discussion around making it a maintenance task.

Thanks again for the thorough review.

Siddharth Shrimali r.siddharth.shrimali@gmail.com

Siddharth ShrimaliMar 25, 2026, 11:46 UTC in reply to Siddharth Shrimali on lore

Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones

Hello Git Community,

I am Siddharth Shrimali, a contributor currently focused on the partial clone subsystem and test modernization (most recently the t0410 series). I would like to submit my updated proposal for GSoC 2026: 'Improve Disk Space Recovery for Partial Clones.'

This is a revised version incorporating feedback from Karthik Nayak. I would appreciate any further feedback or suggestions on the technical approach outlined below.

---
1. PERSONAL INFORMATION
Name:       Siddharth Shrimali
Email:      r.siddharth.shrimali@gmail.com
GitHub:     https://github.com/siddharthshrimali
LinkedIn:   https://www.linkedin.com/in/siddharthshrimali/
University: Walchand College of Engineering, India
Degree:     B.Tech, Computer Science and Engineering
            (2nd year, 4th semester)
Timezone:   IST (UTC+5:30)
---
2. PROJECT ABSTRACT

Git's partial clone feature lets clients work with large repositories without downloading all objects upfront. Objects are fetched lazily from promisor remotes as needed, and git backfill can proactively fill them in. But the reverse does not exist: once blobs have been fetched, there is no way to drop them back to a promised-but-absent state. Over time, clients accumulate large locally-held blobs they no longer need, with no path to reclaim that space short of a full re-clone.

This project implements that missing direction: a mechanism to safely identify blobs that are available on a promisor remote, remove them from local storage, and re-enter them into the promisor contract so that the existing lazy-fetch infrastructure re-fetches them transparently on next access. The result is a reversible partial clone lifecycle.

Project size: 175 hours. Difficulty: Medium to Hard.
Languages: C, Shell (bash/POSIX sh).
Mentors: Christian Couder, Karthik Nayak, Justin Tobler,
         Siddharth Asthana, Ayush Chandekar.
---
3. PROBLEM STATEMENT

When a user clones with --filter=blob:none, Git records the remote as a promisor remote and fetches blobs lazily on demand via promisor_remote_get_direct(). Over months of work, through lazy fetches, explicit backfills, and day-to-day operations, the local repository accumulates blobs that may no longer be needed. There is currently no mechanism to drop these blobs back to a promised-but-absent state. The only option is a fresh re-clone, which is disruptive and expensive.

The gap is the reverse of git backfill: backfill moves objects from promisor-promised to locally-present and this project moves them back.

---
4. GETTING STARTED PROGRESS

The project ideas page listed specific steps to get started. I have completed all of them:

  - Built Git from source and ran the full test suite.
  - Set up a partial clone experiment: created a local promisor remote
    via file://, cloned with --filter=blob:none, triggered lazy fetches,
    and inspected the resulting .promisor sidecar files to understand
    how the promisor contract works at the file level.
  - Studied builtin/backfill.c end to end, along with
    promisor-remote.c, list-objects-filter.c, and
    Documentation/technical/partial-clone.txt.
  - Submitted a micro-patch series (see Section 6) that was accepted
    and merged to master.
  - Read the partial clone and Protocol v2 documentation thoroughly.
---
5. TECHNICAL APPROACH
The full flow, at a high level:
  1. Enumerate locally-held blobs above a size threshold.
  2. Verify each candidate is available on the promisor remote.
  3. Write a .promisor sidecar registering dropped OIDs as promised.
     (This must happen before removal. See Section 5.3.)
  4. Remove confirmed blobs from local storage.
  5. Enforce safety guards.
---
5.1 Blob Enumeration

A naive object walk using traverse_commit_list() on a partial clone would hit missing trees and blobs and trigger a lazy-fetch storm, which is the opposite of what we want. The enumeration must stay strictly local.

We achieve this by passing OBJECT_INFO_SKIP_FETCH_OBJECT to oid_object_info_extended(), the same approach used by remove_fetched_oids() in promisor-remote.c. This ensures we only evaluate blobs that are physically present on disk and never touch the promisor machinery during enumeration.

The goal is to collect blob OIDs that exceed a configurable size threshold and are currently locally present, meaning they are not already in a promised/missing state.

---
5.2 Remote Availability Verification

Before removing anything, we confirm the promisor remote holds each candidate blob. We use the Protocol v2 "object-info" capability, which lets us query the remote for object metadata without downloading the object bodies, in a single batched round trip.

For v1, Protocol v2 object-info will be a hard requirement. The conservative local-graph fallback considered in earlier drafts is brittle in a partial clone context -- walking trees we do not have locally would either fail or trigger fetches -- and the safety guarantee it provides is weaker. Making object-info mandatory ensures we only drop what we can explicitly confirm the remote has.

---
5.3 Re-entering the Promisor Contract (write before delete)

After removal, any access to a dropped blob must trigger a lazy fetch via promisor_remote_get_direct() rather than a hard "object missing" error. To achieve this, we write a new (possibly empty) packfile with a .promisor sidecar that lists the dropped OIDs.

The write-before-delete ordering is the most critical part of the implementation. The .promisor sidecar must be written and fsynced before any object is physically removed. If a crash or interrupt happens in a window where an object is neither locally present nor promised, the repository ends up in an unrecoverable state.

This ordering also handles the race condition where a concurrent operation (such as a merge or rebase) starts mid-cleanup and suddenly needs a blob we just deleted. Because the OID is already in the .promisor file before deletion, the concurrent operation will not hit a fatal "object missing" error -- it will simply trigger the lazy-fetch machinery and re-fetch the blob transparently. The repository stays in a consistent state regardless of when the concurrent operation starts.

For packfile removal, the existing .keep and lock mechanisms in the repack machinery provide protection against GC collisions.

---
5.4 Object Removal
Two cases must be handled:
  a) Loose objects: compute the path via oid_to_path() (correct for
     both SHA-1 and SHA-256) and unlink it.
  b) Packed objects: a single object cannot be surgically removed from
     a packfile. Instead, we drive the repack machinery in
     builtin/repack.c to produce a new pack that excludes the target
     OIDs. The --filter mechanism on git pack-objects already supports
     object exclusion and can be driven with an explicit OID list.
---
5.5 Safety Guards
  - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,
    REVERT_HEAD, or rebase-merge/ is present under .git/. Removing
    objects mid-operation could corrupt an otherwise-recoverable state.
    Note that the write-before-delete ordering (Section 5.3) handles
    the case where such an operation starts after cleanup has begun.
  - Index safety: by default, refuse to drop blobs referenced by the
    current index (read via read_cache()). An explicit flag can override
    this for users who understand the implications.
  - Remote reachability: refuse if the promisor remote is not reachable
    at invocation time. Objects should not be dropped if they cannot be
    immediately re-fetched when needed.
  - Dry-run (--dry-run): enumerate and verify candidates, print what
    would be removed and the estimated space reclaimed, without touching
    any objects.
---
5.6 Command Placement

The project description explicitly leaves placement as an open design question. I will send a design email to the list during the bonding period and defer to community feedback before writing any code.

Based on the discussion with Karthik, git maintenance is the primary candidate. Space reclamation is fundamentally a housekeeping operation that fits naturally alongside existing maintenance tasks such as loose-objects and incremental-repack. It can be registered as an opt-in task (e.g., 'git maintenance run --task=drop-promisor-blobs') and scheduled for periodic execution. The design discussion will finalize the exact interface.

---
6. MICRO-PROJECT

As the required GSoC qualification task, I submitted a two-patch series to modernize t/t3700-add.sh:

  [PATCH v3 1/2] t3700: avoid suppressing git's exit code
  [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics
  Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
  Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/

Patch 1 breaks pipelines of the form "git foo | grep bar" that silently swallow git's exit code, by redirecting git output to a temp file and grepping that. Patch 2 replaces all plain grep / ! grep calls with the test_grep helper for better failure diagnostics. The series was iterated to v3 incorporating structural feedback from Junio C Hamano.

Status: Merged to master.

I have also contributed several other patches to Git since March 2026, including a C code fix in builtin/submodule--helper.c, test modernizations in t3200 and t9123, work on builtin/backfill.c, and notably in t/t0410-partial-clone.sh which is directly relevant to this project. Full details are in Section 9.

---
7. DELIVERABLES

The deliverables section lists the concrete outputs I commit to producing by the end of the project. This is what both the mentors and I can use to track progress and evaluate success.

  1. A design email to git@vger.kernel.org during the bonding period
     covering interface, command placement, safety model, and
     write-before-delete ordering, with community sign-off before any
     implementation begins.
  2. Core logic: blob enumeration using OBJECT_INFO_SKIP_FETCH_OBJECT,
     remote availability verification via Protocol v2 object-info,
     .promisor sidecar write, and safe removal of loose and packed
     objects.
  3. Command integration: wired into the agreed-upon command surface
     (git maintenance task as primary candidate).
  4. A comprehensive test suite covering: basic drop and re-fetch,
     dry-run, size filtering, safety guard rejections, SHA-256
     compatibility, loose and packed object cases, and
     write-before-delete crash safety.
  5. Documentation: man page updates for the modified command(s) and
     updates to Documentation/technical/partial-clone.txt describing the
     new object lifecycle (fetch -> hold -> drop -> re-fetch).
---
8. TIMELINE (175 hours)
In Progress (Now - May 1)
  - Continue submitting patches to the list while selections are pending.
  - Read builtin/repack.c and the Protocol v2 object-info code so the
    bonding period can focus on design rather than catching up on reading.
  - Experiment with .promisor sidecar files hands-on and draft the design
    email early so it is ready to send on day one of bonding.
Community Bonding Period (May 1 - May 26)
  - Go deeper into promisor-remote.c, builtin/backfill.c,
    builtin/repack.c, and list-objects-filter.c with the specific goal
    of understanding the exact call chains relevant to this project.
  - Trace the full lazy-fetch path in a debugger end to end.
  - Send the design proposal to the list and incorporate feedback before
    writing any implementation code.
Week 1-2 (May 27 - June 9) | Blob enumeration
  - Implement blob enumeration using oid_object_info_extended() with
    OBJECT_INFO_SKIP_FETCH_OBJECT, collecting locally-present blob OIDs
    above the size threshold without triggering lazy fetches.
  - Tests: correct OIDs collected, size filter respected, already-missing
    blobs excluded, no lazy fetches triggered during enumeration.
Week 3-4 (June 10 - June 23) | Remote verification
  - Implement remote availability check via Protocol v2 object-info,
    batched for efficiency.
  - Tests: mock promisor remote, batch query correctness, rejection of
    candidates the remote does not have.
Week 5-6 (June 24 - July 7) | Removal and re-promisor
  - Implement .promisor sidecar write for candidate OIDs (before removal).
  - Implement loose object removal (oid_to_path + unlink).
  - Implement packed object removal by driving repack with an OID
    exclusion list.
  - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch
    triggers correctly after drop, write-before-delete ordering safety.
  - Send an interim patch series to the list for early review.
Week 7-8 (July 8 - July 21) | Safety checks and command wiring
  - Implement all safety guards: in-progress operation detection, index
    blob check, remote reachability check.
  - Implement --dry-run.
  - Wire core logic into the agreed command surface with flags:
    --filter, --dry-run, --verbose.
  - Midterm evaluation: working prototype, tests passing, docs draft.
Week 9-10 (July 22 - August 4) | Edge cases and review
  - Handle edge cases: multiple worktrees, alternates, submodules with
    their own promisor remotes, SHA-256 repositories.
  - Address mailing list review feedback from the interim series.
  - Performance testing on a repository with many locally-held blobs.
Week 11-12 (August 5 - August 18) | Documentation and cleanup
  - Write and finalize man page documentation.
  - Update Documentation/technical/partial-clone.txt.
  - Final patch series submission incorporating all review feedback.
Buffer (August 19 - August 25)
  - Address any remaining review comments.
  - Ensure all tests pass on CI for both SHA-1 and SHA-256.
  - Final evaluation submission.
---
9. MY CONTRIBUTIONS TO GIT

I have been contributing to Git since early March 2026. All patches were submitted to git@vger.kernel.org and iterated through review with Junio C Hamano, Jeff King, Patrick Steinhardt, Derrick Stolee, and Phillip Wood.

Merged to master:
  1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)
     Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/
     (See Section 6 for full description. This is my micro-project.)
     Status: Merged to master.
  2. [PATCH] t9123: use test_when_finished for cleanup
     Message-ID: <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/
     Moved bare setup code in t9123 that ran outside any test block into
     a proper test_expect_success block. Used test_when_finished to
     register cleanup of the 'import' directory, ensuring it runs even
     if the test fails, replacing a manual rm -rf that could be skipped
     on failure.
     Status: Merged to master.
  3. [PATCH v2] t: fix "that that" typo in lib-unicode-nfc-nfd.sh
     Message-ID: <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/
     Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.
     v1 over-reached by also fixing "that that" in two other files where
     the phrasing was grammatically intentional; v2 correctly scoped the
     change after re-reading the context.
     Status: Merged to master.
  4. [PATCH v2] submodule--helper: replace malloc with xmalloc
     Message-ID: <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/
     Replaced a raw malloc() in submodule_summary_callback() with
     xmalloc(), which calls die() on allocation failure rather than
     returning NULL and risking a NULL dereference. Also used sizeof(*temp)
     instead of the explicit struct name, and dropped the redundant C cast
     on the xmalloc() return. Improved the commit message in v2 to explain
     the reasoning behind removing the cast, as requested by Junio.
     Status: Merged to master.
  5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID
     Message-ID: <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/
     Replaced a hardcoded 40-zero string in t/t3200-branch.sh with
     $ZERO_OID to make the test hash-algorithm independent. The 40-char
     string caused premature failure under SHA-256 because Git reports
     "malformed object name" (40 vs 64 chars) before reaching the actual
     object-type check. Using $ZERO_OID with test_grep for the "must
     point to a commit" error ensures the test validates the right failure
     mode. Suggested-by Patrick Steinhardt.
     Status: Merged to master.
  6. [PATCH v3] t0410: modernize delete_object helper
     Message-ID: <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/
     Modernized the delete_object helper in t/t0410-partial-clone.sh,
     the primary test file for partial clones and directly relevant to
     this project, by replacing a fragile manual sed-based object path
     calculation with test_oid_to_path, making the helper correct for
     both SHA-1 and SHA-256. Added 'local' variable declarations with
     proper quoting for shell portability. Suggested-by Jeff King, who
     also caught that unquoted 'local' assignments are flagged by
     check-non-portable-shell. This patch prompted Junio to add a note
     to Documentation/SubmittingPatches reminding contributors to run
     'make test' from the top level.
     Helped-by: Pushkar Singh.
     Status: Merged to master.
  7. [PATCH v3] backfill: handle unexpected arguments
     Message-ID: <20260321031643.5185-1-r.siddharth.shrimali@gmail.com>
     Link: https://lore.kernel.org/git/20260321031643.5185-1-r.siddharth.shrimali@gmail.com/
     Added a check in cmd_backfill() to report an error and print the
     short usage string if any unexpected non-option arguments are passed.
     Iterated to v3 incorporating feedback from Junio C Hamano (error
     message style, usage() vs usage_with_options()) and Phillip Wood
     (test_grep usage in tests). The patch was dropped after Derrick
     Stolee pointed out an in-flight series adding revision arguments to
     git backfill, making a hard rejection of positional arguments wrong.
     However, the report directly motivated Derrick to add a test for
     this behavior in his series, where I am credited as Reported-by.
     Status: Dropped, but credited as Reported-by in Derrick Stolee's
     series at:
     https://lore.kernel.org/git/9699650aa7dc04cf1cdc26803caa8304b29c1662.1774266019.git.gitgitgadget@gmail.com/
Collaboration:
  8. Helped with [PATCH v5] help: cleanup the construction of keys_uniq
     Message-ID: <20260311192453.62213-1-amishhhaaaa@gmail.com>
     Link: https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/
     (patch by Amisha Chhajed). Suggested moving strbuf initialization
     and release out of the inner loop to reuse the same buffer across
     iterations, avoiding repeated alloc/free overhead. Listed as
     Suggested-by in the patch.
     Status: Merged to master.

Patch #6 is worth highlighting separately: working on t0410-partial-clone.sh required reading the entire file carefully, understanding how promisor remote tests are structured, and reasoning about hash-algorithm independence in the partial clone context. That work gives me a direct head start on the test infrastructure for this project.

---
10. RELEVANT EXPERIENCE

C and shell (bash/POSIX sh) are my primary languages for this project. My submodule--helper patch (patch #4 above) demonstrates comfort with C memory management patterns in Git's codebase (xmalloc/die() convention, sizeof(*ptr) idiom, implicit void* promotion). My test patches demonstrate hands-on familiarity with Git's shell test framework: test_grep, test_oid_to_path, test_must_fail, test_when_finished, and the portability rules enforced by check-non-portable-shell. I am also familiar with Go, which gives me a general systems programming background.

I have submitted patches through multiple review cycles on the mailing list, incorporated feedback from senior maintainers, and am comfortable with the in-reply-to threading conventions and interdiff-based review process.

You can find all my patches at: https://lore.kernel.org/git/?q=Siddharth+Shrimali

---
11. AVAILABILITY

I can commit 25-30 hours per week to GSoC during the coding period (May-August). My 4th semester exams conclude by mid-May, so I will be partly available at the start and fully available from mid-May onward with no academic conflicts through the end of summer. I have no internships, part-time work, or other commitments planned for this period.

I will send a brief weekly status update to my mentors and post questions or design discussions to the mailing list as they come up, to stay unblocked and keep the community informed of progress.

---
12. WHY THIS PROJECT

I have been using Git for a while, but it was only when I started contributing patches that I actually began reading its internals. Partial clone was one of the first areas I looked at carefully, partly because of the t0410 patch and partly out of genuine curiosity. As I read promisor-remote.c and backfill.c, the missing reverse direction became clear. Backfill exists, but its counterpart does not. That felt like an unfinished thought.

I find the write-before-delete ordering problem genuinely interesting. It is one of those situations where the obvious implementation is wrong, and getting it right requires careful reasoning about crash recovery. The same is true for packed object removal, where you cannot simply delete one object and call it done. These are exactly the kinds of problems I enjoy thinking through.

I also want to be candid that I do not yet know every corner of this codebase. There will be parts of the implementation where I will need guidance. That is exactly why I want to use the bonding period to read, experiment, and deepen my understanding before writing any implementation code, and why the design discussion on the list matters to me. I would much rather understand the problem thoroughly than rush into code that later has to be thrown away.

---

Thank you for your time and for considering my proposal. I have genuinely enjoyed contributing to Git over the past few weeks and I am looking forward to doing more of it, with or without GSoC. If you have any questions or if there is anything missing from this proposal, I am happy to follow up on the list.

---

Siddharth Shrimali r.siddharth.shrimali@gmail.com

Back to recent threads