git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [GSoC Proposal] Improve Disk Space Recovery for Partial Clones

From
Karthik Nayak <karthik.188@gmail.com>
Date
Mar 16, 2026, 23:05 UTC
Message-ID
<CAOLa=ZSqQ=w8EVFMUwn5EUfMmNBgzGzG458Ex1ixiBSTiWnWYg@mail.gmail.com>
In-Reply-To
<CAGWgyh-NASOa+6NPhQHCte2_A7OkNm1r2qAov3Kn1+r1d25hYw@mail.gmail.com>
Siddharth Shrimali <r.siddharth.shrimali@gmail.com> writes:
Hello Siddharth,
Show 108 quoted lines
> Hello Git Community,
>
> I am Siddharth Shrimali, a contributor currently focused on the
> partial clone subsystem and test modernization (most recently the
> t0410 series). I would like to submit my proposal for GSoC 2026:
> 'Improve Disk Space Recovery for Partial Clones.'
>
> I've had a great experience collaborating with the community so far
> and would appreciate any feedback or suggestions on the technical
> approach outlined below.
>
> ---
>
> 1. PERSONAL INFORMATION
>
> Name:       Siddharth Shrimali
> Email:        r.siddharth.shrimali@gmail.com
> GitHub:      https://github.com/siddharthshrimali
> LinkedIn:    https://www.linkedin.com/in/siddharthshrimali/
> University:  Walchand College of Engineering, India
> Degree:      B.Tech, Computer Science and Engineering
>              (2nd year, 4th semester)
> Timezone:  IST (UTC+5:30)
>
> ---
>
> 2. PROJECT ABSTRACT
>
> Git's partial clone feature lets clients work with large repositories
> without downloading all objects upfront. Objects are fetched lazily from
> promisor remotes as needed, and git backfill can proactively fill them
> in. But the reverse does not exist: once blobs have been fetched, there
> is no way to drop them back to a promised-but-absent state. Over time,
> clients accumulate large locally-held blobs they no longer need, with no
> path to reclaim that space short of a full re-clone.
>
> This project implements that missing direction: a mechanism to safely
> identify blobs that are available on a promisor remote, remove them from
> local storage, and re-enter them into the promisor contract so that the
> existing lazy-fetch infrastructure re-fetches them transparently on next
> access. The result is a reversible partial clone lifecycle.
>
> Project size: 175 hours. Difficulty: Medium to Hard.
> Languages: C, shell (bash/POSIX sh).
> Mentors: Christian Couder, Karthik Nayak, Justin Tobler,
>          Siddharth Asthana, Ayush Chandekar.
>
> ---
>
> 3. PROBLEM STATEMENT
>
> When a user clones with --filter=blob:none, Git records the remote as a
> promisor remote and fetches blobs lazily on demand via
> promisor_remote_get_direct(). Over months of work, through lazy fetches,
> explicit backfills, and day-to-day operations, the local repository
> accumulates blobs that may no longer be needed. There is currently no
> mechanism to drop these blobs back to a promised-but-absent state. The
> only option is a fresh re-clone, which is disruptive and expensive.
>
> The gap is the reverse of git backfill: backfill moves objects from
> promisor-promised to locally-present and this project moves them back.
>
> ---
>
> 4. GETTING STARTED PROGRESS
>
> The project ideas page listed specific steps to get started. I have
> completed all of them:
>
>   - Built Git from source and ran the full test suite.
>
>   - Set up a partial clone experiment: created a local promisor remote
>     via file://, cloned with --filter=blob:none, triggered lazy fetches,
>     and inspected the resulting .promisor sidecar files to understand
>     how the promisor contract works at the file level.
>
>   - Studied builtin/backfill.c end to end, along with
>     promisor-remote.c, list-objects-filter.c, and
>     Documentation/technical/partial-clone.txt.
>
>   - Submitted a micro-patch series (see Section 6) that was accepted
>     and merged to master.
>
>   - Read the partial clone and Protocol v2 documentation thoroughly.
>
> ---
>
> 5. TECHNICAL APPROACH
>
> The full flow, at a high level:
>
>   1. Enumerate locally-held blobs above a size threshold.
>   2. Verify each candidate is available on the promisor remote.
>   3. Write a .promisor sidecar registering dropped OIDs as promised.
>      (This must happen before removal. See Section 5.3.)
>   4. Remove confirmed blobs from local storage.
>   5. Enforce safety guards.
>
> ---
>
> 5.1 Blob Enumeration
>
> Walk the object graph from all local refs using traverse_commit_list()
> in list-objects.c, with a size-based filter consistent with the existing
> blob:limit=N syntax from list-objects-filter.c. The goal is to collect
> blob OIDs that exceed the threshold and are currently locally present,
> meaning they resolve via find_pack_entry() or as a loose object, and are
> not already in a promised/missing state.
How does this work when used on a partial clone with 'blob:none'?
Show 9 quoted lines
> ---
>
> 5.2 Remote Availability Verification
>
> Before removing anything, we confirm the promisor remote holds each
> candidate blob. The cleanest way to do this is the Protocol v2
> "object-info" capability, which lets us query the remote for object
> metadata without downloading the object bodies, in a single batched
> round trip
Makes sense.
Show 5 quoted lines
>
> For remotes that do not advertise object-info, a conservative fallback
> applies: if a blob is reachable from a commit the remote has, and the
> remote's advertised filter covers that object, its presence on the
> remote is implied.

How do we know a blob is reachable from a commit that the remote advertises?

Show 16 quoted lines
>
> ---
>
> 5.3 Re-entering the Promisor Contract (write before delete)
>
> After removal, any access to a dropped blob must trigger a lazy fetch
> via promisor_remote_get_direct() rather than a hard "object missing"
> error. To achieve this, we write a new (possibly empty) packfile with a
> .promisor sidecar that lists the dropped OIDs.
>
> The write-before-delete ordering is the most critical part of the
> implementation. The .promisor sidecar must be written and fsynced before
> any object is physically removed. If a crash or interrupt happens in a
> window where an object is neither locally present nor promised, the
> repository ends up in an unrecoverable state.
>
Yeah, this makes sense too.
Show 23 quoted lines
> ---
>
> 5.4 Object Removal
>
> Two cases must be handled:
>
>   a) Loose objects: compute the path via oid_to_path() (correct for
>      both SHA-1 and SHA-256) and unlink it.
>
>   b) Packed objects: a single object cannot be surgically removed from
>      a packfile. Instead, we drive the repack machinery in
>      builtin/repack.c to produce a new pack that excludes the target
>      OIDs. The --filter mechanism on git pack-objects already supports
>      object exclusion and can be driven with an explicit OID list.
>
> ---
>
> 5.5 Safety Guards
>
>   - No in-progress operation: refuse if MERGE_HEAD, CHERRY_PICK_HEAD,
>     REVERT_HEAD, or rebase-merge/ is present under .git/. Removing
>     objects mid-operation could corrupt an otherwise-recoverable state.
>

But couldn't one of these operations start after we start the process of cleanup?

Show 12 quoted lines
>   - Index safety: by default, refuse to drop blobs referenced by the
>     current index (read via read_cache()). An explicit flag can override
>     this for users who understand the implications.
>
>   - Remote reachability: refuse if the promisor remote is not reachable
>     at invocation time. Objects should not be dropped if they cannot be
>     immediately re-fetched when needed.
>
>   - Dry-run (--dry-run): enumerate and verify candidates, print what
>     would be removed and the estimated space reclaimed, without touching
>     any objects.
>
Nice.
Show 14 quoted lines
> ---
>
> 5.6 Command Placement
>
> The project description explicitly leaves placement as an open design
> question. I will send a design email to the list during the bonding
> period and defer to community feedback before writing any code. My
> current thinking is:
>
>   - A new option on git backfill (e.g., --drop-blobs): backfill fills
>     objects in; this flag drains them back out. The symmetry is clean
>     and the command is already familiar to users working with partial
>     clones.
>

I'm not sure if 'git backfill --drop-blobs' makes sense, since the command talks about filling in data and the flag talks about cleaning up data. But this seems to be the closest relative.

>   - An opt-in git maintenance task for scheduled periodic space
>     reclamation, wrapping the same core logic.
>

This would be my top pick, seems like data cleanup is definitely a maintenance task.

Show 296 quoted lines
> ---
>
> 6. MICRO-PROJECT
>
> As the required GSoC qualification task, I submitted a two-patch series
> to modernize t/t3700-add.sh:
>
>   [PATCH v3 1/2] t3700: avoid suppressing git's exit code
>   [PATCH v3 2/2] t3700: use test_grep helper for better diagnostics
>
>   Message-ID: <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
>   Link: https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/
>
> Patch 1 breaks pipelines of the form "git foo | grep bar" that silently
> swallows git's exit code, by redirecting git output to a temp file and
> grepping that. Patch 2 replaces all plain grep / ! grep calls with the
> test_grep helper for better failure diagnostics. The series was iterated
> to v3 incorporating structural feedback from Junio C Hamano.
>
> Status: Merged to master.
>
> I have also contributed several other patches to Git since February
> 2026, including a C code fix in builtin/submodule--helper.c, test
> modernizations in t3200 and t9123, and notably in
> t/t0410-partial-clone.sh which is directly relevant to this project.
> Full details are in Section 9.
>
> ---
>
> 7. DELIVERABLES
>
> The deliverables section lists the concrete outputs I commit to
> producing by the end of the project. This is what both the mentors and
> I can use to track progress and evaluate success.
>
>   1. A design email to git@vger.kernel.org during the bonding period
>      covering interface, command placement, safety model, and
>      write-before-delete ordering, with community sign-off before any
>      implementation begins.
>
>   2. Core logic: blob enumeration, remote availability verification
>      (Protocol v2 object-info + fallback), .promisor sidecar write, and
>      safe removal of loose and packed objects.
>
>   3. Command integration: wired into the agreed-upon command surface
>      (git backfill option and/or git maintenance task).
>
>   4. A comprehensive test suite covering: basic drop and re-fetch,
>      dry-run, size filtering, safety guard rejections, SHA-256
>      compatibility, loose and packed object cases, and
>      write-before-delete crash safety.
>
>   5. Documentation: man page updates for the modified command(s) and
>      updates to Documentation/technical/partial-clone.txt describing the
>      new object lifecycle (fetch -> hold -> drop -> re-fetch).
>
> ---
>
> 8. TIMELINE (175 hours)
>
> In Progress (Now - May 1)
>
>   - Continue submitting patches to the list while selections are pending.
>   - Read builtin/repack.c and the Protocol v2 object-info code so the
>     bonding period can focus on design rather than catching up on reading.
>   - Experiment with .promisor sidecar files hands-on and draft the design
>     email early so it is ready to send on day one of bonding.
>
> Community Bonding Period (May 1 - May 26)
>
>   - Go deeper into promisor-remote.c, builtin/backfill.c,
>     builtin/repack.c, and list-objects-filter.c with the specific goal
>     of understanding the exact call chains relevant to this project.
>   - Trace the full lazy-fetch path in a debugger end to end.
>   - Send the design proposal to the list and incorporate feedback before
>     writing any implementation code.
>
> Week 1-2 (May 27 - June 9) | Blob enumeration
>
>   - Implement blob enumeration via traverse_commit_list() with a size
>     filter, collecting locally-present blob OIDs above the threshold.
>   - Tests: correct OIDs collected, size filter respected, already-missing
>     blobs excluded.
>
> Week 3-4 (June 10 - June 23) | Remote verification
>
>   - Implement remote availability check via Protocol v2 object-info,
>     batched for efficiency.
>   - Implement the conservative reachability-implies-presence fallback.
>   - Tests: mock promisor remote, batch query correctness, fallback path.
>
> Week 5-6 (June 24 - July 7) | Removal and re-promisor
>
>   - Implement .promisor sidecar write for candidate OIDs (before removal).
>   - Implement loose object removal (oid_to_path + unlink).
>   - Implement packed object removal by driving repack with an OID
>     exclusion list.
>   - Tests: loose drop + re-fetch, packed drop + re-fetch, lazy fetch
>     triggers correctly after drop, write-before-delete ordering safety.
>   - Send an interim patch series to the list for early review.
>
> Week 7-8 (July 8 - July 21) | Safety checks and command wiring
>
>   - Implement all safety guards: in-progress operation detection, index
>     blob check, remote reachability check.
>   - Implement --dry-run.
>   - Wire core logic into the agreed command surface with flags:
>     --filter, --dry-run, --verbose.
>   - Midterm evaluation: working prototype, tests passing, docs draft.
>
> Week 9-10 (July 22 - August 4) | Edge cases and review
>
>   - Handle edge cases: multiple worktrees, alternates, submodules with
>     their own promisor remotes, SHA-256 repositories.
>   - Address mailing list review feedback from the interim series.
>   - Performance testing on a repository with many locally-held blobs.
>
> Week 11-12 (August 5 - August 18) | Documentation and cleanup
>
>   - Write and finalize man page documentation.
>   - Update Documentation/technical/partial-clone.txt.
>   - Final patch series submission incorporating all review feedback.
>
> Buffer (August 19 - August 25)
>
>   - Address any remaining review comments.
>   - Ensure all tests pass on CI for both SHA-1 and SHA-256.
>   - Final evaluation submission.
>
> ---
>
> 9. MY CONTRIBUTIONS TO GIT
>
> I have been contributing to Git since early March 2026. All patches
> were submitted to git@vger.kernel.org and iterated through review with
> Junio C Hamano, Jeff King, and Patrick Steinhardt.
>
> Merged to master:
>
>   1. [PATCH v3] t3700: avoid suppressing git's exit code (2-patch series)
>      <20260303204029.52952-1-r.siddharth.shrimali@gmail.com>
>      (See Section 6 for full description. This is my micro-project.)
>      https://lore.kernel.org/git/20260303204029.52952-1-r.siddharth.shrimali@gmail.com/
>      Status: Merged to master.
>
>   2. [PATCH] t9123: use test_when_finished for cleanup
>      <20260305125408.16958-1-r.siddharth.shrimali@gmail.com>
>      Moved bare setup code in t9123 that ran outside any test block into
>      a proper test_expect_success block. Used test_when_finished to
>      register cleanup of the 'import' directory, ensuring it runs even
>      if the test fails, replacing a manual rm -rf that could be skipped
>      on failure.
>      https://lore.kernel.org/git/20260305125408.16958-1-r.siddharth.shrimali@gmail.com/
>      Status: Merged to master.
>
>   3. [PATCH v2] t: fix "that that" typo in lib-unicode-nfc-nfd.sh
>      <20260302192627.83631-1-r.siddharth.shrimali@gmail.com>
>      Fixed an unintentional repeated-word typo in t/lib-unicode-nfc-nfd.sh.
>      v1 over-reached by also fixing "that that" in two other files where
>      the phrasing was grammatically intentional; v2 correctly scoped the
>      change after re-reading the context.
>      https://lore.kernel.org/git/20260302192627.83631-1-r.siddharth.shrimali@gmail.com/
>      Status: Merged to master.
>
> Queued for master / next:
>
>   4. [PATCH v2] submodule--helper: replace malloc with xmalloc
>      <20260310164412.47403-1-r.siddharth.shrimali@gmail.com>
>      Replaced a raw malloc() in submodule_summary_callback() with
>      xmalloc(), which calls die() on allocation failure rather than
>      returning NULL and risking a NULL dereference. Also used sizeof(*temp)
>      instead of the explicit struct name, and dropped the redundant C cast
>      on the xmalloc() return. Improved the commit message in v2 to explain
>      the reasoning behind removing the cast, as requested by Junio.
>      https://lore.kernel.org/git/20260310164412.47403-1-r.siddharth.shrimali@gmail.com/
>      Status: Will merge to next.
>
>   5. [PATCH v2] t3200: replace hardcoded null OID with $ZERO_OID
>      <20260311174120.76871-1-r.siddharth.shrimali@gmail.com>
>      Replaced a hardcoded 40-zero string in t/t3200-branch.sh with
>      $ZERO_OID to make the test hash-algorithm independent. The 40-char
>      string caused premature failure under SHA-256 because Git reports
>      "malformed object name" (40 vs 64 chars) before reaching the actual
>      object-type check. Using $ZERO_OID with test_grep for the "must
>      point to a commit" error ensures the test validates the right failure
>      mode. Suggested-by Patrick Steinhardt.
>      https://lore.kernel.org/git/20260311174120.76871-1-r.siddharth.shrimali@gmail.com/
>      Status: Will merge to master.
>
>   6. [PATCH v3] t0410: modernize delete_object helper
>      <20260313053159.36492-1-r.siddharth.shrimali@gmail.com>
>      Modernized the delete_object helper in t/t0410-partial-clone.sh,
>      the primary test file for partial clones and directly relevant to
>      this project, by replacing a fragile manual sed-based object path
>      calculation with test_oid_to_path, making the helper correct for
>      both SHA-1 and SHA-256. Added 'local' variable declarations with
>      proper quoting for shell portability. Suggested-by Jeff King, who
>      also caught that unquoted 'local' assignments are flagged by
>      check-non-portable-shell. This patch prompted Junio to add a note
>      to Documentation/SubmittingPatches reminding contributors to run
>      'make test' from the top level.
>      Helped-by: Pushkar Singh.
>      https://lore.kernel.org/git/20260313053159.36492-1-r.siddharth.shrimali@gmail.com/
>      Status: Good to go to next.
>
> Collaboration:
>
>   7. Helped with [PATCH v5] help: cleanup the construction of keys_uniq
>      <20260311192453.62213-1-amishhhaaaa@gmail.com>
>      (patch by Amisha Chhajed). Suggested moving strbuf initialization
>      and release out of the inner loop to reuse the same buffer across
>      iterations, avoiding repeated alloc/free overhead. Listed as
>      Suggested-by in the patch.
>      https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/
>      Status: Will merge to next.
>
> Patch #6 is worth highlighting separately: working on
> t0410-partial-clone.sh required reading the entire file carefully,
> understanding how promisor remote tests are structured, and reasoning
> about hash-algorithm independence in the partial clone context. That
> work gives me a direct head start on the test infrastructure for this
> project.
>
> ---
>
> 10. RELEVANT EXPERIENCE
>
> C and shell (bash/POSIX sh) are my primary languages for this project.
> My submodule--helper patch (patch #4 above) demonstrates comfort with C
> memory management patterns in Git's codebase (xmalloc/die() convention,
> sizeof(*ptr) idiom, implicit void* promotion). My test patches
> demonstrate hands-on familiarity with Git's shell test framework:
> test_grep, test_oid_to_path, test_must_fail, test_when_finished, and the
> portability rules enforced by check-non-portable-shell. I am also
> familiar with Go, which gives me a general systems programming background.
>
> I have submitted patches through multiple review cycles on the mailing
> list, incorporated feedback from senior maintainers, and am comfortable
> with the in-reply-to threading conventions and interdiff-based review
> process.
>
> You can find all my patches at:
> https://lore.kernel.org/git/?q=Siddharth+Shrimali
>
> ---
>
> 11. AVAILABILITY
>
> I can commit 25-30 hours per week to GSoC during the coding period
> (May-August). My 4th semester exams conclude by mid-May, so I will be
> partly available at the start and fully available from mid-May onward
> with no academic conflicts through the end of summer. I have no
> internships, part-time work, or other commitments planned for this
> period.
>
> I will send a brief weekly status update to my mentors and post
> questions or design discussions to the mailing list as they come up, to
> stay unblocked and keep the community informed of progress.
>
> ---
>
> 12. WHY THIS PROJECT
>
> I have been using Git for a while, but it was only when I started
> contributing patches that I actually began reading its internals. Partial
> clone was one of the first areas I looked at carefully, partly because of
> the t0410 patch and partly out of genuine curiosity. As I read
> promisor-remote.c and backfill.c, the missing reverse direction became
> clear. Backfill exists, but its counterpart does not. That felt like an
> unfinished thought.
>
> I find the write-before-delete ordering problem genuinely interesting. It
> is one of those situations where the obvious implementation is wrong, and
> getting it right requires careful reasoning about crash recovery. The same
> is true for packed object removal, where you cannot simply delete one
> object and call it done. These are exactly the kinds of problems I enjoy
> thinking through.
>
> I also want to be candid that I do not yet know every corner of this
> codebase. There will be parts of the implementation where I will need
> guidance. That is exactly why I want to use the bonding period to read,
> experiment, and deepen my understanding before writing any implementation
> code, and why the design discussion on the list matters to me. I would
> much rather understand the problem thoroughly than rush into code that
> later has to be thrown away.
>
> ---
>
> Thank you for your time and for considering my proposal. I have genuinely
> enjoyed contributing to Git over the past few weeks and I am looking
> forward to doing more of it, with or without GSoC.
> If you have any questions or if there is anything missing from this
> proposal, I am happy to follow up on the list.
>
> ---
>
The rest of this proposal looks good to me!
> Siddharth Shrimali
> r.siddharth.shrimali@gmail.com

Regards, Karthik

Previous: Siddharth ShrimaliNext: Siddharth Shrimali
Message 2 of 4 in “[GSoC Proposal] Improve Disk Space Recovery for Partial Clones”
  1. Siddharth ShrimaliMar 14, 2026
  2. Karthik NayakMar 16, 2026
  3. Siddharth ShrimaliMar 17, 2026
  4. Siddharth ShrimaliMar 25, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.