From: Amisha Chhajed Date: Tue, 24 Mar 2026 00:09:26 GMT Subject: Re: [GSOC Proposal v2][RFC] Improve disk space recovery for partial clones Message-ID: In-Reply-To: PDF Version: https://docs.google.com/document/d/10H_FybR9Er7iDVwkisIRvjIZT_tEldwSu9jOZo7DRps/edit?usp=sharing On Tue, 24 Mar 2026 at 05:38, Amisha Chhajed wrote: > > Hello! I am Amisha, I am interested in the project Improve disk space > recovery for partial clones, I would appreciate it if I can get > suggestions and improvement comments for my proposal. Thank you so > much. > > Improve disk space recovery for partial clones > > Mentors: Christian Couder, Karthik Nayak, Justin Tobler, Siddharth > Asthana, Ayush Chandekar. > > Project Size: 350 hours > > Difficulty: Medium to Hard. > > Personal Information: > > —---------------------------------------------------------------------------------------------------------------------------- > > Name: Amisha Chhajed > > Email: amishhhaaaa@gmail.com > > github: https://github.com/amishhaa > > Time Zone: UTC +5:30 (IST) > > Education: SVKM's Dwarkadas J. Sanghvi College of Engineering > > Year: 3rd year, 6th semester > > Degree: Bachelor of Technology in Artificial Intelligence and Data Science > > About me and past experience: > > —---------------------------------------------------------------------------------------------------------------------------- > > Hello, I am Amisha, currently in my penultimate year of engineering.I > am deeply passionate about contributing to open source and find great > fulfillment when I see my code helping people. Given this motivation, > I have contributed to various open source projects and the experience > has been extremly rewarding and ethereal. > > Apart from open source, I make art and I like building games. > > > I am currently doing my LFX at OpenTelemetry, my project is about > building a GO CLI tool that runs tests of the dependents of a GO > library to record any regresisons caused by new changes in the library > (more about my project: > https://mentorship.lfx.linuxfoundation.org/project/5f537fc2-548b-487a-99ed-c61f7e8bcd47) > (list of PRs created until now: > https://github.com/open-telemetry/opentelemetry-go-build-tools/issues?q=is%3Apr+author%3Aamishhaa) > inspired by Rust's Crater(https://github.com/rust-lang/crater), this > is a project/tool that I really love as it is the first time i have > made something E2E with a huge impact. I interned at Google for summer > of 2025 under team workspace serving infrastructure. Some open source > contributions that i am most proud of are in bitcoin core(refer > credits: https://bitcoincore.org/en/releases/29.2/ my PR: > https://github.com/bitcoin/bitcoin/pull/33482) and git :) > > My contributions in git: > > * cat-file: exit code of 'git cat-file' is suppressed by piping it > directly into grep. > > Status: Awaiting review > > Mailing List: https://lore.kernel.org/git/20260113180409.36683-1-amishhhaaaa@gmail.com/ > > Log: This was the first patch i ever created for git as a > microproject, made me familiar with the mailing list workflow. > > > * sparse-checkout: optimize string_list construction and add tests to > verify deduplication. > > Status: merged in 'master' > > Mailing List: https://lore.kernel.org/git/20260121130005.72375-1-amishhhaaaa@gmail.com/ > > Log: This is a really important patch for me, improves O(n^2) > complexity to O(n log n) of sparse-checkout by building a sorted > 'string_list' by constructing it unsorted then sorting it followed by > removing duplicates, this triggered a series of patches and uncovered > various bugs when i worked on replacing calls of string_list_sort() > and string_list_remove_duplicates() with string_list_sort_u(). > > > *u-string-list: add unit tests for string-list methods. > > Status: merged in 'master' > > Mailing List: https://lore.kernel.org/git/20260129121220.69267-1-amishhhaaaa@gmail.com/ > > Log: Adding unit tests for string-list methods which i saw were not > present when i was creating a new API string_list_sort_u. > > > * string-list: add string_list_sort_u() that mimics "sort -u" > > Status: merged in 'master' > > Mailing List: https://lore.kernel.org/git/20260129121220.69267-2-amishhhaaaa@gmail.com/ > > Log: Adding a new API string_list_sort_u and cleaning up the callsites > that can directly adopt this new replacement, string_list_sort_u > mimics sort -u. > > > *sparse-checkout: use string_list_sort_u > > Status: merged in 'master' > > Mailing List: https://lore.kernel.org/git/20260212041017.91370-2-amishhhaaaa@gmail.com/ > > Log: Small fix to replace a callsite of string_list_sort and > string_list_remove_duplicates with string_list_sort_u. > > > *help: cleanup the contruction of keys_uniq > > Status: Will merge to 'next'. > > Mailing List: https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/ > > Log: Cleaning up complex callsites of string_list_sort and > string_list_remove_duplicates, this one involved finding a test case > that demonstarted a breakage, > https://lore.kernel.org/git/CAPvEtrenMBMFaMxcCR4VwoyMFU-_Z+bqq5nJaWv5eyn3HRutEA@mail.gmail.com/, > then replacing them with string_list_sort_u, another bug found as an > effort is https://lore.kernel.org/git/CAPvEtrfEZXHxcDf=z60ODfUA8cS81rhF1y7KEZApEBby7aCa1A@mail.gmail.com/, > this is still a pending bug which is good to work around in future, I > have provided a test to demonstrate an existing breakage. > > > History/Background, Overview and Project Theory > > —---------------------------------------------------------------------------------------------------------------------------- > > The "Partial Clone" feature is a performance optimization for Git that > allows Git to function without having a complete copy of the > repository, building partial clone was a community effort, with > contributions referenced here, > https://git-scm.com/docs/partial-clone#_related_links. Most of my > knowledge of this project came from researching and reading blogs like > https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/. > The talk https://www.youtube.com/watch?v=YdstUWcg5j4 by Derrick Stolee > are good sources to learn about how git currently handles objects. > > > There are 4 types of clones that we support, full clone, blobless > clone, treeless clone and shallow clone, each of them have their own > set of configurations that tells rest of git what objects are OK to be > missing from repository as they are promised, git stores objects in > .git/ folder under objects/, these objects can be of 4 types, tag, > tree, commits and blobs. When we partial clone, git writes the partial > clone type in config file inside .git/, for example, > > (base) amishachhajed@Amishas-MacBook-Pro .git % cat config > > [core] > > repositoryformatversion = 1 > > filemode = true > > bare = false > > logallrefupdates = true > > ignorecase = true > > precomposeunicode = true > > [remote "origin"] > > url = https://github.com/python/cpython.git > > fetch = +refs/heads/*:refs/remotes/origin/* > > promisor = true > > partialclonefilter = tree:0 > > [branch "main"] > > remote = origin > > merge = refs/heads/main > > > Each promised object has to be on some of the remote otherwise our > partial clone is broken and we do not account for broken remotes. > > However overtime a lot of objects might be fetched and currently there > is no functionality that, first checks if an object is present on a > promisor and then we can remove it safely to free up our disk space, > fundamentally users who make a partial clone want to save up space so > being able to remove the aquired objects on demand would be a great > addition. > > > This project aims to implement a command 'git evict' that can carry > out safe removal of promised objects from our disk space such that it > can be fetched later on from any of the remotes if needed, > dynamically. > > Proposed Plan: > > —---------------------------------------------------------------------------------------------------------------------------- > > Whenever we are evicting something we are unsure of its usefullness to > users, unless obvious, which git gc already handles. To overcome this, > we can make a new command 'git evict', essentially this command gives > users options and freedom to evict objects that they no longer > require, either it is out of cone or not on a checkout and a lot of > other situations, instead of predicting what users may or may not need > we can give them the choice of removal. In a case where users have a > cone and have set up a maintance task of evicting objects outside of > cone, in that case we can make it automatic. > > > As git currently treats remotes as source of thruth, we also trust > those remotes when evicting blobs because if all the remotes break the > principal of not strictly presenting the promised objects then the > partial clone is inherently broken, we can display a warning that > remote might not be able to fetch your file again if it gets > unavailable on the remote while evicting, however by making a partial > clone we have already placed our trust on the remotes. > > > Locally i attempted implementing 'git evict OID_TO_EVICT' which is > available here https://github.com/git/git/compare/master...amishhaa:git:par-c. > Steps to test it out: > > Partial clone a repository blobless, > > (base) amishachhajed@Amishas-MacBook-Pro % git clone > --filter=blob:none https://github.com/python/cpython.git > > > Get OID for a blob, here i am getting it for README, > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git rev-parse > HEAD:README.rst > > 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec > > > cat-file the blob to make sure it is present on disk > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t > 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec > > remote: Enumerating objects: 1, done. > > remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1 (from 1) > > Receiving objects: 100% (1/1), 3.50 KiB | 3.50 MiB/s, done. > > blob > > Run the git evict command with the object oid we recieved from the > previous step. > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git evict > 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec > > DEBUG: Evicted 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec from pack entry search > > Enumerating objects: 810655, done. > > Counting objects: 100% (810654/810654), done. > > Delta compression using up to 12 threads > > Compressing objects: 100% (199618/199618), done. > > Writing objects: 100% (810654/810654), done. > > Total 810654 (delta 608469), reused 810654 (delta 608469), pack-reused > 0 (from 0) > > > Attempt to cat-file again. > > (Without network it should fail at fetching) > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t > 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec > > fatal: unable to access 'https://github.com/python/cpython.git/': > Could not resolve host: github.com > > fatal: could not fetch 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec from > promisor remote > > > (With network it would re-fetch) > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t > 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec > > remote: Enumerating objects: 1, done. > > remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1 (from 1) > > Receiving objects: 100% (1/1), 3.50 KiB | 3.50 MiB/s, done. > > blob > > > However the current implementation I have locally has no validation on > an OID, to make sure it is a promisr object before attempting to evict > it, hence it fails when i try to evict a commit object, that is not > promised in a partial clone. > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git rev-parse HEAD^{commit} > > 306c556fdbe7034c9a6d87516534eecdb911ad11 > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git evict > 306c556fdbe7034c9a6d87516534eecdb911ad11 > > > (base) amishachhajed@Amishas-MacBook-Pro cpython % git evict > 306c556fdbe7034c9a6d87516534eecdb911ad11 > > Enumerating objects: 810831, done. > > Counting objects: 100% (810831/810831), done. > > Delta compression using up to 12 threads > > Compressing objects: 100% (200113/200113), done. > > Writing objects: 100% (810831/810831), done. > > Total 810831 (delta 608149), reused 810831 (delta 608149), pack-reused > 0 (from 0) > > fatal: bad object refs/heads/main > > > So, when user calls git evict, it internally finds an oidset with > objects to evict based on the flag the user has set, and then skips > those objects in want_object_in_pack() method, so we would be able to > evict everything in a single repack operation. > > > Currently on the user interface i am planning to implement these > flags, inspired heavily from reversing how we partial clone, for say > when partial cloning i set blob:limit= then i might want to > evict blobs greater than that size in future. > > * --outside-cone: Evict blobs that are not part of the current > sparse-checkout cone. > > https://lore.kernel.org/git/735eb76e-44a9-4f79-b769-23a3a07437ae@gmail.com/) > > * --large-only=: Evict blobs that are bigger than a certain size. > > * --tree-depth=: Evict blobs with depth > n > > * --outside-checkout: Evict blobs that are not part of the current checkout. > > > This project can essentially be split into two parts: > > * Given an object, we can use method is_promisor_object() to check if > it is promised, if yes we can repack our packfiles without that > object, we would skip those OIDs in want_object_in_pack(). > > These OIDs that we have to skip will be passed from evict to > want_object_in_pack() method. > > > * Now comes gathering the list of objects for oidset, for each command > written above we would need a different method to find and append the > object to evict, in oidset. For example, for objects > --large-only=, we need to find objects that are larger than the > size specified and mark it for eviction, similar methods to handle > such filters have been written in our codebase and can be used as > reference. > > > * Optionally also add git evict methods to git maintainence. > > Project Timeline: > > —---------------------------------------------------------------------------------------------------------------------------- > > Community Bonding (Until May 24th) > > * Discuss project ideas and design implementation details with > mentors, possible subcommands to keep and the overall architecture of > command, any optimizations we can make when repacking objects. > > * Study different types of partial clones and how refs work, and which > OIDs i can evict in different types of partial clones. > > * All stages would be accompained by necessary documentation and tests. > > > Coding Period > > (May 25 - June 30) > > * Implement a basic git evict command. > > * Implement evict_object method, this evict object method would need > to handle different cases when we created a partial clone, that is, if > we created a treeless clone or a blobless clone, for example if we > have a blobless clone we cannot evict a OID referencing a commit > object or a tag object because then refs would complain. It would need > to handle most of the cases that partial clone handles. > > * evict_object method would only be responsible to evict an OID initially. > > (July 1st - August 16) > > * Implement different ways to gather OIDs for the flag that we are referencing. > > * Pass on those OIDs to evict_objects method which would internally > call evict_object method, that handles all the different object cases. > > > Final Week (August 17 - August 24) > > *This is a buffer period for any unforseen delays and prepare final > report of everything we have accomplished over the summer :) > > Availibility: > > —---------------------------------------------------------------------------------------------------------------------------- > > I would be dedicating 45-50 hours per week of my time to this project > weekly. I dont have any other commitments apart from LFX in month of > May which would be easy to manage given my summer break, so during the > month of May my time commitment would be 25 hours to this project, > incase of extensions i have my internship break next semester so this > would be considered as that hence i do not have any university work > which would be interfering with the timeline even after my summer > break. > > Post-GSOC and Appreciation: > > —---------------------------------------------------------------------------------------------------------------------------- > > I want my journey with git to be a long one, it is very fulfilling for > me to see my code running on many devices, it is like a dream come > true for me, so even post GSOC I intend to keep contributing to git. > > > Contirbuting to git uptill now has been insanely rewarding, i learned > about commands that i did not even know existed, it feels magical to > know the internals of a tool i have been using since years and be a > part of the making as well. > > > I would like to thank the reviewers and maintainers, without them my > journey would'nt have been so easy, i learned some of my best software > development lessons on this very list which i am insanely grateful > for. Thank you. > > > > -- > Thanks, > Amisha -- Thanks, Amisha