{"thread":{"id":"65341","subject":"[GSOC Proposal v2][RFC] Improve disk space recovery for partial clones","startedAt":"2026-03-24T00:08:18Z","lastAt":"2026-03-24T00:09:39Z","messageCount":2,"participants":["Amisha Chhajed"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"539794","messageId":"CAPvEtreC_kRh7NW3JNfNkH+E-T1iK5XQ6aS81dtAVujGD=CzLg@mail.gmail.com","threadId":"65341","inReplyTo":null,"subject":"[GSOC Proposal v2][RFC] Improve disk space recovery for partial clones","fromName":"Amisha Chhajed","fromEmail":"amishhhaaaa@gmail.com","sentAt":"2026-03-24T00:08:06Z","receivedAt":"2026-03-24T00:08:18Z","isPatch":false,"sender":{"key":"amishhhaaaa@gmail.com","avatar":null},"body":"Hello! I am Amisha, I am interested in the project Improve disk space\nrecovery for partial clones, I would appreciate it if I can get\nsuggestions and improvement comments for my proposal. Thank you so\nmuch.\n\nImprove disk space recovery for partial clones\n\nMentors: Christian Couder, Karthik Nayak, Justin Tobler, Siddharth\nAsthana, Ayush Chandekar.\n\nProject Size: 350 hours\n\nDifficulty: Medium to Hard.\n\nPersonal Information:\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nName: Amisha Chhajed\n\nEmail: amishhhaaaa@gmail.com\n\ngithub: https://github.com/amishhaa\n\nTime Zone: UTC +5:30 (IST)\n\nEducation: SVKM's Dwarkadas J. Sanghvi College of Engineering\n\nYear: 3rd year, 6th semester\n\nDegree: Bachelor of Technology in Artificial Intelligence and Data Science\n\nAbout me and past experience:\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nHello, I am Amisha, currently in my penultimate year of engineering.I\nam deeply passionate about contributing to open source and find great\nfulfillment when I see my code helping people. Given this motivation,\nI have contributed to various open source projects and the experience\nhas been extremly rewarding and ethereal.\n\nApart from open source, I make art and I like building games.\n\n\nI am currently doing my LFX at OpenTelemetry, my project is about\nbuilding a GO CLI tool that runs tests of the dependents of a GO\nlibrary to record any regresisons caused by new changes in the library\n(more about my project:\nhttps://mentorship.lfx.linuxfoundation.org/project/5f537fc2-548b-487a-99ed-c61f7e8bcd47)\n(list of PRs created until now:\nhttps://github.com/open-telemetry/opentelemetry-go-build-tools/issues?q=is%3Apr+author%3Aamishhaa)\ninspired by Rust's Crater(https://github.com/rust-lang/crater), this\nis a project/tool that I really love as it is the first time i have\nmade something E2E with a huge impact. I interned at Google for summer\nof 2025 under team workspace serving infrastructure. Some open source\ncontributions that i am most proud of are in bitcoin core(refer\ncredits: https://bitcoincore.org/en/releases/29.2/ my PR:\nhttps://github.com/bitcoin/bitcoin/pull/33482) and git :)\n\nMy contributions in git:\n\n* cat-file: exit code of 'git cat-file' is suppressed by piping it\ndirectly into grep.\n\nStatus: Awaiting review\n\nMailing List: https://lore.kernel.org/git/20260113180409.36683-1-amishhhaaaa@gmail.com/\n\nLog: This was the first patch i ever created for git as a\nmicroproject, made me familiar with the mailing list workflow.\n\n\n* sparse-checkout: optimize string_list construction and add tests to\nverify deduplication.\n\nStatus: merged in 'master'\n\nMailing List: https://lore.kernel.org/git/20260121130005.72375-1-amishhhaaaa@gmail.com/\n\nLog: This is a really important patch for me, improves O(n^2)\ncomplexity to O(n log n) of sparse-checkout by building a sorted\n'string_list' by constructing it unsorted then sorting it followed by\nremoving duplicates, this triggered a series of patches and uncovered\nvarious bugs when i worked on replacing calls of string_list_sort()\nand string_list_remove_duplicates() with string_list_sort_u().\n\n\n*u-string-list: add unit tests for string-list methods.\n\nStatus: merged in 'master'\n\nMailing List: https://lore.kernel.org/git/20260129121220.69267-1-amishhhaaaa@gmail.com/\n\nLog: Adding unit tests for string-list methods which i saw were not\npresent when i was creating a new API string_list_sort_u.\n\n\n* string-list: add string_list_sort_u() that mimics \"sort -u\"\n\nStatus: merged in 'master'\n\nMailing List: https://lore.kernel.org/git/20260129121220.69267-2-amishhhaaaa@gmail.com/\n\nLog: Adding a new API string_list_sort_u and cleaning up the callsites\nthat can directly adopt this new replacement, string_list_sort_u\nmimics sort -u.\n\n\n*sparse-checkout: use string_list_sort_u\n\nStatus: merged in 'master'\n\nMailing List: https://lore.kernel.org/git/20260212041017.91370-2-amishhhaaaa@gmail.com/\n\nLog: Small fix to replace a callsite of string_list_sort and\nstring_list_remove_duplicates with string_list_sort_u.\n\n\n*help: cleanup the contruction of keys_uniq\n\nStatus: Will merge to 'next'.\n\nMailing List: https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/\n\nLog: Cleaning up complex callsites of string_list_sort and\nstring_list_remove_duplicates, this one involved finding a test case\nthat demonstarted a breakage,\nhttps://lore.kernel.org/git/CAPvEtrenMBMFaMxcCR4VwoyMFU-_Z+bqq5nJaWv5eyn3HRutEA@mail.gmail.com/,\nthen replacing them with string_list_sort_u, another bug found as an\neffort is https://lore.kernel.org/git/CAPvEtrfEZXHxcDf=z60ODfUA8cS81rhF1y7KEZApEBby7aCa1A@mail.gmail.com/,\nthis is still a pending bug which is good to work around in future, I\nhave provided a test to demonstrate an existing breakage.\n\n\nHistory/Background, Overview and Project Theory\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nThe \"Partial Clone\" feature is a performance optimization for Git that\nallows Git to function without having a complete copy of the\nrepository, building partial clone was a community effort, with\ncontributions referenced here,\nhttps://git-scm.com/docs/partial-clone#_related_links. Most of my\nknowledge of this project came from researching and reading blogs like\nhttps://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/.\nThe talk https://www.youtube.com/watch?v=YdstUWcg5j4 by Derrick Stolee\nare good sources to learn about how git currently handles objects.\n\n\nThere are 4 types of clones that we support, full clone, blobless\nclone, treeless clone and shallow clone, each of them have their own\nset of configurations that tells rest of git what objects are OK to be\nmissing from repository as they are promised, git stores objects in\n.git/ folder under objects/, these objects can be of 4 types, tag,\ntree, commits and blobs. When we partial clone, git writes the partial\nclone type in config file inside .git/, for example,\n\n(base) amishachhajed@Amishas-MacBook-Pro .git % cat config\n\n[core]\n\nrepositoryformatversion = 1\n\nfilemode = true\n\nbare = false\n\nlogallrefupdates = true\n\nignorecase = true\n\nprecomposeunicode = true\n\n[remote \"origin\"]\n\nurl = https://github.com/python/cpython.git\n\nfetch = +refs/heads/*:refs/remotes/origin/*\n\npromisor = true\n\npartialclonefilter = tree:0\n\n[branch \"main\"]\n\nremote = origin\n\nmerge = refs/heads/main\n\n\nEach promised object has to be on some of the remote otherwise our\npartial clone is broken and we do not account for broken remotes.\n\nHowever overtime a lot of objects might be fetched and currently there\nis no functionality that, first checks if an object is present on a\npromisor and then we can remove it safely to free up our disk space,\nfundamentally users who make a partial clone want to save up space so\nbeing able to remove the aquired objects on demand would be a great\naddition.\n\n\nThis project aims to implement a command 'git evict' that can carry\nout safe removal of promised objects from our disk space such that it\ncan be fetched later on from any of the remotes if needed,\ndynamically.\n\nProposed Plan:\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nWhenever we are evicting something we are unsure of its usefullness to\nusers, unless obvious, which git gc already handles. To overcome this,\nwe can make a new command 'git evict', essentially this command gives\nusers options and freedom to evict objects that they no longer\nrequire, either it is out of cone or not on a checkout and a lot of\nother situations, instead of predicting what users may or may not need\nwe can give them the choice of removal. In a case where users have a\ncone and have set up a maintance task of evicting objects outside of\ncone, in that case we can make it automatic.\n\n\nAs git currently treats remotes as source of thruth, we also trust\nthose remotes when evicting blobs because if all the remotes break the\nprincipal of not strictly presenting the promised objects then the\npartial clone is inherently broken, we can display a warning that\nremote might not be able to fetch your file again if it gets\nunavailable on the remote while evicting, however by making a partial\nclone we have already placed our trust on the remotes.\n\n\nLocally i attempted implementing 'git evict OID_TO_EVICT' which is\navailable here https://github.com/git/git/compare/master...amishhaa:git:par-c.\nSteps to test it out:\n\nPartial clone a repository blobless,\n\n(base) amishachhajed@Amishas-MacBook-Pro % git clone\n--filter=blob:none https://github.com/python/cpython.git\n\n\nGet OID for a blob, here i am getting it for README,\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git rev-parse\nHEAD:README.rst\n\n1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n\n\ncat-file the blob to make sure it is present on disk\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t\n1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n\nremote: Enumerating objects: 1, done.\n\nremote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1 (from 1)\n\nReceiving objects: 100% (1/1), 3.50 KiB | 3.50 MiB/s, done.\n\nblob\n\nRun the git evict command with the object oid we recieved from the\nprevious step.\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git evict\n1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n\nDEBUG: Evicted 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec from pack entry search\n\nEnumerating objects: 810655, done.\n\nCounting objects: 100% (810654/810654), done.\n\nDelta compression using up to 12 threads\n\nCompressing objects: 100% (199618/199618), done.\n\nWriting objects: 100% (810654/810654), done.\n\nTotal 810654 (delta 608469), reused 810654 (delta 608469), pack-reused\n0 (from 0)\n\n\nAttempt to cat-file again.\n\n(Without network it should fail at fetching)\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t\n1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n\nfatal: unable to access 'https://github.com/python/cpython.git/':\nCould not resolve host: github.com\n\nfatal: could not fetch 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec from\npromisor remote\n\n\n(With network it would re-fetch)\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t\n1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n\nremote: Enumerating objects: 1, done.\n\nremote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1 (from 1)\n\nReceiving objects: 100% (1/1), 3.50 KiB | 3.50 MiB/s, done.\n\nblob\n\n\nHowever the current implementation I have locally has no validation on\nan OID, to make sure it is a promisr object before attempting to evict\nit, hence it fails when i try to evict a commit object, that is not\npromised in a partial clone.\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git rev-parse HEAD^{commit}\n\n306c556fdbe7034c9a6d87516534eecdb911ad11\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git evict\n306c556fdbe7034c9a6d87516534eecdb911ad11\n\n\n(base) amishachhajed@Amishas-MacBook-Pro cpython % git evict\n306c556fdbe7034c9a6d87516534eecdb911ad11\n\nEnumerating objects: 810831, done.\n\nCounting objects: 100% (810831/810831), done.\n\nDelta compression using up to 12 threads\n\nCompressing objects: 100% (200113/200113), done.\n\nWriting objects: 100% (810831/810831), done.\n\nTotal 810831 (delta 608149), reused 810831 (delta 608149), pack-reused\n0 (from 0)\n\nfatal: bad object refs/heads/main\n\n\nSo, when user calls git evict, it internally finds an oidset with\nobjects to evict based on the flag the user has set, and then skips\nthose objects in want_object_in_pack() method, so we would be able to\nevict everything in a single repack operation.\n\n\nCurrently on the user interface i am planning to implement these\nflags, inspired heavily from reversing how we partial clone, for say\nwhen partial cloning i set blob:limit=<size> then i might want to\nevict blobs greater than that size in future.\n\n* --outside-cone: Evict blobs that are not part of the current\nsparse-checkout cone.\n\nhttps://lore.kernel.org/git/735eb76e-44a9-4f79-b769-23a3a07437ae@gmail.com/)\n\n* --large-only=<size>: Evict blobs that are bigger than a certain size.\n\n* --tree-depth=<n>: Evict blobs with depth > n\n\n* --outside-checkout: Evict blobs that are not part of the current checkout.\n\n\nThis project can essentially be split into two parts:\n\n* Given an object, we can use method is_promisor_object() to check if\nit is promised, if yes we can repack our packfiles without that\nobject, we would skip those OIDs in want_object_in_pack().\n\nThese OIDs that we have to skip will be passed from evict to\nwant_object_in_pack() method.\n\n\n* Now comes gathering the list of objects for oidset, for each command\nwritten above we would need a different method to find and append the\nobject to evict, in oidset. For example, for objects\n--large-only=<size>, we need to find objects that are larger than the\nsize specified and mark it for eviction, similar methods to handle\nsuch filters have been written in our codebase and can be used as\nreference.\n\n\n* Optionally also add git evict methods to git maintainence.\n\nProject Timeline:\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nCommunity Bonding (Until May 24th)\n\n* Discuss project ideas and design implementation details with\nmentors, possible subcommands to keep and the overall architecture of\ncommand, any optimizations we can make when repacking objects.\n\n* Study different types of partial clones and how refs work, and which\nOIDs i can evict in different types of partial clones.\n\n* All stages would be accompained by necessary documentation and tests.\n\n\nCoding Period\n\n(May 25 - June 30)\n\n* Implement a basic git evict command.\n\n* Implement evict_object method, this evict object method would need\nto handle different cases when we created a partial clone, that is, if\nwe created a treeless clone or a blobless clone, for example if we\nhave a blobless clone we cannot evict a OID referencing a commit\nobject or a tag object because then refs would complain. It would need\nto handle most of the cases that partial clone handles.\n\n* evict_object method would only be responsible to evict an OID initially.\n\n(July 1st - August 16)\n\n* Implement different ways to gather OIDs for the flag that we are referencing.\n\n* Pass on those OIDs to evict_objects method which would internally\ncall evict_object method, that handles all the different object cases.\n\n\nFinal Week (August 17 - August 24)\n\n*This is a buffer period for any unforseen delays and prepare final\nreport of everything we have accomplished over the summer :)\n\nAvailibility:\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nI would be dedicating 45-50 hours per week of my time to this project\nweekly. I dont have any other commitments apart from LFX in month of\nMay which would be easy to manage given my summer break, so during the\nmonth of May my time commitment would be 25 hours to this project,\nincase of extensions i have my internship break next semester so this\nwould be considered as that hence i do not have any university work\nwhich would be interfering with the timeline even after my summer\nbreak.\n\nPost-GSOC and Appreciation:\n\n—----------------------------------------------------------------------------------------------------------------------------\n\nI want my journey with git to be a long one, it is very fulfilling for\nme to see my code running on many devices, it is like a dream come\ntrue for me, so even post GSOC I intend to keep contributing to git.\n\n\nContirbuting to git uptill now has been insanely rewarding, i learned\nabout commands that i did not even know existed, it feels magical to\nknow the internals of a tool i have been using since years and be a\npart of the making as well.\n\n\nI would like to thank the reviewers and maintainers, without them my\njourney would'nt have been so easy, i learned some of my best software\ndevelopment lessons on this very list which i am insanely grateful\nfor. Thank you.\n\n\n\n-- \nThanks,\nAmisha\n"},{"id":"539795","messageId":"CAPvEtrdpJ5AHw+461ffkMmSXrtsi1vpQHKUQAW+9GZwoaifF4w@mail.gmail.com","threadId":"65341","inReplyTo":"CAPvEtreC_kRh7NW3JNfNkH+E-T1iK5XQ6aS81dtAVujGD=CzLg@mail.gmail.com","subject":"Re: [GSOC Proposal v2][RFC] Improve disk space recovery for partial clones","fromName":"Amisha Chhajed","fromEmail":"amishhhaaaa@gmail.com","sentAt":"2026-03-24T00:09:26Z","receivedAt":"2026-03-24T00:09:39Z","isPatch":false,"sender":{"key":"amishhhaaaa@gmail.com","avatar":null},"body":"PDF Version: https://docs.google.com/document/d/10H_FybR9Er7iDVwkisIRvjIZT_tEldwSu9jOZo7DRps/edit?usp=sharing\n\nOn Tue, 24 Mar 2026 at 05:38, Amisha Chhajed <amishhhaaaa@gmail.com> wrote:\n>\n> Hello! I am Amisha, I am interested in the project Improve disk space\n> recovery for partial clones, I would appreciate it if I can get\n> suggestions and improvement comments for my proposal. Thank you so\n> much.\n>\n> Improve disk space recovery for partial clones\n>\n> Mentors: Christian Couder, Karthik Nayak, Justin Tobler, Siddharth\n> Asthana, Ayush Chandekar.\n>\n> Project Size: 350 hours\n>\n> Difficulty: Medium to Hard.\n>\n> Personal Information:\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> Name: Amisha Chhajed\n>\n> Email: amishhhaaaa@gmail.com\n>\n> github: https://github.com/amishhaa\n>\n> Time Zone: UTC +5:30 (IST)\n>\n> Education: SVKM's Dwarkadas J. Sanghvi College of Engineering\n>\n> Year: 3rd year, 6th semester\n>\n> Degree: Bachelor of Technology in Artificial Intelligence and Data Science\n>\n> About me and past experience:\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> Hello, I am Amisha, currently in my penultimate year of engineering.I\n> am deeply passionate about contributing to open source and find great\n> fulfillment when I see my code helping people. Given this motivation,\n> I have contributed to various open source projects and the experience\n> has been extremly rewarding and ethereal.\n>\n> Apart from open source, I make art and I like building games.\n>\n>\n> I am currently doing my LFX at OpenTelemetry, my project is about\n> building a GO CLI tool that runs tests of the dependents of a GO\n> library to record any regresisons caused by new changes in the library\n> (more about my project:\n> https://mentorship.lfx.linuxfoundation.org/project/5f537fc2-548b-487a-99ed-c61f7e8bcd47)\n> (list of PRs created until now:\n> https://github.com/open-telemetry/opentelemetry-go-build-tools/issues?q=is%3Apr+author%3Aamishhaa)\n> inspired by Rust's Crater(https://github.com/rust-lang/crater), this\n> is a project/tool that I really love as it is the first time i have\n> made something E2E with a huge impact. I interned at Google for summer\n> of 2025 under team workspace serving infrastructure. Some open source\n> contributions that i am most proud of are in bitcoin core(refer\n> credits: https://bitcoincore.org/en/releases/29.2/ my PR:\n> https://github.com/bitcoin/bitcoin/pull/33482) and git :)\n>\n> My contributions in git:\n>\n> * cat-file: exit code of 'git cat-file' is suppressed by piping it\n> directly into grep.\n>\n> Status: Awaiting review\n>\n> Mailing List: https://lore.kernel.org/git/20260113180409.36683-1-amishhhaaaa@gmail.com/\n>\n> Log: This was the first patch i ever created for git as a\n> microproject, made me familiar with the mailing list workflow.\n>\n>\n> * sparse-checkout: optimize string_list construction and add tests to\n> verify deduplication.\n>\n> Status: merged in 'master'\n>\n> Mailing List: https://lore.kernel.org/git/20260121130005.72375-1-amishhhaaaa@gmail.com/\n>\n> Log: This is a really important patch for me, improves O(n^2)\n> complexity to O(n log n) of sparse-checkout by building a sorted\n> 'string_list' by constructing it unsorted then sorting it followed by\n> removing duplicates, this triggered a series of patches and uncovered\n> various bugs when i worked on replacing calls of string_list_sort()\n> and string_list_remove_duplicates() with string_list_sort_u().\n>\n>\n> *u-string-list: add unit tests for string-list methods.\n>\n> Status: merged in 'master'\n>\n> Mailing List: https://lore.kernel.org/git/20260129121220.69267-1-amishhhaaaa@gmail.com/\n>\n> Log: Adding unit tests for string-list methods which i saw were not\n> present when i was creating a new API string_list_sort_u.\n>\n>\n> * string-list: add string_list_sort_u() that mimics \"sort -u\"\n>\n> Status: merged in 'master'\n>\n> Mailing List: https://lore.kernel.org/git/20260129121220.69267-2-amishhhaaaa@gmail.com/\n>\n> Log: Adding a new API string_list_sort_u and cleaning up the callsites\n> that can directly adopt this new replacement, string_list_sort_u\n> mimics sort -u.\n>\n>\n> *sparse-checkout: use string_list_sort_u\n>\n> Status: merged in 'master'\n>\n> Mailing List: https://lore.kernel.org/git/20260212041017.91370-2-amishhhaaaa@gmail.com/\n>\n> Log: Small fix to replace a callsite of string_list_sort and\n> string_list_remove_duplicates with string_list_sort_u.\n>\n>\n> *help: cleanup the contruction of keys_uniq\n>\n> Status: Will merge to 'next'.\n>\n> Mailing List: https://lore.kernel.org/git/20260311192453.62213-1-amishhhaaaa@gmail.com/\n>\n> Log: Cleaning up complex callsites of string_list_sort and\n> string_list_remove_duplicates, this one involved finding a test case\n> that demonstarted a breakage,\n> https://lore.kernel.org/git/CAPvEtrenMBMFaMxcCR4VwoyMFU-_Z+bqq5nJaWv5eyn3HRutEA@mail.gmail.com/,\n> then replacing them with string_list_sort_u, another bug found as an\n> effort is https://lore.kernel.org/git/CAPvEtrfEZXHxcDf=z60ODfUA8cS81rhF1y7KEZApEBby7aCa1A@mail.gmail.com/,\n> this is still a pending bug which is good to work around in future, I\n> have provided a test to demonstrate an existing breakage.\n>\n>\n> History/Background, Overview and Project Theory\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> The \"Partial Clone\" feature is a performance optimization for Git that\n> allows Git to function without having a complete copy of the\n> repository, building partial clone was a community effort, with\n> contributions referenced here,\n> https://git-scm.com/docs/partial-clone#_related_links. Most of my\n> knowledge of this project came from researching and reading blogs like\n> https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/.\n> The talk https://www.youtube.com/watch?v=YdstUWcg5j4 by Derrick Stolee\n> are good sources to learn about how git currently handles objects.\n>\n>\n> There are 4 types of clones that we support, full clone, blobless\n> clone, treeless clone and shallow clone, each of them have their own\n> set of configurations that tells rest of git what objects are OK to be\n> missing from repository as they are promised, git stores objects in\n> .git/ folder under objects/, these objects can be of 4 types, tag,\n> tree, commits and blobs. When we partial clone, git writes the partial\n> clone type in config file inside .git/, for example,\n>\n> (base) amishachhajed@Amishas-MacBook-Pro .git % cat config\n>\n> [core]\n>\n> repositoryformatversion = 1\n>\n> filemode = true\n>\n> bare = false\n>\n> logallrefupdates = true\n>\n> ignorecase = true\n>\n> precomposeunicode = true\n>\n> [remote \"origin\"]\n>\n> url = https://github.com/python/cpython.git\n>\n> fetch = +refs/heads/*:refs/remotes/origin/*\n>\n> promisor = true\n>\n> partialclonefilter = tree:0\n>\n> [branch \"main\"]\n>\n> remote = origin\n>\n> merge = refs/heads/main\n>\n>\n> Each promised object has to be on some of the remote otherwise our\n> partial clone is broken and we do not account for broken remotes.\n>\n> However overtime a lot of objects might be fetched and currently there\n> is no functionality that, first checks if an object is present on a\n> promisor and then we can remove it safely to free up our disk space,\n> fundamentally users who make a partial clone want to save up space so\n> being able to remove the aquired objects on demand would be a great\n> addition.\n>\n>\n> This project aims to implement a command 'git evict' that can carry\n> out safe removal of promised objects from our disk space such that it\n> can be fetched later on from any of the remotes if needed,\n> dynamically.\n>\n> Proposed Plan:\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> Whenever we are evicting something we are unsure of its usefullness to\n> users, unless obvious, which git gc already handles. To overcome this,\n> we can make a new command 'git evict', essentially this command gives\n> users options and freedom to evict objects that they no longer\n> require, either it is out of cone or not on a checkout and a lot of\n> other situations, instead of predicting what users may or may not need\n> we can give them the choice of removal. In a case where users have a\n> cone and have set up a maintance task of evicting objects outside of\n> cone, in that case we can make it automatic.\n>\n>\n> As git currently treats remotes as source of thruth, we also trust\n> those remotes when evicting blobs because if all the remotes break the\n> principal of not strictly presenting the promised objects then the\n> partial clone is inherently broken, we can display a warning that\n> remote might not be able to fetch your file again if it gets\n> unavailable on the remote while evicting, however by making a partial\n> clone we have already placed our trust on the remotes.\n>\n>\n> Locally i attempted implementing 'git evict OID_TO_EVICT' which is\n> available here https://github.com/git/git/compare/master...amishhaa:git:par-c.\n> Steps to test it out:\n>\n> Partial clone a repository blobless,\n>\n> (base) amishachhajed@Amishas-MacBook-Pro % git clone\n> --filter=blob:none https://github.com/python/cpython.git\n>\n>\n> Get OID for a blob, here i am getting it for README,\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git rev-parse\n> HEAD:README.rst\n>\n> 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n>\n>\n> cat-file the blob to make sure it is present on disk\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t\n> 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n>\n> remote: Enumerating objects: 1, done.\n>\n> remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1 (from 1)\n>\n> Receiving objects: 100% (1/1), 3.50 KiB | 3.50 MiB/s, done.\n>\n> blob\n>\n> Run the git evict command with the object oid we recieved from the\n> previous step.\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git evict\n> 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n>\n> DEBUG: Evicted 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec from pack entry search\n>\n> Enumerating objects: 810655, done.\n>\n> Counting objects: 100% (810654/810654), done.\n>\n> Delta compression using up to 12 threads\n>\n> Compressing objects: 100% (199618/199618), done.\n>\n> Writing objects: 100% (810654/810654), done.\n>\n> Total 810654 (delta 608469), reused 810654 (delta 608469), pack-reused\n> 0 (from 0)\n>\n>\n> Attempt to cat-file again.\n>\n> (Without network it should fail at fetching)\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t\n> 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n>\n> fatal: unable to access 'https://github.com/python/cpython.git/':\n> Could not resolve host: github.com\n>\n> fatal: could not fetch 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec from\n> promisor remote\n>\n>\n> (With network it would re-fetch)\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git cat-file -t\n> 1d2874e9ca4fdcbfc5ef14b0202dd73d707bc9ec\n>\n> remote: Enumerating objects: 1, done.\n>\n> remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1 (from 1)\n>\n> Receiving objects: 100% (1/1), 3.50 KiB | 3.50 MiB/s, done.\n>\n> blob\n>\n>\n> However the current implementation I have locally has no validation on\n> an OID, to make sure it is a promisr object before attempting to evict\n> it, hence it fails when i try to evict a commit object, that is not\n> promised in a partial clone.\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git rev-parse HEAD^{commit}\n>\n> 306c556fdbe7034c9a6d87516534eecdb911ad11\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git evict\n> 306c556fdbe7034c9a6d87516534eecdb911ad11\n>\n>\n> (base) amishachhajed@Amishas-MacBook-Pro cpython % git evict\n> 306c556fdbe7034c9a6d87516534eecdb911ad11\n>\n> Enumerating objects: 810831, done.\n>\n> Counting objects: 100% (810831/810831), done.\n>\n> Delta compression using up to 12 threads\n>\n> Compressing objects: 100% (200113/200113), done.\n>\n> Writing objects: 100% (810831/810831), done.\n>\n> Total 810831 (delta 608149), reused 810831 (delta 608149), pack-reused\n> 0 (from 0)\n>\n> fatal: bad object refs/heads/main\n>\n>\n> So, when user calls git evict, it internally finds an oidset with\n> objects to evict based on the flag the user has set, and then skips\n> those objects in want_object_in_pack() method, so we would be able to\n> evict everything in a single repack operation.\n>\n>\n> Currently on the user interface i am planning to implement these\n> flags, inspired heavily from reversing how we partial clone, for say\n> when partial cloning i set blob:limit=<size> then i might want to\n> evict blobs greater than that size in future.\n>\n> * --outside-cone: Evict blobs that are not part of the current\n> sparse-checkout cone.\n>\n> https://lore.kernel.org/git/735eb76e-44a9-4f79-b769-23a3a07437ae@gmail.com/)\n>\n> * --large-only=<size>: Evict blobs that are bigger than a certain size.\n>\n> * --tree-depth=<n>: Evict blobs with depth > n\n>\n> * --outside-checkout: Evict blobs that are not part of the current checkout.\n>\n>\n> This project can essentially be split into two parts:\n>\n> * Given an object, we can use method is_promisor_object() to check if\n> it is promised, if yes we can repack our packfiles without that\n> object, we would skip those OIDs in want_object_in_pack().\n>\n> These OIDs that we have to skip will be passed from evict to\n> want_object_in_pack() method.\n>\n>\n> * Now comes gathering the list of objects for oidset, for each command\n> written above we would need a different method to find and append the\n> object to evict, in oidset. For example, for objects\n> --large-only=<size>, we need to find objects that are larger than the\n> size specified and mark it for eviction, similar methods to handle\n> such filters have been written in our codebase and can be used as\n> reference.\n>\n>\n> * Optionally also add git evict methods to git maintainence.\n>\n> Project Timeline:\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> Community Bonding (Until May 24th)\n>\n> * Discuss project ideas and design implementation details with\n> mentors, possible subcommands to keep and the overall architecture of\n> command, any optimizations we can make when repacking objects.\n>\n> * Study different types of partial clones and how refs work, and which\n> OIDs i can evict in different types of partial clones.\n>\n> * All stages would be accompained by necessary documentation and tests.\n>\n>\n> Coding Period\n>\n> (May 25 - June 30)\n>\n> * Implement a basic git evict command.\n>\n> * Implement evict_object method, this evict object method would need\n> to handle different cases when we created a partial clone, that is, if\n> we created a treeless clone or a blobless clone, for example if we\n> have a blobless clone we cannot evict a OID referencing a commit\n> object or a tag object because then refs would complain. It would need\n> to handle most of the cases that partial clone handles.\n>\n> * evict_object method would only be responsible to evict an OID initially.\n>\n> (July 1st - August 16)\n>\n> * Implement different ways to gather OIDs for the flag that we are referencing.\n>\n> * Pass on those OIDs to evict_objects method which would internally\n> call evict_object method, that handles all the different object cases.\n>\n>\n> Final Week (August 17 - August 24)\n>\n> *This is a buffer period for any unforseen delays and prepare final\n> report of everything we have accomplished over the summer :)\n>\n> Availibility:\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> I would be dedicating 45-50 hours per week of my time to this project\n> weekly. I dont have any other commitments apart from LFX in month of\n> May which would be easy to manage given my summer break, so during the\n> month of May my time commitment would be 25 hours to this project,\n> incase of extensions i have my internship break next semester so this\n> would be considered as that hence i do not have any university work\n> which would be interfering with the timeline even after my summer\n> break.\n>\n> Post-GSOC and Appreciation:\n>\n> —----------------------------------------------------------------------------------------------------------------------------\n>\n> I want my journey with git to be a long one, it is very fulfilling for\n> me to see my code running on many devices, it is like a dream come\n> true for me, so even post GSOC I intend to keep contributing to git.\n>\n>\n> Contirbuting to git uptill now has been insanely rewarding, i learned\n> about commands that i did not even know existed, it feels magical to\n> know the internals of a tool i have been using since years and be a\n> part of the making as well.\n>\n>\n> I would like to thank the reviewers and maintainers, without them my\n> journey would'nt have been so easy, i learned some of my best software\n> development lessons on this very list which i am insanely grateful\n> for. Thank you.\n>\n>\n>\n> --\n> Thanks,\n> Amisha\n\n\n\n-- \nThanks,\nAmisha\n"}]}