{"thread":{"id":"59054","subject":"Reducing Git Repository size - git-filter-repo doesn't help","startedAt":"2023-01-09T02:22:31Z","lastAt":"2023-01-12T02:01:55Z","messageCount":11,"participants":["fawaz ahmed0","rsbecker@nexbridge.com","Erik Cervin Edin","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"469928","messageId":"PH7PR12MB779560B6C003FEE76E4F2471F1FE9@PH7PR12MB7795.namprd12.prod.outlook.com","threadId":"59054","inReplyTo":null,"subject":"Reducing Git Repository size - git-filter-repo doesn't help","fromName":"fawaz ahmed0","fromEmail":"fawazahmed0@hotmail.com","sentAt":"2023-01-09T02:22:25Z","receivedAt":"2023-01-09T02:22:31Z","isPatch":false,"sender":{"key":"fawazahmed0@hotmail.com","avatar":null},"body":"Hi,\n\nI have this huge repo: https://github.com/fawazahmed0/currency-api#readme and I am trying to reduce its size.\n\nI have run filter-repo script on this repo ( https://github.com/fawazahmed0/currency-api/blob/1/.github/workflows/cleanup-repo.yml )\nThe commits were reduced from 1k to 600 , but the space used is still same. (i.e size-pack: 6.47 GiB , https://github.com/fawazahmed95/currency-api/actions/runs/3865919157/jobs/6589710845#step:5:1498 )\n\nAlmost all commits of this repo were applied on partially cloned repository: ( https://github.com/fawazahmed0/currency-api/blob/1/.github/workflows/run.yml )\nSo I guess it had never run any git maintenance task in it's life.\n\nI am not sure what needs to be done to reduce it's space utilization. ( https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#:~:text=less%20than%205%20GB%20is%20strongly%20recommended )\n\n\nThanks,\nFawaz Ahmed\n\n\n\n\n\n\n\n"},{"id":"469929","messageId":"017d01d923d3$20b90bd0$622b2370$@nexbridge.com","threadId":"59054","inReplyTo":"PH7PR12MB779560B6C003FEE76E4F2471F1FE9@PH7PR12MB7795.namprd12.prod.outlook.com","subject":"RE: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-01-09T02:36:02Z","receivedAt":"2023-01-09T02:37:59Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On January 8, 2023 9:22 PM, fawaz ahmed0 wrote:\n>I have this huge repo: https://github.com/fawazahmed0/currency-api#readme\n>and I am trying to reduce its size.\n>\n>I have run filter-repo script on this repo (\n>https://github.com/fawazahmed0/currency-\n>api/blob/1/.github/workflows/cleanup-repo.yml ) The commits were reduced\n>from 1k to 600 , but the space used is still same. (i.e size-pack: 6.47 GiB\n,\n>https://github.com/fawazahmed95/currency-\n>api/actions/runs/3865919157/jobs/6589710845#step:5:1498 )\n>\n>Almost all commits of this repo were applied on partially cloned\nrepository: (\n>https://github.com/fawazahmed0/currency-\n>api/blob/1/.github/workflows/run.yml ) So I guess it had never run any git\n>maintenance task in it's life.\n>\n>I am not sure what needs to be done to reduce it's space utilization. (\n>https://docs.github.com/en/repositories/working-with-files/managing-large-\n>files/about-large-files-on-\n>github#:~:text=less%20than%205%20GB%20is%20strongly%20recommended )\n\nThe first thing you can try is 'git gc --aggressive' to reduce the clone\nsize. Github automatically does garbage collection. If this is a question of\nthe size of the working index, look at the sparse-checkout command.\n\n--Randall\n\n"},{"id":"469934","messageId":"PH7PR12MB779523B6D7BA03A8725ADED3F1FE9@PH7PR12MB7795.namprd12.prod.outlook.com","threadId":"59054","inReplyTo":"017d01d923d3$20b90bd0$622b2370$@nexbridge.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"fawaz ahmed0","fromEmail":"fawazahmed0@hotmail.com","sentAt":"2023-01-09T03:29:17Z","receivedAt":"2023-01-09T03:29:24Z","isPatch":false,"sender":{"key":"fawazahmed0@hotmail.com","avatar":null},"body":"\n\n\nFrom: rsbecker@nexbridge.com \nSent: Monday, January 9, 2023 8:06 AM\nTo: 'fawaz ahmed0' ; git@vger.kernel.org \nSubject: RE: Reducing Git Repository size - git-filter-repo doesn't help \n \nOn January 8, 2023 9:22 PM, fawaz ahmed0 wrote:\n>I have this huge repo: https://github.com/fawazahmed0/currency-api#readme\n>and I am trying to reduce its size.\n>\n>I have run filter-repo script on this repo (\n>https://github.com/fawazahmed0/currency-\n>api/blob/1/.github/workflows/cleanup-repo.yml ) The commits were reduced\n>from 1k to 600 , but the space used is still same. (i.e size-pack: 6.47 GiB\n,\n>https://github.com/fawazahmed95/currency-\n>api/actions/runs/3865919157/jobs/6589710845#step:5:1498 )\n>\n>Almost all commits of this repo were applied on partially cloned\nrepository: (\n>https://github.com/fawazahmed0/currency-\n>api/blob/1/.github/workflows/run.yml ) So I guess it had never run any git\n>maintenance task in it's life.\n>\n>I am not sure what needs to be done to reduce it's space utilization. (\n>https://docs.github.com/en/repositories/working-with-files/managing-large-\n>files/about-large-files-on-\n>github#:~:text=less%20than%205%20GB%20is%20strongly%20recommended )\n\nThe first thing you can try is 'git gc --aggressive' to reduce the clone\nsize. Github automatically does garbage collection. If this is a question of\nthe size of the working index, look at the sparse-checkout command.\n\n--Randall\n\n------------\n\nYes I have already tried git gc --aggressive ( https://github.com/fawazahmed0/currency-api/blob/1a1fb65703a2fc352b0ee452ce908bee545698c2/.github/workflows/cleanup-repo.yml#L78 ) . I am using sparse-checkout to manage the repository. \n\nThe thing is the space utilized by the repository is not being reduced, even on deleting the data using git-filter-repo script. \n\nThanks,\nFawaz Ahmed\n\n"},{"id":"469962","messageId":"CA+JQ7M--TeqxH3nj95sAvGTPz9FbHkS2eYfe763w05fUZX0XCA@mail.gmail.com","threadId":"59054","inReplyTo":"PH7PR12MB779560B6C003FEE76E4F2471F1FE9@PH7PR12MB7795.namprd12.prod.outlook.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2023-01-09T10:22:13Z","receivedAt":"2023-01-09T10:24:04Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"On Mon, Jan 9, 2023 at 3:24 AM fawaz ahmed0 wrote:\n> I have this huge repo: https://github.com/fawazahmed0/currency-api#readme and I am trying to reduce its size.\n>\n> I have run filter-repo script on this repo ( https://github.com/fawazahmed0/currency-api/blob/1/.github/workflows/cleanup-repo.yml )\n\nCan you elaborate exactly how you're trying to reduce the repository?\nLooking at the script it seems you're removing /latest? And/or folders\ncorresponding to certain years?\n\n> The commits were reduced from 1k to 600 , but the space used is still same. (i.e size-pack: 6.47 GiB , https://github.com/fawazahmed95/currency-api/actions/runs/3865919157/jobs/6589710845#step:5:1498 )\n\nThe number of commits is actually irrelevant, what matters is really\nonly how much of the tree was pruned. And only if what was pruned\nwasn't duplicated.\n\nSay you commit\n 2018/big.json\n 2019/same-identical-big.json\nand then delete 2018, the size of the repository in its packed state\nwill be virtually identical.\n\nYou can analyze which files and directories are occupying the most\nspace by running\n git filter-repo --analyze\nand checking the output file. (It's somewhere like\n.git/filter-repo/analysis I don't remember exactly)\n\nHowever, it seems that you're using git in a highly unconventional\nmanner and I'd say it's probably worthwhile to consider if it's even\nthe appropriate tool for the task at hand.\n"},{"id":"470019","messageId":"CABPp-BE_FLFL3-s3936zWWfaORBPim14A-oqVNo+8gx+CMduHA@mail.gmail.com","threadId":"59054","inReplyTo":"PH7PR12MB779560B6C003FEE76E4F2471F1FE9@PH7PR12MB7795.namprd12.prod.outlook.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-01-10T02:42:00Z","receivedAt":"2023-01-10T02:43:55Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Jan 8, 2023 at 6:54 PM fawaz ahmed0 wrote:\n>\n> Hi,\n>\n> I have this huge repo: https://github.com/fawazahmed0/currency-api#readme and I am trying to reduce its size.\n>\n> I have run filter-repo script on this repo ( https://github.com/fawazahmed0/currency-api/blob/1/.github/workflows/cleanup-repo.yml )\n\nWhy are you cleaning up in a CI task? filter-repo is intended for a\none-shot \"flag day\" type cleanup, not something you repeatedly do.\nSomething seems a bit off already.\n\n> The commits were reduced from 1k to 600 , but the space used is still same. (i.e size-pack: 6.47 GiB , https://github.com/fawazahmed95/currency-api/actions/runs/3865919157/jobs/6589710845#step:5:1498 )\n\nYou show the ending size, but not the starting. Could you provide\nthat number and how you got it, so we can see what you're measuring\n(especially since below it's not at all clear what you're even\nmeasuring?)\n\nThe reduction in commits suggests it certainly did do some kind of\npruning, and you might also want to look at the output of running\n\"python3 git-filter-repo --analyze\", both before and after filtering,\nto get an idea of what's is/was using lots of space.\n\nTaking a closer look, I suspect you are missing some important\ncleaning. When there are multiple copies of a file in a repository,\ngit only stores one version. Based on\nhttps://github.com/fawazahmed0/currency-api/issues/55, all the files\nthat are now in directories that you are deleting used to be in the\nroot folder under another name. The files with the old name aren't\ngoing to be deleted by your pruning since you only requested that the\nnew names of the files be deleted. If I'm understanding your\nstructure correctly (I didn't clone your repo or try this out; I'm\nmaking inferences based on poking around at the links you provided and\nlooking at that issue), the upshot of that is that your filtering\nprobably won't shrink things much since you are still keeping a copy\nof those files. Again, \"python3 git-filter-repo --analyze\" both\nbefore and after filtering will help you find these kinds of things\nand/or other problems.\n\n> Almost all commits of this repo were applied on partially cloned repository: ( https://github.com/fawazahmed0/currency-api/blob/1/.github/workflows/run.yml )\n> So I guess it had never run any git maintenance task in it's life.\n\nHow exactly are you measuring the size, given that you have a partial\nclone? You don't even have the objects in order to measure, so I\ndon't understand how you are measuring. I'm even suspecting you are\nmeasuring something else entirely; could you clarify all your size\nmeasurements and how you got them?\n\n> I am not sure what needs to be done to reduce it's space utilization. ( https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#:~:text=less%20than%205%20GB%20is%20strongly%20recommended )\n\nNote that git-filter-repo only changes the size of the _local_ repo.\nYou made an additional clone within GitHub Actions, and then\nfilter-repo shrinks *that* clone. Even if you had deleted all copies\nof older files you don't want anymore locally (which is suspect as I\nnoted above), your force pushing isn't going to shrink the size of the\nrepo on the server (i.e. on GitHub) since there are pull requests in\nyour repo that GitHub won't allow you to overwrite via force-push, and\nthose pull requests still hold on to the old history.\n\nYou probably want to read the \"DISCUSSION\" section of the filter-repo\nmanual, and you may also want to see GitHub's documentation on\nshrinking repos, up at\nhttps://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository.\nIt appears you've skipped the whole \"Fully removing the data from\nGitHub\" section of their documentation.\n"},{"id":"470048","messageId":"PH7PR12MB7795ED45E989E2EDE2364049F1FF9@PH7PR12MB7795.namprd12.prod.outlook.com","threadId":"59054","inReplyTo":"CABPp-BE_FLFL3-s3936zWWfaORBPim14A-oqVNo+8gx+CMduHA@mail.gmail.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"fawaz ahmed0","fromEmail":"fawazahmed0@hotmail.com","sentAt":"2023-01-10T08:18:51Z","receivedAt":"2023-01-10T08:19:07Z","isPatch":false,"sender":{"key":"fawazahmed0@hotmail.com","avatar":null},"body":"\n> Note that git-filter-repo only changes the size of the _local_ repo.\n You made an additional clone within GitHub Actions, and then\n filter-repo shrinks *that* clone.  Even if you had deleted all copies\n of older files you don't want anymore locally (which is suspect as I\n noted above), your force pushing isn't going to shrink the size of the\n repo on the server (i.e. on GitHub) since there are pull requests in\n your repo that GitHub won't allow you to overwrite via force-push, and\n those pull requests still hold on to the old history.\n\nThanks for your inputs. Yes, the issue is with the GitHub itself (I have verified it locally), so when I force push this reduced sized repo to Github and reclone it back the size gets back to 4+ GB.\nSo what do you suggest? \n\nI have already contacted Github few days back, and they have cleared the cache etc for this repo.(but that did not reduce any size). ( I will try contacting them again)\n\n\n\n\n> Why are you cleaning up in a CI task?\nTask is yearly cleanup, to keep repo under 5GB limit as recommened by Github ( To avoid receiving email from GitHub : https://github.com/whosonfirst-data/whosonfirst-data/issues/1507 )\n\n\n> Could you provide that number and how you got it, so we can see what you're measuring\nI was checking the repo size here: https://github.com/settings/repositories (it takes around 1 hour for github to show updated size after any commit)\n\n\n\n\n\n\n"},{"id":"470085","messageId":"CABPp-BGh17pH1DUgPOoaDKBcKay3NAyi4js-HnEKAAyYUBSoYQ@mail.gmail.com","threadId":"59054","inReplyTo":"PH7PR12MB7795ED45E989E2EDE2364049F1FF9@PH7PR12MB7795.namprd12.prod.outlook.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-01-11T02:18:00Z","receivedAt":"2023-01-11T02:18:46Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 10, 2023 at 12:18 AM fawaz ahmed0 wrote:\n>\n>\n> > Note that git-filter-repo only changes the size of the _local_ repo.\n> You made an additional clone within GitHub Actions, and then\n> filter-repo shrinks *that* clone. Even if you had deleted all copies\n> of older files you don't want anymore locally (which is suspect as I\n> noted above), your force pushing isn't going to shrink the size of the\n> repo on the server (i.e. on GitHub) since there are pull requests in\n> your repo that GitHub won't allow you to overwrite via force-push, and\n> those pull requests still hold on to the old history.\n>\n> Thanks for your inputs. Yes, the issue is with the GitHub itself (I have verified it locally), so when I force push this reduced sized repo to Github and reclone it back the size gets back to 4+ GB.\n> So what do you suggest?\n\nWell, digging a bit further, I see:\n```\n$ git ls-remote https://github.com/fawazahmed0/currency-api\n5536f2096aec71cee6d35772697ce1efc6148521 HEAD\n5536f2096aec71cee6d35772697ce1efc6148521 refs/heads/1\n7ed955eebe2e5a670566d5fb4e11142bae6afed6 refs/heads/patch-1\n0292ec7340ec1ac8cd11d5839b8a25dd0c1db38f refs/pull/1/head\nfa18a4ae50cc8ad84643d1f1f92da77c35139bbc refs/pull/16/head\nbca74c53f66681f2d74a98d2f785e1610abeb794 refs/pull/18/head\n4da5bcd8c7852bb306059a856f95e9bc1aeaefd3 refs/pull/47/head\n5f1408be857734745a7a3349e9d16a8200e993be refs/pull/49/head\n00ff16478e95f11347bc3e769e310c1a254579c9 refs/pull/57/head\n5e3121acaddc7be810f2b9ad92641a04f9b923fb refs/pull/58/head\n8afe04f52b8174c789691b4097c70f95d33c99c4 refs/pull/59/head\n55b0d38d156145d8f9726746da7bb4d5ba9d988c refs/pull/60/head\nbca74c53f66681f2d74a98d2f785e1610abeb794 refs/reviewable/pr18/r1\n4da5bcd8c7852bb306059a856f95e9bc1aeaefd3 refs/reviewable/pr47/r1\n5f1408be857734745a7a3349e9d16a8200e993be refs/reviewable/pr49/r1\n00ff16478e95f11347bc3e769e310c1a254579c9 refs/reviewable/pr57/r1\n5e3121acaddc7be810f2b9ad92641a04f9b923fb refs/reviewable/pr58/r1\n8afe04f52b8174c789691b4097c70f95d33c99c4 refs/reviewable/pr59/r1\n55b0d38d156145d8f9726746da7bb4d5ba9d988c refs/reviewable/pr60/r1\n```\n\nSo, you have 19 references. And according to the output of the job\nyou linked, you only force pushed refs/heads/1. That means the other\nbranches and refs are holding on to the old history. You'll either\nneed to delete or rewrite refs/heads/patch-1, and\nrefs/reviewable/pr{18,47,49,57,58,59,60}/r1 as well. Rewriting the\nrefs/reviewable stuff may confuse and mess up Reviewable if you ever\nlook at those reviews again; it may be that deleting those refs\nresults in a cleaner error message within Reviewable.\n\nYou also need to have refs/pull/{1,16,18,47,49,57,58,59,60}/head all\nrewritten too, but you don't have access. You need to get GitHub\nsupport to do that. If they don't ask for the exact filtering you\nhave done, and are only \"clearing caches\", then it's not going to do\nanyone any good. Those references need to be filtered the same way,\nor else those special refs need to be deleted. But please do note\nthat you should not ask them to do that until you've already cleaned\nup all the stuff that you can.\n\nThe steps look roughly as follows:\n\n1. Clone the repository. Since you have refs outside of refs/heads/\nand refs/tags/ (namely refs/reviewable/), you'll need a mirror clone.\nAlso, make sure to NOT use a partial clone (which would defeat steps 2\n& 5).\n2. Note the size of the *local* clone ('du -hs' should come in handy).\n3. Run `python3 git-filter-repo --analyze`, and look at the reports it\ngenerates to find out the big files/directories and all the names of\nthose paths.\n4. Filter the repository.\n5. Verify that the local clone actually shrinks from your filtering\noperation (not just has fewer commits, but 'du -hs' now reports a much\nsmaller number). If it does not shrink as much as expected, run\n`python3 git-filter-repo --analyze` again and see if you missed\nalternate historical names of some files or whether you only filtered\nwhat turned out to be small files or whatever.\n\nNote that step 3 is important. Rather than guessing what is big and\ntaking up lots of space, you find out what is big and take action upon\nit. For example, you once committed the node_modules/ folder. I do\nnot know its size (too lazy to do a full clone and find out), but\nthose are often quite large. Creating a commit that deletes that\ndirectory does not remove it from history, only from the current\nversion. So, expunging that directory from history may be important.\nThere may be other nuggets you find too, such as alternate names of\nfiles that also need to be deleted if you've renamed things.\n\nAfter you've succeeded with all the above:\n6. force push *all* references back to GitHub -- or at least the ones\nthat GitHub will permit you to force push (should be everything other\nthan refs/pull/*). If you only force push your \"1\" branch, you still\nleave every other reference holding on to the old history.\n7. Contact GitHub support to ensure they clean up the remaining\nreferences (i.e. the refs/pull/* ones) _and_ clear their caches. They\nwill either have to ask for what filtering you did so they can do it\nas well, or they'll need to nuke those pull requests.\n8. Wait to hear back\n9. Check the `git ls-remote\nhttps://github.com/fawazahmed0/currency-api` output again. If the\nrefs/pull/* lines still exist and still have the same hashes at the\nbeginning of their lines, GitHub did not filter those references and\nthey are still holding on to the old history. Contact them again, and\neither get them to filter those refs the same way you filtered yours,\nor get them to delete those pull requests.\n\n> I have already contacted Github few days back, and they have cleared the cache etc for this repo.(but that did not reduce any size). ( I will try contacting them again)\n\nContacting them now is a waste of their time. Filter your repo first.\nYour whole repo (implying you need a mirror clone rather than a\nregular clone given you have the unusual refs/reviewable/* references\nthat need filtering too). And push the whole repo back, not just a\nsingle branch.\n"},{"id":"470087","messageId":"CABPp-BHBEng3=_mLgdKu+ZNGfSZdcZUZZCX-p9two+af+GecfQ@mail.gmail.com","threadId":"59054","inReplyTo":"CABPp-BGh17pH1DUgPOoaDKBcKay3NAyi4js-HnEKAAyYUBSoYQ@mail.gmail.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-01-11T08:41:00Z","receivedAt":"2023-01-11T08:42:38Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 10, 2023 at 6:18 PM Elijah Newren wrote:\n>\n> On Tue, Jan 10, 2023 at 12:18 AM fawaz ahmed0 wrote:\n> >\n> >\n> > > Note that git-filter-repo only changes the size of the _local_ repo.\n> > You made an additional clone within GitHub Actions, and then\n> > filter-repo shrinks *that* clone. Even if you had deleted all copies\n> > of older files you don't want anymore locally (which is suspect as I\n> > noted above), your force pushing isn't going to shrink the size of the\n> > repo on the server (i.e. on GitHub) since there are pull requests in\n> > your repo that GitHub won't allow you to overwrite via force-push, and\n> > those pull requests still hold on to the old history.\n> >\n> > Thanks for your inputs. Yes, the issue is with the GitHub itself (I have verified it locally), so when I force push this reduced sized repo to Github and reclone it back the size gets back to 4+ GB.\n> > So what do you suggest?\n>\n> Well, digging a bit further, I see:\n> ```\n> $ git ls-remote https://github.com/fawazahmed0/currency-api\n> 5536f2096aec71cee6d35772697ce1efc6148521 HEAD\n> 5536f2096aec71cee6d35772697ce1efc6148521 refs/heads/1\n> 7ed955eebe2e5a670566d5fb4e11142bae6afed6 refs/heads/patch-1\n> 0292ec7340ec1ac8cd11d5839b8a25dd0c1db38f refs/pull/1/head\n> fa18a4ae50cc8ad84643d1f1f92da77c35139bbc refs/pull/16/head\n> bca74c53f66681f2d74a98d2f785e1610abeb794 refs/pull/18/head\n> 4da5bcd8c7852bb306059a856f95e9bc1aeaefd3 refs/pull/47/head\n> 5f1408be857734745a7a3349e9d16a8200e993be refs/pull/49/head\n> 00ff16478e95f11347bc3e769e310c1a254579c9 refs/pull/57/head\n> 5e3121acaddc7be810f2b9ad92641a04f9b923fb refs/pull/58/head\n> 8afe04f52b8174c789691b4097c70f95d33c99c4 refs/pull/59/head\n> 55b0d38d156145d8f9726746da7bb4d5ba9d988c refs/pull/60/head\n> bca74c53f66681f2d74a98d2f785e1610abeb794 refs/reviewable/pr18/r1\n> 4da5bcd8c7852bb306059a856f95e9bc1aeaefd3 refs/reviewable/pr47/r1\n> 5f1408be857734745a7a3349e9d16a8200e993be refs/reviewable/pr49/r1\n> 00ff16478e95f11347bc3e769e310c1a254579c9 refs/reviewable/pr57/r1\n> 5e3121acaddc7be810f2b9ad92641a04f9b923fb refs/reviewable/pr58/r1\n> 8afe04f52b8174c789691b4097c70f95d33c99c4 refs/reviewable/pr59/r1\n> 55b0d38d156145d8f9726746da7bb4d5ba9d988c refs/reviewable/pr60/r1\n> ```\n>\n> So, you have 19 references. And according to the output of the job\n> you linked, you only force pushed refs/heads/1. That means the other\n> branches and refs are holding on to the old history. You'll either\n> need to delete or rewrite refs/heads/patch-1, and\n> refs/reviewable/pr{18,47,49,57,58,59,60}/r1 as well. Rewriting the\n> refs/reviewable stuff may confuse and mess up Reviewable if you ever\n> look at those reviews again; it may be that deleting those refs\n> results in a cleaner error message within Reviewable.\n>\n> You also need to have refs/pull/{1,16,18,47,49,57,58,59,60}/head all\n> rewritten too, but you don't have access. You need to get GitHub\n> support to do that. If they don't ask for the exact filtering you\n> have done, and are only \"clearing caches\", then it's not going to do\n> anyone any good. Those references need to be filtered the same way,\n> or else those special refs need to be deleted. But please do note\n> that you should not ask them to do that until you've already cleaned\n> up all the stuff that you can.\n>\n> The steps look roughly as follows:\n>\n> 1. Clone the repository. Since you have refs outside of refs/heads/\n> and refs/tags/ (namely refs/reviewable/), you'll need a mirror clone.\n> Also, make sure to NOT use a partial clone (which would defeat steps 2\n> & 5).\n> 2. Note the size of the *local* clone ('du -hs' should come in handy).\n> 3. Run `python3 git-filter-repo --analyze`, and look at the reports it\n> generates to find out the big files/directories and all the names of\n> those paths.\n> 4. Filter the repository.\n> 5. Verify that the local clone actually shrinks from your filtering\n> operation (not just has fewer commits, but 'du -hs' now reports a much\n> smaller number). If it does not shrink as much as expected, run\n> `python3 git-filter-repo --analyze` again and see if you missed\n> alternate historical names of some files or whether you only filtered\n> what turned out to be small files or whatever.\n>\n> Note that step 3 is important. Rather than guessing what is big and\n> taking up lots of space, you find out what is big and take action upon\n> it. For example, you once committed the node_modules/ folder. I do\n> not know its size (too lazy to do a full clone and find out), but\n> those are often quite large. Creating a commit that deletes that\n> directory does not remove it from history, only from the current\n> version. So, expunging that directory from history may be important.\n> There may be other nuggets you find too, such as alternate names of\n> files that also need to be deleted if you've renamed things.\n>\n> After you've succeeded with all the above:\n> 6. force push *all* references back to GitHub -- or at least the ones\n> that GitHub will permit you to force push (should be everything other\n> than refs/pull/*). If you only force push your \"1\" branch, you still\n> leave every other reference holding on to the old history.\n> 7. Contact GitHub support to ensure they clean up the remaining\n> references (i.e. the refs/pull/* ones) _and_ clear their caches. They\n> will either have to ask for what filtering you did so they can do it\n> as well, or they'll need to nuke those pull requests.\n> 8. Wait to hear back\n> 9. Check the `git ls-remote\n> https://github.com/fawazahmed0/currency-api` output again. If the\n> refs/pull/* lines still exist and still have the same hashes at the\n> beginning of their lines, GitHub did not filter those references and\n> they are still holding on to the old history. Contact them again, and\n> either get them to filter those refs the same way you filtered yours,\n> or get them to delete those pull requests.\n\nI may need to take this partially back. Since you've already\ncontacted GitHub support, they may have already done filtering on\nthese references and further filtering of those isn't needed. It may\nbe that after filtering the references and your control and pushing\nthose all back, that no further work is needed.\n\n> > I have already contacted Github few days back, and they have cleared the cache etc for this repo.(but that did not reduce any size). ( I will try contacting them again)\n>\n> Contacting them now is a waste of their time. Filter your repo first.\n> Your whole repo (implying you need a mirror clone rather than a\n> regular clone given you have the unusual refs/reviewable/* references\n> that need filtering too). And push the whole repo back, not just a\n> single branch.\n"},{"id":"470101","messageId":"PH7PR12MB77959E90F52C59140ED662DBF1FC9@PH7PR12MB7795.namprd12.prod.outlook.com","threadId":"59054","inReplyTo":"CABPp-BGh17pH1DUgPOoaDKBcKay3NAyi4js-HnEKAAyYUBSoYQ@mail.gmail.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"fawaz ahmed0","fromEmail":"fawazahmed0@hotmail.com","sentAt":"2023-01-11T20:08:32Z","receivedAt":"2023-01-11T20:09:34Z","isPatch":false,"sender":{"key":"fawazahmed0@hotmail.com","avatar":null},"body":"Thank you for looking into this. \n\nI have removed all the unnecessary refs on remote, and rerun the filter-repo and force pushed the changes again.\nTo be be sure, I have also requested Github support to completely nuke the refs/pull and clear cache etc.\n\nAlso would like to let you know that git-filter-repo --analyze (which was also suggested by EriK), causes OOM in my case (also tried running in github actions with huge swap, but it exceeds 6 hours timeout limit).\n\nI have verified size reduction (after running filter-repo) using:\n`git gc && git count-objects -vH`\n\nThank you once again for the help"},{"id":"470131","messageId":"PH7PR12MB779537896DF1C71188E0E683F1FD9@PH7PR12MB7795.namprd12.prod.outlook.com","threadId":"59054","inReplyTo":"PH7PR12MB77959E90F52C59140ED662DBF1FC9@PH7PR12MB7795.namprd12.prod.outlook.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"fawaz ahmed0","fromEmail":"fawazahmed0@hotmail.com","sentAt":"2023-01-12T01:54:06Z","receivedAt":"2023-01-12T01:54:13Z","isPatch":false,"sender":{"key":"fawazahmed0@hotmail.com","avatar":null},"body":"Hi,\n\nI have checked the repository size at here: https://github.com/settings/repositories and it got reduced from 4.7Gb to 3.3Gb.\n\nThank you, have been trying to reduce the repo size from last 15 days.\n\nRegards,\nFawaz Ahmed"},{"id":"470133","messageId":"CABPp-BEKywBvsjDmawGkBMnp+n8A8Nc3DJ3F0ceuvMwh+NFKFA@mail.gmail.com","threadId":"59054","inReplyTo":"PH7PR12MB779537896DF1C71188E0E683F1FD9@PH7PR12MB7795.namprd12.prod.outlook.com","subject":"Re: Reducing Git Repository size - git-filter-repo doesn't help","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-01-12T02:01:38Z","receivedAt":"2023-01-12T02:01:55Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jan 11, 2023 at 5:54 PM fawaz ahmed0 wrote:\n>\n> Hi,\n>\n> I have checked the repository size at here: https://github.com/settings/repositories and it got reduced from 4.7Gb to 3.3Gb.\n>\n> Thank you, have been trying to reduce the repo size from last 15 days.\n>\n> Regards,\n> Fawaz Ahmed\n\nAwesome, glad you got it squared away!\n"}]}