{"thread":{"id":"64100","subject":"Running out of inodes on an NFS which stores repos","startedAt":"2025-09-06T14:18:50Z","lastAt":"2025-09-09T07:39:56Z","messageCount":6,"participants":["Kousik Sanagavarapu","rsbecker@nexbridge.com","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"525692","messageId":"20250906141711.64419-1-five231003@gmail.com","threadId":"64100","inReplyTo":null,"subject":"Running out of inodes on an NFS which stores repos","fromName":"Kousik Sanagavarapu","fromEmail":"five231003@gmail.com","sentAt":"2025-09-06T14:16:12Z","receivedAt":"2025-09-06T14:18:50Z","isPatch":false,"sender":{"key":"five231003@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75560439?v=4"},"body":"Hello everyone,\nAt my $(DAYJOB), we have an NFS which stores different git repos.\nDue to how git stores objects, we have started to run out of inodes on\nthe NFS as the number of repos coming into the NFS increased.\n\nThese git repos come from another service and there are typically\nthousands of them each day. It is important to note that we only store\nthe .git dir and expose a url which is configured as the remote by\ndefault to read and write into this repo.\n\nAll of these are small repos; usually not many files and not many\ncommits too - I'd say ~5 commits on average.\n\nHistorically, when we ran out of inodes, we had implemented a few\nstrategies where we used to repack the objects or archive the older\nrepos and move them into another store and bring them back into this\nNFS and unarchive the repo.\n\nHowever, none of these totally mitigated the issue and we still run\ninto issue as the traffic increases. As a last resort,  we increased\nthe disk size even though there was ton of free space left - just\nfor increasing the number of inodes.\n\nWe can't delete any of these repos, no matter how old, because they are\nvaluable data.\n\nI was wondering if there was some other strategy that we could implement\nhere as this seems like a problem that people might often run into. It\nwould really help to here your thoughts or if you could point me to\nanywhere else.\n\nThanks\n"},{"id":"525693","messageId":"03d101dc1f42$e5380a70$afa81f50$@nexbridge.com","threadId":"64100","inReplyTo":"20250906141711.64419-1-five231003@gmail.com","subject":"RE: Running out of inodes on an NFS which stores repos","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2025-09-06T15:28:22Z","receivedAt":"2025-09-06T15:28:35Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 6, 2025 10:16 AM, Kousik Sanagavarapu wrote:\n>Hello everyone,\n>At my $(DAYJOB), we have an NFS which stores different git repos.\n>Due to how git stores objects, we have started to run out of inodes on the\nNFS as\n>the number of repos coming into the NFS increased.\n>\n>These git repos come from another service and there are typically thousands\nof\n>them each day. It is important to note that we only store the .git dir and\nexpose a url\n>which is configured as the remote by default to read and write into this\nrepo.\n>\n>All of these are small repos; usually not many files and not many commits\ntoo - I'd\n>say ~5 commits on average.\n>\n>Historically, when we ran out of inodes, we had implemented a few\nstrategies\n>where we used to repack the objects or archive the older repos and move\nthem into\n>another store and bring them back into this NFS and unarchive the repo.\n>\n>However, none of these totally mitigated the issue and we still run into\nissue as the\n>traffic increases. As a last resort,  we increased the disk size even\nthough there was\n>ton of free space left - just for increasing the number of inodes.\n>\n>We can't delete any of these repos, no matter how old, because they are\nvaluable\n>data.\n>\n>I was wondering if there was some other strategy that we could implement\nhere as\n>this seems like a problem that people might often run into. It would really\nhelp to\n>here your thoughts or if you could point me to anywhere else.\n\nI would suggest running\n\ngit gc --aggressive\n\non your repos. This might help compress your pack files. I have seen\ncustomers\nwith thousands of pack files who have never run a garbage collection.\n\nAnother thing you might want to try is to use sparse-checkout to only keep\nthe\ndirectories you absolutely need if that is an option. Also, check your /tmp\nand\nlost+found directories. \n\n"},{"id":"525694","messageId":"aLxUkTzuVaZrWDs2@fruit.crustytoothpaste.net","threadId":"64100","inReplyTo":"20250906141711.64419-1-five231003@gmail.com","subject":"Re: Running out of inodes on an NFS which stores repos","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-09-06T15:34:41Z","receivedAt":"2025-09-06T15:34:49Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-09-06 at 14:16:12, Kousik Sanagavarapu wrote:\n> Hello everyone,\n\nHi,\n\n> These git repos come from another service and there are typically\n> thousands of them each day. It is important to note that we only store\n> the .git dir and expose a url which is configured as the remote by\n> default to read and write into this repo.\n> \n> All of these are small repos; usually not many files and not many\n> commits too - I'd say ~5 commits on average.\n> \n> Historically, when we ran out of inodes, we had implemented a few\n> strategies where we used to repack the objects or archive the older\n> repos and move them into another store and bring them back into this\n> NFS and unarchive the repo.\n> \n> However, none of these totally mitigated the issue and we still run\n> into issue as the traffic increases. As a last resort,  we increased\n> the disk size even though there was ton of free space left - just\n> for increasing the number of inodes.\n> \n> We can't delete any of these repos, no matter how old, because they are\n> valuable data.\n> \n> I was wondering if there was some other strategy that we could implement\n> here as this seems like a problem that people might often run into. It\n> would really help to here your thoughts or if you could point me to\n> anywhere else.\n\nThere are a couple things that come to mind here.  You can try to set\n`fetch.unpackLimit` to 1, which will cause of the objects pushed into\nthe repository to end up in a pack.  That means you'll usually have\nonly two files, the pack and index, rather than the loose objects.\n\nIf you have a large number of references, you may wish to convert the\nrepositories to use the reftable backend instead of the files backend\n(via `git refs migrate --ref-format=reftable`), which will also tend to\nuse fewer files on disk.  Note that this requires a relatively new Git,\nso if you need to access these repositories with an older Git version,\ndon't do this.\n\nYou can also periodically repack more frequently if you set\n`gc.autoPackLimit` to a smaller number (in conjunction with\n`fetch.unpackLimit` above).  If you have repositories that are not\npacked at all, running `git gc` (or, if you don't want to remove any\nobjects, `git repack -d --cruft`), which will likely reduce the number\nof loose objects and result in more objects being packed.\n\nFinally, it may be useful to you to reformat the underlying file system\nin a way that has more inodes.  I know ext4 supports a larger inode\nratio for repositories with many small files.  Alternatively, apparently\nbtrfs does not have a fixed inode ratio, so that may be helpful to avoid\nrunning out of inodes.  I can't speak to non-Linux file systems, though.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"525780","messageId":"DCN87S14V9G8.3BAV5XX1BDHKM@gmail.com","threadId":"64100","inReplyTo":"aLxUkTzuVaZrWDs2@fruit.crustytoothpaste.net","subject":"Re: Running out of inodes on an NFS which stores repos","fromName":"Kousik Sanagavarapu","fromEmail":"five231003@gmail.com","sentAt":"2025-09-08T07:05:08Z","receivedAt":"2025-09-08T07:05:12Z","isPatch":false,"sender":{"key":"five231003@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75560439?v=4"},"body":"On Sat Sep 6, 2025 at 9:04 PM IST, brian m. carlson wrote:\n> On 2025-09-06 at 14:16:12, Kousik Sanagavarapu wrote:\n>> Hello everyone,\n>\n> Hi,\n>\n>> These git repos come from another service and there are typically\n>> thousands of them each day. It is important to note that we only store\n>> the .git dir and expose a url which is configured as the remote by\n>> default to read and write into this repo.\n>>\n>> All of these are small repos; usually not many files and not many\n>> commits too - I'd say ~5 commits on average.\n>>\n>> Historically, when we ran out of inodes, we had implemented a few\n>> strategies where we used to repack the objects or archive the older\n>> repos and move them into another store and bring them back into this\n>> NFS and unarchive the repo.\n>>\n>> However, none of these totally mitigated the issue and we still run\n>> into issue as the traffic increases. As a last resort,  we increased\n>> the disk size even though there was ton of free space left - just\n>> for increasing the number of inodes.\n>>\n>> We can't delete any of these repos, no matter how old, because they are\n>> valuable data.\n>>\n>> I was wondering if there was some other strategy that we could implement\n>> here as this seems like a problem that people might often run into. It\n>> would really help to here your thoughts or if you could point me to\n>> anywhere else.\n>\n> There are a couple things that come to mind here.  You can try to set\n> `fetch.unpackLimit` to 1, which will cause of the objects pushed into\n> the repository to end up in a pack.  That means you'll usually have\n> only two files, the pack and index, rather than the loose objects.\n\nThanks for this, I have tried this out and while going through the\nsurrounding documentation, found `transfer.unpackLimit`. This was exactly\nwhat I was looking for.\n\n> If you have a large number of references, you may wish to convert the\n> repositories to use the reftable backend instead of the files backend\n> (via `git refs migrate --ref-format=reftable`), which will also tend to\n> use fewer files on disk.  Note that this requires a relatively new Git,\n> so if you need to access these repositories with an older Git version,\n> don't do this.\n>\n> You can also periodically repack more frequently if you set\n> `gc.autoPackLimit` to a smaller number (in conjunction with\n> `fetch.unpackLimit` above).  If you have repositories that are not\n> packed at all, running `git gc` (or, if you don't want to remove any\n> objects, `git repack -d --cruft`), which will likely reduce the number\n> of loose objects and result in more objects being packed.\n\nYes, I have now set the following config surrounding gc\n\n\t[receive]\n\t\tautogc = true\n\t[gc]\n\t\tauto = 1\n\t\tautopacklimit = 1\n\nCurious to know if this will have any noticable performance impact\nthough. As I mentioned in my previous msg, these are small repos but the\nnumber of repos being created and the operations performed on them are\nlarge - mostly pushes,\n\n> Finally, it may be useful to you to reformat the underlying file system\n> in a way that has more inodes.  I know ext4 supports a larger inode\n> ratio for repositories with many small files.  Alternatively, apparently\n> btrfs does not have a fixed inode ratio, so that may be helpful to avoid\n> running out of inodes.  I can't speak to non-Linux file systems, though.\n\nUnfourtunately, I can't reformat the NFS. It is currently on ext4 and\neven though there are quite a few filesystems which don't impose a\nthreshold on inodes, I can't migrate to them.\n"},{"id":"525888","messageId":"aL904XGUmXmnyXGl@fruit.crustytoothpaste.net","threadId":"64100","inReplyTo":"DCN87S14V9G8.3BAV5XX1BDHKM@gmail.com","subject":"Re: Running out of inodes on an NFS which stores repos","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-09-09T00:29:21Z","receivedAt":"2025-09-09T00:29:23Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-09-08 at 07:05:08, Kousik Sanagavarapu wrote:\n> Yes, I have now set the following config surrounding gc\n> \n> \t[receive]\n> \t\tautogc = true\n> \t[gc]\n> \t\tauto = 1\n> \t\tautopacklimit = 1\n> \n> Curious to know if this will have any noticable performance impact\n> though. As I mentioned in my previous msg, these are small repos but the\n> number of repos being created and the operations performed on them are\n> large - mostly pushes,\n\nThe `transfer.unpackLimit` will not have any impact; it's in use at at\nleast some major forges.  Packed objects can use things like bitmaps and\nother functionality, which forges like for performance.\n\nThe gc settings you have will cause everything to repacked after every\npush, and repacking data can be quite expensive.  At work, we repack\nafter about every 40 pushes or so.  You may wish to use a different\nvalue.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"525915","messageId":"DCO3KXDFLAOW.TJHMTZOP7LQG@gmail.com","threadId":"64100","inReplyTo":"aL904XGUmXmnyXGl@fruit.crustytoothpaste.net","subject":"Re: Running out of inodes on an NFS which stores repos","fromName":"Kousik Sanagavarapu","fromEmail":"five231003@gmail.com","sentAt":"2025-09-09T07:39:53Z","receivedAt":"2025-09-09T07:39:56Z","isPatch":false,"sender":{"key":"five231003@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75560439?v=4"},"body":"On Tue Sep 9, 2025 at 5:59 AM IST, brian m. carlson wrote:\n> On 2025-09-08 at 07:05:08, Kousik Sanagavarapu wrote:\n>> Yes, I have now set the following config surrounding gc\n>>\n>> \t[receive]\n>> \t\tautogc = true\n>> \t[gc]\n>> \t\tauto = 1\n>> \t\tautopacklimit = 1\n>>\n>> Curious to know if this will have any noticable performance impact\n>> though. As I mentioned in my previous msg, these are small repos but the\n>> number of repos being created and the operations performed on them are\n>> large - mostly pushes,\n>\n> The `transfer.unpackLimit` will not have any impact; it's in use at at\n> least some major forges.  Packed objects can use things like bitmaps and\n> other functionality, which forges like for performance.\n\nOh, got it.\n\n> The gc settings you have will cause everything to repacked after every\n> push, and repacking data can be quite expensive.  At work, we repack\n> after about every 40 pushes or so.  You may wish to use a different\n> value.\n\nGot it, thanks for the info. I will try with a higher value and see how\nit goes.\n"}]}