{"thread":{"id":"62669","subject":"git gc does not clean tmp_pack* files","startedAt":"2024-12-19T02:19:24Z","lastAt":"2024-12-28T20:13:47Z","messageCount":8,"participants":["Boomman","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"509321","messageId":"CADMKT2-vsUUi-23YGexvxbW9wj0Br+cQcv34eStnw-0EDRiKMw@mail.gmail.com","threadId":"62669","inReplyTo":null,"subject":"git gc does not clean tmp_pack* files","fromName":"Boomman","fromEmail":"boomman37@gmail.com","sentAt":"2024-12-19T02:19:06Z","receivedAt":"2024-12-19T02:19:24Z","isPatch":false,"sender":{"key":"boomman37@gmail.com","avatar":null},"body":"Hello,\n\nI'm currently hitting an issue trying to garbage collect a git repo in\ncase of low disk space.\nAfter running \"git gc\" a bunch of times I tried to clean up more and\nmore space on the disk not realizing that when \"git gc\" fails it just\nleaves the tmp_pack file there.\n\nD:\\Platform>git gc\n...blah...\nfatal: sha1 file '.git/objects/pack/tmp_pack_FG1inp' write error. Out\nof diskspace\nfatal: failed to run repack\n\nD:\\Platform>git gc\n...blah...\nfatal: sha1 file '.git/objects/pack/tmp_pack_IFvamY' write error. Out\nof diskspace\nfatal: failed to run repack\n\nD:\\Platform>git gc\n...blah...\nfatal: sha1 file '.git/objects/pack/tmp_pack_khHCC9' write error. Out\nof diskspace\nfatal: failed to run repack\n\nD:\\Platform>dir .git\\objects\\pack\\tmp*\n Directory of D:\\Platform\\.git\\objects\\pack\n\n12/18/2024 05:33 PM 7,367,032,832 tmp_pack_FG1inp\n12/18/2024 05:35 PM 3,787,194,368 tmp_pack_IFvamY\n12/18/2024 05:39 PM 7,713,062,912 tmp_pack_khHCC9\n09/11/2024 11:33 AM 3,068,002,304 tmp_pack_XTVFUi\n 4 File(s) 21,935,292,416 bytes\n 0 Dir(s) 339,968 bytes free\n\nI believe that before trying to write *anything* to disk \"git gc\"\nshould try to take exclusive handles on these and wipe them, ideally\nby default. The total size of these tmp* files is multiple times\nlarger than the repo I'm trying to compact, so if the command just did\nthis pre-cleaning I'd not have hit this problem once I cleaned enough\ndisk space.\n\nPlease let me know your thoughts on this.\n\n-Vitaly\n"},{"id":"509323","messageId":"20241219054222.GA2310556@coredump.intra.peff.net","threadId":"62669","inReplyTo":"CADMKT2-vsUUi-23YGexvxbW9wj0Br+cQcv34eStnw-0EDRiKMw@mail.gmail.com","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-19T05:42:22Z","receivedAt":"2024-12-19T05:42:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 18, 2024 at 06:19:06PM -0800, Boomman wrote:\n\n> D:\\Platform>dir .git\\objects\\pack\\tmp*\n> Directory of D:\\Platform\\.git\\objects\\pack\n> \n> 12/18/2024 05:33 PM 7,367,032,832 tmp_pack_FG1inp\n> 12/18/2024 05:35 PM 3,787,194,368 tmp_pack_IFvamY\n> 12/18/2024 05:39 PM 7,713,062,912 tmp_pack_khHCC9\n> 09/11/2024 11:33 AM 3,068,002,304 tmp_pack_XTVFUi\n> 4 File(s) 21,935,292,416 bytes\n> 0 Dir(s) 339,968 bytes free\n> \n> I believe that before trying to write *anything* to disk \"git gc\"\n> should try to take exclusive handles on these and wipe them, ideally\n> by default. The total size of these tmp* files is multiple times\n> larger than the repo I'm trying to compact, so if the command just did\n> this pre-cleaning I'd not have hit this problem once I cleaned enough\n> disk space.\n\ngit-gc does know how to clean up these files, but they are subject to\nthe same mtime grace period that loose objects are. This is to avoid\ndeleting a file that is being actively used by a simultaneous process.\n\nTry \"git gc --prune=now\" if you know there are no other active processes\nin the repository.\n\nWe usually prune things after finishing the repack. So if you're running\nout of disk space to repack, there might be a chicken-and-egg problem.\nYou can run \"git prune\" manually in that case.\n\nPossibly git-gc should prune first for this reason, but I'd be hesitant\nto do so for actual loose objects. It's a little weird that tempfile\ncleanup is lumped in with loose object cleanup, and is mostly\nhistorical. Possibly those should be split.\n\n-Peff\n"},{"id":"509329","messageId":"CADMKT28cZcdW8UJ59vZHoMa0WEbyAccx4A10-jk3MiBkhZLxEg@mail.gmail.com","threadId":"62669","inReplyTo":"20241219054222.GA2310556@coredump.intra.peff.net","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Boomman","fromEmail":"boomman37@gmail.com","sentAt":"2024-12-19T08:26:15Z","receivedAt":"2024-12-19T08:26:32Z","isPatch":false,"sender":{"key":"boomman37@gmail.com","avatar":null},"body":"Yes, if the behavior in case of running out of disk space is to just\nleave the malformed file there, it stands to reason that cleaning up\nthose malformed files should be the first operation to do for gc.\nAt the very least, git should notify the user that they've got all of\nthose tmp_pack files totaling 20+ GB in the object folder before it\nwill declare that it can't write a single byte into a lock file\nbecause previous \"git gc\" calls exhausted all the disk space.\nI know that on Windows it's possible to take an exclusive write lock\non a file while the process is running, so at least on Windows those\ntmp_pack files could be \"soft-try\" cleaned up without affecting other\nrunning git processes, not sure if it's possible for other supported\nOSes.\nhttps://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex\n\n-Vitaly\n\n\n>\n> On Wed, Dec 18, 2024 at 06:19:06PM -0800, Boomman wrote:\n>\n> > D:\\Platform>dir .git\\objects\\pack\\tmp*\n> > Directory of D:\\Platform\\.git\\objects\\pack\n> >\n> > 12/18/2024 05:33 PM 7,367,032,832 tmp_pack_FG1inp\n> > 12/18/2024 05:35 PM 3,787,194,368 tmp_pack_IFvamY\n> > 12/18/2024 05:39 PM 7,713,062,912 tmp_pack_khHCC9\n> > 09/11/2024 11:33 AM 3,068,002,304 tmp_pack_XTVFUi\n> > 4 File(s) 21,935,292,416 bytes\n> > 0 Dir(s) 339,968 bytes free\n> >\n> > I believe that before trying to write *anything* to disk \"git gc\"\n> > should try to take exclusive handles on these and wipe them, ideally\n> > by default. The total size of these tmp* files is multiple times\n> > larger than the repo I'm trying to compact, so if the command just did\n> > this pre-cleaning I'd not have hit this problem once I cleaned enough\n> > disk space.\n>\n> git-gc does know how to clean up these files, but they are subject to\n> the same mtime grace period that loose objects are. This is to avoid\n> deleting a file that is being actively used by a simultaneous process.\n>\n> Try \"git gc --prune=now\" if you know there are no other active processes\n> in the repository.\n>\n> We usually prune things after finishing the repack. So if you're running\n> out of disk space to repack, there might be a chicken-and-egg problem.\n> You can run \"git prune\" manually in that case.\n>\n> Possibly git-gc should prune first for this reason, but I'd be hesitant\n> to do so for actual loose objects. It's a little weird that tempfile\n> cleanup is lumped in with loose object cleanup, and is mostly\n> historical. Possibly those should be split.\n>\n> -Peff\n"},{"id":"509331","messageId":"xmqqmsgrncr6.fsf@gitster.g","threadId":"62669","inReplyTo":"CADMKT28cZcdW8UJ59vZHoMa0WEbyAccx4A10-jk3MiBkhZLxEg@mail.gmail.com","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-19T11:17:01Z","receivedAt":"2024-12-19T11:17:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Boomman writes:\n\n> Yes, if the behavior in case of running out of disk space is to just\n> leave the malformed file there, it stands to reason that cleaning up\n> those malformed files should be the first operation to do for gc.\n\nIt is misleading to call them malformed, isn't it? When a Git\nprocess creates a packfile (or loose object file for that matter),\nthey are written under these tmp_* names. When the processes die\nwithout finalizing these (either removing or renaming into their\nfinal names), they are left behind, and it would be better if we can\nremove it _before_ another process wants to consume more disk space.\n\nBut the issue is how you tell which one of these \"malformed\" files\nare still being written and will be finalized, and which ones are\nleftover ones. You want to remove the latter without molesting the\nformer. And you want to do so in a portable way, possibly even\nacross the network file systems.\n\nI guess, as Peff alluded to, we could do at the beginning of \"gc\" to\nprune only these \"possibly in progress, possibly leftover\" files\nthat are too old, repack the clearly \"finalized\" ones, and then\nprune again, this time including the \"finalized\" ones that are no\nlonger in use, which would help the creation of new packfiles from\nbeing blocked by these leftover files that are hoarding disk quota.\n"},{"id":"509368","messageId":"20241220090533.GC133148@coredump.intra.peff.net","threadId":"62669","inReplyTo":"xmqqmsgrncr6.fsf@gitster.g","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-20T09:05:33Z","receivedAt":"2024-12-20T09:05:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 19, 2024 at 03:17:01AM -0800, Junio C Hamano wrote:\n\n> Boomman writes:\n> \n> > Yes, if the behavior in case of running out of disk space is to just\n> > leave the malformed file there, it stands to reason that cleaning up\n> > those malformed files should be the first operation to do for gc.\n> \n> It is misleading to call them malformed, isn't it? When a Git\n> process creates a packfile (or loose object file for that matter),\n> they are written under these tmp_* names. When the processes die\n> without finalizing these (either removing or renaming into their\n> final names), they are left behind, and it would be better if we can\n> remove it _before_ another process wants to consume more disk space.\n\nWe usually automatically clean up our tempfiles if we encounter an\nerror, but don't do so for partially written packs. I think this is\nmostly historical, though occasionally it can be useful for debugging\n(e.g., indexing a pack coming over the network).\n\nIt might make sense to register them as tempfiles in the usual way,\npossibly with an environment variable option to ask for them to be kept\n(for debugging).\n\nThat's not foolproof, since a process can die without cleaning up after\nitself (e.g., on system crash). But it would mean that a repeatedly\nfailing \"git repack -ad\" does not fill up the disk. And the decision of\nwhen to clean up tempfiles in git-gc is less important.\n\n> But the issue is how you tell which one of these \"malformed\" files\n> are still being written and will be finalized, and which ones are\n> leftover ones. You want to remove the latter without molesting the\n> former. And you want to do so in a portable way, possibly even\n> across the network file systems.\n\nYeah, I think there are two issues being discussed in this thread:\n\n - when to clean up leftover tempfiles\n\n - how to decide which tempfiles are leftover\n\nThe second one is what the OP mentioned for locking. But not only does\nthat have portability questions, I'm not sure it is sufficient. Would we\never write tmp_pack_*, complete our process, and then expect our caller\nto do something with it (meaning there's a race where no process is\nholding the lock)?\n\nI'm not sure. We definitely write \"tmp\" packfiles via pack-objects and\nexpect git-repack to move them to their final names. I think we use a\nslightly different name (\"tmp--pack-*\"), but arguably we should\nconsider cleaning up stale versions of those, too.\n\n-Peff\n"},{"id":"509459","messageId":"CADMKT2_QonR_XbZ5297Df34FrrTpbJp04Qy5BGjPb_Ov=uFF4A@mail.gmail.com","threadId":"62669","inReplyTo":"20241220090533.GC133148@coredump.intra.peff.net","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Boomman","fromEmail":"boomman37@gmail.com","sentAt":"2024-12-21T01:17:50Z","receivedAt":"2024-12-21T01:18:07Z","isPatch":false,"sender":{"key":"boomman37@gmail.com","avatar":null},"body":"For me, two \"git gc\" on a same repo fail to run:\nfatal: gc is already running on machine 'WIN-blah' pid 40304 (use\n--force if not)\n\nIf you're already colliding on this, then I don't see why you can't\nuse a normal looking name without a randomized string like\n\"tmp_garbagecollecting\", so that each execution would at least\noverwrite the same location. In this case --force could append _1\nprobably.\n\n-Vitaly\n\n\nOn Fri, Dec 20, 2024 at 1:05 AM Jeff King wrote:\n>\n> On Thu, Dec 19, 2024 at 03:17:01AM -0800, Junio C Hamano wrote:\n>\n> > Boomman writes:\n> >\n> > > Yes, if the behavior in case of running out of disk space is to just\n> > > leave the malformed file there, it stands to reason that cleaning up\n> > > those malformed files should be the first operation to do for gc.\n> >\n> > It is misleading to call them malformed, isn't it? When a Git\n> > process creates a packfile (or loose object file for that matter),\n> > they are written under these tmp_* names. When the processes die\n> > without finalizing these (either removing or renaming into their\n> > final names), they are left behind, and it would be better if we can\n> > remove it _before_ another process wants to consume more disk space.\n>\n> We usually automatically clean up our tempfiles if we encounter an\n> error, but don't do so for partially written packs. I think this is\n> mostly historical, though occasionally it can be useful for debugging\n> (e.g., indexing a pack coming over the network).\n>\n> It might make sense to register them as tempfiles in the usual way,\n> possibly with an environment variable option to ask for them to be kept\n> (for debugging).\n>\n> That's not foolproof, since a process can die without cleaning up after\n> itself (e.g., on system crash). But it would mean that a repeatedly\n> failing \"git repack -ad\" does not fill up the disk. And the decision of\n> when to clean up tempfiles in git-gc is less important.\n>\n> > But the issue is how you tell which one of these \"malformed\" files\n> > are still being written and will be finalized, and which ones are\n> > leftover ones. You want to remove the latter without molesting the\n> > former. And you want to do so in a portable way, possibly even\n> > across the network file systems.\n>\n> Yeah, I think there are two issues being discussed in this thread:\n>\n> - when to clean up leftover tempfiles\n>\n> - how to decide which tempfiles are leftover\n>\n> The second one is what the OP mentioned for locking. But not only does\n> that have portability questions, I'm not sure it is sufficient. Would we\n> ever write tmp_pack_*, complete our process, and then expect our caller\n> to do something with it (meaning there's a race where no process is\n> holding the lock)?\n>\n> I'm not sure. We definitely write \"tmp\" packfiles via pack-objects and\n> expect git-repack to move them to their final names. I think we use a\n> slightly different name (\"tmp--pack-*\"), but arguably we should\n> consider cleaning up stale versions of those, too.\n>\n> -Peff\n"},{"id":"509699","messageId":"20241228194428.GB1535629@coredump.intra.peff.net","threadId":"62669","inReplyTo":"CADMKT2_QonR_XbZ5297Df34FrrTpbJp04Qy5BGjPb_Ov=uFF4A@mail.gmail.com","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-28T19:44:28Z","receivedAt":"2024-12-28T19:44:30Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 20, 2024 at 05:17:50PM -0800, Boomman wrote:\n\n> For me, two \"git gc\" on a same repo fail to run:\n> fatal: gc is already running on machine 'WIN-blah' pid 40304 (use\n> --force if not)\n> \n> If you're already colliding on this, then I don't see why you can't\n> use a normal looking name without a randomized string like\n> \"tmp_garbagecollecting\", so that each execution would at least\n> overwrite the same location. In this case --force could append _1\n> probably.\n\ngit-gc is not the only thing that writes packs. There might be\nsimultaneous packs written by incoming pushes or fetches, for example.\n\n-Peff\n"},{"id":"509701","messageId":"CADMKT2-kK=CNp4aeL2JciugQdNWed62VRmkc-CTaUNxoMxZNwQ@mail.gmail.com","threadId":"62669","inReplyTo":"20241228194428.GB1535629@coredump.intra.peff.net","subject":"Re: git gc does not clean tmp_pack* files","fromName":"Boomman","fromEmail":"boomman37@gmail.com","sentAt":"2024-12-28T20:13:29Z","receivedAt":"2024-12-28T20:13:47Z","isPatch":false,"sender":{"key":"boomman37@gmail.com","avatar":null},"body":"Right, but you know the *intent* of why each pack was created, right?\nThis is push, this is pull, this is gc. Clearly some of those are\nexpected to not run in parallel in normal scenarios, like gc. I\nimagine there are more: fetch from a single remote, push to a single\nremote? Why not make their packs names contain an operation identifier\nthat's supposed to be unique by default instead of a random string:\ntmp_fetch_myremote, tmp_push_myotherremote. This way you can reduce\nthe number of trash from failed operations, since the same location is\ngoing to be overwritten each call, plus the folder contents will be\nmuch more comprehensible at a glance.\n\n-Vitaly\n\nOn Sat, Dec 28, 2024 at 11:44 AM Jeff King wrote:\n>\n> On Fri, Dec 20, 2024 at 05:17:50PM -0800, Boomman wrote:\n>\n> > For me, two \"git gc\" on a same repo fail to run:\n> > fatal: gc is already running on machine 'WIN-blah' pid 40304 (use\n> > --force if not)\n> >\n> > If you're already colliding on this, then I don't see why you can't\n> > use a normal looking name without a randomized string like\n> > \"tmp_garbagecollecting\", so that each execution would at least\n> > overwrite the same location. In this case --force could append _1\n> > probably.\n>\n> git-gc is not the only thing that writes packs. There might be\n> simultaneous packs written by incoming pushes or fetches, for example.\n>\n> -Peff\n"}]}