{"thread":{"id":"52830","subject":"\"Garbage collect\" old commits in git repository to free disk space","startedAt":"2020-02-18T00:36:33Z","lastAt":"2020-02-18T05:51:11Z","messageCount":2,"participants":["Tomas Mudrunka","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"391959","messageId":"8a4001f7b0e23e7df3172deeb32e0553@spoje.net","threadId":"52830","inReplyTo":null,"subject":"\"Garbage collect\" old commits in git repository to free disk space","fromName":"Tomas Mudrunka","fromEmail":"mudrunka@spoje.net","sentAt":"2020-02-18T00:29:24Z","receivedAt":"2020-02-18T00:36:33Z","isPatch":false,"sender":{"key":"mudrunka@spoje.net","avatar":null},"body":"Hello,\nis there safe way to garbage collect old commits from git repository? \nLets say that i want to always keep only last 100 commits and throw \neverything older away. To achieve similar goal as git clone --depth=100, \nbut on the server side. I had partial success with doing shallow clone \nand then converting to bare repo while removing the shallow flag from \n.git/config. But i didn't liked that solution and wasn't really sure \nwhat consequences in terms of data integrity and forward compatibility \nwith newer git versions might be.\n\nTo tell you more about my USE CASE:\n\nI want to create free opensource sofware similar to dropbox, but based \non git. My idea is following:\n\n1.) Automaticaly pull/commit/push changed files to/from several laptops \nto single git server (and forcefully resolve all conflicts, this will \nwork unless you plan to use it for software development)\n2.) On central server maintain tags indicating latest commits \nsynchronized to individual laptops.\n3.) On server delete old commits that are no longer needed by any laptop \nto sync their worktree. Once synced, delete these commits on laptops as \nwell. (optionaly leaving eg. 1 month or 1GB of old commits in case you \nmight need to rollback. Possibly keep the history only on the server, \nwhile deleting it from clients)\n\nThis way computers can stay in sync forever without running out of disk \nspace, because old commits are removed.\nEg. If i accidentaly add some very big file to synced folder and then \ndelete it, it will eventualy get deleted, once everybody gets in sync \nagain.\n\nI am aware that this is not something which git was designed for, but to \nme it seems like it should be more than doable. Do you think, any of you \ncan give me some hints on how to approach this problem please?\n\n\nThese are some projects which inspired me to explore this route:\n\nhttps://github.com/presslabs/gitfs\nhttps://www.syncany.org/\nhttps://www.cis.upenn.edu/~bcpierce/unison/\nhttps://etckeeper.branchable.com/\n\n-- \nS pozdravem\nBest regards\n      Tomáš Mudruňka - SPOJE.NET s.r.o.\n"},{"id":"391968","messageId":"20200218055108.GE1641086@coredump.intra.peff.net","threadId":"52830","inReplyTo":"8a4001f7b0e23e7df3172deeb32e0553@spoje.net","subject":"Re: \"Garbage collect\" old commits in git repository to free disk space","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-02-18T05:51:08Z","receivedAt":"2020-02-18T05:51:11Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 18, 2020 at 01:29:24AM +0100, Tomas Mudrunka wrote:\n\n> is there safe way to garbage collect old commits from git repository? Lets\n> say that i want to always keep only last 100 commits and throw everything\n> older away. To achieve similar goal as git clone --depth=100, but on the\n> server side. I had partial success with doing shallow clone and then\n> converting to bare repo while removing the shallow flag from .git/config.\n> But i didn't liked that solution and wasn't really sure what consequences in\n> terms of data integrity and forward compatibility with newer git versions\n> might be.\n\nI can't say for sure, but what you did with the shallow file is likely\nto bite you later. Shallow repositories are supposed to know where their\nboundary cutoffs are, and that information is stored in that file.\n\nThe normal answer here is that you'd want to rewrite the history using\ngrafts and git-filter-branch, or the new git-filter-repo. But...\n\n> To tell you more about my USE CASE:\n> \n> I want to create free opensource sofware similar to dropbox, but based on\n> git. My idea is following:\n\n...I think the rewrite would defeat the purpose, since you're relying on\nthe stability of the hashes to let all sides of the conversation figure\nout when they're in sync.\n\n> I am aware that this is not something which git was designed for, but to me\n> it seems like it should be more than doable. Do you think, any of you can\n> give me some hints on how to approach this problem please?\n\nI don't know that there's an easy way. Git is close to what you want,\nbut really is designed to assume the other side has all the reachable\nobjects.  Shallow clones are the feature that's closest to what you\nwant, but:\n\n  - I haven't had good experiences with repeatedly fetching into a\n    shallow clone. I believe the shallow list can grow because the\n    client doesn't realize which commits are reachable from others (and\n    hence are redundant). And I have seen shallow cuts crossing merge\n    boundaries cause a lot of extra objects to be transferred.\n\n  - It's really designed for _some_ repository to have all of the\n    objects. Fetching out of a shallow repository does work, I think,\n    but I would guess isn't very well exercised. So I have no idea what\n    kind of dragons you'd encounter.\n\nAnother option would be to periodically rewrite the history to a point\nthat you think all clients have synced, and then somehow communicate the\nrewrite to them (outside of Git, but it sounds like you'd have software\nwrapping Git). And then they could all do the identical rewrite and keep\ngoing.\n\nAlso look at git-annex if you haven't, which I think supports this kind\nof history truncation. I don't remember how it works exactly, but I\nthink that it may not put the blobs into Git at all, but rather just\npointers. So you're free to remove the actual data, but the pointer\nremains (and will later say \"sorry, I can't get that data for you\").\n\n-Peff\n"}]}