{"thread":{"id":"48273","subject":"File versioning based on shallow Git repositories?","startedAt":"2018-04-12T18:12:42Z","lastAt":"2018-04-13T21:58:16Z","messageCount":9,"participants":["Hallvard Breien Furuseth","Ævar Arnfjörð Bjarmason","Rafael Ascensao","Jakub Narebski","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"344552","messageId":"hbf.20180412fvfi@bombur.uio.no","threadId":"48273","inReplyTo":null,"subject":"File versioning based on shallow Git repositories?","fromName":"Hallvard Breien Furuseth","fromEmail":"h.b.furuseth@usit.uio.no","sentAt":"2018-04-12T18:01:15Z","receivedAt":"2018-04-12T18:12:42Z","isPatch":false,"sender":{"key":"h.b.furuseth@usit.uio.no","avatar":null},"body":"Can I use a shallow Git repo for file versioning, and regularly purge\nhistory older than e.g. 2 weeks? Purged data MUST NOT be recoverable.\n\nOr is there a backup tool based on shallow Git cloning which does this?\nPush/pull to another shallow repo would be nice but is not required.\nThe files are text files up to 1/4 Gb, usually with few changes. \n\n\nIf using Git - I see \"git fetch --depth\" can shorten history now.\nHow do I do that without 'fetch', in the origin repo?\nAlso Documentation/technical/shallow.txt describes some caveats, I'm\nnot sure how relevant they are.\n\nTo purge old data -\n git config core.logallrefupdates false\n git gc --prune=now --aggressive\nAnything else?\n\nI'm guessing that without --aggressive, some expired info might be\ndeduced from studying the packing of the remaining objects. Don't\nknow if we'll be required to be that paranoid.\n\n-- \nHallvard\n"},{"id":"344554","messageId":"87d0z4b6ti.fsf@evledraar.gmail.com","threadId":"48273","inReplyTo":"hbf.20180412fvfi@bombur.uio.no","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-04-12T18:47:21Z","receivedAt":"2018-04-12T18:47:30Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Apr 12 2018, Hallvard Breien Furuseth wrote:\n\n> Can I use a shallow Git repo for file versioning, and regularly purge\n> history older than e.g. 2 weeks? Purged data MUST NOT be recoverable.\n>\n> Or is there a backup tool based on shallow Git cloning which does this?\n> Push/pull to another shallow repo would be nice but is not required.\n> The files are text files up to 1/4 Gb, usually with few changes.\n>\n>\n> If using Git - I see \"git fetch --depth\" can shorten history now.\n> How do I do that without 'fetch', in the origin repo?\n> Also Documentation/technical/shallow.txt describes some caveats, I'm\n> not sure how relevant they are.\n>\n> To purge old data -\n> git config core.logallrefupdates false\n> git gc --prune=now --aggressive\n> Anything else?\n>\n> I'm guessing that without --aggressive, some expired info might be\n> deduced from studying the packing of the remaining objects. Don't\n> know if we'll be required to be that paranoid.\n\nThe shallow feature is not for this use-case, but there's a much easier\nsolution that I've used for exactly this use-case, e.g. taking backups\nof SQL dumps that delta-compress well, and then throwing out old\nbackups.\n\nYou:\n\n1. Create a backup.git repo\n2. Each time you make a backup, checkout a new orphan branch, see \"git\n checkout --orphan\"\n3. You copy the files over, commit them, \"git log\" at this point shows\n one commit no matter if you've done this before.\n4. You create a tag for this backup, e.g. one named after the current\n time, delete the branch.\n5. You then have a retention period for the tags, e.g. only keep the\n last 30 tags if you do daily backups for 30 days of backups.\n\nThen as soon as you delete the tags the old commit will be unreferenced,\nand you can make git-gc delete the data.\n\nYou'll still be able to `git diff` between tags, even though they have\nunrelated histories, and the files will still delta-compress.\n"},{"id":"344556","messageId":"4af21bcd-7a68-50df-4cce-0b050ccaeb90@usit.uio.no","threadId":"48273","inReplyTo":"87d0z4b6ti.fsf@evledraar.gmail.com","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Hallvard Breien Furuseth","fromEmail":"h.b.furuseth@usit.uio.no","sentAt":"2018-04-12T19:36:40Z","receivedAt":"2018-04-12T19:36:47Z","isPatch":false,"sender":{"key":"h.b.furuseth@usit.uio.no","avatar":null},"body":"On 12. april 2018 20:47, Ævar Arnfjörð Bjarmason wrote:\n> 1. Create a backup.git repo\n> 2. Each time you make a backup, checkout a new orphan branch, see \"git\n> checkout --orphan\"\n> 3. You copy the files over, commit them, \"git log\" at this point shows\n> one commit no matter if you've done this before.\n> 4. You create a tag for this backup, e.g. one named after the current\n> time, delete the branch.\n> 5. You then have a retention period for the tags, e.g. only keep the\n> last 30 tags if you do daily backups for 30 days of backups.\n> \n> Then as soon as you delete the tags the old commit will be unreferenced,\n> and you can make git-gc delete the data.\n\nNice!\nWhy the tags though, instead of branches named after the current time?\n\nOne --orphan branch/tag per day with several commits would work for me.\n\nAlso maybe it'll be worthwhile to generate .git/info/grafts in a local\nclone of the repo to get back easily visible history. No grafts in\nthe original repo, grafts mess things up.\n\n-- \nHallvard\n"},{"id":"344557","messageId":"87k1tcf905.fsf@evledraar.gmail.com","threadId":"48273","inReplyTo":"4af21bcd-7a68-50df-4cce-0b050ccaeb90@usit.uio.no","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-04-12T20:46:34Z","receivedAt":"2018-04-12T20:46:44Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Apr 12 2018, Hallvard Breien Furuseth wrote:\n\n> On 12. april 2018 20:47, Ævar Arnfjörð Bjarmason wrote:\n>> 1. Create a backup.git repo\n>> 2. Each time you make a backup, checkout a new orphan branch, see \"git\n>> checkout --orphan\"\n>> 3. You copy the files over, commit them, \"git log\" at this point shows\n>> one commit no matter if you've done this before.\n>> 4. You create a tag for this backup, e.g. one named after the current\n>> time, delete the branch.\n>> 5. You then have a retention period for the tags, e.g. only keep the\n>> last 30 tags if you do daily backups for 30 days of backups.\n>>\n>> Then as soon as you delete the tags the old commit will be unreferenced,\n>> and you can make git-gc delete the data.\n>\n> Nice!\n> Why the tags though, instead of branches named after the current time?\n\nBecause tags are idiomatic in git for a reference that doesn't change,\nbut sure, if you'd like branches that'll work too.\n\n> One --orphan branch/tag per day with several commits would work for me.\n>\n> Also maybe it'll be worthwhile to generate .git/info/grafts in a local\n> clone of the repo to get back easily visible history. No grafts in\n> the original repo, grafts mess things up.\n\nMaybe, I have not tried this with grafts.\n"},{"id":"344558","messageId":"CACUQV592km3SaHiY9uZon4E3jhakmYmzwcejmsnExzaybNm3xw@mail.gmail.com","threadId":"48273","inReplyTo":"87k1tcf905.fsf@evledraar.gmail.com","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Rafael Ascensao","fromEmail":"rafa.almas@gmail.com","sentAt":"2018-04-12T21:07:08Z","receivedAt":"2018-04-12T21:07:54Z","isPatch":false,"sender":{"key":"rafa.almas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1923789?v=4"},"body":"Would initiating a repo with a empty root commit, tag it with 'base' then\n\nuse $ git rebase --onto base master@{30 days ago} master;\n\nbe viable?\n\nThe --orphan & tag is perhaps more robust, since it's \"harder\" to move\ntags around.\n\n--\nRafael Ascensão\n"},{"id":"344560","messageId":"2e700b4e-d52c-afc5-dacd-51405fb51c21@usit.uio.no","threadId":"48273","inReplyTo":"CACUQV592km3SaHiY9uZon4E3jhakmYmzwcejmsnExzaybNm3xw@mail.gmail.com","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Hallvard Breien Furuseth","fromEmail":"h.b.furuseth@usit.uio.no","sentAt":"2018-04-12T21:22:55Z","receivedAt":"2018-04-12T21:23:01Z","isPatch":false,"sender":{"key":"h.b.furuseth@usit.uio.no","avatar":null},"body":"On 12. april 2018 23:07, Rafael Ascensao wrote:\n> Would initiating a repo with a empty root commit, tag it with 'base' then\n> use $ git rebase --onto base master@{30 days ago} master;\n> be viable?\n\nNo... my question was confused from the beginning. With such large files\nI _shouldn't_ have history (or grafts), otherwise Git spends a lot of CPU\ntime creating diffs when I look at a commit, or worse, when I try git log.\nWhich I discovered quickly when trying real data instead of test-data:-)\n\nÆvar's suggestion was exactly right in that respect. Thanks again!\n\n-- \nHallvard\n"},{"id":"344594","messageId":"86efjjmqsi.fsf@gmail.com","threadId":"48273","inReplyTo":"4af21bcd-7a68-50df-4cce-0b050ccaeb90@usit.uio.no","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-13T08:52:45Z","receivedAt":"2018-04-13T08:53:03Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hallvard Breien Furuseth writes:\n\n> Also maybe it'll be worthwhile to generate .git/info/grafts in a local\n> clone of the repo to get back easily visible history. No grafts in\n> the original repo, grafts mess things up.\n\nJust a reminder: modern Git has \"git replace\", a modern and safe\nalternative to the grafts file.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"344601","messageId":"nycvar.QRO.7.76.6.1804131157360.65@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48273","inReplyTo":"86efjjmqsi.fsf@gmail.com","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-04-13T11:12:34Z","receivedAt":"2018-04-13T11:12:57Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Kuba,\n\nOn Fri, 13 Apr 2018, Jakub Narebski wrote:\n\n> Hallvard Breien Furuseth writes:\n> \n> > Also maybe it'll be worthwhile to generate .git/info/grafts in a local\n> > clone of the repo to get back easily visible history. No grafts in\n> > the original repo, grafts mess things up.\n> \n> Just a reminder: modern Git has \"git replace\", a modern and safe\n> alternative to the grafts file.\n\nRight!\n\nMaybe it is time to start deprecating grafts? They *do* cause problems,\nsuch as weird \"missing objects\" problems when trying to fetch into, or\npush from, a repository with grafts. These problems are not shared by the\n`git replace` method.\n\nI just sent out a patch to add a deprecation warning.\n\nCiao,\nDscho\n"},{"id":"344650","messageId":"861sfin50f.fsf@gmail.com","threadId":"48273","inReplyTo":"nycvar.QRO.7.76.6.1804131157360.65@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: File versioning based on shallow Git repositories?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-13T21:57:52Z","receivedAt":"2018-04-13T21:58:16Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hello Johannes,\n\nJohannes Schindelin writes:\n> On Fri, 13 Apr 2018, Jakub Narebski wrote:\n>> Hallvard Breien Furuseth writes:\n>> \n>>> Also maybe it'll be worthwhile to generate .git/info/grafts in a local\n>>> clone of the repo to get back easily visible history. No grafts in\n>>> the original repo, grafts mess things up.\n>> \n>> Just a reminder: modern Git has \"git replace\", a modern and safe\n>> alternative to the grafts file.\n>\n> Right!\n>\n> Maybe it is time to start deprecating grafts? They *do* cause problems,\n> such as weird \"missing objects\" problems when trying to fetch into, or\n> push from, a repository with grafts. These problems are not shared by the\n> `git replace` method.\n\nAlso you can propagate \"git replace\" info with clone / fetch / push.\n\n> I just sent out a patch to add a deprecation warning.\n\nThank you for this.\n\n-- \nJakub Narębski\n"}]}