{"thread":{"id":"3925","subject":"using git on flash media","startedAt":"2006-04-19T23:31:25Z","lastAt":"2006-04-20T00:44:34Z","messageCount":3,"participants":["David Tweed","Linus Torvalds","Josh Boyer"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"18899","messageId":"20060419233125.89318.qmail@web86912.mail.ukl.yahoo.com","threadId":"3925","inReplyTo":null,"subject":"using git on flash media","fromName":"David Tweed","fromEmail":"tweed314@yahoo.co.uk","sentAt":"2006-04-19T23:31:25Z","receivedAt":"2006-04-19T23:31:25Z","isPatch":false,"sender":{"key":"tweed314@yahoo.co.uk","avatar":null},"body":"I was wondering if anyone here could answer two silly\nquestions arising from a lack of knowledge about the\nprecise architecture of git (unfortunately\nwebsearching \"using git flash\" turns up pages about\nmiserable people and Macromedia):\n\nIs it reasonable to use git reasonably intensively\nkeeping it's database on a flash media drive? (Ie, the\nflash drive is plugged into a standard desktop machine\nand the \"actual versions\" of the file that're being\nmanaged are on the machine's hard disc, but all of the\ngit repository data are on a flash media drive. What\nI'm basically checking is that it doesn't, I dunno,\nrewrite files so frequently that on a modern flash\ndrive it would wear out the entire drive unreasonably\nquickly.\n\nLikewise, supposing that there are several machines\nwith the same filesystem tree devoted to the\ngit-chronicled project. Supposing you've got a careful\nuser who checks out the latest tree from the flash\ndrive into the machine's hard disk upon sitting down\nat one of these machines, and commits to the flash\ndrive & properly umounts the flash before pulling it\nout. The user does this switching between several\nmachines randomly. (So the git repository on flash\nacts to ensure that whenever I sit down at a machine\nI'm presented with my latest versions of everything.)\nIs there any obvious problem that could come up which\ncould lead to git getting confused and somehow\ncorrupting the archive contents on the flash drive?\n(Not the kind of catastrophic loss thing that'd be\ncaught by taking regular backups but some corruption\nthat'd silently make parts of the repository\nun-checkoutable in the main flash repository.) I'm\ntrying to imagine some program somehow getting\nconfused and somehow writing some vital piece of\nrepository data to the \"current\" hard disc without\nrealising that means it's not \"always readable all the\ntime\", unlike the flash drive proper.\n\n(I know the answer probably ought to be \"just make all\nthe machines networked and communicate via the network\nrather than a flash drive\", but assume I'm not\namenable to changing.)\n\nMany thanks for any insight,\n\ncheers, david tweed\n\n\nSend instant messages to your online friends http://uk.messenger.yahoo.com \n"},{"id":"18903","messageId":"Pine.LNX.4.64.0604191651110.3701@g5.osdl.org","threadId":"3925","inReplyTo":"20060419233125.89318.qmail@web86912.mail.ukl.yahoo.com","subject":"Re: using git on flash media","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-04-20T00:23:09Z","receivedAt":"2006-04-20T00:23:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 20 Apr 2006, David Tweed wrote:\n>\n> What I'm basically checking is that it doesn't, I dunno, rewrite files \n> so frequently that on a modern flash drive it would wear out the entire \n> drive unreasonably quickly.\n\nThe largely write-once nature of git should mean that the only files that \nget rewritten a lot are\n - the directories get rewritten to, since git creates new objects at a \n   reasonable pace\n - the branch references get rewritten.\n\nIn general, I'd say that git probably does less writing than most other \nSCM's are likely to do.\n\nThat said, when you say \"modern flash drive\", I really suspect you \nshouldn't care deeply any more. Modern flash devices can be rewritten a \nlot more than old ones could (by an order of magnitude or more), and they \nalmost always have wear levelling in hw, making it even less of an issue \n(but if they don't, your biggest issue will be that you should use a \nfilesystem that does it for you).\n\nThat said, if you want to be safe, I think flash memory card vendors \nguarantee only up to 10,000 write cycles (and it used to be much less). \n\nThat's _complete_ rewrites, though, which is more than just a single \nsector write. They tend to guarantee 100,000 single-sector re-writes (ie \nmore like the \"directory update\" things when you create a new object).\n\nAnd assuming you'd count one commit as one \"total rewrite\" (which sounds \nunlikely - but it's certainly more than one sector - I don't know what \nthey consider a total rewrite when they make up their numbers), that \nimplies that to be really safe, you shouldn't do more than 10,000 commits \nbefore you replace your flash. Quite frankly, I suspect that's _way_ more \nconservative than you should be, but hey, since you asked..\n\n10,000 commits is actually a fair number. The kernel has gotten 25,000 in \na year, but the kernel is a pretty active and large project. I suspect \nthat 10,000 commits is quite a lot of years for most projects.\n\nOne rule: NEVER mount your flash with the \"sync\" option, and use \"noatime\" \nto avoid unnecessary inode access time updates (that's especially true for \ngit, where archive atimes aren't interesting, but it's usually a good idea \nfor flash in general). Otherwise you'll get normal accesses ending up \ndoign \"writes\" too and writes will do a lot more of them, and the above \n\"one commit = one rewrite\" rule-of-thumb is suddenly not at all \nconservative.\n\nBtw, backups are still good. Flash or no flash, and whether you're very \nconservative in your flash usage or not.\n\n\t\tLinus\n"},{"id":"18905","messageId":"625fc13d0604191744w1155a6c3hde184a9705669c4b@mail.gmail.com","threadId":"3925","inReplyTo":"Pine.LNX.4.64.0604191651110.3701@g5.osdl.org","subject":"Re: using git on flash media","fromName":"Josh Boyer","fromEmail":"jwboyer@gmail.com","sentAt":"2006-04-20T00:44:34Z","receivedAt":"2006-04-20T00:44:34Z","isPatch":false,"sender":{"key":"jwboyer@gmail.com","avatar":null},"body":"On 4/19/06, Linus Torvalds <torvalds@osdl.org> wrote:\n>\n>\n> On Thu, 20 Apr 2006, David Tweed wrote:\n> >\n> > What I'm basically checking is that it doesn't, I dunno, rewrite files\n> > so frequently that on a modern flash drive it would wear out the entire\n> > drive unreasonably quickly.\n>\n> The largely write-once nature of git should mean that the only files that\n> get rewritten a lot are\n>  - the directories get rewritten to, since git creates new objects at a\n>    reasonable pace\n>  - the branch references get rewritten.\n>\n> In general, I'd say that git probably does less writing than most other\n> SCM's are likely to do.\n>\n> That said, when you say \"modern flash drive\", I really suspect you\n> shouldn't care deeply any more. Modern flash devices can be rewritten a\n> lot more than old ones could (by an order of magnitude or more), and they\n> almost always have wear levelling in hw, making it even less of an issue\n> (but if they don't, your biggest issue will be that you should use a\n> filesystem that does it for you).\n\nFinding one that claims to have HW wear leveling is still hard, so you\ncan never really tell.\n\n> That said, if you want to be safe, I think flash memory card vendors\n> guarantee only up to 10,000 write cycles (and it used to be much less).\n>\n> That's _complete_ rewrites, though, which is more than just a single\n> sector write. They tend to guarantee 100,000 single-sector re-writes (ie\n> more like the \"directory update\" things when you create a new object).\n\nWhen talking about flash, it's the erase cycles that are counted. \nMost NOR chips guarantee 100,000 erase cycles per eraseblock.  NAND\nchips, which are what most flash drives use, can vary between 10,000 -\n100,000 erases depending on the technology used.\n\n> And assuming you'd count one commit as one \"total rewrite\" (which sounds\n> unlikely - but it's certainly more than one sector - I don't know what\n> they consider a total rewrite when they make up their numbers), that\n\nA \"total rewrite\" is simply something that causes an eraseblock to be\nerased.  That can vary alot, but it essentially boils down to needing\nto write new data to a location in an eraseblock that has already been\nwritten to.\n\n> implies that to be really safe, you shouldn't do more than 10,000 commits\n> before you replace your flash. Quite frankly, I suspect that's _way_ more\n> conservative than you should be, but hey, since you asked..\n\nTo be really conservative, just don't do it ;).  Most flash drives\ndon't allow access to the raw flash chips, so while you can use\nsomething like JFFS2 on them still, the benefits of that are somewhat\nlost since you never really know what is going on underneath.\n\n> One rule: NEVER mount your flash with the \"sync\" option, and use \"noatime\"\n> to avoid unnecessary inode access time updates (that's especially true for\n\nRight.\n\njosh\n"}]}