{"thread":{"id":"29585","subject":"Git, Builds, and Filesystem Type","startedAt":"2012-02-09T21:23:18Z","lastAt":"2012-02-10T04:04:13Z","messageCount":8,"participants":["Hilco Wijbenga","Martin Fick"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"184270","messageId":"CAE1pOi1of-hj+87M7RqhFUWA8an14bPG88dAOwhNogmfFvJ=tA@mail.gmail.com","threadId":"29585","inReplyTo":null,"subject":"Git, Builds, and Filesystem Type","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-02-09T21:23:18Z","receivedAt":"2012-02-09T21:23:18Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"Hi all,\n\nI'm thinking about trying out different filesystems over the weekend\nto see if, say, BTRFS or XFS is faster when using Git and running our\nbuild.\n\nCurrently, I'm using ReiserFS and it's not like it's not working. I'm\nvery pleased with ReiserFS but after seeing talks about BTRFS and XFS\nI'm curious if another (newer) FS is better suited to our specific\nenvironment. Anything to make the build a little faster. :-)\n\nFor the record, our (Java) project is quite small. It's 43MB (source\nand images) and the entire directory tree after building is about\n1.6GB (this includes all JARs downloaded by Maven). So we're not\ntalking TBs of data.\n\nAny thoughts on which FSs to include in my tests? Or simply which FS\nmight be more appropriate?\n\nCheers,\nHilco\n"},{"id":"184274","messageId":"201202091453.38564.mfick@codeaurora.org","threadId":"29585","inReplyTo":"CAE1pOi1of-hj+87M7RqhFUWA8an14bPG88dAOwhNogmfFvJ=tA@mail.gmail.com","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2012-02-09T21:53:38Z","receivedAt":"2012-02-09T21:53:38Z","isPatch":false,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"On Thursday, February 09, 2012 02:23:18 pm Hilco Wijbenga \nwrote:\n> For the record, our (Java) project is quite small. It's\n> 43MB (source and images) and the entire directory tree\n> after building is about 1.6GB (this includes all JARs\n> downloaded by Maven). So we're not talking TBs of data.\n> \n> Any thoughts on which FSs to include in my tests? Or\n> simply which FS might be more appropriate?\n\ntmpfs is probably fastest hands down if you can use it (even \nif you have to back it by swap).\n\n-Martin\n\n-- \nEmployee of Qualcomm Innovation Center, Inc. which is a \nmember of Code Aurora Forum\n"},{"id":"184285","messageId":"CAE1pOi387-bimYEG4bjFOjaCwhPeDyLRj7wOJgyuKSCrZ9kBFg@mail.gmail.com","threadId":"29585","inReplyTo":"201202091453.38564.mfick@codeaurora.org","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-02-09T23:24:47Z","receivedAt":"2012-02-09T23:24:47Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 9 February 2012 13:53, Martin Fick wrote:\n> On Thursday, February 09, 2012 02:23:18 pm Hilco Wijbenga\n> wrote:\n>> For the record, our (Java) project is quite small. It's\n>> 43MB (source and images) and the entire directory tree\n>> after building is about 1.6GB (this includes all JARs\n>> downloaded by Maven). So we're not talking TBs of data.\n>>\n>> Any thoughts on which FSs to include in my tests? Or\n>> simply which FS might be more appropriate?\n>\n> tmpfs is probably fastest hands down if you can use it (even\n> if you have to back it by swap).\n\nI don't have quite that much RAM. :-)\n"},{"id":"184287","messageId":"201202091634.36563.mfick@codeaurora.org","threadId":"29585","inReplyTo":"CAE1pOi387-bimYEG4bjFOjaCwhPeDyLRj7wOJgyuKSCrZ9kBFg@mail.gmail.com","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2012-02-09T23:34:36Z","receivedAt":"2012-02-09T23:34:36Z","isPatch":false,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"On Thursday, February 09, 2012 04:24:47 pm Hilco Wijbenga \nwrote:\n> On 9 February 2012 13:53, Martin Fick \n wrote:\n> > On Thursday, February 09, 2012 02:23:18 pm Hilco\n> > Wijbenga\n> > \n> > wrote:\n> >> For the record, our (Java) project is quite small.\n> >> It's 43MB (source and images) and the entire\n> >> directory tree after building is about 1.6GB (this\n> >> includes all JARs downloaded by Maven). So we're not\n> >> talking TBs of data.\n> >> \n> >> Any thoughts on which FSs to include in my tests? Or\n> >> simply which FS might be more appropriate?\n> > \n> > tmpfs is probably fastest hands down if you can use it\n> > (even if you have to back it by swap).\n> \n> I don't have quite that much RAM. :-)\n\nBut I am sure that you have that much disk space which you \ncan allocate to swap, if not you already couldn't build it. \nAnd tmpfs swapping is still likely faster than a persistent \nFS (it will not need to block on syncs). If you are \nbenchmarking, it is likely worth you effort since that will \nprobably mark the upper performance bound,\n\n-Martin\n\n-- \nEmployee of Qualcomm Innovation Center, Inc. which is a \nmember of Code Aurora Forum\n"},{"id":"184305","messageId":"CAE1pOi0=Sf7K=FaHW--z6AHSTKsmBWe-f75gXcdZivW_4T6g4Q@mail.gmail.com","threadId":"29585","inReplyTo":"CAE5ih7_NkyJ6vGbyoKvQy65LFK3-zkXi79Xd6+3Si8DyUi47JQ@mail.gmail.com","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-02-10T01:11:12Z","receivedAt":"2012-02-10T01:11:12Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 9 February 2012 15:24, Luke Diamand wrote:\n> On Thu, Feb 9, 2012 at 9:23 PM, Hilco Wijbenga \n> wrote:\n>>\n>> Hi all,\n>>\n>> I'm thinking about trying out different filesystems over the weekend\n>> to see if, say, BTRFS or XFS is faster when using Git and running our\n>> build.\n>>\n>> Currently, I'm using ReiserFS and it's not like it's not working. I'm\n>> very pleased with ReiserFS but after seeing talks about BTRFS and XFS\n>> I'm curious if another (newer) FS is better suited to our specific\n>> environment. Anything to make the build a little faster. :-)\n>>\n>> For the record, our (Java) project is quite small. It's 43MB (source\n>> and images) and the entire directory tree after building is about\n>> 1.6GB (this includes all JARs downloaded by Maven). So we're not\n>> talking TBs of data.\n>>\n>> Any thoughts on which FSs to include in my tests? Or simply which FS\n>> might be more appropriate?\n>\n>\n> Do people still use reiserfs? I thought development on that pretty much\n> stopped years ago. And reiser4 never made it into the kernel. Read the wiki\n> page for why.\n\nAs I said, reiserfs works fine so I see no need to replace it. I'm not\na big fan of ext3 (I've run out of inodes too many times) and I simply\nhaven't tried ext4. Apparently, it has some architectural problems but\nI'm no expert.\n\n> ext4, FTW!\n>\n> But whatever you use, you might find that the core.preloadindex config\n> option helps. It certainly does for NFS.\n\nI would like to think that my local hard drive has no latency issues?\nWould this really be worthwhile even if I do not use some sort of\ndistributed FS?\n"},{"id":"184306","messageId":"CAE1pOi1O10XeROv+sQRwAAVQ0PneMZTOaEfny-Oz2Sp+=z+aiA@mail.gmail.com","threadId":"29585","inReplyTo":"201202091634.36563.mfick@codeaurora.org","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-02-10T01:25:59Z","receivedAt":"2012-02-10T01:25:59Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 9 February 2012 15:34, Martin Fick wrote:\n> On Thursday, February 09, 2012 04:24:47 pm Hilco Wijbenga\n> wrote:\n>> On 9 February 2012 13:53, Martin Fick\n> wrote:\n>> > On Thursday, February 09, 2012 02:23:18 pm Hilco\n>> > Wijbenga\n>> >\n>> > wrote:\n>> >> For the record, our (Java) project is quite small.\n>> >> It's 43MB (source and images) and the entire\n>> >> directory tree after building is about 1.6GB (this\n>> >> includes all JARs downloaded by Maven). So we're not\n>> >> talking TBs of data.\n>> >>\n>> >> Any thoughts on which FSs to include in my tests? Or\n>> >> simply which FS might be more appropriate?\n>> >\n>> > tmpfs is probably fastest hands down if you can use it\n>> > (even if you have to back it by swap).\n>>\n>> I don't have quite that much RAM. :-)\n>\n> But I am sure that you have that much disk space which you\n> can allocate to swap, if not you already couldn't build it.\n> And tmpfs swapping is still likely faster than a persistent\n> FS (it will not need to block on syncs).  If you are\n> benchmarking, it is likely worth you effort since that will\n> probably mark the upper performance bound,\n\nI found [1]. Is that sort of what you had in mind? That would be quite\ntricky. I have a group of some 60 projects, each with their own\n\"target\" directory which would have to be mounted on tmpfs. And the\n\"target\" directory is created by Maven, not by me. Not to mention that\nI shut down my computer at the end of the day. :-)\n\nI think I would prefer a somewhat more persistent solution. I\ncertainly have enough space for a very big swap partition. So the\nwhole of ~/my-project would fit on tmpfs. I'm just not sure about\nmaking it persistent at the end of the day.\n\n[1] http://code.google.com/p/chromium/wiki/LinuxFasterBuilds\n"},{"id":"184308","messageId":"14de0389-6497-4e74-baa5-129b0c1560a3@email.android.com","threadId":"29585","inReplyTo":"CAE1pOi1O10XeROv+sQRwAAVQ0PneMZTOaEfny-Oz2Sp+=z+aiA@mail.gmail.com","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2012-02-10T02:08:45Z","receivedAt":"2012-02-10T02:08:45Z","isPatch":false,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"\n\n>I found [1]. Is that sort of what you had in mind\n\nyes\n\n>That would be quite\n>tricky. I have a group of some 60 projects, each with their own\n>\"target\" directory which would have to be mounted on tmpfs. And the\n>\"target\" directory is created by Maven, not by me. Not to mention that\n>I shut down my computer at the end of the day. :-)\n\nSounds like a laptop? Hibernate?\n\n>I think I would prefer a somewhat more persistent solution. I\n>certainly have enough space for a very big swap partition. So the\n>whole of ~/my-project would fit on tmpfs. I'm just not sure about\n>making it persistent at the end of the day.\n\n\nLink your .git dir to a persistent location. \n\nBenchmark it first so you know how much hassle it may be worth. If it isn't worth it, xfs or brtfs will likely not be worth it either,\n\n-Martin\nEmployee of Qualcomm Innovation Center,Inc. which is a member of Code Aurora Forum\n"},{"id":"184320","messageId":"CAE1pOi3Qk2qESza3QpwxbediYTOQxZ8WvpcfiE7FpkUZ9CwUjA@mail.gmail.com","threadId":"29585","inReplyTo":"14de0389-6497-4e74-baa5-129b0c1560a3@email.android.com","subject":"Re: Git, Builds, and Filesystem Type","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-02-10T04:04:13Z","receivedAt":"2012-02-10T04:04:13Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 9 February 2012 18:08, Martin Fick wrote:\n>>That would be quite\n>>tricky. I have a group of some 60 projects, each with their own\n>>\"target\" directory which would have to be mounted on tmpfs. And the\n>>\"target\" directory is created by Maven, not by me. Not to mention that\n>>I shut down my computer at the end of the day. :-)\n>\n> Sounds like a laptop?  Hibernate?\n\nNo, a regular desktop that I shut down every night. It makes quite a\nbit of noise and I would rather not waste power.\n\n>>I think I would prefer a somewhat more persistent solution. I\n>>certainly have enough space for a very big swap partition. So the\n>>whole of ~/my-project would fit on tmpfs. I'm just not sure about\n>>making it persistent at the end of the day.\n>\n> Link your .git dir to a persistent location.\n\nI don't think that would help all that much. I would still lose\nanything I had not committed or stashed. And I would still have to\nrebuild every day.\n\nPerhaps something together with unionfs? I'll think about it.\n"}]}