{"thread":{"id":"16036","subject":"Performance impact of a large number of commits","startedAt":"2008-10-24T19:02:26Z","lastAt":"2008-10-25T15:12:33Z","messageCount":7,"participants":["Samuel Abels","david@lang.hm"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"93869","messageId":"1224874946.7566.13.camel@localhost","threadId":"16036","inReplyTo":null,"subject":"Performance impact of a large number of commits","fromName":"Samuel Abels","fromEmail":"newsgroups@debain.org","sentAt":"2008-10-24T19:02:26Z","receivedAt":"2008-10-24T19:02:26Z","isPatch":false,"sender":{"key":"newsgroups@debain.org","avatar":null},"body":"Hi,\n\nI am considering Git to maintain a repository of approximately 300.000\nfiles totaling 1 GB, with a number of ~100.000 commits per day, all in\none single branch. The only operations performed are \"git commit\", \"git\nshow\", and \"git checkout\", and all on one local machine. Does this sound\nlike a reasonable thing to do with Git?\n\n-Samuel\n"},{"id":"93871","messageId":"alpine.DEB.1.10.0810241240100.27333@asgard.lang.hm","threadId":"16036","inReplyTo":"1224874946.7566.13.camel@localhost","subject":"Re: Performance impact of a large number of commits","fromName":"","fromEmail":"david@lang.hm","sentAt":"2008-10-24T19:43:27Z","receivedAt":"2008-10-24T19:43:27Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 24 Oct 2008, Samuel Abels wrote:\n\n> Hi,\n>\n> I am considering Git to maintain a repository of approximately 300.000\n> files totaling 1 GB, with a number of ~100.000 commits per day, all in\n> one single branch. The only operations performed are \"git commit\", \"git\n> show\", and \"git checkout\", and all on one local machine. Does this sound\n> like a reasonable thing to do with Git?\n\n100,000 commits per day??\n\nthat's 1.5 commits/second. what is updating files that quickly?\n\nI suspect that you will have some issues here, but it's going to depend on \nhow many files get updated each 3/4 of a second.\n\nDavid Lang\n"},{"id":"93876","messageId":"1224878202.7566.25.camel@localhost","threadId":"16036","inReplyTo":"alpine.DEB.1.10.0810241240100.27333@asgard.lang.hm","subject":"Re: Performance impact of a large number of commits","fromName":"Samuel Abels","fromEmail":"newsgroups@debain.org","sentAt":"2008-10-24T19:56:42Z","receivedAt":"2008-10-24T19:56:42Z","isPatch":false,"sender":{"key":"newsgroups@debain.org","avatar":null},"body":"On Fri, 2008-10-24 at 12:43 -0700, david@lang.hm wrote:\n> 100,000 commits per day??\n> \n> that's 1.5 commits/second. what is updating files that quickly?\n\nThis is an automated process taking snapshots of rapidly changing files\ncontaining statistical data. Unfortunately, our needs go beyond what a\nversioning file system has to offer, and the data is an unstructured\ntext file (in other words, using a relational database is not a good\noption).\n\n> I suspect that you will have some issues here, but it's going to depend on \n> how many files get updated each 3/4 of a second.\n\nThat would be 5 to 10 changed files per commit, and those are passed to\ngit commit explicitly (i.e., walking the tree to stat files for finding\nchanges is not necessary).\n\n-Samuel\n"},{"id":"93878","messageId":"alpine.DEB.1.10.0810241301430.27333@asgard.lang.hm","threadId":"16036","inReplyTo":"1224878202.7566.25.camel@localhost","subject":"Re: Performance impact of a large number of commits","fromName":"","fromEmail":"david@lang.hm","sentAt":"2008-10-24T20:11:01Z","receivedAt":"2008-10-24T20:11:01Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 24 Oct 2008, Samuel Abels wrote:\n\n> On Fri, 2008-10-24 at 12:43 -0700, david@lang.hm wrote:\n>> 100,000 commits per day??\n>>\n>> that's 1.5 commits/second. what is updating files that quickly?\n>\n> This is an automated process taking snapshots of rapidly changing files\n> containing statistical data. Unfortunately, our needs go beyond what a\n> versioning file system has to offer, and the data is an unstructured\n> text file (in other words, using a relational database is not a good\n> option).\n>\n>> I suspect that you will have some issues here, but it's going to depend on\n>> how many files get updated each 3/4 of a second.\n>\n> That would be 5 to 10 changed files per commit, and those are passed to\n> git commit explicitly (i.e., walking the tree to stat files for finding\n> changes is not necessary).\n\nI suspect that your limits would be filesystem/OS limits more than git \nlimits\n\nat 5-10 files/commit you are going to be creating .5-1m files/day, even \nspread across 256 directories this is going to be a _lot_ of files.\n\npacking this may help (depending on how much the files change), but with \nthis many files the work of doing the packing would be expensive.\n\nDavid Lang\n"},{"id":"93909","messageId":"1224911915.7566.35.camel@localhost","threadId":"16036","inReplyTo":"alpine.DEB.1.10.0810241301430.27333@asgard.lang.hm","subject":"Re: Performance impact of a large number of commits","fromName":"Samuel Abels","fromEmail":"newsgroups@debain.org","sentAt":"2008-10-25T05:18:35Z","receivedAt":"2008-10-25T05:18:35Z","isPatch":false,"sender":{"key":"newsgroups@debain.org","avatar":null},"body":"On Fri, 2008-10-24 at 13:11 -0700, david@lang.hm wrote:\n> > git commit explicitly (i.e., walking the tree to stat files for finding\n> > changes is not necessary).\n> \n> I suspect that your limits would be filesystem/OS limits more than git \n> limits\n> \n> at 5-10 files/commit you are going to be creating .5-1m files/day, even \n> spread across 256 directories this is going to be a _lot_ of files.\n\nThe files are organized in a way that places no more than ~1.000 files\ninto each directory. Will Git create a directory containing a larger\nnumber of object files? I can see that this would be a problem in our\nuse case.\n\n> packing this may help (depending on how much the files change), but with \n> this many files the work of doing the packing would be expensive.\n\nWe can probably do that even if it takes several hours.\n\n-Samuel\n"},{"id":"93910","messageId":"alpine.DEB.1.10.0810242219440.20238@asgard.lang.hm","threadId":"16036","inReplyTo":"1224911915.7566.35.camel@localhost","subject":"Re: Performance impact of a large number of commits","fromName":"","fromEmail":"david@lang.hm","sentAt":"2008-10-25T05:29:58Z","receivedAt":"2008-10-25T05:29:58Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Sat, 25 Oct 2008, Samuel Abels wrote:\n\n> On Fri, 2008-10-24 at 13:11 -0700, david@lang.hm wrote:\n>>> git commit explicitly (i.e., walking the tree to stat files for finding\n>>> changes is not necessary).\n>>\n>> I suspect that your limits would be filesystem/OS limits more than git\n>> limits\n>>\n>> at 5-10 files/commit you are going to be creating .5-1m files/day, even\n>> spread across 256 directories this is going to be a _lot_ of files.\n>\n> The files are organized in a way that places no more than ~1.000 files\n> into each directory. Will Git create a directory containing a larger\n> number of object files? I can see that this would be a problem in our\n> use case.\n\nwhen git stores the copies of the files it does a sha1 hash of the file \ncontents and then stores the file in the directory\n.git/objects/<first two digits of the hash>/<hash>\nthis means that if you have files that have the same content they all fold \ntogeather, but with lots of files changing rapidly the result is a lot of \nfiles in these object directories.\n\nit would be a pretty minor change to git to have it use more directories \n(in fact, there's another thread going on today where people are looking \nat making this configurable, in that case to reduce the number of \ndirectories)\n\nthe other storage format that git has is the pack file. it takes a bunch \nof the objects, does some comparisons between them (to find duplicate bits \nof files), and then stores the result (base files plus deltas to re-create \nother files). the resulting compression is _extremely_ efficiant, and it \ncollapses many file objects into one pack file (addressing the issues of \nmany files in one directory)\n\n>> packing this may help (depending on how much the files change), but with\n>> this many files the work of doing the packing would be expensive.\n>\n> We can probably do that even if it takes several hours.\n\nmy concern is that spending time creating the pack files will mean that \nyou don't have time to insert the new files.\n\nthat being said, there may be other ways of dealing with this data rather \nthan putting it into files and then adding it to the git repository.\n\nGit has a fast-import streaming format that is designed for programs to \nuse that are converting repositories from other SCM systems. if you can \ntell more about what you are doing (how the data is being gathered, are \nthe files re-created for each commit, or are they being modified? if they \nare being modified is it appending data, changing some data, or randomly \nwriting throughout the file? etc) there may be some other options \navailable.\n\nat this point I don't know if git can work for you or not, but I'm pretty \nsure nothing else will have a chance with your size.\n\nDavid Lang\n"},{"id":"93929","messageId":"1224947553.7566.48.camel@localhost","threadId":"16036","inReplyTo":"alpine.DEB.1.10.0810242219440.20238@asgard.lang.hm","subject":"Re: Performance impact of a large number of commits","fromName":"Samuel Abels","fromEmail":"newsgroups@debain.org","sentAt":"2008-10-25T15:12:33Z","receivedAt":"2008-10-25T15:12:33Z","isPatch":false,"sender":{"key":"newsgroups@debain.org","avatar":null},"body":"On Fri, 2008-10-24 at 22:29 -0700, david@lang.hm wrote:\n> when git stores the copies of the files it does a sha1 hash of the file \n> contents and then stores the file in the directory\n> .git/objects/<first two digits of the hash>/<hash>\n\n> it would be a pretty minor change to git to have it use more directories \n\nAh, I see how this works. Well, I'll think of a way to cope with this (I\nmight patch my Git installation, or see how well it performs on an\nindexed file system). If all else fails we'll have to slash the number\nof commits even if this means that some files are not added to the\nhistory.\n\n> my concern is that spending time creating the pack files will mean that \n> you don't have time to insert the new files.\n> \n> that being said, there may be other ways of dealing with this data rather \n> than putting it into files and then adding it to the git repository.\n> \n> Git has a fast-import streaming format that is designed for programs to \n> use that are converting repositories from other SCM systems.\n\nI'm pretty sure that the streaming format won't do us much good, as the\nfiles are re-created from scratch between commits.\n\nThanks a lot for the information, this was very helpful.\n\n-Samuel\n"}]}