{"thread":{"id":"38247","subject":"Cygwin sparse checkout degrades performance","startedAt":"2014-12-24T18:30:42Z","lastAt":"2014-12-24T19:40:58Z","messageCount":2,"participants":["Brian Ericson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"254083","messageId":"549B0652.3020605@ptc.com","threadId":"38247","inReplyTo":null,"subject":"Cygwin sparse checkout degrades performance","fromName":"Brian Ericson","fromEmail":"bericson@ptc.com","sentAt":"2014-12-24T18:30:42Z","receivedAt":"2014-12-24T18:30:42Z","isPatch":false,"sender":{"key":"bericson@ptc.com","avatar":"https://gravatar.com/avatar/8ee326fa7a162bc30cb2b3f4f454773d63c864ca6d4095af9312e9dd856587cd?d=mp&s=160"},"body":"Counter-intuitively, using sparse checkout in Cygwin degrades \"status\" \ntimes as status appears to \"stat\" non-existent files and directories.\n\nTo demonstrate, I created a repo with 100k random files in a \ndir/dir/dir/file structure (on a linux box -- to do this in Cygwin \nrequires piping the result of \"openssl rand\" to \"dos2unix\" as the output \ncontains \"\\r\") and cloned in a Cygwin shell:\n\ngit init test\ncd test\ngit commit --allow-empty -m 'Empty first commit'\nfor i in {1..10}; do for j in {1..10000}; do file=$( openssl rand -hex \n32 | sed 's,^\\(.\\)\\(.\\)\\(.\\),\\1/\\2/\\3/,'); mkdir -p $( dirname $file ); \necho $file > $file ; done & done; wait\ngit add .\ngit commit -m '100000 files'\ngit gc --prune=now --aggressive\n\nI then timed and plotted \"git status\" as sparse checkout step-wisely \nreduced the number of files in the working tree using the folllowing \ncommand:\n\n( ( git status >& /dev/null; time -p git status > /dev/null ) |& sed -n \n'/real/{s/real/100000/p}'; git config core.sparseCheckout true; for i in \n$( seq 90000 -10000 10000 ) 1; do git ls-files | head -n $i | sed \n's,^,/,' > .git/info/sparse-checkout; git read-tree -u -m HEAD; git \nstatus >& /dev/null; ( time -p git status > /dev/null ) |& sed -n \n\"/real/{s/real/$i/p}\"; done; echo '*' > .git/info/sparse-checkout; git \nread-tree -u -m HEAD; rm .git/info/sparse-checkout; git config --unset \ncore.sparseCheckout ) | gnuplot -p -e \"set terminal dumb; set xrange[] \nreverse; set style data dots; set nokey; plot '-' using 1:2\"\n\nVertical bar is time in seconds, horizontal the number of files in the \nworking tree after the sparse checkout.\n\nLinux results (v2.1.0):\n   0.45 \n.+-----+------+-----+------+------+------+------+-----+------+-----++\n        +      +      +     +      +      +      +      + +      +      +\n| |\n    0.4 ++ ++\n| |\n   0.35 ++ ++\n| |\n| |\n    0.3 ++ .                                                           ++\n| |\n        | .                                                     |\n   0.25 ++                  . .                                       ++\n        |                                 . .                          |\n| |\n    0.2 ++                                              . .            ++\n| |\n   0.15 ++                                                           . +.\n| |\n        +      +      +     +      +      +      +      + +      +      +\n    0.1 \n++-----+------+-----+------+------+------+------+-----+------+-----++\n      100000 90000  80000 70000  60000  50000  40000  30000 20000 10000    0\n\nCygwin results (v2.1.1):\n   10 \n++-----+------+------+------+------+------+------+------+------+-----++\n      +      +      +      +      +      +      +      + +      +      +\n| .\n    9 ++ ++\n| .      |\n| |\n    8 ++ .            ++\n| |\n| |\n    7 ++ .                   ++\n| |\n| |\n      |                                  . .                           |\n    6 ++ ++\n| |\n| |\n    5 ++ .                                                             ++\n      .                    . .                                         |\n      +      +      .      +      +      +      +      + +      +      +\n    4 \n++-----+------+------+------+------+------+------+------+------+-----++\n    100000 90000  80000  70000  60000  50000  40000  30000  20000 10000    0\n\nLinux times do what I expect/want (they get better as the number of \nworking tree files decrease), but Cygwin does the opposite: the worst \ntimes are in a working tree with only 1 (sparse) file, and it's double \nwhere I started with no sparse checkout!  I'd hoped sparse checkout \nwould improve the too-slow status times when all files are present...\n\nLooking at strace with a working tree consisting of a single (sparse) \nfile suggests Cygwin is attempting to access the non-existent files and \ndirectories whereas Linux does not appear to do so.  In fact, if I do \nnothing more than \"mkdir -p $( git ls-files | cut -c1-5 | sort -u )\" \nwhen looking at a single (sparse) file, I can drop status times below \n3s, a 3-fold improvement and something at least better than where I started!\n\nIs there a way I can get improved status times using sparse checkout \nwith Cygwin?\n"},{"id":"254084","messageId":"549B16CA.3040107@ptc.com","threadId":"38247","inReplyTo":"549B0652.3020605@ptc.com","subject":"Re: Cygwin sparse checkout degrades performance","fromName":"Brian Ericson","fromEmail":"bericson@ptc.com","sentAt":"2014-12-24T19:40:58Z","receivedAt":"2014-12-24T19:40:58Z","isPatch":false,"sender":{"key":"bericson@ptc.com","avatar":"https://gravatar.com/avatar/8ee326fa7a162bc30cb2b3f4f454773d63c864ca6d4095af9312e9dd856587cd?d=mp&s=160"},"body":"Huh.  The graphs (somehow) ended up incoherently reformatted...  Sorry \nabout that!\n\nHere's the raw data after a second run:\n\nLinux:\n100000 0.49\n  90000 0.27\n  80000 0.27\n  70000 0.28\n  60000 0.23\n  50000 0.21\n  40000 0.21\n  30000 0.19\n  20000 0.19\n  10000 0.16\n      1 0.14\n\nCygwin:\n100000 4.72\n  90000 4.28\n  80000 4.41\n  70000 4.43\n  60000 4.67\n  50000 5.04\n  40000 6.24\n  30000 7.28\n  20000 7.88\n  10000 8.96\n      1 9.43\n\n\n\nOn 12/24/2014 12:30 PM, Brian Ericson wrote:\n> Counter-intuitively, using sparse checkout in Cygwin degrades \"status\"\n> times as status appears to \"stat\" non-existent files and directories.\n>\n> To demonstrate, I created a repo with 100k random files in a\n> dir/dir/dir/file structure (on a linux box -- to do this in Cygwin\n> requires piping the result of \"openssl rand\" to \"dos2unix\" as the output\n> contains \"\\r\") and cloned in a Cygwin shell:\n>\n> git init test\n> cd test\n> git commit --allow-empty -m 'Empty first commit'\n> for i in {1..10}; do for j in {1..10000}; do file=$( openssl rand -hex\n> 32 | sed 's,^\\(.\\)\\(.\\)\\(.\\),\\1/\\2/\\3/,'); mkdir -p $( dirname $file );\n> echo $file > $file ; done & done; wait\n> git add .\n> git commit -m '100000 files'\n> git gc --prune=now --aggressive\n>\n> I then timed and plotted \"git status\" as sparse checkout step-wisely\n> reduced the number of files in the working tree using the folllowing\n> command:\n>\n> ( ( git status >& /dev/null; time -p git status > /dev/null ) |& sed -n\n> '/real/{s/real/100000/p}'; git config core.sparseCheckout true; for i in\n> $( seq 90000 -10000 10000 ) 1; do git ls-files | head -n $i | sed\n> 's,^,/,' > .git/info/sparse-checkout; git read-tree -u -m HEAD; git\n> status >& /dev/null; ( time -p git status > /dev/null ) |& sed -n\n> \"/real/{s/real/$i/p}\"; done; echo '*' > .git/info/sparse-checkout; git\n> read-tree -u -m HEAD; rm .git/info/sparse-checkout; git config --unset\n> core.sparseCheckout ) | gnuplot -p -e \"set terminal dumb; set xrange[]\n> reverse; set style data dots; set nokey; plot '-' using 1:2\"\n>\n> Vertical bar is time in seconds, horizontal the number of files in the\n> working tree after the sparse checkout.\n>\n> Linux results (v2.1.0):\n>    0.45\n> .+-----+------+-----+------+------+------+------+-----+------+-----++\n>         +      +      +     +      +      +      +      + +      +      +\n> | |\n>     0.4 ++ ++\n> | |\n>    0.35 ++ ++\n> | |\n> | |\n>     0.3 ++ .                                                           ++\n> | |\n>         | .                                                     |\n>    0.25 ++                  . .                                       ++\n>         |                                 . .                          |\n> | |\n>     0.2 ++                                              . .            ++\n> | |\n>    0.15 ++                                                           . +.\n> | |\n>         +      +      +     +      +      +      +      + +      +      +\n>     0.1\n> ++-----+------+-----+------+------+------+------+-----+------+-----++\n>       100000 90000  80000 70000  60000  50000  40000  30000 20000\n> 10000    0\n>\n> Cygwin results (v2.1.1):\n>    10\n> ++-----+------+------+------+------+------+------+------+------+-----++\n>       +      +      +      +      +      +      +      + +      +      +\n> | .\n>     9 ++ ++\n> | .      |\n> | |\n>     8 ++ .            ++\n> | |\n> | |\n>     7 ++ .                   ++\n> | |\n> | |\n>       |                                  . .                           |\n>     6 ++ ++\n> | |\n> | |\n>     5 ++ .                                                             ++\n>       .                    . .                                         |\n>       +      +      .      +      +      +      +      + +      +      +\n>     4\n> ++-----+------+------+------+------+------+------+------+------+-----++\n>     100000 90000  80000  70000  60000  50000  40000  30000  20000\n> 10000    0\n>\n> Linux times do what I expect/want (they get better as the number of\n> working tree files decrease), but Cygwin does the opposite: the worst\n> times are in a working tree with only 1 (sparse) file, and it's double\n> where I started with no sparse checkout!  I'd hoped sparse checkout\n> would improve the too-slow status times when all files are present...\n>\n> Looking at strace with a working tree consisting of a single (sparse)\n> file suggests Cygwin is attempting to access the non-existent files and\n> directories whereas Linux does not appear to do so.  In fact, if I do\n> nothing more than \"mkdir -p $( git ls-files | cut -c1-5 | sort -u )\"\n> when looking at a single (sparse) file, I can drop status times below\n> 3s, a 3-fold improvement and something at least better than where I\n> started!\n>\n> Is there a way I can get improved status times using sparse checkout\n> with Cygwin?\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> .\n>\n"}]}