{"thread":{"id":"11168","subject":"Better value for chunk_size when threaded","startedAt":"2007-12-06T23:58:34Z","lastAt":"2007-12-10T13:46:09Z","messageCount":7,"participants":["Jon Smirl","Nicolas Pitre","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"62217","messageId":"9e4733910712061558k19fbc864ia1fb7a3431fd2603@mail.gmail.com","threadId":"11168","inReplyTo":null,"subject":"Better value for chunk_size when threaded","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-06T23:58:34Z","receivedAt":"2007-12-06T23:58:34Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"I tried some various ideas out for chunk_size and the best strategy I\nfound was to simply set it to a constant. How does 20,000 work on\nother CPUs?\n\nI'd turn on default threaded support with this change. With threads=1\nversus non-threaded there is no appreciable difference in the time.\n\nIs there an API to ask how many CPUs are in the system? It would be\nnice to default the number of threads equal to the number of CPUs and\nonly use pack.threads=X to override.\n\nMaking all of this work by default should help when outside people\ndecide to do a massive import.\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 4f44658..4d73be8 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -1645,7 +1645,7 @@ static void ll_find_deltas(struct object_entry\n**list, unsigned list_size,\n }\n\n /* this should be auto-tuned somehow */\n- chunk_size = window * 1000;\n+ chunk_size = 20000;\n\n do {\n unsigned sublist_size = chunk_size;\n\n\nwith chunk_size = 20000, everything is on a q6600 4GB\n\nthreads = 5\ntime git repack -a -d -f --depth=250 --window=250\nreal 6m20.123s\nuser 20m25.841s\nsys 0m5.520s\n\n\nthreads = 4\ntime git repack -a -d -f --depth=250 --window=250\nreal 6m15.525s\nuser 20m20.852s\nsys 0m5.356s\n\nthreads = 4\ntime git repack -a -d -f\nreal 1m31.537s\nuser 3m2.063s\nsys 0m3.064s\n\nthreads = 1\ntime git repack -a -d -f --depth=250 --window=250\nreal 18m46.005s\nuser 18m43.122s\nsys 0m1.228s\n\nthreads = 1\ntime git repack -a -d -f\nreal 2m57.774s\nuser 2m54.211s\nsys 0m1.228s\n\nNon-threaded\ntime git repack -a -d -f --depth=250 --window=250\nreal 18m51.183s\nuser 18m46.538s\nsys 0m1.604s\n\nNon-threaded\ntime git repack -a -d -f\nreal 2m54.849s\nuser 2m51.267s\nsys 0m1.412s\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62225","messageId":"alpine.LFD.0.99999.0712062014060.555@xanadu.home","threadId":"11168","inReplyTo":"9e4733910712061558k19fbc864ia1fb7a3431fd2603@mail.gmail.com","subject":"Re: Better value for chunk_size when threaded","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-07T01:27:44Z","receivedAt":"2007-12-07T01:27:44Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 6 Dec 2007, Jon Smirl wrote:\n\n> I tried some various ideas out for chunk_size and the best strategy I\n> found was to simply set it to a constant. How does 20,000 work on\n> other CPUs?\n\nThat depends on the object size. If you have a repo with big objects \nbut only 1000 of them for example, then the constant doesn't work.\n\nIdeally I'd opt for a value that tend towards around 5 seconds worth of \nwork per segment, or something like that. Maybe using the actual \nobjects size could be another way.\n\n> I'd turn on default threaded support with this change. With threads=1\n> versus non-threaded there is no appreciable difference in the time.\n\nWould need a way to determine pthreads availability from Makefile.\n\n> Is there an API to ask how many CPUs are in the system? It would be\n> nice to default the number of threads equal to the number of CPUs and\n> only use pack.threads=X to override.\n\nIf there is one besides futzing with /proc/cpuinfo I'd like to know \nabout it. Bonus points if it is portable.\n\n\nNicolas\n"},{"id":"62226","messageId":"9e4733910712061737o50a9a5f1ldccdf943bb19319f@mail.gmail.com","threadId":"11168","inReplyTo":"alpine.LFD.0.99999.0712062014060.555@xanadu.home","subject":"Re: Better value for chunk_size when threaded","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T01:37:07Z","receivedAt":"2007-12-07T01:37:07Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 12/6/07, Nicolas Pitre wrote:\n> On Thu, 6 Dec 2007, Jon Smirl wrote:\n>\n> > I tried some various ideas out for chunk_size and the best strategy I\n> > found was to simply set it to a constant. How does 20,000 work on\n> > other CPUs?\n>\n> That depends on the object size. If you have a repo with big objects\n> but only 1000 of them for example, then the constant doesn't work.\n\nHow about defaulting it to 20,000 and allowing an override? It's not\nfatal if we guess wrong, we just want to most common cases to work out\nof the box. 20,000 is definitely better than the current window *\n1000.\n\n> Ideally I'd opt for a value that tend towards around 5 seconds worth of\n> work per segment, or something like that. Maybe using the actual\n> objects size could be another way.\n>\n> > I'd turn on default threaded support with this change. With threads=1\n> > versus non-threaded there is no appreciable difference in the time.\n>\n> Would need a way to determine pthreads availability from Makefile.\n\nconfigure knows if pthreads is there.\n\n> > Is there an API to ask how many CPUs are in the system? It would be\n> > nice to default the number of threads equal to the number of CPUs and\n> > only use pack.threads=X to override.\n>\n> If there is one besides futzing with /proc/cpuinfo I'd like to know\n> about it. Bonus points if it is portable.\n>\n>\n> Nicolas\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62232","messageId":"alpine.LFD.0.99999.0712062139170.555@xanadu.home","threadId":"11168","inReplyTo":"9e4733910712061737o50a9a5f1ldccdf943bb19319f@mail.gmail.com","subject":"Re: Better value for chunk_size when threaded","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-07T03:25:01Z","receivedAt":"2007-12-07T03:25:01Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 6 Dec 2007, Jon Smirl wrote:\n\n> On 12/6/07, Nicolas Pitre wrote:\n> > On Thu, 6 Dec 2007, Jon Smirl wrote:\n> >\n> > > I tried some various ideas out for chunk_size and the best strategy I\n> > > found was to simply set it to a constant. How does 20,000 work on\n> > > other CPUs?\n> >\n> > That depends on the object size. If you have a repo with big objects\n> > but only 1000 of them for example, then the constant doesn't work.\n> \n> How about defaulting it to 20,000 and allowing an override? It's not\n> fatal if we guess wrong, we just want to most common cases to work out\n> of the box. 20,000 is definitely better than the current window *\n> 1000.\n\nSure.\n\n... But I think this can be made much better than that with no guessing \nat all.\n\nSay you have 4 threads. then let's divide the whole object list into 4 \nbig segments and feed those to each thread.\n\nOne thread will always finish before the others. The idea is to find \nthe active thread with the largest amount of remaining objects to \nprocess at that point, and steal half of them and give that to the \nthread that just finished. Repeat for each thread that completes its \nsegment until everything is done.\n\n\nNicolas\n"},{"id":"62262","messageId":"47590AEE.2080705@op5.se","threadId":"11168","inReplyTo":"9e4733910712061558k19fbc864ia1fb7a3431fd2603@mail.gmail.com","subject":"Re: Better value for chunk_size when threaded","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-12-07T08:57:18Z","receivedAt":"2007-12-07T08:57:18Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Jon Smirl wrote:\n> I tried some various ideas out for chunk_size and the best strategy I\n> found was to simply set it to a constant. How does 20,000 work on\n> other CPUs?\n> \n> I'd turn on default threaded support with this change. With threads=1\n> versus non-threaded there is no appreciable difference in the time.\n> \n> Is there an API to ask how many CPUs are in the system? It would be\n> nice to default the number of threads equal to the number of CPUs and\n> only use pack.threads=X to override.\n> \n\nI posted a patch to implement that just yesterday. It might need some\npolishing, but it hasn't received any comments so far.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"62560","messageId":"475D1473.5090809@op5.se","threadId":"11168","inReplyTo":"alpine.LFD.0.99999.0712062014060.555@xanadu.home","subject":"Re: Better value for chunk_size when threaded","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-12-10T10:26:59Z","receivedAt":"2007-12-10T10:26:59Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n> On Thu, 6 Dec 2007, Jon Smirl wrote:\n> \n>> I tried some various ideas out for chunk_size and the best strategy I\n>> found was to simply set it to a constant. How does 20,000 work on\n>> other CPUs?\n> \n> That depends on the object size. If you have a repo with big objects \n> but only 1000 of them for example, then the constant doesn't work.\n> \n> Ideally I'd opt for a value that tend towards around 5 seconds worth of \n> work per segment, or something like that. Maybe using the actual \n> objects size could be another way.\n> \n>> I'd turn on default threaded support with this change. With threads=1\n>> versus non-threaded there is no appreciable difference in the time.\n> \n> Would need a way to determine pthreads availability from Makefile.\n> \n>> Is there an API to ask how many CPUs are in the system? It would be\n>> nice to default the number of threads equal to the number of CPUs and\n>> only use pack.threads=X to override.\n> \n> If there is one besides futzing with /proc/cpuinfo I'd like to know \n> about it. Bonus points if it is portable.\n> \n\nHere is such a one. I've sent it before, using git-send-email, but that\none doesn't seem to work too well for all list-members, probably because\nmy own laptop appears to be the original SMTP-server and its name can't\nbe looked up. Sorry for inlining it here instead of sending it as a mail\non its own, but I have absolutely no idea how to get git-send-email to\ndo ldap authentication and connect to our tls-enabled smtp-server\nwithout using /usr/bin/sendmail and adding my laptop as originating\nsmtp-server.\n\nThis patch replaces the one I sent earlier and *should* work on\neverything from Irix and AIX to Linux, Windows and every other\nposixish system. It passes all tests, both with and without\nTHREADED_DELTA_SEARCH, and causes our weekly repack of our\nmother-ship repos to run roughly 4 times as fast (4 cores, no\nprevious thread config).\n\nExtract with\n sed -n -e /^##SEDMEHERE##/,/##TOHERE##/p -e /^##/d\n\n##SEDMEHERE##\nFrom ddf08303bd7962be385abbd5e964455a90ed6055 Mon Sep 17 00:00:00 2001\nFrom: Andreas Ericsson \nDate: Thu, 6 Dec 2007 22:09:27 +0100\nSubject: [PATCH] pack-objects: Add runtime detection of number of CPU's\n\nPacking objects can be done in parallell nowadays, but\nit's only done if the config option pack.threads is set\nto a value above 1. Because of that, the code-path used\nis sometimes not the most optimal one.\n\nThis patch adds a routine to detect the number of active\nCPU's at runtime, which should provide a better default\nand activate the (hopefully) better codepath more often.\n\nThe code is a rework of \"numcpus.c\", written by one\nPhilip Willoughby . numcpus.c is in\nthe public domain and can presently be downloaded from\nhttp://csgsoft.doc.ic.ac.uk/numcpus/\n\nSigned-off-by: Andreas Ericsson \n---\n builtin-pack-objects.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++-\n 1 files changed, 48 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 250dc56..ccf5198 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -17,6 +17,13 @@\n \n #ifdef THREADED_DELTA_SEARCH\n #include \n+# ifdef _WIN32\n+# define WIN32_LEAN_AND_MEAN\n+# include \n+# endif\n+# if defined(hpux) || defined(__hpux) || defined(_hpux)\n+# include \n+# endif\n #endif\n \n static const char pack_usage[] = \"\\\n@@ -70,7 +77,7 @@ static int progress = 1;\n static int window = 10;\n static uint32_t pack_size_limit;\n static int depth = 50;\n-static int delta_search_threads = 1;\n+static long delta_search_threads;\n static int pack_to_stdout;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -2004,6 +2011,44 @@ static int adjust_perm(const char *path, mode_t mode)\n \treturn adjust_shared_perm(path);\n }\n \n+/*\n+ * this is a disgusting nest of #ifdef's. I just love\n+ * non-portable interfaces. By doing it in two steps\n+ * we can get the function to be fairly coherent anyways\n+ */\n+#ifndef _SC_NPROCESSORS_ONLN\n+# ifdef _SC_NPROC_ONLN\n+# define _SC_NPROCESSORS_ONLN _SC_NPROC_ONLN\n+# elif defined _SC_CRAY_NCPU\n+# define _SC_NPROCESSORS_ONLN _SC_CRAY_NCPU\n+# endif\n+#endif\n+static long active_cpu_count(void)\n+{\n+#ifdef THREADED_DELTA_SEARCH\n+# ifdef _SC_NPROCESSORS_ONLN\n+\tlong ncpus;\n+\n+\tif ((ncpus = (long)sysconf(_SC_NPROCESSORS_ONLN)) > 0)\n+\t\treturn ncpus;\n+# else\n+# ifdef _WIN32\n+\tSYSTEM_INFO info;\n+\tGetSystemInfo(&info);\n+\n+\treturn (long)info.dwNumberOfProcessors;\n+# endif /* _WIN32 */\n+# if defined(hpux) || defined(__hpux) || defined(_hpux)\n+\tstruct pst_dynamic psd;\n+\n+\tif (!pstat_getdynamic(&psd, sizeof(psd), (size_t)1, 0))\n+\t\treturn (long)psd.psd_proc_cnt;\n+# endif /* hpux */\n+# endif /* _SC_NPROCESSORS_ONLN */\n+#endif /* THREADED_DELTA_SEARCH */\n+\treturn 1;\n+}\n+\n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n \tint use_internal_rev_list = 0;\n@@ -2019,6 +2064,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \trp_av[1] = \"--objects\"; /* --thin will make it --objects-edge */\n \trp_ac = 2;\n \n+\tdelta_search_threads = active_cpu_count();\n+\n \tgit_config(git_pack_config);\n \tif (!pack_compression_seen && core_compression_seen)\n \t\tpack_compression_level = core_compression_level;\n-- \n1.5.3.6.2031.gf9bdc\n##TOHERE##\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"62574","messageId":"alpine.LFD.0.99999.0712100843530.555@xanadu.home","threadId":"11168","inReplyTo":"475D1473.5090809@op5.se","subject":"Re: Better value for chunk_size when threaded","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-10T13:46:09Z","receivedAt":"2007-12-10T13:46:09Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 10 Dec 2007, Andreas Ericsson wrote:\n\n> Nicolas Pitre wrote:\n> > If there is one besides futzing with /proc/cpuinfo I'd like to know about\n> > it. Bonus points if it is portable.\n> > \n> \n> Here is such a one. I've sent it before, using git-send-email, but that\n> one doesn't seem to work too well for all list-members, probably because\n> my own laptop appears to be the original SMTP-server and its name can't\n> be looked up. Sorry for inlining it here instead of sending it as a mail\n> on its own, but I have absolutely no idea how to get git-send-email to\n> do ldap authentication and connect to our tls-enabled smtp-server\n> without using /usr/bin/sendmail and adding my laptop as originating\n> smtp-server.\n> \n> This patch replaces the one I sent earlier and *should* work on\n> everything from Irix and AIX to Linux, Windows and every other\n> posixish system. It passes all tests, both with and without\n> THREADED_DELTA_SEARCH, and causes our weekly repack of our\n> mother-ship repos to run roughly 4 times as fast (4 cores, no\n> previous thread config).\n> \n> Extract with\n> sed -n -e /^##SEDMEHERE##/,/##TOHERE##/p -e /^##/d\n> \n> ##SEDMEHERE##\n> > From ddf08303bd7962be385abbd5e964455a90ed6055 Mon Sep 17 00:00:00 2001\n> From: Andreas Ericsson \n> Date: Thu, 6 Dec 2007 22:09:27 +0100\n> Subject: [PATCH] pack-objects: Add runtime detection of number of CPU's\n> \n> Packing objects can be done in parallell nowadays, but\n> it's only done if the config option pack.threads is set\n> to a value above 1. Because of that, the code-path used\n> is sometimes not the most optimal one.\n> \n> This patch adds a routine to detect the number of active\n> CPU's at runtime, which should provide a better default\n> and activate the (hopefully) better codepath more often.\n\nYour patch is whitespace dammaged.\n\nAlso please make it into a separate .c file. One day, maybe index-pack \nwill want to use it as well.\n\n\nNicolas\n"}]}