{"thread":{"id":"190","subject":"Performance of various compressors","startedAt":"2005-04-21T05:06:38Z","lastAt":"2005-04-25T12:17:22Z","messageCount":11,"participants":["Mike Taht","Klaus Robert Suetterlin","Alecs King","H. Peter Anvin","Aaron Lehmann","David A. Wheeler"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"1093","messageId":"426734DE.3040606@timesys.com","threadId":"190","inReplyTo":null,"subject":"Performance of various compressors","fromName":"Mike Taht","fromEmail":"mike.taht@timesys.com","sentAt":"2005-04-21T05:06:38Z","receivedAt":"2005-04-21T05:06:38Z","isPatch":false,"sender":{"key":"mike.taht@timesys.com","avatar":null},"body":"I started rolling a tool to measure various aspects of git performance. \nI will start looking at merge next, and at workloads different from the \nkernel (gcc4 anyone?) ...\n\nThe only data points worth sharing a this point are:\n\nThat doing the compression at a level of 3, rather than the max of 9, \ncuts the cpu time required for a big git commit by over half, and that \nthat actually translates into a win on the I/O to disk. (these tests \nwere performed on a dual opteron 842)\n\nThe benefits of compression aren't very much for git right now.\n\nAnd: A big git commit is I/O bound. But we knew that. Maybe it's \npossible to make it less I/O bound.\n\nGit branch: 7a4c67965de68ae7bc7aa1fde33f8eb9d8114697\nTree: 2.6.11.7 source tree\nBranch: N/a\nMerge File: N/a\nHW: dual opteron 242\nMem: 1GB\nDisk: seagate barracuda\nFilesystem: Reiser3\nGit add: N/a\nCache: Hot\nGit Commit: 44.97user 5.94system 1:45.24elapsed 48%CPU\nGit Merge:\nOptions:\nFeature: Test of compression=9 (std git)\n\ndu -s .git/objects  110106  # du is probably not the right thing\ndu -s --apparent-size .git/objects 58979\n\nGit branch: 9e272677621c91784cf2533123a41745178f0701\nTree: 2.6.11.7 source tree\nBranch: N/a\nMerge File: N/a\nHW: dual opteron 242\nMem: 1GB\nDisk: seagate barracuda\nDisk mode: udma5\nFilesystem: Reiser3\nGit add: N/a\nCache: Hot\nGit Commit: 16.79user 6.15system 1:21.92elapsed 28%CPU\nGit Merge:\nOptions:\nFeature: Test of compression=3 (std git)\n\ndu -s .git/objects  115218\ndu -s --apparent-size .git/objects 64274\n\nThere's some variety in the best/worst case timings for I/O for the \ncompressor=3 case...\n\n16.79user 6.15system 1:21.92elapsed 28%CPU\n16.68user 5.71system 1:13.19elapsed 30%CPU\n-- \n\nMike Taht\n\n\nlastly -\n\nTimings of git commit with tmpfs (note, these were done with an ancient, \n5 hour old version of git and the script)\n\nHot cache, tmpfs .git compression=9\n44.97user 2.76system 0:47.72elapsed 100%CPU\n\nHot cache, tmpfs .git, compression=6\nWed Apr 20 20:18:11 PDT 2005\n23.55user 2.83system 0:26.36elapsed 100%CPU (0avgtext+0avgdata \n0maxresident)k\n0inputs+0outputs (0major+568680minor)pagefaults 0swaps\n109620  .git/objects\n58618   .git/objects\n\n\n"},{"id":"1096","messageId":"426736AF.7000900@timesys.com","threadId":"190","inReplyTo":"426734DE.3040606@timesys.com","subject":"Re: Performance of various compressors","fromName":"Mike Taht","fromEmail":"mike.taht@timesys.com","sentAt":"2005-04-21T05:14:23Z","receivedAt":"2005-04-21T05:14:23Z","isPatch":false,"sender":{"key":"mike.taht@timesys.com","avatar":null},"body":"Just to clarify this was a git add of the linux-2.6.11.7 sources (sorry, \nuntimed) , and timing the git commit.\n\nMo betta data latah.\n\nMike Taht wrote:\n> I started rolling a tool to measure various aspects of git performance. \n> I will start looking at merge next, and at workloads different from the \n> kernel (gcc4 anyone?) ...\n> \n> The only data points worth sharing a this point are:\n> \n> That doing the compression at a level of 3, rather than the max of 9, \n> cuts the cpu time required for a big git commit by over half, and that \n> that actually translates into a win on the I/O to disk. (these tests \n> were performed on a dual opteron 842)\n> \n> The benefits of compression aren't very much for git right now.\n> \n> And: A big git commit is I/O bound. But we knew that. Maybe it's \n> possible to make it less I/O bound.\n> \n> Git branch: 7a4c67965de68ae7bc7aa1fde33f8eb9d8114697\n> Tree: 2.6.11.7 source tree\n> Branch: N/a\n> Merge File: N/a\n> HW: dual opteron 242\n> Mem: 1GB\n> Disk: seagate barracuda\n> Filesystem: Reiser3\n> Git add: N/a\n> Cache: Hot\n> Git Commit: 44.97user 5.94system 1:45.24elapsed 48%CPU\n> Git Merge:\n> Options:\n> Feature: Test of compression=9 (std git)\n> \n> du -s .git/objects  110106  # du is probably not the right thing\n> du -s --apparent-size .git/objects 58979\n> \n> Git branch: 9e272677621c91784cf2533123a41745178f0701\n> Tree: 2.6.11.7 source tree\n> Branch: N/a\n> Merge File: N/a\n> HW: dual opteron 242\n> Mem: 1GB\n> Disk: seagate barracuda\n> Disk mode: udma5\n> Filesystem: Reiser3\n> Git add: N/a\n> Cache: Hot\n> Git Commit: 16.79user 6.15system 1:21.92elapsed 28%CPU\n> Git Merge:\n> Options:\n> Feature: Test of compression=3 (std git)\n> \n> du -s .git/objects  115218\n> du -s --apparent-size .git/objects 64274\n> \n> There's some variety in the best/worst case timings for I/O for the \n> compressor=3 case...\n> \n> 16.79user 6.15system 1:21.92elapsed 28%CPU\n> 16.68user 5.71system 1:13.19elapsed 30%CPU\n\n\n-- \n\nMike Taht\n\n\n   \"The chief contribution of Protestantism to human thought is its \nmassive proof\nthat God is a bore.\n\t-- H.L. Mencken, \"The Aesthetic Recoil,\" American Mercury, July, 1931.\"\n"},{"id":"1097","messageId":"4267387A.6040602@timesys.com","threadId":"190","inReplyTo":"426736AF.7000900@timesys.com","subject":"[PATCH] experimental - Performance of various compressors","fromName":"Mike Taht","fromEmail":"mike.taht@timesys.com","sentAt":"2005-04-21T05:22:02Z","receivedAt":"2005-04-21T05:22:02Z","isPatch":true,"sender":{"key":"mike.taht@timesys.com","avatar":null},"body":"\n\n\nDon't apply this patch and change GIT_COMPRESSION unless you know what \nyou are doing and why you are doing it. You will break an older version \nof git. You may break a newer version of git. You have been warned.\n\nI also note that there's a bzlib out there.\n\ncache.h: 828d660ab82bb35a1ca632a2ba4620dc483889bd\n--- a/cache.h\n+++ b/cache.h\n@@ -16,6 +16,8 @@\n  #include <openssl/sha.h>\n  #include <zlib.h>\n\n+#define GIT_COMPRESSION Z_BEST_COMPRESSION\n+\n  /*\n   * Basic data structures for the directory cache\n   *\nsha1_file.c: 754e8b4e9ea8104df48152f875d6b874304e2a62\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -199,7 +199,7 @@ int write_sha1_file(char *buf, unsigned\n\n         /* Set it up */\n         memset(&stream, 0, sizeof(stream));\n-       deflateInit(&stream, Z_BEST_COMPRESSION);\n+       deflateInit(&stream, GIT_COMPRESSION);\n         size = deflateBound(&stream, len);\n         compressed = malloc(size);\n\nupdate-cache.c: a09883541c745c76413c62109a80f40df4b7a7fb\n--- a/update-cache.c\n+++ b/update-cache.c\n@@ -40,7 +40,7 @@ static int index_fd(unsigned char *sha1,\n         SHA1_Final(sha1, &c);\n\n         memset(&stream, 0, sizeof(stream));\n-       deflateInit(&stream, Z_BEST_COMPRESSION);\n+       deflateInit(&stream, GIT_COMPRESSION);\n\n         /*\n          * ASCII size + nul byte\n\n\n\nMike Taht wrote:\n> Just to clarify this was a git add of the linux-2.6.11.7 sources (sorry, \n> untimed) , and timing the git commit.\n> \n> Mo betta data latah.\n> \n> Mike Taht wrote:\n> \n>> I started rolling a tool to measure various aspects of git \n>> performance. I will start looking at merge next, and at workloads \n>> different from the kernel (gcc4 anyone?) ...\n>>\n>> The only data points worth sharing a this point are:\n>>\n>> That doing the compression at a level of 3, rather than the max of 9, \n>> cuts the cpu time required for a big git commit by over half, and that \n>> that actually translates into a win on the I/O to disk. (these tests \n>> were performed on a dual opteron 842)\n>>\n>> The benefits of compression aren't very much for git right now.\n>>\n>> And: A big git commit is I/O bound. But we knew that. Maybe it's \n>> possible to make it less I/O bound.\n>>\n>> Git branch: 7a4c67965de68ae7bc7aa1fde33f8eb9d8114697\n>> Tree: 2.6.11.7 source tree\n>> Branch: N/a\n>> Merge File: N/a\n>> HW: dual opteron 242\n>> Mem: 1GB\n>> Disk: seagate barracuda\n>> Filesystem: Reiser3\n>> Git add: N/a\n>> Cache: Hot\n>> Git Commit: 44.97user 5.94system 1:45.24elapsed 48%CPU\n>> Git Merge:\n>> Options:\n>> Feature: Test of compression=9 (std git)\n>>\n>> du -s .git/objects  110106  # du is probably not the right thing\n>> du -s --apparent-size .git/objects 58979\n>>\n>> Git branch: 9e272677621c91784cf2533123a41745178f0701\n>> Tree: 2.6.11.7 source tree\n>> Branch: N/a\n>> Merge File: N/a\n>> HW: dual opteron 242\n>> Mem: 1GB\n>> Disk: seagate barracuda\n>> Disk mode: udma5\n>> Filesystem: Reiser3\n>> Git add: N/a\n>> Cache: Hot\n>> Git Commit: 16.79user 6.15system 1:21.92elapsed 28%CPU\n>> Git Merge:\n>> Options:\n>> Feature: Test of compression=3 (std git)\n>>\n>> du -s .git/objects  115218\n>> du -s --apparent-size .git/objects 64274\n>>\n>> There's some variety in the best/worst case timings for I/O for the \n>> compressor=3 case...\n>>\n>> 16.79user 6.15system 1:21.92elapsed 28%CPU\n>> 16.68user 5.71system 1:13.19elapsed 30%CPU\n> \n> \n> \n\n\n-- \n\nMike Taht\n\n\n   \"\"His mind is like a steel trap -- full of mice.\"\n\t-- Foghorn Leghorn\"\n"},{"id":"1114","messageId":"20050421102326.GA22541@xdt04.mpe-garching.mpg.de","threadId":"190","inReplyTo":"4267387A.6040602@timesys.com","subject":"HOWTO: PATCH: don't hardcode path-to-bash, use sys/limits.h","fromName":"Klaus Robert Suetterlin","fromEmail":"robert@mpe.mpg.de","sentAt":"2005-04-21T10:23:26Z","receivedAt":"2005-04-21T10:23:26Z","isPatch":false,"sender":{"key":"robert@mpe.mpg.de","avatar":null},"body":"Hi,\n\nI supply a patch that dehardcodes the path to bash (which is not /bin on all computers) and adds sys/limits.h to provide ULONG_MAX.\n\nIf this is not the right way to supply patches, or if this email misses some crucial point please tell me so, and supply explanation.\n\n-- \nRobert Suetterlin (robert@mpe.mpg.de)\nphone: (+49)89 / 30000-3546   fax: (+49)89 / 30000-3950\n\ncommit 5f6caff82b1f3b5931d92aaff99be6d8dbad10ca\ntree d7ea8aeefbbc2ab63cb5acd41b647b1b5f11fb83\nparent cd1c034369b73da7503da365fa556aab27004814\nauthor Klaus Robert Suetterlin <krs@xdt04.mpe-garching.mpg.de> 1114078431 +0200\ncommitter Klaus Robert Suetterlin <krs@xdt04.mpe-garching.mpg.de> 1114078431 +0200\n\nDon't hardcode the path-to-bash please.\n\nIndex: commit.c\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/commit.c  (mode:100644 sha1:eda45d7e15358ed6f2cd0502de2a08987307fc98)\n+++ d7ea8aeefbbc2ab63cb5acd41b647b1b5f11fb83/commit.c  (mode:100644 sha1:cfe9a8ddf6ee2702e3923cb22240f9f9ed1bd04c)\n@@ -1,3 +1,4 @@\n+#include <sys/limits.h>\n #include \"commit.h\"\n #include \"cache.h\"\n #include <string.h>\nIndex: gitdiff-do\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/gitdiff-do  (mode:100755 sha1:afed4e40b259a61b0f12979ba7326f26743bc553)\n+++ d7ea8aeefbbc2ab63cb5acd41b647b1b5f11fb83/gitdiff-do  (mode:100755 sha1:218dfabeb4a5dcbd2cf58bd6f672f385690ec397)\n@@ -1,4 +1,4 @@\n-#!/bin/bash\n+#!/usr/bin/env bash\n #\n # Make a diff between two GIT trees.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitlog.sh\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/gitlog.sh  (mode:100755 sha1:a496a864f9586e47a4d7bd3ae0af0b3e07b7deb8)\n+++ d7ea8aeefbbc2ab63cb5acd41b647b1b5f11fb83/gitlog.sh  (mode:100755 sha1:7b3aa8a89bc64273c648920ccd1686859754803e)\n@@ -1,4 +1,4 @@\n-#!/bin/bash\n+#!/usr/bin/env bash\n #\n # Make a log of changes in a GIT branch.\n #\n"},{"id":"1132","messageId":"20050421143102.GA830@alc.bsd.st","threadId":"190","inReplyTo":"20050421102326.GA22541@xdt04.mpe-garching.mpg.de","subject":"Re: HOWTO: PATCH: don't hardcode path-to-bash, use sys/limits.h","fromName":"Alecs King","fromEmail":"alecsk@gmail.com","sentAt":"2005-04-21T14:31:02Z","receivedAt":"2005-04-21T14:31:02Z","isPatch":false,"sender":{"key":"alecsk@gmail.com","avatar":"https://avatars.githubusercontent.com/u/84553?v=4"},"body":"On Thu, Apr 21, 2005 at 12:23:26PM +0200, Klaus Robert Suetterlin wrote:\n> Hi,\n> \n> I supply a patch that dehardcodes the path to bash (which is not /bin\n> on all computers) and adds sys/limits.h to provide ULONG_MAX.\n\nHi, i did a similar patch a while back ago. As for ULONG_MAX, not every\nsytem has <sys/limits.h>, i think <limits.h> is the rite place to go.\n\nThe patch below tested on both debian and fbsd.\n\n\ncommit 2deea74db72fb57a8b80e7945f23814112b22723\ntree 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd\nparent cd1c034369b73da7503da365fa556aab27004814\nauthor Alecs King <alecsk ! gmail d@t com> 1114075114 +0800\ncommitter Alecs King <alecsk ! gmail d@t com> 1114075114 +0800\n\ntrivial fix for making it more portable\n\nIndex: commit-tree.c\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/commit-tree.c  (mode:100644 sha1:043c7aa371101a1ea8cfc467279abf6c8acc7fd1)\n+++ 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/commit-tree.c  (mode:100644 sha1:8a1f12dca07041d203ce22442b8470d42d322ef5)\n@@ -252,7 +252,7 @@\n \n \tthen -= offset;\n \n-\tsnprintf(result, maxlen, \"%lu %5.5s\", then, p);\n+\tsnprintf(result, maxlen, \"%lu %5.5s\", (unsigned long) then, p);\n }\n \n static void check_valid(unsigned char *sha1, const char *expect)\nIndex: commit.c\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/commit.c  (mode:100644 sha1:eda45d7e15358ed6f2cd0502de2a08987307fc98)\n+++ 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/commit.c  (mode:100644 sha1:9f0668eb68cec56a738a58fe930ae0ae2960e2b2)\n@@ -1,6 +1,7 @@\n #include \"commit.h\"\n #include \"cache.h\"\n #include <string.h>\n+#include <limits.h>\n \n const char *commit_type = \"commit\";\n \nIndex: gitdiff-do\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/gitdiff-do  (mode:100755 sha1:afed4e40b259a61b0f12979ba7326f26743bc553)\n+++ 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitdiff-do  (mode:100755 sha1:218dfabeb4a5dcbd2cf58bd6f672f385690ec397)\n@@ -1,4 +1,4 @@\n-#!/bin/bash\n+#!/usr/bin/env bash\n #\n # Make a diff between two GIT trees.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitlog.sh\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/gitlog.sh  (mode:100755 sha1:a496a864f9586e47a4d7bd3ae0af0b3e07b7deb8)\n+++ 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitlog.sh  (mode:100755 sha1:7b3aa8a89bc64273c648920ccd1686859754803e)\n@@ -1,4 +1,4 @@\n-#!/bin/bash\n+#!/usr/bin/env bash\n #\n # Make a log of changes in a GIT branch.\n #\nIndex: revision.h\n===================================================================\n--- c0260bfb82da04aeff4e598ced5295d6ae2e262d/revision.h  (mode:100644 sha1:46cc10440be781cea4993aca37ee35e251495084)\n+++ 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/revision.h  (mode:100644 sha1:f0754f5d8ea3da52503b8ea8c16b34566e4ae6e0)\n@@ -10,6 +10,7 @@\n  * definition for this rev, and not just seen it as\n  * a parent target.\n  */\n+#include <limits.h>\n #define marked(rev)\t((rev)->flags & 0xffff)\n #define SEEN 0x10000\n #define USED 0x20000\n\n-- \nAlecs King\n"},{"id":"1180","messageId":"20050421194255.GA8479@alc.bsd.st","threadId":"190","inReplyTo":"20050421143102.GA830@alc.bsd.st","subject":"[PATCH] #!/bin/sh --> #!/usr/bin/env bash","fromName":"Alecs King","fromEmail":"alecsk@gmail.com","sentAt":"2005-04-21T19:42:55Z","receivedAt":"2005-04-21T19:42:55Z","isPatch":true,"sender":{"key":"alecsk@gmail.com","avatar":"https://avatars.githubusercontent.com/u/84553?v=4"},"body":"On Thu, Apr 21, 2005 at 10:31:02PM +0800, Alecs King wrote:\n> On Thu, Apr 21, 2005 at 12:23:26PM +0200, Klaus Robert Suetterlin wrote:\n> > Hi,\n> > \n> > I supply a patch that dehardcodes the path to bash (which is not /bin\n> > on all computers) and adds sys/limits.h to provide ULONG_MAX.\n> \n> Hi, i did a similar patch a while back ago. As for ULONG_MAX, not every\n> sytem has <sys/limits.h>, i think <limits.h> is the rite place to go.\n> \n> The patch below tested on both debian and fbsd.\n>  \n> [snip]\n\nAnd as for bash, only gitdiff-do and gitlog.sh 'explicitly' use bash\ninstead of /bin/sh.  On most Linux distros, /bin/sh is just a symbolic\nlink to bash.  But not on some others.  I found gitlsobj.sh could not\nwork using a plain /bin/sh on fbsd.  To make life easier, i think it\nmight be better if we all explicitly use bash for all shell scripts.\n\npatch below assumes the patch above has been applied.\n\n\ncommit 341cd1241815178d567ce612c97c2bb5a663021a\ntree abb16c39fe8354383b632f7fa9dd4611ff66e1d1\nparent 2deea74db72fb57a8b80e7945f23814112b22723\nauthor Alecs King <alecsk ! gmail d@t com> 1114107613 +0800\ncommitter Alecs King <alecsk ! gmail d@t com> 1114107613 +0800\n\nExplicitly use bash\n#!/bin/sh ==> #!/usr/bin/env bash\n\nIndex: gitXlntree.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitXlntree.sh  (mode:100755 sha1:c474913d09906739d8175f1b430720a3ac67e798)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitXlntree.sh  (mode:100755 sha1:adc01eeb56f394a6168ae1f6f1fe4c40e1c2aecc)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Provide an independent view to the objects database.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitXnormid.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitXnormid.sh  (mode:100755 sha1:c0d53afabe8662ebfc3c697faf08b0a2b43c93f7)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitXnormid.sh  (mode:100755 sha1:9b311aca57bd8b7012f45d730c6fd26d5fb5d2b2)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Internal: Normalize the given ID to a tree ID.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitadd.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitadd.sh  (mode:100755 sha1:3f5e9a2d6b452d596cd853f1585113bdb356a2e3)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitadd.sh  (mode:100755 sha1:6feb7372e95be4546af17e0c6b55d10c9a1c441d)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Add new file to a GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitaddremote.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitaddremote.sh  (mode:100755 sha1:a117b9e8d14b977143caa48c26fc51794e8b7135)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitaddremote.sh  (mode:100755 sha1:bccaa9068063b07d13012477861c6706b7cd40a6)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Add new \"remote\" to the GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitapply.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitapply.sh  (mode:100755 sha1:7703809dc0743c6e4c1fa5b7d922a4efc16b4276)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitapply.sh  (mode:100755 sha1:794ea5ed6acdd34e34742a17cbd784dcbf738289)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Apply a diff generated by git diff.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitcancel.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitcancel.sh  (mode:100755 sha1:74b4083d67eda87d88a6f92c6c66877bba8bda8a)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitcancel.sh  (mode:100755 sha1:c320ee98e2ed0b13a68de3b2ec4e4a8451b5189a)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Cancels current edits in the working tree.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitcommit.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitcommit.sh  (mode:100755 sha1:a13bef2c84492ed75679d7d52bb710df35544f8a)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitcommit.sh  (mode:100755 sha1:0207f402cc5107de2a4685f6fcade081c41d91e9)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Commit into a GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitdiff.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitdiff.sh  (mode:100755 sha1:8e14a868f513f4ec524a2c8974c8d202c6824038)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitdiff.sh  (mode:100755 sha1:e27915d4172717ddd4d01269877312b08ed2acc4)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Make a diff between two GIT trees.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitexport.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitexport.sh  (mode:100755 sha1:5b94424beca55ffe6b5535e4975e6e63c1bae672)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitexport.sh  (mode:100755 sha1:428cd9d845598e320556729b6098505132a4e7c4)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Exports a particular revision from a GIT repository.\n # Copyright (c) Johannes E. Schindelin, 2005\nIndex: gitfork.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitfork.sh  (mode:100755 sha1:b827c3037ac4f3cdfb6708bf8edb60944f59318a)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitfork.sh  (mode:100755 sha1:ce26f985ebb48b6a3127ac8afd427ba30ba5668a)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Create a branch sharing the objects database.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitinit.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitinit.sh  (mode:100755 sha1:9905166859827893e326b01bdc3970ff6d51064d)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitinit.sh  (mode:100755 sha1:bc00e9ee709aabeb4764b77ac4e5a19212fa5857)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Initialize a GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitls.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitls.sh  (mode:100755 sha1:c8d2220eae66addd49493cdb32af21b6c0217b23)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitls.sh  (mode:100755 sha1:a05883b09512bd1d1fe31e1c6d43f01a395c58a1)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # List contents of a particular tree in a GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitlsobj.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitlsobj.sh  (mode:100755 sha1:423a1bc7476bad7bf40f1b3ddb03d83fdcf1f9cd)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitlsobj.sh  (mode:100755 sha1:3f4426eeac7cc5ad51a46632319814fbf62b2cc3)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # List objects of the GIT repository.\n # Copyright (c) Randy Dunlap, 2005\nIndex: gitlsremote.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitlsremote.sh  (mode:100755 sha1:2212be93aaa8a371e83cafb69fa21a7a1b24ed13)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitlsremote.sh  (mode:100755 sha1:29657d7a899ffb425a36ec04bf1c62aa1ecc14d7)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Lists remote GIT repositories\n # Copyright (c) Steven Cole 2005\nIndex: gitmerge-file.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitmerge-file.sh  (mode:100755 sha1:820de487babb76ce419b6823c8fe4c58608d0c8c)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitmerge-file.sh  (mode:100755 sha1:237186eaefc4a503c386e4a0e7c28818e6704db7)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Copyright (c) Linus Torvalds, 2005\n #\nIndex: gitmerge.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitmerge.sh  (mode:100755 sha1:bc68f6cda84cbf1165d71b17d6207b3c46a8cad4)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitmerge.sh  (mode:100755 sha1:92e552700a40c5e1f7339c9b1f261cb39206a3c3)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Merge a branch to the current tree.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitpatch.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitpatch.sh  (mode:100755 sha1:580e3e6b0c23625abd2288be35ee33a787a1ba3c)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitpatch.sh  (mode:100755 sha1:fd00c88133c874ac71a90a045a313363f9f22350)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Make a patch from a given commit.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitpull.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitpull.sh  (mode:100755 sha1:0cafc0270ea91aaf099f398b7e5cd360be9ea086)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitpull.sh  (mode:100755 sha1:7f847f39e0b2aa150fe195d8d4f6f0d62487ae72)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Pulls changes from \"remote\" to the local GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitrm.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitrm.sh  (mode:100755 sha1:3fa31f9a1ae843dcb184b8371ff60f626e8820b3)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitrm.sh  (mode:100755 sha1:e014b979ea7b8f7ae69eabc7dd146c8a7f286d19)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Remove a file from a GIT repository.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitseek.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitseek.sh  (mode:100755 sha1:b80969a4ba040202827ea7532235abab15ca9392)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitseek.sh  (mode:100755 sha1:035b78a93307da8f67f7447ed3a182a6d17d2c50)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Seek the working tree to a given commit.\n # Copyright (c) Petr Baudis, 2005\nIndex: gitstatus.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gitstatus.sh  (mode:100755 sha1:7d5209ea838106eb2ab5bde2704997508a22a4e8)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gitstatus.sh  (mode:100755 sha1:9cfb2ce947082002cff3e5497ca2a994c4bbb101)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Show status of entries in your working tree.\n # Copyright (c) Petr Baudis, 2005\nIndex: gittag.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gittag.sh  (mode:100755 sha1:9e1e200deda54b2401d6d685f0d5305cfbfa38ca)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gittag.sh  (mode:100755 sha1:19c7f3ecffa55f117c27c9a1d8de67f65805f1c7)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Mark certain commit by a tag.\n # Copyright (c) Petr Baudis, 2005\nIndex: gittrack.sh\n===================================================================\n--- 0c92ac3af53457b6b9651cf82d98ce3a7b166dcd/gittrack.sh  (mode:100755 sha1:7509d4adb2b2c50cd2acdf9126fc57cff79e6009)\n+++ abb16c39fe8354383b632f7fa9dd4611ff66e1d1/gittrack.sh  (mode:100755 sha1:bdf33313f4ad0c4f7b6b235fbc68fc85f226a33a)\n@@ -1,4 +1,4 @@\n-#!/bin/sh\n+#!/usr/bin/env bash\n #\n # Makes your working tree track the given branch.\n # Copyright (c) Petr Baudis, 2005\n\n-- \nAlecs King\n"},{"id":"1246","messageId":"4268A9C5.5090102@zytor.com","threadId":"190","inReplyTo":"20050421194255.GA8479@alc.bsd.st","subject":"Re: [PATCH] #!/bin/sh --> #!/usr/bin/env bash","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-04-22T07:37:41Z","receivedAt":"2005-04-22T07:37:41Z","isPatch":true,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Alecs King wrote:\n> \n> And as for bash, only gitdiff-do and gitlog.sh 'explicitly' use bash\n> instead of /bin/sh.  On most Linux distros, /bin/sh is just a symbolic\n> link to bash.  But not on some others.  I found gitlsobj.sh could not\n> work using a plain /bin/sh on fbsd.  To make life easier, i think it\n> might be better if we all explicitly use bash for all shell scripts.\n> \n\nHow about #!/bin/bash (build from .in files if you feel it necessary to \nsupport systems which don't have bash in /bin) instead of doubling the \nnumber of execs?\n\n\t-hpa\n"},{"id":"1319","messageId":"20050422203801.GE7437@vitelus.com","threadId":"190","inReplyTo":"426734DE.3040606@timesys.com","subject":"Re: Performance of various compressors","fromName":"Aaron Lehmann","fromEmail":"aaronl@vitelus.com","sentAt":"2005-04-22T20:38:01Z","receivedAt":"2005-04-22T20:38:01Z","isPatch":false,"sender":{"key":"aaronl@vitelus.com","avatar":null},"body":"On Wed, Apr 20, 2005 at 10:06:38PM -0700, Mike Taht wrote:\n> That doing the compression at a level of 3, rather than the max of 9, \n> cuts the cpu time required for a big git commit by over half, and that \n> that actually translates into a win on the I/O to disk. (these tests \n> were performed on a dual opteron 842)\n\nIf (de)compression is slowing things down, you might want to check out\nlzo (http://www.oberhumer.com/opensource/lzo/). I tested it on the\n2.6.11 kernel source and found that lzo -7 output is only 2% larger\nthan gzip -3, but lzo decompression is almost 3 times faster. The\ndownside is that lzo took 5 times longer to perform the compression at\n-7. Compression with lzo -3 is 3.5 times faster than gzip -3, but it\nproduces a file that's 37% bigger. Unfortunately, lzo has no settings\nin between -3 and -7. I'd expect git to be more sensitive to\ndecompression speeds, though.\n\nBTW, lzo decompression speed is not affected by the compression level.\n"},{"id":"1362","messageId":"4269B424.8010406@dwheeler.com","threadId":"190","inReplyTo":"4268A9C5.5090102@zytor.com","subject":"Re: [PATCH] #!/bin/sh --> #!/usr/bin/env bash","fromName":"David A. Wheeler","fromEmail":"dwheeler@dwheeler.com","sentAt":"2005-04-23T02:34:12Z","receivedAt":"2005-04-23T02:34:12Z","isPatch":true,"sender":{"key":"dwheeler@dwheeler.com","avatar":"https://avatars.githubusercontent.com/u/813150?v=4"},"body":"\n> Alecs King wrote:\n> \n>>\n>> And as for bash, only gitdiff-do and gitlog.sh 'explicitly' use bash\n>> instead of /bin/sh.  On most Linux distros, /bin/sh is just a symbolic\n>> link to bash.  But not on some others.  I found gitlsobj.sh could not\n>> work using a plain /bin/sh on fbsd.  To make life easier, i think it\n>> might be better if we all explicitly use bash for all shell scripts.\n\n\nH. Peter Anvin wrote:\n> How about #!/bin/bash (build from .in files if you feel it necessary to \n> support systems which don't have bash in /bin) instead of doubling the \n> number of execs?\n\nIf # of execs is that critical, it probably should not be in\nbash anyway.  OpenBSD (at least 3.1)'s bash appears to be in\n/usr/local/bin/bash, NOT /bin/bash.\nI'd go with the /bin/env solution for now;\nit maximizes the \"it just works\" factor, and\nwhen it comes time for .in files much of the cogito code (at least)\nwill probably be rewritten in Perl, and anything performance-sensitive\nwill be in C.\n\n--- David A. Wheeler\n"},{"id":"1366","messageId":"4269E82D.9010009@zytor.com","threadId":"190","inReplyTo":"4269B424.8010406@dwheeler.com","subject":"Re: [PATCH] #!/bin/sh --> #!/usr/bin/env bash","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-04-23T06:16:13Z","receivedAt":"2005-04-23T06:16:13Z","isPatch":true,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"David A. Wheeler wrote:\n> \n> If # of execs is that critical, it probably should not be in\n> bash anyway.  OpenBSD (at least 3.1)'s bash appears to be in\n> /usr/local/bin/bash, NOT /bin/bash.\n> I'd go with the /bin/env solution for now;\n> it maximizes the \"it just works\" factor, and\n> when it comes time for .in files much of the cogito code (at least)\n> will probably be rewritten in Perl, and anything performance-sensitive\n> will be in C.\n> \n\nMakes sense.\n\n\t-hpa\n"},{"id":"1607","messageId":"20050425121722.GA63026@xdt04.mpe-garching.mpg.de","threadId":"190","inReplyTo":"20050422203801.GE7437@vitelus.com","subject":"git I/O performance (was: Performance of various compressors)","fromName":"Klaus Robert Suetterlin","fromEmail":"robert@mpe.mpg.de","sentAt":"2005-04-25T12:17:22Z","receivedAt":"2005-04-25T12:17:22Z","isPatch":false,"sender":{"key":"robert@mpe.mpg.de","avatar":null},"body":"I did some statistics on the freebsd /usr/src/sys directory, as I\ndid not have access to the linux kernel sources.\n\nThis is 5435 Files, of about 81MB (according to du -sh).  I did\n\nfind sys/ -type f -exec gzip -9 {} +\nfind sys/ -type f -exec gzip -d {} +\n\nand similar calls to get an impression how different compression\nlevels and compressors will act on the data most likely handled by\ngit backend storage.\n\nOn a 700MHz p3, UDMA33, freebsd 5.3, ffs (soft updates) I get:\n\ncompressor | levels (size, time to compress, time to uncompress)\n-----------+-------------------------------------------------------------------\ngzip       | 9 (28M, 1:19, 30), 6 (28M, 31.7, 30), 3 (30M, 26.1, 28.7)\n           | 1 (31M, 23.6, 29.8)\nbzip2      | 9 (27M, 2:14, 37.4) 6 (27M, 2:11, 38.8) 3 (27M, 2:10, 38.3)\nlzop       | 9 (32M, 2:15, 35.4) 7 (32M, 57.9, 40.3) 3 (39M, 36.0, 44.4)\n\n\nThese speeds are for the case that our work set fits into filesystem\ncaches.  This will be the most common case --- as normal commits will\nnot checkin the whole tree.\n\nThat is.  We should really use gzip -6.  It results in the best\ncompression at a reasonable time.  bzip2 can't really compress those\ntiny files efficiently. lzop is limited by open/close (see below).\n\nBTW. I also did this for the whole /usr/src of freebsd (which is\n35000 files and 350MB, du -sh gives 398MB).  The numbers look best\nfor gzip -6.\n\nThe files we work with seem to have an average uncompressed size\nof 10-15kB and seem to compress by about a factor of three.\n\nSo I did a test in C: open(\"file%d\"), write(file, buf, 10000),\nclose(file).  I repeated this for 35000 files as in the freebsd src\ncase, to get some statistics.  The gprof output tells me, that\nopen+close take the same ammount of time as the write.  (You should\nreally try do to rm test[0-9]* on those 35000 files :))  I wrote\nthe full 10000 bytes, to check for the case when we have no compression\nat all.  When compression gets better we will become more and more\nopen/close limited.\n\nThis means we are open/close limited in git.  Even if we compress\nthe files to zero size, we cannot get faster than by a factor of\ntwo!\n\nEarlier messages in this thread showed that we are also limited by\nfilesytem cache, so we should use compression and efficient prefetch\nto get best performance out of it.  Because even if we cannot get\nfaster than by a factor of two through compression (even delta\ncompression won't help!  It would make things worse IMHO) we can\nget a lot worse (like ten times slower) for large sets on slow\nmachines with few memory.\n\nI also tried to get a better ratio by using the standard db.h btree\ndatabase, so I wouldn't have to open and close all those files.\nUnfortunately the btree is about twice as large as the files, so I\nhad to write twice as much data to disk(800MB).  Also db->put\nis much more complicated than write.  So the test ended up taking\nabout 10% more time, than the open/write/close case.  Maybe in the\ncase of a smaller work set (i.e. 1000 files instead of 35000) this\nmight provide faster backend speeds.  Also one could optimise speed\nby tweaking the acccess method parameters.\n\n\nOn Fri, Apr 22, 2005 at 01:38:01PM -0700, Aaron Lehmann wrote:\n> On Wed, Apr 20, 2005 at 10:06:38PM -0700, Mike Taht wrote:\n> > That doing the compression at a level of 3, rather than the max of 9, \n> > cuts the cpu time required for a big git commit by over half, and that \n> > that actually translates into a win on the I/O to disk. (these tests \n> > were performed on a dual opteron 842)\n> \n> If (de)compression is slowing things down, you might want to check out\n> lzo (http://www.oberhumer.com/opensource/lzo/). I tested it on the\n> 2.6.11 kernel source and found that lzo -7 output is only 2% larger\n> than gzip -3, but lzo decompression is almost 3 times faster. The\n> downside is that lzo took 5 times longer to perform the compression at\n> -7. Compression with lzo -3 is 3.5 times faster than gzip -3, but it\n> produces a file that's 37% bigger. Unfortunately, lzo has no settings\n> in between -3 and -7. I'd expect git to be more sensitive to\n> decompression speeds, though.\n> \n> BTW, lzo decompression speed is not affected by the compression level.\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n\n-- \nRobert Suetterlin (robert@mpe.mpg.de)\nphone: (+49)89 / 30000-3546   fax: (+49)89 / 30000-3950\n"}]}