{"thread":{"id":"34169","subject":"[RFC] speed up git submodule","startedAt":"2013-06-17T00:05:03Z","lastAt":"2013-06-17T08:29:44Z","messageCount":2,"participants":["Fredrik Gustafsson","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"221022","messageId":"1371427503-9678-1-git-send-email-iveqy@iveqy.com","threadId":"34169","inReplyTo":null,"subject":"[RFC] speed up git submodule","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2013-06-17T00:05:03Z","receivedAt":"2013-06-17T00:05:03Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"I've been playing a bit with lua. It's an embedded scripting language\nwith strong c integration. It's small and fast.\n\nThe interesting feature would be to run C-functions direct inside lua. I\nsuppose that would increase speed even more, at the same time as we have\nthe convinence of a interpreted language. Lua is smaller and faster\n(well as always, it depends on what you're doing) than python and ruby.\nPerl is a really pain for the windows folks (I've heard).\n\nA correct implementation for lua support would be to start a\nlua-interpreter from inside git.c (or somewhere) and load the lua code\nfor a specific command. That would make us independent of any target\ninstallation of lua (althought the git binary would increase with the\nlua library around 300 kb).\n\nHowever I did a quick test using lua as a replacement for sh (without\ndirect calls to c-functions) and the result is impressive. (However this\nis the wrong way of using lua, shell scripting is not something lua is\ngood at).\n\nI did some runs on a project with 52 submodules (or 53 if you count the\nones in .gitmodules). These results are pretty typical:\niveqy@kolya:~/projects/eracle_core$ time /home/iveqy/projects/git/git-submodule.lua > /dev/null\n\nreal    0m1.665s\nuser    0m0.276s\nsys     0m0.452s\niveqy@kolya:~/projects/eracle_core$ time git submodule > /dev/null\n\nreal    0m3.413s\nuser    0m0.476s\nsys     0m1.224s\n\nFor me, that speedup does matter.\n\nNOTICE!!!\nThis code is experimental. It does have some known bugs, it does have\nsome style issues. A state of the art complete implementation would\ncontain a few more tests/jumps and less concat (which is extremely\nexpensive in lua) and less git-invokation.\n\nSigned-off-by: Fredrik Gustafsson <iveqy@iveqy.com>\n---\n git-submodule.lua | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 104 insertions(+)\n create mode 100755 git-submodule.lua\n\ndiff --git a/git-submodule.lua b/git-submodule.lua\nnew file mode 100755\nindex 0000000..14f71e6\n--- /dev/null\n+++ b/git-submodule.lua\n@@ -0,0 +1,104 @@\n+#!/usr/bin/lua\n+\n+function run_cmd(cmd)\n+\tlocal f = io.popen(cmd, 'r');\n+\tlocal out = f:read('*a');\n+\tf:close()\n+\treturn out\n+end\n+\n+function fwrite(fmt, ...)\n+\treturn io.write(string.format(fmt, ...))\n+end\n+\n+function read_gitmodules()\n+\tlocal inf = assert(io.open('.gitmodules', 'r'))\n+\tlocal config = inf:read(\"*all\")\n+\tgitmodules = {}\n+\tfor sm in string.gmatch(config, '%[[^]]*%][^%[]*') do\n+\t\tlocal thismod = {}\n+\t\tlocal name = string.match(sm, '%[%s-submodule%s-\"(.+)\"%s-%]')\n+\t\tthismod[\"name\"] = name\n+\t\tlocal path = ''\n+\t\tfor k, v in string.gmatch(sm, '\\n%s*([^=^%s]*)%s*=%s*([^\\n]*)') do\n+\t\t\tif k == 'path' then\n+\t\t\t\tpath = v\n+\t\t\telse\n+\t\t\t\tthismod[k] = v\n+\t\t\tend\n+\t\tend\n+\t\tif path == '' then\n+\t\t\tfwrite(\"No path found for %s in .gitmodules\\n\", name)\n+\t\t\tos.exit(1)\n+\t\tend\n+\t\tgitmodules[path] = thismod\n+\tend\n+\n+\treturn gitmodules\n+end\n+\n+function module_list()\n+\tlocal lsfiles = 'git ls-files --stage --error-unmatch -z || echo \"#unmatched\"'\n+\tlocal out = run_cmd(lsfiles)\n+\tlocal unmerged = ''\n+\tlocal subs = read_gitmodules()\n+\n+\tfor row in string.gmatch(out, '.-\\0') do\n+\t\tif row == '#unmatched' then\n+\t\t\tos.exit(1)\n+\t\tend\n+\n+\t\tlocal mode, sha1, stage, path = string.match(row, '(%d+)%s([0-9a-f]+)%s(.)%s(.*)\\0')\n+\t\tif mode == '160000' then\n+\t\t\tif stage == '0' then\n+\t\t\t\tsubs[path][\"sha1\"] = sha1\n+\t\t\t\tsubs[path][\"stage\"] = stage\n+\t\t\telse\n+\t\t\t\tif unmerged ~= path then\n+\t\t\t\t\tlocal null_sha1 = '0000000000000000000000000000000000000000'\n+\t\t\t\t\tsubs[path][\"sha1\"] = null_sha1\n+\t\t\t\t\tsubs[path][\"stage\"] = 'U'\n+\t\t\t\tend\n+\t\t\t\tunmerged = path\n+\t\t\tend\n+\t\tend\n+\tend\n+\treturn subs\n+end\n+\n+function get_name_rev(path, sha1)\n+\tif sha1 == nil then sha1=\"\" end\n+\tlocal cmd = \"cd \\\"\" .. path .. \"\\\" && (git describe \" .. sha1 ..\n+\t\t\t\t\" 2>/dev/null || git describe --tags \" .. sha1 ..\n+\t\t\t\t\" 2>/dev/null || git describe --contains \" .. sha1 ..\n+\t\t\t\t\" 2>/dev/null || git describe --all --always \" .. sha1 ..\n+\t\t\t\t\" 2>/dev/null) \"\n+\treturn string.gsub(run_cmd(cmd), '\\n', '')\n+end\n+\n+function cmd_status()\n+\tsubs = module_list()\n+\tfor smpath in pairs(subs) do\n+\t\tif (subs[smpath].sha1) then\n+\t\t\tif subs[smpath].stage == 'U' then\n+\t\t\t\tsubs[smpath][\"revname\"] = get_name_rev(smpath, subs[smpath].sha1)\n+\t\t\t\tfwrite(\"U%s %s (%s)\", subs[smpath].sha1, smpath, subs[smpath].revname)\n+\t\t\telseif run_cmd(\"test -z \" .. subs[smpath].url .. \" || ! test -d \" .. smpath ..\"/.git -o -f \" .. smpath .. \"/.git || echo '0' \") ~= '0\\n' then\n+\t\t\t\tsubs[smpath][\"revname\"] = get_name_rev(smpath, subs[smpath].sha1)\n+\t\t\t\tfwrite(\"-%s %s (%s)\\n\", subs[smpath].sha1, smpath, subs[smpath].revname)\n+\t\t\telseif run_cmd(\"git diff-files --ignore-submodules=dirty --quiet -- \" .. smpath .. \" || echo '0'\") ~= '0\\n' then\n+\t\t\t\tp = run_cmd(\"git diff-files --ignore-submodules=dirty --quiet -- \" .. smpath .. \" || echo '0'\")\n+\t\t\t\tsubs[smpath][\"revname\"] = get_name_rev(smpath, subs[smpath].sha1)\n+\t\t\t\tfwrite(\" %s %s (%s)\\n\", subs[smpath].sha1, smpath, subs[smpath].revname)\n+\t\t\telse\n+\t\t\t\tsubs[smpath].sha1 = string.gsub(run_cmd('cd ' .. smpath .. ' && git rev-parse --verify HEAD'), \"\\n\", \"\")\n+\t\t\t\tsubs[smpath][\"revname\"] = get_name_rev(smpath, subs[smpath].sha1)\n+\t\t\t\tfwrite(\"+%s %s (%s)\\n\", subs[smpath].sha1, smpath, subs[smpath].revname)\n+\t\t\tend\n+\t\tend\n+\tend\n+end\n+\n+if arg[1] == nil or arg[1] == 'status' then\n+\tcmd_status()\n+end\n-- \n1.8.3.1.381.g2ab719e.dirty\n"},{"id":"221040","messageId":"87a9mpgmx3.fsf@linux-k42r.v.cablecom.net","threadId":"34169","inReplyTo":"1371427503-9678-1-git-send-email-iveqy@iveqy.com","subject":"Re: [RFC] speed up git submodule","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-17T08:29:44Z","receivedAt":"2013-06-17T08:29:44Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Fredrik Gustafsson <iveqy@iveqy.com> writes:\n\n> The interesting feature would be to run C-functions direct inside lua. I\n> suppose that would increase speed even more, at the same time as we have\n> the convinence of a interpreted language. Lua is smaller and faster\n> (well as always, it depends on what you're doing) than python and ruby.\n> Perl is a really pain for the windows folks (I've heard).\n>\n> A correct implementation for lua support would be to start a\n> lua-interpreter from inside git.c (or somewhere) and load the lua code\n> for a specific command. That would make us independent of any target\n> installation of lua (althought the git binary would increase with the\n> lua library around 300 kb).\n>\n> However I did a quick test using lua as a replacement for sh (without\n> direct calls to c-functions) and the result is impressive. (However this\n> is the wrong way of using lua, shell scripting is not something lua is\n> good at).\n\nOk, so as you say, to really buy us anything you'd have to interface lua\nwith the C code directly.  Otherwise you might as well write it in Perl\ninstead which is already a requirement for a lot of the \"niceties\".\n\nHowever, instead of writing against git's C code, you could also\ninterface with libgit2, either from Lua or Perl...\n\n\nBTW Peff once posted an interface to Lua for the --pretty formatters:\n\n  http://thread.gmane.org/gmane.comp.version-control.git/206335\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"}]}