{"thread":{"id":"38276","subject":"RFC: web UI for commit dependency inference tool","startedAt":"2015-01-04T01:08:03Z","lastAt":"2015-01-19T00:11:39Z","messageCount":2,"participants":["Adam Spiers"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"254272","messageId":"20150104010803.GD4108@pacific.linksys.moosehall","threadId":"38276","inReplyTo":null,"subject":"RFC: web UI for commit dependency inference tool","fromName":"Adam Spiers","fromEmail":"git@adamspiers.org","sentAt":"2015-01-04T01:08:03Z","receivedAt":"2015-01-04T01:08:03Z","isPatch":false,"sender":{"key":"git@adamspiers.org","avatar":"https://avatars.githubusercontent.com/u/100738?v=4"},"body":"Hi all,\n\nThanks to my employer's generous \"Hack Week\" policy[0], I have the\nluxury of being able to spend most of next week hacking on a git\ncommit dependency inference tool which I built 14 months ago but never\ngot round to polishing up or publically announcing.  In this email\nI'll briefly explain the tool and some ideas I have for adding a\nweb-based UI to it next week - any feedback is most welcome.\n\n[0] https://hackweek.suse.com/\n\nBackground theory\n=================\n\nIt is fairly clear that two git commits within a single repo can be\nconsidered \"independent\" from each other in a certain sense, if they\ndo not change the same files, or if they do not change overlapping\nparts of the same file(s).\n\nIn contrast, when a commit changes a line, it is \"dependent\" on not\nonly the commit which last changed that line, but also any commits\nwhich were responsible for providing the surrounding lines of context,\nbecause without those previous versions of the line and its context,\nthe commit's diff might not cleanly apply[1].  So all dependencies of\na commit can be programmatically inferred by running git-blame on the\nlines the commit changes, plus however many lines of context make\nsense for the use case of this particular dependency analysis.\n\nTherefore the dependency calculation is impacted by a \"fuzz\" factor\n(c.f. patch(1)) parameter, i.e. the number of lines of context which\nare considered necessary for the commit's diff to cleanly apply.\n\nAs with many dependency relationships, these dependencies form edges\nin a DAG (directed acyclic graph) whose nodes correspond to commits.\nNote that a node can only depend on a subset of its ancestors.\n\n[1] Depending on how it's being applied, of course.\n\nMotivation\n==========\n\nSometimes it is useful to understand the nature of parts of this DAG,\nas its nature will impact the success or failure of operations\nincluding merge, rebase, cherry-pick etc.\n\nFor example when porting a commit \"A\" between git branches via git\ncherry-pick, it can be useful to programmatically determine in advance\nthe minimum number of other dependent commits which would also need to\nbe cherry-picked to provide the context for commit \"A\" to cleanly\napply.\n\nAnother use case might be to better understand levels of specialism /\ncross-functionality within an agile team.  If I author a commit which\nmodifies (say) lines 34-37 and 102-109 of a file, the authors of the\ndependent commits forms a list which indicates the group of people I\nshould potentially consider asking to review my commit, since I'm\neffectively changing \"their\" code.  Monitoring those relationships\nover time might shed some light on how agile teams should best\ncoordinate efforts on shared code bases.\n\nI'm sure there are other use cases I haven't yet thought of.  At first\nI thought that it might provide a useful way to programmatically\npredict whether operations such as merge / rebase / cherry-pick would\nsucceed, but actually it's probably cheaper and more reliable simply\nto perform the operation and then roll back.\n\nBTW the dependency graph is likely to be semantically incomplete; for\nexample it would not auto-detect dependencies between a commit A which\nchanges code and another commit B which changes documentation or tests\nto reflect the code changes in commit A.  (Although of course it's\nusually best practice to logically group such changes together in a\nsingle commit.)  But this should not stop it from being useful.\n\nCurrent status\n==============\n\nI have written a tool called git-deps which automatically walks this\ngraph:\n\n    https://github.com/aspiers/git-config/blob/master/bin/git-deps\n\nI haven't yet documented it or formally announced it until now, but\nit's a single Python script, and usage is fairly self-explanatory:\n\n    $ git deps -h\n    usage: git-deps [options] COMMIT-ISH [COMMIT-ISH...]\n\n    Auto-detects commits which the given commit(s) depend on.\n\n    optional arguments:\n      -h, --help            show this help message and exit\n      -l, --log             Show commit logs for calculated dependencies\n                            [False]\n      -r, --recurse         Follow dependencies recursively [False]\n      -e COMMITISH, --exclude-commits COMMITISH\n                            Exclude commits which are ancestors of the given\n                            COMMITISH (can be repeated)\n      -c NUM, --context-lines NUM\n                            Number of lines of diff context to use [1]\n      -d, --debug           Show debugging [False]\n\nBy default it will list all dependencies of the given commit-ish(s),\nbut with --recurse it will one dependency (i.e. two SHA1s representing\na graph edge) per line.\n\nThere is still plenty of scope for optimization, e.g. it only takes\npartial advantage of pygit2.\n\nFuture plans\n============\n\n1. Interactive graph visualization\n\n   Currently the output is text only, but I think it would be more\n   useful to visualise the dependencies as an interactive graph where\n   you could zoom in/out, pan around, hover over each node to see\n   commit meta-data, click on leaf nodes to request further recursion,\n   and so on.  Nodes could be coloured according to commit author, and\n   sized according to the diffstats.\n\n   Clearly this should be cross-platform and based on some modern\n   rendering technology, so HTML/CSS/Javascript seems the obvious\n   choice.  Dependency inference is too expensive to generate the full\n   graph as a static web page, so I plan to extend the tool so it can\n   act as a lightweight web server, e.g.\n\n       $ git deps --web --port 8080\n\n   and then you could simply point your browser at http://localhost:8080\n   to interact with the graph.  It might look a little like this:\n\n       http://marvl.infotech.monash.edu/webcola/examples/downwardedges.html\n\n   but with interactive zoom/pan/hover/click functionality like this:\n\n       http://marvl.infotech.monash.edu/webcola/examples/onlinebrowse.html\n\n   Since a lot of the hard work is already done by cola.js in the above\n   examples, most likely I will use that in conjunction with d3.js for\n   rendering:\n\n       http://marvl.infotech.monash.edu/webcola/\n\n   Since the tool is already written in Python, I am considering using a\n   very lightweight web framework such as Flask:\n\n       http://flask.pocoo.org/\n\n   (I suspect Django would be overkill for this application which is\n   essentially stateless.)\n\n   Another approach might be to integrate it into an existing git web\n   frontend written in Python.  However I trawled through\n\n       https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Web_Interfaces\n\n   but couldn't find any Python-based frontend which looked like it was\n   in active development.  Perhaps the most promising I could find was:\n\n       http://git.kaarsemaker.net/goblet/\n\n2. Performance improvements\n\n   The tool should make better use of pygit2, since blame support was\n   not complete when it was originally written.  It also still forks\n   git-merge-base.\n\n3. Documentation\n\n4. Tests\n\n   Yes - embarrassing to admit I wrote this as a quick hack without\n   following TDD.  In my defence, I was doing it in coffee breaks at\n   an openSUSE conference ;-p\n\nRequest for feedback\n====================\n\nAny kind of feedback is very welcome - obviously sooner rather than\nlater, as my Hack Week starts on Monday.  Here's the project page:\n\n    https://hackweek.suse.com/11/projects/366\n\nMany thanks in advance!\nAdam\n"},{"id":"254871","messageId":"20150119001139.GA23327@pacific.linksys.moosehall","threadId":"38276","inReplyTo":"20150104010803.GD4108@pacific.linksys.moosehall","subject":"[ANNOUNCE] git-deps: commit dependency analysis / visualization","fromName":"Adam Spiers","fromEmail":"git@adamspiers.org","sentAt":"2015-01-19T00:11:39Z","receivedAt":"2015-01-19T00:11:39Z","isPatch":false,"sender":{"key":"git@adamspiers.org","avatar":"https://avatars.githubusercontent.com/u/100738?v=4"},"body":"On Sun, Jan 04, 2015 at 01:08:03AM +0000, Adam Spiers wrote:\n> Hi all,\n> \n> Thanks to my employer's generous \"Hack Week\" policy[0], I have the\n> luxury of being able to spend most of next week hacking on a git\n> commit dependency inference tool which I built 14 months ago but never\n> got round to polishing up or publically announcing.  In this email\n> I'll briefly explain the tool and some ideas I have for adding a\n> web-based UI to it next week - any feedback is most welcome.\n\n[snipped]\n\n> Request for feedback\n> ====================\n> \n> Any kind of feedback is very welcome - obviously sooner rather than\n> later, as my Hack Week starts on Monday.  Here's the project page:\n> \n>     https://hackweek.suse.com/11/projects/366\n> \n> Many thanks in advance!\n> Adam\n\nNoone told me this was a stupid idea, so I went ahead and added a web UI\nto the tool :-)\n\nI'm pleased to announce this is ready for testing; here are more\ndetails along with a short screencast demonstration:\n\n    http://blog.adamspiers.org/2015/01/19/git-deps/\n\nCheers,\nAdam\n"}]}