{"thread":{"id":"25449","subject":"Multi-target software repo, plus toolchain tracking","startedAt":"2010-10-14T17:21:42Z","lastAt":"2010-10-14T17:21:42Z","messageCount":1,"participants":["Jason Sewall"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"153526","messageId":"AANLkTineMhWPAMKondq9N26w3p6djaJ_xDyhj+yeawFv@mail.gmail.com","threadId":"25449","inReplyTo":null,"subject":"Multi-target software repo, plus toolchain tracking","fromName":"Jason Sewall","fromEmail":"jasonsewall@gmail.com","sentAt":"2010-10-14T17:21:42Z","receivedAt":"2010-10-14T17:21:42Z","isPatch":false,"sender":{"key":"jasonsewall@gmail.com","avatar":null},"body":"At $WORK, we're trying to figure out how to handle a combinatorial\nexplosion of research projects, backends, and big binary toolchains\nand I'm looking into leverage git for the problem. I'd appreciate the\nlist's advice; here's the basics (I've tried to make it as succinct as\npossible):\n\nWe have a suite composed of multiple sub-projects; each of these\nprojects supports some subset of a bunch of backend targets:\n\nrepo/\n      project1/\n\t\ttargetA/\n\t\ttargetB/\n\t\ttargetD/\n\t\t...\n      project2/\n\t\ttargetC/\n\t\ttargetD/\n\t\t...\n      project3/\n\t\ttargetB/\n\t\t...\n      ...\n\nThe complication is that most of these targets depend on some\n(possibly experimental) toolchain for compilation, and even execution\nin some cases.  We get these tools from upstream as big binary blobs\nwith timestamps.\n\nWe are very interested in tracking how performance changes with\nupdates in our code as well as changes in the toolchain, so it makes\nsense to collect some performance numbers with some code and package\nthose numbers along with info about which version of a given tool was\nused to compile/execute.  Naively, we could just put the tool right in\nthe repository like so:\n\nrepo/\n      project1/\n\t\ttargetA/\n\t\t\ttargetA-toolchain/\n\t\ttargetB/\n\t\t\ttargetB-toolchain/\n\t\t...\n      project3/\n\t\ttargetB/\n\t\t\ttargetB-toolchain/\n\t\t...\n      ...\n\nThe duplication of tools alone is pretty wasteful in disk space, but\nworse still, as we update tools and commit them, the size of the repo\n--- with all of these giant binary toolchains in it --- is likely to\nget pretty huge, and become a burden those who want to clone the repo\nand just work on the tip of some branch.\n\nSo what we really want is to have git make sure to grab the right\nversion of each toolchain whenever we move HEAD; I haven't got tons of\nexperience with submodules, but I think that they are well-suited to\nhandle this problem.\n\nThis is where I'd like to leverage the list's knowledge; there are a\nfew ways to handle this, and I'm not sure which is the best.  Again,\nwhat we want is:\n\n1. For a given checkout of repo/ to have each targetXXX-toolchain/\ndirectory populated, keeping in mind that different projects may not\nall be using the same version of a toolchain\n2. For end-users of the repo/ (cloners) to avoid having to lug around\nevery version of every toolchain; we don't want to 'pollute' the repo\nwith the large (>100mb) toolchain blobs we get from upstream.\n3. If possible, further minimize disk usage by using some sort of\nindirection (soft or hard linking, for example) when multiple projects\n*are* using the same version of a given toolchain.\n4. Minimize maintenance overhead\n\nHere are a few ideas I've had on the subject\n\nI. We could make a separate repo for each tool:\n\ntargetA-toolchain-repo/\ntargetB-toolchain-repo/\n\nand each targetXXX/ directory in the main repo's toolchain\nsubdirectory treats this as a submodule. This satisfies #1, #2, and\npossibly #4, but not #3.\n\nII. Make a separate repo for each tool, but instead of having multiple\nsubmodule checkouts, just have a tools directory with checkouts of all\ncurrently-needed tools, with symlinks from the\nprojectXXX/targetXXX/targetXXX-toolchain to these top-level\nsubmodules.\n\ni.e.\n\nrepo/\n      tools/\n            targetA-toolchain-ver-c0ffee/      <- submodule checkout\nof targetA-toolchain\n            targetB-toolchain-ver-deadbeef/ <- submodule checkout of\ntargetB-toolchain\n            targetB-toolchain-ver-b00b1e5/  <- submodule checkout of\ntargetB-toolchain\n      project1/\n\t\ttargetA/\n\t\t\ttargetA-toolchain/  <- symlink to ../../tools/targetA-toolchain-ver-c0ffee/\n\t\ttargetB/\n\t\t\ttargetB-toolchain/  <- symlink to ../../tools/targetB-toolchain-ver-deadbeef/\n\t\t...\n      project3/\n\t\ttargetB/\n\t\t\ttargetB-toolchain/  <- symlink to ../../tools/targetB-toolchain-ver-b00b1e5/\n\t\t...\n      ...\n\nThis is good on #1, #2, and #3, but has the complication that there's\na bit of maintenance involved - setting up symlinks, deleting unused\nsubmodules in tools/ etc.\n\nI'd really appreciate any comments/advice the list might have on this.\nI know my way around most of git pretty well, but I haven't really\nused submodules before.\n\nJason\n"}]}