{"thread":{"id":"18881","subject":"integrating make and git","startedAt":"2009-04-15T15:19:31Z","lastAt":"2009-04-18T07:03:13Z","messageCount":14,"participants":["E R","Matthieu Moy","Daniel Barkalow","Robin Rosenberg","John Bito","Ben Jackson","David Kågedal","Jeff King","Nguyen Thai Ngoc Duy","Dmitry Potapov","Ealdwulf Wuffinga"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"111362","messageId":"3a69fa7c0904150819x7598dea5ic43bf0991c35ae45@mail.gmail.com","threadId":"18881","inReplyTo":null,"subject":"integrating make and git","fromName":"E R","fromEmail":"pc88mxer@gmail.com","sentAt":"2009-04-15T15:19:31Z","receivedAt":"2009-04-15T15:19:31Z","isPatch":false,"sender":{"key":"pc88mxer@gmail.com","avatar":null},"body":"I have an idea about integrating make with git, and I'm wondering if\nit is a reasonable thing to do.\n\nFirst of all, I am under the impression that git can quickly compute a\nhash of a directory and its contents. Is that correct?\n\nIf so, suppose you using git to manage revision control of a project\nwhich has some components like 'lib1', 'lib2', etc. Typically you\nwould perform something like: make clean; make all and 'make all'\nwould perform 'make lib1' and 'make lib2'. When checking out a\ndifferent revision of the project you would have to perform another\n'make clean' before 'make all' since you aren't sure of what's changed\nand the timestamps of the derived files will be more recent than the\ntimestamps of the source files.\n\nNow suppose that making 'lib1' only depends on the source code in a\ncertain directory. The idea is to associate the hash of the source\ndirectory for lib1 with its the derived files. Make can check this to\ndetermine if the component really needs to be rebuilt. Then as you\nmove around in the repository you can avoid rebuilding components that\nhaven't changed.\n\nGood, bad, ugly?\n"},{"id":"111367","messageId":"vpqbpqx6i22.fsf@bauges.imag.fr","threadId":"18881","inReplyTo":"3a69fa7c0904150819x7598dea5ic43bf0991c35ae45@mail.gmail.com","subject":"Re: integrating make and git","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2009-04-15T15:41:09Z","receivedAt":"2009-04-15T15:41:09Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"E R writes:\n\n> When checking out a\n> different revision of the project you would have to perform another\n> 'make clean' before 'make all' since you aren't sure of what's changed\n> and the timestamps of the derived files will be more recent than the\n> timestamps of the source files.\n\nThe last assumption is incorrect. git checkout will touch the files it\nmodifies, and won't play with timestamp precisely to save you from\nhaving to do \"make clean\" each time you use git.\n\n-- \nMatthieu\n"},{"id":"111370","messageId":"alpine.LNX.1.00.0904151148030.19665@iabervon.org","threadId":"18881","inReplyTo":"3a69fa7c0904150819x7598dea5ic43bf0991c35ae45@mail.gmail.com","subject":"Re: integrating make and git","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-04-15T16:20:00Z","receivedAt":"2009-04-15T16:20:00Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 15 Apr 2009, E R wrote:\n\n> I have an idea about integrating make with git, and I'm wondering if\n> it is a reasonable thing to do.\n> \n> First of all, I am under the impression that git can quickly compute a\n> hash of a directory and its contents. Is that correct?\n> \n> If so, suppose you using git to manage revision control of a project\n> which has some components like 'lib1', 'lib2', etc. Typically you\n> would perform something like: make clean; make all and 'make all'\n> would perform 'make lib1' and 'make lib2'. When checking out a\n> different revision of the project you would have to perform another\n> 'make clean' before 'make all' since you aren't sure of what's changed\n> and the timestamps of the derived files will be more recent than the\n> timestamps of the source files.\n\nNo, the timestamps of the changed source files will be newer than the \ntimestamps of the derived files. Git doesn't backdate files in working \ndirectories, in order to avoid causing the problem you're trying to fix. \n(And because getting the history is so quick and easy with git that \nlooking at dates on files in the filesystem is kind of pointless.)\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"111374","messageId":"3a69fa7c0904150947w25783199n6e304d7b4efcd051@mail.gmail.com","threadId":"18881","inReplyTo":"alpine.LNX.1.00.0904151148030.19665@iabervon.org","subject":"Re: integrating make and git","fromName":"E R","fromEmail":"pc88mxer@gmail.com","sentAt":"2009-04-15T16:47:52Z","receivedAt":"2009-04-15T16:47:52Z","isPatch":false,"sender":{"key":"pc88mxer@gmail.com","avatar":null},"body":"Ok - I was wrong about the timestamps not getting updated. Thanks for\nthat correction.\n\nHowever, what about the idea of associating the result of a build with\nthe hash of the source files used by the build, and using git to\ncompute the hash?\n\nOn Wed, Apr 15, 2009 at 11:20 AM, Daniel Barkalow wrote:\n> On Wed, 15 Apr 2009, E R wrote:\n>\n>> I have an idea about integrating make with git, and I'm wondering if\n>> it is a reasonable thing to do.\n>>\n>> First of all, I am under the impression that git can quickly compute a\n>> hash of a directory and its contents. Is that correct?\n>>\n>> If so, suppose you using git to manage revision control of a project\n>> which has some components like 'lib1', 'lib2', etc. Typically you\n>> would perform something like: make clean; make all and 'make all'\n>> would perform 'make lib1' and 'make lib2'. When checking out a\n>> different revision of the project you would have to perform another\n>> 'make clean' before 'make all' since you aren't sure of what's changed\n>> and the timestamps of the derived files will be more recent than the\n>> timestamps of the source files.\n>\n> No, the timestamps of the changed source files will be newer than the\n> timestamps of the derived files. Git doesn't backdate files in working\n> directories, in order to avoid causing the problem you're trying to fix.\n> (And because getting the history is so quick and easy with git that\n> looking at dates on files in the filesystem is kind of pointless.)\n>\n>        -Daniel\n> *This .sig left intentionally blank*\n>\n"},{"id":"111382","messageId":"200904151930.32816.robin.rosenberg.lists@dewire.com","threadId":"18881","inReplyTo":"3a69fa7c0904150947w25783199n6e304d7b4efcd051@mail.gmail.com","subject":"Re: integrating make and git","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-04-15T17:30:32Z","receivedAt":"2009-04-15T17:30:32Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"onsdag 15 april 2009 18:47:52 skrev E R :\n> Ok - I was wrong about the timestamps not getting updated. Thanks for\n> that correction.\n> \n> However, what about the idea of associating the result of a build with\n> the hash of the source files used by the build, and using git to\n> compute the hash?\n\nTake a look at ccache. It doesn't use Git, but it uses hashes of source, and\ncompiler flags and associates that with the resulting object files, so it\ncan avoid compiling. If you are building largs C/C++ (especially C++)\nprojects you want it. \n\n-- robin\n"},{"id":"111396","messageId":"alpine.LNX.1.00.0904151654330.10753@iabervon.org","threadId":"18881","inReplyTo":"3a69fa7c0904150947w25783199n6e304d7b4efcd051@mail.gmail.com","subject":"Re: integrating make and git","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-04-15T21:01:05Z","receivedAt":"2009-04-15T21:01:05Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 15 Apr 2009, E R wrote:\n\n> Ok - I was wrong about the timestamps not getting updated. Thanks for\n> that correction.\n> \n> However, what about the idea of associating the result of a build with\n> the hash of the source files used by the build, and using git to\n> compute the hash?\n\nIt's a reasonable idea, in general, but may or may not be useful for any \nparticular problem. In general, the objects in your subdirectories are \nalso doing to depend on some but not most things from an include \ndirectory, and so there's not much benefit you can get on a per-directory \ngranularity. On the other hand, I've gotten good results by embedding the \ncommit sha1 in generated object files, which allowed me to exactly \nidentify different builds much later, and even figure out what the source \nthat went into them was.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"111398","messageId":"3ae83b000904151434t5c3c85ebvec391ec29981ff79@mail.gmail.com","threadId":"18881","inReplyTo":"alpine.LNX.1.00.0904151654330.10753@iabervon.org","subject":"Re: integrating make and git","fromName":"John Bito","fromEmail":"jwbito@gmail.com","sentAt":"2009-04-15T21:34:33Z","receivedAt":"2009-04-15T21:34:33Z","isPatch":false,"sender":{"key":"jwbito@gmail.com","avatar":"https://gravatar.com/avatar/e94274c6b71a11e3fa03d91e75937609e66315f23267efd591ef1f98bb860bf9?d=mp&s=160"},"body":"If you're not already using make for a project, think before you\nstart. If you're building something that will go into distribution in\nsource form, you probably should need to use it (via automake &\nautoconf). For the stuff that I'm doing in a more focused\nenvironment, I use boost-build/bjam\n(http://www.boost.org/users/download/boost_jam_3_1_17). This provides\nvery clean organization of release/debug builds as well as a much more\nexpressive language than make.\n\nIt's easy to create makefiles that are quite brittle and the plumbing\nto establish portable builds is really complex. I was away from make\nworking on Java & Ruby for almost ten years. Though I have more than\n10 years of experience with make, I'm very happy to have replaced gobs\nof makefile code with the boost-build package and a few, short\nJamfiles.\n\nYMMV\nJohn\n"},{"id":"111414","messageId":"loom.20090416T034427-809@post.gmane.org","threadId":"18881","inReplyTo":"3a69fa7c0904150819x7598dea5ic43bf0991c35ae45@mail.gmail.com","subject":"Re: integrating make and git","fromName":"Ben Jackson","fromEmail":"ben@ben.com","sentAt":"2009-04-16T03:50:20Z","receivedAt":"2009-04-16T03:50:20Z","isPatch":false,"sender":{"key":"ben@ben.com","avatar":"https://gravatar.com/avatar/df49904dd23b03a5f57d9d53c0bf9fb6f69a14fac075c98f54f26cf1ce960794?d=mp&s=160"},"body":"E R gmail.com> writes:\n\n> Now suppose that making 'lib1' only depends on the source code in a\n> certain directory. The idea is to associate the hash of the source\n> directory for lib1 with its the derived files. Make can check this to\n> determine if the component really needs to be rebuilt.\n\nClearCase has \"wink-ins\" which are very much like this. It knows that a given\nobject was produced from a certain set of sources with a particular command. \nWhen someone wants to recreate that object (not even necessarily the original\nbuilder) it can \"wink in\" the result. Typically a brand new \"view\" (a ClearCase\nworking directory) build will consist of winking in a ton of objects rather than\nbuilding anything. I'm not sure how much of this is due to cleverness in\nclearmake and how much is due to the view being implemented as a virtual\nfilesystem (which can see every repository file being read as part of a build).\n\n--Ben\n"},{"id":"111430","messageId":"8763h5qazf.fsf@krank.kagedal.org","threadId":"18881","inReplyTo":"loom.20090416T034427-809@post.gmane.org","subject":"Re: integrating make and git","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2009-04-16T08:05:56Z","receivedAt":"2009-04-16T08:05:56Z","isPatch":false,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"Ben Jackson writes:\n\n> E R gmail.com> writes:\n>\n>> Now suppose that making 'lib1' only depends on the source code in a\n>> certain directory. The idea is to associate the hash of the source\n>> directory for lib1 with its the derived files. Make can check this to\n>> determine if the component really needs to be rebuilt.\n>\n> ClearCase has \"wink-ins\" which are very much like this. It knows that a given\n> object was produced from a certain set of sources with a particular command. \n> When someone wants to recreate that object (not even necessarily the original\n> builder) it can \"wink in\" the result. Typically a brand new \"view\" (a ClearCase\n> working directory) build will consist of winking in a ton of objects rather than\n> building anything. I'm not sure how much of this is due to cleverness in\n> clearmake and how much is due to the view being implemented as a virtual\n> filesystem (which can see every repository file being read as part of a build).\n\nIt very much depends on implementing its own file system, since it\notherwise would have no idea what the *real* build dependencies are.\n\nThat one of the nice things about clearmake, by the way. You don't\nhave to worry much about describing the dependencies, since it will\nfigure it out all by itself when you first build the project.\n\nBut I don't think there is much in CC for git to copy... :-)\n\n-- \nDavid Kågedal\n"},{"id":"111434","messageId":"20090416082615.GA27365@coredump.intra.peff.net","threadId":"18881","inReplyTo":"200904151930.32816.robin.rosenberg.lists@dewire.com","subject":"Re: integrating make and git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-04-16T08:26:15Z","receivedAt":"2009-04-16T08:26:15Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 15, 2009 at 07:30:32PM +0200, Robin Rosenberg wrote:\n\n> Take a look at ccache. It doesn't use Git, but it uses hashes of source, and\n> compiler flags and associates that with the resulting object files, so it\n> can avoid compiling. If you are building largs C/C++ (especially C++)\n> projects you want it. \n\nIn theory, one could improve something like ccache by asking git the\nsha-1 of the file. Since git maintains a cache based on stat info, you\ncan get away with not looking at the file contents at all (which saves\nCPU time in hashing, but also helps a lot when building from a cold\ncache).\n\nIn practice, this doesn't help because:\n\n 1. ccache looks at more than just the file itself. I believe it\n actually runs it through cpp and hashes that.\n\n 2. People combine ccache with make; if the stat data hasn't changed,\n in most cases, you will skip building before you even get to\n ccache.\n\nBut one could probably design a system to replace both ccache and make\nthat relies on git's fast sha-1 reporting to avoid duplicate work. I\nsuspect nobody has bothered because make+ccache is \"fast enough\" that\nthe added complexity would not be worth it.\n\n-Peff\n"},{"id":"111438","messageId":"vpq63h42a9y.fsf@bauges.imag.fr","threadId":"18881","inReplyTo":"20090416082615.GA27365@coredump.intra.peff.net","subject":"Re: integrating make and git","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2009-04-16T09:55:05Z","receivedAt":"2009-04-16T09:55:05Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Jeff King writes:\n\n> But one could probably design a system to replace both ccache and make\n> that relies on git's fast sha-1 reporting to avoid duplicate work. I\n> suspect nobody has bothered because make+ccache is \"fast enough\" that\n> the added complexity would not be worth it.\n\nAIUI, ClearCase does something similar to that. Call that an immense\nbloatware where everything has to come together, or a nice integration\nof different tools, I never used it, so I don't know (I heard the\nfirst option more than the second ...).\n\n-- \nMatthieu\n"},{"id":"111444","messageId":"fcaeb9bf0904160550u1a2911eai7e51ed92e6f6aa31@mail.gmail.com","threadId":"18881","inReplyTo":"vpq63h42a9y.fsf@bauges.imag.fr","subject":"Re: integrating make and git","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-04-16T12:50:39Z","receivedAt":"2009-04-16T12:50:39Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Apr 16, 2009 at 7:55 PM, Matthieu Moy wrote:\n> Jeff King writes:\n>\n>> But one could probably design a system to replace both ccache and make\n>> that relies on git's fast sha-1 reporting to avoid duplicate work. I\n>> suspect nobody has bothered because make+ccache is \"fast enough\" that\n>> the added complexity would not be worth it.\n>\n> AIUI, ClearCase does something similar to that. Call that an immense\n> bloatware where everything has to come together, or a nice integration\n> of different tools, I never used it, so I don't know (I heard the\n> first option more than the second ...).\n\nIt does help on really big projects (complete rebuild may take one\nday, complete recollect built objects and link them with clearmake\ntake about one hour). C++-based projects may like it due to long time\ncompilation.\n-- \nDuy\n"},{"id":"111510","messageId":"37fcd2780904171024v1b65d621q2725858e76358c36@mail.gmail.com","threadId":"18881","inReplyTo":"8763h5qazf.fsf@krank.kagedal.org","subject":"Re: integrating make and git","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-04-17T17:24:51Z","receivedAt":"2009-04-17T17:24:51Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Thu, Apr 16, 2009 at 12:05 PM, David Kågedal wrote:\n> Ben Jackson writes:\n>\n>> E R gmail.com> writes:\n>>\n>>> Now suppose that making 'lib1' only depends on the source code in a\n>>> certain directory. The idea is to associate the hash of the source\n>>> directory for lib1 with its the derived files. Make can check this to\n>>> determine if the component really needs to be rebuilt.\n>>\n>> ClearCase has \"wink-ins\" which are very much like this.  It knows that a given\n>> object was produced from a certain set of sources with a particular command.\n>> When someone wants to recreate that object (not even necessarily the original\n>> builder) it can \"wink in\" the result.  Typically a brand new \"view\" (a ClearCase\n>> working directory) build will consist of winking in a ton of objects rather than\n>> building anything.  I'm not sure how much of this is due to cleverness in\n>> clearmake and how much is due to the view being implemented as a virtual\n>> filesystem (which can see every repository file being read as part of a build).\n>\n> It very much depends on implementing its own file system, since it\n> otherwise would have no idea what the *real* build dependencies are.\n\nNot necessary... You can use LD_PRELOAD to intercept 'open' (and all\nneeded syscalls), but it works only on those platforms where LD_PRELOAD\nis supported. IIRC, there was some tool that did this, but I have never\nused it. I am pretty happy with ccache :)\n\nDmitry\n"},{"id":"111562","messageId":"efe2b6d70904180003k1ef1afdbi98e21193fb61895@mail.gmail.com","threadId":"18881","inReplyTo":"3a69fa7c0904150819x7598dea5ic43bf0991c35ae45@mail.gmail.com","subject":"Re: integrating make and git","fromName":"Ealdwulf Wuffinga","fromEmail":"ealdwulf@googlemail.com","sentAt":"2009-04-18T07:03:13Z","receivedAt":"2009-04-18T07:03:13Z","isPatch":false,"sender":{"key":"ealdwulf@googlemail.com","avatar":null},"body":"This idea sounds very much like vesta (http://www.vestasys.org) -\nexcept that vesta has its\nown version control system, rather than using git. It has its own\nfilesystem (actually a user-mode nfs server). It's GPL, so in theory\nyou could swipe its filesystem code, or replace its\nvcs with git. Don't know how hard that would be, though - doesn't\nsound trivial. It's worth looking at vesta for ideas, though, even if\nyou don't use the code. The owners are fairly responsive on their IRC\nchannel, too.\n\nEaldwulf\n"}]}