{"thread":{"id":"6024","subject":"updating only changed files source directory?","startedAt":"2006-10-24T01:33:18Z","lastAt":"2006-10-25T19:35:23Z","messageCount":8,"participants":["Han-Wen Nienhuys","Shawn Pearce","Jakub Narebski","Daniel Barkalow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"29795","messageId":"ehjqgf$ggb$1@sea.gmane.org","threadId":"6024","inReplyTo":null,"subject":"updating only changed files source directory?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2006-10-24T01:33:18Z","receivedAt":"2006-10-24T01:33:18Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"\nHello there,\n\nI'm just starting out with GIT. Initially, I want to use experiment \nwith integrating it into our binary builder structure for LilyPond.\n\nThe binary builder roughly does this:\n\n 1. get source code updates from a server to a single, local\n repository. This is currently a git repository that is that\n tracks our CVS server.\n\n 2. copy latest commit from a branch to separate source directory.\n This copy should only update files that changed.\n\n 3. Incrementally compile from that source directory\n\nThe binary builder does this for several branches and several\nplatforms of the project. Due to parallel compilation, it might even\nbe possible that different branches of are being checked out\nconcurrently from a single repository.\n\nFor a VCS, this is slightly nonstandard use, as we don't do any work\nin the working dir, we just compile from it, but have many working\ndirectories.\n\n\nI have some questions and remarks\n\n* Is there a command analogous to git-clone for updating a repository?\nRight now, I'm using a combination of\n\n git-http-fetch -a \n wget /refs/head/ ## dump to /refs/head/\n\nfor all branches I want to know about. I was looking for a command\nthat would update the heads of all branches.\n\n\n* Why is the order of args in git-http-fetch inconsistent with the\norder in git-fetch? in fetch, the repository comes first, in\nhttp-fetch, it comes last\n\n\n* How do I update a source directory?\n\nI can do the following\n\n git --git-dir read-tree \n\n cd \n git --git-dir checkout-index -a -f\n\nUnfortunately, this touches all files, which messes up the timestamps\ntriggering needless recompilation. How can I make checkout-index only\ntouch files that have changed? Or alternatively, make checkout-index\nremember timestamps on files that didn't change?\n\nOf course, I can store the commitish of the last version of the\nsrcdir, and apply the diff between both to the source directory, but \nthat seems somewhat convoluted. Is there a better way?\n\n\n* As far as I can see, there is no reason to have only one index in a\ngit repository. Why isn't it possible to specify an alternate\nindex-file with an option similar to --git-dir ?\n\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"29811","messageId":"20061024055539.GD9724@spearce.org","threadId":"6024","inReplyTo":"ehjqgf$ggb$1@sea.gmane.org","subject":"Re: updating only changed files source directory?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-10-24T05:55:39Z","receivedAt":"2006-10-24T05:55:39Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Han-Wen Nienhuys wrote:\n> For a VCS, this is slightly nonstandard use, as we don't do any work\n> in the working dir, we just compile from it, but have many working\n> directories.\n\nIts not nonstandard use. A lot of projects perform rolling builds\nwhich trigger anytime there are changes; very active projects\nwould always be building and thus would always want to have the\nVCS only update those files which actually changed, to minimize\nthe compile time.\n \n> I have some questions and remarks\n> \n> * Is there a command analogous to git-clone for updating a repository?\n> Right now, I'm using a combination of\n\nYes, its called git-fetch and git-pull. Which leads us to...\n \n> git-http-fetch -a \n> wget /refs/head/ ## dump to /refs/head/\n> \n> for all branches I want to know about. I was looking for a command\n> that would update the heads of all branches.\n\nWhy not use git-fetch?\n\nCreate a .git/remotes file named 'origin' and put in there the URL\nyou want to fetch from and the list of branches you want to download\nand keep current.\n\nThen downloading the changes to the build repository is as simple\nas running `git-fetch` with no parameters (as it defaults to reading\nthe origin file).\n\n> * How do I update a source directory?\n\nAlways keep the source directory on a branch that is not listed\nin the .git/remotes/origin file. This way the fetch will always\nsucceed without failure.\n\nThen you can do after the fetch:\n\n\tgit-reset --hard \n\nand the source directory will be updated to (which\ncould just be a branch name of one of those branches you fetch,\nor could be a full SHA1, or a tag, etc.).\n\nThe reset --hard process will only change the files that really have\nto change. This means it will run in linear time proportional to the\nnumber of files needing to be updated; and only those files which are\ndifferent between the working directory and will have\nnew modification dates. Therefore incremental rebuilds will work.\n \n> * As far as I can see, there is no reason to have only one index in a\n> git repository. Why isn't it possible to specify an alternate\n> index-file with an option similar to --git-dir ?\n\nThe index is key to getting the fast update of the working directory.\nYou can change the index with the (rather undocuments) GIT_INDEX_FILE\nenvironment variable. I do this in a few tools I have written\naround Git, but I don't do it very often.\n\n-- \nShawn.\n"},{"id":"29823","messageId":"ehkgfs$af6$1@sea.gmane.org","threadId":"6024","inReplyTo":"ehjqgf$ggb$1@sea.gmane.org","subject":"Re: updating only changed files source directory?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-24T07:48:39Z","receivedAt":"2006-10-24T07:48:39Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Han-Wen Nienhuys wrote:\n\n> I have some questions and remarks\n\nI see that you are using fairly low level commands (plumbing commands)\n \n> git-http-fetch -a \n> wget /refs/head/ ## dump to /refs/head/\n\ninstead of setting $GIT_DIR/remotes/origin file and using \"git fetch\".\nBTW. \"git fetch\" will not update branch you are on, unless --update-head-ok\noption is used.\n\n> git --git-dir read-tree \n> \n> cd \n> git --git-dir checkout-index -a -f\n\ninstead of \n git --git-dir= checkout \n(-f is Force a re-read of everything)\n \n> * As far as I can see, there is no reason to have only one index in a\n> git repository. Why isn't it possible to specify an alternate\n> index-file with an option similar to --git-dir ?\n\n--git-dir is alternative to setting GIT_DIR. You can use GIT_INDEX_FILE\nto specify alternate index file. Documented in git(7), section\n\"ENVIRONMENT VARIABLES\".\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"29840","messageId":"453DE1F5.5010803@xs4all.nl","threadId":"6024","inReplyTo":"ehkgfs$af6$1@sea.gmane.org","subject":"Re: updating only changed files source directory?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2006-10-24T09:50:45Z","receivedAt":"2006-10-24T09:50:45Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Jakub Narebski escreveu:\n> Han-Wen Nienhuys wrote:\n> \n>> I have some questions and remarks\n> \n> I see that you are using fairly low level commands (plumbing commands)\n> \n>> git-http-fetch -a \n>> wget /refs/head/ ## dump to /refs/head/\n> \n> instead of setting $GIT_DIR/remotes/origin file and using \"git fetch\".\n> BTW. \"git fetch\" will not update branch you are on, unless --update-head-ok\n> option is used.\n\nI tried fetch, but was put off by the warnings because I didn't have \n--update-head-ok. Using lowlevel commands is my way of making sure that \nGit doesn't assume it needs to do anything intelligent.\n\n>> git --git-dir read-tree \n>>\n>> cd \n>> git --git-dir checkout-index -a -f\n> \n> instead of \n> git --git-dir= checkout \n> (-f is Force a re-read of everything)\n\nYes, however,\n\n checkout\n\nchanges the state of the repository, which is something I want to prevent.\n\n>> * As far as I can see, there is no reason to have only one index in a\n>> git repository. Why isn't it possible to specify an alternate\n>> index-file with an option similar to --git-dir ?\n> \n> --git-dir is alternative to setting GIT_DIR. You can use GIT_INDEX_FILE\n> to specify alternate index file. Documented in git(7), section\n> \"ENVIRONMENT VARIABLES\".\n\nSilly me, I overlooked in the manpage. Note that it is standard to put \nthe environment section at the end of the manpage. Right now it's \nsomewhere in the middle.\n\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"29843","messageId":"200610241213.38558.jnareb@gmail.com","threadId":"6024","inReplyTo":"453DE1F5.5010803@xs4all.nl","subject":"Re: updating only changed files source directory?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-24T10:13:38Z","receivedAt":"2006-10-24T10:13:38Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Han-Wen Nienhuys wrote:\n> Jakub Narebski escreveu:\n>> Han-Wen Nienhuys wrote:\n>> \n>> I see that you are using fairly low level commands (plumbing commands)\n>> \n>>> git-http-fetch -a \n>>> wget /refs/head/ ## dump to /refs/head/\n>> \n>> instead of setting $GIT_DIR/remotes/origin file and using \"git fetch\".\n>> BTW. \"git fetch\" will not update branch you are on, unless --update-head-ok\n>> option is used.\n> \n> I tried fetch, but was put off by the warnings because I didn't have \n> --update-head-ok. Using lowlevel commands is my way of making sure that \n> Git doesn't assume it needs to do anything intelligent.\n\nYou can either have additional branch which is not tracking branch\n(you don't fetch into this branch), and on which you are always on,\ncalled for example 'check-out' (and which can be used for git-reset\nsolution to checking out files to external directory), and use\ngit-fetch without --update-head-ok, or (if the repository is bare\nrepository, without working area) use --update-head-ok.\n \n>>> git --git-dir read-tree \n>>>\n>>> cd \n>>> git --git-dir checkout-index -a -f\n>> \n>> instead of \n>> git --git-dir= checkout \n>> (-f is Force a re-read of everything)\n\ngit-checkout-index(1):\n\n -f|--force\n forces overwrite of existing files\n\nSo probably you would get what you want if you lose '-f'.\n\n> Yes, however,\n> \n> git checkout\n> \n> changes the state of the repository, which is something I want to prevent.\n\nWell, git-reset also changes state of repository, but it changes only\nthe branch we have created exactly for this purpose.\n-- \nJakub Narebski\nPoland\n"},{"id":"295002","messageId":"Pine.LNX.4.64.0610241435420.9789@iabervon.org","threadId":"6024","inReplyTo":"ehjqgf$ggb$1@sea.gmane.org","subject":"Re: updating only changed files source directory?","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-10-24T19:12:41Z","receivedAt":"2006-10-24T19:12:41Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 24 Oct 2006, Han-Wen Nienhuys wrote:\n\n> \n> Hello there,\n> \n> I'm just starting out with GIT. Initially, I want to use experiment with\n> integrating it into our binary builder structure for LilyPond.\n> \n> The binary builder roughly does this:\n> \n> 1. get source code updates from a server to a single, local\n> repository. This is currently a git repository that is that\n> tracks our CVS server.\n> \n> 2. copy latest commit from a branch to separate source directory.\n> This copy should only update files that changed.\n> \n> 3. Incrementally compile from that source directory\n\nThe terminology in the git world is, I think, a little different from what \nyou expect. We call the thing that contains all of the tracked information \n(what you're calling the repository) the \"object database\"; what we call \nthe \"repository\" is a bit different: it primarily keeps track of the heads \nof branches, in addition to either containing an object database or \nreferencing an external one. So you need a repository for each source \ndirectory (because it keeps track of what commit is currently in the \nsource directory), but it doesn't need to have its own complete object \ndatabase, which is what you're trying to share between all of them.\n\nYou have a single repository with no source directory that contains the \ndatabase and the heads according to the upstream source, and then each \nsource directory has a repository that contains the head as far as you've \nbuilt it in that directory. You fetch into the single bare repository \nfrom upstream, and then pull into each source directory from the bare \nrepository; this will do the minimal update to the contents of the source \ndirectory automatically.\n\nI think that you want to request a few git features:\n\n - support having a bare repository not on a branch, so that it can fetch \n all heads from its upstream. You're not doing anything branch-specific \n in the bare repository anyway, but git currently wants a valid HEAD to\n accept a path as containing a git repository\n\n - support getting an origin remote configuration with a bare repository\n\n - support cloning a branch of a repository, such that the clone's \n \"origin\" is the upstream's chosen branch, not its \"master\".\n\n - support cloning without generating a \"master\" branch in the clone, and \n instead starting on \"origin\"\n\nThen you do:\n\ngit clone --bare --no-head --with-origin REPOSITORY.git\n\nfor each branch:\n\n git clone --shared --branch= --no-master REPOSITORY.git \n\nWhen you want to update:\n\nGIT_DIR=REPOSITORY.git git fetch\n\nfor each branch:\n\n (cd ; git pull; make)\n\nNote that all of the features you need are in \"clone\" for setting things \nup nicely automatically; if you arrange everything by hand just right, you \ncan already to the updating procedure I give.\n\n\t-Daniel\n"},{"id":"295618","messageId":"453F517A.7060000@xs4all.nl","threadId":"6024","inReplyTo":"Pine.LNX.4.64.0610241435420.9789@iabervon.org","subject":"Re: updating only changed files source directory?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2006-10-25T11:58:50Z","receivedAt":"2006-10-25T11:58:50Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Daniel Barkalow escreveu:\n>> I'm just starting out with GIT. Initially, I want to use experiment with\n>> integrating it into our binary builder structure for LilyPond.\n>>\n>> The binary builder roughly does this:\n>>\n>> 1. get source code updates from a server to a single, local\n>> repository. This is currently a git repository that is that\n>> tracks our CVS server.\n>>\n>> 2. copy latest commit from a branch to separate source directory.\n>> This copy should only update files that changed.\n>>\n>> 3. Incrementally compile from that source directory\n> \n> The terminology in the git world is, I think, a little different from what \n> you expect. We call the thing that contains all of the tracked information \n> (what you're calling the repository) the \"object database\"; what we call \n\nyes, you hit the nail on the head.\n\n> referencing an external one. So you need a repository for each source \n> directory (because it keeps track of what commit is currently in the \n> source directory), but it doesn't need to have its own complete object \n> database, which is what you're trying to share between all of them.\n\nOK. This makes sense; thanks for this pointer.\n\nHow can I set the object database? I found GIT_OBJECT_DIRECTORY, but \ncan I write a config file entry for that?\n\n> built it in that directory. You fetch into the single bare repository \n> from upstream, and then pull into each source directory from the bare \n> repository; this will do the minimal update to the contents of the source \n> directory automatically.\n\nyes, this works. Thanks!\n\n-- \n"},{"id":"294383","messageId":"Pine.LNX.4.64.0610251525060.9789@iabervon.org","threadId":"6024","inReplyTo":"453F517A.7060000@xs4all.nl","subject":"Re: updating only changed files source directory?","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-10-25T19:35:23Z","receivedAt":"2006-10-25T19:35:23Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 25 Oct 2006, Han-Wen Nienhuys wrote:\n\n> How can I set the object database? I found GIT_OBJECT_DIRECTORY, but can I\n> write a config file entry for that?\n\nIf you clone with --shared, it'll do the right thing automatically, which \nis to have the clone's .git/objects/info/alternates be the objects \ndirectory of the bare repository.\n\n(Note that any new objects you create in the clone go into the clone's own \nobjects database. This shouldn't matter for you, unless your build system \nis tagging things or something, but if you end up doing development in a \nsimilarly structured system, it's worth knowing that this doesn't affect \nthe bare repository at all.)\n\n> yes, this works. Thanks!\n\nNo problem. :)\n\n\t-Daniel\n"}]}