{"thread":{"id":"25707","subject":"Looking for a way to set up Git correctly","startedAt":"2010-11-11T03:25:31Z","lastAt":"2010-11-11T19:38:59Z","messageCount":5,"participants":["Dennis","Alex Riesen","Enrico Weigelt","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"155644","messageId":"BD94CB4FDD0C4462804E316F814A3CCA@denny","threadId":"25707","inReplyTo":null,"subject":"Looking for a way to set up Git correctly","fromName":"Dennis","fromEmail":"denny@dennymagicsite.com","sentAt":"2010-11-11T03:25:31Z","receivedAt":"2010-11-11T03:25:31Z","isPatch":false,"sender":{"key":"denny@dennymagicsite.com","avatar":null},"body":"I have a situation.\n\nI have started a web project (call it branch1), and have maintained it \nwithout a version control system for quite some time.\nThen, I copied it to another folder (branch2) and while the project remained \nessentially the same, I have changed a few of internal paths and some \nvariable names inside the files.\nThen, a few months later on, I copied branch2 to a folder called branch3 and \nalso modified some of the variable names and some of the internal structure \nof the files.\n\nThus I ended up with 3 folders on my local HDD with pretty much the same \nfile names and folder structure and everything, and most of the file \ncontent, except those small deltas that made those files different for each \nbranch.\n\nI guess it's never too late, and now I want to put these 3 projects into a \nversion control system, and I chose git.\n\nNow, this can be either really simple or really complicated.  My first \nquestion is:  how do I set the repository up in the proper way where I could \nwork on all 3 projects separately, with additional possibility of working on \nbranch1 only and later committing my changes to branch2 and branch3.  (Since \nprojects are virtually identical, a fix in one branch usually needs to be \npropagated to other branches)\nFirst, I assume I will use a single repository for this.  Then, do I simply \nset up 3 branches and start using them, or is there a way to set git up to \ncapitalize on the projects being nearly identical?\n\nMy second question is that each branch has a huge folder with image data. \nBy huge I mean 1 to 4Gb, depending on the branch.  Since images are not \ndirectly relevant to the development work, is there a way to not include \nthose folders in git?  To be honest though, I probably should include them, \nbut I wanted to ask about this separately as git repository may be get \nlarge, since all 3 branches may grow to 9Gb or so.\n\nThus I am looking for a git way to handle my situation.  Is this simple or \nis is hard?\nAre there any recommendations before I jump in?\nDennis \n"},{"id":"155658","messageId":"AANLkTi=CTR-i2v-hi6dp_OAOf6Y7UO2giQq_Uq-tDs=M@mail.gmail.com","threadId":"25707","inReplyTo":"BD94CB4FDD0C4462804E316F814A3CCA@denny","subject":"Re: Looking for a way to set up Git correctly","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2010-11-11T09:38:47Z","receivedAt":"2010-11-11T09:38:47Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On Thu, Nov 11, 2010 at 04:25, Dennis <denny@dennymagicsite.com> wrote:\n> I have started a web project (call it branch1), and have maintained it\n> without a version control system for quite some time.\n> Then, I copied it to another folder (branch2) and while the project remained\n> essentially the same, I have changed a few of internal paths and some\n> variable names inside the files.\n> Then, a few months later on, I copied branch2 to a folder called branch3 and\n> also modified some of the variable names and some of the internal structure\n> of the files.\n>\n> Thus I ended up with 3 folders on my local HDD with pretty much the same\n> file names and folder structure and everything, and most of the file\n> content, except those small deltas that made those files different for each\n> branch.\n>\n> I guess it's never too late, and now I want to put these 3 projects into a\n> version control system, and I chose git.\n>\n> Now, this can be either really simple or really complicated.  My first\n> question is:  how do I set the repository up in the proper way where I could\n> work on all 3 projects separately, with additional possibility of working on\n> branch1 only and later committing my changes to branch2 and branch3.  (Since\n> projects are virtually identical, a fix in one branch usually needs to be\n> propagated to other branches)\n> First, I assume I will use a single repository for this.  Then, do I simply\n> set up 3 branches and start using them, or is there a way to set git up to\n> capitalize on the projects being nearly identical?\n\nAssuming I've got the relationships of your \"branches\" right:\n\n$ cp -a branch1 branch && cd branch\n$ git init\n$ echo /huge-images/ >.gitignore\n$ git add .gitignore; git add .; git commit; git branch branch1\n$ git checkout -b branch2\n$ cp -a ../branch2 .\n$ git add .; git commit\n$ git checkout -b branch3\n$ cp -a ../branch3 .\n$ git add .; git commit\n\n> My second question is that each branch has a huge folder with image data. By\n> huge I mean 1 to 4Gb, depending on the branch.  Since images are not\n> directly relevant to the development work, is there a way to not include\n> those folders in git?  To be honest though, I probably should include them,\n> but I wanted to ask about this separately as git repository may be get\n> large, since all 3 branches may grow to 9Gb or so.\n>\n> Thus I am looking for a git way to handle my situation.  Is this simple or\n> is is hard?\n\nIf you add the images you will eventually run into problems (heavy\nswapping, for one).\nGit is not really setup to work with big binary files (a file must fit into\nmemory completely).\n"},{"id":"155686","messageId":"20101111132502.GB23423@nibiru.local","threadId":"25707","inReplyTo":"BD94CB4FDD0C4462804E316F814A3CCA@denny","subject":"Re: Looking for a way to set up Git correctly","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-11T13:25:02Z","receivedAt":"2010-11-11T13:25:02Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Dennis <denny@dennymagicsite.com> wrote:\n\nHi,\n\n> Now, this can be either really simple or really complicated.  My first \n> question is:  how do I set the repository up in the proper way where I \n> could work on all 3 projects separately, with additional possibility of \n> working on branch1 only and later committing my changes to branch2 and \n> branch3.  \n\nAs first step you could create 3 separate git repos in each directory\nand add everything to it (git init, git add -A, git commit). Then \nrename the branches properly (so instead of \"master\", they'll be called\n\"branch1\", \"branch2\", \"branch2\" or something like that). Create another\n(maybe bare) repo elsewhere, add it as remote to the three other ones\nand push their branches upwards. Now you have 4 repos, 3 for working\non the individual branches and another for collecting them all (hub model).\nYou could also choose to throw the first three away and only work in\nthe last one.\n\n> (Since projects are virtually identical, a fix in one branch \n> usually needs to be propagated to other branches)\n\nIn your case, cherry-pick might be the right for you.\nYou could also do a little bit refactoring, making a 4th branch which\nthe other 3 are then rebased onto. Then you could do your fixes in that\nbranch and merged into or rebase the other 3 onto that one.\n\n> My second question is that each branch has a huge folder with image data. \n> By huge I mean 1 to 4Gb, depending on the branch.  Since images are not \n> directly relevant to the development work, is there a way to not include \n> those folders in git?\n\nsee .gitignore file.\nnevertheless it might be useful to also have all the images in the\nrepo for backup reasons.\n\nBTW: if you're concerned about disk space, you could add the object dir\nof the 4th (hub) repository to the 3 working repos (run git-gc in the\nhub repo before that!). Next gc runs will remove the objects that are\nalready present in the hub. But beware! If you remove something in the\nhub repo and run git-gc there, you could loose objects in the other repos!\n(maybe it would be wise to add the 3 working repos as remotes in the\nhub and always run an git remote update before git-gc in the hub).\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone:  +49 36207 519931  email: weigelt@metux.de\n mobile: +49 151 27565287  icq:   210169427         skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"155705","messageId":"20101111164606.GA16972@burratino","threadId":"25707","inReplyTo":"20101111132502.GB23423@nibiru.local","subject":"Re: Looking for a way to set up Git correctly","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-11T16:46:06Z","receivedAt":"2010-11-11T16:46:06Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(+cc: Dennis again, Alex)\n\nHi,\n\nEnrico Weigelt wrote:\n> * Dennis <denny@dennymagicsite.com> wrote:\n\n>> Now, this can be either really simple or really complicated.  My first \n>> question is:  how do I set the repository up in the proper way where I \n>> could work on all 3 projects separately, with additional possibility of \n>> working on branch1 only and later committing my changes to branch2 and \n>> branch3.  \n>\n> As first step you could create 3 separate git repos in each directory\n> and add everything to it (git init, git add -A, git commit). Then \n> rename the branches properly (so instead of \"master\", they'll be called\n> \"branch1\", \"branch2\", \"branch2\" or something like that). Create another\n> (maybe bare) repo elsewhere, add it as remote to the three other ones\n> and push their branches upwards.\n\nSo this looks like so:\n\n\tfor i in project1 project2 project3\n\tdo\n\t\t(\n\t\t\tcd \"$i\"\n\t\t\tgit init\n\t\t\tgit add .\n\t\t\tgit commit\n\t\t)\n\tdone\n\tgit init main\n\tcd main\n\tfor i in project1 project2 project3\n\tdo\n\t\tgit fetch ../$i master:$i\n\tdone\n\tmv project1 project2 project3 away/\n\nIf you would like multiple worktrees (one for each branch, maybe) for\nthe main repo, you might want to look into the new-workdir script in\ncontrib/workdir (but do consider the caveats[1]).\n\n>> (Since projects are virtually identical, a fix in one branch \n>> usually needs to be propagated to other branches)\n>\n> In your case, cherry-pick might be the right for you.\n\ne.g., when project3 gets a new fix:\n\n\tgit checkout project1\n\tgit cherry-pick project3\n\n> You could also do a little bit refactoring, making a 4th branch which\n> the other 3 are then rebased onto.\n\nRight, what is the actual relationship between these projects?  Do\nthey actually represent branches in the history of a single project?\n\nSuppose project1 is historically an ancestor to project2, project3,\nand project4, which are independent.  (Maybe project1 is the initial\nversion and projects 2,3,4 are ports to other platforms.)  You could\ntake this into account when initially setting up the branches, like\nthis:\n\n\tgit init main\n\tcd main\n\tGIT_DIR=$(pwd)/.git; export GIT_DIR\n\tGIT_WORK_TREE=../project1 git add .\n\tGIT_WORK_TREE=../project1 git commit\n\tgit branch -m project1\n\tfor i in project2 project3 project4\n\tdo\n\t\tgit checkout -b $i project1\n\t\tGIT_WORK_TREE=../$i git add -A\n\t\tGIT_WORK_TREE=../$i git commit\n\tdone\n\n(and use gitk --all when done to make sure everything looks right)\n\nAlternatively, you can rearrange the history afterwards:\n\n\t$ git cat-file commit project2 | tee project2\n\ttree 76db51024713f6ef191928a8445d48d39ab55434\n\tauthor Junio C Hamano <gitster@pobox.com> 1289324716 -0800\n\tcommitter Junio C Hamano <gitster@pobox.com> 1289324716 -0800\n\n\tproject2: an excellent project\n\t$ git rev-parse project1\n\t$ vi project2\n\t... add a \"parent <object id>\" line\n\t    after the tree line,\n\t    where <object id> is the full object name rev-parse printed ...\n\t$ git hash-object -t commit -w project2\n\t$ git branch -f branch2 <the object name hash-object prints>\n\t... repeat for project3 and project4 ...\n\t$ gitk --all;\t\t# to make sure everything looks right\n\nThis is less convenient than it ought to be.  It would be nice to add\na \"git graft\" command to automate this procedure, which\n\n - interacts well with \"git replace\"\n - doesn't interact poorly with \"git fetch\" like .git/info/grafts does\n - could be more convenient to use than .git/info/grafts.\n\nAs the gitworkflows man page mentions, if you make your fixes on the\noldest branch they apply to (project1) and then merge to all later\nbranches, then the fixes will propagate forward correctly.  See the\n\"Graduation\" and \"Merging upwards\" sections of gitworkflows for details.\n\n>> My second question is that each branch has a huge folder with image data. \n>> By huge I mean 1 to 4Gb, depending on the branch.  Since images are not \n>> directly relevant to the development work, is there a way to not include \n>> those folders in git?\n\nI would suggest tracking a symlink to another repository (or to a\ndirectory tracked through other means, like unison).\n\nHope that helps,\nJonathan\n\n[1] If you have two worktrees for the same project with the\nsame branch checked out at a given moment, the results can be\nconfusing (changes made in one worktree will look like they have\nbeen commited and undone in the other).\n\nThe \"detached HEAD\" feature (which git-checkout.1 explains) and\nmultiple worktrees do not interact so well: the need to preserve\ncommits while no branch was checked out in one worktree will not be\ntaken into account when \"git gc\" runs (explicitly or implicitly!) on\nthe other.  This can be very disconcerting.\n"},{"id":"155735","messageId":"20101111193859.GI16972@burratino","threadId":"25707","inReplyTo":"20101111190724.00vcimqm8w0cw8s0@dennymagicsite.com","subject":"Re: Looking for a way to set up Git correctly","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-11T19:38:59Z","receivedAt":"2010-11-11T19:38:59Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"denny@dennymagicsite.com wrote:\n\n> I am still looking through your replies and getting familiar with\n> git commands.\n\nBy the way, please ignore that GIT_WORK_TREE stuff I did.  It\nprobably works, but it's ugly. :)  That example could have been\nwritten better as\n\n\tgit init everything\n\tGIT_DIR=$(pwd)/everything/.git; export GIT_DIR\n\t(\n\t\tcd common-ancestor\n\t\tgit add -A\n\t\tgit commit\n\t\tgit branch -m ancestor\n\t)\n\t(\n\t\tcd project1\n\t\tgit checkout -b project1 ancestor\n\t\tgit add -A\n\t\tgit commit\n\t)\n\t... etc ..\n\tunset GIT_DIR\n\n\tcd everything\n\tgit checkout project1\n\n[...]\n> From a developer's point of view, working on projectX means making\n> some changes and committing them to the repo for that project.  The\n> developer may not be aware of other pojects existing.\n\nFor concreteness, I am imagining these directories represent various\nversions of the Almquist shell.  The common ancestor is the BSD4.3/Net-2\nversion and various projects may have built from there in different\ndirections: NetBSD sh, FreeBSD sh, dash.  (Yes, I am oversimplifying. :))\n\nNow suppose they have diverged so wildly that it is never possible to\nsynchronize code with each other.  Instead, they can copy fixes, and\nthis is especially convenient when the fixes are phrased as diffs to\nthe common ancestor.\n\nTo facilitate this, Alice revives the BSD4.3/Net-2 sh project with a\n\"fixes only\" policy.  Her daily work might look like this:\n\n $ git fetch netbsd\n $ git log netbsd/for-alice@{1}..netbsd/for-alice; # any good patches today?\n $ git cherry-pick -s 67fd89980; # a good patch.\n ... quick test ...\n $ git cherry-pick -s 897ac8; # another good patch.\n ... quick test ...\n ...\n $ git fetch freebsd\n ... and similarly for the rest of the patch submitters ...\n $ git am emailed-patch\n\nThen to more thoroughly test the result:\n\n $ git checkout -b throwaway;\t# new throw-away branch.[1]\n $ git merge netbsd/master;\t# will the changes work for netbsd?\n ... thorough test ...\n $ git reset --keep master\n $ git merge freebsd/master;\t# how about freebsd?\n ... etc ...\n\nAnd finally she pushes the changes out.\n\n> Without knowing anything about git for a moment, one ideal workflow\n> is where a developer makes changes to projectX that touch the base\n> and projectX specific features.  Then the developer commits them and\n> pushes them to the main repo.  The main repo contains all projects.\n> During the commit, chages to the base automagically get pushed to\n> all projects that share that base\n\nIf it is a matter of what files are touched, then maybe the base is\nactually something like a library, which should be managed as a\nseparate project.  See the \"git submodule\" manual if you would like to\ntry something like this but still keep the projects coupled.\n\nOn the other hand, remaining in the situation from before:\n\nSuppose Sam is the NetBSD sh maintainer.  The first step in working on\na new release might be\n\n $ git fetch ancestor\n $ git log -p HEAD..FETCH_HEAD;\t# fixes look okay?\n $ git pull ancestor\n\nsince Alice tends to include only safe, well tested fixes.\n\nMany changes Sam makes are specific to his project, but today he comes\nup with a fix that might be useful for other ash descendants.\n\nSo instead of commiting directly, he can try:\n\n $ git checkout for-alice;\t# carry the fix to the for-alice branch\n ... test ...\n $ git commit -a;\t\t# commit it.\n\nIf it is not an urgent fix, at this point he might do\n\n $ git checkout master;\t\t# back to the main NetBSD branch, without the fix\n\nand give the other projects some time to work on the patch and come up\nwith a better fix.  Or he might cherry-pick the commit from for-alice,\nand even publish it and encourage others to cherry-pick directly from\nhim to get the fix out ASAP.\n\nNotice that not all changes to the base files are necessarily useful\nfor other descendants of the ancestral program.  So in this example,\npropagation of changes between projects is fairly explicit.\n\n[1] \"git checkout HEAD^0\" would be more convenient.\nSee DETACHED HEAD in the git checkout manual if interested.\n"}]}