{"thread":{"id":"6342","subject":"Restraining git pull/fetch to the current branch","startedAt":"2007-01-11T21:47:53Z","lastAt":"2007-01-15T13:06:55Z","messageCount":5,"participants":["Julian Phillips","Shawn O. Pearce","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"31500","messageId":"Pine.LNX.4.64.0701112127000.12870@beast.quantumfyre.co.uk","threadId":"6342","inReplyTo":null,"subject":"Restraining git pull/fetch to the current branch","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2007-01-11T21:47:53Z","receivedAt":"2007-01-11T21:47:53Z","isPatch":false,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"While trying out git on a large repository (10000s of commits, 1000s of \nbranches, ~2.5Gb when packed) at work I noticed that doing a pull was \ntaking a long time (longer than I was prepared to wait anyway).\n\nA quick test showed that a small repository (1 commit, 24k .git/objects) \nwith 1000 branches took 1m30 to do \"git pull\" (local xfs partition).  I \ndon't know if this is reasonable or not, but all I actually cared \nabout was updating the current branch, which \"git pull origin \n<branch_name>\" did in 0.3s.\n\nSo what I would like to know is: is there any way to make a pull/fetch \nwith no options default to only fetching the current branch? (other than \nscripting \"git pull/fetch origin $(git symbolic-ref HEAD)\" that is)\n\nTIA\n\n-- \nJulian\n\n  ---\nYou can't go home again, unless you set $HOME.\n"},{"id":"31530","messageId":"20070112005921.GE23864@spearce.org","threadId":"6342","inReplyTo":"Pine.LNX.4.64.0701112127000.12870@beast.quantumfyre.co.uk","subject":"Re: Restraining git pull/fetch to the current branch","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-01-12T00:59:21Z","receivedAt":"2007-01-12T00:59:21Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Julian Phillips <julian@quantumfyre.co.uk> wrote:\n> While trying out git on a large repository (10000s of commits, 1000s of \n> branches, ~2.5Gb when packed) at work I noticed that doing a pull was \n> taking a long time (longer than I was prepared to wait anyway).\n> \n> A quick test showed that a small repository (1 commit, 24k .git/objects) \n> with 1000 branches took 1m30 to do \"git pull\" (local xfs partition).  I \n> don't know if this is reasonable or not, but all I actually cared \n> about was updating the current branch, which \"git pull origin \n> <branch_name>\" did in 0.3s.\n> \n> So what I would like to know is: is there any way to make a pull/fetch \n> with no options default to only fetching the current branch? (other than \n> scripting \"git pull/fetch origin $(git symbolic-ref HEAD)\" that is)\n\nNo, but fortunately bash has a fancy alias tool:\n\n\talias gp='git pull origin $(git symbolic-ref HEAD)'\n\nperhaps your shell can help.  :-)\n\n\nLife is going to be painful with that repository with current Git\n(1.5.0 and later) as the new default configuration for a clone is to\ncopy every branch into refs/remotes/origin/*, where * is wildcarded\nagainst the current set of branches on the remote repository.\nIf that takes \"a long time\" you will be processing a lot of refs\nyou don't care about (or need to care about).\n\n-- \nShawn.\n"},{"id":"31532","messageId":"7vejq1yrt2.fsf@assigned-by-dhcp.cox.net","threadId":"6342","inReplyTo":"Pine.LNX.4.64.0701112127000.12870@beast.quantumfyre.co.uk","subject":"Re: Restraining git pull/fetch to the current branch","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-12T01:09:13Z","receivedAt":"2007-01-12T01:09:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Julian Phillips <julian@quantumfyre.co.uk> writes:\n\n> While trying out git on a large repository (10000s of commits, 1000s\n> of branches, ~2.5Gb when packed) at work I noticed that doing a pull\n> was taking a long time (longer than I was prepared to wait anyway).\n\nAre they all real branches?  In other words, does your project\nhave 1000s of active parallel development?\n\n> So what I would like to know is: is there any way to make a pull/fetch\n> with no options default to only fetching the current branch? (other\n> than scripting \"git pull/fetch origin $(git symbolic-ref HEAD)\" that\n> is)\n\nAlso, assuming the answer to the above question is yes, will you\nhave 1000s of branches on your end and will work on any one of\nthem?\n\nThe default configuration created by git-clone makes you track\nall branches from the remote side by putting:\n\n\tremote.origin.fetch = +refs/heads/*:refs/remotes/origin/*\n\nIf you do not care all 1000s branches but only are interested in\nselected few, you could change that configuration to suit your\nneeds better.\n\n    remote.origin.fetch = +refs/heads/stable:refs/remotes/origin/stable\n    remote.origin.fetch = +refs/heads/testing:refs/remotes/origin/testing\n    remote.origin.fetch = +refs/heads/unstable:refs/remotes/origin/unstable\n\n\nI suspect most of the time is being spent in the\nappend-fetch-head loop in fetch_main shell function in\ngit-fetch.sh The true fix would not be to limit the number of\nbranches updated, but to speed that part of the code up.\n"},{"id":"31554","messageId":"Pine.LNX.4.64.0701121332100.1493@reaper.quantumfyre.co.uk","threadId":"6342","inReplyTo":"7vejq1yrt2.fsf@assigned-by-dhcp.cox.net","subject":"Re: Restraining git pull/fetch to the current branch","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2007-01-12T14:08:28Z","receivedAt":"2007-01-12T14:08:28Z","isPatch":false,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"On Thu, 11 Jan 2007, Junio C Hamano wrote:\n\n> Julian Phillips <julian@quantumfyre.co.uk> writes:\n>\n>> While trying out git on a large repository (10000s of commits, 1000s\n>> of branches, ~2.5Gb when packed) at work I noticed that doing a pull\n>> was taking a long time (longer than I was prepared to wait anyway).\n>\n> Are they all real branches?  In other words, does your project\n> have 1000s of active parallel development?\n\n(Oops, over enthusiastic with the 0 there, I mean 100s of branches, about \n880 atm).\n\nThey are mostly topic style branches, with only 20 or so in active use at \nany one time.  The idea of having to cope with 100s of active branches at \nthe same time (given that we currently are using subversion) is quite \nfrankly terrifying.\n\n> Also, assuming the answer to the above question is yes, will you\n> have 1000s of branches on your end and will work on any one of\n> them?\n\nIt would be necessary to have access to all of the currently active \nbranches at least, with the added complication that the set of current \nactive branches changes quite rapidly.\n\n> If you do not care all 1000s branches but only are interested in\n> selected few, you could change that configuration to suit your\n> needs better.\n\nI think the problem here would be keeping track of which branches are \ncurrently active.  Some scheme could probably be derived, but I was hoping \nthat fetching an unchanged branch would be sufficently fast that it would \nbe necessary.  I appear to have been wrong :(\n\n> I suspect most of the time is being spent in the\n> append-fetch-head loop in fetch_main shell function in\n> git-fetch.sh The true fix would not be to limit the number of\n> branches updated, but to speed that part of the code up.\n\nIndeed, each call to append_fetch_head is taking ~1.7s (~1.5s user, ~0.2s \nsys).  So simply looping over all the branches explains the ~27m that a \ncomplete fetch takes. (This is for fetch with no updates).  Given that a \n\"clone orig new\" takes ~8m30 (half of which would seem to be IO), it looks \nlike it may be faster to create a new repository each time instead of \nupdating the old one, which is certainly a viable workaround - but might \nimply that fetch has some room for improvement?\n\n-- \nJulian\n\n  ---\nThere is nothing stranger in a strange land than the stranger who comes\nto visit.\n"},{"id":"31725","messageId":"Pine.LNX.4.64.0701151253190.9916@reaper.quantumfyre.co.uk","threadId":"6342","inReplyTo":"Pine.LNX.4.64.0701121332100.1493@reaper.quantumfyre.co.uk","subject":"Re: Restraining git pull/fetch to the current branch","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2007-01-15T13:06:55Z","receivedAt":"2007-01-15T13:06:55Z","isPatch":false,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"On Fri, 12 Jan 2007, Julian Phillips wrote:\n\n>>  I suspect most of the time is being spent in the\n>>  append-fetch-head loop in fetch_main shell function in\n>>  git-fetch.sh The true fix would not be to limit the number of\n>>  branches updated, but to speed that part of the code up.\n>\n> Indeed, each call to append_fetch_head is taking ~1.7s (~1.5s user, ~0.2s \n> sys).  So simply looping over all the branches explains the ~27m that a \n> complete fetch takes. (This is for fetch with no updates).  Given that a \n> \"clone orig new\" takes ~8m30 (half of which would seem to be IO), it looks \n> like it may be faster to create a new repository each time instead of \n> updating the old one, which is certainly a viable workaround - but might \n> imply that fetch has some room for improvement?\n\nI have had a chance to spend a little more time looking at this.  It would \nappear that the major culprit is show-ref.\n\nRunning \"git show-ref --hash <ref>\" takes ~1.7s, compared to 0.002s for \n\"cat $GIT_DIR/<ref>\".  If I add the following to the top of \nappend_fetch_head a null fetch takes 1m28s instead of ~27m.\n\nlocal_head_=$(cat $GIT_DIR/$local_name_);\nif [ \"$head_\" == \"$local_head_\" ]; then\n \treturn;\nfi\n\nLooking at the code for show-ref it appears to looks at all the refs to \nfind the one you ask for.  This makes fetch O(n^2) in no of branches, \nwhich would seem not strictly necessary - but then I am not really \nfamiliar with the internal working of git.  I noticed that the man page \nfor show-ref says that its use over direct access is encouraged, but as it \nstands it is far too slow to be used in fetch when you have a large \nmany-branched repository...\n\n(In looking at this I also discovered that if you have too many branches \nthen fetch will die with a too long command line error when calling \ngit-fetch-pack.)\n\n-- \nJulian\n\n  ---\nI will never lie to you.\n"}]}