# Integration branching 11 messages from 2008-04-07 to 2008-04-10. Participants: Kai Hendry, Junio C Hamano, Avery Pennarun, Mike Hommey. Thread: https://gitlist.dev/t/13012 ## Kai Hendry, 2008-04-07 13:02 Subject: Integration branching Message-ID: URL: https://gitlist.dev/e/b24851260804070602o11b93923m7ce7d79d5f59547b%40mail.gmail.com ``` I've been asked to setup & host a git repo of an "integration branch" of WebKit for our developers. So I expected it to work like so: git clone git://git.webkit.org/WebKit.git git checkout -b test git-config remote.upload.url ssh://git.webvm.net/srv/git/webkit-test x61:~/aplix/test/WebKit% git push --thin -v upload test Pushing to ssh://git.webvm.net/srv/git/webkit-test Counting objects: 371612, done. Compressing objects: 100% (67851/67851), done. Writing objects: 11% (43886/371612), 25.32 MiB | 88 KiB/ However when uploading the branch it seems to upload _everything_ which is far too big and impractical. Couldn't it just somehow reference the remote branch and just publish the patch of the "small changes"? When doing a git-http-push instead of ssh, git resulted in a "Out of memory" error: 1. git clone git://git.webkit.org/WebKit.git # Check it out 2. git checkout -b test # create a "test" branch and switch to it 3. echo blah >> ChangeLog # change something 4. git commit -a # commit that test change 5. git-config remote.upload.url http://moto@moto.webvm.net/moto.git # configure the upload url 6. git push upload test # fatal: Out of memory, malloc failed Was this the fault of the server(nox) or client(x61)? x61:~% git --version git version 1.5.4.5 nox:~% git --version git version 1.5.4.4 So is there some better way of publishing a branch? Without uploading the whole WebKit? I did try something else, that was to create a webkit repo on the server nox like so: nox:/srv/git/webkit% git --bare init --shared nox:/srv/git/webkit% git --bare fetch git://git.webkit.org/WebKit.git master:master git fetch -f git://git.webkit.org/WebKit.git master:master # again forcing updates And let developers branch from that. This proved a little cumbersome as I need to manually force updates to webkit trunk when developers asked me to do that. However once a developer cloned ssh://git.webvm.net/srv/git/webkit. Creating a test branch 'deleteme' and then `git push origin deleteme` was extremely fast. So I guess this is the right way to achieve my goals? Any suggestions and tips would be greatly appreciated! ``` ## Junio C Hamano, 2008-04-07 17:41 Subject: Re: Integration branching Message-ID: <7vod8lpnpb.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vod8lpnpb.fsf%40gitster.siamese.dyndns.org In-Reply-To: ``` "Kai Hendry" writes: > So I expected it to work like so: > > git clone git://git.webkit.org/WebKit.git > git checkout -b test > > > > git-config remote.upload.url ssh://git.webvm.net/srv/git/webkit-test A set of big questions is how that git.webvm.net/srv/git/webkit-test repository was prepared, what its history look like and how it relates to the history your repository has. I presume that that is different from the repository you are pushing from (which is "x61:aplix/test/WebKit" repository), but other than that you did not give any details to guess further. > x61:~/aplix/test/WebKit% git push --thin -v upload test By the way, that --thin is an implementation detail and not meant for the end user consumption. I do not think it would matter in this case, though. > Pushing to ssh://git.webvm.net/srv/git/webkit-test > Counting objects: 371612, done. > Compressing objects: 100% (67851/67851), done. > Writing objects: 11% (43886/371612), 25.32 MiB | 88 KiB/ > > However when uploading the branch it seems to upload _everything_ > which is far too big and impractical. Couldn't it just somehow > reference the remote branch and just publish the patch of the "small > changes"? What you described is exactly what is supposed to be doing, but again as we do not know how that git.webvm.net:/srv/gi/webkit-test/ repository was prepared, we cannot tell. But I have a theory: http://thread.gmane.org/gmane.comp.version-control.git/77643 and the latest code is still parked in 'pu' branch. ``` ## Kai Hendry, 2008-04-07 18:18 Subject: Re: Integration branching Message-ID: URL: https://gitlist.dev/e/b24851260804071118h4eaadef8i8313c5a4cb26495d%40mail.gmail.com In-Reply-To: <7vod8lpnpb.fsf@gitster.siamese.dyndns.org> ``` On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano wrote: > A set of big questions is how that git.webvm.net/srv/git/webkit-test > repository was prepared, what its history look like and how it relates to > the history your repository has. I presume that that is different from > the repository you are pushing from (which is "x61:aplix/test/WebKit" > repository), but other than that you did not give any details to guess > further. webkit-test was just a `git --bare init --shared` and completely empty with I tried pushing (publishing) the changes to WebKit trunk. With thanks, ``` ## Avery Pennarun, 2008-04-07 18:32 Subject: Re: Integration branching Message-ID: <32541b130804071132q24acff6an853730af4fc7273e@mail.gmail.com> URL: https://gitlist.dev/e/32541b130804071132q24acff6an853730af4fc7273e%40mail.gmail.com In-Reply-To: ``` On Mon, Apr 7, 2008 at 2:18 PM, Kai Hendry wrote: > On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano wrote: > > A set of big questions is how that git.webvm.net/srv/git/webkit-test > > repository was prepared, what its history look like and how it relates to > > the history your repository has. I presume that that is different from > > the repository you are pushing from (which is "x61:aplix/test/WebKit" > > repository), but other than that you did not give any details to guess > > further. > > webkit-test was just a `git --bare init --shared` and completely empty > with I tried pushing (publishing) the changes to WebKit trunk. > > With thanks, Basically every git repository needs to be entirely self-sufficient, that is, it should be possible to check out a copy without referring to any external repositories. So your initial push to an empty repository will always upload all the objects, even if you pulled from some other. The good news is that if your friends who checked out of the main repository want to try out your branch, it *won't* accidentally download all the extra objects over again. git will notice that they already have the same objects (retrieved from the main repo) and download only the new ones. So yes, some space is "wasted" on your extra repo, but disk space is cheap anyway (and you now have a backup of the main repo for free :)) Have fun, Avery ``` ## Junio C Hamano, 2008-04-07 18:44 Subject: Re: Integration branching Message-ID: <7vd4p1pks1.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vd4p1pks1.fsf%40gitster.siamese.dyndns.org In-Reply-To: ``` "Kai Hendry" writes: > On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano wrote: >> A set of big questions is how that git.webvm.net/srv/git/webkit-test >> repository was prepared, what its history look like and how it relates to >> the history your repository has. I presume that that is different from >> the repository you are pushing from (which is "x61:aplix/test/WebKit" >> repository), but other than that you did not give any details to guess >> further. > > webkit-test was just a `git --bare init --shared` and completely empty > with I tried pushing (publishing) the changes to WebKit trunk. That explains it without the "theory" in my previous message. The destination repository must be populated with full history somehow (and one way would have been to start it by cloning from the upstream), but if it was empty that is alright as well, because your push made it up-to-date. IOW, there is nothing to worry about and the system is working as designed. Further pushes will update it incrementally. ``` ## Kai Hendry, 2008-04-08 13:56 Subject: Re: Integration branching Message-ID: URL: https://gitlist.dev/e/b24851260804080656u4c42c31cv3984fa2943d83b8a%40mail.gmail.com In-Reply-To: <32541b130804071132q24acff6an853730af4fc7273e@mail.gmail.com> ``` Ok it makes sense that I need at least one checkout of all the objects. But still it seems that when I branch and publish via http it wants to upload zillions of objects. For example see the attached webkit-http.txt log. I think there is some bug in the http push transport as if the transport is ssh it's much *much* faster: x61:~/aplix/webkit-test/webkit% git-config remote.origin.url ssh://git.webvm.net/srv/git/webkit x61:~/aplix/webkit-test/webkit% git push origin deleteme Counting objects: 5, done. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 289 bytes, done. Total 3 (delta 2), reused 0 (delta 0) To ssh://git.webvm.net/srv/git/webkit * [new branch] deleteme -> deleteme x61:~/aplix/webkit-test/webkit% I need http:// to work as some of the engineers I work with are behind a "corporate firewall" that doesn't allow ssh://. Crazy, I know. x61:~/aplix/webkit-test% git clone git://git.webvm.net/webkit Initialized empty Git repository in /home/hendry/aplix/webkit-test/webkit/.git/ remote: Counting objects: 406041, done. remote: Compressing objects: 100% (98138/98138), done. remote: Total 406041 (delta 295642), reused 398766 (delta 288407) Receiving objects: 100% (406041/406041), 440.93 MiB | 2484 KiB/s, done. Resolving deltas: 100% (295642/295642), done. Checking out files: 100% (44750/44750), done. x61:~/aplix/webkit-test% ll total 12K drwxr-xr-x 3 hendry hendry 4.0K 2008-04-08 13:39 ./ drwxr-xr-x 20 hendry hendry 4.0K 2008-04-08 13:22 ../ drwxr-xr-x 15 hendry hendry 4.0K 2008-04-08 13:54 webkit/ x61:~/aplix/webkit-test% cd webkit x61:~/aplix/webkit-test/webkit% vim .git/config x61:~/aplix/webkit-test/webkit% git-config remote.origin.url http://moto@moto.webvm.net/webkit.git x61:~/aplix/webkit-test/webkit% vim .git/config x61:~/aplix/webkit-test/webkit% ll total 220K drwxr-xr-x 15 hendry hendry 4.0K 2008-04-08 13:54 ./ drwxr-xr-x 3 hendry hendry 4.0K 2008-04-08 13:39 ../ -rwxr-xr-x 1 hendry hendry 1.6K 2008-04-08 13:54 autogen.sh* drwxr-xr-x 12 hendry hendry 4.0K 2008-04-08 13:54 BugsSite/ -rw-r--r-- 1 hendry hendry 104K 2008-04-08 13:54 ChangeLog -rw-r--r-- 1 hendry hendry 19K 2008-04-08 13:54 configure.ac drwxr-xr-x 8 hendry hendry 4.0K 2008-04-08 14:34 .git/ -rw-r--r-- 1 hendry hendry 9.6K 2008-04-08 13:54 GNUmakefile.am drwxr-xr-x 13 hendry hendry 4.0K 2008-04-08 13:54 JavaScriptCore/ drwxr-xr-x 7 hendry hendry 4.0K 2008-04-08 13:54 JavaScriptGlue/ drwxr-xr-x 18 hendry hendry 4.0K 2008-04-08 13:54 LayoutTests/ -rw-r--r-- 1 hendry hendry 1.1K 2008-04-08 13:54 Makefile -rw-r--r-- 1 hendry hendry 1.1K 2008-04-08 13:54 Makefile.shared drwxr-xr-x 3 hendry hendry 4.0K 2008-04-08 13:54 PageLoadTests/ drwxr-xr-x 5 hendry hendry 4.0K 2008-04-08 13:54 PlanetWebKit/ drwxr-xr-x 5 hendry hendry 4.0K 2008-04-08 13:54 SunSpider/ drwxr-xr-x 25 hendry hendry 4.0K 2008-04-08 13:54 WebCore/ drwxr-xr-x 9 hendry hendry 4.0K 2008-04-08 13:54 WebKit/ drwxr-xr-x 4 hendry hendry 4.0K 2008-04-08 13:54 WebKitLibraries/ -rw-r--r-- 1 hendry hendry 3.3K 2008-04-08 13:54 WebKit.pri -rw-r--r-- 1 hendry hendry 474 2008-04-08 13:54 WebKit.pro drwxr-xr-x 14 hendry hendry 4.0K 2008-04-08 13:54 WebKitSite/ drwxr-xr-x 19 hendry hendry 4.0K 2008-04-08 13:54 WebKitTools/ x61:~/aplix/webkit-test/webkit% git checkout -b deleteme Switched to a new branch "deleteme" x61:~/aplix/webkit-test/webkit% vim ChangeLog x61:~/aplix/webkit-test/webkit% git ci Created commit 4b56425: Test 1 files changed, 1 insertions(+), 0 deletions(-) x61:~/aplix/webkit-test/webkit% git push origin deleteme Fetching remote heads... refs/ refs/heads/ refs/heads/origin/ refs/remotes/ refs/remotes/webkit/ refs/remotes/origin/ refs/remotes/debian/ refs/tags/ updating 'refs/heads/deleteme' from 0000000000000000000000000000000000000000 to 4b5642541a73f105cf6e99b0b0c4ee00f0b869c2 sending 29257 objects ``` ## Junio C Hamano, 2008-04-08 18:49 Subject: Re: Integration branching Message-ID: <7vmyo4mbb6.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vmyo4mbb6.fsf%40gitster.siamese.dyndns.org In-Reply-To: ``` "Kai Hendry" writes: > I need http:// to work as some of the engineers I work with are behind > a "corporate firewall" that doesn't allow ssh://. Crazy, I know. It is not "crazy", but simply is "unfortunate". I am not involved in http-push at all myself, but I recall there was somebody working on improving it recently? ``` ## Mike Hommey, 2008-04-08 19:01 Subject: Re: Integration branching Message-ID: <20080408190127.GA28556@glandium.org> URL: https://gitlist.dev/e/20080408190127.GA28556%40glandium.org In-Reply-To: <7vmyo4mbb6.fsf@gitster.siamese.dyndns.org> ``` On Tue, Apr 08, 2008 at 11:49:33AM -0700, Junio C Hamano wrote: > "Kai Hendry" writes: > > > I need http:// to work as some of the engineers I work with are behind > > a "corporate firewall" that doesn't allow ssh://. Crazy, I know. > > It is not "crazy", but simply is "unfortunate". I am not involved in > http-push at all myself, but I recall there was somebody working on > improving it recently? That was me, but it has been stalled by various debian work in progress. I hope to be able to work on http-push again in a couple weeks. Mike ``` ## Kai Hendry, 2008-04-08 20:32 Subject: Re: Integration branching Message-ID: URL: https://gitlist.dev/e/b24851260804081332l74beb7e6o2f48b18f65af7b55%40mail.gmail.com In-Reply-To: <20080408190127.GA28556@glandium.org> ``` Hi Mike! You're busy. :) So do you acknowledge this as a bug or am I doing something wrong? If you know where the problem lies, perhaps I can get a C ninja from work to look at it. Kind regards from the UK, ``` ## Mike Hommey, 2008-04-08 20:43 Subject: Re: Integration branching Message-ID: <20080408204319.GA12323@glandium.org> URL: https://gitlist.dev/e/20080408204319.GA12323%40glandium.org In-Reply-To: ``` On Tue, Apr 08, 2008 at 09:32:31PM +0100, Kai Hendry wrote: > Hi Mike! > > You're busy. :) > > So do you acknowledge this as a bug or am I doing something wrong? > > If you know where the problem lies, perhaps I can get a C ninja from > work to look at it. Could you provide a trace with GIT_CURL_VERBOSE set so that i can get a better idea ? Mike ``` ## Mike Hommey, 2008-04-10 20:04 Subject: Re: Integration branching Message-ID: <20080410200413.GA16951@glandium.org> URL: https://gitlist.dev/e/20080410200413.GA16951%40glandium.org In-Reply-To: <20080408204319.GA12323@glandium.org> ``` On Tue, Apr 08, 2008 at 10:43:19PM +0200, Mike Hommey wrote: > On Tue, Apr 08, 2008 at 09:32:31PM +0100, Kai Hendry wrote: > > Hi Mike! > > > > You're busy. :) > > > > So do you acknowledge this as a bug or am I doing something wrong? > > > > If you know where the problem lies, perhaps I can get a C ninja from > > work to look at it. > > Could you provide a trace with GIT_CURL_VERBOSE set so that i can get a > better idea ? FWIW, it was made clear from off-list discussion with Kai that his problem was a lack of running git update-server-info when required. Mike ```