Integration branching

I've been asked to setup & host a git repo of an "integration branch" of WebKit for our developers.

So I expected it to work like so:

git clone git://git.webkit.org/WebKit.git git checkout -b test

<Make a few small changes and commit>

git-config remote.upload.url ssh://git.webvm.net/srv/git/webkit-test x61:~/aplix/test/WebKit% git push --thin -v upload test Pushing to ssh://git.webvm.net/srv/git/webkit-test Counting objects: 371612, done. Compressing objects: 100% (67851/67851), done. Writing objects: 11% (43886/371612), 25.32 MiB | 88 KiB/

However when uploading the branch it seems to upload _everything_ which is far too big and impractical. Couldn't it just somehow reference the remote branch and just publish the patch of the "small changes"?

When doing a git-http-push instead of ssh, git resulted in a "Out of memory" error:

   1. git clone git://git.webkit.org/WebKit.git # Check it out
   2. git checkout -b test # create a "test" branch and switch to it
   3. echo blah >> ChangeLog # change something
   4. git commit -a # commit that test change
   5. git-config remote.upload.url http://moto@moto.webvm.net/moto.git
# configure the upload url
   6. git push upload test # fatal: Out of memory, malloc failed

Was this the fault of the server(nox) or client(x61)?

x61:~% git --version git version 1.5.4.5 nox:~% git --version git version 1.5.4.4

So is there some better way of publishing a branch? Without uploading the whole WebKit?

I did try something else, that was to create a webkit repo on the server nox like so:

nox:/srv/git/webkit% git --bare init --shared nox:/srv/git/webkit% git --bare fetch git://git.webkit.org/WebKit.git master:master git fetch -f git://git.webkit.org/WebKit.git master:master # again forcing updates

And let developers branch from that. This proved a little cumbersome as I need to manually force updates to webkit trunk when developers asked me to do that. However once a developer cloned ssh://git.webvm.net/srv/git/webkit. Creating a test branch 'deleteme' and then `git push origin deleteme` was extremely fast. So I guess this is the right way to achieve my goals?

Any suggestions and tips would be greatly appreciated!

"Kai Hendry" <hendry@aplixcorp.com> writes:

A set of big questions is how that git.webvm.net/srv/git/webkit-test repository was prepared, what its history look like and how it relates to the history your repository has. I presume that that is different from the repository you are pushing from (which is "x61:aplix/test/WebKit" repository), but other than that you did not give any details to guess further.

By the way, that --thin is an implementation detail and not meant for the end user consumption. I do not think it would matter in this case, though.

What you described is exactly what is supposed to be doing, but again as we do not know how that git.webvm.net:/srv/gi/webkit-test/ repository was prepared, we cannot tell.

But I have a theory:

    http://thread.gmane.org/gmane.comp.version-control.git/77643

and the latest code is still parked in 'pu' branch.

On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano <gitster@pobox.com> wrote:

webkit-test was just a `git --bare init --shared` and completely empty with I tried pushing (publishing) the changes to WebKit trunk.

With thanks,

On Mon, Apr 7, 2008 at 2:18 PM, Kai Hendry <hendry@aplixcorp.com> wrote:

Basically every git repository needs to be entirely self-sufficient, that is, it should be possible to check out a copy without referring to any external repositories. So your initial push to an empty repository will always upload all the objects, even if you pulled from some other.

The good news is that if your friends who checked out of the main repository want to try out your branch, it *won't* accidentally download all the extra objects over again. git will notice that they already have the same objects (retrieved from the main repo) and download only the new ones.

So yes, some space is "wasted" on your extra repo, but disk space is cheap anyway (and you now have a backup of the main repo for free :))

Have fun,

Avery

Ok it makes sense that I need at least one checkout of all the objects. But still it seems that when I branch and publish via http it wants to upload zillions of objects.

For example see the attached webkit-http.txt log. I think there is some bug in the http push transport as if the transport is ssh it's much *much* faster:

x61:~/aplix/webkit-test/webkit% git-config remote.origin.url
ssh://git.webvm.net/srv/git/webkit
x61:~/aplix/webkit-test/webkit% git push origin deleteme
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 289 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To ssh://git.webvm.net/srv/git/webkit
 * [new branch]      deleteme -> deleteme
x61:~/aplix/webkit-test/webkit%

I need http:// to work as some of the engineers I work with are behind a "corporate firewall" that doesn't allow ssh://. Crazy, I know.

x61:~/aplix/webkit-test% git clone git://git.webvm.net/webkit
Initialized empty Git repository in /home/hendry/aplix/webkit-test/webkit/.git/
remote: Counting objects: 406041, done.
remote: Compressing objects: 100% (98138/98138), done.
remote: Total 406041 (delta 295642), reused 398766 (delta 288407)
Receiving objects: 100% (406041/406041), 440.93 MiB | 2484 KiB/s, done.
Resolving deltas: 100% (295642/295642), done.
Checking out files: 100% (44750/44750), done.
x61:~/aplix/webkit-test% ll
total 12K
drwxr-xr-x  3 hendry hendry 4.0K 2008-04-08 13:39 ./
drwxr-xr-x 20 hendry hendry 4.0K 2008-04-08 13:22 ../
drwxr-xr-x 15 hendry hendry 4.0K 2008-04-08 13:54 webkit/
x61:~/aplix/webkit-test% cd webkit
x61:~/aplix/webkit-test/webkit% vim .git/config
x61:~/aplix/webkit-test/webkit% git-config remote.origin.url http://moto@moto.webvm.net/webkit.git
x61:~/aplix/webkit-test/webkit% vim .git/config
x61:~/aplix/webkit-test/webkit% ll
total 220K
drwxr-xr-x 15 hendry hendry 4.0K 2008-04-08 13:54 ./
drwxr-xr-x  3 hendry hendry 4.0K 2008-04-08 13:39 ../
-rwxr-xr-x  1 hendry hendry 1.6K 2008-04-08 13:54 autogen.sh*
drwxr-xr-x 12 hendry hendry 4.0K 2008-04-08 13:54 BugsSite/
-rw-r--r--  1 hendry hendry 104K 2008-04-08 13:54 ChangeLog
-rw-r--r--  1 hendry hendry  19K 2008-04-08 13:54 configure.ac
drwxr-xr-x  8 hendry hendry 4.0K 2008-04-08 14:34 .git/
-rw-r--r--  1 hendry hendry 9.6K 2008-04-08 13:54 GNUmakefile.am
drwxr-xr-x 13 hendry hendry 4.0K 2008-04-08 13:54 JavaScriptCore/
drwxr-xr-x  7 hendry hendry 4.0K 2008-04-08 13:54 JavaScriptGlue/
drwxr-xr-x 18 hendry hendry 4.0K 2008-04-08 13:54 LayoutTests/
-rw-r--r--  1 hendry hendry 1.1K 2008-04-08 13:54 Makefile
-rw-r--r--  1 hendry hendry 1.1K 2008-04-08 13:54 Makefile.shared
drwxr-xr-x  3 hendry hendry 4.0K 2008-04-08 13:54 PageLoadTests/
drwxr-xr-x  5 hendry hendry 4.0K 2008-04-08 13:54 PlanetWebKit/
drwxr-xr-x  5 hendry hendry 4.0K 2008-04-08 13:54 SunSpider/
drwxr-xr-x 25 hendry hendry 4.0K 2008-04-08 13:54 WebCore/
drwxr-xr-x  9 hendry hendry 4.0K 2008-04-08 13:54 WebKit/
drwxr-xr-x  4 hendry hendry 4.0K 2008-04-08 13:54 WebKitLibraries/
-rw-r--r--  1 hendry hendry 3.3K 2008-04-08 13:54 WebKit.pri
-rw-r--r--  1 hendry hendry  474 2008-04-08 13:54 WebKit.pro
drwxr-xr-x 14 hendry hendry 4.0K 2008-04-08 13:54 WebKitSite/
drwxr-xr-x 19 hendry hendry 4.0K 2008-04-08 13:54 WebKitTools/
x61:~/aplix/webkit-test/webkit% git checkout -b deleteme
Switched to a new branch "deleteme"
x61:~/aplix/webkit-test/webkit% vim ChangeLog
x61:~/aplix/webkit-test/webkit% git ci
Created commit 4b56425: Test
 1 files changed, 1 insertions(+), 0 deletions(-)
x61:~/aplix/webkit-test/webkit% git push origin deleteme
Fetching remote heads...
  refs/
  refs/heads/
  refs/heads/origin/
  refs/remotes/
  refs/remotes/webkit/
  refs/remotes/origin/
  refs/remotes/debian/
  refs/tags/
updating 'refs/heads/deleteme'
  from 0000000000000000000000000000000000000000
  to   4b5642541a73f105cf6e99b0b0c4ee00f0b869c2
    sending 29257 objects

"Kai Hendry" <hendry@aplixcorp.com> writes:

It is not "crazy", but simply is "unfortunate". I am not involved in http-push at all myself, but I recall there was somebody working on improving it recently?

On Tue, Apr 08, 2008 at 11:49:33AM -0700, Junio C Hamano wrote:

That was me, but it has been stalled by various debian work in progress.

I hope to be able to work on http-push again in a couple weeks.

Mike

Hi Mike!

You're busy. :)

So do you acknowledge this as a bug or am I doing something wrong?

If you know where the problem lies, perhaps I can get a C ninja from work to look at it.

Kind regards from the UK,

On Tue, Apr 08, 2008 at 09:32:31PM +0100, Kai Hendry wrote:

Could you provide a trace with GIT_CURL_VERBOSE set so that i can get a better idea ?

Mike

On Tue, Apr 08, 2008 at 10:43:19PM +0200, Mike Hommey wrote:

FWIW, it was made clear from off-list discussion with Kai that his problem was a lack of running git update-server-info when required.

Mike

"Kai Hendry" <hendry@aplixcorp.com> writes:

That explains it without the "theory" in my previous message. The destination repository must be populated with full history somehow (and one way would have been to start it by cloning from the upstream), but if it was empty that is alright as well, because your push made it up-to-date. IOW, there is nothing to worry about and the system is working as designed.

Further pushes will update it incrementally.