threads / discuss / 41064

Building Git with HTTPS support: avoiding libcurl?

Subject: Building Git with HTTPS support: avoiding libcurl?

## tl;dr

6 messages between Dec 22, 2015 and Dec 24, 2015.

replies: 5people: 5as markdown or json

Paul Smith· Dec 22, 2015, 15:39 UTC · lore

I'm trying to build Git (2.6.4) on GNU/Linux, but without any requirements (other than basic libc etc.) on the local system. This works fine except for one thing: git-remote-https.

In order to build this I need to have libcurl, but libcurl is a MONSTER library with an enormous number of prerequisites (see below).

Just wondering if anyone has considered an alternative to libcurl; maybe I'm wrong but it seems to me that HTTPS support for Git would require only a tiny fraction of the libcurl features and maybe there's an alternative available which would be more targeted?

I realize this is not a short-term thing in that there won't be an API compatible library that can just be dropped in. This is more a forward -looking question. For now I'm looking to see if I can rebuild libcurl myself without most of these dependencies such as Kerberos, LDAP, etc.

$ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4
        linux-vdso.so.1 =>  (0x00007fff37d81000)
        libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f682b921000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f682b704000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f682b49a000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f682b058000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f682ae0e000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f682abfe000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f682a9ac000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f682a792000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f682a573000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f682a1a9000)
        libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28 (0x00007f6829e8d000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f6829c59000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f6829a23000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f68297a3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f682959e000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f68292cc000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f682909d000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f6828e98000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f6828c8d000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f6828a71000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f6828855000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f6828615000)
        /lib64/ld-linux-x86-64.so.2 (0x0000559b03259000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f68283b0000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f682819c000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f6827f98000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f6827d8e000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f6827b04000)
        libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f6827861000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f682762d000)
        libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f6827418000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f6827210000)
        libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f6826fe6000)
        libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f6826dd7000)
        libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f6826b8c000)
        libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f68268be000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f6826686000)
Dave Borowitz· Dec 22, 2015, 17:08 UTC · re: Paul Smith · lore

Re: Building Git with HTTPS support: avoiding libcurl?

On Tue, Dec 22, 2015 at 7:39 AM, Paul Smith <paul@mad-scientist.net> wrote:
Show 6 quoted lines
> I'm trying to build Git (2.6.4) on GNU/Linux, but without any
> requirements (other than basic libc etc.) on the local system.  This
> works fine except for one thing: git-remote-https.
>
> In order to build this I need to have libcurl, but libcurl is a MONSTER
> library with an enormous number of prerequisites (see below).

Well, IIUC one of the reasons for Git's fork-everything strategy is to avoid having to dynamically link the core git binary against large libraries like libcurl, so the dependency size has been taken into account at least in that sense.

Show 50 quoted lines
> Just wondering if anyone has considered an alternative to libcurl; maybe
> I'm wrong but it seems to me that HTTPS support for Git would require
> only a tiny fraction of the libcurl features and maybe there's an
> alternative available which would be more targeted?
>
> I realize this is not a short-term thing in that there won't be an API
> compatible library that can just be dropped in.  This is more a forward
> -looking question.  For now I'm looking to see if I can rebuild libcurl
> myself without most of these dependencies such as Kerberos, LDAP, etc.
>
>
> $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4
>         linux-vdso.so.1 =>  (0x00007fff37d81000)
>         libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f682b921000)
>         librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f682b704000)
>         libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f682b49a000)
>         libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f682b058000)
>         libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f682ae0e000)
>         liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f682abfe000)
>         libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f682a9ac000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f682a792000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f682a573000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f682a1a9000)
>         libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28 (0x00007f6829e8d000)
>         libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f6829c59000)
>         libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f6829a23000)
>         libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f68297a3000)
>         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f682959e000)
>         libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f68292cc000)
>         libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f682909d000)
>         libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f6828e98000)
>         libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f6828c8d000)
>         libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f6828a71000)
>         libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f6828855000)
>         libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f6828615000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000559b03259000)
>         libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f68283b0000)
>         libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f682819c000)
>         libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f6827f98000)
>         libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f6827d8e000)
>         libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f6827b04000)
>         libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f6827861000)
>         libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f682762d000)
>         libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f6827418000)
>         libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f6827210000)
>         libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f6826fe6000)
>         libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f6826dd7000)
>         libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f6826b8c000)
>         libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f68268be000)
>         libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f6826686000)

Maybe half of those dependencies are crypto libraries, so even if you managed to eliminate libcurl, you'd have a hard time supporting HTTPS without them. Or maybe use something like boringssl instead?

Paul Smith· Dec 22, 2015, 17:30 UTC · re: Dave Borowitz · lore

Re: Building Git with HTTPS support: avoiding libcurl?

On Tue, 2015-12-22 at 09:08 -0800, Dave Borowitz wrote:
> Well, IIUC one of the reasons for Git's fork-everything strategy is to
> avoid having to dynamically link the core git binary against large
> libraries like libcurl, so the dependency size has been taken into
> account at least in that sense.

Sure, and it's great that I still get most Git even if I don't have libcurl.

But without support for cloning https remotes there's a significant hole in Git functionality...

I grok that Git doesn't want to re-invent the wheel and that libcurl is convenient. I just wonder if anyone knows of another wheel, that doesn't come attached to an entire tractor-trailer, that could be used instead :).

Daniel Stenberg· Dec 23, 2015, 10:17 UTC · re: Paul Smith · lore

Re: Building Git with HTTPS support: avoiding libcurl?

On Tue, 22 Dec 2015, Paul Smith wrote:
> I grok that Git doesn't want to re-invent the wheel and that libcurl is 
> convenient.  I just wonder if anyone knows of another wheel, that doesn't 
> come attached to an entire tractor-trailer, that could be used instead :).

But if you would consider another lib, then you could just rebuild your own libcurl instead from source, entirely without any dependencies on other libs! That would be similar to finding another lib with less dependencies. (As already mentioned, you'd still need crypto and TLS support no doubt.)

That huge dependency collection is there much because your distro decided that having a libcurl with all that support is preferable. libcurl itself offers lots of customizability at build-time so you can strip out most of that if you wanted to.

But why do the distros build and provide a libcurl that can do all this?

I think you can look at this from a slightly higher altitude. By re-using a very widely used, well developed and properly documented library (yeah, I claim it is but you don't need to take my word for it) that is available everywhere - git benefits. By having many projects use the same lib, even if no two projects use the exact same feature set, we get more reliable software in the entire ecosystem - with less work.

I would guess that switching out libcurl in git would be a not insignificant amount of work, as no libcurl alternative I'm aware of is even close to this API.

-- 
  / daniel.haxx.se
Johannes Schindelin· Dec 23, 2015, 19:35 UTC · re: Daniel Stenberg · lore

Re: Building Git with HTTPS support: avoiding libcurl?

Hi Daniel,
On Wed, 23 Dec 2015, Daniel Stenberg wrote:
> By re-using a very widely used, well developed and properly documented
> library [libcurl] (yeah, I claim it is but you don't need to take my
> word for it) that is available everywhere - git benefits.

For what it's worth, I fully agree. My perspective: By using a pretty much fully-configured cURL, Git for Windows has supported the largest number of use cases with minimal requirements from myself. It has been a boon. If every library was as easy to use and as high quality, anybody could maintain Git for Windows ;-)

Ciao, a very grateful Dscho

Thiago Farina· Dec 24, 2015, 22:36 UTC · re: Paul Smith · lore

Re: Building Git with HTTPS support: avoiding libcurl?

On Tue, Dec 22, 2015 at 1:39 PM, Paul Smith <paul@mad-scientist.net> wrote:
Show 7 quoted lines
> I'm trying to build Git (2.6.4) on GNU/Linux, but without any
> requirements (other than basic libc etc.) on the local system.  This
> works fine except for one thing: git-remote-https.
>
> In order to build this I need to have libcurl, but libcurl is a MONSTER
> library with an enormous number of prerequisites (see below).
>

I think Git would have to be changed to use raw sockets and implement everything it needs on top of that, like libgit2 already does. Certainly this won't be a trivial task.

-- 
Thiago Farina

← back to recent threads