threads / discuss / 26288

Re: cannot fetch arm git tree

Subject: Re: cannot fetch arm git tree

## tl;dr

15 messages between Jan 16, 2011 and Jan 24, 2011.

replies: 14people: 7as markdown or json

Uwe Kleine-König· Jan 16, 2011, 11:08 UTC · lore
Hello Jello,
On Sun, Jan 16, 2011 at 09:23:15AM +0000, Russell King - ARM Linux wrote:
Show 26 quoted lines
> On Sun, Jan 16, 2011 at 10:28:55AM +0800, Jello huang wrote:
> > Dear Russell,
> > 
> > when i use git fetch,there arm some wrong with it.
> > 
> > jello@jello-laptop:~/git/russell/linux-2.6-arm$git pull
> > error: Unable to find 89e4d4b145bb7e73b4c45671a84b401a5d8694c1 under
> > http://ftp.arm.linux.org.uk/pub/armlinux/kernel/git-cur/linux-2.6-arm.git
> > Cannot obtain needed blob 89e4d4b145bb7e73b4c45671a84b401a5d8694c1
> > while processing commit eda2e5dcc914b4d70f665443efc9780e89a5e5c1.
> > error: Fetch failed.
> > 
> > What is the wrong?
> 
> No idea.  The tree has the object file in one of its pack files:
> 
> | rmk@ZenIV:[linux-2.6-arm.git]:<1020> GIT_DIR=. git cat-file -p 89e4d4b145bb7e73b4c45671a84b401a5d8694c1|head
> | 
> |         List of maintainers and how to submit kernel changes
> | 
> | Please try to follow the guidelines below.  This will make things
> | easier on the maintainers.  Not all of these guidelines matter for every
> | trivial patch so apply some common sense.
> 
> and the pack info file lists all the pack files.  Maybe you have a hash
> collision on a pack file with your repository?

Probably you have a corrupted pack file. Try renaming all packfiles listed in http://ftp.arm.linux.org.uk/pub/armlinux/kernel/git-cur/linux-2.6-arm.git/objects/info/packs (i.e. .git/objects/pack/pack-74405a23171b6debd894d4791e06956d6387022a.pack etc.) and try refetching then. This happend to me after Ctrl-Cing an earlier git-fetch. Git doesn't seem to handle that case.

Best regards Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Russell King - ARM Linux· Jan 16, 2011, 13:42 UTC · lore
On Sun, Jan 16, 2011 at 09:10:17PM +0800, Jello huang wrote:
> yes,git doesn't  handle that case and i rename the pack name,but there is
> also the similar error.Now i just delet the git tree and  clone it again
> tonight .
_Always_ without fail fetch Linus' tree before pulling my tree.

My tree is a rsync clone of the objects and pack files in Linus' tree, plus whatever git decided to build on top of that - for local commits that's individual object files. For remote pulls, that's probably a few small pack files.

There is *no* repacking of my tree. So the only times it gets 'repacked' is when Linus repacks his tree.

Let's say you already have a copy of my tree from a month ago, and Linus has pulled some work from me into his tree, and repacked his tree into one single pack file. At the moment, the largest pack file from Linus is 400MB plus a 50MB index.

You already have most of the contents of that 400MB pack file, but if you're missing even _one_ object which is contained within it, git will have to download the _entire_ 400MB pack file and index file to retrieve it.

However, if you first fetch Linus' tree via the git protocol, it can just request the objects it doesn't have from the git server. That will mean you'll have all the objects in the large pack files before you start trying to pull my tree, and git won't have to download 400MB for the sake of retrieving just maybe 10k that you didn't have.

This isn't something special with my tree - it's a side effect of the http protocol git uses. So, before you fetch _any_ http-based git tree, first make sure you're up to date with Linus'.

(I update my tree from Linus' in rsync mode to make http-based stuff a lot more friendly to people using it - some of whom are stuck behind firewalls which can only do http. Fetching a constantly repacked git tree via http results in hundreds of megabytes needing to be fetched every time.)

So please, whenever possible, always fetch Linus' latest tree _first_ and then mine. Same goes for any other http based tree which doesn't auto-repack.

Jello huang· Jan 17, 2011, 01:49 UTC · re: Russell King - ARM Linux · lore
Russell,thanks for your reply so elaborate and i have gotten the idea now.

On 16 January 2011 21:42, Russell King - ARM Linux <linux@arm.linux.org.uk>wrote:

Show 45 quoted lines
> On Sun, Jan 16, 2011 at 09:10:17PM +0800, Jello huang wrote:
> > yes,git doesn't  handle that case and i rename the pack name,but there is
> > also the similar error.Now i just delet the git tree and  clone it again
> > tonight .
>
> _Always_ without fail fetch Linus' tree before pulling my tree.
>
> My tree is a rsync clone of the objects and pack files in Linus' tree,
> plus whatever git decided to build on top of that - for local commits
> that's individual object files.  For remote pulls, that's probably a few
> small pack files.
>
> There is *no* repacking of my tree.  So the only times it gets 'repacked'
> is when Linus repacks his tree.
>
> Let's say you already have a copy of my tree from a month ago, and Linus
> has pulled some work from me into his tree, and repacked his tree into one
> single pack file.  At the moment, the largest pack file from Linus is
> 400MB plus a 50MB index.
>
> You already have most of the contents of that 400MB pack file, but if
> you're missing even _one_ object which is contained within it, git will
> have to download the _entire_ 400MB pack file and index file to retrieve
> it.
>
> However, if you first fetch Linus' tree via the git protocol, it can just
> request the objects it doesn't have from the git server.  That will mean
> you'll have all the objects in the large pack files before you start trying
> to pull my tree, and git won't have to download 400MB for the sake of
> retrieving just maybe 10k that you didn't have.
>
> This isn't something special with my tree - it's a side effect of the
> http protocol git uses.  So, before you fetch _any_ http-based git tree,
> first make sure you're up to date with Linus'.
>
> (I update my tree from Linus' in rsync mode to make http-based stuff a
> lot more friendly to people using it - some of whom are stuck behind
> firewalls which can only do http.  Fetching a constantly repacked git
> tree via http results in hundreds of megabytes needing to be fetched
> every time.)
>
> So please, whenever possible, always fetch Linus' latest tree _first_
> and then mine.  Same goes for any other http based tree which doesn't
> auto-repack.
>
-- 
JUST DO IT,NOTHING IS IMPOSSIBLE


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Detlef Vollmann· Jan 21, 2011, 13:38 UTC · re: Russell King - ARM Linux · lore
On 01/16/11 14:42, Russell King - ARM Linux wrote:
Show 9 quoted lines
> Let's say you already have a copy of my tree from a month ago, and Linus
> has pulled some work from me into his tree, and repacked his tree into one
> single pack file.  At the moment, the largest pack file from Linus is
> 400MB plus a 50MB index.
>
> You already have most of the contents of that 400MB pack file, but if
> you're missing even _one_ object which is contained within it, git will
> have to download the _entire_ 400MB pack file and index file to retrieve
> it.

I thought this has changed with "smart http" in git 1.6.6. Am I missing something?

   Detlef
Uwe Kleine-König· Jan 21, 2011, 13:47 UTC · re: Detlef Vollmann · lore
Hi Detlef,
On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
Show 12 quoted lines
> On 01/16/11 14:42, Russell King - ARM Linux wrote:
> >Let's say you already have a copy of my tree from a month ago, and Linus
> >has pulled some work from me into his tree, and repacked his tree into one
> >single pack file.  At the moment, the largest pack file from Linus is
> >400MB plus a 50MB index.
> >
> >You already have most of the contents of that 400MB pack file, but if
> >you're missing even _one_ object which is contained within it, git will
> >have to download the _entire_ 400MB pack file and index file to retrieve
> >it.
> I thought this has changed with "smart http" in git 1.6.6.
> Am I missing something?

Well, not all http repos offer smart http. E.g. Russell doesn't[1], probably because the serving machine doesn't have the power to nice serve a repo via git:// or smart http://.

Best regards Uwe

[1] I didn't recheck though
-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Russell King - ARM Linux· Jan 21, 2011, 13:57 UTC · re: Uwe Kleine-König · lore
On Fri, Jan 21, 2011 at 02:47:28PM +0100, Uwe Kleine-König wrote:
Show 18 quoted lines
> Hi Detlef,
> 
> On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
> > On 01/16/11 14:42, Russell King - ARM Linux wrote:
> > >Let's say you already have a copy of my tree from a month ago, and Linus
> > >has pulled some work from me into his tree, and repacked his tree into one
> > >single pack file.  At the moment, the largest pack file from Linus is
> > >400MB plus a 50MB index.
> > >
> > >You already have most of the contents of that 400MB pack file, but if
> > >you're missing even _one_ object which is contained within it, git will
> > >have to download the _entire_ 400MB pack file and index file to retrieve
> > >it.
> > I thought this has changed with "smart http" in git 1.6.6.
> > Am I missing something?
> Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
> probably because the serving machine doesn't have the power to nice
> serve a repo via git:// or smart http://.
What is smart http?  I don't particularly follow git developments.
Detlef Vollmann· Jan 21, 2011, 14:28 UTC · re: Russell King - ARM Linux · lore
On 01/21/11 14:57, Russell King - ARM Linux wrote:
Show 21 quoted lines
> On Fri, Jan 21, 2011 at 02:47:28PM +0100, Uwe Kleine-König wrote:
>> Hi Detlef,
>>
>> On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
>>> On 01/16/11 14:42, Russell King - ARM Linux wrote:
>>>> Let's say you already have a copy of my tree from a month ago, and Linus
>>>> has pulled some work from me into his tree, and repacked his tree into one
>>>> single pack file.  At the moment, the largest pack file from Linus is
>>>> 400MB plus a 50MB index.
>>>>
>>>> You already have most of the contents of that 400MB pack file, but if
>>>> you're missing even _one_ object which is contained within it, git will
>>>> have to download the _entire_ 400MB pack file and index file to retrieve
>>>> it.
>>> I thought this has changed with "smart http" in git 1.6.6.
>>> Am I missing something?
>> Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
>> probably because the serving machine doesn't have the power to nice
>> serve a repo via git:// or smart http://.
>
> What is smart http?  I don't particularly follow git developments.

It seems to be an implementation of the git protocol using HTTP as transport. Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.

   Detlef
Jello huang· Jan 21, 2011, 14:30 UTC · re: Detlef Vollmann · lore
not a lucky dog. i used 1.6.0,there was a large pack.
On 21 January 2011 22:28, Detlef Vollmann <dv@vollmann.ch> wrote:
Show 40 quoted lines
> On 01/21/11 14:57, Russell King - ARM Linux wrote:
>
>> On Fri, Jan 21, 2011 at 02:47:28PM +0100, Uwe Kleine-König wrote:
>>
>>> Hi Detlef,
>>>
>>> On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
>>>
>>>> On 01/16/11 14:42, Russell King - ARM Linux wrote:
>>>>
>>>>> Let's say you already have a copy of my tree from a month ago, and
>>>>> Linus
>>>>> has pulled some work from me into his tree, and repacked his tree into
>>>>> one
>>>>> single pack file.  At the moment, the largest pack file from Linus is
>>>>> 400MB plus a 50MB index.
>>>>>
>>>>> You already have most of the contents of that 400MB pack file, but if
>>>>> you're missing even _one_ object which is contained within it, git will
>>>>> have to download the _entire_ 400MB pack file and index file to
>>>>> retrieve
>>>>> it.
>>>>>
>>>> I thought this has changed with "smart http" in git 1.6.6.
>>>> Am I missing something?
>>>>
>>> Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
>>> probably because the serving machine doesn't have the power to nice
>>> serve a repo via git:// or smart http://.
>>>
>>
>> What is smart http?  I don't particularly follow git developments.
>>
> It seems to be an implementation of the git protocol using
> HTTP as transport.
> Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.
>
>  Detlef
>
>
-- 
JUST DO IT,NOTHING IS IMPOSSIBLE


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Russell King - ARM Linux· Jan 21, 2011, 14:50 UTC · re: Detlef Vollmann · lore
On Fri, Jan 21, 2011 at 03:28:14PM +0100, Detlef Vollmann wrote:
> It seems to be an implementation of the git protocol using
> HTTP as transport.
> Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.
Setting up Smart HTTP
...
   To set it up, it■s best to walk through the instructions on the
   `git-http-backend` documentation page. Basically, you have to install Git
   v1.6.6 or higher on a server with an Apache 2.x webserver (it has to be
   Apache, currently - other CGI servers don■t work, last I checked). Then
   you add something similar to this to your http.conf file:
 SetEnv GIT_PROJECT_ROOT /var/www/git
 SetEnv GIT_HTTP_EXPORT_ALL
 ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/

Great. Deciding that it will be http://servername.example.com/git/ is really damned annoying as that's traditionally where gitweb lives, which requires a different script alias.

It seems that due to a lack of coordination between different git developers, people running webservers have a choice between providing gitweb or this http extension.

I'm really not interested in working out how to bodge this into working along side the existing gitweb setup by adding lots of rewrite rules, so as gitweb got there first I think it has priority, that's what we have and we'll have to live without the smart http extensions.

It's really not that big a deal if you follow the advice I've given.
Detlef Vollmann· Jan 21, 2011, 15:14 UTC · re: Russell King - ARM Linux · lore
On 01/21/11 15:50, Russell King - ARM Linux wrote:
Show 25 quoted lines
> On Fri, Jan 21, 2011 at 03:28:14PM +0100, Detlef Vollmann wrote:
>> It seems to be an implementation of the git protocol using
>> HTTP as transport.
>> Some info on this is at<http://progit.org/2010/03/04/smart-http.html>.
>
> Setting up Smart HTTP
>
> ...
>     To set it up, it■s best to walk through the instructions on the
>     `git-http-backend` documentation page. Basically, you have to install Git
>     v1.6.6 or higher on a server with an Apache 2.x webserver (it has to be
>     Apache, currently - other CGI servers don■t work, last I checked). Then
>     you add something similar to this to your http.conf file:
>
>   SetEnv GIT_PROJECT_ROOT /var/www/git
>   SetEnv GIT_HTTP_EXPORT_ALL
>   ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
>
> Great.  Deciding that it will be http://servername.example.com/git/ is
> really damned annoying as that's traditionally where gitweb lives,
> which requires a different script alias.
>
> It seems that due to a lack of coordination between different git
> developers, people running webservers have a choice between providing
> gitweb or this http extension.

Huh? /git/ is just the example here, you can use any name you want. E.g. I use /auth/ for authenticated users, and others use /gitmob/ or /gitanon/ for non-authenticated users. And you can use something like gitsmart, githttp, or whatever...

   Detlef

_______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Uwe Kleine-König· Jan 21, 2011, 15:25 UTC · re: Russell King - ARM Linux · lore
On Fri, Jan 21, 2011 at 02:50:26PM +0000, Russell King - ARM Linux wrote:
Show 30 quoted lines
> On Fri, Jan 21, 2011 at 03:28:14PM +0100, Detlef Vollmann wrote:
> > It seems to be an implementation of the git protocol using
> > HTTP as transport.
> > Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.
> 
> Setting up Smart HTTP
> 
> ...
>    To set it up, it■s best to walk through the instructions on the
>    `git-http-backend` documentation page. Basically, you have to install Git
>    v1.6.6 or higher on a server with an Apache 2.x webserver (it has to be
>    Apache, currently - other CGI servers don■t work, last I checked). Then
>    you add something similar to this to your http.conf file:
> 
>  SetEnv GIT_PROJECT_ROOT /var/www/git
>  SetEnv GIT_HTTP_EXPORT_ALL
>  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
> 
> Great.  Deciding that it will be http://servername.example.com/git/ is
> really damned annoying as that's traditionally where gitweb lives,
> which requires a different script alias.
> 
> It seems that due to a lack of coordination between different git
> developers, people running webservers have a choice between providing
> gitweb or this http extension.
> 
> I'm really not interested in working out how to bodge this into working
> along side the existing gitweb setup by adding lots of rewrite rules, so
> as gitweb got there first I think it has priority, that's what we have
> and we'll have to live without the smart http extensions.
IIRC it's designed to live along side the http:// clone url.
git-http-backend can still serve dumb http clients including a web
browser.
 
But note that as git-http-backend less info it has to calculate much
more.  So the load it introduces should be comparable to running
git-daemon as should be the times to fetch from it.  So AFAIK the only
reason to run it is that more corporate users can access port 80.

Best regards Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Miles Bader· Jan 24, 2011, 05:01 UTC · re: Russell King - ARM Linux · lore
Russell King - ARM Linux <linux@arm.linux.org.uk> writes:
> I'm really not interested in working out how to bodge this into working
> along side the existing gitweb setup by adding lots of rewrite rules, so
> as gitweb got there first I think it has priority, that's what we have
> and we'll have to live without the smart http extensions.
...
> It's really not that big a deal if you follow the advice I've given.

Smart http is actually a very big deal -- the old git http protocol is almost unusable in practice with big repos, at least over somewhat latency-limited network connections.

If you don't intend to support people pulling over http, then maybe you don't care. But if you do care, it's very much worth a second look.

[My personal reason for caring is that I'm behind a corporate firewall that's latency limited, although it seems to have pretty good bandwidth. With some public repos, pulling via the old http protocol was a multi-hour operation; the new http protocol is typically multiple orders of magnitude faster in these cases.]

-Miles
-- 
Omochiroi!
Johannes Sixt· Jan 24, 2011, 07:21 UTC · re: Russell King - ARM Linux · lore
Am 1/21/2011 15:50, schrieb Russell King - ARM Linux:
Show 10 quoted lines
>  SetEnv GIT_PROJECT_ROOT /var/www/git
>  SetEnv GIT_HTTP_EXPORT_ALL
>  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
> 
> Great.  Deciding that it will be http://servername.example.com/git/ is
> really damned annoying as that's traditionally where gitweb lives,
> which requires a different script alias.
> ...
> I'm really not interested in working out how to bodge this into working
> along side the existing gitweb setup by adding lots of rewrite rules,...

It has been worked out for you already. It's just a single rule (although a bit longish). Look for 'ScriptAliasMatch' in http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

-- Hannes
J.H.· Jan 24, 2011, 07:34 UTC · re: Johannes Sixt · lore
On 01/23/2011 11:21 PM, Johannes Sixt wrote:
Show 15 quoted lines
> Am 1/21/2011 15:50, schrieb Russell King - ARM Linux:
>>  SetEnv GIT_PROJECT_ROOT /var/www/git
>>  SetEnv GIT_HTTP_EXPORT_ALL
>>  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
>>
>> Great.  Deciding that it will be http://servername.example.com/git/ is
>> really damned annoying as that's traditionally where gitweb lives,
>> which requires a different script alias.
>> ...
>> I'm really not interested in working out how to bodge this into working
>> along side the existing gitweb setup by adding lots of rewrite rules,...
> 
> It has been worked out for you already. It's just a single rule (although
> a bit longish). Look for 'ScriptAliasMatch' in
> http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

I was going to say, I have both gitweb and smart http support running in parallel, it's not terribly complicated. The way I'm doing it is a bit older (mainly that long ScriptAliasMatch broken out more), but it's still only 11 lines to the apache config:

# Various environment setup, should be obvious SetEnv GIT_PROJECT_ROOT /path SetEnv GIT_HTTP_EXPORT_ALL 1 SetEnv GIT_EXEC_PATH /usr/libexec/git-core/

# Here so that we can do straight wget fetches for the objects files, # pack files and refs heads. # This isn't strictly neccisary, but if it's done we can use the inbuilt # sendfile() of apache for the sending of those files vs. funneling them # through the CGI AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /path/$1 AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /path/$1 AliasMatch ^/(.*/refs/heads/.*)$ /path/$1

# Meat of how to make this all work ScriptAliasMatch "^/(.*/HEAD)$" /usr/libexec/git-core/git-http-backend/path/$1 ScriptAliasMatch "^/(.*/info/refs)$" /usr/libexec/git-core/git-http-backend/path/$1 ScriptAliasMatch "^/(.*/objects/info/[^/]+)$" /usr/libexec/git-core/git-http-backend/path/$1 ScriptAliasMatch "^/pub/scm/(.*/git-upload-pack)$" /usr/libexec/git-core/git-http-backend/path/$1 ScriptAliasMatch "^/pub/scm/(.*/git-receive-pack)$" /usr/libexec/git-core/git-http-backend/path/$1

Just an example anyway, the paths have been shortened to fit e-mail better.
- John 'Warthog9' Hawley
Russell King - ARM Linux· Jan 24, 2011, 10:53 UTC · re: Johannes Sixt · lore
On Mon, Jan 24, 2011 at 08:21:09AM +0100, Johannes Sixt wrote:
Show 15 quoted lines
> Am 1/21/2011 15:50, schrieb Russell King - ARM Linux:
> >  SetEnv GIT_PROJECT_ROOT /var/www/git
> >  SetEnv GIT_HTTP_EXPORT_ALL
> >  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
> > 
> > Great.  Deciding that it will be http://servername.example.com/git/ is
> > really damned annoying as that's traditionally where gitweb lives,
> > which requires a different script alias.
> > ...
> > I'm really not interested in working out how to bodge this into working
> > along side the existing gitweb setup by adding lots of rewrite rules,...
> 
> It has been worked out for you already. It's just a single rule (although
> a bit longish). Look for 'ScriptAliasMatch' in
> http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

Thanks, added taking two server restarts (one of them took a few minutes for apache to restart as it was soo busy...)

I assume this also means that I need to update the URLs people are using (as the tree wasn't exported under http://hostname.example.com/git/)?

← back to recent threads