threads / discuss / 42117

issue upgrading git from 1.8.2.1 to 2.8.0

Subject: issue upgrading git from 1.8.2.1 to 2.8.0

## tl;dr

6 messages between Apr 22, 2016 and Apr 23, 2016.

replies: 5people: 4as markdown or json

Gennaro Torre· Apr 22, 2016, 22:44 UTC · lore

Hi,

Last week we upgraded git from 1.8.2.1 to 2.8.0. We saw some very weird behavior where git clones and pushes were working, but git fetch was not.

Here is our setup:

We have 10+ nodes caching a few repositories, the repos were originally cloned via git 1.8.2.1 with `git clone --mirror <repo>`

We have 3000+ nodes that take deploys using the git protocol, via `git fetch`, the existing repositories were also cloned via git 1.8.2.1 from the caching nodes.

When we upgraded to 2.8.0, all nodes received the upgrade.

We observed that when we tried to deploy code to the nodes (they run `git fetch` to update the repository) this would fail.

The fix: we deleted the repositories originally cloned via git 1.8.2.1, and did a `git clone <repo>` dropping a fresh repository that was cloned using git 2.8.0. Everything started working correctly. Our running theory here is that there was some incompatibility with the repositories cloned with the old version of git, and trying to run `git fetch` with the newest version of git.

Thanks, Gennaro

Stefan Beller· Apr 22, 2016, 22:48 UTC · re: Gennaro Torre · lore

Re: issue upgrading git from 1.8.2.1 to 2.8.0

On Fri, Apr 22, 2016 at 3:44 PM, Gennaro Torre <gtorre@tumblr.com> wrote:

Show 27 quoted lines

>
> Hi,
>
> Last week we upgraded git from 1.8.2.1 to 2.8.0. We saw some very
> weird behavior where git clones and pushes were working, but git fetch
> was not.
>
> Here is our setup:
>
> We have 10+ nodes caching a few repositories, the repos were
> originally cloned via git 1.8.2.1 with `git clone --mirror <repo>`
>
> We have 3000+ nodes that take deploys using the git protocol, via `git
> fetch`, the existing repositories were also cloned via git 1.8.2.1
> from the caching nodes.
>
> When we upgraded to 2.8.0, all nodes received the upgrade.
>
> We observed that when we tried to deploy code to the nodes (they run
> `git fetch` to update the repository) this would fail.
>
> The fix: we deleted the repositories originally cloned via git
> 1.8.2.1, and did a `git clone <repo>` dropping a fresh repository that
> was cloned using git 2.8.0. Everything started working correctly. Our
> running theory here is that there was some incompatibility with the
> repositories cloned with the old version of git, and trying to run
> `git fetch` with the newest version of git.

Do you still have an old repository? You could compare a new and old repository with e.g. meld to see if there are differences you don't expect.

Although 1.8..2.8 is a lot, Git ought to stay compatible there.

Thanks, Stefan

Show 8 quoted lines

>
>
> Thanks,
> Gennaro
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Junio C Hamano· Apr 22, 2016, 22:57 UTC · re: Stefan Beller · lore

Re: issue upgrading git from 1.8.2.1 to 2.8.0

Stefan Beller <sbeller@google.com> writes:

> Although 1.8..2.8 is a lot, Git ought to stay compatible there.

In principle that is true, but there were deliberate backward incompatible changes at around 2.0 boundary, and those who followed along the upgrade paths were given warnings and advice messages to guide them to adjust their configurations during the incremental upgrade path. It is not all that surprising if somebody who jumped between the two versions in one hop didn't benefit from any such guidance, though.

Offhand, perhaps .git/config left by ancient versions and recent versions may differ in such a way that it affects the particular workflow deployed there?

Jeff King· Apr 22, 2016, 23:14 UTC · re: Gennaro Torre · lore

Re: issue upgrading git from 1.8.2.1 to 2.8.0

On Fri, Apr 22, 2016 at 06:44:02PM -0400, Gennaro Torre wrote:

> When we upgraded to 2.8.0, all nodes received the upgrade.
> 
> We observed that when we tried to deploy code to the nodes (they run
> `git fetch` to update the repository) this would fail.

Did `git fetch` say anything interesting on stderr?

-Peff

Jeff King· Apr 23, 2016, 05:11 UTC · lore

Re: issue upgrading git from 1.8.2.1 to 2.8.0

On Sat, Apr 23, 2016 at 01:01:06AM -0400, Gennaro Torre wrote:

Show 9 quoted lines

> @Jeff King nope nothing interesting, it just hangs:
> 
> [gtorre@host1 A:DEVEL tumblr.old]$ git fetch -vv
> Looking up <cache> ... done.
> Connecting to <cache> (port 9418) ... <ip> done.
> Server supports multi_ack_detailed
> Server supports side-band-64k
> Server supports ofs-delta
> Server version is git/2.8.0

Hmm. If you run with GIT_TRACE_PACKET=1, that may give an indication of why it is hanging. Or perhaps tracing via gdb to see where we are when it hangs.

-Peff

Jeff King· Apr 23, 2016, 05:22 UTC · lore

Re: issue upgrading git from 1.8.2.1 to 2.8.0

On Sat, Apr 23, 2016 at 01:18:50AM -0400, Gennaro Torre wrote:

Show 20 quoted lines

> This is what I see:
> 
> 01:16:15.789969 pkt-line.c:80           packet:        fetch>
> git-upload-pack /tumblr.git\0host=<cache>\0
> 01:16:15.851231 pkt-line.c:80           packet:        fetch<
> a210029455476c3ae0d34748aeaeccf71864fd81 HEAD\0multi_ack thin-pack
> side-band side-band-64k ofs-delta shallow no-progress include-tag
> multi_ack_detailed symref=HEAD:refs/heads/master agent=git/2.8.0
> 01:16:15.851433 pkt-line.c:80           packet:        fetch<
> 511b1feab8d10b81777a75c02dabc94c7da9e0d4 refs/heads/263-fix
> .....
> 01:17:59.497259 pkt-line.c:80           packet:        fetch<
> cd63ca15d47b469e692faea2e04f32a241a6637a refs/tags/hotposts_v0.1
> 01:17:59.497265 pkt-line.c:80           packet:        fetch<
> 4749ee8a11ca093928c8a43fb0147d22177f1d71 refs/tags/hotposts_v0.1^{}
> 01:17:59.497282 pkt-line.c:80           packet:        fetch< 0000
> Server supports multi_ack_detailed
> Server supports side-band-64k
> Server supports ofs-delta
> Server version is git/2.8.0

So that's weird. The client gets the initial ref advertisement and then...does nothing? It should drop into find_common() and start sending want/have lines to the server (or decide it doesn't need any objects and immediately send a flush packet).

Is the client chewing any CPU, or otherwise doing anything according to "strace"? Is it possible to connect to it with gdb and get a backtrace?

-Peff