{"thread":{"id":"55521","subject":"Git push --mirror hangs","startedAt":"2021-04-19T17:11:05Z","lastAt":"2021-04-20T06:30:19Z","messageCount":3,"participants":["Joonas Harjumaki","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"422319","messageId":"AM6PR06MB4263DD69639AFD60663D1DD6A7499@AM6PR06MB4263.eurprd06.prod.outlook.com","threadId":"55521","inReplyTo":null,"subject":"Git push --mirror hangs","fromName":"Joonas Harjumaki","fromEmail":"joonas.harjumaki@fi.abb.com","sentAt":"2021-04-19T17:11:01Z","receivedAt":"2021-04-19T17:11:05Z","isPatch":false,"sender":{"key":"joonas.harjumaki@fi.abb.com","avatar":null},"body":"I'm trying to \"git push --mirror https://[project]@dev.azure.com\" to Azure DevOps repository. The size-pack is 3.70 GiB. I have git version 2.31.1 from http://ppa.launchpad.net/git-core/ppa/ubuntu on Ubuntu 20.04.\niotop or top are not reporting any load.\n\niotop:\nTotal DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s\nCurrent DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s\n\ntop:\ntop - 19:58:05 up 12:33, 1 user, load average: 0,18, 0,27, 0,22\nTasks: 285 total, 1 running, 284 sleeping, 0 stopped, 0 zombie\n%Cpu(s): 1,4 us, 1,4 sy, 0,0 ni, 97,1 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st\nMiB Mem : 9727,7 total, 1069,0 free, 1123,0 used, 7535,8 buff/cache\nMiB Swap: 2048,0 total, 854,2 free, 1193,8 used. 8195,3 avail Mem \n\n PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND \n 84514 ssc600 20 0 14528 4116 3308 R 6,2 0,0 0:00.01 top \n 1 root 20 0 169028 13284 8504 S 0,0 0,1 0:21.42 systemd \n 2 root 20 0 0 0 0 S 0,0 0,0 0:00.02 kthreadd \n 3 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 rcu_gp \n 4 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 rcu_par_gp \n 6 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kworker/0:0H-kblockd \n 9 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 mm_percpu_wq \n 10 root 20 0 0 0 0 S 0,0 0,0 0:42.87 ksoftirqd/0 \n 11 root 20 0 0 0 0 I 0,0 0,0 0:12.28 rcu_preempt \n 12 root rt 0 0 0 0 S 0,0 0,0 0:00.30 migration/0 \n 13 root -51 0 0 0 0 S 0,0 0,0 0:00.00 idle_inject/0 \n 14 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/0 \n 15 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/1 \n\nGit has been stuck for an hour and half after printing these:\nWriting objects: 100% (116519/116519), 3.69 GiB | 2.99 MiB/s, done.\nTotal 116519 (delta 74981), reused 116519 (delta 74981), pack-reused 0\n18:42:45.650572 trace.c:487 performance: 1266.567544063 s: git command: /usr/lib/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --progress\n18:42:45.743254 pkt-line.c:80 packet: git> 0000\n18:42:46.465637 http.c:703 <= Recv header, 0000000013 bytes (0x0000000d)\n18:42:46.465693 http.c:715 <= Recv header: HTTP/2 200\n18:42:46.465707 http.c:703 <= Recv header, 0000000024 bytes (0x00000018)\n18:42:46.465716 http.c:715 <= Recv header: cache-control: private\n18:42:46.465727 http.c:703 <= Recv header, 0000000053 bytes (0x00000035)\n18:42:46.465734 http.c:715 <= Recv header: content-type: application/x-git-receive-pack-result\n18:42:46.465744 http.c:703 <= Recv header, 0000000124 bytes (0x0000007c)\n18:42:46.465752 http.c:715 <= Recv header: p3p: CP=\"CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT\"\n18:42:46.465763 http.c:703 <= Recv header, 0000000055 bytes (0x00000037)\n18:42:46.465770 http.c:715 <= Recv header: x-tfs-processid: [stripped]\n18:42:46.465778 http.c:703 <= Recv header, 0000000064 bytes (0x00000040)\n18:42:46.465785 http.c:715 <= Recv header: strict-transport-security: max-age=31536000; includeSubDomains\n18:42:46.465795 http.c:703 <= Recv header, 0000000050 bytes (0x00000032)\n18:42:46.465801 http.c:715 <= Recv header: activityid: [stripped]\n18:42:46.465810 http.c:703 <= Recv header, 0000000053 bytes (0x00000035)\n18:42:46.465833 http.c:715 <= Recv header: x-tfs-session: [stripped]\n18:42:46.465842 http.c:703 <= Recv header, 0000000051 bytes (0x00000033)\n18:42:46.465849 http.c:715 <= Recv header: x-vss-e2eid: [stripped]\n18:42:46.465858 http.c:703 <= Recv header, 0000000082 bytes (0x00000052)\n18:42:46.465866 http.c:715 <= Recv header: x-vss-userdata: [stripped]\n18:42:46.465899 http.c:703 <= Recv header, 0000000029 bytes (0x0000001d)\n18:42:46.465910 http.c:715 <= Recv header: x-frame-options: SAMEORIGIN\n18:42:46.465919 http.c:703 <= Recv header, 0000000068 bytes (0x00000044)\n18:42:46.465926 http.c:715 <= Recv header: request-context: [stripped]\n18:42:46.465936 http.c:703 <= Recv header, 0000000048 bytes (0x00000030)\n18:42:46.465945 http.c:715 <= Recv header: access-control-expose-headers: Request-Context\n18:42:46.465954 http.c:703 <= Recv header, 0000000033 bytes (0x00000021)\n18:42:46.465961 http.c:715 <= Recv header: x-content-type-options: nosniff\n18:42:46.465972 http.c:703 <= Recv header, 0000000104 bytes (0x00000068)\n18:42:46.465980 http.c:715 <= Recv header: x-msedge-ref: [stripped]\n18:42:46.465990 http.c:703 <= Recv header, 0000000037 bytes (0x00000025)\n18:42:46.465996 http.c:715 <= Recv header: date: Mon, 19 Apr 2021 15:42:45 GMT\n18:42:46.466009 http.c:703 <= Recv header, 0000000002 bytes (0x00000002)\n18:42:46.466017 http.c:715 <= Recv header:\n18:42:46.466294 pkt-line.c:80 packet: sideband< \\2\\15Analyzing objects... (41535/116519)\nremote:\n\nIs this normal and I should just wait, or is there a bug in git or Azure?\n\nJoonas Harjumäki\n"},{"id":"422360","messageId":"YH4tsaCBMJBGp0bC@camp.crustytoothpaste.net","threadId":"55521","inReplyTo":"AM6PR06MB4263DD69639AFD60663D1DD6A7499@AM6PR06MB4263.eurprd06.prod.outlook.com","subject":"Re: Git push --mirror hangs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-04-20T01:26:09Z","receivedAt":"2021-04-20T01:26:49Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-04-19 at 17:11:01, Joonas Harjumaki wrote:\n> Git has been stuck for an hour and half after printing these:\n> Writing objects: 100% (116519/116519), 3.69 GiB | 2.99 MiB/s, done.\n> Total 116519 (delta 74981), reused 116519 (delta 74981), pack-reused 0\n> 18:42:45.650572 trace.c:487 performance: 1266.567544063 s: git command: /usr/lib/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --progress\n> 18:42:45.743254 pkt-line.c:80 packet: git> 0000\n> 18:42:46.465637 http.c:703 <= Recv header, 0000000013 bytes (0x0000000d)\n> 18:42:46.465693 http.c:715 <= Recv header: HTTP/2 200\n> 18:42:46.465707 http.c:703 <= Recv header, 0000000024 bytes (0x00000018)\n> 18:42:46.465716 http.c:715 <= Recv header: cache-control: private\n> 18:42:46.465727 http.c:703 <= Recv header, 0000000053 bytes (0x00000035)\n> 18:42:46.465734 http.c:715 <= Recv header: content-type: application/x-git-receive-pack-result\n> 18:42:46.465744 http.c:703 <= Recv header, 0000000124 bytes (0x0000007c)\n> 18:42:46.465752 http.c:715 <= Recv header: p3p: CP=\"CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT\"\n> 18:42:46.465763 http.c:703 <= Recv header, 0000000055 bytes (0x00000037)\n> 18:42:46.465770 http.c:715 <= Recv header: x-tfs-processid: [stripped]\n> 18:42:46.465778 http.c:703 <= Recv header, 0000000064 bytes (0x00000040)\n> 18:42:46.465785 http.c:715 <= Recv header: strict-transport-security: max-age=31536000; includeSubDomains\n> 18:42:46.465795 http.c:703 <= Recv header, 0000000050 bytes (0x00000032)\n> 18:42:46.465801 http.c:715 <= Recv header: activityid: [stripped]\n> 18:42:46.465810 http.c:703 <= Recv header, 0000000053 bytes (0x00000035)\n> 18:42:46.465833 http.c:715 <= Recv header: x-tfs-session: [stripped]\n> 18:42:46.465842 http.c:703 <= Recv header, 0000000051 bytes (0x00000033)\n> 18:42:46.465849 http.c:715 <= Recv header: x-vss-e2eid: [stripped]\n> 18:42:46.465858 http.c:703 <= Recv header, 0000000082 bytes (0x00000052)\n> 18:42:46.465866 http.c:715 <= Recv header: x-vss-userdata: [stripped]\n> 18:42:46.465899 http.c:703 <= Recv header, 0000000029 bytes (0x0000001d)\n> 18:42:46.465910 http.c:715 <= Recv header: x-frame-options: SAMEORIGIN\n> 18:42:46.465919 http.c:703 <= Recv header, 0000000068 bytes (0x00000044)\n> 18:42:46.465926 http.c:715 <= Recv header: request-context: [stripped]\n> 18:42:46.465936 http.c:703 <= Recv header, 0000000048 bytes (0x00000030)\n> 18:42:46.465945 http.c:715 <= Recv header: access-control-expose-headers: Request-Context\n> 18:42:46.465954 http.c:703 <= Recv header, 0000000033 bytes (0x00000021)\n> 18:42:46.465961 http.c:715 <= Recv header: x-content-type-options: nosniff\n> 18:42:46.465972 http.c:703 <= Recv header, 0000000104 bytes (0x00000068)\n> 18:42:46.465980 http.c:715 <= Recv header: x-msedge-ref: [stripped]\n> 18:42:46.465990 http.c:703 <= Recv header, 0000000037 bytes (0x00000025)\n> 18:42:46.465996 http.c:715 <= Recv header: date: Mon, 19 Apr 2021 15:42:45 GMT\n> 18:42:46.466009 http.c:703 <= Recv header, 0000000002 bytes (0x00000002)\n> 18:42:46.466017 http.c:715 <= Recv header:\n> 18:42:46.466294 pkt-line.c:80 packet: sideband< \\2\\15Analyzing objects... (41535/116519)\n> remote:\n\nI suspect by the content of this sideband packet, that Azure DevOps is\ndoing something with those objects. The text \"Analyzing objects\"\ndoesn't appear in Git, so it's presumably some operation specific to\nAzure DevOps that's being performed.\n\nI would ask them what's going on on their side that's making this so\nslow. It's possible that waiting will help, but it's also possible that\nit won't. Since I don't know what's actually happening on the server\nside, I can't say.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"422369","messageId":"AM6PR06MB426347AC8403658E95038202A7489@AM6PR06MB4263.eurprd06.prod.outlook.com","threadId":"55521","inReplyTo":"YH4tsaCBMJBGp0bC@camp.crustytoothpaste.net","subject":"RE: Git push --mirror hangs","fromName":"Joonas Harjumaki","fromEmail":"joonas.harjumaki@fi.abb.com","sentAt":"2021-04-20T06:30:06Z","receivedAt":"2021-04-20T06:30:19Z","isPatch":false,"sender":{"key":"joonas.harjumaki@fi.abb.com","avatar":null},"body":"Thanks! Pushing the history in parts helped.\n\nJoonas Harjumäki\n\n-----Original Message-----\nFrom: brian m. carlson \nSent: tiistai 20. huhtikuuta 2021 4.26\nTo: Joonas Harjumaki \nCc: git@vger.kernel.org\nSubject: Re: Git push --mirror hangs\n\nOn 2021-04-19 at 17:11:01, Joonas Harjumaki wrote:\n> Git has been stuck for an hour and half after printing these:\n> Writing objects: 100% (116519/116519), 3.69 GiB | 2.99 MiB/s, done.\n> Total 116519 (delta 74981), reused 116519 (delta 74981), pack-reused 0\n> 18:42:45.650572 trace.c:487 performance: 1266.567544063 s: git command: /usr/lib/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --progress\n> 18:42:45.743254 pkt-line.c:80 packet: git> 0000\n> 18:42:46.465637 http.c:703 <= Recv header, 0000000013 bytes (0x0000000d)\n> 18:42:46.465693 http.c:715 <= Recv header: HTTP/2 200\n> 18:42:46.465707 http.c:703 <= Recv header, 0000000024 bytes (0x00000018)\n> 18:42:46.465716 http.c:715 <= Recv header: cache-control: private\n> 18:42:46.465727 http.c:703 <= Recv header, 0000000053 bytes (0x00000035)\n> 18:42:46.465734 http.c:715 <= Recv header: content-type: application/x-git-receive-pack-result\n> 18:42:46.465744 http.c:703 <= Recv header, 0000000124 bytes (0x0000007c)\n> 18:42:46.465752 http.c:715 <= Recv header: p3p: CP=\"CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT\"\n> 18:42:46.465763 http.c:703 <= Recv header, 0000000055 bytes (0x00000037)\n> 18:42:46.465770 http.c:715 <= Recv header: x-tfs-processid: [stripped]\n> 18:42:46.465778 http.c:703 <= Recv header, 0000000064 bytes (0x00000040)\n> 18:42:46.465785 http.c:715 <= Recv header: strict-transport-security: max-age=31536000; includeSubDomains\n> 18:42:46.465795 http.c:703 <= Recv header, 0000000050 bytes (0x00000032)\n> 18:42:46.465801 http.c:715 <= Recv header: activityid: [stripped]\n> 18:42:46.465810 http.c:703 <= Recv header, 0000000053 bytes (0x00000035)\n> 18:42:46.465833 http.c:715 <= Recv header: x-tfs-session: [stripped]\n> 18:42:46.465842 http.c:703 <= Recv header, 0000000051 bytes (0x00000033)\n> 18:42:46.465849 http.c:715 <= Recv header: x-vss-e2eid: [stripped]\n> 18:42:46.465858 http.c:703 <= Recv header, 0000000082 bytes (0x00000052)\n> 18:42:46.465866 http.c:715 <= Recv header: x-vss-userdata: [stripped]\n> 18:42:46.465899 http.c:703 <= Recv header, 0000000029 bytes (0x0000001d)\n> 18:42:46.465910 http.c:715 <= Recv header: x-frame-options: SAMEORIGIN\n> 18:42:46.465919 http.c:703 <= Recv header, 0000000068 bytes (0x00000044)\n> 18:42:46.465926 http.c:715 <= Recv header: request-context: [stripped]\n> 18:42:46.465936 http.c:703 <= Recv header, 0000000048 bytes (0x00000030)\n> 18:42:46.465945 http.c:715 <= Recv header: access-control-expose-headers: Request-Context\n> 18:42:46.465954 http.c:703 <= Recv header, 0000000033 bytes (0x00000021)\n> 18:42:46.465961 http.c:715 <= Recv header: x-content-type-options: nosniff\n> 18:42:46.465972 http.c:703 <= Recv header, 0000000104 bytes (0x00000068)\n> 18:42:46.465980 http.c:715 <= Recv header: x-msedge-ref: [stripped]\n> 18:42:46.465990 http.c:703 <= Recv header, 0000000037 bytes (0x00000025)\n> 18:42:46.465996 http.c:715 <= Recv header: date: Mon, 19 Apr 2021 15:42:45 GMT\n> 18:42:46.466009 http.c:703 <= Recv header, 0000000002 bytes (0x00000002)\n> 18:42:46.466017 http.c:715 <= Recv header:\n> 18:42:46.466294 pkt-line.c:80 packet: sideband< \\2\\15Analyzing objects... (41535/116519)\n> remote:\n\nI suspect by the content of this sideband packet, that Azure DevOps is doing something with those objects. The text \"Analyzing objects\"\ndoesn't appear in Git, so it's presumably some operation specific to Azure DevOps that's being performed.\n\nI would ask them what's going on on their side that's making this so slow. It's possible that waiting will help, but it's also possible that it won't. Since I don't know what's actually happening on the server side, I can't say.\n--\nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"}]}