git/list[1] front-page[2] threads[3] people[4] search[5] about
 

30min Script in git 2.7.4 takes 22+ hrs in git 2.9.3

From
RSRobert Stryker <rstryker@redhat.com>
Date
Apr 27, 2017, 16:36 UTC
Message-ID
<CA+Up40iusByn-R55=2=2Ae8KH1mkj4hGF_E9dX3vn1vboyMwMw@mail.gmail.com>
Hi all:

The following script attempts to merge 4 git repos into one, maintaining tag and branch content (but not SHAs). Each original repo basically gets its own subfolder in the new one. Original repos are first rewritten to have their history think they always belonged in the target subfolder.

The problem: the script takes 30 minutes for one environment including git 2.7.4, and generates a repo of about 30mb. When run by a coworker using git 2.9.3, it takes 22+ hours and generates a 10gb repo.

Clearly something here is very wrong. Either there's a pretty horrible regression or my idea is a pretty bad one ;)

General process for the script:
  - check out 4 repos
  - rewrite their history so they always thought they were in a subfolder
  - copy these 4 rewritten folders to a temporary location
  - get a list of branches and tags for each of the 4 repos
  - initialize a new repo with a readme.md
  - for each unique tag
       - check the 4 rewritten / backed up repos for the tag
       - for each of the 4 rewritten repos:
            - if the tag exists in that repo, merge it into the new
repo in a test branch
           -  git pull --no-edit ../intermediate/oneRewrittenRepo    (SLOW PART)
        - save the tag
   - for each unique branch (same logic)

So... yeah... 30mb + 30 minutes -> 11gb + 22 hours somewhere between these two versions of git?

According to coworker:
during each pass of the Tags' loop it's sitting for a long time on:
 git pull --no-edit ../intermediate/webtools.common
which runs in its turn
git fetch --update-head-ok ../intermediate/webtools.common
which in its turn runs
git-upload-pack ../intermediate/webtools.common
Any ideas here are much appreciated =/
The Script in question is here:
   https://gist.github.com/robstryker/4854fc86ab3714a5e1af353b98cbc768
Next: Jeff King
Message 1 of 3 in “30min Script in git 2.7.4 takes 22+ hrs in git 2.9.3”
  1. Robert StrykerApr 27, 2017
  2. Jeff KingApr 27, 2017
  3. Jeff KingApr 27, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.