git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Delta compression not so effective

From
Marius Storm-Olsen <mstormo@gmail.com>
Date
Mar 1, 2017, 13:51 UTC
Message-ID
<4d2a1852-8c84-2869-78ad-3c863f6dcaf7@gmail.com>

I have just converted an SVN repo to Git (using SubGit), where I feel delta compression has let me down :)

Suffice it to say, this is a "traditional" SVN repo, with an extern/ blown out of proportion with many binary check-ins. BUT, even still, I would expect Git's delta compression to be quite effective, compared to the compression present in SVN. In this case however, the Git repo ends up being 46% larger than the SVN DB.

Details - SVN:
     Commits: 32988
     DB (server) size: 139GB
     Branches: 103
     Tags: 1088
Details - Git:
     $ git count-objects -v
       count: 0
       size: 0
       in-pack: 666515
       packs: 1
       size-pack: 211933109
       prune-packable: 0
       garbage: 0
       size-garbage: 0
     $ du -sh .
       203G    .
     $ java -jar ~/sources/bfg/bfg.jar --delete-folders extern 
--no-blob-protection && \
       git reflog expire --expire=now --all && \
       git gc --prune=now --aggressive
     $ git count-objects -v
       count: 0
       size: 0
       in-pack: 495070
       packs: 1
       size-pack: 5765365
       prune-packable: 0
       garbage: 0
       size-garbage: 0
     $ du -sh .
       5.6G    .
When first importing, I disabled gc to avoid any repacking until 
completed. When done importing, there was 209GB of all loose objects 
(~670k files). With the hopes of quick consolidation, I did a
     git -c gc.autoDetach=0 -c gc.reflogExpire=0 \
           -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 \
           -c gc.rerereunresolved=0 -c gc.pruneExpire=now \
           gc --prune
which brought it down to 206GB in a single pack. I then ran
     git repack -a -d -F --window=350 --depth=250
which took it down to 203GB, where I'm at right now.
However, this is still miles away from the 139GB in SVN's DB.

Any ideas what's going on, and why my results are so terrible, compared to SVN?

Thanks!
-- 
.marius
Next: Junio C Hamano
Message 1 of 15 in “Delta compression not so effective”
  1. Marius Storm-OlsenMar 1, 2017
  2. Junio C HamanoMar 1, 2017
  3. Junio C HamanoMar 1, 2017
  4. Linus TorvaldsMar 1, 2017
  5. Marius Storm-OlsenMar 1, 2017
  6. Linus TorvaldsMar 1, 2017
  7. Martin LanghoffMar 1, 2017
  8. Marius Storm-OlsenMar 2, 2017
  9. Linus TorvaldsMar 2, 2017
  10. Marius Storm-OlsenMar 4, 2017
  11. Linus TorvaldsMar 6, 2017
  12. Marius Storm-OlsenMar 6, 2017
  13. Thomas BraunMar 7, 2017
  14. Martin LanghoffMar 1, 2017
  15. Marius Storm-OlsenMar 1, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.