[linux.conf.au] VCS Interoperability

Hi there good folk of the git community,

Next week, I'll be presenting a summary of the past 2 years work on improving svn interoperability for git. I'm requesting feedback from anyone who cares with regard to what they'd like to hear about.

 http://linux.conf.au/schedule/158/view_talk?day=friday

Jonathan put together an overview for Dmitry in preparation for Google Summer of Code 2011:

 svn-fe is very young, so a sufficiently bored person (meaning: I am
 not advocating that you do this; this is just an excuse to provide
 references to avoid getting stuck when you have questions) could
 probably read its history in full without getting lost.  Here are some
 references on the initial design (i.e., how responsibility is divided
 between the svndump driver and repo_tree and fast_export modules).
 The division of responsibilities between modules mostly survives,
 while the details of mechanism are quite different now.

  announcement and following thread:
  http://thread.gmane.org/gmane.comp.version-control.git/143180/focus=143388
  http://thread.gmane.org/gmane.comp.version-control.git/143187

  first public review (not much big picture stuff yet):
  http://thread.gmane.org/gmane.comp.version-control.git/147587

  second review (focuses on infrastructure, most of which is
  obsolete now :))
  http://thread.gmane.org/gmane.comp.version-control.git/148409

  third review (some serious thoughts about design begin here)
  http://thread.gmane.org/gmane.comp.version-control.git/148866/focus=149097

  fourth review (with a program to test with based on David's original
  program; review mentions some unresolved wishes: appropriate
  incremental import bookkeeping, properties, empty directories)
  http://thread.gmane.org/gmane.comp.version-control.git/149571/focus=149934

  fifth review, which is the one that stuck.
  http://thread.gmane.org/gmane.comp.version-control.git/151086/focus=151144

Also of note is the contributions made by Ram in 2010 that eventually became a headlining feature for Subversion 1.7:

 http://subversion.apache.org/docs/release-notes/1.7.html#svnrdump

Dmitry ended up submitting a number of short and pointy series. My favourite by far was:

 fast-import: improve deltas for blobs
 http://thread.gmane.org/gmane.comp.version-control.git/179774

This optimisation was glaring in hindsight. Respect to Dmitry for spotting it.

I believe a number of people have been contributing to the remote helper infrastructure lately. They also deserve credit.

Maybe a better historian than myself can provide additional links.

The overhanging issue after all this work is that there are about 95 outstanding patches waiting to be detangled and resubmitted. I regret not having time to complete this exercise myself.

I hope the process has been insightful and maybe to renew some interest. Thanks for reading.

-- David Barr

Hi David,

David Michael Barr wrote:

Nice.  As a lay person attending the conference, here are a few things
I think I'd like to hear:
- Why this project is so challenging compared to say, a git-hg bridge
or a git-darcs bridge.  What makes Subversion especially hard to deal
with?
- What is the biggest motivation for developing the svnrdump/ svnrload
combination?  Are there any other usecases for the tools?
- How has this project contributed to the development of the
fast-import infrastructure?  Can these changes be used to improve
other/ future remote helpers?
- You've spoken about exporting Subversion history to Git so far, but
what about the reverse?  Which parts of the picture are still missing?

Thanks.

-- Ram

Hi all,

Thank you for the feedback, it helped a lot with picking a trajectory.

Video is now available: http://youtu.be/0hVuv-wv4Dw
Slides: http://barrbrain.github.com/vcs-interoperability.html

It was my first conference presentation so the usual caveats apply. I was fortunate to have a small but interested audience.

I look forward to constructive criticism so that I can better represent our community to the folk Down Under.

-- David Barr

Hi David,

David Michael Barr wrote:

Thanks for the valuable criticism.  A few thoughts:
1. We'll try to fix this over the next few weeks.
2. Subversion's architecture is well-compartmentalized: is this what
leads to communication gaps between the API layers?
3. What are your thoughts on lib'ifying Git so that others can call
into it using an API?

Cheers!

On Sun, 22 Jan 2012, Ramkumar Ramachandra wrote:

This is something that everyone agrees would be a good thing. There have been many people who have started projects to do so, but they have all stalled.

David Lang

On Jan 22, 2012, at 4:12 PM, david@lang.hm wrote:

I believe libgit2 is still under active development.

http://libgit2.github.com

Hey,

On Sun, Jan 22, 2012 at 3:33 PM, Brian Gernhardt <benji@silverinsanity.com> wrote:

Yes, GitHub alone actually has 2 full time employees and 1 contractor who are entirely dedicated to the project. It also ships with the GitHub for Mac product, so it's used in production by tens of thousands every day. If any of you want to get involved, you can check out the mailing list (libgit2@librelist.org) or (probably more usefully) the GitHub project page:

https://github.com/libgit2/libgit2

Open tickets, contribute code, review PRs, etc.

Scott

I'm thinking libgit2 is where git features that have stablised are formalised. On the other hand, git-core is where features are incubated.

I would like to see fast-import ported to libgit2 when it stabilises ;)

After giving my talk, I feel compelled to reroll the historic vcs-svn series. I'm pushing as I go to my GitHub account:

  https://github.com/barrbrain/git/commits/svn-fe-reroll

-- David Barr

Hi,

David Barr wrote:

Just some quick notes to save some time:

All the commits on the 

 git://repo.or.cz/git/jrn.git svn-fe

branch were known-good in the sense that they seemed sane enough to build on and I do not think they need any reorganization or other work. Maybe that could help bootstrap your efforts?

The svn-fe-pu branch includes some other patches: some optimizations which are probably safe, the demo helper that allows "git clone svn-alpha::<url>", some transport-helper patches to support the latter, and so on. They are not vetted. If anyone sends patches from that branch, or any other patch for that matter, to the list and cc-s me, I'll be happy to review them. Here's a shortlog for convenience:

 David Barr (2):
       fast-import: allow object_table to grow dynamically
       fast-import: allow atom_table to grow dynamically
 
 Dmitry Ivankov (2):
       Arrange a backflow pipe from fast-importer to remote helper stdin
       Add alpha version of remote-svn helper
 
 Jonathan Nieder (12):
       work around ISO C incompatibility between data and function pointers
       ensure initializer elements are computable at load time
       enums: omit trailing comma for portability
       notes: avoid C99-style comment
       notes merge: eliminate OUTPUT macro
       make sure initializers are computable at load time
       fast-import: allow branch_table to grow dynamically
       fast-import: use DIV_ROUND_UP
       svn-fe: add a program that generates a notes-to-svn-revs mapping
       svn-fe: do not rely on /bin/env utility to launch remote helper
       svn-fe: use tabs to indent in remote helper script
       remaining warnings

Test results from the svn-fe branch would be interesting. I am particularly nervous about asking Junio to pull changes to contrib/svn-fe that might break it at the same time as making the interface changes needed for support of the "svnadmin dump --incremental" format without much testing since it would be painful to back them out. But probably that's moot, since after this long while there still hasn't been anyone testing it.

Jonathan

Jonathan Nieder wrote:

Eh, you only live once.

Junio, please pull

  git://repo.or.cz/git/jrn.git svn-fe

to get the following changes since commit ec014eac0e9e6f30cbbca616090fa2ecf74797e7:

  Git 1.7.5 (2011-04-23 23:36:32 -0700)

up to c5bcbcdcfa1e2a1977497cb3a342c0365c8d78d6:

  vcs-svn: reset first_commit_done in fast_export_init (2011-06-23 10:04:36 -0500)

I'd even be okay with pulling this for v1.7.9, but application for the next release would also be fine with me. It simplifies svn-fe a great deal and fulfills a longstanding wish: support for dumps with deltas in them. The cost is that commandline usage of the svn-fe tool becomes a little more complicated since it no longer keeps state itself but instead reads blobs back from fast-import in order to copy them between revisions and apply deltas to them.

Sorry I've taken so long to get to this, and thanks to David for the prodding.

----------------------------------------------------------------
David Barr (9):
      vcs-svn: set up channel to read fast-import cat-blob response
      vcs-svn: quote paths correctly for ls command
      vcs-svn: use mark from previous import for parent commit
      vcs-svn: pass paths through to fast-import
      vcs-svn: drop string_pool
      vcs-svn: drop treap
      vcs-svn: drop obj_pool
      vcs-svn: avoid using ls command twice
      vcs-svn: implement text-delta handling

Dmitry Ivankov (2):
      vcs-svn: do not initialize report_buffer twice
      vcs-svn: reset first_commit_done in fast_export_init

Jonathan Nieder (31):
      vcs-svn: use higher mark numbers for blobs
      vcs-svn: save marks for imported commits
      vcs-svn: add a comment before each commit
      vcs-svn: eliminate repo_tree structure
      vcs-svn: handle filenames with dq correctly
      Merge branch 'db/length-as-hash' (early part) into db/svn-fe-code-purge
      Merge branch 'db/strbufs-for-metadata' into db/svn-fe-code-purge
      Makefile: list one vcs-svn/xdiff object or header per line
      vcs-svn: learn to maintain a sliding view of a file
      vcs-svn: make buffer_read_binary API more convenient
      vcs-svn: skeleton of an svn delta parser
      vcs-svn: parse svndiff0 window header
      vcs-svn: read the preimage when applying deltas
      vcs-svn: read inline data from deltas
      vcs-svn: read instructions from deltas
      vcs-svn: implement copyfrom_data delta instruction
      vcs-svn: verify that deltas consume all inline data
      vcs-svn: let deltas use data from postimage
      vcs-svn: let deltas use data from preimage
      Merge commit 'v1.7.5' into svn-fe
      Merge branch 'db/vcs-svn-incremental' into svn-fe
      Merge branch 'db/svn-fe-code-purge' into svn-fe
      Merge branch 'db/delta-applier' into db/text-delta
      test-svn-fe: split off "test-svn-fe -d" into a separate function
      vcs-svn: cap number of bytes read from sliding view
      Merge branch 'db/delta-applier' into svn-fe
      Merge branch 'db/delta-applier' into db/text-delta
      vcs-svn: guard against overflow when computing preimage length
      vcs-svn: avoid hangs from corrupt deltas
      Merge branch 'db/text-delta' into svn-fe
      Merge branch 'db/text-delta' into svn-fe

 .gitignore                |    3 -
 Makefile                  |   58 +++++---
 contrib/svn-fe/svn-fe.txt |    9 +-
 t/t0080-vcs-svn.sh        |  117 ---------------
 t/t0081-line-buffer.sh    |  106 +-------------
 t/t9010-svn-fe.sh         |  365 ++++++++++++++++++++++++++++++++++++++-------
 t/t9011-svn-da.sh         |  248 ++++++++++++++++++++++++++++++
 test-obj-pool.c           |  116 --------------
 test-string-pool.c        |   31 ----
 test-svn-fe.c             |   50 ++++++-
 test-treap.c              |   70 ---------
 vcs-svn/LICENSE           |    3 +-
 vcs-svn/fast_export.c     |  253 +++++++++++++++++++++++++++++--
 vcs-svn/fast_export.h     |   26 +++-
 vcs-svn/line_buffer.c     |    6 +-
 vcs-svn/line_buffer.h     |    2 +-
 vcs-svn/obj_pool.h        |   61 --------
 vcs-svn/repo_tree.c       |  330 ++++-------------------------------------
 vcs-svn/repo_tree.h       |   12 +-
 vcs-svn/sliding_window.c  |   79 ++++++++++
 vcs-svn/sliding_window.h  |   18 +++
 vcs-svn/string_pool.c     |  102 -------------
 vcs-svn/string_pool.h     |   11 --
 vcs-svn/string_pool.txt   |   43 ------
 vcs-svn/svndiff.c         |  308 ++++++++++++++++++++++++++++++++++++++
 vcs-svn/svndiff.h         |   10 ++
 vcs-svn/svndump.c         |  117 ++++++++++-----
 vcs-svn/trp.h             |  237 -----------------------------
 vcs-svn/trp.txt           |  109 --------------
 29 files changed, 1434 insertions(+), 1466 deletions(-)
 delete mode 100755 t/t0080-vcs-svn.sh
 create mode 100755 t/t9011-svn-da.sh
 delete mode 100644 test-obj-pool.c
 delete mode 100644 test-string-pool.c
 delete mode 100644 test-treap.c
 delete mode 100644 vcs-svn/obj_pool.h
 create mode 100644 vcs-svn/sliding_window.c
 create mode 100644 vcs-svn/sliding_window.h
 delete mode 100644 vcs-svn/string_pool.c
 delete mode 100644 vcs-svn/string_pool.h
 delete mode 100644 vcs-svn/string_pool.txt
 create mode 100644 vcs-svn/svndiff.c
 create mode 100644 vcs-svn/svndiff.h
 delete mode 100644 vcs-svn/trp.h
 delete mode 100644 vcs-svn/trp.txt

Jonathan Nieder wrote:

Oh, and incremental imports, too. ;-)

On Fri, Jan 27, 2012 at 11:46 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:

Thank you, Jonathan. I forgot our prior discussion about avoiding yet another history rewrite. This provides a nice cut-point. I do think we need to gather Dmitry's work on svn-fe and propose a front-end for core so that it is no longer relegated to contrib.

-- David Barr.

David Barr wrote:

Listed at [1]. Thanks for the nice and well maintained overview, Dmitry.

Yep. Should such a remote helper link directly to the vcs-svn lib, or would it make sense to start with a script that makes use of svn-fe (possibly with a different name once it is hoisted out of contrib)?

[1] http://divanorama.github.com/gsoc11/streams.html

Jonathan Nieder <jrnieder@gmail.com> writes:

Thanks, but will pull after 1.7.9 that was scheduled to happen today.

By the way, we should have done a GPG keysigning at the last GitTogether. The above paragraph ("The series simplifies svn-fe a great deal...") could have been recorded in the message body of a signed tag, and we could have started eating our own dogfood, now even Linus and his lieutenants are using the upcoming 1.7.9 feature.

Junio C Hamano <gitster@pobox.com> writes:

There were some conflicts, but I think I resolved correctly. Please double-check what appears in 'pu'.

Thanks.

Jonathan Nieder <jrnieder@gmail.com> writes:

Curious; this series seem to be based on a codebase that is older than 6908e99 (Revert "t0081 (line-buffer): add buffering tests", 2011-03-30).

Not complaining, not asking any question. Just making an observation.