[PATCH 00/38] Virtualization of the refs API

Since the that ref-iterator [1] changes seem to have gotten a positive reception, let's try to keep up the momentum. I hope I'm not overloading the review pipeline...

I think all of the groundwork is in place now to virtualize the refs API. This will open the way to storing refs in ways other than the familiar loose refs / packed refs format, such as David Turner's proposed LMDB-based storage [2].

This is a long patch series, but most of the patches are pretty simple and formulaic. The goal is to implement a `ref_store`. In the language of object-oriented programming, `ref_store` is an abstract base class representing a reference storage backend. It provides methods to read, write, and delete references and symrefs, and to iterate over references, reflogs, and reflog entries, plus a number of other things—19 methods in all.

The one concrete implementation of this class is files_ref_store, which implements the traditional loose/packed refs scheme with caching. After this patch series, about the only things left in `refs/files-backend.c` with external linkage are vtables.

On these 19 methods are built the refs API as used by the rest of Git. The OO interface is not exposed; instead, the OO implementation is wrapped in traditional C functions. In fact, this patch series doesn't change the existing public interface at all, and adds only two new functions!

Somebody who wants to implement a new way to store references needs to implement a new class derived from `ref_store`, including its 19 methods, plus one or two supporting ref_iterator classes, and wire it up to be instantiated when needed.

This patch series opens up a lot of possibilities for improving the internal design of the files backend; I've described some of my plans in recent emails [3,4]. But that can be done separately; for now, the main point of this series is to clear the way for LMDB-based reference storage.

This patch series owes a lot to Ronnie Sahlberg and David Turner, who wrote earlier drafts along the same basic lines. The current patch series differs from theirs in many details of where exactly to draw the line of abstraction. But the most important difference in my opinion is that this series allows multiple ref_stores of different types to coexist. I think this is important for the UI (for example, if somebody converts a submodule to use a different reference storage scheme than the umbrella repository uses), but also because it allows ref_stores to be compounded together internally to decrease the coupling between different parts of the system. I've tried to retain their authorship of any patches that are more or less recognizable from their versions (hopefully without introducing any bugs!) But the spirit of their versions permeates this patch series. Thanks a lot to both of you!

This series applies on top of the "ref-iterator v2" series that I just submitted [1]. It can also be obtained from my GitHub repo [5] as branch "ref-store".

Michael

[1] http://thread.gmane.org/gmane.comp.version-control.git/296322 [2] http://thread.gmane.org/gmane.comp.version-control.git/286572 [3] http://thread.gmane.org/gmane.comp.version-control.git/295961/focus=296096 [4] http://thread.gmane.org/gmane.comp.version-control.git/295961/focus=296186 [5] https://github.com/mhagger/git

David Turner (8):
  rename_ref_available(): add docstring
  refs: add methods for reflog
  refs: add method for initial ref transaction commit
  refs: add method for delete_refs
  refs: add methods to init refs db
  refs: add method to rename refs
  refs: make lock generic
  refs: implement iteration over only per-worktree refs

Michael Haggerty (28):
  resolve_gitlink_ref(): eliminate temporary variable
  refs: rename struct ref_cache to files_ref_store
  refs: create a base class "ref_store" for files_ref_store
  add_packed_ref(): add a files_ref_store argument
  get_packed_ref(): add a files_ref_store argument
  resolve_missing_loose_ref(): add a files_ref_store argument
  {lock,commit,rollback}_packed_refs(): add files_ref_store arguments
  refs: reorder definitions
  resolve_packed_ref(): rename function from resolve_missing_loose_ref()
  resolve_gitlink_packed_ref(): remove function
  read_raw_ref(): take a (struct ref_store *) argument
  resolve_ref_recursively(): new function
  resolve_gitlink_ref(): implement using resolve_ref_recursively()
  resolve_gitlink_ref(): avoid memory allocation in many cases
  resolve_gitlink_ref(): rename path parameter to submodule
  refs: make read_raw_ref() virtual
  refs: make verify_refname_available() virtual
  refs: make pack_refs() virtual
  refs: make create_symref() virtual
  refs: make peel_ref() virtual
  repack_without_refs(): add a files_ref_store argument
  lock_raw_ref(): add a files_ref_store argument
  commit_ref_update(): add a files_ref_store argument
  lock_ref_for_update(): add a files_ref_store argument
  lock_ref_sha1_basic(): add a files_ref_store argument
  split_symref_update(): add a files_ref_store argument
  files_ref_iterator_begin(): take a ref_store argument
  refs: add method iterator_begin

Ronnie Sahlberg (2):
  refs: add a backend method structure
  refs: add a transaction_commit() method

 builtin/init-db.c    |  21 +-
 refs.c               | 294 +++++++++++++++++++++++++-
 refs.h               |  13 +-
 refs/files-backend.c | 567 +++++++++++++++++++++++++++------------------------
 refs/refs-internal.h | 213 +++++++++++++++++--
 5 files changed, 799 insertions(+), 309 deletions(-)