[PATCH 00/30] Read loose references lazily

From: Michael Haggerty <mhagger@alum.mit.edu>

This is the next installment of the ref-api saga. The main result of this patch series is to teach git to read loose references one directory at a time, only when they are needed.

The first nine commits are not very interesting; they convert get_ref_dir() and do_for_each_reflog() to use strbufs and to tighten up their specifications. The search_for_subdir() function extracted in patch 5 will be useful later. Patch 6 slightly reduces the amount of work needed to read loose refs from disk, and also changes the function's API in a way that will be needed later when directories are read one at a time.

Patches 10 - 25 mostly switch a lot of code from using ref_dir pointers to using ref_entry pointers as arguments and return values. This is important, because we want to be able to read directories of loose references on demand; this means that functions deep in the refs code need access to the name of directories of references. Since the name field is part of ref_entry but not ref_dir, this change has to percolate through many functions. The three key patches are patch 18, which removes the "dirname" argument to get_ref_dir(), and patches 21 and 25, which change search_ref_dir() and sort_ref_dir() to take ref_entries as argument. Together, these patches allow search_ref_dir() and sort_ref_dir() to call get_ref_dir() to fill directory entries on demand.

Patch 26-28 makes it possible to get from a ref_entry to the ref_cache of which it is part, and teach get_ref_dir() (renamed to read_loose_refs()) to use this mechanism to access the name of the ref_cache that it is to read.

Finally, patches 29 and 30 teach the code to read loose references lazily. This is done by adding a distinction between REF_DIR_COMPLETE and REF_DIR_INCOMPLETE; the latter is used to stub out directories of loose references that haven't been read yet. The loose references are read transparently into a REF_DIR_INCOMPLETE only when they are needed, at which time the directory entry is transformed into REF_DIR_COMPLETE.

This patch series has to be applied on top of mh/ref-api (or next). It passes all tests. It also includes quite a few assert()s to verify and document function preconditions. The asserts should all be pretty cheap, but if leaving them in is a problem please let me know.

It seems pretty overwhelming, but the changes are implemented in baby steps and most of the patches are quite simple.

Michael Haggerty (30):
  get_ref_dir(): return early if directory cannot be read
  get_ref_dir(): use a strbuf to hold refname
  get_ref_dir(): rename "base" parameter to "dirname"
  get_ref_dir(): require that the dirname argument ends in '/'
  refs.c: extract function search_for_subdir()
  get_ref_dir(): take the containing directory as argument
  do_for_each_reflog(): return early on error
  do_for_each_reflog(): use a strbuf to hold logfile name
  do_for_each_reflog(): reuse strbuf across recursive function calls
  refs: wrap top-level ref_dirs in ref_entries
  get_packed_refs(): return (ref_entry *) instead of (ref_dir *)
  get_loose_refs(): return (ref_entry *) instead of (ref_dir *)
  is_refname_available(): take (ref_entry *) instead of (ref_dir *)
  find_ref(): take (ref_entry *) instead of (ref_dir *)
  read_packed_refs(): take (ref_entry *) instead of (ref_dir *)
  add_ref(): take (ref_entry *) instead of (ref_dir *)
  find_containing_direntry(): use (ref_entry *) instead of (ref_dir *)
  get_ref_dir(): take (ref_entry *) instead of (ref_dir *)
  get_ref_dir(): remove dirname argument
  search_for_subdir(): take (ref_entry *) instead of (ref_dir *)
  search_ref_dir(): take (ref_entry *) instead of (ref_dir *)
  add_entry(): take (ref_entry *) instead of (ref_dir *)
  do_for_each_ref_in_dirs(): take (ref_entry *) instead of (ref_dir *)
  do_for_each_ref_in_dir(): take (ref_entry *) instead of (ref_dir *)
  sort_ref_dir(): take (ref_entry *) instead of (ref_dir *)
  struct ref_dir: store a reference to the enclosing ref_cache
  read_loose_refs(): rename function from get_ref_dir()
  read_loose_refs(): access ref_cache via the ref_dir field
  create_dir_entry(): allow the flag value to be passed as an argument
  refs: read loose references lazily

 refs.c |  550 +++++++++++++++++++++++++++++++++++++++-------------------------
 1 file changed, 332 insertions(+), 218 deletions(-)