git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 42/43] refs: add LMDB refs backend

From
Michael Haggerty <mhagger@alum.mit.edu>
Date
Oct 7, 2015, 18:31 UTC
Message-ID
<561564EC.8070704@alum.mit.edu>
In-Reply-To
<1444182660.7739.77.camel@twopensource.com>
On 10/07/2015 03:51 AM, David Turner wrote:
Show 18 quoted lines
> On Mon, 2015-10-05 at 17:47 +0200, Michael Haggerty wrote:
>> On 09/29/2015 12:02 AM, David Turner wrote:
>>> Add a database backend for refs using LMDB.  [...]
>>
>> I think you have said before that if one writer holds the write lock on
>> the DB, then other writers fail immediately. Is that correct? If so, is
>> that something that can be adjusted? I think it would be preferable for
>> the second writer to retry acquiring the write lock for a little while
>> with a timeout (as we now do when trying to acquire the packed-refs
>> lock). Otherwise you could have the unhappy situation that somebody
>> spends a long time pushing a packfile to a server, only to have the
>> reference update be rejected at the last moment due to a lock conflict
>> with another process that was touching completely different references.
>> We already do before/after consistency checks when updating references,
>> so you wouldn't even have to add such code in the backend yourself.
> 
> No, the second writer waits for the first writer to unlock (or for it to
> crash).
Cool, that's better behavior.
Show 7 quoted lines
> [...]
>> Do you store "peeled" reference values for tags, as is done in
>> packed-refs? I think that is an important optimization.
> 
> No.  Do you happen to know in what situations this is a performance
> benefit, so that I can benchmark?  I suspect it would matter much less
> for the LMDB backend, because lookups are pretty quick.

The reference lookup speed is not relevant here. "Peeling" is applied to references that point at tag objects (a.k.a. annotated tags). It means that the tag object is looked up to see what *it* points at (recursively if necessary) and the result is stored to the packed-refs file in a specially-formatted extra line that looks like

    17f9f635c101aef03874e1de1d8d0322187494b3 refs/tags/v2.6.0
    ^be08dee9738eaaa0423885ed189c2b6ad8368cf0

I think a good command to benchmark would be `git show-refs -d` in a repository with a number of annotated tags. This command's output is similar to the output of `git ls-remote <remote>` and also comes up during reference negotiation when fetching (so its performance is definitely not moot).

Show 8 quoted lines
> [...]
>> Currently we discard the reflog for a reference when the reference is
>> deleted. [...]
>> Have you thought about removing this limitation in the lbdb backend?
> 
> I'm going for feature parity first.  We can always add new functionality
> later.  This particular function would be pretty straightforward to add,
> I think.
+1
Show 18 quoted lines
> [...]
>>> +The rsync and file:// transports don't work yet, because they
>>> +don't use the refs API.
>>
>> Do they fail gracefully?
> 
> Not particularly gracefully.
> 
> rsync: link_stat "/home/dturner/git/t/trash
> directory.t5510-fetch/.git/packed-refs" failed: No such file or
> directory (2)
> rsync error: some files/attrs were not transferred (see previous errors)
> (code 23) at main.c(1183) [sender=3.1.1]
> fatal: Could not run rsync to get refs
> -------------
> 
> The problem is that rsync on the client assumes that packed-refs exists.
> We could hack it to also check for refdb.

I guess this is something that will have to be improved sooner or later, though I guess not as a precondition for merging this patch series.

Show 6 quoted lines
> [...]
>> I'm somewhat surprised that you only register the lmdb backend if it is
>> used in the main repo. I would expect the backend to be registered
>> unconditionally on startup. The cost is trivial, isn't it?
> 
> Yeah, but this was the easiest place to do it.
OK.
> [...]
I'm really happy about your work.

Regarding strategy: I think a good approach would be to get as much of the preparatory work as possible (the abstraction and separation of refs-be-files) to the point where it can be merged before there is too much more code churn in the area. That work is not very controversial, I think, and letting it wait for a long time will increase the likelihood of conflicts with other people's changes. The refs-be-lmdb patches, on the other hand, (1) will take longer to get polished, (2) will take longer to review because other people are not familiar with LDMB, and (3) won't bitrot very fast anyway because they don't overlap as much with areas that other people are likely to work on. So I would advocate working on those at a more deliberate pace and planning for them to be merged as a separate batch.

Michael
-- 
Michael Haggerty
mhagger@alum.mit.edu
Previous: David TurnerNext: Junio C Hamano
Message 83 of 90 in “lmdb ref backend”
  1. 00/43 lmdb ref backendDavid Turner, Sep 28, 2015
  2. 01/43 refs.c: create a public version of verify_refname_availableDavid Turner, Sep 28, 2015
  3. Torsten BögershausenOct 3, 2015
  4. David TurnerOct 3, 2015
  5. Torsten BögershausenOct 3, 2015
  6. Torsten BögershausenOct 4, 2015
  7. Michael HaggertyOct 5, 2015
  8. David TurnerOct 5, 2015
  9. 02/43 refs: make repack_without_refs and is_branch publicDavid Turner, Sep 28, 2015
  10. Michael HaggertyOct 5, 2015
  11. David TurnerOct 5, 2015
  12. 03/43 refs-be-files.c: rename refs to refs-be-filesDavid Turner, Sep 28, 2015
  13. 04/43 refs.c: add a new refs.c file to hold all common refs codeDavid Turner, Sep 28, 2015
  14. 05/43 refs.c: move update_ref to refs.cDavid Turner, Sep 28, 2015
  15. 06/43 refs.c: move delete_ref and delete_refs to the common codeDavid Turner, Sep 28, 2015
  16. 07/43 refs.c: move read_ref_at to the common refs fileDavid Turner, Sep 28, 2015
  17. 08/43 refs.c: move the hidden refs functions to the common codeDavid Turner, Sep 28, 2015
  18. 09/43 refs.c: move dwim and friend functions to the common refs codeDavid Turner, Sep 28, 2015
  19. 10/43 refs.c: move warn_if_dangling_symref* to the common codeDavid Turner, Sep 28, 2015
  20. 11/43 refs.c: move read_ref, read_ref_full and ref_exists to the common codeDavid Turner, Sep 28, 2015
  21. 12/43 refs.c: move resolve_refdup to commonDavid Turner, Sep 28, 2015
  22. 13/43 refs.c: move check_refname_format to the common codeDavid Turner, Sep 28, 2015
  23. 14/43 refs.c: move is_branch to the common codeDavid Turner, Sep 28, 2015
  24. 15/43 refs.c: move prettify_refname to the common codeDavid Turner, Sep 28, 2015
  25. 16/43 refs.c: move ref iterators to the common codeDavid Turner, Sep 28, 2015
  26. 17/43 refs.c: move head_ref_namespaced to the common codeDavid Turner, Sep 28, 2015
  27. 18/43 refs-be-files.c: add a backend method structure with transaction functionsDavid Turner, Sep 28, 2015
  28. Michael HaggertyOct 5, 2015
  29. Junio C HamanoOct 5, 2015
  30. David TurnerOct 6, 2015
  31. 19/43 refs-be-files.c: add methods for misc ref operationsDavid Turner, Sep 28, 2015
  32. 20/43 refs-be-files.c: add methods for the ref iteratorsDavid Turner, Sep 28, 2015
  33. 21/43 refs-be-files.c: add method for for_each_reftype_...David Turner, Sep 28, 2015
  34. 22/43 refs-be-files.c: add do_for_each_per_worktree_refDavid Turner, Sep 28, 2015
  35. Michael HaggertyOct 5, 2015
  36. David TurnerOct 5, 2015
  37. 23/43 refs.c: move refname_is_safe to the common codeDavid Turner, Sep 28, 2015
  38. 24/43 refs.h: document make refname_is_safe and add it to headerDavid Turner, Sep 28, 2015
  39. 25/43 refs.c: move copy_msg to the common codeDavid Turner, Sep 28, 2015
  40. 26/43 refs.c: move peel_object to the common codeDavid Turner, Sep 28, 2015
  41. 27/43 refs.c: move should_autocreate_reflog to common codeDavid Turner, Sep 28, 2015
  42. Junio C HamanoOct 2, 2015
  43. 28/43 refs.c: add ref backend init functionDavid Turner, Sep 28, 2015
  44. Michael HaggertyOct 5, 2015
  45. David TurnerOct 5, 2015
  46. 29/43 refs.c: add methods for reflogDavid Turner, Sep 28, 2015
  47. 30/43 refs-be-files.c: add method to expire reflogsDavid Turner, Sep 28, 2015
  48. Michael HaggertyOct 5, 2015
  49. 31/43 refs.c: add method for initial ref transaction commitDavid Turner, Sep 28, 2015
  50. 32/43 initdb: move safe_create_dir into common codeDavid Turner, Sep 28, 2015
  51. 33/43 refs.c: add method for initializing refs dbDavid Turner, Sep 28, 2015
  52. 34/43 refs.c: make struct ref_transaction genericDavid Turner, Sep 28, 2015
  53. Michael BlumeOct 6, 2015
  54. David TurnerOct 6, 2015
  55. 35/43 refs-be-files.c: add method to rename refsDavid Turner, Sep 28, 2015
  56. 36/43 run-command: track total number of commands runDavid Turner, Sep 28, 2015
  57. 37/43 refs: move some defines from refs-be-files.c to refs.hDavid Turner, Sep 28, 2015
  58. Michael HaggertyOct 5, 2015
  59. 38/43 refs: make some files backend functions publicDavid Turner, Sep 28, 2015
  60. Michael HaggertyOct 5, 2015
  61. David TurnerOct 6, 2015
  62. David TurnerOct 7, 2015
  63. Michael HaggertyOct 7, 2015
  64. Junio C HamanoOct 7, 2015
  65. David TurnerOct 7, 2015
  66. Michael HaggertyOct 7, 2015
  67. 39/43 refs: break out a ref conflict checkDavid Turner, Sep 28, 2015
  68. Michael HaggertyOct 5, 2015
  69. David TurnerOct 6, 2015
  70. 40/43 refs: allow ref backend to be set for cloneDavid Turner, Sep 28, 2015
  71. Michael HaggertyOct 5, 2015
  72. David TurnerOct 6, 2015
  73. Jeff KingOct 6, 2015
  74. David TurnerOct 6, 2015
  75. Carlos Martín NietoOct 12, 2015
  76. Michael HaggertyOct 5, 2015
  77. David TurnerOct 6, 2015
  78. 41/43 refs: add register_refs_backendDavid Turner, Sep 28, 2015
  79. 42/43 refs: add LMDB refs backendDavid Turner, Sep 28, 2015
  80. Junio C HamanoOct 2, 2015
  81. Michael HaggertyOct 5, 2015
  82. David TurnerOct 7, 2015
  83. Michael HaggertyOct 7, 2015
  84. Junio C HamanoOct 7, 2015
  85. David TurnerOct 7, 2015
  86. Michael HaggertyOct 7, 2015
  87. 43/43 refs: tests for db backendDavid Turner, Sep 28, 2015
  88. Dennis KaarsemakerOct 3, 2015
  89. Junio C HamanoOct 5, 2015
  90. David TurnerOct 6, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.