git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

From
Jonathan Nieder <jrnieder@gmail.com>
Date
May 23, 2014, 15:53 UTC
Message-ID
<20140523155353.GA4065@google.com>
In-Reply-To
<537F67DD.5010101@alum.mit.edu>
Hi,
Michael Haggerty wrote:
Show 10 quoted lines
> The status quo is that we have a single reference back end consisting of
> loose references sitting on top of packed references.
>
> But really, loose references and packed references are two relatively
> independent reference back ends [1].  We just happen to use them layered
> on top of each other.
>
> This suggests to me that our current structure is best modeled as two
> independent reference back ends, with a third implementation of the same
> reference API whose job it is to compose the first two.
[...]
> [1] Forget for the sake of this discussion that we can't store symbolic
> references as packed refs.

I find it hard to forget that. ;-) More to the point, the trouble with loose refs and packed refs as independent reference backends is that neither has very good performance characteristics. Enumerating many loose refs is slow. Adding a new packed ref to a large list is also slow. Git currently uses both loose and packed refs in a way that allows each to overcome the limitations of the other, and the fact that it involves two on-disk data structures seems to me like an implementation detail of how it achieves that.

So I believe most git code should not have to know about the difference between loose and packed refs (or the upper and lower layer) to allow the details of the layering can be tuned in low-level ref handling code.

On the other hand, from a code structure perspective I can easily believe that implementing some subset (or maybe even all) of the reference backend API for loose refs and packed refs separately and providing a separate file describing how to compose them might be the cleanest way to write this code. It's more general layering that seems to lie in the direction of madness.

Maybe I'm wrong and people will find lots of use for combinations like
 * loose refs shadowing an sqlite database
 * tdb shadowing mysql
 * etc
It's easy to prove a naysayer wrong with code and I don't want to
discourage that.

For the topic at hand it's relevant because packed-refs have properties that make some operations (certain deletion/ref creation combinations) much less fussy than loose refs, and it would be nice to be able to take advantage of that. In the long term I would like to see git taking advantage of that when someone tries to fetch refs with names that would conflict on the filesystem (e.g., topic, topic/a, topic/b).

Thanks, Jonathan

Previous: Michael HaggertyNext: Michael Haggerty
Message 122 of 139 in “Use ref transactions for all ref updates”
  1. 00/44 Use ref transactions for all ref updatesRonnie Sahlberg, May 15, 2014
  2. 01/44 refs.c: constify the sha arguments for ref_transaction_create|delete|updateRonnie Sahlberg, May 15, 2014
  3. Jonathan NiederMay 15, 2014
  4. 02/44 refs.c: allow passing NULL to ref_transaction_freeRonnie Sahlberg, May 15, 2014
  5. Jonathan NiederMay 15, 2014
  6. Ronnie SahlbergMay 15, 2014
  7. 03/44 refs.c: add a strbuf argument to ref_transaction_commit for error loggingRonnie Sahlberg, May 15, 2014
  8. 04/44 refs.c: add an err argument to repack_without_refsRonnie Sahlberg, May 15, 2014
  9. Jonathan NiederMay 15, 2014
  10. Ronnie SahlbergMay 15, 2014
  11. 05/44 refs.c: make ref_update_reject_duplicates take a strbuf argument for errorsRonnie Sahlberg, May 15, 2014
  12. 06/44 refs.c: add an err argument ro delete_loose_refRonnie Sahlberg, May 15, 2014
  13. Jonathan NiederMay 15, 2014
  14. Ronnie SahlbergMay 15, 2014
  15. 07/44 refs.c: make update_ref_write update a strbuf on failureRonnie Sahlberg, May 15, 2014
  16. 08/44 update-ref.c: log transaction error from the update_refRonnie Sahlberg, May 15, 2014
  17. Jonathan NiederMay 15, 2014
  18. 09/44 refs.c: remove the onerr argument to ref_transaction_commitRonnie Sahlberg, May 15, 2014
  19. 10/44 refs.c: change ref_transaction_update() to do error checking and return statusRonnie Sahlberg, May 15, 2014
  20. Jonathan NiederMay 15, 2014
  21. Ronnie SahlbergMay 15, 2014
  22. 11/44 refs.c: change ref_transaction_create to do error checking and return statusRonnie Sahlberg, May 15, 2014
  23. Jonathan NiederMay 15, 2014
  24. Ronnie SahlbergMay 15, 2014
  25. 12/44 refs.c: ref_transaction_delete to check for error and return statusRonnie Sahlberg, May 15, 2014
  26. Jonathan NiederMay 15, 2014
  27. Ronnie SahlbergMay 15, 2014
  28. 13/44 tag.c: use ref transactions when doing updatesRonnie Sahlberg, May 15, 2014
  29. Jonathan NiederMay 15, 2014
  30. Ronnie SahlbergMay 15, 2014
  31. 14/44 replace.c: use the ref transaction functions for updatesRonnie Sahlberg, May 15, 2014
  32. Jonathan NiederMay 15, 2014
  33. Ronnie SahlbergMay 15, 2014
  34. 15/44 commit.c: use ref transactions for updatesRonnie Sahlberg, May 15, 2014
  35. Jonathan NiederMay 15, 2014
  36. 16/44 sequencer.c: use ref transactions for all ref updatesRonnie Sahlberg, May 15, 2014
  37. Jonathan NiederMay 15, 2014
  38. 17/44 fast-import.c: change update_branch to use ref transactionsRonnie Sahlberg, May 15, 2014
  39. Jonathan NiederMay 15, 2014
  40. Ronnie SahlbergMay 15, 2014
  41. 18/44 branch.c: use ref transaction for all ref updatesRonnie Sahlberg, May 15, 2014
  42. Jonathan NiederMay 15, 2014
  43. 19/44 refs.c: change update_ref to use a transactionRonnie Sahlberg, May 15, 2014
  44. Jonathan NiederMay 15, 2014
  45. 20/44 refs.c: free the transaction before returning when number of updates is 0Ronnie Sahlberg, May 15, 2014
  46. 21/44 refs.c: ref_transaction_commit should not free the transactionRonnie Sahlberg, May 15, 2014
  47. Jonathan NiederMay 16, 2014
  48. Ronnie SahlbergMay 16, 2014
  49. Ronnie SahlbergMay 16, 2014
  50. 22/44 fetch.c: clear errno before calling functions that might set itRonnie Sahlberg, May 15, 2014
  51. Jonathan NiederMay 16, 2014
  52. Ronnie SahlbergMay 16, 2014
  53. Jeff KingMay 16, 2014
  54. 23/44 fetch.c: change s_update_ref to use a ref transactionRonnie Sahlberg, May 15, 2014
  55. Jonathan NiederMay 16, 2014
  56. Ronnie SahlbergMay 16, 2014
  57. Jonathan NiederMay 16, 2014
  58. Ronnie SahlbergMay 19, 2014
  59. 24/44 fetch.c: use a single ref transaction for all ref updatesRonnie Sahlberg, May 15, 2014
  60. Jonathan NiederMay 16, 2014
  61. Ronnie SahlbergMay 19, 2014
  62. 25/44 receive-pack.c: use a reference transaction for updating the refsRonnie Sahlberg, May 15, 2014
  63. Jonathan NiederMay 20, 2014
  64. Ronnie SahlbergMay 20, 2014
  65. Ronnie SahlbergMay 21, 2014
  66. 26/44 fast-import.c: use a ref transaction when dumping tagsRonnie Sahlberg, May 15, 2014
  67. Jonathan NiederMay 20, 2014
  68. Ronnie SahlbergMay 20, 2014
  69. 27/44 walker.c: use ref transaction for ref updatesRonnie Sahlberg, May 15, 2014
  70. Jonathan NiederMay 21, 2014
  71. Ronnie SahlbergMay 21, 2014
  72. 28/44 refs.c: make write_ref_sha1 staticRonnie Sahlberg, May 15, 2014
  73. Jonathan NiederMay 21, 2014
  74. Ronnie SahlbergMay 21, 2014
  75. 29/44 refs.c: make lock_ref_sha1 staticRonnie Sahlberg, May 15, 2014
  76. Jonathan NiederMay 21, 2014
  77. 30/44 refs.c: add transaction.status and track OPEN/CLOSED/ERRORRonnie Sahlberg, May 15, 2014
  78. Jonathan NiederMay 21, 2014
  79. Ronnie SahlbergMay 21, 2014
  80. Jonathan NiederMay 21, 2014
  81. Ronnie SahlbergMay 22, 2014
  82. 31/44 refs.c: remove the update_ref_lock functionRonnie Sahlberg, May 15, 2014
  83. Jonathan NiederMay 21, 2014
  84. 32/44 refs.c: remove the update_ref_write functionRonnie Sahlberg, May 15, 2014
  85. Jonathan NiederMay 21, 2014
  86. Ronnie SahlbergMay 22, 2014
  87. 33/44 refs.c: remove lock_ref_sha1Ronnie Sahlberg, May 15, 2014
  88. Jonathan NiederMay 21, 2014
  89. 34/44 refs.c: make prune_ref use a transaction to delete the refRonnie Sahlberg, May 15, 2014
  90. Jonathan NiederMay 21, 2014
  91. Ronnie SahlbergMay 22, 2014
  92. 35/44 refs.c: make delete_ref use a transactionRonnie Sahlberg, May 15, 2014
  93. Jonathan NiederMay 21, 2014
  94. Ronnie SahlbergMay 22, 2014
  95. Ronnie SahlbergMay 22, 2014
  96. 36/44 refs.c: pass the ref log message to _create/delete/update instead of _commitRonnie Sahlberg, May 15, 2014
  97. Jonathan NiederMay 21, 2014
  98. Ronnie SahlbergMay 22, 2014
  99. 37/44 refs.c: pass NULL as *flags to read_ref_fullRonnie Sahlberg, May 15, 2014
  100. Jonathan NiederMay 21, 2014
  101. 38/44 refs.c: pack all refs before we start to rename a refRonnie Sahlberg, May 15, 2014
  102. Jonathan NiederMay 21, 2014
  103. Ronnie SahlbergMay 22, 2014
  104. Jonathan NiederMay 22, 2014
  105. Ronnie SahlbergMay 22, 2014
  106. 39/44 refs.c: move the check for valid refname to lock_ref_sha1_basicRonnie Sahlberg, May 15, 2014
  107. Jonathan NiederMay 22, 2014
  108. Ronnie SahlbergMay 22, 2014
  109. Jonathan NiederMay 22, 2014
  110. Ronnie SahlbergMay 22, 2014
  111. 40/44 refs.c: call lock_ref_sha1_basic directly from commitRonnie Sahlberg, May 15, 2014
  112. Jonathan NiederMay 22, 2014
  113. 41/44 refs.c: add a new flag for transaction delete for refs we know are packed onlyRonnie Sahlberg, May 15, 2014
  114. Jonathan NiederMay 22, 2014
  115. Ronnie SahlbergMay 22, 2014
  116. Ronnie SahlbergMay 22, 2014
  117. Jonathan NiederMay 22, 2014
  118. Jonathan NiederMay 22, 2014
  119. Ronnie SahlbergMay 23, 2014
  120. Jonathan NiederMay 23, 2014
  121. Michael HaggertyMay 23, 2014
  122. Jonathan NiederMay 23, 2014
  123. Michael HaggertyMay 23, 2014
  124. Junio C HamanoMay 27, 2014
  125. Michael HaggertyMay 28, 2014
  126. Junio C HamanoMay 28, 2014
  127. Michael HaggertyMay 28, 2014
  128. 42/44 refs.c: pass a skip list to name_conflict_fnRonnie Sahlberg, May 15, 2014
  129. Jonathan NiederMay 22, 2014
  130. Ronnie SahlbergMay 27, 2014
  131. 43/44 refs.c: make rename_ref use a transactionRonnie Sahlberg, May 15, 2014
  132. 44/44 refs.c: remove forward declaration of write_ref_sha1Ronnie Sahlberg, May 15, 2014
  133. Jonathan NiederMay 15, 2014
  134. Junio C HamanoMay 15, 2014
  135. Jonathan NiederMay 22, 2014
  136. Jonathan NiederMay 22, 2014
  137. Jonathan NiederMay 22, 2014
  138. Jonathan NiederMay 22, 2014
  139. Ronnie SahlbergMay 27, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.