git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 00/18] reftable: stop using "git-compat-util.h"

From
Patrick Steinhardt <ps@pks.im>
Date
Feb 6, 2025, 07:52 UTC
Message-ID
<20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im>
In-Reply-To
<20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im>
Hi,

this patch series is the final step to fully decouple the reftable library from the rest of the Git codebase. The goal of this is to make the library reusable by other projects like libgit2 by simply copying over the source files, making Git the canonical upstream for reftable functionality.

This patch series stops using all kinds of helpers exposed by our "git-compat-util.h" header and open-codes them instead. In order to keep us from using these helpers by accident the final step is to pull out POSIX-related bits and pieces into a new "compat/posix.h" header, which the reftable library then uses instead of "git-compat-util.h".

The series is built on top of master at 5f8f7081f7 (The third batch, 2025-01-23) with ps/reftable-sign-compare at 33319b0976 (reftable: address trivial -Wsign-compare warnings, 2025-01-20) merged into it. There is a trivial merge conflict with ps/zlib-ng that can be solved like this:

    diff --cc reftable/system.h
    index e4a8944a70,d02eacea8f..0000000000
    --- a/reftable/system.h
    +++ b/reftable/system.h
    @@@ -11,15 -11,9 +11,15 @@@ https://developers.google.com/open-sour
      
      /* This header glues the reftable library to the rest of Git */
      
     -#include "git-compat-util.h"
     +#include "compat/posix.h"
    - #include <zlib.h>
    + #include "compat/zlib-compat.h"
      
     +/*
     + * Return a random 32 bit integer. This function is expected to return
     + * pre-seeded data.
     + */
     +uint32_t reftable_rand(void);
     +
      /*
       * An implementation-specific temporary file. By making this specific to the
       * implementation it becomes possible to tie temporary files into any kind of
Changes in v2:
  - The splitup of Windows headers has broken compilation because some
    of the headers couldn't be found anymore. I've fixed this more
    generally by converting includes in "compat/" to always be relative
    to the project source directory, dropping the platform-specific
    `-Icompat/` include.
  - Explain why we don't port over `EWOULDBLOCK` handling.
  - Fix commit message typos.
  - Link to v1: https://lore.kernel.org/r/20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im
Changes in v3:
  - Fix type of `total_read` variable used to track how many bytes we
    have read in `fd_read_lines()`.
  - Drop the patch use root-relative includes again. Let's rather
    discuss this outside of the scope of this series.
  - Link to v2: https://lore.kernel.org/r/20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im
Changes in v4:
  - Fix a couple of now-invalid relative includes that I missed. This
    fixes the build issue in "seen" with Meson.
  - Link to v3: https://lore.kernel.org/r/20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (18):
      reftable/stack: stop using `read_in_full()`
      reftable/stack: stop using `write_in_full()`
      reftable/blocksource: stop using `xmmap()`
      reftable/record: stop using `COPY_ARRAY()`
      reftable/record: stop using `BUG()` in `reftable_record_init()`
      reftable/record: don't `BUG()` in `reftable_record_cmp()`
      reftable: stop using `BUG()` in trivial cases
      reftable/basics: stop using `st_mult()` in array allocators
      reftable/basics: provide wrappers for big endian conversion
      reftable/reader: stop using `ARRAY_SIZE()` macro
      reftable/system: introduce `reftable_rand()`
      reftable/stack: stop using `sleep_millisec()`
      reftable/basics: stop using `SWAP()` macro
      reftable/basics: stop using `UNUSED` annotation
      compat/mingw: split out POSIX-related bits
      git-compat-util.h: split out POSIX-emulating bits
      reftable: decouple from Git codebase by pulling in "compat/posix.h"
      Makefile: skip reftable library for Coccinelle
 Makefile                                |   2 +-
 compat/{mingw.c => mingw/compat-util.c} |  28 +-
 compat/mingw/compat-util.h              | 220 +++++++++++++
 compat/{mingw.h => mingw/posix.h}       | 216 +------------
 compat/msvc.c                           |   6 -
 compat/msvc/compat-util.c               |   6 +
 compat/msvc/compat-util.h               |   7 +
 compat/{msvc.h => msvc/posix.h}         |   8 +-
 compat/posix.h                          | 541 ++++++++++++++++++++++++++++++++
 config.mak.uname                        |   6 +-
 contrib/buildsystems/CMakeLists.txt     |   2 +-
 git-compat-util.h                       | 535 +------------------------------
 meson.build                             |   8 +-
 reftable/basics.c                       |  19 --
 reftable/basics.h                       | 123 +++++++-
 reftable/block.c                        |  16 +-
 reftable/blocksource.c                  |  21 +-
 reftable/iter.c                         |  20 +-
 reftable/merged.c                       |  27 +-
 reftable/pq.c                           |  40 ++-
 reftable/pq.h                           |   2 +-
 reftable/reader.c                       |  33 +-
 reftable/record.c                       | 109 ++++---
 reftable/record.h                       |   6 +-
 reftable/stack.c                        |  52 ++-
 reftable/system.c                       |   7 +
 reftable/system.h                       |   9 +-
 reftable/writer.c                       |  29 +-
 t/unit-tests/t-reftable-basics.c        |  28 +-
 t/unit-tests/t-reftable-pq.c            |  22 +-
 t/unit-tests/t-reftable-record.c        |  42 ++-
 31 files changed, 1242 insertions(+), 948 deletions(-)
Range-diff versus v3:
 1:  801e7bb30c =  1:  07f44927f1 reftable/stack: stop using `read_in_full()`
 2:  9b1b778a28 =  2:  c62404d62e reftable/stack: stop using `write_in_full()`
 3:  336b5ca3e1 =  3:  f936b06d4b reftable/blocksource: stop using `xmmap()`
 4:  ec74201d8d =  4:  ea203b9060 reftable/record: stop using `COPY_ARRAY()`
 5:  853740bd10 =  5:  c13aaeca16 reftable/record: stop using `BUG()` in `reftable_record_init()`
 6:  c6003aa731 =  6:  397b5321bf reftable/record: don't `BUG()` in `reftable_record_cmp()`
 7:  afbb02bdc7 =  7:  921d4ba9f0 reftable: stop using `BUG()` in trivial cases
 8:  fe34e5e425 =  8:  0c4396cc16 reftable/basics: stop using `st_mult()` in array allocators
 9:  405b6bdac7 =  9:  af50906b6d reftable/basics: provide wrappers for big endian conversion
10:  f0ca7a5f13 = 10:  1e931c747b reftable/reader: stop using `ARRAY_SIZE()` macro
11:  2e8152c509 = 11:  f66900df08 reftable/system: introduce `reftable_rand()`
12:  bdb708349d = 12:  99af9a50da reftable/stack: stop using `sleep_millisec()`
13:  6e59462638 = 13:  bdcd8744df reftable/basics: stop using `SWAP()` macro
14:  7c4e2eeb9b = 14:  41f637bd3f reftable/basics: stop using `UNUSED` annotation
15:  942eeca014 ! 15:  4dbbc71e11 compat/mingw: split out POSIX-related bits
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## compat/mingw.c => compat/mingw/compat-util.c ##
    +@@
    + #define USE_THE_REPOSITORY_VARIABLE
    + #define DISABLE_SIGN_COMPARE_WARNINGS
    + 
    +-#include "../git-compat-util.h"
    +-#include "win32.h"
    ++#include "../../git-compat-util.h"
    ++#include "../win32.h"
    + #include <aclapi.h>
    + #include <sddl.h>
    + #include <conio.h>
    + #include <wchar.h>
    +-#include "../strbuf.h"
    +-#include "../run-command.h"
    +-#include "../abspath.h"
    +-#include "../alloc.h"
    +-#include "win32/lazyload.h"
    +-#include "../config.h"
    +-#include "../environment.h"
    +-#include "../trace2.h"
    +-#include "../symlinks.h"
    +-#include "../wrapper.h"
    +-#include "dir.h"
    +-#include "gettext.h"
    ++#include "../../strbuf.h"
    ++#include "../../run-command.h"
    ++#include "../../abspath.h"
    ++#include "../../alloc.h"
    ++#include "../win32/lazyload.h"
    ++#include "../../config.h"
    ++#include "../../environment.h"
    ++#include "../../trace2.h"
    ++#include "../../symlinks.h"
    ++#include "../../wrapper.h"
    ++#include "../../dir.h"
    ++#include "../../gettext.h"
    + #define SECURITY_WIN32
    + #include <sspi.h>
    + 
     
      ## compat/mingw/compat-util.h (new) ##
     @@
    @@ compat/mingw/posix.h: char *mingw_query_user_email(void);
     -#endif
     +#endif /* COMPAT_MINGW_POSIX_H */
     
    - ## compat/msvc.c => compat/msvc/compat-util.c ##
    + ## compat/msvc.c (deleted) ##
     @@
    - #include <conio.h>
    - #include "../strbuf.h"
    - 
    +-#include "../git-compat-util.h"
    +-#include "win32.h"
    +-#include <conio.h>
    +-#include "../strbuf.h"
    +-
     -#include "mingw.c"
    -+#include "mingw/compat-util.c"
    +
    + ## compat/msvc/compat-util.c (new) ##
    +@@
    ++#include "../../git-compat-util.h"
    ++#include "../win32.h"
    ++#include <conio.h>
    ++#include "../../strbuf.h"
    ++
    ++#include "../mingw/compat-util.c"
     
      ## compat/msvc/compat-util.h (new) ##
     @@
16:  53151b2649 ! 16:  f0e9c6d3ee git-compat-util.h: split out POSIX-emulating bits
    @@ compat/posix.h (new)
     +#define HOST_NAME_MAX 256
     +#endif
     +
    -+#include "sane-ctype.h"
    ++#include "../sane-ctype.h"
     +
     +void git_stable_qsort(void *base, size_t nmemb, size_t size,
     +		      int(*compar)(const void *, const void *));
17:  b63a798513 = 17:  35a975dfe2 reftable: decouple from Git codebase by pulling in "compat/posix.h"
18:  f9edb54708 = 18:  1dde84b32c Makefile: skip reftable library for Coccinelle

--- base-commit: 8047765d092881ec4aef7dfc57772161eee7f0f5 change-id: 20241119-pks-reftable-drop-git-compat-util-470f2bfde562

Previous: Justin ToblerNext: Patrick Steinhardt
Message 80 of 123 in “reftable: stop using "git-compat-util.h"”
  1. 00/19 reftable: stop using "git-compat-util.h"Patrick Steinhardt, Jan 27, 2025
  2. 02/19 reftable/stack: stop using `write_in_full()`Patrick Steinhardt, Jan 27, 2025
  3. 01/19 reftable/stack: stop using `read_in_full()`Patrick Steinhardt, Jan 27, 2025
  4. Justin ToblerJan 27, 2025
  5. Patrick SteinhardtJan 28, 2025
  6. Junio C HamanoJan 28, 2025
  7. Patrick SteinhardtJan 29, 2025
  8. 03/19 reftable/blocksource: stop using `xmmap()`Patrick Steinhardt, Jan 27, 2025
  9. 04/19 reftable/record: stop using `COPY_ARRAY()`Patrick Steinhardt, Jan 27, 2025
  10. 05/19 reftable/record: stop using `BUG()` in `reftable_record_init()`Patrick Steinhardt, Jan 27, 2025
  11. Justin ToblerJan 27, 2025
  12. 06/19 reftable/record: don't `BUG()` in `reftable_record_cmp()`Patrick Steinhardt, Jan 27, 2025
  13. Justin ToblerJan 27, 2025
  14. 07/19 reftable: stop using `BUG()` in trivial casesPatrick Steinhardt, Jan 27, 2025
  15. 08/19 reftable/basics: stop using `st_mult()` in array allocatorsPatrick Steinhardt, Jan 27, 2025
  16. 09/19 reftable/basics: provide wrappers for big endian conversionPatrick Steinhardt, Jan 27, 2025
  17. 10/19 reftable/reader: stop using `ARRAY_SIZE()` macroPatrick Steinhardt, Jan 27, 2025
  18. 11/19 reftable/system: introduce `reftable_rand()`Patrick Steinhardt, Jan 27, 2025
  19. 12/19 reftable/stack: stop using `sleep_millisec()`Patrick Steinhardt, Jan 27, 2025
  20. 13/19 reftable/basics: stop using `SWAP()` macroPatrick Steinhardt, Jan 27, 2025
  21. 14/19 reftable/basics: stop using `UNUSED` annotationPatrick Steinhardt, Jan 27, 2025
  22. 15/19 compat/mingw: split out POSIX-related bitsPatrick Steinhardt, Jan 27, 2025
  23. 16/19 compat/msvc: split out POSIX-related bitsPatrick Steinhardt, Jan 27, 2025
  24. 17/19 git-compat-util.h: split out POSIX-emulating bitsPatrick Steinhardt, Jan 27, 2025
  25. Justin ToblerJan 27, 2025
  26. 18/19 reftable: decouple from Git codebase by pulling in "compat/posix.h"Patrick Steinhardt, Jan 27, 2025
  27. 19/19 Makefile: skip reftable library for CoccinellePatrick Steinhardt, Jan 27, 2025
  28. Junio C HamanoJan 27, 2025
  29. Patrick SteinhardtJan 28, 2025
  30. Junio C HamanoJan 28, 2025
  31. 00/20 reftable: stop using "git-compat-util.h"Patrick Steinhardt, Jan 28, 2025
  32. 01/20 reftable/stack: stop using `read_in_full()`Patrick Steinhardt, Jan 28, 2025
  33. 02/20 reftable/stack: stop using `write_in_full()`Patrick Steinhardt, Jan 28, 2025
  34. 04/20 reftable/record: stop using `COPY_ARRAY()`Patrick Steinhardt, Jan 28, 2025
  35. Justin ToblerJan 29, 2025
  36. Patrick SteinhardtFeb 3, 2025
  37. 03/20 reftable/blocksource: stop using `xmmap()`Patrick Steinhardt, Jan 28, 2025
  38. 05/20 reftable/record: stop using `BUG()` in `reftable_record_init()`Patrick Steinhardt, Jan 28, 2025
  39. 06/20 reftable/record: don't `BUG()` in `reftable_record_cmp()`Patrick Steinhardt, Jan 28, 2025
  40. 07/20 reftable: stop using `BUG()` in trivial casesPatrick Steinhardt, Jan 28, 2025
  41. 08/20 reftable/basics: stop using `st_mult()` in array allocatorsPatrick Steinhardt, Jan 28, 2025
  42. 09/20 reftable/basics: provide wrappers for big endian conversionPatrick Steinhardt, Jan 28, 2025
  43. 10/20 reftable/reader: stop using `ARRAY_SIZE()` macroPatrick Steinhardt, Jan 28, 2025
  44. 11/20 reftable/system: introduce `reftable_rand()`Patrick Steinhardt, Jan 28, 2025
  45. 12/20 reftable/stack: stop using `sleep_millisec()`Patrick Steinhardt, Jan 28, 2025
  46. 13/20 reftable/basics: stop using `SWAP()` macroPatrick Steinhardt, Jan 28, 2025
  47. 14/20 reftable/basics: stop using `UNUSED` annotationPatrick Steinhardt, Jan 28, 2025
  48. 15/20 compat: consistently resolve headers via project rootPatrick Steinhardt, Jan 28, 2025
  49. Johannes SixtJan 29, 2025
  50. Junio C HamanoJan 29, 2025
  51. Patrick SteinhardtFeb 3, 2025
  52. 16/20 compat/mingw: split out POSIX-related bitsPatrick Steinhardt, Jan 28, 2025
  53. 17/20 compat/msvc: split out POSIX-related bitsPatrick Steinhardt, Jan 28, 2025
  54. 18/20 git-compat-util.h: split out POSIX-emulating bitsPatrick Steinhardt, Jan 28, 2025
  55. 19/20 reftable: decouple from Git codebase by pulling in "compat/posix.h"Patrick Steinhardt, Jan 28, 2025
  56. 20/20 Makefile: skip reftable library for CoccinellePatrick Steinhardt, Jan 28, 2025
  57. Junio C HamanoJan 28, 2025
  58. Patrick SteinhardtJan 29, 2025
  59. Junio C HamanoJan 29, 2025
  60. 00/18 reftable: stop using "git-compat-util.h"Patrick Steinhardt, Feb 3, 2025
  61. 01/18 reftable/stack: stop using `read_in_full()`Patrick Steinhardt, Feb 3, 2025
  62. 02/18 reftable/stack: stop using `write_in_full()`Patrick Steinhardt, Feb 3, 2025
  63. 03/18 reftable/blocksource: stop using `xmmap()`Patrick Steinhardt, Feb 3, 2025
  64. 04/18 reftable/record: stop using `COPY_ARRAY()`Patrick Steinhardt, Feb 3, 2025
  65. 05/18 reftable/record: stop using `BUG()` in `reftable_record_init()`Patrick Steinhardt, Feb 3, 2025
  66. 06/18 reftable/record: don't `BUG()` in `reftable_record_cmp()`Patrick Steinhardt, Feb 3, 2025
  67. 07/18 reftable: stop using `BUG()` in trivial casesPatrick Steinhardt, Feb 3, 2025
  68. 08/18 reftable/basics: stop using `st_mult()` in array allocatorsPatrick Steinhardt, Feb 3, 2025
  69. 09/18 reftable/basics: provide wrappers for big endian conversionPatrick Steinhardt, Feb 3, 2025
  70. 10/18 reftable/reader: stop using `ARRAY_SIZE()` macroPatrick Steinhardt, Feb 3, 2025
  71. 11/18 reftable/system: introduce `reftable_rand()`Patrick Steinhardt, Feb 3, 2025
  72. 12/18 reftable/stack: stop using `sleep_millisec()`Patrick Steinhardt, Feb 3, 2025
  73. 13/18 reftable/basics: stop using `SWAP()` macroPatrick Steinhardt, Feb 3, 2025
  74. 14/18 reftable/basics: stop using `UNUSED` annotationPatrick Steinhardt, Feb 3, 2025
  75. 15/18 compat/mingw: split out POSIX-related bitsPatrick Steinhardt, Feb 3, 2025
  76. 16/18 git-compat-util.h: split out POSIX-emulating bitsPatrick Steinhardt, Feb 3, 2025
  77. 17/18 reftable: decouple from Git codebase by pulling in "compat/posix.h"Patrick Steinhardt, Feb 3, 2025
  78. 18/18 Makefile: skip reftable library for CoccinellePatrick Steinhardt, Feb 3, 2025
  79. Justin ToblerFeb 6, 2025
  80. 00/18 reftable: stop using "git-compat-util.h"Patrick Steinhardt, Feb 6, 2025
  81. 02/18 reftable/stack: stop using `write_in_full()`Patrick Steinhardt, Feb 6, 2025
  82. 01/18 reftable/stack: stop using `read_in_full()`Patrick Steinhardt, Feb 6, 2025
  83. 03/18 reftable/blocksource: stop using `xmmap()`Patrick Steinhardt, Feb 6, 2025
  84. 04/18 reftable/record: stop using `COPY_ARRAY()`Patrick Steinhardt, Feb 6, 2025
  85. 05/18 reftable/record: stop using `BUG()` in `reftable_record_init()`Patrick Steinhardt, Feb 6, 2025
  86. 06/18 reftable/record: don't `BUG()` in `reftable_record_cmp()`Patrick Steinhardt, Feb 6, 2025
  87. 07/18 reftable: stop using `BUG()` in trivial casesPatrick Steinhardt, Feb 6, 2025
  88. 08/18 reftable/basics: stop using `st_mult()` in array allocatorsPatrick Steinhardt, Feb 6, 2025
  89. 10/18 reftable/reader: stop using `ARRAY_SIZE()` macroPatrick Steinhardt, Feb 6, 2025
  90. 09/18 reftable/basics: provide wrappers for big endian conversionPatrick Steinhardt, Feb 6, 2025
  91. 11/18 reftable/system: introduce `reftable_rand()`Patrick Steinhardt, Feb 6, 2025
  92. 12/18 reftable/stack: stop using `sleep_millisec()`Patrick Steinhardt, Feb 6, 2025
  93. 13/18 reftable/basics: stop using `SWAP()` macroPatrick Steinhardt, Feb 6, 2025
  94. 14/18 reftable/basics: stop using `UNUSED` annotationPatrick Steinhardt, Feb 6, 2025
  95. Toon ClaesFeb 7, 2025
  96. Patrick SteinhardtFeb 7, 2025
  97. 15/18 compat/mingw: split out POSIX-related bitsPatrick Steinhardt, Feb 6, 2025
  98. 16/18 git-compat-util.h: split out POSIX-emulating bitsPatrick Steinhardt, Feb 6, 2025
  99. 17/18 reftable: decouple from Git codebase by pulling in "compat/posix.h"Patrick Steinhardt, Feb 6, 2025
  100. 18/18 Makefile: skip reftable library for CoccinellePatrick Steinhardt, Feb 6, 2025
  101. 00/18 reftable: stop using "git-compat-util.h"Patrick Steinhardt, Feb 7, 2025
  102. 01/18 reftable/stack: stop using `read_in_full()`Patrick Steinhardt, Feb 7, 2025
  103. 02/18 reftable/stack: stop using `write_in_full()`Patrick Steinhardt, Feb 7, 2025
  104. 03/18 reftable/blocksource: stop using `xmmap()`Patrick Steinhardt, Feb 7, 2025
  105. 04/18 reftable/record: stop using `COPY_ARRAY()`Patrick Steinhardt, Feb 7, 2025
  106. 05/18 reftable/record: stop using `BUG()` in `reftable_record_init()`Patrick Steinhardt, Feb 7, 2025
  107. 06/18 reftable/record: don't `BUG()` in `reftable_record_cmp()`Patrick Steinhardt, Feb 7, 2025
  108. 08/18 reftable/basics: stop using `st_mult()` in array allocatorsPatrick Steinhardt, Feb 7, 2025
  109. 07/18 reftable: stop using `BUG()` in trivial casesPatrick Steinhardt, Feb 7, 2025
  110. 09/18 reftable/basics: provide wrappers for big endian conversionPatrick Steinhardt, Feb 7, 2025
  111. 10/18 reftable/reader: stop using `ARRAY_SIZE()` macroPatrick Steinhardt, Feb 7, 2025
  112. 11/18 reftable/system: introduce `reftable_rand()`Patrick Steinhardt, Feb 7, 2025
  113. 12/18 reftable/stack: stop using `sleep_millisec()`Patrick Steinhardt, Feb 7, 2025
  114. 13/18 reftable/basics: stop using `SWAP()` macroPatrick Steinhardt, Feb 7, 2025
  115. 14/18 reftable/basics: introduce `REFTABLE_UNUSED` annotationPatrick Steinhardt, Feb 7, 2025
  116. 15/18 compat/mingw: split out POSIX-related bitsPatrick Steinhardt, Feb 7, 2025
  117. Johannes SixtFeb 9, 2025
  118. Junio C HamanoFeb 10, 2025
  119. Johannes SchindelinFeb 13, 2025
  120. Patrick SteinhardtFeb 17, 2025
  121. 16/18 git-compat-util.h: split out POSIX-emulating bitsPatrick Steinhardt, Feb 7, 2025
  122. 17/18 reftable: decouple from Git codebase by pulling in "compat/posix.h"Patrick Steinhardt, Feb 7, 2025
  123. 18/18 Makefile: skip reftable library for CoccinellePatrick Steinhardt, Feb 7, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.