git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/7] clone: fix init of refdb with wrong object format

From
Patrick Steinhardt <ps@pks.im>
Date
Dec 6, 2023, 13:09 UTC
Message-ID
<ZXByiXj5rLUHgoAU@tanuki>
In-Reply-To
<cover.1701863960.git.ps@pks.im>
On Wed, Dec 06, 2023 at 01:39:44PM +0100, Patrick Steinhardt wrote:
Show 42 quoted lines
> Hi,
> 
> when using git-clone(1), we initialize the complete repository before we
> know about the object format used by the remote repository. This means
> that we'll potentially create the refdb with the wrong object format in
> case the local default object format and remote object format are not
> the same.
> 
> This isn't much of a problem in the context of the files backend, which
> never records the object format anyway. But it is a problem for the
> reftable backend, which indeed records the object format in the on-disk
> data structures. The result is thus a reftable with wrong object format.
> 
> This patch series aims to solve this issue by initializing the refdb at
> a later point after we have learned about the remote object format. This
> requires some careful reordering of code. Unfortunately, the end result
> is not easily verifiable and thus I didn't add tests here. But it does
> fix cloning of SHA256 repositories with the in-progress state of the
> reftable backend.
> 
> While at it I noticed that this actually fixes a bug with bundle URIs
> when the object formats diverge in this way.
> 
> The series is layed out as follows:
> 
>   - Patch 1 + 2: split out a function to create the refdb and make it
>     possible to skip its initialization in `init_db()`.
> 
>   - Patch 3: allows git-remote-curl(1) to work with repos that get
>     initialized during its lifetime.
> 
>   - Patch 4 - 6: address various corner cases where we access the refdb
>     before we learned about the object format.
> 
>   - Patch 7: move initialization of the refdb to happen after we have
>     learned about the object format.
> 
> This patch series is actually the last incompatibility for the reftable
> backend that I have found. All tests except for the files-backend
> specific ones pass now with the current state I have at [1], which is
> currently at e6f2f592b7 (t: skip tests which are incompatible with
> reftable, 2023-11-24)

I forgot to add the link to the merge request that contains the current reftable backend's implementation in case anybody is interested: https://gitlab.com/gitlab-org/git/-/merge_requests/58.

Patrick
Show 25 quoted lines
> Thanks in advance for your reviews!
> 
> Patrick
> 
> Patrick Steinhardt (7):
>   setup: extract function to create the refdb
>   setup: allow skipping creation of the refdb
>   remote-curl: rediscover repository when fetching refs
>   builtin/clone: fix bundle URIs with mismatching object formats
>   builtin/clone: set up sparse checkout later
>   builtin/clone: skip reading HEAD when retrieving remote
>   builtin/clone: create the refdb with the correct object format
> 
>  builtin/clone.c             |  65 ++++++++++++----------
>  remote-curl.c               |   7 ++-
>  remote.c                    |  26 +++++----
>  remote.h                    |   1 +
>  setup.c                     | 106 +++++++++++++++++++++---------------
>  setup.h                     |   6 +-
>  t/t5558-clone-bundle-uri.sh |  18 ++++++
>  7 files changed, 140 insertions(+), 89 deletions(-)
> 
> -- 
> 2.43.0
> 
Previous: Patrick SteinhardtNext: Junio C Hamano
Message 19 of 33 in “clone: fix init of refdb with wrong object format”
  1. 0/7 clone: fix init of refdb with wrong object formatPatrick Steinhardt, Dec 6, 2023
  2. 1/7 setup: extract function to create the refdbPatrick Steinhardt, Dec 6, 2023
  3. Karthik NayakDec 6, 2023
  4. Patrick SteinhardtDec 7, 2023
  5. Junio C HamanoDec 8, 2023
  6. Patrick SteinhardtDec 11, 2023
  7. Karthik NayakDec 11, 2023
  8. 2/7 setup: allow skipping creation of the refdbPatrick Steinhardt, Dec 6, 2023
  9. 3/7 remote-curl: rediscover repository when fetching refsPatrick Steinhardt, Dec 6, 2023
  10. Junio C HamanoDec 8, 2023
  11. Patrick SteinhardtDec 11, 2023
  12. 4/7 builtin/clone: fix bundle URIs with mismatching object formatsPatrick Steinhardt, Dec 6, 2023
  13. Karthik NayakDec 6, 2023
  14. Patrick SteinhardtDec 7, 2023
  15. Junio C HamanoDec 8, 2023
  16. 5/7 builtin/clone: set up sparse checkout laterPatrick Steinhardt, Dec 6, 2023
  17. 6/7 builtin/clone: skip reading HEAD when retrieving remotePatrick Steinhardt, Dec 6, 2023
  18. 7/7 builtin/clone: create the refdb with the correct object formatPatrick Steinhardt, Dec 6, 2023
  19. Patrick SteinhardtDec 6, 2023
  20. Junio C HamanoDec 10, 2023
  21. Patrick SteinhardtDec 11, 2023
  22. Junio C HamanoDec 11, 2023
  23. Patrick SteinhardtDec 11, 2023
  24. brian m. carlsonDec 11, 2023
  25. 0/7 clone: fix init of refdb with wrong object formatPatrick Steinhardt, Dec 12, 2023
  26. 1/7 setup: extract function to create the refdbPatrick Steinhardt, Dec 12, 2023
  27. 2/7 setup: allow skipping creation of the refdbPatrick Steinhardt, Dec 12, 2023
  28. 3/7 remote-curl: rediscover repository when fetching refsPatrick Steinhardt, Dec 12, 2023
  29. 4/7 builtin/clone: fix bundle URIs with mismatching object formatsPatrick Steinhardt, Dec 12, 2023
  30. 5/7 builtin/clone: set up sparse checkout laterPatrick Steinhardt, Dec 12, 2023
  31. 6/7 builtin/clone: skip reading HEAD when retrieving remotePatrick Steinhardt, Dec 12, 2023
  32. 7/7 builtin/clone: create the refdb with the correct object formatPatrick Steinhardt, Dec 12, 2023
  33. Junio C HamanoDec 12, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.