git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 12/13] bulk-checkin: don't fetch promised objects on write

From
Patrick Steinhardt <ps@pks.im>
Date
Apr 23, 2025, 07:48 UTC
Message-ID
<20250423-pks-object-store-cleanups-v1-12-81f8411a5d08@pks.im>
In-Reply-To
<20250423-pks-object-store-cleanups-v1-0-81f8411a5d08@pks.im>

When writing objects via the bulk-checkin subsystem we first try to figure out whether an object already exists in the repository before we append it to the packfile. This check uses `repo_has_object_file()`, which knows to fetch promised objects by default. As such, if we were about to write an object that is promised, we'd fetch the object via the promisor and then skip writing it. This behaviour doesn't seem sensible: it should be significantly faster to take the locally-written object instead of faulting in objects from the promisor remote.

There is one counter-argument here: it could be that the bulk-checkin mechanism will end up writing an object to disk whose content collides with the object in the promisor remote. The local repository and its promisor remote would now have two objects with different contents but the same name. But the resulting behaviour would be wrong both when we prefer the fetched object, and also when prefering the written object:

  - When we prefer the written object we will now see a different world
    compared to everyone else who has the promised object.
  - When we prefer the fetched object we will end up with an object that
    is different compared to what the user just asked us to write. This
    seems even worse compared to the first scenario.

In an ideal world, we would protect against this by fetching the promised object and then performing a collision check. But this feels exceedingly expensive and ultimately rather pointless, as more common writing paths like `write_loose_object()` don't protect against this scenario either. And in any case we're talking about a local user that has write access to the repository anyway, so if they want to do any kind of mischieve they already can.

Change the behaviour so that we don't fault in the object via the promisor remote. We shouldn't have to worry about hash collisions too much (yet) as the mechanism is only used during local writes anyway. And even if there was a collision, prefering local data that we were just asked to write over data controlled by a potentially untrusted remote feels like the better failure mode.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 bulk-checkin.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bulk-checkin.c b/bulk-checkin.c
index c31c31b18d8..b182c456d69 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -130,7 +130,7 @@ static void flush_batch_fsync(void)
 static int already_written(struct bulk_checkin_packfile *state, struct object_id *oid)
 {
 	/* The object may already exist in the repository */
-	if (repo_has_object_file(the_repository, oid))
+	if (has_object(the_repository, oid, HAS_OBJECT_RECHECK_PACKED))
 		return 1;
 
 	/* Might want to keep the list sorted */
-- 
2.49.0.901.g37484f566f.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 19 of 55 in “object-store: a handful of cleanups”
  1. 00/13 object-store: a handful of cleanupsPatrick Steinhardt, Apr 23, 2025
  2. 01/13 object-store: move `struct packed_git` into "packfile.h"Patrick Steinhardt, Apr 23, 2025
  3. 02/13 object-store: drop `loose_object_path()`Patrick Steinhardt, Apr 23, 2025
  4. 03/13 object-store: move and rename `odb_pack_keep()`Patrick Steinhardt, Apr 23, 2025
  5. Karthik NayakApr 23, 2025
  6. 05/13 object-store: allow fetching objects via `has_object()`Patrick Steinhardt, Apr 23, 2025
  7. Karthik NayakApr 23, 2025
  8. 04/13 object-store: move function declarations to their respective subsystemsPatrick Steinhardt, Apr 23, 2025
  9. 06/13 treewide: trivial conversions of `repo_has_object_file()`Patrick Steinhardt, Apr 23, 2025
  10. 07/13 builtin/index-pack: don't fetch promised objects for collision checkPatrick Steinhardt, Apr 23, 2025
  11. Karthik NayakApr 23, 2025
  12. Patrick SteinhardtApr 25, 2025
  13. Karthik NayakApr 28, 2025
  14. 08/13 builtin/show-ref: don't fetch objects when printing refsPatrick Steinhardt, Apr 23, 2025
  15. 09/13 refs: don't fetch promisor objects in `ref_resolves_to_object()`Patrick Steinhardt, Apr 23, 2025
  16. Karthik NayakApr 23, 2025
  17. 10/13 http-walker: don't fetch objects via promisor remotesPatrick Steinhardt, Apr 23, 2025
  18. 11/13 list-objects: clarify how promised blobs are excludedPatrick Steinhardt, Apr 23, 2025
  19. 12/13 bulk-checkin: don't fetch promised objects on writePatrick Steinhardt, Apr 23, 2025
  20. 13/13 object-store: drop `repo_has_object_file()`Patrick Steinhardt, Apr 23, 2025
  21. Karthik NayakApr 23, 2025
  22. Patrick SteinhardtApr 25, 2025
  23. 00/13 object-store: a handful of cleanupsPatrick Steinhardt, Apr 25, 2025
  24. 01/13 object-store: move `struct packed_git` into "packfile.h"Patrick Steinhardt, Apr 25, 2025
  25. 02/13 object-store: drop `loose_object_path()`Patrick Steinhardt, Apr 25, 2025
  26. 03/13 object-store: move and rename `odb_pack_keep()`Patrick Steinhardt, Apr 25, 2025
  27. 04/13 object-store: move function declarations to their respective subsystemsPatrick Steinhardt, Apr 25, 2025
  28. 05/13 object-store: allow fetching objects via `has_object()`Patrick Steinhardt, Apr 25, 2025
  29. 06/13 treewide: trivial conversions of `repo_has_object_file()`Patrick Steinhardt, Apr 25, 2025
  30. Junio C HamanoApr 28, 2025
  31. 07/13 builtin/index-pack: don't fetch promised objects for collision checkPatrick Steinhardt, Apr 25, 2025
  32. Junio C HamanoApr 28, 2025
  33. Patrick SteinhardtApr 29, 2025
  34. 08/13 builtin/show-ref: don't fetch objects when printing refsPatrick Steinhardt, Apr 25, 2025
  35. Junio C HamanoApr 28, 2025
  36. 09/13 refs: don't fetch promisor objects in `ref_resolves_to_object()`Patrick Steinhardt, Apr 25, 2025
  37. Junio C HamanoApr 28, 2025
  38. 10/13 http-walker: don't fetch objects via promisor remotesPatrick Steinhardt, Apr 25, 2025
  39. Junio C HamanoApr 28, 2025
  40. 11/13 list-objects: clarify how promised blobs are excludedPatrick Steinhardt, Apr 25, 2025
  41. 12/13 bulk-checkin: don't fetch promised objects on writePatrick Steinhardt, Apr 25, 2025
  42. Junio C HamanoApr 28, 2025
  43. Patrick SteinhardtApr 29, 2025
  44. Junio C HamanoApr 29, 2025
  45. 13/13 object-store: drop `repo_has_object_file()`Patrick Steinhardt, Apr 25, 2025
  46. Karthik NayakApr 28, 2025
  47. 0/7 object-store: a handful of cleanupsPatrick Steinhardt, Apr 29, 2025
  48. 1/7 object-store: move `struct packed_git` into "packfile.h"Patrick Steinhardt, Apr 29, 2025
  49. 2/7 object-store: drop `loose_object_path()`Patrick Steinhardt, Apr 29, 2025
  50. 3/7 object-store: move and rename `odb_pack_keep()`Patrick Steinhardt, Apr 29, 2025
  51. 4/7 object-store: move function declarations to their respective subsystemsPatrick Steinhardt, Apr 29, 2025
  52. 5/7 object-store: allow fetching objects via `has_object()`Patrick Steinhardt, Apr 29, 2025
  53. 6/7 treewide: convert users of `repo_has_object_file()` to `has_object()`Patrick Steinhardt, Apr 29, 2025
  54. 7/7 object-store: drop `repo_has_object_file()`Patrick Steinhardt, Apr 29, 2025
  55. Junio C HamanoApr 29, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.