git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 04/10] refs/files: support symbolic and root refs in initial transaction

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 8, 2024, 09:34 UTC
Message-ID
<20241108-pks-refs-optimize-migrations-v1-4-7fd37fa80e35@pks.im>
In-Reply-To
<20241108-pks-refs-optimize-migrations-v1-0-7fd37fa80e35@pks.im>

The "files" backend has implemented special logic when committing the first transactions in an otherwise empty ref store: instead of writing all refs as separate loose files, it instead knows to write them all into a "packed-refs" file directly. This is significantly more efficient than having to write each of the refs as separate "loose" ref.

The only user of this optimization is git-clone(1), which only uses this mechanism to write regular refs. Consequently, the implementation does not know how to handle both symbolic and root refs. While fine in the context of git-clone(1), this keeps us from using the mechanism in more cases.

Adapt the logic to also support symbolic and root refs by using a second transaction that we use for all of the refs that need to be written as loose refs.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 refs/files-backend.c | 44 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 34 insertions(+), 10 deletions(-)
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 3ed18475a72aa4798d15b2912c37b4caabd47aac..116d4259697b20583cb2db34ed47025e8781cd42 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -2995,6 +2995,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,
 	int ret = 0;
 	struct string_list affected_refnames = STRING_LIST_INIT_NODUP;
 	struct ref_transaction *packed_transaction = NULL;
+	struct ref_transaction *loose_transaction = NULL;
 
 	assert(err);
 
@@ -3040,6 +3041,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,
 		if ((update->flags & REF_HAVE_OLD) &&
 		    !is_null_oid(&update->old_oid))
 			BUG("initial ref transaction with old_sha1 set");
+
 		if (refs_verify_refname_available(&refs->base, update->refname,
 						  &affected_refnames, NULL,
 						  err)) {
@@ -3048,26 +3050,48 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,
 		}
 
 		/*
-		 * Add a reference creation for this reference to the
-		 * packed-refs transaction:
+		 * packed-refs don't support symbolic refs and root refs, so we
+		 * have to queue these references via the loose transaction.
 		 */
-		ref_transaction_add_update(packed_transaction, update->refname,
-					   update->flags & ~REF_HAVE_OLD,
-					   &update->new_oid, &update->old_oid,
-					   NULL, NULL, NULL);
+		if (update->new_target || is_root_ref(update->refname)) {
+			if (!loose_transaction) {
+				loose_transaction = ref_store_transaction_begin(&refs->base, 0, err);
+				if (!loose_transaction) {
+					ret = TRANSACTION_GENERIC_ERROR;
+					goto cleanup;
+				}
+			}
+
+			ref_transaction_add_update(loose_transaction, update->refname,
+						   update->flags & ~REF_HAVE_OLD,
+						   update->new_target ? NULL : &update->new_oid, NULL,
+						   update->new_target, NULL, NULL);
+		} else {
+			ref_transaction_add_update(packed_transaction, update->refname,
+						   update->flags & ~REF_HAVE_OLD,
+						   &update->new_oid, &update->old_oid,
+						   NULL, NULL, NULL);
+		}
 	}
 
-	if (packed_refs_lock(refs->packed_ref_store, 0, err)) {
+	if (packed_refs_lock(refs->packed_ref_store, 0, err) ||
+	    ref_transaction_commit(packed_transaction, err)) {
 		ret = TRANSACTION_GENERIC_ERROR;
 		goto cleanup;
 	}
+	packed_refs_unlock(refs->packed_ref_store);
 
-	if (ref_transaction_commit(packed_transaction, err)) {
-		ret = TRANSACTION_GENERIC_ERROR;
+	if (loose_transaction) {
+		if (ref_transaction_prepare(loose_transaction, err) ||
+		    ref_transaction_commit(loose_transaction, err)) {
+			ret = TRANSACTION_GENERIC_ERROR;
+			goto cleanup;
+		}
 	}
 
-	packed_refs_unlock(refs->packed_ref_store);
 cleanup:
+	if (loose_transaction)
+		ref_transaction_free(loose_transaction);
 	if (packed_transaction)
 		ref_transaction_free(packed_transaction);
 	transaction->state = REF_TRANSACTION_CLOSED;
-- 
2.47.0.229.g8f8d6eee53.dirty
Previous: karthik nayakNext: karthik nayak
Message 11 of 42 in “refs: optimize ref format migrations”
  1. 00/10 refs: optimize ref format migrationsPatrick Steinhardt, Nov 8, 2024
  2. 01/10 refs: allow passing flags when setting up a transactionPatrick Steinhardt, Nov 8, 2024
  3. karthik nayakNov 11, 2024
  4. Patrick SteinhardtNov 11, 2024
  5. 02/10 refs/files: move logic to commit initial transactionPatrick Steinhardt, Nov 8, 2024
  6. 03/10 refs: introduce "initial" transaction flagPatrick Steinhardt, Nov 8, 2024
  7. 05/10 refs: use "initial" transaction semantics to migrate refsPatrick Steinhardt, Nov 8, 2024
  8. karthik nayakNov 11, 2024
  9. 06/10 refs: skip collision checks in initial transactionsPatrick Steinhardt, Nov 8, 2024
  10. karthik nayakNov 11, 2024
  11. 04/10 refs/files: support symbolic and root refs in initial transactionPatrick Steinhardt, Nov 8, 2024
  12. karthik nayakNov 11, 2024
  13. Patrick SteinhardtNov 11, 2024
  14. 09/10 reftable/block: rename `block_writer::buf` variablePatrick Steinhardt, Nov 8, 2024
  15. 08/10 reftable/writer: optimize allocations by using a scratch bufferPatrick Steinhardt, Nov 8, 2024
  16. 07/10 refs: don't normalize log messages with `REF_SKIP_CREATE_REFLOG`Patrick Steinhardt, Nov 8, 2024
  17. 10/10 reftable/block: optimize allocations by using scratch bufferPatrick Steinhardt, Nov 8, 2024
  18. karthik nayakNov 11, 2024
  19. Patrick SteinhardtNov 11, 2024
  20. Junio C HamanoNov 20, 2024
  21. Patrick SteinhardtNov 20, 2024
  22. Christian CouderNov 20, 2024
  23. Patrick SteinhardtNov 25, 2024
  24. 00/10 refs: optimize ref format migrationsPatrick Steinhardt, Nov 20, 2024
  25. 01/10 refs: allow passing flags when setting up a transactionPatrick Steinhardt, Nov 20, 2024
  26. Christian CouderNov 20, 2024
  27. 02/10 refs/files: move logic to commit initial transactionPatrick Steinhardt, Nov 20, 2024
  28. 03/10 refs: introduce "initial" transaction flagPatrick Steinhardt, Nov 20, 2024
  29. 04/10 refs/files: support symbolic and root refs in initial transactionPatrick Steinhardt, Nov 20, 2024
  30. 05/10 refs: use "initial" transaction semantics to migrate refsPatrick Steinhardt, Nov 20, 2024
  31. 06/10 refs: skip collision checks in initial transactionsPatrick Steinhardt, Nov 20, 2024
  32. Christian CouderNov 20, 2024
  33. Patrick SteinhardtNov 25, 2024
  34. Kristoffer HaugsbakkNov 20, 2024
  35. Patrick SteinhardtNov 25, 2024
  36. 07/10 refs: don't normalize log messages with `REF_SKIP_CREATE_REFLOG`Patrick Steinhardt, Nov 20, 2024
  37. 08/10 reftable/writer: optimize allocations by using a scratch bufferPatrick Steinhardt, Nov 20, 2024
  38. Christian CouderNov 20, 2024
  39. Patrick SteinhardtNov 25, 2024
  40. 09/10 reftable/block: rename `block_writer::buf` variablePatrick Steinhardt, Nov 20, 2024
  41. 10/10 reftable/block: optimize allocations by using scratch bufferPatrick Steinhardt, Nov 20, 2024
  42. Christian CouderNov 20, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.