git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 3/4] refs.c: add a transaction function to append a reflog entry

From
Stefan Beller <sbeller@google.com>
Date
Nov 27, 2014, 05:34 UTC
Message-ID
<1417066485-24921-4-git-send-email-sbeller@google.com>
In-Reply-To
<1417066485-24921-1-git-send-email-sbeller@google.com>
From: Ronnie Sahlberg <sahlberg@google.com>

Define a new transaction update type, UPDATE_LOG, and a new function transaction_update_reflog. This function will lock the reflog and append an entry to it during transaction commit. We can pass a flag to this function, which can truncate the the reflog file before we write the update.

When performing a reflog transaction update, only write to the reflog iff msg is non-NULL. This can then be combined with REFLOG_TRUNCATE to perform an update that only truncates but does not write. This change only affects whether or not a reflog entry should be generated and written. If msg == NULL then no such entry will be written.

Orthogonal to this we have a boolean flag REFLOG_TRUNCATE which is used to tell the transaction system to "truncate the reflog and thus discard all previous users".

At the current time the only place where we use msg == NULL is also the place, where we use REFLOG_TRUNCATE. Even though these two settings are currently only ever used together it still makes sense to have them through two separate knobs.

This allows future consumers of this API that may want to do things
differently. For example someone can do:
  msg="Reflog truncated by Bob because ..." + REFLOG_TRUNCATE
and have it truncate the log and have it start fresh with an initial message
that explains the log was truncated. This API allows that.

During one transaction we allow to make multiple reflog updates to the same ref. This means we only need to lock the reflog once, during the first update that touches the reflog, and that all further updates can just write the reflog entry since the reflog is already locked.

This allows us to write code (internally in refs.c) such as:
t = transaction_begin()
transaction_reflog_update(t, "foo", REFLOG_TRUNCATE, NULL);
loop-over-something...
   transaction_reflog_update(t, "foo", 0, <message>);
transaction_commit(t)

where we first truncate the reflog and then build the new content one line at a time.

While this technically looks like O(n2) behavior it is not that bad. We only do this loop for transactions that cover a single ref during reflog expire. This means that the linear search inside transaction_update_reflog() will find the match on the very first entry thus making it O(1) and not O(n) or our usecases. Thus the whole expire becomes O(n) instead of O(n2). If in the future we start doing this for many refs in one single transaction we might want to optimize this. But there is no need to complexify the code and optimize for future usecases that might never materialize at this stage.

Instead of recording the line by line reflog updates in memory, use a tempfile: .git/tmp_reflog_XXXXXX which we write the entries to as the transaction is built. Then just rename this file onto the destination reflog file when the transaction is committed.

todo: This patch needs to add an atexit() thingy as well to ensure that any remaining files are unlinked on exit, just like the lock_file() thing does.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
This is a complete rewrite from previous series. Sorry no diff to previous versions.
 refs.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 refs.h |  21 ++++++++++
 2 files changed, 167 insertions(+), 2 deletions(-)
diff --git a/refs.c b/refs.c
index 84e086f..a1af703 100644
--- a/refs.c
+++ b/refs.c
@@ -3517,7 +3517,8 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)
 }
 
 enum transaction_update_type {
-	UPDATE_SHA1 = 0
+	UPDATE_SHA1 = 0,
+	UPDATE_LOG = 1
 };
 
 /**
@@ -3530,11 +3531,18 @@ struct ref_update {
 	enum transaction_update_type update_type;
 	unsigned char new_sha1[20];
 	unsigned char old_sha1[20];
-	int flags; /* REF_NODEREF? */
+	int flags;  /* The flags to transaction_update_ref[log] are defined
+		     * in refs.h
+		     */
 	int have_old; /* 1 if old_sha1 is valid, 0 otherwise */
 	struct ref_lock *lock;
 	int type;
 	char *msg;
+
+	/* used by reflog updates */
+	char *tmp_reflog;
+	int reflog_fd;
+
 	const char refname[FLEX_ARRAY];
 };
 
@@ -3580,6 +3588,7 @@ void transaction_free(struct transaction *transaction)
 		return;
 
 	for (i = 0; i < transaction->nr; i++) {
+		free(transaction->updates[i]->tmp_reflog);
 		free(transaction->updates[i]->msg);
 		free(transaction->updates[i]);
 	}
@@ -3601,6 +3610,116 @@ static struct ref_update *add_update(struct transaction *transaction,
 	return update;
 }
 
+int transaction_update_reflog(struct transaction *transaction,
+			      const char *refname,
+			      const unsigned char *new_sha1,
+			      const unsigned char *old_sha1,
+			      const char *email,
+			      unsigned long timestamp, int tz,
+			      const char *msg, int flags,
+			      struct strbuf *err)
+{
+	struct ref_update *update;
+	struct strbuf buf = STRBUF_INIT;
+	int i;
+
+	if (transaction->state != TRANSACTION_OPEN)
+		die("BUG: update_reflog called for transaction that is not open");
+
+	/* Check if there is another reflog update for this ref already. */
+	for (i = 0; transaction->nr > 0 && i < transaction->nr; i++) {
+		if (transaction->updates[i]->update_type != UPDATE_LOG)
+			continue;
+		if (!strcmp(transaction->updates[i]->refname,
+			    refname)) {
+			break;
+		}
+	}
+	/* When starting the transaction or when we did not find the ref,
+	 * we will need to create a new temporary file. */
+	if (transaction->nr == 0 || i == transaction->nr) {
+		int orig_fd;
+		update = add_update(transaction, refname, UPDATE_LOG);
+
+		orig_fd = open(git_path("logs/%s", refname), O_RDONLY);
+		if (orig_fd < 0) {
+			const char *str = "Cannot open reflog for '%s'. %s";
+
+			strbuf_addf(err, str, refname, strerror(errno));
+			transaction->state = TRANSACTION_CLOSED;
+			return 1;
+		}
+
+		update->tmp_reflog = xstrdup(git_path(".tmp_reflog_XXXXXX"));
+		update->reflog_fd = mkstemp(update->tmp_reflog);
+		if (update->reflog_fd == -1) {
+			const char *str = "Could not create temporary "
+			  "reflog for '%s'. %s";
+
+			close(orig_fd);
+			strbuf_addf(err, str, refname, strerror(errno));
+			transaction->state = TRANSACTION_CLOSED;
+			return 1;
+		}
+		if (adjust_shared_perm(update->tmp_reflog)) {
+			strbuf_addf(err, "Could not fix permission bits for "
+				    "reflog: %s. %s",
+				    update->tmp_reflog, strerror(errno));
+			close(orig_fd);
+			unlink_or_warn(update->tmp_reflog);
+			close(update->reflog_fd);
+			update->reflog_fd = -1;
+			transaction->state = TRANSACTION_CLOSED;
+			return 1;
+		}
+		if (copy_fd(orig_fd, update->reflog_fd)) {
+			strbuf_addf(err, "Could not copy reflog: %s. %s",
+				    refname, strerror(errno));
+			close(orig_fd);
+			unlink_or_warn(update->tmp_reflog);
+			close(update->reflog_fd);
+			update->reflog_fd = -1;
+			transaction->state = TRANSACTION_CLOSED;
+			return 1;
+		}
+		close(orig_fd);
+	} else {
+		update = transaction->updates[i];
+	}
+
+	if (flags & REFLOG_TRUNCATE) {
+		if (lseek(update->reflog_fd, 0, SEEK_SET) < 0 ||
+			ftruncate(update->reflog_fd, 0)) {
+			strbuf_addf(err, "Could not truncate reflog: %s. %s",
+				    refname, strerror(errno));
+			unlink_or_warn(update->tmp_reflog);
+			close(update->reflog_fd);
+			update->reflog_fd = -1;
+			transaction->state = TRANSACTION_CLOSED;
+			return 1;
+		}
+	}
+	if (email)
+		strbuf_addf(&buf, "%s %lu %+05d", email, timestamp, tz);
+
+	if (msg &&
+	    log_ref_write_fd(update->reflog_fd,
+			     old_sha1, new_sha1,
+			     buf.buf, msg)) {
+		strbuf_addf(err, "Could not write to reflog: %s. %s",
+			    refname, strerror(errno));
+		unlink_or_warn(update->tmp_reflog);
+		close(update->reflog_fd);
+		update->reflog_fd = -1;
+		transaction->state = TRANSACTION_CLOSED;
+		strbuf_release(&buf);
+		return 1;
+	}
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 int transaction_update_ref(struct transaction *transaction,
 			   const char *refname,
 			   const unsigned char *new_sha1,
@@ -3815,6 +3934,31 @@ int transaction_commit(struct transaction *transaction,
 	}
 	for (i = 0; i < delnum; i++)
 		unlink_or_warn(git_path("logs/%s", delnames[i]));
+
+	/* Commit all reflog files */
+	for (i = 0; i < n; i++) {
+		struct ref_update *update = updates[i];
+
+		if (update->update_type != UPDATE_LOG)
+			continue;
+		if (update->reflog_fd == -1)
+			continue;
+		if (close(update->reflog_fd) == -1) {
+			error("Could not commit temporary reflog: %s. %s",
+			      update->refname, strerror(errno));
+			update->reflog_fd = -1;
+			continue;
+		}
+		update->reflog_fd = -1;
+		if (rename(update->tmp_reflog,
+			   git_path("logs/%s", update->refname))) {
+			error("Could not commit reflog: %s. %s",
+			      update->refname, strerror(errno));
+			update->reflog_fd = -1;
+			continue;
+		}
+	}
+
 	clear_loose_ref_cache(&ref_cache);
 
 cleanup:
diff --git a/refs.h b/refs.h
index 556adfd..9f70b89 100644
--- a/refs.h
+++ b/refs.h
@@ -328,6 +328,27 @@ int transaction_delete_ref(struct transaction *transaction,
 			   struct strbuf *err);
 
 /*
+ * Flags controlling transaction_update_reflog().
+ * REFLOG_TRUNCATE: Truncate the reflog.
+ *
+ * Flags >= 0x100 are reserved for internal use.
+ */
+#define REFLOG_TRUNCATE 0x01
+/*
+ * Append a reflog entry for refname. If the REFLOG_TRUNCATE flag is set
+ * this update will first truncate the reflog before writing the entry.
+ * If msg is NULL no update will be written to the log.
+ */
+int transaction_update_reflog(struct transaction *transaction,
+			      const char *refname,
+			      const unsigned char *new_sha1,
+			      const unsigned char *old_sha1,
+			      const char *email,
+			      unsigned long timestamp, int tz,
+			      const char *msg, int flags,
+			      struct strbuf *err);
+
+/*
  * Commit all of the changes that have been queued in transaction, as
  * atomically as possible.
  *

-- 
2.2.0.rc3
Previous: Stefan BellerNext: Stefan Beller
Message 30 of 31 in “ref-transactions-reflog”
  1. 00/14 ref-transactions-reflogStefan Beller, Nov 18, 2014
  2. 01/14 refs.c: make ref_transaction_create a wrapper for ref_transaction_updateStefan Beller, Nov 18, 2014
  3. 02/14 refs.c: make ref_transaction_delete a wrapper for ref_transaction_updateStefan Beller, Nov 18, 2014
  4. 03/14 refs.c: rename the transaction functionsStefan Beller, Nov 18, 2014
  5. 04/14 refs.c: add a function to append a reflog entry to a fdStefan Beller, Nov 18, 2014
  6. 05/14 refs.c: add a new update_type field to ref_updateStefan Beller, Nov 18, 2014
  7. 06/14 refs.c: add a transaction function to append a reflog entryStefan Beller, Nov 18, 2014
  8. 07/14 refs.c: add a flag to allow reflog updates to truncate the logStefan Beller, Nov 18, 2014
  9. 08/14 refs.c: only write reflog update if msg is non-NULLStefan Beller, Nov 18, 2014
  10. 09/14 refs.c: allow multiple reflog updates during a single transactionStefan Beller, Nov 18, 2014
  11. 10/14 reflog.c: use a reflog transaction when writing during expireStefan Beller, Nov 18, 2014
  12. 11/14 refs.c: rename log_ref_setup to create_reflogStefan Beller, Nov 18, 2014
  13. 12/14 refs.c: Remove unlock_ref/close_ref/commit_ref from the refs apiStefan Beller, Nov 18, 2014
  14. 13/14 refs.c: remove lock_any_ref_for_updateStefan Beller, Nov 18, 2014
  15. 14/14 refs.c: allow deleting refs with a broken sha1Stefan Beller, Nov 18, 2014
  16. Michael HaggertyNov 18, 2014
  17. Ronnie SahlbergNov 18, 2014
  18. Michael HaggertyNov 18, 2014
  19. Junio C HamanoNov 18, 2014
  20. Michael HaggertyNov 18, 2014
  21. Junio C HamanoNov 18, 2014
  22. Stefan BellerNov 19, 2014
  23. Jonathan NiederNov 20, 2014
  24. Junio C HamanoNov 20, 2014
  25. Michael HaggertyNov 20, 2014
  26. Jonathan NiederNov 20, 2014
  27. 0/4 Using transactions for the reflogStefan Beller, Nov 27, 2014
  28. 1/4 refs.c: rename the transaction functionsStefan Beller, Nov 27, 2014
  29. 2/4 refs.c: add a new update_type field to ref_updateStefan Beller, Nov 27, 2014
  30. 3/4 refs.c: add a transaction function to append a reflog entryStefan Beller, Nov 27, 2014
  31. 4/4 reflog.c: use a reflog transaction when writing during expireStefan Beller, Nov 27, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.