{"thread":{"id":"62635","subject":"[PATCH v2 1/8] refs: include committer info in `ref_update` struct","startedAt":"2024-12-13T10:36:53Z","lastAt":"2024-12-24T10:34:33Z","messageCount":63,"participants":["Karthik Nayak","Christian Couder","Patrick Steinhardt","karthik nayak","Junio C Hamano","Toon Claes"],"isPatch":true,"patchVersion":2,"patchTotal":8},"messages":[{"id":"509068","messageId":"20241213-320-git-refs-migrate-reflogs-v2-1-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 1/8] refs: include committer info in `ref_update` struct","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:46Z","receivedAt":"2024-12-13T10:36:53Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference backends obtain the committer information from\n`git_committer_info(0)` when adding a reflog. The upcoming patches\nintroduce support for migrating reflogs between the reference backends.\nThis requires an interface to creating reflogs, including custom\ncommitter information.\n\nAdd a new field `committer_info` to the `ref_update` struct, which is\nthen used by the reference backends. If there is no `committer_info`\nprovided, the reference backends default to using\n`git_committer_info(0)`. The field itself cannot be set to\n`git_committer_info(0)` since the values are dynamic and must be\nobtained right when the reflog is being committed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 1 +\n refs/files-backend.c | 24 ++++++++++++++----------\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 12 +++++++++++-\n 4 files changed, 27 insertions(+), 11 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 762f3e324d59c60cd4f05c2f257e54de8deb00e5..f003e51c6bf5229bfbce8ce61ffad7cdba0572e0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1151,6 +1151,7 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tfree(transaction->updates[i]->msg);\n+\t\tfree(transaction->updates[i]->committer_info);\n \t\tfree((char *)transaction->updates[i]->new_target);\n \t\tfree((char *)transaction->updates[i]->old_target);\n \t\tfree(transaction->updates[i]);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 64f51f0da905a9a8a1ac4109c6b0a9a85a355db7..6078668c99ee254e794e3ba49689aa34e6022efd 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1858,6 +1858,9 @@ static int log_ref_write_fd(int fd, const struct object_id *old_oid,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint ret = 0;\n \n+\tif (!committer)\n+\t\tcommitter = git_committer_info(0);\n+\n \tstrbuf_addf(&sb, \"%s %s %s\", oid_to_hex(old_oid), oid_to_hex(new_oid), committer);\n \tif (msg && *msg) {\n \t\tstrbuf_addch(&sb, '\\t');\n@@ -1871,8 +1874,10 @@ static int log_ref_write_fd(int fd, const struct object_id *old_oid,\n }\n \n static int files_log_ref_write(struct files_ref_store *refs,\n-\t\t\t const char *refname, const struct object_id *old_oid,\n-\t\t\t const struct object_id *new_oid, const char *msg,\n+\t\t\t const char *refname,\n+\t\t\t const struct object_id *old_oid,\n+\t\t\t const struct object_id *new_oid,\n+\t\t\t const char *committer_info, const char *msg,\n \t\t\t int flags, struct strbuf *err)\n {\n \tint logfd, result;\n@@ -1889,8 +1894,7 @@ static int files_log_ref_write(struct files_ref_store *refs,\n \n \tif (logfd < 0)\n \t\treturn 0;\n-\tresult = log_ref_write_fd(logfd, old_oid, new_oid,\n-\t\t\t\t git_committer_info(0), msg);\n+\tresult = log_ref_write_fd(logfd, old_oid, new_oid, committer_info, msg);\n \tif (result) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \t\tint save_errno = errno;\n@@ -1974,8 +1978,7 @@ static int commit_ref_update(struct files_ref_store *refs,\n \tfiles_assert_main_repository(refs, \"commit_ref_update\");\n \n \tclear_loose_ref_cache(refs);\n-\tif (files_log_ref_write(refs, lock->ref_name,\n-\t\t\t\t&lock->old_oid, oid,\n+\tif (files_log_ref_write(refs, lock->ref_name, &lock->old_oid, oid, NULL,\n \t\t\t\tlogmsg, flags, err)) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \t\tstrbuf_addf(err, \"cannot update the ref '%s': %s\",\n@@ -2007,9 +2010,9 @@ static int commit_ref_update(struct files_ref_store *refs,\n \t\tif (head_ref && (head_flag & REF_ISSYMREF) &&\n \t\t !strcmp(head_ref, lock->ref_name)) {\n \t\t\tstruct strbuf log_err = STRBUF_INIT;\n-\t\t\tif (files_log_ref_write(refs, \"HEAD\",\n-\t\t\t\t\t\t&lock->old_oid, oid,\n-\t\t\t\t\t\tlogmsg, flags, &log_err)) {\n+\t\t\tif (files_log_ref_write(refs, \"HEAD\", &lock->old_oid,\n+\t\t\t\t\t\toid, NULL, logmsg, flags,\n+\t\t\t\t\t\t&log_err)) {\n \t\t\t\terror(\"%s\", log_err.buf);\n \t\t\t\tstrbuf_release(&log_err);\n \t\t\t}\n@@ -2969,7 +2972,8 @@ static int parse_and_write_reflog(struct files_ref_store *refs,\n \t}\n \n \tif (files_log_ref_write(refs, lock->ref_name, &lock->old_oid,\n-\t\t\t\t&update->new_oid, update->msg, update->flags, err)) {\n+\t\t\t\t&update->new_oid, update->committer_info,\n+\t\t\t\tupdate->msg, update->flags, err)) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \n \t\tstrbuf_addf(err, \"cannot update the ref '%s': %s\",\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 58aa56d1b27c85d606ed7c8c0d908e4b87d1066b..0fd95cdacd99e4a728c22f5286f6b3f0f360c110 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -113,6 +113,7 @@ struct ref_update {\n \tvoid *backend_data;\n \tunsigned int type;\n \tchar *msg;\n+\tchar *committer_info;\n \n \t/*\n \t * If this ref_update was split off of a symref update via\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 647ef9b05b1dc9a376ed054330b487f7595c5caa..e882602487c66261d586a94101bb1b4e9a2ed60e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1379,11 +1379,21 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t}\n \n \t\t\tif (create_reflog) {\n+\t\t\t\tstruct ident_split c;\n+\n \t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n \t\t\t\tlog = &logs[logs_nr++];\n \t\t\t\tmemset(log, 0, sizeof(*log));\n \n-\t\t\t\tfill_reftable_log_record(log, &committer_ident);\n+\t\t\t\tif (u->committer_info) {\n+\t\t\t\t\tif (split_ident_line(&c, u->committer_info,\n+\t\t\t\t\t\t\t strlen(u->committer_info)))\n+\t\t\t\t\t\tBUG(\"failed splitting committer info\");\n+\t\t\t\t} else {\n+\t\t\t\t\tc = committer_ident;\n+\t\t\t\t}\n+\n+\t\t\t\tfill_reftable_log_record(log, &c);\n \t\t\t\tlog->update_index = ts;\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n\n-- \n2.47.1\n\n"},{"id":"509070","messageId":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241209-320-git-refs-migrate-reflogs-v1-0-d4bc37ee860f@gmail.com","subject":"[PATCH v2 0/8] refs: add reflog support to `git refs migrate`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:45Z","receivedAt":"2024-12-13T10:36:53Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs migrate` command was introduced in\n25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n2024-06-06) to support migrating from one reference backend to another.\n\nOne limitation of the feature was that it didn't support migrating\nrepositories which contained reflogs. This isn't a requirement on the\nserver side as repositories are stored as bare repositories (which do\nnot contain any reflogs). Clients however generally use reflogs and\nuntil now couldn't use the `git refs migrate` command to migrate their\nrepositories to the new reftable format.\n\nOne of the issues for adding reflog support is that the ref transactions\ndon't support reflogs additions:\n 1. While there is REF_LOG_ONLY flag, there is no function to utilize\n the flag and add reflogs.\n 2. reference backends generally sort the updates by the refname. This\n wouldn't work for reflogs which need to ensure that they maintain the\n order of creation.\n 3. In the files backend, reflog entries are added by obtaining locks\n on the refs themselves. This means each update in the transaction, will\n obtain a ref_lock. This paradigm fails to accompany the fact that there\n could be multiple reflog updates for a refname in a single transaction.\n 4. The backends check for duplicate entries, which doesn't make sense\n in the context of adding multiple reflogs for a given refname.\n\nWe overcome these issue we make the following changes:\n - Update the ref_update structure to also include the committer\n information. Using this, we can add a new function which only adds\n reflog updates to the transaction.\n - Add an index field to the ref_update structure, this will help order\n updates in pre-defined order, this fixes #2.\n - While the ideal fix for #3 would be to actually introduce reflog\n locks, this wouldn't be possible without breaking backward\n compatibility. So we add a count field to the existing ref_lock. With\n this, multiple reflog updates can share a single ref_lock.\n\nOverall, this series is a bit more involved, and I would appreciate it\nif it receives a bit more scrutiny.\n\nThe series is based on top of e66fd72e97 (The fourteenth batch,\n2024-12-06) with `kn/reftable-writer-log-write-verify` merged in.\n\nSigned-off-by: Karthik Nayak \n---\nChanges in v2:\n- Split patch 5 into two separate patches. This should make it easier to\n review and reduce cognitive load in a single patch.\n- In reftable backend, instead of using `strmapint` to ensure we have\n new update_indexes for reflogs with the same refname, we now use the\n already available `update->index` field to increment the update_index.\n- Cleanup the code and follow some of the better practices.\n- Add some clarity to the commit messages.\n- Link to v1: https://lore.kernel.org/r/20241209-320-git-refs-migrate-reflogs-v1-0-d4bc37ee860f@gmail.com\n\n---\nKarthik Nayak (8):\n refs: include committer info in `ref_update` struct\n refs: add `index` field to `struct ref_udpate`\n refs/files: add count field to ref_lock\n refs: extract out refname verification in transactions\n refs: add `committer_info` to `ref_transaction_add_update()`\n refs: introduce the `ref_transaction_update_reflog` function\n refs: allow multiple reflog entries for the same refname\n refs: add support for migrating reflogs\n\n Documentation/git-refs.txt | 2 -\n refs.c | 168 +++++++++++++++++++++++++++++++++------------\n refs.h | 12 ++++\n refs/files-backend.c | 131 +++++++++++++++++++++++------------\n refs/refs-internal.h | 9 +++\n refs/reftable-backend.c | 53 +++++++++++---\n t/t1460-refs-migrate.sh | 73 ++++++++++++++------\n 7 files changed, 328 insertions(+), 120 deletions(-)\n---\n\nRange-diff versus v1:\n\n1: 627104646c ! 1: 81a493ae20 refs: include committer info in `ref_update` struct\n @@ refs/files-backend.c: static int commit_ref_update(struct files_ref_store *refs,\n \t\t\tstruct strbuf log_err = STRBUF_INIT;\n -\t\t\tif (files_log_ref_write(refs, \"HEAD\",\n -\t\t\t\t\t\t&lock->old_oid, oid,\n +-\t\t\t\t\t\tlogmsg, flags, &log_err)) {\n +\t\t\tif (files_log_ref_write(refs, \"HEAD\", &lock->old_oid,\n -+\t\t\t\t\t\toid, git_committer_info(0),\n - \t\t\t\t\t\tlogmsg, flags, &log_err)) {\n ++\t\t\t\t\t\toid, NULL, logmsg, flags,\n ++\t\t\t\t\t\t&log_err)) {\n \t\t\t\terror(\"%s\", log_err.buf);\n \t\t\t\tstrbuf_release(&log_err);\n + \t\t\t}\n @@ refs/files-backend.c: static int parse_and_write_reflog(struct files_ref_store *refs,\n \t}\n \n2: 7bc7c7cb1b = 2: 3c4d53de5c refs: add `index` field to `struct ref_udpate`\n3: 807fff5d50 ! 3: 598ad493af refs/files: add count field to ref_lock\n @@ Commit message\n \n The upcoming patches will add support for reflog updates in ref\n transaction. This means, in a particular transaction we want to have ref\n - updates and reflog updates. For refs, in a given transaction there can\n - only be one update. But, we can theoretically have multiple reflog\n - updates in a given transaction.\n + updates and reflog updates. For a given ref in a given transaction there\n + can be at most one update. But we can theoretically have multiple reflog\n + updates for a given ref in a given transaction. A great example of this\n + would be when migrating reflogs from one backend to another. There we\n + would batch all the reflog updates for a given reference in a single\n + transaction.\n \n The current flow does not support this, because currently refs & reflogs\n are treated as a single entity and capture the lock together. To\n @@ refs/files-backend.c: struct ref_lock {\n \tchar *ref_name;\n \tstruct lock_file lk;\n \tstruct object_id old_oid;\n -+\t/* count keeps track of users of the lock */\n -+\tunsigned int count;\n ++\tunsigned int count; /* track users of the lock (ref update + reflog updates) */\n };\n \n struct files_ref_store {\n @@ refs/files-backend.c: static int lock_ref_for_update(struct files_ref_store *ref\n -\t\tchar *reason;\n +\tlock = strmap_get(&backend_data->ref_locks, update->refname);\n +\tif (lock) {\n -+\t\tlock->count = lock->count + 1;\n ++\t\tlock->count++;\n +\t} else {\n +\t\tret = lock_raw_ref(refs, update->refname, mustexist,\n +\t\t\t\t affected_refnames,\n4: 33473ad609 < -: ---------- refs: extract out refname verification in transactions\n5: ae85d9e340 < -: ---------- refs: introduce the `ref_transaction_update_reflog` function\n-: ---------- > 4: 64a3cbd91d refs: extract out refname verification in transactions\n-: ---------- > 5: 888f96facb refs: add `committer_info` to `ref_transaction_add_update()`\n-: ---------- > 6: 9253e1ceda refs: introduce the `ref_transaction_update_reflog` function\n6: b49872a6b5 ! 7: 6d12784851 refs: allow multiple reflog entries for the same refname\n @@ Metadata\n ## Commit message ##\n refs: allow multiple reflog entries for the same refname\n \n - The reference transaction only allows a update for a given reference to\n - avoid conflicts. This, however, isn't an issue for reflogs. There are no\n - conflicts to be resolved in reflogs and when migrating reflogs between\n - backends we'd have multiple reflog entries for the same refname.\n + The reference transaction only allows a single update for a given\n + reference to avoid conflicts. This, however, isn't an issue for reflogs.\n + There are no conflicts to be resolved in reflogs and when migrating\n + reflogs between backends we'd have multiple reflog entries for the same\n + refname.\n \n So allow multiple reflog updates within a single transaction. Also the\n reflog creation logic isn't exposed to the end user. While this might\n change in the future, currently, this reduces the scope of issues to\n think about.\n \n - This is required to add reflog migration support to `git refs migrate`\n - which currently doesn't support it.\n + In the reftable backend, the writer sorts all updates based on the\n + update_index before writing to the block. When there are multiple\n + reflogs for a given refname, it is essential that the order of the\n + reflogs is maintained. So add the `index` value to the `update_index`.\n + The `index` field is only be set when multiple reflog entries for a\n + given refname are added and as such in most scenarios the old behavior\n + remains.\n +\n + This is required to add reflog migration support to `git refs migrate`.\n \n Signed-off-by: Karthik Nayak \n \n @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n \n \t/*\n @@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n + \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n ++\tuint64_t max_update_index = ts;\n \tconst char *committer_info;\n -+\tstruct strintmap logs_ts;\n \tint ret = 0;\n \n - \tcommitter_info = git_committer_info(0);\n -@@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n - \n - \tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n - \n -+\tstrintmap_init(&logs_ts, ts);\n -+\n - \treftable_writer_set_limits(writer, ts, ts);\n - \n - \tfor (i = 0; i < arg->updates_nr; i++) {\n -@@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n - \n - \t\t\tif (create_reflog) {\n - \t\t\t\tstruct ident_split c;\n -+\t\t\t\tuint64_t update_index;\n - \n - \t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n - \t\t\t\tlog = &logs[logs_nr++];\n @@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t\t}\n \n \t\t\t\tfill_reftable_log_record(log, &c);\n -\t\t\t\tlog->update_index = ts;\n +\n -+\t\t\t\tupdate_index = strintmap_get(&logs_ts, u->refname);\n -+\t\t\t\tlog->update_index = update_index;\n -+\t\t\t\tstrintmap_set(&logs_ts, u->refname, update_index+1);\n ++\t\t\t\t/*\n ++\t\t\t\t * Updates are sorted by the writer. So updates for the same\n ++\t\t\t\t * refname need to contain different update indices.\n ++\t\t\t\t */\n ++\t\t\t\tlog->update_index = ts + u->index;\n ++\n ++\t\t\t\t/*\n ++\t\t\t\t * Note the max update_index so the limit can be set later on.\n ++\t\t\t\t */\n ++\t\t\t\tif (log->update_index > max_update_index)\n ++\t\t\t\t\tmax_update_index = log->update_index;\n +\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n \t\t\t\t u->new_oid.hash, GIT_MAX_RAWSZ);\n @@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n - \n - done:\n - \tassert(ret != REFTABLE_API_ERROR);\n -+\tstrintmap_clear(&logs_ts);\n - \tfor (i = 0; i < logs_nr; i++)\n - \t\treftable_log_record_release(&logs[i]);\n - \tfree(logs);\n + \t * and log blocks.\n + \t */\n + \tif (logs) {\n ++\t\treftable_writer_set_limits(writer, ts, max_update_index);\n ++\n + \t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n + \t\tif (ret < 0)\n + \t\t\tgoto done;\n7: 0df1005b2b ! 8: 06dba479d6 refs: add support for migrating reflogs\n @@ refs.c: int ref_update_check_old_target(const char *referent, struct ref_update\n }\n \n struct migration_data {\n -+\tunsigned int index;\n ++\tunsigned int reflog_index;\n \tstruct ref_store *old_refs;\n \tstruct ref_transaction *transaction;\n \tstruct strbuf *errbuf;\n @@ refs.c: static int migrate_one_ref(const char *refname, const char *referent UNU\n +\tdata.old_refs = migration_data->old_refs;\n +\tdata.transaction = migration_data->transaction;\n +\tdata.errbuf = migration_data->errbuf;\n -+\tdata.index = &migration_data->index;\n ++\tdata.index = &migration_data->reflog_index;\n +\n +\treturn refs_for_each_reflog_ent(migration_data->old_refs, refname,\n +\t\t\t\t\tmigrate_one_reflog_entry, &data);\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \t * Worktrees complicate the migration because every worktree has a\n \t * separate ref storage. While it should be feasible to implement, this\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n - \t * 1. Set up a new temporary directory and initialize it with the new\n - \t * format. This is where all refs will be migrated into.\n - \t *\n --\t * 2. Enumerate all refs and write them into the new ref storage.\n --\t * This operation is safe as we do not yet modify the main\n -+\t * 2. Enumerate all refs and reflogs and write them into the new ref\n -+\t * storage. This operation is safe as we do not yet modify the main\n + \t * This operation is safe as we do not yet modify the main\n \t * repository.\n \t *\n - \t * 3. If we're in dry-run mode then we are done and can hand over the\n +-\t * 3. If we're in dry-run mode then we are done and can hand over the\n ++\t * 3. Enumerate all reflogs and write them into the new ref storage.\n ++\t * This operation is safe as we do not yet modify the main\n ++\t * repository.\n ++\t *\n ++\t * 4. If we're in dry-run mode then we are done and can hand over the\n + \t * directory to the caller for inspection. If not, we now start\n + \t * with the destructive part.\n + \t *\n +-\t * 4. Delete the old ref storage from disk. As we have a copy of refs\n ++\t * 5. Delete the old ref storage from disk. As we have a copy of refs\n + \t * in the new ref storage it's okay(ish) if we now get interrupted\n + \t * as there is an equivalent copy of all refs available.\n + \t *\n +-\t * 5. Move the new ref storage files into place.\n ++\t * 6. Move the new ref storage files into place.\n + \t *\n +-\t * 6. Change the repository format to the new ref format.\n ++\t * 7. Change the repository format to the new ref format.\n + \t */\n + \tstrbuf_addf(&new_gitdir, \"%s/%s\", old_refs->gitdir, \"ref_migration.XXXXXX\");\n + \tif (!mkdtemp(new_gitdir.buf)) {\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \tif (ret < 0)\n \t\tgoto done;\n \n -+\tdata.index = 1;\n ++\tdata.reflog_index = 1;\n +\tret = refs_for_each_reflog(old_refs, migrate_one_reflog, &data);\n +\tif (ret < 0)\n +\t\tgoto done;\n\n\n--- \n\nbase-commit: 09245f4b75863f4e94dac7feebaafce53a26965f\nchange-id: 20241111-320-git-refs-migrate-reflogs-a53e3a6cffc9\n\nThanks\n- Karthik\n\n"},{"id":"509069","messageId":"20241213-320-git-refs-migrate-reflogs-v2-2-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 2/8] refs: add `index` field to `struct ref_udpate`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:47Z","receivedAt":"2024-12-13T10:36:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reftable backend, sorts its updates by refname before applying them,\nthis ensures that the references are stored sorted. When migrating\nreflogs from one backend to another, the order of the reflogs must be\nmaintained. Add a new `index` field to the `ref_update` struct to\nfacilitate this.\n\nThis field is used in the reftable backend's sort comparison function\n`transaction_update_cmp`, to ensure that indexed fields maintain their\norder.\n\nSigned-off-by: Karthik Nayak \n---\n refs/refs-internal.h | 7 +++++++\n refs/reftable-backend.c | 13 +++++++++++--\n 2 files changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 0fd95cdacd99e4a728c22f5286f6b3f0f360c110..f5c733d099f0c6f1076a25f4f77d9d5eb345ec87 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -115,6 +115,13 @@ struct ref_update {\n \tchar *msg;\n \tchar *committer_info;\n \n+\t/*\n+\t * The index overrides the default sort algorithm. This is needed\n+\t * when migrating reflogs and we want to ensure we carry over the\n+\t * same order.\n+\t */\n+\tunsigned int index;\n+\n \t/*\n \t * If this ref_update was split off of a symref update via\n \t * split_symref_update(), then this member points at that\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex e882602487c66261d586a94101bb1b4e9a2ed60e..c008f20be719fec3af6a8f81c821cb9c263764d7 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1279,8 +1279,17 @@ static int reftable_be_transaction_abort(struct ref_store *ref_store UNUSED,\n \n static int transaction_update_cmp(const void *a, const void *b)\n {\n-\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n-\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+\tstruct reftable_transaction_update *update_a = (struct reftable_transaction_update *)a;\n+\tstruct reftable_transaction_update *update_b = (struct reftable_transaction_update *)b;\n+\n+\t/*\n+\t * If there is an index set, it should take preference (default is 0).\n+\t * This ensures that updates with indexes are sorted amongst themselves.\n+\t */\n+\tif (update_a->update->index || update_b->update->index)\n+\t\treturn update_a->update->index - update_b->update->index;\n+\n+\treturn strcmp(update_a->update->refname, update_b->update->refname);\n }\n \n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n\n-- \n2.47.1\n\n"},{"id":"509071","messageId":"20241213-320-git-refs-migrate-reflogs-v2-3-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 3/8] refs/files: add count field to ref_lock","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:48Z","receivedAt":"2024-12-13T10:36:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When refs are updated in the files-backend, a lock is obtained for the\ncorresponding file path. This is the case even for reflogs, i.e. a lock\nis obtained on the reference path instead of the reflog path. This\nworks, since generally, reflogs are updated alongside the ref.\n\nThe upcoming patches will add support for reflog updates in ref\ntransaction. This means, in a particular transaction we want to have ref\nupdates and reflog updates. For a given ref in a given transaction there\ncan be at most one update. But we can theoretically have multiple reflog\nupdates for a given ref in a given transaction. A great example of this\nwould be when migrating reflogs from one backend to another. There we\nwould batch all the reflog updates for a given reference in a single\ntransaction.\n\nThe current flow does not support this, because currently refs & reflogs\nare treated as a single entity and capture the lock together. To\nseparate this, add a count field to ref_lock. With this, multiple\nupdates can hold onto a single ref_lock and the lock will only be\nreleased when all of them release the lock.\n\nThis patch only adds the `count` field to `ref_lock` and adds the logic\nto increment and decrement the lock. In a follow up commit, we'll\nseparate the reflog update logic from ref updates and utilize this\nfunctionality.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 58 +++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 39 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6078668c99ee254e794e3ba49689aa34e6022efd..02cb4907d8659e87a227fed4f60a5f6606be8764 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -71,6 +71,7 @@ struct ref_lock {\n \tchar *ref_name;\n \tstruct lock_file lk;\n \tstruct object_id old_oid;\n+\tunsigned int count; /* track users of the lock (ref update + reflog updates) */\n };\n \n struct files_ref_store {\n@@ -638,9 +639,12 @@ int parse_loose_ref_contents(const struct git_hash_algo *algop,\n \n static void unlock_ref(struct ref_lock *lock)\n {\n-\trollback_lock_file(&lock->lk);\n-\tfree(lock->ref_name);\n-\tfree(lock);\n+\tlock->count--;\n+\tif (!lock->count) {\n+\t\trollback_lock_file(&lock->lk);\n+\t\tfree(lock->ref_name);\n+\t\tfree(lock);\n+\t}\n }\n \n /*\n@@ -696,6 +700,7 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t*lock_p = CALLOC_ARRAY(lock, 1);\n \n \tlock->ref_name = xstrdup(refname);\n+\tlock->count = 1;\n \tfiles_ref_path(refs, &ref_file, refname);\n \n retry:\n@@ -1169,6 +1174,7 @@ static struct ref_lock *lock_ref_oid_basic(struct files_ref_store *refs,\n \t\tgoto error_return;\n \n \tlock->ref_name = xstrdup(refname);\n+\tlock->count = 1;\n \n \tif (raceproof_create_file(ref_file.buf, create_reflock, &lock->lk)) {\n \t\tunable_to_lock_message(ref_file.buf, errno, err);\n@@ -2535,6 +2541,12 @@ static int check_old_oid(struct ref_update *update, struct object_id *oid,\n \treturn -1;\n }\n \n+struct files_transaction_backend_data {\n+\tstruct ref_transaction *packed_transaction;\n+\tint packed_refs_locked;\n+\tstruct strmap ref_locks;\n+};\n+\n /*\n * Prepare for carrying out update:\n * - Lock the reference referred to by update.\n@@ -2557,11 +2569,14 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n {\n \tstruct strbuf referent = STRBUF_INIT;\n \tint mustexist = ref_update_expects_existing_old_ref(update);\n+\tstruct files_transaction_backend_data *backend_data;\n \tint ret = 0;\n \tstruct ref_lock *lock;\n \n \tfiles_assert_main_repository(refs, \"lock_ref_for_update\");\n \n+\tbackend_data = transaction->backend_data;\n+\n \tif ((update->flags & REF_HAVE_NEW) && ref_update_has_null_new_value(update))\n \t\tupdate->flags |= REF_DELETING;\n \n@@ -2572,18 +2587,25 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\tgoto out;\n \t}\n \n-\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t affected_refnames,\n-\t\t\t &lock, &referent,\n-\t\t\t &update->type, err);\n-\tif (ret) {\n-\t\tchar *reason;\n+\tlock = strmap_get(&backend_data->ref_locks, update->refname);\n+\tif (lock) {\n+\t\tlock->count++;\n+\t} else {\n+\t\tret = lock_raw_ref(refs, update->refname, mustexist,\n+\t\t\t\t affected_refnames,\n+\t\t\t\t &lock, &referent,\n+\t\t\t\t &update->type, err);\n+\t\tif (ret) {\n+\t\t\tchar *reason;\n+\n+\t\t\treason = strbuf_detach(err, NULL);\n+\t\t\tstrbuf_addf(err, \"cannot lock ref '%s': %s\",\n+\t\t\t\t ref_update_original_update_refname(update), reason);\n+\t\t\tfree(reason);\n+\t\t\tgoto out;\n+\t\t}\n \n-\t\treason = strbuf_detach(err, NULL);\n-\t\tstrbuf_addf(err, \"cannot lock ref '%s': %s\",\n-\t\t\t ref_update_original_update_refname(update), reason);\n-\t\tfree(reason);\n-\t\tgoto out;\n+\t\tstrmap_put(&backend_data->ref_locks, update->refname, lock);\n \t}\n \n \tupdate->backend_data = lock;\n@@ -2730,11 +2752,6 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \treturn ret;\n }\n \n-struct files_transaction_backend_data {\n-\tstruct ref_transaction *packed_transaction;\n-\tint packed_refs_locked;\n-};\n-\n /*\n * Unlock any references in `transaction` that are still locked, and\n * mark the transaction closed.\n@@ -2767,6 +2784,8 @@ static void files_transaction_cleanup(struct files_ref_store *refs,\n \t\tif (backend_data->packed_refs_locked)\n \t\t\tpacked_refs_unlock(refs->packed_ref_store);\n \n+\t\tstrmap_clear(&backend_data->ref_locks, 0);\n+\n \t\tfree(backend_data);\n \t}\n \n@@ -2796,6 +2815,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto cleanup;\n \n \tCALLOC_ARRAY(backend_data, 1);\n+\tstrmap_init(&backend_data->ref_locks);\n \ttransaction->backend_data = backend_data;\n \n \t/*\n\n-- \n2.47.1\n\n"},{"id":"509072","messageId":"20241213-320-git-refs-migrate-reflogs-v2-4-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 4/8] refs: extract out refname verification in transactions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:49Z","receivedAt":"2024-12-13T10:36:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Unless the `REF_SKIP_REFNAME_VERIFICATION` flag is set for an update,\nthe refname of the update is verified for:\n\n - Ensuring it is not a pseudoref.\n - Checking the refname format.\n\nThese checks will also be needed in a following commit where the\nfunction to add reflog updates to the transaction is introduced. Extract\nthe code out into a new static function.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 37 +++++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f003e51c6bf5229bfbce8ce61ffad7cdba0572e0..9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1196,6 +1196,28 @@ struct ref_update *ref_transaction_add_update(\n \treturn update;\n }\n \n+static int transaction_refname_valid(const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t unsigned int flags, struct strbuf *err)\n+{\n+\tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n+\t\treturn 1;\n+\n+\tif (is_pseudo_ref(refname)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n+\t\t\t refname);\n+\t\treturn 0;\n+\t} else if ((new_oid && !is_null_oid(new_oid)) ?\n+\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n+\t\t !refname_is_safe(refname)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n+\t\t\t refname);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const struct object_id *new_oid,\n@@ -1213,21 +1235,8 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\treturn -1;\n \t}\n \n-\tif (!(flags & REF_SKIP_REFNAME_VERIFICATION) &&\n-\t ((new_oid && !is_null_oid(new_oid)) ?\n-\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n-\t\t\t !refname_is_safe(refname))) {\n-\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n-\t\t\t refname);\n+\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n \t\treturn -1;\n-\t}\n-\n-\tif (!(flags & REF_SKIP_REFNAME_VERIFICATION) &&\n-\t is_pseudo_ref(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n-\t\t\t refname);\n-\t\treturn -1;\n-\t}\n \n \tif (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS)\n \t\tBUG(\"illegal flags 0x%x passed to ref_transaction_update()\", flags);\n\n-- \n2.47.1\n\n"},{"id":"509073","messageId":"20241213-320-git-refs-migrate-reflogs-v2-5-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:50Z","receivedAt":"2024-12-13T10:36:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_transaction_add_update()` creates the `ref_update` struct. To\nfacilitate addition of reflogs in the next commit, the function needs to\naccommodate setting the `committer_info` field in the struct. So modify\nthe function to also take `committer_info` as an argument and set it\naccordingly.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 9 +++++++--\n refs/files-backend.c | 14 ++++++++------\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 6 ++++--\n 4 files changed, 20 insertions(+), 10 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801..428ca256f3e5860554e9a7fa42a8368bb2689b31 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1166,6 +1166,7 @@ struct ref_update *ref_transaction_add_update(\n \t\tconst struct object_id *new_oid,\n \t\tconst struct object_id *old_oid,\n \t\tconst char *new_target, const char *old_target,\n+\t\tconst char *committer_info,\n \t\tconst char *msg)\n {\n \tstruct ref_update *update;\n@@ -1190,8 +1191,12 @@ struct ref_update *ref_transaction_add_update(\n \t\toidcpy(&update->new_oid, new_oid);\n \tif ((flags & REF_HAVE_OLD) && old_oid)\n \t\toidcpy(&update->old_oid, old_oid);\n-\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n+\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n+\t\tif (committer_info)\n+\t\t\tupdate->committer_info = xstrdup(committer_info);\n+\n \t\tupdate->msg = normalize_reflog_message(msg);\n+\t}\n \n \treturn update;\n }\n@@ -1253,7 +1258,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n-\t\t\t\t old_target, msg);\n+\t\t\t\t old_target, NULL, msg);\n \treturn 0;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 02cb4907d8659e87a227fed4f60a5f6606be8764..255fed8354cae982f785b1b85340e2a1eeecf2a6 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1270,7 +1270,7 @@ static void prune_ref(struct files_ref_store *refs, struct ref_to_prune *r)\n \tref_transaction_add_update(\n \t\t\ttransaction, r->name,\n \t\t\tREF_NO_DEREF | REF_HAVE_NEW | REF_HAVE_OLD | REF_IS_PRUNING,\n-\t\t\tnull_oid(), &r->oid, NULL, NULL, NULL);\n+\t\t\tnull_oid(), &r->oid, NULL, NULL, NULL, NULL);\n \tif (ref_transaction_commit(transaction, &err))\n \t\tgoto cleanup;\n \n@@ -2417,7 +2417,7 @@ static int split_head_update(struct ref_update *update,\n \t\t\ttransaction, \"HEAD\",\n \t\t\tupdate->flags | REF_LOG_ONLY | REF_NO_DEREF,\n \t\t\t&update->new_oid, &update->old_oid,\n-\t\t\tNULL, NULL, update->msg);\n+\t\t\tNULL, NULL, update->committer_info, update->msg);\n \n \t/*\n \t * Add \"HEAD\". This insertion is O(N) in the transaction\n@@ -2481,7 +2481,8 @@ static int split_symref_update(struct ref_update *update,\n \t\t\ttransaction, referent, new_flags,\n \t\t\tupdate->new_target ? NULL : &update->new_oid,\n \t\t\tupdate->old_target ? NULL : &update->old_oid,\n-\t\t\tupdate->new_target, update->old_target, update->msg);\n+\t\t\tupdate->new_target, update->old_target, NULL,\n+\t\t\tupdate->msg);\n \n \tnew_update->parent_update = update;\n \n@@ -2914,7 +2915,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t\tpacked_transaction, update->refname,\n \t\t\t\t\tREF_HAVE_NEW | REF_NO_DEREF,\n \t\t\t\t\t&update->new_oid, NULL,\n-\t\t\t\t\tNULL, NULL, NULL);\n+\t\t\t\t\tNULL, NULL, NULL, NULL);\n \t\t}\n \t}\n \n@@ -3094,12 +3095,13 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t\tref_transaction_add_update(loose_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n \t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n-\t\t\t\t\t\t update->new_target, NULL, NULL);\n+\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n+\t\t\t\t\t\t NULL);\n \t\t} else {\n \t\t\tref_transaction_add_update(packed_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n \t\t\t\t\t\t &update->new_oid, &update->old_oid,\n-\t\t\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t\t\t NULL, NULL, update->committer_info, NULL);\n \t\t}\n \t}\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f5c733d099f0c6f1076a25f4f77d9d5eb345ec87..79b287c5ec5c7d8f759869cf93cda405640186dc 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -162,6 +162,7 @@ struct ref_update *ref_transaction_add_update(\n \t\tconst struct object_id *new_oid,\n \t\tconst struct object_id *old_oid,\n \t\tconst char *new_target, const char *old_target,\n+\t\tconst char *committer_info,\n \t\tconst char *msg);\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex c008f20be719fec3af6a8f81c821cb9c263764d7..b2e3ba877de9e59fea5a4d066eb13e60ef22a32b 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1078,7 +1078,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tnew_update = ref_transaction_add_update(\n \t\t\t\t\ttransaction, \"HEAD\",\n \t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n-\t\t\t\t\t&u->new_oid, &u->old_oid, NULL, NULL, u->msg);\n+\t\t\t\t\t&u->new_oid, &u->old_oid, NULL, NULL, NULL,\n+\t\t\t\t\tu->msg);\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n@@ -1161,7 +1162,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t\ttransaction, referent.buf, new_flags,\n \t\t\t\t\tu->new_target ? NULL : &u->new_oid,\n \t\t\t\t\tu->old_target ? NULL : &u->old_oid,\n-\t\t\t\t\tu->new_target, u->old_target, u->msg);\n+\t\t\t\t\tu->new_target, u->old_target,\n+\t\t\t\t\tu->committer_info, u->msg);\n \n \t\t\t\tnew_update->parent_update = u;\n \n\n-- \n2.47.1\n\n"},{"id":"509074","messageId":"20241213-320-git-refs-migrate-reflogs-v2-6-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:51Z","receivedAt":"2024-12-13T10:36:57Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new function `ref_transaction_update_reflog`, for clients to\nadd a reflog update to a transaction. While the existing function\n`ref_transaction_update` also allows clients to add a reflog entry, this\nfunction does a few things more, It:\n - Enforces that only a reflog entry is added and does not update the\n ref itself.\n - Allows the users to also provide the committer information. This\n means clients can add reflog entries with custom committer\n information.\n\nA follow up commit will utilize this function to add reflog support to\n`git refs migrate`.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 44 ++++++++++++++++++++++++++++++++++++++------\n refs.h | 12 ++++++++++++\n refs/files-backend.c | 24 ++++++++++++++++--------\n 3 files changed, 66 insertions(+), 14 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 428ca256f3e5860554e9a7fa42a8368bb2689b31..9f539369bc94a25594adc3e95847f2fe72f58a08 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1203,20 +1203,21 @@ struct ref_update *ref_transaction_add_update(\n \n static int transaction_refname_valid(const char *refname,\n \t\t\t\t const struct object_id *new_oid,\n-\t\t\t\t unsigned int flags, struct strbuf *err)\n+\t\t\t\t unsigned int flags, unsigned int reflog,\n+\t\t\t\t struct strbuf *err)\n {\n \tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n \t\treturn 1;\n \n \tif (is_pseudo_ref(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n-\t\t\t refname);\n+\t\tconst char *what = reflog ? \"reflog for pseudoref\" : \"pseudoref\";\n+\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t} else if ((new_oid && !is_null_oid(new_oid)) ?\n \t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n \t\t !refname_is_safe(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n-\t\t\t refname);\n+\t\tconst char *what = reflog ? \"reflog with bad name\" : \"ref with bad name\";\n+\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t}\n \n@@ -1240,7 +1241,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\treturn -1;\n \t}\n \n-\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n+\tif (!transaction_refname_valid(refname, new_oid, flags, 0, err))\n \t\treturn -1;\n \n \tif (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS)\n@@ -1259,6 +1260,37 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n \t\t\t\t old_target, NULL, msg);\n+\n+\treturn 0;\n+}\n+\n+int ref_transaction_update_reflog(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t const char *committer_info, unsigned int flags,\n+\t\t\t\t const char *msg, unsigned int index,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct ref_update *update;\n+\n+\tassert(err);\n+\n+\tif (!transaction_refname_valid(refname, new_oid, flags, 1, err))\n+\t\treturn -1;\n+\n+\tflags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\n+\tupdate = ref_transaction_add_update(transaction, refname, flags,\n+\t\t\t\t\t new_oid, old_oid, NULL, NULL,\n+\t\t\t\t\t committer_info, msg);\n+\t/*\n+\t * While we do set the old_oid value, we unset the flag to skip\n+\t * old_oid verification which only makes sense for refs.\n+\t */\n+\tupdate->flags &= ~REF_HAVE_OLD;\n+\tupdate->index = index;\n+\n \treturn 0;\n }\n \ndiff --git a/refs.h b/refs.h\nindex a5bedf48cf6de91005a7e8d0bf58ca98350397a6..67f8b3eef3f2101409e5cc6eb2241d99e9f7d95c 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -727,6 +727,18 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t unsigned int flags, const char *msg,\n \t\t\t struct strbuf *err);\n \n+/*\n+ * Similar to`ref_transaction_update`, but this function is only for adding\n+ * a reflog update. Supports providing custom committer information.\n+ */\n+int ref_transaction_update_reflog(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t const char *committer_info, unsigned int flags,\n+\t\t\t\t const char *msg, unsigned int index,\n+\t\t\t\t struct strbuf *err);\n+\n /*\n * Add a reference creation to transaction. new_oid is the value that\n * the reference should have after the update; it must not be\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 255fed8354cae982f785b1b85340e2a1eeecf2a6..c11213f52065bcf2fa7612df8f9500692ee2d02c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3080,10 +3080,12 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * packed-refs don't support symbolic refs and root refs, so we\n-\t\t * have to queue these references via the loose transaction.\n+\t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n+\t\t * so we have to queue these references via the loose transaction.\n \t\t */\n-\t\tif (update->new_target || is_root_ref(update->refname)) {\n+\t\tif (update->new_target ||\n+\t\t is_root_ref(update->refname) ||\n+\t\t (update->flags & REF_LOG_ONLY)) {\n \t\t\tif (!loose_transaction) {\n \t\t\t\tloose_transaction = ref_store_transaction_begin(&refs->base, 0, err);\n \t\t\t\tif (!loose_transaction) {\n@@ -3092,11 +3094,17 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t\t\t}\n \t\t\t}\n \n-\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n-\t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n-\t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n-\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n-\t\t\t\t\t\t NULL);\n+\t\t\tif (update->flags & REF_LOG_ONLY)\n+\t\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n+\t\t\t\t\t\t\t update->flags, &update->new_oid,\n+\t\t\t\t\t\t\t &update->old_oid, NULL, NULL,\n+\t\t\t\t\t\t\t update->committer_info, update->msg);\n+\t\t\telse\n+\t\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n+\t\t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n+\t\t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n+\t\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n+\t\t\t\t\t\t\t NULL);\n \t\t} else {\n \t\t\tref_transaction_add_update(packed_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n\n-- \n2.47.1\n\n"},{"id":"509075","messageId":"20241213-320-git-refs-migrate-reflogs-v2-7-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 7/8] refs: allow multiple reflog entries for the same refname","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:52Z","receivedAt":"2024-12-13T10:36:57Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference transaction only allows a single update for a given\nreference to avoid conflicts. This, however, isn't an issue for reflogs.\nThere are no conflicts to be resolved in reflogs and when migrating\nreflogs between backends we'd have multiple reflog entries for the same\nrefname.\n\nSo allow multiple reflog updates within a single transaction. Also the\nreflog creation logic isn't exposed to the end user. While this might\nchange in the future, currently, this reduces the scope of issues to\nthink about.\n\nIn the reftable backend, the writer sorts all updates based on the\nupdate_index before writing to the block. When there are multiple\nreflogs for a given refname, it is essential that the order of the\nreflogs is maintained. So add the `index` value to the `update_index`.\nThe `index` field is only be set when multiple reflog entries for a\ngiven refname are added and as such in most scenarios the old behavior\nremains.\n\nThis is required to add reflog migration support to `git refs migrate`.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 15 +++++++++++----\n refs/reftable-backend.c | 22 +++++++++++++++++++---\n 2 files changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex c11213f52065bcf2fa7612df8f9500692ee2d02c..8953d1c6d37b13b0db701888b3db92fd87a68aaa 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2611,6 +2611,9 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \n \tupdate->backend_data = lock;\n \n+\tif (update->flags & REF_LOG_ONLY)\n+\t\tgoto out;\n+\n \tif (update->type & REF_ISSYMREF) {\n \t\tif (update->flags & REF_NO_DEREF) {\n \t\t\t/*\n@@ -2829,13 +2832,16 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t */\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *update = transaction->updates[i];\n-\t\tstruct string_list_item *item =\n-\t\t\tstring_list_append(&affected_refnames, update->refname);\n+\t\tstruct string_list_item *item;\n \n \t\tif ((update->flags & REF_IS_PRUNING) &&\n \t\t !(update->flags & REF_NO_DEREF))\n \t\t\tBUG(\"REF_IS_PRUNING set without REF_NO_DEREF\");\n \n+\t\tif (update->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\titem = string_list_append(&affected_refnames, update->refname);\n \t\t/*\n \t\t * We store a pointer to update in item->util, but at\n \t\t * the moment we never use the value of this field\n@@ -3035,8 +3041,9 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \n \t/* Fail if a refname appears more than once in the transaction: */\n \tfor (i = 0; i < transaction->nr; i++)\n-\t\tstring_list_append(&affected_refnames,\n-\t\t\t\t transaction->updates[i]->refname);\n+\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n+\t\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t\t transaction->updates[i]->refname);\n \tstring_list_sort(&affected_refnames);\n \tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -990,8 +990,9 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tif (ret)\n \t\t\tgoto done;\n \n-\t\tstring_list_append(&affected_refnames,\n-\t\t\t\t transaction->updates[i]->refname);\n+\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n+\t\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t\t transaction->updates[i]->refname);\n \t}\n \n \t/*\n@@ -1301,6 +1302,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tuint64_t max_update_index = ts;\n \tconst char *committer_info;\n \tint ret = 0;\n \n@@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t\t}\n \n \t\t\t\tfill_reftable_log_record(log, &c);\n-\t\t\t\tlog->update_index = ts;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Updates are sorted by the writer. So updates for the same\n+\t\t\t\t * refname need to contain different update indices.\n+\t\t\t\t */\n+\t\t\t\tlog->update_index = ts + u->index;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Note the max update_index so the limit can be set later on.\n+\t\t\t\t */\n+\t\t\t\tif (log->update_index > max_update_index)\n+\t\t\t\t\tmax_update_index = log->update_index;\n+\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n \t\t\t\t u->new_oid.hash, GIT_MAX_RAWSZ);\n@@ -1469,6 +1483,8 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t * and log blocks.\n \t */\n \tif (logs) {\n+\t\treftable_writer_set_limits(writer, ts, max_update_index);\n+\n \t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n\n-- \n2.47.1\n\n"},{"id":"509076","messageId":"20241213-320-git-refs-migrate-reflogs-v2-8-f28312cdb6c0@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v2 8/8] refs: add support for migrating reflogs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T10:36:53Z","receivedAt":"2024-12-13T10:36:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs migrate` command was introduced in\n25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n2024-06-06) to support migrating from one reference backend to another.\n\nOne limitation of the command was that it didn't support migrating\nrepositories which contained reflogs. A previous commit, added support\nfor adding reflog updates in ref transactions. Using the added\nfunctionality bake in reflog support for `git refs migrate`.\n\nTo ensure that the order of the reflogs is maintained during the\nmigration, we add the index for each reflog update as we iterate over\nthe reflogs from the old reference backend. This is to ensure that the\norder is maintained in the new backend.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-refs.txt | 2 --\n refs.c | 89 ++++++++++++++++++++++++++++++++--------------\n t/t1460-refs-migrate.sh | 73 +++++++++++++++++++++++++------------\n 3 files changed, 113 insertions(+), 51 deletions(-)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex ce31f93061db5e5d16aca516dd3d15f6527db870..9829984b0a4c4f54ec7f9b6c6c7072f62b1d198d 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -57,8 +57,6 @@ KNOWN LIMITATIONS\n \n The ref format migration has several known limitations in its current form:\n \n-* It is not possible to migrate repositories that have reflogs.\n-\n * It is not possible to migrate repositories that have worktrees.\n \n * There is no way to block concurrent writes to the repository during an\ndiff --git a/refs.c b/refs.c\nindex 9f539369bc94a25594adc3e95847f2fe72f58a08..f19292d50f0003881220e8f7cfcf6c7eb4b2e749 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -30,6 +30,7 @@\n #include \"date.h\"\n #include \"commit.h\"\n #include \"wildmatch.h\"\n+#include \"ident.h\"\n \n /*\n * List of all available backends\n@@ -2673,6 +2674,7 @@ int ref_update_check_old_target(const char *referent, struct ref_update *update,\n }\n \n struct migration_data {\n+\tunsigned int reflog_index;\n \tstruct ref_store *old_refs;\n \tstruct ref_transaction *transaction;\n \tstruct strbuf *errbuf;\n@@ -2708,6 +2710,53 @@ static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n \treturn ret;\n }\n \n+struct reflog_migration_data {\n+\tunsigned int *index;\n+\tconst char *refname;\n+\tstruct ref_store *old_refs;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf *errbuf;\n+};\n+\n+static int migrate_one_reflog_entry(struct object_id *old_oid,\n+\t\t\t\t struct object_id *new_oid,\n+\t\t\t\t const char *committer,\n+\t\t\t\t timestamp_t timestamp, int tz,\n+\t\t\t\t const char *msg, void *cb_data)\n+{\n+\tstruct reflog_migration_data *data = cb_data;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *date;\n+\tint ret;\n+\n+\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n+\t/* committer contains name and email */\n+\tstrbuf_addstr(&sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n+\n+\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n+\t\t\t\t\t new_oid, old_oid, sb.buf,\n+\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n+\t\t\t\t\t (*data->index)++, data->errbuf);\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+\n+static int migrate_one_reflog(const char *refname, void *cb_data)\n+{\n+\tstruct migration_data *migration_data = cb_data;\n+\tstruct reflog_migration_data data;\n+\n+\tdata.refname = refname;\n+\tdata.old_refs = migration_data->old_refs;\n+\tdata.transaction = migration_data->transaction;\n+\tdata.errbuf = migration_data->errbuf;\n+\tdata.index = &migration_data->reflog_index;\n+\n+\treturn refs_for_each_reflog_ent(migration_data->old_refs, refname,\n+\t\t\t\t\tmigrate_one_reflog_entry, &data);\n+}\n+\n static int move_files(const char *from_path, const char *to_path, struct strbuf *errbuf)\n {\n \tstruct strbuf from_buf = STRBUF_INIT, to_buf = STRBUF_INIT;\n@@ -2774,13 +2823,6 @@ static int move_files(const char *from_path, const char *to_path, struct strbuf\n \treturn ret;\n }\n \n-static int count_reflogs(const char *reflog UNUSED, void *payload)\n-{\n-\tsize_t *reflog_count = payload;\n-\t(*reflog_count)++;\n-\treturn 0;\n-}\n-\n static int has_worktrees(void)\n {\n \tstruct worktree **worktrees = get_worktrees();\n@@ -2806,7 +2848,6 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tstruct ref_transaction *transaction = NULL;\n \tstruct strbuf new_gitdir = STRBUF_INIT;\n \tstruct migration_data data;\n-\tsize_t reflog_count = 0;\n \tint did_migrate_refs = 0;\n \tint ret;\n \n@@ -2818,21 +2859,6 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \n \told_refs = get_main_ref_store(repo);\n \n-\t/*\n-\t * We do not have any interfaces that would allow us to write many\n-\t * reflog entries. Once we have them we can remove this restriction.\n-\t */\n-\tif (refs_for_each_reflog(old_refs, count_reflogs, &reflog_count) < 0) {\n-\t\tstrbuf_addstr(errbuf, \"cannot count reflogs\");\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tif (reflog_count) {\n-\t\tstrbuf_addstr(errbuf, \"migrating reflogs is not supported yet\");\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\n \t/*\n \t * Worktrees complicate the migration because every worktree has a\n \t * separate ref storage. While it should be feasible to implement, this\n@@ -2858,17 +2884,21 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t * This operation is safe as we do not yet modify the main\n \t * repository.\n \t *\n-\t * 3. If we're in dry-run mode then we are done and can hand over the\n+\t * 3. Enumerate all reflogs and write them into the new ref storage.\n+\t * This operation is safe as we do not yet modify the main\n+\t * repository.\n+\t *\n+\t * 4. If we're in dry-run mode then we are done and can hand over the\n \t * directory to the caller for inspection. If not, we now start\n \t * with the destructive part.\n \t *\n-\t * 4. Delete the old ref storage from disk. As we have a copy of refs\n+\t * 5. Delete the old ref storage from disk. As we have a copy of refs\n \t * in the new ref storage it's okay(ish) if we now get interrupted\n \t * as there is an equivalent copy of all refs available.\n \t *\n-\t * 5. Move the new ref storage files into place.\n+\t * 6. Move the new ref storage files into place.\n \t *\n-\t * 6. Change the repository format to the new ref format.\n+\t * 7. Change the repository format to the new ref format.\n \t */\n \tstrbuf_addf(&new_gitdir, \"%s/%s\", old_refs->gitdir, \"ref_migration.XXXXXX\");\n \tif (!mkdtemp(new_gitdir.buf)) {\n@@ -2910,6 +2940,11 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tif (ret < 0)\n \t\tgoto done;\n \n+\tdata.reflog_index = 1;\n+\tret = refs_for_each_reflog(old_refs, migrate_one_reflog, &data);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = ref_transaction_commit(transaction, errbuf);\n \tif (ret < 0)\n \t\tgoto done;\ndiff --git a/t/t1460-refs-migrate.sh b/t/t1460-refs-migrate.sh\nindex 1bfff3a7afd5acc470424dfe7ec3e97d45f5c481..f59bc4860f19c4af82dc6f2984bdb69d61fe3ec2 100755\n--- a/t/t1460-refs-migrate.sh\n+++ b/t/t1460-refs-migrate.sh\n@@ -7,23 +7,44 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+# Migrate the provided repository from one format to the other and\n+# verify that the references and logs are migrated over correctly.\n+# Usage: test_migration \n+# is the relative path to the repo to be migrated.\n+# is the ref format to be migrated to.\n+# (true or false) whether to skip reflog verification.\n test_migration () {\n-\tgit -C \"$1\" for-each-ref --include-root-refs \\\n+\trepo=$1 &&\n+\tformat=$2 &&\n+\tskip_reflog_verify=${3:-false} &&\n+\tgit -C \"$repo\" for-each-ref --include-root-refs \\\n \t\t--format='%(refname) %(objectname) %(symref)' >expect &&\n-\tgit -C \"$1\" refs migrate --ref-format=\"$2\" &&\n-\tgit -C \"$1\" for-each-ref --include-root-refs \\\n+\tif ! $skip_reflog_verify\n+\tthen\n+\t git -C \"$repo\" reflog --all >expect_logs &&\n+\t git -C \"$repo\" reflog list >expect_log_list\n+\tfi &&\n+\n+\tgit -C \"$repo\" refs migrate --ref-format=\"$2\" &&\n+\n+\tgit -C \"$repo\" for-each-ref --include-root-refs \\\n \t\t--format='%(refname) %(objectname) %(symref)' >actual &&\n \ttest_cmp expect actual &&\n+\tif ! $skip_reflog_verify\n+\tthen\n+\t\tgit -C \"$repo\" reflog --all >actual_logs &&\n+\t\tgit -C \"$repo\" reflog list >actual_log_list &&\n+\t\ttest_cmp expect_logs actual_logs &&\n+\t\ttest_cmp expect_log_list actual_log_list\n+\tfi &&\n \n-\tgit -C \"$1\" rev-parse --show-ref-format >actual &&\n-\techo \"$2\" >expect &&\n+\tgit -C \"$repo\" rev-parse --show-ref-format >actual &&\n+\techo \"$format\" >expect &&\n \ttest_cmp expect actual\n }\n \n test_expect_success 'setup' '\n-\trm -rf .git &&\n-\t# The migration does not yet support reflogs.\n-\tgit config --global core.logAllRefUpdates false\n+\trm -rf .git\n '\n \n test_expect_success \"superfluous arguments\" '\n@@ -78,19 +99,6 @@ do\n \t\t\ttest_cmp expect err\n \t\t'\n \n-\t\ttest_expect_success \"$from_format -> $to_format: migration with reflog fails\" '\n-\t\t\ttest_when_finished \"rm -rf repo\" &&\n-\t\t\tgit init --ref-format=$from_format repo &&\n-\t\t\ttest_config -C repo core.logAllRefUpdates true &&\n-\t\t\ttest_commit -C repo logged &&\n-\t\t\ttest_must_fail git -C repo refs migrate \\\n-\t\t\t\t--ref-format=$to_format 2>err &&\n-\t\t\tcat >expect <<-EOF &&\n-\t\t\terror: migrating reflogs is not supported yet\n-\t\t\tEOF\n-\t\t\ttest_cmp expect err\n-\t\t'\n-\n \t\ttest_expect_success \"$from_format -> $to_format: migration with worktree fails\" '\n \t\t\ttest_when_finished \"rm -rf repo\" &&\n \t\t\tgit init --ref-format=$from_format repo &&\n@@ -141,7 +149,7 @@ do\n \t\t\ttest_commit -C repo initial &&\n \t\t\ttest-tool -C repo ref-store main update-ref \"\" refs/heads/broken \\\n \t\t\t\t\"$(test_oid 001)\" \"$ZERO_OID\" REF_SKIP_CREATE_REFLOG,REF_SKIP_OID_VERIFICATION &&\n-\t\t\ttest_migration repo \"$to_format\" &&\n+\t\t\ttest_migration repo \"$to_format\" true &&\n \t\t\ttest_oid 001 >expect &&\n \t\t\tgit -C repo rev-parse refs/heads/broken >actual &&\n \t\t\ttest_cmp expect actual\n@@ -195,6 +203,27 @@ do\n \t\t\tgit -C repo rev-parse --show-ref-format >actual &&\n \t\t\ttest_cmp expect actual\n \t\t'\n+\n+\t\ttest_expect_success \"$from_format -> $to_format: reflogs of symrefs with target deleted\" '\n+\t\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t\tgit init --ref-format=$from_format repo &&\n+\t\t\ttest_commit -C repo initial &&\n+\t\t\tgit -C repo branch branch-1 HEAD &&\n+\t\t\tgit -C repo symbolic-ref refs/heads/symref refs/heads/branch-1 &&\n+\t\t\tcat >input <<-EOF &&\n+\t\t\tdelete refs/heads/branch-1\n+\t\t\tEOF\n+\t\t\tgit -C repo update-ref --stdin $to_format: reflogs order is retained\" '\n+\t\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t\tgit init --ref-format=$from_format repo &&\n+\t\t\ttest_commit --date \"100005000 +0700\" --no-tag -C repo initial &&\n+\t\t\ttest_commit --date \"100003000 +0700\" --no-tag -C repo second &&\n+\t\t\ttest_migration repo \"$to_format\"\n+\t\t'\n \tdone\n done\n \n\n-- \n2.47.1\n\n"},{"id":"509092","messageId":"CAP8UFD1MOChEhJhW1i4E3D62k_to5_r3P05LCbFRXMrEunvAfw@mail.gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-6-f28312cdb6c0@gmail.com","subject":"Re: [PATCH v2 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-13T11:44:27Z","receivedAt":"2024-12-13T11:44:40Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Dec 13, 2024 at 11:36 AM Karthik Nayak wrote:\n\n> +int ref_transaction_update_reflog(struct ref_transaction *transaction,\n> + const char *refname,\n> + const struct object_id *new_oid,\n> + const struct object_id *old_oid,\n> + const char *committer_info, unsigned int flags,\n> + const char *msg, unsigned int index,\n> + struct strbuf *err)\n> +{\n> + struct ref_update *update;\n> +\n> + assert(err);\n> +\n> + if (!transaction_refname_valid(refname, new_oid, flags, 1, err))\n> + return -1;\n> +\n> + flags |= REF_LOG_ONLY | REF_NO_DEREF;\n\nIf we could switch the above lines like this:\n\n flags |= REF_LOG_ONLY | REF_NO_DEREF;\n\n if (!transaction_refname_valid(refname, new_oid, flags, 1, err))\n return -1;\n\nmaybe we wouldn't need transaction_refname_valid() to take an\n'unsigned int reflog' argument and we could instead use 'flags &\nREF_LOG_ONLY' inside that function?\n\n> + update = ref_transaction_add_update(transaction, refname, flags,\n> + new_oid, old_oid, NULL, NULL,\n> + committer_info, msg);\n> + /*\n> + * While we do set the old_oid value, we unset the flag to skip\n> + * old_oid verification which only makes sense for refs.\n> + */\n> + update->flags &= ~REF_HAVE_OLD;\n> + update->index = index;\n> +\n> return 0;\n> }\n"},{"id":"509098","messageId":"Z1wnbCoidG-lC7ui@pks.im","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-5-f28312cdb6c0@gmail.com","subject":"Re: [PATCH v2 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T12:24:23Z","receivedAt":"2024-12-13T12:24:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 13, 2024 at 11:36:50AM +0100, Karthik Nayak wrote:\n> The `ref_transaction_add_update()` creates the `ref_update` struct. To\n> facilitate addition of reflogs in the next commit, the function needs to\n> accommodate setting the `committer_info` field in the struct. So modify\n> the function to also take `committer_info` as an argument and set it\n> accordingly.\n\nI was wondering a bit whether we could instead pull out a\n`add_update_internal()` function so that we don't need to modify all\ncallers of `ref_transaction_add_update()`. Because ultimately, we don't\nuse the field anywhere except from `ref_transaction_add_reflog_update()`\nas far as I can see.\n\nThis is more of a thought than a strong opinion, so feel free to ignore.\n\n> @@ -1190,8 +1191,12 @@ struct ref_update *ref_transaction_add_update(\n> \t\toidcpy(&update->new_oid, new_oid);\n> \tif ((flags & REF_HAVE_OLD) && old_oid)\n> \t\toidcpy(&update->old_oid, old_oid);\n> -\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n> +\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n> +\t\tif (committer_info)\n> +\t\t\tupdate->committer_info = xstrdup(committer_info);\n> +\n> \t\tupdate->msg = normalize_reflog_message(msg);\n> +\t}\n> \n> \treturn update;\n> }\n\nThis can use `xstrdup_or_null()` and then drop the condition.\n\nPatrick\n"},{"id":"509099","messageId":"Z1wne2iJAQGk-sG4@pks.im","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-6-f28312cdb6c0@gmail.com","subject":"Re: [PATCH v2 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T12:24:27Z","receivedAt":"2024-12-13T12:24:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 13, 2024 at 11:36:51AM +0100, Karthik Nayak wrote:\n> diff --git a/refs.h b/refs.h\n> index a5bedf48cf6de91005a7e8d0bf58ca98350397a6..67f8b3eef3f2101409e5cc6eb2241d99e9f7d95c 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -727,6 +727,18 @@ int ref_transaction_update(struct ref_transaction *transaction,\n> \t\t\t unsigned int flags, const char *msg,\n> \t\t\t struct strbuf *err);\n> \n> +/*\n> + * Similar to`ref_transaction_update`, but this function is only for adding\n> + * a reflog update. Supports providing custom committer information.\n> + */\n> +int ref_transaction_update_reflog(struct ref_transaction *transaction,\n> +\t\t\t\t const char *refname,\n> +\t\t\t\t const struct object_id *new_oid,\n> +\t\t\t\t const struct object_id *old_oid,\n> +\t\t\t\t const char *committer_info, unsigned int flags,\n> +\t\t\t\t const char *msg, unsigned int index,\n> +\t\t\t\t struct strbuf *err);\n> +\n\nNit: it would be great to explain what the index does in the doc, as it\nis completely non-obvious.\n\nPatrick\n"},{"id":"509100","messageId":"Z1wnf54rEq0POCb3@pks.im","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-7-f28312cdb6c0@gmail.com","subject":"Re: [PATCH v2 7/8] refs: allow multiple reflog entries for the same refname","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T12:24:31Z","receivedAt":"2024-12-13T12:24:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 13, 2024 at 11:36:52AM +0100, Karthik Nayak wrote:\n> The reference transaction only allows a single update for a given\n> reference to avoid conflicts. This, however, isn't an issue for reflogs.\n> There are no conflicts to be resolved in reflogs and when migrating\n> reflogs between backends we'd have multiple reflog entries for the same\n> refname.\n> \n> So allow multiple reflog updates within a single transaction. Also the\n> reflog creation logic isn't exposed to the end user. While this might\n> change in the future, currently, this reduces the scope of issues to\n> think about.\n> \n> In the reftable backend, the writer sorts all updates based on the\n> update_index before writing to the block. When there are multiple\n> reflogs for a given refname, it is essential that the order of the\n> reflogs is maintained. So add the `index` value to the `update_index`.\n> The `index` field is only be set when multiple reflog entries for a\n\ns/only be/only\n\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n> \t\t\t\t}\n> \n> \t\t\t\tfill_reftable_log_record(log, &c);\n> -\t\t\t\tlog->update_index = ts;\n> +\n> +\t\t\t\t/*\n> +\t\t\t\t * Updates are sorted by the writer. So updates for the same\n> +\t\t\t\t * refname need to contain different update indices.\n> +\t\t\t\t */\n> +\t\t\t\tlog->update_index = ts + u->index;\n\nOkay. So instead of tracking things via a map, we now rely on the caller\nto provide the update index. And if they don't provide one then we\ncannot guarantee ordering.\n\nI guess that's a good solution. After all, there will only be a very\nlimited amount of callers in the first place, so I think it's fine to\nshift the responsibility onto them to maintain reflog ordering. They're\nalso the only ones who really know about the actual ordering.\n\n> +\t\t\t\t/*\n> +\t\t\t\t * Note the max update_index so the limit can be set later on.\n> +\t\t\t\t */\n> +\t\t\t\tif (log->update_index > max_update_index)\n> +\t\t\t\t\tmax_update_index = log->update_index;\n\nMakes sense, as well.\n\nPatrick\n"},{"id":"509101","messageId":"Z1wngv1Kox_QQcwK@pks.im","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-8-f28312cdb6c0@gmail.com","subject":"Re: [PATCH v2 8/8] refs: add support for migrating reflogs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-13T12:24:34Z","receivedAt":"2024-12-13T12:24:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 13, 2024 at 11:36:53AM +0100, Karthik Nayak wrote:\n> diff --git a/refs.c b/refs.c\n> index 9f539369bc94a25594adc3e95847f2fe72f58a08..f19292d50f0003881220e8f7cfcf6c7eb4b2e749 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2708,6 +2710,53 @@ static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n> \treturn ret;\n> }\n> \n> +struct reflog_migration_data {\n> +\tunsigned int *index;\n> +\tconst char *refname;\n> +\tstruct ref_store *old_refs;\n> +\tstruct ref_transaction *transaction;\n> +\tstruct strbuf *errbuf;\n> +};\n> +\n> +static int migrate_one_reflog_entry(struct object_id *old_oid,\n> +\t\t\t\t struct object_id *new_oid,\n> +\t\t\t\t const char *committer,\n> +\t\t\t\t timestamp_t timestamp, int tz,\n> +\t\t\t\t const char *msg, void *cb_data)\n> +{\n> +\tstruct reflog_migration_data *data = cb_data;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tconst char *date;\n> +\tint ret;\n> +\n> +\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n> +\t/* committer contains name and email */\n> +\tstrbuf_addstr(&sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n> +\n> +\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n> +\t\t\t\t\t new_oid, old_oid, sb.buf,\n> +\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n> +\t\t\t\t\t (*data->index)++, data->errbuf);\n\nThis is where we now increment the reflog index to ensure a proper\nordering.\n\n> +\tstrbuf_release(&sb);\n> +\n> +\treturn ret;\n> +}\n\nWe're now allocating one buffer per reflog entry. We may want to\noptimize this by having a scratch buffer in `migration_data`, which we\ncould then pass on via `reflog_migration_data`.\n\n> @@ -2910,6 +2940,11 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n> \tif (ret < 0)\n> \t\tgoto done;\n> \n> +\tdata.reflog_index = 1;\n\nI'm a bit surprised that we initialize the relfog entry here, because\nthat means we now have a globally increasing counter across all reflogs.\nCouldn't we initialize the index per reflog instead? It ultimately does\nnot really matter, but feels like the more obvious design to me\n\nAlso, is there any specific reason why we start at 1 and not 0? Just curious.\n\nPatrick\n"},{"id":"509112","messageId":"CAOLa=ZT9KDNaz84DLoRBzY_yaZ3kZ2FUWADohZhM4EcrJgmS+A@mail.gmail.com","threadId":"62635","inReplyTo":"Z1wnbCoidG-lC7ui@pks.im","subject":"Re: [PATCH v2 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T19:43:03Z","receivedAt":"2024-12-13T19:43:05Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Fri, Dec 13, 2024 at 11:36:50AM +0100, Karthik Nayak wrote:\n>> The `ref_transaction_add_update()` creates the `ref_update` struct. To\n>> facilitate addition of reflogs in the next commit, the function needs to\n>> accommodate setting the `committer_info` field in the struct. So modify\n>> the function to also take `committer_info` as an argument and set it\n>> accordingly.\n>\n> I was wondering a bit whether we could instead pull out a\n> `add_update_internal()` function so that we don't need to modify all\n> callers of `ref_transaction_add_update()`. Because ultimately, we don't\n> use the field anywhere except from `ref_transaction_add_reflog_update()`\n> as far as I can see.\n>\n> This is more of a thought than a strong opinion, so feel free to ignore.\n>\n\nYes, that is a possible change, but the number of code changes are\nrelatively low and I didn't think it made so much difference. Also\nbecause we'd now have one more function. But I don't mind doing it\neither, if anyone feels strongly about it, I'll happily make that\nchange.\n\n>> @@ -1190,8 +1191,12 @@ struct ref_update *ref_transaction_add_update(\n>> \t\toidcpy(&update->new_oid, new_oid);\n>> \tif ((flags & REF_HAVE_OLD) && old_oid)\n>> \t\toidcpy(&update->old_oid, old_oid);\n>> -\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n>> +\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n>> +\t\tif (committer_info)\n>> +\t\t\tupdate->committer_info = xstrdup(committer_info);\n>> +\n>> \t\tupdate->msg = normalize_reflog_message(msg);\n>> +\t}\n>>\n>> \treturn update;\n>> }\n>\n> This can use `xstrdup_or_null()` and then drop the condition.\n>\n> Patrick\n\nThat is good improvement, will add.\n\nThanks!\n"},{"id":"509113","messageId":"CAOLa=ZRyr6yG6=UDQ6Lr2K6mSPiMzg+iopPHPwWdjFgyjxgfVQ@mail.gmail.com","threadId":"62635","inReplyTo":"CAP8UFD1MOChEhJhW1i4E3D62k_to5_r3P05LCbFRXMrEunvAfw@mail.gmail.com","subject":"Re: [PATCH v2 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T19:49:23Z","receivedAt":"2024-12-13T19:49:26Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Christian Couder writes:\n\n> On Fri, Dec 13, 2024 at 11:36 AM Karthik Nayak wrote:\n>\n>> +int ref_transaction_update_reflog(struct ref_transaction *transaction,\n>> + const char *refname,\n>> + const struct object_id *new_oid,\n>> + const struct object_id *old_oid,\n>> + const char *committer_info, unsigned int flags,\n>> + const char *msg, unsigned int index,\n>> + struct strbuf *err)\n>> +{\n>> + struct ref_update *update;\n>> +\n>> + assert(err);\n>> +\n>> + if (!transaction_refname_valid(refname, new_oid, flags, 1, err))\n>> + return -1;\n>> +\n>> + flags |= REF_LOG_ONLY | REF_NO_DEREF;\n>\n> If we could switch the above lines like this:\n>\n> flags |= REF_LOG_ONLY | REF_NO_DEREF;\n>\n> if (!transaction_refname_valid(refname, new_oid, flags, 1, err))\n> return -1;\n>\n> maybe we wouldn't need transaction_refname_valid() to take an\n> 'unsigned int reflog' argument and we could instead use 'flags &\n> REF_LOG_ONLY' inside that function?\n>\n\nThe issue is the that this changes existing behavior, since\n`ref_transaction_update()` can also be called with the `REF_LOG_ONLY`\nflag set.\n\nBut, I think it is a worthwhile change, because earlier even for reflog\nupdates we would show 'refusing to update ref ...' as the message. I'll\nadd this in and also make a note in the commit message.\n\nThanks\n\n>> + update = ref_transaction_add_update(transaction, refname, flags,\n>> + new_oid, old_oid, NULL, NULL,\n>> + committer_info, msg);\n>> + /*\n>> + * While we do set the old_oid value, we unset the flag to skip\n>> + * old_oid verification which only makes sense for refs.\n>> + */\n>> + update->flags &= ~REF_HAVE_OLD;\n>> + update->index = index;\n>> +\n>> return 0;\n>> }\n"},{"id":"509114","messageId":"CAOLa=ZQO9NyuwVPP82wCf6BOGLL7U_aLNX=cxKd7H4=Qc2YX-A@mail.gmail.com","threadId":"62635","inReplyTo":"Z1wnf54rEq0POCb3@pks.im","subject":"Re: [PATCH v2 7/8] refs: allow multiple reflog entries for the same refname","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-13T20:02:21Z","receivedAt":"2024-12-13T20:02:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Fri, Dec 13, 2024 at 11:36:52AM +0100, Karthik Nayak wrote:\n>> The reference transaction only allows a single update for a given\n>> reference to avoid conflicts. This, however, isn't an issue for reflogs.\n>> There are no conflicts to be resolved in reflogs and when migrating\n>> reflogs between backends we'd have multiple reflog entries for the same\n>> refname.\n>>\n>> So allow multiple reflog updates within a single transaction. Also the\n>> reflog creation logic isn't exposed to the end user. While this might\n>> change in the future, currently, this reduces the scope of issues to\n>> think about.\n>>\n>> In the reftable backend, the writer sorts all updates based on the\n>> update_index before writing to the block. When there are multiple\n>> reflogs for a given refname, it is essential that the order of the\n>> reflogs is maintained. So add the `index` value to the `update_index`.\n>> The `index` field is only be set when multiple reflog entries for a\n>\n> s/only be/only\n\nThanks.\n\n>> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n>> index b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n>> --- a/refs/reftable-backend.c\n>> +++ b/refs/reftable-backend.c\n>> @@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n>> \t\t\t\t}\n>>\n>> \t\t\t\tfill_reftable_log_record(log, &c);\n>> -\t\t\t\tlog->update_index = ts;\n>> +\n>> +\t\t\t\t/*\n>> +\t\t\t\t * Updates are sorted by the writer. So updates for the same\n>> +\t\t\t\t * refname need to contain different update indices.\n>> +\t\t\t\t */\n>> +\t\t\t\tlog->update_index = ts + u->index;\n>\n> Okay. So instead of tracking things via a map, we now rely on the caller\n> to provide the update index. And if they don't provide one then we\n> cannot guarantee ordering.\n>\n\nWhich works, because for reflog migration, the caller _has_ to specify\nordering.\n\n> I guess that's a good solution. After all, there will only be a very\n> limited amount of callers in the first place, so I think it's fine to\n> shift the responsibility onto them to maintain reflog ordering. They're\n> also the only ones who really know about the actual ordering.\n>\n\nExactly. Currently this is entirely and only used in the migration code.\nPerhaps we expose reflog addition to users, but that is not something\nbeing pursued.\n\n>> +\t\t\t\t/*\n>> +\t\t\t\t * Note the max update_index so the limit can be set later on.\n>> +\t\t\t\t */\n>> +\t\t\t\tif (log->update_index > max_update_index)\n>> +\t\t\t\t\tmax_update_index = log->update_index;\n>\n> Makes sense, as well.\n>\n> Patrick\n\nThanks\n"},{"id":"509121","messageId":"CAOLa=ZQ9SHD3gzTVaznGhkCBjrrJbHm1fDyi1F-h6VZvtdpxgw@mail.gmail.com","threadId":"62635","inReplyTo":"Z1wngv1Kox_QQcwK@pks.im","subject":"Re: [PATCH v2 8/8] refs: add support for migrating reflogs","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T11:09:23Z","receivedAt":"2024-12-15T11:09:25Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Fri, Dec 13, 2024 at 11:36:53AM +0100, Karthik Nayak wrote:\n>> diff --git a/refs.c b/refs.c\n>> index 9f539369bc94a25594adc3e95847f2fe72f58a08..f19292d50f0003881220e8f7cfcf6c7eb4b2e749 100644\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -2708,6 +2710,53 @@ static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n>> \treturn ret;\n>> }\n>>\n>> +struct reflog_migration_data {\n>> +\tunsigned int *index;\n>> +\tconst char *refname;\n>> +\tstruct ref_store *old_refs;\n>> +\tstruct ref_transaction *transaction;\n>> +\tstruct strbuf *errbuf;\n>> +};\n>> +\n>> +static int migrate_one_reflog_entry(struct object_id *old_oid,\n>> +\t\t\t\t struct object_id *new_oid,\n>> +\t\t\t\t const char *committer,\n>> +\t\t\t\t timestamp_t timestamp, int tz,\n>> +\t\t\t\t const char *msg, void *cb_data)\n>> +{\n>> +\tstruct reflog_migration_data *data = cb_data;\n>> +\tstruct strbuf sb = STRBUF_INIT;\n>> +\tconst char *date;\n>> +\tint ret;\n>> +\n>> +\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n>> +\t/* committer contains name and email */\n>> +\tstrbuf_addstr(&sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n>> +\n>> +\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n>> +\t\t\t\t\t new_oid, old_oid, sb.buf,\n>> +\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n>> +\t\t\t\t\t (*data->index)++, data->errbuf);\n>\n> This is where we now increment the reflog index to ensure a proper\n> ordering.\n>\n>> +\tstrbuf_release(&sb);\n>> +\n>> +\treturn ret;\n>> +}\n>\n> We're now allocating one buffer per reflog entry. We may want to\n> optimize this by having a scratch buffer in `migration_data`, which we\n> could then pass on via `reflog_migration_data`.\n>\n\nThat makes sense, let me do that.\n\n>> @@ -2910,6 +2940,11 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n>> \tif (ret < 0)\n>> \t\tgoto done;\n>>\n>> +\tdata.reflog_index = 1;\n>\n> I'm a bit surprised that we initialize the relfog entry here, because\n> that means we now have a globally increasing counter across all reflogs.\n> Couldn't we initialize the index per reflog instead? It ultimately does\n> not really matter, but feels like the more obvious design to me\n\nYes, this was needed cause I initially didn't understand how the\nudpate_index worked and assumed two logs couldn't have the same\nupdate_index. I missed changing it, like you said, it works, but I'll\nfix it.\n\n> Also, is there any specific reason why we start at 1 and not 0? Just curious.\n\nNot really, I wanted to distinguish between index entries vs non-indexed\nentries. But logically, no, I'll remove it, to remove any confusion.\n\n> Patrick\n\nThanks\n"},{"id":"509125","messageId":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com","subject":"[PATCH v3 0/8] refs: add reflog support to `git refs migrate`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:37Z","receivedAt":"2024-12-15T16:25:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs migrate` command was introduced in\n25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n2024-06-06) to support migrating from one reference backend to another.\n\nOne limitation of the feature was that it didn't support migrating\nrepositories which contained reflogs. This isn't a requirement on the\nserver side as repositories are stored as bare repositories (which do\nnot contain any reflogs). Clients however generally use reflogs and\nuntil now couldn't use the `git refs migrate` command to migrate their\nrepositories to the new reftable format.\n\nOne of the issues for adding reflog support is that the ref transactions\ndon't support reflogs additions:\n 1. While there is REF_LOG_ONLY flag, there is no function to utilize\n the flag and add reflogs.\n 2. reference backends generally sort the updates by the refname. This\n wouldn't work for reflogs which need to ensure that they maintain the\n order of creation.\n 3. In the files backend, reflog entries are added by obtaining locks\n on the refs themselves. This means each update in the transaction, will\n obtain a ref_lock. This paradigm fails to accompany the fact that there\n could be multiple reflog updates for a refname in a single transaction.\n 4. The backends check for duplicate entries, which doesn't make sense\n in the context of adding multiple reflogs for a given refname.\n\nWe overcome these issue we make the following changes:\n - Update the ref_update structure to also include the committer\n information. Using this, we can add a new function which only adds\n reflog updates to the transaction.\n - Add an index field to the ref_update structure, this will help order\n updates in pre-defined order, this fixes #2.\n - While the ideal fix for #3 would be to actually introduce reflog\n locks, this wouldn't be possible without breaking backward\n compatibility. So we add a count field to the existing ref_lock. With\n this, multiple reflog updates can share a single ref_lock.\n\nOverall, this series is a bit more involved, and I would appreciate it\nif it receives a bit more scrutiny.\n\nThe series is based on top of e66fd72e97 (The fourteenth batch,\n2024-12-06) with `kn/reftable-writer-log-write-verify` merged in.\n\nSigned-off-by: Karthik Nayak \n---\nChanges in v3:\n- patch 5: Use `xstrdup_or_null` unconditionally. \n- patch 6: In `transaction_refname_valid()` use the transaction flags\n to identify reflogs. Update the documentation to also mention the\n purpose of the `index` field.\n- patch 8: Instead of allocating an strbuf for each reflog entry, we \n store and re-use one in the migration callback data.\n- patch 8: Don't use a global index increment for all reflogs entries,\n instead create and use one per reflog.\n- patch 8: Avoid setting the first reflog index to `1`. This would default\n to `0` as the first index, which is okay, since the index is incremented\n for consequtive reflog entries.\n- Small typo fixes. \n- Thanks to Christian and Patrick for the review! \n- Link to v2: https://lore.kernel.org/all/20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com/\n\nChanges in v2:\n- Split patch 5 into two separate patches. This should make it easier to\n review and reduce cognitive load in a single patch.\n- In reftable backend, instead of using `strmapint` to ensure we have\n new update_indexes for reflogs with the same refname, we now use the\n already available `update->index` field to increment the update_index.\n- Cleanup the code and follow some of the better practices.\n- Add some clarity to the commit messages.\n- Link to v1: https://lore.kernel.org/r/20241209-320-git-refs-migrate-reflogs-v1-0-d4bc37ee860f@gmail.com\n\n---\nKarthik Nayak (8):\n refs: include committer info in `ref_update` struct\n refs: add `index` field to `struct ref_udpate`\n refs/files: add count field to ref_lock\n refs: extract out refname verification in transactions\n refs: add `committer_info` to `ref_transaction_add_update()`\n refs: introduce the `ref_transaction_update_reflog` function\n refs: allow multiple reflog entries for the same refname\n refs: add support for migrating reflogs\n\n Documentation/git-refs.txt | 2 -\n refs.c | 165 +++++++++++++++++++++++++++++++++------------\n refs.h | 14 ++++\n refs/files-backend.c | 131 ++++++++++++++++++++++-------------\n refs/refs-internal.h | 9 +++\n refs/reftable-backend.c | 53 ++++++++++++---\n t/t1460-refs-migrate.sh | 73 ++++++++++++++------\n 7 files changed, 327 insertions(+), 120 deletions(-)\n---\n\nRange-diff versus v2:\n\n1: d9bd20468c = 1: d1c06e34bf refs: include committer info in `ref_update` struct\n2: 8478eeac95 = 2: 33e65a965a refs: add `index` field to `struct ref_udpate`\n3: 913fd320f0 = 3: 34c5beccb6 refs/files: add count field to ref_lock\n4: 66b86b5807 = 4: 4fceb64954 refs: extract out refname verification in transactions\n5: 33ad1774d4 ! 5: 3c5abd0047 refs: add `committer_info` to `ref_transaction_add_update()`\n @@ refs.c: struct ref_update *ref_transaction_add_update(\n \t\toidcpy(&update->old_oid, old_oid);\n -\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n +\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n -+\t\tif (committer_info)\n -+\t\t\tupdate->committer_info = xstrdup(committer_info);\n -+\n ++\t\tupdate->committer_info = xstrdup_or_null(committer_info);\n \t\tupdate->msg = normalize_reflog_message(msg);\n +\t}\n \n6: cdbb15b11a ! 6: 9e12f16b96 refs: introduce the `ref_transaction_update_reflog` function\n @@ Commit message\n means clients can add reflog entries with custom committer\n information.\n \n + The `transaction_refname_valid()` function also modifies the error\n + message selectively based on the type of the update. This change also\n + affects reflog updates which go through `ref_transaction_update()`.\n +\n A follow up commit will utilize this function to add reflog support to\n `git refs migrate`.\n \n Signed-off-by: Karthik Nayak \n \n ## refs.c ##\n -@@ refs.c: struct ref_update *ref_transaction_add_update(\n - \n - static int transaction_refname_valid(const char *refname,\n - \t\t\t\t const struct object_id *new_oid,\n --\t\t\t\t unsigned int flags, struct strbuf *err)\n -+\t\t\t\t unsigned int flags, unsigned int reflog,\n -+\t\t\t\t struct strbuf *err)\n - {\n - \tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n +@@ refs.c: static int transaction_refname_valid(const char *refname,\n \t\treturn 1;\n \n \tif (is_pseudo_ref(refname)) {\n -\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n -\t\t\t refname);\n -+\t\tconst char *what = reflog ? \"reflog for pseudoref\" : \"pseudoref\";\n ++\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n +\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t} else if ((new_oid && !is_null_oid(new_oid)) ?\n @@ refs.c: struct ref_update *ref_transaction_add_update(\n \t\t !refname_is_safe(refname)) {\n -\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n -\t\t\t refname);\n -+\t\tconst char *what = reflog ? \"reflog with bad name\" : \"ref with bad name\";\n ++\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog with bad name\" : \"ref with bad name\";\n +\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t}\n \n -@@ refs.c: int ref_transaction_update(struct ref_transaction *transaction,\n - \t\treturn -1;\n - \t}\n - \n --\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n -+\tif (!transaction_refname_valid(refname, new_oid, flags, 0, err))\n - \t\treturn -1;\n - \n - \tif (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS)\n @@ refs.c: int ref_transaction_update(struct ref_transaction *transaction,\n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n @@ refs.c: int ref_transaction_update(struct ref_transaction *transaction,\n +\n +\tassert(err);\n +\n -+\tif (!transaction_refname_valid(refname, new_oid, flags, 1, err))\n -+\t\treturn -1;\n -+\n +\tflags |= REF_LOG_ONLY | REF_NO_DEREF;\n +\n ++\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n ++\t\treturn -1;\n ++\n +\tupdate = ref_transaction_add_update(transaction, refname, flags,\n +\t\t\t\t\t new_oid, old_oid, NULL, NULL,\n +\t\t\t\t\t committer_info, msg);\n @@ refs.h: int ref_transaction_update(struct ref_transaction *transaction,\n \n +/*\n + * Similar to`ref_transaction_update`, but this function is only for adding\n -+ * a reflog update. Supports providing custom committer information.\n ++ * a reflog update. Supports providing custom committer information. The index\n ++ * field can be utiltized to order updates as desired. When not used, the\n ++ * updates default to being ordered by refname.\n + */\n +int ref_transaction_update_reflog(struct ref_transaction *transaction,\n +\t\t\t\t const char *refname,\n7: dffc14e1a3 ! 7: 4d76cf4773 refs: allow multiple reflog entries for the same refname\n @@ Commit message\n update_index before writing to the block. When there are multiple\n reflogs for a given refname, it is essential that the order of the\n reflogs is maintained. So add the `index` value to the `update_index`.\n - The `index` field is only be set when multiple reflog entries for a\n - given refname are added and as such in most scenarios the old behavior\n + The `index` field is only set when multiple reflog entries for a given\n + refname are added and as such in most scenarios the old behavior\n remains.\n \n This is required to add reflog migration support to `git refs migrate`.\n8: 481d185e6e ! 8: 31cc392d8d refs: add support for migrating reflogs\n @@ refs.c\n \n /*\n * List of all available backends\n -@@ refs.c: int ref_update_check_old_target(const char *referent, struct ref_update *update,\n - }\n - \n - struct migration_data {\n -+\tunsigned int reflog_index;\n +@@ refs.c: struct migration_data {\n \tstruct ref_store *old_refs;\n \tstruct ref_transaction *transaction;\n \tstruct strbuf *errbuf;\n ++\tstruct strbuf sb;\n + };\n + \n + static int migrate_one_ref(const char *refname, const char *referent UNUSED, const struct object_id *oid,\n @@ refs.c: static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n \treturn ret;\n }\n \n +struct reflog_migration_data {\n -+\tunsigned int *index;\n ++\tunsigned int index;\n +\tconst char *refname;\n +\tstruct ref_store *old_refs;\n +\tstruct ref_transaction *transaction;\n +\tstruct strbuf *errbuf;\n ++\tstruct strbuf *sb;\n +};\n +\n +static int migrate_one_reflog_entry(struct object_id *old_oid,\n @@ refs.c: static int migrate_one_ref(const char *refname, const char *referent UNU\n +\t\t\t\t const char *msg, void *cb_data)\n +{\n +\tstruct reflog_migration_data *data = cb_data;\n -+\tstruct strbuf sb = STRBUF_INIT;\n +\tconst char *date;\n +\tint ret;\n +\n +\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n ++\tstrbuf_reset(data->sb);\n +\t/* committer contains name and email */\n -+\tstrbuf_addstr(&sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n ++\tstrbuf_addstr(data->sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n +\n +\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n -+\t\t\t\t\t new_oid, old_oid, sb.buf,\n ++\t\t\t\t\t new_oid, old_oid, data->sb->buf,\n +\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n -+\t\t\t\t\t (*data->index)++, data->errbuf);\n -+\tstrbuf_release(&sb);\n -+\n ++\t\t\t\t\t data->index++, data->errbuf);\n +\treturn ret;\n +}\n +\n @@ refs.c: static int migrate_one_ref(const char *refname, const char *referent UNU\n +\tdata.old_refs = migration_data->old_refs;\n +\tdata.transaction = migration_data->transaction;\n +\tdata.errbuf = migration_data->errbuf;\n -+\tdata.index = &migration_data->reflog_index;\n ++\tdata.sb = &migration_data->sb;\n +\n +\treturn refs_for_each_reflog_ent(migration_data->old_refs, refname,\n +\t\t\t\t\tmigrate_one_reflog_entry, &data);\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \t */\n \tstrbuf_addf(&new_gitdir, \"%s/%s\", old_refs->gitdir, \"ref_migration.XXXXXX\");\n \tif (!mkdtemp(new_gitdir.buf)) {\n +@@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n + \tdata.old_refs = old_refs;\n + \tdata.transaction = transaction;\n + \tdata.errbuf = errbuf;\n ++\tstrbuf_init(&data.sb, 0);\n + \n + \t/*\n + \t * We need to use the internal `do_for_each_ref()` here so that we can\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \tif (ret < 0)\n \t\tgoto done;\n \n -+\tdata.reflog_index = 1;\n +\tret = refs_for_each_reflog(old_refs, migrate_one_reflog, &data);\n +\tif (ret < 0)\n +\t\tgoto done;\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \tret = ref_transaction_commit(transaction, errbuf);\n \tif (ret < 0)\n \t\tgoto done;\n +@@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n + \t}\n + \tref_transaction_free(transaction);\n + \tstrbuf_release(&new_gitdir);\n ++\tstrbuf_release(&data.sb);\n + \treturn ret;\n + }\n + \n \n ## t/t1460-refs-migrate.sh ##\n @@ t/t1460-refs-migrate.sh: export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n\n\n--- \n\nbase-commit: 09245f4b75863f4e94dac7feebaafce53a26965f\nchange-id: 20241111-320-git-refs-migrate-reflogs-a53e3a6cffc9\n\nThanks\n- Karthik\n\n"},{"id":"509126","messageId":"20241215-320-git-refs-migrate-reflogs-v3-1-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 1/8] refs: include committer info in `ref_update` struct","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:38Z","receivedAt":"2024-12-15T16:25:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference backends obtain the committer information from\n`git_committer_info(0)` when adding a reflog. The upcoming patches\nintroduce support for migrating reflogs between the reference backends.\nThis requires an interface to creating reflogs, including custom\ncommitter information.\n\nAdd a new field `committer_info` to the `ref_update` struct, which is\nthen used by the reference backends. If there is no `committer_info`\nprovided, the reference backends default to using\n`git_committer_info(0)`. The field itself cannot be set to\n`git_committer_info(0)` since the values are dynamic and must be\nobtained right when the reflog is being committed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 1 +\n refs/files-backend.c | 24 ++++++++++++++----------\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 12 +++++++++++-\n 4 files changed, 27 insertions(+), 11 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 762f3e324d59c60cd4f05c2f257e54de8deb00e5..f003e51c6bf5229bfbce8ce61ffad7cdba0572e0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1151,6 +1151,7 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tfree(transaction->updates[i]->msg);\n+\t\tfree(transaction->updates[i]->committer_info);\n \t\tfree((char *)transaction->updates[i]->new_target);\n \t\tfree((char *)transaction->updates[i]->old_target);\n \t\tfree(transaction->updates[i]);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 64f51f0da905a9a8a1ac4109c6b0a9a85a355db7..6078668c99ee254e794e3ba49689aa34e6022efd 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1858,6 +1858,9 @@ static int log_ref_write_fd(int fd, const struct object_id *old_oid,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint ret = 0;\n \n+\tif (!committer)\n+\t\tcommitter = git_committer_info(0);\n+\n \tstrbuf_addf(&sb, \"%s %s %s\", oid_to_hex(old_oid), oid_to_hex(new_oid), committer);\n \tif (msg && *msg) {\n \t\tstrbuf_addch(&sb, '\\t');\n@@ -1871,8 +1874,10 @@ static int log_ref_write_fd(int fd, const struct object_id *old_oid,\n }\n \n static int files_log_ref_write(struct files_ref_store *refs,\n-\t\t\t const char *refname, const struct object_id *old_oid,\n-\t\t\t const struct object_id *new_oid, const char *msg,\n+\t\t\t const char *refname,\n+\t\t\t const struct object_id *old_oid,\n+\t\t\t const struct object_id *new_oid,\n+\t\t\t const char *committer_info, const char *msg,\n \t\t\t int flags, struct strbuf *err)\n {\n \tint logfd, result;\n@@ -1889,8 +1894,7 @@ static int files_log_ref_write(struct files_ref_store *refs,\n \n \tif (logfd < 0)\n \t\treturn 0;\n-\tresult = log_ref_write_fd(logfd, old_oid, new_oid,\n-\t\t\t\t git_committer_info(0), msg);\n+\tresult = log_ref_write_fd(logfd, old_oid, new_oid, committer_info, msg);\n \tif (result) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \t\tint save_errno = errno;\n@@ -1974,8 +1978,7 @@ static int commit_ref_update(struct files_ref_store *refs,\n \tfiles_assert_main_repository(refs, \"commit_ref_update\");\n \n \tclear_loose_ref_cache(refs);\n-\tif (files_log_ref_write(refs, lock->ref_name,\n-\t\t\t\t&lock->old_oid, oid,\n+\tif (files_log_ref_write(refs, lock->ref_name, &lock->old_oid, oid, NULL,\n \t\t\t\tlogmsg, flags, err)) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \t\tstrbuf_addf(err, \"cannot update the ref '%s': %s\",\n@@ -2007,9 +2010,9 @@ static int commit_ref_update(struct files_ref_store *refs,\n \t\tif (head_ref && (head_flag & REF_ISSYMREF) &&\n \t\t !strcmp(head_ref, lock->ref_name)) {\n \t\t\tstruct strbuf log_err = STRBUF_INIT;\n-\t\t\tif (files_log_ref_write(refs, \"HEAD\",\n-\t\t\t\t\t\t&lock->old_oid, oid,\n-\t\t\t\t\t\tlogmsg, flags, &log_err)) {\n+\t\t\tif (files_log_ref_write(refs, \"HEAD\", &lock->old_oid,\n+\t\t\t\t\t\toid, NULL, logmsg, flags,\n+\t\t\t\t\t\t&log_err)) {\n \t\t\t\terror(\"%s\", log_err.buf);\n \t\t\t\tstrbuf_release(&log_err);\n \t\t\t}\n@@ -2969,7 +2972,8 @@ static int parse_and_write_reflog(struct files_ref_store *refs,\n \t}\n \n \tif (files_log_ref_write(refs, lock->ref_name, &lock->old_oid,\n-\t\t\t\t&update->new_oid, update->msg, update->flags, err)) {\n+\t\t\t\t&update->new_oid, update->committer_info,\n+\t\t\t\tupdate->msg, update->flags, err)) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \n \t\tstrbuf_addf(err, \"cannot update the ref '%s': %s\",\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 58aa56d1b27c85d606ed7c8c0d908e4b87d1066b..0fd95cdacd99e4a728c22f5286f6b3f0f360c110 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -113,6 +113,7 @@ struct ref_update {\n \tvoid *backend_data;\n \tunsigned int type;\n \tchar *msg;\n+\tchar *committer_info;\n \n \t/*\n \t * If this ref_update was split off of a symref update via\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 647ef9b05b1dc9a376ed054330b487f7595c5caa..e882602487c66261d586a94101bb1b4e9a2ed60e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1379,11 +1379,21 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t}\n \n \t\t\tif (create_reflog) {\n+\t\t\t\tstruct ident_split c;\n+\n \t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n \t\t\t\tlog = &logs[logs_nr++];\n \t\t\t\tmemset(log, 0, sizeof(*log));\n \n-\t\t\t\tfill_reftable_log_record(log, &committer_ident);\n+\t\t\t\tif (u->committer_info) {\n+\t\t\t\t\tif (split_ident_line(&c, u->committer_info,\n+\t\t\t\t\t\t\t strlen(u->committer_info)))\n+\t\t\t\t\t\tBUG(\"failed splitting committer info\");\n+\t\t\t\t} else {\n+\t\t\t\t\tc = committer_ident;\n+\t\t\t\t}\n+\n+\t\t\t\tfill_reftable_log_record(log, &c);\n \t\t\t\tlog->update_index = ts;\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n\n-- \n2.47.1\n\n"},{"id":"509127","messageId":"20241215-320-git-refs-migrate-reflogs-v3-2-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 2/8] refs: add `index` field to `struct ref_udpate`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:39Z","receivedAt":"2024-12-15T16:25:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reftable backend, sorts its updates by refname before applying them,\nthis ensures that the references are stored sorted. When migrating\nreflogs from one backend to another, the order of the reflogs must be\nmaintained. Add a new `index` field to the `ref_update` struct to\nfacilitate this.\n\nThis field is used in the reftable backend's sort comparison function\n`transaction_update_cmp`, to ensure that indexed fields maintain their\norder.\n\nSigned-off-by: Karthik Nayak \n---\n refs/refs-internal.h | 7 +++++++\n refs/reftable-backend.c | 13 +++++++++++--\n 2 files changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 0fd95cdacd99e4a728c22f5286f6b3f0f360c110..f5c733d099f0c6f1076a25f4f77d9d5eb345ec87 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -115,6 +115,13 @@ struct ref_update {\n \tchar *msg;\n \tchar *committer_info;\n \n+\t/*\n+\t * The index overrides the default sort algorithm. This is needed\n+\t * when migrating reflogs and we want to ensure we carry over the\n+\t * same order.\n+\t */\n+\tunsigned int index;\n+\n \t/*\n \t * If this ref_update was split off of a symref update via\n \t * split_symref_update(), then this member points at that\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex e882602487c66261d586a94101bb1b4e9a2ed60e..c008f20be719fec3af6a8f81c821cb9c263764d7 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1279,8 +1279,17 @@ static int reftable_be_transaction_abort(struct ref_store *ref_store UNUSED,\n \n static int transaction_update_cmp(const void *a, const void *b)\n {\n-\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n-\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+\tstruct reftable_transaction_update *update_a = (struct reftable_transaction_update *)a;\n+\tstruct reftable_transaction_update *update_b = (struct reftable_transaction_update *)b;\n+\n+\t/*\n+\t * If there is an index set, it should take preference (default is 0).\n+\t * This ensures that updates with indexes are sorted amongst themselves.\n+\t */\n+\tif (update_a->update->index || update_b->update->index)\n+\t\treturn update_a->update->index - update_b->update->index;\n+\n+\treturn strcmp(update_a->update->refname, update_b->update->refname);\n }\n \n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n\n-- \n2.47.1\n\n"},{"id":"509128","messageId":"20241215-320-git-refs-migrate-reflogs-v3-3-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 3/8] refs/files: add count field to ref_lock","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:40Z","receivedAt":"2024-12-15T16:25:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When refs are updated in the files-backend, a lock is obtained for the\ncorresponding file path. This is the case even for reflogs, i.e. a lock\nis obtained on the reference path instead of the reflog path. This\nworks, since generally, reflogs are updated alongside the ref.\n\nThe upcoming patches will add support for reflog updates in ref\ntransaction. This means, in a particular transaction we want to have ref\nupdates and reflog updates. For a given ref in a given transaction there\ncan be at most one update. But we can theoretically have multiple reflog\nupdates for a given ref in a given transaction. A great example of this\nwould be when migrating reflogs from one backend to another. There we\nwould batch all the reflog updates for a given reference in a single\ntransaction.\n\nThe current flow does not support this, because currently refs & reflogs\nare treated as a single entity and capture the lock together. To\nseparate this, add a count field to ref_lock. With this, multiple\nupdates can hold onto a single ref_lock and the lock will only be\nreleased when all of them release the lock.\n\nThis patch only adds the `count` field to `ref_lock` and adds the logic\nto increment and decrement the lock. In a follow up commit, we'll\nseparate the reflog update logic from ref updates and utilize this\nfunctionality.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 58 +++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 39 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6078668c99ee254e794e3ba49689aa34e6022efd..02cb4907d8659e87a227fed4f60a5f6606be8764 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -71,6 +71,7 @@ struct ref_lock {\n \tchar *ref_name;\n \tstruct lock_file lk;\n \tstruct object_id old_oid;\n+\tunsigned int count; /* track users of the lock (ref update + reflog updates) */\n };\n \n struct files_ref_store {\n@@ -638,9 +639,12 @@ int parse_loose_ref_contents(const struct git_hash_algo *algop,\n \n static void unlock_ref(struct ref_lock *lock)\n {\n-\trollback_lock_file(&lock->lk);\n-\tfree(lock->ref_name);\n-\tfree(lock);\n+\tlock->count--;\n+\tif (!lock->count) {\n+\t\trollback_lock_file(&lock->lk);\n+\t\tfree(lock->ref_name);\n+\t\tfree(lock);\n+\t}\n }\n \n /*\n@@ -696,6 +700,7 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t*lock_p = CALLOC_ARRAY(lock, 1);\n \n \tlock->ref_name = xstrdup(refname);\n+\tlock->count = 1;\n \tfiles_ref_path(refs, &ref_file, refname);\n \n retry:\n@@ -1169,6 +1174,7 @@ static struct ref_lock *lock_ref_oid_basic(struct files_ref_store *refs,\n \t\tgoto error_return;\n \n \tlock->ref_name = xstrdup(refname);\n+\tlock->count = 1;\n \n \tif (raceproof_create_file(ref_file.buf, create_reflock, &lock->lk)) {\n \t\tunable_to_lock_message(ref_file.buf, errno, err);\n@@ -2535,6 +2541,12 @@ static int check_old_oid(struct ref_update *update, struct object_id *oid,\n \treturn -1;\n }\n \n+struct files_transaction_backend_data {\n+\tstruct ref_transaction *packed_transaction;\n+\tint packed_refs_locked;\n+\tstruct strmap ref_locks;\n+};\n+\n /*\n * Prepare for carrying out update:\n * - Lock the reference referred to by update.\n@@ -2557,11 +2569,14 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n {\n \tstruct strbuf referent = STRBUF_INIT;\n \tint mustexist = ref_update_expects_existing_old_ref(update);\n+\tstruct files_transaction_backend_data *backend_data;\n \tint ret = 0;\n \tstruct ref_lock *lock;\n \n \tfiles_assert_main_repository(refs, \"lock_ref_for_update\");\n \n+\tbackend_data = transaction->backend_data;\n+\n \tif ((update->flags & REF_HAVE_NEW) && ref_update_has_null_new_value(update))\n \t\tupdate->flags |= REF_DELETING;\n \n@@ -2572,18 +2587,25 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\tgoto out;\n \t}\n \n-\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t affected_refnames,\n-\t\t\t &lock, &referent,\n-\t\t\t &update->type, err);\n-\tif (ret) {\n-\t\tchar *reason;\n+\tlock = strmap_get(&backend_data->ref_locks, update->refname);\n+\tif (lock) {\n+\t\tlock->count++;\n+\t} else {\n+\t\tret = lock_raw_ref(refs, update->refname, mustexist,\n+\t\t\t\t affected_refnames,\n+\t\t\t\t &lock, &referent,\n+\t\t\t\t &update->type, err);\n+\t\tif (ret) {\n+\t\t\tchar *reason;\n+\n+\t\t\treason = strbuf_detach(err, NULL);\n+\t\t\tstrbuf_addf(err, \"cannot lock ref '%s': %s\",\n+\t\t\t\t ref_update_original_update_refname(update), reason);\n+\t\t\tfree(reason);\n+\t\t\tgoto out;\n+\t\t}\n \n-\t\treason = strbuf_detach(err, NULL);\n-\t\tstrbuf_addf(err, \"cannot lock ref '%s': %s\",\n-\t\t\t ref_update_original_update_refname(update), reason);\n-\t\tfree(reason);\n-\t\tgoto out;\n+\t\tstrmap_put(&backend_data->ref_locks, update->refname, lock);\n \t}\n \n \tupdate->backend_data = lock;\n@@ -2730,11 +2752,6 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \treturn ret;\n }\n \n-struct files_transaction_backend_data {\n-\tstruct ref_transaction *packed_transaction;\n-\tint packed_refs_locked;\n-};\n-\n /*\n * Unlock any references in `transaction` that are still locked, and\n * mark the transaction closed.\n@@ -2767,6 +2784,8 @@ static void files_transaction_cleanup(struct files_ref_store *refs,\n \t\tif (backend_data->packed_refs_locked)\n \t\t\tpacked_refs_unlock(refs->packed_ref_store);\n \n+\t\tstrmap_clear(&backend_data->ref_locks, 0);\n+\n \t\tfree(backend_data);\n \t}\n \n@@ -2796,6 +2815,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto cleanup;\n \n \tCALLOC_ARRAY(backend_data, 1);\n+\tstrmap_init(&backend_data->ref_locks);\n \ttransaction->backend_data = backend_data;\n \n \t/*\n\n-- \n2.47.1\n\n"},{"id":"509129","messageId":"20241215-320-git-refs-migrate-reflogs-v3-4-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 4/8] refs: extract out refname verification in transactions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:41Z","receivedAt":"2024-12-15T16:25:57Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Unless the `REF_SKIP_REFNAME_VERIFICATION` flag is set for an update,\nthe refname of the update is verified for:\n\n - Ensuring it is not a pseudoref.\n - Checking the refname format.\n\nThese checks will also be needed in a following commit where the\nfunction to add reflog updates to the transaction is introduced. Extract\nthe code out into a new static function.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 37 +++++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f003e51c6bf5229bfbce8ce61ffad7cdba0572e0..9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1196,6 +1196,28 @@ struct ref_update *ref_transaction_add_update(\n \treturn update;\n }\n \n+static int transaction_refname_valid(const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t unsigned int flags, struct strbuf *err)\n+{\n+\tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n+\t\treturn 1;\n+\n+\tif (is_pseudo_ref(refname)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n+\t\t\t refname);\n+\t\treturn 0;\n+\t} else if ((new_oid && !is_null_oid(new_oid)) ?\n+\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n+\t\t !refname_is_safe(refname)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n+\t\t\t refname);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const struct object_id *new_oid,\n@@ -1213,21 +1235,8 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\treturn -1;\n \t}\n \n-\tif (!(flags & REF_SKIP_REFNAME_VERIFICATION) &&\n-\t ((new_oid && !is_null_oid(new_oid)) ?\n-\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n-\t\t\t !refname_is_safe(refname))) {\n-\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n-\t\t\t refname);\n+\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n \t\treturn -1;\n-\t}\n-\n-\tif (!(flags & REF_SKIP_REFNAME_VERIFICATION) &&\n-\t is_pseudo_ref(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n-\t\t\t refname);\n-\t\treturn -1;\n-\t}\n \n \tif (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS)\n \t\tBUG(\"illegal flags 0x%x passed to ref_transaction_update()\", flags);\n\n-- \n2.47.1\n\n"},{"id":"509130","messageId":"20241215-320-git-refs-migrate-reflogs-v3-5-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:42Z","receivedAt":"2024-12-15T16:25:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_transaction_add_update()` creates the `ref_update` struct. To\nfacilitate addition of reflogs in the next commit, the function needs to\naccommodate setting the `committer_info` field in the struct. So modify\nthe function to also take `committer_info` as an argument and set it\naccordingly.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++--\n refs/files-backend.c | 14 ++++++++------\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 6 ++++--\n 4 files changed, 18 insertions(+), 10 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801..782bf1090af65196263a3c35ed18d878bb4f2967 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1166,6 +1166,7 @@ struct ref_update *ref_transaction_add_update(\n \t\tconst struct object_id *new_oid,\n \t\tconst struct object_id *old_oid,\n \t\tconst char *new_target, const char *old_target,\n+\t\tconst char *committer_info,\n \t\tconst char *msg)\n {\n \tstruct ref_update *update;\n@@ -1190,8 +1191,10 @@ struct ref_update *ref_transaction_add_update(\n \t\toidcpy(&update->new_oid, new_oid);\n \tif ((flags & REF_HAVE_OLD) && old_oid)\n \t\toidcpy(&update->old_oid, old_oid);\n-\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n+\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n+\t\tupdate->committer_info = xstrdup_or_null(committer_info);\n \t\tupdate->msg = normalize_reflog_message(msg);\n+\t}\n \n \treturn update;\n }\n@@ -1253,7 +1256,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n-\t\t\t\t old_target, msg);\n+\t\t\t\t old_target, NULL, msg);\n \treturn 0;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 02cb4907d8659e87a227fed4f60a5f6606be8764..255fed8354cae982f785b1b85340e2a1eeecf2a6 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1270,7 +1270,7 @@ static void prune_ref(struct files_ref_store *refs, struct ref_to_prune *r)\n \tref_transaction_add_update(\n \t\t\ttransaction, r->name,\n \t\t\tREF_NO_DEREF | REF_HAVE_NEW | REF_HAVE_OLD | REF_IS_PRUNING,\n-\t\t\tnull_oid(), &r->oid, NULL, NULL, NULL);\n+\t\t\tnull_oid(), &r->oid, NULL, NULL, NULL, NULL);\n \tif (ref_transaction_commit(transaction, &err))\n \t\tgoto cleanup;\n \n@@ -2417,7 +2417,7 @@ static int split_head_update(struct ref_update *update,\n \t\t\ttransaction, \"HEAD\",\n \t\t\tupdate->flags | REF_LOG_ONLY | REF_NO_DEREF,\n \t\t\t&update->new_oid, &update->old_oid,\n-\t\t\tNULL, NULL, update->msg);\n+\t\t\tNULL, NULL, update->committer_info, update->msg);\n \n \t/*\n \t * Add \"HEAD\". This insertion is O(N) in the transaction\n@@ -2481,7 +2481,8 @@ static int split_symref_update(struct ref_update *update,\n \t\t\ttransaction, referent, new_flags,\n \t\t\tupdate->new_target ? NULL : &update->new_oid,\n \t\t\tupdate->old_target ? NULL : &update->old_oid,\n-\t\t\tupdate->new_target, update->old_target, update->msg);\n+\t\t\tupdate->new_target, update->old_target, NULL,\n+\t\t\tupdate->msg);\n \n \tnew_update->parent_update = update;\n \n@@ -2914,7 +2915,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t\tpacked_transaction, update->refname,\n \t\t\t\t\tREF_HAVE_NEW | REF_NO_DEREF,\n \t\t\t\t\t&update->new_oid, NULL,\n-\t\t\t\t\tNULL, NULL, NULL);\n+\t\t\t\t\tNULL, NULL, NULL, NULL);\n \t\t}\n \t}\n \n@@ -3094,12 +3095,13 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t\tref_transaction_add_update(loose_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n \t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n-\t\t\t\t\t\t update->new_target, NULL, NULL);\n+\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n+\t\t\t\t\t\t NULL);\n \t\t} else {\n \t\t\tref_transaction_add_update(packed_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n \t\t\t\t\t\t &update->new_oid, &update->old_oid,\n-\t\t\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t\t\t NULL, NULL, update->committer_info, NULL);\n \t\t}\n \t}\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f5c733d099f0c6f1076a25f4f77d9d5eb345ec87..79b287c5ec5c7d8f759869cf93cda405640186dc 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -162,6 +162,7 @@ struct ref_update *ref_transaction_add_update(\n \t\tconst struct object_id *new_oid,\n \t\tconst struct object_id *old_oid,\n \t\tconst char *new_target, const char *old_target,\n+\t\tconst char *committer_info,\n \t\tconst char *msg);\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex c008f20be719fec3af6a8f81c821cb9c263764d7..b2e3ba877de9e59fea5a4d066eb13e60ef22a32b 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1078,7 +1078,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tnew_update = ref_transaction_add_update(\n \t\t\t\t\ttransaction, \"HEAD\",\n \t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n-\t\t\t\t\t&u->new_oid, &u->old_oid, NULL, NULL, u->msg);\n+\t\t\t\t\t&u->new_oid, &u->old_oid, NULL, NULL, NULL,\n+\t\t\t\t\tu->msg);\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n@@ -1161,7 +1162,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t\ttransaction, referent.buf, new_flags,\n \t\t\t\t\tu->new_target ? NULL : &u->new_oid,\n \t\t\t\t\tu->old_target ? NULL : &u->old_oid,\n-\t\t\t\t\tu->new_target, u->old_target, u->msg);\n+\t\t\t\t\tu->new_target, u->old_target,\n+\t\t\t\t\tu->committer_info, u->msg);\n \n \t\t\t\tnew_update->parent_update = u;\n \n\n-- \n2.47.1\n\n"},{"id":"509131","messageId":"20241215-320-git-refs-migrate-reflogs-v3-6-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:43Z","receivedAt":"2024-12-15T16:25:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new function `ref_transaction_update_reflog`, for clients to\nadd a reflog update to a transaction. While the existing function\n`ref_transaction_update` also allows clients to add a reflog entry, this\nfunction does a few things more, It:\n - Enforces that only a reflog entry is added and does not update the\n ref itself.\n - Allows the users to also provide the committer information. This\n means clients can add reflog entries with custom committer\n information.\n\nThe `transaction_refname_valid()` function also modifies the error\nmessage selectively based on the type of the update. This change also\naffects reflog updates which go through `ref_transaction_update()`.\n\nA follow up commit will utilize this function to add reflog support to\n`git refs migrate`.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 39 +++++++++++++++++++++++++++++++++++----\n refs.h | 14 ++++++++++++++\n refs/files-backend.c | 24 ++++++++++++++++--------\n 3 files changed, 65 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 782bf1090af65196263a3c35ed18d878bb4f2967..8b3882cff17e5e3b0376f75654e32f81a23e5cb2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1207,14 +1207,14 @@ static int transaction_refname_valid(const char *refname,\n \t\treturn 1;\n \n \tif (is_pseudo_ref(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n-\t\t\t refname);\n+\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n+\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t} else if ((new_oid && !is_null_oid(new_oid)) ?\n \t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n \t\t !refname_is_safe(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n-\t\t\t refname);\n+\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog with bad name\" : \"ref with bad name\";\n+\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t}\n \n@@ -1257,6 +1257,37 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n \t\t\t\t old_target, NULL, msg);\n+\n+\treturn 0;\n+}\n+\n+int ref_transaction_update_reflog(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t const char *committer_info, unsigned int flags,\n+\t\t\t\t const char *msg, unsigned int index,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct ref_update *update;\n+\n+\tassert(err);\n+\n+\tflags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\n+\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n+\t\treturn -1;\n+\n+\tupdate = ref_transaction_add_update(transaction, refname, flags,\n+\t\t\t\t\t new_oid, old_oid, NULL, NULL,\n+\t\t\t\t\t committer_info, msg);\n+\t/*\n+\t * While we do set the old_oid value, we unset the flag to skip\n+\t * old_oid verification which only makes sense for refs.\n+\t */\n+\tupdate->flags &= ~REF_HAVE_OLD;\n+\tupdate->index = index;\n+\n \treturn 0;\n }\n \ndiff --git a/refs.h b/refs.h\nindex a5bedf48cf6de91005a7e8d0bf58ca98350397a6..b0dfc65ed2e59c4b66967840339f81e7746a96d3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -727,6 +727,20 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t unsigned int flags, const char *msg,\n \t\t\t struct strbuf *err);\n \n+/*\n+ * Similar to`ref_transaction_update`, but this function is only for adding\n+ * a reflog update. Supports providing custom committer information. The index\n+ * field can be utiltized to order updates as desired. When not used, the\n+ * updates default to being ordered by refname.\n+ */\n+int ref_transaction_update_reflog(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t const char *committer_info, unsigned int flags,\n+\t\t\t\t const char *msg, unsigned int index,\n+\t\t\t\t struct strbuf *err);\n+\n /*\n * Add a reference creation to transaction. new_oid is the value that\n * the reference should have after the update; it must not be\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 255fed8354cae982f785b1b85340e2a1eeecf2a6..c11213f52065bcf2fa7612df8f9500692ee2d02c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3080,10 +3080,12 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * packed-refs don't support symbolic refs and root refs, so we\n-\t\t * have to queue these references via the loose transaction.\n+\t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n+\t\t * so we have to queue these references via the loose transaction.\n \t\t */\n-\t\tif (update->new_target || is_root_ref(update->refname)) {\n+\t\tif (update->new_target ||\n+\t\t is_root_ref(update->refname) ||\n+\t\t (update->flags & REF_LOG_ONLY)) {\n \t\t\tif (!loose_transaction) {\n \t\t\t\tloose_transaction = ref_store_transaction_begin(&refs->base, 0, err);\n \t\t\t\tif (!loose_transaction) {\n@@ -3092,11 +3094,17 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t\t\t}\n \t\t\t}\n \n-\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n-\t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n-\t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n-\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n-\t\t\t\t\t\t NULL);\n+\t\t\tif (update->flags & REF_LOG_ONLY)\n+\t\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n+\t\t\t\t\t\t\t update->flags, &update->new_oid,\n+\t\t\t\t\t\t\t &update->old_oid, NULL, NULL,\n+\t\t\t\t\t\t\t update->committer_info, update->msg);\n+\t\t\telse\n+\t\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n+\t\t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n+\t\t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n+\t\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n+\t\t\t\t\t\t\t NULL);\n \t\t} else {\n \t\t\tref_transaction_add_update(packed_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n\n-- \n2.47.1\n\n"},{"id":"509132","messageId":"20241215-320-git-refs-migrate-reflogs-v3-7-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 7/8] refs: allow multiple reflog entries for the same refname","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:44Z","receivedAt":"2024-12-15T16:25:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference transaction only allows a single update for a given\nreference to avoid conflicts. This, however, isn't an issue for reflogs.\nThere are no conflicts to be resolved in reflogs and when migrating\nreflogs between backends we'd have multiple reflog entries for the same\nrefname.\n\nSo allow multiple reflog updates within a single transaction. Also the\nreflog creation logic isn't exposed to the end user. While this might\nchange in the future, currently, this reduces the scope of issues to\nthink about.\n\nIn the reftable backend, the writer sorts all updates based on the\nupdate_index before writing to the block. When there are multiple\nreflogs for a given refname, it is essential that the order of the\nreflogs is maintained. So add the `index` value to the `update_index`.\nThe `index` field is only set when multiple reflog entries for a given\nrefname are added and as such in most scenarios the old behavior\nremains.\n\nThis is required to add reflog migration support to `git refs migrate`.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 15 +++++++++++----\n refs/reftable-backend.c | 22 +++++++++++++++++++---\n 2 files changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex c11213f52065bcf2fa7612df8f9500692ee2d02c..8953d1c6d37b13b0db701888b3db92fd87a68aaa 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2611,6 +2611,9 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \n \tupdate->backend_data = lock;\n \n+\tif (update->flags & REF_LOG_ONLY)\n+\t\tgoto out;\n+\n \tif (update->type & REF_ISSYMREF) {\n \t\tif (update->flags & REF_NO_DEREF) {\n \t\t\t/*\n@@ -2829,13 +2832,16 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t */\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *update = transaction->updates[i];\n-\t\tstruct string_list_item *item =\n-\t\t\tstring_list_append(&affected_refnames, update->refname);\n+\t\tstruct string_list_item *item;\n \n \t\tif ((update->flags & REF_IS_PRUNING) &&\n \t\t !(update->flags & REF_NO_DEREF))\n \t\t\tBUG(\"REF_IS_PRUNING set without REF_NO_DEREF\");\n \n+\t\tif (update->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\titem = string_list_append(&affected_refnames, update->refname);\n \t\t/*\n \t\t * We store a pointer to update in item->util, but at\n \t\t * the moment we never use the value of this field\n@@ -3035,8 +3041,9 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \n \t/* Fail if a refname appears more than once in the transaction: */\n \tfor (i = 0; i < transaction->nr; i++)\n-\t\tstring_list_append(&affected_refnames,\n-\t\t\t\t transaction->updates[i]->refname);\n+\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n+\t\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t\t transaction->updates[i]->refname);\n \tstring_list_sort(&affected_refnames);\n \tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -990,8 +990,9 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tif (ret)\n \t\t\tgoto done;\n \n-\t\tstring_list_append(&affected_refnames,\n-\t\t\t\t transaction->updates[i]->refname);\n+\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n+\t\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t\t transaction->updates[i]->refname);\n \t}\n \n \t/*\n@@ -1301,6 +1302,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tuint64_t max_update_index = ts;\n \tconst char *committer_info;\n \tint ret = 0;\n \n@@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t\t}\n \n \t\t\t\tfill_reftable_log_record(log, &c);\n-\t\t\t\tlog->update_index = ts;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Updates are sorted by the writer. So updates for the same\n+\t\t\t\t * refname need to contain different update indices.\n+\t\t\t\t */\n+\t\t\t\tlog->update_index = ts + u->index;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Note the max update_index so the limit can be set later on.\n+\t\t\t\t */\n+\t\t\t\tif (log->update_index > max_update_index)\n+\t\t\t\t\tmax_update_index = log->update_index;\n+\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n \t\t\t\t u->new_oid.hash, GIT_MAX_RAWSZ);\n@@ -1469,6 +1483,8 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t * and log blocks.\n \t */\n \tif (logs) {\n+\t\treftable_writer_set_limits(writer, ts, max_update_index);\n+\n \t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n\n-- \n2.47.1\n\n"},{"id":"509133","messageId":"20241215-320-git-refs-migrate-reflogs-v3-8-4127fe707b98@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v3 8/8] refs: add support for migrating reflogs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-15T16:25:45Z","receivedAt":"2024-12-15T16:26:00Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs migrate` command was introduced in\n25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n2024-06-06) to support migrating from one reference backend to another.\n\nOne limitation of the command was that it didn't support migrating\nrepositories which contained reflogs. A previous commit, added support\nfor adding reflog updates in ref transactions. Using the added\nfunctionality bake in reflog support for `git refs migrate`.\n\nTo ensure that the order of the reflogs is maintained during the\nmigration, we add the index for each reflog update as we iterate over\nthe reflogs from the old reference backend. This is to ensure that the\norder is maintained in the new backend.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-refs.txt | 2 --\n refs.c | 89 ++++++++++++++++++++++++++++++++--------------\n t/t1460-refs-migrate.sh | 73 +++++++++++++++++++++++++------------\n 3 files changed, 113 insertions(+), 51 deletions(-)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex ce31f93061db5e5d16aca516dd3d15f6527db870..9829984b0a4c4f54ec7f9b6c6c7072f62b1d198d 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -57,8 +57,6 @@ KNOWN LIMITATIONS\n \n The ref format migration has several known limitations in its current form:\n \n-* It is not possible to migrate repositories that have reflogs.\n-\n * It is not possible to migrate repositories that have worktrees.\n \n * There is no way to block concurrent writes to the repository during an\ndiff --git a/refs.c b/refs.c\nindex 8b3882cff17e5e3b0376f75654e32f81a23e5cb2..4a74f7c7bd0314ad8e6c4cbea436df934b2c7f88 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -30,6 +30,7 @@\n #include \"date.h\"\n #include \"commit.h\"\n #include \"wildmatch.h\"\n+#include \"ident.h\"\n \n /*\n * List of all available backends\n@@ -2673,6 +2674,7 @@ struct migration_data {\n \tstruct ref_store *old_refs;\n \tstruct ref_transaction *transaction;\n \tstruct strbuf *errbuf;\n+\tstruct strbuf sb;\n };\n \n static int migrate_one_ref(const char *refname, const char *referent UNUSED, const struct object_id *oid,\n@@ -2705,6 +2707,52 @@ static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n \treturn ret;\n }\n \n+struct reflog_migration_data {\n+\tunsigned int index;\n+\tconst char *refname;\n+\tstruct ref_store *old_refs;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf *errbuf;\n+\tstruct strbuf *sb;\n+};\n+\n+static int migrate_one_reflog_entry(struct object_id *old_oid,\n+\t\t\t\t struct object_id *new_oid,\n+\t\t\t\t const char *committer,\n+\t\t\t\t timestamp_t timestamp, int tz,\n+\t\t\t\t const char *msg, void *cb_data)\n+{\n+\tstruct reflog_migration_data *data = cb_data;\n+\tconst char *date;\n+\tint ret;\n+\n+\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n+\tstrbuf_reset(data->sb);\n+\t/* committer contains name and email */\n+\tstrbuf_addstr(data->sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n+\n+\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n+\t\t\t\t\t new_oid, old_oid, data->sb->buf,\n+\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n+\t\t\t\t\t data->index++, data->errbuf);\n+\treturn ret;\n+}\n+\n+static int migrate_one_reflog(const char *refname, void *cb_data)\n+{\n+\tstruct migration_data *migration_data = cb_data;\n+\tstruct reflog_migration_data data;\n+\n+\tdata.refname = refname;\n+\tdata.old_refs = migration_data->old_refs;\n+\tdata.transaction = migration_data->transaction;\n+\tdata.errbuf = migration_data->errbuf;\n+\tdata.sb = &migration_data->sb;\n+\n+\treturn refs_for_each_reflog_ent(migration_data->old_refs, refname,\n+\t\t\t\t\tmigrate_one_reflog_entry, &data);\n+}\n+\n static int move_files(const char *from_path, const char *to_path, struct strbuf *errbuf)\n {\n \tstruct strbuf from_buf = STRBUF_INIT, to_buf = STRBUF_INIT;\n@@ -2771,13 +2819,6 @@ static int move_files(const char *from_path, const char *to_path, struct strbuf\n \treturn ret;\n }\n \n-static int count_reflogs(const char *reflog UNUSED, void *payload)\n-{\n-\tsize_t *reflog_count = payload;\n-\t(*reflog_count)++;\n-\treturn 0;\n-}\n-\n static int has_worktrees(void)\n {\n \tstruct worktree **worktrees = get_worktrees();\n@@ -2803,7 +2844,6 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tstruct ref_transaction *transaction = NULL;\n \tstruct strbuf new_gitdir = STRBUF_INIT;\n \tstruct migration_data data;\n-\tsize_t reflog_count = 0;\n \tint did_migrate_refs = 0;\n \tint ret;\n \n@@ -2815,21 +2855,6 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \n \told_refs = get_main_ref_store(repo);\n \n-\t/*\n-\t * We do not have any interfaces that would allow us to write many\n-\t * reflog entries. Once we have them we can remove this restriction.\n-\t */\n-\tif (refs_for_each_reflog(old_refs, count_reflogs, &reflog_count) < 0) {\n-\t\tstrbuf_addstr(errbuf, \"cannot count reflogs\");\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tif (reflog_count) {\n-\t\tstrbuf_addstr(errbuf, \"migrating reflogs is not supported yet\");\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\n \t/*\n \t * Worktrees complicate the migration because every worktree has a\n \t * separate ref storage. While it should be feasible to implement, this\n@@ -2855,17 +2880,21 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t * This operation is safe as we do not yet modify the main\n \t * repository.\n \t *\n-\t * 3. If we're in dry-run mode then we are done and can hand over the\n+\t * 3. Enumerate all reflogs and write them into the new ref storage.\n+\t * This operation is safe as we do not yet modify the main\n+\t * repository.\n+\t *\n+\t * 4. If we're in dry-run mode then we are done and can hand over the\n \t * directory to the caller for inspection. If not, we now start\n \t * with the destructive part.\n \t *\n-\t * 4. Delete the old ref storage from disk. As we have a copy of refs\n+\t * 5. Delete the old ref storage from disk. As we have a copy of refs\n \t * in the new ref storage it's okay(ish) if we now get interrupted\n \t * as there is an equivalent copy of all refs available.\n \t *\n-\t * 5. Move the new ref storage files into place.\n+\t * 6. Move the new ref storage files into place.\n \t *\n-\t * 6. Change the repository format to the new ref format.\n+\t * 7. Change the repository format to the new ref format.\n \t */\n \tstrbuf_addf(&new_gitdir, \"%s/%s\", old_refs->gitdir, \"ref_migration.XXXXXX\");\n \tif (!mkdtemp(new_gitdir.buf)) {\n@@ -2889,6 +2918,7 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tdata.old_refs = old_refs;\n \tdata.transaction = transaction;\n \tdata.errbuf = errbuf;\n+\tstrbuf_init(&data.sb, 0);\n \n \t/*\n \t * We need to use the internal `do_for_each_ref()` here so that we can\n@@ -2907,6 +2937,10 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tif (ret < 0)\n \t\tgoto done;\n \n+\tret = refs_for_each_reflog(old_refs, migrate_one_reflog, &data);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = ref_transaction_commit(transaction, errbuf);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2982,6 +3016,7 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t}\n \tref_transaction_free(transaction);\n \tstrbuf_release(&new_gitdir);\n+\tstrbuf_release(&data.sb);\n \treturn ret;\n }\n \ndiff --git a/t/t1460-refs-migrate.sh b/t/t1460-refs-migrate.sh\nindex 1bfff3a7afd5acc470424dfe7ec3e97d45f5c481..f59bc4860f19c4af82dc6f2984bdb69d61fe3ec2 100755\n--- a/t/t1460-refs-migrate.sh\n+++ b/t/t1460-refs-migrate.sh\n@@ -7,23 +7,44 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+# Migrate the provided repository from one format to the other and\n+# verify that the references and logs are migrated over correctly.\n+# Usage: test_migration \n+# is the relative path to the repo to be migrated.\n+# is the ref format to be migrated to.\n+# (true or false) whether to skip reflog verification.\n test_migration () {\n-\tgit -C \"$1\" for-each-ref --include-root-refs \\\n+\trepo=$1 &&\n+\tformat=$2 &&\n+\tskip_reflog_verify=${3:-false} &&\n+\tgit -C \"$repo\" for-each-ref --include-root-refs \\\n \t\t--format='%(refname) %(objectname) %(symref)' >expect &&\n-\tgit -C \"$1\" refs migrate --ref-format=\"$2\" &&\n-\tgit -C \"$1\" for-each-ref --include-root-refs \\\n+\tif ! $skip_reflog_verify\n+\tthen\n+\t git -C \"$repo\" reflog --all >expect_logs &&\n+\t git -C \"$repo\" reflog list >expect_log_list\n+\tfi &&\n+\n+\tgit -C \"$repo\" refs migrate --ref-format=\"$2\" &&\n+\n+\tgit -C \"$repo\" for-each-ref --include-root-refs \\\n \t\t--format='%(refname) %(objectname) %(symref)' >actual &&\n \ttest_cmp expect actual &&\n+\tif ! $skip_reflog_verify\n+\tthen\n+\t\tgit -C \"$repo\" reflog --all >actual_logs &&\n+\t\tgit -C \"$repo\" reflog list >actual_log_list &&\n+\t\ttest_cmp expect_logs actual_logs &&\n+\t\ttest_cmp expect_log_list actual_log_list\n+\tfi &&\n \n-\tgit -C \"$1\" rev-parse --show-ref-format >actual &&\n-\techo \"$2\" >expect &&\n+\tgit -C \"$repo\" rev-parse --show-ref-format >actual &&\n+\techo \"$format\" >expect &&\n \ttest_cmp expect actual\n }\n \n test_expect_success 'setup' '\n-\trm -rf .git &&\n-\t# The migration does not yet support reflogs.\n-\tgit config --global core.logAllRefUpdates false\n+\trm -rf .git\n '\n \n test_expect_success \"superfluous arguments\" '\n@@ -78,19 +99,6 @@ do\n \t\t\ttest_cmp expect err\n \t\t'\n \n-\t\ttest_expect_success \"$from_format -> $to_format: migration with reflog fails\" '\n-\t\t\ttest_when_finished \"rm -rf repo\" &&\n-\t\t\tgit init --ref-format=$from_format repo &&\n-\t\t\ttest_config -C repo core.logAllRefUpdates true &&\n-\t\t\ttest_commit -C repo logged &&\n-\t\t\ttest_must_fail git -C repo refs migrate \\\n-\t\t\t\t--ref-format=$to_format 2>err &&\n-\t\t\tcat >expect <<-EOF &&\n-\t\t\terror: migrating reflogs is not supported yet\n-\t\t\tEOF\n-\t\t\ttest_cmp expect err\n-\t\t'\n-\n \t\ttest_expect_success \"$from_format -> $to_format: migration with worktree fails\" '\n \t\t\ttest_when_finished \"rm -rf repo\" &&\n \t\t\tgit init --ref-format=$from_format repo &&\n@@ -141,7 +149,7 @@ do\n \t\t\ttest_commit -C repo initial &&\n \t\t\ttest-tool -C repo ref-store main update-ref \"\" refs/heads/broken \\\n \t\t\t\t\"$(test_oid 001)\" \"$ZERO_OID\" REF_SKIP_CREATE_REFLOG,REF_SKIP_OID_VERIFICATION &&\n-\t\t\ttest_migration repo \"$to_format\" &&\n+\t\t\ttest_migration repo \"$to_format\" true &&\n \t\t\ttest_oid 001 >expect &&\n \t\t\tgit -C repo rev-parse refs/heads/broken >actual &&\n \t\t\ttest_cmp expect actual\n@@ -195,6 +203,27 @@ do\n \t\t\tgit -C repo rev-parse --show-ref-format >actual &&\n \t\t\ttest_cmp expect actual\n \t\t'\n+\n+\t\ttest_expect_success \"$from_format -> $to_format: reflogs of symrefs with target deleted\" '\n+\t\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t\tgit init --ref-format=$from_format repo &&\n+\t\t\ttest_commit -C repo initial &&\n+\t\t\tgit -C repo branch branch-1 HEAD &&\n+\t\t\tgit -C repo symbolic-ref refs/heads/symref refs/heads/branch-1 &&\n+\t\t\tcat >input <<-EOF &&\n+\t\t\tdelete refs/heads/branch-1\n+\t\t\tEOF\n+\t\t\tgit -C repo update-ref --stdin $to_format: reflogs order is retained\" '\n+\t\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t\tgit init --ref-format=$from_format repo &&\n+\t\t\ttest_commit --date \"100005000 +0700\" --no-tag -C repo initial &&\n+\t\t\ttest_commit --date \"100003000 +0700\" --no-tag -C repo second &&\n+\t\t\ttest_migration repo \"$to_format\"\n+\t\t'\n \tdone\n done\n \n\n-- \n2.47.1\n\n"},{"id":"509139","messageId":"xmqq34io8ptl.fsf@gitster.g","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"Re: [PATCH v3 0/8] refs: add reflog support to `git refs migrate`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-15T23:54:46Z","receivedAt":"2024-12-15T23:54:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Overall, this series is a bit more involved, and I would appreciate it\n> if it receives a bit more scrutiny.\n>\n> The series is based on top of e66fd72e97 (The fourteenth batch,\n> 2024-12-06) with `kn/reftable-writer-log-write-verify` merged in.\n\nt1460.6 does not pass for me. I noticed it after I merged it to\n'seen', but it fails standalone as well.\n\n $ git log --first-parent --boundary master..kn/reflog-migration\n 65a841566a refs: add support for migrating reflogs\n 44ffb130f3 refs: allow multiple reflog entries for the same refname\n 6134624eaf refs: introduce the `ref_transaction_update_reflog` function\n 65a73fce30 refs: add `committer_info` to `ref_transaction_add_update()`\n 5ed33e32c7 refs: extract out refname verification in transactions\n e9851924a2 refs/files: add count field to ref_lock\n cdfa2c379a refs: add `index` field to `struct ref_udpate`\n ee4d52c7f2 refs: include committer info in `ref_update` struct\n df5d7a7ba5 Merge branch 'kn/reftable-writer-log-write-verify' into kn/reflog-migration\n - 49c6b912e2 reftable/writer: ensure valid range for log's update_index\n - e66fd72e97 The fourteenth batch\n\n...\nok 5 - files: migration to same format fails\n\nexpecting success of 1460.6 'files -> reftable: migration with worktree fails':\n test_when_finished \"rm -rf repo\" &&\n git init --ref-format=$from_format repo &&\n git -C repo worktree add wt &&\n test_must_fail git -C repo refs migrate \\\n --ref-format=$to_format 2>err &&\n cat >expect <<-EOF &&\n error: migrating repositories with worktrees is not supported yet\n EOF\n test_cmp expect err\n\n++ test_when_finished 'rm -rf repo'\n++ test 0 = 0\n++ test_cleanup='{ rm -rf repo\n } && (exit \"$eval_ret\"); eval_ret=$?; :'\n++ git init --ref-format=files repo\nInitialized empty Git repository in /home/gitster/w/git.git/t/trash directory.t1460-refs-migrate/repo/.git/\n++ git -C repo worktree add wt\nNo possible source branch, inferring '--orphan'\nPreparing worktree (new branch 'wt')\n++ test_must_fail git -C repo refs migrate --ref-format=reftable\n++ case \"$1\" in\n++ _test_ok=\n++ test_must_fail_acceptable git -C repo refs migrate --ref-format=reftable\n++ test git = env\n++ test git = nongit\n++ case \"$1\" in\n++ return 0\n++ git -C repo refs migrate --ref-format=reftable\n/home/gitster/w/git.git/t/test-lib-functions.sh: line 1175: 3403665 Aborted \"$@\" 2>&7\n++ exit_code=134\n++ test 134 -eq 0\n++ test_match_signal 13 134\n++ test 134 = 141\n++ test 134 = 269\n++ return 1\n++ test 134 -gt 129\n++ test 134 -le 192\n++ echo 'test_must_fail: died by signal 6: git -C repo refs migrate --ref-format=reftable'\ntest_must_fail: died by signal 6: git -C repo refs migrate --ref-format=reftable\n++ return 1\nerror: last command exited with $?=1\nnot ok 6 - files -> reftable: migration with worktree fails\n#\n# test_when_finished \"rm -rf repo\" &&\n# git init --ref-format=$from_format repo &&\n# git -C repo worktree add wt &&\n# test_must_fail git -C repo refs migrate \\\n# --ref-format=$to_format 2>err &&\n# cat >expect <<-EOF &&\n# error: migrating repositories with worktrees is not supported yet\n# EOF\n# test_cmp expect err\n#\n1..6\n"},{"id":"509140","messageId":"Z1_KzlKc7RBfas4L@pks.im","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-8-4127fe707b98@gmail.com","subject":"Re: [PATCH v3 8/8] refs: add support for migrating reflogs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-16T07:25:19Z","receivedAt":"2024-12-16T07:25:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Dec 15, 2024 at 05:25:45PM +0100, Karthik Nayak wrote:\n> @@ -2705,6 +2707,52 @@ static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n> \treturn ret;\n> }\n> \n> +struct reflog_migration_data {\n> +\tunsigned int index;\n> +\tconst char *refname;\n> +\tstruct ref_store *old_refs;\n> +\tstruct ref_transaction *transaction;\n> +\tstruct strbuf *errbuf;\n> +\tstruct strbuf *sb;\n> +};\n> +\n> +static int migrate_one_reflog_entry(struct object_id *old_oid,\n> +\t\t\t\t struct object_id *new_oid,\n> +\t\t\t\t const char *committer,\n> +\t\t\t\t timestamp_t timestamp, int tz,\n> +\t\t\t\t const char *msg, void *cb_data)\n> +{\n> +\tstruct reflog_migration_data *data = cb_data;\n> +\tconst char *date;\n> +\tint ret;\n> +\n> +\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n> +\tstrbuf_reset(data->sb);\n> +\t/* committer contains name and email */\n> +\tstrbuf_addstr(data->sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n> +\n> +\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n> +\t\t\t\t\t new_oid, old_oid, data->sb->buf,\n> +\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n> +\t\t\t\t\t data->index++, data->errbuf);\n> +\treturn ret;\n> +}\n> +\n> +static int migrate_one_reflog(const char *refname, void *cb_data)\n> +{\n> +\tstruct migration_data *migration_data = cb_data;\n> +\tstruct reflog_migration_data data;\n> +\n> +\tdata.refname = refname;\n> +\tdata.old_refs = migration_data->old_refs;\n> +\tdata.transaction = migration_data->transaction;\n> +\tdata.errbuf = migration_data->errbuf;\n> +\tdata.sb = &migration_data->sb;\n\nThe `index` variable isn't getting initialized here anymore, so its\nvalue is essenitally random. I'd propose to use designated initializers\nfor `data` to fix this:\n\n struct reflog_migration_data data = {\n .refname = refname,\n .old_refs = migration_data->old_refs,\n .transaction = migration_data->transaction,\n .errbuf = migration_data->errbuf,\n .sb = &migration_data->sb,\n };\n\nMaybe that fixes the issue that Junio has seen?\n\nPatrick\n"},{"id":"509162","messageId":"CAOLa=ZQHaGO+o=9vda0eA62K1kvdpi0yjm6adxEDhVhn4NFbpA@mail.gmail.com","threadId":"62635","inReplyTo":"xmqq34io8ptl.fsf@gitster.g","subject":"Re: [PATCH v3 0/8] refs: add reflog support to `git refs migrate`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T14:33:24Z","receivedAt":"2024-12-16T14:33:26Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Overall, this series is a bit more involved, and I would appreciate it\n>> if it receives a bit more scrutiny.\n>>\n>> The series is based on top of e66fd72e97 (The fourteenth batch,\n>> 2024-12-06) with `kn/reftable-writer-log-write-verify` merged in.\n>\n> t1460.6 does not pass for me. I noticed it after I merged it to\n> 'seen', but it fails standalone as well.\n>\n\nThanks Junio, seems like this passes on GCC and that is what I was\nusing. Sadly, it also passes on older clang version, which is what the\nCI uses. Unfortunately I assumed that the CI passing [1] should be\nvalidation enough. But I can indeed reproduce this locally with clang.\n\nPatrick posted a fix on the list [1] and also discovered one more while\nwe were discussing off the list. I'll send in the next version with both\nof those included once I validate all the tests once more.\n\n[1]: https://gitlab.com/gitlab-org/git/-/pipelines/1589854339\n[2]: https://lore.kernel.org/r/Z1_KzlKc7RBfas4L@pks.im\n\n[snip]\n"},{"id":"509165","messageId":"xmqqo71b7hkn.fsf@gitster.g","threadId":"62635","inReplyTo":"Z1_KzlKc7RBfas4L@pks.im","subject":"Re: [PATCH v3 8/8] refs: add support for migrating reflogs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-16T15:50:32Z","receivedAt":"2024-12-16T15:50:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> +static int migrate_one_reflog(const char *refname, void *cb_data)\n>> +{\n>> +\tstruct migration_data *migration_data = cb_data;\n>> +\tstruct reflog_migration_data data;\n>> +\n>> +\tdata.refname = refname;\n>> +\tdata.old_refs = migration_data->old_refs;\n>> +\tdata.transaction = migration_data->transaction;\n>> +\tdata.errbuf = migration_data->errbuf;\n>> +\tdata.sb = &migration_data->sb;\n>\n> The `index` variable isn't getting initialized here anymore, so its\n> value is essenitally random. I'd propose to use designated initializers\n> for `data` to fix this:\n>\n> struct reflog_migration_data data = {\n> .refname = refname,\n> .old_refs = migration_data->old_refs,\n> .transaction = migration_data->transaction,\n> .errbuf = migration_data->errbuf,\n> .sb = &migration_data->sb,\n> };\n\nGOod. As long as it is sensible to null-initialize the relevant\nfield and all the other fields not mentioned above, that certainly\nwould give us more predicitable behaviour ;-). I do not offhand\nknow if 0 is the right value to initialize the .index member with,\nthough; didn't you two recently had an exchange about starting with\n0 or 1 or something?\n\nThanks.\n"},{"id":"509166","messageId":"CAOLa=ZQniLNyxmvypm8BwMHyBRNb6E+SM8AXZiJpprpda5A4bg@mail.gmail.com","threadId":"62635","inReplyTo":"xmqqo71b7hkn.fsf@gitster.g","subject":"Re: [PATCH v3 8/8] refs: add support for migrating reflogs","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T15:59:57Z","receivedAt":"2024-12-16T15:59:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Patrick Steinhardt writes:\n>\n>>> +static int migrate_one_reflog(const char *refname, void *cb_data)\n>>> +{\n>>> +\tstruct migration_data *migration_data = cb_data;\n>>> +\tstruct reflog_migration_data data;\n>>> +\n>>> +\tdata.refname = refname;\n>>> +\tdata.old_refs = migration_data->old_refs;\n>>> +\tdata.transaction = migration_data->transaction;\n>>> +\tdata.errbuf = migration_data->errbuf;\n>>> +\tdata.sb = &migration_data->sb;\n>>\n>> The `index` variable isn't getting initialized here anymore, so its\n>> value is essenitally random. I'd propose to use designated initializers\n>> for `data` to fix this:\n>>\n>> struct reflog_migration_data data = {\n>> .refname = refname,\n>> .old_refs = migration_data->old_refs,\n>> .transaction = migration_data->transaction,\n>> .errbuf = migration_data->errbuf,\n>> .sb = &migration_data->sb,\n>> };\n>\n> GOod. As long as it is sensible to null-initialize the relevant\n> field and all the other fields not mentioned above, that certainly\n> would give us more predicitable behaviour ;-). I do not offhand\n> know if 0 is the right value to initialize the .index member with,\n> though; didn't you two recently had an exchange about starting with\n> 0 or 1 or something?\n>\n\nWe did [1] indeed, context is that I set it to '1' to distinguish\nbetween indexed and non-indexed updates. But it wasn't logically needed\nand was confusing so I decided to remove that change (which caused the\nissue here!).\n\n[1]: https://lore.kernel.org/r/CAOLa=ZQ9SHD3gzTVaznGhkCBjrrJbHm1fDyi1F-h6VZvtdpxgw@mail.gmail.com\n\n> Thanks.\n"},{"id":"509171","messageId":"xmqq1py77fm3.fsf@gitster.g","threadId":"62635","inReplyTo":"CAOLa=ZQHaGO+o=9vda0eA62K1kvdpi0yjm6adxEDhVhn4NFbpA@mail.gmail.com","subject":"Re: [PATCH v3 0/8] refs: add reflog support to `git refs migrate`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-16T16:32:52Z","receivedAt":"2024-12-16T16:32:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"karthik nayak writes:\n\n> Patrick posted a fix on the list [1] and also discovered one more while\n> we were discussing off the list. I'll send in the next version with both\n> of those included once I validate all the tests once more.\n\nThanks.\n"},{"id":"509172","messageId":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com","subject":"[PATCH v4 0/8] refs: add reflog support to `git refs migrate`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:25Z","receivedAt":"2024-12-16T16:44:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs migrate` command was introduced in\n25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n2024-06-06) to support migrating from one reference backend to another.\n\nOne limitation of the feature was that it didn't support migrating\nrepositories which contained reflogs. This isn't a requirement on the\nserver side as repositories are stored as bare repositories (which do\nnot contain any reflogs). Clients however generally use reflogs and\nuntil now couldn't use the `git refs migrate` command to migrate their\nrepositories to the new reftable format.\n\nOne of the issues for adding reflog support is that the ref transactions\ndon't support reflogs additions:\n 1. While there is REF_LOG_ONLY flag, there is no function to utilize\n the flag and add reflogs.\n 2. reference backends generally sort the updates by the refname. This\n wouldn't work for reflogs which need to ensure that they maintain the\n order of creation.\n 3. In the files backend, reflog entries are added by obtaining locks\n on the refs themselves. This means each update in the transaction, will\n obtain a ref_lock. This paradigm fails to accompany the fact that there\n could be multiple reflog updates for a refname in a single transaction.\n 4. The backends check for duplicate entries, which doesn't make sense\n in the context of adding multiple reflogs for a given refname.\n\nWe overcome these issue we make the following changes:\n - Update the ref_update structure to also include the committer\n information. Using this, we can add a new function which only adds\n reflog updates to the transaction.\n - Add an index field to the ref_update structure, this will help order\n updates in pre-defined order, this fixes #2.\n - While the ideal fix for #3 would be to actually introduce reflog\n locks, this wouldn't be possible without breaking backward\n compatibility. So we add a count field to the existing ref_lock. With\n this, multiple reflog updates can share a single ref_lock.\n\nOverall, this series is a bit more involved, and I would appreciate it\nif it receives a bit more scrutiny.\n\nThe series is based on top of e66fd72e97 (The fourteenth batch,\n2024-12-06) with `kn/reftable-writer-log-write-verify` merged in.\n\nSigned-off-by: Karthik Nayak \n---\nChanges in v4:\n- Fix broken tests, due to two reasons in patch 8/8:\n - The `index` field in `reflog_migration_data` wasn't initialized to\n 0. This specifically doesn't break the test, but causes undefined\n behavior. Fix this by using designated initializers.\n - The strbuf within `migration_data` wasn't initialized when the flow\n exited early, causing memory leaks. Fix this too by using designated\n initializers.\n- Thanks to Junio for reporting and Patrick for helping shed some light\n on these broken tests.\n- Link to v3: https://lore.kernel.org/r/20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com\n\nChanges in v3:\n- patch 5: Use `xstrdup_or_null` unconditionally.\n- patch 6: In `transaction_refname_valid()` use the transaction flags\n to identify reflogs. Update the documentation to also mention the\n purpose of the `index` field.\n- patch 8: Instead of allocating an strbuf for each reflog entry, we\n store and re-use one in the migration callback data.\n- patch 8: Don't use a global index increment for all reflogs entries,\n instead create and use one per reflog.\n- patch 8: Avoid setting the first reflog index to `1`. This would default\n to `0` as the first index, which is okay, since the index is incremented\n for consequtive reflog entries.\n- Small typo fixes.\n- Thanks to Christian and Patrick for the review!\n- Link to v2: https://lore.kernel.org/all/20241213-320-git-refs-migrate-reflogs-v2-0-f28312cdb6c0@gmail.com/\n\nChanges in v2:\n- Split patch 5 into two separate patches. This should make it easier to\n review and reduce cognitive load in a single patch.\n- In reftable backend, instead of using `strmapint` to ensure we have\n new update_indexes for reflogs with the same refname, we now use the\n already available `update->index` field to increment the update_index.\n- Cleanup the code and follow some of the better practices.\n- Add some clarity to the commit messages.\n- Link to v1: https://lore.kernel.org/r/20241209-320-git-refs-migrate-reflogs-v1-0-d4bc37ee860f@gmail.com\n\n---\nKarthik Nayak (8):\n refs: include committer info in `ref_update` struct\n refs: add `index` field to `struct ref_udpate`\n refs/files: add count field to ref_lock\n refs: extract out refname verification in transactions\n refs: add `committer_info` to `ref_transaction_add_update()`\n refs: introduce the `ref_transaction_update_reflog` function\n refs: allow multiple reflog entries for the same refname\n refs: add support for migrating reflogs\n\n Documentation/git-refs.txt | 2 -\n refs.c | 168 +++++++++++++++++++++++++++++++++------------\n refs.h | 14 ++++\n refs/files-backend.c | 131 +++++++++++++++++++++++------------\n refs/refs-internal.h | 9 +++\n refs/reftable-backend.c | 53 +++++++++++---\n t/t1460-refs-migrate.sh | 73 ++++++++++++++------\n 7 files changed, 329 insertions(+), 121 deletions(-)\n---\n\nRange-diff versus v3:\n\n1: 7989ca0679 = 1: 34fb6a475e refs: include committer info in `ref_update` struct\n2: 12acd7b4bb = 2: 4badd2b8ec refs: add `index` field to `struct ref_udpate`\n3: 0d13c0d09b = 3: 0d8673c2fe refs/files: add count field to ref_lock\n4: d4073cd9dc = 4: dec4b3c6f6 refs: extract out refname verification in transactions\n5: 5a3d242955 = 5: b1cc1bb242 refs: add `committer_info` to `ref_transaction_add_update()`\n6: fedd93f113 = 6: f669e87498 refs: introduce the `ref_transaction_update_reflog` function\n7: b4465ee0c5 = 7: 53c0d2b62b refs: allow multiple reflog entries for the same refname\n8: 8760610904 ! 8: 1cf30a5c3a refs: add support for migrating reflogs\n @@ Commit message\n the reflogs from the old reference backend. This is to ensure that the\n order is maintained in the new backend.\n \n + Helped-by: Patrick Steinhardt \n Signed-off-by: Karthik Nayak \n \n ## Documentation/git-refs.txt ##\n @@ refs.c: static int migrate_one_ref(const char *refname, const char *referent UNU\n +static int migrate_one_reflog(const char *refname, void *cb_data)\n +{\n +\tstruct migration_data *migration_data = cb_data;\n -+\tstruct reflog_migration_data data;\n -+\n -+\tdata.refname = refname;\n -+\tdata.old_refs = migration_data->old_refs;\n -+\tdata.transaction = migration_data->transaction;\n -+\tdata.errbuf = migration_data->errbuf;\n -+\tdata.sb = &migration_data->sb;\n ++\tstruct reflog_migration_data data = {\n ++\t\t.refname = refname,\n ++\t\t.old_refs = migration_data->old_refs,\n ++\t\t.transaction = migration_data->transaction,\n ++\t\t.errbuf = migration_data->errbuf,\n ++\t\t.sb = &migration_data->sb,\n ++\t};\n +\n +\treturn refs_for_each_reflog_ent(migration_data->old_refs, refname,\n +\t\t\t\t\tmigrate_one_reflog_entry, &data);\n @@ refs.c: static int move_files(const char *from_path, const char *to_path, struct\n {\n \tstruct worktree **worktrees = get_worktrees();\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n + \tstruct ref_store *old_refs = NULL, *new_refs = NULL;\n \tstruct ref_transaction *transaction = NULL;\n \tstruct strbuf new_gitdir = STRBUF_INIT;\n - \tstruct migration_data data;\n +-\tstruct migration_data data;\n -\tsize_t reflog_count = 0;\n ++\tstruct migration_data data = {\n ++\t\t.sb = STRBUF_INIT,\n ++\t};\n \tint did_migrate_refs = 0;\n \tint ret;\n \n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \t */\n \tstrbuf_addf(&new_gitdir, \"%s/%s\", old_refs->gitdir, \"ref_migration.XXXXXX\");\n \tif (!mkdtemp(new_gitdir.buf)) {\n -@@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n - \tdata.old_refs = old_refs;\n - \tdata.transaction = transaction;\n - \tdata.errbuf = errbuf;\n -+\tstrbuf_init(&data.sb, 0);\n - \n - \t/*\n - \t * We need to use the internal `do_for_each_ref()` here so that we can\n @@ refs.c: int repo_migrate_ref_storage_format(struct repository *repo,\n \tif (ret < 0)\n \t\tgoto done;\n\n\n--- \n\nbase-commit: 09245f4b75863f4e94dac7feebaafce53a26965f\nchange-id: 20241111-320-git-refs-migrate-reflogs-a53e3a6cffc9\n\nThanks\n- Karthik\n\n"},{"id":"509173","messageId":"20241216-320-git-refs-migrate-reflogs-v4-1-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 1/8] refs: include committer info in `ref_update` struct","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:26Z","receivedAt":"2024-12-16T16:44:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference backends obtain the committer information from\n`git_committer_info(0)` when adding a reflog. The upcoming patches\nintroduce support for migrating reflogs between the reference backends.\nThis requires an interface to creating reflogs, including custom\ncommitter information.\n\nAdd a new field `committer_info` to the `ref_update` struct, which is\nthen used by the reference backends. If there is no `committer_info`\nprovided, the reference backends default to using\n`git_committer_info(0)`. The field itself cannot be set to\n`git_committer_info(0)` since the values are dynamic and must be\nobtained right when the reflog is being committed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 1 +\n refs/files-backend.c | 24 ++++++++++++++----------\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 12 +++++++++++-\n 4 files changed, 27 insertions(+), 11 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 762f3e324d59c60cd4f05c2f257e54de8deb00e5..f003e51c6bf5229bfbce8ce61ffad7cdba0572e0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1151,6 +1151,7 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tfree(transaction->updates[i]->msg);\n+\t\tfree(transaction->updates[i]->committer_info);\n \t\tfree((char *)transaction->updates[i]->new_target);\n \t\tfree((char *)transaction->updates[i]->old_target);\n \t\tfree(transaction->updates[i]);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 64f51f0da905a9a8a1ac4109c6b0a9a85a355db7..6078668c99ee254e794e3ba49689aa34e6022efd 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1858,6 +1858,9 @@ static int log_ref_write_fd(int fd, const struct object_id *old_oid,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint ret = 0;\n \n+\tif (!committer)\n+\t\tcommitter = git_committer_info(0);\n+\n \tstrbuf_addf(&sb, \"%s %s %s\", oid_to_hex(old_oid), oid_to_hex(new_oid), committer);\n \tif (msg && *msg) {\n \t\tstrbuf_addch(&sb, '\\t');\n@@ -1871,8 +1874,10 @@ static int log_ref_write_fd(int fd, const struct object_id *old_oid,\n }\n \n static int files_log_ref_write(struct files_ref_store *refs,\n-\t\t\t const char *refname, const struct object_id *old_oid,\n-\t\t\t const struct object_id *new_oid, const char *msg,\n+\t\t\t const char *refname,\n+\t\t\t const struct object_id *old_oid,\n+\t\t\t const struct object_id *new_oid,\n+\t\t\t const char *committer_info, const char *msg,\n \t\t\t int flags, struct strbuf *err)\n {\n \tint logfd, result;\n@@ -1889,8 +1894,7 @@ static int files_log_ref_write(struct files_ref_store *refs,\n \n \tif (logfd < 0)\n \t\treturn 0;\n-\tresult = log_ref_write_fd(logfd, old_oid, new_oid,\n-\t\t\t\t git_committer_info(0), msg);\n+\tresult = log_ref_write_fd(logfd, old_oid, new_oid, committer_info, msg);\n \tif (result) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \t\tint save_errno = errno;\n@@ -1974,8 +1978,7 @@ static int commit_ref_update(struct files_ref_store *refs,\n \tfiles_assert_main_repository(refs, \"commit_ref_update\");\n \n \tclear_loose_ref_cache(refs);\n-\tif (files_log_ref_write(refs, lock->ref_name,\n-\t\t\t\t&lock->old_oid, oid,\n+\tif (files_log_ref_write(refs, lock->ref_name, &lock->old_oid, oid, NULL,\n \t\t\t\tlogmsg, flags, err)) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \t\tstrbuf_addf(err, \"cannot update the ref '%s': %s\",\n@@ -2007,9 +2010,9 @@ static int commit_ref_update(struct files_ref_store *refs,\n \t\tif (head_ref && (head_flag & REF_ISSYMREF) &&\n \t\t !strcmp(head_ref, lock->ref_name)) {\n \t\t\tstruct strbuf log_err = STRBUF_INIT;\n-\t\t\tif (files_log_ref_write(refs, \"HEAD\",\n-\t\t\t\t\t\t&lock->old_oid, oid,\n-\t\t\t\t\t\tlogmsg, flags, &log_err)) {\n+\t\t\tif (files_log_ref_write(refs, \"HEAD\", &lock->old_oid,\n+\t\t\t\t\t\toid, NULL, logmsg, flags,\n+\t\t\t\t\t\t&log_err)) {\n \t\t\t\terror(\"%s\", log_err.buf);\n \t\t\t\tstrbuf_release(&log_err);\n \t\t\t}\n@@ -2969,7 +2972,8 @@ static int parse_and_write_reflog(struct files_ref_store *refs,\n \t}\n \n \tif (files_log_ref_write(refs, lock->ref_name, &lock->old_oid,\n-\t\t\t\t&update->new_oid, update->msg, update->flags, err)) {\n+\t\t\t\t&update->new_oid, update->committer_info,\n+\t\t\t\tupdate->msg, update->flags, err)) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \n \t\tstrbuf_addf(err, \"cannot update the ref '%s': %s\",\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 58aa56d1b27c85d606ed7c8c0d908e4b87d1066b..0fd95cdacd99e4a728c22f5286f6b3f0f360c110 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -113,6 +113,7 @@ struct ref_update {\n \tvoid *backend_data;\n \tunsigned int type;\n \tchar *msg;\n+\tchar *committer_info;\n \n \t/*\n \t * If this ref_update was split off of a symref update via\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 647ef9b05b1dc9a376ed054330b487f7595c5caa..e882602487c66261d586a94101bb1b4e9a2ed60e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1379,11 +1379,21 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t}\n \n \t\t\tif (create_reflog) {\n+\t\t\t\tstruct ident_split c;\n+\n \t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n \t\t\t\tlog = &logs[logs_nr++];\n \t\t\t\tmemset(log, 0, sizeof(*log));\n \n-\t\t\t\tfill_reftable_log_record(log, &committer_ident);\n+\t\t\t\tif (u->committer_info) {\n+\t\t\t\t\tif (split_ident_line(&c, u->committer_info,\n+\t\t\t\t\t\t\t strlen(u->committer_info)))\n+\t\t\t\t\t\tBUG(\"failed splitting committer info\");\n+\t\t\t\t} else {\n+\t\t\t\t\tc = committer_ident;\n+\t\t\t\t}\n+\n+\t\t\t\tfill_reftable_log_record(log, &c);\n \t\t\t\tlog->update_index = ts;\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n\n-- \n2.47.1\n\n"},{"id":"509174","messageId":"20241216-320-git-refs-migrate-reflogs-v4-2-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 2/8] refs: add `index` field to `struct ref_udpate`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:27Z","receivedAt":"2024-12-16T16:44:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reftable backend, sorts its updates by refname before applying them,\nthis ensures that the references are stored sorted. When migrating\nreflogs from one backend to another, the order of the reflogs must be\nmaintained. Add a new `index` field to the `ref_update` struct to\nfacilitate this.\n\nThis field is used in the reftable backend's sort comparison function\n`transaction_update_cmp`, to ensure that indexed fields maintain their\norder.\n\nSigned-off-by: Karthik Nayak \n---\n refs/refs-internal.h | 7 +++++++\n refs/reftable-backend.c | 13 +++++++++++--\n 2 files changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 0fd95cdacd99e4a728c22f5286f6b3f0f360c110..f5c733d099f0c6f1076a25f4f77d9d5eb345ec87 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -115,6 +115,13 @@ struct ref_update {\n \tchar *msg;\n \tchar *committer_info;\n \n+\t/*\n+\t * The index overrides the default sort algorithm. This is needed\n+\t * when migrating reflogs and we want to ensure we carry over the\n+\t * same order.\n+\t */\n+\tunsigned int index;\n+\n \t/*\n \t * If this ref_update was split off of a symref update via\n \t * split_symref_update(), then this member points at that\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex e882602487c66261d586a94101bb1b4e9a2ed60e..c008f20be719fec3af6a8f81c821cb9c263764d7 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1279,8 +1279,17 @@ static int reftable_be_transaction_abort(struct ref_store *ref_store UNUSED,\n \n static int transaction_update_cmp(const void *a, const void *b)\n {\n-\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n-\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+\tstruct reftable_transaction_update *update_a = (struct reftable_transaction_update *)a;\n+\tstruct reftable_transaction_update *update_b = (struct reftable_transaction_update *)b;\n+\n+\t/*\n+\t * If there is an index set, it should take preference (default is 0).\n+\t * This ensures that updates with indexes are sorted amongst themselves.\n+\t */\n+\tif (update_a->update->index || update_b->update->index)\n+\t\treturn update_a->update->index - update_b->update->index;\n+\n+\treturn strcmp(update_a->update->refname, update_b->update->refname);\n }\n \n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n\n-- \n2.47.1\n\n"},{"id":"509175","messageId":"20241216-320-git-refs-migrate-reflogs-v4-3-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 3/8] refs/files: add count field to ref_lock","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:28Z","receivedAt":"2024-12-16T16:44:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When refs are updated in the files-backend, a lock is obtained for the\ncorresponding file path. This is the case even for reflogs, i.e. a lock\nis obtained on the reference path instead of the reflog path. This\nworks, since generally, reflogs are updated alongside the ref.\n\nThe upcoming patches will add support for reflog updates in ref\ntransaction. This means, in a particular transaction we want to have ref\nupdates and reflog updates. For a given ref in a given transaction there\ncan be at most one update. But we can theoretically have multiple reflog\nupdates for a given ref in a given transaction. A great example of this\nwould be when migrating reflogs from one backend to another. There we\nwould batch all the reflog updates for a given reference in a single\ntransaction.\n\nThe current flow does not support this, because currently refs & reflogs\nare treated as a single entity and capture the lock together. To\nseparate this, add a count field to ref_lock. With this, multiple\nupdates can hold onto a single ref_lock and the lock will only be\nreleased when all of them release the lock.\n\nThis patch only adds the `count` field to `ref_lock` and adds the logic\nto increment and decrement the lock. In a follow up commit, we'll\nseparate the reflog update logic from ref updates and utilize this\nfunctionality.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 58 +++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 39 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6078668c99ee254e794e3ba49689aa34e6022efd..02cb4907d8659e87a227fed4f60a5f6606be8764 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -71,6 +71,7 @@ struct ref_lock {\n \tchar *ref_name;\n \tstruct lock_file lk;\n \tstruct object_id old_oid;\n+\tunsigned int count; /* track users of the lock (ref update + reflog updates) */\n };\n \n struct files_ref_store {\n@@ -638,9 +639,12 @@ int parse_loose_ref_contents(const struct git_hash_algo *algop,\n \n static void unlock_ref(struct ref_lock *lock)\n {\n-\trollback_lock_file(&lock->lk);\n-\tfree(lock->ref_name);\n-\tfree(lock);\n+\tlock->count--;\n+\tif (!lock->count) {\n+\t\trollback_lock_file(&lock->lk);\n+\t\tfree(lock->ref_name);\n+\t\tfree(lock);\n+\t}\n }\n \n /*\n@@ -696,6 +700,7 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t*lock_p = CALLOC_ARRAY(lock, 1);\n \n \tlock->ref_name = xstrdup(refname);\n+\tlock->count = 1;\n \tfiles_ref_path(refs, &ref_file, refname);\n \n retry:\n@@ -1169,6 +1174,7 @@ static struct ref_lock *lock_ref_oid_basic(struct files_ref_store *refs,\n \t\tgoto error_return;\n \n \tlock->ref_name = xstrdup(refname);\n+\tlock->count = 1;\n \n \tif (raceproof_create_file(ref_file.buf, create_reflock, &lock->lk)) {\n \t\tunable_to_lock_message(ref_file.buf, errno, err);\n@@ -2535,6 +2541,12 @@ static int check_old_oid(struct ref_update *update, struct object_id *oid,\n \treturn -1;\n }\n \n+struct files_transaction_backend_data {\n+\tstruct ref_transaction *packed_transaction;\n+\tint packed_refs_locked;\n+\tstruct strmap ref_locks;\n+};\n+\n /*\n * Prepare for carrying out update:\n * - Lock the reference referred to by update.\n@@ -2557,11 +2569,14 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n {\n \tstruct strbuf referent = STRBUF_INIT;\n \tint mustexist = ref_update_expects_existing_old_ref(update);\n+\tstruct files_transaction_backend_data *backend_data;\n \tint ret = 0;\n \tstruct ref_lock *lock;\n \n \tfiles_assert_main_repository(refs, \"lock_ref_for_update\");\n \n+\tbackend_data = transaction->backend_data;\n+\n \tif ((update->flags & REF_HAVE_NEW) && ref_update_has_null_new_value(update))\n \t\tupdate->flags |= REF_DELETING;\n \n@@ -2572,18 +2587,25 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\tgoto out;\n \t}\n \n-\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t affected_refnames,\n-\t\t\t &lock, &referent,\n-\t\t\t &update->type, err);\n-\tif (ret) {\n-\t\tchar *reason;\n+\tlock = strmap_get(&backend_data->ref_locks, update->refname);\n+\tif (lock) {\n+\t\tlock->count++;\n+\t} else {\n+\t\tret = lock_raw_ref(refs, update->refname, mustexist,\n+\t\t\t\t affected_refnames,\n+\t\t\t\t &lock, &referent,\n+\t\t\t\t &update->type, err);\n+\t\tif (ret) {\n+\t\t\tchar *reason;\n+\n+\t\t\treason = strbuf_detach(err, NULL);\n+\t\t\tstrbuf_addf(err, \"cannot lock ref '%s': %s\",\n+\t\t\t\t ref_update_original_update_refname(update), reason);\n+\t\t\tfree(reason);\n+\t\t\tgoto out;\n+\t\t}\n \n-\t\treason = strbuf_detach(err, NULL);\n-\t\tstrbuf_addf(err, \"cannot lock ref '%s': %s\",\n-\t\t\t ref_update_original_update_refname(update), reason);\n-\t\tfree(reason);\n-\t\tgoto out;\n+\t\tstrmap_put(&backend_data->ref_locks, update->refname, lock);\n \t}\n \n \tupdate->backend_data = lock;\n@@ -2730,11 +2752,6 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \treturn ret;\n }\n \n-struct files_transaction_backend_data {\n-\tstruct ref_transaction *packed_transaction;\n-\tint packed_refs_locked;\n-};\n-\n /*\n * Unlock any references in `transaction` that are still locked, and\n * mark the transaction closed.\n@@ -2767,6 +2784,8 @@ static void files_transaction_cleanup(struct files_ref_store *refs,\n \t\tif (backend_data->packed_refs_locked)\n \t\t\tpacked_refs_unlock(refs->packed_ref_store);\n \n+\t\tstrmap_clear(&backend_data->ref_locks, 0);\n+\n \t\tfree(backend_data);\n \t}\n \n@@ -2796,6 +2815,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto cleanup;\n \n \tCALLOC_ARRAY(backend_data, 1);\n+\tstrmap_init(&backend_data->ref_locks);\n \ttransaction->backend_data = backend_data;\n \n \t/*\n\n-- \n2.47.1\n\n"},{"id":"509176","messageId":"20241216-320-git-refs-migrate-reflogs-v4-4-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 4/8] refs: extract out refname verification in transactions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:29Z","receivedAt":"2024-12-16T16:44:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Unless the `REF_SKIP_REFNAME_VERIFICATION` flag is set for an update,\nthe refname of the update is verified for:\n\n - Ensuring it is not a pseudoref.\n - Checking the refname format.\n\nThese checks will also be needed in a following commit where the\nfunction to add reflog updates to the transaction is introduced. Extract\nthe code out into a new static function.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 37 +++++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f003e51c6bf5229bfbce8ce61ffad7cdba0572e0..9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1196,6 +1196,28 @@ struct ref_update *ref_transaction_add_update(\n \treturn update;\n }\n \n+static int transaction_refname_valid(const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t unsigned int flags, struct strbuf *err)\n+{\n+\tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n+\t\treturn 1;\n+\n+\tif (is_pseudo_ref(refname)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n+\t\t\t refname);\n+\t\treturn 0;\n+\t} else if ((new_oid && !is_null_oid(new_oid)) ?\n+\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n+\t\t !refname_is_safe(refname)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n+\t\t\t refname);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const struct object_id *new_oid,\n@@ -1213,21 +1235,8 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\treturn -1;\n \t}\n \n-\tif (!(flags & REF_SKIP_REFNAME_VERIFICATION) &&\n-\t ((new_oid && !is_null_oid(new_oid)) ?\n-\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n-\t\t\t !refname_is_safe(refname))) {\n-\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n-\t\t\t refname);\n+\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n \t\treturn -1;\n-\t}\n-\n-\tif (!(flags & REF_SKIP_REFNAME_VERIFICATION) &&\n-\t is_pseudo_ref(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n-\t\t\t refname);\n-\t\treturn -1;\n-\t}\n \n \tif (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS)\n \t\tBUG(\"illegal flags 0x%x passed to ref_transaction_update()\", flags);\n\n-- \n2.47.1\n\n"},{"id":"509177","messageId":"20241216-320-git-refs-migrate-reflogs-v4-5-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:30Z","receivedAt":"2024-12-16T16:44:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_transaction_add_update()` creates the `ref_update` struct. To\nfacilitate addition of reflogs in the next commit, the function needs to\naccommodate setting the `committer_info` field in the struct. So modify\nthe function to also take `committer_info` as an argument and set it\naccordingly.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++--\n refs/files-backend.c | 14 ++++++++------\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 6 ++++--\n 4 files changed, 18 insertions(+), 10 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801..782bf1090af65196263a3c35ed18d878bb4f2967 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1166,6 +1166,7 @@ struct ref_update *ref_transaction_add_update(\n \t\tconst struct object_id *new_oid,\n \t\tconst struct object_id *old_oid,\n \t\tconst char *new_target, const char *old_target,\n+\t\tconst char *committer_info,\n \t\tconst char *msg)\n {\n \tstruct ref_update *update;\n@@ -1190,8 +1191,10 @@ struct ref_update *ref_transaction_add_update(\n \t\toidcpy(&update->new_oid, new_oid);\n \tif ((flags & REF_HAVE_OLD) && old_oid)\n \t\toidcpy(&update->old_oid, old_oid);\n-\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n+\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n+\t\tupdate->committer_info = xstrdup_or_null(committer_info);\n \t\tupdate->msg = normalize_reflog_message(msg);\n+\t}\n \n \treturn update;\n }\n@@ -1253,7 +1256,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n-\t\t\t\t old_target, msg);\n+\t\t\t\t old_target, NULL, msg);\n \treturn 0;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 02cb4907d8659e87a227fed4f60a5f6606be8764..255fed8354cae982f785b1b85340e2a1eeecf2a6 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1270,7 +1270,7 @@ static void prune_ref(struct files_ref_store *refs, struct ref_to_prune *r)\n \tref_transaction_add_update(\n \t\t\ttransaction, r->name,\n \t\t\tREF_NO_DEREF | REF_HAVE_NEW | REF_HAVE_OLD | REF_IS_PRUNING,\n-\t\t\tnull_oid(), &r->oid, NULL, NULL, NULL);\n+\t\t\tnull_oid(), &r->oid, NULL, NULL, NULL, NULL);\n \tif (ref_transaction_commit(transaction, &err))\n \t\tgoto cleanup;\n \n@@ -2417,7 +2417,7 @@ static int split_head_update(struct ref_update *update,\n \t\t\ttransaction, \"HEAD\",\n \t\t\tupdate->flags | REF_LOG_ONLY | REF_NO_DEREF,\n \t\t\t&update->new_oid, &update->old_oid,\n-\t\t\tNULL, NULL, update->msg);\n+\t\t\tNULL, NULL, update->committer_info, update->msg);\n \n \t/*\n \t * Add \"HEAD\". This insertion is O(N) in the transaction\n@@ -2481,7 +2481,8 @@ static int split_symref_update(struct ref_update *update,\n \t\t\ttransaction, referent, new_flags,\n \t\t\tupdate->new_target ? NULL : &update->new_oid,\n \t\t\tupdate->old_target ? NULL : &update->old_oid,\n-\t\t\tupdate->new_target, update->old_target, update->msg);\n+\t\t\tupdate->new_target, update->old_target, NULL,\n+\t\t\tupdate->msg);\n \n \tnew_update->parent_update = update;\n \n@@ -2914,7 +2915,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t\tpacked_transaction, update->refname,\n \t\t\t\t\tREF_HAVE_NEW | REF_NO_DEREF,\n \t\t\t\t\t&update->new_oid, NULL,\n-\t\t\t\t\tNULL, NULL, NULL);\n+\t\t\t\t\tNULL, NULL, NULL, NULL);\n \t\t}\n \t}\n \n@@ -3094,12 +3095,13 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t\tref_transaction_add_update(loose_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n \t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n-\t\t\t\t\t\t update->new_target, NULL, NULL);\n+\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n+\t\t\t\t\t\t NULL);\n \t\t} else {\n \t\t\tref_transaction_add_update(packed_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n \t\t\t\t\t\t &update->new_oid, &update->old_oid,\n-\t\t\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t\t\t NULL, NULL, update->committer_info, NULL);\n \t\t}\n \t}\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f5c733d099f0c6f1076a25f4f77d9d5eb345ec87..79b287c5ec5c7d8f759869cf93cda405640186dc 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -162,6 +162,7 @@ struct ref_update *ref_transaction_add_update(\n \t\tconst struct object_id *new_oid,\n \t\tconst struct object_id *old_oid,\n \t\tconst char *new_target, const char *old_target,\n+\t\tconst char *committer_info,\n \t\tconst char *msg);\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex c008f20be719fec3af6a8f81c821cb9c263764d7..b2e3ba877de9e59fea5a4d066eb13e60ef22a32b 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1078,7 +1078,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tnew_update = ref_transaction_add_update(\n \t\t\t\t\ttransaction, \"HEAD\",\n \t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n-\t\t\t\t\t&u->new_oid, &u->old_oid, NULL, NULL, u->msg);\n+\t\t\t\t\t&u->new_oid, &u->old_oid, NULL, NULL, NULL,\n+\t\t\t\t\tu->msg);\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n@@ -1161,7 +1162,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t\ttransaction, referent.buf, new_flags,\n \t\t\t\t\tu->new_target ? NULL : &u->new_oid,\n \t\t\t\t\tu->old_target ? NULL : &u->old_oid,\n-\t\t\t\t\tu->new_target, u->old_target, u->msg);\n+\t\t\t\t\tu->new_target, u->old_target,\n+\t\t\t\t\tu->committer_info, u->msg);\n \n \t\t\t\tnew_update->parent_update = u;\n \n\n-- \n2.47.1\n\n"},{"id":"509179","messageId":"20241216-320-git-refs-migrate-reflogs-v4-6-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:31Z","receivedAt":"2024-12-16T16:44:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new function `ref_transaction_update_reflog`, for clients to\nadd a reflog update to a transaction. While the existing function\n`ref_transaction_update` also allows clients to add a reflog entry, this\nfunction does a few things more, It:\n - Enforces that only a reflog entry is added and does not update the\n ref itself.\n - Allows the users to also provide the committer information. This\n means clients can add reflog entries with custom committer\n information.\n\nThe `transaction_refname_valid()` function also modifies the error\nmessage selectively based on the type of the update. This change also\naffects reflog updates which go through `ref_transaction_update()`.\n\nA follow up commit will utilize this function to add reflog support to\n`git refs migrate`.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 39 +++++++++++++++++++++++++++++++++++----\n refs.h | 14 ++++++++++++++\n refs/files-backend.c | 24 ++++++++++++++++--------\n 3 files changed, 65 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 782bf1090af65196263a3c35ed18d878bb4f2967..8b3882cff17e5e3b0376f75654e32f81a23e5cb2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1207,14 +1207,14 @@ static int transaction_refname_valid(const char *refname,\n \t\treturn 1;\n \n \tif (is_pseudo_ref(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n-\t\t\t refname);\n+\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n+\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t} else if ((new_oid && !is_null_oid(new_oid)) ?\n \t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n \t\t !refname_is_safe(refname)) {\n-\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n-\t\t\t refname);\n+\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog with bad name\" : \"ref with bad name\";\n+\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n \t\treturn 0;\n \t}\n \n@@ -1257,6 +1257,37 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \tref_transaction_add_update(transaction, refname, flags,\n \t\t\t\t new_oid, old_oid, new_target,\n \t\t\t\t old_target, NULL, msg);\n+\n+\treturn 0;\n+}\n+\n+int ref_transaction_update_reflog(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t const char *committer_info, unsigned int flags,\n+\t\t\t\t const char *msg, unsigned int index,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct ref_update *update;\n+\n+\tassert(err);\n+\n+\tflags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\n+\tif (!transaction_refname_valid(refname, new_oid, flags, err))\n+\t\treturn -1;\n+\n+\tupdate = ref_transaction_add_update(transaction, refname, flags,\n+\t\t\t\t\t new_oid, old_oid, NULL, NULL,\n+\t\t\t\t\t committer_info, msg);\n+\t/*\n+\t * While we do set the old_oid value, we unset the flag to skip\n+\t * old_oid verification which only makes sense for refs.\n+\t */\n+\tupdate->flags &= ~REF_HAVE_OLD;\n+\tupdate->index = index;\n+\n \treturn 0;\n }\n \ndiff --git a/refs.h b/refs.h\nindex a5bedf48cf6de91005a7e8d0bf58ca98350397a6..b0dfc65ed2e59c4b66967840339f81e7746a96d3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -727,6 +727,20 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t unsigned int flags, const char *msg,\n \t\t\t struct strbuf *err);\n \n+/*\n+ * Similar to`ref_transaction_update`, but this function is only for adding\n+ * a reflog update. Supports providing custom committer information. The index\n+ * field can be utiltized to order updates as desired. When not used, the\n+ * updates default to being ordered by refname.\n+ */\n+int ref_transaction_update_reflog(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t const char *committer_info, unsigned int flags,\n+\t\t\t\t const char *msg, unsigned int index,\n+\t\t\t\t struct strbuf *err);\n+\n /*\n * Add a reference creation to transaction. new_oid is the value that\n * the reference should have after the update; it must not be\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 255fed8354cae982f785b1b85340e2a1eeecf2a6..c11213f52065bcf2fa7612df8f9500692ee2d02c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3080,10 +3080,12 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * packed-refs don't support symbolic refs and root refs, so we\n-\t\t * have to queue these references via the loose transaction.\n+\t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n+\t\t * so we have to queue these references via the loose transaction.\n \t\t */\n-\t\tif (update->new_target || is_root_ref(update->refname)) {\n+\t\tif (update->new_target ||\n+\t\t is_root_ref(update->refname) ||\n+\t\t (update->flags & REF_LOG_ONLY)) {\n \t\t\tif (!loose_transaction) {\n \t\t\t\tloose_transaction = ref_store_transaction_begin(&refs->base, 0, err);\n \t\t\t\tif (!loose_transaction) {\n@@ -3092,11 +3094,17 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t\t\t}\n \t\t\t}\n \n-\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n-\t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n-\t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n-\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n-\t\t\t\t\t\t NULL);\n+\t\t\tif (update->flags & REF_LOG_ONLY)\n+\t\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n+\t\t\t\t\t\t\t update->flags, &update->new_oid,\n+\t\t\t\t\t\t\t &update->old_oid, NULL, NULL,\n+\t\t\t\t\t\t\t update->committer_info, update->msg);\n+\t\t\telse\n+\t\t\t\tref_transaction_add_update(loose_transaction, update->refname,\n+\t\t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n+\t\t\t\t\t\t\t update->new_target ? NULL : &update->new_oid, NULL,\n+\t\t\t\t\t\t\t update->new_target, NULL, update->committer_info,\n+\t\t\t\t\t\t\t NULL);\n \t\t} else {\n \t\t\tref_transaction_add_update(packed_transaction, update->refname,\n \t\t\t\t\t\t update->flags & ~REF_HAVE_OLD,\n\n-- \n2.47.1\n\n"},{"id":"509178","messageId":"20241216-320-git-refs-migrate-reflogs-v4-7-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 7/8] refs: allow multiple reflog entries for the same refname","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:32Z","receivedAt":"2024-12-16T16:44:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference transaction only allows a single update for a given\nreference to avoid conflicts. This, however, isn't an issue for reflogs.\nThere are no conflicts to be resolved in reflogs and when migrating\nreflogs between backends we'd have multiple reflog entries for the same\nrefname.\n\nSo allow multiple reflog updates within a single transaction. Also the\nreflog creation logic isn't exposed to the end user. While this might\nchange in the future, currently, this reduces the scope of issues to\nthink about.\n\nIn the reftable backend, the writer sorts all updates based on the\nupdate_index before writing to the block. When there are multiple\nreflogs for a given refname, it is essential that the order of the\nreflogs is maintained. So add the `index` value to the `update_index`.\nThe `index` field is only set when multiple reflog entries for a given\nrefname are added and as such in most scenarios the old behavior\nremains.\n\nThis is required to add reflog migration support to `git refs migrate`.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 15 +++++++++++----\n refs/reftable-backend.c | 22 +++++++++++++++++++---\n 2 files changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex c11213f52065bcf2fa7612df8f9500692ee2d02c..8953d1c6d37b13b0db701888b3db92fd87a68aaa 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2611,6 +2611,9 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \n \tupdate->backend_data = lock;\n \n+\tif (update->flags & REF_LOG_ONLY)\n+\t\tgoto out;\n+\n \tif (update->type & REF_ISSYMREF) {\n \t\tif (update->flags & REF_NO_DEREF) {\n \t\t\t/*\n@@ -2829,13 +2832,16 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t */\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *update = transaction->updates[i];\n-\t\tstruct string_list_item *item =\n-\t\t\tstring_list_append(&affected_refnames, update->refname);\n+\t\tstruct string_list_item *item;\n \n \t\tif ((update->flags & REF_IS_PRUNING) &&\n \t\t !(update->flags & REF_NO_DEREF))\n \t\t\tBUG(\"REF_IS_PRUNING set without REF_NO_DEREF\");\n \n+\t\tif (update->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\titem = string_list_append(&affected_refnames, update->refname);\n \t\t/*\n \t\t * We store a pointer to update in item->util, but at\n \t\t * the moment we never use the value of this field\n@@ -3035,8 +3041,9 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \n \t/* Fail if a refname appears more than once in the transaction: */\n \tfor (i = 0; i < transaction->nr; i++)\n-\t\tstring_list_append(&affected_refnames,\n-\t\t\t\t transaction->updates[i]->refname);\n+\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n+\t\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t\t transaction->updates[i]->refname);\n \tstring_list_sort(&affected_refnames);\n \tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -990,8 +990,9 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tif (ret)\n \t\t\tgoto done;\n \n-\t\tstring_list_append(&affected_refnames,\n-\t\t\t\t transaction->updates[i]->refname);\n+\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n+\t\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t\t transaction->updates[i]->refname);\n \t}\n \n \t/*\n@@ -1301,6 +1302,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tuint64_t max_update_index = ts;\n \tconst char *committer_info;\n \tint ret = 0;\n \n@@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\t\t}\n \n \t\t\t\tfill_reftable_log_record(log, &c);\n-\t\t\t\tlog->update_index = ts;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Updates are sorted by the writer. So updates for the same\n+\t\t\t\t * refname need to contain different update indices.\n+\t\t\t\t */\n+\t\t\t\tlog->update_index = ts + u->index;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Note the max update_index so the limit can be set later on.\n+\t\t\t\t */\n+\t\t\t\tif (log->update_index > max_update_index)\n+\t\t\t\t\tmax_update_index = log->update_index;\n+\n \t\t\t\tlog->refname = xstrdup(u->refname);\n \t\t\t\tmemcpy(log->value.update.new_hash,\n \t\t\t\t u->new_oid.hash, GIT_MAX_RAWSZ);\n@@ -1469,6 +1483,8 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t * and log blocks.\n \t */\n \tif (logs) {\n+\t\treftable_writer_set_limits(writer, ts, max_update_index);\n+\n \t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n\n-- \n2.47.1\n\n"},{"id":"509180","messageId":"20241216-320-git-refs-migrate-reflogs-v4-8-d7cd3f197453@gmail.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"[PATCH v4 8/8] refs: add support for migrating reflogs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T16:44:33Z","receivedAt":"2024-12-16T16:44:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs migrate` command was introduced in\n25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n2024-06-06) to support migrating from one reference backend to another.\n\nOne limitation of the command was that it didn't support migrating\nrepositories which contained reflogs. A previous commit, added support\nfor adding reflog updates in ref transactions. Using the added\nfunctionality bake in reflog support for `git refs migrate`.\n\nTo ensure that the order of the reflogs is maintained during the\nmigration, we add the index for each reflog update as we iterate over\nthe reflogs from the old reference backend. This is to ensure that the\norder is maintained in the new backend.\n\nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n Documentation/git-refs.txt | 2 -\n refs.c | 92 ++++++++++++++++++++++++++++++++--------------\n t/t1460-refs-migrate.sh | 73 +++++++++++++++++++++++++-----------\n 3 files changed, 115 insertions(+), 52 deletions(-)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex ce31f93061db5e5d16aca516dd3d15f6527db870..9829984b0a4c4f54ec7f9b6c6c7072f62b1d198d 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -57,8 +57,6 @@ KNOWN LIMITATIONS\n \n The ref format migration has several known limitations in its current form:\n \n-* It is not possible to migrate repositories that have reflogs.\n-\n * It is not possible to migrate repositories that have worktrees.\n \n * There is no way to block concurrent writes to the repository during an\ndiff --git a/refs.c b/refs.c\nindex 8b3882cff17e5e3b0376f75654e32f81a23e5cb2..5d541ddc41aa84905e688c92565aa18fbf55323b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -30,6 +30,7 @@\n #include \"date.h\"\n #include \"commit.h\"\n #include \"wildmatch.h\"\n+#include \"ident.h\"\n \n /*\n * List of all available backends\n@@ -2673,6 +2674,7 @@ struct migration_data {\n \tstruct ref_store *old_refs;\n \tstruct ref_transaction *transaction;\n \tstruct strbuf *errbuf;\n+\tstruct strbuf sb;\n };\n \n static int migrate_one_ref(const char *refname, const char *referent UNUSED, const struct object_id *oid,\n@@ -2705,6 +2707,52 @@ static int migrate_one_ref(const char *refname, const char *referent UNUSED, con\n \treturn ret;\n }\n \n+struct reflog_migration_data {\n+\tunsigned int index;\n+\tconst char *refname;\n+\tstruct ref_store *old_refs;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf *errbuf;\n+\tstruct strbuf *sb;\n+};\n+\n+static int migrate_one_reflog_entry(struct object_id *old_oid,\n+\t\t\t\t struct object_id *new_oid,\n+\t\t\t\t const char *committer,\n+\t\t\t\t timestamp_t timestamp, int tz,\n+\t\t\t\t const char *msg, void *cb_data)\n+{\n+\tstruct reflog_migration_data *data = cb_data;\n+\tconst char *date;\n+\tint ret;\n+\n+\tdate = show_date(timestamp, tz, DATE_MODE(NORMAL));\n+\tstrbuf_reset(data->sb);\n+\t/* committer contains name and email */\n+\tstrbuf_addstr(data->sb, fmt_ident(\"\", committer, WANT_BLANK_IDENT, date, 0));\n+\n+\tret = ref_transaction_update_reflog(data->transaction, data->refname,\n+\t\t\t\t\t new_oid, old_oid, data->sb->buf,\n+\t\t\t\t\t REF_HAVE_NEW | REF_HAVE_OLD, msg,\n+\t\t\t\t\t data->index++, data->errbuf);\n+\treturn ret;\n+}\n+\n+static int migrate_one_reflog(const char *refname, void *cb_data)\n+{\n+\tstruct migration_data *migration_data = cb_data;\n+\tstruct reflog_migration_data data = {\n+\t\t.refname = refname,\n+\t\t.old_refs = migration_data->old_refs,\n+\t\t.transaction = migration_data->transaction,\n+\t\t.errbuf = migration_data->errbuf,\n+\t\t.sb = &migration_data->sb,\n+\t};\n+\n+\treturn refs_for_each_reflog_ent(migration_data->old_refs, refname,\n+\t\t\t\t\tmigrate_one_reflog_entry, &data);\n+}\n+\n static int move_files(const char *from_path, const char *to_path, struct strbuf *errbuf)\n {\n \tstruct strbuf from_buf = STRBUF_INIT, to_buf = STRBUF_INIT;\n@@ -2771,13 +2819,6 @@ static int move_files(const char *from_path, const char *to_path, struct strbuf\n \treturn ret;\n }\n \n-static int count_reflogs(const char *reflog UNUSED, void *payload)\n-{\n-\tsize_t *reflog_count = payload;\n-\t(*reflog_count)++;\n-\treturn 0;\n-}\n-\n static int has_worktrees(void)\n {\n \tstruct worktree **worktrees = get_worktrees();\n@@ -2802,8 +2843,9 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tstruct ref_store *old_refs = NULL, *new_refs = NULL;\n \tstruct ref_transaction *transaction = NULL;\n \tstruct strbuf new_gitdir = STRBUF_INIT;\n-\tstruct migration_data data;\n-\tsize_t reflog_count = 0;\n+\tstruct migration_data data = {\n+\t\t.sb = STRBUF_INIT,\n+\t};\n \tint did_migrate_refs = 0;\n \tint ret;\n \n@@ -2815,21 +2857,6 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \n \told_refs = get_main_ref_store(repo);\n \n-\t/*\n-\t * We do not have any interfaces that would allow us to write many\n-\t * reflog entries. Once we have them we can remove this restriction.\n-\t */\n-\tif (refs_for_each_reflog(old_refs, count_reflogs, &reflog_count) < 0) {\n-\t\tstrbuf_addstr(errbuf, \"cannot count reflogs\");\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tif (reflog_count) {\n-\t\tstrbuf_addstr(errbuf, \"migrating reflogs is not supported yet\");\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\n \t/*\n \t * Worktrees complicate the migration because every worktree has a\n \t * separate ref storage. While it should be feasible to implement, this\n@@ -2855,17 +2882,21 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t * This operation is safe as we do not yet modify the main\n \t * repository.\n \t *\n-\t * 3. If we're in dry-run mode then we are done and can hand over the\n+\t * 3. Enumerate all reflogs and write them into the new ref storage.\n+\t * This operation is safe as we do not yet modify the main\n+\t * repository.\n+\t *\n+\t * 4. If we're in dry-run mode then we are done and can hand over the\n \t * directory to the caller for inspection. If not, we now start\n \t * with the destructive part.\n \t *\n-\t * 4. Delete the old ref storage from disk. As we have a copy of refs\n+\t * 5. Delete the old ref storage from disk. As we have a copy of refs\n \t * in the new ref storage it's okay(ish) if we now get interrupted\n \t * as there is an equivalent copy of all refs available.\n \t *\n-\t * 5. Move the new ref storage files into place.\n+\t * 6. Move the new ref storage files into place.\n \t *\n-\t * 6. Change the repository format to the new ref format.\n+\t * 7. Change the repository format to the new ref format.\n \t */\n \tstrbuf_addf(&new_gitdir, \"%s/%s\", old_refs->gitdir, \"ref_migration.XXXXXX\");\n \tif (!mkdtemp(new_gitdir.buf)) {\n@@ -2907,6 +2938,10 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \tif (ret < 0)\n \t\tgoto done;\n \n+\tret = refs_for_each_reflog(old_refs, migrate_one_reflog, &data);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = ref_transaction_commit(transaction, errbuf);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2982,6 +3017,7 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t}\n \tref_transaction_free(transaction);\n \tstrbuf_release(&new_gitdir);\n+\tstrbuf_release(&data.sb);\n \treturn ret;\n }\n \ndiff --git a/t/t1460-refs-migrate.sh b/t/t1460-refs-migrate.sh\nindex 1bfff3a7afd5acc470424dfe7ec3e97d45f5c481..f59bc4860f19c4af82dc6f2984bdb69d61fe3ec2 100755\n--- a/t/t1460-refs-migrate.sh\n+++ b/t/t1460-refs-migrate.sh\n@@ -7,23 +7,44 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+# Migrate the provided repository from one format to the other and\n+# verify that the references and logs are migrated over correctly.\n+# Usage: test_migration \n+# is the relative path to the repo to be migrated.\n+# is the ref format to be migrated to.\n+# (true or false) whether to skip reflog verification.\n test_migration () {\n-\tgit -C \"$1\" for-each-ref --include-root-refs \\\n+\trepo=$1 &&\n+\tformat=$2 &&\n+\tskip_reflog_verify=${3:-false} &&\n+\tgit -C \"$repo\" for-each-ref --include-root-refs \\\n \t\t--format='%(refname) %(objectname) %(symref)' >expect &&\n-\tgit -C \"$1\" refs migrate --ref-format=\"$2\" &&\n-\tgit -C \"$1\" for-each-ref --include-root-refs \\\n+\tif ! $skip_reflog_verify\n+\tthen\n+\t git -C \"$repo\" reflog --all >expect_logs &&\n+\t git -C \"$repo\" reflog list >expect_log_list\n+\tfi &&\n+\n+\tgit -C \"$repo\" refs migrate --ref-format=\"$2\" &&\n+\n+\tgit -C \"$repo\" for-each-ref --include-root-refs \\\n \t\t--format='%(refname) %(objectname) %(symref)' >actual &&\n \ttest_cmp expect actual &&\n+\tif ! $skip_reflog_verify\n+\tthen\n+\t\tgit -C \"$repo\" reflog --all >actual_logs &&\n+\t\tgit -C \"$repo\" reflog list >actual_log_list &&\n+\t\ttest_cmp expect_logs actual_logs &&\n+\t\ttest_cmp expect_log_list actual_log_list\n+\tfi &&\n \n-\tgit -C \"$1\" rev-parse --show-ref-format >actual &&\n-\techo \"$2\" >expect &&\n+\tgit -C \"$repo\" rev-parse --show-ref-format >actual &&\n+\techo \"$format\" >expect &&\n \ttest_cmp expect actual\n }\n \n test_expect_success 'setup' '\n-\trm -rf .git &&\n-\t# The migration does not yet support reflogs.\n-\tgit config --global core.logAllRefUpdates false\n+\trm -rf .git\n '\n \n test_expect_success \"superfluous arguments\" '\n@@ -78,19 +99,6 @@ do\n \t\t\ttest_cmp expect err\n \t\t'\n \n-\t\ttest_expect_success \"$from_format -> $to_format: migration with reflog fails\" '\n-\t\t\ttest_when_finished \"rm -rf repo\" &&\n-\t\t\tgit init --ref-format=$from_format repo &&\n-\t\t\ttest_config -C repo core.logAllRefUpdates true &&\n-\t\t\ttest_commit -C repo logged &&\n-\t\t\ttest_must_fail git -C repo refs migrate \\\n-\t\t\t\t--ref-format=$to_format 2>err &&\n-\t\t\tcat >expect <<-EOF &&\n-\t\t\terror: migrating reflogs is not supported yet\n-\t\t\tEOF\n-\t\t\ttest_cmp expect err\n-\t\t'\n-\n \t\ttest_expect_success \"$from_format -> $to_format: migration with worktree fails\" '\n \t\t\ttest_when_finished \"rm -rf repo\" &&\n \t\t\tgit init --ref-format=$from_format repo &&\n@@ -141,7 +149,7 @@ do\n \t\t\ttest_commit -C repo initial &&\n \t\t\ttest-tool -C repo ref-store main update-ref \"\" refs/heads/broken \\\n \t\t\t\t\"$(test_oid 001)\" \"$ZERO_OID\" REF_SKIP_CREATE_REFLOG,REF_SKIP_OID_VERIFICATION &&\n-\t\t\ttest_migration repo \"$to_format\" &&\n+\t\t\ttest_migration repo \"$to_format\" true &&\n \t\t\ttest_oid 001 >expect &&\n \t\t\tgit -C repo rev-parse refs/heads/broken >actual &&\n \t\t\ttest_cmp expect actual\n@@ -195,6 +203,27 @@ do\n \t\t\tgit -C repo rev-parse --show-ref-format >actual &&\n \t\t\ttest_cmp expect actual\n \t\t'\n+\n+\t\ttest_expect_success \"$from_format -> $to_format: reflogs of symrefs with target deleted\" '\n+\t\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t\tgit init --ref-format=$from_format repo &&\n+\t\t\ttest_commit -C repo initial &&\n+\t\t\tgit -C repo branch branch-1 HEAD &&\n+\t\t\tgit -C repo symbolic-ref refs/heads/symref refs/heads/branch-1 &&\n+\t\t\tcat >input <<-EOF &&\n+\t\t\tdelete refs/heads/branch-1\n+\t\t\tEOF\n+\t\t\tgit -C repo update-ref --stdin $to_format: reflogs order is retained\" '\n+\t\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t\tgit init --ref-format=$from_format repo &&\n+\t\t\ttest_commit --date \"100005000 +0700\" --no-tag -C repo initial &&\n+\t\t\ttest_commit --date \"100003000 +0700\" --no-tag -C repo second &&\n+\t\t\ttest_migration repo \"$to_format\"\n+\t\t'\n \tdone\n done\n \n\n-- \n2.47.1\n\n"},{"id":"509204","messageId":"Z2EhZpL5V0LlhEvj@pks.im","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 0/8] refs: add reflog support to `git refs migrate`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-17T06:59:50Z","receivedAt":"2024-12-17T07:00:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Dec 16, 2024 at 05:44:25PM +0100, Karthik Nayak wrote:\n> Changes in v4:\n> - Fix broken tests, due to two reasons in patch 8/8:\n> - The `index` field in `reflog_migration_data` wasn't initialized to\n> 0. This specifically doesn't break the test, but causes undefined\n> behavior. Fix this by using designated initializers.\n> - The strbuf within `migration_data` wasn't initialized when the flow\n> exited early, causing memory leaks. Fix this too by using designated\n> initializers.\n> - Thanks to Junio for reporting and Patrick for helping shed some light\n> on these broken tests.\n> - Link to v3: https://lore.kernel.org/r/20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com\n\nThe range-diff looks as expected, so this version of the series looks\ngood to me. Thanks!\n\nPatrick\n"},{"id":"509208","messageId":"CAOLa=ZSzGk1FqiciTktTgv4J1uH6CW6VpbxXOXdKmrXJtR6iyQ@mail.gmail.com","threadId":"62635","inReplyTo":"Z2EhZpL5V0LlhEvj@pks.im","subject":"Re: [PATCH v4 0/8] refs: add reflog support to `git refs migrate`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-17T09:35:10Z","receivedAt":"2024-12-17T09:35:12Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Dec 16, 2024 at 05:44:25PM +0100, Karthik Nayak wrote:\n>> Changes in v4:\n>> - Fix broken tests, due to two reasons in patch 8/8:\n>> - The `index` field in `reflog_migration_data` wasn't initialized to\n>> 0. This specifically doesn't break the test, but causes undefined\n>> behavior. Fix this by using designated initializers.\n>> - The strbuf within `migration_data` wasn't initialized when the flow\n>> exited early, causing memory leaks. Fix this too by using designated\n>> initializers.\n>> - Thanks to Junio for reporting and Patrick for helping shed some light\n>> on these broken tests.\n>> - Link to v3: https://lore.kernel.org/r/20241215-320-git-refs-migrate-reflogs-v3-0-4127fe707b98@gmail.com\n>\n> The range-diff looks as expected, so this version of the series looks\n> good to me. Thanks!\n>\n> Patrick\n\nThanks Patrick for your review!\n"},{"id":"509243","messageId":"xmqqv7vixamw.fsf@gitster.g","threadId":"62635","inReplyTo":"CAOLa=ZSzGk1FqiciTktTgv4J1uH6CW6VpbxXOXdKmrXJtR6iyQ@mail.gmail.com","subject":"Re: [PATCH v4 0/8] refs: add reflog support to `git refs migrate`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-17T21:28:07Z","receivedAt":"2024-12-17T21:28:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"karthik nayak writes:\n\n> Patrick Steinhardt writes:\n>\n>> The range-diff looks as expected, so this version of the series looks\n>> good to me. Thanks!\n>>\n>> Patrick\n>\n> Thanks Patrick for your review!\n\nThanks, both. Let's queue it again.\n"},{"id":"509346","messageId":"87bjx7h3q9.fsf@iotcl.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-2-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 2/8] refs: add `index` field to `struct ref_udpate`","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:28:30Z","receivedAt":"2024-12-19T19:28:42Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> The reftable backend, sorts its updates by refname before applying them,\n> this ensures that the references are stored sorted. When migrating\n> reflogs from one backend to another, the order of the reflogs must be\n> maintained. Add a new `index` field to the `ref_update` struct to\n> facilitate this.\n>\n> This field is used in the reftable backend's sort comparison function\n> `transaction_update_cmp`, to ensure that indexed fields maintain their\n> order.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> refs/refs-internal.h | 7 +++++++\n> refs/reftable-backend.c | 13 +++++++++++--\n> 2 files changed, 18 insertions(+), 2 deletions(-)\n>\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 0fd95cdacd99e4a728c22f5286f6b3f0f360c110..f5c733d099f0c6f1076a25f4f77d9d5eb345ec87 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -115,6 +115,13 @@ struct ref_update {\n> \tchar *msg;\n> \tchar *committer_info;\n> \n> +\t/*\n> +\t * The index overrides the default sort algorithm. This is needed\n> +\t * when migrating reflogs and we want to ensure we carry over the\n> +\t * same order.\n> +\t */\n> +\tunsigned int index;\n> +\n> \t/*\n> \t * If this ref_update was split off of a symref update via\n> \t * split_symref_update(), then this member points at that\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index e882602487c66261d586a94101bb1b4e9a2ed60e..c008f20be719fec3af6a8f81c821cb9c263764d7 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -1279,8 +1279,17 @@ static int reftable_be_transaction_abort(struct ref_store *ref_store UNUSED,\n> \n> static int transaction_update_cmp(const void *a, const void *b)\n> {\n> -\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n> -\t\t ((struct reftable_transaction_update *)b)->update->refname);\n> +\tstruct reftable_transaction_update *update_a = (struct reftable_transaction_update *)a;\n> +\tstruct reftable_transaction_update *update_b = (struct reftable_transaction_update *)b;\n> +\n> +\t/*\n> +\t * If there is an index set, it should take preference (default is 0).\n> +\t * This ensures that updates with indexes are sorted amongst themselves.\n> +\t */\n> +\tif (update_a->update->index || update_b->update->index)\n\nWhat if one of both simply isn't set, and the other one is? Then we\ncompare an unset with one that is set? Or am I being too paranoid?\n\n--\nToon\n"},{"id":"509347","messageId":"87a5crh3oy.fsf@iotcl.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-4-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 4/8] refs: extract out refname verification in transactions","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:29:17Z","receivedAt":"2024-12-19T19:29:39Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> Unless the `REF_SKIP_REFNAME_VERIFICATION` flag is set for an update,\n> the refname of the update is verified for:\n>\n> - Ensuring it is not a pseudoref.\n> - Checking the refname format.\n>\n> These checks will also be needed in a following commit where the\n> function to add reflog updates to the transaction is introduced. Extract\n> the code out into a new static function.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> refs.c | 37 +++++++++++++++++++++++--------------\n> 1 file changed, 23 insertions(+), 14 deletions(-)\n>\n> diff --git a/refs.c b/refs.c\n> index f003e51c6bf5229bfbce8ce61ffad7cdba0572e0..9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1196,6 +1196,28 @@ struct ref_update *ref_transaction_add_update(\n> \treturn update;\n> }\n> \n> +static int transaction_refname_valid(const char *refname,\n> +\t\t\t\t const struct object_id *new_oid,\n> +\t\t\t\t unsigned int flags, struct strbuf *err)\n> +{\n> +\tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n> +\t\treturn 1;\n> +\n> +\tif (is_pseudo_ref(refname)) {\n> +\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n> +\t\t\t refname);\n> +\t\treturn 0;\n\nWith this early return you don't need the `else` below? Why did you add\nit?\n\n> +\t} else if ((new_oid && !is_null_oid(new_oid)) ?\n> +\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n> +\t\t !refname_is_safe(refname)) {\n> +\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n> +\t\t\t refname);\n> +\t\treturn 0;\n> +\t}\n\nI see you've swapped order of checking whether it's a pseudoref with\nchecking whether the format is okay. I think this shouldn't make a big\ndifference, but it will give a different error message when attempting\nto update an illformatted pseudoref. For me it makes more sense how\nyou've done it now. But because you mention both checks as bullet points\nin the commit message, do you think it would make sense to say something\nabout them being swapped?\n\n-- \nToon\n"},{"id":"509348","messageId":"878qsbh3nh.fsf@iotcl.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-5-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:30:10Z","receivedAt":"2024-12-19T19:30:30Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> The `ref_transaction_add_update()` creates the `ref_update` struct. To\n> facilitate addition of reflogs in the next commit, the function needs to\n> accommodate setting the `committer_info` field in the struct. So modify\n> the function to also take `committer_info` as an argument and set it\n> accordingly.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> refs.c | 7 +++++--\n> refs/files-backend.c | 14 ++++++++------\n> refs/refs-internal.h | 1 +\n> refs/reftable-backend.c | 6 ++++--\n> 4 files changed, 18 insertions(+), 10 deletions(-)\n>\n> diff --git a/refs.c b/refs.c\n> index 9c9f4940c60d3cdd34ce8f1e668d17b9da3cd801..782bf1090af65196263a3c35ed18d878bb4f2967 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1166,6 +1166,7 @@ struct ref_update *ref_transaction_add_update(\n> \t\tconst struct object_id *new_oid,\n> \t\tconst struct object_id *old_oid,\n> \t\tconst char *new_target, const char *old_target,\n> +\t\tconst char *committer_info,\n> \t\tconst char *msg)\n> {\n> \tstruct ref_update *update;\n> @@ -1190,8 +1191,10 @@ struct ref_update *ref_transaction_add_update(\n> \t\toidcpy(&update->new_oid, new_oid);\n> \tif ((flags & REF_HAVE_OLD) && old_oid)\n> \t\toidcpy(&update->old_oid, old_oid);\n> -\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n> +\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n> +\t\tupdate->committer_info = xstrdup_or_null(committer_info);\n\nWhy only include the committer_info when we're not skipping reflog\nupdates?\n\n-- \nToon\n\n"},{"id":"509349","messageId":"875xnfh3kx.fsf@iotcl.com","threadId":"62635","inReplyTo":"CAOLa=ZT9KDNaz84DLoRBzY_yaZ3kZ2FUWADohZhM4EcrJgmS+A@mail.gmail.com","subject":"Re: [PATCH v2 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:31:42Z","receivedAt":"2024-12-19T19:31:55Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"karthik nayak writes:\n\n> Patrick Steinhardt writes:\n>\n>> On Fri, Dec 13, 2024 at 11:36:50AM +0100, Karthik Nayak wrote:\n>>> The `ref_transaction_add_update()` creates the `ref_update` struct. To\n>>> facilitate addition of reflogs in the next commit, the function needs to\n>>> accommodate setting the `committer_info` field in the struct. So modify\n>>> the function to also take `committer_info` as an argument and set it\n>>> accordingly.\n>>\n>> I was wondering a bit whether we could instead pull out a\n>> `add_update_internal()` function so that we don't need to modify all\n>> callers of `ref_transaction_add_update()`. Because ultimately, we don't\n>> use the field anywhere except from `ref_transaction_add_reflog_update()`\n>> as far as I can see.\n>>\n>> This is more of a thought than a strong opinion, so feel free to ignore.\n>>\n>\n> Yes, that is a possible change, but the number of code changes are\n> relatively low and I didn't think it made so much difference. Also\n> because we'd now have one more function. But I don't mind doing it\n> either, if anyone feels strongly about it, I'll happily make that\n> change.\n\nYes, I agree the number of callsites isn't that large, but on the other\nhand, I see various calls to this function having four `NULL`s in a row\nas arguments. Personally, I think that starts to smell a bit.\n\nNow, before you change anything. I'm not sure what Patrick was\nsuggesting? Would it mean we basically rename\n`ref_transaction_add_update()` to `add_update_internal()` and create a\nnew wrapper function `ref_transaction_add_update()` that simply calls\n`add_update_internal(..., NULL, msg)`? I don't think that's a\ngreat solution either.\n\nAlternively, because ref_transaction_add_update() returns the `struct\nref_update`, why not add a function `ref_update_set_committer` and call\nthat where we need to set the committer? I see this also will help in a\nfuture commit where you call ref_transaction_add_update() differently\ndepending on reflog updates.\n\n--\nToon\n"},{"id":"509350","messageId":"874j2zh3jv.fsf@iotcl.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-0-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 0/8] refs: add reflog support to `git refs migrate`","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:32:20Z","receivedAt":"2024-12-19T19:32:31Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> The `git refs migrate` command was introduced in\n> 25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n> 2024-06-06) to support migrating from one reference backend to another.\n>\n> One limitation of the feature was that it didn't support migrating\n> repositories which contained reflogs. This isn't a requirement on the\n> server side as repositories are stored as bare repositories (which do\n> not contain any reflogs). Clients however generally use reflogs and\n> until now couldn't use the `git refs migrate` command to migrate their\n> repositories to the new reftable format.\n>\n> One of the issues for adding reflog support is that the ref transactions\n> don't support reflogs additions:\n> 1. While there is REF_LOG_ONLY flag, there is no function to utilize\n> the flag and add reflogs.\n> 2. reference backends generally sort the updates by the refname. This\n> wouldn't work for reflogs which need to ensure that they maintain the\n> order of creation.\n> 3. In the files backend, reflog entries are added by obtaining locks\n> on the refs themselves. This means each update in the transaction, will\n> obtain a ref_lock. This paradigm fails to accompany the fact that there\n> could be multiple reflog updates for a refname in a single transaction.\n> 4. The backends check for duplicate entries, which doesn't make sense\n> in the context of adding multiple reflogs for a given refname.\n>\n> We overcome these issue we make the following changes:\n> - Update the ref_update structure to also include the committer\n> information. Using this, we can add a new function which only adds\n> reflog updates to the transaction.\n\nOut of interest, I see various changes happen around committer info. But\nwhy is the committer info more relevant for reflog updates, in contrast\nto normal ref updates?\n\n-- \nToon\n"},{"id":"509351","messageId":"8734ijh3jg.fsf@iotcl.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-6-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:32:35Z","receivedAt":"2024-12-19T19:32:45Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> Introduce a new function `ref_transaction_update_reflog`, for clients to\n> add a reflog update to a transaction. While the existing function\n> `ref_transaction_update` also allows clients to add a reflog entry, this\n> function does a few things more, It:\n> - Enforces that only a reflog entry is added and does not update the\n> ref itself.\n> - Allows the users to also provide the committer information. This\n> means clients can add reflog entries with custom committer\n> information.\n>\n> The `transaction_refname_valid()` function also modifies the error\n> message selectively based on the type of the update. This change also\n> affects reflog updates which go through `ref_transaction_update()`.\n>\n> A follow up commit will utilize this function to add reflog support to\n> `git refs migrate`.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> refs.c | 39 +++++++++++++++++++++++++++++++++++----\n> refs.h | 14 ++++++++++++++\n> refs/files-backend.c | 24 ++++++++++++++++--------\n> 3 files changed, 65 insertions(+), 12 deletions(-)\n>\n> diff --git a/refs.c b/refs.c\n> index 782bf1090af65196263a3c35ed18d878bb4f2967..8b3882cff17e5e3b0376f75654e32f81a23e5cb2 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1207,14 +1207,14 @@ static int transaction_refname_valid(const char *refname,\n> \t\treturn 1;\n> \n> \tif (is_pseudo_ref(refname)) {\n> -\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n> -\t\t\t refname);\n> +\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n\nThese strings are not localized.\n\n> +\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n> \t\treturn 0;\n> \t} else if ((new_oid && !is_null_oid(new_oid)) ?\n> \t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n> \t\t !refname_is_safe(refname)) {\n> -\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n> -\t\t\t refname);\n> +\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog with bad name\" : \"ref with bad name\";\n\nAlso these strings are not localized.\n\n-- \nToon\n"},{"id":"509352","messageId":"871py3h3ia.fsf@iotcl.com","threadId":"62635","inReplyTo":"20241216-320-git-refs-migrate-reflogs-v4-7-d7cd3f197453@gmail.com","subject":"Re: [PATCH v4 7/8] refs: allow multiple reflog entries for the same refname","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-19T19:33:17Z","receivedAt":"2024-12-19T19:33:31Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> The reference transaction only allows a single update for a given\n> reference to avoid conflicts. This, however, isn't an issue for reflogs.\n> There are no conflicts to be resolved in reflogs and when migrating\n> reflogs between backends we'd have multiple reflog entries for the same\n> refname.\n>\n> So allow multiple reflog updates within a single transaction. Also the\n> reflog creation logic isn't exposed to the end user. While this might\n> change in the future, currently, this reduces the scope of issues to\n> think about.\n>\n> In the reftable backend, the writer sorts all updates based on the\n> update_index before writing to the block. When there are multiple\n> reflogs for a given refname, it is essential that the order of the\n> reflogs is maintained. So add the `index` value to the `update_index`.\n> The `index` field is only set when multiple reflog entries for a given\n> refname are added and as such in most scenarios the old behavior\n> remains.\n>\n> This is required to add reflog migration support to `git refs migrate`.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> refs/files-backend.c | 15 +++++++++++----\n> refs/reftable-backend.c | 22 +++++++++++++++++++---\n> 2 files changed, 30 insertions(+), 7 deletions(-)\n>\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index c11213f52065bcf2fa7612df8f9500692ee2d02c..8953d1c6d37b13b0db701888b3db92fd87a68aaa 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -2611,6 +2611,9 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n> \n> \tupdate->backend_data = lock;\n> \n> +\tif (update->flags & REF_LOG_ONLY)\n> +\t\tgoto out;\n> +\n> \tif (update->type & REF_ISSYMREF) {\n> \t\tif (update->flags & REF_NO_DEREF) {\n> \t\t\t/*\n> @@ -2829,13 +2832,16 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n> \t */\n> \tfor (i = 0; i < transaction->nr; i++) {\n> \t\tstruct ref_update *update = transaction->updates[i];\n> -\t\tstruct string_list_item *item =\n> -\t\t\tstring_list_append(&affected_refnames, update->refname);\n> +\t\tstruct string_list_item *item;\n> \n> \t\tif ((update->flags & REF_IS_PRUNING) &&\n> \t\t !(update->flags & REF_NO_DEREF))\n> \t\t\tBUG(\"REF_IS_PRUNING set without REF_NO_DEREF\");\n> \n> +\t\tif (update->flags & REF_LOG_ONLY)\n> +\t\t\tcontinue;\n> +\n> +\t\titem = string_list_append(&affected_refnames, update->refname);\n> \t\t/*\n> \t\t * We store a pointer to update in item->util, but at\n> \t\t * the moment we never use the value of this field\n> @@ -3035,8 +3041,9 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \n> \t/* Fail if a refname appears more than once in the transaction: */\n> \tfor (i = 0; i < transaction->nr; i++)\n> -\t\tstring_list_append(&affected_refnames,\n> -\t\t\t\t transaction->updates[i]->refname);\n> +\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n> +\t\t\tstring_list_append(&affected_refnames,\n> +\t\t\t\t\t transaction->updates[i]->refname);\n> \tstring_list_sort(&affected_refnames);\n> \tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n> \t\tret = TRANSACTION_GENERIC_ERROR;\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -990,8 +990,9 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\tif (ret)\n> \t\t\tgoto done;\n> \n> -\t\tstring_list_append(&affected_refnames,\n> -\t\t\t\t transaction->updates[i]->refname);\n> +\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n> +\t\t\tstring_list_append(&affected_refnames,\n> +\t\t\t\t\t transaction->updates[i]->refname);\n> \t}\n> \n> \t/*\n> @@ -1301,6 +1302,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n> \tstruct reftable_log_record *logs = NULL;\n> \tstruct ident_split committer_ident = {0};\n> \tsize_t logs_nr = 0, logs_alloc = 0, i;\n> +\tuint64_t max_update_index = ts;\n> \tconst char *committer_info;\n> \tint ret = 0;\n> \n> @@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n> \t\t\t\t}\n> \n> \t\t\t\tfill_reftable_log_record(log, &c);\n> -\t\t\t\tlog->update_index = ts;\n> +\n> +\t\t\t\t/*\n> +\t\t\t\t * Updates are sorted by the writer. So updates for the same\n> +\t\t\t\t * refname need to contain different update indices.\n> +\t\t\t\t */\n> +\t\t\t\tlog->update_index = ts + u->index;\n\nDuring my review I was having a hard time figuring out when `u->index`\nwas not 0 and where it is being set. Can you maybe explain a bit?\n\n> +\n> +\t\t\t\t/*\n> +\t\t\t\t * Note the max update_index so the limit can be set later on.\n> +\t\t\t\t */\n> +\t\t\t\tif (log->update_index > max_update_index)\n\nIs there a lot of value in having this if clause? I was a bit confused\nwhy it is here, because I think we can do the assignment to\nmax_update_index unconditionally.\n\n> +\t\t\t\t\tmax_update_index = log->update_index;\n> +\n> \t\t\t\tlog->refname = xstrdup(u->refname);\n> \t\t\t\tmemcpy(log->value.update.new_hash,\n> \t\t\t\t u->new_oid.hash, GIT_MAX_RAWSZ);\n> @@ -1469,6 +1483,8 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n> \t * and log blocks.\n> \t */\n> \tif (logs) {\n> +\t\treftable_writer_set_limits(writer, ts, max_update_index);\n\nSo max_update_index is used to set the limits on the current writer, but\nusing reftable_stack_next_update_index() it's also used to give the next\nstack it's starting point for their range. Now I'm not familiar enough\nwith the code, but are all stacks handled in sequential order? And how\ndoes a stack relate to a reftable file?\n\n> +\n> \t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n> \t\tif (ret < 0)\n> \t\t\tgoto done;\n>\n> -- \n> 2.47.1\n"},{"id":"509353","messageId":"xmqq4j2zifmy.fsf@gitster.g","threadId":"62635","inReplyTo":"8734ijh3jg.fsf@iotcl.com","subject":"Re: [PATCH v4 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-19T20:25:57Z","receivedAt":"2024-12-19T20:26:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Toon Claes writes:\n\n>> +\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n>\n> These strings are not localized.\n>\n>> +\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n>> \t\treturn 0;\n\nAnd the structure forces sentence logo. If \"reflog for pseudoref\"\nwere masculine and \"pseudoref\" were feminine in a language where the\nverb \"update\" conjugates differently based on its object, the\nresulting construction cannot be translated. Rather, we'd need to\ndo something uglier like this:\n\n\tconst char *refusal_msg;\n\tif (flag & REF_LOG_ONLY)\n\t\trefusal_msg = _(\"refusing to update reflog for pseudoref '%s'\");\n\telse\n\t\trefusal_msg = _(\"refusing to update pseudoref '%s'\");\n\t...\n\tstrbuf_addf(err, refusal_msg, refname);\n\n"},{"id":"509372","messageId":"CAOLa=ZRs3uy6ZyMeHCJUGgcDC3u9KXAymoK5SWD8_KJk3jxygw@mail.gmail.com","threadId":"62635","inReplyTo":"87bjx7h3q9.fsf@iotcl.com","subject":"Re: [PATCH v4 2/8] refs: add `index` field to `struct ref_udpate`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T10:09:06Z","receivedAt":"2024-12-20T10:09:08Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> Karthik Nayak writes:\n>> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n>> index e882602487c66261d586a94101bb1b4e9a2ed60e..c008f20be719fec3af6a8f81c821cb9c263764d7 100644\n>> --- a/refs/reftable-backend.c\n>> +++ b/refs/reftable-backend.c\n>> @@ -1279,8 +1279,17 @@ static int reftable_be_transaction_abort(struct ref_store *ref_store UNUSED,\n>>\n>> static int transaction_update_cmp(const void *a, const void *b)\n>> {\n>> -\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n>> -\t\t ((struct reftable_transaction_update *)b)->update->refname);\n>> +\tstruct reftable_transaction_update *update_a = (struct reftable_transaction_update *)a;\n>> +\tstruct reftable_transaction_update *update_b = (struct reftable_transaction_update *)b;\n>> +\n>> +\t/*\n>> +\t * If there is an index set, it should take preference (default is 0).\n>> +\t * This ensures that updates with indexes are sorted amongst themselves.\n>> +\t */\n>> +\tif (update_a->update->index || update_b->update->index)\n>\n> What if one of both simply isn't set, and the other one is? Then we\n> compare an unset with one that is set? Or am I being too paranoid?\n>\n> --\n> Toon\n\nThose are expected scenarios, if one of them contains an index value,\nthen it'll be sorted before the other. At the end, we need:\n1. Values with index to be sorted amongst themselves by index value.\n2. Values without index to be sorted amongst themselves by the refname.\n\nKarthik\n"},{"id":"509373","messageId":"CAOLa=ZRnwqsChx9SiMv_95Leyijh_E7zEhkeHQOea=gjfGZ-Qg@mail.gmail.com","threadId":"62635","inReplyTo":"87a5crh3oy.fsf@iotcl.com","subject":"Re: [PATCH v4 4/8] refs: extract out refname verification in transactions","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T10:30:43Z","receivedAt":"2024-12-20T10:30:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> Karthik Nayak writes:\n>> +static int transaction_refname_valid(const char *refname,\n>> +\t\t\t\t const struct object_id *new_oid,\n>> +\t\t\t\t unsigned int flags, struct strbuf *err)\n>> +{\n>> +\tif (flags & REF_SKIP_REFNAME_VERIFICATION)\n>> +\t\treturn 1;\n>> +\n>> +\tif (is_pseudo_ref(refname)) {\n>> +\t\tstrbuf_addf(err, _(\"refusing to update pseudoref '%s'\"),\n>> +\t\t\t refname);\n>> +\t\treturn 0;\n>\n> With this early return you don't need the `else` below? Why did you add\n> it?\n>\n\nYou mean we could simply have\n\n if { ... check for pseudoref ... }\n\n if { ... check for bad refname ... }\n\n return -1;\n\nthen, you're right, that would work too. No specific reason that I added\nan else.\n\n>> +\t} else if ((new_oid && !is_null_oid(new_oid)) ?\n>> +\t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n>> +\t\t !refname_is_safe(refname)) {\n>> +\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name '%s'\"),\n>> +\t\t\t refname);\n>> +\t\treturn 0;\n>> +\t}\n>\n> I see you've swapped order of checking whether it's a pseudoref with\n> checking whether the format is okay. I think this shouldn't make a big\n> difference, but it will give a different error message when attempting\n> to update an illformatted pseudoref. For me it makes more sense how\n> you've done it now. But because you mention both checks as bullet points\n> in the commit message, do you think it would make sense to say something\n> about them being swapped?\n>\n\nI actually didn't notice that I did swap them. It doesn't change the\nlogic. However, for creation of a pseudoref, in the old flow, we'd check\nif the refname is safe and then go to the section where we check for\npseudorefs. Now we simply skip to the pseudoref check. I'll add more\ndetails in the commit message locally for now and will include it if I\ndo re-roll!\n\n> --\n> Toon\n"},{"id":"509375","messageId":"CAOLa=ZRUQVwkyv+JgCPoqAkUar+0Yt4oYJ-c=tfgCL+W8v6vMw@mail.gmail.com","threadId":"62635","inReplyTo":"878qsbh3nh.fsf@iotcl.com","subject":"Re: [PATCH v4 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T10:44:15Z","receivedAt":"2024-12-20T10:44:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> Karthik Nayak writes:\n\n[snip]\n\n>> {\n>> \tstruct ref_update *update;\n>> @@ -1190,8 +1191,10 @@ struct ref_update *ref_transaction_add_update(\n>> \t\toidcpy(&update->new_oid, new_oid);\n>> \tif ((flags & REF_HAVE_OLD) && old_oid)\n>> \t\toidcpy(&update->old_oid, old_oid);\n>> -\tif (!(flags & REF_SKIP_CREATE_REFLOG))\n>> +\tif (!(flags & REF_SKIP_CREATE_REFLOG)) {\n>> +\t\tupdate->committer_info = xstrdup_or_null(committer_info);\n>\n> Why only include the committer_info when we're not skipping reflog\n> updates?\n>\n> --\n> Toon\n\nThe `committer_info` contains information around\n1. author of a ref update\n2. date/time of the update\n\nThis is only relevant in the context of reflogs. Regular ref updates\ndon't store this information. Hence we only add it for reflogs here.\n"},{"id":"509376","messageId":"CAOLa=ZR3fNRdGyYcvOBi=79WyOEr5VF7KD=p3JuGO6jUCfxANw@mail.gmail.com","threadId":"62635","inReplyTo":"xmqq4j2zifmy.fsf@gitster.g","subject":"Re: [PATCH v4 6/8] refs: introduce the `ref_transaction_update_reflog` function","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T10:55:25Z","receivedAt":"2024-12-20T10:55:27Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Toon Claes writes:\n>\n>>> +\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n>>\n>> These strings are not localized.\n>>\n>>> +\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n>>> \t\treturn 0;\n>\n> And the structure forces sentence logo. If \"reflog for pseudoref\"\n> were masculine and \"pseudoref\" were feminine in a language where the\n> verb \"update\" conjugates differently based on its object, the\n> resulting construction cannot be translated. Rather, we'd need to\n> do something uglier like this:\n>\n> \tconst char *refusal_msg;\n> \tif (flag & REF_LOG_ONLY)\n> \t\trefusal_msg = _(\"refusing to update reflog for pseudoref '%s'\");\n> \telse\n> \t\trefusal_msg = _(\"refusing to update pseudoref '%s'\");\n> \t...\n> \tstrbuf_addf(err, refusal_msg, refname);\n\nIndeed. Will add this in. I think its better to prioritize good\ntranslation here.\n"},{"id":"509378","messageId":"CAOLa=ZRQsVMBahopKZLwFX1qZuJ35EZPBNmoxbidzGkheqSpEw@mail.gmail.com","threadId":"62635","inReplyTo":"871py3h3ia.fsf@iotcl.com","subject":"Re: [PATCH v4 7/8] refs: allow multiple reflog entries for the same refname","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T11:15:07Z","receivedAt":"2024-12-20T11:15:09Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> Karthik Nayak writes:\n>\n>> The reference transaction only allows a single update for a given\n>> reference to avoid conflicts. This, however, isn't an issue for reflogs.\n>> There are no conflicts to be resolved in reflogs and when migrating\n>> reflogs between backends we'd have multiple reflog entries for the same\n>> refname.\n>>\n>> So allow multiple reflog updates within a single transaction. Also the\n>> reflog creation logic isn't exposed to the end user. While this might\n>> change in the future, currently, this reduces the scope of issues to\n>> think about.\n>>\n>> In the reftable backend, the writer sorts all updates based on the\n>> update_index before writing to the block. When there are multiple\n>> reflogs for a given refname, it is essential that the order of the\n>> reflogs is maintained. So add the `index` value to the `update_index`.\n>> The `index` field is only set when multiple reflog entries for a given\n>> refname are added and as such in most scenarios the old behavior\n>> remains.\n>>\n>> This is required to add reflog migration support to `git refs migrate`.\n>>\n>> Signed-off-by: Karthik Nayak \n>> ---\n>> refs/files-backend.c | 15 +++++++++++----\n>> refs/reftable-backend.c | 22 +++++++++++++++++++---\n>> 2 files changed, 30 insertions(+), 7 deletions(-)\n>>\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index c11213f52065bcf2fa7612df8f9500692ee2d02c..8953d1c6d37b13b0db701888b3db92fd87a68aaa 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -2611,6 +2611,9 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n>>\n>> \tupdate->backend_data = lock;\n>>\n>> +\tif (update->flags & REF_LOG_ONLY)\n>> +\t\tgoto out;\n>> +\n>> \tif (update->type & REF_ISSYMREF) {\n>> \t\tif (update->flags & REF_NO_DEREF) {\n>> \t\t\t/*\n>> @@ -2829,13 +2832,16 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n>> \t */\n>> \tfor (i = 0; i < transaction->nr; i++) {\n>> \t\tstruct ref_update *update = transaction->updates[i];\n>> -\t\tstruct string_list_item *item =\n>> -\t\t\tstring_list_append(&affected_refnames, update->refname);\n>> +\t\tstruct string_list_item *item;\n>>\n>> \t\tif ((update->flags & REF_IS_PRUNING) &&\n>> \t\t !(update->flags & REF_NO_DEREF))\n>> \t\t\tBUG(\"REF_IS_PRUNING set without REF_NO_DEREF\");\n>>\n>> +\t\tif (update->flags & REF_LOG_ONLY)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\titem = string_list_append(&affected_refnames, update->refname);\n>> \t\t/*\n>> \t\t * We store a pointer to update in item->util, but at\n>> \t\t * the moment we never use the value of this field\n>> @@ -3035,8 +3041,9 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n>>\n>> \t/* Fail if a refname appears more than once in the transaction: */\n>> \tfor (i = 0; i < transaction->nr; i++)\n>> -\t\tstring_list_append(&affected_refnames,\n>> -\t\t\t\t transaction->updates[i]->refname);\n>> +\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n>> +\t\t\tstring_list_append(&affected_refnames,\n>> +\t\t\t\t\t transaction->updates[i]->refname);\n>> \tstring_list_sort(&affected_refnames);\n>> \tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n>> \t\tret = TRANSACTION_GENERIC_ERROR;\n>> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n>> index b2e3ba877de9e59fea5a4d066eb13e60ef22a32b..bec5962debea7b62572d08f6fa8fd38ab4cd8af6 100644\n>> --- a/refs/reftable-backend.c\n>> +++ b/refs/reftable-backend.c\n>> @@ -990,8 +990,9 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n>> \t\tif (ret)\n>> \t\t\tgoto done;\n>>\n>> -\t\tstring_list_append(&affected_refnames,\n>> -\t\t\t\t transaction->updates[i]->refname);\n>> +\t\tif (!(transaction->updates[i]->flags & REF_LOG_ONLY))\n>> +\t\t\tstring_list_append(&affected_refnames,\n>> +\t\t\t\t\t transaction->updates[i]->refname);\n>> \t}\n>>\n>> \t/*\n>> @@ -1301,6 +1302,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n>> \tstruct reftable_log_record *logs = NULL;\n>> \tstruct ident_split committer_ident = {0};\n>> \tsize_t logs_nr = 0, logs_alloc = 0, i;\n>> +\tuint64_t max_update_index = ts;\n>> \tconst char *committer_info;\n>> \tint ret = 0;\n>>\n>> @@ -1405,7 +1407,19 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n>> \t\t\t\t}\n>>\n>> \t\t\t\tfill_reftable_log_record(log, &c);\n>> -\t\t\t\tlog->update_index = ts;\n>> +\n>> +\t\t\t\t/*\n>> +\t\t\t\t * Updates are sorted by the writer. So updates for the same\n>> +\t\t\t\t * refname need to contain different update indices.\n>> +\t\t\t\t */\n>> +\t\t\t\tlog->update_index = ts + u->index;\n>\n> During my review I was having a hard time figuring out when `u->index`\n> was not 0 and where it is being set. Can you maybe explain a bit?\n>\n\nAs of this patch, there is no users of the index. This patch adds in the\ninfrastructure. The next patch is where we actually set the index.\n\nIn short, the index is only needed for the reftable backend. This is\nbecause reflogs contain a specific order and we need to retain that\norder. In the reftable backend. For optimization, all writes are sorted\nby refnames. The index provided a parallel system to retain the order of\nthe updates. There are no real usecases apart from migration of reflogs\nfrom one backend to another, which is added in the next patch.\n\n>> +\n>> +\t\t\t\t/*\n>> +\t\t\t\t * Note the max update_index so the limit can be set later on.\n>> +\t\t\t\t */\n>> +\t\t\t\tif (log->update_index > max_update_index)\n>\n> Is there a lot of value in having this if clause? I was a bit confused\n> why it is here, because I think we can do the assignment to\n> max_update_index unconditionally.\n>\n\nIt is necessary. For reflogs whose index isn't set, their `update_index`\nwould simply be the `ts` value. So if there are a mix of reflog updates\nwith and without index, we could end up with a scenario where we don't\nset the max to the actual max.\n\n>> +\t\t\t\t\tmax_update_index = log->update_index;\n>> +\n>> \t\t\t\tlog->refname = xstrdup(u->refname);\n>> \t\t\t\tmemcpy(log->value.update.new_hash,\n>> \t\t\t\t u->new_oid.hash, GIT_MAX_RAWSZ);\n>> @@ -1469,6 +1483,8 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n>> \t * and log blocks.\n>> \t */\n>> \tif (logs) {\n>> +\t\treftable_writer_set_limits(writer, ts, max_update_index);\n>\n> So max_update_index is used to set the limits on the current writer, but\n> using reftable_stack_next_update_index() it's also used to give the next\n> stack it's starting point for their range.\n\nUsing `reftable_stack_next_update_index()` would return `ts + 1` as that\nis the next sequential update. This could be lesser than the\nmax_update_index. So we can't use that. Once all the reflogs are\nwritten, the next call to `reftable_stack_next_update_index()` would\nreturn `max_update_index + 1`.\n\n> Now I'm not familiar enough with the code, but are all stacks handled\n> in sequential order?\n\nNot sure I understand your question correctly. Updates are handled as\nper a given index. Each update is also sequentially stored. Tables are\nnamed after the min and max index that they store.\n\n> And how does a stack relate to a reftable file?\n\nThe stack is used to refer to a collection of reftable tables. So for a\ngiven worktree, the tables under '$GIT_DIR/reftable' would constitute a\nstack, where the 'tables.list' would state the tables which are part of\nthe stack\n\n>> +\n>> \t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n>> \t\tif (ret < 0)\n>> \t\t\tgoto done;\n>>\n>> --\n>> 2.47.1\n"},{"id":"509380","messageId":"CAOLa=ZRcp+OxiQxYGJeQ8AEWNk-6SkrRsZdi8bnNtNaWe_zrhg@mail.gmail.com","threadId":"62635","inReplyTo":"874j2zh3jv.fsf@iotcl.com","subject":"Re: [PATCH v4 0/8] refs: add reflog support to `git refs migrate`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T11:23:29Z","receivedAt":"2024-12-20T11:23:30Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> Karthik Nayak writes:\n>\n>> The `git refs migrate` command was introduced in\n>> 25a0023f28 (builtin/refs: new command to migrate ref storage formats,\n>> 2024-06-06) to support migrating from one reference backend to another.\n>>\n>> One limitation of the feature was that it didn't support migrating\n>> repositories which contained reflogs. This isn't a requirement on the\n>> server side as repositories are stored as bare repositories (which do\n>> not contain any reflogs). Clients however generally use reflogs and\n>> until now couldn't use the `git refs migrate` command to migrate their\n>> repositories to the new reftable format.\n>>\n>> One of the issues for adding reflog support is that the ref transactions\n>> don't support reflogs additions:\n>> 1. While there is REF_LOG_ONLY flag, there is no function to utilize\n>> the flag and add reflogs.\n>> 2. reference backends generally sort the updates by the refname. This\n>> wouldn't work for reflogs which need to ensure that they maintain the\n>> order of creation.\n>> 3. In the files backend, reflog entries are added by obtaining locks\n>> on the refs themselves. This means each update in the transaction, will\n>> obtain a ref_lock. This paradigm fails to accompany the fact that there\n>> could be multiple reflog updates for a refname in a single transaction.\n>> 4. The backends check for duplicate entries, which doesn't make sense\n>> in the context of adding multiple reflogs for a given refname.\n>>\n>> We overcome these issue we make the following changes:\n>> - Update the ref_update structure to also include the committer\n>> information. Using this, we can add a new function which only adds\n>> reflog updates to the transaction.\n>\n> Out of interest, I see various changes happen around committer info. But\n> why is the committer info more relevant for reflog updates, in contrast\n> to normal ref updates?\n>\n> --\n> Toon\n\nThe committer info is metadata around a ref update. It is only stored in\nthe reflogs. So they're only relevant for reflogs! For e.g. in the files\nbackend, we can see:\n\n $ cat .git/logs/HEAD | tail -1\n 7a929cb27ca6df69d4db64b008e27e002c691028\nbc8e00178b671da2b845ccbba175f6c093ed6949 Karthik Nayak\n 1734624674 +0100\tcommit: c10\n\nHere the 'Karthik Nayak 1734624673 +0100'\nsection is the committer_info. Whereas if you see the reference itself\n\n $ cat .git/HEAD\n ref: refs/heads/master\n\nIt only contains reference information. So, the `committer_info` is\nnot needed for regular ref updates. Earlier we dynamically obtain the\ninformation when a reflog was being created. But for migration of\nexisting reflogs we need to pass this information from the source to the\nref transaction mechanism. So we pass the `committer_info` through the\nlayers.\n\nThank you for the review.\n"},{"id":"509381","messageId":"CAOLa=ZTx3T=md4VQ6tRb5U41nOHsbhTB5d6VsmyUHFxHgQQ88g@mail.gmail.com","threadId":"62635","inReplyTo":"875xnfh3kx.fsf@iotcl.com","subject":"Re: [PATCH v2 5/8] refs: add `committer_info` to `ref_transaction_add_update()`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T11:31:15Z","receivedAt":"2024-12-20T11:31:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> karthik nayak writes:\n>\n>> Patrick Steinhardt writes:\n>>\n>>> On Fri, Dec 13, 2024 at 11:36:50AM +0100, Karthik Nayak wrote:\n>>>> The `ref_transaction_add_update()` creates the `ref_update` struct. To\n>>>> facilitate addition of reflogs in the next commit, the function needs to\n>>>> accommodate setting the `committer_info` field in the struct. So modify\n>>>> the function to also take `committer_info` as an argument and set it\n>>>> accordingly.\n>>>\n>>> I was wondering a bit whether we could instead pull out a\n>>> `add_update_internal()` function so that we don't need to modify all\n>>> callers of `ref_transaction_add_update()`. Because ultimately, we don't\n>>> use the field anywhere except from `ref_transaction_add_reflog_update()`\n>>> as far as I can see.\n>>>\n>>> This is more of a thought than a strong opinion, so feel free to ignore.\n>>>\n>>\n>> Yes, that is a possible change, but the number of code changes are\n>> relatively low and I didn't think it made so much difference. Also\n>> because we'd now have one more function. But I don't mind doing it\n>> either, if anyone feels strongly about it, I'll happily make that\n>> change.\n>\n> Yes, I agree the number of callsites isn't that large, but on the other\n> hand, I see various calls to this function having four `NULL`s in a row\n> as arguments. Personally, I think that starts to smell a bit.\n>\n\nI agree with your reasoning here..\n\n> Now, before you change anything. I'm not sure what Patrick was\n> suggesting? Would it mean we basically rename\n> `ref_transaction_add_update()` to `add_update_internal()` and create a\n> new wrapper function `ref_transaction_add_update()` that simply calls\n> `add_update_internal(..., NULL, msg)`? I don't think that's a\n> great solution either.\n>\n\nYes, agreed with this too.\n\n> Alternively, because ref_transaction_add_update() returns the `struct\n> ref_update`, why not add a function `ref_update_set_committer` and call\n> that where we need to set the committer? I see this also will help in a\n> future commit where you call ref_transaction_add_update() differently\n> depending on reflog updates.\n>\n\nI think this seems like a nice way to go about it. Currently all the\nlogic pertaining to creating an `ref_update` struct is contained within\n`ref_transaction_add_update()`. So having individual functions would\nmake sense, but the con here is that this doesn't enforce fields to be\nset. But for `committer_info` it does make sense. I'm going to leave it\nfor now since the series is merged to `next`. Maybe something to do in\nthe future as part of #leftoverbits.\n\n> --\n> Toon\n"},{"id":"509389","messageId":"20241220125837.2774153-1-karthik.188@gmail.com","threadId":"62635","inReplyTo":"xmqq4j2zifmy.fsf@gitster.g","subject":"[PATCH] refs: mark invalid refname message for translation","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-20T12:58:37Z","receivedAt":"2024-12-20T12:58:49Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The error message produced by `transaction_refname_valid()` changes based\non whether the update is a ref update or a reflog update, with the use\nof a ternary operator. This breaks translation since the sub-msg is not\nmarked for translation. Fix this by setting the entire message using a\n`if {} else {}` block and marking each message for translation.\n\nSigned-off-by: Karthik Nayak \n---\n\nSince the reflog migration topic has been merged to 'next', I am sending this \nas an individual patch which applies on top of 'kn/reflog-migration'. \n\nJunio, I'd also be happy to re-roll the series if that is better.\n\n refs.c | 16 ++++++++++++----\n 1 file changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 58e543ce39..9c887427f2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1255,14 +1255,22 @@ static int transaction_refname_valid(const char *refname,\n \t\treturn 1;\n \n \tif (is_pseudo_ref(refname)) {\n-\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog for pseudoref\" : \"pseudoref\";\n-\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n+\t\tconst char *refusal_msg;\n+\t\tif (flags & REF_LOG_ONLY)\n+\t\t\trefusal_msg = _(\"refusing to update reflog for pseudoref '%s'\");\n+\t\telse\n+\t\t\trefusal_msg = _(\"refusing to update pseudoref '%s'\");\n+\t\tstrbuf_addf(err, refusal_msg, refname);\n \t\treturn 0;\n \t} else if ((new_oid && !is_null_oid(new_oid)) ?\n \t\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) :\n \t\t !refname_is_safe(refname)) {\n-\t\tconst char *what = flags & REF_LOG_ONLY ? \"reflog with bad name\" : \"ref with bad name\";\n-\t\tstrbuf_addf(err, _(\"refusing to update %s '%s'\"), what, refname);\n+\t\tconst char *refusal_msg;\n+\t\tif (flags & REF_LOG_ONLY)\n+\t\t\trefusal_msg = _(\"refusing to update reflog with bad name '%s'\");\n+\t\telse\n+\t\t\trefusal_msg = _(\"refusing to update ref with bad name '%s'\");\n+\t\tstrbuf_addf(err, refusal_msg, refname);\n \t\treturn 0;\n \t}\n \n-- \n2.47.1\n\n"},{"id":"509401","messageId":"xmqqzfkqe4g7.fsf@gitster.g","threadId":"62635","inReplyTo":"20241220125837.2774153-1-karthik.188@gmail.com","subject":"Re: [PATCH] refs: mark invalid refname message for translation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-20T15:53:28Z","receivedAt":"2024-12-20T15:53:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The error message produced by `transaction_refname_valid()` changes based\n> on whether the update is a ref update or a reflog update, with the use\n> of a ternary operator. This breaks translation since the sub-msg is not\n> marked for translation. Fix this by setting the entire message using a\n> `if {} else {}` block and marking each message for translation.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n>\n> Since the reflog migration topic has been merged to 'next', I am sending this \n> as an individual patch which applies on top of 'kn/reflog-migration'. \n\nThanks, that is the most sensible way to fix up a glitch that was\ndiscovered too late ;-) Will queue.\n"},{"id":"509546","messageId":"87zfkl74k9.fsf@iotcl.com","threadId":"62635","inReplyTo":"xmqqzfkqe4g7.fsf@gitster.g","subject":"Re: [PATCH] refs: mark invalid refname message for translation","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-24T10:34:14Z","receivedAt":"2024-12-24T10:34:33Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Since the reflog migration topic has been merged to 'next', I am sending this \n>> as an individual patch which applies on top of 'kn/reflog-migration'. \n>\n> Thanks, that is the most sensible way to fix up a glitch that was\n> discovered too late ;-) Will queue.\n\nThanks both for dealing with this so swiftly! Error messag is way better\nlike this.\n\n-- \nToon\n"}]}