{"thread":{"id":"46582","subject":"[RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","startedAt":"2017-08-13T19:36:31Z","lastAt":"2017-08-30T13:17:34Z","messageCount":13,"participants":["Richard Maw","Stefan Beller","Junio C Hamano","Michael Haggerty"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"326249","messageId":"20170813193611.4233-1-richard.maw@gmail.com","threadId":"46582","inReplyTo":null,"subject":"[RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:04Z","receivedAt":"2017-08-13T19:36:31Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"Forewarning: I don't consider this work complete\nand am unlikely to find time to finish it any time soon.\nI've mostly sent this because it may include valuable feedback\non how well the ref storage backends works\nfrom trying to use it to change how git namespaces work.\n\nIntroduction\n============\n\nI work on a git server called Gitano,\nand I'd like to add support for git namespaces to:\n\n1.  Allow administrative content to be kept out of the default namespace.\n\n    Gitano keeps some repository configuration in a ref in the repository.\n    I'd like to be able to hide it in a namespace\n    to make fetching all the content easier, such as for mirroring.\n\n2.  Implement repository forking backed by namespaces in the same repo.\n\n    I'd like to be able to add a logical vs physical repository separation\n    so that forking a repository just copies the refs into a namespace.\n\n3.  Any other inventive uses users could think of.\n\n\nIt's possible with some hackery to do all this with the existing namespace code\nbut we like to integrate with a web frontend, and a web frontend needs to do\nmore than just let you push and pull the repository.\n\nUnfortunately namespace handling was never implemented for any other part of git\nand at least gitolite makes use of namespaces,\nand will have to work around it not being implemented fully,\nbut implementing it more fully will break work-arounds.\n\nInstead of continuing to do work-arounds I tried to add support into libgit,\nsince as well as making it less code in the web frontend to work around it,\nit would become common code that other git servers may make use of.\n\n\nThis is not my first attempt to improve the git namespace handling in git.\nI tried last year, but it took me so long that all the ref handling code changed\nand I would have had to start from scratch.\n\nFortunately the pluggable ref backends work provided an easier starting point.\n\nThe included patches work for the use-cases I need to support in gitano,\nbut I don't want it to end up like GIT_NAMESPACE and not be fixable,\nso I've tried to make it work such that everything in the git test suite works.\n\nI was not successful with this due to time constraints\nbut it helped me find many issues and I've listed known issues.\n\nTechnical description\n=====================\n\nThis patch series adds a new refs backend, stacking on top of the files backend,\nbased on whether `core.namespace` is set in git config.\n\nSince config can be done on the command-line or in config files,\nboth per-repo or global, it's a reasonable approach.\n\nThis allows the default namespace of a repository to be changed\nso that operations such as fetch only show refs of that namespace,\nor a smarter git server to intercept the repository path\nand redirect it to a different repository with a namespace set.\n\nIt also mostly works for local git operations,\nbut I've no real idea what they would be useful for\nbeyond what's needed for a git server's web frontend.\n\nIt may make sense in future to allow remotes to have namespaces\nso you can push to a namespace on the server without it being namespace aware,\nbut that's not the goal of this series,\nand may be possible to handle without modification by changing the refspecs.\n\nThe majority of what the namespaced refs backend does\nis to prepend the namespace to requests before the files backend sees them\nand strip namespaces off returned refs before the caller sees them.\n\nIssues\n======\n\nHaving core.namespace set in a repository namespaces the fetch results.\nIf the source repository is local then setting -c core.namespace\nto fetch the content into a different namespace will break the source namespace\nsince the upload-pack also obeys the -c core.namespace on the command-line.\n\nSupporting `clone --namespace` instead of `git -c` might resolve this,\nbut I can't think of an appropriate use-case for wanting to do so\nwhich makes it hard to consider what API would be appropriate.\n\nBugs\n----\n\nMost boil down to how special refs like HEAD are handled.\n\n1.  Logged messages display the namespaced path,\n    which a human may deal with but confuses the test suite.\n\n2.  Reflogs for namespaced HEAD are not updated.\n\n    This is because resolving HEAD to split the transaction's updates\n    to add a log only update to HEAD works by transaction_prepare resolving HEAD\n    using its own ref store rather than the main one,\n    so the namespace translation isn't performed.\n    See split_head_update.\n\n    The fix for this may be to move the transaction mangling out of the backend,\n    unless it should be implied that every backend implementation\n    must be responsible for symbolic ref reflog updates implicitly.\n\n3.  git init always says Reinitialized repository.\n\n    This is because create_default_files checks for whether HEAD already exists\n    directly via fs access rather than through the refs backend,\n    but then creates the symbolic ref using it.\n\n    Without a workaround this causes clones to fail because they don't recognise\n    that the repository is initialized because .git/HEAD is missing.\n    By working around this and creating .git/HEAD during the init_db callback\n    instead create_default_files thinks it's a reinitialisation\n    since it expected to create HEAD itself,\n    and fails to actually create HEAD.\n    The implemented workaround for that is for the namespaced backend's init_db\n    to create both .git/HEAD and the namespaced HEAD.\n\n    This could be fixed by:\n\n    1.  Make create_default_files check whether HEAD exists via the refs backend\n\n        This wouldn't fix .git/HEAD being missing in other cases,\n        but allows namespaced_init_db to only create .git/HEAD\n        and would fix the reinitialisation message.\n\n        I attempted to do this, but running the test suite on the change\n        caused it to corrupt my repository,\n        so I haven't dared try it again.\n\n    2.  Making backends responsible for creating HEAD in init_db.\n\n        I'm unsure whether creating HEAD is the responsibility of the workspace\n        or the responsibility of the refs backend.\n\n4.  delete-refs won't remove namespaced symbolic refs.\n\n    The files backend's delete_refs calls back into refs_delete_ref,\n    which normally bypasses the backend to delete pseudorefs,\n    whether the NODEREF option has been passed or not.\n\n    (see t1405-main-ref-store for a test that triggers this)\n\n    This could be fixed by changing is_pseudoref_syntax to strip any namespace\n    off the front before checking whether it's outside of refs/ and is all caps,\n    however this would then mean that anywhere that calls ref_delete\n    would delete the pseudoref without namespace translation.\n\n    Fixing this requires one of:\n\n    1.  A work-around in the namespaced ref backend\n        so that if it looks like a pseudoref after the namespace is stripped\n        then the NODEREF flag is implicitly added to any deletes.\n    2.  Ref backends to be given domain over pseudorefs.\n    3.  Pseudoref backends.\n\n5.  Per-workspace namespaced refs don't work.\n\n    refs/bisect/ gets moved to refs/namespaces/foo/refs/bisect,\n    but those refs can't be found in iteration\n    because the files backend walks the refs in the common dir\n    rather than the refs in the git dir,\n    and the simple workaround in loose_fill_ref_dir doesn't work.\n\nRichard Maw (7):\n  Expose expand_namespace API\n  Add git_configset_add_standard\n  Add helper for skipping namespace prefixes\n  Autocreate reflogs for namespaced refs\n  Treat namespaced HEAD and refs/bisect as per-worktree\n  Add namespaced ref backend\n  Plumb in namespaced ref store\n\n Makefile                  |   1 +\n cache.h                   |   1 +\n config.c                  |   7 +-\n config.h                  |   1 +\n environment.c             |   2 +-\n git-compat-util.h         |  19 ++\n refs.c                    |   7 +-\n refs/namespaced-backend.c | 619 ++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h      |   1 +\n 9 files changed, 654 insertions(+), 4 deletions(-)\n create mode 100644 refs/namespaced-backend.c\n\n-- \n2.9.0\n\n"},{"id":"326250","messageId":"20170813193611.4233-2-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 1/7] Expose expand_namespace API","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:05Z","receivedAt":"2017-08-13T19:36:33Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"Namespaces will not only be settable with GIT_NAMESPACE,\nso this previously internal helper needs to be made available to other code.\n---\n cache.h       | 1 +\n environment.c | 2 +-\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 71fe092..e01b8a2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -485,6 +485,7 @@ extern char *get_graft_file(void);\n extern int set_git_dir(const char *path);\n extern int get_common_dir_noenv(struct strbuf *sb, const char *gitdir);\n extern int get_common_dir(struct strbuf *sb, const char *gitdir);\n+extern char *expand_namespace(const char *raw_namespace);\n extern const char *get_git_namespace(void);\n extern const char *strip_namespace(const char *namespaced_ref);\n extern const char *get_super_prefix(void);\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10..97dfa8c 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -123,7 +123,7 @@ const char * const local_repo_env[] = {\n \tNULL\n };\n \n-static char *expand_namespace(const char *raw_namespace)\n+char *expand_namespace(const char *raw_namespace)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct strbuf **components, **c;\n-- \n2.9.0\n\n"},{"id":"326251","messageId":"20170813193611.4233-3-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 2/7] Add git_configset_add_standard","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:06Z","receivedAt":"2017-08-13T19:36:34Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"This exposes implementation details of repo_read_config\nso a configset can be populated with a repository's standard config,\nwithout needing to create a full repository.\n\nThis allows config lookup to use the same codepath\nwhether the repository is ready or not,\nwhich allows it to be used during git init.\n---\n config.c | 7 ++++++-\n config.h | 1 +\n 2 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/config.c b/config.c\nindex 231f9a7..d0af812 100644\n--- a/config.c\n+++ b/config.c\n@@ -1764,6 +1764,11 @@ int git_configset_add_file(struct config_set *cs, const char *filename)\n \treturn git_config_from_file(config_set_callback, filename, cs);\n }\n \n+int git_configset_add_standard(struct config_set *cs, const struct config_options *opts)\n+{\n+\treturn config_with_options(config_set_callback, cs, NULL, opts);\n+}\n+\n int git_configset_get_value(struct config_set *cs, const char *key, const char **value)\n {\n \tconst struct string_list *values = NULL;\n@@ -1879,7 +1884,7 @@ static void repo_read_config(struct repository *repo)\n \n \tgit_configset_init(repo->config);\n \n-\tif (config_with_options(config_set_callback, repo->config, NULL, &opts) < 0)\n+\tif (git_configset_add_standard(repo->config, &opts) < 0)\n \t\t/*\n \t\t * config_with_options() normally returns only\n \t\t * zero, as most errors are fatal, and\ndiff --git a/config.h b/config.h\nindex 0352da1..0b7b8d2 100644\n--- a/config.h\n+++ b/config.h\n@@ -151,6 +151,7 @@ struct config_set {\n \n extern void git_configset_init(struct config_set *cs);\n extern int git_configset_add_file(struct config_set *cs, const char *filename);\n+extern int git_configset_add_standard(struct config_set *cs, const struct config_options *opts);\n extern int git_configset_get_value(struct config_set *cs, const char *key, const char **value);\n extern const struct string_list *git_configset_get_value_multi(struct config_set *cs, const char *key);\n extern void git_configset_clear(struct config_set *cs);\n-- \n2.9.0\n\n"},{"id":"326252","messageId":"20170813193611.4233-4-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 3/7] Add helper for skipping namespace prefixes","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:07Z","receivedAt":"2017-08-13T19:36:37Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"Normally your own namespace is known and you only need to skip that prefix,\nbut when you need to classify the type of a ref\nit helps to be able to consider what type of ref it would be\nif it were outside of a namespace.\n---\n git-compat-util.h | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex db9c22d..c5e0a34 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -501,6 +501,25 @@ static inline int skip_prefix_mem(const char *buf, size_t len,\n \treturn 0;\n }\n \n+static inline int skip_namespace(const char *refname, const char **out)\n+{\n+\tconst char *c = refname;\n+\n+\twhile (skip_prefix(c, \"refs/namespaces/\", &c)) {\n+\t\tc = strchr(c, '/');\n+\t\tif (!c)\n+\t\t\treturn 0;\n+\n+\t\tc++;\n+\t}\n+\n+\tif (c == refname)\n+\t\treturn 0;\n+\n+\t*out = c;\n+\treturn 1;\n+}\n+\n /*\n  * If buf ends with suffix, return 1 and subtract the length of the suffix\n  * from *len. Otherwise, return 0 and leave *len untouched.\n-- \n2.9.0\n\n"},{"id":"326253","messageId":"20170813193611.4233-5-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 4/7] Autocreate reflogs for namespaced refs","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:08Z","receivedAt":"2017-08-13T19:36:38Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"Since refs are classified based on their prefix\nbut namespaces have their own prefix,\nit's necessary to skip that prefix to classify their remaining prefix.\n---\n refs.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/refs.c b/refs.c\nindex ba22f4a..2e8bace 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -699,6 +699,7 @@ int should_autocreate_reflog(const char *refname)\n \tcase LOG_REFS_ALWAYS:\n \t\treturn 1;\n \tcase LOG_REFS_NORMAL:\n+\t\t(void) skip_namespace(refname, &refname);\n \t\treturn starts_with(refname, \"refs/heads/\") ||\n \t\t\tstarts_with(refname, \"refs/remotes/\") ||\n \t\t\tstarts_with(refname, \"refs/notes/\") ||\n-- \n2.9.0\n\n"},{"id":"326254","messageId":"20170813193611.4233-6-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 5/7] Treat namespaced HEAD and refs/bisect as per-worktree","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:09Z","receivedAt":"2017-08-13T19:36:44Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"---\n refs.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/refs.c b/refs.c\nindex 2e8bace..9a3dcfb 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -536,6 +536,7 @@ int dwim_log(const char *str, int len, unsigned char *sha1, char **log)\n \n static int is_per_worktree_ref(const char *refname)\n {\n+\t(void) skip_namespace(refname, &refname);\n \treturn !strcmp(refname, \"HEAD\") ||\n \t\tstarts_with(refname, \"refs/bisect/\");\n }\n-- \n2.9.0\n\n"},{"id":"326255","messageId":"20170813193611.4233-7-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 6/7] Add namespaced ref backend","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:10Z","receivedAt":"2017-08-13T19:36:46Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"---\n Makefile                  |   1 +\n refs/namespaced-backend.c | 619 ++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h      |   1 +\n 3 files changed, 621 insertions(+)\n create mode 100644 refs/namespaced-backend.c\n\ndiff --git a/Makefile b/Makefile\nindex 461c845..0c417c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -842,6 +842,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/files-backend.o\n LIB_OBJS += refs/iterator.o\n+LIB_OBJS += refs/namespaced-backend.o\n LIB_OBJS += refs/ref-cache.o\n LIB_OBJS += ref-filter.o\n LIB_OBJS += remote.o\ndiff --git a/refs/namespaced-backend.c b/refs/namespaced-backend.c\nnew file mode 100644\nindex 0000000..bcea2ca\n--- /dev/null\n+++ b/refs/namespaced-backend.c\n@@ -0,0 +1,619 @@\n+\n+#include \"../cache.h\"\n+#include \"../config.h\"\n+#include \"../refs.h\"\n+#include \"refs-internal.h\"\n+#include \"../repository.h\"\n+#include \"../iterator.h\"\n+\n+/* Namespace backend intended to stack on top of existing ref store */\n+\n+extern struct ref_storage_be refs_be_namespaced;\n+\n+struct namespaced_ref_store {\n+\tstruct ref_store base;\n+\tstruct ref_store *lower;\n+\tchar *prefix;\n+};\n+\n+static struct namespaced_ref_store *namespaced_downcast(\n+\t\tstruct ref_store *ref_store, const char *caller)\n+{\n+\tstruct namespaced_ref_store *refs;\n+\n+\tif (ref_store->be != &refs_be_namespaced)\n+\t\tdie(\"BUG: ref_store is type \\\"%s\\\" not \\\"namespaced\\\" in %s\",\n+\t\t    ref_store->be->name, caller);\n+\n+\trefs = (struct namespaced_ref_store *)ref_store;\n+\n+\treturn refs;\n+}\n+\n+int namespaced_ref_store_create(const char *gitdir,\n+                                struct ref_store **lower)\n+{\n+\tstruct namespaced_ref_store *refs = NULL;\n+\tstruct ref_store *ref_store;\n+\tchar *config = NULL, *prefix = NULL;\n+\tint ret;\n+\tstruct config_options opts;\n+\tstruct config_set cs;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tret = get_common_dir(&sb, gitdir);\n+\tif (ret < 0) {\n+\t\tgoto cleanup;\n+\t}\n+\n+\topts.respect_includes = 1;\n+\topts.commondir =  sb.buf;\n+\topts.git_dir = gitdir;\n+\tmemset(&cs, 0, sizeof(cs));\n+\tgit_configset_init(&cs);\n+\n+\tret = git_configset_add_standard(&cs, &opts);\n+\tif (ret < 0) {\n+\t\tgoto cleanup;\n+\t}\n+\n+\tret = git_configset_get_string(&cs, \"core.namespace\", &config);\n+\tif (ret != 0) {\n+\t\tgoto cleanup;\n+\t}\n+\n+\tprefix = expand_namespace(config);\n+\tassert(prefix);\n+\n+\trefs = xcalloc(1, sizeof(*refs));\n+\tref_store = &refs->base;\n+\trefs->prefix = prefix;\n+\tprefix = NULL;\n+\n+\tbase_ref_store_init(ref_store, &refs_be_namespaced);\n+\n+\trefs->lower = *lower;\n+\t*lower = ref_store;\n+\trefs = NULL;\n+\n+cleanup:\n+\tfree(refs);\n+\tgit_configset_clear(&cs);\n+\tfree(prefix);\n+\tfree(config);\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+\n+static void prepend_prefix(struct ref_transaction *transaction,\n+                           const char *prefix)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tsize_t prefixlen;\n+\tint i;\n+\n+\tstrbuf_addstr(&sb, prefix);\n+\tprefixlen = sb.len;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *oldupdate, *newupdate;\n+\n+\t\toldupdate = transaction->updates[i];\n+\n+\t\tif (ref_type(oldupdate->refname) == REF_TYPE_PSEUDOREF)\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_addstr(&sb, oldupdate->refname);\n+\t\tFLEX_ALLOC_STR(newupdate, refname, sb.buf);\n+\t\tmemcpy(newupdate, oldupdate,\n+\t\t       ((char*)&newupdate->refname) - (char*)newupdate);\n+\t\ttransaction->updates[i] = newupdate;\n+\t\tfree(oldupdate);\n+\t\tstrbuf_setlen(&sb, prefixlen);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n+\n+static char *add_namespace(\n+\t\tstruct strbuf *sb, struct namespaced_ref_store *refs,\n+\t\tconst char *refname)\n+{\n+\tif (ref_type(refname) != REF_TYPE_PSEUDOREF)\n+\t\tstrbuf_addstr(sb, refs->prefix);\n+\tif (refname)\n+\t\tstrbuf_addstr(sb, refname);\n+\treturn sb->buf;\n+}\n+\n+static int namespaced_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"init_db\");\n+\tint ret;\n+\t\n+\tret = refs->lower->be->init_db(refs->lower, err);\n+\tif (ret != 0)\n+\t\treturn ret;\n+\n+\t/* TODO: Needs to add an un-namespaced HEAD symlink,\n+\t         is_git_directory assumes it's not a repo without it.\n+\t         Can't change is_git_directory to resolve ref via backend\n+\t         since it would need to make the backend and can't free it. */\n+\tret = refs->lower->be->create_symref(\n+\t\t\trefs->lower, \"HEAD\", \"refs/heads/master\", NULL);\n+\tif (ret != 0)\n+\t\treturn ret;\n+\n+\tret = refs_create_symref(ref_store, \"HEAD\", \"refs/heads/master\", NULL);\n+\n+\treturn ret;\n+}\n+\n+static int namespaced_transaction_prepare(struct ref_store *ref_store,\n+                                          struct ref_transaction *transaction,\n+                                          struct strbuf *err)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"transaction_prepare\");\n+\tint ret;\n+\n+\tprepend_prefix(transaction, refs->prefix);\n+\n+\tret = refs->lower->be->transaction_prepare(refs->lower, transaction,\n+\t                                           err);\n+\n+\t/* TODO: Fix missing ref update for namespaced HEAD */\n+\n+\treturn ret;\n+}\n+static int namespaced_transaction_finish(struct ref_store *ref_store,\n+                                         struct ref_transaction *transaction,\n+                                         struct strbuf *err)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"transaction_finish\");\n+\treturn refs->lower->be->transaction_finish(refs->lower, transaction,\n+\t                                           err);\n+}\n+static int namespaced_transaction_abort(struct ref_store *ref_store,\n+                                        struct ref_transaction *transaction,\n+                                        struct strbuf *err)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"transaction_abort\");\n+\treturn refs->lower->be->transaction_abort(refs->lower, transaction,\n+\t                                          err);\n+}\n+static int namespaced_initial_transaction_commit(\n+\t\tstruct ref_store *ref_store,\n+\t\tstruct ref_transaction *transaction, struct strbuf *err)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"initial_transaction_commit\");\n+\tint ret;\n+\n+\tprepend_prefix(transaction, refs->prefix);\n+\n+\tret = refs->lower->be->initial_transaction_commit(\n+\t\t\trefs->lower, transaction, err);\n+\n+\t/* TODO: Fix missing ref update for namespaced HEAD */\n+\n+\treturn ret;\n+}\n+static int namespaced_pack_refs(struct ref_store *ref_store, unsigned int flags)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"pack_refs\");\n+\treturn refs->lower->be->pack_refs(refs->lower, flags);\n+}\n+static int namespaced_peel_ref(struct ref_store *ref_store, const char *refname,\n+                               unsigned char *sha1)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"peel_ref\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->peel_ref(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname), sha1);\n+\t\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static int namespaced_create_symref(struct ref_store *ref_store,\n+                                    const char *refname,\n+                                    const char *target,\n+                                    const char *logmsg)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"create_symref\");\n+\tstruct strbuf rsb = STRBUF_INIT, tsb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->create_symref(\n+\t\t\trefs->lower, add_namespace(&rsb, refs, refname),\n+\t\t\tadd_namespace(&tsb, refs, target), logmsg);\n+\t\n+\tstrbuf_release(&rsb);\n+\tstrbuf_release(&tsb);\n+\n+\treturn ret;\n+}\n+static int namespaced_delete_refs(struct ref_store *ref_store, const char *msg,\n+                                  struct string_list *refnames,\n+                                  unsigned int flags)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"delete_refs\");\n+\tstruct string_list prefixed = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *it;\n+\tint ret;\n+\n+\tfor_each_string_list_item(it, refnames) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t/* TODO: Pseudorefs aren't namespaced,\n+\t\t         so add_namespace may do nothing but strdup */\n+\t\tadd_namespace(&sb, refs, it->string);\n+\t\tstring_list_append_nodup(&prefixed, strbuf_detach(&sb, NULL));\n+\n+\t\tstrbuf_release(&sb);\n+\t}\n+\t\n+\tret = refs->lower->be->delete_refs(refs->lower, msg, &prefixed, flags);\n+\n+\tstring_list_clear(&prefixed, 1);\n+\n+\treturn ret;\n+}\n+static int namespaced_rename_ref(struct ref_store *ref_store,\n+                                 const char *oldref, const char *newref,\n+                                 const char *logmsg)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"rename_ref\");\n+\tstruct strbuf osb = STRBUF_INIT, nsb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->rename_ref(\n+\t\t\trefs->lower, add_namespace(&osb, refs, oldref),\n+\t\t\tadd_namespace(&nsb, refs, newref), logmsg);\n+\t\n+\tstrbuf_release(&osb);\n+\tstrbuf_release(&nsb);\n+\n+\treturn ret;\n+}\n+\n+extern struct ref_iterator_vtable namespaced_ref_iterator_vtable;\n+\n+struct namespaced_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct ref_iterator *lower;\n+\tconst char *prefix;\n+};\n+\n+static struct namespaced_ref_iterator *nsiter_downcast(\n+\t\tstruct ref_iterator *ref_iterator, const char *caller)\n+{\n+\tstruct namespaced_ref_iterator *iter;\n+\n+\tif (ref_iterator->vtable != &namespaced_ref_iterator_vtable)\n+\t\tdie(\"BUG: ref_iterator is not \\\"namespaced\\\" in %s\",\n+\t\t    caller);\n+\n+\titer = (struct namespaced_ref_iterator *)ref_iterator;\n+\n+\treturn iter;\n+}\n+static int namespaced_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct namespaced_ref_iterator *iter = nsiter_downcast(ref_iterator,\n+\t                                                       \"advance\");\n+\tint ret;\n+\n+\twhile ((ret = iter->lower->vtable->advance(iter->lower)) == ITER_OK) {\n+\t\t/* Pseudorefs are not namespaced */\n+\t\tif (ref_type(iter->lower->refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\titer->base.oid = iter->lower->oid;\n+\t\t\titer->base.flags = iter->lower->flags;\n+\t\t\titer->base.refname = iter->lower->refname;\n+\t\t\treturn ITER_OK;\n+\t\t}\n+\t\t    \n+\t\t/* Standard ref iterator is pre-filtered,\n+\t\t   but the same function is used for reflogs\n+\t\t   which has no pre-filtering. */\n+\t\tif (!starts_with(iter->lower->refname, iter->prefix))\n+\t\t\tcontinue;\n+\n+\t\titer->base.oid = iter->lower->oid;\n+\t\titer->base.flags = iter->lower->flags;\n+\t\tassert(skip_prefix(iter->lower->refname, iter->prefix,\n+\t\t                   &iter->base.refname));\n+\t\treturn ITER_OK;\n+\t}\n+\n+\titer->lower = NULL;\n+\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\tret = ITER_ERROR;\n+\n+\treturn ret;\n+}\n+static int namespaced_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+                                        struct object_id *peeled)\n+{\n+\tstruct namespaced_ref_iterator *iter = nsiter_downcast(ref_iterator,\n+\t                                                       \"peel\");\n+\treturn iter->lower->vtable->peel(iter->lower, peeled);\n+}\n+static int namespaced_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct namespaced_ref_iterator *iter = nsiter_downcast(ref_iterator,\n+\t                                                       \"abort\");\n+\tint ret = ITER_DONE;\n+\tif (iter->lower)\n+\t\tret = iter->lower->vtable->abort(iter->lower);\n+\n+\tbase_ref_iterator_free(ref_iterator);\n+\treturn ret;\n+}\n+struct ref_iterator_vtable namespaced_ref_iterator_vtable = {\n+\tnamespaced_ref_iterator_advance,\n+\tnamespaced_ref_iterator_peel,\n+\tnamespaced_ref_iterator_abort,\n+};\n+\n+static struct ref_iterator *make_namespaced_iterator(\n+\t\tstruct ref_iterator *lower,\n+\t\tconst char *prefix)\n+{\n+\tstruct ref_iterator *ref_iterator;\n+\tstruct namespaced_ref_iterator *iter;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tref_iterator = &iter->base;\n+\tbase_ref_iterator_init(ref_iterator, &namespaced_ref_iterator_vtable);\n+\n+\titer->lower = lower;\n+\titer->prefix = prefix;\n+\n+\treturn ref_iterator;\n+}\n+static struct ref_iterator *namespaced_iterator_begin(\n+\t\tstruct ref_store *ref_store, const char *prefix,\n+\t\tunsigned int flags)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"iterator_begin\");\n+\tstruct ref_iterator *lower, *ret;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t/* TODO: Pseudorefs aren't namespaced, but appear in the namespace.\n+\t         Pseudorefs can only appear if there's no prefix or it's \"\"\n+\t         at which point stripping the namespace off and filtering works,\n+\t         so if we've got a prefix we prepend the namespace,\n+\t         and if we don't we filter post-hoc.\n+\t*/\n+\tif (prefix && *prefix)\n+\t\tprefix = add_namespace(&sb, refs, prefix);\n+\n+\tlower = refs->lower->be->iterator_begin(\n+\t\t\trefs->lower, prefix, flags);\n+\n+\tret = make_namespaced_iterator(lower, refs->prefix);\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static int namespaced_read_raw_ref(struct ref_store *ref_store,\n+                                   const char *refname, unsigned char *sha1,\n+                                   struct strbuf *referent, unsigned int *type)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"read_raw_ref\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->read_raw_ref(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname), sha1,\n+\t\t\treferent, type);\n+\n+\tif (ret == 0 && (*type & REF_ISSYMREF) == REF_ISSYMREF) {\n+\t\tconst char *stripped_ref;\n+\t\tif (skip_prefix(referent->buf, refs->prefix, &stripped_ref)) {\n+\t\t\tstruct strbuf stripped = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&stripped, stripped_ref);\n+\t\t\tstrbuf_swap(referent, &stripped);\n+\t\t\tstrbuf_release(&stripped);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static struct ref_iterator *namespaced_reflog_iterator_begin(\n+\t\tstruct ref_store *ref_store)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"reflog_iterator_begin\");\n+\tstruct ref_iterator *lower;\n+\n+\tlower = refs->lower->be->reflog_iterator_begin(\n+\t\t\trefs->lower);\n+\n+\treturn make_namespaced_iterator(lower, refs->prefix);\n+}\n+\n+static int namespaced_for_each_reflog_ent(struct ref_store *ref_store,\n+                                          const char *refname,\n+                                          each_reflog_ent_fn fn,\n+                                          void *cb_data)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"for_each_reflog_ent\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->for_each_reflog_ent(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname),\n+\t\t\tfn, cb_data);\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static int namespaced_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n+                                                  const char *refname,\n+                                                  each_reflog_ent_fn fn,\n+                                                  void *cb_data)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"for_each_reflog_ent_reverse\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->for_each_reflog_ent_reverse(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname),\n+\t\t\tfn, cb_data);\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static int namespaced_reflog_exists(struct ref_store *ref_store,\n+                                    const char *refname)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"reflog_exists\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->reflog_exists(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname));\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static int namespaced_create_reflog(struct ref_store *ref_store,\n+                                    const char *refname, int force_create,\n+                                    struct strbuf *err)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"create_reflog\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->create_reflog(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname),\n+\t\t\tforce_create, err);\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+static int namespaced_delete_reflog(struct ref_store *ref_store,\n+                                    const char *refname)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"delete_reflog\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = refs->lower->be->delete_reflog(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname));\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+\n+struct wrap_expiry_cb_data {\n+\tchar *prefix;\n+\treflog_expiry_prepare_fn *prepare_fn;\n+\treflog_expiry_should_prune_fn *should_prune_fn;\n+\treflog_expiry_cleanup_fn *cleanup_fn;\n+\tvoid *cb_data;\n+};\n+void wrap_expiry_prepare(const char *refname, const struct object_id *oid,\n+                         void *cb_data)\n+{\n+\tstruct wrap_expiry_cb_data *cbd = cb_data;\n+\tskip_prefix(refname, cbd->prefix, &refname);\n+\tcbd->prepare_fn(refname, oid, cbd->cb_data);\n+}\n+int wrap_expiry_should_prune(struct object_id *ooid, struct object_id *noid,\n+                             const char *email, timestamp_t timestamp, int tz,\n+                             const char *message, void *cb_data)\n+{\n+\tstruct wrap_expiry_cb_data *cbd = cb_data;\n+\treturn cbd->should_prune_fn(ooid, noid, email, timestamp, tz,\n+\t                            message, cbd->cb_data);\n+}\n+void wrap_expiry_cleanup(void *cb_data)\n+{\n+\tstruct wrap_expiry_cb_data *cbd = cb_data;\n+\tcbd->cleanup_fn(cbd->cb_data);\n+}\n+static int namespaced_reflog_expire(\n+\t\tstruct ref_store *ref_store, const char *refname,\n+\t\tconst unsigned char *sha1, unsigned int flags,\n+\t\treflog_expiry_prepare_fn prepare_fn,\n+\t\treflog_expiry_should_prune_fn should_prune_fn,\n+\t\treflog_expiry_cleanup_fn cleanup_fn,\n+\t\tvoid *policy_cb_data)\n+{\n+\tstruct namespaced_ref_store *refs = namespaced_downcast(\n+\t\t\tref_store, \"reflog_expire\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct wrap_expiry_cb_data cbd = {\n+\t\trefs->prefix,\n+\t\tprepare_fn,\n+\t\tshould_prune_fn,\n+\t\tcleanup_fn,\n+\t\tpolicy_cb_data,\n+\t};\n+\tint ret;\n+\n+\tret = refs->lower->be->reflog_expire(\n+\t\t\trefs->lower, add_namespace(&sb, refs, refname), sha1,\n+\t\t\tflags, wrap_expiry_prepare, wrap_expiry_should_prune,\n+\t\t\twrap_expiry_cleanup, &cbd);\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+\n+struct ref_storage_be refs_be_namespaced = {\n+\tNULL,\n+\t\"namespaced\",\n+\tNULL, /* Initialised by namespaced_ref_store_create with different API */\n+\tnamespaced_init_db,\n+\tnamespaced_transaction_prepare,\n+\tnamespaced_transaction_finish,\n+\tnamespaced_transaction_abort,\n+\tnamespaced_initial_transaction_commit,\n+\tnamespaced_pack_refs,\n+\tnamespaced_peel_ref,\n+\tnamespaced_create_symref,\n+\tnamespaced_delete_refs,\n+\tnamespaced_rename_ref,\n+\tnamespaced_iterator_begin,\n+\tnamespaced_read_raw_ref,\n+\tnamespaced_reflog_iterator_begin,\n+\tnamespaced_for_each_reflog_ent,\n+\tnamespaced_for_each_reflog_ent_reverse,\n+\tnamespaced_reflog_exists,\n+\tnamespaced_create_reflog,\n+\tnamespaced_delete_reflog,\n+\tnamespaced_reflog_expire,\n+};\n+\n+\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 192f9f8..8391eda 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -655,6 +655,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+int namespaced_ref_store_create(const char *gitdir, struct ref_store **lower);\n \n /*\n  * A representation of the reference store for the main repository or\n-- \n2.9.0\n\n"},{"id":"326256","messageId":"20170813193611.4233-8-richard.maw@gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"[PATCH 7/7] Plumb in namespaced ref store","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-13T19:36:11Z","receivedAt":"2017-08-13T19:36:49Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"---\n refs.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9a3dcfb..e80244f 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -647,7 +647,6 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,\n \tstruct strbuf err = STRBUF_INIT;\n \n \tif (ref_type(refname) == REF_TYPE_PSEUDOREF) {\n-\t\tassert(refs == get_main_ref_store());\n \t\treturn delete_pseudoref(refname, old_sha1);\n \t}\n \n@@ -986,7 +985,6 @@ int refs_update_ref(struct ref_store *refs, const char *msg,\n \tint ret = 0;\n \n \tif (ref_type(refname) == REF_TYPE_PSEUDOREF) {\n-\t\tassert(refs == get_main_ref_store());\n \t\tret = write_pseudoref(refname, new_sha1, old_sha1, &err);\n \t} else {\n \t\tt = ref_store_transaction_begin(refs, &err);\n@@ -1589,6 +1587,9 @@ static struct ref_store *ref_store_init(const char *gitdir,\n \t\tdie(\"BUG: reference backend %s is unknown\", be_name);\n \n \trefs = be->init(gitdir, flags);\n+\n+\tnamespaced_ref_store_create(gitdir, &refs);\n+\n \treturn refs;\n }\n \n-- \n2.9.0\n\n"},{"id":"326279","messageId":"CAGZ79kZz16kwwMOxoi+6UksT2CaPBU5O+r3QzxfrUBf8c3whhg@mail.gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-08-14T17:00:09Z","receivedAt":"2017-08-14T17:00:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> Technical description\n> =====================\n>\n> This patch series adds a new refs backend, stacking on top of the files backend,\n> based on whether `core.namespace` is set in git config.\n\nCurrently there is another Big Thing getting started in in the refs backend.\nhttps://public-inbox.org/git/CAJo=hJtg0PAVHT1phbArdra8+4LfnEEuaj3fBid==BXkZghi8g@mail.gmail.com/\nMaybe it is worth looking into that as well? Reftables solve\nthe problem of scaling (i.e. a repo containing a million refs works\njust fine, and fast(!) for reading/writing refs, which is not the case\nfor packed refs)\n"},{"id":"326413","messageId":"xmqqefscsqj1.fsf@gitster.mtv.corp.google.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-15T17:13:22Z","receivedAt":"2017-08-15T17:13:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Richard Maw <richard.maw@gmail.com> writes:\n\n> This is not my first attempt to improve the git namespace handling in git.\n> I tried last year, but it took me so long that all the ref handling code changed\n> and I would have had to start from scratch.\n>\n> Fortunately the pluggable ref backends work provided an easier starting point.\n\nYeah, I also made an ultra-brief foray into ref backends a few weeks\nago, and found that Michael did an excellent job identifying the\nbuilding blocks backends may want to implement differently and\nabstracting out major parts of the ref processing.  I also hit some\nof the same issues you mention, e.g. \"HEAD\" and other funny refs.\n\nI do suspect that the current GIT_NAMESPACE thing may have outlived\nits usefulness and with the pluggable ref backend thing in place, we\nmay want to redesign how support for multiple views into the same\nrepository is done.  I do not have a need for such a thing myself,\nbut I am glad somebody is looking into it ;-)\n\n"},{"id":"327122","messageId":"20170824102016.GA2021@mimir","threadId":"46582","inReplyTo":"xmqqefscsqj1.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-24T10:20:16Z","receivedAt":"2017-08-24T10:22:12Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"On Tue, Aug 15, 2017 at 10:13:22AM -0700, Junio C Hamano wrote:\n> Richard Maw <richard.maw@gmail.com> writes:\n> \n> > This is not my first attempt to improve the git namespace handling in git.\n> > I tried last year, but it took me so long that all the ref handling code changed\n> > and I would have had to start from scratch.\n> >\n> > Fortunately the pluggable ref backends work provided an easier starting point.\n> \n> Yeah, I also made an ultra-brief foray into ref backends a few weeks\n> ago, and found that Michael did an excellent job identifying the\n> building blocks backends may want to implement differently and\n> abstracting out major parts of the ref processing.  I also hit some\n> of the same issues you mention, e.g. \"HEAD\" and other funny refs.\n> \n> I do suspect that the current GIT_NAMESPACE thing may have outlived\n> its usefulness and with the pluggable ref backend thing in place, we\n> may want to redesign how support for multiple views into the same\n> repository is done.  I do not have a need for such a thing myself,\n> but I am glad somebody is looking into it ;-)\n\nIt was great to be able to get something mostly working this time around,\nwhich would not have been possible without the pluggable ref backends.\n\nI've no intention of giving up just yet,\nthough it'll be a while before I can devote significant time to it.\n\nI'll be keeping an eye on the refdb backend.\nIf it in the process fixes the issues I'd been having that'd be so much better\nsince I've not got the time or community standing to champion big changes.\n"},{"id":"327136","messageId":"CAMy9T_HNGEv+HX7Zyj+g9i=vmrB+hTx7NJdd=FVatZRUv1b-yQ@mail.gmail.com","threadId":"46582","inReplyTo":"20170813193611.4233-1-richard.maw@gmail.com","subject":"Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2017-08-24T16:17:07Z","receivedAt":"2017-08-24T16:17:15Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On Sun, Aug 13, 2017 at 9:36 PM, Richard Maw <richard.maw@gmail.com> wrote:\n> Forewarning: I don't consider this work complete\n> and am unlikely to find time to finish it any time soon.\n> I've mostly sent this because it may include valuable feedback\n> on how well the ref storage backends works\n> from trying to use it to change how git namespaces work.\n>\n> Introduction\n> ============\n>\n> I work on a git server called Gitano,\n> and I'd like to add support for git namespaces to: [...]\n\nThanks so much for your efforts and your description of the problems\nthat you faced. That will be really valuable for whomever might follow\nup on your work (even if it is you :-) ).\n\n> Unfortunately namespace handling was never implemented for any other part of git\n> and at least gitolite makes use of namespaces,\n> and will have to work around it not being implemented fully,\n> but implementing it more fully will break work-arounds.\n\nI agree that the current namespace feature is not a great foundation\nfor future work.\n\n> [...]\n> Fortunately the pluggable ref backends work provided an easier starting point.\n\n:-) I'm glad my years-long obsession is finally yielding fruit.\n\nFirst a general comment about the approach...\n\nI've always thought that a workable \"Git with namespaces\" would\nprobably look more like git worktrees:\n\n* One main repository holding all of the objects and all of the\nnon-pseudo references.\n\n* One lightweight directory per namespace-view, holding the\n\"core.namespace\" config and the pseudorefs. HEAD should probably be\nstored in the main repository (?). Both the main repository and the\nnamespace-view directories would probably be bare, though perhaps\nsomebody can think of an application for allowing non-bare\nrepositories.\n\nEven though this scheme implies the need for extra directories, I\nthink that it would make it easier to fix a lot of your problems:\n\n* Each namespace-view could define its own namespace,\nquasi-permanently. You wouldn't have to pass it via the environment.\n(You might even want to *forbid* changing the namespace via the\nenvironment or command line!) So fetches and pushes from one namespace\nto another would work correctly.\n\n* There would be one place for each namespace-view's pseudorefs. You\nwouldn't have to squeeze them into a single reference tree.\n\n* The main repository would know all of the references of all of the\nnamespace views, so maintenance tools like `git gc` would work without\nchanges.\n\n* The namespace-view directories wouldn't be mistakable for full\nrepositories, so tools like `git gc` would refuse to run in them. I\nthink this would also make it a little bit harder for reference values\nto \"leak\" from one namespace-view to another.\n\n* Remote access could use different paths for different\nnamespace-views. The externally-visible path need not, of course, be\nthe same as the path of the namespace-view's directory.\n\nBy the way, there are certainly still places in the code that don't go\nthrough the refs API (e.g., the one that Junio found). That's because\nthe refs API has still not been used for anything very interesting, so\nthe bugs haven't been flushed out. I see you've found some more.\nThat's because you're doing something interesting :-)\n\n> [...]\n> Bugs\n> ----\n>\n> Most boil down to how special refs like HEAD are handled.\n>\n> 1.  Logged messages display the namespaced path,\n>     which a human may deal with but confuses the test suite.\n\nI think it's clear that the logged messages should reflect the shorter\nreference names, and it is the test suite that needs to be fixed.\n\n> 2.  Reflogs for namespaced HEAD are not updated.\n>\n>     This is because resolving HEAD to split the transaction's updates\n>     to add a log only update to HEAD works by transaction_prepare resolving HEAD\n>     using its own ref store rather than the main one,\n>     so the namespace translation isn't performed.\n>     See split_head_update.\n>\n>     The fix for this may be to move the transaction mangling out of the backend,\n>     unless it should be implied that every backend implementation\n>     must be responsible for symbolic ref reflog updates implicitly.\n\nIt probably makes sense for the namespace layer to do this step.\n\nI think there is a similar problem with `split_symref_update()`. Here\nthe problem is trickier, because you don't know how to split the\nupdate until you have locked the symref, but the locking necessarily\nhas to happen in the main-repo backend. So I think there will be\nplaces where the main-repo backend needs to call back to the namespace\nlayer for some things, like deciding what reference names to use in\nerror messages and things.\n\nYou'd also want to prevent actions in a namespace-view from affecting\nreferences outside of that namespace. For example, you shouldn't be\nable to follow a symref from a namespace-view ref to another reference\nin a different namespace. This also implies some cooperation between\nthe file-level backend and the namespace layer.\n\nI guess it is also clear that symrefs on disk have to contain the full\nreference names of their targets, but when they are read via a\nnamespace-view, the caller should see the short name as the target.\n\n[...]\n\nMichael\n"},{"id":"327397","messageId":"20170830131725.GE2021@mimir","threadId":"46582","inReplyTo":"CAMy9T_HNGEv+HX7Zyj+g9i=vmrB+hTx7NJdd=FVatZRUv1b-yQ@mail.gmail.com","subject":"Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend","fromName":"Richard Maw","fromEmail":"richard.maw@gmail.com","sentAt":"2017-08-30T13:17:25Z","receivedAt":"2017-08-30T13:17:34Z","isPatch":true,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"On Thu, Aug 24, 2017 at 06:17:07PM +0200, Michael Haggerty wrote:\n> On Sun, Aug 13, 2017 at 9:36 PM, Richard Maw <richard.maw@gmail.com> wrote:\n> > [...]\n> > Fortunately the pluggable ref backends work provided an easier starting point.\n> \n> :-) I'm glad my years-long obsession is finally yielding fruit.\n> \n> First a general comment about the approach...\n> \n> I've always thought that a workable \"Git with namespaces\" would\n> probably look more like git worktrees:\n> \n> * One main repository holding all of the objects and all of the\n> non-pseudo references.\n> \n> * One lightweight directory per namespace-view, holding the\n> \"core.namespace\" config and the pseudorefs. HEAD should probably be\n> stored in the main repository (?).\n\nI had pondered something like this as a later extension,\nsince it's an iterable way of finding out which namespaces exist in a repository\nand allows you to explicitly say which namespaces are meant to exist\nsince looking for all (refs/namespaces/[^/]+/).*HEAD\nwill also return HEADs that were created by requesting a symbolic ref creation.\n\nGiven how many of my problems have been because\nnamespaced head isn't located at ${GITDIR}/HEAD\nI'm not sure how this helps.\n\n> Both the main repository and the namespace-view directories would probably be\n> bare, though perhaps somebody can think of an application for allowing\n> non-bare repositories.\n\nThe only application I've come up with so far is:\n\n1.  Your git server stores configuration in git repositories (gitano does).\n2.  The configuration is stored in a git namespace (gitano wants to do this).\n3.  Instead of making a clone of the namespace, making changes and pushing that\n    it would be nicer to create a workspace to make the changes in instead.\n\nIt's not a particularly strong application,\nsince normally you'd be administering by doing a clone remotely,\nand even if that doesn't work you can do a clone locally and push that.\n\nSimilarly you should be able to create a workspace\nand check out the namespace's head yourself.\n\n> Even though this scheme implies the need for extra directories, I\n> think that it would make it easier to fix a lot of your problems:\n> \n> * Each namespace-view could define its own namespace,\n> quasi-permanently. You wouldn't have to pass it via the environment.\n> (You might even want to *forbid* changing the namespace via the\n> environment or command line!) So fetches and pushes from one namespace\n> to another would work correctly.\n> \n> * There would be one place for each namespace-view's pseudorefs. You\n> wouldn't have to squeeze them into a single reference tree.\n\nI'm not sure that un-namespaced pseudorefs are a problem.\nAt least for the git server use-case you wouldn't typically have them.\nIt would be nice for it to just work of course.\n\n> * The main repository would know all of the references of all of the\n> namespace views, so maintenance tools like `git gc` would work without\n> changes.\n\nThe same is/will be true for worktrees from walking the gitdirs\nstored in the commondir.\n\n> * The namespace-view directories wouldn't be mistakable for full\n> repositories, so tools like `git gc` would refuse to run in them. I\n> think this would also make it a little bit harder for reference values\n> to \"leak\" from one namespace-view to another.\n> \n> * Remote access could use different paths for different\n> namespace-views. The externally-visible path need not, of course, be\n> the same as the path of the namespace-view's directory.\n\nThis talk of worktrees got me thinking about an alternative implementation\nwhere instead of mangling refs to add and remove prefixes\na separate git directory is maintained, like worktrees,\nand the namespace handling in the refs backend would be\nto create a files (or reftable) backend for the namespace's git directory.\n\nHaving a separate namespave-view would be like having a bare worktree,\nwhich would presumably be just the gitfile.\n\nThis would require extra tooling to create namespaces\nsince you can't just create a sub-namespace by copying some refs\nand adding a symbolic ref for HEAD.\n\nHowever since you can't (or couldn't last time I tried to find a way to)\npush a symbolic ref, so the git server needs extra tooling anyway.\nYou would lose the ability to trivially see sub-namespaces by fetching refs,\nbut the only use I can think of for that is mirroring for backup.\n\n> By the way, there are certainly still places in the code that don't go\n> through the refs API (e.g., the one that Junio found). That's because\n> the refs API has still not been used for anything very interesting, so\n> the bugs haven't been flushed out. I see you've found some more.\n> That's because you're doing something interesting :-)\n\n> > [...]\n> > Bugs\n> > ----\n> >\n> > Most boil down to how special refs like HEAD are handled.\n> >\n> > 1.  Logged messages display the namespaced path,\n> >     which a human may deal with but confuses the test suite.\n> \n> I think it's clear that the logged messages should reflect the shorter\n> reference names, and it is the test suite that needs to be fixed.\n\nIt's error messages from inside the files backend.\n\nThe namespaced backend gets passed refs with the namespace prepended\nand when it produces a message including the ref name the prefix is added.\n\nI'd not put too much thought into ways of fixing this\nsince if something goes really wrong it's probably more useful\nto know the real ref name.\n\n> > 2.  Reflogs for namespaced HEAD are not updated.\n> >\n> >     This is because resolving HEAD to split the transaction's updates\n> >     to add a log only update to HEAD works by transaction_prepare resolving HEAD\n> >     using its own ref store rather than the main one,\n> >     so the namespace translation isn't performed.\n> >     See split_head_update.\n> >\n> >     The fix for this may be to move the transaction mangling out of the backend,\n> >     unless it should be implied that every backend implementation\n> >     must be responsible for symbolic ref reflog updates implicitly.\n> \n> It probably makes sense for the namespace layer to do this step.\n> \n> I think there is a similar problem with `split_symref_update()`. Here\n> the problem is trickier, because you don't know how to split the\n> update until you have locked the symref, but the locking necessarily\n> has to happen in the main-repo backend. So I think there will be\n> places where the main-repo backend needs to call back to the namespace\n> layer for some things, like deciding what reference names to use in\n> error messages and things.\n\nIt sounds like perhaps struct ref_store needs a pointer to the \"top\" backend\nfor things like refs_resolve_refdup,\nbut also to store a \"parent\" and add a render_ref_fn:\n\n    typedef const char *render_ref_fn(struct ref_store *ref_store,\n                                      const char *refname,\n                                      struct strbuf *result);\n\nThis would return the rendered ref.\nPassive backends are just:\n\n    const char *backend_render_ref(struct ref_store *ref_store,\n                                   const char *refname, struct strbuf *result)\n    {\n        if (ref_store->parent == NULL)\n            return refname;\n        return render_ref(ref_store->parent, refname, result)\n    }\n\nThe namespaced backend would be:\n\n    const char *namespaced_render_ref(struct ref_store *ref_store,\n                                      const char *refname,\n                                      struct strbuf *result)\n    {\n        struct namespaced_ref_store *refs = namespaced_downcast(\n                        ref_store, \"render_ref\");\n\n        if (skip_prefix(refname, refs->prefix, &refname) == 0)\n            return NULL;\n\n        if (ref_store->parent == NULL)\n            return refname;\n\n        return render_ref(ref_store->parent, refname, result)\n    }\n\nIf a backend did anything more weird it could put the result in *result\nand return a pointer into that.\n\n> You'd also want to prevent actions in a namespace-view from affecting\n> references outside of that namespace. For example, you shouldn't be\n> able to follow a symref from a namespace-view ref to another reference\n> in a different namespace. This also implies some cooperation between\n> the file-level backend and the namespace layer.\n\nIf it's created via the namespaced ref backend it can only point within.\nIf it's created from a parent namespace then it may be sufficient\nto catch it at resolve time and treat it as a broken.\n\nThe files backend using REF_ISBROKEN for if a ref resolves to the null OID\nseems like sufficient precedent for using it this way.\n\n> I guess it is also clear that symrefs on disk have to contain the full\n> reference names of their targets, but when they are read via a\n> namespace-view, the caller should see the short name as the target.\n> \n> [...]\n> \n> Michael\n"}]}