{"thread":{"id":"5027","subject":"[PATCH] Allow fetching from multiple repositories at once","startedAt":"2006-07-28T05:44:21Z","lastAt":"2006-07-28T14:51:30Z","messageCount":8,"participants":["Petr Baudis","Junio C Hamano","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"24281","messageId":"20060728054341.15864.35862.stgit@machine","threadId":"5027","inReplyTo":null,"subject":"[PATCH] Allow fetching from multiple repositories at once","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-07-28T05:44:21Z","receivedAt":"2006-07-28T05:44:21Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"This patch enables fetching multiple repositories at once over the Git\nprotocol (and SSH, and locally if git-fetch-pack is your cup of coffee\nthere). This is done especially for the xorg people who have tons of\nrepositories and dislike pulls much slower than they were used to with CVS.\nI'm eager to hear how this affects the situation.\n\nIt's kind of \"superproject\" thing, basically, taking reverse approach than\nthe subproject ideas. However, in practice I think it can be used for\nsubprojects quite well and perhaps if I find some spare time during the day\nI will add the lightweight subproject support to Cogito, using this.\n\nSo, you need some kind of porcelain for this. The idea is that instead of\ntelling git-fetch-pack a single repository, you pass multiple --repo=\nparameters and refs always \"belong\" to the latest mentioned repository;\nwhen outputting the new ref values, the appropriate repo is mentioned near\neach ref. In order for this to be useful, on your local side you should\nshare the objects database in some way - either using alternates (then\nyou must fetch to an object database reachable from everywhere) or symlinked\nobject databases.\n\nYou still need to pass git-fetch-pack some URL in addition to the\nrepositories - it is used only for git_connect(), the purpose is that\nrepositories must be local directories so if you want to talk remote, you\nneed to do something like\n\n\tgit-fetch-pack git://kernel.org/pub/scm/git/git.git --repo=/pub/scm/git/git.git master next --repo=/pub/scm/cogito/cogito.git master\n\nThe implementation is, well... rather hackish to say the least. :-)\nWe can do something nicer if it turns out to be a boost huge enough\nin reality too.\n\nSigned-off-by: Petr Baudis <pasky@suse.cz>\n---\n\n cache.h       |    7 ++++--\n connect.c     |   11 ++++++---\n daemon.c      |    9 +++++---\n fetch-pack.c  |   58 +++++++++++++++++++++++++++++++++----------------\n path.c        |    5 ++++\n peek-remote.c |    4 ++-\n send-pack.c   |    4 ++-\n sha1_file.c   |    6 +++--\n upload-pack.c |   68 +++++++++++++++++++++++++++++++++++++++++++++++++--------\n 9 files changed, 127 insertions(+), 45 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 457d1d0..ea2f5e8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -225,6 +225,7 @@ int git_config_perm(const char *var, con\n int adjust_shared_perm(const char *path);\n int safe_create_leading_directories(char *path);\n char *enter_repo(char *path, int strict);\n+int security_repo_check(int export_all_trees);\n \n /* Read and unpack a sha1 file into memory, write memory to a sha1 file */\n extern int sha1_object_info(const unsigned char *, char *, unsigned long *);\n@@ -298,6 +299,7 @@ extern struct alternate_object_database \n \tchar base[FLEX_ARRAY]; /* more */\n } *alt_odb_list;\n extern void prepare_alt_odb(void);\n+extern int link_alt_odb_entry(const char * entry, int len, const char * relative_base, int depth, int allow_self);\n \n extern struct packed_git {\n \tstruct packed_git *next;\n@@ -325,6 +327,7 @@ struct ref {\n \tunsigned char new_sha1[20];\n \tunsigned char force;\n \tstruct ref *peer_ref; /* when renaming */\n+\tchar *repo; /* when doing multi-repo operations */\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n@@ -334,11 +337,11 @@ #define REF_TAGS\t(1u << 2)\n \n extern int git_connect(int fd[2], char *url, const char *prog);\n extern int finish_connect(pid_t pid);\n-extern int path_match(const char *path, int nr, char **match);\n+extern int path_match(const char *path, const char *repo, int nr, char **match, char **match_repo);\n extern int match_refs(struct ref *src, struct ref *dst, struct ref ***dst_tail,\n \t\t      int nr_refspec, char **refspec, int all);\n extern int get_ack(int fd, unsigned char *result_sha1);\n-extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match, unsigned int flags);\n+extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match, char *repo, unsigned int flags);\n extern int server_supports(const char *feature);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1);\ndiff --git a/connect.c b/connect.c\nindex 4422a0d..63f095c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -45,9 +45,8 @@ static int check_ref(const char *name, i\n  */\n struct ref **get_remote_heads(int in, struct ref **list,\n \t\t\t      int nr_match, char **match,\n-\t\t\t      unsigned int flags)\n+\t\t\t      char *repo, unsigned int flags)\n {\n-\t*list = NULL;\n \tfor (;;) {\n \t\tstruct ref *ref;\n \t\tunsigned char old_sha1[20];\n@@ -74,11 +73,12 @@ struct ref **get_remote_heads(int in, st\n \n \t\tif (!check_ref(name, name_len, flags))\n \t\t\tcontinue;\n-\t\tif (nr_match && !path_match(name, nr_match, match))\n+\t\tif (nr_match && !path_match(name, NULL, nr_match, match, NULL))\n \t\t\tcontinue;\n \t\tref = xcalloc(1, sizeof(*ref) + len - 40);\n \t\tmemcpy(ref->old_sha1, old_sha1, 20);\n \t\tmemcpy(ref->name, buffer + 41, len - 40);\n+\t\tref->repo = repo;\n \t\t*list = ref;\n \t\tlist = &ref->next;\n \t}\n@@ -112,7 +112,8 @@ int get_ack(int fd, unsigned char *resul\n \tdie(\"git-fetch_pack: expected ACK/NAK, got '%s'\", line);\n }\n \n-int path_match(const char *path, int nr, char **match)\n+int path_match(const char *path, const char *repo, int nr, char **match,\n+               char **match_repo)\n {\n \tint i;\n \tint pathlen = strlen(path);\n@@ -121,6 +122,8 @@ int path_match(const char *path, int nr,\n \t\tchar *s = match[i];\n \t\tint len = strlen(s);\n \n+\t\tif (match_repo && repo != match_repo[i])\n+\t\t\tcontinue;\n \t\tif (!len || len > pathlen)\n \t\t\tcontinue;\n \t\tif (memcmp(path + pathlen - len, s, len))\ndiff --git a/daemon.c b/daemon.c\nindex e4ec676..179f2fe 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -233,6 +233,7 @@ static int upload(char *dir)\n {\n \t/* Timeout as string */\n \tchar timeout_buf[64];\n+\tchar checkexport_buf[64];\n \tconst char *path;\n \n \tloginfo(\"Request for '%s'\", dir);\n@@ -250,8 +251,7 @@ static int upload(char *dir)\n \t * path_ok() uses enter_repo() and does whitelist checking.\n \t * We only need to make sure the repository is exported.\n \t */\n-\n-\tif (!export_all_trees && access(\"git-daemon-export-ok\", F_OK)) {\n+\tif (!security_repo_check(export_all_trees)) {\n \t\tlogerror(\"'%s': repository not exported.\", path);\n \t\terrno = EACCES;\n \t\treturn -1;\n@@ -265,8 +265,11 @@ static int upload(char *dir)\n \n \tsnprintf(timeout_buf, sizeof timeout_buf, \"--timeout=%u\", timeout);\n \n+\tstrcpy(checkexport_buf, \"--check-export\");\n+\tif (export_all_trees) checkexport_buf[1] = '\\0'; /* XXX */\n+\n \t/* git-upload-pack only ever reads stuff, so this is safe */\n-\texecl_git_cmd(\"upload-pack\", \"--strict\", timeout_buf, \".\", NULL);\n+\texecl_git_cmd(\"upload-pack\", \"--strict\", timeout_buf, checkexport_buf, \".\", NULL);\n \treturn -1;\n }\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex b7824db..2b02d0f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -281,7 +281,8 @@ static void mark_recent_complete_commits\n \t}\n }\n \n-static void filter_refs(struct ref **refs, int nr_match, char **match)\n+static void filter_refs(struct ref **refs, int nr_match, char **match,\n+                        char **match_repo)\n {\n \tstruct ref **return_refs;\n \tstruct ref *newlist = NULL;\n@@ -312,7 +313,7 @@ static void filter_refs(struct ref **ref\n \t\t\tcontinue;\n \t\t}\n \t\telse {\n-\t\t\tint order = path_match(ref->name, nr_match, match);\n+\t\t\tint order = path_match(ref->name, ref->repo, nr_match, match, match_repo);\n \t\t\tif (order) {\n \t\t\t\treturn_refs[order-1] = ref;\n \t\t\t\tcontinue; /* we will link it later */\n@@ -337,7 +338,8 @@ static void filter_refs(struct ref **ref\n \t*refs = newlist;\n }\n \n-static int everything_local(struct ref **refs, int nr_match, char **match)\n+static int everything_local(struct ref **refs, int nr_match, char **match,\n+                            char **match_repo)\n {\n \tstruct ref *ref;\n \tint retval;\n@@ -386,7 +388,7 @@ static int everything_local(struct ref *\n \t\t}\n \t}\n \n-\tfilter_refs(refs, nr_match, match);\n+\tfilter_refs(refs, nr_match, match, match_repo);\n \n \tfor (retval = 1, ref = *refs; ref ; ref = ref->next) {\n \t\tconst unsigned char *remote = ref->old_sha1;\n@@ -414,13 +416,15 @@ static int everything_local(struct ref *\n \treturn retval;\n }\n \n-static int fetch_pack(int fd[2], int nr_match, char **match)\n+static int fetch_pack(int fd[2], int nr_repo, char **repo,\n+                      int nr_match, char **match, char **match_repo)\n {\n-\tstruct ref *ref;\n+\tstruct ref *ref = NULL, **refn;\n \tunsigned char sha1[20];\n \tint status;\n+\tint i;\n \n-\tget_remote_heads(fd[0], &ref, 0, NULL, 0);\n+\trefn = get_remote_heads(fd[0], &ref, 0, NULL, nr_repo ? repo[0] : NULL, 0);\n \tif (server_supports(\"multi_ack\")) {\n \t\tif (verbose)\n \t\t\tfprintf(stderr, \"Server supports multi_ack\\n\");\n@@ -431,11 +435,19 @@ static int fetch_pack(int fd[2], int nr_\n \t\t\tfprintf(stderr, \"Server supports side-band\\n\");\n \t\tuse_sideband = 1;\n \t}\n+\tif (!server_supports(\"switch-repo\") && nr_repo > 1)\n+\t\tdie(\"server does not support fetching multiple repositories at once\");\n+\n+\tfor (i = 1; i < nr_repo; i++) {\n+\t\tpacket_write(fd[1], \"switch-repo %s\", repo[i]);\n+\t\trefn = get_remote_heads(fd[0], refn, 0, NULL, repo[i], 0);\n+\t}\n+\n \tif (!ref) {\n \t\tpacket_flush(fd[1]);\n \t\tdie(\"no matching remote head\");\n \t}\n-\tif (everything_local(&ref, nr_match, match)) {\n+\tif (everything_local(&ref, nr_match, match, match_repo)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n@@ -456,8 +468,9 @@ static int fetch_pack(int fd[2], int nr_\n \n  all_done:\n \twhile (ref) {\n-\t\tprintf(\"%s %s\\n\",\n-\t\t       sha1_to_hex(ref->old_sha1), ref->name);\n+\t\tprintf(\"%s %s%s%s\\n\",\n+\t\t       sha1_to_hex(ref->old_sha1), ref->name,\n+\t\t       ref->repo ? \" \" : \"\", ref->repo ? : \"\");\n \t\tref = ref->next;\n \t}\n \treturn 0;\n@@ -465,15 +478,16 @@ static int fetch_pack(int fd[2], int nr_\n \n int main(int argc, char **argv)\n {\n-\tint i, ret, nr_heads;\n-\tchar *dest = NULL, **heads;\n+\tint i, ret, nr_repo = 0, nr_heads = 0;\n+\tchar *dest = NULL;\n+\tchar **repo = xmalloc(argc * sizeof(*repo));\n+\tchar **heads = xmalloc(argc * sizeof(*heads));\n+\tchar **heads_repo = xmalloc(argc * sizeof(*heads_repo));\n \tint fd[2];\n \tpid_t pid;\n \n \tsetup_git_directory();\n \n-\tnr_heads = 0;\n-\theads = NULL;\n \tfor (i = 1; i < argc; i++) {\n \t\tchar *arg = argv[i];\n \n@@ -498,16 +512,22 @@ int main(int argc, char **argv)\n \t\t\t\tfetch_all = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strncmp(\"--repo=\", arg, 7)) {\n+\t\t\t\trepo[nr_repo++] = arg + 7;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(\"-v\", arg)) {\n \t\t\t\tverbose = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tusage(fetch_pack_usage);\n \t\t}\n-\t\tdest = arg;\n-\t\theads = argv + i + 1;\n-\t\tnr_heads = argc - i - 1;\n-\t\tbreak;\n+\t\tif (!dest) {\n+\t\t\tdest = arg;\n+\t\t\tcontinue;\n+\t\t}\n+\t\theads_repo[nr_heads] = nr_repo ? repo[nr_repo - 1] : NULL;\n+\t\theads[nr_heads++] = arg;\n \t}\n \tif (!dest)\n \t\tusage(fetch_pack_usage);\n@@ -516,7 +536,7 @@ int main(int argc, char **argv)\n \tpid = git_connect(fd, dest, exec);\n \tif (pid < 0)\n \t\treturn 1;\n-\tret = fetch_pack(fd, nr_heads, heads);\n+\tret = fetch_pack(fd, nr_repo, repo, nr_heads, heads, heads_repo);\n \tclose(fd[0]);\n \tclose(fd[1]);\n \tfinish_connect(pid);\ndiff --git a/path.c b/path.c\nindex db8905f..3021164 100644\n--- a/path.c\n+++ b/path.c\n@@ -243,6 +243,11 @@ char *enter_repo(char *path, int strict)\n \treturn NULL;\n }\n \n+int security_repo_check(int export_all_trees)\n+{\n+\treturn (export_all_trees || access(\"git-daemon-export-ok\", F_OK));\n+}\n+\n int adjust_shared_perm(const char *path)\n {\n \tstruct stat st;\ndiff --git a/peek-remote.c b/peek-remote.c\nindex 2b30980..7d8ae3c 100644\n--- a/peek-remote.c\n+++ b/peek-remote.c\n@@ -9,9 +9,9 @@ static const char *exec = \"git-upload-pa\n \n static int peek_remote(int fd[2], unsigned flags)\n {\n-\tstruct ref *ref;\n+\tstruct ref *ref = NULL;\n \n-\tget_remote_heads(fd[0], &ref, 0, NULL, flags);\n+\tget_remote_heads(fd[0], &ref, 0, NULL, NULL, flags);\n \tpacket_flush(fd[1]);\n \n \twhile (ref) {\ndiff --git a/send-pack.c b/send-pack.c\nindex 10bc8bc..24c47c9 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -232,14 +232,14 @@ static int receive_status(int in)\n \n static int send_pack(int in, int out, int nr_refspec, char **refspec)\n {\n-\tstruct ref *ref;\n+\tstruct ref *ref = NULL;\n \tint new_refs;\n \tint ret = 0;\n \tint ask_for_status_report = 0;\n \tint expect_status_report = 0;\n \n \t/* No funny business with the matcher */\n-\tremote_tail = get_remote_heads(in, &remote_refs, 0, NULL, REF_NORMAL);\n+\tremote_tail = get_remote_heads(in, &remote_refs, 0, NULL, NULL, REF_NORMAL);\n \tget_local_heads();\n \n \t/* Does the other end support the reporting? */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 8f279d8..1ca39e1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -229,7 +229,7 @@ static void read_info_alternates(const c\n  * SHA1, an extra slash for the first level indirection, and the\n  * terminating NUL.\n  */\n-static int link_alt_odb_entry(const char * entry, int len, const char * relative_base, int depth)\n+int link_alt_odb_entry(const char * entry, int len, const char * relative_base, int depth, int allow_self)\n {\n \tstruct stat st;\n \tconst char *objdir = get_object_directory();\n@@ -279,7 +279,7 @@ static int link_alt_odb_entry(const char\n \t\t\treturn -1;\n \t\t}\n \t}\n-\tif (!memcmp(ent->base, objdir, pfxlen)) {\n+\tif (!allow_self && !memcmp(ent->base, objdir, pfxlen)) {\n \t\tfree(ent);\n \t\treturn -1;\n \t}\n@@ -325,7 +325,7 @@ static void link_alt_odb_entries(const c\n \t\t\t\t\t\trelative_base, last);\n \t\t\t} else {\n \t\t\t\tlink_alt_odb_entry(last, cp - last,\n-\t\t\t\t\t\trelative_base, depth);\n+\t\t\t\t\t\trelative_base, depth, 0);\n \t\t\t}\n \t\t}\n \t\twhile (cp < ep && *cp == sep)\ndiff --git a/upload-pack.c b/upload-pack.c\nindex bbd6bd6..3440978 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -9,7 +9,7 @@ #include \"object.h\"\n #include \"commit.h\"\n #include \"exec_cmd.h\"\n \n-static const char upload_pack_usage[] = \"git-upload-pack [--strict] [--timeout=nn] <dir>\";\n+static const char upload_pack_usage[] = \"git-upload-pack [--strict] [--timeout=nn] [--check-export] <dir>\";\n \n #define THEY_HAVE (1U << 0)\n #define OUR_REF (1U << 1)\n@@ -20,6 +20,8 @@ static struct object_array have_obj;\n static struct object_array want_obj;\n static unsigned int timeout = 0;\n static int use_sideband = 0;\n+static int check_export = 0;\n+static int strict = 0;\n \n static void reset_timeout(void)\n {\n@@ -392,7 +394,7 @@ static int get_common_commits(void)\n \t}\n }\n \n-static void receive_needs(void)\n+static char *receive_needs(void)\n {\n \tstatic char line[1000];\n \tint len;\n@@ -403,8 +405,10 @@ static void receive_needs(void)\n \t\tlen = packet_read_line(0, line, sizeof(line));\n \t\treset_timeout();\n \t\tif (!len)\n-\t\t\treturn;\n+\t\t\treturn NULL;\n \n+\t\tif (!strncmp(\"switch-repo \", line, 12))\n+\t\t\treturn line+12;\n \t\tif (strncmp(\"want \", line, 5) ||\n \t\t    get_sha1_hex(line+5, sha1_buf))\n \t\t\tdie(\"git-upload-pack: protocol error, \"\n@@ -436,7 +440,7 @@ static void receive_needs(void)\n \n static int send_ref(const char *refname, const unsigned char *sha1)\n {\n-\tstatic const char *capabilities = \"multi_ack thin-pack side-band\";\n+\tstatic const char *capabilities = \"multi_ack thin-pack side-band switch-repo\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n@@ -461,11 +465,52 @@ static int send_ref(const char *refname,\n \n static int upload_pack(void)\n {\n-\treset_timeout();\n-\thead_ref(send_ref);\n-\tfor_each_ref(send_ref);\n-\tpacket_flush(1);\n-\treceive_needs();\n+\tint multirepo = 0;\n+\n+\twhile (1) {\n+\t\tchar *repo;\n+\t\tchar cwd[PATH_MAX];\n+\n+\t\treset_timeout();\n+\t\thead_ref(send_ref);\n+\t\tfor_each_ref(send_ref);\n+\t\tpacket_flush(1);\n+\t\trepo = receive_needs();\n+\t\tif (!repo)\n+\t\t\tbreak;\n+\t\tmultirepo++;\n+\n+\t\tfprintf(stderr, \"git-upload-pack: switching to repo %s\", repo);\n+\n+\t\t/* So that we still find objects of the original repository... */\n+\t\tgetcwd(cwd, PATH_MAX);\n+\t\tif (strlen(cwd) < PATH_MAX - 8)\n+\t\t\tstrcat(cwd, \"/objects\");\n+\t\tlink_alt_odb_entry(cwd, strlen(cwd), NULL, 0, 1);\n+\n+\t\tif (!enter_repo(repo, strict) || !security_repo_check(!check_export))\n+\t\t\tdie(\"git-upload-pack: security violation\");\n+\t}\n+\n+\tif (multirepo) {\n+#define ALTENV_SIZE 65536\n+\t\t/* Propagate all the repositories to the children */\n+\t\tchar altenv[ALTENV_SIZE], *p = altenv;\n+\t\tstruct alternate_object_database *alt;\n+\t\tstrcpy(p, ALTERNATE_DB_ENVIRONMENT \"=\");\n+\t\tp += sizeof(ALTERNATE_DB_ENVIRONMENT);\n+\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\t\tstrncpy(p, alt->base, alt->name - alt->base);\n+\t\t\tp += alt->name - alt->base;\n+\t\t\tif (p - altenv < ALTENV_SIZE)\n+\t\t\t\t*p++ = ':';\n+\t\t\tif (p - altenv >= ALTENV_SIZE)\n+\t\t\t\tdie(\"fetching too many repositories\");\n+\t\t}\n+\t\tp[-1] = '\\0';\n+\t\tputenv(altenv);\n+\t}\n+\n \tif (!want_obj.nr)\n \t\treturn 0;\n \tget_common_commits();\n@@ -477,7 +522,6 @@ int main(int argc, char **argv)\n {\n \tchar *dir;\n \tint i;\n-\tint strict = 0;\n \n \tfor (i = 1; i < argc; i++) {\n \t\tchar *arg = argv[i];\n@@ -492,6 +536,10 @@ int main(int argc, char **argv)\n \t\t\ttimeout = atoi(arg+10);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--check-export\")) {\n+\t\t\tcheck_export = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--\")) {\n \t\t\ti++;\n \t\t\tbreak;\n"},{"id":"24282","messageId":"20060728054636.GL13776@pasky.or.cz","threadId":"5027","inReplyTo":"20060728054341.15864.35862.stgit@machine","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-07-28T05:46:36Z","receivedAt":"2006-07-28T05:46:36Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Fri, Jul 28, 2006 at 07:44:21AM CEST, I got a letter\nwhere Petr Baudis <pasky@suse.cz> said that...\n> So, you need some kind of porcelain for this. The idea is that instead of\n> telling git-fetch-pack a single repository, you pass multiple --repo=\n> parameters and refs always \"belong\" to the latest mentioned repository;\n> when outputting the new ref values, the appropriate repo is mentioned near\n> each ref. In order for this to be useful, on your local side you should\n> share the objects database in some way - either using alternates (then\n> you must fetch to an object database reachable from everywhere) or symlinked\n> object databases.\n> \n> You still need to pass git-fetch-pack some URL in addition to the\n> repositories - it is used only for git_connect(), the purpose is that\n> repositories must be local directories so if you want to talk remote, you\n> need to do something like\n> \n> \tgit-fetch-pack git://kernel.org/pub/scm/git/git.git --repo=/pub/scm/git/git.git master next --repo=/pub/scm/cogito/cogito.git master\n\nThis is a simple \"porcelain\" for it:\n\n-->8--\n\n#!/usr/bin/perl\nuse warnings;\nuse strict;\n\n# Remember that you must ensure the obj database is shared - either symlink it\n# or setup alternates!\n#my $remoteurl = 'git://git.kernel.org/pub/scm/git/git.git';\n#my %config = (\n#\t'/pub/scm/git/git.git' => {\n#\t\t'next' => {\n#\t\t\t'/home/xpasky/q/gg' => 'origin'\n#\t\t},\n#\t\t'master' => {\n#\t\t\t'/home/xpasky/q/gg' => 'origin2',\n#\t\t\t'/home/xpasky/q/gg2' => 'origin'\n#\t\t}\n#\t},\n#\t'/pub/scm/cogito/cogito-doc.git' => {\n#\t\tmaster => {\n#\t\t\t'/home/xpasky/q/gg2' => 'origin2'\n#\t\t}\n#\t}\n#);\nmy $remoteurl = 'puturlofremotehosthere';\nmy %config = (\n\t'remoterepo1' => {\n\t\tbranch1 => {\n\t\t\t'localrepo1' => 'origin'\n\t\t},\n\t\tbranch2 => {\n\t\t\t'localrepo1' => 'origin2',\n\t\t\t'localrepo2' => 'origin'\n\t\t}\n\t},\n\t'remoterepo2' => {\n\t\tmaster => {\n\t\t\t'localrepo2' => 'origin2'\n\t\t}\n\t}\n);\n\nmy @args = ($remoteurl);\nforeach my $repo (keys %config) {\n\tpush (@args, '--repo='.$repo);\n\tforeach my $branch (keys %{$config{$repo}}) {\n\t\tpush (@args, $branch);\n\t}\n}\n\nopen (F, '-|', 'git-fetch-pack', @args) or die \"$!\";\nwhile (<F>) {\n\tchomp;\n\tsplit / /, $_;\n\tmy ($sha, $ref, $repo) = @_;\n\t$ref =~ s#^refs/heads/##;\n\tforeach my $lrepo (keys %{$config{$repo}->{$ref}}) {\n\t\tsystem(\"GIT_DIR=$lrepo git-update-ref $config{$repo}->{$ref}->{$lrepo} $sha\");\n\t}\n}\nclose (F);\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"},{"id":"24289","messageId":"7v7j1ymbbd.fsf@assigned-by-dhcp.cox.net","threadId":"5027","inReplyTo":"20060728054341.15864.35862.stgit@machine","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-07-28T07:35:18Z","receivedAt":"2006-07-28T07:35:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This is quite heavyweight.\n\nI'll take a look at it eventually ;-), but I'd like to take a\ndeeper look at merge-recursive by Johannes and Alex first.\nPerhaps I can get to it over the weekend, but Saturday is\nalready booked for friends' wedding, so it might take a while.\n"},{"id":"24291","messageId":"Pine.LNX.4.63.0607281045430.29667@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5027","inReplyTo":"20060728054341.15864.35862.stgit@machine","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-07-28T08:51:56Z","receivedAt":"2006-07-28T08:51:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 28 Jul 2006, Petr Baudis wrote:\n\n> This patch enables fetching multiple repositories at once over the Git\n> protocol (and SSH, and locally if git-fetch-pack is your cup of coffee\n> there). This is done especially for the xorg people who have tons of\n> repositories and dislike pulls much slower than they were used to with CVS.\n> I'm eager to hear how this affects the situation.\n\nSo the scenario is: one remote repository (probably shared), and multiple \nlocal repositories, all tracking different branches?\n\nSo, why not setup a single local (master) repository, setup all the other \nrepos with the local master as alternate, and write a simple script which \nfirst fetches all branches into the master, and then pulls into the other \nlocal repos from that master?\n\nThe beauty of it is: you can still pull/push directly from the remote \nrepo, if you want.\n\nCiao,\nDscho\n"},{"id":"24301","messageId":"20060728140058.GM13776@pasky.or.cz","threadId":"5027","inReplyTo":"Pine.LNX.4.63.0607281045430.29667@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-07-28T14:00:58Z","receivedAt":"2006-07-28T14:00:58Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  Hi,\n\nDear diary, on Fri, Jul 28, 2006 at 10:51:56AM CEST, I got a letter\nwhere Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...\n> So the scenario is: one remote repository (probably shared), and multiple \n> local repositories, all tracking different branches?\n> \n> So, why not setup a single local (master) repository, setup all the other \n> repos with the local master as alternate, and write a simple script which \n> first fetches all branches into the master, and then pulls into the other \n> local repos from that master?\n\n  Nope, the scenario is many remote repositories and any number of local\nrepositories. Look at http://gitweb.freedesktop.org/ - as far as I know,\nmany people have many (most?) of those repositories cloned and fetching\nupdates is just a huge pain. This makes no assumptions about the number\nof local repositories, you can even fetch into various branch of a\nsingle local repository - only if you fetch into multiple repositories,\nall of them must see the objects you fetched.\n\n  The alternative of squashing all the remote repositories into a single\none is probably not very attractive since you get a huge branches tree\ninstead, varying hooks will get impractical, it won't look good in\ngitweb, git clone will by default clone all the stuff, you will need to\nimpose tag namespaces, permissions might get tricky and so on and so on.\n\n> The beauty of it is: you can still pull/push directly from the remote \n> repo, if you want.\n\n  This should let you do that too.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"},{"id":"24302","messageId":"20060728140410.GN13776@pasky.or.cz","threadId":"5027","inReplyTo":"20060728054341.15864.35862.stgit@machine","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-07-28T14:04:10Z","receivedAt":"2006-07-28T14:04:10Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Fri, Jul 28, 2006 at 07:44:21AM CEST, I got a letter\nwhere Petr Baudis <pasky@suse.cz> said that...\n> @@ -461,11 +465,52 @@ static int send_ref(const char *refname,\n>  \n>  static int upload_pack(void)\n>  {\n> -\treset_timeout();\n> -\thead_ref(send_ref);\n> -\tfor_each_ref(send_ref);\n> -\tpacket_flush(1);\n> -\treceive_needs();\n> +\tint multirepo = 0;\n> +\n> +\twhile (1) {\n> +\t\tchar *repo;\n> +\t\tchar cwd[PATH_MAX];\n> +\n> +\t\treset_timeout();\n> +\t\thead_ref(send_ref);\n> +\t\tfor_each_ref(send_ref);\n> +\t\tpacket_flush(1);\n> +\t\trepo = receive_needs();\n> +\t\tif (!repo)\n> +\t\t\tbreak;\n> +\t\tmultirepo++;\n> +\n> +\t\tfprintf(stderr, \"git-upload-pack: switching to repo %s\", repo);\n> +\n> +\t\t/* So that we still find objects of the original repository... */\n> +\t\tgetcwd(cwd, PATH_MAX);\n> +\t\tif (strlen(cwd) < PATH_MAX - 8)\n> +\t\t\tstrcat(cwd, \"/objects\");\n> +\t\tlink_alt_odb_entry(cwd, strlen(cwd), NULL, 0, 1);\n> +\n> +\t\tif (!enter_repo(repo, strict) || !security_repo_check(!check_export))\n> +\t\t\tdie(\"git-upload-pack: security violation\");\n> +\t}\n> +\n> +\tif (multirepo) {\n> +#define ALTENV_SIZE 65536\n> +\t\t/* Propagate all the repositories to the children */\n> +\t\tchar altenv[ALTENV_SIZE], *p = altenv;\n> +\t\tstruct alternate_object_database *alt;\n> +\t\tstrcpy(p, ALTERNATE_DB_ENVIRONMENT \"=\");\n> +\t\tp += sizeof(ALTERNATE_DB_ENVIRONMENT);\n> +\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n> +\t\t\tstrncpy(p, alt->base, alt->name - alt->base);\n> +\t\t\tp += alt->name - alt->base;\n> +\t\t\tif (p - altenv < ALTENV_SIZE)\n> +\t\t\t\t*p++ = ':';\n> +\t\t\tif (p - altenv >= ALTENV_SIZE)\n> +\t\t\t\tdie(\"fetching too many repositories\");\n> +\t\t}\n> +\t\tp[-1] = '\\0';\n> +\t\tputenv(altenv);\n> +\t}\n> +\n>  \tif (!want_obj.nr)\n>  \t\treturn 0;\n>  \tget_common_commits();\n\n  Note that you need to be more careful about ALTENV_SIZE checking here,\nand I'm not sure if we even need to abuse the alternates database here;\nonly later I added setting up the alternates variable since I realized\nwe are executing external tools here, and things would be simpler if we\ncould get away by just doing that.\n\n  I'm sorry, I don't have time to send the updated patch anymore. :-(\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"},{"id":"24303","messageId":"Pine.LNX.4.63.0607281608520.29667@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5027","inReplyTo":"20060728140058.GM13776@pasky.or.cz","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-07-28T14:13:28Z","receivedAt":"2006-07-28T14:13:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 28 Jul 2006, Petr Baudis wrote:\n\n> Dear diary, on Fri, Jul 28, 2006 at 10:51:56AM CEST, I got a letter\n> where Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...\n> > So the scenario is: one remote repository (probably shared), and multiple \n> > local repositories, all tracking different branches?\n> > \n> > So, why not setup a single local (master) repository, setup all the other \n> > repos with the local master as alternate, and write a simple script which \n> > first fetches all branches into the master, and then pulls into the other \n> > local repos from that master?\n> \n>   Nope, the scenario is many remote repositories and any number of local\n> repositories.\n\nAh, that clarifies it for me.\n\n> Look at http://gitweb.freedesktop.org/ - as far as I know, many people \n> have many (most?) of those repositories cloned and fetching updates is \n> just a huge pain. This makes no assumptions about the number of local \n> repositories, you can even fetch into various branch of a single local \n> repository - only if you fetch into multiple repositories, all of them \n> must see the objects you fetched.\n> \n>   The alternative of squashing all the remote repositories into a single\n> one is probably not very attractive since you get a huge branches tree\n> instead, varying hooks will get impractical, it won't look good in\n> gitweb, git clone will by default clone all the stuff, you will need to\n> impose tag namespaces, permissions might get tricky and so on and so on.\n\nYou misunderstood me: I was talking about a (hidden) local multiplexer.\n\nBut in the scenario you painted, wouldn't it be practical to have one \nconsolidated repo _in addition_ to the small ones? Of course, the refs in \nthe consolidated one would have to be prefixed with the name of the repo \nthey come from.\n\n> > The beauty of it is: you can still pull/push directly from the remote \n> > repo, if you want.\n> \n>   This should let you do that too.\n\nOf course. But the solution I proposed does not need a patch, and _still_ \nlets you do said operations. Just wanted to point that out.\n\nCiao,\nDscho\n"},{"id":"24304","messageId":"20060728145130.GO13776@pasky.or.cz","threadId":"5027","inReplyTo":"Pine.LNX.4.63.0607281608520.29667@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Allow fetching from multiple repositories at once","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-07-28T14:51:30Z","receivedAt":"2006-07-28T14:51:30Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Hi!\n\nDear diary, on Fri, Jul 28, 2006 at 04:13:28PM CEST, I got a letter\nwhere Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...\n> You misunderstood me: I was talking about a (hidden) local multiplexer.\n\nYes; I just wanted to cover another obvious alternative.  :-)\n\n> But in the scenario you painted, wouldn't it be practical to have one \n> consolidated repo _in addition_ to the small ones? Of course, the refs in \n> the consolidated one would have to be prefixed with the name of the repo \n> they come from.\n\nYou still have problems with tags, for example, and pushing may get\ntricky. It's also non-intuitive for users (I'm seeing some repositories\nin gitweb but fetching from something completely different, and look at\nthose weird things I have to do with the branches, or git will clone\n_all_ the xorg projects at once for me, eeek).\n\nOf course you could hide all that elaborately in the porcelain but I\nthink this turns to be eventually much less-impact solution which is\nsignificantly easier to maintain for the repository admins, intuitive\nfor users and also simpler for porcelains supporting at least\nlight-weight subprojects.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"}]}