{"thread":{"id":"37727","subject":"[PATCH] clone: --dissociate option to mark that reference is only temporary","startedAt":"2014-10-14T19:57:07Z","lastAt":"2014-10-17T12:47:07Z","messageCount":12,"participants":["Junio C Hamano","Marc Branchaud","Johannes Sixt","Jakub Narębski"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"250626","messageId":"xmqqa94yzap8.fsf@gitster.dls.corp.google.com","threadId":"37727","inReplyTo":null,"subject":"[PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-14T19:57:07Z","receivedAt":"2014-10-14T19:57:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"While use of the --reference option to borrow objects from an\nexisting local repository of the same project is an effective way to\nreduce traffic when cloning a project over the network, it makes the\nresulting \"borrowing\" repository dependent on the \"borrowed\"\nrepository. After running\n\n\tgit clone --reference=P $URL Q\n\nthe resulting repository Q will be broken if the borrowed repository\nP disappears.\n\nThe way to allow the borrowed repository to be removed is to repack\nthe borrowing repository (i.e. run \"git repack -a -d\" in Q); while\npower users may know it very well, it is not easily discoverable.\n\nTeach a new \"--dissociate\" option to \"git clone\" to run this\nrepacking for the user.\n\nSigned-off-by: Junio C Hamano \n---\n\n * This comes from\n http://thread.gmane.org/gmane.comp.version-control.git/243918/focus=245397\n which is one of the low-hanging entries in the leftover-bits list\n http://git-blame.blogspot.com/p/leftover-bits.html\n\n Yes, I must have been really bored to do this ;-)\n\n Documentation/git-clone.txt | 11 +++++++++--\n builtin/clone.c | 25 +++++++++++++++++++++++++\n t/t5700-clone-reference.sh | 17 +++++++++++++++++\n 3 files changed, 51 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 0363d00..f1f2a3f 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git clone' [--template=]\n \t [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]\n \t [-o ] [-b ] [-u ] [--reference ]\n-\t [--separate-git-dir ]\n+\t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch]\n \t [--recursive | --recurse-submodules] [--] \n \t []\n@@ -98,7 +98,14 @@ objects from the source repository into a pack in the cloned repository.\n \trequire fewer objects to be copied from the repository\n \tbeing cloned, reducing network and local storage costs.\n +\n-*NOTE*: see the NOTE for the `--shared` option.\n+*NOTE*: see the NOTE for the `--shared` option, and also the\n+`--dissociate` option.\n+\n+--dissociate::\n+\tBorrow the objects from reference repositories specified\n+\twith the `--reference` options only to reduce network\n+\ttransfer and stop borrowing from them after a clone is made\n+\tby making necessary local copies of borrowed objects.\n \n --quiet::\n -q::\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex bbd169c..780fbd5 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -48,6 +48,7 @@ static int option_verbosity;\n static int option_progress = -1;\n static struct string_list option_config;\n static struct string_list option_reference;\n+static int option_dissociate;\n \n static int opt_parse_reference(const struct option *opt, const char *arg, int unset)\n {\n@@ -93,6 +94,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"create a shallow clone of that depth\")),\n \tOPT_BOOL(0, \"single-branch\", &option_single_branch,\n \t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\tOPT_BOOL(0, \"dissociate\", &option_dissociate,\n+\t\t N_(\"use --reference only while cloning\")),\n \tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n@@ -736,6 +739,21 @@ static void write_refspec_config(const char* src_ref_prefix,\n \tstrbuf_release(&value);\n }\n \n+static void dissociate_from_references(void)\n+{\n+\tstruct child_process cmd;\n+\n+\tmemset(&cmd, 0, sizeof(cmd));\n+\targv_array_pushl(&cmd.args, \"repack\", \"-a\", \"-d\", NULL);\n+\tcmd.git_cmd = 1;\n+\tcmd.out = -1;\n+\tcmd.no_stdin = 1;\n+\tif (run_command(&cmd))\n+\t\tdie(_(\"cannot repack to clean up\"));\n+\tif (unlink(git_path(\"objects/info/alternates\")) && errno != ENOENT)\n+\t\tdie_errno(_(\"cannot unlink temporary alternates file\"));\n+}\n+\n int cmd_clone(int argc, const char **argv, const char *prefix)\n {\n \tint is_bundle = 0, is_local;\n@@ -883,6 +901,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tif (option_reference.nr)\n \t\tsetup_reference();\n+\telse if (option_dissociate) {\n+\t\twarning(_(\"--dissociate given, but there is no --reference\"));\n+\t\toption_dissociate = 0;\n+\t}\n \n \tfetch_pattern = value.buf;\n \trefspec = parse_fetch_refspec(1, &fetch_pattern);\n@@ -996,6 +1018,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \ttransport_unlock_pack(transport);\n \ttransport_disconnect(transport);\n \n+\tif (option_dissociate)\n+\t\tdissociate_from_references();\n+\n \tjunk_mode = JUNK_LEAVE_REPO;\n \terr = checkout();\n \ndiff --git a/t/t5700-clone-reference.sh b/t/t5700-clone-reference.sh\nindex 6537911..3e783fc 100755\n--- a/t/t5700-clone-reference.sh\n+++ b/t/t5700-clone-reference.sh\n@@ -198,4 +198,21 @@ test_expect_success 'clone using repo pointed at by gitfile as reference' '\n \ttest_cmp expected \"$base_dir/O/.git/objects/info/alternates\"\n '\n \n+test_expect_success 'clone and dissociate from reference' '\n+\tgit init P &&\n+\t(\n+\t\tcd P &&\ttest_commit one\n+\t) &&\n+\tgit clone P Q &&\n+\t(\n+\t\tcd Q && test_commit two\n+\t) &&\n+\tgit clone --no-local --reference=P Q R &&\n+\tgit clone --no-local --reference=P --dissociate Q S &&\n+\t# removing the reference P would corrupt R but not S\n+\trm -fr P &&\n+\ttest_must_fail git -C R fsck &&\n+\tgit -C S fsck\n+'\n+\n test_done\n-- \n2.1.2-488-g6ab273f\n"},{"id":"250678","messageId":"543E85FA.6050404@xiplink.com","threadId":"37727","inReplyTo":"xmqqa94yzap8.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2014-10-15T14:34:34Z","receivedAt":"2014-10-15T14:34:34Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 14-10-14 03:57 PM, Junio C Hamano wrote:\n> While use of the --reference option to borrow objects from an\n> existing local repository of the same project is an effective way to\n> reduce traffic when cloning a project over the network, it makes the\n> resulting \"borrowing\" repository dependent on the \"borrowed\"\n> repository. After running\n> \n> \tgit clone --reference=P $URL Q\n> \n> the resulting repository Q will be broken if the borrowed repository\n> P disappears.\n> \n> The way to allow the borrowed repository to be removed is to repack\n> the borrowing repository (i.e. run \"git repack -a -d\" in Q); while\n> power users may know it very well, it is not easily discoverable.\n> \n> Teach a new \"--dissociate\" option to \"git clone\" to run this\n> repacking for the user.\n\nAfter reading the above I thought the option would be better named\n\"--derference\". It seemed to me to be something one would like to run after\nthe first --reference clone.\n\nLooking more closely I see that's not the case. In fact, the option only\nmakes sense if --reference is also used.\n\nI think things would be more understandable if the option was \"--dissociate\n\" and was an explicit alternative to --reference:\n\t[[--reference | --dissociate] ]\n\nI'm still not liking the name \"--dissociate\" though. The original suggestion\nof \"--borrow\" is better. Perhaps \"--library\" or \"--local-cache\"? I dunno...\n\nSo now I'm wondering if the implementation would be more efficient as an\nextension of the --local operation. That is, instead of a post-clone repack,\ndo a --local clone first followed by a simple \"git fetch\" from the source repo.\n\n\t\tM.\n\n\n> Signed-off-by: Junio C Hamano \n> ---\n> \n> * This comes from\n> http://thread.gmane.org/gmane.comp.version-control.git/243918/focus=245397\n> which is one of the low-hanging entries in the leftover-bits list\n> http://git-blame.blogspot.com/p/leftover-bits.html\n> \n> Yes, I must have been really bored to do this ;-)\n> \n> Documentation/git-clone.txt | 11 +++++++++--\n> builtin/clone.c | 25 +++++++++++++++++++++++++\n> t/t5700-clone-reference.sh | 17 +++++++++++++++++\n> 3 files changed, 51 insertions(+), 2 deletions(-)\n> \n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 0363d00..f1f2a3f 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -12,7 +12,7 @@ SYNOPSIS\n> 'git clone' [--template=]\n> \t [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]\n> \t [-o ] [-b ] [-u ] [--reference ]\n> -\t [--separate-git-dir ]\n> +\t [--dissociate] [--separate-git-dir ]\n> \t [--depth ] [--[no-]single-branch]\n> \t [--recursive | --recurse-submodules] [--] \n> \t []\n> @@ -98,7 +98,14 @@ objects from the source repository into a pack in the cloned repository.\n> \trequire fewer objects to be copied from the repository\n> \tbeing cloned, reducing network and local storage costs.\n> +\n> -*NOTE*: see the NOTE for the `--shared` option.\n> +*NOTE*: see the NOTE for the `--shared` option, and also the\n> +`--dissociate` option.\n> +\n> +--dissociate::\n> +\tBorrow the objects from reference repositories specified\n> +\twith the `--reference` options only to reduce network\n> +\ttransfer and stop borrowing from them after a clone is made\n> +\tby making necessary local copies of borrowed objects.\n> \n> --quiet::\n> -q::\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index bbd169c..780fbd5 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -48,6 +48,7 @@ static int option_verbosity;\n> static int option_progress = -1;\n> static struct string_list option_config;\n> static struct string_list option_reference;\n> +static int option_dissociate;\n> \n> static int opt_parse_reference(const struct option *opt, const char *arg, int unset)\n> {\n> @@ -93,6 +94,8 @@ static struct option builtin_clone_options[] = {\n> \t\t N_(\"create a shallow clone of that depth\")),\n> \tOPT_BOOL(0, \"single-branch\", &option_single_branch,\n> \t\t N_(\"clone only one branch, HEAD or --branch\")),\n> +\tOPT_BOOL(0, \"dissociate\", &option_dissociate,\n> +\t\t N_(\"use --reference only while cloning\")),\n> \tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n> \t\t N_(\"separate git dir from working tree\")),\n> \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n> @@ -736,6 +739,21 @@ static void write_refspec_config(const char* src_ref_prefix,\n> \tstrbuf_release(&value);\n> }\n> \n> +static void dissociate_from_references(void)\n> +{\n> +\tstruct child_process cmd;\n> +\n> +\tmemset(&cmd, 0, sizeof(cmd));\n> +\targv_array_pushl(&cmd.args, \"repack\", \"-a\", \"-d\", NULL);\n> +\tcmd.git_cmd = 1;\n> +\tcmd.out = -1;\n> +\tcmd.no_stdin = 1;\n> +\tif (run_command(&cmd))\n> +\t\tdie(_(\"cannot repack to clean up\"));\n> +\tif (unlink(git_path(\"objects/info/alternates\")) && errno != ENOENT)\n> +\t\tdie_errno(_(\"cannot unlink temporary alternates file\"));\n> +}\n> +\n> int cmd_clone(int argc, const char **argv, const char *prefix)\n> {\n> \tint is_bundle = 0, is_local;\n> @@ -883,6 +901,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \n> \tif (option_reference.nr)\n> \t\tsetup_reference();\n> +\telse if (option_dissociate) {\n> +\t\twarning(_(\"--dissociate given, but there is no --reference\"));\n> +\t\toption_dissociate = 0;\n> +\t}\n> \n> \tfetch_pattern = value.buf;\n> \trefspec = parse_fetch_refspec(1, &fetch_pattern);\n> @@ -996,6 +1018,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \ttransport_unlock_pack(transport);\n> \ttransport_disconnect(transport);\n> \n> +\tif (option_dissociate)\n> +\t\tdissociate_from_references();\n> +\n> \tjunk_mode = JUNK_LEAVE_REPO;\n> \terr = checkout();\n> \n> diff --git a/t/t5700-clone-reference.sh b/t/t5700-clone-reference.sh\n> index 6537911..3e783fc 100755\n> --- a/t/t5700-clone-reference.sh\n> +++ b/t/t5700-clone-reference.sh\n> @@ -198,4 +198,21 @@ test_expect_success 'clone using repo pointed at by gitfile as reference' '\n> \ttest_cmp expected \"$base_dir/O/.git/objects/info/alternates\"\n> '\n> \n> +test_expect_success 'clone and dissociate from reference' '\n> +\tgit init P &&\n> +\t(\n> +\t\tcd P &&\ttest_commit one\n> +\t) &&\n> +\tgit clone P Q &&\n> +\t(\n> +\t\tcd Q && test_commit two\n> +\t) &&\n> +\tgit clone --no-local --reference=P Q R &&\n> +\tgit clone --no-local --reference=P --dissociate Q S &&\n> +\t# removing the reference P would corrupt R but not S\n> +\trm -fr P &&\n> +\ttest_must_fail git -C R fsck &&\n> +\tgit -C S fsck\n> +'\n> +\n> test_done\n> \n"},{"id":"250692","messageId":"xmqqwq81w8az.fsf@gitster.dls.corp.google.com","threadId":"37727","inReplyTo":"543E85FA.6050404@xiplink.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-15T17:29:24Z","receivedAt":"2014-10-15T17:29:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Marc Branchaud writes:\n\n> I think things would be more understandable if the option was \"--dissociate\n> \" and was an explicit alternative to --reference:\n> \t[[--reference | --dissociate] ]\n>\n> I'm still not liking the name \"--dissociate\" though. The original suggestion\n> of \"--borrow\" is better. Perhaps \"--library\" or \"--local-cache\"? I dunno...\n\nI was not thinking when I originally started the topic with\n\"--borrow\", until I realized that it would not make much sense,\nprimarily because we allow multiple references.\n\nWhat should this command line do, and how would you implement such a\nbehaviour?\n\n $ git clone \\\n --reference=/local/pool/linux.git \\\n --borrow=../my/neighbour/linux-hack.git \\\n git://git.kernel.org/...../linux.git\n\nWith \"do the usual --reference thing, but then dissociate the result\nfrom referents\" option, there is no ambiguity and that is why I did\nnot go with the \"--borrow\" option suggested in the original thread.\n\n> So now I'm wondering if the implementation would be more efficient as an\n> extension of the --local operation. That is, instead of a post-clone repack,\n> do a --local clone first followed by a simple \"git fetch\" from the source repo.\n\nThe network overhead may be comparable to the \"--reference\"\noptimization, but if your \"clone --local\" ends up copying (instead\nof hard-linking), the initial cost to copy locally would be a pure\nextra price over \"clone --reference and then --dissociate\". If the\nlocal clone uses hard-linking, it would be cheaper, but it still\ncosts more than dropping an entry into .git/objects/info/alternates,\nI would imagine. You will pay with your scheme the same cost to run\n\"repack -a -d\", which is paid by \"--dissociate\" at the end of clone,\neventually at the first \"gc\", so there is no efficiency advantage,\neither.\n\nThe above is my knee-jerk assessment without any measuring, though.\n"},{"id":"250701","messageId":"543ECE88.7050703@kdbg.org","threadId":"37727","inReplyTo":"xmqqa94yzap8.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2014-10-15T19:44:08Z","receivedAt":"2014-10-15T19:44:08Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 14.10.2014 um 21:57 schrieb Junio C Hamano:\n> +static void dissociate_from_references(void)\n> +{\n> +\tstruct child_process cmd;\n> +\n> +\tmemset(&cmd, 0, sizeof(cmd));\n\nWe have CHILD_PROCESS_INIT these days.\n\n> +\targv_array_pushl(&cmd.args, \"repack\", \"-a\", \"-d\", NULL);\n> +\tcmd.git_cmd = 1;\n> +\tcmd.out = -1;\n\nThis requests a pipe, but you don't use it nor need it.\n\n> +\tcmd.no_stdin = 1;\n> +\tif (run_command(&cmd))\n> +\t\tdie(_(\"cannot repack to clean up\"));\n> +\tif (unlink(git_path(\"objects/info/alternates\")) && errno != ENOENT)\n> +\t\tdie_errno(_(\"cannot unlink temporary alternates file\"));\n> +}\n\nUnless you have a secret plan, you can do it even shorter with our\nhelpers:\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 8649090..81efb07 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -743,14 +743,9 @@ static void write_refspec_config(const char *src_ref_prefix,\n \n static void dissociate_from_references(void)\n {\n-\tstruct child_process cmd;\n-\n-\tmemset(&cmd, 0, sizeof(cmd));\n-\targv_array_pushl(&cmd.args, \"repack\", \"-a\", \"-d\", NULL);\n-\tcmd.git_cmd = 1;\n-\tcmd.out = -1;\n-\tcmd.no_stdin = 1;\n-\tif (run_command(&cmd))\n+\tstatic const char* argv[] = { \"repack\", \"-a\", \"-d\", NULL };\n+\n+\tif (run_command_v_opt(argv, RUN_GIT_CMD|RUN_COMMAND_NO_STDIN))\n \t\tdie(_(\"cannot repack to clean up\"));\n \tif (unlink(git_path(\"objects/info/alternates\")) && errno != ENOENT)\n \t\tdie_errno(_(\"cannot unlink temporary alternates file\"));\n"},{"id":"250705","messageId":"543EDE6A.7040500@xiplink.com","threadId":"37727","inReplyTo":"xmqqwq81w8az.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2014-10-15T20:51:54Z","receivedAt":"2014-10-15T20:51:54Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 14-10-15 01:29 PM, Junio C Hamano wrote:\n> Marc Branchaud writes:\n> \n>> I think things would be more understandable if the option was \"--dissociate\n>> \" and was an explicit alternative to --reference:\n>> \t[[--reference | --dissociate] ]\n>>\n>> I'm still not liking the name \"--dissociate\" though. The original suggestion\n>> of \"--borrow\" is better. Perhaps \"--library\" or \"--local-cache\"? I dunno...\n> \n> I was not thinking when I originally started the topic with\n> \"--borrow\", until I realized that it would not make much sense,\n> primarily because we allow multiple references.\n\nI hadn't realized that was possible. There's no indication in the man page\nthat multiple --references are allowed (or forbidden, for that matter).\n\n> What should this command line do, and how would you implement such a\n> behaviour?\n> \n> $ git clone \\\n> --reference=/local/pool/linux.git \\\n> --borrow=../my/neighbour/linux-hack.git \\\n> git://git.kernel.org/...../linux.git\n> \n> With \"do the usual --reference thing, but then dissociate the result\n> from referents\" option, there is no ambiguity and that is why I did\n> not go with the \"--borrow\" option suggested in the original thread.\n\nI had not considered this case. My limited imagination has a hard time\ncoming up with a scenario where more than one --reference (or\n--borrow/--dissociate) would make sense. In this example, the --borrow seems\nuseless. How would clone decide that it even needed objects from the\nneighbour repo? None of the refs on gko need any of the neighbour's unique\nobjects. (I get the feeling I don't understand how clone works...)\n\n>> So now I'm wondering if the implementation would be more efficient as an\n>> extension of the --local operation. That is, instead of a post-clone repack,\n>> do a --local clone first followed by a simple \"git fetch\" from the source repo.\n> \n> The network overhead may be comparable to the \"--reference\"\n> optimization, but if your \"clone --local\" ends up copying (instead\n> of hard-linking), the initial cost to copy locally would be a pure\n> extra price over \"clone --reference and then --dissociate\". If the\n> local clone uses hard-linking, it would be cheaper, but it still\n> costs more than dropping an entry into .git/objects/info/alternates,\n> I would imagine. You will pay with your scheme the same cost to run\n> \"repack -a -d\", which is paid by \"--dissociate\" at the end of clone,\n> eventually at the first \"gc\", so there is no efficiency advantage,\n> either.\n> \n> The above is my knee-jerk assessment without any measuring, though.\n\nThat makes sense to me, at least. I agree with your assessment.\n\n\t\tM.\n"},{"id":"250707","messageId":"xmqqsiipuifi.fsf@gitster.dls.corp.google.com","threadId":"37727","inReplyTo":"543EDE6A.7040500@xiplink.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-15T21:33:37Z","receivedAt":"2014-10-15T21:33:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Marc Branchaud writes:\n\n> On 14-10-15 01:29 PM, Junio C Hamano wrote:\n>\n>> $ git clone \\\n>> --reference=/local/pool/linux.git \\\n>> --borrow=../my/neighbour/linux-hack.git \\\n>> git://git.kernel.org/...../linux.git\n>> \n>> With \"do the usual --reference thing, but then dissociate the result\n>> from referents\" option, there is no ambiguity and that is why I did\n>> not go with the \"--borrow\" option suggested in the original thread.\n>\n> I had not considered this case. My limited imagination has a hard time\n> coming up with a scenario where more than one --reference (or\n> In this example, the --borrow seems\n> useless. How would clone decide that it even needed objects from the\n> neighbour repo? None of the refs on gko need any of the neighbour's unique\n> objects.\n\nA probable scenario might go like this.\n\n The company-wide pool is designed for everybody's use and will\n stay, even if it lags behind because it fetches every other day,\n so it is safe to keep referring to via alternates. My neighbour\n is following the linux-next repository and has changes that are\n meant to land \"in the future\" to the mainline, but it can\n disappear without notice so I cannot afford to depend on its\n presense forever.\n\nUnder that particular scenario, what should happen is fairly clear;\nwe want to dissociate from neibour's immediately after clone is\ndone, while being still dependent on the shared pool. But there is\nthe question of \"how would you implement such a behaviour\" (even if\nyou know that is the single only behaviour you would want to see).\n\nAlso I am not confident enough that it is the only plausible way any\nuser may want to mix reference and borrow together.\n"},{"id":"250708","messageId":"xmqqoatduiew.fsf@gitster.dls.corp.google.com","threadId":"37727","inReplyTo":"543ECE88.7050703@kdbg.org","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-15T21:33:59Z","receivedAt":"2014-10-15T21:33:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Unless you have a secret plan, you can do it even shorter with our\n> helpers:\n\nThanks. No there isn't a secret plan. It was just me being rusty.\n\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 8649090..81efb07 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -743,14 +743,9 @@ static void write_refspec_config(const char *src_ref_prefix,\n> \n> static void dissociate_from_references(void)\n> {\n> -\tstruct child_process cmd;\n> -\n> -\tmemset(&cmd, 0, sizeof(cmd));\n> -\targv_array_pushl(&cmd.args, \"repack\", \"-a\", \"-d\", NULL);\n> -\tcmd.git_cmd = 1;\n> -\tcmd.out = -1;\n> -\tcmd.no_stdin = 1;\n> -\tif (run_command(&cmd))\n> +\tstatic const char* argv[] = { \"repack\", \"-a\", \"-d\", NULL };\n> +\n> +\tif (run_command_v_opt(argv, RUN_GIT_CMD|RUN_COMMAND_NO_STDIN))\n> \t\tdie(_(\"cannot repack to clean up\"));\n> \tif (unlink(git_path(\"objects/info/alternates\")) && errno != ENOENT)\n> \t\tdie_errno(_(\"cannot unlink temporary alternates file\"));\n"},{"id":"250710","messageId":"543EEAC4.8070204@xiplink.com","threadId":"37727","inReplyTo":"xmqqsiipuifi.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2014-10-15T21:44:36Z","receivedAt":"2014-10-15T21:44:36Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 14-10-15 05:33 PM, Junio C Hamano wrote:\n> Marc Branchaud writes:\n> \n>> On 14-10-15 01:29 PM, Junio C Hamano wrote:\n>>\n>>> $ git clone \\\n>>> --reference=/local/pool/linux.git \\\n>>> --borrow=../my/neighbour/linux-hack.git \\\n>>> git://git.kernel.org/...../linux.git\n>>>\n>>> With \"do the usual --reference thing, but then dissociate the result\n>>> from referents\" option, there is no ambiguity and that is why I did\n>>> not go with the \"--borrow\" option suggested in the original thread.\n>>\n>> I had not considered this case. My limited imagination has a hard time\n>> coming up with a scenario where more than one --reference (or\n>> In this example, the --borrow seems\n>> useless. How would clone decide that it even needed objects from the\n>> neighbour repo? None of the refs on gko need any of the neighbour's unique\n>> objects.\n> \n> A probable scenario might go like this.\n> \n> The company-wide pool is designed for everybody's use and will\n> stay, even if it lags behind because it fetches every other day,\n> so it is safe to keep referring to via alternates. My neighbour\n> is following the linux-next repository and has changes that are\n> meant to land \"in the future\" to the mainline, but it can\n> disappear without notice so I cannot afford to depend on its\n> presense forever.\n> \n> Under that particular scenario, what should happen is fairly clear;\n> we want to dissociate from neibour's immediately after clone is\n> done, while being still dependent on the shared pool.\n\nYes, but we're cloning gko, not the neighbour. Doesn't that mean that the\nclone operation won't know about any of the neighbour's refs? In order to\nget any of the neighbour's refs (and its unique objects) you have to either\nclone the neighbour directly or (post-clone) fetch from it, no?\n\n\t\tM.\n"},{"id":"250711","messageId":"xmqqfvepuhmy.fsf@gitster.dls.corp.google.com","threadId":"37727","inReplyTo":"543EEAC4.8070204@xiplink.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-15T21:50:45Z","receivedAt":"2014-10-15T21:50:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Marc Branchaud writes:\n\n> Yes, but we're cloning gko, not the neighbour. Doesn't that mean that the\n> clone operation won't know about any of the neighbour's refs?\n\nNo. --reference (and a natural implementation of --borrow, I would imagine)\npeeks the refs of the repository we borrow from and that is how\nclone can say \"I already have objects reachable from these refs, so\nplease send me the remainder\" to the repository it is cloning from.\n"},{"id":"250754","messageId":"543FE3BE.9010307@xiplink.com","threadId":"37727","inReplyTo":"xmqqfvepuhmy.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2014-10-16T15:26:54Z","receivedAt":"2014-10-16T15:26:54Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 14-10-15 05:50 PM, Junio C Hamano wrote:\n> Marc Branchaud writes:\n> \n>> Yes, but we're cloning gko, not the neighbour. Doesn't that mean that the\n>> clone operation won't know about any of the neighbour's refs?\n> \n> No. --reference (and a natural implementation of --borrow, I would imagine)\n> peeks the refs of the repository we borrow from and that is how\n> clone can say \"I already have objects reachable from these refs, so\n> please send me the remainder\" to the repository it is cloning from.\n\nBy \"know about\" I meant \"want to use\". Sorry for being a bit dense about\nthis; let me try again.\n\n(BTW, it occurs to me that your patch -- if I read it right -- doesn't\nfulfill your scenario since it disassociates the clone from all repos,\nregardless of whether they are specified with --reference or --borrow. In\nthe following I assume a --borrow that only disassociates from the specified\nrepo and leaves the --reference repo(s) alone.)\n\nSince we're cloning gko's refs, all of the neighbour's unique refs and\nobjects can be ignored. Even though paths to the neighbour and the local\npool will be in the clone's alternates file, any refs the clone operation\ncreates won't need to use any objects from the neighbour. The clone\noperation gives us no way to refer to the neighbour's unique objects.\n\nI just don't see what difference the --borrow option makes. Consider the two\ncases:\n\nWith just --reference=/local/pool/linux.git:\n\t1. Set up the alternates file with that path.\n\t2. Copy gko's refs into refs/remotes/origin/.\n\t3. Set up refs/heads/master to refer to gko's HEAD.\n\t4. Checkout refs/heads/master (uses objects from local pool).\n\nWith both that --reference and --borrow=../my/neighbour/linux-hack.git:\n\t1. Set up the alternates file with both paths.\n\t2. Copy gko's refs into refs/remotes/origin/.\n\t3. Set up refs/heads/master to refer to gko's HEAD.\n\t4. Checkout refs/heads/master (uses objects from local pool).\n\t5. Disassociate ourselves from the neighbour repo.\n\nIn both cases the first four actions have no need of the neighbour repo. The\nsecond case's fifth action surgically removes the neighbour as an alternate\nobject store, and we're left with the same clone we got in the first case.\nWhat was the point?\n\nIt seems that in order to make something like --borrow useful, \"git clone\"\nwould somehow need to know which of the neighbour's refs you want to *also*\nclone, then copy any unique objects from the neighbour before disassociating\nfrom it.\n\n\t\tM.\n"},{"id":"250764","messageId":"xmqqegu7u85m.fsf@gitster.dls.corp.google.com","threadId":"37727","inReplyTo":"xmqqwq81w8az.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-16T19:27:49Z","receivedAt":"2014-10-16T19:27:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Marc Branchaud writes:\n>\n>> I think things would be more understandable if the option was \"--dissociate\n>> \" and was an explicit alternative to --reference:\n>> \t[[--reference | --dissociate] ]\n>>\n>> I'm still not liking the name \"--dissociate\" though. The original suggestion\n>> of \"--borrow\" is better. Perhaps \"--library\" or \"--local-cache\"? I dunno...\n>\n> I was not thinking when I originally started the topic with\n> \"--borrow\", until I realized that it would not make much sense,\n> primarily because we allow multiple references.\n>\n> What should this command line do, and how would you implement such a\n> behaviour?\n>\n> $ git clone \\\n> --reference=/local/pool/linux.git \\\n> --borrow=../my/neighbour/linux-hack.git \\\n> git://git.kernel.org/...../linux.git\n>\n> With \"do the usual --reference thing, but then dissociate the result\n> from referents\" option, there is no ambiguity and that is why I did\n> not go with the \"--borrow\" option suggested in the original thread.\n\nAnother approach we could take is to add --borrow and then forbid\nmixing --reference and --borrow on the same command line, until\nsomebody comes up with an implementation to allow us dissociate from\nborrowed repositories while still depending on referenced ones, at\nwhich time we can lift that limitation.\n\nBut if that future extension is not going to happen, there is not\nmuch difference. The end result will be either\n\n - the one that is very clear to the users that they cannot\n selectively dissociate because there is no such option documented\n (i.e. --reference, --dissociate and no --borrow); and\n\n - the other one that gives a slight hope to the users that the\n combination might work (i.e. with --reference, --borrow and no\n --dissociate) but refuses to do so when it actually is run.\n\nBetween the two, the former might actually be easier to the users to\nunderstand, as it keeps their expectation in line with the reality.\n\nSo I dunno.\n\nI certainly am *not* going to do the selective dissociation change\nmyself. Do we have a volunteer (hint: this probably does not fall\ninto \"low-hanging fruit\" category)?\n"},{"id":"250797","messageId":"54410FCB.6070908@gmail.com","threadId":"37727","inReplyTo":"543FE3BE.9010307@xiplink.com","subject":"Re: [PATCH] clone: --dissociate option to mark that reference is only temporary","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2014-10-17T12:47:07Z","receivedAt":"2014-10-17T12:47:07Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On 2014-10-16 17:26, Marc Branchaud wrote:\n\n> I just don't see what difference the --borrow option makes. Consider the two\n> cases:\n>\n> With just --reference=/local/pool/linux.git:\n> \t1. Set up the alternates file with that path.\n\n x. Fetch object from origin not present in pool\n\n> \t2. Copy gko's refs into refs/remotes/origin/.\n> \t3. Set up refs/heads/master to refer to gko's HEAD.\n> \t4. Checkout refs/heads/master (uses objects from local pool).\n>\n> With both that --reference and --borrow=../my/neighbour/linux-hack.git:\n> \t1. Set up the alternates file with both paths.\n\n x. Fetch objects from origin not present in either pool\n or neighbour repo (\"have\" pool and neighbour)\n\n> \t2. Copy gko's refs into refs/remotes/origin/.\n> \t3. Set up refs/heads/master to refer to gko's HEAD.\n> \t4. Checkout refs/heads/master (uses objects from local pool).\n> \t5. Disassociate ourselves from the neighbour repo.\n which means roughly:\n\n 5.1. Remove neighbour repo from alternates\n 5.2. Fetch required objects from neighbour repo\n (\"want\" neighbour, have ???)\n\nIt is possible that because of technical limitations --reference and \n--borrow / dissociate / --temporary-reference / --object-cache are to be\nmutually exclusive.\n\n> In both cases the first four actions have no need of the neighbour repo. The\n> second case's fifth action surgically removes the neighbour as an alternate\n> object store, and we're left with the same clone we got in the first case.\n> What was the point?\n\nYou are missing fetching object from your list of actions.\n\n> It seems that in order to make something like --borrow useful, \"git clone\"\n> would somehow need to know which of the neighbour's refs you want to *also*\n> clone, then copy any unique objects from the neighbour before disassociating\n> from it.\n>\n> \t\tM.\n\n-- \nJakub Narębski\n"}]}