{"thread":{"id":"39146","subject":"git clone --depth: shallow clone problems","startedAt":"2015-04-23T14:53:04Z","lastAt":"2015-05-29T15:52:05Z","messageCount":5,"participants":["Michal Pomorski","Duy Nguyen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"259899","messageId":"CALxdAKiyRMR-jF2KcBQGYgXUb+D8=YaESU3E9KN+zBLx=0UCNw@mail.gmail.com","threadId":"39146","inReplyTo":null,"subject":"git clone --depth: shallow clone problems","fromName":"Michal Pomorski","fromEmail":"misieck@gmail.com","sentAt":"2015-04-23T14:53:04Z","receivedAt":"2015-04-23T14:53:04Z","isPatch":false,"sender":{"key":"misieck@gmail.com","avatar":null},"body":"tl: skip to the second paragraph\n\nSo here is what I just experienced:\nWe had an emergency error in an application at work and as the\nresponsible developer was unavailable, I was asked to check it out and\nlook into it.\nWe are a small branch of a bigger company and our connection to the\ncompany's source servers is really slow, so just to quickly start it\nup, I decided to take a shallow clone (that's what it is for, right?).\nAfter a while we realized, the clone I have made was not sufficient\nand was missing some newest work done on the project.\nI did \"git fetch --unshallow\" which finished surprisingly quickly, and\nit did not bring any newer commits.\nUnaware of the fine print hiding in the documentation of git clone, I\nblamed the repo (and in extension the person who provided me the\naddress to it). After coming to a realization, a process which cost me\ntime and embarrassment, I understood what is supposed to be the\n\"correct\" behaviour:\n\"--single-branch\nClone only the history leading to the tip of a single branch, either\nspecified by the --branch option or the primary branch remote's HEAD\npoints at. When creating a shallow clone with the --depth option, this\nis the default, unless --no-single-branch is given to fetch the\nhistories near the tips of all branches.\"\nOf course, the new commits were put on a custom branch, and I knew\nthat all the time, but I expected the branch to show up eventually, at\nleast after git fetch --unshallow.\n\n\nI hope you can see the faults in the usability of the commands I was exposed to:\n1) git clone --depth should:\n -warn about only fetching the current HEAD (name it by a real name\nif applicable)\nand/or\n -require the --branch option so that it is not left to chance\n(current HEAD could be anything; is it really meaningful to talk about\nthe current HEAD on a server?)\nand/or\n -make the --no-single-branch the default...\nand maybe\n -have the option to clone the most recent commits.\n\n2) git fetch --unshallow should convert the clone into an actual\nequivalent of a normal, not shallow clone.\n\n3) The documentation should be improved. The behaviour of --shallow is\ndescribed partly in the description of --no-single-branch. This is the\ndocumentation equivalent of the infamous \"come from\" control flow\nstructure.\n\nRegards,\nMichal Pomorski\n"},{"id":"262119","messageId":"20150526124702.GA12101@lanh","threadId":"39146","inReplyTo":"CALxdAKiyRMR-jF2KcBQGYgXUb+D8=YaESU3E9KN+zBLx=0UCNw@mail.gmail.com","subject":"Re: git clone --depth: shallow clone problems","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-05-26T12:47:02Z","receivedAt":"2015-05-26T12:47:02Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Apr 23, 2015 at 04:53:04PM +0200, Michal Pomorski wrote:\n> tl: skip to the second paragraph\n> \n> So here is what I just experienced:\n> We had an emergency error in an application at work and as the\n> responsible developer was unavailable, I was asked to check it out and\n> look into it.\n> We are a small branch of a bigger company and our connection to the\n> company's source servers is really slow, so just to quickly start it\n> up, I decided to take a shallow clone (that's what it is for, right?).\n> After a while we realized, the clone I have made was not sufficient\n> and was missing some newest work done on the project.\n> I did \"git fetch --unshallow\" which finished surprisingly quickly, and\n> it did not bring any newer commits.\n> Unaware of the fine print hiding in the documentation of git clone, I\n> blamed the repo (and in extension the person who provided me the\n> address to it). After coming to a realization, a process which cost me\n> time and embarrassment, I understood what is supposed to be the\n> \"correct\" behaviour:\n> \"--single-branch\n> Clone only the history leading to the tip of a single branch, either\n> specified by the --branch option or the primary branch remote's HEAD\n> points at. When creating a shallow clone with the --depth option, this\n> is the default, unless --no-single-branch is given to fetch the\n> histories near the tips of all branches.\"\n> Of course, the new commits were put on a custom branch, and I knew\n> that all the time, but I expected the branch to show up eventually, at\n> least after git fetch --unshallow.\n> \n>\n> I hope you can see the faults in the usability of the commands I was\n> exposed to:\n> 1) git clone --depth should:\n> -warn about only fetching the current HEAD (name it by a real name\n> if applicable)\n> and/or\n> -require the --branch option so that it is not left to chance\n> (current HEAD could be anything; is it really meaningful to talk about\n> the current HEAD on a server?)\n> and/or\n> -make the --no-single-branch the default...\n> and maybe\n> -have the option to clone the most recent commits.\n\n--single-branch being the default was because we (at least I, stilll)\nbelieved it was the common case, so I don't think we should revert\nit. But I can see --unshallow documentation is misleading. How about\nthis? I think it's better than what we have.\n\n-- 8< --\nSubject: [PATCH] fetch-options.txt: clarify what --unshallow does\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/fetch-options.txt | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 45583d8..63d3452 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -14,12 +14,17 @@\n \tbranch history. Tags for the deepened commits are not fetched.\n \n --unshallow::\n-\tIf the source repository is complete, convert a shallow\n-\trepository to a complete one, removing all the limitations\n+\tIf the source repository is complete, convert all shallow\n+\trefs to complete ones, removing all the limitations\n \timposed by shallow repositories.\n +\n If the source repository is shallow, fetch as much as possible so that\n-the current repository has the same history as the source repository.\n+the all existing refs of current repository have the same history as in\n+the source repository.\n++\n+Note that if the repository is created with --single-branch, which is\n+default for a shallow clone, only one ref is completed. `--unshallow`\n+does not fetch all remaining refs from source repository.\n \n --update-shallow::\n \tBy default when fetching from a shallow repository,\n-- \n2.3.0.rc1.137.g477eb31\n-- 8< --\n\n> 2) git fetch --unshallow should convert the clone into an actual\n> equivalent of a normal, not shallow clone.\n\nI was thinking of some way to undo --single-branch too. I don't think\nit should be the job of --unshallow, maybe a new option, but I can't\nthink of a name better than --really-unshallow :P\n\nIt's been a while and no one responds to this, perhaps people are not\ninterested in such an option?\n\n> 3) The documentation should be improved. The behaviour of --shallow is\n> described partly in the description of --no-single-branch. This is the\n> documentation equivalent of the infamous \"come from\" control flow\n> structure.\n\nYes. Like this?\n\n-- 8< --\nSubject: [PATCH] clone.txt: mention --single-branch in --depth\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-clone.txt | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex f1f2a3f..9c320da 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -185,7 +185,9 @@ objects from the source repository into a pack in the cloned repository.\n \n --depth ::\n \tCreate a 'shallow' clone with a history truncated to the\n-\tspecified number of revisions.\n+\tspecified number of revisions. --single-branch is\n+\tautomatically specified unless the user overrides it with\n+\t--no-single-branch\n \n --[no-]single-branch::\n \tClone only the history leading to the tip of a single branch,\n-- \n2.3.0.rc1.137.g477eb31\n\n-- 8< --\n"},{"id":"262128","messageId":"xmqqmw0rl78r.fsf@gitster.dls.corp.google.com","threadId":"39146","inReplyTo":"20150526124702.GA12101@lanh","subject":"Re: git clone --depth: shallow clone problems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-05-26T14:54:12Z","receivedAt":"2015-05-26T14:54:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> --unshallow::\n> -\tIf the source repository is complete, convert a shallow\n> -\trepository to a complete one, removing all the limitations\n> +\tIf the source repository is complete, convert all shallow\n> +\trefs to complete ones, removing all the limitations\n\nDefine \"shallow ref\", or better yet explain without introducing such\na new term (I do not think shallow-ness is a property of a ref, by\nthe way---I think you meant all refs that can reach the points the\nhistory is cauterised by .git/shallow, but we shouldn't assume that\nthe target audience of this paragraph to know Git as well as I do).\n\n> \timposed by shallow repositories.\n> +\n> If the source repository is shallow, fetch as much as possible so that\n> -the current repository has the same history as the source repository.\n> +the all existing refs of current repository have the same history as in\n> +the source repository.\n\nMakes sense, modulo \"so that the all existing refs\" sounds strange\nto my ears (\"all the existing refs\" perhaps).\n\n> ++\n> +Note that if the repository is created with --single-branch, which is\n> +default for a shallow clone, only one ref is completed. `--unshallow`\n> +does not fetch all remaining refs from source repository.\n\nI do not think this \"Note\" is being as helpful as it could be.\n\n - If the repository was created with --single-branch but if the\n user later fetched and started tracking other branches, the\n statement \"only one ref is completed\" is untrue; the true version\n is \"only the existing refs are completed\", which leads to a more\n important point. It says the same thing as \"all existing refs\"\n above and does not add any useful information.\n\n - The last sentence is less than useful but merely frustrating---it\n says what you cannot do with this option, strongly hints that the\n writer of the sentence knows what the reader wants to achieve,\n but without saying what other way the reader go to achieve it.\n\nPerhaps replace that Note paragraph with something along this line?\n\n +\n By fetching and tracking refs that you haven't been tracking,\n you can add these new refs to \"all refs\" to be unshallowed.\n\n>> 2) git fetch --unshallow should convert the clone into an actual\n>> equivalent of a normal, not shallow clone.\n>\n> I was thinking of some way to undo --single-branch too. I don't think\n> it should be the job of --unshallow, maybe a new option, but I can't\n> think of a name better than --really-unshallow :P\n\nIsn't that just the matter of updating remote.origin.fetch? I do\nnot think it belongs to \"clone\" (or \"fetch\"). Perhaps it belongs to\n\"remote\", where it already shows with \"git remote -v\" what is\nfetched and pushed.\n\n> --depth ::\n> \tCreate a 'shallow' clone with a history truncated to the\n> -\tspecified number of revisions.\n> +\tspecified number of revisions. --single-branch is\n> +\tautomatically specified unless the user overrides it with\n> +\t--no-single-branch\n\nYeah, something like that would be a definite improvement.\n\nBy the way, while composing this message, I scratched my head after\ntyping\n\n $ git clone --shallow=4 --no-local ./git.git ./playpen\n\nIs it just me or do we want to add such a synonym?\n\nThanks.\n"},{"id":"262406","messageId":"CACsJy8AO7UNF-SnuqohOPDy2dOvfg=bA+b0rdVihabxBLAz19Q@mail.gmail.com","threadId":"39146","inReplyTo":"xmqqmw0rl78r.fsf@gitster.dls.corp.google.com","subject":"Re: git clone --depth: shallow clone problems","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-05-29T11:04:31Z","receivedAt":"2015-05-29T11:04:31Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 26, 2015 at 9:54 PM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>> --unshallow::\n>> - If the source repository is complete, convert a shallow\n>> - repository to a complete one, removing all the limitations\n>> + If the source repository is complete, convert all shallow\n>> + refs to complete ones, removing all the limitations\n>\n> Define \"shallow ref\", or better yet explain without introducing such\n> a new term (I do not think shallow-ness is a property of a ref, by\n> the way---I think you meant all refs that can reach the points the\n> history is cauterised by .git/shallow, but we shouldn't assume that\n> the target audience of this paragraph to know Git as well as I do).\n\nOK maybe\n\nMake sure all existing refs have the same history as the ones from the\nsource repository. If the source repository is complete, this removes\nall limitations imposed by shallow repositories.\n\n?\n\n>> imposed by shallow repositories.\n>> +\n>> If the source repository is shallow, fetch as much as possible so that\n>> -the current repository has the same history as the source repository.\n>> +the all existing refs of current repository have the same history as in\n>> +the source repository.\n>\n> Makes sense, modulo \"so that the all existing refs\" sounds strange\n> to my ears (\"all the existing refs\" perhaps).\n\nWill fix. Although I think the new paragraph above already covers this case too.\n\n>> ++\n>> +Note that if the repository is created with --single-branch, which is\n>> +default for a shallow clone, only one ref is completed. `--unshallow`\n>> +does not fetch all remaining refs from source repository.\n>\n> I do not think this \"Note\" is being as helpful as it could be.\n>\n> - If the repository was created with --single-branch but if the\n> user later fetched and started tracking other branches, the\n> statement \"only one ref is completed\" is untrue; the true version\n> is \"only the existing refs are completed\", which leads to a more\n> important point. It says the same thing as \"all existing refs\"\n> above and does not add any useful information.\n>\n> - The last sentence is less than useful but merely frustrating---it\n> says what you cannot do with this option, strongly hints that the\n> writer of the sentence knows what the reader wants to achieve,\n> but without saying what other way the reader go to achieve it.\n>\n> Perhaps replace that Note paragraph with something along this line?\n>\n> +\n> By fetching and tracking refs that you haven't been tracking,\n> you can add these new refs to \"all refs\" to be unshallowed.\n\nIt was meant to emphasize --unshallow would not create a true clone,\nif people miss the keywords \"all existing\" refs.. Maybe..\n\nYou may need to fetch and track other refs from the source repository\nif you want to make full clone.\n\n?\n\n>\n>>> 2) git fetch --unshallow should convert the clone into an actual\n>>> equivalent of a normal, not shallow clone.\n>>\n>> I was thinking of some way to undo --single-branch too. I don't think\n>> it should be the job of --unshallow, maybe a new option, but I can't\n>> think of a name better than --really-unshallow :P\n>\n> Isn't that just the matter of updating remote.origin.fetch? I do\n> not think it belongs to \"clone\" (or \"fetch\"). Perhaps it belongs to\n> \"remote\", where it already shows with \"git remote -v\" what is\n> fetched and pushed.\n\nYeah, perhaps showing an advice about git-remote is enough if the\ncurrent ref spec does not fetch everything.\n\n>> --depth ::\n>> Create a 'shallow' clone with a history truncated to the\n>> - specified number of revisions.\n>> + specified number of revisions. --single-branch is\n>> + automatically specified unless the user overrides it with\n>> + --no-single-branch\n>\n> Yeah, something like that would be a definite improvement.\n>\n> By the way, while composing this message, I scratched my head after\n> typing\n>\n> $ git clone --shallow=4 --no-local ./git.git ./playpen\n>\n> Is it just me or do we want to add such a synonym?\n\nI have a series to define shallow boundaries by tags or dates, in\naddition to history depth. It may take a few more months until I'm\nfinished with my ongoing topics. Then you could think about what\n--shallow means exactly ;-)\n-- \nDuy\n"},{"id":"262429","messageId":"xmqqiobbe5zu.fsf@gitster.dls.corp.google.com","threadId":"39146","inReplyTo":"CACsJy8AO7UNF-SnuqohOPDy2dOvfg=bA+b0rdVihabxBLAz19Q@mail.gmail.com","subject":"Re: git clone --depth: shallow clone problems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-05-29T15:52:05Z","receivedAt":"2015-05-29T15:52:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Tue, May 26, 2015 at 9:54 PM, Junio C Hamano wrote:\n>> Duy Nguyen writes:\n>>\n>>> --unshallow::\n>>> - If the source repository is complete, convert a shallow\n>>> - repository to a complete one, removing all the limitations\n>>> + If the source repository is complete, convert all shallow\n>>> + refs to complete ones, removing all the limitations\n>>\n>> Define \"shallow ref\", or better yet explain without introducing such\n>> a new term (I do not think shallow-ness is a property of a ref, by\n>> the way---I think you meant all refs that can reach the points the\n>> history is cauterised by .git/shallow, but we shouldn't assume that\n>> the target audience of this paragraph to know Git as well as I do).\n>\n> OK maybe\n>\n> Make sure all existing refs have the same history as the ones from the\n> source repository. If the source repository is complete, this removes\n> all limitations imposed by shallow repositories.\n\nOK, I can understand that description a lot easier than the original\none.\n\n>>> ++\n>>> +Note that if the repository is created with --single-branch, which is\n>>> +default for a shallow clone, only one ref is completed. `--unshallow`\n>>> +does not fetch all remaining refs from source repository.\n>>\n>> I do not think this \"Note\" is being as helpful as it could be.\n>>\n>> - If the repository was created with --single-branch but if the\n>> user later fetched and started tracking other branches, the\n>> statement \"only one ref is completed\" is untrue; the true version\n>> is \"only the existing refs are completed\", which leads to a more\n>> important point. It says the same thing as \"all existing refs\"\n>> above and does not add any useful information.\n>>\n>> - The last sentence is less than useful but merely frustrating---it\n>> says what you cannot do with this option, strongly hints that the\n>> writer of the sentence knows what the reader wants to achieve,\n>> but without saying what other way the reader go to achieve it.\n>>\n>> Perhaps replace that Note paragraph with something along this line?\n>>\n>> +\n>> By fetching and tracking refs that you haven't been tracking,\n>> you can add these new refs to \"all refs\" to be unshallowed.\n>\n> It was meant to emphasize --unshallow would not create a true clone,\n> if people miss the keywords \"all existing\" refs.. Maybe..\n\nI know; it is not helpful to describe what it does *NOT* do, without\nsaying what the reader is likely trying to find out.\n\n> You may need to fetch and track other refs from the source repository\n> if you want to make full clone.\n\nThat is much better.\n"}]}