{"thread":{"id":"42667","subject":"Problem with --shallow-submodules option","startedAt":"2016-06-20T13:26:37Z","lastAt":"2016-06-30T21:05:07Z","messageCount":8,"participants":["Istvan Zakar","Stefan Beller","Fredrik Gustafsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"289677","messageId":"loom.20160620T145755-931@post.gmane.org","threadId":"42667","inReplyTo":null,"subject":"Problem with --shallow-submodules option","fromName":"Istvan Zakar","fromEmail":"istvan.zakar@gmail.com","sentAt":"2016-06-20T13:06:39Z","receivedAt":"2016-06-20T13:26:37Z","isPatch":false,"sender":{"key":"istvan.zakar@gmail.com","avatar":null},"body":"Hello,\n\nI'm working on a relatively big project with many submodules. During \ncloning for testing I tried to decrease the amount of data need to be \nfetched from the server by using --shallow-submodules option in the clone \ncommand. It seems to check out the tip of the remote repo, and if it's not \nthe commit registered in the superproject the submodule update fails \n(obviously). Can I somehow tell to fetch that exact commit I need for my \nsuperproject?\n\nThanks,\n Istvan\n\n"},{"id":"289689","messageId":"CAGZ79kZyEzp92JP_Bp2te1XO=PB0+fwFn57MrBPuWe25PQKOog@mail.gmail.com","threadId":"42667","inReplyTo":"loom.20160620T145755-931@post.gmane.org","subject":"Re: Problem with --shallow-submodules option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-06-20T17:45:57Z","receivedAt":"2016-06-20T17:46:13Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jun 20, 2016 at 6:06 AM, Istvan Zakar wrote:\n> Hello,\n>\n> I'm working on a relatively big project with many submodules. During\n> cloning for testing I tried to decrease the amount of data need to be\n> fetched from the server by using --shallow-submodules option in the clone\n> command. It seems to check out the tip of the remote repo, and if it's not\n> the commit registered in the superproject the submodule update fails\n> (obviously).\n\nYes that is broken as the depth of a submodule is counted from its own HEAD\nnot from the superprojects sha1 as it should.\n\nSo it does\n\n git clone --depth=1 \n\n if HEAD != recorded gitlink sha1,\n git fetch \n\n git checkout \n\n> Can I somehow tell to fetch that exact commit I need for my\n> superproject?\n\nSome servers support fetching by direct sha1, which is what we make use\nof here, then it sort-of works.\n\nIf the server doesn't support the capability to fetch an arbitrary sha1,\nthe submodule command fails, with a message such as\n\n error: no such remote ref $sha1\n Fetched in submodule path '', but it did not contain\n$sha1. Direct fetching of that commit failed.\n\nSo if it breaks for you now, I would suggest not using that switch, I\ndon't think there is a quick\nworkaround.\n\n>\n> Thanks,\n> Istvan\n\nThanks,\nStefan\n\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"289726","messageId":"CAPV8XuZvDkcEwxRB0HwaihVo7QzqsoHTRCdV7sqxkT31-RWkmA@mail.gmail.com","threadId":"42667","inReplyTo":"CAGZ79kZyEzp92JP_Bp2te1XO=PB0+fwFn57MrBPuWe25PQKOog@mail.gmail.com","subject":"Re: Problem with --shallow-submodules option","fromName":"Istvan Zakar","fromEmail":"istvan.zakar@gmail.com","sentAt":"2016-06-21T06:32:14Z","receivedAt":"2016-06-21T06:34:07Z","isPatch":false,"sender":{"key":"istvan.zakar@gmail.com","avatar":null},"body":"Hi,\n\nThanks for the answer.\nSo it means that it is a setting on the server side which can be\nactivated? (I guess it depends on the version of the server)\nI did some reading in the topic. Are you talking about this setting\n\"uploadpack.allowReachableSHA1InWant\", or did I misunderstood what I\nread?\n\nThanks,\n Istvan\n\nOn 20 June 2016 at 19:45, Stefan Beller wrote:\n> On Mon, Jun 20, 2016 at 6:06 AM, Istvan Zakar wrote:\n>> Hello,\n>>\n>> I'm working on a relatively big project with many submodules. During\n>> cloning for testing I tried to decrease the amount of data need to be\n>> fetched from the server by using --shallow-submodules option in the clone\n>> command. It seems to check out the tip of the remote repo, and if it's not\n>> the commit registered in the superproject the submodule update fails\n>> (obviously).\n>\n> Yes that is broken as the depth of a submodule is counted from its own HEAD\n> not from the superprojects sha1 as it should.\n>\n> So it does\n>\n> git clone --depth=1 \n>\n> if HEAD != recorded gitlink sha1,\n> git fetch \n>\n> git checkout \n>\n>> Can I somehow tell to fetch that exact commit I need for my\n>> superproject?\n>\n> Some servers support fetching by direct sha1, which is what we make use\n> of here, then it sort-of works.\n>\n> If the server doesn't support the capability to fetch an arbitrary sha1,\n> the submodule command fails, with a message such as\n>\n> error: no such remote ref $sha1\n> Fetched in submodule path '', but it did not contain\n> $sha1. Direct fetching of that commit failed.\n>\n> So if it breaks for you now, I would suggest not using that switch, I\n> don't think there is a quick\n> workaround.\n>\n>>\n>> Thanks,\n>> Istvan\n>\n> Thanks,\n> Stefan\n>\n>>\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"289806","messageId":"CAGZ79kYvMjDaPnet1J_QrHARYtNqLwRhs7f08cDZ7P6dT8Rs6A@mail.gmail.com","threadId":"42667","inReplyTo":"CAPV8XuZvDkcEwxRB0HwaihVo7QzqsoHTRCdV7sqxkT31-RWkmA@mail.gmail.com","subject":"Re: Problem with --shallow-submodules option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-06-21T20:35:40Z","receivedAt":"2016-06-21T20:42:55Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jun 20, 2016 at 11:32 PM, Istvan Zakar wrote:\n> Hi,\n>\n> Thanks for the answer.\n> So it means that it is a setting on the server side which can be\n> activated? (I guess it depends on the version of the server)\n> I did some reading in the topic. Are you talking about this setting\n> \"uploadpack.allowReachableSHA1InWant\", or did I misunderstood what I\n> read?\n\nNo that's exactly what I meant; sorry for not spelling that out.\n\nThanks,\nStefan\n\n>\n> Thanks,\n> Istvan\n>\n> On 20 June 2016 at 19:45, Stefan Beller wrote:\n>> On Mon, Jun 20, 2016 at 6:06 AM, Istvan Zakar wrote:\n>>> Hello,\n>>>\n>>> I'm working on a relatively big project with many submodules. During\n>>> cloning for testing I tried to decrease the amount of data need to be\n>>> fetched from the server by using --shallow-submodules option in the clone\n>>> command. It seems to check out the tip of the remote repo, and if it's not\n>>> the commit registered in the superproject the submodule update fails\n>>> (obviously).\n>>\n>> Yes that is broken as the depth of a submodule is counted from its own HEAD\n>> not from the superprojects sha1 as it should.\n>>\n>> So it does\n>>\n>> git clone --depth=1 \n>>\n>> if HEAD != recorded gitlink sha1,\n>> git fetch \n>>\n>> git checkout \n>>\n>>> Can I somehow tell to fetch that exact commit I need for my\n>>> superproject?\n>>\n>> Some servers support fetching by direct sha1, which is what we make use\n>> of here, then it sort-of works.\n>>\n>> If the server doesn't support the capability to fetch an arbitrary sha1,\n>> the submodule command fails, with a message such as\n>>\n>> error: no such remote ref $sha1\n>> Fetched in submodule path '', but it did not contain\n>> $sha1. Direct fetching of that commit failed.\n>>\n>> So if it breaks for you now, I would suggest not using that switch, I\n>> don't think there is a quick\n>> workaround.\n>>\n>>>\n>>> Thanks,\n>>> Istvan\n>>\n>> Thanks,\n>> Stefan\n>>\n>>>\n>>> --\n>>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>>> the body of a message to majordomo@vger.kernel.org\n>>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"289857","messageId":"20160622153145.GB16644@paksenarrion.iveqy.com","threadId":"42667","inReplyTo":"loom.20160620T145755-931@post.gmane.org","subject":"Re: Problem with --shallow-submodules option","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2016-06-22T15:31:45Z","receivedAt":"2016-06-22T15:34:48Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Mon, Jun 20, 2016 at 01:06:39PM +0000, Istvan Zakar wrote:\n> I'm working on a relatively big project with many submodules. During \n> cloning for testing I tried to decrease the amount of data need to be \n> fetched from the server by using --shallow-submodules option in the clone \n> command. It seems to check out the tip of the remote repo, and if it's not \n> the commit registered in the superproject the submodule update fails \n> (obviously). Can I somehow tell to fetch that exact commit I need for my \n> superproject?\n\nMaybe. http://stackoverflow.com/questions/2144406/git-shallow-submodules\ngives a good overview of this problem.\n\ngit fetches a branch and is shallow from that branch, which might be an\nother sha1 than the one the submodule points to, (as you say). This\nis/was one of the drawbacks with this method. However the since git 2.8,\ngit will try to fetch the sha1 direct (and not the branch). So then it\nwill work, if(!), the server supports direct access to sha1. This was\npreviously not allowed due to security concerns (if I recall correctly).\n\nSo the answer is, yes this will work if you've a recent version of git\nand support on the server side for doing this. Unfortunately I'm not\nsure which git version is needed on the server side for this to work.\n\n-- \nFredrik Gustafsson\n\nphone: +46 733-608274\ne-mail: iveqy@iveqy.com\nwebsite: http://www.iveqy.com\n"},{"id":"290593","messageId":"CAPV8XuZ4wTWBPkwB4grmx-oznnx2koiCFvVLZ-oG+E2v1ipPBw@mail.gmail.com","threadId":"42667","inReplyTo":"20160622153145.GB16644@paksenarrion.iveqy.com","subject":"Re: Problem with --shallow-submodules option","fromName":"Istvan Zakar","fromEmail":"istvan.zakar@gmail.com","sentAt":"2016-06-30T13:27:14Z","receivedAt":"2016-06-30T13:27:19Z","isPatch":false,"sender":{"key":"istvan.zakar@gmail.com","avatar":null},"body":"Hello,\n\nThanks for your answers. I tested it after the changes were made on\nthe git server, and it seems to be working. But some other issue came\nup.\n\nWe have quite many submodules in our project so I did some comaprision:\n\nIf I do a clone with these parameters:\n--jobs 20 --recurse-submodules\n\nThe clone lasts ~53 seconds, and the total size of the folder is around 2 GB.\n\nIf I add the shallow-submodules option, the size of the folder will be\na bit below 1GB, so the size decreased as I expected, but the time of\nthe clone itself increased to 90 seconds. It seems the last step of\nthe command, checking out the submodules is executed one-by-one, and\nnot in parallel, so it seems at this step the jobs parameter does not\nhave effect.\n\nIs it intentional, or there is some option I missed?\n\nI'm using git 2.9.0 on client side.\n\nThanks,\n Istvan\n\nps: if I update the submodules with --depth 1 parameter in parallel\nusing xargs it lasts about 18 seconds, so it's a workaround for this\nissue, but it would be nice to do it with a single command.\n\n\n\n\nOn 22 June 2016 at 17:31, Fredrik Gustafsson wrote:\n> On Mon, Jun 20, 2016 at 01:06:39PM +0000, Istvan Zakar wrote:\n>> I'm working on a relatively big project with many submodules. During\n>> cloning for testing I tried to decrease the amount of data need to be\n>> fetched from the server by using --shallow-submodules option in the clone\n>> command. It seems to check out the tip of the remote repo, and if it's not\n>> the commit registered in the superproject the submodule update fails\n>> (obviously). Can I somehow tell to fetch that exact commit I need for my\n>> superproject?\n>\n> Maybe. http://stackoverflow.com/questions/2144406/git-shallow-submodules\n> gives a good overview of this problem.\n>\n> git fetches a branch and is shallow from that branch, which might be an\n> other sha1 than the one the submodule points to, (as you say). This\n> is/was one of the drawbacks with this method. However the since git 2.8,\n> git will try to fetch the sha1 direct (and not the branch). So then it\n> will work, if(!), the server supports direct access to sha1. This was\n> previously not allowed due to security concerns (if I recall correctly).\n>\n> So the answer is, yes this will work if you've a recent version of git\n> and support on the server side for doing this. Unfortunately I'm not\n> sure which git version is needed on the server side for this to work.\n>\n> --\n> Fredrik Gustafsson\n>\n> phone: +46 733-608274\n> e-mail: iveqy@iveqy.com\n> website: http://www.iveqy.com\n"},{"id":"290616","messageId":"CAGZ79kZRJ-O3825r-oXn4YG7Xp6+SXmttBxhzjADgRK5yNa=xQ@mail.gmail.com","threadId":"42667","inReplyTo":"CAPV8XuZ4wTWBPkwB4grmx-oznnx2koiCFvVLZ-oG+E2v1ipPBw@mail.gmail.com","subject":"Re: Problem with --shallow-submodules option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-06-30T20:57:50Z","receivedAt":"2016-06-30T20:58:18Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jun 30, 2016 at 6:27 AM, Istvan Zakar wrote:\n> Hello,\n>\n> Thanks for your answers. I tested it after the changes were made on\n> the git server, and it seems to be working. But some other issue came\n> up.\n>\n> We have quite many submodules in our project so I did some comaprision:\n>\n> If I do a clone with these parameters:\n> --jobs 20 --recurse-submodules\n>\n> The clone lasts ~53 seconds, and the total size of the folder is around 2 GB.\n>\n> If I add the shallow-submodules option, the size of the folder will be\n> a bit below 1GB, so the size decreased as I expected, but the time of\n> the clone itself increased to 90 seconds. It seems the last step of\n> the command, checking out the submodules is executed one-by-one, and\n> not in parallel, so it seems at this step the jobs parameter does not\n> have effect.\n>\n> Is it intentional, or there is some option I missed?\n\nIt was intentional at the time of submitting the patches.\nThe checkout phase is a bit complicated as it combines the\nnewly cloned submodules as well as the submodules to incrementally\nfetch into one bucket and treats them the same.\n\nAnd for submodules that were fetched incrementally you may run into problems\nwhen combining that with the local state (e.g. rebase or merge configured in\n`submodule..update` or passed on the command line), which requires\nhuman interaction (resolving the merge conflict), which we want to present one\nat a time to the user.\n\nThe handling for the user is not quite clear, when to stop, see:\n15ffb7cde48b73b3d5ce259443db7d2e0ba13750 (submodule update: continue\nwhen a checkout fails)\n877449c136539cf8b9b4ed9cfe33a796b7b93f93 (git-submodule.sh: clarify\nthe \"should we die now\" logic)\n\nSo we want to die as soon as we see a merge conflict or other\nerror that is likely to require some human interaction.\nTo do that properly we need to have complicated logic or just update\none submodule at a time.\n\nFor initial checkouts we know that there will be no merge conflicts, i.e.\nit will be a \"checkout -f\" (with an implicit must_die_on_failure=no)\nSo we could run all checkouts of submodules in parallel, too. We'd\njust need to write the patch for that.\n\nAs the cloning is already done in parallel, we can hook into the initial\ncheckout there easily. I'd build that on top of [1], creating a similar commit.\nIn the successful case of `update_clone_task_finished` (the case with\n`!result` -> return 0;) we would need to add the checkout command to\nthe queue instead of just finishing.\n\n[1] https://github.com/gitster/git/commit/665b35eccd39fefd714cb5c332277a6b94fd9386\n\n\n>\n> I'm using git 2.9.0 on client side.\n>\n> Thanks,\n> Istvan\n>\n> ps: if I update the submodules with --depth 1 parameter in parallel\n> using xargs it lasts about 18 seconds, so it's a workaround for this\n> issue, but it would be nice to do it with a single command.\n>\n>\n>\n>\n> On 22 June 2016 at 17:31, Fredrik Gustafsson wrote:\n>> On Mon, Jun 20, 2016 at 01:06:39PM +0000, Istvan Zakar wrote:\n>>> I'm working on a relatively big project with many submodules. During\n>>> cloning for testing I tried to decrease the amount of data need to be\n>>> fetched from the server by using --shallow-submodules option in the clone\n>>> command. It seems to check out the tip of the remote repo, and if it's not\n>>> the commit registered in the superproject the submodule update fails\n>>> (obviously). Can I somehow tell to fetch that exact commit I need for my\n>>> superproject?\n>>\n>> Maybe. http://stackoverflow.com/questions/2144406/git-shallow-submodules\n>> gives a good overview of this problem.\n>>\n>> git fetches a branch and is shallow from that branch, which might be an\n>> other sha1 than the one the submodule points to, (as you say). This\n>> is/was one of the drawbacks with this method. However the since git 2.8,\n>> git will try to fetch the sha1 direct (and not the branch). So then it\n>> will work, if(!), the server supports direct access to sha1. This was\n>> previously not allowed due to security concerns (if I recall correctly).\n>>\n>> So the answer is, yes this will work if you've a recent version of git\n>> and support on the server side for doing this. Unfortunately I'm not\n>> sure which git version is needed on the server side for this to work.\n>>\n>> --\n>> Fredrik Gustafsson\n>>\n>> phone: +46 733-608274\n>> e-mail: iveqy@iveqy.com\n>> website: http://www.iveqy.com\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"290617","messageId":"CAPV8XuYvt7v9EciZUsb5qgyWDjUFm2fxVfe16xivu-0GcMF8=g@mail.gmail.com","threadId":"42667","inReplyTo":"CAGZ79kZRJ-O3825r-oXn4YG7Xp6+SXmttBxhzjADgRK5yNa=xQ@mail.gmail.com","subject":"Re: Problem with --shallow-submodules option","fromName":"Istvan Zakar","fromEmail":"istvan.zakar@gmail.com","sentAt":"2016-06-30T21:04:59Z","receivedAt":"2016-06-30T21:05:07Z","isPatch":false,"sender":{"key":"istvan.zakar@gmail.com","avatar":null},"body":"Hi,\n\nThanks for the clarification, it makes sense now.\n\nThanks,\n Istvan\n\n\nOn 30 June 2016 at 22:57, Stefan Beller wrote:\n> On Thu, Jun 30, 2016 at 6:27 AM, Istvan Zakar wrote:\n>> Hello,\n>>\n>> Thanks for your answers. I tested it after the changes were made on\n>> the git server, and it seems to be working. But some other issue came\n>> up.\n>>\n>> We have quite many submodules in our project so I did some comaprision:\n>>\n>> If I do a clone with these parameters:\n>> --jobs 20 --recurse-submodules\n>>\n>> The clone lasts ~53 seconds, and the total size of the folder is around 2 GB.\n>>\n>> If I add the shallow-submodules option, the size of the folder will be\n>> a bit below 1GB, so the size decreased as I expected, but the time of\n>> the clone itself increased to 90 seconds. It seems the last step of\n>> the command, checking out the submodules is executed one-by-one, and\n>> not in parallel, so it seems at this step the jobs parameter does not\n>> have effect.\n>>\n>> Is it intentional, or there is some option I missed?\n>\n> It was intentional at the time of submitting the patches.\n> The checkout phase is a bit complicated as it combines the\n> newly cloned submodules as well as the submodules to incrementally\n> fetch into one bucket and treats them the same.\n>\n> And for submodules that were fetched incrementally you may run into problems\n> when combining that with the local state (e.g. rebase or merge configured in\n> `submodule..update` or passed on the command line), which requires\n> human interaction (resolving the merge conflict), which we want to present one\n> at a time to the user.\n>\n> The handling for the user is not quite clear, when to stop, see:\n> 15ffb7cde48b73b3d5ce259443db7d2e0ba13750 (submodule update: continue\n> when a checkout fails)\n> 877449c136539cf8b9b4ed9cfe33a796b7b93f93 (git-submodule.sh: clarify\n> the \"should we die now\" logic)\n>\n> So we want to die as soon as we see a merge conflict or other\n> error that is likely to require some human interaction.\n> To do that properly we need to have complicated logic or just update\n> one submodule at a time.\n>\n> For initial checkouts we know that there will be no merge conflicts, i.e.\n> it will be a \"checkout -f\" (with an implicit must_die_on_failure=no)\n> So we could run all checkouts of submodules in parallel, too. We'd\n> just need to write the patch for that.\n>\n> As the cloning is already done in parallel, we can hook into the initial\n> checkout there easily. I'd build that on top of [1], creating a similar commit.\n> In the successful case of `update_clone_task_finished` (the case with\n> `!result` -> return 0;) we would need to add the checkout command to\n> the queue instead of just finishing.\n>\n> [1] https://github.com/gitster/git/commit/665b35eccd39fefd714cb5c332277a6b94fd9386\n>\n>\n>>\n>> I'm using git 2.9.0 on client side.\n>>\n>> Thanks,\n>> Istvan\n>>\n>> ps: if I update the submodules with --depth 1 parameter in parallel\n>> using xargs it lasts about 18 seconds, so it's a workaround for this\n>> issue, but it would be nice to do it with a single command.\n>>\n>>\n>>\n>>\n>> On 22 June 2016 at 17:31, Fredrik Gustafsson wrote:\n>>> On Mon, Jun 20, 2016 at 01:06:39PM +0000, Istvan Zakar wrote:\n>>>> I'm working on a relatively big project with many submodules. During\n>>>> cloning for testing I tried to decrease the amount of data need to be\n>>>> fetched from the server by using --shallow-submodules option in the clone\n>>>> command. It seems to check out the tip of the remote repo, and if it's not\n>>>> the commit registered in the superproject the submodule update fails\n>>>> (obviously). Can I somehow tell to fetch that exact commit I need for my\n>>>> superproject?\n>>>\n>>> Maybe. http://stackoverflow.com/questions/2144406/git-shallow-submodules\n>>> gives a good overview of this problem.\n>>>\n>>> git fetches a branch and is shallow from that branch, which might be an\n>>> other sha1 than the one the submodule points to, (as you say). This\n>>> is/was one of the drawbacks with this method. However the since git 2.8,\n>>> git will try to fetch the sha1 direct (and not the branch). So then it\n>>> will work, if(!), the server supports direct access to sha1. This was\n>>> previously not allowed due to security concerns (if I recall correctly).\n>>>\n>>> So the answer is, yes this will work if you've a recent version of git\n>>> and support on the server side for doing this. Unfortunately I'm not\n>>> sure which git version is needed on the server side for this to work.\n>>>\n>>> --\n>>> Fredrik Gustafsson\n>>>\n>>> phone: +46 733-608274\n>>> e-mail: iveqy@iveqy.com\n>>> website: http://www.iveqy.com\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"}]}