{"thread":{"id":"40314","subject":"--progress option for git submodule update?","startedAt":"2015-09-09T22:52:53Z","lastAt":"2015-09-11T23:34:21Z","messageCount":5,"participants":["Vitali Lovich","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"269671","messageId":"88E7FC00-9A87-4E20-89D8-4BF5997F7B07@gmail.com","threadId":"40314","inReplyTo":null,"subject":"--progress option for git submodule update?","fromName":"Vitali Lovich","fromEmail":"vlovich@gmail.com","sentAt":"2015-09-09T22:52:53Z","receivedAt":"2015-09-09T22:52:53Z","isPatch":false,"sender":{"key":"vlovich@gmail.com","avatar":null},"body":"Hi,\n\nGit submodule doesn’t have a --progress option like regular clone/fetch does. This means that it can hang a long time without output as it’s transferring data, particularly for large repositories.\nThis is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).\n\nI’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of. The only alternative is a hacky solution to clone locally first with the —progress option\n& then somehow set up the submodule to use the local clone as a reference.\n\nThanks,\nVitali"},{"id":"269672","messageId":"CAGZ79kYRYqVE35_i5+DvqOj7G6LvhBQgsQok5gabLY6x20F80w@mail.gmail.com","threadId":"40314","inReplyTo":"88E7FC00-9A87-4E20-89D8-4BF5997F7B07@gmail.com","subject":"Re: --progress option for git submodule update?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-09-09T23:40:32Z","receivedAt":"2015-09-09T23:40:32Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"So submodules...\n\nI am currently working on improving submodules (some basic performance\nimprovements have been done, soon to be merged upstream, I currently\ntry to get parallelism working for git fetch --recurse-submodules and for\ngit submodule update eventually. I sent some early working patches for that,\nbut I am doing a whole new redesign without threads now).\n\nOn Wed, Sep 9, 2015 at 3:52 PM, Vitali Lovich wrote:\n> Hi,\n>\n> Git submodule doesn’t have a --progress option like regular clone/fetch does. This means that it can hang a long time without output as it’s transferring data, particularly for large repositories.\n\nFor repositories with nested submodules it is impossible to estimate\nthe progress because you don't know how many there are.\nSay you have a layout like:\n\nA\n -> B\n-> C\n-> D\n -> E\n -> F\n\nwhereas each letter is a repository and B,C,D are submodules of A and\nE,F are submodules of D.\nSo if D is not cloned yet, it looks like A has only 3 submodules, but\nin fact we need to update 5\nsubmodules.\n\n> This is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).\n\nMaybe a better error-out-if-hanging would be better IMHO ?\nAnother option would be to enumerate the submodules and give the\ncurrently estimated upper bound ?\n\nDoh! I see what you're missing now after rereading the email closely.\nYou can add a --quiet option,\nbut --verbose or --progress just errors out, but you want that as a\npossible argument for git clone\ninside the git submodule update code.\n\nThanks,\nStefan\n\n>\n> I’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of. The only alternative is a hacky solution to clone locally first with the —progress option\n> & then somehow set up the submodule to use the local clone as a reference.\n>\n> Thanks,\n> Vitali--\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"269675","messageId":"68DDAE70-85F2-4873-BDBD-373985A49815@gmail.com","threadId":"40314","inReplyTo":"CAGZ79kYRYqVE35_i5+DvqOj7G6LvhBQgsQok5gabLY6x20F80w@mail.gmail.com","subject":"Re: --progress option for git submodule update?","fromName":"Vitali Lovich","fromEmail":"vlovich@gmail.com","sentAt":"2015-09-10T03:06:03Z","receivedAt":"2015-09-10T03:06:03Z","isPatch":false,"sender":{"key":"vlovich@gmail.com","avatar":null},"body":"\n> On Sep 9, 2015, at 4:40 PM, Stefan Beller wrote:\n> \n> So submodules...\n> \n> I am currently working on improving submodules (some basic performance\n> improvements have been done, soon to be merged upstream, I currently\n> try to get parallelism working for git fetch --recurse-submodules and for\n> git submodule update eventually. I sent some early working patches for that,\n> but I am doing a whole new redesign without threads now).\nThat sounds exciting. Can’t wait.\n\n> On Wed, Sep 9, 2015 at 3:52 PM, Vitali Lovich wrote:\n>> Hi,\n>> \n>> Git submodule doesn’t have a --progress option like regular clone/fetch does. This means that it can hang a long time without output as it’s transferring data, particularly for large repositories.\n> \n> For repositories with nested submodules it is impossible to estimate\n> the progress because you don't know how many there are.\n> Say you have a layout like:\n> \n> A\n> -> B\n> -> C\n> -> D\n> -> E\n> -> F\n> \n> whereas each letter is a repository and B,C,D are submodules of A and\n> E,F are submodules of D.\n> So if D is not cloned yet, it looks like A has only 3 submodules, but\n> in fact we need to update 5\n> submodules.\nI don’t think I’m asking for an overall --progress. As you point out that is very difficult/an intractable problem. I was thinking it would just report the progress for each submodule it encounters as it fetches/clones.\nThe added benefit to that is that if there’s a lot of submodules, an overall progress might get stuck at a long time at a given percentage whereas it’s less likely cloning just a single module would depending on the size of repositories.\n\n>> This is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).\n> \n> Maybe a better error-out-if-hanging would be better IMHO ?\n> Another option would be to enumerate the submodules and give the\n> currently estimated upper bound ?\nThat’s much more difficult & I’m still left with the original problem where I have to set a very conservative upper-bound which wastes valuable machine time & causes extra contention for automation resources.\n\n> Doh! I see what you're missing now after rereading the email closely.\n> You can add a --quiet option,\n> but --verbose or --progress just errors out, but you want that as a\n> possible argument for git clone\n> inside the git submodule update code.\nYes exactly.\n\n> \n> Thanks,\n> Stefan\n> \n>> \n>> I’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of. The only alternative is a hacky solution to clone locally first with the —progress option\n>> & then somehow set up the submodule to use the local clone as a reference.\n>> \n>> Thanks,\n>> Vitali--\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"269803","messageId":"CAGZ79kbH+917v6pmCC3w4rovEVarHp+w1tYthMwkMU2hrq=VdQ@mail.gmail.com","threadId":"40314","inReplyTo":"68DDAE70-85F2-4873-BDBD-373985A49815@gmail.com","subject":"Re: --progress option for git submodule update?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-09-11T23:05:16Z","receivedAt":"2015-09-11T23:05:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Sep 9, 2015 at 8:06 PM, Vitali Lovich wrote:\n>> Doh! I see what you're missing now after rereading the email closely.\n>> You can add a --quiet option,\n>> but --verbose or --progress just errors out, but you want that as a\n>> possible argument for git clone\n>> inside the git submodule update code.\n> Yes exactly.\n\nInstead of cloning with submodules, you could also clone only the\nsuperproject and then do a git fetch --recuse-submodules=yes -v\ninstead soonish.\n"},{"id":"269811","messageId":"8B026E5E-5303-45EC-984D-3F944DB6F2DB@gmail.com","threadId":"40314","inReplyTo":"CAGZ79kbH+917v6pmCC3w4rovEVarHp+w1tYthMwkMU2hrq=VdQ@mail.gmail.com","subject":"Re: --progress option for git submodule update?","fromName":"Vitali Lovich","fromEmail":"vlovich@gmail.com","sentAt":"2015-09-11T23:34:21Z","receivedAt":"2015-09-11T23:34:21Z","isPatch":false,"sender":{"key":"vlovich@gmail.com","avatar":null},"body":"& then do a git submodule init?\n\n> On Sep 11, 2015, at 4:05 PM, Stefan Beller wrote:\n> \n> On Wed, Sep 9, 2015 at 8:06 PM, Vitali Lovich wrote:\n>>> Doh! I see what you're missing now after rereading the email closely.\n>>> You can add a --quiet option,\n>>> but --verbose or --progress just errors out, but you want that as a\n>>> possible argument for git clone\n>>> inside the git submodule update code.\n>> Yes exactly.\n> \n> Instead of cloning with submodules, you could also clone only the\n> superproject and then do a git fetch --recuse-submodules=yes -v\n> instead soonish.\n"}]}