{"thread":{"id":"27838","subject":"git-ftp: retry, sftp support","startedAt":"2011-07-18T02:35:54Z","lastAt":"2011-07-18T10:51:33Z","messageCount":5,"participants":["Martin Langhoff","Jakub Narebski","Timo Besenreuther"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"171534","messageId":"CACPiFCL22yr096_nNfjvfP_bkJRC7HA65GUF12wedzV=cz-_kg@mail.gmail.com","threadId":"27838","inReplyTo":null,"subject":"git-ftp: retry, sftp support","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2011-07-18T02:35:54Z","receivedAt":"2011-07-18T02:35:54Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"René, Timo,\n\nThanks for git-ftp -- it has saved me from going crazy with low cost\nhosting setups that only support ftp.\n\nAlong the way, I scratched some personal itches; so here are the patches.\n\nOne thing I have not been able to fix: it uses one connection per\nfile. This creates significant overhead for large trees, and in\nsome cases it triggers firewalls' DoS countermeasures. An initial\nattempt at fixing it using ncftpspooler did not pan out. It will\nprobably require rewriting the script in Python or Perl to use\nlibcurl bindings.\n\n[ Unfortunately, I am behind the great fw of china right now, and I\ncan't seem to smtp out via gmail from here. So patches attached. ]\n\ncheers,\n\n\nmartin\n\n\n\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- Software Architect - OLPC\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n\n\nFrom e193aa15ae0b59296a04dc9910d36c7d9059ff3f Mon Sep 17 00:00:00 2001\nFrom: Martin Langhoff \nDate: Sun, 10 Jul 2011 11:14:50 -0400\nSubject: [PATCH 1/2] Add support for --retry parameter -- which gets passed\n along to curl\n\nMakes a difference with unreliable connections and hosts.\n---\n git-ftp.sh | 27 ++++++++++++++++++++++++---\n 1 files changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/git-ftp.sh b/git-ftp.sh\nindex cfadc5f..5dda3a4 100755\n--- a/git-ftp.sh\n+++ b/git-ftp.sh\n@@ -58,6 +58,7 @@ OPTIONS:\n -D, --dry-run Dry run: Does not upload anything\n -a, --all Uploads all files, ignores deployed SHA1 hash\n -c, --catchup Updates SHA1 hash without uploading files\n+ -r, --retry Number of retries (see curl manpage)\n -f, --force Force, does not ask questions\n -v, --verbose Verbose\n \n@@ -123,17 +124,17 @@ upload_file() {\n if [ -z ${dest_file} ]; then\n dest_file=${source_file}\n fi\n- ${CURL_BIN} -T ${source_file} --user ${REMOTE_USER}:${REMOTE_PASSWD} --ftp-create-dirs -# ftp://${REMOTE_HOST}/${REMOTE_PATH}${dest_file}\n+ ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -T ${source_file} --user ${REMOTE_USER}:${REMOTE_PASSWD} --ftp-create-dirs -# ftp://${REMOTE_HOST}/${REMOTE_PATH}${dest_file}\n }\n \n remove_file() {\n file=${1}\n- ${CURL_BIN} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} -Q \"-DELE ${REMOTE_PATH}${file}\" ftp://${REMOTE_HOST} > /dev/null 2>&1\n+ ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} -Q \"-DELE ${REMOTE_PATH}${file}\" ftp://${REMOTE_HOST} > /dev/null 2>&1\n }\n \n get_file_content() {\n source_file=${1}\n- ${CURL_BIN} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} ftp://${REMOTE_HOST}/${REMOTE_PATH}${source_file}\n+ ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} ftp://${REMOTE_HOST}/${REMOTE_PATH}${source_file}\n }\n \n while test $# != 0\n@@ -176,6 +177,22 @@ do\n ;;\n esac\n ;;\n+ -r|--retry*)\n+ case \"$#,$1\" in\n+ *,*=*)\n+ CURL_RETRY_INT=`expr \"z$1\" : 'z-[^=]*=\\(.*\\)'`\n+ ;;\n+ 1,*)\n+ usage\n+ ;;\n+ *)\n+ if [ ! `echo \"${2}\" | egrep '^-' | wc -l` -eq 1 ]; then\n+ CURL_RETRY_INT=\"$2\"\n+ shift\n+ fi\n+ ;;\n+\t esac\n+\t ;;\n -a|--all)\n IGNORE_DEPLOYED=1\n ;;\n@@ -309,6 +326,10 @@ write_log \"Host is '${REMOTE_HOST}'\"\n write_log \"User is '${REMOTE_USER}'\"\n write_log \"Path is '${REMOTE_PATH}'\"\n \n+if [ -n \"$CURL_RETRY_INT\" ];then\n+ CURL_RETRY='--retry'\n+fi\n+\n DEPLOYED_SHA1=\"\"\n if [ ${IGNORE_DEPLOYED} -ne 1 ]; then\n # Get the last commit (SHA) we deployed if not ignored or not found\n-- \n1.7.6\n\n\n\nFrom c2bbf06d43bfb4fd0df461f09ccd0dd895282d04 Mon Sep 17 00:00:00 2001\nFrom: Martin Langhoff \nDate: Sun, 10 Jul 2011 12:57:49 -0400\nSubject: [PATCH 2/2] Add sftp support\n\ncurl already does the heavy lifting. This is non-ideal\nin that it passes the password around in the cmdline.\n\nThis is shell however -- almost anything we do with a password means\npassing params around in horrifying ways.\n\nPending: a way to use ssh keys.\n---\n git-ftp.sh | 17 +++++++++--------\n 1 files changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/git-ftp.sh b/git-ftp.sh\nindex 5dda3a4..9c57e4d 100755\n--- a/git-ftp.sh\n+++ b/git-ftp.sh\n@@ -49,7 +49,7 @@ DESCRIPTION:\n \n URL:\n . default host.example.com[:][/]\n- . FTP ftp://host.example.com[:][/]\n+ . FTP (s)ftp://host.example.com[:][/]\n \n OPTIONS:\n -h, --help Show this message\n@@ -65,6 +65,7 @@ OPTIONS:\n EXAMPLES:\n . git ftp -u john ftp://ftp.example.com:4445/public_ftp -p -v\n . git ftp -p -u john -v ftp.example.com:4445:/public_ftp \n+ . git ftp -u john sftp://ftp.secureexample.com/public_ftp -p -v\n EOF\n exit 0\n }\n@@ -124,17 +125,17 @@ upload_file() {\n if [ -z ${dest_file} ]; then\n dest_file=${source_file}\n fi\n- ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -T ${source_file} --user ${REMOTE_USER}:${REMOTE_PASSWD} --ftp-create-dirs -# ftp://${REMOTE_HOST}/${REMOTE_PATH}${dest_file}\n+ ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -T ${source_file} --user ${REMOTE_USER}:${REMOTE_PASSWD} --ftp-create-dirs -# ${REMOTE_PROTOCOL}://${REMOTE_HOST}/${REMOTE_PATH}${dest_file}\n }\n \n remove_file() {\n file=${1}\n- ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} -Q \"-DELE ${REMOTE_PATH}${file}\" ftp://${REMOTE_HOST} > /dev/null 2>&1\n+ ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} -Q \"-DELE ${REMOTE_PATH}${file}\" ${REMOTE_PROTOCOL}://${REMOTE_HOST} > /dev/null 2>&1\n }\n \n get_file_content() {\n source_file=${1}\n- ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} ftp://${REMOTE_HOST}/${REMOTE_PATH}${source_file}\n+ ${CURL_BIN} ${CURL_RETRY} ${CURL_RETRY_INT} -s --user ${REMOTE_USER}:${REMOTE_PASSWD} ${REMOTE_PROTOCOL}://${REMOTE_HOST}/${REMOTE_PATH}${source_file}\n }\n \n while test $# != 0\n@@ -305,7 +306,7 @@ if [ ${HAS_ERROR} -ne 0 ]; then\n fi\n \n # Split protocol from url \n-REMOTE_PROTOCOL=`expr \"${URL}\" : \"\\(ftp\\).*\"`\n+REMOTE_PROTOCOL=`expr \"${URL}\" : \"\\(sftp\\|ftp\\).*\"`\n \n # Check supported protocol\n if [ -z ${REMOTE_PROTOCOL} ]; then\n@@ -333,7 +334,7 @@ fi\n DEPLOYED_SHA1=\"\"\n if [ ${IGNORE_DEPLOYED} -ne 1 ]; then\n # Get the last commit (SHA) we deployed if not ignored or not found\n- write_log \"Retrieving last commit from ftp://${REMOTE_HOST}/${REMOTE_PATH}\"\n+ write_log \"Retrieving last commit from ${REMOTE_PROTOCOL}://${REMOTE_HOST}/${REMOTE_PATH}\"\n DEPLOYED_SHA1=\"`get_file_content ${DEPLOYED_SHA1_FILE}`\"\n if [ $? -ne 0 ]; then\n write_info \"Could not get last commit or it does not exist\"\n@@ -394,7 +395,7 @@ if [ $CATCHUP -ne 1 ]; then\n # File exits?\n if [ -f ${file} ]; then \n # Uploading file\n- write_info \"[${done_items} of ${total_items}] Uploading ${file} to ftp://${REMOTE_HOST}/${REMOTE_PATH}${file}\"\n+ write_info \"[${done_items} of ${total_items}] Uploading ${file} to ${REMOTE_PROTOCOL}://${REMOTE_HOST}/${REMOTE_PATH}${file}\"\n if [ ${DRY_RUN} -ne 1 ]; then\n upload_file ${file}\n check_exit_status \"Could not upload\"\n@@ -414,7 +415,7 @@ fi\n \n # if successful, remember the SHA1 of last commit\n DEPLOYED_SHA1=`${GIT_BIN} log -n 1 --pretty=%H`\n-write_info \"Uploading commit log to ftp://${REMOTE_HOST}/${REMOTE_PATH}${DEPLOYED_SHA1_FILE}\"\n+write_info \"Uploading commit log to ${REMOTE_PROTOCOL}://${REMOTE_HOST}/${REMOTE_PATH}${DEPLOYED_SHA1_FILE}\"\n if [ ${DRY_RUN} -ne 1 ]; then\n echo \"${DEPLOYED_SHA1}\" | upload_file - ${DEPLOYED_SHA1_FILE}\n check_exit_status \"Could not upload\"\n-- \n1.7.6\n\n"},{"id":"171559","messageId":"m3vcv0yqb4.fsf@localhost.localdomain","threadId":"27838","inReplyTo":"CACPiFCL22yr096_nNfjvfP_bkJRC7HA65GUF12wedzV=cz-_kg@mail.gmail.com","subject":"Re: git-ftp: retry, sftp support","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-07-18T07:57:35Z","receivedAt":"2011-07-18T07:57:35Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Martin Langhoff writes:\n\n> René, Timo,\n> \n> Thanks for git-ftp -- it has saved me from going crazy with low cost\n> hosting setups that only support ftp.\n\nCould you give us a link? It isn't in git core, is it?\n\nHow git-ftp differs from ftp / ftps remote helper (git-remote-ftp etc.)?\n\n\n-- \nJakub Narębski\n"},{"id":"171563","messageId":"CACPiFCK4B96cTAz8JEBMOpktZG2R5HsYA2YgEdRuqO55XTOVcg@mail.gmail.com","threadId":"27838","inReplyTo":"m3vcv0yqb4.fsf@localhost.localdomain","subject":"Re: git-ftp: retry, sftp support","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2011-07-18T08:09:59Z","receivedAt":"2011-07-18T08:09:59Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"2011/7/18 Jakub Narebski :\n> Martin Langhoff writes:\n>\n>> René, Timo,\n>>\n>> Thanks for git-ftp -- it has saved me from going crazy with low cost\n>> hosting setups that only support ftp.\n>\n> Could you give us a link?  It isn't in git core, is it?\n\ngit remote -v\norigin\tgit://github.com/BeezyT/git-ftp.git (fetch)\norigin\tgit://github.com/BeezyT/git-ftp.git (push)\n\n> How git-ftp differs from ftp / ftps remote helper (git-remote-ftp etc.)?\n\nAIUI, the ftp/ftps remote helpers are to keep a git _repo_ on a server\nthat runs ftp.\n\nThis git-ftp is a \"deploy the tip of my branch onto a production\nserver\" tool. The usage model is\n\n - hack on your html/php website on your dev machine, in a git checkout\n - commit your code\n - use git-ftp to publish to the hosting server\n\nMaybe it should be called \"git ftpdeploy\". It's a handy trick. I\nfound it via http://stackoverflow.com/questions/2950107/git-push-into-production-ftp\n\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- Software Architect - OLPC\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"171567","messageId":"138AFDD9-CE91-4019-BAF3-B6F301D4225E@gmail.com","threadId":"27838","inReplyTo":"CACPiFCK4B96cTAz8JEBMOpktZG2R5HsYA2YgEdRuqO55XTOVcg@mail.gmail.com","subject":"Re: git-ftp: retry, sftp support","fromName":"Timo Besenreuther","fromEmail":"timo.besenreuther@gmail.com","sentAt":"2011-07-18T08:58:24Z","receivedAt":"2011-07-18T08:58:24Z","isPatch":false,"sender":{"key":"timo.besenreuther@gmail.com","avatar":null},"body":"Hey guys,\n\nI think there's some confusion here.\n\ngit-ftp is a bash tool written by Rene Moser (github resmo) to sync a git repo with an FTP server.\nIt is not in git core. I just added some minor improvements. My github repo is just a fork.\n\nRegards,\nTimo\n\n\nOn Jul 18, 2011, at 10:09 AM, Martin Langhoff wrote:\n\n> 2011/7/18 Jakub Narebski :\n>> Martin Langhoff writes:\n>> \n>>> René, Timo,\n>>> \n>>> Thanks for git-ftp -- it has saved me from going crazy with low cost\n>>> hosting setups that only support ftp.\n>> \n>> Could you give us a link? It isn't in git core, is it?\n> \n> git remote -v\n> origin\tgit://github.com/BeezyT/git-ftp.git (fetch)\n> origin\tgit://github.com/BeezyT/git-ftp.git (push)\n> \n>> How git-ftp differs from ftp / ftps remote helper (git-remote-ftp etc.)?\n> \n> AIUI, the ftp/ftps remote helpers are to keep a git _repo_ on a server\n> that runs ftp.\n> \n> This git-ftp is a \"deploy the tip of my branch onto a production\n> server\" tool. The usage model is\n> \n> - hack on your html/php website on your dev machine, in a git checkout\n> - commit your code\n> - use git-ftp to publish to the hosting server\n> \n> Maybe it should be called \"git ftpdeploy\". It's a handy trick. I\n> found it via http://stackoverflow.com/questions/2950107/git-push-into-production-ftp\n> \n> \n> \n> \n> m\n> -- \n> martin.langhoff@gmail.com\n> martin@laptop.org -- Software Architect - OLPC\n> - ask interesting questions\n> - don't get distracted with shiny stuff - working code first\n> - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"171569","messageId":"CACPiFCJHx7Nb2N+FfL6Cu4ULEqwhJxY4-GUNWLA2ds8zL8WfwA@mail.gmail.com","threadId":"27838","inReplyTo":"138AFDD9-CE91-4019-BAF3-B6F301D4225E@gmail.com","subject":"Re: git-ftp: retry, sftp support","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2011-07-18T10:51:33Z","receivedAt":"2011-07-18T10:51:33Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Mon, Jul 18, 2011 at 4:58 AM, Timo Besenreuther\n wrote:\n> git-ftp is a bash tool written by Rene Moser (github resmo) to sync a git repo with an FTP server.\n> It is not in git core. I just added some minor improvements. My github repo is just a fork.\n\nYup - I know it's not in git core, but I enjoy using it, so wanted to\nraise awareness of it, as well of https://github.com/ezyang/git-ftp\n\nHave you seen the patches? Mergeable?\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- Software Architect - OLPC\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"}]}