{"thread":{"id":"49033","subject":"[RFC PATCH 0/1] Introduce git-recover","startedAt":"2018-08-04T14:23:01Z","lastAt":"2018-08-05T01:34:28Z","messageCount":8,"participants":["Edward Thomson","Junio C Hamano","Robert P. J. Day","Todd Zullinger","Jonathan Nieder"],"isPatch":true,"patchVersion":1,"patchTotal":1},"messages":[{"id":"354488","messageId":"20180804142247.GA7@e3c0ce5ceb57","threadId":"49033","inReplyTo":null,"subject":"[RFC PATCH 0/1] Introduce git-recover","fromName":"Edward Thomson","fromEmail":"ethomson@edwardthomson.com","sentAt":"2018-08-04T14:22:47Z","receivedAt":"2018-08-04T14:23:01Z","isPatch":true,"sender":{"key":"ethomson@edwardthomson.com","avatar":"https://avatars.githubusercontent.com/u/1130014?v=4"},"body":"Hello-\n\nI created a simple shell script a while back to help people recover\nfiles that they deleted from their working directory (but had been added\nto the repository), which looks for unreachable blobs in the object\ndatabase and places them in the working directory (either en masse,\ninteractively, or via command-line arguments).\n\nThis has been available at https://github.com/ethomson/git-recover for\nabout a year, and in that time, someone has suggested that I propose\nthis as part of git itself.  So I thought I'd see if there's any\ninterest in this.\n\nIf there is, I'd like to get a sense of the amount of work required to\nmake this suitable for inclusion.  There are some larger pieces of work\nrequired -- at a minimum, I think this requires:\n\n- Tests -- there are none, which is fine with me but probably less fine\n  for inclusion here.\n- Documentation -- the current README is below but it will need proper\n  documentation that can be rendered into manpages, etc, by the tools.\n- Remove bashisms -- there are many.\n\nAgain, this may not be particularly interesting, but I thought I'd send\nit along in case it is.\n\nCheers-\n-ed\n\n-----------------------------------------------------------------------\n\ngit-recover allows you to recover some files that you've accidentally\ndeleted from your working directory.  It helps you find files that exist\nin the repository's object database - because you ran git add - but were\nnever committed.\n\nGetting Started\n---------------\nThe simplest way to use git-recover is in interactive mode - simply run\n`git-recover -i` and it will show you all the files that you can recover\nand prompt you to act.\n\nUsing git-recover\n-----------------\nRunning git-recover without any arguments will list all the files (git\n\"blobs\") that were recently orphaned, by their ID.  (Their filename is not \nknown.)\n\nYou can examine these blobs by running `git show <objectid>`.  If you\nfind one that you want to recover, you can provide the ID as the argument\nto git-recover.  You can specify the `--filename` option to write the\nfile out and apply any filters that are set up in the repository.  For\nexample:\n\n    git-recover 38762cf7f55934b34d179ae6a4c80cadccbb7f0a \\\n        --filename shattered.pdf\n\nYou can also specify multiple files to recover, each with an optional\noutput filename:\n\n    git-recover 38762c --filename one.txt cafebae --filename bae.txt\n\nIf you want to recover _all_ the orphaned blobs in your repository, run\n`git-recover --all`.  This will write all the orphaned files to the\ncurrent working directory, so it's best to run this inside a temporary\ndirectory beneath your working directory.  For example:\n\n    mkdir _tmp && cd _tmp && git-recover --all\n\nBy default, git-recover limits itself to recently created orphaned blobs.\nIf you want to see _all_ orphaned files that have been created in your\nrepository (but haven't yet been garbage collected), you can run:\n\n    git-recover --full\n\nOptions\n-------\n    git-recover [-a] [-i] [--full] [<id> [-f <filename>] ...]\n\n-a, --all\nWrite all orphaned blobs to the current working directory.  Each file will\nbe named using its 40 character object ID.\n\n-i, --interactive\nDisplay information about each orphaned blob and prompt to recover it.\n\n--full\nList or recover all orphaned blobs, even those that are in packfiles.  By \ndefault, `git-recover` will only look at loose object files, which limits\nit to the most recently created files.  Examining packfiles may be slow,\nespecially in large repositories.\n\n<id>\nThe object ID (or its abbreviation) to recover.  The file will be written to\nthe current working directory and named using its 40 character object ID,\nunless the `-f` option is specified.\n\n-f <filename>, --filename <filename>\nWhen specified after an object ID, the file written will use this filename.\nIn addition, any filters (for example: CRLF conversion or Git-LFS) will be\nrun according to the `gitattributes` configuration.\n\n\nEdward Thomson (1):\n  recover: restoration of deleted worktree files\n\n git-recover.sh | 311 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 311 insertions(+)\n create mode 100755 git-recover.sh\n\n-- \n2.0.0 (libgit2)\n\n"},{"id":"354489","messageId":"20180804142416.GA6@5f28dc333bbd","threadId":"49033","inReplyTo":"20180804142247.GA7@e3c0ce5ceb57","subject":"[RFC PATCH 1/1] recover: restoration of deleted worktree files","fromName":"Edward Thomson","fromEmail":"ethomson@edwardthomson.com","sentAt":"2018-08-04T14:24:16Z","receivedAt":"2018-08-04T14:24:28Z","isPatch":true,"sender":{"key":"ethomson@edwardthomson.com","avatar":"https://avatars.githubusercontent.com/u/1130014?v=4"},"body":"Introduce git-recover, a simple script to aide in restoration of deleted\nworktree files.  This will look for unreachable blobs in the object\ndatabase and prompt users to restore them to disk, either interactively\nor on the command-line.\n---\n git-recover.sh | 311 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 311 insertions(+)\n create mode 100755 git-recover.sh\n\ndiff --git a/git-recover.sh b/git-recover.sh\nnew file mode 100755\nindex 000000000..651d4116f\n--- /dev/null\n+++ b/git-recover.sh\n@@ -0,0 +1,311 @@\n+#!/usr/bin/env bash\n+#\n+# This program helps recover files in your repository that were deleted\n+# from the working tree.\n+#\n+# Copyright (c) 2017-2018 Edward Thomson.\n+\n+set -e\n+\n+IFS=$'\\n'\n+\n+PROGNAME=$(echo \"$0\" | sed -e 's/.*\\///')\n+GIT_DIR=$(git rev-parse --git-dir)\n+\n+DO_RECOVER=0\n+DO_FULL=0\n+DO_INTERACTIVE=0\n+BLOBS=()\n+FILENAMES=()\n+\n+function die_usage {\n+\techo \"usage: $PROGNAME [-a] [-i] [--full] [<id> [-f <filename>] ...]\" >&2\n+\texit 1\n+}\n+\n+while [[ $# -gt 0 ]]; do\n+\tcase \"$1\" in\n+\t-a|--all)\n+\t\tDO_RECOVER=1\n+\t\t;;\n+\t-i|--interactive)\n+\t\tDO_INTERACTIVE=1\n+\t\t;;\n+\t--full)\n+\t\tDO_FULL=1\n+\t\t;;\n+\t*)\n+\t\tif [ \"${1:0:1}\" == \"-\" ]; then\n+\t\t\techo \"$PROGNAME: unknown argument: $1\" >&2\n+\t\t\tdie_usage\n+\t\tfi\n+\t\tBLOBS+=(\"$1\")\n+\n+\t\tshift\n+\t\tif [ \"$1\" == \"-f\" ] || [ \"$1\" == \"--filename\" ]; then\n+\t\t\tshift\n+\t\t\tif [ $# == 0 ]; then\n+\t\t\t\tdie_usage\n+\t\t\tfi\n+\t\t\tFILENAMES+=(\"$1\")\n+\t\t\tshift\n+\t\telse\n+\t\t\tFILENAMES+=(\"\")\n+\t\tfi\n+\t\tcontinue\n+\t;;\n+\tesac\n+\tshift\n+done\n+\n+if [ ${#BLOBS[@]} != 0 ] && [ $DO_RECOVER == 1 ]; then\n+\tdie_usage\n+elif [ ${#BLOBS[@]} != 0 ]; then\n+\tDO_RECOVER=1\n+fi\n+\n+case \"$OSTYPE\" in\n+\tdarwin*|freebsd*) IS_BSD=1 ;;\n+\t*) IS_BSD=0 ;;\n+esac\n+\n+function expand_given_blobs() {\n+\tfor i in \"${!BLOBS[@]}\"; do\n+\t\tID=$(git rev-parse --verify \"${BLOBS[$i]}\" 2>/dev/null || true)\n+\n+\t\tif [ -z \"$ID\" ]; then\n+\t\t\techo \"$PROGNAME: ${BLOBS[$i]} is not a valid object.\" 1>&2\n+\t\t\texit 1\n+\t\tfi\n+\n+\t\tTYPE=$(git cat-file -t \"${ID}\" 2>/dev/null || true)\n+\n+\t\tif [ \"$TYPE\" != \"blob\" ]; then\n+\t\t\techo \"$PROGNAME: ${BLOBS[$i]} is not a blob.\" 1>&2\n+\t\t\texit\n+\t\tfi\n+\n+\t\tBLOBS[$i]=$ID\n+\tdone\n+}\n+\n+# find all the unreachable blobs\n+function find_unreachable() {\n+\tFULLNESS=\"--no-full\"\n+\n+\tif [ $DO_FULL == 1 ]; then FULLNESS=\"--full\"; fi\n+\n+\tBLOBS=($(git fsck --unreachable --no-reflogs \\\n+\t\t\"${FULLNESS}\" --no-progress | sed -ne 's/^unreachable blob //p'))\n+}\n+\n+function read_one_file {\n+\tBLOB=$1\n+\tFILTER_NAME=$2\n+\tARGS=()\n+\n+\tif [ -z \"$FILTER_NAME\" ]; then\n+\t\tARGS+=(\"blob\")\n+\telse\n+\t\tARGS+=(\"--filters\" \"--path=$FILTER_NAME\")\n+\tfi\n+\n+\tgit cat-file \"${ARGS[@]}\" \"$BLOB\"\n+}\n+\n+function write_one_file {\n+\tBLOB=$1\n+\tFILTER_NAME=$2\n+\tOUTPUT_NAME=$3\n+\n+\tABBREV=$(git rev-parse --short \"${BLOB}\")\n+\n+\techo -n \"Writing $ABBREV: \"\n+\tread_one_file \"$BLOB\" \"$FILTER_NAME\" > \"$OUTPUT_NAME\"\n+\techo \"$OUTPUT_NAME.\"\n+}\n+\n+function unique_filename {\n+\tif [ ! -f \"${BLOB}\" ]; then\n+\t\techo \"$BLOB\"\n+\telse\n+\t\tcnt=1\n+\t\twhile true\n+\t\tdo\n+\t\t\tfn=\"${BLOB}~${cnt}\"\n+\t\t\tif [ ! -f \"${fn}\" ]; then\n+\t\t\t\techo \"${fn}\"\n+\t\t\t\tbreak\n+\t\t\tfi\n+\t\t\tcnt=$((cnt+1))\n+\t\tdone\n+\tfi\n+}\n+\n+function write_recoverable {\n+\tfor i in \"${!BLOBS[@]}\"; do\n+\t\tBLOB=${BLOBS[$i]}\n+\t\tFILTER_NAME=${FILENAMES[$i]}\n+\t\tOUTPUT_NAME=${FILENAMES[$i]:-$(unique_filename)}\n+\n+\t\twrite_one_file \"$BLOB\" \"$FILTER_NAME\" \"$OUTPUT_NAME\"\n+\tdone\n+}\n+\n+function file_time {\n+\tif [ $IS_BSD == 1 ]; then\n+\t\tstat -f %c \"$1\"\n+\telse\n+\t\tstat -c %Y \"$1\"\n+\tfi\n+}\n+\n+function timestamp_to_s {\n+\tif [ $IS_BSD == 1 ]; then\n+\t\tdate -r \"$1\"\n+\telse\n+\t\tdate -d @\"$1\"\n+\tfi\n+}\n+\n+function sort_by_timestamp {\n+\t# sort blobs in loose objects by their timestamp (packed blobs last)\n+\tBLOB_AND_TIMESTAMPS=($(for BLOB in \"${BLOBS[@]}\"; do\n+\t\tLOOSE=\"${BLOB::2}/${BLOB:2}\"\n+\t\tTIME=$(file_time \"$GIT_DIR/objects/$LOOSE\" 2>/dev/null || true)\n+\t\techo \"$BLOB $TIME\"\n+\tdone | sort -k2 -r))\n+}\n+\n+function print_recoverable {\n+\techo \"Recoverable orphaned git blobs:\"\n+\techo \"\"\n+\n+\tsort_by_timestamp\n+\tfor BLOB_AND_TIMESTAMP in \"${BLOB_AND_TIMESTAMPS[@]}\"; do\n+\t\tBLOB=${BLOB_AND_TIMESTAMP::40}\n+\t\tTIME=${BLOB_AND_TIMESTAMP:41}\n+\t\tDATE=$([ ! -z \"$TIME\" ] && timestamp_to_s \"$TIME\" || echo \"(Unknown)\") \n+\n+\t\techo \"$BLOB  $DATE\"\n+\tdone\n+}\n+\n+function prompt_for_filename {\n+\twhile true\n+\tdo\n+\t\techo -n \"Filename (return to skip): \"\n+\t\tread -r FILENAME\n+\n+\t\tif [ -f \"$FILENAME\" ]; then\n+\t\t\techo -n \"File exists, overwrite? [y,N]: \"\n+\t\t\tread -r overwrite\n+\n+\t\t\tcase \"$overwrite\" in\n+\t\t\t[yY]*)\n+\t\t\t\treturn 0\n+\t\t\t\t;;\n+\t\t\tesac\n+\n+\t\t\techo\n+\t\telse\n+\t\t\treturn 0\n+\t\tfi\n+\tdone\n+}\n+\n+function view_file {\n+\tread_one_file \"${BLOB}\" | ${PAGER:-less}\n+}\n+\n+function show_summary {\n+\tFILETYPE=$(read_one_file \"${BLOB}\" | file -b -)\n+\tIS_TEXT=$(echo \"${FILETYPE}\" | grep -c ' text$' 2>/dev/null || true)\n+\n+\tif [ \"$IS_TEXT\" == \"1\" ]; then\n+\t\tread_one_file \"${BLOB}\"\n+\telse\n+\t\tread_one_file \"${BLOB}\" | hexdump -C\n+\tfi\n+}\n+\n+function interactive {\n+\techo \"Recoverable orphaned git blobs:\"\n+\n+\tsort_by_timestamp\n+\tfor BLOB_AND_TIMESTAMP in \"${BLOB_AND_TIMESTAMPS[@]}\"; do\n+\t\techo\n+\n+\t\tBLOB=${BLOB_AND_TIMESTAMP::40}\n+\t\tTIME=${BLOB_AND_TIMESTAMP:41}\n+\t\tDATE=$([ ! -z \"$TIME\" ] && timestamp_to_s \"$TIME\" || echo \"(Unknown)\") \n+\n+\t\techo \"$BLOB  ($DATE)\"\n+\t\tshow_summary \"${BLOB}\" | head -4 | sed -e 's/^/> /'\n+\t\techo\n+\n+\t\twhile true\n+\t\tdo\n+\t\t\techo -n \"Recover this file? [y,n,v,f,q,?]: \"\n+\t\t\tread -r ans || return 1\n+\n+\t\t\tcase \"$ans\" in\n+\t\t\t[yY]*)\n+\t\t\t\twrite_one_file \"${BLOB}\" \"\" \"$(unique_filename)\"\n+\t\t\t\tbreak\n+\t\t\t\t;;\n+\t\t\t[nN]*)\n+\t\t\t\tbreak\n+\t\t\t\t;;\n+\t\t\t[vV]*)\n+\t\t\t\tview_file \"${BLOB}\"\n+\t\t\t\techo\n+\t\t\t\t;;\n+\t\t\t[fF]*)\n+\t\t\t\tprompt_for_filename\n+\n+\t\t\t\tif [ \"$FILENAME\" == \"\" ]; then\n+\t\t\t\t\tbreak\n+\t\t\t\tfi\n+\n+\t\t\t\twrite_one_file \"${BLOB}\" \"${FILENAME}\" \"${FILENAME}\"\n+\t\t\t\tbreak\n+\t\t\t\t;;\n+\t\t\t\\?*)\n+\t\t\t\techo\n+\t\t\t\techo \"Do you want to recover this file?\"\n+\t\t\t\techo \" y: yes, write the file to ${BLOB}\"\n+\t\t\t\techo \" n: no, skip this file and see the next orphaned file\"\n+\t\t\t\techo \" v: view the file\"\n+\t\t\t\techo \" f: prompt for a filename to use for recovery\"\n+\t\t\t\techo \" q: quit\"\n+\t\t\t\techo\n+\t\t\t\t;;\n+\t\t\t[qQ]*)\n+\t\t\t\treturn 0\n+\t\t\t\t;;\n+\t\t\tesac\n+\t\tdone\n+\tdone\n+}\n+\n+\n+if [ ${#BLOBS[@]} != 0 ]; then\n+\texpand_given_blobs\n+else\n+\tfind_unreachable\n+fi\n+\n+if [ ${#BLOBS[@]} == 0 ]; then\n+\techo \"$PROGNAME: no recoverable orphaned blobs.\"\n+\texit\n+fi\n+\n+if [ $DO_INTERACTIVE == 1 ]; then\n+\tinteractive\n+elif [ $DO_RECOVER == 1 ]; then\n+\twrite_recoverable\n+else\n+\tprint_recoverable\n+fi\n+\n-- \n2.0.0 (libgit2)\n\n"},{"id":"354491","messageId":"xmqqpnyyt9di.fsf@gitster-ct.c.googlers.com","threadId":"49033","inReplyTo":"20180804142416.GA6@5f28dc333bbd","subject":"Re: [RFC PATCH 1/1] recover: restoration of deleted worktree files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-04T15:54:49Z","receivedAt":"2018-08-04T15:58:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson <ethomson@edwardthomson.com> writes:\n\n> Introduce git-recover, a simple script to aide in restoration of deleted\n> worktree files.  This will look for unreachable blobs in the object\n> database and prompt users to restore them to disk, either interactively\n> or on the command-line.\n> ---\n>  git-recover.sh | 311 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>  1 file changed, 311 insertions(+)\n>  create mode 100755 git-recover.sh\n\nMy first reaction was to say that I am not going to take a new\ncommand written only for bash with full bashism, even if it came\nwith docs, tests nor Makefile integration, for Git itself.  Then I\nreconsidered, as not everything related to Git is git-core, and all\nof the above traits are sign of this patch _not_ meant for git-core.\n\nIn other words, I think this patch can be a fine addition to\nsomebody else's project (i.e. random collection of scripts that may\nhelp Git users), so let's see how I can offer comments/inputs to\nhelp you improve it.  So I won't comment on lang, log message, or\nshell scripting style---these are project convention and the git-core\nconvention won't be relevant to this patch.\n\n> diff --git a/git-recover.sh b/git-recover.sh\n> new file mode 100755\n> index 000000000..651d4116f\n> --- /dev/null\n> +++ b/git-recover.sh\n> @@ -0,0 +1,311 @@\n> +#!/usr/bin/env bash\n> +#\n> +# This program helps recover files in your repository that were deleted\n> +# from the working tree.\n> +#\n> +# Copyright (c) 2017-2018 Edward Thomson.\n> +\n> +set -e\n> +\n> +IFS=$'\\n'\n> +\n> +PROGNAME=$(echo \"$0\" | sed -e 's/.*\\///')\n> +GIT_DIR=$(git rev-parse --git-dir)\n> +\n> +DO_RECOVER=0\n> +DO_FULL=0\n> +DO_INTERACTIVE=0\n> +BLOBS=()\n> +FILENAMES=()\n> +\n> +function die_usage {\n> +\techo \"usage: $PROGNAME [-a] [-i] [--full] [<id> [-f <filename>] ...]\" >&2\n> +\texit 1\n> +}\n> +\n> +while [[ $# -gt 0 ]]; do\n> +\tcase \"$1\" in\n> +\t-a|--all)\n> +\t\tDO_RECOVER=1\n> +\t\t;;\n> +\t-i|--interactive)\n> +\t\tDO_INTERACTIVE=1\n> +\t\t;;\n> +\t--full)\n> +\t\tDO_FULL=1\n> +\t\t;;\n> +\t*)\n> +\t\tif [ \"${1:0:1}\" == \"-\" ]; then\n> +\t\t\techo \"$PROGNAME: unknown argument: $1\" >&2\n> +\t\t\tdie_usage\n> +\t\tfi\n> +\t\tBLOBS+=(\"$1\")\n> +\n> +\t\tshift\n> +\t\tif [ \"$1\" == \"-f\" ] || [ \"$1\" == \"--filename\" ]; then\n> +\t\t\tshift\n> +\t\t\tif [ $# == 0 ]; then\n> +\t\t\t\tdie_usage\n> +\t\t\tfi\n> +\t\t\tFILENAMES+=(\"$1\")\n> +\t\t\tshift\n> +\t\telse\n> +\t\t\tFILENAMES+=(\"\")\n> +\t\tfi\n\nYou do not want to take \"--file=Makefile\" (i.e. abbreviated option\nname, and value as part of the option arg after '=')?\n\n> +\t\tcontinue\n> +\t;;\n> +\tesac\n> +\tshift\n> +done\n\nSo, as a user, I can run this with \"-a\" but no blob object names to\nrun it in DO_RECOVER mode, or I can give one or more \"blob spec\"\nwhere I say object id, optionally followed by one \"-f filename\"; in\nthe latter mode, BLOBS[] and FILENAMES[] array would have the same\nnumber of elements, corresponding to each other.\n\n> +if [ ${#BLOBS[@]} != 0 ] && [ $DO_RECOVER == 1 ]; then\n> +\tdie_usage\n> +elif [ ${#BLOBS[@]} != 0 ]; then\n> +\tDO_RECOVER=1\n> +fi\n\nIf I did not say \"-a\" but did not give \"blob spec\", then I am\nimplicitly asking for \"-a\" to work in DO_RECOVER mode.\n\nI think I understood what the program wants to do so far.\n\n> +case \"$OSTYPE\" in\n> +\tdarwin*|freebsd*) IS_BSD=1 ;;\n> +\t*) IS_BSD=0 ;;\n> +esac\n> +\n> +function expand_given_blobs() {\n> +\tfor i in \"${!BLOBS[@]}\"; do\n> +\t\tID=$(git rev-parse --verify \"${BLOBS[$i]}\" 2>/dev/null || true)\n> +\n> +\t\tif [ -z \"$ID\" ]; then\n> +\t\t\techo \"$PROGNAME: ${BLOBS[$i]} is not a valid object.\" 1>&2\n> +\t\t\texit 1\n> +\t\tfi\n> +\n> +\t\tTYPE=$(git cat-file -t \"${ID}\" 2>/dev/null || true)\n\nAn earlier \"set -e\" makes \"|| true\" ugliness required.  I suspect\nuse of \"set -e\" overall is a loss (vs explicit error checking).\n\n> +\t\tif [ \"$TYPE\" != \"blob\" ]; then\n> +\t\t\techo \"$PROGNAME: ${BLOBS[$i]} is not a blob.\" 1>&2\n> +\t\t\texit\n> +\t\tfi\n\nA user may have given us 11f5bcd9 and this function makes sure such\nan object exists in the object store *and* is a blob.  Otherwise\nit dies.  The main objective of this function is to turn that user\nsupplied object name to a full hex that is known to refer to an\nexisting blob.\n\n> +\t\tBLOBS[$i]=$ID\n> +\tdone\n\nI find a disconnect between this being a loop and the attiude \"we\nwon't tolerate any erroneous input\".  If a user is feeding dozens of\nblob object names, wouldn't it be more helpful to give a warning, go\non and help the user with the rest?\n\n> +}\n> +\n> +# find all the unreachable blobs\n> +function find_unreachable() {\n> +\tFULLNESS=\"--no-full\"\n> +\n> +\tif [ $DO_FULL == 1 ]; then FULLNESS=\"--full\"; fi\n> +\n> +\tBLOBS=($(git fsck --unreachable --no-reflogs \\\n> +\t\t\"${FULLNESS}\" --no-progress | sed -ne 's/^unreachable blob //p'))\n> +}\n\nIf you are going to do a full sweep with fsck anyway, perhaps have\nmake it do the work of writing out lost-found, so that you can\niterate over them?\n\nAs a blob that is only reachable from a commit in reflog and not in\nthe histories that are alive is something a user would want to recover,\nthe use of --no-reflogs option makes sense to me.\n\n> +function read_one_file {\n> +\tBLOB=$1\n> +\tFILTER_NAME=$2\n> +\tARGS=()\n> +\n> +\tif [ -z \"$FILTER_NAME\" ]; then\n> +\t\tARGS+=(\"blob\")\n> +\telse\n> +\t\tARGS+=(\"--filters\" \"--path=$FILTER_NAME\")\n> +\tfi\n> +\n> +\tgit cat-file \"${ARGS[@]}\" \"$BLOB\"\n> +}\n\nWe get a blob object name and optional \"--filename=name\"; drives\ncat-file possibly with \"--filters --path=name\" to grab its contents.\nI find it a good thinking to use \"--filters\" that does the equivalent\nof the smudge codepath, as you eventually want to materialize the\nfound contents as a working tree file ...\n\n> +function write_one_file {\n> +\tBLOB=$1\n> +\tFILTER_NAME=$2\n> +\tOUTPUT_NAME=$3\n> +\n> +\tABBREV=$(git rev-parse --short \"${BLOB}\")\n> +\n> +\techo -n \"Writing $ABBREV: \"\n> +\tread_one_file \"$BLOB\" \"$FILTER_NAME\" > \"$OUTPUT_NAME\"\n> +\techo \"$OUTPUT_NAME.\"\n> +}\n\n... which happens here.\n\n> +function unique_filename {\n> +\tif [ ! -f \"${BLOB}\" ]; then\n> +\t\techo \"$BLOB\"\n> +\telse\n> +\t\tcnt=1\n> +\t\twhile true\n> +\t\tdo\n> +\t\t\tfn=\"${BLOB}~${cnt}\"\n> +\t\t\tif [ ! -f \"${fn}\" ]; then\n> +\t\t\t\techo \"${fn}\"\n> +\t\t\t\tbreak\n> +\t\t\tfi\n> +\t\t\tcnt=$((cnt+1))\n> +\t\tdone\n> +\tfi\n> +}\n\nThe function comes up with a name for a given blob to be written in\nthe directory, in which the user happened to have started the\ncommand.  The function cannot be used unless we are processing each\nblob fully before moving onto the next one. In other words,\n\"resurrect abcde --file=Makefile abcde --file=README.pdf\" would\nleave the same blob in the BLOBS[] array twice, so that two\ndifferent --filters can be attempted, but the calling code cannot\nfirst call this function for all the BLOBS[] elements to assign them\nunique-filename, as this function depends on \"test -f\" to be able to\nsee what happened to the previous elements in the BLOBS[] array.\n\n> +function write_recoverable {\n> +\tfor i in \"${!BLOBS[@]}\"; do\n> +\t\tBLOB=${BLOBS[$i]}\n> +\t\tFILTER_NAME=${FILENAMES[$i]}\n> +\t\tOUTPUT_NAME=${FILENAMES[$i]:-$(unique_filename)}\n> +\n> +\t\twrite_one_file \"$BLOB\" \"$FILTER_NAME\" \"$OUTPUT_NAME\"\n> +\tdone\n> +}\n\nAnd we do that for all in BLOBS[].\n\n> +function interactive {\n> +\techo \"Recoverable orphaned git blobs:\"\n> +\n> +\tsort_by_timestamp\n> +\tfor BLOB_AND_TIMESTAMP in \"${BLOB_AND_TIMESTAMPS[@]}\"; do\n> +\t\techo\n> +\n> +\t\tBLOB=${BLOB_AND_TIMESTAMP::40}\n> +\t\tTIME=${BLOB_AND_TIMESTAMP:41}\n> +\t\tDATE=$([ ! -z \"$TIME\" ] && timestamp_to_s \"$TIME\" || echo \"(Unknown)\") \n> +\n> +\t\techo \"$BLOB  ($DATE)\"\n> +\t\tshow_summary \"${BLOB}\" | head -4 | sed -e 's/^/> /'\n> +\t\techo\n> +\n> +\t\twhile true\n> +\t\tdo\n> +\t\t\techo -n \"Recover this file? [y,n,v,f,q,?]: \"\n> +\t\t\tread -r ans || return 1\n> +\n> +\t\t\tcase \"$ans\" in\n> +\t\t\t[yY]*)\n> +\t\t\t\twrite_one_file \"${BLOB}\" \"\" \"$(unique_filename)\"\n> +\t\t\t\tbreak\n> +\t\t\t\t;;\n> +\t\t\t[nN]*)\n> +\t\t\t\tbreak\n> +\t\t\t\t;;\n> +\t\t\t[vV]*)\n> +\t\t\t\tview_file \"${BLOB}\"\n> +\t\t\t\techo\n> +\t\t\t\t;;\n> +\t\t\t[fF]*)\n> +\t\t\t\tprompt_for_filename\n> +\n> +\t\t\t\tif [ \"$FILENAME\" == \"\" ]; then\n> +\t\t\t\t\tbreak\n> +\t\t\t\tfi\n> +\n> +\t\t\t\twrite_one_file \"${BLOB}\" \"${FILENAME}\" \"${FILENAME}\"\n> +\t\t\t\tbreak\n> +\t\t\t\t;;\n> +\t\t\t\\?*)\n> +\t\t\t\techo\n> +\t\t\t\techo \"Do you want to recover this file?\"\n> +\t\t\t\techo \" y: yes, write the file to ${BLOB}\"\n> +\t\t\t\techo \" n: no, skip this file and see the next orphaned file\"\n> +\t\t\t\techo \" v: view the file\"\n> +\t\t\t\techo \" f: prompt for a filename to use for recovery\"\n> +\t\t\t\techo \" q: quit\"\n> +\t\t\t\techo\n> +\t\t\t\t;;\n> +\t\t\t[qQ]*)\n> +\t\t\t\treturn 0\n> +\t\t\t\t;;\n> +\t\t\tesac\n> +\t\tdone\n> +\tdone\n\nShows a bit of snippet from an orphaned blob, offer to write to\ndisk, to show it in full to give larger clue on its contents, etc.\nI am not sure the value of [yY] that does not do the prompt thing,\npossibly using the generated uniq name as the default---after all,\nthis is interactive.\n\nOne thing that I am mildly disappointed about this script is that it\ndoes not give much over lost-found service fsck gives, other than\nthe \"interactive\" loop.  A dedicated \"resurrect\" command should do a\nlot more.\n\nLet me throw one piece of idea out, which may or may not work well.\n\n\"fsck\" finds unreachable blobs, but it should be finding trees that\nare also unreachable that contain them, recursively.  A command to\ntruly help users recover lost blobs should be taking advantage of\nthat fact and spending cycles to exploit it to help them.  For\nexample, you could (and this won't happen inside a bash script)\n\n - enumerate unreachable trees and blobs;\n - identify _a_ tree the sought-blob appears in;\n - identify _a_ tree that tree appears in;\n - recursively do the above until you find no more.\n\nThat gives you _a_ path to the blob in _a_ tree.  That top tree\nmight be the toplevel of a working tree (it may be referenced by a\ncommit that is unreachable, for example), or the commit and its top\nlevel tree may already have been lost to gc and what you have might\nbe a tree structure representing a subtree (e.g. you thought that\nyou found \"Makefile\", but it turns out not to be the top-level\nMakefile, but \"t/Makefile\").\n\nBut any such name is 1000x better than a random name derived from\nthe object name.\n"},{"id":"354492","messageId":"20180804161956.GA6@1032a7a09014","threadId":"49033","inReplyTo":"xmqqpnyyt9di.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH 1/1] recover: restoration of deleted worktree files","fromName":"Edward Thomson","fromEmail":"ethomson@edwardthomson.com","sentAt":"2018-08-04T16:19:56Z","receivedAt":"2018-08-04T16:20:05Z","isPatch":true,"sender":{"key":"ethomson@edwardthomson.com","avatar":"https://avatars.githubusercontent.com/u/1130014?v=4"},"body":"On Sat, Aug 04, 2018 at 08:54:49AM -0700, Junio C Hamano wrote:\n> \n> My first reaction was to say that I am not going to take a new\n> command written only for bash with full bashism, even if it came\n> with docs, tests nor Makefile integration, for Git itself.  Then I\n> reconsidered, as not everything related to Git is git-core, and all\n> of the above traits are sign of this patch _not_ meant for git-core.\n\nYes, obviously I was not suggesting that this would be mergeable with\nthe bashims, as I mentioned in my cover letter.\n\nIn any case, it sounds like you're not particularly interested in\nthis, although I certainly appreciate you taking the time to suggest\nimprovements despite that.  There's some good feedback there.\n\nCheers-\n-ed\n"},{"id":"354493","messageId":"alpine.LFD.2.21.1808041214550.28242@localhost.localdomain","threadId":"49033","inReplyTo":"xmqqpnyyt9di.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH 1/1] recover: restoration of deleted worktree files","fromName":"Robert P. J. Day","fromEmail":"rpjday@crashcourse.ca","sentAt":"2018-08-04T16:17:03Z","receivedAt":"2018-08-04T16:46:02Z","isPatch":true,"sender":{"key":"rpjday@crashcourse.ca","avatar":"https://avatars.githubusercontent.com/u/226084077?v=4"},"body":"On Sat, 4 Aug 2018, Junio C Hamano wrote:\n\n> Edward Thomson <ethomson@edwardthomson.com> writes:\n>\n> > Introduce git-recover, a simple script to aide in restoration of\n> > deleted worktree files.  This will look for unreachable blobs in\n> > the object database and prompt users to restore them to disk,\n> > either interactively or on the command-line.\n\n> >  git-recover.sh | 311 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> >  1 file changed, 311 insertions(+)\n> >  create mode 100755 git-recover.sh\n>\n> My first reaction was to say that I am not going to take a new\n> command written only for bash with full bashism, even if it came\n> with docs, tests nor Makefile integration, for Git itself.  Then I\n> reconsidered, as not everything related to Git is git-core, and all\n> of the above traits are sign of this patch _not_ meant for git-core.\n>\n> In other words, I think this patch can be a fine addition to\n> somebody else's project (i.e. random collection of scripts that may\n> help Git users), so let's see how I can offer comments/inputs to\n> help you improve it.  So I won't comment on lang, log message, or\n> shell scripting style---these are project convention and the\n> git-core convention won't be relevant to this patch.\n\n  not sure how relevant this is, but fedora bundles a bunch of neat\nutilities into two packages: git-tools and git-extras. i have no idea\nwhat relationship those packages have to official git, or who decides\nwhat goes into them.\n\nrday\n\n-- \n\n========================================================================\nRobert P. J. Day                                 Ottawa, Ontario, CANADA\n                  http://crashcourse.ca/dokuwiki\n\nTwitter:                                       http://twitter.com/rpjday\nLinkedIn:                               http://ca.linkedin.com/in/rpjday\n========================================================================\n"},{"id":"354494","messageId":"xmqqh8kat6wd.fsf@gitster-ct.c.googlers.com","threadId":"49033","inReplyTo":"20180804161956.GA6@1032a7a09014","subject":"Re: [RFC PATCH 1/1] recover: restoration of deleted worktree files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-04T16:48:18Z","receivedAt":"2018-08-04T16:48:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson <ethomson@edwardthomson.com> writes:\n\n> In any case, it sounds like you're not particularly interested in\n> this, although I certainly appreciate you taking the time to suggest\n> improvements despite that.  There's some good feedback there.\n\nNot in its current shape.  But do not take this in a wrong way.  It\nmay be useful in a third-party script collection in its current\nshape already.\n\nMore importantly, I am not opposed to have a \"resurrect\" utility in\nthe core distribution.  It just has to be a lot better than what\n\"grep -e 'I think I wrote this string' .git/lost-found/other/*\"\ngives us.\n\nFilename discovery (perhaps from lost trees, which was the idea I\nwrote in the message I am responding to, but others may come up with\nbetter alternatibve approaches) is a must, but not primarily because\nsuch a grep won't find the path to which the contents should go.\nWhen a user says \"I think I wrote this string in the file I am\nlooking for\", s/he already knows what s/he wants to recover (i.e. it\nwas a README file at the top-level).  Filename discovery is a must\nbecause grepping in the raw blob contents without smudge filter\nchain applied may not find what we want in the first place, and for\nthat to happen, we need to have a filename.\n\n\tSide note.  That may mean that even working in the\n\tdo-recover mode, the script may want to take a filename,\n\tletting the user to say \"pretend all lost blobs are of this\n\ttype, as that is the type of the blob I just lost and am\n\tinterested in, and a filename will help you find an\n\tappropriate smudge and/or textconv filter to help me\"\n\nThat makes me realize that I did not mention one more thing, other\nthan the \"interactibve loop\", I did like in the script over what\nlost-found gives us: smudge filter support.  I do not very often\nwork with contents that needs clean/smudge other than in one project\n(obviously not \"git.git\"), and I can see how it is essential in\nhelping the user to find the contents the user is looking for.\n\nThanks.\n"},{"id":"354499","messageId":"20180804173340.GK3764@zaya.teonanacatl.net","threadId":"49033","inReplyTo":"alpine.LFD.2.21.1808041214550.28242@localhost.localdomain","subject":"Re: [RFC PATCH 1/1] recover: restoration of deleted worktree files","fromName":"Todd Zullinger","fromEmail":"tmz@pobox.com","sentAt":"2018-08-04T17:33:40Z","receivedAt":"2018-08-04T17:41:32Z","isPatch":true,"sender":{"key":"tmz@pobox.com","avatar":"https://avatars.githubusercontent.com/u/806319?v=4"},"body":"Hi,\n\nRobert P. J. Day wrote:\n> On Sat, 4 Aug 2018, Junio C Hamano wrote:\n>> In other words, I think this patch can be a fine addition to\n>> somebody else's project (i.e. random collection of scripts that may\n>> help Git users), so let's see how I can offer comments/inputs to\n>> help you improve it.  So I won't comment on lang, log message, or\n>> shell scripting style---these are project convention and the\n>> git-core convention won't be relevant to this patch.\n> \n>   not sure how relevant this is, but fedora bundles a bunch of neat\n> utilities into two packages: git-tools and git-extras. i have no idea\n> what relationship those packages have to official git, or who decides\n> what goes into them.\n\nFor anyone curious, those packages (git-extras and\ngit-tools) are both entirely separate projects upstream and\nin the fedora packaging.  A git-recover script may well be a\ngood fit in one of those upstream projects.\n\nThe git-(extras|tools) package names are a bit confusing\nIMO.  But it's probably more confusing that they each add a\nnumber of git-* commands in the default PATH the way they're\npackaged.\n\nWe do package some bits from contrib/ (e.g. completion,\nsubtree, etc.) in the fedora git packages.  We don't add\nscripts and commands from outside of the git tarballs as\npart of the fedora git package, though.\n\nSo far, I don't recall anyone filing a bug report about\ncommands from git-extras or git-tools against git.  So it\nseems that users of those additional packages aren't being\nconfused, thankfully.\n\n-- \nTodd\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nBetween two evils, I always pick the one I never tried before.\n    -- Mae West\n\n"},{"id":"354513","messageId":"20180805013422.GC258270@aiede.svl.corp.google.com","threadId":"49033","inReplyTo":"20180804142247.GA7@e3c0ce5ceb57","subject":"Re: [RFC PATCH 0/1] Introduce git-recover","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-05T01:34:22Z","receivedAt":"2018-08-05T01:34:28Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nEdward Thomson wrote:\n\n> I created a simple shell script a while back to help people recover\n> files that they deleted from their working directory (but had been added\n> to the repository), which looks for unreachable blobs in the object\n> database and places them in the working directory (either en masse,\n> interactively, or via command-line arguments).\n\nCool!  Most of this belongs in the commit message, which is part of why\nI always discourage having a separate cover letter in single-patch\nseries.\n\n> This has been available at https://github.com/ethomson/git-recover for\n> about a year, and in that time, someone has suggested that I propose\n> this as part of git itself.  So I thought I'd see if there's any\n> interest in this.\n>\n> If there is, I'd like to get a sense of the amount of work required to\n> make this suitable for inclusion.  There are some larger pieces of work\n> required -- at a minimum, I think this requires:\n>\n> - Tests -- there are none, which is fine with me but probably less fine\n>   for inclusion here.\n> - Documentation -- the current README is below but it will need proper\n>   documentation that can be rendered into manpages, etc, by the tools.\n> - Remove bashisms -- there are many.\n\nOne possible path in that direction would be to \"stage\" the code in\ncontrib/ first, while documenting the intention of graduating to a\ncommand in git itself.  Then the list can pitch in with those tasks.\nThere are good reasons for a tool to exist outside of Git, so I\nwouldn't recommend this unless we have a clear plan for its graduation\nthat we've agreed upon as a project, but thought I should mention it\nas a mechanism in case we decide to do that.\n\nThe trend these days for Git commands has been to prefer to have them\nin C.  Portable shell is a perfectly fine stopping point on the way\nthere, though.\n\nMy more fundamental main thought is separate from those logistics: how\ndoes this relate to \"git fsck --lost-found\"?  What would your ideal\ninterface to solve this problem look like?  Can we make Git's commands\ncomplement each other in a good way to solve it well?\n\nThanks,\nJonathan\n"}]}