{"thread":{"id":"58251","subject":"[PATCH 0/7] Generalize 'scalar diagnose' into 'git bugreport --diagnose'","startedAt":"2022-08-01T21:15:04Z","lastAt":"2022-08-18T19:12:32Z","messageCount":94,"participants":["Victoria Dye via GitGitGadget","Junio C Hamano","Ævar Arnfjörð Bjarmason","Victoria Dye","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"460376","messageId":"pull.1310.git.1659388498.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":null,"subject":"[PATCH 0/7] Generalize 'scalar diagnose' into 'git bugreport --diagnose'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:51Z","receivedAt":"2022-08-01T21:15:04Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"As part of the preparation for moving Scalar out of 'contrib/' and into Git,\nthis series moves the functionality of 'scalar diagnose' into a new option\n('--diagnose') for 'git bugreport'. This change further aligns Scalar with\nthe objective [1] of having it only contain functionality and settings that\nbenefit large Git repositories, but not all repositories. The diagnostics\nreported by 'scalar diagnose' relevant for investigating issues in any Git\nrepository, so generating them should be part of a \"normal\" Git builtin.\n\nAn alternative implementation considered was creating a new 'git diagnose'\nbuiltin, but the new command would end up duplicating much of\n'builtin/bugreport.c'. Although that issue could be overcome with\nrefactoring, I didn't see a major UX benefit of 'git diagnose' vs 'git\nbugreport --diagnose', so I went with the latter, simpler approach.\n\nFinally, despite 'scalar diagnose' now being nothing more than a wrapper for\n'git bugreport --diagnose', it is not being deprecated in this series.\nAlthough deprecation -> removal could be a future cleanup effort, 'scalar\ndiagnose' is kept around for now as an alias for users already accustomed to\nusing it in 'scalar'.\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1275.v2.git.1657584367.gitgitgadget@gmail.com/\n\nVictoria Dye (7):\n scalar: use \"$GIT_UNZIP\" in 'scalar diagnose' test\n builtin/bugreport.c: create '--diagnose' option\n builtin/bugreport.c: avoid size_t overflow\n builtin/bugreport.c: add directory to archiver more gently\n builtin/bugreport.c: add '--no-report' option\n scalar: use 'git bugreport --diagnose' in 'scalar diagnose'\n scalar: update technical doc roadmap\n\n Documentation/git-bugreport.txt | 17 +-\n Documentation/technical/scalar.txt | 9 +-\n builtin/bugreport.c | 302 ++++++++++++++++++++++++++++-\n contrib/scalar/scalar.c | 271 +-------------------------\n contrib/scalar/t/t9099-scalar.sh | 8 +-\n t/t0091-bugreport.sh | 29 +++\n 6 files changed, 358 insertions(+), 278 deletions(-)\n\n\nbase-commit: 23b219f8e3f2adfb0441e135f0a880e6124f766c\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1310%2Fvdye%2Fscalar%2Fgeneralize-diagnose-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1310/vdye/scalar/generalize-diagnose-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1310\n-- \ngitgitgadget\n"},{"id":"460377","messageId":"a7a766de29b53610d19e56f270863230f3a2eab6.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 1/7] scalar: use \"$GIT_UNZIP\" in 'scalar diagnose' test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:52Z","receivedAt":"2022-08-01T21:15:06Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUse the \"$GIT_UNZIP\" test variable rather than verbatim 'unzip' to unzip the\n'scalar diagnose' archive. Using \"$GIT_UNZIP\" is needed to run the Scalar\ntests on systems where 'unzip' is not in the system path.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/t/t9099-scalar.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh\nindex 10b1172a8aa..fac86a57550 100755\n--- a/contrib/scalar/t/t9099-scalar.sh\n+++ b/contrib/scalar/t/t9099-scalar.sh\n@@ -109,14 +109,14 @@ test_expect_success UNZIP 'scalar diagnose' '\n \tsed -n \"s/.*$SQ\\\\(.*\\\\.zip\\\\)$SQ.*/\\\\1/p\" zip_path &&\n \tzip_path=$(cat zip_path) &&\n \ttest -n \"$zip_path\" &&\n-\tunzip -v \"$zip_path\" &&\n+\t\"$GIT_UNZIP\" -v \"$zip_path\" &&\n \tfolder=${zip_path%.zip} &&\n \ttest_path_is_missing \"$folder\" &&\n-\tunzip -p \"$zip_path\" diagnostics.log >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n \ttest_file_not_empty out &&\n-\tunzip -p \"$zip_path\" packs-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n \tgrep \"$(pwd)/.git/objects\" out &&\n-\tunzip -p \"$zip_path\" objects-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n \tgrep \"^Total: [1-9]\" out\n '\n \n-- \ngitgitgadget\n\n"},{"id":"460378","messageId":"932dc8cddacf1841996cd3e0d512b95828213015.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 2/7] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:53Z","receivedAt":"2022-08-01T21:15:10Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a '--diagnose' option for 'git bugreport' to collect additional\ninformation about the repository and write it to a zipped archive.\n\nThe \"diagnose\" functionality was originally implemented for Scalar in\naa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\ndiagnostics gathered are not specific to Scalar-cloned repositories and\ncould be useful when diagnosing issues in any Git repository.\n\nNote that, while this patch appears large, it is mostly copied directly out\nof 'scalar.c'. Specifically, the functions\n\n- dir_file_stats_objects()\n- dir_file_stats()\n- count_files()\n- loose_objs_stats()\n- add_directory_to_archiver()\n- get_disk_info()\n\nare all copied verbatim from 'scalar.c'. The 'create_diagnostics_archive()'\nfunction is a mostly unmodified copy of 'cmd_diagnose()', with the primary\nchanges being that 'zip_path' is an input and \"Enlistment root\" is corrected\nto \"Repository root\" in the logs.\n\nThe remainder of the patch is made up of adding the '--diagnose' option to\n'cmd_bugreport()' (including generation of the archive's 'zip_path'),\nupdating documentation, and adding a test. Note that the test is\n'test_expect_failure' due to bugs in the original 'scalar diagnose'. These\nwill be fixed in subsequent patches.\n\nSuggested-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n Documentation/git-bugreport.txt | 11 +-\n builtin/bugreport.c | 282 +++++++++++++++++++++++++++++++-\n t/t0091-bugreport.sh | 20 +++\n 3 files changed, 309 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-bugreport.txt b/Documentation/git-bugreport.txt\nindex d8817bf3cec..b55658bc287 100644\n--- a/Documentation/git-bugreport.txt\n+++ b/Documentation/git-bugreport.txt\n@@ -8,7 +8,7 @@ git-bugreport - Collect information for user to file a bug report\n SYNOPSIS\n --------\n [verse]\n-'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n+'git bugreport' []\n \n DESCRIPTION\n -----------\n@@ -31,6 +31,9 @@ The following information is captured automatically:\n - A list of enabled hooks\n - $SHELL\n \n+Additional information may be gathered into a separate zip archive using the\n+`--diagnose` option.\n+\n This tool is invoked via the typical Git setup process, which means that in some\n cases, it might not be able to launch - for example, if a relevant config file\n is unreadable. In this kind of scenario, it may be helpful to manually gather\n@@ -49,6 +52,12 @@ OPTIONS\n \tnamed 'git-bugreport-'. This should take the form of a\n \tstrftime(3) format string; the current local time will be used.\n \n+--diagnose::\n+\tCreate a zip archive of information about the repository including logs\n+\tand certain statistics describing the data shape of the repository. The\n+\tarchive is written to the same output directory as the bug report and is\n+\tnamed 'git-diagnostics-'.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 9de32bc96e7..35b1fc48bf1 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -5,6 +5,10 @@\n #include \"compat/compiler.h\"\n #include \"hook.h\"\n #include \"hook-list.h\"\n+#include \"dir.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"archive.h\"\n \n \n static void get_system_info(struct strbuf *sys_info)\n@@ -59,7 +63,7 @@ static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n }\n \n static const char * const bugreport_usage[] = {\n-\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n+\tN_(\"git bugreport []\"),\n \tNULL\n };\n \n@@ -91,6 +95,259 @@ static void get_header(struct strbuf *buf, const char *title)\n \tstrbuf_addf(buf, \"\\n\\n[%s]\\n\", title);\n }\n \n+static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_name, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\tstruct stat st;\n+\n+\tif (!stat(full_path, &st))\n+\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n+\t\t\t (uintmax_t)st.st_size);\n+}\n+\n+static int dir_file_stats(struct object_directory *object_dir, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\n+\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n+\n+\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n+\t\t\t\t data);\n+\n+\treturn 0;\n+}\n+\n+static int count_files(char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count = 0;\n+\n+\tif (!dir)\n+\t\treturn 0;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n+\t\t\tcount++;\n+\n+\tclosedir(dir);\n+\treturn count;\n+}\n+\n+static void loose_objs_stats(struct strbuf *buf, const char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count;\n+\tint total = 0;\n+\tunsigned char c;\n+\tstruct strbuf count_path = STRBUF_INIT;\n+\tsize_t base_path_len;\n+\n+\tif (!dir)\n+\t\treturn;\n+\n+\tstrbuf_addstr(buf, \"Object directory stats for \");\n+\tstrbuf_add_absolute_path(buf, path);\n+\tstrbuf_addstr(buf, \":\\n\");\n+\n+\tstrbuf_add_absolute_path(&count_path, path);\n+\tstrbuf_addch(&count_path, '/');\n+\tbase_path_len = count_path.len;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) &&\n+\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n+\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n+\t\t\tstrbuf_setlen(&count_path, base_path_len);\n+\t\t\tstrbuf_addstr(&count_path, e->d_name);\n+\t\t\ttotal += (count = count_files(count_path.buf));\n+\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n+\t\t}\n+\n+\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n+\n+\tstrbuf_release(&count_path);\n+\tclosedir(dir);\n+}\n+\n+static int add_directory_to_archiver(struct strvec *archiver_args,\n+\t\t\t\t const char *path, int recurse)\n+{\n+\tint at_root = !*path;\n+\tDIR *dir = opendir(at_root ? \".\" : path);\n+\tstruct dirent *e;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t len;\n+\tint res = 0;\n+\n+\tif (!dir)\n+\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\n+\tif (!at_root)\n+\t\tstrbuf_addf(&buf, \"%s/\", path);\n+\tlen = buf.len;\n+\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n+\n+\twhile (!res && (e = readdir(dir))) {\n+\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&buf, len);\n+\t\tstrbuf_addstr(&buf, e->d_name);\n+\n+\t\tif (e->d_type == DT_REG)\n+\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n+\t\telse if (e->d_type != DT_DIR)\n+\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n+\t\t\t\t \"directory\"), buf.buf);\n+\t\telse if (recurse &&\n+\t\t\t add_directory_to_archiver(archiver_args,\n+\t\t\t\t\t\t buf.buf, recurse) < 0)\n+\t\t\tres = -1;\n+\t}\n+\n+\tclosedir(dir);\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+#ifndef WIN32\n+#include \n+#endif\n+\n+static int get_disk_info(struct strbuf *out)\n+{\n+#ifdef WIN32\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n+\tDWORD serial_number, component_length, flags;\n+\tULARGE_INTEGER avail2caller, total, avail;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n+\t\terror(_(\"could not determine free disk size for '%s'\"),\n+\t\t buf.buf);\n+\t\tstrbuf_release(&buf);\n+\t\treturn -1;\n+\t}\n+\n+\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n+\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n+\t\t\t\t &serial_number, &component_length, &flags,\n+\t\t\t\t fs_name, sizeof(fs_name))) {\n+\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n+\t\tstrbuf_release(&buf);\n+\t\treturn -1;\n+\t}\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n+\tstrbuf_addch(out, '\\n');\n+\tstrbuf_release(&buf);\n+#else\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct statvfs stat;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (statvfs(buf.buf, &stat) < 0) {\n+\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n+\t\t\t buf.buf);\n+\t\tstrbuf_release(&buf);\n+\t\treturn -1;\n+\t}\n+\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n+\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n+\tstrbuf_release(&buf);\n+#endif\n+\treturn 0;\n+}\n+\n+static int create_diagnostics_archive(struct strbuf *zip_path)\n+{\n+\tstruct strvec archiver_args = STRVEC_INIT;\n+\tchar **argv_copy = NULL;\n+\tint stdout_fd = -1, archiver_fd = -1;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res = 0;\n+\n+\tstdout_fd = dup(1);\n+\tif (stdout_fd < 0) {\n+\t\tres = error_errno(_(\"could not duplicate stdout\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tarchiver_fd = xopen(zip_path->buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n+\tif (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) {\n+\t\tres = error_errno(_(\"could not redirect output\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tinit_zip_archiver();\n+\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n+\tget_version_info(&buf, 1);\n+\n+\tstrbuf_addf(&buf, \"Repository root: %s\\n\", the_repository->worktree);\n+\tget_disk_info(&buf);\n+\twrite_or_die(stdout_fd, buf.buf, buf.len);\n+\tstrvec_pushf(&archiver_args,\n+\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n+\t\t (int)buf.len, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n+\tdir_file_stats(the_repository->objects->odb, &buf);\n+\tforeach_alt_odb(dir_file_stats, &buf);\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n+\tloose_objs_stats(&buf, \".git/objects\");\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n+\t\tgoto diagnose_cleanup;\n+\n+\tstrvec_pushl(&archiver_args, \"--prefix=\",\n+\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n+\n+\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n+\targv_copy = xmemdupz(archiver_args.v,\n+\t\t\t sizeof(char *) * archiver_args.nr);\n+\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n+\t\t\t the_repository, NULL, 0);\n+\tif (res) {\n+\t\terror(_(\"failed to write archive\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tif (!res)\n+\t\tfprintf(stderr, \"\\n\"\n+\t\t\t\"Diagnostics complete.\\n\"\n+\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n+\t\t\tzip_path->buf);\n+\n+diagnose_cleanup:\n+\tif (archiver_fd >= 0) {\n+\t\tclose(1);\n+\t\tdup2(stdout_fd, 1);\n+\t}\n+\tfree(argv_copy);\n+\tstrvec_clear(&archiver_args);\n+\tstrbuf_release(&buf);\n+\n+\treturn res;\n+}\n+\n int cmd_bugreport(int argc, const char **argv, const char *prefix)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n@@ -98,16 +355,20 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tint report = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tint diagnose = 0;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tconst char *user_relative_path = NULL;\n \tchar *prefixed_filename;\n+\tsize_t output_path_len;\n \n \tconst struct option bugreport_options[] = {\n+\t\tOPT_BOOL(0, \"diagnose\", &diagnose,\n+\t\t\t N_(\"generate a diagnostics zip archive\")),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n-\t\t\t N_(\"specify a destination for the bugreport file\")),\n+\t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n-\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\t\t N_(\"specify a strftime format suffix for the filename(s)\")),\n \t\tOPT_END()\n \t};\n \n@@ -119,6 +380,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t option_output ? option_output : \"\");\n \tstrbuf_addstr(&report_path, prefixed_filename);\n \tstrbuf_complete(&report_path, '/');\n+\toutput_path_len = report_path.len;\n \n \tstrbuf_addstr(&report_path, \"git-bugreport-\");\n \tstrbuf_addftime(&report_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n@@ -133,6 +395,20 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t report_path.buf);\n \t}\n \n+\t/* Prepare diagnostics, if requested */\n+\tif (diagnose) {\n+\t\tstruct strbuf zip_path = STRBUF_INIT;\n+\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n+\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\t\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\t\tif (create_diagnostics_archive(&zip_path))\n+\t\t\tdie_errno(_(\"unable to create diagnostics archive %s\"), zip_path.buf);\n+\n+\t\tstrbuf_release(&zip_path);\n+\t}\n+\n \t/* Prepare the report contents */\n \tget_bug_template(&buffer);\n \ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 08f5fe9caef..3cf983aa67f 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -78,4 +78,24 @@ test_expect_success 'indicates populated hooks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose -o report -s test >out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\tgrep \"Available space\" out &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"460379","messageId":"e8abfdfa892f5670ed411ed6daef26a4ffa1bfe6.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:54Z","receivedAt":"2022-08-01T21:15:11Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAvoid size_t overflow when reporting the available disk space in\n'get_disk_info' by casting the block size and available block count to\n'uint64_t' before multiplying them. Without this change, 'st_mult' would\n(correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\nbytes of available space.\n\nSigned-off-by: Victoria Dye \n---\n builtin/bugreport.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 35b1fc48bf1..720889a37ad 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -258,7 +258,7 @@ static int get_disk_info(struct strbuf *out)\n \t}\n \n \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n+\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n \tstrbuf_release(&buf);\n #endif\n-- \ngitgitgadget\n\n"},{"id":"460380","messageId":"4bc290fbf43e0193aae288b79249014d899ea34a.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 4/7] builtin/bugreport.c: add directory to archiver more gently","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:55Z","receivedAt":"2022-08-01T21:15:13Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIf a directory added to the '--diagnose' archiver does not exist, warn and\nreturn 0 from 'add_directory_to_archiver()' rather than failing with a fatal\nerror. This handles a failure edge case where the '.git/logs' has not yet\nbeen created when running 'git bugreport --diagnose', but extends to any\nsituation where a directory may be missing in the '.git' dir.\n\nNow, when a directory is missing a warning is captured in the diagnostic\nlogs. This provides a user with more complete information than if 'git\nbugreport' simply failed with an error.\n\nSigned-off-by: Victoria Dye \n---\n builtin/bugreport.c | 8 +++++++-\n t/t0091-bugreport.sh | 11 ++++++++++-\n 2 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 720889a37ad..dea11f91386 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -176,12 +176,18 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \t\t\t\t const char *path, int recurse)\n {\n \tint at_root = !*path;\n-\tDIR *dir = opendir(at_root ? \".\" : path);\n+\tDIR *dir;\n \tstruct dirent *e;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t len;\n \tint res = 0;\n \n+\tif (!file_exists(at_root ? \".\" : path)) {\n+\t\twarning(_(\"directory '%s' does not exist, will not be archived\"), path);\n+\t\treturn 0;\n+\t}\n+\n+\tdir = opendir(at_root ? \".\" : path);\n \tif (!dir)\n \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n \ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 3cf983aa67f..e9db89ef2c8 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -78,7 +78,7 @@ test_expect_success 'indicates populated hooks' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n+test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n \ttest_when_finished rm -rf report &&\n \n \tgit bugreport --diagnose -o report -s test >out &&\n@@ -98,4 +98,13 @@ test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n \tgrep \"^Total: [0-9][0-9]*\" out\n '\n \n+test_expect_success '--diagnose warns when archived dir does not exist' '\n+\ttest_when_finished rm -rf report &&\n+\n+\t# Remove logs - not guaranteed to exist\n+\trm -rf .git/logs &&\n+\tgit bugreport --diagnose -o report -s test 2>err &&\n+\tgrep \"directory .\\.git/logs. does not exist, will not be archived\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"460381","messageId":"d6527049a4f62f3b1f008706443a11a0e0816074.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 5/7] builtin/bugreport.c: add '--no-report' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:56Z","receivedAt":"2022-08-01T21:15:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd a '--no-report' option to 'git bugreport' to avoid writing the\n'git-bugreport-.txt' file. This gives users the option of creating\nonly the diagnostic archive with '--diagnose' and mirroring the behavior of\nthe original 'scalar diagnose' as closely as possible.\n\nIf a user specifies '--no-report' *without* also specifying '--diagnose',\nthe 'git bugreport' operation is a no-op; a warning message is printed and\nthe command returns with a non-error exit code.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/git-bugreport.txt | 6 ++++++\n builtin/bugreport.c | 16 +++++++++++++++-\n 2 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-bugreport.txt b/Documentation/git-bugreport.txt\nindex b55658bc287..5eae7a4f950 100644\n--- a/Documentation/git-bugreport.txt\n+++ b/Documentation/git-bugreport.txt\n@@ -58,6 +58,12 @@ OPTIONS\n \tarchive is written to the same output directory as the bug report and is\n \tnamed 'git-diagnostics-'.\n \n+--no-report::\n+\tDo not write out a 'git-bugreport-.txt' file. This option is\n+\tintended for use with `--diagnose` when only the diagnostic archive is\n+\tneeded. If `--no-report` is used without `--diagnose`, `git bugreport`\n+\tis a no-op.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex dea11f91386..5ecff70276a 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -361,7 +361,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tint report = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n-\tint diagnose = 0;\n+\tint diagnose = 0, skip_summary = 0;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tconst char *user_relative_path = NULL;\n@@ -371,6 +371,8 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tconst struct option bugreport_options[] = {\n \t\tOPT_BOOL(0, \"diagnose\", &diagnose,\n \t\t\t N_(\"generate a diagnostics zip archive\")),\n+\t\tOPT_BOOL(0, \"no-report\", &skip_summary,\n+\t\t\t N_(\"do not create a summary report\")),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n \t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n@@ -381,6 +383,11 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \targc = parse_options(argc, argv, prefix, bugreport_options,\n \t\t\t bugreport_usage, 0);\n \n+\tif (skip_summary && !diagnose) {\n+\t\twarning(_(\"Nothing to do!\"));\n+\t\treturn 0;\n+\t}\n+\n \t/* Prepare the path to put the result */\n \tprefixed_filename = prefix_filename(prefix,\n \t\t\t\t\t option_output ? option_output : \"\");\n@@ -415,6 +422,13 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_release(&zip_path);\n \t}\n \n+\tif (skip_summary) {\n+\t\tfree(prefixed_filename);\n+\t\tstrbuf_release(&buffer);\n+\t\tstrbuf_release(&report_path);\n+\t\treturn 0;\n+\t}\n+\n \t/* Prepare the report contents */\n \tget_bug_template(&buffer);\n \n-- \ngitgitgadget\n\n"},{"id":"460382","messageId":"4eb3c43d48832150c53ee56c679ab13464185974.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 6/7] scalar: use 'git bugreport --diagnose' in 'scalar diagnose'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:57Z","receivedAt":"2022-08-01T21:15:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nReplace implementation of 'scalar diagnose' with an internal invocation of\n'git bugreport --diagnose --no-report'. The '--diagnose' option of 'git\nbugreport' was implemented to mirror what 'scalar diagnose' does, taking\nmost of its code directly from 'scalar.c'. Remove the now-duplicate code in\n'scalar.c' and have 'scalar diagnose' call 'git bugreport' to create the\ndiagnostics archive.\n\nThis introduces two (minor) changes to the output of 'scalar diagnose':\nchanging \"Enlistment root\" to \"Repository root\" in 'diagnostics.log'\n(\"enlistment root\" was inaccurate anyway, as the reported path always\npointed to the root of the repository), and changing the prefix of the zip\narchive from 'scalar_' to 'git-diagnostics-'.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 271 ++--------------------------------------\n 1 file changed, 7 insertions(+), 264 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 97e71fe19cd..7b1953605bd 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,7 +11,6 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"archive.h\"\n #include \"object-store.h\"\n \n /*\n@@ -262,99 +261,6 @@ static int unregister_dir(void)\n \treturn res;\n }\n \n-static int add_directory_to_archiver(struct strvec *archiver_args,\n-\t\t\t\t\t const char *path, int recurse)\n-{\n-\tint at_root = !*path;\n-\tDIR *dir = opendir(at_root ? \".\" : path);\n-\tstruct dirent *e;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t len;\n-\tint res = 0;\n-\n-\tif (!dir)\n-\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n-\n-\tif (!at_root)\n-\t\tstrbuf_addf(&buf, \"%s/\", path);\n-\tlen = buf.len;\n-\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n-\n-\twhile (!res && (e = readdir(dir))) {\n-\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n-\t\t\tcontinue;\n-\n-\t\tstrbuf_setlen(&buf, len);\n-\t\tstrbuf_addstr(&buf, e->d_name);\n-\n-\t\tif (e->d_type == DT_REG)\n-\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n-\t\telse if (e->d_type != DT_DIR)\n-\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n-\t\t\t\t \"directory\"), buf.buf);\n-\t\telse if (recurse &&\n-\t\t\t add_directory_to_archiver(archiver_args,\n-\t\t\t\t\t\t buf.buf, recurse) < 0)\n-\t\t\tres = -1;\n-\t}\n-\n-\tclosedir(dir);\n-\tstrbuf_release(&buf);\n-\treturn res;\n-}\n-\n-#ifndef WIN32\n-#include \n-#endif\n-\n-static int get_disk_info(struct strbuf *out)\n-{\n-#ifdef WIN32\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n-\tDWORD serial_number, component_length, flags;\n-\tULARGE_INTEGER avail2caller, total, avail;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n-\t\terror(_(\"could not determine free disk size for '%s'\"),\n-\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n-\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n-\t\t\t\t &serial_number, &component_length, &flags,\n-\t\t\t\t fs_name, sizeof(fs_name))) {\n-\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n-\tstrbuf_addch(out, '\\n');\n-\tstrbuf_release(&buf);\n-#else\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tstruct statvfs stat;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (statvfs(buf.buf, &stat) < 0) {\n-\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n-\t\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n-\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n-\tstrbuf_release(&buf);\n-#endif\n-\treturn 0;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\n@@ -595,83 +501,6 @@ cleanup:\n \treturn res;\n }\n \n-static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n-\t\t\t\t const char *file_name, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\tstruct stat st;\n-\n-\tif (!stat(full_path, &st))\n-\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n-\t\t\t (uintmax_t)st.st_size);\n-}\n-\n-static int dir_file_stats(struct object_directory *object_dir, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\n-\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n-\n-\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n-\t\t\t\t data);\n-\n-\treturn 0;\n-}\n-\n-static int count_files(char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count = 0;\n-\n-\tif (!dir)\n-\t\treturn 0;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n-\t\t\tcount++;\n-\n-\tclosedir(dir);\n-\treturn count;\n-}\n-\n-static void loose_objs_stats(struct strbuf *buf, const char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count;\n-\tint total = 0;\n-\tunsigned char c;\n-\tstruct strbuf count_path = STRBUF_INIT;\n-\tsize_t base_path_len;\n-\n-\tif (!dir)\n-\t\treturn;\n-\n-\tstrbuf_addstr(buf, \"Object directory stats for \");\n-\tstrbuf_add_absolute_path(buf, path);\n-\tstrbuf_addstr(buf, \":\\n\");\n-\n-\tstrbuf_add_absolute_path(&count_path, path);\n-\tstrbuf_addch(&count_path, '/');\n-\tbase_path_len = count_path.len;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) &&\n-\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n-\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n-\t\t\tstrbuf_setlen(&count_path, base_path_len);\n-\t\t\tstrbuf_addstr(&count_path, e->d_name);\n-\t\t\ttotal += (count = count_files(count_path.buf));\n-\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n-\t\t}\n-\n-\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n-\n-\tstrbuf_release(&count_path);\n-\tclosedir(dir);\n-}\n-\n static int cmd_diagnose(int argc, const char **argv)\n {\n \tstruct option options[] = {\n@@ -681,106 +510,20 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tN_(\"scalar diagnose []\"),\n \t\tNULL\n \t};\n-\tstruct strbuf zip_path = STRBUF_INIT;\n-\tstruct strvec archiver_args = STRVEC_INIT;\n-\tchar **argv_copy = NULL;\n-\tint stdout_fd = -1, archiver_fd = -1;\n-\ttime_t now = time(NULL);\n-\tstruct tm tm;\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct strbuf diagnostics_path = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\t usage, 0);\n \n-\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n-\n-\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n-\tstrbuf_addftime(&zip_path,\n-\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n-\tstrbuf_addstr(&zip_path, \".zip\");\n-\tswitch (safe_create_leading_directories(zip_path.buf)) {\n-\tcase SCLD_EXISTS:\n-\tcase SCLD_OK:\n-\t\tbreak;\n-\tdefault:\n-\t\terror_errno(_(\"could not create directory for '%s'\"),\n-\t\t\t zip_path.buf);\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\tstdout_fd = dup(1);\n-\tif (stdout_fd < 0) {\n-\t\tres = error_errno(_(\"could not duplicate stdout\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tarchiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n-\tif (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) {\n-\t\tres = error_errno(_(\"could not redirect output\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tinit_zip_archiver();\n-\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n-\tget_version_info(&buf, 1);\n-\n-\tstrbuf_addf(&buf, \"Enlistment root: %s\\n\", the_repository->worktree);\n-\tget_disk_info(&buf);\n-\twrite_or_die(stdout_fd, buf.buf, buf.len);\n-\tstrvec_pushf(&archiver_args,\n-\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n-\t\t (int)buf.len, buf.buf);\n+\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_path);\n+\tstrbuf_addstr(&diagnostics_path, \"/.scalarDiagnostics\");\n \n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n-\tdir_file_stats(the_repository->objects->odb, &buf);\n-\tforeach_alt_odb(dir_file_stats, &buf);\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n-\tloose_objs_stats(&buf, \".git/objects\");\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n-\t\tgoto diagnose_cleanup;\n-\n-\tstrvec_pushl(&archiver_args, \"--prefix=\",\n-\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n-\n-\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n-\targv_copy = xmemdupz(archiver_args.v,\n-\t\t\t sizeof(char *) * archiver_args.nr);\n-\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n-\t\t\t the_repository, NULL, 0);\n-\tif (res) {\n-\t\terror(_(\"failed to write archive\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tif (!res)\n-\t\tfprintf(stderr, \"\\n\"\n-\t\t \"Diagnostics complete.\\n\"\n-\t\t \"All of the gathered info is captured in '%s'\\n\",\n-\t\t zip_path.buf);\n-\n-diagnose_cleanup:\n-\tif (archiver_fd >= 0) {\n-\t\tclose(1);\n-\t\tdup2(stdout_fd, 1);\n-\t}\n-\tfree(argv_copy);\n-\tstrvec_clear(&archiver_args);\n-\tstrbuf_release(&zip_path);\n-\tstrbuf_release(&buf);\n+\tif (run_git(\"bugreport\", \"--diagnose\", \"--no-report\",\n+\t\t \"-s\", \"%Y%m%d_%H%M%S\", \"-o\", diagnostics_path.buf, NULL) < 0)\n+\t\tres = -1;\n \n+\tstrbuf_release(&diagnostics_path);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"460383","messageId":"86d40a4bd15dc5f9198998f761f4c4cfa8f31e1d.1659388498.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH 7/7] scalar: update technical doc roadmap","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-01T21:14:58Z","receivedAt":"2022-08-01T21:15:19Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate the Scalar roadmap to reflect the completion of generalizing 'scalar\ndiagnose' into 'git bugreport --diagnose'.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/technical/scalar.txt | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt\nindex 08bc09c225a..1fa2f6d5f91 100644\n--- a/Documentation/technical/scalar.txt\n+++ b/Documentation/technical/scalar.txt\n@@ -84,6 +84,9 @@ series have been accepted:\n \n - `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n \n+- `scalar-generalize-diagnose`: Move the functionality of `scalar diagnose`\n+ into `git bugreport --diagnose`.\n+\n Roughly speaking (and subject to change), the following series are needed to\n \"finish\" this initial version of Scalar:\n \n@@ -91,12 +94,6 @@ Roughly speaking (and subject to change), the following series are needed to\n and implement `scalar help`. At the end of this series, Scalar should be\n feature-complete from the perspective of a user.\n \n-- Generalize features not specific to Scalar: In the spirit of making Scalar\n- configure only what is needed for large repo performance, move common\n- utilities into other parts of Git. Some of this will be internal-only, but one\n- major change will be generalizing `scalar diagnose` for use with any Git\n- repository.\n-\n - Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of\n `git`, including updates to build and install it with the rest of Git. This\n change will incorporate Scalar into the Git CI and test framework, as well as\n-- \ngitgitgadget\n"},{"id":"460387","messageId":"xmqqfsifodg7.fsf@gitster.g","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Generalize 'scalar diagnose' into 'git bugreport --diagnose'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T21:34:32Z","receivedAt":"2022-08-01T21:34:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> As part of the preparation for moving Scalar out of 'contrib/' and into Git,\n> this series moves the functionality of 'scalar diagnose' into a new option\n> ('--diagnose') for 'git bugreport'. This change further aligns Scalar with\n> the objective [1] of having it only contain functionality and settings that\n> benefit large Git repositories, but not all repositories. The diagnostics\n> reported by 'scalar diagnose' relevant for investigating issues in any Git\n> repository, so generating them should be part of a \"normal\" Git builtin.\n\n;-)\n"},{"id":"460388","messageId":"xmqqbkt3ocvn.fsf@gitster.g","threadId":"58251","inReplyTo":"a7a766de29b53610d19e56f270863230f3a2eab6.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] scalar: use \"$GIT_UNZIP\" in 'scalar diagnose' test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T21:46:52Z","receivedAt":"2022-08-01T21:46:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> From: Victoria Dye \n>\n> Use the \"$GIT_UNZIP\" test variable rather than verbatim 'unzip' to unzip the\n> 'scalar diagnose' archive. Using \"$GIT_UNZIP\" is needed to run the Scalar\n> tests on systems where 'unzip' is not in the system path.\n\nMakes sense. It makes it more in line with how a handful of tests\nin t/ already use the zip archive.\n\n"},{"id":"460389","messageId":"xmqqtu6vmwxb.fsf@gitster.g","threadId":"58251","inReplyTo":"932dc8cddacf1841996cd3e0d512b95828213015.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] builtin/bugreport.c: create '--diagnose' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T22:16:48Z","receivedAt":"2022-08-01T22:16:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> From: Victoria Dye \n>\n> Create a '--diagnose' option for 'git bugreport' to collect additional\n> information about the repository and write it to a zipped archive.\n>\n> The \"diagnose\" functionality was originally implemented for Scalar in\n> aa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\n> diagnostics gathered are not specific to Scalar-cloned repositories and\n> could be useful when diagnosing issues in any Git repository.\n>\n> Note that, while this patch appears large, it is mostly copied directly out\n> of 'scalar.c'. Specifically, the functions\n>\n> - dir_file_stats_objects()\n> - dir_file_stats()\n> - count_files()\n> - loose_objs_stats()\n> - add_directory_to_archiver()\n> - get_disk_info()\n\nYup. As this does not \"move\" code across from older place to the\nnew home, it takes a bit of processing to verify the above claim,\nbut\n\n $ git blame -C -C -C -s -b master.. -- builtin/bugreport.c\n\nshows that these are largely verbatim copies.\n\n> +#ifndef WIN32\n> +#include \n> +#endif\n> +\n> +static int get_disk_info(struct strbuf *out)\n> +{\n> +#ifdef WIN32\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +...\n> +\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n> +\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n> +...\n> +#else\n> +...\n> +\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n> +\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n> +...\n> +#endif\n> +\treturn 0;\n> +}\n\nAs a proper part of Git, this part should probably be factored out\nso that a platform specific helper function, implemented in compat/\nlayer, grabs \"available disk space\" number in off_t and the caller\nof the above function becomes\n\n\tstrbuf_realpath(&dir, \".\", 1);\n\tstrbuf_addf(out, \"Available space on '%s:' \", dir.buf);\n\tstrbuf_humanise_bytes(out, get_disk_size(dir.buf));\n\nor something, without having to have #ifdef droppings.\n\n> +static int create_diagnostics_archive(struct strbuf *zip_path)\n> +{\n\nLarge part of this function is also lifted from scalar, and it looks\nOK. One thing I noticed is that \"res\" is explicitly initialized to\n0, but given that the way the code is structured to use the \"we\nprocess sequencially in successful case, and branch out by 'goto'\nimmediately when we see a breakage\" pattern, it may be better to\ninitialize it to -1 (i.e. assume error), or even better, leave it\nuninitialized (i.e. let the compiler notice if a jump to cleanup is\nmade without setting res appropriately).\n\n> +diagnose_cleanup:\n> +\tif (archiver_fd >= 0) {\n> +\t\tclose(1);\n> +\t\tdup2(stdout_fd, 1);\n> +\t}\n> +\tfree(argv_copy);\n> +\tstrvec_clear(&archiver_args);\n> +\tstrbuf_release(&buf);\n\nHmph, stdout_fd is a copy of the file descriptor 1 that was saved\naway at the beginning. Then archiver_fd was created to write into\nthe zip archive, and during the bulk of the function it was dup2'ed\nto the file descriptor 1, to make anything written to the latter\nappear in the zip output.\n\nWhen we successfully opened archive_fd but failed to dup2(), we may\nclose a wrong file desciptor 1 here, but we recover from that by\nusing the saved-away stdout_fd, so we'd be OK. If we did dup2(),\nthen we would be OK, too.\n\nI am wondering if archiver_fd itself is leaking here, though.\n\nAlso, if we failed to open archiver_fd, then we have stdout_fd\nleaking here, I suspect.\n\n> +\treturn res;\n> +}\n\nOther than that, looks good to me.\n"},{"id":"460390","messageId":"xmqqpmhjmwu4.fsf@gitster.g","threadId":"58251","inReplyTo":"e8abfdfa892f5670ed411ed6daef26a4ffa1bfe6.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T22:18:43Z","receivedAt":"2022-08-01T22:18:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> From: Victoria Dye \n>\n> Avoid size_t overflow when reporting the available disk space in\n> 'get_disk_info' by casting the block size and available block count to\n> 'uint64_t' before multiplying them. Without this change, 'st_mult' would\n> (correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\n> bytes of available space.\n\nSane. But shouldn't the cast be to off_t, which is what\nstrbuf_humanise_bytes() takes anyway?\n\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/bugreport.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/bugreport.c b/builtin/bugreport.c\n> index 35b1fc48bf1..720889a37ad 100644\n> --- a/builtin/bugreport.c\n> +++ b/builtin/bugreport.c\n> @@ -258,7 +258,7 @@ static int get_disk_info(struct strbuf *out)\n> \t}\n> \n> \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n> -\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n> +\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n> \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n> \tstrbuf_release(&buf);\n> #endif\n"},{"id":"460391","messageId":"xmqqles7mwnc.fsf@gitster.g","threadId":"58251","inReplyTo":"4bc290fbf43e0193aae288b79249014d899ea34a.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/7] builtin/bugreport.c: add directory to archiver more gently","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T22:22:47Z","receivedAt":"2022-08-01T22:22:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> \tint at_root = !*path;\n> -\tDIR *dir = opendir(at_root ? \".\" : path);\n> +\tDIR *dir;\n> \tstruct dirent *e;\n> \tstruct strbuf buf = STRBUF_INIT;\n> \tsize_t len;\n> \tint res = 0;\n> \n> +\tif (!file_exists(at_root ? \".\" : path)) {\n> +\t\twarning(_(\"directory '%s' does not exist, will not be archived\"), path);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tdir = opendir(at_root ? \".\" : path);\n> \tif (!dir)\n> \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n\nI am not sure if TOCTTOU is how we want to be more gentle. Do we\nrather want to do something like this\n\n\tdir = opendir(...);\n\tif (!dir) {\n\t\tif (errno == ENOENT) {\n\t\t\twarning(_(\"not archiving missing directory '%s'\", path);\n\t\t return 0;\n\t\t}\n return error_errno(_(\"cannot open directory '%s'\"), path);\n\t}\n\nor am I missing something subtle?\n\nThanks.\n\n> diff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\n> index 3cf983aa67f..e9db89ef2c8 100755\n> --- a/t/t0091-bugreport.sh\n> +++ b/t/t0091-bugreport.sh\n> @@ -78,7 +78,7 @@ test_expect_success 'indicates populated hooks' '\n> \ttest_cmp expect actual\n> '\n> \n> -test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n> +test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n> \ttest_when_finished rm -rf report &&\n> \n> \tgit bugreport --diagnose -o report -s test >out &&\n> @@ -98,4 +98,13 @@ test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n> \tgrep \"^Total: [0-9][0-9]*\" out\n> '\n> \n> +test_expect_success '--diagnose warns when archived dir does not exist' '\n> +\ttest_when_finished rm -rf report &&\n> +\n> +\t# Remove logs - not guaranteed to exist\n> +\trm -rf .git/logs &&\n> +\tgit bugreport --diagnose -o report -s test 2>err &&\n> +\tgrep \"directory .\\.git/logs. does not exist, will not be archived\" err\n> +'\n> +\n> test_done\n"},{"id":"460393","messageId":"xmqqbkt3mw8w.fsf@gitster.g","threadId":"58251","inReplyTo":"d6527049a4f62f3b1f008706443a11a0e0816074.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] builtin/bugreport.c: add '--no-report' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T22:31:27Z","receivedAt":"2022-08-01T22:31:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> From: Victoria Dye \n>\n> Add a '--no-report' option to 'git bugreport' to avoid writing the\n> 'git-bugreport-.txt' file. This gives users the option of creating\n> only the diagnostic archive with '--diagnose' and mirroring the behavior of\n> the original 'scalar diagnose' as closely as possible.\n>\n> If a user specifies '--no-report' *without* also specifying '--diagnose',\n> the 'git bugreport' operation is a no-op; a warning message is printed and\n> the command returns with a non-error exit code.\n\nI think this makes sense from scalar side, and I have no objection\nagainst this \"--no-report\" feature existing, but I wonder if those\nwho want to send report may want to have a handy way to tell the\ncommand to \"include\" the diag archive in their report (instead of\ncreating separate report and diagnose files, having to attach two\nfiles to their message). Perhaps that is unneeded, or perhaps that\ncomes in later patches in the series, I dunno.\n\n> +--no-report::\n> +\tDo not write out a 'git-bugreport-.txt' file. This option is\n> +\tintended for use with `--diagnose` when only the diagnostic archive is\n> +\tneeded. If `--no-report` is used without `--diagnose`, `git bugreport`\n> +\tis a no-op.\n\nI wonder if thinking it this way may make the UI simpler to explain.\n\nThe \"git bugreport\" is capable of showing report and diagnose with\nthese two orthogonal options, i.e.\n\n\t--report::\twrites bugreport file\n\t--diagnose::\twrites diagnostic archive\n\nAnd for backward compatibility reasons, the command pretends as if\nyou gave it \"--report\" when you run it without either.\n\nThat way, \"bugreport --diagnose\" will just show diagnostic archive\nwithout having to pass \"--no-report\". There is no need for \"nothing\nto do\", either.\n\n"},{"id":"460401","messageId":"220802.86ilnbgzr8.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"e8abfdfa892f5670ed411ed6daef26a4ffa1bfe6.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-02T02:03:55Z","receivedAt":"2022-08-02T02:12:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n>\n> Avoid size_t overflow when reporting the available disk space in\n> 'get_disk_info' by casting the block size and available block count to\n> 'uint64_t' before multiplying them. Without this change, 'st_mult' would\n> (correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\n> bytes of available space.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/bugreport.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/bugreport.c b/builtin/bugreport.c\n> index 35b1fc48bf1..720889a37ad 100644\n> --- a/builtin/bugreport.c\n> +++ b/builtin/bugreport.c\n> @@ -258,7 +258,7 @@ static int get_disk_info(struct strbuf *out)\n> \t}\n> \n> \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n> -\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n> +\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n\nDoesn't this remove the overflow guard on 64 bit systems to support\nthose 32 bit systems?\n\nI also don't tthink it's correct that this would \"correctly\nreport...\". Before this we were simply assuming that \"size_t\" and\n\"unsigned long\" & \"fsblkcnt_t\" would all yield the same thing.\n\nBut I don't think per [1] and [2] that POSIX is giving us any guarantees\nin that regard, even on 32 bit systems, but perhaps it's a reasonable\nassumption in practice.\n\n1. https://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/statvfs.h.html\n2. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html\n"},{"id":"460402","messageId":"220802.86edxzgz2z.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"932dc8cddacf1841996cd3e0d512b95828213015.1659388498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] builtin/bugreport.c: create '--diagnose' option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-02T02:17:52Z","receivedAt":"2022-08-02T02:26:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n> [...]\n> Documentation/git-bugreport.txt | 11 +-\n> builtin/bugreport.c | 282 +++++++++++++++++++++++++++++++-\n> t/t0091-bugreport.sh | 20 +++\n> 3 files changed, 309 insertions(+), 4 deletions(-)\n> [...]\n\nMaybe it's not easy in this case, but I wonder if this series can't be\nre-arranged in a way that more directly benefits from the diff move\ndetection.\n\nE.g. if we moved the unchanged functions to a new repo-disk-usage.c or\nsomething we could have an intermediate step of having both use that,\nand then going forward would work towards a better lib/built-in\nsplit-up...\n\n> --- a/Documentation/git-bugreport.txt\n> +++ b/Documentation/git-bugreport.txt\n> @@ -8,7 +8,7 @@ git-bugreport - Collect information for user to file a bug report\n> SYNOPSIS\n> --------\n> [verse]\n> -'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n> +'git bugreport' []\n> [...]\n> static const char * const bugreport_usage[] = {\n> -\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n> +\tN_(\"git bugreport []\"),\n> \tNULL\n> };\n\nWe have some built-ins that punt on re-listing the synopsis in the -h\noutput, but we always list the full usage in the SYNOPSIS.\n\nI think both of these hunks should be dropped, instead we should\n(presumably) add a \"git bugreport --diagnose\" to this, and if it\ncombines (or not) with other options, let's update both accordingly.\n\n> [...]\n> +\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n>\n\nIs the \"scalar-diagnose\" here a mistake?\n"},{"id":"460403","messageId":"220802.867d3rgx8n.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Generalize 'scalar diagnose' into 'git bugreport --diagnose'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-02T02:49:15Z","receivedAt":"2022-08-02T03:08:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n\n> [...] I didn't see a major UX benefit of 'git diagnose' vs 'git\n> bugreport --diagnose', so I went with the latter, simpler approach.\n\nI really wanted to like this, but I find the end result here really\nconfusing from a UX perspective.\n\nYou can now run \"git bugreport --diagnose\", which creates a giant *.zip\nfile to go along with your *.txt, but your *.txt makes no reference to\nit.\n\nShould you ... attach it to your bug report to this mailing list, do\nsomething else?\n\nThe documentation doesn't offer much in the way of hints, other than\nsuggesting (with --no-report) that this --diagnose is for something\nentirely different (and that's how \"scalar\" uses it).\n\nI know what it's really for after reading this series, but for \"git\nbugreport\" in particular we should be really careful about not making\nthe UX confusing.\n\nThe generated *.zip contains some really deep info about your repo (and\nnot just metadata, e.g. copies of the index, various logs etc.), someone\ne.g. in a proprietary setting really doesn't want to be sharing that\ninfo.\n\nSo I would like to see real integration into \"git bugreport\", i.e. for\nus to smartly report more repository metrics, e.g. approx number of\nloose objects, the sort of state \"__git_ps1\" might report, etc.\n\nBut I think the end-state here makes things much more confusing for\nusers.\n\n> An alternative implementation considered was creating a new 'git diagnose'\n> builtin, but the new command would end up duplicating much of\n> 'builtin/bugreport.c'.\n\nIt seems we always \"return\" from cmd_bugreport() quite quickly, and we\nbasically only share the code to create the output directory. Just\nduplicating or sharing that seems like a much better approach for now\nthan creating the above UX confusion.\n\nNote that you can also share code between multiple built-ins, even in\nthe same file (see e.g. builtin/{checkout,log}.c). So we could even\nshare something like the safe_create_leading_directories() calling code\nin bugreport.c without libifying it.\n\n> Finally, despite 'scalar diagnose' now being nothing more than a wrapper for\n> 'git bugreport --diagnose', it is not being deprecated in this series.\n> Although deprecation -> removal could be a future cleanup effort, 'scalar\n> diagnose' is kept around for now as an alias for users already accustomed to\n> using it in 'scalar'.\n\nWe don't have a \"make install\" to get a \"scalar\" onto user's systems\nyet, do we really need to worry about those users?\n\nOr is this a reference to the out-of-tree version of \"scalar\", not\ngit.git's?\n\n"},{"id":"460445","messageId":"0f175c9c-726b-4f73-ecd9-ed7df9dee865@github.com","threadId":"58251","inReplyTo":"xmqqtu6vmwxb.fsf@gitster.g","subject":"Re: [PATCH 2/7] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-02T15:40:21Z","receivedAt":"2022-08-02T15:41:03Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> From: Victoria Dye \n>>\n>> Create a '--diagnose' option for 'git bugreport' to collect additional\n>> information about the repository and write it to a zipped archive.\n>>\n>> The \"diagnose\" functionality was originally implemented for Scalar in\n>> aa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\n>> diagnostics gathered are not specific to Scalar-cloned repositories and\n>> could be useful when diagnosing issues in any Git repository.\n>>\n>> Note that, while this patch appears large, it is mostly copied directly out\n>> of 'scalar.c'. Specifically, the functions\n>>\n>> - dir_file_stats_objects()\n>> - dir_file_stats()\n>> - count_files()\n>> - loose_objs_stats()\n>> - add_directory_to_archiver()\n>> - get_disk_info()\n> \n> Yup. As this does not \"move\" code across from older place to the\n> new home, it takes a bit of processing to verify the above claim,\n> but\n> \n> $ git blame -C -C -C -s -b master.. -- builtin/bugreport.c\n> \n> shows that these are largely verbatim copies.\n> \n>> +#ifndef WIN32\n>> +#include \n>> +#endif\n>> +\n>> +static int get_disk_info(struct strbuf *out)\n>> +{\n>> +#ifdef WIN32\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +...\n>> +\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n>> +\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n>> +...\n>> +#else\n>> +...\n>> +\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n>> +\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n>> +...\n>> +#endif\n>> +\treturn 0;\n>> +}\n> \n> As a proper part of Git, this part should probably be factored out\n> so that a platform specific helper function, implemented in compat/\n> layer, grabs \"available disk space\" number in off_t and the caller\n> of the above function becomes\n> \n> \tstrbuf_realpath(&dir, \".\", 1);\n> \tstrbuf_addf(out, \"Available space on '%s:' \", dir.buf);\n> \tstrbuf_humanise_bytes(out, get_disk_size(dir.buf));\n> \n> or something, without having to have #ifdef droppings.\n> \n\nThis makes sense, I'll probably follow an approach similar to what was done\nwith 'compat/compiler.h' in [1] (unless adding to 'git-compat-util.h' would\nbe more appropriate?).\n\n[1] https://lore.kernel.org/git/20200416211807.60811-6-emilyshaffer@google.com/\n\n>> +static int create_diagnostics_archive(struct strbuf *zip_path)\n>> +{\n> \n> Large part of this function is also lifted from scalar, and it looks\n> OK. One thing I noticed is that \"res\" is explicitly initialized to\n> 0, but given that the way the code is structured to use the \"we\n> process sequencially in successful case, and branch out by 'goto'\n> immediately when we see a breakage\" pattern, it may be better to\n> initialize it to -1 (i.e. assume error), or even better, leave it\n> uninitialized (i.e. let the compiler notice if a jump to cleanup is\n> made without setting res appropriately).\n> \n\nI'll go with the \"uninitialized\" approach in the re-roll; I like the\nsimplicity of relying on the compiler to determine if it's unassigned.\n\n>> +diagnose_cleanup:\n>> +\tif (archiver_fd >= 0) {\n>> +\t\tclose(1);\n>> +\t\tdup2(stdout_fd, 1);\n>> +\t}\n>> +\tfree(argv_copy);\n>> +\tstrvec_clear(&archiver_args);\n>> +\tstrbuf_release(&buf);\n> \n> Hmph, stdout_fd is a copy of the file descriptor 1 that was saved\n> away at the beginning. Then archiver_fd was created to write into\n> the zip archive, and during the bulk of the function it was dup2'ed\n> to the file descriptor 1, to make anything written to the latter\n> appear in the zip output.\n> \n> When we successfully opened archive_fd but failed to dup2(), we may\n> close a wrong file desciptor 1 here, but we recover from that by\n> using the saved-away stdout_fd, so we'd be OK. If we did dup2(),\n> then we would be OK, too.\n> \n> I am wondering if archiver_fd itself is leaking here, though.\n> \n> Also, if we failed to open archiver_fd, then we have stdout_fd\n> leaking here, I suspect.\n> \n\nIf I'm not mistaken, both 'archiver_fd' and 'stdout_fd' are always leaked if\nthey're successfully created (they're never 'close()'d). There's also an\nunnecessary check for 'archiver_fd < 0', since 'xopen()' will die if it\ncan't open the file. And, as you mentioned, the wrong file descriptor 1 is\nclosed if the 'dup2()' of 'archiver_fd' fails.\n\nI'll clean this up for V2, thanks.\n\n>> +\treturn res;\n>> +}\n> \n> Other than that, looks good to me.\n\n"},{"id":"460446","messageId":"760ebe95-552f-c0d7-18c3-f19671fdd192@github.com","threadId":"58251","inReplyTo":"xmqqles7mwnc.fsf@gitster.g","subject":"Re: [PATCH 4/7] builtin/bugreport.c: add directory to archiver more gently","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-02T15:43:11Z","receivedAt":"2022-08-02T15:43:18Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> \tint at_root = !*path;\n>> -\tDIR *dir = opendir(at_root ? \".\" : path);\n>> +\tDIR *dir;\n>> \tstruct dirent *e;\n>> \tstruct strbuf buf = STRBUF_INIT;\n>> \tsize_t len;\n>> \tint res = 0;\n>> \n>> +\tif (!file_exists(at_root ? \".\" : path)) {\n>> +\t\twarning(_(\"directory '%s' does not exist, will not be archived\"), path);\n>> +\t\treturn 0;\n>> +\t}\n>> +\n>> +\tdir = opendir(at_root ? \".\" : path);\n>> \tif (!dir)\n>> \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n> \n> I am not sure if TOCTTOU is how we want to be more gentle. Do we\n> rather want to do something like this\n> \n> \tdir = opendir(...);\n> \tif (!dir) {\n> \t\tif (errno == ENOENT) {\n> \t\t\twarning(_(\"not archiving missing directory '%s'\", path);\n> \t\t return 0;\n> \t\t}\n> return error_errno(_(\"cannot open directory '%s'\"), path);\n> \t}\n> \n> or am I missing something subtle?\n> \n\nThe \"gentleness\" was meant to be a reference only to the error -> warning\nchange, the TOCTTOU change was just a miss by me. I'll fix it in the next\nversion, thanks!\n\n> Thanks.\n> \n>> diff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\n>> index 3cf983aa67f..e9db89ef2c8 100755\n>> --- a/t/t0091-bugreport.sh\n>> +++ b/t/t0091-bugreport.sh\n>> @@ -78,7 +78,7 @@ test_expect_success 'indicates populated hooks' '\n>> \ttest_cmp expect actual\n>> '\n>> \n>> -test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n>> +test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n>> \ttest_when_finished rm -rf report &&\n>> \n>> \tgit bugreport --diagnose -o report -s test >out &&\n>> @@ -98,4 +98,13 @@ test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n>> \tgrep \"^Total: [0-9][0-9]*\" out\n>> '\n>> \n>> +test_expect_success '--diagnose warns when archived dir does not exist' '\n>> +\ttest_when_finished rm -rf report &&\n>> +\n>> +\t# Remove logs - not guaranteed to exist\n>> +\trm -rf .git/logs &&\n>> +\tgit bugreport --diagnose -o report -s test 2>err &&\n>> +\tgrep \"directory .\\.git/logs. does not exist, will not be archived\" err\n>> +'\n>> +\n>> test_done\n\n"},{"id":"460496","messageId":"c075e483-6e00-5de4-b923-af8983ba384e@github.com","threadId":"58251","inReplyTo":"xmqqpmhjmwu4.fsf@gitster.g","subject":"Re: [PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-02T16:26:37Z","receivedAt":"2022-08-02T16:26:54Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> From: Victoria Dye \n>>\n>> Avoid size_t overflow when reporting the available disk space in\n>> 'get_disk_info' by casting the block size and available block count to\n>> 'uint64_t' before multiplying them. Without this change, 'st_mult' would\n>> (correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\n>> bytes of available space.\n> \n> Sane. But shouldn't the cast be to off_t, which is what\n> strbuf_humanise_bytes() takes anyway?\n> \n\nI chose 'uint64_t' to mimic 'throughput_string()' [1], but the signed\n'off_t' is a better choice given its use in 'strbuf_humanise_bytes()'.\n\nOn a related note, while writing this I made the (unsubstantiated)\nassumption that 'off_t' would be a 64-bit int, even on 32-bit systems. Your\ncomment prompted me to confirm that assumption; while 'off_t' is not always\nguaranteed to be an int64 by default [2], Git is compiled with '#define\n_FILE_OFFSET_BITS 64' [3] so 'off_t' is equivalent to 'off64_t'.\n\nI'll update the casts to 'off_t' in V2. Thanks!\n\n[1] https://lore.kernel.org/git/20171110173956.25105-4-newren@gmail.com/\n[2] https://www.gnu.org/software/libc/manual/html_mono/libc.html#index-off_005ft\n[3] https://lore.kernel.org/git/7vr6smc1de.fsf@assigned-by-dhcp.cox.net/\n\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/bugreport.c | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/builtin/bugreport.c b/builtin/bugreport.c\n>> index 35b1fc48bf1..720889a37ad 100644\n>> --- a/builtin/bugreport.c\n>> +++ b/builtin/bugreport.c\n>> @@ -258,7 +258,7 @@ static int get_disk_info(struct strbuf *out)\n>> \t}\n>> \n>> \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n>> -\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n>> +\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n>> \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n>> \tstrbuf_release(&buf);\n>> #endif\n\n"},{"id":"460497","messageId":"9fdac51b-ff8c-9564-d349-a71e42708479@github.com","threadId":"58251","inReplyTo":"220802.86ilnbgzr8.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-02T16:26:42Z","receivedAt":"2022-08-02T16:26:57Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> From: Victoria Dye \n>>\n>> Avoid size_t overflow when reporting the available disk space in\n>> 'get_disk_info' by casting the block size and available block count to\n>> 'uint64_t' before multiplying them. Without this change, 'st_mult' would\n>> (correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\n>> bytes of available space.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/bugreport.c | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/builtin/bugreport.c b/builtin/bugreport.c\n>> index 35b1fc48bf1..720889a37ad 100644\n>> --- a/builtin/bugreport.c\n>> +++ b/builtin/bugreport.c\n>> @@ -258,7 +258,7 @@ static int get_disk_info(struct strbuf *out)\n>> \t}\n>> \n>> \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n>> -\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n>> +\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n> \n> Doesn't this remove the overflow guard on 64 bit systems to support\n> those 32 bit systems?\n> \n\nIt does, but the total disk space available on a system should be able to\nfit into a 64-bit integer. I considered adding an explicit\n'unsigned_mult_overflows', but decided against it because it's almost\ncertainly overkill for such an implausible edge case.\n\n> I also don't tthink it's correct that this would \"correctly\n> report...\". Before this we were simply assuming that \"size_t\" and\n> \"unsigned long\" & \"fsblkcnt_t\" would all yield the same thing.\n> \n\nThe point I was making is that, if your 'size_t' is 32 bits, but you have\nmore than ~4GB of disk space available on your system, the result of the\nmultiplication will overflow 'size_t'. So, 'st_mult' failing because it\ndetects an overflow is \"correct\", rather than e.g. a false positive.\n\n> But I don't think per [1] and [2] that POSIX is giving us any guarantees\n> in that regard, even on 32 bit systems, but perhaps it's a reasonable\n> assumption in practice.\n> \n> 1. https://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/statvfs.h.html\n> 2. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html\n\n"},{"id":"460500","messageId":"f3235afe-25cc-21a4-fc35-56e35d6be0ce@github.com","threadId":"58251","inReplyTo":"xmqqbkt3mw8w.fsf@gitster.g","subject":"Re: [PATCH 5/7] builtin/bugreport.c: add '--no-report' option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-02T19:46:27Z","receivedAt":"2022-08-02T19:46:35Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> From: Victoria Dye \n>>\n>> Add a '--no-report' option to 'git bugreport' to avoid writing the\n>> 'git-bugreport-.txt' file. This gives users the option of creating\n>> only the diagnostic archive with '--diagnose' and mirroring the behavior of\n>> the original 'scalar diagnose' as closely as possible.\n>>\n>> If a user specifies '--no-report' *without* also specifying '--diagnose',\n>> the 'git bugreport' operation is a no-op; a warning message is printed and\n>> the command returns with a non-error exit code.\n> \n> I think this makes sense from scalar side, and I have no objection\n> against this \"--no-report\" feature existing, but I wonder if those\n> who want to send report may want to have a handy way to tell the\n> command to \"include\" the diag archive in their report (instead of\n> creating separate report and diagnose files, having to attach two\n> files to their message). Perhaps that is unneeded, or perhaps that\n> comes in later patches in the series, I dunno.\n> \n\nI tried finding where in the documentation there are instructions on sending\na bug report to the mailing list, but didn't see anything (otherwise, I'd\nadd some info on '--diagnose' there). Maybe Emily would know?\n\nIf instructions like that don't exist, I'll update the command documentation\nhere to clarify that '--diagnose' generates an attachment that includes more\ncomplete repository information to aid in debugging.\n\n>> +--no-report::\n>> +\tDo not write out a 'git-bugreport-.txt' file. This option is\n>> +\tintended for use with `--diagnose` when only the diagnostic archive is\n>> +\tneeded. If `--no-report` is used without `--diagnose`, `git bugreport`\n>> +\tis a no-op.\n> \n> I wonder if thinking it this way may make the UI simpler to explain.\n> \n> The \"git bugreport\" is capable of showing report and diagnose with\n> these two orthogonal options, i.e.\n> \n> \t--report::\twrites bugreport file\n> \t--diagnose::\twrites diagnostic archive\n> \n> And for backward compatibility reasons, the command pretends as if\n> you gave it \"--report\" when you run it without either.\n> \n> That way, \"bugreport --diagnose\" will just show diagnostic archive\n> without having to pass \"--no-report\". There is no need for \"nothing\n> to do\", either.\n> \n\nI like the simplicity of this, but I'd imagine that a user would want to\ngenerate diagnostics *with* a report more often than without one. The cases\nI can think of for \"standalone diagnostics\" are: internally in 'scalar\ndiagnose', someone requesting more info after an initial bug report, or a\nuser looking into something on their own.\n\nMaybe I could replace '--no-report' with '--diagnostics-only'? Then the\nthree modes of use would be:\n\n- 'git bugreport': report only (most common usage)\n- 'git bugreport --diagnose': report + diagnostics\n- 'git bugreport --diagnostics-only': diagnostics only (least common usage)\n\nIt would eliminate the need for \"nothing to do\" while making it (I think?)\nclearer to a user why you would want to use any of these options.\n"},{"id":"460501","messageId":"392ae6f1-c2f1-ab0f-c832-f8e53800506a@github.com","threadId":"58251","inReplyTo":"220802.867d3rgx8n.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 0/7] Generalize 'scalar diagnose' into 'git bugreport --diagnose'","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-02T19:48:34Z","receivedAt":"2022-08-02T19:48:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> [...] I didn't see a major UX benefit of 'git diagnose' vs 'git\n>> bugreport --diagnose', so I went with the latter, simpler approach.\n> \n> I really wanted to like this, but I find the end result here really\n> confusing from a UX perspective.\n> \n> You can now run \"git bugreport --diagnose\", which creates a giant *.zip\n> file to go along with your *.txt, but your *.txt makes no reference to\n> it.\n> \n> Should you ... attach it to your bug report to this mailing list, do\n> something else?\n> \n> The documentation doesn't offer much in the way of hints, other than\n> suggesting (with --no-report) that this --diagnose is for something\n> entirely different (and that's how \"scalar\" uses it).\n> \n> I know what it's really for after reading this series, but for \"git\n> bugreport\" in particular we should be really careful about not making\n> the UX confusing.\n> \n> The generated *.zip contains some really deep info about your repo (and\n> not just metadata, e.g. copies of the index, various logs etc.), someone\n> e.g. in a proprietary setting really doesn't want to be sharing that\n> info.\n> \n> So I would like to see real integration into \"git bugreport\", i.e. for\n> us to smartly report more repository metrics, e.g. approx number of\n> loose objects, the sort of state \"__git_ps1\" might report, etc.\n> \n> But I think the end-state here makes things much more confusing for\n> users.\n> \n\nThe \"confusing UX\" you describe here doesn't seem to be an inherent issue\nwith the implementation (nor is it as insurmountable as you're implying), it\nlargely appears to be an issue of under-documentation. I'll improve that in\nV2 [1], but I want clarify what I was/am going for here as well.\n\nIn the context of a bug report, the diagnostics are intended to be used as\nsupplemental information to aid in debugging (i.e., attached with the report\nin the email to the list). They're especially valuable when a bug reporter\nisn't very familiar with Git internals and they can't reproduce the issue. A\nlot of bugs can be investigated without those diagnostics, though, which is\nwhy '--diagnose' isn't \"on\" by default.\n\nThere are also valid use-cases (beyond 'scalar diagnose') for '--no-report':\nsomeone requests more information after looking into an already-generated\nreport, or a developer wants to investigate a bug on their own and use the\ndiagnostics as a \"starting point\" to guide more in-depth debugging. \n\nAs for the proprietary data issue, I'd be open to having an option to\nconfigure which diagnostics a user wants (either something like '--diagnose\n' or a separate option entirely). I'm pretty indifferent on the UI,\nthough, so I'll defer to other contributors on 1) if they want that feature,\nand 2) what they think that should look like.\n\n[1] https://lore.kernel.org/git/f3235afe-25cc-21a4-fc35-56e35d6be0ce@github.com/\n\n>> An alternative implementation considered was creating a new 'git diagnose'\n>> builtin, but the new command would end up duplicating much of\n>> 'builtin/bugreport.c'.\n> \n> It seems we always \"return\" from cmd_bugreport() quite quickly, and we\n> basically only share the code to create the output directory. Just\n> duplicating or sharing that seems like a much better approach for now\n> than creating the above UX confusion.\n> \n> Note that you can also share code between multiple built-ins, even in\n> the same file (see e.g. builtin/{checkout,log}.c). So we could even\n> share something like the safe_create_leading_directories() calling code\n> in bugreport.c without libifying it.\n> \n\nYou deleted the part where I addressed this suggestion directly:\n\n> Although that issue could be overcome with refactoring, I didn't see a\n> major UX benefit of 'git diagnose' vs 'git bugreport --diagnose', so I\n> went with the latter, simpler approach.\n\nAnd, in the process of writing down my thoughts on the UX above, I've become\nmore convinced that including '--diagnose' in 'git bugreport' is the better\nway to present this functionality to users.\n\n>> Finally, despite 'scalar diagnose' now being nothing more than a wrapper for\n>> 'git bugreport --diagnose', it is not being deprecated in this series.\n>> Although deprecation -> removal could be a future cleanup effort, 'scalar\n>> diagnose' is kept around for now as an alias for users already accustomed to\n>> using it in 'scalar'.\n> \n> We don't have a \"make install\" to get a \"scalar\" onto user's systems\n> yet, do we really need to worry about those users?\n> \n> Or is this a reference to the out-of-tree version of \"scalar\", not\n> git.git's?\n> \n\nIn practice, it's the \"out-of-tree Scalar\" users that would care the most.\nThat said, with Scalar in the Git tree (albeit 'contrib/' and not built by\ndefault), I think it's reasonable to want to avoid breaking changes if\npossible. The continued existence of 'scalar diagnose' wouldn't really be\nhurting anyone anyway, so there's no pressing need to deprecate it now.\n"},{"id":"460502","messageId":"xmqqwnbqfjx4.fsf@gitster.g","threadId":"58251","inReplyTo":"c075e483-6e00-5de4-b923-af8983ba384e@github.com","subject":"Re: [PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-02T20:51:51Z","receivedAt":"2022-08-02T20:52:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> I chose 'uint64_t' to mimic 'throughput_string()' [1], but the signed\n> 'off_t' is a better choice given its use in 'strbuf_humanise_bytes()'.\n>\n> On a related note, while writing this I made the (unsubstantiated)\n> assumption that 'off_t' would be a 64-bit int, even on 32-bit systems. Your\n> comment prompted me to confirm that assumption; while 'off_t' is not always\n> guaranteed to be an int64 by default [2], Git is compiled with '#define\n> _FILE_OFFSET_BITS 64' [3] so 'off_t' is equivalent to 'off64_t'.\n\nOffset into a single file can be smaller than the size of the whole\ndisk, after all, so from that point of view, use of off_t in\n\"humanise_bytes\" API may be something we would want to fix\neventually to reduce confusion.\n\nI do not particularly mind casting up to uint64, especially if that\nmatches the code lifted from scalar. As long as our longer term\nplan is to update strbuf_humanise_bytes() to take the widest\npossible unsigned integer, what we do right now would not make that\nmuch of a difference.\n\nThanks.\n"},{"id":"460541","messageId":"220803.86tu6tfqvf.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"9fdac51b-ff8c-9564-d349-a71e42708479@github.com","subject":"Re: [PATCH 3/7] builtin/bugreport.c: avoid size_t overflow","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-03T12:25:23Z","receivedAt":"2022-08-03T12:34:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 02 2022, Victoria Dye wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n>> \n>>> From: Victoria Dye \n>>>\n>>> Avoid size_t overflow when reporting the available disk space in\n>>> 'get_disk_info' by casting the block size and available block count to\n>>> 'uint64_t' before multiplying them. Without this change, 'st_mult' would\n>>> (correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\n>>> bytes of available space.\n>>>\n>>> Signed-off-by: Victoria Dye \n>>> ---\n>>> builtin/bugreport.c | 2 +-\n>>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>>\n>>> diff --git a/builtin/bugreport.c b/builtin/bugreport.c\n>>> index 35b1fc48bf1..720889a37ad 100644\n>>> --- a/builtin/bugreport.c\n>>> +++ b/builtin/bugreport.c\n>>> @@ -258,7 +258,7 @@ static int get_disk_info(struct strbuf *out)\n>>> \t}\n>>> \n>>> \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n>>> -\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n>>> +\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n>> \n>> Doesn't this remove the overflow guard on 64 bit systems to support\n>> those 32 bit systems?\n>> \n>\n> It does, but the total disk space available on a system should be able to\n> fit into a 64-bit integer. I considered adding an explicit\n> 'unsigned_mult_overflows', but decided against it because it's almost\n> certainly overkill for such an implausible edge case.\n\nYeah it's probably overkill, and maybe this is good as-is & we don't\nneed to worry here.\n\nBut that's quite different from what the patch says, it's not \"avoid\nsize_t overflow\" but e.g.:\n\n\tbugreport.c: don't do size_t overflow check before casting to 32bit\n\n\tIt's a hassle to support the check on 32 bit systems, and we\n\tdon't think this is something we'll run into in practice [...]\n\nPerhaps?\n\n>> I also don't tthink it's correct that this would \"correctly\n>> report...\". Before this we were simply assuming that \"size_t\" and\n>> \"unsigned long\" & \"fsblkcnt_t\" would all yield the same thing.\n>> \n>\n> The point I was making is that, if your 'size_t' is 32 bits, but you have\n> more than ~4GB of disk space available on your system, the result of the\n> multiplication will overflow 'size_t'. So, 'st_mult' failing because it\n> detects an overflow is \"correct\", rather than e.g. a false positive.\n\nI think it would be useful to document these assumptions in the commit\nmessage, POSIX just says \"blkcnt_t and off_t shall be signed integer\ntypes\", and \"size_t shall be an unsigned integer type.\".\n\nDo other bits of the standard(s) that I've missed say that off_t's\nsigned type must be double the width of size_t's unsigned, or is it one\nof those things that's not standardized but can be relied on in\npractice?\n\nWe have a related assertion in 37ee680d9b9 (http.postbuffer: allow full\nrange of ssize_t values, 2017-04-11) (xcurl_off_t()). Perhaps you want\nto do something similar to sanity check your assumptions here?\n\n1. https://pubs.opengroup.org/onlinepubs/009604599/basedefs/sys/types.h.html\n"},{"id":"460542","messageId":"220803.86pmhhfpjb.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"392ae6f1-c2f1-ab0f-c832-f8e53800506a@github.com","subject":"Re: [PATCH 0/7] Generalize 'scalar diagnose' into 'git bugreport --diagnose'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-03T12:34:04Z","receivedAt":"2022-08-03T13:02:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 02 2022, Victoria Dye wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Aug 01 2022, Victoria Dye via GitGitGadget wrote:\n>> \n>>> [...] I didn't see a major UX benefit of 'git diagnose' vs 'git\n>>> bugreport --diagnose', so I went with the latter, simpler approach.\n>> \n>> I really wanted to like this, but I find the end result here really\n>> confusing from a UX perspective.\n>> \n>> You can now run \"git bugreport --diagnose\", which creates a giant *.zip\n>> file to go along with your *.txt, but your *.txt makes no reference to\n>> it.\n>> \n>> Should you ... attach it to your bug report to this mailing list, do\n>> something else?\n>> \n>> The documentation doesn't offer much in the way of hints, other than\n>> suggesting (with --no-report) that this --diagnose is for something\n>> entirely different (and that's how \"scalar\" uses it).\n>> \n>> I know what it's really for after reading this series, but for \"git\n>> bugreport\" in particular we should be really careful about not making\n>> the UX confusing.\n>> \n>> The generated *.zip contains some really deep info about your repo (and\n>> not just metadata, e.g. copies of the index, various logs etc.), someone\n>> e.g. in a proprietary setting really doesn't want to be sharing that\n>> info.\n>> \n>> So I would like to see real integration into \"git bugreport\", i.e. for\n>> us to smartly report more repository metrics, e.g. approx number of\n>> loose objects, the sort of state \"__git_ps1\" might report, etc.\n>> \n>> But I think the end-state here makes things much more confusing for\n>> users.\n>> \n>\n> The \"confusing UX\" you describe here doesn't seem to be an inherent issue\n> with the implementation (nor is it as insurmountable as you're implying),\n\nI'm not implying or saying that it's insurmountable.\n\nI think in principle having such a mode in \"git bugreport\" would make\nsense.\n\nBut the UX here seems to be an afterthought. So I wonder if we shouldn't\nhold off on it and just have a new *--helper or something instead.\n\n> it\n> largely appears to be an issue of under-documentation. I'll improve that in\n> V2 [1], but I want clarify what I was/am going for here as well.\n>\n> In the context of a bug report, the diagnostics are intended to be used as\n> supplemental information to aid in debugging (i.e., attached with the report\n> in the email to the list).\n\nPer https://lore.kernel.org/git/?q=n%3Azip we don't block *.zip\nattachments, but we have fairly low size limits (to the point of\nblocking some large patches).\n\nE.g. e448263716f (po/git.pot: this is now a generated file, 2022-05-26)\n(https://lore.kernel.org/git/20220526145035.18958-7-worldhello.net@gmail.com/)\nwas blocked, and it's just over 500k.\n\nAside from the data-sharing issues that seems like a good addition to\ngit-bugreport, i.e. tell the user if the attachment would be blocked due\nto its size...\n\n> They're especially valuable when a bug reporter\n> isn't very familiar with Git internals and they can't reproduce the issue. A\n> lot of bugs can be investigated without those diagnostics, though, which is\n> why '--diagnose' isn't \"on\" by default.\n>\n> There are also valid use-cases (beyond 'scalar diagnose') for '--no-report':\n> someone requests more information after looking into an already-generated\n> report, or a developer wants to investigate a bug on their own and use the\n> diagnostics as a \"starting point\" to guide more in-depth debugging. \n\nYes, it's useful in a lot of cases. I'm just saying that we really need\nto bridge the gap of telling the user what they should be doing with\nthis new file....\n\n> As for the proprietary data issue, I'd be open to having an option to\n> configure which diagnostics a user wants (either something like '--diagnose\n> ' or a separate option entirely). I'm pretty indifferent on the UI,\n> though, so I'll defer to other contributors on 1) if they want that feature,\n> and 2) what they think that should look like.\n\nI think it's really important that a bug report feature doesn't submit\nprivate data by default. The name of a \"--diagnose\" option heavily\nimplies some aggregate metrics etc.\n\nWe then attach what to the user are opaque binary files, but which\ncontain even the file contants of the repository (index files).\n\nI think scalar was developer for \"internal\" use-cases where such sharing\nwasn't an issue for anyone, but I don't want anyone to get in trouble\nbecause they shared their proprietary source code on the ML while trying\nto file a bug report, so defaults really matter.\n\nEspecially because if there's one thing we've learned from bug reports &\nquestions to this & git-users is that the one thing you can rely on is\nthat users routinely don't carefully scrutinize documentation before\ntrying out various features...\n\n> [1] https://lore.kernel.org/git/f3235afe-25cc-21a4-fc35-56e35d6be0ce@github.com/\n>\n>>> An alternative implementation considered was creating a new 'git diagnose'\n>>> builtin, but the new command would end up duplicating much of\n>>> 'builtin/bugreport.c'.\n>> \n>> It seems we always \"return\" from cmd_bugreport() quite quickly, and we\n>> basically only share the code to create the output directory. Just\n>> duplicating or sharing that seems like a much better approach for now\n>> than creating the above UX confusion.\n>> \n>> Note that you can also share code between multiple built-ins, even in\n>> the same file (see e.g. builtin/{checkout,log}.c). So we could even\n>> share something like the safe_create_leading_directories() calling code\n>> in bugreport.c without libifying it.\n>> \n>\n> You deleted the part where I addressed this suggestion directly:\n\nYes, the \"Although that...\" sentence, but I commented on the UX\ntrade-off elsewhere.\n\n>> Although that issue could be overcome with refactoring, I didn't see a\n>> major UX benefit of 'git diagnose' vs 'git bugreport --diagnose', so I\n>> went with the latter, simpler approach.\n>\n> And, in the process of writing down my thoughts on the UX above, I've become\n> more convinced that including '--diagnose' in 'git bugreport' is the better\n> way to present this functionality to users.\n\nWe're on the same page there, we're just discussing if/how to make the\nend-to-end process clearer to users.\n\nI don't think it's clear in v1, and is sorely missing something like the\ndiscussion we have around \"ANONYMIZING\" in git-fast-export(1), and we\nreally should have a \"safe by default\". Everything I'm noting here would\nbe addressed by e.g.:\n\n * The *.txt output would say \"you can additionally attach a diagnostic\n *.zip\" file etc. etc., noting if/when the user would do that.\n\n * We'd have that --diagnostics be safe by default in the way\n --anonymize is, e.g. including stats about number of refs & the like,\n not their contents.\n\n * We could also have e.g. \"--diagnostics --no-anonymize\", or\n \"--diagnostics=full\" or whatever, which would be some approximation\n of the current output.\n\n>>> Finally, despite 'scalar diagnose' now being nothing more than a wrapper for\n>>> 'git bugreport --diagnose', it is not being deprecated in this series.\n>>> Although deprecation -> removal could be a future cleanup effort, 'scalar\n>>> diagnose' is kept around for now as an alias for users already accustomed to\n>>> using it in 'scalar'.\n>> \n>> We don't have a \"make install\" to get a \"scalar\" onto user's systems\n>> yet, do we really need to worry about those users?\n>> \n>> Or is this a reference to the out-of-tree version of \"scalar\", not\n>> git.git's?\n>> \n>\n> In practice, it's the \"out-of-tree Scalar\" users that would care the most.\n> That said, with Scalar in the Git tree (albeit 'contrib/' and not built by\n> default), I think it's reasonable to want to avoid breaking changes if\n> possible. The continued existence of 'scalar diagnose' wouldn't really be\n> hurting anyone anyway, so there's no pressing need to deprecate it now.\n\nI'm fine with keeping it, but just found the juxtaposition of doing that\n& previous discussions about scalar to be surprising.\n\nWhen it was submitted (from my recollection, I see\nDocumentation/technical/scalar.txt is rather light on the topic) the\nidea was (and this is from my own recollection, I haven't dug up ML\nreferences) that scalar would be \"going away\", and that the reason to\nhave it in-tree was to inform the design of the main \"git\" tooling for\nthe use-cases scalar is addressing.\n\nSo, a bit similar to Cogito (although that was never in-tree).\n\nI'm fine with just keeping it as an alias for \"git bugreport\n--whatever\", but think it would make even more sense to change the\ninterface a bit as we integrate it to the rest of git, and then \"die()\"\nsaying \"'scalar xyz' is now 'git abc', mostly....\".\n\nExisting scalar users used to 'scalar xyz' would then try the new 'git\nabc', and report if/how that fits their use-case.\n\nThen at some point if we do that with all of scalar's features we'd be\nat the point of 'git rm'-ing it, as it would have served its stated\npurpose.\n\nJust my 0.02, I think in any case having the scalar.txt doc's \"Roadmap\"\nupdated to address this would be very useful.\n"},{"id":"460586","messageId":"ad5b60bf11ef099e709dc9b69d80c28e196c4543.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 01/10] scalar-diagnose: use \"$GIT_UNZIP\" in test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:34Z","receivedAt":"2022-08-04T01:45:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUse the \"$GIT_UNZIP\" test variable rather than verbatim 'unzip' to unzip the\n'scalar diagnose' archive. Using \"$GIT_UNZIP\" is needed to run the Scalar\ntests on systems where 'unzip' is not in the system path.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/t/t9099-scalar.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh\nindex 10b1172a8aa..fac86a57550 100755\n--- a/contrib/scalar/t/t9099-scalar.sh\n+++ b/contrib/scalar/t/t9099-scalar.sh\n@@ -109,14 +109,14 @@ test_expect_success UNZIP 'scalar diagnose' '\n \tsed -n \"s/.*$SQ\\\\(.*\\\\.zip\\\\)$SQ.*/\\\\1/p\" zip_path &&\n \tzip_path=$(cat zip_path) &&\n \ttest -n \"$zip_path\" &&\n-\tunzip -v \"$zip_path\" &&\n+\t\"$GIT_UNZIP\" -v \"$zip_path\" &&\n \tfolder=${zip_path%.zip} &&\n \ttest_path_is_missing \"$folder\" &&\n-\tunzip -p \"$zip_path\" diagnostics.log >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n \ttest_file_not_empty out &&\n-\tunzip -p \"$zip_path\" packs-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n \tgrep \"$(pwd)/.git/objects\" out &&\n-\tunzip -p \"$zip_path\" objects-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n \tgrep \"^Total: [1-9]\" out\n '\n \n-- \ngitgitgadget\n\n"},{"id":"460587","messageId":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.git.1659388498.gitgitgadget@gmail.com","subject":"[PATCH v2 00/10] Generalize 'scalar diagnose' into 'git diagnose' and 'git bugreport --diagnose'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:33Z","receivedAt":"2022-08-04T01:45:56Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"As part of the preparation for moving Scalar out of 'contrib/' and into Git,\nthis series moves the functionality of 'scalar diagnose' into a new builtin\n('git diagnose') and a new option ('--diagnose') for 'git bugreport'. This\nchange further aligns Scalar with the objective [1] of having it only\ncontain functionality and settings that benefit large Git repositories, but\nnot all repositories. The diagnostics reported by 'scalar diagnose' relevant\nfor investigating issues in any Git repository, so generating them should be\npart of a \"normal\" Git builtin.\n\nThe series is organized as follows:\n\n * Miscellaneous fixes for the existing 'scalar diagnose' implementation\n * Moving the code for generating diagnostics into a common location in the\n Git tree\n * Implementing 'git diagnose'\n * Implementing 'git bugreport --diagnose'\n * Updating the Scalar roadmap\n\nFinally, despite 'scalar diagnose' now being nothing more than a wrapper for\n'git bugreport --diagnose', it is not being deprecated in this series.\nAlthough deprecation -> removal could be a future cleanup effort, 'scalar\ndiagnose' is kept around for now as an alias for users already accustomed to\nusing it in 'scalar'.\n\n\nChanges since V1\n================\n\n * Reorganized patches to fix minor issues (e.g., more gently adding\n directories to the archive) of 'scalar diagnose' in 'scalar.c', before\n the code is moved out of that file.\n * (Almost) entirely redesigned the UI for generating diagnostics. The new\n approach avoids cluttering 'git bugreport' with a mode that doesn't\n actually generate a report. Now, there are distinct options for different\n use cases: generating extra diagnostics with a bug report ('git bugreport\n --diagnose') and generating diagnostics for personal debugging/addition\n to an existing bug report ('git diagnose').\n * Moved 'get_disk_info()' into 'compat/'.\n * Moved 'create_diagnostics_archive()' into a new 'diagnose.c', as it now\n has multiple callers.\n * Updated command & option documentation to more clearly guide users on how\n to use the new options.\n * Added the '--all' (and '--diagnose=all') option to change the default\n behavior of diagnostics generation to exclude '.git' directory contents.\n For many bug reporters, this would reveal private repository contents\n they don't want to expose to the public mailing list. This has the added\n benefit of creating much smaller archives by default, which will be more\n likely to successfully send to the mailing list.\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1275.v2.git.1657584367.gitgitgadget@gmail.com/\n\nVictoria Dye (10):\n scalar-diagnose: use \"$GIT_UNZIP\" in test\n scalar-diagnose: avoid 32-bit overflow of size_t\n scalar-diagnose: add directory to archiver more gently\n scalar-diagnose: move 'get_disk_info()' to 'compat/'\n scalar-diagnose: move functionality to common location\n builtin/diagnose.c: create 'git diagnose' builtin\n builtin/diagnose.c: gate certain data behind '--all'\n builtin/bugreport.c: create '--diagnose' option\n scalar-diagnose: use 'git diagnose --all'\n scalar: update technical doc roadmap\n\n .gitignore | 1 +\n Documentation/git-bugreport.txt | 18 ++\n Documentation/git-diagnose.txt | 60 +++++++\n Documentation/technical/scalar.txt | 9 +-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/bugreport.c | 47 ++++-\n builtin/diagnose.c | 62 +++++++\n compat/disk.h | 56 ++++++\n contrib/scalar/scalar.c | 272 +----------------------------\n contrib/scalar/t/t9099-scalar.sh | 8 +-\n diagnose.c | 218 +++++++++++++++++++++++\n diagnose.h | 9 +\n git-compat-util.h | 1 +\n git.c | 1 +\n t/t0091-bugreport.sh | 44 +++++\n t/t0092-diagnose.sh | 43 +++++\n 17 files changed, 575 insertions(+), 277 deletions(-)\n create mode 100644 Documentation/git-diagnose.txt\n create mode 100644 builtin/diagnose.c\n create mode 100644 compat/disk.h\n create mode 100644 diagnose.c\n create mode 100644 diagnose.h\n create mode 100755 t/t0092-diagnose.sh\n\n\nbase-commit: 23b219f8e3f2adfb0441e135f0a880e6124f766c\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1310%2Fvdye%2Fscalar%2Fgeneralize-diagnose-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1310/vdye/scalar/generalize-diagnose-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1310\n\nRange-diff vs v1:\n\n 1: a7a766de29b ! 1: ad5b60bf11e scalar: use \"$GIT_UNZIP\" in 'scalar diagnose' test\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - scalar: use \"$GIT_UNZIP\" in 'scalar diagnose' test\n + scalar-diagnose: use \"$GIT_UNZIP\" in test\n \n Use the \"$GIT_UNZIP\" test variable rather than verbatim 'unzip' to unzip the\n 'scalar diagnose' archive. Using \"$GIT_UNZIP\" is needed to run the Scalar\n 3: e8abfdfa892 ! 2: 7956dc24b30 builtin/bugreport.c: avoid size_t overflow\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - builtin/bugreport.c: avoid size_t overflow\n + scalar-diagnose: avoid 32-bit overflow of size_t\n \n - Avoid size_t overflow when reporting the available disk space in\n + Avoid 32-bit size_t overflow when reporting the available disk space in\n 'get_disk_info' by casting the block size and available block count to\n - 'uint64_t' before multiplying them. Without this change, 'st_mult' would\n - (correctly) report size_t overflow on 32-bit systems at or exceeding 2^32\n + 'off_t' before multiplying them. Without this change, 'st_mult' would\n + (correctly) report a size_t overflow on 32-bit systems at or exceeding 2^32\n bytes of available space.\n \n + Note that 'off_t' is a 64-bit integer even on 32-bit systems due to the\n + inclusion of '#define _FILE_OFFSET_BITS 64' in 'git-compat-util.h' (see\n + b97e911643 (Support for large files on 32bit systems., 2007-02-17)).\n +\n + Helped-by: Junio C Hamano \n Signed-off-by: Victoria Dye \n \n - ## builtin/bugreport.c ##\n -@@ builtin/bugreport.c: static int get_disk_info(struct strbuf *out)\n + ## contrib/scalar/scalar.c ##\n +@@ contrib/scalar/scalar.c: static int get_disk_info(struct strbuf *out)\n \t}\n \n \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n -\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n -+\tstrbuf_humanise_bytes(out, (uint64_t)stat.f_bsize * (uint64_t)stat.f_bavail);\n ++\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n \tstrbuf_release(&buf);\n #endif\n 4: 4bc290fbf43 ! 3: 23349bfaf8f builtin/bugreport.c: add directory to archiver more gently\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - builtin/bugreport.c: add directory to archiver more gently\n + scalar-diagnose: add directory to archiver more gently\n \n - If a directory added to the '--diagnose' archiver does not exist, warn and\n - return 0 from 'add_directory_to_archiver()' rather than failing with a fatal\n - error. This handles a failure edge case where the '.git/logs' has not yet\n - been created when running 'git bugreport --diagnose', but extends to any\n + If a directory added to the 'scalar diagnose' archiver does not exist, warn\n + and return 0 from 'add_directory_to_archiver()' rather than failing with a\n + fatal error. This handles a failure edge case where the '.git/logs' has not\n + yet been created when running 'scalar diagnose', but extends to any\n situation where a directory may be missing in the '.git' dir.\n \n Now, when a directory is missing a warning is captured in the diagnostic\n - logs. This provides a user with more complete information than if 'git\n - bugreport' simply failed with an error.\n + logs. This provides a user with more complete information than if 'scalar\n + diagnose' simply failed with an error.\n \n + Helped-by: Junio C Hamano \n Signed-off-by: Victoria Dye \n \n - ## builtin/bugreport.c ##\n -@@ builtin/bugreport.c: static int add_directory_to_archiver(struct strvec *archiver_args,\n - \t\t\t\t const char *path, int recurse)\n + ## contrib/scalar/scalar.c ##\n +@@ contrib/scalar/scalar.c: static int add_directory_to_archiver(struct strvec *archiver_args,\n + \t\t\t\t\t const char *path, int recurse)\n {\n \tint at_root = !*path;\n -\tDIR *dir = opendir(at_root ? \".\" : path);\n @@ builtin/bugreport.c: static int add_directory_to_archiver(struct strvec *archive\n \tsize_t len;\n \tint res = 0;\n \n -+\tif (!file_exists(at_root ? \".\" : path)) {\n -+\t\twarning(_(\"directory '%s' does not exist, will not be archived\"), path);\n -+\t\treturn 0;\n -+\t}\n -+\n +-\tif (!dir)\n +\tdir = opendir(at_root ? \".\" : path);\n - \tif (!dir)\n ++\tif (!dir) {\n ++\t\tif (errno == ENOENT) {\n ++\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n ++\t\t\treturn 0;\n ++\t\t}\n \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n ++\t}\n \n -\n - ## t/t0091-bugreport.sh ##\n -@@ t/t0091-bugreport.sh: test_expect_success 'indicates populated hooks' '\n - \ttest_cmp expect actual\n - '\n - \n --test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n -+test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n - \ttest_when_finished rm -rf report &&\n - \n - \tgit bugreport --diagnose -o report -s test >out &&\n -@@ t/t0091-bugreport.sh: test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n - \tgrep \"^Total: [0-9][0-9]*\" out\n - '\n - \n -+test_expect_success '--diagnose warns when archived dir does not exist' '\n -+\ttest_when_finished rm -rf report &&\n -+\n -+\t# Remove logs - not guaranteed to exist\n -+\trm -rf .git/logs &&\n -+\tgit bugreport --diagnose -o report -s test 2>err &&\n -+\tgrep \"directory .\\.git/logs. does not exist, will not be archived\" err\n -+'\n -+\n - test_done\n + \tif (!at_root)\n + \t\tstrbuf_addf(&buf, \"%s/\", path);\n -: ----------- > 4: 05bba1e699f scalar-diagnose: move 'get_disk_info()' to 'compat/'\n 6: 4eb3c43d488 ! 5: 3a0cb33c658 scalar: use 'git bugreport --diagnose' in 'scalar diagnose'\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - scalar: use 'git bugreport --diagnose' in 'scalar diagnose'\n + scalar-diagnose: move functionality to common location\n \n - Replace implementation of 'scalar diagnose' with an internal invocation of\n - 'git bugreport --diagnose --no-report'. The '--diagnose' option of 'git\n - bugreport' was implemented to mirror what 'scalar diagnose' does, taking\n - most of its code directly from 'scalar.c'. Remove the now-duplicate code in\n - 'scalar.c' and have 'scalar diagnose' call 'git bugreport' to create the\n - diagnostics archive.\n + Move the core functionality of 'scalar diagnose' into a new 'diagnose.[c,h]'\n + library to prepare for new callers in the main Git tree generating\n + diagnostic archives. These callers will be introduced in subsequent patches.\n \n - This introduces two (minor) changes to the output of 'scalar diagnose':\n - changing \"Enlistment root\" to \"Repository root\" in 'diagnostics.log'\n - (\"enlistment root\" was inaccurate anyway, as the reported path always\n - pointed to the root of the repository), and changing the prefix of the zip\n - archive from 'scalar_' to 'git-diagnostics-'.\n + While this patch appears large, it is mostly made up of moving code out of\n + 'scalar.c' and into 'diagnose.c'. Specifically, the functions\n +\n + - dir_file_stats_objects()\n + - dir_file_stats()\n + - count_files()\n + - loose_objs_stats()\n + - add_directory_to_archiver()\n +\n + are all copied verbatim from 'scalar.c'. The 'create_diagnostics_archive()'\n + function is a mostly identical (partial) copy of 'cmd_diagnose()', with the\n + primary changes being that 'zip_path' is an input and \"Enlistment root\" is\n + corrected to \"Repository root\" in the archiver log.\n \n Signed-off-by: Victoria Dye \n \n + ## Makefile ##\n +@@ Makefile: LIB_OBJS += ctype.o\n + LIB_OBJS += date.o\n + LIB_OBJS += decorate.o\n + LIB_OBJS += delta-islands.o\n ++LIB_OBJS += diagnose.o\n + LIB_OBJS += diff-delta.o\n + LIB_OBJS += diff-merges.o\n + LIB_OBJS += diff-lib.o\n +\n ## contrib/scalar/scalar.c ##\n @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n -#include \"archive.h\"\n - #include \"object-store.h\"\n +-#include \"object-store.h\"\n +-#include \"compat/disk.h\"\n ++#include \"diagnose.h\"\n \n /*\n + * Remove the deepest subdirectory in the provided path string. Path must not\n @@ contrib/scalar/scalar.c: static int unregister_dir(void)\n \treturn res;\n }\n @@ contrib/scalar/scalar.c: static int unregister_dir(void)\n -\t\t\t\t\t const char *path, int recurse)\n -{\n -\tint at_root = !*path;\n --\tDIR *dir = opendir(at_root ? \".\" : path);\n +-\tDIR *dir;\n -\tstruct dirent *e;\n -\tstruct strbuf buf = STRBUF_INIT;\n -\tsize_t len;\n -\tint res = 0;\n -\n --\tif (!dir)\n +-\tdir = opendir(at_root ? \".\" : path);\n +-\tif (!dir) {\n +-\t\tif (errno == ENOENT) {\n +-\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n +-\t\t\treturn 0;\n +-\t\t}\n -\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n +-\t}\n -\n -\tif (!at_root)\n -\t\tstrbuf_addf(&buf, \"%s/\", path);\n @@ contrib/scalar/scalar.c: static int unregister_dir(void)\n -\tstrbuf_release(&buf);\n -\treturn res;\n -}\n --\n --#ifndef WIN32\n --#include \n --#endif\n --\n --static int get_disk_info(struct strbuf *out)\n --{\n --#ifdef WIN32\n --\tstruct strbuf buf = STRBUF_INIT;\n --\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n --\tDWORD serial_number, component_length, flags;\n --\tULARGE_INTEGER avail2caller, total, avail;\n --\n --\tstrbuf_realpath(&buf, \".\", 1);\n --\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n --\t\terror(_(\"could not determine free disk size for '%s'\"),\n --\t\t buf.buf);\n --\t\tstrbuf_release(&buf);\n --\t\treturn -1;\n --\t}\n --\n --\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n --\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n --\t\t\t\t &serial_number, &component_length, &flags,\n --\t\t\t\t fs_name, sizeof(fs_name))) {\n --\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n --\t\tstrbuf_release(&buf);\n --\t\treturn -1;\n --\t}\n --\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n --\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n --\tstrbuf_addch(out, '\\n');\n --\tstrbuf_release(&buf);\n --#else\n --\tstruct strbuf buf = STRBUF_INIT;\n --\tstruct statvfs stat;\n --\n --\tstrbuf_realpath(&buf, \".\", 1);\n --\tif (statvfs(buf.buf, &stat) < 0) {\n --\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n --\t\t\t buf.buf);\n --\t\tstrbuf_release(&buf);\n --\t\treturn -1;\n --\t}\n --\n --\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n --\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n --\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n --\tstrbuf_release(&buf);\n --#endif\n --\treturn 0;\n --}\n -\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n @@ contrib/scalar/scalar.c: cleanup:\n {\n \tstruct option options[] = {\n @@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n - \t\tN_(\"scalar diagnose []\"),\n \t\tNULL\n \t};\n --\tstruct strbuf zip_path = STRBUF_INIT;\n + \tstruct strbuf zip_path = STRBUF_INIT;\n -\tstruct strvec archiver_args = STRVEC_INIT;\n -\tchar **argv_copy = NULL;\n -\tint stdout_fd = -1, archiver_fd = -1;\n --\ttime_t now = time(NULL);\n --\tstruct tm tm;\n + \ttime_t now = time(NULL);\n + \tstruct tm tm;\n -\tstruct strbuf buf = STRBUF_INIT;\n -+\tstruct strbuf diagnostics_path = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n - \t\t\t usage, 0);\n - \n --\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n --\n --\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n --\tstrbuf_addftime(&zip_path,\n --\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n --\tstrbuf_addstr(&zip_path, \".zip\");\n --\tswitch (safe_create_leading_directories(zip_path.buf)) {\n --\tcase SCLD_EXISTS:\n --\tcase SCLD_OK:\n --\t\tbreak;\n --\tdefault:\n --\t\terror_errno(_(\"could not create directory for '%s'\"),\n --\t\t\t zip_path.buf);\n --\t\tgoto diagnose_cleanup;\n --\t}\n +@@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n + \t\t\t zip_path.buf);\n + \t\tgoto diagnose_cleanup;\n + \t}\n -\tstdout_fd = dup(1);\n -\tif (stdout_fd < 0) {\n -\t\tres = error_errno(_(\"could not duplicate stdout\"));\n @@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n -\tstrvec_pushf(&archiver_args,\n -\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n -\t\t (int)buf.len, buf.buf);\n -+\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_path);\n -+\tstrbuf_addstr(&diagnostics_path, \"/.scalarDiagnostics\");\n - \n +-\n -\tstrbuf_reset(&buf);\n -\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n -\tdir_file_stats(the_repository->objects->odb, &buf);\n @@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n -\t\terror(_(\"failed to write archive\"));\n -\t\tgoto diagnose_cleanup;\n -\t}\n --\n + \n -\tif (!res)\n -\t\tfprintf(stderr, \"\\n\"\n -\t\t \"Diagnostics complete.\\n\"\n -\t\t \"All of the gathered info is captured in '%s'\\n\",\n -\t\t zip_path.buf);\n --\n --diagnose_cleanup:\n ++\tres = create_diagnostics_archive(&zip_path);\n + \n + diagnose_cleanup:\n -\tif (archiver_fd >= 0) {\n -\t\tclose(1);\n -\t\tdup2(stdout_fd, 1);\n -\t}\n -\tfree(argv_copy);\n -\tstrvec_clear(&archiver_args);\n --\tstrbuf_release(&zip_path);\n + \tstrbuf_release(&zip_path);\n -\tstrbuf_release(&buf);\n -+\tif (run_git(\"bugreport\", \"--diagnose\", \"--no-report\",\n -+\t\t \"-s\", \"%Y%m%d_%H%M%S\", \"-o\", diagnostics_path.buf, NULL) < 0)\n -+\t\tres = -1;\n - \n -+\tstrbuf_release(&diagnostics_path);\n +-\n \treturn res;\n }\n \n +\n + ## diagnose.c (new) ##\n +@@\n ++#include \"diagnose.h\"\n ++#include \"compat/disk.h\"\n ++#include \"archive.h\"\n ++#include \"dir.h\"\n ++#include \"help.h\"\n ++#include \"strvec.h\"\n ++#include \"object-store.h\"\n ++#include \"packfile.h\"\n ++\n ++static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n ++\t\t\t\t const char *file_name, void *data)\n ++{\n ++\tstruct strbuf *buf = data;\n ++\tstruct stat st;\n ++\n ++\tif (!stat(full_path, &st))\n ++\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n ++\t\t\t (uintmax_t)st.st_size);\n ++}\n ++\n ++static int dir_file_stats(struct object_directory *object_dir, void *data)\n ++{\n ++\tstruct strbuf *buf = data;\n ++\n ++\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n ++\n ++\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n ++\t\t\t\t data);\n ++\n ++\treturn 0;\n ++}\n ++\n ++static int count_files(char *path)\n ++{\n ++\tDIR *dir = opendir(path);\n ++\tstruct dirent *e;\n ++\tint count = 0;\n ++\n ++\tif (!dir)\n ++\t\treturn 0;\n ++\n ++\twhile ((e = readdir(dir)) != NULL)\n ++\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n ++\t\t\tcount++;\n ++\n ++\tclosedir(dir);\n ++\treturn count;\n ++}\n ++\n ++static void loose_objs_stats(struct strbuf *buf, const char *path)\n ++{\n ++\tDIR *dir = opendir(path);\n ++\tstruct dirent *e;\n ++\tint count;\n ++\tint total = 0;\n ++\tunsigned char c;\n ++\tstruct strbuf count_path = STRBUF_INIT;\n ++\tsize_t base_path_len;\n ++\n ++\tif (!dir)\n ++\t\treturn;\n ++\n ++\tstrbuf_addstr(buf, \"Object directory stats for \");\n ++\tstrbuf_add_absolute_path(buf, path);\n ++\tstrbuf_addstr(buf, \":\\n\");\n ++\n ++\tstrbuf_add_absolute_path(&count_path, path);\n ++\tstrbuf_addch(&count_path, '/');\n ++\tbase_path_len = count_path.len;\n ++\n ++\twhile ((e = readdir(dir)) != NULL)\n ++\t\tif (!is_dot_or_dotdot(e->d_name) &&\n ++\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n ++\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n ++\t\t\tstrbuf_setlen(&count_path, base_path_len);\n ++\t\t\tstrbuf_addstr(&count_path, e->d_name);\n ++\t\t\ttotal += (count = count_files(count_path.buf));\n ++\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n ++\t\t}\n ++\n ++\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n ++\n ++\tstrbuf_release(&count_path);\n ++\tclosedir(dir);\n ++}\n ++\n ++static int add_directory_to_archiver(struct strvec *archiver_args,\n ++\t\t\t\t const char *path, int recurse)\n ++{\n ++\tint at_root = !*path;\n ++\tDIR *dir;\n ++\tstruct dirent *e;\n ++\tstruct strbuf buf = STRBUF_INIT;\n ++\tsize_t len;\n ++\tint res = 0;\n ++\n ++\tdir = opendir(at_root ? \".\" : path);\n ++\tif (!dir) {\n ++\t\tif (errno == ENOENT) {\n ++\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n ++\t\t\treturn 0;\n ++\t\t}\n ++\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n ++\t}\n ++\n ++\tif (!at_root)\n ++\t\tstrbuf_addf(&buf, \"%s/\", path);\n ++\tlen = buf.len;\n ++\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n ++\n ++\twhile (!res && (e = readdir(dir))) {\n ++\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n ++\t\t\tcontinue;\n ++\n ++\t\tstrbuf_setlen(&buf, len);\n ++\t\tstrbuf_addstr(&buf, e->d_name);\n ++\n ++\t\tif (e->d_type == DT_REG)\n ++\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n ++\t\telse if (e->d_type != DT_DIR)\n ++\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n ++\t\t\t\t \"directory\"), buf.buf);\n ++\t\telse if (recurse &&\n ++\t\t\t add_directory_to_archiver(archiver_args,\n ++\t\t\t\t\t\t buf.buf, recurse) < 0)\n ++\t\t\tres = -1;\n ++\t}\n ++\n ++\tclosedir(dir);\n ++\tstrbuf_release(&buf);\n ++\treturn res;\n ++}\n ++\n ++int create_diagnostics_archive(struct strbuf *zip_path)\n ++{\n ++\tstruct strvec archiver_args = STRVEC_INIT;\n ++\tchar **argv_copy = NULL;\n ++\tint stdout_fd = -1, archiver_fd = -1;\n ++\tstruct strbuf buf = STRBUF_INIT;\n ++\tint res;\n ++\n ++\tstdout_fd = dup(STDOUT_FILENO);\n ++\tif (stdout_fd < 0) {\n ++\t\tres = error_errno(_(\"could not duplicate stdout\"));\n ++\t\tgoto diagnose_cleanup;\n ++\t}\n ++\n ++\tarchiver_fd = xopen(zip_path->buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n ++\tif (dup2(archiver_fd, STDOUT_FILENO) < 0) {\n ++\t\tres = error_errno(_(\"could not redirect output\"));\n ++\t\tgoto diagnose_cleanup;\n ++\t}\n ++\n ++\tinit_zip_archiver();\n ++\tstrvec_pushl(&archiver_args, \"git-diagnose\", \"--format=zip\", NULL);\n ++\n ++\tstrbuf_reset(&buf);\n ++\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n ++\tget_version_info(&buf, 1);\n ++\n ++\tstrbuf_addf(&buf, \"Repository root: %s\\n\", the_repository->worktree);\n ++\tget_disk_info(&buf);\n ++\twrite_or_die(stdout_fd, buf.buf, buf.len);\n ++\tstrvec_pushf(&archiver_args,\n ++\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n ++\t\t (int)buf.len, buf.buf);\n ++\n ++\tstrbuf_reset(&buf);\n ++\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n ++\tdir_file_stats(the_repository->objects->odb, &buf);\n ++\tforeach_alt_odb(dir_file_stats, &buf);\n ++\tstrvec_push(&archiver_args, buf.buf);\n ++\n ++\tstrbuf_reset(&buf);\n ++\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n ++\tloose_objs_stats(&buf, \".git/objects\");\n ++\tstrvec_push(&archiver_args, buf.buf);\n ++\n ++\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n ++\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n ++\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n ++\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n ++\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n ++\t\tgoto diagnose_cleanup;\n ++\n ++\tstrvec_pushl(&archiver_args, \"--prefix=\",\n ++\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n ++\n ++\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n ++\targv_copy = xmemdupz(archiver_args.v,\n ++\t\t\t sizeof(char *) * archiver_args.nr);\n ++\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n ++\t\t\t the_repository, NULL, 0);\n ++\tif (res) {\n ++\t\terror(_(\"failed to write archive\"));\n ++\t\tgoto diagnose_cleanup;\n ++\t}\n ++\n ++\tif (!res)\n ++\t\tfprintf(stderr, \"\\n\"\n ++\t\t\t\"Diagnostics complete.\\n\"\n ++\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n ++\t\t\tzip_path->buf);\n ++\n ++diagnose_cleanup:\n ++\tif (archiver_fd >= 0) {\n ++\t\tdup2(stdout_fd, STDOUT_FILENO);\n ++\t\tclose(stdout_fd);\n ++\t\tclose(archiver_fd);\n ++\t}\n ++\tfree(argv_copy);\n ++\tstrvec_clear(&archiver_args);\n ++\tstrbuf_release(&buf);\n ++\n ++\treturn res;\n ++}\n +\n + ## diagnose.h (new) ##\n +@@\n ++#ifndef DIAGNOSE_H\n ++#define DIAGNOSE_H\n ++\n ++#include \"cache.h\"\n ++#include \"strbuf.h\"\n ++\n ++int create_diagnostics_archive(struct strbuf *zip_path);\n ++\n ++#endif /* DIAGNOSE_H */\n -: ----------- > 6: 73e139ee377 builtin/diagnose.c: create 'git diagnose' builtin\n -: ----------- > 7: a3e62a4a041 builtin/diagnose.c: gate certain data behind '--all'\n 2: 932dc8cddac ! 8: d81e7c10997 builtin/bugreport.c: create '--diagnose' option\n @@ Commit message\n Create a '--diagnose' option for 'git bugreport' to collect additional\n information about the repository and write it to a zipped archive.\n \n - The \"diagnose\" functionality was originally implemented for Scalar in\n - aa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\n - diagnostics gathered are not specific to Scalar-cloned repositories and\n - could be useful when diagnosing issues in any Git repository.\n + The '--diagnose' option behaves effectively as an alias for simultaneously\n + running 'git bugreport' and 'git diagnose'. In the documentation, users are\n + explicitly recommended to attach the diagnostics alongside a bug report to\n + provide additional context to readers, ideally reducing some back-and-forth\n + between reporters and those debugging the issue.\n \n - Note that, while this patch appears large, it is mostly copied directly out\n - of 'scalar.c'. Specifically, the functions\n -\n - - dir_file_stats_objects()\n - - dir_file_stats()\n - - count_files()\n - - loose_objs_stats()\n - - add_directory_to_archiver()\n - - get_disk_info()\n -\n - are all copied verbatim from 'scalar.c'. The 'create_diagnostics_archive()'\n - function is a mostly unmodified copy of 'cmd_diagnose()', with the primary\n - changes being that 'zip_path' is an input and \"Enlistment root\" is corrected\n - to \"Repository root\" in the logs.\n -\n - The remainder of the patch is made up of adding the '--diagnose' option to\n - 'cmd_bugreport()' (including generation of the archive's 'zip_path'),\n - updating documentation, and adding a test. Note that the test is\n - 'test_expect_failure' due to bugs in the original 'scalar diagnose'. These\n - will be fixed in subsequent patches.\n + Note that '--diagnose' may take an optional string arg (either 'basic' or\n + 'all'). If specified without the arg or with 'basic', the behavior\n + corresponds to running 'git diagnose' without '--all'; this default is meant\n + to help reduce unintentional leaking of sensitive information). However, a\n + user can still manually specify '--diagnose=all' to generate the equivalent\n + archive to one created with 'git diagnose --all'.\n \n Suggested-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Victoria Dye \n \n ## Documentation/git-bugreport.txt ##\n -@@ Documentation/git-bugreport.txt: git-bugreport - Collect information for user to file a bug report\n - SYNOPSIS\n +@@ Documentation/git-bugreport.txt: SYNOPSIS\n --------\n [verse]\n --'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n -+'git bugreport' []\n + 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n ++\t\t[--diagnose[=(basic|all)]]\n \n DESCRIPTION\n -----------\n @@ Documentation/git-bugreport.txt: The following information is captured automatic\n - $SHELL\n \n +Additional information may be gathered into a separate zip archive using the\n -+`--diagnose` option.\n ++`--diagnose` option, and can be attached alongside the bugreport document to\n ++provide additional context to readers.\n +\n This tool is invoked via the typical Git setup process, which means that in some\n cases, it might not be able to launch - for example, if a relevant config file\n @@ Documentation/git-bugreport.txt: OPTIONS\n \tnamed 'git-bugreport-'. This should take the form of a\n \tstrftime(3) format string; the current local time will be used.\n \n -+--diagnose::\n ++--diagnose[=(basic|all)]::\n +\tCreate a zip archive of information about the repository including logs\n +\tand certain statistics describing the data shape of the repository. The\n +\tarchive is written to the same output directory as the bug report and is\n +\tnamed 'git-diagnostics-'.\n +++\n ++By default, `--diagnose` (equivalent to `--diagnose=basic`) will collect only\n ++statistics and summarized data about the repository and filesystem. Specifying\n ++`--diagnose=all` will create an archive with the same contents generated by `git\n ++diagnose --all`; this archive will be much larger, and will contain potentially\n ++sensitive information about the repository. See linkgit:git-diagnose[1] for more\n ++details on the contents of the diagnostic archive.\n +\n GIT\n ---\n @@ builtin/bugreport.c\n #include \"compat/compiler.h\"\n #include \"hook.h\"\n #include \"hook-list.h\"\n -+#include \"dir.h\"\n -+#include \"object-store.h\"\n -+#include \"packfile.h\"\n -+#include \"archive.h\"\n ++#include \"diagnose.h\"\n \n ++enum diagnose_mode {\n ++\tDIAGNOSE_NONE,\n ++\tDIAGNOSE_BASIC,\n ++\tDIAGNOSE_ALL\n ++};\n \n static void get_system_info(struct strbuf *sys_info)\n -@@ builtin/bugreport.c: static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n - }\n - \n - static const char * const bugreport_usage[] = {\n --\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n -+\tN_(\"git bugreport []\"),\n - \tNULL\n - };\n - \n + {\n @@ builtin/bugreport.c: static void get_header(struct strbuf *buf, const char *title)\n \tstrbuf_addf(buf, \"\\n\\n[%s]\\n\", title);\n }\n \n -+static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n -+\t\t\t\t const char *file_name, void *data)\n -+{\n -+\tstruct strbuf *buf = data;\n -+\tstruct stat st;\n -+\n -+\tif (!stat(full_path, &st))\n -+\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n -+\t\t\t (uintmax_t)st.st_size);\n -+}\n -+\n -+static int dir_file_stats(struct object_directory *object_dir, void *data)\n -+{\n -+\tstruct strbuf *buf = data;\n -+\n -+\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n -+\n -+\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n -+\t\t\t\t data);\n -+\n -+\treturn 0;\n -+}\n -+\n -+static int count_files(char *path)\n -+{\n -+\tDIR *dir = opendir(path);\n -+\tstruct dirent *e;\n -+\tint count = 0;\n -+\n -+\tif (!dir)\n -+\t\treturn 0;\n -+\n -+\twhile ((e = readdir(dir)) != NULL)\n -+\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n -+\t\t\tcount++;\n -+\n -+\tclosedir(dir);\n -+\treturn count;\n -+}\n -+\n -+static void loose_objs_stats(struct strbuf *buf, const char *path)\n -+{\n -+\tDIR *dir = opendir(path);\n -+\tstruct dirent *e;\n -+\tint count;\n -+\tint total = 0;\n -+\tunsigned char c;\n -+\tstruct strbuf count_path = STRBUF_INIT;\n -+\tsize_t base_path_len;\n -+\n -+\tif (!dir)\n -+\t\treturn;\n -+\n -+\tstrbuf_addstr(buf, \"Object directory stats for \");\n -+\tstrbuf_add_absolute_path(buf, path);\n -+\tstrbuf_addstr(buf, \":\\n\");\n -+\n -+\tstrbuf_add_absolute_path(&count_path, path);\n -+\tstrbuf_addch(&count_path, '/');\n -+\tbase_path_len = count_path.len;\n -+\n -+\twhile ((e = readdir(dir)) != NULL)\n -+\t\tif (!is_dot_or_dotdot(e->d_name) &&\n -+\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n -+\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n -+\t\t\tstrbuf_setlen(&count_path, base_path_len);\n -+\t\t\tstrbuf_addstr(&count_path, e->d_name);\n -+\t\t\ttotal += (count = count_files(count_path.buf));\n -+\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n -+\t\t}\n -+\n -+\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n -+\n -+\tstrbuf_release(&count_path);\n -+\tclosedir(dir);\n -+}\n -+\n -+static int add_directory_to_archiver(struct strvec *archiver_args,\n -+\t\t\t\t const char *path, int recurse)\n -+{\n -+\tint at_root = !*path;\n -+\tDIR *dir = opendir(at_root ? \".\" : path);\n -+\tstruct dirent *e;\n -+\tstruct strbuf buf = STRBUF_INIT;\n -+\tsize_t len;\n -+\tint res = 0;\n -+\n -+\tif (!dir)\n -+\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n -+\n -+\tif (!at_root)\n -+\t\tstrbuf_addf(&buf, \"%s/\", path);\n -+\tlen = buf.len;\n -+\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n -+\n -+\twhile (!res && (e = readdir(dir))) {\n -+\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n -+\t\t\tcontinue;\n -+\n -+\t\tstrbuf_setlen(&buf, len);\n -+\t\tstrbuf_addstr(&buf, e->d_name);\n -+\n -+\t\tif (e->d_type == DT_REG)\n -+\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n -+\t\telse if (e->d_type != DT_DIR)\n -+\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n -+\t\t\t\t \"directory\"), buf.buf);\n -+\t\telse if (recurse &&\n -+\t\t\t add_directory_to_archiver(archiver_args,\n -+\t\t\t\t\t\t buf.buf, recurse) < 0)\n -+\t\t\tres = -1;\n -+\t}\n -+\n -+\tclosedir(dir);\n -+\tstrbuf_release(&buf);\n -+\treturn res;\n -+}\n -+\n -+#ifndef WIN32\n -+#include \n -+#endif\n -+\n -+static int get_disk_info(struct strbuf *out)\n ++static int option_parse_diagnose(const struct option *opt,\n ++\t\t\t\t const char *arg, int unset)\n +{\n -+#ifdef WIN32\n -+\tstruct strbuf buf = STRBUF_INIT;\n -+\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n -+\tDWORD serial_number, component_length, flags;\n -+\tULARGE_INTEGER avail2caller, total, avail;\n -+\n -+\tstrbuf_realpath(&buf, \".\", 1);\n -+\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n -+\t\terror(_(\"could not determine free disk size for '%s'\"),\n -+\t\t buf.buf);\n -+\t\tstrbuf_release(&buf);\n -+\t\treturn -1;\n -+\t}\n ++\tenum diagnose_mode *diagnose = opt->value;\n +\n -+\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n -+\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n -+\t\t\t\t &serial_number, &component_length, &flags,\n -+\t\t\t\t fs_name, sizeof(fs_name))) {\n -+\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n -+\t\tstrbuf_release(&buf);\n -+\t\treturn -1;\n -+\t}\n -+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n -+\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n -+\tstrbuf_addch(out, '\\n');\n -+\tstrbuf_release(&buf);\n -+#else\n -+\tstruct strbuf buf = STRBUF_INIT;\n -+\tstruct statvfs stat;\n ++\tBUG_ON_OPT_NEG(unset);\n +\n -+\tstrbuf_realpath(&buf, \".\", 1);\n -+\tif (statvfs(buf.buf, &stat) < 0) {\n -+\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n -+\t\t\t buf.buf);\n -+\t\tstrbuf_release(&buf);\n -+\t\treturn -1;\n -+\t}\n ++\tif (!arg || !strcmp(arg, \"basic\"))\n ++\t\t*diagnose = DIAGNOSE_BASIC;\n ++\telse if (!strcmp(arg, \"all\"))\n ++\t\t*diagnose = DIAGNOSE_ALL;\n ++\telse\n ++\t\tdie(_(\"diagnose mode must be either 'basic' or 'all'\"));\n +\n -+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n -+\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n -+\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n -+\tstrbuf_release(&buf);\n -+#endif\n +\treturn 0;\n +}\n -+\n -+static int create_diagnostics_archive(struct strbuf *zip_path)\n -+{\n -+\tstruct strvec archiver_args = STRVEC_INIT;\n -+\tchar **argv_copy = NULL;\n -+\tint stdout_fd = -1, archiver_fd = -1;\n -+\tstruct strbuf buf = STRBUF_INIT;\n -+\tint res = 0;\n -+\n -+\tstdout_fd = dup(1);\n -+\tif (stdout_fd < 0) {\n -+\t\tres = error_errno(_(\"could not duplicate stdout\"));\n -+\t\tgoto diagnose_cleanup;\n -+\t}\n -+\n -+\tarchiver_fd = xopen(zip_path->buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n -+\tif (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) {\n -+\t\tres = error_errno(_(\"could not redirect output\"));\n -+\t\tgoto diagnose_cleanup;\n -+\t}\n -+\n -+\tinit_zip_archiver();\n -+\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n -+\n -+\tstrbuf_reset(&buf);\n -+\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n -+\tget_version_info(&buf, 1);\n -+\n -+\tstrbuf_addf(&buf, \"Repository root: %s\\n\", the_repository->worktree);\n -+\tget_disk_info(&buf);\n -+\twrite_or_die(stdout_fd, buf.buf, buf.len);\n -+\tstrvec_pushf(&archiver_args,\n -+\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n -+\t\t (int)buf.len, buf.buf);\n -+\n -+\tstrbuf_reset(&buf);\n -+\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n -+\tdir_file_stats(the_repository->objects->odb, &buf);\n -+\tforeach_alt_odb(dir_file_stats, &buf);\n -+\tstrvec_push(&archiver_args, buf.buf);\n -+\n -+\tstrbuf_reset(&buf);\n -+\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n -+\tloose_objs_stats(&buf, \".git/objects\");\n -+\tstrvec_push(&archiver_args, buf.buf);\n -+\n -+\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n -+\t\tgoto diagnose_cleanup;\n -+\n -+\tstrvec_pushl(&archiver_args, \"--prefix=\",\n -+\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n -+\n -+\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n -+\targv_copy = xmemdupz(archiver_args.v,\n -+\t\t\t sizeof(char *) * archiver_args.nr);\n -+\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n -+\t\t\t the_repository, NULL, 0);\n -+\tif (res) {\n -+\t\terror(_(\"failed to write archive\"));\n -+\t\tgoto diagnose_cleanup;\n -+\t}\n -+\n -+\tif (!res)\n -+\t\tfprintf(stderr, \"\\n\"\n -+\t\t\t\"Diagnostics complete.\\n\"\n -+\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n -+\t\t\tzip_path->buf);\n -+\n -+diagnose_cleanup:\n -+\tif (archiver_fd >= 0) {\n -+\t\tclose(1);\n -+\t\tdup2(stdout_fd, 1);\n -+\t}\n -+\tfree(argv_copy);\n -+\tstrvec_clear(&archiver_args);\n -+\tstrbuf_release(&buf);\n -+\n -+\treturn res;\n -+}\n +\n int cmd_bugreport(int argc, const char **argv, const char *prefix)\n {\n @@ builtin/bugreport.c: int cmd_bugreport(int argc, const char **argv, const char *\n \tint report = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n -+\tint diagnose = 0;\n ++\tenum diagnose_mode diagnose = DIAGNOSE_NONE;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tconst char *user_relative_path = NULL;\n @@ builtin/bugreport.c: int cmd_bugreport(int argc, const char **argv, const char *\n +\tsize_t output_path_len;\n \n \tconst struct option bugreport_options[] = {\n -+\t\tOPT_BOOL(0, \"diagnose\", &diagnose,\n -+\t\t\t N_(\"generate a diagnostics zip archive\")),\n ++\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"(basic|all)\"),\n ++\t\t\t N_(\"create an additional zip archive of detailed diagnostics\"),\n ++\t\t\t PARSE_OPT_NONEG | PARSE_OPT_OPTARG, option_parse_diagnose),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n -\t\t\t N_(\"specify a destination for the bugreport file\")),\n +\t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n @@ builtin/bugreport.c: int cmd_bugreport(int argc, const char **argv, const char *\n \t}\n \n +\t/* Prepare diagnostics, if requested */\n -+\tif (diagnose) {\n ++\tif (diagnose != DIAGNOSE_NONE) {\n +\t\tstruct strbuf zip_path = STRBUF_INIT;\n +\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n +\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n +\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n +\t\tstrbuf_addstr(&zip_path, \".zip\");\n +\n -+\t\tif (create_diagnostics_archive(&zip_path))\n ++\t\tif (create_diagnostics_archive(&zip_path, diagnose == DIAGNOSE_ALL))\n +\t\t\tdie_errno(_(\"unable to create diagnostics archive %s\"), zip_path.buf);\n +\n +\t\tstrbuf_release(&zip_path);\n @@ t/t0091-bugreport.sh: test_expect_success 'indicates populated hooks' '\n \ttest_cmp expect actual\n '\n \n -+test_expect_failure UNZIP '--diagnose creates diagnostics zip archive' '\n ++test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n +\ttest_when_finished rm -rf report &&\n +\n +\tgit bugreport --diagnose -o report -s test >out &&\n @@ t/t0091-bugreport.sh: test_expect_success 'indicates populated hooks' '\n +\tgrep \".git/objects\" out &&\n +\n +\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n -+\tgrep \"^Total: [0-9][0-9]*\" out\n ++\tgrep \"^Total: [0-9][0-9]*\" out &&\n ++\n ++\t# Should not include .git directory contents by default\n ++\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n ++'\n ++\n ++test_expect_success UNZIP '--diagnose=basic excludes .git dir contents' '\n ++\ttest_when_finished rm -rf report &&\n ++\n ++\tgit bugreport --diagnose=basic -o report -s test >out &&\n ++\n ++\t# Should not include .git directory contents\n ++\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n ++'\n ++\n ++test_expect_success UNZIP '--diagnose=all includes .git dir contents' '\n ++\ttest_when_finished rm -rf report &&\n ++\n ++\tgit bugreport --diagnose=all -o report -s test >out &&\n ++\n ++\t# Should include .git directory contents\n ++\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n ++\n ++\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n ++\ttest_file_not_empty out\n +'\n +\n test_done\n 5: d6527049a4f < -: ----------- builtin/bugreport.c: add '--no-report' option\n -: ----------- > 9: 6834bdcaea8 scalar-diagnose: use 'git diagnose --all'\n 7: 86d40a4bd15 ! 10: 14925c3feed scalar: update technical doc roadmap\n @@ Commit message\n scalar: update technical doc roadmap\n \n Update the Scalar roadmap to reflect the completion of generalizing 'scalar\n - diagnose' into 'git bugreport --diagnose'.\n + diagnose' into 'git diagnose' and 'git bugreport --diagnose'.\n \n Signed-off-by: Victoria Dye \n \n @@ Documentation/technical/scalar.txt: series have been accepted:\n - `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n \n +- `scalar-generalize-diagnose`: Move the functionality of `scalar diagnose`\n -+ into `git bugreport --diagnose`.\n ++ into `git diagnose` and `git bugreport --diagnose`.\n +\n Roughly speaking (and subject to change), the following series are needed to\n \"finish\" this initial version of Scalar:\n\n-- \ngitgitgadget\n"},{"id":"460588","messageId":"7956dc24b30247ee2e401fca61897522483c5c3e.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 02/10] scalar-diagnose: avoid 32-bit overflow of size_t","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:35Z","receivedAt":"2022-08-04T01:45:59Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAvoid 32-bit size_t overflow when reporting the available disk space in\n'get_disk_info' by casting the block size and available block count to\n'off_t' before multiplying them. Without this change, 'st_mult' would\n(correctly) report a size_t overflow on 32-bit systems at or exceeding 2^32\nbytes of available space.\n\nNote that 'off_t' is a 64-bit integer even on 32-bit systems due to the\ninclusion of '#define _FILE_OFFSET_BITS 64' in 'git-compat-util.h' (see\nb97e911643 (Support for large files on 32bit systems., 2007-02-17)).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 97e71fe19cd..04046452284 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -348,7 +348,7 @@ static int get_disk_info(struct strbuf *out)\n \t}\n \n \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n+\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n \tstrbuf_release(&buf);\n #endif\n-- \ngitgitgadget\n\n"},{"id":"460589","messageId":"23349bfaf8fc5f5001f1ed1fa19e9b3909466ae3.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 03/10] scalar-diagnose: add directory to archiver more gently","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:36Z","receivedAt":"2022-08-04T01:46:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIf a directory added to the 'scalar diagnose' archiver does not exist, warn\nand return 0 from 'add_directory_to_archiver()' rather than failing with a\nfatal error. This handles a failure edge case where the '.git/logs' has not\nyet been created when running 'scalar diagnose', but extends to any\nsituation where a directory may be missing in the '.git' dir.\n\nNow, when a directory is missing a warning is captured in the diagnostic\nlogs. This provides a user with more complete information than if 'scalar\ndiagnose' simply failed with an error.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 04046452284..b9092f0b612 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -266,14 +266,20 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \t\t\t\t\t const char *path, int recurse)\n {\n \tint at_root = !*path;\n-\tDIR *dir = opendir(at_root ? \".\" : path);\n+\tDIR *dir;\n \tstruct dirent *e;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t len;\n \tint res = 0;\n \n-\tif (!dir)\n+\tdir = opendir(at_root ? \".\" : path);\n+\tif (!dir) {\n+\t\tif (errno == ENOENT) {\n+\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n+\t\t\treturn 0;\n+\t\t}\n \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\t}\n \n \tif (!at_root)\n \t\tstrbuf_addf(&buf, \"%s/\", path);\n-- \ngitgitgadget\n\n"},{"id":"460590","messageId":"05bba1e699ff242d496fb22d0735f23a901f8357.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 04/10] scalar-diagnose: move 'get_disk_info()' to 'compat/'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:37Z","receivedAt":"2022-08-04T01:46:12Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nMove 'get_disk_info()' function into 'compat/'. Although Scalar-specific\ncode is generally not part of the main Git tree, 'get_disk_info()' will be\nused in subsequent patches by additional callers beyond 'scalar diagnose'.\nThis patch prepares for that change, at which point this platform-specific\ncode should be part of 'compat/' as a matter of convention.\n\nThe function is copied *mostly* verbatim, with two exceptions:\n\n* '#ifdef WIN32' is replaced with '#ifdef GIT_WINDOWS_NATIVE' to allow\n 'statvfs' to be used with Cygwin.\n* the 'struct strbuf buf' and 'int res' (as well as their corresponding\n cleanup & return) are moved outside of the '#ifdef' block.\n\nSigned-off-by: Victoria Dye \n---\n compat/disk.h | 56 +++++++++++++++++++++++++++++++++++++++++\n contrib/scalar/scalar.c | 53 +-------------------------------------\n git-compat-util.h | 1 +\n 3 files changed, 58 insertions(+), 52 deletions(-)\n create mode 100644 compat/disk.h\n\ndiff --git a/compat/disk.h b/compat/disk.h\nnew file mode 100644\nindex 00000000000..50a32e3d8a4\n--- /dev/null\n+++ b/compat/disk.h\n@@ -0,0 +1,56 @@\n+#ifndef COMPAT_DISK_H\n+#define COMPAT_DISK_H\n+\n+#include \"git-compat-util.h\"\n+\n+static int get_disk_info(struct strbuf *out)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res = 0;\n+\n+#ifdef GIT_WINDOWS_NATIVE\n+\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n+\tDWORD serial_number, component_length, flags;\n+\tULARGE_INTEGER avail2caller, total, avail;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n+\t\terror(_(\"could not determine free disk size for '%s'\"),\n+\t\t buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n+\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n+\t\t\t\t &serial_number, &component_length, &flags,\n+\t\t\t\t fs_name, sizeof(fs_name))) {\n+\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n+\tstrbuf_addch(out, '\\n');\n+#else\n+\tstruct statvfs stat;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (statvfs(buf.buf, &stat) < 0) {\n+\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n+\t\t\t buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n+\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n+#endif\n+\n+cleanup:\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+#endif /* COMPAT_DISK_H */\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex b9092f0b612..607fedefd82 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -13,6 +13,7 @@\n #include \"help.h\"\n #include \"archive.h\"\n #include \"object-store.h\"\n+#include \"compat/disk.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -309,58 +310,6 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n \n-#ifndef WIN32\n-#include \n-#endif\n-\n-static int get_disk_info(struct strbuf *out)\n-{\n-#ifdef WIN32\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n-\tDWORD serial_number, component_length, flags;\n-\tULARGE_INTEGER avail2caller, total, avail;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n-\t\terror(_(\"could not determine free disk size for '%s'\"),\n-\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n-\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n-\t\t\t\t &serial_number, &component_length, &flags,\n-\t\t\t\t fs_name, sizeof(fs_name))) {\n-\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n-\tstrbuf_addch(out, '\\n');\n-\tstrbuf_release(&buf);\n-#else\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tstruct statvfs stat;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (statvfs(buf.buf, &stat) < 0) {\n-\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n-\t\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n-\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n-\tstrbuf_release(&buf);\n-#endif\n-\treturn 0;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 58d7708296b..9a62e3a0d2d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -258,6 +258,7 @@ static inline int is_xplatform_dir_sep(int c)\n #include \n #include \n #include \n+#include \n #include \n #ifndef NO_SYS_SELECT_H\n #include \n-- \ngitgitgadget\n\n"},{"id":"460591","messageId":"73e139ee377f9c50e671b0d94a28b93c1db28a69.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 06/10] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:39Z","receivedAt":"2022-08-04T01:46:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a 'git diagnose' builtin to generate a standalone zip archive of\nrepository diagnostics.\n\nThe \"diagnose\" functionality was originally implemented for Scalar in\naa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\ndiagnostics gathered are not specific to Scalar-cloned repositories and\ncan be useful when diagnosing issues in any Git repository.\n\nSigned-off-by: Victoria Dye \n---\n .gitignore | 1 +\n Documentation/git-diagnose.txt | 52 ++++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diagnose.c | 58 ++++++++++++++++++++++++++++++++++\n git.c | 1 +\n t/t0092-diagnose.sh | 28 ++++++++++++++++\n 7 files changed, 142 insertions(+)\n create mode 100644 Documentation/git-diagnose.txt\n create mode 100644 builtin/diagnose.c\n create mode 100755 t/t0092-diagnose.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 42fd7253b44..80b530bbed2 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -53,6 +53,7 @@\n /git-cvsimport\n /git-cvsserver\n /git-daemon\n+/git-diagnose\n /git-diff\n /git-diff-files\n /git-diff-index\ndiff --git a/Documentation/git-diagnose.txt b/Documentation/git-diagnose.txt\nnew file mode 100644\nindex 00000000000..b12ef98f013\n--- /dev/null\n+++ b/Documentation/git-diagnose.txt\n@@ -0,0 +1,52 @@\n+git-diagnose(1)\n+================\n+\n+NAME\n+----\n+git-diagnose - Generate a zip archive of diagnostic information\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\n+DESCRIPTION\n+-----------\n+Collects detailed information about the user's machine, Git client, and\n+repository state and packages that information into a zip archive. The\n+generated archive can then, for example, be shared with the Git mailing list to\n+help debug an issue or serve as a reference for independent debugging.\n+\n+The following information is captured in the archive:\n+\n+ * 'git version --build-options'\n+ * The path to the repository root\n+ * The available disk space on the filesystem\n+ * The name and size of each packfile, including those in alternate object\n+ stores\n+ * The total count of loose objects, as well as counts broken down by\n+ `.git/objects` subdirectory\n+ * The contents of the `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n+ `.git/objects/info` directories\n+\n+This tool differs from linkgit:git-bugreport[1] in that it collects much more\n+detailed information with a greater focus on reporting the size and data shape\n+of repository contents.\n+\n+OPTIONS\n+-------\n+-o ::\n+--output-directory ::\n+\tPlace the resulting diagnostics archive in `` instead of the\n+\tcurrent directory.\n+\n+-s ::\n+--suffix ::\n+\tSpecify an alternate suffix for the diagnostics archive name, to create\n+\ta file named 'git-diagnostics-'. This should take the\n+\tform of a strftime(3) format string; the current local time will be\n+\tused.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex ad27a0bd70c..9ceaf55582a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1154,6 +1154,7 @@ BUILTIN_OBJS += builtin/credential-cache.o\n BUILTIN_OBJS += builtin/credential-store.o\n BUILTIN_OBJS += builtin/credential.o\n BUILTIN_OBJS += builtin/describe.o\n+BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n BUILTIN_OBJS += builtin/diff-tree.o\ndiff --git a/builtin.h b/builtin.h\nindex 40e9ecc8485..8901a34d6bf 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -144,6 +144,7 @@ int cmd_credential_cache(int argc, const char **argv, const char *prefix);\n int cmd_credential_cache_daemon(int argc, const char **argv, const char *prefix);\n int cmd_credential_store(int argc, const char **argv, const char *prefix);\n int cmd_describe(int argc, const char **argv, const char *prefix);\n+int cmd_diagnose(int argc, const char **argv, const char *prefix);\n int cmd_diff_files(int argc, const char **argv, const char *prefix);\n int cmd_diff_index(int argc, const char **argv, const char *prefix);\n int cmd_diff(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/diagnose.c b/builtin/diagnose.c\nnew file mode 100644\nindex 00000000000..c545c6bae1d\n--- /dev/null\n+++ b/builtin/diagnose.c\n@@ -0,0 +1,58 @@\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"diagnose.h\"\n+\n+\n+static const char * const diagnose_usage[] = {\n+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n+\tNULL\n+};\n+\n+int cmd_diagnose(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct strbuf zip_path = STRBUF_INIT;\n+\ttime_t now = time(NULL);\n+\tstruct tm tm;\n+\tchar *option_output = NULL;\n+\tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n+\tchar *prefixed_filename;\n+\n+\tconst struct option diagnose_options[] = {\n+\t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n+\t\t\t N_(\"specify a destination for the diagnostics archive\")),\n+\t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n+\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, diagnose_options,\n+\t\t\t diagnose_usage, 0);\n+\n+\t/* Prepare the path to put the result */\n+\tprefixed_filename = prefix_filename(prefix,\n+\t\t\t\t\t option_output ? option_output : \"\");\n+\tstrbuf_addstr(&zip_path, prefixed_filename);\n+\tstrbuf_complete(&zip_path, '/');\n+\n+\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\tswitch (safe_create_leading_directories(zip_path.buf)) {\n+\tcase SCLD_OK:\n+\tcase SCLD_EXISTS:\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(_(\"could not create leading directories for '%s'\"),\n+\t\t zip_path.buf);\n+\t}\n+\n+\t/* Prepare diagnostics */\n+\tif (create_diagnostics_archive(&zip_path))\n+\t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n+\t\t\t zip_path.buf);\n+\n+\tfree(prefixed_filename);\n+\tstrbuf_release(&zip_path);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex e5d62fa5a92..0b9d8ef7677 100644\n--- a/git.c\n+++ b/git.c\n@@ -522,6 +522,7 @@ static struct cmd_struct commands[] = {\n \t{ \"credential-cache--daemon\", cmd_credential_cache_daemon },\n \t{ \"credential-store\", cmd_credential_store },\n \t{ \"describe\", cmd_describe, RUN_SETUP },\n+\t{ \"diagnose\", cmd_diagnose, RUN_SETUP_GENTLY },\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\ndiff --git a/t/t0092-diagnose.sh b/t/t0092-diagnose.sh\nnew file mode 100755\nindex 00000000000..fa05bf6046f\n--- /dev/null\n+++ b/t/t0092-diagnose.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='git diagnose'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success UNZIP 'creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test >out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\tgrep \"Available space\" out &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"460592","messageId":"3a0cb33c6583b591b1bbb51a3886f18023813d5b.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 05/10] scalar-diagnose: move functionality to common location","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:38Z","receivedAt":"2022-08-04T01:46:17Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nMove the core functionality of 'scalar diagnose' into a new 'diagnose.[c,h]'\nlibrary to prepare for new callers in the main Git tree generating\ndiagnostic archives. These callers will be introduced in subsequent patches.\n\nWhile this patch appears large, it is mostly made up of moving code out of\n'scalar.c' and into 'diagnose.c'. Specifically, the functions\n\n- dir_file_stats_objects()\n- dir_file_stats()\n- count_files()\n- loose_objs_stats()\n- add_directory_to_archiver()\n\nare all copied verbatim from 'scalar.c'. The 'create_diagnostics_archive()'\nfunction is a mostly identical (partial) copy of 'cmd_diagnose()', with the\nprimary changes being that 'zip_path' is an input and \"Enlistment root\" is\ncorrected to \"Repository root\" in the archiver log.\n\nSigned-off-by: Victoria Dye \n---\n Makefile | 1 +\n contrib/scalar/scalar.c | 202 +------------------------------------\n diagnose.c | 216 ++++++++++++++++++++++++++++++++++++++++\n diagnose.h | 9 ++\n 4 files changed, 228 insertions(+), 200 deletions(-)\n create mode 100644 diagnose.c\n create mode 100644 diagnose.h\n\ndiff --git a/Makefile b/Makefile\nindex 1624471badc..ad27a0bd70c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -932,6 +932,7 @@ LIB_OBJS += ctype.o\n LIB_OBJS += date.o\n LIB_OBJS += decorate.o\n LIB_OBJS += delta-islands.o\n+LIB_OBJS += diagnose.o\n LIB_OBJS += diff-delta.o\n LIB_OBJS += diff-merges.o\n LIB_OBJS += diff-lib.o\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 607fedefd82..3983def760a 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,9 +11,7 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"archive.h\"\n-#include \"object-store.h\"\n-#include \"compat/disk.h\"\n+#include \"diagnose.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -263,53 +261,6 @@ static int unregister_dir(void)\n \treturn res;\n }\n \n-static int add_directory_to_archiver(struct strvec *archiver_args,\n-\t\t\t\t\t const char *path, int recurse)\n-{\n-\tint at_root = !*path;\n-\tDIR *dir;\n-\tstruct dirent *e;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t len;\n-\tint res = 0;\n-\n-\tdir = opendir(at_root ? \".\" : path);\n-\tif (!dir) {\n-\t\tif (errno == ENOENT) {\n-\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n-\t\t\treturn 0;\n-\t\t}\n-\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n-\t}\n-\n-\tif (!at_root)\n-\t\tstrbuf_addf(&buf, \"%s/\", path);\n-\tlen = buf.len;\n-\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n-\n-\twhile (!res && (e = readdir(dir))) {\n-\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n-\t\t\tcontinue;\n-\n-\t\tstrbuf_setlen(&buf, len);\n-\t\tstrbuf_addstr(&buf, e->d_name);\n-\n-\t\tif (e->d_type == DT_REG)\n-\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n-\t\telse if (e->d_type != DT_DIR)\n-\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n-\t\t\t\t \"directory\"), buf.buf);\n-\t\telse if (recurse &&\n-\t\t\t add_directory_to_archiver(archiver_args,\n-\t\t\t\t\t\t buf.buf, recurse) < 0)\n-\t\t\tres = -1;\n-\t}\n-\n-\tclosedir(dir);\n-\tstrbuf_release(&buf);\n-\treturn res;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\n@@ -550,83 +501,6 @@ cleanup:\n \treturn res;\n }\n \n-static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n-\t\t\t\t const char *file_name, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\tstruct stat st;\n-\n-\tif (!stat(full_path, &st))\n-\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n-\t\t\t (uintmax_t)st.st_size);\n-}\n-\n-static int dir_file_stats(struct object_directory *object_dir, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\n-\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n-\n-\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n-\t\t\t\t data);\n-\n-\treturn 0;\n-}\n-\n-static int count_files(char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count = 0;\n-\n-\tif (!dir)\n-\t\treturn 0;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n-\t\t\tcount++;\n-\n-\tclosedir(dir);\n-\treturn count;\n-}\n-\n-static void loose_objs_stats(struct strbuf *buf, const char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count;\n-\tint total = 0;\n-\tunsigned char c;\n-\tstruct strbuf count_path = STRBUF_INIT;\n-\tsize_t base_path_len;\n-\n-\tif (!dir)\n-\t\treturn;\n-\n-\tstrbuf_addstr(buf, \"Object directory stats for \");\n-\tstrbuf_add_absolute_path(buf, path);\n-\tstrbuf_addstr(buf, \":\\n\");\n-\n-\tstrbuf_add_absolute_path(&count_path, path);\n-\tstrbuf_addch(&count_path, '/');\n-\tbase_path_len = count_path.len;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) &&\n-\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n-\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n-\t\t\tstrbuf_setlen(&count_path, base_path_len);\n-\t\t\tstrbuf_addstr(&count_path, e->d_name);\n-\t\t\ttotal += (count = count_files(count_path.buf));\n-\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n-\t\t}\n-\n-\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n-\n-\tstrbuf_release(&count_path);\n-\tclosedir(dir);\n-}\n-\n static int cmd_diagnose(int argc, const char **argv)\n {\n \tstruct option options[] = {\n@@ -637,12 +511,8 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tNULL\n \t};\n \tstruct strbuf zip_path = STRBUF_INIT;\n-\tstruct strvec archiver_args = STRVEC_INIT;\n-\tchar **argv_copy = NULL;\n-\tint stdout_fd = -1, archiver_fd = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n-\tstruct strbuf buf = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n@@ -663,79 +533,11 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\t\t zip_path.buf);\n \t\tgoto diagnose_cleanup;\n \t}\n-\tstdout_fd = dup(1);\n-\tif (stdout_fd < 0) {\n-\t\tres = error_errno(_(\"could not duplicate stdout\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tarchiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n-\tif (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) {\n-\t\tres = error_errno(_(\"could not redirect output\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tinit_zip_archiver();\n-\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n-\tget_version_info(&buf, 1);\n-\n-\tstrbuf_addf(&buf, \"Enlistment root: %s\\n\", the_repository->worktree);\n-\tget_disk_info(&buf);\n-\twrite_or_die(stdout_fd, buf.buf, buf.len);\n-\tstrvec_pushf(&archiver_args,\n-\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n-\t\t (int)buf.len, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n-\tdir_file_stats(the_repository->objects->odb, &buf);\n-\tforeach_alt_odb(dir_file_stats, &buf);\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n-\tloose_objs_stats(&buf, \".git/objects\");\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n-\t\tgoto diagnose_cleanup;\n-\n-\tstrvec_pushl(&archiver_args, \"--prefix=\",\n-\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n-\n-\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n-\targv_copy = xmemdupz(archiver_args.v,\n-\t\t\t sizeof(char *) * archiver_args.nr);\n-\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n-\t\t\t the_repository, NULL, 0);\n-\tif (res) {\n-\t\terror(_(\"failed to write archive\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n \n-\tif (!res)\n-\t\tfprintf(stderr, \"\\n\"\n-\t\t \"Diagnostics complete.\\n\"\n-\t\t \"All of the gathered info is captured in '%s'\\n\",\n-\t\t zip_path.buf);\n+\tres = create_diagnostics_archive(&zip_path);\n \n diagnose_cleanup:\n-\tif (archiver_fd >= 0) {\n-\t\tclose(1);\n-\t\tdup2(stdout_fd, 1);\n-\t}\n-\tfree(argv_copy);\n-\tstrvec_clear(&archiver_args);\n \tstrbuf_release(&zip_path);\n-\tstrbuf_release(&buf);\n-\n \treturn res;\n }\n \ndiff --git a/diagnose.c b/diagnose.c\nnew file mode 100644\nindex 00000000000..6c3774afb19\n--- /dev/null\n+++ b/diagnose.c\n@@ -0,0 +1,216 @@\n+#include \"diagnose.h\"\n+#include \"compat/disk.h\"\n+#include \"archive.h\"\n+#include \"dir.h\"\n+#include \"help.h\"\n+#include \"strvec.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_name, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\tstruct stat st;\n+\n+\tif (!stat(full_path, &st))\n+\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n+\t\t\t (uintmax_t)st.st_size);\n+}\n+\n+static int dir_file_stats(struct object_directory *object_dir, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\n+\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n+\n+\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n+\t\t\t\t data);\n+\n+\treturn 0;\n+}\n+\n+static int count_files(char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count = 0;\n+\n+\tif (!dir)\n+\t\treturn 0;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n+\t\t\tcount++;\n+\n+\tclosedir(dir);\n+\treturn count;\n+}\n+\n+static void loose_objs_stats(struct strbuf *buf, const char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count;\n+\tint total = 0;\n+\tunsigned char c;\n+\tstruct strbuf count_path = STRBUF_INIT;\n+\tsize_t base_path_len;\n+\n+\tif (!dir)\n+\t\treturn;\n+\n+\tstrbuf_addstr(buf, \"Object directory stats for \");\n+\tstrbuf_add_absolute_path(buf, path);\n+\tstrbuf_addstr(buf, \":\\n\");\n+\n+\tstrbuf_add_absolute_path(&count_path, path);\n+\tstrbuf_addch(&count_path, '/');\n+\tbase_path_len = count_path.len;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) &&\n+\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n+\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n+\t\t\tstrbuf_setlen(&count_path, base_path_len);\n+\t\t\tstrbuf_addstr(&count_path, e->d_name);\n+\t\t\ttotal += (count = count_files(count_path.buf));\n+\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n+\t\t}\n+\n+\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n+\n+\tstrbuf_release(&count_path);\n+\tclosedir(dir);\n+}\n+\n+static int add_directory_to_archiver(struct strvec *archiver_args,\n+\t\t\t\t const char *path, int recurse)\n+{\n+\tint at_root = !*path;\n+\tDIR *dir;\n+\tstruct dirent *e;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t len;\n+\tint res = 0;\n+\n+\tdir = opendir(at_root ? \".\" : path);\n+\tif (!dir) {\n+\t\tif (errno == ENOENT) {\n+\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n+\t\t\treturn 0;\n+\t\t}\n+\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\t}\n+\n+\tif (!at_root)\n+\t\tstrbuf_addf(&buf, \"%s/\", path);\n+\tlen = buf.len;\n+\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n+\n+\twhile (!res && (e = readdir(dir))) {\n+\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&buf, len);\n+\t\tstrbuf_addstr(&buf, e->d_name);\n+\n+\t\tif (e->d_type == DT_REG)\n+\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n+\t\telse if (e->d_type != DT_DIR)\n+\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n+\t\t\t\t \"directory\"), buf.buf);\n+\t\telse if (recurse &&\n+\t\t\t add_directory_to_archiver(archiver_args,\n+\t\t\t\t\t\t buf.buf, recurse) < 0)\n+\t\t\tres = -1;\n+\t}\n+\n+\tclosedir(dir);\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+int create_diagnostics_archive(struct strbuf *zip_path)\n+{\n+\tstruct strvec archiver_args = STRVEC_INIT;\n+\tchar **argv_copy = NULL;\n+\tint stdout_fd = -1, archiver_fd = -1;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res;\n+\n+\tstdout_fd = dup(STDOUT_FILENO);\n+\tif (stdout_fd < 0) {\n+\t\tres = error_errno(_(\"could not duplicate stdout\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tarchiver_fd = xopen(zip_path->buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n+\tif (dup2(archiver_fd, STDOUT_FILENO) < 0) {\n+\t\tres = error_errno(_(\"could not redirect output\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tinit_zip_archiver();\n+\tstrvec_pushl(&archiver_args, \"git-diagnose\", \"--format=zip\", NULL);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n+\tget_version_info(&buf, 1);\n+\n+\tstrbuf_addf(&buf, \"Repository root: %s\\n\", the_repository->worktree);\n+\tget_disk_info(&buf);\n+\twrite_or_die(stdout_fd, buf.buf, buf.len);\n+\tstrvec_pushf(&archiver_args,\n+\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n+\t\t (int)buf.len, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n+\tdir_file_stats(the_repository->objects->odb, &buf);\n+\tforeach_alt_odb(dir_file_stats, &buf);\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n+\tloose_objs_stats(&buf, \".git/objects\");\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n+\t\tgoto diagnose_cleanup;\n+\n+\tstrvec_pushl(&archiver_args, \"--prefix=\",\n+\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n+\n+\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n+\targv_copy = xmemdupz(archiver_args.v,\n+\t\t\t sizeof(char *) * archiver_args.nr);\n+\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n+\t\t\t the_repository, NULL, 0);\n+\tif (res) {\n+\t\terror(_(\"failed to write archive\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tif (!res)\n+\t\tfprintf(stderr, \"\\n\"\n+\t\t\t\"Diagnostics complete.\\n\"\n+\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n+\t\t\tzip_path->buf);\n+\n+diagnose_cleanup:\n+\tif (archiver_fd >= 0) {\n+\t\tdup2(stdout_fd, STDOUT_FILENO);\n+\t\tclose(stdout_fd);\n+\t\tclose(archiver_fd);\n+\t}\n+\tfree(argv_copy);\n+\tstrvec_clear(&archiver_args);\n+\tstrbuf_release(&buf);\n+\n+\treturn res;\n+}\ndiff --git a/diagnose.h b/diagnose.h\nnew file mode 100644\nindex 00000000000..e86e8a3c962\n--- /dev/null\n+++ b/diagnose.h\n@@ -0,0 +1,9 @@\n+#ifndef DIAGNOSE_H\n+#define DIAGNOSE_H\n+\n+#include \"cache.h\"\n+#include \"strbuf.h\"\n+\n+int create_diagnostics_archive(struct strbuf *zip_path);\n+\n+#endif /* DIAGNOSE_H */\n-- \ngitgitgadget\n\n"},{"id":"460593","messageId":"a3e62a4a0418abbff5015302f0102ffacccf0ca4.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 07/10] builtin/diagnose.c: gate certain data behind '--all'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:40Z","receivedAt":"2022-08-04T01:46:19Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate 'git diagnose' to *not* include '.git/' directory contents by\ndefault, instead requiring specification of a '--all' option to include it.\nWhile helpful for debugging, the archived '.git/' directory contents may be\nsensitive, as they can be used to reconstruct an entire repository.\n\nTo guard against users inadvertently including this information in\ndiagnostics and sharing it (e.g., with the mailing list), '.git/' directory\ncontents will only be included if '--all' is specified.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/git-diagnose.txt | 12 ++++++++++--\n builtin/diagnose.c | 8 ++++++--\n contrib/scalar/scalar.c | 2 +-\n diagnose.c | 14 ++++++++------\n diagnose.h | 2 +-\n t/t0092-diagnose.sh | 17 ++++++++++++++++-\n 6 files changed, 42 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-diagnose.txt b/Documentation/git-diagnose.txt\nindex b12ef98f013..374b7402511 100644\n--- a/Documentation/git-diagnose.txt\n+++ b/Documentation/git-diagnose.txt\n@@ -9,6 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\t [-a | --all]\n \n DESCRIPTION\n -----------\n@@ -26,8 +27,6 @@ The following information is captured in the archive:\n stores\n * The total count of loose objects, as well as counts broken down by\n `.git/objects` subdirectory\n- * The contents of the `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n- `.git/objects/info` directories\n \n This tool differs from linkgit:git-bugreport[1] in that it collects much more\n detailed information with a greater focus on reporting the size and data shape\n@@ -47,6 +46,15 @@ OPTIONS\n \tform of a strftime(3) format string; the current local time will be\n \tused.\n \n+-a::\n+--all::\n+\tInclude more complete repository diagnostic information in the archive.\n+\tSpecifically, this will add copies of `.git`, `.git/hooks`, `.git/info`,\n+\t`.git/logs`, and `.git/objects/info` directories to the output archive.\n+\tThis additional data may be sensitive; a user can reconstruct the full\n+\tcontents of the diagnosed repository with this information. Users should\n+\texercise caution when sharing an archive generated with this option.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/diagnose.c b/builtin/diagnose.c\nindex c545c6bae1d..0a2a63fdfbc 100644\n--- a/builtin/diagnose.c\n+++ b/builtin/diagnose.c\n@@ -4,7 +4,7 @@\n \n \n static const char * const diagnose_usage[] = {\n-\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ] [-a|--all]\"),\n \tNULL\n };\n \n@@ -13,6 +13,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \tstruct strbuf zip_path = STRBUF_INIT;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tint include_everything = 0;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tchar *prefixed_filename;\n@@ -22,6 +23,9 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"specify a destination for the diagnostics archive\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n \t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\tOPT_BOOL_F('a', \"all\", &include_everything,\n+\t\t\t N_(\"collect complete diagnostic information\"),\n+\t\t\t PARSE_OPT_NONEG),\n \t\tOPT_END()\n \t};\n \n@@ -48,7 +52,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \t}\n \n \t/* Prepare diagnostics */\n-\tif (create_diagnostics_archive(&zip_path))\n+\tif (create_diagnostics_archive(&zip_path, include_everything))\n \t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n \t\t\t zip_path.buf);\n \ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 3983def760a..b10955531ce 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -534,7 +534,7 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tgoto diagnose_cleanup;\n \t}\n \n-\tres = create_diagnostics_archive(&zip_path);\n+\tres = create_diagnostics_archive(&zip_path, 1);\n \n diagnose_cleanup:\n \tstrbuf_release(&zip_path);\ndiff --git a/diagnose.c b/diagnose.c\nindex 6c3774afb19..6be53d7a1f8 100644\n--- a/diagnose.c\n+++ b/diagnose.c\n@@ -131,7 +131,7 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n \n-int create_diagnostics_archive(struct strbuf *zip_path)\n+int create_diagnostics_archive(struct strbuf *zip_path, int include_everything)\n {\n \tstruct strvec archiver_args = STRVEC_INIT;\n \tchar **argv_copy = NULL;\n@@ -176,11 +176,13 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n \tloose_objs_stats(&buf, \".git/objects\");\n \tstrvec_push(&archiver_args, buf.buf);\n \n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n+\t/* Only include this if explicitly requested */\n+\tif (include_everything &&\n+\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n \t\tgoto diagnose_cleanup;\n \n \tstrvec_pushl(&archiver_args, \"--prefix=\",\ndiff --git a/diagnose.h b/diagnose.h\nindex e86e8a3c962..c0c5daf65e7 100644\n--- a/diagnose.h\n+++ b/diagnose.h\n@@ -4,6 +4,6 @@\n #include \"cache.h\"\n #include \"strbuf.h\"\n \n-int create_diagnostics_archive(struct strbuf *zip_path);\n+int create_diagnostics_archive(struct strbuf *zip_path, int include_everything);\n \n #endif /* DIAGNOSE_H */\ndiff --git a/t/t0092-diagnose.sh b/t/t0092-diagnose.sh\nindex fa05bf6046f..3c808698d3f 100755\n--- a/t/t0092-diagnose.sh\n+++ b/t/t0092-diagnose.sh\n@@ -22,7 +22,22 @@ test_expect_success UNZIP 'creates diagnostics zip archive' '\n \tgrep \".git/objects\" out &&\n \n \t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n-\tgrep \"^Total: [0-9][0-9]*\" out\n+\tgrep \"^Total: [0-9][0-9]*\" out &&\n+\n+\t# Should not include .git directory contents\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--all includes .git data in archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test --all >out &&\n+\n+\t# Should include .git directory contents\n+\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n+\ttest_file_not_empty out\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"460594","messageId":"6834bdcaea838cc49f209efd785bf2bdf09e9c08.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 09/10] scalar-diagnose: use 'git diagnose --all'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:42Z","receivedAt":"2022-08-04T01:46:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nReplace implementation of 'scalar diagnose' with an internal invocation of\n'git diagnose --all'. This simplifies the implementation of 'cmd_diagnose'\nby making it a direct alias of 'git diagnose' and removes some code in\n'scalar.c' that is duplicated in 'builtin/diagnose.c'. The simplicity of the\nalias also sets up a clean deprecation path for 'scalar diagnose' (in favor\nof 'git diagnose'), if that is desired in the future.\n\nThis introduces one minor change to the output of 'scalar diagnose', which\nis that the prefix of the created zip archive is changed from 'scalar_' to\n'git-diagnostics-'.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 29 +++++++----------------------\n 1 file changed, 7 insertions(+), 22 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex b10955531ce..fe2a0e9decb 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,7 +11,6 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"diagnose.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -510,34 +509,20 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tN_(\"scalar diagnose []\"),\n \t\tNULL\n \t};\n-\tstruct strbuf zip_path = STRBUF_INIT;\n-\ttime_t now = time(NULL);\n-\tstruct tm tm;\n+\tstruct strbuf diagnostics_root = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\t usage, 0);\n \n-\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n-\n-\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n-\tstrbuf_addftime(&zip_path,\n-\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n-\tstrbuf_addstr(&zip_path, \".zip\");\n-\tswitch (safe_create_leading_directories(zip_path.buf)) {\n-\tcase SCLD_EXISTS:\n-\tcase SCLD_OK:\n-\t\tbreak;\n-\tdefault:\n-\t\terror_errno(_(\"could not create directory for '%s'\"),\n-\t\t\t zip_path.buf);\n-\t\tgoto diagnose_cleanup;\n-\t}\n+\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_root);\n+\tstrbuf_addstr(&diagnostics_root, \"/.scalarDiagnostics\");\n \n-\tres = create_diagnostics_archive(&zip_path, 1);\n+\tif (run_git(\"diagnose\", \"--all\", \"-s\", \"%Y%m%d_%H%M%S\",\n+\t\t \"-o\", diagnostics_root.buf, NULL) < 0)\n+\t\tres = -1;\n \n-diagnose_cleanup:\n-\tstrbuf_release(&zip_path);\n+\tstrbuf_release(&diagnostics_root);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"460595","messageId":"d81e7c10997e9e8dc211d241019fbafa6b25fb04.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 08/10] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:41Z","receivedAt":"2022-08-04T01:46:23Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a '--diagnose' option for 'git bugreport' to collect additional\ninformation about the repository and write it to a zipped archive.\n\nThe '--diagnose' option behaves effectively as an alias for simultaneously\nrunning 'git bugreport' and 'git diagnose'. In the documentation, users are\nexplicitly recommended to attach the diagnostics alongside a bug report to\nprovide additional context to readers, ideally reducing some back-and-forth\nbetween reporters and those debugging the issue.\n\nNote that '--diagnose' may take an optional string arg (either 'basic' or\n'all'). If specified without the arg or with 'basic', the behavior\ncorresponds to running 'git diagnose' without '--all'; this default is meant\nto help reduce unintentional leaking of sensitive information). However, a\nuser can still manually specify '--diagnose=all' to generate the equivalent\narchive to one created with 'git diagnose --all'.\n\nSuggested-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n Documentation/git-bugreport.txt | 18 +++++++++++++\n builtin/bugreport.c | 47 +++++++++++++++++++++++++++++++--\n t/t0091-bugreport.sh | 44 ++++++++++++++++++++++++++++++\n 3 files changed, 107 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-bugreport.txt b/Documentation/git-bugreport.txt\nindex d8817bf3cec..a4d984a77be 100644\n--- a/Documentation/git-bugreport.txt\n+++ b/Documentation/git-bugreport.txt\n@@ -9,6 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\t\t[--diagnose[=(basic|all)]]\n \n DESCRIPTION\n -----------\n@@ -31,6 +32,10 @@ The following information is captured automatically:\n - A list of enabled hooks\n - $SHELL\n \n+Additional information may be gathered into a separate zip archive using the\n+`--diagnose` option, and can be attached alongside the bugreport document to\n+provide additional context to readers.\n+\n This tool is invoked via the typical Git setup process, which means that in some\n cases, it might not be able to launch - for example, if a relevant config file\n is unreadable. In this kind of scenario, it may be helpful to manually gather\n@@ -49,6 +54,19 @@ OPTIONS\n \tnamed 'git-bugreport-'. This should take the form of a\n \tstrftime(3) format string; the current local time will be used.\n \n+--diagnose[=(basic|all)]::\n+\tCreate a zip archive of information about the repository including logs\n+\tand certain statistics describing the data shape of the repository. The\n+\tarchive is written to the same output directory as the bug report and is\n+\tnamed 'git-diagnostics-'.\n++\n+By default, `--diagnose` (equivalent to `--diagnose=basic`) will collect only\n+statistics and summarized data about the repository and filesystem. Specifying\n+`--diagnose=all` will create an archive with the same contents generated by `git\n+diagnose --all`; this archive will be much larger, and will contain potentially\n+sensitive information about the repository. See linkgit:git-diagnose[1] for more\n+details on the contents of the diagnostic archive.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 9de32bc96e7..1907258d61d 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -5,7 +5,13 @@\n #include \"compat/compiler.h\"\n #include \"hook.h\"\n #include \"hook-list.h\"\n+#include \"diagnose.h\"\n \n+enum diagnose_mode {\n+\tDIAGNOSE_NONE,\n+\tDIAGNOSE_BASIC,\n+\tDIAGNOSE_ALL\n+};\n \n static void get_system_info(struct strbuf *sys_info)\n {\n@@ -91,6 +97,23 @@ static void get_header(struct strbuf *buf, const char *title)\n \tstrbuf_addf(buf, \"\\n\\n[%s]\\n\", title);\n }\n \n+static int option_parse_diagnose(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tenum diagnose_mode *diagnose = opt->value;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\n+\tif (!arg || !strcmp(arg, \"basic\"))\n+\t\t*diagnose = DIAGNOSE_BASIC;\n+\telse if (!strcmp(arg, \"all\"))\n+\t\t*diagnose = DIAGNOSE_ALL;\n+\telse\n+\t\tdie(_(\"diagnose mode must be either 'basic' or 'all'\"));\n+\n+\treturn 0;\n+}\n+\n int cmd_bugreport(int argc, const char **argv, const char *prefix)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n@@ -98,16 +121,21 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tint report = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tenum diagnose_mode diagnose = DIAGNOSE_NONE;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tconst char *user_relative_path = NULL;\n \tchar *prefixed_filename;\n+\tsize_t output_path_len;\n \n \tconst struct option bugreport_options[] = {\n+\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"(basic|all)\"),\n+\t\t\t N_(\"create an additional zip archive of detailed diagnostics\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_OPTARG, option_parse_diagnose),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n-\t\t\t N_(\"specify a destination for the bugreport file\")),\n+\t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n-\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\t\t N_(\"specify a strftime format suffix for the filename(s)\")),\n \t\tOPT_END()\n \t};\n \n@@ -119,6 +147,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t option_output ? option_output : \"\");\n \tstrbuf_addstr(&report_path, prefixed_filename);\n \tstrbuf_complete(&report_path, '/');\n+\toutput_path_len = report_path.len;\n \n \tstrbuf_addstr(&report_path, \"git-bugreport-\");\n \tstrbuf_addftime(&report_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n@@ -133,6 +162,20 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t report_path.buf);\n \t}\n \n+\t/* Prepare diagnostics, if requested */\n+\tif (diagnose != DIAGNOSE_NONE) {\n+\t\tstruct strbuf zip_path = STRBUF_INIT;\n+\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n+\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\t\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\t\tif (create_diagnostics_archive(&zip_path, diagnose == DIAGNOSE_ALL))\n+\t\t\tdie_errno(_(\"unable to create diagnostics archive %s\"), zip_path.buf);\n+\n+\t\tstrbuf_release(&zip_path);\n+\t}\n+\n \t/* Prepare the report contents */\n \tget_bug_template(&buffer);\n \ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 08f5fe9caef..1a63b3803f3 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -78,4 +78,48 @@ test_expect_success 'indicates populated hooks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose -o report -s test >out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\tgrep \"Available space\" out &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out &&\n+\n+\t# Should not include .git directory contents by default\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--diagnose=basic excludes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose=basic -o report -s test >out &&\n+\n+\t# Should not include .git directory contents\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--diagnose=all includes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose=all -o report -s test >out &&\n+\n+\t# Should include .git directory contents\n+\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n+\ttest_file_not_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"460596","messageId":"14925c3feede27315a22645fc97e4546c1e6094e.1659577543.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v2 10/10] scalar: update technical doc roadmap","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-04T01:45:43Z","receivedAt":"2022-08-04T01:46:46Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate the Scalar roadmap to reflect the completion of generalizing 'scalar\ndiagnose' into 'git diagnose' and 'git bugreport --diagnose'.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/technical/scalar.txt | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt\nindex 08bc09c225a..f6353375f08 100644\n--- a/Documentation/technical/scalar.txt\n+++ b/Documentation/technical/scalar.txt\n@@ -84,6 +84,9 @@ series have been accepted:\n \n - `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n \n+- `scalar-generalize-diagnose`: Move the functionality of `scalar diagnose`\n+ into `git diagnose` and `git bugreport --diagnose`.\n+\n Roughly speaking (and subject to change), the following series are needed to\n \"finish\" this initial version of Scalar:\n \n@@ -91,12 +94,6 @@ Roughly speaking (and subject to change), the following series are needed to\n and implement `scalar help`. At the end of this series, Scalar should be\n feature-complete from the perspective of a user.\n \n-- Generalize features not specific to Scalar: In the spirit of making Scalar\n- configure only what is needed for large repo performance, move common\n- utilities into other parts of Git. Some of this will be internal-only, but one\n- major change will be generalizing `scalar diagnose` for use with any Git\n- repository.\n-\n - Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of\n `git`, including updates to build and install it with the rest of Git. This\n change will incorporate Scalar into the Git CI and test framework, as well as\n-- \ngitgitgadget\n"},{"id":"460598","messageId":"220804.868ro4fryg.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"23349bfaf8fc5f5001f1ed1fa19e9b3909466ae3.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 03/10] scalar-diagnose: add directory to archiver more gently","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-04T06:19:53Z","receivedAt":"2022-08-04T06:22:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n>\n> If a directory added to the 'scalar diagnose' archiver does not exist, warn\n> and return 0 from 'add_directory_to_archiver()' rather than failing with a\n> fatal error. This handles a failure edge case where the '.git/logs' has not\n> yet been created when running 'scalar diagnose', but extends to any\n> situation where a directory may be missing in the '.git' dir.\n>\n> Now, when a directory is missing a warning is captured in the diagnostic\n> logs. This provides a user with more complete information than if 'scalar\n> diagnose' simply failed with an error.\n>\n> Helped-by: Junio C Hamano \n> Signed-off-by: Victoria Dye \n> ---\n> contrib/scalar/scalar.c | 10 ++++++++--\n> 1 file changed, 8 insertions(+), 2 deletions(-)\n>\n> diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\n> index 04046452284..b9092f0b612 100644\n> --- a/contrib/scalar/scalar.c\n> +++ b/contrib/scalar/scalar.c\n> @@ -266,14 +266,20 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n> \t\t\t\t\t const char *path, int recurse)\n> {\n> \tint at_root = !*path;\n> -\tDIR *dir = opendir(at_root ? \".\" : path);\n> +\tDIR *dir;\n> \tstruct dirent *e;\n> \tstruct strbuf buf = STRBUF_INIT;\n> \tsize_t len;\n> \tint res = 0;\n> \n> -\tif (!dir)\n> +\tdir = opendir(at_root ? \".\" : path);\n> +\tif (!dir) {\n> +\t\tif (errno == ENOENT) {\n\nPer [1] I think this is incorrect or overly strict. Let's not spew\nwarnings if the user \"rm -rf .git/hooks\" or whatever.\n\nIt might be valuable to note in some file in the archive such oddities\nwe find, but warning() won't give us that.\n\n> +\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n\nIn any case, this should be e.g.:\n\n\twarning_errno(_(\"could not archive directory '%s'\"), path);\n\nYou already have an errno, so using *_errno() will add the standard\ninformation about what the issue is.\n\n> +\t\t\treturn 0;\n> +\t\t}\n> \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n> +\t}\n> \n> \tif (!at_root)\n> \t\tstrbuf_addf(&buf, \"%s/\", path);\n\n1. https://lore.kernel.org/git/220610.86ilp9s1x7.gmgdl@evledraar.gmail.com/\n"},{"id":"460599","messageId":"220804.864jysfrtr.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"3a0cb33c6583b591b1bbb51a3886f18023813d5b.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/10] scalar-diagnose: move functionality to common location","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-04T06:24:25Z","receivedAt":"2022-08-04T06:25:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n> [...]\n> diff --git a/diagnose.h b/diagnose.h\n> new file mode 100644\n> index 00000000000..e86e8a3c962\n> --- /dev/null\n> +++ b/diagnose.h\n> @@ -0,0 +1,9 @@\n> +#ifndef DIAGNOSE_H\n> +#define DIAGNOSE_H\n> +\n> +#include \"cache.h\"\n\nWe don't need cache.h here, just ...\n\n> +#include \"strbuf.h\"\n\n...this, also a matter of preference, but we could also just skip the\nincludes here and use a forward decl (as is common in other headers):\n\n\tstruct strbuf;\n\n> +\n> +int create_diagnostics_archive(struct strbuf *zip_path);\n> +\n> +#endif /* DIAGNOSE_H */\n\n"},{"id":"460600","messageId":"220804.86zggkecmk.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"73e139ee377f9c50e671b0d94a28b93c1db28a69.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/10] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-04T06:27:08Z","receivedAt":"2022-08-04T06:39:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n>\n> Create a 'git diagnose' builtin to generate a standalone zip archive of\n> repository diagnostics.\n\nIt's good to have this as a built-in separate from \"git bugreport\",\nbut...\n\n> +git-diagnose - Generate a zip archive of diagnostic information\n\n...I'd really prefer for this not to squat on such a common name we\nmight regret having reserved later for such very specific\nfunctionality. I'd think e.g. these would be better:\n\n\tgit mk-diagnostics-zip\n\nOr maybe:\n\n\tgit archive-interesting-for-report\n\nIf I had to guess what a \"git diagnose\" did, I'd probably think:\n\n * It analyzes your config, and suggests redundancies/alternatives\n * It does some perf tests / heuritics, and e.g. suggests you turn on\n the commit-graph writing.\n\netc., this (arguably even too generic then) made sense as \"scalar\ndiagnose\" because scalar is all about being an opinionated interface\ntargeted at performance\", so there's an implied \"my repo's performance\"\nfollowing a \"scalar diagnose\".\n\nBut as a top-level command-name I think we should pick something more\nspecific to what it does, which is (I haven't fully read ahead in the\nre-roll, but I'm assuming is) mostly/entirely to be a \"helper\" for\n\"scalar diagnose\" and/or \"git bugreport\".\n\n> +\tswitch (safe_create_leading_directories(zip_path.buf)) {\n> +\tcase SCLD_OK:\n> +\tcase SCLD_EXISTS:\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tdie(_(\"could not create leading directories for '%s'\"),\n> +\t\t zip_path.buf);\n\nThis seems to be carrying forward a minor bug from bugreport.c we should\nprobably fix: we should use die_errno() here (and maybe lead with a\ncommit to fix bugreport.c's version).\n\nThe strbuf*() before that also seems like a good candidate for a utility\nfunction in your new diagnose library, i.e. to have both bugreport.c and\ndiagnose.c pass it the prefix/suffix/format, then try to create that\ndirectory, then replace the copy/pasting here with a one-line call to\nthat now-shared function.\n\nThe two codepaths only seem to differ in the prefix & suffix from a\nquick skim, the rest is all copy/pasted...\n\n"},{"id":"460601","messageId":"220804.86v8r8ec4s.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"a3e62a4a0418abbff5015302f0102ffacccf0ca4.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/10] builtin/diagnose.c: gate certain data behind '--all'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-04T06:39:47Z","receivedAt":"2022-08-04T06:50:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n> [...]\n> --------\n> [verse]\n> 'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n> +\t [-a | --all]\n\nI have some local patches that...\n\n> static const char * const diagnose_usage[] = {\n> -\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n> +\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ] [-a|--all]\"),\n> \tNULL\n> };\n\n...spot when we have SYNOPSIS & -h discrepancies. In this case we break\nwith a \\n after in the SYNOPSIS, but you don't add a \"\\n\" and\nindentation here in the -h output. The two should be the same.\n\n> @@ -13,6 +13,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n> \tstruct strbuf zip_path = STRBUF_INIT;\n> \ttime_t now = time(NULL);\n> \tstruct tm tm;\n> +\tint include_everything = 0;\n> \tchar *option_output = NULL;\n> \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n> \tchar *prefixed_filename;\n> @@ -22,6 +23,9 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n> \t\t\t N_(\"specify a destination for the diagnostics archive\")),\n> \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n> \t\t\t N_(\"specify a strftime format suffix for the filename\")),\n> +\t\tOPT_BOOL_F('a', \"all\", &include_everything,\n> +\t\t\t N_(\"collect complete diagnostic information\"),\n> +\t\t\t PARSE_OPT_NONEG),\n\nNice to have a \"stats only\" by default and some way to add the whole\nshebang optionally...\n\n> +int create_diagnostics_archive(struct strbuf *zip_path, int include_everything)\n\n...but maybe...\n> {\n> \tstruct strvec archiver_args = STRVEC_INIT;\n> \tchar **argv_copy = NULL;\n> @@ -176,11 +176,13 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n> \tloose_objs_stats(&buf, \".git/objects\");\n> \tstrvec_push(&archiver_args, buf.buf);\n> \n> -\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n> +\t/* Only include this if explicitly requested */\n> +\tif (include_everything &&\n> +\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n> \t\tgoto diagnose_cleanup;\n\n...this should be --include-gitdir-extract or something, instead of\n\"--all\" and \"--include-everything\".\n\nI'd think that \"all\" would be a thing that would actually tar up my\nentire .git directory as-is (in a way that would pass git fsck on the\nother end (unless that's the bug being reported...)).\n\nAside: Since we are getting the churn of adding this, then re-indenting\n it maybe a prep step of adding a add_directories_to_archiver() would be\n useful, which would just have a data-driven:\n\n\t{\n\t\t{ \".git\" },\n\t\t[...],\n\t\t{ \".git/logs, 1 },\n\t\tNULL\n\t},\n\nThen loop over that, making it easy to add/declare new subdirs to add.\n"},{"id":"460602","messageId":"220804.86r11webp8.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"6834bdcaea838cc49f209efd785bf2bdf09e9c08.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/10] scalar-diagnose: use 'git diagnose --all'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-04T06:54:01Z","receivedAt":"2022-08-04T06:59:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n>\n> Replace implementation of 'scalar diagnose' with an internal invocation of\n> 'git diagnose --all'. This simplifies the implementation of 'cmd_diagnose'\n> by making it a direct alias of 'git diagnose' and removes some code in\n> 'scalar.c' that is duplicated in 'builtin/diagnose.c'. The simplicity of the\n> alias also sets up a clean deprecation path for 'scalar diagnose' (in favor\n> of 'git diagnose'), if that is desired in the future.\n>\n> This introduces one minor change to the output of 'scalar diagnose', which\n> is that the prefix of the created zip archive is changed from 'scalar_' to\n> 'git-diagnostics-'.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> contrib/scalar/scalar.c | 29 +++++++----------------------\n> 1 file changed, 7 insertions(+), 22 deletions(-)\n>\n> diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\n> index b10955531ce..fe2a0e9decb 100644\n> --- a/contrib/scalar/scalar.c\n> +++ b/contrib/scalar/scalar.c\n> @@ -11,7 +11,6 @@\n> #include \"dir.h\"\n> #include \"packfile.h\"\n> #include \"help.h\"\n> -#include \"diagnose.h\"\n> \n> /*\n> * Remove the deepest subdirectory in the provided path string. Path must not\n> @@ -510,34 +509,20 @@ static int cmd_diagnose(int argc, const char **argv)\n> \t\tN_(\"scalar diagnose []\"),\n> \t\tNULL\n> \t};\n> -\tstruct strbuf zip_path = STRBUF_INIT;\n> -\ttime_t now = time(NULL);\n> -\tstruct tm tm;\n> +\tstruct strbuf diagnostics_root = STRBUF_INIT;\n> \tint res = 0;\n> \n> \targc = parse_options(argc, argv, NULL, options,\n> \t\t\t usage, 0);\n> \n> -\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n> -\n> -\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n> -\tstrbuf_addftime(&zip_path,\n> -\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n> -\tstrbuf_addstr(&zip_path, \".zip\");\n> -\tswitch (safe_create_leading_directories(zip_path.buf)) {\n> -\tcase SCLD_EXISTS:\n> -\tcase SCLD_OK:\n> -\t\tbreak;\n> -\tdefault:\n> -\t\terror_errno(_(\"could not create directory for '%s'\"),\n> -\t\t\t zip_path.buf);\n> -\t\tgoto diagnose_cleanup;\n\nJust spotting this now, but we had ad error, but we \"goto\ndiagnose_cleanup\", but that will use our \"res = 0\" above.\n\nIs this untested already or in this series (didn't go back to look). But\nmaybe a moot point, the post-image replacement uses die()..\n\n> -\t}\n> +\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_root);\n> +\tstrbuf_addstr(&diagnostics_root, \"/.scalarDiagnostics\");\n> \n> -\tres = create_diagnostics_archive(&zip_path, 1);\n> +\tif (run_git(\"diagnose\", \"--all\", \"-s\", \"%Y%m%d_%H%M%S\",\n> +\t\t \"-o\", diagnostics_root.buf, NULL) < 0)\n> +\t\tres = -1;\n\nThe code handling here seems really odd, issues:\n\n * This *can* return -1, if start_command() fails, but that's by far the\n rarer case, usually it would be 0 or >0 (only <0 if we can't start\n the command at all).\n\n * You should not be returning -1 from cmd_*() in general (we have\n outstanding issues with it, but those should be fixed). It will yield\n an exit code of 255 (but it's not portable)).\n\n * If you're going to return -1 at all, why override <0 with -1, just\n \"res = run_git(...)\" instead?\n\nI think all-in-all this should be:\n\n\tres = run_git(...);\n\nThen:\n\n> \n> -diagnose_cleanup:\n> -\tstrbuf_release(&zip_path);\n> +\tstrbuf_release(&diagnostics_root);\n> \treturn res;\n\n\treturn res < 0 ? -res : res;\n\nOr whatever.\n"},{"id":"460646","messageId":"xmqq35ec6igq.fsf@gitster.g","threadId":"58251","inReplyTo":"220804.868ro4fryg.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 03/10] scalar-diagnose: add directory to archiver more gently","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-04T17:12:37Z","receivedAt":"2022-08-04T17:12:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> +\tdir = opendir(at_root ? \".\" : path);\n>> +\tif (!dir) {\n>> +\t\tif (errno == ENOENT) {\n>\n> Per [1]\n\n\"Per [1]\" somehow sounds more like a reference to an authoritative\nsource, at least to me. Every time you use it, I have to see what\nit refers to, and after realizing that you used it as a replacement\nof \"I said it already in [1]\" again, it leaves a funny feeling.\n\n> I think this is incorrect or overly strict. Let's not spew\n> warnings if the user \"rm -rf .git/hooks\" or whatever.\n\nThe above is doing the right thing even in that situation, doesn't\nit? If there is no \".git/hooks\" that is fine. We get ENOENT, give\na warning to indicate that we found an unusual situation, and return\nwithout failing. If we got something other than ENOENT, we fail with\nerror_errno(), because opendir() failed for a reason other than \"No\nsuch file or directory\".\n\n> You already have an errno, so using *_errno() will add the standard\n> information about what the issue is.\n\nReading the code aloud, slowly, may help. When errno says ENOENT,\nwe know opendir() failed because of \"No such file or directory\",\nso \"path\" was missing. So let's say 'not archiving a missing directory'\".\n\nENOENT or \"No such file or directory\" is an implementation detail\nthat does not help the end user.\n\nThe other side, i.e. when the errno is *not* ENOENT, already uses\nerror_errno().\n\nSo, I am puzzled.\n"},{"id":"460647","messageId":"xmqqwnbn6i0v.fsf@gitster.g","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/10] Generalize 'scalar diagnose' into 'git diagnose' and 'git bugreport --diagnose'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-04T17:22:08Z","receivedAt":"2022-08-04T17:22:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> * (Almost) entirely redesigned the UI for generating diagnostics. The new\n> approach avoids cluttering 'git bugreport' with a mode that doesn't\n> actually generate a report. Now, there are distinct options for different\n> use cases: generating extra diagnostics with a bug report ('git bugreport\n> --diagnose') and generating diagnostics for personal debugging/addition\n> to an existing bug report ('git diagnose').\n\nAn additional command gives us far more design flexibility, and in\nthis case I think it may be worth it. It has a risk of confusing\nusers between \"git bugreport --diag\" and \"git diagnose --report\" a\nway to send a report with diagnostic information, though.\n\n\n"},{"id":"460661","messageId":"220804.8635ebu55j.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"xmqq35ec6igq.fsf@gitster.g","subject":"Re: [PATCH v2 03/10] scalar-diagnose: add directory to archiver more gently","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-04T20:12:56Z","receivedAt":"2022-08-04T20:26:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 04 2022, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>>> +\tdir = opendir(at_root ? \".\" : path);\n>>> +\tif (!dir) {\n>>> +\t\tif (errno == ENOENT) {\n>>\n>> Per [1]\n>\n> \"Per [1]\" somehow sounds more like a reference to an authoritative\n> source, at least to me. Every time you use it, I have to see what\n> it refers to, and after realizing that you used it as a replacement\n> of \"I said it already in [1]\" again, it leaves a funny feeling.\n\n\"Per\" in the sense of \"Per what I noted in [1]\".\n\n>> I think this is incorrect or overly strict. Let's not spew\n>> warnings if the user \"rm -rf .git/hooks\" or whatever.\n>\n> The above is doing the right thing even in that situation, doesn't\n> it? If there is no \".git/hooks\" that is fine. We get ENOENT, give\n> a warning to indicate that we found an unusual situation, and return\n> without failing. If we got something other than ENOENT, we fail with\n> error_errno(), because opendir() failed for a reason other than \"No\n> such file or directory\".\n\nI'm mainly noting that the point of this step is to produce an archive\nfor the consumption of the remote end.\n\nTherefore it seems to me like it would me much more useful to note these\n\"oddities\" in some log that we're about to zip up, rather than issue a\nwarning().\n\nThe \"per [1]\" was a reference to the (paraphrasing) \"maybe that's not\nneeded at all\", but you seemed to think so. But for the purposes of the\ndiscussion here let's assume we keep it.\n\n>> You already have an errno, so using *_errno() will add the standard\n>> information about what the issue is.\n>\n> Reading the code aloud, slowly, may help. When errno says ENOENT,\n> we know opendir() failed because of \"No such file or directory\",\n> so \"path\" was missing. So let's say 'not archiving a missing directory'\".\n>\n> ENOENT or \"No such file or directory\" is an implementation detail\n> that does not help the end user.\n>\n> The other side, i.e. when the errno is *not* ENOENT, already uses\n> error_errno().\n>\n> So, I am puzzled.\n\nI'm pointing out that we don't need to include that part in the message,\nbecause warning_errno() will already give us that for free. I.e.:\n\n\twarning: could not archive directory '': No such file or directory\n\nv.s.:\n\n\twarning: could not archive missing directory ''\n\nThe advantages of doing so being:\n\n * It's clear (at least to the keen eye) that it's using the \"errno\n format\", so you know it's not just saying \"could not for \", it specifically got ENOENT.\n\n * The i18n for the strerror() comes from the C library, which will be\n translated already, whereas a new git.pot message won't be (but we'll\n hopefully bridge the gap eventually).\n\n * This way we we can share the message, whatever the errno happens to\n be, so we could e.g.:\n\n errno = ENOENT;\n warning_errno(_(\"could not archive directory '%s'\"), \"\");\n errno = ENOMEM;\n error_errno(_(\"could not archive directory '%s'\"), \"\");\n\n Whereas putting the reason for why we couldn't (which just duplicates\n the errno) in the message forces the messages & i18n to diverge.\n"},{"id":"460670","messageId":"xmqqsfmb3ed9.fsf@gitster.g","threadId":"58251","inReplyTo":"220804.8635ebu55j.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 03/10] scalar-diagnose: add directory to archiver more gently","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-04T21:09:22Z","receivedAt":"2022-08-04T21:09:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I'm mainly noting that the point of this step is to produce an archive\n> for the consumption of the remote end.\n>\n> Therefore it seems to me like it would me much more useful to note these\n> \"oddities\" in some log that we're about to zip up, rather than issue a\n> warning().\n\nHmph, the receiving end that inspects the archive will know that a\ndirectory did not get archived, but they cannot tell if that is\nbecause it did not exist, or because it was unreadable, and the\ntrouble the user may be having can well be the result of having\nthe directory unreadable. So from that point of view, in addition\nto these warnings and errors, it would be helpful to record the\nerrors we encounter while we generate the diagnostic archive in the\narchive itself for inspection.\n\nBut the warning on the local side has merit to warn the user of an\nunusual situation. \"I am puzzled why the hook I thought I wrote did\nnot trigger, but the diag tool says I do not have .git/hooks at all\"\nis a welcome side effect, even though it may not be the primary\neffect we are aiming to gain by having these warning messages.\n\n> I'm pointing out that we don't need to include that part in the message,\n> because warning_errno() will already give us that for free. I.e.:\n>\n> \twarning: could not archive directory '': No such file or directory\n>\n> v.s.:\n>\n> \twarning: could not archive missing directory ''\n>\n> The advantages of doing so being:\n>\n> * It's clear (at least to the keen eye) that it's using the \"errno\n> format\", so you know it's not just saying \"could not for reason>\", it specifically got ENOENT.\n\nFunny. I find it much clearer if we can use our own message without\nhaving to rely on whatever strerror(error) gives us. We know better\nthan the C library why we got ENOENT and be more readable. They say\n\"No such file or directory\" because from ENOENT alone they cannot\ntell you if it was a file or a directory that was missing, but we\nhave a better context like \"we were trying to create an archive\" and\n\"we tried opendir, expecting that the thing is a directory\".\n\n> * The i18n for the strerror() comes from the C library, which will be\n> translated already, whereas a new git.pot message won't be (but we'll\n> hopefully bridge the gap eventually).\n\nAs I said above, I do not think it is an advantage in this case that\nstrerror() is translated, as the point of having a separate message\nis because we can be more to-the-point, and we do not need to use\nthe strerror() result in there.\n\n> * This way we we can share the message, whatever the errno happens to\n> be, so we could e.g.:\n>\n> errno = ENOENT;\n> warning_errno(_(\"could not archive directory '%s'\"), \"\");\n> errno = ENOMEM;\n> error_errno(_(\"could not archive directory '%s'\"), \"\");\n>\n> Whereas putting the reason for why we couldn't (which just duplicates\n> the errno) in the message forces the messages & i18n to diverge.\n\nAnd an added clarity that we can use a separate message is something\nI think is worth having, compared to the cost of having an extra\nmessage over the more generic one for any errno.\n"},{"id":"460736","messageId":"7cd85665-3827-4f83-4e39-564e161fdea7@github.com","threadId":"58251","inReplyTo":"73e139ee377f9c50e671b0d94a28b93c1db28a69.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/10] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-05T19:11:12Z","receivedAt":"2022-08-05T19:11:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2022 9:45 PM, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> Create a 'git diagnose' builtin to generate a standalone zip archive of\n> repository diagnostics.\n\n> + * The contents of the `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n> + `.git/objects/info` directories\n\nYou remove these lines in the next patch, which is called \"gate certain data\nbehind '--all'\" but maybe we shouldn't have this functionality now and\ninstead add it in the future.\n\nThe biggest reason for the --all option is that these contents will likely\ninclude private IP (path names and branch names, but not file contents) that\nthe user would probably not want to share with the public mailing list, but\nmight want to share with a trusted Git expert in order to resolve a problem.\nYou mention earlier that\n\n The generated archive can then, for example, be shared with the\n Git mailing list to help debug an issue or serve as a reference for\n independent debugging.\n\nSo, if you're sending a v3, then moving this out of this patch and into the\nnext one would be a good way to be sure that this possibly-private data is\nnot mentioned as something to share super publicly.\n\n(Of course, this requires making the change to create_diagnostics_archive()\nin advance of creating the builtin, so maybe this reorganization isn't\nworth it.)\n\n> @@ -0,0 +1,58 @@\n> +#include \"builtin.h\"\n> +#include \"parse-options.h\"\n> +#include \"diagnose.h\"\n> +\n> +\n\nnit: double empty line\n\n> +++ b/t/t0092-diagnose.sh\n> @@ -0,0 +1,28 @@\n> +#!/bin/sh\n> +\n> +test_description='git diagnose'\n> +\n> +TEST_PASSES_SANITIZE_LEAK=true\n> +. ./test-lib.sh\n> +\n> +test_expect_success UNZIP 'creates diagnostics zip archive' '\n> +\ttest_when_finished rm -rf report &&\n> +\n> +\tgit diagnose -o report -s test >out &&\n> +\n> +\tzip_path=report/git-diagnostics-test.zip &&\n> +\tgrep \"Available space\" out &&\n> +\ttest_path_is_file \"$zip_path\" &&\n\nnit: 'grep' the output immediately after the 'git diagnose' command and\nkeep the zip_path use immediately after its definition.\n\nThanks,\n-Stolee\n"},{"id":"460738","messageId":"3dc402e1-1f27-8a24-544d-d90d403a7da0@github.com","threadId":"58251","inReplyTo":"d81e7c10997e9e8dc211d241019fbafa6b25fb04.1659577543.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/10] builtin/bugreport.c: create '--diagnose' option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-05T19:35:24Z","receivedAt":"2022-08-05T19:35:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2022 9:45 PM, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n\n> +--diagnose[=(basic|all)]::\n> +\tCreate a zip archive of information about the repository including logs\n\nlogs? I think the reflogs are not included unless \"all\" is specified. Perhaps\nwe can unify this description with the beginning of git-diagnose.txt:\n\n Collects detailed information about the user's machine, Git client, and\n repository state and packages that information into a zip archive.\n\nresulting in\n\n\tCreate a zip archive containing information about the user's machine,\n\tGit client, and repository state.\n\n> +\tand certain statistics describing the data shape of the repository. The\n> +\tarchive is written to the same output directory as the bug report and is\n> +\tnamed 'git-diagnostics-'.\n> ++\n> +By default, `--diagnose` (equivalent to `--diagnose=basic`) will collect only\n> +statistics and summarized data about the repository and filesystem. Specifying\n> +`--diagnose=all` will create an archive with the same contents generated by `git\n> +diagnose --all`; this archive will be much larger, and will contain potentially\n> +sensitive information about the repository. See linkgit:git-diagnose[1] for more\n> +details on the contents of the diagnostic archive.\n\nPerhaps here (and git-diagnose.txt) should be really explicit about sharing the\n\"all\" mode output only with trusted parties. Let the user decide what level of\ntrust is necessary depending on their situation (we don't need to say \"open source\nrepos are fine to share\" or something).\n\n> +enum diagnose_mode {\n> +\tDIAGNOSE_NONE,\n> +\tDIAGNOSE_BASIC,\n> +\tDIAGNOSE_ALL\n> +};\n\nThis enum makes me think that it might be nice to use this in diagnose.h\nalong with an array that pairs strings with the enum. We could unify the\noptions by having 'git diagnose --mode=(basic|all)' which could be\nextended in the future with another mode that might be in between the two.\n\nIt may also be a waste of time to set up that infrastructure without it\nactually mattering in the future, but I thought I'd mention it as an\nalternative, in case that inspires you.\n\n> static void get_system_info(struct strbuf *sys_info)\n> {\n> @@ -91,6 +97,23 @@ static void get_header(struct strbuf *buf, const char *title)\n> \tstrbuf_addf(buf, \"\\n\\n[%s]\\n\", title);\n> }\n> \n> +static int option_parse_diagnose(const struct option *opt,\n> +\t\t\t\t const char *arg, int unset)\n> +{\n> +\tenum diagnose_mode *diagnose = opt->value;\n> +\n> +\tBUG_ON_OPT_NEG(unset);\n> +\n> +\tif (!arg || !strcmp(arg, \"basic\"))\n> +\t\t*diagnose = DIAGNOSE_BASIC;\n> +\telse if (!strcmp(arg, \"all\"))\n> +\t\t*diagnose = DIAGNOSE_ALL;\n\nShould we allow \"none\" to reset the value to DIAGNOSE_NONE?\n\n> +\telse\n> +\t\tdie(_(\"diagnose mode must be either 'basic' or 'all'\"));\n\nI wondered initially if this should be a usage() call instead. But we have\nplenty of examples of using die() to report an issue with a single option\nor a combination of options.\n\n> \tconst struct option bugreport_options[] = {\n> +\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"(basic|all)\"),\n> +\t\t\t N_(\"create an additional zip archive of detailed diagnostics\"),\n> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_OPTARG, option_parse_diagnose),\n\nThe biggest reason for this to be an OPT_CALLBACK_F is because of the\n'--diagnose' option (without '='), so an OPT_STRING would not be\nappropriate here.\n\n> @@ -119,6 +147,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n> \t\t\t\t\t option_output ? option_output : \"\");\n> \tstrbuf_addstr(&report_path, prefixed_filename);\n> \tstrbuf_complete(&report_path, '/');\n> +\toutput_path_len = report_path.len;\n\nPerhaps this should be renamed to output_dir_len, since we know this is\na directory that will contain all of the output files.\n\n> +\t/* Prepare diagnostics, if requested */\n> +\tif (diagnose != DIAGNOSE_NONE) {\n> +\t\tstruct strbuf zip_path = STRBUF_INIT;\n> +\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n> +\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n> +\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n> +\t\tstrbuf_addstr(&zip_path, \".zip\");\n> +\n> +\t\tif (create_diagnostics_archive(&zip_path, diagnose == DIAGNOSE_ALL))\n\n(Just pausing to say this could be create_diagnostics_archive(&zip_path, diagnose)\nif we use the enum inside diagnose.c.\n\nThanks,\n-Stolee\n"},{"id":"460739","messageId":"41f85d57-596a-5e42-756c-93fe0f45185f@github.com","threadId":"58251","inReplyTo":"220804.86zggkecmk.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 06/10] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-05T19:38:19Z","receivedAt":"2022-08-05T19:38:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/4/2022 2:27 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> From: Victoria Dye \n>>\n>> Create a 'git diagnose' builtin to generate a standalone zip archive of\n>> repository diagnostics.\n> \n> It's good to have this as a built-in separate from \"git bugreport\",\n> but...\n> \n>> +git-diagnose - Generate a zip archive of diagnostic information\n> \n> ...I'd really prefer for this not to squat on such a common name we\n> might regret having reserved later for such very specific\n> functionality. I'd think e.g. these would be better:\n> \n> \tgit mk-diagnostics-zip\n> \n> Or maybe:\n> \n> \tgit archive-interesting-for-report\n\nThese are not realistic replacements.\n\n> If I had to guess what a \"git diagnose\" did, I'd probably think:\n> \n> * It analyzes your config, and suggests redundancies/alternatives\n> * It does some perf tests / heuritics, and e.g. suggests you turn on\n> the commit-graph writing.\n\nThese sound like great options to add in the future, such as:\n\n --perf-test: Run performance tests on your repository using different\n Git config options and recommend certain settings.\n\n(This --perf-test option would be a great way to get wider adoption\nof parallel checkout, since its optimal settings are so machine\ndependent.)\n\nThe thing is, even if we did these other things, it would result in\nsome kind of document that summarizes the repository shape and features.\nThat kind of data is exactly what this version of 'git diagnose' does.\n\nFor now, it leaves the human reader responsible for making decisions\nbased on those documents, but they have been incredibly helpful when we\nare _diagnosing_ issues users are having with their repositories.\n\nThanks,\n-Stolee\n"},{"id":"460921","messageId":"82af0381-ecdc-adf4-1339-ec35c18118a1@github.com","threadId":"58251","inReplyTo":"xmqqwnbn6i0v.fsf@gitster.g","subject":"Re: [PATCH v2 00/10] Generalize 'scalar diagnose' into 'git diagnose' and 'git bugreport --diagnose'","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-09T16:17:18Z","receivedAt":"2022-08-09T16:17:25Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> * (Almost) entirely redesigned the UI for generating diagnostics. The new\n>> approach avoids cluttering 'git bugreport' with a mode that doesn't\n>> actually generate a report. Now, there are distinct options for different\n>> use cases: generating extra diagnostics with a bug report ('git bugreport\n>> --diagnose') and generating diagnostics for personal debugging/addition\n>> to an existing bug report ('git diagnose').\n> \n> An additional command gives us far more design flexibility, and in\n> this case I think it may be worth it. It has a risk of confusing\n> users between \"git bugreport --diag\" and \"git diagnose --report\" a\n> way to send a report with diagnostic information, though.\n\nThis is an interesting point and something I think users could plausibly run\ninto. I can think of a few of ways to address that:\n\n1. Do nothing, wait for feedback from users.\n2. Create hidden option(s) '--report' and/or '--bugreport' in 'git diagnose'\n that trigger a warning (or advice?) along the lines of \"did you mean 'git\n bugreport --diagnose'?\" and exit with 'usage()'.\n3. Create visible options '--report' and/or '--bugreport' in 'git diagnose'\n that invoke 'git bugreport --diagnose'.\n\nI'm leaning towards option 2, but I'd also understand not wanting to clutter\nbuiltins with options it *doesn't* use for the sake of advising a user.\n\n> \n> \n\n"},{"id":"460924","messageId":"xmqqmtcdqs2y.fsf@gitster.g","threadId":"58251","inReplyTo":"82af0381-ecdc-adf4-1339-ec35c18118a1@github.com","subject":"Re: [PATCH v2 00/10] Generalize 'scalar diagnose' into 'git diagnose' and 'git bugreport --diagnose'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-09T16:50:29Z","receivedAt":"2022-08-09T16:50:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> This is an interesting point and something I think users could plausibly run\n> into. I can think of a few of ways to address that:\n>\n> 1. Do nothing, wait for feedback from users.\n> 2. Create hidden option(s) '--report' and/or '--bugreport' in 'git diagnose'\n> that trigger a warning (or advice?) along the lines of \"did you mean 'git\n> bugreport --diagnose'?\" and exit with 'usage()'.\n> 3. Create visible options '--report' and/or '--bugreport' in 'git diagnose'\n> that invoke 'git bugreport --diagnose'.\n>\n> I'm leaning towards option 2, but I'd also understand not wanting to clutter\n> builtins with options it *doesn't* use for the sake of advising a user.\n\nI was leaning towards option 1, actually ;-)\n\nI'm offline today, so please expect no changes to the public\nrepositories.\n\nThanks.\n"},{"id":"460925","messageId":"73f168c4-543c-9ff3-5bf2-9a4c789d7fad@github.com","threadId":"58251","inReplyTo":"220804.86r11webp8.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 09/10] scalar-diagnose: use 'git diagnose --all'","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-09T16:54:24Z","receivedAt":"2022-08-09T16:54:30Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> From: Victoria Dye \n>>\n>> Replace implementation of 'scalar diagnose' with an internal invocation of\n>> 'git diagnose --all'. This simplifies the implementation of 'cmd_diagnose'\n>> by making it a direct alias of 'git diagnose' and removes some code in\n>> 'scalar.c' that is duplicated in 'builtin/diagnose.c'. The simplicity of the\n>> alias also sets up a clean deprecation path for 'scalar diagnose' (in favor\n>> of 'git diagnose'), if that is desired in the future.\n>>\n>> This introduces one minor change to the output of 'scalar diagnose', which\n>> is that the prefix of the created zip archive is changed from 'scalar_' to\n>> 'git-diagnostics-'.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> contrib/scalar/scalar.c | 29 +++++++----------------------\n>> 1 file changed, 7 insertions(+), 22 deletions(-)\n>>\n>> diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\n>> index b10955531ce..fe2a0e9decb 100644\n>> --- a/contrib/scalar/scalar.c\n>> +++ b/contrib/scalar/scalar.c\n>> @@ -11,7 +11,6 @@\n>> #include \"dir.h\"\n>> #include \"packfile.h\"\n>> #include \"help.h\"\n>> -#include \"diagnose.h\"\n>> \n>> /*\n>> * Remove the deepest subdirectory in the provided path string. Path must not\n>> @@ -510,34 +509,20 @@ static int cmd_diagnose(int argc, const char **argv)\n>> \t\tN_(\"scalar diagnose []\"),\n>> \t\tNULL\n>> \t};\n>> -\tstruct strbuf zip_path = STRBUF_INIT;\n>> -\ttime_t now = time(NULL);\n>> -\tstruct tm tm;\n>> +\tstruct strbuf diagnostics_root = STRBUF_INIT;\n>> \tint res = 0;\n>> \n>> \targc = parse_options(argc, argv, NULL, options,\n>> \t\t\t usage, 0);\n>> \n>> -\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n>> -\n>> -\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n>> -\tstrbuf_addftime(&zip_path,\n>> -\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n>> -\tstrbuf_addstr(&zip_path, \".zip\");\n>> -\tswitch (safe_create_leading_directories(zip_path.buf)) {\n>> -\tcase SCLD_EXISTS:\n>> -\tcase SCLD_OK:\n>> -\t\tbreak;\n>> -\tdefault:\n>> -\t\terror_errno(_(\"could not create directory for '%s'\"),\n>> -\t\t\t zip_path.buf);\n>> -\t\tgoto diagnose_cleanup;\n> \n> Just spotting this now, but we had ad error, but we \"goto\n> diagnose_cleanup\", but that will use our \"res = 0\" above.\n> \n> Is this untested already or in this series (didn't go back to look). But\n> maybe a moot point, the post-image replacement uses die()..\n\nNice catch - this does appear to be a pre-existing bug in 'scalar diagnose'.\nGiven that both 'git diagnose' and 'git bugreport --diagnose' handle this\ncase more appropriately, though, I agree that it's a bit of a moot point and\nnot worth the churn created by a bugfix patch.\n\n> \n>> -\t}\n>> +\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_root);\n>> +\tstrbuf_addstr(&diagnostics_root, \"/.scalarDiagnostics\");\n>> \n>> -\tres = create_diagnostics_archive(&zip_path, 1);\n>> +\tif (run_git(\"diagnose\", \"--all\", \"-s\", \"%Y%m%d_%H%M%S\",\n>> +\t\t \"-o\", diagnostics_root.buf, NULL) < 0)\n>> +\t\tres = -1;\n> \n> The code handling here seems really odd, issues:\n> \n> * This *can* return -1, if start_command() fails, but that's by far the\n> rarer case, usually it would be 0 or >0 (only <0 if we can't start\n> the command at all).\n> \n> * You should not be returning -1 from cmd_*() in general (we have\n> outstanding issues with it, but those should be fixed). It will yield\n> an exit code of 255 (but it's not portable)).\n> \n> * If you're going to return -1 at all, why override <0 with -1, just\n> \"res = run_git(...)\" instead?\n\nThanks for the info, I'll replace the hardcoded '-1' return value with\nsomething derived from 'res' in the next version.\n\n> \n> I think all-in-all this should be:\n> \n> \tres = run_git(...);\n> \n> Then:\n> \n>> \n>> -diagnose_cleanup:\n>> -\tstrbuf_release(&zip_path);\n>> +\tstrbuf_release(&diagnostics_root);\n>> \treturn res;\n> \n> \treturn res < 0 ? -res : res;\n> \n> Or whatever.\n\n"},{"id":"460954","messageId":"a45f5693-7186-2953-6620-3f1359a12238@github.com","threadId":"58251","inReplyTo":"3dc402e1-1f27-8a24-544d-d90d403a7da0@github.com","subject":"Re: [PATCH v2 08/10] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-09T23:53:16Z","receivedAt":"2022-08-09T23:53:47Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee wrote:\n> On 8/3/2022 9:45 PM, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye \n> \n>> +--diagnose[=(basic|all)]::\n>> +\tCreate a zip archive of information about the repository including logs\n> \n> logs? I think the reflogs are not included unless \"all\" is specified. Perhaps\n> we can unify this description with the beginning of git-diagnose.txt:\n> \n> Collects detailed information about the user's machine, Git client, and\n> repository state and packages that information into a zip archive.\n> \n> resulting in\n> \n> \tCreate a zip archive containing information about the user's machine,\n> \tGit client, and repository state.\n> \n\n[...]\n\n>> +\tand certain statistics describing the data shape of the repository. The\n>> +\tarchive is written to the same output directory as the bug report and is\n>> +\tnamed 'git-diagnostics-'.\n>> ++\n>> +By default, `--diagnose` (equivalent to `--diagnose=basic`) will collect only\n>> +statistics and summarized data about the repository and filesystem. Specifying\n>> +`--diagnose=all` will create an archive with the same contents generated by `git\n>> +diagnose --all`; this archive will be much larger, and will contain potentially\n>> +sensitive information about the repository. See linkgit:git-diagnose[1] for more\n>> +details on the contents of the diagnostic archive.\n> \n> Perhaps here (and git-diagnose.txt) should be really explicit about sharing the\n> \"all\" mode output only with trusted parties. Let the user decide what level of\n> trust is necessary depending on their situation (we don't need to say \"open source\n> repos are fine to share\" or something).\n\nBoth of these documentation suggestions make sense to me; I'll update\naccordingly in V3.\n\n> \n>> +enum diagnose_mode {\n>> +\tDIAGNOSE_NONE,\n>> +\tDIAGNOSE_BASIC,\n>> +\tDIAGNOSE_ALL\n>> +};\n> \n> This enum makes me think that it might be nice to use this in diagnose.h\n> along with an array that pairs strings with the enum. We could unify the\n> options by having 'git diagnose --mode=(basic|all)' which could be\n> extended in the future with another mode that might be in between the two.\n> \n> It may also be a waste of time to set up that infrastructure without it\n> actually mattering in the future, but I thought I'd mention it as an\n> alternative, in case that inspires you.\n\nYour suggestion is more extensible than the boolean \"include_everything\" I'm\nusing right now in 'create_diagnostics_archive()'. I'll incorporate it into\nmy next re-roll, thanks!\n\n> \n>> static void get_system_info(struct strbuf *sys_info)\n>> {\n>> @@ -91,6 +97,23 @@ static void get_header(struct strbuf *buf, const char *title)\n>> \tstrbuf_addf(buf, \"\\n\\n[%s]\\n\", title);\n>> }\n>> \n>> +static int option_parse_diagnose(const struct option *opt,\n>> +\t\t\t\t const char *arg, int unset)\n>> +{\n>> +\tenum diagnose_mode *diagnose = opt->value;\n>> +\n>> +\tBUG_ON_OPT_NEG(unset);\n>> +\n>> +\tif (!arg || !strcmp(arg, \"basic\"))\n>> +\t\t*diagnose = DIAGNOSE_BASIC;\n>> +\telse if (!strcmp(arg, \"all\"))\n>> +\t\t*diagnose = DIAGNOSE_ALL;\n> \n> Should we allow \"none\" to reset the value to DIAGNOSE_NONE?\n\nAs far as I can tell, while some builtins have options that match the\ndefault behavior of the command (e.g., '--no-autosquash' in 'git rebase'),\nthose options typically exist to override a config setting (e.g.,\n'rebase.autosquash'). No config exists for 'bugreport --diagnose' (and I\ndon't think it would make sense to add one), so '--diagnose=none' would only\nbe used to override another '--diagnose' specification in the same\ncommand/alias (e.g., 'git bugreport --diagnose=basic --diagnose=none'). \n\nThat use case seems unlikely, but if there's precedent or use cases I'm not\naccounting for, I'm happy to add the option.\n\n> \n>> +\telse\n>> +\t\tdie(_(\"diagnose mode must be either 'basic' or 'all'\"));\n> \n> I wondered initially if this should be a usage() call instead. But we have\n> plenty of examples of using die() to report an issue with a single option\n> or a combination of options.\n\nAfter looking at other 'OPT_CALLBACK_F' examples (e.g., the 'diff_opt_stat'\noptions in 'diff.c'), I think this should at least change to 'return\nerror()' to get a more appropriate return code (129):\n\n\telse\n\t\treturn error(_(\"invalid --%s mode '%s'\"), opt->long_name, arg);\n\nI also looked into using 'usage_msg_opt()' to print both an error message\nand the usage string, but doing so would require making the 'bugreport'\noptions structure & variables static so they're accessible by\n'option_parse_diagnose()'. If you think it would be valuable to include that\ninformation, I'll add an extra refactoring patch to include it.\n\n> \n>> \tconst struct option bugreport_options[] = {\n>> +\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"(basic|all)\"),\n>> +\t\t\t N_(\"create an additional zip archive of detailed diagnostics\"),\n>> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_OPTARG, option_parse_diagnose),\n> \n> The biggest reason for this to be an OPT_CALLBACK_F is because of the\n> '--diagnose' option (without '='), so an OPT_STRING would not be\n> appropriate here.\n> \n>> @@ -119,6 +147,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n>> \t\t\t\t\t option_output ? option_output : \"\");\n>> \tstrbuf_addstr(&report_path, prefixed_filename);\n>> \tstrbuf_complete(&report_path, '/');\n>> +\toutput_path_len = report_path.len;\n> \n> Perhaps this should be renamed to output_dir_len, since we know this is\n> a directory that will contain all of the output files.\n\nAgreed, 'path' is a bit ambiguous and 'dir' more clearly refers to the\ndirectory containing the output files. \n\n> \n>> +\t/* Prepare diagnostics, if requested */\n>> +\tif (diagnose != DIAGNOSE_NONE) {\n>> +\t\tstruct strbuf zip_path = STRBUF_INIT;\n>> +\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n>> +\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n>> +\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n>> +\t\tstrbuf_addstr(&zip_path, \".zip\");\n>> +\n>> +\t\tif (create_diagnostics_archive(&zip_path, diagnose == DIAGNOSE_ALL))\n> \n> (Just pausing to say this could be create_diagnostics_archive(&zip_path, diagnose)\n> if we use the enum inside diagnose.c.\n> \n> Thanks,\n> -Stolee\n\n"},{"id":"460979","messageId":"77f445f7-3934-c165-3160-e09d7b884774@github.com","threadId":"58251","inReplyTo":"a45f5693-7186-2953-6620-3f1359a12238@github.com","subject":"Re: [PATCH v2 08/10] builtin/bugreport.c: create '--diagnose' option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-10T12:52:23Z","receivedAt":"2022-08-10T12:52:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/9/22 7:53 PM, Victoria Dye wrote:\n> Derrick Stolee wrote:\n>> On 8/3/2022 9:45 PM, Victoria Dye via GitGitGadget wrote:\n\n>>> +static int option_parse_diagnose(const struct option *opt,\n>>> +\t\t\t\t const char *arg, int unset)\n>>> +{\n>>> +\tenum diagnose_mode *diagnose = opt->value;\n>>> +\n>>> +\tBUG_ON_OPT_NEG(unset);\n>>> +\n>>> +\tif (!arg || !strcmp(arg, \"basic\"))\n>>> +\t\t*diagnose = DIAGNOSE_BASIC;\n>>> +\telse if (!strcmp(arg, \"all\"))\n>>> +\t\t*diagnose = DIAGNOSE_ALL;\n>>\n>> Should we allow \"none\" to reset the value to DIAGNOSE_NONE?\n> \n> As far as I can tell, while some builtins have options that match the\n> default behavior of the command (e.g., '--no-autosquash' in 'git rebase'),\n> those options typically exist to override a config setting (e.g.,\n> 'rebase.autosquash'). No config exists for 'bugreport --diagnose' (and I\n> don't think it would make sense to add one), so '--diagnose=none' would only\n> be used to override another '--diagnose' specification in the same\n> command/alias (e.g., 'git bugreport --diagnose=basic --diagnose=none'). \n\nAh, so --diagnose=none isn't valuable because --no-diagnose would be\nthe better way to write the same thing. You would need to remove the\nPARSE_OPT_NONEG from your OPT_CALLBACK_F() to allow that (and then do\nthe appropriate logic with the \"unset\" parameter).\n\nThe reason to have these things is basically so users can create\naliases (say 'git br' expands to 'git bugreport --diagnose=all', but\nthey want to run 'git br --no-diagnose' to clear that --diagnose=all).\n\nThanks,\n-Stolee\n"},{"id":"461002","messageId":"19fe9b29-a860-fb59-ecb4-0d72b61167fe@github.com","threadId":"58251","inReplyTo":"77f445f7-3934-c165-3160-e09d7b884774@github.com","subject":"Re: [PATCH v2 08/10] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-10T16:13:43Z","receivedAt":"2022-08-10T16:13:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee wrote:\n> On 8/9/22 7:53 PM, Victoria Dye wrote:\n>> Derrick Stolee wrote:\n>>> On 8/3/2022 9:45 PM, Victoria Dye via GitGitGadget wrote:\n> \n>>>> +static int option_parse_diagnose(const struct option *opt,\n>>>> +\t\t\t\t const char *arg, int unset)\n>>>> +{\n>>>> +\tenum diagnose_mode *diagnose = opt->value;\n>>>> +\n>>>> +\tBUG_ON_OPT_NEG(unset);\n>>>> +\n>>>> +\tif (!arg || !strcmp(arg, \"basic\"))\n>>>> +\t\t*diagnose = DIAGNOSE_BASIC;\n>>>> +\telse if (!strcmp(arg, \"all\"))\n>>>> +\t\t*diagnose = DIAGNOSE_ALL;\n>>>\n>>> Should we allow \"none\" to reset the value to DIAGNOSE_NONE?\n>>\n>> As far as I can tell, while some builtins have options that match the\n>> default behavior of the command (e.g., '--no-autosquash' in 'git rebase'),\n>> those options typically exist to override a config setting (e.g.,\n>> 'rebase.autosquash'). No config exists for 'bugreport --diagnose' (and I\n>> don't think it would make sense to add one), so '--diagnose=none' would only\n>> be used to override another '--diagnose' specification in the same\n>> command/alias (e.g., 'git bugreport --diagnose=basic --diagnose=none'). \n> \n> Ah, so --diagnose=none isn't valuable because --no-diagnose would be\n> the better way to write the same thing. You would need to remove the\n> PARSE_OPT_NONEG from your OPT_CALLBACK_F() to allow that (and then do\n> the appropriate logic with the \"unset\" parameter).\n\nI'm not sure I follow. I wasn't suggesting a difference in value between\n'--no-diagnose' and '--diagnose=none'. My point was that, when there's an\noption variant that \"resets\" the value to the default (like\n'--no-autosquash', '--no-recurse-submodules', etc.), it usually *also*\ncorresponds to an overridable config setting ('rebase.autosquash',\n'push.recurseSubmodules'). No such 'bugreport.diagnose' config exists (or,\nIMO, should exist), so the need for a \"reset to default\" option seemed\nweaker. \n\nI used boolean options as my examples, but they aren't intended to imply a\nmeaningful difference between '--no-diagnose' amd '--diagnose=none'.\n\n> \n> The reason to have these things is basically so users can create\n> aliases (say 'git br' expands to 'git bugreport --diagnose=all', but\n> they want to run 'git br --no-diagnose' to clear that --diagnose=all).\n\nI considered that usage (\"'--diagnose=none' would only be used to override\nanother '--diagnose' specification in the same command/alias\"), but wasn't\nsure how common it would be for this particular option. It sounds like you\ncan see it being useful, so I'll include '--diagnose=none' in the next\nversion.\n\n> \n> Thanks,\n> -Stolee\n\n"},{"id":"461003","messageId":"87b19226-2542-2ea0-e16a-ce8e57d6b589@github.com","threadId":"58251","inReplyTo":"19fe9b29-a860-fb59-ecb4-0d72b61167fe@github.com","subject":"Re: [PATCH v2 08/10] builtin/bugreport.c: create '--diagnose' option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-10T16:47:31Z","receivedAt":"2022-08-10T16:47:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/10/2022 12:13 PM, Victoria Dye wrote:\n> Derrick Stolee wrote:\n>> On 8/9/22 7:53 PM, Victoria Dye wrote:\n>>> Derrick Stolee wrote:\n>>>> On 8/3/2022 9:45 PM, Victoria Dye via GitGitGadget wrote:\n>>\n>>>>> +static int option_parse_diagnose(const struct option *opt,\n>>>>> +\t\t\t\t const char *arg, int unset)\n>>>>> +{\n>>>>> +\tenum diagnose_mode *diagnose = opt->value;\n>>>>> +\n>>>>> +\tBUG_ON_OPT_NEG(unset);\n>>>>> +\n>>>>> +\tif (!arg || !strcmp(arg, \"basic\"))\n>>>>> +\t\t*diagnose = DIAGNOSE_BASIC;\n>>>>> +\telse if (!strcmp(arg, \"all\"))\n>>>>> +\t\t*diagnose = DIAGNOSE_ALL;\n>>>>\n>>>> Should we allow \"none\" to reset the value to DIAGNOSE_NONE?\n>>>\n>>> As far as I can tell, while some builtins have options that match the\n>>> default behavior of the command (e.g., '--no-autosquash' in 'git rebase'),\n>>> those options typically exist to override a config setting (e.g.,\n>>> 'rebase.autosquash'). No config exists for 'bugreport --diagnose' (and I\n>>> don't think it would make sense to add one), so '--diagnose=none' would only\n>>> be used to override another '--diagnose' specification in the same\n>>> command/alias (e.g., 'git bugreport --diagnose=basic --diagnose=none'). \n>>\n>> Ah, so --diagnose=none isn't valuable because --no-diagnose would be\n>> the better way to write the same thing. You would need to remove the\n>> PARSE_OPT_NONEG from your OPT_CALLBACK_F() to allow that (and then do\n>> the appropriate logic with the \"unset\" parameter).\n> \n> I'm not sure I follow. I wasn't suggesting a difference in value between\n> '--no-diagnose' and '--diagnose=none'. My point was that, when there's an\n> option variant that \"resets\" the value to the default (like\n> '--no-autosquash', '--no-recurse-submodules', etc.), it usually *also*\n> corresponds to an overridable config setting ('rebase.autosquash',\n> 'push.recurseSubmodules'). No such 'bugreport.diagnose' config exists (or,\n> IMO, should exist), so the need for a \"reset to default\" option seemed\n> weaker. \n\nYou're right that these make the most sense when there can be a\nnon-CLI source of a setting to unset, which is a better reason than\nthe alias reason I gave.\n \n> I used boolean options as my examples, but they aren't intended to imply a\n> meaningful difference between '--no-diagnose' amd '--diagnose=none'.\n> \n>>\n>> The reason to have these things is basically so users can create\n>> aliases (say 'git br' expands to 'git bugreport --diagnose=all', but\n>> they want to run 'git br --no-diagnose' to clear that --diagnose=all).\n> \n> I considered that usage (\"'--diagnose=none' would only be used to override\n> another '--diagnose' specification in the same command/alias\"), but wasn't\n> sure how common it would be for this particular option. It sounds like you\n> can see it being useful, so I'll include '--diagnose=none' in the next\n> version.\n\nThis change would make it easier to add a config option in the future,\nthough I doubt we will need it as 'git bugreport' should be used too\ninfrequently to want to set up such config in advance.\n\nThanks,\n-Stolee\n"},{"id":"461044","messageId":"f5ceb9c71903c41c47d715ec75ced9375c643ac3.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 01/11] scalar-diagnose: use \"$GIT_UNZIP\" in test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:23Z","receivedAt":"2022-08-10T23:34:41Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUse the \"$GIT_UNZIP\" test variable rather than verbatim 'unzip' to unzip the\n'scalar diagnose' archive. Using \"$GIT_UNZIP\" is needed to run the Scalar\ntests on systems where 'unzip' is not in the system path.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/t/t9099-scalar.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh\nindex 10b1172a8aa..fac86a57550 100755\n--- a/contrib/scalar/t/t9099-scalar.sh\n+++ b/contrib/scalar/t/t9099-scalar.sh\n@@ -109,14 +109,14 @@ test_expect_success UNZIP 'scalar diagnose' '\n \tsed -n \"s/.*$SQ\\\\(.*\\\\.zip\\\\)$SQ.*/\\\\1/p\" zip_path &&\n \tzip_path=$(cat zip_path) &&\n \ttest -n \"$zip_path\" &&\n-\tunzip -v \"$zip_path\" &&\n+\t\"$GIT_UNZIP\" -v \"$zip_path\" &&\n \tfolder=${zip_path%.zip} &&\n \ttest_path_is_missing \"$folder\" &&\n-\tunzip -p \"$zip_path\" diagnostics.log >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n \ttest_file_not_empty out &&\n-\tunzip -p \"$zip_path\" packs-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n \tgrep \"$(pwd)/.git/objects\" out &&\n-\tunzip -p \"$zip_path\" objects-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n \tgrep \"^Total: [1-9]\" out\n '\n \n-- \ngitgitgadget\n\n"},{"id":"461045","messageId":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v2.git.1659577543.gitgitgadget@gmail.com","subject":"[PATCH v3 00/11] Generalize 'scalar diagnose' into 'git diagnose' and 'git bugreport --diagnose'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:22Z","receivedAt":"2022-08-10T23:34:42Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"As part of the preparation for moving Scalar out of 'contrib/' and into Git,\nthis series moves the functionality of 'scalar diagnose' into a new builtin\n('git diagnose') and a new option ('--diagnose') for 'git bugreport'. This\nchange further aligns Scalar with the objective [1] of having it only\ncontain functionality and settings that benefit large Git repositories, but\nnot all repositories. The diagnostics reported by 'scalar diagnose' relevant\nfor investigating issues in any Git repository, so generating them should be\npart of a \"normal\" Git builtin.\n\nThe series is organized as follows:\n\n * Miscellaneous fixes for the existing 'scalar diagnose' implementation\n * Moving the code for generating diagnostics into a common location in the\n Git tree\n * Implementing 'git diagnose'\n * Implementing 'git bugreport --diagnose'\n * Updating the Scalar roadmap\n\nFinally, despite 'scalar diagnose' now being nothing more than a wrapper for\n'git bugreport --diagnose', it is not being deprecated in this series.\nAlthough deprecation -> removal could be a future cleanup effort, 'scalar\ndiagnose' is kept around for now as an alias for users already accustomed to\nusing it in 'scalar'.\n\n\nChanges since V2\n================\n\n * Replaced 'int include_everything' arg to 'create_diagnostic_archive()'\n with 'enum diagnose_mode mode'.\n * Replaced '--all' with configurable '--mode' option in 'git diagnose';\n moved 'option_parse_diagnose()' into 'diagnose.c' so that it can be used\n for both 'git bugreport --diagnose' and 'git diagnose --mode'.\n * Split \"builtin/diagnose.c: gate certain data behind '--all'\" (formerly\n patch 7/10) into \"diagnose.c: add option to configure archive contents\"\n (patch 6/11) and \"builtin/diagnose.c: add '--mode' option\" (patch 8/11).\n * Added '--no-diagnose' for 'git bugreport'. I was initially going to use\n '--diagnose=none', but '--no-diagnose' was easier to configure when using\n the shared 'option_parse_diagnose()' function .\n * Updated usage strings, option descriptions, and documentation files for\n 'mode' option. To avoid needing to keep multiple lists of valid 'mode'\n values up-to-date, format mode value as everywhere except option\n description in 'git-diagnose.txt', where the values are listed. The\n documentation of '--diagnose' in 'git-bugreport.txt' links to\n 'git-diagnose.txt' and explicitly calls out that details on 'mode' can be\n found there.\n * Reworded 'git diagnose' and 'git bugreport' command & option\n documentation.\n * Added additional checks to 't0091-bugreport.sh' and 't0092-diagnose.sh'\n tests\n * Moved '#include \"cache.h\" from 'diagnose.h' to 'diagnose.c'.\n * Fixed '--output-directory' usage string in 'builtin/diagnose.c'.\n * Replaced 'die()' with 'die_errno()' in error triggered when leading\n directories of archive cannot be created.\n * Changed hardcoded '-1' error exit code in 'scalar diagnose' to returning\n the exit code from 'git diagnose --mode=all'.\n\n\nChanges since V1\n================\n\n * Reorganized patches to fix minor issues (e.g., more gently adding\n directories to the archive) of 'scalar diagnose' in 'scalar.c', before\n the code is moved out of that file.\n * (Almost) entirely redesigned the UI for generating diagnostics. The new\n approach avoids cluttering 'git bugreport' with a mode that doesn't\n actually generate a report. Now, there are distinct options for different\n use cases: generating extra diagnostics with a bug report ('git bugreport\n --diagnose') and generating diagnostics for personal debugging/addition\n to an existing bug report ('git diagnose').\n * Moved 'get_disk_info()' into 'compat/'.\n * Moved 'create_diagnostics_archive()' into a new 'diagnose.c', as it now\n has multiple callers.\n * Updated command & option documentation to more clearly guide users on how\n to use the new options.\n * Added the '--all' (and '--diagnose=all') option to change the default\n behavior of diagnostics generation to exclude '.git' directory contents.\n For many bug reporters, this would reveal private repository contents\n they don't want to expose to the public mailing list. This has the added\n benefit of creating much smaller archives by default, which will be more\n likely to successfully send to the mailing list.\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1275.v2.git.1657584367.gitgitgadget@gmail.com/\n\nVictoria Dye (11):\n scalar-diagnose: use \"$GIT_UNZIP\" in test\n scalar-diagnose: avoid 32-bit overflow of size_t\n scalar-diagnose: add directory to archiver more gently\n scalar-diagnose: move 'get_disk_info()' to 'compat/'\n scalar-diagnose: move functionality to common location\n diagnose.c: add option to configure archive contents\n builtin/diagnose.c: create 'git diagnose' builtin\n builtin/diagnose.c: add '--mode' option\n builtin/bugreport.c: create '--diagnose' option\n scalar-diagnose: use 'git diagnose --mode=all'\n scalar: update technical doc roadmap\n\n .gitignore | 1 +\n Documentation/git-bugreport.txt | 18 ++\n Documentation/git-diagnose.txt | 65 +++++++\n Documentation/technical/scalar.txt | 9 +-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/bugreport.c | 27 ++-\n builtin/diagnose.c | 61 +++++++\n compat/disk.h | 56 ++++++\n contrib/scalar/scalar.c | 271 +----------------------------\n contrib/scalar/t/t9099-scalar.sh | 8 +-\n diagnose.c | 254 +++++++++++++++++++++++++++\n diagnose.h | 17 ++\n git-compat-util.h | 1 +\n git.c | 1 +\n t/t0091-bugreport.sh | 48 +++++\n t/t0092-diagnose.sh | 60 +++++++\n 17 files changed, 622 insertions(+), 278 deletions(-)\n create mode 100644 Documentation/git-diagnose.txt\n create mode 100644 builtin/diagnose.c\n create mode 100644 compat/disk.h\n create mode 100644 diagnose.c\n create mode 100644 diagnose.h\n create mode 100755 t/t0092-diagnose.sh\n\n\nbase-commit: 4af7188bc97f70277d0f10d56d5373022b1fa385\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1310%2Fvdye%2Fscalar%2Fgeneralize-diagnose-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1310/vdye/scalar/generalize-diagnose-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1310\n\nRange-diff vs v2:\n\n 1: ad5b60bf11e = 1: f5ceb9c7190 scalar-diagnose: use \"$GIT_UNZIP\" in test\n 2: 7956dc24b30 = 2: 78a93eb95bb scalar-diagnose: avoid 32-bit overflow of size_t\n 3: 23349bfaf8f = 3: 22ee8ea5a1e scalar-diagnose: add directory to archiver more gently\n 4: 05bba1e699f = 4: 18f2ba4e0cd scalar-diagnose: move 'get_disk_info()' to 'compat/'\n 5: 3a0cb33c658 ! 5: 7a51fad87a8 scalar-diagnose: move functionality to common location\n @@ Commit message\n primary changes being that 'zip_path' is an input and \"Enlistment root\" is\n corrected to \"Repository root\" in the archiver log.\n \n + Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Victoria Dye \n \n ## Makefile ##\n @@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n \n ## diagnose.c (new) ##\n @@\n ++#include \"cache.h\"\n +#include \"diagnose.h\"\n +#include \"compat/disk.h\"\n +#include \"archive.h\"\n @@ diagnose.h (new)\n +#ifndef DIAGNOSE_H\n +#define DIAGNOSE_H\n +\n -+#include \"cache.h\"\n +#include \"strbuf.h\"\n +\n +int create_diagnostics_archive(struct strbuf *zip_path);\n -: ----------- > 6: 0a6c55696d8 diagnose.c: add option to configure archive contents\n 6: 73e139ee377 ! 7: bf3c073a985 builtin/diagnose.c: create 'git diagnose' builtin\n @@ Commit message\n diagnostics gathered are not specific to Scalar-cloned repositories and\n can be useful when diagnosing issues in any Git repository.\n \n + Helped-by: Ævar Arnfjörð Bjarmason \n + Helped-by: Derrick Stolee \n Signed-off-by: Victoria Dye \n \n ## .gitignore ##\n @@ Documentation/git-diagnose.txt (new)\n + stores\n + * The total count of loose objects, as well as counts broken down by\n + `.git/objects` subdirectory\n -+ * The contents of the `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n -+ `.git/objects/info` directories\n +\n +This tool differs from linkgit:git-bugreport[1] in that it collects much more\n +detailed information with a greater focus on reporting the size and data shape\n @@ builtin/diagnose.c (new)\n +#include \"parse-options.h\"\n +#include \"diagnose.h\"\n +\n -+\n +static const char * const diagnose_usage[] = {\n -+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n ++\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n +\tNULL\n +};\n +\n @@ builtin/diagnose.c (new)\n +\tcase SCLD_EXISTS:\n +\t\tbreak;\n +\tdefault:\n -+\t\tdie(_(\"could not create leading directories for '%s'\"),\n -+\t\t zip_path.buf);\n ++\t\tdie_errno(_(\"could not create leading directories for '%s'\"),\n ++\t\t\t zip_path.buf);\n +\t}\n +\n +\t/* Prepare diagnostics */\n -+\tif (create_diagnostics_archive(&zip_path))\n ++\tif (create_diagnostics_archive(&zip_path, DIAGNOSE_STATS))\n +\t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n +\t\t\t zip_path.buf);\n +\n @@ t/t0092-diagnose.sh (new)\n +\ttest_when_finished rm -rf report &&\n +\n +\tgit diagnose -o report -s test >out &&\n ++\tgrep \"Available space\" out &&\n +\n +\tzip_path=report/git-diagnostics-test.zip &&\n -+\tgrep \"Available space\" out &&\n +\ttest_path_is_file \"$zip_path\" &&\n +\n +\t# Check zipped archive content\n @@ t/t0092-diagnose.sh (new)\n +\tgrep \".git/objects\" out &&\n +\n +\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n ++\tgrep \"^Total: [0-9][0-9]*\" out &&\n ++\n ++\t# Should not include .git directory contents by default\n ++\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n +\tgrep \"^Total: [0-9][0-9]*\" out\n +'\n +\n 7: a3e62a4a041 ! 8: 3da0cb725c9 builtin/diagnose.c: gate certain data behind '--all'\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - builtin/diagnose.c: gate certain data behind '--all'\n + builtin/diagnose.c: add '--mode' option\n \n - Update 'git diagnose' to *not* include '.git/' directory contents by\n - default, instead requiring specification of a '--all' option to include it.\n - While helpful for debugging, the archived '.git/' directory contents may be\n - sensitive, as they can be used to reconstruct an entire repository.\n + Create '--mode=' option in 'git diagnose' to allow users to optionally\n + select non-default diagnostic information to include in the output archive.\n + Additionally, document the currently-available modes, emphasizing the\n + importance of not sharing a '--mode=all' archive publicly due to the\n + presence of sensitive information.\n \n - To guard against users inadvertently including this information in\n - diagnostics and sharing it (e.g., with the mailing list), '.git/' directory\n - contents will only be included if '--all' is specified.\n + Note that the option parsing callback - 'option_parse_diagnose()' - is added\n + to 'diagnose.c' rather than 'builtin/diagnose.c' so that it may be reused in\n + future callers configuring a diagnostics archive.\n \n + Helped-by: Derrick Stolee \n Signed-off-by: Victoria Dye \n \n ## Documentation/git-diagnose.txt ##\n @@ Documentation/git-diagnose.txt: SYNOPSIS\n --------\n [verse]\n 'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n -+\t [-a | --all]\n ++\t [--mode=]\n \n DESCRIPTION\n -----------\n +@@ Documentation/git-diagnose.txt: repository state and packages that information into a zip archive. The\n + generated archive can then, for example, be shared with the Git mailing list to\n + help debug an issue or serve as a reference for independent debugging.\n + \n +-The following information is captured in the archive:\n ++By default, the following information is captured in the archive:\n + \n + * 'git version --build-options'\n + * The path to the repository root\n @@ Documentation/git-diagnose.txt: The following information is captured in the archive:\n - stores\n * The total count of loose objects, as well as counts broken down by\n `.git/objects` subdirectory\n -- * The contents of the `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n -- `.git/objects/info` directories\n \n ++Additional information can be collected by selecting a different diagnostic mode\n ++using the `--mode` option.\n ++\n This tool differs from linkgit:git-bugreport[1] in that it collects much more\n detailed information with a greater focus on reporting the size and data shape\n + of repository contents.\n @@ Documentation/git-diagnose.txt: OPTIONS\n \tform of a strftime(3) format string; the current local time will be\n \tused.\n \n -+-a::\n -+--all::\n -+\tInclude more complete repository diagnostic information in the archive.\n -+\tSpecifically, this will add copies of `.git`, `.git/hooks`, `.git/info`,\n -+\t`.git/logs`, and `.git/objects/info` directories to the output archive.\n -+\tThis additional data may be sensitive; a user can reconstruct the full\n -+\tcontents of the diagnosed repository with this information. Users should\n -+\texercise caution when sharing an archive generated with this option.\n ++--mode=(stats|all)::\n ++\tSpecify the type of diagnostics that should be collected. The default behavior\n ++\tof 'git diagnose' is equivalent to `--mode=stats`.\n +++\n ++The `--mode=all` option collects everything included in `--mode=stats`, as well\n ++as copies of `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n ++`.git/objects/info` directories. This additional information may be sensitive,\n ++as it can be used to reconstruct the full contents of the diagnosed repository.\n ++Users should exercise caution when sharing an archive generated with\n ++`--mode=all`.\n +\n GIT\n ---\n @@ Documentation/git-diagnose.txt: OPTIONS\n \n ## builtin/diagnose.c ##\n @@\n - \n + #include \"diagnose.h\"\n \n static const char * const diagnose_usage[] = {\n --\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n -+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ] [-a|--all]\"),\n +-\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n ++\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ] [--mode=]\"),\n \tNULL\n };\n \n @@ builtin/diagnose.c: int cmd_diagnose(int argc, const char **argv, const char *pr\n \tstruct strbuf zip_path = STRBUF_INIT;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n -+\tint include_everything = 0;\n ++\tenum diagnose_mode mode = DIAGNOSE_STATS;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tchar *prefixed_filename;\n @@ builtin/diagnose.c: int cmd_diagnose(int argc, const char **argv, const char *pr\n \t\t\t N_(\"specify a destination for the diagnostics archive\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n \t\t\t N_(\"specify a strftime format suffix for the filename\")),\n -+\t\tOPT_BOOL_F('a', \"all\", &include_everything,\n -+\t\t\t N_(\"collect complete diagnostic information\"),\n -+\t\t\t PARSE_OPT_NONEG),\n ++\t\tOPT_CALLBACK_F(0, \"mode\", &mode, N_(\"(stats|all)\"),\n ++\t\t\t N_(\"specify the content of the diagnostic archive\"),\n ++\t\t\t PARSE_OPT_NONEG, option_parse_diagnose),\n \t\tOPT_END()\n \t};\n \n @@ builtin/diagnose.c: int cmd_diagnose(int argc, const char **argv, const char *pr\n \t}\n \n \t/* Prepare diagnostics */\n --\tif (create_diagnostics_archive(&zip_path))\n -+\tif (create_diagnostics_archive(&zip_path, include_everything))\n +-\tif (create_diagnostics_archive(&zip_path, DIAGNOSE_STATS))\n ++\tif (create_diagnostics_archive(&zip_path, mode))\n \t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n \t\t\t zip_path.buf);\n \n \n - ## contrib/scalar/scalar.c ##\n -@@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n - \t\tgoto diagnose_cleanup;\n - \t}\n - \n --\tres = create_diagnostics_archive(&zip_path);\n -+\tres = create_diagnostics_archive(&zip_path, 1);\n - \n - diagnose_cleanup:\n - \tstrbuf_release(&zip_path);\n -\n ## diagnose.c ##\n -@@ diagnose.c: static int add_directory_to_archiver(struct strvec *archiver_args,\n - \treturn res;\n - }\n +@@\n + #include \"object-store.h\"\n + #include \"packfile.h\"\n \n --int create_diagnostics_archive(struct strbuf *zip_path)\n -+int create_diagnostics_archive(struct strbuf *zip_path, int include_everything)\n ++struct diagnose_option {\n ++\tenum diagnose_mode mode;\n ++\tconst char *option_name;\n ++};\n ++\n ++static struct diagnose_option diagnose_options[] = {\n ++\t{ DIAGNOSE_STATS, \"stats\" },\n ++\t{ DIAGNOSE_ALL, \"all\" },\n ++};\n ++\n ++int option_parse_diagnose(const struct option *opt, const char *arg, int unset)\n ++{\n ++\tint i;\n ++\tenum diagnose_mode *diagnose = opt->value;\n ++\n ++\tif (!arg) {\n ++\t\t*diagnose = unset ? DIAGNOSE_NONE : DIAGNOSE_STATS;\n ++\t\treturn 0;\n ++\t}\n ++\n ++\tfor (i = 0; i < ARRAY_SIZE(diagnose_options); i++) {\n ++\t\tif (!strcmp(arg, diagnose_options[i].option_name)) {\n ++\t\t\t*diagnose = diagnose_options[i].mode;\n ++\t\t\treturn 0;\n ++\t\t}\n ++\t}\n ++\n ++\treturn error(_(\"invalid --%s value '%s'\"), opt->long_name, arg);\n ++}\n ++\n + static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n + \t\t\t\t const char *file_name, void *data)\n {\n - \tstruct strvec archiver_args = STRVEC_INIT;\n - \tchar **argv_copy = NULL;\n -@@ diagnose.c: int create_diagnostics_archive(struct strbuf *zip_path)\n - \tloose_objs_stats(&buf, \".git/objects\");\n - \tstrvec_push(&archiver_args, buf.buf);\n - \n --\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n --\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n --\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n --\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n --\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n -+\t/* Only include this if explicitly requested */\n -+\tif (include_everything &&\n -+\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n - \t\tgoto diagnose_cleanup;\n - \n - \tstrvec_pushl(&archiver_args, \"--prefix=\",\n \n ## diagnose.h ##\n -@@\n - #include \"cache.h\"\n - #include \"strbuf.h\"\n +@@ diagnose.h: enum diagnose_mode {\n + \tDIAGNOSE_ALL\n + };\n \n --int create_diagnostics_archive(struct strbuf *zip_path);\n -+int create_diagnostics_archive(struct strbuf *zip_path, int include_everything);\n ++int option_parse_diagnose(const struct option *opt, const char *arg, int unset);\n ++\n + int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode);\n \n #endif /* DIAGNOSE_H */\n \n ## t/t0092-diagnose.sh ##\n @@ t/t0092-diagnose.sh: test_expect_success UNZIP 'creates diagnostics zip archive' '\n - \tgrep \".git/objects\" out &&\n \n - \t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n + \t# Should not include .git directory contents by default\n + \t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n -\tgrep \"^Total: [0-9][0-9]*\" out\n -+\tgrep \"^Total: [0-9][0-9]*\" out &&\n ++'\n ++\n ++test_expect_success UNZIP '--mode=stats excludes .git dir contents' '\n ++\ttest_when_finished rm -rf report &&\n +\n -+\t# Should not include .git directory contents\n ++\tgit diagnose -o report -s test --mode=stats >out &&\n ++\n ++\t# Includes pack quantity/size info\n ++\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n ++\tgrep \".git/objects\" out &&\n ++\n ++\t# Does not include .git directory contents\n +\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n +'\n +\n -+test_expect_success UNZIP '--all includes .git data in archive' '\n ++test_expect_success UNZIP '--mode=all includes .git dir contents' '\n +\ttest_when_finished rm -rf report &&\n +\n -+\tgit diagnose -o report -s test --all >out &&\n ++\tgit diagnose -o report -s test --mode=all >out &&\n ++\n ++\t# Includes pack quantity/size info\n ++\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n ++\tgrep \".git/objects\" out &&\n +\n -+\t# Should include .git directory contents\n ++\t# Includes .git directory contents\n +\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n +\n +\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n 8: d81e7c10997 ! 9: 1a1eb2c9806 builtin/bugreport.c: create '--diagnose' option\n @@ Commit message\n provide additional context to readers, ideally reducing some back-and-forth\n between reporters and those debugging the issue.\n \n - Note that '--diagnose' may take an optional string arg (either 'basic' or\n - 'all'). If specified without the arg or with 'basic', the behavior\n - corresponds to running 'git diagnose' without '--all'; this default is meant\n - to help reduce unintentional leaking of sensitive information). However, a\n - user can still manually specify '--diagnose=all' to generate the equivalent\n - archive to one created with 'git diagnose --all'.\n + Note that '--diagnose' may take an optional string arg (either 'stats' or\n + 'all'). If specified without the arg, the behavior corresponds to running\n + 'git diagnose' without '--mode'. As with 'git diagnose', this default is\n + intended to help reduce unintentional leaking of sensitive information).\n + Users can also explicitly specify '--diagnose=(stats|all)' to generate the\n + respective archive created by 'git diagnose --mode=(stats|all)'.\n \n Suggested-by: Ævar Arnfjörð Bjarmason \n + Helped-by: Derrick Stolee \n Signed-off-by: Victoria Dye \n \n ## Documentation/git-bugreport.txt ##\n @@ Documentation/git-bugreport.txt: SYNOPSIS\n --------\n [verse]\n 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n -+\t\t[--diagnose[=(basic|all)]]\n ++\t\t[--diagnose[=]]\n \n DESCRIPTION\n -----------\n @@ Documentation/git-bugreport.txt: OPTIONS\n \tnamed 'git-bugreport-'. This should take the form of a\n \tstrftime(3) format string; the current local time will be used.\n \n -+--diagnose[=(basic|all)]::\n -+\tCreate a zip archive of information about the repository including logs\n -+\tand certain statistics describing the data shape of the repository. The\n -+\tarchive is written to the same output directory as the bug report and is\n -+\tnamed 'git-diagnostics-'.\n ++--no-diagnose::\n ++--diagnose[=]::\n ++\tCreate a zip archive of supplemental information about the user's\n ++\tmachine, Git client, and repository state. The archive is written to the\n ++\tsame output directory as the bug report and is named\n ++\t'git-diagnostics-'.\n ++\n -+By default, `--diagnose` (equivalent to `--diagnose=basic`) will collect only\n -+statistics and summarized data about the repository and filesystem. Specifying\n -+`--diagnose=all` will create an archive with the same contents generated by `git\n -+diagnose --all`; this archive will be much larger, and will contain potentially\n -+sensitive information about the repository. See linkgit:git-diagnose[1] for more\n -+details on the contents of the diagnostic archive.\n ++Without `mode` specified, the diagnostic archive will contain the default set of\n ++statistics reported by `git diagnose`. An optional `mode` value may be specified\n ++to change which information is included in the archive. See\n ++linkgit:git-diagnose[1] for the list of valid values for `mode` and details\n ++about their usage.\n +\n GIT\n ---\n @@ builtin/bugreport.c\n #include \"hook-list.h\"\n +#include \"diagnose.h\"\n \n -+enum diagnose_mode {\n -+\tDIAGNOSE_NONE,\n -+\tDIAGNOSE_BASIC,\n -+\tDIAGNOSE_ALL\n -+};\n \n static void get_system_info(struct strbuf *sys_info)\n - {\n -@@ builtin/bugreport.c: static void get_header(struct strbuf *buf, const char *title)\n - \tstrbuf_addf(buf, \"\\n\\n[%s]\\n\", title);\n +@@ builtin/bugreport.c: static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n }\n \n -+static int option_parse_diagnose(const struct option *opt,\n -+\t\t\t\t const char *arg, int unset)\n -+{\n -+\tenum diagnose_mode *diagnose = opt->value;\n -+\n -+\tBUG_ON_OPT_NEG(unset);\n -+\n -+\tif (!arg || !strcmp(arg, \"basic\"))\n -+\t\t*diagnose = DIAGNOSE_BASIC;\n -+\telse if (!strcmp(arg, \"all\"))\n -+\t\t*diagnose = DIAGNOSE_ALL;\n -+\telse\n -+\t\tdie(_(\"diagnose mode must be either 'basic' or 'all'\"));\n -+\n -+\treturn 0;\n -+}\n -+\n - int cmd_bugreport(int argc, const char **argv, const char *prefix)\n - {\n - \tstruct strbuf buffer = STRBUF_INIT;\n + static const char * const bugreport_usage[] = {\n +-\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n ++\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ] [--diagnose[=]\"),\n + \tNULL\n + };\n + \n @@ builtin/bugreport.c: int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tint report = -1;\n \ttime_t now = time(NULL);\n @@ builtin/bugreport.c: int cmd_bugreport(int argc, const char **argv, const char *\n +\tsize_t output_path_len;\n \n \tconst struct option bugreport_options[] = {\n -+\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"(basic|all)\"),\n -+\t\t\t N_(\"create an additional zip archive of detailed diagnostics\"),\n -+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_OPTARG, option_parse_diagnose),\n ++\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"mode\"),\n ++\t\t\t N_(\"create an additional zip archive of detailed diagnostics (default 'stats')\"),\n ++\t\t\t PARSE_OPT_OPTARG, option_parse_diagnose),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n -\t\t\t N_(\"specify a destination for the bugreport file\")),\n +\t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n @@ builtin/bugreport.c: int cmd_bugreport(int argc, const char **argv, const char *\n +\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n +\t\tstrbuf_addstr(&zip_path, \".zip\");\n +\n -+\t\tif (create_diagnostics_archive(&zip_path, diagnose == DIAGNOSE_ALL))\n ++\t\tif (create_diagnostics_archive(&zip_path, diagnose))\n +\t\t\tdie_errno(_(\"unable to create diagnostics archive %s\"), zip_path.buf);\n +\n +\t\tstrbuf_release(&zip_path);\n @@ t/t0091-bugreport.sh: test_expect_success 'indicates populated hooks' '\n +\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n +'\n +\n -+test_expect_success UNZIP '--diagnose=basic excludes .git dir contents' '\n ++test_expect_success UNZIP '--diagnose=stats excludes .git dir contents' '\n +\ttest_when_finished rm -rf report &&\n +\n -+\tgit bugreport --diagnose=basic -o report -s test >out &&\n ++\tgit bugreport --diagnose=stats -o report -s test >out &&\n ++\n ++\t# Includes pack quantity/size info\n ++\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n ++\tgrep \".git/objects\" out &&\n +\n -+\t# Should not include .git directory contents\n ++\t# Does not include .git directory contents\n +\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n +'\n +\n @@ t/t0091-bugreport.sh: test_expect_success 'indicates populated hooks' '\n +\n +\tgit bugreport --diagnose=all -o report -s test >out &&\n +\n -+\t# Should include .git directory contents\n ++\t# Includes .git directory contents\n +\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n +\n +\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n 9: 6834bdcaea8 ! 10: d22674752f0 scalar-diagnose: use 'git diagnose --all'\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - scalar-diagnose: use 'git diagnose --all'\n + scalar-diagnose: use 'git diagnose --mode=all'\n \n Replace implementation of 'scalar diagnose' with an internal invocation of\n - 'git diagnose --all'. This simplifies the implementation of 'cmd_diagnose'\n - by making it a direct alias of 'git diagnose' and removes some code in\n - 'scalar.c' that is duplicated in 'builtin/diagnose.c'. The simplicity of the\n - alias also sets up a clean deprecation path for 'scalar diagnose' (in favor\n - of 'git diagnose'), if that is desired in the future.\n + 'git diagnose --mode=all'. This simplifies the implementation of\n + 'cmd_diagnose' by making it a direct alias of 'git diagnose' and removes\n + some code in 'scalar.c' that is duplicated in 'builtin/diagnose.c'. The\n + simplicity of the alias also sets up a clean deprecation path for 'scalar\n + diagnose' (in favor of 'git diagnose'), if that is desired in the future.\n \n This introduces one minor change to the output of 'scalar diagnose', which\n is that the prefix of the created zip archive is changed from 'scalar_' to\n 'git-diagnostics-'.\n \n + Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Victoria Dye \n \n ## contrib/scalar/scalar.c ##\n @@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n +\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_root);\n +\tstrbuf_addstr(&diagnostics_root, \"/.scalarDiagnostics\");\n \n --\tres = create_diagnostics_archive(&zip_path, 1);\n -+\tif (run_git(\"diagnose\", \"--all\", \"-s\", \"%Y%m%d_%H%M%S\",\n -+\t\t \"-o\", diagnostics_root.buf, NULL) < 0)\n -+\t\tres = -1;\n +-\tres = create_diagnostics_archive(&zip_path, DIAGNOSE_ALL);\n ++\tres = run_git(\"diagnose\", \"--mode=all\", \"-s\", \"%Y%m%d_%H%M%S\",\n ++\t\t \"-o\", diagnostics_root.buf, NULL);\n \n -diagnose_cleanup:\n -\tstrbuf_release(&zip_path);\n 10: 14925c3feed = 11: b64475f5b17 scalar: update technical doc roadmap\n\n-- \ngitgitgadget\n"},{"id":"461046","messageId":"78a93eb95bb0c43778501f03837bffdcf74e4c99.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 02/11] scalar-diagnose: avoid 32-bit overflow of size_t","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:24Z","receivedAt":"2022-08-10T23:34:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAvoid 32-bit size_t overflow when reporting the available disk space in\n'get_disk_info' by casting the block size and available block count to\n'off_t' before multiplying them. Without this change, 'st_mult' would\n(correctly) report a size_t overflow on 32-bit systems at or exceeding 2^32\nbytes of available space.\n\nNote that 'off_t' is a 64-bit integer even on 32-bit systems due to the\ninclusion of '#define _FILE_OFFSET_BITS 64' in 'git-compat-util.h' (see\nb97e911643 (Support for large files on 32bit systems., 2007-02-17)).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 97e71fe19cd..04046452284 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -348,7 +348,7 @@ static int get_disk_info(struct strbuf *out)\n \t}\n \n \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n+\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n \tstrbuf_release(&buf);\n #endif\n-- \ngitgitgadget\n\n"},{"id":"461047","messageId":"22ee8ea5a1e04a42ad359be4eb5ebc96bd5e5fa2.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 03/11] scalar-diagnose: add directory to archiver more gently","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:25Z","receivedAt":"2022-08-10T23:34:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIf a directory added to the 'scalar diagnose' archiver does not exist, warn\nand return 0 from 'add_directory_to_archiver()' rather than failing with a\nfatal error. This handles a failure edge case where the '.git/logs' has not\nyet been created when running 'scalar diagnose', but extends to any\nsituation where a directory may be missing in the '.git' dir.\n\nNow, when a directory is missing a warning is captured in the diagnostic\nlogs. This provides a user with more complete information than if 'scalar\ndiagnose' simply failed with an error.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 04046452284..b9092f0b612 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -266,14 +266,20 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \t\t\t\t\t const char *path, int recurse)\n {\n \tint at_root = !*path;\n-\tDIR *dir = opendir(at_root ? \".\" : path);\n+\tDIR *dir;\n \tstruct dirent *e;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t len;\n \tint res = 0;\n \n-\tif (!dir)\n+\tdir = opendir(at_root ? \".\" : path);\n+\tif (!dir) {\n+\t\tif (errno == ENOENT) {\n+\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n+\t\t\treturn 0;\n+\t\t}\n \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\t}\n \n \tif (!at_root)\n \t\tstrbuf_addf(&buf, \"%s/\", path);\n-- \ngitgitgadget\n\n"},{"id":"461048","messageId":"18f2ba4e0cdd9aea213dc7d0e908c6c0d8eccfd0.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 04/11] scalar-diagnose: move 'get_disk_info()' to 'compat/'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:26Z","receivedAt":"2022-08-10T23:34:57Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nMove 'get_disk_info()' function into 'compat/'. Although Scalar-specific\ncode is generally not part of the main Git tree, 'get_disk_info()' will be\nused in subsequent patches by additional callers beyond 'scalar diagnose'.\nThis patch prepares for that change, at which point this platform-specific\ncode should be part of 'compat/' as a matter of convention.\n\nThe function is copied *mostly* verbatim, with two exceptions:\n\n* '#ifdef WIN32' is replaced with '#ifdef GIT_WINDOWS_NATIVE' to allow\n 'statvfs' to be used with Cygwin.\n* the 'struct strbuf buf' and 'int res' (as well as their corresponding\n cleanup & return) are moved outside of the '#ifdef' block.\n\nSigned-off-by: Victoria Dye \n---\n compat/disk.h | 56 +++++++++++++++++++++++++++++++++++++++++\n contrib/scalar/scalar.c | 53 +-------------------------------------\n git-compat-util.h | 1 +\n 3 files changed, 58 insertions(+), 52 deletions(-)\n create mode 100644 compat/disk.h\n\ndiff --git a/compat/disk.h b/compat/disk.h\nnew file mode 100644\nindex 00000000000..50a32e3d8a4\n--- /dev/null\n+++ b/compat/disk.h\n@@ -0,0 +1,56 @@\n+#ifndef COMPAT_DISK_H\n+#define COMPAT_DISK_H\n+\n+#include \"git-compat-util.h\"\n+\n+static int get_disk_info(struct strbuf *out)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res = 0;\n+\n+#ifdef GIT_WINDOWS_NATIVE\n+\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n+\tDWORD serial_number, component_length, flags;\n+\tULARGE_INTEGER avail2caller, total, avail;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n+\t\terror(_(\"could not determine free disk size for '%s'\"),\n+\t\t buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n+\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n+\t\t\t\t &serial_number, &component_length, &flags,\n+\t\t\t\t fs_name, sizeof(fs_name))) {\n+\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n+\tstrbuf_addch(out, '\\n');\n+#else\n+\tstruct statvfs stat;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (statvfs(buf.buf, &stat) < 0) {\n+\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n+\t\t\t buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n+\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n+#endif\n+\n+cleanup:\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+#endif /* COMPAT_DISK_H */\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex b9092f0b612..607fedefd82 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -13,6 +13,7 @@\n #include \"help.h\"\n #include \"archive.h\"\n #include \"object-store.h\"\n+#include \"compat/disk.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -309,58 +310,6 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n \n-#ifndef WIN32\n-#include \n-#endif\n-\n-static int get_disk_info(struct strbuf *out)\n-{\n-#ifdef WIN32\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n-\tDWORD serial_number, component_length, flags;\n-\tULARGE_INTEGER avail2caller, total, avail;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n-\t\terror(_(\"could not determine free disk size for '%s'\"),\n-\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n-\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n-\t\t\t\t &serial_number, &component_length, &flags,\n-\t\t\t\t fs_name, sizeof(fs_name))) {\n-\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n-\tstrbuf_addch(out, '\\n');\n-\tstrbuf_release(&buf);\n-#else\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tstruct statvfs stat;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (statvfs(buf.buf, &stat) < 0) {\n-\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n-\t\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n-\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n-\tstrbuf_release(&buf);\n-#endif\n-\treturn 0;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 58d7708296b..9a62e3a0d2d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -258,6 +258,7 @@ static inline int is_xplatform_dir_sep(int c)\n #include \n #include \n #include \n+#include \n #include \n #ifndef NO_SYS_SELECT_H\n #include \n-- \ngitgitgadget\n\n"},{"id":"461049","messageId":"7a51fad87a8a21058ac95970d9dd66c731f07cf1.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 05/11] scalar-diagnose: move functionality to common location","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:27Z","receivedAt":"2022-08-10T23:34:59Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nMove the core functionality of 'scalar diagnose' into a new 'diagnose.[c,h]'\nlibrary to prepare for new callers in the main Git tree generating\ndiagnostic archives. These callers will be introduced in subsequent patches.\n\nWhile this patch appears large, it is mostly made up of moving code out of\n'scalar.c' and into 'diagnose.c'. Specifically, the functions\n\n- dir_file_stats_objects()\n- dir_file_stats()\n- count_files()\n- loose_objs_stats()\n- add_directory_to_archiver()\n\nare all copied verbatim from 'scalar.c'. The 'create_diagnostics_archive()'\nfunction is a mostly identical (partial) copy of 'cmd_diagnose()', with the\nprimary changes being that 'zip_path' is an input and \"Enlistment root\" is\ncorrected to \"Repository root\" in the archiver log.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n Makefile | 1 +\n contrib/scalar/scalar.c | 202 +------------------------------------\n diagnose.c | 217 ++++++++++++++++++++++++++++++++++++++++\n diagnose.h | 8 ++\n 4 files changed, 228 insertions(+), 200 deletions(-)\n create mode 100644 diagnose.c\n create mode 100644 diagnose.h\n\ndiff --git a/Makefile b/Makefile\nindex 2ec9b2dc6bb..ed66cb70e5a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -932,6 +932,7 @@ LIB_OBJS += ctype.o\n LIB_OBJS += date.o\n LIB_OBJS += decorate.o\n LIB_OBJS += delta-islands.o\n+LIB_OBJS += diagnose.o\n LIB_OBJS += diff-delta.o\n LIB_OBJS += diff-merges.o\n LIB_OBJS += diff-lib.o\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 607fedefd82..3983def760a 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,9 +11,7 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"archive.h\"\n-#include \"object-store.h\"\n-#include \"compat/disk.h\"\n+#include \"diagnose.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -263,53 +261,6 @@ static int unregister_dir(void)\n \treturn res;\n }\n \n-static int add_directory_to_archiver(struct strvec *archiver_args,\n-\t\t\t\t\t const char *path, int recurse)\n-{\n-\tint at_root = !*path;\n-\tDIR *dir;\n-\tstruct dirent *e;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t len;\n-\tint res = 0;\n-\n-\tdir = opendir(at_root ? \".\" : path);\n-\tif (!dir) {\n-\t\tif (errno == ENOENT) {\n-\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n-\t\t\treturn 0;\n-\t\t}\n-\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n-\t}\n-\n-\tif (!at_root)\n-\t\tstrbuf_addf(&buf, \"%s/\", path);\n-\tlen = buf.len;\n-\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n-\n-\twhile (!res && (e = readdir(dir))) {\n-\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n-\t\t\tcontinue;\n-\n-\t\tstrbuf_setlen(&buf, len);\n-\t\tstrbuf_addstr(&buf, e->d_name);\n-\n-\t\tif (e->d_type == DT_REG)\n-\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n-\t\telse if (e->d_type != DT_DIR)\n-\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n-\t\t\t\t \"directory\"), buf.buf);\n-\t\telse if (recurse &&\n-\t\t\t add_directory_to_archiver(archiver_args,\n-\t\t\t\t\t\t buf.buf, recurse) < 0)\n-\t\t\tres = -1;\n-\t}\n-\n-\tclosedir(dir);\n-\tstrbuf_release(&buf);\n-\treturn res;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\n@@ -550,83 +501,6 @@ cleanup:\n \treturn res;\n }\n \n-static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n-\t\t\t\t const char *file_name, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\tstruct stat st;\n-\n-\tif (!stat(full_path, &st))\n-\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n-\t\t\t (uintmax_t)st.st_size);\n-}\n-\n-static int dir_file_stats(struct object_directory *object_dir, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\n-\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n-\n-\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n-\t\t\t\t data);\n-\n-\treturn 0;\n-}\n-\n-static int count_files(char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count = 0;\n-\n-\tif (!dir)\n-\t\treturn 0;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n-\t\t\tcount++;\n-\n-\tclosedir(dir);\n-\treturn count;\n-}\n-\n-static void loose_objs_stats(struct strbuf *buf, const char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count;\n-\tint total = 0;\n-\tunsigned char c;\n-\tstruct strbuf count_path = STRBUF_INIT;\n-\tsize_t base_path_len;\n-\n-\tif (!dir)\n-\t\treturn;\n-\n-\tstrbuf_addstr(buf, \"Object directory stats for \");\n-\tstrbuf_add_absolute_path(buf, path);\n-\tstrbuf_addstr(buf, \":\\n\");\n-\n-\tstrbuf_add_absolute_path(&count_path, path);\n-\tstrbuf_addch(&count_path, '/');\n-\tbase_path_len = count_path.len;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) &&\n-\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n-\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n-\t\t\tstrbuf_setlen(&count_path, base_path_len);\n-\t\t\tstrbuf_addstr(&count_path, e->d_name);\n-\t\t\ttotal += (count = count_files(count_path.buf));\n-\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n-\t\t}\n-\n-\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n-\n-\tstrbuf_release(&count_path);\n-\tclosedir(dir);\n-}\n-\n static int cmd_diagnose(int argc, const char **argv)\n {\n \tstruct option options[] = {\n@@ -637,12 +511,8 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tNULL\n \t};\n \tstruct strbuf zip_path = STRBUF_INIT;\n-\tstruct strvec archiver_args = STRVEC_INIT;\n-\tchar **argv_copy = NULL;\n-\tint stdout_fd = -1, archiver_fd = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n-\tstruct strbuf buf = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n@@ -663,79 +533,11 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\t\t zip_path.buf);\n \t\tgoto diagnose_cleanup;\n \t}\n-\tstdout_fd = dup(1);\n-\tif (stdout_fd < 0) {\n-\t\tres = error_errno(_(\"could not duplicate stdout\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tarchiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n-\tif (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) {\n-\t\tres = error_errno(_(\"could not redirect output\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tinit_zip_archiver();\n-\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n-\tget_version_info(&buf, 1);\n-\n-\tstrbuf_addf(&buf, \"Enlistment root: %s\\n\", the_repository->worktree);\n-\tget_disk_info(&buf);\n-\twrite_or_die(stdout_fd, buf.buf, buf.len);\n-\tstrvec_pushf(&archiver_args,\n-\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n-\t\t (int)buf.len, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n-\tdir_file_stats(the_repository->objects->odb, &buf);\n-\tforeach_alt_odb(dir_file_stats, &buf);\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n-\tloose_objs_stats(&buf, \".git/objects\");\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n-\t\tgoto diagnose_cleanup;\n-\n-\tstrvec_pushl(&archiver_args, \"--prefix=\",\n-\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n-\n-\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n-\targv_copy = xmemdupz(archiver_args.v,\n-\t\t\t sizeof(char *) * archiver_args.nr);\n-\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n-\t\t\t the_repository, NULL, 0);\n-\tif (res) {\n-\t\terror(_(\"failed to write archive\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n \n-\tif (!res)\n-\t\tfprintf(stderr, \"\\n\"\n-\t\t \"Diagnostics complete.\\n\"\n-\t\t \"All of the gathered info is captured in '%s'\\n\",\n-\t\t zip_path.buf);\n+\tres = create_diagnostics_archive(&zip_path);\n \n diagnose_cleanup:\n-\tif (archiver_fd >= 0) {\n-\t\tclose(1);\n-\t\tdup2(stdout_fd, 1);\n-\t}\n-\tfree(argv_copy);\n-\tstrvec_clear(&archiver_args);\n \tstrbuf_release(&zip_path);\n-\tstrbuf_release(&buf);\n-\n \treturn res;\n }\n \ndiff --git a/diagnose.c b/diagnose.c\nnew file mode 100644\nindex 00000000000..509d582f0ea\n--- /dev/null\n+++ b/diagnose.c\n@@ -0,0 +1,217 @@\n+#include \"cache.h\"\n+#include \"diagnose.h\"\n+#include \"compat/disk.h\"\n+#include \"archive.h\"\n+#include \"dir.h\"\n+#include \"help.h\"\n+#include \"strvec.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_name, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\tstruct stat st;\n+\n+\tif (!stat(full_path, &st))\n+\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n+\t\t\t (uintmax_t)st.st_size);\n+}\n+\n+static int dir_file_stats(struct object_directory *object_dir, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\n+\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n+\n+\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n+\t\t\t\t data);\n+\n+\treturn 0;\n+}\n+\n+static int count_files(char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count = 0;\n+\n+\tif (!dir)\n+\t\treturn 0;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n+\t\t\tcount++;\n+\n+\tclosedir(dir);\n+\treturn count;\n+}\n+\n+static void loose_objs_stats(struct strbuf *buf, const char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count;\n+\tint total = 0;\n+\tunsigned char c;\n+\tstruct strbuf count_path = STRBUF_INIT;\n+\tsize_t base_path_len;\n+\n+\tif (!dir)\n+\t\treturn;\n+\n+\tstrbuf_addstr(buf, \"Object directory stats for \");\n+\tstrbuf_add_absolute_path(buf, path);\n+\tstrbuf_addstr(buf, \":\\n\");\n+\n+\tstrbuf_add_absolute_path(&count_path, path);\n+\tstrbuf_addch(&count_path, '/');\n+\tbase_path_len = count_path.len;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) &&\n+\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n+\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n+\t\t\tstrbuf_setlen(&count_path, base_path_len);\n+\t\t\tstrbuf_addstr(&count_path, e->d_name);\n+\t\t\ttotal += (count = count_files(count_path.buf));\n+\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n+\t\t}\n+\n+\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n+\n+\tstrbuf_release(&count_path);\n+\tclosedir(dir);\n+}\n+\n+static int add_directory_to_archiver(struct strvec *archiver_args,\n+\t\t\t\t const char *path, int recurse)\n+{\n+\tint at_root = !*path;\n+\tDIR *dir;\n+\tstruct dirent *e;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t len;\n+\tint res = 0;\n+\n+\tdir = opendir(at_root ? \".\" : path);\n+\tif (!dir) {\n+\t\tif (errno == ENOENT) {\n+\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n+\t\t\treturn 0;\n+\t\t}\n+\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\t}\n+\n+\tif (!at_root)\n+\t\tstrbuf_addf(&buf, \"%s/\", path);\n+\tlen = buf.len;\n+\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n+\n+\twhile (!res && (e = readdir(dir))) {\n+\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&buf, len);\n+\t\tstrbuf_addstr(&buf, e->d_name);\n+\n+\t\tif (e->d_type == DT_REG)\n+\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n+\t\telse if (e->d_type != DT_DIR)\n+\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n+\t\t\t\t \"directory\"), buf.buf);\n+\t\telse if (recurse &&\n+\t\t\t add_directory_to_archiver(archiver_args,\n+\t\t\t\t\t\t buf.buf, recurse) < 0)\n+\t\t\tres = -1;\n+\t}\n+\n+\tclosedir(dir);\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+int create_diagnostics_archive(struct strbuf *zip_path)\n+{\n+\tstruct strvec archiver_args = STRVEC_INIT;\n+\tchar **argv_copy = NULL;\n+\tint stdout_fd = -1, archiver_fd = -1;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res;\n+\n+\tstdout_fd = dup(STDOUT_FILENO);\n+\tif (stdout_fd < 0) {\n+\t\tres = error_errno(_(\"could not duplicate stdout\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tarchiver_fd = xopen(zip_path->buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n+\tif (dup2(archiver_fd, STDOUT_FILENO) < 0) {\n+\t\tres = error_errno(_(\"could not redirect output\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tinit_zip_archiver();\n+\tstrvec_pushl(&archiver_args, \"git-diagnose\", \"--format=zip\", NULL);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n+\tget_version_info(&buf, 1);\n+\n+\tstrbuf_addf(&buf, \"Repository root: %s\\n\", the_repository->worktree);\n+\tget_disk_info(&buf);\n+\twrite_or_die(stdout_fd, buf.buf, buf.len);\n+\tstrvec_pushf(&archiver_args,\n+\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n+\t\t (int)buf.len, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n+\tdir_file_stats(the_repository->objects->odb, &buf);\n+\tforeach_alt_odb(dir_file_stats, &buf);\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n+\tloose_objs_stats(&buf, \".git/objects\");\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n+\t\tgoto diagnose_cleanup;\n+\n+\tstrvec_pushl(&archiver_args, \"--prefix=\",\n+\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n+\n+\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n+\targv_copy = xmemdupz(archiver_args.v,\n+\t\t\t sizeof(char *) * archiver_args.nr);\n+\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n+\t\t\t the_repository, NULL, 0);\n+\tif (res) {\n+\t\terror(_(\"failed to write archive\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tif (!res)\n+\t\tfprintf(stderr, \"\\n\"\n+\t\t\t\"Diagnostics complete.\\n\"\n+\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n+\t\t\tzip_path->buf);\n+\n+diagnose_cleanup:\n+\tif (archiver_fd >= 0) {\n+\t\tdup2(stdout_fd, STDOUT_FILENO);\n+\t\tclose(stdout_fd);\n+\t\tclose(archiver_fd);\n+\t}\n+\tfree(argv_copy);\n+\tstrvec_clear(&archiver_args);\n+\tstrbuf_release(&buf);\n+\n+\treturn res;\n+}\ndiff --git a/diagnose.h b/diagnose.h\nnew file mode 100644\nindex 00000000000..06dca69bdac\n--- /dev/null\n+++ b/diagnose.h\n@@ -0,0 +1,8 @@\n+#ifndef DIAGNOSE_H\n+#define DIAGNOSE_H\n+\n+#include \"strbuf.h\"\n+\n+int create_diagnostics_archive(struct strbuf *zip_path);\n+\n+#endif /* DIAGNOSE_H */\n-- \ngitgitgadget\n\n"},{"id":"461050","messageId":"bf3c073a985cc0cc6b26ae96a5a6287949d5315d.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 07/11] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:29Z","receivedAt":"2022-08-10T23:35:02Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a 'git diagnose' builtin to generate a standalone zip archive of\nrepository diagnostics.\n\nThe \"diagnose\" functionality was originally implemented for Scalar in\naa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\ndiagnostics gathered are not specific to Scalar-cloned repositories and\ncan be useful when diagnosing issues in any Git repository.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n .gitignore | 1 +\n Documentation/git-diagnose.txt | 50 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diagnose.c | 57 ++++++++++++++++++++++++++++++++++\n git.c | 1 +\n t/t0092-diagnose.sh | 32 +++++++++++++++++++\n 7 files changed, 143 insertions(+)\n create mode 100644 Documentation/git-diagnose.txt\n create mode 100644 builtin/diagnose.c\n create mode 100755 t/t0092-diagnose.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 42fd7253b44..80b530bbed2 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -53,6 +53,7 @@\n /git-cvsimport\n /git-cvsserver\n /git-daemon\n+/git-diagnose\n /git-diff\n /git-diff-files\n /git-diff-index\ndiff --git a/Documentation/git-diagnose.txt b/Documentation/git-diagnose.txt\nnew file mode 100644\nindex 00000000000..ce07dd0725d\n--- /dev/null\n+++ b/Documentation/git-diagnose.txt\n@@ -0,0 +1,50 @@\n+git-diagnose(1)\n+================\n+\n+NAME\n+----\n+git-diagnose - Generate a zip archive of diagnostic information\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\n+DESCRIPTION\n+-----------\n+Collects detailed information about the user's machine, Git client, and\n+repository state and packages that information into a zip archive. The\n+generated archive can then, for example, be shared with the Git mailing list to\n+help debug an issue or serve as a reference for independent debugging.\n+\n+The following information is captured in the archive:\n+\n+ * 'git version --build-options'\n+ * The path to the repository root\n+ * The available disk space on the filesystem\n+ * The name and size of each packfile, including those in alternate object\n+ stores\n+ * The total count of loose objects, as well as counts broken down by\n+ `.git/objects` subdirectory\n+\n+This tool differs from linkgit:git-bugreport[1] in that it collects much more\n+detailed information with a greater focus on reporting the size and data shape\n+of repository contents.\n+\n+OPTIONS\n+-------\n+-o ::\n+--output-directory ::\n+\tPlace the resulting diagnostics archive in `` instead of the\n+\tcurrent directory.\n+\n+-s ::\n+--suffix ::\n+\tSpecify an alternate suffix for the diagnostics archive name, to create\n+\ta file named 'git-diagnostics-'. This should take the\n+\tform of a strftime(3) format string; the current local time will be\n+\tused.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex ed66cb70e5a..d34f680c065 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1153,6 +1153,7 @@ BUILTIN_OBJS += builtin/credential-cache.o\n BUILTIN_OBJS += builtin/credential-store.o\n BUILTIN_OBJS += builtin/credential.o\n BUILTIN_OBJS += builtin/describe.o\n+BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n BUILTIN_OBJS += builtin/diff-tree.o\ndiff --git a/builtin.h b/builtin.h\nindex 40e9ecc8485..8901a34d6bf 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -144,6 +144,7 @@ int cmd_credential_cache(int argc, const char **argv, const char *prefix);\n int cmd_credential_cache_daemon(int argc, const char **argv, const char *prefix);\n int cmd_credential_store(int argc, const char **argv, const char *prefix);\n int cmd_describe(int argc, const char **argv, const char *prefix);\n+int cmd_diagnose(int argc, const char **argv, const char *prefix);\n int cmd_diff_files(int argc, const char **argv, const char *prefix);\n int cmd_diff_index(int argc, const char **argv, const char *prefix);\n int cmd_diff(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/diagnose.c b/builtin/diagnose.c\nnew file mode 100644\nindex 00000000000..832493bba65\n--- /dev/null\n+++ b/builtin/diagnose.c\n@@ -0,0 +1,57 @@\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"diagnose.h\"\n+\n+static const char * const diagnose_usage[] = {\n+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n+\tNULL\n+};\n+\n+int cmd_diagnose(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct strbuf zip_path = STRBUF_INIT;\n+\ttime_t now = time(NULL);\n+\tstruct tm tm;\n+\tchar *option_output = NULL;\n+\tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n+\tchar *prefixed_filename;\n+\n+\tconst struct option diagnose_options[] = {\n+\t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n+\t\t\t N_(\"specify a destination for the diagnostics archive\")),\n+\t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n+\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, diagnose_options,\n+\t\t\t diagnose_usage, 0);\n+\n+\t/* Prepare the path to put the result */\n+\tprefixed_filename = prefix_filename(prefix,\n+\t\t\t\t\t option_output ? option_output : \"\");\n+\tstrbuf_addstr(&zip_path, prefixed_filename);\n+\tstrbuf_complete(&zip_path, '/');\n+\n+\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\tswitch (safe_create_leading_directories(zip_path.buf)) {\n+\tcase SCLD_OK:\n+\tcase SCLD_EXISTS:\n+\t\tbreak;\n+\tdefault:\n+\t\tdie_errno(_(\"could not create leading directories for '%s'\"),\n+\t\t\t zip_path.buf);\n+\t}\n+\n+\t/* Prepare diagnostics */\n+\tif (create_diagnostics_archive(&zip_path, DIAGNOSE_STATS))\n+\t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n+\t\t\t zip_path.buf);\n+\n+\tfree(prefixed_filename);\n+\tstrbuf_release(&zip_path);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex e5d62fa5a92..0b9d8ef7677 100644\n--- a/git.c\n+++ b/git.c\n@@ -522,6 +522,7 @@ static struct cmd_struct commands[] = {\n \t{ \"credential-cache--daemon\", cmd_credential_cache_daemon },\n \t{ \"credential-store\", cmd_credential_store },\n \t{ \"describe\", cmd_describe, RUN_SETUP },\n+\t{ \"diagnose\", cmd_diagnose, RUN_SETUP_GENTLY },\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\ndiff --git a/t/t0092-diagnose.sh b/t/t0092-diagnose.sh\nnew file mode 100755\nindex 00000000000..b6923726fd7\n--- /dev/null\n+++ b/t/t0092-diagnose.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='git diagnose'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success UNZIP 'creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test >out &&\n+\tgrep \"Available space\" out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out &&\n+\n+\t# Should not include .git directory contents by default\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+\tgrep \"^Total: [0-9][0-9]*\" out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"461051","messageId":"0a6c55696d88cde666c11cd6b5d723c9e75a3b76.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 06/11] diagnose.c: add option to configure archive contents","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:28Z","receivedAt":"2022-08-10T23:35:03Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate 'create_diagnostics_archive()' to take an argument 'mode'. When\narchiving diagnostics for a repository, 'mode' is used to selectively\ninclude/exclude information based on its value. The initial options for\n'mode' are:\n\n* DIAGNOSE_NONE: do not collect any diagnostics or create an archive\n (no-op).\n* DIAGNOSE_STATS: collect basic repository metadata (Git version, repo path,\n filesystem available space) as well as sizing and count statistics for the\n repository's objects and packfiles.\n* DIAGNOSE_ALL: collect basic repository metadata, sizing/count statistics,\n and copies of the '.git', '.git/hooks', '.git/info', '.git/logs', and\n '.git/objects/info' directories.\n\nThese modes are introduced to provide users the option to collect\ndiagnostics without the sensitive information included in copies of '.git'\ndir contents. At the moment, only 'scalar diagnose' uses\n'create_diagnostics_archive()' (with a hardcoded 'DIAGNOSE_ALL' mode to\nmatch existing functionality), but more callers will be introduced in\nsubsequent patches.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 2 +-\n diagnose.c | 19 +++++++++++++------\n diagnose.h | 9 ++++++++-\n 3 files changed, 22 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 3983def760a..d538b8b8f14 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -534,7 +534,7 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tgoto diagnose_cleanup;\n \t}\n \n-\tres = create_diagnostics_archive(&zip_path);\n+\tres = create_diagnostics_archive(&zip_path, DIAGNOSE_ALL);\n \n diagnose_cleanup:\n \tstrbuf_release(&zip_path);\ndiff --git a/diagnose.c b/diagnose.c\nindex 509d582f0ea..aadc3d4b21f 100644\n--- a/diagnose.c\n+++ b/diagnose.c\n@@ -132,7 +132,7 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n \n-int create_diagnostics_archive(struct strbuf *zip_path)\n+int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode)\n {\n \tstruct strvec archiver_args = STRVEC_INIT;\n \tchar **argv_copy = NULL;\n@@ -140,6 +140,11 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n \tstruct strbuf buf = STRBUF_INIT;\n \tint res;\n \n+\tif (mode == DIAGNOSE_NONE) {\n+\t\tres = 0;\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n \tstdout_fd = dup(STDOUT_FILENO);\n \tif (stdout_fd < 0) {\n \t\tres = error_errno(_(\"could not duplicate stdout\"));\n@@ -177,11 +182,13 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n \tloose_objs_stats(&buf, \".git/objects\");\n \tstrvec_push(&archiver_args, buf.buf);\n \n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n+\t/* Only include this if explicitly requested */\n+\tif (mode == DIAGNOSE_ALL &&\n+\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n \t\tgoto diagnose_cleanup;\n \n \tstrvec_pushl(&archiver_args, \"--prefix=\",\ndiff --git a/diagnose.h b/diagnose.h\nindex 06dca69bdac..9bb6049bf0c 100644\n--- a/diagnose.h\n+++ b/diagnose.h\n@@ -2,7 +2,14 @@\n #define DIAGNOSE_H\n \n #include \"strbuf.h\"\n+#include \"parse-options.h\"\n \n-int create_diagnostics_archive(struct strbuf *zip_path);\n+enum diagnose_mode {\n+\tDIAGNOSE_NONE,\n+\tDIAGNOSE_STATS,\n+\tDIAGNOSE_ALL\n+};\n+\n+int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode);\n \n #endif /* DIAGNOSE_H */\n-- \ngitgitgadget\n\n"},{"id":"461052","messageId":"1a1eb2c980635415c04d5c8d9a62bd972482d7dc.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 09/11] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:31Z","receivedAt":"2022-08-10T23:35:07Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a '--diagnose' option for 'git bugreport' to collect additional\ninformation about the repository and write it to a zipped archive.\n\nThe '--diagnose' option behaves effectively as an alias for simultaneously\nrunning 'git bugreport' and 'git diagnose'. In the documentation, users are\nexplicitly recommended to attach the diagnostics alongside a bug report to\nprovide additional context to readers, ideally reducing some back-and-forth\nbetween reporters and those debugging the issue.\n\nNote that '--diagnose' may take an optional string arg (either 'stats' or\n'all'). If specified without the arg, the behavior corresponds to running\n'git diagnose' without '--mode'. As with 'git diagnose', this default is\nintended to help reduce unintentional leaking of sensitive information).\nUsers can also explicitly specify '--diagnose=(stats|all)' to generate the\nrespective archive created by 'git diagnose --mode=(stats|all)'.\n\nSuggested-by: Ævar Arnfjörð Bjarmason \nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n Documentation/git-bugreport.txt | 18 +++++++++++++\n builtin/bugreport.c | 27 ++++++++++++++++---\n t/t0091-bugreport.sh | 48 +++++++++++++++++++++++++++++++++\n 3 files changed, 90 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-bugreport.txt b/Documentation/git-bugreport.txt\nindex d8817bf3cec..eca726e5791 100644\n--- a/Documentation/git-bugreport.txt\n+++ b/Documentation/git-bugreport.txt\n@@ -9,6 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\t\t[--diagnose[=]]\n \n DESCRIPTION\n -----------\n@@ -31,6 +32,10 @@ The following information is captured automatically:\n - A list of enabled hooks\n - $SHELL\n \n+Additional information may be gathered into a separate zip archive using the\n+`--diagnose` option, and can be attached alongside the bugreport document to\n+provide additional context to readers.\n+\n This tool is invoked via the typical Git setup process, which means that in some\n cases, it might not be able to launch - for example, if a relevant config file\n is unreadable. In this kind of scenario, it may be helpful to manually gather\n@@ -49,6 +54,19 @@ OPTIONS\n \tnamed 'git-bugreport-'. This should take the form of a\n \tstrftime(3) format string; the current local time will be used.\n \n+--no-diagnose::\n+--diagnose[=]::\n+\tCreate a zip archive of supplemental information about the user's\n+\tmachine, Git client, and repository state. The archive is written to the\n+\tsame output directory as the bug report and is named\n+\t'git-diagnostics-'.\n++\n+Without `mode` specified, the diagnostic archive will contain the default set of\n+statistics reported by `git diagnose`. An optional `mode` value may be specified\n+to change which information is included in the archive. See\n+linkgit:git-diagnose[1] for the list of valid values for `mode` and details\n+about their usage.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 9de32bc96e7..530895be55f 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -5,6 +5,7 @@\n #include \"compat/compiler.h\"\n #include \"hook.h\"\n #include \"hook-list.h\"\n+#include \"diagnose.h\"\n \n \n static void get_system_info(struct strbuf *sys_info)\n@@ -59,7 +60,7 @@ static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n }\n \n static const char * const bugreport_usage[] = {\n-\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n+\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ] [--diagnose[=]\"),\n \tNULL\n };\n \n@@ -98,16 +99,21 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tint report = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tenum diagnose_mode diagnose = DIAGNOSE_NONE;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tconst char *user_relative_path = NULL;\n \tchar *prefixed_filename;\n+\tsize_t output_path_len;\n \n \tconst struct option bugreport_options[] = {\n+\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"mode\"),\n+\t\t\t N_(\"create an additional zip archive of detailed diagnostics (default 'stats')\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_diagnose),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n-\t\t\t N_(\"specify a destination for the bugreport file\")),\n+\t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n-\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\t\t N_(\"specify a strftime format suffix for the filename(s)\")),\n \t\tOPT_END()\n \t};\n \n@@ -119,6 +125,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t option_output ? option_output : \"\");\n \tstrbuf_addstr(&report_path, prefixed_filename);\n \tstrbuf_complete(&report_path, '/');\n+\toutput_path_len = report_path.len;\n \n \tstrbuf_addstr(&report_path, \"git-bugreport-\");\n \tstrbuf_addftime(&report_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n@@ -133,6 +140,20 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t report_path.buf);\n \t}\n \n+\t/* Prepare diagnostics, if requested */\n+\tif (diagnose != DIAGNOSE_NONE) {\n+\t\tstruct strbuf zip_path = STRBUF_INIT;\n+\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n+\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\t\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\t\tif (create_diagnostics_archive(&zip_path, diagnose))\n+\t\t\tdie_errno(_(\"unable to create diagnostics archive %s\"), zip_path.buf);\n+\n+\t\tstrbuf_release(&zip_path);\n+\t}\n+\n \t/* Prepare the report contents */\n \tget_bug_template(&buffer);\n \ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 08f5fe9caef..b6d2f591acd 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -78,4 +78,52 @@ test_expect_success 'indicates populated hooks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose -o report -s test >out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\tgrep \"Available space\" out &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out &&\n+\n+\t# Should not include .git directory contents by default\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--diagnose=stats excludes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose=stats -o report -s test >out &&\n+\n+\t# Includes pack quantity/size info\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t# Does not include .git directory contents\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--diagnose=all includes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose=all -o report -s test >out &&\n+\n+\t# Includes .git directory contents\n+\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n+\ttest_file_not_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"461053","messageId":"3da0cb725c927d08dd9486286e06bdb76896f5b7.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 08/11] builtin/diagnose.c: add '--mode' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:30Z","receivedAt":"2022-08-10T23:35:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate '--mode=' option in 'git diagnose' to allow users to optionally\nselect non-default diagnostic information to include in the output archive.\nAdditionally, document the currently-available modes, emphasizing the\nimportance of not sharing a '--mode=all' archive publicly due to the\npresence of sensitive information.\n\nNote that the option parsing callback - 'option_parse_diagnose()' - is added\nto 'diagnose.c' rather than 'builtin/diagnose.c' so that it may be reused in\nfuture callers configuring a diagnostics archive.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n Documentation/git-diagnose.txt | 17 ++++++++++++++++-\n builtin/diagnose.c | 8 ++++++--\n diagnose.c | 30 ++++++++++++++++++++++++++++++\n diagnose.h | 2 ++\n t/t0092-diagnose.sh | 30 +++++++++++++++++++++++++++++-\n 5 files changed, 83 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-diagnose.txt b/Documentation/git-diagnose.txt\nindex ce07dd0725d..3ec8cc7ad72 100644\n--- a/Documentation/git-diagnose.txt\n+++ b/Documentation/git-diagnose.txt\n@@ -9,6 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\t [--mode=]\n \n DESCRIPTION\n -----------\n@@ -17,7 +18,7 @@ repository state and packages that information into a zip archive. The\n generated archive can then, for example, be shared with the Git mailing list to\n help debug an issue or serve as a reference for independent debugging.\n \n-The following information is captured in the archive:\n+By default, the following information is captured in the archive:\n \n * 'git version --build-options'\n * The path to the repository root\n@@ -27,6 +28,9 @@ The following information is captured in the archive:\n * The total count of loose objects, as well as counts broken down by\n `.git/objects` subdirectory\n \n+Additional information can be collected by selecting a different diagnostic mode\n+using the `--mode` option.\n+\n This tool differs from linkgit:git-bugreport[1] in that it collects much more\n detailed information with a greater focus on reporting the size and data shape\n of repository contents.\n@@ -45,6 +49,17 @@ OPTIONS\n \tform of a strftime(3) format string; the current local time will be\n \tused.\n \n+--mode=(stats|all)::\n+\tSpecify the type of diagnostics that should be collected. The default behavior\n+\tof 'git diagnose' is equivalent to `--mode=stats`.\n++\n+The `--mode=all` option collects everything included in `--mode=stats`, as well\n+as copies of `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n+`.git/objects/info` directories. This additional information may be sensitive,\n+as it can be used to reconstruct the full contents of the diagnosed repository.\n+Users should exercise caution when sharing an archive generated with\n+`--mode=all`.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/diagnose.c b/builtin/diagnose.c\nindex 832493bba65..cd260c20155 100644\n--- a/builtin/diagnose.c\n+++ b/builtin/diagnose.c\n@@ -3,7 +3,7 @@\n #include \"diagnose.h\"\n \n static const char * const diagnose_usage[] = {\n-\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ] [--mode=]\"),\n \tNULL\n };\n \n@@ -12,6 +12,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \tstruct strbuf zip_path = STRBUF_INIT;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tenum diagnose_mode mode = DIAGNOSE_STATS;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tchar *prefixed_filename;\n@@ -21,6 +22,9 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"specify a destination for the diagnostics archive\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n \t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\tOPT_CALLBACK_F(0, \"mode\", &mode, N_(\"(stats|all)\"),\n+\t\t\t N_(\"specify the content of the diagnostic archive\"),\n+\t\t\t PARSE_OPT_NONEG, option_parse_diagnose),\n \t\tOPT_END()\n \t};\n \n@@ -47,7 +51,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \t}\n \n \t/* Prepare diagnostics */\n-\tif (create_diagnostics_archive(&zip_path, DIAGNOSE_STATS))\n+\tif (create_diagnostics_archive(&zip_path, mode))\n \t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n \t\t\t zip_path.buf);\n \ndiff --git a/diagnose.c b/diagnose.c\nindex aadc3d4b21f..00c3e9438e2 100644\n--- a/diagnose.c\n+++ b/diagnose.c\n@@ -8,6 +8,36 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n+struct diagnose_option {\n+\tenum diagnose_mode mode;\n+\tconst char *option_name;\n+};\n+\n+static struct diagnose_option diagnose_options[] = {\n+\t{ DIAGNOSE_STATS, \"stats\" },\n+\t{ DIAGNOSE_ALL, \"all\" },\n+};\n+\n+int option_parse_diagnose(const struct option *opt, const char *arg, int unset)\n+{\n+\tint i;\n+\tenum diagnose_mode *diagnose = opt->value;\n+\n+\tif (!arg) {\n+\t\t*diagnose = unset ? DIAGNOSE_NONE : DIAGNOSE_STATS;\n+\t\treturn 0;\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(diagnose_options); i++) {\n+\t\tif (!strcmp(arg, diagnose_options[i].option_name)) {\n+\t\t\t*diagnose = diagnose_options[i].mode;\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn error(_(\"invalid --%s value '%s'\"), opt->long_name, arg);\n+}\n+\n static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n \t\t\t\t const char *file_name, void *data)\n {\ndiff --git a/diagnose.h b/diagnose.h\nindex 9bb6049bf0c..7a4951a7863 100644\n--- a/diagnose.h\n+++ b/diagnose.h\n@@ -10,6 +10,8 @@ enum diagnose_mode {\n \tDIAGNOSE_ALL\n };\n \n+int option_parse_diagnose(const struct option *opt, const char *arg, int unset);\n+\n int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode);\n \n #endif /* DIAGNOSE_H */\ndiff --git a/t/t0092-diagnose.sh b/t/t0092-diagnose.sh\nindex b6923726fd7..fca9b58489c 100755\n--- a/t/t0092-diagnose.sh\n+++ b/t/t0092-diagnose.sh\n@@ -26,7 +26,35 @@ test_expect_success UNZIP 'creates diagnostics zip archive' '\n \n \t# Should not include .git directory contents by default\n \t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n-\tgrep \"^Total: [0-9][0-9]*\" out\n+'\n+\n+test_expect_success UNZIP '--mode=stats excludes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test --mode=stats >out &&\n+\n+\t# Includes pack quantity/size info\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t# Does not include .git directory contents\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--mode=all includes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test --mode=all >out &&\n+\n+\t# Includes pack quantity/size info\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t# Includes .git directory contents\n+\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n+\ttest_file_not_empty out\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"461054","messageId":"d22674752f04a543185c29bbc51511f176682fdf.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 10/11] scalar-diagnose: use 'git diagnose --mode=all'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:32Z","receivedAt":"2022-08-10T23:35:10Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nReplace implementation of 'scalar diagnose' with an internal invocation of\n'git diagnose --mode=all'. This simplifies the implementation of\n'cmd_diagnose' by making it a direct alias of 'git diagnose' and removes\nsome code in 'scalar.c' that is duplicated in 'builtin/diagnose.c'. The\nsimplicity of the alias also sets up a clean deprecation path for 'scalar\ndiagnose' (in favor of 'git diagnose'), if that is desired in the future.\n\nThis introduces one minor change to the output of 'scalar diagnose', which\nis that the prefix of the created zip archive is changed from 'scalar_' to\n'git-diagnostics-'.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 28 ++++++----------------------\n 1 file changed, 6 insertions(+), 22 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex d538b8b8f14..68571ce195f 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,7 +11,6 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"diagnose.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -510,34 +509,19 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tN_(\"scalar diagnose []\"),\n \t\tNULL\n \t};\n-\tstruct strbuf zip_path = STRBUF_INIT;\n-\ttime_t now = time(NULL);\n-\tstruct tm tm;\n+\tstruct strbuf diagnostics_root = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\t usage, 0);\n \n-\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n-\n-\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n-\tstrbuf_addftime(&zip_path,\n-\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n-\tstrbuf_addstr(&zip_path, \".zip\");\n-\tswitch (safe_create_leading_directories(zip_path.buf)) {\n-\tcase SCLD_EXISTS:\n-\tcase SCLD_OK:\n-\t\tbreak;\n-\tdefault:\n-\t\terror_errno(_(\"could not create directory for '%s'\"),\n-\t\t\t zip_path.buf);\n-\t\tgoto diagnose_cleanup;\n-\t}\n+\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_root);\n+\tstrbuf_addstr(&diagnostics_root, \"/.scalarDiagnostics\");\n \n-\tres = create_diagnostics_archive(&zip_path, DIAGNOSE_ALL);\n+\tres = run_git(\"diagnose\", \"--mode=all\", \"-s\", \"%Y%m%d_%H%M%S\",\n+\t\t \"-o\", diagnostics_root.buf, NULL);\n \n-diagnose_cleanup:\n-\tstrbuf_release(&zip_path);\n+\tstrbuf_release(&diagnostics_root);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"461055","messageId":"b64475f5b170666b76f28f86f9a9391c426b2d83.1660174473.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v3 11/11] scalar: update technical doc roadmap","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-10T23:34:33Z","receivedAt":"2022-08-10T23:35:13Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate the Scalar roadmap to reflect the completion of generalizing 'scalar\ndiagnose' into 'git diagnose' and 'git bugreport --diagnose'.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/technical/scalar.txt | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt\nindex 08bc09c225a..f6353375f08 100644\n--- a/Documentation/technical/scalar.txt\n+++ b/Documentation/technical/scalar.txt\n@@ -84,6 +84,9 @@ series have been accepted:\n \n - `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n \n+- `scalar-generalize-diagnose`: Move the functionality of `scalar diagnose`\n+ into `git diagnose` and `git bugreport --diagnose`.\n+\n Roughly speaking (and subject to change), the following series are needed to\n \"finish\" this initial version of Scalar:\n \n@@ -91,12 +94,6 @@ Roughly speaking (and subject to change), the following series are needed to\n and implement `scalar help`. At the end of this series, Scalar should be\n feature-complete from the perspective of a user.\n \n-- Generalize features not specific to Scalar: In the spirit of making Scalar\n- configure only what is needed for large repo performance, move common\n- utilities into other parts of Git. Some of this will be internal-only, but one\n- major change will be generalizing `scalar diagnose` for use with any Git\n- repository.\n-\n - Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of\n `git`, including updates to build and install it with the rest of Git. This\n change will incorporate Scalar into the Git CI and test framework, as well as\n-- \ngitgitgadget\n"},{"id":"461056","messageId":"xmqq8rnviqhz.fsf@gitster.g","threadId":"58251","inReplyTo":"0a6c55696d88cde666c11cd6b5d723c9e75a3b76.1660174473.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 06/11] diagnose.c: add option to configure archive contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-11T00:16:24Z","receivedAt":"2022-08-11T00:16:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> @@ -177,11 +182,13 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n> \tloose_objs_stats(&buf, \".git/objects\");\n> \tstrvec_push(&archiver_args, buf.buf);\n> \n> -\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n> +\t/* Only include this if explicitly requested */\n> +\tif (mode == DIAGNOSE_ALL &&\n> +\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n> +\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n> \t\tgoto diagnose_cleanup;\n\nAt first glance, it looks as if this part fails silently, but\nadd_directory_to_archiver() states what failed there, so we show\nnecessary error messages and do not silently fail, which is good.\n\nThere is a \"failed to write archive\" message after write_archive()\ncall returns non-zero, but presumably write_archive() itself gives\ndiagnostics (like \"oh, I was told to archive this file but I cannot\nread it\") when it does so, so in a sense, giving the concluding\n\"failed to write\" only in that case might make the error messages\nuneven. If we fail to enlist \".git/hooks\" directory, we may want to\nsay why we failed to do so, and then want to see the concluding\n\"failed to write\" at the end, just like the case where write_archive()\nfailed.\n\nIt is a truely minor point, and if it turns out to be worth fixing,\nit can be easily done by moving the diagnose_clean_up label a bit\nhigher, i.e.\n\n\t...\n\tres = write_archive(...);\n\ndiagnose_cleanup:\n\tif (res)\n\t\terror(_(\"failed to write archive\"));\n\telse\n \tfprintf(stderr, \"\\n\"\n\t\t\t\"Diagnostics complete.\\n\"\n\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n\t\t\tzip_path->buf);\n\n\tif (archiver_fd >= 0) {\n\t\t... restore FD#1 and close stdout_fd and archiver_fd\n\t}\n\t...\n\n\nOther than that, this new patch looks good to me.\n\nThanks.\n"},{"id":"461067","messageId":"220811.86ilmzqchn.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"0a6c55696d88cde666c11cd6b5d723c9e75a3b76.1660174473.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 06/11] diagnose.c: add option to configure archive contents","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-11T10:51:28Z","receivedAt":"2022-08-11T10:51:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 10 2022, Victoria Dye via GitGitGadget wrote:\n\n> index 06dca69bdac..9bb6049bf0c 100644\n> --- a/diagnose.h\n> +++ b/diagnose.h\n> @@ -2,7 +2,14 @@\n> #define DIAGNOSE_H\n> \n> #include \"strbuf.h\"\n> +#include \"parse-options.h\"\n\nThis is a stray include that isn't needed at this point, some mistake,\nor needed by a subsequent patch?\n"},{"id":"461068","messageId":"220811.86edxnqcdk.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"1a1eb2c980635415c04d5c8d9a62bd972482d7dc.1660174473.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 09/11] builtin/bugreport.c: create '--diagnose' option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-11T10:53:48Z","receivedAt":"2022-08-11T10:54:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 10 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n>\n> Create a '--diagnose' option for 'git bugreport' to collect additional\n> information about the repository and write it to a zipped archive.\n> [...]\n> 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n> +\t\t[--diagnose[=]]\n> [...]\n> static const char * const bugreport_usage[] = {\n> -\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n> +\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ] [--diagnose[=]\"),\n> \tNULL\n> };\n\nThis still has the SYNOPSIS v.s. -h discrepancy noted in\nhttps://lore.kernel.org/git/220804.86v8r8ec4s.gmgdl@evledraar.gmail.com/\n"},{"id":"461070","messageId":"220811.865yizqbnt.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"41f85d57-596a-5e42-756c-93fe0f45185f@github.com","subject":"Re: [PATCH v2 06/10] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-11T11:06:24Z","receivedAt":"2022-08-11T11:09:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Aug 05 2022, Derrick Stolee wrote:\n\n> On 8/4/2022 2:27 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Thu, Aug 04 2022, Victoria Dye via GitGitGadget wrote:\n>> \n>>> From: Victoria Dye \n>>>\n>>> Create a 'git diagnose' builtin to generate a standalone zip archive of\n>>> repository diagnostics.\n>> \n>> It's good to have this as a built-in separate from \"git bugreport\",\n>> but...\n>> \n>>> +git-diagnose - Generate a zip archive of diagnostic information\n>> \n>> ...I'd really prefer for this not to squat on such a common name we\n>> might regret having reserved later for such very specific\n>> functionality. I'd think e.g. these would be better:\n>> \n>> \tgit mk-diagnostics-zip\n>> \n>> Or maybe:\n>> \n>> \tgit archive-interesting-for-report\n>\n> These are not realistic replacements.\n\nMaybe:\n\n\tgit diagnose create-zip\n\n?\n\n>> If I had to guess what a \"git diagnose\" did, I'd probably think:\n>> \n>> * It analyzes your config, and suggests redundancies/alternatives\n>> * It does some perf tests / heuritics, and e.g. suggests you turn on\n>> the commit-graph writing.\n>\n> These sound like great options to add in the future, such as:\n>\n> --perf-test: Run performance tests on your repository using different\n> Git config options and recommend certain settings.\n>\n> (This --perf-test option would be a great way to get wider adoption\n> of parallel checkout, since its optimal settings are so machine\n> dependent.)\n\n...\n\n> The thing is, even if we did these other things, it would result in\n> some kind of document that summarizes the repository shape and features.\n> That kind of data is exactly what this version of 'git diagnose' does.\n\nI think a command like \"git diagnose\" that had options to do other\nunrelated stuff, but by default created a zip archive when given no\noptions would be rather confusing.\n\nYes, it makes sense to emit some human-readable summary, but to zip it\nup as well? That's something we just need for the \"git bugreport\"\ncase...\n\n> For now, it leaves the human reader responsible for making decisions\n> based on those documents, but they have been incredibly helpful when we\n> are _diagnosing_ issues users are having with their repositories.\n\nThis is orthagonal to what I'm pointing out. You're basically saying the\nuser can just read the documentation to find out what this built-in\ndoes.\n\nThat's true, what I'm pointing out is that it's unfortunate that such\nhighly specific functionality is squatting on such a short & generic\nname, but just e.g. adding a \"create-zip\" sub-command would address\nthat.\n"},{"id":"461077","messageId":"f9a5f658-7c6f-bb3a-d6df-0ab7d2908395@github.com","threadId":"58251","inReplyTo":"220811.86edxnqcdk.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 09/11] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-11T15:40:42Z","receivedAt":"2022-08-11T15:47:28Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Aug 10 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> From: Victoria Dye \n>>\n>> Create a '--diagnose' option for 'git bugreport' to collect additional\n>> information about the repository and write it to a zipped archive.\n>> [...]\n>> 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n>> +\t\t[--diagnose[=]]\n>> [...]\n>> static const char * const bugreport_usage[] = {\n>> -\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n>> +\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ] [--diagnose[=]\"),\n>> \tNULL\n>> };\n> \n> This still has the SYNOPSIS v.s. -h discrepancy noted in\n> https://lore.kernel.org/git/220804.86v8r8ec4s.gmgdl@evledraar.gmail.com/\n\nThe discrepancy you pointed out was on 'git diagnose' (which has since been\nfixed), this is a pre-existing one in 'git bugreport'. I decided against\nfixing *this* one because it didn't really fit into any of the patches in\nthis series, so it would need its own patch. When balancing \"leave things\nbetter than you found them\" vs. \"stay focused on the purpose of the series\",\nI leaned towards the latter to avoid setting a precedent for other 'git\nbugreport'-related scope creep.\n\nIf you have the patches to audit this sort of thing, I think a nice place to\nfix this might be in a dedicated series fixing discrepancies tree-wide. Even\nbetter, you could include the patches in your tree that detect them as part\nof e.g. the 'static-analysis' CI job.\n"},{"id":"461080","messageId":"9d1b0cb9-5c21-c101-8597-2fe166cb6abe@github.com","threadId":"58251","inReplyTo":"220811.86ilmzqchn.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 06/11] diagnose.c: add option to configure archive contents","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-11T15:43:47Z","receivedAt":"2022-08-11T15:52:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Aug 10 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> index 06dca69bdac..9bb6049bf0c 100644\n>> --- a/diagnose.h\n>> +++ b/diagnose.h\n>> @@ -2,7 +2,14 @@\n>> #define DIAGNOSE_H\n>> \n>> #include \"strbuf.h\"\n>> +#include \"parse-options.h\"\n> \n> This is a stray include that isn't needed at this point, some mistake,\n> or needed by a subsequent patch?\n\nIt's needed by patch 8 [1]. If I re-roll again, I'll move this '#include' to\nthat patch. Thanks!\n\n[1] https://lore.kernel.org/git/3da0cb725c927d08dd9486286e06bdb76896f5b7.1660174473.git.gitgitgadget@gmail.com/\n"},{"id":"461099","messageId":"220811.86fsi2plk7.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"f9a5f658-7c6f-bb3a-d6df-0ab7d2908395@github.com","subject":"Re: [PATCH v3 09/11] builtin/bugreport.c: create '--diagnose' option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-11T20:30:22Z","receivedAt":"2022-08-11T20:33:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 11 2022, Victoria Dye wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Wed, Aug 10 2022, Victoria Dye via GitGitGadget wrote:\n>> \n>>> From: Victoria Dye \n>>>\n>>> Create a '--diagnose' option for 'git bugreport' to collect additional\n>>> information about the repository and write it to a zipped archive.\n>>> [...]\n>>> 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n>>> +\t\t[--diagnose[=]]\n>>> [...]\n>>> static const char * const bugreport_usage[] = {\n>>> -\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n>>> +\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ] [--diagnose[=]\"),\n>>> \tNULL\n>>> };\n>> \n>> This still has the SYNOPSIS v.s. -h discrepancy noted in\n>> https://lore.kernel.org/git/220804.86v8r8ec4s.gmgdl@evledraar.gmail.com/\n>\n> The discrepancy you pointed out was on 'git diagnose' (which has since been\n> fixed),\n\nAh, sorry. I missed that & conflated the two.\n\n> this is a pre-existing one in 'git bugreport'. I decided against\n> fixing *this* one because it didn't really fit into any of the patches in\n> this series, so it would need its own patch. When balancing \"leave things\n> better than you found them\" vs. \"stay focused on the purpose of the series\",\n> I leaned towards the latter to avoid setting a precedent for other 'git\n> bugreport'-related scope creep.\n\nIn any case, I'm pointing out the difference in one of them having\n\\n-wrapping inconsistent with the other, which is an addition in this\nseries, sorry about not being clear.\n\nI see that there's also the difference in how they format \"--suffix\",\nbut that's pre-existing & we can leave it for now. I think that's what\nyou're pointing out here as pre-existing.\n\n> If you have the patches to audit this sort of thing, I think a nice place to\n> fix this might be in a dedicated series fixing discrepancies tree-wide. Even\n> better, you could include the patches in your tree that detect them as part\n> of e.g. the 'static-analysis' CI job.\n\nYeah I do have those, and will probably submit those sooner than later,\nand I'll end up spotting differences once they land on \"master\"\n(e.g. [1] is one such case).\n\nBut this one is just one I eyeballed during review.\n\n1. https://lore.kernel.org/git/220811.86o7wrov26.gmgdl@evledraar.gmail.com/\n"},{"id":"461126","messageId":"32f2cadc-556e-1cd5-a238-c8f1cdaaf470@github.com","threadId":"58251","inReplyTo":"xmqq8rnviqhz.fsf@gitster.g","subject":"Re: [PATCH v3 06/11] diagnose.c: add option to configure archive contents","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-12T17:00:30Z","receivedAt":"2022-08-12T17:00:38Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> @@ -177,11 +182,13 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n>> \tloose_objs_stats(&buf, \".git/objects\");\n>> \tstrvec_push(&archiver_args, buf.buf);\n>> \n>> -\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n>> -\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n>> -\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n>> -\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n>> -\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n>> +\t/* Only include this if explicitly requested */\n>> +\tif (mode == DIAGNOSE_ALL &&\n>> +\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n>> +\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n>> +\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n>> +\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n>> +\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n>> \t\tgoto diagnose_cleanup;\n> \n> At first glance, it looks as if this part fails silently, but\n> add_directory_to_archiver() states what failed there, so we show\n> necessary error messages and do not silently fail, which is good.\n> \n> There is a \"failed to write archive\" message after write_archive()\n> call returns non-zero, but presumably write_archive() itself gives\n> diagnostics (like \"oh, I was told to archive this file but I cannot\n> read it\") when it does so, so in a sense, giving the concluding\n> \"failed to write\" only in that case might make the error messages\n> uneven. If we fail to enlist \".git/hooks\" directory, we may want to\n> say why we failed to do so, and then want to see the concluding\n> \"failed to write\" at the end, just like the case where write_archive()\n> failed.\n> \n> It is a truely minor point, and if it turns out to be worth fixing,\n> it can be easily done by moving the diagnose_clean_up label a bit\n> higher, i.e.\n> \n> \t...\n> \tres = write_archive(...);\n> \n> diagnose_cleanup:\n> \tif (res)\n> \t\terror(_(\"failed to write archive\"));\n> \telse\n> \tfprintf(stderr, \"\\n\"\n> \t\t\t\"Diagnostics complete.\\n\"\n> \t\t\t\"All of the gathered info is captured in '%s'\\n\",\n> \t\t\tzip_path->buf);\n> \n> \tif (archiver_fd >= 0) {\n> \t\t... restore FD#1 and close stdout_fd and archiver_fd\n> \t}\n> \t...\n>\n\nI like this idea, since I think there's value in indicating both the cause\n(\"could not open directory\") and effect (\"failed to write archive\") of the\nerror. I'll include this and [1] in a re-roll. Thanks!\n\n[1] https://lore.kernel.org/git/9d1b0cb9-5c21-c101-8597-2fe166cb6abe@github.com/\n\n> \n> Other than that, this new patch looks good to me.\n> \n> Thanks.\n\n"},{"id":"461136","messageId":"f5ceb9c71903c41c47d715ec75ced9375c643ac3.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 01/11] scalar-diagnose: use \"$GIT_UNZIP\" in test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:09Z","receivedAt":"2022-08-12T20:10:28Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUse the \"$GIT_UNZIP\" test variable rather than verbatim 'unzip' to unzip the\n'scalar diagnose' archive. Using \"$GIT_UNZIP\" is needed to run the Scalar\ntests on systems where 'unzip' is not in the system path.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/t/t9099-scalar.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh\nindex 10b1172a8aa..fac86a57550 100755\n--- a/contrib/scalar/t/t9099-scalar.sh\n+++ b/contrib/scalar/t/t9099-scalar.sh\n@@ -109,14 +109,14 @@ test_expect_success UNZIP 'scalar diagnose' '\n \tsed -n \"s/.*$SQ\\\\(.*\\\\.zip\\\\)$SQ.*/\\\\1/p\" zip_path &&\n \tzip_path=$(cat zip_path) &&\n \ttest -n \"$zip_path\" &&\n-\tunzip -v \"$zip_path\" &&\n+\t\"$GIT_UNZIP\" -v \"$zip_path\" &&\n \tfolder=${zip_path%.zip} &&\n \ttest_path_is_missing \"$folder\" &&\n-\tunzip -p \"$zip_path\" diagnostics.log >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n \ttest_file_not_empty out &&\n-\tunzip -p \"$zip_path\" packs-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n \tgrep \"$(pwd)/.git/objects\" out &&\n-\tunzip -p \"$zip_path\" objects-local.txt >out &&\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n \tgrep \"^Total: [1-9]\" out\n '\n \n-- \ngitgitgadget\n\n"},{"id":"461137","messageId":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v3.git.1660174473.gitgitgadget@gmail.com","subject":"[PATCH v4 00/11] Generalize 'scalar diagnose' into 'git diagnose' and 'git bugreport --diagnose'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:08Z","receivedAt":"2022-08-12T20:10:29Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"As part of the preparation for moving Scalar out of 'contrib/' and into Git,\nthis series moves the functionality of 'scalar diagnose' into a new builtin\n('git diagnose') and a new option ('--diagnose') for 'git bugreport'. This\nchange further aligns Scalar with the objective [1] of having it only\ncontain functionality and settings that benefit large Git repositories, but\nnot all repositories. The diagnostics reported by 'scalar diagnose' relevant\nfor investigating issues in any Git repository, so generating them should be\npart of a \"normal\" Git builtin.\n\nThe series is organized as follows:\n\n * Miscellaneous fixes for the existing 'scalar diagnose' implementation\n * Moving the code for generating diagnostics into a common location in the\n Git tree\n * Implementing 'git diagnose'\n * Implementing 'git bugreport --diagnose'\n * Updating the Scalar roadmap\n\nFinally, despite 'scalar diagnose' now being nothing more than a wrapper for\n'git bugreport --diagnose', it is not being deprecated in this series.\nAlthough deprecation -> removal could be a future cleanup effort, 'scalar\ndiagnose' is kept around for now as an alias for users already accustomed to\nusing it in 'scalar'.\n\n\nChanges since V3\n================\n\n * Moved 'parse-options.h' import in 'diagnose.h' into the commit where it's\n first needed.\n * Removed a redundant 'if (!res)' condition gating the final \"Diagnostics\n complete\" message in 'create_diagnostics_archive()'.\n * Improved error reporting in 'create_diagnostics_archive()'. I was\n originally going to modify the \"failed to write archive\" error to trigger\n whenever 'create_diagnostics_archive()' returned a nonzero value [2].\n However, while working on it I realized the message would no longer be\n tied to a failure of 'write_archive()', making it less helpful in\n pinpointing an issue. To address the original issue\n ('add_directory_to_archiver()' silently failing in\n 'create_diagnostics_archive()'), I instead refactored those calls into a\n loop and added the error message. Now, there's exactly one error message\n printed for each possible early exit scenario from\n 'create_diagnostics_archive()', hopefully avoiding both redundancy &\n under-reporting.\n\n\nChanges since V2\n================\n\n * Replaced 'int include_everything' arg to 'create_diagnostic_archive()'\n with 'enum diagnose_mode mode'.\n * Replaced '--all' with configurable '--mode' option in 'git diagnose';\n moved 'option_parse_diagnose()' into 'diagnose.c' so that it can be used\n for both 'git bugreport --diagnose' and 'git diagnose --mode'.\n * Split \"builtin/diagnose.c: gate certain data behind '--all'\" (formerly\n patch 7/10) into \"diagnose.c: add option to configure archive contents\"\n (patch 6/11) and \"builtin/diagnose.c: add '--mode' option\" (patch 8/11).\n * Added '--no-diagnose' for 'git bugreport'. I was initially going to use\n '--diagnose=none', but '--no-diagnose' was easier to configure when using\n the shared 'option_parse_diagnose()' function .\n * Updated usage strings, option descriptions, and documentation files for\n 'mode' option. To avoid needing to keep multiple lists of valid 'mode'\n values up-to-date, format mode value as everywhere except option\n description in 'git-diagnose.txt', where the values are listed. The\n documentation of '--diagnose' in 'git-bugreport.txt' links to\n 'git-diagnose.txt' and explicitly calls out that details on 'mode' can be\n found there.\n * Reworded 'git diagnose' and 'git bugreport' command & option\n documentation.\n * Added additional checks to 't0091-bugreport.sh' and 't0092-diagnose.sh'\n tests\n * Moved '#include \"cache.h\" from 'diagnose.h' to 'diagnose.c'.\n * Fixed '--output-directory' usage string in 'builtin/diagnose.c'.\n * Replaced 'die()' with 'die_errno()' in error triggered when leading\n directories of archive cannot be created.\n * Changed hardcoded '-1' error exit code in 'scalar diagnose' to returning\n the exit code from 'git diagnose --mode=all'.\n\n\nChanges since V1\n================\n\n * Reorganized patches to fix minor issues (e.g., more gently adding\n directories to the archive) of 'scalar diagnose' in 'scalar.c', before\n the code is moved out of that file.\n * (Almost) entirely redesigned the UI for generating diagnostics. The new\n approach avoids cluttering 'git bugreport' with a mode that doesn't\n actually generate a report. Now, there are distinct options for different\n use cases: generating extra diagnostics with a bug report ('git bugreport\n --diagnose') and generating diagnostics for personal debugging/addition\n to an existing bug report ('git diagnose').\n * Moved 'get_disk_info()' into 'compat/'.\n * Moved 'create_diagnostics_archive()' into a new 'diagnose.c', as it now\n has multiple callers.\n * Updated command & option documentation to more clearly guide users on how\n to use the new options.\n * Added the '--all' (and '--diagnose=all') option to change the default\n behavior of diagnostics generation to exclude '.git' directory contents.\n For many bug reporters, this would reveal private repository contents\n they don't want to expose to the public mailing list. This has the added\n benefit of creating much smaller archives by default, which will be more\n likely to successfully send to the mailing list.\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1275.v2.git.1657584367.gitgitgadget@gmail.com/\n[2]\nhttps://lore.kernel.org/git/32f2cadc-556e-1cd5-a238-c8f1cdaaf470@github.com/\n\nVictoria Dye (11):\n scalar-diagnose: use \"$GIT_UNZIP\" in test\n scalar-diagnose: avoid 32-bit overflow of size_t\n scalar-diagnose: add directory to archiver more gently\n scalar-diagnose: move 'get_disk_info()' to 'compat/'\n scalar-diagnose: move functionality to common location\n diagnose.c: add option to configure archive contents\n builtin/diagnose.c: create 'git diagnose' builtin\n builtin/diagnose.c: add '--mode' option\n builtin/bugreport.c: create '--diagnose' option\n scalar-diagnose: use 'git diagnose --mode=all'\n scalar: update technical doc roadmap\n\n .gitignore | 1 +\n Documentation/git-bugreport.txt | 18 ++\n Documentation/git-diagnose.txt | 65 +++++++\n Documentation/technical/scalar.txt | 9 +-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/bugreport.c | 27 ++-\n builtin/diagnose.c | 61 +++++++\n compat/disk.h | 56 ++++++\n contrib/scalar/scalar.c | 271 +----------------------------\n contrib/scalar/t/t9099-scalar.sh | 8 +-\n diagnose.c | 269 ++++++++++++++++++++++++++++\n diagnose.h | 17 ++\n git-compat-util.h | 1 +\n git.c | 1 +\n t/t0091-bugreport.sh | 48 +++++\n t/t0092-diagnose.sh | 60 +++++++\n 17 files changed, 637 insertions(+), 278 deletions(-)\n create mode 100644 Documentation/git-diagnose.txt\n create mode 100644 builtin/diagnose.c\n create mode 100644 compat/disk.h\n create mode 100644 diagnose.c\n create mode 100644 diagnose.h\n create mode 100755 t/t0092-diagnose.sh\n\n\nbase-commit: 4af7188bc97f70277d0f10d56d5373022b1fa385\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1310%2Fvdye%2Fscalar%2Fgeneralize-diagnose-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1310/vdye/scalar/generalize-diagnose-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1310\n\nRange-diff vs v3:\n\n 1: f5ceb9c7190 = 1: f5ceb9c7190 scalar-diagnose: use \"$GIT_UNZIP\" in test\n 2: 78a93eb95bb = 2: 78a93eb95bb scalar-diagnose: avoid 32-bit overflow of size_t\n 3: 22ee8ea5a1e = 3: 22ee8ea5a1e scalar-diagnose: add directory to archiver more gently\n 4: 18f2ba4e0cd = 4: 18f2ba4e0cd scalar-diagnose: move 'get_disk_info()' to 'compat/'\n 5: 7a51fad87a8 ! 5: c19f3632d4f scalar-diagnose: move functionality to common location\n @@ diagnose.c (new)\n +\t\tgoto diagnose_cleanup;\n +\t}\n +\n -+\tif (!res)\n -+\t\tfprintf(stderr, \"\\n\"\n -+\t\t\t\"Diagnostics complete.\\n\"\n -+\t\t\t\"All of the gathered info is captured in '%s'\\n\",\n -+\t\t\tzip_path->buf);\n ++\tfprintf(stderr, \"\\n\"\n ++\t\t\"Diagnostics complete.\\n\"\n ++\t\t\"All of the gathered info is captured in '%s'\\n\",\n ++\t\tzip_path->buf);\n +\n +diagnose_cleanup:\n +\tif (archiver_fd >= 0) {\n 6: 0a6c55696d8 ! 6: 710b67e5776 diagnose.c: add option to configure archive contents\n @@ Commit message\n match existing functionality), but more callers will be introduced in\n subsequent patches.\n \n + Finally, refactor from a hardcoded set of 'add_directory_to_archiver()'\n + calls to iterative invocations gated by 'DIAGNOSE_ALL'. This allows for\n + easier future modification of the set of directories to archive and improves\n + error reporting when 'add_directory_to_archiver()' fails.\n +\n Helped-by: Derrick Stolee \n Signed-off-by: Victoria Dye \n \n @@ contrib/scalar/scalar.c: static int cmd_diagnose(int argc, const char **argv)\n \tstrbuf_release(&zip_path);\n \n ## diagnose.c ##\n +@@\n + #include \"object-store.h\"\n + #include \"packfile.h\"\n + \n ++struct archive_dir {\n ++\tconst char *path;\n ++\tint recursive;\n ++};\n ++\n + static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n + \t\t\t\t const char *file_name, void *data)\n + {\n @@ diagnose.c: static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n @@ diagnose.c: static int add_directory_to_archiver(struct strvec *archiver_args,\n {\n \tstruct strvec archiver_args = STRVEC_INIT;\n \tchar **argv_copy = NULL;\n -@@ diagnose.c: int create_diagnostics_archive(struct strbuf *zip_path)\n + \tint stdout_fd = -1, archiver_fd = -1;\n \tstruct strbuf buf = STRBUF_INIT;\n - \tint res;\n - \n +-\tint res;\n ++\tint res, i;\n ++\tstruct archive_dir archive_dirs[] = {\n ++\t\t{ \".git\", 0 },\n ++\t\t{ \".git/hooks\", 0 },\n ++\t\t{ \".git/info\", 0 },\n ++\t\t{ \".git/logs\", 1 },\n ++\t\t{ \".git/objects/info\", 0 }\n ++\t};\n ++\n +\tif (mode == DIAGNOSE_NONE) {\n +\t\tres = 0;\n +\t\tgoto diagnose_cleanup;\n +\t}\n -+\n + \n \tstdout_fd = dup(STDOUT_FILENO);\n \tif (stdout_fd < 0) {\n - \t\tres = error_errno(_(\"could not duplicate stdout\"));\n @@ diagnose.c: int create_diagnostics_archive(struct strbuf *zip_path)\n \tloose_objs_stats(&buf, \".git/objects\");\n \tstrvec_push(&archiver_args, buf.buf);\n @@ diagnose.c: int create_diagnostics_archive(struct strbuf *zip_path)\n -\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n -\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n -\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n +-\t\tgoto diagnose_cleanup;\n +\t/* Only include this if explicitly requested */\n -+\tif (mode == DIAGNOSE_ALL &&\n -+\t ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n -+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0))))\n - \t\tgoto diagnose_cleanup;\n ++\tif (mode == DIAGNOSE_ALL) {\n ++\t\tfor (i = 0; i < ARRAY_SIZE(archive_dirs); i++) {\n ++\t\t\tif (add_directory_to_archiver(&archiver_args,\n ++\t\t\t\t\t\t archive_dirs[i].path,\n ++\t\t\t\t\t\t archive_dirs[i].recursive)) {\n ++\t\t\t\tres = error_errno(_(\"could not add directory '%s' to archiver\"),\n ++\t\t\t\t\t\t archive_dirs[i].path);\n ++\t\t\t\tgoto diagnose_cleanup;\n ++\t\t\t}\n ++\t\t}\n ++\t}\n \n \tstrvec_pushl(&archiver_args, \"--prefix=\",\n + \t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n \n ## diagnose.h ##\n @@\n - #define DIAGNOSE_H\n \n #include \"strbuf.h\"\n -+#include \"parse-options.h\"\n \n -int create_diagnostics_archive(struct strbuf *zip_path);\n +enum diagnose_mode {\n 7: bf3c073a985 = 7: b58d13325b2 builtin/diagnose.c: create 'git diagnose' builtin\n 8: 3da0cb725c9 ! 8: 82be069e5e9 builtin/diagnose.c: add '--mode' option\n @@ builtin/diagnose.c: int cmd_diagnose(int argc, const char **argv, const char *pr\n \n \n ## diagnose.c ##\n -@@\n - #include \"object-store.h\"\n - #include \"packfile.h\"\n +@@ diagnose.c: struct archive_dir {\n + \tint recursive;\n + };\n \n +struct diagnose_option {\n +\tenum diagnose_mode mode;\n @@ diagnose.c\n {\n \n ## diagnose.h ##\n +@@\n + #define DIAGNOSE_H\n + \n + #include \"strbuf.h\"\n ++#include \"parse-options.h\"\n + \n + enum diagnose_mode {\n + \tDIAGNOSE_NONE,\n @@ diagnose.h: enum diagnose_mode {\n \tDIAGNOSE_ALL\n };\n 9: 1a1eb2c9806 = 9: 718e3f43484 builtin/bugreport.c: create '--diagnose' option\n 10: d22674752f0 = 10: 94b32eacdd5 scalar-diagnose: use 'git diagnose --mode=all'\n 11: b64475f5b17 = 11: 728f8b81fd0 scalar: update technical doc roadmap\n\n-- \ngitgitgadget\n"},{"id":"461138","messageId":"78a93eb95bb0c43778501f03837bffdcf74e4c99.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 02/11] scalar-diagnose: avoid 32-bit overflow of size_t","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:10Z","receivedAt":"2022-08-12T20:10:32Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAvoid 32-bit size_t overflow when reporting the available disk space in\n'get_disk_info' by casting the block size and available block count to\n'off_t' before multiplying them. Without this change, 'st_mult' would\n(correctly) report a size_t overflow on 32-bit systems at or exceeding 2^32\nbytes of available space.\n\nNote that 'off_t' is a 64-bit integer even on 32-bit systems due to the\ninclusion of '#define _FILE_OFFSET_BITS 64' in 'git-compat-util.h' (see\nb97e911643 (Support for large files on 32bit systems., 2007-02-17)).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 97e71fe19cd..04046452284 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -348,7 +348,7 @@ static int get_disk_info(struct strbuf *out)\n \t}\n \n \tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail));\n+\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n \tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n \tstrbuf_release(&buf);\n #endif\n-- \ngitgitgadget\n\n"},{"id":"461139","messageId":"22ee8ea5a1e04a42ad359be4eb5ebc96bd5e5fa2.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 03/11] scalar-diagnose: add directory to archiver more gently","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:11Z","receivedAt":"2022-08-12T20:10:35Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIf a directory added to the 'scalar diagnose' archiver does not exist, warn\nand return 0 from 'add_directory_to_archiver()' rather than failing with a\nfatal error. This handles a failure edge case where the '.git/logs' has not\nyet been created when running 'scalar diagnose', but extends to any\nsituation where a directory may be missing in the '.git' dir.\n\nNow, when a directory is missing a warning is captured in the diagnostic\nlogs. This provides a user with more complete information than if 'scalar\ndiagnose' simply failed with an error.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 04046452284..b9092f0b612 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -266,14 +266,20 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \t\t\t\t\t const char *path, int recurse)\n {\n \tint at_root = !*path;\n-\tDIR *dir = opendir(at_root ? \".\" : path);\n+\tDIR *dir;\n \tstruct dirent *e;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t len;\n \tint res = 0;\n \n-\tif (!dir)\n+\tdir = opendir(at_root ? \".\" : path);\n+\tif (!dir) {\n+\t\tif (errno == ENOENT) {\n+\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n+\t\t\treturn 0;\n+\t\t}\n \t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\t}\n \n \tif (!at_root)\n \t\tstrbuf_addf(&buf, \"%s/\", path);\n-- \ngitgitgadget\n\n"},{"id":"461140","messageId":"18f2ba4e0cdd9aea213dc7d0e908c6c0d8eccfd0.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 04/11] scalar-diagnose: move 'get_disk_info()' to 'compat/'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:12Z","receivedAt":"2022-08-12T20:10:38Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nMove 'get_disk_info()' function into 'compat/'. Although Scalar-specific\ncode is generally not part of the main Git tree, 'get_disk_info()' will be\nused in subsequent patches by additional callers beyond 'scalar diagnose'.\nThis patch prepares for that change, at which point this platform-specific\ncode should be part of 'compat/' as a matter of convention.\n\nThe function is copied *mostly* verbatim, with two exceptions:\n\n* '#ifdef WIN32' is replaced with '#ifdef GIT_WINDOWS_NATIVE' to allow\n 'statvfs' to be used with Cygwin.\n* the 'struct strbuf buf' and 'int res' (as well as their corresponding\n cleanup & return) are moved outside of the '#ifdef' block.\n\nSigned-off-by: Victoria Dye \n---\n compat/disk.h | 56 +++++++++++++++++++++++++++++++++++++++++\n contrib/scalar/scalar.c | 53 +-------------------------------------\n git-compat-util.h | 1 +\n 3 files changed, 58 insertions(+), 52 deletions(-)\n create mode 100644 compat/disk.h\n\ndiff --git a/compat/disk.h b/compat/disk.h\nnew file mode 100644\nindex 00000000000..50a32e3d8a4\n--- /dev/null\n+++ b/compat/disk.h\n@@ -0,0 +1,56 @@\n+#ifndef COMPAT_DISK_H\n+#define COMPAT_DISK_H\n+\n+#include \"git-compat-util.h\"\n+\n+static int get_disk_info(struct strbuf *out)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res = 0;\n+\n+#ifdef GIT_WINDOWS_NATIVE\n+\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n+\tDWORD serial_number, component_length, flags;\n+\tULARGE_INTEGER avail2caller, total, avail;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n+\t\terror(_(\"could not determine free disk size for '%s'\"),\n+\t\t buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n+\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n+\t\t\t\t &serial_number, &component_length, &flags,\n+\t\t\t\t fs_name, sizeof(fs_name))) {\n+\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n+\tstrbuf_addch(out, '\\n');\n+#else\n+\tstruct statvfs stat;\n+\n+\tstrbuf_realpath(&buf, \".\", 1);\n+\tif (statvfs(buf.buf, &stat) < 0) {\n+\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n+\t\t\t buf.buf);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n+\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n+\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n+#endif\n+\n+cleanup:\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+#endif /* COMPAT_DISK_H */\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex b9092f0b612..607fedefd82 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -13,6 +13,7 @@\n #include \"help.h\"\n #include \"archive.h\"\n #include \"object-store.h\"\n+#include \"compat/disk.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -309,58 +310,6 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n \n-#ifndef WIN32\n-#include \n-#endif\n-\n-static int get_disk_info(struct strbuf *out)\n-{\n-#ifdef WIN32\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tchar volume_name[MAX_PATH], fs_name[MAX_PATH];\n-\tDWORD serial_number, component_length, flags;\n-\tULARGE_INTEGER avail2caller, total, avail;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) {\n-\t\terror(_(\"could not determine free disk size for '%s'\"),\n-\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_setlen(&buf, offset_1st_component(buf.buf));\n-\tif (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name),\n-\t\t\t\t &serial_number, &component_length, &flags,\n-\t\t\t\t fs_name, sizeof(fs_name))) {\n-\t\terror(_(\"could not get info for '%s'\"), buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, avail2caller.QuadPart);\n-\tstrbuf_addch(out, '\\n');\n-\tstrbuf_release(&buf);\n-#else\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tstruct statvfs stat;\n-\n-\tstrbuf_realpath(&buf, \".\", 1);\n-\tif (statvfs(buf.buf, &stat) < 0) {\n-\t\terror_errno(_(\"could not determine free disk size for '%s'\"),\n-\t\t\t buf.buf);\n-\t\tstrbuf_release(&buf);\n-\t\treturn -1;\n-\t}\n-\n-\tstrbuf_addf(out, \"Available space on '%s': \", buf.buf);\n-\tstrbuf_humanise_bytes(out, (off_t)stat.f_bsize * (off_t)stat.f_bavail);\n-\tstrbuf_addf(out, \" (mount flags 0x%lx)\\n\", stat.f_flag);\n-\tstrbuf_release(&buf);\n-#endif\n-\treturn 0;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 58d7708296b..9a62e3a0d2d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -258,6 +258,7 @@ static inline int is_xplatform_dir_sep(int c)\n #include \n #include \n #include \n+#include \n #include \n #ifndef NO_SYS_SELECT_H\n #include \n-- \ngitgitgadget\n\n"},{"id":"461141","messageId":"c19f3632d4f2f966517a276e7096742c8477125c.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 05/11] scalar-diagnose: move functionality to common location","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:13Z","receivedAt":"2022-08-12T20:10:42Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nMove the core functionality of 'scalar diagnose' into a new 'diagnose.[c,h]'\nlibrary to prepare for new callers in the main Git tree generating\ndiagnostic archives. These callers will be introduced in subsequent patches.\n\nWhile this patch appears large, it is mostly made up of moving code out of\n'scalar.c' and into 'diagnose.c'. Specifically, the functions\n\n- dir_file_stats_objects()\n- dir_file_stats()\n- count_files()\n- loose_objs_stats()\n- add_directory_to_archiver()\n\nare all copied verbatim from 'scalar.c'. The 'create_diagnostics_archive()'\nfunction is a mostly identical (partial) copy of 'cmd_diagnose()', with the\nprimary changes being that 'zip_path' is an input and \"Enlistment root\" is\ncorrected to \"Repository root\" in the archiver log.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n Makefile | 1 +\n contrib/scalar/scalar.c | 202 +------------------------------------\n diagnose.c | 216 ++++++++++++++++++++++++++++++++++++++++\n diagnose.h | 8 ++\n 4 files changed, 227 insertions(+), 200 deletions(-)\n create mode 100644 diagnose.c\n create mode 100644 diagnose.h\n\ndiff --git a/Makefile b/Makefile\nindex 2ec9b2dc6bb..ed66cb70e5a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -932,6 +932,7 @@ LIB_OBJS += ctype.o\n LIB_OBJS += date.o\n LIB_OBJS += decorate.o\n LIB_OBJS += delta-islands.o\n+LIB_OBJS += diagnose.o\n LIB_OBJS += diff-delta.o\n LIB_OBJS += diff-merges.o\n LIB_OBJS += diff-lib.o\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 607fedefd82..3983def760a 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,9 +11,7 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"archive.h\"\n-#include \"object-store.h\"\n-#include \"compat/disk.h\"\n+#include \"diagnose.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -263,53 +261,6 @@ static int unregister_dir(void)\n \treturn res;\n }\n \n-static int add_directory_to_archiver(struct strvec *archiver_args,\n-\t\t\t\t\t const char *path, int recurse)\n-{\n-\tint at_root = !*path;\n-\tDIR *dir;\n-\tstruct dirent *e;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t len;\n-\tint res = 0;\n-\n-\tdir = opendir(at_root ? \".\" : path);\n-\tif (!dir) {\n-\t\tif (errno == ENOENT) {\n-\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n-\t\t\treturn 0;\n-\t\t}\n-\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n-\t}\n-\n-\tif (!at_root)\n-\t\tstrbuf_addf(&buf, \"%s/\", path);\n-\tlen = buf.len;\n-\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n-\n-\twhile (!res && (e = readdir(dir))) {\n-\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n-\t\t\tcontinue;\n-\n-\t\tstrbuf_setlen(&buf, len);\n-\t\tstrbuf_addstr(&buf, e->d_name);\n-\n-\t\tif (e->d_type == DT_REG)\n-\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n-\t\telse if (e->d_type != DT_DIR)\n-\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n-\t\t\t\t \"directory\"), buf.buf);\n-\t\telse if (recurse &&\n-\t\t\t add_directory_to_archiver(archiver_args,\n-\t\t\t\t\t\t buf.buf, recurse) < 0)\n-\t\t\tres = -1;\n-\t}\n-\n-\tclosedir(dir);\n-\tstrbuf_release(&buf);\n-\treturn res;\n-}\n-\n /* printf-style interface, expects `=` argument */\n static int set_config(const char *fmt, ...)\n {\n@@ -550,83 +501,6 @@ cleanup:\n \treturn res;\n }\n \n-static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n-\t\t\t\t const char *file_name, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\tstruct stat st;\n-\n-\tif (!stat(full_path, &st))\n-\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n-\t\t\t (uintmax_t)st.st_size);\n-}\n-\n-static int dir_file_stats(struct object_directory *object_dir, void *data)\n-{\n-\tstruct strbuf *buf = data;\n-\n-\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n-\n-\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n-\t\t\t\t data);\n-\n-\treturn 0;\n-}\n-\n-static int count_files(char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count = 0;\n-\n-\tif (!dir)\n-\t\treturn 0;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n-\t\t\tcount++;\n-\n-\tclosedir(dir);\n-\treturn count;\n-}\n-\n-static void loose_objs_stats(struct strbuf *buf, const char *path)\n-{\n-\tDIR *dir = opendir(path);\n-\tstruct dirent *e;\n-\tint count;\n-\tint total = 0;\n-\tunsigned char c;\n-\tstruct strbuf count_path = STRBUF_INIT;\n-\tsize_t base_path_len;\n-\n-\tif (!dir)\n-\t\treturn;\n-\n-\tstrbuf_addstr(buf, \"Object directory stats for \");\n-\tstrbuf_add_absolute_path(buf, path);\n-\tstrbuf_addstr(buf, \":\\n\");\n-\n-\tstrbuf_add_absolute_path(&count_path, path);\n-\tstrbuf_addch(&count_path, '/');\n-\tbase_path_len = count_path.len;\n-\n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name) &&\n-\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n-\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n-\t\t\tstrbuf_setlen(&count_path, base_path_len);\n-\t\t\tstrbuf_addstr(&count_path, e->d_name);\n-\t\t\ttotal += (count = count_files(count_path.buf));\n-\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n-\t\t}\n-\n-\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n-\n-\tstrbuf_release(&count_path);\n-\tclosedir(dir);\n-}\n-\n static int cmd_diagnose(int argc, const char **argv)\n {\n \tstruct option options[] = {\n@@ -637,12 +511,8 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tNULL\n \t};\n \tstruct strbuf zip_path = STRBUF_INIT;\n-\tstruct strvec archiver_args = STRVEC_INIT;\n-\tchar **argv_copy = NULL;\n-\tint stdout_fd = -1, archiver_fd = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n-\tstruct strbuf buf = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n@@ -663,79 +533,11 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\t\t zip_path.buf);\n \t\tgoto diagnose_cleanup;\n \t}\n-\tstdout_fd = dup(1);\n-\tif (stdout_fd < 0) {\n-\t\tres = error_errno(_(\"could not duplicate stdout\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tarchiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n-\tif (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) {\n-\t\tres = error_errno(_(\"could not redirect output\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n-\n-\tinit_zip_archiver();\n-\tstrvec_pushl(&archiver_args, \"scalar-diagnose\", \"--format=zip\", NULL);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n-\tget_version_info(&buf, 1);\n-\n-\tstrbuf_addf(&buf, \"Enlistment root: %s\\n\", the_repository->worktree);\n-\tget_disk_info(&buf);\n-\twrite_or_die(stdout_fd, buf.buf, buf.len);\n-\tstrvec_pushf(&archiver_args,\n-\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n-\t\t (int)buf.len, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n-\tdir_file_stats(the_repository->objects->odb, &buf);\n-\tforeach_alt_odb(dir_file_stats, &buf);\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tstrbuf_reset(&buf);\n-\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n-\tloose_objs_stats(&buf, \".git/objects\");\n-\tstrvec_push(&archiver_args, buf.buf);\n-\n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n-\t\tgoto diagnose_cleanup;\n-\n-\tstrvec_pushl(&archiver_args, \"--prefix=\",\n-\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n-\n-\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n-\targv_copy = xmemdupz(archiver_args.v,\n-\t\t\t sizeof(char *) * archiver_args.nr);\n-\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n-\t\t\t the_repository, NULL, 0);\n-\tif (res) {\n-\t\terror(_(\"failed to write archive\"));\n-\t\tgoto diagnose_cleanup;\n-\t}\n \n-\tif (!res)\n-\t\tfprintf(stderr, \"\\n\"\n-\t\t \"Diagnostics complete.\\n\"\n-\t\t \"All of the gathered info is captured in '%s'\\n\",\n-\t\t zip_path.buf);\n+\tres = create_diagnostics_archive(&zip_path);\n \n diagnose_cleanup:\n-\tif (archiver_fd >= 0) {\n-\t\tclose(1);\n-\t\tdup2(stdout_fd, 1);\n-\t}\n-\tfree(argv_copy);\n-\tstrvec_clear(&archiver_args);\n \tstrbuf_release(&zip_path);\n-\tstrbuf_release(&buf);\n-\n \treturn res;\n }\n \ndiff --git a/diagnose.c b/diagnose.c\nnew file mode 100644\nindex 00000000000..f0dcbfe1a2a\n--- /dev/null\n+++ b/diagnose.c\n@@ -0,0 +1,216 @@\n+#include \"cache.h\"\n+#include \"diagnose.h\"\n+#include \"compat/disk.h\"\n+#include \"archive.h\"\n+#include \"dir.h\"\n+#include \"help.h\"\n+#include \"strvec.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_name, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\tstruct stat st;\n+\n+\tif (!stat(full_path, &st))\n+\t\tstrbuf_addf(buf, \"%-70s %16\" PRIuMAX \"\\n\", file_name,\n+\t\t\t (uintmax_t)st.st_size);\n+}\n+\n+static int dir_file_stats(struct object_directory *object_dir, void *data)\n+{\n+\tstruct strbuf *buf = data;\n+\n+\tstrbuf_addf(buf, \"Contents of %s:\\n\", object_dir->path);\n+\n+\tfor_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects,\n+\t\t\t\t data);\n+\n+\treturn 0;\n+}\n+\n+static int count_files(char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count = 0;\n+\n+\tif (!dir)\n+\t\treturn 0;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG)\n+\t\t\tcount++;\n+\n+\tclosedir(dir);\n+\treturn count;\n+}\n+\n+static void loose_objs_stats(struct strbuf *buf, const char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *e;\n+\tint count;\n+\tint total = 0;\n+\tunsigned char c;\n+\tstruct strbuf count_path = STRBUF_INIT;\n+\tsize_t base_path_len;\n+\n+\tif (!dir)\n+\t\treturn;\n+\n+\tstrbuf_addstr(buf, \"Object directory stats for \");\n+\tstrbuf_add_absolute_path(buf, path);\n+\tstrbuf_addstr(buf, \":\\n\");\n+\n+\tstrbuf_add_absolute_path(&count_path, path);\n+\tstrbuf_addch(&count_path, '/');\n+\tbase_path_len = count_path.len;\n+\n+\twhile ((e = readdir(dir)) != NULL)\n+\t\tif (!is_dot_or_dotdot(e->d_name) &&\n+\t\t e->d_type == DT_DIR && strlen(e->d_name) == 2 &&\n+\t\t !hex_to_bytes(&c, e->d_name, 1)) {\n+\t\t\tstrbuf_setlen(&count_path, base_path_len);\n+\t\t\tstrbuf_addstr(&count_path, e->d_name);\n+\t\t\ttotal += (count = count_files(count_path.buf));\n+\t\t\tstrbuf_addf(buf, \"%s : %7d files\\n\", e->d_name, count);\n+\t\t}\n+\n+\tstrbuf_addf(buf, \"Total: %d loose objects\", total);\n+\n+\tstrbuf_release(&count_path);\n+\tclosedir(dir);\n+}\n+\n+static int add_directory_to_archiver(struct strvec *archiver_args,\n+\t\t\t\t const char *path, int recurse)\n+{\n+\tint at_root = !*path;\n+\tDIR *dir;\n+\tstruct dirent *e;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t len;\n+\tint res = 0;\n+\n+\tdir = opendir(at_root ? \".\" : path);\n+\tif (!dir) {\n+\t\tif (errno == ENOENT) {\n+\t\t\twarning(_(\"could not archive missing directory '%s'\"), path);\n+\t\t\treturn 0;\n+\t\t}\n+\t\treturn error_errno(_(\"could not open directory '%s'\"), path);\n+\t}\n+\n+\tif (!at_root)\n+\t\tstrbuf_addf(&buf, \"%s/\", path);\n+\tlen = buf.len;\n+\tstrvec_pushf(archiver_args, \"--prefix=%s\", buf.buf);\n+\n+\twhile (!res && (e = readdir(dir))) {\n+\t\tif (!strcmp(\".\", e->d_name) || !strcmp(\"..\", e->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&buf, len);\n+\t\tstrbuf_addstr(&buf, e->d_name);\n+\n+\t\tif (e->d_type == DT_REG)\n+\t\t\tstrvec_pushf(archiver_args, \"--add-file=%s\", buf.buf);\n+\t\telse if (e->d_type != DT_DIR)\n+\t\t\twarning(_(\"skipping '%s', which is neither file nor \"\n+\t\t\t\t \"directory\"), buf.buf);\n+\t\telse if (recurse &&\n+\t\t\t add_directory_to_archiver(archiver_args,\n+\t\t\t\t\t\t buf.buf, recurse) < 0)\n+\t\t\tres = -1;\n+\t}\n+\n+\tclosedir(dir);\n+\tstrbuf_release(&buf);\n+\treturn res;\n+}\n+\n+int create_diagnostics_archive(struct strbuf *zip_path)\n+{\n+\tstruct strvec archiver_args = STRVEC_INIT;\n+\tchar **argv_copy = NULL;\n+\tint stdout_fd = -1, archiver_fd = -1;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint res;\n+\n+\tstdout_fd = dup(STDOUT_FILENO);\n+\tif (stdout_fd < 0) {\n+\t\tres = error_errno(_(\"could not duplicate stdout\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tarchiver_fd = xopen(zip_path->buf, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n+\tif (dup2(archiver_fd, STDOUT_FILENO) < 0) {\n+\t\tres = error_errno(_(\"could not redirect output\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tinit_zip_archiver();\n+\tstrvec_pushl(&archiver_args, \"git-diagnose\", \"--format=zip\", NULL);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"Collecting diagnostic info\\n\\n\");\n+\tget_version_info(&buf, 1);\n+\n+\tstrbuf_addf(&buf, \"Repository root: %s\\n\", the_repository->worktree);\n+\tget_disk_info(&buf);\n+\twrite_or_die(stdout_fd, buf.buf, buf.len);\n+\tstrvec_pushf(&archiver_args,\n+\t\t \"--add-virtual-file=diagnostics.log:%.*s\",\n+\t\t (int)buf.len, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=packs-local.txt:\");\n+\tdir_file_stats(the_repository->objects->odb, &buf);\n+\tforeach_alt_odb(dir_file_stats, &buf);\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, \"--add-virtual-file=objects-local.txt:\");\n+\tloose_objs_stats(&buf, \".git/objects\");\n+\tstrvec_push(&archiver_args, buf.buf);\n+\n+\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n+\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n+\t\tgoto diagnose_cleanup;\n+\n+\tstrvec_pushl(&archiver_args, \"--prefix=\",\n+\t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\n+\n+\t/* `write_archive()` modifies the `argv` passed to it. Let it. */\n+\targv_copy = xmemdupz(archiver_args.v,\n+\t\t\t sizeof(char *) * archiver_args.nr);\n+\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n+\t\t\t the_repository, NULL, 0);\n+\tif (res) {\n+\t\terror(_(\"failed to write archive\"));\n+\t\tgoto diagnose_cleanup;\n+\t}\n+\n+\tfprintf(stderr, \"\\n\"\n+\t\t\"Diagnostics complete.\\n\"\n+\t\t\"All of the gathered info is captured in '%s'\\n\",\n+\t\tzip_path->buf);\n+\n+diagnose_cleanup:\n+\tif (archiver_fd >= 0) {\n+\t\tdup2(stdout_fd, STDOUT_FILENO);\n+\t\tclose(stdout_fd);\n+\t\tclose(archiver_fd);\n+\t}\n+\tfree(argv_copy);\n+\tstrvec_clear(&archiver_args);\n+\tstrbuf_release(&buf);\n+\n+\treturn res;\n+}\ndiff --git a/diagnose.h b/diagnose.h\nnew file mode 100644\nindex 00000000000..06dca69bdac\n--- /dev/null\n+++ b/diagnose.h\n@@ -0,0 +1,8 @@\n+#ifndef DIAGNOSE_H\n+#define DIAGNOSE_H\n+\n+#include \"strbuf.h\"\n+\n+int create_diagnostics_archive(struct strbuf *zip_path);\n+\n+#endif /* DIAGNOSE_H */\n-- \ngitgitgadget\n\n"},{"id":"461142","messageId":"710b67e5776363d199ed5043d019386819d44e7e.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 06/11] diagnose.c: add option to configure archive contents","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:14Z","receivedAt":"2022-08-12T20:10:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate 'create_diagnostics_archive()' to take an argument 'mode'. When\narchiving diagnostics for a repository, 'mode' is used to selectively\ninclude/exclude information based on its value. The initial options for\n'mode' are:\n\n* DIAGNOSE_NONE: do not collect any diagnostics or create an archive\n (no-op).\n* DIAGNOSE_STATS: collect basic repository metadata (Git version, repo path,\n filesystem available space) as well as sizing and count statistics for the\n repository's objects and packfiles.\n* DIAGNOSE_ALL: collect basic repository metadata, sizing/count statistics,\n and copies of the '.git', '.git/hooks', '.git/info', '.git/logs', and\n '.git/objects/info' directories.\n\nThese modes are introduced to provide users the option to collect\ndiagnostics without the sensitive information included in copies of '.git'\ndir contents. At the moment, only 'scalar diagnose' uses\n'create_diagnostics_archive()' (with a hardcoded 'DIAGNOSE_ALL' mode to\nmatch existing functionality), but more callers will be introduced in\nsubsequent patches.\n\nFinally, refactor from a hardcoded set of 'add_directory_to_archiver()'\ncalls to iterative invocations gated by 'DIAGNOSE_ALL'. This allows for\neasier future modification of the set of directories to archive and improves\nerror reporting when 'add_directory_to_archiver()' fails.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 2 +-\n diagnose.c | 39 +++++++++++++++++++++++++++++++--------\n diagnose.h | 8 +++++++-\n 3 files changed, 39 insertions(+), 10 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex 3983def760a..d538b8b8f14 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -534,7 +534,7 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tgoto diagnose_cleanup;\n \t}\n \n-\tres = create_diagnostics_archive(&zip_path);\n+\tres = create_diagnostics_archive(&zip_path, DIAGNOSE_ALL);\n \n diagnose_cleanup:\n \tstrbuf_release(&zip_path);\ndiff --git a/diagnose.c b/diagnose.c\nindex f0dcbfe1a2a..9270056db2f 100644\n--- a/diagnose.c\n+++ b/diagnose.c\n@@ -8,6 +8,11 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n+struct archive_dir {\n+\tconst char *path;\n+\tint recursive;\n+};\n+\n static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n \t\t\t\t const char *file_name, void *data)\n {\n@@ -132,13 +137,25 @@ static int add_directory_to_archiver(struct strvec *archiver_args,\n \treturn res;\n }\n \n-int create_diagnostics_archive(struct strbuf *zip_path)\n+int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode)\n {\n \tstruct strvec archiver_args = STRVEC_INIT;\n \tchar **argv_copy = NULL;\n \tint stdout_fd = -1, archiver_fd = -1;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tint res;\n+\tint res, i;\n+\tstruct archive_dir archive_dirs[] = {\n+\t\t{ \".git\", 0 },\n+\t\t{ \".git/hooks\", 0 },\n+\t\t{ \".git/info\", 0 },\n+\t\t{ \".git/logs\", 1 },\n+\t\t{ \".git/objects/info\", 0 }\n+\t};\n+\n+\tif (mode == DIAGNOSE_NONE) {\n+\t\tres = 0;\n+\t\tgoto diagnose_cleanup;\n+\t}\n \n \tstdout_fd = dup(STDOUT_FILENO);\n \tif (stdout_fd < 0) {\n@@ -177,12 +194,18 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n \tloose_objs_stats(&buf, \".git/objects\");\n \tstrvec_push(&archiver_args, buf.buf);\n \n-\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n-\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n-\t\tgoto diagnose_cleanup;\n+\t/* Only include this if explicitly requested */\n+\tif (mode == DIAGNOSE_ALL) {\n+\t\tfor (i = 0; i < ARRAY_SIZE(archive_dirs); i++) {\n+\t\t\tif (add_directory_to_archiver(&archiver_args,\n+\t\t\t\t\t\t archive_dirs[i].path,\n+\t\t\t\t\t\t archive_dirs[i].recursive)) {\n+\t\t\t\tres = error_errno(_(\"could not add directory '%s' to archiver\"),\n+\t\t\t\t\t\t archive_dirs[i].path);\n+\t\t\t\tgoto diagnose_cleanup;\n+\t\t\t}\n+\t\t}\n+\t}\n \n \tstrvec_pushl(&archiver_args, \"--prefix=\",\n \t\t oid_to_hex(the_hash_algo->empty_tree), \"--\", NULL);\ndiff --git a/diagnose.h b/diagnose.h\nindex 06dca69bdac..998775857a0 100644\n--- a/diagnose.h\n+++ b/diagnose.h\n@@ -3,6 +3,12 @@\n \n #include \"strbuf.h\"\n \n-int create_diagnostics_archive(struct strbuf *zip_path);\n+enum diagnose_mode {\n+\tDIAGNOSE_NONE,\n+\tDIAGNOSE_STATS,\n+\tDIAGNOSE_ALL\n+};\n+\n+int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode);\n \n #endif /* DIAGNOSE_H */\n-- \ngitgitgadget\n\n"},{"id":"461143","messageId":"b58d13325b2eaef32fac89fb2be8e29004560c45.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 07/11] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:15Z","receivedAt":"2022-08-12T20:10:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a 'git diagnose' builtin to generate a standalone zip archive of\nrepository diagnostics.\n\nThe \"diagnose\" functionality was originally implemented for Scalar in\naa5c79a331 (scalar: implement `scalar diagnose`, 2022-05-28). However, the\ndiagnostics gathered are not specific to Scalar-cloned repositories and\ncan be useful when diagnosing issues in any Git repository.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n .gitignore | 1 +\n Documentation/git-diagnose.txt | 50 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diagnose.c | 57 ++++++++++++++++++++++++++++++++++\n git.c | 1 +\n t/t0092-diagnose.sh | 32 +++++++++++++++++++\n 7 files changed, 143 insertions(+)\n create mode 100644 Documentation/git-diagnose.txt\n create mode 100644 builtin/diagnose.c\n create mode 100755 t/t0092-diagnose.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 42fd7253b44..80b530bbed2 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -53,6 +53,7 @@\n /git-cvsimport\n /git-cvsserver\n /git-daemon\n+/git-diagnose\n /git-diff\n /git-diff-files\n /git-diff-index\ndiff --git a/Documentation/git-diagnose.txt b/Documentation/git-diagnose.txt\nnew file mode 100644\nindex 00000000000..ce07dd0725d\n--- /dev/null\n+++ b/Documentation/git-diagnose.txt\n@@ -0,0 +1,50 @@\n+git-diagnose(1)\n+================\n+\n+NAME\n+----\n+git-diagnose - Generate a zip archive of diagnostic information\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\n+DESCRIPTION\n+-----------\n+Collects detailed information about the user's machine, Git client, and\n+repository state and packages that information into a zip archive. The\n+generated archive can then, for example, be shared with the Git mailing list to\n+help debug an issue or serve as a reference for independent debugging.\n+\n+The following information is captured in the archive:\n+\n+ * 'git version --build-options'\n+ * The path to the repository root\n+ * The available disk space on the filesystem\n+ * The name and size of each packfile, including those in alternate object\n+ stores\n+ * The total count of loose objects, as well as counts broken down by\n+ `.git/objects` subdirectory\n+\n+This tool differs from linkgit:git-bugreport[1] in that it collects much more\n+detailed information with a greater focus on reporting the size and data shape\n+of repository contents.\n+\n+OPTIONS\n+-------\n+-o ::\n+--output-directory ::\n+\tPlace the resulting diagnostics archive in `` instead of the\n+\tcurrent directory.\n+\n+-s ::\n+--suffix ::\n+\tSpecify an alternate suffix for the diagnostics archive name, to create\n+\ta file named 'git-diagnostics-'. This should take the\n+\tform of a strftime(3) format string; the current local time will be\n+\tused.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex ed66cb70e5a..d34f680c065 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1153,6 +1153,7 @@ BUILTIN_OBJS += builtin/credential-cache.o\n BUILTIN_OBJS += builtin/credential-store.o\n BUILTIN_OBJS += builtin/credential.o\n BUILTIN_OBJS += builtin/describe.o\n+BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n BUILTIN_OBJS += builtin/diff-tree.o\ndiff --git a/builtin.h b/builtin.h\nindex 40e9ecc8485..8901a34d6bf 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -144,6 +144,7 @@ int cmd_credential_cache(int argc, const char **argv, const char *prefix);\n int cmd_credential_cache_daemon(int argc, const char **argv, const char *prefix);\n int cmd_credential_store(int argc, const char **argv, const char *prefix);\n int cmd_describe(int argc, const char **argv, const char *prefix);\n+int cmd_diagnose(int argc, const char **argv, const char *prefix);\n int cmd_diff_files(int argc, const char **argv, const char *prefix);\n int cmd_diff_index(int argc, const char **argv, const char *prefix);\n int cmd_diff(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/diagnose.c b/builtin/diagnose.c\nnew file mode 100644\nindex 00000000000..832493bba65\n--- /dev/null\n+++ b/builtin/diagnose.c\n@@ -0,0 +1,57 @@\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"diagnose.h\"\n+\n+static const char * const diagnose_usage[] = {\n+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n+\tNULL\n+};\n+\n+int cmd_diagnose(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct strbuf zip_path = STRBUF_INIT;\n+\ttime_t now = time(NULL);\n+\tstruct tm tm;\n+\tchar *option_output = NULL;\n+\tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n+\tchar *prefixed_filename;\n+\n+\tconst struct option diagnose_options[] = {\n+\t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n+\t\t\t N_(\"specify a destination for the diagnostics archive\")),\n+\t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n+\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, diagnose_options,\n+\t\t\t diagnose_usage, 0);\n+\n+\t/* Prepare the path to put the result */\n+\tprefixed_filename = prefix_filename(prefix,\n+\t\t\t\t\t option_output ? option_output : \"\");\n+\tstrbuf_addstr(&zip_path, prefixed_filename);\n+\tstrbuf_complete(&zip_path, '/');\n+\n+\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\tswitch (safe_create_leading_directories(zip_path.buf)) {\n+\tcase SCLD_OK:\n+\tcase SCLD_EXISTS:\n+\t\tbreak;\n+\tdefault:\n+\t\tdie_errno(_(\"could not create leading directories for '%s'\"),\n+\t\t\t zip_path.buf);\n+\t}\n+\n+\t/* Prepare diagnostics */\n+\tif (create_diagnostics_archive(&zip_path, DIAGNOSE_STATS))\n+\t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n+\t\t\t zip_path.buf);\n+\n+\tfree(prefixed_filename);\n+\tstrbuf_release(&zip_path);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex e5d62fa5a92..0b9d8ef7677 100644\n--- a/git.c\n+++ b/git.c\n@@ -522,6 +522,7 @@ static struct cmd_struct commands[] = {\n \t{ \"credential-cache--daemon\", cmd_credential_cache_daemon },\n \t{ \"credential-store\", cmd_credential_store },\n \t{ \"describe\", cmd_describe, RUN_SETUP },\n+\t{ \"diagnose\", cmd_diagnose, RUN_SETUP_GENTLY },\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\ndiff --git a/t/t0092-diagnose.sh b/t/t0092-diagnose.sh\nnew file mode 100755\nindex 00000000000..b6923726fd7\n--- /dev/null\n+++ b/t/t0092-diagnose.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='git diagnose'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success UNZIP 'creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test >out &&\n+\tgrep \"Available space\" out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out &&\n+\n+\t# Should not include .git directory contents by default\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+\tgrep \"^Total: [0-9][0-9]*\" out\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"461144","messageId":"718e3f434841ee4697a505d83f2e4acbeed364fd.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 09/11] builtin/bugreport.c: create '--diagnose' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:17Z","receivedAt":"2022-08-12T20:10:59Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate a '--diagnose' option for 'git bugreport' to collect additional\ninformation about the repository and write it to a zipped archive.\n\nThe '--diagnose' option behaves effectively as an alias for simultaneously\nrunning 'git bugreport' and 'git diagnose'. In the documentation, users are\nexplicitly recommended to attach the diagnostics alongside a bug report to\nprovide additional context to readers, ideally reducing some back-and-forth\nbetween reporters and those debugging the issue.\n\nNote that '--diagnose' may take an optional string arg (either 'stats' or\n'all'). If specified without the arg, the behavior corresponds to running\n'git diagnose' without '--mode'. As with 'git diagnose', this default is\nintended to help reduce unintentional leaking of sensitive information).\nUsers can also explicitly specify '--diagnose=(stats|all)' to generate the\nrespective archive created by 'git diagnose --mode=(stats|all)'.\n\nSuggested-by: Ævar Arnfjörð Bjarmason \nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n Documentation/git-bugreport.txt | 18 +++++++++++++\n builtin/bugreport.c | 27 ++++++++++++++++---\n t/t0091-bugreport.sh | 48 +++++++++++++++++++++++++++++++++\n 3 files changed, 90 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-bugreport.txt b/Documentation/git-bugreport.txt\nindex d8817bf3cec..eca726e5791 100644\n--- a/Documentation/git-bugreport.txt\n+++ b/Documentation/git-bugreport.txt\n@@ -9,6 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git bugreport' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\t\t[--diagnose[=]]\n \n DESCRIPTION\n -----------\n@@ -31,6 +32,10 @@ The following information is captured automatically:\n - A list of enabled hooks\n - $SHELL\n \n+Additional information may be gathered into a separate zip archive using the\n+`--diagnose` option, and can be attached alongside the bugreport document to\n+provide additional context to readers.\n+\n This tool is invoked via the typical Git setup process, which means that in some\n cases, it might not be able to launch - for example, if a relevant config file\n is unreadable. In this kind of scenario, it may be helpful to manually gather\n@@ -49,6 +54,19 @@ OPTIONS\n \tnamed 'git-bugreport-'. This should take the form of a\n \tstrftime(3) format string; the current local time will be used.\n \n+--no-diagnose::\n+--diagnose[=]::\n+\tCreate a zip archive of supplemental information about the user's\n+\tmachine, Git client, and repository state. The archive is written to the\n+\tsame output directory as the bug report and is named\n+\t'git-diagnostics-'.\n++\n+Without `mode` specified, the diagnostic archive will contain the default set of\n+statistics reported by `git diagnose`. An optional `mode` value may be specified\n+to change which information is included in the archive. See\n+linkgit:git-diagnose[1] for the list of valid values for `mode` and details\n+about their usage.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 9de32bc96e7..530895be55f 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -5,6 +5,7 @@\n #include \"compat/compiler.h\"\n #include \"hook.h\"\n #include \"hook-list.h\"\n+#include \"diagnose.h\"\n \n \n static void get_system_info(struct strbuf *sys_info)\n@@ -59,7 +60,7 @@ static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n }\n \n static const char * const bugreport_usage[] = {\n-\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ]\"),\n+\tN_(\"git bugreport [-o|--output-directory ] [-s|--suffix ] [--diagnose[=]\"),\n \tNULL\n };\n \n@@ -98,16 +99,21 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \tint report = -1;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tenum diagnose_mode diagnose = DIAGNOSE_NONE;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tconst char *user_relative_path = NULL;\n \tchar *prefixed_filename;\n+\tsize_t output_path_len;\n \n \tconst struct option bugreport_options[] = {\n+\t\tOPT_CALLBACK_F(0, \"diagnose\", &diagnose, N_(\"mode\"),\n+\t\t\t N_(\"create an additional zip archive of detailed diagnostics (default 'stats')\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_diagnose),\n \t\tOPT_STRING('o', \"output-directory\", &option_output, N_(\"path\"),\n-\t\t\t N_(\"specify a destination for the bugreport file\")),\n+\t\t\t N_(\"specify a destination for the bugreport file(s)\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n-\t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\t\t N_(\"specify a strftime format suffix for the filename(s)\")),\n \t\tOPT_END()\n \t};\n \n@@ -119,6 +125,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t option_output ? option_output : \"\");\n \tstrbuf_addstr(&report_path, prefixed_filename);\n \tstrbuf_complete(&report_path, '/');\n+\toutput_path_len = report_path.len;\n \n \tstrbuf_addstr(&report_path, \"git-bugreport-\");\n \tstrbuf_addftime(&report_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n@@ -133,6 +140,20 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix)\n \t\t report_path.buf);\n \t}\n \n+\t/* Prepare diagnostics, if requested */\n+\tif (diagnose != DIAGNOSE_NONE) {\n+\t\tstruct strbuf zip_path = STRBUF_INIT;\n+\t\tstrbuf_add(&zip_path, report_path.buf, output_path_len);\n+\t\tstrbuf_addstr(&zip_path, \"git-diagnostics-\");\n+\t\tstrbuf_addftime(&zip_path, option_suffix, localtime_r(&now, &tm), 0, 0);\n+\t\tstrbuf_addstr(&zip_path, \".zip\");\n+\n+\t\tif (create_diagnostics_archive(&zip_path, diagnose))\n+\t\t\tdie_errno(_(\"unable to create diagnostics archive %s\"), zip_path.buf);\n+\n+\t\tstrbuf_release(&zip_path);\n+\t}\n+\n \t/* Prepare the report contents */\n \tget_bug_template(&buffer);\n \ndiff --git a/t/t0091-bugreport.sh b/t/t0091-bugreport.sh\nindex 08f5fe9caef..b6d2f591acd 100755\n--- a/t/t0091-bugreport.sh\n+++ b/t/t0091-bugreport.sh\n@@ -78,4 +78,52 @@ test_expect_success 'indicates populated hooks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success UNZIP '--diagnose creates diagnostics zip archive' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose -o report -s test >out &&\n+\n+\tzip_path=report/git-diagnostics-test.zip &&\n+\tgrep \"Available space\" out &&\n+\ttest_path_is_file \"$zip_path\" &&\n+\n+\t# Check zipped archive content\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" diagnostics.log >out &&\n+\ttest_file_not_empty out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" objects-local.txt >out &&\n+\tgrep \"^Total: [0-9][0-9]*\" out &&\n+\n+\t# Should not include .git directory contents by default\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--diagnose=stats excludes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose=stats -o report -s test >out &&\n+\n+\t# Includes pack quantity/size info\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t# Does not include .git directory contents\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--diagnose=all includes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit bugreport --diagnose=all -o report -s test >out &&\n+\n+\t# Includes .git directory contents\n+\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n+\ttest_file_not_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"461145","messageId":"94b32eacdd565df665c40bc49929c128b93cc626.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 10/11] scalar-diagnose: use 'git diagnose --mode=all'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:18Z","receivedAt":"2022-08-12T20:11:00Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nReplace implementation of 'scalar diagnose' with an internal invocation of\n'git diagnose --mode=all'. This simplifies the implementation of\n'cmd_diagnose' by making it a direct alias of 'git diagnose' and removes\nsome code in 'scalar.c' that is duplicated in 'builtin/diagnose.c'. The\nsimplicity of the alias also sets up a clean deprecation path for 'scalar\ndiagnose' (in favor of 'git diagnose'), if that is desired in the future.\n\nThis introduces one minor change to the output of 'scalar diagnose', which\nis that the prefix of the created zip archive is changed from 'scalar_' to\n'git-diagnostics-'.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.c | 28 ++++++----------------------\n 1 file changed, 6 insertions(+), 22 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c\nindex d538b8b8f14..68571ce195f 100644\n--- a/contrib/scalar/scalar.c\n+++ b/contrib/scalar/scalar.c\n@@ -11,7 +11,6 @@\n #include \"dir.h\"\n #include \"packfile.h\"\n #include \"help.h\"\n-#include \"diagnose.h\"\n \n /*\n * Remove the deepest subdirectory in the provided path string. Path must not\n@@ -510,34 +509,19 @@ static int cmd_diagnose(int argc, const char **argv)\n \t\tN_(\"scalar diagnose []\"),\n \t\tNULL\n \t};\n-\tstruct strbuf zip_path = STRBUF_INIT;\n-\ttime_t now = time(NULL);\n-\tstruct tm tm;\n+\tstruct strbuf diagnostics_root = STRBUF_INIT;\n \tint res = 0;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\t usage, 0);\n \n-\tsetup_enlistment_directory(argc, argv, usage, options, &zip_path);\n-\n-\tstrbuf_addstr(&zip_path, \"/.scalarDiagnostics/scalar_\");\n-\tstrbuf_addftime(&zip_path,\n-\t\t\t\"%Y%m%d_%H%M%S\", localtime_r(&now, &tm), 0, 0);\n-\tstrbuf_addstr(&zip_path, \".zip\");\n-\tswitch (safe_create_leading_directories(zip_path.buf)) {\n-\tcase SCLD_EXISTS:\n-\tcase SCLD_OK:\n-\t\tbreak;\n-\tdefault:\n-\t\terror_errno(_(\"could not create directory for '%s'\"),\n-\t\t\t zip_path.buf);\n-\t\tgoto diagnose_cleanup;\n-\t}\n+\tsetup_enlistment_directory(argc, argv, usage, options, &diagnostics_root);\n+\tstrbuf_addstr(&diagnostics_root, \"/.scalarDiagnostics\");\n \n-\tres = create_diagnostics_archive(&zip_path, DIAGNOSE_ALL);\n+\tres = run_git(\"diagnose\", \"--mode=all\", \"-s\", \"%Y%m%d_%H%M%S\",\n+\t\t \"-o\", diagnostics_root.buf, NULL);\n \n-diagnose_cleanup:\n-\tstrbuf_release(&zip_path);\n+\tstrbuf_release(&diagnostics_root);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"461146","messageId":"82be069e5e965c7193f15bb38ef523dc4b7b3592.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 08/11] builtin/diagnose.c: add '--mode' option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:16Z","receivedAt":"2022-08-12T20:11:03Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nCreate '--mode=' option in 'git diagnose' to allow users to optionally\nselect non-default diagnostic information to include in the output archive.\nAdditionally, document the currently-available modes, emphasizing the\nimportance of not sharing a '--mode=all' archive publicly due to the\npresence of sensitive information.\n\nNote that the option parsing callback - 'option_parse_diagnose()' - is added\nto 'diagnose.c' rather than 'builtin/diagnose.c' so that it may be reused in\nfuture callers configuring a diagnostics archive.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n Documentation/git-diagnose.txt | 17 ++++++++++++++++-\n builtin/diagnose.c | 8 ++++++--\n diagnose.c | 30 ++++++++++++++++++++++++++++++\n diagnose.h | 3 +++\n t/t0092-diagnose.sh | 30 +++++++++++++++++++++++++++++-\n 5 files changed, 84 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-diagnose.txt b/Documentation/git-diagnose.txt\nindex ce07dd0725d..3ec8cc7ad72 100644\n--- a/Documentation/git-diagnose.txt\n+++ b/Documentation/git-diagnose.txt\n@@ -9,6 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n+\t [--mode=]\n \n DESCRIPTION\n -----------\n@@ -17,7 +18,7 @@ repository state and packages that information into a zip archive. The\n generated archive can then, for example, be shared with the Git mailing list to\n help debug an issue or serve as a reference for independent debugging.\n \n-The following information is captured in the archive:\n+By default, the following information is captured in the archive:\n \n * 'git version --build-options'\n * The path to the repository root\n@@ -27,6 +28,9 @@ The following information is captured in the archive:\n * The total count of loose objects, as well as counts broken down by\n `.git/objects` subdirectory\n \n+Additional information can be collected by selecting a different diagnostic mode\n+using the `--mode` option.\n+\n This tool differs from linkgit:git-bugreport[1] in that it collects much more\n detailed information with a greater focus on reporting the size and data shape\n of repository contents.\n@@ -45,6 +49,17 @@ OPTIONS\n \tform of a strftime(3) format string; the current local time will be\n \tused.\n \n+--mode=(stats|all)::\n+\tSpecify the type of diagnostics that should be collected. The default behavior\n+\tof 'git diagnose' is equivalent to `--mode=stats`.\n++\n+The `--mode=all` option collects everything included in `--mode=stats`, as well\n+as copies of `.git`, `.git/hooks`, `.git/info`, `.git/logs`, and\n+`.git/objects/info` directories. This additional information may be sensitive,\n+as it can be used to reconstruct the full contents of the diagnosed repository.\n+Users should exercise caution when sharing an archive generated with\n+`--mode=all`.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/diagnose.c b/builtin/diagnose.c\nindex 832493bba65..cd260c20155 100644\n--- a/builtin/diagnose.c\n+++ b/builtin/diagnose.c\n@@ -3,7 +3,7 @@\n #include \"diagnose.h\"\n \n static const char * const diagnose_usage[] = {\n-\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n+\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ] [--mode=]\"),\n \tNULL\n };\n \n@@ -12,6 +12,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \tstruct strbuf zip_path = STRBUF_INIT;\n \ttime_t now = time(NULL);\n \tstruct tm tm;\n+\tenum diagnose_mode mode = DIAGNOSE_STATS;\n \tchar *option_output = NULL;\n \tchar *option_suffix = \"%Y-%m-%d-%H%M\";\n \tchar *prefixed_filename;\n@@ -21,6 +22,9 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"specify a destination for the diagnostics archive\")),\n \t\tOPT_STRING('s', \"suffix\", &option_suffix, N_(\"format\"),\n \t\t\t N_(\"specify a strftime format suffix for the filename\")),\n+\t\tOPT_CALLBACK_F(0, \"mode\", &mode, N_(\"(stats|all)\"),\n+\t\t\t N_(\"specify the content of the diagnostic archive\"),\n+\t\t\t PARSE_OPT_NONEG, option_parse_diagnose),\n \t\tOPT_END()\n \t};\n \n@@ -47,7 +51,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix)\n \t}\n \n \t/* Prepare diagnostics */\n-\tif (create_diagnostics_archive(&zip_path, DIAGNOSE_STATS))\n+\tif (create_diagnostics_archive(&zip_path, mode))\n \t\tdie_errno(_(\"unable to create diagnostics archive %s\"),\n \t\t\t zip_path.buf);\n \ndiff --git a/diagnose.c b/diagnose.c\nindex 9270056db2f..beb0a8741ba 100644\n--- a/diagnose.c\n+++ b/diagnose.c\n@@ -13,6 +13,36 @@ struct archive_dir {\n \tint recursive;\n };\n \n+struct diagnose_option {\n+\tenum diagnose_mode mode;\n+\tconst char *option_name;\n+};\n+\n+static struct diagnose_option diagnose_options[] = {\n+\t{ DIAGNOSE_STATS, \"stats\" },\n+\t{ DIAGNOSE_ALL, \"all\" },\n+};\n+\n+int option_parse_diagnose(const struct option *opt, const char *arg, int unset)\n+{\n+\tint i;\n+\tenum diagnose_mode *diagnose = opt->value;\n+\n+\tif (!arg) {\n+\t\t*diagnose = unset ? DIAGNOSE_NONE : DIAGNOSE_STATS;\n+\t\treturn 0;\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(diagnose_options); i++) {\n+\t\tif (!strcmp(arg, diagnose_options[i].option_name)) {\n+\t\t\t*diagnose = diagnose_options[i].mode;\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn error(_(\"invalid --%s value '%s'\"), opt->long_name, arg);\n+}\n+\n static void dir_file_stats_objects(const char *full_path, size_t full_path_len,\n \t\t\t\t const char *file_name, void *data)\n {\ndiff --git a/diagnose.h b/diagnose.h\nindex 998775857a0..7a4951a7863 100644\n--- a/diagnose.h\n+++ b/diagnose.h\n@@ -2,6 +2,7 @@\n #define DIAGNOSE_H\n \n #include \"strbuf.h\"\n+#include \"parse-options.h\"\n \n enum diagnose_mode {\n \tDIAGNOSE_NONE,\n@@ -9,6 +10,8 @@ enum diagnose_mode {\n \tDIAGNOSE_ALL\n };\n \n+int option_parse_diagnose(const struct option *opt, const char *arg, int unset);\n+\n int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode);\n \n #endif /* DIAGNOSE_H */\ndiff --git a/t/t0092-diagnose.sh b/t/t0092-diagnose.sh\nindex b6923726fd7..fca9b58489c 100755\n--- a/t/t0092-diagnose.sh\n+++ b/t/t0092-diagnose.sh\n@@ -26,7 +26,35 @@ test_expect_success UNZIP 'creates diagnostics zip archive' '\n \n \t# Should not include .git directory contents by default\n \t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n-\tgrep \"^Total: [0-9][0-9]*\" out\n+'\n+\n+test_expect_success UNZIP '--mode=stats excludes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test --mode=stats >out &&\n+\n+\t# Includes pack quantity/size info\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t# Does not include .git directory contents\n+\t! \"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\"\n+'\n+\n+test_expect_success UNZIP '--mode=all includes .git dir contents' '\n+\ttest_when_finished rm -rf report &&\n+\n+\tgit diagnose -o report -s test --mode=all >out &&\n+\n+\t# Includes pack quantity/size info\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" packs-local.txt >out &&\n+\tgrep \".git/objects\" out &&\n+\n+\t# Includes .git directory contents\n+\t\"$GIT_UNZIP\" -l \"$zip_path\" | grep \".git/\" &&\n+\n+\t\"$GIT_UNZIP\" -p \"$zip_path\" .git/HEAD >out &&\n+\ttest_file_not_empty out\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"461147","messageId":"728f8b81fd09a3d46f87d98a6eac38c46765d688.1660335019.git.gitgitgadget@gmail.com","threadId":"58251","inReplyTo":"pull.1310.v4.git.1660335019.gitgitgadget@gmail.com","subject":"[PATCH v4 11/11] scalar: update technical doc roadmap","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-12T20:10:19Z","receivedAt":"2022-08-12T20:11:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate the Scalar roadmap to reflect the completion of generalizing 'scalar\ndiagnose' into 'git diagnose' and 'git bugreport --diagnose'.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/technical/scalar.txt | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt\nindex 08bc09c225a..f6353375f08 100644\n--- a/Documentation/technical/scalar.txt\n+++ b/Documentation/technical/scalar.txt\n@@ -84,6 +84,9 @@ series have been accepted:\n \n - `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n \n+- `scalar-generalize-diagnose`: Move the functionality of `scalar diagnose`\n+ into `git diagnose` and `git bugreport --diagnose`.\n+\n Roughly speaking (and subject to change), the following series are needed to\n \"finish\" this initial version of Scalar:\n \n@@ -91,12 +94,6 @@ Roughly speaking (and subject to change), the following series are needed to\n and implement `scalar help`. At the end of this series, Scalar should be\n feature-complete from the perspective of a user.\n \n-- Generalize features not specific to Scalar: In the spirit of making Scalar\n- configure only what is needed for large repo performance, move common\n- utilities into other parts of Git. Some of this will be internal-only, but one\n- major change will be generalizing `scalar diagnose` for use with any Git\n- repository.\n-\n - Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of\n `git`, including updates to build and install it with the rest of Git. This\n change will incorporate Scalar into the Git CI and test framework, as well as\n-- \ngitgitgadget\n"},{"id":"461148","messageId":"xmqq7d3db43z.fsf@gitster.g","threadId":"58251","inReplyTo":"c19f3632d4f2f966517a276e7096742c8477125c.1660335019.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 05/11] scalar-diagnose: move functionality to common location","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-12T20:26:24Z","receivedAt":"2022-08-12T20:28:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> +\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n> +\t\t\t the_repository, NULL, 0);\n> +\tif (res) {\n> +\t\terror(_(\"failed to write archive\"));\n> +\t\tgoto diagnose_cleanup;\n> +\t}\n> +\n> +\tfprintf(stderr, \"\\n\"\n> +\t\t\"Diagnostics complete.\\n\"\n> +\t\t\"All of the gathered info is captured in '%s'\\n\",\n> +\t\tzip_path->buf);\n> +\n> +diagnose_cleanup:\n> +\tif (archiver_fd >= 0) {\n> +\t\tdup2(stdout_fd, STDOUT_FILENO);\n> +\t\tclose(stdout_fd);\n> +\t\tclose(archiver_fd);\n> +\t}\n\nHmph, after reading \n\nhttps://lore.kernel.org/git/32f2cadc-556e-1cd5-a238-c8f1cdaaf470@github.com/\n\nI would have expected to see the above part more like:\n\n res = write_archive(...);\n\n diagnose_cleanup:\n if (res)\n error(...);\n else\n fprintf(stderr, \"Diag complete\");\n\n if (archiver_fd >= 0) {\n ...\n\n"},{"id":"461149","messageId":"xmqq35e1b3up.fsf@gitster.g","threadId":"58251","inReplyTo":"710b67e5776363d199ed5043d019386819d44e7e.1660335019.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/11] diagnose.c: add option to configure archive contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-12T20:31:58Z","receivedAt":"2022-08-12T20:32:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> -int create_diagnostics_archive(struct strbuf *zip_path)\n> +int create_diagnostics_archive(struct strbuf *zip_path, enum diagnose_mode mode)\n> {\n> \tstruct strvec archiver_args = STRVEC_INIT;\n> \tchar **argv_copy = NULL;\n> \tint stdout_fd = -1, archiver_fd = -1;\n> \tstruct strbuf buf = STRBUF_INIT;\n> -\tint res;\n> +\tint res, i;\n> +\tstruct archive_dir archive_dirs[] = {\n> +\t\t{ \".git\", 0 },\n> +\t\t{ \".git/hooks\", 0 },\n> +\t\t{ \".git/info\", 0 },\n> +\t\t{ \".git/logs\", 1 },\n> +\t\t{ \".git/objects/info\", 0 }\n> +\t};\n> +\n> +\tif (mode == DIAGNOSE_NONE) {\n> +\t\tres = 0;\n> +\t\tgoto diagnose_cleanup;\n> +\t}\n> \n> \tstdout_fd = dup(STDOUT_FILENO);\n> \tif (stdout_fd < 0) {\n> @@ -177,12 +194,18 @@ int create_diagnostics_archive(struct strbuf *zip_path)\n> \tloose_objs_stats(&buf, \".git/objects\");\n> \tstrvec_push(&archiver_args, buf.buf);\n> \n> -\tif ((res = add_directory_to_archiver(&archiver_args, \".git\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/hooks\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/info\", 0)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/logs\", 1)) ||\n> -\t (res = add_directory_to_archiver(&archiver_args, \".git/objects/info\", 0)))\n> -\t\tgoto diagnose_cleanup;\n> +\t/* Only include this if explicitly requested */\n> +\tif (mode == DIAGNOSE_ALL) {\n> +\t\tfor (i = 0; i < ARRAY_SIZE(archive_dirs); i++) {\n> +\t\t\tif (add_directory_to_archiver(&archiver_args,\n> +\t\t\t\t\t\t archive_dirs[i].path,\n> +\t\t\t\t\t\t archive_dirs[i].recursive)) {\n> +\t\t\t\tres = error_errno(_(\"could not add directory '%s' to archiver\"),\n> +\t\t\t\t\t\t archive_dirs[i].path);\n> +\t\t\t\tgoto diagnose_cleanup;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n\nEven without the \"only include under DIAGNOSE_ALL\" support added by\nthis step, the table-driven organization is much nicer. The earlier\n\"move to common\" step aimed to be as close as pure move, so this\nstep is our first opportunity to make this clean-up, so I do not\nmind too much about this step doing two unrelated things (one is to\nclean-up the if (A||B||C) into a loop over A, B and C, the other is\nto introduce the diagnose_mode) at once.\n\nThanks.\n\n\n"},{"id":"461151","messageId":"a728c894-4e4a-44c3-a1f5-f87c8bfae13a@github.com","threadId":"58251","inReplyTo":"xmqq7d3db43z.fsf@gitster.g","subject":"Re: [PATCH v4 05/11] scalar-diagnose: move functionality to common location","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-12T21:00:40Z","receivedAt":"2022-08-12T21:00:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> +\tres = write_archive(archiver_args.nr, (const char **)argv_copy, NULL,\n>> +\t\t\t the_repository, NULL, 0);\n>> +\tif (res) {\n>> +\t\terror(_(\"failed to write archive\"));\n>> +\t\tgoto diagnose_cleanup;\n>> +\t}\n>> +\n>> +\tfprintf(stderr, \"\\n\"\n>> +\t\t\"Diagnostics complete.\\n\"\n>> +\t\t\"All of the gathered info is captured in '%s'\\n\",\n>> +\t\tzip_path->buf);\n>> +\n>> +diagnose_cleanup:\n>> +\tif (archiver_fd >= 0) {\n>> +\t\tdup2(stdout_fd, STDOUT_FILENO);\n>> +\t\tclose(stdout_fd);\n>> +\t\tclose(archiver_fd);\n>> +\t}\n> \n> Hmph, after reading \n> \n> https://lore.kernel.org/git/32f2cadc-556e-1cd5-a238-c8f1cdaaf470@github.com/\n> \n> I would have expected to see the above part more like:\n> \n> res = write_archive(...);\n> \n> diagnose_cleanup:\n> if (res)\n> error(...);\n> else\n> fprintf(stderr, \"Diag complete\");\n> \n> if (archiver_fd >= 0) {\n> ...\n> \n\nI originally planned to implement it this way, but instead went with adding\nan error printout explicitly for failed 'add_directory_to_archiver()' calls\n(in patch 6 [1]). I elaborated on the thought process/reasoning for\nmodifying the approach in the cover letter [2]:\n\n> Improved error reporting in 'create_diagnostics_archive()'. I was\n> originally going to modify the \"failed to write archive\" error to trigger\n> whenever 'create_diagnostics_archive()' returned a nonzero value.\n> However, while working on it I realized the message would no longer be\n> tied to a failure of 'write_archive()', making it less helpful in\n> pinpointing an issue. To address the original issue\n> ('add_directory_to_archiver()' silently failing in\n> 'create_diagnostics_archive()'), I instead refactored those calls into a\n> loop and added the error message. Now, there's exactly one error message\n> printed for each possible early exit scenario from\n> 'create_diagnostics_archive()', hopefully avoiding both redundancy &\n> under-reporting.\n\nTo add a bit more context: when I used the \"move 'diagnose_cleanup'\"\napproach, I felt that the added message was either redundant or too general\nto help a user identify an issue. Redundancy appeared when, for example,\n'dup2()' returned a nonzero code; if that happened, we'd get the printouts:\n\n$ git diagnose --suffix test\nerror: could not redirect output: \nerror: could not write archive\nerror: unable to create diagnostics archive 'git-diagnostics-test.zip': \n\nThe first two are from 'create_diagnostics_archive()', and the second\ndoesn't really give us information that we don't get out of the third (in\n'builtin/diagnose.c').\n\nConversely, a failure in 'add_directory_to_archiver()' and 'write_archive()'\nwould give us the same printouts (at least within the scope of\n'diagnose.c'/'builtin/diagnose.c'):\n\n$ git diagnose --suffix test\nerror: could not write archive\nerror: unable to create diagnostics archive 'git-diagnostics-test.zip: \n\nPreviously, \"could not write archive\" would point someone debugging to\n'write_archive()'; now, it's unclear.\n\nI ended up settling on adding the error message directly to the\n'add_directory_to_archiver()' loop in patch 6 because it meant that:\n\n1. 'create_diagnostics_archive()' would only ever print one error message;\n the others printed would indicate (IMO more clearly) where in the call\n stack the error happened\n2. there would be a unique error message for each condition that caused\n 'create_diagnostics_archive()' to exit early\n\nApologies for not sending another reply with these details before\nre-rolling. I'll be more direct when changing plans in the future.\n\nThanks!\n\n[1] https://lore.kernel.org/git/710b67e5776363d199ed5043d019386819d44e7e.1660335019.git.gitgitgadget@gmail.com/\n[2] https://lore.kernel.org/git/pull.1310.v4.git.1660335019.gitgitgadget@gmail.com/\n"},{"id":"461155","messageId":"xmqqk07d9n0p.fsf@gitster.g","threadId":"58251","inReplyTo":"a728c894-4e4a-44c3-a1f5-f87c8bfae13a@github.com","subject":"Re: [PATCH v4 05/11] scalar-diagnose: move functionality to common location","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-12T21:20:54Z","receivedAt":"2022-08-12T21:21:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n>> Improved error reporting in 'create_diagnostics_archive()'. I was\n>> originally going to modify the \"failed to write archive\" error to trigger\n>> whenever 'create_diagnostics_archive()' returned a nonzero value.\n>> However, while working on it I realized the message would no longer be\n>> tied to a failure of 'write_archive()', making it less helpful in\n>> pinpointing an issue. To address the original issue\n>> ('add_directory_to_archiver()' silently failing in\n>> 'create_diagnostics_archive()'), I instead refactored those calls into a\n>> loop and added the error message. Now, there's exactly one error message\n>> printed for each possible early exit scenario from\n>> 'create_diagnostics_archive()', hopefully avoiding both redundancy &\n>> under-reporting.\n\nAh, I see. I probably should have read the cover letter before\nresponding. I try to understand the new iteration _without_ relying\non the cover letter first, to ensure that the resulting history is\nstill understandable; when I see something questionable, however, I\nshould see if cover letter gives more context and clues. Sorry for\nthe noise.\n\n"},{"id":"461456","messageId":"220818.86edxd8k8q.gmgdl@evledraar.gmail.com","threadId":"58251","inReplyTo":"b58d13325b2eaef32fac89fb2be8e29004560c45.1660335019.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 07/11] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-18T18:43:22Z","receivedAt":"2022-08-18T18:45:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Aug 12 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n> [...]\n\nThis is correct:\n\n> +'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n\n...\n\n> +\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n\nBut this is not, it's missing () around the short v.s. long option, and\nwe should have a space surrounding the \"|\" as well.\n"},{"id":"461461","messageId":"xmqqzgg1nz6v.fsf@gitster.g","threadId":"58251","inReplyTo":"220818.86edxd8k8q.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 07/11] builtin/diagnose.c: create 'git diagnose' builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-18T19:12:24Z","receivedAt":"2022-08-18T19:12:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Fri, Aug 12 2022, Victoria Dye via GitGitGadget wrote:\n>\n>> From: Victoria Dye \n>> [...]\n>\n> This is correct:\n>\n>> +'git diagnose' [(-o | --output-directory) ] [(-s | --suffix) ]\n>\n> ...\n>\n>> +\tN_(\"git diagnose [-o|--output-directory ] [-s|--suffix ]\"),\n>\n> But this is not, it's missing () around the short v.s. long option, and\n> we should have a space surrounding the \"|\" as well.\n\nYou are commenting on what appears inside N_() and I agree that it\nshould match the other one.\n\nIt is kind of sad that our usage strings do not allow proper\ntranslation and instead force translaters to _know_ that they are\nsupposed to touch _only_ the placeholder strings (path and format in\nthis case).\n\nNot your fault, or Victoria's, of course ;-)\n"}]}