{"thread":{"id":"56595","subject":"What's cooking in git.git (Sep 2021, #08; Mon, 27)","startedAt":"2021-09-28T00:52:06Z","lastAt":"2021-10-08T23:01:48Z","messageCount":40,"participants":["Junio C Hamano","Ævar Arnfjörð Bjarmason","Elijah Newren","Han-Wen Nienhuys","Derrick Stolee","Neeraj Singh","Jeff King","Fabian Stelzer","Carlo Arenas"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"437220","messageId":"xmqq8rzhmsi7.fsf@gitster.g","threadId":"56595","inReplyTo":null,"subject":"What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T00:52:00Z","receivedAt":"2021-09-28T00:52:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Here are the topics that have been cooking in my tree.  Commits\nprefixed with '+' are in 'next' (being in 'next' is a sign that a\ntopic is stable enough to be used and are candidate to be in a\nfuture release).  Commits prefixed with '-' are only in 'seen',\nwhich means nothing more than that I have found them of interest for\nsome reason (like \"it may have hard-to-resolve conflicts with\nanother topic already in flight\" or \"this may turn out to be\nuseful\").  Do not read too much into a topic being in (or not in)\n'seen'.  The ones marked with '.' do not appear in any of the\nintegration branches, but I am still holding onto them.\n\n\n\nCopies of the source code to Git live in many repositories, and the\nfollowing is a list of the ones I push into or their mirrors.  Some\nrepositories have only a subset of branches.\n\nWith maint, master, next, seen, todo:\n\n\tgit://git.kernel.org/pub/scm/git/git.git/\n\tgit://repo.or.cz/alt-git.git/\n\thttps://kernel.googlesource.com/pub/scm/git/git/\n\thttps://github.com/git/git/\n\thttps://gitlab.com/git-vcs/git/\n\nWith all the integration branches and topics broken out:\n\n\thttps://github.com/gitster/git/\n\nEven though the preformatted documentation in HTML and man format\nare not sources, they are published in these repositories for\nconvenience (replace \"htmldocs\" with \"manpages\" for the manual\npages):\n\n\tgit://git.kernel.org/pub/scm/git/git-htmldocs.git/\n\thttps://github.com/gitster/git-htmldocs.git/\n\nRelease tarballs are available at:\n\n\thttps://www.kernel.org/pub/software/scm/git/\n\n--------------------------------------------------\n[New Topics]\n\n* ab/bundle-remove-verbose-option (2021-09-23) 1 commit\n - bundle: remove ignored & undocumented \"--verbose\" flag\n\n Doc update.\n\n Will merge to 'next'.\n\n\n* ab/config-based-hooks-1 (2021-09-27) 8 commits\n - hook-list.h: add a generated list of hooks, like config-list.h\n - hook.c users: use \"hook_exists()\" instead of \"find_hook()\"\n - hook.c: add a hook_exists() wrapper and use it in bugreport.c\n - hook.[ch]: move find_hook() from run-command.c to hook.c\n - Makefile: remove an out-of-date comment\n - Makefile: don't perform \"mv $@+ $@\" dance for $(GENERATED_H)\n - Makefile: stop hardcoding {command,config}-list.h\n - Makefile: mark \"check\" target as .PHONY\n\n\n* ab/http-pinned-public-key-mismatch (2021-09-27) 1 commit\n - http: check CURLE_SSL_PINNEDPUBKEYNOTMATCH when emitting errors\n\n HTTPS error handling updates.\n\n Will merge to 'next'?\n\n\n* ab/make-compdb-fix (2021-09-27) 1 commit\n - Makefile: pass -Wno-pendantic under GENERATE_COMPILATION_DATABASE=yes\n\n Build update.\n\n Will merge to 'next'.\n\n\n* jk/ref-paranoia (2021-09-27) 16 commits\n - refs: drop \"broken\" flag from for_each_fullref_in()\n - ref-filter: drop broken-ref code entirely\n - ref-filter: stop setting FILTER_REFS_INCLUDE_BROKEN\n - repack, prune: drop GIT_REF_PARANOIA settings\n - refs: turn on GIT_REF_PARANOIA by default\n - refs: omit dangling symrefs when using GIT_REF_PARANOIA\n - refs: add DO_FOR_EACH_OMIT_DANGLING_SYMREFS flag\n - refs-internal.h: reorganize DO_FOR_EACH_* flag documentation\n - refs-internal.h: move DO_FOR_EACH_* flags next to each other\n - t5312: be more assertive about command failure\n - t5312: test non-destructive repack\n - t5312: create bogus ref as necessary\n - t5312: drop \"verbose\" helper\n - t5600: provide detached HEAD for corruption failures\n - t5516: don't use HEAD ref for invalid ref-deletion tests\n - t7900: clean up some more broken refs\n\n The ref iteration code used to optionally allow dangling refs to be\n shown, which has been tightened up.\n\n Will merge to 'next'?\n\n\n* pw/rebase-reread-todo-after-editing (2021-09-24) 2 commits\n - rebase: fix todo-list rereading\n - sequencer.c: factor out a function\n\n The code to re-read the edited todo list in \"git rebase -i\" was\n made more robust.\n\n Will merge to 'next'.\n\n\n* rs/close-pack-leakfix (2021-09-24) 1 commit\n - packfile: release bad_objects in close_pack()\n\n Leakfix.\n\n Will merge to 'next'.\n\n--------------------------------------------------\n[Stalled]\n\n* ab/pack-objects-stdin (2021-07-09) 5 commits\n - pack-objects.c: make use of REV_INFO_STDIN_LINE_PROCESS\n - pack-objects.c: do stdin parsing via revision.c's API\n - revision.[ch]: add a \"handle_stdin_line\" API\n - revision.h: refactor \"disable_stdin\" and \"read_from_stdin\"\n - upload-pack: run is_repository_shallow() before setup_revisions()\n\n Introduce handle_stdin_line callback to revision API and uses it.\n\n Expecting a reroll.\n\n\n* es/superproject-aware-submodules (2021-08-19) 5 commits\n . fixup! introduce submodule.superprojectGitDir record\n . submodule: record superproject gitdir during 'update'\n . submodule: record superproject gitdir during absorbgitdirs\n . introduce submodule.superprojectGitDir record\n . t7400-submodule-basic: modernize inspect() helper\n\n A configuration variable in a submodule points at the location of\n the superproject it is bound to (RFC).\n\n Kicked out of 'seen' tentatively to see how well a conflicting\n ar/submodule-update topic fares.\n\n\n* ab/fsck-unexpected-type (2021-09-22) 17 commits\n - fsck: report invalid object type-path combinations\n - fsck: don't hard die on invalid object types\n - object-file.c: stop dying in parse_loose_header()\n - object-file.c: return ULHR_TOO_LONG on \"header too long\"\n - object-file.c: use \"enum\" return type for unpack_loose_header()\n - object-file.c: simplify unpack_loose_short_header()\n - object-file.c: make parse_loose_header_extended() public\n - object-file.c: return -1, not \"status\" from unpack_loose_header()\n - object-file.c: don't set \"typep\" when returning non-zero\n - cat-file tests: test for current --allow-unknown-type behavior\n - cat-file tests: add corrupt loose object test\n - cat-file tests: test for missing/bogus object with -t, -s and -p\n - cat-file tests: move bogus_* variable declarations earlier\n - fsck tests: test for garbage appended to a loose object\n - fsck tests: test current hash/type mismatch behavior\n - fsck tests: refactor one test to use a sub-repo\n - fsck tests: add test for fsck-ing an unknown type\n\n \"git fsck\" has been taught to report mismatch between expected and\n actual types of an object better.\n\n Needs review.\n\n--------------------------------------------------\n[Cooking]\n\n* ab/make-clean-depend-dirs (2021-09-22) 1 commit\n - Makefile: clean .depend dirs under COMPUTE_HEADER_DEPENDENCIES != yes\n\n \"make clean\" has been updated to remove leftover .depend/\n directories, even when it is not told to use them to compute header\n dependencies.\n\n Will merge to 'next'.\n\n\n* bs/ls-files-opt-help-text-update (2021-09-22) 1 commit\n - ls-files: use imperative mood for -X and -z option description\n\n Help test for \"ls-files\" options have been updated.\n\n Will merge to 'next'.\n\n\n* ds/perf-test-built-path-fix (2021-09-22) 1 commit\n - t/perf/run: fix bin-wrappers computation\n\n Perf test fix.\n\n Will merge to 'next'.\n\n\n* gc/doc-first-contribution-reroll (2021-09-22) 1 commit\n - MyFirstContribution: Document --range-diff option when writing v2\n\n Doc update.\n\n Will merge to 'next'.\n\n\n* jk/grep-haystack-is-read-only (2021-09-22) 5 commits\n - grep: store grep_source buffer as const\n - grep: mark \"haystack\" buffers as const\n - grep: stop modifying buffer in grep_source_1()\n - grep: stop modifying buffer in show_line()\n - grep: stop modifying buffer in strip_timestamp\n (this branch is used by hm/paint-hits-in-log-grep.)\n\n Code clean-up in the \"grep\" machinery.\n\n Will merge to 'next'.\n\n\n* jk/http-redact-fix (2021-09-22) 1 commit\n - http: match headers case-insensitively when redacting\n\n Sensitive data in the HTTP trace were supposed to be redacted, but\n we failed to do so in HTTP/2 requests.\n\n Will merge to 'next'.\n\n\n* js/win-lazyload-buildfix (2021-09-27) 2 commits\n - lazyload.h: use an even more generic function pointer than FARPROC\n - lazyload.h: fix warnings about mismatching function pointer types\n\n Compilation fix.\n\n Will merge to 'next'.\n\n\n* ab/auto-depend-with-pedantic (2021-09-22) 1 commit\n - Makefile: make COMPUTE_HEADER_DEPENDENCIES=auto work with DEVOPTS=pedantic\n\n Improve build procedure for developers.\n\n Will merge to 'next'.\n\n\n* ab/make-sparse-for-real (2021-09-22) 1 commit\n - Makefile: make the \"sparse\" target non-.PHONY\n\n Prevent \"make sparse\" from running for the source files that\n haven't been modified.\n\n Will merge to 'next'?\n\n\n* bs/difftool-msg-tweak (2021-09-22) 1 commit\n - difftool: fix word spacing in the usage strings\n\n Message tweak.\n\n Will merge to 'next'.\n\n\n* ew/midx-doc-update (2021-09-24) 1 commit\n - doc/technical: update note about core.multiPackIndex\n\n Doc tweak.\n\n Will merge to 'next'.\n\n\n* jt/no-abuse-alternate-odb-for-submodules (2021-09-22) 10 commits\n . submodule: trace adding submodule ODB as alternate\n . refs: change refs_for_each_ref_in() to take repo\n . submodule: pass repo to check_has_commit()\n . object-file: only register submodule ODB if needed\n . merge-{ort,recursive}: remove add_submodule_odb()\n . refs: teach refs_for_each_ref() arbitrary repos\n . refs iterator: support non-the_repository advance\n . refs: add repo paramater to _iterator_peel()\n . refs: make _advance() check struct repo, not flag\n . Merge branch 'jt/add-submodule-odb-clean-up' into jt/no-abuse-alternate-odb-for-submodules\n (this branch uses jt/add-submodule-odb-clean-up.)\n\n Follow through the work to use the repo interface to access\n submodule objects in-process, instead of abusing the alternate\n object database interface.\n\n Expecting a reroll on top of the jk/ref-paranoia topic.\n\n\n* tp/send-email-completion (2021-09-22) 3 commits\n - send-email docs: add format-patch options\n - send-email: move bash completions to core script\n - send-email: terminate --git-completion-helper with LF\n\n The command line complation for \"git send-email\" options have been\n tweaked to make it easier to keep it in sync with the command itself.\n\n\n* hm/paint-hits-in-log-grep (2021-09-24) 3 commits\n - pretty: colorize pattern matches in commit messages\n - grep: refactor next_match() and match_one_pattern() for external use\n - Merge branch 'jk/grep-haystack-is-read-only' into hm/paint-hits-in-log-grep\n (this branch uses jk/grep-haystack-is-read-only.)\n\n \"git log --grep=string --author=name\" learns to highlight hits just\n like \"git grep string\" does.\n\n\n* ab/repo-settings-cleanup (2021-09-22) 5 commits\n - repository.h: don't use a mix of int and bitfields\n - repo-settings.c: simplify the setup\n - read-cache & fetch-negotiator: check \"enum\" values in switch()\n - environment.c: remove test-specific \"ignore_untracked...\" variable\n - wrapper.c: add x{un,}setenv(), and use xsetenv() in environment.c\n\n Code cleanup.\n\n Will merge to 'next'.\n\n\n* ah/connect-parse-feature-v0-fix (2021-09-27) 1 commit\n - connect: also update offset for features without values\n\n Protocol v0 clients can get stuck parsing a malformed feature line.\n\n Will merge to 'next'.\n\n\n* ah/unreak-revisions (2021-09-20) 2 commits\n - log: UNLEAK original pending objects\n - log: UNLEAK rev to silence a large number of leaks\n\n Mark a few structures with UNLEAK() to help leak detection CI jobs.\n\n Will merge to 'next'.\n\n\n* ar/submodule-update (2021-09-20) 8 commits\n - submodule--helper: rename helper functions\n - submodule--helper: remove unused helpers\n - submodule--helper: remove update-clone subcommand\n - submodule: move core cmd_update() logic to C\n - submodule--helper: refactor get_submodule_displaypath()\n - submodule--helper: rename helpers for update-clone\n - submodule--helper: get remote names from any repository\n - submodule--helper: split up ensure_core_worktree()\n\n Rewrite of \"git submodule update\" in C.\n\n Expecting a reroll.\n Stomps on a handful of other topics and comes with an overly large step.\n\n\n* da/difftool (2021-09-23) 4 commits\n - difftool: add a missing space to the run_dir_diff() comments\n - difftool: create a tmpdir path without repeated slashes\n - Merge branch 'da/difftool-dir-diff-symlink-fix' into da/difftool\n - Merge branch 'ab/retire-option-argument' into da/difftool\n (this branch uses da/difftool-dir-diff-symlink-fix.)\n\n Code clean-up in \"git difftool\".\n\n Will merge to 'next'?\n\n\n* da/difftool-dir-diff-symlink-fix (2021-09-23) 1 commit\n - difftool: fix symlink-file writing in dir-diff mode\n (this branch is used by da/difftool.)\n\n \"git difftool --dir-diff\" mishandled symbolic links.\n\n Will merge to 'next'.\n\n\n* en/removing-untracked-fixes (2021-09-27) 12 commits\n - Documentation: call out commands that nuke untracked files/directories\n - Comment important codepaths regarding nuking untracked files/dirs\n - unpack-trees: avoid nuking untracked dir in way of locally deleted file\n - unpack-trees: avoid nuking untracked dir in way of unmerged file\n - Change unpack_trees' 'reset' flag into an enum\n - Remove ignored files by default when they are in the way\n - unpack-trees: make dir an internal-only struct\n - unpack-trees: introduce preserve_ignored to unpack_trees_options\n - read-tree, merge-recursive: overwrite ignored files by default\n - checkout, read-tree: fix leak of unpack_trees_options.dir\n - t2500: add various tests for nuking untracked files\n - Merge branch 'en/am-abort-fix' into en/removing-untracked-fixes\n\n Various fixes in code paths that move untracked files away to make room.\n\n Will merge to 'next'?\n\n\n* en/typofixes (2021-09-20) 2 commits\n  (merged to 'next' on 2021-09-20 at 48648dafa3)\n + merge-ort: fix completely wrong comment\n + trace2.h: fix trivial comment typo\n\n Typofixes.\n\n Will merge to 'master'.\n\n\n* ks/submodule-add-message-fix (2021-09-20) 1 commit\n - submodule--helper: fix incorrect newlines in an error message\n\n Message regression fix.\n\n Will merge to 'next'.\n\n\n* tb/commit-graph-usage-fix (2021-09-22) 2 commits\n - builtin/multi-pack-index.c: disable top-level --[no-]progress\n - builtin/commit-graph.c: don't accept common --[no-]progress\n\n Regression fix for \"git commit-graph\" command line parsing.\n\n Will merge to 'next'.\n\n\n* ws/refer-to-forkpoint-config-in-rebase-doc (2021-09-20) 1 commit\n - Document `rebase.forkpoint` in rebase man page\n\n Doc update.\n\n Will merge to 'next'?\n\n\n* jk/clone-unborn-head-in-bare (2021-09-20) 1 commit\n  (merged to 'next' on 2021-09-20 at 93c93b8d51)\n + clone: handle unborn branch in bare repos\n\n \"git clone\" from a repository whose HEAD is unborn into a bare\n repository didn't follow the branch name the other side used, which\n is corrected.\n\n Will merge to 'master'.\n\n\n* jk/reduce-malloc-in-v2-servers (2021-09-15) 12 commits\n  (merged to 'next' on 2021-09-16 at 40cfe41efc)\n + ls-refs: reject unknown arguments\n + serve: reject commands used as capabilities\n + serve: reject bogus v2 \"command=ls-refs=foo\"\n + docs/protocol-v2: clarify some ls-refs ref-prefix details\n + ls-refs: ignore very long ref-prefix counts\n + serve: drop \"keys\" strvec\n + serve: provide \"receive\" function for session-id capability\n + serve: provide \"receive\" function for object-format capability\n + serve: add \"receive\" method for v2 capabilities table\n + serve: return capability \"value\" from get_capability()\n + serve: rename is_command() to parse_command()\n + Merge branch 'ab/serve-cleanup' into jk/reduce-malloc-in-v2-servers\n\n Code cleanup to limit memory consumption and tighten protocol\n message parsing.\n\n Will merge to 'master'.\n\n\n* ns/batched-fsync (2021-09-27) 8 commits\n - core.fsyncobjectfiles: performance tests for add and stash\n - core.fsyncobjectfiles: tests for batch mode\n - unpack-objects: use the bulk-checkin infrastructure\n - update-index: use the bulk-checkin infrastructure\n - core.fsyncobjectfiles: add windows support for batch mode\n - core.fsyncobjectfiles: batched disk flushes\n - bulk-checkin: rename 'state' variable and separate 'plugged' boolean\n - object-file.c: do not rename in a temp odb\n\n The \"core.fsyncobjectfiles\" configuration variable can now be set\n to \"batch\" for improved performance.\n\n Will merge to 'next'?\n\n\n* jh/builtin-fsmonitor-part1 (2021-09-20) 7 commits\n - t/helper/simple-ipc: convert test-simple-ipc to use start_bg_command\n - run-command: create start_bg_command\n - simple-ipc/ipc-win32: add Windows ACL to named pipe\n - simple-ipc/ipc-win32: add trace2 debugging\n - simple-ipc: move definition of ipc_active_state outside of ifdef\n - simple-ipc: preparations for supporting binary messages.\n - trace2: add trace2_child_ready() to report on background children\n\n Built-in fsmonitor (part 1).\n\n\n* cb/cvsserver (2021-09-16) 3 commits\n  (merged to 'next' on 2021-09-23 at 98f5f3f9cf)\n + Documentation: cleanup git-cvsserver\n + git-cvsserver: protect against NULL in crypt(3)\n + git-cvsserver: use crypt correctly to compare password hashes\n\n \"git cvsserver\" had a long-standing bug in its authentication code,\n which has finally been corrected (it is unclear and is a separate\n question if anybody is seriously using it, though).\n\n Will merge to 'master'.\n\n\n* cb/unicode-14 (2021-09-17) 1 commit\n  (merged to 'next' on 2021-09-20 at 7873b310ad)\n + unicode: update the width tables to Unicode 14\n\n The unicode character width table (used for output alignment) has\n been updated.\n\n Will merge to 'master'.\n\n\n* ab/align-parse-options-help (2021-09-22) 4 commits\n - parse-options: properly align continued usage output\n - git rev-parse --parseopt tests: add more usagestr tests\n - send-pack: properly use parse_options() API for usage string\n - parse-options API users: align usage output in C-strings\n\n When \"git cmd -h\" shows more than one line of usage text (e.g.\n the cmd subcommand may take sub-sub-command), parse-options API\n learned to align these lines, even across i18n/l10n.\n\n Will merge to 'next'?\n\n\n* ab/help-config-vars (2021-09-23) 9 commits\n - help: move column config discovery to help.c library\n - help / completion: make \"git help\" do the hard work\n - help tests: test --config-for-completion option & output\n - help: simplify by moving to OPT_CMDMODE()\n - help: correct logic error in combining --all and --guides\n - help: correct logic error in combining --all and --config\n - help tests: add test for --config output\n - help: correct usage & behavior of \"git help --guides\"\n - help: correct the usage string in -h and documentation\n\n Teach \"git help -c\" into helping the command line completion of\n configuration variables.\n\n Will merge to 'next'?\n\n\n* tb/repack-write-midx (2021-09-16) 8 commits\n - builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n - builtin/repack.c: make largest pack preferred\n - builtin/repack.c: support writing a MIDX while repacking\n - builtin/repack.c: extract showing progress to a variable\n - builtin/repack.c: keep track of existing packs unconditionally\n - midx: preliminary support for `--refs-snapshot`\n - builtin/multi-pack-index.c: support `--stdin-packs` mode\n - midx: expose `write_midx_file_only()` publicly\n\n \"git repack\" has been taught to generate multi-pack reachability\n bitmaps.\n\n Under review.\n cf. <YU0TS+KsWY36oeKU@nand.local>\n\n\n* ds/add-rm-with-sparse-index (2021-09-24) 13 commits\n - advice: update message to suggest '--sparse'\n - mv: refuse to move sparse paths\n - rm: skip sparse paths with missing SKIP_WORKTREE\n - rm: add --sparse option\n - add: update --renormalize to skip sparse paths\n - add: update --chmod to skip sparse paths\n - add: implement the --sparse option\n - add: skip tracked paths outside sparse-checkout cone\n - add: fail when adding an untracked sparse file\n - dir: fix pattern matching on dirs\n - dir: select directories correctly\n - t1092: behavior for adding sparse files\n - t3705: test that 'sparse_entry' is unstaged\n\n \"git add\", \"git mv\", and \"git rm\" have been adjusted to avoid\n updating paths outside of the sparse-checkout definition unless\n the user specifies a \"--sparse\" option.\n\n Will merge to 'next'?\n\n\n* tb/midx-write-propagate-namehash (2021-09-17) 7 commits\n - t5326: test propagating hashcache values\n - p5326: generate pack bitmaps before writing the MIDX bitmap\n - p5326: don't set core.multiPackIndex unnecessarily\n - p5326: create missing 'perf-tag' tag\n - midx.c: respect 'pack.writeBitmapHashcache' when writing bitmaps\n - pack-bitmap.c: propagate namehash values from existing bitmaps\n - t/helper/test-bitmap.c: add 'dump-hashes' mode\n\n \"git multi-pack-index write --bitmap\" learns to propagate the\n hashcache from original bitmap to resulting bitmap.\n\n Will merge to 'next'.\n\n\n* pw/rebase-of-a-tag-fix (2021-09-22) 10 commits\n - rebase: dereference tags\n - rebase: use lookup_commit_reference_by_name()\n - rebase: use our standard error return value\n - t3407: rework rebase --quit tests\n - t3407: strengthen rebase --abort tests\n - t3407: use test_path_is_missing\n - t3407: rename a variable\n - t3407: use test_cmp_rev\n - t3407: use test_commit\n - t3407: run tests in $TEST_DIRECTORY\n\n \"git rebase <upstream> <tag>\" failed when aborted in the middle, as\n it mistakenly tried to write the tag object instead of peeling it\n to HEAD.\n\n Will merge to 'next'.\n\n\n* en/zdiff3 (2021-09-20) 2 commits\n - update documentation for new zdiff3 conflictStyle\n - xdiff: implement a zealous diff3, or \"zdiff3\"\n\n \"Zealous diff3\" style of merge conflict presentation has been added.\n\n\n* en/stash-df-fix (2021-09-10) 3 commits\n  (merged to 'next' on 2021-09-20 at 513c11fb11)\n + stash: restore untracked files AFTER restoring tracked files\n + stash: avoid feeding directories to update-index\n + t3903: document a pair of directory/file bugs\n\n Fix \"git stash\" corner case where the tentative change involves\n changing a directory to a file (or vice versa).\n\n Will merge to 'master'.\n\n\n* jt/add-submodule-odb-clean-up (2021-09-09) 3 commits\n - revision: remove \"submodule\" from opt struct\n - repository: support unabsorbed in repo_submodule_init\n - submodule: remove unnecessary unabsorbed fallback\n (this branch is used by jt/no-abuse-alternate-odb-for-submodules.)\n\n More code paths that uses the hack to add submodule's object\n database to the set of alternate object store have been cleaned up.\n\n Will merge to 'next'.\n\n\n* jx/ci-l10n (2021-09-09) 1 commit\n  (merged to 'next' on 2021-09-23 at b2d7f5eecb)\n + ci: new github-action for git-l10n code review\n\n CI help for l10n.\n\n Will merge to 'master'.\n\n\n* js/scalar (2021-09-14) 15 commits\n - scalar: accept -C and -c options before the subcommand\n - scalar: implement the `version` command\n - scalar: implement the `delete` command\n - scalar: teach 'reconfigure' to optionally handle all registered enlistments\n - scalar: allow reconfiguring an existing enlistment\n - scalar: implement the `run` command\n - scalar: teach 'clone' to support the --single-branch option\n - scalar: implement the `clone` subcommand\n - scalar: implement 'scalar list'\n - scalar: let 'unregister' handle a deleted enlistment directory gracefully\n - scalar: 'unregister' stops background maintenance\n - scalar: 'register' sets recommended config and starts maintenance\n - scalar: create test infrastructure\n - scalar: start documenting the command\n - scalar: create a rudimentary executable\n\n Add pieces from \"scalar\" to contrib/.\n\n Will merge to 'next'?\n\n\n* ab/sanitize-leak-ci (2021-09-23) 2 commits\n - tests: add a test mode for SANITIZE=leak, run it in CI\n - Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n\n CI learns to run the leak sanitizer builds.\n\n Will merge to 'next'?\n\n\n* ms/customizable-ident-expansion (2021-09-01) 1 commit\n - keyword expansion: make \"$Id$\" string configurable\n\n Instead of \"$Id$\", user-specified string (like $FreeBSD$) can be\n used as an in-blob placeholder for keyword expansion.\n\n\n* js/retire-preserve-merges (2021-09-07) 11 commits\n  (merged to 'next' on 2021-09-10 at f645ffd7a3)\n + sequencer: restrict scope of a formerly public function\n + rebase: remove a no-longer-used function\n + rebase: stop mentioning the -p option in comments\n + rebase: remove obsolete code comment\n + rebase: drop the internal `rebase--interactive` command\n + git-svn: drop support for `--preserve-merges`\n + rebase: drop support for `--preserve-merges`\n + pull: remove support for `--rebase=preserve`\n + tests: stop testing `git rebase --preserve-merges`\n + remote: warn about unhandled branch.<name>.rebase values\n + t5520: do not use `pull.rebase=preserve`\n\n The \"--preserve-merges\" option of \"git rebase\" has been removed.\n\n Will cook in 'next'.\n\n\n* en/remerge-diff (2021-08-31) 7 commits\n - doc/diff-options: explain the new --remerge-diff option\n - show, log: provide a --remerge-diff capability\n - tmp-objdir: new API for creating and removing primary object dirs\n - merge-ort: capture and print ll-merge warnings in our preferred fashion\n - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n - merge-ort: add ability to record conflict messages in a file\n - merge-ort: mark a few more conflict messages as omittable\n\n A new presentation for two-parent merge \"--remerge-diff\" can be\n used to show the difference between mechanical (and possibly\n conflicted) merge results and the recorded resolution.\n\n Will merge to 'next'?\n\n\n* sg/test-split-index-fix (2021-09-07) 7 commits\n - read-cache: fix GIT_TEST_SPLIT_INDEX\n - tests: disable GIT_TEST_SPLIT_INDEX for sparse index tests\n - read-cache: look for shared index files next to the index, too\n - t1600-index: disable GIT_TEST_SPLIT_INDEX\n - t1600-index: don't run git commands upstream of a pipe\n - t1600-index: remove unnecessary redirection\n - Merge branch 'ds/sparse-index-ignored-files' into sg/test-split-index-fix\n\n Test updates.\n\n Will merge to 'next'?\n\n\n* ab/refs-errno-cleanup (2021-08-25) 4 commits\n - refs: make errno output explicit for refs_resolve_ref_unsafe\n - refs: explicitly return failure_errno from parse_loose_ref_contents\n - branch tests: test for errno propagating on failing read\n - refs: add failure_errno to refs_read_raw_ref() signature\n (this branch uses ab/refs-files-cleanup and hn/refs-errno-cleanup.)\n\n The \"remainder\" of hn/refs-errno-cleanup topic.\n\n\n* ab/lib-subtest (2021-09-22) 9 commits\n - test-lib tests: get rid of copy/pasted mock test code\n - test-lib tests: assert 1 exit code, not non-zero\n - test-lib tests: refactor common part of check_sub_test_lib_test*()\n - test-lib tests: avoid subshell for \"test_cmp\" for readability\n - test-lib tests: don't provide a description for the sub-tests\n - test-lib tests: split up \"write and run\" into two functions\n - test-lib tests: move \"run_sub_test\" to a new lib-subtest.sh\n - Merge branch 'ps/t0000-output-directory-fix' into ab/lib-subtest\n - Merge branch 'jk/t0000-subtests-fix' into ab/lib-subtest\n\n Updates to the tests in t0000 to test the test framework.\n\n Will merge to 'next'?\n\n\n* ab/only-single-progress-at-once (2021-09-22) 8 commits\n - progress.c: add & assert a \"global_progress\" variable\n - pack-bitmap-write.c: add a missing stop_progress()\n - progress.c: add temporary variable from progress struct\n - progress.c: stop eagerly fflush(stderr) when not a terminal\n - progress.c: call progress_interval() from progress_test_force_update()\n - progress.c: move signal handler functions lower\n - progress.c tests: test some invalid usage\n - progress.c tests: make start/stop verbs on stdin\n\n Further tweaks on progress API.\n\n\n* fs/ssh-signing (2021-09-10) 9 commits\n - ssh signing: test that gpg fails for unknown keys\n - ssh signing: tests for logs, tags & push certs\n - ssh signing: duplicate t7510 tests for commits\n - ssh signing: verify signatures using ssh-keygen\n - ssh signing: provide a textual signing_key_id\n - ssh signing: retrieve a default key from ssh-agent\n - ssh signing: add ssh key format and signing code\n - ssh signing: add test prereqs\n - ssh signing: preliminary refactoring and clean-up\n\n Use ssh public crypto for object and push-cert signing.\n\n On hold.\n cf. <pull.1041.v8.git.git.1631304462.gitgitgadget@gmail.com>\n cf. <532d97e7-8c91-df6a-6d90-70668256f513@gigacodes.de>\n\n\n* cf/fetch-set-upstream-while-detached (2021-07-06) 1 commit\n - fetch: fix segfault on --set-upstream while on a detached HEAD\n\n \"git fetch --set-upstream\" while on detached HEAD segfaulted\n instead of noticing that such an operation did not make sense.\n\n Expecting a reroll.\n cf. <xmqqsg0ri5mq.fsf@gitster.g>\n\n\n* pw/diff-color-moved-fix (2021-08-05) 13 commits\n - diff: drop unused options parameter from cmp_in_block_with_wsd()\n - diff --color-moved: intern strings\n - diff: use designated initializers for emitted_diff_symbol\n - diff --color-moved-ws=allow-indentation-change: improve hash lookups\n - diff --color-moved: stop clearing potential moved blocks\n - diff --color-moved: shrink potential moved blocks as we go\n - diff --color-moved: unify moved block growth functions\n - diff --color-moved: call comparison function directly\n - diff --color-moved-ws=allow-indentation-change: simplify and optimize\n - diff: simplify allow-indentation-change delta calculation\n - diff --color-moved: avoid false short line matches and bad zerba coloring\n - diff --color-moved=zebra: fix alternate coloring\n - diff --color-moved: add perf tests\n\n Originally merged to 'next' on 2021-08-05\n\n Long-overdue correctness and performance update to \"diff\n --color-moved\" feature.\n\n Expecting a reroll.\n cf. <8bec1a6d-5052-50c3-4100-e6348289d581@gmail.com>\n\n\n* hn/reftable (2021-09-10) 20 commits\n - fixup! reftable: implement stack, a mutable database of reftable files.\n - Add \"test-tool dump-reftable\" command.\n - reftable: add dump utility\n - reftable: implement stack, a mutable database of reftable files.\n - reftable: implement refname validation\n - reftable: add merged table view\n - reftable: add a heap-based priority queue for reftable records\n - reftable: reftable file level tests\n - reftable: read reftable files\n - reftable: generic interface to tables\n - reftable: write reftable files\n - reftable: a generic binary tree implementation\n - reftable: reading/writing blocks\n - Provide zlib's uncompress2 from compat/zlib-compat.c\n - reftable: (de)serialization for the polymorphic record type.\n - reftable: add blocksource, an abstraction for random access reads\n - reftable: utility functions\n - reftable: add error related functionality\n - reftable: RFC: add LICENSE\n - hash.h: provide constants for the hash IDs\n\n The \"reftable\" backend for the refs API, without integrating into\n the refs subsystem.\n\n Will merge to 'next'?\n\n\n* ab/refs-files-cleanup (2021-08-25) 13 commits\n  (merged to 'next' on 2021-09-23 at eb5668523f)\n + refs/files: remove unused \"errno != ENOTDIR\" condition\n + refs/files: remove unused \"errno == EISDIR\" code\n + refs/files: remove unused \"oid\" in lock_ref_oid_basic()\n + refs API: remove OID argument to reflog_expire()\n + reflog expire: don't lock reflogs using previously seen OID\n + refs/files: add a comment about refs_reflog_exists() call\n + refs: make repo_dwim_log() accept a NULL oid\n + refs/debug: re-indent argument list for \"prepare\"\n + refs/files: remove unused \"skip\" in lock_raw_ref() too\n + refs/files: remove unused \"extras/skip\" in lock_ref_oid_basic()\n + refs: drop unused \"flags\" parameter to lock_ref_oid_basic()\n + refs/files: remove unused REF_DELETING in lock_ref_oid_basic()\n + refs/packet: add missing BUG() invocations to reflog callbacks\n (this branch is used by ab/refs-errno-cleanup and hn/refs-errno-cleanup.)\n\n Continued work on top of the hn/refs-errno-cleanup topic.\n\n Will merge to 'master'.\n\n\n* hn/refs-errno-cleanup (2021-08-25) 4 commits\n  (merged to 'next' on 2021-09-23 at 502e6b6b08)\n + refs: make errno output explicit for read_raw_ref_fn\n + refs/files-backend: stop setting errno from lock_ref_oid_basic\n + refs: remove EINVAL errno output from specification of read_raw_ref_fn\n + refs file backend: move raceproof_create_file() here\n (this branch is used by ab/refs-errno-cleanup; uses ab/refs-files-cleanup.)\n\n Futz with the way 'errno' is relied on in the refs API to carry the\n failure modes up the call chain.\n\n Will merge to 'master'.\n\n--------------------------------------------------\n[Discarded]\n\n* ao/p4-avoid-decoding (2021-04-12) 2 commits\n . git-p4: do not decode data from perforce by default\n . git-p4: avoid decoding more data from perforce\n\n \"git p4\" in Python-2 days used to accept a lot more kinds of data\n from Perforce server as uninterrupted byte sequence, but after\n switching to Python-3, too many things are expected to be in UTF-8,\n which broke traditional use cases.\n\n Have been stalled for too long.\n cf. <20210504220153.1d9f0cb2@ado-tr>\n\n\n* tv/p4-fallback-encoding (2021-04-30) 1 commit\n . git-p4: git-p4.fallbackEncoding to specify non UTF-8 charset\n\n \"git p4\" learns the fallbackEncoding configuration variable to\n safely accept changeset descriptions that aren't written in UTF-8.\n\n Have been stalled for too long.\n cf. <CAKu1iLUaLuAZWqjNK4tfhhR=YaSt4MdQ+90ZY-JcEh_SeHyYCw@mail.gmail.com>\n\n\n* jh/builtin-fsmonitor (2021-09-03) 37 commits\n . fixup! fsmonitor--daemon: implement handle_client callback\n . SQUASH??? https://github.com/git/git/runs/3438543601?check_suite_focus=true#step:5:136\n . BANDAID: sparse fixes\n . t7527: test FS event reporing on MacOS WRT case and Unicode\n . fsmonitor: handle shortname for .git\n . t7527: test status with untracked-cache and fsmonitor--daemon\n . fsmonitor: force update index after large responses\n . fsmonitor: enhance existing comments\n . fsmonitor--daemon: use a cookie file to sync with file system\n . fsmonitor--daemon: periodically truncate list of modified files\n . t7527: create test for fsmonitor--daemon\n . t/perf/p7519: add fsmonitor--daemon test cases\n . t/perf: avoid copying builtin fsmonitor files into test repo\n . t/perf/p7519: speed up test using \"test-tool touch\"\n . t/helper/test-touch: add helper to touch a series of files\n . fsmonitor--daemon: implement handle_client callback\n . fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS\n . fsmonitor-fs-listen-macos: add macos header files for FSEvent\n . fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows\n . fsmonitor--daemon: create token-based changed path cache\n . fsmonitor--daemon: define token-ids\n . fsmonitor--daemon: add pathname classification\n . fsmonitor: do not try to operate on bare repos\n . fsmonitor--daemon: implement 'start' command\n . fsmonitor--daemon: implement 'run' command\n . fsmonitor-fs-listen-macos: stub in backend for MacOS\n . fsmonitor-fs-listen-win32: stub in backend for Windows\n . t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n . fsmonitor--daemon: implement 'stop' and 'status' commands\n . fsmonitor--daemon: add a built-in fsmonitor daemon\n . fsmonitor: use IPC to query the builtin FSMonitor daemon\n . fsmonitor: config settings are repository-specific\n . help: include fsmonitor--daemon feature flag in version info\n . fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n . fsmonitor--daemon: update fsmonitor documentation\n . fsmonitor--daemon: man page\n . simple-ipc: preparations for supporting binary messages.\n\n An attempt to write and ship with a watchman equivalent tailored\n for our use.\n\n Will be rerolled in pieces.\n\n\n* ab/config-based-hooks-base (2021-09-09) 36 commits\n . hooks: fix a TOCTOU in \"did we run a hook?\" heuristic\n . receive-pack: convert receive hooks to hook.h\n . post-update: use hook.h library\n . receive-pack: convert 'update' hook to hook.h\n . hooks: allow callers to capture output\n . run-command: allow capturing of collated output\n . reference-transaction: use hook.h to run hooks\n . hook tests: use a modern style for \"pre-push\" tests\n . hook tests: test for exact \"pre-push\" hook input\n . transport: convert pre-push hook to hook.h\n . hook: convert 'post-rewrite' hook in sequencer.c to hook.h\n . hook: provide stdin by string_list or callback\n . run-command: add stdin callback for parallelization\n . am: convert 'post-rewrite' hook to hook.h\n . hook: support passing stdin to hooks\n . run-command: allow stdin for run_processes_parallel\n . run-command: remove old run_hook_{le,ve}() hook API\n . receive-pack: convert push-to-checkout hook to hook.h\n . read-cache: convert post-index-change to use hook.h\n . commit: convert {pre-commit,prepare-commit-msg} hook to hook.h\n . git-p4: use 'git hook' to run hooks\n . send-email: use 'git hook run' for 'sendemail-validate'\n . git hook run: add an --ignore-missing flag\n . merge: convert post-merge to use hook.h\n . hooks: convert 'post-checkout' hook to hook library\n . am: convert applypatch to use hook.h\n . rebase: convert pre-rebase to use hook.h\n . gc: use hook library for pre-auto-gc hook\n . hook: add 'run' subcommand\n . hook-list.h: add a generated list of hooks, like config-list.h\n . hook.c users: use \"hook_exists()\" instead of \"find_hook()\"\n . hook.c: add a hook_exists() wrapper and use it in bugreport.c\n . hook.[ch]: move find_hook() from run-command.c to hook.c\n . Makefile: remove an out-of-date comment\n . Makefile: stop hardcoding {command,config}-list.h\n . Makefile: mark \"check\" target as .PHONY\n (this branch is used by es/config-based-hooks.)\n\n Restructuring of (a subset of) Emily's config-based-hooks series,\n to demonstrate that a series can be presented as a more logical and\n focused progression.\n\n Will be rerolled in pieces.\n\n\n* es/config-based-hooks (2021-09-09) 6 commits\n . hook: allow out-of-repo 'git hook' invocations\n . hook: include hooks from the config\n . hook: introduce \"git hook list\"\n . hook: allow parallel hook execution\n . fixup! hook: run a list of hooks instead\n . hook: run a list of hooks instead\n (this branch uses ab/config-based-hooks-base.)\n\n Revamp the hooks subsystem to allow multiple of them to trigger\n upon the same event and control via the configuration variables.\n\n\n* cb/make-compdb-fix (2021-09-22) 1 commit\n . Makefile: avoid breaking compilation database generation with DEVELOPER\n\n Adjust to recent change to use -pedantic for developer builds.\n\n Replaced by the ab/make-compdb-fix topic that uses the same\n approach as the ab/auto-depend-with-pedantic topic.\n\n"},{"id":"437242","messageId":"87czotzaru.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T01:57:01Z","receivedAt":"2021-09-28T02:36:45Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Junio C Hamano wrote:\n\nUpdates on my topics, CC'd the people I mentioned.\n\n> * ab/config-based-hooks-1 (2021-09-27) 8 commits\n>  - hook-list.h: add a generated list of hooks, like config-list.h\n>  - hook.c users: use \"hook_exists()\" instead of \"find_hook()\"\n>  - hook.c: add a hook_exists() wrapper and use it in bugreport.c\n>  - hook.[ch]: move find_hook() from run-command.c to hook.c\n>  - Makefile: remove an out-of-date comment\n>  - Makefile: don't perform \"mv $@+ $@\" dance for $(GENERATED_H)\n>  - Makefile: stop hardcoding {command,config}-list.h\n>  - Makefile: mark \"check\" target as .PHONY\n\nThanks for picking this up. Looks like my split-up plan is working in\nthat this has received a lot of careful review.\n\nI think per René's the v2 should be good-to-go, barring any new\nundiscovered issues:\nhttps://lore.kernel.org/git/0e07cee4-079a-af12-880f-d4a99300df28@web.de/\n\n> * ab/http-pinned-public-key-mismatch (2021-09-27) 1 commit\n>  - http: check CURLE_SSL_PINNEDPUBKEYNOTMATCH when emitting errors\n>\n>  HTTPS error handling updates.\n>\n>  Will merge to 'next'?\n\nI'd like it merged. I think a fair summary of Jeff King's\nhttps://lore.kernel.org/git/YU5CJC9XJvQITfr8@coredump.intra.peff.net/ is\n\"meh\" though :)\n\nI think the translation improvement + mentioning the specific git config\nvariable in play is a pretty minor but still worthwhile change.\n\n> * ab/make-compdb-fix (2021-09-27) 1 commit\n>  - Makefile: pass -Wno-pendantic under GENERATE_COMPILATION_DATABASE=yes\n>\n>  Build update.\n>\n>  Will merge to 'next'.\n\nThanks, FWIW I thought Carlos's cb/make-compdb-fix was better, but see\nfrom this E-Mail & other context that you'd looked at both and preferred\nthe consistency with my COMPUTE_HEADER_DEPENDENCIES fix.\n\nIn any case both fix the immediate \"pedantic\" fallout.\n\n> * ab/pack-objects-stdin (2021-07-09) 5 commits\n>  - pack-objects.c: make use of REV_INFO_STDIN_LINE_PROCESS\n>  - pack-objects.c: do stdin parsing via revision.c's API\n>  - revision.[ch]: add a \"handle_stdin_line\" API\n>  - revision.h: refactor \"disable_stdin\" and \"read_from_stdin\"\n>  - upload-pack: run is_repository_shallow() before setup_revisions()\n>\n>  Introduce handle_stdin_line callback to revision API and uses it.\n>\n>  Expecting a reroll.\n\nI said I'd re-roll this into the larger \"git bundle --stdin\" topic I had\nin mind, but when doing so found one memory leak, which I really wanted\nthe SANITIZE=leak interface to fix.\n\nI think it's best to eject/discard this given the amount of other\noutstanding stuff I've got. I don't really *need* this for anything\nelse, I'll loop back to it sometime after the SANITIZE=leak testing has\nlanded.\n\n> * ab/fsck-unexpected-type (2021-09-22) 17 commits\n> [...]\n>  \"git fsck\" has been taught to report mismatch between expected and\n>  actual types of an object better.\n>\n>  Needs review.\n\nThe \"needs review\" note here pre-dates v6, which got a thorough review,\nand Taylor seemed happy with the v7, aside from a minor issue I just\nre-rolled v8 with a fix for:\n\nhttps://lore.kernel.org/git/cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com/\n\n> * ab/make-clean-depend-dirs (2021-09-22) 1 commit\n> [...]\n>  Will merge to 'next'.\n\nThanks!\n\n> * ab/auto-depend-with-pedantic (2021-09-22) 1 commit\n>  - Makefile: make COMPUTE_HEADER_DEPENDENCIES=auto work with DEVOPTS=pedantic\n>\n>  Improve build procedure for developers.\n>\n>  Will merge to 'next'.\n\nThanks!\n\n> * ab/make-sparse-for-real (2021-09-22) 1 commit\n>  - Makefile: make the \"sparse\" target non-.PHONY\n>\n>  Prevent \"make sparse\" from running for the source files that\n>  haven't been modified.\n>\n>  Will merge to 'next'?\n\nI just submitted a hopefully final v4, please ignore the v3 :)\nhttps://lore.kernel.org/git/patch-v4-1.1-f31fa3e8282-20210928T014023Z-avarab@gmail.com/T/#u\n\nThat leaves \"sparse\" as the bug-for-bug compatible old target, with a\nnew \"sparse-incr\" for those (like me) who want the incremental\nversion. The feedback on v2 was mainly about the change in behavior\nbreaking existing workflows, so v4 should get around that nicely.\n\n> * ab/repo-settings-cleanup (2021-09-22) 5 commits\n> [...]\n>  Will merge to 'next'.\n\nThanks!\n\n> * ab/align-parse-options-help (2021-09-22) 4 commits\n>  - parse-options: properly align continued usage output\n>  - git rev-parse --parseopt tests: add more usagestr tests\n>  - send-pack: properly use parse_options() API for usage string\n>  - parse-options API users: align usage output in C-strings\n>\n>  When \"git cmd -h\" shows more than one line of usage text (e.g.\n>  the cmd subcommand may take sub-sub-command), parse-options API\n>  learned to align these lines, even across i18n/l10n.\n>\n>  Will merge to 'next'?\n\nI think it's ready for that.\n\n> * ab/help-config-vars (2021-09-23) 9 commits\n>  - help: move column config discovery to help.c library\n>  - help / completion: make \"git help\" do the hard work\n>  - help tests: test --config-for-completion option & output\n>  - help: simplify by moving to OPT_CMDMODE()\n>  - help: correct logic error in combining --all and --guides\n>  - help: correct logic error in combining --all and --config\n>  - help tests: add test for --config output\n>  - help: correct usage & behavior of \"git help --guides\"\n>  - help: correct the usage string in -h and documentation\n>\n>  Teach \"git help -c\" into helping the command line completion of\n>  configuration variables.\n>\n>  Will merge to 'next'?\n\nI think so too, you had some feedback on the v3, I think partially\naddressed by the \"simplify by moving to OPT_CMDMODE()\" step.\n\nFor further improving the usage info in the rare case of combining\ne.g. --all and --config I'd prefer to just leave it for a future\nimprovement.\n\n> * ab/sanitize-leak-ci (2021-09-23) 2 commits\n>  - tests: add a test mode for SANITIZE=leak, run it in CI\n>  - Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS\n>\n>  CI learns to run the leak sanitizer builds.\n>\n>  Will merge to 'next'?\n\nYes please! As noted in some other threads I've got more memory leak\nfixes waiting on this, and since the v8 to avoid testing the split index\nit doesn't seem to be causing any trouble:\nhttps://lore.kernel.org/git/cover-v8-0.2-00000000000-20210923T091819Z-avarab@gmail.com/\n\n> * ab/refs-errno-cleanup (2021-08-25) 4 commits\n>  - refs: make errno output explicit for refs_resolve_ref_unsafe\n>  - refs: explicitly return failure_errno from parse_loose_ref_contents\n>  - branch tests: test for errno propagating on failing read\n>  - refs: add failure_errno to refs_read_raw_ref() signature\n>  (this branch uses ab/refs-files-cleanup and hn/refs-errno-cleanup.)\n>\n>  The \"remainder\" of hn/refs-errno-cleanup topic.\n\nI think this should be ready too, but if you'd prefer to wait a bit\nafter ab/refs-files-cleanup and ab/refs-files-cleanup land I understand,\nit's a lot of ref code changes...\n\n> * ab/lib-subtest (2021-09-22) 9 commits\n>  - test-lib tests: get rid of copy/pasted mock test code\n>  - test-lib tests: assert 1 exit code, not non-zero\n>  - test-lib tests: refactor common part of check_sub_test_lib_test*()\n>  - test-lib tests: avoid subshell for \"test_cmp\" for readability\n>  - test-lib tests: don't provide a description for the sub-tests\n>  - test-lib tests: split up \"write and run\" into two functions\n>  - test-lib tests: move \"run_sub_test\" to a new lib-subtest.sh\n>  - Merge branch 'ps/t0000-output-directory-fix' into ab/lib-subtest\n>  - Merge branch 'jk/t0000-subtests-fix' into ab/lib-subtest\n>\n>  Updates to the tests in t0000 to test the test framework.\n>\n>  Will merge to 'next'?\n\nYes please, it was waiting on review for a long time. Carlo had some\nfeedback on it that I re-rolled v4 addressing, resolving what I think\nwas the main sticking point all along. I.e. conflating a test assertion\nmode with this t0000 cleanup/split-up:\n\nhttps://lore.kernel.org/git/cover-v4-0.7-00000000000-20210922T111734Z-avarab@gmail.com/\n\n> * ab/only-single-progress-at-once (2021-09-22) 8 commits\n>  - progress.c: add & assert a \"global_progress\" variable\n>  - pack-bitmap-write.c: add a missing stop_progress()\n>  - progress.c: add temporary variable from progress struct\n>  - progress.c: stop eagerly fflush(stderr) when not a terminal\n>  - progress.c: call progress_interval() from progress_test_force_update()\n>  - progress.c: move signal handler functions lower\n>  - progress.c tests: test some invalid usage\n>  - progress.c tests: make start/stop verbs on stdin\n>\n>  Further tweaks on progress API.\n\nI think per\nhttps://lore.kernel.org/git/cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com/\nthis should be ready to progress.\n\nI.e. it was held off due to comment from SZEDER that once clarified I\nunderstand to be a general concern that we should test the 8/8 step. As\nnoted in the re-rolled 8/8 I've tested this thoroughly in a way that\nwould break if that BUG() was going to bite us.\n\n> * ab/refs-files-cleanup (2021-08-25) 13 commits\n> [...]\n>\n>  Continued work on top of the hn/refs-errno-cleanup topic.\n>\n>  Will merge to 'master'.\n\n[...]\n\n> * hn/refs-errno-cleanup (2021-08-25) 4 commits\n> [...]\n>\n>  Will merge to 'master'.\n\nYay, thanks!\n"},{"id":"437255","messageId":"CABPp-BGuzd_TH57-1RvwJQD5r3S3ZkJcuiPnU8aWee8pgzUBEw@mail.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-28T06:46:40Z","receivedAt":"2021-09-28T06:46:54Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"[Did some slight re-ordering of topics]\n\nOn Mon, Sep 27, 2021 at 5:53 PM Junio C Hamano <gitster@pobox.com> wrote:\n\n> * ds/add-rm-with-sparse-index (2021-09-24) 13 commits\n>  - advice: update message to suggest '--sparse'\n>  - mv: refuse to move sparse paths\n>  - rm: skip sparse paths with missing SKIP_WORKTREE\n>  - rm: add --sparse option\n>  - add: update --renormalize to skip sparse paths\n>  - add: update --chmod to skip sparse paths\n>  - add: implement the --sparse option\n>  - add: skip tracked paths outside sparse-checkout cone\n>  - add: fail when adding an untracked sparse file\n>  - dir: fix pattern matching on dirs\n>  - dir: select directories correctly\n>  - t1092: behavior for adding sparse files\n>  - t3705: test that 'sparse_entry' is unstaged\n>\n>  \"git add\", \"git mv\", and \"git rm\" have been adjusted to avoid\n>  updating paths outside of the sparse-checkout definition unless\n>  the user specifies a \"--sparse\" option.\n>\n>  Will merge to 'next'?\n\nIt would be nice to see the --diff-filter=u change, which you also\nseemed to like[1]; but after that, yeah this is ready to merge down.\n\n[1] https://lore.kernel.org/git/xmqq35pppwsm.fsf@gitster.g/\n\n> * js/scalar (2021-09-14) 15 commits\n>  - scalar: accept -C and -c options before the subcommand\n>  - scalar: implement the `version` command\n>  - scalar: implement the `delete` command\n>  - scalar: teach 'reconfigure' to optionally handle all registered enlistments\n>  - scalar: allow reconfiguring an existing enlistment\n>  - scalar: implement the `run` command\n>  - scalar: teach 'clone' to support the --single-branch option\n>  - scalar: implement the `clone` subcommand\n>  - scalar: implement 'scalar list'\n>  - scalar: let 'unregister' handle a deleted enlistment directory gracefully\n>  - scalar: 'unregister' stops background maintenance\n>  - scalar: 'register' sets recommended config and starts maintenance\n>  - scalar: create test infrastructure\n>  - scalar: start documenting the command\n>  - scalar: create a rudimentary executable\n>\n>  Add pieces from \"scalar\" to contrib/.\n>\n>  Will merge to 'next'?\n\nI feel bad for taking so long to take a look, but I finally responded\nwith a few comments.  Mostly, I'm glad it's a contrib thing; a lot of\nthe config makes sense to me (even if some of it is 'meh' for my\nrepository sizes or setup), but two of the config settings would be\nvery objectionable if they were a core git command.  Since it's\ncontrib, though, it's probably fine to be very opinionated...and\nperhaps even excessively so.  ;-)\n\nHowever, since Johannes has been away for a couple weeks, maybe give\nhim a chance to return and respond to myself and others (and perhaps\npush any updates that occurred to him while on vacation) before\nmerging down?\n\n> * en/remerge-diff (2021-08-31) 7 commits\n>  - doc/diff-options: explain the new --remerge-diff option\n>  - show, log: provide a --remerge-diff capability\n>  - tmp-objdir: new API for creating and removing primary object dirs\n>  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n>  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n>  - merge-ort: add ability to record conflict messages in a file\n>  - merge-ort: mark a few more conflict messages as omittable\n>\n>  A new presentation for two-parent merge \"--remerge-diff\" can be\n>  used to show the difference between mechanical (and possibly\n>  conflicted) merge results and the recorded resolution.\n>\n>  Will merge to 'next'?\n\nIt has been a month that it's been cooking with no issues brought up,\nand it's been in production for nearly a year...\n\nBut just this morning I pinged peff and jrnieder if they might have\ntime to respectively look at the tmp-objdir stuff (patch 5, plus its\nintegration into log-tree.c in patch 7) and the ll-merge.[ch] changes\n(patch 3).  I don't know if either will have time to do it, but\nperhaps wait half a week or so to see if they'll mention they have\ntime?  Otherwise, yeah, it's probably time to merge this down.\n\n> * en/removing-untracked-fixes (2021-09-27) 12 commits\n>  - Documentation: call out commands that nuke untracked files/directories\n>  - Comment important codepaths regarding nuking untracked files/dirs\n>  - unpack-trees: avoid nuking untracked dir in way of locally deleted file\n>  - unpack-trees: avoid nuking untracked dir in way of unmerged file\n>  - Change unpack_trees' 'reset' flag into an enum\n>  - Remove ignored files by default when they are in the way\n>  - unpack-trees: make dir an internal-only struct\n>  - unpack-trees: introduce preserve_ignored to unpack_trees_options\n>  - read-tree, merge-recursive: overwrite ignored files by default\n>  - checkout, read-tree: fix leak of unpack_trees_options.dir\n>  - t2500: add various tests for nuking untracked files\n>  - Merge branch 'en/am-abort-fix' into en/removing-untracked-fixes\n>\n>  Various fixes in code paths that move untracked files away to make room.\n>\n>  Will merge to 'next'?\n\nI just sent out v3 this morning with five new patches (included in\nyour list above).  While I think my patches are good, and I'd like to\nsee them merged down to next so I can send my\ncurrent-working-directory-deletion fixes that build on top of it, I'm\na little surprised you're proposing to merge this series down this\nquickly instead of waiting a little longer for review of the new\npatches.  I'm not complaining...but was that intentional?\n"},{"id":"437261","messageId":"87v92lxhh4.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"CABPp-BGuzd_TH57-1RvwJQD5r3S3ZkJcuiPnU8aWee8pgzUBEw@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T07:45:25Z","receivedAt":"2021-09-28T07:54:52Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Elijah Newren wrote:\n\n> [Did some slight re-ordering of topics]\n>\n> On Mon, Sep 27, 2021 at 5:53 PM Junio C Hamano <gitster@pobox.com> wrote:\n> [...]\n>> * en/remerge-diff (2021-08-31) 7 commits\n>>  - doc/diff-options: explain the new --remerge-diff option\n>>  - show, log: provide a --remerge-diff capability\n>>  - tmp-objdir: new API for creating and removing primary object dirs\n>>  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n>>  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n>>  - merge-ort: add ability to record conflict messages in a file\n>>  - merge-ort: mark a few more conflict messages as omittable\n>>\n>>  A new presentation for two-parent merge \"--remerge-diff\" can be\n>>  used to show the difference between mechanical (and possibly\n>>  conflicted) merge results and the recorded resolution.\n>>\n>>  Will merge to 'next'?\n>\n> It has been a month that it's been cooking with no issues brought up,\n> and it's been in production for nearly a year...\n>\n> But just this morning I pinged peff and jrnieder if they might have\n> time to respectively look at the tmp-objdir stuff (patch 5, plus its\n> integration into log-tree.c in patch 7) and the ll-merge.[ch] changes\n> (patch 3).  I don't know if either will have time to do it, but\n> perhaps wait half a week or so to see if they'll mention they have\n> time?  Otherwise, yeah, it's probably time to merge this down.\n\nI haven't poked at it much, but haven't you and Neeraj Singh (CC'd)\nindependently come up with two slightly different changes in\ntmp-objdir.c to do the same thing? See the tmp-objdir.c part of:\n\nhttp://lore.kernel.org/git/543ea3569342165363c1602ce36683a54dce7a0b.1632527609.git.gitgitgadget@gmail.com\n\nAnd your:\n\nhttp://lore.kernel.org/git/67d3b2b09f9ddda616cdd0d1b12ab7afc73670ed.1630376800.git.gitgitgadget@gmail.com\n\nI.e. yours has the object database managed outside, his has it added to\n\"struct tmp_objdir\", but it's the same objdir dance isn't it?\n\nI started reading the combined code in \"seen\" and found it quite\nconfusing until I saw what was going on.\n\nFor both, if you can agree on some common API: I'd prefer if the\n\"%s/incoming-XXXXXX\" part of the \"tmp_objdir_create()\" was passed as\nsome template, perhaps just the string \"incoming\" as a prefix?\n\nI.e. this was for receive-pack.c originally, now it's for bulk-checkin.c\nand log.c, if either of those segfault or is long-running it's quite\nconfusing to have something called \"incoming\" if you're manually\ninspecting it.  Perhaps \"core-fsyncObjectFiles-batch\" (or even\n\"core.fsyncObjectFiles=batch\") and \"log-remerge-diff\" as prefixes for\nthe two, and \"incoming\" for the one existing caller in \"master\"?\n\n>> * en/removing-untracked-fixes (2021-09-27) 12 commits\n>>  - Documentation: call out commands that nuke untracked files/directories\n>>  - Comment important codepaths regarding nuking untracked files/dirs\n>>  - unpack-trees: avoid nuking untracked dir in way of locally deleted file\n>>  - unpack-trees: avoid nuking untracked dir in way of unmerged file\n>>  - Change unpack_trees' 'reset' flag into an enum\n>>  - Remove ignored files by default when they are in the way\n>>  - unpack-trees: make dir an internal-only struct\n>>  - unpack-trees: introduce preserve_ignored to unpack_trees_options\n>>  - read-tree, merge-recursive: overwrite ignored files by default\n>>  - checkout, read-tree: fix leak of unpack_trees_options.dir\n>>  - t2500: add various tests for nuking untracked files\n>>  - Merge branch 'en/am-abort-fix' into en/removing-untracked-fixes\n>>\n>>  Various fixes in code paths that move untracked files away to make room.\n>>\n>>  Will merge to 'next'?\n>\n> I just sent out v3 this morning with five new patches (included in\n> your list above).  While I think my patches are good, and I'd like to\n> see them merged down to next so I can send my\n> current-working-directory-deletion fixes that build on top of it, I'm\n> a little surprised you're proposing to merge this series down this\n> quickly instead of waiting a little longer for review of the new\n> patches.  I'm not complaining...but was that intentional?\n\n"},{"id":"437264","messageId":"87ilylxg85.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"CABPp-BGuzd_TH57-1RvwJQD5r3S3ZkJcuiPnU8aWee8pgzUBEw@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T08:07:41Z","receivedAt":"2021-09-28T08:21:51Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Elijah Newren wrote:\n\n[Aside: I quite liked the method of replying to What's Cooking per-topic\nin Johannes's replies to\nhttps://lore.kernel.org/git/xmqqsfyf5b74.fsf@gitster.g/; I'm never quite\nsure if I should have one E-Mail with all of my comments on other\nin-flight work, use existing commentary as a springboard etc. Doing the\nlatter here]\n\n>> * js/scalar (2021-09-14) 15 commits\n>>  - scalar: accept -C and -c options before the subcommand\n>>  - scalar: implement the `version` command\n>>  - scalar: implement the `delete` command\n>>  - scalar: teach 'reconfigure' to optionally handle all registered enlistments\n>>  - scalar: allow reconfiguring an existing enlistment\n>>  - scalar: implement the `run` command\n>>  - scalar: teach 'clone' to support the --single-branch option\n>>  - scalar: implement the `clone` subcommand\n>>  - scalar: implement 'scalar list'\n>>  - scalar: let 'unregister' handle a deleted enlistment directory gracefully\n>>  - scalar: 'unregister' stops background maintenance\n>>  - scalar: 'register' sets recommended config and starts maintenance\n>>  - scalar: create test infrastructure\n>>  - scalar: start documenting the command\n>>  - scalar: create a rudimentary executable\n>>\n>>  Add pieces from \"scalar\" to contrib/.\n>>\n>>  Will merge to 'next'?\n>\n> I feel bad for taking so long to take a look, but I finally responded\n> with a few comments.  Mostly, I'm glad it's a contrib thing; a lot of\n> the config makes sense to me (even if some of it is 'meh' for my\n> repository sizes or setup), but two of the config settings would be\n> very objectionable if they were a core git command.  Since it's\n> contrib, though, it's probably fine to be very opinionated...and\n> perhaps even excessively so.  ;-)\n>\n> However, since Johannes has been away for a couple weeks, maybe give\n> him a chance to return and respond to myself and others (and perhaps\n> push any updates that occurred to him while on vacation) before\n> merging down?\n\nSince the thread at\nhttps://lore.kernel.org/git/87ilz44kdk.fsf@evledraar.gmail.com/ I've\nbeen running with an altered version of Johannes's patches that does the\nMakefile integration differently, per my\nhttps://lore.kernel.org/git/87ilz44kdk.fsf@evledraar.gmail.com/.\n\nI've been trickling in some of the Makefile changes that make that\neasier (still quite possible on top of \"master\", I just had some\ncleanups). It builds[1], tests[2], installs[3] and participates in any\nauxiliary targets like \"check-docs\", \"sparse\"[4] (and now my\n\"sparse-incr\") etc.\n\nAFAIKT Johannes's version is so far just doing [1] and [2], and just\noptionally if you \"make\" in the contrib subdirectory.\n\nJunio seemed positive on that direction in [1][2]. Re your comments in\n[3]: Right now \"scalar\" doesn't define any of its own config, but one\nway out of some of your comments seems to be to do that.\n\nIf it's using its own build system in contrib/scalar that'll happen how\nexactly? We keep the main manpage in contrib/scalar/scalar.txt but have\na Documentation/config/scalar.txt? Have \"git help --config\" grow some\noptional mode for scalar-specific config? Add a \"scalar help-config\"?\n\nIn case my opinion on it isn't painfully obvious at this point I think\nthe answer for both current & future integration reasons should be to\njust have its assets & build logic intergrated with existing locations\nand logic.\n\n1. https://lore.kernel.org/git/xmqqilz32hhr.fsf@gitster.g/\n2. https://lore.kernel.org/git/xmqq1r5qzv35.fsf@gitster.g/\n3. https://lore.kernel.org/git/CABPp-BG_wupp1o5bBSYOJSvF3eJjf=TbX0RBHqqKuD+3F8s6hw@mail.gmail.com/\n"},{"id":"437265","messageId":"87ee99xg4o.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"da/difftool (was: Re: What's cooking in git.git (Sep 2021, #08; Mon, 27))","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T08:22:30Z","receivedAt":"2021-09-28T08:23:56Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Junio C Hamano wrote:\n\n> * da/difftool (2021-09-23) 4 commits\n>  - difftool: add a missing space to the run_dir_diff() comments\n>  - difftool: create a tmpdir path without repeated slashes\n>  - Merge branch 'da/difftool-dir-diff-symlink-fix' into da/difftool\n>  - Merge branch 'ab/retire-option-argument' into da/difftool\n>  (this branch uses da/difftool-dir-diff-symlink-fix.)\n>\n>  Code clean-up in \"git difftool\".\n>\n>  Will merge to 'next'?\n\nI think so, I looked it over and left a comment about a weird edge case\nin the pre-image to do with negative return codes, but all of the logic\nseems OK to me.\n"},{"id":"437266","messageId":"87a6jxxfrt.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"ns/batched-fsync & en/remerge-diff (was Re: What's cooking in git.git (Sep 2021, #08; Mon, 27))","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T08:23:56Z","receivedAt":"2021-09-28T08:31:39Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Junio C Hamano wrote:\n\n> * ns/batched-fsync (2021-09-27) 8 commits\n>  - core.fsyncobjectfiles: performance tests for add and stash\n>  - core.fsyncobjectfiles: tests for batch mode\n>  - unpack-objects: use the bulk-checkin infrastructure\n>  - update-index: use the bulk-checkin infrastructure\n>  - core.fsyncobjectfiles: add windows support for batch mode\n>  - core.fsyncobjectfiles: batched disk flushes\n>  - bulk-checkin: rename 'state' variable and separate 'plugged' boolean\n>  - object-file.c: do not rename in a temp odb\n>\n>  The \"core.fsyncobjectfiles\" configuration variable can now be set\n>  to \"batch\" for improved performance.\n>\n>  Will merge to 'next'?\n\nThis version was significantly re-rolled in response to my feedback, I\nhaven't had time to look at this latest version in detail, but will try\nto do so soon.\n\nI did note in\nhttps://lore.kernel.org/git/87v92lxhh4.fsf@evledraar.gmail.com/ that it\none of Elijah's in-flight seem to have independently come up with\nslightly incompatible versions of the same tmp-objdir.c API....\n\n> * en/remerge-diff (2021-08-31) 7 commits\n>  - doc/diff-options: explain the new --remerge-diff option\n>  - show, log: provide a --remerge-diff capability\n>  - tmp-objdir: new API for creating and removing primary object dirs\n>  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n>  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n>  - merge-ort: add ability to record conflict messages in a file\n>  - merge-ort: mark a few more conflict messages as omittable\n>\n>  A new presentation for two-parent merge \"--remerge-diff\" can be\n>  used to show the difference between mechanical (and possibly\n>  conflicted) merge results and the recorded resolution.\n>\n>  Will merge to 'next'?\n\n...i.e. this series\n"},{"id":"437267","messageId":"875yulxfow.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"sg/test-split-index-fix (was Re: What's cooking in git.git (Sep 2021, #08; Mon, 27))","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T08:31:40Z","receivedAt":"2021-09-28T08:33:23Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Junio C Hamano wrote:\n\n> * sg/test-split-index-fix (2021-09-07) 7 commits\n>  - read-cache: fix GIT_TEST_SPLIT_INDEX\n>  - tests: disable GIT_TEST_SPLIT_INDEX for sparse index tests\n>  - read-cache: look for shared index files next to the index, too\n>  - t1600-index: disable GIT_TEST_SPLIT_INDEX\n>  - t1600-index: don't run git commands upstream of a pipe\n>  - t1600-index: remove unnecessary redirection\n>  - Merge branch 'ds/sparse-index-ignored-files' into sg/test-split-index-fix\n>\n>  Test updates.\n>\n>  Will merge to 'next'?\n\nIt LGTM. It introduces a \"new\" memory leak that I re-rolled\nab/sanitize-leak-ci for, but on closer inspection it's a leak that was\nthere all along, but we didn't see due to the issue the series fixes:\nThe GIT_TEST_SPLIT_INDEX test mode was broken.\n"},{"id":"437268","messageId":"871r59xfgk.fsf@evledraar.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"hn/reftable (Re: What's cooking in git.git (Sep 2021, #08; Mon, 27))","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T08:35:36Z","receivedAt":"2021-09-28T08:38:28Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Junio C Hamano wrote:\n\n> * hn/reftable (2021-09-10) 20 commits\n>  - fixup! reftable: implement stack, a mutable database of reftable files.\n>  - Add \"test-tool dump-reftable\" command.\n>  - reftable: add dump utility\n>  - reftable: implement stack, a mutable database of reftable files.\n>  - reftable: implement refname validation\n>  - reftable: add merged table view\n>  - reftable: add a heap-based priority queue for reftable records\n>  - reftable: reftable file level tests\n>  - reftable: read reftable files\n>  - reftable: generic interface to tables\n>  - reftable: write reftable files\n>  - reftable: a generic binary tree implementation\n>  - reftable: reading/writing blocks\n>  - Provide zlib's uncompress2 from compat/zlib-compat.c\n>  - reftable: (de)serialization for the polymorphic record type.\n>  - reftable: add blocksource, an abstraction for random access reads\n>  - reftable: utility functions\n>  - reftable: add error related functionality\n>  - reftable: RFC: add LICENSE\n>  - hash.h: provide constants for the hash IDs\n>\n>  The \"reftable\" backend for the refs API, without integrating into\n>  the refs subsystem.\n>\n>  Will merge to 'next'?\n\nI think we've reached approximately \"good enough\" with this for the next\nsteps, and can hopefully fix any remaining nits (such as my [1])\npost-merge.\n\nMaybe hold off until Han-Wen gets a chance to ack it, and whether he's\nok with the proposed fixup(s)?\n\nWasn't there an outstanding \"some of this in reftable/* should be static\nfunctions\" from someone, Carlo? (CC'd). In any case that sort of thing\ncould also be a post-cleanup, I couldn't find a reference to that\ndiscussion in anything except my vague memory of it as I wrote this.\n\n1. https://lore.kernel.org/git/87wnn62nhp.fsf@evledraar.gmail.com/\n"},{"id":"437280","messageId":"CAFQ2z_NxqLVfSWNtcXdyWQsTXrTrWGq7Qof=wdx9mmeqOAJS_Q@mail.gmail.com","threadId":"56595","inReplyTo":"871r59xfgk.fsf@evledraar.gmail.com","subject":"Re: hn/reftable (Re: What's cooking in git.git (Sep 2021, #08; Mon, 27))","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-09-28T12:18:59Z","receivedAt":"2021-09-28T12:19:41Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Sep 28, 2021 at 10:38 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> > * hn/reftable (2021-09-10) 20 commits\n> >  - fixup! reftable: implement stack, a mutable database of reftable files.\n> >  - Add \"test-tool dump-reftable\" command.\n> >  - reftable: add dump utility\n> >  - reftable: implement stack, a mutable database of reftable files.\n> >  - reftable: implement refname validation\n> >  - reftable: add merged table view\n> >  - reftable: add a heap-based priority queue for reftable records\n> >  - reftable: reftable file level tests\n> >  - reftable: read reftable files\n> >  - reftable: generic interface to tables\n> >  - reftable: write reftable files\n> >  - reftable: a generic binary tree implementation\n> >  - reftable: reading/writing blocks\n> >  - Provide zlib's uncompress2 from compat/zlib-compat.c\n> >  - reftable: (de)serialization for the polymorphic record type.\n> >  - reftable: add blocksource, an abstraction for random access reads\n> >  - reftable: utility functions\n> >  - reftable: add error related functionality\n> >  - reftable: RFC: add LICENSE\n> >  - hash.h: provide constants for the hash IDs\n> >\n> >  The \"reftable\" backend for the refs API, without integrating into\n> >  the refs subsystem.\n> >\n> >  Will merge to 'next'?\n>\n> I think we've reached approximately \"good enough\" with this for the next\n> steps, and can hopefully fix any remaining nits (such as my [1])\n> post-merge.\n\nThere is still an RFC in 02/19. Maybe we can get agreement that this\nis OK and drop the RFC ?\n\n> Maybe hold off until Han-Wen gets a chance to ack it, and whether he's\n> ok with the proposed fixup(s)?\n\nregarding uncompress2: I thought it was functionality best left to\nzlib to implement, but I imagine git.git offers something similar.\nSounds good to use that.\n\n> Wasn't there an outstanding \"some of this in reftable/* should be static\n> functions\" from someone, Carlo? (CC'd). In any case that sort of thing\n> could also be a post-cleanup, I couldn't find a reference to that\n> discussion in anything except my vague memory of it as I wrote this.\n\nI think I have addressed all outstanding issues in my github PR, and\nI'll send it out once I see CI pass.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"437303","messageId":"97eb5075-2ef3-8891-138a-44a035437404@gmail.com","threadId":"56595","inReplyTo":"CABPp-BGuzd_TH57-1RvwJQD5r3S3ZkJcuiPnU8aWee8pgzUBEw@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-28T13:31:28Z","receivedAt":"2021-09-28T13:31:40Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/28/2021 2:46 AM, Elijah Newren wrote:\n> [Did some slight re-ordering of topics]\n> \n> On Mon, Sep 27, 2021 at 5:53 PM Junio C Hamano <gitster@pobox.com> wrote:\n> \n>> * ds/add-rm-with-sparse-index (2021-09-24) 13 commits\n>>  - advice: update message to suggest '--sparse'\n>>  - mv: refuse to move sparse paths\n>>  - rm: skip sparse paths with missing SKIP_WORKTREE\n>>  - rm: add --sparse option\n>>  - add: update --renormalize to skip sparse paths\n>>  - add: update --chmod to skip sparse paths\n>>  - add: implement the --sparse option\n>>  - add: skip tracked paths outside sparse-checkout cone\n>>  - add: fail when adding an untracked sparse file\n>>  - dir: fix pattern matching on dirs\n>>  - dir: select directories correctly\n>>  - t1092: behavior for adding sparse files\n>>  - t3705: test that 'sparse_entry' is unstaged\n>>\n>>  \"git add\", \"git mv\", and \"git rm\" have been adjusted to avoid\n>>  updating paths outside of the sparse-checkout definition unless\n>>  the user specifies a \"--sparse\" option.\n>>\n>>  Will merge to 'next'?\n> \n> It would be nice to see the --diff-filter=u change, which you also\n> seemed to like[1]; but after that, yeah this is ready to merge down.\n> \n> [1] https://lore.kernel.org/git/xmqq35pppwsm.fsf@gitster.g/\n\nYes, I agree. I was waiting to see if more comments trickled in, but\nit seems stable now. Do you want me to re-roll the whole series, or\ndo you want to apply the fixup below?\n\nThanks,\n-Stolee\n\n\n---- >8 ----\n\nFrom d279bd580ad3b66187f9a8c0370acc5bca7cc5b6 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee <dstolee@microsoft.com>\nDate: Tue, 28 Sep 2021 09:30:10 -0400\nSubject: [PATCH] fixup! t1092: behavior for adding sparse files\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex a00e42fa233..ca91c6a67f8 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -192,7 +192,7 @@ test_sparse_unstaged () {\n \tfor repo in sparse-checkout sparse-index\n \tdo\n \t\t# Skip \"unmerged\" paths\n-\t\tgit -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n+\t\tgit -C $repo diff --staged --diff-filter=u -- \"$file\" >diff &&\n \t\ttest_must_be_empty diff || return 1\n \tdone\n }\n-- \n2.33.0.vfs.0.0\n\n\n"},{"id":"437331","messageId":"xmqqo88clixa.fsf@gitster.g","threadId":"56595","inReplyTo":"CABPp-BGuzd_TH57-1RvwJQD5r3S3ZkJcuiPnU8aWee8pgzUBEw@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T17:16:33Z","receivedAt":"2021-09-28T17:16:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren <newren@gmail.com> writes:\n\n>> * js/scalar (2021-09-14) 15 commits\n> ...\n> However, since Johannes has been away for a couple weeks, maybe give\n> him a chance to return and respond to myself and others (and perhaps\n> push any updates that occurred to him while on vacation) before\n> merging down?\n\nFair enough.\n\n>> * en/remerge-diff (2021-08-31) 7 commits\n>>  - doc/diff-options: explain the new --remerge-diff option\n>>  - show, log: provide a --remerge-diff capability\n>>  - tmp-objdir: new API for creating and removing primary object dirs\n>>  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n>>  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n>>  - merge-ort: add ability to record conflict messages in a file\n>>  - merge-ort: mark a few more conflict messages as omittable\n>>\n>>  A new presentation for two-parent merge \"--remerge-diff\" can be\n>>  used to show the difference between mechanical (and possibly\n>>  conflicted) merge results and the recorded resolution.\n>>\n>>  Will merge to 'next'?\n>\n> It has been a month that it's been cooking with no issues brought up,\n> and it's been in production for nearly a year...\n\nPlease do not read that much for being in \"seen\".  Until a topic\nhits 'next', where some orgs package and ship to their internal\naudience, I am not sure if it can be called \"cooking\".\n\nBut your using it on your folks in the production (how big is your\naudience, I don't know) does count ;-)\n\n> But just this morning I pinged peff and jrnieder if they might have\n> time to respectively look at the tmp-objdir stuff (patch 5, plus its\n> integration into log-tree.c in patch 7) and the ll-merge.[ch] changes\n> (patch 3).  I don't know if either will have time to do it, but\n> perhaps wait half a week or so to see if they'll mention they have\n> time?\n\nSure.\n"},{"id":"437334","messageId":"xmqqilykliiz.fsf@gitster.g","threadId":"56595","inReplyTo":"87v92lxhh4.fsf@evledraar.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T17:25:08Z","receivedAt":"2021-09-28T17:25:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> I haven't poked at it much, but haven't you and Neeraj Singh (CC'd)\n> independently come up with two slightly different changes in\n> tmp-objdir.c to do the same thing? See the tmp-objdir.c part of:\n>\n> http://lore.kernel.org/git/543ea3569342165363c1602ce36683a54dce7a0b.1632527609.git.gitgitgadget@gmail.com\n>\n> And your:\n>\n> http://lore.kernel.org/git/67d3b2b09f9ddda616cdd0d1b12ab7afc73670ed.1630376800.git.gitgitgadget@gmail.com\n>\n> I.e. yours has the object database managed outside, his has it added to\n> \"struct tmp_objdir\", but it's the same objdir dance isn't it?\n\nThey touch the same tmp-objdir, but unlike the original use in the\nreceive-pack (i.e. responding to \"git push\") with the intention to\nadd the objects collected in them back to the primary, remerge-diff\nwants to discard what was added there at the end.  I do not think it\nwould directly help the bulk-fsync stuff (but I didn't quite see why\nbulk-fsync stuff needed to _add_ new functions to tmp-objdir API,\ninstead of just being a customer of tmp-objdir API), where it wants\nto do the same _migrate() dance in the end.\n"},{"id":"437335","messageId":"xmqqczoslieh.fsf@gitster.g","threadId":"56595","inReplyTo":"87ilylxg85.fsf@evledraar.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T17:27:50Z","receivedAt":"2021-09-28T17:27:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> On Mon, Sep 27 2021, Elijah Newren wrote:\n>\n> [Aside: I quite liked the method of replying to What's Cooking per-topic\n> in Johannes's replies to\n> https://lore.kernel.org/git/xmqqsfyf5b74.fsf@gitster.g/; I'm never quite\n> sure if I should have one E-Mail with all of my comments on other\n> in-flight work, use existing commentary as a springboard etc. Doing the\n> latter here]\n\nYup.  Splitting them out may be more work to produce comments, but\nit lets you Cc relevant folks and would be easier for those who\nparticipate in the discussion.\n"},{"id":"437336","messageId":"xmqq7df0li4n.fsf@gitster.g","threadId":"56595","inReplyTo":"97eb5075-2ef3-8891-138a-44a035437404@gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T17:33:44Z","receivedAt":"2021-09-28T17:33:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> Yes, I agree. I was waiting to see if more comments trickled in, but\n> it seems stable now. Do you want me to re-roll the whole series, or\n> do you want to apply the fixup below?\n\nLet us wait for a bit more trickling in, but in the meantime, here\nis what I'd keep.  Thanks.\n\n[ds/add-rm-with-sparse-index] $ git range-diff @{1}...\n 1:  f1309dd7be !  1:  edd2cd345f t1092: behavior for adding sparse files\n    @@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {\n     +\tfor repo in sparse-checkout sparse-index\n     +\tdo\n     +\t\t# Skip \"unmerged\" paths\n    -+\t\tgit -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n    ++\t\tgit -C $repo diff --staged --diff-filter=u -- \"$file\" >diff &&\n     +\t\ttest_must_be_empty diff || return 1\n     +\tdone\n     +}\n 2:  9a8c340809 =  2:  f6526728f9 dir: select directories correctly\n 3:  2a4e495cd4 =  3:  ed4958477b dir: fix pattern matching on dirs\n ...\n\n"},{"id":"437360","messageId":"98d25b30-9961-3356-69f0-890f9b1006da@gmail.com","threadId":"56595","inReplyTo":"xmqq7df0li4n.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-28T20:16:57Z","receivedAt":"2021-09-28T20:17:01Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/28/2021 1:33 PM, Junio C Hamano wrote:\n> Derrick Stolee <stolee@gmail.com> writes:\n> \n>> Yes, I agree. I was waiting to see if more comments trickled in, but\n>> it seems stable now. Do you want me to re-roll the whole series, or\n>> do you want to apply the fixup below?\n> \n> Let us wait for a bit more trickling in, but in the meantime, here\n> is what I'd keep.  Thanks.\n> \n> [ds/add-rm-with-sparse-index] $ git range-diff @{1}...\n>  1:  f1309dd7be !  1:  edd2cd345f t1092: behavior for adding sparse files\n>     @@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {\n>      +\tfor repo in sparse-checkout sparse-index\n>      +\tdo\n>      +\t\t# Skip \"unmerged\" paths\n>     -+\t\tgit -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n>     ++\t\tgit -C $repo diff --staged --diff-filter=u -- \"$file\" >diff &&\n>      +\t\ttest_must_be_empty diff || return 1\n>      +\tdone\n>      +}\n>  2:  9a8c340809 =  2:  f6526728f9 dir: select directories correctly\n>  3:  2a4e495cd4 =  3:  ed4958477b dir: fix pattern matching on dirs\n\nThat's exactly the diff I was hoping for. I'll keep an eye for more\nfeedback.\n\nThanks,\n-Stolee\n"},{"id":"437364","messageId":"xmqqfstojudr.fsf@gitster.g","threadId":"56595","inReplyTo":"87czotzaru.fsf@evledraar.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T20:52:00Z","receivedAt":"2021-09-28T20:52:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n>> * ab/http-pinned-public-key-mismatch (2021-09-27) 1 commit\n>>  - http: check CURLE_SSL_PINNEDPUBKEYNOTMATCH when emitting errors\n>>\n>>  HTTPS error handling updates.\n>>\n>>  Will merge to 'next'?\n>\n> I'd like it merged. I think a fair summary of Jeff King's\n> https://lore.kernel.org/git/YU5CJC9XJvQITfr8@coredump.intra.peff.net/ is\n> \"meh\" though :)\n\nMine is also \"I do not mind seeing it in or out.\"\n\n>> * ab/pack-objects-stdin (2021-07-09) 5 commits\n>>  - pack-objects.c: make use of REV_INFO_STDIN_LINE_PROCESS\n>>  - pack-objects.c: do stdin parsing via revision.c's API\n>>  - revision.[ch]: add a \"handle_stdin_line\" API\n>>  - revision.h: refactor \"disable_stdin\" and \"read_from_stdin\"\n>>  - upload-pack: run is_repository_shallow() before setup_revisions()\n>>\n>>  Introduce handle_stdin_line callback to revision API and uses it.\n>>\n>>  Expecting a reroll.\n>\n> I said I'd re-roll this into the larger \"git bundle --stdin\" topic I had\n> in mind, but when doing so found one memory leak, which I really wanted\n> the SANITIZE=leak interface to fix.\n>\n> I think it's best to eject/discard this given the amount of other\n> outstanding stuff I've got. I don't really *need* this for anything\n> else, I'll loop back to it sometime after the SANITIZE=leak testing has\n> landed.\n\nOK, will mark as \"Retracted for now\".\n\n>> * ab/fsck-unexpected-type (2021-09-22) 17 commits\n>> [...]\n>>  \"git fsck\" has been taught to report mismatch between expected and\n>>  actual types of an object better.\n>>\n>>  Needs review.\n>\n> The \"needs review\" note here pre-dates v6, which got a thorough review,\n> and Taylor seemed happy with the v7, aside from a minor issue I just\n> re-rolled v8 with a fix for:\n>\n> https://lore.kernel.org/git/cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com/\n\nThanks.\n"},{"id":"437365","messageId":"20210928210059.vy5isvmcj75vufdu@neerajsi-x1.localdomain","threadId":"56595","inReplyTo":"xmqqilykliiz.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Neeraj Singh","fromEmail":"nksingh85@gmail.com","sentAt":"2021-09-28T21:00:59Z","receivedAt":"2021-09-28T21:01:04Z","isPatch":false,"sender":{"key":"nksingh85@gmail.com","avatar":null},"body":"On Tue, Sep 28, 2021 at 10:25:08AM -0700, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n> \n> > I haven't poked at it much, but haven't you and Neeraj Singh (CC'd)\n> > independently come up with two slightly different changes in\n> > tmp-objdir.c to do the same thing? See the tmp-objdir.c part of:\n> >\n> > http://lore.kernel.org/git/543ea3569342165363c1602ce36683a54dce7a0b.1632527609.git.gitgitgadget@gmail.com\n> >\n> > And your:\n> >\n> > http://lore.kernel.org/git/67d3b2b09f9ddda616cdd0d1b12ab7afc73670ed.1630376800.git.gitgitgadget@gmail.com\n> >\n> > I.e. yours has the object database managed outside, his has it added to\n> > \"struct tmp_objdir\", but it's the same objdir dance isn't it?\n> \n> They touch the same tmp-objdir, but unlike the original use in the\n> receive-pack (i.e. responding to \"git push\") with the intention to\n> add the objects collected in them back to the primary, remerge-diff\n> wants to discard what was added there at the end.  I do not think it\n> would directly help the bulk-fsync stuff (but I didn't quite see why\n> bulk-fsync stuff needed to _add_ new functions to tmp-objdir API,\n> instead of just being a customer of tmp-objdir API), where it wants\n> to do the same _migrate() dance in the end.\n\nBoth Elijah and I needed the same functionality of having a writable\nODB in the current process, which wasn't previously provided by tmp-objdir.\n\nI'm making a new patch which is an amalgamation of Elijah's version and mine.\nI'll rebase and resend my patch series with that version and I'll also make\na modified version of Elijah's branch available on github.\n\nThanks,\nNeeraj\n"},{"id":"437389","messageId":"xmqq8rzgi8a7.fsf@gitster.g","threadId":"56595","inReplyTo":"20210928210059.vy5isvmcj75vufdu@neerajsi-x1.localdomain","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T23:34:40Z","receivedAt":"2021-09-28T23:34:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Neeraj Singh <nksingh85@gmail.com> writes:\n\n> Both Elijah and I needed the same functionality of having a writable\n> ODB in the current process, which wasn't previously provided by tmp-objdir.\n>\n> I'm making a new patch which is an amalgamation of Elijah's version and mine.\n> I'll rebase and resend my patch series with that version and I'll also make\n> a modified version of Elijah's branch available on github.\n\nSounds good.  Thanks.\n"},{"id":"437391","messageId":"YVOn3hDsb5pnxR53@coredump.intra.peff.net","threadId":"56595","inReplyTo":"CABPp-BGuzd_TH57-1RvwJQD5r3S3ZkJcuiPnU8aWee8pgzUBEw@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-28T23:40:14Z","receivedAt":"2021-09-28T23:40:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 27, 2021 at 11:46:40PM -0700, Elijah Newren wrote:\n\n> > * en/remerge-diff (2021-08-31) 7 commits\n> >  - doc/diff-options: explain the new --remerge-diff option\n> >  - show, log: provide a --remerge-diff capability\n> >  - tmp-objdir: new API for creating and removing primary object dirs\n> >  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n> >  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n> >  - merge-ort: add ability to record conflict messages in a file\n> >  - merge-ort: mark a few more conflict messages as omittable\n> >\n> >  A new presentation for two-parent merge \"--remerge-diff\" can be\n> >  used to show the difference between mechanical (and possibly\n> >  conflicted) merge results and the recorded resolution.\n> >\n> >  Will merge to 'next'?\n> \n> It has been a month that it's been cooking with no issues brought up,\n> and it's been in production for nearly a year...\n> \n> But just this morning I pinged peff and jrnieder if they might have\n> time to respectively look at the tmp-objdir stuff (patch 5, plus its\n> integration into log-tree.c in patch 7) and the ll-merge.[ch] changes\n> (patch 3).  I don't know if either will have time to do it, but\n> perhaps wait half a week or so to see if they'll mention they have\n> time?  Otherwise, yeah, it's probably time to merge this down.\n\nSorry to take so long. I think this is a very exciting topic, and I\nappreciate being called into to look at tmp-objdir stuff, because it can\nbe quite subtle.\n\nI just left a rather long-ish mail in the thread, but the gist of it is\nthat I'm worried that there's some possibility of corruption there if\nthe diff code writes objects. I didn't do a proof-of-concept there, but\nI worked one up just now. Try this:\n\n  # make a repo\n  git init repo\n  cd repo\n  echo base >file\n  git add file\n  git commit -m base\n\n  # two diverging branches\n  echo main >file\n  git commit -am main\n  git checkout -b side HEAD^\n  echo side >file\n  git commit -am side\n\n  # we get a conflict, and we resolve\n  git merge main\n  echo resolved >file\n  git commit -am merged\n\n  # now imagine we had a file with a diff driver. I stuffed it\n  # in here after the fact, but it could have been here all along,\n  # or come as part of the merge, etc.\n  echo whatever >unrelated\n  echo \"unrelated diff=foo\" >.gitattributes\n  git add .\n  git commit --amend --no-edit\n\n  # set up the diff driver not just to do a textconv, but to cache the\n  # result. This will require writing out new objects for the cache\n  # as part of the diff operation.\n  git config diff.foo.textconv \"$PWD/upcase\"\n  git config diff.foo.cachetextconv true\n  cat >upcase <<\\EOF &&\n  #!/bin/sh\n  tr a-z A-Z <$1\n  EOF\n  chmod +x upcase\n\n  # now show the diff\n  git log -1 --remerge-diff\n\n  # and make sure the repo is still OK\n  git fsck\n\nThe remerge diff will correctly show the textconv'd content (because\nit's not in either parent, and hence an evil merge):\n\n  diff --git a/unrelated b/unrelated\n  new file mode 100644\n  index 0000000..982793c\n  --- /dev/null\n  +++ b/unrelated\n  @@ -0,0 +1 @@\n  +WHATEVER\n\nbut then fsck shows that our cache is corrupted:\n\n  Checking object directories: 100% (256/256), done.\n  error: refs/notes/textconv/foo: invalid sha1 pointer 1e9b3ecffca73411001043fe914a7ec0e7291043\n  error: refs/notes/textconv/foo: invalid reflog entry 1e9b3ecffca73411001043fe914a7ec0e7291043\n  dangling tree 569b6e414203d2f50bdaafc1585eae11e28afc37\n\nNow I'll admit the textconv-cache is a pretty seldom-used feature. And\nthat there _probably_ aren't a lot of other ways that the diff code\nwould try to write objects or references. But I think it speaks to the\nkind of subtle interactions we should worry about (and certainly\ncorrupting the repository is a pretty bad outcome, though at least in\nthis case it's \"just\" a cache and we could blow away that ref manually).\n\n-Peff\n"},{"id":"437394","messageId":"YVOqJTU5HmnXlC5P@coredump.intra.peff.net","threadId":"56595","inReplyTo":"YVOn3hDsb5pnxR53@coredump.intra.peff.net","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-28T23:49:57Z","receivedAt":"2021-09-28T23:50:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 28, 2021 at 07:40:14PM -0400, Jeff King wrote:\n\n> but then fsck shows that our cache is corrupted:\n> \n>   Checking object directories: 100% (256/256), done.\n>   error: refs/notes/textconv/foo: invalid sha1 pointer 1e9b3ecffca73411001043fe914a7ec0e7291043\n>   error: refs/notes/textconv/foo: invalid reflog entry 1e9b3ecffca73411001043fe914a7ec0e7291043\n>   dangling tree 569b6e414203d2f50bdaafc1585eae11e28afc37\n> \n> Now I'll admit the textconv-cache is a pretty seldom-used feature. And\n> that there _probably_ aren't a lot of other ways that the diff code\n> would try to write objects or references. But I think it speaks to the\n> kind of subtle interactions we should worry about (and certainly\n> corrupting the repository is a pretty bad outcome, though at least in\n> this case it's \"just\" a cache and we could blow away that ref manually).\n\nI haven't looked at the way the bulk-fsync code uses tmp_objdir at all\nyet, but this is the same kind of thing to be watching out for. I've\nadded Neeraj to the cc. The full reproduction showing the problem with\nElijah's patch is in the replied-to mail:\n\n  https://lore.kernel.org/git/YVOn3hDsb5pnxR53@coredump.intra.peff.net/\n\nThe batch-fsync stuff might be OK in that the goal there is presumably\nnot to throw away the result, but to eventually migrate it into the odb.\nSo the unintended interaction to look out for there might be more like:\n\n  - some code foo() wants to write a bunch of objects; it opens up a\n    tmp_objdir to batch them\n\n  - some other code bar() deep in the call-stack wants to write an\n    object for whatever reason (maybe it's a caching textconv). Its\n    object ends up in the tmp_objdir, too.\n\n  - bar() writes a ref pointing to the object's oid, thinking it has\n    fully written the object.\n\n  - foo() encounters an error and aborts, rolling back the tmp_objdir,\n    including the object written by bar(). Now our ref is corrupt.\n\nAs I said, I haven't looked at how the bulk-fsync stuff uses\ntmp_objdir. But if it's doing the same \"globally all object writes go to\nthe tmpdir\" then it's at risk of this sort of thing.\n\n-Peff\n"},{"id":"437395","messageId":"CANQDOddZ-KYTB4q0nYNDinis8aKktm6Ek6F+mJouTV-yRtTpUw@mail.gmail.com","threadId":"56595","inReplyTo":"xmqq8rzgi8a7.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Neeraj Singh","fromEmail":"nksingh85@gmail.com","sentAt":"2021-09-28T23:53:13Z","receivedAt":"2021-09-28T23:53:30Z","isPatch":false,"sender":{"key":"nksingh85@gmail.com","avatar":null},"body":"On Tue, Sep 28, 2021 at 4:34 PM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Neeraj Singh <nksingh85@gmail.com> writes:\n>\n> > Both Elijah and I needed the same functionality of having a writable\n> > ODB in the current process, which wasn't previously provided by tmp-objdir.\n> >\n> > I'm making a new patch which is an amalgamation of Elijah's version and mine.\n> > I'll rebase and resend my patch series with that version and I'll also make\n> > a modified version of Elijah's branch available on github.\n>\n> Sounds good.  Thanks.\n\nI just posted the new series to the list.\n\nElijah,\n\nHere's a branch of your changes based on the amalgamated tmp-objdir code:\nhttps://github.com/neerajsi-msft/git/commits/neerajsi/remerge-diff\n\nThis commit adapts your code to use the amalgamated API:\nhttps://github.com/neerajsi-msft/git/commit/725328fe1d8be8326d2ddef78e164ca21450b100\n\nThanks,\nNeeraj\n"},{"id":"437439","messageId":"CABPp-BF-tEk-ef3tVD_27a2UbcrRsZw8y+49+v0H5E39UPgPrg@mail.gmail.com","threadId":"56595","inReplyTo":"xmqqo88clixa.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-29T06:42:45Z","receivedAt":"2021-09-29T06:42:59Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 28, 2021 at 10:16 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Elijah Newren <newren@gmail.com> writes:\n>\n> >> * js/scalar (2021-09-14) 15 commits\n> > ...\n> > However, since Johannes has been away for a couple weeks, maybe give\n> > him a chance to return and respond to myself and others (and perhaps\n> > push any updates that occurred to him while on vacation) before\n> > merging down?\n>\n> Fair enough.\n>\n> >> * en/remerge-diff (2021-08-31) 7 commits\n> >>  - doc/diff-options: explain the new --remerge-diff option\n> >>  - show, log: provide a --remerge-diff capability\n> >>  - tmp-objdir: new API for creating and removing primary object dirs\n> >>  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n> >>  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n> >>  - merge-ort: add ability to record conflict messages in a file\n> >>  - merge-ort: mark a few more conflict messages as omittable\n> >>\n> >>  A new presentation for two-parent merge \"--remerge-diff\" can be\n> >>  used to show the difference between mechanical (and possibly\n> >>  conflicted) merge results and the recorded resolution.\n> >>\n> >>  Will merge to 'next'?\n> >\n> > It has been a month that it's been cooking with no issues brought up,\n> > and it's been in production for nearly a year...\n>\n> Please do not read that much for being in \"seen\".  Until a topic\n> hits 'next', where some orgs package and ship to their internal\n> audience, I am not sure if it can be called \"cooking\".\n\nAs a heads up, I found this topic in an email titled \"What's\nCooking...\", and in particular it was listed in a section of that\nemail that was titled \"[Cooking]\".  ;-)\n\nSorry, I couldn't resist.\n\nBut yeah, I know \"seen\" doesn't mean much.  My comment was reflecting\nthat while leaving a topic in seen for a while sometimes results in\nmore information coming forth to inform the decision about whether to\nmerge the series to next, or drop it, or ask the submitter for\nchanges, there comes a point where further waiting is unlikely to\nprovide any additional information that would help inform that\ndecision.  I was starting to wonder whether we'd get any (more)\nfeedback.  But I'm glad we waited a little longer.  :-)\n\n> But your using it on your folks in the production (how big is your\n> audience, I don't know) does count ;-)\n\n~100 opt-in users (apparently shot up in the past few months; used to\nbe ~50).  I have a hard time gauging how heavily this particular\nfeature was/is used by them, however:\n\n  * The feature was explicitly announced, but I did it because I\nthought it was cool and a nice demonstration of merge-ort features,\nnot because there were explicit requests for it\n  * I did make \"-p\" imply \"--remerge-diff\" for git-log in our internal\nopt-in distribution, so that should increase usage significantly.\n  * A bug in an internal tool for a while was accidentally running\n`git log -p --name-status $COMMIT --not\n--remotes=ACCIDENTALLY_INVALID_REMOTE_NAME` for a while, including in\nbig repos; in combination with previous point, this increased usage\nand scope of remerge-diff usage a fair amount as that command would\nrun on all commits in history.\n  * Most merges here are done by either GitHub (Enterprise) or Gerrit,\nboth of which require the user to rebase and re-push if there are any\nconflicts.  This makes remerge-diff boring for most merges.  There are\nsome notable exceptions where merges are done differently, they're\njust much less common.  (--remerge-diff-only for cherry-picks and\nreverts is a bit more interesting, but that's not part of this\nseries...and I also haven't gotten much feedback on it.)\n  * I have only received feedback from two users about the\nremerge-diff feature, one only indirectly, and both reports actually\nended up being about underlying bugs in early versions of merge-ort[+]\n\n[+] The bugs were (1) my QSORT in write_tree() was not originally\nusing tree_entry_order() and thus was writing trees with the wrong\nordering of elements; this was a bug that was fixed before\nwrite_tree() was ever submitted upstream, and (2) and the bug fixed by\n72b3091040f8 (merge-ort: use STABLE_QSORT instead of QSORT where\nrequired, 2021-03-20)\n\n> > But just this morning I pinged peff and jrnieder if they might have\n> > time to respectively look at the tmp-objdir stuff (patch 5, plus its\n> > integration into log-tree.c in patch 7) and the ll-merge.[ch] changes\n> > (patch 3).  I don't know if either will have time to do it, but\n> > perhaps wait half a week or so to see if they'll mention they have\n> > time?\n>\n> Sure.\n\nApparently that was a good idea; Peff and Ævar have both chimed in\nwith feedback.  :-)\n"},{"id":"437442","messageId":"fd244131-84d4-7152-c88c-45f9ada6b609@gigacodes.de","threadId":"56595","inReplyTo":"xmqq8rzhmsi7.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Fabian Stelzer","fromEmail":"fs@gigacodes.de","sentAt":"2021-09-29T08:12:38Z","receivedAt":"2021-09-29T08:12:50Z","isPatch":false,"sender":{"key":"fs@gigacodes.de","avatar":"https://avatars.githubusercontent.com/u/564858?v=4"},"body":"On 28.09.21 02:52, Junio C Hamano wrote > * fs/ssh-signing (2021-09-10)\n9 commits\n>  - ssh signing: test that gpg fails for unknown keys\n>  - ssh signing: tests for logs, tags & push certs\n>  - ssh signing: duplicate t7510 tests for commits\n>  - ssh signing: verify signatures using ssh-keygen\n>  - ssh signing: provide a textual signing_key_id\n>  - ssh signing: retrieve a default key from ssh-agent\n>  - ssh signing: add ssh key format and signing code\n>  - ssh signing: add test prereqs\n>  - ssh signing: preliminary refactoring and clean-up\n> \n>  Use ssh public crypto for object and push-cert signing.\n> \n>  On hold.\n>  cf. <pull.1041.v8.git.git.1631304462.gitgitgadget@gmail.com>\n>  cf. <532d97e7-8c91-df6a-6d90-70668256f513@gigacodes.de>\n> \n>\nOpenssh 8.8 has been released a few days ago and includes the needed fix\nfor the find-principal segfault.\nI ran the full git testsuite against it without issues.\n\nAlso, we (~30developers) have been running this patch with\nopenssh-portable (2d678c5e3bdc2f5c99f7af5122e9d054925d560d / post 8.7 -\npre 8.8) in our organization for the last 2 weeks without problems.\n\nThe only issues we saw with our users are related to some misleading\nopenssh error messages.\nFor example if you configure a public key and the private key is not\navailable via the ssh-agent the error message is: \"invalid format\".\nOr if the public key contains a typo (forgot a char in copy&pase) it\nwill error with \"no such file or directory\".\nI will need to dig a bit deeper into openssh to see if we can make these\nmore specific without breaking any compatibility. Both errors originate\nfrom some lower level lib functions which i don't want to change.\n\nBut vverall i think this is ready for some broader usage/testing via next.\n\nI'd suggest to send the additional patches for valid-before/after\nfunctionality in a new patchset for review after.\n\nBest regards,\nFabian\n"},{"id":"437475","messageId":"20210929184339.GA19712@neerajsi-x1.localdomain","threadId":"56595","inReplyTo":"YVOn3hDsb5pnxR53@coredump.intra.peff.net","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Neeraj Singh","fromEmail":"nksingh85@gmail.com","sentAt":"2021-09-29T18:43:39Z","receivedAt":"2021-09-29T18:43:44Z","isPatch":false,"sender":{"key":"nksingh85@gmail.com","avatar":null},"body":"On Tue, Sep 28, 2021 at 07:40:14PM -0400, Jeff King wrote:\n> On Mon, Sep 27, 2021 at 11:46:40PM -0700, Elijah Newren wrote:\n> \n> > > * en/remerge-diff (2021-08-31) 7 commits\n> > >  - doc/diff-options: explain the new --remerge-diff option\n> > >  - show, log: provide a --remerge-diff capability\n> > >  - tmp-objdir: new API for creating and removing primary object dirs\n> > >  - merge-ort: capture and print ll-merge warnings in our preferred fashion\n> > >  - ll-merge: add API for capturing warnings in a strbuf instead of stderr\n> > >  - merge-ort: add ability to record conflict messages in a file\n> > >  - merge-ort: mark a few more conflict messages as omittable\n> > >\n> > >  A new presentation for two-parent merge \"--remerge-diff\" can be\n> > >  used to show the difference between mechanical (and possibly\n> > >  conflicted) merge results and the recorded resolution.\n> > >\n> > >  Will merge to 'next'?\n> > \n> > It has been a month that it's been cooking with no issues brought up,\n> > and it's been in production for nearly a year...\n> > \n> > But just this morning I pinged peff and jrnieder if they might have\n> > time to respectively look at the tmp-objdir stuff (patch 5, plus its\n> > integration into log-tree.c in patch 7) and the ll-merge.[ch] changes\n> > (patch 3).  I don't know if either will have time to do it, but\n> > perhaps wait half a week or so to see if they'll mention they have\n> > time?  Otherwise, yeah, it's probably time to merge this down.\n> \n> Sorry to take so long. I think this is a very exciting topic, and I\n> appreciate being called into to look at tmp-objdir stuff, because it can\n> be quite subtle.\n> \n> I just left a rather long-ish mail in the thread, but the gist of it is\n> that I'm worried that there's some possibility of corruption there if\n> the diff code writes objects. I didn't do a proof-of-concept there, but\n> I worked one up just now. Try this:\n> \n>   # make a repo\n>   git init repo\n>   cd repo\n>   echo base >file\n>   git add file\n>   git commit -m base\n> \n>   # two diverging branches\n>   echo main >file\n>   git commit -am main\n>   git checkout -b side HEAD^\n>   echo side >file\n>   git commit -am side\n> \n>   # we get a conflict, and we resolve\n>   git merge main\n>   echo resolved >file\n>   git commit -am merged\n> \n>   # now imagine we had a file with a diff driver. I stuffed it\n>   # in here after the fact, but it could have been here all along,\n>   # or come as part of the merge, etc.\n>   echo whatever >unrelated\n>   echo \"unrelated diff=foo\" >.gitattributes\n>   git add .\n>   git commit --amend --no-edit\n> \n>   # set up the diff driver not just to do a textconv, but to cache the\n>   # result. This will require writing out new objects for the cache\n>   # as part of the diff operation.\n>   git config diff.foo.textconv \"$PWD/upcase\"\n>   git config diff.foo.cachetextconv true\n>   cat >upcase <<\\EOF &&\n>   #!/bin/sh\n>   tr a-z A-Z <$1\n>   EOF\n>   chmod +x upcase\n> \n>   # now show the diff\n>   git log -1 --remerge-diff\n> \n>   # and make sure the repo is still OK\n>   git fsck\n> \n> The remerge diff will correctly show the textconv'd content (because\n> it's not in either parent, and hence an evil merge):\n> \n>   diff --git a/unrelated b/unrelated\n>   new file mode 100644\n>   index 0000000..982793c\n>   --- /dev/null\n>   +++ b/unrelated\n>   @@ -0,0 +1 @@\n>   +WHATEVER\n> \n> but then fsck shows that our cache is corrupted:\n> \n>   Checking object directories: 100% (256/256), done.\n>   error: refs/notes/textconv/foo: invalid sha1 pointer 1e9b3ecffca73411001043fe914a7ec0e7291043\n>   error: refs/notes/textconv/foo: invalid reflog entry 1e9b3ecffca73411001043fe914a7ec0e7291043\n>   dangling tree 569b6e414203d2f50bdaafc1585eae11e28afc37\n> \n> Now I'll admit the textconv-cache is a pretty seldom-used feature. And\n> that there _probably_ aren't a lot of other ways that the diff code\n> would try to write objects or references. But I think it speaks to the\n> kind of subtle interactions we should worry about (and certainly\n> corrupting the repository is a pretty bad outcome, though at least in\n> this case it's \"just\" a cache and we could blow away that ref manually).\n> \n> -Peff\n\nIt seems to me that one problem is that the new \"primary\" objdir code doesn't\ndisable ref updates since the GIT_QUARANTINE_ENVIRONMENT setting isn't set.\nIf we fix that we should be forbidding ref updates.\n\nI've included a path that fixes my test case. This is on top of:\nhttps://lore.kernel.org/git/CANQDOddqwVtWfC4eEP3fJB4sUiszGX8bLqoEVLcMf=v+jzx19g@mail.gmail.com/\n\nFrom 38be7f6d31da8df3f205b35d25dd0a505aa75a8a Mon Sep 17 00:00:00 2001\nFrom: Neeraj Singh <neerajsi@microsoft.com>\nDate: Wed, 29 Sep 2021 11:24:05 -0700\nSubject: [PATCH] tmp-objdir: disable ref updates when replacing the primary\n odb\n\nWhen creating a subprocess with a temporary ODB, we set the\nGIT_QUARANTINE_ENVIRONMENT env var to tell child Git processes not\nto update refs, since the tmp-objdir may go away.\n\nIntroduce a similar mechanism for in-process temporary ODBs when\nwe call tmp_objdir_replace_primary_odb.\n\nPeff's test case was invoking ref updates via the cachetextconv\nsetting. That particular code silently does nothing when a ref\nupdate is forbidden\n\nReported-by: Jeff King <peff@peff.net>\n\nSigned-off-by: Neeraj Singh <neerajsi@microsoft.com>\n---\n object-file.c |  7 +++++++\n refs.c        | 22 +++++++++++++++++++++-\n refs.h        |  5 +++++\n 3 files changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 990381abee..e3e5cec3c8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -770,6 +770,12 @@ struct object_directory *set_temporary_primary_odb(const char *dir, int will_des\n \tnew_odb->will_destroy = will_destroy;\n \tnew_odb->next = the_repository->objects->odb;\n \tthe_repository->objects->odb = new_odb;\n+\n+\t/*\n+\t * Disable ref updates while a temporary odb is active, since\n+\t * the objects in the database may roll back.\n+\t */\n+\trefs_disable_updates();\n \treturn new_odb->next;\n }\n \n@@ -786,6 +792,7 @@ void restore_primary_odb(struct object_directory *restore_odb, const char *old_p\n \n \tthe_repository->objects->odb = restore_odb;\n \tfree_object_directory(cur_odb);\n+\trefs_enable_updates();\n }\n \n /*\ndiff --git a/refs.c b/refs.c\nindex 8b9f7c3a80..98026b7341 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,6 +19,8 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+static int ref_update_disabled_count;\n+\n /*\n  * List of all available backends\n  */\n@@ -1045,6 +1047,24 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \tfree(transaction);\n }\n \n+void refs_disable_updates(void)\n+{\n+\t++ref_update_disabled_count;\n+}\n+\n+void refs_enable_updates(void)\n+{\n+\tif (!ref_update_disabled_count)\n+\t\tBUG(\"ref updates are not disabled\");\n+\n+\t--ref_update_disabled_count;\n+}\n+\n+int ref_updates_are_enabled(void)\n+{\n+\treturn !ref_update_disabled_count;\n+}\n+\n struct ref_update *ref_transaction_add_update(\n \t\tstruct ref_transaction *transaction,\n \t\tconst char *refname, unsigned int flags,\n@@ -2126,7 +2146,7 @@ int ref_transaction_prepare(struct ref_transaction *transaction,\n \t\tbreak;\n \t}\n \n-\tif (getenv(GIT_QUARANTINE_ENVIRONMENT)) {\n+\tif (getenv(GIT_QUARANTINE_ENVIRONMENT) || !ref_updates_are_enabled()) {\n \t\tstrbuf_addstr(err,\n \t\t\t      _(\"ref updates forbidden inside quarantine environment\"));\n \t\treturn -1;\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e..acd7e275c5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -840,6 +840,11 @@ int ref_storage_backend_exists(const char *name);\n \n struct ref_store *get_main_ref_store(struct repository *r);\n \n+/* Helpers to mark updates to the refs as forbidden */\n+void refs_disable_updates(void);\n+void refs_enable_updates(void);\n+int ref_updates_are_enabled(void);\n+\n /**\n  * Submodules\n  * ----------\n-- \n2.25.1\n\n"},{"id":"437495","messageId":"CAPUEspj6q_N-ZPRzUv8jx_5_eP0gDO5TyTpi5-kPkfeoVhkjvg@mail.gmail.com","threadId":"56595","inReplyTo":"CAFQ2z_NxqLVfSWNtcXdyWQsTXrTrWGq7Qof=wdx9mmeqOAJS_Q@mail.gmail.com","subject":"Re: hn/reftable (Re: What's cooking in git.git (Sep 2021, #08; Mon, 27))","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-09-30T05:06:02Z","receivedAt":"2021-09-30T05:07:06Z","isPatch":false,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Tue, Sep 28, 2021 at 5:19 AM Han-Wen Nienhuys <hanwen@google.com> wrote:\n> On Tue, Sep 28, 2021 at 10:38 AM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n> > > * hn/reftable (2021-09-10) 20 commits\n> > >  - fixup! reftable: implement stack, a mutable database of reftable files.\n> > >  - Add \"test-tool dump-reftable\" command.\n> > >  - reftable: add dump utility\n> > >  - reftable: implement stack, a mutable database of reftable files.\n> > >  - reftable: implement refname validation\n> > >  - reftable: add merged table view\n> > >  - reftable: add a heap-based priority queue for reftable records\n> > >  - reftable: reftable file level tests\n> > >  - reftable: read reftable files\n> > >  - reftable: generic interface to tables\n> > >  - reftable: write reftable files\n> > >  - reftable: a generic binary tree implementation\n> > >  - reftable: reading/writing blocks\n> > >  - Provide zlib's uncompress2 from compat/zlib-compat.c\n> > >  - reftable: (de)serialization for the polymorphic record type.\n> > >  - reftable: add blocksource, an abstraction for random access reads\n> > >  - reftable: utility functions\n> > >  - reftable: add error related functionality\n> > >  - reftable: RFC: add LICENSE\n> > >  - hash.h: provide constants for the hash IDs\n> > >\n> > >  The \"reftable\" backend for the refs API, without integrating into\n> > >  the refs subsystem.\n> > >\n> > >  Will merge to 'next'?\n> >\n> > I think we've reached approximately \"good enough\" with this for the next\n> > steps, and can hopefully fix any remaining nits (such as my [1])\n> > post-merge.\n>\n> There is still an RFC in 02/19. Maybe we can get agreement that this\n> is OK and drop the RFC ?\n>\n> > Maybe hold off until Han-Wen gets a chance to ack it, and whether he's\n> > ok with the proposed fixup(s)?\n>\n> regarding uncompress2: I thought it was functionality best left to\n> zlib to implement, but I imagine git.git offers something similar.\n> Sounds good to use that.\n>\n> > Wasn't there an outstanding \"some of this in reftable/* should be static\n> > functions\" from someone, Carlo? (CC'd). In any case that sort of thing\n> > could also be a post-cleanup, I couldn't find a reference to that\n> > discussion in anything except my vague memory of it as I wrote this.\n>\n> I think I have addressed all outstanding issues in my github PR, and\n> I'll send it out once I see CI pass.\n\nIt was actually raised by Ramsay[1], and might have been a red herring,\nbut there was for sure no follow up.\n\nI have a few minor fixups that will post for discussion, and might be even\nok to merge to next without squashing if included on top, but I am not\nsure if all those issues could be addressed post-cleanup, since IMHO\nonce this gets into master, then it will be considered a stable API\nand it can't be changed.\n\nFor example. while looking at Ramsay message noticed the API for \"reader\"\nin libreftable might be better with s/init_reader/reader_init/g\n\nCarlo\n\n[1] https://lore.kernel.org/git/bc387e32-321e-4726-2a02-2e6cf6ed5250@ramsayjones.plus.com/\n\nCC +Ramsay\n"},{"id":"437510","messageId":"YVVyUkwYNfkEqNfU@coredump.intra.peff.net","threadId":"56595","inReplyTo":"20210929184339.GA19712@neerajsi-x1.localdomain","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-30T08:16:18Z","receivedAt":"2021-09-30T08:16:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 29, 2021 at 11:43:39AM -0700, Neeraj Singh wrote:\n\n> It seems to me that one problem is that the new \"primary\" objdir code doesn't\n> disable ref updates since the GIT_QUARANTINE_ENVIRONMENT setting isn't set.\n> If we fix that we should be forbidding ref updates.\n> \n> I've included a path that fixes my test case. This is on top of:\n> https://lore.kernel.org/git/CANQDOddqwVtWfC4eEP3fJB4sUiszGX8bLqoEVLcMf=v+jzx19g@mail.gmail.com/\n\nAh, right, I forgot we had that \"forbid ref updates in quarantine\" magic\n(despite being the one who added it).\n\nI do think this improves the safety, but things are still a bit more\ndangerous because we're handling it all in a single process, which sees\nboth the quarantine and non-quarantine states. I wrote more details in\nthis reply a few minutes ago:\n\n  https://lore.kernel.org/git/YVVmssXlaFM6yD5W@coredump.intra.peff.net/\n\n(sorry, it's long; search for the paragraph starting with \"Whereas doing\nit in-process\").\n\n> When creating a subprocess with a temporary ODB, we set the\n> GIT_QUARANTINE_ENVIRONMENT env var to tell child Git processes not\n> to update refs, since the tmp-objdir may go away.\n> \n> Introduce a similar mechanism for in-process temporary ODBs when\n> we call tmp_objdir_replace_primary_odb.\n> \n> Peff's test case was invoking ref updates via the cachetextconv\n> setting. That particular code silently does nothing when a ref\n> update is forbidden\n\nOh good. I was worried that it would generate ugly errors, rather than\nsilently skipping the update.\n\n> @@ -2126,7 +2146,7 @@ int ref_transaction_prepare(struct ref_transaction *transaction,\n>  \t\tbreak;\n>  \t}\n>  \n> -\tif (getenv(GIT_QUARANTINE_ENVIRONMENT)) {\n> +\tif (getenv(GIT_QUARANTINE_ENVIRONMENT) || !ref_updates_are_enabled()) {\n>  \t\tstrbuf_addstr(err,\n>  \t\t\t      _(\"ref updates forbidden inside quarantine environment\"));\n>  \t\treturn -1;\n\nI think the overall goal of this patch makes sense, but this\nconditional, along with tmp-objdir reaching out to the refs code, feels\nfunny. Should we perhaps have a single:\n\n  int tmp_objdir_is_primary(struct repository *r)\n  {\n\tif (the_tmp_objdir &&\n\t    !strcmp(the_tmp_objdir->path.buf, r->objects->odb->path))\n\t\treturn 1; /* our temporary is the primary */\n\n\tif (getenv(GIT_QUARANTINE_PATH))\n\t\treturn 1; /* our caller put us in quarantine */\n\n\treturn 0;\n  }\n\nThen it's all nicely abstracted and the ref code does not have to know\nthe details of GIT_QUARANTINE_PATH in the first place.\n\nIf you do got that route, the strcmp() might need to be a little more\ncareful about whether r->objects can be NULL (I didn't check).\nAlternatively, I kind of wonder if \"struct object_directory\" should just\nhave a flag that says \"is_temporary\".\n\n-Peff\n"},{"id":"437579","messageId":"xmqqo889ah6z.fsf@gitster.g","threadId":"56595","inReplyTo":"fd244131-84d4-7152-c88c-45f9ada6b609@gigacodes.de","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-30T21:26:12Z","receivedAt":"2021-09-30T21:26:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Fabian Stelzer <fs@gigacodes.de> writes:\n\n> On 28.09.21 02:52, Junio C Hamano wrote > * fs/ssh-signing (2021-09-10)\n> 9 commits\n>>  - ssh signing: test that gpg fails for unknown keys\n>>  - ssh signing: tests for logs, tags & push certs\n>>  - ssh signing: duplicate t7510 tests for commits\n>>  - ssh signing: verify signatures using ssh-keygen\n>>  - ssh signing: provide a textual signing_key_id\n>>  - ssh signing: retrieve a default key from ssh-agent\n>>  - ssh signing: add ssh key format and signing code\n>>  - ssh signing: add test prereqs\n>>  - ssh signing: preliminary refactoring and clean-up\n>> \n>>  Use ssh public crypto for object and push-cert signing.\n>> \n>>  On hold.\n>>  cf. <pull.1041.v8.git.git.1631304462.gitgitgadget@gmail.com>\n>>  cf. <532d97e7-8c91-df6a-6d90-70668256f513@gigacodes.de>\n>> \n>>\n> Openssh 8.8 has been released a few days ago and includes the needed fix\n> for the find-principal segfault.\n> I ran the full git testsuite against it without issues.\n>\n> Also, we (~30developers) have been running this patch with\n> openssh-portable (2d678c5e3bdc2f5c99f7af5122e9d054925d560d / post 8.7 -\n> pre 8.8) in our organization for the last 2 weeks without problems.\n> ...\n> But vverall i think this is ready for some broader usage/testing via next.\n>\n> I'd suggest to send the additional patches for valid-before/after\n> functionality in a new patchset for review after.\n\nOK, so with the new OpenSSH, there wasn't anything that contradicts\nwhat the above set of patches wanted to do and the series is good to\ngo?  Let's merge it to 'next' for wider audience, then.\n\nThanks.\n"},{"id":"437620","messageId":"CABPp-BH6ZzC9p94xda3SqfL0JjxoVAb3oV57a9cpg2ZDc=5NNA@mail.gmail.com","threadId":"56595","inReplyTo":"YVVyUkwYNfkEqNfU@coredump.intra.peff.net","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T07:50:14Z","receivedAt":"2021-10-01T07:50:30Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 30, 2021 at 1:16 AM Jeff King <peff@peff.net> wrote:\n>\n> On Wed, Sep 29, 2021 at 11:43:39AM -0700, Neeraj Singh wrote:\n>\n> > It seems to me that one problem is that the new \"primary\" objdir code doesn't\n> > disable ref updates since the GIT_QUARANTINE_ENVIRONMENT setting isn't set.\n> > If we fix that we should be forbidding ref updates.\n> >\n> > I've included a path that fixes my test case. This is on top of:\n> > https://lore.kernel.org/git/CANQDOddqwVtWfC4eEP3fJB4sUiszGX8bLqoEVLcMf=v+jzx19g@mail.gmail.com/\n>\n> Ah, right, I forgot we had that \"forbid ref updates in quarantine\" magic\n> (despite being the one who added it).\n>\n> I do think this improves the safety, but things are still a bit more\n> dangerous because we're handling it all in a single process, which sees\n> both the quarantine and non-quarantine states. I wrote more details in\n> this reply a few minutes ago:\n>\n>   https://lore.kernel.org/git/YVVmssXlaFM6yD5W@coredump.intra.peff.net/\n>\n> (sorry, it's long; search for the paragraph starting with \"Whereas doing\n> it in-process\").\n>\n> > When creating a subprocess with a temporary ODB, we set the\n> > GIT_QUARANTINE_ENVIRONMENT env var to tell child Git processes not\n> > to update refs, since the tmp-objdir may go away.\n> >\n> > Introduce a similar mechanism for in-process temporary ODBs when\n> > we call tmp_objdir_replace_primary_odb.\n> >\n> > Peff's test case was invoking ref updates via the cachetextconv\n> > setting. That particular code silently does nothing when a ref\n> > update is forbidden\n>\n> Oh good. I was worried that it would generate ugly errors, rather than\n> silently skipping the update.\n>\n> > @@ -2126,7 +2146,7 @@ int ref_transaction_prepare(struct ref_transaction *transaction,\n> >               break;\n> >       }\n> >\n> > -     if (getenv(GIT_QUARANTINE_ENVIRONMENT)) {\n> > +     if (getenv(GIT_QUARANTINE_ENVIRONMENT) || !ref_updates_are_enabled()) {\n> >               strbuf_addstr(err,\n> >                             _(\"ref updates forbidden inside quarantine environment\"));\n> >               return -1;\n>\n> I think the overall goal of this patch makes sense, but this\n> conditional, along with tmp-objdir reaching out to the refs code, feels\n> funny. Should we perhaps have a single:\n>\n>   int tmp_objdir_is_primary(struct repository *r)\n>   {\n>         if (the_tmp_objdir &&\n>             !strcmp(the_tmp_objdir->path.buf, r->objects->odb->path))\n>                 return 1; /* our temporary is the primary */\n>\n>         if (getenv(GIT_QUARANTINE_PATH))\n>                 return 1; /* our caller put us in quarantine */\n>\n>         return 0;\n>   }\n>\n> Then it's all nicely abstracted and the ref code does not have to know\n> the details of GIT_QUARANTINE_PATH in the first place.\n>\n> If you do got that route, the strcmp() might need to be a little more\n> careful about whether r->objects can be NULL (I didn't check).\n> Alternatively, I kind of wonder if \"struct object_directory\" should just\n> have a flag that says \"is_temporary\".\n\nActually, wouldn't this be the safest approach, for my particular\nusecase?  By having the quarantine in place, no refs will be updated,\nwhich removes the risk of new refs mentioning objects that will go\naway.  The only issue that could arise would be from new objects\nreferencing objects that will go away.  But the new objects are also\nwritten to the temporary object store when it remains the primary, and\nthus those new objects will also be deleted when the temporary object\nstore is.\n\nIn contrast, moving the temporary object store to the end of the\nalternates list would allow new objects to be written that reference\nthe objects that will go away.  Using pretend_object_file() will also\nallow new objects to be written that reference the objects that will\ngo away.\n\nSaid another way, I don't think anything should be writing a critical\nfile that needs to be durable when we're in the middle of a\n\"read-only\" process like git-log.  The only things written should be\ntemporary stuff, like the automatic remerge calculation from\nmerge-ort, the textconv cache optimization stuff, or perhaps future\ngitattributes transformation caching.  All that stuff can safely be\nblown away at the completion of each merge.\n"},{"id":"437711","messageId":"xmqq7dew8yq6.fsf@gitster.g","threadId":"56595","inReplyTo":"CABPp-BH6ZzC9p94xda3SqfL0JjxoVAb3oV57a9cpg2ZDc=5NNA@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-01T17:02:41Z","receivedAt":"2021-10-01T17:02:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren <newren@gmail.com> writes:\n\n> Said another way, I don't think anything should be writing a critical\n> file that needs to be durable when we're in the middle of a\n> \"read-only\" process like git-log.  The only things written should be\n> temporary stuff, like the automatic remerge calculation from\n> merge-ort, the textconv cache optimization stuff, or perhaps future\n> gitattributes transformation caching.  All that stuff can safely be\n> blown away at the completion of each merge.\n\nThe textconv cache can be populated/written during \"git log -p\" into\nthe object store to persist.  With or without \"--remerge-diff\", we\ncan make design decision to either \n\n - use temporary object store to discard everything we create at the\n   end in one-go, or\n\n - write them into the object store to let later gc to handle the\n   crufts.\n\nThe former will disable persistent write access to the cache.  It\nstill allows accesses the cached data during the same process,\nthough.  We so far deemed that textconv cache, when the user enables\nit, is valuable enough to make persistent.  Perhaps remerge-diff's\ntentative merge result may fall into the same category?  Some folks\nmay want to cache, others may not.\n\nIf we were to use the same notes-cache mechanism, we record the tree\nobject (perhaps the object name) as the cached value for the merge\ncommit in question.  Hopefully most of the merges are clean merges,\nand \"caching\" the results of the recreation of the merge would cost\nalmost nothing.  We need objects to record the fact that \"this merge\nhas cached result\" in the notes-cache, but the tree object that\nrepresents the cached result is already in the history the merge\nbelongs to.\n\n"},{"id":"437715","messageId":"CANQDOdcBEQQS4c22AiXhAB6sEABWP-kRhmq8Vdd6KG0GemSa9g@mail.gmail.com","threadId":"56595","inReplyTo":"xmqq7dew8yq6.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Neeraj Singh","fromEmail":"nksingh85@gmail.com","sentAt":"2021-10-01T17:39:29Z","receivedAt":"2021-10-01T17:39:46Z","isPatch":false,"sender":{"key":"nksingh85@gmail.com","avatar":null},"body":"On Fri, Oct 1, 2021 at 10:02 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Elijah Newren <newren@gmail.com> writes:\n>\n> > Said another way, I don't think anything should be writing a critical\n> > file that needs to be durable when we're in the middle of a\n> > \"read-only\" process like git-log.  The only things written should be\n> > temporary stuff, like the automatic remerge calculation from\n> > merge-ort, the textconv cache optimization stuff, or perhaps future\n> > gitattributes transformation caching.  All that stuff can safely be\n> > blown away at the completion of each merge.\n>\n> The textconv cache can be populated/written during \"git log -p\" into\n> the object store to persist.  With or without \"--remerge-diff\", we\n> can make design decision to either\n>\n>  - use temporary object store to discard everything we create at the\n>    end in one-go, or\n>\n>  - write them into the object store to let later gc to handle the\n>    crufts.\n>\n> The former will disable persistent write access to the cache.  It\n> still allows accesses the cached data during the same process,\n> though.  We so far deemed that textconv cache, when the user enables\n> it, is valuable enough to make persistent.  Perhaps remerge-diff's\n> tentative merge result may fall into the same category?  Some folks\n> may want to cache, others may not.\n>\n> If we were to use the same notes-cache mechanism, we record the tree\n> object (perhaps the object name) as the cached value for the merge\n> commit in question.  Hopefully most of the merges are clean merges,\n> and \"caching\" the results of the recreation of the merge would cost\n> almost nothing.  We need objects to record the fact that \"this merge\n> has cached result\" in the notes-cache, but the tree object that\n> represents the cached result is already in the history the merge\n> belongs to.\n>\n\nElijah,\nTo Junio's point, I'm also curious about why it's so important to\naggressively purge the\nmechanical merge cache that you want to do it after each diff\noperation rather than once\nat the end.\n"},{"id":"437716","messageId":"CABPp-BEhLQ3b8OCn3EbTa=g7uh_YXmkQuuGazn=OsT=bG3ot1w@mail.gmail.com","threadId":"56595","inReplyTo":"xmqq7dew8yq6.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T18:12:22Z","receivedAt":"2021-10-01T18:12:48Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 1, 2021 at 10:02 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Elijah Newren <newren@gmail.com> writes:\n>\n> > Said another way, I don't think anything should be writing a critical\n> > file that needs to be durable when we're in the middle of a\n> > \"read-only\" process like git-log.  The only things written should be\n> > temporary stuff, like the automatic remerge calculation from\n> > merge-ort, the textconv cache optimization stuff, or perhaps future\n> > gitattributes transformation caching.  All that stuff can safely be\n> > blown away at the completion of each merge.\n>\n> The textconv cache can be populated/written during \"git log -p\" into\n> the object store to persist.  With or without \"--remerge-diff\", we\n> can make design decision to either\n>\n>  - use temporary object store to discard everything we create at the\n>    end in one-go, or\n>\n>  - write them into the object store to let later gc to handle the\n>    crufts.\n>\n> The former will disable persistent write access to the cache.  It\n> still allows accesses the cached data during the same process,\n> though.  We so far deemed that textconv cache, when the user enables\n> it, is valuable enough to make persistent.  Perhaps remerge-diff's\n> tentative merge result may fall into the same category?  Some folks\n> may want to cache, others may not.\n>\n> If we were to use the same notes-cache mechanism, we record the tree\n> object (perhaps the object name) as the cached value for the merge\n> commit in question.  Hopefully most of the merges are clean merges,\n> and \"caching\" the results of the recreation of the merge would cost\n> almost nothing.  We need objects to record the fact that \"this merge\n> has cached result\" in the notes-cache, but the tree object that\n> represents the cached result is already in the history the merge\n> belongs to.\n\nI'm not sure this performance caching for remerge-diff makes any\nsense, for multiple reasons:\n\n1) Does the cache become invalidated when the merge algorithm changes\nor when config options change (--remerge-diff shows the difference\nbetween what an automatic merge would do, but an automatic merge's\nresults depend on the current algorithm and various merge.*, diff.*,\netc. config options the user can set).  Do we re-cache each variant?\nDo we expire the cache at some point?\n\n2) The performance aspect seems suspect at best, and likely to backfire.  Badly:\n\n2a) Let's say there's 100,000 merge commits in your history (so small\nhistory roughly the size of linux.git).  `git log --remerge-diff` thus\nrepeats 100,000 merges, creating loose objects for them.  Perhaps only\n1 in 10 merges needs any loose objects created (because the rest of\nthe merges were clean).  But those that do have conflicts will need to\ncreate both new blobs and any necessary new trees that use those new\nblobs.  Who knows how many blobs and trees are needed, but let's just\nuse 10 total blobs+trees per merge that has conflicts as a guesstimate\nof the average.  Thus `git log --remerge-diff` will need to create\n100,000 * 1/10 * 10 = 100,000 loose objects while it runs.  Preserving\nall those files to cache the results slows things down\nconsiderably...until they are gc'ed.  If we do the notes-cache thing,\nthen yes these objects would become packed.  Perhaps you're suggested\ndoing an automatic gc (or several) as part of running --remerge-diff,\nbut that's super slow itself.  When I implemented that, it felt worse\nto me than just letting loose objects pile up.\n\n2b) Note that my implementation didn't just clear out the tmp-objdir\nat the _end_ of the remerge-diff (which was the easiest to implement),\nbut cleared it out after each merge, because that was much faster with\nas many objects as can accumulate.\n\n2c) Is --remerge-diff slow enough to even merit the complicated effort\nof caching results?  --remerge-diff is faster than --cc in my testing,\nso why is it even worth the pain to cache?\n\n\n\nSo with that out of the way, let's return to discussing the textconv\ncache.  If the remerge-diff results aren't cached, isn't it unsafe to\nallow the textconv cache to persist anything while remerge-diff is\nrunning because it could create corruption?  We could still let the\ntextconv cache persist results when remerge-diff is not in use, but I\nthink we _want_ anything written by the textconv cache during\nremerge-diff to be thrown away because there's a risk it references a\ntemporary object created by remerge-diff that will be deleted.\n\nWith all that in mind, to me, it seems like using the tmp-objdir as\nthe primary object store in combination with the refs quarantine is\nactually the safest solution for this usecase.  Am I missing\nsomething?\n"},{"id":"437717","messageId":"CABPp-BFe+A0rC+n3DDYEPXw3HAjhC7wFU2KX4bw1LV3wO3a6WA@mail.gmail.com","threadId":"56595","inReplyTo":"CANQDOdcBEQQS4c22AiXhAB6sEABWP-kRhmq8Vdd6KG0GemSa9g@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T18:15:27Z","receivedAt":"2021-10-01T18:15:59Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 1, 2021 at 10:39 AM Neeraj Singh <nksingh85@gmail.com> wrote:\n>\n> On Fri, Oct 1, 2021 at 10:02 AM Junio C Hamano <gitster@pobox.com> wrote:\n> >\n> > Elijah Newren <newren@gmail.com> writes:\n> >\n> > > Said another way, I don't think anything should be writing a critical\n> > > file that needs to be durable when we're in the middle of a\n> > > \"read-only\" process like git-log.  The only things written should be\n> > > temporary stuff, like the automatic remerge calculation from\n> > > merge-ort, the textconv cache optimization stuff, or perhaps future\n> > > gitattributes transformation caching.  All that stuff can safely be\n> > > blown away at the completion of each merge.\n> >\n> > The textconv cache can be populated/written during \"git log -p\" into\n> > the object store to persist.  With or without \"--remerge-diff\", we\n> > can make design decision to either\n> >\n> >  - use temporary object store to discard everything we create at the\n> >    end in one-go, or\n> >\n> >  - write them into the object store to let later gc to handle the\n> >    crufts.\n> >\n> > The former will disable persistent write access to the cache.  It\n> > still allows accesses the cached data during the same process,\n> > though.  We so far deemed that textconv cache, when the user enables\n> > it, is valuable enough to make persistent.  Perhaps remerge-diff's\n> > tentative merge result may fall into the same category?  Some folks\n> > may want to cache, others may not.\n> >\n> > If we were to use the same notes-cache mechanism, we record the tree\n> > object (perhaps the object name) as the cached value for the merge\n> > commit in question.  Hopefully most of the merges are clean merges,\n> > and \"caching\" the results of the recreation of the merge would cost\n> > almost nothing.  We need objects to record the fact that \"this merge\n> > has cached result\" in the notes-cache, but the tree object that\n> > represents the cached result is already in the history the merge\n> > belongs to.\n> >\n>\n> Elijah,\n> To Junio's point, I'm also curious about why it's so important to\n> aggressively purge the\n> mechanical merge cache that you want to do it after each diff\n> operation rather than once\n> at the end.\n\nPerformance; I don't want to accumulate hundreds of thousands of loose\nobjects and pay the penalty from them sitting around, especially when\nI know they're cruft.  (My original implementation from a year ago did\njust clean out once at the end.)\n\nFor more details, see\nhttps://lore.kernel.org/git/CABPp-BGsjq3ts4A6wKLYcopD9rknM+LXXi8qR_SLEpmU+x7KNQ@mail.gmail.com/,\nstarting with \"It'd be _much_ slower\".  Or the email I just sent to\nJunio.\n"},{"id":"437738","messageId":"xmqqr1d42ykj.fsf@gitster.g","threadId":"56595","inReplyTo":"CABPp-BEhLQ3b8OCn3EbTa=g7uh_YXmkQuuGazn=OsT=bG3ot1w@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-01T22:02:36Z","receivedAt":"2021-10-01T22:02:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren <newren@gmail.com> writes:\n\n> So with that out of the way, let's return to discussing the textconv\n> cache.  If the remerge-diff results aren't cached, isn't it unsafe to\n> allow the textconv cache to persist anything while remerge-diff is\n> running because it could create corruption?\n\nI do not think anybody involved in this thread thinks it is\npractical to annotate each write_object_file() with \"this is\ntemporary\" vs \"this is to persist\", so it is a given that it would\nbe all-or-none.  If we want write_object_file() called while we are\nrunning remerge-diff to write to a temporary object store, we have\nto accept any other write_object_file() called by somebody else,\nlike textconv cache, must become temporary.\n\nIt may be sufficient to plug ref updates (which would cover the\nfinialization of notes-cache used by the textconv cache) to avoid\ncorruption, but that might give us a pointless and unpleasant error\nmessage, so it may be necessary to teach the notes-cache stuff to\nallow getting existing cached data while disabling it to accept\ncache updates.\n\nThanks.\n\n"},{"id":"437746","messageId":"CABPp-BG9JNeFJ_gzepntcieJAMzT1OKNes9zEOiWY3JMXh4BMg@mail.gmail.com","threadId":"56595","inReplyTo":"xmqqr1d42ykj.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T23:05:38Z","receivedAt":"2021-10-01T23:05:53Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 1, 2021 at 3:02 PM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Elijah Newren <newren@gmail.com> writes:\n>\n> > So with that out of the way, let's return to discussing the textconv\n> > cache.  If the remerge-diff results aren't cached, isn't it unsafe to\n> > allow the textconv cache to persist anything while remerge-diff is\n> > running because it could create corruption?\n>\n> I do not think anybody involved in this thread thinks it is\n> practical to annotate each write_object_file() with \"this is\n> temporary\" vs \"this is to persist\", so it is a given that it would\n> be all-or-none.  If we want write_object_file() called while we are\n> running remerge-diff to write to a temporary object store, we have\n> to accept any other write_object_file() called by somebody else,\n> like textconv cache, must become temporary.\n\nOk, good.\n\nBut is this write_object_file() specific?  I think\npretend_object_file() has the same problem where the textconv cache\ncould reference a pretend_object_file() and thus write objects and/or\nrefs that become corrupt.  (In particular, if the userdiff fires on\nany of the new files created by the three-way content merges, then I\nthink we hit the exact same problem)  So, I think as long as there are\nany \"temporary\" objects being fed to diff, the textconv cache needs to\nalso be considered temporary.  Or am I misunderstanding something?\n\n> It may be sufficient to plug ref updates (which would cover the\n> finialization of notes-cache used by the textconv cache) to avoid\n> corruption, but that might give us a pointless and unpleasant error\n> message, so it may be necessary to teach the notes-cache stuff to\n> allow getting existing cached data while disabling it to accept\n> cache updates.\n\nAll we need to do here is set the quarantine environment, as per\nNeeraj's report[1], it already handles all of this:\n\n\"\"\"\nPeff's test case was invoking ref updates via the cachetextconv\nsetting. That particular code silently does nothing when a ref\nupdate is forbidden\n\"\"\"\n\nLooking at the code, this appears to happen because\nref_transaction_prepare() doesn't print an error in the quarantine\nenvironment, but returns one.  That error is passed all the way up the\nstack to notes_cache_write() and then to fill_textconv().\nfill_textconv() is the first level in the stack that ignores the\nreturn code of what it's calling, namely notes_cache_write(), which\nmeans that things work fine.  There's even a comment a few lines\nstating, \"ignore errors, as we might be in a readonly repository\".\nSo, I think we're good as long as we ensure the quarantine environment\nis setup.\n\n[1] https://lore.kernel.org/git/20210929184339.GA19712@neerajsi-x1.localdomain/\n"},{"id":"437882","messageId":"CABPp-BE4jXyPVAWo7h7fuVHDuatr+mw6CgcWQ1rF7oT0RPMqXg@mail.gmail.com","threadId":"56595","inReplyTo":"YVOn3hDsb5pnxR53@coredump.intra.peff.net","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-04T13:45:00Z","receivedAt":"2021-10-04T13:54:52Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 28, 2021 at 4:40 PM Jeff King <peff@peff.net> wrote:\n>\n> I just left a rather long-ish mail in the thread, but the gist of it is\n> that I'm worried that there's some possibility of corruption there if\n> the diff code writes objects. I didn't do a proof-of-concept there, but\n> I worked one up just now. Try this:\n\nThanks for the testcase; it's very helpful.\n\nLet's make it a little more interesting, though...\n\n>\n>   # make a repo\n>   git init repo\n>   cd repo\n>   echo base >file\n>   git add file\n>   git commit -m base\n>\n>   # two diverging branches\n>   echo main >file\n>   git commit -am main\n>   git checkout -b side HEAD^\n>   echo side >file\n>   git commit -am side\n>\n>   # we get a conflict, and we resolve\n>   git merge main\n>   echo resolved >file\n>   git commit -am merged\n>\n>   # now imagine we had a file with a diff driver. I stuffed it\n>   # in here after the fact, but it could have been here all along,\n>   # or come as part of the merge, etc.\n>   echo whatever >unrelated\n>   echo \"unrelated diff=foo\" >.gitattributes\n\nIf we change this line to:\n\n  echo \"file diff=foo\" >.gitattributes\n\nThen the userdiff will fire on the file involved in the remerge-diff,\nincluding on the file included in the automatic 3-way content\nconflict, giving us a diff that looks like this:\n\ndiff --git a/file b/file\nindex 5c66bc5..2ab19ae 100644\n--- a/file\n+++ b/file\n@@ -1,7 +1 @@\n-<<<<<<< 7D4147C (SIDE)\n-SIDE\n-||||||| BE54C04\n-BASE\n-=======\n-MAIN\n->>>>>>> 03FC4E3 (MAIN)\n+RESOLVED\n\n>   git add .\n>   git commit --amend --no-edit\n>\n>   # set up the diff driver not just to do a textconv, but to cache the\n>   # result. This will require writing out new objects for the cache\n>   # as part of the diff operation.\n>   git config diff.foo.textconv \"$PWD/upcase\"\n>   git config diff.foo.cachetextconv true\n>   cat >upcase <<\\EOF &&\n>   #!/bin/sh\n>   tr a-z A-Z <$1\n>   EOF\n>   chmod +x upcase\n>\n>   # now show the diff\n>   git log -1 --remerge-diff\n>\n>   # and make sure the repo is still OK\n>   git fsck\n>\n> The remerge diff will correctly show the textconv'd content (because\n> it's not in either parent, and hence an evil merge):\n>\n>   diff --git a/unrelated b/unrelated\n>   new file mode 100644\n>   index 0000000..982793c\n>   --- /dev/null\n>   +++ b/unrelated\n>   @@ -0,0 +1 @@\n>   +WHATEVER\n>\n> but then fsck shows that our cache is corrupted:\n>\n>   Checking object directories: 100% (256/256), done.\n>   error: refs/notes/textconv/foo: invalid sha1 pointer 1e9b3ecffca73411001043fe914a7ec0e7291043\n>   error: refs/notes/textconv/foo: invalid reflog entry 1e9b3ecffca73411001043fe914a7ec0e7291043\n>   dangling tree 569b6e414203d2f50bdaafc1585eae11e28afc37\n\n\n> Now I'll admit the textconv-cache is a pretty seldom-used feature. And\n> that there _probably_ aren't a lot of other ways that the diff code\n> would try to write objects or references. But I think it speaks to the\n> kind of subtle interactions we should worry about (and certainly\n> corrupting the repository is a pretty bad outcome, though at least in\n> this case it's \"just\" a cache and we could blow away that ref manually).\n\n\nI implemented the pretend_object_file() solution on Saturday and tried\nthis out, and found with the above change that it has corruption\nproblems as well.  This really doesn't feel much safer too me.\n\nThe move-the-tmp-objdir-to-just-being-an-alternate-instead-of-a-primary\nsolution, as suggested at\nhttps://lore.kernel.org/git/YVOiggCWAdZcxAb6@coredump.intra.peff.net/\nhas the same problem.\n\nBut, more importantly, neither of these solutions can be made safe.\nEven if we adopt the GIT_QUARANTINE_PATH suggestion from Neeraj\n(https://lore.kernel.org/git/20210929184339.GA19712@neerajsi-x1.localdomain/),\nwe still have objects referencing now-missing objects.\n\n*However*, if we use the tmp-objdir-as-primary solution, and leave it\nas a primary, and use your modification to the GIT_QUARANTINE_PATH\nidea from Neeraj (i.e. create a \"the-tmp-objdir\" and have the refs\ncode check for it and turn on the quarantine automatically when one\nexists), then we prevent adding or modifying refs to point to bad\nobjects, and any new objects that textconv might write that could\ndepend on transient objects, will themselves be transient and go away.\nSo, this solution is safe from corruption.  I'll be implementing it.\n\n(not sure how quickly I'll get it done, as my time is mostly limited\nto early mornings or evenings or Saturdays right now...)\n"},{"id":"438241","messageId":"xmqq4k9spk8p.fsf@gitster.g","threadId":"56595","inReplyTo":"CANQDOddZ-KYTB4q0nYNDinis8aKktm6Ek6F+mJouTV-yRtTpUw@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-07T22:01:42Z","receivedAt":"2021-10-07T22:01:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Neeraj Singh <nksingh85@gmail.com> writes:\n\n> Elijah,\n>\n> Here's a branch of your changes based on the amalgamated tmp-objdir code:\n> https://github.com/neerajsi-msft/git/commits/neerajsi/remerge-diff\n>\n> This commit adapts your code to use the amalgamated API:\n> https://github.com/neerajsi-msft/git/commit/725328fe1d8be8326d2ddef78e164ca21450b100\n\nIt seems that the discussion petered out at this point.\n\nRight now I have a version of ns/remerge-diff before this adjustment\nin 'seen', and Neeraj's latest version is kept out of 'seen' as they\ndo not play well together without an adjustment like that.\n\nWhat's the good way forward?  I do not deeply care which one goes\nfirst, but I have a feeling that the need by remerge-diff that wants\nto discard temporary objects would involve more work to make it safe\nthan the need by batched fsync where newly created objects will not\nbe discarded but merely moved to the primary store before the end of\nthe operation, so from that point of view, it seems simpler and\nsafer to queue ns/batched-fsync topic first (especially given that\nit is a no-op until the end-user opts into the experiment), and have\na remerge-diff that uses the infrastructure from Neeraj's topic.\n\nWhat's your take on the rebase Neeraj made, Elijah (at the URL\nabove)?\n\nThanks.\n\n\n\n\n"},{"id":"438275","messageId":"CABPp-BE=7+XifO3mAzNB90Y0vatvhwk7wwM4ptBrg79BhiFqSg@mail.gmail.com","threadId":"56595","inReplyTo":"xmqq4k9spk8p.fsf@gitster.g","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-08T06:51:57Z","receivedAt":"2021-10-08T06:52:14Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Oct 7, 2021 at 3:01 PM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Neeraj Singh <nksingh85@gmail.com> writes:\n>\n> > Elijah,\n> >\n> > Here's a branch of your changes based on the amalgamated tmp-objdir code:\n> > https://github.com/neerajsi-msft/git/commits/neerajsi/remerge-diff\n> >\n> > This commit adapts your code to use the amalgamated API:\n> > https://github.com/neerajsi-msft/git/commit/725328fe1d8be8326d2ddef78e164ca21450b100\n>\n> It seems that the discussion petered out at this point.\n>\n> Right now I have a version of ns/remerge-diff before this adjustment\n> in 'seen', and Neeraj's latest version is kept out of 'seen' as they\n> do not play well together without an adjustment like that.\n>\n> What's the good way forward?  I do not deeply care which one goes\n> first, but I have a feeling that the need by remerge-diff that wants\n> to discard temporary objects would involve more work to make it safe\n> than the need by batched fsync where newly created objects will not\n> be discarded but merely moved to the primary store before the end of\n> the operation, so from that point of view, it seems simpler and\n> safer to queue ns/batched-fsync topic first (especially given that\n> it is a no-op until the end-user opts into the experiment), and have\n> a remerge-diff that uses the infrastructure from Neeraj's topic.\n>\n> What's your take on the rebase Neeraj made, Elijah (at the URL\n> above)?\n\nI meant to dig further, but nearly all my git time in the last week\nand a half was attempting to keep up with other patch reviews.  My git\ntime is fast disappearing in the near term, and it's not clear how\nmuch, if any, time I'll have to work on patches (or even continued\nreviewing) before, say, mid-November.  I most likely won't be able to\ndo any discussion-prep work in advance of the Git Contributor's\nSummit, and might not even be able to attend anymore.\n\nI had looked over Neeraj's patches and they looked reasonable.  I\nthought there might be some tweaks that I could try out, but at this\npoint, just take what he has and keep my topic as expecting an update.\nI'll circle back eventually.\n\nSorry for the delay.\n"},{"id":"438376","messageId":"CANQDOdeSjKsGUYacGwx-fnigqiocbyDOfz2O8im6-7txDCuZzQ@mail.gmail.com","threadId":"56595","inReplyTo":"CABPp-BE=7+XifO3mAzNB90Y0vatvhwk7wwM4ptBrg79BhiFqSg@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Neeraj Singh","fromEmail":"nksingh85@gmail.com","sentAt":"2021-10-08T22:30:22Z","receivedAt":"2021-10-08T22:30:37Z","isPatch":false,"sender":{"key":"nksingh85@gmail.com","avatar":null},"body":"Thanks Junio.  I'd be happy to have my change go in first.  I'm hoping\nthat we'll be able to get a lot of mileage with batch mode in Git For\nWindows early on.\n\nThanks,\nNeeraj\n\nOn Thu, Oct 7, 2021 at 11:52 PM Elijah Newren <newren@gmail.com> wrote:\n>\n> On Thu, Oct 7, 2021 at 3:01 PM Junio C Hamano <gitster@pobox.com> wrote:\n> >\n> > Neeraj Singh <nksingh85@gmail.com> writes:\n> >\n> > > Elijah,\n> > >\n> > > Here's a branch of your changes based on the amalgamated tmp-objdir code:\n> > > https://github.com/neerajsi-msft/git/commits/neerajsi/remerge-diff\n> > >\n> > > This commit adapts your code to use the amalgamated API:\n> > > https://github.com/neerajsi-msft/git/commit/725328fe1d8be8326d2ddef78e164ca21450b100\n> >\n> > It seems that the discussion petered out at this point.\n> >\n> > Right now I have a version of ns/remerge-diff before this adjustment\n> > in 'seen', and Neeraj's latest version is kept out of 'seen' as they\n> > do not play well together without an adjustment like that.\n> >\n> > What's the good way forward?  I do not deeply care which one goes\n> > first, but I have a feeling that the need by remerge-diff that wants\n> > to discard temporary objects would involve more work to make it safe\n> > than the need by batched fsync where newly created objects will not\n> > be discarded but merely moved to the primary store before the end of\n> > the operation, so from that point of view, it seems simpler and\n> > safer to queue ns/batched-fsync topic first (especially given that\n> > it is a no-op until the end-user opts into the experiment), and have\n> > a remerge-diff that uses the infrastructure from Neeraj's topic.\n> >\n> > What's your take on the rebase Neeraj made, Elijah (at the URL\n> > above)?\n>\n> I meant to dig further, but nearly all my git time in the last week\n> and a half was attempting to keep up with other patch reviews.  My git\n> time is fast disappearing in the near term, and it's not clear how\n> much, if any, time I'll have to work on patches (or even continued\n> reviewing) before, say, mid-November.  I most likely won't be able to\n> do any discussion-prep work in advance of the Git Contributor's\n> Summit, and might not even be able to attend anymore.\n>\n> I had looked over Neeraj's patches and they looked reasonable.  I\n> thought there might be some tweaks that I could try out, but at this\n> point, just take what he has and keep my topic as expecting an update.\n> I'll circle back eventually.\n>\n> Sorry for the delay.\n"},{"id":"438381","messageId":"xmqq4k9ri0iv.fsf@gitster.g","threadId":"56595","inReplyTo":"CABPp-BE=7+XifO3mAzNB90Y0vatvhwk7wwM4ptBrg79BhiFqSg@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2021, #08; Mon, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-08T23:01:44Z","receivedAt":"2021-10-08T23:01:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren <newren@gmail.com> writes:\n\n> I had looked over Neeraj's patches and they looked reasonable.  I\n> thought there might be some tweaks that I could try out, but at this\n> point, just take what he has and keep my topic as expecting an update.\n> I'll circle back eventually.\n\nThanks.\n\nI've fetched Neeraj's remerge-diff rebase, and structured three\ntopics: \n\n . 'ns/tmp-objdir' topic is to hold the bottommost two tmp-objdir\n   patches.\n\n . 'ns/batched-fsync' topic forks from 'master', merges\n   'ns/tmp-objdir', and then applies its own patches.\n\n . 'ns/remerge-diff' topic forks from 'master', merges\n   'ns/tmp-objdir', and then applies its own patches.\n\n"}]}