{"thread":{"id":"44178","subject":"What's cooking in git.git (Sep 2016, #08; Tue, 27)","startedAt":"2016-09-27T23:23:21Z","lastAt":"2016-10-04T22:36:54Z","messageCount":11,"participants":["Junio C Hamano","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"302784","messageId":"xmqq8tudkjvn.fsf@gitster.mtv.corp.google.com","threadId":"44178","inReplyTo":null,"subject":"What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-27T23:23:08Z","receivedAt":"2016-09-27T23:23:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Here are the topics that have been cooking.  Commits prefixed with\n'-' are only in 'pu' (proposed updates) while commits prefixed with\n'+' are in 'next'.  The ones marked with '.' do not appear in any of\nthe integration branches, but I am still holding onto them.\n\nYou can find the changes described here in the integration branches\nof the repositories listed at\n\n    http://git-blame.blogspot.com/p/git-public-repositories.html\n\n--------------------------------------------------\n[Graduated to \"master\"]\n\n* mh/diff-indent-heuristic (2016-09-19) 8 commits\n  (merged to 'next' on 2016-09-22 at e71d742)\n + blame: honor the diff heuristic options and config\n + parse-options: add parse_opt_unknown_cb()\n + diff: improve positioning of add/delete blocks in diffs\n + xdl_change_compact(): introduce the concept of a change group\n + recs_match(): take two xrecord_t pointers as arguments\n + is_blank_line(): take a single xrecord_t as argument\n + xdl_change_compact(): only use heuristic if group can't be matched\n + xdl_change_compact(): fix compaction heuristic to adjust ixo\n\n Output from \"git diff\" can be made easier to read by selecting\n which lines are common and which lines are added/deleted\n intelligently when the lines before and after the changed section\n are the same.  A command line option is added to help with the\n experiment to find a good heuristics.\n * mh/diff-indent-heuristic (2016-09-27) 1 commit\n - xdiff: rename \"struct group\" to \"struct xdlgroup\"\n\n Clean-up for a recently graduated topic.\n\n\n* rs/cocci (2016-09-15) 3 commits\n  (merged to 'next' on 2016-09-22 at aa54fa4)\n + use strbuf_addstr() for adding constant strings to a strbuf, part 2\n + add coccicheck make target\n + contrib/coccinelle: fix semantic patch for oid_to_hex_r()\n\n Code cleanup.\n\n\n* ep/doc-check-ref-format-example (2016-09-21) 1 commit\n  (merged to 'next' on 2016-09-22 at 6d0d79e)\n + git-check-ref-format.txt: fixup documentation\n\n A shell script example in check-ref-format documentation has been\n fixed.\n\n\n* jk/clone-recursive-progress (2016-09-22) 1 commit\n  (merged to 'next' on 2016-09-22 at 8310c42)\n + clone: pass --progress decision to recursive submodules\n\n \"git clone --recurse-submodules\" lost the progress eye-candy in\n recent update, which has been corrected.\n\n\n* jk/doc-cvs-update (2016-09-22) 3 commits\n  (merged to 'next' on 2016-09-22 at c0f949f)\n + docs/cvs-migration: mention cvsimport caveats\n + docs/cvs-migration: update link to cvsps homepage\n + docs/cvsimport: prefer cvs-fast-export to parsecvs\n\n Documentation around tools to import from CVS was fairly outdated.\n\n\n* js/regexec-buf (2016-09-21) 3 commits\n  (merged to 'next' on 2016-09-22 at 2ee2477)\n + regex: use regexec_buf()\n + regex: add regexec_buf() that can work on a non NUL-terminated string\n + regex: -G<pattern> feeds a non NUL-terminated string to regexec() and fails\n\n Some codepaths in \"git diff\" used regexec(3) on a buffer that was\n mmap(2)ed, which may not have a terminating NUL, leading to a read\n beyond the end of the mapped region.  This was fixed by introducing\n a regexec_buf() helper that takes a <ptr,len> pair with REG_STARTEND\n extension.\n\n\n* jt/format-patch-rfc (2016-09-21) 1 commit\n  (merged to 'next' on 2016-09-22 at 3b39442)\n + format-patch: add \"--rfc\" for the common case of [RFC PATCH]\n\n In some projects, it is common to use \"[RFC PATCH]\" as the subject\n prefix for a patch meant for discussion rather than application.  A\n new option \"--rfc\" was a short-hand for \"--subject-prefix=RFC PATCH\"\n to help the participants of such projects.\n\n\n* ls/travis-homebrew-path-fix (2016-09-22) 1 commit\n  (merged to 'next' on 2016-09-22 at 310e620)\n + travis-ci: ask homebrew for its path instead of hardcoding it\n\n The procedure to build Git on Mac OS X for Travis CI hardcoded the\n internal directory structure we assumed HomeBrew uses, which was a\n no-no.  The procedure has been updated to ask HomeBrew things we\n need to know to fix this.\n\n\n* mm/config-color-ui-default-to-auto (2016-09-16) 1 commit\n  (merged to 'next' on 2016-09-22 at 4eac0cb)\n + Documentation/config: default for color.* is color.ui\n\n Documentation for individual configuration variables to control use\n of color (like `color.grep`) said that their default value is\n 'false', instead of saying their default is taken from `color.ui`.\n When we updated the default value for color.ui from 'false' to\n 'auto' quite a while ago, all of them broke.  This has been\n corrected.\n\n\n* nd/checkout-disambiguation (2016-09-21) 3 commits\n  (merged to 'next' on 2016-09-22 at ebfa365)\n + checkout: fix ambiguity check in subdir\n + checkout.txt: document a common case that ignores ambiguation rules\n + checkout: add some spaces between code and comment\n\n \"git checkout <word>\" does not follow the usual disambiguation\n rules when the <word> can be both a rev and a path, to allow\n checking out a branch 'foo' in a project that happens to have a\n file 'foo' in the working tree without having to disambiguate.\n This was poorly documented and the check was incorrect when the\n command was run from a subdirectory.\n\n\n* rs/c-auto-resets-attributes (2016-09-19) 1 commit\n  (merged to 'next' on 2016-09-22 at 68f2e4a)\n + pretty: let %C(auto) reset all attributes\n\n The pretty-format specifier \"%C(auto)\" used by the \"log\" family of\n commands to enable coloring of the output is taught to also issue a\n color-reset sequence to the output.\n\n\n* rs/checkout-init-macro (2016-09-22) 1 commit\n  (merged to 'next' on 2016-09-22 at 6513755)\n + introduce CHECKOUT_INIT\n\n Code cleanup.\n\n\n* rt/rebase-i-broken-insn-advise (2016-09-07) 1 commit\n  (merged to 'next' on 2016-09-23 at 0d12484)\n + rebase -i: improve advice on bad instruction lines\n\n When \"git rebase -i\" is given a broken instruction, it told the\n user to fix it with \"--edit-todo\", but didn't say what the step\n after that was (i.e. \"--continue\").\n\n\n* tg/add-chmod+x-fix (2016-09-21) 6 commits\n  (merged to 'next' on 2016-09-22 at 6afdd21)\n + t3700-add: do not check working tree file mode without POSIXPERM\n + t3700-add: create subdirectory gently\n + add: modify already added files when --chmod is given\n + read-cache: introduce chmod_index_entry\n + update-index: add test for chmod flags\n + Merge branch 'ib/t3700-add-chmod-x-updates' into tg/add-chmod+x-fix\n\n \"git add --chmod=+x <pathspec>\" added recently only toggled the\n executable bit for paths that are either new or modified. This has\n been corrected to flip the executable bit for all paths that match\n the given pathspec.\n\n\n* va/i18n-more (2016-09-21) 6 commits\n  (merged to 'next' on 2016-09-22 at bea26e8)\n + i18n: stash: mark messages for translation\n + i18n: notes-merge: mark die messages for translation\n + i18n: ident: mark hint for translation\n + i18n: i18n: diff: mark die messages for translation\n + i18n: connect: mark die messages for translation\n + i18n: commit: mark message for translation\n\n Even more i18n.\n\n--------------------------------------------------\n[New Topics]\n\n* mh/diff-indent-heuristic (2016-09-27) 1 commit\n  (merged to 'next' on 2016-09-27 at 3d6fb66)\n + xdiff: rename \"struct group\" to \"struct xdlgroup\"\n\n Clean-up for a recently graduated topic.\n\n Will merge to 'master'.\n\n\n* rs/cocci (2016-09-27) 3 commits\n - use strbuf_add_unique_abbrev() for adding short hashes, part 2\n - use strbuf_addstr() instead of strbuf_addf() with \"%s\", part 2\n - gitignore: ignore output files of coccicheck make target\n\n Code clean-up with help from coccinelle tool continues.\n\n Will merge to 'next'.\n\n\n* jc/verify-loose-object-header (2016-09-26) 2 commits\n  (merged to 'next' on 2016-09-27 at 2947f95)\n + unpack_sha1_header(): detect malformed object header\n + streaming: make sure to notice corrupt object\n\n Codepaths that read from an on-disk loose object were too loose in\n validating what they are reading is a proper object file and\n sometimes read past the data they read from the disk, which has\n been corrected.  H/t to Gustavo Grieco for reporting.\n\n Will merge to 'master'.\n\n\n* jk/ambiguous-short-object-names (2016-09-27) 11 commits\n - get_short_sha1: make default disambiguation configurable\n - get_short_sha1: list ambiguous objects on error\n - for_each_abbrev: drop duplicate objects\n - sha1_array: let callbacks interrupt iteration\n - get_short_sha1: mark ambiguity error for translation\n - get_short_sha1: NUL-terminate hex prefix\n - get_short_sha1: refactor init of disambiguation code\n - get_short_sha1: parse tags when looking for treeish\n - get_sha1: propagate flags to child functions\n - get_sha1: avoid repeating ourselves via ONLY_TO_DIE\n - get_sha1: detect buggy calls with multiple disambiguators\n\n When given an abbreviated object name that is not (or more\n realistically, \"no longer\") unique, we gave a fatal error\n \"ambiguous argument\".  This error is now accompanied by hints that\n lists the objects that begins with the given prefix.  During the\n course of development of this new feature, numerous minor bugs were\n uncovered and corrected, the most notable one of which is that we\n gave \"short SHA1 xxxx is ambiguous.\" twice without good reason.\n\n Will merge to 'next'.\n\n\n* rs/copy-array (2016-09-25) 2 commits\n  (merged to 'next' on 2016-09-27 at c92e020)\n + use COPY_ARRAY\n + add COPY_ARRAY\n\n Code cleanup.\n\n Will merge to 'master'.\n\n\n* rs/git-gui-use-modern-git-merge-syntax (2016-09-26) 2 commits\n  (merged to 'next' on 2016-09-27 at f55850d)\n + Merge branch 'rs/use-modern-git-merge-syntax' of git-gui into rs/git-gui-use-modern-git-merge-syntax\n + git-gui: stop using deprecated merge syntax\n\n The original command line syntax for \"git merge\", which was \"git\n merge <msg> HEAD <parent>...\", has been deprecated for quite some\n time, and \"git gui\" was the last in-tree user of the syntax.  This\n is finally fixed, so that we can move forward with the deprecation.\n\n Will merge to 'master'.\n\n\n* va/git-gui-i18n (2016-09-26) 3 commits\n  (merged to 'next' on 2016-09-27 at ab0f66f)\n + Merge branch 'va/i18n' of ../git-gui into va/git-gui-i18n\n + git-gui: l10n: add Portuguese translation\n + git-gui i18n: mark strings for translation\n\n \"git gui\" l10n to Portuguese.\n\n Will merge to 'master'.\n\n\n* va/i18n-perl-scripts (2016-09-25) 11 commits\n - i18n: difftool: mark warnings for translation\n - i18n: send-email: mark string with interpolation for translation\n - i18n: send-email: mark warnings and errors for translation\n - i18n: send-email: mark strings for translation\n - i18n: add--interactive: mark edit_hunk_manually message for translation\n - i18n: add--interactive: i18n of help_patch_cmd\n - i18n: add--interactive: mark message for translation\n - i18n: add--interactive: mark plural strings\n - i18n: add--interactive: mark strings with interpolation for translation\n - i18n: add--interactive: mark simple here documents for translation\n - i18n: add--interactive: mark strings for translation\n\n Porcelain scripts written in Perl are getting internationalized.\n\n Waiting for a reroll.\n cf. <1474913721.1035.9.camel@sapo.pt>\n\n\n* vn/revision-shorthand-for-side-branch-log (2016-09-27) 1 commit\n - revision: new rev^-n shorthand for rev^n..rev\n\n \"git log rev^..rev\" is an often-used revision range specification\n to show what was done on a side branch merged at rev, which has\n gained a short-hand \"rev^-1\".  In general \"rev^-$n\" is the same as\n \"^rev^$n rev\", i.e. what was done while the history leading to nth\n parent was looking the other way.\n\n Will merge to 'next'.\n\n\n* dt/mailinfo (2016-09-26) 1 commit\n  (merged to 'next' on 2016-09-27 at 59e95db)\n + add David Turner's Two Sigma address\n\n Will merge to 'master'.\n\n\n* dt/tree-fsck (2016-09-27) 2 commits\n - fsck: handle bad trees like other errors\n - tree-walk: be more specific about corrupt tree errors\n\n The codepath in \"git fsck\" to detect malformed tree objects has\n been updated not to die but keep going after detecting them.\n\n Will merge to 'next'.\n\n\n* jc/latin-1 (2016-09-26) 2 commits\n - utf8: accept \"latin-1\" as ISO-8859-1\n - utf8: refactor code to decide fallback encoding\n\n Some platforms no longer understand \"latin-1\" that is still seen in\n the wild in e-mail headers; replace them with \"iso-8859-1\" that is\n more widely known when conversion fails from/to it.\n\n Will merge to 'next'.\n\n\n* jc/worktree-config (2016-09-27) 1 commit\n - worktree: honor configuration variables\n\n \"git worktree\", even though it used the default_abbrev setting that\n ought to be affected by core.abbrev configuration variable, ignored\n the variable setting.  The command has been taught to read the\n default set of configuration variables to correct this.\n\n Will merge to 'next'.\n\n\n* mg/gpg-richer-status (2016-09-27) 1 commit\n - gpg-interface: use more status letters\n\n The GPG verification status shown in \"%G?\" pretty format specifier\n was not rich enough to differentiate a signature made by an expired\n key, a signature made by a revoked key, etc.  New output letters\n have been assigned to express them.\n\n Will merge to 'next'.\n\n\n* pb/rev-list-reverse-with-count (2016-09-27) 1 commit\n - rev-list-options: clarify the usage of --reverse\n\n Doc update.\n\n Will merge to 'next'.\n\n--------------------------------------------------\n[Stalled]\n\n* jc/bundle (2016-03-03) 6 commits\n - index-pack: --clone-bundle option\n - Merge branch 'jc/index-pack' into jc/bundle\n - bundle v3: the beginning\n - bundle: keep a copy of bundle file name in the in-core bundle header\n - bundle: plug resource leak\n - bundle doc: 'verify' is not about verifying the bundle\n\n The beginning of \"split bundle\", which could be one of the\n ingredients to allow \"git clone\" traffic off of the core server\n network to CDN.\n\n While I think it would make it easier for people to experiment and\n build on if the topic is merged to 'next', I am at the same time a\n bit reluctant to merge an unproven new topic that introduces a new\n file format, which we may end up having to support til the end of\n time.  It is likely that to support a \"prime clone from CDN\", it\n would need a lot more than just \"these are the heads and the pack\n data is over there\", so this may not be sufficient.\n\n Will discard.\n\n\n* jc/attr (2016-05-25) 18 commits\n - attr: support quoting pathname patterns in C style\n - attr: expose validity check for attribute names\n - attr: add counted string version of git_attr()\n - attr: add counted string version of git_check_attr()\n - attr: retire git_check_attrs() API\n - attr: convert git_check_attrs() callers to use the new API\n - attr: convert git_all_attrs() to use \"struct git_attr_check\"\n - attr: (re)introduce git_check_attr() and struct git_attr_check\n - attr: rename function and struct related to checking attributes\n - attr.c: plug small leak in parse_attr_line()\n - attr.c: tighten constness around \"git_attr\" structure\n - attr.c: simplify macroexpand_one()\n - attr.c: mark where #if DEBUG ends more clearly\n - attr.c: complete a sentence in a comment\n - attr.c: explain the lack of attr-name syntax check in parse_attr()\n - attr.c: update a stale comment on \"struct match_attr\"\n - attr.c: use strchrnul() to scan for one line\n - commit.c: use strchrnul() to scan for one line\n (this branch is used by jc/attr-more, sb/pathspec-label and sb/submodule-default-paths.)\n\n The attributes API has been updated so that it can later be\n optimized using the knowledge of which attributes are queried.\n\n I wanted to polish this topic further to make the attribute\n subsystem thread-ready, but because other topics depend on this\n topic and they do not (yet) need it to be thread-ready.\n\n As the authors of topics that depend on this seem not in a hurry,\n let's discard this and dependent topics and restart them some other\n day.\n\n Will discard.\n\n\n* jc/attr-more (2016-06-09) 8 commits\n - attr.c: outline the future plans by heavily commenting\n - attr.c: always pass check[] to collect_some_attrs()\n - attr.c: introduce empty_attr_check_elems()\n - attr.c: correct ugly hack for git_all_attrs()\n - attr.c: rename a local variable check\n - fixup! d5ad6c13\n - attr.c: pass struct git_attr_check down the callchain\n - attr.c: add push_stack() helper\n (this branch uses jc/attr; is tangled with sb/pathspec-label and sb/submodule-default-paths.)\n\n The beginning of long and tortuous journey to clean-up attribute\n subsystem implementation.\n\n Needs to be redone.\n Will discard.\n\n\n* sb/submodule-default-paths (2016-06-20) 5 commits\n - completion: clone can recurse into submodules\n - clone: add --init-submodule=<pathspec> switch\n - submodule update: add `--init-default-path` switch\n - Merge branch 'sb/pathspec-label' into sb/submodule-default-paths\n - Merge branch 'jc/attr' into sb/submodule-default-paths\n (this branch uses jc/attr and sb/pathspec-label; is tangled with jc/attr-more.)\n\n Allow specifying the set of submodules the user is interested in on\n the command line of \"git clone\" that clones the superproject.\n\n Will discard.\n\n\n* sb/pathspec-label (2016-06-03) 6 commits\n - pathspec: disable preload-index when attribute pathspec magic is in use\n - pathspec: allow escaped query values\n - pathspec: allow querying for attributes\n - pathspec: move prefix check out of the inner loop\n - pathspec: move long magic parsing out of prefix_pathspec\n - Documentation: fix a typo\n (this branch is used by sb/submodule-default-paths; uses jc/attr; is tangled with jc/attr-more.)\n\n The pathspec mechanism learned \":(attr:X)$pattern\" pathspec magic\n to limit paths that match $pattern further by attribute settings.\n The preload-index mechanism is disabled when the new pathspec magic\n is in use (at least for now), because the attribute subsystem is\n not thread-ready.\n\n Will discard.\n\n\n* mh/connect (2016-06-06) 10 commits\n - connect: [host:port] is legacy for ssh\n - connect: move ssh command line preparation to a separate function\n - connect: actively reject git:// urls with a user part\n - connect: change the --diag-url output to separate user and host\n - connect: make parse_connect_url() return the user part of the url as a separate value\n - connect: group CONNECT_DIAG_URL handling code\n - connect: make parse_connect_url() return separated host and port\n - connect: re-derive a host:port string from the separate host and port variables\n - connect: call get_host_and_port() earlier\n - connect: document why we sometimes call get_port after get_host_and_port\n\n Rewrite Git-URL parsing routine (hopefully) without changing any\n behaviour.\n\n It has been two months without any support.  We may want to discard\n this.\n\n\n* pb/bisect (2016-08-23) 27 commits\n . bisect--helper: remove the dequote in bisect_start()\n . bisect--helper: retire `--bisect-auto-next` subcommand\n . bisect--helper: retire `--bisect-autostart` subcommand\n . bisect--helper: retire `--check-and-set-terms` subcommand\n . bisect--helper: retire `--bisect-write` subcommand\n . bisect--helper: `bisect_replay` shell function in C\n . bisect--helper: `bisect_log` shell function in C\n . bisect--helper: retire `--write-terms` subcommand\n . bisect--helper: retire `--check-expected-revs` subcommand\n . bisect--helper: `bisect_state` & `bisect_head` shell function in C\n . bisect--helper: `bisect_autostart` shell function in C\n . bisect--helper: retire `--next-all` subcommand\n . bisect--helper: retire `--bisect-clean-state` subcommand\n . bisect--helper: `bisect_next` and `bisect_auto_next` shell function in C\n . bisect--helper: `bisect_start` shell function partially in C\n . bisect--helper: `get_terms` & `bisect_terms` shell function in C\n . bisect--helper: `bisect_next_check` & bisect_voc shell function in C\n . bisect--helper: `check_and_set_terms` shell function in C\n . bisect--helper: `bisect_write` shell function in C\n . bisect--helper: `is_expected_rev` & `check_expected_revs` shell function in C\n . bisect--helper: `bisect_reset` shell function in C\n . wrapper: move is_empty_file() and rename it as is_empty_or_missing_file()\n . t6030: explicitly test for bisection cleanup\n . bisect--helper: `bisect_clean_state` shell function in C\n . bisect--helper: `write_terms` shell function in C\n . bisect: rewrite `check_term_format` shell function in C\n . bisect--helper: use OPT_CMDMODE instead of OPT_BOOL\n\n GSoC \"bisect\" topic.\n\n I'd prefer to see early part solidified so that reviews can focus\n on the later part that is still in flux.  We are almost there but\n not quite yet.\n\n\n* kn/ref-filter-branch-list (2016-05-17) 17 commits\n - branch: implement '--format' option\n - branch: use ref-filter printing APIs\n - branch, tag: use porcelain output\n - ref-filter: allow porcelain to translate messages in the output\n - ref-filter: add `:dir` and `:base` options for ref printing atoms\n - ref-filter: make remote_ref_atom_parser() use refname_atom_parser_internal()\n - ref-filter: introduce symref_atom_parser() and refname_atom_parser()\n - ref-filter: introduce refname_atom_parser_internal()\n - ref-filter: make \"%(symref)\" atom work with the ':short' modifier\n - ref-filter: add support for %(upstream:track,nobracket)\n - ref-filter: make %(upstream:track) prints \"[gone]\" for invalid upstreams\n - ref-filter: introduce format_ref_array_item()\n - ref-filter: move get_head_description() from branch.c\n - ref-filter: modify \"%(objectname:short)\" to take length\n - ref-filter: implement %(if:equals=<string>) and %(if:notequals=<string>)\n - ref-filter: include reference to 'used_atom' within 'atom_value'\n - ref-filter: implement %(if), %(then), and %(else) atoms\n\n The code to list branches in \"git branch\" has been consolidated\n with the more generic ref-filter API.\n\n Rerolled.\n Needs review.\n\n\n* sb/bisect (2016-04-15) 22 commits\n . SQUASH???\n . bisect: get back halfway shortcut\n . bisect: compute best bisection in compute_relevant_weights()\n . bisect: use a bottom-up traversal to find relevant weights\n . bisect: prepare for different algorithms based on find_all\n . bisect: rename count_distance() to compute_weight()\n . bisect: make total number of commits global\n . bisect: introduce distance_direction()\n . bisect: extract get_distance() function from code duplication\n . bisect: use commit instead of commit list as arguments when appropriate\n . bisect: replace clear_distance() by unique markers\n . bisect: use struct node_data array instead of int array\n . bisect: get rid of recursion in count_distance()\n . bisect: make algorithm behavior independent of DEBUG_BISECT\n . bisect: make bisect compile if DEBUG_BISECT is set\n . bisect: plug the biggest memory leak\n . bisect: add test for the bisect algorithm\n . t6030: generalize test to not rely on current implementation\n . t: use test_cmp_rev() where appropriate\n . t/test-lib-functions.sh: generalize test_cmp_rev\n . bisect: allow 'bisect run' if no good commit is known\n . bisect: write about `bisect next` in documentation\n\n The internal algorithm used in \"git bisect\" to find the next commit\n to check has been optimized greatly.\n\n Was expecting a reroll, but now pb/bisect topic starts removinging\n more and more parts from git-bisect.sh, this needs to see a fresh\n reroll.\n\n Will discard.\n cf. <1460294354-7031-1-git-send-email-s-beyer@gmx.net>\n\n\n* sg/completion-updates (2016-02-28) 21 commits\n . completion: cache the path to the repository\n . completion: extract repository discovery from __gitdir()\n . completion: don't guard git executions with __gitdir()\n . completion: consolidate silencing errors from git commands\n . completion: don't use __gitdir() for git commands\n . completion: respect 'git -C <path>'\n . completion: fix completion after 'git -C <path>'\n . completion: don't offer commands when 'git --opt' needs an argument\n . rev-parse: add '--absolute-git-dir' option\n . completion: list short refs from a remote given as a URL\n . completion: don't list 'HEAD' when trying refs completion outside of a repo\n . completion: list refs from remote when remote's name matches a directory\n . completion: respect 'git --git-dir=<path>' when listing remote refs\n . completion: fix most spots not respecting 'git --git-dir=<path>'\n . completion: ensure that the repository path given on the command line exists\n . completion tests: add tests for the __git_refs() helper function\n . completion tests: check __gitdir()'s output in the error cases\n . completion tests: consolidate getting path of current working directory\n . completion tests: make the $cur variable local to the test helper functions\n . completion tests: don't add test cruft to the test repository\n . completion: improve __git_refs()'s in-code documentation\n\n Has been waiting for a reroll for too long.\n cf. <1456754714-25237-1-git-send-email-szeder@ira.uka.de>\n\n Will discard.\n\n\n* ec/annotate-deleted (2015-11-20) 1 commit\n - annotate: skip checking working tree if a revision is provided\n\n Usability fix for annotate-specific \"<file> <rev>\" syntax with deleted\n files.\n\n Has been waiting for a review for too long without seeing anything.\n\n Will discard.\n\n\n* dk/gc-more-wo-pack (2016-01-13) 4 commits\n - gc: clean garbage .bitmap files from pack dir\n - t5304: ensure non-garbage files are not deleted\n - t5304: test .bitmap garbage files\n - prepare_packed_git(): find more garbage\n\n Follow-on to dk/gc-idx-wo-pack topic, to clean up stale\n .bitmap and .keep files.\n\n Has been waiting for a reroll for too long.\n cf. <xmqq60ypbeng.fsf@gitster.mtv.corp.google.com>\n\n Will discard.\n\n\n* jc/diff-b-m (2015-02-23) 5 commits\n . WIPWIP\n . WIP: diff-b-m\n - diffcore-rename: allow easier debugging\n - diffcore-rename.c: add locate_rename_src()\n - diffcore-break: allow debugging\n\n \"git diff -B -M\" produced incorrect patch when the postimage of a\n completely rewritten file is similar to the preimage of a removed\n file; such a resulting file must not be expressed as a rename from\n other place.\n\n The fix in this patch is broken, unfortunately.\n\n Will discard.\n\n--------------------------------------------------\n[Cooking]\n\n* jk/verify-packfile-gently (2016-09-22) 1 commit\n  (merged to 'next' on 2016-09-26 at f5abba5)\n + verify_packfile: check pack validity before accessing data\n\n A low-level function verify_packfile() was meant to show errors\n that were detected without dying itself, but under some conditions\n it didn't and died instead, which has been fixed.\n\n Will merge to 'master'.\n\n\n* jt/fetch-pack-in-vain-count-with-stateless (2016-09-23) 1 commit\n  (merged to 'next' on 2016-09-26 at 9645629)\n + fetch-pack: do not reset in_vain on non-novel acks\n\n When \"git fetch\" tries to find where the history of the repository\n it runs inhas diverged from what the other side has, it has a\n mechanism to avoid digging too deep into irrelevant side branches.\n This however did not work well over the \"smart-http\" transport due\n to a design bug, which has been fixed.\n\n Will merge to 'master'.\n\n\n* ik/gitweb-force-highlight (2016-09-25) 2 commits\n  (merged to 'next' on 2016-09-27 at cbb8391)\n + gitweb: use highlight's shebang detection\n + gitweb: remove unused guess_file_syntax() parameter\n\n \"gitweb\" can spawn \"highlight\" to show blob contents with\n (programming) language-specific syntax highlighting, but only\n when the language is known.  \"highlight\" can however be told\n to make the guess itself by giving it \"--force\" option, which\n has been enabled.\n\n Will merge to 'master'.\n\n\n* jk/ident-ai-canonname-could-be-null (2016-09-23) 1 commit\n  (merged to 'next' on 2016-09-26 at 0eefb29)\n + ident: handle NULL ai_canonname\n\n In the codepath that comes up with the hostname to be used in an\n e-mail when the user didn't tell us, we looked at ai_canonname\n field in struct addrinfo without making sure it is not NULL first.\n\n Will merge to 'master'.\n\n\n* jc/blame-reverse (2016-06-14) 2 commits\n  (merged to 'next' on 2016-09-22 at d1a8e9c)\n + blame: dwim \"blame --reverse OLD\" as \"blame --reverse OLD..\"\n + blame: improve diagnosis for \"--reverse NEW\"\n\n It is a common mistake to say \"git blame --reverse OLD path\",\n expecting that the command line is dwimmed as if asking how lines\n in path in an old revision OLD have survived up to the current\n commit.\n\n Will hold to see if it is broken.\n\n\n* nd/init-core-worktree-in-multi-worktree-world (2016-09-25) 5 commits\n  (merged to 'next' on 2016-09-27 at 619f7f3)\n + init: kill git_link variable\n + init: do not set unnecessary core.worktree\n + init: kill set_git_dir_init()\n + init: call set_git_dir_init() from within init_db()\n + init: correct re-initialization from a linked worktree\n\n \"git init\" tried to record core.worktree in the repository's\n 'config' file when GIT_WORK_TREE environment variable was set and\n it was different from where GIT_DIR appears as \".git\" at its top,\n but the logic was faulty when .git is a \"gitdir:\" file that points\n at the real place, causing trouble in working trees that are\n managed by \"git worktree\".  This has been corrected.\n\n Will merge to 'master'.\n\n\n* jt/mailinfo-fold-in-body-headers (2016-09-21) 3 commits\n  (merged to 'next' on 2016-09-26 at 4235eb6)\n + mailinfo: handle in-body header continuations\n + mailinfo: make is_scissors_line take plain char *\n + mailinfo: separate in-body header processing\n\n When \"git format-patch --stdout\" output is placed as an in-body\n header and it uses the RFC2822 header folding, \"git am\" failed to\n put the header line back into a single logical line.  The\n underlying \"git mailinfo\" was taught to handle this properly.\n\n Will merge to 'master'.\n\n\n* kd/mailinfo-quoted-string (2016-09-26) 3 commits\n - mailinfo: unescape quoted-pair in header fields\n - SQUASH???\n - t5100-mailinfo: replace common path prefix with variable\n\n An e-mail author named that spelled a backslash-quoted double quote\n in the human readable part \"My \\\"double quoted\\\" name\" was not\n unquoted correctly.\n\n Waiting for a reroll.\n cf. <20160926194455.GB19089@ikke.info>\n\n\n* js/libify-require-clean-work-tree (2016-09-12) 5 commits\n - wt-status: teach has_{unstaged,uncommitted}_changes() about submodules\n - Export also the has_un{staged,committed}_changed() functions\n - Make the require_clean_work_tree() function truly reusable\n - pull: make code more similar to the shell script again\n - pull: drop confusing prefix parameter of die_on_unclean_work_tree()\n\n The require_clean_work_tree() helper was recreated in C when \"git\n pull\" was rewritten from shell; the helper is now made available to\n other callers in preparation for upcoming \"rebase -i\" work.\n\n Waiting for comments.\n Modulo a few minor nits, this looked almost ready.\n cf. <xmqqtwdl2bhm.fsf@gitster.mtv.corp.google.com>\n cf. <xmqqpoo92bdr.fsf@gitster.mtv.corp.google.com>\n\n\n* bw/ls-files-recurse-submodules (2016-09-21) 2 commits\n - ls-files: add pathspec matching for submodules\n - ls-files: optionally recurse into submodules\n\n \"git ls-files\" learned \"--recurse-submodules\" option that can be\n used to get a listing of tracked files across submodules (i.e. this\n only works with \"--cached\" option, not for listing untracked or\n ignored files).  This would be a useful tool to sit on the upstream\n side of a pipe that is read with xargs to work on all working tree\n files from the top-level superproject.\n\n Waiting for the discussion to conclude.\n\n\n* ls/filter-process (2016-09-23) 11 commits\n - convert: add filter.<driver>.process option\n - convert: make apply_filter() adhere to standard Git error handling\n - convert: modernize tests\n - convert: quote filter names in error messages\n - pkt-line: add functions to read/write flush terminated packet streams\n - pkt-line: add packet_write_gently()\n - pkt-line: add packet_flush_gently()\n - pkt-line: add packet_write_fmt_gently()\n - run-command: move check_pipe() from write_or_die to run_command\n - pkt-line: extract set_packet_header()\n - pkt-line: rename packet_write() to packet_write_fmt()\n\n The smudge/clean filter API expect an external process is spawned\n to filter the contents for each path that has a filter defined.  A\n new type of \"process\" filter API has been added to allow the first\n request to run the filter for a path to spawn a single process, and\n all filtering need is served by this single process for multiple\n paths, reducing the process creation overhead.\n\n\n* hv/submodule-not-yet-pushed-fix (2016-09-14) 2 commits\n - serialize collection of refs that contain submodule changes\n - serialize collection of changed submodules\n\n The code in \"git push\" to compute if any commit being pushed in the\n superproject binds a commit in a submodule that hasn't been pushed\n out was overly inefficient, making it unusable even for a small\n project that does not have any submodule but have a reasonable\n number of refs.\n\n The last two in the original series seem to break a few tests when\n queued to 'pu', and dropped for now.\n\n Waiting for a reroll.\n\n\n* sg/fix-versioncmp-with-common-suffix (2016-09-08) 5 commits\n - versioncmp: cope with common leading parts in versionsort.prereleaseSuffix\n - versioncmp: pass full tagnames to swap_prereleases()\n - t7004-tag: add version sort tests to show prerelease reordering issues\n - t7004-tag: use test_config helper\n - t7004-tag: delete unnecessary tags with test_when_finished\n\n The prereleaseSuffix feature of version comparison that is used in\n \"git tag -l\" did not correctly when two or more prereleases for the\n same release were present (e.g. when 2.0, 2.0-beta1, and 2.0-beta2\n are there and the code needs to compare 2.0-beta1 and 2.0-beta2).\n\n Waiting for a reroll.\n cf. <20160908223727.Horde.jVOOJ278ssZ3qkyjkmyqZD-@webmail.informatik.kit.edu>\n\n\n* cp/completion-negative-refs (2016-08-24) 1 commit\n  (merged to 'next' on 2016-09-22 at abd1585)\n + completion: support excluding refs\n\n The command-line completion script (in contrib/) learned to\n complete \"git cmd ^mas<HT>\" to complete the negative end of\n reference to \"git cmd ^master\".\n\n Will hold to see if it is broken.\n\n\n* sb/push-make-submodule-check-the-default (2016-08-24) 1 commit\n - push: change submodule default to check\n\n Turn the default of \"push.recurseSubmodules\" to \"check\".\n\n Will hold to wait for hv/submodule-not-yet-pushed-fix\n\n This reveals that the \"check\" mode is too inefficient to use in\n real projects, even in ones as small as git itself.\n cf. <xmqqh9aaot49.fsf@gitster.mtv.corp.google.com>\n\n\n* ak/curl-imap-send-explicit-scheme (2016-08-17) 1 commit\n  (merged to 'next' on 2016-09-22 at 4449584)\n + imap-send: Tell cURL to use imap:// or imaps://\n\n When we started cURL to talk to imap server when a new enough\n version of cURL library is available, we forgot to explicitly add\n imap(s):// before the destination.  To some folks, that didn't work\n and the library tried to make HTTP(s) requests instead.\n\n Will hold to see if it is broken.\n\n\n* jk/pack-objects-optim-mru (2016-08-11) 4 commits\n  (merged to 'next' on 2016-09-21 at 97b919b)\n + pack-objects: use mru list when iterating over packs\n + pack-objects: break delta cycles before delta-search phase\n + sha1_file: make packed_object_info public\n + provide an initializer for \"struct object_info\"\n\n Originally merged to 'next' on 2016-08-11\n\n \"git pack-objects\" in a repository with many packfiles used to\n spend a lot of time looking for/at objects in them; the accesses to\n the packfiles are now optimized by checking the most-recently-used\n packfile first.\n\n Will hold to see if people scream.\n\n\n* dp/autoconf-curl-ssl (2016-06-28) 1 commit\n  (merged to 'next' on 2016-09-22 at 9c5aeec)\n + ./configure.ac: detect SSL in libcurl using curl-config\n\n The ./configure script generated from configure.ac was taught how\n to detect support of SSL by libcurl better.\n\n Will hold to see if it is broken.\n\n\n* jc/pull-rebase-ff (2016-07-28) 1 commit\n - pull: fast-forward \"pull --rebase=true\"\n\n \"git pull --rebase\", when there is no new commits on our side since\n we forked from the upstream, should be able to fast-forward without\n invoking \"git rebase\", but it didn't.\n\n Needs a real log message and a few tests.\n\n\n* ex/deprecate-empty-pathspec-as-match-all (2016-06-22) 1 commit\n  (merged to 'next' on 2016-09-21 at e19148e)\n + pathspec: warn on empty strings as pathspec\n\n Originally merged to 'next' on 2016-07-13\n\n An empty string used as a pathspec element has always meant\n 'everything matches', but it is too easy to write a script that\n finds a path to remove in $path and run 'git rm \"$paht\"', which\n ends up removing everything.  Start warning about this use of an\n empty string used for 'everything matches' and ask users to use a\n more explicit '.' for that instead.\n\n The hope is that existing users will not mind this change, and\n eventually the warning can be turned into a hard error, upgrading\n the deprecation into removal of this (mis)feature.\n\n Will hold to see if people scream.\n\n\n* nd/shallow-deepen (2016-06-13) 27 commits\n  (merged to 'next' on 2016-09-22 at f0cf3e3)\n + fetch, upload-pack: --deepen=N extends shallow boundary by N commits\n + upload-pack: add get_reachable_list()\n + upload-pack: split check_unreachable() in two, prep for get_reachable_list()\n + t5500, t5539: tests for shallow depth excluding a ref\n + clone: define shallow clone boundary with --shallow-exclude\n + fetch: define shallow boundary with --shallow-exclude\n + upload-pack: support define shallow boundary by excluding revisions\n + refs: add expand_ref()\n + t5500, t5539: tests for shallow depth since a specific date\n + clone: define shallow clone boundary based on time with --shallow-since\n + fetch: define shallow boundary with --shallow-since\n + upload-pack: add deepen-since to cut shallow repos based on time\n + shallow.c: implement a generic shallow boundary finder based on rev-list\n + fetch-pack: use a separate flag for fetch in deepening mode\n + fetch-pack.c: mark strings for translating\n + fetch-pack: use a common function for verbose printing\n + fetch-pack: use skip_prefix() instead of starts_with()\n + upload-pack: move rev-list code out of check_non_tip()\n + upload-pack: make check_non_tip() clean things up on error\n + upload-pack: tighten number parsing at \"deepen\" lines\n + upload-pack: use skip_prefix() instead of starts_with()\n + upload-pack: move \"unshallow\" sending code out of deepen()\n + upload-pack: remove unused variable \"backup\"\n + upload-pack: move \"shallow\" sending code out of deepen()\n + upload-pack: move shallow deepen code out of receive_needs()\n + transport-helper.c: refactor set_helper_option()\n + remote-curl.c: convert fetch_git() to use argv_array\n\n The existing \"git fetch --depth=<n>\" option was hard to use\n correctly when making the history of an existing shallow clone\n deeper.  A new option, \"--deepen=<n>\", has been added to make this\n easier to use.  \"git clone\" also learned \"--shallow-since=<date>\"\n and \"--shallow-exclude=<tag>\" options to make it easier to specify\n \"I am interested only in the recent N months worth of history\" and\n \"Give me only the history since that version\".\n\n Will hold to see if it is broken.\n\n\n* jc/merge-drop-old-syntax (2015-04-29) 1 commit\n - merge: drop 'git merge <message> HEAD <commit>' syntax\n\n Stop supporting \"git merge <message> HEAD <commit>\" syntax that has\n been deprecated since October 2007, and issues a deprecation\n warning message since v2.5.0.\n\n It has been reported that git-gui still uses the deprecated syntax,\n which needs to be fixed before this final step can proceed.\n cf. <5671DB28.8020901@kdbg.org>\n\n Will hold to wait for rs/git-gui-use-modern-git-merge-syntax\n\n--------------------------------------------------\n[Discarded]\n\n* jn/fix-connect-unexpected-hangup-diag (2016-09-08) 1 commit\n . connect: tighten check for unexpected early hang up\n\n Now part of jt/accept-capability-advertisement-when-fetching-from-void\n topic.\n"},{"id":"303038","messageId":"CAGZ79kY6c-vwSP7-1Gz4jwWO-z_yT2oFbG4cgZb-JAae=Sk-cA@mail.gmail.com","threadId":"44178","inReplyTo":"xmqq8tudkjvn.fsf@gitster.mtv.corp.google.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-01T01:33:44Z","receivedAt":"2016-10-01T01:33:50Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">\n>\n> * jc/attr (2016-05-25) 18 commits\n>  - attr: support quoting pathname patterns in C style\n>  - attr: expose validity check for attribute names\n>  - attr: add counted string version of git_attr()\n>  - attr: add counted string version of git_check_attr()\n>  - attr: retire git_check_attrs() API\n>  - attr: convert git_check_attrs() callers to use the new API\n>  - attr: convert git_all_attrs() to use \"struct git_attr_check\"\n>  - attr: (re)introduce git_check_attr() and struct git_attr_check\n>  - attr: rename function and struct related to checking attributes\n>  - attr.c: plug small leak in parse_attr_line()\n>  - attr.c: tighten constness around \"git_attr\" structure\n>  - attr.c: simplify macroexpand_one()\n>  - attr.c: mark where #if DEBUG ends more clearly\n>  - attr.c: complete a sentence in a comment\n>  - attr.c: explain the lack of attr-name syntax check in parse_attr()\n>  - attr.c: update a stale comment on \"struct match_attr\"\n>  - attr.c: use strchrnul() to scan for one line\n>  - commit.c: use strchrnul() to scan for one line\n>  (this branch is used by jc/attr-more, sb/pathspec-label and sb/submodule-default-paths.)\n>\n>  The attributes API has been updated so that it can later be\n>  optimized using the knowledge of which attributes are queried.\n>\n>  I wanted to polish this topic further to make the attribute\n>  subsystem thread-ready, but because other topics depend on this\n>  topic and they do not (yet) need it to be thread-ready.\n>\n>  As the authors of topics that depend on this seem not in a hurry,\n>  let's discard this and dependent topics and restart them some other\n>  day.\n>\n>  Will discard.\n\nSo I just realized this is a big hint for me to pick up that topic; I assumed\nyou'd want to tackle the attr subsystem eventually, so all I was doing, was\nwaiting for your motivation to look at attr stuff to come back.\n\nSo what is the actual lacking stuff here?\n\nThanks,\nStefan\n"},{"id":"303104","messageId":"xmqqk2dp71d2.fsf@gitster.mtv.corp.google.com","threadId":"44178","inReplyTo":"CAGZ79kY6c-vwSP7-1Gz4jwWO-z_yT2oFbG4cgZb-JAae=Sk-cA@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-03T18:07:21Z","receivedAt":"2016-10-03T18:07:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> * jc/attr (2016-05-25) 18 commits\n>> ...\n>>  The attributes API has been updated so that it can later be\n>>  optimized using the knowledge of which attributes are queried.\n>>\n>>  I wanted to polish this topic further to make the attribute\n>>  subsystem thread-ready, but because other topics depend on this\n>>  topic and they do not (yet) need it to be thread-ready.\n>>\n>>  As the authors of topics that depend on this seem not in a hurry,\n>>  let's discard this and dependent topics and restart them some other\n>>  day.\n>>\n>>  Will discard.\n>\n> So I just realized this is a big hint for me to pick up that topic; I assumed\n> you'd want to tackle the attr subsystem eventually, so all I was doing, was\n> waiting for your motivation to look at attr stuff to come back.\n>\n> So what is the actual lacking stuff here?\n\nQuite a bit.  Do you want a grand vision, or just the minimum that\nwill hopefully futureproof us?\n\nThe current attribute API is optimized for a very narrow use case\nwhere a single thread looks up a single set of attributes for\nadjacent paths, without mixing lookups for other attributes of\ndistant paths.\n\nTake \"conversion\" codepath for an example.  \"git checkout\" will\niterate over the paths in the index and is interested in the eol,\ntext and filter attributes (perhaps more, but the details do not\nchange the overall picture that much).  When it checks dir/fileA, it\nis expected that it would next want to check dir/fileB, before it\nwould want to check dir2/fileC and it would move much later to\notherdir/fileD.  It also is expected that it would want to learn the\nsame set of attributes, simply because the codepath is doing the\nsame operation over these paths (i.e. learn how the Git \"clean\"\nrepresentation needs to be converted to the external \"smudged\"\nrepresentation).\n\nBased on the \"adjacent paths\" assumption, the attribute subsystem\nhas a single cache that holds the contents of the .gitattributes\nfiles that matter to the current query.  This has to be split up if\nwe ever want to do a parallel checkout with multiple threads.  One\nthread may be walking the first half of the index, while another one\nmay be walking the second half of the index.  They would benefit\nfrom the same optimization that keeps track of the contents of\n.gitattributes files that matters to each of their traversal.  If\nthe first thread is responsible for working on dir/fileA, it will\nwork on dir/fileB next, before going to dir2/fileC, but it does not\nwant to share the cache with the other thread that would be scanning\nentries far-away from what it's scanning, like otherdir/fileD, if it\nevicts the cached information for dir/* that is still useful for the\nfirst thread.\n\nAnother thing we would want to see is to take advantage of the\n\"lookup is for single set of attributes from a single codepath\" for\nfurther optimization.  The way each of the callers is structured is\nto first declare a set of attributes it is interested in by\npreparing git_attr_check_elem[] array and then make repeated calls\nto git_attr_check() passing it and a path.\n\nThe current implementation however does not tie the cache to this\ngit_attr_check_elem[] array but has only one single global cache.\nThe cache _could_ be used to query any attribute because of it, and\nthat leads to inefficiency.  It has everything it read from relevant\n.gitattributes files, even the entries that do not affect any\nattributes that a single codepath showed its interest in.  I am\nhoping that we can do better by having a per <thread, callsite>\ncache of .gitattributes files, so that a caller in one thread (say,\n\"git checkout\" that scans the first half of the index) that asks for\n\"eol\" and \"filter\" would use a cache that does not have entries\nirrelevant for the attributes the caller is intereseted in, and that\nis tied to the directory hierarchy the caller is asking about\n(i.e. what prepare_attr_stack() does).\n\nUp to 079186123a (\"attr: retire git_check_attrs() API\", 2016-05-16)\nof the series gives us \"struct git_attr_check\" to replace the\ngit_attr_check_elem[] array.  I originally hoped that this struct\ncan hold the per-callsite cache itself, before we hit the threading\nissue too early (IIRC, that was preload-index code) and realize that\nthe cache needs to be not just per callsite but needs to be per\n<thread, callsite>.  This new structure cannot be used to store the\ncache itself, but this change is probably a necessary first step for\nallowing the API in multi-threaded context.  git_attr_check_elem[]\narray was static and had slots to receive returned values, which\nwould not have worked in threaded environment.  We'd further need to\nchange it so that the inquiry keys (which are \"interned\" git_attr\nobjects) of \"git_attr_check\" are initialized just once before\nstarting to make repeated calls to git_attr_check(), but the\nmechanism to return values would be thread-safe.  git_check_attr()\ncall may have to gain an extra/separate variable for the caller to\nspecify an array to return values, or something.\n\nThe way \"interned\" git_attr objects are managed needs to become\nthread-safe by protecting their creation and registering with mutex,\nbut that is relatively isolated and straightforward conversion so I\ndidn't pay any attention to that in my series.  In the final state,\nit of course needs to be taken care of.\n\nSo that's the overall \"grand vision\" picture the series leading up\nto the tip of jc/attr-more was trying to lead us to.\n\nThe minimum that would future-proof us, that is still missing from\nthe series, would probably be to separate the query parameter\n\"struct git_attr_check\" and the return values from git_check_attr().\nOnce it is done, I think we do not need to update the caller when we\nupdate the attr.c infrastructure to be thread-safe.\n"},{"id":"303109","messageId":"CAGZ79ka8dO1AHJftKAqD6LvxJSP+8yGGa7Citcdxxrnc5DMeYg@mail.gmail.com","threadId":"44178","inReplyTo":"xmqqk2dp71d2.fsf@gitster.mtv.corp.google.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-03T19:55:55Z","receivedAt":"2016-10-03T19:56:03Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 3, 2016 at 11:07 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>>> * jc/attr (2016-05-25) 18 commits\n>>> ...\n>>>  The attributes API has been updated so that it can later be\n>>>  optimized using the knowledge of which attributes are queried.\n>>>\n>>>  I wanted to polish this topic further to make the attribute\n>>>  subsystem thread-ready, but because other topics depend on this\n>>>  topic and they do not (yet) need it to be thread-ready.\n>>>\n>>>  As the authors of topics that depend on this seem not in a hurry,\n>>>  let's discard this and dependent topics and restart them some other\n>>>  day.\n>>>\n>>>  Will discard.\n>>\n>> So I just realized this is a big hint for me to pick up that topic; I assumed\n>> you'd want to tackle the attr subsystem eventually, so all I was doing, was\n>> waiting for your motivation to look at attr stuff to come back.\n>>\n>> So what is the actual lacking stuff here?\n>\n> Quite a bit.  Do you want a grand vision, or just the minimum that\n> will hopefully futureproof us?\n\nThat is a good question. I do want the bare minimum at least; time\npermitting I can do more.\n\nI looked through jc/attr and I think all commits up to\n0a5aad (2016-05-25, attr.c: plug small leak in parse_attr_line())\nare better off if we'd include them today, no matter if we go with the\ngrand vision or the bare minimum viable, because all commits\nup to that one are fixing real issues (memory or readability issues)\n\nStarting with its child b649c7a50c (attr: rename function and struct\nrelated to checking attributes) we start off going on an adventure which\nmay lead to the grand vision implemented, so these should be held off\nuntil we have implemented a viable way.\n\n>\n> The current attribute API is optimized for a very narrow use case\n> where a single thread looks up a single set of attributes for\n> adjacent paths, without mixing lookups for other attributes of\n> distant paths.\n>\n> Take \"conversion\" codepath for an example.  \"git checkout\" will\n> iterate over the paths in the index and is interested in the eol,\n> text and filter attributes (perhaps more, but the details do not\n> change the overall picture that much).  When it checks dir/fileA, it\n> is expected that it would next want to check dir/fileB, before it\n> would want to check dir2/fileC and it would move much later to\n> otherdir/fileD.  It also is expected that it would want to learn the\n> same set of attributes, simply because the codepath is doing the\n> same operation over these paths (i.e. learn how the Git \"clean\"\n> representation needs to be converted to the external \"smudged\"\n> representation).\n>\n> Based on the \"adjacent paths\" assumption, the attribute subsystem\n> has a single cache that holds the contents of the .gitattributes\n> files that matter to the current query.  This has to be split up if\n> we ever want to do a parallel checkout with multiple threads.  One\n> thread may be walking the first half of the index, while another one\n> may be walking the second half of the index.  They would benefit\n> from the same optimization that keeps track of the contents of\n> .gitattributes files that matters to each of their traversal.  If\n> the first thread is responsible for working on dir/fileA, it will\n> work on dir/fileB next, before going to dir2/fileC, but it does not\n> want to share the cache with the other thread that would be scanning\n> entries far-away from what it's scanning, like otherdir/fileD, if it\n> evicts the cached information for dir/* that is still useful for the\n> first thread.\n>\n> Another thing we would want to see is to take advantage of the\n> \"lookup is for single set of attributes from a single codepath\" for\n> further optimization.  The way each of the callers is structured is\n> to first declare a set of attributes it is interested in by\n> preparing git_attr_check_elem[] array and then make repeated calls\n> to git_attr_check() passing it and a path.\n>\n> The current implementation however does not tie the cache to this\n> git_attr_check_elem[] array but has only one single global cache.\n> The cache _could_ be used to query any attribute because of it, and\n> that leads to inefficiency.  It has everything it read from relevant\n> .gitattributes files, even the entries that do not affect any\n> attributes that a single codepath showed its interest in.  I am\n> hoping that we can do better by having a per <thread, callsite>\n> cache of .gitattributes files, so that a caller in one thread (say,\n> \"git checkout\" that scans the first half of the index) that asks for\n> \"eol\" and \"filter\" would use a cache that does not have entries\n> irrelevant for the attributes the caller is intereseted in, and that\n> is tied to the directory hierarchy the caller is asking about\n> (i.e. what prepare_attr_stack() does).\n>\n> Up to 079186123a (\"attr: retire git_check_attrs() API\", 2016-05-16)\n> of the series gives us \"struct git_attr_check\" to replace the\n> git_attr_check_elem[] array.  I originally hoped that this struct\n> can hold the per-callsite cache itself, before we hit the threading\n> issue too early (IIRC, that was preload-index code) and realize that\n> the cache needs to be not just per callsite but needs to be per\n> <thread, callsite>.  This new structure cannot be used to store the\n> cache itself, but this change is probably a necessary first step for\n> allowing the API in multi-threaded context.  git_attr_check_elem[]\n> array was static and had slots to receive returned values, which\n> would not have worked in threaded environment.  We'd further need to\n> change it so that the inquiry keys (which are \"interned\" git_attr\n> objects) of \"git_attr_check\" are initialized just once before\n> starting to make repeated calls to git_attr_check(), but the\n> mechanism to return values would be thread-safe.  git_check_attr()\n> call may have to gain an extra/separate variable for the caller to\n> specify an array to return values, or something.\n\nI am looking at the tip of jc/attr-more and for a minimum\nthread safety we'd want to change the call sites to be aware of the\nthreads, i.e. instead of doing\n\n    if (!check)\n        check = git_attr_check_initl(\"crlf\", \"ident\",\n                    \"filter\", \"eol\", \"text\",\n                    NULL);\n\nWe'd rather call\n\n        struct git_attr_check *check;\n        check = git_attr_check_lookup_or_initl_threadsafe(\n                \"crlf\", \"ident\", \"filter\", \"eol\", \"text\", NULL);\n         if (!git_check_attr(path, check)) {\n             ...\n\nSo we would make all init functions (git_attr_check_initl,\ngit_attr_check_alloc) to be aware of the threads and we would\nnot have a static variable to keep the state as that is global unfortunately,\nbut rather have a threadsafe lookup function. Though this makes me\nwonder how much performance we need to care about here as the version\nwith the static variable seems to be optimized a lot. So maybe we'd add this\nlookup function in attr.h so the respective callers can inline it?\n\n>\n> The way \"interned\" git_attr objects are managed needs to become\n> thread-safe by protecting their creation and registering with mutex,\n> but that is relatively isolated and straightforward conversion so I\n> didn't pay any attention to that in my series.  In the final state,\n> it of course needs to be taken care of.\n>\n> So that's the overall \"grand vision\" picture the series leading up\n> to the tip of jc/attr-more was trying to lead us to.\n>\n> The minimum that would future-proof us, that is still missing from\n> the series, would probably be to separate the query parameter\n> \"struct git_attr_check\" and the return values from git_check_attr().\n\nNot sure what you mean here with separating as a preparation for\nthe thread safety. As I understand it we can still keep the thread local\nstates in git_attr_check, we'd just have to route each thread to its\nown part of the memory in there?\n\n> Once it is done, I think we do not need to update the caller when we\n> update the attr.c infrastructure to be thread-safe.\n"},{"id":"303110","messageId":"xmqq1szx6vch.fsf@gitster.mtv.corp.google.com","threadId":"44178","inReplyTo":"CAGZ79ka8dO1AHJftKAqD6LvxJSP+8yGGa7Citcdxxrnc5DMeYg@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-03T20:17:18Z","receivedAt":"2016-10-03T20:17:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> The minimum that would future-proof us, that is still missing from\n>> the series, would probably be to separate the query parameter\n>> \"struct git_attr_check\" and the return values from git_check_attr().\n>\n> Not sure what you mean here with separating as a preparation for\n> the thread safety. As I understand it we can still keep the thread local\n> states in git_attr_check, we'd just have to route each thread to its\n> own part of the memory in there?\n\nFor example, look at what you did in your pathspec-label topic.\n\n    static int match_attrs(const char *name, int namelen,\n                           const struct pathspec_item *item)\n    {\n            int i;\n\n            git_check_attr_counted(name, namelen, item->attr_check);\n            for (i = 0; i < item->attr_match_nr; i++) {\n                    const char *value;\n                    int matched;\n                    enum attr_match_mode match_mode;\n\n                    value = item->attr_check->check[i].value;\n                    match_mode = item->attr_match[i].match_mode;\n\nEach pathspec item has an attr_check member that wants to see a\nspecific set of attributes for a path being matched.  Each element\nof the item->attr_check->check[] array is <attr, value> pair, where\n<attr> is a constant for the purpose of the codepath (i.e. no matter\nwhich thread is asking, and no matter for which path the question is\nbeing asked, it asks for a fixed attribute that was computed when\nthe pathspec was parsed).  But <value> is a slot to return the\nfinding back to the caller.\n\nSo you can never keep this code structure and have this function\ncalled more than once, specifically, you cannot make\ngit_check_attr_counted() call from multiple threads, at one time.\n\nInstead the calling convention needs to be updated to allow this\ncaller of git_check_attr_counted() to pass a separate array that is\non its stack, e.g.\n\n\tconst char *v[... some size ...];\n\n\tgit_check_attr_counted(name, namelen, item->attr_check, v);\n        for (i = 0; i < item->attr_match_nr; i++) {\n        \tconst char *value;\n\n                value = v[i];\n        \tmatch_mode = item->attr_match[i].match_mode;\n\nWe could do that API update way before we make the attribute\nsubsystem's implementation thread-safe, and if we did so now,\nthen the caller will not have to change.\n\nThat is what I meant as \"future-proofing\", i.e. future-proofing the\ncallers.\n\nAnd from that point of view, I think 0a5aadcce4 is not an ideal\nplace to stop.  We'd want at least up to 079186123a but probably\neven more, e.g. to 48d93f7f42, I would think.\n"},{"id":"303136","messageId":"xmqqwphp5f9q.fsf@gitster.mtv.corp.google.com","threadId":"44178","inReplyTo":"CAGZ79ka8dO1AHJftKAqD6LvxJSP+8yGGa7Citcdxxrnc5DMeYg@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-03T20:49:53Z","receivedAt":"2016-10-03T20:50:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> I am looking at the tip of jc/attr-more and for a minimum\n> thread safety we'd want to change the call sites to be aware of the\n> threads, i.e. instead of doing\n>\n      static struct git_attr_check *check;\t\n>     if (!check)\n>         check = git_attr_check_initl(\"crlf\", \"ident\",\n>                     \"filter\", \"eol\", \"text\",\n>                     NULL);\n>\n> We'd rather call\n>\n>         struct git_attr_check *check;\n>         check = git_attr_check_lookup_or_initl_threadsafe(\n>                 \"crlf\", \"ident\", \"filter\", \"eol\", \"text\", NULL);\n>          if (!git_check_attr(path, check)) {\n>              ...\n\nI actually am hoping that the \"static\" can be kept so that we can\nminimize the cost of interning these symbols into struct git_attr.\n\nThe initialization would thus become something like:\n\n\tstatic struct git_attr_check *check;\n\tgit_attr_check_initl(&check, \"crlf\", \"ident\", ..., NULL);\n\nThe structure will have an array of git_attr[], once interned will\nbe shared and used by everybody.  _initl() will need to make sure\nthat the \"check\" pointer is updated atomically so that multiple\npeople racing to initialize this variable will not step on each\nothers' toes.\n\nThen the use site would do something like\n\n\tconst char *result[... some size ...];\n\tgit_check_attr(path, check, result[]);\n\nto receive result regardless of anybody else who is using the same\n\"check\" structure to ask the same question (for different paths).\n"},{"id":"303145","messageId":"CAGZ79kYt+Z=ff1b2G+wWRAGGS=je+dpksfmMXj0fWwYVvHk8Cg@mail.gmail.com","threadId":"44178","inReplyTo":"xmqqwphp5f9q.fsf@gitster.mtv.corp.google.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-03T21:49:51Z","receivedAt":"2016-10-03T21:49:57Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 3, 2016 at 1:49 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>> I am looking at the tip of jc/attr-more and for a minimum\n>> thread safety we'd want to change the call sites to be aware of the\n>> threads, i.e. instead of doing\n>>\n>       static struct git_attr_check *check;\n>>     if (!check)\n>>         check = git_attr_check_initl(\"crlf\", \"ident\",\n>>                     \"filter\", \"eol\", \"text\",\n>>                     NULL);\n>>\n>> We'd rather call\n>>\n>>         struct git_attr_check *check;\n>>         check = git_attr_check_lookup_or_initl_threadsafe(\n>>                 \"crlf\", \"ident\", \"filter\", \"eol\", \"text\", NULL);\n>>          if (!git_check_attr(path, check)) {\n>>              ...\n>\n> I actually am hoping that the \"static\" can be kept so that we can\n> minimize the cost of interning these symbols into struct git_attr.\n>\n> The initialization would thus become something like:\n>\n>         static struct git_attr_check *check;\n>         git_attr_check_initl(&check, \"crlf\", \"ident\", ..., NULL);\n>\n> The structure will have an array of git_attr[], once interned will\n> be shared and used by everybody.  _initl() will need to make sure\n> that the \"check\" pointer is updated atomically so that multiple\n> people racing to initialize this variable will not step on each\n> others' toes.\n\nI see and that mutex to protect the first argument of git_attr_check_initl\nis unrelated to the actual pointer; we can use a global mutex for that,\nor rather a static scoped mutex in attr.c, such that all parallel\nracing git_attr_check_initl try to acquire that init_lock and only\none succeeds and that one makes sure to first initialize a git_attr_check\ncompletely and then atomically storing the pointer to the &check location.\n\n>\n> Then the use site would do something like\n>\n>         const char *result[... some size ...];\n\n... some size ... depends on the git_attr_check->check_nr\nand not on the path as I first assumed. So when we still\nwant to go fast here:\n\n    static struct git_attr_check *check;\n\n    /* hard code 2 as it has to hold results for crlf and ident only */\n    const char *results[2];\n\n    if (!check)\n        git_attr_check_initl(&check, \"crlf\", \"ident\", NULL);\n\n    if (nr != check->check_nr)\n        ALLOC_GROW(results, check->check_nr, alloc);\n\n    git_check_attr(path, check, &result);\n    // result[i] contains the result for the i-th element of check\n\n    // Note: git_attr_check_elem seems to be useless now, as the\n    // results are not stored in there, we only make use of the `attr` key.\n"},{"id":"303147","messageId":"xmqqh98t5c69.fsf@gitster.mtv.corp.google.com","threadId":"44178","inReplyTo":"CAGZ79kYt+Z=ff1b2G+wWRAGGS=je+dpksfmMXj0fWwYVvHk8Cg@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-03T21:56:46Z","receivedAt":"2016-10-03T21:56:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>     // Note: git_attr_check_elem seems to be useless now, as the\n>     // results are not stored in there, we only make use of the `attr` key.\n\nI do not think git_attr_check_elem would be visible to the callers,\nonce we split the \"inquiry\" and \"result\" like the code illustrated\nabove.  I actually doubt that the type would even internally need to\nsurvive such a rewrite.\n\nThe point of \"future-proofing\" the callers is to hide such\nimplementation details from them.  We know that the current API will\nneed to be updated at least once to prepare the implementation of\nthe API so that it has some chance of becoming thread-safe, and I\nthink we know enough how the updated API should look like to the\ncallers.  I was hoping the minimum future-proofing would allow us to\nupdate the current \"attr\" API users only once, without having to\nupdate them again when we make it ready to be used in threaded\nenvironment.\n"},{"id":"303152","messageId":"CAGZ79kY+yB-AxRUt0rArGw6DaLDsMZDjKt2jQh9B=P-79T+qYQ@mail.gmail.com","threadId":"44178","inReplyTo":"xmqqh98t5c69.fsf@gitster.mtv.corp.google.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-03T22:17:30Z","receivedAt":"2016-10-03T22:17:35Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 3, 2016 at 2:56 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>>     // Note: git_attr_check_elem seems to be useless now, as the\n>>     // results are not stored in there, we only make use of the `attr` key.\n>\n> I do not think git_attr_check_elem would be visible to the callers,\n> once we split the \"inquiry\" and \"result\" like the code illustrated\n> above.  I actually doubt that the type would even internally need to\n> survive such a rewrite.\n\nSo how would we go about git_all_attrs then?\n\nI think those (only builtin/check-attr.c as well as Documentation) would\nneed to read these names off of git_attr_check_elem.\n\nSo instead we could do a\n\nint git_all_attrs(const char *path, char *result_keys[], char *result_values[],\n                        int nr, int alloc)\n\nInternally git_all_attrs would use nr/alloc to resize result_{key,value} to an\nappropriate size and then fill it with keys/values.\n\nAlthough I do not think check-attr needs to be fast as it is a debugging\ntool rather than a daily used tool, but it would fit into the current\nline of thinking?\n\n\n>\n> The point of \"future-proofing\" the callers is to hide such\n> implementation details from them.  We know that the current API will\n> need to be updated at least once to prepare the implementation of\n> the API so that it has some chance of becoming thread-safe, and I\n> think we know enough how the updated API should look like to the\n> callers.\n\nI don't think we have the same idea how it should look like, as e.g.\nit is unclear what we do with the `const struct git_attr` in the\ngit_attr_check to me.\n\n> I was hoping the minimum future-proofing would allow us to\n> update the current \"attr\" API users only once, without having to\n> update them again when we make it ready to be used in threaded\n> environment.\n\nOk, so let's first define how the future proofed API should look like\nand then we can go towards it.\n"},{"id":"303250","messageId":"xmqqh98sw0tv.fsf@gitster.mtv.corp.google.com","threadId":"44178","inReplyTo":"CAGZ79kY+yB-AxRUt0rArGw6DaLDsMZDjKt2jQh9B=P-79T+qYQ@mail.gmail.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-04T16:11:56Z","receivedAt":"2016-10-04T16:12:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> So how would we go about git_all_attrs then?\n\nI think you arrived the same conclusion, but the use of\ngit_attr_check_elem[] in the original implementation of\ngit_all_attrs() does not translate to the use of struct\ngit_attr_check in the new world order we have been discussing.  The\nlatter is about \"here are the attributes I want to know about\",\ni.e. the question being asked and then we would need to separate out\nthe answer from it to prepare for the threaded future.\n\nThe question git_all_attrs() asks is quite different. The caller\ndoes not even know what set of attributes it is interested in, so if\nyou ever want to allow more than one callers pass the same callsite\nto the function at the same time, they need their own copy of both\n<attr>s and <value>s.  I'd expect they'd ask about a path and then\nreceive an array of (<attr>,<value>).  This is essentially what\nhappens in today's \"array of git_attr_check_elem[]\" interface.\n\nThat is fundamentally different from the normal use of\ngit_check_attr(), where <attr>s are part of questions being asked.\n\n> int git_all_attrs(const char *path, char *result_keys[], char *result_values[],\n>                         int nr, int alloc)\n\nOr wrap each (<attr>,<value>) in a struct and return an array of it.\nI do not see a reason why the above cannot be\n\n\tstruct { const struct git_attr *a; const char *v; }\n\t**git_all_attr(const char *path);\n\nwhich returns an array of struct's with an sentinel element at the\nend (e.g. the 'a' field set to NULL or something).\n"},{"id":"303325","messageId":"CAGZ79kbmSVF42C4FxgC6Bvp0edfq1_XOZKWgwb13JZgug6NsBQ@mail.gmail.com","threadId":"44178","inReplyTo":"xmqqh98sw0tv.fsf@gitster.mtv.corp.google.com","subject":"Re: What's cooking in git.git (Sep 2016, #08; Tue, 27)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-04T22:36:48Z","receivedAt":"2016-10-04T22:36:54Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Oct 4, 2016 at 9:11 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>\n> The question git_all_attrs() asks is quite different.\n\nAnd as far as I can tell it is only used by builtin/check-attr.c which we do not\nintend to convert into a threaded beast in the foreseeable future I'd expect.\n\nSo maybe we can just punt on redefining the API there, i.e.\nwe keep the question and answer together in a struct git_attr_check.\n\n\n<some return value here>\ngit_all_attr(const char *path); /* no other arguments except path*/\n\nThat looks great, and I incorporated that into the proposal I sent out\nas an RFC.\n"}]}