{"thread":{"id":"20872","subject":"[PATCHv5 00/14] git notes","startedAt":"2009-09-08T02:26:48Z","lastAt":"2009-09-12T23:37:49Z","messageCount":58,"participants":["Johan Herland","Junio C Hamano","Johannes Schindelin","Shawn O. Pearce","Sverre Rabbelier","Stephen R. van den Berg","Geert Bosch","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"122645","messageId":"1252376822-6138-1-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":null,"subject":"[PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:48Z","receivedAt":"2009-09-08T02:26:48Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Yet another iteration of the 'git notes' feature. Rebased on top of 'next':\n- Patches 1-9 are unchanged from (patches 1-7, 11-12 of) the last iteration.\n- Patch 10 teaches the notes code to free its data structures on request.\n- Patch 11 introduces the 16-tree notes lookup code that handles SHA1-based\n fanout schemes. This is pretty much unchanged from patch 8 in the previous\n iteration.\n- Patch 12 adds selftests that verify correct parsing of notes trees with\n various SHA1-based fanouts.\n- Patch 13 introduces a flexible parser for a variety of date-based and\n SHA1-based fanout schemes. This is the interesting part, as far as this\n iteration is concerned.\n- Patch 14 adds selftests that verify correct parsing of notes trees with\n various date-based fanouts.\n\nNote that the series does not yet include code for _writing_ notes into a\nsuitably structured notes tree. That will be done in a later iteration.\n\nI have some performance numbers that I will send in a separate email.\n\n\nHave fun! :)\n\n...Johan\n\n\nJohan Herland (9):\n Teach \"-m \" and \"-F \" to \"git notes edit\"\n fast-import: Add support for importing commit notes\n t3302-notes-index-expensive: Speed up create_repo()\n Add flags to get_commit_notes() to control the format of the note string\n Teach notes code to free its internal data structures on request.\n Teach the notes lookup code to parse notes trees with various fanout schemes\n Selftests verifying semantics when loading notes trees with various fanouts\n Allow flexible organization of notes trees, using both commit date and SHA1\n Add test cases for date-based fanouts\n\nJohannes Schindelin (5):\n Introduce commit notes\n Add a script to edit/inspect notes\n Speed up git notes lookup\n Add an expensive test for git-notes\n Add '%N'-format for pretty-printing commit notes\n\n .gitignore | 1 +\n Documentation/config.txt | 13 +\n Documentation/git-fast-import.txt | 45 +++-\n Documentation/git-notes.txt | 60 ++++\n Documentation/pretty-formats.txt | 1 +\n Makefile | 3 +\n cache.h | 4 +\n command-list.txt | 1 +\n commit.c | 1 +\n config.c | 5 +\n environment.c | 1 +\n fast-import.c | 88 +++++-\n git-notes.sh | 121 +++++++\n notes.c | 673 +++++++++++++++++++++++++++++++++++++\n notes.h | 12 +\n pretty.c | 10 +\n t/t3301-notes.sh | 150 ++++++++\n t/t3302-notes-index-expensive.sh | 118 +++++++\n t/t3303-notes-subtrees.sh | 201 +++++++++++\n t/t9300-fast-import.sh | 166 +++++++++\n 20 files changed, 1664 insertions(+), 10 deletions(-)\n create mode 100644 Documentation/git-notes.txt\n create mode 100755 git-notes.sh\n create mode 100644 notes.c\n create mode 100644 notes.h\n create mode 100755 t/t3301-notes.sh\n create mode 100755 t/t3302-notes-index-expensive.sh\n create mode 100755 t/t3303-notes-subtrees.sh\n"},{"id":"122646","messageId":"1252376822-6138-2-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 01/14] Introduce commit notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:49Z","receivedAt":"2009-09-08T02:26:49Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nCommit notes are blobs which are shown together with the commit\nmessage. These blobs are taken from the notes ref, which you can\nconfigure by the config variable core.notesRef, which in turn can\nbe overridden by the environment variable GIT_NOTES_REF.\n\nThe notes ref is a branch which contains \"files\" whose names are\nthe names of the corresponding commits (i.e. the SHA-1).\n\nThe rationale for putting this information into a ref is this: we\nwant to be able to fetch and possibly union-merge the notes,\nmaybe even look at the date when a note was introduced, and we\nwant to store them efficiently together with the other objects.\n\nThis patch has been improved by the following contributions:\n- Thomas Rast: fix core.notesRef documentation\n- Tor Arne Vestbø: fix printing of multi-line notes\n- Alex Riesen: Using char array instead of char pointer costs less BSS\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Thomas Rast \nSigned-off-by: Tor Arne Vestbø \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n Documentation/config.txt | 13 +++++++++\n Makefile | 2 +\n cache.h | 4 +++\n commit.c | 1 +\n config.c | 5 +++\n environment.c | 1 +\n notes.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++\n notes.h | 7 +++++\n pretty.c | 5 +++\n 9 files changed, 106 insertions(+), 0 deletions(-)\n create mode 100644 notes.c\n create mode 100644 notes.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d445395..728d787 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -443,6 +443,19 @@ On some file system/operating system combinations, this is unreliable.\n Set this config setting to 'rename' there; However, This will remove the\n check that makes sure that existing object files will not get overwritten.\n \n+core.notesRef::\n+\tWhen showing commit messages, also show notes which are stored in\n+\tthe given ref. This ref is expected to contain files named\n+\tafter the full SHA-1 of the commit they annotate.\n++\n+If such a file exists in the given ref, the referenced blob is read, and\n+appended to the commit message, separated by a \"Notes:\" line. If the\n+given ref itself does not exist, it is not an error, but means that no\n+notes should be printed.\n++\n+This setting defaults to \"refs/notes/commits\", and can be overridden by\n+the `GIT_NOTES_REF` environment variable.\n+\n add.ignore-errors::\n \tTells 'git-add' to continue adding files when some files cannot be\n \tadded due to indexing errors. Equivalent to the '--ignore-errors'\ndiff --git a/Makefile b/Makefile\nindex ce882d0..37b8f85 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -426,6 +426,7 @@ LIB_H += ll-merge.h\n LIB_H += log-tree.h\n LIB_H += mailmap.h\n LIB_H += merge-recursive.h\n+LIB_H += notes.h\n LIB_H += object.h\n LIB_H += pack.h\n LIB_H += pack-refs.h\n@@ -509,6 +510,7 @@ LIB_OBJS += match-trees.o\n LIB_OBJS += merge-file.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += name-hash.o\n+LIB_OBJS += notes.o\n LIB_OBJS += object.o\n LIB_OBJS += pack-check.o\n LIB_OBJS += pack-refs.o\ndiff --git a/cache.h b/cache.h\nindex ae324c9..917138b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -371,6 +371,8 @@ static inline enum object_type object_type(unsigned int mode)\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n+#define GIT_NOTES_REF_ENVIRONMENT \"GIT_NOTES_REF\"\n+#define GIT_NOTES_DEFAULT_REF \"refs/notes/commits\"\n \n extern int is_bare_repository_cfg;\n extern int is_bare_repository(void);\n@@ -565,6 +567,8 @@ enum object_creation_mode {\n \n extern enum object_creation_mode object_creation_mode;\n \n+extern char *notes_ref_name;\n+\n extern int grafts_replace_parents;\n \n #define GIT_REPO_VERSION 0\ndiff --git a/commit.c b/commit.c\nindex a6c6f70..a0a77a6 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -5,6 +5,7 @@\n #include \"utf8.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n+#include \"notes.h\"\n \n int save_commit_buffer = 1;\n \ndiff --git a/config.c b/config.c\nindex e87edea..70a7d34 100644\n--- a/config.c\n+++ b/config.c\n@@ -467,6 +467,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.notesref\")) {\n+\t\tnotes_ref_name = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.pager\"))\n \t\treturn git_config_string(&pager_program, var, value);\n \ndiff --git a/environment.c b/environment.c\nindex 5de6837..571ab56 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -49,6 +49,7 @@ enum push_default_type push_default = PUSH_DEFAULT_MATCHING;\n #define OBJECT_CREATION_MODE OBJECT_CREATION_USES_HARDLINKS\n #endif\n enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n+char *notes_ref_name;\n int grafts_replace_parents = 1;\n \n /* Parallel index stat data preload? */\ndiff --git a/notes.c b/notes.c\nnew file mode 100644\nindex 0000000..401966d\n--- /dev/null\n+++ b/notes.c\n@@ -0,0 +1,68 @@\n+#include \"cache.h\"\n+#include \"commit.h\"\n+#include \"notes.h\"\n+#include \"refs.h\"\n+#include \"utf8.h\"\n+#include \"strbuf.h\"\n+\n+static int initialized;\n+\n+void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n+\t\tconst char *output_encoding)\n+{\n+\tstatic const char utf8[] = \"utf-8\";\n+\tstruct strbuf name = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\tchar *msg, *msg_p;\n+\tunsigned long linelen, msglen;\n+\tenum object_type type;\n+\n+\tif (!initialized) {\n+\t\tconst char *env = getenv(GIT_NOTES_REF_ENVIRONMENT);\n+\t\tif (env)\n+\t\t\tnotes_ref_name = getenv(GIT_NOTES_REF_ENVIRONMENT);\n+\t\telse if (!notes_ref_name)\n+\t\t\tnotes_ref_name = GIT_NOTES_DEFAULT_REF;\n+\t\tif (notes_ref_name && read_ref(notes_ref_name, sha1))\n+\t\t\tnotes_ref_name = NULL;\n+\t\tinitialized = 1;\n+\t}\n+\n+\tif (!notes_ref_name)\n+\t\treturn;\n+\n+\tstrbuf_addf(&name, \"%s:%s\", notes_ref_name,\n+\t\t\tsha1_to_hex(commit->object.sha1));\n+\tif (get_sha1(name.buf, sha1))\n+\t\treturn;\n+\n+\tif (!(msg = read_sha1_file(sha1, &type, &msglen)) || !msglen ||\n+\t\t\ttype != OBJ_BLOB)\n+\t\treturn;\n+\n+\tif (output_encoding && *output_encoding &&\n+\t\t\tstrcmp(utf8, output_encoding)) {\n+\t\tchar *reencoded = reencode_string(msg, output_encoding, utf8);\n+\t\tif (reencoded) {\n+\t\t\tfree(msg);\n+\t\t\tmsg = reencoded;\n+\t\t\tmsglen = strlen(msg);\n+\t\t}\n+\t}\n+\n+\t/* we will end the annotation by a newline anyway */\n+\tif (msglen && msg[msglen - 1] == '\\n')\n+\t\tmsglen--;\n+\n+\tstrbuf_addstr(sb, \"\\nNotes:\\n\");\n+\n+\tfor (msg_p = msg; msg_p < msg + msglen; msg_p += linelen + 1) {\n+\t\tlinelen = strchrnul(msg_p, '\\n') - msg_p;\n+\n+\t\tstrbuf_addstr(sb, \" \");\n+\t\tstrbuf_add(sb, msg_p, linelen);\n+\t\tstrbuf_addch(sb, '\\n');\n+\t}\n+\n+\tfree(msg);\n+}\ndiff --git a/notes.h b/notes.h\nnew file mode 100644\nindex 0000000..79d21b6\n--- /dev/null\n+++ b/notes.h\n@@ -0,0 +1,7 @@\n+#ifndef NOTES_H\n+#define NOTES_H\n+\n+void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n+\t\tconst char *output_encoding);\n+\n+#endif\ndiff --git a/pretty.c b/pretty.c\nindex f5983f8..e25db81 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -6,6 +6,7 @@\n #include \"string-list.h\"\n #include \"mailmap.h\"\n #include \"log-tree.h\"\n+#include \"notes.h\"\n #include \"color.h\"\n \n static char *user_format;\n@@ -975,5 +976,9 @@ void pretty_print_commit(enum cmit_fmt fmt, const struct commit *commit,\n \t */\n \tif (fmt == CMIT_FMT_EMAIL && sb->len <= beginning_of_body)\n \t\tstrbuf_addch(sb, '\\n');\n+\n+\tif (fmt != CMIT_FMT_ONELINE)\n+\t\tget_commit_notes(commit, sb, encoding);\n+\n \tfree(reencoded);\n }\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122647","messageId":"1252376822-6138-3-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 02/14] Add a script to edit/inspect notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:50Z","receivedAt":"2009-09-08T02:26:50Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nThe script 'git notes' allows you to edit and show commit notes, by\ncalling either\n\n\tgit notes show \n\nor\n\n\tgit notes edit \n\nThis patch has been improved by the following contributions:\n- Tor Arne Vestbø: fix printing of multi-line notes\n- Michael J Gruber: test and handle empty notes gracefully\n- Thomas Rast:\n - only clean up message file when editing\n - use GIT_EDITOR and core.editor over VISUAL/EDITOR\n - t3301: fix confusing quoting in test for valid notes ref\n - t3301: use test_must_fail instead of !\n - refuse to edit notes outside refs/notes/\n- Junio C Hamano: tests: fix \"export var=val\"\n- Christian Couder: documentation: fix 'linkgit' macro in \"git-notes.txt\"\n- Johan Herland: minor cleanup and bugfixing in git-notes.sh (v2)\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Tor Arne Vestbø \nSigned-off-by: Michael J Gruber \nSigned-off-by: Thomas Rast \nSigned-off-by: Christian Couder \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Documentation/git-notes.txt | 46 +++++++++++++++++\n Makefile | 1 +\n command-list.txt | 1 +\n git-notes.sh | 73 +++++++++++++++++++++++++++\n t/t3301-notes.sh | 114 +++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 236 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-notes.txt\n create mode 100755 git-notes.sh\n create mode 100755 t/t3301-notes.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex c446290..703241b 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -86,6 +86,7 @@ git-mktag\n git-mktree\n git-name-rev\n git-mv\n+git-notes\n git-pack-redundant\n git-pack-objects\n git-pack-refs\ndiff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt\nnew file mode 100644\nindex 0000000..7136016\n--- /dev/null\n+++ b/Documentation/git-notes.txt\n@@ -0,0 +1,46 @@\n+git-notes(1)\n+============\n+\n+NAME\n+----\n+git-notes - Add/inspect commit notes\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git-notes' (edit | show) [commit]\n+\n+DESCRIPTION\n+-----------\n+This command allows you to add notes to commit messages, without\n+changing the commit. To discern these notes from the message stored\n+in the commit object, the notes are indented like the message, after\n+an unindented line saying \"Notes:\".\n+\n+To disable commit notes, you have to set the config variable\n+core.notesRef to the empty string. Alternatively, you can set it\n+to a different ref, something like \"refs/notes/bugzilla\". This setting\n+can be overridden by the environment variable \"GIT_NOTES_REF\".\n+\n+\n+SUBCOMMANDS\n+-----------\n+\n+edit::\n+\tEdit the notes for a given commit (defaults to HEAD).\n+\n+show::\n+\tShow the notes for a given commit (defaults to HEAD).\n+\n+\n+Author\n+------\n+Written by Johannes Schindelin \n+\n+Documentation\n+-------------\n+Documentation by Johannes Schindelin\n+\n+GIT\n+---\n+Part of the linkgit:git[7] suite\ndiff --git a/Makefile b/Makefile\nindex 37b8f85..8e8376c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -316,6 +316,7 @@ SCRIPT_SH += git-merge-one-file.sh\n SCRIPT_SH += git-merge-resolve.sh\n SCRIPT_SH += git-mergetool.sh\n SCRIPT_SH += git-mergetool--lib.sh\n+SCRIPT_SH += git-notes.sh\n SCRIPT_SH += git-parse-remote.sh\n SCRIPT_SH += git-pull.sh\n SCRIPT_SH += git-quiltimport.sh\ndiff --git a/command-list.txt b/command-list.txt\nindex fb03a2e..4296941 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -74,6 +74,7 @@ git-mktag plumbingmanipulators\n git-mktree plumbingmanipulators\n git-mv mainporcelain common\n git-name-rev plumbinginterrogators\n+git-notes mainporcelain\n git-pack-objects plumbingmanipulators\n git-pack-redundant plumbinginterrogators\n git-pack-refs ancillarymanipulators\ndiff --git a/git-notes.sh b/git-notes.sh\nnew file mode 100755\nindex 0000000..f06c254\n--- /dev/null\n+++ b/git-notes.sh\n@@ -0,0 +1,73 @@\n+#!/bin/sh\n+\n+USAGE=\"(edit | show) [commit]\"\n+. git-sh-setup\n+\n+test -n \"$3\" && usage\n+\n+test -z \"$1\" && usage\n+ACTION=\"$1\"; shift\n+\n+test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"$(git config core.notesref)\"\n+test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"refs/notes/commits\"\n+\n+COMMIT=$(git rev-parse --verify --default HEAD \"$@\") ||\n+die \"Invalid commit: $@\"\n+\n+case \"$ACTION\" in\n+edit)\n+\tif [ \"${GIT_NOTES_REF#refs/notes/}\" = \"$GIT_NOTES_REF\" ]; then\n+\t\tdie \"Refusing to edit notes in $GIT_NOTES_REF (outside of refs/notes/)\"\n+\tfi\n+\n+\tMSG_FILE=\"$GIT_DIR/new-notes-$COMMIT\"\n+\tGIT_INDEX_FILE=\"$MSG_FILE.idx\"\n+\texport GIT_INDEX_FILE\n+\n+\ttrap '\n+\t\ttest -f \"$MSG_FILE\" && rm \"$MSG_FILE\"\n+\t\ttest -f \"$GIT_INDEX_FILE\" && rm \"$GIT_INDEX_FILE\"\n+\t' 0\n+\n+\tGIT_NOTES_REF= git log -1 $COMMIT | sed \"s/^/#/\" > \"$MSG_FILE\"\n+\n+\tCURRENT_HEAD=$(git show-ref \"$GIT_NOTES_REF\" | cut -f 1 -d ' ')\n+\tif [ -z \"$CURRENT_HEAD\" ]; then\n+\t\tPARENT=\n+\telse\n+\t\tPARENT=\"-p $CURRENT_HEAD\"\n+\t\tgit read-tree \"$GIT_NOTES_REF\" || die \"Could not read index\"\n+\t\tgit cat-file blob :$COMMIT >> \"$MSG_FILE\" 2> /dev/null\n+\tfi\n+\n+\tcore_editor=\"$(git config core.editor)\"\n+\t${GIT_EDITOR:-${core_editor:-${VISUAL:-${EDITOR:-vi}}}} \"$MSG_FILE\"\n+\n+\tgrep -v ^# < \"$MSG_FILE\" | git stripspace > \"$MSG_FILE\".processed\n+\tmv \"$MSG_FILE\".processed \"$MSG_FILE\"\n+\tif [ -s \"$MSG_FILE\" ]; then\n+\t\tBLOB=$(git hash-object -w \"$MSG_FILE\") ||\n+\t\t\tdie \"Could not write into object database\"\n+\t\tgit update-index --add --cacheinfo 0644 $BLOB $COMMIT ||\n+\t\t\tdie \"Could not write index\"\n+\telse\n+\t\ttest -z \"$CURRENT_HEAD\" &&\n+\t\t\tdie \"Will not initialise with empty tree\"\n+\t\tgit update-index --force-remove $COMMIT ||\n+\t\t\tdie \"Could not update index\"\n+\tfi\n+\n+\tTREE=$(git write-tree) || die \"Could not write tree\"\n+\tNEW_HEAD=$(echo Annotate $COMMIT | git commit-tree $TREE $PARENT) ||\n+\t\tdie \"Could not annotate\"\n+\tgit update-ref -m \"Annotate $COMMIT\" \\\n+\t\t\"$GIT_NOTES_REF\" $NEW_HEAD $CURRENT_HEAD\n+;;\n+show)\n+\tgit rev-parse -q --verify \"$GIT_NOTES_REF\":$COMMIT > /dev/null ||\n+\t\tdie \"No note for commit $COMMIT.\"\n+\tgit show \"$GIT_NOTES_REF\":$COMMIT\n+;;\n+*)\n+\tusage\n+esac\ndiff --git a/t/t3301-notes.sh b/t/t3301-notes.sh\nnew file mode 100755\nindex 0000000..73e53be\n--- /dev/null\n+++ b/t/t3301-notes.sh\n@@ -0,0 +1,114 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Johannes E. Schindelin\n+#\n+\n+test_description='Test commit notes'\n+\n+. ./test-lib.sh\n+\n+cat > fake_editor.sh << \\EOF\n+echo \"$MSG\" > \"$1\"\n+echo \"$MSG\" >& 2\n+EOF\n+chmod a+x fake_editor.sh\n+VISUAL=./fake_editor.sh\n+export VISUAL\n+\n+test_expect_success 'cannot annotate non-existing HEAD' '\n+\t(MSG=3 && export MSG && test_must_fail git notes edit)\n+'\n+\n+test_expect_success setup '\n+\t: > a1 &&\n+\tgit add a1 &&\n+\ttest_tick &&\n+\tgit commit -m 1st &&\n+\t: > a2 &&\n+\tgit add a2 &&\n+\ttest_tick &&\n+\tgit commit -m 2nd\n+'\n+\n+test_expect_success 'need valid notes ref' '\n+\t(MSG=1 GIT_NOTES_REF=/ && export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes edit) &&\n+\t(MSG=2 GIT_NOTES_REF=/ && export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes show)\n+'\n+\n+test_expect_success 'refusing to edit in refs/heads/' '\n+\t(MSG=1 GIT_NOTES_REF=refs/heads/bogus &&\n+\t export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes edit)\n+'\n+\n+test_expect_success 'refusing to edit in refs/remotes/' '\n+\t(MSG=1 GIT_NOTES_REF=refs/remotes/bogus &&\n+\t export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes edit)\n+'\n+\n+# 1 indicates caught gracefully by die, 128 means git-show barked\n+test_expect_success 'handle empty notes gracefully' '\n+\tgit notes show ; test 1 = $?\n+'\n+\n+test_expect_success 'create notes' '\n+\tgit config core.notesRef refs/notes/commits &&\n+\tMSG=b1 git notes edit &&\n+\ttest ! -f .git/new-notes &&\n+\ttest 1 = $(git ls-tree refs/notes/commits | wc -l) &&\n+\ttest b1 = $(git notes show) &&\n+\tgit show HEAD^ &&\n+\ttest_must_fail git notes show HEAD^\n+'\n+\n+cat > expect << EOF\n+commit 268048bfb8a1fb38e703baceb8ab235421bf80c5\n+Author: A U Thor \n+Date: Thu Apr 7 15:14:13 2005 -0700\n+\n+ 2nd\n+\n+Notes:\n+ b1\n+EOF\n+\n+test_expect_success 'show notes' '\n+\t! (git cat-file commit HEAD | grep b1) &&\n+\tgit log -1 > output &&\n+\ttest_cmp expect output\n+'\n+test_expect_success 'create multi-line notes (setup)' '\n+\t: > a3 &&\n+\tgit add a3 &&\n+\ttest_tick &&\n+\tgit commit -m 3rd &&\n+\tMSG=\"b3\n+c3c3c3c3\n+d3d3d3\" git notes edit\n+'\n+\n+cat > expect-multiline << EOF\n+commit 1584215f1d29c65e99c6c6848626553fdd07fd75\n+Author: A U Thor \n+Date: Thu Apr 7 15:15:13 2005 -0700\n+\n+ 3rd\n+\n+Notes:\n+ b3\n+ c3c3c3c3\n+ d3d3d3\n+EOF\n+\n+printf \"\\n\" >> expect-multiline\n+cat expect >> expect-multiline\n+\n+test_expect_success 'show multi-line notes' '\n+\tgit log -2 > output &&\n+\ttest_cmp expect-multiline output\n+'\n+\n+test_done\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122653","messageId":"1252376822-6138-4-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 03/14] Speed up git notes lookup","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:51Z","receivedAt":"2009-09-08T02:26:51Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nTo avoid looking up each and every commit in the notes ref's tree\nobject, which is very expensive, speed things up by slurping the tree\nobject's contents into a hash_map.\n\nThe idea for the hashmap singleton is from David Reiss, initial\nbenchmarking by Jeff King.\n\nNote: the implementation allows for arbitrary entries in the notes\ntree object, ignoring those that do not reference a valid object. This\nallows you to annotate arbitrary branches, or objects.\n\nThis patch has been improved by the following contributions:\n- Junio C Hamano: fixed an obvious error in initialize_hash_map()\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n notes.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 102 insertions(+), 10 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 401966d..9172154 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -4,15 +4,112 @@\n #include \"refs.h\"\n #include \"utf8.h\"\n #include \"strbuf.h\"\n+#include \"tree-walk.h\"\n+\n+struct entry {\n+\tunsigned char commit_sha1[20];\n+\tunsigned char notes_sha1[20];\n+};\n+\n+struct hash_map {\n+\tstruct entry *entries;\n+\toff_t count, size;\n+};\n \n static int initialized;\n+static struct hash_map hash_map;\n+\n+static int hash_index(struct hash_map *map, const unsigned char *sha1)\n+{\n+\tint i = ((*(unsigned int *)sha1) % map->size);\n+\n+\tfor (;;) {\n+\t\tunsigned char *current = map->entries[i].commit_sha1;\n+\n+\t\tif (!hashcmp(sha1, current))\n+\t\t\treturn i;\n+\n+\t\tif (is_null_sha1(current))\n+\t\t\treturn -1 - i;\n+\n+\t\tif (++i == map->size)\n+\t\t\ti = 0;\n+\t}\n+}\n+\n+static void add_entry(const unsigned char *commit_sha1,\n+\t\tconst unsigned char *notes_sha1)\n+{\n+\tint index;\n+\n+\tif (hash_map.count + 1 > hash_map.size >> 1) {\n+\t\tint i, old_size = hash_map.size;\n+\t\tstruct entry *old = hash_map.entries;\n+\n+\t\thash_map.size = old_size ? old_size << 1 : 64;\n+\t\thash_map.entries = (struct entry *)\n+\t\t\txcalloc(sizeof(struct entry), hash_map.size);\n+\n+\t\tfor (i = 0; i < old_size; i++)\n+\t\t\tif (!is_null_sha1(old[i].commit_sha1)) {\n+\t\t\t\tindex = -1 - hash_index(&hash_map,\n+\t\t\t\t\t\told[i].commit_sha1);\n+\t\t\t\tmemcpy(hash_map.entries + index, old + i,\n+\t\t\t\t\tsizeof(struct entry));\n+\t\t\t}\n+\t\tfree(old);\n+\t}\n+\n+\tindex = hash_index(&hash_map, commit_sha1);\n+\tif (index < 0) {\n+\t\tindex = -1 - index;\n+\t\thash_map.count++;\n+\t}\n+\n+\thashcpy(hash_map.entries[index].commit_sha1, commit_sha1);\n+\thashcpy(hash_map.entries[index].notes_sha1, notes_sha1);\n+}\n+\n+static void initialize_hash_map(const char *notes_ref_name)\n+{\n+\tunsigned char sha1[20], commit_sha1[20];\n+\tunsigned mode;\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tvoid *buf;\n+\n+\tif (!notes_ref_name || read_ref(notes_ref_name, commit_sha1) ||\n+\t get_tree_entry(commit_sha1, \"\", sha1, &mode))\n+\t\treturn;\n+\n+\tbuf = fill_tree_descriptor(&desc, sha1);\n+\tif (!buf)\n+\t\tdie(\"Could not read %s for notes-index\", sha1_to_hex(sha1));\n+\n+\twhile (tree_entry(&desc, &entry))\n+\t\tif (!get_sha1(entry.path, commit_sha1))\n+\t\t\tadd_entry(commit_sha1, entry.sha1);\n+\tfree(buf);\n+}\n+\n+static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n+{\n+\tint index;\n+\n+\tif (!hash_map.size)\n+\t\treturn NULL;\n+\n+\tindex = hash_index(&hash_map, commit_sha1);\n+\tif (index < 0)\n+\t\treturn NULL;\n+\treturn hash_map.entries[index].notes_sha1;\n+}\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tconst char *output_encoding)\n {\n \tstatic const char utf8[] = \"utf-8\";\n-\tstruct strbuf name = STRBUF_INIT;\n-\tunsigned char sha1[20];\n+\tunsigned char *sha1;\n \tchar *msg, *msg_p;\n \tunsigned long linelen, msglen;\n \tenum object_type type;\n@@ -23,17 +120,12 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\t\tnotes_ref_name = getenv(GIT_NOTES_REF_ENVIRONMENT);\n \t\telse if (!notes_ref_name)\n \t\t\tnotes_ref_name = GIT_NOTES_DEFAULT_REF;\n-\t\tif (notes_ref_name && read_ref(notes_ref_name, sha1))\n-\t\t\tnotes_ref_name = NULL;\n+\t\tinitialize_hash_map(notes_ref_name);\n \t\tinitialized = 1;\n \t}\n \n-\tif (!notes_ref_name)\n-\t\treturn;\n-\n-\tstrbuf_addf(&name, \"%s:%s\", notes_ref_name,\n-\t\t\tsha1_to_hex(commit->object.sha1));\n-\tif (get_sha1(name.buf, sha1))\n+\tsha1 = lookup_notes(commit->object.sha1);\n+\tif (!sha1)\n \t\treturn;\n \n \tif (!(msg = read_sha1_file(sha1, &type, &msglen)) || !msglen ||\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122648","messageId":"1252376822-6138-5-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 04/14] Add an expensive test for git-notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:52Z","receivedAt":"2009-09-08T02:26:52Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\ngit-notes have the potential of being pretty expensive, so test with\na lot of commits. A lot. So to make things cheaper, you have to\nopt-in explicitely, by setting the environment variable\nGIT_NOTES_TIMING_TESTS.\n\nThis patch has been improved by the following contributions:\n- Junio C Hamano: tests: fix \"export var=val\"\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n t/t3302-notes-index-expensive.sh | 98 ++++++++++++++++++++++++++++++++++++++\n 1 files changed, 98 insertions(+), 0 deletions(-)\n create mode 100755 t/t3302-notes-index-expensive.sh\n\ndiff --git a/t/t3302-notes-index-expensive.sh b/t/t3302-notes-index-expensive.sh\nnew file mode 100755\nindex 0000000..0ef3e95\n--- /dev/null\n+++ b/t/t3302-notes-index-expensive.sh\n@@ -0,0 +1,98 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Johannes E. Schindelin\n+#\n+\n+test_description='Test commit notes index (expensive!)'\n+\n+. ./test-lib.sh\n+\n+test -z \"$GIT_NOTES_TIMING_TESTS\" && {\n+\tsay Skipping timing tests\n+\ttest_done\n+\texit\n+}\n+\n+create_repo () {\n+\tnumber_of_commits=$1\n+\tnr=0\n+\tparent=\n+\ttest -d .git || {\n+\tgit init &&\n+\ttree=$(git write-tree) &&\n+\twhile [ $nr -lt $number_of_commits ]; do\n+\t\ttest_tick &&\n+\t\tcommit=$(echo $nr | git commit-tree $tree $parent) ||\n+\t\t\treturn\n+\t\tparent=\"-p $commit\"\n+\t\tnr=$(($nr+1))\n+\tdone &&\n+\tgit update-ref refs/heads/master $commit &&\n+\t{\n+\t\tGIT_INDEX_FILE=.git/temp; export GIT_INDEX_FILE;\n+\t\tgit rev-list HEAD | cat -n | sed \"s/^[ \t][ \t]*/ /g\" |\n+\t\twhile read nr sha1; do\n+\t\t\tblob=$(echo note $nr | git hash-object -w --stdin) &&\n+\t\t\techo $sha1 | sed \"s/^/0644 $blob 0\t/\"\n+\t\tdone | git update-index --index-info &&\n+\t\ttree=$(git write-tree) &&\n+\t\ttest_tick &&\n+\t\tcommit=$(echo notes | git commit-tree $tree) &&\n+\t\tgit update-ref refs/notes/commits $commit\n+\t} &&\n+\tgit config core.notesRef refs/notes/commits\n+\t}\n+}\n+\n+test_notes () {\n+\tcount=$1 &&\n+\tgit config core.notesRef refs/notes/commits &&\n+\tgit log | grep \"^ \" > output &&\n+\ti=1 &&\n+\twhile [ $i -le $count ]; do\n+\t\techo \" $(($count-$i))\" &&\n+\t\techo \" note $i\" &&\n+\t\ti=$(($i+1));\n+\tdone > expect &&\n+\tgit diff expect output\n+}\n+\n+cat > time_notes << \\EOF\n+\tmode=$1\n+\ti=1\n+\twhile [ $i -lt $2 ]; do\n+\t\tcase $1 in\n+\t\tno-notes)\n+\t\t\tGIT_NOTES_REF=non-existing; export GIT_NOTES_REF\n+\t\t;;\n+\t\tnotes)\n+\t\t\tunset GIT_NOTES_REF\n+\t\t;;\n+\t\tesac\n+\t\tgit log >/dev/null\n+\t\ti=$(($i+1))\n+\tdone\n+EOF\n+\n+time_notes () {\n+\tfor mode in no-notes notes\n+\tdo\n+\t\techo $mode\n+\t\t/usr/bin/time sh ../time_notes $mode $1\n+\tdone\n+}\n+\n+for count in 10 100 1000 10000; do\n+\n+\tmkdir $count\n+\t(cd $count;\n+\n+\ttest_expect_success \"setup $count\" \"create_repo $count\"\n+\n+\ttest_expect_success 'notes work' \"test_notes $count\"\n+\n+\ttest_expect_success 'notes timing' \"time_notes 100\"\n+\t)\n+done\n+\n+test_done\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122649","messageId":"1252376822-6138-6-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 05/14] Teach \"-m \" and \"-F \" to \"git notes edit\"","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:53Z","receivedAt":"2009-09-08T02:26:53Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The \"-m\" and \"-F\" options are already the established method\n(in both git-commit and git-tag) to specify a commit/tag message\nwithout invoking the editor. This patch teaches \"git notes edit\"\nto respect the same options for specifying a notes message without\ninvoking the editor.\n\nMultiple \"-m\" and/or \"-F\" options are concatenated as separate\nparagraphs.\n\nThe patch also updates the \"git notes\" documentation and adds\nselftests for the new functionality. Unfortunately, the added\nselftests include a couple of lines with trailing whitespace\n(without these the test will fail). This may cause git to warn\nabout \"whitespace errors\".\n\nThis patch has been improved by the following contributions:\n- Thomas Rast: fix trailing whitespace in t3301\n\nSigned-off-by: Johan Herland \n---\n Documentation/git-notes.txt | 16 ++++++++++-\n git-notes.sh | 64 +++++++++++++++++++++++++++++++++++++-----\n t/t3301-notes.sh | 36 ++++++++++++++++++++++++\n 3 files changed, 107 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt\nindex 7136016..94cceb1 100644\n--- a/Documentation/git-notes.txt\n+++ b/Documentation/git-notes.txt\n@@ -8,7 +8,7 @@ git-notes - Add/inspect commit notes\n SYNOPSIS\n --------\n [verse]\n-'git-notes' (edit | show) [commit]\n+'git-notes' (edit [-F | -m ] | show) [commit]\n \n DESCRIPTION\n -----------\n@@ -33,6 +33,20 @@ show::\n \tShow the notes for a given commit (defaults to HEAD).\n \n \n+OPTIONS\n+-------\n+-m ::\n+\tUse the given note message (instead of prompting).\n+\tIf multiple `-m` (or `-F`) options are given, their\n+\tvalues are concatenated as separate paragraphs.\n+\n+-F ::\n+\tTake the note message from the given file. Use '-' to\n+\tread the note message from the standard input.\n+\tIf multiple `-F` (or `-m`) options are given, their\n+\tvalues are concatenated as separate paragraphs.\n+\n+\n Author\n ------\n Written by Johannes Schindelin \ndiff --git a/git-notes.sh b/git-notes.sh\nindex f06c254..e642e47 100755\n--- a/git-notes.sh\n+++ b/git-notes.sh\n@@ -1,16 +1,59 @@\n #!/bin/sh\n \n-USAGE=\"(edit | show) [commit]\"\n+USAGE=\"(edit [-F | -m ] | show) [commit]\"\n . git-sh-setup\n \n-test -n \"$3\" && usage\n-\n test -z \"$1\" && usage\n ACTION=\"$1\"; shift\n \n test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"$(git config core.notesref)\"\n test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"refs/notes/commits\"\n \n+MESSAGE=\n+while test $# != 0\n+do\n+\tcase \"$1\" in\n+\t-m)\n+\t\ttest \"$ACTION\" = \"edit\" || usage\n+\t\tshift\n+\t\tif test \"$#\" = \"0\"; then\n+\t\t\tdie \"error: option -m needs an argument\"\n+\t\telse\n+\t\t\tif [ -z \"$MESSAGE\" ]; then\n+\t\t\t\tMESSAGE=\"$1\"\n+\t\t\telse\n+\t\t\t\tMESSAGE=\"$MESSAGE\n+\n+$1\"\n+\t\t\tfi\n+\t\t\tshift\n+\t\tfi\n+\t\t;;\n+\t-F)\n+\t\ttest \"$ACTION\" = \"edit\" || usage\n+\t\tshift\n+\t\tif test \"$#\" = \"0\"; then\n+\t\t\tdie \"error: option -F needs an argument\"\n+\t\telse\n+\t\t\tif [ -z \"$MESSAGE\" ]; then\n+\t\t\t\tMESSAGE=\"$(cat \"$1\")\"\n+\t\t\telse\n+\t\t\t\tMESSAGE=\"$MESSAGE\n+\n+$(cat \"$1\")\"\n+\t\t\tfi\n+\t\t\tshift\n+\t\tfi\n+\t\t;;\n+\t-*)\n+\t\tusage\n+\t\t;;\n+\t*)\n+\t\tbreak\n+\t\t;;\n+\tesac\n+done\n+\n COMMIT=$(git rev-parse --verify --default HEAD \"$@\") ||\n die \"Invalid commit: $@\"\n \n@@ -29,19 +72,24 @@ edit)\n \t\ttest -f \"$GIT_INDEX_FILE\" && rm \"$GIT_INDEX_FILE\"\n \t' 0\n \n-\tGIT_NOTES_REF= git log -1 $COMMIT | sed \"s/^/#/\" > \"$MSG_FILE\"\n-\n \tCURRENT_HEAD=$(git show-ref \"$GIT_NOTES_REF\" | cut -f 1 -d ' ')\n \tif [ -z \"$CURRENT_HEAD\" ]; then\n \t\tPARENT=\n \telse\n \t\tPARENT=\"-p $CURRENT_HEAD\"\n \t\tgit read-tree \"$GIT_NOTES_REF\" || die \"Could not read index\"\n-\t\tgit cat-file blob :$COMMIT >> \"$MSG_FILE\" 2> /dev/null\n \tfi\n \n-\tcore_editor=\"$(git config core.editor)\"\n-\t${GIT_EDITOR:-${core_editor:-${VISUAL:-${EDITOR:-vi}}}} \"$MSG_FILE\"\n+\tif [ -z \"$MESSAGE\" ]; then\n+\t\tGIT_NOTES_REF= git log -1 $COMMIT | sed \"s/^/#/\" > \"$MSG_FILE\"\n+\t\tif [ ! -z \"$CURRENT_HEAD\" ]; then\n+\t\t\tgit cat-file blob :$COMMIT >> \"$MSG_FILE\" 2> /dev/null\n+\t\tfi\n+\t\tcore_editor=\"$(git config core.editor)\"\n+\t\t${GIT_EDITOR:-${core_editor:-${VISUAL:-${EDITOR:-vi}}}} \"$MSG_FILE\"\n+\telse\n+\t\techo \"$MESSAGE\" > \"$MSG_FILE\"\n+\tfi\n \n \tgrep -v ^# < \"$MSG_FILE\" | git stripspace > \"$MSG_FILE\".processed\n \tmv \"$MSG_FILE\".processed \"$MSG_FILE\"\ndiff --git a/t/t3301-notes.sh b/t/t3301-notes.sh\nindex 73e53be..1e34f48 100755\n--- a/t/t3301-notes.sh\n+++ b/t/t3301-notes.sh\n@@ -110,5 +110,41 @@ test_expect_success 'show multi-line notes' '\n \tgit log -2 > output &&\n \ttest_cmp expect-multiline output\n '\n+test_expect_success 'create -m and -F notes (setup)' '\n+\t: > a4 &&\n+\tgit add a4 &&\n+\ttest_tick &&\n+\tgit commit -m 4th &&\n+\techo \"xyzzy\" > note5 &&\n+\tgit notes edit -m spam -F note5 -m \"foo\n+bar\n+baz\"\n+'\n+\n+whitespace=\" \"\n+cat > expect-m-and-F << EOF\n+commit 15023535574ded8b1a89052b32673f84cf9582b8\n+Author: A U Thor \n+Date: Thu Apr 7 15:16:13 2005 -0700\n+\n+ 4th\n+\n+Notes:\n+ spam\n+$whitespace\n+ xyzzy\n+$whitespace\n+ foo\n+ bar\n+ baz\n+EOF\n+\n+printf \"\\n\" >> expect-m-and-F\n+cat expect-multiline >> expect-m-and-F\n+\n+test_expect_success 'show -m and -F notes' '\n+\tgit log -3 > output &&\n+\ttest_cmp expect-m-and-F output\n+'\n \n test_done\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122651","messageId":"1252376822-6138-7-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 06/14] fast-import: Add support for importing commit notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:54Z","receivedAt":"2009-09-08T02:26:54Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Introduce a 'notemodify' subcommand of the 'commit' command. This subcommand\nis similar to 'filemodify', except that no mode is supplied (all notes have\nmode 0644), and the path is set to the hex SHA1 of the given \"comittish\".\n\nThis enables fast import of note objects along with their associated commits,\nsince the notes can now be named using the mark references of their\ncorresponding commits.\n\nThe patch also includes a test case of the added functionality.\n\nSigned-off-by: Johan Herland \nAcked-by: Shawn O. Pearce \n---\n Documentation/git-fast-import.txt | 45 +++++++++--\n fast-import.c | 88 +++++++++++++++++++-\n t/t9300-fast-import.sh | 166 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 289 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex f1c94b4..bb198c2 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -325,7 +325,7 @@ change to the project.\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n-\t(filemodify | filedelete | filecopy | filerename | filedeleteall)*\n+\t(filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)*\n \tLF?\n ....\n \n@@ -348,14 +348,13 @@ commit message use a 0 length data. Commit messages are free-form\n and are not interpreted by Git. Currently they must be encoded in\n UTF-8, as fast-import does not permit other encodings to be specified.\n \n-Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`\n-and `filedeleteall` commands\n+Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`,\n+`filedeleteall` and `notemodify` commands\n may be included to update the contents of the branch prior to\n creating the commit. These commands may be supplied in any order.\n However it is recommended that a `filedeleteall` command precede\n-all `filemodify`, `filecopy` and `filerename` commands in the same\n-commit, as `filedeleteall`\n-wipes the branch clean (see below).\n+all `filemodify`, `filecopy`, `filerename` and `notemodify` commands in\n+the same commit, as `filedeleteall` wipes the branch clean (see below).\n \n The `LF` after the command is optional (it used to be required).\n \n@@ -604,6 +603,40 @@ more memory per active branch (less than 1 MiB for even most large\n projects); so frontends that can easily obtain only the affected\n paths for a commit are encouraged to do so.\n \n+`notemodify`\n+^^^^^^^^^^^^\n+Included in a `commit` command to add a new note (annotating a given\n+commit) or change the content of an existing note. This command has\n+two different means of specifying the content of the note.\n+\n+External data format::\n+\tThe data content for the note was already supplied by a prior\n+\t`blob` command. The frontend just needs to connect it to the\n+\tcommit that is to be annotated.\n++\n+....\n+\t'N' SP SP LF\n+....\n++\n+Here `` can be either a mark reference (`:`)\n+set by a prior `blob` command, or a full 40-byte SHA-1 of an\n+existing Git blob object.\n+\n+Inline data format::\n+\tThe data content for the note has not been supplied yet.\n+\tThe frontend wants to supply it as part of this modify\n+\tcommand.\n++\n+....\n+\t'N' SP 'inline' SP LF\n+\tdata\n+....\n++\n+See below for a detailed description of the `data` command.\n+\n+In both formats `` is any of the commit specification\n+expressions also accepted by `from` (see above).\n+\n `mark`\n ~~~~~~\n Arranges for fast-import to save a reference to the current object, allowing\ndiff --git a/fast-import.c b/fast-import.c\nindex dcfb8fa..1e91358 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -22,8 +22,8 @@ Format of STDIN stream:\n ('author' sp name sp '<' email '>' sp when lf)?\n 'committer' sp name sp '<' email '>' sp when lf\n commit_msg\n- ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?\n- ('merge' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)*\n+ ('from' sp committish lf)?\n+ ('merge' sp committish lf)*\n file_change*\n lf?;\n commit_msg ::= data;\n@@ -41,15 +41,18 @@ Format of STDIN stream:\n file_obm ::= 'M' sp mode sp (hexsha1 | idnum) sp path_str lf;\n file_inm ::= 'M' sp mode sp 'inline' sp path_str lf\n data;\n+ note_obm ::= 'N' sp (hexsha1 | idnum) sp committish lf;\n+ note_inm ::= 'N' sp 'inline' sp committish lf\n+ data;\n \n new_tag ::= 'tag' sp tag_str lf\n- 'from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf\n+ 'from' sp committish lf\n ('tagger' sp name sp '<' email '>' sp when lf)?\n tag_msg;\n tag_msg ::= data;\n \n reset_branch ::= 'reset' sp ref_str lf\n- ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?\n+ ('from' sp committish lf)?\n lf?;\n \n checkpoint ::= 'checkpoint' lf\n@@ -88,6 +91,7 @@ Format of STDIN stream:\n # stream formatting is: \\, \" and LF. Otherwise these values\n # are UTF8.\n #\n+ committish ::= (ref_str | hexsha1 | sha1exp_str | idnum);\n ref_str ::= ref;\n sha1exp_str ::= sha1exp;\n tag_str ::= tag;\n@@ -2053,6 +2057,80 @@ static void file_change_cr(struct branch *b, int rename)\n \t\tleaf.tree);\n }\n \n+static void note_change_n(struct branch *b)\n+{\n+\tconst char *p = command_buf.buf + 2;\n+\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstruct object_entry *oe = oe;\n+\tstruct branch *s;\n+\tunsigned char sha1[20], commit_sha1[20];\n+\tuint16_t inline_data = 0;\n+\n+\t/* or 'inline' */\n+\tif (*p == ':') {\n+\t\tchar *x;\n+\t\toe = find_mark(strtoumax(p + 1, &x, 10));\n+\t\thashcpy(sha1, oe->sha1);\n+\t\tp = x;\n+\t} else if (!prefixcmp(p, \"inline\")) {\n+\t\tinline_data = 1;\n+\t\tp += 6;\n+\t} else {\n+\t\tif (get_sha1_hex(p, sha1))\n+\t\t\tdie(\"Invalid SHA1: %s\", command_buf.buf);\n+\t\toe = find_object(sha1);\n+\t\tp += 40;\n+\t}\n+\tif (*p++ != ' ')\n+\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n+\n+\t/* */\n+\ts = lookup_branch(p);\n+\tif (s) {\n+\t\thashcpy(commit_sha1, s->sha1);\n+\t} else if (*p == ':') {\n+\t\tuintmax_t commit_mark = strtoumax(p + 1, NULL, 10);\n+\t\tstruct object_entry *commit_oe = find_mark(commit_mark);\n+\t\tif (commit_oe->type != OBJ_COMMIT)\n+\t\t\tdie(\"Mark :%\" PRIuMAX \" not a commit\", commit_mark);\n+\t\thashcpy(commit_sha1, commit_oe->sha1);\n+\t} else if (!get_sha1(p, commit_sha1)) {\n+\t\tunsigned long size;\n+\t\tchar *buf = read_object_with_reference(commit_sha1,\n+\t\t\tcommit_type, &size, commit_sha1);\n+\t\tif (!buf || size < 46)\n+\t\t\tdie(\"Not a valid commit: %s\", p);\n+\t\tfree(buf);\n+\t} else\n+\t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n+\n+\tif (inline_data) {\n+\t\tstatic struct strbuf buf = STRBUF_INIT;\n+\n+\t\tif (p != uq.buf) {\n+\t\t\tstrbuf_addstr(&uq, p);\n+\t\t\tp = uq.buf;\n+\t\t}\n+\t\tread_next_command();\n+\t\tparse_data(&buf);\n+\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t} else if (oe) {\n+\t\tif (oe->type != OBJ_BLOB)\n+\t\t\tdie(\"Not a blob (actually a %s): %s\",\n+\t\t\t\ttypename(oe->type), command_buf.buf);\n+\t} else {\n+\t\tenum object_type type = sha1_object_info(sha1, NULL);\n+\t\tif (type < 0)\n+\t\t\tdie(\"Blob not found: %s\", command_buf.buf);\n+\t\tif (type != OBJ_BLOB)\n+\t\t\tdie(\"Not a blob (actually a %s): %s\",\n+\t\t\t typename(type), command_buf.buf);\n+\t}\n+\n+\ttree_content_set(&b->branch_tree, sha1_to_hex(commit_sha1), sha1,\n+\t\tS_IFREG | 0644, NULL);\n+}\n+\n static void file_change_deleteall(struct branch *b)\n {\n \trelease_tree_content_recursive(b->branch_tree.tree);\n@@ -2222,6 +2300,8 @@ static void parse_new_commit(void)\n \t\t\tfile_change_cr(b, 1);\n \t\telse if (!prefixcmp(command_buf.buf, \"C \"))\n \t\t\tfile_change_cr(b, 0);\n+\t\telse if (!prefixcmp(command_buf.buf, \"N \"))\n+\t\t\tnote_change_n(b);\n \t\telse if (!strcmp(\"deleteall\", command_buf.buf))\n \t\t\tfile_change_deleteall(b);\n \t\telse {\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex d33fc55..2f5c323 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1089,6 +1089,172 @@ test_expect_success 'P: fail on blob mark in gitlink' '\n test_must_fail git fast-import input < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data <expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+first (:3)\n+EOF\n+test_expect_success \\\n+\t'Q: verify first commit' \\\n+\t'git cat-file commit notes-test~2 | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+second (:5)\n+EOF\n+test_expect_success \\\n+\t'Q: verify second commit' \\\n+\t'git cat-file commit notes-test^ | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+third (:6)\n+EOF\n+test_expect_success \\\n+\t'Q: verify third commit' \\\n+\t'git cat-file commit notes-test | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+notes (:9)\n+EOF\n+test_expect_success \\\n+\t'Q: verify notes commit' \\\n+\t'git cat-file commit refs/notes/foobar | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect.unsorted <expect\n+test_expect_success \\\n+\t'Q: verify notes tree' \\\n+\t'git cat-file -p refs/notes/foobar^{tree} | sed \"s/ [0-9a-f]*\t/ /\" >actual &&\n+\t test_cmp expect actual'\n+\n+echo \"$note1_data\" >expect\n+test_expect_success \\\n+\t'Q: verify note for first commit' \\\n+\t'git cat-file blob refs/notes/foobar:$commit1 >actual && test_cmp expect actual'\n+\n+echo \"$note2_data\" >expect\n+test_expect_success \\\n+\t'Q: verify note for second commit' \\\n+\t'git cat-file blob refs/notes/foobar:$commit2 >actual && test_cmp expect actual'\n+\n+echo \"$note3_data\" >expect\n+test_expect_success \\\n+\t'Q: verify note for third commit' \\\n+\t'git cat-file blob refs/notes/foobar:$commit3 >actual && test_cmp expect actual'\n+\n+###\n ### series R (feature and option)\n ###\n \n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122650","messageId":"1252376822-6138-8-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 07/14] t3302-notes-index-expensive: Speed up create_repo()","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:55Z","receivedAt":"2009-09-08T02:26:55Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Creating repos with 10/100/1000/10000 commits and notes takes a lot of time.\nHowever, using git-fast-import to do the job is a lot more efficient than\nusing plumbing commands to do the same.\n\nThis patch decreases the overall run-time of this test on my machine from\n~3 to ~1 minutes.\n\nSigned-off-by: Johan Herland \nAcked-by: Johannes Schindelin \n---\n t/t3302-notes-index-expensive.sh | 74 ++++++++++++++++++++++++--------------\n 1 files changed, 47 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t3302-notes-index-expensive.sh b/t/t3302-notes-index-expensive.sh\nindex 0ef3e95..ee84fc4 100755\n--- a/t/t3302-notes-index-expensive.sh\n+++ b/t/t3302-notes-index-expensive.sh\n@@ -16,30 +16,50 @@ test -z \"$GIT_NOTES_TIMING_TESTS\" && {\n create_repo () {\n \tnumber_of_commits=$1\n \tnr=0\n-\tparent=\n \ttest -d .git || {\n \tgit init &&\n-\ttree=$(git write-tree) &&\n-\twhile [ $nr -lt $number_of_commits ]; do\n-\t\ttest_tick &&\n-\t\tcommit=$(echo $nr | git commit-tree $tree $parent) ||\n-\t\t\treturn\n-\t\tparent=\"-p $commit\"\n-\t\tnr=$(($nr+1))\n-\tdone &&\n-\tgit update-ref refs/heads/master $commit &&\n-\t{\n-\t\tGIT_INDEX_FILE=.git/temp; export GIT_INDEX_FILE;\n-\t\tgit rev-list HEAD | cat -n | sed \"s/^[ \t][ \t]*/ /g\" |\n-\t\twhile read nr sha1; do\n-\t\t\tblob=$(echo note $nr | git hash-object -w --stdin) &&\n-\t\t\techo $sha1 | sed \"s/^/0644 $blob 0\t/\"\n-\t\tdone | git update-index --index-info &&\n-\t\ttree=$(git write-tree) &&\n+\t(\n+\t\twhile [ $nr -lt $number_of_commits ]; do\n+\t\t\tnr=$(($nr+1))\n+\t\t\tmark=$(($nr+$nr))\n+\t\t\tnotemark=$(($mark+1))\n+\t\t\ttest_tick &&\n+\t\t\tcat < $GIT_COMMITTER_DATE\n+data <> note_commit\n+\t\tdone &&\n \t\ttest_tick &&\n-\t\tcommit=$(echo notes | git commit-tree $tree) &&\n-\t\tgit update-ref refs/notes/commits $commit\n-\t} &&\n+\t\tcat < $GIT_COMMITTER_DATE\n+data < output &&\n-\ti=1 &&\n-\twhile [ $i -le $count ]; do\n-\t\techo \" $(($count-$i))\" &&\n-\t\techo \" note $i\" &&\n-\t\ti=$(($i+1));\n+\ti=$count &&\n+\twhile [ $i -gt 0 ]; do\n+\t\techo \" commit #$i\" &&\n+\t\techo \" note for commit #$i\" &&\n+\t\ti=$(($i-1));\n \tdone > expect &&\n-\tgit diff expect output\n+\ttest_cmp expect output\n }\n \n cat > time_notes << \\EOF\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122652","messageId":"1252376822-6138-9-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 08/14] Add flags to get_commit_notes() to control the format of the note string","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:56Z","receivedAt":"2009-09-08T02:26:56Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"This patch adds the following flags to get_commit_notes() for adjusting the\nformat of the produced note string:\n- NOTES_SHOW_HEADER: Print \"Notes:\" line before the notes contents\n- NOTES_INDENT: Indent notes contents by 4 spaces\n\nSuggested-by: Johannes Schindelin \nSigned-off-by: Johan Herland \n---\n notes.c | 8 +++++---\n notes.h | 5 ++++-\n pretty.c | 3 ++-\n 3 files changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 9172154..84c30c1 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -106,7 +106,7 @@ static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n }\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n-\t\tconst char *output_encoding)\n+\t\tconst char *output_encoding, int flags)\n {\n \tstatic const char utf8[] = \"utf-8\";\n \tunsigned char *sha1;\n@@ -146,12 +146,14 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \tif (msglen && msg[msglen - 1] == '\\n')\n \t\tmsglen--;\n \n-\tstrbuf_addstr(sb, \"\\nNotes:\\n\");\n+\tif (flags & NOTES_SHOW_HEADER)\n+\t\tstrbuf_addstr(sb, \"\\nNotes:\\n\");\n \n \tfor (msg_p = msg; msg_p < msg + msglen; msg_p += linelen + 1) {\n \t\tlinelen = strchrnul(msg_p, '\\n') - msg_p;\n \n-\t\tstrbuf_addstr(sb, \" \");\n+\t\tif (flags & NOTES_INDENT)\n+\t\t\tstrbuf_addstr(sb, \" \");\n \t\tstrbuf_add(sb, msg_p, linelen);\n \t\tstrbuf_addch(sb, '\\n');\n \t}\ndiff --git a/notes.h b/notes.h\nindex 79d21b6..7f3eed4 100644\n--- a/notes.h\n+++ b/notes.h\n@@ -1,7 +1,10 @@\n #ifndef NOTES_H\n #define NOTES_H\n \n+#define NOTES_SHOW_HEADER 1\n+#define NOTES_INDENT 2\n+\n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n-\t\tconst char *output_encoding);\n+\t\tconst char *output_encoding, int flags);\n \n #endif\ndiff --git a/pretty.c b/pretty.c\nindex e25db81..01eadd0 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -978,7 +978,8 @@ void pretty_print_commit(enum cmit_fmt fmt, const struct commit *commit,\n \t\tstrbuf_addch(sb, '\\n');\n \n \tif (fmt != CMIT_FMT_ONELINE)\n-\t\tget_commit_notes(commit, sb, encoding);\n+\t\tget_commit_notes(commit, sb, encoding,\n+\t\t\t\t NOTES_SHOW_HEADER | NOTES_INDENT);\n \n \tfree(reencoded);\n }\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122658","messageId":"1252376822-6138-10-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 09/14] Add '%N'-format for pretty-printing commit notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:57Z","receivedAt":"2009-09-08T02:26:57Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nSigned-off-by: Johan Herland \n---\n Documentation/pretty-formats.txt | 1 +\n pretty.c | 4 ++++\n 2 files changed, 5 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/pretty-formats.txt b/Documentation/pretty-formats.txt\nindex 2a845b1..5fb10b3 100644\n--- a/Documentation/pretty-formats.txt\n+++ b/Documentation/pretty-formats.txt\n@@ -123,6 +123,7 @@ The placeholders are:\n - '%s': subject\n - '%f': sanitized subject line, suitable for a filename\n - '%b': body\n+- '%N': commit notes\n - '%Cred': switch color to red\n - '%Cgreen': switch color to green\n - '%Cblue': switch color to blue\ndiff --git a/pretty.c b/pretty.c\nindex 01eadd0..7f350bb 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -702,6 +702,10 @@ static size_t format_commit_item(struct strbuf *sb, const char *placeholder,\n \tcase 'd':\n \t\tformat_decoration(sb, commit);\n \t\treturn 1;\n+\tcase 'N':\n+\t\tget_commit_notes(commit, sb, git_log_output_encoding ?\n+\t\t\t git_log_output_encoding : git_commit_encoding, 0);\n+\t\treturn 1;\n \t}\n \n \t/* For the rest we have to parse the commit header. */\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122659","messageId":"1252376822-6138-11-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 10/14] Teach notes code to free its internal data structures on request.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:58Z","receivedAt":"2009-09-08T02:26:58Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"There's no need to be rude to memory-concious callers...\n\nSigned-off-by: Johan Herland \n---\n notes.c | 7 +++++++\n notes.h | 2 ++\n 2 files changed, 9 insertions(+), 0 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 84c30c1..008c3d4 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -160,3 +160,10 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \n \tfree(msg);\n }\n+\n+void free_commit_notes()\n+{\n+\tfree(hash_map.entries);\n+\tmemset(&hash_map, 0, sizeof(struct hash_map));\n+\tinitialized = 0;\n+}\ndiff --git a/notes.h b/notes.h\nindex 7f3eed4..41802e5 100644\n--- a/notes.h\n+++ b/notes.h\n@@ -7,4 +7,6 @@\n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tconst char *output_encoding, int flags);\n \n+void free_commit_notes();\n+\n #endif\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122655","messageId":"1252376822-6138-12-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 11/14] Teach the notes lookup code to parse notes trees with various fanout schemes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:26:59Z","receivedAt":"2009-09-08T02:26:59Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The semantics used when parsing notes trees (with regards to fanout subtrees)\nfollow Dscho's proposal fairly closely:\n- No concatenation/merging of notes is performed. If there are several notes\n objects referencing a given commit, only one of those objects are used.\n- If a notes object for a given commit is present in the \"root\" notes tree,\n no subtrees are consulted; the object in the root tree is used directly.\n- If there are more than one subtree that prefix-matches the given commit,\n only the subtree with the longest matching prefix is consulted. This\n means that if the given commit is e.g. \"deadbeef\", and the notes tree have\n subtrees \"de\" and \"dead\", then the following paths in the notes tree are\n searched: \"deadbeef\", \"dead/beef\". Note that \"de/adbeef\" is NOT searched.\n- Fanout directories (subtrees) must references a whole number of bytes\n from the SHA1 sum they subdivide. E.g. subtrees \"dead\" and \"de\" are\n acceptable; \"d\" and \"dea\" are not.\n- Multiple levels of fanout are allowed. All the above rules apply\n recursively. E.g. \"de/adbeef\" is preferred over \"de/adbe/ef\", etc.\n\nThis patch changes the in-memory datastructure for holding parsed notes:\nInstead of holding all note (and subtree) entries in a hash table, a\nsimple 16-tree structure is used instead. The tree structure consists of\n16-arrays as internal nodes, and note/subtree entries as leaf nodes. The\ntree is traversed by indexing subsequent nibbles of the search key until\na leaf node is encountered. If a subtree entry is encountered while\nsearching for a note, the subtree is unpacked into the 16-tree structure,\nand the search continues into that subtree.\n\nThe new algorithm performs significantly better in the cases where only\na fraction of the notes need to be looked up (this is assumed to be the\ncommon case for notes lookup). The new code even performs marginally\nbetter in the worst case (where _all_ the notes are looked up).\n\nIn addition to this, comes the massive performance win associated with\norganizing the notes tree according to some fanout scheme. Even a simple\n2/38 fanout scheme is dramatically quicker to traverse (going from tens of\nseconds to sub-second runtimes).\n\nAs for memory usage, the new code is marginally better than the old code in\nthe worst case, but in the case of looking up only some notes from a notes\ntree with proper fanout, the new code uses only a small fraction of the\nmemory needed to hold the entire notes tree.\n\nHowever, there is one casualty of this patch. The old notes lookup code was\nable to parse notes that were associated with non-SHA1s (e.g. refs). The new\ncode requires the referenced object to be named by a SHA1 sum. Still, this\nis not considered a major setback, since the notes infrastructure was not\noriginally intended to annotate objects outside the Git object database.\n\nSigned-off-by: Johan Herland \n---\n notes.c | 317 +++++++++++++++++++++++++++++++++++++++++++++++++--------------\n 1 files changed, 248 insertions(+), 69 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 008c3d4..6926aa6 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -6,103 +6,282 @@\n #include \"strbuf.h\"\n #include \"tree-walk.h\"\n\n-struct entry {\n-\tunsigned char commit_sha1[20];\n-\tunsigned char notes_sha1[20];\n+/*\n+ * Use a non-balancing simple 16-tree structure with struct int_node as\n+ * internal nodes, and struct leaf_node as leaf nodes. Each int_node has a\n+ * 16-array of pointers to its children.\n+ * The bottom 2 bits of each pointer is used to identify the pointer type\n+ * - ptr & 3 == 0 - NULL pointer, assert(ptr == NULL)\n+ * - ptr & 3 == 1 - pointer to next internal node - cast to struct int_node *\n+ * - ptr & 3 == 2 - pointer to note entry - cast to struct leaf_node *\n+ * - ptr & 3 == 3 - pointer to subtree entry - cast to struct leaf_node *\n+ *\n+ * The root node is a statically allocated struct int_node.\n+ */\n+struct int_node {\n+\tvoid *a[16];\n };\n\n-struct hash_map {\n-\tstruct entry *entries;\n-\toff_t count, size;\n+/*\n+ * Leaf nodes come in two variants, note entries and subtree entries,\n+ * distinguished by the LSb of the leaf node pointer (see above).\n+ * As a note entry, the key is the SHA1 of the referenced commit, and the\n+ * value is the SHA1 of the note object.\n+ * As a subtree entry, the key is the prefix SHA1 (w/trailing NULs) of the\n+ * referenced commit, using the last byte of the key to store the length of\n+ * the prefix. The value is the SHA1 of the tree object containing the notes\n+ * subtree.\n+ */\n+struct leaf_node {\n+\tunsigned char key_sha1[20];\n+\tunsigned char val_sha1[20];\n };\n\n-static int initialized;\n-static struct hash_map hash_map;\n+#define PTR_TYPE_NULL 0\n+#define PTR_TYPE_INTERNAL 1\n+#define PTR_TYPE_NOTE 2\n+#define PTR_TYPE_SUBTREE 3\n\n-static int hash_index(struct hash_map *map, const unsigned char *sha1)\n-{\n-\tint i = ((*(unsigned int *)sha1) % map->size);\n+#define GET_PTR_TYPE(ptr) ((uintptr_t) (ptr) & 3)\n+#define CLR_PTR_TYPE(ptr) ((void *) ((uintptr_t) (ptr) & ~3))\n+#define SET_PTR_TYPE(ptr, type) ((void *) ((uintptr_t) (ptr) | (type)))\n\n-\tfor (;;) {\n-\t\tunsigned char *current = map->entries[i].commit_sha1;\n+#define GET_NIBBLE(n, sha1) (((sha1[n >> 1]) >> ((n & 0x01) << 2)) & 0x0f)\n\n-\t\tif (!hashcmp(sha1, current))\n-\t\t\treturn i;\n+#define SUBTREE_SHA1_PREFIXCMP(key_sha1, subtree_sha1) \\\n+\t(memcmp(key_sha1, subtree_sha1, subtree_sha1[19]))\n\n-\t\tif (is_null_sha1(current))\n-\t\t\treturn -1 - i;\n+static struct int_node root_node;\n\n-\t\tif (++i == map->size)\n-\t\t\ti = 0;\n+static int initialized;\n+\n+static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n+\t\tunsigned int n);\n+\n+/*\n+ * To find a leaf_node:\n+ * 1. Start at the root node, with n = 0\n+ * 2. Use the nth nibble of the key as an index into a:\n+ * - If a[n] is an int_node, recurse into that node and increment n\n+ * - If a leaf_node with matching key, return leaf_node (assert note entry)\n+ * - If a matching subtree entry, unpack that subtree entry (and remove it);\n+ * restart search at the current level.\n+ * - Otherwise, we end up at a NULL pointer, or a non-matching leaf_node.\n+ * Backtrack out of the recursion, one level at a time and check a[0]:\n+ * - If a[0] at the current level is a matching subtree entry, unpack that\n+ * subtree entry (and remove it); restart search at the current level.\n+ */\n+static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n+\t\tconst unsigned char *key_sha1)\n+{\n+\tstruct leaf_node *l;\n+\tunsigned char i = GET_NIBBLE(n, key_sha1);\n+\tvoid *p = tree->a[i];\n+\n+\tswitch(GET_PTR_TYPE(p)) {\n+\tcase PTR_TYPE_INTERNAL:\n+\t\tl = note_tree_find(CLR_PTR_TYPE(p), n + 1, key_sha1);\n+\t\tif (l)\n+\t\t\treturn l;\n+\t\tbreak;\n+\tcase PTR_TYPE_NOTE:\n+\t\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n+\t\tif (!hashcmp(key_sha1, l->key_sha1))\n+\t\t\treturn l; /* return note object matching given key */\n+\t\tbreak;\n+\tcase PTR_TYPE_SUBTREE:\n+\t\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n+\t\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n+\t\t\t/* unpack tree and resume search */\n+\t\t\ttree->a[i] = NULL;\n+\t\t\tload_subtree(l, tree, n);\n+\t\t\tfree(l);\n+\t\t\treturn note_tree_find(tree, n, key_sha1);\n+\t\t}\n+\t\tbreak;\n+\tcase PTR_TYPE_NULL:\n+\tdefault:\n+\t\tassert(!p);\n+\t\tbreak;\n \t}\n+\n+\t/*\n+\t * Did not find key at this (or any lower) level.\n+\t * Check if there's a matching subtree entry in tree->a[0].\n+\t * If so, unpack tree and resume search.\n+\t */\n+\tp = tree->a[0];\n+\tif (GET_PTR_TYPE(p) != PTR_TYPE_SUBTREE)\n+\t\treturn NULL;\n+\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n+\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n+\t\t/* unpack tree and resume search */\n+\t\ttree->a[0] = NULL;\n+\t\tload_subtree(l, tree, n);\n+\t\tfree(l);\n+\t\treturn note_tree_find(tree, n, key_sha1);\n+\t}\n+\treturn NULL;\n }\n\n-static void add_entry(const unsigned char *commit_sha1,\n-\t\tconst unsigned char *notes_sha1)\n+/*\n+ * To insert a leaf_node:\n+ * 1. Start at the root node, with n = 0\n+ * 2. Use the nth nibble of the key as an index into a:\n+ * - If a[n] is NULL, store the tweaked pointer directly into a[n]\n+ * - If a[n] is an int_node, recurse into that node and increment n\n+ * - If a[n] is a leaf_node:\n+ * 1. Check if they're equal, and handle that (abort? overwrite?)\n+ * 2. Create a new int_node, and store both leaf_nodes there\n+ * 3. Store the new int_node into a[n].\n+ */\n+static int note_tree_insert(struct int_node *tree, unsigned char n,\n+\t\tconst struct leaf_node *entry, unsigned char type)\n {\n-\tint index;\n-\n-\tif (hash_map.count + 1 > hash_map.size >> 1) {\n-\t\tint i, old_size = hash_map.size;\n-\t\tstruct entry *old = hash_map.entries;\n-\n-\t\thash_map.size = old_size ? old_size << 1 : 64;\n-\t\thash_map.entries = (struct entry *)\n-\t\t\txcalloc(sizeof(struct entry), hash_map.size);\n-\n-\t\tfor (i = 0; i < old_size; i++)\n-\t\t\tif (!is_null_sha1(old[i].commit_sha1)) {\n-\t\t\t\tindex = -1 - hash_index(&hash_map,\n-\t\t\t\t\t\told[i].commit_sha1);\n-\t\t\t\tmemcpy(hash_map.entries + index, old + i,\n-\t\t\t\t\tsizeof(struct entry));\n-\t\t\t}\n-\t\tfree(old);\n+\tstruct int_node *new_node;\n+\tconst struct leaf_node *l;\n+\tint ret;\n+\tunsigned char i = GET_NIBBLE(n, entry->key_sha1);\n+\tvoid *p = tree->a[i];\n+\tassert(GET_PTR_TYPE(entry) == PTR_TYPE_NULL);\n+\tswitch(GET_PTR_TYPE(p)) {\n+\tcase PTR_TYPE_NULL:\n+\t\tassert(!p);\n+\t\ttree->a[i] = SET_PTR_TYPE(entry, type);\n+\t\treturn 0;\n+\tcase PTR_TYPE_INTERNAL:\n+\t\treturn note_tree_insert(CLR_PTR_TYPE(p), n + 1, entry, type);\n+\tdefault:\n+\t\tassert(GET_PTR_TYPE(p) == PTR_TYPE_NOTE ||\n+\t\t\tGET_PTR_TYPE(p) == PTR_TYPE_SUBTREE);\n+\t\tl = (const struct leaf_node *) CLR_PTR_TYPE(p);\n+\t\tif (!hashcmp(entry->key_sha1, l->key_sha1))\n+\t\t\treturn -1; /* abort insert on matching key */\n+\t\tnew_node = (struct int_node *)\n+\t\t\txcalloc(sizeof(struct int_node), 1);\n+\t\tret = note_tree_insert(new_node, n + 1,\n+\t\t\tCLR_PTR_TYPE(p), GET_PTR_TYPE(p));\n+\t\tif (ret) {\n+\t\t\tfree(new_node);\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttree->a[i] = SET_PTR_TYPE(new_node, PTR_TYPE_INTERNAL);\n+\t\treturn note_tree_insert(new_node, n + 1, entry, type);\n \t}\n+}\n\n-\tindex = hash_index(&hash_map, commit_sha1);\n-\tif (index < 0) {\n-\t\tindex = -1 - index;\n-\t\thash_map.count++;\n+/* Free the entire notes data contained in the given tree */\n+static void note_tree_free(struct int_node *tree)\n+{\n+\tunsigned int i;\n+\tfor (i = 0; i < 16; i++) {\n+\t\tvoid *p = tree->a[i];\n+\t\tswitch(GET_PTR_TYPE(p)) {\n+\t\tcase PTR_TYPE_INTERNAL:\n+\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n+\t\t\t/* fall through */\n+\t\tcase PTR_TYPE_NOTE:\n+\t\tcase PTR_TYPE_SUBTREE:\n+\t\t\tfree(CLR_PTR_TYPE(p));\n+\t\t}\n \t}\n+}\n\n-\thashcpy(hash_map.entries[index].commit_sha1, commit_sha1);\n-\thashcpy(hash_map.entries[index].notes_sha1, notes_sha1);\n+/*\n+ * Convert a partial SHA1 hex string to the corresponding partial SHA1 value.\n+ * - hex - Partial SHA1 segment in ASCII hex format\n+ * - hex_len - Length of above segment. Must be multiple of 2 between 0 and 40\n+ * - sha1 - Partial SHA1 value is written here\n+ * - sha1_len - Max #bytes to store in sha1, Must be >= hex_len / 2, and < 20\n+ * Returns -1 on error (invalid arguments or invalid SHA1 (not in hex format).\n+ * Otherwise, returns number of bytes written to sha1 (i.e. hex_len / 2).\n+ * Pads sha1 with NULs up to sha1_len (not included in returned length).\n+ */\n+static int get_sha1_hex_segment(const char *hex, unsigned int hex_len,\n+\t\tunsigned char *sha1, unsigned int sha1_len)\n+{\n+\tunsigned int i, len = hex_len >> 1;\n+\tif (hex_len % 2 != 0 || len > sha1_len)\n+\t\treturn -1;\n+\tfor (i = 0; i < len; i++) {\n+\t\tunsigned int val = (hexval(hex[0]) << 4) | hexval(hex[1]);\n+\t\tif (val & ~0xff)\n+\t\t\treturn -1;\n+\t\t*sha1++ = val;\n+\t\thex += 2;\n+\t}\n+\tfor (; i < sha1_len; i++)\n+\t\t*sha1++ = 0;\n+\treturn len;\n }\n\n-static void initialize_hash_map(const char *notes_ref_name)\n+static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n+\t\tunsigned int n)\n {\n-\tunsigned char sha1[20], commit_sha1[20];\n-\tunsigned mode;\n+\tunsigned char commit_sha1[20];\n+\tunsigned int prefix_len;\n+\tint status;\n+\tvoid *buf;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n-\tvoid *buf;\n+\n+\tbuf = fill_tree_descriptor(&desc, subtree->val_sha1);\n+\tif (!buf)\n+\t\tdie(\"Could not read %s for notes-index\",\n+\t\t sha1_to_hex(subtree->val_sha1));\n+\n+\tprefix_len = subtree->key_sha1[19];\n+\tassert(prefix_len * 2 >= n);\n+\tmemcpy(commit_sha1, subtree->key_sha1, prefix_len);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tint len = get_sha1_hex_segment(entry.path, strlen(entry.path),\n+\t\t\t\tcommit_sha1 + prefix_len, 20 - prefix_len);\n+\t\tif (len < 0)\n+\t\t\tcontinue; /* entry.path is not a SHA1 sum. Skip */\n+\t\tlen += prefix_len;\n+\n+\t\t/*\n+\t\t * If commit SHA1 is complete (len == 20), assume note object\n+\t\t * If commit SHA1 is incomplete (len < 20), assume note subtree\n+\t\t */\n+\t\tif (len <= 20) {\n+\t\t\tunsigned char type = PTR_TYPE_NOTE;\n+\t\t\tstruct leaf_node *l = (struct leaf_node *)\n+\t\t\t\txcalloc(sizeof(struct leaf_node), 1);\n+\t\t\thashcpy(l->key_sha1, commit_sha1);\n+\t\t\thashcpy(l->val_sha1, entry.sha1);\n+\t\t\tif (len < 20) {\n+\t\t\t\tl->key_sha1[19] = (unsigned char) len;\n+\t\t\t\ttype = PTR_TYPE_SUBTREE;\n+\t\t\t}\n+\t\t\tstatus = note_tree_insert(node, n, l, type);\n+\t\t\tassert(!status);\n+\t\t}\n+\t}\n+\tfree(buf);\n+}\n+\n+static void initialize_notes(const char *notes_ref_name)\n+{\n+\tunsigned char sha1[20], commit_sha1[20];\n+\tunsigned mode;\n+\tstruct leaf_node root_tree;\n\n \tif (!notes_ref_name || read_ref(notes_ref_name, commit_sha1) ||\n \t get_tree_entry(commit_sha1, \"\", sha1, &mode))\n \t\treturn;\n\n-\tbuf = fill_tree_descriptor(&desc, sha1);\n-\tif (!buf)\n-\t\tdie(\"Could not read %s for notes-index\", sha1_to_hex(sha1));\n-\n-\twhile (tree_entry(&desc, &entry))\n-\t\tif (!get_sha1(entry.path, commit_sha1))\n-\t\t\tadd_entry(commit_sha1, entry.sha1);\n-\tfree(buf);\n+\thashclr(root_tree.key_sha1);\n+\thashcpy(root_tree.val_sha1, sha1);\n+\tload_subtree(&root_tree, &root_node, 0);\n }\n\n static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n {\n-\tint index;\n-\n-\tif (!hash_map.size)\n-\t\treturn NULL;\n-\n-\tindex = hash_index(&hash_map, commit_sha1);\n-\tif (index < 0)\n-\t\treturn NULL;\n-\treturn hash_map.entries[index].notes_sha1;\n+\tstruct leaf_node *found = note_tree_find(&root_node, 0, commit_sha1);\n+\tif (found)\n+\t\treturn found->val_sha1;\n+\treturn NULL;\n }\n\n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n@@ -120,7 +299,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\t\tnotes_ref_name = getenv(GIT_NOTES_REF_ENVIRONMENT);\n \t\telse if (!notes_ref_name)\n \t\t\tnotes_ref_name = GIT_NOTES_DEFAULT_REF;\n-\t\tinitialize_hash_map(notes_ref_name);\n+\t\tinitialize_notes(notes_ref_name);\n \t\tinitialized = 1;\n \t}\n\n@@ -163,7 +342,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n\n void free_commit_notes()\n {\n-\tfree(hash_map.entries);\n-\tmemset(&hash_map, 0, sizeof(struct hash_map));\n+\tnote_tree_free(&root_node);\n+\tmemset(&root_node, 0, sizeof(struct int_node));\n \tinitialized = 0;\n }\n--\n1.6.4.304.g1365c.dirty\n"},{"id":"122654","messageId":"1252376822-6138-13-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 12/14] Selftests verifying semantics when loading notes trees with various fanouts","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:27:00Z","receivedAt":"2009-09-08T02:27:00Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Add selftests verifying:\n- that we are able to parse notes trees with various fanout schemes\n- that notes trees with conflicting fanout schemes are parsed as expected\n\nSigned-off-by: Johan Herland \n---\n t/t3303-notes-subtrees.sh | 137 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 137 insertions(+), 0 deletions(-)\n create mode 100755 t/t3303-notes-subtrees.sh\n\ndiff --git a/t/t3303-notes-subtrees.sh b/t/t3303-notes-subtrees.sh\nnew file mode 100755\nindex 0000000..d24203e\n--- /dev/null\n+++ b/t/t3303-notes-subtrees.sh\n@@ -0,0 +1,137 @@\n+#!/bin/sh\n+\n+test_description='Test commit notes organized in subtrees'\n+\n+. ./test-lib.sh\n+\n+number_of_commits=100\n+\n+start_note_commit () {\n+\ttest_tick &&\n+\tcat < $GIT_COMMITTER_DATE\n+data < output &&\n+\ti=$number_of_commits &&\n+\twhile [ $i -gt 0 ]; do\n+\t\techo \" commit #$i\" &&\n+\t\techo \" note for commit #$i\" &&\n+\t\ti=$(($i-1));\n+\tdone > expect &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success \"setup: create $number_of_commits commits\" '\n+\n+\t(\n+\t\tnr=0 &&\n+\t\twhile [ $nr -lt $number_of_commits ]; do\n+\t\t\tnr=$(($nr+1)) &&\n+\t\t\ttest_tick &&\n+\t\t\tcat < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data <From rule #6, we see that the following example notes tree is valid:\n- y2009m09/0123456789abcdef0123456789abcdef012345678\n- y2009m09/123456789abcdef0123456789abcdef0123456789\n- y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n- y2008m01/d31/34/56789abcdef0123456789abcdef0123456789ab\n- y2008m01/d16/4567/89abcdef0123456789abcdef0123456789abc\n- y2008m01/d16/5678/9abcdef0123456789abcdef0123456789abcd\n\nConversely the following structure is invalid (violates rule #6):\n- y2009m09/0123456789abcdef0123456789abcdef012345678\n- y2009m09/12/3456789abcdef0123456789abcdef0123456789\n- y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n- y2008m01/34/56789abcdef0123456789abcdef0123456789ab\n- y2008m01/d16/45/6789abcdef0123456789abcdef0123456789abc\n- y2008/m01d16/5678/9abcdef0123456789abcdef0123456789abcd\n\nThe flexibility added by this patch adds considerable complexity to the notes\ntree parser, but the runtime and memory usage is not significantly affected\n(except for the effects introduced by the chosen notes tree structure).\n\nInternally, the 16-tree data structure introduced in earlier patches is still\nused to hold the SHA1-based fanout levels and the note entries themselves.\nHowever, this patch adds a hierarchical date-based linked-list structure\naround the 16-tree structure that mirrors the fanout scheme used in the\nactual notes tree.\n\nSigned-off-by: Johan Herland \n---\n notes.c | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 364 insertions(+), 39 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 6926aa6..a3e3f83 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -7,6 +7,70 @@\n #include \"tree-walk.h\"\n \n /*\n+ * Format of entries in the notes tree structure:\n+ *\n+ * note-entry ::= (period sep)? sha1-spec\n+ * period ::= year sep?\n+ * (month sep?\n+ * (date sep?)?\n+ * )?;\n+ * year ::= 'y' yearnum;\n+ * month ::= 'm' monthnum;\n+ * date ::= 'd' datenum;\n+ * yearnum ::= # 4-digit decimal year, from annotated commit's timestamp;\n+ * monthnum ::= # 2-digit decimal month, from annotated commit's timestamp;\n+ * datenum ::= # 2-digit decimal date, from annotated commit's timestamp;\n+ * sha1-spec ::= (hex-fragment sep?){20}\n+ * sep ::= '/';\n+ * hex-fragment ::= # Fragment of hexsha1 (2 bytes);\n+ * hexsha1 ::= # SHA1 of annotated commit in hex format (40 bytes);\n+ *\n+ * Thus, the following example note entries are all valid:\n+ * - 0123456789abcdef0123456789abcdef012345678\n+ * - 01/23456789abcdef0123456789abcdef012345678\n+ * - 0123/456789abcdef0123456789abcdef012345678\n+ * - 01/23/456789abcdef0123456789abcdef012345678\n+ * - 0123/4567/89abcdef0123456789abcdef012345678\n+ * - 0123/45/67/89abcdef0123456789abcdef012345678\n+ * - y2009/0123456789abcdef0123456789abcdef012345678\n+ * - y2009/m09/01/23456789abcdef0123456789abcdef012345678\n+ * - y2009/m09/d01/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m09/01/23/456789abcdef0123456789abcdef012345678\n+ * - y2009m09/d01/0123/4567/89abcdef0123456789abcdef012345678\n+ * - y2009/m09d01/01/23/45/67/89abcdef0123456789abcdef012345678\n+ * - y2009m09d01/0123456789abcdef0123456789abcdef012345678\n+ *\n+ * and the following example note entries are all invalid:\n+ * - 0/123456789abcdef0123456789abcdef012345678\n+ * - 012/3456789abcdef0123456789abcdef012345678\n+ * - 01/234/56789abcdef0123456789abcdef012345678\n+ * - y20090123456789abcdef0123456789abcdef012345678\n+ * - y20/09/m09/01/23456789abcdef0123456789abcdef012345678\n+ * - y20/09m09/d01/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m/09/01/23/456789abcdef0123456789abcdef012345678\n+ * - y2009/d01/0123/4567/89abcdef0123456789abcdef012345678\n+ * - m09/y2009/d01/01/23/45/67/89abcdef0123456789abcdef012345678\n+ *\n+ * In addition to the above per-entry rules, we require that _all_ entries at\n+ * a given level in the notes tree (levels are separated by '/') follow the\n+ * exact same format at that level. Thus the following structure is valid:\n+ * - y2009m09/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m09/123456789abcdef0123456789abcdef0123456789\n+ * - y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n+ * - y2008m01/d31/34/56789abcdef0123456789abcdef0123456789ab\n+ * - y2008m01/d16/4567/89abcdef0123456789abcdef0123456789abc\n+ * - y2008m01/d16/5678/9abcdef0123456789abcdef0123456789abcd\n+ *\n+ * but the following structure is invalid:\n+ * - y2009m09/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m09/12/3456789abcdef0123456789abcdef0123456789\n+ * - y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n+ * - y2008m01/34/56789abcdef0123456789abcdef0123456789ab\n+ * - y2008m01/d16/45/6789abcdef0123456789abcdef0123456789abc\n+ * - y2008/m01d16/5678/9abcdef0123456789abcdef0123456789abcd\n+ */\n+\n+/*\n * Use a non-balancing simple 16-tree structure with struct int_node as\n * internal nodes, and struct leaf_node as leaf nodes. Each int_node has a\n * 16-array of pointers to its children.\n@@ -17,9 +81,45 @@\n * - ptr & 3 == 3 - pointer to subtree entry - cast to struct leaf_node *\n *\n * The root node is a statically allocated struct int_node.\n+ *\n+ * In order to allow date-based fanout schemes in addition to the original\n+ * SHA1-based fanout schemes, we need to overload this structure, as follows:\n+ * If the first pointer in the 16-array is ~0 (i.e. 0xffffffff on 32-bit\n+ * systems and 0xffffffffffffffff on 64-bit systems), then the int_node is NOT\n+ * to be interpreted as a 16-array of child node pointers. Rather, the int_node\n+ * now represents a period-based node with the following properties:\n+ * - The node has a pointer to a \"child\" node of type struct int_node, which is\n+ * EITHER a \"regular\" int_node object representing the root node of a 16-tree\n+ * structure holding notes associated with commits with timestamps within\n+ * that time period, OR another period-based int_node representing some\n+ * subdivision of the time period.\n+ * - The node also has a pointer to a \"previous\" period-based int_node, which\n+ * represents the previous time period for which there exist note objects.\n+ * - The node has a pointer to a \"parent\" node, which is the period-based\n+ * int_node that has this int_node as one of its children. This is needed\n+ * when traversing the date-based int_nodes looking for a period matching the\n+ * given commit. For top-level objects, this is set to NULL.\n+ * - The node stores the SHA1 sum of the tree object that represents its child\n+ * (within the notes tree structure). Thus, we keep a reference to the child\n+ * structure that without necessarily allocating the child node (and\n+ * underlying structure).\n+ * - Finally, the node has a period string, which indicates the time period of\n+ * the notes contained within, typically of the form \"YYYY\", \"YYYY-MM\" or\n+ * \"YYYY-MM-DD\", depending on the granularity of the corresponding\n+ * period-based entries in the notes tree structure.\n */\n struct int_node {\n-\tvoid *a[16];\n+\tunion {\n+\t\tvoid *a[16];\n+\t\tstruct {\n+\t\t\tvoid *magic; /* ~0 \"enables\" this part of the union */\n+\t\t\tstruct int_node *child;\n+\t\t\tstruct int_node *prev;\n+\t\t\tstruct int_node *parent;\n+\t\t\tunsigned char tree_sha1[20];\n+\t\t\tchar period[11]; /* Enough to hold \"YYYY-MM-DD\" */\n+\t\t};\n+\t};\n };\n \n /*\n@@ -51,12 +151,18 @@ struct leaf_node {\n #define SUBTREE_SHA1_PREFIXCMP(key_sha1, subtree_sha1) \\\n \t(memcmp(key_sha1, subtree_sha1, subtree_sha1[19]))\n \n+#define SUBTREE_DATE_PREFIXCMP(commit_date, subtree_date) \\\n+\t(prefixcmp(commit_date, subtree_date))\n+\n static struct int_node root_node;\n \n+static struct int_node *cur_node;\n+\n static int initialized;\n \n-static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n-\t\tunsigned int n);\n+static void load_subtree(const unsigned char *sha1,\n+\t\tconst unsigned char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, struct int_node *parent, int n);\n \n /*\n * To find a leaf_node:\n@@ -94,7 +200,8 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n \t\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n \t\t\t/* unpack tree and resume search */\n \t\t\ttree->a[i] = NULL;\n-\t\t\tload_subtree(l, tree, n);\n+\t\t\tload_subtree(l->val_sha1, l->key_sha1, l->key_sha1[19],\n+\t\t\t\t tree, NULL, (int) n);\n \t\t\tfree(l);\n \t\t\treturn note_tree_find(tree, n, key_sha1);\n \t\t}\n@@ -117,7 +224,8 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n \tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n \t\t/* unpack tree and resume search */\n \t\ttree->a[0] = NULL;\n-\t\tload_subtree(l, tree, n);\n+\t\tload_subtree(l->val_sha1, l->key_sha1, l->key_sha1[19], tree,\n+\t\t\t NULL, (int) n);\n \t\tfree(l);\n \t\treturn note_tree_find(tree, n, key_sha1);\n \t}\n@@ -173,16 +281,28 @@ static int note_tree_insert(struct int_node *tree, unsigned char n,\n /* Free the entire notes data contained in the given tree */\n static void note_tree_free(struct int_node *tree)\n {\n-\tunsigned int i;\n-\tfor (i = 0; i < 16; i++) {\n-\t\tvoid *p = tree->a[i];\n-\t\tswitch(GET_PTR_TYPE(p)) {\n-\t\tcase PTR_TYPE_INTERNAL:\n-\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n-\t\t\t/* fall through */\n-\t\tcase PTR_TYPE_NOTE:\n-\t\tcase PTR_TYPE_SUBTREE:\n-\t\t\tfree(CLR_PTR_TYPE(p));\n+\tif (tree->magic == (void *) ~0) {\n+\t\tif (tree->prev) {\n+\t\t\tnote_tree_free(tree->prev);\n+\t\t\tfree(tree->prev);\n+\t\t}\n+\t\tif (tree->child) {\n+\t\t\tnote_tree_free(tree->child);\n+\t\t\tfree(tree->child);\n+\t\t}\n+\t}\n+\telse {\n+\t\tunsigned int i;\n+\t\tfor (i = 0; i < 16; i++) {\n+\t\t\tvoid *p = tree->a[i];\n+\t\t\tswitch(GET_PTR_TYPE(p)) {\n+\t\t\tcase PTR_TYPE_INTERNAL:\n+\t\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n+\t\t\t\t/* fall through */\n+\t\t\tcase PTR_TYPE_NOTE:\n+\t\t\tcase PTR_TYPE_SUBTREE:\n+\t\t\t\tfree(CLR_PTR_TYPE(p));\n+\t\t\t}\n \t\t}\n \t}\n }\n@@ -215,29 +335,139 @@ static int get_sha1_hex_segment(const char *hex, unsigned int hex_len,\n \treturn len;\n }\n \n-static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n-\t\tunsigned int n)\n+/*\n+ * Parse year/month/date strings, and generate the corresponding period string\n+ * for the given path entry:\n+ * - prefix must follow one of these forms: \"\", \"YYYY\", \"YYYY-MM\"\n+ * - path should follow one of these forms: \"yYYYY\", \"yYYYYmMM\", \"yYYYYmMMdDD\",\n+ * \"mMMdDD\", \"mMM\" or \"dDD\"\n+ * The resulting string (which follows the form \"YYYY\", \"YYYY-MM\" or\n+ * \"YYYY-MM-DD\") is returned as a static string. If path is not valid in the\n+ * given (prefix) context, NULL is returned.\n+ */\n+static const char *parse_period(const char *prefix, unsigned int prefix_len,\n+\t\tconst char *path, unsigned int path_len)\n+{\n+\tstatic char result[11];\n+\tchar expect_type; /* y/m/d for year/month/day-based fanout */\n+\tunsigned int expect_len, value;\n+\tchar *endptr, *target = result;\n+\n+\tswitch (prefix_len) {\n+\tcase 0:\n+\t\t/* No prefix, expect year-based fanout in path */\n+\t\texpect_type = 'y';\n+\t\texpect_len = 4;\n+\t\tbreak;\n+\tcase 4:\n+\t\t/* Year in prefix, expect month-based fanout in path */\n+\t\texpect_type = 'm';\n+\t\texpect_len = 2;\n+\t\tbreak;\n+\tcase 7:\n+\t\t/* \"YYYY-MM\" in prefix, expect day-based fanout in path */\n+\t\texpect_type = 'd';\n+\t\texpect_len = 2;\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"Date-based notes tree loading invoked with invalid \"\n+\t\t \"prefix '%.*s'\", prefix_len, prefix);\n+\t}\n+\n+\tif (path[0] != expect_type) {\n+\t\twarning(\"Unexpected entry path in date-based notes tree: '%s' \"\n+\t\t\t\"(skipping)\", path);\n+\t\treturn NULL;\n+\t}\n+\tvalue = (unsigned int) strtoul(path + 1, &endptr, 10);\n+\tswitch (expect_type) {\n+\tcase 'y':\n+\t\tif (value < 1969 || value >= 3000) {\n+\t\t\twarning(\"Invalid year value in date-based notes tree:\"\n+\t\t\t\t\" '%s' (skipping)\", path);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tbreak;\n+\tcase 'm':\n+\t\tif (value < 1 || value > 12) {\n+\t\t\twarning(\"Invalid month value in date-based notes tree:\"\n+\t\t\t\t\" '%s' (skipping)\", path);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tbreak;\n+\tcase 'd':\n+\t\tif (value < 1 || value > 31) {\n+\t\t\twarning(\"Invalid day value in date-based notes tree:\"\n+\t\t\t\t\" '%s' (skipping)\", path);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tbreak;\n+\t}\n+\n+\tif (prefix == result) {\n+\t\ttarget = result + prefix_len;\n+\t\tprefix = NULL;\n+\t\tprefix_len = 0;\n+\t}\n+\tprefix_len = snprintf(target, 11, \"%.*s%s%0*u\", prefix_len, prefix,\n+\t\t\t expect_len == 2 ? \"-\" : \"\", expect_len, value);\n+\tprefix_len += target - result;\n+\tassert(prefix_len < 11);\n+\n+\tif (*endptr) /* there are more components in this path */\n+\t\treturn parse_period(result, prefix_len, endptr,\n+\t\t\t\t path_len - (endptr - path));\n+\treturn result;\n+}\n+\n+static void load_date_subtree(struct tree_desc *tree_desc,\n+\t\tconst char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, struct int_node *parent)\n+{\n+\tstruct name_entry entry;\n+\tstruct int_node *cur_node = NULL;\n+\tstruct int_node *new_node;\n+\n+\twhile (tree_entry(tree_desc, &entry)) {\n+\t\tconst char *period = parse_period(\n+\t\t\tprefix, prefix_len, entry.path, strlen(entry.path));\n+\t\tif (!period)\n+\t\t\tcontinue;\n+\t\tif (tree_desc->size) /* this is not the last tree entry */\n+\t\t\tnew_node = (struct int_node *)\n+\t\t\t\txmalloc(sizeof(struct int_node));\n+\t\telse /* this is the last entry, store directly into node */\n+\t\t\tnew_node = node;\n+\n+\t\tnew_node->magic = (void *) ~0;\n+\t\tnew_node->child = NULL;\n+\t\tnew_node->prev = cur_node;\n+\t\tnew_node->parent = parent;\n+\t\thashcpy(new_node->tree_sha1, entry.sha1);\n+\t\tstrcpy(new_node->period, period);\n+\t\tcur_node = new_node;\n+\t}\n+\tassert(!cur_node || cur_node == node);\n+}\n+\n+static void load_sha1_subtree(struct tree_desc *tree_desc,\n+\t\tconst unsigned char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, unsigned char n)\n {\n \tunsigned char commit_sha1[20];\n-\tunsigned int prefix_len;\n \tint status;\n-\tvoid *buf;\n-\tstruct tree_desc desc;\n \tstruct name_entry entry;\n \n-\tbuf = fill_tree_descriptor(&desc, subtree->val_sha1);\n-\tif (!buf)\n-\t\tdie(\"Could not read %s for notes-index\",\n-\t\t sha1_to_hex(subtree->val_sha1));\n-\n-\tprefix_len = subtree->key_sha1[19];\n \tassert(prefix_len * 2 >= n);\n-\tmemcpy(commit_sha1, subtree->key_sha1, prefix_len);\n-\twhile (tree_entry(&desc, &entry)) {\n+\tmemcpy(commit_sha1, prefix, prefix_len);\n+\twhile (tree_entry(tree_desc, &entry)) {\n \t\tint len = get_sha1_hex_segment(entry.path, strlen(entry.path),\n \t\t\t\tcommit_sha1 + prefix_len, 20 - prefix_len);\n-\t\tif (len < 0)\n+\t\tif (len < 0) {\n+\t\t\twarning(\"Invalid value in notes tree: '%s' (skipping)\",\n+\t\t\t\tentry.path);\n \t\t\tcontinue; /* entry.path is not a SHA1 sum. Skip */\n+\t\t}\n \t\tlen += prefix_len;\n \n \t\t/*\n@@ -258,6 +488,42 @@ static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n \t\t\tassert(!status);\n \t\t}\n \t}\n+}\n+\n+static void load_subtree(const unsigned char *sha1,\n+\t\tconst unsigned char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, struct int_node *parent, int n)\n+{\n+\tvoid *buf;\n+\tstruct tree_desc desc;\n+\n+\tbuf = fill_tree_descriptor(&desc, sha1);\n+\tif (!buf)\n+\t\tdie(\"Could not read notes subtree at %s\", sha1_to_hex(sha1));\n+\t/*\n+\t * After fill_tree_descriptor(), we can peek at the first tree entry\n+\t * in desc.entry.\n+\t */\n+\tswitch (desc.entry.path[0]) {\n+\tcase 'd':\n+\t\tif (strlen(desc.entry.path) != 3)\n+\t\t\tbreak;\n+\t\t/* fall-through */\n+\tcase 'm':\n+\tcase 'y':\n+\t\t/* path cannot be a SHA1 fragment */\n+\t\tload_date_subtree(&desc, (const char *) prefix, prefix_len,\n+\t\t\t\t node, parent);\n+\t\tfree(buf);\n+\t\treturn;\n+\t}\n+\tif (n < 0) {\n+\t\t/* Arriving from a date-based subtree; reset prefix */\n+\t\tn = 0;\n+\t\tprefix = NULL;\n+\t\tprefix_len = 0;\n+\t}\n+\tload_sha1_subtree(&desc, prefix, prefix_len, node, n);\n \tfree(buf);\n }\n \n@@ -265,23 +531,81 @@ static void initialize_notes(const char *notes_ref_name)\n {\n \tunsigned char sha1[20], commit_sha1[20];\n \tunsigned mode;\n-\tstruct leaf_node root_tree;\n \n \tif (!notes_ref_name || read_ref(notes_ref_name, commit_sha1) ||\n \t get_tree_entry(commit_sha1, \"\", sha1, &mode))\n \t\treturn;\n \n-\thashclr(root_tree.key_sha1);\n-\thashcpy(root_tree.val_sha1, sha1);\n-\tload_subtree(&root_tree, &root_node, 0);\n+\tload_subtree(sha1, NULL, 0, &root_node, NULL, 0);\n+\tcur_node = &root_node;\n }\n \n-static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n+static unsigned char *lookup_notes(const struct commit *commit)\n {\n-\tstruct leaf_node *found = note_tree_find(&root_node, 0, commit_sha1);\n-\tif (found)\n-\t\treturn found->val_sha1;\n-\treturn NULL;\n+\tstruct int_node *node = cur_node, *seen_node = cur_node;\n+\tstruct leaf_node *found;\n+\tconst char *short_date;\n+\n+\tif (!node)\n+\t\treturn NULL;\n+\n+\t/* Convert commit->date to YYYY-MM-DD format */\n+\tshort_date = show_date(commit->date, 0, DATE_SHORT);\n+\n+\twhile (node->magic == (void *) ~0) { /* date-based node */\n+\t\tint cmp = SUBTREE_DATE_PREFIXCMP(short_date, node->period);\n+\t\tif (cmp == 0) {\n+\t\t\t/* Search inside child node */\n+\t\t\tif (!node->child) {\n+\t\t\t\t/* Must unpack child node first */\n+\t\t\t\tnode->child = (struct int_node *)\n+\t\t\t\t\txcalloc(sizeof(struct int_node), 1);\n+\t\t\t\tload_subtree(node->tree_sha1,\n+\t\t\t\t\t(const unsigned char *) node->period,\n+\t\t\t\t\tstrlen(node->period), node->child,\n+\t\t\t\t\tnode, -1);\n+\t\t\t}\n+\t\t\tseen_node = node;\n+\t\t\tnode = node->child;\n+\t\t}\n+\t\telse if (cmp > 0) {\n+\t\t\t/* Search in past node */\n+\t\t\tif (node->prev)\n+\t\t\t\tnode = node->prev;\n+\t\t\telse\n+\t\t\t\tnode = node->parent;\n+\t\t}\n+\t\telse {\n+\t\t\t/* Search in future node */\n+\t\t\tif (!node->parent) {\n+\t\t\t\t/* Restart from root_node */\n+\t\t\t\tseen_node = node;\n+\t\t\t\tnode = &root_node;\n+\t\t\t}\n+\t\t\telse\n+\t\t\t\tnode = node->parent;\n+\t\t}\n+\t\tif (!node || node == seen_node) {\n+\t\t\t/* We've been here before, give up search */\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\twhile (cur_node &&\n+\t SUBTREE_DATE_PREFIXCMP(cur_node->period, seen_node->period) < 0)\n+\t{\n+\t\t/*\n+\t\t * We're about to move cur_node backwards in history. We are\n+\t\t * unlikely to need this cur_node in the future, so free() it.\n+\t\t */\n+\t\tnote_tree_free(cur_node->child);\n+\t\tcur_node->child = NULL;\n+\t\tcur_node = cur_node->parent;\n+\t}\n+\tcur_node = seen_node;\n+\n+\t/* Drill down further with SHA1-based lookup */\n+\tfound = note_tree_find(node, 0, commit->object.sha1);\n+\treturn found ? found->val_sha1 : NULL;\n }\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n@@ -303,7 +627,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tinitialized = 1;\n \t}\n \n-\tsha1 = lookup_notes(commit->object.sha1);\n+\tsha1 = lookup_notes(commit);\n \tif (!sha1)\n \t\treturn;\n \n@@ -342,6 +666,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \n void free_commit_notes()\n {\n+\tcur_node = NULL;\n \tnote_tree_free(&root_node);\n \tmemset(&root_node, 0, sizeof(struct int_node));\n \tinitialized = 0;\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122656","messageId":"1252376822-6138-15-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"[PATCHv5 14/14] Add test cases for date-based fanouts","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T02:27:02Z","receivedAt":"2009-09-08T02:27:02Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Signed-off-by: Johan Herland \n---\n t/t3303-notes-subtrees.sh | 64 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 64 insertions(+), 0 deletions(-)\n\ndiff --git a/t/t3303-notes-subtrees.sh b/t/t3303-notes-subtrees.sh\nindex d24203e..d3cbf6d 100755\n--- a/t/t3303-notes-subtrees.sh\n+++ b/t/t3303-notes-subtrees.sh\n@@ -134,4 +134,68 @@ test_expect_success 'verify notes in 4/36-fanout overriding 2/38-fanout' 'verify\n test_expect_success 'test notes in 2/38-fanout overriding 2/2/36-fanout' 'test_preferred \"s|^..|&/|\" \"s|^\\(..\\)\\(..\\)|\\1/\\2/|\"'\n test_expect_success 'verify notes in 2/38-fanout overriding 2/2/36-fanout' 'verify_notes'\n \n+test_date_based () {\n+\t(\n+\t\tstart_note_commit &&\n+\t\tnr=$number_of_commits &&\n+\t\tgit log --format=\"%H %ct\" refs/heads/master |\n+\t\twhile read sha1 date_t; do\n+\t\t\tdate=$(date -u -d \"@$date_t\" +\"$1\")\n+\t\t\tnote_path=\"$date/$(echo \"$sha1\" | sed \"$2\")\"\n+\t\t\tcat < I have some performance numbers that I will send in a separate email.\n\nOk, here we go:\n\nTest scenario:\nLinux kernel repo with 157118 commits, 1 note per commit, with notes\norganized into various fanout schemes.\nHardware is Intel Core 2 Quad with 4GB RAM.\n\nThe tests were done on the following algorithms:\n\n- \"before\": This is the state of the notes code after applying patches 1-9.\n It uses the original notes-in-hash-map implementation, and does not grok\n any fanout scheme.\n\n- \"16tree\": This is the state of the notes code after applying patch 10.\n It uses the 16-tree data structure that parses the SHA-1 based fanout\n schemes.\n\n- \"flexible\": This is the state of the notes code after applying the entire\n patch series. This code parses a variety of date- and SHA1-based fanout\n schemes.\n\nFurthermore, the following notes tree structures were tested:\n\n- \"no-notes\": Testing without any notes at all. This is only present as a\n baseline, and to verify that the notes code does not negatively affect\n performance when not in use.\n\n- \"no-fanout\": All notes stored directly inside the root notes tree object.\n\n- \"2_38\": All notes stored in a SHA1-based 2/38 fanout scheme.\n\n- \"2_2_36\": All notes stored in a SHA1-based 2/2/36 fanout scheme.\n\n- \"ym\": Notes are organized within \"yYYYYmMM\"-named subtrees, where \"YYYY\"\n and \"MM\" are the year and month (respectively) from the annotated commit's\n commit date.\n\n- \"ym_2_38\": Same as above, but with a 2/38 SHA1-based fanout scheme within\n the \"yYYYYmMM\"-named subtrees.\n\n- \"ymd\": Notes are organized within \"yYYYYmMMdDD\"-named subtrees.\n\n- \"ymd_2_38\": Same as above, but with a 2/38 SHA1-based fanout scheme within\n the \"yYYYYmMMdDD\"-named subtrees.\n\n- \"y_m\": Notes are organized within two-level \"yYYYY/mMM\" subtrees.\n\n- \"y_m_2_38\": Same as above, but with a 2/38 SHA1-based fanout scheme within\n the \"yYYYY/mMM\"-named subtrees.\n\n- \"y_m_d\": Notes are organized within three-level \"yYYYY/mMM/dDD\" subtrees.\n\n- \"y_m_d_2_38\": Same as above, but with a 2/38 SHA1-based fanout scheme\n within the \"yYYYY/mMM/dDD\"-named subtrees.\n\n\nHere are the runtime numbers, the first column shows the runtime for 100\nrepetitions of \"git log -n10\" (which we assume to be a common use case),\nand the second column shows the runtime from a single run of\n\"git log --all\" (which is somewhat closer to a worst case).\n\n\nAlgorithm / Notes tree git log -n10 (x100) git log --all\n------------------------------------------------------------\nbefore / no-notes 4.78s 63.90s\nbefore / no-fanout 56.85s 65.69s\n\n16tree / no-notes 4.77s 64.18s\n16tree / no-fanout 30.35s 65.39s\n16tree / 2_38 5.57s 65.42s\n16tree / 2_2_36 5.19s 65.76s\n\nflexible / no-notes 4.78s 63.91s\nflexible / no-fanout 30.34s 65.57s\nflexible / 2_38 5.57s 65.46s\nflexible / 2_2_36 5.18s 65.72s\nflexible / ym 5.13s 65.66s\nflexible / ym_2_38 5.08s 65.63s\nflexible / ymd 5.30s 65.45s\nflexible / ymd_2_38 5.29s 65.90s\nflexible / y_m 5.11s 65.72s\nflexible / y_m_2_38 5.08s 65.67s\nflexible / y_m_d 5.06s 65.50s\nflexible / y_m_d_2_38 5.07s 65.79s\n\n\nFinally, I have also looked at the memory consumption of the various\nalgorithms and fanout schemes:\n\nThe memory usage was measured by calculating the #bytes dynamically\nallocated for the notes data structure, and printing the current\nusage every time get_commit_notes() was called during a complete run\nof \"git log --all\".\n\nThe results are attached as two gnuplot graphs, one with regular\naxes, and one with logarithmic axes.\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122662","messageId":"7vd462qdeg.fsf@alter.siamese.dyndns.org","threadId":"20872","inReplyTo":"200909080512.34634.johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-08T04:16:39Z","receivedAt":"2009-09-08T04:16:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> Furthermore, the following notes tree structures were tested:\n>\n> - \"no-notes\": Testing without any notes at all. This is only present as a\n> baseline, and to verify that the notes code does not negatively affect\n> performance when not in use.\n\nMinor nit.\n\nFor this to be a baseline, you would need to have another algorithm before\n\"before\", i.e., without any of these notes implementation.\n\nComparison with \"before\" alone is not meaningful. That is like starting\nwith a state with unknown performance regression compared to the stock\nversion, and then boast improvements made by various variations.\n\nYou would need to compare overhead of various \"algorithms\" with the stock\ngit in \"no-notes\" case as well. It would give us the true performance\ncost of supporting notes.\n"},{"id":"122688","messageId":"200909081054.02523.johan@herland.net","threadId":"20872","inReplyTo":"7vd462qdeg.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T08:54:02Z","receivedAt":"2009-09-08T08:54:02Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 08 September 2009, Junio C Hamano wrote:\n> Johan Herland writes:\n> > Furthermore, the following notes tree structures were tested:\n> >\n> > - \"no-notes\": Testing without any notes at all. This is only present as\n> > a baseline, and to verify that the notes code does not negatively\n> > affect performance when not in use.\n> \n> Minor nit.\n> \n> For this to be a baseline, you would need to have another algorithm\n> before \"before\", i.e., without any of these notes implementation.\n> \n> Comparison with \"before\" alone is not meaningful. That is like starting\n> with a state with unknown performance regression compared to the stock\n> version, and then boast improvements made by various variations.\n> \n> You would need to compare overhead of various \"algorithms\" with the stock\n> git in \"no-notes\" case as well. It would give us the true performance\n> cost of supporting notes.\n\nTrue. Here is the same table with the baseline ('next') entry on top:\n\n\nAlgorithm / Notes tree git log -n10 (x100) git log --all\n------------------------------------------------------------\nnext / no-notes 4.77s 63.84s\n\nbefore / no-notes 4.78s 63.90s\nbefore / no-fanout 56.85s 65.69s\n\n16tree / no-notes 4.77s 64.18s\n16tree / no-fanout 30.35s 65.39s\n16tree / 2_38 5.57s 65.42s\n16tree / 2_2_36 5.19s 65.76s\n\nflexible / no-notes 4.78s 63.91s\nflexible / no-fanout 30.34s 65.57s\nflexible / 2_38 5.57s 65.46s\nflexible / 2_2_36 5.18s 65.72s\nflexible / ym 5.13s 65.66s\nflexible / ym_2_38 5.08s 65.63s\nflexible / ymd 5.30s 65.45s\nflexible / ymd_2_38 5.29s 65.90s\nflexible / y_m 5.11s 65.72s\nflexible / y_m_2_38 5.08s 65.67s\nflexible / y_m_d 5.06s 65.50s\nflexible / y_m_d_2_38 5.07s 65.79s\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122691","messageId":"alpine.DEB.1.00.0909081100020.4330@intel-tinevez-2-302","threadId":"20872","inReplyTo":"200909081054.02523.johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-09-08T09:32:04Z","receivedAt":"2009-09-08T09:32:04Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 8 Sep 2009, Johan Herland wrote:\n\n> Algorithm / Notes tree git log -n10 (x100) git log --all\n> ------------------------------------------------------------\n> next / no-notes 4.77s 63.84s\n> \n> before / no-notes 4.78s 63.90s\n> before / no-fanout 56.85s 65.69s\n> \n> 16tree / no-notes 4.77s 64.18s\n> 16tree / no-fanout 30.35s 65.39s\n> 16tree / 2_38 5.57s 65.42s\n> 16tree / 2_2_36 5.19s 65.76s\n> \n> flexible / no-notes 4.78s 63.91s\n> flexible / no-fanout 30.34s 65.57s\n> flexible / 2_38 5.57s 65.46s\n> flexible / 2_2_36 5.18s 65.72s\n> flexible / ym 5.13s 65.66s\n> flexible / ym_2_38 5.08s 65.63s\n> flexible / ymd 5.30s 65.45s\n> flexible / ymd_2_38 5.29s 65.90s\n> flexible / y_m 5.11s 65.72s\n> flexible / y_m_2_38 5.08s 65.67s\n> flexible / y_m_d 5.06s 65.50s\n> flexible / y_m_d_2_38 5.07s 65.79s\n\nIt's good to see that the no-notes behaves roughly like baseline.\n\nI can see that some people may think that date-based fan-out is the cat's \nass, but I have to warn that we have no idea how notes will be used, and \nthe date-based fan-out is rather limiting in certain respects:\n\n- for the typical nightly-build-generated notes, this fan-out is pretty \n inefficient memory-wise.\n\n- I find the restriction to commits rather limiting.\n\n- most of the performance difference between the date-based and the SHA-1 \n based fan-out looks to me as if the issue was the top-level tree. \n Basically, this tree has to be read _every_ time _anybody_ wants to read \n a note.\n\n Maybe a finer-grained fan-out (finer than 16-bits) could help. After \n all, if you have 16 different notes, chances are that they have 16 \n different first letters, but all have the same commit year. That's \n where the top-level notes with a fan-out perform incredibly bad.\n\n But I think that having a dynamic fan-out that can even put blobs into \n the top-level tree (nothing prevents us from doing that, right?) would \n _outperform_ the date-based one, at least with less than 1 note/commit \n (and maybe even then, because the year-based fan-out results in pretty \n varying entropies per fan-out depth).\n\n The real question for me, therefore, is: what is the optimal way to \n strike the balance between size of the tree objects (which we want to \n be small, so that unpacking them is fast) and depth of the fan-out \n (which we want to be shallow to avoid reading worst-case 39 tree objects \n to get at one single note).\n\n- related to the previous point is my gut feeling that the date-based \n fan-out has nothing to do with any theoretical optimum. I am pretty \n certain that the optimal fan-out strategy depends heavily on the SHA-1s \n of the annotated objects (if you have 10,000 notes in 2009, but only 1 \n in 2008, the year-based fan-out _must_ be suboptimal) and maybe is \n something like a sibling to the Fibonacci heap.\n\n- I'd love to see performance numbers for less than 157118 notes. Don't \n get me wrong, it is good to see the worst-case scenario in terms of \n notes/commits ratio. But it will hardly be the common case, and I \n very much would like to optimize for the common case.\n\n So, I'd appreciate if you could do the tests with something like 500 \n notes, randomly spread over the commits (rationale: my original \n understanding was that the notes could amend commit messages, and that \n is much more likely to be done with relatively old commits that you \n cannot change anymore).\n\nPlease understand that I might not have the time to participate in this \nthread as much as I would like to. The next 4 days will be especially \nhard.\n\nCiao,\nDscho\n"},{"id":"122696","messageId":"200909081436.30761.johan@herland.net","threadId":"20872","inReplyTo":"alpine.DEB.1.00.0909081100020.4330@intel-tinevez-2-302","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T12:36:30Z","receivedAt":"2009-09-08T12:36:30Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Hi,\n\nOn Tuesday 08 September 2009, Johannes Schindelin wrote:\n> On Tue, 8 Sep 2009, Johan Herland wrote:\n> > Algorithm / Notes tree git log -n10 (x100) git log --all\n> > ------------------------------------------------------------\n> > next / no-notes 4.77s 63.84s\n> >\n> > before / no-notes 4.78s 63.90s\n> > before / no-fanout 56.85s 65.69s\n> >\n> > 16tree / no-notes 4.77s 64.18s\n> > 16tree / no-fanout 30.35s 65.39s\n> > 16tree / 2_38 5.57s 65.42s\n> > 16tree / 2_2_36 5.19s 65.76s\n> >\n> > flexible / no-notes 4.78s 63.91s\n> > flexible / no-fanout 30.34s 65.57s\n> > flexible / 2_38 5.57s 65.46s\n> > flexible / 2_2_36 5.18s 65.72s\n> > flexible / ym 5.13s 65.66s\n> > flexible / ym_2_38 5.08s 65.63s\n> > flexible / ymd 5.30s 65.45s\n> > flexible / ymd_2_38 5.29s 65.90s\n> > flexible / y_m 5.11s 65.72s\n> > flexible / y_m_2_38 5.08s 65.67s\n> > flexible / y_m_d 5.06s 65.50s\n> > flexible / y_m_d_2_38 5.07s 65.79s\n>\n> It's good to see that the no-notes behaves roughly like baseline.\n\nAgreed.\n\n> I can see that some people may think that date-based fan-out is the\n> cat's ass, but I have to warn that we have no idea how notes will be\n> used,\n\nI don't agree. Although we will certainly see many more use cases for \nnotes, I believe that the vast majority of them can be placed in one of \ntwo categories:\n\n1. Looking up a few (1 - 10) notes on an individual basis. In this case, \nperformance will always be \"good enough\", and I don't believe it's \nworth spending much time optimizing for this case.\n\n2. Looking up many notes in a sequence based on the chronology of the \nobjects (commits) they annotate. This is what 'git log' does, and I \nbelieve this is the case we should optimize for.\n\nOnce you work from this assumption, it is clear that date-based fanout \nbeats pure SHA1-based fanout, because it changes the notes lookup \nprocess from random-access to near-streaming. This is clearly reflected \nin the memory usage graphs I posted.\n\nAlso note that the \"flexible\" code to some degree resolves the whole \ndate-based fanout vs. SHA1-based fanout discussion: We are now free to \nchoose -- at runtime -- which notes tree structure is more optimal for \na given collection of notes.\n\n> and the date-based fan-out is rather limiting in certain respects:\n>\n> - for the typical nightly-build-generated notes, this fan-out is\n> pretty inefficient memory-wise.\n\nYes and no. A y/m/d/40 structure with 1 note for every y/m/d is indeed \nwasteful, but using a y/40 structure instead creates a much better \nsituation with a \"healthy\" ~365 notes per year level. And the y/40 \nstill preserves some of the 'streaming' aspects mentioned above.\n\n> - I find the restriction to commits rather limiting.\n\nI see your point, but I don't agree until I see a compelling case for \nannotating a non-commit.\n\nIn any case, the flexible lookup code still allow us to organize notes \npurely by SHA1-based fanout, so annotated trees/blobs could still be \nsupported by the same code (modulo a small s/struct commit/struct \nobject/ patch) provided that they are stored in a notes tree that \nsimply does not use date-based fanout.\n\n> - most of the performance difference between the date-based and the\n> SHA-1 based fan-out looks to me as if the issue was the top-level\n> tree. Basically, this tree has to be read _every_ time _anybody_\n> wants to read a note.\n\nNot sure what you're trying to say here. The top-level notes tree is \nread (as in fill_tree_descriptor()) exactly _once_. After that, it is \ncached by the internal data structure (until free_commit_notes() or \nend-of-process).\n\nIn the SHA1-based case, each notes lookup does indeed start at the root \nof the data structure (corresponding to the top-level tree), but in the \ndate-based case, we keep a pointer (cur_node) to the innermost \ndate-based tree that was most recently used. Thus, if the next note is \nwithin the same date-based tree (which I assume is the common case), \nthen we don't need to look at the root of the data structure.\n\n> Maybe a finer-grained fan-out (finer than 16-bits) could help. \n> After all, if you have 16 different notes, chances are that they have\n> 16 different first letters, but all have the same commit year. \n> That's where the top-level notes with a fan-out perform incredibly\n> bad.\n\nNot really, the first lookup would start at the root node, and navigate \ninto the year node, but all subsequent lookups would start directly at \nthe year node (and only backtrack if the commit year doesn't match the \nyear node).\n\nBTW, when you mention \"finer than 16-bits\", do you mean moving from a \n16-tree to a, say, 32-tree or 64-tree structure? That will complicate \nthe tree navigation somewhat, and increase memory waste. (Remember, I \nstarted out with a 256-tree idea, but scrapped it because of memory \nwaste).\n\n> But I think that having a dynamic fan-out that can even put blobs\n> into the top-level tree (nothing prevents us from doing that, right?)\n\nWell, the \"flexible\" code does add the new requirement that all entries \nin a notes (sub)tree object must follow the same scheme, i.e. you \ncannot have:\n\n /12/34567890123456789012345678901234567890\n /2345/678901234567890123456789012345678901\n\nbut you can have\n\n /12/34567890123456789012345678901234567890\n /23/45/678901234567890123456789012345678901\n\n> would _outperform_ the date-based one, at least with less than 1\n> note/commit (and maybe even then, because the year-based fan-out\n> results in pretty varying entropies per fan-out depth).\n>\n> The real question for me, therefore, is: what is the optimal way to\n> strike the balance between size of the tree objects (which we want\n> to be small, so that unpacking them is fast) and depth of the\n> fan-out (which we want to be shallow to avoid reading worst-case 39\n> tree objects to get at one single note).\n\ns/39/19/ (each fanout must use at least 2 chars of the 40-char SHA1)\n\nYes, the challenge is indeed striking the correct balance. I believe \nthat the notes code should be taught to write (and automatically \nre-organize) the notes tree so that it is optimized for the current \ncollection of notes.\n\n> - related to the previous point is my gut feeling that the date-based\n> fan-out has nothing to do with any theoretical optimum. I am\n> pretty certain that the optimal fan-out strategy depends heavily on\n> the SHA-1s of the annotated objects (if you have 10,000 notes in\n> 2009, but only 1 in 2008, the year-based fan-out _must_ be\n> suboptimal) and maybe is something like a sibling to the Fibonacci\n> heap.\n\nYes, it is trivial to create scenarios where any rigid date-based fanout \nis suboptimal. However, it is equally trivial to create scenarios where \nany SHA1-based fanout will perform worse than a carefully chosen \ndate-based fanout. I believe the best way forward is to design for \nflexibility in the notes tree structure, and then teach the notes \n_writing_ code to choose a notes tree structure that is good/optimal \nfor the given set of notes.\n\n> - I'd love to see performance numbers for less than 157118 notes. \n> Don't get me wrong, it is good to see the worst-case scenario in\n> terms of notes/commits ratio. But it will hardly be the common case,\n> and I very much would like to optimize for the common case.\n>\n> So, I'd appreciate if you could do the tests with something like\n> 500 notes, randomly spread over the commits (rationale: my original\n> understanding was that the notes could amend commit messages, and\n> that is much more likely to be done with relatively old commits that\n> you cannot change anymore).\n\nOk. I will try to test that.\n\n> Please understand that I might not have the time to participate in\n> this thread as much as I would like to. The next 4 days will be\n> especially hard.\n\nThanks for the feedback! I appreciate any time you're able to spend on \nthis. And I don't mind waiting for a few days for more feedback.\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122709","messageId":"alpine.DEB.1.00.0909081741590.4330@intel-tinevez-2-302","threadId":"20872","inReplyTo":"200909081436.30761.johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-09-08T15:53:06Z","receivedAt":"2009-09-08T15:53:06Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 8 Sep 2009, Johan Herland wrote:\n\n> On Tuesday 08 September 2009, Johannes Schindelin wrote:\n\n> > I can see that some people may think that date-based fan-out is the \n> > cat's ass, but I have to warn that we have no idea how notes will be \n> > used,\n> \n> I don't agree. Although we will certainly see many more use cases for \n> notes, I believe that the vast majority of them can be placed in one of \n> two categories:\n\nMy experience with Git is that having beliefs how my work is used was a \nconstant source of surprise.\n\n> > - I find the restriction to commits rather limiting.\n> \n> I see your point, but I don't agree until I see a compelling case for \n> annotating a non-commit.\n\nMy point is that it is too late by then, if you don't allow for a flexible \nand still efficient scheme.\n\n> > - most of the performance difference between the date-based and the \n> > SHA-1 based fan-out looks to me as if the issue was the top-level \n> > tree. Basically, this tree has to be read _every_ time _anybody_ \n> > wants to read a note.\n> \n> Not sure what you're trying to say here. The top-level notes tree is \n> read (as in fill_tree_descriptor()) exactly _once_. After that, it is \n> cached by the internal data structure (until free_commit_notes() or \n> end-of-process).\n\nBy that reasoning, we do not need any fan-out scheme.\n\nKeep in mind: reading a large tree object takes a long time. That's why \nwe started fan-out. Reading a large number of tree objects also takes a \nlong time. That's why I propagated flexible fan-out that is only read-in \non demand.\n\n> > But I think that having a dynamic fan-out that can even put blobs into \n> > the top-level tree (nothing prevents us from doing that, right?)\n> \n> Well, the \"flexible\" code does add the new requirement that all entries \n> in a notes (sub)tree object must follow the same scheme, i.e. you \n> cannot have:\n> \n> /12/34567890123456789012345678901234567890\n> /2345/678901234567890123456789012345678901\n> \n> but you can have\n> \n> /12/34567890123456789012345678901234567890\n> /23/45/678901234567890123456789012345678901\n\nUmm, why? Is there any good technical reason?\n\n> > The real question for me, therefore, is: what is the optimal way to \n> > strike the balance between size of the tree objects (which we want to \n> > be small, so that unpacking them is fast) and depth of the fan-out \n> > (which we want to be shallow to avoid reading worst-case 39 tree \n> > objects to get at one single note).\n> \n> s/39/19/ (each fanout must use at least 2 chars of the 40-char SHA1)\n\nThat is another unnecessary restriction that could cost you dearly. Just \nthink what happens if it turns out that the optimal number of tree items \nis closer to 16 than to 255...\n\n> Yes, the challenge is indeed striking the correct balance. I believe \n> that the notes code should be taught to write (and automatically \n> re-organize) the notes tree so that it is optimized for the current \n> collection of notes.\n\nOf course! I never thought that the user should be allowed to make the \nchoice how to organize the notes.\n\nCiao,\nDscho\n"},{"id":"122727","messageId":"7vocplxjov.fsf@alter.siamese.dyndns.org","threadId":"20872","inReplyTo":"alpine.DEB.1.00.0909081100020.4330@intel-tinevez-2-302","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-08T20:31:12Z","receivedAt":"2009-09-08T20:31:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Hi,\n>\n> On Tue, 8 Sep 2009, Johan Herland wrote:\n>\n>> Algorithm / Notes tree git log -n10 (x100) git log --all\n>> ------------------------------------------------------------\n>> next / no-notes 4.77s 63.84s\n>> \n>> before / no-notes 4.78s 63.90s\n>> before / no-fanout 56.85s 65.69s\n>> \n>> 16tree / no-notes 4.77s 64.18s\n>> 16tree / no-fanout 30.35s 65.39s\n>> 16tree / 2_38 5.57s 65.42s\n>> 16tree / 2_2_36 5.19s 65.76s\n>> \n>> flexible / no-notes 4.78s 63.91s\n>> flexible / no-fanout 30.34s 65.57s\n>> flexible / 2_38 5.57s 65.46s\n>> flexible / 2_2_36 5.18s 65.72s\n>> flexible / ym 5.13s 65.66s\n>> flexible / ym_2_38 5.08s 65.63s\n>> flexible / ymd 5.30s 65.45s\n>> flexible / ymd_2_38 5.29s 65.90s\n>> flexible / y_m 5.11s 65.72s\n>> flexible / y_m_2_38 5.08s 65.67s\n>> flexible / y_m_d 5.06s 65.50s\n>> flexible / y_m_d_2_38 5.07s 65.79s\n>\n> It's good to see that the no-notes behaves roughly like baseline.\n>\n> I can see that some people may think that date-based fan-out is the cat's \n> ass,\n\nActually, my knee-jerk reaction was that 4.77 (next) vs 5.57 (16tree with\n2_38) is already a good enough performance/simplicity tradeoff, and 5.57\nvs 5.08 (16tree with ym_2_38) probably does not justify the risk of worst\ncase behaviour that can come from possible mismatch between the access\npattern and the date-optimized tree layout.\n\nBut that only argues against supporting _only_ date-optimized layout.\n\nSupport of \"flexible layout\" is not that flexible as its name suggests;\none single note tree needs to have a uniform fanout strategy. But it is\nnot unusably rigid either; you only need to be extra careful when merging\ntwo notes trees. We can leave the heuristics to choose what the optimum\nlayout to later rounds.\n\n> - I find the restriction to commits rather limiting.\n\nYeah, we would not want to be surprised to find many people want to\nannotate non-commits with this mechanism.\n\n> - most of the performance difference between the date-based and the SHA-1 \n> based fan-out looks to me as if the issue was the top-level tree. \n> Basically, this tree has to be read _every_ time _anybody_ wants to read \n> a note.\n\nA comparison between 'next' and another algorithm that opens the top-level\nnotes tree object and returns \"I did not find any note\" without doing\nanything else would reveal that cost. But when you are doing \"log -n10\"\n(or \"log --all\"), you would read the notes top-level tree once, and it is\nlikely to be cached in the obj_hash[] (or in delta_base cache) already for\nthe remaining invocations, even if notes mechanism does not do its own\ncache, which I think it does, no?\n\n> - I'd love to see performance numbers for less than 157118 notes. Don't \n> get me wrong, it is good to see the worst-case scenario in terms of \n> notes/commits ratio. But it will hardly be the common case, and I \n> very much would like to optimize for the common case.\n>\n> So, I'd appreciate if you could do the tests with something like 500 \n> notes, randomly spread over the commits (rationale: my original \n> understanding was that the notes could amend commit messages, and that \n> is much more likely to be done with relatively old commits that you \n> cannot change anymore).\n\nHmph, is that a typical use case? How does it relate to CC's object\nreplacement mechanism?\n\nAlso Gitney talked about annotating commits in the code-review thing.\nWhat's the expected notes density and distribution in that application?\n"},{"id":"122732","messageId":"20090908211046.GV1033@spearce.org","threadId":"20872","inReplyTo":"7vocplxjov.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-09-08T21:10:46Z","receivedAt":"2009-09-08T21:10:46Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> Also Gitney talked about annotating commits in the code-review thing.\n> What's the expected notes density and distribution in that application?\n\nUh, try one note per commit in a project. A few merges won't need\na note, but nearly every single non-merge commit would.\n\nConsider a project with a velocity of about 200 non-merge\ncommits/day; the object count goes up fast.\n\nOne idea we are starting to kick around might double or quadruple\nthat number. If we store metadata about every version of every\ncommit ever proposed to a project, we need a lot more notes than\ncommits. Right now we have this sort of distribution from one of\nour servers:\n\nversions | commits \n---------+---------\n 1 | 9262\n 2 | 2626\n 3 | 1053\n 4 | 424\n 5 | 224\n 6 | 124\n 7 | 57\n 8 | 38\n 9 | 28\n 10 | 14\n 11 | 12\n 12 | 10\n 13 | 5\n 14 | 6\n 15 | 2\n 16 | 3\n 17 | 2\n 21 | 1\n 32 | 1\n\nSo most commits (66%) would have only 1 version (and 1 note)\nrelated to them in the note tree, but if I use the same note tree\nfor final commits as individual revisions considered, at least 18%\nof the commits in the final history of the project would actually\nhave two notes, and 7.5% would have 3 notes.\n\n-- \nShawn.\n"},{"id":"122734","messageId":"fabb9a1e0909081436x72c71b0bv51d8f198cd82f289@mail.gmail.com","threadId":"20872","inReplyTo":"20090908211046.GV1033@spearce.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-09-08T21:36:36Z","receivedAt":"2009-09-08T21:36:36Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, Sep 8, 2009 at 23:10, Shawn O. Pearce wrote:\n> So most commits (66%) would have only 1 version (and 1 note)\n> related to them in the note tree, but if I use the same note tree\n> for final commits as individual revisions considered, at least 18%\n> of the commits in the final history of the project would actually\n> have two notes, and 7.5% would have 3 notes.\n\nYou could however store all that information in one note, yes? Since\nthe 'latest version' is the one committed, you can include the notes\nfor all the previous ones at commit time?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"122735","messageId":"20090908213944.GX1033@spearce.org","threadId":"20872","inReplyTo":"fabb9a1e0909081436x72c71b0bv51d8f198cd82f289@mail.gmail.com","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-09-08T21:39:44Z","receivedAt":"2009-09-08T21:39:44Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sverre Rabbelier wrote:\n> On Tue, Sep 8, 2009 at 23:10, Shawn O. Pearce wrote:\n> > So most commits (66%) would have only 1 version (and 1 note)\n> > related to them in the note tree, but if I use the same note tree\n> > for final commits as individual revisions considered, at least 18%\n> > of the commits in the final history of the project would actually\n> > have two notes, and 7.5% would have 3 notes.\n> \n> You could however store all that information in one note, yes? Since\n> the 'latest version' is the one committed, you can include the notes\n> for all the previous ones at commit time?\n\nUh, but the natural way to index those is by commit, and each\ndifferent revision of a change is a different commit. Why delete\nthe prior revision information and move it to the final commit note?\nSomeone who has the prior revisions in their reflog and is doing\n`git log -g --notes` might want to see that annotation.\n\n-- \nShawn.\n"},{"id":"122736","messageId":"200909082340.05318.johan@herland.net","threadId":"20872","inReplyTo":"7vocplxjov.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T21:40:05Z","receivedAt":"2009-09-08T21:40:05Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 08 September 2009, Junio C Hamano wrote:\n> Johannes Schindelin writes:\n> > On Tue, 8 Sep 2009, Johan Herland wrote:\n> >> Algorithm / Notes tree git log -n10 (x100) git log --all\n> >> ------------------------------------------------------------\n> >> next / no-notes 4.77s 63.84s\n> >>\n> >> before / no-notes 4.78s 63.90s\n> >> before / no-fanout 56.85s 65.69s\n> >>\n> >> 16tree / no-notes 4.77s 64.18s\n> >> 16tree / no-fanout 30.35s 65.39s\n> >> 16tree / 2_38 5.57s 65.42s\n> >> 16tree / 2_2_36 5.19s 65.76s\n> >>\n> >> flexible / no-notes 4.78s 63.91s\n> >> flexible / no-fanout 30.34s 65.57s\n> >> flexible / 2_38 5.57s 65.46s\n> >> flexible / 2_2_36 5.18s 65.72s\n> >> flexible / ym 5.13s 65.66s\n> >> flexible / ym_2_38 5.08s 65.63s\n> >> flexible / ymd 5.30s 65.45s\n> >> flexible / ymd_2_38 5.29s 65.90s\n> >> flexible / y_m 5.11s 65.72s\n> >> flexible / y_m_2_38 5.08s 65.67s\n> >> flexible / y_m_d 5.06s 65.50s\n> >> flexible / y_m_d_2_38 5.07s 65.79s\n> >\n> > I can see that some people may think that date-based fan-out is the\n> > cat's ass,\n> \n> Actually, my knee-jerk reaction was that 4.77 (next) vs 5.57 (16tree with\n> 2_38) is already a good enough performance/simplicity tradeoff, and 5.57\n> vs 5.08 (16tree with ym_2_38) probably does not justify the risk of worst\n> case behaviour that can come from possible mismatch between the access\n> pattern and the date-optimized tree layout.\n\nYes, 16tree / 2_38 looks like a reasonable tradeoff when you look at the \nabsolute numbers, but it's also interesting to highlight the actual cost of \ndoing the notes lookup. In that case, we see that 16tree / 2_38 costs 0.80s, \nwhereas flexible / ym_2_38 only costs 0.31s, i.e. less than half the cost of \nthe former...\n\n> But that only argues against supporting _only_ date-optimized layout.\n> \n> Support of \"flexible layout\" is not that flexible as its name suggests;\n> one single note tree needs to have a uniform fanout strategy.\n\nActually, the uniform strategy is only required at each separate level. You \nare free to vary the strategy within independent subtrees. I.e. in the case \nwhere you have 1 note from 2007, and 1000 notes from 2008, you are free to \nuse a mix of date-based and SHA1-based structures, like this:\n\n y2007/1234567...\n y2008/m01/d01/2345678...\n y2008/m01/d01/3456789...\n y2008/m01/d02/45/67890...\n y2008/m01/d02/56/78901...\n y2008/m01/d02/67/89012...\n ...\n\n> > - I find the restriction to commits rather limiting.\n> \n> Yeah, we would not want to be surprised to find many people want to\n> annotate non-commits with this mechanism.\n\nWe could arbitrarily set the \"commit date\" for non-commit objects to the \nepoch, so that they can still be represented in a date-based fanout. (Of \ncourse, the notes code should be smart enough to choose a more optimal \nfanout if the number of non-commit notes is significant).\n\n> > - most of the performance difference between the date-based and the\n> > SHA-1 based fan-out looks to me as if the issue was the top-level tree.\n> > Basically, this tree has to be read _every_ time _anybody_ wants to\n> > read a note.\n> \n> A comparison between 'next' and another algorithm that opens the\n> top-level notes tree object and returns \"I did not find any note\"\n> without doing anything else would reveal that cost. But when you are\n> doing \"log -n10\" (or \"log --all\"), you would read the notes top-level\n> tree once, and it is likely to be cached in the obj_hash[] (or in\n> delta_base cache) already for the remaining invocations, even if notes\n> mechanism does not do its own cache, which I think it does, no?\n\nYes it does, since Dscho's original hash_map based implementation, in fact.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122739","messageId":"fabb9a1e0909081457q21d892fby66bdffc5b0199d32@mail.gmail.com","threadId":"20872","inReplyTo":"20090908213944.GX1033@spearce.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-09-08T21:57:34Z","receivedAt":"2009-09-08T21:57:34Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, Sep 8, 2009 at 23:39, Shawn O. Pearce wrote:\n> Uh, but the natural way to index those is by commit, and each\n> different revision of a change is a different commit.  Why delete\n> the prior revision information and move it to the final commit note?\n\nAh, I didn't realize you would push the notes before the final revision is made.\n\n> Someone who has the prior revisions in their reflog and is doing\n> `git log -g --notes` might want to see that annotation.\n\nIt would make more sense to have multiple notes then, but wouldn't you\nwant them to annotate the original commit, rather than the final one?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"122741","messageId":"200909090046.45213.johan@herland.net","threadId":"20872","inReplyTo":"alpine.DEB.1.00.0909081741590.4330@intel-tinevez-2-302","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-08T22:46:45Z","receivedAt":"2009-09-08T22:46:45Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 08 September 2009, Johannes Schindelin wrote:\n> On Tue, 8 Sep 2009, Johan Herland wrote:\n> > On Tuesday 08 September 2009, Johannes Schindelin wrote:\n> > > I can see that some people may think that date-based fan-out is the\n> > > cat's ass, but I have to warn that we have no idea how notes will be\n> > > used,\n> >\n> > I don't agree. Although we will certainly see many more use cases for\n> > notes, I believe that the vast majority of them can be placed in one of\n> > two categories:\n> \n> My experience with Git is that having beliefs how my work is used was a\n> constant source of surprise.\n\nAnd you believe that a system that only allows SHA1-based fanout schemes is \nbetter equipped to tackle such surprises than a system that provides both \ndate-based and SHA1-based fanout schemes?\n\n> > > - I find the restriction to commits rather limiting.\n> >\n> > I see your point, but I don't agree until I see a compelling case for\n> > annotating a non-commit.\n> \n> My point is that it is too late by then, if you don't allow for a\n> flexible and still efficient scheme.\n\nAs I replied to Junio, we could use the epoch as a \"commit date\" for tree \nand blob objects, thus making them representable in a date-based fanout \nscheme (although if there are a signficant number of non-commit notes, the \ncode should be smart enough to find a better (SHA1-based probably) fanout \nscheme for those notes).\n\n> > > - most of the performance difference between the date-based and the\n> > > SHA-1 based fan-out looks to me as if the issue was the top-level\n> > > tree. Basically, this tree has to be read _every_ time _anybody_\n> > > wants to read a note.\n> >\n> > Not sure what you're trying to say here. The top-level notes tree is\n> > read (as in fill_tree_descriptor()) exactly _once_. After that, it is\n> > cached by the internal data structure (until free_commit_notes() or\n> > end-of-process).\n> \n> By that reasoning, we do not need any fan-out scheme.\n> \n> Keep in mind: reading a large tree object takes a long time. That's why\n> we started fan-out. Reading a large number of tree objects also takes a\n> long time. That's why I propagated flexible fan-out that is only read-in\n> on demand.\n\nNot sure where you're going with this. Of course we want to strike an \noptimal balance between the size of tree objects and the number of tree \nobjects. Nobody is arguing about that. Both SHA1-based and (in the most \ncommon cases) date-based schemes can be used to achieve this balance. But \nusing date-based fanout has the added advantage of providing better \nperformance (both runtime- and memory-wise) when looking up notes in a \nchronological order.\n\n> > > But I think that having a dynamic fan-out that can even put blobs\n> > > into the top-level tree (nothing prevents us from doing that, right?)\n> >\n> > Well, the \"flexible\" code does add the new requirement that all entries\n> > in a notes (sub)tree object must follow the same scheme, i.e. you\n> > cannot have:\n> >\n> > /12/34567890123456789012345678901234567890\n> > /2345/678901234567890123456789012345678901\n> >\n> > but you can have\n> >\n> > /12/34567890123456789012345678901234567890\n> > /23/45/678901234567890123456789012345678901\n> \n> Umm, why? Is there any good technical reason?\n\nIn the date-based parts of notes tree, there are very good reasons for doing \nso: The code peeks at the first tree entry in order to determine what kind \nof date-based fanout is used in the current tree object. Subsequent entries \n(in that tree object) that do not follow the same format are \nskipped/ignored.\n\nThe SHA1-based fanout code has not changed since the last iteration, so this \nextra requirement is not absolutely necessary for the SHA1-based parts of \nthe notes tree. However, the extra requirement does guarantee that commit \nnotes have exactly one unique location in the notes tree, and thus relieves \nus of having to keep searching for alternative notes locations, and \nconcatenate the notes found.\n\n> > > The real question for me, therefore, is: what is the optimal way to\n> > > strike the balance between size of the tree objects (which we want to\n> > > be small, so that unpacking them is fast) and depth of the fan-out\n> > > (which we want to be shallow to avoid reading worst-case 39 tree\n> > > objects to get at one single note).\n> >\n> > s/39/19/ (each fanout must use at least 2 chars of the 40-char SHA1)\n> \n> That is another unnecessary restriction that could cost you dearly. Just\n> think what happens if it turns out that the optimal number of tree items\n> is closer to 16 than to 255...\n\nThe code can easily be rewritten to allow for \"odd\" fanouts (1/39, 1/1/38, \netc.). Feel free to submit a patch.\n\nI was, however, naive enough to assume that when git.git decided on using \n2/38 fanout for its loose objects, then some performance-related thoughts\nwent into that decision. If there are indications that multiple-of-2-type \nfanouts are not optimal, we should probably reconsider.\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122791","messageId":"20090910062311.GA27466@cuci.nl","threadId":"20872","inReplyTo":"200909090046.45213.johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2009-09-10T06:23:11Z","receivedAt":"2009-09-10T06:23:11Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Johan Herland wrote:\n>I was, however, naive enough to assume that when git.git decided on using \n>2/38 fanout for its loose objects, then some performance-related thoughts\n>went into that decision. If there are indications that multiple-of-2-type \n\nI presume that that choice was based on the fact that on a typical filesystem\n(e.g. ext2 old-style) without directory indexing, the sort-of optimal fanout\nshould aim for no more than 100-200 directory entries per directory.\n-- \nSincerely,\n Stephen R. van den Berg.\n\nMommy, what happens to your files when you die?\n"},{"id":"122797","messageId":"200909101125.35451.johan@herland.net","threadId":"20872","inReplyTo":"200909081436.30761.johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-10T09:25:35Z","receivedAt":"2009-09-10T09:25:35Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 08 September 2009, Johan Herland wrote:\n> On Tuesday 08 September 2009, Johannes Schindelin wrote:\n> > On Tue, 8 Sep 2009, Johan Herland wrote:\n> > > Algorithm / Notes tree git log -n10 (x100) git log --all\n> > > ------------------------------------------------------------\n> > > next / no-notes 4.77s 63.84s\n> > >\n> > > before / no-notes 4.78s 63.90s\n> > > before / no-fanout 56.85s 65.69s\n> > >\n> > > 16tree / no-notes 4.77s 64.18s\n> > > 16tree / no-fanout 30.35s 65.39s\n> > > 16tree / 2_38 5.57s 65.42s\n> > > 16tree / 2_2_36 5.19s 65.76s\n> > >\n> > > flexible / no-notes 4.78s 63.91s\n> > > flexible / no-fanout 30.34s 65.57s\n> > > flexible / 2_38 5.57s 65.46s\n> > > flexible / 2_2_36 5.18s 65.72s\n> > > flexible / ym 5.13s 65.66s\n> > > flexible / ym_2_38 5.08s 65.63s\n> > > flexible / ymd 5.30s 65.45s\n> > > flexible / ymd_2_38 5.29s 65.90s\n> > > flexible / y_m 5.11s 65.72s\n> > > flexible / y_m_2_38 5.08s 65.67s\n> > > flexible / y_m_d 5.06s 65.50s\n> > > flexible / y_m_d_2_38 5.07s 65.79s\n\n[snip]\n\n> > - I'd love to see performance numbers for less than 157118 notes.\n> > Don't get me wrong, it is good to see the worst-case scenario in\n> > terms of notes/commits ratio. But it will hardly be the common case,\n> > and I very much would like to optimize for the common case.\n> >\n> > So, I'd appreciate if you could do the tests with something like\n> > 500 notes, randomly spread over the commits (rationale: my original\n> > understanding was that the notes could amend commit messages, and\n> > that is much more likely to be done with relatively old commits that\n> > you cannot change anymore).\n> \n> Ok. I will try to test that.\n\nHere are the results of the 500-notes-in-kernel-repo test:\n\nAlgorithm / Notes tree git log -n10 (x100) git log --all\n\nnext / no-notes 4.83s 64.78s\n\nbefore / no-notes 4.84s 64.76s\nbefore / no-fanout 4.98s 64.89s\n\n16tree / no-notes 4.84s 64.61s\n16tree / no-fanout 4.92s 64.68s\n16tree / 2_38 4.85s 64.45s\n16tree / 2_2_36 4.85s 64.63s\n\nflexible / no-notes 4.84s 64.82s\nflexible / no-fanout 4.91s 65.01s\nflexible / 2_38 4.85s 64.93s\nflexible / 2_2_36 4.85s 64.63s\nflexible / ym 4.83s 64.63s\nflexible / ym_2_38 4.86s 64.72s\nflexible / ymd 4.91s 64.74s\nflexible / ymd_2_38 4.91s 64.56s\nflexible / y_m 4.86s 64.76s\nflexible / y_m_2_38 4.86s 64.71s\nflexible / y_m_d 4.86s 64.73s\nflexible / y_m_d_2_38 4.84s 64.50s\n\nI don't like the noise level in the second column ('git log --all'). Then \nagain, I don't find that column very interesting (it's mostly there to \nverify that we don't have any abysmal worst-case behaviours in the notes \ncode).\n\nThe first column is fairly nice and tidy, though. At a first glance it shows \npretty much the same results as the 157000-notes table previously posted. \nObviously the abysmal performance of no-fanout is gone (500 notes in a \nsingle tree object is not _that_ bad), although a 2/38-fanout is still a \nbetter choice for 500 notes (but 2/2/36 does not provide any additional \nimprovement).\n\n>From this we can start to guess that the threshold for moving from no fanout \nto 2/38 is somewhere below 500 notes, while the theshold for moving from \n2/38 to 2/2/36 is between 500 and ~157000 notes (probably much closer to \n157000 than to 500; I wouldn't be surprised if ~256 entries per level turns \nout to be good a threshold).\n\nThe date-based fanout performs on par with the SHA1-based fanout, although \nit's hard to say anything conclusively when the numbers are as close as \nthis. However, the ymd and ymd_2_38 fanout probably show signs of too much \noverhead (too many levels) at only 500 notes. This is not surprising.\n\nMy gut feeling tells me that moving from 'no-fanout' to either '2_38' or \n'ym' is a good idea at ~256 notes. Then, if we went with '2_38', we'd have \nto switch to '2_2_36' at ~64K notes (i.e. when each /38 level reaches ~256 \nnotes) However, it seems that with 'ym', we could stick with it for much \nlonger before having to consider switching to a different fanout alternative \n(probably 'ym_2_38' or 'y_m_d').\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122806","messageId":"8445CEA3-AC5D-4A38-9C73-B4E14BD4864C@adacore.com","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2009-09-10T14:00:18Z","receivedAt":"2009-09-10T14:00:18Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"On Sep 7, 2009, at 22:26, Johan Herland wrote:\n> Yet another iteration of the 'git notes' feature. Rebased on top of \n> 'next':\n> - Patches 1-9 are unchanged from (patches 1-7, 11-12 of) the last \n> iteration.\n> - Patch 10 teaches the notes code to free its data structures on \n> request.\n> - Patch 11 introduces the 16-tree notes lookup code that handles \n> SHA1-based\n> fanout schemes. This is pretty much unchanged from patch 8 in the \n> previous\n> iteration.\n> - Patch 12 adds selftests that verify correct parsing of notes trees \n> with\n> various SHA1-based fanouts.\n> - Patch 13 introduces a flexible parser for a variety of date-based \n> and\n> SHA1-based fanout schemes. This is the interesting part, as far as \n> this\n> iteration is concerned.\n> - Patch 14 adds selftests that verify correct parsing of notes trees \n> with\n> various date-based fanouts.\n>\n> Note that the series does not yet include code for _writing_ notes \n> into a\n> suitably structured notes tree. That will be done in a later \n> iteration.\n>\n> I have some performance numbers that I will send in a separate email.\n\nHi Johan,\n\nI've been following this series with some interest, and am curious\nwhy notes need to be stored in a separate data structure from regular\nobjects. Note that I'm not questioning the design (and certainly would\nnot want to, this late in the process), rather I'd like to learn\nabout the reasons.\n\nI've wondered about this as well in the context of refs, reflog and\ngit config. In a completely unified model, every change to the\nrepository (except for the index, pack indices and working directory)\nwould be a commit of the .git/ directory (again excluding indices).\nOne of the advantages (besides allowing configuration management\nof the repository itself in addition to its contents) would be that\nno locking is ever required.\n\nThis would be just an implementation detail without necessarily\naffecting the user interface other than direct inspection/modification\nof the .git directory, which is a similar to the move to packed refs.\nAgain, I'm not proposing to change anything, just wondering about\ndesign rationale.\n\n -Geert\n"},{"id":"122807","messageId":"4AA9089D.3000507@drmicha.warpmail.net","threadId":"20872","inReplyTo":"8445CEA3-AC5D-4A38-9C73-B4E14BD4864C@adacore.com","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2009-09-10T14:09:33Z","receivedAt":"2009-09-10T14:09:33Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Geert Bosch venit, vidit, dixit 10.09.2009 16:00:\n> On Sep 7, 2009, at 22:26, Johan Herland wrote:\n...\n> \n> Hi Johan,\n> \n> I've been following this series with some interest, and am curious\n> why notes need to be stored in a separate data structure from regular\n> objects. Note that I'm not questioning the design (and certainly would\n\nIt's not separate, that's the point. They're stored as objects in trees,\njust like anything else. The discussion about the structure is about how\nto organize the tree structure, not actual subdirectories under .git/.\n\n> not want to, this late in the process), rather I'd like to learn\n> about the reasons.\n> \n> I've wondered about this as well in the context of refs, reflog and\n> git config. In a completely unified model, every change to the\n> repository (except for the index, pack indices and working directory)\n> would be a commit of the .git/ directory (again excluding indices).\n> One of the advantages (besides allowing configuration management\n> of the repository itself in addition to its contents) would be that\n> no locking is ever required.\n\n...and one of the disadvantages that you're not in control of your\nconfig any more, if you pull from upstream. config and reflog are\nsomething inherently private and local. The reflog does not even make\nsense other than in a local (per repo) context.\n\nFor the config, one may think up a solution where parts of config are\nshared (by storing them as objects and referencing them) and git asks\nyou before changing anything on pull/fetch. In a sense git submodule\ndoes that already.\n\nCheers,\nMichael\n"},{"id":"122809","messageId":"DAAED274-A4D5-40A2-9A5F-3F84D6D424AF@adacore.com","threadId":"20872","inReplyTo":"4AA9089D.3000507@drmicha.warpmail.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2009-09-10T14:12:10Z","receivedAt":"2009-09-10T14:12:10Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn Sep 10, 2009, at 10:09, Michael J Gruber wrote:\n\n> It's not separate, that's the point. They're stored as objects in \n> trees,\n> just like anything else. The discussion about the structure is about \n> how\n> to organize the tree structure, not actual subdirectories under .git/.\n\nArghh, sorry for the noise.\n\n -Geert\n"},{"id":"122927","messageId":"7viqfpowc0.fsf@alter.siamese.dyndns.org","threadId":"20872","inReplyTo":"1252376822-6138-1-git-send-email-johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-12T00:11:59Z","receivedAt":"2009-09-12T00:11:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> Yet another iteration of the 'git notes' feature. Rebased on top of 'next':\n\nBy the way I didn't pick this up as it did not apply to any of my\nbranches.\n"},{"id":"122950","messageId":"200909121750.00733.johan@herland.net","threadId":"20872","inReplyTo":"200909080512.34634.johan@herland.net","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T15:50:00Z","receivedAt":"2009-09-12T15:50:00Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 08 September 2009, Johan Herland wrote:\n> Algorithm / Notes tree git log -n10 (x100) git log --all\n> ------------------------------------------------------------\n> before / no-notes 4.78s 63.90s\n> before / no-fanout 56.85s 65.69s\n> \n> 16tree / no-notes 4.77s 64.18s\n> 16tree / no-fanout 30.35s 65.39s\n> 16tree / 2_38 5.57s 65.42s\n> 16tree / 2_2_36 5.19s 65.76s\n> \n> flexible / no-notes 4.78s 63.91s\n> flexible / no-fanout 30.34s 65.57s\n> flexible / 2_38 5.57s 65.46s\n> flexible / 2_2_36 5.18s 65.72s\n> flexible / ym 5.13s 65.66s\n> flexible / ym_2_38 5.08s 65.63s\n> flexible / ymd 5.30s 65.45s\n> flexible / ymd_2_38 5.29s 65.90s\n> flexible / y_m 5.11s 65.72s\n> flexible / y_m_2_38 5.08s 65.67s\n> flexible / y_m_d 5.06s 65.50s\n> flexible / y_m_d_2_38 5.07s 65.79s\n\nOk, I have been pondering this back and forth, and I'm not sure what to \nthink. It seems allowing (not mandating) date-based fanout gives a slight \nruntime advantage if used correctly, but I'm not sure the slight runtime \nimprovement is worth the added code complexity and worse maintainability. \nI'm starting to lean against SHA1-based fanout being \"good enough\".\n\nBut when we look at the memory consumption, it's clear that SHA1-based \nfanout loses out (because you cannot throw away subtrees without fear that \nthey will be needed again soon). Then again, memory consumption has not been \nthe major focus of the git project, and 14 MB (for holding all ~157000 notes \nin the kernel repo example) is not excessive for an average desktop \ncomputer.\n\nShawn, do you have any additional defence for the date-based fanout? Are \nthere untested reasonable scenarios that would show the benefits of date-\nbased fanout? How does the plan for notes usage in your code-review thingy \ncompare to my test scenario?\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122949","messageId":"200909121752.07523.johan@herland.net","threadId":"20872","inReplyTo":"7viqfpowc0.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T15:52:07Z","receivedAt":"2009-09-12T15:52:07Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Saturday 12 September 2009, Junio C Hamano wrote:\n> Johan Herland writes:\n> > Yet another iteration of the 'git notes' feature. Rebased on top of\n> > 'next':\n> \n> By the way I didn't pick this up as it did not apply to any of my\n> branches.\n\nWeird... I just rebased it from an old 'next' to current 'next' without any \nconflicts.\n\nWill resend immediately.\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122951","messageId":"1252771728-27206-1-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:34Z","receivedAt":"2009-09-12T16:08:34Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The sixth iteration of the 'git notes' feature. Rebased on top of current\n'next', otherwise unchanged from v5.\n\n\nHave fun! :)\n\n...Johan\n\n\nJohan Herland (9):\n Teach \"-m \" and \"-F \" to \"git notes edit\"\n fast-import: Add support for importing commit notes\n t3302-notes-index-expensive: Speed up create_repo()\n Add flags to get_commit_notes() to control the format of the note string\n Teach notes code to free its internal data structures on request.\n Teach the notes lookup code to parse notes trees with various fanout schemes\n Selftests verifying semantics when loading notes trees with various fanouts\n Allow flexible organization of notes trees, using both commit date and SHA1\n Add test cases for various date-based fanouts\n\nJohannes Schindelin (5):\n Introduce commit notes\n Add a script to edit/inspect notes\n Speed up git notes lookup\n Add an expensive test for git-notes\n Add '%N'-format for pretty-printing commit notes\n\n .gitignore | 1 +\n Documentation/config.txt | 13 +\n Documentation/git-fast-import.txt | 45 +++-\n Documentation/git-notes.txt | 60 ++++\n Documentation/pretty-formats.txt | 1 +\n Makefile | 3 +\n cache.h | 4 +\n command-list.txt | 1 +\n commit.c | 1 +\n config.c | 5 +\n environment.c | 1 +\n fast-import.c | 88 +++++-\n git-notes.sh | 121 +++++++\n notes.c | 673 +++++++++++++++++++++++++++++++++++++\n notes.h | 12 +\n pretty.c | 10 +\n t/t3301-notes.sh | 150 ++++++++\n t/t3302-notes-index-expensive.sh | 118 +++++++\n t/t3303-notes-subtrees.sh | 201 +++++++++++\n t/t9300-fast-import.sh | 166 +++++++++\n 20 files changed, 1664 insertions(+), 10 deletions(-)\n create mode 100644 Documentation/git-notes.txt\n create mode 100755 git-notes.sh\n create mode 100644 notes.c\n create mode 100644 notes.h\n create mode 100755 t/t3301-notes.sh\n create mode 100755 t/t3302-notes-index-expensive.sh\n create mode 100755 t/t3303-notes-subtrees.sh\n"},{"id":"122952","messageId":"1252771728-27206-2-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 01/14] Introduce commit notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:35Z","receivedAt":"2009-09-12T16:08:35Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nCommit notes are blobs which are shown together with the commit\nmessage. These blobs are taken from the notes ref, which you can\nconfigure by the config variable core.notesRef, which in turn can\nbe overridden by the environment variable GIT_NOTES_REF.\n\nThe notes ref is a branch which contains \"files\" whose names are\nthe names of the corresponding commits (i.e. the SHA-1).\n\nThe rationale for putting this information into a ref is this: we\nwant to be able to fetch and possibly union-merge the notes,\nmaybe even look at the date when a note was introduced, and we\nwant to store them efficiently together with the other objects.\n\nThis patch has been improved by the following contributions:\n- Thomas Rast: fix core.notesRef documentation\n- Tor Arne Vestbø: fix printing of multi-line notes\n- Alex Riesen: Using char array instead of char pointer costs less BSS\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Thomas Rast \nSigned-off-by: Tor Arne Vestbø \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n Documentation/config.txt | 13 +++++++++\n Makefile | 2 +\n cache.h | 4 +++\n commit.c | 1 +\n config.c | 5 +++\n environment.c | 1 +\n notes.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++\n notes.h | 7 +++++\n pretty.c | 5 +++\n 9 files changed, 106 insertions(+), 0 deletions(-)\n create mode 100644 notes.c\n create mode 100644 notes.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex cc156b8..32b0cdf 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -458,6 +458,19 @@ On some file system/operating system combinations, this is unreliable.\n Set this config setting to 'rename' there; However, This will remove the\n check that makes sure that existing object files will not get overwritten.\n \n+core.notesRef::\n+\tWhen showing commit messages, also show notes which are stored in\n+\tthe given ref. This ref is expected to contain files named\n+\tafter the full SHA-1 of the commit they annotate.\n++\n+If such a file exists in the given ref, the referenced blob is read, and\n+appended to the commit message, separated by a \"Notes:\" line. If the\n+given ref itself does not exist, it is not an error, but means that no\n+notes should be printed.\n++\n+This setting defaults to \"refs/notes/commits\", and can be overridden by\n+the `GIT_NOTES_REF` environment variable.\n+\n add.ignore-errors::\n \tTells 'git-add' to continue adding files when some files cannot be\n \tadded due to indexing errors. Equivalent to the '--ignore-errors'\ndiff --git a/Makefile b/Makefile\nindex bde2acd..09180ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -429,6 +429,7 @@ LIB_H += ll-merge.h\n LIB_H += log-tree.h\n LIB_H += mailmap.h\n LIB_H += merge-recursive.h\n+LIB_H += notes.h\n LIB_H += object.h\n LIB_H += pack.h\n LIB_H += pack-refs.h\n@@ -513,6 +514,7 @@ LIB_OBJS += match-trees.o\n LIB_OBJS += merge-file.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += name-hash.o\n+LIB_OBJS += notes.o\n LIB_OBJS += object.o\n LIB_OBJS += pack-check.o\n LIB_OBJS += pack-refs.o\ndiff --git a/cache.h b/cache.h\nindex 30a7a16..3d1a355 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -372,6 +372,8 @@ static inline enum object_type object_type(unsigned int mode)\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n+#define GIT_NOTES_REF_ENVIRONMENT \"GIT_NOTES_REF\"\n+#define GIT_NOTES_DEFAULT_REF \"refs/notes/commits\"\n \n extern int is_bare_repository_cfg;\n extern int is_bare_repository(void);\n@@ -566,6 +568,8 @@ enum object_creation_mode {\n \n extern enum object_creation_mode object_creation_mode;\n \n+extern char *notes_ref_name;\n+\n extern int grafts_replace_parents;\n \n #define GIT_REPO_VERSION 0\ndiff --git a/commit.c b/commit.c\nindex a6c6f70..a0a77a6 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -5,6 +5,7 @@\n #include \"utf8.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n+#include \"notes.h\"\n \n int save_commit_buffer = 1;\n \ndiff --git a/config.c b/config.c\nindex f21530c..42bef56 100644\n--- a/config.c\n+++ b/config.c\n@@ -467,6 +467,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.notesref\")) {\n+\t\tnotes_ref_name = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.pager\"))\n \t\treturn git_config_string(&pager_program, var, value);\n \ndiff --git a/environment.c b/environment.c\nindex 5de6837..571ab56 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -49,6 +49,7 @@ enum push_default_type push_default = PUSH_DEFAULT_MATCHING;\n #define OBJECT_CREATION_MODE OBJECT_CREATION_USES_HARDLINKS\n #endif\n enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n+char *notes_ref_name;\n int grafts_replace_parents = 1;\n \n /* Parallel index stat data preload? */\ndiff --git a/notes.c b/notes.c\nnew file mode 100644\nindex 0000000..401966d\n--- /dev/null\n+++ b/notes.c\n@@ -0,0 +1,68 @@\n+#include \"cache.h\"\n+#include \"commit.h\"\n+#include \"notes.h\"\n+#include \"refs.h\"\n+#include \"utf8.h\"\n+#include \"strbuf.h\"\n+\n+static int initialized;\n+\n+void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n+\t\tconst char *output_encoding)\n+{\n+\tstatic const char utf8[] = \"utf-8\";\n+\tstruct strbuf name = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\tchar *msg, *msg_p;\n+\tunsigned long linelen, msglen;\n+\tenum object_type type;\n+\n+\tif (!initialized) {\n+\t\tconst char *env = getenv(GIT_NOTES_REF_ENVIRONMENT);\n+\t\tif (env)\n+\t\t\tnotes_ref_name = getenv(GIT_NOTES_REF_ENVIRONMENT);\n+\t\telse if (!notes_ref_name)\n+\t\t\tnotes_ref_name = GIT_NOTES_DEFAULT_REF;\n+\t\tif (notes_ref_name && read_ref(notes_ref_name, sha1))\n+\t\t\tnotes_ref_name = NULL;\n+\t\tinitialized = 1;\n+\t}\n+\n+\tif (!notes_ref_name)\n+\t\treturn;\n+\n+\tstrbuf_addf(&name, \"%s:%s\", notes_ref_name,\n+\t\t\tsha1_to_hex(commit->object.sha1));\n+\tif (get_sha1(name.buf, sha1))\n+\t\treturn;\n+\n+\tif (!(msg = read_sha1_file(sha1, &type, &msglen)) || !msglen ||\n+\t\t\ttype != OBJ_BLOB)\n+\t\treturn;\n+\n+\tif (output_encoding && *output_encoding &&\n+\t\t\tstrcmp(utf8, output_encoding)) {\n+\t\tchar *reencoded = reencode_string(msg, output_encoding, utf8);\n+\t\tif (reencoded) {\n+\t\t\tfree(msg);\n+\t\t\tmsg = reencoded;\n+\t\t\tmsglen = strlen(msg);\n+\t\t}\n+\t}\n+\n+\t/* we will end the annotation by a newline anyway */\n+\tif (msglen && msg[msglen - 1] == '\\n')\n+\t\tmsglen--;\n+\n+\tstrbuf_addstr(sb, \"\\nNotes:\\n\");\n+\n+\tfor (msg_p = msg; msg_p < msg + msglen; msg_p += linelen + 1) {\n+\t\tlinelen = strchrnul(msg_p, '\\n') - msg_p;\n+\n+\t\tstrbuf_addstr(sb, \" \");\n+\t\tstrbuf_add(sb, msg_p, linelen);\n+\t\tstrbuf_addch(sb, '\\n');\n+\t}\n+\n+\tfree(msg);\n+}\ndiff --git a/notes.h b/notes.h\nnew file mode 100644\nindex 0000000..79d21b6\n--- /dev/null\n+++ b/notes.h\n@@ -0,0 +1,7 @@\n+#ifndef NOTES_H\n+#define NOTES_H\n+\n+void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n+\t\tconst char *output_encoding);\n+\n+#endif\ndiff --git a/pretty.c b/pretty.c\nindex f5983f8..e25db81 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -6,6 +6,7 @@\n #include \"string-list.h\"\n #include \"mailmap.h\"\n #include \"log-tree.h\"\n+#include \"notes.h\"\n #include \"color.h\"\n \n static char *user_format;\n@@ -975,5 +976,9 @@ void pretty_print_commit(enum cmit_fmt fmt, const struct commit *commit,\n \t */\n \tif (fmt == CMIT_FMT_EMAIL && sb->len <= beginning_of_body)\n \t\tstrbuf_addch(sb, '\\n');\n+\n+\tif (fmt != CMIT_FMT_ONELINE)\n+\t\tget_commit_notes(commit, sb, encoding);\n+\n \tfree(reencoded);\n }\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122958","messageId":"1252771728-27206-3-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 02/14] Add a script to edit/inspect notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:36Z","receivedAt":"2009-09-12T16:08:36Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nThe script 'git notes' allows you to edit and show commit notes, by\ncalling either\n\n\tgit notes show \n\nor\n\n\tgit notes edit \n\nThis patch has been improved by the following contributions:\n- Tor Arne Vestbø: fix printing of multi-line notes\n- Michael J Gruber: test and handle empty notes gracefully\n- Thomas Rast:\n - only clean up message file when editing\n - use GIT_EDITOR and core.editor over VISUAL/EDITOR\n - t3301: fix confusing quoting in test for valid notes ref\n - t3301: use test_must_fail instead of !\n - refuse to edit notes outside refs/notes/\n- Junio C Hamano: tests: fix \"export var=val\"\n- Christian Couder: documentation: fix 'linkgit' macro in \"git-notes.txt\"\n- Johan Herland: minor cleanup and bugfixing in git-notes.sh (v2)\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Tor Arne Vestbø \nSigned-off-by: Michael J Gruber \nSigned-off-by: Thomas Rast \nSigned-off-by: Christian Couder \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Documentation/git-notes.txt | 46 +++++++++++++++++\n Makefile | 1 +\n command-list.txt | 1 +\n git-notes.sh | 73 +++++++++++++++++++++++++++\n t/t3301-notes.sh | 114 +++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 236 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-notes.txt\n create mode 100755 git-notes.sh\n create mode 100755 t/t3301-notes.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 47672b0..c7f9960 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -86,6 +86,7 @@ git-mktag\n git-mktree\n git-name-rev\n git-mv\n+git-notes\n git-pack-redundant\n git-pack-objects\n git-pack-refs\ndiff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt\nnew file mode 100644\nindex 0000000..7136016\n--- /dev/null\n+++ b/Documentation/git-notes.txt\n@@ -0,0 +1,46 @@\n+git-notes(1)\n+============\n+\n+NAME\n+----\n+git-notes - Add/inspect commit notes\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git-notes' (edit | show) [commit]\n+\n+DESCRIPTION\n+-----------\n+This command allows you to add notes to commit messages, without\n+changing the commit. To discern these notes from the message stored\n+in the commit object, the notes are indented like the message, after\n+an unindented line saying \"Notes:\".\n+\n+To disable commit notes, you have to set the config variable\n+core.notesRef to the empty string. Alternatively, you can set it\n+to a different ref, something like \"refs/notes/bugzilla\". This setting\n+can be overridden by the environment variable \"GIT_NOTES_REF\".\n+\n+\n+SUBCOMMANDS\n+-----------\n+\n+edit::\n+\tEdit the notes for a given commit (defaults to HEAD).\n+\n+show::\n+\tShow the notes for a given commit (defaults to HEAD).\n+\n+\n+Author\n+------\n+Written by Johannes Schindelin \n+\n+Documentation\n+-------------\n+Documentation by Johannes Schindelin\n+\n+GIT\n+---\n+Part of the linkgit:git[7] suite\ndiff --git a/Makefile b/Makefile\nindex 09180ac..6d84be1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -318,6 +318,7 @@ SCRIPT_SH += git-merge-one-file.sh\n SCRIPT_SH += git-merge-resolve.sh\n SCRIPT_SH += git-mergetool.sh\n SCRIPT_SH += git-mergetool--lib.sh\n+SCRIPT_SH += git-notes.sh\n SCRIPT_SH += git-parse-remote.sh\n SCRIPT_SH += git-pull.sh\n SCRIPT_SH += git-quiltimport.sh\ndiff --git a/command-list.txt b/command-list.txt\nindex fb03a2e..4296941 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -74,6 +74,7 @@ git-mktag plumbingmanipulators\n git-mktree plumbingmanipulators\n git-mv mainporcelain common\n git-name-rev plumbinginterrogators\n+git-notes mainporcelain\n git-pack-objects plumbingmanipulators\n git-pack-redundant plumbinginterrogators\n git-pack-refs ancillarymanipulators\ndiff --git a/git-notes.sh b/git-notes.sh\nnew file mode 100755\nindex 0000000..f06c254\n--- /dev/null\n+++ b/git-notes.sh\n@@ -0,0 +1,73 @@\n+#!/bin/sh\n+\n+USAGE=\"(edit | show) [commit]\"\n+. git-sh-setup\n+\n+test -n \"$3\" && usage\n+\n+test -z \"$1\" && usage\n+ACTION=\"$1\"; shift\n+\n+test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"$(git config core.notesref)\"\n+test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"refs/notes/commits\"\n+\n+COMMIT=$(git rev-parse --verify --default HEAD \"$@\") ||\n+die \"Invalid commit: $@\"\n+\n+case \"$ACTION\" in\n+edit)\n+\tif [ \"${GIT_NOTES_REF#refs/notes/}\" = \"$GIT_NOTES_REF\" ]; then\n+\t\tdie \"Refusing to edit notes in $GIT_NOTES_REF (outside of refs/notes/)\"\n+\tfi\n+\n+\tMSG_FILE=\"$GIT_DIR/new-notes-$COMMIT\"\n+\tGIT_INDEX_FILE=\"$MSG_FILE.idx\"\n+\texport GIT_INDEX_FILE\n+\n+\ttrap '\n+\t\ttest -f \"$MSG_FILE\" && rm \"$MSG_FILE\"\n+\t\ttest -f \"$GIT_INDEX_FILE\" && rm \"$GIT_INDEX_FILE\"\n+\t' 0\n+\n+\tGIT_NOTES_REF= git log -1 $COMMIT | sed \"s/^/#/\" > \"$MSG_FILE\"\n+\n+\tCURRENT_HEAD=$(git show-ref \"$GIT_NOTES_REF\" | cut -f 1 -d ' ')\n+\tif [ -z \"$CURRENT_HEAD\" ]; then\n+\t\tPARENT=\n+\telse\n+\t\tPARENT=\"-p $CURRENT_HEAD\"\n+\t\tgit read-tree \"$GIT_NOTES_REF\" || die \"Could not read index\"\n+\t\tgit cat-file blob :$COMMIT >> \"$MSG_FILE\" 2> /dev/null\n+\tfi\n+\n+\tcore_editor=\"$(git config core.editor)\"\n+\t${GIT_EDITOR:-${core_editor:-${VISUAL:-${EDITOR:-vi}}}} \"$MSG_FILE\"\n+\n+\tgrep -v ^# < \"$MSG_FILE\" | git stripspace > \"$MSG_FILE\".processed\n+\tmv \"$MSG_FILE\".processed \"$MSG_FILE\"\n+\tif [ -s \"$MSG_FILE\" ]; then\n+\t\tBLOB=$(git hash-object -w \"$MSG_FILE\") ||\n+\t\t\tdie \"Could not write into object database\"\n+\t\tgit update-index --add --cacheinfo 0644 $BLOB $COMMIT ||\n+\t\t\tdie \"Could not write index\"\n+\telse\n+\t\ttest -z \"$CURRENT_HEAD\" &&\n+\t\t\tdie \"Will not initialise with empty tree\"\n+\t\tgit update-index --force-remove $COMMIT ||\n+\t\t\tdie \"Could not update index\"\n+\tfi\n+\n+\tTREE=$(git write-tree) || die \"Could not write tree\"\n+\tNEW_HEAD=$(echo Annotate $COMMIT | git commit-tree $TREE $PARENT) ||\n+\t\tdie \"Could not annotate\"\n+\tgit update-ref -m \"Annotate $COMMIT\" \\\n+\t\t\"$GIT_NOTES_REF\" $NEW_HEAD $CURRENT_HEAD\n+;;\n+show)\n+\tgit rev-parse -q --verify \"$GIT_NOTES_REF\":$COMMIT > /dev/null ||\n+\t\tdie \"No note for commit $COMMIT.\"\n+\tgit show \"$GIT_NOTES_REF\":$COMMIT\n+;;\n+*)\n+\tusage\n+esac\ndiff --git a/t/t3301-notes.sh b/t/t3301-notes.sh\nnew file mode 100755\nindex 0000000..73e53be\n--- /dev/null\n+++ b/t/t3301-notes.sh\n@@ -0,0 +1,114 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Johannes E. Schindelin\n+#\n+\n+test_description='Test commit notes'\n+\n+. ./test-lib.sh\n+\n+cat > fake_editor.sh << \\EOF\n+echo \"$MSG\" > \"$1\"\n+echo \"$MSG\" >& 2\n+EOF\n+chmod a+x fake_editor.sh\n+VISUAL=./fake_editor.sh\n+export VISUAL\n+\n+test_expect_success 'cannot annotate non-existing HEAD' '\n+\t(MSG=3 && export MSG && test_must_fail git notes edit)\n+'\n+\n+test_expect_success setup '\n+\t: > a1 &&\n+\tgit add a1 &&\n+\ttest_tick &&\n+\tgit commit -m 1st &&\n+\t: > a2 &&\n+\tgit add a2 &&\n+\ttest_tick &&\n+\tgit commit -m 2nd\n+'\n+\n+test_expect_success 'need valid notes ref' '\n+\t(MSG=1 GIT_NOTES_REF=/ && export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes edit) &&\n+\t(MSG=2 GIT_NOTES_REF=/ && export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes show)\n+'\n+\n+test_expect_success 'refusing to edit in refs/heads/' '\n+\t(MSG=1 GIT_NOTES_REF=refs/heads/bogus &&\n+\t export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes edit)\n+'\n+\n+test_expect_success 'refusing to edit in refs/remotes/' '\n+\t(MSG=1 GIT_NOTES_REF=refs/remotes/bogus &&\n+\t export MSG GIT_NOTES_REF &&\n+\t test_must_fail git notes edit)\n+'\n+\n+# 1 indicates caught gracefully by die, 128 means git-show barked\n+test_expect_success 'handle empty notes gracefully' '\n+\tgit notes show ; test 1 = $?\n+'\n+\n+test_expect_success 'create notes' '\n+\tgit config core.notesRef refs/notes/commits &&\n+\tMSG=b1 git notes edit &&\n+\ttest ! -f .git/new-notes &&\n+\ttest 1 = $(git ls-tree refs/notes/commits | wc -l) &&\n+\ttest b1 = $(git notes show) &&\n+\tgit show HEAD^ &&\n+\ttest_must_fail git notes show HEAD^\n+'\n+\n+cat > expect << EOF\n+commit 268048bfb8a1fb38e703baceb8ab235421bf80c5\n+Author: A U Thor \n+Date: Thu Apr 7 15:14:13 2005 -0700\n+\n+ 2nd\n+\n+Notes:\n+ b1\n+EOF\n+\n+test_expect_success 'show notes' '\n+\t! (git cat-file commit HEAD | grep b1) &&\n+\tgit log -1 > output &&\n+\ttest_cmp expect output\n+'\n+test_expect_success 'create multi-line notes (setup)' '\n+\t: > a3 &&\n+\tgit add a3 &&\n+\ttest_tick &&\n+\tgit commit -m 3rd &&\n+\tMSG=\"b3\n+c3c3c3c3\n+d3d3d3\" git notes edit\n+'\n+\n+cat > expect-multiline << EOF\n+commit 1584215f1d29c65e99c6c6848626553fdd07fd75\n+Author: A U Thor \n+Date: Thu Apr 7 15:15:13 2005 -0700\n+\n+ 3rd\n+\n+Notes:\n+ b3\n+ c3c3c3c3\n+ d3d3d3\n+EOF\n+\n+printf \"\\n\" >> expect-multiline\n+cat expect >> expect-multiline\n+\n+test_expect_success 'show multi-line notes' '\n+\tgit log -2 > output &&\n+\ttest_cmp expect-multiline output\n+'\n+\n+test_done\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122954","messageId":"1252771728-27206-4-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 03/14] Speed up git notes lookup","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:37Z","receivedAt":"2009-09-12T16:08:37Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nTo avoid looking up each and every commit in the notes ref's tree\nobject, which is very expensive, speed things up by slurping the tree\nobject's contents into a hash_map.\n\nThe idea for the hashmap singleton is from David Reiss, initial\nbenchmarking by Jeff King.\n\nNote: the implementation allows for arbitrary entries in the notes\ntree object, ignoring those that do not reference a valid object. This\nallows you to annotate arbitrary branches, or objects.\n\nThis patch has been improved by the following contributions:\n- Junio C Hamano: fixed an obvious error in initialize_hash_map()\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n notes.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 102 insertions(+), 10 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 401966d..9172154 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -4,15 +4,112 @@\n #include \"refs.h\"\n #include \"utf8.h\"\n #include \"strbuf.h\"\n+#include \"tree-walk.h\"\n+\n+struct entry {\n+\tunsigned char commit_sha1[20];\n+\tunsigned char notes_sha1[20];\n+};\n+\n+struct hash_map {\n+\tstruct entry *entries;\n+\toff_t count, size;\n+};\n \n static int initialized;\n+static struct hash_map hash_map;\n+\n+static int hash_index(struct hash_map *map, const unsigned char *sha1)\n+{\n+\tint i = ((*(unsigned int *)sha1) % map->size);\n+\n+\tfor (;;) {\n+\t\tunsigned char *current = map->entries[i].commit_sha1;\n+\n+\t\tif (!hashcmp(sha1, current))\n+\t\t\treturn i;\n+\n+\t\tif (is_null_sha1(current))\n+\t\t\treturn -1 - i;\n+\n+\t\tif (++i == map->size)\n+\t\t\ti = 0;\n+\t}\n+}\n+\n+static void add_entry(const unsigned char *commit_sha1,\n+\t\tconst unsigned char *notes_sha1)\n+{\n+\tint index;\n+\n+\tif (hash_map.count + 1 > hash_map.size >> 1) {\n+\t\tint i, old_size = hash_map.size;\n+\t\tstruct entry *old = hash_map.entries;\n+\n+\t\thash_map.size = old_size ? old_size << 1 : 64;\n+\t\thash_map.entries = (struct entry *)\n+\t\t\txcalloc(sizeof(struct entry), hash_map.size);\n+\n+\t\tfor (i = 0; i < old_size; i++)\n+\t\t\tif (!is_null_sha1(old[i].commit_sha1)) {\n+\t\t\t\tindex = -1 - hash_index(&hash_map,\n+\t\t\t\t\t\told[i].commit_sha1);\n+\t\t\t\tmemcpy(hash_map.entries + index, old + i,\n+\t\t\t\t\tsizeof(struct entry));\n+\t\t\t}\n+\t\tfree(old);\n+\t}\n+\n+\tindex = hash_index(&hash_map, commit_sha1);\n+\tif (index < 0) {\n+\t\tindex = -1 - index;\n+\t\thash_map.count++;\n+\t}\n+\n+\thashcpy(hash_map.entries[index].commit_sha1, commit_sha1);\n+\thashcpy(hash_map.entries[index].notes_sha1, notes_sha1);\n+}\n+\n+static void initialize_hash_map(const char *notes_ref_name)\n+{\n+\tunsigned char sha1[20], commit_sha1[20];\n+\tunsigned mode;\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tvoid *buf;\n+\n+\tif (!notes_ref_name || read_ref(notes_ref_name, commit_sha1) ||\n+\t get_tree_entry(commit_sha1, \"\", sha1, &mode))\n+\t\treturn;\n+\n+\tbuf = fill_tree_descriptor(&desc, sha1);\n+\tif (!buf)\n+\t\tdie(\"Could not read %s for notes-index\", sha1_to_hex(sha1));\n+\n+\twhile (tree_entry(&desc, &entry))\n+\t\tif (!get_sha1(entry.path, commit_sha1))\n+\t\t\tadd_entry(commit_sha1, entry.sha1);\n+\tfree(buf);\n+}\n+\n+static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n+{\n+\tint index;\n+\n+\tif (!hash_map.size)\n+\t\treturn NULL;\n+\n+\tindex = hash_index(&hash_map, commit_sha1);\n+\tif (index < 0)\n+\t\treturn NULL;\n+\treturn hash_map.entries[index].notes_sha1;\n+}\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tconst char *output_encoding)\n {\n \tstatic const char utf8[] = \"utf-8\";\n-\tstruct strbuf name = STRBUF_INIT;\n-\tunsigned char sha1[20];\n+\tunsigned char *sha1;\n \tchar *msg, *msg_p;\n \tunsigned long linelen, msglen;\n \tenum object_type type;\n@@ -23,17 +120,12 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\t\tnotes_ref_name = getenv(GIT_NOTES_REF_ENVIRONMENT);\n \t\telse if (!notes_ref_name)\n \t\t\tnotes_ref_name = GIT_NOTES_DEFAULT_REF;\n-\t\tif (notes_ref_name && read_ref(notes_ref_name, sha1))\n-\t\t\tnotes_ref_name = NULL;\n+\t\tinitialize_hash_map(notes_ref_name);\n \t\tinitialized = 1;\n \t}\n \n-\tif (!notes_ref_name)\n-\t\treturn;\n-\n-\tstrbuf_addf(&name, \"%s:%s\", notes_ref_name,\n-\t\t\tsha1_to_hex(commit->object.sha1));\n-\tif (get_sha1(name.buf, sha1))\n+\tsha1 = lookup_notes(commit->object.sha1);\n+\tif (!sha1)\n \t\treturn;\n \n \tif (!(msg = read_sha1_file(sha1, &type, &msglen)) || !msglen ||\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122953","messageId":"1252771728-27206-5-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 04/14] Add an expensive test for git-notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:38Z","receivedAt":"2009-09-12T16:08:38Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\ngit-notes have the potential of being pretty expensive, so test with\na lot of commits. A lot. So to make things cheaper, you have to\nopt-in explicitely, by setting the environment variable\nGIT_NOTES_TIMING_TESTS.\n\nThis patch has been improved by the following contributions:\n- Junio C Hamano: tests: fix \"export var=val\"\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Johan Herland \nSigned-off-by: Junio C Hamano \n---\n t/t3302-notes-index-expensive.sh | 98 ++++++++++++++++++++++++++++++++++++++\n 1 files changed, 98 insertions(+), 0 deletions(-)\n create mode 100755 t/t3302-notes-index-expensive.sh\n\ndiff --git a/t/t3302-notes-index-expensive.sh b/t/t3302-notes-index-expensive.sh\nnew file mode 100755\nindex 0000000..0ef3e95\n--- /dev/null\n+++ b/t/t3302-notes-index-expensive.sh\n@@ -0,0 +1,98 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Johannes E. Schindelin\n+#\n+\n+test_description='Test commit notes index (expensive!)'\n+\n+. ./test-lib.sh\n+\n+test -z \"$GIT_NOTES_TIMING_TESTS\" && {\n+\tsay Skipping timing tests\n+\ttest_done\n+\texit\n+}\n+\n+create_repo () {\n+\tnumber_of_commits=$1\n+\tnr=0\n+\tparent=\n+\ttest -d .git || {\n+\tgit init &&\n+\ttree=$(git write-tree) &&\n+\twhile [ $nr -lt $number_of_commits ]; do\n+\t\ttest_tick &&\n+\t\tcommit=$(echo $nr | git commit-tree $tree $parent) ||\n+\t\t\treturn\n+\t\tparent=\"-p $commit\"\n+\t\tnr=$(($nr+1))\n+\tdone &&\n+\tgit update-ref refs/heads/master $commit &&\n+\t{\n+\t\tGIT_INDEX_FILE=.git/temp; export GIT_INDEX_FILE;\n+\t\tgit rev-list HEAD | cat -n | sed \"s/^[ \t][ \t]*/ /g\" |\n+\t\twhile read nr sha1; do\n+\t\t\tblob=$(echo note $nr | git hash-object -w --stdin) &&\n+\t\t\techo $sha1 | sed \"s/^/0644 $blob 0\t/\"\n+\t\tdone | git update-index --index-info &&\n+\t\ttree=$(git write-tree) &&\n+\t\ttest_tick &&\n+\t\tcommit=$(echo notes | git commit-tree $tree) &&\n+\t\tgit update-ref refs/notes/commits $commit\n+\t} &&\n+\tgit config core.notesRef refs/notes/commits\n+\t}\n+}\n+\n+test_notes () {\n+\tcount=$1 &&\n+\tgit config core.notesRef refs/notes/commits &&\n+\tgit log | grep \"^ \" > output &&\n+\ti=1 &&\n+\twhile [ $i -le $count ]; do\n+\t\techo \" $(($count-$i))\" &&\n+\t\techo \" note $i\" &&\n+\t\ti=$(($i+1));\n+\tdone > expect &&\n+\tgit diff expect output\n+}\n+\n+cat > time_notes << \\EOF\n+\tmode=$1\n+\ti=1\n+\twhile [ $i -lt $2 ]; do\n+\t\tcase $1 in\n+\t\tno-notes)\n+\t\t\tGIT_NOTES_REF=non-existing; export GIT_NOTES_REF\n+\t\t;;\n+\t\tnotes)\n+\t\t\tunset GIT_NOTES_REF\n+\t\t;;\n+\t\tesac\n+\t\tgit log >/dev/null\n+\t\ti=$(($i+1))\n+\tdone\n+EOF\n+\n+time_notes () {\n+\tfor mode in no-notes notes\n+\tdo\n+\t\techo $mode\n+\t\t/usr/bin/time sh ../time_notes $mode $1\n+\tdone\n+}\n+\n+for count in 10 100 1000 10000; do\n+\n+\tmkdir $count\n+\t(cd $count;\n+\n+\ttest_expect_success \"setup $count\" \"create_repo $count\"\n+\n+\ttest_expect_success 'notes work' \"test_notes $count\"\n+\n+\ttest_expect_success 'notes timing' \"time_notes 100\"\n+\t)\n+done\n+\n+test_done\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122961","messageId":"1252771728-27206-6-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 05/14] Teach \"-m \" and \"-F \" to \"git notes edit\"","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:39Z","receivedAt":"2009-09-12T16:08:39Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The \"-m\" and \"-F\" options are already the established method\n(in both git-commit and git-tag) to specify a commit/tag message\nwithout invoking the editor. This patch teaches \"git notes edit\"\nto respect the same options for specifying a notes message without\ninvoking the editor.\n\nMultiple \"-m\" and/or \"-F\" options are concatenated as separate\nparagraphs.\n\nThe patch also updates the \"git notes\" documentation and adds\nselftests for the new functionality. Unfortunately, the added\nselftests include a couple of lines with trailing whitespace\n(without these the test will fail). This may cause git to warn\nabout \"whitespace errors\".\n\nThis patch has been improved by the following contributions:\n- Thomas Rast: fix trailing whitespace in t3301\n\nSigned-off-by: Johan Herland \n---\n Documentation/git-notes.txt | 16 ++++++++++-\n git-notes.sh | 64 +++++++++++++++++++++++++++++++++++++-----\n t/t3301-notes.sh | 36 ++++++++++++++++++++++++\n 3 files changed, 107 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt\nindex 7136016..94cceb1 100644\n--- a/Documentation/git-notes.txt\n+++ b/Documentation/git-notes.txt\n@@ -8,7 +8,7 @@ git-notes - Add/inspect commit notes\n SYNOPSIS\n --------\n [verse]\n-'git-notes' (edit | show) [commit]\n+'git-notes' (edit [-F | -m ] | show) [commit]\n \n DESCRIPTION\n -----------\n@@ -33,6 +33,20 @@ show::\n \tShow the notes for a given commit (defaults to HEAD).\n \n \n+OPTIONS\n+-------\n+-m ::\n+\tUse the given note message (instead of prompting).\n+\tIf multiple `-m` (or `-F`) options are given, their\n+\tvalues are concatenated as separate paragraphs.\n+\n+-F ::\n+\tTake the note message from the given file. Use '-' to\n+\tread the note message from the standard input.\n+\tIf multiple `-F` (or `-m`) options are given, their\n+\tvalues are concatenated as separate paragraphs.\n+\n+\n Author\n ------\n Written by Johannes Schindelin \ndiff --git a/git-notes.sh b/git-notes.sh\nindex f06c254..e642e47 100755\n--- a/git-notes.sh\n+++ b/git-notes.sh\n@@ -1,16 +1,59 @@\n #!/bin/sh\n \n-USAGE=\"(edit | show) [commit]\"\n+USAGE=\"(edit [-F | -m ] | show) [commit]\"\n . git-sh-setup\n \n-test -n \"$3\" && usage\n-\n test -z \"$1\" && usage\n ACTION=\"$1\"; shift\n \n test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"$(git config core.notesref)\"\n test -z \"$GIT_NOTES_REF\" && GIT_NOTES_REF=\"refs/notes/commits\"\n \n+MESSAGE=\n+while test $# != 0\n+do\n+\tcase \"$1\" in\n+\t-m)\n+\t\ttest \"$ACTION\" = \"edit\" || usage\n+\t\tshift\n+\t\tif test \"$#\" = \"0\"; then\n+\t\t\tdie \"error: option -m needs an argument\"\n+\t\telse\n+\t\t\tif [ -z \"$MESSAGE\" ]; then\n+\t\t\t\tMESSAGE=\"$1\"\n+\t\t\telse\n+\t\t\t\tMESSAGE=\"$MESSAGE\n+\n+$1\"\n+\t\t\tfi\n+\t\t\tshift\n+\t\tfi\n+\t\t;;\n+\t-F)\n+\t\ttest \"$ACTION\" = \"edit\" || usage\n+\t\tshift\n+\t\tif test \"$#\" = \"0\"; then\n+\t\t\tdie \"error: option -F needs an argument\"\n+\t\telse\n+\t\t\tif [ -z \"$MESSAGE\" ]; then\n+\t\t\t\tMESSAGE=\"$(cat \"$1\")\"\n+\t\t\telse\n+\t\t\t\tMESSAGE=\"$MESSAGE\n+\n+$(cat \"$1\")\"\n+\t\t\tfi\n+\t\t\tshift\n+\t\tfi\n+\t\t;;\n+\t-*)\n+\t\tusage\n+\t\t;;\n+\t*)\n+\t\tbreak\n+\t\t;;\n+\tesac\n+done\n+\n COMMIT=$(git rev-parse --verify --default HEAD \"$@\") ||\n die \"Invalid commit: $@\"\n \n@@ -29,19 +72,24 @@ edit)\n \t\ttest -f \"$GIT_INDEX_FILE\" && rm \"$GIT_INDEX_FILE\"\n \t' 0\n \n-\tGIT_NOTES_REF= git log -1 $COMMIT | sed \"s/^/#/\" > \"$MSG_FILE\"\n-\n \tCURRENT_HEAD=$(git show-ref \"$GIT_NOTES_REF\" | cut -f 1 -d ' ')\n \tif [ -z \"$CURRENT_HEAD\" ]; then\n \t\tPARENT=\n \telse\n \t\tPARENT=\"-p $CURRENT_HEAD\"\n \t\tgit read-tree \"$GIT_NOTES_REF\" || die \"Could not read index\"\n-\t\tgit cat-file blob :$COMMIT >> \"$MSG_FILE\" 2> /dev/null\n \tfi\n \n-\tcore_editor=\"$(git config core.editor)\"\n-\t${GIT_EDITOR:-${core_editor:-${VISUAL:-${EDITOR:-vi}}}} \"$MSG_FILE\"\n+\tif [ -z \"$MESSAGE\" ]; then\n+\t\tGIT_NOTES_REF= git log -1 $COMMIT | sed \"s/^/#/\" > \"$MSG_FILE\"\n+\t\tif [ ! -z \"$CURRENT_HEAD\" ]; then\n+\t\t\tgit cat-file blob :$COMMIT >> \"$MSG_FILE\" 2> /dev/null\n+\t\tfi\n+\t\tcore_editor=\"$(git config core.editor)\"\n+\t\t${GIT_EDITOR:-${core_editor:-${VISUAL:-${EDITOR:-vi}}}} \"$MSG_FILE\"\n+\telse\n+\t\techo \"$MESSAGE\" > \"$MSG_FILE\"\n+\tfi\n \n \tgrep -v ^# < \"$MSG_FILE\" | git stripspace > \"$MSG_FILE\".processed\n \tmv \"$MSG_FILE\".processed \"$MSG_FILE\"\ndiff --git a/t/t3301-notes.sh b/t/t3301-notes.sh\nindex 73e53be..1e34f48 100755\n--- a/t/t3301-notes.sh\n+++ b/t/t3301-notes.sh\n@@ -110,5 +110,41 @@ test_expect_success 'show multi-line notes' '\n \tgit log -2 > output &&\n \ttest_cmp expect-multiline output\n '\n+test_expect_success 'create -m and -F notes (setup)' '\n+\t: > a4 &&\n+\tgit add a4 &&\n+\ttest_tick &&\n+\tgit commit -m 4th &&\n+\techo \"xyzzy\" > note5 &&\n+\tgit notes edit -m spam -F note5 -m \"foo\n+bar\n+baz\"\n+'\n+\n+whitespace=\" \"\n+cat > expect-m-and-F << EOF\n+commit 15023535574ded8b1a89052b32673f84cf9582b8\n+Author: A U Thor \n+Date: Thu Apr 7 15:16:13 2005 -0700\n+\n+ 4th\n+\n+Notes:\n+ spam\n+$whitespace\n+ xyzzy\n+$whitespace\n+ foo\n+ bar\n+ baz\n+EOF\n+\n+printf \"\\n\" >> expect-m-and-F\n+cat expect-multiline >> expect-m-and-F\n+\n+test_expect_success 'show -m and -F notes' '\n+\tgit log -3 > output &&\n+\ttest_cmp expect-m-and-F output\n+'\n \n test_done\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122960","messageId":"1252771728-27206-7-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 06/14] fast-import: Add support for importing commit notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:40Z","receivedAt":"2009-09-12T16:08:40Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Introduce a 'notemodify' subcommand of the 'commit' command. This subcommand\nis similar to 'filemodify', except that no mode is supplied (all notes have\nmode 0644), and the path is set to the hex SHA1 of the given \"comittish\".\n\nThis enables fast import of note objects along with their associated commits,\nsince the notes can now be named using the mark references of their\ncorresponding commits.\n\nThe patch also includes a test case of the added functionality.\n\nSigned-off-by: Johan Herland \nAcked-by: Shawn O. Pearce \n---\n Documentation/git-fast-import.txt | 45 +++++++++--\n fast-import.c | 88 +++++++++++++++++++-\n t/t9300-fast-import.sh | 166 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 289 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex f1c94b4..bb198c2 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -325,7 +325,7 @@ change to the project.\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n-\t(filemodify | filedelete | filecopy | filerename | filedeleteall)*\n+\t(filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)*\n \tLF?\n ....\n \n@@ -348,14 +348,13 @@ commit message use a 0 length data. Commit messages are free-form\n and are not interpreted by Git. Currently they must be encoded in\n UTF-8, as fast-import does not permit other encodings to be specified.\n \n-Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`\n-and `filedeleteall` commands\n+Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`,\n+`filedeleteall` and `notemodify` commands\n may be included to update the contents of the branch prior to\n creating the commit. These commands may be supplied in any order.\n However it is recommended that a `filedeleteall` command precede\n-all `filemodify`, `filecopy` and `filerename` commands in the same\n-commit, as `filedeleteall`\n-wipes the branch clean (see below).\n+all `filemodify`, `filecopy`, `filerename` and `notemodify` commands in\n+the same commit, as `filedeleteall` wipes the branch clean (see below).\n \n The `LF` after the command is optional (it used to be required).\n \n@@ -604,6 +603,40 @@ more memory per active branch (less than 1 MiB for even most large\n projects); so frontends that can easily obtain only the affected\n paths for a commit are encouraged to do so.\n \n+`notemodify`\n+^^^^^^^^^^^^\n+Included in a `commit` command to add a new note (annotating a given\n+commit) or change the content of an existing note. This command has\n+two different means of specifying the content of the note.\n+\n+External data format::\n+\tThe data content for the note was already supplied by a prior\n+\t`blob` command. The frontend just needs to connect it to the\n+\tcommit that is to be annotated.\n++\n+....\n+\t'N' SP SP LF\n+....\n++\n+Here `` can be either a mark reference (`:`)\n+set by a prior `blob` command, or a full 40-byte SHA-1 of an\n+existing Git blob object.\n+\n+Inline data format::\n+\tThe data content for the note has not been supplied yet.\n+\tThe frontend wants to supply it as part of this modify\n+\tcommand.\n++\n+....\n+\t'N' SP 'inline' SP LF\n+\tdata\n+....\n++\n+See below for a detailed description of the `data` command.\n+\n+In both formats `` is any of the commit specification\n+expressions also accepted by `from` (see above).\n+\n `mark`\n ~~~~~~\n Arranges for fast-import to save a reference to the current object, allowing\ndiff --git a/fast-import.c b/fast-import.c\nindex dcfb8fa..1e91358 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -22,8 +22,8 @@ Format of STDIN stream:\n ('author' sp name sp '<' email '>' sp when lf)?\n 'committer' sp name sp '<' email '>' sp when lf\n commit_msg\n- ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?\n- ('merge' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)*\n+ ('from' sp committish lf)?\n+ ('merge' sp committish lf)*\n file_change*\n lf?;\n commit_msg ::= data;\n@@ -41,15 +41,18 @@ Format of STDIN stream:\n file_obm ::= 'M' sp mode sp (hexsha1 | idnum) sp path_str lf;\n file_inm ::= 'M' sp mode sp 'inline' sp path_str lf\n data;\n+ note_obm ::= 'N' sp (hexsha1 | idnum) sp committish lf;\n+ note_inm ::= 'N' sp 'inline' sp committish lf\n+ data;\n \n new_tag ::= 'tag' sp tag_str lf\n- 'from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf\n+ 'from' sp committish lf\n ('tagger' sp name sp '<' email '>' sp when lf)?\n tag_msg;\n tag_msg ::= data;\n \n reset_branch ::= 'reset' sp ref_str lf\n- ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?\n+ ('from' sp committish lf)?\n lf?;\n \n checkpoint ::= 'checkpoint' lf\n@@ -88,6 +91,7 @@ Format of STDIN stream:\n # stream formatting is: \\, \" and LF. Otherwise these values\n # are UTF8.\n #\n+ committish ::= (ref_str | hexsha1 | sha1exp_str | idnum);\n ref_str ::= ref;\n sha1exp_str ::= sha1exp;\n tag_str ::= tag;\n@@ -2053,6 +2057,80 @@ static void file_change_cr(struct branch *b, int rename)\n \t\tleaf.tree);\n }\n \n+static void note_change_n(struct branch *b)\n+{\n+\tconst char *p = command_buf.buf + 2;\n+\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstruct object_entry *oe = oe;\n+\tstruct branch *s;\n+\tunsigned char sha1[20], commit_sha1[20];\n+\tuint16_t inline_data = 0;\n+\n+\t/* or 'inline' */\n+\tif (*p == ':') {\n+\t\tchar *x;\n+\t\toe = find_mark(strtoumax(p + 1, &x, 10));\n+\t\thashcpy(sha1, oe->sha1);\n+\t\tp = x;\n+\t} else if (!prefixcmp(p, \"inline\")) {\n+\t\tinline_data = 1;\n+\t\tp += 6;\n+\t} else {\n+\t\tif (get_sha1_hex(p, sha1))\n+\t\t\tdie(\"Invalid SHA1: %s\", command_buf.buf);\n+\t\toe = find_object(sha1);\n+\t\tp += 40;\n+\t}\n+\tif (*p++ != ' ')\n+\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n+\n+\t/* */\n+\ts = lookup_branch(p);\n+\tif (s) {\n+\t\thashcpy(commit_sha1, s->sha1);\n+\t} else if (*p == ':') {\n+\t\tuintmax_t commit_mark = strtoumax(p + 1, NULL, 10);\n+\t\tstruct object_entry *commit_oe = find_mark(commit_mark);\n+\t\tif (commit_oe->type != OBJ_COMMIT)\n+\t\t\tdie(\"Mark :%\" PRIuMAX \" not a commit\", commit_mark);\n+\t\thashcpy(commit_sha1, commit_oe->sha1);\n+\t} else if (!get_sha1(p, commit_sha1)) {\n+\t\tunsigned long size;\n+\t\tchar *buf = read_object_with_reference(commit_sha1,\n+\t\t\tcommit_type, &size, commit_sha1);\n+\t\tif (!buf || size < 46)\n+\t\t\tdie(\"Not a valid commit: %s\", p);\n+\t\tfree(buf);\n+\t} else\n+\t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n+\n+\tif (inline_data) {\n+\t\tstatic struct strbuf buf = STRBUF_INIT;\n+\n+\t\tif (p != uq.buf) {\n+\t\t\tstrbuf_addstr(&uq, p);\n+\t\t\tp = uq.buf;\n+\t\t}\n+\t\tread_next_command();\n+\t\tparse_data(&buf);\n+\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t} else if (oe) {\n+\t\tif (oe->type != OBJ_BLOB)\n+\t\t\tdie(\"Not a blob (actually a %s): %s\",\n+\t\t\t\ttypename(oe->type), command_buf.buf);\n+\t} else {\n+\t\tenum object_type type = sha1_object_info(sha1, NULL);\n+\t\tif (type < 0)\n+\t\t\tdie(\"Blob not found: %s\", command_buf.buf);\n+\t\tif (type != OBJ_BLOB)\n+\t\t\tdie(\"Not a blob (actually a %s): %s\",\n+\t\t\t typename(type), command_buf.buf);\n+\t}\n+\n+\ttree_content_set(&b->branch_tree, sha1_to_hex(commit_sha1), sha1,\n+\t\tS_IFREG | 0644, NULL);\n+}\n+\n static void file_change_deleteall(struct branch *b)\n {\n \trelease_tree_content_recursive(b->branch_tree.tree);\n@@ -2222,6 +2300,8 @@ static void parse_new_commit(void)\n \t\t\tfile_change_cr(b, 1);\n \t\telse if (!prefixcmp(command_buf.buf, \"C \"))\n \t\t\tfile_change_cr(b, 0);\n+\t\telse if (!prefixcmp(command_buf.buf, \"N \"))\n+\t\t\tnote_change_n(b);\n \t\telse if (!strcmp(\"deleteall\", command_buf.buf))\n \t\t\tfile_change_deleteall(b);\n \t\telse {\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex d33fc55..2f5c323 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1089,6 +1089,172 @@ test_expect_success 'P: fail on blob mark in gitlink' '\n test_must_fail git fast-import input < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data <expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+first (:3)\n+EOF\n+test_expect_success \\\n+\t'Q: verify first commit' \\\n+\t'git cat-file commit notes-test~2 | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+second (:5)\n+EOF\n+test_expect_success \\\n+\t'Q: verify second commit' \\\n+\t'git cat-file commit notes-test^ | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+third (:6)\n+EOF\n+test_expect_success \\\n+\t'Q: verify third commit' \\\n+\t'git cat-file commit notes-test | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect < $GIT_COMMITTER_DATE\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\n+notes (:9)\n+EOF\n+test_expect_success \\\n+\t'Q: verify notes commit' \\\n+\t'git cat-file commit refs/notes/foobar | sed 1d >actual &&\n+\ttest_cmp expect actual'\n+\n+cat >expect.unsorted <expect\n+test_expect_success \\\n+\t'Q: verify notes tree' \\\n+\t'git cat-file -p refs/notes/foobar^{tree} | sed \"s/ [0-9a-f]*\t/ /\" >actual &&\n+\t test_cmp expect actual'\n+\n+echo \"$note1_data\" >expect\n+test_expect_success \\\n+\t'Q: verify note for first commit' \\\n+\t'git cat-file blob refs/notes/foobar:$commit1 >actual && test_cmp expect actual'\n+\n+echo \"$note2_data\" >expect\n+test_expect_success \\\n+\t'Q: verify note for second commit' \\\n+\t'git cat-file blob refs/notes/foobar:$commit2 >actual && test_cmp expect actual'\n+\n+echo \"$note3_data\" >expect\n+test_expect_success \\\n+\t'Q: verify note for third commit' \\\n+\t'git cat-file blob refs/notes/foobar:$commit3 >actual && test_cmp expect actual'\n+\n+###\n ### series R (feature and option)\n ###\n \n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122962","messageId":"1252771728-27206-8-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 07/14] t3302-notes-index-expensive: Speed up create_repo()","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:41Z","receivedAt":"2009-09-12T16:08:41Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Creating repos with 10/100/1000/10000 commits and notes takes a lot of time.\nHowever, using git-fast-import to do the job is a lot more efficient than\nusing plumbing commands to do the same.\n\nThis patch decreases the overall run-time of this test on my machine from\n~3 to ~1 minutes.\n\nSigned-off-by: Johan Herland \nAcked-by: Johannes Schindelin \n---\n t/t3302-notes-index-expensive.sh | 74 ++++++++++++++++++++++++--------------\n 1 files changed, 47 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t3302-notes-index-expensive.sh b/t/t3302-notes-index-expensive.sh\nindex 0ef3e95..ee84fc4 100755\n--- a/t/t3302-notes-index-expensive.sh\n+++ b/t/t3302-notes-index-expensive.sh\n@@ -16,30 +16,50 @@ test -z \"$GIT_NOTES_TIMING_TESTS\" && {\n create_repo () {\n \tnumber_of_commits=$1\n \tnr=0\n-\tparent=\n \ttest -d .git || {\n \tgit init &&\n-\ttree=$(git write-tree) &&\n-\twhile [ $nr -lt $number_of_commits ]; do\n-\t\ttest_tick &&\n-\t\tcommit=$(echo $nr | git commit-tree $tree $parent) ||\n-\t\t\treturn\n-\t\tparent=\"-p $commit\"\n-\t\tnr=$(($nr+1))\n-\tdone &&\n-\tgit update-ref refs/heads/master $commit &&\n-\t{\n-\t\tGIT_INDEX_FILE=.git/temp; export GIT_INDEX_FILE;\n-\t\tgit rev-list HEAD | cat -n | sed \"s/^[ \t][ \t]*/ /g\" |\n-\t\twhile read nr sha1; do\n-\t\t\tblob=$(echo note $nr | git hash-object -w --stdin) &&\n-\t\t\techo $sha1 | sed \"s/^/0644 $blob 0\t/\"\n-\t\tdone | git update-index --index-info &&\n-\t\ttree=$(git write-tree) &&\n+\t(\n+\t\twhile [ $nr -lt $number_of_commits ]; do\n+\t\t\tnr=$(($nr+1))\n+\t\t\tmark=$(($nr+$nr))\n+\t\t\tnotemark=$(($mark+1))\n+\t\t\ttest_tick &&\n+\t\t\tcat < $GIT_COMMITTER_DATE\n+data <> note_commit\n+\t\tdone &&\n \t\ttest_tick &&\n-\t\tcommit=$(echo notes | git commit-tree $tree) &&\n-\t\tgit update-ref refs/notes/commits $commit\n-\t} &&\n+\t\tcat < $GIT_COMMITTER_DATE\n+data < output &&\n-\ti=1 &&\n-\twhile [ $i -le $count ]; do\n-\t\techo \" $(($count-$i))\" &&\n-\t\techo \" note $i\" &&\n-\t\ti=$(($i+1));\n+\ti=$count &&\n+\twhile [ $i -gt 0 ]; do\n+\t\techo \" commit #$i\" &&\n+\t\techo \" note for commit #$i\" &&\n+\t\ti=$(($i-1));\n \tdone > expect &&\n-\tgit diff expect output\n+\ttest_cmp expect output\n }\n \n cat > time_notes << \\EOF\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122955","messageId":"1252771728-27206-9-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 08/14] Add flags to get_commit_notes() to control the format of the note string","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:42Z","receivedAt":"2009-09-12T16:08:42Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"This patch adds the following flags to get_commit_notes() for adjusting the\nformat of the produced note string:\n- NOTES_SHOW_HEADER: Print \"Notes:\" line before the notes contents\n- NOTES_INDENT: Indent notes contents by 4 spaces\n\nSuggested-by: Johannes Schindelin \nSigned-off-by: Johan Herland \n---\n notes.c | 8 +++++---\n notes.h | 5 ++++-\n pretty.c | 3 ++-\n 3 files changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 9172154..84c30c1 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -106,7 +106,7 @@ static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n }\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n-\t\tconst char *output_encoding)\n+\t\tconst char *output_encoding, int flags)\n {\n \tstatic const char utf8[] = \"utf-8\";\n \tunsigned char *sha1;\n@@ -146,12 +146,14 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \tif (msglen && msg[msglen - 1] == '\\n')\n \t\tmsglen--;\n \n-\tstrbuf_addstr(sb, \"\\nNotes:\\n\");\n+\tif (flags & NOTES_SHOW_HEADER)\n+\t\tstrbuf_addstr(sb, \"\\nNotes:\\n\");\n \n \tfor (msg_p = msg; msg_p < msg + msglen; msg_p += linelen + 1) {\n \t\tlinelen = strchrnul(msg_p, '\\n') - msg_p;\n \n-\t\tstrbuf_addstr(sb, \" \");\n+\t\tif (flags & NOTES_INDENT)\n+\t\t\tstrbuf_addstr(sb, \" \");\n \t\tstrbuf_add(sb, msg_p, linelen);\n \t\tstrbuf_addch(sb, '\\n');\n \t}\ndiff --git a/notes.h b/notes.h\nindex 79d21b6..7f3eed4 100644\n--- a/notes.h\n+++ b/notes.h\n@@ -1,7 +1,10 @@\n #ifndef NOTES_H\n #define NOTES_H\n \n+#define NOTES_SHOW_HEADER 1\n+#define NOTES_INDENT 2\n+\n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n-\t\tconst char *output_encoding);\n+\t\tconst char *output_encoding, int flags);\n \n #endif\ndiff --git a/pretty.c b/pretty.c\nindex e25db81..01eadd0 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -978,7 +978,8 @@ void pretty_print_commit(enum cmit_fmt fmt, const struct commit *commit,\n \t\tstrbuf_addch(sb, '\\n');\n \n \tif (fmt != CMIT_FMT_ONELINE)\n-\t\tget_commit_notes(commit, sb, encoding);\n+\t\tget_commit_notes(commit, sb, encoding,\n+\t\t\t\t NOTES_SHOW_HEADER | NOTES_INDENT);\n \n \tfree(reencoded);\n }\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122963","messageId":"1252771728-27206-10-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 09/14] Add '%N'-format for pretty-printing commit notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:43Z","receivedAt":"2009-09-12T16:08:43Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"From: Johannes Schindelin \n\nSigned-off-by: Johan Herland \n---\n Documentation/pretty-formats.txt | 1 +\n pretty.c | 4 ++++\n 2 files changed, 5 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/pretty-formats.txt b/Documentation/pretty-formats.txt\nindex 2a845b1..5fb10b3 100644\n--- a/Documentation/pretty-formats.txt\n+++ b/Documentation/pretty-formats.txt\n@@ -123,6 +123,7 @@ The placeholders are:\n - '%s': subject\n - '%f': sanitized subject line, suitable for a filename\n - '%b': body\n+- '%N': commit notes\n - '%Cred': switch color to red\n - '%Cgreen': switch color to green\n - '%Cblue': switch color to blue\ndiff --git a/pretty.c b/pretty.c\nindex 01eadd0..7f350bb 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -702,6 +702,10 @@ static size_t format_commit_item(struct strbuf *sb, const char *placeholder,\n \tcase 'd':\n \t\tformat_decoration(sb, commit);\n \t\treturn 1;\n+\tcase 'N':\n+\t\tget_commit_notes(commit, sb, git_log_output_encoding ?\n+\t\t\t git_log_output_encoding : git_commit_encoding, 0);\n+\t\treturn 1;\n \t}\n \n \t/* For the rest we have to parse the commit header. */\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122956","messageId":"1252771728-27206-11-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 10/14] Teach notes code to free its internal data structures on request.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:44Z","receivedAt":"2009-09-12T16:08:44Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"There's no need to be rude to memory-concious callers...\n\nSigned-off-by: Johan Herland \n---\n notes.c | 7 +++++++\n notes.h | 2 ++\n 2 files changed, 9 insertions(+), 0 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 84c30c1..008c3d4 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -160,3 +160,10 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \n \tfree(msg);\n }\n+\n+void free_commit_notes()\n+{\n+\tfree(hash_map.entries);\n+\tmemset(&hash_map, 0, sizeof(struct hash_map));\n+\tinitialized = 0;\n+}\ndiff --git a/notes.h b/notes.h\nindex 7f3eed4..41802e5 100644\n--- a/notes.h\n+++ b/notes.h\n@@ -7,4 +7,6 @@\n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tconst char *output_encoding, int flags);\n \n+void free_commit_notes();\n+\n #endif\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122959","messageId":"1252771728-27206-12-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 11/14] Teach the notes lookup code to parse notes trees with various fanout schemes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:45Z","receivedAt":"2009-09-12T16:08:45Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The semantics used when parsing notes trees (with regards to fanout subtrees)\nfollow Dscho's proposal fairly closely:\n- No concatenation/merging of notes is performed. If there are several notes\n objects referencing a given commit, only one of those objects are used.\n- If a notes object for a given commit is present in the \"root\" notes tree,\n no subtrees are consulted; the object in the root tree is used directly.\n- If there are more than one subtree that prefix-matches the given commit,\n only the subtree with the longest matching prefix is consulted. This\n means that if the given commit is e.g. \"deadbeef\", and the notes tree have\n subtrees \"de\" and \"dead\", then the following paths in the notes tree are\n searched: \"deadbeef\", \"dead/beef\". Note that \"de/adbeef\" is NOT searched.\n- Fanout directories (subtrees) must references a whole number of bytes\n from the SHA1 sum they subdivide. E.g. subtrees \"dead\" and \"de\" are\n acceptable; \"d\" and \"dea\" are not.\n- Multiple levels of fanout are allowed. All the above rules apply\n recursively. E.g. \"de/adbeef\" is preferred over \"de/adbe/ef\", etc.\n\nThis patch changes the in-memory datastructure for holding parsed notes:\nInstead of holding all note (and subtree) entries in a hash table, a\nsimple 16-tree structure is used instead. The tree structure consists of\n16-arrays as internal nodes, and note/subtree entries as leaf nodes. The\ntree is traversed by indexing subsequent nibbles of the search key until\na leaf node is encountered. If a subtree entry is encountered while\nsearching for a note, the subtree is unpacked into the 16-tree structure,\nand the search continues into that subtree.\n\nThe new algorithm performs significantly better in the cases where only\na fraction of the notes need to be looked up (this is assumed to be the\ncommon case for notes lookup). The new code even performs marginally\nbetter in the worst case (where _all_ the notes are looked up).\n\nIn addition to this, comes the massive performance win associated with\norganizing the notes tree according to some fanout scheme. Even a simple\n2/38 fanout scheme is dramatically quicker to traverse (going from tens of\nseconds to sub-second runtimes).\n\nAs for memory usage, the new code is marginally better than the old code in\nthe worst case, but in the case of looking up only some notes from a notes\ntree with proper fanout, the new code uses only a small fraction of the\nmemory needed to hold the entire notes tree.\n\nHowever, there is one casualty of this patch. The old notes lookup code was\nable to parse notes that were associated with non-SHA1s (e.g. refs). The new\ncode requires the referenced object to be named by a SHA1 sum. Still, this\nis not considered a major setback, since the notes infrastructure was not\noriginally intended to annotate objects outside the Git object database.\n\nCc: Johannes Schindelin \nSigned-off-by: Johan Herland \n---\n notes.c | 317 +++++++++++++++++++++++++++++++++++++++++++++++++--------------\n 1 files changed, 248 insertions(+), 69 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 008c3d4..6926aa6 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -6,103 +6,282 @@\n #include \"strbuf.h\"\n #include \"tree-walk.h\"\n \n-struct entry {\n-\tunsigned char commit_sha1[20];\n-\tunsigned char notes_sha1[20];\n+/*\n+ * Use a non-balancing simple 16-tree structure with struct int_node as\n+ * internal nodes, and struct leaf_node as leaf nodes. Each int_node has a\n+ * 16-array of pointers to its children.\n+ * The bottom 2 bits of each pointer is used to identify the pointer type\n+ * - ptr & 3 == 0 - NULL pointer, assert(ptr == NULL)\n+ * - ptr & 3 == 1 - pointer to next internal node - cast to struct int_node *\n+ * - ptr & 3 == 2 - pointer to note entry - cast to struct leaf_node *\n+ * - ptr & 3 == 3 - pointer to subtree entry - cast to struct leaf_node *\n+ *\n+ * The root node is a statically allocated struct int_node.\n+ */\n+struct int_node {\n+\tvoid *a[16];\n };\n \n-struct hash_map {\n-\tstruct entry *entries;\n-\toff_t count, size;\n+/*\n+ * Leaf nodes come in two variants, note entries and subtree entries,\n+ * distinguished by the LSb of the leaf node pointer (see above).\n+ * As a note entry, the key is the SHA1 of the referenced commit, and the\n+ * value is the SHA1 of the note object.\n+ * As a subtree entry, the key is the prefix SHA1 (w/trailing NULs) of the\n+ * referenced commit, using the last byte of the key to store the length of\n+ * the prefix. The value is the SHA1 of the tree object containing the notes\n+ * subtree.\n+ */\n+struct leaf_node {\n+\tunsigned char key_sha1[20];\n+\tunsigned char val_sha1[20];\n };\n \n-static int initialized;\n-static struct hash_map hash_map;\n+#define PTR_TYPE_NULL 0\n+#define PTR_TYPE_INTERNAL 1\n+#define PTR_TYPE_NOTE 2\n+#define PTR_TYPE_SUBTREE 3\n \n-static int hash_index(struct hash_map *map, const unsigned char *sha1)\n-{\n-\tint i = ((*(unsigned int *)sha1) % map->size);\n+#define GET_PTR_TYPE(ptr) ((uintptr_t) (ptr) & 3)\n+#define CLR_PTR_TYPE(ptr) ((void *) ((uintptr_t) (ptr) & ~3))\n+#define SET_PTR_TYPE(ptr, type) ((void *) ((uintptr_t) (ptr) | (type)))\n \n-\tfor (;;) {\n-\t\tunsigned char *current = map->entries[i].commit_sha1;\n+#define GET_NIBBLE(n, sha1) (((sha1[n >> 1]) >> ((n & 0x01) << 2)) & 0x0f)\n \n-\t\tif (!hashcmp(sha1, current))\n-\t\t\treturn i;\n+#define SUBTREE_SHA1_PREFIXCMP(key_sha1, subtree_sha1) \\\n+\t(memcmp(key_sha1, subtree_sha1, subtree_sha1[19]))\n \n-\t\tif (is_null_sha1(current))\n-\t\t\treturn -1 - i;\n+static struct int_node root_node;\n \n-\t\tif (++i == map->size)\n-\t\t\ti = 0;\n+static int initialized;\n+\n+static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n+\t\tunsigned int n);\n+\n+/*\n+ * To find a leaf_node:\n+ * 1. Start at the root node, with n = 0\n+ * 2. Use the nth nibble of the key as an index into a:\n+ * - If a[n] is an int_node, recurse into that node and increment n\n+ * - If a leaf_node with matching key, return leaf_node (assert note entry)\n+ * - If a matching subtree entry, unpack that subtree entry (and remove it);\n+ * restart search at the current level.\n+ * - Otherwise, we end up at a NULL pointer, or a non-matching leaf_node.\n+ * Backtrack out of the recursion, one level at a time and check a[0]:\n+ * - If a[0] at the current level is a matching subtree entry, unpack that\n+ * subtree entry (and remove it); restart search at the current level.\n+ */\n+static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n+\t\tconst unsigned char *key_sha1)\n+{\n+\tstruct leaf_node *l;\n+\tunsigned char i = GET_NIBBLE(n, key_sha1);\n+\tvoid *p = tree->a[i];\n+\n+\tswitch(GET_PTR_TYPE(p)) {\n+\tcase PTR_TYPE_INTERNAL:\n+\t\tl = note_tree_find(CLR_PTR_TYPE(p), n + 1, key_sha1);\n+\t\tif (l)\n+\t\t\treturn l;\n+\t\tbreak;\n+\tcase PTR_TYPE_NOTE:\n+\t\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n+\t\tif (!hashcmp(key_sha1, l->key_sha1))\n+\t\t\treturn l; /* return note object matching given key */\n+\t\tbreak;\n+\tcase PTR_TYPE_SUBTREE:\n+\t\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n+\t\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n+\t\t\t/* unpack tree and resume search */\n+\t\t\ttree->a[i] = NULL;\n+\t\t\tload_subtree(l, tree, n);\n+\t\t\tfree(l);\n+\t\t\treturn note_tree_find(tree, n, key_sha1);\n+\t\t}\n+\t\tbreak;\n+\tcase PTR_TYPE_NULL:\n+\tdefault:\n+\t\tassert(!p);\n+\t\tbreak;\n \t}\n+\n+\t/*\n+\t * Did not find key at this (or any lower) level.\n+\t * Check if there's a matching subtree entry in tree->a[0].\n+\t * If so, unpack tree and resume search.\n+\t */\n+\tp = tree->a[0];\n+\tif (GET_PTR_TYPE(p) != PTR_TYPE_SUBTREE)\n+\t\treturn NULL;\n+\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n+\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n+\t\t/* unpack tree and resume search */\n+\t\ttree->a[0] = NULL;\n+\t\tload_subtree(l, tree, n);\n+\t\tfree(l);\n+\t\treturn note_tree_find(tree, n, key_sha1);\n+\t}\n+\treturn NULL;\n }\n \n-static void add_entry(const unsigned char *commit_sha1,\n-\t\tconst unsigned char *notes_sha1)\n+/*\n+ * To insert a leaf_node:\n+ * 1. Start at the root node, with n = 0\n+ * 2. Use the nth nibble of the key as an index into a:\n+ * - If a[n] is NULL, store the tweaked pointer directly into a[n]\n+ * - If a[n] is an int_node, recurse into that node and increment n\n+ * - If a[n] is a leaf_node:\n+ * 1. Check if they're equal, and handle that (abort? overwrite?)\n+ * 2. Create a new int_node, and store both leaf_nodes there\n+ * 3. Store the new int_node into a[n].\n+ */\n+static int note_tree_insert(struct int_node *tree, unsigned char n,\n+\t\tconst struct leaf_node *entry, unsigned char type)\n {\n-\tint index;\n-\n-\tif (hash_map.count + 1 > hash_map.size >> 1) {\n-\t\tint i, old_size = hash_map.size;\n-\t\tstruct entry *old = hash_map.entries;\n-\n-\t\thash_map.size = old_size ? old_size << 1 : 64;\n-\t\thash_map.entries = (struct entry *)\n-\t\t\txcalloc(sizeof(struct entry), hash_map.size);\n-\n-\t\tfor (i = 0; i < old_size; i++)\n-\t\t\tif (!is_null_sha1(old[i].commit_sha1)) {\n-\t\t\t\tindex = -1 - hash_index(&hash_map,\n-\t\t\t\t\t\told[i].commit_sha1);\n-\t\t\t\tmemcpy(hash_map.entries + index, old + i,\n-\t\t\t\t\tsizeof(struct entry));\n-\t\t\t}\n-\t\tfree(old);\n+\tstruct int_node *new_node;\n+\tconst struct leaf_node *l;\n+\tint ret;\n+\tunsigned char i = GET_NIBBLE(n, entry->key_sha1);\n+\tvoid *p = tree->a[i];\n+\tassert(GET_PTR_TYPE(entry) == PTR_TYPE_NULL);\n+\tswitch(GET_PTR_TYPE(p)) {\n+\tcase PTR_TYPE_NULL:\n+\t\tassert(!p);\n+\t\ttree->a[i] = SET_PTR_TYPE(entry, type);\n+\t\treturn 0;\n+\tcase PTR_TYPE_INTERNAL:\n+\t\treturn note_tree_insert(CLR_PTR_TYPE(p), n + 1, entry, type);\n+\tdefault:\n+\t\tassert(GET_PTR_TYPE(p) == PTR_TYPE_NOTE ||\n+\t\t\tGET_PTR_TYPE(p) == PTR_TYPE_SUBTREE);\n+\t\tl = (const struct leaf_node *) CLR_PTR_TYPE(p);\n+\t\tif (!hashcmp(entry->key_sha1, l->key_sha1))\n+\t\t\treturn -1; /* abort insert on matching key */\n+\t\tnew_node = (struct int_node *)\n+\t\t\txcalloc(sizeof(struct int_node), 1);\n+\t\tret = note_tree_insert(new_node, n + 1,\n+\t\t\tCLR_PTR_TYPE(p), GET_PTR_TYPE(p));\n+\t\tif (ret) {\n+\t\t\tfree(new_node);\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttree->a[i] = SET_PTR_TYPE(new_node, PTR_TYPE_INTERNAL);\n+\t\treturn note_tree_insert(new_node, n + 1, entry, type);\n \t}\n+}\n \n-\tindex = hash_index(&hash_map, commit_sha1);\n-\tif (index < 0) {\n-\t\tindex = -1 - index;\n-\t\thash_map.count++;\n+/* Free the entire notes data contained in the given tree */\n+static void note_tree_free(struct int_node *tree)\n+{\n+\tunsigned int i;\n+\tfor (i = 0; i < 16; i++) {\n+\t\tvoid *p = tree->a[i];\n+\t\tswitch(GET_PTR_TYPE(p)) {\n+\t\tcase PTR_TYPE_INTERNAL:\n+\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n+\t\t\t/* fall through */\n+\t\tcase PTR_TYPE_NOTE:\n+\t\tcase PTR_TYPE_SUBTREE:\n+\t\t\tfree(CLR_PTR_TYPE(p));\n+\t\t}\n \t}\n+}\n \n-\thashcpy(hash_map.entries[index].commit_sha1, commit_sha1);\n-\thashcpy(hash_map.entries[index].notes_sha1, notes_sha1);\n+/*\n+ * Convert a partial SHA1 hex string to the corresponding partial SHA1 value.\n+ * - hex - Partial SHA1 segment in ASCII hex format\n+ * - hex_len - Length of above segment. Must be multiple of 2 between 0 and 40\n+ * - sha1 - Partial SHA1 value is written here\n+ * - sha1_len - Max #bytes to store in sha1, Must be >= hex_len / 2, and < 20\n+ * Returns -1 on error (invalid arguments or invalid SHA1 (not in hex format).\n+ * Otherwise, returns number of bytes written to sha1 (i.e. hex_len / 2).\n+ * Pads sha1 with NULs up to sha1_len (not included in returned length).\n+ */\n+static int get_sha1_hex_segment(const char *hex, unsigned int hex_len,\n+\t\tunsigned char *sha1, unsigned int sha1_len)\n+{\n+\tunsigned int i, len = hex_len >> 1;\n+\tif (hex_len % 2 != 0 || len > sha1_len)\n+\t\treturn -1;\n+\tfor (i = 0; i < len; i++) {\n+\t\tunsigned int val = (hexval(hex[0]) << 4) | hexval(hex[1]);\n+\t\tif (val & ~0xff)\n+\t\t\treturn -1;\n+\t\t*sha1++ = val;\n+\t\thex += 2;\n+\t}\n+\tfor (; i < sha1_len; i++)\n+\t\t*sha1++ = 0;\n+\treturn len;\n }\n \n-static void initialize_hash_map(const char *notes_ref_name)\n+static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n+\t\tunsigned int n)\n {\n-\tunsigned char sha1[20], commit_sha1[20];\n-\tunsigned mode;\n+\tunsigned char commit_sha1[20];\n+\tunsigned int prefix_len;\n+\tint status;\n+\tvoid *buf;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n-\tvoid *buf;\n+\n+\tbuf = fill_tree_descriptor(&desc, subtree->val_sha1);\n+\tif (!buf)\n+\t\tdie(\"Could not read %s for notes-index\",\n+\t\t sha1_to_hex(subtree->val_sha1));\n+\n+\tprefix_len = subtree->key_sha1[19];\n+\tassert(prefix_len * 2 >= n);\n+\tmemcpy(commit_sha1, subtree->key_sha1, prefix_len);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tint len = get_sha1_hex_segment(entry.path, strlen(entry.path),\n+\t\t\t\tcommit_sha1 + prefix_len, 20 - prefix_len);\n+\t\tif (len < 0)\n+\t\t\tcontinue; /* entry.path is not a SHA1 sum. Skip */\n+\t\tlen += prefix_len;\n+\n+\t\t/*\n+\t\t * If commit SHA1 is complete (len == 20), assume note object\n+\t\t * If commit SHA1 is incomplete (len < 20), assume note subtree\n+\t\t */\n+\t\tif (len <= 20) {\n+\t\t\tunsigned char type = PTR_TYPE_NOTE;\n+\t\t\tstruct leaf_node *l = (struct leaf_node *)\n+\t\t\t\txcalloc(sizeof(struct leaf_node), 1);\n+\t\t\thashcpy(l->key_sha1, commit_sha1);\n+\t\t\thashcpy(l->val_sha1, entry.sha1);\n+\t\t\tif (len < 20) {\n+\t\t\t\tl->key_sha1[19] = (unsigned char) len;\n+\t\t\t\ttype = PTR_TYPE_SUBTREE;\n+\t\t\t}\n+\t\t\tstatus = note_tree_insert(node, n, l, type);\n+\t\t\tassert(!status);\n+\t\t}\n+\t}\n+\tfree(buf);\n+}\n+\n+static void initialize_notes(const char *notes_ref_name)\n+{\n+\tunsigned char sha1[20], commit_sha1[20];\n+\tunsigned mode;\n+\tstruct leaf_node root_tree;\n \n \tif (!notes_ref_name || read_ref(notes_ref_name, commit_sha1) ||\n \t get_tree_entry(commit_sha1, \"\", sha1, &mode))\n \t\treturn;\n \n-\tbuf = fill_tree_descriptor(&desc, sha1);\n-\tif (!buf)\n-\t\tdie(\"Could not read %s for notes-index\", sha1_to_hex(sha1));\n-\n-\twhile (tree_entry(&desc, &entry))\n-\t\tif (!get_sha1(entry.path, commit_sha1))\n-\t\t\tadd_entry(commit_sha1, entry.sha1);\n-\tfree(buf);\n+\thashclr(root_tree.key_sha1);\n+\thashcpy(root_tree.val_sha1, sha1);\n+\tload_subtree(&root_tree, &root_node, 0);\n }\n \n static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n {\n-\tint index;\n-\n-\tif (!hash_map.size)\n-\t\treturn NULL;\n-\n-\tindex = hash_index(&hash_map, commit_sha1);\n-\tif (index < 0)\n-\t\treturn NULL;\n-\treturn hash_map.entries[index].notes_sha1;\n+\tstruct leaf_node *found = note_tree_find(&root_node, 0, commit_sha1);\n+\tif (found)\n+\t\treturn found->val_sha1;\n+\treturn NULL;\n }\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n@@ -120,7 +299,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\t\tnotes_ref_name = getenv(GIT_NOTES_REF_ENVIRONMENT);\n \t\telse if (!notes_ref_name)\n \t\t\tnotes_ref_name = GIT_NOTES_DEFAULT_REF;\n-\t\tinitialize_hash_map(notes_ref_name);\n+\t\tinitialize_notes(notes_ref_name);\n \t\tinitialized = 1;\n \t}\n \n@@ -163,7 +342,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \n void free_commit_notes()\n {\n-\tfree(hash_map.entries);\n-\tmemset(&hash_map, 0, sizeof(struct hash_map));\n+\tnote_tree_free(&root_node);\n+\tmemset(&root_node, 0, sizeof(struct int_node));\n \tinitialized = 0;\n }\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122957","messageId":"1252771728-27206-13-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 12/14] Selftests verifying semantics when loading notes trees with various fanouts","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:46Z","receivedAt":"2009-09-12T16:08:46Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Add selftests verifying:\n- that we are able to parse notes trees with various fanout schemes\n- that notes trees with conflicting fanout schemes are parsed as expected\n\nSigned-off-by: Johan Herland \n---\n t/t3303-notes-subtrees.sh | 137 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 137 insertions(+), 0 deletions(-)\n create mode 100755 t/t3303-notes-subtrees.sh\n\ndiff --git a/t/t3303-notes-subtrees.sh b/t/t3303-notes-subtrees.sh\nnew file mode 100755\nindex 0000000..d24203e\n--- /dev/null\n+++ b/t/t3303-notes-subtrees.sh\n@@ -0,0 +1,137 @@\n+#!/bin/sh\n+\n+test_description='Test commit notes organized in subtrees'\n+\n+. ./test-lib.sh\n+\n+number_of_commits=100\n+\n+start_note_commit () {\n+\ttest_tick &&\n+\tcat < $GIT_COMMITTER_DATE\n+data < output &&\n+\ti=$number_of_commits &&\n+\twhile [ $i -gt 0 ]; do\n+\t\techo \" commit #$i\" &&\n+\t\techo \" note for commit #$i\" &&\n+\t\ti=$(($i-1));\n+\tdone > expect &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success \"setup: create $number_of_commits commits\" '\n+\n+\t(\n+\t\tnr=0 &&\n+\t\twhile [ $nr -lt $number_of_commits ]; do\n+\t\t\tnr=$(($nr+1)) &&\n+\t\t\ttest_tick &&\n+\t\t\tcat < $GIT_COMMITTER_DATE\n+data < $GIT_COMMITTER_DATE\n+data <From rule #6, we see that the following example notes tree is valid:\n- y2009m09/0123456789abcdef0123456789abcdef012345678\n- y2009m09/123456789abcdef0123456789abcdef0123456789\n- y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n- y2008m01/d31/34/56789abcdef0123456789abcdef0123456789ab\n- y2008m01/d16/4567/89abcdef0123456789abcdef0123456789abc\n- y2008m01/d16/5678/9abcdef0123456789abcdef0123456789abcd\n\nConversely the following structure is invalid (violates rule #6):\n- y2009m09/0123456789abcdef0123456789abcdef012345678\n- y2009m09/12/3456789abcdef0123456789abcdef0123456789\n- y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n- y2008m01/34/56789abcdef0123456789abcdef0123456789ab\n- y2008m01/d16/45/6789abcdef0123456789abcdef0123456789abc\n- y2008/m01d16/5678/9abcdef0123456789abcdef0123456789abcd\n\nThe flexibility added by this patch adds considerable complexity to the notes\ntree parser, but the runtime and memory usage is not significantly affected\n(except for the effects introduced by the chosen notes tree structure).\n\nInternally, the 16-tree data structure introduced in earlier patches is still\nused to hold the SHA1-based fanout levels and the note entries themselves.\nHowever, this patch adds a hierarchical date-based linked-list structure\naround the 16-tree structure that mirrors the fanout scheme used in the\nactual notes tree.\n\nSigned-off-by: Johan Herland \n---\n notes.c | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 364 insertions(+), 39 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 6926aa6..a3e3f83 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -7,6 +7,70 @@\n #include \"tree-walk.h\"\n \n /*\n+ * Format of entries in the notes tree structure:\n+ *\n+ * note-entry ::= (period sep)? sha1-spec\n+ * period ::= year sep?\n+ * (month sep?\n+ * (date sep?)?\n+ * )?;\n+ * year ::= 'y' yearnum;\n+ * month ::= 'm' monthnum;\n+ * date ::= 'd' datenum;\n+ * yearnum ::= # 4-digit decimal year, from annotated commit's timestamp;\n+ * monthnum ::= # 2-digit decimal month, from annotated commit's timestamp;\n+ * datenum ::= # 2-digit decimal date, from annotated commit's timestamp;\n+ * sha1-spec ::= (hex-fragment sep?){20}\n+ * sep ::= '/';\n+ * hex-fragment ::= # Fragment of hexsha1 (2 bytes);\n+ * hexsha1 ::= # SHA1 of annotated commit in hex format (40 bytes);\n+ *\n+ * Thus, the following example note entries are all valid:\n+ * - 0123456789abcdef0123456789abcdef012345678\n+ * - 01/23456789abcdef0123456789abcdef012345678\n+ * - 0123/456789abcdef0123456789abcdef012345678\n+ * - 01/23/456789abcdef0123456789abcdef012345678\n+ * - 0123/4567/89abcdef0123456789abcdef012345678\n+ * - 0123/45/67/89abcdef0123456789abcdef012345678\n+ * - y2009/0123456789abcdef0123456789abcdef012345678\n+ * - y2009/m09/01/23456789abcdef0123456789abcdef012345678\n+ * - y2009/m09/d01/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m09/01/23/456789abcdef0123456789abcdef012345678\n+ * - y2009m09/d01/0123/4567/89abcdef0123456789abcdef012345678\n+ * - y2009/m09d01/01/23/45/67/89abcdef0123456789abcdef012345678\n+ * - y2009m09d01/0123456789abcdef0123456789abcdef012345678\n+ *\n+ * and the following example note entries are all invalid:\n+ * - 0/123456789abcdef0123456789abcdef012345678\n+ * - 012/3456789abcdef0123456789abcdef012345678\n+ * - 01/234/56789abcdef0123456789abcdef012345678\n+ * - y20090123456789abcdef0123456789abcdef012345678\n+ * - y20/09/m09/01/23456789abcdef0123456789abcdef012345678\n+ * - y20/09m09/d01/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m/09/01/23/456789abcdef0123456789abcdef012345678\n+ * - y2009/d01/0123/4567/89abcdef0123456789abcdef012345678\n+ * - m09/y2009/d01/01/23/45/67/89abcdef0123456789abcdef012345678\n+ *\n+ * In addition to the above per-entry rules, we require that _all_ entries at\n+ * a given level in the notes tree (levels are separated by '/') follow the\n+ * exact same format at that level. Thus the following structure is valid:\n+ * - y2009m09/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m09/123456789abcdef0123456789abcdef0123456789\n+ * - y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n+ * - y2008m01/d31/34/56789abcdef0123456789abcdef0123456789ab\n+ * - y2008m01/d16/4567/89abcdef0123456789abcdef0123456789abc\n+ * - y2008m01/d16/5678/9abcdef0123456789abcdef0123456789abcd\n+ *\n+ * but the following structure is invalid:\n+ * - y2009m09/0123456789abcdef0123456789abcdef012345678\n+ * - y2009m09/12/3456789abcdef0123456789abcdef0123456789\n+ * - y2008m01/d31/23/456789abcdef0123456789abcdef0123456789a\n+ * - y2008m01/34/56789abcdef0123456789abcdef0123456789ab\n+ * - y2008m01/d16/45/6789abcdef0123456789abcdef0123456789abc\n+ * - y2008/m01d16/5678/9abcdef0123456789abcdef0123456789abcd\n+ */\n+\n+/*\n * Use a non-balancing simple 16-tree structure with struct int_node as\n * internal nodes, and struct leaf_node as leaf nodes. Each int_node has a\n * 16-array of pointers to its children.\n@@ -17,9 +81,45 @@\n * - ptr & 3 == 3 - pointer to subtree entry - cast to struct leaf_node *\n *\n * The root node is a statically allocated struct int_node.\n+ *\n+ * In order to allow date-based fanout schemes in addition to the original\n+ * SHA1-based fanout schemes, we need to overload this structure, as follows:\n+ * If the first pointer in the 16-array is ~0 (i.e. 0xffffffff on 32-bit\n+ * systems and 0xffffffffffffffff on 64-bit systems), then the int_node is NOT\n+ * to be interpreted as a 16-array of child node pointers. Rather, the int_node\n+ * now represents a period-based node with the following properties:\n+ * - The node has a pointer to a \"child\" node of type struct int_node, which is\n+ * EITHER a \"regular\" int_node object representing the root node of a 16-tree\n+ * structure holding notes associated with commits with timestamps within\n+ * that time period, OR another period-based int_node representing some\n+ * subdivision of the time period.\n+ * - The node also has a pointer to a \"previous\" period-based int_node, which\n+ * represents the previous time period for which there exist note objects.\n+ * - The node has a pointer to a \"parent\" node, which is the period-based\n+ * int_node that has this int_node as one of its children. This is needed\n+ * when traversing the date-based int_nodes looking for a period matching the\n+ * given commit. For top-level objects, this is set to NULL.\n+ * - The node stores the SHA1 sum of the tree object that represents its child\n+ * (within the notes tree structure). Thus, we keep a reference to the child\n+ * structure that without necessarily allocating the child node (and\n+ * underlying structure).\n+ * - Finally, the node has a period string, which indicates the time period of\n+ * the notes contained within, typically of the form \"YYYY\", \"YYYY-MM\" or\n+ * \"YYYY-MM-DD\", depending on the granularity of the corresponding\n+ * period-based entries in the notes tree structure.\n */\n struct int_node {\n-\tvoid *a[16];\n+\tunion {\n+\t\tvoid *a[16];\n+\t\tstruct {\n+\t\t\tvoid *magic; /* ~0 \"enables\" this part of the union */\n+\t\t\tstruct int_node *child;\n+\t\t\tstruct int_node *prev;\n+\t\t\tstruct int_node *parent;\n+\t\t\tunsigned char tree_sha1[20];\n+\t\t\tchar period[11]; /* Enough to hold \"YYYY-MM-DD\" */\n+\t\t};\n+\t};\n };\n \n /*\n@@ -51,12 +151,18 @@ struct leaf_node {\n #define SUBTREE_SHA1_PREFIXCMP(key_sha1, subtree_sha1) \\\n \t(memcmp(key_sha1, subtree_sha1, subtree_sha1[19]))\n \n+#define SUBTREE_DATE_PREFIXCMP(commit_date, subtree_date) \\\n+\t(prefixcmp(commit_date, subtree_date))\n+\n static struct int_node root_node;\n \n+static struct int_node *cur_node;\n+\n static int initialized;\n \n-static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n-\t\tunsigned int n);\n+static void load_subtree(const unsigned char *sha1,\n+\t\tconst unsigned char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, struct int_node *parent, int n);\n \n /*\n * To find a leaf_node:\n@@ -94,7 +200,8 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n \t\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n \t\t\t/* unpack tree and resume search */\n \t\t\ttree->a[i] = NULL;\n-\t\t\tload_subtree(l, tree, n);\n+\t\t\tload_subtree(l->val_sha1, l->key_sha1, l->key_sha1[19],\n+\t\t\t\t tree, NULL, (int) n);\n \t\t\tfree(l);\n \t\t\treturn note_tree_find(tree, n, key_sha1);\n \t\t}\n@@ -117,7 +224,8 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n \tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n \t\t/* unpack tree and resume search */\n \t\ttree->a[0] = NULL;\n-\t\tload_subtree(l, tree, n);\n+\t\tload_subtree(l->val_sha1, l->key_sha1, l->key_sha1[19], tree,\n+\t\t\t NULL, (int) n);\n \t\tfree(l);\n \t\treturn note_tree_find(tree, n, key_sha1);\n \t}\n@@ -173,16 +281,28 @@ static int note_tree_insert(struct int_node *tree, unsigned char n,\n /* Free the entire notes data contained in the given tree */\n static void note_tree_free(struct int_node *tree)\n {\n-\tunsigned int i;\n-\tfor (i = 0; i < 16; i++) {\n-\t\tvoid *p = tree->a[i];\n-\t\tswitch(GET_PTR_TYPE(p)) {\n-\t\tcase PTR_TYPE_INTERNAL:\n-\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n-\t\t\t/* fall through */\n-\t\tcase PTR_TYPE_NOTE:\n-\t\tcase PTR_TYPE_SUBTREE:\n-\t\t\tfree(CLR_PTR_TYPE(p));\n+\tif (tree->magic == (void *) ~0) {\n+\t\tif (tree->prev) {\n+\t\t\tnote_tree_free(tree->prev);\n+\t\t\tfree(tree->prev);\n+\t\t}\n+\t\tif (tree->child) {\n+\t\t\tnote_tree_free(tree->child);\n+\t\t\tfree(tree->child);\n+\t\t}\n+\t}\n+\telse {\n+\t\tunsigned int i;\n+\t\tfor (i = 0; i < 16; i++) {\n+\t\t\tvoid *p = tree->a[i];\n+\t\t\tswitch(GET_PTR_TYPE(p)) {\n+\t\t\tcase PTR_TYPE_INTERNAL:\n+\t\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n+\t\t\t\t/* fall through */\n+\t\t\tcase PTR_TYPE_NOTE:\n+\t\t\tcase PTR_TYPE_SUBTREE:\n+\t\t\t\tfree(CLR_PTR_TYPE(p));\n+\t\t\t}\n \t\t}\n \t}\n }\n@@ -215,29 +335,139 @@ static int get_sha1_hex_segment(const char *hex, unsigned int hex_len,\n \treturn len;\n }\n \n-static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n-\t\tunsigned int n)\n+/*\n+ * Parse year/month/date strings, and generate the corresponding period string\n+ * for the given path entry:\n+ * - prefix must follow one of these forms: \"\", \"YYYY\", \"YYYY-MM\"\n+ * - path should follow one of these forms: \"yYYYY\", \"yYYYYmMM\", \"yYYYYmMMdDD\",\n+ * \"mMMdDD\", \"mMM\" or \"dDD\"\n+ * The resulting string (which follows the form \"YYYY\", \"YYYY-MM\" or\n+ * \"YYYY-MM-DD\") is returned as a static string. If path is not valid in the\n+ * given (prefix) context, NULL is returned.\n+ */\n+static const char *parse_period(const char *prefix, unsigned int prefix_len,\n+\t\tconst char *path, unsigned int path_len)\n+{\n+\tstatic char result[11];\n+\tchar expect_type; /* y/m/d for year/month/day-based fanout */\n+\tunsigned int expect_len, value;\n+\tchar *endptr, *target = result;\n+\n+\tswitch (prefix_len) {\n+\tcase 0:\n+\t\t/* No prefix, expect year-based fanout in path */\n+\t\texpect_type = 'y';\n+\t\texpect_len = 4;\n+\t\tbreak;\n+\tcase 4:\n+\t\t/* Year in prefix, expect month-based fanout in path */\n+\t\texpect_type = 'm';\n+\t\texpect_len = 2;\n+\t\tbreak;\n+\tcase 7:\n+\t\t/* \"YYYY-MM\" in prefix, expect day-based fanout in path */\n+\t\texpect_type = 'd';\n+\t\texpect_len = 2;\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"Date-based notes tree loading invoked with invalid \"\n+\t\t \"prefix '%.*s'\", prefix_len, prefix);\n+\t}\n+\n+\tif (path[0] != expect_type) {\n+\t\twarning(\"Unexpected entry path in date-based notes tree: '%s' \"\n+\t\t\t\"(skipping)\", path);\n+\t\treturn NULL;\n+\t}\n+\tvalue = (unsigned int) strtoul(path + 1, &endptr, 10);\n+\tswitch (expect_type) {\n+\tcase 'y':\n+\t\tif (value < 1969 || value >= 3000) {\n+\t\t\twarning(\"Invalid year value in date-based notes tree:\"\n+\t\t\t\t\" '%s' (skipping)\", path);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tbreak;\n+\tcase 'm':\n+\t\tif (value < 1 || value > 12) {\n+\t\t\twarning(\"Invalid month value in date-based notes tree:\"\n+\t\t\t\t\" '%s' (skipping)\", path);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tbreak;\n+\tcase 'd':\n+\t\tif (value < 1 || value > 31) {\n+\t\t\twarning(\"Invalid day value in date-based notes tree:\"\n+\t\t\t\t\" '%s' (skipping)\", path);\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tbreak;\n+\t}\n+\n+\tif (prefix == result) {\n+\t\ttarget = result + prefix_len;\n+\t\tprefix = NULL;\n+\t\tprefix_len = 0;\n+\t}\n+\tprefix_len = snprintf(target, 11, \"%.*s%s%0*u\", prefix_len, prefix,\n+\t\t\t expect_len == 2 ? \"-\" : \"\", expect_len, value);\n+\tprefix_len += target - result;\n+\tassert(prefix_len < 11);\n+\n+\tif (*endptr) /* there are more components in this path */\n+\t\treturn parse_period(result, prefix_len, endptr,\n+\t\t\t\t path_len - (endptr - path));\n+\treturn result;\n+}\n+\n+static void load_date_subtree(struct tree_desc *tree_desc,\n+\t\tconst char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, struct int_node *parent)\n+{\n+\tstruct name_entry entry;\n+\tstruct int_node *cur_node = NULL;\n+\tstruct int_node *new_node;\n+\n+\twhile (tree_entry(tree_desc, &entry)) {\n+\t\tconst char *period = parse_period(\n+\t\t\tprefix, prefix_len, entry.path, strlen(entry.path));\n+\t\tif (!period)\n+\t\t\tcontinue;\n+\t\tif (tree_desc->size) /* this is not the last tree entry */\n+\t\t\tnew_node = (struct int_node *)\n+\t\t\t\txmalloc(sizeof(struct int_node));\n+\t\telse /* this is the last entry, store directly into node */\n+\t\t\tnew_node = node;\n+\n+\t\tnew_node->magic = (void *) ~0;\n+\t\tnew_node->child = NULL;\n+\t\tnew_node->prev = cur_node;\n+\t\tnew_node->parent = parent;\n+\t\thashcpy(new_node->tree_sha1, entry.sha1);\n+\t\tstrcpy(new_node->period, period);\n+\t\tcur_node = new_node;\n+\t}\n+\tassert(!cur_node || cur_node == node);\n+}\n+\n+static void load_sha1_subtree(struct tree_desc *tree_desc,\n+\t\tconst unsigned char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, unsigned char n)\n {\n \tunsigned char commit_sha1[20];\n-\tunsigned int prefix_len;\n \tint status;\n-\tvoid *buf;\n-\tstruct tree_desc desc;\n \tstruct name_entry entry;\n \n-\tbuf = fill_tree_descriptor(&desc, subtree->val_sha1);\n-\tif (!buf)\n-\t\tdie(\"Could not read %s for notes-index\",\n-\t\t sha1_to_hex(subtree->val_sha1));\n-\n-\tprefix_len = subtree->key_sha1[19];\n \tassert(prefix_len * 2 >= n);\n-\tmemcpy(commit_sha1, subtree->key_sha1, prefix_len);\n-\twhile (tree_entry(&desc, &entry)) {\n+\tmemcpy(commit_sha1, prefix, prefix_len);\n+\twhile (tree_entry(tree_desc, &entry)) {\n \t\tint len = get_sha1_hex_segment(entry.path, strlen(entry.path),\n \t\t\t\tcommit_sha1 + prefix_len, 20 - prefix_len);\n-\t\tif (len < 0)\n+\t\tif (len < 0) {\n+\t\t\twarning(\"Invalid value in notes tree: '%s' (skipping)\",\n+\t\t\t\tentry.path);\n \t\t\tcontinue; /* entry.path is not a SHA1 sum. Skip */\n+\t\t}\n \t\tlen += prefix_len;\n \n \t\t/*\n@@ -258,6 +488,42 @@ static void load_subtree(struct leaf_node *subtree, struct int_node *node,\n \t\t\tassert(!status);\n \t\t}\n \t}\n+}\n+\n+static void load_subtree(const unsigned char *sha1,\n+\t\tconst unsigned char *prefix, unsigned int prefix_len,\n+\t\tstruct int_node *node, struct int_node *parent, int n)\n+{\n+\tvoid *buf;\n+\tstruct tree_desc desc;\n+\n+\tbuf = fill_tree_descriptor(&desc, sha1);\n+\tif (!buf)\n+\t\tdie(\"Could not read notes subtree at %s\", sha1_to_hex(sha1));\n+\t/*\n+\t * After fill_tree_descriptor(), we can peek at the first tree entry\n+\t * in desc.entry.\n+\t */\n+\tswitch (desc.entry.path[0]) {\n+\tcase 'd':\n+\t\tif (strlen(desc.entry.path) != 3)\n+\t\t\tbreak;\n+\t\t/* fall-through */\n+\tcase 'm':\n+\tcase 'y':\n+\t\t/* path cannot be a SHA1 fragment */\n+\t\tload_date_subtree(&desc, (const char *) prefix, prefix_len,\n+\t\t\t\t node, parent);\n+\t\tfree(buf);\n+\t\treturn;\n+\t}\n+\tif (n < 0) {\n+\t\t/* Arriving from a date-based subtree; reset prefix */\n+\t\tn = 0;\n+\t\tprefix = NULL;\n+\t\tprefix_len = 0;\n+\t}\n+\tload_sha1_subtree(&desc, prefix, prefix_len, node, n);\n \tfree(buf);\n }\n \n@@ -265,23 +531,81 @@ static void initialize_notes(const char *notes_ref_name)\n {\n \tunsigned char sha1[20], commit_sha1[20];\n \tunsigned mode;\n-\tstruct leaf_node root_tree;\n \n \tif (!notes_ref_name || read_ref(notes_ref_name, commit_sha1) ||\n \t get_tree_entry(commit_sha1, \"\", sha1, &mode))\n \t\treturn;\n \n-\thashclr(root_tree.key_sha1);\n-\thashcpy(root_tree.val_sha1, sha1);\n-\tload_subtree(&root_tree, &root_node, 0);\n+\tload_subtree(sha1, NULL, 0, &root_node, NULL, 0);\n+\tcur_node = &root_node;\n }\n \n-static unsigned char *lookup_notes(const unsigned char *commit_sha1)\n+static unsigned char *lookup_notes(const struct commit *commit)\n {\n-\tstruct leaf_node *found = note_tree_find(&root_node, 0, commit_sha1);\n-\tif (found)\n-\t\treturn found->val_sha1;\n-\treturn NULL;\n+\tstruct int_node *node = cur_node, *seen_node = cur_node;\n+\tstruct leaf_node *found;\n+\tconst char *short_date;\n+\n+\tif (!node)\n+\t\treturn NULL;\n+\n+\t/* Convert commit->date to YYYY-MM-DD format */\n+\tshort_date = show_date(commit->date, 0, DATE_SHORT);\n+\n+\twhile (node->magic == (void *) ~0) { /* date-based node */\n+\t\tint cmp = SUBTREE_DATE_PREFIXCMP(short_date, node->period);\n+\t\tif (cmp == 0) {\n+\t\t\t/* Search inside child node */\n+\t\t\tif (!node->child) {\n+\t\t\t\t/* Must unpack child node first */\n+\t\t\t\tnode->child = (struct int_node *)\n+\t\t\t\t\txcalloc(sizeof(struct int_node), 1);\n+\t\t\t\tload_subtree(node->tree_sha1,\n+\t\t\t\t\t(const unsigned char *) node->period,\n+\t\t\t\t\tstrlen(node->period), node->child,\n+\t\t\t\t\tnode, -1);\n+\t\t\t}\n+\t\t\tseen_node = node;\n+\t\t\tnode = node->child;\n+\t\t}\n+\t\telse if (cmp > 0) {\n+\t\t\t/* Search in past node */\n+\t\t\tif (node->prev)\n+\t\t\t\tnode = node->prev;\n+\t\t\telse\n+\t\t\t\tnode = node->parent;\n+\t\t}\n+\t\telse {\n+\t\t\t/* Search in future node */\n+\t\t\tif (!node->parent) {\n+\t\t\t\t/* Restart from root_node */\n+\t\t\t\tseen_node = node;\n+\t\t\t\tnode = &root_node;\n+\t\t\t}\n+\t\t\telse\n+\t\t\t\tnode = node->parent;\n+\t\t}\n+\t\tif (!node || node == seen_node) {\n+\t\t\t/* We've been here before, give up search */\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\twhile (cur_node &&\n+\t SUBTREE_DATE_PREFIXCMP(cur_node->period, seen_node->period) < 0)\n+\t{\n+\t\t/*\n+\t\t * We're about to move cur_node backwards in history. We are\n+\t\t * unlikely to need this cur_node in the future, so free() it.\n+\t\t */\n+\t\tnote_tree_free(cur_node->child);\n+\t\tcur_node->child = NULL;\n+\t\tcur_node = cur_node->parent;\n+\t}\n+\tcur_node = seen_node;\n+\n+\t/* Drill down further with SHA1-based lookup */\n+\tfound = note_tree_find(node, 0, commit->object.sha1);\n+\treturn found ? found->val_sha1 : NULL;\n }\n \n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n@@ -303,7 +627,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tinitialized = 1;\n \t}\n \n-\tsha1 = lookup_notes(commit->object.sha1);\n+\tsha1 = lookup_notes(commit);\n \tif (!sha1)\n \t\treturn;\n \n@@ -342,6 +666,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \n void free_commit_notes()\n {\n+\tcur_node = NULL;\n \tnote_tree_free(&root_node);\n \tmemset(&root_node, 0, sizeof(struct int_node));\n \tinitialized = 0;\n-- \n1.6.4.304.g1365c.dirty\n"},{"id":"122965","messageId":"1252771728-27206-15-git-send-email-johan@herland.net","threadId":"20872","inReplyTo":"200909121752.07523.johan@herland.net","subject":"[PATCHv6 14/14] Add test cases for various date-based fanouts","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T16:08:48Z","receivedAt":"2009-09-12T16:08:48Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Signed-off-by: Johan Herland \n---\n t/t3303-notes-subtrees.sh | 64 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 64 insertions(+), 0 deletions(-)\n\ndiff --git a/t/t3303-notes-subtrees.sh b/t/t3303-notes-subtrees.sh\nindex d24203e..d3cbf6d 100755\n--- a/t/t3303-notes-subtrees.sh\n+++ b/t/t3303-notes-subtrees.sh\n@@ -134,4 +134,68 @@ test_expect_success 'verify notes in 4/36-fanout overriding 2/38-fanout' 'verify\n test_expect_success 'test notes in 2/38-fanout overriding 2/2/36-fanout' 'test_preferred \"s|^..|&/|\" \"s|^\\(..\\)\\(..\\)|\\1/\\2/|\"'\n test_expect_success 'verify notes in 2/38-fanout overriding 2/2/36-fanout' 'verify_notes'\n \n+test_date_based () {\n+\t(\n+\t\tstart_note_commit &&\n+\t\tnr=$number_of_commits &&\n+\t\tgit log --format=\"%H %ct\" refs/heads/master |\n+\t\twhile read sha1 date_t; do\n+\t\t\tdate=$(date -u -d \"@$date_t\" +\"$1\")\n+\t\t\tnote_path=\"$date/$(echo \"$sha1\" | sed \"$2\")\"\n+\t\t\tcat < wrote:\n> Shawn, do you have any additional defence for the date-based fanout?\n\nNo.\n\nThe only defense I have for it is \"it sounds like a nice theory\ngiven access patterns\", and the note about memory usage you made,\nbut which I clipped to keep this email shorter. :-)\n\nIt was only a theory I tossed out there in a back-seat-driver\nsort of way. Your results show my hunch was correct, it may help.\nBut they also say it may not help enough to justify the complexity,\nso I now agree with you that SHA-1 fan out may be good enough.\n\n> Are \n> there untested reasonable scenarios that would show the benefits of date-\n> based fanout?\n\nI don't think there are, your tests were pretty good at covering\nthings.\n\n> How does the plan for notes usage in your code-review thingy \n> compare to my test scenario?\n\nI think your tests may still have been too low in volume, 115k notes\nisn't a lot. Based on the distributions I was looking at before,\nI could be seeing a growth of >100k notes/year. Ask me again in\n5 years if 115k notes is a lot. :-)\n\nBut we all know that SHA-1 distributes data quite well, so the SHA-1\nfan-out may just need to change from 2_38 to 2_2_2_34 (or something)\nto handle that larger volume.\n\n-- \nShawn.\n"},{"id":"122974","messageId":"200909122035.47051.johan@herland.net","threadId":"20872","inReplyTo":"20090912181150.GN1033@spearce.org","subject":"Re: [PATCHv5 00/14] git notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T18:35:46Z","receivedAt":"2009-09-12T18:35:46Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Saturday 12 September 2009, Shawn O. Pearce wrote:\n> Johan Herland wrote:\n> > Shawn, do you have any additional defence for the date-based fanout?\n> \n> No.\n> \n> The only defense I have for it is \"it sounds like a nice theory\n> given access patterns\", and the note about memory usage you made,\n> but which I clipped to keep this email shorter. :-)\n> \n> It was only a theory I tossed out there in a back-seat-driver\n> sort of way. Your results show my hunch was correct, it may help.\n> But they also say it may not help enough to justify the complexity,\n> so I now agree with you that SHA-1 fan out may be good enough.\n\nOk, so I guess we can drop the flexible part of notes code. Junio: Feel free \nto drop the two last patches from the jh/notes series.\n\n> > How does the plan for notes usage in your code-review thingy\n> > compare to my test scenario?\n> \n> I think your tests may still have been too low in volume, 115k notes\n> isn't a lot. Based on the distributions I was looking at before,\n> I could be seeing a growth of >100k notes/year. Ask me again in\n> 5 years if 115k notes is a lot. :-)\n> \n> But we all know that SHA-1 distributes data quite well, so the SHA-1\n> fan-out may just need to change from 2_38 to 2_2_2_34 (or something)\n> to handle that larger volume.\n\nYes, I expect that the optimal number of entries per tree level is ~256, so \nif we add an upper threshold at ~300 (where we start using another fanout \nlevel), and a lower threshold at ~200 (where we consolidate subtrees and put \nall into this level), the (still-to-be-written) writing part of the notes \ncode should automatically adjust the notes tree to the optimal layout.\n\nWith those assumptions, and a growth of 100k notes/year, a 2/2/36 fanout \nshould last you ~150 years, and a 2/2/2/34 fanout should be enough for the \nnext ~40,000 years... ;)\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"122975","messageId":"7vmy50f1mf.fsf@alter.siamese.dyndns.org","threadId":"20872","inReplyTo":"1252771728-27206-11-git-send-email-johan@herland.net","subject":"Re: [PATCHv6 10/14] Teach notes code to free its internal data structures on request.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-12T18:40:08Z","receivedAt":"2009-09-12T18:40:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> There's no need to be rude to memory-concious callers...\n\nWill squash this in.\n\n-- >8 --\nFrom: Junio C Hamano \nDate: Sat, 12 Sep 2009 11:34:24 -0700\nSubject: [PATCH] notes.[ch] fixup: avoid old-style declaration\n\nSigned-off-by: Junio C Hamano \n---\n notes.c | 2 +-\n notes.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex 008c3d4..9ed2c87 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -161,7 +161,7 @@ void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \tfree(msg);\n }\n \n-void free_commit_notes()\n+void free_commit_notes(void)\n {\n \tfree(hash_map.entries);\n \tmemset(&hash_map, 0, sizeof(struct hash_map));\ndiff --git a/notes.h b/notes.h\nindex 41802e5..d1dd1d1 100644\n--- a/notes.h\n+++ b/notes.h\n@@ -7,6 +7,6 @@\n void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n \t\tconst char *output_encoding, int flags);\n \n-void free_commit_notes();\n+void free_commit_notes(void);\n \n #endif\n-- \n1.6.5.rc0.82.g1c5d9\n"},{"id":"122976","messageId":"7viqfof1kc.fsf@alter.siamese.dyndns.org","threadId":"20872","inReplyTo":"1252771728-27206-14-git-send-email-johan@herland.net","subject":"Re: [PATCHv6 13/14] Allow flexible organization of notes trees, using both commit date and SHA1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-12T18:41:23Z","receivedAt":"2009-09-12T18:41:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> This is a major expansion of the notes lookup code to allow for variations\n> in the notes tree organization. The variations allowed include mixing fanout\n> schemes based on the commit dates of the annotated commits (aka. date-based\n> fanout) with fanout schemes based on the SHA1 of the annotated commits (aka.\n> SHA1-based fanout).\n\nWill squash this in.\n\n-- >8 --\nFrom: Junio C Hamano \nDate: Sat, 12 Sep 2009 11:36:42 -0700\nSubject: [PATCH] notes.[ch] fixup: avoid unnamed struct members\n\nSigned-off-by: Junio C Hamano \n---\n notes.c | 82 +++++++++++++++++++++++++++++++-------------------------------\n 1 files changed, 41 insertions(+), 41 deletions(-)\n\ndiff --git a/notes.c b/notes.c\nindex c573ffa..9a92853 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -118,8 +118,8 @@ struct int_node {\n \t\t\tstruct int_node *parent;\n \t\t\tunsigned char tree_sha1[20];\n \t\t\tchar period[11]; /* Enough to hold \"YYYY-MM-DD\" */\n-\t\t};\n-\t};\n+\t\t} s;\n+\t} u;\n };\n \n /*\n@@ -182,7 +182,7 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n {\n \tstruct leaf_node *l;\n \tunsigned char i = GET_NIBBLE(n, key_sha1);\n-\tvoid *p = tree->a[i];\n+\tvoid *p = tree->u.a[i];\n \n \tswitch(GET_PTR_TYPE(p)) {\n \tcase PTR_TYPE_INTERNAL:\n@@ -199,7 +199,7 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n \t\tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n \t\tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n \t\t\t/* unpack tree and resume search */\n-\t\t\ttree->a[i] = NULL;\n+\t\t\ttree->u.a[i] = NULL;\n \t\t\tload_subtree(l->val_sha1, l->key_sha1, l->key_sha1[19],\n \t\t\t\t tree, NULL, (int) n);\n \t\t\tfree(l);\n@@ -214,16 +214,16 @@ static struct leaf_node *note_tree_find(struct int_node *tree, unsigned char n,\n \n \t/*\n \t * Did not find key at this (or any lower) level.\n-\t * Check if there's a matching subtree entry in tree->a[0].\n+\t * Check if there's a matching subtree entry in tree->u.a[0].\n \t * If so, unpack tree and resume search.\n \t */\n-\tp = tree->a[0];\n+\tp = tree->u.a[0];\n \tif (GET_PTR_TYPE(p) != PTR_TYPE_SUBTREE)\n \t\treturn NULL;\n \tl = (struct leaf_node *) CLR_PTR_TYPE(p);\n \tif (!SUBTREE_SHA1_PREFIXCMP(key_sha1, l->key_sha1)) {\n \t\t/* unpack tree and resume search */\n-\t\ttree->a[0] = NULL;\n+\t\ttree->u.a[0] = NULL;\n \t\tload_subtree(l->val_sha1, l->key_sha1, l->key_sha1[19], tree,\n \t\t\t NULL, (int) n);\n \t\tfree(l);\n@@ -250,12 +250,12 @@ static int note_tree_insert(struct int_node *tree, unsigned char n,\n \tconst struct leaf_node *l;\n \tint ret;\n \tunsigned char i = GET_NIBBLE(n, entry->key_sha1);\n-\tvoid *p = tree->a[i];\n+\tvoid *p = tree->u.a[i];\n \tassert(GET_PTR_TYPE(entry) == PTR_TYPE_NULL);\n \tswitch(GET_PTR_TYPE(p)) {\n \tcase PTR_TYPE_NULL:\n \t\tassert(!p);\n-\t\ttree->a[i] = SET_PTR_TYPE(entry, type);\n+\t\ttree->u.a[i] = SET_PTR_TYPE(entry, type);\n \t\treturn 0;\n \tcase PTR_TYPE_INTERNAL:\n \t\treturn note_tree_insert(CLR_PTR_TYPE(p), n + 1, entry, type);\n@@ -273,7 +273,7 @@ static int note_tree_insert(struct int_node *tree, unsigned char n,\n \t\t\tfree(new_node);\n \t\t\treturn -1;\n \t\t}\n-\t\ttree->a[i] = SET_PTR_TYPE(new_node, PTR_TYPE_INTERNAL);\n+\t\ttree->u.a[i] = SET_PTR_TYPE(new_node, PTR_TYPE_INTERNAL);\n \t\treturn note_tree_insert(new_node, n + 1, entry, type);\n \t}\n }\n@@ -281,20 +281,20 @@ static int note_tree_insert(struct int_node *tree, unsigned char n,\n /* Free the entire notes data contained in the given tree */\n static void note_tree_free(struct int_node *tree)\n {\n-\tif (tree->magic == (void *) ~0) {\n-\t\tif (tree->prev) {\n-\t\t\tnote_tree_free(tree->prev);\n-\t\t\tfree(tree->prev);\n+\tif (tree->u.s.magic == (void *) ~0) {\n+\t\tif (tree->u.s.prev) {\n+\t\t\tnote_tree_free(tree->u.s.prev);\n+\t\t\tfree(tree->u.s.prev);\n \t\t}\n-\t\tif (tree->child) {\n-\t\t\tnote_tree_free(tree->child);\n-\t\t\tfree(tree->child);\n+\t\tif (tree->u.s.magic) {\n+\t\t\tnote_tree_free(tree->u.s.magic);\n+\t\t\tfree(tree->u.s.magic);\n \t\t}\n \t}\n \telse {\n \t\tunsigned int i;\n \t\tfor (i = 0; i < 16; i++) {\n-\t\t\tvoid *p = tree->a[i];\n+\t\t\tvoid *p = tree->u.a[i];\n \t\t\tswitch(GET_PTR_TYPE(p)) {\n \t\t\tcase PTR_TYPE_INTERNAL:\n \t\t\t\tnote_tree_free(CLR_PTR_TYPE(p));\n@@ -439,12 +439,12 @@ static void load_date_subtree(struct tree_desc *tree_desc,\n \t\telse /* this is the last entry, store directly into node */\n \t\t\tnew_node = node;\n \n-\t\tnew_node->magic = (void *) ~0;\n-\t\tnew_node->child = NULL;\n-\t\tnew_node->prev = cur_node;\n-\t\tnew_node->parent = parent;\n-\t\thashcpy(new_node->tree_sha1, entry.sha1);\n-\t\tstrcpy(new_node->period, period);\n+\t\tnew_node->u.s.magic = (void *) ~0;\n+\t\tnew_node->u.s.magic = NULL;\n+\t\tnew_node->u.s.prev = cur_node;\n+\t\tnew_node->u.s.parent = parent;\n+\t\thashcpy(new_node->u.s.tree_sha1, entry.sha1);\n+\t\tstrcpy(new_node->u.s.period, period);\n \t\tcur_node = new_node;\n \t}\n \tassert(!cur_node || cur_node == node);\n@@ -552,38 +552,38 @@ static unsigned char *lookup_notes(const struct commit *commit)\n \t/* Convert commit->date to YYYY-MM-DD format */\n \tshort_date = show_date(commit->date, 0, DATE_SHORT);\n \n-\twhile (node->magic == (void *) ~0) { /* date-based node */\n-\t\tint cmp = SUBTREE_DATE_PREFIXCMP(short_date, node->period);\n+\twhile (node->u.s.magic == (void *) ~0) { /* date-based node */\n+\t\tint cmp = SUBTREE_DATE_PREFIXCMP(short_date, node->u.s.period);\n \t\tif (cmp == 0) {\n \t\t\t/* Search inside child node */\n-\t\t\tif (!node->child) {\n+\t\t\tif (!node->u.s.magic) {\n \t\t\t\t/* Must unpack child node first */\n-\t\t\t\tnode->child = (struct int_node *)\n+\t\t\t\tnode->u.s.magic = (struct int_node *)\n \t\t\t\t\txcalloc(sizeof(struct int_node), 1);\n-\t\t\t\tload_subtree(node->tree_sha1,\n-\t\t\t\t\t(const unsigned char *) node->period,\n-\t\t\t\t\tstrlen(node->period), node->child,\n+\t\t\t\tload_subtree(node->u.s.tree_sha1,\n+\t\t\t\t\t(const unsigned char *) node->u.s.period,\n+\t\t\t\t\tstrlen(node->u.s.period), node->u.s.magic,\n \t\t\t\t\tnode, -1);\n \t\t\t}\n \t\t\tseen_node = node;\n-\t\t\tnode = node->child;\n+\t\t\tnode = node->u.s.magic;\n \t\t}\n \t\telse if (cmp > 0) {\n \t\t\t/* Search in past node */\n-\t\t\tif (node->prev)\n-\t\t\t\tnode = node->prev;\n+\t\t\tif (node->u.s.prev)\n+\t\t\t\tnode = node->u.s.prev;\n \t\t\telse\n-\t\t\t\tnode = node->parent;\n+\t\t\t\tnode = node->u.s.parent;\n \t\t}\n \t\telse {\n \t\t\t/* Search in future node */\n-\t\t\tif (!node->parent) {\n+\t\t\tif (!node->u.s.parent) {\n \t\t\t\t/* Restart from root_node */\n \t\t\t\tseen_node = node;\n \t\t\t\tnode = &root_node;\n \t\t\t}\n \t\t\telse\n-\t\t\t\tnode = node->parent;\n+\t\t\t\tnode = node->u.s.parent;\n \t\t}\n \t\tif (!node || node == seen_node) {\n \t\t\t/* We've been here before, give up search */\n@@ -591,15 +591,15 @@ static unsigned char *lookup_notes(const struct commit *commit)\n \t\t}\n \t}\n \twhile (cur_node &&\n-\t SUBTREE_DATE_PREFIXCMP(cur_node->period, seen_node->period) < 0)\n+\t SUBTREE_DATE_PREFIXCMP(cur_node->u.s.period, seen_node->u.s.period) < 0)\n \t{\n \t\t/*\n \t\t * We're about to move cur_node backwards in history. We are\n \t\t * unlikely to need this cur_node in the future, so free() it.\n \t\t */\n-\t\tnote_tree_free(cur_node->child);\n-\t\tcur_node->child = NULL;\n-\t\tcur_node = cur_node->parent;\n+\t\tnote_tree_free(cur_node->u.s.magic);\n+\t\tcur_node->u.s.magic = NULL;\n+\t\tcur_node = cur_node->u.s.parent;\n \t}\n \tcur_node = seen_node;\n \n-- \n1.6.5.rc0.82.g1c5d9\n"},{"id":"122998","messageId":"200909130021.38296.johan@herland.net","threadId":"20872","inReplyTo":"7vmy50f1mf.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCHv6 10/14] Teach notes code to free its internal data structures on request.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T22:21:38Z","receivedAt":"2009-09-12T22:21:38Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Saturday 12 September 2009, Junio C Hamano wrote:\n> Johan Herland writes:\n> > There's no need to be rude to memory-concious callers...\n> \n> Will squash this in.\n> \n> -- >8 --\n> From: Junio C Hamano \n> Date: Sat, 12 Sep 2009 11:34:24 -0700\n> Subject: [PATCH] notes.[ch] fixup: avoid old-style declaration\n> \n> Signed-off-by: Junio C Hamano \n> ---\n> notes.c | 2 +-\n> notes.h | 2 +-\n> 2 files changed, 2 insertions(+), 2 deletions(-)\n> \n> diff --git a/notes.c b/notes.c\n> index 008c3d4..9ed2c87 100644\n> --- a/notes.c\n> +++ b/notes.c\n> @@ -161,7 +161,7 @@ void get_commit_notes(const struct commit *commit,\n> struct strbuf *sb, free(msg);\n> }\n> \n> -void free_commit_notes()\n> +void free_commit_notes(void)\n> {\n> \tfree(hash_map.entries);\n> \tmemset(&hash_map, 0, sizeof(struct hash_map));\n> diff --git a/notes.h b/notes.h\n> index 41802e5..d1dd1d1 100644\n> --- a/notes.h\n> +++ b/notes.h\n> @@ -7,6 +7,6 @@\n> void get_commit_notes(const struct commit *commit, struct strbuf *sb,\n> \t\tconst char *output_encoding, int flags);\n> \n> -void free_commit_notes();\n> +void free_commit_notes(void);\n> \n> #endif\n\nThanks,\n\nAcked-by: Johan Herland \n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"123002","messageId":"200909130033.28666.johan@herland.net","threadId":"20872","inReplyTo":"7viqfof1kc.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCHv6 13/14] Allow flexible organization of notes trees, using both commit date and SHA1","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-09-12T22:33:28Z","receivedAt":"2009-09-12T22:33:28Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Saturday 12 September 2009, Junio C Hamano wrote:\n> Johan Herland writes:\n> > This is a major expansion of the notes lookup code to allow for\n> > variations in the notes tree organization. The variations allowed\n> > include mixing fanout schemes based on the commit dates of the\n> > annotated commits (aka. date-based fanout) with fanout schemes based on\n> > the SHA1 of the annotated commits (aka. SHA1-based fanout).\n\nNote that this patch is about to be removed from this series, cf. today's \ndiscussion with Shawn elsewhere in this thread.\n\n> Will squash this in.\n\nI agree with the attempt, but not all of the hunks are good:\n\n> @@ -281,20 +281,20 @@ static int note_tree_insert(struct int_node *tree,\n> unsigned char n, /* Free the entire notes data contained in the given\n> tree */\n> static void note_tree_free(struct int_node *tree)\n> {\n> -\tif (tree->magic == (void *) ~0) {\n> -\t\tif (tree->prev) {\n> -\t\t\tnote_tree_free(tree->prev);\n> -\t\t\tfree(tree->prev);\n> +\tif (tree->u.s.magic == (void *) ~0) {\n> +\t\tif (tree->u.s.prev) {\n> +\t\t\tnote_tree_free(tree->u.s.prev);\n> +\t\t\tfree(tree->u.s.prev);\n> \t\t}\n> -\t\tif (tree->child) {\n> -\t\t\tnote_tree_free(tree->child);\n> -\t\t\tfree(tree->child);\n> +\t\tif (tree->u.s.magic) {\n> +\t\t\tnote_tree_free(tree->u.s.magic);\n> +\t\t\tfree(tree->u.s.magic);\n\nHere, you are replacing tree->child with tree->u.s.magic. Shouldn't that be \ntree->u.s.child instead?\n\n> @@ -439,12 +439,12 @@ static void load_date_subtree(struct tree_desc\n> *tree_desc, else /* this is the last entry, store directly into node */\n> \t\t\tnew_node = node;\n> \n> -\t\tnew_node->magic = (void *) ~0;\n> -\t\tnew_node->child = NULL;\n> -\t\tnew_node->prev = cur_node;\n> -\t\tnew_node->parent = parent;\n> -\t\thashcpy(new_node->tree_sha1, entry.sha1);\n> -\t\tstrcpy(new_node->period, period);\n> +\t\tnew_node->u.s.magic = (void *) ~0;\n> +\t\tnew_node->u.s.magic = NULL;\n\nSame as above: new_node->u.s.child\n\n> +\t\tnew_node->u.s.prev = cur_node;\n> +\t\tnew_node->u.s.parent = parent;\n> +\t\thashcpy(new_node->u.s.tree_sha1, entry.sha1);\n> +\t\tstrcpy(new_node->u.s.period, period);\n> \t\tcur_node = new_node;\n> \t}\n> \tassert(!cur_node || cur_node == node);\n> @@ -552,38 +552,38 @@ static unsigned char *lookup_notes(const struct\n> commit *commit) /* Convert commit->date to YYYY-MM-DD format */\n> \tshort_date = show_date(commit->date, 0, DATE_SHORT);\n> \n> -\twhile (node->magic == (void *) ~0) { /* date-based node */\n> -\t\tint cmp = SUBTREE_DATE_PREFIXCMP(short_date, node->period);\n> +\twhile (node->u.s.magic == (void *) ~0) { /* date-based node */\n> +\t\tint cmp = SUBTREE_DATE_PREFIXCMP(short_date, node->u.s.period);\n> \t\tif (cmp == 0) {\n> \t\t\t/* Search inside child node */\n> -\t\t\tif (!node->child) {\n> +\t\t\tif (!node->u.s.magic) {\n> \t\t\t\t/* Must unpack child node first */\n> -\t\t\t\tnode->child = (struct int_node *)\n> +\t\t\t\tnode->u.s.magic = (struct int_node *)\n> \t\t\t\t\txcalloc(sizeof(struct int_node), 1);\n> -\t\t\t\tload_subtree(node->tree_sha1,\n> -\t\t\t\t\t(const unsigned char *) node->period,\n> -\t\t\t\t\tstrlen(node->period), node->child,\n> +\t\t\t\tload_subtree(node->u.s.tree_sha1,\n> +\t\t\t\t\t(const unsigned char *) node->u.s.period,\n> +\t\t\t\t\tstrlen(node->u.s.period), node->u.s.magic,\n> \t\t\t\t\tnode, -1);\n> \t\t\t}\n> \t\t\tseen_node = node;\n> -\t\t\tnode = node->child;\n> +\t\t\tnode = node->u.s.magic;\n\nSame again, 4 times in the above hunk.\n\n> @@ -591,15 +591,15 @@ static unsigned char *lookup_notes(const struct\n> commit *commit) }\n> \t}\n> \twhile (cur_node &&\n> -\t SUBTREE_DATE_PREFIXCMP(cur_node->period, seen_node->period) < 0)\n> +\t SUBTREE_DATE_PREFIXCMP(cur_node->u.s.period,\n> seen_node->u.s.period) < 0) {\n> \t\t/*\n> \t\t * We're about to move cur_node backwards in history. We are\n> \t\t * unlikely to need this cur_node in the future, so free() it.\n> \t\t */\n> -\t\tnote_tree_free(cur_node->child);\n> -\t\tcur_node->child = NULL;\n> -\t\tcur_node = cur_node->parent;\n> +\t\tnote_tree_free(cur_node->u.s.magic);\n> +\t\tcur_node->u.s.magic = NULL;\n\n...and another one here.\n\n> +\t\tcur_node = cur_node->u.s.parent;\n> \t}\n> \tcur_node = seen_node;\n> \n\n\nBut as I said above, you may want to drop 13/14 and 14/14 completely, \ninstead.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"123009","messageId":"7vzl8zbupe.fsf@alter.siamese.dyndns.org","threadId":"20872","inReplyTo":"200909130033.28666.johan@herland.net","subject":"Re: [PATCHv6 13/14] Allow flexible organization of notes trees, using both commit date and SHA1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-12T23:37:49Z","receivedAt":"2009-09-12T23:37:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> But as I said above, you may want to drop 13/14 and 14/14 completely, \n> instead.\n\nThanks. Will do.\n"}]}