{"thread":{"id":"8421","subject":"Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","startedAt":"2007-06-04T00:51:05Z","lastAt":"2007-06-10T19:42:48Z","messageCount":52,"participants":["Johan Herland","Matthias Lederhofer","Junio C Hamano","Steven Grimm","Johannes Schindelin","Pierre Habouzit","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"43927","messageId":"200706040251.05286.johan@herland.net","threadId":"8421","inReplyTo":null,"subject":"Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:51:05Z","receivedAt":"2007-06-04T00:51:05Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"(In response to the earlier thread on the git 'notes' feature/idea: \"[PATCH \n00/15] git-note: A mechanism for providing free-form after-the-fact \nannotations on commits\")\n\nOk, this is getting too big for me to ship all in one patch series, so I'm \ngonna have to split it up. Right now I'm thinking three patch series, but \nas I'm not near finished yet, that may change.\n\nThe three patch series I'm talking about are:\n\n1. Refactoring the git tag object. This lays down part of the groundwork \nneeded to support the second incarnation of git 'notes'. This is all ready \nto go and should follow shortly after this mail.\n\n2. Introducing soft references (softrefs). Softrefs is the general mechanism \nbehind the \"reverse mapping\" needed to support 'notes', as discussed in the \nprevious thread. I'm still working on this patch series, and it will \nhopefully be ready soon-ish.\n\n3. Reimplementing git 'notes' on top of (1) and (2). This should be fairly \neasy (once (2) is done), as it's mostly a matter of building a simple \nporcelain on top of (1) and (2).\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"43928","messageId":"200706040251.52613.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.05286.johan@herland.net","subject":"[PATCH 0/6] Refactor the tag object","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:51:52Z","receivedAt":"2007-06-04T00:51:52Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"This patch series implements part of the ground work for the 'notes'\nfeature discussed earlier in the thread \"[PATCH 00/15] git-note: A\nmechanism for providing free-form after-the-fact annotations on commits\".\n\nThe following patches refactors the tag object by:\n1. Making the \"tag\" header optional\n2. Introducing a new optional \"keywords\" header\n3. Making the \"tagger\" header mandatory as far as possible\n4. Do better and more thorough verification of tag objects\n\nUnfortunately the first patch in the series is bigger than I would have\nliked, but I couldn't find an easy way to split it up.\n\nHere's the shortlog:\n\nJohan Herland (6):\n Refactor git tag objects; make \"tag\" header optional; introduce new optional \"keywords\" header\n git-show: When showing tag objects with no tag name, show tag object's SHA1 instead of an empty string\n git-fsck: Do thorough verification of tag objects.\n Documentation/git-mktag: Document the changes in tag object structure\n git-mktag tests: Fix and expand the mktag tests according to the new tag object structure\n Add fsck_verify_ref_to_tag_object() to verify that refname matches name stored in tag object\n\n Documentation/git-mktag.txt | 42 +++++--\n builtin-fsck.c | 35 ++++++\n builtin-log.c | 2 +-\n mktag.c | 148 +++++-------------------\n t/t3800-mktag.sh | 204 ++++++++++++++++++++++++++++++---\n tag.c | 266 +++++++++++++++++++++++++++++++++++--------\n tag.h | 4 +-\n 7 files changed, 507 insertions(+), 194 deletions(-)\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"43929","messageId":"200706040252.55152.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH 1/6] Refactor git tag objects; make \"tag\" header optional; introduce new optional \"keywords\" header","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:52:54Z","receivedAt":"2007-06-04T00:52:54Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"\nIn order to support ref-less tag objects (aka. 'notes'), we want to do\nsome changes to the tag object structure. The new structure implemented\nin this patch is backward-compatible in the way that all existing tag\nobjects (valid in the old implementation) will remain valid in the new\nimplementation. The following changes are done:\n\n1. Make the \"tag\" header optional. The \"tag\" header contains the tag name,\n which is optional for 'notes'. The new semantics for the \"tag\" header\n are as follows: The tag header _must_ be given for signed tags (this\n is already enforced by git-tag.sh). When the tag header is not given,\n its value defaults to the empty string.\n\n2. Introduce a new optional \"keywords\" header. The \"keywords\" header is a\n comma-separated list of free-form values. However, two certain value\n have special meaning: \"tag\" and \"note\": When the \"keywords\" header is\n missing, its default value is set to \"tag\" if a \"tag\" header is\n present; else the default \"keywords\" value is set to \"note\". The\n \"keywords\" header is meant to be used by porcelains for classifying\n different types of tag objects. This classification may then be used to\n filter tag objects in the presentation layer (e.g. by implementing\n extra filter options to --decorate, etc.)\n\n3. Make the \"tagger\" header mandatory. This header has already for a long\n time been \"de facto\" mandatory, in that it is automatically generated\n by git-tag.sh. This patch verifies the existence of the header when\n creating tag objects. However, since there exists old tags without\n the \"tagger\" header, the verification should not be done when parsing\n tag objects.\n\n4. Consolidate the parsing and verification of tag objects. Currently\n parsing is done in tag.c:parse_tag_buffer(), and verification is done\n with a very similar piece of code in mktag.c:verify_tag(). This patch\n unifies the parsing and verification of tag objects in a new function\n parse_and_verify_tag_buffer() which is then called from the other\n places.\n\nSigned-off-by: Johan Herland \n---\n mktag.c | 148 ++++++++---------------------------\n tag.c | 266 +++++++++++++++++++++++++++++++++++++++++++++++++++------------\n tag.h | 4 +-\n 3 files changed, 251 insertions(+), 167 deletions(-)\n\ndiff --git a/mktag.c b/mktag.c\nindex 9310111..cde6036 100644\n--- a/mktag.c\n+++ b/mktag.c\n@@ -2,128 +2,42 @@\n #include \"tag.h\"\n \n /*\n- * A signature file has a very simple fixed format: four lines\n- * of \"object \" + \"type \" + \"tag \" +\n+ * Tag object data has the following format: two mandatory lines of\n+ * \"object \" + \"type \", plus two optional lines of\n+ * \"tag \" + \"keywords \", plus a mandatory line of\n * \"tagger \", followed by a blank line, a free-form tag\n- * message and a signature block that git itself doesn't care about,\n- * but that can be verified with gpg or similar.\n+ * message and an optional signature block that git itself doesn't\n+ * care about, but that can be verified with gpg or similar.\n *\n- * The first three lines are guaranteed to be at least 63 bytes:\n- * \"object \\n\" is 48 bytes, \"type tag\\n\" at 9 bytes is the\n- * shortest possible type-line, and \"tag .\\n\" at 6 bytes is the\n- * shortest single-character-tag line. \n+ * represents the object pointed to by this tag, is\n+ * the type of the object pointed to (\"tag\", \"blob\", \"tree\" or \"commit\"),\n+ * is the name of this tag object (and must correspond to the\n+ * name of the corresponding ref (if any) in '.git/refs/'). is\n+ * a comma-separated list of keywords associated with this tag object, and\n+ * holds the \"name \" of the tag creator and timestamp\n+ * of when the tag object was created (analogous to \"committer\" in commit\n+ * objects).\n *\n- * We also artificially limit the size of the full object to 8kB.\n- * Just because I'm a lazy bastard, and if you can't fit a signature\n- * in that size, you're doing something wrong.\n- */\n-\n-/* Some random size */\n-#define MAXSIZE (8192)\n-\n-/*\n- * We refuse to tag something we can't verify. Just because.\n+ * The first two lines are guaranteed to be at least 57 bytes:\n+ * \"object \\n\" is 48 bytes, and \"type tag\\n\" at 9 bytes is\n+ * the shortest possible \"type\" line. The tagger line is at least\n+ * \"tagger \\n\" (8 bytes), and a blank line is also needed (1 byte).\n+ * Therefore a tag object _must_ have >= 66 bytes.\n+ *\n+ * If \"tag \" is omitted, defaults to the empty string.\n+ * If \"keywords \" is omitted, defaults to \"tag\" if\n+ * a was given, \"note\" otherwise.\n */\n-static int verify_object(unsigned char *sha1, const char *expected_type)\n-{\n-\tint ret = -1;\n-\tenum object_type type;\n-\tunsigned long size;\n-\tvoid *buffer = read_sha1_file(sha1, &type, &size);\n-\n-\tif (buffer) {\n-\t\tif (type == type_from_string(expected_type))\n-\t\t\tret = check_sha1_signature(sha1, buffer, size, expected_type);\n-\t\tfree(buffer);\n-\t}\n-\treturn ret;\n-}\n-\n-#ifdef NO_C99_FORMAT\n-#define PD_FMT \"%d\"\n-#else\n-#define PD_FMT \"%td\"\n-#endif\n-\n-static int verify_tag(char *buffer, unsigned long size)\n-{\n-\tint typelen;\n-\tchar type[20];\n-\tunsigned char sha1[20];\n-\tconst char *object, *type_line, *tag_line, *tagger_line;\n-\n-\tif (size < 64)\n-\t\treturn error(\"wanna fool me ? you obviously got the size wrong !\");\n-\n-\tbuffer[size] = 0;\n-\n-\t/* Verify object line */\n-\tobject = buffer;\n-\tif (memcmp(object, \"object \", 7))\n-\t\treturn error(\"char%d: does not start with \\\"object \\\"\", 0);\n-\n-\tif (get_sha1_hex(object + 7, sha1))\n-\t\treturn error(\"char%d: could not get SHA1 hash\", 7);\n-\n-\t/* Verify type line */\n-\ttype_line = object + 48;\n-\tif (memcmp(type_line - 1, \"\\ntype \", 6))\n-\t\treturn error(\"char%d: could not find \\\"\\\\ntype \\\"\", 47);\n-\n-\t/* Verify tag-line */\n-\ttag_line = strchr(type_line, '\\n');\n-\tif (!tag_line)\n-\t\treturn error(\"char\" PD_FMT \": could not find next \\\"\\\\n\\\"\", type_line - buffer);\n-\ttag_line++;\n-\tif (memcmp(tag_line, \"tag \", 4) || tag_line[4] == '\\n')\n-\t\treturn error(\"char\" PD_FMT \": no \\\"tag \\\" found\", tag_line - buffer);\n-\n-\t/* Get the actual type */\n-\ttypelen = tag_line - type_line - strlen(\"type \\n\");\n-\tif (typelen >= sizeof(type))\n-\t\treturn error(\"char\" PD_FMT \": type too long\", type_line+5 - buffer);\n-\n-\tmemcpy(type, type_line+5, typelen);\n-\ttype[typelen] = 0;\n-\n-\t/* Verify that the object matches */\n-\tif (verify_object(sha1, type))\n-\t\treturn error(\"char%d: could not verify object %s\", 7, sha1_to_hex(sha1));\n-\n-\t/* Verify the tag-name: we don't allow control characters or spaces in it */\n-\ttag_line += 4;\n-\tfor (;;) {\n-\t\tunsigned char c = *tag_line++;\n-\t\tif (c == '\\n')\n-\t\t\tbreak;\n-\t\tif (c > ' ')\n-\t\t\tcontinue;\n-\t\treturn error(\"char\" PD_FMT \": could not verify tag name\", tag_line - buffer);\n-\t}\n-\n-\t/* Verify the tagger line */\n-\ttagger_line = tag_line;\n-\n-\tif (memcmp(tagger_line, \"tagger\", 6) || (tagger_line[6] == '\\n'))\n-\t\treturn error(\"char\" PD_FMT \": could not find \\\"tagger\\\"\", tagger_line - buffer);\n-\n-\t/* TODO: check for committer info + blank line? */\n-\t/* Also, the minimum length is probably + \"tagger .\", or 63+8=71 */\n-\n-\t/* The actual stuff afterwards we don't care about.. */\n-\treturn 0;\n-}\n-\n-#undef PD_FMT\n \n int main(int argc, char **argv)\n {\n \tunsigned long size = 4096;\n \tchar *buffer = xmalloc(size);\n+\tstruct tag result_tag;\n \tunsigned char result_sha1[20];\n \n \tif (argc != 1)\n-\t\tusage(\"git-mktag < signaturefile\");\n+\t\tusage(\"git-mktag < tag_data_file\");\n \n \tsetup_git_directory();\n \n@@ -132,16 +46,18 @@ int main(int argc, char **argv)\n \t\tdie(\"could not read from stdin\");\n \t}\n \n-\t/* Verify it for some basic sanity: it needs to start with\n-\t \"object \\ntype\\ntagger \" */\n-\tif (verify_tag(buffer, size) < 0)\n-\t\tdie(\"invalid tag signature file\");\n+\t/* Verify tag object data */\n+\tif (parse_and_verify_tag_buffer(&result_tag, buffer, size, 1)) {\n+\t\tfree(buffer);\n+\t\tdie(\"invalid tag data file\");\n+\t}\n \n-\tif (write_sha1_file(buffer, size, tag_type, result_sha1) < 0)\n+\tif (write_sha1_file(buffer, size, tag_type, result_sha1) < 0) {\n+\t\tfree(buffer);\n \t\tdie(\"unable to write tag file\");\n+\t}\n \n \tfree(buffer);\n-\n \tprintf(\"%s\\n\", sha1_to_hex(result_sha1));\n \treturn 0;\n }\ndiff --git a/tag.c b/tag.c\nindex bbacd59..9c95e0b 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -33,65 +33,231 @@ struct tag *lookup_tag(const unsigned char *sha1)\n return (struct tag *) obj;\n }\n \n-int parse_tag_buffer(struct tag *item, void *data, unsigned long size)\n+/*\n+ * We refuse to tag something we can't verify. Just because.\n+ */\n+static int verify_object(unsigned char *sha1, const char *expected_type)\n+{\n+\tint ret = -1;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *buffer = read_sha1_file(sha1, &type, &size);\n+\n+\tif (buffer) {\n+\t\tif (type == type_from_string(expected_type))\n+\t\t\tret = check_sha1_signature(sha1, buffer, size, expected_type);\n+\t\tfree(buffer);\n+\t}\n+\treturn ret;\n+}\n+\n+/*\n+ * Perform parsing and verification of tag object data.\n+ *\n+ * The 'item' parameter may be set to NULL if only verification is desired.\n+ */\n+int parse_and_verify_tag_buffer(struct tag *item, const char *data, const unsigned long size, int thorough_verify)\n {\n-\tint typelen, taglen;\n+#ifdef NO_C99_FORMAT\n+#define PD_FMT \"%d\"\n+#else\n+#define PD_FMT \"%td\"\n+#endif\n+\n \tunsigned char sha1[20];\n-\tconst char *type_line, *tag_line, *sig_line;\n \tchar type[20];\n+\tconst char *type_line, *tag_line, *keywords_line, *tagger_line;\n+\tunsigned long type_len, tag_len, keywords_len, tagger_len;\n+\tconst char *header_end, *end = data + size;\n \n- if (item->object.parsed)\n- return 0;\n- item->object.parsed = 1;\n-\n-\tif (size < 64)\n-\t\treturn -1;\n-\tif (memcmp(\"object \", data, 7) || get_sha1_hex((char *) data + 7, sha1))\n-\t\treturn -1;\n-\n-\ttype_line = (char *) data + 48;\n-\tif (memcmp(\"\\ntype \", type_line-1, 6))\n-\t\treturn -1;\n-\n-\ttag_line = strchr(type_line, '\\n');\n-\tif (!tag_line || memcmp(\"tag \", ++tag_line, 4))\n-\t\treturn -1;\n-\n-\tsig_line = strchr(tag_line, '\\n');\n-\tif (!sig_line)\n-\t\treturn -1;\n-\tsig_line++;\n-\n-\ttypelen = tag_line - type_line - strlen(\"type \\n\");\n-\tif (typelen >= 20)\n-\t\treturn -1;\n-\tmemcpy(type, type_line + 5, typelen);\n-\ttype[typelen] = '\\0';\n-\ttaglen = sig_line - tag_line - strlen(\"tag \\n\");\n-\titem->tag = xmalloc(taglen + 1);\n-\tmemcpy(item->tag, tag_line + 4, taglen);\n-\titem->tag[taglen] = '\\0';\n-\n-\tif (!strcmp(type, blob_type)) {\n-\t\titem->tagged = &lookup_blob(sha1)->object;\n-\t} else if (!strcmp(type, tree_type)) {\n-\t\titem->tagged = &lookup_tree(sha1)->object;\n-\t} else if (!strcmp(type, commit_type)) {\n-\t\titem->tagged = &lookup_commit(sha1)->object;\n-\t} else if (!strcmp(type, tag_type)) {\n-\t\titem->tagged = &lookup_tag(sha1)->object;\n-\t} else {\n-\t\terror(\"Unknown type %s\", type);\n-\t\titem->tagged = NULL;\n+\tif (item) {\n+\t\tif (item->object.parsed)\n+\t\t\treturn 0;\n+\t\titem->object.parsed = 1;\n \t}\n \n-\tif (item->tagged && track_object_refs) {\n-\t\tstruct object_refs *refs = alloc_object_refs(1);\n-\t\trefs->ref[0] = item->tagged;\n-\t\tset_object_refs(&item->object, refs);\n+\tif (size < 66)\n+\t\treturn error(\"failed preliminary size check\");\n+\n+\t/* Verify mandatory object line */\n+\tif (memcmp(data, \"object \", 7))\n+\t\treturn error(\"char%d: does not start with \\\"object \\\"\", 0);\n+\n+\tif (get_sha1_hex(data + 7, sha1))\n+\t\treturn error(\"char%d: could not get SHA1 hash\", 7);\n+\n+\t/* Verify mandatory type line */\n+\ttype_line = data + 48;\n+\tif (memcmp(type_line - 1, \"\\ntype \", 6))\n+\t\treturn error(\"char%d: could not find \\\"\\\\ntype \\\"\", 47);\n+\n+\t/* Verify optional tag line */\n+\ttag_line = memchr(type_line, '\\n', end - type_line);\n+\tif (!tag_line++)\n+\t\treturn error(\"char\" PD_FMT \": could not find \\\"\\\\n\\\" after \\\"type\\\"\", type_line - data);\n+\tif (end - tag_line < 4)\n+\t\treturn error(\"char\" PD_FMT \": premature end of data\", tag_line - data);\n+\tif (memcmp(\"tag \", tag_line, 4))\n+\t\tkeywords_line = tag_line; /* no tag name given */\n+\telse { /* tag name given */\n+\t\tkeywords_line = memchr(tag_line, '\\n', end - tag_line);\n+\t\tif (!keywords_line++)\n+\t\t\treturn error(\"char\" PD_FMT \": could not find \\\"\\\\n\\\" after \\\"tag\\\"\", tag_line - data);\n+\t}\n+\n+\t/* Verify optional keywords line */\n+\tif (end - keywords_line < 9)\n+\t\treturn error(\"char\" PD_FMT \": premature end of data\", keywords_line - data);\n+\tif (memcmp(\"keywords \", keywords_line, 9))\n+\t\ttagger_line = keywords_line; /* no keywords given */\n+\telse { /* keywords given */\n+\t\ttagger_line = memchr(keywords_line, '\\n', end - keywords_line);\n+\t\tif (!tagger_line++)\n+\t\t\treturn error(\"char\" PD_FMT \": could not find \\\"\\\\n\\\" after \\\"keywords\\\"\", keywords_line - data);\n+\t}\n+\n+\tif (thorough_verify) {\n+\t\t/*\n+\t\t * Verify mandatory tagger line, but only when we're checking\n+\t\t * thoroughly, i.e. on inserting a new tag, and on fsck.\n+\t\t * There are existing tag objects without a tagger line (most\n+\t\t * notably the \"v0.99\" tag in the main git repo), and we don't\n+\t\t * want to fail parsing on these.\n+\t\t */\n+\t\tif (end - tagger_line < 7)\n+\t\t\treturn error(\"char\" PD_FMT \": premature end of data\", tagger_line - data);\n+\t\tif (memcmp(\"tagger \", tagger_line, 7))\n+\t\t\treturn error(\"char\" PD_FMT \": could not find \\\"tagger \\\"\", tagger_line - data);\n+\t\theader_end = memchr(tagger_line, '\\n', end - tagger_line);\n+\t\tif (!header_end++)\n+\t\t\treturn error(\"char\" PD_FMT \": could not find \\\"\\\\n\\\" after \\\"tagger\\\"\", tagger_line - data);\n+\t\tif (end - header_end < 1)\n+\t\t\treturn error(\"char\" PD_FMT \": premature end of data\", header_end - data);\n+\t\tif (*header_end != '\\n') /* header must end with \"\\n\\n\" */\n+\t\t\treturn error(\"char\" PD_FMT \": could not find blank line after header section\", header_end - data);\n+\t}\n+\telse {\n+\t\t/* Treat tagger line as optional */\n+\t\tif (end - tagger_line >= 7 && !memcmp(\"tagger \", tagger_line, 7)) {\n+\t\t\t/* Found tagger line */\n+\t\t\theader_end = memchr(tagger_line, '\\n', end - tagger_line);\n+\t\t\tif (!header_end++)\n+\t\t\t\treturn error(\"char\" PD_FMT \": could not find \\\"\\\\n\\\" after \\\"tagger\\\"\", tagger_line - data);\n+\t\t}\n+\t\telse /* No tagger line */\n+\t\t\theader_end = tagger_line;\n \t}\n \n+\tif (end - header_end < 1)\n+\t\treturn error(\"char\" PD_FMT \": premature end of data\", header_end - data);\n+\tif (*header_end != '\\n') /* header must end with \"\\n\\n\" */\n+\t\treturn error(\"char\" PD_FMT \": could not find blank line after header section\", header_end - data);\n+\n+\t/* Calculate lengths of header fields */\n+\ttype_len = tag_line == type_line ? 0 : /* 0 if not given, > 0 if given */\n+\t\t\t(tag_line - type_line) - strlen(\"type \\n\");\n+\ttag_len = keywords_line == tag_line ? 0 : /* 0 if not given, > 0 if given */\n+\t\t\t(keywords_line - tag_line) - strlen(\"tag \\n\");\n+\tkeywords_len = tagger_line == keywords_line ? 0 : /* 0 if not given, > 0 if given */\n+\t\t\t(tagger_line - keywords_line) - strlen(\"keywords \\n\");\n+\ttagger_len = header_end == tagger_line ? 0 : /* 0 if not given, > 0 if given */\n+\t\t\t(header_end - tagger_line) - strlen(\"tagger \\n\");\n+\n+\t/* Get the actual type */\n+\tif (type_len >= sizeof(type))\n+\t\treturn error(\"char\" PD_FMT \": type too long\", (type_line + 5) - data);\n+\tmemcpy(type, type_line + 5, type_len);\n+\ttype[type_len] = '\\0';\n+\n+\tif (thorough_verify) {\n+\t\t/* Verify that the object matches */\n+\t\tif (verify_object(sha1, type))\n+\t\t\treturn error(\"char%d: could not verify object %s\", 7, sha1_to_hex(sha1));\n+\n+\t\t/* Verify the tag name: we don't allow control characters or spaces in it */\n+\t\tif (tag_len > 0) { /* tag name was given */\n+\t\t\ttag_line += 4; /* skip past \"tag \" */\n+\t\t\tfor (;;) {\n+\t\t\t\tunsigned char c = *tag_line++;\n+\t\t\t\tif (c == '\\n')\n+\t\t\t\t\tbreak;\n+\t\t\t\tif (c > ' ' && c != 0x7f)\n+\t\t\t\t\tcontinue;\n+\t\t\t\treturn error(\"char\" PD_FMT \": could not verify tag name\", tag_line - data);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Verify the keywords line: we don't allow control characters or spaces in it, or two subsequent commas */\n+\t\tif (keywords_len > 0) { /* keywords line was given */\n+\t\t\tkeywords_line += 9; /* skip past \"keywords \" */\n+\t\t\tfor (;;) {\n+\t\t\t\tunsigned char c = *keywords_line++;\n+\t\t\t\tif (c == '\\n')\n+\t\t\t\t\tbreak;\n+\t\t\t\tif (c == ',' && *keywords_line == ',')\n+\t\t\t\t\treturn error(\"char\" PD_FMT \": found empty keyword\", keywords_line - data);\n+\t\t\t\tif (c > ' ' && c != 0x7f)\n+\t\t\t\t\tcontinue;\n+\t\t\t\treturn error(\"char\" PD_FMT \": could not verify keywords\", keywords_line - data);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Verify the tagger line */\n+\t\t/* TODO: check for committer/tagger info */\n+\n+\t\t/* The actual stuff afterwards we don't care about.. */\n+\t}\n+\n+\tif (item) { /* Store parsed information into item */\n+\t\tif (tag_len > 0) { /* optional tag name was given */\n+\t\t\titem->tag = xmalloc(tag_len + 1);\n+\t\t\tmemcpy(item->tag, tag_line + 4, tag_len);\n+\t\t\titem->tag[tag_len] = '\\0';\n+\t\t}\n+\t\telse { /* optional tag name not given */\n+\t\t\titem->tag = xmalloc(1);\n+\t\t\titem->tag[0] = '\\0';\n+\t\t}\n+\n+\t\tif (keywords_len > 0) { /* optional keywords string was given */\n+\t\t\titem->keywords = xmalloc(keywords_len + 1);\n+\t\t\tmemcpy(item->keywords, keywords_line + 9, keywords_len);\n+\t\t\titem->keywords[keywords_len] = '\\0';\n+\t\t}\n+\t\telse { /* optional keywords string not given. Set default keywords */\n+\t\t\t/* if tag name is set, use \"tag\"; otherwise use \"note\" */\n+\t\t\tconst char *default_kw = item->tag ? \"tag\" : \"note\";\n+\t\t\titem->keywords = xmalloc(strlen(default_kw) + 1);\n+\t\t\tmemcpy(item->keywords, default_kw, strlen(default_kw) + 1);\n+\t\t}\n+\n+\t\tif (!strcmp(type, blob_type)) {\n+\t\t\titem->tagged = &lookup_blob(sha1)->object;\n+\t\t} else if (!strcmp(type, tree_type)) {\n+\t\t\titem->tagged = &lookup_tree(sha1)->object;\n+\t\t} else if (!strcmp(type, commit_type)) {\n+\t\t\titem->tagged = &lookup_commit(sha1)->object;\n+\t\t} else if (!strcmp(type, tag_type)) {\n+\t\t\titem->tagged = &lookup_tag(sha1)->object;\n+\t\t} else {\n+\t\t\terror(\"Unknown type %s\", type);\n+\t\t\titem->tagged = NULL;\n+\t\t}\n+\n+\t\tif (item->tagged && track_object_refs) {\n+\t\t\tstruct object_refs *refs = alloc_object_refs(1);\n+\t\t\trefs->ref[0] = item->tagged;\n+\t\t\tset_object_refs(&item->object, refs);\n+\t\t}\n+\t}\n \treturn 0;\n+\n+#undef PD_FMT\n+}\n+\n+int parse_tag_buffer(struct tag *item, void *data, unsigned long size)\n+{\n+\treturn parse_and_verify_tag_buffer(item, (const char *) data, size, 0);\n }\n \n int parse_tag(struct tag *item)\ndiff --git a/tag.h b/tag.h\nindex 7a0cb00..6853594 100644\n--- a/tag.h\n+++ b/tag.h\n@@ -8,11 +8,13 @@ extern const char *tag_type;\n struct tag {\n \tstruct object object;\n \tstruct object *tagged;\n-\tchar *tag;\n+\tchar *tag; /* optional, may be empty (\"\") */\n+\tchar *keywords; /* optional, defaults to tag ? \"tag\" : \"note\" */\n \tchar *signature; /* not actually implemented */\n };\n \n extern struct tag *lookup_tag(const unsigned char *sha1);\n+extern int parse_and_verify_tag_buffer(struct tag *item, const char *data, const unsigned long size, int thorough_verify);\n extern int parse_tag_buffer(struct tag *item, void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n extern struct object *deref_tag(struct object *, const char *, int);\n-- \n1.5.2\n"},{"id":"43930","messageId":"200706040253.26845.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH 2/6] git-show: When showing tag objects with no tag name, show tag object's SHA1 instead of an empty string","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:53:26Z","receivedAt":"2007-06-04T00:53:26Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"This is a consequence of making the \"tag\" header in tag objects optional.\n\nSigned-off-by: Johan Herland \n---\n builtin-log.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-log.c b/builtin-log.c\nindex 3744712..1a0f111 100644\n--- a/builtin-log.c\n+++ b/builtin-log.c\n@@ -181,7 +181,7 @@ int cmd_show(int argc, const char **argv, const char *prefix)\n \t\t\tprintf(\"%stag %s%s\\n\\n\",\n \t\t\t\t\tdiff_get_color(rev.diffopt.color_diff,\n \t\t\t\t\t\tDIFF_COMMIT),\n-\t\t\t\t\tt->tag,\n+\t\t\t\t\t*(t->tag) ? t->tag : name,\n \t\t\t\t\tdiff_get_color(rev.diffopt.color_diff,\n \t\t\t\t\t\tDIFF_RESET));\n \t\t\tret = show_object(o->sha1, 1);\n-- \n1.5.2\n"},{"id":"43931","messageId":"200706040253.51920.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:53:51Z","receivedAt":"2007-06-04T00:53:51Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Teach git-fsck to do the same kind of verification on tag objects that is\nalready done by git-mktag.\n\nSigned-off-by: Johan Herland \n---\n builtin-fsck.c | 14 ++++++++++++++\n 1 files changed, 14 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex cbbcaf0..a8914ae 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -344,6 +344,20 @@ static int fsck_commit(struct commit *commit)\n static int fsck_tag(struct tag *tag)\n {\n \tstruct object *tagged = tag->tagged;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tchar *data = (char *) read_sha1_file(tag->object.sha1, &type, &size);\n+\tif (!data)\n+\t\treturn error(\"Could not read tag %s\", sha1_to_hex(tag->object.sha1));\n+\tif (type != OBJ_TAG) {\n+\t\tfree(data);\n+\t\treturn error(\"Internal error: Tag %s not a tag\", sha1_to_hex(tag->object.sha1));\n+\t}\n+\tif (parse_and_verify_tag_buffer(0, data, size, 1)) { /* Thoroughly verify tag object */\n+\t\tfree(data);\n+\t\treturn error(\"Tag %s failed thorough tag object verification\", sha1_to_hex(tag->object.sha1));\n+\t}\n+\tfree(data);\n \n \tif (!tagged) {\n \t\treturn objerror(&tag->object, \"could not load tagged object\");\n-- \n1.5.2\n"},{"id":"43932","messageId":"200706040254.13631.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH 4/6] Documentation/git-mktag: Document the changes in tag object structure","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:54:13Z","receivedAt":"2007-06-04T00:54:13Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The new structure of tag objects is documented.\n\nAlso some much-needed cleanup is done. E.g. remove the paragraph on the\n8kB limit, since this limit was removed ages ago.\n\nSigned-off-by: Johan Herland \n---\n Documentation/git-mktag.txt | 42 ++++++++++++++++++++++++++++++------------\n 1 files changed, 30 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-mktag.txt b/Documentation/git-mktag.txt\nindex 2860a3d..411105d 100644\n--- a/Documentation/git-mktag.txt\n+++ b/Documentation/git-mktag.txt\n@@ -8,40 +8,58 @@ git-mktag - Creates a tag object\n \n SYNOPSIS\n --------\n-'git-mktag' < signature_file\n+[verse]\n+'git-mktag' < tag_data_file\n+\n \n DESCRIPTION\n -----------\n-Reads a tag contents on standard input and creates a tag object\n+Reads tag object data on standard input and creates a tag object\n that can also be used to sign other objects.\n \n The output is the new tag's identifier.\n \n-Tag Format\n+\n+DISCUSSION\n ----------\n-A tag signature file has a very simple fixed format: three lines of\n+Tag object data has the following format\n \n+[verse]\n object \n type \n- tag \n+ tag (optional)\n+ keywords (optional)\n+ tagger \n+\n+followed by a blank line and a free-form message and an optional signature\n+that git itself doesn't care about, but that may be verified with gpg or\n+similar.\n \n-followed by some 'optional' free-form signature that git itself\n-doesn't care about, but that can be verified with gpg or similar.\n+In the above listing, `` represents the object pointed to by this tag,\n+`` is the type of the object pointed to (\"tag\", \"blob\", \"tree\" or\n+\"commit\"), `` is the name of this tag object (and must correspond\n+to the name of the corresponding ref (if any) in `.git/refs/`). ``\n+is a comma-separated list of keywords associated with this tag object, and\n+`` holds the \"`name `\" of the tag creator and timestamp\n+of when the tag object was created (analogous to \"committer\" in commit\n+objects).\n \n-The size of the full object is artificially limited to 8kB. (Just\n-because I'm a lazy bastard, and if you can't fit a signature in that\n-size, you're doing something wrong)\n+If \"`tag `\" is omitted, defaults to the empty string.\n+If \"`keywords `\" is omitted, defaults to \"`tag`\" if\n+a was given, \"`note`\" otherwise.\n \n \n Author\n ------\n Written by Linus Torvalds \n \n+\n Documentation\n --------------\n-Documentation by David Greaves, Junio C Hamano and the git-list .\n+Documentation by Johan Herland, David Greaves, Junio C Hamano and the\n+git-list .\n+\n \n GIT\n ---\n Part of the gitlink:git[7] suite\n-\n-- \n1.5.2\n"},{"id":"43933","messageId":"200706040254.34554.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH 5/6] git-mktag tests: Fix and expand the mktag tests according to the new tag object structure","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:54:34Z","receivedAt":"2007-06-04T00:54:34Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The existing tests are updated to reflect the changes in the tag object.\n\nAdditionally some more tests are added to test the new \"keywords\" header,\nand to test the more thorough verification routine.\n\nSigned-off-by: Johan Herland \n---\n t/t3800-mktag.sh | 204 ++++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 files changed, 190 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t3800-mktag.sh b/t/t3800-mktag.sh\nindex 7c7e433..f6e3d10 100755\n--- a/t/t3800-mktag.sh\n+++ b/t/t3800-mktag.sh\n@@ -34,7 +34,7 @@ too short for a tag\n EOF\n \n cat >expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig\n printf \"type tagsssssssssssssssssssssssssssssss\" >>tag.sig\n \n cat >expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <expect.pat <tag.sig <.git/refs/tags/mytag 2>message'\n \n ############################################################\n-# 14. check mytag\n+# 20. check mytag\n \n test_expect_success \\\n 'check mytag' \\\n 'git-tag -l | grep mytag'\n \n+############################################################\n+# 21. create valid tag #2\n+\n+cat >tag.sig <.git/refs/tags/mytag 2>message'\n+\n+############################################################\n+# 22. create valid tag #3\n+\n+cat >tag.sig <.git/refs/tags/mytag 2>message'\n+\n+############################################################\n+# 23. create valid tag #4\n+\n+cat >tag.sig <.git/refs/tags/mytag 2>message'\n+\n \n test_done\n-- \n1.5.2\n"},{"id":"43934","messageId":"200706040254.56482.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH 6/6] Add fsck_verify_ref_to_tag_object() to verify that refname matches name stored in tag object","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T00:54:56Z","receivedAt":"2007-06-04T00:54:56Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Monday 28 May 2007, Junio C Hamano wrote:\n> However it would be a good\n> idea to add logic to fsck to warn upon inconsistencis (perhaps\n> by mistake) between refname and tag's true name.\n>\n> The check would say something like:\n>\n> If an annotated (signed or unsigned) tag has a \"tag\"\n> line to give it the official $name, and if it is pointed\n> at by a ref, the refname must end with \"/$name\".\n> Otherwise we warn.\n>\n> Trivially, the above rule says that having v2.6.22 tag under\n> refs/tags/v2.6.20 is a mistake we would want to be warned upon.\n\nThis patch adds the check described by Junio.\n\nSigned-off-by: Johan Herland \n---\n builtin-fsck.c | 21 +++++++++++++++++++++\n 1 files changed, 21 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex a8914ae..379317e 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -515,6 +515,25 @@ static int fsck_handle_reflog(const char *logname, const unsigned char *sha1, in\n \treturn 0;\n }\n \n+static void fsck_verify_ref_to_tag_object(const char *refname, struct object *obj)\n+{\n+\t/* Verify that refname matches the name stored in obj's \"tag\" header */\n+\tstruct tag *tagobj = (struct tag *) parse_object(obj->sha1);\n+\tsize_t tagname_len = strlen(tagobj->tag);\n+\tsize_t refname_len = strlen(refname);\n+\n+\tif (!tagname_len) return; /* No tag name stored in tagobj. Nothing to do. */\n+\n+\tif (tagname_len < refname_len &&\n+\t !memcmp(tagobj->tag, refname + (refname_len - tagname_len), tagname_len) &&\n+\t refname[(refname_len - tagname_len) - 1] == '/') {\n+\t\t/* OK: tag name is \"$name\", and refname ends with \"/$name\" */\n+\t\treturn;\n+\t}\n+\telse\n+\t\terror(\"%s: Mismatch between tag ref and tag object's name %s\", refname, tagobj->tag);\n+}\n+\n static int fsck_handle_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n {\n \tstruct object *obj;\n@@ -529,6 +548,8 @@ static int fsck_handle_ref(const char *refname, const unsigned char *sha1, int f\n \t\t/* We'll continue with the rest despite the error.. */\n \t\treturn 0;\n \t}\n+\tif (obj->type == OBJ_TAG) /* ref to tag object */\n+\t\tfsck_verify_ref_to_tag_object(refname, obj);\n \tdefault_refs++;\n \tobj->used = 1;\n \tmark_reachable(obj, REACHABLE);\n-- \n1.5.2\n"},{"id":"43940","messageId":"20070604055655.GB15148@moooo.ath.cx","threadId":"8421","inReplyTo":"200706040253.51920.johan@herland.net","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Matthias Lederhofer","fromEmail":"matled@gmx.net","sentAt":"2007-06-04T05:56:55Z","receivedAt":"2007-06-04T05:56:55Z","isPatch":true,"sender":{"key":"matled@gmx.net","avatar":null},"body":"Johan Herland wrote:\n> diff --git a/builtin-fsck.c b/builtin-fsck.c\n> index cbbcaf0..a8914ae 100644\n> --- a/builtin-fsck.c\n> +++ b/builtin-fsck.c\n> @@ -344,6 +344,20 @@ static int fsck_commit(struct commit *commit)\n> static int fsck_tag(struct tag *tag)\n> {\n> \tstruct object *tagged = tag->tagged;\n> +\tenum object_type type;\n> +\tunsigned long size;\n> +\tchar *data = (char *) read_sha1_file(tag->object.sha1, &type, &size);\n> +\tif (!data)\n> +\t\treturn error(\"Could not read tag %s\", sha1_to_hex(tag->object.sha1));\n> +\tif (type != OBJ_TAG) {\n> +\t\tfree(data);\n> +\t\treturn error(\"Internal error: Tag %s not a tag\", sha1_to_hex(tag->object.sha1));\n> +\t}\n> +\tif (parse_and_verify_tag_buffer(0, data, size, 1)) { /* Thoroughly verify tag object */\n> +\t\tfree(data);\n> +\t\treturn error(\"Tag %s failed thorough tag object verification\", sha1_to_hex(tag->object.sha1));\n> +\t}\n> +\tfree(data);\n> \n> \tif (!tagged) {\n> \t\treturn objerror(&tag->object, \"could not load tagged object\");\n\nThe objerror() function prints the sha1 and object type, I think this\none should be used instead of error() here.\n"},{"id":"43942","messageId":"20070604060812.GC15148@moooo.ath.cx","threadId":"8421","inReplyTo":"200706040252.55152.johan@herland.net","subject":"Re: [PATCH 1/6] Refactor git tag objects; make \"tag\" header optional; introduce new optional \"keywords\" header","fromName":"Matthias Lederhofer","fromEmail":"matled@gmx.net","sentAt":"2007-06-04T06:08:12Z","receivedAt":"2007-06-04T06:08:12Z","isPatch":true,"sender":{"key":"matled@gmx.net","avatar":null},"body":"Johan Herland wrote:\n> 1. Make the \"tag\" header optional. The \"tag\" header contains the tag name,\n> which is optional for 'notes'. The new semantics for the \"tag\" header\n> are as follows: The tag header _must_ be given for signed tags (this\n> is already enforced by git-tag.sh). When the tag header is not given,\n> its value defaults to the empty string.\n\nWhy must signed tags have a tag header? Will notes optionally have a\ntag header?\n"},{"id":"43950","messageId":"200706040930.14287.johan@herland.net","threadId":"8421","inReplyTo":"20070604060812.GC15148@moooo.ath.cx","subject":"Re: [PATCH 1/6] Refactor git tag objects; make \"tag\" header optional; introduce new optional \"keywords\" header","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T07:30:14Z","receivedAt":"2007-06-04T07:30:14Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Monday 04 June 2007, Matthias Lederhofer wrote:\n> Johan Herland wrote:\n> > 1. Make the \"tag\" header optional. The \"tag\" header contains the tag \nname,\n> > which is optional for 'notes'. The new semantics for the \"tag\" header\n> > are as follows: The tag header _must_ be given for signed tags (this\n> > is already enforced by git-tag.sh). When the tag header is not given,\n> > its value defaults to the empty string.\n> \n> Why must signed tags have a tag header? Will notes optionally have a\n> tag header?\n\nThe purpose of signing a tag is to cryptographically verify the thing \npointed at by the tag. But you also want to protect the tag itself. In \norder to make it harder for someone to rename a signed tag (thereby opening \nthe door to replacing it with a different - possibly signed - malicious \ntag), you want to include the tag name in the signed data. This allows us \nto verify that the tag ref (as stored in '.git/refs') is identical to the\ntag name stored inside the signed object.\n\n\nYes, 'notes' will optionally have a \"tag\" header. When I originally designed \nnotes, I didn't think anybody would want to name their notes, but Linus \nrequested it, and there's no technical argument against it. Note that if \nyou name your note, and put a ref to it (under '.git/refs'), there's \ntechnically no distinction between a tag object and a note object, except \nwhat you choose to put in the \"keywords\" header, of course.\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"43953","messageId":"200706040951.06620.johan@herland.net","threadId":"8421","inReplyTo":"20070604055655.GB15148@moooo.ath.cx","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-04T07:51:06Z","receivedAt":"2007-06-04T07:51:06Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Teach git-fsck to do the same kind of verification on tag objects that is\nalready done by git-mktag.\n\nSigned-off-by: Johan Herland \n---\n\nMatthias Lederhofer wrote:\n> The objerror() function prints the sha1 and object type, I think this\n> one should be used instead of error() here.\n\nOf course, you're right. Like this, I hope:\n\n builtin-fsck.c | 14 ++++++++++++++\n 1 files changed, 14 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex cbbcaf0..71a5fd5 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -344,6 +344,20 @@ static int fsck_commit(struct commit *commit)\n static int fsck_tag(struct tag *tag)\n {\n \tstruct object *tagged = tag->tagged;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tchar *data = (char *) read_sha1_file(tag->object.sha1, &type, &size);\n+\tif (!data)\n+\t\treturn objerror(&tag->object, \"could not read tag\");\n+\tif (type != OBJ_TAG) {\n+\t\tfree(data);\n+\t\treturn objerror(&tag->object, \"not a tag (internal error)\");\n+\t}\n+\tif (parse_and_verify_tag_buffer(0, data, size, 1)) { /* thoroughly verify tag object */\n+\t\tfree(data);\n+\t\treturn objerror(&tag->object, \"failed thorough tag object verification\");\n+\t}\n+\tfree(data);\n \n \tif (!tagged) {\n \t\treturn objerror(&tag->object, \"could not load tagged object\");\n-- \n1.5.2\n"},{"id":"44000","messageId":"7vzm3fihpu.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"Re: [PATCH 0/6] Refactor the tag object","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-04T20:32:13Z","receivedAt":"2007-06-04T20:32:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan, I gave only a cursory look at the series so far (that is,\nI looked at your log message and had a very quick pass over the\ncode to see what the basic idea is and if it is sound, without\nreally reading/reviewing the code). It all looks good.\n"},{"id":"44129","messageId":"7vtztl7dqi.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"200706040951.06620.johan@herland.net","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-06T07:18:13Z","receivedAt":"2007-06-06T07:18:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> Teach git-fsck to do the same kind of verification on tag objects that is\n> already done by git-mktag.\n\nThe tagger field was introduced mid July 2005; any repository\nwith a tag object older than that would now get non-zero exit\nfrom fsck.\n\nThis won't practically be problem in newer repositories, but it\nis somewhat annoying. Perhaps do this only under the new -v\noption to git-fsck, say \"warning\" not \"error\", and not exit with\nnon-zero because of this?\n"},{"id":"44134","messageId":"200706061006.33139.johan@herland.net","threadId":"8421","inReplyTo":"7vtztl7dqi.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-06T08:06:32Z","receivedAt":"2007-06-06T08:06:32Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Teach git-fsck to do the same kind of verification on tag objects that is\nalready done by git-mktag.\n\nSigned-off-by: Johan Herland \n---\n\nOn Wednesday 06 June 2007, Junio C Hamano wrote:\n> The tagger field was introduced mid July 2005; any repository\n> with a tag object older than that would now get non-zero exit\n> from fsck.\n> \n> This won't practically be problem in newer repositories, but it\n> is somewhat annoying. Perhaps do this only under the new -v\n> option to git-fsck, say \"warning\" not \"error\", and not exit with\n> non-zero because of this?\n\nLike this?\n\nOr would you rather switch around the \"verbose\" and the\n\"parse_and_verify_tag_buffer()\" (i.e. not even attempt the thorough\nverification unless in verbose mode)?\n\n\nHave fun!\n\n...Johan\n\n builtin-fsck.c | 13 +++++++++++++\n 1 files changed, 13 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex bacae5d..fb9a8bb 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -359,11 +359,24 @@ static int fsck_commit(struct commit *commit)\n static int fsck_tag(struct tag *tag)\n {\n \tstruct object *tagged = tag->tagged;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tchar *data = (char *) read_sha1_file(tag->object.sha1, &type, &size);\n \n \tif (verbose)\n \t\tfprintf(stderr, \"Checking tag %s\\n\",\n \t\t\tsha1_to_hex(tag->object.sha1));\n \n+\tif (!data)\n+\t\treturn objerror(&tag->object, \"could not read tag\");\n+\tif (type != OBJ_TAG) {\n+\t\tfree(data);\n+\t\treturn objerror(&tag->object, \"not a tag (internal error)\");\n+\t}\n+\tif (parse_and_verify_tag_buffer(0, data, size, 1) && verbose)\n+\t\tobjwarning(&tag->object, \"failed thorough tag object verification\");\n+\tfree(data);\n+\n \tif (!tagged) {\n \t\treturn objerror(&tag->object, \"could not load tagged object\");\n \t}\n-- \n1.5.2\n\n\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44147","messageId":"7vejkp5ua1.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"200706061006.33139.johan@herland.net","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-06T09:03:50Z","receivedAt":"2007-06-06T09:03:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n>> This won't practically be problem in newer repositories, but it\n>> is somewhat annoying. Perhaps do this only under the new -v\n>> option to git-fsck, say \"warning\" not \"error\", and not exit with\n>> non-zero because of this?\n>\n> Like this?\n>\n> Or would you rather switch around the \"verbose\" and the\n> \"parse_and_verify_tag_buffer()\" (i.e. not even attempt the thorough\n> verification unless in verbose mode)?\n\nActually I was thinking about doing something like this.\n\n-\tif (parse_and_verify_tag_buffer(0, data, size, 1) && verbose)\n+\tif (parse_and_verify_tag_buffer(0, data, size, verbose))\n"},{"id":"44151","messageId":"7vwsyh4ewh.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"7vejkp5ua1.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-06T09:21:18Z","receivedAt":"2007-06-06T09:21:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Johan Herland writes:\n> ...\n>> Or would you rather switch around the \"verbose\" and the\n>> \"parse_and_verify_tag_buffer()\" (i.e. not even attempt the thorough\n>> verification unless in verbose mode)?\n>\n> Actually I was thinking about doing something like this.\n>\n> -\tif (parse_and_verify_tag_buffer(0, data, size, 1) && verbose)\n> +\tif (parse_and_verify_tag_buffer(0, data, size, verbose))\n\nWell, after running fsck with --verbose, I take the whole\nsuggestion back. I think it is a good idea to do the \"thorough\"\ntag validation in general, and it should not be buried under the\nverbose output, which is almost useless unless in a very narrow\nspecial case that you are really trying to see which exact\nobject is corrupt.\n\nSo I think your original patch to signal error on thorough tag\nvalidation failure is probably a good approach in general.\nPeople need to know that in git.git fsck would return non-zero\nbecause of v0.99 tag, but the people who get hit/annoyed by this\nought to be minority. It may be the case that a major portion\nof git users currently are the ones who futz with the git.git\nrepository, but there would be a serious problem if it continues\nto be the case ;-)\n"},{"id":"44154","messageId":"200706061226.00689.johan@herland.net","threadId":"8421","inReplyTo":"7vwsyh4ewh.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-06T10:26:00Z","receivedAt":"2007-06-06T10:26:00Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Wednesday 06 June 2007, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Johan Herland writes:\n> > ...\n> >> Or would you rather switch around the \"verbose\" and the\n> >> \"parse_and_verify_tag_buffer()\" (i.e. not even attempt the thorough\n> >> verification unless in verbose mode)?\n> >\n> > Actually I was thinking about doing something like this.\n> >\n> > -\tif (parse_and_verify_tag_buffer(0, data, size, 1) && verbose)\n> > +\tif (parse_and_verify_tag_buffer(0, data, size, verbose))\n> \n> Well, after running fsck with --verbose, I take the whole\n> suggestion back. I think it is a good idea to do the \"thorough\"\n> tag validation in general, and it should not be buried under the\n> verbose output, which is almost useless unless in a very narrow\n> special case that you are really trying to see which exact\n> object is corrupt.\n\nTake your pick among my patches or feel free to roll your own. :)\n\n> So I think your original patch to signal error on thorough tag\n> validation failure is probably a good approach in general.\n> People need to know that in git.git fsck would return non-zero\n> because of v0.99 tag, but the people who get hit/annoyed by this\n> ought to be minority. It may be the case that a major portion\n> of git users currently are the ones who futz with the git.git\n> repository, but there would be a serious problem if it continues\n> to be the case ;-)\n\nI also noticed that a number of the early tags in the kernel repo use the \nancient format, and would thus fail fsck.\n\n\nCould we replace the v0.99 tag (and other ancient tags) with \"correct\" \nversions, and then encourage users who have already cloned to delete their \nv0.99 tag and re-pull? New clones would of course never see the old tag at \nall. This sure as hell sounds similar to inserting foot into mouth before \nshooting oneself in said foot, but it might still be worth considering...\n\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44155","messageId":"7vr6op4bgy.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"200706061226.00689.johan@herland.net","subject":"Re: [PATCH 3/6] git-fsck: Do thorough verification of tag objects.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-06T10:35:25Z","receivedAt":"2007-06-06T10:35:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> I also noticed that a number of the early tags in the kernel repo use the \n> ancient format, and would thus fail fsck.\n>\n> \n> Could we replace the v0.99 tag (and other ancient tags) with \"correct\" \n> versions, and then encourage users who have already cloned to delete their \n> v0.99 tag and re-pull? New clones would of course never see the old tag at \n> all. This sure as hell sounds similar to inserting foot into mouth before \n> shooting oneself in said foot, but it might still be worth considering...\n> \n\nI actually think that is not too bad. In the course of git\ndevelopment, the kernel folks had to do the wholesale repository\nconversion twice (once when the order of hashing and compression\nwas swapped, another when the flat tree was made hierarchical),\nI think. Compared to that, tags are not referred to by other\nentities, so it's much easier to \"convert\" (iow, re-sign).\n"},{"id":"44268","messageId":"200706080013.50644.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.52613.johan@herland.net","subject":"[PATCH] Fix bug in tag parsing when thorough verification was in effect","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-07T22:13:50Z","receivedAt":"2007-06-07T22:13:50Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The code that was enabled by passing a non-zero 'thorough_verify' argument\nto parse_and_verify_tag_buffer() moved the 'tag_line' and 'keywords_line'\npointer variables forward in memory while checking for illegal chars.\nThese pointers were later used when setting the respective members on\nthe parsed tag object.\n\nThe fix refactors the verification loop so as to use offsets to the\n'tag_line' and 'keywords_line' pointers, instead of moving the pointers\ndirectly.\n\nThe patch also includes cleanup of the code associated with moving the\nvarious '*_line' pointers past their initial header field identifier.\nThese operations are now done along with the calculation of their\ncorresponding '*_len' variables.\n\nThe patch also includes minor changes to expected output in associated\ntestcases.\n\nThe bug was discovered by inspection. Currently none of the callers of\nparse_and_verify_tag_buffer() that use thorough_verify != 0, also use\nthe 'tag' and 'keywords' members of the parsed tag object.\n\nSigned-off-by: Johan Herland \n---\n\nThis goes on top of the existing \"Refactor the tag object\" patch series.\n\n\nHave fun!\n\n...Johan\n\n t/t3800-mktag.sh | 8 ++++----\n tag.c | 49 ++++++++++++++++++++++++++-----------------------\n 2 files changed, 30 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t3800-mktag.sh b/t/t3800-mktag.sh\nindex f6e3d10..ac9008a 100755\n--- a/t/t3800-mktag.sh\n+++ b/t/t3800-mktag.sh\n@@ -186,7 +186,7 @@ tagger bar@baz.com\n EOF\n \n cat >expect.pat <expect.pat <expect.pat <expect.pat < 0 if given */\n-\t\t\t(tag_line - type_line) - strlen(\"type \\n\");\n-\ttag_len = keywords_line == tag_line ? 0 : /* 0 if not given, > 0 if given */\n-\t\t\t(keywords_line - tag_line) - strlen(\"tag \\n\");\n-\tkeywords_len = tagger_line == keywords_line ? 0 : /* 0 if not given, > 0 if given */\n-\t\t\t(tagger_line - keywords_line) - strlen(\"keywords \\n\");\n-\ttagger_len = header_end == tagger_line ? 0 : /* 0 if not given, > 0 if given */\n-\t\t\t(header_end - tagger_line) - strlen(\"tagger \\n\");\n+\t/*\n+\t * Advance header field pointers past their initial identifier.\n+\t * Calculate lengths of header fields (0 for fields that are not given).\n+\t */\n+\ttype_line += strlen(\"type \");\n+\ttype_len = tag_line > type_line ? ( tag_line - type_line) - 1 : 0;\n+\ttag_line += strlen(\"tag \");\n+\ttag_len = keywords_line > tag_line ? (keywords_line - tag_line) - 1 : 0;\n+\tkeywords_line += strlen(\"keywords \");\n+\tkeywords_len = tagger_line > keywords_line ? ( tagger_line - keywords_line) - 1 : 0;\n+\ttagger_line += strlen(\"tagger \");\n+\ttagger_len = header_end > tagger_line ? ( header_end - tagger_line) - 1 : 0;\n \n \t/* Get the actual type */\n \tif (type_len >= sizeof(type))\n-\t\treturn error(\"char\" PD_FMT \": type too long\", (type_line + 5) - data);\n-\tmemcpy(type, type_line + 5, type_len);\n+\t\treturn error(\"char\" PD_FMT \": type too long\", (type_line) - data);\n+\tmemcpy(type, type_line, type_len);\n \ttype[type_len] = '\\0';\n \n \tif (thorough_verify) {\n+\t\tunsigned long i;\n+\n \t\t/* Verify that the object matches */\n \t\tif (verify_object(sha1, type))\n \t\t\treturn error(\"char%d: could not verify object %s\", 7, sha1_to_hex(sha1));\n \n \t\t/* Verify the tag name: we don't allow control characters or spaces in it */\n \t\tif (tag_len > 0) { /* tag name was given */\n-\t\t\ttag_line += 4; /* skip past \"tag \" */\n-\t\t\tfor (;;) {\n-\t\t\t\tunsigned char c = *tag_line++;\n+\t\t\tfor (i = 0; i < tag_len; ++i) {\n+\t\t\t\tunsigned char c = tag_line[i];\n \t\t\t\tif (c == '\\n')\n \t\t\t\t\tbreak;\n \t\t\t\tif (c > ' ' && c != 0x7f)\n \t\t\t\t\tcontinue;\n-\t\t\t\treturn error(\"char\" PD_FMT \": could not verify tag name\", tag_line - data);\n+\t\t\t\treturn error(\"char\" PD_FMT \": could not verify tag name\", tag_line + i - data);\n \t\t\t}\n \t\t}\n \n \t\t/* Verify the keywords line: we don't allow control characters or spaces in it, or two subsequent commas */\n \t\tif (keywords_len > 0) { /* keywords line was given */\n-\t\t\tkeywords_line += 9; /* skip past \"keywords \" */\n-\t\t\tfor (;;) {\n-\t\t\t\tunsigned char c = *keywords_line++;\n+\t\t\tfor (i = 0; i < keywords_len; ++i) {\n+\t\t\t\tunsigned char c = keywords_line[i];\n \t\t\t\tif (c == '\\n')\n \t\t\t\t\tbreak;\n-\t\t\t\tif (c == ',' && *keywords_line == ',')\n-\t\t\t\t\treturn error(\"char\" PD_FMT \": found empty keyword\", keywords_line - data);\n+\t\t\t\tif (c == ',' && keywords_line[i + 1] == ',') /* consecutive commas */\n+\t\t\t\t\treturn error(\"char\" PD_FMT \": found empty keyword\", keywords_line + i - data);\n \t\t\t\tif (c > ' ' && c != 0x7f)\n \t\t\t\t\tcontinue;\n-\t\t\t\treturn error(\"char\" PD_FMT \": could not verify keywords\", keywords_line - data);\n+\t\t\t\treturn error(\"char\" PD_FMT \": could not verify keywords\", keywords_line + i - data);\n \t\t\t}\n \t\t}\n \n@@ -211,7 +214,7 @@ int parse_and_verify_tag_buffer(struct tag *item, const char *data, const unsign\n \tif (item) { /* Store parsed information into item */\n \t\tif (tag_len > 0) { /* optional tag name was given */\n \t\t\titem->tag = xmalloc(tag_len + 1);\n-\t\t\tmemcpy(item->tag, tag_line + 4, tag_len);\n+\t\t\tmemcpy(item->tag, tag_line, tag_len);\n \t\t\titem->tag[tag_len] = '\\0';\n \t\t}\n \t\telse { /* optional tag name not given */\n@@ -221,7 +224,7 @@ int parse_and_verify_tag_buffer(struct tag *item, const char *data, const unsign\n \n \t\tif (keywords_len > 0) { /* optional keywords string was given */\n \t\t\titem->keywords = xmalloc(keywords_len + 1);\n-\t\t\tmemcpy(item->keywords, keywords_line + 9, keywords_len);\n+\t\t\tmemcpy(item->keywords, keywords_line, keywords_len);\n \t\t\titem->keywords[keywords_len] = '\\0';\n \t\t}\n \t\telse { /* optional keywords string not given. Set default keywords */\n-- \n1.5.2\n"},{"id":"44465","messageId":"200706092019.13185.johan@herland.net","threadId":"8421","inReplyTo":"200706040251.05286.johan@herland.net","subject":"[PATCH 0/7] Introduce soft references (softrefs)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:19:13Z","receivedAt":"2007-06-09T18:19:13Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"This patch series introduces soft references (softrefs); a mechanism for\ndeclaring reachability between arbitrary (but existing) git objects.\nSoftrefs are meant to provide the mechanism for \"reverse mapping\" that\nwe determined was needed for tag objects (especially 'notes'). The patch\nseries also teaches git-mktag to create softrefs for all tag objects.\n\nSee the Discussion section in the git-softref manual page (patch #4/7) or\nthe comments in the header file (patch #1/7) for more details on the\ndesign of softrefs.\n\nI've added some informal performance data at the bottom of this mail [1].\n\nNote that this patch series is incomplete in that the following things\nhave yet to be implemented:\n\n1. Clone/fetch/push of softrefs\n\n2. Packing of softrefs\n\n3. General integration of softrefs into parts of git where they might be\n useful\n\n4. Find appropriate value for MAX_UNSORTED_ENTRIES\n\n\nThere are also some questions connected to the above list of todos:\n\n1. Just how should softrefs affect reachability? Should softrefs be\n used/followed in _all_ reachability computations? If not, which?\n\n2. How should softrefs propagate. I suggest they are pretty much always\n propagated under clone/fetch/push. (Note that the softrefs merge\n algorithm in softrefs.c removes duplicates and softrefs between\n non-existing objects, so pre-filtering of the softrefs to be\n clones/fetched/pushed may not be necessary)\n\n3. Where can softrefs be used to improve performance by replacing existing\n techniques?\n\n4. How to best pack softrefs? Keeping them in the same pack as the objects\n they refer to seems to be a good idea, but more thought needs to be put\n into this before we can make an implementation\n\n5. How to find _all_ (even unreachable) tag objects in repo for\n 'git-softref --rebuild-tags'?\n\n6. Optimization. Pretty much nothing has been done so far. Performance\n seems to be acceptable for now. Probably needs more testing to\n determine bottlenecks\n\n\nNOTE: After the 7 patches, I will send an _optional_ patch\nthat changes the softrefs entries from text format (82 bytes per entry)\nto binary format (40 bytes per entry). The patch is optional, because\nI want the list to decide if we want the (marginal) speedup and\nsimplified code provided by the patch, or if we want to keep the\nread-/maintainability of the text format. Currently I'm in favour of\nkeeping the text format, but I'm far from sure.\n\n\nFinally, here's the shortlog: (This patch series of course goes on top of\nthe previous \"Refactor the tag object\" patch series, although there isn't\nreally that many dependencies between them):\n\nJohan Herland (7):\n Softrefs: Add softrefs header file with API documentation\n Softrefs: Add implementation of softrefs API\n Softrefs: Add git-softref, a builtin command for adding, listing and administering softrefs\n Softrefs: Add manual page documenting git-softref and softrefs subsystem in general\n Softrefs: Add testcases for basic softrefs behaviour\n Softrefs: Administrivia associated with softrefs subsystem and git-softref builtin\n Teach git-mktag to register softrefs for all tag objects\n\n .gitignore | 1 +\n Documentation/cmd-list.perl | 7 +-\n Documentation/git-softref.txt | 119 +++++++\n Makefile | 6 +-\n builtin-softref.c | 167 ++++++++++\n builtin.h | 1 +\n git.c | 1 +\n mktag.c | 11 +-\n softrefs.c | 712 +++++++++++++++++++++++++++++++++++++++++\n softrefs.h | 188 +++++++++++\n t/t3050-softrefs.sh | 314 ++++++++++++++++++\n 11 files changed, 1521 insertions(+), 6 deletions(-)\n create mode 100644 Documentation/git-softref.txt\n create mode 100644 builtin-softref.c\n create mode 100644 softrefs.c\n create mode 100644 softrefs.h\n create mode 100755 t/t3050-softrefs.sh\n\n\nHave fun!\n\n...Johan\n\n\n[1] Informal performance measurements\n\nI prepared a linux kernel repo (holding 57274 commits) with 10 tag objects,\nand created softrefs from every commit to every tag object (572740 softrefs\nin total). The resulting softrefs db was 46964680 bytes. The experiment was\ndone on a 32-bit Intel Pentium 4 (3 GHz w/HyperThreading) with 1 GB RAM:\n\n\n========\nOperations on unsorted softrefs:\n(572740 (10 per commit) entries in random/unsorted order)\n========\n\nListing all softrefs\n(sequential reading of unsorted softrefs file)\n--------\n$ /usr/bin/time git softref --list > /dev/null\n0.44user 0.02system 0:00.47elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+11786minor)pagefaults 0swaps\n\nListing HEAD's softrefs\n(sequential reading of unsorted softrefs file)\n--------\n$ /usr/bin/time git softref --list HEAD > /dev/null\n0.11user 0.01system 0:00.14elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+11790minor)pagefaults 0swaps\n\nSorting softrefs\n--------\n$ /usr/bin/time git softref --merge-unsorted\n2.73user 4.97system 0:07.77elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+15833minor)pagefaults 0swaps\n\nSorting softrefs into existing sorted file\n(throwing away duplicates)\n--------\n$ /usr/bin/time git softref --merge-unsorted\n3.49user 5.12system 0:08.64elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+27300minor)pagefaults 0swaps\n\n\n========\nOperations on sorted softrefs:\n(572740 (10 per commit) entries in sorted order)\n========\n\nListing all softrefs\n(sequential reading of sorted softrefs file)\n--------\n$ /usr/bin/time git softref --list > /dev/null\n0.43user 0.02system 0:00.48elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+11786minor)pagefaults 0swaps\n\nListing HEAD's softrefs\n(256-fanout followed by binary search in sorted softrefs file)\n--------\n$/usr/bin/time git softref --list HEAD > /dev/null\n0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+334minor)pagefaults 0swaps\n\nSorting softrefs\n(no-op)\n--------\n$ /usr/bin/time git softref --merge-unsorted\n0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+312minor)pagefaults 0swaps\n\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44466","messageId":"200706092021.43314.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 1/7] Softrefs: Add softrefs header file with API documentation","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:21:43Z","receivedAt":"2007-06-09T18:21:43Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"See patch for documentation.\n\nSigned-off-by: Johan Herland \n---\n softrefs.h | 188 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 188 insertions(+), 0 deletions(-)\n create mode 100644 softrefs.h\n\ndiff --git a/softrefs.h b/softrefs.h\nnew file mode 100644\nindex 0000000..db0f8b9\n--- /dev/null\n+++ b/softrefs.h\n@@ -0,0 +1,188 @@\n+#ifndef SOFTREFS_H\n+#define SOFTREFS_H\n+\n+/*\n+ * Softrefs is a general mechanism for declaring a relationship between two\n+ * existing arbitrary objects in the repo. Softrefs differ from the existing\n+ * reachability relationship in that a softref may be created after _both_ of\n+ * the involved objects have been added to the repo. In contrast, the regular\n+ * reachability relationship depends on the reachable object's name being\n+ * stored _inside_ the other object. A reachability relationship can therefore\n+ * not be created at a later time without violating the immutability of git\n+ * objects.\n+ *\n+ * Softrefs are defined as going _from_ one object _to_ another object. Once\n+ * a softref between two objects has been created, the \"to\" object is\n+ * considered reachable from the \"from\" object.\n+ *\n+ * Also, softrefs are stored in a way that makes it easy and quick to find all\n+ * the \"to\" objects reachable from a given \"from\" object.\n+ *\n+ * The softrefs db consists of two files: .git/softrefs.unsorted and\n+ * .git/softrefs.sorted. Both files use the same format; one softref per line\n+ * of the form \" \\n\". Each sha1 sum is 40 bytes long; this\n+ * makes each entry exactly 82 bytes long (including the space between the sha1\n+ * sums and the terminating linefeed).\n+ *\n+ * The entries in .git/softrefs.sorted are sorted on , in order to\n+ * make lookup fast.\n+ *\n+ * The entries in .git/softrefs.unsorted are _not_ sorted. This is to make\n+ * insertion fast.\n+ *\n+ * When softrefs are created (by calling add_softref()/add_softrefs()), they\n+ * are appended to .git/softrefs.unsorted. When .git/softrefs.unsorted reach a\n+ * certain number of entries (determined by MAX_UNSORTED_ENTRIES), all the\n+ * entries in .git/softrefs.unsorted are merged into .git/softrefs.sorted.\n+ *\n+ * Soft references are used as a reverse mapping between tag objects and their\n+ * corresponding tagged objects. For each tag object, a soft reference _to_\n+ * the tag object _from_ the tagged object is created. Given an arbitrary\n+ * object X in the database, softrefs allow for easy lookup of which tag\n+ * objects that point to object X.\n+ */\n+\n+/*\n+ * Simple list of softrefs\n+ */\n+struct softref_list {\n+\tstruct softref_list *next;\n+\tunsigned char from_sha1[20];\n+\tunsigned char to_sha1[20];\n+};\n+\n+/* Callback function type; used as parameter to for_each_softref()\n+ *\n+ * The functions takes the following arguments:\n+ * - from_sha1 - The SHA1 of the 'from' object in the current softref\n+ * - to_sha1 - The SHA1 of the 'to' object in the current softref\n+ * - cb_data - as passed to for_each_softref()\n+ *\n+ * Return non-zero to stop for_each_softref() from iterating through.\n+ */\n+typedef int each_softref_fn(\n+\tconst unsigned char *from_sha1,\n+\tconst unsigned char *to_sha1,\n+\tvoid *cb_data);\n+\n+/*\n+ * Invoke 'fn' with 'cb_data' for each object pointed to by 'from_sha1'\n+ *\n+ * If 'from_sha1' is NULL, 'fn' is invoked for _all_ softrefs in the db.\n+ *\n+ * If 'fn' returns non-zero for any given softref, iteration is stopped and the\n+ * same return value is returned from this function. If other problems are\n+ * encountered while iterating, -1 is returned. If all matching entries were\n+ * iterated successfully, and 'fn' returned 0 for all of them, 0 is returned.\n+ */\n+extern int for_each_softref_with_from(\n+\tconst unsigned char *from_sha1, each_softref_fn fn, void *cb_data);\n+\n+/*\n+ * Invoke 'fn' with 'cb_data' for each softref stored in the db\n+ *\n+ * This function is identical to calling for_each_softref_with_from() with\n+ * NULL as the first parameter.\n+ */\n+extern int for_each_softref(each_softref_fn fn, void *cb_data);\n+\n+/*\n+ * Initialize/prepare the softrefs db for a lot of read-only access\n+ *\n+ * You may call this function before doing repeated calls to accessor functions\n+ * such as:\n+ * - for_each_softref_with_from()\n+ * - for_each_softref()\n+ * - lookup_softref()\n+ * - has_softref()\n+ *\n+ * This function is purely optional, although it may improve performance when\n+ * accessor functions are called repeatedly. The change in performance is\n+ * caused by:\n+ * 1. Merging unsorted softref entries into the sorted db file,\n+ * 2. Doing open() and mmap() on the sorted db file (in order to avoid doing\n+ * this on each subsequent call to an accessor function).\n+ *\n+ * When done accessing the softrefs db, the caller _must_ call\n+ * deinit_softrefs_access() to properly deinitialize internal structures.\n+ */\n+extern void init_softrefs_access();\n+\n+/*\n+ * Deinitialize internal structures associated with init_softrefs_access()\n+ *\n+ * Call this function when finished accessing softrefs after a call to\n+ * init_softrefs_access().\n+ */\n+extern void deinit_softrefs_access();\n+\n+/*\n+ * Look up the given object id in the softrefs db\n+ *\n+ * Returns a list of all the matching softrefs, i.e. softrefs whose from_sha1\n+ * is identical to the given. If the given from_sha1 is NULL, all softrefs are\n+ * returned.\n+ *\n+ * The entired softref_list returned (i.e. all elements retrievable by\n+ * following the next pointer) must be free()d by the caller.\n+ *\n+ * You should consider using one of the for_each_softref*() functions instead,\n+ * as those might save you some memory.\n+ */\n+extern struct softref_list *lookup_softref(const unsigned char *from_sha1);\n+\n+/*\n+ * Delete (i.e. free()) all elements in the given softref_list\n+ */\n+extern void delete_softref_list(struct softref_list *list);\n+\n+/*\n+ * Return 1 if there exists a softref between 'from_sha1' and 'to_sha1'\n+ *\n+ * Otherwise, return 0.\n+ */\n+extern int has_softref(\n+\tconst unsigned char *from_sha1, const unsigned char *to_sha1);\n+\n+/*\n+ * Add all the softrefs given in the given 'list' to the db.\n+ *\n+ * Returns the number of softrefs added, or -1 on failure to add any softrefs.\n+ */\n+extern int add_softrefs(const struct softref_list *list);\n+\n+/*\n+ * Add a softref between 'from_sha1' and 'to_sha1'\n+ *\n+ * 'from_sha1' and 'to_sha1' are two 20-byte object ids.\n+ * Returns 0 on success, 1 if the softref already exists, -1 on failure.\n+ */\n+extern int add_softref(\n+\tconst unsigned char *from_sha1, const unsigned char *to_sha1);\n+\n+/*\n+ * Merge softrefs found in the given unsorted softrefs file into the sorted db\n+ *\n+ * If 'unsorted_file' is NULL, the internal unsorted db file is merged.\n+ *\n+ * Note that this routine is automatically invoked by add_softrefs() and\n+ * add_softref() to control the size of the unsorted db file.\n+ *\n+ * If 'unsorted_file' is NULL, the merging is only done if the number of\n+ * softrefs in the unsorted db file exceed a fixed threshold (see\n+ * MAX_UNSORTED_ENTRIES). However, if 'force' is set, the merging will be done\n+ * regardless. Passing anything other than NULL for 'unsorted_file'\n+ * automatically turns on 'force'.\n+ *\n+ * Returns 0 on success; non-zero if problems were encountered.\n+ */\n+extern int merge_unsorted_softrefs(const char *unsorted_file, int force);\n+\n+/*\n+ * Merge softrefs found in the given sorted softrefs file into the sorted db\n+ *\n+ * Returns 0 on success; non-zero if problems were encountered.\n+ */\n+extern int merge_sorted_softrefs(const char *sorted_file);\n+\n+#endif /* SOFTREFS_H */\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44467","messageId":"200706092022.21234.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 2/7] Softrefs: Add implementation of softrefs API","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:22:21Z","receivedAt":"2007-06-09T18:22:21Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"This code tries to implement the softrefs API as straightforwardly as\npossible. Virtually no optimization has been done, although I do have\na feeling the code has ok performance as is.\n\nAll functions that do not appear in the API docs have some comments\nattached to them.\n\nThere are also a couple of things to be considered before inclusion:\n- File locking. Currently no locking is performed on softrefs files\n before reading or writing entries.\n- Packing. We need a plan for how softrefs should be included in packs,\n at which supporting code must be added to this implementation.\n\nSigned-off-by: Johan Herland \n---\n softrefs.c | 712 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 712 insertions(+), 0 deletions(-)\n create mode 100644 softrefs.c\n\ndiff --git a/softrefs.c b/softrefs.c\nnew file mode 100644\nindex 0000000..c7308c8\n--- /dev/null\n+++ b/softrefs.c\n@@ -0,0 +1,712 @@\n+#include \"cache.h\"\n+#include \"softrefs.h\"\n+\n+/* constants */\n+static const char * UNSORTED_FILENAME = \"softrefs.unsorted\";\n+static const char * SORTED_FILENAME = \"softrefs.sorted\";\n+static const unsigned int MAX_UNSORTED_ENTRIES = 1000;\n+\n+\n+/* softref entry as it appears in a softrefs file */\n+struct softrefs_entry {\n+\tchar from_sha1_hex[40];\n+\tchar space;\n+\tchar to_sha1_hex[40];\n+\tchar lf;\n+};\n+\n+/* simple encapsulation of a softrefs file */\n+struct softrefs_file {\n+\tchar *filename;\n+\tint fd;\n+\tstruct softrefs_entry *data; /* mmap()ed softrefs_entry objects */\n+\tunsigned long data_len; /* # of softrefs_entry objects in data */\n+};\n+\n+/* Internal file opened/closed by (de)init_softrefs_access() */\n+static struct softrefs_file *internal_file = 0;\n+\n+/*\n+ * Open and mmap() the given filename, Assign the file descriptior, data\n+ * pointer and data length to the given softrefs_file object.\n+ * Return 0 on success, -1 on failure.\n+ *\n+ * Note that a non-existing file is not a failure per se, but is rather treated\n+ * as an empty file, i.e. there will be no data in the file structure\n+ * (data_len == 0), but 0/sucess will be returned.\n+ *\n+ * The caller must _always_ call close_softrefs_file() with the same\n+ * softrefs_file argument after processing the file data, even if no file\n+ * is actually opened and/or this function returns -1.\n+ */\n+static int open_softrefs_file(const char *filename, struct softrefs_file *file)\n+{\n+\tstruct stat st;\n+\n+\t/* Default \"failure\" values */\n+\tfile->filename = xstrdup(filename);\n+\tfile->fd = -1;\n+\tfile->data = MAP_FAILED;\n+\tfile->data_len = 0;\n+\n+\t/* FIXME: File locking!? */\n+\tif (access(file->filename, F_OK))\n+\t\treturn 0;\n+\tfile->fd = open(file->filename, O_RDONLY);\n+\tif (file->fd < 0)\n+\t\treturn error(\"Failed to open softrefs file %s: %s\",\n+\t\t\t\tfile->filename, strerror(errno));\n+\tif (fstat(file->fd, &st))\n+\t\treturn error(\"Failed to fstat softrefs file %s: %s\",\n+\t\t\t\tfile->filename, strerror(errno));\n+\tif (st.st_size == 0) /* Empty file. No need to call mmap() */\n+\t\treturn 0;\n+\tif (st.st_size % sizeof(struct softrefs_entry))\n+\t\treturn error(\"Refuse to mmap softrefs file %s: File does not have whole number of softref entries\",\n+\t\t\t\tfile->filename);\n+\n+\tfile->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, file->fd, 0);\n+\tif (file->data == MAP_FAILED)\n+\t\treturn error(\"Failed to mmap softrefs file %s: %s\",\n+\t\t\t\tfile->filename, strerror(errno));\n+\n+\tfile->data_len = st.st_size / sizeof(struct softrefs_entry);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Close the softrefs file identified by the given softrefs_file object.\n+ * Return 0 on success, non-zero on failure.\n+ */\n+static int close_softrefs_file(const struct softrefs_file *file)\n+{\n+\tint ret = 0;\n+\tif (file->data != MAP_FAILED &&\n+\t munmap(file->data, file->data_len * sizeof(struct softrefs_entry)))\n+\t{\n+\t\tret = error(\"Failed to munmap softrefs file %s: %s\",\n+\t\t\t\tfile->filename, strerror(errno));\n+\t}\n+\tif (file->fd != -1 && close(file->fd))\n+\t\tret = error(\"Failed to close softrefs file %s: %s\",\n+\t\t\t\tfile->filename, strerror(errno));\n+\tfree(file->filename);\n+\treturn ret;\n+}\n+\n+/*\n+ * Write the given softrefs_entry to the given file descriptor, which must be\n+ * open and writable.\n+ *\n+ * Returns 0 on success, non-zero on failure.\n+ */\n+static int write_entry(int fd, const struct softrefs_entry *entry)\n+{\n+\tif (write(fd, (const void *) entry, sizeof(struct softrefs_entry))\n+\t\t< sizeof(struct softrefs_entry))\n+\t{\n+\t\treturn error(\"Failed to write entry '%.40s -> %.40s' to softrefs file descriptor %i: %s\",\n+\t\t\t\tentry->from_sha1_hex, entry->to_sha1_hex,\n+\t\t\t\tfd, strerror(errno));\n+\t}\n+\treturn 0;\n+}\n+\n+/* See softrefs.h for documentation */\n+void init_softrefs_access()\n+{\n+\tif (internal_file) /* already initialized */\n+\t\treturn;\n+\n+\t/* Force merge into sorted, so that we only have one file to search */\n+\tif (merge_unsorted_softrefs(NULL, 1))\n+\t\treturn; /* merge failed */\n+\n+\tinternal_file = xmalloc(sizeof(struct softrefs_file));\n+\tif (open_softrefs_file(git_path(SORTED_FILENAME), internal_file)) {\n+\t\tfree(internal_file);\n+\t\tinternal_file = 0;\n+\t}\n+}\n+\n+/* See softrefs.h for documentation */\n+void deinit_softrefs_access()\n+{\n+\tif (!internal_file) /* already deinitialized */\n+\t\treturn;\n+\tclose_softrefs_file(internal_file);\n+\tinternal_file = 0;\n+}\n+\n+/* comparison between a SHA1 sum and a softrefs entry */\n+static int sha1_to_entry_cmp(\n+\tconst unsigned char *from_sha1, const struct softrefs_entry *entry)\n+{\n+\tunsigned char sha1[20];\n+\tget_sha1_hex(entry->from_sha1_hex, sha1);\n+\treturn hashcmp(from_sha1, sha1);\n+}\n+\n+/* comparison between softrefs entries */\n+static int softrefs_entry_cmp(\n+\t\tconst struct softrefs_entry *a, const struct softrefs_entry *b)\n+{\n+\tunsigned char sa[20], sb[20];\n+\tint ret;\n+\tget_sha1_hex(a->from_sha1_hex, sa);\n+\tget_sha1_hex(b->from_sha1_hex, sb);\n+\tret = hashcmp(sa, sb);\n+\tif (!ret) {\n+\t\tget_sha1_hex(a->to_sha1_hex, sa);\n+\t\tget_sha1_hex(b->to_sha1_hex, sb);\n+\t\tret = hashcmp(sa, sb);\n+\t}\n+\treturn ret;\n+}\n+\n+/* comparison between softrefs entries as invoked by qsort() */\n+static int softrefs_entry_qsort_cmp(const void *a, const void *b)\n+{\n+\tconst struct softrefs_entry *na = *((const struct softrefs_entry **) a);\n+\tconst struct softrefs_entry *nb = *((const struct softrefs_entry **) b);\n+\treturn softrefs_entry_cmp(na, nb);\n+}\n+\n+\n+/*\n+ * Sequentially process given 'file' starting at index 'i'\n+ *\n+ * For each entry matching 'from_sha1' (if NULL, match all entries), invoke\n+ * callback function 'fn' with the from_sha1 and to_sha1 of the matching\n+ * softref. Keep going until 'fn' returns non-zero, or end of file is reached.\n+ *\n+ * If the 'stop_at_first_non_match' flag is set, processing will stop when the\n+ * first non-matching entry is encountered.\n+ *\n+ * Returns result of 'fn' if non-zero; otherwise 0 on success and -1 on failure.\n+ */\n+static int do_for_each_sequential(\n+\t\tconst unsigned char *from_sha1,\n+\t\teach_softref_fn fn, void *cb_data,\n+\t\tstruct softrefs_file *file,\n+\t\tunsigned long i,\n+\t\tint stop_at_first_non_match)\n+{\n+\tunsigned char f_sha1[20], t_sha1[20]; /* Holds sha1 per entry */\n+\tint ret = 0;\n+\tfor (; i < file->data_len; ++i) { /* Step through file, starting at i */\n+\t\t/* sanity check entry */\n+\t\tif (file->data[i].space != ' ' || file->data[i].lf != '\\n') {\n+\t\t\tret = error(\"Entry #%lu in softrefs file %s failed sanity check\",\n+\t\t\t\t\ti, file->filename);\n+\t\t\tbreak;\n+\t\t}\n+\t\t/* retrieve SHA1 values */\n+\t\tif (get_sha1_hex(file->data[i].from_sha1_hex, f_sha1) ||\n+\t\t get_sha1_hex(file->data[i].to_sha1_hex, t_sha1)) {\n+\t\t\tret = error(\"Failed to read SHA1 values from entry #%lu in softrefs file %s\",\n+\t\t\t\t\ti, file->filename);\n+\t\t\tbreak;\n+\t\t}\n+\t\t/* Compare to lookup value */\n+\t\tif (!from_sha1 || !hashcmp(from_sha1, f_sha1)) {\n+\t\t\tif ((ret = fn(f_sha1, t_sha1, cb_data)))\n+\t\t\t\tbreak; /* bail out if callback returns != 0 */\n+\t\t}\n+\t\telse if (stop_at_first_non_match)\n+\t\t\tbreak;\n+\t}\n+\treturn ret;\n+}\n+\n+/* Invoke callback 'fn' for each matching entry in UNSORTED_FILENAME */\n+static int do_for_each_unsorted(\n+\t\tconst unsigned char *from_sha1,\n+\t\teach_softref_fn fn, void *cb_data)\n+{\n+\tstruct softrefs_file file;\n+\tint ret = 0;\n+\n+\tif (internal_file)\n+\t\t/*\n+\t\t * internal_file is open. Unsorted entries are merged just\n+\t\t * before opening internal_file (in init_softrefs_access()).\n+\t\t * Since internal_file is still open, no entries have been\n+\t\t * added since last merge, meaning that there can be no\n+\t\t * unsorted entries in the db, and thus no unsorted file.\n+\t\t * Therefore return immediate success.\n+\t\t */\n+\t\treturn 0;\n+\n+\tif (!(ret = open_softrefs_file(git_path(UNSORTED_FILENAME), &file)))\n+\t\tret = do_for_each_sequential(from_sha1, fn, cb_data, &file, 0, 0);\n+\n+\tclose_softrefs_file(&file);\n+\treturn ret;\n+}\n+\n+/* Invoke callback 'fn' for each matching entry in SORTED_FILENAME */\n+static int do_for_each_sorted(\n+\t\tconst unsigned char *from_sha1,\n+\t\teach_softref_fn fn, void *cb_data)\n+{\n+\tstruct softrefs_file *file;\n+\tunsigned long i, left, right;\n+\tint cmp_result;\n+\tint ret = 0;\n+\n+\tif (internal_file) /* use already open internal_file */\n+\t\tfile = internal_file;\n+\telse { /* open file ourselves */\n+\t\tfile = xmalloc(sizeof(struct softrefs_file));\n+\t\tif ((ret = open_softrefs_file(git_path(SORTED_FILENAME), file)))\n+\t\t\tgoto done;\n+\t}\n+\tif (!file->data_len) /* no entries */\n+\t\tgoto done;\n+\n+\tif (!from_sha1) { /* match _all_ entries; do sequential walk instead */\n+\t\tret = do_for_each_sequential(from_sha1, fn, cb_data, file, 0, 0);\n+\t\tgoto done;\n+\t}\n+\n+\t/* Calculate first index by 256-fanout */\n+\tleft = 0;\n+\tright = file->data_len;\n+\ti = (from_sha1[0] * file->data_len) / 256;\n+\n+\t/* Binary search */\n+\twhile ((cmp_result = sha1_to_entry_cmp(from_sha1, &(file->data[i])))) {\n+\t\tif (right - left <= 1) /* not found; give up */\n+\t\t\tgoto done;\n+\t\tif (cmp_result > 0) /* go right */\n+\t\t\tleft = i + 1;\n+\t\telse /* go left */\n+\t\t\tright = i;\n+\t\ti = (left + right) / 2;\n+\t}\n+\n+\t/* i points to a matching entry, but not necessarily the first */\n+\twhile (i >= 1 && sha1_to_entry_cmp(from_sha1, &(file->data[i - 1])) == 0)\n+\t\t--i;\n+\n+\t/* i points to the first matching entry */\n+\t/* do sequential processing from i, stopping at first non-match */\n+\tret = do_for_each_sequential(from_sha1, fn, cb_data, file, i, 1);\n+\n+done:\n+\tif (!internal_file) { /* only close if we opened ourselves */\n+\t\tclose_softrefs_file(file);\n+\t\tfree(file);\n+\t}\n+\treturn ret;\n+}\n+\n+/* See softrefs.h for documentation */\n+int for_each_softref_with_from(\n+\t\tconst unsigned char *from_sha1,\n+\t\teach_softref_fn fn, void *cb_data)\n+{\n+\tint ret = do_for_each_unsorted(from_sha1, fn, cb_data);\n+\tif (ret)\n+\t\treturn ret;\n+\tret = do_for_each_sorted(from_sha1, fn, cb_data);\n+\treturn ret;\n+}\n+\n+/* See softrefs.h for documentation */\n+int for_each_softref(each_softref_fn fn, void *cb_data)\n+{\n+\treturn for_each_softref_with_from(0, fn, cb_data);\n+}\n+\n+static int lookup_softref_helper(\n+\t\tconst unsigned char *from_sha1, const unsigned char *to_sha1,\n+\t\tvoid *cb_data)\n+{\n+\tstruct softref_list **prev = (struct softref_list **) cb_data;\n+\n+\tstruct softref_list *current = xmalloc(sizeof(struct softref_list));\n+\tcurrent->next = *prev;\n+\thashcpy(current->from_sha1, from_sha1);\n+\thashcpy(current->to_sha1, to_sha1);\n+\t*prev = current;\n+\treturn 0;\n+}\n+\n+/* See softrefs.h for documentation */\n+struct softref_list *lookup_softref(const unsigned char *from_sha1)\n+{\n+\tstruct softref_list *result = NULL;\n+\tstruct softref_list **p = &result;\n+\tif (for_each_softref_with_from(\n+\t\t\tfrom_sha1, lookup_softref_helper, (void *) p))\n+\t{\n+\t\tdelete_softref_list(result);\n+\t\tresult = NULL;\n+\t}\n+\treturn result;\n+}\n+\n+/* See softrefs.h for documentation */\n+void delete_softref_list(struct softref_list *list)\n+{\n+\twhile (list) {\n+\t\tstruct softref_list *next = list->next;\n+\t\tfree(list);\n+\t\tlist = next;\n+\t}\n+}\n+\n+static int has_softref_helper(\n+\t\tconst unsigned char *from_sha1, const unsigned char *to_sha1,\n+\t\tvoid *cb_data)\n+{\n+\tconst unsigned char *needle = (const unsigned char *) cb_data;\n+\tif (!hashcmp(to_sha1, needle))\n+\t\treturn 1; /* found */\n+\treturn 0; /* keep going */\n+}\n+\n+/* See softrefs.h for documentation */\n+int has_softref(const unsigned char *from_sha1, const unsigned char *to_sha1)\n+{\n+\tint ret = for_each_softref_with_from(\n+\t\t\tfrom_sha1, has_softref_helper, (void *) to_sha1);\n+\treturn ret == 1 ? 1 : 0;\n+}\n+\n+\n+/*\n+ * Merge the unsorted softref entries in unsorted_filename into sorted_filename\n+ *\n+ * Returns 0 on success; non-zero on failure.\n+ *\n+ * If sorted_filename does not exist, the entries in unsorted_filename will be\n+ * sorted and stored into sorted_filename.\n+ * If unsorted_filename does not exist, this function will do nothing and\n+ * return 0.\n+ */\n+static int merge_unsorted_into_sorted(\n+\t\tconst char *unsorted_filename, const char *sorted_filename)\n+{\n+\tstruct softrefs_file unsorted, sorted;\n+\tchar *result_filename = 0;\n+\tint result_fd = -1;\n+\tint ret = 0;\n+\tunsigned long i, j;\n+\t/* array of pointers to softrefs_entries in unsorted file */\n+\tstruct softrefs_entry **to_insert;\n+\t/* keep track of last processed entry, to remove duplicates */\n+\tstruct softrefs_entry *prev = NULL;\n+\n+\t/* Open input files */\n+\tdeinit_softrefs_access();\n+\topen_softrefs_file(unsorted_filename, &unsorted);\n+\tif (!unsorted.data_len) { /* no unsorted entries; nothing to do */\n+\t\tclose_softrefs_file(&unsorted);\n+\t\treturn 0;\n+\t}\n+\topen_softrefs_file(sorted_filename, &sorted);\n+\n+\t/* Sort the unsorted entries */\n+\tto_insert = xmalloc(sizeof(struct softrefs_entry *) * unsorted.data_len);\n+\tfor (i = 0; i < unsorted.data_len; ++i)\n+\t\tto_insert[i] = &(unsorted.data[i]);\n+\tqsort(to_insert, unsorted.data_len, sizeof(struct softrefs_entry *),\n+\t\t\tsoftrefs_entry_qsort_cmp);\n+\n+\t/* Create result file */\n+\tresult_filename = xmalloc(strlen(sorted_filename) + 4);\n+\tsprintf(result_filename, \"%s.new\", sorted_filename);\n+\tresult_fd = open(result_filename, O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0666);\n+\tif (result_fd < 0) {\n+\t\tret = error(\"Failed to open merge result file %s: %s\",\n+\t\t\t\tresult_filename, strerror(errno));\n+\t\tgoto done;\n+\t}\n+\n+\ti = 0; /* index into to_insert (the sorted version of unsorted.data) */\n+\tj = 0; /* index into sorted.data */\n+\twhile (!ret && (i < unsorted.data_len || j < sorted.data_len)) {\n+\t\t/* there are still entries in either list */\n+\t\tstruct softrefs_entry *cur;\n+\t\tunsigned char from_sha1[20], to_sha1[20];\n+\t\tif (i < unsorted.data_len && j < sorted.data_len) {\n+\t\t\t/* there are still entries in _both_ lists */\n+\t\t\t/* choose \"lowest\" entry from either list */\n+\t\t\tif (softrefs_entry_cmp(to_insert[i], &(sorted.data[j])) < 0)\n+\t\t\t\tcur = to_insert[i++];\n+\t\t\telse\n+\t\t\t\tcur = &(sorted.data[j++]);\n+\t\t}\n+\t\telse if (i < unsorted.data_len) /* entries left in to_insert */\n+\t\t\tcur = to_insert[i++];\n+\t\telse /* entries left in sorted.data */\n+\t\t\tcur = &(sorted.data[j++]);\n+\n+\t\tif (prev && !softrefs_entry_cmp(prev, cur))\n+\t\t\tcontinue; /* skip writing if prev == cur */\n+\t\tprev = cur;\n+\n+\t\t/* skip writing if softref involves a non-existing object */\n+\t\tif (get_sha1_hex(cur->from_sha1_hex, from_sha1) ||\n+\t\t\t!has_sha1_file(from_sha1) ||\n+\t\t get_sha1_hex(cur->to_sha1_hex, to_sha1) ||\n+\t\t\t!has_sha1_file( to_sha1))\n+\t\t{\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tret = write_entry(result_fd, cur);\n+\t}\n+\n+done:\n+\tif (result_fd >= 0 && close(result_fd))\n+\t\tret = error(\"Failed to close merge result file %s: %s\",\n+\t\t\t\tresult_filename, strerror(errno));\n+\tclose_softrefs_file(&sorted);\n+\tclose_softrefs_file(&unsorted);\n+\tif (ret) { /* Failure. Delete result_filename */\n+\t\tif (result_filename && unlink(result_filename))\n+\t\t\terror(\"Failed to remove merge result file %s: %s\",\n+\t\t\t\t\tresult_filename, strerror(errno));\n+\t}\n+\telse { /* Success. Replace sorted_filename with result_filename */\n+\t\tif (rename(result_filename, sorted_filename))\n+\t\t\tret = error(\"Failed to replace sorted softrefs file %s: %s\",\n+\t\t\t\t\tsorted_filename, strerror(errno));\n+\t}\n+\treturn ret;\n+}\n+\n+/*\n+ * Merge the sorted softref entries in 'from_file' into 'to_file'\n+ *\n+ * Returns 0 on success; non-zero on failure.\n+ *\n+ * If to_file does not exist, from_file will be copied into to_file.\n+ * If from_file does not exist, this function will do nothing and return 0.\n+ */\n+static int merge_sorted_into_sorted(const char *from_file, const char *to_file)\n+{\n+\tstruct softrefs_file file1, file2;\n+\tchar *result_filename = 0;\n+\tint result_fd = -1;\n+\tint ret = 0;\n+\tunsigned long i, j;\n+\t/* keep track of last processed entry, to remove duplicates */\n+\tstruct softrefs_entry *prev = NULL;\n+\n+\t/* Open input files */\n+\tdeinit_softrefs_access();\n+\topen_softrefs_file(from_file, &file1);\n+\tif (!file1.data_len) { /* no entries; nothing to do */\n+\t\tclose_softrefs_file(&file1);\n+\t\treturn 0;\n+\t}\n+\topen_softrefs_file(to_file, &file2);\n+\n+\t/* Create result file */\n+\tresult_filename = xmalloc(strlen(to_file) + 4);\n+\tsprintf(result_filename, \"%s.new\", to_file);\n+\tresult_fd = open(result_filename, O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0666);\n+\tif (result_fd < 0) {\n+\t\tret = error(\"Failed to open merge result file %s: %s\",\n+\t\t\t\tresult_filename, strerror(errno));\n+\t\tgoto done;\n+\t}\n+\n+\ti = 0; /* index into file1.data */\n+\tj = 0; /* index into file2.data */\n+\twhile (!ret && (i < file1.data_len || j < file2.data_len)) {\n+\t\t/* there are still entries in either list */\n+\t\tstruct softrefs_entry *cur;\n+\t\tunsigned char from_sha1[20], to_sha1[20];\n+\t\tif (i < file1.data_len && j < file2.data_len) {\n+\t\t\t/* there are still entries in _both_ lists */\n+\t\t\t/* choose \"lowest\" entry from either list */\n+\t\t\tif (softrefs_entry_cmp(&(file1.data[i]), &(file2.data[j])) < 0)\n+\t\t\t\tcur = &(file1.data[i++]);\n+\t\t\telse\n+\t\t\t\tcur = &(file2.data[j++]);\n+\t\t}\n+\t\telse if (i < file1.data_len) /* entries left in file1.data */\n+\t\t\tcur = &(file1.data[i++]);\n+\t\telse /* entries left in file2.data */\n+\t\t\tcur = &(file2.data[j++]);\n+\n+\t\tif (prev && !softrefs_entry_cmp(prev, cur))\n+\t\t\tcontinue; /* skip writing if cur and prev are duplicates */\n+\t\tprev = cur;\n+\n+\t\t/* skip writing if softref involves a non-existing object */\n+\t\tif (get_sha1_hex(cur->from_sha1_hex, from_sha1) ||\n+\t\t\t!has_sha1_file(from_sha1) ||\n+\t\t get_sha1_hex(cur->to_sha1_hex, to_sha1) ||\n+\t\t\t!has_sha1_file( to_sha1))\n+\t\t{\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tret = write_entry(result_fd, cur);\n+\t}\n+\n+done:\n+\tif (result_fd >= 0 && close(result_fd))\n+\t\tret = error(\"Failed to close merge result file %s: %s\",\n+\t\t\t\tresult_filename, strerror(errno));\n+\tclose_softrefs_file(&file2);\n+\tclose_softrefs_file(&file1);\n+\tif (ret) { /* Failure. Delete result_filename */\n+\t\tif (result_filename && unlink(result_filename))\n+\t\t\terror(\"Failed to remove merge result file %s: %s\",\n+\t\t\t\t\tresult_filename, strerror(errno));\n+\t}\n+\telse { /* Success. Replace to_file with result_filename */\n+\t\tif (rename(result_filename, to_file))\n+\t\t\tret = error(\"Failed to replace sorted softrefs file %s: %s\",\n+\t\t\t\t\tto_file, strerror(errno));\n+\t}\n+\treturn ret;\n+}\n+\n+/* See softrefs.h for documentation */\n+int add_softrefs(const struct softref_list *list)\n+{\n+\tstruct softrefs_entry entry;\n+\tint fd;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\t/* Close internal softrefs file, if initialized. */\n+\tdeinit_softrefs_access();\n+\n+\t/* FIXME: File locking!? */\n+\tfd = open(git_path(UNSORTED_FILENAME), O_WRONLY|O_APPEND|O_CREAT, 0666);\n+\tif (fd < 0)\n+\t\treturn error(\"Failed to open softrefs file %s: %s\",\n+\t\t\t\tgit_path(UNSORTED_FILENAME), strerror(errno));\n+\tif (fstat(fd, &st))\n+\t\treturn error(\"Failed to fstat softrefs file %s: %s\",\n+\t\t\t\tgit_path(UNSORTED_FILENAME), strerror(errno));\n+\tif (st.st_size % sizeof(struct softrefs_entry))\n+\t\treturn error(\"Refuse to edit softrefs file %s: File does not have whole number of softref entries\",\n+\t\t\t\tgit_path(UNSORTED_FILENAME));\n+\n+\t/* File is open; start writing entries */\n+\twhile (list) {\n+\t\tif (!hashcmp(list->from_sha1, list->to_sha1)) {\n+\t\t\t/* self-reference: from_sha1 == to_sha1 */\n+\t\t\terror(\"Cannot add self-reference (%s -> %s)\",\n+\t\t\t\t\tsha1_to_hex(list->from_sha1),\n+\t\t\t\t\tsha1_to_hex(list->to_sha1));\n+\t\t}\n+\t\telse if (has_softref(list->from_sha1, list->to_sha1)) {\n+\t\t\t/* softref exists already */\n+\t\t\t/* nada */;\n+\t\t}\n+\t\telse { /* softref is ok */\n+\t\t\tstrcpy(entry.from_sha1_hex, sha1_to_hex(list->from_sha1));\n+\t\t\tstrcpy(entry.to_sha1_hex, sha1_to_hex(list->to_sha1));\n+\t\t\tentry.space = ' ';\n+\t\t\tentry.lf = '\\n';\n+\t\t\tif (write_entry(fd, &entry))\n+\t\t\t\terror(\"Failed to write entry to softrefs file %s: %s\",\n+\t\t\t\t\t\tgit_path(UNSORTED_FILENAME),\n+\t\t\t\t\t\tstrerror(errno));\n+\t\t\telse /* write_entry() succeeded */\n+\t\t\t\tret++;\n+\t\t}\n+\t\tlist = list->next;\n+\t}\n+\n+\t/* finished writing entries */\n+\tif (close(fd))\n+\t\treturn error(\"Failed to close softrefs file %s: %s\",\n+\t\t\t\tgit_path(UNSORTED_FILENAME), strerror(errno));\n+\n+\tmerge_unsorted_softrefs(NULL, 0);\n+\treturn ret;\n+}\n+\n+/* See softrefs.h for documentation */\n+int add_softref(const unsigned char *from_sha1, const unsigned char *to_sha1)\n+{\n+\tstruct softref_list l;\n+\tint ret;\n+\n+\tif (!hashcmp(from_sha1, to_sha1))\n+\t\treturn error(\"Cannot add self-reference (%s -> %s)\",\n+\t\t\tsha1_to_hex(from_sha1), sha1_to_hex(to_sha1));\n+\n+\thashcpy(l.from_sha1, from_sha1);\n+\thashcpy(l.to_sha1, to_sha1);\n+\tl.next = NULL;\n+\tret = add_softrefs(&l);\n+\tswitch (ret) {\n+\t\tcase 0: return 1;\n+\t\tcase 1: return 0;\n+\t\tdefault: return -1;\n+\t}\n+}\n+\n+/* See softrefs.h for documentation */\n+int merge_unsorted_softrefs(const char *unsorted_file, int force)\n+{\n+\tstruct stat st;\n+\tint num_entries;\n+\tint delete_file = 0; /* set to true to delete unsorted_file afterwards */\n+\tint ret = 0;\n+\n+\tif (unsorted_file == NULL) { /* use UNSORTED_FILENAME */\n+\t\tunsorted_file = git_path(UNSORTED_FILENAME);\n+\t\tdelete_file = 1;\n+\t\tif (access(unsorted_file, F_OK))\n+\t\t\t/* UNSORTED_FILENAME doesn't exist; nothing to do */\n+\t\t\treturn 0;\n+\t}\n+\telse {\n+\t\tforce = 1; /* no threshold on merging external file */\n+\t\tif (access(unsorted_file, F_OK))\n+\t\t\t/* external unsorted file doesn't exist; failure */\n+\t\t\treturn error(\"Failed to access softrefs file %s: %s\",\n+\t\t\t\t\tunsorted_file, strerror(errno));\n+\t}\n+\n+\tif (stat(unsorted_file, &st))\n+\t\treturn error(\"Failed to stat() softrefs file %s: %s\",\n+\t\t\t\tunsorted_file, strerror(errno));\n+\tif (st.st_size % sizeof(struct softrefs_entry))\n+\t\treturn error(\"Corrupt softrefs file %s: Aborting\",\n+\t\t\t\tunsorted_file);\n+\tif (st.st_size == 0) /* file is empty; nothing to do */\n+\t\treturn 0;\n+\tnum_entries = st.st_size / sizeof(struct softrefs_entry);\n+\tif (force || num_entries > MAX_UNSORTED_ENTRIES) { /* do it */\n+\t\tret = merge_unsorted_into_sorted(\n+\t\t\t\tunsorted_file, git_path(SORTED_FILENAME));\n+\t\tif (!ret && delete_file && unlink(unsorted_file))\n+\t\t\terror(\"Failed to remove unsorted softrefs file %s: %s\",\n+\t\t\t\t\tunsorted_file, strerror(errno));\n+\t}\n+\treturn ret;\n+}\n+\n+/* See softrefs.h for documentation */\n+int merge_sorted_softrefs(const char *sorted_file)\n+{\n+\tstruct stat st;\n+\tif (access(sorted_file, F_OK)) /* external file doesn't exist; FAIL */\n+\t\treturn error(\"Failed to access softrefs file %s: %s\",\n+\t\t\t\tsorted_file, strerror(errno));\n+\tif (stat(sorted_file, &st))\n+\t\treturn error(\"Failed to stat() softrefs file %s: %s\",\n+\t\t\t\tsorted_file, strerror(errno));\n+\tif (st.st_size % sizeof(struct softrefs_entry))\n+\t\treturn error(\"Corrupt softrefs file %s: Aborting\", sorted_file);\n+\tif (st.st_size == 0) /* file is empty; nothing to do */\n+\t\treturn 0;\n+\treturn merge_sorted_into_sorted(sorted_file, git_path(SORTED_FILENAME));\n+}\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44468","messageId":"200706092022.46933.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 3/7] Softrefs: Add git-softref, a builtin command for adding, listing and administering softrefs","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:22:46Z","receivedAt":"2007-06-09T18:22:46Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"git-softref is meant to be used from shell scripts that need to interact\nwith the softrefs database. The builtin command provides most of the\nfunctionality present in the softrefs C API.\n\nDocumentation to follow in a subsequent patch.\n\nSigned-off-by: Johan Herland \n---\n builtin-softref.c | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 167 insertions(+), 0 deletions(-)\n create mode 100644 builtin-softref.c\n\ndiff --git a/builtin-softref.c b/builtin-softref.c\nnew file mode 100644\nindex 0000000..f95db4e\n--- /dev/null\n+++ b/builtin-softref.c\n@@ -0,0 +1,167 @@\n+/*\n+ * git softref builtin command\n+ *\n+ * Add, list and administer soft references (softrefs)\n+ *\n+ * Copyright (c) 2007 Johan Herland\n+ */\n+\n+#include \"cache.h\"\n+#include \"tag.h\"\n+#include \"refs.h\"\n+#include \"softrefs.h\"\n+\n+static const char builtin_softref_usage[] =\n+\t\"git-softref [ --list []\"\n+\t\t\" | --has \"\n+\t\t\" | --add \"\n+\t\t\" | --rebuild-tags\"\n+\t\t\" | --merge-unsorted []\"\n+\t\t\" | --merge-sorted ]\";\n+\n+static int list_helper(\n+\t\tconst unsigned char *from_sha1,\n+\t\tconst unsigned char *to_sha1,\n+\t\tvoid *cb_data)\n+{\n+\tprintf(\"%s %s\\n\", sha1_to_hex(from_sha1), sha1_to_hex(to_sha1));\n+\treturn 0;\n+}\n+\n+int rebuild_tags_helper(\n+\t\tconst char *refname,\n+\t\tconst unsigned char *sha1,\n+\t\tint flags,\n+\t\tvoid *cb_data)\n+{\n+\tstruct object *o = parse_object(sha1);\n+\tif (o && o->type == OBJ_TAG) {\n+\t\tstruct tag *t = (struct tag *) o;\n+\t\tstruct softref_list **prev = (struct softref_list **) cb_data;\n+\t\tstruct softref_list *current = xmalloc(sizeof(struct softref_list));\n+\t\tcurrent->next = *prev;\n+\t\thashcpy(current->from_sha1, t->tagged->sha1);\n+\t\thashcpy(current->to_sha1, t->object.sha1);\n+\t\t*prev = current;\n+\t}\n+\treturn 0;\n+}\n+\n+int cmd_softref(int argc, const char **argv, const char *prefix)\n+{\n+\tint i;\n+\tint show_usage = 0, list = 0, has = 0, add = 0, rebuild_tags = 0,\n+\t merge_unsorted = 0, merge_sorted = 0;\n+\tconst char *from_name = NULL, *to_name = NULL, *softrefs_file = NULL;\n+\tunsigned char from_sha1[20], to_sha1[20];\n+\n+\tgit_config(git_default_config);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tif (!strcmp(arg, \"--list\")) {\n+\t\t\tlist = 1;\n+\t\t\tif (i + 1 < argc) /* given */\n+\t\t\t\tfrom_name = argv[++i];\n+\t\t}\n+\t\telse if (!strcmp(arg, \"--has\")) {\n+\t\t\thas = 1;\n+\t\t\tif (i + 2 >= argc)\n+\t\t\t\tshow_usage = error(\"--has needs two arguments: and \");\n+\t\t\telse {\n+\t\t\t\tfrom_name = argv[++i];\n+\t\t\t\tto_name = argv[++i];\n+\t\t\t}\n+\t\t}\n+\t\telse if (!strcmp(arg, \"--add\")) {\n+\t\t\tadd = 1;\n+\t\t\tif (i + 2 >= argc)\n+\t\t\t\tshow_usage = error(\"--add needs two arguments: and \");\n+\t\t\telse {\n+\t\t\t\tfrom_name = argv[++i];\n+\t\t\t\tto_name = argv[++i];\n+\t\t\t}\n+\t\t}\n+\t\telse if (!strcmp(arg, \"--rebuild-tags\"))\n+\t\t\trebuild_tags = 1;\n+\t\telse if (!strcmp(arg, \"--merge-unsorted\")) {\n+\t\t\tmerge_unsorted = 1;\n+\t\t\tif (i + 1 < argc) /* given */\n+\t\t\t\tsoftrefs_file = argv[++i];\n+\t\t}\n+\t\telse if (!strcmp(arg, \"--merge-sorted\")) {\n+\t\t\tmerge_sorted = 1;\n+\t\t\tif (i + 1 >= argc)\n+\t\t\t\tshow_usage = error(\"--merge-sorted needs one argument: \");\n+\t\t\telse\n+\t\t\t\tsoftrefs_file = argv[++i];\n+\t\t}\n+\t\telse\n+\t\t\tshow_usage = error(\"Unknown argument '%s'\", arg);\n+\t}\n+\n+\t/* default to --list if no command given; fail if more than one */\n+\tswitch(list + has + add + rebuild_tags + merge_unsorted + merge_sorted) {\n+\t\tcase 0: list = 1; break;\n+\t\tcase 1: break;\n+\t\tdefault: show_usage = 1;\n+\t}\n+\tif (show_usage)\n+\t\tusage(builtin_softref_usage);\n+\n+\tif (list) {\n+\t\tif (from_name) { /* show from_name's softrefs */\n+\t\t\tif (get_sha1(from_name, from_sha1))\n+\t\t\t\tdie(\"Not a valid object name %s\", from_name);\n+\t\t\tif (for_each_softref_with_from(from_sha1, list_helper, 0))\n+\t\t\t\tdie(\"Error encountered while listing softrefs\");\n+\t\t}\n+\t\telse if (for_each_softref(list_helper, 0)) /* show all softrefs */\n+\t\t\tdie(\"Error encountered while listing softrefs\");\n+\t}\n+\telse if (has) {\n+\t\tif (get_sha1(from_name, from_sha1) || !has_sha1_file(from_sha1))\n+\t\t\tdie(\"Not a valid object name %s\", from_name);\n+\t\tif (get_sha1(to_name, to_sha1) || !has_sha1_file(to_sha1))\n+\t\t\tdie(\"Not a valid object name %s\", to_name);\n+\t\treturn has_softref(from_sha1, to_sha1);\n+\t}\n+\telse if (add) {\n+\t\tif (get_sha1(from_name, from_sha1) || !has_sha1_file(from_sha1))\n+\t\t\tdie(\"Not a valid object name %s\", from_name);\n+\t\tif (get_sha1(to_name, to_sha1) || !has_sha1_file(to_sha1))\n+\t\t\tdie(\"Not a valid object name %s\", to_name);\n+\t\tif (add_softref(from_sha1, to_sha1) < 0)\n+\t\t\tdie(\"Failed to create softref from %s to %s\",\n+\t\t\t\tfrom_name, to_name);\n+\t}\n+\telse if (rebuild_tags) {\n+\t\t/*\n+\t\t * Find all tag objects, and add their corresponding softrefs\n+\t\t *\n+\t\t * For now, we'll have to settle for referenced tag objects as\n+\t\t * it seems to be non-trivial to find _all_ the tag objects in\n+\t\t * the db (including unreachables).\n+\t\t */\n+\t\tstruct softref_list *to_add = NULL;\n+\t\tstruct softref_list **p = &to_add;\n+\t\tint ret;\n+\t\tif (for_each_tag_ref(rebuild_tags_helper, (void *) p)) {\n+\t\t\tdelete_softref_list(to_add);\n+\t\t\tdie(\"Failed to find tag objects\");\n+\t\t}\n+\t\tret = add_softrefs(to_add);\n+\t\tdelete_softref_list(to_add);\n+\t\tif (ret < 0)\n+\t\t\tdie(\"Failed to add softrefs for tag objects\");\n+\t\tprintf(\"Added %i missing softrefs for tag objects.\\n\", ret);\n+\t}\n+\telse if (merge_unsorted) {\n+\t\treturn merge_unsorted_softrefs(softrefs_file, 1);\n+\t}\n+\telse if (merge_sorted) {\n+\t\treturn merge_sorted_softrefs(softrefs_file);\n+\t}\n+\n+\treturn 0;\n+}\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44469","messageId":"200706092023.13537.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 4/7] Softrefs: Add manual page documenting git-softref and softrefs subsystem in general","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:23:13Z","receivedAt":"2007-06-09T18:23:13Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Signed-off-by: Johan Herland \n---\n Documentation/git-softref.txt | 119 +++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 119 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-softref.txt\n\ndiff --git a/Documentation/git-softref.txt b/Documentation/git-softref.txt\nnew file mode 100644\nindex 0000000..6a3e13b\n--- /dev/null\n+++ b/Documentation/git-softref.txt\n@@ -0,0 +1,119 @@\n+git-softref(1)\n+==============\n+\n+NAME\n+----\n+git-softref - Create, list and administer soft references\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git-softref' --list []\n+'git-softref' --has \n+'git-softref' --add \n+'git-softref' --rebuild-tags\n+'git-softref' --merge-unsorted []\n+'git-softref' --merge-sorted \n+\n+\n+DESCRIPTION\n+-----------\n+Query and administer soft references in a git repository.\n+\n+Soft references are used to declare reachability between already existing\n+objects. An object (called the 'to-object') may be declared reachable from\n+another object (the 'from-object') without affecting the immutability of either\n+object.\n+\n+The `--list` option will list existing softrefs in the database. If given the\n+optional , the list is limited to softrefs from the given object.\n+The `--has` option is used to check for the existence of a softref between two\n+given objects, similarly the `--add` option is used to add such a softref.\n+\n+The `--rebuild-tags` option is used to generate softrefs for all tag objects in\n+the repository reachable from tag refs. Tag objects use softrefs to declare\n+reachability 'from' the tagged object, 'to' the tag object. This allows for tag\n+objects to the cloned/fetched/pushed along with their associated objects.\n+\n+Finally, the `--merge-unsorted` and `--merge-sorted` options are used to merge\n+softrefs files into the sorted softrefs db. The filename argument must point\n+to an existing file in unsorted/sorted softrefs format. The softrefs entries\n+in this file will be merged into the sorted softrefs db. The `--merge-unsorted`\n+option may be used 'without' a filename, in which case the currently unsorted\n+portion of the softrefs db will be merged into the sorted db. Note that this\n+last operation is also done regularly by default when adding softrefs, so\n+there is no need to invoke this option during regular use.\n+\n+\n+OPTIONS\n+-------\n+--list []::\n+\tList all softrefs that have the given ''.\n+\tIf '' is not given, list 'all' softrefs in the repository.\n+\n+--has ::\n+\tReturn with exit code 1 if the given softref exists in the repository.\n+\tReturn with exit code 0 otherwise.\n+\n+--add ::\n+\tAdd a softref from the given '' to the given ''.\n+\tThe '' will from now on be considered reachable from the\n+\t''.\n+\n+--rebuild-tags::\n+\tAutomatically generate softrefs for all tag objects reachable from\n+\ttag refs in the repository.\n+\n+--merge-unsorted []::\n+\tMerge the softrefs in the given unsorted '' into the\n+\tsorted softrefs db. If a filename is not given, force a merge of the\n+\tinternal unsorted softrefs store into the sorted softrefs db.\n+\n+--merge-sorted ::\n+\tMerge the softrefs in the given sorted '' into the\n+\tsorted softrefs db.\n+\n+\n+DISCUSSION\n+----------\n+Soft references (softrefs) is a general mechanism for declaring a relationship\n+between two existing arbitrary objects in the repo. Softrefs differ from the\n+existing reachability relationship in that a softref may be created after\n+'both' of the involved objects have been added to the repo. In contrast,\n+regular reachability depends on the reachable object's name being stored\n+'inside' the other object. A reachability relationship can therefore not be\n+created at a later time without violating the immutability of git objects.\n+\n+Softrefs are defined as going 'from' one object 'to' another object. Once\n+a softref between two objects has been created, the \"to\" object is considered\n+reachable from the \"from\" object.\n+\n+Also, softrefs are stored in a way that makes it easy and quick to find all\n+the \"to\" objects reachable from a given \"from\" object.\n+\n+The softrefs db consists of two files: `.git/softrefs.unsorted` and\n+`.git/softrefs.sorted`. Both files use the same format; one softref per line\n+of the form \"` \\n`\". Each sha1 sum is 40 bytes long; this\n+makes each entry exactly 82 bytes long.\n+\n+The entries in `.git/softrefs.sorted` are sorted on ``, in order to\n+make lookup fast. This file is also known as the \"sorted softrefs db\".\n+\n+The entries in `.git/softrefs.unsorted` are 'not' sorted. This is to make\n+insertion fast. This file is also known as the \"unsorted softrefs db\".\n+\n+When softrefs are created, they are appended to `.git/softrefs.unsorted`.\n+When `.git/softrefs.unsorted` reach a certain number of entries, all the\n+entries in `.git/softrefs.unsorted` are automatically merged into\n+`.git/softrefs.sorted`.\n+\n+\n+Author\n+------\n+Written by Johan Herland .\n+\n+\n+GIT\n+---\n+Part of the gitlink:git[7] suite\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44470","messageId":"200706092023.38661.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 5/7] Softrefs: Add testcases for basic softrefs behaviour","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:23:38Z","receivedAt":"2007-06-09T18:23:38Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Adds testing of the basic options available to the git-softref command.\n\nSigned-off-by: Johan Herland \n---\n t/t3050-softrefs.sh | 314 +++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 314 insertions(+), 0 deletions(-)\n create mode 100755 t/t3050-softrefs.sh\n\ndiff --git a/t/t3050-softrefs.sh b/t/t3050-softrefs.sh\nnew file mode 100755\nindex 0000000..a925178\n--- /dev/null\n+++ b/t/t3050-softrefs.sh\n@@ -0,0 +1,314 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Johan Herland\n+#\n+\n+test_description='Basic functionality of soft references'\n+. ./test-lib.sh\n+\n+\n+# Prepare repo and create some notes\n+\n+test_expect_success 'Populating repo with test data' '\n+\techo \"foo\" > foo &&\n+\tgit-add foo &&\n+\ttest_tick &&\n+\tgit-commit -m \"Initial commit\" &&\n+\tgit-tag -a -m \"Tagging initial commit\" footag &&\n+\techo \"bar\" >> foo &&\n+\ttest_tick &&\n+\tgit-commit -m \"Second commit\" foo &&\n+\tgit-tag -a -m \"Tagging second commit\" bartag\n+'\n+\n+# At this point we should have:\n+# - commit @ 301711b66fe71164f646b798706a2c1f7024da8d (\"Initial commit\")\n+# - tag @ ad60bc179c6874af6d97f181c67f11adcca5122b (\"footag\")\n+# - commit @ 9671cbee7ad26528645b2665c8f74d39a6288864 (\"Second commit\")\n+# - tag @ a927fc832d42f1f64d8318e8acec43545d9562de (\"bartag\")\n+# - The tag creation should also have created softrefs:\n+# - From \"Initial commit\" to \"footag\"\n+# - From \"Second commit\" to \"bartag\"\n+\n+# Testing git-softref --list\n+\n+test_expect_success 'Testing git-softref --list on initial test data (1)' '\n+\tcat > expected_output << EOF &&\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tgit-softref > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list > actual_output 2>&1 &&\n+\tcmp actual_output expected_output\n+'\n+\n+test_expect_success 'Testing git-softref --list on initial test data (2)' '\n+\tcat > expected_output << EOF &&\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tgit-softref --list 9671cbee7ad26528645b2665c8f74d39a6288864 > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list HEAD > actual_output 2>&1 &&\n+\tcmp actual_output expected_output\n+'\n+\n+test_expect_success 'Testing git-softref --list on initial test data (3)' '\n+\tcat > expected_output << EOF &&\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+EOF\n+\tgit-softref --list 301711b66fe71164f646b798706a2c1f7024da8d > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list HEAD^ > actual_output 2>&1 &&\n+\tcmp actual_output expected_output\n+'\n+\n+test_expect_success 'Testing git-softref --list on initial test data (4)' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\tgit-softref --list footag > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list bartag > actual_output 2>&1 &&\n+\tcmp actual_output expected_output\n+'\n+\n+# Testing git-softref --has\n+\n+test_expect_success 'Testing git-softref --has on initial test data' '\n+\t(git-softref --has 301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b;\n+\ttest \"$?\" = \"1\") &&\n+\t(git-softref --has HEAD^ footag;\n+\ttest \"$?\" = \"1\") &&\n+\t(git-softref --has footag^{} footag;\n+\ttest \"$?\" = \"1\") &&\n+\t(git-softref --has 9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de;\n+\ttest \"$?\" = \"1\") &&\n+\t(git-softref --has HEAD bartag;\n+\ttest \"$?\" = \"1\") &&\n+\t(git-softref --has bartag^{} bartag;\n+\ttest \"$?\" = \"1\") &&\n+\t(git-softref --has ad60bc179c6874af6d97f181c67f11adcca5122b 301711b66fe71164f646b798706a2c1f7024da8d;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has a927fc832d42f1f64d8318e8acec43545d9562de 9671cbee7ad26528645b2665c8f74d39a6288864;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has HEAD HEAD^;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has HEAD^ HEAD;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has footag HEAD;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has bartag HEAD;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has footag HEAD^;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has bartag HEAD^;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has footag bartag;\n+\ttest \"$?\" = \"0\") &&\n+\t(git-softref --has bartag footag;\n+\ttest \"$?\" = \"0\")\n+'\n+\n+# Testing git-softref --rebuild-tags\n+\n+test_expect_success 'Testing git-softref --rebuild-tags on initial test data' '\n+\tcat > expected_output << EOF &&\n+Added 0 missing softrefs for tag objects.\n+EOF\n+\tcat .git/softrefs.* | sort > expected_softrefs &&\n+\tgit-softref --rebuild-tags > actual_output 2>&1 &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+# Testing git-softref --add\n+\n+test_expect_success 'Testing git-softref --add with existing softref' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\tcat .git/softrefs.* | sort > expected_softrefs &&\n+\tgit-softref --add HEAD bartag > actual_output 2>&1 &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+test_expect_success 'Testing git-softref --add with self-refential softref' '\n+\tcat > expected_output << EOF &&\n+error: Cannot add self-reference (9671cbee7ad26528645b2665c8f74d39a6288864 -> 9671cbee7ad26528645b2665c8f74d39a6288864)\n+fatal: Failed to create softref from HEAD to HEAD\n+EOF\n+\tcat .git/softrefs.* | sort > expected_softrefs &&\n+\t(git-softref --add HEAD HEAD > actual_output 2>&1; test \"$?\" != \"0\") &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+test_expect_success 'Testing git-softref --add with non-existing objects (1)' '\n+\tcat > expected_output << EOF &&\n+fatal: Not a valid object name 1234567890123456789012345678901234567890\n+EOF\n+\tcat .git/softrefs.* | sort > expected_softrefs &&\n+\t(git-softref --add 1234567890123456789012345678901234567890 HEAD > actual_output 2>&1;\n+\t\ttest \"$?\" != \"0\") &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs &&\n+\t(git-softref --add HEAD 1234567890123456789012345678901234567890 > actual_output 2>&1;\n+\t\ttest \"$?\" != \"0\") &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+test_expect_success 'Testing git-softref --add with non-existing objects (2)' '\n+\tcat > expected_output << EOF &&\n+fatal: Not a valid object name HEAD^^^\n+EOF\n+\tcat .git/softrefs.* | sort > expected_softrefs &&\n+\t(git-softref --add HEAD^^^ HEAD > actual_output 2>&1; test \"$?\" != \"0\") &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+test_expect_success 'Testing git-softref --add with valid arguments (1)' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\tcat > new_softref << EOF\n+301711b66fe71164f646b798706a2c1f7024da8d 9671cbee7ad26528645b2665c8f74d39a6288864\n+EOF\n+\tcat .git/softrefs.* new_softref | sort > expected_softrefs &&\n+\tgit-softref --add HEAD^ HEAD > actual_output 2>&1 &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+test_expect_success 'Testing git-softref --add with valid arguments (2)' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\tcat > new_softref << EOF\n+ad60bc179c6874af6d97f181c67f11adcca5122b a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tcat .git/softrefs.* new_softref | sort > expected_softrefs &&\n+\tgit-softref --add footag bartag > actual_output 2>&1 &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+# Removing softrefs\n+\n+test_expect_success 'Removing all softrefs' '\n+\trm .git/softrefs.*\n+'\n+\n+# Testing git-softref --list and --has\n+\n+test_expect_success 'Testing git-softref with no softrefs' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\tgit-softref > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list 9671cbee7ad26528645b2665c8f74d39a6288864 > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list HEAD > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list HEAD^ > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list footag > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --list bartag > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --has HEAD bartag > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tgit-softref --has HEAD^ footag > actual_output 2>&1 &&\n+\tcmp actual_output expected_output\n+'\n+\n+# Testing git-softref --rebuild-tags\n+# (Should recreated missing softrefs for tag objects reachable from 'refs/tags')\n+\n+test_expect_success 'Testing git-softref --rebuild-tags to rebuild missing tag softrefs' '\n+\tcat > expected_output << EOF &&\n+Added 2 missing softrefs for tag objects.\n+EOF\n+\tcat > new_softref << EOF\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tcat .git/softrefs.* new_softref | sort > expected_softrefs &&\n+\tgit-softref --rebuild-tags > actual_output 2>&1 &&\n+\tcat .git/softrefs.* | sort > actual_softrefs &&\n+\tcmp actual_output expected_output &&\n+\tcmp actual_softrefs expected_softrefs\n+'\n+\n+# Testing git-softref --merge-unsorted\n+\n+test_expect_success 'Testing git-softref --merge-unsorted' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\trm .git/softrefs*\n+\tcat > .git/softrefs.unsorted << EOF\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+EOF\n+\tcat > expected_softrefs << EOF\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tgit-softref --merge-unsorted > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tcmp .git/softrefs.sorted expected_softrefs &&\n+\ttest ! -e .git/softrefs.unsorted\n+'\n+\n+# Testing git-softref --merge-unsorted \n+\n+test_expect_success 'Testing git-softref --merge-unsorted ' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\trm .git/softrefs*\n+\tcat > new_softrefs << EOF\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+EOF\n+\tcat > expected_softrefs << EOF\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tgit-softref --merge-unsorted new_softrefs > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tcmp .git/softrefs.sorted expected_softrefs &&\n+\ttest -e new_softrefs\n+'\n+\n+# Testing git-softref --merge-sorted \n+\n+test_expect_success 'Testing git-softref --merge-sorted ' '\n+\tcat > expected_output << EOF &&\n+EOF\n+\trm .git/softrefs*\n+\tcat > new_softrefs << EOF\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tcat > expected_softrefs << EOF\n+301711b66fe71164f646b798706a2c1f7024da8d ad60bc179c6874af6d97f181c67f11adcca5122b\n+9671cbee7ad26528645b2665c8f74d39a6288864 a927fc832d42f1f64d8318e8acec43545d9562de\n+EOF\n+\tgit-softref --merge-sorted new_softrefs > actual_output 2>&1 &&\n+\tcmp actual_output expected_output &&\n+\tcmp .git/softrefs.sorted expected_softrefs &&\n+\ttest -e new_softrefs\n+'\n+\n+# FIXME: More testing needed on how softrefs interact with the rest of git\n+\n+test_done\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44471","messageId":"200706092024.01593.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 6/7] Softrefs: Administrivia associated with softrefs subsystem and git-softref builtin","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:24:01Z","receivedAt":"2007-06-09T18:24:01Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Also cleans up sorting and whitespace in Documentation/cmd-list.perl\n\nSigned-off-by: Johan Herland \n---\n .gitignore | 1 +\n Documentation/cmd-list.perl | 7 ++++---\n Makefile | 6 ++++--\n builtin.h | 1 +\n git.c | 1 +\n 5 files changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/.gitignore b/.gitignore\nindex 27e5aeb..7fd6904 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -119,6 +119,7 @@ git-show\n git-show-branch\n git-show-index\n git-show-ref\n+git-softref\n git-ssh-fetch\n git-ssh-pull\n git-ssh-push\ndiff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl\nindex a181f75..4e1f45b 100755\n--- a/Documentation/cmd-list.perl\n+++ b/Documentation/cmd-list.perl\n@@ -90,6 +90,7 @@ git-clean mainporcelain\n git-clone mainporcelain\n git-commit mainporcelain\n git-commit-tree plumbingmanipulators\n+git-config ancillarymanipulators\n git-convert-objects ancillarymanipulators\n git-count-objects ancillaryinterrogators\n git-cvsexportcommit foreignscminterface\n@@ -101,13 +102,13 @@ git-diff-files plumbinginterrogators\n git-diff-index plumbinginterrogators\n git-diff mainporcelain\n git-diff-tree plumbinginterrogators\n-git-fast-import\t\t\t\tancillarymanipulators\n+git-fast-import ancillarymanipulators\n git-fetch mainporcelain\n git-fetch-pack synchingrepositories\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n git-format-patch mainporcelain\n-git-fsck\t ancillaryinterrogators\n+git-fsck ancillaryinterrogators\n git-gc mainporcelain\n git-get-tar-commit-id ancillaryinterrogators\n git-grep mainporcelain\n@@ -155,7 +156,6 @@ git-receive-pack synchelpers\n git-reflog ancillarymanipulators\n git-relink ancillarymanipulators\n git-repack ancillarymanipulators\n-git-config ancillarymanipulators\n git-remote ancillarymanipulators\n git-request-pull foreignscminterface\n git-rerere ancillaryinterrogators\n@@ -174,6 +174,7 @@ git-show-branch ancillaryinterrogators\n git-show-index plumbinginterrogators\n git-show-ref plumbinginterrogators\n git-sh-setup purehelpers\n+git-softref ancillarymanipulators\n git-ssh-fetch synchingrepositories\n git-ssh-upload synchingrepositories\n git-status mainporcelain\ndiff --git a/Makefile b/Makefile\nindex 0f75955..22e3e53 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -296,7 +296,7 @@ LIB_H = \\\n \trun-command.h strbuf.h tag.h tree.h git-compat-util.h revision.h \\\n \ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h \\\n \tutf8.h reflog-walk.h patch-ids.h attr.h decorate.h progress.h \\\n-\tmailmap.h remote.h\n+\tmailmap.h remote.h softrefs.h\n \n DIFF_OBJS = \\\n \tdiff.o diff-lib.o diffcore-break.o diffcore-order.o \\\n@@ -318,7 +318,8 @@ LIB_OBJS = \\\n \twrite_or_die.o trace.o list-objects.o grep.o match-trees.o \\\n \talloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \\\n \tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o utf8.o \\\n-\tconvert.o attr.o decorate.o progress.o mailmap.o symlinks.o remote.o\n+\tconvert.o attr.o decorate.o progress.o mailmap.o symlinks.o remote.o \\\n+\tsoftrefs.o\n \n BUILTIN_OBJS = \\\n \tbuiltin-add.o \\\n@@ -370,6 +371,7 @@ BUILTIN_OBJS = \\\n \tbuiltin-runstatus.o \\\n \tbuiltin-shortlog.o \\\n \tbuiltin-show-branch.o \\\n+\tbuiltin-softref.o \\\n \tbuiltin-stripspace.o \\\n \tbuiltin-symbolic-ref.o \\\n \tbuiltin-tar-tree.o \\\ndiff --git a/builtin.h b/builtin.h\nindex da4834c..beae52c 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -67,6 +67,7 @@ extern int cmd_runstatus(int argc, const char **argv, const char *prefix);\n extern int cmd_shortlog(int argc, const char **argv, const char *prefix);\n extern int cmd_show(int argc, const char **argv, const char *prefix);\n extern int cmd_show_branch(int argc, const char **argv, const char *prefix);\n+extern int cmd_softref(int argc, const char **argv, const char *prefix);\n extern int cmd_stripspace(int argc, const char **argv, const char *prefix);\n extern int cmd_symbolic_ref(int argc, const char **argv, const char *prefix);\n extern int cmd_tar_tree(int argc, const char **argv, const char *prefix);\ndiff --git a/git.c b/git.c\nindex 29b55a1..96cc0b8 100644\n--- a/git.c\n+++ b/git.c\n@@ -283,6 +283,7 @@ static void handle_internal_command(int argc, const char **argv, char **envp)\n \t\t{ \"shortlog\", cmd_shortlog, RUN_SETUP | USE_PAGER },\n \t\t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t\t{ \"show\", cmd_show, RUN_SETUP | USE_PAGER },\n+\t\t{ \"softref\", cmd_softref, RUN_SETUP },\n \t\t{ \"stripspace\", cmd_stripspace },\n \t\t{ \"symbolic-ref\", cmd_symbolic_ref, RUN_SETUP },\n \t\t{ \"tar-tree\", cmd_tar_tree },\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44472","messageId":"200706092024.23548.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH 7/7] Teach git-mktag to register softrefs for all tag objects","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:24:23Z","receivedAt":"2007-06-09T18:24:23Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"For each tag object created, we create a corresponding softref from the\ntagged object to the tag object itself. This is needed to enable efficient\nlookup of which tag objects that point to a given commit/object.\n\nSigned-off-by: Johan Herland \n---\n mktag.c | 11 ++++++++++-\n 1 files changed, 10 insertions(+), 1 deletions(-)\n\ndiff --git a/mktag.c b/mktag.c\nindex af0cfa6..db8a6b8 100644\n--- a/mktag.c\n+++ b/mktag.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"tag.h\"\n+#include \"softrefs.h\"\n \n /*\n * Tag object data has the following format: two mandatory lines of\n@@ -32,6 +33,7 @@ int main(int argc, char **argv)\n {\n \tunsigned long size = 4096;\n \tchar *buffer = xmalloc(size);\n+\tstruct tag result_tag;\n \tunsigned char result_sha1[20];\n \n \tif (argc != 1)\n@@ -46,7 +48,7 @@ int main(int argc, char **argv)\n \tbuffer[size] = 0;\n \n \t/* Verify tag object data */\n-\tif (parse_and_verify_tag_buffer(0, buffer, size, 1)) {\n+\tif (parse_and_verify_tag_buffer(&result_tag, buffer, size, 1)) {\n \t\tfree(buffer);\n \t\tdie(\"invalid tag data file\");\n \t}\n@@ -57,6 +59,13 @@ int main(int argc, char **argv)\n \t}\n \n \tfree(buffer);\n+\n+\t/* Create reverse mapping softref */\n+\tif (add_softref(result_tag.tagged->sha1, result_sha1) < 0) {\n+\t\tdie(\"unable to create softref for resulting tag object %s\",\n+\t\t\tsha1_to_hex(result_sha1));\n+\t}\n+\n \tprintf(\"%s\\n\", sha1_to_hex(result_sha1));\n \treturn 0;\n }\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44473","messageId":"200706092025.30156.johan@herland.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"[PATCH] Change softrefs file format from text (82 bytes per entry) to binary (40 bytes per entry)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T18:25:30Z","receivedAt":"2007-06-09T18:25:30Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"The text-based softrefs file format uses 82 bytes per entry (40 bytes\nfrom_sha1 in hex, 1 byte SP, 40 bytes to_sha1 in hex, 1 byte LF).\n\nThe binary softrefs file format uses 40 bytes per entry (20 bytes\nfrom_sha1, 20 bytes to_sha1).\n\nMoving to a binary format increases performance slightly, but sacrifices\neasy readability of the softrefs files.\n\nSigned-off-by: Johan Herland \n---\n\nTo illustrate the change in performance from changing the softrefs file\nformat, I prepared a linux repo (holding 57274 commits) with 10 tag\nobjects, and created softrefs from every commit to every tag object\n(572740 softrefs in total). The resulting softrefs db was 46964680 bytes\nwhen using the text format, and 22909600 bytes when using the binary\nformat. The experiment was done on a 32-bit Intel Pentium 4\n(3 GHz w/HyperThreading) with 1 GB RAM:\n\n\n========\nOperations on unsorted softrefs:\n(572740 (10 per commit) entries in random/unsorted order)\n========\n\nListing all softrefs\n(sequential reading of unsorted softrefs file)\n--------\n[text format]\n$ /usr/bin/time git softref --list > /dev/null\n0.44user 0.02system 0:00.47elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+11786minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --list > /dev/null\n0.35user 0.01system 0:00.38elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+5913minor)pagefaults 0swaps\n\nListing HEAD's softrefs\n(sequential reading of unsorted softrefs file)\n--------\n[text format]\n$ /usr/bin/time git softref --list HEAD > /dev/null\n0.11user 0.01system 0:00.14elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+11790minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --list HEAD > /dev/null\n0.02user 0.01system 0:00.03elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+5918minor)pagefaults 0swaps\n\nSorting softrefs\n--------\n[text format]\n$ /usr/bin/time git softref --merge-unsorted\n2.73user 4.97system 0:07.77elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+15833minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --merge-unsorted\n1.78user 5.00system 0:06.79elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+9961minor)pagefaults 0swaps\n\nSorting softrefs into existing sorted file\n(throwing away duplicates)\n--------\n[text format]\n$ /usr/bin/time git softref --merge-unsorted\n3.49user 5.12system 0:08.64elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+27300minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --merge-unsorted\n2.03user 4.92system 0:07.05elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+15556minor)pagefaults 0swaps\n\n\n========\nOperations on sorted softrefs:\n(572740 (10 per commit) entries in sorted order)\n========\n\nListing all softrefs\n(sequential reading of sorted softrefs file)\n--------\n[text format]\n$ /usr/bin/time git softref --list > /dev/null\n0.43user 0.02system 0:00.48elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+11786minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --list > /dev/null\n0.37user 0.00system 0:00.38elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+5914minor)pagefaults 0swaps\n\nListing HEAD's softrefs\n(256-fanout followed by binary search in sorted softrefs file)\n--------\n[text format]\n$/usr/bin/time git softref --list HEAD > /dev/null\n0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+334minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --list HEAD > /dev/null\n0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+333minor)pagefaults 0swaps\n\nSorting softrefs\n(no-op)\n--------\n[text format]\n$ /usr/bin/time git softref --merge-unsorted\n0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+312minor)pagefaults 0swaps\n[binary format]\n$ /usr/bin/time git softref --merge-unsorted\n0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+312minor)pagefaults 0swaps\n\n\nAs expected, the binary format almost halved the number of pagefaults for\ncases causing the entire softrefs db to be read (the reason for this, of\ncourse, being the halved size of the db).\n\nFor the most common use case (looking up a given commit in a sorted db)\nthe binary format has no measurable effect on performance.\n\n\n...Johan\n\n\n Documentation/git-softref.txt | 6 ++--\n softrefs.c | 78 +++++++++++------------------------------\n softrefs.h | 7 ++--\n 3 files changed, 27 insertions(+), 64 deletions(-)\n\ndiff --git a/Documentation/git-softref.txt b/Documentation/git-softref.txt\nindex 6a3e13b..e21aaf3 100644\n--- a/Documentation/git-softref.txt\n+++ b/Documentation/git-softref.txt\n@@ -93,9 +93,9 @@ Also, softrefs are stored in a way that makes it easy and quick to find all\n the \"to\" objects reachable from a given \"from\" object.\n \n The softrefs db consists of two files: `.git/softrefs.unsorted` and\n-`.git/softrefs.sorted`. Both files use the same format; one softref per line\n-of the form \"` \\n`\". Each sha1 sum is 40 bytes long; this\n-makes each entry exactly 82 bytes long.\n+`.git/softrefs.sorted`. Both files use the same binary format; ``\n+followed by `` per entry. Each sha1 sum is 20 bytes long; this\n+makes each entry exactly 40 bytes long.\n \n The entries in `.git/softrefs.sorted` are sorted on ``, in order to\n make lookup fast. This file is also known as the \"sorted softrefs db\".\ndiff --git a/softrefs.c b/softrefs.c\nindex c7308c8..8bb3a83 100644\n--- a/softrefs.c\n+++ b/softrefs.c\n@@ -9,10 +9,8 @@ static const unsigned int MAX_UNSORTED_ENTRIES = 1000;\n \n /* softref entry as it appears in a softrefs file */\n struct softrefs_entry {\n-\tchar from_sha1_hex[40];\n-\tchar space;\n-\tchar to_sha1_hex[40];\n-\tchar lf;\n+\tunsigned char from_sha1[20];\n+\tunsigned char to_sha1[20];\n };\n \n /* simple encapsulation of a softrefs file */\n@@ -106,8 +104,9 @@ static int write_entry(int fd, const struct softrefs_entry *entry)\n \tif (write(fd, (const void *) entry, sizeof(struct softrefs_entry))\n \t\t< sizeof(struct softrefs_entry))\n \t{\n-\t\treturn error(\"Failed to write entry '%.40s -> %.40s' to softrefs file descriptor %i: %s\",\n-\t\t\t\tentry->from_sha1_hex, entry->to_sha1_hex,\n+\t\treturn error(\"Failed to write entry '%s -> %s' to softrefs file descriptor %i: %s\",\n+\t\t\t\tsha1_to_hex(entry->from_sha1),\n+\t\t\t\tsha1_to_hex(entry->to_sha1),\n \t\t\t\tfd, strerror(errno));\n \t}\n \treturn 0;\n@@ -139,29 +138,13 @@ void deinit_softrefs_access()\n \tinternal_file = 0;\n }\n \n-/* comparison between a SHA1 sum and a softrefs entry */\n-static int sha1_to_entry_cmp(\n-\tconst unsigned char *from_sha1, const struct softrefs_entry *entry)\n-{\n-\tunsigned char sha1[20];\n-\tget_sha1_hex(entry->from_sha1_hex, sha1);\n-\treturn hashcmp(from_sha1, sha1);\n-}\n-\n /* comparison between softrefs entries */\n static int softrefs_entry_cmp(\n \t\tconst struct softrefs_entry *a, const struct softrefs_entry *b)\n {\n-\tunsigned char sa[20], sb[20];\n-\tint ret;\n-\tget_sha1_hex(a->from_sha1_hex, sa);\n-\tget_sha1_hex(b->from_sha1_hex, sb);\n-\tret = hashcmp(sa, sb);\n-\tif (!ret) {\n-\t\tget_sha1_hex(a->to_sha1_hex, sa);\n-\t\tget_sha1_hex(b->to_sha1_hex, sb);\n-\t\tret = hashcmp(sa, sb);\n-\t}\n+\tint ret = hashcmp(a->from_sha1, b->from_sha1);\n+\tif (!ret)\n+\t\tret = hashcmp(a->to_sha1, b->to_sha1);\n \treturn ret;\n }\n \n@@ -193,26 +176,15 @@ static int do_for_each_sequential(\n \t\tunsigned long i,\n \t\tint stop_at_first_non_match)\n {\n-\tunsigned char f_sha1[20], t_sha1[20]; /* Holds sha1 per entry */\n \tint ret = 0;\n \tfor (; i < file->data_len; ++i) { /* Step through file, starting at i */\n-\t\t/* sanity check entry */\n-\t\tif (file->data[i].space != ' ' || file->data[i].lf != '\\n') {\n-\t\t\tret = error(\"Entry #%lu in softrefs file %s failed sanity check\",\n-\t\t\t\t\ti, file->filename);\n-\t\t\tbreak;\n-\t\t}\n-\t\t/* retrieve SHA1 values */\n-\t\tif (get_sha1_hex(file->data[i].from_sha1_hex, f_sha1) ||\n-\t\t get_sha1_hex(file->data[i].to_sha1_hex, t_sha1)) {\n-\t\t\tret = error(\"Failed to read SHA1 values from entry #%lu in softrefs file %s\",\n-\t\t\t\t\ti, file->filename);\n-\t\t\tbreak;\n-\t\t}\n \t\t/* Compare to lookup value */\n-\t\tif (!from_sha1 || !hashcmp(from_sha1, f_sha1)) {\n-\t\t\tif ((ret = fn(f_sha1, t_sha1, cb_data)))\n+\t\tif (!from_sha1 || !hashcmp(from_sha1, file->data[i].from_sha1)) {\n+\t\t\tif ((ret = fn(file->data[i].from_sha1,\n+\t\t\t\t\tfile->data[i].to_sha1, cb_data)))\n+\t\t\t{\n \t\t\t\tbreak; /* bail out if callback returns != 0 */\n+\t\t\t}\n \t\t}\n \t\telse if (stop_at_first_non_match)\n \t\t\tbreak;\n@@ -277,7 +249,7 @@ static int do_for_each_sorted(\n \ti = (from_sha1[0] * file->data_len) / 256;\n \n \t/* Binary search */\n-\twhile ((cmp_result = sha1_to_entry_cmp(from_sha1, &(file->data[i])))) {\n+\twhile ((cmp_result = hashcmp(from_sha1, file->data[i].from_sha1))) {\n \t\tif (right - left <= 1) /* not found; give up */\n \t\t\tgoto done;\n \t\tif (cmp_result > 0) /* go right */\n@@ -288,7 +260,7 @@ static int do_for_each_sorted(\n \t}\n \n \t/* i points to a matching entry, but not necessarily the first */\n-\twhile (i >= 1 && sha1_to_entry_cmp(from_sha1, &(file->data[i - 1])) == 0)\n+\twhile (i >= 1 && !hashcmp(from_sha1, file->data[i - 1].from_sha1))\n \t\t--i;\n \n \t/* i points to the first matching entry */\n@@ -432,7 +404,6 @@ static int merge_unsorted_into_sorted(\n \twhile (!ret && (i < unsorted.data_len || j < sorted.data_len)) {\n \t\t/* there are still entries in either list */\n \t\tstruct softrefs_entry *cur;\n-\t\tunsigned char from_sha1[20], to_sha1[20];\n \t\tif (i < unsorted.data_len && j < sorted.data_len) {\n \t\t\t/* there are still entries in _both_ lists */\n \t\t\t/* choose \"lowest\" entry from either list */\n@@ -451,10 +422,8 @@ static int merge_unsorted_into_sorted(\n \t\tprev = cur;\n \n \t\t/* skip writing if softref involves a non-existing object */\n-\t\tif (get_sha1_hex(cur->from_sha1_hex, from_sha1) ||\n-\t\t\t!has_sha1_file(from_sha1) ||\n-\t\t get_sha1_hex(cur->to_sha1_hex, to_sha1) ||\n-\t\t\t!has_sha1_file( to_sha1))\n+\t\tif (!has_sha1_file(cur->from_sha1) ||\n+\t\t !has_sha1_file(cur->to_sha1))\n \t\t{\n \t\t\tcontinue;\n \t\t}\n@@ -523,7 +492,6 @@ static int merge_sorted_into_sorted(const char *from_file, const char *to_file)\n \twhile (!ret && (i < file1.data_len || j < file2.data_len)) {\n \t\t/* there are still entries in either list */\n \t\tstruct softrefs_entry *cur;\n-\t\tunsigned char from_sha1[20], to_sha1[20];\n \t\tif (i < file1.data_len && j < file2.data_len) {\n \t\t\t/* there are still entries in _both_ lists */\n \t\t\t/* choose \"lowest\" entry from either list */\n@@ -542,10 +510,8 @@ static int merge_sorted_into_sorted(const char *from_file, const char *to_file)\n \t\tprev = cur;\n \n \t\t/* skip writing if softref involves a non-existing object */\n-\t\tif (get_sha1_hex(cur->from_sha1_hex, from_sha1) ||\n-\t\t\t!has_sha1_file(from_sha1) ||\n-\t\t get_sha1_hex(cur->to_sha1_hex, to_sha1) ||\n-\t\t\t!has_sha1_file( to_sha1))\n+\t\tif (!has_sha1_file(cur->from_sha1) ||\n+\t\t !has_sha1_file(cur->to_sha1))\n \t\t{\n \t\t\tcontinue;\n \t\t}\n@@ -608,10 +574,8 @@ int add_softrefs(const struct softref_list *list)\n \t\t\t/* nada */;\n \t\t}\n \t\telse { /* softref is ok */\n-\t\t\tstrcpy(entry.from_sha1_hex, sha1_to_hex(list->from_sha1));\n-\t\t\tstrcpy(entry.to_sha1_hex, sha1_to_hex(list->to_sha1));\n-\t\t\tentry.space = ' ';\n-\t\t\tentry.lf = '\\n';\n+\t\t\thashcpy(entry.from_sha1, list->from_sha1);\n+\t\t\thashcpy(entry.to_sha1, list->to_sha1);\n \t\t\tif (write_entry(fd, &entry))\n \t\t\t\terror(\"Failed to write entry to softrefs file %s: %s\",\n \t\t\t\t\t\tgit_path(UNSORTED_FILENAME),\ndiff --git a/softrefs.h b/softrefs.h\nindex db0f8b9..89d25ce 100644\n--- a/softrefs.h\n+++ b/softrefs.h\n@@ -19,10 +19,9 @@\n * the \"to\" objects reachable from a given \"from\" object.\n *\n * The softrefs db consists of two files: .git/softrefs.unsorted and\n- * .git/softrefs.sorted. Both files use the same format; one softref per line\n- * of the form \" \\n\". Each sha1 sum is 40 bytes long; this\n- * makes each entry exactly 82 bytes long (including the space between the sha1\n- * sums and the terminating linefeed).\n+ * .git/softrefs.sorted. Both files use the same binary format; \n+ * followed by per entry. Each sha1 sum is 20 bytes long; this\n+ * makes each entry exactly 40 bytes long.\n *\n * The entries in .git/softrefs.sorted are sorted on , in order to\n * make lookup fast.\n-- \n1.5.2.1.144.gabc40\n"},{"id":"44510","messageId":"466B305A.5080802@midwinter.com","threadId":"8421","inReplyTo":"200706040251.05286.johan@herland.net","subject":"Re: Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-06-09T22:57:30Z","receivedAt":"2007-06-09T22:57:30Z","isPatch":false,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"(Resending this in plaintext; sorry to those who got it twice.)\n\nBeing able to specify relationships between commits after the fact seems \nlike a very useful facility.\n\nDoes it make sense to have type information to record what the \nrelationship between two objects means? Without that, it seems like \nit'll be hard to build much of a tool set on top of this feature, since \nno two tools that made use of it could unambiguously query just their \nown softrefs.\n\nA few use cases for relationships-after-the-fact come to mind in \naddition to the one the patch itself mentions:\n\nA facility like this could replace the info/grafts file, or at least \nprovide another way to turn a regular commit into a merge commit. Just \nput a \"manually specified merge parent\" ref between the target revision \nand the one you want git to think you've merged from. That would scale a \nlot better than info/grafts does, I suspect, if only by virtue of being \nO(log n) searchable thanks to the sorting.\n\nOne could easily imagine recording a \"cherry picked\" softref, which \ncould, e.g., be the rebase machinery to skip over an already-applied \nrevision. IMO the lack of any tool-readable history about cherry picking \n-- which is, after all, a sort of merge, at least conceptually -- is a \nshortcoming in present-day git. (Not a huge one, but if nothing else \nit'd be great to see cherry picking represented in, e.g., the gitk \nhistory display.)\n\nIt might be possible to annotate rebases to make pulling from rebased \nbranches less troublesome. If you have\n\nA--B--C--D\n \\\n -E--F--G\n\nand you rebase E onto D, a \"rebased from\" softref could be recorded \nbetween E and E':\n\nA--B--C--D\n \\ \\\n -E....E'--F'--G'\n\nThen a pulling client could potentially use that information to cleanly \nreplay the rebase operation to keep its history straight. Perhaps if you \ncould record historical rebases like that, you could do away with the \ncurrent gotchas involving rebasing shared repositories. One negative \nside effect would be that you'd end up needing to keep E around where \nbefore you'd have been able to throw it away, but it should delta \ncompress well, and you can, I think, still prune revisions F and G in \nthe above picture. Or maybe it's enough to just keep E's SHA1 around \nwithout actually retaining its contents.\n\nBut in any event, this seems like the start of a useful new set of \ncapabilities for git.\n\n-Steve\n"},{"id":"44511","messageId":"200706100116.46062.johan@herland.net","threadId":"8421","inReplyTo":"466B305A.5080802@midwinter.com","subject":"Re: Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-09T23:16:45Z","receivedAt":"2007-06-09T23:16:45Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 10 June 2007, Steven Grimm wrote:\n> Being able to specify relationships between commits after the fact seems \n> like a very useful facility.\n> \n> Does it make sense to have type information to record what the \n> relationship between two objects means? Without that, it seems like \n> it'll be hard to build much of a tool set on top of this feature, since \n> no two tools that made use of it could unambiguously query just their \n> own softrefs.\n\nActually MadCoder/Pierre had a similar idea on IRC. He wanted to separate \nsoftrefs into namespaces, so that softrefs for tags could live in a \ndifferent place than softrefs associated with his \"gits\" bug tracker.\n\nI haven't thought very much about this, but it's certainly possible to do \nsomething like this. What do the rest of y'all think?\n\n> A few use cases for relationships-after-the-fact come to mind in \n> addition to the one the patch itself mentions:\n> \n> A facility like this could replace the info/grafts file, or at least \n> provide another way to turn a regular commit into a merge commit. Just \n> put a \"manually specified merge parent\" ref between the target revision \n> and the one you want git to think you've merged from. That would scale a \n> lot better than info/grafts does, I suspect, if only by virtue of being \n> O(log n) searchable thanks to the sorting.\n\nYes, I _knew_ this was similar to grafts in some way :) While working on \nthis, I tried to see if I could leverage grafts somewhere in my design, but \nI found them to be too commit-bound and specific. But when you look at it \nthe other way it seems to make more sense.\n\n> One could easily imagine recording a \"cherry picked\" softref, which \n> could, e.g., be the rebase machinery to skip over an already-applied \n> revision. IMO the lack of any tool-readable history about cherry picking \n> -- which is, after all, a sort of merge, at least conceptually -- is a \n> shortcoming in present-day git. (Not a huge one, but if nothing else \n> it'd be great to see cherry picking represented in, e.g., the gitk \n> history display.)\n> \n> It might be possible to annotate rebases to make pulling from rebased \n> branches less troublesome. If you have\n> \n> A--B--C--D\n> \\\n> -E--F--G\n> \n> and you rebase E onto D, a \"rebased from\" softref could be recorded \n> between E and E':\n> \n> A--B--C--D\n> \\ \\\n> -E....E'--F'--G'\n> \n> Then a pulling client could potentially use that information to cleanly \n> replay the rebase operation to keep its history straight. Perhaps if you \n> could record historical rebases like that, you could do away with the \n> current gotchas involving rebasing shared repositories. One negative \n> side effect would be that you'd end up needing to keep E around where \n> before you'd have been able to throw it away, but it should delta \n> compress well, and you can, I think, still prune revisions F and G in \n> the above picture. Or maybe it's enough to just keep E's SHA1 around \n> without actually retaining its contents.\n\nWhoa. I hadn't even imagined this, but I guess you're right. I actually \nthought about solving the same problem (using a much worse method) way back \nin May [1], but I'd since totally forgotten about it. \n\n> But in any event, this seems like the start of a useful new set of \n> capabilities for git.\n\nThanks a lot for sharing your ideas. :)\n\n\nHave fun!\n\n...Johan\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/46137\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44514","messageId":"7vk5ucd6of.fsf_-_@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"200706092019.13185.johan@herland.net","subject":"Comment on weak refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-09T23:55:28Z","receivedAt":"2007-06-09T23:55:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The patch series to look-up and maintain \"softref\" relationship\nis trivially clean. Although I probably would have many nits to\npick, I do not think it is _wrong_ in a major way per-se. I\nwould not even mind saying that I liked the basic concept, until\nI thought things a bit deeper.\n\nHere are some initial notes I took while reading your patches.\n\n\nSemantics\n---------\n\nNot all \"softref\" relationship is equal. \"This object is\nreferred to by these tags\" is one obvious application, and only\nbecause we already try to follow tags when git-fetch happens\nanyway, it looks natural to make everybody follow such a softref\nrelationship.\n\nHowever, as Pierre Habouzit wants to, we may want to make a bug\ntracking sheet (the details of the implementation of such a bug\ntracking sheet does not matter in this discussion -- it could be\na blob, a commit, or a tag) refer to commits using this\nmechanism, by pointing at the blob from commits after the fact\n(i.e. \"later it was verified that this commit fixes the bug\ndescribed in this bug entry\").\n\n\tSide note: I am assuming a simplest implementation where\n\tone blob would always capture the latest status of the\n\tbug. refs/bugs/12127 would point at the latest version\n\tof bug 12127's tracking sheet. An alternative\n\timplementation would be to represent each entry of the\n\ttracking sheet for a single bug as a blob, and have\n\tmultiple blobs associated to a commit on the main\n\tproject, or the other way around, but then you would\n\tneed a way to give order between referers to a single\n\treferent, which I do not find in your proposed\n\t\"softref\".\n\nMost users who want to download and compile the main project do\nnot care about bug tracker objects. You would need to have a\nway to describe what kind of relationship a particular softlink\nrepresents, and adjust the definition of reachability for the\npurposes of traversal of objects.\n\nIt gets worse when you actually start using softrefs. I do not\nthink you would have a limited set of softrefs, such as\n\"reverse-tag-lookup-softref\", \"bug-tracker-softref\". For\nexample, a typical bug tracking sheet may look like this:\n\n - Hey I found a bug, you can reproduce like so... I am\n testing commit _A_.\n - It appears that commit _B_ introduced it; it does not\n reproduce with anything older.\n -\tHere is a patch to fix it; please try.\n - Oh, it seems to fix. Committed as _C_.\n - No, it does not work for me, and the approach to fix is\n wrong; reopening the bug.\n - Ok, take 2 with different approach. Committed as _D_.\n \tplease try.\n - Thanks, it finally fixes it. Closing the bug.\n\nThe bug will be associated with commits A, B, C and D. The\nquestions maintainers would want to ask are:\n\n - What caused this bug?\n - Which versions (or earlier) have this bug?\n - Which versions (or later) have proper fix?\n - What alternate approaches were taken to fix this bug?\n - In this upcoming release, which bugs have been fixed?\n - What bugs are still open after this release?\n\nDepending on what you want to find out, you would need to ask\nwhich commits are related to this bug tracking sheet object, and\nthe answer has to be different. Some \"softref\" relation should\nextend to its ancestry (when \"this fixes\" is attached to a\ncommit, its children ought to inherit that property), some\nshouldn't (\"this is what broke it\" should not propagate to its\nparent nor child).\n\nIt is also unclear how relationship \"softref\" introduces is\npropagated across repositories (not objects the softref binds,\nbut the fact that such a binding between two objects exists need\nto be propagated). I would imagine that your assumption is\nsimply \"to take union\". IOW, if you say A refers to B and\ntransfer object A to the other side, in addition to transfering\nobject B (if the other side does not have it yet), you would\ntell the other side that B is related to A and have the other\nside add that to its set of softrefs. Techinically it is a\nsimple and easy to implement semantics, but I suspect that would\nnot necessarily be useful in practice. Maybe two people would\ndisagree if A and B are related or not. Maybe you first think A\nand B are related and then later change your mind. Should\n\"softref\" relationships be versioned?\n\n\nReachability\n------------\n\nThe association brought in between referent and referer by\nsoftref is \"weak\", in that referer needs to exist only if\nreferent need to be there. This has the following\nconsequences.\n\nFsck/prune/lost-found\n.....................\n\nThe current object traversal starts from \"known tip objects\"\n(i.e. refs, HEAD, index, and reflog entries when not doing\nlost-found) and follows the reachability link embedded in\nreferer objects (i.e. tag to tagged object, commit to tree, tree\nto tree and blob). We only need to extend this \"reachability\"\nwith softref. If a referer object refers to another object via\na softref, we consider referent reachable and we are done.\n\nThis should be reasonably straightforward, except that we\nprobably would need to worry about circular references that\nsoftlink makes possible.\n\npush/fetch/rev-list --objects\n.............................\n\nWe walk the revision range (object transfers essentially starts\ntraversal from the tips of the sender until it meets what the\nreceiver already has), enumerating the reachable objects. I\nsuspect that adding reachability with softref to this scheme has\nconsequences on performance.\n\nImagine:\n\n\tA---B---C---D---E\n\nThe sender's tip is at E and the receiver claims to have C. In\nthe sender's repository, E is associated with A (somebody\nnoticed that E fixes regression introduced by A, and added a\nsoftlink to make A reachable from E). Currently we only need to\nknow C is reachable from E to decide that we do not have to send\nA again, but with softlink we would need to.\n\nThe ancestry chain of referent and referer do not have to share\nany common commits. Imagine a bug tracking system where each\nbug's tracking sheet is represented as a DAG of commits (this\nwill allow you to merge and split bugs easily). This history\nwould not share any tree nor blob with the history of the source\ncode of the project. And you would make a commit in the main\nproject associated with objects in the bug tracking project\nusing softrefs. As sender and receiver exchanges the commit\nancestry information on the main project, both ends may need to\nnegotiate which objects in the bug tracking history are already\npresent in the reciever.\n\nOne attractive point of softref is that you do not have to\nanchor referents with explicit refs. E.g. if a commit in the\nmain project is associated with bug tracking entries in the \"bug\ntracking\" project via softrefs, that is enough to keep the bug\ntracking objects alive. But I suspect this property makes the\nenumeration of \"what do we have on this end\" costly. I dunno.\n\nCome to think of it, and this is off topic for \"softref\", I\nthink using an isolated commit DAG for each bug is probably a\nvery natural way to implement a bug tracking system. If you\nwant to refer to commits on the mainline, you can refer to them\nby their object names, just like mainline commit log messages\nwould refer to earlier commits in the text (e.g. \"This fixes the\nregression introduced by commit ABC\"). The new text entry for\ntraditional BTS will go to the commit message (a bug \"project\"\ncommit does not have to have anything in its tree), and you can\nuse \"git log\" to view what the ordinary BTS would provide, and\nit will distribute the tracking. You do not necessarily\nneed/benefit from \"softref\" for this, though. It could be that\nsuch a bug \"project\" commit might have a commit from the main\nproject in its tree (it would look like a subproject) whose tree\nentry name may be something like \"fixes/yyyy-mm-yy-author\".\n"},{"id":"44522","messageId":"200706100325.32846.johan@herland.net","threadId":"8421","inReplyTo":"7vk5ucd6of.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: Comment on weak refs","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-10T01:25:32Z","receivedAt":"2007-06-10T01:25:32Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 10 June 2007, Junio C Hamano wrote:\n> The patch series to look-up and maintain \"softref\" relationship\n> is trivially clean. Although I probably would have many nits to\n> pick, I do not think it is _wrong_ in a major way per-se. I\n> would not even mind saying that I liked the basic concept, until\n> I thought things a bit deeper.\n> \n> Here are some initial notes I took while reading your patches.\n> \n> \n> Semantics\n> ---------\n> \n> Not all \"softref\" relationship is equal. \"This object is\n> referred to by these tags\" is one obvious application, and only\n> because we already try to follow tags when git-fetch happens\n> anyway, it looks natural to make everybody follow such a softref\n> relationship.\n> \n> However, as Pierre Habouzit wants to, we may want to make a bug\n> tracking sheet (the details of the implementation of such a bug\n> tracking sheet does not matter in this discussion -- it could be\n> a blob, a commit, or a tag) refer to commits using this\n> mechanism, by pointing at the blob from commits after the fact\n> (i.e. \"later it was verified that this commit fixes the bug\n> described in this bug entry\").\n> \n> \tSide note: I am assuming a simplest implementation where\n> \tone blob would always capture the latest status of the\n> \tbug. refs/bugs/12127 would point at the latest version\n> \tof bug 12127's tracking sheet. An alternative\n> \timplementation would be to represent each entry of the\n> \ttracking sheet for a single bug as a blob, and have\n> \tmultiple blobs associated to a commit on the main\n> \tproject, or the other way around, but then you would\n> \tneed a way to give order between referers to a single\n> \treferent, which I do not find in your proposed\n> \t\"softref\".\n> \n> Most users who want to download and compile the main project do\n> not care about bug tracker objects. You would need to have a\n> way to describe what kind of relationship a particular softlink\n> represents, and adjust the definition of reachability for the\n> purposes of traversal of objects.\n\nYes, I'm starting to see that it's not a good idea to put _all_ softrefs in \none bag.\n\n> It gets worse when you actually start using softrefs. I do not\n> think you would have a limited set of softrefs, such as\n> \"reverse-tag-lookup-softref\", \"bug-tracker-softref\". For\n> example, a typical bug tracking sheet may look like this:\n> \n> - Hey I found a bug, you can reproduce like so... I am\n> testing commit _A_.\n> - It appears that commit _B_ introduced it; it does not\n> reproduce with anything older.\n> -\tHere is a patch to fix it; please try.\n> - Oh, it seems to fix. Committed as _C_.\n> - No, it does not work for me, and the approach to fix is\n> wrong; reopening the bug.\n> - Ok, take 2 with different approach. Committed as _D_.\n> \tplease try.\n> - Thanks, it finally fixes it. Closing the bug.\n> \n> The bug will be associated with commits A, B, C and D. The\n> questions maintainers would want to ask are:\n> \n> - What caused this bug?\n> - Which versions (or earlier) have this bug?\n> - Which versions (or later) have proper fix?\n> - What alternate approaches were taken to fix this bug?\n> - In this upcoming release, which bugs have been fixed?\n> - What bugs are still open after this release?\n> \n> Depending on what you want to find out, you would need to ask\n> which commits are related to this bug tracking sheet object, and\n> the answer has to be different. Some \"softref\" relation should\n> extend to its ancestry (when \"this fixes\" is attached to a\n> commit, its children ought to inherit that property), some\n> shouldn't (\"this is what broke it\" should not propagate to its\n> parent nor child).\n\nWe're getting a little ahead of ourselves, aren't we? IMHO, it would be up \nto the bug system to determine which (and how many) connections to make \nbetween the bug reports and the commits (or even if softrefs would be the \ncorrect mechanism for these connections at all). We shouldn't necessarily \nbase the softrefs design on how we imagine a hypothetical bug system to \nwork. But Pierre might have something to say on how he would want to use \nsoftrefs, and his system is hopefully _less_ hypothetical. :)\n\nBut I can see the use of letting the user/porcelain/bugtracker define \nclasses/namespaces of softrefs (at runtime).\n\n> It is also unclear how relationship \"softref\" introduces is\n> propagated across repositories (not objects the softref binds,\n> but the fact that such a binding between two objects exists need\n> to be propagated). I would imagine that your assumption is\n> simply \"to take union\". IOW, if you say A refers to B and\n> transfer object A to the other side, in addition to transfering\n> object B (if the other side does not have it yet), you would\n> tell the other side that B is related to A and have the other\n> side add that to its set of softrefs. Techinically it is a\n> simple and easy to implement semantics, but I suspect that would\n> not necessarily be useful in practice. Maybe two people would\n> disagree if A and B are related or not.\n\nYes, I see that different classes of softrefs would have different semantics \nfor propagation. we could probably try to set up some sane defaults, and \nthen let users put rules in their configs for how they would want to \npropagate the various softrefs classes.\n\n> Maybe you first think A \n> and B are related and then later change your mind. Should\n> \"softref\" relationships be versioned?\n\nIntriguing idea. Not immediately sure how we would implement it though...\n\n> Reachability\n> ------------\n> \n> The association brought in between referent and referer by\n> softref is \"weak\", in that referer needs to exist only if\n> referent need to be there. This has the following\n> consequences.\n>\n> Fsck/prune/lost-found\n> .....................\n> \n> The current object traversal starts from \"known tip objects\"\n> (i.e. refs, HEAD, index, and reflog entries when not doing\n> lost-found) and follows the reachability link embedded in\n> referer objects (i.e. tag to tagged object, commit to tree, tree\n> to tree and blob). We only need to extend this \"reachability\"\n> with softref. If a referer object refers to another object via\n> a softref, we consider referent reachable and we are done.\n\nAgreed.\n\n> This should be reasonably straightforward, except that we\n> probably would need to worry about circular references that\n> softlink makes possible.\n\nIsn't there a .used flag on objects we could easily check to see if we've \nseen an object before, thus preventing us from following the circular \nreference?\n\n> push/fetch/rev-list --objects\n> .............................\n> \n> We walk the revision range (object transfers essentially starts\n> traversal from the tips of the sender until it meets what the\n> receiver already has), enumerating the reachable objects. I\n> suspect that adding reachability with softref to this scheme has\n> consequences on performance.\n> \n> Imagine:\n> \n> \tA---B---C---D---E\n> \n> The sender's tip is at E and the receiver claims to have C. In\n> the sender's repository, E is associated with A (somebody\n> noticed that E fixes regression introduced by A, and added a\n> softlink to make A reachable from E). Currently we only need to\n> know C is reachable from E to decide that we do not have to send\n> A again, but with softlink we would need to.\n\nHmm. First of all, I'm not sure it would be useful to add a _direct_ link \nbetween E and A, but even so...\nI'm thinking we can do the regular/current reachability calculation first, \nand after it's done, we analyze the softrefs to see if there are more \nobjects to be fetched. In that scenario, we wouldn't need to send A again, \nsince it's already in our repo.\n\n> The ancestry chain of referent and referer do not have to share\n> any common commits. Imagine a bug tracking system where each\n> bug's tracking sheet is represented as a DAG of commits (this\n> will allow you to merge and split bugs easily). This history\n> would not share any tree nor blob with the history of the source\n> code of the project. And you would make a commit in the main\n> project associated with objects in the bug tracking project\n> using softrefs. As sender and receiver exchanges the commit\n> ancestry information on the main project, both ends may need to\n> negotiate which objects in the bug tracking history are already\n> present in the reciever.\n\nAs above, if the receving side gets the list of involved softrefs, it can \nmake the decision on which objects it needs to fetch.\n\nHmm. Thinking about it, this process would of course need to be recursive, \nwhich could potentially adversely affect the runtime of fetch...\n\n> One attractive point of softref is that you do not have to\n> anchor referents with explicit refs. E.g. if a commit in the\n> main project is associated with bug tracking entries in the \"bug\n> tracking\" project via softrefs, that is enough to keep the bug\n> tracking objects alive. But I suspect this property makes the\n> enumeration of \"what do we have on this end\" costly. I dunno.\n\nYeah, there are still may open questions. But I'm glad to see that most \npeople seem to find the basic concept useful, at least.\n\n\nThanks for taking the time and effort to comment.\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44527","messageId":"Pine.LNX.4.64.0706100716480.4059@racer.site","threadId":"8421","inReplyTo":"200706100325.32846.johan@herland.net","subject":"Re: Comment on weak refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-10T06:33:17Z","receivedAt":"2007-06-10T06:33:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 10 Jun 2007, Johan Herland wrote:\n\n> On Sunday 10 June 2007, Junio C Hamano wrote:\n>\n> > Maybe you first think A and B are related and then later change your \n> > mind. Should \"softref\" relationships be versioned?\n> \n> Intriguing idea. Not immediately sure how we would implement it \n> though...\n\nHas my lightweight annotation patch reached you?\n\nI like my approach better than yours, because it is\n\n1) a way, way smaller patch, and\n2) it automatically includes the versionability.\n\nAfter thinking about it a little more (my plane was slow, and as a result \nI am allowed to spend 8 more hours in Paris), I think that a small but \ncrucial change would make this thing even more useful:\n\nInstead of having \"core.showAnnotations\" be a boolean config, it might be \nbetter to have \"core.annotationsRef\" instead, overrideable by the \nenvironment variable GIT_ANNOTATION_REF.\n\nWith this, you can have different refs for different kinds of annotations.\n\nFor example, some people might add bugtracker comments (even comments like \n\"this commit was bad: introduced bug #798, solved by commit 9899fdadc..\"). \nThose comments could live in refs/annotations/bugs. To see them, just say \n\n\tGIT_ANNOTATION_REF=refs/annotations/bugs gitk\n\nVoila.\n\nI am quite certain that treating annotations as branches, containing \nfan-out directories for the reverse lookup. I am even quite certain that \nin most cases, a working-directory-less merging is possible for such \nannotations.\n\nCiao,\nDscho\n"},{"id":"44531","messageId":"Pine.LNX.4.64.0706100750440.4059@racer.site","threadId":"8421","inReplyTo":"200706092021.43314.johan@herland.net","subject":"Re: [PATCH 1/7] Softrefs: Add softrefs header file with API documentation","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-10T06:58:49Z","receivedAt":"2007-06-10T06:58:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 9 Jun 2007, Johan Herland wrote:\n\n> See patch for documentation.\n\nThis is preposterous. Either you substitute the patch for a documentation, \nor you document it in the commit message. I consider commit messages like \n\"See patch for documentation\" as reasonable as all those CVS \"** no \nmessage **\" abominations.\n\n> + * The softrefs db consists of two files: .git/softrefs.unsorted and\n> + * .git/softrefs.sorted. Both files use the same format; one softref per line\n> + * of the form \" \\n\". Each sha1 sum is 40 bytes long; this\n> + * makes each entry exactly 82 bytes long (including the space between the sha1 + * sums and the terminating linefeed).\n> + *\n> + * The entries in .git/softrefs.sorted are sorted on , in order to\n> + * make lookup fast.\n> + *\n> + * The entries in .git/softrefs.unsorted are _not_ sorted. This is to make\n> + * insertion fast.\n\nThis sure sounds like you buy the disadvantages of both. Last time I \nchecked, it was recommended to look at your needs and pick _one_ \nappropriate data structure fitting _all_ your needs.\n\nBesides, your lines are way too long. Yes, it is not in \nDocumentation/SubmittingPatches, but even just a cursory look into the \nexisting source shows you that it is mostly 80-chars-per-line. I think it \ngoes without saying that you should try to imitate the existing practices \nin any project, and since you have to read the source to get acquainted \nwith it _anyway_, it would only be a duplication to have it in \nSubmittingPatches, too.\n\nCiao,\nDscho\n"},{"id":"44536","messageId":"7vzm389rvl.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"Pine.LNX.4.64.0706100750440.4059@racer.site","subject":"Re: [PATCH 1/7] Softrefs: Add softrefs header file with API documentation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-10T07:43:26Z","receivedAt":"2007-06-10T07:43:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Besides, your lines are way too long. Yes, it is not in \n> Documentation/SubmittingPatches,...\n> ... since you have to read the source to get acquainted \n> with it _anyway_, it would only be a duplication to have it in \n> SubmittingPatches, too.\n\nWell, maybe we should do this.\n\ndiff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches\nindex 01354c2..4bdfdfe 100644\n--- a/Documentation/SubmittingPatches\n+++ b/Documentation/SubmittingPatches\n@@ -5,6 +5,7 @@ Checklist (and a short version for the impatient):\n \t- make commits of logical units\n \t- check for unnecessary whitespace with \"git diff --check\"\n \t before committing\n+\t- tab width is 8, the terminal is 80-columns wide.\n \t- do not check in commented out code or unneeded files\n \t- provide a meaningful commit message\n \t- the first line of the commit message should be a short\n@@ -82,6 +83,14 @@ option).\n Another thing: NULL pointers shall be written as NULL, not as 0.\n \n \n+(1b) Tab width is 8, Terminal is 80-column wide.\n+\n+We generally follow the same coding style guidelines as the\n+Linux kernel project. Lines are indented with Tabs, each of\n+which are 8 columns wide. Lines should fit on 80-column wide\n+terminals.\n+\n+\n (2) Generate your patch using git tools out of your commits.\n \n git based diff tools (git, Cogito, and StGIT included) generate\n"},{"id":"44541","messageId":"Pine.LNX.4.64.0706100851440.4059@racer.site","threadId":"8421","inReplyTo":"7vzm389rvl.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/7] Softrefs: Add softrefs header file with API documentation","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-10T07:54:32Z","receivedAt":"2007-06-10T07:54:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 10 Jun 2007, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > Besides, your lines are way too long. Yes, it is not in \n> > Documentation/SubmittingPatches,...\n> > ... since you have to read the source to get acquainted \n> > with it _anyway_, it would only be a duplication to have it in \n> > SubmittingPatches, too.\n> \n> Well, maybe we should do this.\n>\n> [...]\n\nBut where to stop?\n\nMany people want to put an opening curly bracket in its own line. Other \nindenting is subject for discussion, too. White space after operators, but \nnot after function names should be included, too.\n\nI know you mean good, but I think it is not a bad idea to let people get \nfamiliar with the code (and the formatting rules) first. This way we can \neven tell who did, and who did not do that, before submitting a patch.\n\nCiao,\nDscho\n"},{"id":"44542","messageId":"Pine.LNX.4.64.0706100900420.4059@racer.site","threadId":"8421","inReplyTo":"200706092025.30156.johan@herland.net","subject":"Re: [PATCH] Change softrefs file format from text (82 bytes per entry) to binary (40 bytes per entry)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-10T08:02:11Z","receivedAt":"2007-06-10T08:02:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 9 Jun 2007, Johan Herland wrote:\n\n> The text-based softrefs file format uses 82 bytes per entry (40 bytes \n> from_sha1 in hex, 1 byte SP, 40 bytes to_sha1 in hex, 1 byte LF).\n> \n> The binary softrefs file format uses 40 bytes per entry (20 bytes \n> from_sha1, 20 bytes to_sha1).\n> \n> Moving to a binary format increases performance slightly, but sacrifices \n> easy readability of the softrefs files.\n\nIt is bad style to introduce one type, and then change it to another in a \nbackwards-incompatible way. Either you make it backwards compatible, or \nyou start with the second format, never even mentioning that you had \nanother format.\n\nCiao,\nDscho\n"},{"id":"44548","messageId":"20070610082935.GD2951@artemis","threadId":"8421","inReplyTo":"200706100116.46062.johan@herland.net","subject":"Re: Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2007-06-10T08:29:35Z","receivedAt":"2007-06-10T08:29:35Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Sun, Jun 10, 2007 at 01:16:45AM +0200, Johan Herland wrote:\n> On Sunday 10 June 2007, Steven Grimm wrote:\n> > Being able to specify relationships between commits after the fact seems \n> > like a very useful facility.\n> > \n> > Does it make sense to have type information to record what the \n> > relationship between two objects means? Without that, it seems like \n> > it'll be hard to build much of a tool set on top of this feature, since \n> > no two tools that made use of it could unambiguously query just their \n> > own softrefs.\n> \n> Actually MadCoder/Pierre had a similar idea on IRC. He wanted to separate \n> softrefs into namespaces, so that softrefs for tags could live in a \n> different place than softrefs associated with his \"gits\" bug tracker.\n> \n> I haven't thought very much about this, but it's certainly possible to do \n> something like this. What do the rest of y'all think?\n\n Well, if we're two with the same idea, it's a good one, no ? :)\n\n In fact, the namespace idea like I told you on IRC isn't _that_\nbrilliant. But I'm sure recording a softref with:\n\n \n\n token would help classify the softref. And I'm sure we'll end up with:\n\n \n\n with the flags to say what behaviour (e.g.) the reachability resolver\nshould have wrt that link ?\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"44549","messageId":"7vir9w8b4a.fsf@assigned-by-dhcp.cox.net","threadId":"8421","inReplyTo":"Pine.LNX.4.64.0706100900420.4059@racer.site","subject":"Re: [PATCH] Change softrefs file format from text (82 bytes per entry) to binary (40 bytes per entry)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-06-10T08:30:45Z","receivedAt":"2007-06-10T08:30:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> On Sat, 9 Jun 2007, Johan Herland wrote:\n>\n>> The text-based softrefs file format uses 82 bytes per entry (40 bytes \n>> from_sha1 in hex, 1 byte SP, 40 bytes to_sha1 in hex, 1 byte LF).\n>> \n>> The binary softrefs file format uses 40 bytes per entry (20 bytes \n>> from_sha1, 20 bytes to_sha1).\n>> \n>> Moving to a binary format increases performance slightly, but sacrifices \n>> easy readability of the softrefs files.\n>\n> It is bad style to introduce one type, and then change it to another in a \n> backwards-incompatible way. Either you make it backwards compatible, or \n> you start with the second format, never even mentioning that you had \n> another format.\n\nWhile I agree with that in principle, I think you are being a\nbit too harsh to a set of patches that shows possible\nalternatives for an idea that is not even in any unreleased\nversion of git.\n\nGot out of the wrong side of bed this morning?\n"},{"id":"44564","messageId":"20070610090350.GE2951@artemis","threadId":"8421","inReplyTo":"200706100325.32846.johan@herland.net","subject":"Re: Comment on weak refs","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2007-06-10T09:03:50Z","receivedAt":"2007-06-10T09:03:50Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Sun, Jun 10, 2007 at 03:25:32AM +0200, Johan Herland wrote:\n> On Sunday 10 June 2007, Junio C Hamano wrote:\n> > It gets worse when you actually start using softrefs. I do not\n> > think you would have a limited set of softrefs, such as\n> > \"reverse-tag-lookup-softref\", \"bug-tracker-softref\". For\n> > example, a typical bug tracking sheet may look like this:\n> > \n> > - Hey I found a bug, you can reproduce like so... I am\n> > testing commit _A_.\n> > - It appears that commit _B_ introduced it; it does not\n> > reproduce with anything older.\n> > -\tHere is a patch to fix it; please try.\n> > - Oh, it seems to fix. Committed as _C_.\n> > - No, it does not work for me, and the approach to fix is\n> > wrong; reopening the bug.\n> > - Ok, take 2 with different approach. Committed as _D_.\n> > \tplease try.\n> > - Thanks, it finally fixes it. Closing the bug.\n> > \n> > The bug will be associated with commits A, B, C and D. The\n> > questions maintainers would want to ask are:\n> > \n> > - What caused this bug?\n> > - Which versions (or earlier) have this bug?\n> > - Which versions (or later) have proper fix?\n> > - What alternate approaches were taken to fix this bug?\n> > - In this upcoming release, which bugs have been fixed?\n> > - What bugs are still open after this release?\n> > \n> > Depending on what you want to find out, you would need to ask\n> > which commits are related to this bug tracking sheet object, and\n> > the answer has to be different. Some \"softref\" relation should\n> > extend to its ancestry (when \"this fixes\" is attached to a\n> > commit, its children ought to inherit that property), some\n> > shouldn't (\"this is what broke it\" should not propagate to its\n> > parent nor child).\n> \n> We're getting a little ahead of ourselves, aren't we? IMHO, it would be up \n> to the bug system to determine which (and how many) connections to make \n> between the bug reports and the commits (or even if softrefs would be the \n> correct mechanism for these connections at all). We shouldn't necessarily \n> base the softrefs design on how we imagine a hypothetical bug system to \n> work. But Pierre might have something to say on how he would want to use \n> softrefs, and his system is hopefully _less_ hypothetical. :)\n\n To be fair, I'm still struggling with the storage backend yet, trying\nto make things fast enough (My current import rate of mails is 10 per\nsecond, wich is not that brilliant I guess), and also to design some\nsimple things like \"answering\" to a bug.\n\n For now, my design is the following, I've a 'bts' branch where the\nbugs reports (plain mailboxes) go. Grit is able to manage as many branch\nthe user wants, bts is just the default name for it. Then, for a bts\nbranch, you have $GIT_DIR/grit/.index and\n$GIT_DIR/grit//. The former is the index of the tip of the bts\nbranch, and the latter contains some bits of the tip of the branch\ncheckouted (can be seen as some kind of cache, useful to run mutt -f on\na mbox e.g.).\n\n Bugs have the sha id of the hash of the first imported mail, and are\nput in sha[:2]/sha[2:] files, à la .git/objects/. I also should have a\nsecond file with annotations about the bug, format not really clear for\nnow, as \"one file per bug\" could be quite inefficient. OTOH if I mix too\nmany bugs in the same file, the merge risk is bigger (but I suppose I\ncould use a specific merge strategy on this).\n\n Here is the sole non hypothetical thing yet. My plans then was to use\n\"links\" (softlinks or not, I'm speaking generically, I hope softrefs\nwill match my needs, I don't know yet) between specific commits, and\nbugs. Links would somehow carry information on wether this is an\n\"opening\" tag (like: this bug is present starting at that commit), an\ninformationnal tag (like: this commit helps fixing that bug, but is not\nenough), or a closing/fixing tag (like: this commit fixes it). A fourth\nkind may be also used aka a not-found tag (like: well this commit does\nnot fixes the bug, but for sure it's not there anymore at that commit).\n\n Though, softlinks do not need to \"carry\" the information for real,\nthey just need to be linked somehow to the bug, bug that would have the\nannotations for those softlinks in them.\n\n What is somehow flawed for me, is that when someone \"answers\" to the\nbug or changes a bit of information about it, it generate a \"new\"\ncommit, and I would need to move the softlinks to the new commit object\nit generated to shorten the path and go directly to the last version of\nthe bug status file.\n\n So to be of use for me, yes, I guess I would really like the\nversionning of softlinks. If I use them at all, I don't know yet.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"44570","messageId":"Pine.LNX.4.64.0706101022210.4059@racer.site","threadId":"8421","inReplyTo":"7vir9w8b4a.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Change softrefs file format from text (82 bytes per entry) to binary (40 bytes per entry)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-10T09:46:09Z","receivedAt":"2007-06-10T09:46:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 10 Jun 2007, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > On Sat, 9 Jun 2007, Johan Herland wrote:\n> >\n> >> The text-based softrefs file format uses 82 bytes per entry (40 bytes \n> >> from_sha1 in hex, 1 byte SP, 40 bytes to_sha1 in hex, 1 byte LF).\n> >> \n> >> The binary softrefs file format uses 40 bytes per entry (20 bytes \n> >> from_sha1, 20 bytes to_sha1).\n> >> \n> >> Moving to a binary format increases performance slightly, but sacrifices \n> >> easy readability of the softrefs files.\n> >\n> > It is bad style to introduce one type, and then change it to another in a \n> > backwards-incompatible way. Either you make it backwards compatible, or \n> > you start with the second format, never even mentioning that you had \n> > another format.\n> \n> While I agree with that in principle, I think you are being a\n> bit too harsh to a set of patches that shows possible\n> alternatives for an idea that is not even in any unreleased\n> version of git.\n> \n> Got out of the wrong side of bed this morning?\n\nPossibly. Except it was not a bed, but an airplane passenger seat.\n\nAnd it did not help that I totally disagree with the approach: \"sorted \nlist does not do this well, unsorted not that... let's take both!\"\n\nSo, please take my words with a grain of salt. But I still think that \n_what_ I was saying is correct.\n\nYes, the patch series shows an alternative, but I would have wished for \neither a smaller quick-n-dirty proof-of-concept implementation ([RFC]), or \na better thought-through one ([PATCH]).\n\nCiao,\nDscho\n"},{"id":"44602","messageId":"200706101541.44362.johan@herland.net","threadId":"8421","inReplyTo":"Pine.LNX.4.64.0706100716480.4059@racer.site","subject":"Re: Comment on weak refs","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-10T13:41:44Z","receivedAt":"2007-06-10T13:41:44Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 10 June 2007, Johannes Schindelin wrote:\n> On Sun, 10 Jun 2007, Johan Herland wrote:\n> > On Sunday 10 June 2007, Junio C Hamano wrote:\n> Has my lightweight annotation patch reached you?\n> \n> I like my approach better than yours, because it is\n> \n> 1) a way, way smaller patch, and\n> 2) it automatically includes the versionability.\n\nI see your point, but your lightweight annotations are solving a different \nproblem, aren't they? They do provide the after-the-fact annotations that \nsort of sparked of these discussions, but I can't see how your patch is a \nreplacement of the general \"relationships between arbitrary objects\" \nconcept that softrefs try to solve.\n\nOf course, it might be that the lightweight annotations are \"good enough\" \nfor the use cases we currently see, and that softrefs are a bit overkill. \nWe'll just have to see what features people (like Pierre) really need.\n\n> After thinking about it a little more (my plane was slow, and as a result \n> I am allowed to spend 8 more hours in Paris), I think that a small but \n> crucial change would make this thing even more useful:\n> \n> Instead of having \"core.showAnnotations\" be a boolean config, it might be \n> better to have \"core.annotationsRef\" instead, overrideable by the \n> environment variable GIT_ANNOTATION_REF.\n> \n> With this, you can have different refs for different kinds of annotations.\n> \n> For example, some people might add bugtracker comments (even comments like \n> \"this commit was bad: introduced bug #798, solved by commit 9899fdadc..\"). \n> Those comments could live in refs/annotations/bugs. To see them, just say \n> \n> \tGIT_ANNOTATION_REF=refs/annotations/bugs gitk\n> \n> Voila.\n\nNice. Something similar should be possible to do with softrefs as well.\n\n> I am quite certain that treating annotations as branches, containing \n> fan-out directories for the reverse lookup. I am even quite certain that \n> in most cases, a working-directory-less merging is possible for such \n> annotations.\n\nI'm not convinced about the working-directory-less merging. AFAICS the \nlightweight annotations will behave pretty much like the \"regular\" version \ncontrolled filesystem, and you'll have the same kind of conflicts when you \nmerge stuff between repos. I'd be glad to be proven wrong, of course.\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44606","messageId":"200706101600.02673.johan@herland.net","threadId":"8421","inReplyTo":"Pine.LNX.4.64.0706100750440.4059@racer.site","subject":"Re: [PATCH 1/7] Softrefs: Add softrefs header file with API documentation","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-10T14:00:02Z","receivedAt":"2007-06-10T14:00:02Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 10 June 2007, Johannes Schindelin wrote:\n> Hi,\n> \n> On Sat, 9 Jun 2007, Johan Herland wrote:\n> \n> > See patch for documentation.\n> \n> This is preposterous. Either you substitute the patch for a documentation, \n> or you document it in the commit message. I consider commit messages like \n> \"See patch for documentation\" as reasonable as all those CVS \"** no \n> message **\" abominations.\n\nWell, I could have copied documentation from the header file into the commit\nmessage\n\n> > + * The softrefs db consists of two files: .git/softrefs.unsorted and\n> > + * .git/softrefs.sorted. Both files use the same format; one softref per line\n> > + * of the form \" \\n\". Each sha1 sum is 40 bytes long; this\n> > + * makes each entry exactly 82 bytes long (including the space between the sha1 + * sums and the terminating linefeed).\n> > + *\n> > + * The entries in .git/softrefs.sorted are sorted on , in order to\n> > + * make lookup fast.\n> > + *\n> > + * The entries in .git/softrefs.unsorted are _not_ sorted. This is to make\n> > + * insertion fast.\n> \n> This sure sounds like you buy the disadvantages of both. Last time I \n> checked, it was recommended to look at your needs and pick _one_ \n> appropriate data structure fitting _all_ your needs.\n\nFirst, the unsorted file is bounded in size to make sure it never gets\nlarge enough to really impact performance\nSecond, I'd ask you to look at the performance numbers (in patch #0)\nbefore commenting on the performance.\n\n> Besides, your lines are way too long. Yes, it is not in \n> Documentation/SubmittingPatches, but even just a cursory look into the \n> existing source shows you that it is mostly 80-chars-per-line. I think it \n> goes without saying that you should try to imitate the existing practices \n> in any project, and since you have to read the source to get acquainted \n> with it _anyway_, it would only be a duplication to have it in \n> SubmittingPatches, too.\n\nI have indeed tried to follow the 80-chars-per-line rule. In softrefs.h\nI fail to see a _single_ line exceeding 80 chars per line. In the other\nfiles I believe the number of long lines is comparable to other files\nin the git repo.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44608","messageId":"200706101603.43027.johan@herland.net","threadId":"8421","inReplyTo":"Pine.LNX.4.64.0706100900420.4059@racer.site","subject":"Re: [PATCH] Change softrefs file format from text (82 bytes per entry) to binary (40 bytes per entry)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-10T14:03:42Z","receivedAt":"2007-06-10T14:03:42Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 10 June 2007, Johannes Schindelin wrote:\n> On Sat, 9 Jun 2007, Johan Herland wrote:\n> > The text-based softrefs file format uses 82 bytes per entry (40 bytes \n> > from_sha1 in hex, 1 byte SP, 40 bytes to_sha1 in hex, 1 byte LF).\n> > \n> > The binary softrefs file format uses 40 bytes per entry (20 bytes \n> > from_sha1, 20 bytes to_sha1).\n> > \n> > Moving to a binary format increases performance slightly, but sacrifices \n> > easy readability of the softrefs files.\n> \n> It is bad style to introduce one type, and then change it to another in a \n> backwards-incompatible way. Either you make it backwards compatible, or \n> you start with the second format, never even mentioning that you had \n> another format.\n\nAs Junio correctly pointed out, this patch is only here to demonstrate an\nalternative solution. Whether or not this patch should be used should be\ndetermined _long_ before (even thinking about) putting this into a release.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44611","messageId":"20070610140859.GB6730@artemis.madism.org","threadId":"8421","inReplyTo":"200706101541.44362.johan@herland.net","subject":"Re: Comment on weak refs","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2007-06-10T14:09:00Z","receivedAt":"2007-06-10T14:09:00Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":" Sorry for the noise, but I'm really pissed with git@vger recently.\nThat mail I'm answering to, never made it to the list. Neither did my\nquite long answer to Martin, who kindly forwared it back to me so that I\ncan send it again, sadly git@vger just does not wants mail, whereas its\nSMTP seems to accept mail:\n\n Jun 10 16:02:09 pan postfix/smtp[20560]: DE84FCAD9: to=, relay=vger.kernel.org[209.132.176.167]:25, delay=4, delays=0.31/0.02/0.5/3.1, dsn=2.7.0, status=sent (250 2.7.0 nothing apparently wrong in the message. BF:; S1754569AbXFJOCJ)\n\n I've sent a mail to postmaster@vger.kernel.org a week ago, but it\nseems it remained a dead letter. Is anyone able to tell what's going\nwrong ? it's _really_ irritating, and let me want to give up\ndiscussions, as I _hate_ losing mail (I usually don't keep a copy of\nmails I send to a mail list as I expect it to send it back to me, and\nwell, what would be a copy worth if nobody can read the mail anyway ?).\n\n So if anyone knows what can be done ....\n\n\nOn Sun, Jun 10, 2007 at 03:41:44PM +0200, Johan Herland wrote:\n> On Sunday 10 June 2007, Johannes Schindelin wrote:\n> > On Sun, 10 Jun 2007, Johan Herland wrote:\n> > > On Sunday 10 June 2007, Junio C Hamano wrote:\n> > Has my lightweight annotation patch reached you?\n> > \n> > I like my approach better than yours, because it is\n> > \n> > 1) a way, way smaller patch, and\n> > 2) it automatically includes the versionability.\n> \n> I see your point, but your lightweight annotations are solving a different \n> problem, aren't they? They do provide the after-the-fact annotations that \n> sort of sparked of these discussions, but I can't see how your patch is a \n> replacement of the general \"relationships between arbitrary objects\" \n> concept that softrefs try to solve.\n> \n> Of course, it might be that the lightweight annotations are \"good enough\" \n> for the use cases we currently see, and that softrefs are a bit overkill. \n> We'll just have to see what features people (like Pierre) really need.\n> \n> > After thinking about it a little more (my plane was slow, and as a result \n> > I am allowed to spend 8 more hours in Paris), I think that a small but \n> > crucial change would make this thing even more useful:\n> > \n> > Instead of having \"core.showAnnotations\" be a boolean config, it might be \n> > better to have \"core.annotationsRef\" instead, overrideable by the \n> > environment variable GIT_ANNOTATION_REF.\n> > \n> > With this, you can have different refs for different kinds of annotations.\n> > \n> > For example, some people might add bugtracker comments (even comments like \n> > \"this commit was bad: introduced bug #798, solved by commit 9899fdadc..\"). \n> > Those comments could live in refs/annotations/bugs. To see them, just say \n> > \n> > \tGIT_ANNOTATION_REF=refs/annotations/bugs gitk\n> > \n> > Voila.\n> \n> Nice. Something similar should be possible to do with softrefs as well.\n> \n> > I am quite certain that treating annotations as branches, containing \n> > fan-out directories for the reverse lookup. I am even quite certain that \n> > in most cases, a working-directory-less merging is possible for such \n> > annotations.\n> \n> I'm not convinced about the working-directory-less merging. AFAICS the \n> lightweight annotations will behave pretty much like the \"regular\" version \n> controlled filesystem, and you'll have the same kind of conflicts when you \n> merge stuff between repos. I'd be glad to be proven wrong, of course.\n> \n> \n> Have fun!\n> \n> ....Johan\n> \n> -- \n> Johan Herland, \n> www.herland.net\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"44613","messageId":"20070610142523.GD6730@artemis.madism.org","threadId":"8421","inReplyTo":"20070610140859.GB6730@artemis.madism.org","subject":"Re: Comment on weak refs","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2007-06-10T14:25:23Z","receivedAt":"2007-06-10T14:25:23Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Sun, Jun 10, 2007 at 04:08:59PM +0200, Pierre Habouzit wrote:\n> Sorry for the noise, but I'm really pissed with git@vger recently.\n> That mail I'm answering to, never made it to the list. Neither did my\n> quite long answer to Martin, who kindly forwared it back to me so that I\n> can send it again, sadly git@vger just does not wants mail, whereas its\n> SMTP seems to accept mail:\n> \n> Jun 10 16:02:09 pan postfix/smtp[20560]: DE84FCAD9: to=, relay=vger.kernel.org[209.132.176.167]:25, delay=4, delays=0.31/0.02/0.5/3.1, dsn=2.7.0, status=sent (250 2.7.0 nothing apparently wrong in the message. BF:; S1754569AbXFJOCJ)\n> \n> I've sent a mail to postmaster@vger.kernel.org a week ago, but it\n> seems it remained a dead letter. Is anyone able to tell what's going\n> wrong ? it's _really_ irritating, and let me want to give up\n> discussions, as I _hate_ losing mail (I usually don't keep a copy of\n> mails I send to a mail list as I expect it to send it back to me, and\n> well, what would be a copy worth if nobody can read the mail anyway ?).\n> \n> So if anyone knows what can be done ....\n\n SOrry, *I* screwed up. I did not checked first if the archived had the\nmails, and it had. So as I don't see the mails come, it's definitely a\nproblem in between, and is not necessarilly vger.kernel.org's fault, and\nI apologies for the noise.\n\n So if anyone sent a mail to the list (without me in Cc) hoping that I\nwould get the mail and never answered, it's time to send it again :/\n\n That is also the reason why some of my mails have been sent many\ntimes: because I thought they were eaten at some point. That should not\nhappen again, sorry about that.\n\nCheers,\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"44614","messageId":"f4h1ot$83m$1@sea.gmane.org","threadId":"8421","inReplyTo":"200706092021.43314.johan@herland.net","subject":"Re: [PATCH 1/7] Softrefs: Add softrefs header file with API documentation","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-06-10T14:27:41Z","receivedAt":"2007-06-10T14:27:41Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johan Herland wrote:\n\n> + * When softrefs are created (by calling add_softref()/add_softrefs()), they\n> + * are appended to .git/softrefs.unsorted. When .git/softrefs.unsorted reach a\n> + * certain number of entries (determined by MAX_UNSORTED_ENTRIES), all the\n> + * entries in .git/softrefs.unsorted are merged into .git/softrefs.sorted.\n\nPerhaps git-gc should also sort softrefs (if it doesn't do it yet)?\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"44615","messageId":"200706101631.27600.johan@herland.net","threadId":"8421","inReplyTo":"20070610082935.GD2951@artemis","subject":"Re: Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-10T14:31:27Z","receivedAt":"2007-06-10T14:31:27Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 10 June 2007, Pierre Habouzit wrote:\n> On Sun, Jun 10, 2007 at 01:16:45AM +0200, Johan Herland wrote:\n> > On Sunday 10 June 2007, Steven Grimm wrote:\n> > > Being able to specify relationships between commits after the fact seems \n> > > like a very useful facility.\n> > > \n> > > Does it make sense to have type information to record what the \n> > > relationship between two objects means? Without that, it seems like \n> > > it'll be hard to build much of a tool set on top of this feature, since \n> > > no two tools that made use of it could unambiguously query just their \n> > > own softrefs.\n> > \n> > Actually MadCoder/Pierre had a similar idea on IRC. He wanted to separate \n> > softrefs into namespaces, so that softrefs for tags could live in a \n> > different place than softrefs associated with his \"gits\" bug tracker.\n> > \n> > I haven't thought very much about this, but it's certainly possible to do \n> > something like this. What do the rest of y'all think?\n> \n> Well, if we're two with the same idea, it's a good one, no ? :)\n> \n> In fact, the namespace idea like I told you on IRC isn't _that_\n> brilliant. But I'm sure recording a softref with:\n> \n> \n> \n> token would help classify the softref. And I'm sure we'll end up with:\n> \n> \n> \n> with the flags to say what behaviour (e.g.) the reachability resolver\n> should have wrt that link ?\n\nInteresting. But I'm not sure I want to give up the fixed-length softref\nrecords as I imagine it makes the lookup and processing _much_ faster.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"44616","messageId":"200706101645.41601.johan@herland.net","threadId":"8421","inReplyTo":"f4h1ot$83m$1@sea.gmane.org","subject":"[PATCH] Teach git-gc to merge unsorted softrefs","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-06-10T14:45:41Z","receivedAt":"2007-06-10T14:45:41Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Signed-off-by: Johan Herland \n---\n\nOn Sunday 10 June 2007, Jakub Narebski wrote:\n> Perhaps git-gc should also sort softrefs (if it doesn't do it yet)?\n\nSure.\n\n builtin-gc.c | 4 ++++\n 1 files changed, 4 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 45025fb..30e1e44 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -12,6 +12,7 @@\n \n #include \"cache.h\"\n #include \"run-command.h\"\n+#include \"softrefs.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -87,6 +88,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (i != argc)\n \t\tusage(builtin_gc_usage);\n \n+\tif (merge_unsorted_softrefs(NULL, 1))\n+\t\treturn error(\"failed to merge unsorted softrefs\");\n+\n \tif (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))\n \t\treturn error(FAILED_RUN, argv_pack_refs[0]);\n \n-- \n1.5.2.1.144.gabc40\n"},{"id":"44619","messageId":"f4h581$htu$1@sea.gmane.org","threadId":"8421","inReplyTo":"7vk5ucd6of.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: Comment on weak refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-06-10T15:26:57Z","receivedAt":"2007-06-10T15:26:57Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> Semantics\n> ---------\n> \n> Not all \"softref\" relationship is equal.  \"This object is\n> referred to by these tags\" is one obvious application, and only\n> because we already try to follow tags when git-fetch happens\n> anyway, it looks natural to make everybody follow such a softref\n> relationship.\n\nOr \"this onject is referred to by these _notes_\", where notes differ\nfrom tags on what is more important: name or comment (message). For\ntags most important is the name, for notes most important is the\ncomment (which might be bug message, but might be correction to\ncommit message, or additional acked-by).\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"44633","messageId":"466C5438.3020600@midwinter.com","threadId":"8421","inReplyTo":"200706101631.27600.johan@herland.net","subject":"Re: Refactoring the tag object; Introducing soft references (softrefs); Git 'notes' (take 2)","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-06-10T19:42:48Z","receivedAt":"2007-06-10T19:42:48Z","isPatch":false,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Johan Herland wrote:\n> Interesting. But I'm not sure I want to give up the fixed-length softref\n> records as I imagine it makes the lookup and processing _much_ faster.\n> \n\nThe token (really a namespace identifier) could be defined as a string \nwith a fixed, probably small, maximum size. Or, better IMO, it could be \nan integer with a bunch of enumerated values for internal git uses and a \nrange reserved for unofficial / experimental use. I agree that \nfixed-length records seem like a win here, assuming this is the general \nstorage layout we end up with when all is said and done.\n\n-Steve\n"}]}