{"thread":{"id":"49486","subject":"[RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer","startedAt":"2018-10-04T23:01:42Z","lastAt":"2018-10-16T06:18:39Z","messageCount":10,"participants":["Josh Steadmon","Junio C Hamano","steadmon@google.com"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"359645","messageId":"cover.1538693039.git.steadmon@google.com","threadId":"49486","inReplyTo":null,"subject":"[RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2018-10-04T23:01:27Z","receivedAt":"2018-10-04T23:01:42Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"libFuzzer[1] is a fuzzing engine included in recent versions of LLVM. It\nis used by OSS-Fuzz[2] for continuous fuzzing of OSS projects.\n\nThis series adds two basic fuzzing targets covering packfile header and\nindex code. It is not particularly portable, and requires the use of\nLLVM v4.0 (the latest version available on my workstation). I would\nparticularly appreciate advice on how to make the Makefile more\nportable.\n\n[1]: https://llvm.org/docs/LibFuzzer.html\n[2]: https://github.com/google/oss-fuzz\n\nJosh Steadmon (2):\n  fuzz: Add basic fuzz testing target.\n  fuzz: Add fuzz testing for packfile indices.\n\n .gitignore          |  3 +++\n Makefile            | 33 ++++++++++++++++++++++++++++++++-\n fuzz-pack-headers.c | 14 ++++++++++++++\n fuzz-pack-idx.c     | 13 +++++++++++++\n packfile.c          | 44 +++++++++++++++++++++++++-------------------\n packfile.h          | 13 +++++++++++++\n 6 files changed, 100 insertions(+), 20 deletions(-)\n create mode 100644 fuzz-pack-headers.c\n create mode 100644 fuzz-pack-idx.c\n\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359646","messageId":"bfe4f2f965528ca2a45864857051b9835970ff07.1538693039.git.steadmon@google.com","threadId":"49486","inReplyTo":"cover.1538693039.git.steadmon@google.com","subject":"[RFC PATCH 1/2] fuzz: Add basic fuzz testing target.","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2018-10-04T23:01:28Z","receivedAt":"2018-10-04T23:01:48Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Signed-off-by: Josh Steadmon <steadmon@google.com>\n---\n .gitignore          |  2 ++\n Makefile            | 30 +++++++++++++++++++++++++++++-\n fuzz-pack-headers.c | 14 ++++++++++++++\n 3 files changed, 45 insertions(+), 1 deletion(-)\n create mode 100644 fuzz-pack-headers.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 9d1363a1eb..87a28b3115 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -1,3 +1,5 @@\n+/fuzz_corpora\n+/fuzz-pack-headers\n /GIT-BUILD-OPTIONS\n /GIT-CFLAGS\n /GIT-LDFLAGS\ndiff --git a/Makefile b/Makefile\nindex 13e1c52478..10bb82b115 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -590,6 +590,8 @@ XDIFF_OBJS =\n VCSSVN_OBJS =\n GENERATED_H =\n EXTRA_CPPFLAGS =\n+FUZZ_OBJS =\n+FUZZ_PROGRAMS =\n LIB_OBJS =\n PROGRAM_OBJS =\n PROGRAMS =\n@@ -682,6 +684,10 @@ SCRIPTS = $(SCRIPT_SH_INS) \\\n \n ETAGS_TARGET = TAGS\n \n+FUZZ_OBJS += fuzz-pack-headers.o\n+\n+FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n+\n # Empty...\n EXTRA_PROGRAMS =\n \n@@ -2250,6 +2256,7 @@ TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(VCSSVN_OBJS) \\\n+\t$(FUZZ_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2931,7 +2938,7 @@ profile-clean:\n cocciclean:\n \t$(RM) contrib/coccinelle/*.cocci.patch*\n \n-clean: profile-clean coverage-clean cocciclean\n+clean: profile-clean coverage-clean cocciclean fuzz-clean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB)\n@@ -3061,3 +3068,24 @@ cover_db: coverage-report\n cover_db_html: cover_db\n \tcover -report html -outputdir cover_db_html cover_db\n \n+\n+### Fuzz testing\n+#\n+.PHONY: fuzz-clean fuzz-objs fuzz-compile\n+\n+FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address\n+FUZZ_LDFLAGS = $(FUZZ_CFLAGS)\n+\n+\n+fuzz-clean:\n+\t$(RM) $(FUZZ_PROGRAMS) $(FUZZ_OBJS)\n+\n+fuzz-objs: $(FUZZ_OBJS)\n+\n+fuzz-compile:\n+\t$(MAKE) CC=clang LD=clang CFLAGS=\"$(FUZZ_CFLAGS)\" \\\n+\t\tLDFLAGS=\"$(FUZZ_LDFLAGS)\" all fuzz-objs\n+\n+$(FUZZ_PROGRAMS): fuzz-compile\n+\tclang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \\\n+\t\t$(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@\ndiff --git a/fuzz-pack-headers.c b/fuzz-pack-headers.c\nnew file mode 100644\nindex 0000000000..99da1d0fd3\n--- /dev/null\n+++ b/fuzz-pack-headers.c\n@@ -0,0 +1,14 @@\n+#include \"packfile.h\"\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n+{\n+\tenum object_type type;\n+\tunsigned long len;\n+\n+\tunpack_object_header_buffer((const unsigned char *)data,\n+\t\t\t\t    (unsigned long)size, &type, &len);\n+\n+\treturn 0;\n+}\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359647","messageId":"60b9daad5c5132811875b6094abf4d87852e4263.1538693039.git.steadmon@google.com","threadId":"49486","inReplyTo":"cover.1538693039.git.steadmon@google.com","subject":"[RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices.","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2018-10-04T23:01:29Z","receivedAt":"2018-10-04T23:01:52Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Signed-off-by: Josh Steadmon <steadmon@google.com>\n---\n .gitignore      |  1 +\n Makefile        |  5 ++++-\n fuzz-pack-idx.c | 13 +++++++++++++\n packfile.c      | 44 +++++++++++++++++++++++++-------------------\n packfile.h      | 13 +++++++++++++\n 5 files changed, 56 insertions(+), 20 deletions(-)\n create mode 100644 fuzz-pack-idx.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 87a28b3115..64b3377d40 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -1,5 +1,6 @@\n /fuzz_corpora\n /fuzz-pack-headers\n+/fuzz-pack-idx\n /GIT-BUILD-OPTIONS\n /GIT-CFLAGS\n /GIT-LDFLAGS\ndiff --git a/Makefile b/Makefile\nindex 10bb82b115..f7e6be8f19 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -685,6 +685,7 @@ SCRIPTS = $(SCRIPT_SH_INS) \\\n ETAGS_TARGET = TAGS\n \n FUZZ_OBJS += fuzz-pack-headers.o\n+FUZZ_OBJS += fuzz-pack-idx.o\n \n FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n \n@@ -3071,7 +3072,7 @@ cover_db_html: cover_db\n \n ### Fuzz testing\n #\n-.PHONY: fuzz-clean fuzz-objs fuzz-compile\n+.PHONY: fuzz-clean fuzz-objs fuzz-compile fuzz-all\n \n FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address\n FUZZ_LDFLAGS = $(FUZZ_CFLAGS)\n@@ -3089,3 +3090,5 @@ fuzz-compile:\n $(FUZZ_PROGRAMS): fuzz-compile\n \tclang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \\\n \t\t$(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@\n+\n+fuzz-all: $(FUZZ_PROGRAMS)\ndiff --git a/fuzz-pack-idx.c b/fuzz-pack-idx.c\nnew file mode 100644\nindex 0000000000..0c3d777aac\n--- /dev/null\n+++ b/fuzz-pack-idx.c\n@@ -0,0 +1,13 @@\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n+{\n+\tstruct packed_git p;\n+\n+\tload_idx(\"fuzz-input\", GIT_SHA1_RAWSZ, (void *)data, size, &p);\n+\n+\treturn 0;\n+}\ndiff --git a/packfile.c b/packfile.c\nindex 841b36182f..86074a76e9 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -80,10 +80,8 @@ void pack_report(void)\n static int check_packed_git_idx(const char *path, struct packed_git *p)\n {\n \tvoid *idx_map;\n-\tstruct pack_idx_header *hdr;\n \tsize_t idx_size;\n-\tuint32_t version, nr, i, *index;\n-\tint fd = git_open(path);\n+\tint fd = git_open(path), ret;\n \tstruct stat st;\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n@@ -101,16 +99,32 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \n-\thdr = idx_map;\n+\tret = load_idx(path, hashsz, idx_map, idx_size, p);\n+\n+\tif (ret)\n+\t\tmunmap(idx_map, idx_size);\n+\n+\treturn ret;\n+}\n+\n+int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n+\t     size_t idx_size, struct packed_git *p)\n+{\n+\tstruct pack_idx_header *hdr = idx_map;\n+\tuint32_t version, nr, i, *index;\n+\n+\tif (idx_size < 4 * 256 + hashsz + hashsz)\n+\t\treturn error(\"index file %s is too small\", path);\n+\tif (idx_map == NULL)\n+\t\treturn error(\"empty data\");\n+\n \tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n \t\tversion = ntohl(hdr->idx_version);\n-\t\tif (version < 2 || version > 2) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (version < 2 || version > 2)\n \t\t\treturn error(\"index file %s is version %\"PRIu32\n \t\t\t\t     \" and is not supported by this binary\"\n \t\t\t\t     \" (try upgrading GIT to a newer version)\",\n \t\t\t\t     path, version);\n-\t\t}\n \t} else\n \t\tversion = 1;\n \n@@ -120,10 +134,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\tindex += 2;  /* skip index header */\n \tfor (i = 0; i < 256; i++) {\n \t\tuint32_t n = ntohl(index[i]);\n-\t\tif (n < nr) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (n < nr)\n \t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\t}\n \t\tnr = n;\n \t}\n \n@@ -135,10 +147,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\t *  - hash of the packfile\n \t\t *  - file checksum\n \t\t */\n-\t\tif (idx_size != 4*256 + nr * (hashsz + 4) + hashsz + hashsz) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (idx_size != 4 * 256 + nr * (hashsz + 4) + hashsz + hashsz)\n \t\t\treturn error(\"wrong index v1 file size in %s\", path);\n-\t\t}\n \t} else if (version == 2) {\n \t\t/*\n \t\t * Minimum size:\n@@ -157,20 +167,16 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\tunsigned long max_size = min_size;\n \t\tif (nr)\n \t\t\tmax_size += (nr - 1)*8;\n-\t\tif (idx_size < min_size || idx_size > max_size) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (idx_size < min_size || idx_size > max_size)\n \t\t\treturn error(\"wrong index v2 file size in %s\", path);\n-\t\t}\n \t\tif (idx_size != min_size &&\n \t\t    /*\n \t\t     * make sure we can deal with large pack offsets.\n \t\t     * 31-bit signed offset won't be enough, neither\n \t\t     * 32-bit unsigned one will be.\n \t\t     */\n-\t\t    (sizeof(off_t) <= 4)) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\t    (sizeof(off_t) <= 4))\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n-\t\t}\n \t}\n \n \tp->index_version = version;\ndiff --git a/packfile.h b/packfile.h\nindex 442625723d..6c4037605d 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -164,4 +164,17 @@ extern int has_pack_index(const unsigned char *sha1);\n  */\n extern int is_promisor_object(const struct object_id *oid);\n \n+/*\n+ * Expose a function for fuzz testing.\n+ *\n+ * load_idx() parses a block of memory as a packfile index and puts the results\n+ * into a struct packed_git.\n+ *\n+ * This function should not be used directly. It is exposed here only so that we\n+ * have a convenient entry-point for fuzz testing. For real uses, you should\n+ * probably use open_pack_index() or parse_pack_index() instead.\n+ */\n+extern int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n+\t\t    size_t idx_size, struct packed_git *p);\n+\n #endif\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359974","messageId":"xmqqva6amtt7.fsf@gitster-ct.c.googlers.com","threadId":"49486","inReplyTo":"bfe4f2f965528ca2a45864857051b9835970ff07.1538693039.git.steadmon@google.com","subject":"Re: [RFC PATCH 1/2] fuzz: Add basic fuzz testing target.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-10T02:14:44Z","receivedAt":"2018-10-10T02:14:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon <steadmon@google.com> writes:\n\n> +FUZZ_OBJS += fuzz-pack-headers.o\n> +\n> +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n> +\n> ...\n> +### Fuzz testing\n> +#\n> +.PHONY: fuzz-clean fuzz-objs fuzz-compile\n\nI take it that you anticipate the fuzz programs in the future all\nbe named fuzz-$(blah), whose source is fuzz-$(blah).o (even though\nwe may grow some common code that may be linked with them, which can\nbe done by tweaking the rule for the $(FUZZ_PROGRAMS) target).  Am I\nreading you correctly?  Would fuzz-{clean,objs,compile} risk squatting\non nicer names we may want to use for $(blah) down the line?\n\n> + ...\n> +$(FUZZ_PROGRAMS): fuzz-compile\n> +\tclang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \\\n> +\t\t$(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@\n\nIs the expected usage pattern to know a single fuzz-* program the\nbuilder wants to build, to run \"make fuzz-pack-headers\"?  If not, it\nalso would be a good idea to have something like\n\n    fuzz-build-all:: $(FUZZ_PROGRAMS)\n    .PHONY: fuzz-build-all\n\nperhaps?\n\nAlso, in the final version we unleash to general developer audience,\nwe'd want to support \"make V=1\" (and \"make\" that is \"$(QUIET)\").\n"},{"id":"359975","messageId":"xmqqr2gymtl4.fsf@gitster-ct.c.googlers.com","threadId":"49486","inReplyTo":"60b9daad5c5132811875b6094abf4d87852e4263.1538693039.git.steadmon@google.com","subject":"Re: [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-10T02:19:35Z","receivedAt":"2018-10-10T02:19:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon <steadmon@google.com> writes:\n\n>  ### Fuzz testing\n>  #\n> -.PHONY: fuzz-clean fuzz-objs fuzz-compile\n> +.PHONY: fuzz-clean fuzz-objs fuzz-compile fuzz-all\n> ...\n>  FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address\n> ...\n> +\n> +fuzz-all: $(FUZZ_PROGRAMS)\n\nI guess I read your mind ;-) Please do this in 1/2 instead of adding\nit at 2/2 as \"oops, we'd need it more and more as we add these\".\n\n\n\n"},{"id":"360371","messageId":"cover.1539391439.git.steadmon@google.com","threadId":"49486","inReplyTo":"cover.1538693039.git.steadmon@google.com","subject":"[PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz","fromName":"","fromEmail":"steadmon@google.com","sentAt":"2018-10-13T00:58:39Z","receivedAt":"2018-10-13T00:58:45Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Josh Steadmon <steadmon@google.com>\n\nV2 of this series pulls the compiler flags out of the Makefile, to be\nprovided by the user depending on the combination of compiler and\nfuzzing engine in use. This also makes it more compatible with\nOSS-Fuzz's build process.\n\nJosh Steadmon (2):\n  fuzz: Add basic fuzz testing target.\n  fuzz: Add fuzz testing for packfile indices.\n\n .gitignore          |  3 +++\n Makefile            | 33 +++++++++++++++++++++++++++++++++\n fuzz-pack-headers.c | 14 ++++++++++++++\n fuzz-pack-idx.c     | 13 +++++++++++++\n packfile.c          | 44 +++++++++++++++++++++++++-------------------\n packfile.h          | 13 +++++++++++++\n 6 files changed, 101 insertions(+), 19 deletions(-)\n create mode 100644 fuzz-pack-headers.c\n create mode 100644 fuzz-pack-idx.c\n\nRange-diff against v1:\n1:  9456c41798 ! 1:  446d8081b1 fuzz: Add basic fuzz testing target.\n    @@ -32,6 +32,9 @@\n      \n     +FUZZ_OBJS += fuzz-pack-headers.o\n     +\n    ++# Always build fuzz objects even if not testing, to prevent bit-rot.\n    ++all:: $(FUZZ_OBJS)\n    ++\n     +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n     +\n      # Empty...\n    @@ -46,14 +49,13 @@\n      \tgit.o\n      ifndef NO_CURL\n     @@\n    - cocciclean:\n    - \t$(RM) contrib/coccinelle/*.cocci.patch*\n    - \n    --clean: profile-clean coverage-clean cocciclean\n    -+clean: profile-clean coverage-clean cocciclean fuzz-clean\n    - \t$(RM) *.res\n    - \t$(RM) $(OBJECTS)\n      \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB)\n    + \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n    + \t$(RM) $(TEST_PROGRAMS) $(NO_INSTALL)\n    ++\t$(RM) $(FUZZ_PROGRAMS)\n    + \t$(RM) -r bin-wrappers $(dep_dirs)\n    + \t$(RM) -r po/build/\n    + \t$(RM) *.pyc *.pyo */*.pyc */*.pyo command-list.h $(ETAGS_TARGET) tags cscope*\n     @@\n      cover_db_html: cover_db\n      \tcover -report html -outputdir cover_db_html cover_db\n    @@ -61,24 +63,24 @@\n     +\n     +### Fuzz testing\n     +#\n    -+.PHONY: fuzz-clean fuzz-objs fuzz-compile\n    -+\n    -+FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address\n    -+FUZZ_LDFLAGS = $(FUZZ_CFLAGS)\n    -+\n    -+\n    -+fuzz-clean:\n    -+\t$(RM) $(FUZZ_PROGRAMS) $(FUZZ_OBJS)\n    -+\n    -+fuzz-objs: $(FUZZ_OBJS)\n    ++# Building fuzz targets generally requires a special set of compiler flags that\n    ++# are not necessarily appropriate for general builds, and that vary greatly\n    ++# depending on the compiler version used.\n    ++#\n    ++# An example command to build against libFuzzer from LLVM 4.0.0:\n    ++#\n    ++# make CC=clang CXX=clang++ \\\n    ++#      CFLAGS=\"-fsanitize-coverage=trace-pc-guard -fsanitize=address\" \\\n    ++#      LIB_FUZZING_ENGINE=/usr/lib/llvm-4.0/lib/libFuzzer.a \\\n    ++#      fuzz-all\n    ++#\n    ++.PHONY: fuzz-all\n     +\n    -+fuzz-compile:\n    -+\t$(MAKE) CC=clang LD=clang CFLAGS=\"$(FUZZ_CFLAGS)\" \\\n    -+\t\tLDFLAGS=\"$(FUZZ_LDFLAGS)\" all fuzz-objs\n    ++$(FUZZ_PROGRAMS): all\n    ++\t$(QUIET_LINK)$(CXX) $(CFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) \\\n    ++\t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n     +\n    -+$(FUZZ_PROGRAMS): fuzz-compile\n    -+\tclang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \\\n    -+\t\t$(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@\n    ++fuzz-all: $(FUZZ_PROGRAMS)\n     \n      diff --git a/fuzz-pack-headers.c b/fuzz-pack-headers.c\n      new file mode 100644\n2:  581eb8f817 ! 2:  c7b5a03d81 fuzz: Add fuzz testing for packfile indices.\n    @@ -24,23 +24,8 @@\n      FUZZ_OBJS += fuzz-pack-headers.o\n     +FUZZ_OBJS += fuzz-pack-idx.o\n      \n    - FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n    - \n    -@@\n    - \n    - ### Fuzz testing\n    - #\n    --.PHONY: fuzz-clean fuzz-objs fuzz-compile\n    -+.PHONY: fuzz-clean fuzz-objs fuzz-compile fuzz-all\n    - \n    - FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address\n    - FUZZ_LDFLAGS = $(FUZZ_CFLAGS)\n    -@@\n    - $(FUZZ_PROGRAMS): fuzz-compile\n    - \tclang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \\\n    - \t\t$(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@\n    -+\n    -+fuzz-all: $(FUZZ_PROGRAMS)\n    + # Always build fuzz objects even if not testing, to prevent bit-rot.\n    + all:: $(FUZZ_OBJS)\n     \n      diff --git a/fuzz-pack-idx.c b/fuzz-pack-idx.c\n      new file mode 100644\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"360372","messageId":"446d8081b1a05250e69ae56a4fa41923039c69f6.1539391439.git.steadmon@google.com","threadId":"49486","inReplyTo":"cover.1539391439.git.steadmon@google.com","subject":"[PATCH v2 1/2] fuzz: Add basic fuzz testing target.","fromName":"","fromEmail":"steadmon@google.com","sentAt":"2018-10-13T00:58:40Z","receivedAt":"2018-10-13T00:58:48Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Josh Steadmon <steadmon@google.com>\n\nfuzz-pack-headers.c provides a fuzzing entry point compatible with\nlibFuzzer (and possibly other fuzzing engines).\n\nSigned-off-by: Josh Steadmon <steadmon@google.com>\n---\n .gitignore          |  2 ++\n Makefile            | 32 ++++++++++++++++++++++++++++++++\n fuzz-pack-headers.c | 14 ++++++++++++++\n 3 files changed, 48 insertions(+)\n create mode 100644 fuzz-pack-headers.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 9d1363a1eb..87a28b3115 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -1,3 +1,5 @@\n+/fuzz_corpora\n+/fuzz-pack-headers\n /GIT-BUILD-OPTIONS\n /GIT-CFLAGS\n /GIT-LDFLAGS\ndiff --git a/Makefile b/Makefile\nindex 13e1c52478..7f5a19b8ab 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -590,6 +590,8 @@ XDIFF_OBJS =\n VCSSVN_OBJS =\n GENERATED_H =\n EXTRA_CPPFLAGS =\n+FUZZ_OBJS =\n+FUZZ_PROGRAMS =\n LIB_OBJS =\n PROGRAM_OBJS =\n PROGRAMS =\n@@ -682,6 +684,13 @@ SCRIPTS = $(SCRIPT_SH_INS) \\\n \n ETAGS_TARGET = TAGS\n \n+FUZZ_OBJS += fuzz-pack-headers.o\n+\n+# Always build fuzz objects even if not testing, to prevent bit-rot.\n+all:: $(FUZZ_OBJS)\n+\n+FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n+\n # Empty...\n EXTRA_PROGRAMS =\n \n@@ -2250,6 +2259,7 @@ TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(VCSSVN_OBJS) \\\n+\t$(FUZZ_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2937,6 +2947,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS) $(NO_INSTALL)\n+\t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) -r bin-wrappers $(dep_dirs)\n \t$(RM) -r po/build/\n \t$(RM) *.pyc *.pyo */*.pyc */*.pyo command-list.h $(ETAGS_TARGET) tags cscope*\n@@ -3061,3 +3072,24 @@ cover_db: coverage-report\n cover_db_html: cover_db\n \tcover -report html -outputdir cover_db_html cover_db\n \n+\n+### Fuzz testing\n+#\n+# Building fuzz targets generally requires a special set of compiler flags that\n+# are not necessarily appropriate for general builds, and that vary greatly\n+# depending on the compiler version used.\n+#\n+# An example command to build against libFuzzer from LLVM 4.0.0:\n+#\n+# make CC=clang CXX=clang++ \\\n+#      CFLAGS=\"-fsanitize-coverage=trace-pc-guard -fsanitize=address\" \\\n+#      LIB_FUZZING_ENGINE=/usr/lib/llvm-4.0/lib/libFuzzer.a \\\n+#      fuzz-all\n+#\n+.PHONY: fuzz-all\n+\n+$(FUZZ_PROGRAMS): all\n+\t$(QUIET_LINK)$(CXX) $(CFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) \\\n+\t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n+\n+fuzz-all: $(FUZZ_PROGRAMS)\ndiff --git a/fuzz-pack-headers.c b/fuzz-pack-headers.c\nnew file mode 100644\nindex 0000000000..99da1d0fd3\n--- /dev/null\n+++ b/fuzz-pack-headers.c\n@@ -0,0 +1,14 @@\n+#include \"packfile.h\"\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n+{\n+\tenum object_type type;\n+\tunsigned long len;\n+\n+\tunpack_object_header_buffer((const unsigned char *)data,\n+\t\t\t\t    (unsigned long)size, &type, &len);\n+\n+\treturn 0;\n+}\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"360373","messageId":"c7b5a03d810bee0289cccedd5f433fca2867904c.1539391439.git.steadmon@google.com","threadId":"49486","inReplyTo":"cover.1539391439.git.steadmon@google.com","subject":"[PATCH v2 2/2] fuzz: Add fuzz testing for packfile indices.","fromName":"","fromEmail":"steadmon@google.com","sentAt":"2018-10-13T00:58:41Z","receivedAt":"2018-10-13T00:58:50Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Josh Steadmon <steadmon@google.com>\n\nBreaks the majority of check_packed_git_idx() into a separate function,\nload_idx(). The latter function operates on arbitrary buffers, which\nmakes it suitable as a fuzzing test target.\n\nSigned-off-by: Josh Steadmon <steadmon@google.com>\n---\n .gitignore      |  1 +\n Makefile        |  1 +\n fuzz-pack-idx.c | 13 +++++++++++++\n packfile.c      | 44 +++++++++++++++++++++++++-------------------\n packfile.h      | 13 +++++++++++++\n 5 files changed, 53 insertions(+), 19 deletions(-)\n create mode 100644 fuzz-pack-idx.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 87a28b3115..64b3377d40 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -1,5 +1,6 @@\n /fuzz_corpora\n /fuzz-pack-headers\n+/fuzz-pack-idx\n /GIT-BUILD-OPTIONS\n /GIT-CFLAGS\n /GIT-LDFLAGS\ndiff --git a/Makefile b/Makefile\nindex 7f5a19b8ab..1b3d893090 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -685,6 +685,7 @@ SCRIPTS = $(SCRIPT_SH_INS) \\\n ETAGS_TARGET = TAGS\n \n FUZZ_OBJS += fuzz-pack-headers.o\n+FUZZ_OBJS += fuzz-pack-idx.o\n \n # Always build fuzz objects even if not testing, to prevent bit-rot.\n all:: $(FUZZ_OBJS)\ndiff --git a/fuzz-pack-idx.c b/fuzz-pack-idx.c\nnew file mode 100644\nindex 0000000000..0c3d777aac\n--- /dev/null\n+++ b/fuzz-pack-idx.c\n@@ -0,0 +1,13 @@\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n+\n+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n+{\n+\tstruct packed_git p;\n+\n+\tload_idx(\"fuzz-input\", GIT_SHA1_RAWSZ, (void *)data, size, &p);\n+\n+\treturn 0;\n+}\ndiff --git a/packfile.c b/packfile.c\nindex 841b36182f..86074a76e9 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -80,10 +80,8 @@ void pack_report(void)\n static int check_packed_git_idx(const char *path, struct packed_git *p)\n {\n \tvoid *idx_map;\n-\tstruct pack_idx_header *hdr;\n \tsize_t idx_size;\n-\tuint32_t version, nr, i, *index;\n-\tint fd = git_open(path);\n+\tint fd = git_open(path), ret;\n \tstruct stat st;\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n@@ -101,16 +99,32 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \n-\thdr = idx_map;\n+\tret = load_idx(path, hashsz, idx_map, idx_size, p);\n+\n+\tif (ret)\n+\t\tmunmap(idx_map, idx_size);\n+\n+\treturn ret;\n+}\n+\n+int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n+\t     size_t idx_size, struct packed_git *p)\n+{\n+\tstruct pack_idx_header *hdr = idx_map;\n+\tuint32_t version, nr, i, *index;\n+\n+\tif (idx_size < 4 * 256 + hashsz + hashsz)\n+\t\treturn error(\"index file %s is too small\", path);\n+\tif (idx_map == NULL)\n+\t\treturn error(\"empty data\");\n+\n \tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n \t\tversion = ntohl(hdr->idx_version);\n-\t\tif (version < 2 || version > 2) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (version < 2 || version > 2)\n \t\t\treturn error(\"index file %s is version %\"PRIu32\n \t\t\t\t     \" and is not supported by this binary\"\n \t\t\t\t     \" (try upgrading GIT to a newer version)\",\n \t\t\t\t     path, version);\n-\t\t}\n \t} else\n \t\tversion = 1;\n \n@@ -120,10 +134,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\tindex += 2;  /* skip index header */\n \tfor (i = 0; i < 256; i++) {\n \t\tuint32_t n = ntohl(index[i]);\n-\t\tif (n < nr) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (n < nr)\n \t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\t}\n \t\tnr = n;\n \t}\n \n@@ -135,10 +147,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\t *  - hash of the packfile\n \t\t *  - file checksum\n \t\t */\n-\t\tif (idx_size != 4*256 + nr * (hashsz + 4) + hashsz + hashsz) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (idx_size != 4 * 256 + nr * (hashsz + 4) + hashsz + hashsz)\n \t\t\treturn error(\"wrong index v1 file size in %s\", path);\n-\t\t}\n \t} else if (version == 2) {\n \t\t/*\n \t\t * Minimum size:\n@@ -157,20 +167,16 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\tunsigned long max_size = min_size;\n \t\tif (nr)\n \t\t\tmax_size += (nr - 1)*8;\n-\t\tif (idx_size < min_size || idx_size > max_size) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\tif (idx_size < min_size || idx_size > max_size)\n \t\t\treturn error(\"wrong index v2 file size in %s\", path);\n-\t\t}\n \t\tif (idx_size != min_size &&\n \t\t    /*\n \t\t     * make sure we can deal with large pack offsets.\n \t\t     * 31-bit signed offset won't be enough, neither\n \t\t     * 32-bit unsigned one will be.\n \t\t     */\n-\t\t    (sizeof(off_t) <= 4)) {\n-\t\t\tmunmap(idx_map, idx_size);\n+\t\t    (sizeof(off_t) <= 4))\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n-\t\t}\n \t}\n \n \tp->index_version = version;\ndiff --git a/packfile.h b/packfile.h\nindex 442625723d..6c4037605d 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -164,4 +164,17 @@ extern int has_pack_index(const unsigned char *sha1);\n  */\n extern int is_promisor_object(const struct object_id *oid);\n \n+/*\n+ * Expose a function for fuzz testing.\n+ *\n+ * load_idx() parses a block of memory as a packfile index and puts the results\n+ * into a struct packed_git.\n+ *\n+ * This function should not be used directly. It is exposed here only so that we\n+ * have a convenient entry-point for fuzz testing. For real uses, you should\n+ * probably use open_pack_index() or parse_pack_index() instead.\n+ */\n+extern int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n+\t\t    size_t idx_size, struct packed_git *p);\n+\n #endif\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"360374","messageId":"20181013005918.GD12177@google.com","threadId":"49486","inReplyTo":"xmqqva6amtt7.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH 1/2] fuzz: Add basic fuzz testing target.","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2018-10-13T00:59:18Z","receivedAt":"2018-10-13T00:59:26Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2018.10.10 11:14, Junio C Hamano wrote:\n> Josh Steadmon <steadmon@google.com> writes:\n> \n> > +FUZZ_OBJS += fuzz-pack-headers.o\n> > +\n> > +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))\n> > +\n> > ...\n> > +### Fuzz testing\n> > +#\n> > +.PHONY: fuzz-clean fuzz-objs fuzz-compile\n> \n> I take it that you anticipate the fuzz programs in the future all\n> be named fuzz-$(blah), whose source is fuzz-$(blah).o (even though\n> we may grow some common code that may be linked with them, which can\n> be done by tweaking the rule for the $(FUZZ_PROGRAMS) target).  Am I\n> reading you correctly?  Would fuzz-{clean,objs,compile} risk squatting\n> on nicer names we may want to use for $(blah) down the line?\n\nYes, that's correct. I've reworked the rules to be more compatible with\nhow OSS-Fuzz expects to build these targets, and now \"fuzz-all\" is\nthe only remaining special target.\n\n> > + ...\n> > +$(FUZZ_PROGRAMS): fuzz-compile\n> > +\tclang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \\\n> > +\t\t$(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@\n> \n> Is the expected usage pattern to know a single fuzz-* program the\n> builder wants to build, to run \"make fuzz-pack-headers\"?  If not, it\n> also would be a good idea to have something like\n> \n>     fuzz-build-all:: $(FUZZ_PROGRAMS)\n>     .PHONY: fuzz-build-all\n> \n> perhaps?\n> \n> Also, in the final version we unleash to general developer audience,\n> we'd want to support \"make V=1\" (and \"make\" that is \"$(QUIET)\").\n\nDone and done.\n"},{"id":"360586","messageId":"xmqqbm7uza6f.fsf@gitster-ct.c.googlers.com","threadId":"49486","inReplyTo":"cover.1539391439.git.steadmon@google.com","subject":"Re: [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-16T06:18:32Z","receivedAt":"2018-10-16T06:18:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"steadmon@google.com writes:\n\n> From: Josh Steadmon <steadmon@google.com>\n>\n> V2 of this series pulls the compiler flags out of the Makefile, to be\n> provided by the user depending on the combination of compiler and\n> fuzzing engine in use. This also makes it more compatible with\n> OSS-Fuzz's build process.\n\nThanks, will replace.  I think we can merge this to 'next' and down\nand build incrementally on top.\n"}]}