{"thread":{"id":"26132","subject":"[PATCH 0/4] teach vcs-svn/line_buffer to handle multiple input files","startedAt":"2010-12-24T08:05:05Z","lastAt":"2011-03-01T00:41:32Z","messageCount":25,"participants":["Jonathan Nieder","David Michael Barr","Junio C Hamano","Sverre Rabbelier"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"158545","messageId":"20101224080505.GA29681@burratino","threadId":"26132","inReplyTo":null,"subject":"[PATCH 0/4] teach vcs-svn/line_buffer to handle multiple input files","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-24T08:05:05Z","receivedAt":"2010-12-24T08:05:05Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi David et al,\n\nThis collection of patches comes from the svndiff0 series[1].  They\nare not urgent --- the motivation is for svn-fe to be able to keep\nseparate track of input from stdin (svnrdump) and the report-fd (blobs\nfrom fast-import) for the coming Text-delta support --- but ideally I\nwould like to see them applied at the start of the next merge window,\nsince they change API that other patches use.  See [*] below for the\nopen question.\n\nThe idea: instead of keeping the input file handle and input buffer as\nglobal variables, pack them in a struct and let the calling program\nkeep track of them.\n\nPatch 1 makes a previously global buffer local to the two functions\nthat use it.  Performance impact should be negligibile.  Ideally the\nbuffer would not be needed at all --- there is enough buffering at\nlower layers already --- but stdio does not provide the calls that\nwould be needed to eliminate it (in particular a wrapper for\nsendfile(2)).\n\nPatch 2 replaces a use of the obj_pool library with a strbuf.  The\nmain immediate effect is to improve error handling behavior (more\nimportantly, this is needed for patches 3 and 4 since obj_pool is\ndefined to be global).\n\nPatches 3 and 4 are the main patches, collecting input state in a\nstruct and moving resposibility for that struct to the calling\nprogram, respectively.\n\nThe patches have already received a lot of testing.\n\n[*]\nI am not sure whether this is the right approach for reading from the\nreport-fd.  To avoid deadlock, we cannot issue a blocking read(2)\nafter the trailing newline has been read from an expected line or the\nnth byte has been read in fixed-length input.  This would rule out\nfread/fgets if implemented as follows with too large an internal\nbuffer:\n\n 1. fill internal buffer completely (or stop when an error or\n    end of file is encountered)\n 2. fill caller's buffer from internal buffer\n\nglibc fread/fgets do not work that way (and in fact will never\ndeadlock for us).  What about other platforms?  The standards (C,\nPOSIX) do not make it obvious.\n\n - maybe setvbuf(f, NULL, _IOLBF, 0) would make fgets safe to use\n - maybe setvbuf(f, NULL, _IONBF, 0) would make fread safe to use.\n   On the other hand, it is not clear to me what unbuffered input\n   means in this context.  That flag does not do anything meaningful\n   on glibc, for example, for input streams.\n\nIf all else fails, setting the O_NONBLOCK flag with fcntl (this\ncould presumably be implemented as SetNamedPipeHandleState(...,\nPIPE_NOWAIT) on Windows) would avoid trouble.  After any failing\noperation we would have to check that the error is EAGAIN and\nclear the error indicator.  Simple.  But it would be even better\nto learn that that is not needed.\n\nSo far I have been playing it safe with the read(fd, buf, 1) trick\nbut that does not have great performance, as David noticed[2].\n\nThoughts?\n\nJonathan Nieder (4):\n  vcs-svn: Eliminate global byte_buffer[] array\n  vcs-svn: Replace buffer_read_string memory pool with a strbuf\n  vcs-svn: Collect line_buffer data in a struct\n  vcs-svn: Teach line_buffer to handle multiple input files\n\n test-line-buffer.c      |   17 ++++++-----\n vcs-svn/fast_export.c   |    6 ++--\n vcs-svn/fast_export.h   |    5 +++-\n vcs-svn/line_buffer.c   |   66 ++++++++++++++++++++--------------------------\n vcs-svn/line_buffer.h   |   25 +++++++++++++-----\n vcs-svn/line_buffer.txt |    5 ++-\n vcs-svn/svndump.c       |   29 +++++++++++---------\n 7 files changed, 82 insertions(+), 71 deletions(-)\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/158731\n[2] http://colabti.org/irclogger/irclogger_log/git-devel?date=2010-12-18\n"},{"id":"158546","messageId":"20101224080853.GB29681@burratino","threadId":"26132","inReplyTo":"20101224080505.GA29681@burratino","subject":"[PATCH 1/4] vcs-svn: eliminate global byte_buffer","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-24T08:08:53Z","receivedAt":"2010-12-24T08:08:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Date: Sun, 10 Oct 2010 21:37:10 -0500\n\nThe data stored in byte_buffer[] is always either discarded or\nwritten to stdout immediately.  No need for it to persist between\nfunction calls.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nFor the line_buffer library.  Decreases BSS size.  Increase stack\noverhead of two I/O functions by 4096 bytes.  Performance effect\nhasn't been measured.  Alas there is no stdio wrapper for sendfile; if\nthere were, we could eliminate the buffer altogether.\n\nOf course the goal is to make line_buffer more easily reusable, by\neliminating _all_ global state.  I assume David would like this but I\ndon't remember if he said so.\n\n vcs-svn/line_buffer.c |    3 ++-\n 1 files changed, 2 insertions(+), 1 deletions(-)\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex 1543567..f22c94f 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -14,7 +14,6 @@\n obj_pool_gen(blob, char, 4096)\n \n static char line_buffer[LINE_BUFFER_LEN];\n-static char byte_buffer[COPY_BUFFER_LEN];\n static FILE *infile;\n \n int buffer_init(const char *filename)\n@@ -68,6 +67,7 @@ char *buffer_read_string(uint32_t len)\n \n void buffer_copy_bytes(uint32_t len)\n {\n+\tchar byte_buffer[COPY_BUFFER_LEN];\n \tuint32_t in;\n \twhile (len > 0 && !feof(infile) && !ferror(infile)) {\n \t\tin = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;\n@@ -83,6 +83,7 @@ void buffer_copy_bytes(uint32_t len)\n \n void buffer_skip_bytes(uint32_t len)\n {\n+\tchar byte_buffer[COPY_BUFFER_LEN];\n \tuint32_t in;\n \twhile (len > 0 && !feof(infile) && !ferror(infile)) {\n \t\tin = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;\n-- \n1.7.2.3.554.gc9b5c.dirty\n"},{"id":"158547","messageId":"20101224081700.GC29681@burratino","threadId":"26132","inReplyTo":"20101224080505.GA29681@burratino","subject":"[PATCH 2/4] vcs-svn: replace buffer_read_string memory pool with a strbuf","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-24T08:17:00Z","receivedAt":"2010-12-24T08:17:00Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Date: Sat, 6 Nov 2010 12:01:28 -0500\n\nobj_pool is inherently global and does not use the standard growing\nfactor alloc_nr, which makes it feel out of place in the git codebase.\nPlus it is overkill for this application: all that is needed is a\nbuffer that can grow between requests to accomodate larger strings.\nUse a strbuf instead.\n\nAs a side effect, this improves the error handling: allocation\nfailures will result in a clean exit instead of segfaults.  It would\nbe nice to add a test case (using ulimit or failmalloc) but that can\nwait for another day.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nRequires jn/thinner-wrapper (from master) if contrib/svn-fe/svn-fe is\nto build without linking to libz et al.\n\nThe initial size of the per-line buffer shrinks from 4096 to 0 (well,\nmaybe 16 or so).  strbuf_fread is not inline.  I haven't looked into\nthe effect on performance from these changes.\n\nI find obj_pool tricky to use correctly (see 3c93983, vcs-svn: fix\nintermittent repo_tree corruption, 2010-12-05 for example) so I look\nforward to eliminating obj_pool from the vcs-svn/ dir altogether.\nExcitingly enough, David has already done that, it seems[1].\n\n[1] git://github.com/barrbrain/git.git vcs-svn-incremental\n\n vcs-svn/line_buffer.c |   16 ++++++----------\n 1 files changed, 6 insertions(+), 10 deletions(-)\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex f22c94f..6f32f28 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -5,15 +5,13 @@\n \n #include \"git-compat-util.h\"\n #include \"line_buffer.h\"\n-#include \"obj_pool.h\"\n+#include \"strbuf.h\"\n \n #define LINE_BUFFER_LEN 10000\n #define COPY_BUFFER_LEN 4096\n \n-/* Create memory pool for char sequence of known length */\n-obj_pool_gen(blob, char, 4096)\n-\n static char line_buffer[LINE_BUFFER_LEN];\n+static struct strbuf blob_buffer = STRBUF_INIT;\n static FILE *infile;\n \n int buffer_init(const char *filename)\n@@ -58,11 +56,9 @@ char *buffer_read_line(void)\n \n char *buffer_read_string(uint32_t len)\n {\n-\tchar *s;\n-\tblob_free(blob_pool.size);\n-\ts = blob_pointer(blob_alloc(len + 1));\n-\ts[fread(s, 1, len, infile)] = '\\0';\n-\treturn ferror(infile) ? NULL : s;\n+\tstrbuf_reset(&blob_buffer);\n+\tstrbuf_fread(&blob_buffer, len, infile);\n+\treturn ferror(infile) ? NULL : blob_buffer.buf;\n }\n \n void buffer_copy_bytes(uint32_t len)\n@@ -94,5 +90,5 @@ void buffer_skip_bytes(uint32_t len)\n \n void buffer_reset(void)\n {\n-\tblob_reset();\n+\tstrbuf_release(&blob_buffer);\n }\n-- \n1.7.2.3.554.gc9b5c.dirty\n"},{"id":"158548","messageId":"20101224081844.GD29681@burratino","threadId":"26132","inReplyTo":"20101224080505.GA29681@burratino","subject":"[PATCH 3/4] vcs-svn: collect line_buffer data in a struct","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-24T08:18:44Z","receivedAt":"2010-12-24T08:18:44Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Date: Sun, 10 Oct 2010 21:39:21 -0500\n\nPrepare for the line_buffer lib to support input from multiple files,\nby collecting global state in a struct that can be easily passed\naround.\n\nNo API change yet.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nA simple search-and-replace kind of job.  If gcc is smart, there\nshould be no change in code size or performance, but I haven't\nchecked.  \n\n vcs-svn/line_buffer.c |   45 ++++++++++++++++++++++-----------------------\n vcs-svn/line_buffer.h |   11 +++++++++++\n 2 files changed, 33 insertions(+), 23 deletions(-)\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex 6f32f28..e7bc230 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -7,17 +7,16 @@\n #include \"line_buffer.h\"\n #include \"strbuf.h\"\n \n-#define LINE_BUFFER_LEN 10000\n #define COPY_BUFFER_LEN 4096\n-\n-static char line_buffer[LINE_BUFFER_LEN];\n-static struct strbuf blob_buffer = STRBUF_INIT;\n-static FILE *infile;\n+static struct line_buffer buf_ = LINE_BUFFER_INIT;\n+static struct line_buffer *buf;\n \n int buffer_init(const char *filename)\n {\n-\tinfile = filename ? fopen(filename, \"r\") : stdin;\n-\tif (!infile)\n+\tbuf = &buf_;\n+\n+\tbuf->infile = filename ? fopen(filename, \"r\") : stdin;\n+\tif (!buf->infile)\n \t\treturn -1;\n \treturn 0;\n }\n@@ -25,10 +24,10 @@ int buffer_init(const char *filename)\n int buffer_deinit(void)\n {\n \tint err;\n-\tif (infile == stdin)\n-\t\treturn ferror(infile);\n-\terr = ferror(infile);\n-\terr |= fclose(infile);\n+\tif (buf->infile == stdin)\n+\t\treturn ferror(buf->infile);\n+\terr = ferror(buf->infile);\n+\terr |= fclose(buf->infile);\n \treturn err;\n }\n \n@@ -36,13 +35,13 @@ int buffer_deinit(void)\n char *buffer_read_line(void)\n {\n \tchar *end;\n-\tif (!fgets(line_buffer, sizeof(line_buffer), infile))\n+\tif (!fgets(buf->line_buffer, sizeof(buf->line_buffer), buf->infile))\n \t\t/* Error or data exhausted. */\n \t\treturn NULL;\n-\tend = line_buffer + strlen(line_buffer);\n+\tend = buf->line_buffer + strlen(buf->line_buffer);\n \tif (end[-1] == '\\n')\n \t\tend[-1] = '\\0';\n-\telse if (feof(infile))\n+\telse if (feof(buf->infile))\n \t\t; /* No newline at end of file.  That's fine. */\n \telse\n \t\t/*\n@@ -51,23 +50,23 @@ char *buffer_read_line(void)\n \t\t * but for now let's return an error.\n \t\t */\n \t\treturn NULL;\n-\treturn line_buffer;\n+\treturn buf->line_buffer;\n }\n \n char *buffer_read_string(uint32_t len)\n {\n-\tstrbuf_reset(&blob_buffer);\n-\tstrbuf_fread(&blob_buffer, len, infile);\n-\treturn ferror(infile) ? NULL : blob_buffer.buf;\n+\tstrbuf_reset(&buf->blob_buffer);\n+\tstrbuf_fread(&buf->blob_buffer, len, buf->infile);\n+\treturn ferror(buf->infile) ? NULL : buf->blob_buffer.buf;\n }\n \n void buffer_copy_bytes(uint32_t len)\n {\n \tchar byte_buffer[COPY_BUFFER_LEN];\n \tuint32_t in;\n-\twhile (len > 0 && !feof(infile) && !ferror(infile)) {\n+\twhile (len > 0 && !feof(buf->infile) && !ferror(buf->infile)) {\n \t\tin = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;\n-\t\tin = fread(byte_buffer, 1, in, infile);\n+\t\tin = fread(byte_buffer, 1, in, buf->infile);\n \t\tlen -= in;\n \t\tfwrite(byte_buffer, 1, in, stdout);\n \t\tif (ferror(stdout)) {\n@@ -81,14 +80,14 @@ void buffer_skip_bytes(uint32_t len)\n {\n \tchar byte_buffer[COPY_BUFFER_LEN];\n \tuint32_t in;\n-\twhile (len > 0 && !feof(infile) && !ferror(infile)) {\n+\twhile (len > 0 && !feof(buf->infile) && !ferror(buf->infile)) {\n \t\tin = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;\n-\t\tin = fread(byte_buffer, 1, in, infile);\n+\t\tin = fread(byte_buffer, 1, in, buf->infile);\n \t\tlen -= in;\n \t}\n }\n \n void buffer_reset(void)\n {\n-\tstrbuf_release(&blob_buffer);\n+\tstrbuf_release(&buf->blob_buffer);\n }\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex 9c78ae1..4ae1133 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -1,6 +1,17 @@\n #ifndef LINE_BUFFER_H_\n #define LINE_BUFFER_H_\n \n+#include \"strbuf.h\"\n+\n+#define LINE_BUFFER_LEN 10000\n+\n+struct line_buffer {\n+\tchar line_buffer[LINE_BUFFER_LEN];\n+\tstruct strbuf blob_buffer;\n+\tFILE *infile;\n+};\n+#define LINE_BUFFER_INIT {\"\", STRBUF_INIT, NULL}\n+\n int buffer_init(const char *filename);\n int buffer_deinit(void);\n char *buffer_read_line(void);\n-- \n1.7.2.3.554.gc9b5c.dirty\n"},{"id":"158549","messageId":"20101224082846.GE29681@burratino","threadId":"26132","inReplyTo":"20101224080505.GA29681@burratino","subject":"[PATCH 4/4] vcs-svn: teach line_buffer to handle multiple input files","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-24T08:28:46Z","receivedAt":"2010-12-24T08:28:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Date: Sun, 10 Oct 2010 21:41:06 -0500\n\nCollect the line_buffer state in a newly public line_buffer struct.\nCallers can use multiple line_buffers to manage input from multiple\nfiles at a time.\n\nsvn-fe's delta applier will use this to stream a delta from svnrdump\nand the preimage it applies to from fast-import at the same time.\n\nThe tests don't take advantage of the new features, but I think that's\nokay.  It is easier to find lingering examples of nonreentrant code by\nsearching for \"static\" in line_buffer.c.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nThat's it.  The API change feels inevitable to me, perhaps because it\nhas been in use for so long.  Hopefully it can be a good basis for\nfuture changes:\n\n - buffer_tmpfile_{init,rewind,prepare_to_read}, for when files are\n   just a place to stash data for later\n - buffer_fdopen to read from a file descriptor\n - buffer_interactive for deadlock avoidance (if that turns out to\n   be needed)\n\nI would be happy to hear any thoughts you have, even as basic as\n\"looks ok\" or \"confusing\".\n\nthanks,\nJonathan\n\n test-line-buffer.c      |   17 +++++++++--------\n vcs-svn/fast_export.c   |    6 +++---\n vcs-svn/fast_export.h   |    5 ++++-\n vcs-svn/line_buffer.c   |   20 ++++++++------------\n vcs-svn/line_buffer.h   |   14 +++++++-------\n vcs-svn/line_buffer.txt |    5 +++--\n vcs-svn/svndump.c       |   29 ++++++++++++++++-------------\n 7 files changed, 50 insertions(+), 46 deletions(-)\n\ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex c11bf7f..f9af892 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -22,25 +22,26 @@ static uint32_t strtouint32(const char *s)\n \n int main(int argc, char *argv[])\n {\n+\tstruct line_buffer buf = LINE_BUFFER_INIT;\n \tchar *s;\n \n \tif (argc != 1)\n \t\tusage(\"test-line-buffer < input.txt\");\n-\tif (buffer_init(NULL))\n+\tif (buffer_init(&buf, NULL))\n \t\tdie_errno(\"open error\");\n-\twhile ((s = buffer_read_line())) {\n-\t\ts = buffer_read_string(strtouint32(s));\n+\twhile ((s = buffer_read_line(&buf))) {\n+\t\ts = buffer_read_string(&buf, strtouint32(s));\n \t\tfputs(s, stdout);\n \t\tfputc('\\n', stdout);\n-\t\tbuffer_skip_bytes(1);\n-\t\tif (!(s = buffer_read_line()))\n+\t\tbuffer_skip_bytes(&buf, 1);\n+\t\tif (!(s = buffer_read_line(&buf)))\n \t\t\tbreak;\n-\t\tbuffer_copy_bytes(strtouint32(s) + 1);\n+\t\tbuffer_copy_bytes(&buf, strtouint32(s) + 1);\n \t}\n-\tif (buffer_deinit())\n+\tif (buffer_deinit(&buf))\n \t\tdie(\"input error\");\n \tif (ferror(stdout))\n \t\tdie(\"output error\");\n-\tbuffer_reset();\n+\tbuffer_reset(&buf);\n \treturn 0;\n }\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 6cfa256..260cf50 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -63,14 +63,14 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log,\n \tprintf(\"progress Imported commit %\"PRIu32\".\\n\\n\", revision);\n }\n \n-void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len)\n+void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len, struct line_buffer *input)\n {\n \tif (mode == REPO_MODE_LNK) {\n \t\t/* svn symlink blobs start with \"link \" */\n-\t\tbuffer_skip_bytes(5);\n+\t\tbuffer_skip_bytes(input, 5);\n \t\tlen -= 5;\n \t}\n \tprintf(\"blob\\nmark :%\"PRIu32\"\\ndata %\"PRIu32\"\\n\", mark, len);\n-\tbuffer_copy_bytes(len);\n+\tbuffer_copy_bytes(input, len);\n \tfputc('\\n', stdout);\n }\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex 2aaaea5..054e7d5 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -1,11 +1,14 @@\n #ifndef FAST_EXPORT_H_\n #define FAST_EXPORT_H_\n \n+#include \"line_buffer.h\"\n+\n void fast_export_delete(uint32_t depth, uint32_t *path);\n void fast_export_modify(uint32_t depth, uint32_t *path, uint32_t mode,\n \t\t\tuint32_t mark);\n void fast_export_commit(uint32_t revision, uint32_t author, char *log,\n \t\t\tuint32_t uuid, uint32_t url, unsigned long timestamp);\n-void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len);\n+void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len,\n+\t\t      struct line_buffer *input);\n \n #endif\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex e7bc230..806932b 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -8,20 +8,16 @@\n #include \"strbuf.h\"\n \n #define COPY_BUFFER_LEN 4096\n-static struct line_buffer buf_ = LINE_BUFFER_INIT;\n-static struct line_buffer *buf;\n \n-int buffer_init(const char *filename)\n+int buffer_init(struct line_buffer *buf, const char *filename)\n {\n-\tbuf = &buf_;\n-\n \tbuf->infile = filename ? fopen(filename, \"r\") : stdin;\n \tif (!buf->infile)\n \t\treturn -1;\n \treturn 0;\n }\n \n-int buffer_deinit(void)\n+int buffer_deinit(struct line_buffer *buf)\n {\n \tint err;\n \tif (buf->infile == stdin)\n@@ -32,7 +28,7 @@ int buffer_deinit(void)\n }\n \n /* Read a line without trailing newline. */\n-char *buffer_read_line(void)\n+char *buffer_read_line(struct line_buffer *buf)\n {\n \tchar *end;\n \tif (!fgets(buf->line_buffer, sizeof(buf->line_buffer), buf->infile))\n@@ -53,14 +49,14 @@ char *buffer_read_line(void)\n \treturn buf->line_buffer;\n }\n \n-char *buffer_read_string(uint32_t len)\n+char *buffer_read_string(struct line_buffer *buf, uint32_t len)\n {\n \tstrbuf_reset(&buf->blob_buffer);\n \tstrbuf_fread(&buf->blob_buffer, len, buf->infile);\n \treturn ferror(buf->infile) ? NULL : buf->blob_buffer.buf;\n }\n \n-void buffer_copy_bytes(uint32_t len)\n+void buffer_copy_bytes(struct line_buffer *buf, uint32_t len)\n {\n \tchar byte_buffer[COPY_BUFFER_LEN];\n \tuint32_t in;\n@@ -70,13 +66,13 @@ void buffer_copy_bytes(uint32_t len)\n \t\tlen -= in;\n \t\tfwrite(byte_buffer, 1, in, stdout);\n \t\tif (ferror(stdout)) {\n-\t\t\tbuffer_skip_bytes(len);\n+\t\t\tbuffer_skip_bytes(buf, len);\n \t\t\treturn;\n \t\t}\n \t}\n }\n \n-void buffer_skip_bytes(uint32_t len)\n+void buffer_skip_bytes(struct line_buffer *buf, uint32_t len)\n {\n \tchar byte_buffer[COPY_BUFFER_LEN];\n \tuint32_t in;\n@@ -87,7 +83,7 @@ void buffer_skip_bytes(uint32_t len)\n \t}\n }\n \n-void buffer_reset(void)\n+void buffer_reset(struct line_buffer *buf)\n {\n \tstrbuf_release(&buf->blob_buffer);\n }\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex 4ae1133..fb37390 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -12,12 +12,12 @@ struct line_buffer {\n };\n #define LINE_BUFFER_INIT {\"\", STRBUF_INIT, NULL}\n \n-int buffer_init(const char *filename);\n-int buffer_deinit(void);\n-char *buffer_read_line(void);\n-char *buffer_read_string(uint32_t len);\n-void buffer_copy_bytes(uint32_t len);\n-void buffer_skip_bytes(uint32_t len);\n-void buffer_reset(void);\n+int buffer_init(struct line_buffer *buf, const char *filename);\n+int buffer_deinit(struct line_buffer *buf);\n+char *buffer_read_line(struct line_buffer *buf);\n+char *buffer_read_string(struct line_buffer *buf, uint32_t len);\n+void buffer_copy_bytes(struct line_buffer *buf, uint32_t len);\n+void buffer_skip_bytes(struct line_buffer *buf, uint32_t len);\n+void buffer_reset(struct line_buffer *buf);\n \n #endif\ndiff --git a/vcs-svn/line_buffer.txt b/vcs-svn/line_buffer.txt\nindex 8906fb1..f8eaa4d 100644\n--- a/vcs-svn/line_buffer.txt\n+++ b/vcs-svn/line_buffer.txt\n@@ -14,14 +14,15 @@ Calling sequence\n \n The calling program:\n \n+ - initializes a `struct line_buffer` to LINE_BUFFER_INIT\n  - specifies a file to read with `buffer_init`\n  - processes input with `buffer_read_line`, `buffer_read_string`,\n    `buffer_skip_bytes`, and `buffer_copy_bytes`\n  - closes the file with `buffer_deinit`, perhaps to start over and\n    read another file.\n \n-Before exiting, the caller can use `buffer_reset` to deallocate\n-resources for the benefit of profiling tools.\n+When finished, the caller can use `buffer_reset` to deallocate\n+resources.\n \n Functions\n ---------\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 53d0215..3bba0fe 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -30,6 +30,8 @@\n /* Create memory pool for log messages */\n obj_pool_gen(log, char, 4096)\n \n+static struct line_buffer input = LINE_BUFFER_INIT;\n+\n static char* log_copy(uint32_t length, char *log)\n {\n \tchar *buffer;\n@@ -113,14 +115,14 @@ static void read_props(void)\n \tuint32_t key = ~0;\n \tchar *val = NULL;\n \tchar *t;\n-\twhile ((t = buffer_read_line()) && strcmp(t, \"PROPS-END\")) {\n+\twhile ((t = buffer_read_line(&input)) && strcmp(t, \"PROPS-END\")) {\n \t\tif (!strncmp(t, \"K \", 2)) {\n \t\t\tlen = atoi(&t[2]);\n-\t\t\tkey = pool_intern(buffer_read_string(len));\n-\t\t\tbuffer_read_line();\n+\t\t\tkey = pool_intern(buffer_read_string(&input, len));\n+\t\t\tbuffer_read_line(&input);\n \t\t} else if (!strncmp(t, \"V \", 2)) {\n \t\t\tlen = atoi(&t[2]);\n-\t\t\tval = buffer_read_string(len);\n+\t\t\tval = buffer_read_string(&input, len);\n \t\t\tif (key == keys.svn_log) {\n \t\t\t\t/* Value length excludes terminating nul. */\n \t\t\t\trev_ctx.log = log_copy(len + 1, val);\n@@ -135,7 +137,7 @@ static void read_props(void)\n \t\t\t\tnode_ctx.type = REPO_MODE_LNK;\n \t\t\t}\n \t\t\tkey = ~0;\n-\t\t\tbuffer_read_line();\n+\t\t\tbuffer_read_line(&input);\n \t\t}\n \t}\n }\n@@ -177,9 +179,10 @@ static void handle_node(void)\n \t\tnode_ctx.type = node_ctx.srcMode;\n \n \tif (node_ctx.mark)\n-\t\tfast_export_blob(node_ctx.type, node_ctx.mark, node_ctx.textLength);\n+\t\tfast_export_blob(node_ctx.type,\n+\t\t\t\t node_ctx.mark, node_ctx.textLength, &input);\n \telse if (node_ctx.textLength != LENGTH_UNKNOWN)\n-\t\tbuffer_skip_bytes(node_ctx.textLength);\n+\t\tbuffer_skip_bytes(&input, node_ctx.textLength);\n }\n \n static void handle_revision(void)\n@@ -198,7 +201,7 @@ void svndump_read(const char *url)\n \tuint32_t key;\n \n \treset_dump_ctx(pool_intern(url));\n-\twhile ((t = buffer_read_line())) {\n+\twhile ((t = buffer_read_line(&input))) {\n \t\tval = strstr(t, \": \");\n \t\tif (!val)\n \t\t\tcontinue;\n@@ -250,7 +253,7 @@ void svndump_read(const char *url)\n \t\t\tnode_ctx.propLength = atoi(val);\n \t\t} else if (key == keys.content_length) {\n \t\t\tlen = atoi(val);\n-\t\t\tbuffer_read_line();\n+\t\t\tbuffer_read_line(&input);\n \t\t\tif (active_ctx == REV_CTX) {\n \t\t\t\tread_props();\n \t\t\t} else if (active_ctx == NODE_CTX) {\n@@ -258,7 +261,7 @@ void svndump_read(const char *url)\n \t\t\t\tactive_ctx = REV_CTX;\n \t\t\t} else {\n \t\t\t\tfprintf(stderr, \"Unexpected content length header: %\"PRIu32\"\\n\", len);\n-\t\t\t\tbuffer_skip_bytes(len);\n+\t\t\t\tbuffer_skip_bytes(&input, len);\n \t\t\t}\n \t\t}\n \t}\n@@ -270,7 +273,7 @@ void svndump_read(const char *url)\n \n void svndump_init(const char *filename)\n {\n-\tbuffer_init(filename);\n+\tbuffer_init(&input, filename);\n \trepo_init();\n \treset_dump_ctx(~0);\n \treset_rev_ctx(0);\n@@ -285,7 +288,7 @@ void svndump_deinit(void)\n \treset_dump_ctx(~0);\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n-\tif (buffer_deinit())\n+\tif (buffer_deinit(&input))\n \t\tfprintf(stderr, \"Input error\\n\");\n \tif (ferror(stdout))\n \t\tfprintf(stderr, \"Output error\\n\");\n@@ -294,7 +297,7 @@ void svndump_deinit(void)\n void svndump_reset(void)\n {\n \tlog_reset();\n-\tbuffer_reset();\n+\tbuffer_reset(&input);\n \trepo_reset();\n \treset_dump_ctx(~0);\n \treset_rev_ctx(0);\n-- \n1.7.2.3.554.gc9b5c.dirty\n"},{"id":"158827","messageId":"20110103004900.GA30506@burratino","threadId":"26132","inReplyTo":"20101224080505.GA29681@burratino","subject":"Re: [PATCH 0/4] teach vcs-svn/line_buffer to handle multiple input files","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T00:49:00Z","receivedAt":"2011-01-03T00:49:00Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> [*]\n> I am not sure whether this is the right approach for reading from the\n> report-fd.  To avoid deadlock, we cannot issue a blocking read(2)\n> after the trailing newline has been read from an expected line or the\n> nth byte has been read in fixed-length input.\n\nI think this isn't a problem (even though I haven't found any\ncomforting text in the standards themselves).\n\nTo convince myself so, I wrote a couple of new tests.  The change\ndescriptions explain the reasoning.\n\nThese four patches apply on top of\n\n  [PATCH 4/4] vcs-svn: teach line_buffer to handle multiple input files\n\nand are numbered accordingly.  Thoughts welcome, as always.\n\nJonathan Nieder (4):\n  vcs-svn: make test-line-buffer input format more flexible\n  tests: give vcs-svn/line_buffer its own test script\n  vcs-svn: tweak test-line-buffer to not assume line-oriented input\n  t0081 (line-buffer): add buffering test\n\n t/t0080-vcs-svn.sh     |   54 ---------------\n t/t0081-line-buffer.sh |  174 ++++++++++++++++++++++++++++++++++++++++++++++++\n test-line-buffer.c     |   76 +++++++++++++++------\n 3 files changed, 230 insertions(+), 74 deletions(-)\n create mode 100755 t/t0081-line-buffer.sh\n"},{"id":"158828","messageId":"20110103005016.GB30506@burratino","threadId":"26132","inReplyTo":"20110103004900.GA30506@burratino","subject":"[PATCH 5/8] vcs-svn: make test-line-buffer input format more flexible","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T00:50:16Z","receivedAt":"2011-01-03T00:50:16Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Imitate the input format of test-obj-pool to support arbitrary\nsequences of commands rather than alternating read/copy.  This should\nmake it easier to add tests that exercise other line_buffer functions.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n t/t0080-vcs-svn.sh |   18 ++++++++--------\n test-line-buffer.c |   56 +++++++++++++++++++++++++++++++++------------------\n 2 files changed, 45 insertions(+), 29 deletions(-)\n\ndiff --git a/t/t0080-vcs-svn.sh b/t/t0080-vcs-svn.sh\nindex d3225ad..8be9700 100755\n--- a/t/t0080-vcs-svn.sh\n+++ b/t/t0080-vcs-svn.sh\n@@ -85,40 +85,40 @@ test_expect_success 'line buffer' '\n \tprintf \"%s\\n\" \"\" foo >expected6 &&\n \n \ttest-line-buffer <<-\\EOF >actual1 &&\n-\t5\n+\tread 5\n \tHELLO\n \tEOF\n \n \ttest-line-buffer <<-\\EOF >actual2 &&\n-\t0\n+\tread 0\n \n-\t5\n+\tcopy 5\n \tHELLO\n \tEOF\n \n \tq_to_nul <<-\\EOF |\n-\t1\n+\tread 1\n \tQ\n \tEOF\n \ttest-line-buffer >actual3 &&\n \n \tq_to_nul <<-\\EOF |\n-\t0\n+\tread 0\n \n-\t1\n+\tcopy 1\n \tQ\n \tEOF\n \ttest-line-buffer >actual4 &&\n \n \ttest-line-buffer <<-\\EOF >actual5 &&\n-\t5\n+\tread 5\n \tfoo\n \tEOF\n \n \ttest-line-buffer <<-\\EOF >actual6 &&\n-\t0\n+\tread 0\n \n-\t5\n+\tcopy 5\n \tfoo\n \tEOF\n \ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex f9af892..383f35b 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -1,11 +1,5 @@\n /*\n  * test-line-buffer.c: code to exercise the svn importer's input helper\n- *\n- * Input format:\n- *\tnumber NL\n- *\t(number bytes) NL\n- *\tnumber NL\n- *\t...\n  */\n \n #include \"git-compat-util.h\"\n@@ -20,28 +14,50 @@ static uint32_t strtouint32(const char *s)\n \treturn (uint32_t) n;\n }\n \n+static void handle_command(const char *command, const char *arg, struct line_buffer *buf)\n+{\n+\tswitch (*command) {\n+\tcase 'c':\n+\t\tif (!prefixcmp(command, \"copy \")) {\n+\t\t\tbuffer_copy_bytes(buf, strtouint32(arg) + 1);\n+\t\t\treturn;\n+\t\t}\n+\tcase 'r':\n+\t\tif (!prefixcmp(command, \"read \")) {\n+\t\t\tconst char *s = buffer_read_string(buf, strtouint32(arg));\n+\t\t\tprintf(\"%s\\n\", s);\n+\t\t\tbuffer_skip_bytes(buf, 1);\t/* consume newline */\n+\t\t\treturn;\n+\t\t}\n+\tdefault:\n+\t\tdie(\"unrecognized command: %s\", command);\n+\t}\n+}\n+\n+static void handle_line(const char *line, struct line_buffer *stdin_buf)\n+{\n+\tconst char *arg = strchr(line, ' ');\n+\tif (!arg)\n+\t\tdie(\"no argument in line: %s\", line);\n+\thandle_command(line, arg + 1, stdin_buf);\n+}\n+\n int main(int argc, char *argv[])\n {\n-\tstruct line_buffer buf = LINE_BUFFER_INIT;\n+\tstruct line_buffer stdin_buf = LINE_BUFFER_INIT;\n \tchar *s;\n \n \tif (argc != 1)\n-\t\tusage(\"test-line-buffer < input.txt\");\n-\tif (buffer_init(&buf, NULL))\n+\t\tusage(\"test-line-buffer < script\");\n+\n+\tif (buffer_init(&stdin_buf, NULL))\n \t\tdie_errno(\"open error\");\n-\twhile ((s = buffer_read_line(&buf))) {\n-\t\ts = buffer_read_string(&buf, strtouint32(s));\n-\t\tfputs(s, stdout);\n-\t\tfputc('\\n', stdout);\n-\t\tbuffer_skip_bytes(&buf, 1);\n-\t\tif (!(s = buffer_read_line(&buf)))\n-\t\t\tbreak;\n-\t\tbuffer_copy_bytes(&buf, strtouint32(s) + 1);\n-\t}\n-\tif (buffer_deinit(&buf))\n+\twhile ((s = buffer_read_line(&stdin_buf)))\n+\t\thandle_line(s, &stdin_buf);\n+\tif (buffer_deinit(&stdin_buf))\n \t\tdie(\"input error\");\n \tif (ferror(stdout))\n \t\tdie(\"output error\");\n-\tbuffer_reset(&buf);\n+\tbuffer_reset(&stdin_buf);\n \treturn 0;\n }\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158829","messageId":"20110103005107.GC30506@burratino","threadId":"26132","inReplyTo":"20110103004900.GA30506@burratino","subject":"[PATCH 6/8] tests: give vcs-svn/line_buffer its own test script","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T00:51:07Z","receivedAt":"2011-01-03T00:51:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Split the line_buffer test into small pieces and move it to its\nown file as preparation for adding more tests.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n t/t0080-vcs-svn.sh     |   54 --------------------------------------\n t/t0081-line-buffer.sh |   67 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 67 insertions(+), 54 deletions(-)\n create mode 100755 t/t0081-line-buffer.sh\n\ndiff --git a/t/t0080-vcs-svn.sh b/t/t0080-vcs-svn.sh\nindex 8be9700..99a314b 100755\n--- a/t/t0080-vcs-svn.sh\n+++ b/t/t0080-vcs-svn.sh\n@@ -76,60 +76,6 @@ test_expect_success 'obj pool: high-water mark' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success 'line buffer' '\n-\techo HELLO >expected1 &&\n-\tprintf \"%s\\n\" \"\" HELLO >expected2 &&\n-\techo >expected3 &&\n-\tprintf \"%s\\n\" \"\" Q | q_to_nul >expected4 &&\n-\tprintf \"%s\\n\" foo \"\" >expected5 &&\n-\tprintf \"%s\\n\" \"\" foo >expected6 &&\n-\n-\ttest-line-buffer <<-\\EOF >actual1 &&\n-\tread 5\n-\tHELLO\n-\tEOF\n-\n-\ttest-line-buffer <<-\\EOF >actual2 &&\n-\tread 0\n-\n-\tcopy 5\n-\tHELLO\n-\tEOF\n-\n-\tq_to_nul <<-\\EOF |\n-\tread 1\n-\tQ\n-\tEOF\n-\ttest-line-buffer >actual3 &&\n-\n-\tq_to_nul <<-\\EOF |\n-\tread 0\n-\n-\tcopy 1\n-\tQ\n-\tEOF\n-\ttest-line-buffer >actual4 &&\n-\n-\ttest-line-buffer <<-\\EOF >actual5 &&\n-\tread 5\n-\tfoo\n-\tEOF\n-\n-\ttest-line-buffer <<-\\EOF >actual6 &&\n-\tread 0\n-\n-\tcopy 5\n-\tfoo\n-\tEOF\n-\n-\ttest_cmp expected1 actual1 &&\n-\ttest_cmp expected2 actual2 &&\n-\ttest_cmp expected3 actual3 &&\n-\ttest_cmp expected4 actual4 &&\n-\ttest_cmp expected5 actual5 &&\n-\ttest_cmp expected6 actual6\n-'\n-\n test_expect_success 'string pool' '\n \techo a does not equal b >expected.differ &&\n \techo a equals a >expected.match &&\ndiff --git a/t/t0081-line-buffer.sh b/t/t0081-line-buffer.sh\nnew file mode 100755\nindex 0000000..13ac735\n--- /dev/null\n+++ b/t/t0081-line-buffer.sh\n@@ -0,0 +1,67 @@\n+#!/bin/sh\n+\n+test_description=\"Test the svn importer's input handling routines.\n+\"\n+. ./test-lib.sh\n+\n+test_expect_success 'read greeting' '\n+\techo HELLO >expect &&\n+\ttest-line-buffer <<-\\EOF >actual &&\n+\tread 5\n+\tHELLO\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '0-length read, send along greeting' '\n+\tprintf \"%s\\n\" \"\" HELLO >expect &&\n+\ttest-line-buffer <<-\\EOF >actual &&\n+\tread 0\n+\n+\tcopy 5\n+\tHELLO\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'buffer_read_string copes with trailing null byte' '\n+\techo >expect &&\n+\tq_to_nul <<-\\EOF | test-line-buffer >actual &&\n+\tread 1\n+\tQ\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '0-length read, copy null byte' '\n+\tprintf \"%s\\n\" \"\" Q | q_to_nul >expect &&\n+\tq_to_nul <<-\\EOF | test-line-buffer >actual &&\n+\tread 0\n+\n+\tcopy 1\n+\tQ\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'long reads are truncated' '\n+\tprintf \"%s\\n\" foo \"\" >expect &&\n+\ttest-line-buffer <<-\\EOF >actual &&\n+\tread 5\n+\tfoo\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'long copies are truncated' '\n+\tprintf \"%s\\n\" \"\" foo >expect &&\n+\ttest-line-buffer <<-\\EOF >actual &&\n+\tread 0\n+\n+\tcopy 5\n+\tfoo\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158830","messageId":"20110103005228.GD30506@burratino","threadId":"26132","inReplyTo":"20110103004900.GA30506@burratino","subject":"[PATCH 7/8] vcs-svn: tweak test-line-buffer to not assume line-oriented input","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T00:52:28Z","receivedAt":"2011-01-03T00:52:28Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Do not expect an implicit newline after each input record.\nUse a separate command to exercise buffer_skip_bytes.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n t/t0081-line-buffer.sh |   27 +++++++++++++--------------\n test-line-buffer.c     |   10 +++++++---\n 2 files changed, 20 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t0081-line-buffer.sh b/t/t0081-line-buffer.sh\nindex 13ac735..68d6163 100755\n--- a/t/t0081-line-buffer.sh\n+++ b/t/t0081-line-buffer.sh\n@@ -7,45 +7,44 @@ test_description=\"Test the svn importer's input handling routines.\n test_expect_success 'read greeting' '\n \techo HELLO >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n-\tread 5\n+\tread 6\n \tHELLO\n \tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success '0-length read, send along greeting' '\n-\tprintf \"%s\\n\" \"\" HELLO >expect &&\n+\techo HELLO >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n \tread 0\n-\n-\tcopy 5\n+\tcopy 6\n \tHELLO\n \tEOF\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'buffer_read_string copes with trailing null byte' '\n-\techo >expect &&\n+test_expect_success 'buffer_read_string copes with null byte' '\n+\t>expect &&\n \tq_to_nul <<-\\EOF | test-line-buffer >actual &&\n-\tread 1\n+\tread 2\n \tQ\n \tEOF\n \ttest_cmp expect actual\n '\n \n-test_expect_success '0-length read, copy null byte' '\n-\tprintf \"%s\\n\" \"\" Q | q_to_nul >expect &&\n+test_expect_success 'skip, copy null byte' '\n+\techo Q | q_to_nul >expect &&\n \tq_to_nul <<-\\EOF | test-line-buffer >actual &&\n-\tread 0\n-\n-\tcopy 1\n+\tskip 2\n+\tQ\n+\tcopy 2\n \tQ\n \tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success 'long reads are truncated' '\n-\tprintf \"%s\\n\" foo \"\" >expect &&\n+\techo foo >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n \tread 5\n \tfoo\n@@ -56,7 +55,7 @@ test_expect_success 'long reads are truncated' '\n test_expect_success 'long copies are truncated' '\n \tprintf \"%s\\n\" \"\" foo >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n-\tread 0\n+\tread 1\n \n \tcopy 5\n \tfoo\ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex 383f35b..da0bc65 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -19,14 +19,18 @@ static void handle_command(const char *command, const char *arg, struct line_buf\n \tswitch (*command) {\n \tcase 'c':\n \t\tif (!prefixcmp(command, \"copy \")) {\n-\t\t\tbuffer_copy_bytes(buf, strtouint32(arg) + 1);\n+\t\t\tbuffer_copy_bytes(buf, strtouint32(arg));\n \t\t\treturn;\n \t\t}\n \tcase 'r':\n \t\tif (!prefixcmp(command, \"read \")) {\n \t\t\tconst char *s = buffer_read_string(buf, strtouint32(arg));\n-\t\t\tprintf(\"%s\\n\", s);\n-\t\t\tbuffer_skip_bytes(buf, 1);\t/* consume newline */\n+\t\t\tfputs(s, stdout);\n+\t\t\treturn;\n+\t\t}\n+\tcase 's':\n+\t\tif (!prefixcmp(command, \"skip \")) {\n+\t\t\tbuffer_skip_bytes(buf, strtouint32(arg));\n \t\t\treturn;\n \t\t}\n \tdefault:\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158831","messageId":"20110103010716.GE30506@burratino","threadId":"26132","inReplyTo":"20110103004900.GA30506@burratino","subject":"[PATCH 8/8] t0081 (line-buffer): add buffering tests","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T01:07:16Z","receivedAt":"2011-01-03T01:07:16Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"POSIX makes the behavior of read(2) from a pipe fairly clear: a read\nfrom an empty pipe will block until there is data available and any\nother read will not block, prefering to return a partial result.\nLikewise, fread(3) and fgets(3) are clearly specified to act as\nthough implemented by calling fgetc(3) in a simple loop.  But the\nbuffering behavior of fgetc is less clear.\n\nLuckily, no sane platform is going to implement fgetc by calling the\nequivalent of read(2) more than once.  fgetc has to be able to\nreturn without filling its buffer to preserve errno when errors are\nencountered anyway.  So let's assume the simpler behavior (trust) but\nadd some tests to catch insane platforms that violate that when they\ncome (verify).\n\nFirst check that fread can handle a 0-length read from an empty fifo.\nThe writing end of the fifo is opened in advance in a subshell since\neven 0-length reads are allowed to block when the writing end of a\npipe is not open.\n\nNext try short inputs from a pipe that is not filled all the way.\n\nLastly (two tests) try very large inputs from a pipe that will not fit\nin the relevant buffers.  The first of these tests reads a little\nmore than 8192 bytes, which is BUFSIZ (the size of stdio's buffers)\non this Linux machine.  The second reads a little over 64 KiB (the\npipe capacity on Linux) and is not run unless requested by setting\nthe GIT_REMOTE_SVN_TEST_BIG_FILES environment variable.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n t/t0081-line-buffer.sh |  110 +++++++++++++++++++++++++++++++++++++++++++++++-\n test-line-buffer.c     |   22 ++++++++-\n 2 files changed, 128 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t0081-line-buffer.sh b/t/t0081-line-buffer.sh\nindex 68d6163..33a728e 100755\n--- a/t/t0081-line-buffer.sh\n+++ b/t/t0081-line-buffer.sh\n@@ -1,10 +1,76 @@\n #!/bin/sh\n \n test_description=\"Test the svn importer's input handling routines.\n+\n+These tests exercise the line_buffer library, but their real purpose\n+is to check the assumptions that library makes of the platform's input\n+routines.  Processes engaged in bi-directional communication would\n+hang if fread or fgets is too greedy.\n+\n+While at it, check that input of newlines and null bytes are handled\n+correctly.\n \"\n . ./test-lib.sh\n \n-test_expect_success 'read greeting' '\n+test -n \"$GIT_REMOTE_SVN_TEST_BIG_FILES\" && test_set_prereq EXPENSIVE\n+\n+generate_tens_of_lines () {\n+\ttens=$1 &&\n+\tline=$2 &&\n+\n+\ti=0 &&\n+\twhile test $i -lt \"$tens\"\n+\tdo\n+\t\tfor j in a b c d e f g h i j\n+\t\tdo\n+\t\t\techo \"$line\"\n+\t\tdone &&\n+\t\t: $((i = $i + 1)) ||\n+\t\treturn\n+\tdone\n+}\n+\n+long_read_test () {\n+\t: each line is 10 bytes, including newline &&\n+\tline=abcdefghi &&\n+\techo \"$line\" >expect &&\n+\n+\tif ! test_declared_prereq PIPE\n+\tthen\n+\t\techo >&4 \"long_read_test: need to declare PIPE prerequisite\"\n+\t\treturn 127\n+\tfi &&\n+\ttens_of_lines=$(($1 / 100 + 1)) &&\n+\tlines=$(($tens_of_lines * 10)) &&\n+\treadsize=$((($lines - 1) * 10 + 3)) &&\n+\tcopysize=7 &&\n+\trm -f input &&\n+\tmkfifo input &&\n+\t{\n+\t\t{\n+\t\t\tgenerate_tens_of_lines $tens_of_lines \"$line\" &&\n+\t\t\tsleep 100\n+\t\t} >input &\n+\t} &&\n+\ttest-line-buffer input <<-EOF >output &&\n+\tread $readsize\n+\tcopy $copysize\n+\tEOF\n+\tkill $! &&\n+\ttest_line_count = $lines output &&\n+\ttail -n 1 <output >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+test_expect_success 'setup: have pipes?' '\n+      rm -f frob &&\n+      if mkfifo frob\n+      then\n+\t\ttest_set_prereq PIPE\n+      fi\n+'\n+\n+test_expect_success 'hello world' '\n \techo HELLO >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n \tread 6\n@@ -13,6 +79,21 @@ test_expect_success 'read greeting' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success PIPE '0-length read, no input available' '\n+\t>expect &&\n+\trm -f input &&\n+\tmkfifo input &&\n+\t{\n+\t\tsleep 100 >input &\n+\t} &&\n+\ttest-line-buffer input <<-\\EOF >actual &&\n+\tread 0\n+\tcopy 0\n+\tEOF\n+\tkill $! &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success '0-length read, send along greeting' '\n \techo HELLO >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n@@ -23,6 +104,33 @@ test_expect_success '0-length read, send along greeting' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success PIPE '1-byte read, no input available' '\n+\tprintf \"%s\" ab >expect &&\n+\trm -f input &&\n+\tmkfifo input &&\n+\t{\n+\t\t{\n+\t\t\tprintf \"%s\" a &&\n+\t\t\tprintf \"%s\" b &&\n+\t\t\tsleep 100\n+\t\t} >input &\n+\t} &&\n+\ttest-line-buffer input <<-\\EOF >actual &&\n+\tread 1\n+\tcopy 1\n+\tEOF\n+\tkill $! &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success PIPE 'long read (around 8192 bytes)' '\n+\tlong_read_test 8192\n+'\n+\n+test_expect_success PIPE,EXPENSIVE 'longer read (around 65536 bytes)' '\n+\tlong_read_test 65536\n+'\n+\n test_expect_success 'buffer_read_string copes with null byte' '\n \t>expect &&\n \tq_to_nul <<-\\EOF | test-line-buffer >actual &&\ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex da0bc65..ec19b13 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -49,15 +49,31 @@ static void handle_line(const char *line, struct line_buffer *stdin_buf)\n int main(int argc, char *argv[])\n {\n \tstruct line_buffer stdin_buf = LINE_BUFFER_INIT;\n+\tstruct line_buffer file_buf = LINE_BUFFER_INIT;\n+\tstruct line_buffer *input = &stdin_buf;\n+\tconst char *filename;\n \tchar *s;\n \n-\tif (argc != 1)\n-\t\tusage(\"test-line-buffer < script\");\n+\tif (argc == 1)\n+\t\tfilename = NULL;\n+\telse if (argc == 2)\n+\t\tfilename = argv[1];\n+\telse\n+\t\tusage(\"test-line-buffer [file] < script\");\n \n \tif (buffer_init(&stdin_buf, NULL))\n \t\tdie_errno(\"open error\");\n+\tif (filename) {\n+\t\tif (buffer_init(&file_buf, filename))\n+\t\t\tdie_errno(\"error opening %s\", filename);\n+\t\tinput = &file_buf;\n+\t}\n+\n \twhile ((s = buffer_read_line(&stdin_buf)))\n-\t\thandle_line(s, &stdin_buf);\n+\t\thandle_line(s, input);\n+\n+\tif (filename && buffer_deinit(&file_buf))\n+\t\tdie(\"error reading from %s\", filename);\n \tif (buffer_deinit(&stdin_buf))\n \t\tdie(\"input error\");\n \tif (ferror(stdout))\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158833","messageId":"20110103013457.GB30791@burratino","threadId":"26132","inReplyTo":"20110103010716.GE30506@burratino","subject":"Re: [PATCH 8/8] t0081 (line-buffer): add buffering tests","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T01:34:57Z","receivedAt":"2011-01-03T01:34:57Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> First check that fread can handle a 0-length read from an empty fifo.\n> The writing end of the fifo is opened in advance in a subshell since\n> even 0-length reads are allowed to block when the writing end of a\n> pipe is not open.\n\nErm, what I mean here is that open(O_RDONLY) will block until the\nwriting end has been opened.\n"},{"id":"158834","messageId":"20110103030328.GA10143@burratino","threadId":"26132","inReplyTo":"20110103004900.GA30506@burratino","subject":"[PATCHES 9-12/12] line_buffer: more wrappers around stdio functions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T03:03:28Z","receivedAt":"2011-01-03T03:03:28Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"This might be the last batch of line_buffer patches for the moment[1].\nIt introduces wrappers around strbuf_fread, fgetc, fdopen, and tmpfile\nso the line_buffer lib can do what those functions do.\n\nThe motivation in the background is delta application.  We need:\n\n - binary-safe input, since svn (like most version control systems)\n   gets used to store binary files from time to time.\n - character-oriented input (fgetc) as a basic convenience, needed in\n   particular for reading variable-length integers in svndiff blocks.\n - input from file descriptors, to read information requested from\n   fast-import (in particular, delta preimages).\n - temporary files, to store the result of delta application and\n   retrieve its length so svn-fe can write\n\n\tdata <length>\n\t... delta application result ...\n\n   to fast-import.\n\nThe ideas behind the third and fourth patches (patches 11 and 12) are\nfrom David Barr's earlier work in the same direction.  Patches are\nbased against\n\n  [PATCH 8/8] t0081 (line-buffer): add buffering tests\n\nso we can reuse some of the testing infrastructure.  They are numbered\naccordingly for easy application.\n\nEach patch introduces new API.  I would be happy if you can find an\ninfelicity or two so we can fix the functions now before people get\nused to them.\n\nJonathan Nieder (4):\n  vcs-svn: add binary-safe read function\n  vcs-svn: allow character-oriented input\n  vcs-svn: allow input from file descriptor\n  vcs-svn: teach line_buffer about temporary files\n\n t/t0081-line-buffer.sh  |   27 +++++++++++++++++++++++++++\n test-line-buffer.c      |   21 ++++++++++++++++++---\n vcs-svn/line_buffer.c   |   43 +++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/line_buffer.h   |   10 +++++++++-\n vcs-svn/line_buffer.txt |   31 +++++++++++++++++++++++++++----\n 5 files changed, 124 insertions(+), 8 deletions(-)\n\n[1] There's another patch to use 64-bit offsets but the API changes\nare more obvious.\n"},{"id":"158835","messageId":"20110103030546.GB10143@burratino","threadId":"26132","inReplyTo":"20110103030328.GA10143@burratino","subject":"[PATCH 09/12] vcs-svn: add binary-safe read function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T03:05:46Z","receivedAt":"2011-01-03T03:05:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"buffer_read_string works well for non line-oriented input except for\none problem: it does not tell the caller how many bytes were actually\nwritten.  This means that unless one is very careful about checking\nfor errors (and eof) the calling program cannot tell the difference\nbetween the string \"foo\" followed by an early end of file and the\nstring \"foo\\0bar\\0baz\".\n\nSo introduce a variant that reports the length, too, a thinner wrapper\naround strbuf_fread.  Its result is written to a strbuf so the caller\ndoes not need to keep track of the number of bytes read.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n t/t0081-line-buffer.sh |   18 ++++++++++++++++++\n test-line-buffer.c     |   10 ++++++++++\n vcs-svn/line_buffer.c  |    6 ++++++\n vcs-svn/line_buffer.h  |    1 +\n 4 files changed, 35 insertions(+), 0 deletions(-)\n\ndiff --git a/t/t0081-line-buffer.sh b/t/t0081-line-buffer.sh\nindex 33a728e..a8eeb20 100755\n--- a/t/t0081-line-buffer.sh\n+++ b/t/t0081-line-buffer.sh\n@@ -151,6 +151,15 @@ test_expect_success 'skip, copy null byte' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'read null byte' '\n+\techo \">QhelloQ\" | q_to_nul >expect &&\n+\tq_to_nul <<-\\EOF | test-line-buffer >actual &&\n+\tbinary 8\n+\tQhelloQ\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'long reads are truncated' '\n \techo foo >expect &&\n \ttest-line-buffer <<-\\EOF >actual &&\n@@ -171,4 +180,13 @@ test_expect_success 'long copies are truncated' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'long binary reads are truncated' '\n+\techo \">foo\" >expect &&\n+\ttest-line-buffer <<-\\EOF >actual &&\n+\tbinary 5\n+\tfoo\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_done\ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex ec19b13..19bf2d4 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -3,6 +3,7 @@\n  */\n \n #include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n #include \"vcs-svn/line_buffer.h\"\n \n static uint32_t strtouint32(const char *s)\n@@ -17,6 +18,15 @@ static uint32_t strtouint32(const char *s)\n static void handle_command(const char *command, const char *arg, struct line_buffer *buf)\n {\n \tswitch (*command) {\n+\tcase 'b':\n+\t\tif (!prefixcmp(command, \"binary \")) {\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addch(&sb, '>');\n+\t\t\tbuffer_read_binary(buf, &sb, strtouint32(arg));\n+\t\t\tfwrite(sb.buf, 1, sb.len, stdout);\n+\t\t\tstrbuf_release(&sb);\n+\t\t\treturn;\n+\t\t}\n \tcase 'c':\n \t\tif (!prefixcmp(command, \"copy \")) {\n \t\t\tbuffer_copy_bytes(buf, strtouint32(arg));\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex 806932b..661b007 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -56,6 +56,12 @@ char *buffer_read_string(struct line_buffer *buf, uint32_t len)\n \treturn ferror(buf->infile) ? NULL : buf->blob_buffer.buf;\n }\n \n+void buffer_read_binary(struct line_buffer *buf,\n+\t\t\t\tstruct strbuf *sb, uint32_t size)\n+{\n+\tstrbuf_fread(sb, size, buf->infile);\n+}\n+\n void buffer_copy_bytes(struct line_buffer *buf, uint32_t len)\n {\n \tchar byte_buffer[COPY_BUFFER_LEN];\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex fb37390..0c2d3d9 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -16,6 +16,7 @@ int buffer_init(struct line_buffer *buf, const char *filename);\n int buffer_deinit(struct line_buffer *buf);\n char *buffer_read_line(struct line_buffer *buf);\n char *buffer_read_string(struct line_buffer *buf, uint32_t len);\n+void buffer_read_binary(struct line_buffer *buf, struct strbuf *sb, uint32_t len);\n void buffer_copy_bytes(struct line_buffer *buf, uint32_t len);\n void buffer_skip_bytes(struct line_buffer *buf, uint32_t len);\n void buffer_reset(struct line_buffer *buf);\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158836","messageId":"20110103030632.GC10143@burratino","threadId":"26132","inReplyTo":"20110103030328.GA10143@burratino","subject":"[PATCH 10/12] vcs-svn: allow character-oriented input","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T03:06:32Z","receivedAt":"2011-01-03T03:06:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"buffer_read_char can be used in place of buffer_read_string(1) to\navoid consuming valuable static buffer space.  The delta applier will\nuse this to read variable-length integers one byte at a time.\n\nUnderneath, it is fgetc, wrapped so the line_buffer library can\nmaintain its role as gatekeeper of input.\n\nLater it might be worth checking if fgetc_unlocked is faster ---\nmost line_buffer functions are not thread-safe anyway.\n\nHelpd-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n vcs-svn/line_buffer.c |    5 +++++\n vcs-svn/line_buffer.h |    1 +\n 2 files changed, 6 insertions(+), 0 deletions(-)\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex 661b007..37ec56e 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -27,6 +27,11 @@ int buffer_deinit(struct line_buffer *buf)\n \treturn err;\n }\n \n+int buffer_read_char(struct line_buffer *buf)\n+{\n+\treturn fgetc(buf->infile);\n+}\n+\n /* Read a line without trailing newline. */\n char *buffer_read_line(struct line_buffer *buf)\n {\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex 0c2d3d9..0a59c73 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -16,6 +16,7 @@ int buffer_init(struct line_buffer *buf, const char *filename);\n int buffer_deinit(struct line_buffer *buf);\n char *buffer_read_line(struct line_buffer *buf);\n char *buffer_read_string(struct line_buffer *buf, uint32_t len);\n+int buffer_read_char(struct line_buffer *buf);\n void buffer_read_binary(struct line_buffer *buf, struct strbuf *sb, uint32_t len);\n void buffer_copy_bytes(struct line_buffer *buf, uint32_t len);\n void buffer_skip_bytes(struct line_buffer *buf, uint32_t len);\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158837","messageId":"20110103030938.GD10143@burratino","threadId":"26132","inReplyTo":"20110103030328.GA10143@burratino","subject":"[PATCH 11/12] vcs-svn: allow input from file descriptor","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T03:09:38Z","receivedAt":"2011-01-03T03:09:38Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Based-on-patch-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nNo fd_buffer.  This is essentially the way David did it in the first\nplace; sorry to have doubted.\n\n t/t0081-line-buffer.sh  |    9 +++++++++\n test-line-buffer.c      |   11 ++++++++---\n vcs-svn/line_buffer.c   |    8 ++++++++\n vcs-svn/line_buffer.h   |    1 +\n vcs-svn/line_buffer.txt |    9 +++++----\n 5 files changed, 31 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t0081-line-buffer.sh b/t/t0081-line-buffer.sh\nindex a8eeb20..550fad0 100755\n--- a/t/t0081-line-buffer.sh\n+++ b/t/t0081-line-buffer.sh\n@@ -131,6 +131,15 @@ test_expect_success PIPE,EXPENSIVE 'longer read (around 65536 bytes)' '\n \tlong_read_test 65536\n '\n \n+test_expect_success 'read from file descriptor' '\n+\trm -f input &&\n+\techo hello >expect &&\n+\techo hello >input &&\n+\techo copy 6 |\n+\ttest-line-buffer \"&4\" 4<input >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'buffer_read_string copes with null byte' '\n \t>expect &&\n \tq_to_nul <<-\\EOF | test-line-buffer >actual &&\ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex 19bf2d4..25b20b9 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -69,13 +69,18 @@ int main(int argc, char *argv[])\n \telse if (argc == 2)\n \t\tfilename = argv[1];\n \telse\n-\t\tusage(\"test-line-buffer [file] < script\");\n+\t\tusage(\"test-line-buffer [file | &fd] < script\");\n \n \tif (buffer_init(&stdin_buf, NULL))\n \t\tdie_errno(\"open error\");\n \tif (filename) {\n-\t\tif (buffer_init(&file_buf, filename))\n-\t\t\tdie_errno(\"error opening %s\", filename);\n+\t\tif (*filename == '&') {\n+\t\t\tif (buffer_fdinit(&file_buf, strtouint32(filename + 1)))\n+\t\t\t\tdie_errno(\"error opening fd %s\", filename + 1);\n+\t\t} else {\n+\t\t\tif (buffer_init(&file_buf, filename))\n+\t\t\t\tdie_errno(\"error opening %s\", filename);\n+\t\t}\n \t\tinput = &file_buf;\n \t}\n \ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex 37ec56e..e29a81a 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -17,6 +17,14 @@ int buffer_init(struct line_buffer *buf, const char *filename)\n \treturn 0;\n }\n \n+int buffer_fdinit(struct line_buffer *buf, int fd)\n+{\n+\tbuf->infile = fdopen(fd, \"r\");\n+\tif (!buf->infile)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int buffer_deinit(struct line_buffer *buf)\n {\n \tint err;\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex 0a59c73..630d83c 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -13,6 +13,7 @@ struct line_buffer {\n #define LINE_BUFFER_INIT {\"\", STRBUF_INIT, NULL}\n \n int buffer_init(struct line_buffer *buf, const char *filename);\n+int buffer_fdinit(struct line_buffer *buf, int fd);\n int buffer_deinit(struct line_buffer *buf);\n char *buffer_read_line(struct line_buffer *buf);\n char *buffer_read_string(struct line_buffer *buf, uint32_t len);\ndiff --git a/vcs-svn/line_buffer.txt b/vcs-svn/line_buffer.txt\nindex f8eaa4d..4e8fb71 100644\n--- a/vcs-svn/line_buffer.txt\n+++ b/vcs-svn/line_buffer.txt\n@@ -27,10 +27,11 @@ resources.\n Functions\n ---------\n \n-`buffer_init`::\n-\tOpen the named file for input.  If filename is NULL,\n-\tstart reading from stdin.  On failure, returns -1 (with\n-\terrno indicating the nature of the failure).\n+`buffer_init`, `buffer_fdinit`::\n+\tOpen the named file or file descriptor for input.\n+\tbuffer_init(buf, NULL) prepares to read from stdin.\n+\tOn failure, returns -1 (with errno indicating the nature\n+\tof the failure).\n \n `buffer_deinit`::\n \tStop reading from the current file (closing it unless\n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"158838","messageId":"20110103031059.GE10143@burratino","threadId":"26132","inReplyTo":"20110103030328.GA10143@burratino","subject":"[PATCH 12/12] vcs-svn: teach line_buffer about temporary files","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-03T03:10:59Z","receivedAt":"2011-01-03T03:10:59Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"It can sometimes be useful to write information temporarily to file,\nto read back later.  These functions allow a program to use the\nline_buffer facilities when doing so.\n\nIt works like this:\n\n 1. find a unique filename with buffer_tmpfile_init.\n 2. rewind with buffer_tmpfile_rewind.  This returns a stdio\n    handle for writing.\n 3. when finished writing, declare so with\n    buffer_tmpfile_prepare_to_read.  The return value indicates\n    how many bytes were written.\n 4. read whatever portion of the file is needed.\n 5. if finished, remove the temporary file with buffer_deinit.\n    otherwise, go back to step 2,\n\nThe svn support would use this to buffer the postimage from delta\napplication until the length is known and fast-import can receive\nthe resulting blob.\n\nBased-on-patch-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nDavid, this is another piece of infrastructure from early svn-fe3\nhistory.  I've cleaned up the API a little bit but the idea is the\nsame.  Thank you.\n\n vcs-svn/line_buffer.c   |   24 ++++++++++++++++++++++++\n vcs-svn/line_buffer.h   |    7 ++++++-\n vcs-svn/line_buffer.txt |   22 ++++++++++++++++++++++\n 3 files changed, 52 insertions(+), 1 deletions(-)\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex e29a81a..aedf105 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -25,6 +25,14 @@ int buffer_fdinit(struct line_buffer *buf, int fd)\n \treturn 0;\n }\n \n+int buffer_tmpfile_init(struct line_buffer *buf)\n+{\n+\tbuf->infile = tmpfile();\n+\tif (!buf->infile)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int buffer_deinit(struct line_buffer *buf)\n {\n \tint err;\n@@ -35,6 +43,22 @@ int buffer_deinit(struct line_buffer *buf)\n \treturn err;\n }\n \n+FILE *buffer_tmpfile_rewind(struct line_buffer *buf)\n+{\n+\trewind(buf->infile);\n+\treturn buf->infile;\n+}\n+\n+long buffer_tmpfile_prepare_to_read(struct line_buffer *buf)\n+{\n+\tlong pos = ftell(buf->infile);\n+\tif (pos < 0)\n+\t\treturn error(\"ftell error: %s\", strerror(errno));\n+\tif (fseek(buf->infile, 0, SEEK_SET))\n+\t\treturn error(\"seek error: %s\", strerror(errno));\n+\treturn pos;\n+}\n+\n int buffer_read_char(struct line_buffer *buf)\n {\n \treturn fgetc(buf->infile);\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex 630d83c..96ce966 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -15,12 +15,17 @@ struct line_buffer {\n int buffer_init(struct line_buffer *buf, const char *filename);\n int buffer_fdinit(struct line_buffer *buf, int fd);\n int buffer_deinit(struct line_buffer *buf);\n+void buffer_reset(struct line_buffer *buf);\n+\n+int buffer_tmpfile_init(struct line_buffer *buf);\n+FILE *buffer_tmpfile_rewind(struct line_buffer *buf);\t/* prepare to write. */\n+long buffer_tmpfile_prepare_to_read(struct line_buffer *buf);\n+\n char *buffer_read_line(struct line_buffer *buf);\n char *buffer_read_string(struct line_buffer *buf, uint32_t len);\n int buffer_read_char(struct line_buffer *buf);\n void buffer_read_binary(struct line_buffer *buf, struct strbuf *sb, uint32_t len);\n void buffer_copy_bytes(struct line_buffer *buf, uint32_t len);\n void buffer_skip_bytes(struct line_buffer *buf, uint32_t len);\n-void buffer_reset(struct line_buffer *buf);\n \n #endif\ndiff --git a/vcs-svn/line_buffer.txt b/vcs-svn/line_buffer.txt\nindex 4e8fb71..e89cc41 100644\n--- a/vcs-svn/line_buffer.txt\n+++ b/vcs-svn/line_buffer.txt\n@@ -24,6 +24,28 @@ The calling program:\n When finished, the caller can use `buffer_reset` to deallocate\n resources.\n \n+Using temporary files\n+---------------------\n+\n+Temporary files provide a place to store data that should not outlive\n+the calling program.  A program\n+\n+ - initializes a `struct line_buffer` to LINE_BUFFER_INIT\n+ - requests a temporary file with `buffer_tmpfile_init`\n+ - acquires an output handle by calling `buffer_tmpfile_rewind`\n+ - uses standard I/O functions like `fprintf` and `fwrite` to fill\n+   the temporary file\n+ - declares writing is over with `buffer_tmpfile_prepare_to_read`\n+ - can re-read what was written with `buffer_read_line`,\n+   `buffer_read_string`, and so on\n+ - can reuse the temporary file by calling `buffer_tmpfile_rewind`\n+   again\n+ - removes the temporary file with `buffer_deinit`, perhaps to\n+   reuse the line_buffer for some other file.\n+\n+When finished, the calling program can use `buffer_reset` to deallocate\n+resources.\n+\n Functions\n ---------\n \n-- \n1.7.4.rc0.580.g89dc.dirty\n"},{"id":"159783","messageId":"20110122064235.GA10304@burratino","threadId":"26132","inReplyTo":"20110103031059.GE10143@burratino","subject":"[FYI/PATCH] vcs-svn: give control over temporary file names","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-22T06:42:35Z","receivedAt":"2011-01-22T06:42:35Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Allow users of the line_buffer library to specify what directory and\nfilename to use for temporary files.  For example:\n\n\tstruct line_buffer tmp = LINE_BUFFER_INIT;\n\tif (buffer_tmpfile_init(&tmp, \".git\", \"svnfe_blob.XXXXXX\"))\n\t\tdie_errno(\"opening temporary file\");\n\t...\n\tif (buffer_deinit(&tmp))\n\t\tdie_errno(\"removing temporary file\");\n\nOn Windows, something like this is needed if users without write\npermission to the root directory are to be able to use temporary\nfiles.\n\nUnlike the implementation using tmpfile, this would not take\ncare of automatically removing the temporary file on exit.  The\nuser is responsible for now for installing appropriate signal and\natexit handlers to take care of that.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nI have an odd feeling this will be necessary at some point but I'm not\nsure if it's a good idea or not.  Although tmpfile(3) is a limited\ninterface, it's pretty much exactly what we want.\n\nSee [1] for tmpfile on Windows.\n\nThis is just to get the idea out there.  I don't think this patch's\nmoment has come (though I'd be interested in your thoughts either\nway).\n\n[1] http://msdn.microsoft.com/en-us/library/x8x7sakw.aspx\n\n vcs-svn/line_buffer.c |   35 ++++++++++++++++++++++++++++++-----\n vcs-svn/line_buffer.h |    6 ++++--\n 2 files changed, 34 insertions(+), 7 deletions(-)\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex aedf105..4131e08 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -3,7 +3,7 @@\n  * See LICENSE for details.\n  */\n \n-#include \"git-compat-util.h\"\n+#include \"cache.h\"\n #include \"line_buffer.h\"\n #include \"strbuf.h\"\n \n@@ -25,12 +25,33 @@ int buffer_fdinit(struct line_buffer *buf, int fd)\n \treturn 0;\n }\n \n-int buffer_tmpfile_init(struct line_buffer *buf)\n+int buffer_tmpfile_init(struct line_buffer *buf,\n+\t\t\tconst char *dirname, const char *pattern)\n {\n-\tbuf->infile = tmpfile();\n-\tif (!buf->infile)\n-\t\treturn -1;\n+\tint fd, saved_errno;\n+\tint mode = 0444;\t/* just remove write permission. */\n+\tstrbuf_addstr(&buf->temp_filename, dirname);\n+\tstrbuf_addch(&buf->temp_filename, '/');\n+\tstrbuf_addstr(&buf->temp_filename, pattern);\n+\n+\tfd = git_mkstemp_mode(buf->temp_filename.buf, mode);\n+\tif (fd < 0) {\n+\t\tsaved_errno = errno;\n+\t\tgoto fail_mktemp;\n+\t}\n+\tbuf->infile = fdopen(fd, \"r+\");\n+\tif (!buf->infile) {\n+\t\tsaved_errno = errno;\n+\t\tgoto fail_fdopen;\n+\t}\n \treturn 0;\n+\n+fail_fdopen:\n+\tclose(fd);\n+fail_mktemp:\n+\tstrbuf_reset(&buf->temp_filename);\n+\terrno = saved_errno;\n+\treturn -1;\n }\n \n int buffer_deinit(struct line_buffer *buf)\n@@ -40,6 +61,9 @@ int buffer_deinit(struct line_buffer *buf)\n \t\treturn ferror(buf->infile);\n \terr = ferror(buf->infile);\n \terr |= fclose(buf->infile);\n+\tif (buf->temp_filename.len)\n+\t\terr |= unlink_or_warn(buf->temp_filename.buf);\n+\tstrbuf_reset(&buf->temp_filename);\n \treturn err;\n }\n \n@@ -129,4 +153,5 @@ void buffer_skip_bytes(struct line_buffer *buf, uint32_t len)\n void buffer_reset(struct line_buffer *buf)\n {\n \tstrbuf_release(&buf->blob_buffer);\n+\tstrbuf_release(&buf->temp_filename);\n }\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex 96ce966..bd1a621 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -9,15 +9,17 @@ struct line_buffer {\n \tchar line_buffer[LINE_BUFFER_LEN];\n \tstruct strbuf blob_buffer;\n \tFILE *infile;\n+\tstruct strbuf temp_filename;\n };\n-#define LINE_BUFFER_INIT {\"\", STRBUF_INIT, NULL}\n+#define LINE_BUFFER_INIT {\"\", STRBUF_INIT, NULL, STRBUF_INIT}\n \n int buffer_init(struct line_buffer *buf, const char *filename);\n int buffer_fdinit(struct line_buffer *buf, int fd);\n int buffer_deinit(struct line_buffer *buf);\n void buffer_reset(struct line_buffer *buf);\n \n-int buffer_tmpfile_init(struct line_buffer *buf);\n+int buffer_tmpfile_init(struct line_buffer *buf,\n+\t\tconst char *directory, const char *pattern);\n FILE *buffer_tmpfile_rewind(struct line_buffer *buf);\t/* prepare to write. */\n long buffer_tmpfile_prepare_to_read(struct line_buffer *buf);\n \n-- \n1.7.4.rc2\n"},{"id":"162309","messageId":"20110226114435.GB12231@elie","threadId":"26132","inReplyTo":"20101224080505.GA29681@burratino","subject":"[PULL svn-fe] fast-import 'ls', line-buffer changes","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-02-26T11:44:42Z","receivedAt":"2011-02-26T11:44:42Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Junio,\n\nPlease pull\n\n  git://repo.or.cz/git/jrn.git svn-fe\n\nto get the following changes on top of master + the old jn/svn-fe.\n\nThese are patches from the threads $gmane/164146 and $gmane/167536,\nproviding some plumbing to support incremental import in svn-fe.  They\nare probably far from perfect but they seem to work okay so far, and\nI'd be fine with putting any fixes on top of them.\n\nThoughts, suggestions, etc welcome as always.\n\nDavid Barr (1):\n      fast-import: add 'ls' command\n\nJonathan Nieder (13):\n      vcs-svn: eliminate global byte_buffer\n      vcs-svn: replace buffer_read_string memory pool with a strbuf\n      vcs-svn: collect line_buffer data in a struct\n      vcs-svn: teach line_buffer to handle multiple input files\n      vcs-svn: make test-line-buffer input format more flexible\n      tests: give vcs-svn/line_buffer its own test script\n      vcs-svn: tweak test-line-buffer to not assume line-oriented input\n      t0081 (line-buffer): add buffering tests\n      vcs-svn: add binary-safe read function\n      vcs-svn: allow character-oriented input\n      vcs-svn: allow input from file descriptor\n      vcs-svn: teach line_buffer about temporary files\n      Merge commit 'jn/svn-fe' of git://github.com/gitster/git into svn-fe\n\n Documentation/git-fast-import.txt |   63 +++++++++++-\n fast-import.c                     |  162 +++++++++++++++++++++++++++++-\n t/t0080-vcs-svn.sh                |   54 ----------\n t/t0081-line-buffer.sh            |  201 +++++++++++++++++++++++++++++++++++++\n t/t9010-svn-fe.sh                 |    1 +\n t/t9300-fast-import.sh            |   92 +++++++++++++++--\n test-line-buffer.c                |   90 +++++++++++++----\n vcs-svn/fast_export.c             |    6 +-\n vcs-svn/fast_export.h             |    5 +-\n vcs-svn/line_buffer.c             |  105 +++++++++++++-------\n vcs-svn/line_buffer.h             |   33 +++++--\n vcs-svn/line_buffer.txt           |   36 ++++++-\n vcs-svn/svndump.c                 |   23 +++--\n 13 files changed, 722 insertions(+), 149 deletions(-)\n"},{"id":"162310","messageId":"34AB7FDB-87F9-4C46-AC90-D15A55103F02@cordelta.com","threadId":"26132","inReplyTo":"20110226114435.GB12231@elie","subject":"Re: [PULL svn-fe] fast-import 'ls', line-buffer changes","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2011-02-26T12:03:33Z","receivedAt":"2011-02-26T12:03:33Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"\n\nHi,\n\n> Please pull\n> \n>  git://repo.or.cz/git/jrn.git svn-fe\n> \n> to get the following changes on top of master + the old jn/svn-fe.\n> \n> These are patches from the threads $gmane/164146 and $gmane/167536,\n> providing some plumbing to support incremental import in svn-fe.  They\n> are probably far from perfect but they seem to work okay so far, and\n> I'd be fine with putting any fixes on top of them.\n> \n> Thoughts, suggestions, etc welcome as always.\n\nI'll try to put together a new topic based on this branch.\nThe target is incremental imports and removal of svn-fe's internal tree representation.\nThe patches that I intend to pick are as follows:\n\nDavid Barr (9):\nvcs-svn: drop obj_pool.h\nvcs-svn: drop trp.h\nvcs-svn: drop string_pool\nvcs-svn: factor out usage of string_pool\nvcs-svn: implement perfect hash for top-level keys\nvcs-svn: implement perfect hash for node-prop keys\nvcs-svn: avoid using ls command twice\nvcs-svn: pass paths through to fast-import\nvcs-svn: use mark from previous import for parent commit\n\nJonathan Nieder (7):\nvcs-svn: handle filenames with dq correctly\nvcs-svn: add a comment before each commit\nvcs-svn: simplify repo_modify_path and repo_copy\nvcs-svn: prepare to eliminate repo_tree structure\nvcs-svn: save marks for imported commits\nvcs-svn: use higher mark numbers for blobs\nvcs-svn: introduce repo_read_path to check the content at a path\n"},{"id":"162395","messageId":"7vhbbolm5q.fsf@alter.siamese.dyndns.org","threadId":"26132","inReplyTo":"20110226114435.GB12231@elie","subject":"Re: [PULL svn-fe] fast-import 'ls', line-buffer changes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-02-28T06:15:45Z","receivedAt":"2011-02-28T06:15:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> Hi Junio,\n>\n> Please pull\n>\n>   git://repo.or.cz/git/jrn.git svn-fe\n>\n> to get the following changes on top of master + the old jn/svn-fe.\n>\n> These are patches from the threads $gmane/164146 and $gmane/167536,\n> providing some plumbing to support incremental import in svn-fe.  They\n> are probably far from perfect but they seem to work okay so far, and\n> I'd be fine with putting any fixes on top of them.\n>\n> Thoughts, suggestions, etc welcome as always.\n\nAs I was not involved in the thread heavily, I'll just pull this into\n'master', trusting that the responsible parties will be able to handle\npotential fallouts to fast-import users, if any, promptly.\n\n... Yikes.  I said the above and then my build for \"master\" breaks with\n\n  fast-import.c: In function 'dereference':\n  fast-import.c:2885: error: pointer of type 'void *' used in arithmetic\n  fast-import.c:2890: error: pointer of type 'void *' used in arithmetic\n\nforcing me to redo all three integration branches.  What an un-fun.\n\nNot pulled.\n\n fast-import.c |    4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 6c37b84..cfddb7a 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2882,12 +2882,12 @@ static struct object_entry *dereference(struct object_entry *oe,\n \tswitch (oe->type) {\n \tcase OBJ_TAG:\n \t\tif (size < 40 + strlen(\"object \") ||\n-\t\t    get_sha1_hex(buf + strlen(\"object \"), sha1))\n+\t\t    get_sha1_hex((char *)buf + strlen(\"object \"), sha1))\n \t\t\tdie(\"Invalid SHA1 in tag: %s\", command_buf.buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tif (size < 40 + strlen(\"tree \") ||\n-\t\t    get_sha1_hex(buf + strlen(\"tree \"), sha1))\n+\t\t    get_sha1_hex((char *)buf + strlen(\"tree \"), sha1))\n \t\t\tdie(\"Invalid SHA1 in commit: %s\", command_buf.buf);\n \t}\n \n"},{"id":"162477","messageId":"20110228213210.GD1942@elie","threadId":"26132","inReplyTo":"7vhbbolm5q.fsf@alter.siamese.dyndns.org","subject":"[PATCH svn-fe] fast-import: make code \"-Wpointer-arith\" clean","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-02-28T21:32:10Z","receivedAt":"2011-02-28T21:32:10Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"The dereference() function to peel a tree-ish and find the underlying\ntree expects arithmetic to (void *) to work on byte addresses.  We\nshould be reading the text of objects through a char * anyway.\n\nNoticed-by: Junio C Hamano <gitster@pobox.com>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nJunio C Hamano wrote:\n\n> ... Yikes.  I said the above and then my build for \"master\" breaks with\n>\n>   fast-import.c: In function 'dereference':\n>   fast-import.c:2885: error: pointer of type 'void *' used in arithmetic\n>   fast-import.c:2890: error: pointer of type 'void *' used in arithmetic\n\nThis should fix it, I suppose?  A\n\n\t-std=c99 -O3 -Wall -W\n\t-Wno-sign-compare\n\t-Wno-unused-parameter\n\t-Wno-missing-field-initializers\n\t-Wno-empty-body\n\t-Wno-pointer-to-int-cast\n\t-Wno-type-limits\n\t-Wno-unused-but-set-variable\n\t-Wold-style-definition -Wpointer-arith -Wvla\n\t-Wdeclaration-after-statement -Werror\n\nbuild passes, except for an \"unsigned long expire\" in builtin-reflog\nthat confuses this copy of gcc.\n\n fast-import.c |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 6c37b84..e1268b8 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2848,7 +2848,7 @@ static struct object_entry *dereference(struct object_entry *oe,\n \t\t\t\t\tunsigned char sha1[20])\n {\n \tunsigned long size;\n-\tvoid *buf = NULL;\n+\tchar *buf = NULL;\n \tif (!oe) {\n \t\tenum object_type type = sha1_object_info(sha1, NULL);\n \t\tif (type < 0)\n-- \n1.7.4.1\n"},{"id":"162479","messageId":"AANLkTikD8qQKdhXN9dx+hk8r=eGRBtc3eRSFU2vDxmsP@mail.gmail.com","threadId":"26132","inReplyTo":"20110228213210.GD1942@elie","subject":"Re: [PATCH svn-fe] fast-import: make code \"-Wpointer-arith\" clean","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-02-28T21:36:33Z","receivedAt":"2011-02-28T21:36:33Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Mon, Feb 28, 2011 at 22:32, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> This should fix it, I suppose?  A\n>\n>        -std=c99 -O3 -Wall -W\n>        -Wno-sign-compare\n>        -Wno-unused-parameter\n>        -Wno-missing-field-initializers\n>        -Wno-empty-body\n>        -Wno-pointer-to-int-cast\n>        -Wno-type-limits\n>        -Wno-unused-but-set-variable\n>        -Wold-style-definition -Wpointer-arith -Wvla\n>        -Wdeclaration-after-statement -Werror\n>\n> build passes, except for an \"unsigned long expire\" in builtin-reflog\n> that confuses this copy of gcc.\n\nHow come this slipped by unnoticed (except by Junio)? Is the default\nMakefile not strict enough?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"162483","messageId":"7voc5vhl2a.fsf@alter.siamese.dyndns.org","threadId":"26132","inReplyTo":"AANLkTikD8qQKdhXN9dx+hk8r=eGRBtc3eRSFU2vDxmsP@mail.gmail.com","subject":"Re: [PATCH svn-fe] fast-import: make code \"-Wpointer-arith\" clean","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-02-28T22:05:17Z","receivedAt":"2011-02-28T22:05:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sverre Rabbelier <srabbelier@gmail.com> writes:\n\n> How come this slipped by unnoticed (except by Junio)? Is the default\n> Makefile not strict enough?\n\nNo, it's just me being a tad overly paranoid.\n"},{"id":"162491","messageId":"20110228231511.GF1942@elie","threadId":"26132","inReplyTo":"20110228213210.GD1942@elie","subject":"Re: [PATCH svn-fe] fast-import: make code \"-Wpointer-arith\" clean","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-02-28T23:15:11Z","receivedAt":"2011-02-28T23:15:11Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> This should fix it, I suppose?\n\nPushed out.  Thanks again.\n"},{"id":"162499","messageId":"7vipw3fz9f.fsf@alter.siamese.dyndns.org","threadId":"26132","inReplyTo":"20110228231511.GF1942@elie","subject":"Re: [PATCH svn-fe] fast-import: make code \"-Wpointer-arith\" clean","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-03-01T00:41:32Z","receivedAt":"2011-03-01T00:41:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> Jonathan Nieder wrote:\n>\n>> This should fix it, I suppose?\n>\n> Pushed out.  Thanks again.\n\nThanks, pulled.\n"}]}