git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v5 07/15] pkt-line: add functions to read/write flush terminated packet streams

From
larsxschneider@gmail.com <larsxschneider@gmail.com>
Date
Aug 10, 2016, 13:04 UTC
Message-ID
<20160810130411.12419-8-larsxschneider@gmail.com>
In-Reply-To
<20160810130411.12419-1-larsxschneider@gmail.com>
From: Lars Schneider <larsxschneider@gmail.com>

packet_write_stream_with_flush_from_fd() and packet_write_stream_with_flush_from_buf() write a stream of packets. All content packets use the maximal packet size except for the last one. After the last content packet a `flush` control packet is written.

packet_read_till_flush() reads arbitary sized packets until it detects a `flush` packet.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 pkt-line.c | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 pkt-line.h |  7 +++++
 2 files changed, 96 insertions(+)
diff --git a/pkt-line.c b/pkt-line.c
index e6a0924..4b526e1 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -181,6 +181,45 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)
 	va_end(args);
 }
 
+int packet_write_stream_with_flush_from_fd(int fd_in, int fd_out)
+{
+	int err = 0;
+	ssize_t bytes_to_write;
+	while (!err) {
+		bytes_to_write = xread(fd_in, packet_write_buffer, PKTLINE_DATA_MAXLEN);
+		if (bytes_to_write < 0)
+			return COPY_READ_ERROR;
+		if (bytes_to_write == 0)
+			break;
+		if (bytes_to_write > PKTLINE_DATA_MAXLEN)
+			return COPY_WRITE_ERROR;
+		err = packet_write_gently(fd_out, packet_write_buffer, bytes_to_write);
+	}
+	if (!err)
+		err = packet_flush_gently(fd_out);
+	return err;
+}
+
+int packet_write_stream_with_flush_from_buf(const char *src_in, size_t len, int fd_out)
+{
+	int err = 0;
+	size_t bytes_written = 0;
+	size_t bytes_to_write;
+	while (!err) {
+		if ((len - bytes_written) > PKTLINE_DATA_MAXLEN)
+			bytes_to_write = PKTLINE_DATA_MAXLEN;
+		else
+			bytes_to_write = len - bytes_written;
+		if (bytes_to_write == 0)
+			break;
+		err = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);
+		bytes_written += bytes_to_write;
+	}
+	if (!err)
+		err = packet_flush_gently(fd_out);
+	return err;
+}
+
 static int get_packet_data(int fd, char **src_buf, size_t *src_size,
 			   void *dst, unsigned size, int options)
 {
@@ -290,3 +329,53 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)
 {
 	return packet_read_line_generic(-1, src, src_len, dst_len);
 }
+
+ssize_t packet_read_till_flush(int fd_in, struct strbuf *sb_out)
+{
+	int len, ret;
+	int options = PACKET_READ_GENTLE_ON_EOF;
+	char linelen[4];
+
+	size_t oldlen = sb_out->len;
+	size_t oldalloc = sb_out->alloc;
+
+	for (;;) {
+		/* Read packet header */
+		ret = get_packet_data(fd_in, NULL, NULL, linelen, 4, options);
+		if (ret < 0)
+			goto done;
+		len = packet_length(linelen);
+		if (len < 0)
+			die("protocol error: bad line length character: %.4s", linelen);
+		if (!len) {
+			/* Found a flush packet - Done! */
+			packet_trace("0000", 4, 0);
+			break;
+		}
+		len -= 4;
+
+		/* Read packet content */
+		strbuf_grow(sb_out, len);
+		ret = get_packet_data(fd_in, NULL, NULL, sb_out->buf + sb_out->len, len, options);
+		if (ret < 0)
+			goto done;
+
+		if (ret != len) {
+			error("protocol error: incomplete read (expected %d, got %d)", len, ret);
+			goto done;
+		}
+
+		packet_trace(sb_out->buf + sb_out->len, len, 0);
+		sb_out->len += len;
+	}
+
+done:
+	if (ret < 0) {
+		if (oldalloc == 0)
+			strbuf_release(sb_out);
+		else
+			strbuf_setlen(sb_out, oldlen);
+		return ret;  /* unexpected EOF */
+	}
+	return sb_out->len - oldlen;
+}
diff --git a/pkt-line.h b/pkt-line.h
index b6c8bcd..89063ee 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -25,6 +25,8 @@ void packet_buf_flush(struct strbuf *buf);
 void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
 int packet_flush_gently(int fd);
 int packet_write_gently_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
+int packet_write_stream_with_flush_from_fd(int fd_in, int fd_out);
+int packet_write_stream_with_flush_from_buf(const char *src_in, size_t len, int fd_out);
 
 /*
  * Read a packetized line into the buffer, which must be at least size bytes
@@ -77,6 +79,11 @@ char *packet_read_line(int fd, int *size);
  */
 char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
 
+/*
+ * Reads a stream of variable sized packets until a flush packet is detected.
+ */
+ssize_t packet_read_till_flush(int fd_in, struct strbuf *sb_out);
+
 #define DEFAULT_PACKET_MAX 1000
 #define LARGE_PACKET_MAX 65520
 #define PKTLINE_DATA_MAXLEN (LARGE_PACKET_MAX - 4)
-- 
2.9.2
Previous: larsxschneider@gmail.comNext: larsxschneider@gmail.com
Message 27 of 50 in “Git filter protocol”
  1. 00/15 Git filter protocollarsxschneider@gmail.com, Aug 10, 2016
  2. 14/15 convert: add filter.<driver>.process optionlarsxschneider@gmail.com, Aug 10, 2016
  3. Stefan BellerAug 12, 2016
  4. Jeff KingAug 12, 2016
  5. Stefan BellerAug 12, 2016
  6. Lars SchneiderAug 12, 2016
  7. Junio C HamanoAug 12, 2016
  8. Lars SchneiderAug 12, 2016
  9. Junio C HamanoAug 12, 2016
  10. Lars SchneiderAug 12, 2016
  11. Stefan BellerAug 12, 2016
  12. Lars SchneiderAug 12, 2016
  13. 08/15 pkt-line: rename packet_write() to packet_write_fmt()larsxschneider@gmail.com, Aug 10, 2016
  14. 05/15 pkt-line: add packet_write_gently_fmt()larsxschneider@gmail.com, Aug 10, 2016
  15. Jeff KingAug 10, 2016
  16. Junio C HamanoAug 10, 2016
  17. Lars SchneiderAug 10, 2016
  18. Junio C HamanoAug 10, 2016
  19. Lars SchneiderAug 10, 2016
  20. Jeff KingAug 10, 2016
  21. 15/15 read-cache: make sure file handles are not inherited by child processeslarsxschneider@gmail.com, Aug 10, 2016
  22. Johannes SchindelinAug 18, 2016
  23. 01/15 pkt-line: extract set_packet_header()larsxschneider@gmail.com, Aug 10, 2016
  24. 09/15 pack-protocol: fix maximum pkt-line sizelarsxschneider@gmail.com, Aug 10, 2016
  25. 06/15 pkt-line: add packet_flush_gently()larsxschneider@gmail.com, Aug 10, 2016
  26. 10/15 convert: quote filter names in error messageslarsxschneider@gmail.com, Aug 10, 2016
  27. 07/15 pkt-line: add functions to read/write flush terminated packet streamslarsxschneider@gmail.com, Aug 10, 2016
  28. 12/15 convert: generate large test files only oncelarsxschneider@gmail.com, Aug 10, 2016
  29. 11/15 convert: modernize testslarsxschneider@gmail.com, Aug 10, 2016
  30. 04/15 pkt-line: add packet_write_gently()larsxschneider@gmail.com, Aug 10, 2016
  31. Jeff KingAug 10, 2016
  32. Lars SchneiderAug 10, 2016
  33. Jeff KingAug 10, 2016
  34. Junio C HamanoAug 10, 2016
  35. Lars SchneiderAug 10, 2016
  36. Junio C HamanoAug 10, 2016
  37. Lars SchneiderAug 10, 2016
  38. 02/15 pkt-line: call packet_trace() only if a packet is actually sendlarsxschneider@gmail.com, Aug 10, 2016
  39. Jeff KingAug 10, 2016
  40. Lars SchneiderAug 10, 2016
  41. Jeff KingAug 10, 2016
  42. Lars SchneiderAug 10, 2016
  43. Jeff KingAug 10, 2016
  44. 13/15 convert: make apply_filter() adhere to standard Git error handlinglarsxschneider@gmail.com, Aug 10, 2016
  45. 03/15 pkt-line: add `gentle` parameter to format_packet()larsxschneider@gmail.com, Aug 10, 2016
  46. Jeff KingAug 10, 2016
  47. Lars SchneiderAug 10, 2016
  48. Jeff KingAug 10, 2016
  49. Lars SchneiderAug 10, 2016
  50. Jeff KingAug 10, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.