git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 14/19] pkt-line: provide a LARGE_PACKET_MAX static buffer

From
Jeff King <peff@peff.net>
Date
Feb 20, 2013, 20:02 UTC
Message-ID
<20130220200257.GN25647@sigill.intra.peff.net>
In-Reply-To
<20130220195147.GA25332@sigill.intra.peff.net>

Most of the callers of packet_read_line just read into a static 1000-byte buffer (callers which handle arbitrary binary data already use LARGE_PACKET_MAX). This works fine in practice, because:

  1. The only variable-sized data in these lines is a ref
     name, and refs tend to be a lot shorter than 1000
     characters.
  2. When sending ref lines, git-core always limits itself
     to 1000 byte packets.

However, the only limit given in the protocol specification in Documentation/technical/protocol-common.txt is LARGE_PACKET_MAX; the 1000 byte limit is mentioned only in pack-protocol.txt, and then only describing what we write, not as a specific limit for readers.

This patch lets us bump the 1000-byte limit to LARGE_PACKET_MAX. Even though git-core will never write a packet where this makes a difference, there are two good reasons to do this:

  1. Other git implementations may have followed
     protocol-common.txt and used a larger maximum size. We
     don't bump into it in practice because it would involve
     very long ref names.
  2. We may want to increase the 1000-byte limit one day.
     Since packets are transferred before any capabilities,
     it's difficult to do this in a backwards-compatible
     way. But if we bump the size of buffer the readers can
     handle, eventually older versions of git will be
     obsolete enough that we can justify bumping the
     writers, as well. We don't have plans to do this
     anytime soon, but there is no reason not to start the
     clock ticking now.

Just bumping all of the reading bufs to LARGE_PACKET_MAX would waste memory. Instead, since most readers just read into a temporary buffer anyway, let's provide a single static buffer that all callers can use. We can further wrap this detail away by having the packet_read_line wrapper just use the buffer transparently and return a pointer to the static storage. That covers most of the cases, and the remaining ones already read into their own LARGE_PACKET_MAX buffers.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/archive.c        | 15 +++++++--------
 builtin/fetch-pack.c     |  7 +++----
 builtin/receive-pack.c   |  6 +++---
 builtin/upload-archive.c |  7 ++-----
 connect.c                |  4 ++--
 daemon.c                 |  4 ++--
 fetch-pack.c             | 12 ++++++------
 pkt-line.c               |  9 +++++++--
 pkt-line.h               |  9 +++++++--
 send-pack.c              |  7 +++----
 upload-pack.c            | 12 +++++-------
 11 files changed, 47 insertions(+), 45 deletions(-)
diff --git a/builtin/archive.c b/builtin/archive.c
index d381ac4..49178f1 100644
--- a/builtin/archive.c
+++ b/builtin/archive.c
@@ -27,8 +27,8 @@ static int run_remote_archiver(int argc, const char **argv,
 			       const char *remote, const char *exec,
 			       const char *name_hint)
 {
-	char buf[LARGE_PACKET_MAX];
-	int fd[2], i, len, rv;
+	char *buf;
+	int fd[2], i, rv;
 	struct transport *transport;
 	struct remote *_remote;
 
@@ -53,19 +53,18 @@ static int run_remote_archiver(int argc, const char **argv,
 		packet_write(fd[1], "argument %s\n", argv[i]);
 	packet_flush(fd[1]);
 
-	len = packet_read_line(fd[0], buf, sizeof(buf));
-	if (!len)
+	buf = packet_read_line(fd[0], NULL);
+	if (!buf)
 		die(_("git archive: expected ACK/NAK, got EOF"));
 	if (strcmp(buf, "ACK")) {
-		if (len > 5 && !prefixcmp(buf, "NACK "))
+		if (!prefixcmp(buf, "NACK "))
 			die(_("git archive: NACK %s"), buf + 5);
-		if (len > 4 && !prefixcmp(buf, "ERR "))
+		if (!prefixcmp(buf, "ERR "))
 			die(_("remote error: %s"), buf + 4);
 		die(_("git archive: protocol error"));
 	}
 
-	len = packet_read_line(fd[0], buf, sizeof(buf));
-	if (len)
+	if (packet_read_line(fd[0], NULL))
 		die(_("git archive: expected a flush"));
 
 	/* Now, start reading from fd[0] and spit it out to stdout */
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index f73664f..c21cc2c 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -100,12 +100,11 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 			/* in stateless RPC mode we use pkt-line to read
 			 * from stdin, until we get a flush packet
 			 */
-			static char line[1000];
 			for (;;) {
-				int n = packet_read_line(0, line, sizeof(line));
-				if (!n)
+				char *line = packet_read_line(0, NULL);
+				if (!line)
 					break;
-				string_list_append(&sought, xmemdupz(line, n));
+				string_list_append(&sought, xstrdup(line));
 			}
 		}
 		else {
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 6679e63..ccebd74 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -754,14 +754,14 @@ static struct command *read_head_info(void)
 	struct command *commands = NULL;
 	struct command **p = &commands;
 	for (;;) {
-		static char line[1000];
+		char *line;
 		unsigned char old_sha1[20], new_sha1[20];
 		struct command *cmd;
 		char *refname;
 		int len, reflen;
 
-		len = packet_read_line(0, line, sizeof(line));
-		if (!len)
+		line = packet_read_line(0, &len);
+		if (!line)
 			break;
 		if (len < 83 ||
 		    line[40] != ' ' ||
diff --git a/builtin/upload-archive.c b/builtin/upload-archive.c
index 7d367b5..2a94675 100644
--- a/builtin/upload-archive.c
+++ b/builtin/upload-archive.c
@@ -21,8 +21,6 @@ int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix)
 {
 	struct argv_array sent_argv = ARGV_ARRAY_INIT;
 	const char *arg_cmd = "argument ";
-	char buf[4096];
-	int len;
 
 	if (argc != 2)
 		usage(upload_archive_usage);
@@ -33,9 +31,8 @@ int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix)
 	/* put received options in sent_argv[] */
 	argv_array_push(&sent_argv, "git-upload-archive");
 	for (;;) {
-		/* This will die if not enough free space in buf */
-		len = packet_read_line(0, buf, sizeof(buf));
-		if (len == 0)
+		char *buf = packet_read_line(0, NULL);
+		if (!buf)
 			break;	/* got a flush */
 		if (sent_argv.argc > MAX_ARGS)
 		    die("Too many options (>%d)", MAX_ARGS - 1);
diff --git a/connect.c b/connect.c
index fe8eb01..611ffb4 100644
--- a/connect.c
+++ b/connect.c
@@ -72,11 +72,11 @@ struct ref **get_remote_heads(int in, struct ref **list,
 	for (;;) {
 		struct ref *ref;
 		unsigned char old_sha1[20];
-		static char buffer[1000];
 		char *name;
 		int len, name_len;
+		char *buffer = packet_buffer;
 
-		len = packet_read(in, buffer, sizeof(buffer),
+		len = packet_read(in, packet_buffer, sizeof(packet_buffer),
 				  PACKET_READ_GENTLE_ON_EOF |
 				  PACKET_READ_CHOMP_NEWLINE);
 		if (len < 0)
diff --git a/daemon.c b/daemon.c
index 4f5cd61..3f70e79 100644
--- a/daemon.c
+++ b/daemon.c
@@ -604,7 +604,7 @@ static int execute(void)
 
 static int execute(void)
 {
-	static char line[1000];
+	char *line = packet_buffer;
 	int pktlen, len, i;
 	char *addr = getenv("REMOTE_ADDR"), *port = getenv("REMOTE_PORT");
 
@@ -612,7 +612,7 @@ static int execute(void)
 		loginfo("Connection from %s:%s", addr, port);
 
 	alarm(init_timeout ? init_timeout : timeout);
-	pktlen = packet_read(0, line, sizeof(line), 0);
+	pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);
 	alarm(0);
 
 	len = strlen(line);
diff --git a/fetch-pack.c b/fetch-pack.c
index f830db2..66ff9ad 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -172,8 +172,8 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)
 		 * shallow and unshallow commands every time there
 		 * is a block of have lines exchanged.
 		 */
-		char line[1000];
-		while (packet_read_line(fd, line, sizeof(line))) {
+		char *line;
+		while ((line = packet_read_line(fd, NULL))) {
 			if (!prefixcmp(line, "shallow "))
 				continue;
 			if (!prefixcmp(line, "unshallow "))
@@ -215,8 +215,8 @@ static enum ack_type get_ack(int fd, unsigned char *result_sha1)
 
 static enum ack_type get_ack(int fd, unsigned char *result_sha1)
 {
-	static char line[1000];
-	int len = packet_read_line(fd, line, sizeof(line));
+	int len;
+	char *line = packet_read_line(fd, &len);
 
 	if (!len)
 		die("git fetch-pack: expected ACK/NAK, got EOF");
@@ -346,11 +346,11 @@ static int find_common(struct fetch_pack_args *args,
 	state_len = req_buf.len;
 
 	if (args->depth > 0) {
-		char line[1024];
+		char *line;
 		unsigned char sha1[20];
 
 		send_request(args, fd[1], &req_buf);
-		while (packet_read_line(fd[0], line, sizeof(line))) {
+		while ((line = packet_read_line(fd[0], NULL))) {
 			if (!prefixcmp(line, "shallow ")) {
 				if (get_sha1_hex(line + 8, sha1))
 					die("invalid shallow line: %s", line);
diff --git a/pkt-line.c b/pkt-line.c
index dc11c40..55fb688 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "pkt-line.h"
 
+char packet_buffer[LARGE_PACKET_MAX];
 static const char *packet_trace_prefix = "git";
 static const char trace_key[] = "GIT_TRACE_PACKET";
 
@@ -174,9 +175,13 @@ int packet_read_line(int fd, char *buffer, unsigned size)
 	return len;
 }
 
-int packet_read_line(int fd, char *buffer, unsigned size)
+char *packet_read_line(int fd, int *len_p)
 {
-	return packet_read(fd, buffer, size, PACKET_READ_CHOMP_NEWLINE);
+	int len = packet_read(fd, packet_buffer, sizeof(packet_buffer),
+			      PACKET_READ_CHOMP_NEWLINE);
+	if (len_p)
+		*len_p = len;
+	return len ? packet_buffer : NULL;
 }
 
 int packet_get_line(struct strbuf *out,
diff --git a/pkt-line.h b/pkt-line.h
index 6927ea5..fa93e32 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -54,12 +54,17 @@ int packet_read_line(int fd, char *buffer, unsigned size);
 
 /*
  * Convenience wrapper for packet_read that is not gentle, and sets the
- * CHOMP_NEWLINE option.
+ * CHOMP_NEWLINE option. The return value is NULL for a flush packet,
+ * and otherwise points to a static buffer (that may be overwritten by
+ * subsequent calls). If the size parameter is not NULL, the length of the
+ * packet is written to it.
  */
-int packet_read_line(int fd, char *buffer, unsigned size);
+char *packet_read_line(int fd, int *size);
+
 
 #define DEFAULT_PACKET_MAX 1000
 #define LARGE_PACKET_MAX 65520
+extern char packet_buffer[LARGE_PACKET_MAX];
 
 int packet_get_line(struct strbuf *out, char **src_buf, size_t *src_len);
 
diff --git a/send-pack.c b/send-pack.c
index 8c230bf..7d172ef 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -106,9 +106,8 @@ static int receive_status(int in, struct ref *refs)
 static int receive_status(int in, struct ref *refs)
 {
 	struct ref *hint;
-	char line[1000];
 	int ret = 0;
-	int len = packet_read_line(in, line, sizeof(line));
+	char *line = packet_read_line(in, NULL);
 	if (prefixcmp(line, "unpack "))
 		return error("did not receive remote status");
 	if (strcmp(line, "unpack ok")) {
@@ -119,8 +118,8 @@ static int receive_status(int in, struct ref *refs)
 	while (1) {
 		char *refname;
 		char *msg;
-		len = packet_read_line(in, line, sizeof(line));
-		if (!len)
+		line = packet_read_line(in, NULL);
+		if (!line)
 			break;
 		if (prefixcmp(line, "ok ") && prefixcmp(line, "ng ")) {
 			error("invalid ref status from remote: %s", line);
diff --git a/upload-pack.c b/upload-pack.c
index 7446cb7..bc241ba 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -408,7 +408,6 @@ static int get_common_commits(void)
 
 static int get_common_commits(void)
 {
-	static char line[1000];
 	unsigned char sha1[20];
 	char last_hex[41];
 	int got_common = 0;
@@ -418,10 +417,10 @@ static int get_common_commits(void)
 	save_commit_buffer = 0;
 
 	for (;;) {
-		int len = packet_read_line(0, line, sizeof(line));
+		char *line = packet_read_line(0, NULL);
 		reset_timeout();
 
-		if (!len) {
+		if (!line) {
 			if (multi_ack == 2 && got_common
 			    && !got_other && ok_to_give_up()) {
 				sent_ready = 1;
@@ -567,8 +566,7 @@ static void receive_needs(void)
 static void receive_needs(void)
 {
 	struct object_array shallows = OBJECT_ARRAY_INIT;
-	static char line[1000];
-	int len, depth = 0;
+	int depth = 0;
 	int has_non_tip = 0;
 
 	shallow_nr = 0;
@@ -576,9 +574,9 @@ static void receive_needs(void)
 		struct object *o;
 		const char *features;
 		unsigned char sha1_buf[20];
-		len = packet_read_line(0, line, sizeof(line));
+		char *line = packet_read_line(0, NULL);
 		reset_timeout();
-		if (!len)
+		if (!line)
 			break;
 
 		if (!prefixcmp(line, "shallow ")) {

-- 
1.8.2.rc0.9.g352092c
Previous: Jeff KingNext: Jeff King
Message 34 of 40 in “[PATCHv3 0/19] pkt-line cleanups and fixes”
  1. Jeff KingFeb 20, 2013
  2. 01/19 upload-pack: use get_sha1_hex to parse "shallow" linesJeff King, Feb 20, 2013
  3. 02/19 upload-pack: do not add duplicate objects to shallow listJeff King, Feb 20, 2013
  4. 03/19 upload-pack: remove packet debugging harnessJeff King, Feb 20, 2013
  5. 04/19 fetch-pack: fix out-of-bounds buffer offset in get_ackJeff King, Feb 20, 2013
  6. 05/19 send-pack: prefer prefixcmp over memcmp in receive_statusJeff King, Feb 20, 2013
  7. 06/19 upload-archive: do not copy repo nameJeff King, Feb 20, 2013
  8. 07/19 upload-archive: use argv_array to store client argumentsJeff King, Feb 20, 2013
  9. 08/19 write_or_die: raise SIGPIPE when we get EPIPEJeff King, Feb 20, 2013
  10. Jonathan NiederFeb 20, 2013
  11. Jeff KingFeb 20, 2013
  12. Jonathan NiederFeb 20, 2013
  13. Jeff KingFeb 20, 2013
  14. Jonathan NiederFeb 20, 2013
  15. Jeff KingFeb 20, 2013
  16. Junio C HamanoFeb 20, 2013
  17. [BUG] MSVC: error box when interrupting `gitlog` by quitting lessMarat Radchenko, Mar 28, 2014
  18. Marat RadchenkoMar 28, 2014
  19. Jeff KingMar 28, 2014
  20. Marat RadchenkoMar 28, 2014
  21. Jeff KingMar 28, 2014
  22. Johannes SixtMar 28, 2014
  23. MSVC: link in invalidcontinue.obj for better POSIX compatibilityMarat Radchenko, Mar 28, 2014
  24. Junio C HamanoMar 28, 2014
  25. Marat RadchenkoMar 28, 2014
  26. Junio C HamanoMar 28, 2014
  27. MSVC: link in invalidcontinue.obj for better POSIX compatibilityMarat Radchenko, Mar 28, 2014
  28. Junio C HamanoMar 28, 2014
  29. 09/19 pkt-line: move a misplaced commentJeff King, Feb 20, 2013
  30. 10/19 pkt-line: drop safe_write functionJeff King, Feb 20, 2013
  31. 11/19 pkt-line: provide a generic reading function with optionsJeff King, Feb 20, 2013
  32. 12/19 pkt-line: teach packet_read_line to chomp newlinesJeff King, Feb 20, 2013
  33. 13/19 pkt-line: move LARGE_PACKET_MAX definition from sidebandJeff King, Feb 20, 2013
  34. 14/19 pkt-line: provide a LARGE_PACKET_MAX static bufferJeff King, Feb 20, 2013
  35. 15/19 pkt-line: share buffer/descriptor reading implementationJeff King, Feb 20, 2013
  36. Eric SunshineFeb 22, 2013
  37. 16/19 teach get_remote_heads to read from a memory bufferJeff King, Feb 20, 2013
  38. 17/19 remote-curl: pass buffer straight to get_remote_headsJeff King, Feb 20, 2013
  39. 18/19 remote-curl: move ref-parsing code up in fileJeff King, Feb 20, 2013
  40. 19/19 remote-curl: always parse incoming refsJeff King, Feb 20, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.