git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC/PATCH 1/2] bundle: allowing to read from an unseekable fd

From
Phil Hord <phil.hord@gmail.com>
Date
Oct 14, 2011, 12:31 UTC
Message-ID
<CABURp0qnhNQ06pNHw_6_PkT2+WNPdQ1WzUHF8p6q3Hx3JkoC8A@mail.gmail.com>
In-Reply-To
<7vpqi034l0.fsf@alter.siamese.dyndns.org>
On Thu, Oct 13, 2011 at 6:32 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 33 quoted lines
> The current code opens a given file with fopen(), reads it until the end
> of the header and runs ftell(), and reopens the same file with open() and
> seeks to skip the header. This structure makes it hard to retarget the
> code to read from input that is not seekable, such as a network socket.
>
> This patch by itself does not reach that goal yet, but I think it is a
> right step in that direction.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>
>  * It would be nice if we can avoid byte-by-byte reading from the file
>   descriptor by over-reading into the strbuf and pass the remainder to
>   the caller of read_bundle_header(), but I suspect that it would require
>   us to carry the "here is the remainder from the previous read" buffer
>   around throughout the transport layer. Parsing of the header wouldn't
>   be performance critical compared to the computation cost of actually
>   reading the rest of the bundle, hopefully, so...
>
>  bundle.c |   99 ++++++++++++++++++++++++++++++++++++++++----------------------
>  1 files changed, 64 insertions(+), 35 deletions(-)
>
> diff --git a/bundle.c b/bundle.c
> index f48fd7d..3aa715c 100644
> --- a/bundle.c
> +++ b/bundle.c
> @@ -23,49 +23,78 @@ static void add_to_ref_list(const unsigned char *sha1, const char *name,
>        list->nr++;
>  }
>
> -/* returns an fd */
> +/* Eventually this should go to strbuf.[ch] */
> +static int strbuf_readline_fd(struct strbuf *sb, int fd)

A size limiter would be useful here. Since it's readline, maybe the limit can even be hardcoded.

Without a limit, calling this with something stupid like "/dev/zero" will consume all memory and never return.

Phil
Previous: Junio C Hamano
Message 5 of 5 in “bundle: allowing to read from an unseekable fd”
  1. 1/2 bundle: allowing to read from an unseekable fdJunio C Hamano, Oct 13, 2011
  2. 2/2 bundle: add parse_bundle_header() helper functionJunio C Hamano, Oct 13, 2011
  3. Phil HordOct 14, 2011
  4. Junio C HamanoOct 14, 2011
  5. Phil HordOct 14, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.