git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/2] Function stripspace now gets a buffer instead file descriptors.

From
CRCarlos Rica <jasampler@gmail.com>
Date
Jul 12, 2007, 00:14 UTC
Message-ID
<1b46aba20707111714v63bf921dh15000e2629bcf260@mail.gmail.com>
In-Reply-To
<7vd4yy1svw.fsf@assigned-by-dhcp.cox.net>
2007/7/12, Junio C Hamano <gitster@pobox.com>:
Show 11 quoted lines
> Carlos Rica <jasampler@gmail.com> writes:
> > @@ -28,52 +26,67 @@ static int cleanup(char *line, int len)
> >   * Remove empty lines from the beginning and end
> >   * and also trailing spaces from every line.
> >   *
> > + * Note that the buffer will not be null-terminated.
> > + *
>
> The name of the sentinel character '\0' is NUL, not null (which
> is a different word, used to call a pointer that points
> nowhere).  The buffer will not be "NUL-terminated".
Thank you Junio, I will use it on the future.
Show 20 quoted lines
> >  int cmd_stripspace(int argc, const char **argv, const char *prefix)
> >  {
> > -     stripspace(stdin, stdout, 0);
> > +     char *buffer;
> > +     unsigned long size;
> > +
> > +     size = 1024;
> > +     buffer = xmalloc(size);
> > +     if (read_pipe(0, &buffer, &size))
> > +             die("could not read the input");
>
> The command used to be capable of streaming and filtering a few
> hundred gigabytes of text on a machine with small address space,
> as it operated one line at a time, but now it cannot as it has
> to hold everything in core before starting.
>
> I do not think we miss that loss of capability too much, but I
> wonder if we can be a bit more clever about it, perhaps feeding
> a chunk at a time.  Not a very strong request, but just
> wondering if it is an easy change.

I did those changes because I was needing those tests that I had written before in order to develop the function. After that, we now can restore the previous function with file descriptors to make it capable of filter a few hundred gigabytes of text, provided that the text does not have long long lines on it.

Indeed, the implementation for composing a tag (header, cleaned message and optional signature) in "builtin-tag.c", now pass it to the function write_sha1_file as a buffer on memory, so it won't support sizes bigger than memory available on the system. Messages should not be so big, but I don't know how to limit those.

Previous: Junio C Hamano
Message 7 of 7 in “Function stripspace now gets a buffer instead file descriptors.”
  1. 1/2 Function stripspace now gets a buffer instead file descriptors.Carlos Rica, Jul 11, 2007
  2. Johannes SchindelinJul 11, 2007
  3. Junio C HamanoJul 11, 2007
  4. Bill LearJul 11, 2007
  5. Carlos RicaJul 11, 2007
  6. Junio C HamanoJul 12, 2007
  7. Carlos RicaJul 12, 2007

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.