git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2] bulk-checkin: only support blobs in index_bulk_checkin

From
EBEric W. Biederman <ebiederm@gmail.com>
Date
Sep 20, 2023, 12:24 UTC
Message-ID
<87zg1h58xa.fsf@gmail.froward.int.ebiederm.org>
In-Reply-To
<xmqqr0mtcosy.fsf@gitster.g>
Junio C Hamano <gitster@pobox.com> writes:
Show 12 quoted lines
> "Eric W. Biederman" <ebiederm@gmail.com> writes:
>
>> As far as I can tell this extra pass defeats most of the purpose of
>> streaming, and it is much easier to implement with in memory buffers.
>
> The purpose of streaming being the ability to hash and compute the
> object name without having to hold the entirety of the object, I am
> not sure the above is a good argument.  You can run multiple passes
> by streaming the same data twice if you needed to, and how much
> easier the implementation may become if you can assume that you can
> hold everything in-core, what you cannot fit in-core would not fit
> in-core, so ...
Yes this wording needs to be clarified.

If streaming to handle objects that don't fit in memory is the purpose, I agree there are slow multi-pass ways to deal with trees, commits and tags.

If writing directly to the pack is the purpose, using an in-core buffer for trees, commits, and tags is better.

I will put on the wording on the back burner and see what I come up with.

Show 12 quoted lines
>> So if it is needed to write commits, trees, and tags directly to pack
>> files writing a separate function to do the would be needed.
>
> But I am OK with this conclusion.  As the way to compute the
> fallback hashes for different types of objects are very different,
> compared to a single-hash world where as long as you come up with a
> serialization you have only a single way to hash and name the
> object.  We would end up having separate helper functions per target
> type anyway, even if we kept a single entry point function like
> index_stream().  The single entry point function will only be used
> to just dispatch to type specific ones, so renaming what we have today
> and making it clear they are for "blobs" does make sense.

Good. I am glad I am able to step back and successfully explain the whys of things.

Eric
Previous: Junio C HamanoNext: Eric W. Biederman
Message 3 of 12 in “bulk-checkin: only support blobs in index_bulk_checkin”
  1. bulk-checkin: only support blobs in index_bulk_checkinEric W. Biederman, Sep 20, 2023
  2. Junio C HamanoSep 20, 2023
  3. Eric W. BiedermanSep 20, 2023
  4. bulk-checkin: only support blobs in index_bulk_checkinEric W. Biederman, Sep 26, 2023
  5. Junio C HamanoSep 26, 2023
  6. Taylor BlauSep 27, 2023
  7. Junio C HamanoSep 27, 2023
  8. Taylor BlauSep 27, 2023
  9. Junio C HamanoSep 27, 2023
  10. Eric W. BiedermanSep 27, 2023
  11. Eric W. BiedermanSep 27, 2023
  12. Oswald BuddenhagenSep 28, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.