Re: [PATCH v6 06/13] pkt-line: add functions to read/write flush terminated packet streams

From: Stefan Beller <sbeller@google.com>
Date: Aug 26, 2016, 17:02 UTC
Message-ID: <CAGZ79kYD7KSzdhe-60fs5iUKmYcMd6+V3fSN513+sr15q+g39Q@mail.gmail.com>
In-Reply-To: <CA+P7+xrYM_r+Q_27hWM2FwbreV90AqPgoiH-qPZN0hoPiHpKhQ@mail.gmail.com>

On Thu, Aug 25, 2016 at 5:55 PM, Jacob Keller <jacob.keller@gmail.com> wrote:

Show 18 quoted lines

> On Thu, Aug 25, 2016 at 3:31 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> What is wrong about that?  4*80k = 320kB overhead for length fields
>> to transfer 5GB worth of data?  I do not think it is worth worrying
>> about it.
>>
>> But I am more surprised by seeing that "why not a single huge
>> packet" suggestion immediately after you talked about "without the
>> possibility to intervene".  They do not seem to be remotely related;
>> in fact, they are going into opposite directions.
>>
>> Puzzled.
>
> Stefan's argument to me is thus "If we're already going to ignore
> sideband packets here, why not go all the way and make variable length
> packets and send a single packet of a maximum length? Doing thus will
> solve some set of future problems nicely and makes this code easier."
>
> I'm not sure I agree myself, but that's the logic as I understand it.

Yeah. To me it seems this design explicitly makes it hard for side bands. As we do not need sidebands for local transfers, this is fine for sure.

(If we wanted to make it sideband friendly, I'd expect you could register callbacks for either all packets or for a given sideband until the next flush comes.)

So as hinted by this design, we want a protocol that
* doesn't care about sidebands
* cares about large data (hence maybe throughput)
* has easy/clean interface

And one large packet would suffice for these three points as well and additionally has benefits for the network stuff.

The 320kB additional transmission are negligible overhead, so I was not concerned about the size, but rather the code being bloated, i.e. we need one layer of additional code to cope with the repetitive packets.

--- My background is mostly submodule related, and whenever I come up with a shiny novel idea that would help submodules tremendously, someone (mostly Peff) comes along and suggests a broader more generic thing, that works just as well for submodules but is applicable to all of Git.

So I picked up that way of thinking: If we write code here, that helps with a very special niche of things, can we make so, that the rest also benefits?

I may not understand all the requirements in this case, but to me it looks like the "one packet" approach covers all requirements, but has the huge potential to make other parts better in the long run.

-- 
Thanks,
Stefan