git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 4/4] convert: add "status=delayed" to filter process protocol

From
Lars Schneider <larsxschneider@gmail.com>
Date
Apr 18, 2017, 16:14 UTC
Message-ID
<1D510C6F-A830-48BE-880B-62F4212F4A7F@gmail.com>
In-Reply-To
<20170412174610.GB49694@Ida>
Show 10 quoted lines
> On 12. Apr 2017, at 19:46, Taylor Blau <ttaylorr@github.com> wrote:
> 
> I think this is a great approach and one that I'd be happy to implement in LFS.
> The additional capability isn't too complex, so I think other similar filters to
> LFS shouldn't have a hard time implementing it either.
> 
> I left a few comments, mostly expressing approval to the documentation changes.
> I'll leave the C review to someone more expert than me.
> 
> +1 from me on the protocol changes.
Thanks!
Show 8 quoted lines
>> +Delay
>> +^^^^^
>> +
>> +If the filter supports the "delay" capability, then Git can send the
>> +flag "delay-able" after the filter command and pathname.
> 
> Nit: I think either way is fine, but `can_delay` will save us 1 byte per each
> new checkout entry.
1 byte is no convincing argument to me but since you are a native speaker I trust your "can-delay" suggestion. I prefer dashes over underscores, though, for consistency with the rest of the protocol.
Show 9 quoted lines
>> +"delay-id", a number that identifies the blob, and a flush packet. The
>> +filter acknowledges this number with a "success" status and a flush
>> +packet.
> 
> I mentioned this in another thread, but I'd prefer, if possible, that we use the
> pathname as a unique identifier for referring back to a particular checkout
> entry. I think adding an additional identifier adds unnecessary complication to
> the protocol and introduces a forced mapping on the filter side from id to
> path.
I agree! I answered in the other thread. Let's keep the discussion there.
Show 10 quoted lines
> Both Git and the filter are going to have to keep these paths in memory
> somewhere, be that in-process, or on disk. That being said, I can see potential
> troubles with a large number of long paths that exceed the memory available to
> Git or the filter when stored in a hashmap/set.
> 
> On Git's side, I think trading that for some CPU time might make sense. If Git
> were to SHA1 each path and store that in a hashmap, it would consume more CPU
> time, but less memory to store each path. Git and the filter could then exchange
> path names, and Git would simply SHA1 the pathname each time it needed to refer
> back to memory associated with that entry in a hashmap.
I would be surprised if this would be necessary. If we filter delay 50,000 files (= a lot!) with a path length of 1000 characters (= very long!) then we would use 50MB plus some hashmap data structures. Modern machines should have enough RAM I would think...

Thanks, Lars

Previous: Taylor BlauNext: Taylor Blau
Message 17 of 18 in “convert: add "status=delayed" to filter process protocol”
  1. 0/4 convert: add "status=delayed" to filter process protocolLars Schneider, Apr 9, 2017
  2. 1/4 t0021: keep filter log files on comparisonLars Schneider, Apr 9, 2017
  3. 3/4 t0021: write "OUT" only on successLars Schneider, Apr 9, 2017
  4. 2/4 t0021: make debug log file name configurableLars Schneider, Apr 9, 2017
  5. 4/4 convert: add "status=delayed" to filter process protocolLars Schneider, Apr 9, 2017
  6. Lars SchneiderApr 10, 2017
  7. Eric WongApr 10, 2017
  8. Lars SchneiderApr 10, 2017
  9. Torsten BögershausenApr 10, 2017
  10. Lars SchneiderApr 11, 2017
  11. Torsten BögershausenApr 12, 2017
  12. Lars SchneiderApr 18, 2017
  13. Torsten BögershausenApr 19, 2017
  14. Lars SchneiderMay 21, 2017
  15. Taylor BlauApr 12, 2017
  16. Taylor BlauApr 12, 2017
  17. Lars SchneiderApr 18, 2017
  18. Taylor BlauApr 18, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.