git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] list-objects-filter: merge filter data structs

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
May 29, 2019, 20:57 UTC
Message-ID
<e9147614-80f9-4c18-b431-539e2376295d@jeffhostetler.com>
In-Reply-To
<xmqq4l5dyrcu.fsf@gitster-ct.c.googlers.com>
On 5/29/2019 3:48 PM, Junio C Hamano wrote:
Show 62 quoted lines
> Matthew DeVore <matvore@comcast.net> writes:
> 
>> Simplify the filter execution data logic and structs by putting all
>> execution data for all filter types in a single struct. This results in
>> a tiny overhead for each filter instance, and in exchange, invoking
>> filters is not only easier but the list-objects-filter public API is
>> simpler and more opaque.
> 
> Hmmm...
> 
>> +struct filter_data {
>> +	/* Used by all filter types. */
>>   	struct oidset *omits;
>> +
>> +	enum list_objects_filter_result (*filter_object_fn)(
>> +		struct repository *r,
>> +		enum list_objects_filter_situation filter_situation,
>> +		struct object *obj,
>> +		const char *pathname,
>> +		const char *filename,
>> +		struct filter_data *filter_data);
>> +
>> +	/* BEGIN tree:<depth> filter data */
>> +
>> +	/*
>> +	 * Maps trees to the minimum depth at which they were seen. It is not
>> +	 * necessary to re-traverse a tree at deeper or equal depths than it has
>> +	 * already been traversed.
>> +	 *
>> +	 * We can't use LOFR_MARK_SEEN for tree objects since this will prevent
>> +	 * it from being traversed at shallower depths.
>> +	 */
>> +	struct oidmap seen_at_depth;
>> +
>> +	unsigned long exclude_depth;
>> +	unsigned long current_depth;
>> +
>> +	/* BEGIN blobs:limit=<limit> filter data */
>> +
>> +	unsigned long max_bytes;
>> +
>> +	/* BEGIN sparse:... filter data */
>> +
>> +	struct exclude_list el;
>> +
>> +	size_t nr, alloc;
>> +	struct frame *array_frame;
>>   };
> 
> I am hoping that I am not misreading the intention but you do not
> plan to use the above so that you can say "apply 'tree:depth=4' and
> 'blobs:limit=1G' at the same time" by filling the fields in a single
> struct, do you?  For combined filter, you'll still have multiple
> instances of filter_data struct, strung together in a list that says
> "all of these must be satisfied" or something like that, right?
> 
> And if that is the case, I am not sure why the above "struct with
> all these fields" is a good idea.  If these three (and probably we
> will have more as the system evolves) sets of fields in this outer
> struct for different filters were enclosed in a union, that would be
> a different story, though.
> 

I'm not sure I like the combined structure as proposed. But let's think about it.

I think part of problem with my original version was putting the filter_fn and filter_free_fn in the traversal_context rather than inside the filter_*_data structure.

I did a simple combined structure for the list_objects_filter_options and kind of regretted it because it wasn't obvious which data fields were defined or undefined in each filter constructor. But it was convenient when parsing the command line.

I think having a combined structure with a union enclosing a structure for the data fields in each filter type would be worth considering. That way you have a somewhat self-documenting sub-structure for each filter type that indicates which fields are defined.

I'd also suggest keeping the "oidset omits" inside each of the sub-structures, but that's just me.

BTW, I don't see a free_fn. That may collapse out with your proposal but I wanted to ask.

Thanks Jeff

Previous: Junio C HamanoNext: Matthew DeVore
Message 7 of 41 in “Filter combination”
  1. 0/5 Filter combinationMatthew DeVore, May 22, 2019
  2. 1/5 list-objects-filter: refactor into a context structMatthew DeVore, May 22, 2019
  3. Emily ShafferMay 24, 2019
  4. Matthew DeVoreMay 28, 2019
  5. list-objects-filter: merge filter data structsMatthew DeVore, May 28, 2019
  6. Junio C HamanoMay 29, 2019
  7. Jeff HostetlerMay 29, 2019
  8. Matthew DeVoreMay 29, 2019
  9. list-objects-filter: merge filter data structsMatthew DeVore, May 30, 2019
  10. Junio C HamanoMay 30, 2019
  11. Matthew DeVoreMay 30, 2019
  12. Matthew DeVoreMay 30, 2019
  13. 2/5 list-objects-filter-options: error is localizeableMatthew DeVore, May 22, 2019
  14. Emily ShafferMay 24, 2019
  15. Matthew DeVoreMay 28, 2019
  16. 3/5 list-objects-filter: implement composite filtersMatthew DeVore, May 22, 2019
  17. Jeff HostetlerMay 24, 2019
  18. Junio C HamanoMay 28, 2019
  19. Matthew DeVoreMay 29, 2019
  20. Jeff HostetlerMay 29, 2019
  21. Matthew DeVoreMay 29, 2019
  22. Jeff HostetlerMay 30, 2019
  23. Matthew DeVoreMay 31, 2019
  24. Jeff HostetlerJun 3, 2019
  25. Matthew DeVoreJun 1, 2019
  26. Emily ShafferMay 28, 2019
  27. Matthew DeVoreMay 31, 2019
  28. Jeff KingMay 31, 2019
  29. Matthew DeVoreJun 1, 2019
  30. Jeff KingJun 3, 2019
  31. Matthew DeVoreJun 3, 2019
  32. Jeff KingJun 4, 2019
  33. Matthew DeVoreJun 4, 2019
  34. Jeff KingJun 4, 2019
  35. Matthew DeVoreJun 4, 2019
  36. Jeff KingJun 4, 2019
  37. Matthew DeVoreJun 4, 2019
  38. Jeff KingJun 9, 2019
  39. 4/5 list-objects-filter-options: move error check upMatthew DeVore, May 22, 2019
  40. 5/5 list-objects-filter-options: allow mult. --filterMatthew DeVore, May 22, 2019
  41. Matthew DeVoreJun 6, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.