git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/6] path-walk: introduce an object walk by path

From
karthik nayak <karthik.188@gmail.com>
Date
Nov 1, 2024, 13:42 UTC
Message-ID
<CAOLa=ZRJd7CXNFi1V=4UN_CuNFHqSsOGE5S0AnVeBMvPJJy72w@mail.gmail.com>
In-Reply-To
<draft-87r07v14kl.fsf@archlinux.mail-host-address-is-not-set>
karthik nayak <karthik.188@gmail.com> writes:
Show 51 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> From: Derrick Stolee <stolee@gmail.com>
>>
>> In anticipation of a few planned applications, introduce the most basic form
>> of a path-walk API. It currently assumes that there are no UNINTERESTING
>> objects, and does not include any complicated filters. It calls a function
>> pointer on groups of tree and blob objects as grouped by path. This only
>> includes objects the first time they are discovered, so an object that
>> appears at multiple paths will not be included in two batches.
>>
>> These batches are collected in 'struct type_and_oid_list' objects, which
>> store an object type and an oid_array of objects.
>>
>> The data structures are documented in 'struct path_walk_context', but in
>> summary the most important are:
>>
>>   * 'paths_to_lists' is a strmap that connects a path to a
>>     type_and_oid_list for that path. To avoid conflicts in path names,
>>     we make sure that tree paths end in "/" (except the root path with
>>     is an empty string) and blob paths do not end in "/".
>>
>>   * 'path_stack' is a string list that is added to in an append-only
>>     way. This stores the stack of our depth-first search on the heap
>>     instead of using recursion.
>>
>>   * 'path_stack_pushed' is a strmap that stores path names that were
>>     already added to 'path_stack', to avoid repeating paths in the
>>     stack. Mostly, this saves us from quadratic lookups from doing
>>     unsorted checks into the string_list.
>>
>> The coupling of 'path_stack' and 'path_stack_pushed' is protected by the
>> push_to_stack() method. Call this instead of inserting into these
>> structures directly.
>>
>> The walk_objects_by_path() method initializes these structures and
>> starts walking commits from the given rev_info struct. The commits are
>> used to find the list of root trees which populate the start of our
>> depth-first search.
>
> Isn't this more of breadth-first search? Reading through the code, the
> algorithm seems something like:
>
> - For each commit in list of commits (from rev_info)
>   - Tackle each root tree, add root path to the stack.
> - For each path in stack left
>   - Call the callback provided by client.
>   - Find all its first level children, add each to the stack.
>
> So wouldn't this go through the tree in level by level basis? Making it
> a BFS?

My bad here, thinking more about it, it is DFS indeed. Although we add all the children of a level to the stack, we pop each of them from the stack and end up traversing down that level.

Show 6 quoted lines
>
> Apart from this, the patch itself looks solid. I ended up writing a
> small client to play with this API, and was very pleased how quickly I
> could get it running.
>
> [snip]
Previous: Derrick Stolee via GitGitGadget
Message 67 of 67 in “PATH WALK I: The path-walk API”
  1. 0/6 PATH WALK I: The path-walk APIDerrick Stolee via GitGitGadget, Oct 31, 2024
  2. 1/6 path-walk: introduce an object walk by pathDerrick Stolee via GitGitGadget, Oct 31, 2024
  3. karthik nayakNov 1, 2024
  4. Derrick StoleeNov 1, 2024
  5. 2/6 test-lib-functions: add test_cmp_sortedDerrick Stolee via GitGitGadget, Oct 31, 2024
  6. 3/6 t6601: add helper for testing path-walk APIDerrick Stolee via GitGitGadget, Oct 31, 2024
  7. karthik nayakNov 1, 2024
  8. Jonathan TanNov 1, 2024
  9. Derrick StoleeNov 4, 2024
  10. Jonathan TanNov 4, 2024
  11. Derrick StoleeNov 8, 2024
  12. Patrick SteinhardtNov 6, 2024
  13. Derrick StoleeNov 8, 2024
  14. 4/6 path-walk: allow consumer to specify object typesDerrick Stolee via GitGitGadget, Oct 31, 2024
  15. 5/6 path-walk: visit tags and cached objectsDerrick Stolee via GitGitGadget, Oct 31, 2024
  16. karthik nayakNov 1, 2024
  17. Derrick StoleeNov 4, 2024
  18. 6/6 path-walk: mark trees and blobs as UNINTERESTINGDerrick Stolee via GitGitGadget, Oct 31, 2024
  19. Derrick StoleeOct 31, 2024
  20. Taylor BlauNov 1, 2024
  21. Derrick StoleeNov 4, 2024
  22. Jeff KingNov 4, 2024
  23. Junio C HamanoNov 5, 2024
  24. Derrick StoleeNov 8, 2024
  25. Junio C HamanoNov 11, 2024
  26. Derrick StoleeNov 11, 2024
  27. Jeff KingNov 11, 2024
  28. Junio C HamanoNov 11, 2024
  29. Jeff KingNov 11, 2024
  30. 0/6 PATH WALK I: The path-walk APIDerrick Stolee via GitGitGadget, Nov 9, 2024
  31. 1/6 path-walk: introduce an object walk by pathDerrick Stolee via GitGitGadget, Nov 9, 2024
  32. 2/6 test-lib-functions: add test_cmp_sortedDerrick Stolee via GitGitGadget, Nov 9, 2024
  33. 3/6 t6601: add helper for testing path-walk APIDerrick Stolee via GitGitGadget, Nov 9, 2024
  34. Taylor BlauNov 21, 2024
  35. 4/6 path-walk: allow consumer to specify object typesDerrick Stolee via GitGitGadget, Nov 9, 2024
  36. Taylor BlauNov 21, 2024
  37. 5/6 path-walk: visit tags and cached objectsDerrick Stolee via GitGitGadget, Nov 9, 2024
  38. 6/6 path-walk: mark trees and blobs as UNINTERESTINGDerrick Stolee via GitGitGadget, Nov 9, 2024
  39. Taylor BlauNov 21, 2024
  40. Patrick SteinhardtNov 25, 2024
  41. Junio C HamanoNov 26, 2024
  42. Patrick SteinhardtNov 26, 2024
  43. Junio C HamanoNov 26, 2024
  44. 0/7 PATH WALK I: The path-walk APIDerrick Stolee via GitGitGadget, Dec 6, 2024
  45. 1/7 path-walk: introduce an object walk by pathDerrick Stolee via GitGitGadget, Dec 6, 2024
  46. Patrick SteinhardtDec 13, 2024
  47. Derrick StoleeDec 18, 2024
  48. Patrick SteinhardtDec 27, 2024
  49. 2/7 test-lib-functions: add test_cmp_sortedDerrick Stolee via GitGitGadget, Dec 6, 2024
  50. 3/7 t6601: add helper for testing path-walk APIDerrick Stolee via GitGitGadget, Dec 6, 2024
  51. 4/7 path-walk: allow consumer to specify object typesDerrick Stolee via GitGitGadget, Dec 6, 2024
  52. 5/7 path-walk: visit tags and cached objectsDerrick Stolee via GitGitGadget, Dec 6, 2024
  53. Patrick SteinhardtDec 13, 2024
  54. Derrick StoleeDec 18, 2024
  55. 6/7 path-walk: mark trees and blobs as UNINTERESTINGDerrick Stolee via GitGitGadget, Dec 6, 2024
  56. 7/7 path-walk: reorder object visitsDerrick Stolee via GitGitGadget, Dec 6, 2024
  57. Patrick SteinhardtDec 13, 2024
  58. 0/7 PATH WALK I: The path-walk APIDerrick Stolee via GitGitGadget, Dec 20, 2024
  59. 1/7 path-walk: introduce an object walk by pathDerrick Stolee via GitGitGadget, Dec 20, 2024
  60. Patrick SteinhardtDec 27, 2024
  61. 2/7 test-lib-functions: add test_cmp_sortedDerrick Stolee via GitGitGadget, Dec 20, 2024
  62. 3/7 t6601: add helper for testing path-walk APIDerrick Stolee via GitGitGadget, Dec 20, 2024
  63. 4/7 path-walk: allow consumer to specify object typesDerrick Stolee via GitGitGadget, Dec 20, 2024
  64. 5/7 path-walk: visit tags and cached objectsDerrick Stolee via GitGitGadget, Dec 20, 2024
  65. 6/7 path-walk: mark trees and blobs as UNINTERESTINGDerrick Stolee via GitGitGadget, Dec 20, 2024
  66. 7/7 path-walk: reorder object visitsDerrick Stolee via GitGitGadget, Dec 20, 2024
  67. karthik nayakNov 1, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.