git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v6 09/40] Add initial external odb support

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Oct 4, 2017, 00:15 UTC
Message-ID
<CAGf8dgLD3JR4_oqEM09D6j+izrnXoj8YPr7JfBwKrx2qj=Vkkw@mail.gmail.com>
In-Reply-To
<CAP8UFD05QzCn53SjbbgaToVWyd=3N+HayDvR4OZNaP7YKbEBfg@mail.gmail.com>

On Tue, Oct 3, 2017 at 2:45 AM, Christian Couder <christian.couder@gmail.com> wrote:

Show 20 quoted lines
> Yeah, some people need the faster solution, but my opinion is that
> many other people would prefer the single shot protocol.
> If all you want to do is a simple resumable clone using bundles for
> example, then the long running process solution is very much overkill.
>
> For example with filters there are people using them to do keyword
> expansion (maybe to emulate the way Subversion and CVS substitutes
> keywords like $Id$, $Author$ and so on). It would be really bad to
> deprecate the single shot filters and tell those people they now have
> to use long running processes because we don't want to maintain the
> small code that make single shot filters work.
>
> The Microsoft GVFS use case is just one use case that is very far from
> what most people need. And my opinion is that many more people could
> benefit from the single shot protocol. For example many people and
> admins could benefit from resumable clones using bundles and, if I
> remove the single shot protocol, this use case will be unnecessarily
> more difficult to implement in the same way as keyword expansion would
> be unnecessarily more difficult to implement if we removed the single
> shot filters.

The idea that some users will prefer writing to the single-shot protocol is reasonable to me, but I think that providing a contrib/ Perl script that wraps something that speaks the single-shot protocol is sufficient. This results in less C code, and a better separation of concerns (I prefer 1 exit point and 1 adapter over 2 exit points).

Show 24 quoted lines
> I agree that your patch set already includes some infrastructure that
> could be used by my work, and your patch sets are perhaps implementing
> some of this infrastructure better than in my work (I haven't taken a
> deep look). But I really think that the right approach is to focus
> first on designing a flexible protocol between Git and external
> stores. Then the infrastructure work should be related to improving or
> enabling the flexible protocol and the communication between Git and
> external stores.
>
> Doing infrastructure work first and improving things on top of this
> new infrastructure without relying first on a design of the protocol
> between Git and external stores is not the best approach as I think we
> might over engineer some infrastructure work or base some user
> interfaces on the infrastructure work and not on the end goal.
>
> For example if we improve the current protocol, which is not
> necessarily a bad thing in itself, we might forget that for resumable
> clone it is much better if we just let external stores and helpers
> handle the transfer.
>
> I am not saying that doing infrastructure work is bad or will not in
> the end let us reach our goals, but I see it as something that is
> potentially distracting, or misleading, from focusing first on the
> protocol between Git and external stores.

I think that the infrastructure really needs to be considered when designing the protocol. In particular, we had to consider the needs of the connectivity check in fsck and the repacking in GC when designing what the promisor remote (or ODB, in this case) needs to tell us and what, if any, postprocessing needs to be done. In the end, I settled on tracking which objects came from the promisor remote and which did not, which works in my design (which I have tried to ensure that it fits in our and Microsoft's use case). But that design won't work in what I understand to be the ODB case, at least from what I understand, because (at least) (i) you can have multiple ODBs, and (ii) Git does not have direct access to the objects stored within the ODBs. So some more design needs to be done.

Previous: Christian CouderNext: Christian Couder
Message 44 of 49 in “Add initial experimental external ODB support”
  1. 00/40 Add initial experimental external ODB supportChristian Couder, Sep 16, 2017
  2. 01/40 builtin/clone: get rid of 'value' strbufChristian Couder, Sep 16, 2017
  3. 04/40 t0021/rot13-filter: improve error messageChristian Couder, Sep 16, 2017
  4. 05/40 t0021/rot13-filter: add packet_initialize()Christian Couder, Sep 16, 2017
  5. 07/40 Add Git/Packet.pm from parts of t0021/rot13-filter.plChristian Couder, Sep 16, 2017
  6. 11/40 t0400: add 'put_raw_obj' instruction to odb-helper scriptChristian Couder, Sep 16, 2017
  7. 10/40 odb-helper: add odb_helper_init() to send 'init' instructionChristian Couder, Sep 16, 2017
  8. 16/40 Add t0410 to test external ODB transferChristian Couder, Sep 16, 2017
  9. 14/40 t0400: add test for external odb write supportChristian Couder, Sep 16, 2017
  10. 19/40 lib-httpd: add list.shChristian Couder, Sep 16, 2017
  11. 22/40 pack-objects: don't pack objects in external odbsChristian Couder, Sep 16, 2017
  12. 26/40 odb-helper: add init_object_process()Christian Couder, Sep 16, 2017
  13. 29/40 odb-helper: add put_object_process()Christian Couder, Sep 16, 2017
  14. 30/40 Add t0470 to test passing raw objectsChristian Couder, Sep 16, 2017
  15. 31/40 odb-helper: add have_object_process()Christian Couder, Sep 16, 2017
  16. 32/40 Add t0480 to test "have" capability and raw objectsChristian Couder, Sep 16, 2017
  17. 33/40 external-odb: use 'odb=magic' attribute to mark odb blobsChristian Couder, Sep 16, 2017
  18. 38/40 Add tests for 'clone --initial-refspec'Christian Couder, Sep 16, 2017
  19. 39/40 Add t0430 to test cloning using bundlesChristian Couder, Sep 16, 2017
  20. 40/40 Doc/external-odb: explain transfering objects and metadataChristian Couder, Sep 16, 2017
  21. 37/40 clone: disable external odb before initial cloneChristian Couder, Sep 16, 2017
  22. 35/40 clone: add 'initial' param to write_remote_refs()Christian Couder, Sep 16, 2017
  23. 36/40 clone: add --initial-refspec optionChristian Couder, Sep 16, 2017
  24. 34/40 Add Documentation/technical/external-odb.txtChristian Couder, Sep 16, 2017
  25. 25/40 odb-helper: add 'script_mode' to 'struct odb_helper'Christian Couder, Sep 16, 2017
  26. 28/40 Add t0460 to test passing git objectsChristian Couder, Sep 16, 2017
  27. 23/40 Add t0420 to test transfer to HTTP external odbChristian Couder, Sep 16, 2017
  28. 21/40 odb-helper: add odb_helper_get_raw_object()Christian Couder, Sep 16, 2017
  29. 20/40 lib-httpd: add apache-e-odb.confChristian Couder, Sep 16, 2017
  30. 18/40 lib-httpd: add upload.shChristian Couder, Sep 16, 2017
  31. 24/40 external-odb: add 'get_direct' supportChristian Couder, Sep 16, 2017
  32. 27/40 Add t0450 to test 'get_direct' mechanismChristian Couder, Sep 16, 2017
  33. 15/40 Add GIT_NO_EXTERNAL_ODB env variableChristian Couder, Sep 16, 2017
  34. 13/40 external-odb: accept only blobs for nowChristian Couder, Sep 16, 2017
  35. 17/40 lib-httpd: pass config file to start_httpd()Christian Couder, Sep 16, 2017
  36. 12/40 external odb: add 'put_raw_obj' supportChristian Couder, Sep 16, 2017
  37. 06/40 t0021/rot13-filter: add capability functionsChristian Couder, Sep 16, 2017
  38. 09/40 Add initial external odb supportChristian Couder, Sep 16, 2017
  39. Jonathan TanSep 19, 2017
  40. Christian CouderSep 27, 2017
  41. Jonathan TanSep 29, 2017
  42. Ben PeartOct 2, 2017
  43. Christian CouderOct 3, 2017
  44. Jonathan TanOct 4, 2017
  45. 03/40 t0021/rot13-filter: improve 'if .. elsif .. else' styleChristian Couder, Sep 16, 2017
  46. 08/40 sha1_file: prepare for external odbsChristian Couder, Sep 16, 2017
  47. 02/40 t0021/rot13-filter: refactor packet reading functionsChristian Couder, Sep 16, 2017
  48. Ben PeartOct 2, 2017
  49. Christian CouderOct 3, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.