git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC/PATCH v4 00/49] Add initial experimental external ODB support

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Jul 12, 2017, 19:06 UTC
Message-ID
<20170712120647.6340f75a@twelve2.svl.corp.google.com>
In-Reply-To
<20170620075523.26961-1-chriscool@tuxfamily.org>

On Tue, 20 Jun 2017 09:54:34 +0200 Christian Couder <christian.couder@gmail.com> wrote:

Show 6 quoted lines
> Git can store its objects only in the form of loose objects in
> separate files or packed objects in a pack file.
> 
> To be able to better handle some kind of objects, for example big
> blobs, it would be nice if Git could store its objects in other object
> databases (ODB).

Thanks for this, and sorry for the late reply. It's good to know that others are thinking about "missing" objects in repos too.

>   - "have": the helper should respond with the sha1, size and type of
>     all the objects the external ODB contains, one object per line.

This should work well if we are not caching this "have" information locally (that is, if the object store can be accessed with low latency), but I am not sure if this will work otherwise. I see that you have proposed a local cache-using method later in the e-mail - my comments on that are below.

>   - "get <sha1>": the helper should then read from the external ODB
>     the content of the object corresponding to <sha1> and pass it to
> Git.

This makes sense - I have some patches [1] that implement this with the "fault_in" mechanism described in your e-mail.

[1] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/
Show 35 quoted lines
> * Transfering information
> 
> To tranfer information about the blobs stored in external ODB, some
> special refs, called "odb ref", similar as replace refs, are used in
> the tests of this series, but in general nothing forces the helper to
> use that mechanism.
> 
> The external odb helper is responsible for using and creating the refs
> in refs/odbs/<odbname>/, if it wants to do that. It is free for
> example to just create one ref, as it is also free to create many
> refs. Git would just transmit the refs that have been created by this
> helper, if Git is asked to do so.
> 
> For now in the tests there is one odb ref per blob, as it is simple
> and as it is similar to what git-lfs does. Each ref name is
> refs/odbs/<odbname>/<sha1> where <sha1> is the sha1 of the blob stored
> in the external odb named <odbname>.
> 
> These odb refs point to a blob that is stored in the Git
> repository and contain information about the blob stored in the
> external odb. This information can be specific to the external odb.
> The repos can then share this information using commands like:
> 
> `git fetch origin "refs/odbs/<odbname>/*:refs/odbs/<odbname>/*"`
> 
> At the end of the current patch series, "git clone" is teached a
> "--initial-refspec" option, that asks it to first fetch some specified
> refs. This is used in the tests to fetch the odb refs first.
> 
> This way only one "git clone" command can setup a repo using the
> external ODB mechanism as long as the right helper is installed on the
> machine and as long as the following options are used:
> 
>   - "--initial-refspec <odbrefspec>" to fetch the odb refspec
>   - "-c odb.<odbname>.command=<helper>" to configure the helper

A method like this means that information about every object is downloaded, regardless of which branches were actually cloned, and regardless of what parameters (e.g. max blob size) were used to control the objects that were actually cloned.

We could make, say, one "odb ref" per size and branch - for example,
"refs/odbs/master/0", "refs/odbs/master/1k", "refs/odbs/master/1m", etc.
- and have the client know which one to download. But this wouldn't
scale if we introduce different object filters in the clone and fetch
commands.

I think that it is best to have upload-pack send this information together with the packfile, since it knows exactly what objects were omitted, and therefore what information the client needs. As discussed in a sibling e-mail, clone/fetch already needs to be modified to omit objects anyway.

Previous: Christian CouderNext: Christian Couder
Message 63 of 64 in “Add initial experimental external ODB support”
  1. 00/49 Add initial experimental external ODB supportChristian Couder, Jun 20, 2017
  2. 01/49 builtin/clone: get rid of 'value' strbufChristian Couder, Jun 20, 2017
  3. 03/49 t0021/rot13-filter: improve 'if .. elsif .. else' styleChristian Couder, Jun 20, 2017
  4. 02/49 t0021/rot13-filter: refactor packet reading functionsChristian Couder, Jun 20, 2017
  5. 06/49 Git/Packet.pm: improve error messageChristian Couder, Jun 20, 2017
  6. 05/49 t0021/rot13-filter: use Git/Packet.pmChristian Couder, Jun 20, 2017
  7. 13/49 external-odb: accept only blobs for nowChristian Couder, Jun 20, 2017
  8. 15/49 Add GIT_NO_EXTERNAL_ODB env variableChristian Couder, Jun 20, 2017
  9. 12/49 external odb: add write supportChristian Couder, Jun 20, 2017
  10. 16/49 Add t0410 to test external ODB transferChristian Couder, Jun 20, 2017
  11. 18/49 lib-httpd: add upload.shChristian Couder, Jun 20, 2017
  12. 14/49 t0400: add test for external odb write supportChristian Couder, Jun 20, 2017
  13. 19/49 lib-httpd: add list.shChristian Couder, Jun 20, 2017
  14. 26/49 odb-helper: add script_modeChristian Couder, Jun 20, 2017
  15. 27/49 Documentation: add read-object-protocol.txtChristian Couder, Jun 20, 2017
  16. 32/49 t04*: add 'get_cap' support to helpersChristian Couder, Jun 20, 2017
  17. 30/49 odb-helper: add read_object_process()Christian Couder, Jun 20, 2017
  18. Ben PeartJul 10, 2017
  19. 34/49 odb-helper: fix odb_helper_fetch_object() for read_objectChristian Couder, Jun 20, 2017
  20. 33/49 odb-helper: call odb_helper_lookup() with 'have' capabilityChristian Couder, Jun 20, 2017
  21. 38/49 Add t0470 to test passing plain objectsChristian Couder, Jun 20, 2017
  22. 37/49 odb-helper: add read_packetized_plain_object_to_fd()Christian Couder, Jun 20, 2017
  23. 40/49 Add t0480 to test "have" capability and plain objectsChristian Couder, Jun 20, 2017
  24. 31/49 external-odb: add external_odb_get_capabilities()Christian Couder, Jun 20, 2017
  25. 29/49 Add t0410 to test read object mechanismChristian Couder, Jun 20, 2017
  26. 45/49 clone: add initial param to write_remote_refs()Christian Couder, Jun 20, 2017
  27. 46/49 clone: add --initial-refspec optionChristian Couder, Jun 20, 2017
  28. 48/49 Add test for 'clone --initial-refspec'Christian Couder, Jun 20, 2017
  29. 43/49 odb-helper: advertise 'put' capabilityChristian Couder, Jun 20, 2017
  30. 41/49 external-odb: add external_odb_do_fetch_object()Christian Couder, Jun 20, 2017
  31. 47/49 clone: disable external odb before initial cloneChristian Couder, Jun 20, 2017
  32. 42/49 odb-helper: advertise 'have' capabilityChristian Couder, Jun 20, 2017
  33. 49/49 t: add t0430 to test cloning using bundlesChristian Couder, Jun 20, 2017
  34. 39/49 odb-helper: add write_object_process()Christian Couder, Jun 20, 2017
  35. 44/49 odb-helper: add have_object_process()Christian Couder, Jun 20, 2017
  36. 35/49 Add t0460 to test passing git objectsChristian Couder, Jun 20, 2017
  37. 36/49 odb-helper: add read_packetized_git_object_to_fd()Christian Couder, Jun 20, 2017
  38. 23/49 t0420: add test with HTTP external odbChristian Couder, Jun 20, 2017
  39. 24/49 odb-helper: start fault in implementationChristian Couder, Jun 20, 2017
  40. 28/49 contrib: add long-running-read-object/example.plChristian Couder, Jun 20, 2017
  41. 25/49 external-odb: add external_odb_fault_in_object()Christian Couder, Jun 20, 2017
  42. 22/49 pack-objects: don't pack objects in external odbsChristian Couder, Jun 20, 2017
  43. 20/49 lib-httpd: add apache-e-odb.confChristian Couder, Jun 20, 2017
  44. 09/49 Add initial external odb supportChristian Couder, Jun 20, 2017
  45. Ben PeartJun 23, 2017
  46. 17/49 lib-httpd: pass config file to start_httpd()Christian Couder, Jun 20, 2017
  47. 21/49 odb-helper: add 'store_plain_objects' to 'struct odb_helper'Christian Couder, Jun 20, 2017
  48. 07/49 Git/Packet.pm: add packet_initialize()Christian Couder, Jun 20, 2017
  49. Ben PeartJun 23, 2017
  50. 10/49 external odb foreachChristian Couder, Jun 20, 2017
  51. 08/49 Git/Packet: add capability functionsChristian Couder, Jun 20, 2017
  52. 11/49 t0400: add 'put' command to odb-helper scriptChristian Couder, Jun 20, 2017
  53. 04/49 Add Git/Packet.pm from parts of t0021/rot13-filter.plChristian Couder, Jun 20, 2017
  54. Christian CouderJun 20, 2017
  55. Ben PeartJun 23, 2017
  56. Christian CouderJul 1, 2017
  57. Christian CouderJul 1, 2017
  58. Junio C HamanoJul 1, 2017
  59. Christian CouderJul 2, 2017
  60. Junio C HamanoJul 3, 2017
  61. Ben PeartJul 6, 2017
  62. Christian CouderSep 15, 2017
  63. Jonathan TanJul 12, 2017
  64. Christian CouderSep 15, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.