threads / patch / 35841

patchIntroduce experimental remote object access mode

Subject: [PATCH] Introduce experimental remote object access mode

## tl;dr

3 messages between Feb 11, 2014 and Feb 12, 2014. Diffs are folded; open one to read it.

replies: 2people: 2as markdown or json

Shawn Pearce· Feb 11, 2014, 08:54 UTC · lore

Make it easy to experiment what remote access to objects would be like if the network ran at say 1 ms round trip latency to obtain any object not on the local repository.

  $ time git ls-tree -r HEAD
  real 0m0.059s
  $ time GIT_RTT=1 git ls-tree -r HEAD
  real 0m27.283s

Yes kids, slowing down loose object access by just 1ms if all objects are remote can take a simple ls-tree from 59ms to more than enough time to drink tea or coffee.

Why would you do this? Perhaps you need more time in your day to consume tea or coffee. Set GIT_RTT and enjoy a beverage.

So-not-signed-off-by: this author or anyone else
---
  :-)
 sha1_file.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
Show changes to sha1_file.c +14 −0
diff --git a/sha1_file.c b/sha1_file.c
index 6e8c05d..9bdcbc3 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -38,6 +38,7 @@ const unsigned char null_sha1[20];

 static const char *no_log_pack_access = "no_log_pack_access";
 static const char *log_pack_access;
+static useconds_t rtt;

 /*
  * This is meant to hold a *small* number of objects that you would
@@ -436,9 +437,20 @@ void prepare_alt_odb(void)
  read_info_alternates(get_object_directory(), 0);
 }

+static void apply_rtt()
+{
+ if (!rtt) {
+ char *rtt_str = getenv("GIT_RTT");
+ rtt = rtt_str ? strtoul(rtt_str, NULL, 10) * 1000 : 1;
+ }
+ if (rtt > 1)
+ usleep(rtt);
+}
+
 static int has_loose_object_local(const unsigned char *sha1)
 {
  char *name = sha1_file_name(sha1);
+ apply_rtt();
  return !access(name, F_OK);
 }

@@ -1303,6 +1315,7 @@ void prepare_packed_git(void)

  if (prepare_packed_git_run_once)
  return;
+
  prepare_packed_git_one(get_object_directory(), 1);
  prepare_alt_odb();
  for (alt = alt_odb_list; alt; alt = alt->next) {
@@ -1439,6 +1452,7 @@ static int open_sha1_file(const unsigned char *sha1)
  struct alternate_object_database *alt;

  fd = git_open_noatime(name);
+ apply_rtt();
  if (fd >= 0)
  return fd;

-- 
1.9.0.rc1.175.g0b1dcb5
Junio C Hamano· Feb 11, 2014, 19:29 UTC · re: Shawn Pearce · lore

Re: [PATCH] Introduce experimental remote object access mode

Shawn Pearce <spearce@spearce.org> writes:
> Why would you do this? Perhaps you need more time in your day
> to consume tea or coffee. Set GIT_RTT and enjoy a beverage.

So the conclusion is that it is not practical to do a lazy fetch if it is done extremely naively at "we want this object --- wait a bit and we'll give you" level?

I am wondering if we can do a bit better, like "we want this object --- wait a bit, ah that's a commit, so it is likely that you may want the trees and blobs associated with it, too, if not right now but in a near future, let me push a pack that holds them to you"?

Show 57 quoted lines
>
> So-not-signed-off-by: this author or anyone else
> ---
>
>   :-)
>
>  sha1_file.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/sha1_file.c b/sha1_file.c
> index 6e8c05d..9bdcbc3 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -38,6 +38,7 @@ const unsigned char null_sha1[20];
>
>  static const char *no_log_pack_access = "no_log_pack_access";
>  static const char *log_pack_access;
> +static useconds_t rtt;
>
>  /*
>   * This is meant to hold a *small* number of objects that you would
> @@ -436,9 +437,20 @@ void prepare_alt_odb(void)
>   read_info_alternates(get_object_directory(), 0);
>  }
>
> +static void apply_rtt()
> +{
> + if (!rtt) {
> + char *rtt_str = getenv("GIT_RTT");
> + rtt = rtt_str ? strtoul(rtt_str, NULL, 10) * 1000 : 1;
> + }
> + if (rtt > 1)
> + usleep(rtt);
> +}
> +
>  static int has_loose_object_local(const unsigned char *sha1)
>  {
>   char *name = sha1_file_name(sha1);
> + apply_rtt();
>   return !access(name, F_OK);
>  }
>
> @@ -1303,6 +1315,7 @@ void prepare_packed_git(void)
>
>   if (prepare_packed_git_run_once)
>   return;
> +
>   prepare_packed_git_one(get_object_directory(), 1);
>   prepare_alt_odb();
>   for (alt = alt_odb_list; alt; alt = alt->next) {
> @@ -1439,6 +1452,7 @@ static int open_sha1_file(const unsigned char *sha1)
>   struct alternate_object_database *alt;
>
>   fd = git_open_noatime(name);
> + apply_rtt();
>   if (fd >= 0)
>   return fd;
Shawn Pearce· Feb 12, 2014, 20:55 UTC · re: Junio C Hamano · lore

Re: [PATCH] Introduce experimental remote object access mode

On Tue, Feb 11, 2014 at 11:29 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 8 quoted lines
> Shawn Pearce <spearce@spearce.org> writes:
>
>> Why would you do this? Perhaps you need more time in your day
>> to consume tea or coffee. Set GIT_RTT and enjoy a beverage.
>
> So the conclusion is that it is not practical to do a lazy fetch if
> it is done extremely naively at "we want this object --- wait a bit
> and we'll give you" level?

Yes, this is what I thought when someone proposed this hack in sha1_file.c to me on Monday. So I ran a quick experiment to see if my instinct was right.

> I am wondering if we can do a bit better, like "we want this object
> --- wait a bit, ah that's a commit, so it is likely that you may
> want the trees and blobs associated with it, too, if not right now
> but in a near future, let me push a pack that holds them to you"?
Ah, smart observation. That might work. However I doubt it.

I implemented a version of Git on top of Google Bigtable (and Apache HBase and Apache Cassandra) multiple times using JGit. tl;dr: this approach doesn't work in practice.

The naive implementation for these distributed NoSQL systems is to store each object in its own row keyed by SHA-1, and lookup the object when you want it. This is very slow and is more or less what this stupid patch shows. Worse, none of them were able to even get close to the 1ms latency I used in this example.

In another implementation (which I published into JGit as the "DHT" backend) I stored a group of related commits together in a row. Row target sizes were in the 1-2 MiB range when using pack style compression for commits, so the average row held hundreds of commits. Reading one commit would actually slurp back a number of related commits. The idea was if we need commit A now we will need B, C, D, E (its parents and ancestors) soon as the application walks the revision history, like rev-list or pack-objects.

For a process like pack-objects this almost seems to work. If commits are together we can slurp a group at a time to amortize the round trip latency. Unfortunately the application can still go through hundreds of commits faster than the real world RTT is. So I tried to fix this by storing an extra metadata pointer in each row to identify the next row, so next block of commits could start loading right away. Its still slow, as the application can scan through data faster than the RTT.

At least for pack generation the traversal code does commits and builds up a list of all root trees. The root trees can be async loaded in batches, but the depth first traversal is still a killer. There are stalls while the application waits for the next subtree, even if you cluster the trees also into groups using depth first traversal the way the packer produces pack files today.

Junio's idea to cluster data by commit and its related trees and blobs is just a different data organization. It may be necessary to make two copies of the data, one clustered by commits and another by commit+tree+blob to satisfy different access patterns. And in the commit+tree+blob case you may need multiple redundant copies of blobs near commits that use them if those commits are frequently accessed. Its a lot of redundant disk space.

We always say disk is cheap, but disk is slow and not getting faster. SSDs are helping, but SSDs are expensive and have size limitations compared to spinning disk. Just making many copies of data isn't necessarily a solution.

← back to recent threads