{"thread":{"id":"42720","subject":"[RFC] Native access to Git LFS cache","startedAt":"2016-06-27T05:38:27Z","lastAt":"2016-06-28T16:01:30Z","messageCount":11,"participants":["larsxschneider@gmail.com","Duy Nguyen","Junio C Hamano","Johannes Schindelin","Lars Schneider","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"290217","messageId":"1467005913-6503-1-git-send-email-larsxschneider@gmail.com","threadId":"42720","inReplyTo":null,"subject":"[RFC] Native access to Git LFS cache","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-06-27T05:38:33Z","receivedAt":"2016-06-27T05:38:27Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider <larsxschneider@gmail.com>\n\nHi,\n\nI found a way to make Git LFS faster up to a factor of 100x in\nrepositories with a large number of Git LFS files. I am looking\nfor comments if my approach would be acceptable by the Git community.\n\n## What is Git LFS?\nGit LFS [1] is an extension to Git that handles large files for Git\nrepositories. The project gained quite some momentum as almost all major\nGit hosting services support it (GitHub [1], Atlassian Bitbucket [2],\nGitLab [4]).\n\n## What is the problem with Git LFS?\nGit LFS is an application that is executed via Git clean/smudge filter.\nThe process invocation of these filters requires noticeable time (especially\non Windows) even if the Git LFS executable only accesses its local cache.\n\nBased on my previous findings [5] Steve Streeting (@sinbad) improved the\nclone times of Git LFS repositories with a lot of files by a factor of 10\nor more [6][7].\n\nUnfortunately that fix helps only with cloning. Any local Git operation\nthat invokes the clean/smudge filter (e.g. switching branches) is still\nslow. Even on the Git mailing list a user reported that issue [8].\n\n\n## Proposed solution\nGit LFS caches its objects under .git/lfs/objects. Most of the time Git\nLFS objects are already available in the cache (e.g. if you switch branches\nback and forth). I implemented these \"cache hits\" natively in Git.\nPlease note that this implementation is just a quick and dirty proof of\nconcept. If the Git community agrees that this kind of approach would be\nacceptable then I will start to work on a proper patch series with cross\nplatform support and unit tests.\n\n\n## Performance tests\nI executed both test runs on a 2,5 GHz Intel Core i7 with SSD and OS X.\nA test run is the consecutive execution of four Git commands:\n 1. clone the repo\n 2. checkout to the \"removed-files\" branch\n 3. timed: checkout the \"master\" branch\n 4. timed: checkout \"removed-files\" branch\n\nTest command:\nset -x; git lfs clone https://github.com/larsxschneider/lfstest-manyfiles.git repo; cd repo; git checkout removed-files; time git checkout master; time git checkout removed-files\n\nI compiled Git with the following flags:\nNO_GETTEXT=YesPlease NEEDS_SSL_WITH_CRYPTO=YesPlease make -j 8 CFLAGS=\"-I/usr/local/opt/openssl/include\" LDFLAGS=\"-L/usr/local/opt/openssl/lib\"\n\n\n### TEST RUN A -- Default Git 2.9 (ab7797d) and Git LFS 1.2.1\n+ git lfs clone https://github.com/larsxschneider/lfstest-manyfiles.git repo\nCloning into 'repo'...\nwarning: templates not found /Users/lars/share/git-core/templates\nremote: Counting objects: 15012, done.\nremote: Total 15012 (delta 0), reused 0 (delta 0), pack-reused 15012\nReceiving objects: 100% (15012/15012), 2.02 MiB | 1.77 MiB/s, done.\nChecking connectivity... done.\nChecking out files: 100% (15001/15001), done.\nGit LFS: (15000 of 15000 files) 0 B / 77.04 KB\n+ cd repo\n+ git checkout removed-files\nBranch removed-files set up to track remote branch removed-files from origin.\nSwitched to a new branch 'removed-files'\n+ git checkout master\nChecking out files: 100% (12000/12000), done.\nSwitched to branch 'master'\nYour branch is up-to-date with 'origin/master'.\n\nreal    6m2.979s\nuser    2m39.066s\nsys 2m41.610s\n+ git checkout removed-files\nSwitched to branch 'removed-files'\nYour branch is up-to-date with 'origin/removed-files'.\n\nreal    0m1.310s\nuser    0m0.385s\nsys 0m0.881s\n\n\n### TEST RUN B -- Default Git 2.9 with native LFS cache and Git LFS 1.2.1\nhttps://github.com/larsxschneider/git/tree/lfs-cache\n+ git lfs clone https://github.com/larsxschneider/lfstest-manyfiles.git repo\nCloning into 'repo'...\nwarning: templates not found /Users/lars/share/git-core/templates\nremote: Counting objects: 15012, done.\nremote: Total 15012 (delta 0), reused 0 (delta 0), pack-reused 15012\nReceiving objects: 100% (15012/15012), 2.02 MiB | 1.44 MiB/s, done.\nChecking connectivity... done.\nGit LFS: (15001 of 15000 files) 0 B / 77.04 KB\n+ cd repo\n+ git checkout removed-files\nBranch removed-files set up to track remote branch removed-files from origin.\nSwitched to a new branch 'removed-files'\n+ git checkout master\nChecking out files: 100% (12000/12000), done.\nSwitched to branch 'master'\nYour branch is up-to-date with 'origin/master'.\n\nreal    0m2.267s\nuser    0m0.295s\nsys 0m1.948s\n+ git checkout removed-files\nSwitched to branch 'removed-files'\nYour branch is up-to-date with 'origin/removed-files'.\n\nreal    0m0.715s\nuser    0m0.072s\nsys 0m0.672s\n\n\n### Results\nDefault Git:                      6m2.979s + 0m1.310s = 364s\nGit with native LFS cache access: 0m2.267s + 0m0.715s = 4s\n\nThe native cache solution is almost 100x faster when switching branches\non my local machine with a test repository containing 15,000 Git LFS files.\nBased on my previous experience with Git LFS clone I expect even more\ndramatic results on Windows.\n\nThanks,\nLars\n\n[1] https://git-lfs.github.com/\n[2] https://github.com/blog/1986-announcing-git-large-file-storage-lfs\n[3] http://blogs.atlassian.com/2016/02/git-lfs-for-designers-game-developers-architects/\n[4] https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/\n[5] https://github.com/github/git-lfs/issues/931#issuecomment-172939381\n[6] https://github.com/github/git-lfs/pull/988\n[7] https://developer.atlassian.com/blog/2016/04/git-lfs-12-clone-faster/\n[8] http://article.gmane.org/gmane.comp.version-control.git/297809\n\n\n---\n cache.h     |   2 +\n convert.c   | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++----------\n csum-file.c |  31 +++++++++++++\n csum-file.h |   2 +\n hex.c       |  16 +++++++\n 5 files changed, 172 insertions(+), 24 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 6049f86..57fdb18 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1196,6 +1196,8 @@ extern char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n extern char *sha1_to_hex(const unsigned char *sha1);\t/* static buffer result! */\n extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as sha1_to_hex */\n\n+extern char *sha256_to_hex_r(char *out, const unsigned char *sha1);\n+\n extern int interpret_branch_name(const char *str, int len, struct strbuf *);\n extern int get_sha1_mb(const char *str, unsigned char *sha1);\n\ndiff --git a/convert.c b/convert.c\nindex b1614bf..006e9c4 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -18,6 +18,10 @@\n #define CONVERT_STAT_BITS_TXT_CRLF  0x2\n #define CONVERT_STAT_BITS_BIN       0x4\n\n+const char *LFS_VERSION_MARKER = \"version https://git-lfs.github.com/spec/v1\\n\";\n+const char *LFS_OID_MARKER = \"oid sha256:\";\n+\n+\n enum crlf_action {\n \tCRLF_UNDEFINED,\n \tCRLF_BINARY,\n@@ -427,6 +431,79 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \treturn (write_err || status);\n }\n\n+static int cached_lfs_smudge(const char *src, const char *cmd,\n+\t\t\t\t\t\t\t struct strbuf *lfsbuf)\n+{\n+\tint ret = 0;\n+\tif (src &&\n+\t\tstrlen(src) > strlen(LFS_VERSION_MARKER) &&\n+\t\t!strncmp(LFS_VERSION_MARKER, src, strlen(LFS_VERSION_MARKER)) &&\n+\t\t!strcmp(\"git-lfs smudge %f\", cmd)\n+\t) {\n+\t\tconst char *lfs_oid_found = strstr(src, LFS_OID_MARKER);\n+\t\tif (lfs_oid_found) {\n+\t\t\tconst char *lfs_oid = lfs_oid_found + strlen(LFS_OID_MARKER);\n+\n+\t\t\t// Construct path to LFS object\n+\t\t\tstrbuf_reset(lfsbuf);\n+\t\t\tstrbuf_addstr(lfsbuf, git_pathdup(\"lfs/objects/\"));\n+\t\t\tstrbuf_add(lfsbuf, lfs_oid, 2);\n+\t\t\tstrbuf_addch(lfsbuf, '/');\n+\t\t\tstrbuf_add(lfsbuf, lfs_oid+2, 2);\n+\t\t\tstrbuf_addch(lfsbuf, '/');\n+\t\t\tstrbuf_add(lfsbuf, lfs_oid, 64);\n+\n+\t\t\tif (access(lfsbuf->buf, F_OK) != -1) {\n+\t\t\t\t// LFS object found in local LFS cache\n+\t\t\t\tret = 1;\n+\t\t\t}\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n+static int cached_lfs_clean(const char *path, const char *cmd,\n+\t\t\t\t\t\t\tstruct strbuf *lfsbuf)\n+{\n+\tint ret = 0;\n+\tif (path && !strcmp(\"git-lfs clean %f\", cmd)) {\n+\n+\t\t// TODO: Is there an easy way to access the content of the last\n+\t\t// known committed state of this file in the Git repo? If yes,\n+\t\t// then we could read the last known Git LFS OID, construct a\n+\t\t// path in the Git LFS cache and compare this file against \"path\".\n+\t\t// If both files are equal then we can use the last known committed\n+\t\t// state as \"clean\" and we could get rid of the SHA256 dependency\n+\t\t// here.\n+\t\tssize_t file_size;\n+\t\tunsigned char sha256[64];\n+\t\tsha256fd(path, &sha256, &file_size);\n+\n+\t\tchar lfs_oid[64];\n+\t\tsha256_to_hex_r(lfs_oid, sha256);\n+\n+\t\t// Construct path to LFS object\n+\t\tstrbuf_reset(lfsbuf);\n+\t\tstrbuf_addstr(lfsbuf, git_pathdup(\"lfs/objects/\"));\n+\t\tstrbuf_add(lfsbuf, lfs_oid, 2);\n+\t\tstrbuf_addch(lfsbuf, '/');\n+\t\tstrbuf_add(lfsbuf, lfs_oid+2, 2);\n+\t\tstrbuf_addch(lfsbuf, '/');\n+\t\tstrbuf_add(lfsbuf, lfs_oid, 64);\n+\n+\t\tif (access(lfsbuf->buf, F_OK) != -1) {\n+\t\t\t// LFS object found in local LFS cache\n+\t\t\tstrbuf_reset(lfsbuf);\n+\t\t\tstrbuf_addstr(lfsbuf, LFS_VERSION_MARKER);\n+\t\t\tstrbuf_addstr(lfsbuf, LFS_OID_MARKER);\n+\t\t\tstrbuf_add(lfsbuf, lfs_oid, 64);\n+\t\t\tstrbuf_addf(lfsbuf, \"\\nsize %d\\n\", file_size);\n+\t\t\tret = 1;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n static int apply_filter(const char *path, const char *src, size_t len, int fd,\n                         struct strbuf *dst, const char *cmd)\n {\n@@ -437,6 +514,7 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \t * (child --> cmd) --> us\n \t */\n \tint ret = 1;\n+\tstruct strbuf lfsbuf = STRBUF_INIT;\n \tstruct strbuf nbuf = STRBUF_INIT;\n \tstruct async async;\n \tstruct filter_params params;\n@@ -447,37 +525,56 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \tif (!dst)\n \t\treturn 1;\n\n-\tmemset(&async, 0, sizeof(async));\n-\tasync.proc = filter_buffer_or_fd;\n-\tasync.data = &params;\n-\tasync.out = -1;\n-\tparams.src = src;\n-\tparams.size = len;\n-\tparams.fd = fd;\n-\tparams.cmd = cmd;\n-\tparams.path = path;\n-\n-\tfflush(NULL);\n-\tif (start_async(&async))\n-\t\treturn 0;\t/* error was already reported */\n-\n-\tif (strbuf_read(&nbuf, async.out, len) < 0) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n-\t\tret = 0;\n-\t}\n-\tif (close(async.out)) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n-\t\tret = 0;\n+\t// TODO: check if git config \"lfs.native-cache\" is true\n+\tif (cached_lfs_smudge(src, cmd, &lfsbuf)) {\n+\t\tfd = open(lfsbuf.buf, O_RDONLY);\n+\t\tif (strbuf_read(&nbuf, fd, len) < 0) {\n+\t\t\terror(\"reading from cached LFS object failed\", lfsbuf.buf);\n+\t\t\tret = 0;\n+\t\t}\n+\t\tif (close(fd)) {\n+\t\t\terror(\"closing cached LFS object failed\", lfsbuf.buf);\n+\t\t\tret = 0;\n+\t\t}\n \t}\n-\tif (finish_async(&async)) {\n-\t\terror(\"external filter %s failed\", cmd);\n-\t\tret = 0;\n+\t// TODO: check if git config \"lfs.native-cache\" is true\n+\telse if (cached_lfs_clean(path, cmd, &lfsbuf)) {\n+\t\tstrbuf_reset(&nbuf);\n+\t\tstrbuf_addstr(&nbuf, lfsbuf.buf);\n+\t} else {\n+\t\tmemset(&async, 0, sizeof(async));\n+\t\tasync.proc = filter_buffer_or_fd;\n+\t\tasync.data = &params;\n+\t\tasync.out = -1;\n+\t\tparams.src = src;\n+\t\tparams.size = len;\n+\t\tparams.fd = fd;\n+\t\tparams.cmd = cmd;\n+\t\tparams.path = path;\n+\n+\t\tfflush(NULL);\n+\t\tif (start_async(&async))\n+\t\t\treturn 0;\t/* error was already reported */\n+\n+\t\tif (strbuf_read(&nbuf, async.out, len) < 0) {\n+\t\t\terror(\"read from external filter %s failed\", cmd);\n+\t\t\tret = 0;\n+\t\t}\n+\t\tif (close(async.out)) {\n+\t\t\terror(\"read from external filter %s failed\", cmd);\n+\t\t\tret = 0;\n+\t\t}\n+\t\tif (finish_async(&async)) {\n+\t\t\terror(\"external filter %s failed\", cmd);\n+\t\t\tret = 0;\n+\t\t}\n \t}\n\n \tif (ret) {\n \t\tstrbuf_swap(dst, &nbuf);\n \t}\n \tstrbuf_release(&nbuf);\n+\tstrbuf_release(&lfsbuf);\n \treturn ret;\n }\n\ndiff --git a/csum-file.c b/csum-file.c\nindex a172199..2ca4d7f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -11,6 +11,37 @@\n #include \"progress.h\"\n #include \"csum-file.h\"\n\n+void sha256fd(const char *name, unsigned char *sha256, ssize_t *file_size)\n+{\n+\tint fd;\n+\tstruct stat st;\n+\tfd = open(name, O_RDONLY);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to open '%s'\", name);\n+\tfstat(fd, &st);\n+\tsize_t size = xsize_t(st.st_size);\n+\t*file_size = size;\n+\n+\tSHA256_CTX ctx;\n+\tSHA256_Init(&ctx);\n+\tunsigned char fd_buffer[8192];\n+\n+\twhile (size > 0) {\n+\t\tssize_t rsize = size < sizeof(fd_buffer) ? size : sizeof(fd_buffer);\n+\t\tssize_t ret = read_in_full(fd, fd_buffer, rsize);\n+\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"%s: sha256 file read error\", name);\n+\t\tif (ret != rsize)\n+\t\t\tdie(\"failed to read %d bytes from '%s'\", (int)rsize, name);\n+\t\tSHA256_Update(&ctx, fd_buffer, rsize);\n+\t\tsize -= rsize;\n+\t}\n+\n+\tSHA256_Final(sha256, &ctx);\n+\tclose(fd);\n+}\n+\n static void flush(struct sha1file *f, const void *buf, unsigned int count)\n {\n \tif (0 <= f->check_fd && count)  {\ndiff --git a/csum-file.h b/csum-file.h\nindex 7530927..bad9262 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -39,6 +39,8 @@ extern void sha1flush(struct sha1file *f);\n extern void crc32_begin(struct sha1file *);\n extern uint32_t crc32_end(struct sha1file *);\n\n+extern void sha256fd(const char *name, unsigned char *sha256, ssize_t *file_size);\n+\n static inline void sha1write_u8(struct sha1file *f, uint8_t data)\n {\n \tsha1write(f, &data, sizeof(data));\ndiff --git a/hex.c b/hex.c\nindex 0519f85..73e2077 100644\n--- a/hex.c\n+++ b/hex.c\n@@ -88,3 +88,19 @@ char *oid_to_hex(const struct object_id *oid)\n {\n \treturn sha1_to_hex(oid->hash);\n }\n+\n+char *sha256_to_hex_r(char *buffer, const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tchar *buf = buffer;\n+\tint i;\n+\n+\tfor (i = 0; i < 32; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t*buf = '\\0';\n+\n+\treturn buffer;\n+}\n--\n2.5.1\n\n"},{"id":"290241","messageId":"CACsJy8B7ie_oQRZ7Pw3NTo4YyXwZOer-iDUVxDPSY_uhrQi1bw@mail.gmail.com","threadId":"42720","inReplyTo":"1467005913-6503-1-git-send-email-larsxschneider@gmail.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-27T15:53:30Z","receivedAt":"2016-06-27T15:54:18Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n> ## Proposed solution\n> Git LFS caches its objects under .git/lfs/objects. Most of the time Git\n> LFS objects are already available in the cache (e.g. if you switch branches\n> back and forth). I implemented these \"cache hits\" natively in Git.\n> Please note that this implementation is just a quick and dirty proof of\n> concept. If the Git community agrees that this kind of approach would be\n> acceptable then I will start to work on a proper patch series with cross\n> platform support and unit tests.\n\nWould it be possible to move all this code to a separate daemon?\nInstead of spawning a new process to do the filtering, you send a\ncommand \"convert this\" over maybe unix socket and either receive the\nwhole result over the socket, or receive a path of the result.\n\nI don't think hard coding \"git-lfs\" is a good way to go (if you keep\nthat in the final impl. of course). I guess the costly part is\nspawning processes and going through the same process initialization\nfor every object. If we keep a daemon running, all that is gone. You\nstill have to pay for extra context switches and memory copy (unless\nyou send the path, but then it could be racy), but I think that's\nnegligible. And all smudge/clean filters can do caching and more if\nthey want to.\n-- \nDuy\n"},{"id":"290245","messageId":"xmqqmvm6bom5.fsf@gitster.mtv.corp.google.com","threadId":"42720","inReplyTo":"1467005913-6503-1-git-send-email-larsxschneider@gmail.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-27T16:09:06Z","receivedAt":"2016-06-27T16:09:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"larsxschneider@gmail.com writes:\n\n> Unfortunately that fix helps only with cloning. Any local Git operation\n> that invokes the clean/smudge filter (e.g. switching branches) is still\n> slow.\n\nDo you know where the slowness comes from?  Does Joey's new\nclean/smudge interface help GitLFS?\n\nYou are not likely to get anything that knows that a blob object may\nbe named as anything other than SHA-1(\"blob <len>\" + <contents>) to\nGit core.  The remote-object-store idea that was floated by Peff and\nChristian started running with at least maintains that object naming\nproperty and has a better chance of interacting better with the core,\nbut LFS, Annex or anything that would not preserve the object naming\nwould not.\n\nPersonally, I view a surrogate blob left by LFS in the tree object\nand filtered via clean/smudge a \"smarter\" kind of symbolic link that\npoints outside what Git controls.  The area outside what Git\ncontrols is left to be managed by whatever the add-on does; Git\nshouldn't even be aware of how they are structured and/or managed.\n\n\n"},{"id":"290356","messageId":"alpine.DEB.2.20.1606281139490.12947@virtualbox","threadId":"42720","inReplyTo":"CACsJy8B7ie_oQRZ7Pw3NTo4YyXwZOer-iDUVxDPSY_uhrQi1bw@mail.gmail.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-06-28T09:40:09Z","receivedAt":"2016-06-28T09:40:47Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Mon, 27 Jun 2016, Duy Nguyen wrote:\n\n> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n> > ## Proposed solution\n> > Git LFS caches its objects under .git/lfs/objects. Most of the time\n> > Git LFS objects are already available in the cache (e.g. if you switch\n> > branches back and forth). I implemented these \"cache hits\" natively in\n> > Git.  Please note that this implementation is just a quick and dirty\n> > proof of concept. If the Git community agrees that this kind of\n> > approach would be acceptable then I will start to work on a proper\n> > patch series with cross platform support and unit tests.\n> \n> Would it be possible to move all this code to a separate daemon?\n> Instead of spawning a new process to do the filtering, you send a\n> command \"convert this\" over maybe unix socket and either receive the\n> whole result over the socket, or receive a path of the result.\n\nUnix sockets are not really portable...\n\nCiao,\nDscho\n"},{"id":"290375","messageId":"CACsJy8DCovHoutyEYwaxdCTU4K-h8u_JTOwt8GUiNTkkXzGz3w@mail.gmail.com","threadId":"42720","inReplyTo":"alpine.DEB.2.20.1606281139490.12947@virtualbox","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-28T13:11:43Z","receivedAt":"2016-06-28T13:12:18Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jun 28, 2016 at 11:40 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi Duy,\n>\n> On Mon, 27 Jun 2016, Duy Nguyen wrote:\n>\n>> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n>> > ## Proposed solution\n>> > Git LFS caches its objects under .git/lfs/objects. Most of the time\n>> > Git LFS objects are already available in the cache (e.g. if you switch\n>> > branches back and forth). I implemented these \"cache hits\" natively in\n>> > Git.  Please note that this implementation is just a quick and dirty\n>> > proof of concept. If the Git community agrees that this kind of\n>> > approach would be acceptable then I will start to work on a proper\n>> > patch series with cross platform support and unit tests.\n>>\n>> Would it be possible to move all this code to a separate daemon?\n>> Instead of spawning a new process to do the filtering, you send a\n>> command \"convert this\" over maybe unix socket and either receive the\n>> whole result over the socket, or receive a path of the result.\n>\n> Unix sockets are not really portable...\n\nIt's the same situation as index-helper. I expect you guys will\nreplace the transport with named pipe or similar.\n-- \nDuy\n"},{"id":"290376","messageId":"alpine.DEB.2.20.1606281513380.12947@virtualbox","threadId":"42720","inReplyTo":"CACsJy8DCovHoutyEYwaxdCTU4K-h8u_JTOwt8GUiNTkkXzGz3w@mail.gmail.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-06-28T13:14:43Z","receivedAt":"2016-06-28T13:15:33Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Tue, 28 Jun 2016, Duy Nguyen wrote:\n\n> On Tue, Jun 28, 2016 at 11:40 AM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n> >\n> > On Mon, 27 Jun 2016, Duy Nguyen wrote:\n> >\n> >> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n> >> > ## Proposed solution\n> >> > Git LFS caches its objects under .git/lfs/objects. Most of the time\n> >> > Git LFS objects are already available in the cache (e.g. if you\n> >> > switch branches back and forth). I implemented these \"cache hits\"\n> >> > natively in Git.  Please note that this implementation is just a\n> >> > quick and dirty proof of concept. If the Git community agrees that\n> >> > this kind of approach would be acceptable then I will start to work\n> >> > on a proper patch series with cross platform support and unit\n> >> > tests.\n> >>\n> >> Would it be possible to move all this code to a separate daemon?\n> >> Instead of spawning a new process to do the filtering, you send a\n> >> command \"convert this\" over maybe unix socket and either receive the\n> >> whole result over the socket, or receive a path of the result.\n> >\n> > Unix sockets are not really portable...\n> \n> It's the same situation as index-helper. I expect you guys will\n> replace the transport with named pipe or similar.\n\nYes, I will have to work on that. But I might need to ask for a change in\nthe design if I hit some obstacle there: named pipes are not the same at\nall as Unix sockets.\n\nRead: it will be painful, and not a general solution. So every new Unix\nsocket that you introduce will introduce new problems for me.\n\nCiao,\nDscho\n"},{"id":"290377","messageId":"9603B6F9-3BEF-4779-84C3-6DC61D7FBC46@gmail.com","threadId":"42720","inReplyTo":"xmqqmvm6bom5.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-06-28T13:22:59Z","receivedAt":"2016-06-28T13:23:13Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jun 2016, at 18:09, Junio C Hamano <gitster@pobox.com> wrote:\n> \n> larsxschneider@gmail.com writes:\n> \n>> Unfortunately that fix helps only with cloning. Any local Git operation\n>> that invokes the clean/smudge filter (e.g. switching branches) is still\n>> slow.\n> \n> Do you know where the slowness comes from?  Does Joey's new\n> clean/smudge interface help GitLFS?\n\nI am pretty sure the startup time of the external clean/smudge process\ncauses the slowness and consequently I don't think Joey's patch would help. \nThe following tests makes me believe that:\n\nI ran the same test as in my original email using the repo with 15,000 \nLFS files. Instead of the LFS binary I use the fast and simple shell \nbuilt-in `true` command:\n\n$ git -c filter.lfs.smudge=true -c filter.lfs.clean=true clone https://github.com/larsxschneider/lfstest-manyfiles.git\n$ cd lfstest-manyfiles/\n$ time git -c filter.lfs.smudge=true -c filter.lfs.clean=true checkout removed-files\n\nreal\t0m47.030s\nuser\t0m29.521s\nsys\t0m16.993s\n\nIt still takes 47 seconds to switch the branch. Does this test prove my\npoint or do you see a flaw in the test?\n\n\n> You are not likely to get anything that knows that a blob object may\n> be named as anything other than SHA-1(\"blob <len>\" + <contents>) to\n> Git core.  The remote-object-store idea that was floated by Peff and\n> Christian started running with at least maintains that object naming\n> property and has a better chance of interacting better with the core,\n> but LFS, Annex or anything that would not preserve the object naming\n> would not.\n> \n> Personally, I view a surrogate blob left by LFS in the tree object\n> and filtered via clean/smudge a \"smarter\" kind of symbolic link that\n> points outside what Git controls.  The area outside what Git\n> controls is left to be managed by whatever the add-on does; Git\n> shouldn't even be aware of how they are structured and/or managed.\n\nI understand and somewhat anticipated your point of view. I will try\nto find a less intrusive solution.\n\n@Christian/Peff: \nIs there a place to look for more info about your remote-object-store idea? \n\nThanks,\nLars"},{"id":"290379","messageId":"5491DA7A-4B1E-4015-879F-F01FF7481143@gmail.com","threadId":"42720","inReplyTo":"alpine.DEB.2.20.1606281513380.12947@virtualbox","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-06-28T13:43:38Z","receivedAt":"2016-06-28T13:43:51Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 28 Jun 2016, at 15:14, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> \n> Hi Duy,\n> \n> On Tue, 28 Jun 2016, Duy Nguyen wrote:\n> \n>> On Tue, Jun 28, 2016 at 11:40 AM, Johannes Schindelin\n>> <Johannes.Schindelin@gmx.de> wrote:\n>>> \n>>> On Mon, 27 Jun 2016, Duy Nguyen wrote:\n>>> \n>>>> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n>>>>> ## Proposed solution\n>>>>> Git LFS caches its objects under .git/lfs/objects. Most of the time\n>>>>> Git LFS objects are already available in the cache (e.g. if you\n>>>>> switch branches back and forth). I implemented these \"cache hits\"\n>>>>> natively in Git.  Please note that this implementation is just a\n>>>>> quick and dirty proof of concept. If the Git community agrees that\n>>>>> this kind of approach would be acceptable then I will start to work\n>>>>> on a proper patch series with cross platform support and unit\n>>>>> tests.\n>>>> \n>>>> Would it be possible to move all this code to a separate daemon?\n>>>> Instead of spawning a new process to do the filtering, you send a\n>>>> command \"convert this\" over maybe unix socket and either receive the\n>>>> whole result over the socket, or receive a path of the result.\n>>> \n>>> Unix sockets are not really portable...\n>> \n>> It's the same situation as index-helper. I expect you guys will\n>> replace the transport with named pipe or similar.\n> \n> Yes, I will have to work on that. But I might need to ask for a change in\n> the design if I hit some obstacle there: named pipes are not the same at\n> all as Unix sockets.\n> \n> Read: it will be painful, and not a general solution. So every new Unix\n> socket that you introduce will introduce new problems for me.\n\nThanks Duy for your suggestion. I considered a daemon, but a daemon makes\nit always harder for the user as the user needs to ensure the daemon is \nrunning! Plus, Dscho's concerns regarding Windows.\n\nI think the core problem is that we invoke the filter for every file:\nhttps://github.com/git/git/blob/master/convert.c#L461-L475\n\nCouldn't we start the filter executable at the beginning of the Git process\nand communicate with it via stdin/stdout whenever we hit the Git filter \ncode? Would that work?\n\nAlternatively, do you see a way to add a \"plugin\" system to Git? Where Git\ncould be configured to dynamically load a \"filter\" library?\n\n@Dscho:\nDo you have a recommendation for interprocess communication that works \nwithout trouble on Windows? \n\nThanks,\nLars"},{"id":"290380","messageId":"CAP8UFD1Zr+Fu3j=kzkbHzHm2TH0fY93r2GmAE-rzqQfg4Aa4Xw@mail.gmail.com","threadId":"42720","inReplyTo":"9603B6F9-3BEF-4779-84C3-6DC61D7FBC46@gmail.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T13:53:47Z","receivedAt":"2016-06-28T13:54:00Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Jun 28, 2016 at 3:22 PM, Lars Schneider\n<larsxschneider@gmail.com> wrote:\n>\n> @Christian/Peff:\n> Is there a place to look for more info about your remote-object-store idea?\n\nYou may want to take a look at:\n\nhttps://github.com/chriscool/git/commits/external-odb\n\nI just updated it and I may send an updated RFC series from this\nbranch to the list soon.\n"},{"id":"290386","messageId":"CACsJy8BLhO3R=8rAqLwAXc3LGLVVr4KGtWOm4MAh7bKDMn+uPw@mail.gmail.com","threadId":"42720","inReplyTo":"alpine.DEB.2.20.1606281513380.12947@virtualbox","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-28T15:50:14Z","receivedAt":"2016-06-28T15:50:49Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jun 28, 2016 at 3:14 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi Duy,\n>\n> On Tue, 28 Jun 2016, Duy Nguyen wrote:\n>\n>> On Tue, Jun 28, 2016 at 11:40 AM, Johannes Schindelin\n>> <Johannes.Schindelin@gmx.de> wrote:\n>> >\n>> > On Mon, 27 Jun 2016, Duy Nguyen wrote:\n>> >\n>> >> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n>> >> > ## Proposed solution\n>> >> > Git LFS caches its objects under .git/lfs/objects. Most of the time\n>> >> > Git LFS objects are already available in the cache (e.g. if you\n>> >> > switch branches back and forth). I implemented these \"cache hits\"\n>> >> > natively in Git.  Please note that this implementation is just a\n>> >> > quick and dirty proof of concept. If the Git community agrees that\n>> >> > this kind of approach would be acceptable then I will start to work\n>> >> > on a proper patch series with cross platform support and unit\n>> >> > tests.\n>> >>\n>> >> Would it be possible to move all this code to a separate daemon?\n>> >> Instead of spawning a new process to do the filtering, you send a\n>> >> command \"convert this\" over maybe unix socket and either receive the\n>> >> whole result over the socket, or receive a path of the result.\n>> >\n>> > Unix sockets are not really portable...\n>>\n>> It's the same situation as index-helper. I expect you guys will\n>> replace the transport with named pipe or similar.\n>\n> Yes, I will have to work on that. But I might need to ask for a change in\n> the design if I hit some obstacle there: named pipes are not the same at\n> all as Unix sockets.\n>\n> Read: it will be painful, and not a general solution. So every new Unix\n> socket that you introduce will introduce new problems for me.\n\nI thought we could have a drop-in replacement (or maybe a higher\nabstraction that would be sufficient for git). Thanks for pointing it\nout.\n-- \nDuy\n"},{"id":"290387","messageId":"CACsJy8AVSa+=PgYwPAx3HmFJ3Gb51K9TJxtGSQjZWTH=DazvLA@mail.gmail.com","threadId":"42720","inReplyTo":"5491DA7A-4B1E-4015-879F-F01FF7481143@gmail.com","subject":"Re: [RFC] Native access to Git LFS cache","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-28T16:00:41Z","receivedAt":"2016-06-28T16:01:30Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jun 28, 2016 at 3:43 PM, Lars Schneider\n<larsxschneider@gmail.com> wrote:\n>\n>> On 28 Jun 2016, at 15:14, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n>>\n>> Hi Duy,\n>>\n>> On Tue, 28 Jun 2016, Duy Nguyen wrote:\n>>\n>>> On Tue, Jun 28, 2016 at 11:40 AM, Johannes Schindelin\n>>> <Johannes.Schindelin@gmx.de> wrote:\n>>>>\n>>>> On Mon, 27 Jun 2016, Duy Nguyen wrote:\n>>>>\n>>>>> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:\n>>>>>> ## Proposed solution\n>>>>>> Git LFS caches its objects under .git/lfs/objects. Most of the time\n>>>>>> Git LFS objects are already available in the cache (e.g. if you\n>>>>>> switch branches back and forth). I implemented these \"cache hits\"\n>>>>>> natively in Git.  Please note that this implementation is just a\n>>>>>> quick and dirty proof of concept. If the Git community agrees that\n>>>>>> this kind of approach would be acceptable then I will start to work\n>>>>>> on a proper patch series with cross platform support and unit\n>>>>>> tests.\n>>>>>\n>>>>> Would it be possible to move all this code to a separate daemon?\n>>>>> Instead of spawning a new process to do the filtering, you send a\n>>>>> command \"convert this\" over maybe unix socket and either receive the\n>>>>> whole result over the socket, or receive a path of the result.\n>>>>\n>>>> Unix sockets are not really portable...\n>>>\n>>> It's the same situation as index-helper. I expect you guys will\n>>> replace the transport with named pipe or similar.\n>>\n>> Yes, I will have to work on that. But I might need to ask for a change in\n>> the design if I hit some obstacle there: named pipes are not the same at\n>> all as Unix sockets.\n>>\n>> Read: it will be painful, and not a general solution. So every new Unix\n>> socket that you introduce will introduce new problems for me.\n>\n> Thanks Duy for your suggestion. I considered a daemon, but a daemon makes\n> it always harder for the user as the user needs to ensure the daemon is\n> running! Plus, Dscho's concerns regarding Windows.\n>\n> I think the core problem is that we invoke the filter for every file:\n> https://github.com/git/git/blob/master/convert.c#L461-L475\n>\n> Couldn't we start the filter executable at the beginning of the Git process\n> and communicate with it via stdin/stdout whenever we hit the Git filter\n> code? Would that work?\n\nYeah if one filter process per one git process still brings\nsignificant perf. gain for you, why not, it's simpler than daemon.\nThough you may want to look at Christian's external odb first in the\nother mail. Note though that external odb may still spawn process a\nlot (because the design is you cache objects locally once and you\ndon't have to spawn again). Whether that fits in lfs scheme, I have no\nidea (I have never used git-lfs myself).\n\n> Alternatively, do you see a way to add a \"plugin\" system to Git? Where Git\n> could be configured to dynamically load a \"filter\" library?\n\nI don't think plugins as .so files are welcome because it would force\nus to freeze some ABI. So far all git extension has always been via an\nexternal process.\n-- \nDuy\n"}]}