{"thread":{"id":"64990","subject":"[PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","startedAt":"2026-02-13T12:47:28Z","lastAt":"2026-02-20T16:48:58Z","messageCount":9,"participants":["George Hu","Chris Torek","Phillip Wood","Jeff King","Ed Maste","Collin Funk"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"535925","messageId":"20260213124656.218396-1-integral@archlinux.org","threadId":"64990","inReplyTo":null,"subject":"[PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"George Hu","fromEmail":"integral@archlinux.org","sentAt":"2026-02-13T12:46:56Z","receivedAt":"2026-02-13T12:47:28Z","isPatch":true,"sender":{"key":"integral@archlinux.org","avatar":"https://gravatar.com/avatar/8f3d5e9fafe5e5809d88513c99befff5acd3d1c0182cbd2546cb449ece82b02b?d=mp&s=160"},"body":"The `sendfile()` system call copies data between one file descriptor\nand another within the kernel, which is more efficient than the\ncombination of `read()` and `write()`.\n\nSigned-off-by: George Hu \n---\n copy.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/copy.c b/copy.c\nindex b668209b6c..d4b7cde764 100644\n--- a/copy.c\n+++ b/copy.c\n@@ -7,8 +7,23 @@\n #include \"strbuf.h\"\n #include \"abspath.h\"\n \n+#ifdef __linux__\n+# include \n+#endif\n+\n int copy_fd(int ifd, int ofd)\n {\n+#ifdef __linux__\n+\tstruct stat ifd_st;\n+\tsize_t ifd_len;\n+\tssize_t ret = 0;\n+\n+\tfstat(ifd, &ifd_st);\n+\tifd_len = ifd_st.st_size;\n+\n+\twhile (ifd_len && (ret = sendfile(ofd, ifd, NULL, ifd_len)) > 0)\n+\t\tifd_len -= (size_t)ret;\n+#else\n \twhile (1) {\n \t\tchar buffer[8192];\n \t\tssize_t len = xread(ifd, buffer, sizeof(buffer));\n@@ -19,6 +34,8 @@ int copy_fd(int ifd, int ofd)\n \t\tif (write_in_full(ofd, buffer, len) < 0)\n \t\t\treturn COPY_WRITE_ERROR;\n \t}\n+#endif\n+\n \treturn 0;\n }\n \n-- \n2.53.0\n\n"},{"id":"535938","messageId":"CAPx1GvfsnZiUkkXJSAtt4HECT6f+zb1+vtiT+UikcAq0WuFR8Q@mail.gmail.com","threadId":"64990","inReplyTo":"20260213124656.218396-1-integral@archlinux.org","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2026-02-13T15:36:20Z","receivedAt":"2026-02-13T15:36:34Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Fri, Feb 13, 2026 at 4:47 AM George Hu wrote:\n> The `sendfile()` system call copies data between one file descriptor\n> and another within the kernel, which is more efficient than the\n> combination of `read()` and `write()`.\n\nsendfile() is found on other systems (notably BSDs), so perhaps ...\n\n> Signed-off-by: George Hu \n> ---\n> copy.c | 17 +++++++++++++++++\n> 1 file changed, 17 insertions(+)\n>\n> diff --git a/copy.c b/copy.c\n> index b668209b6c..d4b7cde764 100644\n> --- a/copy.c\n> +++ b/copy.c\n> @@ -7,8 +7,23 @@\n> #include \"strbuf.h\"\n> #include \"abspath.h\"\n>\n> +#ifdef __linux__\n\n... this and the subsequent ifdef should be based on the feature,\nrather than the OS.\n\nChris\n"},{"id":"536005","messageId":"e2a5c920-7f15-4ea8-9390-cf42ac33082d@archlinux.org","threadId":"64990","inReplyTo":"CAPx1GvfsnZiUkkXJSAtt4HECT6f+zb1+vtiT+UikcAq0WuFR8Q@mail.gmail.com","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"George Hu","fromEmail":"integral@archlinux.org","sentAt":"2026-02-14T09:21:59Z","receivedAt":"2026-02-14T09:22:48Z","isPatch":true,"sender":{"key":"integral@archlinux.org","avatar":"https://gravatar.com/avatar/8f3d5e9fafe5e5809d88513c99befff5acd3d1c0182cbd2546cb449ece82b02b?d=mp&s=160"},"body":"On 2/13/26 11:36 PM, Chris Torek wrote:\n\n> On Fri, Feb 13, 2026 at 4:47 AM George Hu wrote:\n>> The `sendfile()` system call copies data between one file descriptor\n>> and another within the kernel, which is more efficient than the\n>> combination of `read()` and `write()`.\n> sendfile() is found on other systems (notably BSDs), so perhaps ...\n>\n>> Signed-off-by: George Hu \n>> ---\n>> copy.c | 17 +++++++++++++++++\n>> 1 file changed, 17 insertions(+)\n>>\n>> diff --git a/copy.c b/copy.c\n>> index b668209b6c..d4b7cde764 100644\n>> --- a/copy.c\n>> +++ b/copy.c\n>> @@ -7,8 +7,23 @@\n>> #include \"strbuf.h\"\n>> #include \"abspath.h\"\n>>\n>> +#ifdef __linux__\n> ... this and the subsequent ifdef should be based on the feature,\n> rather than the OS.\n>\n> Chris\n\nHello,\n\nAlthough the `sendfile()` system call exists in both Linux and BSDs, \ntheir semantics and APIs differ.\nThe Linux prototype of `sendfile()` is:\n\nssize_t sendfile(int out_fd, int in_fd, off_t *_Nullable offset, size_t \ncount);\n\nWhile FreeBSD exposes:\n\nint sendfile(int fd, int s, off_t offset, size_t nbytes, struct sf_hdtr \n*hdtr, off_t *sbytes, int flags);\n\nGeorge\n"},{"id":"536026","messageId":"bf0b3c41-9784-4494-a932-68abfa60cea6@gmail.com","threadId":"64990","inReplyTo":"20260213124656.218396-1-integral@archlinux.org","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2026-02-14T16:43:27Z","receivedAt":"2026-02-14T16:43:32Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 13/02/2026 12:46, George Hu wrote:\n> The `sendfile()` system call copies data between one file descriptor\n> and another within the kernel, which is more efficient than the\n> combination of `read()` and `write()`.\n\nDoes git copy any files big enough that this makes a noticeable difference?\n\n> int copy_fd(int ifd, int ofd)\n> {\n> +#ifdef __linux__\n\nOur normal practice when a function has platform specific \nimplementations is to host those implementations under compat/\n(see the implementations of trace2_collect_process_information() for an \nexample)\n\n> +\tstruct stat ifd_st;\n> +\tsize_t ifd_len;\n> +\tssize_t ret = 0;\n> +\n> +\tfstat(ifd, &ifd_st);\n\nWhat happens if fstat() fails?\n\n> +\tifd_len = ifd_st.st_size;\n> +\n> +\twhile (ifd_len && (ret = sendfile(ofd, ifd, NULL, ifd_len)) > 0)\n> +\t\tifd_len -= (size_t)ret;\n\nThis does not propagate errors to the caller, if sendfile() fails the \nfunction returns 0. write_in_full() handles non-blocking writes, we \nshould do the same here if we see EAGAIN. The man page lists various \nrestrictions on the file descriptors passed to sendfile() - I'm not sure \nthat they affect the uses of copy_file() in git but to be safe we should \nfall back to the read()/write() loop if we see EINVAL.\n\nThanks\n\nPhillip\n\n> +#else\n> \twhile (1) {\n> \t\tchar buffer[8192];\n> \t\tssize_t len = xread(ifd, buffer, sizeof(buffer));\n> @@ -19,6 +34,8 @@ int copy_fd(int ifd, int ofd)\n> \t\tif (write_in_full(ofd, buffer, len) < 0)\n> \t\t\treturn COPY_WRITE_ERROR;\n> \t}\n> +#endif\n> +\n> \treturn 0;\n> }\n> \n\n"},{"id":"536027","messageId":"CAPx1GvcdFN5gn97xxuNLRBy2sV2gK3Wm56Pw3tQGENN8_+BKnw@mail.gmail.com","threadId":"64990","inReplyTo":"e2a5c920-7f15-4ea8-9390-cf42ac33082d@archlinux.org","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2026-02-14T16:50:00Z","receivedAt":"2026-02-14T16:50:14Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"Ah, more importantly, FreeBSD's sendfile only operates on sockets.\n\nBoth systems also need fallback code for un-handled cases.\n\nChris\n\nOn Sat, Feb 14, 2026 at 1:22 AM George Hu wrote:\n>\n> On 2/13/26 11:36 PM, Chris Torek wrote:\n>\n> > On Fri, Feb 13, 2026 at 4:47 AM George Hu wrote:\n> >> The `sendfile()` system call copies data between one file descriptor\n> >> and another within the kernel, which is more efficient than the\n> >> combination of `read()` and `write()`.\n> > sendfile() is found on other systems (notably BSDs), so perhaps ...\n> >\n> >> Signed-off-by: George Hu \n> >> ---\n> >> copy.c | 17 +++++++++++++++++\n> >> 1 file changed, 17 insertions(+)\n> >>\n> >> diff --git a/copy.c b/copy.c\n> >> index b668209b6c..d4b7cde764 100644\n> >> --- a/copy.c\n> >> +++ b/copy.c\n> >> @@ -7,8 +7,23 @@\n> >> #include \"strbuf.h\"\n> >> #include \"abspath.h\"\n> >>\n> >> +#ifdef __linux__\n> > ... this and the subsequent ifdef should be based on the feature,\n> > rather than the OS.\n> >\n> > Chris\n>\n> Hello,\n>\n> Although the `sendfile()` system call exists in both Linux and BSDs,\n> their semantics and APIs differ.\n> The Linux prototype of `sendfile()` is:\n>\n> ssize_t sendfile(int out_fd, int in_fd, off_t *_Nullable offset, size_t\n> count);\n>\n> While FreeBSD exposes:\n>\n> int sendfile(int fd, int s, off_t offset, size_t nbytes, struct sf_hdtr\n> *hdtr, off_t *sbytes, int flags);\n>\n> George\n"},{"id":"536039","messageId":"77a887d0-51bf-4bf6-af8f-d5555dab2fe2@archlinux.org","threadId":"64990","inReplyTo":"bf0b3c41-9784-4494-a932-68abfa60cea6@gmail.com","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"George Hu","fromEmail":"integral@archlinux.org","sentAt":"2026-02-15T06:23:16Z","receivedAt":"2026-02-15T06:23:24Z","isPatch":true,"sender":{"key":"integral@archlinux.org","avatar":"https://gravatar.com/avatar/8f3d5e9fafe5e5809d88513c99befff5acd3d1c0182cbd2546cb449ece82b02b?d=mp&s=160"},"body":"On 2/15/26 12:43 AM, Phillip Wood wrote:\n\n> On 13/02/2026 12:46, George Hu wrote:\n>> The `sendfile()` system call copies data between one file descriptor\n>> and another within the kernel, which is more efficient than the\n>> combination of `read()` and `write()`.\n>\n> Does git copy any files big enough that this makes a noticeable \n> difference?\n>\n>>   int copy_fd(int ifd, int ofd)\n>>   {\n>> +#ifdef __linux__\n>\n> Our normal practice when a function has platform specific \n> implementations is to host those implementations under compat/\n> (see the implementations of trace2_collect_process_information() for \n> an example)\n>\n\nThe Linux implementation of `trace2_collect_process_information()` \nresides in compat/linux with a stub version in compat/stub. After moving \nthe Linux-specifc `copy_fd()` implementation into compat/linux, where \nshould the generic implementation be placed?\n\n>> +    struct stat ifd_st;\n>> +    size_t ifd_len;\n>> +    ssize_t ret = 0;\n>> +\n>> +    fstat(ifd, &ifd_st);\n>\n> What happens if fstat() fails?\n>\n>> +    ifd_len = ifd_st.st_size;\n>> +\n>> +    while (ifd_len && (ret = sendfile(ofd, ifd, NULL, ifd_len)) > 0)\n>> +        ifd_len -= (size_t)ret;\n>\n> This does not propagate errors to the caller, if sendfile() fails the \n> function returns 0. write_in_full() handles non-blocking writes, we \n> should do the same here if we see EAGAIN. The man page lists various \n> restrictions on the file descriptors passed to sendfile() - I'm not \n> sure that they affect the uses of copy_file() in git but to be safe we \n> should fall back to the read()/write() loop if we see EINVAL.\n>\n\nAccording to the manual, `sendfile()` returns -1 on failure; a return \nvalue of 0 indicates EOF.\n\nThere are error cases besides EAGAIN and EINVAL. Maybe we should fall \nback to the read() / write() loop for errors other than EAGAIN?\n\nSincerely,\nGeorge\n\n> Thanks\n>\n> Phillip\n>\n>> +#else\n>>       while (1) {\n>>           char buffer[8192];\n>>           ssize_t len = xread(ifd, buffer, sizeof(buffer));\n>> @@ -19,6 +34,8 @@ int copy_fd(int ifd, int ofd)\n>>           if (write_in_full(ofd, buffer, len) < 0)\n>>               return COPY_WRITE_ERROR;\n>>       }\n>> +#endif\n>> +\n>>       return 0;\n>>   }\n>\n"},{"id":"536041","messageId":"20260215074306.GA86282@coredump.intra.peff.net","threadId":"64990","inReplyTo":"20260213124656.218396-1-integral@archlinux.org","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-02-15T07:43:06Z","receivedAt":"2026-02-15T07:43:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 13, 2026 at 08:46:56PM +0800, George Hu wrote:\n\n> The `sendfile()` system call copies data between one file descriptor\n> and another within the kernel, which is more efficient than the\n> combination of `read()` and `write()`.\n\nOK, but...does this efficiency matter for the callers of copy_file() and\nfriends? Just skimming over grep results, it mostly seems to be used\nwith files we'd expect to be small. Some possible exceptions I can see:\n\n - bundle-uri with a file:// uri will use it (so this could be a big\n packfile)\n\n - in \"clone --local\" mode without hardlinks available, we might copy a\n packfile\n\nTo some degree, if it's easy to use sendfile(), we should just do so if\nit might be a bit faster. But as responses from others showed, there are\nsome complications we'd have to deal with (portability, fallback, etc).\nSo I think it would be a lot more compelling if we could show a\nmeasurable speedup for some real world operation.\n\n-Peff\n"},{"id":"536535","messageId":"CAPyFy2Dj6z9OuYFF9Tpn98mrd3Hs6BD75B73B+jkqp9aVM8X_w@mail.gmail.com","threadId":"64990","inReplyTo":"CAPx1GvcdFN5gn97xxuNLRBy2sV2gK3Wm56Pw3tQGENN8_+BKnw@mail.gmail.com","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2026-02-20T16:35:45Z","receivedAt":"2026-02-20T16:35:58Z","isPatch":true,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Sat, 14 Feb 2026 at 11:50, Chris Torek wrote:\n>\n> Ah, more importantly, FreeBSD's sendfile only operates on sockets.\n\nTrue. If benchmarking shows this is profitable then we'd want to use\ncopy_file_range(2) on FreeBSD.\n"},{"id":"536537","messageId":"87o6ljtl6w.fsf@gmail.com","threadId":"64990","inReplyTo":"CAPyFy2Dj6z9OuYFF9Tpn98mrd3Hs6BD75B73B+jkqp9aVM8X_w@mail.gmail.com","subject":"Re: [PATCH] copy.c: use `sendfile()` for in-kernel file copying on Linux","fromName":"Collin Funk","fromEmail":"collin.funk1@gmail.com","sentAt":"2026-02-20T16:48:55Z","receivedAt":"2026-02-20T16:48:58Z","isPatch":true,"sender":{"key":"collin.funk1@gmail.com","avatar":"https://avatars.githubusercontent.com/u/65689063?v=4"},"body":"Ed Maste writes:\n\n> On Sat, 14 Feb 2026 at 11:50, Chris Torek wrote:\n>>\n>> Ah, more importantly, FreeBSD's sendfile only operates on sockets.\n>\n> True. If benchmarking shows this is profitable then we'd want to use\n> copy_file_range(2) on FreeBSD.\n\nWe use copy_file_range in GNU Coreutils. Note that it has quite a few\nissues [1], including a recent one affecting files larger than INT_MAX\nbytes [2][3].\n\nCoreutils has Gnulib to work around this stuff, and the performance\nimprovement is meaningful for 'cp'. Based on earlier messages in this\nthread, I am not sure if it is worth dealing with in this case.\n\nCollin\n\n[1] https://lwn.net/Articles/789527/\n[2] https://sourceware.org/PR33245\n[3] https://bugs.gnu.org/79139\n"}]}