threads / patch / 46187

patchcheckout: don't write merge results into the object database

Subject: [PATCH] checkout: don't write merge results into the object database

## tl;dr

4 messages between Jun 15, 2017 and Jun 15, 2017. Diffs are folded; open one to read it.

replies: 3people: 3as markdown or json

René Scharfe· Jun 15, 2017, 11:33 UTC · lore

Merge results need to be written to the worktree, of course, but we don't necessarily need object entries for them, especially if they contain conflict markers. Use pretend_sha1_file() to create fake blobs to pass to make_cache_entry() and checkout_entry() instead.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 builtin/checkout.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Show changes to builtin/checkout.c +3 −3

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 1624eed7e7..f51c15af9f 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -215,7 +215,7 @@ static int checkout_merged(int pos, const struct checkout *state)
 
 	/*
 	 * NEEDSWORK:
-	 * There is absolutely no reason to write this as a blob object
+	 * There is absolutely no reason to build a fake blob object
 	 * and create a phony cache entry.  This hack is primarily to get
 	 * to the write_entry() machinery that massages the contents to
 	 * work-tree format and writes out which only allows it for a
@@ -225,8 +225,8 @@ static int checkout_merged(int pos, const struct checkout *state)
 	 * (it also writes the merge result to the object database even
 	 * when it may contain conflicts).
 	 */
-	if (write_sha1_file(result_buf.ptr, result_buf.size,
-			    blob_type, oid.hash))
+	if (pretend_sha1_file(result_buf.ptr, result_buf.size,
+			      OBJ_BLOB, oid.hash))
 		die(_("Unable to add merge result for '%s'"), path);
 	free(result_buf.ptr);
 	ce = make_cache_entry(mode, oid.hash, path, 2, 0);

-- 
2.13.0

Jeff King· Jun 15, 2017, 13:57 UTC · re: René Scharfe · lore

Re: [PATCH] checkout: don't write merge results into the object database

On Thu, Jun 15, 2017 at 01:33:42PM +0200, René Scharfe wrote:

> Merge results need to be written to the worktree, of course, but we
> don't necessarily need object entries for them, especially if they
> contain conflict markers.  Use pretend_sha1_file() to create fake
> blobs to pass to make_cache_entry() and checkout_entry() instead.

Conceptually this makes sense, although I have a comment below.

Out of curiosity, did this come up when looking into the cherry-pick segfault/error from a few hours ago?

Show 10 quoted lines

> @@ -225,8 +225,8 @@ static int checkout_merged(int pos, const struct checkout *state)
>  	 * (it also writes the merge result to the object database even
>  	 * when it may contain conflicts).
>  	 */
> -	if (write_sha1_file(result_buf.ptr, result_buf.size,
> -			    blob_type, oid.hash))
> +	if (pretend_sha1_file(result_buf.ptr, result_buf.size,
> +			      OBJ_BLOB, oid.hash))
>  		die(_("Unable to add merge result for '%s'"), path);
>  	free(result_buf.ptr);

I wondered if pretend_sha1_file() makes a copy of the buffer, and indeed it does. So this is correct.

But that raises an interesting question: how big are these objects, and is it a good idea to store them in RAM? Obviously they're in RAM already, but I'm not sure if it's one-at-a-time. We could be bumping the peak RAM used if there's a large number of these entries. Touching the on-disk odb does feel hacky, but it also means they behave like other objects.

If it is a concern, I think it could be solved by "unpretending" after our call to checkout_entry completes. That would need a new call in sha1_file.c, but it should be easy to write.

-Peff

René Scharfe· Jun 15, 2017, 15:48 UTC · re: Jeff King · lore

Re: [PATCH] checkout: don't write merge results into the object database

Am 15.06.2017 um 15:57 schrieb Jeff King:

Show 11 quoted lines

> On Thu, Jun 15, 2017 at 01:33:42PM +0200, René Scharfe wrote:
> 
>> Merge results need to be written to the worktree, of course, but we
>> don't necessarily need object entries for them, especially if they
>> contain conflict markers.  Use pretend_sha1_file() to create fake
>> blobs to pass to make_cache_entry() and checkout_entry() instead.
> 
> Conceptually this makes sense, although I have a comment below.
> 
> Out of curiosity, did this come up when looking into the cherry-pick
> segfault/error from a few hours ago?

No, it came up in the discussion of Dscho's memory leak plug series [1].

Show 24 quoted lines

>> @@ -225,8 +225,8 @@ static int checkout_merged(int pos, const struct checkout *state)
>>   	 * (it also writes the merge result to the object database even
>>   	 * when it may contain conflicts).
>>   	 */
>> -	if (write_sha1_file(result_buf.ptr, result_buf.size,
>> -			    blob_type, oid.hash))
>> +	if (pretend_sha1_file(result_buf.ptr, result_buf.size,
>> +			      OBJ_BLOB, oid.hash))
>>   		die(_("Unable to add merge result for '%s'"), path);
>>   	free(result_buf.ptr);
> 
> I wondered if pretend_sha1_file() makes a copy of the buffer, and indeed
> it does. So this is correct.
> 
> But that raises an interesting question: how big are these objects, and
> is it a good idea to store them in RAM? Obviously they're in RAM
> already, but I'm not sure if it's one-at-a-time. We could be bumping the
> peak RAM used if there's a large number of these entries. Touching the
> on-disk odb does feel hacky, but it also means they behave like other
> objects.
> 
> If it is a concern, I think it could be solved by "unpretending" after
> our call to checkout_entry completes. That would need a new call in
> sha1_file.c, but it should be easy to write.

Good point; we'd accumulate fake entries that we'll never need again. The patch should clean them up.

Alternatively we could finally address the NEEDSWORK comment and provide a way to checkout a file without faking an index entry..

René

[1] https://public-inbox.org/git/2704e145927c851c4163a68cfdfd5ada48fff21d.1493906085.git.johannes.schindelin@gmx.de/

Junio C Hamano· Jun 15, 2017, 21:56 UTC · re: René Scharfe · lore

Re: [PATCH] checkout: don't write merge results into the object database

René Scharfe <l.s.r@web.de> writes:

Show 5 quoted lines

>> If it is a concern, I think it could be solved by "unpretending" after
>> our call to checkout_entry completes. That would need a new call in
>> sha1_file.c, but it should be easy to write.
>
> Good point; we'd accumulate fake entries that we'll never need again.

Hopefully we are not pretending to have the same object from two callsites; this one may knows the merged one no longer needs to be in core, but some other callsite wanted to pretend a blob with the same contents is in the object store, what happens to it? I do not think we want to refcount ;-)

> The patch should clean them up.
>
> Alternatively we could finally address the NEEDSWORK comment and
> provide a way to checkout a file without faking an index entry..

Yeah, we should not need index entry, and we should not need the blob object name.

    ... thinks a bit more ...

Having said that, in the longer term, it may be safe to write an actual object out to the object store. The convert-to-working-tree backends currently work only on raw bytes, but it is not inplausible for some new interfaces to want to pass the object name to the backend and tell it either togive raw (converted) bytes back, or to write out the bytes directly to the filesystem, bypassing the main Git process. If we "pretend" in this process, not just we accumulate cruft in-core as Peff points out, we risk giving out an invalid object name to such external mechanisms.

I do not think it is too bad to leave a handful of temporary blobs that are written out by "git checkout -m <other-branch>" in the object store to be GC'ed.