threads / rfc / 32702

RFC patchRevoke write access to refs and odb after importing another repo's odb

Subject: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

## tl;dr

7 messages between Jan 23, 2013 and Jan 24, 2013. Diffs are folded; open one to read it.

replies: 6people: 3as markdown or json

Nguyễn Thái Ngọc Duy· Jan 23, 2013, 13:34 UTC · lore

add_submodule_odb() can be used to import objects from another repository temporarily. After this point we don't know which objects are ours, which are external. If we create an object that refers to an external object, next time git runs, it may find a hole in the object graph because the external repository may not be imported. The same goes for pointing a ref to an external SHA-1.

To protect ourselves, once add_submodule_odb() is used:
 - trees, tags and commits cannot be created
 - refs cannot be updated

In certain cases that submodule code knows that it's safe to write, it can turn the readonly flag off.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 I think this is a good safety check. It catches at least a case in
 t7405.3. I did not investigate further though.
 cache.h      | 1 +
 refs.c       | 2 ++
 sha1_file.c  | 2 ++
 submodule.c  | 7 +++++++
 5 files changed, 16 insertions(+)
Show changes to 4 files +12 −0

cache.h, refs.c, sha1_file.c, submodule.c

diff --git a/cache.h b/cache.h
index c257953..772d229 100644
--- a/cache.h
+++ b/cache.h
@@ -753,6 +753,7 @@ extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
+extern int git_repo_readonly();
 
 /* global flag to enable extra checks when accessing packed objects */
 extern int do_check_packed_object_crc;
diff --git a/refs.c b/refs.c
index 541fec2..22b13f4 100644
--- a/refs.c
+++ b/refs.c
@@ -1711,6 +1711,8 @@ struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha
 struct ref_lock *lock_any_ref_for_update(const char *refname,
 					 const unsigned char *old_sha1, int flags)
 {
+	if (git_repo_readonly())
+		die("repository in read-only mode, cannot update refs");
 	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
 		return NULL;
 	return lock_ref_sha1_basic(refname, old_sha1, flags, NULL);
diff --git a/sha1_file.c b/sha1_file.c
index 40b2329..b9e8b59 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -2575,6 +2575,8 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign
 	char hdr[32];
 	int hdrlen;
 
+	if (git_repo_readonly() && strcmp(type, "blob"))
+		die("repository in read-only mode, cannot update object database");
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
 	 */
diff --git a/submodule.c b/submodule.c
index 2f55436..5eba597 100644
--- a/submodule.c
+++ b/submodule.c
@@ -19,6 +19,7 @@ static struct string_list changed_submodule_paths;
 static int initialized_fetch_ref_tips;
 static struct sha1_array ref_tips_before_fetch;
 static struct sha1_array ref_tips_after_fetch;
+static int readonly;
 
 /*
  * The following flag is set if the .gitmodules file is unmerged. We then
@@ -30,6 +31,11 @@ static struct sha1_array ref_tips_after_fetch;
  */
 static int gitmodules_is_unmerged;
 
+int git_repo_readonly()
+{
+	return readonly;
+}
+
 static int add_submodule_odb(const char *path)
 {
 	struct strbuf objects_directory = STRBUF_INIT;
@@ -67,6 +73,7 @@ static int add_submodule_odb(const char *path)
 	/* add possible alternates from the submodule */
 	read_info_alternates(objects_directory.buf, 0);
 	prepare_alt_odb();
+	readonly = 1;
 done:
 	strbuf_release(&objects_directory);
 	return ret;
-- 
1.8.0.rc2.23.g1fb49df
Duy Nguyen· Jan 23, 2013, 13:38 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

On Wed, Jan 23, 2013 at 8:34 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
Show 11 quoted lines
> add_submodule_odb() can be used to import objects from another
> repository temporarily. After this point we don't know which objects
> are ours, which are external. If we create an object that refers to an
> external object, next time git runs, it may find a hole in the object
> graph because the external repository may not be imported. The same
> goes for pointing a ref to an external SHA-1.
>
> To protect ourselves, once add_submodule_odb() is used:
>
>  - trees, tags and commits cannot be created
>  - refs cannot be updated

.. and soon after I pressed send, I realized I missed at least two other places write should not be allowed:

 - index
 - reflog

But the general idea is probably more important than implementation details at this stage.

-- 
Duy
Junio C Hamano· Jan 23, 2013, 17:01 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
Show 18 quoted lines
> add_submodule_odb() can be used to import objects from another
> repository temporarily. After this point we don't know which objects
> are ours, which are external. If we create an object that refers to an
> external object, next time git runs, it may find a hole in the object
> graph because the external repository may not be imported. The same
> goes for pointing a ref to an external SHA-1.
>
> To protect ourselves, once add_submodule_odb() is used:
>
>  - trees, tags and commits cannot be created
>  - refs cannot be updated
>
> In certain cases that submodule code knows that it's safe to write, it
> can turn the readonly flag off.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  I think this is a good safety check.

Two step implementation to bring "read-only" support into a testable shape and then flip that bit in add_submdule_odb() would be a sensible approach.

I however have this suspicion that this will become a losing battle and we would be better off getting rid of add_submodule_odb(); instead operations that work across repositories will be done as a subprocess, which will get us back closer to one of the original design goals of submodule support to have a clear separation between the superproject and its submodules.

Jens Lehmann· Jan 23, 2013, 20:38 UTC · re: Junio C Hamano · lore

Re: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

Am 23.01.2013 18:01, schrieb Junio C Hamano:
Show 24 quoted lines
> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
> 
>> add_submodule_odb() can be used to import objects from another
>> repository temporarily. After this point we don't know which objects
>> are ours, which are external. If we create an object that refers to an
>> external object, next time git runs, it may find a hole in the object
>> graph because the external repository may not be imported. The same
>> goes for pointing a ref to an external SHA-1.
>>
>> To protect ourselves, once add_submodule_odb() is used:
>>
>>  - trees, tags and commits cannot be created
>>  - refs cannot be updated
>>
>> In certain cases that submodule code knows that it's safe to write, it
>> can turn the readonly flag off.
>>
>> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>> ---
>>  I think this is a good safety check.
> 
> Two step implementation to bring "read-only" support into a testable
> shape and then flip that bit in add_submdule_odb() would be a
> sensible approach.

I agree this is a worthwhile change so nobody accidentally screws things up.

>>  It catches at least a case in
>>  t7405.3. I did not investigate further though.

This is a false positive. The merge algorithm picked a fast-forward in a submodule as a proper merge result and records that in a gitlink. But as Duy pointed out this could be easily fixed by turning the readonly flag off in that case.

Show 6 quoted lines
> I however have this suspicion that this will become a losing battle
> and we would be better off getting rid of add_submodule_odb();
> instead operations that work across repositories will be done as a
> subprocess, which will get us back closer to one of the original
> design goals of submodule support to have a clear separation between
> the superproject and its submodules.

Please don't. While I agree with your goal, I'd be unhappy to do that because of the performance drop (especially on fork-challenged operating systems).

Junio C Hamano· Jan 23, 2013, 21:06 UTC · re: Jens Lehmann · lore

Re: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

Jens Lehmann <Jens.Lehmann@web.de> writes:
> This is a false positive. The merge algorithm picked a fast-forward
> in a submodule as a proper merge result and records that in a
> gitlink. But as Duy pointed out this could be easily fixed by
> turning the readonly flag off in that case.

I see that as "easily circumvented and not an effective protection", though.

In theory, adding a gitlink to the index, removing a gitlink to the index and modifying an existing gitlink in the index to another gitlink in the index and writing the resulting in-core index out to the on-disk index should be allowed, even after objects from the submodule object database have contaminated our in-core object pool, as long as you do not run cache_tree_update(). I am not sure if that single loophole would be sufficient, though.

Duy Nguyen· Jan 24, 2013, 05:58 UTC · re: Junio C Hamano · lore

Re: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

On Thu, Jan 24, 2013 at 4:06 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 17 quoted lines
> Jens Lehmann <Jens.Lehmann@web.de> writes:
>
>> This is a false positive. The merge algorithm picked a fast-forward
>> in a submodule as a proper merge result and records that in a
>> gitlink. But as Duy pointed out this could be easily fixed by
>> turning the readonly flag off in that case.
>
> I see that as "easily circumvented and not an effective protection",
> though.
>
> In theory, adding a gitlink to the index, removing a gitlink to the
> index and modifying an existing gitlink in the index to another
> gitlink in the index and writing the resulting in-core index out to
> the on-disk index should be allowed, even after objects from the
> submodule object database have contaminated our in-core object pool,
> as long as you do not run cache_tree_update().  I am not sure if that
> single loophole would be sufficient, though.

The problem is we don't know which entries are updated in index. We don't keep track of them. And I think in the unpack-trees case, we scape the whole index then copy over, making it look like the whole index is updated (even with the same content). One way to check this is verify the source of all non-gitlink entries in index before writing to disk (only when readonly flag is on, of course). sha1_object_info_extended() should help (or be extended to do the job). Hmm.. if we do this, we could also verify if new sha-1 objects do not refer to an external source, if so allow them to be created.

-- 
Duy
Duy Nguyen· Jan 24, 2013, 01:30 UTC · re: Junio C Hamano · lore

Re: [PATCH/RFC] Revoke write access to refs and odb after importing another repo's odb

On Thu, Jan 24, 2013 at 12:01 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 6 quoted lines
> I however have this suspicion that this will become a losing battle
> and we would be better off getting rid of add_submodule_odb();
> instead operations that work across repositories will be done as a
> subprocess, which will get us back closer to one of the original
> design goals of submodule support to have a clear separation between
> the superproject and its submodules.

It does not have to be subprocess. Thomas Rast did some work on support multithread access to object db by basically replicating all datastructure per thread. If that work is complete, we have something like "odb container" that could be used to access objects from another repository and it won't contaminate the original odb. The same thing can be done for ref and index access.

-- 
Duy

← back to recent threads