Re: [PATCH 0/3] Another approach to large transactions

From: Stefan Beller <sbeller@google.com>
Date: Apr 21, 2015, 17:31 UTC
Message-ID: <CAGZ79kbev7OS_-xZAizSr0SSwdAZfn4U-3QRy7GprexJt=CzZw@mail.gmail.com>
In-Reply-To: <xmqqa8y1iekw.fsf@gitster.dls.corp.google.com>

On Tue, Apr 21, 2015 at 10:19 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 18 quoted lines

> Stefan Beller <sbeller@google.com> writes:
>
>> On Mon, Apr 20, 2015 at 3:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>
>>> On the core management side, xmalloc() and friends retry upon
>>> failure, after attempting to free the resource.  I wonder if your
>>> codepath can do something similar to that, perhaps?
>>
>> But then we'd need to think about which fds can be 'garbage collected'.
>> The lock files certainly can be closed and reopened.
>
> ... and that is the natural thing to garbage collect, no?  After
> all, this approach allows lock-file subsystem to keep fds open even
> when they do not absolutely have to, so they are the best candidates
> to be shot down first when things gets tight.
>
> A good thing is that you have a list of all them already for use by
> the atexit handler ;-)

Yes, but when such a garbage collection is in place, it is part of the API, which means you need to check all places, where lock files are used, so that you have the pattern

   open lock file
   ...
   if lock_file->lk == -1
       lock_file_reopen(...)
   use(lock_file->lk)

I think that could be easily integrated into the lock_file API when the API
users don't directly read the values of the lock file struct, but rather call
a method
    int lock_file_get_fd(lock);

which guarantees to return a valid fd, but that is not long term stable. You can use the fd for the next action, but it may become garbage collected again.

So then you don't know how long the fd is valid as the garbage collection may be either triggered from the lock file code or by yourself?

There is some uncertainty on when which data is valid, which is why I dislike this approach and rather prefer to make it explicit. If we run into such a problem in another place, we can still think about a general solution in the lock file API.