Keep reflogs for deleted (remote tracking) branches?

> As far as I can tell, even "core.logAllRefUpdates=always" does *not*
> keep any reflog entries around, even temporarily (until reflog
> expiry), once a ref  is deleted - do I understand that correctly? Is
> this behavior intentional / reasoned, or just a consequence of the
> fact that it's *hard* to keep "managing" per-branch reflogs for
> branches that don't exist?
>
> I am planning a workaround using server hooks to "back up" refs that
> are being deleted from specific namespaces, in my specific case, and I
> imagine that a system like github keeps track of deleted stuff itself
> for a while, but I find this "per-ref reflog disappearance" behavior
> puzzling / out-of-character, so wanted to make sure I'm not missing
> something.

> On Tue, Mar 8, 2022 at 12:28 PM Tao Klerks <tao@klerks.biz> wrote:
>> As far as I can tell, even "core.logAllRefUpdates=always" does *not*
>> keep any reflog entries around, even temporarily (until reflog
>> expiry), once a ref  is deleted - do I understand that correctly? Is
>> this behavior intentional / reasoned, or just a consequence of the
>> fact that it's *hard* to keep "managing" per-branch reflogs for
>> branches that don't exist?
>>
>> I am planning a workaround using server hooks to "back up" refs that
>> are being deleted from specific namespaces, in my specific case, and I
>> imagine that a system like github keeps track of deleted stuff itself
>> for a while, but I find this "per-ref reflog disappearance" behavior
>> puzzling / out-of-character, so wanted to make sure I'm not missing
>> something.
>
> I think this behavior is motivated by directory/file conflicts. If you
> have a reflog file in refs/logs/foo, you can't create a reflog for
> refs/foo/bar, because that would live in refs/logs/foo/bar
>
> At Google, we keep reflogs in a completely different storage system
> altogether, which avoids this problem, and I wouldn't be surprised if
> other large hosting providers do something similar.

>
>
> On Tue, Mar 08 2022, Han-Wen Nienhuys wrote:
>
> > On Tue, Mar 8, 2022 at 12:28 PM Tao Klerks <tao@klerks.biz> wrote:
> >> As far as I can tell, even "core.logAllRefUpdates=always" does *not*
> >> keep any reflog entries around, even temporarily (until reflog
> >> expiry), once a ref  is deleted - do I understand that correctly? Is
> >> this behavior intentional / reasoned, or just a consequence of the
> >> fact that it's *hard* to keep "managing" per-branch reflogs for
> >> branches that don't exist?
> >>
> >> I am planning a workaround using server hooks to "back up" refs that
> >> are being deleted from specific namespaces, in my specific case, and I
> >> imagine that a system like github keeps track of deleted stuff itself
> >> for a while, but I find this "per-ref reflog disappearance" behavior
> >> puzzling / out-of-character, so wanted to make sure I'm not missing
> >> something.
> >
> > I think this behavior is motivated by directory/file conflicts. If you
> > have a reflog file in refs/logs/foo, you can't create a reflog for
> > refs/foo/bar, because that would live in refs/logs/foo/bar
> >
> > At Google, we keep reflogs in a completely different storage system
> > altogether, which avoids this problem, and I wouldn't be surprised if
> > other large hosting providers do something similar.

>
> I once worked on a system where:
>
>  * References would be "archived", i.e. just a backup system that would
>    run "git fetch" without pruning.
>
>  * You were only allowed to push to either existing branches like
>    "master", or names with exactly one slash in them, e.g. "avar/topic",
>    not "avar/topic/nested", for that you'd need "avar/topic-nested" or
>    whatever.
>
> The second item neatly avoids D/F conflicts, at the cost of some
> grumbling from people who can't use their preferred branch name.
>
> And you can easily implement backups without that constraint by fetching
> refs/* to refs/YYYYMMDD-HHMMSS/* or whatever, and have some manual
> pruning process in place for those "secondly refs".

>
> Another way to solve a similar problem is to have
> pre-receive/post-receive hooks log attempted/successful pushes, which
> along with an appropriate "gc" policy will allow you to manually look up
> these older branches (or even to fetch them, if you publish the log and
> set uploadpack.allowAnySHA1InWant=true).

> On Tue, Mar 8, 2022 at 2:05 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> More generally I have not really run into this as a practical
>> problem.
>
> That's fair, nor have I - but I *have* come reasonably close: one
> person accidentally deletes a branch that someone else had prepared
> *without even realizing*, and the initial author is not available, and
> I only find out about it a few hours later. Dangling commit hunt, here
> we come. (the original author became available and re-pushed before it
> came to that)

>> Another way to solve a similar problem is to have
>> pre-receive/post-receive hooks log attempted/successful pushes, which
>> along with an appropriate "gc" policy will allow you to manually look up
>> these older branches (or even to fetch them, if you publish the log and
>> set uploadpack.allowAnySHA1InWant=true).
>
> Yep, that's closer to my expected plan, thanks - my intent is to back
> up, on force-push and/or deletion, into a specific refspace with a
> cleanup policy, using a server hook. So after something is "deleted"
> (or force-pushed away), it can be easily recovered for a period of eg
> 3 months in that refspace, eg
> "refs/force-push-backups/YYYY-MM-DD-<BRANCHNAME>-<HASHPREFIX>".
>
> My question is specifically about the, in my opinion, very surprising
> behavior of deleting reflogs along with deleted branches - I mainly
> provided the example use-case for context.

>
> I think you might find it interesting to have pre-receive hooks
> e.g. reject pushes if you're deleting a topic whose commits aren't
> entirely <your author> i.e. just something like:
>
>     git push -o ireallymeanit=1 --delete topic
>
> I.e. it's an easy to implement extra safety check that people can always
> opt-out of, print a scary message and most people will think twice :)

> > My question is specifically about the, in my opinion, very surprising
> > behavior of deleting reflogs along with deleted branches - I mainly
> > provided the example use-case for context.
>
> Yes it's quite a mess, e.g. if you follow the rabit hole at the
> recent[1].
>
> One fundimental problem (discussed in various places around the reftable
> backend) is that we carry N meanings for an empty reflog:
>
> A. "This is an active branch, but we have expired the entries".
>
> B. "I manually created this, knowing that the various core.* configs
>    around reflog will say "oh, a reflog exists, let's log to it" (in
>    some cases).
>
> C. Another is: This is "stale" log, i.e. no branch exists, but the log
>    is there.
>
> Which is one reason[2] we'd delete them on branch deletion, because
> otherwise we'd start logging again when a branch is re-created, which
> possibly isn't what we wanted.
>
> 1. https://lore.kernel.org/git/de5e2b0e290791d0a4f58a893d8571b5fc8c4f1a.1646952843.git.avarab@gmail.com
> 2. I'm not saying this was intended, and haven't looked into this case,
>    just that's it it's an emergent effect of how these files are treated
>    now.

> Hi folks,
> 
> I have a practical question in case I missed something.
> 
> Imagine a small team (10ppl) working on a single centralized repo, in
> github for example. They regularly create new branches, and typically
> delete them eventually - after merging, or at other times when
> branches were a dead end or whatever. The members of this team all
> have a "simple" git remote configuration, the result of a "git clone"
> with no special configuration. One exception is that they have set
> "fetch.prune" to "true", because otherwise remote branches that have
> been deleted (in the context of completed merges, or arbitrarily by
> other team members) accumulate locally and having to explicitly prune
> them from time to time is a pain. Every time someone says "why do I
> still see these branches in my repo?", someone else replies "oh, just
> run 'git config fetch.prune true'".
> 
> Now, one day someone deletes a branch accidentally from the server,
> and the sole author of that branch has gone on vacation (or has an IT
> failure, or has left the company, or whatever). Other team members
> have seen this branch go by, it's appeared in their "fetch" output,
> but no-one remembers checking it out, so it's not in their main
> "HEAD" reflogs.
> 
> Even though the ref was at one point on every team member's computer,
> and they still undoubtedly have a dangling commit in their repos,
> they're going to have a hard time finding it - there are many dangling
> commits in any given repo.
> 
> Now my question: is there any way to (temporarily) keep a reflog for
> that deleted/pruned branch, in those team members' repos?
> 
> As far as I can tell, even "core.logAllRefUpdates=always" does *not*
> keep any reflog entries around, even temporarily (until reflog
> expiry), once a ref  is deleted - do I understand that correctly? Is
> this behavior intentional / reasoned, or just a consequence of the
> fact that it's *hard* to keep "managing" per-branch reflogs for
> branches that don't exist?
> 
> I am planning a workaround using server hooks to "back up" refs that
> are being deleted from specific namespaces, in my specific case, and I
> imagine that a system like github keeps track of deleted stuff itself
> for a while, but I find this "per-ref reflog disappearance" behavior
> puzzling / out-of-character, so wanted to make sure I'm not missing
> something.
> 
> Thanks,
> Tao
> 

>
>
> On 3/8/22 6:27 AM, Tao Klerks wrote:
> >
> > I have a practical question in case I missed something.
> >
>
> Have you considered having each team member have their own
> private fork of the repo?  Then their branches are theirs
> alone and no one else needs to see or collide with them.