Should object repacking only update server-info for packs instead of doing it for refs?

I have been experimenting with trying to run git geometric repacking after every push on our servers and have noticed that even a NOOP geometric repack takes approximately 30s on one of my large repositories. Further investigation determined that it was the update-server-info call that was taking that 30s, so running repacking with the -n instead only takes 6ms in this case! I dug a bit deeper and found that it was the update_info_refs() call that takes all the time. This seemed a bit off to me. Yes, it makes sense that would be slow, I have almost 2Mrefs, but I had to ask myself, why is this being done from git repack? It seems counterintuitive that an operation designed to repack objects would be performing maintenance of any sort on refs? Should this be improved somehow? Should there be a -packs and a -refs switch to git update-server-info, and should git repack call it with -packs? Should git pack-refs be calling git update-server-info with -refs instead?

Thanks,

-Martin