git switch has fatal dependency on default fetch config

Changing the default fetch refspec for a remote breaks git switch:

% git branch -r
  origin/HEAD -> origin/master
  origin/feature
  origin/master
% git remote set-branches origin master
% git switch -c feature --track origin/feature
fatal: cannot set up tracking information; starting point 'origin/feature' is not a branch
% git remote set-branches --add origin feature
% git switch -c feature --track origin/feature
branch 'feature' set up to track 'origin/feature'.
Switched to a new branch 'feature'

It seems like I should be able to fetch a remote branch and track it without having to monkey around with my default fetch config. Is there a reason git switch has a hard dependency on the default remote fetch refspec configuration?