Why are ref_lists sorted?

Having got a C implementation of fetch far enough to be able to tell me that I'm up-to-date (when using git:// and not using remotes or branches files) I thought I'd compare performance with the fetch--tool based version. I was a bit surprised at what I saw.

Hot-cache time went from ~30s to ~3s, which didn't seem too bad. However what did puzzle me was where the C version was spending it's time. Being a work in progress it's a bit chatty in places, but there is an up-front delay of just over 1s (i.e. before my first message, which is the name of the remote to fetch).

A bit of investigation showed this to be due to the first attempt to read a ref causing the packed refs to be loaded. In my test repo the packed-refs file has over 9000 entries, but I still thought that it would load faster than that. It turns out that the overhead is from sorting the refs when building the ref_list. If I remove the code for sorting the entries I lose that initial 1s delay, without appearing to break anything in the fetch. However I assume that it's there for a reason ...

So my questions are:

1) what have I broken by removing the sort?
2) is it worth trying to optimise the sort?