How to checkout a revision that contains a deleted submodule?

Maybe this is a FAQ, but I couldn't figure it out!

I have a repo which has a couple of submodules.

At some point in the past I deleted one of those submodules:

    git rm sub2
    git add -u
    git commit -m 'Deleting sub2'
    git push origin
    ...
    ... more commits and pushes...

Now I go and clone the head revision. This gives me a clone which has
nothing present in .git/modules/sub2.
    login on some other machine
    git clone git@my.repo:thing
    cd thing
    ls .git/modules
    <sub2 not present>

So when I go and checkout an old revision where sub2 is still around I get:
    git checkout oldrevision
    fatal: not a git repository: sub2/../.git/modules/sub2

What am I doing wrong? What set of commands do I need to use to ensure that this will always do the right thing?

Thanks Luke