The "vast majority" of over-80s and care home residents should be vaccinated before local areas move on to giving jabs to lower priority groups, a cabinet minister has told Sky News.