A man who stood in the rain as he saw his sick father for the last time has said he's "devastated" and "very angry" that he followed lockdown rules while Downing Street staff held a Christmas party.