Halfpenny wary of Brown threat