Former prime minister Dr Mahathir Mohamad would never have let a member of his cabinet be called "unfit" or "foolish", a former minister says.