It is time to take drastic action against the Filipino armed men after their intrusion turned deadly in Lahad Datu and Semporna, Sabah, said former prime minister Dr Mahathir Mohamad.