Prime Minister Dr Mahathir Mohamad is concerned that Beijing might resort to force if the situation in Hong Kong persists or escalates.
"If they cannot handle, if this thing doesn't stop, and the demands become more and more for autonomy or independence, I think the Chinese will not tolerate that," Mahathir said during an interview with Japan's NHK public television.
"I never thought that a country with two different systems can really work for any length of the time, and sure enough this has happened."
Hong Kong has been rocked by massive protests for the last three months ever since a controversial extradition bill was proposed.
The proposed legislation would have allowed Hong Kong residents to be sent to mainland China for trial, and was largely viewed as eroding the autonomy the former British colony has enjoyed since returned to Chinese control in 1997.
The demonstrations have seen violent clashes between the police, protestors and counter-protest forces.
The territory's leader Carrie Lam on Wednesday announced the withdrawal of the extradition bill.
The announcement, in an internal meeting with pro-establishment lawmakers and Hong Kong delegates of China’s National People’s Congress, came just days after she told business leaders she had caused “unforgivable havoc” by introducing the bill.
If she had a choice she would apologise and resign, according to a leaked audio recording.
But activists have vowed not to stop protesting until their other demands are met, including an independent probe into alleged police brutality against protesters, the unconditional release of those detained and greater democracy.
Mahathir was in Japan's ancient capital of Kyoto on Friday to receive an honorary doctorate from Doshisha University and visit high-tech companies.