The swift removal of race-based affirmative action that is present in the country now will be potentially destabilising for the nation, a Universiti Malaya academician says.