Prime Minister Abdullah Ahmad Badawi, who a day ago arrested three people under the draconian Internal Security Act, has brushed aside calls from within his own Umno party to step down, warning dissenters to stop all talk of an early leadership change.