The problem is that, at present, we will not willy-nilly just eradicate the IRS. Not going to happen today or tomorrow. However, we can starve the beast.
First, the flat tax. Now, the flat tax would eliminate MUCH of the IRS's power almost instantly without appearing to do so. You may find yourself asking, "how?" Glad you asked. With a flat tax there are no tax loop holes, no hiddent tax traps, no complicated mathematical equations that only God can understand and then even He is left scratching his head wondering why we do this to ourselves. The flat tax means calculations are simple and then if you don't pay enough they know instantly and audits lose their bite.
Second, now comes the good part. You see, the IRS is now weakened and starving. No need to have somebody go over millions of documents so no need to have a ton of people on staff to check these overly complex documents or go busting down doors because somebody was short five bucks. So, the layoffs begin because you can't justify keeping such a large force around to make sure somebody did a simple X * N calculation correctly.
Third, now comes the better part. With a weakend IRS their clout goes down. When this happens it gets easier to lower the tax. Say we start off with an N% tax. Well, look at that; another election is coming around. My party is doing rather poorly in the poles so let's pass a 1% tax cut. It'll just be for a year and then we'll go back to 15%. Oh, damn, we're getting stomped in the polls. We'll make that tax cut permanant; (N - 1)% is still a good amount of money coming in. Wash, rince, and repeat. The tax rate is slowly killed off and eventually we get the near zero to zero tax rate we want.
A flat tax may not KILL the income tax completely but it will eventually give us something better. And any small victory is better than no victory at all.