Mike, I had a fun time daily driving my '03 M3 and traveled to Charlotte >20 times and have to saw that it's a very practical car with plenty of juice in stock form. The only downside is that it can be jarring, although mine had stock shocks before I sold it with 62k.
I really do miss the car and actually would like to buy another one in a few years when it's more practical for me as a third car, especially if the prices keep plummeting.
My experience, although brief, autocrossing the car was really a lot of fun; more so than I thought it would be. I never ended up taking the plunge on 18" wheels and r-comps, but having raced about a 1/2 dozen events in TIR on RE01R's, I was surprised at how well the car did running about 4-5 seconds off the quick AS S2k's on a long course like Laurinburg. Keep in mind I had little RWD autocross experience and all stock parts.
So doing a little simple bench racing mathematics (fwiw), I'd say an experienced rear wheel driver could surely knock a second or two off my time letting the car 'hang out' a little and a set of stickies would get things pretty competitive IMO. I'm not sure about the tire wear but Chuck recommended a large front sway bar to alleviate the tire destruction.
I'm pretty confident you can find a clean '03.5 or newer model around $20k as they've seriously taken a huge hit in value in the last year.
If you get a change of heart and decide the E46 M3 is not for you, don't watch this video:
http://www.youtube.com/watch?v=9ZGwpifULe4