You don't have to take that pulley off...
BUT - don't flog the car either. The point is you need to invest in exhaust modifications - SOON - or you are running a risk of this happening:
On the list of performance modifications - a pulley drop is about #6, to be honest - when talking about a bone stock car.
You need to get (in no certain order):
1) An exhaust manifold upgrade - either PLog, headers, something
2) A catless - or hi-flow catted downpipe - get that U-bend gone ASAP
3) An improved air intake system - either home-made or complete system purchased - but it needs to be a FWI for best results.
4) Colder plugs - Autolite 104's or similar
5) A custom tuned PCM - preferably mated with some form of active, real-time scanning tool.
6) THEN - when you have done these things, at minimum, and you have A/F ratios dialed in, KR tuned out as much as you can, etc etc - THEN - you can drop a pulley size. The drop to a 3.4 is the most popular - but you have other options - personally, I think the 3.5 is
underrated for beginning performance gains - but if you do these things, you can (typically) run a 3.4 without any negative ramifications.
-Swash