It ended up being a vacuum leak on the intake manifold vacuum line tee. The one under the supercharger pully. I fixed it and the problem went away.

I fixed another problem too where the car would start up, then stall right away. It was a bad IAC.