You've about hit the nail on the head. This sound system produces power fine, but quality is garbage. When I had this system, I ended up dialing the rear back and installing a rear 10" sub with a small amp to fill in the low frequencies. Had a buddy of mine fabricate a custom housing that fit where our awkward trunk "cubbie" on the right hand side and it looked stock and never got in the way.
It filled in the frequencies nicely and wasn't too loud. I'm not into super loud bass, I just want to hear the whole soundstage.
All the wiring diagrams are free if you create a free AutoZone account. Just Google entertainment schematic 2004 pontiac grand prix autozone.
I ended up replacing the center speaker and adding a filter for the noise. Though, in the end, I ended up running new speaker wires everywhere, used a 30 band eq, passive and active crossovers, and replaced all speakers so I went a bit overboard... Haha