You really need the proper harness. I'm betting by tapping into that wire your amp is on all the time regardless of key position. 04+ do not have a switched 12V to tap into, it's the same circuit as the interior dimmer lights etc.

Btw, that factory unit is worth it's weight in gold around here.