Check your grounds. If you have a faulty ground it will act as another load and it will take battery voltage from the amp.

Are you using a capacitor?