You'd have to cut the speaker wires on the harness that attaches to the HU and make sure they're all covered well so they don't short circuit. Other than that, you should be fine. Just make sure they're secured well and will not short.
On that same note, if you ever decide to switch back to HU power then you'd have to do the same short protection on the speaker wire going to the amp.