My catalog only shows 4 nuts holding the rails in place. Better check and make sure nothing is holding them in place, there is a MAP Sensor bracket on the back side, it maybe attached or in the way.

Otherwise consider an 04 is now 9 years old, if they haven't been removed since being built they're not gonna play nice, you have to convince them Just be careful, don't break anything. Mine were a pain on my 99, but they listened after some work.