All 4 bolts must be tight on the caliper.

2 for the bracket and 2 for the caliper itself, if you left them loose the caliper bolts will fall out and so will the caliper.

The brakes will always be nearly touching the rotor.