You only need to remove the pulley.

Not the entire shaft.