I assume the sleeve is metal (haven't seen the inside of my wheel before). Could you get in there with a dremel tool and just cut slots on opposite sides (like cutting it down the middle but w/o going through the shaft) and then use a chisel or punch to crack it in half around the steering shaft?
If done right you shouldn't damage the shaft.