This is another one of those things that pisses me off to no end.
If the c-clip used on the axle has sharp edges it get's caught when trying to remove.

I always take the c-clip off and file the outside edges round before install...makes it super easy to remove.