Pull harder..

seriously, I just had mine off to paint/clear.

They are splined, so it takes a good bit of pressure. Easiest way i found was to pull up on the arm itself, and it should pop right out, (the arm should flex up some, just use it as leverage