Yeah you'd have to fill in the holes and paint it.
If you do ever take them off, just remember they were there for a reason, mainly to help prevent those assholes that whip there doors open from putting dents into yours.