Removing the rough casting finish wont hurt anything but maybe your hands and arms from all the labor needed, not to mention cost of materials and tools.
Just dont change the size or shape of any of the ports.