If you have enough damage that the cam is worn then something else failed. As for the lifters you are 100% safe to replace them on a roller setup without worry on hurting the cam. Soak the lifters beforehand in a pan of oil to fill them up. This will prevent any damage internally to the lifters during first startup. Also you will need to replace the rocker bolts with new ones as they are torque to yield. If you plan on modding in the future you can get re-usable comp cams rocker bolts from vendors like Zzperformance.