Metal/sludge stuck to the magnet is normal for the first time the pan has been dropped.

If you drop the pan now and see more metal/sludge, then I would be concerned.