If you overfilled your transmission with fluid, it will cause it to foam. If the leak is substantial enough, you could be losing prime with the pump or just foaming the fluid due to air getting in. The "yellow stuff" on the fittings is sealant, and I wouldn't use teflon tape there at all personally. I would spend the $5-$10 for another connector, clean up the threads in the radiator and install without the teflon tape. Post pictures if it still leaks after trying this. Hopefully you didn't strip anything out or mess up the threads. I doubt it, but I've seen it before.
