The vacuum going to the booster should have a rectifier (one way valve). Sounds like your's is not working.

Edited: I guess better term is a check valve.