I'd think it's your booster if it makes air sounds, because it uses engine vacuum. If the pedal comes up a little bit under higher than usual vacuum it may be normal, I've driven vehicles that do that. When you let off the gas vacuum in the engine is higher because the RPM's are still high yet the throttle position is at idle. Try hitting the brakes real quick with the engine running in park, and listen to the sound it makes. It should sound like it's sucking air from the inside of the car. Is it the same noise as what you're describing it does while driving?
And, welcome
