Probabilistic programming systems enable users to encode model structure and naturally reason about uncertainties, which can be leveraged towards improved Bayesian optimization (BO) methods. Here we present a probabilistic program embedding of BO that is capable of addressing main issues such as problematic domains (noisy, non-smooth, high-dimensional) and the neglected inner-optimization. Not only can we utilize programmable structure to incorporate domain knowledge to aid optimization, but dealing with uncertainties and implementing advanced BO techniques become trivial, crucial for use in practice (particularly for non-experts). We demonstrate the efficacy of the approach on optimization benchmarks and a real-world drug development scenario.