An ideal software system in computer graphics should be a combination of innovative ideas, solid software engineering and rapid development. However, in reality these requirements are seldom met simultaneously. In this paper, we present early results on an open-source library named Taichi (http://taichi.graphics), which alleviates this practical issue by providing an accessible, portable, extensible, and high-performance infrastructure that is reusable and tailored for computer graphics. As a case study, we share our experience in building a novel physical simulation system using Taichi.