The new NASA Astrophysics Data System (ADS) is designed with a serviceoriented architecture (SOA) that consists of multiple customized Apache Solr search engine instances plus a collection of microservices, containerized using Docker, and deployed in Amazon Web Services (AWS). For complex systems, like the ADS, this loosely coupled architecture can lead to a more scalable, reliable and resilient system if some fundamental questions are addressed. After having experimented with different AWS environments and deployment methods, we decided in December 2017 to go with Kubernetes as our container orchestration. Defining the best strategy to properly setup Kubernetes has shown to be challenging: automatic scaling services and load balancing traffic can lead to errors whose origin is difficult to identify, monitoring and logging the activity that happens across multiple layers for a single request needs to be carefully addressed, and the best workflow for a Continuous Integration and Delivery (CI/CD) system is not self-evident. We present here how we tackle these challenges and our plans for the future.
@article{arxiv.1901.05463,
title = {Fundamentals of effective cloud management for the new NASA Astrophysics Data System},
author = {Sergi Blanco-Cuaresma and Alberto Accomazzi and Michael J. Kurtz and Edwin Henneken and Carolyn S. Grant and Donna M. Thompson and Roman Chyla and Stephen McDonald and Golnaz Shapurian and Timothy W. Hostetler and Matthew R. Templeton and Kelly E. Lockhart and Kris Bukovi and Nathan Rapport},
journal= {arXiv preprint arXiv:1901.05463},
year = {2019}
}
Comments
To appear in the proceedings of the 28th annual international Astronomical Data Analysis Software & Systems (ADASS XXVIII)