MAGIC Containerization and Virtualization Speaker Series

April 30, 2018

The Middleware and Grid Interagency Coordination Team (MAGIC), an open team under NITRD's Large Scale Networking Interagency Working Group, meets monthly to discuss current and future technologies and collaboration activities in the distributed computing. See for more information on MAGIC.

MAGIC is composed of experts from academia, industry, and government, and led by federal co-chairs, Richard Carlson, Program Director, Advanced Scientific Computing Research, Office of Science, Department of Energy and Rajiv Ramnath, Program Director, Office of Advanced Cyberinfrastructure, National Science Foundation.

MAGIC is conducting an in-depth examination of the impact of and opportunities for virtualization and containerization technologies and services on computing ecosystems. The effort was divided into 4 sessions: (1) various packages that are in common use; (2) adoption and usage of container technologies in science communities; (3) adoption and deployment by resource providers; and (4) discussion of research and operational deployment challenges.

On February 7, 2018, MAGIC kicked off the series with talks on current containerization and virtualization technologies:

  • Docker: "Strategic Overview of Docker", Douglas Thain, Associate Professor, Computer Science and Engineering, University of Notre Dame.
  • Kubernetes: "Kubernetes Overview", Rick Wagner, Professional Services Manager, Globus.
  • Shifter: "Containers for HPC", Shane Canon, Project Engineer, Data & Analytics Services Group, NERSC.
  • Singularity: "Singularity Containers", Dave Godlove, Software Engineer, Singularity.
  • Sylabs: "Sylabs Inc., An Introduction to the company behind Singularity", Greg Kurtzer, CEO, Sylabs, Inc.
  • SPIN: "NERSC Spin Project Overview", Cory Snavely, Lead, Infrastructure Services Group, NERSC.

On March 7, MAGIC hosted presentations on science community adoption and use of container technologies:

  • Advanced Light Source (ALS) Science: "Synchrotrons and Supercomputers", Dula Parkinson, Research Scientist, Experimental Systems Group, LBL.
  • Common Workflow Language (CWL): "Portable workflow creation and deployment with the CWL Standards", Michael R. Crusoe, Lead and co-founder, Common Workflow Language, a project of the Software Freedom Conservancy, and CEO, Octiog, Inc.
  • Earth System Grid Federation (ESGF) and Distributed Resources for the ESGF Advanced Management (DREAM) Project: "The Magic of Dreams", Luca Cinquini, Data Scientist, NASA Jet Propulsion Laboratory.
  • High-energy physics (HEP) Cloud: "Container Technology for HEPCloud", Panagiotis Spentzouris, Head, Scientific Computing division, Office of the CIO, Fermi National Accelerator Laboratory (Fermilab); Burt Holzman, Assistant Division Head, Scientific Computing division, Fermilab; Anthony Tiradani, Group Leader, Grid and Cloud Operations group, Fermilab.
  • US ATLAS: "Atlas Computing Model", Wei Yang, Ph.D. Information Systems Specialist, Stanford University/SLAC National Accelerator Laboratory.
  • Laser Interferometer Gravitational-Wave Observatory (LIGO): "Containing BayesWave: Deploying LIGO analyses on the OSG", James Clark, LIGO Research Scientist, Georgia Institute of Technology.

On April 4, 2018, MAGIC’s third session featured presentations by resource providers:

  • Comet: "Comet Virtual Clusters and Containers", Philip Papadopoulos, Chief Technology Officer, San Diego Supercomputer Center.
  • Open Science Grid (OSG): "Containers on OSG", Frank Wuerthwein, Director, OSG; Professor of Physics, University of California, San Diego; High Throughput Computing Lead San Diego Supercomputer Center.
  • Large Hadron Collider (LHC): "Moving CMS to a Container-Based Infrastructure", Brian Bockelman, Research Assistant Professor, Computer Science and Engineering Department, University of Nebraska-Lincoln.
  • OpenStack: "OpenStack and Container Integrations", Chris Hoge, Senior Strategic Program Manager, OpenStack Foundation.
  • AWS Container Services: "Amazon ECS," Yaniv Donenfeld, Development Manager, Container Services, Amazon Web Services.
  • Leadership Computing Facility (LCF): "Singularity: DOE LCF HPC Container Usage", Adam Simpson, High Performance Computing (HPC) Support Specialist and Programmer, Oak Ridge Leadership Computing Facility; Derek Jenson, HPC Systems Administrator, Argonne Leadership Computing Facility; Jared Punzel, HPC Systems Administrator, Argonne Leadership Computing Facility.

The fourth and final session of the deep dive into virtualization and containerization technologies and services will take place at MAGIC's May 2, 2018 meeting. During this session, participants will discuss challenges and issues and the direction and activities of federal agencies, private industry, and standards organizations. Other discussion topics include: file system access; building, validating, sharing, and customizing containers; root access versus user access space; and standardization.

MAGIC holds public meetings on the first Wednesday of each month (12-2 pm ET). If you would like more information on MAGIC's activities, please contact technical coordinator Joyce Lee at

To learn more about the Middleware and Grid Interagency Coordination Team please visit MAGIC Team page.