VHPC’13 [email protected] FLEXIBLE STORAGE FOR HPC CLOUDS WITH AARRCCHHIIPPEELLAAGGOO AANNDD CCEEPPHH. VVAANNGGEELLIISS KKOOUUKKIISS TECHNICAL LEAD, SYNNEFO 1 Outline VHPC’13 [email protected] Archipelago overview Storagge resources,, clones/snappshots HPC workflow with Archipelago RReessoouurrccee CCoommppoossiittiioonn Archipelago Implementation Flexible I/O pipelines Integgration with Syynnefo Future directions 2 Archippelaggo overview VHPC’13 [email protected] Distributed Storage System - Poweringg storagge in clouds Decouples storage resources from storage backends - FFiilleess // IImmaaggeess // VVoolluummeess // SSnnaappsshhoottss Unified way to provision, handle, and present resources Decouples logic from actual physical storage - Software-Defined Storagge 3 Archippelaggo loggic VHPC’13 [email protected] Thin provisioning, with clones and snapshots - Independent from the underlying storage technology Hash-based data deduplication PPlluuggggaabbllee aarrcchhiitteeccttuurree - Multiple endpoint (northbound) drivers - MMullttiiplle bbackkendd ((soutthhbboundd)) ddriivers Multiple storage backends - Unified management - wwiitthh ssttoorraaggee mmiiggrraattiioonnss 4 Unified view of resources VHPC’13 [email protected] Files - User files, with Dropbox-like syncing Images - TTeemmppllaatteess ffoorr VVMM ccrreeaattiioonn Volumes - Live disks, as seen from VMs Snapshots - PPoiintt-iin-ttiime snapshhotts off VVollumes 5 VHPC’13 [email protected] Spawn mmyy oowwnn UUbbuunnttuu Freeze 6 VHPC’13 [email protected] Clone UUbbuunnttuu ++ uusseerr ddaattaa Snapshot 7 The bigg ppicture VHPC’13 [email protected] Archipelago Core Storagge backend 1 Storagge backend 2 Storagge backend 3 (e.g., Ceph cluster 1) (e.g., Ceph cluster 2) (e.g., NFS over NAS) 8 VHPC’13 [email protected] 9 End-to-end workflow with unified storagge VHPC’13 [email protected] 10