Cloud Telecons 09/26/2016

From Earth Science Information Partners (ESIP)

Telecon Info[edit | edit source]

To start the online portion of the Personal Conference meeting

  1. Go to
  2. You can also dial in using your phone: United States +1 (571) 317-3112
  3. Access Code: 445-841-573

September 26, 2016 ESIP cloud computing cluster telecon recap[edit | edit source]

Participants:Bruce Caron, Fei Hu, Caller o2, Chaowei Yang, Frack Greguska, Gill, Kevin M, Jeffrey R Hall, Joseph C Jacob, Li Angela W., Nga T Quach, Stephana Klene, Vakhnin Andrey A , Walter Jeff (this list might miss somebody else. If there is anyone else, please help add them here. Thanks!)

Presentation: NEXUS Framework for Data Ingestion and Processing (Frank Greguska)[edit | edit source]

Two scalable databases:

  • Cassandra: break into some tiles -> database -> tiles
  • Solr DB cluster: Index chunks spatially -> identify data quickly -> real time data analytics

NEXUS Interface: draw bounding box

Big data problems: 4 V’s : volume 14.6PB, velocity 16.0 TB/Day, Variety 9,462 datasets, veracity different level products have different quality

Upcoming Big data mission: surface water and Ocean Topography(SWOT)

  • Tiling algorithms : Solr for spatial extents and metadata; cassandra: tile data; horizontal scalable


  • configurable streams: multiple tiling algorithms, L2 Swath data, L3/L4 Gridded data
  • customized tiling algorithms


  • Horizontally scalable streams
  • Cloud: OpenStack, Amazon => NEXUS deployed on cloud
how do you deploy NEXUS on cloud? How to scale up and scale down


  • pluggable validation checks: there are actual data in the tile
  • data transformation

How to implement?

  • Spring XD: Zookeeper, Kafka for node communication, Redis
  • Spring Cloud Data Flow

If you have any more questions, please send them to