Cloud Telecons 09/26/2016
Telecon Info
To start the online portion of the Personal Conference meeting
- Go to https://global.gotomeeting.com/join/445841573
- You can also dial in using your phone: United States +1 (571) 317-3112
- Access Code: 445-841-573
September 26, 2016 ESIP cloud computing cluster telecon recap
Participants:Bruce Caron, Fei Hu, Caller o2, Chaowei Yang, Frack Greguska, Gill, Kevin M, Jeffrey R Hall, Joseph C Jacob, Li Angela W., Nga T Quach, Stephana Klene, Vakhnin Andrey A , Walter Jeff
(this list might miss somebody else. If there is anyone else, please help add them here. Thanks!)
Presentation: NEXUS Framework for Data Ingestion and Processing (Frank Greguska)
Two scalable databases:
- Cassandra: break into some tiles -> database -> tiles
- Solr DB cluster: Index chunks spatially -> identify data quickly -> real time data analytics
NEXUS Interface: draw bounding box
Big data problems: 4 V’s : volume 14.6PB, velocity 16.0 TB/Day, Variety 9,462 datasets, veracity different level products have different quality
Upcoming Big data mission: surface water and Ocean Topography(SWOT)
Volume:
- Tiling algorithms : Solr for spatial extents and metadata; cassandra: tile data; horizontal scalable
Variety:
- configurable streams: multiple tiling algorithms, L2 Swath data, L3/L4 Gridded data
- customized tiling algorithms
Velocity:
- Horizontally scalable streams
- Cloud: OpenStack, Amazon => NEXUS deployed on cloud
- how do you deploy NEXUS on cloud? How to scale up and scale down
Veracity:
- pluggable validation checks: there are actual data in the tile
- data transformation
How to implement?
- Spring XD: Zookeeper, Kafka for node communication, Redis
- Spring Cloud Data Flow
If you have any more questions, please send them to greguska@jpl.nasa.gov
