Difference between revisions of "Interoperability and Technology/Tech Dive Webinar Series"

From Earth Science Information Partners (ESIP)
m
 
(85 intermediate revisions by 3 users not shown)
Line 1: Line 1:
'''Bold text'''= Tech Dive Webinars =
+
'''[[Interoperability and Technology/Past Tech Dive Webinar Series|Past Tech Dive Webinars]] (2015-2022)'''
  
<br />
+
==February 15th: "Creating Great Data Products in the Cloud"==  
==10 September 2020: "SELFIE and geoconnex.us update" Dave Blodgett==
 
  
'''Time:'''
+
[https://radiant.earth/about/ (Jed Sundwall])
13, August 2020 (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
  
'''GoToMeeting''':
+
[[File:ITI_Feb2024.png|thumb|IT&I Cloud Data Products 2024]]
https://www.gotomeeting.com/join/533510693
 
  
'''Summary:'''
+
[https://www.media.mit.edu/people/shariful/overview/ Sharif Islam]
This month, I’ll be providing an update on the outcomes of the Second Environmental Linked Features Interoperability Experiment (<nowiki>https://github.com/opengeospatial/SELFIE</nowiki>) and a project using the SELFIE outcomes, <nowiki>https://geoconnex.us</nowiki>. The SELFIE project explored a Web architecture for linking environmental features and observational data with a focus on adoptability and W3C spatial data on the Web best practices. It is wrapped up and in review with the OGC Geosemantics Domain Working Group. Geoconnex.us, US-focused effort being developed by the Ineternet of Water Project at Duke and USGS Water Mission Area, is building on the outcomes of SELFIE to build a system of linked data and knowledge network of earth science data.
 
  
==13 August 2020: "ESIP Summer Meeting Highlights"==
+
Competition within the public cloud sector has reliably led to reduction in object storage costs, continual improvement in performance, and a commodification of services that have made cloud-based object storage a viable solution to share almost any volume of data. Assuming that this is true, what are the best ways to create data products in a cloud environment? This presentation will include an overview of lessons learned from Radiant Earth as they’ve advocated for adoption of cloud-native geospatial data formats and best practices.
  
'''Time'''
+
'''<u>Recording</u>''':<br />
Thursday, 13, August 2020 (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
<html>
==11 June 2020: "ESIP Collaboration Infrastructure 2.0" Erin Robinson, Ike Hecht, Lucas Cioffi, and Sheila Rabun==
+
<iframe width="560" height="315" src="https://www.youtube.com/embed/4cWGJcOcAEA?si=NYWSSB7DiGK2nrMN" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
'''Summary:'''
+
'''<u>Minutes:</u>'''
ESIP has been a virtual organization for over 20 years with two in person meetings a year, but our collaborative infrastructure was lagging. Due to COVID-19, ESIP leadership decided to move our next two in-person meetings to virtual meetings ([https://www.esipfed.org/monday-updates/special-esip-update-2020-esip-summer-2021-esip-winter-meetings-going-virtual announcement]) and to invest funding that would have been spent on travel into the ESIP collaborative infrastructure. We are currently upgrading a few components:
 
  
*Mediawiki upgrade from v1.19 to 1.34 of the ESIP wiki
+
Jed is executive director of Radiant Earth – Focus is on human cooperation on a global scale.
*Utilizing QiqoChat to bring together our asynchronous workspaces with our virtual conferences and meetings
 
*Becoming an ORCID member to gain access to ORCID API keyes to integrate ORCID authentication into the wiki
 
  
We will have three brief presentations on these components from [https://www.mediawiki.org/wiki/User:Tosfos Ike Hecht], [http://www.wikiworks.com/ WikiWorks] on the ESIP Wiki, [https://www.linkedin.com/in/lucascioffi/ Lucas Cioffi], [https://esip.qiqochat.com/ QiQoChat] lead developer on the technical side of QiQoChat and [https://orcid.org/0000-0002-1196-6279 Sheila Rabun], ORCID US Community Specialist on the ORCID API. [https://www.esipfed.org/about/leadership/staff/erin-robinson Erin Robinson], executive director of ESIP will introduce the session. 
+
Two major initiatives – Cloud Native Geospatial foundation and Source Cooperative
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/gLTlTvGEjh8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
Cloud native geospatial is about adoption of efficient approaches
 +
Source is about providing easy and accessible infrastructure
  
==14 May 2020: "CUAHSI HydroShare Update" Jerad Bales, Anthony Castronova, Jeff Horsburgh==
+
What does "Cloud Native" mean? https://guide.cloudnativegeo.org/
 +
partial reads, parallel reads, easy access to metadata
  
'''Summary:'''
+
Leveraging market pressure to make object stores cheaper and more scalable.
HydroShare (https://www.hydroshare.org) is a platform for sharing hydrologic resources (data, models, model instances, geographic coverages, etc.), enabling the scientific community to more easily and freely share products, including the data, models, and workflow scripts used to create scientific publications. HydroShare also includes a variety of social functions, such as resource sharing within a specified group, publication with a DOI, and support for integrating external applications to view and use resources without downloading them. This presentation will provide an overview of HydroShare, details of CUAHSI Compute resources which can be accessed through HydroShare or in a standalone mode, and the metadata model used in HydroShare. This presentation will also describe some community resources held by HydroShare, including comprehensive information on recent hurricanes and the complete Critical Zone Observatory data library.
 
  
More info: https://www.hydroshare.org
+
"Pace Layering" – https://jods.mitpress.mit.edu/pub/issue3-brand/release/2
  
'''About the presenters:'''
+
Observation: Software is getting cheaper and cheaper to build – it gets harder to create software monopolies in the way Microsoft or ESRI have.
  
Jerad Bales is the Executive Director of the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI), Tony Castronova is Hydrologic Scientist at CUAHSI, and Jeff Horsburgh is an associate professor in the Civil and Environmental Engineering department at Utah State University
+
This leads to a lot of diversity and a proliferation of "primitive" standards and defacto interoperability arrangements.
  
'''Slides:''' [https://www.hydroshare.org/resource/c2d7d41310af472aaec9dce57928487e/ On HydroShare.]
+
'''Source Cooperative'''
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/UvCDhHWFOT0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
Borrowed a lot from github architecturally.  
  
==9 April 2020: "Unidata Science Gateway" Julien Chastang==
+
Repository with a README
  
'''Summary:'''
+
Browser of contents in the browser.
With the goal of better serving our community, Unidata is investigating how its technologies can best make use of cloud computing. The observation that science students and professionals are spending too much time distracted by software that is difficult to access, install, and use, motivates Unidata’s investigation. In addition, cloud computing can tackle a class of problems that cannot be approached by traditional, local computing methods because of its ability to scale and its capacity to store large quantities of data. Cloud computing accelerates scientific workflows, discoveries, and collaborations by reducing research and data friction. We aim to improve “time to science” with the NSF-funded Jetstream cloud. We describe a Unidata science gateway on Jetstream. With the aid of open-source cloud computing projects such as OpenStack, Docker, and JupyterHub, we deploy a variety of scientific computing resources on Jetstream for our scientific community. These systems can be leveraged with data-proximate Jupyter notebooks, and remote visualization clients such as the Unidata Integrated Data Viewer (IDV) and AWIPS CAVE. This gateway will enable students and scientists to spend less time managing their software and more time doing science.
 
  
More info: https://science-gateway.unidata.ucar.edu/
+
Within this, what makes a great data product?
  
Slides: https://doi.org/10.6084/m9.figshare.12124065.v1
+
"Our data model is the Web"
  
'''About the presenter:'''
+
People will deal with messy data if it's super valuable.
I am a scientific software developer for the Unidata Program Center at UCAR (University Corporation for Atmospheric Research) in Boulder, Colorado. I have been employed at UCAR  since 1999 and at Unidata since 2010. I obtained a bachelor's degree in molecular, cellular and developmental biology in 1994 and a master's degree in computer science in 2000. I am passionate about the application of computing technology to science and math. During my employment at Unidata I have advocated for open-source, cloud computing and Python related technologies. I began at Unidata as a software developer supporting the Integrated Data Viewer (IDV). More recently, I have been focused on Unidata science gateway efforts with the objective of facilitating science for the Unidata community with web technologies.
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/hdGu8XPW6Rg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
Case in point, IRS 990 data on non-profits was shared in a TON of xml schemas. People came together to organize it and work with it.
  
==12 March 2020: "DGGS in action: provision of rapid response during Australian bushfires and other applications" Shane Crossman and Irina Bastrakova==
+
Story about a building footprint data released in the morning – had been matched up into at least four products by the end of the day.
  
'''Summary'''
+
Shout out to:
Everything has a location. Location can be defined using descriptive terms (e.g. place names), geometry (e.g. geographic coordinates) and/or index notations (e.g. statistical boundaries). However, existing approaches and disconnected infrastructures limit our ability to discover, access and integrate spatial data across organisation and jurisdiction boundaries to produce up-to-date reliable information. The Location Index project (LOC-I) aims to introduce a consistent way to access, analyse and use location data to support the effective integration of socio-economic, statistical and environmental data from multiple data providers to support the spatially enabled delivery of Government policies and initiatives.
+
https://www.geoffmulgan.com/ and https://jscaseddon.co/
 
The devastation caused by the Australian Bushfires highlighted the need for a new approach for rapid data integration. The total burnt area during Autumn-Summer 2019-2020 is 72,000 square miles, which is an equivalent to a half of Montana or North Dakota and Delaware areas combined. Rapid response in provision of information on areas affected by the bushfires was required to support evaluation of the impact, and also planning the recovery process and support for families, businesses and the environment. This presentation will discuss application of the Discrete Global Grid System (DGGS) in bringing together diverse complex information from multiple sources to support the response process. The presentation will also discuss testing of the DGGS capability in other use cases.
 
  
Slides here: https://doi.org/10.6084/m9.figshare.12032592.v1
+
https://jscaseddon.co/2024/02/science-for-steering-vs-for-decision-making/  
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/AF_9JBuPdHI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
"We don't have institutions that are tasked with producing great data products and making them available to the world!"
  
==13 February 2020: "Urban Flooding Open Knowledge Network": Mike Johnson==
+
https://radiant.earth/blog/2023/05/we-dont-talk-about-open-data/
 +
[[File:Meme hackathons.png|thumb]]
  
We’re back this month with a webinar on the Urban Flooding Open Knowledge Network project from Mike Johnson, a graduate student from UC Santa Barbara. This is an exciting stakeholder-driven knowledge network project with emphasis on prototyping interfaces and web resources, some of which Mike will demonstrate for us.
+
"There's a server somewhere where there's some stuff" – This is very different from a local hard drive where everything is indexed.  
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/HuJBKp0yCOo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
A cloud native approach puts the index (metadata) up front in a way that you can figure out what you need.
  
==12 December 2019: "Location Integration Project -- DGGS and Linked Data": Matthew Purss and Shane Crossman (postponed)==
+
A file's metadata gives you the information you need to ask for just the part of a file that you actually need.
  
'''Summary'''
+
But there are other files where you don't need to do range requests. Instead, the file is broken up into many many objects that are indexed.
Following presentations on DGGS and similar technologies for indexing, this presentation will explore use cases and technical implementations for using DGGS and linked data together in the Loc-I project.
 
  
==14 November 2019: "Location Integration Project": Matthew Purss and Shane Crossman==
+
In both cases, the metadata is a map to the content. Figuring out the right size of the content's bits is kind of an art form.
  
'''Summary'''
+
https://www.goodreads.com/en/book/show/172366
Everything has a location. Location can be defined using descriptive terms (e.g. place names, suburb names, and rivers), position and geometry (e.g. geographic coordinates like latitude and longitude) and/or index notations (e.g. mesh block or parcel identifiers). A street address is useful for service delivery; land parcels for revenue raising and investment; coordinates and grids for positioning and monitoring changes in the landscape; various administrative boundaries for law enforcement and service management; and statistical (e.g. health, society or economy) information to analyse and improve life in Australian. However, existing approaches and disconnected infrastructures coupled with the myriad of ways to describe and store location information limit our ability to discover, access and integrate spatial data across organization and jurisdiction boundaries to produce reliable and actionable information. The Location Index project (LOC-I) aims to introduce a consistent way to access, analyse and use location data to support the effective integration of socio-economic, statistical and environmental data from multiple data providers to support the spatially enabled delivery of Government policies and initiatives.
 
 
The flexible, and standards based, spatial data infrastructures being developed by Geoscience Australia and its partners under LOC-I include Internet of Things (IoT), Linked Data and Discrete Global Grid Systems (DGGS) technologies integrated with cloud-based data discovery and access tools. This will enable LOC-I to democratize spatial data applications, where people will be able to do spatial data integration operations without needing GIS specialists. This presentation will provide an overview of LOC-I, DGGS and how DGGS technologies can be used as a tool to spatially enable and integrate socio-economic and geospatial data together.
 
  
http://locationindex.org
+
Q: > I was thinking of your example of Warren Buffett's daily spreadsheet (gedanken experiment)... How do you see data quality or data importance (incl. data provider trustworthiness) being effectively conveyed to users?
  
[http://wiki.esipfed.org/images/f/fb/Loc-I_webinar_-_15Nov2019.pptx| Download the slides here.]
+
A: We want to focus on verification of who people are and relying on reputational considerations to establish importance.  
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/fRSYsuNP1X0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
Q: > I agree with you about the importance of social factors in how people make decisions. What do you think the implications are of this for metadata for open data on the cloud?
  
==10 October 2019: "ELFIE: Environmental Linked Features Interoperability Experiment":==
+
A: Tracking data's impact and use is an important thing to keep track of. Using metadata as concrete records of observations and how it has been used is where this becomes important.
  
'''Summary'''
+
Q: > What about the really important kernels of information that we use to, say calibrate remote sensing products, that are really small but super important? How do we make sure those don't get drowned?
This webinar will provide an update on activities of the OGC Environmental Linked Features Interoperability Experiment (ELFIE) and a preview of activities of the Second ELFIE (SELFIE). This will be an interactive demonstration-based session focused on practical application of JSON-LD to link features and observations.  
+
A: We need to be careful not to overemphasize "everything is open" if we can't keep really important datasets in the spotlight.  
  
Presenter: [https://www.usgs.gov/staff-profiles/david-l-blodgett David Blodgett, Civil Engineer, U.S. Geological Survey Water Mission Area]
+
==January 11th: "Using Earth Observations for Sustainable Development"==
  
[https://opengeospatial.github.io/ELFIE/ More info on ELFIE here.]
+
"Using Earth Observation Technologies when Assessing Environmental, Social, Policy and Technical factors to Support Sustainable Development in Developing Countries"
  
'''Recording'''
+
[https://www.media.mit.edu/people/shariful/overview/ Sharif Islam]
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/lvy9vdv96-s" frameborder="0" allowfullscreen></iframe></html>
+
Earth Observation (EO) technologies, such as satellites and remote sensing, provide a comprehensive view of the Earth's surface, enabling real-time monitoring and data acquisition. Within the environmental domain, EO facilitates tracking land use changes, deforestation, and biodiversity,   thereby   supporting   evidence-based   conservation   efforts.   Social   factors, encompassing population dynamics and urbanization trends, can be analyzed to inform inclusive and resilient development strategies. EO also assumes a crucial role in policy formulation by furnishing accurate and up-to-date information on environmental conditions, thereby supporting informed decision-making. Furthermore, technical aspects, like infrastructure development and resource management, benefit from EO's ability to provide detailed insights into terrain characteristics and natural resource distribution. The integration of Earth Observation across these domains yields a comprehensive understanding of the intricate interplay between environmental, social, policy, and technical factors, fostering a more sustainable and informed approach to development initiatives. In this presentation, I will discuss our lab's work in Bangladesh, Angola, and other countries, covering topics such as coastal erosion, drought, and air pollution.
  
==12 September 2019: "STARE: SpatioTemporal Adaptive Resolution Encoding for Scalable Integrative Analysis": Michael Rilee==
+
'''<u>Recording</u>''':<br />
 +
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/PhEg9bTd1JU?si=EUfOaz3nzEFdOOsb" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
'''Summary'''
+
'''<u>Minutes:</u>'''
Title: STARE: SpatioTemporal Adaptive Resolution Encoding for Scalable Integrative Analysis
 
  
Abstract: The Earth Science enterprise has generated a great volume and variety of high-quality, high-spatiotemporal resolution data (e.g. level 1 and 2 swath data), that could be integrated to open wide arenas of new, improved scientific advances. Yet aligning and integrating different kinds of Earth Science data is a laborious process, leading most researchers to focus on more generic, high level data products that are more easily compared. Dealing with the great volume and variety is the goal of the NASA/ACCESS-17 STARE project. STARE is a unifying indexing scheme addressing variety and is well suited for applying distributed storage and computing resources to address volume. We will describe the STARE approach and how it relates to other geolocation schemes, e.g. DGGS or simulation grids.
+
Plan to share data from NASA and USGS that was used in his PHD work.
  
Presenter: Michael Rilee, Ph.D., Rilee Systems Technologies LLC.
+
Applied the EVDT Environment, Vulnerability, Decision Technology Framework.
  
Bio: Michael Rilee, PI of the STARE project, has been involved with high-end computing and modeling and simulation for about 25 years. For about the past 5 years or so he has been researching advanced computing techniques for a range of Earth Science applications including parallel computing, array databases, regridding, kriging, and atmospheric chemistry. Before that he was involved with several NASA efforts involving spacecraft autonomy, on-board data processing, and high-end computing. He was initially trained in plasma astrophysics, receiving his Ph.D. from Cornell University.
+
Studied a variety of hazards – coastal erosion, air pollution, drought, deforestation, etc.
  
[https://esip.figshare.com/articles/STARE_Spatiotemporal_Adaptive_Resolution_Encoding_for_Integrative_Compatibility_Across_Variety_of_Volume/8942291 See poster about STARE here.]
+
'''Coastal Erosion in Bangladesh:'''
[http://odl.unl.edu/index.php/map_interface_user_guide/ See demo here.]
 
[https://www.researchgate.net/profile/Kwo-Sen_Kuo See related research here.]
 
  
'''Recording'''
+
* Displacement, loss of land, major economic drain
 +
* Studied the situation in the Bay of Bengal
 +
* Used LANDSAT to study coastal erosion from the 80s to the present
 +
* Coastal erosion rates upwards of 300m/yr!
 +
* Combined survey data and landsat observations
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/uodeCiYc6Mg" frameborder="0" allowfullscreen></iframe></html>
+
'''Air Pollution and mortality in South Asia'''
  
==8 August 2019: "The Challenge of Location and How Discrete Global Grid Systems can enable Spatial Data Integration.": Matthew Purss==
+
* Able to show change in air pollution over time using remote sensing
  
'''Summary'''
+
'''Drought in Angola and Brazil'''
Everything has a location. Location can be defined using descriptive terms (e.g. place names, suburb names, and rivers), position and geometry (e.g. geographic coordinates like latitude and longitude) and/or index notations (e.g. mesh block or parcel identifiers). A street address is useful for service delivery; land parcels for revenue raising and investment; coordinates and grids for positioning and monitoring changes in the landscape; various administrative boundaries for law enforcement and service management; and statistical (e.g. health, society or economy) information to analyse and improve life in Australian. However, existing approaches and disconnected infrastructures coupled with the myriad of ways to describe and store location information limit our ability to discover, access and integrate spatial data across organisation and jurisdiction boundaries to produce reliable and actionable information. The Location Index project (LOC-I) aims to introduce a consistent way to access, analyse and use location data to support the effective integration of socio-economic, statistical and environmental data from multiple data providers to support the spatially enabled delivery of Government policies and initiatives.
 
 
The flexible, and standards based, spatial data infrastructures being developed by Geoscience Australia and its partners under LOC-I include Internet of Things (IoT), Linked Data and Discrete Global Grid Systems (DGGS) technologies integrated with cloud-based data discovery and access tools. This will enable LOC-I to democratise spatial data applications, where people will be able to do spatial data integration operations without needing GIS specialists. This presentation will provide an overview of LOC-I, DGGS and how DGGS technologies can be used as a tool to spatially enable and integrate socio-economic and geospatial data together.
 
  
Dr. Matthew Brian John Purss is a Senior Advisor on Geospatial Standards at Geoscience Australia. He is a founding co-chair of the Open Geospatial Consortium's (OGC) Discrete Global Grid Systems (DGGS) Domain and Standards Working Groups. He is a geophysicist with over 20 years' experience in the exploration, research and government sectors and holds a PhD in Exploration Geophysics from Monash University where he studied grid based approaches to magnetic and electromagnetic modelling applications. Prior to initiating and leading the international standardisation of DGGS infrastructures.
+
Used SMAP (Soil Moisture Active Passive)  
  
http://locationindex.org
+
Developed the same index as the US Drought Monitor
  
[[http://wiki.esipfed.org/images/c/c4/ESIP_DGGS_Webinar_presentation_2019_08_08.pptx|Download the slides here.]]
+
Able to apply SMAP observations over time
  
'''Recording'''
+
Applied a social vulnerability model using these data to identify vulnerable populations.
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/Gbe6-N_Zisw" frameborder="0" allowfullscreen></iframe></html>
+
'''Deforestation in Ghana'''
  
==11 July 2019: "ADIwg Open Source Metadata Toolkit": Josh Bradley, Dennis Walworth==
+
Used LANDSAT to identify land converted from forest to mining and urban.
  
'''Summary'''
+
Significant amounts of land to mining (gold mining and others)
Josh Bradley is a data manager for Fish & Wildlife Service, Science Data Applications and project manager for the ADIwg mdTools project.
 
Dennis Walworth is the data manager for the USGS Alaska Science Center and collaborator on the ADIwg mdTools project.
 
  
*[https://docs.google.com/presentation/d/e/2PACX-1vTAXbGlKL8QZ4G8pSHDOoeOLrfH3xWHNmspLeaMAocJI2zl0fKn5g-x7PAirekLAi3vImIL6752T0k7/pub?start=false&loop=false&delayms=3000&slide=id.p1 See slides here.]
+
'''Water hyacinth in a major fishery lake in Benin.'''
*mdEditor website: https://www.mdeditor.org/
 
*Development version: https://dev.mdeditor.org/
 
*mdEditor User Manual: https://adiwg.gitbooks.io/mdeditor/content/
 
*GitHub Issues: https://github.com/adiwg/mdEditor/issues
 
  
'''Recording'''
+
Impact on fishery and transportation
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/ClRmCU2Ae5c" frameborder="0" allowfullscreen></iframe></html>
+
Rotting hyacinth is a big issue
  
==6 June 2019: "Google Dataset Search: Facilitating data discovery in an open ecosystem.": Chris Gorgolewski==
+
Helped develop a DSS to guide management practices
  
'''Summary'''
+
'''Mangrove loss in Brazil'''  
  
There are thousands of data repositories on the Web, providing access to millions of datasets. National and regional governments, scientific publishers and consortia, commercial data providers, and others publish data for fields ranging from social science to life science to high-energy physics to climate science and more. Access to this data is critical to facilitating reproducibility of research results, enabling scientists to build on others’ work, and providing data journalists easier access to information and its provenance. In this talk, I will discuss recently launched Google Dataset Search, which provides search capabilities over potentially all dataset repositories on the Web. I will talk about the open ecosystem for describing and citing datasets that we hope to encourage and the technical details on how we went about building Dataset Search. Finally, I will highlight research challenges in building a vibrant, heterogeneous, and open ecosystem where data becomes a first-class citizen.
+
Combined information from economic impacts, urban plans, and remote sensing to help build a decision support tool.  
  
https://toolbox.google.com/datasetsearch
+
==November 9th: "Persistent Unique Well Identifiers: Why does California need well IDs?"==
https://www.blog.google/products/search/making-it-easier-discover-datasets/
 
  
'''Recording'''
+
[[File:ITI_Nov_Wells.png|thumb|IT&I CA Wells November 2023]]
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/1PSE2hQ7mBo?start=3" frameborder="0" allowfullscreen></iframe></html>
+
[https://cawaterdata.org/teams/hannah-ake/ Hannah Ake]
  
==9 May 2019: "The SpatioTemporal Asset Catalog (STAC) specification": Chris Holmes==
+
Groundwater is a critical resource for farms, urban and rural communities, and ecosystems in California, supplying approximately 40 percent of California's total water supply in average water years, and in some regions of the state, up to 60 percent in dry years. Regardless of water year type – some communities rely entirely on groundwater for drinking water supplies year-round. However, California lacks a uniform well identification system, which has real impacts on those who manage and depend upon groundwater. Clearly identifying wells, both existing and newly constructed, is vital to maintaining a statewide well inventory that can be more easily monitored to ensure the wellbeing of people, the environment, and the economy, while supporting the sustainable use of groundwater. A uniform well ID program has not yet been accomplished at a scale like California, but it is achievable, as evidenced by great successes in other states. Learn more about why a well ID program will be so important to tackle in California and offer your thoughts about how to untangle some of the particularly thorny technical challenges.
  
'''Summary'''
+
'''<u>Recording</u>''':<br />
 +
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/dvxOHh86QVQ?si=GtgSG62nbj2aVMR0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
The SpatioTemporal Asset Catalog (STAC) specification is an emerging standard to make it easier to find geospatial information. It aims to enable a cloud-native geospatial future by providing a common layer of metadata for search and discovery, while playing well with the web and existing geospatial standards.
+
'''<u>Minutes:</u>'''
  
See the the [http://stacspec.org/ SpatioTemporal Asset Catalog (STAC) specification] web site here.
+
*Groundwater is 40-60% of California's Water supply
 +
*~2 Million groundwater wells!
 +
*As many as 15k new wells are constructed each year
 +
Sustainable groundwater management act frames groundwater sustainability agencies that develop groundwater sustainability plans
  
'''Time''': Thursday, 9, May, 2019 (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
There is a need to account for groundwater use to ensure the plans are achieved.
  
'''Recording'''
+
Problem: There is no dedicated funding (or central coordinator) to create and maintain a statewide well inventory.
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/emXgkNutUTo" frameborder="0" allowfullscreen></iframe></html>
+
  
 +
*Department of Water Resources develops standards
 +
*State Water Resources Control Board has statewide ordinance
 +
*Cities and local districts adopt local ordinance
 +
* Local enforcement agency administers and enforces ordinance
  
==11 Apr 2019: "Pachyderm": John Karabaic==
+
There are a lot of IDs in use. 5 different identifiers can be used for the same well.
  
'''Summary'''
+
Solution: Create a well inventory that is statewide but is a compound (single id that stands in for many others) id from multiple id systems. – A meaningless identifier that links multiple others to each other.
  
See the Pachyderm one pager here: https://cdn2.hubspot.net/hubfs/4751021/Business_One_Pager.pdf
+
There are a number of states with well id programs.  
  
"Pachyderm lets you deploy and manage multi-stage, language-agnostic data pipelines while maintaining complete reproducibility and provenance."
+
*Trying to learn from what other states have done.
  
'''Time''': Thursday, 11, April, 2019 (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
Going forward with some kind of identifier system that spans all local and federal identifier systems.
  
'''Join meeting''':  
+
*Q: Will this include federal wells? – Yes!
 +
*Q: Will this actually be a new well identifier minted by someone? – Yes.
 +
* Q: If someone drills a well do they have to register it? – Yes, but it's the local enforcing agency that collects the information.
 +
*Q: What if a well is deepened? Do we update the ID? – This has caused real problems in the past. We end up with multiple IDs for the same hole that go through time.
 +
**Seems to make sense to make a new one to keep things simple.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
Link mentioned early in the talk:
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker''': Michael Masters (Pachyderm)
+
https://groundwateraccounting.org/
See Docs Here:
 
http://docs.pachyderm.io/en/latest/getting_started/local_installation.html 
 
  
Also see intro talk here: https://changelog.com/practicalai/23
+
Reference during Q&A
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/za0JNgL4JGA" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></html>
+
https://docs.ogc.org/per/20-067.html#_cerdi_vvg_selfie_demonstration
  
==14 Mar 2019: "Integrating SciServer and OceanSpy to enable easy access to oceanographic model output": Mattia Almansi (Johns Hopkins University)==
+
==October 26th: "Improving standards and documentation publishing methods: Why can’t we cross the finish line?"==
  
'''Time''': Thursday, 14, March, 2019 (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
[[File:ITI_Oct_ogc.png|thumb|IT&I OGC October 2023]]
  
'''Summary''': OceanSpy is an open-source and user-friendly Python package that enables scientists and interested amateurs to use ocean model data sets with out-of-the-box analysis tools. OceanSpy builds on software packages developed by the Pangeo community (in particular xarray, dask, and xgcm). I will show how OceanSpy can be used on model outputs stored on the Johns Hopkins SciServer system, negating the need for the user to own a computing cluster or even download the data. OceanSpy accelerates and facilitates exploration (including visualization) of terascale data. It is designed to operate on petascale ocean simulations hosted by SciServer in the coming months.
+
[https://www.ogc.org/about/team/scott-simmons/ Scott Simmons]
  
'''Join meeting''':
+
OGC and the rest of the Standards community have been promising for YEARS that our Standards and supporting documentation will be more friendly to the users that need this material the most. Progress has been made on many fronts, but why are we still not finished with a promise made in 2015 that all OGC Standards will be available in implementer-friendly views first, ugly virtual printed paper second?This topic bugs me as much as it bugs our whole community. Some of the problems are institutional (often from our Government members across the globe), others are due to lack of resources, but I think that most are due to a lack of clear reward to motivate people to do things differently.Major progress is being made in some areas. The OGC APIs have landing pages that include focused and relevant content for users/implementers and it takes some effort to find the owning Standard. OGC Developer Resources are growing quickly with sample code, running examples, and multiple views of API resources in OpenAPI, Swagger, and ReDoc.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
[https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fportal.ogc.org%2Ffiles%2F%3Fartifact_id%3D106445&data=05%7C01%7Cdblodgett%40usgs.gov%7Cfb8c43b89a0e45fbb4ff08dbd56a8577%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C638338425710712359%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Z1GilUJ3gQaBmwi7QPaz3%2Fa2ycaqRcsJlNfoYN4aG0U%3D&reserved=0 Slides]
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker''': Mattia is a PhD student at Johns Hopkins University under Prof. Tom Haine.  He's a physical oceanographer, primarily interested in the dynamics governing the circulation of the Subpolar North Atlantic. To address the targets of his research he uses high-resolution numerical simulations and available observations.
 
  
===Links===
+
'''<u>Recording</u>''':<br />
 +
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/HJ7TbhcVs-U?si=hcUmpoIaTz4zNodo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
*[https://github.com/malmans2/oceanspy OceanSpy Github Page]
+
'''<u>Minutes:</u>'''
*[http://www.sciserver.org/ SciServer web page]
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/GO_hNIcFw8o" frameborder="0" allowfullscreen></iframe></html>
+
(missing first ~15 minutes of recording -- apologies)
  
==14 Feb 2019: "Cloud Native Geoprocessing of Earth Observation Satellite Data with Pangeo": Scott Henderson (University of Washington)==
+
Circa 2015 OGC GeoRabble
  
'''Summary''': NASA has estimated that by 2025, it will be storing upwards of 250 Petabytes (PB) of its data using commercial Cloud services (e.g. Amazon Web Services [AWS]). This presentation will focus on efforts funded by a NASA ACCESS 2017 grant to transition the Earth Science community into Cloud computing by developing technologies that build on top of the growing Pangeo ecosystem. In particular, the integration of JupyterHub with Kubernetes and several high-level Python packages (i.e. Xarray, Dask, Rasterio, Intake, PyViz), are enabling Cloud-native workflows that circumvent the bottleneck of downloading large amounts of data. These tools work best with emerging Cloud-native storage solutions for satellite imagery (i.e. NASA’s CMR, STAC, COGs). In this presentation, Scott gives update on the Pangeo project and showcases a few example workflows using large public archives of optical and radar satellite data.
+
*Took a critical look at the status of publishing standards.
 +
*Couldn't we format these specs in a kind of tutorial form?
  
'''Speaker''': Scott Henderson has a PhD in Geological Sciences from Cornell University and is currently a postdoctoral fellow at the University of Washington eScience Institute. Scott studies geologic hazards with satellite-based synthetic aperture radar.
+
*Lots of snippets and tutorial content in the specs.
 +
** E.g. http://opengeospatial.github.io/e-learning/index.html
 +
*Multiple representations of specifications – that OGC staff could maintain
  
===Links===
+
9 years later
  
*[https://docs.google.com/presentation/d/1evNXCddIllXUt4a5jKfmlO2197sp8T6I9L650cYZcsk/edit?usp=sharing Google Slides]
+
*What makes this hard?
*[https://github.com/scottyhq/esip-tech-dive Landsat NDVI (data on AWS, compute on Google)]
+
**Standards must be unambiguous AND procurable.
*[https://github.com/pangeo-data/pangeo-tutorial-agu-2018 AGU 2018 Tutorial material (various examples)]
+
** The modular specification is a model for this balance.
*[https://github.com/scottyhq/grfn_pangeo_demo Getting ready for NISAR data]
 
*[https://github.com/scottyhq/stac-intake-landsat STAC catalogs, Intake, mosaics]
 
*[https://medium.com/pangeo/cloud-native-geoprocessing-of-earth-observation-satellite-data-with-pangeo-997692d91ca2 blog post for context]
 
  
===Recording===
+
Standards are based around testable requirements that relate to conformance classes.
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/yQon6Wh-lN4" frameborder="0" allowfullscreen></iframe></html>
 
  
==[CANCELLED DUE TO US GOV SHUTDOWN] 10 Jan 2019==
+
Swaggerhub and ReDoc as a way to show a richer collection of information for multiple users.
  
==13 Dec 2018: "Developing JupyterLab Extensions": Ian Rose (Berkeley)==
+
Specification are much more modular (core and extensions)
  
'''Summary''': JupyterLab extensions can customize or enhance any part of JupyterLab. They can provide new themes, file viewers and editors, or renderers for rich outputs in notebooks. Extensions can add items to the menu or command palette, keyboard shortcuts, or settings in the settings system. Extensions can provide an API for other extensions to use and can depend on other extensions. In fact, the whole of JupyterLab itself is simply a collection of extensions that are no more powerful or privileged than any custom extension. In this talk Ian will demonstrate how to build a JuptyerLab extension.  
+
Developer website: https://developer.ogc.org/
  
'''Time''': Thursday, 13 Dec, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)  
+
Going to be including persistent  demonstrator (example implementations) that are "in the wild".
  
'''Join meeting''':  
+
https://www.ogc.org/initiatives/open-science/
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
Moving to an "OGC Building Blocks" model that are registered across multiple platforms and linked to lots of examples.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker''': Ian Rose is a postdoctoral fellow at the Berkeley Institute for Data Science, where he is a core developer on Project Jupyter, working on JupyterLab.  He has a PhD in geophysics from Berkeley.
+
Building blocks are richly described and nuanced but linked back to specific requirements in a specification.
  
'''Links''':
+
https://blocks.ogc.org/
  
*https://jupyterlab.readthedocs.io/en/stable/
+
https://sn80uo0zmbg.typeform.com/to/gcwDDNB6?typeform-source=blocks.ogc.org
*https://jupyterlab.readthedocs.io/en/stable/user/extensions.html
 
*https://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html
 
  
===Recording===
+
A lot of this focused on APIs – what about data models?
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/cUEutudPmAo" frameborder="0" allowfullscreen></iframe></html>
 
  
 +
*Worked on APIs first because it was current. Also thinking about how to apply similar concepts to data models.
  
==8 November 2018: "Intake: Lightweight tools for loading and sharing data in data science projects": Martin Durant (Anaconda)==
+
==September 14th: "Water data standardization: Navigating the AntiCommons" ==
  
'''Summary''': Intake is a set of free open-source Python tools that help load data from a variety of formats into familiar containers like Pandas dataframes, Xarray datasets, and more.  Boilerplate data loading code can be transformed into reusable Intake plugins.  Datasets can be described for easy reuse and sharing using Intake catalog files.  Martin will give an overview of Intake and demonstrate use via Jupyter Notebooks.  
+
[[File:ITI_Sept_IoW_Kyle-Onda.png|thumb|IT&I IoW September 2023]]
  
'''Time''': Thursday, 8 November, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
[https://internetofwater.org/about/people/kyle-onda/ Kyle Onda]
  
'''Join meeting''':
+
We all know interoperability rests on data standards and API standards. Many open standards are less prominent in the open water data space than proprietary solutions. This is because proprietary data management solutions are often bundled with very easy to use implementing software and more importantly—client software that address basic use cases. We’re giving people blueprints when they need houses. Community standards making processes should invest in end-user tools if they want to gain traction. The good news is that some of the newest generation of standards is much easier to develop around which has led to some reference implementations that are much easier to create end-user tools around than previously.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
'''<u>Recording</u>''':<br />
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
+
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/miFwXB-E1V8?si=Xr2SF_okCxLv2lL2" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
'''Speaker''': Martin Durant is a data scientist and software developer at Anaconda who specializes in Python data engineering tools, instruction and solutions.  He also has a PhD in Astrophysics.
+
'''<u>Minutes:</u>'''
  
'''Links''':
+
AntiCommons – name comes from social science background
  
*https://intake.readthedocs.io
+
Tragedy of the commons - two solutions, enclose (privatize) or regulate
*https://github.com/ContinuumIO/intake
 
  
===Recording===
+
Tragedy of the anticommons - as opposed to common resources, these are resources that don't get used up – as in open data. Inefficiency and under utilization is common.
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/PSD7r3JFml0" frameborder="0" allowfullscreen></iframe></html>
 
  
 +
Two solutions. expropriation (like imminent domain or public data), incentivize
  
==11 October 2018: "SpatioTemporal Feature Registry: ESIP idea campaign and working example in USGS": Sky Bristol (USGS)==
+
Example – consolidate urban sprawl into higher density housing to get more open space and room for business.
  
'''Summary''': Building a National Biogeographic Map, an analysis platform for exploring biodiversity conservation measures and stressors, sparked the need for a reliable data source with a wide variety of identified places. These need to be assembled in a sustainable and robust way that keeps track of provenance and processing steps so we can build reports for decisionmakers that are trustworthy and consistent. We started an idea campaign in the ESIP Lab with some questions we have about how best to do this work. This talk will share what we have in place so far, including a registry of sources, a data processing pipeline, an integrated index, a REST API, and a working web application. Technologies include USGS ScienceBase, PostgreSQL/PostGIS, ElasticSearch, Python Flask, and other Python processing codes using GDAL and other libraries.
+
Introducing the Internet of Water.  
 +
 +
Noting that in PNW, there are >800 USGS stream gages and >400 from other organizations. Only USGS are very broadly known about.
  
'''Time''': Thursday, 11 October, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
Thinking about open data as an anticommons – environmental data is normally publically available but only in ways that are convenient to data providers and the software that they use.
  
'''Join meeting''':
+
Discussion of the variety of standardized vs bespoke modes of data dissemination.
No live meeting, as something came up at the last minute and Sky can't make it.  He did record a 30 minute presentation, however.  See "Recording" below.  See also the sample Jupyter notebooks in the "Links" section below.
 
  
'''Speaker''': Sky Bristol is the branch chief for Biogeographic Characterization in the USGS Core Science Analytics, Synthesis, and Library Program. Unless he can be skiing bumps, he likes doing cool things to help people make better decisions using data.
+
Example of Nebraska – GUI with download and separate custom API
 +
USGS has the same basic scheme where an ETL goes from data management software to a custom web service system.
  
'''Links''':
+
What's going on here? Limited resources lead to focus on existing users and needs and administration ease.
https://github.com/skybristol/notebooks/blob/master/SFR%20Exploration.ipynb
 
  
 +
Tools that meet this need tend to not focus on the needs of new user and standardization.
  
===Recording===
+
Most organizations don't need standards – they need software. Both server and CLIENT software.  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/J7T59-H_W4o" frameborder="0" allowfullscreen></iframe></html>
 
  
==9 August 2018: "EarthSim: Flexible Environmental Simulation Workflows Entirely Within Jupyter Notebooks": Dharhas Pothina (ERDC)==
+
New specs and efforts ARE heading in this direction.
 +
OGC-API, SensorThings, etc.
  
'''Summary''': Building environmental simulation workflows is typically a slow process involving multiple  proprietary desktop tools that do not interoperate well. In this work, we demonstrate building flexible, lightweight workflows entirely in Jupyter notebooks.  The goal is to provide a set of tools that can easily be reconfigured and repurposed as needed to rapidly solve specific emerging issues. As part of this work, extensive improvements were made to several general-purpose open source packages, including support for annotating and editing plots and maps in Bokeh and HoloViews, rendering large triangular meshes and regridding large raster data in HoloViews, GeoViews, and Datashader, and widget libraries for Param.  
+
Promising developments around proxying non standard APIs and in use of structured data "decoration" to make documentation more standard.
  
'''Time''': Thursday, August 9, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
==August 10th: "Learning to love the upside down: Quarto and the two data science worlds"==
  
'''Join meeting''':  
+
[[File:ITI_August_Quarto.png|thumb|IT&I Quarto August 10th]]
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
[https://cscheid.net/v2/ Carlos Scheidegger]
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker''':  
+
There are two wonderful data science worlds. You can be a jupyter expert: you work on jupyter notebooks, with access to myriad Julia, Python, and R packages, and excellent technical documentation systems. You can also be a knitr and rmarkdown expert: you work on rmarkdown notebooks, with access to myriad Julia, Python, and R packages, and excellent technical documentation systems.
Dharhas Pothina is the Associate Technical Director of the Information Technology Laboratory, US Army Engineer Research and Development Center, in Vicksburg, MS. He was formerly the Water Informatics Lead at the Texas Water Development Board. He holds a Ph.D in Civil Engineering from the University of Texas at Austin.
 
  
'''Links''':
+
But what if your colleague works on the wrong side of the fence? What if you spent years learning one of them, only to find that the job you love is in an organization that uses the other? In this talk, I’m going to tell you about quarto, a system for technical communication (software documentation, academic papers, websites, etc) that aspires to let you choose any of these worlds.
https://pyviz.github.io/EarthSim/
 
  
===Recording===
+
If you’re one to worry about Conway’s law and what this two-worlds situation does to an organization’s talent pool, or if you live in one side of the world and want to be able to collaborate with folks on the other side, I think you’ll find something of value in what I have to say.
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/etf8M_uW39E" frameborder="0" allowfullscreen></iframe></html>
 
  
 +
I’m also going to complain about software, mostly the one I write. Mostly.
  
==14 June 2018: "Analysis of Massive Underwater Video Data in the Cloud using Pangeo": Tim Crone (Lamont)==
+
Slides: https://cscheid.net/static/2023-esip-quarto-talk/
  
'''Summary''': An open-source environment for parallel analysis of massive (100TB) image data in the Cloud is now available via the Pangeo environment, which allows you to apply the power of the Python ecosystem from your browser.  Technologies include JupyterHub, Kubernetes, Docker, and Dask distributed. 
+
'''<u>Recording</u>''':<br />
 +
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/uQ3yZjM1bj8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
'''Time''': Thursday, June 14, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
'''<u>Minutes:</u>'''
  
'''Join meeting''':
+
Carlos was in a tenure computer science position at University of Arizona.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
Hating bad software makes a software developer a good developer.  
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker(s)''':  
+
Two data science worlds:  
Tim Crone is a marine geophysicist at Lamont-Doherty Earth Observatory studying spatial variations in the tidal triggering of microearthquakes within ridge systems, and problems in acoustics associated with high-temperature hydrothermal vents and seafloor seismic networks.  He has recently deployed the Pangeo framework on the Microsoft Azure Cloud.
 
  
'''Links''':
+
tidyverse (with R and markdown)
  
*https://pangeo-data.github.io
+
*Cohesive, hard to run things out of order.
 +
*Doesn't store output.
  
===Recording===
+
Jupyter (python and notebooks)
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/_6eeymc4c7g" frameborder="0" allowfullscreen></iframe></html>
 
  
 +
*Notebook saves intermediate outputs.
 +
*State can be messed up easily – cells aren't linear steps.
  
 +
Quarto:
  
 +
*Acts as a compatibility layer for tidyverse and jupyter ecosystems.
 +
*Emulates RMarkdown with multi language support.
  
==10 May 2018: "NetCDF-CF Advances - Simple Geometries, Swaths, and Groups": Dave Blodgett (USGS), Tim Whiteaker (UT Austin), Aleksander Jelanek (HDF Group) and Daniel Lee (EUMETSAT)==
+
Rant:
  
'''Summary''': Simple geometry (points, lines, and polygons) has now been accepted as part of the Open Geospatial Consortium’s NetCDF-CF specification. This a major enhancement to a widely used standard whose utility has previously been limited to time-series of point or (raster) coverage data only. Advances on Groups and Swaths will also be presented.  
+
Quarto gets you a webpage and PDF output.  
  
'''Time''': Thursday, May 10, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
+
– note that the PDF requirement is not great.
  
'''Join meeting''':
+
Quarto is kind of just a huge wrapper around pandoc.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
Quarto documentation is intractably hard to build out.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker(s)''':
+
Consider Conways Law – that an organization that creates a large system will create a system that is a copy of the organization's communication structure.
Dave Blodgett is a project coordinator with the USGS Office of Water, Geointelligence Branch. He holds a B.S. in civil and environmental engineering and an M.S. in water resources engineering form the UW-Madison. Tim Whiteaker is a research scientist at the University of Texas at Austin. Tim develops innovative cyberinfrastructure for solving water resource engineering challenges. Aleksander Jelanek is a Senior Informatics Architect with the HDF Group, and Daniel Lee is a Software Engineer at EUMETSAT.
 
  
'''Links''':
+
– Quarto is meant to allow whole organizations with different technical tools exist in the same communication structure (same system).
 
+
   
*https://github.com/cf-convention/cf-conventions/pull/115
+
Quarto tries to make kinda hard things easy while not making really hard things impossible.
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/79e3GC74y_w" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
==12 April, 2018: "Jetstream: A free national science and engineering cloud environment on XSEDE": Jeremy Fischer, Indiana University==
 
 
 
'''Summary''': Jetstream, a national science and engineering cloud, adds cloud-based, on-demand computing and data analysis resources to the national XSEDE cyberinfrastructure.  A description of Jetstream current and planned capabilities, and how to gain access, will be presented.
 
 
 
'''Time''': Thursday, April 12, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Jeremy Fischer is a Senior Technical Advisor at Indiana University.  He works primarily on the Jetstream project, as the technical evangelist getting researchers and educators on the system. In this role, he is  the jack of all trades doing unix sys admin work, cloud image maintenance, support, training, documentation, and anything else that needs to happen.
 
 
 
'''Links''':
 
 
 
*https://jetstream-cloud.org/about/index.php
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/-CGKHXvDVyM" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
 
 
 
 
==8 March, 2018: "Zarr: A simple, open, scalable solution for big NetCDF/HDF data on the Cloud": Alistair Miles, University of Oxford.==
 
 
 
'''Summary''': The motivation, current status and future plans for Zarr
 
will be discussed, along with a demo of basic functionality.
 
 
 
'''Time''': Thursday, March 8, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Alistair Miles is the Head of Epidemiological Informatics
 
for the Kwiatkowski group at the University of Oxford. Before joining
 
the University of Oxford, Alistair was a research scientist at the
 
e-Science Centre at the UK Science and Technology Facilities Council,
 
where he was involved in a range of computing research projects,
 
primarily in the areas of Web and semantic technology, and also in the
 
engineering of production software systems.  He is the lead developer
 
for Zarr.
 
 
 
'''Links''':
 
 
 
*https://github.com/zarr-developers/zarr
 
*[https://www.youtube.com/watch?v=8WtaYvqhxHc 5 minute demo of Dask + Zarr + S3]
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/np_p4JBAIYI" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
==8 February, 2018:  "The National Data Service Labs Workbench": Craig Willis, NCSA==
 
 
 
'''Summary''': The National Data Service Labs Workbench is a platform designed to share, discover, evaluate, develop, and test research data management and analysis tools. Community members can recommend or contribute tools as well as drive the direction, and the Workbench is evolving into a platform for data access, education and training.
 
 
 
'''Time''': Thursday, February 8, 2018, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Craig Willis is the Technical Coordinator for the National Data Service and a senior research programmer at the National Center for Supercomputer Applications (NCSA) at the University of Illinois.
 
 
 
'''Links''':
 
 
 
*http://www.nationaldataservice.org/
 
*https://www.workbench.nationaldataservice.org
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/WzZKpbwth_g" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
==11 January 2018:  "The Pangeo Project": Ryan Abernathy (Lamont) and Matthew Rocklin (Anaconda)==
 
 
 
'''Summary''':  Pangeo is a scalable, low-barrier-for-entry science platform, with cloud-optimized storage for large multidimensional datasets, such as simulation (met, ocean, hydrologic, climate) model output.  Technologies include JupyterHub, Kubernetes, Xarray, Dask, and Zarr. The Pangeo environment has been deployed on the NCAR Cheyenne supercomputer, on Google Cloud and on AWS.
 
 
 
'''Speaker(s)''':
 
Ryan Abernathy is a physical oceanographer at Lamont/Columbia, and Matthew Rocklin is a open-source developer for Anaconda. 
 
 
 
'''Links''':
 
 
 
*[https://github.com/pangeo-data/pangeo/issues pangeo discussion on github issues]
 
*[https://www.youtube.com/watch?v=rSOJKbfNBNk 3 minute demo of pangeo on Google Cloud]
 
*[http://matthewrocklin.com/blog/work/2018/01/22/pangeo-2 Blog post on pangeo]
 
*[http://matthewrocklin.com/blog/work/2018/02/06/hdf-in-the-cloud Blog post on big multidimensional data on the Cloud]
 
 
 
===Recording===
 
Note: This talk was given live at the ESIP winter meeting in North Bethesday, MD, USA.
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/mDrjGxaXQT4" frameborder="0" allowfullscreen></iframe></html>
 
 
 
==14 December  2017:  "Mini-Hack-Session: Developing and extending Jupyter Widgets": Jason Grout, Bloomberg==
 
 
 
'''Summary''': [https://github.com/jupyter-widgets/ipywidgets Jupyter widgets] (aka ipywidgets) enables building interactive GUIs for Python code using standard form controls (sliders, dropdowns, textboxes, etc.), as well providing a framework for building complex interactive controls such as interactive [https://github.com/bloomberg/bqplot 2d graphs], [https://github.com/maartenbreddels/ipyvolume 3d graphics], [https://github.com/ellisonbg/ipyleaflet maps], and [http://jupyter.org/widgets more]. Jason will walk through the thought and technical processes involved with developing new widget capability.
 
 
 
'''Time''': Thursday, Dec 14, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Jason Grout is scientific software developer at Bloomberg. He has been a member of the Project Jupyter team since it's inception in 2014 and a core developer for the Jupyter widgets project. He has a PhD in mathematics from Brigham Young University.
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/msJig1fr7Lw" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
==9 November 2017:  "Jupyter Widgets": Jason Grout, Bloomberg==
 
 
 
'''Summary''': [https://github.com/jupyter-widgets/ipywidgets Jupyter widgets] (aka ipywidgets) enables building interactive GUIs for Python code using standard form controls (sliders, dropdowns, textboxes, etc.), as well providing a framework for building complex interactive controls such as interactive [https://github.com/bloomberg/bqplot 2d graphs], [https://github.com/maartenbreddels/ipyvolume 3d graphics], [https://github.com/ellisonbg/ipyleaflet maps], and [http://jupyter.org/widgets more]. The latest developments in Jupyter widgets will be discussed as well as plans for the future.
 
 
 
'''Time''': Thursday, November 9, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Jason Grout is scientific software developer at Bloomberg. He has been a member of the Project Jupyter team since it's inception in 2014 and a core developer for the Jupyter widgets project. He has a PhD in mathematics from Brigham Young University.
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/CVcrTRQkTxo" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
==12 October 2017:  "Research Workspace: A web-based tool for data sharing, documentation, analysis, and publication": Rob Bochenek, Axiom Data Science==
 
 
 
'''Summary''': The Research Workspace (RW) is a web-based tool designed to support collaborative science and data management tasks throughout the data lifecycle. The RW provides a secure environment for organizing, sharing, documenting, and analyzing scientific datasets, and for publishing datasets through a DataONE member node. As a shared, cloud-based storage environment, the RW is designed for collaborative organization and management of project content. Multiple levels of access and read/write permissions provide transparent, controlled access and oversight to collaborators, funders, and project managers. A custom metadata editor exports standards-compliant metadata (ISO 19115-2 and 19110) and includes tools for easily adding keywords, taxonomic information, keywords, spatial boundaries, and contact information to metadata records. An integrated Jupyter notebooks environment allows R- and Python-based analysis scripts to be written in and run on the RW and to access data in the RW or any data set resource that is hosted within the Axiom Data Science cyber-infrastructure stack.  These notebooks serve as transparent, reproducible, and easily-shareable computational analysis and processing tools. Finally, datasets in the RW that have undergone sufficient curation and documentation can be exported to the Research Workspace DataONE Member Node for long-term preservation and broader discoverability and accessibility.  More information about the RW and it's capabilities can be found in the help documents -
 
 
 
'''Time''': Thursday, October 12, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Rob Bochenek is an information architect at Axiom Data Science.  Rob has been developing data management and cyber infrastructure solutions for research programs and organizations for the past fifteen years. He is a graduate of the University of Michigan with a degrees in aerospace engineering and mathematics. Early in his career Rob spent five years at the Exxon Valdez Oil Spill Trustee Council leading the data management team in processing, documenting and organizing the informational products produced from the scientific research funded to understand and monitor the ecological effects of the oil spill. Based upon that experience, Rob founded Axiom in 2006 to develop more generalized and holistic solutions for data management. He specializes in scientific geospatial information management with applications to physical/biological modeling and decision support data warehouse knowledge systems.
 
 
 
'''Links''':
 
 
 
*http://www.axiomdatascience.com/about/
 
*https://researchworkspace.com/help/
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/YOQgAMo0pYc" frameborder="0" allowfullscreen></iframe></html>
 
 
 
==14 September 2017:  "JupyterHub and JupyterLab Developments": Brian Granger, Cal Poly==
 
 
 
'''Summary''': The latest developments in JupyterHub and JupyterLab will be discussed as well as the roadmap for the future.
 
 
 
'''Time''': Thursday, September 14, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Brian Granger is an Associate Professor of Physics at Cal Poly State University in San Luis Obispo, CA. He has a background in theoretical atomic, molecular and optical physics, with a PhD from the University of Colorado. His current research interests include quantum computing, parallel and distributed computing and interactive computing environments for scientific and technical computing. He is a core developer of the Jupyter project and is an active contributor to a number of other open source projects focused on scientific computing in Python.
 
 
 
 
 
'''Links''':
 
 
 
*https://jupyterhub.readthedocs.io/en/latest/
 
*https://github.com/jupyterlab/jupyterlab
 
*https://jupyter.org
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/K1AsGeak51A" frameborder="0" allowfullscreen></iframe></html>
 
 
 
==[BONUS] 31 August 2017:  "ERDDAP 5 min Lightning Talks", Jenn Sevadjian, Jim Potemra, Conor Delaney, Kevin O'Brien, John Kerfoot, Stephanie Petillo, Charles Carleton, Eli Hunter==
 
 
 
'''Summary''': A series of 5 minute lightning talks on how people are using ERDDAP to solve environmental data problems.
 
 
 
'''Time''': Thursday, August 31, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''': Jenn Sevadjian, Jim Potemra, Conor Delaney, Kevin O'Brien, John Kerfoot, Stephanie Petillo, Charles Carleton, Eli Hunter
 
 
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/2-ydBByYB0M" frameborder="0" allowfullscreen></iframe></html>
 
 
 
 
 
==10 August 2017:  "ERDDAP: Easier access to scientific data": Bob Simons, NOAA==
 
 
 
'''Summary''': ERDDAP is a free, open source data server that gives you a simple, consistent way to download subsets of gridded and tabular scientific datasets in common file formats and make graphs and maps.
 
 
 
'''Time''': Thursday, August 10, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Speaker(s)''':
 
Bob Simons is an IT Specialist with NOAA's Environmental Research Division.
 
 
 
'''Links''':
 
 
 
*https://coastwatch.pfeg.noaa.gov/erddap/index.html
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/H541G1XXZrU" frameborder="0" allowfullscreen></iframe></html>
 
 
 
==13 July 2017:  "GeoServer Developments": Jody Garnett and Kevin Smith, Boundless==
 
 
 
'''Summary''': The latest developments in GeoServer will be discussed as well as plans for the future.
 
 
 
'''Time''': Thursday, July 13, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Jody Garnett is the Community Lead and Kevin Smith is the GeoWebCache Lead at Boundless.
 
 
 
'''Links''':
 
 
 
*http://geoserver.org/
 
*https://boundlessgeo.com/geoserver/
 
 
 
===Recording===
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/q4g0RXnadAU" frameborder="0" allowfullscreen></iframe></html>
 
 
 
==6 June 2017: "Installing JupyterHub in the Cloud using Kubernetes Helm": Yuvi Panda==
 
 
 
'''Summary''': Yuvi Panda will show how to deploy JupyterHub in the Cloud using Kubernetes Helm.
 
 
 
'''Time''': Tuesday, June 6, 2017, (Time: 3PM Eastern, 2PM Central, 1PM Mountain, 12PM Pacific)
 
 
 
'''Join meeting''':
 
 
 
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
 
 
'''Speaker(s)''':
 
Yuvi Panda is a developer with 15 years of experience and 400+ followers on GitHub.  He worked formerly with Wikimedia, and is currently working with the Data Science Education Program at UC Berkeley to make it easier for people who don't consider
 
themselves programmers to write code. He has been very involved with creating the Helm Chart for JupyterHub.  
 
  
===Recording===
+
Quarto can convert jupyter notebooks (with cached outputs) into markdown and vice versa.
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/aUwMlSIjtdg" frameborder="0" allowfullscreen></iframe></html>
 
  
 +
Issue is, you need to know a variety of other languages (YAML, CSS, Javascript, LaTeX, etc.)
  
 +
– "unavoidable but kinda gross"
  
'''Links''':
+
You can edit Quarto in RStudio or VS Code, or any text editor.
  
*https://github.com/yuvipanda
+
For collaboration, Quarto projects can use jupyter or knitr engines. E.g. in a single web page, you can build one page with jupyter and another page with knitr.
*https://jupyterhub.readthedocs.io/en/latest/
 
*https://gitter.im/jupyterhub/jupyterhub
 
*https://daemonza.github.io/2017/02/20/using-helm-to-deploy-to-kubernetes/
 
*https://github.com/kubernetes/helm
 
  
==11 May 2017:  "TerriaJS: A Free, Open-Source Library for Building Web-based Geospatial Data Explorers": Kevin Ring, CSIRO/Data61, Australia==
+
– you can embed a ipynb cell in a notebook.
  
'''Summary''': The library behind the Australian National Map. 3D and 2D geospatial visualization based on Cesium and Leaflet. Visualise WMS, WMTS, WFS, KML, GeoJSON, CSV, CZML, GPX, and many more spatial formats out of the box, or easily add your own. Present a dynamic catalog from your existing WMS, ArcGIS, CKAN, CSW, Socrata, WMTS or WFS server, curate your catalog by hand, or use any combination thereof. Explore time-varying WMS layers, watch vehicles move smoothly across the map, and observe your CSV data change over time.
+
Orchestrating computation is hard – quarto has to take input from existing computation – which can be awkward / complex.
  
'''Time''': Thursday, May 11, 2017, (5:00pm ET | 4:00pm CT | 3:00pm MT | 2:00pm PT | 07:00am Sydney Time)
+
Quarto is extensible – CSS themes, OJS for interactive webpages, Pandoc extensions.
  
'''Join meeting''':  
+
Can also write your own shortcodes.
 +
==July 13th 2023: "Tools to Assist Simulation Based Researchers in Deciding What Project Outputs to Preserve and Share"==
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
[[File:ITI_July_EarthCube.png|thumb|IT&I EarthCube Model RCN July 13th]]
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker(s)''':  
+
[https://staff.ucar.edu/users/schuster Doug Schuster]
  
Kevin Ring is a Principal Software Engineer at CSIRO's Data61, and is the lead developer for TerriaJS. Previously, he helped found the Cesium project while working at Analytical Graphics, Inc. (AGI) and developed its streaming terrain and imagery engine.
+
This presentation will highlight findings from the NSF EarthCube
 +
Research Coordination Network project titled “What About Model Data? -
 +
Best Practices for Preservation and Replicability”
 +
(https://modeldatarcn.github.io/), which suggest that most simulation
 +
based research projects only need to preserve and share selected model
 +
outputs, along with the full simulation experiment workflow to
 +
communicate knowledge. Challenges related to meeting community open science
 +
expectations will also be highlighted.
  
'''Links''':
+
Slides available here: [[File:ModelDataRCN-2023-07-13-ESIP-IT&I_v2.pdf|thumb]]
  
*http://terria.io/
+
https://modeldatarcn.github.io/
*https://github.com/TerriaJS/terriajs
 
*http://nationalmap.gov.au/
 
  
===Recording===
+
Rubric: https://modeldatarcn.github.io/rubrics-worksheets/Descriptor-classifications-worksheet-v2.0.pdf
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PL8X9E6I5_i8gmLI7r6huyQNr0oC1mCadA" frameborder="0" allowfullscreen></iframe></html>
 
  
==13 April 2017:  "Processing Planetary-Scale Data in the Cloud": Drew Bollinger, Development Seed==
+
''Open science expectations for simulation based research. Frontiers in Climate, 2021. https://doi.org/10.3389/fclim.2021.763420''
  
'''Summary''': Modern cloud-based infrastructure has had a huge effect on our ability to process, manipulate, and publish satellite imagery at scale. We'll discuss current methods of making imagery available across different platforms and how this is supported by the efforts of groups like AWS to publish open satellite data including MODIS, Landsat and more.  
+
'''<u>Recording</u>''':<br />
 +
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/ulk0mQSQNzQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
'''Time''': Thursday, April 13, 2017, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
'''<u>Minutes:</u>'''
  
'''Join meeting''':  
+
Primary motivation: What are data management requirements for simulation projects?
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
Project ran May 2020 to Jul 2022
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Presenter''':
+
We clearly shouldn't preserve ALL data / output from projects. It's just too expensive.
  
Drew Bollinger is a data analyst and software developer, with experience running advanced statistical and spatial analysis on large and small data sets, as well as building visualizations for data storytelling.
+
Project broke down components of data associated with a different project
 +
Forcings, code/documentation, selected outputs.
  
'''Links''':
+
But what outputs to share?!?
  
*https://github.com/sat-utils/sat-api
+
Project developed a rubric of what to preserve / share.
*https://github.com/sat-utils
 
*https://github.com/developmentseed/landsat-util
 
*https://libra.developmentseed.org/
 
  
===Slides===
+
"Is your project a data production project or a knowledge production project"
http://drewbo.com/talks/esip-2017/#0
 
  
===Recording===
+
"How hard is it to rerun your workflow?"
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/PO2z37XX1Gg" frameborder="0" allowfullscreen></iframe></html>
 
  
==9 March 2017:  "Introduction to Esri Story Maps": Christine White, Esri==
+
"How much will it cost to store and serve the data?"
  
'''Summary''': Today, multi-media communication plays a pivotal role in how an audience experiences, understands, and shares your message. Story Maps bring a narrative to life by weaving maps,  text, images, video, and other content into a creative and memorable story. Christine will share several examples of effective Story Maps and then walk through how you can create and configure your own.
+
Rubric gives guidance on how much of a project's outputs should be preserved.
  
'''Time''': Thursday, March 9, 2017, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
So this is all well and good, but it falls onto PIs and funding agencies.
  
'''Join meeting''':
+
What are the ethical and professional considerations of these trade offs?
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
What are the incentives in place currently? Sharing is not necessarily seen as a benefit to the author.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker(s)''':  
+
==June 8 2023: "Reproducible Data Pipelines in Modern Data Science: what they are, how to use them, and examples you can use!"==
  
Christine is a Technical Advisor and science team member at Esri. She loves using art and technology to communicate about the challenges and opportunities for our future. Christine also serves as the Vice President of ESIP. One of her favorite things about ESIP is how its members offer their unique perspectives (stories) and shared knowledge to collaborate.
+
[[File:ITI_June_Pipeline.png|thumb|IT&I Reproducible Pipelines June 8th]]
  
===GoToMeeting Recording===
+
[https://www.usgs.gov/staff-profiles/julie-padilla Julie Padilla]
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/wqQW2xVw0hA" frameborder="0" allowfullscreen></iframe></html>
+
Modern scientific workflows face common challenges including accommodating growing volumes and complexity of data and the need to update analyses as new data becomes available or project needs change. The use of better practices around reproducible workflows and the use of automated data analysis pipelines can help overcome these challenges and more efficiently translate open data to actionable scientific insights. These data pipelines are transparent, reproducible, and robust to changes in the data or analysis, and therefore promote efficient, open science. In this presentation, participants will learn what makes a reproducible data pipeline and what differentiates it from a workflow as well as the key organizational concepts for effective pipeline development.
  
===Slides===
+
'''<u>Recording</u>''':<br />
Christine gave her presentation as a live StoryMap, available here:
+
<html>
https://www.arcgis.com/apps/MapSeries/index.html?appid=5a99a82a19c84dbab641a22ddd3d329b
+
<iframe width="560" height="315" src="https://www.youtube.com/embed/K8EOY_HLlho" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
==9 February 2017: "Web AppBuilder for ArcGIS": Derek Law, ESRI==
+
'''<u>Minutes:</u>'''
  
'''Summary''': Web AppBuilder for ArcGIS is a pure HTML5/JavaScript-based application that allows you to create your own intuitive, fast, and beautiful web apps without writing a single line of code. The app uses new ArcGIS platform features and modern browser technology to provide both flexible and powerful capabilities such as 3D visualization of data. In addition, developers have an opportunity to create custom tools and themes through the extensibility framework.
+
Motivation –
 +
what if we find bad data in an input,
 +
what if we need to rerun something with new data,  
 +
can we reproduce findings from previous work?
  
'''Time''': Thursday, February 9, 2017, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
Need to be able to "trace" what we did and the way we do it needs to be reliable.
  
'''Join meeting''':
+
A "workflow" is a sequence of steps going from start to finish of some activity or process.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
A "pipeline" is a programmatic implementation of a workflow that requires little to no interaction.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Speaker(s)''':
+
In a pipeline, if one workflow step or input gets changed, we can track what is "downstream" of it.
  
Derek Law is an Product Manager at ESRI. He has over 15 years experience with geospatial software and web application development.
+
Note that different steps of the workflow may be influenced by  different people. So a given step of a pipeline could be contributed by different programmers. But each person would be contributing a component of a consistent pipeline.
  
===GoToMeeting Recording===
+
There is a difference between writing scripts to building a reproducible pipeline.
 +
Better to break it into steps. Script -> organize -> encapsulate into functions -> assemble pipeline.
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/7uKbfMSX6Sw" frameborder="0" allowfullscreen></iframe></html>
+
Focus is on R targets – snakemake is equivalent in python.
  
===Slides===
+
Key concepts for going from script to workflow:
https://speakerdeck.com/esipfed/esri-webapp-builder-derek-law-esri
+
Functions stored separate from workflow script.
 +
Steps clearly organized in script.
 +
Can wrap steps in pipeline steps to track them.
  
==19 January 2017:  "Introduction to Google Earth Engine": Jess Walker, USGS==
+
Pipeline software keeps track of whether things have changed and what needs to be rerun.
 +
Allows visualization of the workflow inputs, functions, and steps.
  
'''Summary''': Google Earth Engine is a cloud-based geospatial processing platform that unites multiple petabytes of publicly accessible imagery and a massive computational infrastructure with a web-based integrated development environment (IDE).  Users can harness the unprecedented combination of data and computing resources to conduct complex geospatial analyses on planetary scales.  
+
How do steps of the pipeline get related to eachother?
 +
They are named and the target names get passed to downstream targets.
  
'''Time''': Thursday, January 19, 2017, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
Chat questions about branching.
 +
Dynamic branching lets you run the same target for a list of inputs in a map/reduce pattern.
  
'''Join meeting''':
+
Pipelines can have outputs that are reports that render pipeline results in a nice form.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
Pipeline templates:
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
+
A pipeline can adopt from a standard template that is pre-determined.  
 +
Helps enforce best practices and have a quick and easy starting point.
  
'''Speaker(s)''':
+
Note that USGS data science has a template for a common pattern.
  
Jessica Walker is a postdoctoral researcher with the USGS Western Geographic Science Center in Tucson, AZ.  Her research investigates the recovery of post-wildfire landscapes in Alaska and across the southwestern US using time series of remote sensing imagery.  
+
What's a best practice for tracking container function and reproducibility?
 +
Versioned Git / Docker for code and environment.
 +
For data, it is context dependent. Generally, try to pull down citeable / persistent sources. If sources are not persistent, you can cache inputs for later reuse / reproducibility.
 +
 +
Data change detection / cacheing is a really tricky thing but many people are working on the problem. https://cboettig.github.io/contentid/, https://dvc.org/
  
 +
https://learning.nceas.ucsb.edu/2021-11-delta/session-3-programmatic-metadata-and-data-access.html#reproducible-data-access
  
===GoToMeeting Recording===
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/m47eHiOL0ZI" frameborder="0" allowfullscreen></iframe></html>
+
==11 May 2023: "Software Procurement Has Failed Us Completely, But No More!"==
  
===Slides===
+
[[File:ITI_May_Software.png|thumb|IT&I Software Procurement May 11th]]
https://speakerdeck.com/esipfed/introduction-to-google-earth-engine-jessica-walker-usgs
 
  
==8 December 2016: "Vector Tile Maps": Sam Matthews, Mapbox==
+
[https://waldo.jaquith.org/ Waldo Jaquith]
  
'''Summary''': Vector tiles make huge maps fast while offering full design flexibility. They are the vector data equivalent of image tiles for web mapping, applying the strengths of tiling – developed for caching, scaling and serving map imagery rapidly – to vector data.  A general overview of vector tiles will be presented.  
+
The way we buy custom software is terrible for everybody involved, and has become a major obstacle to agencies achieving their missions. There are solutions, if we would just use them! By combining the standard practices of user research, Agile software development, open source, modular procurement, and time & materials contracts, we can make procurement once again serve the needs of government.
  
'''Speaker(s)''':  
+
Slides available here: [[File:2023-05-Jaquith.pdf|thumb]]
  
Sam Matthews is a Mapbox engineer  focused on improving the speed and reliability of maps. He works with the Mapnik team to generate vector tiles and maintains the upload pipeline behind Mapbox Studio. He is passionate about making open source tools as welcoming as possible through clear docs and zero assumptions.
+
'''<u>Recording</u>''':<br />
 +
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/V4-3WZ5hN5k" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
'''Time''': Thursday, December 8, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
'''<u>Minutes:</u>'''
  
'''Join meeting''':
+
Recognizing that software procurement is one of the primary ways that Software / IT systems advance, Waldo went into trying to understand that space as a software developer.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
'''Healthcare.gov '''
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
===GoToMeeting Recording===
+
Contract given to CGI Federal for $93M – cost ~1.7B by launch.<br>
https://www.youtube.com/watch?v=wN2-ms2PwBs
+
Low single digits numbers of people actually made it through the system.<br>
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/wN2-ms2PwBs" frameborder="0" allowfullscreen></iframe></html>
+
Senior leaders were given the impression that things were in good shape.<br>
 +
The developers working on the site knew it wasn't going to work out (per IG report).<br>
 +
*strategic misrepresentation – things are represented as more rosy as you go up the chain of command<br>
 +
On launch, things went very badly and the recovery was actually quite quick and positive.<br>
  
===Slides===
+
Waldo recommends reading the IG report on healthcare.gov.
https://speakerdeck.com/esipfed/vector-tile-maps-sam-matthews-mapbox
+
This article: <br>
 +
''("The OIG Report Analyzing Healthcare.gov's Launch: What's There And What's Not", Health Affairs Blog, February 24, 2016. https://dx.doi.org/10.1377/hblog20160224.053370<nowiki/>)'' provides a path to the IG report: <br>
 +
''(HealthCare.gov - CMS Management of the Federal Marketplace: An OIG Case Study (OEI-06-14-00350), https://oig.hhs.gov/oei/reports/oei-06-14-00350.pdf<nowiki/>)''
 +
and additional perspective.
  
==10 November 2016:  "Introducing 3D Tiles": Todd Smith, AGI==
+
'''Rhode Island Unified Health Infrastructure'''
  
'''Summary''': 3D Tiles are an open specification for streaming massive heterogeneous 3D geospatial datasets. To expand on Cesium’s terrain and imagery streaming, 3D Tiles will be used to stream 3D content, including buildings, trees, point clouds, and vector data.  
+
($364M to DeLoitte) "Big Bang" deployment – they let people running old systems go on the day of the new system launch.
 +
They "outsourced" a mission critical function to a contractor.
  
'''Speaker(s)''':
+
We don't tend to hear about relatively smaller projects because they are less likely to fail and garner less attention.
  
Todd Smith is the Cesium Product Manager, and helps define and manage the Cesium product line. Todd has been with the AGI team from the beginning and has been in the web mapping world for over 15 years.  He is a Penn State GIS graduate.  
+
Outsourcing as started in the ~90s was one thing when the outsourcing was for internal agency software. It's different when the systems are actually public interfaces to stakeholders or are otherwise mission critical.  
  
 +
'''[[File:2023-05-Jaquith.pdf|See slides with big numbers and study sources!!]]'''
  
'''Time''': Thursday, November 10, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
It's common for software to meet contract requirements but NOT meet end user needs.
  
'''Join meeting''':
+
Requirements complexity is fractal. There is no complete / comprehensive set of requirements.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
… federal contractors interpreting requirements as children trying to resist getting out the door ...
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
===GoToMeeting Recording===
+
There is little to no potential to update or improve requirements due to contract structure.
https://www.youtube.com/watch?v=0upb4E12CPE
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/0upb4E12CPE" frameborder="0" allowfullscreen></iframe></html>
 
  
===Slides===
+
'''Demos not memos!'''
https://speakerdeck.com/esipfed
 
  
==13 October 2016: "EarthCube Integration and Test Environment (ECITE)": Phil Yang, GMU==
+
Memorable statements:
 +
*Outsourced ability to accomplish agency’s mission
 +
*Load bearing software systems on which the agency depends to complete their mission.
 +
*Mission of many agencies is mediated by technology.
  
'''Summary''': An outgrowth of activities of the EarthCube Technology Architecture Committee (TAC)'s Testbed Working Group (TWG), ECITE provides an integration test-bed for technology and science projects for both EarthCube funded projects and community technology demonstrations.  ECITE consists of a seamless federated system of scalable and location independent distributed computational resources (nodes) across the US. The hybrid federated system provides a robust set of distributed resources utilizing including both public and private cloud capabilities.
+
But no more! – approach developed by 18F
  
'''Speaker(s)''': Chaowei Phil Yang is a Professor at George Mason University where he founded the NSF Spatiotemporal Innovation Center with colleagues from Harvard and UC-Santa Barbara. He advised over 30 graduate students and has placed over 20 geoinformatics professors around the world.  His research interest are utilizing spatiotemporal principles to optimize computing infrastructure for geospatial science applications of national and international significance. (http://cpgis.gmu.edu/homepage/)
+
System of six parts –
  
'''Time''': Thursday, October 13, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
1. User-centered design  <br>
 +
2. Agile software development <br> 
 +
3. Product ownership  <br>
 +
4. DevOps  <br>
 +
5. Building out of loosely coupled parts  <br>
 +
6. Modular contracting  <br>
  
'''Join meeting''':  
+
[[File:Agil-control-model.png|frame|Roles for government and vendors in agile contracting]]
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
"You don't know what people need till you talk to them."
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
===GoToMeeting Recording===
+
Basic premise of agile is good. Focus is on finished software being developed every two weeks.
https://www.youtube.com/watch?v=kYi-22hXY6k
 
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/kYi-22hXY6k" frameborder="0" allowfullscreen></iframe></html>
 
  
===Slides===
+
Constantly delivering a usable product... e.g. A skateboard is more usable than a car part.
https://speakerdeck.com/esipfed
 
  
 +
Key roles for government staff around operations are too often overlooked.
  
==8 September 2016:  "Apache Open Climate Workbench": Lewis McGibbney and Kyo Lee, NASA JPL/Apache OCW==
+
Product team needs to include an Agency Product Owner. Allows government representation in software development iteration.
  
'''Summary''': Apache [http://climate.apache.org Open Climate Workbench] (OCW) is an effort to develop software that performs climate model evaluation using model outputs from a variety of different sources the [http://esgf.llnl.gov/ Earth System Grid Federation], the [http://www.cordex.org/ Coordinated Regional Climate Downscaling Experiment], the [http://nca2014.globalchange.gov/ U.S. National Climate Assessment] and the [http://www.narccap.ucar.edu/ North American Regional Climate Change Assessment Program] and temporal/spatial scales with remote sensing data from [http://www.nasa.gov NASA], [http://www.noaa.gov NOAA] and other agencies. The toolkit includes capabilities for rebinning, metrics computation and visualization.
+
Build out of loosely coupled / interchangeable components. Allows you to do smaller things and form big coherent systems that can evolve.
  
'''Speaker(s)''': Lewis McGibbney, NASA JPL/Apache OCW; currently a Data Scientist at the NASA Jet Propulsion Laboratory in Pasadena, California, Lewis works in the Computer Science and Data Intensive Applications Group (398M). He enjoys floating up and down the tide of technologies at the Apache Software Foundation having a real enthusiasm for Web Search and Information Retrieval in particular. You'll find him on community mailing lists including Nutch, Gora, Any23, OODT, Open Climate Workbench, Tika, Usergrid and a number of incubating mailing lists including CommonsRDF, HTrace and Joshua. Lewis is currently a Project Management Committee member and Committer on OCW.
+
Modular contracts allow big projects that are delivered through many small task orders or contracts. The contract document is kind of a fill in the blank template and doesn't have to be hard.
  
'''Speaker(s)''': Huikyo Lee, NASA JPL/Apache OCW; currently a Climate Data Scientist at the NASA Jet Propulsion Laboratory in Pasadena, California, Huikyo has lead development of Regional Climate Model Evaluation System (http://rcmes.jpl.nasa.gov), an open-source software toolkit based on Open Climate Workbench to facilitate systematic evaluation of climate models using observational datasets from a variety of sources.
+
Westrum typology of cultures article is relevant is relevant: http://dx.doi.org/10.1136/qshc.2003.009522
  
'''Time''': Thursday, September 8, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
==13 April 2023: "Evolution of open source geospatial python."==
  
'''Join meeting''':  
+
[[File:Iti april geospatialpython 720.png|thumb|IT&I Python Open Source April 13th]]
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
[https://github.com/tomkralidis Tom Kralidis]
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
===GoToMeeting Recording===
+
Free and Open Source Software in the Geospatial ecosystems (i.e. FOSS4G) play a key role in geospatial systems and services.  Python has become the lingua franca for scientific and geospatial software and tooling.  This rant and rave will provide an overview of the evolution of FOSS4G and Python, focusing on popular projects in support of Open Standards.
https://www.youtube.com/watch?v=YA8SZiG9JZk
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/YA8SZiG9JZk" frameborder="0" allowfullscreen></iframe></html>
+
Slides: https://geopython.github.io/presentation
  
===Slides===
+
'''<u>Recording</u>''':<br />
https://speakerdeck.com/esipfed/apache-ocw
+
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/HTouLSzKGto" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
==11 August 2016: "Community Data Analysis Tools (CDAT)": Charles Doutriaux, LLNL==
+
'''<u>Minutes:</u>'''
  
'''Summary''': CDAT is a rich set of visual-data exploration and analysis capabilities well-suited for earth science data analysis problems. It integrates many tools and technology to offer scientist a start-to-finish environment for their work. From reading in various data format, to publication-quality output of their analysis.
+
Mapserver has been around for 23 years!
  
'''Speaker''': Charles Doutriaux is a senior Lawrence Livermore National Laboratory research computer scientist, where he is known for his work in climate analytics, informatics, and management systems supporting model intercomparison projects. He works closely with many international climate scientists and shares in the recognition of the Intergovernmental Panel on Climate Change 2007 Nobel Peace Prize. He has co­-authored over 30 peer­-reviewed articles. He presented his work to many scientific conferences. Aside from everything Python-related, his research interests include climate attribution and detection, visualization, and data analysis. Doutriaux has a master's degree in "Climate and Physico-­Chemistry of the Atmosphere" from the University Joseph Fourier in Grenoble. He’s a member of the AGU and AMS. You can contact him at doutriaux1@llnl.gov.
+
Why Python for Geospatial?
 +
Ubiquity
 +
Cross OS compatible
 +
Legible and easy to understand what it's doing
 +
Support ecosystem is strong (PyPI, etc.)
 +
Balance of performance and ease of implementation
 +
Python: fast enough, and fast in human time -- more intensive workloads can glue to C/C++
  
'''Time''': Thursday, August 11, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
The new generation of OGC services – based on JSON, so the API interoperates with client environments / objects at a much more direct level.
  
'''Join meeting''':
+
The geopython ecosystem has a number of low level components that are used across multiple projects.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
pygeoapi is an OGC API reference implementation and an OSGeo project.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
+
E.g. https://github.com/developmentseed/geojson-pydantic
  
===GoToMeeting Recording===
+
pygeoapi implements OGC API - Environmental Data Retrieval (EDR) https://ogcapi.ogc.org/edr/overview.html
https://www.youtube.com/watch?v=nh2dqAHt5jY
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/nh2dqAHt5jY" frameborder="0" allowfullscreen></iframe></html>
+
pygeoapi has a plugin architecture.
 +
https://pygeoapi.io/
 +
https://code.usgs.gov/wma/nhgf/pygeoapi-plugin-cookiecutter
  
===Slides===
+
pycsw is an OGC CSW and OGC API - Records implementation.
 +
Works with pygeometa for metadata creation and maintenance.
 +
https://geopython.github.io/pygeometa/
  
==13 July 2016:  "The NOAA OneStop Data Discovery and Access Framework Project": Ken Casey, NOAA/NCEI==
+
There's a real trade off to "the shiny object" vs the long term sustainability of an approach. Geopython has generally erred on the side of "does it work in a virtualenv out of the box".
  
'''Summary''': The OneStop Project is designed to improve NOAA's data discovery and access framework.  Focusing on all layers of the framework and not just the user interface, OneStop is addressing data format and metadata best practices, ensuring more data are available through modern web services, working to improve the relevance of dataset searches, and improving both collection-level metadata management and granule level metadata systems to accommodate the wide variety and vast scale of NOAA's data.
+
How does pycsw work with STAC and other catalog APIs?
 +
pycsw can convert between various representations of the same basic metadata resource.
  
'''Speaker''': Ken Casey is the Deputy Director of the Data Stewardship Division in the NOAA National Centers for Environmental Information (NCEI).  He leads the OneStop project, is active within NOAA's Big Earth Data Initiative and Big Data Project.  Ken serves on a variety of national and international science and data management panels including the US Group on Earth Observations Data Management Working Group and the Group for High Resolution Sea Surface Temperature (GHRSST) Science Team.  He co-chairs the Committee on Earth Observing Satellites SST Virtual Constellation and represents NCEI in the Federation of Earth Science Information Partners (ESIP).  He holds a PhD in Physical Oceanography from the University of Rhode Island.  
+
"That's a pattern… People can implement things the way they want."
  
'''Time''': Wednesday, July 13, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
'''<u>Chat Highlights:</u>'''
  
'''Join meeting''':  
+
*You can also write a C program that is slower than Python if you aren't careful =).
 +
*https://www.ogc.org/standards/ has lots of useful details
 +
*For anyone interested in geojson API development in Python, I just recently came across this https://github.com/developmentseed/geojson-pydantic
 +
* OGC API - Environmental Data Retrieval (EDR) https://ogcapi.ogc.org/edr/overview.html
 +
*Our team has a pygeoapi plugin cookiecutter that we are hopeful others can get some mileage out of. https://code.usgs.gov/wma/nhgf/pygeoapi-plugin-cookiecutter
 +
*I'm going to post this here and run: https://twitter.com/GdalOrg/status/1613589544737148944
 +
**''100% agreed. That's unfortunate, but PyPI is not designed to deal with binary wheels of beasts like me which depend of ~ 80 direct or indirect other native libraries. Best solution or least worst solution depending on each one's view is "conda install -c conda-forge gdal"''
 +
*General question here - you mentioned getting away from GDAL in a previous project. What are your thoughts on GDAL's role in geospatial python moving forward, and how will pygeoapi accommodate that?
 +
*Never, ever works with the wheels!
 +
* Kitware has some pre-compiled wheels as well: https://github.com/girder/large_image
 +
*In the pangeo.io project, our go to tools are geopandas for tabular geospatial data, xarray/rioxarray for n-dimensional array data, dask for parallelization, and holoviz for interactive visualization.  We use the conda-forge channel pretty much exclusively to build out environments
 +
*If you work on Windows, good luck getting the Python gdal/geos-based tools installed without Conda
 +
*data formats and standards are what make it difficult to get away from GDAL -- it just supports so many different backends!  Picking those apart and cutting legacy formats or developing more modular tools to deal with each of those things "natively" in python would be required to get away from the large dependency on something like GDAL.
 +
*Sustainability and maintainability is always good to ask yourself "how easy will it be to replace this dependency when it no longer works?"
 +
*No one should build gdal alone (unless it is winter and you need a source of heat). Join us at https://github.com/conda-forge/gdal-feedstock
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
 
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
===GoToMeeting Recording===
+
==9 Mar 2023: "Meeting Data Where it Lives: the power of virtual access patterns"==
https://youtu.be/wp7trIRFDOs
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/wp7trIRFDOs" frameborder="0" allowfullscreen></iframe></html>
+
[https://github.com/mikejohnson51 Mike Johnson] (Lynker, NOAA-affiliate) will rant and rave about the VRT and VSI (curl and S3) virtual data access patterns and how he's used them to work with LCMAP and 3DEP data in integrated climate and data analysis workflows.
  
===Slides===
+
'''<u>Recording</u>''':<br />
https://speakerdeck.com/esipfed/noaa-one-stop-ken-casey-ncei
+
<html>
 +
<iframe width="560" height="315" src="https://www.youtube.com/embed/auK_gPR-e7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 +
</html>
  
==9 June 2016: "Dive into Docker":  Kyle Wilcox, Dave Foster and Shane StClair: Axiom Data Science==
+
'''<u>Minutes:</u>'''
  
'''Summary''': Docker is an open platform for distributed applications that has taken the world by storm, making it easy to deploy services with complicated dependencies.  In this presentation you will learn what Docker is, why it will make your life easier, how to build a container, and how to install containers.
+
* VRT stands for "ViRTual"
 +
*VSI stands for "Virtual System Interface"
 +
*Framed by FAIR
  
'''Speaker''': Kyle Wilcox, Dave Foster and Shane StClair are developers at Axiom Data Science.  Axiom Data Science works with organizations to improve the long term management, reuse and impact of their scientific data resources.  They have built Docker containers for many of the key services used by the U.S. Integrated Ocean Observing System (US-IOOS).  
+
LCMAP – requires fairly complex URLs to access specific data elements.
  
'''Time''': June 9, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
3DEP - need to understand tiling scheme to access data across domains.
  
'''Join meeting''':
+
Note some large packages (zip files) where only one small file is actually desired.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
NWM datasets in NetCDF files that change name (with time step) daily as they are archived.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
 
  
'''Links''':
 
  
*http://www.docker.com/
+
Implications for Findability, Availability, and Reuse – note that interoperability is actually pretty good once you have the data.
  
===GoToMeeting Recording===
+
VRT: – an XML "metadata" wrapper around one or more tif files.
https://youtu.be/mDR_x0E5az0
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/mDR_x0E5az0" frameborder="0" allowfullscreen></iframe></html>
+
Use case 1: download all of 3DEP tiles and wrap in a VRT xml file.
  
===Slides===
+
*VRT has an overall aggregated grid "shape"
https://speakerdeck.com/esipfed/dive-into-docker-kyle-wilcox-shane-stclair-dave-foster-axiom-data-science
+
*Includes references to all the individual files.
 +
*Can access the dataset through the vrt wrapper to work across all the times.
 +
*Creates a seamless collection of subdatasets
 +
* Major improvement to accessibility.
 +
If you have to download the data is that "reuse" of the data??
  
==12 May 2016: "Leaflet Time Dimension":  Biel Frontera, SOCIB==
+
VSI: – allows virtualization of data from remote resources available as a few protocols (S3/http/compressed)
  
'''Summary''': Leaflet.TimeDimension is a free, open-source Leaflet.js plugin that enables visualization of spatial data with a temporal dimension. It can manage different types of layers (WMS, GeoJSON, Overlay) and it can be easily extended.  It meet some common needs, enabling web maps using observational and forecasting layers generated by a THREDDS server (via ncWMS), animating trajectories of drifters, gliders, follow a simulated oil spill, and other time dependent mapping applications. 
+
Wide variety of GDAL utilities to access VSI files – zip, tar, 7zip
  
'''Speaker''': Biel Frontera was trained as a mathematician, and has spent most of his career developing software.  He is a free software enthusiast and has worked for the last 3 years on data visualization and geospatial software issues for SOCIB, the Baleric Islands Coastal Observing and Forecasting System.  
+
Use case 2: Access a tif file remotely without downloading all the data in the file.
  
'''Time''': May 12, 2016, (3:00pm ET | 2:00pm CT | 1:00pm MT | 12:00am PT)
+
*Uses vsi to access a single tif file
  
'''Join meeting''':  
+
Use case 3: Use vsi within a vrt to remotely access contents of remote tif files.
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/533510693
+
*Note that the vrt file doesn't actually have to be local itself.
*regular phone: United States: +1 (408) 650-3123, Access Code: 533-510-693
+
*If the tiles that the vrt points to update, the vrt will update by default.
 +
*Can easily access and reuse data without actually copying it around.
  
'''Links''':
+
Use case 4: OGR using vsi to access a shapefile in a tar.gz file remotely.
  
*https://github.com/socib/Leaflet.TimeDimension
+
*Can create a nested url pattern to access contents of the tar.gz remotely.
*http://apps.socib.es/Leaflet.TimeDimension/examples/
 
*http://www.socib.eu/
 
  
===GoToMeeting Recording===
+
Use case 5: NWM shortrange forecast of streamflow in a netcdf file.
https://www.youtube.com/watch?v=US5FUUPqlww
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/US5FUUPqlww" frameborder="0" allowfullscreen></iframe></html>
+
* Appending "HDF5:" to the front of a vsicurl url allows access to a netcdf file directly.
 +
*The access url pattern is SUPER tricky to get right.
  
===Slides===
+
Use case 5: "flat catalogs"
https://speakerdeck.com/esipfed/leatlet-time-dimension-biel-frontera-socib
 
  
==21 Apr 2016:  "The New Geoplatform.gov":  Tod Dabolt, DOI==
+
*Stores a flat (denormalized) table of data variables with the information required to construct URLs.
 +
*Can search based on rudimentary metadata within the catalog.
 +
* Can access and reuse data from any host in the same workflow.
  
'''Summary''': Geoplatform.gov was recently rebuilt from the ground up. Tod will talk about new features of the platform and plans for the future.
+
Use case 6: access NWM current and archived data from a variety of cloud data stores.
  
'''Speaker''': Tod Dabolt is the acting Geographic Information Officer for the Department of Interior, and the technical lead on Geoplatform.gov.  
+
*Leveraging the flat catalog content to fix up urls and data access nuances.
  
'''Time''': April 21, 2016, (2:00pm ET | 1:00pm CT | 12:00pm MT | 11:00am PT)
+
Flat catalog improves findability down at the level of individual data variables.
  
'''Join meeting''':  
+
Take Aways / discussion:
  
*computer, tablet or smartphone: https://www.gotomeeting.com/join/271218861
+
Question about the flat catalog:
*regular phone: United States: +1 (872) 240-3212, Access Code: 271-218-861
 
  
'''Links''':
+
"Minimal set of shortcuts" to get at this fast access mechanism.
  
*http://www.geoplatform.gov
+
Is the flat catalog manually curated?
  
===GoToMeeting Recording===
+
More or less – all are automated but some custom logic is required to add additional content.
https://www.youtube.com/watch?v=f-ABUpy4Qvk
 
  
<html><iframe width="560" height="315" src="https://www.youtube.com/embed/f-ABUpy4Qvk" frameborder="0" allowfullscreen></iframe></html>
+
Would be great to systematize creation of this flat catalog more broadly.
  
===Slides===
+
Question: Could some “examples” be posted either in this doc or elsewhere (or links to examples), for a beginner to copy/paste some code and see for themselves, begin to think about how we’d use this? Something super basic please.
https://speakerdeck.com/esipfed/the-new-geoplatform-tod-dabolt-doi
 
<html><script async class="speakerdeck-embed" data-id="9c2ac038b60a4943bab8a3005350b95e" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script></html>
 
  
==13 Oct 2015: Raj Pandya on AGU's Thriving Earth Exchange and Sharing Solutions==
+
GDAL documentation is good but doesn't have many examples.
  
The Thriving Earth Exchange is a network and platform that connects community leaders, sponsors, and scientists and helps them combine science and local knowledge to solve on-the-ground challenges related to natural hazards, natural resources, and climate change.  I’ll talk about the general principles on which we are building TEX and describe the basic modules that are part of the TEX. Drawing on the lessons learned from our pilots, I'll talk about how we are developing modules and launching new projects with several partners. I’ll describe a range of projects – from a community monitoring effort in Denver to a Pamiri Mountain project to integrate climate projections into traditional calendars. I’ll introduce our nascent “share” module, and describe our partnership with Amazon Web Services to move prototype community-based solutions to the cloud to enhance their adaptability. And, just to live up to the name, I’ll  frame it all around a small rant about the loading-dock model of science and a rave about more participatory approaches.
+
climateR has a workflow that shows how the catalog was built.
  
===Slides===
 
  
[[Media:2015-10-13 ESIP RantRave RajPandya.pdf| PDF]]
+
What about authentication issues?
  
==13 Aug 2015: Rich Signell on Catalog-driven Workflows for Science==
+
*S3 is handled at a session level.
 +
*Earthengine can be handled similarly.
 +
How much word of mouth or human-to-human interaction is required for the catalog.
  
"Catalog-driven, reproducible workflows for ocean science: Comparing
+
*If there is a stable entrypoint (S3 bucket for example) some automation is possible.
sea level forecasts along the US Coastline"
+
*If entrypoints change, configuration needs to be changed based on human intervention.
  
Rich Signell
+
==9 Feb 2023: "February 2023 - Rants & Raves"==
  
Filipe Fernandes
+
The conversation built on the "rants and raves" session from the 2023 January ESIP Meeting, starting with very short presentations and an in-depth discussion on interoperability and the Committee's next steps.
  
The USGS Integrated Ocean Observing System (US-IOOS) requires that
+
'''<u>Recording</u>''':<br />
data providers use standard web services (OPeNDAP+CF, OGC WMS, OGC
+
<html>
SOS) for distributing model products and insitu observations.   The
+
<iframe width="560" height="315" src="https://www.youtube.com/embed/cS7TrLmSu5U" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
services are captured in ISO metadata records and searchable via
+
</html>
standard catalog services (OGC CSW).
 
  
This presentation will demonstrate how to use this system in a
+
'''<u>Minutes:</u>'''
reproducible Jupyter Notebook, discovering, accessing and using model
 
and observed water levels along the US Coastline, using a free python
 
environment that can be installed on Mac, Windows and Linux in less
 
than 10 minutes.
 
  
===Slides===
+
*Mike Mahoney: Make Reproducibility Easy
 +
*Dave Blodgett: FAIR data and Science Data Gateways
 +
*Doug Fils:  Web architecture and Semantic Web
 +
*Megan Carter: Opening Doors for Collaboration
 +
*Yuhan (Douglas) Rao: Where are we for AI-ready data?
  
[https://speakerdeck.com/rsignell/catalog-driven-reproducible-workflows-for-ocean-science Speaker Deck] | [[Media:2015-08-13 ESIP RantRave.pdf| PDF]]
+
I had a couple major take aways from the Winter Meeting:
  
==11 June 2015: [http://www.nationaldataservice.org/projects/labs.html NDS Labs], Matt Turk==
+
*We have come a long way in IT interoperability but most of our tools are based on tried and true fundamentals. We should all know more about those fundamentals.
 +
*There are a TON of unique entry points to things that, at the end of the day, do more or less the same thing. These are opportunities to work together and share tools.
 +
*The “shiny object” is a great way to build enthusiasm and trigger ideas and we need to better capture that enthusiasm and grow some shared knowledge base.
  
Matt is a member of the NDS Labs technical advisory committee and will present NDS Labs as a platform for exploring data services -- enabling the separation of data and its representation, and how NDS Labs is functioning as an emerging platform for such separation.
+
So with that, I want to suggest three core activities:
  
===Slides===
+
#We seek out presentations that explore foundational aspects of interoperability. I want to help build an awareness of the basics that we all kind of know but either take for granted, haven’t learned yet, or straight up forgot.
[[Media:2015-06-11 ESIP RantRave NDSLabs.pdf| PDF]]
+
#We ask for speakers to explore how a given solution fits into multiple domain’s information systems and to discuss the tension between the diversity of use cases that are accommodated by an IT solution targeted at interoperability. We are especially interested to learn about the expense / risk of adopting dependencies vs the efficiency that can be gained from adopting pre-built dependencies.
 +
#We look for opportunities to take small but meaningful steps to record the core aspects of these sessions in the form of web resources like the ESIP wiki or even Wikipedia. On this front, we will aim to construct a summary wiki page from each meeting assembled from a working notes document and the presenting authors contribution.
 +
__FORCETOC__

Latest revision as of 10:30, February 19, 2024

Past Tech Dive Webinars (2015-2022)

February 15th: "Creating Great Data Products in the Cloud"

(Jed Sundwall)

IT&I Cloud Data Products 2024

Sharif Islam

Competition within the public cloud sector has reliably led to reduction in object storage costs, continual improvement in performance, and a commodification of services that have made cloud-based object storage a viable solution to share almost any volume of data. Assuming that this is true, what are the best ways to create data products in a cloud environment? This presentation will include an overview of lessons learned from Radiant Earth as they’ve advocated for adoption of cloud-native geospatial data formats and best practices.

Recording:

Minutes:

Jed is executive director of Radiant Earth – Focus is on human cooperation on a global scale.

Two major initiatives – Cloud Native Geospatial foundation and Source Cooperative

Cloud native geospatial is about adoption of efficient approaches Source is about providing easy and accessible infrastructure

What does "Cloud Native" mean? https://guide.cloudnativegeo.org/ partial reads, parallel reads, easy access to metadata

Leveraging market pressure to make object stores cheaper and more scalable.

"Pace Layering" – https://jods.mitpress.mit.edu/pub/issue3-brand/release/2

Observation: Software is getting cheaper and cheaper to build – it gets harder to create software monopolies in the way Microsoft or ESRI have.

This leads to a lot of diversity and a proliferation of "primitive" standards and defacto interoperability arrangements.

Source Cooperative

Borrowed a lot from github architecturally.

Repository with a README

Browser of contents in the browser.

Within this, what makes a great data product?

"Our data model is the Web"

People will deal with messy data if it's super valuable.

Case in point, IRS 990 data on non-profits was shared in a TON of xml schemas. People came together to organize it and work with it.

Story about a building footprint data released in the morning – had been matched up into at least four products by the end of the day.

Shout out to: https://www.geoffmulgan.com/ and https://jscaseddon.co/

https://jscaseddon.co/2024/02/science-for-steering-vs-for-decision-making/

"We don't have institutions that are tasked with producing great data products and making them available to the world!"

https://radiant.earth/blog/2023/05/we-dont-talk-about-open-data/

Meme hackathons.png

"There's a server somewhere where there's some stuff" – This is very different from a local hard drive where everything is indexed.

A cloud native approach puts the index (metadata) up front in a way that you can figure out what you need.

A file's metadata gives you the information you need to ask for just the part of a file that you actually need.

But there are other files where you don't need to do range requests. Instead, the file is broken up into many many objects that are indexed.

In both cases, the metadata is a map to the content. Figuring out the right size of the content's bits is kind of an art form.

https://www.goodreads.com/en/book/show/172366

Q: > I was thinking of your example of Warren Buffett's daily spreadsheet (gedanken experiment)... How do you see data quality or data importance (incl. data provider trustworthiness) being effectively conveyed to users?

A: We want to focus on verification of who people are and relying on reputational considerations to establish importance.

Q: > I agree with you about the importance of social factors in how people make decisions. What do you think the implications are of this for metadata for open data on the cloud?

A: Tracking data's impact and use is an important thing to keep track of. Using metadata as concrete records of observations and how it has been used is where this becomes important.

Q: > What about the really important kernels of information that we use to, say calibrate remote sensing products, that are really small but super important? How do we make sure those don't get drowned? A: We need to be careful not to overemphasize "everything is open" if we can't keep really important datasets in the spotlight.

January 11th: "Using Earth Observations for Sustainable Development"

"Using Earth Observation Technologies when Assessing Environmental, Social, Policy and Technical factors to Support Sustainable Development in Developing Countries"

Sharif Islam

Earth Observation (EO) technologies, such as satellites and remote sensing, provide a comprehensive view of the Earth's surface, enabling real-time monitoring and data acquisition. Within the environmental domain, EO facilitates tracking land use changes, deforestation, and biodiversity,   thereby   supporting   evidence-based   conservation   efforts.   Social   factors, encompassing population dynamics and urbanization trends, can be analyzed to inform inclusive and resilient development strategies. EO also assumes a crucial role in policy formulation by furnishing accurate and up-to-date information on environmental conditions, thereby supporting informed decision-making. Furthermore, technical aspects, like infrastructure development and resource management, benefit from EO's ability to provide detailed insights into terrain characteristics and natural resource distribution. The integration of Earth Observation across these domains yields a comprehensive understanding of the intricate interplay between environmental, social, policy, and technical factors, fostering a more sustainable and informed approach to development initiatives. In this presentation, I will discuss our lab's work in Bangladesh, Angola, and other countries, covering topics such as coastal erosion, drought, and air pollution.

Recording:

Minutes:

Plan to share data from NASA and USGS that was used in his PHD work.

Applied the EVDT Environment, Vulnerability, Decision Technology Framework.

Studied a variety of hazards – coastal erosion, air pollution, drought, deforestation, etc.

Coastal Erosion in Bangladesh:

  • Displacement, loss of land, major economic drain
  • Studied the situation in the Bay of Bengal
  • Used LANDSAT to study coastal erosion from the 80s to the present
  • Coastal erosion rates upwards of 300m/yr!
  • Combined survey data and landsat observations

Air Pollution and mortality in South Asia

  • Able to show change in air pollution over time using remote sensing

Drought in Angola and Brazil

Used SMAP (Soil Moisture Active Passive)

Developed the same index as the US Drought Monitor

Able to apply SMAP observations over time

Applied a social vulnerability model using these data to identify vulnerable populations.

Deforestation in Ghana

Used LANDSAT to identify land converted from forest to mining and urban.

Significant amounts of land to mining (gold mining and others)

Water hyacinth in a major fishery lake in Benin.

Impact on fishery and transportation

Rotting hyacinth is a big issue

Helped develop a DSS to guide management practices

Mangrove loss in Brazil

Combined information from economic impacts, urban plans, and remote sensing to help build a decision support tool.

November 9th: "Persistent Unique Well Identifiers: Why does California need well IDs?"

IT&I CA Wells November 2023

Hannah Ake

Groundwater is a critical resource for farms, urban and rural communities, and ecosystems in California, supplying approximately 40 percent of California's total water supply in average water years, and in some regions of the state, up to 60 percent in dry years. Regardless of water year type – some communities rely entirely on groundwater for drinking water supplies year-round. However, California lacks a uniform well identification system, which has real impacts on those who manage and depend upon groundwater. Clearly identifying wells, both existing and newly constructed, is vital to maintaining a statewide well inventory that can be more easily monitored to ensure the wellbeing of people, the environment, and the economy, while supporting the sustainable use of groundwater. A uniform well ID program has not yet been accomplished at a scale like California, but it is achievable, as evidenced by great successes in other states. Learn more about why a well ID program will be so important to tackle in California and offer your thoughts about how to untangle some of the particularly thorny technical challenges.

Recording:

Minutes:

  • Groundwater is 40-60% of California's Water supply
  • ~2 Million groundwater wells!
  • As many as 15k new wells are constructed each year

Sustainable groundwater management act frames groundwater sustainability agencies that develop groundwater sustainability plans

There is a need to account for groundwater use to ensure the plans are achieved.

Problem: There is no dedicated funding (or central coordinator) to create and maintain a statewide well inventory.

  • Department of Water Resources develops standards
  • State Water Resources Control Board has statewide ordinance
  • Cities and local districts adopt local ordinance
  • Local enforcement agency administers and enforces ordinance

There are a lot of IDs in use. 5 different identifiers can be used for the same well.

Solution: Create a well inventory that is statewide but is a compound (single id that stands in for many others) id from multiple id systems. – A meaningless identifier that links multiple others to each other.

There are a number of states with well id programs.

  • Trying to learn from what other states have done.

Going forward with some kind of identifier system that spans all local and federal identifier systems.

  • Q: Will this include federal wells? – Yes!
  • Q: Will this actually be a new well identifier minted by someone? – Yes.
  • Q: If someone drills a well do they have to register it? – Yes, but it's the local enforcing agency that collects the information.
  • Q: What if a well is deepened? Do we update the ID? – This has caused real problems in the past. We end up with multiple IDs for the same hole that go through time.
    • Seems to make sense to make a new one to keep things simple.

Link mentioned early in the talk:

https://groundwateraccounting.org/

Reference during Q&A

https://docs.ogc.org/per/20-067.html#_cerdi_vvg_selfie_demonstration

October 26th: "Improving standards and documentation publishing methods: Why can’t we cross the finish line?"

IT&I OGC October 2023

Scott Simmons

OGC and the rest of the Standards community have been promising for YEARS that our Standards and supporting documentation will be more friendly to the users that need this material the most. Progress has been made on many fronts, but why are we still not finished with a promise made in 2015 that all OGC Standards will be available in implementer-friendly views first, ugly virtual printed paper second?This topic bugs me as much as it bugs our whole community. Some of the problems are institutional (often from our Government members across the globe), others are due to lack of resources, but I think that most are due to a lack of clear reward to motivate people to do things differently.Major progress is being made in some areas. The OGC APIs have landing pages that include focused and relevant content for users/implementers and it takes some effort to find the owning Standard. OGC Developer Resources are growing quickly with sample code, running examples, and multiple views of API resources in OpenAPI, Swagger, and ReDoc.

Slides


Recording:

Minutes:

(missing first ~15 minutes of recording -- apologies)

Circa 2015 OGC GeoRabble

  • Took a critical look at the status of publishing standards.
  • Couldn't we format these specs in a kind of tutorial form?

9 years later

  • What makes this hard?
    • Standards must be unambiguous AND procurable.
    • The modular specification is a model for this balance.

Standards are based around testable requirements that relate to conformance classes.

Swaggerhub and ReDoc as a way to show a richer collection of information for multiple users.

Specification are much more modular (core and extensions)

Developer website: https://developer.ogc.org/

Going to be including persistent  demonstrator (example implementations) that are "in the wild".

https://www.ogc.org/initiatives/open-science/

Moving to an "OGC Building Blocks" model that are registered across multiple platforms and linked to lots of examples.

Building blocks are richly described and nuanced but linked back to specific requirements in a specification.

https://blocks.ogc.org/

https://sn80uo0zmbg.typeform.com/to/gcwDDNB6?typeform-source=blocks.ogc.org

A lot of this focused on APIs – what about data models?

  • Worked on APIs first because it was current. Also thinking about how to apply similar concepts to data models.

September 14th: "Water data standardization: Navigating the AntiCommons"

IT&I IoW September 2023

Kyle Onda

We all know interoperability rests on data standards and API standards. Many open standards are less prominent in the open water data space than proprietary solutions. This is because proprietary data management solutions are often bundled with very easy to use implementing software and more importantly—client software that address basic use cases. We’re giving people blueprints when they need houses. Community standards making processes should invest in end-user tools if they want to gain traction. The good news is that some of the newest generation of standards is much easier to develop around which has led to some reference implementations that are much easier to create end-user tools around than previously.

Recording:

Minutes:

AntiCommons – name comes from social science background

Tragedy of the commons - two solutions, enclose (privatize) or regulate

Tragedy of the anticommons - as opposed to common resources, these are resources that don't get used up – as in open data. Inefficiency and under utilization is common.

Two solutions. expropriation (like imminent domain or public data), incentivize

Example – consolidate urban sprawl into higher density housing to get more open space and room for business.

Introducing the Internet of Water.

Noting that in PNW, there are >800 USGS stream gages and >400 from other organizations. Only USGS are very broadly known about.

Thinking about open data as an anticommons – environmental data is normally publically available but only in ways that are convenient to data providers and the software that they use.

Discussion of the variety of standardized vs bespoke modes of data dissemination.

Example of Nebraska – GUI with download and separate custom API USGS has the same basic scheme where an ETL goes from data management software to a custom web service system.

What's going on here? Limited resources lead to focus on existing users and needs and administration ease.

Tools that meet this need tend to not focus on the needs of new user and standardization.

Most organizations don't need standards – they need software. Both server and CLIENT software.

New specs and efforts ARE heading in this direction. OGC-API, SensorThings, etc.

Promising developments around proxying non standard APIs and in use of structured data "decoration" to make documentation more standard.

August 10th: "Learning to love the upside down: Quarto and the two data science worlds"

IT&I Quarto August 10th

Carlos Scheidegger

There are two wonderful data science worlds. You can be a jupyter expert: you work on jupyter notebooks, with access to myriad Julia, Python, and R packages, and excellent technical documentation systems. You can also be a knitr and rmarkdown expert: you work on rmarkdown notebooks, with access to myriad Julia, Python, and R packages, and excellent technical documentation systems.

But what if your colleague works on the wrong side of the fence? What if you spent years learning one of them, only to find that the job you love is in an organization that uses the other? In this talk, I’m going to tell you about quarto, a system for technical communication (software documentation, academic papers, websites, etc) that aspires to let you choose any of these worlds.

If you’re one to worry about Conway’s law and what this two-worlds situation does to an organization’s talent pool, or if you live in one side of the world and want to be able to collaborate with folks on the other side, I think you’ll find something of value in what I have to say.

I’m also going to complain about software, mostly the one I write. Mostly.

Slides: https://cscheid.net/static/2023-esip-quarto-talk/

Recording:

Minutes:

Carlos was in a tenure computer science position at University of Arizona.

Hating bad software makes a software developer a good developer.

Two data science worlds:

tidyverse (with R and markdown)

  • Cohesive, hard to run things out of order.
  • Doesn't store output.

Jupyter (python and notebooks)

  • Notebook saves intermediate outputs.
  • State can be messed up easily – cells aren't linear steps.

Quarto:

  • Acts as a compatibility layer for tidyverse and jupyter ecosystems.
  • Emulates RMarkdown with multi language support.

Rant:

Quarto gets you a webpage and PDF output.

– note that the PDF requirement is not great.

Quarto is kind of just a huge wrapper around pandoc.

Quarto documentation is intractably hard to build out.

Consider Conways Law – that an organization that creates a large system will create a system that is a copy of the organization's communication structure.

– Quarto is meant to allow whole organizations with different technical tools exist in the same communication structure (same system).

Quarto tries to make kinda hard things easy while not making really hard things impossible.

Quarto can convert jupyter notebooks (with cached outputs) into markdown and vice versa.

Issue is, you need to know a variety of other languages (YAML, CSS, Javascript, LaTeX, etc.)

– "unavoidable but kinda gross"

You can edit Quarto in RStudio or VS Code, or any text editor.

For collaboration, Quarto projects can use jupyter or knitr engines. E.g. in a single web page, you can build one page with jupyter and another page with knitr.

– you can embed a ipynb cell in a notebook.

Orchestrating computation is hard – quarto has to take input from existing computation – which can be awkward / complex.

Quarto is extensible – CSS themes, OJS for interactive webpages, Pandoc extensions.

Can also write your own shortcodes.

July 13th 2023: "Tools to Assist Simulation Based Researchers in Deciding What Project Outputs to Preserve and Share"

IT&I EarthCube Model RCN July 13th

Doug Schuster

This presentation will highlight findings from the NSF EarthCube Research Coordination Network project titled “What About Model Data? - Best Practices for Preservation and Replicability” (https://modeldatarcn.github.io/), which suggest that most simulation based research projects only need to preserve and share selected model outputs, along with the full simulation experiment workflow to communicate knowledge. Challenges related to meeting community open science expectations will also be highlighted.

Slides available here: File:ModelDataRCN-2023-07-13-ESIP-IT&I v2.pdf

https://modeldatarcn.github.io/

Rubric: https://modeldatarcn.github.io/rubrics-worksheets/Descriptor-classifications-worksheet-v2.0.pdf

Open science expectations for simulation based research. Frontiers in Climate, 2021. https://doi.org/10.3389/fclim.2021.763420

Recording:

Minutes:

Primary motivation: What are data management requirements for simulation projects?

Project ran May 2020 to Jul 2022

We clearly shouldn't preserve ALL data / output from projects. It's just too expensive.

Project broke down components of data associated with a different project Forcings, code/documentation, selected outputs.

But what outputs to share?!?

Project developed a rubric of what to preserve / share.

"Is your project a data production project or a knowledge production project"

"How hard is it to rerun your workflow?"

"How much will it cost to store and serve the data?"

Rubric gives guidance on how much of a project's outputs should be preserved.

So this is all well and good, but it falls onto PIs and funding agencies.

What are the ethical and professional considerations of these trade offs?

What are the incentives in place currently? Sharing is not necessarily seen as a benefit to the author.

June 8 2023: "Reproducible Data Pipelines in Modern Data Science: what they are, how to use them, and examples you can use!"

IT&I Reproducible Pipelines June 8th

Julie Padilla

Modern scientific workflows face common challenges including accommodating growing volumes and complexity of data and the need to update analyses as new data becomes available or project needs change. The use of better practices around reproducible workflows and the use of automated data analysis pipelines can help overcome these challenges and more efficiently translate open data to actionable scientific insights. These data pipelines are transparent, reproducible, and robust to changes in the data or analysis, and therefore promote efficient, open science. In this presentation, participants will learn what makes a reproducible data pipeline and what differentiates it from a workflow as well as the key organizational concepts for effective pipeline development.

Recording:

Minutes:

Motivation – what if we find bad data in an input, what if we need to rerun something with new data, can we reproduce findings from previous work?

Need to be able to "trace" what we did and the way we do it needs to be reliable.

A "workflow" is a sequence of steps going from start to finish of some activity or process.

A "pipeline" is a programmatic implementation of a workflow that requires little to no interaction.

In a pipeline, if one workflow step or input gets changed, we can track what is "downstream" of it.

Note that different steps of the workflow may be influenced by different people. So a given step of a pipeline could be contributed by different programmers. But each person would be contributing a component of a consistent pipeline.

There is a difference between writing scripts to building a reproducible pipeline. Better to break it into steps. Script -> organize -> encapsulate into functions -> assemble pipeline.

Focus is on R targets – snakemake is equivalent in python.

Key concepts for going from script to workflow: Functions stored separate from workflow script. Steps clearly organized in script. Can wrap steps in pipeline steps to track them.

Pipeline software keeps track of whether things have changed and what needs to be rerun. Allows visualization of the workflow inputs, functions, and steps.

How do steps of the pipeline get related to eachother? They are named and the target names get passed to downstream targets.

Chat questions about branching. Dynamic branching lets you run the same target for a list of inputs in a map/reduce pattern.

Pipelines can have outputs that are reports that render pipeline results in a nice form.

Pipeline templates: A pipeline can adopt from a standard template that is pre-determined. Helps enforce best practices and have a quick and easy starting point.

Note that USGS data science has a template for a common pattern.

What's a best practice for tracking container function and reproducibility? Versioned Git / Docker for code and environment. For data, it is context dependent. Generally, try to pull down citeable / persistent sources. If sources are not persistent, you can cache inputs for later reuse / reproducibility.

Data change detection / cacheing is a really tricky thing but many people are working on the problem. https://cboettig.github.io/contentid/, https://dvc.org/

https://learning.nceas.ucsb.edu/2021-11-delta/session-3-programmatic-metadata-and-data-access.html#reproducible-data-access


11 May 2023: "Software Procurement Has Failed Us Completely, But No More!"

IT&I Software Procurement May 11th

Waldo Jaquith

The way we buy custom software is terrible for everybody involved, and has become a major obstacle to agencies achieving their missions. There are solutions, if we would just use them! By combining the standard practices of user research, Agile software development, open source, modular procurement, and time & materials contracts, we can make procurement once again serve the needs of government.

Slides available here: File:2023-05-Jaquith.pdf

Recording:

Minutes:

Recognizing that software procurement is one of the primary ways that Software / IT systems advance, Waldo went into trying to understand that space as a software developer.

Healthcare.gov

Contract given to CGI Federal for $93M – cost ~1.7B by launch.
Low single digits numbers of people actually made it through the system.
Senior leaders were given the impression that things were in good shape.
The developers working on the site knew it wasn't going to work out (per IG report).

  • strategic misrepresentation – things are represented as more rosy as you go up the chain of command

On launch, things went very badly and the recovery was actually quite quick and positive.

Waldo recommends reading the IG report on healthcare.gov. This article:
("The OIG Report Analyzing Healthcare.gov's Launch: What's There And What's Not", Health Affairs Blog, February 24, 2016. https://dx.doi.org/10.1377/hblog20160224.053370) provides a path to the IG report:
(HealthCare.gov - CMS Management of the Federal Marketplace: An OIG Case Study (OEI-06-14-00350), https://oig.hhs.gov/oei/reports/oei-06-14-00350.pdf) and additional perspective.

Rhode Island Unified Health Infrastructure

($364M to DeLoitte) "Big Bang" deployment – they let people running old systems go on the day of the new system launch. They "outsourced" a mission critical function to a contractor.

We don't tend to hear about relatively smaller projects because they are less likely to fail and garner less attention.

Outsourcing as started in the ~90s was one thing when the outsourcing was for internal agency software. It's different when the systems are actually public interfaces to stakeholders or are otherwise mission critical.

File:2023-05-Jaquith.pdf

It's common for software to meet contract requirements but NOT meet end user needs.

Requirements complexity is fractal. There is no complete / comprehensive set of requirements.

… federal contractors interpreting requirements as children trying to resist getting out the door ...

There is little to no potential to update or improve requirements due to contract structure.

Demos not memos!

Memorable statements:

  • Outsourced ability to accomplish agency’s mission
  • Load bearing software systems on which the agency depends to complete their mission.
  • Mission of many agencies is mediated by technology.

But no more! – approach developed by 18F

System of six parts –

1. User-centered design
2. Agile software development
3. Product ownership
4. DevOps
5. Building out of loosely coupled parts
6. Modular contracting

Roles for government and vendors in agile contracting

"You don't know what people need till you talk to them."

Basic premise of agile is good. Focus is on finished software being developed every two weeks.

Constantly delivering a usable product... e.g. A skateboard is more usable than a car part.

Key roles for government staff around operations are too often overlooked.

Product team needs to include an Agency Product Owner. Allows government representation in software development iteration.

Build out of loosely coupled / interchangeable components. Allows you to do smaller things and form big coherent systems that can evolve.

Modular contracts allow big projects that are delivered through many small task orders or contracts. The contract document is kind of a fill in the blank template and doesn't have to be hard.

Westrum typology of cultures article is relevant is relevant: http://dx.doi.org/10.1136/qshc.2003.009522

13 April 2023: "Evolution of open source geospatial python."

IT&I Python Open Source April 13th

Tom Kralidis

Free and Open Source Software in the Geospatial ecosystems (i.e. FOSS4G) play a key role in geospatial systems and services. Python has become the lingua franca for scientific and geospatial software and tooling. This rant and rave will provide an overview of the evolution of FOSS4G and Python, focusing on popular projects in support of Open Standards.

Slides: https://geopython.github.io/presentation

Recording:

Minutes:

Mapserver has been around for 23 years!

Why Python for Geospatial? Ubiquity Cross OS compatible Legible and easy to understand what it's doing Support ecosystem is strong (PyPI, etc.) Balance of performance and ease of implementation Python: fast enough, and fast in human time -- more intensive workloads can glue to C/C++

The new generation of OGC services – based on JSON, so the API interoperates with client environments / objects at a much more direct level.

The geopython ecosystem has a number of low level components that are used across multiple projects.

pygeoapi is an OGC API reference implementation and an OSGeo project. E.g. https://github.com/developmentseed/geojson-pydantic

pygeoapi implements OGC API - Environmental Data Retrieval (EDR) https://ogcapi.ogc.org/edr/overview.html

pygeoapi has a plugin architecture. https://pygeoapi.io/ https://code.usgs.gov/wma/nhgf/pygeoapi-plugin-cookiecutter

pycsw is an OGC CSW and OGC API - Records implementation. Works with pygeometa for metadata creation and maintenance. https://geopython.github.io/pygeometa/

There's a real trade off to "the shiny object" vs the long term sustainability of an approach. Geopython has generally erred on the side of "does it work in a virtualenv out of the box".

How does pycsw work with STAC and other catalog APIs? pycsw can convert between various representations of the same basic metadata resource.

"That's a pattern… People can implement things the way they want."

Chat Highlights:

  • You can also write a C program that is slower than Python if you aren't careful =).
  • https://www.ogc.org/standards/ has lots of useful details
  • For anyone interested in geojson API development in Python, I just recently came across this https://github.com/developmentseed/geojson-pydantic
  • OGC API - Environmental Data Retrieval (EDR) https://ogcapi.ogc.org/edr/overview.html
  • Our team has a pygeoapi plugin cookiecutter that we are hopeful others can get some mileage out of. https://code.usgs.gov/wma/nhgf/pygeoapi-plugin-cookiecutter
  • I'm going to post this here and run: https://twitter.com/GdalOrg/status/1613589544737148944
    • 100% agreed. That's unfortunate, but PyPI is not designed to deal with binary wheels of beasts like me which depend of ~ 80 direct or indirect other native libraries. Best solution or least worst solution depending on each one's view is "conda install -c conda-forge gdal"
  • General question here - you mentioned getting away from GDAL in a previous project. What are your thoughts on GDAL's role in geospatial python moving forward, and how will pygeoapi accommodate that?
  • Never, ever works with the wheels!
  • Kitware has some pre-compiled wheels as well: https://github.com/girder/large_image
  • In the pangeo.io project, our go to tools are geopandas for tabular geospatial data, xarray/rioxarray for n-dimensional array data, dask for parallelization, and holoviz for interactive visualization. We use the conda-forge channel pretty much exclusively to build out environments
  • If you work on Windows, good luck getting the Python gdal/geos-based tools installed without Conda
  • data formats and standards are what make it difficult to get away from GDAL -- it just supports so many different backends! Picking those apart and cutting legacy formats or developing more modular tools to deal with each of those things "natively" in python would be required to get away from the large dependency on something like GDAL.
  • Sustainability and maintainability is always good to ask yourself "how easy will it be to replace this dependency when it no longer works?"
  • No one should build gdal alone (unless it is winter and you need a source of heat). Join us at https://github.com/conda-forge/gdal-feedstock


9 Mar 2023: "Meeting Data Where it Lives: the power of virtual access patterns"

Mike Johnson (Lynker, NOAA-affiliate) will rant and rave about the VRT and VSI (curl and S3) virtual data access patterns and how he's used them to work with LCMAP and 3DEP data in integrated climate and data analysis workflows.

Recording:

Minutes:

  • VRT stands for "ViRTual"
  • VSI stands for "Virtual System Interface"
  • Framed by FAIR

LCMAP – requires fairly complex URLs to access specific data elements.

3DEP - need to understand tiling scheme to access data across domains.

Note some large packages (zip files) where only one small file is actually desired.

NWM datasets in NetCDF files that change name (with time step) daily as they are archived.


Implications for Findability, Availability, and Reuse – note that interoperability is actually pretty good once you have the data.

VRT: – an XML "metadata" wrapper around one or more tif files.

Use case 1: download all of 3DEP tiles and wrap in a VRT xml file.

  • VRT has an overall aggregated grid "shape"
  • Includes references to all the individual files.
  • Can access the dataset through the vrt wrapper to work across all the times.
  • Creates a seamless collection of subdatasets
  • Major improvement to accessibility.

If you have to download the data is that "reuse" of the data??

VSI: – allows virtualization of data from remote resources available as a few protocols (S3/http/compressed)

Wide variety of GDAL utilities to access VSI files – zip, tar, 7zip

Use case 2: Access a tif file remotely without downloading all the data in the file.

  • Uses vsi to access a single tif file

Use case 3: Use vsi within a vrt to remotely access contents of remote tif files.

  • Note that the vrt file doesn't actually have to be local itself.
  • If the tiles that the vrt points to update, the vrt will update by default.
  • Can easily access and reuse data without actually copying it around.

Use case 4: OGR using vsi to access a shapefile in a tar.gz file remotely.

  • Can create a nested url pattern to access contents of the tar.gz remotely.

Use case 5: NWM shortrange forecast of streamflow in a netcdf file.

  • Appending "HDF5:" to the front of a vsicurl url allows access to a netcdf file directly.
  • The access url pattern is SUPER tricky to get right.

Use case 5: "flat catalogs"

  • Stores a flat (denormalized) table of data variables with the information required to construct URLs.
  • Can search based on rudimentary metadata within the catalog.
  • Can access and reuse data from any host in the same workflow.

Use case 6: access NWM current and archived data from a variety of cloud data stores.

  • Leveraging the flat catalog content to fix up urls and data access nuances.

Flat catalog improves findability down at the level of individual data variables.

Take Aways / discussion:

Question about the flat catalog:

"Minimal set of shortcuts" to get at this fast access mechanism.

Is the flat catalog manually curated?

More or less – all are automated but some custom logic is required to add additional content.

Would be great to systematize creation of this flat catalog more broadly.

Question: Could some “examples” be posted either in this doc or elsewhere (or links to examples), for a beginner to copy/paste some code and see for themselves, begin to think about how we’d use this? Something super basic please.

GDAL documentation is good but doesn't have many examples.

climateR has a workflow that shows how the catalog was built.


What about authentication issues?

  • S3 is handled at a session level.
  • Earthengine can be handled similarly.

How much word of mouth or human-to-human interaction is required for the catalog.

  • If there is a stable entrypoint (S3 bucket for example) some automation is possible.
  • If entrypoints change, configuration needs to be changed based on human intervention.

9 Feb 2023: "February 2023 - Rants & Raves"

The conversation built on the "rants and raves" session from the 2023 January ESIP Meeting, starting with very short presentations and an in-depth discussion on interoperability and the Committee's next steps.

Recording:

Minutes:

  • Mike Mahoney: Make Reproducibility Easy
  • Dave Blodgett: FAIR data and Science Data Gateways
  • Doug Fils: Web architecture and Semantic Web
  • Megan Carter: Opening Doors for Collaboration
  • Yuhan (Douglas) Rao: Where are we for AI-ready data?

I had a couple major take aways from the Winter Meeting:

  • We have come a long way in IT interoperability but most of our tools are based on tried and true fundamentals. We should all know more about those fundamentals.
  • There are a TON of unique entry points to things that, at the end of the day, do more or less the same thing. These are opportunities to work together and share tools.
  • The “shiny object” is a great way to build enthusiasm and trigger ideas and we need to better capture that enthusiasm and grow some shared knowledge base.

So with that, I want to suggest three core activities:

  1. We seek out presentations that explore foundational aspects of interoperability. I want to help build an awareness of the basics that we all kind of know but either take for granted, haven’t learned yet, or straight up forgot.
  2. We ask for speakers to explore how a given solution fits into multiple domain’s information systems and to discuss the tension between the diversity of use cases that are accommodated by an IT solution targeted at interoperability. We are especially interested to learn about the expense / risk of adopting dependencies vs the efficiency that can be gained from adopting pre-built dependencies.
  3. We look for opportunities to take small but meaningful steps to record the core aspects of these sessions in the form of web resources like the ESIP wiki or even Wikipedia. On this front, we will aim to construct a summary wiki page from each meeting assembled from a working notes document and the presenting authors contribution.