Difference between revisions of "ESIP Cloud Computing Testbed Initiative"

From Earth Science Information Partners (ESIP)
m (1 revision imported)
 
(12 intermediate revisions by 2 users not shown)
Line 1: Line 1:
A progress report and discussion about what is needed to support a cloud computing testbed for producing lessons learned and best practice for ESIP members assess, test, and adopt cloud platforms for their research, development, and education: 1) discuss the matrix of requirements for cloud computing by ESIP members, 2) identify the possibility of cloud resources (Amazon, Microsoft, Nebula, Private Cloud), 3) identify potential applications to be running on potential cloud resources, 4) a execution plan for the testbed with a target of presenting in the summer meeting.  
+
ESIP Cloud Computing Testbed Initiative
* Intro (5 minutes) All
+
 
* Cluster overview (5 minutes and email list sign up) Phil
+
1. Intro (5 minutes) All
* Testbed intro (5 minutes) Phil
+
 
* What should the testbed be? (15 minutes) Thomas/All
+
* Phil initiated a discussion about what can CC offer to the ESIP
** virtual?
+
* Cloud Computing Cluster Overview (5 minutes and email list sign up)
** real physical infrastructure?
+
* Ken talked about launching the CC testbed task
** collaborative? how, whom, what?  
+
* Phil then went through the CC cluster Wiki Web pages.
** etc.  
+
* Academia, industry, government collaborate together on cloud computing.
* [[Cloud Computing Testbed Requirements]] (5 minutes) Phil
+
* Applications, resources and problems about CC to share are the purposes of the meeting.
* Science and Application Use Cases Ken/Phil/All
+
* Introduction about testbed CC cluster: telecon, email list.
** P&S Testbeds
+
* Collaboration on what is needed of a testbed cloud computing platform:
** OpenDAP
+
* Matrix of requirements for CC by ESIP members;
** Portals
+
* Identify the cloud resources.
** Models
+
* An execution plan for the CC testbed to be presented in the summer meeting.
** etc.
+
 
* Testbed Ideas (30 minutes) Rick/All
+
(Speaker X)
** Commercial Cloud Environment
+
Need to test applications on Nebula and EC2
** Government Cloud Environment
+
 
** ESIP Cloud Environment
+
(Long)
* Testbed Coordination (10 minutes) Rick/Thomas
+
Air processing algorithm put into Nebula cloud, Amazon cloud from Level 1 to Level 3 data. Want to move more data into cloud for processing without much increase on cost. Hope to try out more cloud platforms.
** Interoperability
+
 
** Gaps for Adaptation
+
(David)
* Milestones & Collaboration (15 minutes) Phil/All
+
Has OpenDAP experience with Amazon. Amazon uses separate system for data storage. In EC2 users run OpenDAP server.
** Platform
+
 
** Use Case Identification
+
(Peter)
** Other resources ready
+
Mention about Commonality of data volume, CC utilization.
** Collaboration
+
 
** EarthCube
+
(Mike Little)
** ESIP Summer Meeting
+
Issues need to be addressed before Nebula could be used: Ethernet interface. Latency problem is worse than Amazon.
 +
 
 +
(..)
 +
Mention about Geoportal.
 +
 
 +
(Speaker Y)
 +
CC is needed to bring the 3h time needed to process satellite images to 15 mins.
 +
 
 +
(Rick)
 +
How about building our own (community) private cloud?
 +
 
 +
(Speaker Z)
 +
Broadcast a “call for proposals” with this content:
 +
Here are some of our use cases and needs. How can we best do this using CC? Steps?, Ideas?
 +
 
 +
 
 +
(Speaker U)
 +
Need to have documentation about “best practices”, i.e., build a CC knowledge base.
 +
 
 +
 
 +
2. Testbed intro (5 minutes)  
 +
 
 +
3. Science and Application Use Cases (20 minutes) Ken/Phil/All
 +
(Ken)
 +
* ESIP testbed goes towards cloud initial effort:
 +
o make CC one of testbed efforts.
 +
o Testbed to help answer questions about CC.
 +
o Have a testbed cloud infrastructure, provide an environment for new testbed users to leverage, to facilitate new testbed missions.
 +
o Provide IaaS to benefit ESIP members.
 +
o Initially five ESIP tasks can be put on the CC testbed.
 +
o Capability to interactively enable users to request resources for hosting applications.
 +
o IaaS eases hardware requirements of new applications.
 +
 
 +
 
 +
4. Testbed Ideas (10 minutes) Rick/All
 +
(Rick)
 +
* Whether we need more than one environment for testbed?
 +
* Can we get resources?
 +
* Commercial? Self-developed?
 +
* The feasibility of options?
 +
* Is there benefit for government-owned CC?
 +
* Government owned: whether or not to ask for, who will pay for? Is it feasible to go to government agencies for free computing resources?
 +
Who and are they willing to offer?
 +
 
 +
(Long)
 +
* Maintaining is more important. Getting resources may be possible.
 +
* Commercial Cloud Environment
 +
* Government Cloud Environment
 +
* Community Cloud Environment
 +
 
 +
(Rick)
 +
Move on to commercial.
 +
 
 +
(Dave)
 +
Microsoft platform worked for OpenDAP applications.
 +
 
 +
(Rick)
 +
Offerings from commercial companies are available, do we think it is good idea?
 +
 
 +
(Ken)
 +
Would it be possible to consider hosting the testbed tasks by commercial companies.
 +
 
 +
(..)
 +
To evaluate testbed against commercial and Nebula CC if it’s desirable to have own CC infrastructure.
 +
 
 +
(..)
 +
The size of the node change options: hundreds nodes vs ten thousand nodes require a different scale human resources.
 +
 
 +
(..)
 +
Commercial options might be good for large cloud computing size for skills and cost effectiveness.
 +
 
 +
Whether the community desires to pursue community built cloud? It takes time to put resources in place.
 +
 
 +
(Phil)
 +
Keep working on involving industry providers for ESIP community.
 +
 
 +
(Thomas)
 +
How big CC will be? Number of nodes?
 +
The size of the CC testbed?
 +
 
 +
(Phil)
 +
The purpose is for testing, learning, gaining experience.
 +
 
 +
(..)
 +
What does CC provide?
 +
 
 +
(Emily)
 +
Testbed is for testing purposes, not for operational (in response to the issue of the feasibility of CC testbed to support science research computing).
 +
 
 +
(Thomas)
 +
Requirements of testbed CC? Expectations from the community.
 +
 
 +
(Ken)
 +
Proposal for testbed CC application.
 +
 
 +
(..)
 +
Use testbed platform to test environment for scaling up applications.
 +
 
 +
(Phil)
 +
Which applications? What requirements?
 +
 
 +
(Ken)
 +
Up to now, discussion about resources donated to testbed CC.
 +
 
 +
(..)
 +
Differentiate testbed CC from commercial, existing CC platform? Why do we need to?
 +
 
 +
(..)
 +
Testbed is a platform for gaining experience and knowledge.
 +
 
 +
(Phil)
 +
One possibility is leveraging testbed CC to produce OpenDAP server image for deploying to other platforms.
 +
 
 +
5. What should the testbed be? (15 minutes) Thomas/All
 +
 
 +
▪ virtual?
 +
 
 +
▪ real physical infrastructure?
 +
 
 +
 
 +
▪ collaborative? how, whom, what?
 +
 
 +
▪ etc.
 +
 
 +
 
 +
6. Testbed Coordination (10 minutes) Rick/Thomas
 +
 
 +
Interoperability
 +
 
 +
Gaps for Adaptation
 +
 
 +
 
 +
7. Cloud Computing Testbed Requirements (5 minutes) Phil
 +
 
 +
 
 +
8. Milestones & Collaboration & Products & Services (15 minutes) Phil/All
 +
 
 +
* Talk to commercial and government for CC resources.
 +
 
 +
* One possible hard deadline is in May to demo on EarthCube, or ESIP summer meeting.
 +
 
 +
* Provide well configured images for ESIP members.
 +
 
 +
* Documents of best practices are beneficial.
 +
 
 +
* Best use cases
 +
 
 +
* Possibility of CC testbed for Big data and other challenges.
 +
 
 +
 
 +
9. ESIP Summer Meeting

Latest revision as of 09:53, October 8, 2021

ESIP Cloud Computing Testbed Initiative

1. Intro (5 minutes) All

  • Phil initiated a discussion about what can CC offer to the ESIP
  • Cloud Computing Cluster Overview (5 minutes and email list sign up)
  • Ken talked about launching the CC testbed task
  • Phil then went through the CC cluster Wiki Web pages.
  • Academia, industry, government collaborate together on cloud computing.
  • Applications, resources and problems about CC to share are the purposes of the meeting.
  • Introduction about testbed CC cluster: telecon, email list.
  • Collaboration on what is needed of a testbed cloud computing platform:
  • Matrix of requirements for CC by ESIP members;
  • Identify the cloud resources.
  • An execution plan for the CC testbed to be presented in the summer meeting.

(Speaker X) Need to test applications on Nebula and EC2

(Long) Air processing algorithm put into Nebula cloud, Amazon cloud from Level 1 to Level 3 data. Want to move more data into cloud for processing without much increase on cost. Hope to try out more cloud platforms.

(David) Has OpenDAP experience with Amazon. Amazon uses separate system for data storage. In EC2 users run OpenDAP server.

(Peter) Mention about Commonality of data volume, CC utilization.

(Mike Little) Issues need to be addressed before Nebula could be used: Ethernet interface. Latency problem is worse than Amazon.

(..) Mention about Geoportal.

(Speaker Y) CC is needed to bring the 3h time needed to process satellite images to 15 mins.

(Rick) How about building our own (community) private cloud?

(Speaker Z) Broadcast a “call for proposals” with this content: Here are some of our use cases and needs. How can we best do this using CC? Steps?, Ideas?


(Speaker U) Need to have documentation about “best practices”, i.e., build a CC knowledge base.


2. Testbed intro (5 minutes)

3. Science and Application Use Cases (20 minutes) Ken/Phil/All (Ken)

  • ESIP testbed goes towards cloud initial effort:

o make CC one of testbed efforts. o Testbed to help answer questions about CC. o Have a testbed cloud infrastructure, provide an environment for new testbed users to leverage, to facilitate new testbed missions. o Provide IaaS to benefit ESIP members. o Initially five ESIP tasks can be put on the CC testbed. o Capability to interactively enable users to request resources for hosting applications. o IaaS eases hardware requirements of new applications.


4. Testbed Ideas (10 minutes) Rick/All (Rick)

  • Whether we need more than one environment for testbed?
  • Can we get resources?
  • Commercial? Self-developed?
  • The feasibility of options?
  • Is there benefit for government-owned CC?
  • Government owned: whether or not to ask for, who will pay for? Is it feasible to go to government agencies for free computing resources?

Who and are they willing to offer?

(Long)

  • Maintaining is more important. Getting resources may be possible.
  • Commercial Cloud Environment
  • Government Cloud Environment
  • Community Cloud Environment

(Rick) Move on to commercial.

(Dave) Microsoft platform worked for OpenDAP applications.

(Rick) Offerings from commercial companies are available, do we think it is good idea?

(Ken) Would it be possible to consider hosting the testbed tasks by commercial companies.

(..) To evaluate testbed against commercial and Nebula CC if it’s desirable to have own CC infrastructure.

(..) The size of the node change options: hundreds nodes vs ten thousand nodes require a different scale human resources.

(..) Commercial options might be good for large cloud computing size for skills and cost effectiveness.

Whether the community desires to pursue community built cloud? It takes time to put resources in place.

(Phil) Keep working on involving industry providers for ESIP community.

(Thomas) How big CC will be? Number of nodes? The size of the CC testbed?

(Phil) The purpose is for testing, learning, gaining experience.

(..) What does CC provide?

(Emily) Testbed is for testing purposes, not for operational (in response to the issue of the feasibility of CC testbed to support science research computing).

(Thomas) Requirements of testbed CC? Expectations from the community.

(Ken) Proposal for testbed CC application.

(..) Use testbed platform to test environment for scaling up applications.

(Phil) Which applications? What requirements?

(Ken) Up to now, discussion about resources donated to testbed CC.

(..) Differentiate testbed CC from commercial, existing CC platform? Why do we need to?

(..) Testbed is a platform for gaining experience and knowledge.

(Phil) One possibility is leveraging testbed CC to produce OpenDAP server image for deploying to other platforms.

5. What should the testbed be? (15 minutes) Thomas/All

▪ virtual?

▪ real physical infrastructure?


▪ collaborative? how, whom, what?

▪ etc.


6. Testbed Coordination (10 minutes) Rick/Thomas

▪ Interoperability

▪ Gaps for Adaptation


7. Cloud Computing Testbed Requirements (5 minutes) Phil


8. Milestones & Collaboration & Products & Services (15 minutes) Phil/All

  • Talk to commercial and government for CC resources.
  • One possible hard deadline is in May to demo on EarthCube, or ESIP summer meeting.
  • Provide well configured images for ESIP members.
  • Documents of best practices are beneficial.
  • Best use cases
  • Possibility of CC testbed for Big data and other challenges.


9. ESIP Summer Meeting