Merge branch 'main' into 31-file-part3advanced_ufunc

pangeo-data · Nov 6, 2023 · 722cbe5 · 722cbe5
2 parents 784ab2a + d944e11
commit 722cbe5
Show file tree

Hide file tree

Showing 25 changed files with 2,535 additions and 104 deletions.
diff --git a/.all-contributorsrc b/.all-contributorsrc
@@ -50,7 +50,7 @@
       "login": "jdries",
       "name": "Jeroen Dries",
       "avatar_url": "https://avatars.githubusercontent.com/u/5937096?v=4",
-      "profile": "https://remotesensing.vito.be/",
+      "profile": "https://remotesensing.vito.be/team/jeroen-dries",
       "contributions": [
         "ideas",
         "design",
@@ -258,6 +258,20 @@
         "doc",
         "tutorial"
       ]
+    },
+    {
+      "login": "dthiex",
+      "name": "Daniel Thiex",
+      "avatar_url": "https://avatars.githubusercontent.com/u/60705209?v=4",
+      "profile": "https://github.com/dthiex",
+      "contributions": [
+        "ideas",
+        "design",
+        "code",
+        "content",
+        "doc",
+        "tutorial"
+      ]
     }
   ],
   "contributorsPerLine": 7,

diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # Unlocking the Power of Space Data with Pangeo & OpenEO
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-20-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-21-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 This repository contains the documentation and jupyter notebooks used for delivering tutorial to BiDS'23.
@@ -19,9 +19,9 @@ The programmes for each workshop are given below for your information. Each work
 | Time | Activity | 
 | ---- | -------- | 
 | 09:00  | 👋 Welcome  (5 minutes) | 
-| 9:05   | Introduction and Motivation (15 minutes) |
-| 9:20   | Overview of the Pangeo ecosystem (20 minutes) |
-| 9:40   | Understanding Xarray to avoid common pitfalls (30 minutes) |
+| 09:05   | Introduction and Motivation (15 minutes) |
+| 09:20   | Overview of the Pangeo ecosystem (20 minutes) |
+| 09:40   | Understanding Xarray to avoid common pitfalls (30 minutes) |
 | 10:10  | Interactive Visualization with Hvplot (15 minutes) |
 | 10:30  | Break (30 minutes)  | |
 
@@ -32,8 +32,8 @@ The programmes for each workshop are given below for your information. Each work
 | 11:00 | 👋 Introduction and motivation (5 minutes) | 
 | 11:05 | Getting started with OpenEO (10 minutes) | 
 | 11:15 | Accessing and processing data with OpenEO (30 minutes) | 
-| 11:45 | ntegrate custom code into your workflow using User Defined Functions (30 minutes) | 
-| 12:15 | Q&A session - feedbacks (20 minutes) | 
+| 11:45 | Integrate custom code into your workflow using User Defined Functions (30 minutes) | 
+| 12:15 | Q&A session - feedbacks (15 minutes) | 
 | 12:30 | 🍽️ Lunch | 
 
 ## Unlocking the Power of Space Data with Pangeo & OpenEO
@@ -75,20 +75,21 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/aljacob"><img src="https://avatars.githubusercontent.com/u/7644251?v=4?s=100" width="100px;" alt="Alexander Jacob"/><br /><sub><b>Alexander Jacob</b></sub></a><br /><a href="#ideas-aljacob" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-aljacob" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=aljacob" title="Code">💻</a> <a href="#content-aljacob" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=aljacob" title="Documentation">📖</a> <a href="#tutorial-aljacob" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/aminraz"><img src="https://avatars.githubusercontent.com/u/48209586?v=4?s=100" width="100px;" alt="Amin Razbani"/><br /><sub><b>Amin Razbani</b></sub></a><br /><a href="#ideas-aminraz" title="Ideas, Planning, & Feedback">🤔</a> <a href="#userTesting-aminraz" title="User Testing">📓</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://www.linkedin.com/in/annefouilloux/"><img src="https://avatars.githubusercontent.com/u/8168508?v=4?s=100" width="100px;" alt="Anne Fouilloux"/><br /><sub><b>Anne Fouilloux</b></sub></a><br /><a href="#ideas-annefou" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-annefou" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=annefou" title="Code">💻</a> <a href="#content-annefou" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=annefou" title="Documentation">📖</a> <a href="#tutorial-annefou" title="Tutorials">✅</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/dthiex"><img src="https://avatars.githubusercontent.com/u/60705209?v=4?s=100" width="100px;" alt="Daniel Thiex"/><br /><sub><b>Daniel Thiex</b></sub></a><br /><a href="#ideas-dthiex" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-dthiex" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=dthiex" title="Code">💻</a> <a href="#content-dthiex" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=dthiex" title="Documentation">📖</a> <a href="#tutorial-dthiex" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/guillaumeeb"><img src="https://avatars.githubusercontent.com/u/17138587?v=4?s=100" width="100px;" alt="Guillaume Eynard-Bontemps"/><br /><sub><b>Guillaume Eynard-Bontemps</b></sub></a><br /><a href="#ideas-guillaumeeb" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-guillaumeeb" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=guillaumeeb" title="Code">💻</a> <a href="#content-guillaumeeb" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=guillaumeeb" title="Documentation">📖</a> <a href="#tutorial-guillaumeeb" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://remotesensing.vito.be/"><img src="https://avatars.githubusercontent.com/u/5937096?v=4?s=100" width="100px;" alt="Jeroen Dries"/><br /><sub><b>Jeroen Dries</b></sub></a><br /><a href="#ideas-jdries" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-jdries" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=jdries" title="Code">💻</a> <a href="#content-jdries" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=jdries" title="Documentation">📖</a> <a href="#tutorial-jdries" title="Tutorials">✅</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://github.com/keewis"><img src="https://avatars.githubusercontent.com/u/14808389?v=4?s=100" width="100px;" alt="Justus Magin"/><br /><sub><b>Justus Magin</b></sub></a><br /><a href="#ideas-keewis" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-keewis" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=keewis" title="Code">💻</a> <a href="#content-keewis" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=keewis" title="Documentation">📖</a> <a href="#tutorial-keewis" title="Tutorials">✅</a></td>
     </tr>
     <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/keewis"><img src="https://avatars.githubusercontent.com/u/14808389?v=4?s=100" width="100px;" alt="Justus Magin"/><br /><sub><b>Justus Magin</b></sub></a><br /><a href="#ideas-keewis" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-keewis" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=keewis" title="Code">💻</a> <a href="#content-keewis" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=keewis" title="Documentation">📖</a> <a href="#tutorial-keewis" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://mohr.ws"><img src="https://avatars.githubusercontent.com/u/8262166?v=4?s=100" width="100px;" alt="Matthias Mohr"/><br /><sub><b>Matthias Mohr</b></sub></a><br /><a href="#ideas-m-mohr" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-m-mohr" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=m-mohr" title="Code">💻</a> <a href="#content-m-mohr" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=m-mohr" title="Documentation">📖</a> <a href="#tutorial-m-mohr" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://www.linkedin.com/in/micheleclaus/"><img src="https://avatars.githubusercontent.com/u/31700619?v=4?s=100" width="100px;" alt="Michele Claus"/><br /><sub><b>Michele Claus</b></sub></a><br /><a href="#ideas-clausmichele" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-clausmichele" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=clausmichele" title="Code">💻</a> <a href="#content-clausmichele" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=clausmichele" title="Documentation">📖</a> <a href="#tutorial-clausmichele" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/micafer"><img src="https://avatars.githubusercontent.com/u/4972699?v=4?s=100" width="100px;" alt="Miguel Caballer Fernandez"/><br /><sub><b>Miguel Caballer Fernandez</b></sub></a><br /><a href="#infra-micafer" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#platform-micafer" title="Packaging/porting to new platform">📦</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/ofk123"><img src="https://avatars.githubusercontent.com/u/98769588?v=4?s=100" width="100px;" alt="Ola"/><br /><sub><b>Ola</b></sub></a><br /><a href="#ideas-ofk123" title="Ideas, Planning, & Feedback">🤔</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://pratichhya.github.io/"><img src="https://avatars.githubusercontent.com/u/39898768?v=4?s=100" width="100px;" alt="Pratichhya "/><br /><sub><b>Pratichhya </b></sub></a><br /><a href="#ideas-Pratichhya" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-Pratichhya" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=Pratichhya" title="Code">💻</a> <a href="#content-Pratichhya" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=Pratichhya" title="Documentation">📖</a> <a href="#tutorial-Pratichhya" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://uk.linkedin.com/in/sebastianlunavalero/en"><img src="https://avatars.githubusercontent.com/u/5345517?v=4?s=100" width="100px;" alt="Sebastian Luna-Valero"/><br /><sub><b>Sebastian Luna-Valero</b></sub></a><br /><a href="#infra-sebastian-luna-valero" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#platform-sebastian-luna-valero" title="Packaging/porting to new platform">📦</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="http://stefaanlippens.net"><img src="https://avatars.githubusercontent.com/u/44946?v=4?s=100" width="100px;" alt="Stefaan Lippens"/><br /><sub><b>Stefaan Lippens</b></sub></a><br /><a href="#ideas-soxofaan" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-soxofaan" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=soxofaan" title="Code">💻</a> <a href="#content-soxofaan" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=soxofaan" title="Documentation">📖</a> <a href="#tutorial-soxofaan" title="Tutorials">✅</a></td>
     </tr>
     <tr>
+      <td align="center" valign="top" width="14.28%"><a href="http://stefaanlippens.net"><img src="https://avatars.githubusercontent.com/u/44946?v=4?s=100" width="100px;" alt="Stefaan Lippens"/><br /><sub><b>Stefaan Lippens</b></sub></a><br /><a href="#ideas-soxofaan" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-soxofaan" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=soxofaan" title="Code">💻</a> <a href="#content-soxofaan" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=soxofaan" title="Documentation">📖</a> <a href="#tutorial-soxofaan" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/tinaok"><img src="https://avatars.githubusercontent.com/u/46813815?v=4?s=100" width="100px;" alt="Tina Odaka"/><br /><sub><b>Tina Odaka</b></sub></a><br /><a href="#ideas-tinaok" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-tinaok" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=tinaok" title="Code">💻</a> <a href="#content-tinaok" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=tinaok" title="Documentation">📖</a> <a href="#tutorial-tinaok" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/ValentinaHutter"><img src="https://avatars.githubusercontent.com/u/85164505?v=4?s=100" width="100px;" alt="ValentinaHutter"/><br /><sub><b>ValentinaHutter</b></sub></a><br /><a href="#ideas-ValentinaHutter" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-ValentinaHutter" title="Design">🎨</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=ValentinaHutter" title="Code">💻</a> <a href="#content-ValentinaHutter" title="Content">🖋</a> <a href="https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/commits?author=ValentinaHutter" title="Documentation">📖</a> <a href="#tutorial-ValentinaHutter" title="Tutorials">✅</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/sustr4"><img src="https://avatars.githubusercontent.com/u/1583737?v=4?s=100" width="100px;" alt="Zdeněk Šustr"/><br /><sub><b>Zdeněk Šustr</b></sub></a><br /><a href="#infra-sustr4" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#platform-sustr4" title="Packaging/porting to new platform">📦</a></td>

diff --git a/tutorial/_config.yml b/tutorial/_config.yml
@@ -23,7 +23,7 @@ execute:
 launch_buttons:
   notebook_interface: "jupyterlab"
   binderhub_url: "https://notebooks.gesis.org/binder/"  # The URL for your BinderHub (e.g., https://mybinder.org)
-  jupyterhub_url: "https://pangeo-foss4g.vm.fedcloud.eu/jupyterhub/"  # The URL for your JupyterHub. (e.g., https://datahub.berkeley.edu)
+  jupyterhub_url: "http://pangeo-eosc.vm.fedcloud.eu/jupyterhub/"  # The URL for your JupyterHub. (e.g., https://datahub.berkeley.edu)
 
 # Define the name of the latex output file for PDF builds
 latex:

diff --git a/tutorial/_toc.yml b/tutorial/_toc.yml
@@ -29,6 +29,8 @@ parts:
 
 - caption: Part 2 - OpenEO
   chapters:
+  - file: part2/openEO-getting_started
+    title: Getting started
   - file: part2/openEO-boa_sentinel_2
     title: Sentinel-2 Bottom Of Atmosphere (BOA)
   - file: part2/openEO-Corine_Land_Cover_Alps
@@ -54,11 +56,16 @@ parts:
     title: How to exploit data with openEO
   - file: part3/data_exploitability_pangeo
     title: How to exploit data on Pangeo
+  - file: part3/udf_intro_openeo
+    title: openEO UDF - Introduction
+  - file: part3/peak_valley
+    title: Advanced openEO UDF + ufunc (XArray) - Peak Valley Case
   - file: part3/advanced_udf
-    title: Custom algorithms - UDF (OpenEO) and ufunc (Xarray)
+    title: Advanced openEO UDF - Machine Learning
   - file: part3/scaling_openeo
     title: Scaling with OpenEO
 
+
 - caption: Beyond the workshop
   chapters:
   - file: afterword/resources

diff --git a/tutorial/before/copernicus.md b/tutorial/before/copernicus.md
@@ -1,6 +1,10 @@
 # Copernicus Data Space Ecosystem
 
-[The Copernicus Data Space Ecosystem](https://dataspace.copernicus.eu) is a data infrastructure for accessing and analyzing Sentinel satellite imagery. Most of the Pangeo & OpenEO training materials can be executed using the Jupyter Notebooks in the Copernicus Dataspace ecosystem.
+[The Copernicus Data Space Ecosystem](https://dataspace.copernicus.eu) is the public service that provides access to the
+full archives of Copernicus Sentinel data. 
+Next to data access, it also offers services for analyzing satellite imagery.
+Most of the Pangeo & openEO training materials can be executed using the Jupyter Notebooks 
+in the Copernicus Dataspace ecosystem.
 
 ## Registration
 
@@ -23,4 +27,10 @@ Then you can choose among the 3 available flavors (as shown on the figure below)
 
 For the tutorial, we suggest you use the "Large Server".
 
+## Getting access to openEO in Copernicus data space
+
+The openEO endpoint for Copernicus is documented here:
+https://documentation.dataspace.copernicus.eu/APIs/openEO/openEO.html
+
+The endpoint you can use in your scripts is: `openeo.dataspace.copernicus.eu` .
 
diff --git a/tutorial/before/setup.md b/tutorial/before/setup.md
@@ -4,11 +4,11 @@ The setup for the tutorials will be described below.
 
 # Setup: how to run the tutorial
 
-This tutorial's goal is to provide a wide introduction to the Open EO and Pangeo ecosystem. 
-While participants will see all the core libraries and possibilities offered by the OpenEO and Pangeo software ecosystem, it is also important that they get some insights about what a OpenEO and Pangeo platform is and how to use it. 
+This tutorial's goal is to provide a wide introduction to the openEO and Pangeo ecosystem. 
+While participants will see all the core libraries and possibilities offered by the openEO and Pangeo software ecosystem, it is also important that they get some insights about what a openEO and Pangeo platform is and how to use it. 
 
 
-We recommend running this tutorial using the OpenEO, CDEC and Pangeo-EOSC infrastructure.  We recommend users to create logins and try to connect them before the tutorial.   
+We recommend running this tutorial using the openEO, CDEC and Pangeo-EOSC infrastructure.  We recommend users to create logins and try to connect them before the tutorial.   
 Creating account and how to run the turorial no these infrastructure are detailed in next pages.  
 
 This page will describe the ways of running the tutorial:

diff --git a/tutorial/data/s2_meadow.nc b/tutorial/data/s2_meadow.nc
diff --git a/tutorial/figures/openeo_networkio.png b/tutorial/figures/openeo_networkio.png
diff --git a/tutorial/figures/sampling_openeo.png b/tutorial/figures/sampling_openeo.png
diff --git a/tutorial/part1/overview.md b/tutorial/part1/overview.md
@@ -1,12 +1,20 @@
 # Overview of the Pangeo ecosystem
 
+- You can find a short introduction to Pangeo in the slides below (also available in [Zenodo](https://doi.org/10.5281/zenodo.10070519)).
+
+<style>
+.responsive-wrap iframe{ max-width: 100%;}
+</style>
+<div class="responsive-wrap">
+<!-- this is the embed code provided by Google -->
+  <iframe src="https://docs.google.com/presentation/d/1-GRd3OivgD723XYUe_4DRzW1hSpYAJ6H33ytlzVezrI/embed?start=false&loop=false&delayms=3000" frameborder="0" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>
+<!-- Google embed ends -->
+</div>
 
-- You can find a short introduction to Pangeo [here](https://doi.org/10.5281/zenodo.10070519)
 - Please follow the explanation at [https://pangeo-data.github.io/pangeo-openeo-BiDS-2023/before/EOSC.html](https://pangeo-data.github.io/pangeo-openeo-BiDS-2023/before/EOSC.html) and register yourself to Pangeo-EOSC cluster, and connect to Pangeo-EOSC cluster.  
 
 - Once you are connected, please open a terminal, and type:
 
-
 ```
 git clone https://github.com/pangeo-data/pangeo-openeo-BiDS-2023.git
 ```

diff --git a/tutorial/part1/visualization.ipynb b/tutorial/part1/visualization.ipynb
@@ -17,7 +17,7 @@
     "\n",
     "### Authors\n",
     "\n",
-    "- Pier Lorenzo Marasco, Ispra (Italy), [@pl-marasco](https://github.com/pl-marasco)\n",
+    "- Pier Lorenzo Marasco, Provare LTD (UK), [@pl-marasco](https://github.com/pl-marasco)\n",
     "\n",
     "### Contributors\n",
     "\n",

diff --git a/tutorial/part1/xarray_pitfalls.ipynb b/tutorial/part1/xarray_pitfalls.ipynb
@@ -17,7 +17,7 @@
     "\n",
     "### Authors\n",
     "\n",
-    "- Pier Lorenzo Marasco, Ispra (Italy), [@pl-marasco](https://github.com/pl-marasco)\n",
+    "- Pier Lorenzo Marasco, Provare LTD (UK), [@pl-marasco](https://github.com/pl-marasco)\n",
     "\n",
     "### Contributors\n",
     "\n",