Merge pull request #79 from AccelerationConsortium/update-1.5

Update 1.5

docs/course-data.yaml

@@ -499,6 +499,7 @@ robotics:
         - "Perform liquid transfer between vials with an automated liquid handler (Science Jubilee and OT-2)"
 
     - module_name: "3.4 Mobile robotics"
+      tutorial_file: 3.4-mobile-robotics.md
       topics:
         - "ROS"
         - "Isaac Sim"
@@ -510,6 +511,7 @@ robotics:
         - "Define asynchrony in the context of hardware control for automonous laboratories"
 
     - module_name: "3.5 Computer vision"
+      tutorial_file: 3.5-computer-vision.md
       topics:
         - "OpenCV"
         - "AprilTags"
@@ -518,6 +520,7 @@ robotics:
         - "Use a motorized microscope and OpenCV to search for regions of interest in a sample"
 
     - module_name: "3.6 Solid sample transfer"
+      tutorial_file: 3.6-solid-sample-transfer.md
       topics:
         - "Workflow orchestration"
         - "ROS"

docs/courses/hello-world/1.5-data-logging.md

@@ -27,92 +27,181 @@ Chemistry and materials research data is precious. By making your data **F**inda
 
 *Defining FAIR Data in materials science. Reproduced from https://doi.org/10.1557/s43577-023-00498-4*
 
-### Writing to MongoDB Using the Data API
+### # Updating Pico W to Use AWS Lambda for MongoDB Interaction
 
-```{warning}
-The Data API is now deprecated (as of 2024-09-10). While the end of life for the Data API is 2025-09-30, the Data API can no longer be enabled for users/database for which it's not already enabled. If you already have it enabled for a particular instance, it should work until the end-of-life date. See https://github.com/AccelerationConsortium/ac-microcourses/issues/45 for context. We are working quickly to adjust the tutorials and assignments based on this change. This also affects the final module.
-```
+For storing our data, we will be using [MongoDB](https://www.mongodb.com/) as before, but instead of using MongoDB's Data API directly, we'll be using [AWS Lambda](https://aws.amazon.com/lambda/) as an intermediary. AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. It's an excellent choice for creating lightweight APIs and handling intermittent requests, which is perfect for our IoT application.
 
-For storing our data, we will be using [MongoDB](https://www.mongodb.com/), a popular ["NoSQL"](https://www.mongodb.com/nosql-explained) database. It's important to note that MongoDB is just one of many excellent choices for databases. It's a document-oriented database, which means it stores data in JSON-like documents. MongoDB is a popular choice for internet-of-things (IoT) applications due to its ease of setup, use, and scalability. Additionally, MongoDB offers a [Data API](https://docs.atlas.mongodb.com/data-api/) that allows for direct reading and writing of data from a microcontroller.
+By using AWS Lambda in conjunction with [Amazon API Gateway](https://aws.amazon.com/api-gateway/), we can create a secure and scalable API that acts as a bridge between our Pico W and MongoDB. This approach offers several advantages:
 
-For the purposes of this tutorial, we have set up a free-tier test database through MongoDB Atlas. We provide an API key for test purposes. To prevent potential misuse from distributing a public API key (which is generally not a good practice), we have granted only write permissions, and the database is configured to automatically delete entries once a certain storage threshold is reached.
+1. Enhanced security: We can implement more robust authentication and authorization mechanisms.
+2. Flexibility: We can easily add data processing or validation logic before writing to MongoDB.
+3. Cost-effective: Lambda's pay-per-use model can be more economical for intermittent data logging.
 
-✅ Copy the following code into a new file on the microcontroller called `write_mongodb.py` and run the file. Note that you will need [`netman.py`](https://github.com/sparks-baird/self-driving-lab-demo/blob/main/src/public_mqtt_sdl_demo/lib/netman.py) [[permalink](https://github.com/sparks-baird/self-driving-lab-demo/blob/0ff0adec3e997c096990de594844d73a9ce18fd6/src/public_mqtt_sdl_demo/lib/netman.py)] and a file named `my_secrets.py` with your WiFi credentials and course ID.
+For this tutorial, we've set up a Lambda function that connects to our MongoDB database. The function is triggered via an API Gateway endpoint, which the Pico W will call to log data. To maintain security, the API Gateway is configured with an API key, and the Lambda function has the necessary permissions to write to MongoDB.
 
+✅ Copy the following code into a new file on the microcontroller called `write_aws_lambda.py` and run the file. Note that you will need [`netman.py`](https://github.com/sparks-baird/self-driving-lab-demo/blob/main/src/public_mqtt_sdl_demo/lib/netman.py) [[permalink](https://github.com/sparks-baird/self-driving-lab-demo/blob/0ff0adec3e997c096990de594844d73a9ce18fd6/src/public_mqtt_sdl_demo/lib/netman.py)] and a file named `my_secrets.py` with your WiFi credentials, course ID, and the AWS API Gateway URL and API key.
 
-```python
-# based on https://medium.com/@johnlpage/introduction-to-microcontrollers-and-the-pi-pico-w-f7a2d9ad1394
-from netman import connectWiFi
-import urequests
+This guide explains how to modify the Pico W code to use AWS Lambda as a data API interface for MongoDB, instead of using MongoDB's Data API directly.
 
-from my_secrets import SSID, PASSWORD, COURSE_ID
+## Setting up AWS Lambda and API Gateway
 
-connectWiFi(SSID, PASSWORD, country="US")
+## Required Reading Materials
 
-DATA_API_KEY = "UT4cdinBetBaNqCBc5hISkaArhllv5dWfzXgbYsLYzpv79nqNhVwVsudQU5ZUmBE"  # Public API key for demo purposes only
-CLUSTER_NAME = "test-cluster"
-DATABASE_NAME = "test-db"
-COLLECTION_NAME = "write-to-me"
+- [AWS Lambda Documentation](https://docs.aws.amazon.com/lambda/)
+- [AWS API Gateway Documentation](https://docs.aws.amazon.com/apigateway/)
+- [Raspberry Pi Pico W Documentation](https://www.raspberrypi.com/documentation/microcontrollers/raspberry-pi-pico.html)
 
-ENDPOINT_BASE_URL = (
-    "https://us-east-2.aws.data.mongodb-api.com/app/data-ibmqs/endpoint/data/v1"
-)
+## Required Python Files
 
-endpoint_url = f"{ENDPOINT_BASE_URL}/action/insertOne"
+1. `main.py`: Contains the modified code above
+2. `netman.py`: Module for WiFi connection (remains unchanged)
+3. `my_secrets.py`: Contains WiFi credentials and COURSE_ID
 
-headers = {"api-key": DATA_API_KEY}
-document = {"course_id": COURSE_ID}
+## Setting up AWS Lambda and API Gateway for MongoDB Integration
+
+Follow these steps to set up AWS Lambda and API Gateway to interact with MongoDB:
+
+1. Create an AWS Lambda function:
+   - Go to AWS Lambda console and click "Create function".
+   - Choose "Author from scratch" and give your function a name.
+   - Select Python as the runtime.
+   - Create a new role with basic Lambda permissions.
 
-payload = {
-    "dataSource": CLUSTER_NAME,
-    "database": DATABASE_NAME,
-    "collection": COLLECTION_NAME,
-    "document": document,
-}
+2. Set up the Lambda function:
+   - In the function code, paste the Python code provided in the "Lambda Function Code" section below.
+   - Add the `pymongo` layer to your Lambda function for MongoDB connectivity.
+   - In the function configuration, add an environment variable for your MongoDB connection string.
 
-print(f"sending document to {CLUSTER_NAME}:{DATABASE_NAME}:{COLLECTION_NAME}")
+3. Create an API Gateway:
+   - Go to API Gateway console and create a new REST API.
+   - Create a new resource and add a POST method.
+   - Set the integration type to "Lambda Function" and select your created function.
+
+4. Set up API Key:
+   - In API Gateway, create a new API key.
+   - Associate this key with your API and deployed stage.
+
+5. Deploy the API:
+   - Deploy your API to a new stage (e.g., "prod").
+   - Note the invoke URL - this will be your `AWS_API_GATEWAY_URL`.
+   - Note the API key - this will be your `AWS_API_KEY`.
+
+6. Update your Pico W code:
+   - Set up the AWS API Gateway URL and API key in your `my_secrets.py` file.
+   - Prepare the document data to be sent in your main script.
+   - Send a POST request to the AWS API Gateway.
+   - Handle the response and implement retry logic if necessary.
+
+7. Ensure security:
+   - Verify that the Lambda function has appropriate permissions to access MongoDB.
+   - Keep your API key secure and do not share it publicly.
+
+By following these steps, you'll have set up a serverless API that allows your Pico W to securely write data to MongoDB through AWS Lambda.
+## Code Modifications
+
+Modify the `main.py` file as follows:
+
+```python
+from netman import connectWiFi
+import urequests
+from my_secrets import SSID, PASSWORD, COURSE_ID, AWS_API_GATEWAY_URL, AWS_API_KEY
+
+connectWiFi(SSID, PASSWORD, country="US")
+
+headers = {"x-api-key": AWS_API_KEY}
+document = {"course_id": COURSE_ID}
+
+print(f"Sending document to AWS Lambda")
 
 num_retries = 3
 for _ in range(num_retries):
-    response = urequests.post(endpoint_url, headers=headers, json=payload)
+    response = urequests.post(AWS_API_GATEWAY_URL, headers=headers, json=document)
     txt = str(response.text)
     status_code = response.status_code
 
+    if status_code != 200:
+        print("Retrying in 5 seconds...")
+        time.sleep(5)
+
     print(f"Response: ({status_code}), msg = {txt}")
 
     response.close()
 
-    if status_code == 201:
+    if status_code == 200:
         print("Added Successfully")
         break
 
     print("Retrying...")
 ```
 
+
+### Example of `my_secrets.py`:
+
+```python
+SSID = "Your_WiFi_SSID"
+PASSWORD = "Your_WiFi_Password"
+COURSE_ID = "Your_Course_ID"
+AWS_API_GATEWAY_URL = "Your_API_Gateway_Invoke_URL"
+AWS_API_KEY = "Your_API_Key"
+```
+
+## Example Lambda Function (Python)
+```python
+import json
+import pymongo
+import os
+
+
+def lambda_handler(event, context):
+    try:
+        client = pymongo.MongoClient(os.environ["MONGODB_URI"])
+        db = client["your_database_name"]
+        collection = db["your_collection_name"]
+
+        body = json.loads(event["body"])
+        result = collection.insert_one(body)
+
+        return {
+            "statusCode": 200,
+            "body": json.dumps(
+                {
+                    "message": "Document inserted successfully",
+                    "id": str(result.inserted_id),
+                }
+            ),
+        }
+    except Exception as e:
+        return {
+            "statusCode": 500,
+            "body": json.dumps(
+                {"message": "Error inserting document", "error": str(e)}
+            ),
+        }
+```
+
 The output should look something like the following (MAC address and IP address redacted, and your insertedId will be different):
 
 > ```python
 > MAC address: ***
 > connected
 > ip = ***
-> sending document to test-cluster:test-db:write-to-me
-> Response: (201), msg = {"insertedId":"6594bfbfb3c925d2fdfbb7e8"}
+> sending document to AWS Lambda
+> Response: (200), msg = {"message": "Document inserted successfully"}
 > Added Successfully
 > ```
 
-### Reading from MongoDB Using PyMongo
+### Reading Data via AWS Lambda
 
-✅ Run the code from [the companion notebook](./1.5.1-pymongo.ipynb) to read data from a MongoDB database.
+✅ Run the code from [the companion notebook](./1.5.1-aws-lambda-read.ipynb) to read data from MongoDB through AWS Lambda.
 
-<!-- ✅ Copy the following code into a new file called `read_mongodb.py` and run it on the microcontroller. -->
 
 ### Reading Material
 
 ✅ Read [Community action on FAIR data will fuel a revolution in materials research](https://doi.org/10.1557/s43577-023-00498-4)
 
 ✅ Watch [the following video](https://youtu.be/cdSENQPsAiI?si=KkwmGzoQmQP-CsQo) about materials databases:
 
-<iframe width="560" height="315" src="https://www.youtube.com/embed/cdSENQPsAiI?si=h1dw65aA1tNamyQV" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+[Watch the video about materials databases on YouTube](https://www.youtube.com/watch?v=cdSENQPsAiI)
 
 ### Additional Resources
 - FAIR data principles [website](https://www.go-fair.org/fair-principles/) and [manuscript](https://doi.org/10.1038/sdata.2016.18)

docs/courses/hello-world/1.5.1-aws-lambda-read.ipynb

@@ -0,0 +1,144 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Reading Data from MongoDB via AWS Lambda\n",
+    "\n",
+    "This notebook demonstrates how to read data from MongoDB using an AWS Lambda function and API Gateway."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "First, let's install the required libraries and import them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install requests\n",
+    "import requests\n",
+    "import json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configuration\n",
+    "\n",
+    "Replace the placeholder values with your actual AWS API Gateway URL and API Key."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "AWS_API_GATEWAY_URL = \"https://your-api-gateway-url.execute-api.region.amazonaws.com/stage/resource\"\n",
+    "AWS_API_KEY = \"your-api-key\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Function to Read Data\n",
+    "\n",
+    "Let's create a function to read data from MongoDB via our AWS Lambda function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def read_data_from_mongodb(query={}):\n",
+    "    headers = {\n",
+    "        \"x-api-key\": AWS_API_KEY,\n",
+    "        \"Content-Type\": \"application/json\"\n",
+    "    }\n",
+    "    \n",
+    "    response = requests.post(AWS_API_GATEWAY_URL, headers=headers, json=query)\n",
+    "    \n",
+    "    if response.status_code == 200:\n",
+    "        return response.json()\n",
+    "    else:\n",
+    "        print(f\"Error: {response.status_code}, {response.text}\")\n",
+    "        return None"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Reading Data\n",
+    "\n",
+    "Now, let's use our function to read data from MongoDB."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Read all documents\n",
+    "result = read_data_from_mongodb()\n",
+    "print(json.dumps(result, indent=2))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Read documents with a specific query\n",
+    "query = {\"course_id\": \"DEMO101\"}\n",
+    "result = read_data_from_mongodb(query)\n",
+    "print(json.dumps(result, indent=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "This notebook demonstrated how to read data from MongoDB using an AWS Lambda function through an API Gateway. Remember to keep your API key secure and not share it publicly."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}