Data Pipeline AWS Lambda Setup
This guide will help you setup an AWS Lambda Environment to convert designer friendly sheets into machine readible sheets through the Google Sheets add-on.
Lambda Layer Setup
Creating a Layer for Lambda functions. This default layer will be given to you as part of the project. But if any need to add additional libraries to your AWS Lambda Layer, here are some simple instructions.
Setup
Layers are a way of including packages for python that the scripts can access on AWS.
You can configure your Lambda function to use additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package.
To include libraries in a layer, place them in the directory structure that corresponds to your programming language.
Creating a Layer with the command line
Run these commands in a Linux environment that supports pip and python
mkdir folder
cd folder
virtualenv v-env
source ./v-env/bin/activate
pip install pandas (and any other packages you want)
deactivate
mkdir python
cd python
cp -r ../v-env/lib/python3.8/site-packages/* . (Or similar)
cd ..
zip -r panda_layer.zip python
Add the layer to the layers section of the lambda function (at the bottom)
Here’s a bit of test code to make sure it’s working for pandas.
import numpy as np
import pandas as pd
def lambda_handler(event, context):
df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),columns=["a", "b", "c"])
number = np.pi
print(df2)
print(number)
Additional Help!
AWS has up to date documentation to help you create layers. If you are unfamiliar with the process, please take a moment to visit their nicely laid out docs.
Test Configurations
After deploying the functions to AWS, you’ll want to configure the functions to be tested. This will allow you to get more detailed information during troubleshooting such as memory used, execution time, and any uncaught exceptions.
Update the uid for both these parameters.
"rawQueryString": "uid=GOOGLE SHEET ID",
"queryStringParameters": { "uid": "GOOGLE SHEET ID" }
Test config For the main lambda function
{
"version": "2.0",
"routeKey": "ANY /idlekit-data-pipeline",
"rawPath": "/default/idlekit-data-pipeline",
"rawQueryString": "uid=1rwZke4LAMuetxTKRd2LQC3WpP0kRDs-X0b9Q8Lzi9kM",
"cookies": [
"awsccc=eyJlIjoxLCJwIjoxLCJmIjoxLCJhIjoxLCJpIjoiNTc0ZWU4OTItMDM4ZC00ZmU4LWEwZGUtN2FhZTk4MWJhMGNlIiwidiI6IjEifQ=="
],
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-US,en;q=0.9",
"cache-control": "max-age=0",
"content-length": "0",
"host": "ouye95yqd5.execute-api.us-west-2.amazonaws.com",
"sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
"sec-ch-ua-mobile": "?0",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"x-amzn-trace-id": "Root=1-60dca88f-3a90afac63b90744402d17db",
"x-forwarded-for": "142.179.207.207",
"x-forwarded-port": "443",
"x-forwarded-proto": "https"
},
"queryStringParameters": {
"uid": "1rwZke4LAMuetxTKRd2LQC3WpP0kRDs-X0b9Q8Lzi9kM"
},
"requestContext": {
"accountId": "635489503008",
"apiId": "ouye95yqd5",
"domainName": "ouye95yqd5.execute-api.us-west-2.amazonaws.com",
"domainPrefix": "ouye95yqd5",
"http": {
"method": "GET",
"path": "/default/idlekit-datapipeline-status-reporter",
"protocol": "HTTP/1.1",
"sourceIp": "142.179.207.207",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
},
"requestId": "Bv9GeiwjvHcEMhA=",
"routeKey": "ANY /idlekit-datapipeline-status-reporter",
"stage": "default",
"time": "30/Jun/2021:17:23:27 +0000",
"timeEpoch": 1625073807620
},
"isBase64Encoded": false
}
```
Test config for the status reporter