Skip to content

Data Pipeline AWS Lambda Setup

This guide will help you setup an AWS Lambda Environment to convert designer friendly sheets into machine readible sheets through the Google Sheets add-on.

Lambda Layer Setup

Creating a Layer for Lambda functions. This default layer will be given to you as part of the project. But if any need to add additional libraries to your AWS Lambda Layer, here are some simple instructions.

Setup

Layers are a way of including packages for python that the scripts can access on AWS.

You can configure your Lambda function to use additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package.

To include libraries in a layer, place them in the directory structure that corresponds to your programming language.

Creating a Layer with the command line

Run these commands in a Linux environment that supports pip and python

mkdir folder
cd folder
virtualenv v-env
source ./v-env/bin/activate
pip install pandas    (and any other packages you want)
deactivate

mkdir python
cd python
cp -r ../v-env/lib/python3.8/site-packages/* .      (Or similar)
cd ..
zip -r panda_layer.zip python
Create a new layer on AWS and upload the zip file you just created. This file is specifically for python.

Add the layer to the layers section of the lambda function (at the bottom)

Here’s a bit of test code to make sure it’s working for pandas.

import numpy as np
import pandas as pd
def lambda_handler(event, context):
    df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),columns=["a", "b", "c"])
    number = np.pi
    print(df2)
    print(number)

Additional Help!

AWS has up to date documentation to help you create layers. If you are unfamiliar with the process, please take a moment to visit their nicely laid out docs.

AWS Lambda Layer Creation

Test Configurations

After deploying the functions to AWS, you’ll want to configure the functions to be tested. This will allow you to get more detailed information during troubleshooting such as memory used, execution time, and any uncaught exceptions.

Update the uid for both these parameters.

"rawQueryString": "uid=GOOGLE SHEET ID",

"queryStringParameters": { "uid": "GOOGLE SHEET ID" }

Test config For the main lambda function

{
  "version": "2.0",
  "routeKey": "ANY /idlekit-data-pipeline",
  "rawPath": "/default/idlekit-data-pipeline",
  "rawQueryString": "uid=1rwZke4LAMuetxTKRd2LQC3WpP0kRDs-X0b9Q8Lzi9kM",
  "cookies": [
    "awsccc=eyJlIjoxLCJwIjoxLCJmIjoxLCJhIjoxLCJpIjoiNTc0ZWU4OTItMDM4ZC00ZmU4LWEwZGUtN2FhZTk4MWJhMGNlIiwidiI6IjEifQ=="
  ],
  "headers": {
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "en-US,en;q=0.9",
    "cache-control": "max-age=0",
    "content-length": "0",
    "host": "ouye95yqd5.execute-api.us-west-2.amazonaws.com",
    "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
    "sec-ch-ua-mobile": "?0",
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "none",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
    "x-amzn-trace-id": "Root=1-60dca88f-3a90afac63b90744402d17db",
    "x-forwarded-for": "142.179.207.207",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https"
  },
  "queryStringParameters": {
    "uid": "1rwZke4LAMuetxTKRd2LQC3WpP0kRDs-X0b9Q8Lzi9kM"
  },
  "requestContext": {
    "accountId": "635489503008",
    "apiId": "ouye95yqd5",
    "domainName": "ouye95yqd5.execute-api.us-west-2.amazonaws.com",
    "domainPrefix": "ouye95yqd5",
    "http": {
      "method": "GET",
      "path": "/default/idlekit-datapipeline-status-reporter",
      "protocol": "HTTP/1.1",
      "sourceIp": "142.179.207.207",
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
    },
    "requestId": "Bv9GeiwjvHcEMhA=",
    "routeKey": "ANY /idlekit-datapipeline-status-reporter",
    "stage": "default",
    "time": "30/Jun/2021:17:23:27 +0000",
    "timeEpoch": 1625073807620
  },
  "isBase64Encoded": false
}
 ```

Test config for the status reporter
{ "version": "2.0", "routeKey": "ANY /idlekit-datapipeline-status-reporter", "rawPath": "/default/idlekit-datapipeline-status-reporter", "rawQueryString": "uid=1234", "cookies": [ "awsccc=eyJlIjoxLCJwIjoxLCJmIjoxLCJhIjoxLCJpIjoiNTc0ZWU4OTItMDM4ZC00ZmU4LWEwZGUtN2FhZTk4MWJhMGNlIiwidiI6IjEifQ==" ], "headers": { "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9", "accept-encoding": "gzip, deflate, br", "accept-language": "en-US,en;q=0.9", "cache-control": "max-age=0", "content-length": "0", "host": "ouye95yqd5.execute-api.us-west-2.amazonaws.com", "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"", "sec-ch-ua-mobile": "?0", "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "none", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36", "x-amzn-trace-id": "Root=1-60dca88f-3a90afac63b90744402d17db", "x-forwarded-for": "142.179.207.207", "x-forwarded-port": "443", "x-forwarded-proto": "https" }, "queryStringParameters": { "uid": "1rwZke4LAMuetxTKRd2LQC3WpP0kRDs-X0b9Q8Lzi9kM" }, "requestContext": { "accountId": "635489503008", "apiId": "ouye95yqd5", "domainName": "ouye95yqd5.execute-api.us-west-2.amazonaws.com", "domainPrefix": "ouye95yqd5", "http": { "method": "GET", "path": "/default/idlekit-datapipeline-status-reporter", "protocol": "HTTP/1.1", "sourceIp": "142.179.207.207", "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" }, "requestId": "Bv9GeiwjvHcEMhA=", "routeKey": "ANY /idlekit-datapipeline-status-reporter", "stage": "default", "time": "30/Jun/2021:17:23:27 +0000", "timeEpoch": 1625073807620 }, "isBase64Encoded": false } ```