Skip to content

Data Pipeline

The data pipeline is a collection of scripts and Google sheets that together allow designers to create game content in a human readible set of Google sheets.

Overview of how the Data Pipeline works

The data pipeline is a set of tools used to transform design friendly sheets into beamable sheets. Once the conversion is finished, continue as normal to upload the beamable sheets to a realm. (Setting up Google Sheets for Beamable)[~/manual/gettingstarted/showcasesetup/gettingstartedbeamablesheets/index.html]

Setup

Everything you need to get Data Pipeline up and running is provided. Just follow the steps below and reach out to your support contact if anything is confusing or not working correctly.

  • Setting up a Local Python Environment
  • Setting a remote Lambda Function on AWS
  • Setting up a Google Sheet Add-On

Respositories

The main repository for the Data Pipeline project can be found here.

https://github.com/Idlekit/idlekit-data-pipeline

Inside you'll find a number of folders relating to each specific part of the pipeline.

  • pipeline

    • This is the bulk of the code base.
    • Contains the local/aws version of the code

    • Start with main.py

      • The pipeline helper classes are found in the idlekit folder

        We suggest you extend these classes rather than modify them for easier updates

  • google_appscript

    • The scripts needed to add menus to Google Sheets to call the functions on AWS Lambda

  • status_reporter

    • A script to scrape the status of the AWS Lambda function's progress and report it to the Google Appscript sidebar for the Data Pipeline.

  • misc_tools

    • Contains tools such a scripts to diif Google Sheets

Best Practices

It's best to do development locally, then when changes to your pipeline are finished uploaded them to AWS to run the conversion script from Google Sheets.