Airjordan

An airflow pipeline for building and scoring NBA daily fantasy models, including:

ETL to load nba stats
generating model features
scoring model predictions

Dependencies

This project requires the following dependencies in order to work:

A postgres sql database to store nba stats in ETL job: Follow these instructions to set up postgres Cloud SQL instance on GCP.
Google Cloud Storage bucket to store model objects: Follow these instructions to set up a service account to access files in GCS bucket. You’ll also need to download private keys required to access your bucket.
Docker and docker compose: Learn how to install docker and install docker-compose*.
API key for mysportsfeed: Get an API key for nba data feeds at mysportsfeed.com. Low cost options are available for personal use.
Environment file: Create a .env file in the project root directory (see the .env.template file) to store your mysportsfeed api key, postgres credentials and path to GCS json credentials file.

* If there are issues docker permissions, follow these instructions.

Run using docker (recommended)

This implementation of the airflow pipeline uses airflow’s CeleryExecutor which allows for parallel execution and easy scaling up of worker nodes.

docker-compose up --build

The airflow webserver is available on port 8080 on the host machine.

Run using pipenv

Note by default, this local developer version of the airflow pipeline, using pipenv, uses the SequentialExecutor which does not allow for parallel execution.

# install dependencies
pipenv install --dev 
# activate environment
pipenv shell
# setup airflow
export AIRFLOW_HOME=$(pwd)
airflow db init
airflow users create --username airflow --password airflow --firstname Anonymous --lastname Admin --role Admin --email admin@example.org
# run airflow
airflow webserver
airflow scheduler