ML at Scale with Spark on AWS

The Challenge

Training models on large tables needs a cluster, and setting one up by hand is slow and hard to repeat. We wanted a setup that anyone can create and destroy with one command.

Our Solution

We wrote Terraform code that creates an AWS EMR (Elastic MapReduce) cluster with Spark, JupyterHub and S3 storage. On top of it we trained Logistic Regression and Random Forest models with Spark MLlib on King County food inspection data.

Key Results

• Logistic Regression: AUC 0.953 and F1 0.876
• Random Forest: AUC 0.940 and F1 0.882
• Cluster and storage defined as code with Terraform