AWS E-Commerce Data Pipeline

Scope note: learning project, not a production system. Built to practice serverless AWS infrastructure and infrastructure-as-code with Terraform.

github

Goal

Process 540k+ e-commerce transactions through a serverless AWS pipeline — from raw CSV to live JSON endpoints — with all infrastructure defined as Terraform code.

Approach

  1. Upload the UCI Online Retail dataset (541k transactions, Dec 2010–Dec 2011) to a versioned S3 bucket.
  2. Run a PySpark job on AWS Glue to clean the data (remove cancellations, invalid rows) and compute 9 aggregation tables written as Parquet.
  3. Serve results through a Lambda function + API Gateway — public HTTP endpoints returning JSON.
  4. Define all 15 infrastructure resources in Terraform: one command to deploy, one to tear down.

Guiding questions

  • Can a full analytics pipeline run on AWS free tier at ~$0/month?
  • Is Terraform practical for managing a small but complete serverless stack?
  • How well does PySpark on Glue handle aggregations like RFM segmentation and cohort retention?

Data

Headline numbers

Raw transactions541,909
Clean rows397,884
Countries37
API endpoints9
Infra resources15Terraform-managed
Monthly cost~$0free tier

Architecture

CSV541k rows
→
S3raw bucket
→
GluePySpark
→
S3parquet
→
LambdaPython
→
API GWHTTP

▲ all resources managed by Terraform

1
IngestS3 (raw bucket)

Raw CSV (541k transactions) uploaded to a versioned S3 bucket.

2
TransformAWS Glue (PySpark)

PySpark job cleans, casts types, removes cancellations, and computes 9 aggregation tables.

3
StoreS3 (processed)

Results written as Parquet — columnar, compressed, fast to query.

4
ServeLambda + API Gateway

Serverless Python function reads Parquet from S3 and returns JSON via a public HTTP endpoint.

5
ProvisionTerraform

All infrastructure defined as code — reproducible, version-controlled, one-command deploy/teardown.

Analytics computed

The PySpark job computes 9 aggregation tables from the cleaned data, grouped into three categories.

Core Metrics

Revenue by Country

Total revenue, orders, unique customers per market

Monthly Trends

Revenue and order volume — captures seasonality

Top Products

Top 20 SKUs by revenue with quantity sold

Customer Analytics

RFM Segmentation

Recency × Frequency × Monetary scoring → Champion, Loyal, At Risk, Lost

Cohort Retention

What % of each monthly cohort returns in following months

Pareto / Revenue Concentration

Do 20% of customers drive 80% of revenue?

Behavioral

Day × Hour Heatmap

When do people buy? Revenue by day-of-week and hour

Average Order Value

AOV trend — is the business growing value or volume?

Basket Size Distribution

Items per order and average revenue per bucket

Live API

The endpoints below hit a real AWS Lambda function that reads Parquet files from S3. Click any endpoint to see live data from the pipeline.

https://87rc15jlf7.execute-api.eu-west-1.amazonaws.com

Conclusion

  • The PySpark job processes 540k rows and computes 9 analytical views (RFM segmentation, cohort retention, Pareto analysis) served through a serverless API at near-zero cost.
  • The entire stack (15 resources) is defined in Terraform — one terraform apply to deploy, one terraform destroy to tear down.
AWSS3GluePySparkLambdaAPI GatewayTerraformPythonPlotly