GitHub GitHub top language GitHub language count GitHub last commit ViewCount


Solving analytical questions on the semi-structured MovieLens dataset containing a million records using Spark and Scala. This features the use of Spark RDD, Spark SQL and Spark Dataframes executed on Spark-Shell (REPL) using Scala API. We aim to draw useful insights about users and movies by leveraging different forms of Spark APIs.

Table of Contents

Major Components

Apache Spark Logo Scala


  • Linux (Ubuntu 15.04)
  • Hadoop 2.7.2
  • Spark 2.0.2
  • Scala 2.11

Installation steps

  1. Simply clone the repository

    git clone
  2. In the repo, Navigate to Spark RDD, Spark SQL or Spark Dataframe locations as needed.

  3. Run the execute script to view results

  4. The will pass the scala code through spark-shell and then display the findings in the terminal from the results folder.

Analytical Queries

Spark RDD

Spark SQL

Spark DataFrames


  • Import Data from URL: Scala
  • Save table without defining DDL in Hive
  • Broadcast Variable example
  • Accumulator example
  • Databricks Community Edition

Note: The results were collected and repartitioned into the same text file: This is not a recommended practice since performance is highly impacted but it is done here for the sake of readability.


This project was featured on Data Machina Issue #130 listed at number 3 under ScalaTOR. Thank you for the listing


This repository is licensed under Apache License 2.0 - see License for more details

Movies Analytics In Spark And Scala

Data cleaning, pre-processing, and Analytics on a million movies using Spark and Scala.

Movies Analytics In Spark And Scala Info

⭐ Stars 37
🔗 Source Code
🕒 Last Update 9 months ago
🕒 Created 4 years ago
🐞 Open Issues 0
➗ Star-Issue Ratio Infinity
😎 Author Thomas-George-T