Intro to Big Data Systems

Introduction to the use, design, and implementation of database and data-intensive systems.

Course Overview

This course covers how to build data applications, and key principles on how to build and scale big data systems. Taught by Shiva Shivakumar.

A few key topics:

  1. How to use a database and SQL (Structured Query Language)? Spotify stores and manages 1 Billion+ songs/podcasts for 100M+ users and 10M+ artists. Let’s build Spotify’s features with SQL. E.g., artist search, popular songs, song recommendations. Intro to SQL alternatives.
  2. How to optimize queries with well-designed algorithms and data structures? Speed up Spotify queries with row and column stores, parallelism, BigSort, Hashing. See how OpenAI’s GPT and Google Maps index and query complex data types like text and geo data.
  3. How to build reliable Transactions? Ticketmaster-like orgs sell Taylor Swift concert tickets to 1 million+ fans across ~50 concerts in transactions – no double selling, guaranteeing tickets after payment. How does it work?
  4. How distributed big data systems scale? Lessons from Google Ads and Discord in scaling.

Key Dates

Lectures: Tues/Thurs
4:30 PM - 5:50 PM
NVIDIA Auditorium
Exam#1: Tuesday, November 2
Class Timings (4:30 - 5:50 PM)
Exam#2: Monday, December 11
7-10 PM

Course Logistics and Policies

Prerequisites CS 103 and CS 107 (or equivalent). Basic knowledege of OS and algorithms (in RAM)

Grading Projects: 40% (10 + 30), Exam #1: 20%, Exam #2: 35%, PSETs: 5%

For students taking the course on a credit/no-credit basis: you need to score equivalent to at least a C grade to pass the course. We cannot provide the exact score threshold since the course is curved at the end of the quarter.

We will be offering extra credit for in class participation.

Ed and Gradescope: please access course Ed and Gradescope on the canvas tab.

Conflict in exams or course schedule Due to the large course enrollment number, we won’t be able to accommodate alternate exam schedules for those who have exam conflicts (both midterm and final). Please make sure you do not have a conflict in exam schedules when enrolling in CS 145.

Late Days You can use a total of two late days (24 hours each, not pro-rated) shared between both project deadlines. You do not lose any credit when using a late day. If you run out of late days and need additional time, you can submit after the deadline -- you will receive a 10% deduction for the 1st 24 hours after the deadline, 25% deduction for the next 24 hours, and zero credit after that. It's by 24 hours, no proration (for seconds, hours, etc)

Lectures Lectures occur on Tues/Thurs 4:30-5:50 p.m. in NVIDIA Auditorium. NOTE that while attendance is not mandatory, we will be giving out extra credit for students with insightful in-class participation.

Lecture Videos Lecture videos will be recorded and posted on Canvas.

Honor Code/Collaboration Policy

Students must adhere to The Stanford Honor Code and The Stanford Honor Code as it pertains to CS courses.

We encourage students to form study groups. Students may discuss and work on homework problems in groups. However, each student must write down the solution independently, and without referring to written notes from the joint session.

It is an honor code violation to copy, refer to, or look at written or code solutions from a previous year, including but not limited to: official solutions from a previous year, solutions posted online, and solutions you or someone else may have written up in a previous year. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo.

The teaching staff will be using plagiarism detection software and if we have reason to believe that you are in violation of the honor code, we will follow the university policy to report it.


Dates Topics Lectures Run Colabs Reading from Course Notes Events
9/26 SQL basics Intro/Logistics, SQL slides Colab #1 Section 1 (SQL) Pg 1-16
9/28 SQL labs (JOINs, Aggregates) Lecture 1 slides, SQL Project 1 released on Ed Colab #1 Section 1 (SQL) Pg 17-23 SQL Project 1 released on Ed. Signup for GCP credits
10/3 SQL intermediate (Common Table Expressions, Window functions) SQL slides Colab #1 Section 2,3 (SQL) Pg 24-36 PSET 0 Due (Pre-reqs)
10/5 Hybrid of SQL and Alternatives SQL slides Colab #1 Section 4 (SQL) Finish Section 1 PSET 1 Due (Basic SQL)
10/10 Systems Primer & nanoDB Storage, BigSort, Hashing Algorithms nanoDB slides Colab #2 Section 1 (NanoDB) Cases 6, 6.1, 6.2 PSET 2 Due
10/12 IO Complexity, Basic Indexing nanoDB slides Colab #3 (Hashing) Cases 7-8.3 SQL Project 1 Due. SQL Project 2 Released
10/17 Intermediate Indexing nanoDB slides Colab #2 Section 2 (NanoDB) Finish Section 2
10/19 JOIN Algorithms nanoDB slides Colab #2 Section 3 (NanoDB) Cases 10.1, 10.2
10/24 nanoDB slides NanoDB Finish Section 3
10/26 File systems, MapReduce/Spark. Tips for Project 2 Spark slides Lecture PSET #3 due -- no extensions. SQL Project 2 Proposal due
10/31 Midterm review Midterm review Lecture
11/2 Midterm in class