Intro to Big Data Systems

Introduction to the use, design, and implementation of database and data-intensive systems.

Course Overview

This course covers how to build data applications, and key principles on how to build and scale big data systems. Taught by Shiva Shivakumar.

A few key topics:

  1. How to use a database and SQL (Structured Query Language)? Spotify stores and manages 1 Billion+ songs/podcasts for 100M+ users and 10M+ artists. Let’s build Spotify’s features with SQL. E.g., artist search, popular songs, song recommendations. Intro to SQL alternatives.
  2. How to optimize queries with well-designed algorithms and data structures? Speed up Spotify queries with row and column stores, parallelism, BigSort, Hashing. See how OpenAI’s GPT and Google Maps index and query complex data types like text and geo data.
  3. How to build reliable Transactions? Ticketmaster-like orgs sell Taylor Swift concert tickets to 1 million+ fans across ~50 concerts in transactions – no double selling, guaranteeing tickets after payment. How does it work?
  4. How distributed big data systems scale? Lessons from Google Ads and Discord in scaling.

Weekly Schedule and Materials

How to navigate CS145 in Fall24?

Key Dates

Lectures: Tues/Thurs
1:30 PM - 2:50 PM

Exam#1: Thursday, October 31
Class Timings (1:30 - 2:50 PM)
Exam#2: Tuesday, December 10
3:30-6:30PM

Course Logistics and Policies

Prerequisites CS 103 and CS 107 (or equivalent). Basic knowledege of OS and algorithms (in RAM)

Grading base (for all): Exam #1: 20 points, Exam #2: 30 points, PSETs: 10 points, Project1: 10 points, Project2: 30 points

We cannot provide the exact score threshold since the course is curved at the end of the quarter. For credit/no-credit you need to score equivalent to at least a C grade to pass the course.

We will offer extra credit for students with insightful in-class and Ed participation.

Canvas, Ed, Gradescope: Lecture videos will be posted on Canvas. Also, please access course Ed, Gradescope, Office Hour schedules on Canvas.

Audit requests (Stanford only): For audit requests and access to internal tools, please fill form

Conflict in exams or course schedule Due to the large course enrollment number, we won’t be able to accommodate alternate exam schedules for those who have exam conflicts (both midterm and final). Please make sure you do not have a conflict in exam schedules when enrolling in CS 145.

Late Days You can use a total of two late days (24 hours each, not pro-rated) shared between both project deadlines. You do not lose any credit when using a late day. If you run out of late days and need additional time, you can submit after the deadline -- you will receive a 10% deduction for the 1st 24 hours after the deadline, 25% deduction for the next 24 hours, and zero credit after that. It's by 24 hours, no proration (for seconds, hours, etc)

Honor Code/Collaboration Policy

Students must adhere to The Stanford Honor Code and The Stanford Honor Code as it pertains to CS courses.

We encourage students to form study groups. Students may discuss and work on homework problems in groups. However, each student must write down the solution independently, and without referring to written notes from the joint session.

It is an honor code violation to copy, refer to, or look at written or code solutions from a previous year, including but not limited to: official solutions from a previous year, solutions posted online, and solutions you or someone else may have written up in a previous year. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo.

The teaching staff will be using plagiarism detection software and if we have reason to believe that you are in violation of the honor code, we will follow the university policy to report it.