Why there are two different kinds of data work, and why one system cannot do both well.
Data at scale,
from first principles
When data is too big for one computer, how do we keep it, change it and ask questions of it? And why are there so many systems for this — Hadoop, Spark, Redshift, Cassandra, Databricks? This document starts with one table and one disk. It ends with a full data platform, where each system does the job it was made for.
TopicBig data systems and data warehouses
Main questionHow do many computers act as one store of data?
You needWhat a table is · what a disk is
Key parts03 MapReduce · 05 Spark · 06 Columns · 08 Cassandra
Chapters11 + glossary
Reading timeAbout 60 minutes
FP-00
Start here
What you will understand, and the path to it.
You will understand
You will understand
How many computers split data, keep copies, and send the work to the data.
You will understand
How MapReduce, Hadoop and Spark run one job on thousands of computers, and how Spark recovers from failures.
You will understand
Why storing data by column makes analytical questions 10 to 1000 times cheaper, and how a warehouse like Redshift uses this.
You will understand
How Cassandra finds the right computer for a key with no central map, and what "quorum" guarantees.
You will understand
How a lakehouse such as Databricks turns plain files into reliable tables, and how all these systems fit together.
The path
Each chapter uses only the chapters before it. Amber chapters are the key parts.
How to read the boxes
KEY POINT
The idea of the section in one sentence.
ANALOGY
A picture from daily life, and where it stops working.
COMMON MISTAKE
A wrong idea that many people have.
NOTE
A simplification or an assumption.
FP-01
Two kinds of question
The split that decides which system you need.
Every system in this document answers one of two kinds of question. When you know which kind a system is for, you can predict how it stores data and what it is bad at.
One table
An online shop keeps a list of its orders. Each line in the list is one order. We call a line a row. Each named field, such as date or amount, holds one kind of value for every order. We call a field a column. Rows and columns together make a table.
People ask questions of tables in a language called SQL, which stands for Structured Query Language. You will see a few short SQL lines in this document. Each one comes with its meaning in words.
Table · orders
Question A: "What is the status of order 1042?" It reads one row, and all six columns of that row.
Look at the two questions the shop asks.
- Question A: "What is the status of order 1042?" or "Set order 1042 to shipped." It reads or changes one row. The website asks thousands of these each second. Each answer must come back in a few milliseconds, because a customer waits.
- Question B: "What were the total sales for each country last year?" It reads three columns (date, country, amount), but it reads them for every row of the year. Only a few people in the company ask it, and a wait of some seconds is acceptable.
Question A is the work that keeps the business running. We call it OLTP, which stands for Online Transaction Processing. Question B is the work that helps people understand the business. We call it OLAP, which stands for Online Analytical Processing.
| OLTP (question A) | OLAP (question B) | |
|---|---|---|
| Rows in each question | One or a few | Millions or billions |
| Columns in each question | Usually all | Usually a few |
| Questions each second | Thousands or more | A few |
| Acceptable wait | Milliseconds | Seconds or minutes |
| Writes | Many small changes | Large loads, not often |
KEY POINT
OLTP reads and writes a few whole rows, very often and very fast. OLAP reads a few columns across very many rows, less often. A system built for one shape of work is slow at the other.
NOTE
Real companies have both kinds of work, and some systems try to do both. The split is still the base of everything that follows. Chapter 06 shows why one storage layout cannot be best for both.
Check"How many customers bought more than three times this year?" OLTP or OLAP?
OLAP. The question reads the customer column of every order in the year, then counts. No single row is the answer. "Add a new order for Ana" is OLTP: it writes one row.
FP-02
Too big for one computer
Split the data, copy the data, move the work.
A shop with ten thousand orders fits on one laptop. A delivery company that records every scan of every parcel can have billions of rows. At some size, one computer is not enough. This chapter shows where that size is, and what we do then.
The disk is the limit
A computer keeps data on a disk. A disk reads data at a fixed speed. A traditional spinning disk reads about 200 megabytes (MB) each second. One terabyte (TB) is one million MB.
Now do the sum. To read 10 TB from one disk takes 10,000,000 MB ÷ 200 MB per second = 50,000 seconds. That is almost 14 hours, for one question. Faster disks help, but data grows faster than disks get faster.
Bigger computer, or more computers?
There are two ways to get more power.
- Buy one bigger computer, with more disks and more memory. We call this scaling up. It is simple, but there is a largest computer you can buy, and the price grows faster than the size.
- Connect many ordinary computers with a network, and make them work together. We call this scaling out. Each computer in the group is a node. The whole group is a cluster.
Scaling out has no fixed limit. If you have 1,000 nodes, each with its own disk, the 14-hour read takes about 50 seconds. But scaling out creates three new problems. Every system in this document is a set of answers to these three problems.
Problem 1: where is each piece?
You must split the data, and each node keeps some parts. One such part is a partition. The system must always know which node has which partition, or must be able to calculate it.
Problem 2: computers fail
Suppose one computer fails, on average, once in 1,000 days (about three years). With one computer, that is rare. With 1,000 computers, you expect about one failure every day. So in a cluster, failure is normal, not special.
The answer is to keep more than one copy of each partition, on different nodes. Each copy is a replica. If one node fails, another replica has the data.
Problem 3: the network is slow
Sending data between nodes is slower than reading a local disk, and all nodes share the network. If every node sends its data to one place, that place becomes the limit again. So the rule is: send the program to the data, not the data to the program. A program is a few kilobytes; the data is terabytes. We call this rule data locality.
Model · read all data once, on a cluster
Time to read all data
13.9 h
Data on each node10 TBFailures expected each day0.001Chance at least one node fails today0.1%
This model uses 200 MB per second for each disk and one failure in 1,000 days for each node. Try this: set 10 TB, then move the nodes from 1 to 1,000. The time falls from hours to seconds. Now look at the failure line. What does a 1,000-node system need that a 1-node system does not?
KEY POINT
When data is too big for one computer, split it into partitions across many nodes. Keep replicas of each partition. Send the work to the nodes that hold the data.
ANALOGY
In a national election, nobody carries all the paper votes to one hall. Each polling station counts its own votes. Then each station sends only its totals to the centre. The work goes to the data, and only small results travel.
Where this stops working: in an election the votes are already split by place, and a polling station rarely disappears in the middle of the count. In a cluster, the software must decide how to split the data, and it must expect a node to fail during the work.
COMMON MISTAKE
"Twice the nodes always means half the time." This is true only for the part of the work that each node does alone. When the nodes must combine their results, data must cross the network. That part does not get faster in the same way. The next chapter is about that part.
CheckYou have 100 TB on 500 nodes. Each node reads 200 MB per second. How long does one full read take?
100 TB is 100,000,000 MB. Each node holds 200,000 MB. At 200 MB per second, each node needs 1,000 seconds, and all nodes work at the same time. So the read takes about 1,000 seconds, or about 17 minutes.
FP-03
MapReduce: one job, many nodes
How separate nodes produce one combined answer. Hadoop and Spark both rest on this chapter.
Chapter 02 put the data on many nodes and sent the work to each node. Each node can now count its own part. But the question wants one answer: total sales for each country, across all nodes. Somehow the parts must come together, and the programmer must not write network code for every question. MapReduce is a pattern that solves this.
Pairs of a label and a value
MapReduce works on pairs. Each pair has a label and a value, for example ("UK", 42) or ("cat", 1). We call this a key-value pair. The label is the key. The key says which group the value belongs to.
The three steps
The programmer writes two small functions. The system does everything else.
- Map. The programmer writes a function that takes one input record and gives out zero or more key-value pairs. The system runs it on every node, on that node's own data. The nodes do not talk to each other in this step. We call this step map.
- Shuffle. The system moves every pair across the network, so that all pairs with the same key arrive at the same node. Then that node sorts the pairs, so equal keys are next to each other. We call this step the shuffle.
- Reduce. The programmer writes a second function. It gets one key and the list of all values for that key, and gives out the result for that key. We call this step reduce. A node that runs it is a reducer.
The classic first example counts how often each word appears in a large set of text. Use the step-through to follow it.
- The input is three lines of text. Each line is on a different node. (A real job has billions of lines on thousands of nodes.)
- Map. Each node reads only its own line. For each word, it gives out the pair (word, 1). The nodes do not talk to each other, so this step gets faster with every node you add.
- Shuffle. Each pair travels to one reducer. The key decides which one: cat, dog and ran go to reducer A (coral lines); sat and the go to reducer B (amber, dashed). Look: every node sends to every reducer.
- Each reducer sorts the pairs it received. Now all the values for one key are together in one list.
- Reduce. Each reducer adds the values in each list. The answer is complete: the 3, cat 2, sat 2, dog 1, ran 1. No reducer needed data from the other reducer.
How the shuffle chooses a reducer
Every node must send "cat" to the same reducer, without asking anyone. For this, the system uses a calculation that turns any text into a number. The same text always gives the same number. We call this calculation a hash.
Then the system divides the hash by the number of reducers and keeps the remainder. The remainder is the reducer number. With two reducers, the remainder is 0 (reducer A) or 1 (reducer B).
| Key | Hash (made-up numbers) | Remainder after ÷ 2 | Reducer |
|---|---|---|---|
| cat | 14 | 0 | A |
| dog | 8 | 0 | A |
| ran | 22 | 0 | A |
| sat | 9 | 1 | B |
| the | 31 | 1 | B |
reducer = hash(key) mod RThe reducer number is the remainder when you divide the hash of the key by R, the number of reducers. "mod" means "the remainder after division".
Node 1 and node 3 both calculate hash("cat") = 14, so both send "cat" to reducer A. No node needs a list of where keys go. Remember this trick: Cassandra, in chapter 08, uses the same idea in a better form.
The same pattern answers business questions
Word count looks like a toy, but change the two functions and you get question B from chapter 01: total sales for each country.
- Map: for each order, give out
(country, amount). Order 1041 gives("UK", 42). - Shuffle: all UK amounts go to one reducer, all FR amounts to one reducer, and so on.
- Reduce: add the amounts. UK gets 42 + 75 + 60 = 177.
In SQL, this is SELECT country, SUM(amount) FROM orders GROUP BY country, which means "for each country, add up the amounts". Every GROUP BY and every join of two large tables needs a shuffle.
KEY PART · WHY THE SHUFFLE IS THE EXPENSIVE STEP
Map and reduce run on each node alone. The shuffle is the only step where data crosses the network, and it goes from every node to every reducer. With 1,000 map nodes and 1,000 reducers, that is up to 1,000,000 separate transfers. The data is also sorted and written to disk on the way.
So the size of the shuffle usually decides how long a job takes. Most tuning of big data jobs is about making the shuffle smaller.
Making the shuffle smaller
Suppose node 1 had the text "the cat the". The map step gives (the, 1) twice. Instead of sending two pairs, the node can add them first and send (the, 2). A reduce step that runs on the map node, before the shuffle, is a combiner.
This is correct for a sum, a count, a maximum or a minimum. For these, the order and grouping of the steps do not change the result. It is not correct for an average. For example, the average of (2, 4) is 3 and the average of (6) is 6. The average of those two averages is 4.5. But the true average of 2, 4 and 6 is 4. To get an average, send (sum, count) pairs and divide only at the end.
When a node fails
A map task reads only its own input part, and nothing else depends on it until the shuffle. So if a node fails in the map step, the system starts that map task again. It uses another node that holds a replica of the same input. The programmer does nothing. This is why MapReduce can run on thousands of cheap nodes that fail often.
We call the whole pattern — map, shuffle, reduce, plus the automatic restart — MapReduce. Google described it in a paper in 2004.
KEY POINT
MapReduce splits a job into two steps. In the map step, each node works alone on its own data. In the reduce step, one function runs for each key. The shuffle joins them: it moves all values for one key to one node. The shuffle is where the network cost is.
ANALOGY
Think of the post. Each local post office sorts its letters into bags, one bag for each destination city (map, and the choice of reducer). Lorries carry the bags between cities (shuffle). Each city office then handles all letters for its own streets (reduce).
Where this stops working: post offices do not combine letters into a result. And they do not all send to all other offices at the same moment. In MapReduce the shuffle is all-to-all and happens in one burst, which is why it is so heavy.
COMMON MISTAKE
"MapReduce is a product." MapReduce is a pattern: a way to split a job. Hadoop (next chapter) has one implementation of it. Spark and every data warehouse use the same map–shuffle–reduce shape inside, even when you only write SQL.
COMMON MISTAKE
"If a job is slow, add more reducers." Suppose one key has most of the data: for example, 90% of orders are from the UK. Then one reducer gets 90% of the work, however many reducers you add. Uneven keys are a real limit. Chapter 11 returns to this.
CheckYou have weather readings: (city, temperature). How do you find the highest temperature in each city with MapReduce? Can you use a combiner?
Map gives out (city, temperature) for each reading. The shuffle sends all readings for one city to one reducer. Reduce keeps the largest value in each list. Yes, a combiner is safe: the maximum of the local maximums is the true maximum.
CheckWhy must all values for "London" arrive at the same reducer?
If two reducers each got some London readings, each would give out its own "highest London temperature". There would be two answers and neither would be sure to be right. One key, one reducer, one answer.
FP-04
Hadoop: storage and work on the same nodes
The first open system that put chapters 02 and 03 together.
MapReduce needs a place where data is already split into parts, copied, and tracked. Google described its own file system in 2003 and MapReduce in 2004. Engineers then built free, open versions, with most early work at Yahoo from 2006. We call this set of software Hadoop. It has three main parts.
Part 1: a file system across many nodes
Hadoop stores one large file across many nodes. We call this file system HDFS, which stands for Hadoop Distributed File System. It works like this:
- HDFS cuts each file into pieces of a fixed size, 128 MB by default. A fixed-size piece of stored data is a block.
- It copies each block to three different nodes, by default. These are the replicas from chapter 02.
- The nodes that store blocks are DataNodes. One special node keeps the list of which blocks make each file, and where each copy is. This node is the NameNode.
- One file of 512 MB arrives. It is too big to keep on one node in a large system, and one disk would read it slowly.
- HDFS cuts the file into four blocks of 128 MB: B1, B2, B3 and B4.
- HDFS copies each block to three different DataNodes. Count the B1 boxes: there are three, on DN1, DN2 and DN4.
- The NameNode keeps the list of where every copy is. A program that wants B3 asks the NameNode, then reads directly from DN1, DN3 or DN4.
- DN3 fails. B2 and B3 each lost one copy. But each still has two copies on other nodes, so no data is lost and reads continue.
- The NameNode sees the failure and makes new copies (amber): B2 on DN1 and B3 on DN5. Every block has three copies again.
Part 2: sharing the nodes between jobs
Many teams run jobs on one cluster at the same time. Some software must give each job its share of the processors and memory on each node. In Hadoop, this part is YARN, which stands for Yet Another Resource Negotiator.
Part 3: MapReduce, near the data
Hadoop's MapReduce engine asks the NameNode where each block is. It then starts each map task on a DataNode that already holds that block. This is data locality from chapter 02: the block is read from the local disk, not across the network.
SQL on top
Writing map and reduce functions for every question is slow work, and most analysts know SQL, not Java. So engineers at Facebook built a layer that takes SQL and turns it into MapReduce jobs. This layer is Hive. Hive also stored the list of tables and their columns, and later systems still use that list format.
The weak point: disk between every step
Each MapReduce job reads its input from HDFS and writes its output back to HDFS, with three copies. A real question often needs a chain of jobs: join two tables, then group, then sort. Machine-learning programs are worse: they read the same data 10 or 100 times in a loop. Each step in the chain pays again: a full disk read, a full disk write, three copies across the network, and a job start. A question that needs ten jobs pays ten times.
KEY POINT
Hadoop puts storage (HDFS) and work (MapReduce) on the same nodes, so each map task reads a local block. It survives node failures through three replicas. Its weak point is that every job writes its full result to disk before the next job can start.
NOTE
Hadoop is still maintained: version 3.5.0 came out in April 2026. But most new projects now keep data in cloud storage (chapter 09) instead of HDFS, and run Spark (next chapter) instead of Hadoop MapReduce. You will still meet Hadoop in large companies that built their own data centres in the 2010s, and you will meet its ideas everywhere.
CheckA 1 GB file goes into HDFS with 128 MB blocks and three replicas. How many blocks are there, and how much disk space does the file use in total?
1 GB is 1,024 MB. 1,024 ÷ 128 = 8 blocks. Each block has three copies, so there are 24 block copies, and the file uses about 3 GB of disk.
FP-05
Spark: plan first, keep data in memory
The same shuffle as MapReduce, but no disk between steps, and a new way to survive failures.
Chapter 04 ended with a problem. Hadoop writes every intermediate result to disk, three times, before the next step can start. Spark started at the University of California, Berkeley, in 2009 to remove that cost. It keeps the map–shuffle–reduce idea, and changes three things.
Change 1: keep intermediate data in memory
A computer's memory reads data many times faster than a spinning disk: often 10 gigabytes per second or more, against about 0.2. Memory is smaller and is lost when power stops, so Hadoop did not trust it. Spark keeps the result of one step in memory and gives it directly to the next step. Only the shuffle still writes files, to the node's own local disk.
Change 2: describe the whole job, then run it
In Spark, you first describe all the steps. Spark does not run any of them yet. It only records them. Look at this short program, in the Python language:
orders = spark.read.parquet("s3://shop/orders") # step: read uk = orders.filter(orders.country == "UK") # step: keep UK rows by_month = uk.groupBy("month").sum("amount") # step: total per month by_month.show() # now Spark runs all three
The first three lines only add steps to a plan. A step that makes a new dataset from an old one is a transformation. The last line asks for a result. A request for a result is an action. Only an action makes Spark do work. We call this way of working lazy evaluation.
Why wait? Because when Spark sees the whole plan, it can improve it before it runs. In this example, it can read only the country, month and amount columns, and drop non-UK rows while it reads. The plan is a set of steps joined by arrows, with no arrow going back to an earlier step. We call this kind of plan a DAG, which stands for directed acyclic graph.
Narrow steps and wide steps
Some steps need only one input partition to make one output partition. "Keep UK rows" is one: each node filters its own rows. We call this a narrow step. Spark runs a chain of narrow steps in one pass over the data, on the same node, with no waiting.
Other steps need data from all partitions. "Total per month" is one: all March rows must meet in one place. We call this a wide step. A wide step needs a shuffle, the same as in MapReduce.
Spark cuts the plan into parts at each wide step. Each part is a stage. Inside a stage, data flows through memory. Between stages, there is a shuffle.
Change 3: remember the recipe, not the result
This is the most important idea in Spark, and it answers a real question. Hadoop survived failures because every result had three copies on disk. Spark keeps results in memory, with no copies. So what happens when a node fails and its memory is gone?
Spark keeps, for every partition, the list of steps that made it from the input data. We call this list the lineage. The input data is safe in replicated storage. So if a partition is lost, Spark runs the same steps again, but only for that one partition.
- The program describes three steps. The dashed boxes are only a plan: no data has moved yet. This plan is also the lineage of every partition.
- An action starts the job. Each node reads its own part of the input files from storage.
- Filter and convert are narrow steps. Each node runs them on its own part, in memory, in one pass. Nothing goes to disk.
- Node B fails. Part 2 was only in node B's memory, so part 2 is lost at every step (red). Parts 1 and 3 are not affected.
- Spark looks at the lineage of part 2: read, then filter, then convert. It runs those three steps again for part 2 only, on node C (amber).
- All parts are complete. The job paid for one partition again, not for three copies of every result.
KEY PART · THE TRADE
Hadoop pays for safety all the time: it writes three copies of every intermediate result, even when nothing fails. Spark pays for safety only when a failure happens: it recomputes the lost part. Failures are common in a large cluster, but each one touches only a small part of the data. So paying only on failure is much cheaper.
For a wide step, a lost output partition needs input from all partitions of the earlier stage. Spark keeps the shuffle files on local disk for this reason, so it does not need to repeat the earlier stage.
Tables, not only functions
Early Spark code worked on collections of objects. Today most Spark code uses a table with named columns, like the orders table above. We call it a DataFrame. You can also write plain SQL. Because Spark knows the columns, it can rewrite your plan into a faster plan before it runs. It moves filters early, reads fewer columns and picks a good way to join. The part of Spark that does this is a query optimizer.
Some programs use the same data many times, for example in a learning loop. You can tell Spark to keep that data in memory after the first use. We call this caching. This is where Spark's speed against Hadoop is largest.
Model · a loop that reads the same data many times
Spark is faster by
35×
Hadoop: chain of jobs35.0 minSpark: read once, then cache60 s
This is a simplified model of one node with 10 GB of data. Hadoop pays 50 s to read from disk, about 150 s to write three copies, and 10 s to start each job. Spark reads from disk once (50 s), then reads from memory at 10 GB per second (1 s for each pass). It leaves out the calculation itself. Try 1 pass: the difference is small. Try 50. Why does the gap grow?
KEY POINT
Spark keeps the map–shuffle–reduce shape and adds three things. It waits for an action, then runs an improved plan. It passes data between narrow steps in memory. And it rebuilds a lost partition from its lineage, instead of keeping copies of every result.
ANALOGY
A kitchen can protect itself in two ways. It can photocopy every half-made dish three times (Hadoop). Or it can keep the recipe, and if a plate falls, cook that one plate again (Spark).
Where this stops working: if the recipe is very long — hundreds of steps — cooking one plate again takes a long time. Spark lets you save a result to storage at a chosen point, which cuts the lineage short. We call that a checkpoint.
COMMON MISTAKE
"Spark keeps all the data in memory, so the data must fit in memory." This is wrong. Spark works on one partition at a time on each processor, and it writes to local disk when memory is full. Shuffles always write to disk. Memory helps most when data is used again and again. A one-pass job on data much larger than memory runs well.
CheckIn the four-line program above, which line makes Spark start work? Which step needs a shuffle?
Only by_month.show(), because it is an action. The first three lines are transformations and only build the plan. groupBy("month") is a wide step: all rows for one month must meet in one place, so it needs a shuffle. The filter is a narrow step.
NOTE
Spark is open source and maintained by the Apache Software Foundation. The current line is 4.2, first released in July 2026. Spark is the engine inside Databricks (chapter 09), and inside the main cloud services for data processing.
FP-06
Store columns, not rows
One change in layout on disk makes analytical questions many times cheaper.
Chapters 02 to 05 made a cluster read faster by using more disks. This chapter reduces how much each disk must read. Go back to question B from chapter 01: total sales for each country last year. It uses three columns. A real orders table can have 50 columns. If we could read only the three we need, we would read about 16 times less.
Disks read in blocks
A disk does not read one value at a time. It reads a whole block (chapter 04), for example 1 MB. If one value you need is inside a block, you pay to read the whole block. So the question is: what is next to what on the disk?
When the values of each row sit together, we call it a row store. Most databases for OLTP work this way, because question A wants a whole row, and one read gets it. When the values of each column sit together, we call it a column store. Column stores are built for OLAP.
Worked example: 1 billion orders, 50 columns, about 8 bytes per value. The table is 400 GB. A row store must read all 400 GB to add up amounts by country and date. A column store reads 3 columns: 3 × 1 billion × 8 bytes = 24 GB. That is about 17 times less, before the next two tricks.
Trick 1: similar values compress well
In a row store, a block holds a mix: dates, names, amounts, text. In a column store, a block holds one kind of value, and the values are often similar or repeat. Similar values are easy to make smaller. Making data smaller without losing information is compression. Two simple methods:
- Store a repeated value once with a count. The country column
UK UK UK UK FR FRbecomesUK×4 FR×2. We call this run-length encoding. It works best when the table is sorted, so equal values are next to each other. - Replace each long text value with a small number, and keep one list of the texts. "United Kingdom" becomes 1, "France" becomes 2. We call this dictionary encoding.
Analytical columns often become 3 to 10 times smaller. Less data on disk means less data to read, so compression makes the question faster, not only the storage cheaper.
Trick 2: skip blocks you do not need
For each block, the system can keep the smallest and the largest value in it. Before it reads a block, it checks these two numbers. If the question asks for March 2025 and a block holds only dates from 2023, the system skips that block. We call this list of smallest and largest values a zone map.
| Block of the date column | Smallest date | Largest date | Question: March 2025 |
|---|---|---|---|
| 1 | 1 Jan 2025 | 28 Feb 2025 | Skip |
| 2 | 1 Mar 2025 | 19 Mar 2025 | Read |
| 3 | 19 Mar 2025 | 8 Apr 2025 | Read |
| 4 | 8 Apr 2025 | 30 May 2025 | Skip |
Zone maps help only when the data on disk is in order of that column. If dates were mixed at random, every block would hold dates from all months, and no block could be skipped. So the order in which you store rows matters a lot.
Model · 400 GB table, 50 columns
Column store reads less by
67×
Row store reads400 GBColumn store reads6.0 GB
This model gives the row store no compression and no skipping, to show each effect on its own. Try this: set 50 columns. The column store still wins, by compression only. Now set 1 column, 10× compression and 90% skipped. The three effects multiply.
An open file format for columns
Column storage is not only inside databases. A common file format stores a table by columns, in groups of rows, with smallest and largest values for each group. This format is Parquet. Spark, warehouses and lakehouses all read it. You saw it in the Spark program: spark.read.parquet(...).
KEY POINT
A column store keeps each column's values together. An analytical question then reads only the columns it uses, those columns compress well, and zone maps let it skip whole blocks. The three savings multiply.
ANALOGY
A school keeps one folder for each pupil, with all their marks inside (row store). To find the average maths mark, a teacher must open every folder. Another school keeps one book for each subject, with every pupil's mark in it (column store). The teacher opens only the maths book.
Where this stops working: when a new pupil arrives, the second school must write in every subject book. That is why column stores are slow at adding or changing single rows. Real column stores collect new rows and write them in large groups.
COMMON MISTAKE
"Column stores are better databases." They are worse for OLTP. To read or change one whole order, a column store must touch 50 separate places, one for each column. A row store touches one. This is the reason, promised in chapter 01, why one layout cannot be best for both kinds of work.
CheckA question uses 5 of 50 columns. The data compresses 4 times. Zone maps skip 75% of blocks. What share of the row store's bytes does the column store read?
Columns: 5 ÷ 50 = 1/10. Compression: 1/4. Skipping 75% leaves 1/4. Multiply: 1/10 × 1/4 × 1/4 = 1/160. The column store reads 400 GB ÷ 160 = 2.5 GB.
FP-07
The data warehouse, and Redshift
A column store, spread over a cluster, that speaks SQL.
Analysts want to ask question-B kinds of question in SQL, get answers in seconds, and work on clean data from every system in the company. They do not want to write Spark programs, and they must not slow down the systems that run the business. The answer is a separate database built only for OLAP.
A separate copy, built for questions
A database that keeps copies of data from many operational systems, organised for analytical questions, is a data warehouse. Data arrives by a regular process with three steps. Copy the data out of each source system. Clean and reshape it. Load it into the warehouse. We call this process ETL, which stands for Extract, Transform, Load.
Modern warehouses are fast enough to do the reshaping themselves. So many teams now load the raw copy first and transform it inside the warehouse with SQL. We call this order ELT.
How the tables are arranged
A warehouse usually has one very large table of events, with one row for each sale, scan or click. Each row holds numbers (amount, quantity) and short codes that point to other tables. We call this table a fact table. The smaller tables describe the codes: who the customer is, what the product is, which store. Each one is a dimension table. Drawn on paper, the fact table sits in the middle with dimensions around it. We call this shape a star schema.
Redshift: the parts
Amazon Web Services (AWS) launched Redshift, its warehouse service, in 2013. It uses chapters 02, 03 and 06 together. It is a column store on a cluster. It runs each SQL question as map–shuffle–reduce work, on all nodes at the same time. We call this way of working MPP, which stands for massively parallel processing.
- The leader node receives SQL, makes the plan, sends the pieces out, and combines the results.
- Each compute node stores and processes part of the data. Each compute node is divided into a few slices. A slice is one share of the node's processors, memory and disk, with its own part of every table.
Choice 1: which slice gets each row
When rows are loaded, Redshift must place each one on a slice. This placement decides how much data must move when two tables are joined. The main choices are:
- KEY: hash one column, as in the shuffle in chapter 03. All rows with the same value of that column go to the same slice. The column is the distribution key.
- EVEN: deal the rows out in turn, like playing cards. The slices get equal amounts, but related rows end up in different places.
- ALL: put a full copy of the table on every node. This is good for small dimension tables.
- AUTO: let Redshift choose, and change the choice as the table grows.
Join orders with customers
KEY on customer_id: the hash sends customer 7 and every order of customer 7 to the same node. Each node can join its own rows.
Choice 2: the order of rows on disk
Chapter 06 showed that zone maps work only when the data is in order. In Redshift, you choose one or more columns, and Redshift stores the rows sorted by them. These columns are the sort key. Most tables use a date as the sort key, because most questions ask about a period of time.
Storage apart from the processors
In the first Redshift design, each node kept its data on its own disks. To add processors, you had to add disks and move data, and the reverse. Newer node types keep all data in S3, AWS's low-cost service for storing files. Chapter 09 explains this kind of storage. They use each node's fast local disk as a short-term copy of the data in use. We call this managed storage. You can then change the number of nodes without moving the data. A further option, Redshift Serverless, adds and removes processing power for you, and charges only for use.
Redshift can also read tables that sit as files in S3, in the open formats you will meet in chapter 09. Since August 2026, this includes version 3 of the Apache Iceberg format.
KEY POINT
A data warehouse such as Redshift is a column store spread over many nodes, which runs each SQL question in parallel on all of them. Two design choices decide its speed: where rows go (distribution key) and in what order they are stored (sort key).
COMMON MISTAKE
"Redshift speaks a form of PostgreSQL's SQL, so we can use it as the application's database." Its SQL looks familiar, but its storage is a column store for OLAP. Thousands of small single-row writes each second, the normal OLTP load, make it slow. Load data into it in large batches.
CheckYou often join a 5-billion-row orders table with a 200-row table of countries. Which distribution style should the countries table have?
ALL. A full copy of 200 rows on every node costs almost nothing. Then every node can join its own orders with the countries, and no order rows need to move.
FP-08
Cassandra: OLTP that never stops
A different problem: huge numbers of small writes and reads by key, with no single point of failure.
Until now, every system served OLAP. Now think about question A at a much larger size. A delivery company records every scan of every parcel: millions of writes each minute. A customer opens the tracking page and wants the scans for one parcel, now. The system must keep working when a node fails, and even when a whole data centre fails. A traditional database on one large computer cannot do this.
Cassandra is a database built for this job. Facebook built it in 2008, using ideas from two earlier systems: Amazon's Dynamo and Google's Bigtable. It is now an open-source Apache project. It answers three questions in its own way.
Question 1: which node has the row, with no central list?
HDFS had a NameNode with a list of every block. For thousands of small requests each second, one central list is a limit and a single point of failure. Cassandra uses the hash trick from chapter 03 instead: any node can calculate where a row lives.
Each table has one column, or a set of columns, that decides the node. This is the partition key. In the delivery example it is the parcel number. Cassandra hashes the partition key into a large number. That number is the row's token.
But chapter 03's rule, "hash mod number of nodes", has a problem here. Suppose you have 4 nodes and add a fifth. The remainder after ÷ 4 and after ÷ 5 differs for about 80% of all tokens. So 80% of the data would have to move to a new node. For a database that never stops, that is not acceptable.
Cassandra uses a ring instead. Picture all possible tokens around a circle, like the minutes on a clock. Each node takes a position on the circle. A row belongs to the first node you reach when you go clockwise from the row's token. We call this method consistent hashing. When you add a node, it takes over only part of the range of one neighbour. The other nodes keep their data.
- Six nodes sit on a ring of tokens from 0 to 99. Each node owns the part of the ring that ends at its position. For example, N3 at 40 owns tokens 26 to 40. (Real tokens go up to about 9 billion billion, and each node takes many small positions, but the rule is the same.)
- A new scan for parcel P42 arrives. Cassandra hashes the partition key "P42" and gets token 37 (amber dot).
- Go clockwise from 37. The first node is N3, at 40. N3 owns this row. Every node can calculate this without asking anyone.
- The table keeps three copies. Cassandra puts the next two copies on the next two nodes clockwise: N4 and N5. The number of copies is the replication factor.
- The client sent the write to N1. N1 acts for the client and sends the write to N3, N4 and N5. The client asked for QUORUM, so N1 answers "done" after two nodes say yes. N5 is slow (dashed line), and gets the write a little later.
- Another client reads P42 through N6, also with QUORUM, so N6 asks two of the three copies: N4 and N5. N5 still has the old value, but N4 has the new one. Each value carries the time it was written, so N6 returns the newest, and tells N5 to update.
Question 2: how many copies must agree?
The number of copies of each row is the replication factor, written N. N = 3 is common. The node that receives a client's request and talks to the copies for it is the coordinator. Any node can be the coordinator.
For each read and each write, the client chooses how many copies must answer before the coordinator replies. This choice is the consistency level. Write W for the number of copies that must confirm a write, and R for the number that must answer a read. The most important level is a majority of the copies. For N = 3, that is 2. We call a majority a quorum.
KEY PART · WHY R + W > N GIVES THE NEWEST VALUE
Take N = 3 copies, W = 2 and R = 2. The write reached at least 2 of the 3 copies. The read asks 2 of the 3 copies. Two groups of 2, chosen from only 3 nodes, must share at least one node. Count it: 2 + 2 = 4, and 4 is more than 3, so at least one node is in both groups. That shared node has the new value, and the timestamp makes it win.
Now take W = 1 and R = 1. The write may reach only N3 so far, and the read may ask only N5. 1 + 1 = 2, which is not more than 3. The groups may not share a node, so the read can return the old value.
R + W > NIf the number of copies you read plus the number of copies you wrote is more than the total number of copies, every read sees the newest write.
When R + W is not more than N, the copies still become equal soon. Cassandra sends late writes to nodes that come back, repairs old copies during reads, and runs a regular repair process. "All copies become equal after a short time, if writes stop" is eventual consistency. It does not mean data is lost. It means a read can be a little out of date.
Model · copies, writes and reads
R + W compared with N
4 > 3
Every read sees the newest write
Nodes that can be down, writes still work1Nodes that can be down, reads still work1
Try this: set W to 1 and R to 1. Writes and reads now work with two nodes down, but reads can be out of date. Then set W to N and R to 1. Reads are fast and always new, but one failed node stops all writes. Every setting trades one thing for another.
Question 3: how to write very fast?
A spinning disk is slow when it jumps to many places, and fast when it writes in one long line. So Cassandra never changes data in place. A write goes through these steps:
- Add the write to the end of a file on disk, in arrival order. This file is the commit log. It is there only to recover after a crash.
- Put the write in a sorted table in memory. This table is the memtable. Now the write is complete, and the client gets its answer.
- When the memtable is full, write it to disk in one piece, in sorted order. This file is never changed after that. We call it an SSTable, which stands for sorted string table.
- Over time there are many SSTables. In the background, Cassandra merges them into fewer, larger ones, and drops old values. We call this compaction.
This design — append to a log, sort in memory, write sorted files, merge later — is the LSM tree, which stands for log-structured merge tree. A delete is also a write: Cassandra writes a marker that says "this row is deleted". The marker is a tombstone. Compaction removes the old data and, later, the tombstone.
Design the table for the question
Cassandra's language looks like SQL. It is called CQL, which stands for Cassandra Query Language. But the rules are different. There are no joins, and a fast query must name the partition key, so that the coordinator can calculate the nodes. So you design one table for each question, and you store the same data in more than one table if two questions need it.
CREATE TABLE scans_by_parcel ( parcel_id text, scan_time timestamp, place text, status text, PRIMARY KEY ((parcel_id), scan_time) ) WITH CLUSTERING ORDER BY (scan_time DESC);
In words: the partition key is parcel_id, so all scans of one parcel are on the same three nodes. Inside that partition, rows are sorted by scan_time, newest first. A column that sets the order of rows inside one partition is a clustering column.
- Fast: "all scans for parcel P42, newest first". One partition, one set of three nodes, already sorted.
- Slow: "all scans with status delayed". The status does not decide the node, so every node must search. Cassandra refuses this by default. If you need it, make a second table with status in the partition key, or copy the data to an OLAP system.
KEY POINT
Cassandra places each row by hashing its partition key onto a ring. It keeps N copies on the next nodes. Each request chooses how many copies must answer. Writes only add to the end of files. This gives very fast writes and no single point of failure. The price: you must design tables around your questions, and large scans are slow.
ANALOGY
A group of clerks sits around a round table. Each clerk handles names whose code falls between their own seat and the seat before them. Every clerk knows the seat plan. So any clerk can take your form and pass it to the right person, and to the next two clerks for safety. A new clerk who sits down takes over part of one neighbour's range only.
Where this stops working: the code is a hash, not the alphabet. "Smith" and "Smyth" end up far apart on the ring. So Cassandra cannot cheaply give you "all names from S to T": neighbouring keys are scattered on purpose, to spread the load.
COMMON MISTAKE
"Cassandra handles big data, so we can do our analytics in it." Cassandra handles a large volume of data, but it answers questions about one partition at a time. Question B, which reads every row, is very slow in it. The usual design copies Cassandra's data into a lake or warehouse for analytics.
COMMON MISTAKE
"Eventual consistency means Cassandra can lose my data." A confirmed write is in the commit log of at least W nodes. Eventual consistency only means that, with a low R and W, a read for a short time can return the value from before the write.
CheckN = 3. A team writes with consistency ONE and reads with ONE. A user changes their address and refreshes the page at once. Can they see the old address? What setting fixes this?
Yes. 1 + 1 = 2, which is not more than 3, so the read can go to a copy that has not got the write yet. Writing and reading with QUORUM fixes it: 2 + 2 = 4, which is more than 3.
CheckWhy does adding a sixth node to a five-node ring move much less data than "hash mod 5" changing to "hash mod 6"?
On the ring, the new node takes over only the tokens between its position and the node before it, from one neighbour. With "mod", the remainder changes for most tokens, so most rows move to a different node.
NOTE
The current stable line is Cassandra 5.0, which added a new kind of index on columns other than the key, and vector search. Cassandra 6 is in pre-release in 2026. Its main change, called Accord, adds full transactions across several partitions. This document describes the classic design, which 6 keeps.
FP-09
The lake, the lakehouse, and Databricks
Cheap files in the cloud, plus a log that makes them behave like tables.
A warehouse stores data in its own format, inside its own system. That has three costs. You pay the warehouse for storage. Only the warehouse can read the data. And it suits tables of numbers and text, but not images, raw logs, or the Python programs that data scientists use. Many teams wanted one cheap place for all their data, which any engine could read.
Object storage
All large clouds sell a service that stores whole files, called objects, by name. You can put a file, get a file, list files and delete a file. You cannot change bytes in the middle of a file: you write a new one. In return, the service is very durable, has no practical size limit, and costs a few cents for each gigabyte each month. We call this object storage. AWS's version is S3. Google and Microsoft have their own.
The data lake
The simple idea: put all data, raw or cleaned, as files in object storage, usually as Parquet. Then read it with Spark or another engine. A store like this is a data lake. It is cheap and open. But plain files are not tables, and teams soon met these problems:
- A job that writes 100 files and fails after 60 leaves 60 files behind. Readers see half a result.
- A reader that lists the folder while a writer is busy sees a mix of old and new files.
- Two writers at the same time can overwrite each other's work.
- To delete one customer's data, for example on a legal request, you must find and rewrite every file that holds it, by hand.
Databases solved these problems decades ago. A database change is all-or-nothing, keeps the data valid, does not mix with other changes in progress, and is permanent once confirmed. These four promises are called ACID: atomic, consistent, isolated, durable. A plain lake has none of them.
A log next to the files
The answer is small and clever. Next to the data files, keep a folder with a numbered list of changes. Each change is one small file that says "add these data files" and "remove those data files". The table, at any moment, is the set of files that the list says is current. We call this list the transaction log. A set of rules for data files plus a transaction log is an open table format.
The trick that makes it safe: object storage can create a file only if no file with that name exists. Two writers both try to create change number 5. Only one succeeds. The other sees the failure, reads the new state, and tries again as change number 6.
- A job writes two data files, A and B. Then it writes change 0 to the log: "add A, B". A reader reads the log and sees version 0: files A and B.
- A second job writes files C and D (dashed). It has not written its change yet. A reader still reads the log, which does not list C and D, so the reader does not see them. No half result is visible.
- The job creates change 1: "add C, D". That one small file is the moment of commit. Now every reader sees version 1: A, B, C and D. The change is all-or-nothing.
- An update changes some rows in A. Object storage cannot edit A, so the job writes a new file A2 with the changed rows, then creates change 2: "remove A, add A2". File A still exists, but version 2 does not include it.
- A reader can ask for an older version. It reads the log only up to change 1, and gets A, B, C and D. This is called time travel. It works because old files stay until a clean-up removes them.
- A job wrote file E and then crashed before it created a change. No change lists E, so no reader ever sees it. A later clean-up deletes such files.
Reading an older version of a table from its log is time travel. The log also stores the list of columns and their types, and statistics such as smallest and largest values for each file. So an engine can skip files the same way a zone map skips blocks, without listing millions of files.
The open table format that Databricks created and released in 2019 is Delta Lake. Another, started at Netflix and now widely supported, is Apache Iceberg. They solve the same problem in slightly different ways, and from 2025 they have moved closer together.
The lakehouse
Put the parts together: open columnar files in object storage, a transaction log that gives ACID changes, and a fast SQL engine on top. You get most of what a warehouse gives, on storage that is cheap and that any engine can read. This design is a lakehouse.
Databricks
The people who created Spark founded a company in 2013 to sell it as a managed service. That company and its platform are Databricks. The platform runs on AWS, Azure or Google Cloud, and keeps the data in object storage in your own cloud account. Its main parts are:
- Spark clusters that start when a job needs them and stop after it, for data processing in Python, SQL or Scala.
- Delta Lake as the default table format, with support for Iceberg tables as well.
- A second engine for SQL, written in C++ for speed, that runs the same plans as Spark. Databricks calls it Photon. SQL warehouses built on it serve analysts and dashboards.
- A catalog that keeps the names of all tables, files and models, controls who may read what, and records which table was made from which. This is Unity Catalog.
- Notebooks, scheduled pipelines and tools for machine learning, all on the same tables.
Bronze, silver, gold
Lakehouse teams often arrange tables in three layers. Bronze holds raw data as it arrived. Silver holds cleaned data: duplicates removed, types fixed, sources joined. Gold holds tables ready for a business use, such as daily totals for a dashboard. Each layer is made from the one before it by a Spark job. We call this arrangement the medallion architecture.
KEY POINT
A lakehouse keeps open columnar files in cheap object storage, and adds a transaction log that makes each change all-or-nothing. Many engines can then use the same files as reliable tables. Databricks is a platform built around this design, with Spark as its engine.
COMMON MISTAKE
"Delta Lake is a storage service, like S3." Delta Lake is a format: Parquet files plus a folder of change files, in your own object storage. Any engine that understands the format can read the table, with or without Databricks.
CheckA job writes three new Parquet files and crashes before it writes change 5. The table was at version 4. What do readers see? Is any data damaged?
Readers see version 4, exactly as before. The three files are not in any change, so they are invisible. No data is damaged. A clean-up job deletes the unlisted files later, and the job can run again.
FP-10
Putting it together
One company, one parcel scan, five systems.
Each system in this document answers the three problems from chapter 02: where each piece is, what happens when nodes fail, and how to keep network traffic low. Each one answers them for one of the two kinds of work from chapter 01. A real company uses several of them together. Follow one scan of one parcel through a typical modern platform.
- The scanner app writes one row: parcel P42 arrived at the Paris depot. The row goes to Cassandra. The partition key P42 decides the three nodes (chapter 08).
- A customer opens the tracking page. It reads partition P42 with QUORUM and gets the scans, newest first, in a few milliseconds. This is OLTP.
- Every few minutes, a job copies new scans as Parquet files into a bronze Delta table in object storage. Cassandra is not asked to do analytics.
- A Spark job cleans the bronze rows, removes duplicate scans, and joins depot details. Narrow steps run in memory; the join needs a shuffle (chapter 05). It writes a silver table with one change in the log (chapter 09).
- Another Spark job groups the silver rows by depot and day, and writes a small gold table. This is the map–shuffle–reduce pattern from chapter 03.
- The gold table goes into Redshift, or a Databricks SQL warehouse reads it in place. Analysts ask question-B kinds of question in SQL, on a column store with a sort key on date (chapters 06 and 07).
- A data scientist trains a model that predicts late parcels, with Spark and Python, on the silver table. It uses the same files, with no extra copy.
Ten years ago, steps 3 to 5 often ran on Hadoop, with HDFS for storage and MapReduce or Hive for processing. Today the same steps usually run on Spark, with object storage instead of HDFS.
The systems side by side
| System | Work | Layout | How a job runs | Good at | Weak at |
|---|---|---|---|---|---|
| Hadoop | OLAP, batch | Files in 128 MB blocks, 3 copies | MapReduce, disk between jobs | Very large batch jobs on owned hardware | Chains of jobs, loops, quick answers |
| Spark | OLAP, batch, some streaming | Reads files (often Parquet) | Plan, then stages in memory, shuffle between stages | Large transformations, loops, ML | Small single-row reads and writes |
| Redshift | OLAP | Column store with distribution and sort keys | MPP SQL on slices | Fast SQL for many analysts | Many small writes; non-table data |
| Cassandra | OLTP | Hash ring, N copies, LSM tree | One partition per request | Huge write volume, always on | Joins, scans, ad-hoc questions |
| Databricks | OLAP, ML, streaming | Parquet plus a transaction log, in object storage | Spark and Photon on separate compute | One platform for data engineering, SQL and ML | OLTP; cost needs watching |
A short history
- 2003–04Google describes its distributed file system, then MapReduce.
- 2006Hadoop starts as an open-source project, with most early work at Yahoo.
- 2008Facebook releases Cassandra as open source.
- 2009–10Spark starts at UC Berkeley, and is released as open source.
- 2013Redshift becomes generally available. Databricks is founded.
- 2019Delta Lake is released as open source.
- 2024–25Cassandra 5.0 (September 2024). Spark 4.0 (May 2025). Databricks adds managed Iceberg tables.
- 2026Hadoop 3.5.0 (April). Spark 4.2 (July). Redshift reads and writes Iceberg v3 (August). Cassandra 6 in pre-release.
Which system for which job
- If you must read and write single items by key, very fast, at very high volume, and never stop: a store like Cassandra.
- If analysts need fast SQL on structured data: a warehouse such as Redshift, or a lakehouse SQL engine.
- If you must transform large raw data, or train models in Python: Spark, often on Databricks.
- If you already run a Hadoop cluster: it keeps working, and most teams plan a move to object storage and Spark.
- If your data fits on one good computer, perhaps up to a few terabytes: a single-node database is often simpler, cheaper and faster than any cluster.
KEY POINT
No single system does everything well, because OLTP and OLAP want opposite layouts. A modern platform uses a key-based store for live work, a lakehouse for shared storage and processing, and a column-store SQL engine for analysts. All of them rest on partitions, replicas, hashing, the shuffle and columns.
FP-11
Where the simple picture stops
What this document left out, and what to learn next.
Uneven keys
Every system here assumes that the hash spreads work evenly. Sometimes one key holds a large part of the data: one very big customer, one busy depot, or the value "unknown". Then one node gets far more work than the others. The whole job waits for that node. This is data skew. Fixes include adding a random number to the hot key so it spreads over several nodes, and combining the parts at the end.
Network splits
Sometimes the network breaks a cluster into two groups that cannot talk to each other. A store with copies must then choose. It can answer from the copies it can reach, which may be out of date. Or it can refuse to answer until the network returns. It cannot both always answer and always be correct during the split. We call this rule the CAP theorem (consistency, availability, and tolerance of network splits). Cassandra's consistency levels let each request choose its side.
Data that never stops arriving
This document mostly showed batch work: take a fixed set of data, process it, finish. Many companies also process each event within seconds of its arrival. We call this stream processing. Its main tools are Apache Kafka, which keeps an ordered log of events, and engines such as Apache Flink and Spark's own streaming mode. The ideas of partitions, replicas and the shuffle all apply, plus new ideas about time and order.
Other systems you will meet
- Snowflake and Google BigQuery: cloud warehouses that separated storage and compute from the start.
- ClickHouse: a very fast open-source column store, often used for logs and events.
- Trino (formerly Presto): a SQL engine that reads lake tables in place.
- DuckDB: a column-store engine that runs inside one program on one computer, and handles surprisingly large data.
- DynamoDB and ScyllaDB: key-based stores close in design to Cassandra.
What to learn next
- Read Spark's plan for a real query (
explain()) and find the shuffles. - Design a Cassandra table for two different questions, and see why you need two tables.
- Look at the
_delta_logfolder of a real Delta table, and read one change file. - Compare a Redshift question with and without a good sort key, and look at how many blocks it reads.
Deep dive · Inside Spark's speed
Three parts of Spark do most of the work. The query optimizer rewrites the plan with rules, and chooses join methods from table statistics. Spark also generates compact machine-level code for whole stages, so data does not pass through general-purpose objects. And since version 3, it can change the plan during the run. After a shuffle, it looks at the real sizes of the partitions. Then it merges small ones, splits skewed ones, or changes the join method. This is called adaptive query execution.
Deep dive · Why Cassandra reads can get slow
A read may need to check the memtable and several SSTables. Each SSTable has a small structure called a Bloom filter. It says "this key is certainly not here" or "it may be here", so most files are skipped. But tables with many deletes collect tombstones, and a read must step over every tombstone in its range until compaction removes them. Queue-like designs, where rows are written and soon deleted, are a classic cause of slow Cassandra reads.
Deep dive · Keeping a warehouse fast
Over time, deletes and updates leave gaps, and new rows arrive out of order. Redshift sorts and cleans tables in the background, and keeps statistics that the optimizer uses to choose plans. It can also add temporary extra clusters when many analysts ask questions at the same time. Most of this is now automatic, but the distribution key and the sort key are still design choices that you should understand.
Deep dive · Delta Lake and Iceberg
Both formats keep Parquet files and a log of changes. Delta Lake's log is a numbered list of change files with regular summary files. Iceberg uses a tree of metadata files: a table points to a snapshot, which points to lists of data files with statistics. Iceberg also hides the details of how a table is split into folders, so you can change that split later without rewriting the data. Version 3 of the Iceberg format shares several features with Delta Lake, such as a compact way to mark deleted rows. Both projects aim to let one copy of the data be read as either format.
FP-GL
Glossary
Every term, in the order it first appears. Type to filter.
- Row
- One line of a table, for example one order. FP-01
- Column
- One named field of a table, with one kind of value for every row. FP-01
- SQL
- Structured Query Language: the standard language for asking questions of tables. FP-01
- OLTP
- Online Transaction Processing: many small reads and writes of whole rows, very fast. FP-01
- OLAP
- Online Analytical Processing: few large questions that read some columns of very many rows. FP-01
- Scaling up
- Getting more power by buying one bigger computer. FP-02
- Scaling out
- Getting more power by connecting more ordinary computers. FP-02
- Node
- One computer in a cluster. FP-02
- Cluster
- A group of nodes, connected by a network, that work as one system. FP-02
- Partition
- One part of the data, kept on one node. FP-02
- Replica
- A copy of a partition on a different node, so data survives a failure. FP-02
- Data locality
- The rule of sending the program to the node that holds the data, not the reverse. FP-02
- Key-value pair
- A label (the key) and a value, for example ("UK", 42). FP-03
- Map
- The step where each node turns its own records into key-value pairs, alone. FP-03
- Shuffle
- The step that moves all pairs with the same key to the same node, across the network. FP-03
- Reduce
- The step that combines all values for one key into a result. FP-03
- Hash
- A calculation that turns any value into a number; the same value always gives the same number. FP-03
- Combiner
- A reduce step that runs on the map node before the shuffle, to send less data. FP-03
- MapReduce
- The pattern of map, shuffle and reduce, with automatic restart of failed tasks. FP-03
- Hadoop
- Open-source software that stores data across a cluster and runs MapReduce on it. FP-04
- HDFS
- Hadoop Distributed File System: stores large files as blocks with copies on many nodes. FP-04
- Block
- A fixed-size piece of stored data, such as 128 MB in HDFS or 1 MB in Redshift. FP-04
- DataNode
- An HDFS node that stores blocks. FP-04
- NameNode
- The HDFS node that keeps the list of which blocks make each file and where they are. FP-04
- YARN
- The part of Hadoop that shares processors and memory between jobs. FP-04
- Hive
- A layer that turns SQL into Hadoop jobs, and keeps a list of tables. FP-04
- Transformation
- In Spark, a step that makes a new dataset from an old one. It only adds to the plan. FP-05
- Action
- In Spark, a request for a result. It makes Spark run the plan. FP-05
- Lazy evaluation
- Waiting until a result is needed before doing any work. FP-05
- DAG
- Directed acyclic graph: a plan of steps joined by arrows, with no arrow back to an earlier step. FP-05
- Narrow step
- A step where each output partition needs only one input partition. No shuffle. FP-05
- Wide step
- A step where output partitions need data from all input partitions. Needs a shuffle. FP-05
- Stage
- A part of a Spark plan between two shuffles, run in memory. FP-05
- Lineage
- The list of steps that made a partition, used to rebuild it after a failure. FP-05
- DataFrame
- A Spark table with named columns. FP-05
- Query optimizer
- The part of a system that rewrites a plan into a faster plan before it runs. FP-05
- Caching
- Keeping data in memory after first use, so later steps read it fast. FP-05
- Checkpoint
- A result saved to storage on purpose, to cut a long lineage short. FP-05
- Row store
- A layout on disk that keeps the values of each row together. Good for OLTP. FP-06
- Column store
- A layout on disk that keeps the values of each column together. Good for OLAP. FP-06
- Compression
- Making data smaller without losing information. FP-06
- Run-length encoding
- Storing a repeated value once, with a count. FP-06
- Dictionary encoding
- Replacing long values with small numbers, plus one list of the values. FP-06
- Zone map
- The smallest and largest value in each block, used to skip blocks. FP-06
- Parquet
- An open file format that stores a table by columns, with statistics. FP-06
- Data warehouse
- A separate database, built for OLAP, with copies of data from many systems. FP-07
- ETL
- Extract, Transform, Load: copy data out, reshape it, then load it into the warehouse. FP-07
- ELT
- Extract, Load, Transform: load raw data first, then reshape it inside the warehouse. FP-07
- Fact table
- A large table with one row for each event, holding numbers and codes. FP-07
- Dimension table
- A small table that describes the codes in a fact table. FP-07
- Star schema
- A fact table in the middle, joined to dimension tables around it. FP-07
- Redshift
- Amazon's data warehouse service: a column store that runs SQL in parallel on a cluster. FP-07
- MPP
- Massively parallel processing: running one question on all nodes at the same time. FP-07
- Leader node
- The Redshift node that plans each question and combines the results. FP-07
- Compute node
- A Redshift node that stores and processes part of the data. FP-07
- Slice
- One share of a compute node, with its own part of every table. FP-07
- Distribution key
- The column whose hash decides which slice gets each row. FP-07
- Sort key
- The column or columns that set the order of rows on disk. FP-07
- Managed storage
- Redshift keeps all data in S3 and uses local disks as a short-term copy, so compute can change size alone. FP-07
- Cassandra
- An open-source database for OLTP at very large scale, with no single point of failure. FP-08
- Partition key
- In Cassandra, the column or columns whose hash decides which nodes hold a row. FP-08
- Token
- The number that Cassandra gets by hashing a partition key. FP-08
- Consistent hashing
- Placing nodes and keys on a ring of tokens; a key belongs to the next node clockwise. FP-08
- Replication factor
- The number of copies of each row, N. FP-08
- Coordinator
- The node that receives a client's request and talks to the copies for it. FP-08
- Consistency level
- How many copies must answer before a read or write completes. FP-08
- Quorum
- A majority of the copies, for example 2 of 3. FP-08
- Eventual consistency
- All copies become equal after a short time, if writes stop. FP-08
- Commit log
- A file on disk where each write is added to the end, for recovery after a crash. FP-08
- Memtable
- A sorted table in memory that holds recent writes. FP-08
- SSTable
- Sorted string table: a sorted file on disk that is never changed after it is written. FP-08
- Compaction
- Merging many SSTables into fewer, and dropping old values. FP-08
- LSM tree
- Log-structured merge tree: append to a log, sort in memory, write sorted files, merge later. FP-08
- Tombstone
- A marker written to say that a row is deleted. FP-08
- CQL
- Cassandra Query Language: looks like SQL, but has no joins and needs the partition key. FP-08
- Clustering column
- A column that sets the order of rows inside one Cassandra partition. FP-08
- Object storage
- A cloud service that stores whole files by name, cheaply and at any size, such as S3. FP-09
- Data lake
- All of a company's data kept as files in object storage. FP-09
- ACID
- Atomic, consistent, isolated, durable: the four promises of a reliable database change. FP-09
- Transaction log
- A numbered list of changes that says which data files make up a table. FP-09
- Open table format
- Rules for data files plus a transaction log, that any engine can read. FP-09
- Time travel
- Reading an older version of a table from its transaction log. FP-09
- Delta Lake
- The open table format created by Databricks. FP-09
- Apache Iceberg
- An open table format started at Netflix and now widely supported. FP-09
- Lakehouse
- Object storage plus an open table format plus a fast SQL engine. FP-09
- Databricks
- A company and platform, from the creators of Spark, built around the lakehouse. FP-09
- Photon
- Databricks' fast SQL engine, written in C++. FP-09
- Unity Catalog
- Databricks' catalog for table names, permissions and lineage. FP-09
- Medallion architecture
- Bronze (raw), silver (clean) and gold (business-ready) layers of tables. FP-09
- Data skew
- When one key holds much more data than others, so one node does most of the work. FP-11
- CAP theorem
- During a network split, a store with copies cannot always answer and always be correct. FP-11
- Stream processing
- Processing each event within seconds of arrival, without a fixed end. FP-11