William Friedman: when codebreaking became a science

Breaking codes with statistics, part three of six.

By the start of the twentieth century, codebreaking had a toolkit. Count the letters, as al-Kindi had described. Look for repeated fragments and measure the gaps between them, as Kasiski had published. In skilled hands these methods worked, but they depended heavily on intuition, experience and luck. Two codebreakers given the same message might take quite different routes, and neither could easily explain why one guess was better than another.

William Friedman changed that. He turned codebreaking into something that could be measured, taught and repeated.

An unlikely start

Friedman was born in 1891 in the Russian Empire and grew up in the United States. He trained in genetics, not languages or mathematics. In 1915 he took a job at Riverbank Laboratories, a private research estate outside Chicago funded by a wealthy businessman, George Fabyan, to work on plant breeding.

Riverbank also ran a project trying to find hidden messages in the works of Shakespeare, part of an old theory that Francis Bacon had written the plays. Friedman was drawn into the cipher work. So was Elizebeth Smith, a researcher on the same project, whom he married and who became a formidable codebreaker in her own right. The Shakespeare theory went nowhere. The skills the two of them built did not.

When the United States entered the First World War, Riverbank became a training ground for army codebreakers, with Friedman as one of the teachers.

Counting coincidences

Friedman's most influential idea is called the index of coincidence. He set it out in a monograph written in 1920.

The idea is easy to state. Take a piece of text, pick two letters from it at random, and ask how likely they are to be the same letter. In a text made of truly random letters, matches are rare, because every letter is as likely as any other. In ordinary English, matches happen noticeably more often, because a few letters such as "e" and "t" turn up far more than the rest.

That single number becomes a powerful instrument.

• Language or noise. A scrambled text that swaps letters one for one keeps the matching rate of the language underneath. A text scrambled with many alphabets drifts towards the random rate. Measuring the rate tells you which kind of cipher you are facing.
• Finding the key length. For a cipher of the Vigenère kind, try splitting the text into groups by a guessed keyword length. When the guess is right, each group behaves like ordinary language and the matching rate jumps. When it is wrong, the rate stays low.
• Lining up messages. Slide two scrambled messages against each other and count how often the letters match. If both were scrambled the same way from the same starting point, the matching rate rises to that of the language. Codebreakers could tell when two messages shared a setting without knowing what either said.

This was a different kind of codebreaking. It did not depend on guessing a word or spotting a lucky repeat. It looked at the statistics of the whole text and asked whether they looked like language or like chance.

A science, with a name

Friedman is credited with coining the word "cryptanalysis" for the analysis of secret writing by people who do not hold the key. The new word fitted the new approach: a discipline with methods that could be written down, checked and taught to others, rather than a private art.

In 1930 he was asked to lead the US Army's new Signal Intelligence Service. He hired a small team of young mathematicians and trained them in his methods. That group went on to help break PURPLE, the machine cipher used for Japan's most sensitive diplomatic messages, before the United States entered the Second World War.

Friedman's training texts were used to teach generations of codebreakers. He spent much of his later career shaping how the United States protected its own communications, as well as how it read other people's.

What it teaches

Friedman's lesson is that protection has to be judged by the statistics of what it produces, not by how complicated it looks. A cipher can be elaborate, slow to use and impressive on paper, and still give itself away to a single, well-chosen count. Anything that claims to keep data secure has to hold up against an adversary who counts and measures everything.

Next in the series: Kullback, one of Friedman's first recruits, and the statistics he wrote down for codebreakers.

If you hold data that has to stand up to that kind of scrutiny, join the waitlist for early access.