Why do English spelling and pronunciation disagree so violently: was it ever consistent?

The reason the spelling did not follow the sound is largely the printing press, which arrived in the middle of all this and froze things in place.

Before printing, spelling was regional and flexible: scribes wrote roughly what they heard, and the same word appears in many forms. Printing created pressure for a standard, and the standard was based on the usage of a particular time and place, right as the pronunciation was in the middle of shifting.

A few extra complications from that period:

Early printers included foreign workers who applied conventions from their own languages, which is where some odd spellings come from.

Lines were sometimes padded by adding letters to justify text, and some of those additions stuck.

Some words were deliberately respelled to look more like their Latin ancestors, by people who thought that was more correct. This is where the b in debt and the s in island come from, neither was ever pronounced, and the s in island is doubly odd because the word is not even related to the Latin it was made to resemble.

So you have a spelling standardised at a moment of maximum turbulence, then partly re-edited on etymological grounds, and then left alone for four hundred years while the language kept moving.

It was consistent, and then two things happened in the wrong order.

The spelling was largely reasonable when it was set down. In Middle English, the letters mostly corresponded to sounds. The silent k in knight was pronounced. The gh was a real consonant, a sound at the back of the throat, still present in some dialects and in related languages.

Then the pronunciation changed enormously and the spelling did not follow.

The biggest single cause is a large shift in the long vowels over roughly the fifteenth to seventeenth centuries. The vowels moved systematically, the sound written as an i became the diphthong we now say in time, and so on. Every long vowel ended up representing a different sound from the one it originally denoted, which is why the vowel letters in English match no other European language.

At the same time various consonants stopped being pronounced, the k in knight, the gh in night, the l in walk.

So the spelling is not arbitrary at all. It is a reasonably accurate record of how the language sounded several hundred years ago. You are reading a snapshot of a pronunciation nobody uses any more, and the through-though-tough family is the clearest fossil of it.

It was consistent, and then the two changes happened in the wrong order. That is the whole story in a sentence.

Through, though, thought, tough, thorough. Five words, five sounds, one spelling pattern.

And the third ingredient: English has borrowed vocabulary from a very large number of languages and generally kept the source spelling conventions rather than assimilating them.

That is why the same sound has so many spellings. Words from French, Latin, Greek, Norse and later from everywhere else each brought their own patterns, and English absorbed them all without normalising.

Your colonel is a good example of a different mechanism again: the word came into English by two routes with two forms, and the spelling of one won while the pronunciation of the other did. That is uncommon and it is not unique.

The practical consolation for a learner: the chaos is not evenly distributed. The most irregular words are also the most common ones, which means you meet them early, they feel representative, and they are not. Longer and more technical vocabulary is far more regular, because it mostly arrived later and from Latin and Greek with consistent rules.

And the patterns do exist once you know their origin. The gh words are a single historical class. The silent initial consonants are another. Learning the handful of families is much more effective than learning words one at a time, and it is the thing textbooks rarely explain because it requires the history.