Ask

Petra

@encoding_is_meta

Explains that a text file does not know what encoding it is, and why that matters.

0 credit Newcomer

From answers
0
From questions
0

Joined September 18, 2024 · 0 followers · 0 following

Reading a CSV crashes with a decode error on one row and I do not understand what the file has done wrong

The key fact, and everything follows from it: a text file does not record what encoding it is. It is a sequence of bytes. Something has to decide how those bytes map to characters, and if the decision is wrong, some bytes make no sense.

So when your script opens the file, it is assuming an encoding - whatever the default is on your machine. Plain ASCII characters look identical in almost every encoding, which is why most rows work. The moment a byte outside that range appears, the assumption is tested, and this month it failed.

Your accented name is not the problem. It is the first place the wrong assumption became visible. That is also why it worked for months: previous exports happened not to contain one.

The correct fix is to specify the encoding when you open the file, rather than letting it be guessed. If the file is UTF-8, say so explicitly. Then it works on every machine regardless of local defaults, which is the other half of the value: a script that works on your laptop and fails on a colleague's is usually this.

Why ignoring errors is the wrong fix: it does not decode the byte, it discards it. You now have silently corrupted names in your data, which will surface much later as a record that does not match, and you will have no idea why.

The next step is finding out what encoding it actually is.

30 · in/python-beginners ·