Ask
29

Reading a CSV crashes with a decode error on one row and I do not understand what the file has done wrong

A monthly export I read with a short script. Most months it works. This month it fails partway through with an error about a byte that cannot be decoded in position something.

I opened the file in a spreadsheet and it looks completely normal. I found the row it stops on and it contains a name with an accent in it, which does not seem like it should be a crisis in the year we are in.

I have found suggestions to add an argument telling it to ignore errors, which does make it run - and now I have names with missing characters, which feels like I have hidden the problem rather than solved it.

What is actually going on, and what is the correct fix rather than the one that silences it?

3 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @utf8_sig_sam · 3w ago

    For an export that opens fine in a spreadsheet, there are two likely candidates and both are common.

    A legacy single-byte encoding, which is what a lot of older business systems still produce. If the file came out of an accounting package or an export written years ago, this is very likely. Try the common Western European one and see if the accented characters come out correct.

    UTF-8 with a byte order mark. Certain spreadsheet applications write three extra bytes at the start of the file to flag it as UTF-8. Decoding as plain UTF-8 works but leaves an invisible character at the start of your first column name, which produces the maddening bug where the first column does not match by name and everything else does. There is a specific encoding name for reading this that strips the mark, and using it is the fix.

    How to tell rather than guess: open the file in a text editor that shows encoding, or read the first few bytes in Python and look at them. That takes a minute and removes the guessing.

    One warning about trying encodings until it stops erroring: some wrong encodings do not error, they just produce wrong characters. A single-byte encoding will happily decode any byte to something, so the script runs and your names are subtly mangled. Always look at the accented row after you change it, rather than trusting that no exception means correct.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @encoding_is_meta · 3w ago

    The key fact, and everything follows from it: a text file does not record what encoding it is. It is a sequence of bytes. Something has to decide how those bytes map to characters, and if the decision is wrong, some bytes make no sense.

    So when your script opens the file, it is assuming an encoding - whatever the default is on your machine. Plain ASCII characters look identical in almost every encoding, which is why most rows work. The moment a byte outside that range appears, the assumption is tested, and this month it failed.

    Your accented name is not the problem. It is the first place the wrong assumption became visible. That is also why it worked for months: previous exports happened not to contain one.

    The correct fix is to specify the encoding when you open the file, rather than letting it be guessed. If the file is UTF-8, say so explicitly. Then it works on every machine regardless of local defaults, which is the other half of the value: a script that works on your laptop and fails on a colleague's is usually this.

    Why ignoring errors is the wrong fix: it does not decode the byte, it discards it. You now have silently corrupted names in your data, which will surface much later as a record that does not match, and you will have no idea why.

    The next step is finding out what encoding it actually is.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @detect_then_decide · 3w ago

    The pattern worth adopting so this stops recurring: decode once, at the boundary, and work in text after that.

    Concretely, for a recurring monthly file:

    Pin the encoding in your script with a comment saying where it came from. Not a guess left implicit, a stated fact about that source.

    Ask the person producing it what it is, if you can. It is a reasonable question and often they can change it to UTF-8 permanently, which is the real fix. This is the same class of solution as asking for ISO dates, and it works about as often.

    Fail loudly if it changes. If a future file arrives in a different encoding, you want the exception rather than silently mangled data. This is the argument against making the script tolerant.

    If you genuinely receive files from many sources with unknown encodings, there are libraries that guess by statistical analysis. They are useful and they are guesses: good enough for a one-off exploration, not something to build a monthly pipeline on without checking the output.

    And keep the original file. When you find out three months later that a name was corrupted, being able to re-read the raw bytes correctly is the difference between fixing it and reconstructing it.

    20
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report