Reading a CSV crashes with a decode error on one row and I do not understand what the file has done wrong
A monthly export I read with a short script. Most months it works. This month it fails partway through with an error about a byte that cannot be decoded in position something.
I opened the file in a spreadsheet and it looks completely normal. I found the row it stops on and it contains a name with an accent in it, which does not seem like it should be a crisis in the year we are in.
I have found suggestions to add an argument telling it to ignore errors, which does make it run - and now I have names with missing characters, which feels like I have hidden the problem rather than solved it.
What is actually going on, and what is the correct fix rather than the one that silences it?
@utf8_sig_sam · 3w ago
For an export that opens fine in a spreadsheet, there are two likely candidates and both are common.
A legacy single-byte encoding, which is what a lot of older business systems still produce. If the file came out of an accounting package or an export written years ago, this is very likely. Try the common Western European one and see if the accented characters come out correct.
UTF-8 with a byte order mark. Certain spreadsheet applications write three extra bytes at the start of the file to flag it as UTF-8. Decoding as plain UTF-8 works but leaves an invisible character at the start of your first column name, which produces the maddening bug where the first column does not match by name and everything else does. There is a specific encoding name for reading this that strips the mark, and using it is the fix.
How to tell rather than guess: open the file in a text editor that shows encoding, or read the first few bytes in Python and look at them. That takes a minute and removes the guessing.
One warning about trying encodings until it stops erroring: some wrong encodings do not error, they just produce wrong characters. A single-byte encoding will happily decode any byte to something, so the script runs and your names are subtly mangled. Always look at the accented row after you change it, rather than trusting that no exception means correct.
Reply
Report