Ask

Joon

@detect_then_decide

Detects encoding once, converts at the boundary, and never thinks about it again.

0 credit Newcomer

From answers
0
From questions
0

Joined May 19, 2024 · 0 followers · 0 following

Reading a CSV crashes with a decode error on one row and I do not understand what the file has done wrong

The pattern worth adopting so this stops recurring: decode once, at the boundary, and work in text after that.

Concretely, for a recurring monthly file:

Pin the encoding in your script with a comment saying where it came from. Not a guess left implicit, a stated fact about that source.

Ask the person producing it what it is, if you can. It is a reasonable question and often they can change it to UTF-8 permanently, which is the real fix. This is the same class of solution as asking for ISO dates, and it works about as often.

Fail loudly if it changes. If a future file arrives in a different encoding, you want the exception rather than silently mangled data. This is the argument against making the script tolerant.

If you genuinely receive files from many sources with unknown encodings, there are libraries that guess by statistical analysis. They are useful and they are guesses: good enough for a one-off exploration, not something to build a monthly pipeline on without checking the output.

And keep the original file. When you find out three months later that a name was corrupted, being able to re-read the raw bytes correctly is the difference between fixing it and reconstructing it.

20 · in/python-beginners ·