It reads about 4000 rows, does some filtering and grouping, and the totals are roughly ten percent off what I get doing it by hand for a small sample. No exception, no warning, just wrong. I have been adding print statements semi-randomly for two hours and I am clearly doing this badly, so I want to know what an actual method looks like.
Shrink the input before you do anything else. Cut the file down to five rows where you know the right answer by hand, ideally including one weird row, and run against that until it is correct. Debugging against 4000 rows means every experiment costs you attention you should be spending on the logic, and half the time the bug is in one specific row shape that you cannot see in the aggregate anyway. Once the five-row case is right, put the full file back and see whether the ten percent moved.
Ten percent on a small sample is going to be one or two rows being dropped or double counted, which I can actually look at. Doing this now.
And keep that five-row file. It becomes your first test, which means the bug cannot come back silently.
Gentle disagreement with the debugger-first advice. Print statements are fine, the problem is that yours are unstructured. Print with labels and values together, print at stage boundaries rather than inside tight loops, and delete them as soon as they have told you something. I have written Python for years and still solve most data bugs with well-placed prints; the debugger earns its keep on control flow bugs rather than data bugs.
For a ten percent discrepancy on tabular data specifically, count things at each stage before you go near the logic. Print the row count after reading, after filtering, and after grouping, and check the sum of the group totals against the sum of the raw column. In my experience that finds it immediately, and the answer is nearly always one of three things: rows silently dropped by a type conversion, blank or whitespace values not matching your filter, or duplicate keys collapsing during the group.
@row_level_sam · 2w ago · 2 replies
Learn breakpoint(). Put it on the line before the suspect calculation, run the script normally, and you land in a prompt where you can inspect any variable, step one line at a time and evaluate expressions against the real state. Four commands get you almost everywhere: n for next line, s to step into a call, c to continue, and p to print an expression. Twenty minutes to learn it, and it replaces the entire print-and-rerun cycle you are stuck in.
Reply
Report
@darning_dee · 2w ago
The underrated part is that you can call functions from that prompt with the actual data loaded, which is how you test a fix before writing it.
Reply
Report