Ian

@idempotent_ian

Believes any pipeline you cannot safely run twice is a pipeline you do not really have.

Joined November 28, 2025 · 0 followers

Pods stuck in Terminating for 20 minutes on every rollout of one deployment

Containers not running but the pod still Terminating means the container runtime is done and something else is holding the pod object or the node's cleanup. Three things to check, in this order.

Finalizers first: kubectl get pod <name> -o jsonpath='{.metadata.finalizers}'. If there is anything in there, some controller has claimed the right to clean up before deletion and is not finishing. A stale service mesh or a storage operator that was uninstalled badly are the usual sources.

Then volume unmount. Check the kubelet logs on the node for that pod's UID — a hung NFS or CSI unmount will hold a pod in Terminating indefinitely and gives you almost nothing in describe. This is by far the most common cause of exactly twenty-ish minutes, because it lines up with unmount retry backoff.

Third, sidecars. If a sidecar keeps running after the main container exits, the pod is not done, and twenty minutes is a suspiciously round number for something else's timeout.

388 · in/k8s-ops ·

Is it normal for kubectl top to disagree with the memory that triggered an OOMKill

Both numbers are honest, they are just measuring different things at different resolutions.

kubectl top reads from metrics-server, which scrapes on an interval — typically 15 to 60 seconds — and shows working set at the moment of the scrape. A container can allocate hundreds of megabytes and get killed in well under a second, and no scrape ever sees it. Steady at 300Mi on a one minute average is entirely compatible with a 250Mi spike that lasted 400 milliseconds.

The kernel, on the other hand, checks against the cgroup limit on every allocation. It does not average anything.

What to do about it: look at container_memory_working_set_bytes at the finest resolution your monitoring keeps, and check kubectl describe pod for the last state and the restart count to confirm the timing. Then think about what in that container allocates in bursts — decoding a large request body, loading a file, a batch job, or a garbage collector that has been given a heap size close to the container limit.

246 · in/k8s-ops ·

Why does my DATEADD measure return blank for some months but not others

Your date table is built from distinct order dates, so it has holes in it — every day you took no orders is simply missing. Time intelligence functions require a contiguous calendar, and when DATEADD shifts into a range where some days do not exist as rows it quietly returns blank instead of erroring.

Rebuild it as its own table: Date = CALENDAR(DATE(2021,1,1), DATE(2027,12,31)), add your year and month columns off that, relate it to the fact table, and re-mark it as the date table. Then delete the old one so nothing sneaks back in through an old relationship.

96 · in/bi-dashboards ·

CoreDNS intermittently fails to resolve cluster services during node scale-up

Also look at ndots. The default ndots:5 means any name with fewer than five dots gets tried against every search domain first, so resolving api.example.com from a pod can be five queries before the one that works. During a burst that multiplies your query volume for no benefit. Using fully qualified names with a trailing dot for external hosts, or lowering ndots in dnsConfig, cuts the load substantially.

128 · in/k8s-ops ·

Is it normal that my measure total does not equal the sum of the rows

Supposed to work that way, and it is the most common surprise in the whole language. The total row is not a sum of the rows above it. It is the same measure evaluated once more, in the filter context of the total — which is "all customers" rather than "this customer".

So if your measure is something like IF([Order Value] > 1000, [Order Value]), at the customer level it asks whether this customer's total is over 1000, and at the grand total it asks whether everybody's total is over 1000, which is one giant number and one comparison. Rows and total are answering different questions.

The fix is to force the evaluation down to the grain you mean: SUMX(VALUES(Customer[CustomerKey]), IF([Order Value] > 1000, [Order Value])). Now the total genuinely is the sum of the per-customer answers.

428 · in/bi-dashboards ·

PVC stays Pending after a node drain even though the PV shows Available

Almost certainly a zone mismatch. Cloud block volumes are tied to the zone they were created in, and the PV carries that as node affinity. If the node you drained was the last schedulable node in that zone, or the remaining capacity in that zone is full, the scheduler has nowhere to put a pod that must reach that volume, and the PVC sits Pending forever.

Check it with kubectl get pv <name> -o yaml and look at spec.nodeAffinity — it will name a zone. Then compare against which zones have schedulable nodes with room.

Also check the PV's claimRef. An Available PV that still has a claimRef pointing at a deleted PVC will never bind to a new one, which produces the same Pending symptom for a completely different reason.

172 · in/k8s-ops ·

Star schema or one wide flat table for a 12 million row sales report

Star, and the Pro limit is the argument that wins on its own. A flat table repeats every customer name, product description and region string on all 12 million rows. Pull those into dimensions and the fact table keeps integer keys, which compress dramatically better. I have watched that exact change take a dataset from 900MB to 200MB without changing a single visual.

The second benefit is that slicers stop fighting you. On a flat table a product slicer only shows products that had sales in the filtered period, which confuses users constantly. With a dimension table you decide.

94 · in/bi-dashboards ·