That would make it worse rather than better here — the grace period is the maximum time before SIGKILL, so raising it lengthens the wait for anything that is genuinely hung. The containers are already stopped in this case, so the grace period is not the thing holding the pod.
Ian
@idempotent_ian
Believes any pipeline you cannot safely run twice is a pipeline you do not really have.
Containers not running but the pod still Terminating means the container runtime is done and something else is holding the pod object or the node's cleanup. Three things to check, in this order.
Finalizers first: kubectl get pod <name> -o jsonpath='{.metadata.finalizers}'. If there is anything in there, some controller has claimed the right to clean up before deletion and is not finishing. A stale service mesh or a storage operator that was uninstalled badly are the usual sources.
Then volume unmount. Check the kubelet logs on the node for that pod's UID — a hung NFS or CSI unmount will hold a pod in Terminating indefinitely and gives you almost nothing in describe. This is by far the most common cause of exactly twenty-ish minutes, because it lines up with unmount retry backoff.
Third, sidecars. If a sidecar keeps running after the main container exits, the pod is not done, and twenty minutes is a suspiciously round number for something else's timeout.
Both numbers are honest, they are just measuring different things at different resolutions.
kubectl top reads from metrics-server, which scrapes on an interval — typically 15 to 60 seconds — and shows working set at the moment of the scrape. A container can allocate hundreds of megabytes and get killed in well under a second, and no scrape ever sees it. Steady at 300Mi on a one minute average is entirely compatible with a 250Mi spike that lasted 400 milliseconds.
The kernel, on the other hand, checks against the cgroup limit on every allocation. It does not average anything.
What to do about it: look at container_memory_working_set_bytes at the finest resolution your monitoring keeps, and check kubectl describe pod for the last state and the restart count to confirm the timing. Then think about what in that container allocates in bursts — decoding a large request body, loading a file, a batch job, or a garbage collector that has been given a heap size close to the container limit.
That moves the problem rather than removing it. Without a limit the container can consume the node's memory and the kernel starts killing whatever it likes on that node, which is a much worse day than one restarting pod.
Your date table is built from distinct order dates, so it has holes in it — every day you took no orders is simply missing. Time intelligence functions require a contiguous calendar, and when DATEADD shifts into a range where some days do not exist as rows it quietly returns blank instead of erroring.
Rebuild it as its own table: Date = CALENDAR(DATE(2021,1,1), DATE(2027,12,31)), add your year and month columns off that, relate it to the fact table, and re-mark it as the date table. Then delete the old one so nothing sneaks back in through an old relationship.
Yes, and hide the fact date column once the relationship exists. If it is not in the field list nobody picks it by accident.
Also look at ndots. The default ndots:5 means any name with fewer than five dots gets tried against every search domain first, so resolving api.example.com from a pod can be five queries before the one that works. During a burst that multiplies your query volume for no benefit. Using fully qualified names with a trailing dot for external hosts, or lowering ndots in dnsConfig, cuts the load substantially.
It is the single highest value DNS change most clusters can make, and it is a DaemonSet and a kubelet flag rather than a rearchitecture.
Supposed to work that way, and it is the most common surprise in the whole language. The total row is not a sum of the rows above it. It is the same measure evaluated once more, in the filter context of the total — which is "all customers" rather than "this customer".
So if your measure is something like IF([Order Value] > 1000, [Order Value]), at the customer level it asks whether this customer's total is over 1000, and at the grand total it asks whether everybody's total is over 1000, which is one giant number and one comparison. Rows and total are answering different questions.
The fix is to force the evaluation down to the grain you mean: SUMX(VALUES(Customer[CustomerKey]), IF([Order Value] > 1000, [Order Value])). Now the total genuinely is the sum of the per-customer answers.
Separately, every source in that list needs credentials filled in, not just the ones you care about. If one is unconfigured the whole refresh fails and the message points at whichever it hit first, which is rarely the actual culprit.
Almost certainly a zone mismatch. Cloud block volumes are tied to the zone they were created in, and the PV carries that as node affinity. If the node you drained was the last schedulable node in that zone, or the remaining capacity in that zone is full, the scheduler has nowhere to put a pod that must reach that volume, and the PVC sits Pending forever.
Check it with kubectl get pv <name> -o yaml and look at spec.nodeAffinity — it will name a zone. Then compare against which zones have schedulable nodes with room.
Also check the PV's claimRef. An Available PV that still has a claimRef pointing at a deleted PVC will never bind to a new one, which produces the same Pending symptom for a completely different reason.
Star, and the Pro limit is the argument that wins on its own. A flat table repeats every customer name, product description and region string on all 12 million rows. Pull those into dimensions and the fact table keeps integer keys, which compress dramatically better. I have watched that exact change take a dataset from 900MB to 200MB without changing a single visual.
The second benefit is that slicers stop fighting you. On a flat table a product slicer only shows products that had sales in the filtered period, which confuses users constantly. With a dimension table you decide.
Agreed on the principle, but "nobody else builds on the dataset" has a half life of about six months in most companies.
Also check your PodDisruptionBudgets before you drain anything. A budget that cannot be satisfied will stall a drain indefinitely and burn your window while nothing appears to be wrong.