Some of my pods were recreated overnight and I cannot find out why

The distinction between restarted and recreated is worth pulling apart, because your description says the latter and it narrows the causes a lot.

A container that crashes is restarted in place: same pod, same name, restart count goes up. A pod that is recreated has a new name and a new creation timestamp, which means the old one was deleted. Common reasons:

• Node pressure eviction. The node ran short of memory or disk and the kubelet evicted pods to recover.
• The node went away — rebooted, lost network, was scaled down by an autoscaler, or was replaced by a rolling node upgrade.
• A controller deleted it — a deployment rollout, a spot instance reclaim, a descheduler, or a policy tool.

Overnight, with a handful of pods and no deployment change, an autoscaler shrinking the node pool or a maintenance window replacing nodes is the leading suspect. Check the node names on the old and new pods: if they moved, the node is the story.

To stop having this problem, the two changes worth making now rather than after the next occurrence:

Ship events somewhere durable. Whether that is your logging stack or a small exporter, it converts "the evidence expired" into "search yesterday at 3am". This is the highest-value hour of work in this whole area.

Set requests and limits deliberately. A great many mystery evictions are pods with no memory request being placed on a node that then runs out. Requests are what the scheduler uses to decide where things fit, and pods without them are the first to be evicted under pressure.

Also worth checking whether your nodes are being replaced on a schedule. Managed node pools with automatic upgrades will recreate pods overnight as designed, and it looks exactly like a mystery until you know it is happening.

The first thing to know is the reason you could not find it: events are garbage collected after about an hour by default. By morning, the evidence of an overnight event is gone from the cluster entirely.

That is the single most important fact in this whole area, and it drives everything else. The cluster is not a record-keeping system; it is a control loop that knows the current state.

Where to look, in order of what survives:

• kubectl describe pod on a current pod shows recent events and, crucially, the last state of previously terminated containers with exit codes.
• kubectl get pod -o yaml shows lastState.terminated with a reason and exit code, which survives longer than the events do.
• The node's kernel log. If the machine ran out of memory, the kernel's out-of-memory killer logged it there, and that record persists.
• Your log aggregation, if you ship events and logs off-cluster. Which is the real answer to your question.

A pod recreated rather than restarted in place also points somewhere specific — that is the scheduler or a controller replacing it, not a container crashing.

One quick check that often resolves it immediately: compare the number of nodes now with what you expect, and look at node ages.

If a node is two hours old and the recreated pods are two hours old, the pods did not do anything — the node was replaced underneath them. That is a five second check and it eliminates the entire application-level investigation.