Small cluster, no log stack yet, everything goes to stdout and we read it with kubectl. Service crash looped for about an hour overnight, the node got rebooted by the platform, and by the time I looked there was nothing to read. Before I sign up for a hosted logging bill I would like to understand what the default retention actually is and what I am choosing between.
Specifics, because the defaults are knowable. The kubelet rotates container logs at containerLogMaxSize, which defaults to 10Mi, and keeps containerLogMaxFiles, which defaults to 5. So roughly 50 MiB per container on disk in the best case.
Two things that make it worse than that sounds. kubectl logs only reads the current file, so if your container wrote 40 MiB the command returns at most the last 10 MiB. And --previous holds exactly one prior instance, so in a crash loop restarting every thirty seconds it is overwritten constantly, which is why yours was empty by morning.
The docs also say plainly that when a pod is evicted from a node the containers go with it, logs included. A node reboot takes everything.
And the crash loop case is the worst case, because the thing you most want to read is the thing being overwritten fastest by its own restarts.
So the answer to how long does it keep logs is somewhere between 50 MiB and until the pod moves, whichever comes first. That is bleaker than I expected and useful to know.
Rough sizing before you go shopping. Our three node cluster with about forty pods produces a few gigabytes of logs a day at normal verbosity, and it went up by a multiple the week somebody left debug logging on in a hot path. Self hosted Loki with thirty day retention on object storage costs us less per month than a couple of coffees, plus the operational cost of running it, which is not zero but is small. The hosted options quoted us per gigabyte ingested, which is fine at three gigabytes a day and terrifying at thirty.
Whatever you pick, set a retention period on day one and an ingestion budget alert on day two.
Disagreeing with ship everything and sort it out later, which is where most teams start and it is expensive in a way that creeps. Our bill quadrupled in a quarter and about 70 percent of the volume was health check logging and one library logging every HTTP request at info including successful ones.
What actually worked: structured logs so you can filter at the agent, drop rules for health checks at the collector, sampling on high volume happy paths, and keeping errors and warnings at full fidelity forever. You want the 3am crash loop, you do not want a million 200 OK lines.
Drop rules at the collector rather than in the app, too. When you need the noisy logging back for an incident you can turn it on without a deploy.
Cheap middle path if you do not want a hosted bill yet: a log shipper as a DaemonSet writing to Loki with object storage behind it. Storage is the cheap part and you can set retention to whatever you can justify. It is not zero maintenance, but it took an afternoon to stand up and it means the next 3am crash is readable at 9am. That alone paid for the afternoon.
Separate from logging, make the crash itself say something. A terminationMessagePath that captures the fatal error means kubectl describe pod tells you why the last container died even when the logs are gone, and restart reason plus exit code in the pod status is often enough to identify an OOMKill or a failed liveness probe without any log stack at all.
@shallow_well_shea · 4w ago
There is no retention. There is a file on a disk that gets rotated and a pod that can be deleted at any moment. If it has not left the node, it does not exist.
Reply
Report