Pasha Sviderski

Today I’m solving a classic distributed systems problem: how to reliably stream logs from multiple replicas in chronological order? Surprisingly, none of the mainstream container tools do this. docker service logs, kubectl logs, stern, kubetail, k9s - they all just interleave streams as they arrive. The standard workaround is to pipe through | sort 🙃 To do it properly, you can't immediately print a log entry because a slower stream might have an earlier timestamp that hasn't arrived yet. You need to know when it's "safe" to print. Here's how I'm solving this for the new 'uc logs' command. You need two things: 1. Low watermark Track the minimum timestamp across all log streams and safely print buffered logs with timestamps below this watermark. > watermark = min(latest_timestamp for each stream) > sort and print all buffered logs where timestamp < watermark 2. Heartbeat timestamps What if a container becomes quiet? The entire stream gets stuck because the watermark wouldn’t advance. The solution is to periodically send empty heartbeat entries with the current timestamp from the servers. This is a promise: "I have no logs older than this" This lets the client advance the watermark immediately instead of blocking the output. 500ms–1s interval works well in practice. What about unsynchronised clocks? Not much you can do without making things messy. Just detect it, warn the user, and call it a day.

42,282 görüntüleme