Logs and timestamps

Overview

A log is a written record of things that happened on a computer: a service started, a user logged in, a request arrived, an error was thrown. Each record is a line (or a structured object) with a timestamp saying when it happened, plus details about what happened. Logs are how a running system remembers its own past, long after the program that produced the event has moved on or restarted.

Logs matter because almost nothing in computing leaves a permanent physical trace. If a server is slow at 03:00, or someone's account is accessed from an unexpected place, the only way to find out what occurred is to read the records the system left behind. Operators use logs to debug crashes and performance problems. Security teams use them to detect attacks and to reconstruct exactly what an intruder did, in what order. Auditors use them to prove that controls were followed. This lesson builds directly on the file-system and storage ideas from earlier Computer & OS Fundamentals lessons (the previous lesson, Compression and archives, covered how older logs are rotated and compressed to save space) — logs are just files that grow over time and eventually get archived.

The single most important detail in a log line is usually the timestamp, and timestamps are surprisingly easy to get wrong. A time like 14:30 is meaningless on its own: 14:30 where, and on whose clock? To make timestamps trustworthy and comparable, the industry leans on a few conventions: the ISO 8601 text format, the UTC time standard, and Unix epoch seconds for machine use. Keeping every machine's clock aligned is the job of NTP (the Network Time Protocol). With those pieces in place, you can take logs from a firewall, a web server, and a database, line their events up on one timeline, and see the whole story.

Why it matters

Logs are the backbone of operations, security, and accountability in real software.

• Defenders detect attacks by noticing anomalies in logs — a burst of failed logins, requests from a strange user agent, a login at 04:00 from a new country. After an incident, they rebuild a minute-by-minute timeline from timestamps to answer: when did the intruder get in, what did they touch, and when were they stopped.
• Operators and developers use logs to debug problems that only show up in production, where you cannot attach a debugger. The timestamp tells you whether the database error happened before or after the deploy.
• Auditors and compliance rely on logs as evidence that access was controlled and policies were followed. Standards such as PCI DSS explicitly require time-synchronized, retained logs.

None of this works if the times are wrong. If two servers disagree by even a few minutes, a real attack can look like two unrelated events, and a forensic timeline can put effects before their causes. That is why synchronized, unambiguous timestamps are not a nicety — they are what makes a pile of log lines into usable evidence. And because logs are evidence, altering them (outside an authorized, documented process) destroys their value and is treated as a serious offense in any security engagement.

Core concepts

1. Log sources: where events come from

Definition. A log source is any component that records its own events. Different sources answer different questions.

• System logs record OS-level events: services starting and stopping, hardware errors, the kernel complaining. On Linux these live under /var/log/ (and in the journald journal); on Windows they appear in the Event Log.
• Authentication logs record who got in and who failed: successful logins, failed passwords, privilege escalation (sudo). This is the first place a defender looks during a suspected break-in. On Linux these are often in /var/log/auth.log or /var/log/secure.
• Application and web logs record what a specific program did: HTTP requests with status codes and client IPs, payment attempts, background-job results.

Why it works this way. Each layer can only see its own world, so you correlate across sources: a web log shows a suspicious request, the auth log shows whether it led to a login, the system log shows whether a service then crashed.

When NOT to rely on one source. Never conclude "nothing happened" from a single log. An attacker who only touched the web tier leaves nothing in the auth log; absence in one place is not absence everywhere.

Pitfall. Logs are often sampled or rate-limited under heavy load. A gap can mean "nothing happened" or "too much happened to record" — those are very different.

   ENTRY POINT                 WHAT SEES IT
   -----------                 ------------
   user types password  --->   auth log      (login success/fail)
   browser sends GET    --->   web log       (URL, status, client IP)
   service crashes      --->   system log    (kernel/service error)

   One real incident usually leaves traces in SEVERAL logs.
   Correlating them = the whole story.

Knowledge check (explain in your own words): Why might a successful attack on a website show up in the web log but leave the authentication log completely empty?

2. Timestamp formats: ISO 8601, UTC, and Unix epoch

Definition. A timestamp records the instant an event occurred. The format is how that instant is written down.

• ISO 8601 is an international standard text format: 2026-06-11T14:30:00Z. It goes from largest unit to smallest (year, month, day, then time), uses T to separate the date from the time, and is sortable as plain text — alphabetical order equals chronological order.
• UTC (Coordinated Universal Time) is the global reference clock, independent of any country's time zone or daylight saving. In ISO 8601, the trailing Z ("Zulu") means "this time is in UTC." A time with an offset like 2026-06-11T16:30:00+02:00 is the same instant expressed in a +2 zone.
• Unix epoch time is the machine-friendly form: a single integer counting seconds since 1970-01-01T00:00:00Z. For example 1749652200. It has no time zone ambiguity at all because it is defined in UTC, and arithmetic on it is trivial (subtract two values to get a duration in seconds).

How it works internally. Computers store time as epoch seconds (or milliseconds/nanoseconds) — a plain counter. ISO 8601 strings are produced from that counter for humans to read. Converting between them is just math plus a calendar.

When to use which. Store and transmit time as UTC (epoch or ISO 8601 with Z). Convert to local time only at the moment of display, for the human reading it. Never store local time.

Pitfall. A timestamp with no zone or offset (2026-06-11 14:30:00) is ambiguous — you cannot know what instant it refers to without out-of-band knowledge of the server's configured zone, which may change.

  Same instant, three notations:

  ISO 8601 (UTC)   2026-06-11T14:30:00Z
  ISO 8601 (+02)   2026-06-11T16:30:00+02:00
  Unix epoch       1749652200
         |
         +-- all point to the exact same point on the global timeline

Knowledge check (predict the output): You sort these three text strings alphabetically: 2026-06-11T09:00:00Z, 2026-06-11T14:30:00Z, 2026-06-11T08:59:59Z. What order do they come out in, and is that the same as chronological order? Why does ISO 8601 make these match?

3. Time zones and clock synchronization (NTP)

Definition. A time zone is a regional offset from UTC (e.g. New York is UTC-5 or UTC-4 in summer). NTP, the Network Time Protocol, is the service that keeps a machine's clock continuously aligned with authoritative time servers.

Why it matters for logs. To correlate events across machines, every machine must agree on what time it is. If the web server's clock is 4 minutes ahead of the database's, a request and the query it triggered will look 4 minutes apart — or even out of order. NTP shrinks that disagreement to milliseconds.

How it works internally. Clocks drift — a cheap crystal oscillator gains or loses a few seconds a day. NTP periodically asks reference servers for the true time, measures the network round-trip delay, and gently slews (speeds up or slows down) the local clock to stay aligned, rather than jumping it.

When NOT to: never "fix" a drifted clock by hand-setting it backward on a running system. Jumping the clock backward can make new log lines appear before old ones and confuse software that assumes time only moves forward. Let NTP correct it gradually.

Pitfall. A virtual machine that was paused/suspended can wake with a badly wrong clock until NTP catches up — log lines from that window are suspect.

  Without sync                      With NTP
  ------------                      --------
  Web  clock: 14:34   ---\         Web  clock: 14:30:02
  DB   clock: 14:30      > 4 min   DB   clock: 14:30:02
  events look unrelated  /         events line up on ONE timeline

Knowledge check (concept): The quiz for this lesson asks why NTP matters; go one level deeper here — describe a concrete two-server scenario where a 5-minute clock skew makes a single attack look like two separate, unrelated events.

4. Detection and integrity

Definition. Detection is using logs to notice that something bad is happening or has happened. Integrity is the guarantee that the logs themselves have not been altered.

• Detection works by spotting patterns: many failed logins from one IP (brute force), one IP touching many ports (a scan), a spike of 500 errors, or a request at an impossible hour. The related exercises Count suspicious IPs in a sample log and Detecting port scans are exactly this kind of pattern-finding over raw log text.
• Integrity matters because logs are evidence. If an attacker (or a careless tester) edits or deletes log lines, the timeline becomes unreliable and the incident may be unprovable. Protections include shipping logs off the host immediately (so the only copy is not on the machine being attacked), append-only storage, and write-once archives.

Pitfall. Treating your own clock and your own host's logs as ground truth during an incident. If the host is compromised, its local logs may already be tampered with — prefer the copy sent to a separate log server.

Knowledge check (find-the-issue): A teammate says "the server log shows nothing at the time of the breach, so nothing happened then." Give two different innocent-and-not-innocent reasons the log could be empty for that window.

Syntax notes

Timestamps you will read and write constantly. The pieces of an ISO 8601 timestamp:

  2026-06-11T14:30:00Z
  |________| |______||
   date       time   |
   YYYY-MM-DD HH:MM:SS
              (24-hour)
                      |
                      +-- Z = UTC. Or an offset like +02:00 / -05:00

Unix epoch is just an integer of seconds since the 1970 UTC epoch:

  1749652200   <-- seconds since 1970-01-01T00:00:00Z

A typical Linux authentication log line, annotated:

  Jun 11 14:30:00 web01 sshd[2931]: Failed password for invalid user admin from 203.0.113.9 port 51514 ssh2
  |____________|  |___| |________|   |__________________________________________________________________|
   timestamp      host  program/pid   message (what happened, and from where)

Note that the classic syslog timestamp (Jun 11 14:30:00) has no year and no time zone — a real-world ambiguity that modern formats fix. Prefer ISO 8601 with Z wherever you control the format.

Lesson

Logs are the system's memory of what happened. They are central to both detecting attacks and investigating incidents.

What gets logged

• System logs (/var/log/ on Linux, the Windows Event Log): service start and stop, authentication attempts, errors.
• Authentication logs: logins, sudo use, failed passwords. This is the first place a defender looks.
• Application and web logs: requests, status codes, client IP addresses.

Timestamps

Every log line is stamped with a time. Two things matter here.

Format. ISO 8601 (for example, 2026-06-11T14:30:00Z) is unambiguous. Many other formats are not.

Time zone. Z means UTC, as opposed to local time. To correlate events across systems, you need a shared time base. That is why NTP (time synchronization) matters.

Unix epoch time, the number of seconds since 1970-01-01 UTC, is the machine-friendly form.

Why both sides care

• Defenders detect attacks by spotting anomalies in logs, such as bursts of failed logins or unusual user agents. They build incident timelines from timestamps.
• Pentesters and red teams know their actions are logged and factor that into the engagement. Critically, they never tamper with logs outside explicit authorization. Tampering destroys evidence and is usually out of scope.
• Forensics reconstructs events from log timelines, so timestamp integrity is paramount.

Code examples

Below is a small, realistic worked example: a slice of raw logs from two systems during a suspected brute-force attempt, followed by the consolidated, normalized timeline a defender would build from them. This is the concept-track "code" — sample data plus the analyst output, not a program.

# --- Raw web server access log (web01), timestamps in UTC (Z) ---
203.0.113.9 - - [2026-06-11T14:29:58Z] "POST /login HTTP/1.1" 401 512
203.0.113.9 - - [2026-06-11T14:29:59Z] "POST /login HTTP/1.1" 401 512
203.0.113.9 - - [2026-06-11T14:30:00Z] "POST /login HTTP/1.1" 401 512
203.0.113.9 - - [2026-06-11T14:30:01Z] "POST /login HTTP/1.1" 200 4096

# --- Raw auth log (app server app02), syslog format, server zone = UTC ---
Jun 11 14:30:01 app02 app[8821]: AUTH success user=alice src=203.0.113.9
Jun 11 14:30:03 app02 app[8821]: ROLE change user=alice new_role=admin

# --- Normalized incident timeline (everything converted to ISO 8601 / UTC) ---
TIME (UTC)             SOURCE  EVENT
2026-06-11T14:29:58Z   web01   POST /login -> 401  (failed) from 203.0.113.9
2026-06-11T14:29:59Z   web01   POST /login -> 401  (failed) from 203.0.113.9
2026-06-11T14:30:00Z   web01   POST /login -> 401  (failed) from 203.0.113.9
2026-06-11T14:30:01Z   web01   POST /login -> 200  (SUCCESS) from 203.0.113.9
2026-06-11T14:30:01Z   app02   AUTH success user=alice from 203.0.113.9
2026-06-11T14:30:03Z   app02   ROLE change user=alice -> admin

What it does. The raw logs come from two machines in two different formats. The defender's job is to normalize them — convert every timestamp to one format (ISO 8601 / UTC) — and then merge them into one ordered list.

Expected output / what you learn from it. Read top to bottom: three failed logins in three seconds from the same IP (a brute-force pattern), immediately followed by a success on the same IP, the matching AUTH success for user alice on the app server in the same second, and two seconds later that account being promoted to admin. The story is now obvious: the attacker guessed alice's password and escalated privileges. This conclusion is only possible because both servers used UTC and had synchronized clocks — the cross-server events land in the correct order.

Key edge cases. (1) If app02 had been on local time (say UTC+2) without an offset, its 14:30:01 would actually be 12:30:01Z, and the AUTH success would appear before the web success — breaking the causal chain. (2) If the clocks were skewed by even a few seconds, the ROLE change might appear before the login. (3) The syslog line has no year; you must supply it from the log's date range or rotation metadata.

Line by line

Walk the normalized timeline one event at a time and watch the picture form.

How values change as you read down: the status code moves 401 -> 401 -> 401 -> 200 (failure becoming success), the source IP stays constant (203.0.113.9, the thread that ties everything together), and the privilege jumps from a normal user to admin.

Why the result is trustworthy: every timestamp is in UTC with synchronized clocks, so step 4 (web success) and step 5 (auth success) correctly share the same second and step 6 correctly comes after. Reorder or skew those and the causal chain — guess, succeed, escalate — falls apart. The timeline is the evidence, and its correctness rests entirely on consistent timestamps.

Common mistakes

Mistake 1 — Storing local time instead of UTC.

WRONG:  logged at "2026-06-11 14:30:00"   (no zone; server happens to be in Berlin, UTC+2)

Why it is wrong: six months later daylight saving shifts, or the server moves to another region, and now you cannot tell whether 14:30 meant 12:30Z or 13:30Z. Correlating with a UTC server becomes guesswork.

RIGHT:  logged at "2026-06-11T12:30:00Z"   (UTC, explicit Z)

Prevent it: configure services to log in UTC; if you must show local time, convert at display time only.

Mistake 2 — Assuming clocks are in sync. Believing two servers agree on the time without running NTP. A 3-minute skew silently splits one incident into two. Recognize it by checking that your hosts run a time-sync service and by comparing a known shared event (e.g. the same request seen in two logs) — if their recorded times differ, you have skew. Prevent it by enabling NTP everywhere and monitoring offset.

Mistake 3 — Sorting non-ISO timestamps as text. Formats like 06/11/2026 2:30 PM do not sort chronologically as strings (12/01 sorts before 06/11, and 2:30 PM sorts before 2:30 AM alphabetically). Learners write a quick text sort and get a scrambled timeline.

WRONG:  sort lines like "06/11/2026 2:30 PM" as plain text  -> wrong order
RIGHT:  use ISO 8601 ("2026-06-11T14:30:00Z") -> text order == time order

Mistake 4 — Trusting a single log source. Concluding "the attacker did nothing on the database" because the web log is quiet about it. Different layers see different things; check the relevant source. Prevent it by always asking "which log would record this action?" before concluding it did not happen.

Mistake 5 — Editing logs to "clean up." Deleting noisy lines (or, during a test, removing one's own activity) destroys evidence and breaks integrity. Even with good intentions, never alter logs in place; filter a copy for readability and keep the original untouched.

Debugging tips

When a timeline looks wrong, work through these checks:

• Events out of order or impossibly fast/slow? Suspect a time zone or skew problem first. Find one event that appears in two logs (a request ID, an IP, a session ID) and compare its timestamp in each. A constant offset between them is clock skew or a zone mismatch.
• A timestamp with no Z and no offset? Treat it as unknown zone until you confirm the source machine's configured time zone. Do not assume UTC just because you want it to be.
• Syslog line missing a year? Derive the year from the file's rotation date or the surrounding entries; do not guess.
• Gap in the logs? Ask: was the service down, was logging rate-limited under load, did the log rotate (older entries moved to a .gz archive — see the Compression and archives lesson), or was a section removed? Check file sizes, rotation timestamps, and whether a remote copy on a log server shows the same gap.
• Two tools disagree on what time an event happened? Confirm both are interpreting the timestamp's zone the same way; many tools silently assume local time for zone-less timestamps.
• Questions to ask when it doesn't work: Is NTP running and synchronized on every host? Are all timestamps the same format and zone before I sorted? Am I looking at the right log source for this action? Could this log have been altered, and is there an off-host copy to compare against?

Memory safety

Robustness, validation, and integrity (this is theory, so focus on correctness and trust):

• Validate timestamps before trusting them. When ingesting logs, reject or flag lines with no zone/offset, with times in the future, or with times before the service existed — these signal misconfiguration, skew, or tampering.
• Normalize early, store as UTC. Convert every incoming timestamp to one canonical form (ISO 8601 / UTC or epoch) as soon as you read it, so all later comparisons and sorts are apples-to-apples.
• Protect log integrity. Logs are evidence. Ship them off the host to a separate, append-only or write-once store as soon as they are written, so a compromised machine cannot quietly rewrite its own history. Restrict who can write to and delete logs.
• Mind retention and privacy. Logs often contain IP addresses, usernames, and request paths — but they must never contain passwords, session tokens, full card numbers, or API keys. Scrub secrets at the source; logging a secret turns your log store into a new place to be breached.
• Handle clock changes safely. Use a monotonic source for measuring durations within one process, and let NTP slew (not jump) the wall clock so log ordering on a single host stays sane.

Real-world uses

Concrete uses:

• Incident response (security). When a breach is suspected, responders pull logs from firewalls, web servers, auth systems, and endpoints, normalize the timestamps to UTC, and build one timeline to answer when/what/how. The brute-force-then-escalate example above is a textbook case.
• SIEM and centralized logging. Tools like the ELK/Elastic stack, Splunk, or a cloud logging service collect logs from hundreds of machines into one place. They depend completely on synchronized UTC timestamps to merge and search across sources.
• Operations and SRE. A latency spike at 14:32:10Z is traced by lining up app logs, database slow-query logs, and deploy events to find the trigger.
• Compliance and audit. PCI DSS and similar standards mandate time-synchronized logs retained for a defined period as proof that access was monitored.

Professional best-practice habits:

Beginner rules

• Always log time in UTC; show local time only when displaying to a person.
• Use ISO 8601 (...Z) so logs sort chronologically as text.
• Include the source (host, service) on every line so events can be correlated.
• Never log secrets (passwords, tokens, keys, full PANs).

Advanced practices

• Run NTP on every host and monitor the clock offset; alert if it drifts.
• Centralize logs to an off-host, append-only/write-once store immediately for integrity.
• Attach a correlation/request ID to related events so they can be joined across services without relying solely on time.
• Define and enforce retention and rotation (compress and archive old logs; keep enough history to investigate slow-burn intrusions).

Practice tasks

Beginner 1 — Convert and label. Given the timestamps 2026-06-11T14:30:00Z, 1749652200 (Unix epoch), and Jun 11 14:30:00, write each one's format name and state which are unambiguous and which are not. Requirement: explain in one sentence why the syslog line is ambiguous. Concepts: ISO 8601, Unix epoch, time-zone ambiguity. Hint: look for an explicit zone marker.

Beginner 2 — Sort a timeline. You are handed five ISO 8601 / UTC log lines in scrambled order. Put them in chronological order using only plain text sorting, then write one sentence explaining why text sorting works here. Input example: lines timestamped 14:30:01Z, 14:29:58Z, 14:30:03Z, 14:29:59Z, 14:30:00Z. Expected output: the lines from earliest to latest. Concepts: ISO 8601 sortability. Hint: alphabetical order should equal time order — verify it does.

Intermediate 1 — Merge two logs. Given a short web log (timestamps in UTC) and a short auth log (syslog format, server zone = UTC), produce a single normalized timeline in ISO 8601 / UTC, then state in one or two sentences what likely happened. Requirement: every line must end up in the same format and be globally ordered. Constraints: do not alter the original lines; build a separate merged view. Concepts: normalization, correlation, log sources. Hint: convert first, merge second.

Intermediate 2 — Find the skew. Two servers each logged the same request (it carries the same request ID req-7f3a). Server A recorded it at 2026-06-11T14:30:00Z; server B at 2026-06-11T14:34:10Z. State the clock skew, explain which event would wrongly appear first if you sorted by time, and name the fix. Concepts: clock skew, NTP, correlation by ID. Hint: the same event cannot truly happen at two different instants.

Challenge — Spot the tampering / the gap. You are given a log excerpt covering 14:25–14:40 UTC, but there are no lines at all between 14:30 and 14:35, and one line at 14:37 has a timestamp of 14:21. List at least three different explanations for the gap (innocent and malicious), describe how you would investigate each (file sizes, rotation/archive timestamps, an off-host copy on a log server), and explain why the out-of-order 14:21 line is a red flag. Constraints: propose investigation steps only — do not fabricate a conclusion. Concepts: integrity, detection, NTP/skew, rotation/archives. Hint: compare against a copy the suspect host could not have edited.

Summary

• Logs are timestamped records of events from several sources — system, authentication, and application/web logs — and you correlate across them to see a full incident.
• Timestamps are the load-bearing detail. Use ISO 8601 (2026-06-11T14:30:00Z) for human-readable, text-sortable times; UTC as the single storage standard; and Unix epoch seconds for unambiguous machine arithmetic.
• Time zones and clocks must agree. A timestamp with no zone is ambiguous, and unsynchronized clocks (no NTP) scramble or split a timeline. NTP keeps every host aligned so cross-server events land in the right order.
• Common mistakes: storing local instead of UTC, assuming clocks are in sync, sorting non-ISO timestamps as text, trusting a single source, and editing logs.
• Logs are evidence. Detect attacks by spotting patterns (failed-login bursts, port scans), protect integrity by shipping logs off-host and never altering them, and never log secrets.
• Remember: normalize to UTC early, sort with ISO 8601, keep clocks synced, and ask "which log would record this?" before concluding nothing happened.

Practice with these exercises

• Count suspicious IPs in a sample log
• Distinct ports touched
• Is this a port scan?