Technology detection and attack-surface mapping

Overview

Security objective. The asset you are learning to reason about is an organisation's external attack surface — every domain, host, port, service, and web application that a stranger on the internet can reach. The threat is that unknown or forgotten exposure (a stale admin panel, an outdated CMS, a test server that was never taken down) becomes an attacker's easiest way in. In this lesson you learn to detect what is exposed and what it runs, so a defender can shrink that surface and a penetration tester can prioritise honestly. You are mapping — not attacking.

This is the third recon lesson. It builds directly on your prerequisites:

• From DNS, WHOIS, and subdomain discovery you already have a list of names and IPs — the raw inventory of where the target lives.
• From OSINT: search dorking, leaks, and metadata you have context — technologies mentioned in job posts, leaked paths, employee-facing systems.

Those two lessons produced scattered facts. This lesson turns them into a picture. First you fingerprint each live host to learn what software it runs. Then you screenshot web hosts in bulk so you can triage hundreds of them by eye. Finally you consolidate everything into one attack-surface map and use it to decide where limited active-testing time should go.

Everything here is passive-to-light-touch reconnaissance performed only against systems you own or are explicitly authorised to test. The output is a map and a plan, produced inside your engagement's scope.

Why it matters

In authorised professional work, the map is the deliverable that makes everything after it defensible.

• Penetration testers are paid for a fixed number of days. A consolidated attack-surface map is how they justify spending those days on the 3 hosts that matter instead of the 300 that don't. It also means every later finding traces back to a concrete place on the map, which is what makes a report credible.
• Blue teams and asset owners use the same techniques defensively as attack-surface management: continuously enumerating their own exposure so a forgotten staging box or an expired-but-still-live subdomain gets found by them before an outsider finds it.
• Scope discipline is a professional obligation. A clean map shows, at a glance, whether a host is inside or outside the authorised scope — which is exactly the check that keeps testers on the right side of the law and their contract.

Recon that stays as a pile of tool output is noise. Recon that becomes an organised, evidence-backed map is the foundation the whole engagement stands on.

Core concepts

1. Technology fingerprinting

Definition. Identifying the software (and ideally versions) a host runs by reading the clues its responses leak.

Plain explanation. Servers and applications are chatty. They announce themselves in Server and X-Powered-By headers, in cookie names (PHPSESSID, JSESSIONID, csrftoken), in HTML/JavaScript artefacts (framework class names, /wp-content/ paths), in the hash of their favicon, and in the exact wording of their default error pages.

How it works. A tool sends ordinary HTTP requests and pattern-matches the responses against a database of known signatures. Server: Apache/2.4.41 plus a wp-json link plus a wordpress_logged_in cookie is a confident "WordPress on Apache" fingerprint.

When / when not. Use it to route your attention toward relevant known issues for that stack. Do not treat a version banner as ground truth: banners can be blank, forged, or set by a proxy in front of the real server. Fingerprinting narrows the search; it does not prove a vulnerability.

Pitfall. A version number in a banner is a hint, not a verdict. Concluding "nginx 1.18 → vulnerable to CVE-X" without confirming the actual behaviour produces false findings that destroy a report's credibility.

2. Mass screenshotting (visual triage)

Definition. Automatically rendering many web hosts to images so a human can scan them all quickly.

Plain explanation. After subdomain discovery you might have hundreds of live web hosts. Opening each in a browser is impossible. A screenshotting tool loads each URL headlessly and saves a picture; you then flip through a grid and your eye instantly catches login panels, default "It works!" pages, admin dashboards, and error screens.

How it works. The tool drives a headless browser against each host in your own target list, captures the rendered page, and builds an HTML report of thumbnails.

When / when not. Ideal for triage of a large in-scope host list. Not a vulnerability scan — a screenshot tells you what a page looks like, not that it is exploitable. Never point it at hosts outside your authorised scope.

Pitfall. Loading arbitrary pages executes their JavaScript in your browser context. Run mass screenshotting from an isolated lab VM, not your daily machine, and only against authorised targets.

3. Consolidation into an attack-surface map

Definition. Linking every recon fact into one connected structure: domain → IP → port → service → stack → endpoints.

Plain explanation. Individually, "subdomain X exists", "IP Y has port 443 open", and "host Z runs WordPress" are trivia. Joined together — this subdomain resolves to this IP, which exposes this port, running this service, which is this app, which has these interesting endpoints — they become a target you can reason about.

How it works. You merge outputs from DNS enumeration, port scanning, and fingerprinting keyed on the host, and record it in a structured form (a table, CSV, or small database) with a note of how each fact was learned.

When / when not. Always, on any engagement. The one time it is not enough on its own is when you forget to mark scope — an unscoped map invites testing something you have no authorisation for.

Pitfall. Recording conclusions without evidence. Every row should say where the data came from so a finding can be traced and retested.

4. Map drives plan (prioritisation)

Definition. Using the finished map to decide where to spend active-testing effort.

Plain explanation. The map reveals concentration and value: an auth endpoint, an admin panel, an upload form, or an outdated public-facing app is worth more attention than a static marketing page. Prioritisation converts the map into an ordered plan.

Pitfall. Chasing the shiniest technology instead of the highest-risk exposure. "Interesting" is not the same as "reachable and impactful."

EXTERNAL ATTACK-SURFACE THREAT MODEL (authorised lab)

   Internet (untrusted)
        |  entry points: DNS names, open ports, web endpoints
        v
  +--------------------- trust boundary: the engagement scope --------------------+
  |                                                                               |
  |  ASSET: the target's reachable hosts and web apps                            |
  |                                                                               |
  |   app-01.lab.local ---> 10.10.0.11 : 443/tcp ---> nginx + WordPress          |
  |        endpoints: /wp-login.php  /wp-admin/  /wp-json/                        |
  |                                                                               |
  |   dev-02.lab.local  ---> 10.10.0.12 : 8080/tcp ---> Tomcat (default page)    |
  |        endpoints: /manager/  (exposed admin -> HIGH interest)                 |
  |                                                                               |
  +-------------------------------------------------------------------------------+
        ^
        |  RECON POSTURE: passive/light HTTP only, from an isolated lab VM,
        |  every fact logged with source + timestamp, nothing outside scope touched.

Knowledge check.

1. What asset is protected here? The organisation's reachable external hosts and web apps — the attack surface a defender wants to shrink.
2. Where is the trust boundary? At the engagement scope: the line between the untrusted internet and the specific hosts you are authorised to test. Anything outside it is off-limits.
3. What insecure assumption typically puts a host on this map? "Nobody knows this staging/admin box is here." Obscurity is not access control — if you can reach and fingerprint it, so can an attacker.

Syntax notes

The core operation is reading fingerprint clues out of an HTTP response. Below is the shape of a lab request/response (host names are placeholders for your own lab):

GET / HTTP/1.1
Host: app-01.lab.local
User-Agent: recon-lab/1.0
Connection: close

--- response (fingerprint clues annotated) ---
HTTP/1.1 200 OK
Server: nginx/1.18.0            # web server + version hint
X-Powered-By: PHP/7.4.3         # backend language hint
Set-Cookie: wordpress_test_...  # app hint: WordPress
Link: <http://app-01.lab.local/wp-json/>; rel="https://api.w.org/"   # REST API -> WordPress

A typical fingerprinting tool invocation, run only against your own lab host, looks like:

# Read the tool's own docs before use; -a controls how noisy it is.
whatweb --aggression 1 http://app-01.lab.local     # 1 = passive/stealthy

And a consolidation record — the row you write for the map — is just structured text:

host=app-01.lab.local ip=10.10.0.11 port=443 service=https \
  stack="nginx/1.18 + WordPress" endpoints="/wp-login.php,/wp-admin/" \
  source="whatweb+nmap 2026-07-07" scope=IN

Lesson

The output of recon should be a single, organised attack-surface map: everything reachable, what it runs, and where to focus.

Technology detection

Fingerprint what a host runs from its responses:

• HTTP headers, such as Server and X-Powered-By.
• Cookies.
• HTML and JavaScript frameworks.
• Favicon hashes.
• Error pages.

Tools like WhatWeb and Wappalyzer automate this. Knowing the stack (for example, a specific CMS and version) points you straight at relevant known issues.

Screenshotting at scale

You may discover dozens or hundreds of web hosts. Automated screenshotting renders every host so you can eyeball them all quickly.

This helps you spot:

• Login panels.
• Default pages.
• Development and admin interfaces worth a closer look.

Building the map

Consolidate everything into one picture:

• Domains and subdomains → resolved IPs.
• Open ports and identified services per host.
• Technology stack per web app.
• Interesting endpoints (admin, API, upload, auth).

Why a map

A map turns recon into a plan. It shows where the surface is concentrated, which targets are highest-value, and how to spend limited active-testing time.

A finding always traces back to a place on this map.

Code examples

Below is a small, self-contained lab pipeline that fingerprints hosts you control and consolidates the results into a map. It is written in Python 3 (standard library only) so it runs anywhere. It talks only to hosts listed in a scope file — that guard is the point.

(1) WARNING: intentionally vulnerable — use only in a local, isolated, authorized lab. Do not deploy.

This is the insecure pattern to recognise — a recon script with no scope check that will happily hit any host handed to it:

# INSECURE PATTERN — DO NOT USE. No authorisation boundary.
import sys, urllib.request

for host in sys.argv[1:]:                 # anything on the command line...
    r = urllib.request.urlopen(host)      # ...gets requested, in or out of scope
    print(host, r.headers.get("Server"))
# Problem: nothing stops this from touching third-party systems you are
# not authorised to test. That is the whole risk of recon tooling.

(2) SECURE version — scope-gated fingerprint + map builder

#!/usr/bin/env python3
"""Lab-only attack-surface mapper: fingerprint IN-SCOPE hosts, build a map.
Usage: python3 mapper.py scope.txt
scope.txt: one authorised base URL per line, e.g.  http://app-01.lab.local
"""
import sys
import urllib.request
import urllib.error
from urllib.parse import urlsplit

HINTS = {  # header/cookie/body substring -> human-readable stack guess
    "wordpress": "WordPress", "wp-": "WordPress",
    "php": "PHP", "nginx": "nginx", "apache": "Apache",
    "express": "Node/Express", "tomcat": "Apache Tomcat",
}


def load_scope(path):
    """Read the allow-list of authorised hosts. This is the trust boundary."""
    allowed = {}
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            url = line.strip()
            if not url or url.startswith("#"):
                continue
            host = urlsplit(url).hostname
            if host:
                allowed[host] = url
    if not allowed:
        raise SystemExit("scope file is empty: nothing is authorised")
    return allowed


def in_scope(url, allowed):
    """Refuse any host not explicitly on the allow-list."""
    return urlsplit(url).hostname in allowed


def fingerprint(url):
    """Send one benign GET and extract fingerprint clues."""
    req = urllib.request.Request(url, headers={"User-Agent": "recon-lab/1.0"})
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            headers = {k.lower(): v for k, v in resp.headers.items()}
            body_head = resp.read(2048).decode("utf-8", "replace").lower()
            code = resp.status
    except urllib.error.URLError as err:
        return {"url": url, "error": str(err)}

    blob = " ".join([
        headers.get("server", ""),
        headers.get("x-powered-by", ""),
        headers.get("set-cookie", ""),
        body_head,
    ]).lower()
    stacks = sorted({v for k, v in HINTS.items() if k in blob})
    return {
        "url": url,
        "status": code,
        "server": headers.get("server", ""),
        "stack": stacks or ["unknown"],
    }


def main():
    if len(sys.argv) != 2:
        raise SystemExit("usage: python3 mapper.py scope.txt")
    allowed = load_scope(sys.argv[1])

    print("host,status,server,stack,scope")
    for host, url in allowed.items():
        if not in_scope(url, allowed):          # defence in depth
            print(f"{host},SKIPPED,,,OUT-OF-SCOPE")
            continue
        row = fingerprint(url)
        if "error" in row:
            print(f"{host},ERROR,{row['error']},,IN")
            continue
        print(f"{host},{row['status']},{row['server']},"
              f"{'|'.join(row['stack'])},IN")


if __name__ == "__main__":
    main()

(3) VERIFY — prove the scope gate rejects bad input and accepts good input

# verify_scope.py — run: python3 verify_scope.py
from mapper import load_scope, in_scope

# Build a tiny scope by writing a temp file.
import tempfile, os
fd, path = tempfile.mkstemp(suffix=".txt")
with os.fdopen(fd, "w") as fh:
    fh.write("http://app-01.lab.local\n")
allowed = load_scope(path)
os.unlink(path)

# GOOD input: an authorised host is accepted.
assert in_scope("http://app-01.lab.local/wp-login.php", allowed) is True

# BAD input: an unauthorised host is REJECTED, even if it looks similar.
assert in_scope("http://app-01.lab.local.evil.example", allowed) is False
assert in_scope("http://not-in-scope.example", allowed) is False

print("scope gate OK: authorised accepted, everything else rejected")

Expected behaviour. Against a lab host you run yourself, python3 mapper.py scope.txt prints one CSV row per authorised host, e.g. app-01.lab.local,200,nginx/1.18.0,PHP|WordPress|nginx,IN. verify_scope.py prints scope gate OK... and exits 0; if the gate ever let an out-of-scope host through, an assert would fail loudly. The security property being demonstrated is that the tool cannot reach a host you did not authorise, and that app-01.lab.local.evil.example (a look-alike) is correctly treated as a different, out-of-scope host.

Line by line

Walking the secure mapper.py:

1. load_scope(path) reads the allow-list file — this is your engagement scope made executable. Blank lines and # comments are skipped; each URL is reduced to its hostname with urlsplit. If the file yields no hosts, the program exits: no scope means nothing is authorised, so it refuses to run.
2. in_scope(url, allowed) returns true only when the URL's hostname is a key in the allow-list. Because it compares the parsed hostname, a look-alike like app-01.lab.local.evil.example has hostname app-01.lab.local.evil.example — not in the map — so it is rejected. This is the trust boundary enforced in one line.
3. fingerprint(url) sends a single, benign GET with an honest User-Agent, then reads only the headers and the first 2 KB of body. It never logs in, never submits forms, never follows into other hosts.
4. It lowercases the Server, X-Powered-By, Set-Cookie values and the body snippet into one blob, then checks each signature in HINTS. Matches accumulate into stacks.
5. main() loads scope first, prints a CSV header, then loops. Before every request it re-checks in_scope (defence in depth — even though the loop only iterates over the allow-list, the explicit check documents and enforces the boundary). Errors are captured, never crash the run.

Trace for one authorised host:

For the look-alike app-01.lab.local.evil.example, step 2 returns False, so no request is ever sent — exactly the behaviour the VERIFY step asserts.

Common mistakes

Debugging tips

When the mapper misbehaves, work through these:

• SystemExit: scope file is empty — your scope.txt has only blanks/comments, or the URLs lack a scheme so urlsplit finds no hostname. Fix: one full URL per line, e.g. http://app-01.lab.local.
• Every host prints ERROR ... URLError — the lab host isn't running or the name doesn't resolve. Check: is your lab VM/container up? Does the name resolve (an /etc/hosts entry or DNS)? Try the URL in curl -I first.
• stack is always unknown — the host isn't leaking recognisable clues (good hardening!) or your HINTS don't cover it. Inspect the raw headers with curl -I <url> and add signatures deliberately.
• A host you expected to skip got requested — verify it is genuinely absent from the allow-list; remember hostnames are case-sensitive as dictionary keys here (normalise to lower-case if your lab mixes cases).
• Timeouts on a host you know is up — a firewall or WAF may be dropping the odd User-Agent; that itself is a fingerprint clue worth noting, not a bug to force past.

Questions to ask when it fails: Is the target actually in scope and reachable? Am I reading the response I think I am (status code, redirects)? Is the missing signal a tool gap or genuine hardening on the target?

Memory safety

Security & safety — detection and logging for recon activity.

Recon is noisy from the defender's side, so as a responsible tester you should keep your own honest record, and as a blue-teamer you should know what to watch for.

What to log (for accountability and traceability):

• Timestamp (with timezone) of each request.
• Source (your testing host/IP) and the target host/URL.
• The resource touched and the result (HTTP status, or skipped/out-of-scope).
• The security decision made — e.g. "rejected: host not in scope allow-list."
• A correlation/engagement ID tying the action to the authorised job.

What to NEVER log: captured passwords, session cookies or tokens, API keys, private keys, full payment card numbers, or unneeded personal data you happened to see in a page. If a screenshot or response body contains such secrets, redact before storing. Use placeholders like API_KEY=<development-placeholder> in any shared notes.

Events that signal recon/abuse (defender's view):

• A burst of requests enumerating many subdomains or paths from one source.
• Sequential hits to fingerprint-revealing URLs (/wp-json/, /manager/, favicon, common error paths).
• Headless-browser User-Agents sweeping many hosts (mass screenshotting).
• Requests to hosts that should be internal-only.

Detection sources: web/reverse-proxy access logs, WAF logs, and DNS query logs.

How false positives arise: legitimate uptime monitors, search-engine crawlers, security scanners you run yourself, and CDN health checks all look a bit like recon. That's why correlation matters — one favicon fetch is noise; a single source hitting hundreds of hosts with a headless UA in two minutes is a signal. As a tester, announcing your source IP and time window to the client up front prevents your authorised work from triggering an incident.

Real-world uses

Authorised use case. A company hires a testing firm for an external assessment of *.corp.example (scope defined in writing). The tester enumerates subdomains, port-scans in-scope IPs, fingerprints each web host, mass-screenshots them from an isolated VM, and consolidates everything into a map. The map immediately surfaces a forgotten staging.corp.example running an outdated CMS with an exposed admin login — the highest-priority item, found in hours, all evidence traceable. The same map, handed to the client's blue team, becomes their attack-surface-management baseline.

Best-practice habits.

Misconceptions to keep straight: decoding a token or reading a version banner is not verification of anything — it's just reading a claim. Passing an automated fingerprint or scanner does not prove a system is secure. And nothing is ever "completely secure" — you report exposure and risk, not certainty.

Practice tasks

All tasks are lab-only: run them against hosts you own or a deliberately-vulnerable practice VM/container (e.g. one you spin up yourself). Authorisation checklist before any task: (1) you own or have written permission for every target; (2) targets are on localhost / your lab network only; (3) you have a scope allow-list file; (4) you know how to reset the lab afterward.

Beginner 1 — Read one fingerprint.

• Objective: fingerprint a single lab host by hand.
• Requirements: start a lab web app (your own container). Use curl -I http://<lab-host>/ to capture the response headers.
• Output: a 3-line note listing the Server, any X-Powered-By, and one cookie name, plus your one-sentence stack guess.
• Constraints: headers only; no login attempts.
• Hints: the Set-Cookie name often names the framework.
• Concepts: fingerprinting, evidence-with-source.

Beginner 2 — Build a scope file and run the mapper.

• Objective: enforce a trust boundary before touching anything.
• Requirements: write scope.txt with your one lab host; run the lesson's mapper.py against it.
• Output: the CSV the tool prints, with the correct IN marker.
• Constraints: exactly one authorised host.
• Hints: full URLs with http://, one per line.
• Concepts: scope allow-list, consolidation record.

Intermediate 1 — Prove the scope gate.

• Objective: demonstrate the boundary rejects bad input.
• Requirements: run the lesson's verify_scope.py; then add a second look-alike hostname of your own and confirm it is rejected.
• Input/Output: asserts pass; script prints the OK line.
• Constraints: no network calls to any out-of-scope host.
• Hints: rejection happens at hostname parsing, before any request.
• Concepts: mitigation verification, defence in depth.

Intermediate 2 — Consolidate a mini map.

• Objective: turn scattered facts into one structured map.
• Requirements: for 2-3 lab hosts, merge DNS/IP, one open port, the fingerprinted stack, and one interesting endpoint into a single CSV or table, each row carrying a source and scope column.
• Output: the map, plus a two-line prioritisation: which host you'd test first and why (reachability + impact).
• Constraints: every fact must cite where it came from.
• Hints: an exposed admin/auth endpoint outranks a static page.
• Concepts: consolidation, map-drives-plan, traceability.

Challenge — From map to defensive remediation plan.

• Objective: flip the tester's map into a defender's action list, then verify a fix.
• Requirements: take your mini map, pick the highest-risk exposure (e.g. an exposed admin panel or an outdated public app on a lab host). Write: the finding (what/why/insecure assumption/impact/how to recognise), a concrete remediation (e.g. restrict the admin path to an internal network / add auth / take the stale host down), and a mitigation verification step (re-run the mapper and curl to show the exposure is gone).
• Output: a short before/after: the map row when exposed, the same row after remediation showing it no longer appears or now requires auth, plus which log entry would have detected the original recon.
• Constraints: lab-only; defensive conclusion required — you must end by closing the exposure and proving it, not exploiting it.
• Hints: verification means re-testing, not assuming. If curl still returns the admin page, the fix isn't done.
• Concepts: remediation, mitigation verification, detection/logging, honest reporting.

Lab cleanup / reset: stop and remove the lab containers/VMs you started (docker rm -f <name> or revert the VM snapshot), delete captured screenshots and any response bodies that may hold sensitive data, and clear temporary scope files. Leave no live test host running.

Summary

• Fingerprinting reads a host's leaked clues — Server/X-Powered-By headers, cookies, framework artefacts, favicon hashes, error pages — to guess its stack. Treat banners as leads, never as proof.
• Mass screenshotting renders many in-scope web hosts to images for fast visual triage; run it from an isolated lab VM, never as a substitute for real verification.
• Consolidation links domain → IP → port → service → stack → endpoints into one structured map, every fact carrying its source and scope marker.
• The map drives the plan: prioritise by reachability and impact (auth, admin, upload, outdated public apps), not by novelty.
• Key commands/shape: curl -I <url> and a whatweb --aggression 1 style call for fingerprinting; a scope allow-list that gates every request is the trust boundary in code.
• Common mistakes: trusting version banners, running without a scope file, keeping loose tool output, and claiming "secure" from a clean scan. Decoding a banner or token is not verification, and nothing is ever "completely secure."
• Remember: recon is only valuable when it is organised, evidence-backed, in-scope, and turned toward shrinking the attack surface — map, prioritise, and for defenders, remediate and verify.

Practice with these exercises

• Final Project: HTTP status-line parser