OSINT: search dorking, leaks, and metadata

Overview

Security objective. The asset you are protecting is an organization's exposed public footprint: forgotten documents, leaked API keys, revealing metadata, and internet-facing services that were never meant to be found. The threat is an attacker assembling this scattered public data into a map of where to strike next — before they ever send a single packet at the target. In this lesson you learn to detect what is leaking so a defender can shut it down, and to do so within an authorized engagement.

What OSINT is. OSINT means open-source intelligence: information that anyone can collect from public sources — search engines, code repositories, published documents, and pre-collected scan databases. No login to the target, no exploit, no special access. You are reading data the world already published.

Where it fits. This lesson builds directly on your prerequisite, Passive vs active reconnaissance. There you learned the dividing line: passive recon reads data without touching the target's systems, while active recon sends traffic to the target. OSINT is the flagship example of passive recon. When you query Shodan, Shodan scanned the target, not you — so reading its results stays passive. When you type a target's IP into your own scanner, that is active. Keep that distinction in mind throughout; it determines what you are allowed to do and when.

Where it leads. Your next lesson, Technology detection and attack-surface mapping, takes the raw footprint you gather here and organizes it into a structured attack surface. OSINT feeds that map.

Why a beginner should care. OSINT is cheap, quiet, and legal when scoped correctly — and it is the single best way to find out what your own organization is leaking. Every technique here is also a defensive audit you can run on yourself.

Why it matters

In authorized professional work, OSINT is almost always the first phase of an engagement, and it pays for itself:

• It is cheap and quiet. You build a picture of the target using public data before spending effort — or making noise — with active tools. Less noise means the client's blue team sees a realistic adversary, and you avoid tripping alarms prematurely.
• It finds the fastest wins. A leaked API key or an exposed admin portal found via a search dork can be a higher-severity finding than anything you would discover by hours of active scanning.
• It is a defensive service. Mature security teams run the same dorks, secret scans, and Shodan queries against themselves on a schedule. Attack-surface management is largely institutionalized OSINT. Teaching a client to do this is part of the deliverable.
• It sets scope and expectations. Recon reveals hosts and services the client may have forgotten they own (shadow IT). Surfacing them early prevents scope surprises later.

Done carelessly, OSINT crosses legal and ethical lines: pulling data on individuals, testing hosts that turn out to be third-party (a cloud provider, a partner), or acting on leaked credentials outside scope. The professional skill is not just finding — it is finding within authorization and turning findings into remediation.

Core concepts

OSINT is a collection of separate techniques that share one property: the data is already public. Learn each one on its own.

1. Search-engine dorking

Definition. Dorking (a.k.a. Google dorking) means using a search engine's advanced operators to narrow results to material that is exposed but not meant to be prominent.

Plain explanation. Search engines index far more than an organization intends — old documents, directory listings, login portals, backup files. Operators let you filter for exactly those.

How it works. Common operators:

You combine them: site:example.com filetype:pdf finds indexed PDFs on that domain.

When / when not. Use it against domains that are in your authorized scope, or against your own organization. Do not use it to hunt for exposed material on out-of-scope third parties, and never act on something like an exposed login just because a dork found it — finding is passive, logging in is not.

Pitfall. Dorks find what the search engine cached. Absence of a result does not prove absence of the file; the file may exist but be un-indexed. Dorking is a detection aid, not proof of security.

2. Exposed secrets in public code

Definition. Secrets (API keys, passwords, tokens, private keys) accidentally committed to public repositories.

Plain explanation. Developers sometimes hard-code a key to "make it work," then push the repo public. Even after deleting the key, it often remains in the repository's commit history.

How it works. Automated scanners look for high-entropy strings and known key formats (e.g., a string shaped like AKIA… for one cloud provider). The key stays valid until someone rotates (revokes and re-issues) it.

When / when not. In scope, a live leaked secret is a real, often high-severity finding — you report it; you do not use it to access systems unless the rules of engagement explicitly authorize proving impact. Out of scope, you stop and report responsibly.

Pitfall. A found key may be a placeholder, expired, or a decoy. Never assume validity, and never test validity by logging in outside authorization.

3. Document metadata

Definition. Hidden fields embedded in files (PDF, Office documents, images) — authors, usernames, software versions, sometimes internal file paths or GPS coordinates.

Plain explanation. When someone saves a document, the tool stamps in metadata that the reader never sees in the body. Published publicly, that metadata leaks internal details.

How it works. Tools read the file's embedded fields. From a batch of an organization's public PDFs you might learn its username convention (jsmith), its software stack, and internal server names in file paths.

When / when not. Extract metadata from documents already published by the in-scope organization. Defensively, advise clients to strip metadata before publishing.

Pitfall. Metadata can be stale or spoofed. Treat it as a lead, not a fact.

4. Internet-scan databases (Shodan, Censys)

Definition. Services that continuously scan the whole internet and let you query their stored results — open ports, service banners, TLS certificates, exposed devices.

Plain explanation. Instead of you scanning a target (active), these services already scanned everything. You search their database. That keeps your action passive.

How it works. You query by IP, domain, port, or product. Results show what services a host exposes and often the software version from the banner.

When / when not. Reading Shodan/Censys about an in-scope target is passive and generally fine. Connecting to a device you found is active and needs authorization.

Pitfall. Scan data can be hours or days stale, and an IP may belong to a shared cloud host, not your target. Confirm ownership before treating a result as "the target."

5. Privacy and scope ethics

OSINT aggregates already-public data, but aggregation itself is sensitive — compiling a dossier on an individual is invasive even from public pieces. Stay focused on the organization's assets. Do not profile people beyond what the engagement's scope requires.

Threat model

           PUBLIC INTERNET (untrusted)
  +--------------------------------------------------+
  |  Entry points an attacker reads (all passive):   |
  |   - Search-engine index (dorks)                  |
  |   - Public code repos + commit history (secrets) |
  |   - Published documents (metadata)               |
  |   - Shodan / Censys scan databases (services)    |
  +--------------------------------------------------+
                        |
                        |  aggregation (attacker builds a map)
                        v
  ============ TRUST BOUNDARY: authorized scope =====
  |  ORG ASSETS you are allowed to assess:           |
  |   - Owned domains / IP ranges                     |
  |   - Org-published documents                       |
  |   - Org's own repositories                        |
  |                                                   |
  |  OUT OF SCOPE (do not touch): third parties,      |
  |  cloud-shared IPs, individuals' personal data     |
  +--------------------------------------------------+

The attacker never crosses the boundary during OSINT — that is what makes it passive. The danger is that aggregated public data tells them exactly where to cross it later.

Knowledge check

1. What asset is being protected in an OSINT engagement? (The organization's exposed public footprint — leaked files, secrets, metadata, and forgotten services.)
2. Where is the trust boundary, and does OSINT cross it? (The boundary is authorized scope; passive OSINT reads public sources outside the target's systems and does not cross it.)
3. Which insecure assumption most often causes a secret leak? ("This repo is private / no one will look at old commits" — assuming obscurity equals safety.)

Syntax notes

There is no single language here — the "syntax" is query operators and safe command shapes. Annotated examples:

Search dork (type into a search engine):

site:example.com filetype:pdf
  |            |
  |            +-- restrict to PDF files
  +-- restrict to this authorized domain only

Combine and exclude:

site:example.com intitle:"index of" -github
  |               |                  |
  |               |                  +-- drop noisy results mentioning github
  |               +-- pages whose title suggests a directory listing
  +-- scope to the domain you are authorized to assess

Read internet-scan data (conceptual query, run against the scan service, not the target):

hostname:example.com
  +-- returns services the scan DB already recorded for this host

Key rule reflected in the syntax: every query is anchored to a scope you are authorized for (site:/hostname: your target), and every query reads a public index — none of them send traffic to the target.

Lesson

OSINT stands for open-source intelligence — anything you can find from public sources. It makes up most of passive reconnaissance (gathering information without interacting with the target's systems).

Search engine dorking

Advanced search operators narrow your results to exposed material. Common ones are site:, filetype:, intitle:, and inurl:.

Used legally, dorking surfaces material an organization forgot was indexed by search engines, such as:

• Publicly exposed documents
• Login portals
• Directory listings

Exposed secrets in code

Public code repositories sometimes leak sensitive data in their source or commit history, including:

• API keys
• Credentials
• Internal hostnames

This matters both ways. Offensively, leaked secrets are findings (when in scope). Defensively, advise clients to scan their own repositories and rotate any leaked keys.

Document metadata

Office files and PDFs embed extra details that are not visible in the document body, such as authors, usernames, software versions, and sometimes internal file paths.

Extracting this metadata from public documents can reveal:

• Usernames — useful for understanding how accounts are named
• The software the organization uses

Internet scan data

Shodan and Censys continuously scan the internet and let you query their results. You can look up exposed services, banners, certificates, and devices using default credentials.

Reading this data is passive: these services performed the scanning, not you.

Ethics

OSINT compiles data that is already public. Even so, compiling it about individuals raises privacy concerns.

Stay focused on the authorized organization and its assets. Do not profile people.

Code examples

OSINT itself is mostly querying, but the defensive side is automatable. Below is the INSECURE → SECURE → VERIFY shape applied to the most common OSINT finding: a secret leaked in a file. Everything runs locally on files you own.

(1) The insecure pattern — a config file with a hard-coded secret

WARNING: intentionally vulnerable — use only in a local, isolated, authorized lab. Do not deploy.

# config.ini  -- INSECURE: secret committed in plaintext, would leak if pushed public
[api]
endpoint = https://api.internal.example
api_key = sk_live_51H8xQeReAlLoOkInGpLaCeHoLdEr

If this file is committed to a public repo, a dork or secret scanner finds api_key, and the key stays valid until rotated. That is the vulnerability.

(2) The secure fix — keep secrets out of source, load from the environment, ignore secret files

# config.ini  -- SECURE: no secret in the file, only a reference
[api]
endpoint = https://api.internal.example
api_key_env = APP_API_KEY   # value lives in the environment, never in git

# .gitignore  -- prevent real secret files from ever being committed
.env
secrets.*
*.pem

# supply the secret at runtime, in the lab, via a placeholder
export APP_API_KEY="<development-placeholder>"

(3) VERIFY — a local scanner that must REJECT the insecure file and ACCEPT the secure one

#!/usr/bin/env bash
# scan_for_secrets.sh -- lab-only pre-commit style check on files you own.
# Exits non-zero if a likely live secret is found. No network calls.
set -euo pipefail

target="${1:?usage: scan_for_secrets.sh <file>}"

# Patterns that indicate a REAL secret value (not an env-var reference).
# 'sk_live_' + long token, or a private key header.
if grep -Eq 'sk_live_[A-Za-z0-9]{16,}|-----BEGIN [A-Z ]*PRIVATE KEY-----' "$target"; then
  echo "REJECT: possible live secret in $target"
  exit 1
fi

echo "ACCEPT: no live secret pattern in $target"
exit 0

Expected behavior.

• Run against the insecure file: ./scan_for_secrets.sh config_insecure.ini prints REJECT: possible live secret in config_insecure.ini and exits 1.
• Run against the secure file: ./scan_for_secrets.sh config_secure.ini prints ACCEPT: no live secret pattern in config_secure.ini and exits 0.

The scanner proves the fix: bad input (a real key) is rejected; good input (an env reference) is accepted. Wiring this into a pre-commit hook stops the leak at the source — the strongest remediation, because the secret never becomes public in the first place.

Line by line

Walking through the verification script, which is the load-bearing part:

Trace with the two inputs:

The key insight: the secure file references a secret by name rather than embedding its value, so the value-shaped regex cannot match it. That is exactly the property we want the fix to have, and the script proves it.

Common mistakes

Real mistakes beginners make with OSINT, and how to correct them:

1. Treating a dork hit as permission to log in.

• Wrong: Find an exposed admin portal via inurl:admin, then try credentials.
• Why wrong: Finding (reading a public index) is passive; logging in is active access, and doing it without authorization is unlawful.
• Corrected: Record the exposed portal as a finding, note it in the report, stop. Only interact if the rules of engagement explicitly authorize it.
• Recognize/prevent: Ask before every action, "Does this send traffic to, or attempt access on, the target?" If yes, it is active — check scope first.

2. Assuming a found secret is fake — or assuming it is real and using it.

• Wrong: Either ignoring a key as "probably a placeholder," or immediately using it to prove access.
• Why wrong: You cannot tell validity by looking, and testing it by authenticating is unauthorized access.
• Corrected: Report the leak, recommend immediate rotation, and let the owner confirm validity on their side.
• Recognize/prevent: Your job is detection and remediation advice, not exploitation.

3. Assuming a Shodan/dork result belongs to the target.

• Wrong: An IP appears in a query, so you treat it as in-scope and start probing.
• Why wrong: The IP may be a shared cloud provider, a CDN, or a third party. Probing it is testing someone else's system.
• Corrected: Verify ownership (WHOIS, the client's asset list) before treating any host as in-scope.
• Recognize/prevent: Maintain an explicit authorized-asset list and check every host against it.

4. Profiling individuals.

• Wrong: Building a detailed dossier on named employees from social media.
• Why wrong: Aggregating public data about people is privacy-invasive and usually outside a technical engagement's purpose.
• Corrected: Focus on organizational assets; touch personal data only when scope explicitly requires it (e.g., an authorized phishing simulation) and minimize it.

5. Believing "no dork results" means "nothing is exposed."

• Wrong: Empty results interpreted as a clean bill of health.
• Why wrong: Search engines index a subset of what exists; un-indexed files still leak.
• Corrected: Combine methods (dorks + scan DBs + your own asset inventory) and state limitations in the report.

Debugging tips

When OSINT work does not go as expected:

• A dork returns nothing. Try broader operators first (site: alone), then add filters. Different search engines index differently — try more than one. Remember: no result is not proof of absence.
• A dork returns too much noise. Add - exclusions (-www, -blog) and combine operators (site: + filetype:) to narrow.
• A Shodan/Censys result seems wrong or stale. Check the scan timestamp. Cross-check the IP's owner with WHOIS. Scan data lags reality by hours or days.
• A found IP does not match the client's assets. It is likely a shared or third-party host. Do not probe it. Confirm against the authorized asset list.
• Your secret scanner flags a false positive. Inspect the match — is it a placeholder, a test fixture, or an env-var reference? Tighten the regex to require a value shape (as in the lesson script), not just the word key.
• Your secret scanner misses a real key (false negative). Your patterns are too narrow. Add the specific formats your target's providers use, and consider entropy-based detection for random tokens.

Questions to ask when something fails:

1. Is this action passive (reading a public source) or active (touching the target)?
2. Is this host/asset actually in my authorized scope?
3. Am I confusing "not indexed" with "not present"?
4. Could this match be a placeholder or a third party?
5. Have I recorded what I did, when, and against what, for the report?

Memory safety

Security & safety: detection and logging.

OSINT is quiet by nature, which cuts both ways — the defender may not see it happening. If you run OSINT tooling (secret scanners, dork automation, scan-DB pulls) as part of an authorized engagement or as a defender auditing your own org, log it properly.

What to log (for every automated OSINT/scan action):

• Timestamp (with timezone) of the query or scan.
• Source — who/which tool/account ran it, and from where (operator, IP, engagement ID).
• Resource — the domain, file, or host queried (the in-scope asset).
• Result — found / not found, and a reference to the finding (an ID), not the sensitive value itself.
• Security decision — e.g., "secret detected → commit blocked," "asset out of scope → skipped."
• Correlation ID — to tie related actions across a session or engagement together.

What to NEVER log:

• The leaked secret value itself (API keys, tokens, passwords, private keys) — log a redacted fingerprint or an ID, never the raw key.
• Session cookies or authentication tokens.
• Full PANs (card numbers), full national IDs, or other regulated data.
• Unneeded PII about individuals — collect and store the minimum scope requires.

Which events signal abuse (from a defender's view):

• A spike in a secret-scanner finding known org keys appearing in public repos — a real leak in progress.
• Repeated external queries in scan databases for your org's assets (where observable) can precede active attacks.
• Newly indexed sensitive URLs (intitle:"index of" on your domain) appearing in search results.

How false positives arise: placeholder/test keys, env-var references mistaken for values, shared-host IPs attributed to your org, and stale scan data. Tune detections to require value-shaped matches and to verify asset ownership, and always let a human confirm before acting on a flagged secret.

Real-world uses

Authorized use case. A company hires a security firm for an external assessment. Before any active scanning, the tester runs OSINT against the authorized domains only: dorks surface a forgotten filetype:xls budget sheet and an intitle:"index of" directory listing; a secret scan of the company's public repos finds an old cloud key in commit history; document metadata reveals the jsmith-style username convention; Shodan shows a database port exposed on an owned IP. None of this touches the target's systems. The tester reports each item with remediation: remove/deindex the files, rotate the key, strip metadata before publishing, and firewall the exposed port. The client then adopts these scans as a recurring self-audit.

Professional best-practice habits.

Defensive framing. The most valuable real-world application is turning OSINT inward: attack-surface management is scheduled, institutionalized OSINT against your own organization so you find and fix leaks before an attacker aggregates them.

Practice tasks

All tasks are lab-only: use domains you own, files you create, or intentionally-vulnerable practice environments. Each ends in a defensive conclusion — remediate and verify.

Authorization checklist (do this before any task):

• I am using only assets I own or am explicitly authorized to assess.
• All scanning targets are localhost, my own files, my own repos, or a sanctioned lab.
• I will report/remediate findings, not exploit them.

Beginner 1 — Dork operator drill (no target).

• Objective: Show you can construct precise dorks.
• Requirements: Write five dork queries that would find, respectively: PDFs on a domain, directory listings, login pages, config files, and results excluding the www subdomain — all templated with example.com as a placeholder domain.
• Output: Five query strings plus one sentence each on what it detects.
• Constraints: Do not run them against any real third party; templates only.
• Hints: Combine site:, filetype:, intitle:, inurl:, and -.
• Concepts: dorking, scope anchoring.
• Defensive conclusion: For each query, state one remediation the org would apply if it returned a hit (deindex, remove, restrict).

Beginner 2 — Metadata awareness on your own file.

• Objective: See what metadata a document leaks.
• Requirements: Create a document you own, then inspect its embedded metadata fields (author, software, timestamps).
• Output: A short list of the metadata fields present.
• Constraints: Only your own files.
• Hints: Most office suites show document properties; the point is awareness, not tooling.
• Concepts: document metadata, information leakage.
• Defensive conclusion: Strip the metadata, re-inspect, and confirm the sensitive fields are gone (verification step).

Intermediate 1 — Build and test a secret scanner.

• Objective: Detect a leaked secret before it is committed.
• Requirements: Extend the lesson's scan_for_secrets.sh to detect one additional key format (a placeholder shape you define). Create one file with a fake live-shaped secret and one with only an env reference.
• Input/Output: Scanner rejects the secret file (exit 1) and accepts the env-reference file (exit 0).
• Constraints: No network calls; placeholder secrets only.
• Hints: Require a value shape (prefix + length), not just the word key.
• Concepts: secret detection, false positives, mitigation verification.
• Defensive conclusion: Describe how you would wire it into a pre-commit hook so the leak never reaches a public repo.

Intermediate 2 — Passive vs active classifier.

• Objective: Reinforce the trust boundary.
• Requirements: Take eight recon actions (mix of dorking, reading Shodan, running a port scan on the target, logging into a found portal, WHOIS lookup, etc.) and classify each passive or active, with a one-line justification.
• Output: A table: action → passive/active → why.
• Constraints: Classification only; do not perform the active ones.
• Hints: Ask "does this send traffic to, or access, the target's systems?"
• Concepts: passive vs active recon, authorization.
• Defensive conclusion: For each active action, name the authorization you would need before doing it.

Challenge — Self-audit playbook.

• Objective: Produce a repeatable defensive OSINT self-audit for an organization you own or a lab org.
• Requirements: Write a short playbook covering (a) dork templates for your own domain, (b) a repo secret-scan step with rotation guidance, (c) a metadata-stripping policy for published documents, (d) a scan-DB review step for owned IPs, and (e) a logging spec (what to log, what never to log). For each, include a verification step proving the control works.
• Output: A one-page playbook with the five sections and their verifications.
• Constraints: Owned/lab assets only; placeholders for any secrets; redacted references in logs.
• Hints: Reuse the memory_safety logging rules and the INSECURE→SECURE→VERIFY shape.
• Concepts: attack-surface management, detection/logging, mitigation verification, least privilege.
• Defensive conclusion: State how often the audit runs and who owns remediation — turning one-off recon into ongoing defense.

Summary

• OSINT is open-source intelligence — assembling a target picture from public sources, almost entirely passively (you read data others published; you do not touch the target). This is the flagship of the passive-recon concept from your prerequisite.
• Four core techniques: search dorking (site:, filetype:, inurl:, intitle:, -), leaked secrets in public code and commit history, document metadata (authors, usernames, paths), and internet-scan databases (Shodan, Censys — they scanned, so reading is passive).
• The trust boundary is authorization. Finding is passive; logging in, probing, or using a found credential is active and needs explicit scope. Verify every host is an owned/in-scope asset before treating it as the target.
• Defensive is the point. Every technique doubles as a self-audit; institutionalized OSINT is attack-surface management. For any leak, follow INSECURE → SECURE → VERIFY: e.g., a hard-coded key → env-var reference + .gitignore → a scanner that rejects the value and accepts the reference.
• Log the right things (timestamp, source, resource, result, security decision, correlation id) and never log the secret value, cookies, tokens, full PANs, or unneeded PII. Expect false positives from placeholders, env references, shared IPs, and stale scan data — have a human confirm before acting.
• Common mistakes: treating a find as permission to access; using or dismissing a found secret; attributing a shared-host IP to the target; profiling individuals; and reading "no results" as "nothing exposed."
• Remember: decoding or reading public data is not the same as being authorized to act on it, passing an automated scan does not prove security, and nothing is ever "completely secure." Stay in scope, minimize personal data, and turn findings into remediation.

Practice with these exercises

• Base64 encoded length
• Valid base64 character?