Deobfuscate a single-byte-XOR payload

Overview

Single-byte XOR is the laziest obfuscation there is, and precisely because it is cheap it shows up constantly in commodity malware, dropper scripts and embedded configuration blobs. Breaking it needs no cryptanalysis: there are only 256 possible keys, so you try all of them and score each result. The insight worth carrying away is that the scoring function does the real work — decoded English or ASCII configuration text is almost entirely printable, while decoding with a wrong key produces near-random bytes. Everything in this lesson operates on a fixed sample baked into the exercise; nothing touches a live system.

Why it matters

Analysts meet obfuscated blobs daily — a configuration section in a sample, a string table hidden from strings, a payload in a document macro. Recognising the pattern and recovering the plaintext in a few lines of C is a routine triage step, and it teaches a transferable idea: when the key space is tiny, brute force plus a good scoring heuristic beats cleverness. It is also a clean lesson in why XOR is an obfuscation, not encryption.

Core concepts

XOR is self-inverse. (p ^ k) ^ k == p, so the same operation encodes and decodes. That is what makes it attractive to lazy authors and trivial to undo.

The key space is 256. A single byte means 256 candidates, so exhaustive search is instant. Compare that with a real cipher, where the key space is the entire point.

Scoring is the algorithm. For each candidate key, decode and count how many bytes look like plausible text — printable ASCII (0x20..0x7E) plus tab, newline and carriage return. The key producing the highest score is almost always correct.

Ambiguity is real. Short buffers, or buffers whose plaintext is not text at all, can score several keys equally. A robust tool reports the best few candidates rather than asserting one, and a human confirms.

Key 0 is a trap. XOR with 0 is the identity, so an already-plaintext buffer scores perfectly at k = 0. That is the correct answer, but it means "high score" alone does not prove obfuscation was present.

Stronger scoring. Frequency analysis (English letter distributions) or looking for expected substrings beats raw printability when the plaintext is prose rather than configuration data.

Syntax notes

#include <stddef.h>

/* how many bytes decode to plausible text under key k */
static int printable_score(const unsigned char *b, size_t n, int k) {
    int s = 0;
    for (size_t i = 0; i < n; i++) {
        unsigned char c = (unsigned char)(b[i] ^ k);
        if ((c >= 0x20 && c <= 0x7E) || c == '\t' || c == '\n' || c == '\r') s++;
    }
    return s;
}

/* try all 256 keys, keep the best-scoring one */
int recover_xor_key(const unsigned char *b, size_t n) {
    int best_k = 0, best_s = -1;
    for (int k = 0; k < 256; k++) {
        int s = printable_score(b, n, k);
        if (s > best_s) { best_s = s; best_k = k; }
    }
    return best_k;
}

Key points:

• unsigned char throughout — a signed char sign-extends and breaks the comparisons.
• The loop runs k = 0..255 inclusive; stopping at 255 exclusive would miss key 0xFF.
• Strict > in the comparison keeps the lowest key on a tie, which makes the result deterministic.

Lesson

Droppers and simple loaders often hide a string (a URL, a command, a second-stage key) behind a single-byte XOR. Because XOR is its own inverse — (x ^ k) ^ k == x — the same one-line loop both hides and reveals the data.

That also makes it trivial to break: a defender simply tries all 256 keys and keeps the one whose decode is mostly printable ASCII. The run below encodes a mock C2 config, then recovers the key from the bytes alone.

Code examples

#include <stdio.h>
#include <string.h>
/* Single-byte XOR is reversible, so it hides nothing from a defender. */
static int printable_score(const unsigned char *b, size_t n, int k){
    int s=0; for(size_t i=0;i<n;i++){ unsigned char c=(unsigned char)(b[i]^k);
        if((c>=0x20&&c<=0x7e)||c=='\n'||c=='\t') s++; } return s;
}
static int recover_xor_key(const unsigned char *b, size_t n){
    int best=0,bs=-1; for(int k=0;k<256;k++){ int s=printable_score(b,n,k);
        if(s>bs){bs=s;best=k;} } return best;
}
int main(void){
    const char *msg = "beacon->198.51.100.23:8443 sleep=60 jitter=15 task=exfil";
    unsigned char blob[128]; size_t n=strlen(msg);
    unsigned char key=0x80;                       /* attacker's key */
    for(size_t i=0;i<n;i++) blob[i]=(unsigned char)(msg[i]^key);
    int found=recover_xor_key(blob,n);            /* defender recovers it */
    char decoded[128];
    for(size_t i=0;i<n;i++) decoded[i]=(char)(blob[i]^found);
    decoded[n]='\0';
    printf("recovered key = 0x%02X\n", found);
    printf("decoded       = %s\n", decoded);
    return 0;
}

Line by line

Common mistakes

Sign-extending bytes; assuming the first key that produces some printable text is correct (score them all).

Debugging tips

Compiler errors and warnings:

• -Wchar-subscripts or comparison warnings if the buffer is char rather than unsigned char.
• -Wsign-compare mixing int i with a size_t n.

Runtime symptoms:

• The recovered key is wrong for high-bit keys. The buffer is a signed char, so b[i] sign-extends before the XOR. Use unsigned char.
• Key 0xFF is never chosen. The loop bound is wrong — it must be k < 256, covering 0 through 255.
• Every key scores the same. The buffer is too short, or its plaintext is binary rather than text; printability scoring only works when the plaintext is text.
• The tool picks 0 on an already-plaintext buffer. That is correct behaviour, not a bug — XOR with 0 is the identity.
• Results differ between runs. You used >= in the comparison, so later equal-scoring keys replace earlier ones. Use > for determinism.

Technique: encode a known string with a known key inside your test, then confirm the recovery returns that key. A round-trip test is far more convincing than eyeballing decoded output.

Memory safety

• unsigned char is a correctness and safety matter. Signed char sign-extends on promotion, so byte comparisons behave unpredictably for values above 0x7F.
• Bounds come from the caller. These functions take an explicit n because the buffer is binary and may legitimately contain 0x00 — never use strlen on a blob, or the scan stops at the first NUL and reports on a fraction of the data.
• Read-only input. const unsigned char * documents that the analysis does not modify the sample; keeping evidence unmodified is standard forensic practice.
• Decode into a separate output buffer, sized by the caller, rather than in place — and bound the write by the destination capacity, not the source length.
• The sample is untrusted data, not code. Everything here reads bytes; nothing executes, interprets or writes them anywhere the sample could influence.

Real-world uses

Concrete uses: Commodity malware families routinely XOR their configuration blocks, command-and-control addresses and embedded strings with a single byte to defeat naive strings output. Incident responders recover those values during triage. The same technique appears in CTF challenges, in unpacking simple installers, and in recovering obfuscated strings from firmware images.

Professional best practices:

Beginner:

• Always work on a copy of the sample and treat it as read-only data.
• Report the top few candidate keys rather than asserting a single answer.

Intermediate:

• Use frequency analysis or expected-substring matching when printability alone is ambiguous.
• Extend the idea to multi-byte (repeating-key) XOR by splitting the buffer into columns by key position and solving each column independently.
• Record which key was used in your notes — reproducibility matters when the finding ends up in a report.

Practice tasks

1. (Beginner) Apply a key. Implement void xor_apply(const unsigned char *in, size_t n, unsigned char key, unsigned char *out). Requirements: the output buffer is caller-supplied; do not modify the input. Concepts: XOR self-inverse.

2. (Beginner) Score printability. Implement printable_score counting printable ASCII plus tab/newline/CR. Concepts: the scoring heuristic.

3. (Intermediate) Recover the key. Implement recover_xor_key over all 256 candidates and verify with a round-trip test (encode with a known key, recover it). Concepts: exhaustive search, deterministic tie-breaking.

4. (Intermediate) Report ambiguity. Return the top three candidate keys with their scores, and construct a short buffer where several keys tie. Concepts: honest reporting under uncertainty.

Summary

Single-byte XOR is obfuscation, not encryption: the key space is 256, XOR is its own inverse, so exhaustive search recovers the plaintext instantly. The real work is the scoring function — decoded text is overwhelmingly printable while a wrong key yields near-random bytes, so the highest-scoring key is almost always correct. Use unsigned char so high-bit keys do not sign-extend, pass an explicit length because a binary blob may contain NUL bytes, and treat ambiguity honestly by reporting several candidates on short buffers. Note that key 0 scoring perfectly simply means the buffer was never obfuscated.

Practice with these exercises

• Apply a single-byte XOR key
• Recover a single-byte XOR key