Spot packed data with a byte-entropy proxy

Overview

Packed, compressed or encrypted data looks statistically different from ordinary code and text: its bytes are spread evenly across all 256 values, while normal content clusters heavily around a few. That difference is what entropy measures — and you can capture it without any floating-point maths at all. Two integer signals computed from a single 256-entry histogram do the job: how many distinct byte values appear, and how dominant the most common byte is. High distinct count plus low dominance means high entropy, which is a strong triage flag for a packed section.

Why it matters

Entropy is the first thing an analyst checks on an unknown binary, because it answers "is this hiding something?" in one number. A PE section with entropy near 8 bits per byte is almost certainly packed or encrypted. Doing it with integers rather than log2 also avoids a real portability trap: the maths library is not always linked by default, so an integer proxy is both simpler and more portable — a practical lesson in choosing your tools to fit the environment.

Core concepts

What entropy captures. Shannon entropy measures unpredictability in bits per byte. Uniformly random data approaches 8.0; English text sits near 4.5; a long run of zeros approaches 0.

The histogram is the shared foundation. One pass over the buffer filling int counts[256] gives you everything — both signals below read from it, so the cost is a single scan.

Distinct byte count. How many of the 256 possible values appear at least once. Compressed or encrypted data typically uses nearly all 256; plain ASCII text uses perhaps 70.

Maximum frequency. The count of the most common byte. In normal data one value (space, zero padding, a common opcode) dominates; in high-entropy data no value stands out, so the maximum is close to n/256.

Combining them. "Many distinct values and no dominant one" is a far more reliable flag than either signal alone — padding of a single repeated byte has low distinct count and a huge maximum, and would fool a naive test.

Entropy is a hint, not a verdict. Legitimately compressed resources — images, archives, already-compressed assets — also score high. The signal narrows where to look; a human decides.

Syntax notes

#include <stddef.h>

/* both signals come from ONE histogram pass */
static void histogram(const unsigned char *b, size_t n, int counts[256]) {
    for (int i = 0; i < 256; i++) counts[i] = 0;
    for (size_t i = 0; i < n; i++) counts[b[i]]++;   /* unsigned char indexes safely 0..255 */
}

int distinct_byte_count(const unsigned char *b, size_t n) {
    int c[256]; histogram(b, n, c);
    int d = 0;
    for (int i = 0; i < 256; i++) if (c[i]) d++;
    return d;
}

int max_byte_frequency(const unsigned char *b, size_t n) {
    int c[256]; histogram(b, n, c);
    int m = 0;
    for (int i = 0; i < 256; i++) if (c[i] > m) m = c[i];
    return m;
}

Key points:

• Indexing with unsigned char is what keeps the index inside 0..255; a signed char can index negatively.
• No <math.h>, so no -lm linking problem — the integer proxy is deliberately portable.
• The histogram must be zeroed explicitly; a stack array is not zero-initialised.

Lesson

Compressed and encrypted data looks statistically flat — it uses nearly all 256 byte values, none dominating. Plaintext and code use a small alphabet with sharp peaks (spaces, e, common opcodes).

True Shannon entropy needs a logarithm, but two integer signals capture the same intuition: how many distinct byte values appear, and how tall the most-common byte's count is. High distinct + low peak = looks encrypted.

Code examples

#include <stdio.h>
#include <string.h>
static int distinct_byte_count(const unsigned char *b,size_t n){int s[256]={0},d=0;for(size_t i=0;i<n;i++)if(!s[b[i]]){s[b[i]]=1;d++;}return d;}
static int max_byte_frequency(const unsigned char *b,size_t n){int c[256]={0},m=0;for(size_t i=0;i<n;i++){int v=++c[b[i]];if(v>m)m=v;}return m;}
int main(void){
    const char *text = "the quick brown fox jumps over the lazy dog again and again";
    unsigned char enc[128];                       /* pseudo-encrypted: spread bytes */
    for(int i=0;i<128;i++) enc[i]=(unsigned char)((i*167+13)&0xff);
    size_t tn=strlen(text);
    printf("plaintext : distinct=%d  peak=%d\n",
           distinct_byte_count((const unsigned char*)text,tn), max_byte_frequency((const unsigned char*)text,tn));
    printf("encrypted : distinct=%d  peak=%d\n",
           distinct_byte_count(enc,128), max_byte_frequency(enc,128));
    puts("high distinct + low peak  =>  looks encrypted/packed");
    return 0;
}

Line by line

Common mistakes

Comparing entropy on tiny buffers (too little data to be meaningful); indexing the histogram with a signed char.

Debugging tips

Compiler errors and warnings:

• -Wchar-subscripts — you indexed the histogram with a plain char. Cast to unsigned char; this is a genuine out-of-bounds risk, not a style nit.
• Link error undefined reference to log2 if you reach for <math.h> without -lm — precisely the trap the integer approach avoids.

Runtime symptoms:

• Wildly wrong counts, or a crash. The histogram was never zeroed, or was indexed with a signed char (negative index).
• Distinct count caps out around 128. You treated the buffer as text and stopped at the first NUL, or used a signed type so high bytes wrapped.
• Everything reports high entropy. You checked only the distinct count; add the maximum-frequency test.
• Legitimate compressed files are flagged. Expected — entropy cannot distinguish packing from ordinary compression. It is a triage hint.
• Results differ by buffer size. Small buffers cannot express high entropy; require a minimum length (a few hundred bytes) before drawing conclusions.

Technique: test three buffers — all one byte value, plain ASCII text, and pseudo-random bytes. The three should be clearly separated by both signals.

Memory safety

• Histogram indexing is the safety-critical line. counts[b[i]] is in bounds only because b is unsigned char. With a signed char, any byte above 0x7F becomes a negative index — an out-of-bounds write that corrupts the stack.
• Zero the array explicitly. int c[256]; on the stack is uninitialised; reading it before writing is undefined behaviour and the results are meaningless.
• Never use strlen on a binary blob. Pass an explicit n; a sample legitimately contains NUL bytes, and stopping early would silently analyse a fraction of the data.
• Counter overflow. int counts are fine for buffers under two billion bytes, but for very large files use a wider counter.
• Read-only. const unsigned char * keeps the evidence unmodified, which matters when the same sample is analysed by several tools.

Real-world uses

Concrete uses: Malware triage tools compute per-section entropy of PE and ELF binaries; a .text section at 7.9 bits per byte indicates a packer. Forensic tools use it to spot encrypted containers among ordinary files. Data-loss-prevention systems flag high-entropy blobs leaving a network. The same histogram feeds file-type identification and compression-ratio estimation.

Professional best practices:

Beginner:

• Always index a histogram with unsigned char.
• Require a minimum buffer size before reporting an entropy verdict.

Intermediate:

• Compute entropy per section or per sliding window rather than over a whole file — a packed stub inside an otherwise normal binary is invisible in a whole-file average.
• Combine entropy with other signals (section names, imports, string counts); on its own it produces false positives on any compressed resource.
• Prefer integer proxies when they suffice; they avoid floating-point and library-linking issues entirely.

Practice tasks

1. (Beginner) Build the histogram. Implement histogram and print the ten most common byte values for a sample buffer. Concepts: zeroing, unsigned char indexing.

2. (Beginner) Two signals. Implement distinct_byte_count and max_byte_frequency. Example: a buffer of one repeated byte -> distinct 1, max n. Concepts: reading the histogram.

3. (Intermediate) Classify three buffers. Score all-one-byte, ASCII text, and pseudo-random buffers, and write down thresholds that separate them. Concepts: choosing a threshold from data.

4. (Intermediate) Sliding window. Report the highest-entropy 256-byte window in a larger buffer, so a small packed region inside normal data is not averaged away. Concepts: windowed analysis.

Summary

Packed, compressed and encrypted data spread their bytes evenly across all 256 values, and you can detect that with integers alone — no log2, and therefore no maths-library linking problem. One histogram pass yields both signals: the distinct byte count (near 256 for high-entropy data, far lower for text) and the maximum byte frequency (small when no value dominates). Requiring both together avoids the obvious false positive of a long run of padding. Index the histogram with unsigned char or a high byte becomes a negative index, zero the array before use, and remember entropy is a triage hint — ordinary compressed resources score high too.

Practice with these exercises

• Count distinct byte values
• Frequency of the most common byte