Detect a NOP sled

Overview

A NOP sled is a long run of no-operation instructions placed before a payload so that an imprecise jump still lands somewhere harmless and slides forward into the real code. On x86 the classic single-byte NOP is 0x90, so detecting a sled reduces to a run-length question: what is the longest run of 0x90 in this buffer, and does it exceed a threshold that ordinary code would never produce? This is a detection exercise operating on a fixed buffer — the point is recognising the pattern in captured data, not producing it.

Why it matters

Long NOP runs are a recognisable artefact in network captures and memory dumps, and run-length detection is a building block for many other indicators: repeated padding, zero-filled regions, and stuck sensor values all use the same scan. It also teaches threshold design — where to draw the line between a normal artefact and a suspicious one, and why that line is a judgement call rather than a constant.

Core concepts

Run-length tracking. Walk the buffer keeping the current run length and the best seen so far. Increment on a match, reset to zero on anything else, and update the maximum as you go.

Update the maximum inside the loop. Checking only after the loop misses a run that ends before the buffer does. Compare-and-update on every match is the simplest correct form.

Thresholds are judgement, not truth. Compilers do emit 0x90 for alignment padding, typically a handful of bytes. A run of 8 is unremarkable; a run of 200 is not. The threshold encodes what you consider normal for the data you are examining.

Short-circuit when only a verdict is needed. If the question is "is there a sled at least t long?", return as soon as the run reaches t rather than scanning the rest of the buffer.

Single-byte NOPs are only the simplest case. Real sleds use multi-byte NOP encodings or semantically-equivalent instruction sequences precisely to defeat naive 0x90 scanning — which is why this signal is one input among several, not a detector on its own.

Guard the threshold. A threshold below 1 is meaningless; clamp it so a caller cannot make the function trivially true.

Syntax notes

#include <stddef.h>

/* longest run of 0x90 anywhere in the buffer */
int longest_nop_run(const unsigned char *b, size_t n) {
    int best = 0, run = 0;
    for (size_t i = 0; i < n; i++) {
        if (b[i] == 0x90) { run++; if (run > best) best = run; }  /* update INSIDE the loop */
        else run = 0;
    }
    return best;
}

/* verdict only - stop as soon as the threshold is met */
int has_nop_sled(const unsigned char *b, size_t n, int threshold) {
    if (threshold < 1) threshold = 1;         /* clamp: a <1 threshold is meaningless */
    int run = 0;
    for (size_t i = 0; i < n; i++) {
        if (b[i] == 0x90) { if (++run >= threshold) return 1; }   /* early exit */
        else run = 0;
    }
    return 0;
}

Key points:

• Updating best inside the loop is what makes a run ending mid-buffer count.
• The verdict function exits early; the measuring function must scan everything.
• unsigned char so the 0x90 comparison is unambiguous.

Lesson

In classic stack-smashing exploits, the payload is preceded by a NOP sled: a long run of 0x90 bytes so that an imprecise jump still slides into the shellcode. A long 0x90 run inside a packet or file is a strong exploit indicator.

The run below measures the longest sled in a mock payload and turns it into a yes/no alert above a threshold.

Code examples

#include <stdio.h>
#include <string.h>
#include <stddef.h>
static int longest_nop_run(const unsigned char *b,size_t n){int best=0,r=0;for(size_t i=0;i<n;i++){if(b[i]==0x90){if(++r>best)best=r;}else r=0;}return best;}
static int has_nop_sled(const unsigned char *b,size_t n,int th){if(th<1)th=1;int r=0;for(size_t i=0;i<n;i++){if(b[i]==0x90){if(++r>=th)return 1;}else r=0;}return 0;}
int main(void){
    unsigned char payload[64];
    memset(payload,0x90,48);                       /* 48-byte NOP sled ... */
    memcpy(payload+48,"\x31\xc0\x50\x68//sh",12);  /* ...then 'shellcode' */
    printf("longest NOP run : %d\n", longest_nop_run(payload,sizeof payload));
    printf("sled >= 16?     : %s\n", has_nop_sled(payload,sizeof payload,16)?"YES (exploit indicator)":"no");
    return 0;
}

Line by line

Common mistakes

Not resetting the run on a non-0x90 byte; off-by-one between > and >= at the threshold.

Debugging tips

Compiler errors and warnings:

• -Wsign-compare between size_t i and an int length; keep n a size_t.
• -Wchar-subscripts style warnings if the buffer is a plain char.

Runtime symptoms:

• The longest run is reported as the run at the very end only. You updated best after the loop instead of inside it.
• A total count is returned rather than the longest run. The else run = 0 reset is missing, so the counter accumulates every NOP in the buffer.
• has_nop_sled always returns 1. The threshold was 0 or negative and not clamped.
• Nothing is detected in a real sample. The sled uses multi-byte NOP encodings rather than 0x90; this signal only covers the simplest case.
• Compiler padding triggers alerts. Your threshold is too low — alignment padding legitimately produces short 0x90 runs.

Technique: test three buffers — no NOPs, a single long run, and several separated runs where the longest is in the middle. The third catches the update-outside-the-loop bug immediately.

Memory safety

• Explicit length, always. These buffers are binary and contain NUL bytes; using strlen would stop at the first one and analyse a fraction of the data.
• unsigned char for byte comparisons so no value sign-extends unexpectedly.
• Read-only. const unsigned char * documents that the detector does not modify the captured sample — important when the same buffer feeds several detectors.
• No allocation, so nothing to leak; the run counters are scalars.
• Counter width. int is fine for buffers up to two billion bytes; use a wider type if you might scan larger regions.
• Bytes are data, never instructions. This code counts values in a buffer; nothing here interprets or executes them, which is what keeps the exercise purely analytical.

Real-world uses

Concrete uses: Network intrusion-detection systems flag long NOP runs in packet payloads as a shellcode indicator. Memory-forensics tools scan process dumps for sled-like regions. The identical run-length scan finds zero-filled sectors in disk forensics, detects stuck values in sensor telemetry, and identifies padding regions in file-format analysis.

Professional best practices:

Beginner:

• Update the maximum inside the loop.
• Clamp the threshold so a caller cannot make the test trivially true.

Intermediate:

• Treat a NOP-run hit as one weak signal among many; on its own it produces false positives on padding and false negatives on multi-byte sleds.
• Report the offset and length of the run, not just a boolean — analysts need to look at what follows it.
• Choose thresholds from measurements of your own normal data rather than copying a number from an article.

Practice tasks

1. (Beginner) Longest run. Implement longest_nop_run updating the maximum inside the loop. Example: runs of 3, 12 and 5 -> 12. Concepts: run-length tracking.

2. (Beginner) Verdict with early exit. Implement has_nop_sled with a clamped threshold. Concepts: short-circuiting, input clamping.

3. (Intermediate) Report the location. Return the offset where the longest run begins as well as its length. Concepts: actionable output.

4. (Intermediate) Generalise the byte. Parameterise the scan to find the longest run of any single value, and use it to locate zero-padding. Concepts: generalising a detector.

Summary

Detecting a NOP sled is a run-length problem: increment on 0x90, reset on anything else, and update the maximum inside the loop so a run ending mid-buffer still counts. A verdict-only variant can exit as soon as the threshold is reached, and that threshold should be clamped so a caller cannot pass 0 and make it trivially true. Choose the threshold deliberately — compilers emit short 0x90 runs as alignment padding, so a low bar produces constant false positives, while multi-byte NOP encodings evade the check entirely. Treat it as one weak signal among several, and report the offset so an analyst can examine what follows.

Practice with these exercises

• Longest NOP run
• Flag a NOP sled over a threshold