charlesreid1.com blog

Nmap Host Discovery: All the Ways to Ask "Is Anyone There?"

Posted in Security

permalink

This is a companion post to Building an Nmap Short Course from Scratch. Where that post was about the meta - course design, lab infrastructure - this one drills into the actual first-lecture material: how Nmap decides whether a host is up.

Full lecture notes: Nmap/Short Course/Lecture 1.

Why Host Discovery Matters

Before you can scan ports, identify services, or check for vulnerabilities, you have to figure out which IP addresses on the target network actually have a machine behind them. Scanning IPs that aren't responding is a waste of time, generates a lot of unnecessary network noise, and can tip off defenders.

Think of it as making a map of active settlements before deciding which ones to explore in detail.

Ethics First

The obvious but necessary caveat: Nmap must only be used on networks where you have explicit, written authorization to scan. Unauthorized scanning can be interpreted as an attack, and in many jurisdictions is illegal.

Everything in this post assumes an isolated lab environment - which is exactly what we set up in the previous post.

The Default: Nmap's Multi-Probe Approach

If you run Nmap as a privileged user (root or sudo) without specifying any discovery options, it will fire four probes at each target:

  1. ICMP echo request (a classic ping)
  2. TCP SYN packet to port 443
  3. TCP ACK packet to port 80
  4. ICMP timestamp request

If any of the four gets a response, Nmap considers the host up.

The multi-probe approach exists because different firewalls block different things. ICMP is commonly blocked at the network edge. Port 443 might be allowed inbound because there is a web server behind it. Port 80 might respond with a TCP RST because there is nothing listening. Any one of these signals is enough.

Unprivileged users can't send raw packets, so Nmap falls back to attempting TCP connect() calls to ports 80 and 443. Less accurate, but works without root.

-sn: Just Tell Me What's Alive

Nmap's default behavior is to do host discovery and then port scan whatever comes back alive. If you only want the host discovery step, use -sn:

nmap -sn 192.168.1.0/24

-sn means "scan, no port scan." (In older versions it was -sP, "scan ping.") It runs the multi-probe discovery and prints just the list of live hosts. Very fast, very quiet compared to a full port scan, and often the first thing you run.

Everything else in this post uses -sn unless otherwise noted.

The -P Family: Picking a Specific Probe

If you want to control exactly which probe Nmap sends, use one of the -P flags.

-PE: ICMP Echo (the plain ping)

sudo nmap -sn -PE 192.168.1.100

The most familiar probe. Sends an ICMP echo request, expects an ICMP echo reply. Works when firewalls allow ICMP. Frequently blocked at network perimeters.

-PP: ICMP Timestamp

sudo nmap -sn -PP 192.168.1.101

Sends an ICMP timestamp request (type 13), expects a timestamp reply (type 14). Useful when echo requests are blocked but timestamp requests aren't - some firewall rules block ICMP type 8 (echo) but forget about type 13.

-PM: ICMP Address Mask

sudo nmap -sn -PM 192.168.1.102

Sends an ICMP address mask request. Very rarely used legitimately these days, which is exactly why it sometimes gets through firewalls that block the more common ICMP types.

The pattern: try the obvious probe, fall back to the less-obvious ones if the obvious one fails.

-PS[ports]: TCP SYN Ping

sudo nmap -sn -PS 192.168.1.0/24
sudo nmap -sn -PS22,80,443 192.168.1.50

Sends a TCP SYN packet to the given ports (default 80 if you don't specify). A response - either SYN/ACK meaning "port open" or RST meaning "port closed" - tells Nmap the host is alive.

This one is the workhorse against firewalled targets. Firewalls typically allow inbound traffic to common service ports (80, 443, 22) because there are legitimate reasons for outsiders to reach those ports. TCP SYN pings ride on that permitted traffic.

-PA[ports]: TCP ACK Ping

sudo nmap -sn -PA 192.168.1.0/24
sudo nmap -sn -PA21 192.168.1.55

Sends a TCP packet with the ACK flag set. This is a weird packet - an ACK with no prior SYN - so most operating systems respond with a TCP RST regardless of whether the port is open. If you see the RST, the host is alive.

Useful against stateful firewalls that block unsolicited SYN packets (because they aren't part of any tracked connection) but let ACKs through (because ACKs look like the middle of an established connection the firewall might have lost track of).

-PU[ports]: UDP Ping

sudo nmap -sn -PU 192.168.1.0/24
sudo nmap -sn -PU53,161 192.168.1.60

Sends a UDP packet to the given ports (default 40125, chosen because it's usually closed). If the port is closed, the host should respond with ICMP port unreachable, telling you it's alive. If the port is open, you might not get a response at all.

UDP ping is less reliable for host discovery on its own, but very useful when the target runs UDP services (DNS on 53, SNMP on 161) and when other probes are all blocked.

-PR: ARP Ping

sudo nmap -sn -PR 192.168.1.0/24

The gold standard when you are on the same Ethernet segment as your targets. ARP is the layer-2 protocol that resolves IP addresses to MAC addresses, and hosts cannot refuse to answer ARP - if they did, they would be unable to talk to anything on the local network.

Nmap automatically uses -PR for local-segment targets when run by a privileged user, unless you tell it not to (--send-ip). It's fast (no round trip past the switch) and 100% reliable for hosts that are up.

Target Specification

Independent of the probe type, you need to tell Nmap which IPs to scan.

Single addresses

nmap 192.168.1.1
nmap scanme.nmap.org

Hostnames get resolved via DNS. scanme.nmap.org is a target the Nmap project maintains specifically for people to practice against.

CIDR ranges

nmap -sn 192.168.1.0/24
nmap -sn 10.0.0.0/8

Standard CIDR notation. /24 is 256 IPs, /8 is 16.7 million (be careful).

Numeric ranges and lists

nmap -sn 192.168.1.1-100        # .1 through .100
nmap -sn 192.168.1.1,2,10,50    # specific IPs
nmap -sn 192.168.1,2,3.1-254    # cross product

That last one scans .1-.254 for each of 192.168.1.x, 192.168.2.x, and 192.168.3.x. Useful for scanning a handful of adjacent subnets.

From a file

nmap -sn -iL targets.txt

One target per line in the file. Best option when you have a large or irregular list of targets, or when the target list comes from another tool.

Excluding targets

nmap -sn 192.168.1.0/24 --exclude 192.168.1.1,192.168.1.100
nmap -sn 192.168.1.0/24 --exclude-file dontscan.txt

Critical for avoiding accidental scans on the CEO's laptop or the production database when you're supposed to be scanning a specific subnet.

Timing: -T0 through -T5

Nmap has six timing templates that control how aggressively it sends packets:

  • -T0 (paranoid): Extremely slow. One probe every few minutes. Used for IDS evasion in serious red-team engagements.
  • -T1 (sneaky): Slow. IDS evasion, but not as extreme.
  • -T2 (polite): Slower than default. Reduces bandwidth and target load. Good for scanning production infrastructure.
  • -T3 (normal): The default. Reasonable timing for most networks.
  • -T4 (aggressive): Faster. Assumes a reliable network. Good balance for lab work.
  • -T5 (insane): Very fast. Sacrifices accuracy for speed. Can overwhelm slow networks or fragile targets.
sudo nmap -sn -T4 192.168.1.0/24

For host discovery specifically, -T4 is usually the right choice in a lab environment. On production, use -T3 or -T2 and be patient.

Reading the Output

A successful scan looks like this:

Starting Nmap 7.94 ( https://nmap.org ) at 2025-05-27 14:00 PDT
Host 192.168.1.1 is up (0.00050s latency).
MAC Address: AA:BB:CC:DD:EE:FF (Realtek Semiconductor)
Host 192.168.1.10 is up (0.00080s latency).
MAC Address: 11:22:33:44:55:66 (VMware)
Nmap done: 256 IP addresses (2 hosts up) scanned in 2.10 seconds

The MAC address only shows up when you're on the same Ethernet segment. The vendor in parentheses comes from Nmap's built-in OUI database - useful for spotting VMs, or figuring out which switch port a device is behind.

When Hosts Don't Appear

"Host is down" in Nmap's output really means "Nmap didn't get a response from any of the probes it sent." That's not the same as "the host is offline." Common reasons a live host doesn't appear:

  • Restrictive firewall. The most common culprit. Dropping all probe types silently is a valid (if aggressive) defensive posture.
  • Host-based firewall. Windows Firewall, iptables, and similar can block probes even when the network firewall is permissive.
  • Wrong scan for the environment. ICMP-only discovery against a target that only accepts TCP-80 - the host won't appear even though a -PS80 scan would find it in an instant.
  • Unprivileged Nmap. Non-root Nmap has very limited discovery options. Always run as root (or via sudo) for real scanning.
  • Network layer issues. Routing problems, wrong subnet mask on the scanner, VLAN mismatches - anything that prevents packets from reaching the target.

The right response to a "no hosts up" result on a network you know is alive: try a different discovery method. If ICMP fails, try -PS22,80,443. If TCP fails, try -PA. If nothing works, you're probably up against a very well-configured firewall.

From Discovery to Deeper Scans

Once you have a list of live hosts, the natural next step is port scanning them to see what services are running. The clean way to chain this is with grepable output:

nmap -sn -oG - 192.168.1.0/24 | awk '/Up$/{print $2}' > live_hosts.txt
nmap -sV -iL live_hosts.txt

The first command produces a machine-parseable list of live IPs. The second feeds that list into a service-detection scan. This two-phase approach is efficient (you don't waste time port-scanning dead IPs) and organized (you have a saved list of live hosts to work from later).

Service detection, port scanning, NSE scripting - all of that comes in later lectures of the course. But it all starts with knowing who's home.

References

Tags:    security    nmap    host discovery    ping    arp    networking    pentesting   

Building an Nmap Short Course from Scratch

Posted in Security

permalink

We spent a good chunk of late May 2025 building a short course on Nmap from scratch - 12 lectures, 12 companion labs, plus the entire virtual lab infrastructure the students would use to run the labs. The whole thing lives on our wiki under Nmap/Short Course.

This post is not about the Nmap material itself (that comes in the next post). It is about the design decisions behind the course - why 12 lectures, why a fully isolated cloud lab, why Vagrant + Docker + Ansible instead of picking one, and what we would do differently if we started over.

Course Shape

The course is organized into three modules:

  • Module 1: Nmap Mastery - Beyond the Basics. The core Nmap material: host discovery, port scanning, service and OS detection, the scripting engine (NSE), and output formats.
  • Module 2: Red Team Nmap - Offensive Recon & Vuln Identification. Using Nmap for reconnaissance in an authorized engagement: fingerprinting, vulnerability enumeration via NSE, and integrating results into a broader recon workflow.
  • Module 3: Blue Team Nmap - Auditing, Defense & Network Monitoring. The other side: using Nmap for asset inventory, compliance checks, detecting unauthorized services, and pairing Nmap output with IDS rules.

Twelve lectures split across those three modules. Every lecture has a companion lab. The labs share a single virtual environment that gets richer over the course - by the time students are in Module 3 they are scanning the same infrastructure they attacked in Module 2.

Why Twelve Lectures

We picked 12 because it maps cleanly to a compressed summer session (a lecture + lab per week for a 12-week course, or two per week for a 6-week intensive). It also gave us enough room to introduce Nmap options in the order they build on each other, without cramming.

Twelve is a round number, each lecture is a coherent unit, and the whole thing still fits in a summer course.

The Lab Environment: The Big Boy

The lab infrastructure is the part we spent the most time on. Our requirements:

  • Students should be able to scan without touching any network they don't have explicit permission to scan
  • The lab should have a variety of realistic services and vulnerabilities, not just one target
  • Adding a new lab scenario should be a few lines of code, not a reinstall
  • The instructor should be able to reset the whole environment to a known-good state before every class

The design we landed on:

A single large EC2 instance as the lab host. We call it "the big boy" in our notes. Something like an m5.xlarge if the budget allows. Ubuntu Server LTS on the host, because Vagrant/libvirt/KVM has the smoothest experience there. Storage is EBS gp3, 80-100 GB.

Nested virtualization on that host. The EC2 instance runs KVM (via libvirt), which runs full VMs for the more heavyweight targets (and one attacker VM per student), plus Docker for lightweight containerized services. Vagrant orchestrates the VMs, Docker Compose orchestrates the containers, and both live on the same private virtual network (192.168.50.0/24 in our example).

Ansible for configuration management. Every target service, every firewall rule, every open port is defined in an Ansible playbook. Changing the lab is editing YAML, not clicking around in Docker or SSHing into VMs and running commands.

Students SSH into an attacker VM. They do not SSH into the EC2 host directly, and they do not connect via VPN. Each student (or shared pair of students) gets a preconfigured attacker VM with Nmap and the other course tools installed. That VM sits inside the lab network and can reach all the targets.

The recommendation we did not take: a VPN-based approach where students connect to the EC2 host and run Nmap from their own laptops. We ruled it out because it puts environment consistency on the student's shoulders - their local Nmap version, their local firewall, their local OS. The attacker-VM approach guarantees everyone is running the same tool from the same place.

Why This Stack

The "why Vagrant + Docker + Ansible instead of picking one" question comes up a lot. Short version:

  • Vagrant is the right tool for full VMs that need to look and behave like real hosts (a Windows target, an outdated Linux with a vulnerable SSH). Vagrant plays well with libvirt/KVM on Linux.
  • Docker Compose is the right tool for lightweight service targets: a vulnerable web app, an FTP server, a Samba share. One container, one service, one IP.
  • Ansible is the right tool for configuration - install this software, open this port, run this service - and it works identically on Vagrant VMs and Docker containers.

Each tool does waht it is best at. Trying to make Docker do full-VM work is possible but painful. Trying to make Vagrant manage 30 tiny services is possible but slow. Ansible glues them together with the same configuration language.

Cost Model

The EC2 instance is the main cost driver. Some things we do to keep it reasonable:

  • Stop the instance when not in use. Evenings, weekends, and between class sessions. We only pay for the EBS storage during those times, not the compute.
  • Start small on the instance size. If it is not enough, we upgrade.
  • Use Elastic IP or a cheap domain. Use an easy to remember domain like nmap-lab.our-course.net.

Spot Instances would save a lot more, but we ruled them out because they can be terminated with little notice, and getting evicted 15 minutes into a lab session is not the experience we want to give students.

Notes for the Instructor

A few things we would tell anyone building a course like this from scratch:

Version everything. The whole lab is a Git repo: Vagrantfile, docker-compose.yml, Ansible playbooks, and the Ansible inventory. Every branch is a different lab scenario. Rolling back is git checkout.

Test the lab reset every time. Before every session, tear the whole environment down (docker-compose down -v && vagrant destroy -f) and bring it back up from scratch. This catches "works on my machine" bugs.

Write the lab handout after you build the lab. The lab handout is the source of truth for what students see and do. If you write the handout first and then build the lab to match it, you will end up writing two things that don't quite line up.

Instrument the attacker VM. We put a shell history file that survives resets, plus a script that logs every Nmap command run during the session, so students can review what they did. Also useful when a student says "I typed the command exactly and it didn't work."

What This Cost Us

The course took roughly two solid weeks to build end-to-end - one week on the lab infrastructure, one week on the lecture content and lab exercises.

expected to run more than once. If we were doing it as a one-off, we would have skipped the Ansible layer and hard-coded the target configurations, saving maybe two or three days at the cost of a much worse experience if we ever wanted to change anything.

If you are considering doing this: yes, do it. The single biggest teaching lever is being able to show students a live network with real services responding to their scans, and this setup does that without ever putting them or you in legal jeopardy.

References

Tags:    security    nmap    teaching    curriculum    aws    vagrant    docker    ansible    pentesting   

Reading Rick Perlstein's Nixonland

Posted in Reading

permalink

We spent April and May 2025 slowly working our way through Rick Perlstein's Nixonland: The Rise of a President and the Fracturing of America (2008). It is 750 pages of dense political history covering roughly 1965 to 1972 - from the Watts riots through Nixon's re-election - and it is the second book in Perlstein's four-volume history of American conservatism.

Full notes on the wiki: Nixonland.

What The Book Actually Is

The title is misleading. Nixonland is not really a biography of Richard Nixon. Nixon is the frame - his loss in 1960, his loss in 1962, his political resurrection, his 1968 victory, his first term, his 1972 landslide - but the book is about the country around him. It is about how a nation that voted for Lyndon Johnson in a landslide in 1964 voted for Nixon in a landslide in 1972.

Perlstein's argument, compressed brutally: the disintegration of the New Deal Democratic coalition was driven less by policy failure than by cultural conflict, and Nixon's genius was recognizing early that the resentments of white middle-class voters at what they saw as the excesses of the counterculture, the civil rights movement, and liberal elites could be organized into a durable political majority. Everything Nixon did in office - the Southern Strategy, the silent majority speeches, the culture war framing, the paranoid style, the enemies list - was in service of that recognition.

Perlstein clearly sees the Nixon coalition as a tragedy for American politics. But his account of how Nixon built it is scrupulously detailed and painfully accurate.

Structure

The book has four Books (parts), each covering roughly two years:

  • Book I (1965-1967): The Watts riots, the rise of Reagan in California, the long hot summers, and the collapse of the LBJ consensus
  • Book II (1967-1968): The 1968 campaign - Tet, LBJ's withdrawal, RFK's assassination, MLK's assassination, the Chicago convention, and Nixon's narrow victory
  • Book III (1969-1970): Nixon's first two years - Vietnam, Cambodia, Kent State, the polarization, the culture war getting organized
  • Book IV (1971-1972): The lead-up to the 1972 landslide - George Wallace, the Democratic primary chaos, McGovern's nomination, the sabotage operations that would become Watergate

Every chapter is worth reading. The pacing gets faster as the book goes on - Book I is a long buildup, Book IV is a chaotic sprint.

Favorite Chapters

Chapter 5: Long, Hot Summer. The 1965 Watts riots, and Perlstein's argument that this was the moment the Democratic Party's Northern white base started to fracture. The riots were shocking - to Californians who thought the racial politics of the South did not apply to them, to Northern liberals who thought the Civil Rights Act had solved the problem, and to Black Americans who watched their neighborhood burn while the National Guard occupied it. Nothing was the same after Watts.

Chapter 15: Wednesday, August 28, 1968. The night of the Chicago police riot at the Democratic National Convention. Perlstein's account of that day is journalism at its best - hour-by-hour, with enough context that you understand not just what happened but what each side thought was happening. The image that the country took away - hippies in Grant Park, police clubbing them on live TV, the convention nominating Humphrey in the middle of the chaos - was the Democratic Party losing its future.

Chapter 30: The Party of Jefferson, Jackson, and George Wallace. The title juxtaposes the two founding icons of the Democratic Party with George Wallace, who was making a serious run at the 1972 Democratic nomination on a platform that was recognizably a throwback to the segregationist South. The chapter is about the identity crisis this produced. Wallace had a real constituency - white working-class voters, especially in the South and the outer suburbs of Northern cities, who felt abandoned by the national Democratic Party. The Democratic establishment eventually settled on the strategy of clearing the field for McGovern, on the theory that McGovern would be the easiest opponent for Nixon to beat. They were right about McGovern being easy to beat, and it did not help them.

Chapter 32: Celebrities. The chapter on the strange celebrity-politician overlap of the 1972 campaign - Warren Beatty raising money for McGovern, John Wayne stumping for Nixon, the music-industry mobilization for both sides. Perlstein uses it to make the point that mass media had transformed political campaigns into a hybrid of policy fight and cultural performance, and Nixon's team understood this better than McGovern's did.

The Wallace Chapter Is the Key

Chpater 30 gets our vote for the most illuminating chapter of the book. The reason is that Wallace exposes the whole logic of the political realignment Perlstein is describing.

Wallace was not a Republican. He was a lifelong Democrat, running in the 1972 Democratic primary. His base was traditional Democratic voters - white Southerners, white working-class voters in the industrial North, union members. His platform was populist: protectionist on trade, hawkish on Vietnam, savagely opposed to busing and to the cultural changes of the 1960s.

When Wallace won primaries in Michigan and Maryland in 1972, it was clear that a significant portion of the Democratic base wanted what he was selling. But the national Democratic Party had committed to a different vision - the McGovern coalition of professionals, young voters, minorities, and cultural liberals. Those two visions were incompatible. Wallace was shot in Maryland in May 1972 and dropped out. The Wallace voters did not just go home. Many of them voted for Nixon that November.

The core Nixon strategy, Perlstein argues, was to keep the Wallace voters in the Republican coalition permanently. He succeeded. The realignment took another decade or two to fully resolve, but the shape of it - the Republican Party becoming the party of culturally conservative white working-class voters, the Democratic Party becoming the party of cosmopolitan educated professionals - was visible by 1972.

Everything about American politics for the last 50 years has been downstream of this realignment. If you want to understand why American politics looks the way it does now, Nixonland is where that story starts.

What This Book Is Not

Nixonland is not:

  • A biography of Nixon (there are better ones - John Farrell's Richard Nixon: The Life is the current standard)
  • A history of Watergate (Perlstein covers it briefly at the end, but the whole scandal properly belongs to Perlstein's next volume, The Invisible Bridge)
  • A neutral academic history (Perlstein has a clear point of view)
  • Short (it is 750 pages, and it takes as long as it needs)

What it is: the definitive book on the political and cultural disintegration of the 1960s, and the definitive book on how the Nixon coalition was built. Both of those things are worth understanding.

Reading Order

Perlstein's four-volume history:

  1. Before the Storm: Barry Goldwater and the Unmaking of the American Consensus (2001) - covers 1960-1964
  2. Nixonland (2008) - covers 1965-1972
  3. The Invisible Bridge: The Fall of Nixon and the Rise of Reagan (2014) - covers 1973-1976
  4. Reaganland: America's Right Turn 1976-1980 (2020) - covers 1976-1980

Total page count for all four: over 3,000 pages. This is a commitment. But it is one of the best accounts of American politics (with a 10,000 foot view) from 1960 to 1980 that we have found. Perlstein's voice is sharp and opinionated but also highly enjoyable and hard to match.

Companion Reading

Other reading in a similar vein:

  • Robert Caro, The Years of Lyndon Johnson (5 vols.). Covers the same period from a very different angle - LBJ's biography. Volume 4, The Passage of Power, and the not-yet-published Volume 5 cover the years Perlstein covers.

References

Wiki:

  • Our wiki notes on Nixonland: Nixonland
  • Our wiki notes on the sequel, The Invisible Bridge: Nixonland

Blog posts:

Books:

  • Perlstein, Rick. Nixonland: The Rise of a President and the Fracturing of America. Scribner, 2008.

Tags:    reading    history    nixon    perlstein    american politics    1960s    1970s   

March 2022

How to Read Ulysses

July 2020

Applied Gitflow

September 2019

Mocking AWS in Unit Tests

May 2018

Current Projects

November 2017

A Hard(y) Math Problem