Scenario 2: auditing web servers over HTTP

The scenario

Your team wants a fast, repeatable web-server hygiene check that runs across every HTTP and HTTPS service on the network. For each web service it should report:

Missing security response headers (Strict-Transport-Security, X-Content-Type-Options, Content-Security-Policy, X-Frame-Options).
Exposed sensitive files such as /.git/HEAD, /.env, /server-status and /phpinfo.php.

The hard part isn’t sending the requests. It’s false positives. Many servers answer 200 OK for every URL (single-page apps, custom 404 pages). A script that reports “/.env exists!” whenever it sees a 200 is worse than useless. So we’ll only report a file when the content matches what that file should look like.

Step 1: the lab target

Create a small site with a deliberately exposed .git directory:

mkdir site && cd site
echo "<h1>Demo</h1>" > index.html
mkdir .git && echo "ref: refs/heads/main" > .git/HEAD
python3 -m http.server 8000

Python’s server sends no security headers, and it will serve the .git/HEAD file. Perfect: two real findings and nothing else.

Step 2: the script

Save as http-hygiene.nse:

local http      = require "http"
local shortport = require "shortport"
local stdnse    = require "stdnse"

description = [[
Checks a web server for missing security response headers and for a short list
of commonly exposed sensitive files. A file is only reported when the response
body matches a known signature, which keeps false positives low.
]]

---
-- @usage
-- nmap -p 80,443,8000 --script http-hygiene <target>
-- nmap -p 80 --script http-hygiene --script-args http-hygiene.basepath=/app/ <target>
--
-- @args http-hygiene.basepath  Path prefix to test under. Default: "/".
--
-- @output
-- PORT     STATE SERVICE
-- 8000/tcp open  http-alt
-- | http-hygiene:
-- |   missing_headers:
-- |     x-content-type-options
-- |     content-security-policy
-- |     x-frame-options
-- |   exposed_files:
-- |_    /.git/HEAD (git repository metadata)

author = "Your Name"
license = "Same as Nmap--See https://nmap.org/book/man-legal.html"
categories = {"discovery", "safe"}

portrule = shortport.http

local WANTED_HEADERS = {
  "strict-transport-security",
  "x-content-type-options",
  "content-security-policy",
  "x-frame-options",
}

-- plain = true means "this should be a text file, not an HTML page"
local PROBES = {
  {path = "/.git/HEAD",     sig = "^ref: refs/",            what = "git repository metadata", plain = true},
  {path = "/.env",          sig = "^[A-Z][A-Z0-9_]*=",      what = "environment file",        plain = true},
  {path = "/server-status", sig = "Apache Server Status",   what = "Apache server-status page"},
  {path = "/phpinfo.php",   sig = "PHP Version",            what = "phpinfo() output"},
}

action = function(host, port)
  local base = stdnse.get_script_args(SCRIPT_NAME .. ".basepath") or "/"
  if base:sub(-1) ~= "/" then base = base .. "/" end

  -- 1. Fetch the root and check headers
  local root = http.get(host, port, base)
  if not (root and root.status) then
    stdnse.debug1("no HTTP response from %s:%d", host.ip, port.number)
    return nil
  end

  local missing = {}
  for _, name in ipairs(WANTED_HEADERS) do
    -- HSTS only means something over TLS
    local applicable = (name ~= "strict-transport-security") or shortport.ssl(host, port)
    if applicable and not root.header[name] then
      missing[#missing + 1] = name
    end
  end

  -- 2. Probe for sensitive files, but only trust signature matches
  local exposed = {}
  for _, probe in ipairs(PROBES) do
    local resp = http.get(host, port, base .. probe.path:sub(2), {redirect_ok = false})
    if resp and resp.status == 200 and resp.body then
      local body = resp.body:sub(1, 4096)
      local looks_html = body:lower():find("<html", 1, true) ~= nil
      if body:find(probe.sig) and not (probe.plain and looks_html) then
        exposed[#exposed + 1] = ("%s (%s)"):format(probe.path, probe.what)
      end
    end
  end

  if #missing == 0 and #exposed == 0 then return nil end

  local out = stdnse.output_table()
  if #missing > 0 then out.missing_headers = missing end
  if #exposed > 0 then out.exposed_files = exposed end
  return out
end

Step 3: run it

nmap -p 8000 --script ./http-hygiene.nse 127.0.0.1

PORT     STATE SERVICE
8000/tcp open  http-alt
| http-hygiene:
|   missing_headers:
|     x-content-type-options
|     content-security-policy
|     x-frame-options
|   exposed_files:
|_    /.git/HEAD (git repository metadata)

strict-transport-security isn’t listed because the lab server is plain HTTP, and HSTS only applies to HTTPS. We deliberately skipped it.

Walk-through

shortport.http selects the targets. It matches HTTP and HTTPS services and a list of typical web ports (80, 443, 8000, 8080, 8088 and a few more), so it fires on our lab servers without needing -sV. A web server on an unusual port such as 8001 is only picked up if -sV identifies it as HTTP.
Header names are lower-case. The http library normalises them, so root.header["x-frame-options"] works whatever the server sent.
shortport.ssl(host, port) is a predicate you can call inside an action. It’s how we skip HSTS on plain HTTP.
Signatures beat status codes. A page must contain ref: refs/ to count as a git file. And for text files (plain = true) an HTML response is rejected outright, which kills the “SPA returns index.html for everything” false positive.
redirect_ok = false. We don’t want a login redirect to be mistaken for the file.
We cap what we read. resp.body:sub(1, 4096) means a multi-megabyte response can’t slow the signature check.
Silence when clean. If nothing is wrong we return nil. A scan of 5,000 web servers should only print the ones that need attention.

Where signatures fall short

Now add a catch-all server that returns the same page for every URL, like a single-page app. Save this as spa_server.py and run it on port 8080:

from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/html")
        self.end_headers()
        self.wfile.write(b"<html><body>PHP Version soon. Single page app.</body></html>")

HTTPServer(("127.0.0.1", 8080), H).serve_forever()

Run nmap -p 8080 --script ./http-hygiene.nse 127.0.0.1. You should see:

8080/tcp open  http-proxy
| http-hygiene:
|   missing_headers:
|     x-content-type-options
|     content-security-policy
|     x-frame-options
|   exposed_files:
|_    /phpinfo.php (phpinfo() output)

The .git/HEAD and .env probes were correctly rejected, because those are plain text files and the server answered with HTML. But /phpinfo.php is a false positive: the catch-all page happens to contain the text PHP Version, and phpinfo.php isn’t marked plain. Signatures alone have a ceiling. Fixing it is the lab below, and lesson 10 shows one solution.

Baseline probe. Before the real probes, request a random path like /nse-<random> (math.random, or a fixed unlikely string). If it returns 200, the server answers everything with 200: skip the probes for HTML-based signatures, or require that the probe body differs from the baseline body.
Two-phrase signatures. Change phpinfo.php to require both PHP Version and Configuration.
New probes. Add /.DS_Store (signature: begins with the bytes \0\0\0\1Bud1) and /robots.txt (report only if it contains Disallow: with a sensitive-looking path such as /admin).
Script argument. Add http-hygiene.probes to let the user add their own paths.

Reporting findings with `vulns`

If you’d rather have the standard State: VULNERABLE output, wrap a finding with the vulns library from lesson 6:

local vulns = require "vulns"

local vuln = {
  title = "Exposed .git directory",
  state = vulns.STATE.NOT_VULN,
  risk_factor = "Medium",
  description = [[Repository metadata is publicly downloadable.]],
}
local report = vulns.Report:new(SCRIPT_NAME, host, port)
-- inside your loop, when .git/HEAD matches:
--   vuln.state = vulns.STATE.VULN
-- and at the end:
return report:make_output(vuln)

Use vulns when a finding is a vulnerability others might triage in bulk. Use a plain table for descriptive audit data like the header list.

Checkpoint

Why not simply report any path that returns HTTP 200?

Many servers return 200 for everything (single-page apps, wildcard error pages), so the result would be mostly false positives. Matching content against a known signature, and rejecting HTML for files that should be plain text, keeps the findings trustworthy.

Why check root.status and not just root?

Because http.get returns a table even when the request failed. status is only set when a real HTTP response was received.

The scenario

Step 1: the lab target

Step 2: the script

Step 3: run it

Walk-through

Where signatures fall short

Reporting findings with vulns

Checkpoint

Reporting findings with `vulns`