_nginX quirks

Nginx quirks the docs don't tell you

Category: devops

Nginx is everywhere, but its behavior is... unique. Here are the gotchas that cost us hours so they don't cost you.

Nginx powers half the internet. It's fast, reliable, and occasionally infuriating. After wrestling with Nginx configurations for 15+ years, here are the quirks that burned us so they don't burn you.

Symlinks: They're Bookmarks, Not Shortcuts

If you come from Windows or macOS, you think of symlinks as "shortcuts to files." That's not how Nginx sees them.

Linux folks know: symlinks are more like bookmarks that point to the actual location. Nginx follows those bookmarks, and that changes everything.

# Your site structure
/var/www/mysite -> /var/www/releases/v1.2.3
/var/www/releases/v1.2.3/index.html

When Nginx serves files, it resolves the symlink first. Your logs show /var/www/releases/v1.2.3/index.html, not /var/www/mysite/index.html.

Why this matters: Security rules, file permissions, and path-based logic all use the resolved path, not the symlink path. It's like asking for directions on the French countryside. You could go this way, or that, but just do it right and you'll be fine.

Location blocks: order is everything (sometimes)

Nginx doesn't process location blocks in the order you write them. It has its own priority system:

# This doesn't work like you think
location /api/ {
    proxy_pass http://backend;
}

location /api/health {
    return 200 "OK";
}

The /api/health request hits the first block, not the second. Why? Prefix matches (/api/) are processed before exact matches when they're longer.

Fix it with specificity:

# Exact match wins
location = /api/health {
    return 200 "OK";
}

location /api/ {
    proxy_pass http://backend;
}

try_files: More powerful than you think

Most people use try_files for SPAs:

try_files $uri $uri/ /index.html;

But it can do much more:

# Serve static files, fall back to app, then 404
try_files $uri @app =404;

location @app {
    proxy_pass http://backend;
}

The @app syntax creates a named location. Use it for complex fallback logic.

Proxy headers: the Devils in the details

Your backend app needs to know about the original request. These headers aren't automatic:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Without these: Your app thinks every request comes from 127.0.0.1 over HTTP. Good luck debugging CORS or SSL redirects.

Buffer sizes: where "big" requests quietly break

Default Nginx buffers are tiny — designed for an era when headers fit in 4KB and a "large" response was a few kilobytes of HTML. Modern web apps routinely exceed these limits: auth cookies, CSP headers, JWT tokens, large JSON payloads, file uploads.

When they do, the errors are anything but obvious.

# Sane defaults for modern APIs and SPAs

client_max_body_size 50M;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

Symptom: Everything works in local testing. In production, behind Nginx, you get a plain 404 Not Found. No logs that make sense. The app is healthy. The upstream responds fine when you curl it directly. You're hair gets more gray. There's a bitter taste in your mouth. Suddenly your gaming-chair feels like sitting on a pile of rocks...of course you want to smash your computer but to what avail?

The three-layer misdirection

Here's what actually happens, step by step:

1. Your app sends a response with headers larger than Nginx's default 4k/8k buffer(!) With cookies, CSP, auth tokens... it adds up fast.

2. Nginx can't read the response headers, it treats this as an upstream failure and generates a 502

3. Your config has error_page 502 503 504 /50x.html

4. That error page doesn't exist on disk

5. Nginx can't serve the error page either, so it falls back to its own 404

How to diagnose it

If you're getting unexplained, infuriating 404s that only appear through Nginx (not when hitting the app directly), check the error log:

sudo tail -f /var/log/nginx/error.log

Look for upstream sent too big header while reading response header from upstream. That's the tell.

SSL configuration that actually works

Let's Encrypt makes SSL easy, but the Nginx config still matters:

server {
    listen 443 ssl http2;
    server_name example.com;

    ssl_certificate /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;

    # Modern SSL settings
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
}

Don't forget the redirect:

server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}

The trailing slash trap

This catches everyone:

# These are DIFFERENT to Nginx
location /api {
    proxy_pass http://backend;
}

location /api/ {
    proxy_pass http://backend/;
}

Request to /api/users:

• First block: proxies to http://backend/users

• Second block: proxies to http://backend/api/users

Rule: Match your location and proxy_pass trailing slashes.

Log debugging that actually helps

Nginx errors are cryptic. Make them useful:

# In http block
log_format detailed '$remote_addr - $remote_user [$time_local] '
                   '"$request" $status $body_bytes_sent '
                   '"$http_referer" "$http_user_agent" '
                   'rt=$request_time uct="$upstream_connect_time" '
                   'uht="$upstream_header_time" urt="$upstream_response_time"';

# In server block
access_log /var/log/nginx/site.log detailed;
error_log /var/log/nginx/site-error.log debug;

Now you can see timing data and trace request flow.

What we learned the hard way

After years of Nginx surprises:

• Test your config: nginx -t catches syntax, not logic errors.

• Reload, don't restart: nginx -s reload keeps connections alive.

• Monitor worker processes: If they're restarting, you have memory leaks or crashes.

• Use upstream blocks: They enable health checks and load balancing.

• Keep configs simple: Complex rules breed subtle bugs.

Nginx is powerful. These quirks are the price of that power. Now you know what to watch for.