URL Encoding: Why Context Determines What You Encode

The Wrong Question

Most introductions to URL encoding start with "which characters need to be encoded?" This is the wrong question. The correct question is: which URI component am I constructing, and what are the encoding rules for that component?

The same character — a slash (/), a plus (+), a colon (:) — may be literal in one component and must be percent-encoded in another. There is no universal "safe" or "unsafe" character list. Context determines encoding.

URI Syntax: Five Components with Different Rules

RFC 3986 defines a URI as five components:

  scheme://authority/path?query#fragment
  \____/   \_______/ \__/ \___/ \______/
    |          |       |     |      |
  scheme   authority  path  query  fragment

Each component has its own grammar and its own set of characters that may appear literally.

Component-Specific Allowed Characters

The unreserved set (always literal, never needs encoding):

A-Z  a-z  0-9  -  .  _  ~

Everything else — including characters that are literal in one component — must be percent-encoded when it appears as data rather than as a delimiter in a given component.

The Consequence

A / is literal in a path (it separates segments). But if your path segment's value contains a literal slash (e.g., a filename reports/2024), that slash must be encoded as %2F to prevent the parser from creating an extra segment.

Intended: /files/reports%2F2024
Parsed as: path segment "files", segment "reports/2024"

Wrong: /files/reports/2024
Parsed as: path segment "files", segment "reports", segment "2024"

Percent-Encoding Mechanics

Percent-encoding replaces each octet (byte) with % followed by two uppercase hex digits:

Space (0x20) → %20
é (UTF-8: 0xC3 0xA9) → %C3%A9
🚀 (UTF-8: 0xF0 0x9F 0x9A 0x80) → %F0%9F%9A%80

The pipeline for non-ASCII characters:

1. Encode the character to UTF-8 bytes (this is universally agreed upon since RFC 3987/IRI)
2. Percent-encode each byte that is not in the allowed set for the target component

This two-step process is why encodeURIComponent("中文") produces %E4%B8%AD%E6%96%87 — it's the three UTF-8 bytes of "中" and the three bytes of "文", each percent-encoded.

Two Standards for Query Strings

This is the most common source of bugs: there are two different encoding standards for query string values.

RFC 3986 Percent-Encoding

Used by: URI specification, encodeURIComponent(), Go net/url, Python urllib.parse.quote()

• Space → %20
• All reserved characters encoded when used as data

application/x-www-form-urlencoded

Used by: HTML form submission, URLSearchParams, Java URLEncoder, Python urllib.parse.urlencode()

• Space → + (not %20)
• Asterisk * is not encoded (historically)
• Tilde ~ encoding varies by implementation vintage

// RFC 3986 style
encodeURIComponent("hello world")  // "hello%20world"

// Form encoding style
new URLSearchParams({q: "hello world"}).toString()  // "q=hello+world"

Both are correct in their context. The bug happens when you mix them: construct a query string with encodeURIComponent (producing %20) but decode with a form parser (treating + as space and %20 as literal %20), or vice versa.

Which to Use

Cross-Language API Comparison

The same task — encode a query parameter value — has different APIs with different behaviors:

// JavaScript
encodeURIComponent(value)       // RFC 3986 for component
encodeURI(fullUrl)              // encodes spaces but NOT :/?#[]@!$&'()*+,;=
new URLSearchParams({k: value}) // x-www-form-urlencoded

# Python
from urllib.parse import quote, urlencode

quote(value)                    # RFC 3986, default safe='/'
quote(value, safe='')           # RFC 3986, encode everything except unreserved
urlencode({'k': value})         # x-www-form-urlencoded (space → +)

// Go
import "net/url"

url.PathEscape(value)           // RFC 3986 for path segment
url.QueryEscape(value)          // x-www-form-urlencoded (space → +)
(&url.URL{RawQuery: ...}).String() // full URL construction

// Java
import java.net.URLEncoder;
import java.net.URI;

URLEncoder.encode(value, StandardCharsets.UTF_8)  // x-www-form-urlencoded (space → +)
// Java has NO built-in RFC 3986 encoder — you must replace + with %20 manually
// or use URI constructor which encodes during construction
new URI("https", "example.com", "/path", "q=" + value, null).toASCIIString()

Note the Java trap: URLEncoder produces form encoding, not URI encoding. If you use it to build a URI path, spaces become + which are literal plus signs in path context.

encodeURI vs encodeURIComponent: When Each Is Wrong

encodeURI is designed for encoding a complete URI where you want to preserve the structural delimiters. It does not encode:

: / ? # [ ] @ ! $ & ' ( ) * + , ; =

This means encodeURI is wrong for encoding a parameter value that contains & or =:

const value = "a=1&b=2";

// Wrong: & and = are not encoded, creating phantom parameters
encodeURI("https://api.example.com/search?filter=" + value)
// → "https://api.example.com/search?filter=a=1&b=2"
// Parsed as: filter=a, extra params: 1&b=2

// Correct: encode the value separately
"https://api.example.com/search?filter=" + encodeURIComponent(value)
// → "https://api.example.com/search?filter=a%3D1%26b%3D2"

encodeURIComponent is wrong for encoding a complete URL:

// Wrong: encodes : and / in the scheme and path
encodeURIComponent("https://example.com/path")
// → "https%3A%2F%2Fexample.com%2Fpath"

The correct pattern: construct URLs by encoding each component value separately, then assemble them with literal delimiters.

Double Encoding: The Silent Corruption

Double encoding occurs when an already-encoded string is encoded again:

Original:     "hello world"
Single encode: "hello%20world"
Double encode: "hello%2520world"  ← %25 is the encoding of %

After one decode pass, you get hello%20world (still encoded). After two decode passes, you get hello world. The bug manifests when the number of encode/decode passes doesn't match.

Common Causes

1. Framework auto-encoding: you manually encode a parameter, then pass it to a library that encodes it again
2. Logging/debugging: copying an encoded value from logs and pasting it into code that will encode it
3. Redirect chains: each redirect handler encodes the target URL

Detection

If you see %25 followed by two hex digits in a URL, that's almost certainly double encoding. The %25 means a literal % was encoded, suggesting the original %XX sequence was treated as data rather than as an encoding.

URL Normalization and Comparison

RFC 3986 §6 defines normalization rules for comparing URIs. Two URIs that look different may be equivalent:

http://EXAMPLE.COM/a%2f → http://example.com/a%2f  (case normalization)
http://example.com/%41   → http://example.com/A     (percent-decode unreserved)
http://example.com/a/../b → http://example.com/b    (path normalization)
http://example.com:80/   → http://example.com/      (default port removal)

Important: %2F (encoded slash) and / (literal slash) are not equivalent in path context. A server may route /a%2Fb differently from /a/b. Normalization rules do not allow decoding reserved characters across component boundaries.

Security Boundaries

Percent-Encoding Is Not Sanitization

URL encoding ensures syntactic correctness. It does not prevent:

• SQL injection: '; DROP TABLE-- when percent-encoded is %27%3B%20DROP%20TABLE--. After the server decodes it, the payload is intact.
• XSS: <script>alert(1)</script> survives encoding/decoding round-trips
• Path traversal: ..%2F..%2Fetc%2Fpasswd — if a server decodes before checking path boundaries, this traverses

Decode-Order Vulnerabilities

A classic attack pattern:

1. Attacker sends %252e%252e%252fetc%252fpasswd
2. First decode pass (e.g., load balancer): %2e%2e%2fetc%2fpasswd
3. Second decode pass (e.g., application): ../../etc/passwd

The load balancer's path check saw no ../ because it was still double-encoded. The application decoded twice and got a traversal path.

Defense: decode exactly once, at the point where you interpret the value. Never decode a value that has already been decoded.

Open Redirect via Encoding

https://trusted.com/redirect?url=https%3A%2F%2Fevil.com

If the redirect handler decodes the url parameter and redirects without validation, the user lands on evil.com. The encoding makes it harder for humans to spot the malicious destination in the raw URL.

IRI, Punycode, and Browser Display

Browsers display URLs using a complex rendering pipeline:

1. IRI (RFC 3987): allows Unicode characters in the displayed URL
2. Punycode (RFC 3492): encodes Unicode domain names into ASCII-compatible form (münchen.de → xn--mnchen-3ya.de)
3. Display heuristics: browsers decode percent-encoded characters for display but encode them for transmission

This means the URL you see in the address bar is not necessarily what the browser sends:

Display:     https://example.com/路径/文件
Wire format: https://example.com/%E8%B7%AF%E5%BE%84/%E6%96%87%E4%BB%B6

For domain names, the encoding uses Punycode (not percent-encoding):

Display:     https://münchen.de/
Wire format: https://xn--mnchen-3ya.de/

This distinction matters for phishing detection: visually similar Unicode characters (homoglyphs) can create domains that look like trusted sites but resolve to different servers.

Summary

URL encoding is not "replace special characters with percent codes." It is a context-dependent operation governed by RFC 3986's component grammar. The same character requires different treatment depending on whether it appears in a scheme, authority, path segment, query parameter key, query parameter value, or fragment.

The two most common bugs:

1. Using the wrong encoding standard (RFC 3986 vs form-urlencoded) for the context
2. Encoding or decoding the wrong number of times (double encoding, missing decode)

Both bugs are silent — they produce valid-looking URLs that fail only in specific edge cases (parameter values containing +, &, =, /, or %).