Text Encodings

Text on the wire is bytes - UTF-8 is the default for Node 24 APIs, and TextEncoder/TextDecoder provide Web-standard conversion with explicit error handling for malformed input.

Recipe

const encoder = new TextEncoder(); // always UTF-8
const decoder = new TextDecoder('utf-8', { fatal: true });
 
const bytes = encoder.encode('Hello 世界');
const text = decoder.decode(bytes);

const buf = Buffer.from('café', 'utf8');
console.log(buf.toString('utf8'));

When to reach for this:

• Parsing upload filenames with non-ASCII characters
• Converting fetch response bytes to strings
• Validating CSV/XML declared as UTF-8 from third parties
• Interoping with browser TextEncoder in isomorphic code

Working Example

import { Buffer } from 'node:buffer';
 
function decodeUtf8Strict(bytes: Uint8Array): string {
  const decoder = new TextDecoder('utf-8', { fatal: true });
  try {
    return decoder.decode(bytes);
  } catch {
    throw new Error('Invalid UTF-8 sequence');
  }
}
 
function decodeUtf8Lossy(bytes: Uint8Array): string {
  return new TextDecoder('utf-8', { fatal: false }).decode(bytes);
}
 
const valid = Buffer.from('hello', 'utf8');
const invalid = Buffer.from([0xff, 0xfe, 0xfd]);
 
console.log(decodeUtf8Strict(valid));
console.log(decodeUtf8Lossy(invalid)); // replacement chars

// HTTP Content-Type should specify charset=utf-8
const encoder = new TextEncoder();
const body = encoder.encode(JSON.stringify({ greeting: '你好' }));

What this demonstrates:

• fatal: true rejects bad sequences - right for security-sensitive parsers
• fatal: false substitutes U+FFFD - OK for user-generated display with warnings
• TextEncoder only supports UTF-8 per spec - use iconv-lite for legacy encodings
• Buffer.toString('utf8') replaces invalid bytes by default - similar to non-fatal decoder

Deep Dive

How It Works

• JavaScript strings are UTF-16 internally; UTF-8 is the interchange encoding for I/O.
• BOM - UTF-8 BOM (EF BB BF) is optional; strip when parsing JSON that must start with {.
• Normalization - NFC vs NFD matters for string comparison (usernames, filenames).
• Buffer encodings - hex, base64, base64url, latin1 for specific protocols - not general text.

Encoding Selection

TypeScript Notes

function assertUtf8Json(raw: Buffer): unknown {
  const text = new TextDecoder('utf-8', { fatal: true }).decode(raw);
  return JSON.parse(text);
}

Gotchas

• Assuming binary string mode preserves bytes - legacy footgun. Fix: use Buffer or Uint8Array for binary.
• Double encoding UTF-8 - encoding already-valid strings for "safety". Fix: encode once at system boundary.
• Ignoring charset in Content-Type - ISO-8859-1 defaults in old clients. Fix: force UTF-8 end-to-end.
• Filename encoding on multipart - RFC 5987 filename* UTF-8 percent encoding. Fix: use framework parsers tested for i18n.
• Comparing unnormalized Unicode - visually identical usernames differ at byte level. Fix: str.normalize('NFC') before compare.

Alternatives

FAQs

Related

• Buffers Basics - allocation and slices
• Binary Protocols - non-text payloads
• Buffer Security - leaks in error messages
• Streaming HTTP Responses - charset headers

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.