What is NLP?
NLP (Natural Language Processing) is the field that develops computational methods for analyzing, representing, retrieving, transforming, or generating human language. It includes rule-based, statistical, neural, and hybrid systems rather than one model architecture.
Quick Facts
| Full Name | Natural Language Processing |
|---|---|
| Created | 1950s (origins in computational linguistics) |
| Specification | Official Specification |
How It Works
Define the task and output contract first
Document classification returns one or more labels; sequence labeling returns a label per Token; entity extraction returns typed spans; retrieval returns ranked items; generation returns a sequence or structured object. These outputs are not interchangeable. Define label meanings, unknown and abstain states, overlap rules, span boundaries, normalization, and the downstream decision before choosing a model.
Representation choices are observable behavior
Normalization, sentence splitting, Tokenization, Unicode handling, truncation, and chat templates change what the model receives and how offsets map back to source text. English whitespace rules do not transfer to Chinese, Japanese, code-switching, emoji, or combining characters. Version preprocessing with the model and state whether offsets count bytes, Unicode code points, UTF-16 units, Tokens, or grapheme clusters.
Evaluation follows the task, language, and decision
Classification may use macro-F1 and calibration; extraction needs exact or overlap-aware span metrics; retrieval needs recall and ranking metrics; generation often combines deterministic checks, reference-based measures, model graders, and human review. Split data by user, document source, conversation, or time when random rows would leak near duplicates or future information. Report language, dialect, domain, length, and rare-label slices.
Production NLP includes lifecycle controls
A benchmark score does not cover malformed input, prompt injection, privacy, stale labels, schema changes, or delayed outcomes. Preserve source and model versions, minimize sensitive text, authorize retrieval before generation, validate structured results, monitor drift and abstention, and keep a deterministic fallback. LLMs can broaden one interface, but smaller models or rules can be faster, cheaper, and easier to audit for bounded tasks.
Key Characteristics
- Covers analysis, extraction, retrieval, transformation, and generation tasks
- Uses rule-based, statistical, neural, or hybrid methods
- Requires explicit language, domain, input, label, and output contracts
- Depends on Tokenization, normalization, and offset conventions
- Uses task-specific metrics rather than one universal NLP score
- Must account for ambiguity, multilingual variation, privacy, and distribution shift
Common Use Cases
- Extracting typed entities and relations from contracts or support records
- Classifying messages under an explicit, reviewable label taxonomy
- Retrieving and ranking passages while preserving source and access metadata
- Translating or summarizing text with terminology and factuality checks
- Generating conversational or structured output under application validation
Example
Loading code...Frequently Asked Questions
What is the difference between NLP, NLU, and NLG?
NLP is the broad field of computational methods for human language. NLU is an application label for analysis tasks such as intent, entities, relations, or meaning representations, while NLG produces language from data or context. The boundary is not a universal protocol, and one system can combine both with retrieval, rules, and human review.
Is a large language model the same as NLP?
No. An LLM is one model family used within NLP. NLP also includes lexicons, parsers, statistical classifiers, sequence taggers, retrieval systems, translation systems, and deterministic rules. For a bounded extraction or classification task, a smaller model or rule-based baseline may be easier to validate and operate.
Why must an NLP task define its label and offset contract?
A label such as neutral, organization, or relevant is meaningful only under written annotation rules. Span outputs also need an offset unit: UTF-8 bytes, Unicode code points, UTF-16 units, Tokens, and grapheme clusters can disagree. Without those contracts, training examples, APIs, evaluation code, and user interfaces can refer to different text regions or outcomes.
How should an NLP system be evaluated?
Choose metrics for the output and decision: macro-F1 and calibration for classification, exact span F1 for extraction, ranking metrics for retrieval, and task-specific checks plus human review for generation. Keep model-selection and final test data separate, prevent user or document leakage, and report results by language, domain, length, rare labels, and risk.
What commonly breaks when NLP moves to production?
Failures include changed language mix, new jargon, label-policy drift, preprocessing mismatch, truncated context, malformed input, privacy leakage, and downstream use of an uncalibrated score as a decision. Monitor input and outcome slices, preserve versions, support abstention, validate outputs, and keep rollback and human-review paths.