NLP Guide: Tasks, Tokenization, Models, and Evaluation

TL;DR

Natural language processing turns language into outputs that software can use: labels, entity spans, ranked documents, translations, or generated answers. A useful NLP system starts with a task contract and representative evaluation data. Model choice comes after that. Tokenization is not necessarily word segmentation, a base encoder is not a trained NER system, and fluent output is not evidence of correctness. This guide connects the main tasks to model selection, Unicode boundaries, and a runnable classification evaluator.

What NLP systems actually produce

Natural language processing is the field of computational methods for analyzing and generating human language. Natural language understanding, or NLU, emphasizes interpretation; natural language generation, or NLG, emphasizes producing text. These are useful task descriptions, not mutually exclusive software architectures.

Consider a support message: “The refund is approved, but it has not reached my bank.” A system might classify it as a refund-status request, extract a transaction identifier from the conversation, retrieve the relevant policy, and draft a response. Each step has a different correct output and failure cost.

NER finds mentions. Linking “Apple” to a particular company identifier is entity linking; deciding that “it” refers to that company is coreference resolution. Extracting a relationship between two entities is another task. Keeping these contracts separate makes errors easier to diagnose.

Speech recognition can supply text to an NLP pipeline, but transcript errors remain part of the application. If an amount is transcribed incorrectly, perfect downstream extraction still yields the wrong business result.

Build the task contract before selecting a model

A task contract defines what the system must return and how a reviewer can decide whether it is correct. “Understand customer feedback” is too broad to evaluate; “assign exactly one routing label from this versioned list” is testable.

For a support router, define billing, refund, and shipping with examples and counterexamples. Decide how to handle a message containing both a delivery complaint and a refund request. Possible policies include routing by the requested next action, allowing multiple labels, or escalating ambiguity. These choices create different datasets and metrics.

Annotation instructions should cover negation, quoted text, mixed intent, insufficient context, and new topics. Review disagreements before training: a model cannot learn a consistent boundary from contradictory labels. Keep genuine disagreement visible instead of silently replacing every difficult case with a guessed answer.

Version the label schema, dataset, preprocessing, model, tokenizer, and decision threshold together. A score produced under a two-class sentiment scheme cannot be compared directly with a later score that adds neutral and mixed classes.

Tokenization, words, and Unicode offsets

Tokenization converts text into the units a particular pipeline consumes. A tokenizer may emit words, subwords, characters, or byte-based units. Model token IDs are determined by its vocabulary and preprocessing; splitting on spaces is not a replacement.

Chinese word segmentation answers a linguistic question about word boundaries. Subword tokenization answers a model-input question about vocabulary units. A model trained with a particular tokenizer should receive that tokenizer's output, rather than text pre-segmented by an unrelated tool unless that is part of its documented training pipeline.

Why offsets need their own contract

An entity span must identify a range in a specific version of the input. Use a documented convention such as a half-open range [start, end), and state the unit.

The emoji occupies four UTF-8 bytes, one code point, and two UTF-16 code units. A browser highlighting an entity cannot interpret backend byte offsets as JavaScript string indexes.

Unicode UAX #29 distinguishes grapheme, word, and sentence boundaries and allows tailored segmentation. For example, e followed by a combining acute accent contains two code points but commonly forms one grapheme cluster. Counting runes does not implement grapheme segmentation.

Normalization, HTML decoding, whitespace cleanup, or redaction can change offsets. Preserve the original text and either retain an explicit mapping or declare that spans refer to the transformed text. Do not normalize only one side of a training or evaluation comparison.

Preprocessing depends on the task

Lowercasing and stop-word removal can help particular lexical baselines, but they are not universal prerequisites. Removing “not” can invert sentiment; removing punctuation can destroy invoice identifiers; lowercasing may remove a useful entity signal. Pretrained models generally need preprocessing compatible with their training configuration.

For sequence labeling, map subword positions back to the annotation spans and document how repeated subwords receive labels. A high token-level score may hide broken whole-entity boundaries, especially when most tokens have the non-entity label.

Choose models through controlled comparisons

Rules, statistical models, and neural models remain useful options. Their history is not a ladder that automatically makes every later approach preferable.

TF-IDF weights terms by their frequency in a document and rarity across documents. It provides a useful baseline because it reveals how much of a task is solvable from lexical evidence. Fit the vocabulary and any learned preprocessing on training data only; including test documents leaks information into the comparison.

BERT, GPT-style models, and T5

BERT pretrains bidirectional representations and adapts them to downstream tasks. A checkpoint such as bert-base-chinese is a base encoder, not proof of a trained person/location/organization extractor. A usable NER package needs a compatible task head, label mapping, tokenizer, and evidence that it works on the target domain.

GPT-style decoder models generate autoregressively, predicting subsequent tokens from the available context. They can also classify by producing a label or structured object. Calling them “generation only” confuses a model mechanism with the tasks it can perform. Conversely, a fluent explanation does not establish that a classification is correct.

T5 studies a text-to-text framework in which many tasks share a textual input/output interface. That interface simplifies task expression, but it does not remove task-specific datasets, validation, or evaluation.

For architecture details, continue with the Transformer guide. For answers that need external knowledge and source attribution, the RAG guide explains the additional retrieval and evidence stages.

Data splits and evaluation that reflect deployment

Evaluation should estimate performance on future, independent inputs. Randomly splitting individual messages can overstate quality when messages from the same conversation, customer, document template, or near-duplicate group appear on both sides.

Use a grouping rule that matches the deployment question. For new conversations, keep each conversation in one partition. For future traffic, consider a chronological holdout and document any entities that legitimately recur. Grouping by customer is useful when testing unseen customers, but it answers a stricter question than serving future messages from existing customers.

Training data fits parameters. Validation data selects prompts, thresholds, and models. Test data is reserved for the final comparison. If repeated error inspection changes your design, those examples have become development data; retain a separate holdout.

Classification metrics

For a given class, a true positive is a correct prediction of that class, a false positive is an incorrect prediction of it, and a false negative is a missed example of it. Start from the raw Confusion Matrix, then interpret Precision and Recall with their different denominators. F1 Score summarizes those two at one threshold; ROC-AUC evaluates score ranking across thresholds; Matthews Correlation Coefficient adds a symmetric summary that uses all four binary cells. They answer different questions and should not be treated as interchangeable leaderboards.

Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
F1        = 2TP / (2TP + FP + FN)
Macro F1  = mean of per-class F1 over the declared label set
Accuracy  = correct predictions / all examples

The scikit-learn F1 reference documents averaging and undefined-score policies. Macro F1 gives each declared class equal weight; weighted F1 weights by true support. Specify the label set and zero-division policy so that a missing class cannot silently disappear from the average.

Accuracy alone can reward ignoring rare classes. In a hypothetical dataset containing 95 routine messages and five urgent complaints, always predicting “routine” gives 95% accuracy and zero recall for urgent complaints. This is arithmetic, not a benchmark result.

F1 also does not encode business cost. Report the false-positive rate or precision at the operating point needed by the application, the relevant recall, and sample counts. For abstaining systems, report coverage and quality on accepted cases together; dropping every difficult case must not look like unconditional improvement. Classification Accuracy gives each exact hard-label match one vote, while Balanced Accuracy averages Recall across the declared classes. Specificity isolates how many actual negatives are correctly rejected. These views are complementary: none removes the need to inspect raw counts and error costs.

Accuracy alone can reward ignoring rare classes. In a hypothetical dataset containing 95 routine messages and five urgent complaints, always predicting “routine” gives 95% accuracy and zero recall for urgent complaints. This is arithmetic, not a benchmark result.

Other tasks need other checks

For score-producing classifiers, a Precision-Recall Curve shows how positive-alert reliability and coverage change across thresholds. When scores are presented as probabilities, Log Loss evaluates the probability assigned to every observed label; it answers a different question from hard-label Accuracy, F1, or ranking area.

Other tasks need other checks

Report uncertainty using a method appropriate to the sampling unit, such as resampling conversations rather than correlated individual messages. A tiny test slice with a perfect score remains weak evidence.

A runnable Go evaluator for a fixed-label task

This Go 1.23 example evaluates six invented predictions. It does not call or train an NLP model. The purpose is to make the label contract, confusion matrix, and averaging policy inspectable before comparing real models.

The example requires exactly one gold label and one prediction per row. Unknown labels, duplicate label definitions, and empty datasets are errors. An absent class receives zero precision, recall, and F1 and remains in the macro average. A production report should additionally record invalid-output and abstention rates instead of dropping those requests.

Save as main.go, then run go run main.go.

package main

import (
	"fmt"
	"log"
	"strings"
	"unicode/utf16"
	"unicode/utf8"
)

type Example struct {
	Gold, Predicted string
}

type ClassScore struct {
	Label                 string
	Support               int
	Precision, Recall, F1 float64
}

type Report struct {
	Matrix           [][]int
	Classes          []ClassScore
	Accuracy, MacroF1 float64
}

func ratio(n, d int) float64 {
	if d == 0 {
		return 0
	}
	return float64(n) / float64(d)
}

func evaluate(labels []string, rows []Example) (Report, error) {
	var result Report
	if len(labels) == 0 || len(rows) == 0 {
		return result, fmt.Errorf("labels and examples must be nonempty")
	}
	index := make(map[string]int, len(labels))
	for i, label := range labels {
		if strings.TrimSpace(label) == "" {
			return Report{}, fmt.Errorf("empty label")
		}
		if _, exists := index[label]; exists {
			return Report{}, fmt.Errorf("duplicate label %q", label)
		}
		index[label] = i
		result.Matrix = append(result.Matrix, make([]int, len(labels)))
	}
	correct := 0
	for i, row := range rows {
		gold, goldOK := index[row.Gold]
		predicted, predictedOK := index[row.Predicted]
		if !goldOK || !predictedOK {
			return Report{}, fmt.Errorf("unknown label at row %d", i)
		}
		result.Matrix[gold][predicted]++
		if gold == predicted {
			correct++
		}
	}
	for i, label := range labels {
		tp, support, predicted := result.Matrix[i][i], 0, 0
		for j := range labels {
			support += result.Matrix[i][j]
			predicted += result.Matrix[j][i]
		}
		score := ClassScore{
			Label: label, Support: support,
			Precision: ratio(tp, predicted),
			Recall:    ratio(tp, support),
			F1:        ratio(2*tp, support+predicted),
		}
		result.Classes = append(result.Classes, score)
		result.MacroF1 += score.F1
	}
	result.MacroF1 /= float64(len(labels))
	result.Accuracy = ratio(correct, len(rows))
	return result, nil
}

func main() {
	labels := []string{"billing", "refund", "shipping"}
	rows := []Example{
		{"billing", "billing"}, {"billing", "billing"},
		{"billing", "refund"}, {"refund", "refund"},
		{"refund", "billing"}, {"shipping", "billing"},
	}
	report, err := evaluate(labels, rows)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("accuracy=%.4f macro_f1=%.4f\n", report.Accuracy, report.MacroF1)
	fmt.Printf("rows=gold columns=predicted labels=%v matrix=%v\n", labels, report.Matrix)
	for _, score := range report.Classes {
		fmt.Printf("%s support=%d precision=%.4f recall=%.4f f1=%.4f\n",
			score.Label, score.Support, score.Precision, score.Recall, score.F1)
	}
	text := "退款🙂"
	fmt.Printf("utf8_bytes=%d code_points=%d utf16_units=%d\n",
		len(text), utf8.RuneCountInString(text), len(utf16.Encode([]rune(text))))
}

Output:

accuracy=0.5000 macro_f1=0.3571
rows=gold columns=predicted labels=[billing refund shipping] matrix=[[2 1 0] [1 1 0] [1 0 0]]
billing support=3 precision=0.5000 recall=0.6667 f1=0.5714
refund support=2 precision=0.5000 recall=0.5000 f1=0.5000
shipping support=1 precision=0.0000 recall=0.0000 f1=0.0000
utf8_bytes=10 code_points=3 utf16_units=4

The matrix exposes the important failure: no shipping request was recognized. The Unicode line demonstrates counting units for valid UTF-8; it is not an offset converter or grapheme segmenter. Before integrating actual predictions, join them to gold records by stable example ID and verify one-to-one coverage. Equal slice lengths alone do not prove alignment.

Deploy the complete pipeline, then monitor its boundaries

Deployment includes preprocessing, inference, validation, and the downstream decision. Measure the full path rather than model latency alone.

For a fixed-label classifier, reject outputs outside the schema. For an extractor, validate span boundaries and confirm extracted text matches the specified source range. For a generator, check required facts and source support; valid JSON only establishes structure.

Keep model confidence separate from calibrated probability. A score of 0.9 is not automatically a 90% chance of correctness. Select thresholds on validation data using observed error costs, and monitor whether calibration changes across languages or domains.

Track language, document length, truncation, class distribution, invalid outputs, abstention, latency, and reviewed business outcomes. Avoid logging raw sensitive text when a redacted failure record and controlled evaluation set are sufficient. Monitor every language you serve: aggregate results can hide failure on a smaller population.

A practical rollout compares the candidate with the existing system on the same traffic sample, initially without changing user-visible decisions. Keep a versioned rollback of the whole pipeline. Reverting only the model while leaving a changed tokenizer or label mapping can introduce a new failure.

FAQ

What is the difference between NLP, NLU, and NLG?

NLP covers processing human language. NLU emphasizes interpreting meaning, such as intent classification; NLG emphasizes producing language, such as summaries. A single application can combine both, and these terms do not prescribe a model architecture.

Does Chinese NLP always require word segmentation?

No. A word-based pipeline needs segmentation, but a pretrained model must use its own compatible tokenizer, which may produce characters, subwords, or byte-based tokens. Word boundaries, model tokens, and character offsets are separate contracts.

Can a base BERT checkpoint perform named entity recognition?

A base encoder produces representations, not a ready-made inventory of person and organization labels. Reliable NER requires a task-trained head or another trained extraction method, matching tokenization, label definitions, and evaluation on the target language and domain.

What accuracy should sentiment analysis achieve?

There is no universal target. Specify the domain, language, label scheme, class balance, and held-out sample first. Compare against a baseline, report per-class precision and recall, and measure the cost of false positives and missed cases.

When should I use a small classifier instead of an LLM?

A small classifier is a strong candidate for stable labels, sufficient representative data, and tight latency or deployment constraints. An LLM is useful for flexible instructions and generation. Compare them on the same held-out cases, including invalid outputs, latency, and total operating cost.

References and further reading

• Unicode UAX #29 — grapheme, word, and sentence boundaries.
• BERT — bidirectional pretraining and downstream adaptation.
• T5 — the text-to-text transfer learning framework.
• scikit-learn F1 documentation — averaging, label selection, and undefined scores.
• Evaluation beyond ROUGE and BLEU — assessing generated text beyond surface overlap.