TL;DR

An AI accelerator should be selected as a system for a defined workload, not as a chip with the largest peak number. Freeze the model, precision, traffic shape, quality gate, latency objectives, software stack, and cost boundary first. Qualify compatible GPU, TPU, and custom-silicon platforms, then compare SLO-qualified goodput, accepted-output cost, operational reliability, and migration risk. No platform wins across every model size, sequence length, batch size, or workload phase.

Table of Contents

Key Takeaways

  • No universal winner exists. Accelerator fit changes with model, precision, sequence lengths, concurrency, latency targets, and software.
  • Memory is a first-class constraint. Weight, activation, workspace, and KV cache capacity can disqualify a platform before peak compute matters.
  • Prefill and Decode are different workloads. Prefill can expose compute limits, while low-concurrency Decode is often dominated by data movement.
  • A chip is not a deployment. Runtime, compiler, kernels, topology, host, networking, and observability are part of the measured system.
  • Goodput is more useful than raw throughput. Count only outputs that meet latency, quality, safety, and validity requirements.
  • TCO includes change. Migration engineering, idle capacity, failure recovery, and portability can outweigh a lower hourly rate.

What Is an AI Accelerator?

An AI accelerator is specialized hardware and its supporting software stack for executing machine-learning workloads more efficiently than a general-purpose processor alone. The category includes GPUs, cloud TPUs and ASICs, inference-specialized processors, wafer-scale systems, and edge NPUs.

The useful unit of comparison is usually a deployable system, not a bare chip:

flowchart LR A["Model artifact and workload"] --> B["Compiler and runtime"] B --> C["Kernels and precision"] C --> D["Accelerator memory"] D --> E["Node topology and interconnect"] E --> F["Serving scheduler"] F --> G["Quality-qualified output"]

This boundary prevents a common error: comparing one vendor's per-chip dense FLOPS with another vendor's sparse system result or managed-service throughput. Those quantities describe different objects.

Architecture Families Solve Different Problems

Platform family Typical strength Typical constraint to prove
General-purpose GPU Broad framework support, programmable kernels, training and serving flexibility Cost, power, availability, and utilization under the target load
Cloud TPU or custom ASIC Hardware-software co-design and scale within a provider's supported stack Operator coverage, compiler behavior, cloud portability, and service availability
Inference-specialized accelerator Predictable execution or a memory system tuned for serving Model coverage, scale-out behavior, compiler maturity, and workload flexibility
Wafer-scale or spatial system High on-chip communication or large tightly coupled compute fabric Placement constraints, compilation, utilization, failure domain, and procurement
Edge NPU Local execution, low data movement, device power envelope Model size, supported operations, memory, update path, and quality at reduced precision

These are tendencies, not rankings. A specialized architecture may beat a GPU on one qualified workload and lose when the model, batch size, sequence length, or software revision changes.

Start With a Workload Contract

A workload contract makes every candidate solve the same problem. Without it, a benchmark can be optimized for whichever system is being promoted.

Freeze these fields before requesting results:

Contract area Required identity
Model Artifact checksum, architecture, adapter, tokenizer, prompt template
Numeric format Weight, activation, accumulation, and KV-cache precision
Request distribution Input and output length histograms, modalities, feature shapes
Traffic Arrival process, concurrency, burst behavior, cancellation, priority classes
Service objectives TTFT, inter-token latency, end-to-end percentiles, availability
Acceptance Task score, safety slices, structured-output validity, tolerance
Memory Weights, activations, runtime workspace, KV cache, fragmentation headroom
Runtime Framework, serving engine, compiler, driver, firmware, kernels
Topology Accelerator count, host, links, network, storage, placement
Operations Warmup, autoscaling, telemetry, fault injection, recovery, rollback
Cost boundary Rental or amortization, energy, network, labor, idle and reserve capacity

For an autoregressive LLM, use traces from production or a versioned synthetic generator. An average prompt length is not enough: long-tail contexts and output lengths can dominate memory and latency.

Separate Hard Qualification From Optimization

Reject a candidate before performance ranking if it cannot:

  1. load the exact model and precision without an unapproved transformation;
  2. execute every required operator or validated fallback;
  3. meet the minimum quality and safety gates;
  4. expose enough metrics to diagnose latency and failures;
  5. recover within the operational objective;
  6. be obtained in the required region and capacity class.

Only qualified candidates should enter cost and performance comparison.

Match the Workload to the Hardware

Hardware fit is determined by the workload's arithmetic intensity, memory footprint, communication pattern, and latency target. Peak tensor throughput describes only one ceiling.

Prefill and Decode Stress Different Resources

Autoregressive inference has two phases:

  • Prefill processes prompt tokens in parallel. Large matrix operations can use substantial compute, and long contexts increase attention work and temporary state.
  • Decode produces tokens iteratively. At low concurrency, repeatedly reading weights and cache state can make memory movement more important than peak arithmetic.

Batching can raise arithmetic intensity, but it also changes queueing and per-request latency. The correct question is not "How many tokens per second can this chip produce?" It is "How much accepted work can this system complete while respecting our latency distribution?"

The cross-platform study The xPU-athalon found that the optimal measured platform changed with batch size, sequence length, and model size. Its results also show why idle power, communication energy, compilation time, and software maturity belong in selection. The numerical results apply to the measured platforms and revisions; the general lesson is to preserve workload identity.

Capacity, Bandwidth, and Topology Are Separate

Three constraints are often collapsed into "GPU memory":

  1. Capacity determines whether weights, activations, workspace, and live cache fit.
  2. Bandwidth limits how quickly data can reach compute units.
  3. Topology determines the communication cost after a model spans devices or nodes.

Tensor parallelism can make a model fit, but collectives may consume the latency saved by distributing computation. Report the exact link topology and accelerator count instead of attaching a system result to one chip.

Treat Software as Part of the Accelerator

Software compatibility is a release contract, not a checkbox. A platform is qualified only when the exact model graph, numeric formats, kernels, framework, driver, firmware, and serving features work together.

For example:

  • NVIDIA Blackwell documentation describes a dual-die design, Transformer Engine, NVLink, RAS, and confidential-computing capabilities. These architecture facts do not independently prove application performance.
  • Google Cloud documents TPU v6e as a 256-chip 2D-torus Pod with specific supported Slice shapes and runtime configuration. A per-chip specification does not describe every Slice or serving behavior.
  • AWS documents Trainium2 through NeuronCore-v3, device memory, NeuronLink, logical-core configuration, and the Neuron software stack. Suitability depends on compilation and supported execution paths.
  • AMD's ROCm compatibility matrix binds hardware to specific operating systems, firmware, drivers, Python versions, frameworks, and serving engines. "Supports ROCm" is therefore too vague for reproducibility.

Build an operator and feature matrix from a real compile-and-run test:

Capability Evidence required
Model load Artifact checksum and successful load log
Operators Native kernel, compiled lowering, or measured fallback
Numeric format Actual kernel path and quality result
Dynamic shapes Tested range, recompilation behavior, failure boundary
Serving Batching, streaming, cache, cancellation, adapters, observability
Debugging Profiler, trace, memory report, error localization
Upgrade Pinned compatibility matrix and rollback procedure

Build a Reproducible Benchmark

A fair accelerator benchmark is a versioned experiment whose inputs, environment, and acceptance policy can be reconstructed. Capture identity before collecting a result.

json 
{
  "benchmarkId": "support-llm-serving/17",
  "model": {
    "artifactSha256": "sha256:...",
    "tokenizerSha256": "sha256:...",
    "precision": {
      "weights": "int8",
      "activations": "bf16",
      "kvCache": "fp8"
    }
  },
  "workload": {
    "datasetRevision": "traffic-sample/42",
    "arrivalProcess": "replayed-timestamps",
    "concurrency": [1, 8, 32, 128],
    "warmupRequests": 200
  },
  "objectives": {
    "p95TtftMs": 800,
    "p95InterTokenMs": 80,
    "minimumTaskAcceptance": 0.97
  },
  "system": {
    "accelerator": "candidate-system",
    "count": 8,
    "topology": "declared-by-provider",
    "runtimeRevision": "sha256:...",
    "driverRevision": "pinned"
  },
  "measurement": {
    "powerBoundary": "whole-system-wall-ac",
    "costBoundary": "compute-network-storage-operations",
    "runnerRevision": "sha256:..."
  }
}

Run enough repetitions to characterize variation and confidence for your decision; there is no universal run count. Randomize candidate order when shared infrastructure or time trends can bias results. Preserve raw request-level observations so percentiles and failures can be recomputed.

Use an Evidence Ladder

Evidence What it can establish What it cannot establish
Datasheet Supported formats, theoretical limits, capacity, link design Application latency, quality, utilization, or cost
Vendor benchmark Performance for the vendor's disclosed configuration Cross-vendor superiority outside that configuration
Standard benchmark Comparable result within the same rules, model, scenario, division, and category Your private workload or migration cost
Reproduced microbenchmark Kernel, bandwidth, communication, or compiler behavior End-to-end product performance
Workload replay Behavior under your model, traffic, and SLO Future workload changes without further testing
Canary Production behavior at controlled exposure Long-term capacity or every failure mode

Measure Goodput, Quality, and Cost Together

Accepted-output goodput counts completed work only when it satisfies the release policy. Raw throughput can increase while user-visible performance regresses.

For request-level serving:

text 
accepted(request) =
  completed
  AND quality_pass
  AND safety_pass
  AND schema_valid
  AND TTFT <= objective
  AND inter_token_latency <= objective
  AND end_to_end_latency <= objective

accepted_goodput = sum(accepted requests) / measured_seconds

Also report:

  • TTFT, inter-token latency, and end-to-end p50/p95/p99;
  • request, input-token, output-token, and accepted-task throughput;
  • queue time, cancellation, timeout, and error rates;
  • weight, activation, workspace, and KV-cache memory;
  • accelerator, host, network, and storage utilization;
  • quality and safety results by risk slice;
  • measured energy with an explicit boundary;
  • cost per accepted request, accepted output token, or business outcome.

Do not hide the trade-off behind one weighted score. Keep hard constraints visible and show the Pareto frontier for cost, latency, quality, and portability.

Read Vendor and MLPerf Evidence Correctly

Benchmark identity determines whether two results are comparable. Product names alone do not.

MLPerf Inference: Datacenter provides standardized scenarios, load generation, quality targets, and submission metadata. Use it with these boundaries:

  • compare the same benchmark, scenario, quality target, and rules version;
  • keep Closed and Open divisions separate;
  • distinguish Available, Preview, and Research/Development/Internal systems;
  • preserve accelerator count, host system, software stack, and submission code;
  • treat MLPerf Power as whole-system wall power for the accompanying benchmark;
  • never substitute TDP or power-supply rating for measured benchmark energy.

Vendor claims are still useful for candidate discovery and architecture understanding. Label them as vendor evidence, retain their model and system configuration, and reproduce the decision-critical claim on your workload.

Calculate Complete TCO

Complete TCO is the cost of delivering accepted work over a defined period. It is not purchase price divided by peak FLOPS.

text 
complete_cost =
  compute_rental_or_amortization
  + power_and_cooling
  + network_and_storage
  + software_and_support
  + migration_engineering
  + operations_and_incidents
  + idle_and_reserved_capacity
  + egress_and_exit_cost
  + rollback_capacity

cost_per_accepted_unit =
  complete_cost / accepted_requests_or_tokens

Use observed utilization rather than assuming every accelerator runs at peak load. Include compiler warmup, maintenance, failed requests, capacity fragmentation, reservation commitments, and multi-region standby. If power is measured, state whether the boundary covers the accelerator, node, rack, or facility and whether cooling overhead is included.

For cloud services, keep price and availability in a replaceable worksheet rather than permanent prose. For owned systems, declare depreciation period, financing, utilization, residual value, and data-center assumptions. A lower chip or instance price does not prove a lower accepted-outcome cost.

Prove Migration and Rollback

Migration is an engineering experiment with explicit exit criteria. Porting a model file is only the first step.

Migration Proof

  1. Compile and execute representative models, custom operators, and dynamic shapes.
  2. Reproduce baseline outputs and investigate every quality delta.
  3. Exercise streaming, cancellation, batching, cache pressure, and overload.
  4. Profile communication, memory, and fallback paths.
  5. Inject worker, link, host, and dependency failures.
  6. Measure cold start, scale-up, repair, and restore time.
  7. Run shadow traffic before a guarded canary.
  8. Keep the previous platform warm enough to meet the rollback objective.

Lock-in has several dimensions: model-format portability, custom kernels, compiler behavior, cloud APIs, orchestration, observability, reservations, and data egress. Record which layer owns each dependency.

Decision Framework

Select an accelerator with gates first and optimization second.

flowchart TD A["Freeze workload and acceptance contract"] --> B["Discover candidate systems"] B --> C{"Model, operators, region, capacity supported?"} C -->|No| D["Reject or document required migration"] C -->|Yes| E["Reproduce benchmark environment"] E --> F{"Quality, safety, and reliability gates pass?"} F -->|No| D F -->|Yes| G["Measure latency, goodput, energy, and complete TCO"] G --> H["Evaluate portability and migration risk"] H --> I["Shadow, canary, and rollback test"] I --> J["Select for this workload revision"]

Use a decision table that preserves evidence:

Dimension Gate or metric Evidence
Compatibility All required models, operators, and features work Compile logs, tests, compatibility matrix
Quality No unacceptable task or safety regression Versioned evaluation report
Latency Percentile objectives under target traffic Request-level benchmark
Capacity Stable concurrency and memory headroom Load test and memory trace
Efficiency Accepted goodput per system and per measured energy Benchmark plus acceptance join
Reliability Recovery and rollback objectives Fault-injection report
Economics Cost per accepted unit at observed utilization Versioned TCO worksheet
Portability Migration effort and exit path Proof-of-port and dependency inventory

The decision expires when the model, traffic distribution, runtime, price contract, region, or quality policy changes materially. Re-run the affected qualification steps instead of treating a past winner as permanent.

Frequently Asked Questions

Is a GPU always the safest default for AI workloads?

A GPU often offers broad framework and kernel support, but "safest" depends on the workload and organization. A managed TPU or custom accelerator may reduce cost or improve availability for a stable supported workload. Conversely, rapid model changes, custom operations, multi-cloud requirements, and debugging needs may make GPU portability more valuable. Prove the trade-off rather than encoding it as a universal default.

Can tokens per second compare different AI accelerators?

Only when the model artifact, tokenizer, prompt and output distributions, precision, serving engine, concurrency, sampling, hardware count, quality gate, and latency objectives match. A single-user Decode rate and a heavily batched aggregate rate answer different questions. Report both phase-aware latency and SLO-qualified aggregate goodput.

Should peak FLOPS or memory bandwidth be used for shortlisting?

Both can identify impossible or promising candidates, but neither selects a system alone. Peak FLOPS must declare precision, sparsity, and aggregation level. Memory bandwidth must be interpreted with capacity, access pattern, cache, kernels, and arithmetic intensity. Use specifications to build hypotheses, then test the full workload.

How should accelerator power efficiency be compared?

Measure energy over the same accepted workload and define the boundary. Whole-system wall energy includes effects that accelerator TDP omits. Idle behavior matters when demand is variable. MLPerf power results apply only to their accompanying benchmark, system, and scenario; they should not be converted into a universal chip efficiency ranking.

How often should accelerator selection be revisited?

Revisit it after material changes to the model, precision, context distribution, concurrency, SLO, runtime, driver, pricing, availability, or application quality policy. Use a versioned contract so the team can rerun only affected gates while retaining historical evidence.

Summary

AI accelerator selection is a workload qualification and systems-engineering problem. Define the behavior and traffic contract, reject incompatible platforms, benchmark qualified systems reproducibly, and count only work that passes product requirements. Compare complete systems with complete costs, then prove migration, failure recovery, and rollback. That method remains useful as individual GPU, TPU, and custom-silicon products change.

Sources and Further Reading