LLM Comparison 2026: DeepSeek, Qwen, and Llama Guide

TL;DR

There is no durable "best LLM" table. A production choice is a versioned claim about an exact model, tokenizer, prompt, tool set, dataset, harness, serving configuration, price schedule, and date.

As of May 16, 2026:

• DeepSeek V4-Pro and V4-Flash publish open weights, 1M-token service context, and exact API model IDs. Pro is the larger quality-oriented release; Flash is the smaller throughput-oriented release.
• Qwen3.5-397B-A17B is an Apache-2.0 open-weight, native multimodal model with 397B total and 17B active parameters. Its downloadable checkpoint and hosted Qwen3.5-Plus service do not have identical context contracts.
• Llama 4 Scout and Maverick are open-weight multimodal MoE models under Meta's custom community license. Their 10M and 1M context claims are capacity limits to test, not proof of reliable retrieval at every position.
• GPT-5.5 and Claude Opus 4.7 are useful hosted reference points, but their architectures are not publicly specified. Do not fill missing facts with inference.

The practical decision rule is simple: freeze release identity, run the same representative tasks, calculate cost per accepted task, audit legal and operational constraints, then canary with an explicit rollback trigger.

What This Comparison Can and Cannot Prove

This guide separates three kinds of evidence:

Vendor benchmark tables are useful release evidence, but models, prompts, scaffolds, token budgets, and grading rules often differ. Combining the highest number from each vendor into one leaderboard creates false comparability. For a deeper treatment, see Why AI Benchmarks Fail.

Dated Release Ledger

The following facts were checked against first-party material available by May 16, 2026. Recheck every row before procurement or deployment.

Two boundaries matter:

1. A hosted alias may change behavior without changing your application code. Prefer a dated snapshot when the provider offers one, and retain a replayable evaluation record.
2. A checkpoint name is not a complete self-hosted identity. Record the weight digest, tokenizer revision, chat template, quantization artifact, inference engine, engine version, and runtime flags.

Architecture: Useful Constraint, Weak Verdict

Mixture-of-Experts separates total stored parameters from the subset activated for a token. That can improve the capability-to-compute trade-off, but three shortcuts are unsafe:

• Active parameters are not total memory. The whole checkpoint normally remains resident or distributed across storage and accelerators.
• Active parameters are not FLOPs. Attention, routing, communication, sequence length, batch shape, kernel quality, and speculative decoding also affect work.
• Architecture does not determine task quality. Data, training, post-training, tools, and serving configuration can dominate a workload result.

Qwen3.5 combines Gated DeltaNet linear-attention layers with sparse MoE and gated attention. DeepSeek V4 reports token-wise compression and DeepSeek Sparse Attention. Llama 4 reports multimodal early fusion and sparse experts. These designs create testable hypotheses about throughput, memory traffic, modalities, and context behavior; they do not establish a winner before measurement.

Closed providers do not publish enough architecture detail to place GPT-5.5 or Claude Opus 4.7 in the same architecture table. "Undisclosed" is the correct value.

Benchmark Reality: One Harness, One Claim

NIST CAISI's May 2026 evaluation is valuable because it reports multiple models under a common evaluation program. A small excerpt illustrates both the value and the limit:

CAISI used controlled agent scaffolding and weighted-token budgets, and explicitly noted that SWE-Bench scores vary with system prompts, scaffolds, and token budgets. It also found that DeepSeek V4-Pro looked stronger on DeepSeek's selected self-reported suite than on CAISI's precommitted suite. This is not evidence that vendor results are fabricated; it is evidence that the measurement contract changes the result.

Claude Opus 4.7 is absent from this CAISI table, so substituting an Anthropic score would break the common-harness comparison. OpenAI and Anthropic release tables should likewise be labeled as provider-reported rather than merged into this table.

For your own evaluation, preserve at least:

provider + model/snapshot + weight digest + tokenizer
prompt hash + tool schema hash + retrieval index version
dataset hash + split + harness commit + grader version
sampling/reasoning settings + token/tool/time budgets
attempt-level result + latency + usage + cost + failure class

Long Context: Capacity Is Not Retrieval Quality

A context-window limit tells you how much a configuration accepts. It does not prove that the model will:

• retrieve a small fact equally well at every position;
• preserve constraints across a long agent trajectory;
• distinguish relevant evidence from distractors;
• fit system instructions, tool results, images, and output inside the same budget;
• deliver acceptable latency and KV-cache cost near the limit.

Test the lengths and structures your product actually uses. Include evidence at the beginning, middle, and end; conflicting passages; reordered sections; multi-hop questions; and abstention cases. Report accuracy by length bucket and position rather than one aggregate. Long Context and Lost in the Middle explains why advertised capacity and effective use diverge.

Long context also does not eliminate retrieval. Retrieval can provide provenance, freshness, access control, selective deletion, and lower per-request cost even when the source technically fits in one prompt.

Self-Hosting Memory: Calculate a Lower Bound First

For P stored parameters at b bits per parameter, the raw weight lower bound is:

weight_GiB = P * b / 8 / 2^30

This is not a hardware prescription. Add:

• quantization scales, zero points, and packing overhead;
• embeddings, output heads, and any tensors kept at higher precision;
• KV cache for the real sequence length, batch, layer count, and cache precision;
• runtime workspaces, kernels, graph capture, and fragmentation;
• tensor or expert parallel communication buffers;
• replicas, failover capacity, observability, and safety headroom.

For sparse models, use total parameters to estimate stored weights. Active parameters are more relevant to token-level expert computation, and still do not capture attention or cross-device communication. A model-serving test should measure prefill latency, decode throughput, queue time, tail latency, memory high-water mark, and quality after quantization. A checkpoint that loads successfully has not passed a capacity test.

Cost: Use Cost per Accepted Task

Provider prices are inputs, not conclusions. On May 1, 2026, NIST cited DeepSeek V4-Pro developer prices of $1.74 per million uncached input tokens, $0.0145 for cached input, and $3.48 for output. NIST still found V4-Pro between 53% less expensive and 41% more expensive than GPT-5.4 mini across the comparable benchmarks. Different token consumption and success rates changed the result.

Use:

request_cost =
  uncached_input_tokens * uncached_input_rate
  + cached_input_tokens * cached_input_rate
  + output_tokens * output_rate
  + tool, retrieval, sandbox, and validation cost

cost_per_accepted_task =
  (all attempts + retries + fallbacks + review cost) / accepted tasks

Hold the acceptance threshold constant. A cheaper attempt that fails twice, invokes expensive tools, or needs human repair may cost more than a higher-priced model. Record p50 and p95 latency, because retries and long tails also consume user time and concurrency.

For self-hosting, include accelerator amortization or rental, idle capacity, orchestration, storage, networking, engineering time, incident response, upgrades, and compliance. AI Inference Cost Economics develops this calculation further.

Open Weight Is Not One Legal Category

These releases have materially different terms:

• DeepSeek V4: the published weight repository uses MIT.
• Qwen3.5-397B-A17B: the model card declares Apache-2.0.
• Llama 4: Meta uses the Llama 4 Community License, including attribution, acceptable-use obligations, and a separate permission condition for products above 700 million monthly active users.
• Hosted models: access is governed by service terms, usage policies, data-processing terms, regional availability, and provider controls.

"Open weights," "source available," and "Open Source AI" are not interchangeable. The Open Source Initiative's Open Source AI Definition also considers access to preferred modification forms and sufficient information about training data. A permissive weight license does not remove privacy, export, sector, data-rights, or downstream model obligations.

Create a release-specific bill of materials: model card URL, immutable artifact digest, license text and revision, code dependencies, training-data disclosures, acceptable-use policy, provider terms, and internal approval. See the Open-Source AI License Compliance Guide.

A Workload-First Evaluation Protocol

1. Write the decision contract

Define the workload, user population, prohibited failures, human-review boundary, data residency, latency SLO, budget, and rollback owner. Decide what "accepted" means before running candidates.

2. Build a representative, versioned dataset

Sample real task shapes without leaking production secrets. Preserve easy, typical, difficult, adversarial, multilingual, long-context, tool-failure, and abstention cases. Keep a hidden holdout and hash every released dataset.

3. Freeze the execution envelope

Use the same tool permissions, retrieval corpus, output schema, maximum attempts, wall-clock limit, and judge policy. If a provider-specific setting must differ, record it and narrow the claim.

4. Score quality and operations together

Measure task success with executable or human-reviewed rubrics, then segment failures. Also capture p50/p95 latency, time to first token, tokens, cache hits, tool calls, retries, safety violations, and cost per accepted task. Add confidence intervals when sample size supports them.

5. Stress the failure boundaries

Test prompt injection, unavailable tools, stale retrieval, malformed structured output, overloaded queues, provider timeouts, context truncation, policy refusal, and fallback behavior. A model that scores well only on the happy path is not release-ready.

6. Shadow, canary, and stop

Replay recent traffic in shadow mode, then expose a small canary with bounded spend and permissions. In LLMOps, the release should have predefined rollback thresholds and enough identity and evidence to reproduce why the route changed.

Runnable Go Release Gate

The gate below does not call a provider. It demonstrates the control plane that should consume already-scored evaluation output. It refuses an identity mismatch and checks task success, p95 latency, cost per accepted task, and safety violations.

package main

import (
	"fmt"
	"os"
)

type Identity struct {
	Provider    string
	Model       string
	Revision    string
	Tokenizer   string
	PromptHash  string
	DatasetHash string
	Harness     string
}

type Result struct {
	Identity
	Attempts         int
	Accepted         int
	P95LatencyMS     int
	TotalCostUSD     float64
	SafetyViolations int
}

type Limits struct {
	MinSuccessRate       float64
	MaxP95LatencyMS      int
	MaxCostPerAcceptedUS float64
}

func gate(got Result, want Identity, limits Limits) error {
	if got.Identity != want {
		return fmt.Errorf("release identity mismatch: got %+v", got.Identity)
	}
	if got.Attempts <= 0 || got.Accepted <= 0 || got.Accepted > got.Attempts {
		return fmt.Errorf("invalid attempt counts")
	}

	successRate := float64(got.Accepted) / float64(got.Attempts)
	costPerAccepted := got.TotalCostUSD / float64(got.Accepted)

	if successRate < limits.MinSuccessRate {
		return fmt.Errorf("success rate %.3f below %.3f", successRate, limits.MinSuccessRate)
	}
	if got.P95LatencyMS > limits.MaxP95LatencyMS {
		return fmt.Errorf("p95 latency %dms above %dms", got.P95LatencyMS, limits.MaxP95LatencyMS)
	}
	if costPerAccepted > limits.MaxCostPerAcceptedUS {
		return fmt.Errorf("cost per accepted task $%.4f above $%.4f", costPerAccepted, limits.MaxCostPerAcceptedUS)
	}
	if got.SafetyViolations != 0 {
		return fmt.Errorf("%d safety violations", got.SafetyViolations)
	}

	fmt.Printf("release approved: success=%.1f%% cost/accepted=$%.4f\n",
		successRate*100, costPerAccepted)
	return nil
}

func main() {
	identity := Identity{
		Provider:    "example-provider",
		Model:       "candidate-model",
		Revision:    "2026-05-16",
		Tokenizer:   "sha256:tokenizer",
		PromptHash:  "sha256:prompt",
		DatasetHash: "sha256:dataset",
		Harness:     "eval-harness@4f3a2b1",
	}
	result := Result{
		Identity:         identity,
		Attempts:         1000,
		Accepted:         950,
		P95LatencyMS:     2180,
		TotalCostUSD:     28.50,
		SafetyViolations: 0,
	}
	limits := Limits{
		MinSuccessRate:       0.92,
		MaxP95LatencyMS:      3000,
		MaxCostPerAcceptedUS: 0.04,
	}

	if err := gate(result, identity, limits); err != nil {
		fmt.Fprintln(os.Stderr, "release blocked:", err)
		os.Exit(1)
	}
}

In a real pipeline, sign the evaluation artifact, compare against a baseline and confidence interval, and keep policy-critical slices as hard gates rather than averaging them away.

Decision Matrix by Workload

Use these as hypotheses to test:

Do not route by brand alone. A model portfolio should be derived from measured task classes, with fallback semantics that preserve safety and result quality.

Common Failure Modes

FAQ

Which LLM is best for coding in May 2026?

No model is best for every coding workload. Use repository-level tasks with your languages, build system, tools, review policy, and failure cost. Record the model and harness identity; a public score from another scaffold is not a deployment guarantee.

Is DeepSeek V4 truly open source?

DeepSeek publishes V4 weights under MIT. Whether a release satisfies the broader Open Source AI Definition requires more than a weight license, including the preferred form for modification and sufficient training-data information. Audit the exact artifact and all dependencies rather than using one label as the legal conclusion.

How much memory do Qwen3.5 and Llama 4 require?

Calculate raw storage from total parameters and precision, then add metadata, higher-precision tensors, KV cache, runtime workspace, parallel communication, replicas, and headroom. Quantization changes both memory and behavior, so benchmark the final artifact on the intended stack.

What is the cheapest high-quality API?

The answer changes with workload, price region, cache hit rate, output length, retry behavior, and acceptance threshold. Compare cost per accepted task at the same quality bar.

Should I use Llama 4 or Qwen3.5 for self-hosting?

Qwen3.5 offers Apache-2.0 weights, native multimodality, and broad declared language coverage. Llama 4 offers different checkpoint sizes, declared context capacities, ecosystem support, and a custom license. Treat these as candidate filters, then test quality, latency, capacity, operability, and legal fit.

Source Notes

Checked through May 16, 2026:

• DeepSeek V4 Preview release: model sizes, 1M service context, API IDs, and open-weight links.
• DeepSeek V4-Pro model repository: model card, artifacts, technical report, and repository license.
• Qwen3.5 release and Qwen3.5-397B-A17B model card: architecture, size, modalities, language coverage, context, and Apache-2.0 declaration.
• Llama 4 model card and license: Scout and Maverick specifications and obligations.
• NIST CAISI evaluation of DeepSeek V4-Pro: controlled benchmark results, methodology caveats, and task-level cost findings.
• Introducing GPT-5.5 and Introducing Claude Opus 4.7: hosted reference availability, provider-reported evaluations, model IDs, and pricing.
• Stanford HELM Long Context: transparent long-context evaluation methodology.
• Open Source AI Definition 1.0: the distinction between weight access and Open Source AI.

Related Resources

• LLM glossary
• Model quantization
• Advanced local LLM deployment with Ollama
• Long-context failure modes
• AI inference cost economics
• Open-source AI license compliance