Direct Answer

Choose a multimodal architecture by the observable contract of the workload, not by whether a vendor calls a model “native.” A single vision-language model API can simplify integration and improve cross-modal reasoning. A modular pipeline can expose evidence and scale specialized stages independently. A hybrid can combine both, but it is not a free best-of-both-worlds default.

The correct choice is the simplest candidate that meets quality, evidence, latency, privacy, recovery, and cost requirements on representative traffic.

Separate Three Meanings of “Architecture”

Many comparisons mix three different layers:

  1. Product interface: one API accepts text, images, audio, or video.
  2. Model architecture: encoders, projectors, Transformer backbones, codecs, diffusion or flow heads, and task-specific paths inside a model artifact.
  3. Application architecture: parsers, retrieval, business rules, model calls, caches, review, and downstream effects around the model.

A unified API does not prove an encoder-free network. An end-to-end model can still contain heterogeneous components. A modular application can call one model, while a “single model” may internally execute a graph.

The M* serving paper models modern multimodal systems as computation graphs with input-dependent walks, including encoders, backbones, codecs, and non-autoregressive loops. Qwen2-VL documents a dynamic-resolution visual path and multimodal positional encoding. Architecture labels should therefore be treated as scoped descriptions, not universal implementation facts.

Three Candidate Designs

Unified-model application

flowchart LR A["Authorized mixed input"] --> B["One multimodal model endpoint"] B --> C["Typed output + evidence request"] C --> D["Application validation"]

This design minimizes application-level handoffs. It is useful for open-ended comparison across modalities, visual explanation, ambiguous input, and prototypes where a unified interface accelerates learning.

Its risks include limited intermediate evidence, a broad failure domain, provider-specific media rules, variable tokenization, difficult capacity prediction, and coupling to one model contract. The model still cannot authorize an object, prove that a cited region supports a claim, or commit an external effect exactly once.

Modular application pipeline

flowchart LR A["Input"] --> B["Decode / parse"] B --> C["OCR / ASR / vision"] C --> D["Canonical artifacts"] D --> E["Retrieve / reason"] E --> F["Validate / review"]

This design separates stages that have different data, hardware, quality metrics, or ownership. Deterministic parsing can handle a born-digital PDF, a speech recognizer can own timestamps, and a computer vision layout model can preserve page geometry before an LLM performs semantic extraction.

Its risks are contract drift, lossy handoffs, coordinate errors, duplicated queues, and cascading failures. “Modular” does not mean deterministic: OCR, ASR, detection, ranking, and generation can all be probabilistic.

Routed or hybrid system

flowchart TD A["Input contract"] --> B["Cheap deterministic checks"] B --> C{"Known measurable slice?"} C -->|"Yes"| D["Specialized pipeline"] C -->|"No"| E["Unified multimodal model"] D --> F["Common evidence contract"] E --> F F --> G{"Pass release policy?"} G -->|"Yes"| H["Accept"] G -->|"No"| I["Review or reject"]

A hybrid is useful when the workload contains separable slices. It is not justified by a vague belief that “simple tasks use the cheap model and hard tasks use the smart model.” The router must be evaluated for false-easy decisions, and both paths must emit the same evidence and lifecycle contract.

Decision Dimensions

Dimension Unified-model bias Modular-pipeline bias
Cross-modal relationship answer depends on joint interpretation modalities can be processed independently
Evidence final answer and coarse regions are sufficient field-level spans, timestamps, layout, or stage lineage required
Replacement one contract changes together components need independent upgrades
Hardware one provider or serving stack is acceptable stages need different accelerators or CPU paths
Latency fewer network hops dominate early exits, batching, or parallel stages dominate
Privacy one approved boundary can receive all inputs modalities need different residency or retention
Failure isolation a broad fallback is acceptable partial replay and stage ownership are required
Volume mixed or uncertain stable high-volume slices can be optimized separately

These are hypotheses to test. A pipeline may have higher tail latency because of queueing across stages. A unified model may be cheaper if it avoids several services. A specialized parser may outperform a large VLM on one document family and fail badly on another.

Define an Architecture-Neutral Contract

Make candidates interchangeable at the application boundary.

text 
Request
  subject, tenant, purpose
  source digest and media descriptors
  task type and requested output schema
  latency, cost, retention, and review policy

Result
  normalized output
  source revision and evidence references
  component/model revisions
  validation findings
  status: accepted | review | rejected | outcome_unknown
  usage, latency, and attempt identity

Avoid exposing internal provider labels as business fields. A customer workflow should not depend on whether one backend calls a unit an image token, media part, crop, or frame.

Compare the Right Quality Layers

OmniDocBench demonstrates why a single page-level score is insufficient: text, layout, tables, formulas, reading order, document type, and attributes reveal different weaknesses. The same principle applies beyond documents.

For each candidate, report:

  • perception: transcription, detection, localization, temporal alignment;
  • reasoning: supported claims, cross-modal consistency, abstention;
  • structure: schema validity, semantic field correctness, ordering;
  • evidence: source coverage, citation precision, coordinate error;
  • operations: p50/p95/p99 latency, throughput, queue time, retries, OOM;
  • risk: unauthorized access, injection success, harmful action, deletion;
  • economics: cost per accepted outcome, including review and failed attempts.

Do not compare an end-to-end model's full-document score with one OCR component's character accuracy. Normalize to the same business task and source distribution.

Serving and Capacity Are Part of the Choice

Multimodal inference may include heterogeneous compute, request-dependent paths, large media transfer, and non-text generation. The M* work is useful here because it treats component placement, scheduling, batching, transport, loops, and streaming as first-class serving concerns.

For each architecture, load-test:

  • media upload and decode time;
  • per-stage service time and queue time;
  • input-dependent path frequency;
  • memory at target resolution, context, batch, and concurrency;
  • backpressure and cancellation propagation;
  • warm and cold behavior;
  • partial failure and retry amplification;
  • capacity during provider or accelerator loss.

A local model is not automatically private or cheap. Downloads, telemetry, logs, browser extensions, model licenses, power, idle accelerators, and operator time remain part of the boundary and total cost.

Failure and Recovery Design

Unified-model failures

  • media or context rejected by a provider;
  • answer has no inspectable intermediate artifact;
  • output contract changes between snapshots;
  • provider outage or policy change affects the whole path;
  • a retry repeats a downstream action after an ambiguous timeout.

Mitigate with model adapters, immutable request records, bounded output, independent validation, a tested fallback, and idempotent effect execution.

Modular-pipeline failures

  • stage schemas or coordinates drift;
  • a crop removes decisive context;
  • one stale stage version contaminates a new run;
  • queues amplify tail latency;
  • retries combine artifacts from different source or model revisions.

Mitigate with content-addressed artifacts, versioned schemas, lineage checks, stage deadlines, compatible replay, and a canonical result model.

Hybrid failures

  • the router sends a difficult case to the weak path;
  • fallback shares the same root cause;
  • only one path implements deletion or authorization correctly;
  • metrics are aggregated so routing harm is invisible.

Report metrics by route and slice. Shadow routing decisions before they affect users, and preserve a simple baseline for rollback.

Runnable Go Decision Gate

This program evaluates synthetic results against a product policy. It deliberately does not prefer “native,” “pipeline,” or “hybrid.” A candidate must pass every hard gate; the winner is then the lowest measured cost per accepted outcome.

go 
package main

import (
	"fmt"
	"sort"
)

type Result struct {
	Name             string
	TaskSuccess      float64
	EvidenceCoverage float64
	HarmfulErrorRate float64
	P95LatencyMS     int
	CostPerAccepted  float64
	DeletionConforms bool
	ReplayConforms   bool
}

func eligible(result Result) bool {
	return result.TaskSuccess >= 0.90 &&
		result.EvidenceCoverage >= 0.92 &&
		result.HarmfulErrorRate <= 0.01 &&
		result.P95LatencyMS <= 1800 &&
		result.DeletionConforms &&
		result.ReplayConforms
}

func main() {
	results := []Result{
		{Name: "unified", TaskSuccess: 0.92, EvidenceCoverage: 0.86, HarmfulErrorRate: 0.009, P95LatencyMS: 1100, CostPerAccepted: 0.021, DeletionConforms: true, ReplayConforms: true},
		{Name: "modular", TaskSuccess: 0.91, EvidenceCoverage: 0.96, HarmfulErrorRate: 0.007, P95LatencyMS: 1450, CostPerAccepted: 0.016, DeletionConforms: true, ReplayConforms: true},
		{Name: "hybrid", TaskSuccess: 0.94, EvidenceCoverage: 0.95, HarmfulErrorRate: 0.006, P95LatencyMS: 2050, CostPerAccepted: 0.027, DeletionConforms: true, ReplayConforms: true},
	}

	passed := make([]Result, 0, len(results))
	for _, result := range results {
		if eligible(result) {
			passed = append(passed, result)
		}
	}
	if len(passed) == 0 {
		fmt.Println("no architecture passed")
		return
	}

	sort.Slice(passed, func(i, j int) bool {
		return passed[i].CostPerAccepted < passed[j].CostPerAccepted
	})
	fmt.Printf("selected=%s cost_per_accepted=%.3f\n",
		passed[0].Name,
		passed[0].CostPerAccepted,
	)
}

Expected output:

text 
selected=modular cost_per_accepted=0.016

The fixture demonstrates the method, not a market conclusion. Change the workload or policy and another architecture may win.

Migration Without a Rewrite

  1. Put the current system behind an architecture-neutral request and result contract.
  2. Capture representative, permission-safe fixtures and current production traces.
  3. Add the new candidate in shadow mode without changing downstream effects.
  4. Compare output, evidence, failure, latency, and cost by slice.
  5. Route only the slices where the candidate passes.
  6. Keep old artifacts readable until replay, deletion, and rollback are proven.
  7. Remove a component only after its unique evidence and recovery role has been replaced.

Frequently Asked Questions

Is a unified multimodal API an end-to-end model?

Not necessarily. The API boundary may hide encoders, speech codecs, retrieval, tools, routing, or multiple models. Use “unified API” for the product surface and reserve architectural claims for published evidence.

Are pipelines more auditable?

They can expose more intermediate artifacts, but auditability requires versioned lineage, access controls, retention, and interpretable checks. A directory full of unversioned OCR files is not an audit trail.

Are pipelines deterministic?

Only some stages may be. PDF parsing and checksums can be deterministic for pinned inputs and versions; OCR, ASR, detection, ranking, and LLM generation may not be. Describe determinism per stage.

When should a hybrid be rejected?

Reject it when routing does not outperform a simple baseline, when the two paths cannot share one evidence and lifecycle contract, or when extra operational cost exceeds the measured quality gain.

How do model benchmarks affect the decision?

They identify candidates and failure slices. They do not replace evaluation on your inputs, prompts, schema, hardware, permissions, and business loss. Preserve benchmark conditions whenever citing a result.

Primary Sources