Eino Production Deployment and Observability in Practice

TL;DR

An Eino application is production-ready only when model calls, tools, state, and rollout behavior have explicit operational contracts. Pin the Eino version, propagate one deadline, bound concurrency at each dependency, retry only safe operations, instrument Eino callbacks, and evaluate answer quality outside the request path. This guide uses Eino v0.9.21 and standard Go service patterns; it does not rely on undocumented debugging or evaluation packages.

Table of Contents

1. Define the Production Contract
2. Choose Synchronous or Durable Execution
3. Bound Time, Concurrency, and Retries
4. Instrument Eino Callbacks
5. Runnable Callback Example
6. Design Traces and Metrics
7. Evaluate Quality Before Release
8. Deploy and Roll Back Safely
9. Incident Checklist
10. Frequently Asked Questions
11. Primary Sources

Define the Production Contract

A production contract states what the service promises before choosing Kubernetes settings or tracing vendors.

Define these invariants:

Eino supplies composition, agents, callbacks, streaming, and checkpoint hooks. It does not decide these business contracts. Treat the Eino component model as an execution layer inside the service boundary.

Choose Synchronous or Durable Execution

Execution mode follows task semantics, not a blanket "agents should be asynchronous" rule.

Synchronous request

Use a synchronous HTTP or RPC request when:

• the result fits the caller's latency budget
• cancellation should stop useful work immediately
• no long-lived external side effect needs recovery
• the response can be regenerated safely

The handler owns the root context.Context. Every model, retriever, and tool call must derive from it so a disconnected client does not leave work running.

Durable job

Use a durable queue and state machine when:

• execution may exceed gateway timeouts
• a process restart must not lose progress
• tools create externally visible side effects
• a human approval or delayed callback can suspend work

Persist a job ID, input version, prompt and tool configuration versions, current step, idempotency keys, and terminal result. A queue provides delivery, not exactly-once business effects; each side-effecting operation still needs its own deduplication contract.

Bound Time, Concurrency, and Retries

Capacity control must exist at the scarce dependency, not only at the HTTP server.

Use separate limits for:

• active end-user runs
• model requests per provider and model
• calls to each downstream tool
• embedding and reranking batches
• background evaluation jobs

A global goroutine limit can still overload one provider while leaving others idle. Per-dependency semaphores make saturation visible and preserve isolation.

Deadline hierarchy

The child deadline must never exceed the remaining parent deadline:

request deadline
  -> orchestration deadline
     -> model attempt deadline
     -> tool attempt deadline

Reserve time for output validation and serialization. A model call that consumes the entire request deadline leaves no budget to cancel tools or return a controlled error.

Retry rules

Retry only when all three conditions hold:

1. the error is transient and classified
2. the operation is idempotent or protected by an idempotency key
3. enough deadline remains for backoff and another attempt

Never retry authentication failures, validation failures, policy denials, or arbitrary side effects. Record attempt count separately from logical operation count.

Instrument Eino Callbacks

Eino v0.9.21 exposes five callback timings around component execution:

• OnStart
• OnEnd
• OnError
• OnStartWithStreamInput
• OnEndWithStreamOutput

Attach handlers at the narrowest useful scope:

Global registration is not thread-safe and should run once during startup. Callback input and output pointers are shared; mutating them can cause data races. Streaming callback copies must be closed after consumption.

The callback layer is an instrumentation hook, not an observability backend. Adapt it to OpenTelemetry, metrics, and structured logs according to your data policy.

Runnable Callback Example

This program builds a two-node Eino chain and measures every node through an invocation-scoped callback. It compiles against Eino v0.9.21 and uses no external API.

package main

import (
	"context"
	"fmt"
	"log"
	"strings"
	"time"

	"github.com/cloudwego/eino/callbacks"
	"github.com/cloudwego/eino/compose"
)

type startedAtKey struct{}

func main() {
	ctx := context.Background()
	chain := compose.NewChain[string, string]().
		AppendLambda(
			compose.InvokableLambda(func(
				ctx context.Context,
				input string,
			) (string, error) {
				if err := ctx.Err(); err != nil {
					return "", err
				}
				return strings.TrimSpace(input), nil
			}),
			compose.WithNodeName("normalize"),
		).
		AppendLambda(
			compose.InvokableLambda(func(
				ctx context.Context,
				input string,
			) (string, error) {
				if err := ctx.Err(); err != nil {
					return "", err
				}
				return "answer: " + input, nil
			}),
			compose.WithNodeName("answer"),
		)

	runnable, err := chain.Compile(
		ctx,
		compose.WithGraphName("support_request"),
		compose.WithMaxRunSteps(8),
	)
	if err != nil {
		log.Fatal(err)
	}

	handler := callbacks.NewHandlerBuilder().
		OnStartFn(func(
			ctx context.Context,
			info *callbacks.RunInfo,
			_ callbacks.CallbackInput,
		) context.Context {
			return context.WithValue(ctx, startedAtKey{}, time.Now())
		}).
		OnEndFn(func(
			ctx context.Context,
			info *callbacks.RunInfo,
			_ callbacks.CallbackOutput,
		) context.Context {
			startedAt, _ := ctx.Value(startedAtKey{}).(time.Time)
			if info != nil {
				log.Printf(
					"component=%s name=%s duration=%s",
					info.Component,
					info.Name,
					time.Since(startedAt),
				)
			}
			return ctx
		}).
		OnErrorFn(func(
			ctx context.Context,
			info *callbacks.RunInfo,
			runErr error,
		) context.Context {
			if info != nil {
				log.Printf(
					"component=%s name=%s error=%q",
					info.Component,
					info.Name,
					runErr,
				)
			}
			return ctx
		}).
		Build()

	runCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()

	output, err := runnable.Invoke(
		runCtx,
		"  diagnose checkout timeout  ",
		compose.WithCallbacks(handler),
	)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(output)
	// Output: answer: diagnose checkout timeout
}

Use this module pin:

require github.com/cloudwego/eino v0.9.21

In an OpenTelemetry adapter, create a span on callback start, store it in the returned context, record bounded attributes on end or error, and then end it. Use low-cardinality span names such as eino.chat_model or eino.tool; put model and tool identity in attributes rather than embedding user IDs or request text in span names.

Design Traces and Metrics

OpenTelemetry defines a trace as nested spans connected through context. A useful Eino request trace is:

agent.request
  eino.agent
    eino.chat_model
    eino.tool
      downstream.rpc
    eino.chat_model
  output.validate

Record enough data to diagnose behavior without turning telemetry into a prompt archive.

Recommended attributes

• application, environment, release, and prompt version
• Eino component, node, agent, model, and tool name
• attempt number and result class
• input and output byte buckets
• prompt, completion, and total token counts when the provider reports them
• queue time, model latency, tool latency, and time to first token
• policy decision, timeout, cancellation, and fallback reason

Avoid by default

• raw prompts and model output
• credentials, authorization headers, cookies, or signed URLs
• unrestricted tool arguments and results
• personal data or tenant identifiers with broad cardinality

Hashing a sensitive value does not automatically make it safe. Define retention, access, deletion, sampling, and regional storage rules before collection.

Service-level indicators

Separate system health from answer quality:

An HTTP 200 with an unsupported answer is operational success but product failure. Both layers need independent gates.

Evaluate Quality Before Release

Evaluation should run outside the online path against versioned datasets.

Maintain at least four suites:

1. Golden tasks with deterministic expected fields or outcomes
2. Adversarial tasks covering prompt injection, cross-tenant access, and destructive actions
3. Regression tasks captured from real failures after sanitization
4. Operational tasks that simulate timeout, rate limit, partial stream, and tool failure

For free-form answers, define a rubric and calibrate any LLM judge against human labels. Store model revision, prompt version, tool catalog, data snapshot, temperature, retry policy, and judge version with every result.

A release gate should compare the candidate against the current production baseline. Do not accept a quality gain that violates latency, cost, or safety budgets, and do not claim a framework performance advantage from unmatched workloads.

Deploy and Roll Back Safely

An agent release includes more than a binary:

• application commit
• Eino and provider SDK versions
• model identifier and revision
• system prompts and Skill content
• tool schemas and policies
• retrieval corpus and index version
• evaluator and threshold versions

Use a canary or version-routed rollout. Keep in-flight durable jobs bound to the version that created them unless the state format has an explicit migration. Rollback must restore the whole execution bundle, not just the Go binary.

Before increasing traffic, verify:

• startup, readiness, and dependency checks
• cancellation and graceful shutdown
• provider and tool rate-limit behavior
• trace propagation across HTTP, RPC, queue, and MCP boundaries
• output validation and policy-denial handling
• old and new version coexistence

Incident Checklist

When latency or failures rise:

1. Check request volume, queue time, and admission-control rejection.
2. Split latency by model, retriever, and tool rather than using one aggregate.
3. Compare attempt count with logical calls to detect retry amplification.
4. Inspect deadline propagation and abandoned stream readers.
5. Check provider quotas, connection pools, and downstream saturation.
6. Confirm prompt, model, tool schema, and index versions for affected traces.
7. Sample sanitized failed paths and compare them with the release baseline.
8. Roll back the complete execution bundle when the regression is version-linked.

Frequently Asked Questions

Does Eino provide a production observability backend?

Eino provides callback hooks and execution metadata. You still choose and operate the trace, metric, and log backends, define semantic attributes, and enforce privacy controls.

Should callbacks be global or request-scoped?

Use one startup-time global handler for mandatory baseline telemetry. Add request-scoped handlers for tenant-aware sampling or temporary diagnostics. Avoid mutating global handlers after execution begins.

How should streaming failures be measured?

Track errors returned while consuming the stream, not only errors returned when the stream is created. Eino's callback documentation notes that mid-stream errors are not reported through OnError; the consumer must record them and close the reader.

What should a readiness check test?

Readiness should prove the process can accept work without running a costly model request. Check configuration, policy and key availability, required stores, and local queues; monitor provider reachability separately to avoid cascading health-check load.

Summary

Production Eino engineering is contract engineering. Bound time, concurrency, retries, state, authorization, and telemetry before increasing traffic. Use the callback APIs that v0.9.21 actually exposes, connect them to standard observability infrastructure, and keep quality evaluation separate from request success. A safe release can be measured, reproduced, and rolled back as one versioned bundle.

Related Resources

• Eino Streaming and Callback System
• Eino Multi-Agent Coordination
• Eino RAG Pipeline in Production
• OpenTelemetry Glossary

Primary Sources

• Eino v0.9.21 callback package
• Eino v0.9.21 ADK API
• CloudWeGo Callback and Trace guide
• OpenTelemetry Trace API specification
• OpenTelemetry Generative AI semantic conventions