Documentation
¶
Overview ¶
Package model provides interfaces and adapters for integrating Large Language Models (LLMs) into agent workflows.
Overview ¶
The model package defines a common interface for LLM providers, enabling you to:
- Use multiple LLM providers (OpenAI, Anthropic, LangChainGo adapters)
- Support tool/function calling
- Stream responses in real-time
- Handle structured outputs
Supported Providers ¶
- OpenAI: GPT-4, GPT-3.5-turbo via OpenAI API
- Anthropic: Claude models via AWS Bedrock
- LangChainGo: Adapter for LangChain-compatible models
Quick Start ¶
Using OpenAI:
import (
"github.com/hupe1980/agentmesh/pkg/model"
"github.com/hupe1980/agentmesh/pkg/model/openai"
)
llm := openai.NewModel(
openai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
openai.WithModelName("gpt-4"),
)
response, _ := llm.Generate(ctx, []message.Message{
message.NewSystemMessageFromText("You are a helpful assistant"),
message.NewHumanMessageFromText("Hello!"),
})
Using Anthropic (AWS Bedrock):
import "github.com/hupe1980/agentmesh/pkg/model/anthropic"
llm := anthropic.NewModel(
anthropic.WithModelID("anthropic.claude-v2"),
anthropic.WithRegion("us-east-1"),
)
Tool Calling ¶
Models that implement ToolAware can invoke tools:
import "github.com/hupe1980/agentmesh/pkg/tool"
weatherTool, _ := tool.NewFuncTool("get_weather", "Get weather", weatherFunc)
// Check if model supports tools
if toolAware, ok := llm.(model.ToolAware); ok {
response, _ := toolAware.GenerateWithTools(ctx, messages, []tool.Interface{weatherTool})
}
Streaming ¶
Stream responses token-by-token:
seq := llm.Generate(ctx, &model.Request{
Messages: messages,
Stream: true,
})
for res, err := range seq {
if err != nil {
log.Fatalf("Error: %v", err)
}
if res.Partial {
fmt.Print(res.Message.String())
} else {
fmt.Println(res.Message.String())
}
}
Model Interface ¶
All models implement the core Interface:
type Interface interface {
Generate(ctx context.Context, messages []message.Message, opts ...Option) (message.Message, error)
Stream(ctx context.Context, messages []message.Message, opts ...Option) <-chan Event
}
Tool-Aware Models ¶
Models with function calling implement ToolAware:
type ToolAware interface {
Interface
GenerateWithTools(ctx context.Context, messages []message.Message, tools []tool.Interface, opts ...Option) (message.Message, error)
}
Configuration Options ¶
Customize model behavior with options:
llm.Generate(ctx, messages, model.WithTemperature(0.7), model.WithMaxTokens(500), model.WithTopP(0.9), )
Custom Model Adapters ¶
Implement the Interface to add new providers:
type CustomModel struct {
client *CustomClient
}
func (m *CustomModel) Generate(ctx context.Context, messages []message.Message, opts ...Option) (message.Message, error) {
// Convert messages to provider format
// Call provider API
// Convert response to message.Message
return response, nil
}
func (m *CustomModel) Stream(ctx context.Context, messages []message.Message, opts ...Option) <-chan Event {
ch := make(chan Event)
go func() {
defer close(ch)
// Stream from provider
}()
return ch
}
Package model provides sentinel errors for the model package. Note: Most model provider packages (openai, anthropic, etc.) define their own package-specific errors for better error identification.
Package model provides model execution with lifecycle management.
The Executor pattern separates model execution from graph orchestration:
- Executor: Handles execution lifecycle (observability, error handling)
- Model: Core generation logic (API calls, streaming)
- ModelNode: Graph orchestration (state extraction, routing)
This separation enables:
- Reusable execution logic across different contexts (graphs, chains, direct calls)
- Custom executor implementations (retry, caching, rate limiting)
- Clean testing boundaries (test execution independent of graph/state)
- Centralized observability handling
Architecture:
┌─────────────┐
│ ModelNode │ Graph layer: state extraction, routing
└──────┬──────┘
│ delegates to
┌──────▼──────┐
│ Executor │ Execution layer: lifecycle, observability
└──────┬──────┘
│ calls
┌──────▼──────┐
│ Model │ Core layer: API calls, streaming
└─────────────┘
Example (basic usage):
executor := model.NewExecutor(openaiModel) resp, err := model.Last(executor.Generate(ctx, req))
Example (streaming):
for resp, err := range executor.Generate(ctx, req) {
if err != nil { return err }
fmt.Print(resp.Message.Content())
}
Example (custom executor):
type RetryExecutor struct {
wrapped model.Executor
maxRetries int
}
func (e *RetryExecutor) Generate(ctx context.Context, req *Request) iter.Seq2[*Response, error] {
// Implement retry logic wrapping e.wrapped
}
Index ¶
- Variables
- func CapabilityScore(has, needs Capabilities) int
- func LastStructured[T any](seq iter.Seq2[*Response, error]) (*T, error)
- type BestMatchRouter
- type Capabilities
- type CapabilityDetector
- type CapabilityRouter
- type CapabilityRouterOption
- type CircuitBreaker
- type CircuitState
- type ComplexityEstimator
- type CompositeRouter
- type CompositeRouterOption
- type ConditionalRouter
- type CostBasedRouter
- type CostRouterOption
- type DefaultCapabilityDetector
- type DefaultExecutor
- type Executor
- type ExecutorOption
- type ExecutorWrapper
- type FallbackRoutedModel
- type FallbackRouter
- func (r *FallbackRouter) AvailableModels() []Model
- func (r *FallbackRouter) CircuitState(m Model) CircuitState
- func (r *FallbackRouter) RecordFailure(m Model)
- func (r *FallbackRouter) RecordSuccess(m Model)
- func (r *FallbackRouter) ResetAllCircuits()
- func (r *FallbackRouter) ResetCircuit(m Model)
- func (r *FallbackRouter) Route(ctx context.Context, req *Request) (Model, error)
- type FallbackRouterOption
- type HeuristicEstimator
- type Logprobs
- type MLComplexityEstimator
- type Middleware
- type MiddlewareFunc
- type Model
- type PriorityRouter
- type PriorityRouterOption
- type RandomRouter
- type RandomRouterOption
- type Request
- type Response
- type RoutedModel
- type RoutedModelOption
- type Router
- type RouterFunc
- type StaticRouter
- type TokenLogprob
- type TopLogprob
- type UsageInfo
- type WeightedRouter
- type WeightedRouterOption
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNoModelAvailable is returned when no model can handle the request. ErrNoModelAvailable = errors.New("model: no model available") // ErrRoutingFailed is returned when routing fails. ErrRoutingFailed = errors.New("model: routing failed") )
Router errors
var ( // ErrNoResponse is returned when a model generates no responses. ErrNoResponse = errors.New("model: no response generated") )
Functions ¶
func CapabilityScore ¶
func CapabilityScore(has, needs Capabilities) int
CapabilityScore calculates a score for how well a model matches requirements. Higher score means better match. Returns 0 if requirements are not met.
func LastStructured ¶
LastStructured extracts the last response from an iterator and unmarshals the response message content into the specified type T. This is useful for extracting structured output from model generation calls.
Example:
type MovieReview struct {
Title string `json:"title"`
Rating int `json:"rating"`
Summary string `json:"summary"`
}
outputSchema, _ := schema.NewOutputSchema("review", MovieReview{})
req := &model.Request{
Messages: []message.Message{...},
OutputSchema: &outputSchema,
}
review, err := model.LastStructured[MovieReview](mdl.Generate(ctx, req))
if err != nil {
return err
}
fmt.Printf("Rating: %d/5\n", review.Rating)
Types ¶
type BestMatchRouter ¶
type BestMatchRouter struct {
// contains filtered or unexported fields
}
BestMatchRouter routes to the model with the highest capability score.
func NewBestMatchRouter ¶
func NewBestMatchRouter(models []Model, opts ...CapabilityRouterOption) *BestMatchRouter
NewBestMatchRouter creates a router that selects the best matching model.
type Capabilities ¶
type Capabilities struct {
// Streaming indicates if the model supports streaming responses.
// When true, Generate() will yield incremental chunks with Partial=true.
Streaming bool
// Tools indicates if the model supports tool/function calling.
// When true, you can include tools in Request.Tools.
Tools bool
// StructuredOutput indicates if the model supports JSON schema-constrained output.
// When true, you can provide a schema in Request.Metadata["schema"].
StructuredOutput bool
// NativeReasoning indicates if the model exposes its internal reasoning process.
// When true, Response.Reasoning will be populated with the model's thinking.
// Examples: OpenAI o1/o3, Gemini 2.0 thinking mode, Claude extended thinking.
NativeReasoning bool
// Logprobs indicates if the model can provide token-level log probabilities.
// When true, Response.Logprobs may be populated (if explicitly requested).
Logprobs bool
// Vision indicates if the model can process image inputs.
// When true, you can include message.ImagePart in your messages.
Vision bool
// Audio indicates if the model can process audio inputs.
Audio bool
// MaxContextTokens is the maximum context window size in tokens.
// This is the total budget for input + output tokens.
// Zero means unknown or unlimited.
MaxContextTokens int
// MaxOutputTokens is the maximum number of tokens the model can generate.
// This may be constrained by configuration or model architecture.
// Zero means unknown or uses default limits.
MaxOutputTokens int
// SupportedModalities lists the input types the model accepts.
// Common values: "text", "image", "audio", "video"
SupportedModalities []string
}
Capabilities describes the features and limitations of a model. This allows callers to discover what a model supports before using it.
type CapabilityDetector ¶
type CapabilityDetector interface {
// Detect returns the capabilities required to handle the request.
Detect(ctx context.Context, req *Request) (Capabilities, error)
}
CapabilityDetector detects required capabilities from a request.
type CapabilityRouter ¶
type CapabilityRouter struct {
// contains filtered or unexported fields
}
CapabilityRouter routes requests to models based on required capabilities. It inspects the request to determine what capabilities are needed and selects a model that supports all of them.
func NewCapabilityRouter ¶
func NewCapabilityRouter(models []Model, opts ...CapabilityRouterOption) *CapabilityRouter
NewCapabilityRouter creates a new capability-based router. Models are checked in order; the first one satisfying all requirements is selected.
type CapabilityRouterOption ¶
type CapabilityRouterOption func(*CapabilityRouter)
CapabilityRouterOption configures a CapabilityRouter.
func WithCapabilityDetector ¶
func WithCapabilityDetector(d CapabilityDetector) CapabilityRouterOption
WithCapabilityDetector sets a custom capability detector.
func WithCapabilityFallback ¶
func WithCapabilityFallback(m Model) CapabilityRouterOption
WithCapabilityFallback sets a fallback model when no model matches.
type CircuitBreaker ¶
type CircuitBreaker struct {
// contains filtered or unexported fields
}
CircuitBreaker tracks failures for a model and opens the circuit when threshold is exceeded.
func NewCircuitBreaker ¶
func NewCircuitBreaker(threshold int, resetTimeout time.Duration) *CircuitBreaker
NewCircuitBreaker creates a new circuit breaker.
func (*CircuitBreaker) IsOpen ¶
func (cb *CircuitBreaker) IsOpen() bool
IsOpen returns true if the circuit is open and requests should be blocked.
func (*CircuitBreaker) RecordFailure ¶
func (cb *CircuitBreaker) RecordFailure()
RecordFailure records a failed request.
func (*CircuitBreaker) RecordSuccess ¶
func (cb *CircuitBreaker) RecordSuccess()
RecordSuccess records a successful request.
func (*CircuitBreaker) Reset ¶
func (cb *CircuitBreaker) Reset()
Reset resets the circuit breaker to closed state.
func (*CircuitBreaker) State ¶
func (cb *CircuitBreaker) State() CircuitState
State returns the current circuit state.
type CircuitState ¶
type CircuitState int
CircuitState represents the state of a circuit breaker.
const ( // CircuitClosed means the circuit is healthy and requests flow normally. CircuitClosed CircuitState = iota // CircuitOpen means the circuit has tripped and requests are blocked. CircuitOpen // CircuitHalfOpen means the circuit is testing if the model has recovered. CircuitHalfOpen )
type ComplexityEstimator ¶
type ComplexityEstimator interface {
// Estimate returns a complexity score between 0.0 (simple) and 1.0 (complex).
Estimate(ctx context.Context, req *Request) (float64, error)
}
ComplexityEstimator estimates the complexity of a request on a scale of 0.0 to 1.0.
type CompositeRouter ¶
type CompositeRouter struct {
// contains filtered or unexported fields
}
CompositeRouter chains multiple routers together. Each router is tried in order; the first to return a model wins. This implements the Chain of Responsibility pattern.
func NewCompositeRouter ¶
func NewCompositeRouter(routers []Router, opts ...CompositeRouterOption) *CompositeRouter
NewCompositeRouter creates a new composite router. Routers are tried in order until one returns a valid model.
Example:
router := NewCompositeRouter(
NewCapabilityRouter(models), // First check capabilities
NewCostBasedRouter(cheap, expensive), // Then optimize for cost
NewFallbackRouter(allModels), // Finally, fallback chain
)
func (*CompositeRouter) Add ¶
func (r *CompositeRouter) Add(router Router)
Add appends a router to the chain.
func (*CompositeRouter) Routers ¶
func (r *CompositeRouter) Routers() []Router
Routers returns the list of routers in the chain.
type CompositeRouterOption ¶
type CompositeRouterOption func(*CompositeRouter)
CompositeRouterOption configures a CompositeRouter.
func WithCompositeFallback ¶
func WithCompositeFallback(m Model) CompositeRouterOption
WithCompositeFallback sets a fallback model when all routers fail.
type ConditionalRouter ¶
type ConditionalRouter struct {
// contains filtered or unexported fields
}
ConditionalRouter routes based on a condition function. If the condition is true, it uses the primary router; otherwise, the alternative.
func NewConditionalRouter ¶
func NewConditionalRouter( condition func(ctx context.Context, req *Request) bool, primary Router, alternative Router, ) *ConditionalRouter
NewConditionalRouter creates a router that chooses based on a condition.
type CostBasedRouter ¶
type CostBasedRouter struct {
// contains filtered or unexported fields
}
CostBasedRouter routes requests to models based on estimated complexity. Simple requests go to cheaper/faster models, complex ones to more capable models.
func NewCostBasedRouter ¶
func NewCostBasedRouter(cheap, expensive Model, opts ...CostRouterOption) *CostBasedRouter
NewCostBasedRouter creates a new cost-based router. The cheap model handles simple requests, expensive model handles complex ones.
type CostRouterOption ¶
type CostRouterOption func(*CostBasedRouter)
CostRouterOption configures a CostBasedRouter.
func WithComplexityEstimator ¶
func WithComplexityEstimator(e ComplexityEstimator) CostRouterOption
WithComplexityEstimator sets a custom complexity estimator.
func WithComplexityThreshold ¶
func WithComplexityThreshold(threshold float64) CostRouterOption
WithComplexityThreshold sets the threshold above which expensive model is used. Default is 0.3 (30% complexity).
type DefaultCapabilityDetector ¶
type DefaultCapabilityDetector struct {
// ReasoningKeywords are words that indicate reasoning capability is needed.
ReasoningKeywords []string
// TokensPerWord is used to estimate token count (default: 1.3)
TokensPerWord float64
}
DefaultCapabilityDetector detects capabilities from request content.
func (*DefaultCapabilityDetector) Detect ¶
func (d *DefaultCapabilityDetector) Detect(ctx context.Context, req *Request) (Capabilities, error)
Detect analyzes the request to determine required capabilities.
type DefaultExecutor ¶
type DefaultExecutor struct {
// contains filtered or unexported fields
}
DefaultExecutor is the standard implementation of Executor. It provides full lifecycle management with observability.
type Executor ¶
type Executor interface {
// Generate executes a model generation with full lifecycle management.
// It handles observability and error recovery automatically.
// Returns an iterator that yields responses. For non-streaming requests,
// a single response is yielded. For streaming requests, incremental
// responses are yielded as they become available.
//
// The iterator pattern (iter.Seq2) provides a unified interface for both
// streaming and non-streaming execution, allowing consumers to use the
// same code path regardless of Request.Stream setting.
//
// Example (non-streaming):
// resp, err := model.Last(executor.Generate(ctx, req))
//
// Example (streaming):
// for resp, err := range executor.Generate(ctx, req) {
// if err != nil { return err }
// // Process incremental response
// }
Generate(ctx context.Context, req *Request) iter.Seq2[*Response, error]
}
Executor handles the complete lifecycle of model generation requests. It wraps a Model with observability and error handling.
This interface allows users to provide custom executor implementations for specialized behavior while maintaining consistent lifecycle management.
Example custom implementations:
- RetryExecutor: Adds retry logic with exponential backoff
- CachedExecutor: Caches responses for deterministic requests
- RateLimitedExecutor: Enforces rate limiting
- CircuitBreakerExecutor: Implements circuit breaker pattern
func Chain ¶
func Chain(executor Executor, middleware ...Middleware) Executor
Chain applies multiple middleware to an executor in order.
func NewExecutor ¶
func NewExecutor(mdl Model, opts ...ExecutorOption) Executor
NewExecutor creates a new default model executor. Returns an Executor interface for maximum flexibility.
The executor wraps the model with:
- Observability (tracing, metrics, logging)
- Error handling and recovery
- Middleware support via Chain()
Example:
executor := model.NewExecutor(myModel,
model.WithExecutorName("react-model"))
resp, err := model.Last(executor.Generate(ctx, req))
type ExecutorOption ¶
type ExecutorOption func(*DefaultExecutor)
ExecutorOption configures a DefaultExecutor.
func WithExecutorName ¶
func WithExecutorName(name string) ExecutorOption
WithExecutorName sets the executor name for observability. The name is used in traces, logs, and metrics to identify this executor.
Example:
executor := model.NewExecutor(myModel,
model.WithExecutorName("my-agent-model"))
type ExecutorWrapper ¶
type ExecutorWrapper struct {
// contains filtered or unexported fields
}
ExecutorWrapper wraps a function as an Executor.
type FallbackRoutedModel ¶
type FallbackRoutedModel struct {
*RoutedModel
// contains filtered or unexported fields
}
FallbackRoutedModel extends RoutedModel to automatically record successes/failures. It wraps FallbackRouter and updates circuit breakers based on generation results.
func NewFallbackRoutedModel ¶
func NewFallbackRoutedModel(router *FallbackRouter, opts ...RoutedModelOption) *FallbackRoutedModel
NewFallbackRoutedModel creates a RoutedModel that automatically manages circuit breakers.
func (*FallbackRoutedModel) RecordFailure ¶
func (m *FallbackRoutedModel) RecordFailure(model Model)
RecordFailure forwards to the fallback router.
func (*FallbackRoutedModel) RecordSuccess ¶
func (m *FallbackRoutedModel) RecordSuccess(model Model)
RecordSuccess forwards to the fallback router.
type FallbackRouter ¶
type FallbackRouter struct {
// contains filtered or unexported fields
}
FallbackRouter tries models in order until one succeeds. It implements a circuit breaker pattern to avoid repeatedly trying failed models.
func NewFallbackRouter ¶
func NewFallbackRouter(models []Model, opts ...FallbackRouterOption) *FallbackRouter
NewFallbackRouter creates a new fallback router with circuit breakers. Models are tried in order; the first available (non-tripped) model is returned.
func (*FallbackRouter) AvailableModels ¶
func (r *FallbackRouter) AvailableModels() []Model
AvailableModels returns models with non-open circuits.
func (*FallbackRouter) CircuitState ¶
func (r *FallbackRouter) CircuitState(m Model) CircuitState
CircuitState returns the circuit breaker state for the given model.
func (*FallbackRouter) RecordFailure ¶
func (r *FallbackRouter) RecordFailure(m Model)
RecordFailure records a failed request for the given model.
func (*FallbackRouter) RecordSuccess ¶
func (r *FallbackRouter) RecordSuccess(m Model)
RecordSuccess records a successful request for the given model.
func (*FallbackRouter) ResetAllCircuits ¶
func (r *FallbackRouter) ResetAllCircuits()
ResetAllCircuits resets all circuit breakers.
func (*FallbackRouter) ResetCircuit ¶
func (r *FallbackRouter) ResetCircuit(m Model)
ResetCircuit resets the circuit breaker for the given model.
type FallbackRouterOption ¶
type FallbackRouterOption func(*FallbackRouter)
FallbackRouterOption configures a FallbackRouter.
func WithFailureThreshold ¶
func WithFailureThreshold(threshold int) FallbackRouterOption
WithFailureThreshold sets the number of failures before tripping the circuit.
func WithResetTimeout ¶
func WithResetTimeout(timeout time.Duration) FallbackRouterOption
WithResetTimeout sets the duration before a tripped circuit resets.
type HeuristicEstimator ¶
type HeuristicEstimator struct {
// ComplexKeywords are words that indicate higher complexity.
// Default set includes common analytical and technical terms.
ComplexKeywords []string
// WordCountWeight is the weight given to word count in complexity.
// Higher values mean longer messages increase complexity more.
WordCountWeight float64
// ContextWeight is the weight given to conversation length.
ContextWeight float64
// ToolWeight is the additional complexity when tools are involved.
ToolWeight float64
// SchemaWeight is the additional complexity for structured output.
SchemaWeight float64
}
HeuristicEstimator estimates complexity using simple heuristics. This provides a fast, no-API-call estimation based on message characteristics.
type Logprobs ¶
type Logprobs struct {
// Content contains log probabilities for each generated token
Content []TokenLogprob
}
Logprobs contains log probability information for generated tokens. This data is useful for understanding model confidence and decision-making.
type MLComplexityEstimator ¶
type MLComplexityEstimator struct {
// contains filtered or unexported fields
}
MLComplexityEstimator uses a lightweight model to estimate complexity. This provides more accurate estimation at the cost of an API call.
func NewMLComplexityEstimator ¶
func NewMLComplexityEstimator(classifier Model) *MLComplexityEstimator
NewMLComplexityEstimator creates a new ML-based complexity estimator. The classifier should be a fast, cheap model (e.g., GPT-4o-mini).
type Middleware ¶
Middleware intercepts and extends model execution.
type MiddlewareFunc ¶
MiddlewareFunc is a function adapter for Middleware.
func (MiddlewareFunc) Wrap ¶
func (f MiddlewareFunc) Wrap(next Executor) Executor
Wrap implements the Middleware interface.
type Model ¶
type Model interface {
// Generate performs a request that yields Response chunks as an iterator.
// For streaming, iterate over all chunks to get incremental updates.
// For blocking, consume only the final response using Last() or similar helpers.
//
// ERROR HANDLING:
// - Fatal errors (API failures, context canceled, invalid input) are returned
// in the second return value (err) as per Go iterator convention
// - When err != nil, iteration stops and no more responses are yielded
// - Response content is always valid when err == nil
//
// Streaming usage:
// req := &model.Request{
// Messages: messages,
// Tools: myTools,
// }
// for resp, err := range model.Generate(ctx, req) {
// if err != nil {
// return fmt.Errorf("generation failed: %w", err)
// }
// fmt.Print(resp.Message.String()) // Process each chunk
// }
//
// Blocking usage (helper required):
// req := &model.Request{Messages: messages}
// resp, err := Last(model.Generate(ctx, req))
// if err != nil {
// return fmt.Errorf("generation failed: %w", err)
// }
// fmt.Println(resp.Message.String()) // Process final message
// fmt.Println("Reasoning:", resp.Reasoning) // Access native reasoning
// fmt.Println("Finish reason:", resp.FinishReason) // Why generation stopped
// fmt.Printf("Tokens used: %d\n", resp.Usage.TotalTokens)
// if resp.Logprobs != nil {
// // Access token probabilities for confidence analysis
// }
//
// The iterator will yield:
// - Multiple responses with incremental content (streaming mode)
// - A single final response (blocking mode)
//
// Context cancellation is respected and will stop iteration.
Generate(ctx context.Context, req *Request) iter.Seq2[*Response, error]
// Capabilities returns the features and limitations of this model.
// Use this to discover what the model supports before attempting to use
// optional features like tools, structured output, or native reasoning.
//
// Example:
// caps := model.Capabilities()
// if caps.Tools {
// req.Tools = myTools
// }
// if caps.NativeReasoning {
// // Model will populate Response.Reasoning automatically
// }
Capabilities() Capabilities
}
Model defines the contract for language model backends using Go 1.23+ iterators. The unified Generate method supports both streaming and blocking consumption patterns.
type PriorityRouter ¶
type PriorityRouter struct {
// contains filtered or unexported fields
}
PriorityRouter routes to models based on priority order with health checking. Unlike FallbackRouter, it doesn't use circuit breakers but allows explicit health checks.
func NewPriorityRouter ¶
func NewPriorityRouter(models []Model, opts ...PriorityRouterOption) *PriorityRouter
NewPriorityRouter creates a new priority-based router. Models are tried in order; the first healthy model is selected.
type PriorityRouterOption ¶
type PriorityRouterOption func(*PriorityRouter)
PriorityRouterOption configures a PriorityRouter.
func WithHealthCheck ¶
func WithHealthCheck(fn func(ctx context.Context, m Model) bool) PriorityRouterOption
WithHealthCheck sets a function to check if a model is healthy.
type RandomRouter ¶
type RandomRouter struct {
// contains filtered or unexported fields
}
RandomRouter randomly selects from a pool of models. Useful for load balancing or A/B testing.
func NewRandomRouter ¶
func NewRandomRouter(models []Model, opts ...RandomRouterOption) *RandomRouter
NewRandomRouter creates a router that randomly selects a model.
type RandomRouterOption ¶
type RandomRouterOption func(*RandomRouter)
RandomRouterOption configures a RandomRouter.
func WithRandomFunc ¶
func WithRandomFunc(fn func(n int) int) RandomRouterOption
WithRandomFunc sets a custom random function.
type Request ¶
type Request struct {
// Messages is the conversation history to send to the model
Messages []message.Message
// Tools is an optional list of tools the model can call during generation.
// Only relevant if Capabilities().Tools is true.
Tools []tool.Tool
// Instructions is an optional instruction text sent with this request.
// When provided, it will be prepended to the messages (or sent via provider-specific
// system parameter). This is sent per-request and not stored in conversation history,
// making it more token-efficient than adding a system message to the state.
//
// Instructions supports dynamic resolution from graph state via template placeholders,
// but by the time it reaches the model, it's already resolved to a plain string.
//
// If empty, only the messages from the Messages field are used.
Instructions string
// OutputSchema is an optional structured output schema with metadata.
// When provided (and Capabilities().StructuredOutput is true), the model will
// be constrained to generate valid JSON matching this schema.
//
// This field provides additional metadata compared to Schema:
// - Name: Schema identifier for logging/debugging
// - Strict: Whether to enforce strict schema validation (provider-specific)
// - Description: Human-readable schema description
// - Schema: The actual JSON Schema definition
//
// If both Schema and OutputSchema are provided, OutputSchema takes precedence.
//
// Example:
// type Result struct {
// Answer string `json:"answer" jsonschema:"required"`
// Confidence float64 `json:"confidence" jsonschema:"required"`
// }
// outputSchema, _ := schema.NewOutputSchema("result", Result{})
// req.OutputSchema = &outputSchema
OutputSchema *schema.OutputSchema
// Stream indicates whether to use streaming mode.
// When true, Generate() will yield incremental chunks with Partial=true.
// When false, Generate() will yield only the final complete response.
Stream bool
// Metadata contains additional request-specific parameters.
// Use this for provider-specific options not covered by standard fields.
Metadata map[string]any
}
Request contains all parameters for a model generation request. This struct allows passing messages, tools, and other options in a single call.
type Response ¶
type Response struct {
// Message is the actual message content generated by the model
Message message.Message
// Reasoning contains the model's internal reasoning/thinking process.
// This is populated by reasoning-capable models like:
// - OpenAI o1/o3 (reasoning_content)
// - Google Gemini 2.0 Flash (thinking mode)
// - Anthropic Claude 3.5+ (extended thinking)
// Empty for models without native reasoning support.
Reasoning string
// FinishReason indicates why the model stopped generating tokens.
// Common values: "stop" (natural completion), "length" (max tokens),
// "tool_calls" (model wants to call a tool), "content_filter" (content policy).
// Empty string if not provided by the model.
FinishReason string
// Logprobs contains the log probabilities of generated tokens.
// Useful for understanding model confidence and alternative token choices.
// Nil if log probabilities were not requested or not supported.
Logprobs *Logprobs
// Usage provides token consumption information for cost tracking and observability
Usage *UsageInfo
// Metadata contains additional model-specific information
Metadata map[string]any
// Partial indicates whether this is an incremental streaming chunk (true)
// or the final complete response (false).
// For streaming mode: true for all chunks except the last one.
// For blocking mode: always false (single complete response).
// Useful for UI state management (e.g., showing spinners until !Partial).
Partial bool
}
Response wraps the message with additional metadata from model generation. This provides a clean way to access reasoning, usage statistics, and other model-specific information without polluting the message structure.
func Collect ¶
Collect gathers all responses from an iterator into a slice. The final error (if any) is returned separately.
Example:
req := &model.Request{Messages: messages}
responses, err := model.Collect(model.Generate(ctx, req))
if err != nil {
return err
}
for _, resp := range responses {
fmt.Print(resp.Message.String())
}
func Last ¶
Last consumes an iterator and returns only the final response and error. This is the standard way to use Generate() in blocking/non-streaming mode.
Example:
req := &model.Request{Messages: messages}
resp, err := model.Last(model.Generate(ctx, req))
if err != nil {
return err
}
fmt.Println(resp.Message.String())
if resp.Reasoning != "" {
fmt.Println("Reasoning:", resp.Reasoning)
}
type RoutedModel ¶
type RoutedModel struct {
// contains filtered or unexported fields
}
RoutedModel wraps a Router to implement the Model interface. Each Generate call first routes to select the appropriate model, making routing transparent to callers.
func NewRoutedModel ¶
func NewRoutedModel(router Router, opts ...RoutedModelOption) *RoutedModel
NewRoutedModel creates a new RoutedModel that wraps a Router.
func (*RoutedModel) Capabilities ¶
func (m *RoutedModel) Capabilities() Capabilities
Capabilities returns the aggregate capabilities of routable models.
func (*RoutedModel) Generate ¶
Generate implements the Model interface by routing to an appropriate model.
func (*RoutedModel) Router ¶
func (m *RoutedModel) Router() Router
Router returns the underlying router.
type RoutedModelOption ¶
type RoutedModelOption func(*RoutedModel)
RoutedModelOption configures a RoutedModel.
func WithFallbackModel ¶
func WithFallbackModel(m Model) RoutedModelOption
WithFallbackModel sets a fallback model to use when routing fails.
func WithRouteCallback ¶
func WithRouteCallback(fn func(ctx context.Context, req *Request, selected Model)) RoutedModelOption
WithRouteCallback sets a callback invoked after routing decision. Useful for logging, metrics, or debugging routing decisions.
func WithRoutedCapabilities ¶
func WithRoutedCapabilities(caps Capabilities) RoutedModelOption
WithRoutedCapabilities sets the capabilities exposed by the RoutedModel. By default, capabilities are empty. Set this to reflect the aggregate capabilities of all routable models.
type Router ¶
type Router interface {
// Route returns the model to use for the given request.
// Returns ErrNoModelAvailable if no suitable model can be found.
Route(ctx context.Context, req *Request) (Model, error)
}
Router selects an appropriate model for the given request. Implementations can base routing decisions on request content, metadata, cost constraints, capabilities, or any other criteria.
type RouterFunc ¶
RouterFunc is an adapter to use a function as a Router.
type StaticRouter ¶
type StaticRouter struct {
// contains filtered or unexported fields
}
StaticRouter always returns the same model. Useful as a simple fallback or for testing.
func NewStaticRouter ¶
func NewStaticRouter(m Model) *StaticRouter
NewStaticRouter creates a router that always returns the given model.
type TokenLogprob ¶
type TokenLogprob struct {
// Token is the actual token string
Token string
// Logprob is the log probability of this token
Logprob float64
// Bytes is the byte representation of the token (optional)
Bytes []byte
// TopLogprobs contains alternative tokens and their log probabilities.
// Useful for understanding what other tokens the model considered.
TopLogprobs []TopLogprob
}
TokenLogprob represents log probability information for a single token.
type TopLogprob ¶
TopLogprob represents an alternative token choice with its probability.
type UsageInfo ¶
type UsageInfo struct {
// PromptTokens is the number of tokens in the input prompt
PromptTokens int
// CompletionTokens is the number of tokens in the generated completion
CompletionTokens int
// ReasoningTokens is the number of tokens used for internal reasoning.
// Only applicable for reasoning-capable models (e.g., OpenAI o1/o3).
// Zero for models without separate reasoning token tracking.
ReasoningTokens int
// TotalTokens is the sum of prompt, completion, and reasoning tokens
TotalTokens int
}
UsageInfo tracks token consumption for observability and cost tracking.
type WeightedRouter ¶
type WeightedRouter struct {
// contains filtered or unexported fields
}
WeightedRouter selects models based on weights. Higher weights mean higher probability of selection.
func NewWeightedRouter ¶
func NewWeightedRouter(models []Model, weights []int, opts ...WeightedRouterOption) *WeightedRouter
NewWeightedRouter creates a router that selects based on weights. weights[i] corresponds to models[i]. Higher weight = more likely selection.
type WeightedRouterOption ¶
type WeightedRouterOption func(*WeightedRouter)
WeightedRouterOption configures a WeightedRouter.
func WithWeightedRandomFunc ¶
func WithWeightedRandomFunc(fn func(n int) int) WeightedRouterOption
WithWeightedRandomFunc sets a custom random function for weighted selection.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package amazonbedrock provides a model adapter for Amazon Bedrock foundation models.
|
Package amazonbedrock provides a model adapter for Amazon Bedrock foundation models. |
|
Package anthropic provides integration with the Anthropic AI API for large language models.
|
Package anthropic provides integration with the Anthropic AI API for large language models. |
|
Package gemini provides Google Gemini model integration for the agentmesh framework.
|
Package gemini provides Google Gemini model integration for the agentmesh framework. |
|
Package langchaingo provides an adapter for using LangChainGo (github.com/tmc/langchaingo) models within AgentMesh workflows.
|
Package langchaingo provides an adapter for using LangChainGo (github.com/tmc/langchaingo) models within AgentMesh workflows. |
|
Package middleware provides reusable middleware for model executors.
|
Package middleware provides reusable middleware for model executors. |
|
Package ollama provides sentinel errors for the Ollama model package.
|
Package ollama provides sentinel errors for the Ollama model package. |
|
Package openai provides integration with the OpenAI API for large language models.
|
Package openai provides integration with the OpenAI API for large language models. |