Documentation
¶
Overview ¶
Package proxy provides the transparent reverse proxy implementation.
Package proxy provides the transparent proxy functionality for the LLM API.
Index ¶
- func CacheKeyFromRequest(r *http.Request) string
- func CacheKeyFromRequestWithVary(r *http.Request, vary string) string
- func Chain(h http.Handler, middleware ...Middleware) http.Handler
- type APIConfig
- type APIProviderConfig
- type AuditLogger
- type CacheMetricType
- type CacheStatsAggregator
- type CacheStatsAggregatorConfig
- type CacheStatsStore
- type ConnectionConfig
- type ErrorResponse
- type Middleware
- type Project
- type ProjectActiveChecker
- type ProjectStore
- type Proxy
- type ProxyConfig
- type ProxyMetrics
- type TimeoutConfig
- type TokenValidator
- type TransparentProxy
- func NewTransparentProxy(config ProxyConfig, validator TokenValidator, store ProjectStore) (*TransparentProxy, error)
- func NewTransparentProxyWithAudit(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func NewTransparentProxyWithLogger(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func NewTransparentProxyWithLoggerAndObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func NewTransparentProxyWithObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func (p *TransparentProxy) Cache() httpCache
- func (p *TransparentProxy) Handler() http.Handler
- func (p *TransparentProxy) LoggingMiddleware() Middleware
- func (p *TransparentProxy) Metrics() ProxyMetrics
- func (p *TransparentProxy) MetricsMiddleware() Middleware
- func (p *TransparentProxy) SetCacheStatsAggregator(agg *CacheStatsAggregator)
- func (p *TransparentProxy) SetMetrics(m *ProxyMetrics)
- func (p *TransparentProxy) Shutdown(ctx context.Context) error
- func (p *TransparentProxy) TimeoutMiddleware(timeout time.Duration) Middleware
- func (p *TransparentProxy) ValidateRequestMiddleware() Middleware
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CacheKeyFromRequest ¶
func CacheKeyFromRequestWithVary ¶
CacheKeyFromRequestWithVary generates a cache key using the specified Vary header from the upstream response. This enables per-response driven cache key generation.
Behavior:
- When vary is empty or "*", this function returns a key that ignores header values (treated as "no vary" for key generation purposes).
- Header names parsed from vary are normalized to canonical MIME header names.
Types ¶
type APIConfig ¶
type APIConfig struct {
// APIs contains configurations for different API providers
APIs map[string]*APIProviderConfig `yaml:"apis"`
// DefaultAPI is the default API provider to use if not specified
DefaultAPI string `yaml:"default_api"`
}
APIConfig represents the top-level configuration for API proxying
func LoadAPIConfigFromFile ¶
LoadAPIConfigFromFile loads API configuration from a YAML file
func (*APIConfig) GetProxyConfigForAPI ¶
func (c *APIConfig) GetProxyConfigForAPI(apiName string) (*ProxyConfig, error)
GetProxyConfigForAPI returns a ProxyConfig for the specified API provider
type APIProviderConfig ¶
type APIProviderConfig struct {
// BaseURL is the target API base URL
BaseURL string `yaml:"base_url"`
// AllowedEndpoints is a list of endpoint prefixes that are allowed to be accessed
AllowedEndpoints []string `yaml:"allowed_endpoints"`
// AllowedMethods is a list of HTTP methods that are allowed
AllowedMethods []string `yaml:"allowed_methods"`
// Timeouts for various operations
Timeouts TimeoutConfig `yaml:"timeouts"`
// Connection settings
Connection ConnectionConfig `yaml:"connection"`
// ParamWhitelist is a map of parameter names to allowed values
ParamWhitelist map[string][]string `yaml:"param_whitelist"`
AllowedOrigins []string `yaml:"allowed_origins"`
RequiredHeaders []string `yaml:"required_headers"`
}
APIProviderConfig represents the configuration for a specific API provider
type AuditLogger ¶
AuditLogger defines the interface for audit event logging
type CacheMetricType ¶
type CacheMetricType int
CacheMetricType represents the kind of cache metric to increment.
const ( CacheMetricHit CacheMetricType = iota CacheMetricMiss CacheMetricBypass CacheMetricStore )
type CacheStatsAggregator ¶
type CacheStatsAggregator struct {
// contains filtered or unexported fields
}
CacheStatsAggregator aggregates cache hit events and periodically flushes them to the database. It uses a buffered channel for non-blocking enqueue and drops events when the buffer is full.
func NewCacheStatsAggregator ¶
func NewCacheStatsAggregator(config CacheStatsAggregatorConfig, store CacheStatsStore, logger *zap.Logger) *CacheStatsAggregator
NewCacheStatsAggregator creates a new aggregator with the given configuration.
func (*CacheStatsAggregator) RecordCacheHit ¶
func (a *CacheStatsAggregator) RecordCacheHit(tokenID string)
RecordCacheHit enqueues a cache hit event for the given token. This is non-blocking; if the buffer is full, the event is dropped.
func (*CacheStatsAggregator) Start ¶
func (a *CacheStatsAggregator) Start()
Start begins the background aggregation worker.
type CacheStatsAggregatorConfig ¶
type CacheStatsAggregatorConfig struct {
BufferSize int // Size of the buffered channel (default: 1000)
FlushInterval time.Duration // How often to flush stats to DB (default: 5s)
BatchSize int // Max events before flush (default: 100)
}
CacheStatsAggregatorConfig holds configuration for the cache stats aggregator.
func DefaultCacheStatsAggregatorConfig ¶
func DefaultCacheStatsAggregatorConfig() CacheStatsAggregatorConfig
DefaultCacheStatsAggregatorConfig returns the default configuration.
type CacheStatsStore ¶
type CacheStatsStore interface {
// IncrementCacheHitCountBatch increments cache_hit_count for multiple tokens.
// The deltas map has token IDs as keys and increment values as values.
IncrementCacheHitCountBatch(ctx context.Context, deltas map[string]int) error
}
CacheStatsStore defines the interface for persisting cache hit counts.
Consistency invariant: Under normal operation, CacheHitCount ≤ RequestCount should always hold for any token. Cache hits are only recorded when responses are served from cache, which requires a prior upstream request that incremented RequestCount.
If CacheHitCount > RequestCount is observed, it indicates a system issue that should be investigated (e.g., request count increment failed while cache hit was recorded). The Admin UI uses safeSub to display max(0, RequestCount-CacheHitCount) to handle this edge case gracefully.
type ConnectionConfig ¶
type ConnectionConfig struct {
// MaxIdleConns is the maximum number of idle connections
MaxIdleConns int `yaml:"max_idle_conns"`
// MaxIdleConnsPerHost is the maximum number of idle connections per host
MaxIdleConnsPerHost int `yaml:"max_idle_conns_per_host"`
}
ConnectionConfig contains connection settings for the proxy
type ErrorResponse ¶
type ErrorResponse struct {
Error string `json:"error"`
Description string `json:"description,omitempty"`
Code string `json:"code,omitempty"`
}
ErrorResponse is the standard format for error responses
type Middleware ¶
Middleware defines a function that wraps an http.Handler
func CircuitBreakerMiddleware ¶
func CircuitBreakerMiddleware(failureThreshold int, cooldown time.Duration, isTransient func(status int) bool) Middleware
CircuitBreakerMiddleware returns a middleware that opens the circuit after N consecutive failures. While open, it returns 503 immediately. After a cooldown, it closes and allows requests again.
func ProjectActiveGuardMiddleware ¶
func ProjectActiveGuardMiddleware(enforceActive bool, checker ProjectActiveChecker, auditLogger AuditLogger) Middleware
ProjectActiveGuardMiddleware creates middleware that enforces project active status If enforceActive is false, the middleware passes through all requests without checking If enforceActive is true, inactive projects receive a 403 Forbidden response
type Project ¶
type Project struct {
ID string `json:"id"`
Name string `json:"name"`
APIKey string `json:"api_key"` // Provider-agnostic API key. Encrypted when ENCRYPTION_KEY is set.
IsActive bool `json:"is_active"`
DeactivatedAt *time.Time `json:"deactivated_at,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
Project represents a project for the management API and proxy (copied from database/models.go)
type ProjectActiveChecker ¶
type ProjectActiveChecker interface {
// GetProjectActive checks if a project is active
GetProjectActive(ctx context.Context, projectID string) (bool, error)
}
ProjectActiveChecker defines the interface for checking project active status
type ProjectStore ¶
type ProjectStore interface {
// GetAPIKeyForProject retrieves the API key for a project
GetAPIKeyForProject(ctx context.Context, projectID string) (string, error)
// GetProjectActive checks if a project is active
GetProjectActive(ctx context.Context, projectID string) (bool, error)
// Management API CRUD
ListProjects(ctx context.Context) ([]Project, error)
CreateProject(ctx context.Context, project Project) error
GetProjectByID(ctx context.Context, projectID string) (Project, error)
UpdateProject(ctx context.Context, project Project) error
DeleteProject(ctx context.Context, projectID string) error
}
ProjectStore defines the interface for retrieving and managing project information (extended for management API)
type Proxy ¶
type Proxy interface {
// Handler returns an http.Handler for the proxy
Handler() http.Handler
// Shutdown gracefully shuts down the proxy
Shutdown(ctx context.Context) error
}
Proxy defines the interface for a transparent HTTP proxy
type ProxyConfig ¶
type ProxyConfig struct {
// TargetBaseURL is the base URL of the API to proxy to
TargetBaseURL string
// AllowedEndpoints is a whitelist of endpoints that can be accessed
AllowedEndpoints []string
// AllowedMethods is a whitelist of HTTP methods that can be used
AllowedMethods []string
// RequestTimeout is the maximum duration for a complete request
RequestTimeout time.Duration
// ResponseHeaderTimeout is the time to wait for response headers
ResponseHeaderTimeout time.Duration
// FlushInterval is how often to flush streaming responses
FlushInterval time.Duration
// MaxIdleConns is the maximum number of idle connections
MaxIdleConns int
// MaxIdleConnsPerHost is the maximum number of idle connections per host
MaxIdleConnsPerHost int
// IdleConnTimeout is how long to keep idle connections alive
IdleConnTimeout time.Duration
// LogLevel controls the verbosity of logging
LogLevel string
// LogFormat controls the log output format (json or console)
LogFormat string
// LogFile specifies a file path for logs (stdout if empty)
LogFile string
// SetXForwardedFor determines whether to set the X-Forwarded-For header
SetXForwardedFor bool
// ParamWhitelist is a map of parameter names to allowed values
ParamWhitelist map[string][]string
// AllowedOrigins is a list of allowed CORS origins for this provider
AllowedOrigins []string
// RequiredHeaders is a list of required request headers (case-insensitive)
RequiredHeaders []string
// Project active guard configuration
EnforceProjectActive bool // Whether to enforce project active status
// --- HTTP cache (global, opt-in; set programmatically, not via YAML) ---
// HTTPCacheEnabled toggles the proxy cache for GET/HEAD based on HTTP semantics
HTTPCacheEnabled bool
// HTTPCacheDefaultTTL is used only when upstream allows caching but omits explicit TTL
HTTPCacheDefaultTTL time.Duration
// HTTPCacheMaxObjectBytes is a guardrail for maximum cacheable response size
HTTPCacheMaxObjectBytes int64
// RedisCacheURL enables Redis-backed cache when non-empty (e.g., redis://localhost:6379/0)
RedisCacheURL string
// RedisCacheKeyPrefix allows namespacing cache keys (default: llmproxy:cache:)
RedisCacheKeyPrefix string
}
ProxyConfig contains configuration for the proxy
func (*ProxyConfig) Validate ¶
func (c *ProxyConfig) Validate() error
Validate checks that the ProxyConfig is valid and returns an error if not.
type ProxyMetrics ¶
type ProxyMetrics struct {
RequestCount int64
ErrorCount int64
TotalResponseTime time.Duration
// Cache metrics (provider-agnostic counters)
CacheHits int64 // Cache hits (responses served from cache)
CacheMisses int64 // Cache misses (responses fetched from upstream)
CacheBypass int64 // Cache bypassed (e.g., due to authorization)
CacheStores int64 // Cache stores (responses stored in cache)
// contains filtered or unexported fields
}
ProxyMetrics tracks proxy usage statistics
type TimeoutConfig ¶
type TimeoutConfig struct {
// Request is the overall request timeout
Request time.Duration `yaml:"request"`
// ResponseHeader is the timeout for receiving response headers
ResponseHeader time.Duration `yaml:"response_header"`
// IdleConnection is the timeout for idle connections
IdleConnection time.Duration `yaml:"idle_connection"`
// FlushInterval controls how often to flush streaming responses
FlushInterval time.Duration `yaml:"flush_interval"`
}
TimeoutConfig contains timeout settings for the proxy
type TokenValidator ¶
type TokenValidator interface {
// ValidateToken validates a token and returns the associated project ID
ValidateToken(ctx context.Context, token string) (string, error)
// ValidateTokenWithTracking validates a token, increments its usage, and returns the project ID
ValidateTokenWithTracking(ctx context.Context, token string) (string, error)
}
TokenValidator defines the interface for token validation
type TransparentProxy ¶
type TransparentProxy struct {
// contains filtered or unexported fields
}
TransparentProxy implements the Proxy interface for transparent proxying
func NewTransparentProxy ¶
func NewTransparentProxy(config ProxyConfig, validator TokenValidator, store ProjectStore) (*TransparentProxy, error)
NewTransparentProxy creates a new proxy instance with an internally configured logger based on the provided ProxyConfig.
func NewTransparentProxyWithAudit ¶
func NewTransparentProxyWithAudit(config ProxyConfig, validator TokenValidator, store ProjectStore, logger *zap.Logger, auditLogger AuditLogger, obsCfg middleware.ObservabilityConfig) (*TransparentProxy, error)
NewTransparentProxyWithAudit creates a proxy with audit logging capabilities.
func NewTransparentProxyWithLogger ¶
func NewTransparentProxyWithLogger(config ProxyConfig, validator TokenValidator, store ProjectStore, logger *zap.Logger) (*TransparentProxy, error)
NewTransparentProxyWithLogger allows providing a custom logger. If logger is nil a new one is created based on the ProxyConfig.
func NewTransparentProxyWithLoggerAndObservability ¶
func NewTransparentProxyWithLoggerAndObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, logger *zap.Logger, obsCfg middleware.ObservabilityConfig) (*TransparentProxy, error)
NewTransparentProxyWithLoggerAndObservability creates a proxy with observability middleware using an existing logger.
func NewTransparentProxyWithObservability ¶
func NewTransparentProxyWithObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, obsCfg middleware.ObservabilityConfig) (*TransparentProxy, error)
NewTransparentProxyWithObservability creates a new proxy with observability middleware.
func (*TransparentProxy) Cache ¶
func (p *TransparentProxy) Cache() httpCache
Cache returns the HTTP cache instance for management operations. Returns nil if caching is disabled.
func (*TransparentProxy) Handler ¶
func (p *TransparentProxy) Handler() http.Handler
Handler returns the HTTP handler for the proxy
func (*TransparentProxy) LoggingMiddleware ¶
func (p *TransparentProxy) LoggingMiddleware() Middleware
LoggingMiddleware logs request details
func (*TransparentProxy) Metrics ¶
func (p *TransparentProxy) Metrics() ProxyMetrics
Metrics returns a copy of the current proxy metrics. Returns a value copy to ensure thread-safety when reading metrics.
func (*TransparentProxy) MetricsMiddleware ¶
func (p *TransparentProxy) MetricsMiddleware() Middleware
MetricsMiddleware collects metrics about requests
func (*TransparentProxy) SetCacheStatsAggregator ¶
func (p *TransparentProxy) SetCacheStatsAggregator(agg *CacheStatsAggregator)
SetCacheStatsAggregator sets the cache stats aggregator for per-token cache hit tracking.
func (*TransparentProxy) SetMetrics ¶
func (p *TransparentProxy) SetMetrics(m *ProxyMetrics)
SetMetrics overwrites the current metrics (primarily for testing).
func (*TransparentProxy) Shutdown ¶
func (p *TransparentProxy) Shutdown(ctx context.Context) error
Shutdown gracefully shuts down the proxy
func (*TransparentProxy) TimeoutMiddleware ¶
func (p *TransparentProxy) TimeoutMiddleware(timeout time.Duration) Middleware
TimeoutMiddleware adds a timeout to requests
func (*TransparentProxy) ValidateRequestMiddleware ¶
func (p *TransparentProxy) ValidateRequestMiddleware() Middleware
ValidateRequestMiddleware validates the incoming request against allowed endpoints and methods