fix: updated logger and model caching #release

[thecodacus]

Optimize Model List Fetching and Improve Provider Management

Overview

This PR optimizes the LLM provider management by implementing lazy loading and caching of dynamic models. Instead of fetching models from all providers on every LLM call, we now only fetch models from the selected provider. Additionally, we've added a caching mechanism for model lists and improved logging throughout the system.

Key Changes

1. Optimized Model List Fetching

• Implemented lazy loading for dynamic models - only fetch from selected provider
• Added caching mechanism for provider model lists
• Moved model list management logic to LLMManager class

2. Universal Logging System

• Enhanced logging with structured scopes using Chalk
• Added cross-platform colored console output that works in terminals
• Implemented consistent debug levels and scoped loggers
• Unified logging format across different environments

Technical Details

Model Fetching Optimization

Key implementation changes (simplified):

// Before
const MODEL_LIST = await getModelList({ apiKeys, providerSettings, serverEnv });
const modelDetails = MODEL_LIST.find((m) => m.name === currentModel);

// After
const modelsList = [
  ...(provider.staticModels || []),
  ...(await LLMManager.getInstance().getModelListFromProvider(provider, {
    apiKeys,
    providerSettings,
    serverEnv
  }))
];

Caching Implementation

class BaseProvider {
  cachedDynamicModels?: {
    cacheId: string;
    models: ModelInfo[];
  };

  getModelsFromCache(options: ProviderOptions): ModelInfo[] | null {
    if (!this.cachedDynamicModels) return null;
    
    const cacheKey = this.getDynamicModelsCacheKey(options);
    if (cacheKey !== this.cachedDynamicModels.cacheId) {
      this.cachedDynamicModels = undefined;
      return null;
    }
    
    return this.cachedDynamicModels.models;
  }
}

Enhanced Cross-Platform Logging

// Universal logging with Chalk for consistent terminal output
const chalk = new Chalk({ level: 3 });

function formatText(text: string, color: string, bg: string) {
  return chalk.bgHex(bg)(chalk.hex(color)(text));
}

// Usage in logger
const labelText = formatText(` ${level.toUpperCase()} `, textColor, bgColor);
console.log(`${labelText}`, allMessages);

// Scoped logger usage example
const logger = createScopedLogger('stream-text');
logger.info(`Sending llm call to ${provider.name}`); // Outputs with consistent colors

Migration Impact

• No breaking changes in the API
• Cached models are automatically invalidated when provider settings change
• Existing code paths continue to work with fallback mechanisms

Testing

• Verified model caching behavior across multiple requests
• Tested cache invalidation scenarios
• Confirmed logging output in different environments
• Validated fallback behavior for unknown models

Future Improvements

• Add cache expiration mechanism
• Implement background refresh for cached models
• Add metrics for cache hit/miss rates
• Consider alternative color schemes for different terminal themes
• Add log rotation and persistence options
• Consider implementing a shared cache for multi-instance deployments

[dustinwloring1988]

worked for static models but the dynmic models tried to call claude-3-5-sonnet-latest when using ollama, then it tried the one I had selected but never got a response. I can look into this more later one today.