Ruby LLM Evals

LLM evaluation engine for Rails.

Test, compare, and improve your LLM prompts within your Rails application.

What you get

• Provider: see available providers
• Model: see available models. In case you're selecting a local provider (eg. Ollama), you can enter the model name in a text field.
• Instructions: optional, the system prompt.
• Message: message template.
• Temperature: optional, controls randomness (0.0 to 1.0). Lower values make output more focused and deterministic.
• Params: optional, additional provider-specific parameters as JSON (e.g., {"max_tokens": 1000}).
• Thinking budget: optional, sets a maximum token budget for thinking/reasoning.

Mounting it

# Gemfile
gem "ruby_llm-evals"

# config/routes.rb
mount RubyLLM::Evals::Engine, at: "/evals"

Details

21 stars on GitHub · maintained by sinaptia. Source: https://github.com/sinaptia/ruby_llm-evals