Custom Metrics on Dashboard

Problem:
I have a a Playwright test for an onboarding flow currently assesses LLM-generated content using an LLM-as-a-judge method with a binary pass/fail outcome (score > 60). While functional, this approach lacks the ability to track score trends over time, which would provide more valuable insights.

Suggested solution:
Implement a custom metrics system that:

  1. Captures the actual LLM evaluation scores from tests

  1. Exposes these scores to Checkly's dashboard

  1. Enables visualization of individual scores per run and aggregated metrics (7-day and 14-day averages)

  1. Creates an extensible framework for any future custom metrics beyond just LLM evaluation scores

Please authenticate to join the conversation.

Upvoters
Status

In Review

Board

πŸ’‘ Feature Request

Date

9 months ago

Author

Berk Durmus

Subscribe to post

Get notified by email when there are changes.