Make AI context explicit and close the loop from generated check to validated fix

I recently used Checkly’s AI Assistant to create a monitoring plan for WERESK, my production SaaS platform. The first-run experience was impressive: after exploring three public endpoints, Checkly generated and deployed eleven URL, API, browser, DNS and heartbeat checks within minutes.

However, creating the checks happened much faster than understanding how they worked, how reliable they were and which AI surface I should use when something failed.

The problem

One generated browser check for the anonymous demo flow failed immediately. The application itself loaded correctly, but the generated test expected profile or card elements to be visible on the first page:

[data-testid="profile-card"], [data-testid="profile-row"], table tbody tr, [class*="profile"], [class*="card"]

In the actual application, /demo initially displays a static dashboard. The user must first click “Profiles” before the profiles table and search field become visible. The generated check therefore represented an incorrect user flow and used broad CSS selectors that did not correspond to the actual DOM structure.

Rocky AI correctly recognized that the selector might be wrong and allowed me to add application context. However, the next action was not obvious. There was no direct way to ask AI to update the generated check, review a proposed change, rerun it and save the working version.

At the same time, Checkly presents several AI entry points:

• the global AI Assistant used to explore a platform and create monitoring plans;

• Rocky AI and its Context field used to analyse failed checks;

• “Ask AI” inside the support widget, apparently intended for product help and documentation.

For a new user, it is not immediately clear:

• what context each AI assistant can access;

• whether it understands the monitored application, the Checkly account, the current check or only Checkly documentation;

• whether it can analyse, explain, modify or deploy resources;

• where a user should continue after an AI-generated check fails;

• whether context added to Rocky AI is available when editing or regenerating the check.

There is also no contextual tutorial explaining these boundaries during the first real monitoring workflow.

Suggested solution

I would make the scope and capabilities of every AI surface explicit and connect them into a single recovery workflow.

For example, each AI interface could state its current context:

• “Working with your Checkly account and monitoring plan”

• “Analysing this check and its latest failed run”

• “Answering questions about Checkly”

• “Using the application context you provided”

For AI-generated checks, the failed-run experience could offer a guided loop:

1. Detect whether the failure most likely comes from the monitored application, test design, locator or environment.

2. Show the evidence used for that classification.

3. Let the user add missing application context.

4. Generate a proposed patch to the check.

5. Present the code diff and explain what will change.

6. Run the updated check without immediately replacing the active version.

7. Compare the new result with the failed run.

8. Save the change only after successful validation.

In my case, this workflow could have identified that the page loaded correctly, discovered the visible “Profiles” navigation control, proposed clicking it before checking the table, and replaced the generic CSS locator with stable Playwright locators based on roles, headings and accessible names.

Contextual onboarding

Instead of a separate generic product tour, I would introduce guidance at the moment when the user needs it:

• after the first monitoring plan is generated;

• when the first check fails;

• when switching between AI Assistant, Rocky AI and support AI;

• before applying an AI-generated modification;

• when a heartbeat is waiting for its first ping;

• when credentials or a repository integration would expand the available functionality.

The onboarding goal should not only be “time to first check.” A generated check is not valuable until the user understands it and trusts that it monitors the intended behaviour.

A more meaningful activation metric could be:

Time from creating an AI-generated monitoring plan to the first reliable, understood and successfully validated check.

Supporting metrics could include:

• time required to classify the first failure;

• percentage of generated checks that pass after validation;

• percentage of failed generated checks successfully repaired in-product;

• number of users who abandon or disable generated checks;

• frequency of prompts submitted to the wrong AI surface;

• time from an initial failed run to a successful rerun;

• retention of AI-generated checks after seven or thirty days.

Expected value

This would preserve the speed and initial “wow” effect of AI-generated monitoring while adding confidence and control. It would also help users learn the product through a real task instead of requiring them to understand Checkly’s internal structure beforehand.

The core opportunity is to close the loop:

Explore → generate → run → diagnose → propose a fix → review → rerun → validate

Checkly already provides many of the individual components. Connecting them into one understandable workflow would make AI-generated monitoring feel less like a fast starting point and more like a reliable product capability.