We shipped some big new updates for our Checkly AI features to help you assess and triage any Playwright failures better and (much) quicker.

Impact Analysis and Root Cause for Playwright Checks

We’ve upgraded our model and added more data sources like network logs and your script code to get higher fidelity answers to the question: “Why did my Playwright-based check fail and who’s impacted?”

For each failure of a Playwright-based check you now get:

• An upgraded End-user Impact Analysis that uses your company’s domain specific wording. More on that below.

• A succinct summary of Test Steps to drastically cut down time finding out where your script failed.

• A Root Cause to determine what caused the error, saving a lot of time scrolling through your code, traces, videos, screenshots and logs.

• An Error Category classification to immediately point your nose in the right direction.

These upgrades are now live.

Add Domain Knowledge with the Tag Description Editor

To make the end-user impact analysis as descriptive as possible, you can enrich any tags you use in Checkly in the Tag Description Editor. This data is then used as context by Checkly AI to give you more accurate descriptions that match your organization’s internal lingo.

You can access this editor under the AI settings tab.

Bring Your Own Model

By default, Checkly AI works with OpenAI GPT-4.1. To address data privacy concerns and general AI / data governance, you can now bring your own LLM from a preferred supplier. We have built-in support for Azure-hosted models and Google Gemini models.

This feature is available on request for Enterprise accounts. If your preferred model is not listed, we can add new models and providers on request.

If you’re an Enterprise customer, go to the Labs section and turn on “Checkly AI.” Then, opt in to the feature on the “AI” screen in account settings.

Want more info? Book a call with our co-founder Tim.