We just released Checkly’s first two AI-assisted features in beta!

First up is Playwright Impact Analysis. This feature generates a concise summary for every failing Playwright test focusing on end-user impact and aims to answer questions like “who is impacted?”, “which features are broken?” and “what steps led to this failure?”.

If your failed Playwright test has a trace file, you will find an Analyze with AI button that kicks off the analysis.

The second is the Explain Error button, displayed above every Playwright error message. Clicking this button uses AI to explain why a script failed, how to fix it with a snipper of example code and displays any best practices and links to helpful documentation, like the official Playwright docs.

Both features are now in beta and free for all users during this phase.

If you’re an Enterprise customer, go to the Labs section and turn on “Checkly AI.” Then, opt in to the feature on the “AI” screen in account settings.

Under the hood, we are using OpenAI GPT-4.1 but we can enable custom providers like Azure and Google Gemini for Enterprise users.

Book a call with our co-founder Tim, who can help you set it up and answer all your questions.