Global playwright.config.ts support (viewport, setup, teardown, storageState ...)

To truly unite E2E testing and monitoring, Checkly CLI has to support the global configuration defined in playwright.config.ts, so that it can be used in both scenarios.playwright.config.ts is very useful to handle stuff like:Common viewport for testsCommon setup, teardown for testsCommon storageState for testsTurn screenshot, trace, video on/offAnd all the other useful playwright.config features that make playwright awesome

Younes Jallouli 5 months ago

4

Dec 31

Browser Checks

+1

Dark. Mode.

💡 For general support requests and bug reports, please go to checklyhq.com/support Is your feature request related to a problem? Please describe. My eyes at night. It hurz Describe the solution you'd like Dark mode. Describe alternatives you've considered Wearing sunglasses looking at Checkly, at night. Additional context 🌚 original GH issue link https://github.com/checkly/public-roadmap/issues/124

Tim Nolet 7 months ago

General

Multi-step API check

Describe the solution you'd like For a CRUD API set, it makes sense to test these in a multi-step sequence. For example, to do a POST to create an object, assert that it was created in the response, then POST to delete the object and assert the response. Describe alternatives you've considered Could potentially do this in setup/teardown scripts, although then it is less clear in the monitoring. original GH issue link https://github.com/checkly/public-roadmap/issues/46

Tim Nolet 7 months ago

3

Dec 31

API Checks

Static IPs

💡 For general support requests and bug reports, please go to checklyhq.com/supportIs your feature request related to a problem? Please describe.I would like to browser-checks for our internal tools (Blocked IPs, Some in private network). Which is not open public? I don't have much control on web server level to these tools. Currently, there is no way to have checks to these.Documentation suggesting whitelisting AWS-IPs doesn't make sense, this basically means opening to internet.Describe the solution you'd likeStatic IP sets. I don't know how your infrastructure looks like, if it's possible to run check runners behind few NATs. We can have IPs sets which can be whitelisted in our LB/firewalls. Any ways these IPs can be changed if your infra needs some update, So an API endpoint which list current IPs help more, to automate IP whitelisting.If NAT is not a possible solution, having an open AWS SNS, Even-Bridge some sort where I can subscribe in our AWS accounts to load new IPs to SGs using lambda.Having private-link or VPC peer to our VPC. This can help to have checks on private network itself.original GH issue link https://github.com/checkly/public-roadmap/issues/154

Tim Nolet 7 months ago

Dec 31

General

Subscribe to status updates

Is your feature request related to a problem? Please describe. I want to allow users to subscribe to status updates from our service. I want to subscribe to Checkly's status so I know when the service is down or degraded, because: This might impact how our customers see the status or our service; We want to know that our services might not be monitored for the duration of the inciden. Describe the solution you'd like An option on Checkly's status page allowing users to subscribe to updates. Ideally, they would be able to choose between multiple ways to get notifications, like RSS/Atom feeds, email, Slack, maybe even webhook - see example from Github's status page: Which is similar to Sentry's - it seems they use Atlassian Status Page. I see #297 asks for a specific type of update, but I'd like to have more options than just email. original GH issue link https://github.com/checkly/public-roadmap/issues/304

Tim Nolet 7 months ago

10

Status Pages

Smarter retries strategies: same region and backoff

The current retry strategy for all checks is fairly limited. We want to expand the possibilities for retrying with two options:- same region: right now, retries are always scheduled on a different region than the original attempt. This is not ideal for geo-fenced checks or region-aware websites rendering different content for different regions.- backoff strategies: we want to enable multiple retries according to a strategy the user can define: linear or exponential so the target system has more time to recover from an initial failure.

Tim Nolet 4 months ago

6

Aug 31

Scheduling

Support for `toHaveScreenshot` / visual regression testing

Is your feature request related to a problem? Please describe. I have a particular use case for toHaveScreenshot in that my Checkly tests goes through our UI to generate a PDF export of a particular page that you're on. I'd love to be able to compare a screenshot of the PDF generated to a snapshot, as it's impossible to run typical assertions on the information shown in PDFs.Describe the solution you'd like I understand that Checkly agents run in an isolated environment for each test. However, toHaveScreenshot could be supported in a way similar to Buildkite in how it allows you to manage artifacts between builds and build steps, it would be cool if the Checkly agent could detect snapshots generated from toHaveScreenshot, store them somewhere, and then pull them down for the next test to verify against. This could be automatic, or something that the test does, wherer you test can import some kind of Checkly npm package that exports some functions to do these particular tasks.Describe alternatives you've considered What I'm trying to achieve here I can write this code myself. I.e i can store the screenshot snapshots in S3 at the end of my test and write them to disk when the test starts again but that's not ideal at scale if I need to do this in multiple tests. There are visual comparison libraries out there but they aren't in the Checkly runtimes. Adding some kind of visual comparison npm package to the runtimes would suffice too!!!original GH issue link https://github.com/checkly/public-roadmap/issues/282

Tim Nolet 7 months ago

1

Nov 9

Browser Checks

Store tokens and other credentials across check runs

Any check that requires auth / credentials that expire needs to fetch these credentials on every check run. While this is the safest way to make sure checks are authenticated, it creates friction:- extra runtime in a browser check or setup script.- extra load on any authentication endpoint.- waste of general cloud resource.A user should be able to store tokens securely in a check run and fetch or revalidate the credentials when or before they expire.

Tim Nolet 6 months ago

General

Increase Response time limit for checks

💡 For general support requests and bug reports, please go to checklyhq.com/support Is your feature request related to a problem? Please describe. We have a time out of apis defined at our load balancer level as 60 sec, checkly times out the request at 30 sec max with which we are not able to identify if the system is not responding or checkly servers not able to communicate with our servers. Describe the solution you'd like Providing an option of custom timeout (maximum configurable to 100 sec) Describe alternatives you've considered NA for now Additional context NA original GH issue link https://github.com/checkly/public-roadmap/issues/207

Tim Nolet 7 months ago

7

Scheduling

Show planned maintenance windows in dashboard

Is your feature request related to a problem? Please describe. we're very happy with the new dashboards, which have become integral to our communication (https://status.qfield.cloud/). There is, unfortunately, a big shortcoming for us. A planned maintenance window is only visible once it becomes active. To really become fully operational as a status page, these should be visible ahead of time. Describe the solution you'd like show planned maintenance as non-active planned incidents Describe alternatives you've considered A minimal solution could be to show planned maintenance as in the admin dashboard Additional context potentially part of #242 and #240 and #65 original GH issue link https://github.com/checkly/public-roadmap/issues/302

Tim Nolet 7 months ago

Status Pages

Scheduled checks at specific times

It would be great to be able to run checks at specific times (like a cron job schedule), as we have an API that publishes certain customer-critical data daily at specific times. So instead of every X minutes which can be any time depending on when the check is first created, we would be able to run reliably at specific times.

Yawar Amin 4 months ago

6

Scheduling

Automated Account Creation

HI Checkly Team, We have lots of Checkly accounts created under our org and are looking to get more in the future. We have a tooling suite for creating checks on Checkly and its pretty much streamlined except for the account creation part. We are looking to streamline account creations as well, as the current process is to email our Checkly representative and that has a turn-around time of ~24hours. We are looking to have this automated via API, preferably get it to be a Terraform resource. We are currently using terraform to create Checkly resources, but whatever way would be faster to get this to become a reality would be great

Giancarlo 6 months ago

4

Auth / Access / Roles

add ability for tcp and icmp checks.

💡 For general support requests and bug reports, please go to checklyhq.com/support Is your feature request related to a problem? Please describe. We do http monitoring but we have some api's that are not http but are at tcp level. Having monitoring for tcp connections basic check on URL and any Port is great. Can integrate tls as well. Describe the solution you'd like as part of addition of api checks, we can add tcp as an option in the drop down. Describe alternatives you've considered using datadog right now . Additional context Add any other context or screenshots about the feature request here. original GH issue link https://github.com/checkly/public-roadmap/issues/96

Tim Nolet 7 months ago

1

Check Types

Customisation of NPM Dependencies

Summary:Allow users to customise or override the NPM dependencies within the Checkly runtime environment.Description:Currently, users are restricted to a set list of packages and versions in the Checkly runtime, which may not cover all the specific use cases or compatibility issues they encounter. As a Checkly user, I need to be able to dynamically use packages in my own runtime. I process dependency updates in projects regularly to use the latest versions for new features and security updates.

Alex Noyes 3 months ago

Runtimes

Private Dashboards

As a Checkly administrator I would like to create private dashboards (View only TV mode style) and share them with third parties using individual username/password or other secure methods that ensure the dashboards are only viewable by authorised users. These users should not count towards the "employee or user" count for billing but may be bundled as part of billing packages. original GH issue link https://github.com/checkly/public-roadmap/issues/23

Tim Nolet 7 months ago

1

Jun 1

Status Pages

Alerting based on p99 response time

💡 For general support requests and bug reports, please go to checklyhq.com/support Is your feature request related to a problem? Please describe. Instead of alerting on a single request being slow, I'd like to alert if the average or p95 time exceeds a limit. Describe the solution you'd like When setting Assertions, I would like to be able to chose "average response time" or "p95 response time" as the Property value for a Response Time check. Describe alternatives you've considered Checking the dashboard manually? I'm not sure what other options there are. Additional context A single slow request may not be worth an alert. But several slow requests will raise our average/p95 response times and are worth investigating. original GH issue link https://github.com/checkly/public-roadmap/issues/224

Tim Nolet 7 months ago

Alerting

Uptime % history on dashboards

Right now you can see up to 30 days back.It will be nice to have a 30,90,180,365 days option as we have a guaranteed SLA calculated per year.

Amir Jaron About 2 months ago

2

Status Pages

Mark/Delete failures as false-positive

We use Checkly statistics to monitor our uptime %.Sometimes we have failures that are false positives (since there was a bug in the test or specific to the test environment, and not in production, or just things didn't really break).We don't want to count those failures as downtime, hopefully with adding a note to why. Thanks!

Amir Jaron About 2 months ago

1

Insights

Create variables at deploy time for API checks

ScenarioI want to be able to set variables to be used on an API checkCurrent usageWe have a set up script to auth against our api which pulls in the correct key/secret values from variables saved on the group, This is done based on check nameIssueWe are removing groups so we can have finer grain control at the check level i.e. so group settings dont override check settingsIn removing groups we now have no way to deploy variables fo our API checksShort term we will manually add them into global variables via the gui but this absolutely should be deployable with npx checkly deploySolutionEither.....Be able to deploy global env vars via npx checkly deployorBe able to store variables on an API check as you can with browser checks

Jonathan-Dalton 2 months ago

1

CLI

Failed runs classified as & reported as degradations

One of our checks was failing intermittently when running from one location, so it was not flagged under failures. We only came across these failed runs when looking into the monitoring results of that particular check.Based on the above monitoring results, we see a clear degradation as whenever the check runs from a particular location first, it fails. It would be great if such failed runs are flagged as degradation of the test as, currently, there is no way for us to be aware of the situation unless we drill into the monitoring results.

Krishna 2 months ago

4

Insights

+1