Configurable Alert Debouncing for Degraded Response Time Thresholds

Problem Statement
Currently, when a user sets a "degraded" response time threshold (e.g., 10s) in Checkly and connects an alert channel, an alert is triggered every time a check exceeds this threshold. This can lead to alert fatigue, as a single transient issue or a brief period of instability may result in multiple, potentially redundant alerts being sent in rapid succession.

Proposed Solution
Introduce a configurable alert debouncing mechanism for degraded response time thresholds. This feature would allow users to specify that an alert for degraded performance should only be sent after a defined number (x) of consecutive or cumulative degraded check results, rather than after every individual occurrence.

Feature Details

• Allow users to set a parameter (e.g., "Send alert after X degraded checks") when configuring a degraded response time threshold.

• Support both consecutive and rolling window logic for degraded check counting:

  • Consecutive: Alert is sent only if X degraded results occur in a row.

  • Rolling window: Alert is sent if X degraded results occur within the last Y check runs or time window.

• Reset the counter after an alert is sent, or provide an option to suppress further alerts until a recovery condition is met (e.g., a check returns to "OK" status).

• Optionally, provide visibility in the UI or via API to see current degraded check counts and pending alert status.

User Value

• Reduces alert noise and fatigue, especially during transient or intermittent issues.

• Allows teams to focus on actionable incidents rather than being overwhelmed by redundant alerts.

• Increases flexibility and control over alerting behavior, aligning with best practices in incident management and observability.

Example Scenario
A user sets a degraded threshold of 10s and configures "Send alert after 3 consecutive degraded checks." If three checks in a row exceed 10s, a single alert is triggered. If only one or two checks are degraded, no alert is sent, reducing unnecessary notifications for minor blips.