Insights Rules | Yarken Help

Overview

Insights Rules enable proactive monitoring of spend and usage data by continuously scanning for unusual patterns, risks, and optimization opportunities. Instead of relying on manual reviews or static dashboards, Insights Rules automate anomaly detection and surface actionable signals as data is updated.

Each Insights Rule defines:

What to monitor — the metric and data source
How anomalies are detected — the detection method and thresholds
Who is notified — the notification channel or group
How often rule should run — hourly, daily, weekly, or monthly

Once configured and enabled:

Rules run automatically at a defined frequency
An Insight is generated when behavior falls outside the expected range
Insights appear on the Insights page for review, acknowledgment, investigation, and resolution

Insights Rules are designed to:

Scale across cloud and non‑cloud spend
Support multiple detection methods to accommodate stable, volatile, seasonal, and structurally changing data patterns.

Concepts

Understanding the following core concepts is essential for configuring and interpreting Insights Rules effectively:

Anomaly
A data point or pattern that deviates significantly from expected or historical behavior based on the selected detection method.
Baseline
The expected range of values learned from historical data, against which current values are compared to identify deviations.
Volatility
The degree of variation in a dataset over time (low, moderate, or high), which influences the choice of detection method.
Seasonality
Recurring, predictable patterns in the data (for example, month-end billing cycles or annual renewals).
False positive
An Insight triggered for behavior that appears abnormal but is actually expected or acceptable (for example, Q4 bonus incorrectly flagged as an anomaly).
Change point
A structural shift in the data that represents a new normal rather than a temporary spike or drop (example, spend increase following a new SaaS contract).
Cooldown days

Cooldown days define how long the system waits before generating a new Insight for the same condition after one has already been triggered.

How Insights Rules work

Insights Rules follow a consistent lifecycle from configuration to resolution.

1. Rule configuration

An admin defines the rule by selecting:

Data source — Spend Cube or Cloud Cube
Metric to monitor
Detection method
Dimensions and filters
Thresholds, severity, and notification settings
Processing frequency and cooldown period

2. Automated execution

The rule runs automatically based on the configured schedule and analyzes incoming and historical data.

3. Anomaly evaluation

The selected detection method compares observed values against the learned baseline or defined thresholds.

4. Insight generation

If abnormal behavior is detected, an Insight is created and displayed in the Insights page for review.

5. Insight lifecycle management

Each Insight moves through defined statuses:
New → Acknowledged → Resolved
Insights can be reopened if additional investigation is required.

Anomaly detection methods

The Insights feature supports multiple detection methods. Each method works differently and is suited for specific types of cost or usage data.

Method	Description	Suitable for	Why use it	Limitations
Z-Score - Statistical anomaly detection	Measures how far a data point deviates from the mean, expressed in standard deviations. Values that exceed a defined threshold (for example, > 3σ) are flagged as anomalies. Uses a rolling mean, typically over a ~12-month window. Flags points that are more than 2.5 standard deviations from the rolling mean.	Stable datasets with predictable patterns. Examples: Labor, hardware, facilities, and power cost pools	Simple and effective for detecting sharp spikes or sudden drops	Sensitive to natural variance and may over-flag anomalies in noisy data Does not account for seasonality (for example, Q4 bonuses) or recent trend shifts
Moving Average - Trend detection	Smooths data over a rolling window (for example, 7 days) to establish a baseline Flags deviations above or below the moving average Uses a rolling window of ~7 months Typically flags deviations of 22–25% from the rolling average	Data with short-term fluctuations but a clear overall trend Examples: Moderately volatile cost pools such as Software, Outside Services, and Telecom	Captures gradual changes and emerging trends Filters out minor noise Fast and TBM-friendly	Can miss long-term seasonality Less effective with irregular or highly variable patterns
Prophet - Seasonal forecasting	Uses forecasting models to account for seasonality, trends, and holidays. Predicts expected values and flags deviations using 95% confidence intervals.	Datasets with clear seasonality Examples: Weekday vs. weekend usage, month-end billing patterns	Handles complex seasonality effectively Adapts over time and reduces false positives Supports holiday effects (for example, Q4 spikes)	Requires sufficient historical data May overfit if data is highly irregular Computationally intensive and less intuitive for some stakeholders
CUSUM - Change point detection	Uses a cumulative sum method to detect shifts in the mean level of a time series Flags structural changes when deviations exceed a defined threshold (for example, ~2 standard deviations)	Identifying sudden, sustained shifts in baseline behavior Examples: Service adoption changes, software, outside services, telecom (primary), hardware and facilities (secondary, for policy-driven changes), contract-driven shifts	Effective at detecting persistent changes, not just one-time spikes Well-suited for structural cost changes (for example, software spend jumping to a new baseline after SaaS rollout) Fast and TBM-friendly	May not detect small, temporary, or short-lived spikes Best for long-term structural changes rather than transient anomalies

Anomaly detection categories

Insights detects multiple categories of anomalies in your data source and the detection method used depends on your data characteristics. Each category highlights a specific type of deviation.

Anomaly category	Description	Example
Spend spike or drop	Sudden increase or decrease in spend compared to the baseline.	AWS spend increased by 80% yesterday.
Usage spike or drop	Unexpected short-term rise or fall in resource usage.	S3 usage doubled last week.
Forecast deviation	Actual spend or usage deviates significantly from the forecasted value.	Actual spend is 45% higher than forecast.
Budget overrun	Spend exceeds the allocated budget threshold.	Team A reached 90% of the monthly budget by day ten.
Zero usage resource	Resource incurs cost while showing no usage.	Twenty instances incurred cost but were unused.
Tagging violation	Spend detected on resources with missing or incorrect tags.	Spend identified on untagged production resources.
New vendor detected	Spend appears from a vendor not present in historical data.	First recorded spend on Datadog.
Seasonality drift	Data deviates from the normal seasonal pattern.	Weekday costs suddenly match weekend levels.
Change point detected	Permanent shift in baseline spend or usage rather than a temporary spike.	Daily network costs double after a new service launch and remain elevated.

Note: To detect the same anomaly category using different detection methods (for example, Spend Spike using both Z-score and Threshold), create separate rules.

When to use which detection method

Note: Detection methods should be selected only after confirming that the prerequisites for creating Insight Rules are met, including sufficient historical data, consistent data availability, and appropriate metric selection. Using a method that does not align with data readiness may lead to unreliable Insights.

Selecting the appropriate detection method is key to reducing noise and avoiding false positives. The method you choose should align with how your data behaves over time.

Data behavior	Recommended detection method
Low volatility, stable data	Z-Score to detect sudden spikes or drops; optionally use CUSUM (Change Point Detection) to identify long-term shifts
Moderate volatility with gradual trends	Moving Average to smooth fluctuations and detect trend changes; optionally complement with CUSUM
Strong seasonality	Prophet to account for recurring weekly, monthly, or quarterly patterns
Contract-driven or structural changes	CUSUM (Change Point Detection) to identify permanent shifts and establish a new baseline

Next step

Create an Insight Rule

Manage Insights Rules