# 5. Report Analytics — Test Cases

User Type: **Super Administrator**
Source: *Mi Digital Academy - Education CRM Features Document*
Spec: report_analytics.md — every feature, sub-feature, and rule covered

## Test Execution Policy

- Zero tolerance: any deviation from the documented behavior is a defect.
- Every failed test is logged with a Bug ID, the feature, the sub-feature, the expected vs actual result, and the severity; 100% of bugs are fixed before the group passes.
- 100% pass rate is required for the group to be marked complete.

## Coverage Matrix

| Feature | Sub-feature / Rule | Test IDs |
|---------|--------------------|----------|
| 5.1 Report Open Rates | Open rate per report type: the percentage of delivered reports opened | TC-SA-AR-05-001 |
| 5.1 Report Open Rates | Channel breakdown: open rates per channel (email, in-app, PDF) | TC-SA-AR-05-002 |
| 5.1 Report Open Rates | Segment breakdown: open rates per recipient segment | TC-SA-AR-05-003 |
| 5.1 Report Open Rates | Trend over time: open rate trends across periods | TC-SA-AR-05-004 |
| 5.1 Report Open Rates | Benchmark comparison: open rates vs. platform averages | TC-SA-AR-05-005 |
| 5.1 Report Open Rates | Rule: an open is counted once per recipient per report | TC-SA-AR-05-006 |
| 5.1 Report Open Rates | Rule: low-volume reports are marked "insufficient sample" | TC-SA-AR-05-007 |
| 5.1 Report Open Rates | Rule: PDF open rate is measured by download + view | TC-SA-AR-05-008 |
| 5.1 Report Open Rates | Rule: analytics access is event-logged | TC-SA-AR-05-009 |
| 5.2 Alert Response Times | Acknowledgement time: the time from delivery to acknowledgement | TC-SA-AR-05-010 |
| 5.2 Alert Response Times | Response time: the time from delivery to a recorded resolution action | TC-SA-AR-05-011 |
| 5.2 Alert Response Times | Per alert type and severity: response times broken down | TC-SA-AR-05-012 |
| 5.2 Alert Response Times | Escalation rate: the share of alerts that required escalation | TC-SA-AR-05-013 |
| 5.2 Alert Response Times | Unacknowledged share: the share of alerts never acknowledged | TC-SA-AR-05-014 |
| 5.2 Alert Response Times | Rule: undelivered alerts are excluded and counted separately | TC-SA-AR-05-015 |
| 5.2 Alert Response Times | Rule: response times are shown as medians | TC-SA-AR-05-016 |
| 5.2 Alert Response Times | Rule: analytics access is event-logged | TC-SA-AR-05-017 |
| 5.3 Engagement with Reports | Time spent: the average time spent per report | TC-SA-AR-05-018 |
| 5.3 Engagement with Reports | Sections viewed: which report sections are viewed | TC-SA-AR-05-019 |
| 5.3 Engagement with Reports | Actions taken: actions triggered from report content | TC-SA-AR-05-020 |
| 5.3 Engagement with Reports | Follow-through rate: the share of recommended actions completed | TC-SA-AR-05-021 |
| 5.3 Engagement with Reports | Per report type: engagement broken down by report type | TC-SA-AR-05-022 |
| 5.3 Engagement with Reports | Rule: time spent is capped per session | TC-SA-AR-05-023 |
| 5.3 Engagement with Reports | Rule: an action is attributed only within the attribution window | TC-SA-AR-05-024 |
| 5.3 Engagement with Reports | Rule: individual recipient-level data is not exposed | TC-SA-AR-05-025 |
| 5.3 Engagement with Reports | Rule: analytics access is event-logged | TC-SA-AR-05-026 |
| 5.4 Report Effectiveness Metrics | Engagement impact: the change in engagement after report delivery | TC-SA-AR-05-027 |
| 5.4 Report Effectiveness Metrics | Performance correlation: the correlation between delivery and performance change | TC-SA-AR-05-028 |
| 5.4 Report Effectiveness Metrics | Alert-to-improvement rate: the share of alerted students who improve | TC-SA-AR-05-029 |
| 5.4 Report Effectiveness Metrics | Program summary: the composite effectiveness score per program | TC-SA-AR-05-030 |
| 5.4 Report Effectiveness Metrics | Period comparison: effectiveness compared across periods | TC-SA-AR-05-031 |
| 5.4 Report Effectiveness Metrics | Rule: correlation is labeled as correlation, not causation | TC-SA-AR-05-032 |
| 5.4 Report Effectiveness Metrics | Rule: programs with insufficient history are marked "insufficient data" | TC-SA-AR-05-033 |
| 5.4 Report Effectiveness Metrics | Rule: composite score weights are configurable and versioned | TC-SA-AR-05-034 |
| 5.4 Report Effectiveness Metrics | Rule: analytics access is event-logged | TC-SA-AR-05-035 |

### TC-SA-AR-05-001 — Open rate per report type: the percentage of delivered reports opened
**Type:** Positive
**Covers:** 5.1 → Open rate per report type: the percentage of delivered reports opened
**Preconditions:** A Super Admin account is active; reports have been delivered and some opened.
**Steps:**
1. As a Super Admin, open Report Analytics → Report Open Rates.
2. Observe the result and verify the full behavior: the open rate per report type is shown as a percentage of delivered reports.
**Expected Result:** The open rate per report type is shown — the percentage of delivered reports opened, delivered exactly as documented.
**Priority:** Critical

### TC-SA-AR-05-002 — Channel breakdown: open rates per channel (email, in-app, PDF)
**Type:** Positive
**Covers:** 5.1 → Channel breakdown: open rates per channel (email, in-app, PDF)
**Preconditions:** A Super Admin account is active; reports were delivered on multiple channels.
**Steps:**
1. As a Super Admin, open the channel breakdown.
2. Observe the result and verify the full behavior: open rates are shown per channel (email, in-app, PDF).
**Expected Result:** The channel breakdown is shown — open rates per channel are visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-003 — Segment breakdown: open rates per recipient segment
**Type:** Positive
**Covers:** 5.1 → Segment breakdown: open rates per recipient segment
**Preconditions:** A Super Admin account is active; reports were delivered to multiple segments.
**Steps:**
1. As a Super Admin, filter the open rates by segment.
2. Observe the result and verify the full behavior: open rates are shown per recipient segment.
**Expected Result:** The segment breakdown is shown — open rates per segment are visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-004 — Trend over time: open rate trends across periods
**Type:** Positive
**Covers:** 5.1 → Trend over time: open rate trends across periods
**Preconditions:** A Super Admin account is active; open rate data exists for multiple periods.
**Steps:**
1. As a Super Admin, view the open rate trend across periods.
2. Observe the result and verify the full behavior: the trend is shown across the standard period boundaries.
**Expected Result:** The trend is shown — open rates across periods are visible, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-005 — Benchmark comparison: open rates vs. platform averages
**Type:** Positive
**Covers:** 5.1 → Benchmark comparison: open rates vs. platform averages
**Preconditions:** A Super Admin account is active; platform average open rates are computed.
**Steps:**
1. As a Super Admin, compare a report type's open rate against the platform average.
2. Observe the result and verify the full behavior: the comparison is shown with both values.
**Expected Result:** The benchmark comparison is shown — the report type's rate vs. the platform average, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-006 — Rule: an open is counted once per recipient per report
**Type:** Positive
**Covers:** 5.1 → Rule: an open is counted once per recipient per report; repeated opens do not inflate the rate
**Preconditions:** A Super Admin account is active; a recipient opened the same report three times.
**Steps:**
1. As a Super Admin, view the open rate for that report.
2. Observe the result and verify the full behavior: the recipient counts as one open.
**Expected Result:** The open is counted once — repeated opens do not inflate the rate, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-007 — Rule: low-volume reports are marked "insufficient sample"
**Type:** Positive
**Covers:** 5.1 → Rule: open rates require a minimum delivery count; low-volume reports are marked "insufficient sample"
**Preconditions:** A Super Admin account is active; a report type was delivered to only 3 recipients.
**Steps:**
1. As a Super Admin, view the open rate for the low-volume report type.
2. Observe the result and verify the full behavior: the report type is marked "insufficient sample".
**Expected Result:** The low-volume report is marked "insufficient sample" — no false rate is shown, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-008 — Rule: PDF open rate is measured by download + view
**Type:** Positive
**Covers:** 5.1 → Rule: PDF open rate is measured by download + view, not just download
**Preconditions:** A Super Admin account is active; some PDFs were downloaded but not viewed.
**Steps:**
1. As a Super Admin, view the PDF channel open rate.
2. Observe the result and verify the full behavior: only downloaded-and-viewed PDFs count as opens.
**Expected Result:** The PDF open rate uses download + view — download-only does not count, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-009 — Rule: analytics access is event-logged
**Type:** Positive
**Covers:** 5.1 → event logging (action performed)
**Preconditions:** A Super Admin account is active; the event log is accessible.
**Steps:**
1. As a Super Admin, open the Report Open Rates view.
2. Open the event log.
3. Observe the result and verify the full behavior: the access is recorded with the actor and timestamp.
**Expected Result:** The analytics access is event-logged — the actor and timestamp are recorded, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-010 — Acknowledgement time: the time from delivery to acknowledgement
**Type:** Positive
**Covers:** 5.2 → Acknowledgement time: the time from alert delivery to acknowledgement
**Preconditions:** A Super Admin account is active; alerts have been delivered and acknowledged.
**Steps:**
1. As a Super Admin, open Alert Response Times.
2. Observe the result and verify the full behavior: the acknowledgement time is shown as the time from delivery to acknowledgement.
**Expected Result:** The acknowledgement time is shown — measured from delivery to acknowledgement, delivered exactly as documented.
**Priority:** Critical

### TC-SA-AR-05-011 — Response time: the time from delivery to a recorded resolution action
**Type:** Positive
**Covers:** 5.2 → Response time: the time from alert delivery to a recorded resolution action
**Preconditions:** A Super Admin account is active; alerts have been resolved.
**Steps:**
1. As a Super Admin, open Alert Response Times.
2. Observe the result and verify the full behavior: the response time is shown as the time from delivery to resolution action.
**Expected Result:** The response time is shown — measured from delivery to resolution, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-012 — Per alert type and severity: response times broken down
**Type:** Positive
**Covers:** 5.2 → Per alert type and severity: response times broken down by type and severity
**Preconditions:** A Super Admin account is active; alerts of multiple types and severities exist.
**Steps:**
1. As a Super Admin, filter response times by alert type and severity.
2. Observe the result and verify the full behavior: response times are broken down by type and severity.
**Expected Result:** The breakdown is shown — response times per type and severity are visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-013 — Escalation rate: the share of alerts that required escalation
**Type:** Positive
**Covers:** 5.2 → Escalation rate: the share of alerts that required escalation
**Preconditions:** A Super Admin account is active; some alerts were escalated.
**Steps:**
1. As a Super Admin, view the escalation rate.
2. Observe the result and verify the full behavior: the share of alerts that required escalation is shown.
**Expected Result:** The escalation rate is shown — the share of escalated alerts is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-014 — Unacknowledged share: the share of alerts never acknowledged
**Type:** Positive
**Covers:** 5.2 → Unacknowledged share: the share of alerts never acknowledged
**Preconditions:** A Super Admin account is active; some alerts were never acknowledged past the cutoff.
**Steps:**
1. As a Super Admin, view the unacknowledged share.
2. Observe the result and verify the full behavior: the share of alerts never acknowledged (at the cutoff) is shown.
**Expected Result:** The unacknowledged share is shown — measured at the defined cutoff, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-015 — Rule: undelivered alerts are excluded and counted separately
**Type:** Positive
**Covers:** 5.2 → Rule: response time is measured from delivery; undelivered alerts are excluded and counted separately
**Preconditions:** A Super Admin account is active; some alerts failed delivery.
**Steps:**
1. As a Super Admin, view the response times.
2. Observe the result and verify the full behavior: undelivered alerts are excluded from response times and counted separately.
**Expected Result:** The undelivered alerts are excluded — they are counted separately, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-016 — Rule: response times are shown as medians
**Type:** Positive
**Covers:** 5.2 → Rule: response times are shown as medians to avoid outlier skew
**Preconditions:** A Super Admin account is active; response time data includes outliers.
**Steps:**
1. As a Super Admin, view the response times.
2. Observe the result and verify the full behavior: the values shown are medians, not means.
**Expected Result:** The medians are shown — response times are presented as medians, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-017 — Rule: analytics access is event-logged
**Type:** Positive
**Covers:** 5.2 → event logging (action performed)
**Preconditions:** A Super Admin account is active; the event log is accessible.
**Steps:**
1. As a Super Admin, open the Alert Response Times view.
2. Open the event log.
3. Observe the result and verify the full behavior: the access is recorded with the actor and timestamp.
**Expected Result:** The analytics access is event-logged — the actor and timestamp are recorded, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-018 — Time spent: the average time spent per report
**Type:** Positive
**Covers:** 5.3 → Time spent: the average time spent per report
**Preconditions:** A Super Admin account is active; recipients have spent time on reports.
**Steps:**
1. As a Super Admin, open Engagement with Reports.
2. Observe the result and verify the full behavior: the average time spent per report is shown.
**Expected Result:** The time spent is shown — the average per report is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-019 — Sections viewed: which report sections are viewed
**Type:** Positive
**Covers:** 5.3 → Sections viewed: which report sections are viewed
**Preconditions:** A Super Admin account is active; recipients have viewed report sections.
**Steps:**
1. As a Super Admin, view the sections viewed breakdown.
2. Observe the result and verify the full behavior: the viewed sections per report are shown.
**Expected Result:** The sections viewed are shown — per-section view data is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-020 — Actions taken: actions triggered from report content
**Type:** Positive
**Covers:** 5.3 → Actions taken: actions triggered from report content
**Preconditions:** A Super Admin account is active; recipients have taken actions from reports.
**Steps:**
1. As a Super Admin, view the actions taken.
2. Observe the result and verify the full behavior: the actions triggered from report content are shown.
**Expected Result:** The actions taken are shown — actions from report content are visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-021 — Follow-through rate: the share of recommended actions completed
**Type:** Positive
**Covers:** 5.3 → Follow-through rate: the share of recommended actions completed
**Preconditions:** A Super Admin account is active; reports contained trackable recommendations.
**Steps:**
1. As a Super Admin, view the follow-through rate.
2. Observe the result and verify the full behavior: the share of completed recommended actions is shown.
**Expected Result:** The follow-through rate is shown — the share of completed recommendations is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-022 — Per report type: engagement broken down by report type
**Type:** Positive
**Covers:** 5.3 → Per report type: engagement broken down by report type
**Preconditions:** A Super Admin account is active; multiple report types have engagement data.
**Steps:**
1. As a Super Admin, filter engagement by report type.
2. Observe the result and verify the full behavior: engagement metrics are broken down by report type.
**Expected Result:** The per-report-type breakdown is shown — engagement per type is visible, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-023 — Rule: time spent is capped per session
**Type:** Positive
**Covers:** 5.3 → Rule: time spent is capped per session to exclude abandoned tabs
**Preconditions:** A Super Admin account is active; a recipient left a report open for 8 hours.
**Steps:**
1. As a Super Admin, view the time spent for that report.
2. Observe the result and verify the full behavior: the session time is capped, not 8 hours.
**Expected Result:** The time spent is capped — the abandoned tab does not inflate the metric, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-024 — Rule: an action is attributed only within the attribution window
**Type:** Positive
**Covers:** 5.3 → Rule: an action is attributed to a report only when taken within the attribution window
**Preconditions:** A Super Admin account is active; a recipient took an action 10 days after viewing a report (window is 48 hours).
**Steps:**
1. As a Super Admin, view the actions taken for that report.
2. Observe the result and verify the full behavior: the late action is not attributed to the report.
**Expected Result:** The attribution window is enforced — the late action is not attributed, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-025 — Rule: individual recipient-level data is not exposed
**Type:** Negative
**Covers:** 5.3 → Rule: engagement data is aggregated; individual recipient-level data is not exposed in this view
**Preconditions:** A Super Admin account is active; the engagement view is open.
**Steps:**
1. As a Super Admin, inspect the engagement view for recipient-level detail.
2. Observe the result and verify the full behavior: only aggregated data is shown; no individual recipient-level data is exposed.
**Expected Result:** The data is aggregated — no individual recipient-level data is exposed, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-026 — Rule: analytics access is event-logged
**Type:** Positive
**Covers:** 5.3 → event logging (action performed)
**Preconditions:** A Super Admin account is active; the event log is accessible.
**Steps:**
1. As a Super Admin, open the Engagement with Reports view.
2. Open the event log.
3. Observe the result and verify the full behavior: the access is recorded with the actor and timestamp.
**Expected Result:** The analytics access is event-logged — the actor and timestamp are recorded, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-027 — Engagement impact: the change in engagement after report delivery
**Type:** Positive
**Covers:** 5.4 → Engagement impact: the change in recipient engagement after report delivery
**Preconditions:** A Super Admin account is active; a report program has delivery and engagement history.
**Steps:**
1. As a Super Admin, open Report Effectiveness Metrics.
2. Observe the result and verify the full behavior: the engagement impact (change after delivery) is shown per program.
**Expected Result:** The engagement impact is shown — the post-delivery engagement change is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-028 — Performance correlation: the correlation between delivery and performance change
**Type:** Positive
**Covers:** 5.4 → Performance correlation: the correlation between report delivery and performance change
**Preconditions:** A Super Admin account is active; a report program has performance history.
**Steps:**
1. As a Super Admin, view the performance correlation.
2. Observe the result and verify the full behavior: the correlation between delivery and performance change is shown.
**Expected Result:** The performance correlation is shown — the correlation metric is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-029 — Alert-to-improvement rate: the share of alerted students who improve
**Type:** Positive
**Covers:** 5.4 → Alert-to-improvement rate: the share of alerted students who improve after the alert
**Preconditions:** A Super Admin account is active; a critical alert program has post-alert performance history.
**Steps:**
1. As a Super Admin, view the alert-to-improvement rate.
2. Observe the result and verify the full behavior: the share of alerted students who improved within the window is shown.
**Expected Result:** The alert-to-improvement rate is shown — the share of improving students is visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-030 — Program summary: the composite effectiveness score per program
**Type:** Positive
**Covers:** 5.4 → Program summary: the composite effectiveness score per report/alert program
**Preconditions:** A Super Admin account is active; multiple programs have effectiveness data.
**Steps:**
1. As a Super Admin, view the program summary.
2. Observe the result and verify the full behavior: the composite effectiveness score is shown per program.
**Expected Result:** The composite score is shown — per-program effectiveness scores are visible, delivered exactly as documented.
**Priority:** High

### TC-SA-AR-05-031 — Period comparison: effectiveness compared across periods
**Type:** Positive
**Covers:** 5.4 → Period comparison: effectiveness compared across periods
**Preconditions:** A Super Admin account is active; effectiveness data exists for multiple periods.
**Steps:**
1. As a Super Admin, compare effectiveness across periods.
2. Observe the result and verify the full behavior: the period-over-period comparison is shown.
**Expected Result:** The period comparison is shown — effectiveness across periods is visible, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-032 — Rule: correlation is labeled as correlation, not causation
**Type:** Positive
**Covers:** 5.4 → Rule: correlation is not presented as causation; the metric is labeled as correlation
**Preconditions:** A Super Admin account is active; the performance correlation view is open.
**Steps:**
1. As a Super Admin, inspect the correlation metric's labeling.
2. Observe the result and verify the full behavior: the metric is labeled as correlation, with no causal language.
**Expected Result:** The labeling is correct — the metric is presented as correlation, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-033 — Rule: programs with insufficient history are marked "insufficient data"
**Type:** Positive
**Covers:** 5.4 → Rule: programs with insufficient history are marked "insufficient data" rather than scored
**Preconditions:** A Super Admin account is active; a new program has only 1 period of history.
**Steps:**
1. As a Super Admin, view the program summary for the new program.
2. Observe the result and verify the full behavior: the program is marked "insufficient data" with no score.
**Expected Result:** The new program is marked "insufficient data" — no false score is shown, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-034 — Rule: composite score weights are configurable and versioned
**Type:** Positive
**Covers:** 5.4 → Rule: the composite score weights are configurable by the Super Admin and versioned
**Preconditions:** A Super Admin account is active; the composite score weights are accessible.
**Steps:**
1. As a Super Admin, change the composite score weights and save.
2. Check the weight version history.
3. Observe the result and verify the full behavior: the new weights are applied and the change is versioned.
**Expected Result:** The weights are configurable and versioned — the change is applied and recorded, delivered exactly as documented.
**Priority:** Medium

### TC-SA-AR-05-035 — Rule: analytics access is event-logged
**Type:** Positive
**Covers:** 5.4 → event logging (action performed)
**Preconditions:** A Super Admin account is active; the event log is accessible.
**Steps:**
1. As a Super Admin, open the Report Effectiveness Metrics view.
2. Open the event log.
3. Observe the result and verify the full behavior: the access is recorded with the actor and timestamp.
**Expected Result:** The analytics access is event-logged — the actor and timestamp are recorded, delivered exactly as documented.
**Priority:** Medium