# 2. AI Prediction Configuration — Test Cases

User Type: **Super Administrator**
Source: *Mi Digital Academy - Education CRM Features Document*
Spec: ai_prediction_configuration.md — every feature, sub-feature, and rule covered

## Test Execution Policy

- Zero tolerance: any deviation from the documented behavior is a defect.
- Every failed test is logged with a Bug ID, the feature, the sub-feature, the expected vs actual result, and the severity; 100% of bugs are fixed before the group passes.
- 100% pass rate is required for the group to be marked complete.

## Coverage Matrix

| Feature | Sub-feature / Rule | Test IDs |
|---------|--------------------|----------|
| 2.1 Prediction Algorithm Settings | Algorithm selection (regression, ensemble, per-exam-type) | TC-SA-TE-10-001 |
| 2.1 Prediction Algorithm Settings | Algorithm parameters (weights, decay, smoothing) | TC-SA-TE-10-002 |
| 2.1 Prediction Algorithm Settings | Per-exam-type algorithm overrides | TC-SA-TE-10-003 |
| 2.1 Prediction Algorithm Settings | Algorithm version control (active, rollback) | TC-SA-TE-10-004 |
| 2.1 Prediction Algorithm Settings | Rule: no parameters → documented defaults | TC-SA-TE-10-005 |
| 2.1 Prediction Algorithm Settings | Rule: override precedence over global | TC-SA-TE-10-006 |
| 2.1 Prediction Algorithm Settings | Rule: rollback restores parameters exactly | TC-SA-TE-10-007 |
| 2.1 Prediction Algorithm Settings | Rule: in-use version not deleted | TC-SA-TE-10-008 |
| 2.1 Prediction Algorithm Settings | Audit logging of prediction algorithm settings | TC-SA-TE-10-009 |
| 2.2 Data Inputs for Prediction | Mock test data inclusion | TC-SA-TE-11-001 |
| 2.2 Data Inputs for Prediction | Practice data inclusion | TC-SA-TE-11-002 |
| 2.2 Data Inputs for Prediction | Attendance data inclusion | TC-SA-TE-11-003 |
| 2.2 Data Inputs for Prediction | Data freshness requirement | TC-SA-TE-11-004 |
| 2.2 Data Inputs for Prediction | Rule: no data → omitted, not zero-filled | TC-SA-TE-11-005 |
| 2.2 Data Inputs for Prediction | Rule: older than freshness → excluded | TC-SA-TE-11-006 |
| 2.2 Data Inputs for Prediction | Rule: insufficient data → no prediction, prompt | TC-SA-TE-11-007 |
| 2.2 Data Inputs for Prediction | Rule: change applies to new predictions | TC-SA-TE-11-008 |
| 2.2 Data Inputs for Prediction | Audit logging of data inputs for prediction | TC-SA-TE-11-009 |
| 2.3 Confidence Interval Configuration | Confidence interval width | TC-SA-TE-12-001 |
| 2.3 Confidence Interval Configuration | Confidence display format | TC-SA-TE-12-002 |
| 2.3 Confidence Interval Configuration | Low-confidence threshold | TC-SA-TE-12-003 |
| 2.3 Confidence Interval Configuration | Minimum data points for a prediction | TC-SA-TE-12-004 |
| 2.3 Confidence Interval Configuration | Rule: below minimum → not shown, prompt | TC-SA-TE-12-005 |
| 2.3 Confidence Interval Configuration | Rule: low-confidence always labeled | TC-SA-TE-12-006 |
| 2.3 Confidence Interval Configuration | Rule: width of zero rejected | TC-SA-TE-12-007 |
| 2.3 Confidence Interval Configuration | Rule: format change applies to new predictions | TC-SA-TE-12-008 |
| 2.3 Confidence Interval Configuration | Audit logging of confidence interval configuration | TC-SA-TE-12-009 |
| 2.4 Real-Time Update Frequency | Update frequency (real-time, hourly, daily, per-event) | TC-SA-TE-13-001 |
| 2.4 Real-Time Update Frequency | Event-triggered updates | TC-SA-TE-13-002 |
| 2.4 Real-Time Update Frequency | Update batch size | TC-SA-TE-13-003 |
| 2.4 Real-Time Update Frequency | Update window (quiet hours) | TC-SA-TE-13-004 |
| 2.4 Real-Time Update Frequency | Rule: event-triggered precedence | TC-SA-TE-13-005 |
| 2.4 Real-Time Update Frequency | Rule: batch split across runs | TC-SA-TE-13-006 |
| 2.4 Real-Time Update Frequency | Rule: quiet window defers to end | TC-SA-TE-13-007 |
| 2.4 Real-Time Update Frequency | Rule: failed batch retried, last good retained | TC-SA-TE-13-008 |
| 2.4 Real-Time Update Frequency | Audit logging of real-time update frequency | TC-SA-TE-13-009 |
| 2.5 Prediction Accuracy Tracking & Calibration | Accuracy tracking (predicted vs actual) | TC-SA-TE-14-001 |
| 2.5 Prediction Accuracy Tracking & Calibration | Calibration curve | TC-SA-TE-14-002 |
| 2.5 Prediction Accuracy Tracking & Calibration | Drift detection | TC-SA-TE-14-003 |
| 2.5 Prediction Accuracy Tracking & Calibration | Calibration report (per exam type, cohort) | TC-SA-TE-14-004 |
| 2.5 Prediction Accuracy Tracking & Calibration | Rule: no actual result → excluded | TC-SA-TE-14-005 |
| 2.5 Prediction Accuracy Tracking & Calibration | Rule: drift beyond threshold → alert | TC-SA-TE-14-006 |
| 2.5 Prediction Accuracy Tracking & Calibration | Rule: insufficient exams → "insufficient data" | TC-SA-TE-14-007 |
| 2.5 Prediction Accuracy Tracking & Calibration | Rule: never backfilled with estimates | TC-SA-TE-14-008 |
| 2.5 Prediction Accuracy Tracking & Calibration | Audit logging of prediction accuracy tracking & calibration | TC-SA-TE-14-009 |

## 2.1 Prediction Algorithm Settings

### TC-SA-TE-10-001 — Algorithm selection (regression, ensemble, per-exam-type)
**Type:** Positive
**Covers:** 2.1 → Algorithm selection
**Preconditions:** A Super Admin account is active; the Target Exam feature is enabled.
**Steps:**
1. As a Super Admin, open Target Exam Management → AI Prediction → Algorithm and select the algorithm (regression, ensemble, per-exam-type).
2. Verify predictions are generated by the selected algorithm.
**Expected Result:** Algorithm selection — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-10-002 — Algorithm parameters (weights, decay, smoothing)
**Type:** Positive
**Covers:** 2.1 → Algorithm parameters
**Preconditions:** An algorithm is selected.
**Steps:**
1. Set the algorithm parameters (weights, decay, smoothing).
2. Verify predictions use the configured parameters.
**Expected Result:** Algorithm parameters — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-10-003 — Per-exam-type algorithm overrides
**Type:** Positive
**Covers:** 2.1 → Per-exam-type overrides
**Preconditions:** A global algorithm is configured.
**Steps:**
1. Define a per-exam-type algorithm override.
2. Verify predictions for that exam type use the override.
**Expected Result:** Per-exam-type algorithm overrides — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-10-004 — Algorithm version control (active, rollback)
**Type:** Positive
**Covers:** 2.1 → Version control
**Preconditions:** Multiple algorithm versions exist.
**Steps:**
1. Set the active algorithm version and verify new predictions use it.
2. Roll back to a previous version and verify new predictions use it.
**Expected Result:** Algorithm version control — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-10-005 — Rule: no parameters → documented defaults
**Type:** Edge
**Covers:** 2.1 → Rule: defaults
**Preconditions:** An algorithm is selected with no parameters set.
**Steps:**
1. Generate a prediction.
2. Verify the algorithm uses the documented defaults.
**Expected Result:** An algorithm with no parameters set uses the documented defaults — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-10-006 — Rule: override precedence over global
**Type:** Negative
**Covers:** 2.1 → Rule: precedence
**Preconditions:** A per-exam-type override and a global algorithm exist.
**Steps:**
1. Generate a prediction for the exam type with the override.
2. Verify the per-exam-type override takes precedence over the global algorithm.
**Expected Result:** A per-exam-type override takes precedence over the global algorithm — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-10-007 — Rule: rollback restores parameters exactly
**Type:** Negative
**Covers:** 2.1 → Rule: rollback fidelity
**Preconditions:** A rollback is performed to a previous version.
**Steps:**
1. Roll back to the previous version.
2. Verify the previous version's parameters are restored exactly.
**Expected Result:** A rollback restores the previous version's parameters exactly — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-10-008 — Rule: in-use version not deleted
**Type:** Negative
**Covers:** 2.1 → Rule: in-use protection
**Preconditions:** An algorithm version is in use by active predictions.
**Steps:**
1. Attempt to delete the in-use version.
2. Verify it is not deleted.
**Expected Result:** An algorithm version in use by active predictions is not deleted — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-10-009 — Audit logging of prediction algorithm settings
**Type:** Positive
**Covers:** 2.1 → Audit logging
**Preconditions:** Prediction algorithm setting changes have been made.
**Steps:**
1. Open the audit trail and filter by prediction algorithm settings.
2. Verify entries show the setting, the change, and the timestamp.
**Expected Result:** Audit logging of prediction algorithm settings — delivered exactly as documented.
**Priority:** Critical

## 2.2 Data Inputs for Prediction

### TC-SA-TE-11-001 — Mock test data inclusion
**Type:** Positive
**Covers:** 2.2 → Mock test data
**Preconditions:** A Super Admin is configuring prediction data inputs.
**Steps:**
1. Open AI Prediction → Data Inputs and configure the mock test data inclusion (all, recent N, weighted by recency).
2. Verify predictions incorporate the configured mock test data.
**Expected Result:** Mock test data inclusion — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-11-002 — Practice data inclusion
**Type:** Positive
**Covers:** 2.2 → Practice data
**Preconditions:** Data inputs are being configured.
**Steps:**
1. Configure the practice data inclusion (quizzes, SRS recall, assignments).
2. Verify predictions incorporate the configured practice data.
**Expected Result:** Practice data inclusion — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-11-003 — Attendance data inclusion
**Type:** Positive
**Covers:** 2.2 → Attendance data
**Preconditions:** Data inputs are being configured.
**Steps:**
1. Configure the attendance data inclusion (live sessions, plan adherence).
2. Verify predictions incorporate the configured attendance data.
**Expected Result:** Attendance data inclusion — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-11-004 — Data freshness requirement
**Type:** Positive
**Covers:** 2.2 → Data freshness
**Preconditions:** Data inputs are being configured.
**Steps:**
1. Set the data freshness requirement (max age of input data).
2. Verify predictions enforce the freshness requirement.
**Expected Result:** Data freshness requirement — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-11-005 — Rule: no data → omitted, not zero-filled
**Type:** Negative
**Covers:** 2.2 → Rule: no-data omission
**Preconditions:** A data source has no data for a student.
**Steps:**
1. Generate a prediction for the student.
2. Verify the source is omitted (not zero-filled).
**Expected Result:** A data source with no data for a student is omitted (not zero-filled) — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-11-006 — Rule: older than freshness → excluded
**Type:** Negative
**Covers:** 2.2 → Rule: freshness enforcement
**Preconditions:** A data source is older than the freshness requirement.
**Steps:**
1. Generate a prediction.
2. Verify the stale source is excluded.
**Expected Result:** A data source older than the freshness requirement is excluded — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-11-007 — Rule: insufficient data → no prediction, prompt
**Type:** Negative
**Covers:** 2.2 → Rule: minimum data
**Preconditions:** A student has insufficient data (below the minimum).
**Steps:**
1. Generate a prediction for the student.
2. Verify no prediction is produced and the student is prompted to practice.
**Expected Result:** A student with insufficient data (below the minimum) gets no prediction (prompted to practice) — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-11-008 — Rule: change applies to new predictions
**Type:** Negative
**Covers:** 2.2 → Rule: change scope
**Preconditions:** A data input change is made; existing predictions exist.
**Steps:**
1. Change a data input.
2. Verify the change applies to new predictions (not retroactively).
**Expected Result:** A data input change applies to new predictions (not retroactively) — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-11-009 — Audit logging of data inputs for prediction
**Type:** Positive
**Covers:** 2.2 → Audit logging
**Preconditions:** Data input changes have been made.
**Steps:**
1. Open the audit trail and filter by data inputs for prediction.
2. Verify entries show the setting, the change, and the timestamp.
**Expected Result:** Audit logging of data inputs for prediction — delivered exactly as documented.
**Priority:** Critical

## 2.3 Confidence Interval Configuration

### TC-SA-TE-12-001 — Confidence interval width
**Type:** Positive
**Covers:** 2.3 → Interval width
**Preconditions:** A Super Admin is configuring confidence intervals.
**Steps:**
1. Open AI Prediction → Confidence Intervals and set the confidence interval width (e.g., ±5, ±10 points).
2. Verify predictions display the configured width.
**Expected Result:** Confidence interval width — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-12-002 — Confidence display format
**Type:** Positive
**Covers:** 2.3 → Display format
**Preconditions:** Confidence intervals are configured.
**Steps:**
1. Choose the confidence display format (range, band, high/medium/low label).
2. Verify predictions display confidence per the format.
**Expected Result:** Confidence display format — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-12-003 — Low-confidence threshold
**Type:** Positive
**Covers:** 2.3 → Low-confidence threshold
**Preconditions:** Confidence intervals are configured.
**Steps:**
1. Define the low-confidence threshold.
2. Verify predictions below the threshold are flagged.
**Expected Result:** Low-confidence threshold — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-12-004 — Minimum data points for a prediction
**Type:** Positive
**Covers:** 2.3 → Minimum data points
**Preconditions:** Confidence intervals are configured.
**Steps:**
1. Set the minimum data points for a prediction.
2. Verify the minimum is enforced.
**Expected Result:** Minimum data points for a prediction — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-12-005 — Rule: below minimum → not shown, prompt
**Type:** Negative
**Covers:** 2.3 → Rule: minimum gate
**Preconditions:** A student is below the minimum data points.
**Steps:**
1. View the prediction for the student.
2. Verify the prediction is not shown and the student is prompted to practice.
**Expected Result:** A prediction below the minimum data points is not shown (prompt to practice) — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-12-006 — Rule: low-confidence always labeled
**Type:** Negative
**Covers:** 2.3 → Rule: labeling
**Preconditions:** A prediction is below the low-confidence threshold.
**Steps:**
1. View the low-confidence prediction.
2. Verify it is always labeled, never shown as precise.
**Expected Result:** A low-confidence prediction is always labeled, never shown as precise — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-12-007 — Rule: width of zero rejected
**Type:** Negative
**Covers:** 2.3 → Rule: zero width
**Preconditions:** A Super Admin is setting the interval width.
**Steps:**
1. Enter a confidence interval width of zero.
2. Verify it is rejected (a range is always shown).
**Expected Result:** A confidence interval width of zero is rejected (a range is always shown) — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-12-008 — Rule: format change applies to new predictions
**Type:** Edge
**Covers:** 2.3 → Rule: format scope
**Preconditions:** A confidence display format change is made.
**Steps:**
1. Change the display format.
2. Verify the change applies to all new predictions.
**Expected Result:** A confidence display format change applies to all new predictions — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-12-009 — Audit logging of confidence interval configuration
**Type:** Positive
**Covers:** 2.3 → Audit logging
**Preconditions:** Confidence interval changes have been made.
**Steps:**
1. Open the audit trail and filter by confidence interval configuration.
2. Verify entries show the setting, the change, and the timestamp.
**Expected Result:** Audit logging of confidence interval configuration — delivered exactly as documented.
**Priority:** Critical

## 2.4 Real-Time Update Frequency

### TC-SA-TE-13-001 — Update frequency (real-time, hourly, daily, per-event)
**Type:** Positive
**Covers:** 2.4 → Update frequency
**Preconditions:** A Super Admin is configuring update frequency.
**Steps:**
1. Open AI Prediction → Update Frequency and set the update frequency (real-time, hourly, daily, per-event).
2. Verify predictions update per the frequency.
**Expected Result:** Update frequency — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-13-002 — Event-triggered updates
**Type:** Positive
**Covers:** 2.4 → Event-triggered updates
**Preconditions:** Update frequency is configured.
**Steps:**
1. Configure the event-triggered updates (mock test completed, plan milestone).
2. Trigger an event and verify the prediction updates.
**Expected Result:** Event-triggered updates — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-13-003 — Update batch size
**Type:** Positive
**Covers:** 2.4 → Batch size
**Preconditions:** Update frequency is configured.
**Steps:**
1. Set the update batch size (students per batch).
2. Verify updates run in batches of the configured size.
**Expected Result:** Update batch size — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-13-004 — Update window (quiet hours)
**Type:** Positive
**Covers:** 2.4 → Update window
**Preconditions:** Update frequency is configured.
**Steps:**
1. Set the update window (quiet hours to avoid peak load).
2. Verify updates respect the window.
**Expected Result:** Update window — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-13-005 — Rule: event-triggered precedence
**Type:** Negative
**Covers:** 2.4 → Rule: precedence
**Preconditions:** An event-triggered update and a scheduled frequency both apply.
**Steps:**
1. Trigger an event.
2. Verify the event-triggered update takes precedence over the scheduled frequency.
**Expected Result:** An event-triggered update takes precedence over the scheduled frequency — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-13-006 — Rule: batch split across runs
**Type:** Edge
**Covers:** 2.4 → Rule: batch split
**Preconditions:** An update batch exceeds the batch size.
**Steps:**
1. Run an update with more students than the batch size.
2. Verify the batch is split across multiple runs.
**Expected Result:** A batch exceeding the batch size is split across multiple runs — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-13-007 — Rule: quiet window defers to end
**Type:** Edge
**Covers:** 2.4 → Rule: quiet window
**Preconditions:** An update is due during the quiet window.
**Steps:**
1. Let an update fall due during the quiet window.
2. Verify it is deferred to the window end.
**Expected Result:** An update during the quiet window is deferred to the window end — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-13-008 — Rule: failed batch retried, last good retained
**Type:** Negative
**Covers:** 2.4 → Rule: failure handling
**Preconditions:** An update batch fails.
**Steps:**
1. Let an update batch fail.
2. Verify it is retried and the last good prediction is retained.
**Expected Result:** A failed update batch is retried; the last good prediction is retained — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-13-009 — Audit logging of real-time update frequency
**Type:** Positive
**Covers:** 2.4 → Audit logging
**Preconditions:** Update frequency changes have been made.
**Steps:**
1. Open the audit trail and filter by real-time update frequency.
2. Verify entries show the setting, the change, and the timestamp.
**Expected Result:** Audit logging of real-time update frequency — delivered exactly as documented.
**Priority:** Critical

## 2.5 Prediction Accuracy Tracking & Calibration

### TC-SA-TE-14-001 — Accuracy tracking (predicted vs actual)
**Type:** Positive
**Covers:** 2.5 → Accuracy tracking
**Preconditions:** A Super Admin is viewing accuracy; completed exams exist.
**Steps:**
1. Open AI Prediction → Accuracy & Calibration.
2. Verify the predicted vs actual score comparison is shown per student/exam.
**Expected Result:** Accuracy tracking — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-14-002 — Calibration curve
**Type:** Positive
**Covers:** 2.5 → Calibration curve
**Preconditions:** Accuracy data exists.
**Steps:**
1. Inspect the calibration curve.
2. Verify it shows predicted probability vs observed frequency.
**Expected Result:** Calibration curve — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-14-003 — Drift detection
**Type:** Positive
**Covers:** 2.5 → Drift detection
**Preconditions:** Accuracy data over time exists.
**Steps:**
1. Check the drift detection.
2. Verify accuracy degradation over time is detected.
**Expected Result:** Drift detection — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-14-004 — Calibration report (per exam type, cohort)
**Type:** Positive
**Covers:** 2.5 → Calibration report
**Preconditions:** Accuracy data exists.
**Steps:**
1. Review the calibration report per exam type/cohort.
2. Verify the report is generated per exam type and cohort.
**Expected Result:** Calibration report — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-14-005 — Rule: no actual result → excluded
**Type:** Negative
**Covers:** 2.5 → Rule: exclusion
**Preconditions:** A student has no actual result yet.
**Steps:**
1. Review the accuracy tracking.
2. Verify the student is excluded from accuracy metrics.
**Expected Result:** A student with no actual result yet is excluded from accuracy tracking — delivered exactly as documented.
**Priority:** High

### TC-SA-TE-14-006 — Rule: drift beyond threshold → alert
**Type:** Negative
**Covers:** 2.5 → Rule: drift alert
**Preconditions:** Accuracy drifts beyond the threshold.
**Steps:**
1. Let accuracy drift beyond the threshold.
2. Verify an alert is raised (retraining recommended).
**Expected Result:** A drift beyond the threshold raises an alert (retraining recommended) — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-14-007 — Rule: insufficient exams → "insufficient data"
**Type:** Edge
**Covers:** 2.5 → Rule: insufficient data
**Preconditions:** A calibration report has insufficient completed exams.
**Steps:**
1. Generate the calibration report.
2. Verify it shows "insufficient data".
**Expected Result:** A calibration report with insufficient completed exams shows "insufficient data" — delivered exactly as documented.
**Priority:** Medium

### TC-SA-TE-14-008 — Rule: never backfilled with estimates
**Type:** Negative
**Covers:** 2.5 → Rule: no backfill
**Preconditions:** An accuracy metric has missing actuals.
**Steps:**
1. Review the accuracy metric.
2. Verify it is never backfilled with estimated actuals.
**Expected Result:** An accuracy metric is never backfilled with estimated actuals — delivered exactly as documented.
**Priority:** Critical

### TC-SA-TE-14-009 — Audit logging of prediction accuracy tracking & calibration
**Type:** Positive
**Covers:** 2.5 → Audit logging
**Preconditions:** Accuracy/calibration configuration changes have been made.
**Steps:**
1. Open the audit trail and filter by prediction accuracy tracking & calibration.
2. Verify entries show the setting, the change, and the timestamp.
**Expected Result:** Audit logging of prediction accuracy tracking & calibration — delivered exactly as documented.
**Priority:** Critical