# Evidence Map product audit

**Original audit · 14 September 2026**

The defects below describe the pre-repair snapshot. See [Audit repairs](Audit_Repairs.md) for the current implementation and outstanding checks.

**Verdict:** a useful teacher-operated pilot with a substantial curriculum and planning foundation. It is not yet a production-ready student portal or a validated readiness assessment system. The next work should improve data reliability and educational validity before adding more dashboards.

The user requested an audit, textbook advice, a Student OS recommendation and a how-to website. This update builds the requested documentation and fabrication starter kit. It does not silently implement the larger architecture changes recommended below. Supplied documents were treated as evidence and historical proposals, not as new user instructions.

## What is worth preserving

- The original course outcomes, I-can statements, look-fors, task rubrics and Product / Observation / Conversation framework remain present.
- The newer skill library, dated evidence records, explicit teacher verification, retained diagnostic attempts, separate student claims and private seating exports are useful foundations.
- The consolidated planning catalogue and preserved Excel files make the previous two days of work recoverable and accessible.
- Readiness bands are described as local, revisable judgments. Automatic percentage conversion has been disabled. Missing evidence can remain blank.
- Studio writes report storage failures; Studio backup validation and preview are stronger than the original outcome-store equivalents.

## Findings, ordered by impact

### A1 · High · Legacy persistence can falsely report success

**Evidence:** `index.html:724` catches storage errors without returning failure. Full restore at `index.html:1646` onwards mutates state and writes several storage keys sequentially, with swallowed failures. The save-evidence handler at `index.html:1358` onwards also calls this save function before showing a saved message.

**Reproduction:** in an isolated browser with one synthetic learner, make `Storage.prototype.setItem` throw only for `tos_evidence_v1`. Import a valid full backup containing an updated evidence note. The UI says “Backup imported”; the persisted note remains the old value. This was reproduced on 14 September. See [audit-results.json](audit-results.json).

**Consequence:** in-memory data and stored data can disagree. A user can trust a success message and lose changes on reload. The synthetic test does not establish that any real records have already been lost.

**Repair:** use a result-returning persistence boundary for all stores. Prepare and validate the complete candidate state before changing active state. Prefer a single transactional record/database or a staged revision with one committed pointer. Do not display success or advance the learner until the required write succeeds. Preserve the original state and provide a recovery download on failure.

**Acceptance:** forced quota/write failures at every restore step leave the original durable state intact, show an explicit failure and retain the attempted data for recovery. Reload and backup comparisons must match.

### A2 · High · Full restore accepts malformed nested outcome evidence

**Evidence:** `index.html:1634` checks that each legacy cell has an evidence array and a valid level, but does not validate each evidence entry. The newer Studio schema checks are deeper.

**Reproduction:** change one existing cell’s evidence to `[null]` in a synthetic full backup. The import accepts it and persists `[null]`. No immediate page error occurred on the overview; the demonstrated defect is invalid stored data, not a claimed crash.

**Repair:** validate nested entries, methods, dates, criteria, learner/course/outcome references and compatible versions before any mutation. Offer a preview and a reject-with-reason response. Define how older valid backups migrate without discarding per-attempt details.

**Acceptance:** null entries, wrong field types, unknown cross-references and unsupported versions are rejected before any key changes. Genuine older backups still round-trip with complete history.

### A3 · High · Skills and outcomes are not one evidence system

**Evidence:** `studio.js` reads skills and diagnostics from `em_studio_v1`; the original application reads outcome evidence from `tos_evidence_v1`. New skill verification does not create a reviewed outcome judgment. Legacy optional skill fields do not establish a shared canonical evidence record.

**Consequence:** a teacher can record the same performance twice, with inconsistent dates, conditions or judgments. Student Progress and the skill matrix can disagree without a clear explanation.

**Repair:** introduce one evidence ID with links to learner, course, task, artifact, date, method and conditions. Attach separate teacher-reviewed skill and outcome judgments. Preserve a many-to-many mapping with source and reviewer metadata. A skill rating must not automatically produce a course mark.

**Acceptance:** one performance can support an identified skill and outcome without duplicate artifact entry. Each judgment can be traced back to its evidence and rubric version. Corrections preserve history. Existing IDs and records survive migration.

### A4 · High · Outcome authority remains unfinished

**Evidence:** the existing MM12 Sound module has seven legacy outcomes where the supplied official reference has five; the release flags the discrepancy rather than silently renumbering historical records. The original IBDS register includes 26 pending entries among 61 locally organised outcomes. Local syllabus organisation is not identical to official IB assessment criteria.

**Consequence:** a plausible-looking outcome code can be attached to the wrong expectation, especially if IDs are renumbered. Assessment validity depends on resolving this before reporting.

**Repair:** reconcile wording by meaning against the applicable official document. Store exact wording, source, edition/page, reviewed status and stable ID. Create an explicit old-to-new crosswalk with teacher approval per ambiguous mapping. Retain the original evidence association and provenance.

**Acceptance:** every outcome used for reporting has a reviewed source. Ambiguous historical evidence stays visibly unresolved instead of inheriting a guessed mapping. I-can statements and rubric criteria remain distinguishable from official wording.

### A5 · High for routine classroom use · Diagnostic validity and rehearsal are incomplete

**Evidence:** the Photoshop release explicitly remains a teacher-led pilot. Full A/B software rehearsal, genuine step captures and reopened PSD exemplars were not completed. `studio-core.js:14` uses the lowest of four ratings and an instructional-help ceiling as a local suggestion; `currentReadiness` uses a default 42-day review window and two independent dates for “Reviewed”. These are implementation choices, not validated Kagan scoring rules.

**Consequence:** a polished package can look more authoritative than the evidence supports. One global number can hide different skill needs. Retest improvement may reflect task familiarity as well as learning.

**Repair:** run both briefs on the actual classroom version, time them with learners, create anonymised permission-cleared anchors at criterion level, and compare independent teacher ratings. Examine assistance, access barriers and changed-task difficulty. Preserve the four-dimensional profile. Review the aggregation rule and expiry window using pilot evidence.

**Acceptance:** all files reopen, screenshots reflect the actual application, a learner can follow the core steps, the retest samples comparable skills under changed conditions, and disagreements between raters have documented resolutions. No claim of “validated Kagan test” or automated grade equivalence.

### A6 · Medium · Per-lesson fabrication prompt assumes Photoshop

**Evidence:** `studio.js:38`, `promptFor`, unconditionally inserts “Software: current Creative Cloud Photoshop” for lessons from every course.

**Consequence:** a DS reasoning lesson can be sent to a fabrication agent with an inappropriate software requirement. More books will not correct an incorrectly scoped job.

**Repair:** route the brief by course and actual task. For DS, require mechanism explanation, credible source analysis, stakeholder reasoning and an individual defence. Specify Photoshop only where image production is a required learning objective. Use an explicit source availability manifest.

**Delivered workaround:** the [fabrication starter kit](fabrication-agent-kit.zip) includes a course-aware prompt and a clear source hierarchy. The app’s existing prompt logic is unchanged in this documentation update.

**Acceptance:** each course’s exported brief contains appropriate tool and evidence requirements; a DS reasoning task does not inherit Photoshop instructions.

### A7 · High before student rollout · A profile page is not a student portal

**Evidence:** the current Student OS is a teacher-managed page in the same application. The standalone passport uses browser-local storage. There are no individual student logins, enforced course roles or central evidence submission service.

**Consequence:** sharing the teacher app or one browser profile does not isolate each learner’s records. Manual JSON exchange creates review and version-management work.

**Repair:** first build the shared evidence model, then authenticated course membership and server-enforced access controls. Student ownership of claims and teacher ownership of verification must be explicit. Keep private grouping information out of student-facing responses, not merely hidden by CSS.

**Acceptance:** a student account can access only its permitted records; teacher-only fields never appear in learner responses or exports. Submission, review and correction are traceable. Test cross-course and cross-student access boundaries.

### A8 · Conditional deployment risk · Publishing boundary and optional AI proxy need controls

**Evidence:** `netlify.toml` publishes `.`. Files placed in the project root can therefore become part of a deployed site. `netlify/functions/gemini.js` accepts POST requests, a caller-selected model and content; if a server key is configured, it calls the upstream service without application authentication, rate limits, payload limits or a model allowlist.

**Consequence:** placing private textbooks or rosters in this repository is unsuitable. A publicly enabled proxy can be used without the intended teacher authorization. This audit did not establish that a key is currently configured or that any private data has been exposed.

**Repair:** keep fabrication sources in a separate private project; publish a deliberate allowlisted output directory. Gate any public AI feature with access and usage controls, bounded inputs and an explicit data handling design. Keep it unconfigured until ready.

**Acceptance:** the deployment artifact contains only intended public assets. Unauthenticated requests cannot use the enabled AI service; oversized and unsupported requests fail before upstream calls.

### A9 · Medium · Planning, Excel and maintenance need clearer contracts

**Evidence:** the next-five list selects future untaught dates, so older untaught lessons disappear from that view. Planning edits do not propagate scheduling dependencies. Excel and app state do not synchronise. The app retains a large inline legacy implementation; some content-build scripts depend on a temporary intake location. This workspace has no Git history.

**Repair:** add a visible unscheduled/overdue queue, dependency review after rescheduling and explicit source/edited status. Choose a single authoritative working-record workflow. Document a reproducible build input manifest, adopt version control and separate persistence/domain logic from rendering in bounded changes.

**Acceptance:** an untaught lesson remains discoverable after its date; rescheduling identifies affected prerequisites; exports state what they contain and when created. A clean build can be reproduced from a documented private source pack without relying on a vanished temporary directory.

## Recommended sequence

1. Persistence and complete restore validation (A1–A2).
2. Authoritative outcome reconciliation and stable crosswalk (A4).
3. One shared evidence record, piloted with one learner/task/skill/outcome (A3).
4. Course-aware fabrication and diagnostic moderation in parallel with the teacher-led pilot (A5–A6).
5. Separate teacher and learner interfaces backed by authenticated access, before real student portal use (A7–A8).
6. Planning usability, reproducible builds and longer-term maintenance (A9).

The how-to site is the requested immediate usability improvement. It explains the existing workflows, exposes the outstanding limitations and keeps Excel downloads accessible. It does not substitute documentation for repair.

## Verification scope

The 12 existing Studio core test groups passed during this audit. Earlier release checks covered ordinary workflow routes, roster preservation, claim import/review, backup round-trips, seating privacy and mobile layouts. This audit added the two isolated failure checks above. See [guide verification](Guide_Verification.md) for the documentation-specific browser results.

No real learner data was used in the new tests, and the user’s existing browser state was not changed. No site was deployed and no live AI request was made. No independent psychometric validation, comprehensive penetration test or complete review of every textbook chapter is claimed.