Methods

How DormWay Studied Fall 2026 Course AI Rules

Research note accompanying DormWay’s Fall 2026 course-policy study

Riley
Founder & CEO•

Research note accompanying DormWay’s Fall 2026 course-policy study

DormWay conducted and sponsored this analysis using its own stored syllabus corpus. This note explains what was screened, how the reference sample was reviewed and what the results can support. The study uses AI-agent source review; it was not independently human-coded or externally peer reviewed.

The source snapshot was taken October 14, 2026, Pacific time. This is a single-term corpus study, not a matched comparison of policy changes over time.

Findings and their denominator

These are sample-corrected estimates for the eligible, policy-evaluable portion of a filtered DormWay transcript collection.

FindingEstimate95% sampling interval
Contains an operative AI-use policy75.3%73.2–77.4%
Default prohibition with exceptions or task-limited permission33.9%31.1–36.6%
Prohibits AI for assessed work without an operative assessed-work exception23.3%21.1–25.5%
Mixed or unclear permission regime11.7%9.2–14.2%

These measures overlap: policy presence includes several policy categories. They should not be drawn as slices of one pie. The 33.9% category does not mean every course requires advance approval. Assessed-work prohibitions can allow private study. Mixed/unclear includes incomplete or ambiguous scope; it is not a contradiction rate, a count of confused students or proof that an instructor gives impossible instructions. No policy found does not establish permission.

Building the candidate frame

The read-only snapshot contained 45,208 records labeled syllabus. The study isolated stored source transcription content from generated summaries and advice, required a linked direct upload, screened for source-body Fall/Autumn 2026 evidence and removed exact whitespace-normalized transcript duplicates. The resulting frame contained 12,920 candidate transcripts.

The unit is a distinct normalized stored transcript, not a unique course, instructor, original file or institution. Different transcriptions can refer to the same uploaded file. Missing, oversized and other frame-excluded records were not exhaustively re-extracted. The collection includes international courses and reflects material entering DormWay’s service; it is not a nationally representative sample of US colleges or students.

Automated screening

All 12,920 candidates received an output, with exact frame-ID/source-hash coverage. Of these, 9,466 were classified from AI-related source passages using a fixed Claude Sonnet 5 prompt, and 3,454 received a lexical no-hit screen. There were no missing outputs; 34 classifications abstained on policy evaluation.

All 87 records whose AI passages exceeded the original input cap received a complete-window pass with the same prompt and model. Failed or slow batch records were recovered through individual requests. No final input-cap truncations remained, but relevant context outside the AI-related passages can still be missed. Automated screening does not establish that all 12,920 candidates are genuine, complete Fall 2026 syllabi.

The random reference sample

A fixed seed selected 600 candidates by simple random sampling before policy outcomes were counted. All selections were retained; exclusions and uncertain records were not replaced. Three AI agents each reviewed 200 stored sources. Another agent independently reviewed an overlapping 120, with documented adjudication.

The locked sample contained 542 source-eligible records, of which 540 supported policy evaluation. The remaining dispositions were 45 source exclusions, 13 source-uncertain records and two eligible sources with insufficient policy evidence. The 540 evaluable sources included 404 with an operative policy and 136 without one.

Measure against the 540 evaluable reference casesResult95% Wilson interval
Policy-presence accuracy531/540 = 98.3%96.9–99.1%
Policy-positive precision396/397 = 99.7%98.6–100.0%
Policy-positive recall396/404 = 98.0%96.1–99.0%
Exact policy-category accuracy437/540 = 80.9%77.4–84.0%

Binary confusion counts were 396 true positives, 135 true negatives, one false positive and eight false negatives. Policy detection and precise category assignment are different tasks: the 98.3% result must not describe exact category accuracy. The precision interval’s upper bound rounds to 100.0%; its unrounded bound is below 100%.

These comparisons use an agent-reviewed reference, not human ground truth. Initial independent-agent category agreement of 95.0% on the overlap is a separate consistency measure. A post-outcome diagnostic found three missed policies among the 128 evaluable reference cases that used the lexical no-hit shortcut. Those errors are already included in overall accuracy and the aggregate correction; the diagnostic did not change the classifier or estimator.

Correcting the aggregate estimates

Raw automated output marked 8,865 of 12,920 candidates policy-positive, or 68.6%. That raw percentage includes source-ineligible or unresolved candidates and is not the reported eligible-domain prevalence.

A predeclared generalized-difference estimator uses the probability sample’s signed residuals to correct automated totals for observed positive and negative errors. Eligibility is estimated separately from the sample; the classifier does not establish source eligibility. The estimated eligible, evaluable total is 11,628 and the estimated operative-policy total is 8,757.3. Their ratio is 75.3%.

Intervals use simple-random-sampling without-replacement residual linearization with a finite-population correction. Category estimates use the same fixed method. The prompt and correction coefficients were not tuned to maximize reference agreement. Separately corrected exclusive categories sum to approximately 100.1%, reflecting their differing auxiliaries; totals were not silently normalized.

The 13 source-uncertain and two policy-unevaluable sample cases give extreme point sensitivity bounds of 73.3–76.0% if all are assumed eligible and respectively policy-negative or policy-positive. Those are sensitivity bounds, not confidence limits.

Original-file audit and sensitivity

Files were retrieved for all 600 sampled transcript units, representing 594 distinct byte hashes. All 584 available ingestion hashes matched downloaded bytes; 16 lacked a stored hash. Independent extraction yielded substantive body text for 582 records. Eighteen lacked usable independent body text. A hash match verifies file identity, not transcription completeness, and every page was not visually recoded.

One original PDF contained an AI policy omitted from its stored transcript. That record was already outside the primary 540 because term evidence was uncertain. Separate original-source adjudication added one evaluable prohibition for sensitivity analysis: 405 policy-positive cases out of 541. The primary reference remained unchanged. The corrected sensitivity estimate is 75.4% (73.3–77.4%), compared with the primary 75.3%.

Named public examples

These examples were separately checked on official Fall 2026 course sites. They were not added to the random sample’s denominator and do not establish a hypothetical student’s actual timetable. Course levels and learning objectives differ.

  • TAMIU ENGL 3307 prohibits AI throughout its creative-writing process. COMM 3332 encourages ideas, draft copy and imagery with attribution and verification while prohibiting quiz use and direct pasting of AI-generated body copy. Its Fall 2026 page retains one stale 2025 presentation-date sentence; no date inference here depends on it.
  • UT Austin CS 380I, page 5, permits programming and exam assistance under understanding and disclosure conditions. CS 439 requires explicit written permission for generated code/text solutions; it is not summarized here as a blanket ban on every possible AI use.
  • UT Austin CS 109 contains conditional instructor-authorized use alongside broader integrity restrictions. Specific authorization can reasonably be an exception; the study does not allege irreconcilable instructions or unfair enforcement.
  • TAMIU ACC 4370 states course failure and Honor Council referral for breach of its AI restriction. This describes a written consequence, not a disciplinary outcome observed by DormWay.

Employer evidence, policy context and prior work

LinkedIn’s 2026 labor-market report reports 70% year-over-year growth in US jobs requiring AI literacy in its observed data. This is growth, not the share of all jobs. USC’s October 6 memorandum requires syllabus AI rules from Spring 2027 while preserving faculty discretion; it is not a nationwide mandate or endorsement of this study.

The subject has prior research, including Igor Chirikov’s 2026 working paper on more than 31,000 syllabi and Handshake’s April 2026 report on student and workforce expectations. DormWay makes no first/largest claim. Different cohorts, dates and definitions prevent interpreting percentages across studies—or across DormWay’s older articles—as a measured trend.

What the study does not measure

The findings document written rules and the decisions those rules require. There was no student survey or measurement of confusion, stress, compliance time, misconduct, actual sanctions, grades or employment outcomes. Sampling intervals do not capture all extraction errors, reference-coding errors, missing-frame coverage or extrapolation beyond the collection. The study does not establish that AI restrictions reduce employability or that policies tightened this fall.

DormWay’s recommendation for clearer instructions and opportunities to practice responsible AI use is its interpretation. Public materials contain aggregates and official public examples, not private uploaded documents or student-linked records.

Read the study · Student checklist

The Course AI Use Label, separate from the results

The Course AI Use Label, now available in the DormWay app, and the downloadable template are DormWay’s communication format for course AI rules. No institution has been verified as adopting it, and this study did not test whether it improves comprehension or changes misconduct or learning outcomes. The fixed statistical codebook, sample and estimates are unchanged. The label’s status vocabulary is not the research classification scheme. A later pilot could measure correct identification of task permission and conditions, time to find the governing instruction, and errors, using an appropriate comparison before claiming effectiveness.

The label acknowledges existing work such as the AI Assessment Scale and Mercer’s traffic-light guidance. The original template does not reproduce either framework’s diagrams or level descriptions; no endorsement, validated mapping or partnership is asserted. Template and guide.

Sources & Methodology

About Riley

Founder & CEO

Riley made DormWay to solve his own problems, and in the process is solving all college students'. A fourth-year at U-M with 100K+ followers across platforms, Riley taught himself to code while building DormWay.