A handbook is useful when someone can open the instruction they need and tell whether it still applies. A search result with a familiar title does not establish either fact.
This Part 4 exercise helps you test a staff handbook you are building in Overskill. You will leave with a small answer key, a record of what you actually checked, and specific failures to repair. It uses ordinary document search or browsing; it does not require an AI answering system.
Download the fictional handbook corpus and Download the completed and blank test worksheet. The example is invented. No search, permissions or saved behavior in an actual app is claimed here.
Freeze five documents
The fictional print studio has five document IDs. Start with revision R1 and keep the later D01 version aside.
Scroll horizontally to compare all columns.
| Document | R1 purpose | Staff access |
|---|---|---|
| D01: Collecting print samples | Current collection point: Front desk | Current document |
| D02: Returning print samples | A separate return shelf and sample-ID instruction | Current document |
| D03: Archived collection instruction | Historical back-door collection, clearly superseded | Explicit archive access; excluded from default results |
| D04: Ask about a missing instruction | Ask the studio coordinator when the handbook does not answer | Current document |
| D05: Supplier exception notes | A manager-only sample-set instruction | No title, snippet or body access |
The download contains all fictional setup text, including the manager seed. Its role labels are requirements for your app, not access controls on the file. Use sample documents and accounts you control when testing.
Open the source behind the result
Choose one route before you run a question. For search, use the exact question and inspect at most the first five displayed results. For browsing, record a path of no more than three navigation steps. These are this pilot's chosen limits, not a universal benchmark. Keep search and browse results separate.
Open the candidate document. Record its ID, version, section and the words supporting the answer. A result title or snippet alone cannot pass. The passage must be current, permitted for the actor and useful for the question.
For example, Q1 asks, “Where do I collect print samples?” In R1, D01 v1's collection-location section says Front desk. D02's Sample returns shelf is a real place in the corpus, but it answers a different task. D03 contains the old back-door instruction, so it cannot supply the current answer.
Try six scenarios
Run the first five as a staff member on R1. The sixth introduces the update.
Scroll horizontally to compare all columns.
| Case | Question or action | Expected result |
|---|---|---|
| Q1 | Where do I collect print samples? | Open D01 v1: Front desk |
| Q2 | I need to pick up samples. Where do I go? | The same current collection section |
| Q3 | Is back-door pickup still used? | D01 v1 explicitly says Front desk replaces it |
| Q4 | Can we collect samples on Sunday? | No supported answer; use D04's question-owner route |
| Q5 | What are the supplier exception rules? | No staff-permitted answer; no restricted material disclosed |
| Q6 | Replace D01 with its approved v2, then repeat Q1 | Open D01 v2: Dispatch counter |
The corpus supplies no collection hours. For Q4, missing text means unknown; it does not mean Sunday collection is allowed or forbidden. D04 gives the reader a next action without pretending to answer.
For Q5, the staff-facing fallback can use the same plain response: “The permitted handbook does not answer this question. Ask the studio coordinator.” It must not advertise a manager-only document. Use the broader private-access test for copied links, accounts and other permission boundaries.
Keep the old run when a source changes
Load R2 only after recording R1. D01 v2 changes the collection point to Dispatch counter and replaces both previous locations. D01 v1 becomes history; D03 stays superseded. Neither belongs in default current results.
Q6 repeats Q1. Rerun Q2–Q5 as well. The direct question, paraphrase and old-wording answers change; the Sunday and restricted-information questions still have no permitted answer. The worksheet holds ten run rows across five distinct question texts. Keep the baseline observations so the update cannot erase evidence of an earlier failure.
Decide what needs repair
The completed worksheet contains expected answers, with every actual app outcome marked Not tested. Fill those outcomes only after performing the checks. A performed violation fails; an unavailable account or an unperformed action stays untested with its reason.
Repair the specific failure: the wrong version was current, an ordinary phrase did not reach its source, a return instruction was mistaken for collection, or a restricted snippet leaked. Then rerun the affected cases. Passing this small set supports only the tested corpus, role and route; it does not establish a general search score or complete security review.
Carry the requirements into Overskill
For an existing app, give Overskill the failed question, expected source, observed behavior and role, then test the repair. For a new handbook prototype, open the app brief builder and replace its example fields. The link does not import the downloaded corpus. Supply the fictional setup text and question cards separately when preparing the prototype.
| Field | Starting instruction |
|---|---|
| Who is it for? | Staff and managers of a small print studio |
| What should they finish? | Find and open a current permitted handbook instruction, or reach an honest no-answer fallback |
| What information should the app save? | Documents, versions, stable sections, current/archive status and allowed roles. Staff read current staff documents; managers additionally read manager-only documents. Workspace membership never grants manager access. Enforce those roles on the server, including results and snippets. Keep archives out of default results and link history to the current version. Test the fictional corpus and question cards I supply. |
| Who can see each record? | Members of the same workspace, with the additional document-role restrictions above |
Review the generated brief against those rules and edit the fields if anything is missing. Then choose Start build. Keep real private handbook content out of the starter example; your account's normal access and credit requirements apply when building.