Margaret Ellison, advisor (synthetic persona)Senior advisor, founder and executive practice, New YorkAbout $820M across 183 households; four relationships above $50M; 7 of those households are in this prototypeWho's who

Evals

How the agents are known to be right, and known to stay right: a corpus a person can check, an expected side that shares no code with the engines, a score per decision, and a golden file the check suite enforces.

300 households and 585 captured messages, generated from a fixed seed so every sentence is one of a known set and its ground truth is reviewed by hand.
The expected side reads the CSV files with its own reader and imports nothing from the engines. If it did, a bug in the engines would be in the answer key too.
Scored per (household, rule) and per (household, kind), never as one average. The counts below are the golden file; a change to a rule, an extractor or the corpus fails the suite until someone reads the diff.

The corpus

300

Households

585

Captured messages

406

Opportunities detected

Data problems

Arithmetic, engine against the eval's own: liquidity months agree on 300 of 300 households, total assets on 300 of 300. The corpus itself lives under public/samples and can be dropped on the Sources screen.

How it is built, in four steps

1. Generate

A sample generator writes 300 households and their messages from a fixed seed and a fixed set of sentences. Reproducible, and small enough to read.

2. Label

A labels file says, per sentence, which rule should fire and which discovery kind it yields, with the reason written beside the ones that fire nothing.

3. Expect, without the engines

The expected side derives every count from the CSV, the rule parameters, the firm edits and each advisor's connections, with its own reader and arithmetic.

4. Score and freeze

The engines run through the real import path and are scored per decision. The counts are frozen in a golden file the check suite enforces on every push.

Compliance desks, per household and per message

100.0%

Precision across rules

100.0%

Recall across rules

300

Reported as cannot evaluate, correctly

50 of 50

On an unhealthy source, not swept

Rule	Expected	Found	Correct	Precision	Recall
Unlogged client complaint complaint-identification-4513	48	48	48	100.0%	100.0%
Concentration rising toward the household's ceiling finra-2111-drift	44	44	44	100.0%	100.0%
Holdings drifted outside the stated profile finra-2111-suitability	78	78	78	100.0%	100.0%
Correspondence not reviewed under written supervisory procedures finra-3110-correspondence	64	64	64	100.0%	100.0%
Performance or testimonial content in advisory-capacity material sec-marketing-206-4-1	40	40	40	100.0%	100.0%
Possible diminished capacity or financial exploitation senior-investor-2165	23	23	23	100.0%	100.0%

A rule whose source is not connected for the household's advisor must come back as cannot evaluate, never as clear. The eval expects that from the advisor files, not from the engine.

Cannot evaluate	Expected	Found	Correct	Precision	Recall
Concentration rising toward the household's ceiling finra-2111-drift	100	100	100	100.0%	100.0%
Holdings drifted outside the stated profile finra-2111-suitability	100	100	100	100.0%	100.0%
Possible diminished capacity or financial exploitation senior-investor-2165	100	100	100	100.0%	100.0%

Discovery, per household and kind

100.0%

Precision across kinds

100.0%

Recall across kinds

430

Candidates expected

Kinds scored

School or university fees23
Cash the client wants working34
Marriage, birth, divorce or a death32
Inheritance or windfall33
Company sale, vesting or funding round34
Property sale or purchase101
Relocation or cross-border move27
Retirement or pension decision85
Someone new acting for the client61

Kind	Expected	Found	Correct	Precision	Recall
School or university fees education	23	23	23	100.0%	100.0%
Cash the client wants working excess-cash	34	34	34	100.0%	100.0%
Marriage, birth, divorce or a death family-change	32	32	32	100.0%	100.0%
Inheritance or windfall inheritance	33	33	33	100.0%	100.0%
Company sale, vesting or funding round liquidity-event	34	34	34	100.0%	100.0%
Property sale or purchase property-sale	101	101	101	100.0%	100.0%
Relocation or cross-border move relocation	27	27	27	100.0%	100.0%
Retirement or pension decision retirement	85	85	85	100.0%	100.0%
Someone new acting for the client third-party	61	61	61	100.0%	100.0%

Research and retrieval

300

Briefings

711

Things not established

189

Inferences with a confidence

yes

Every claim cites a record

406 of 406

Opportunities cited

105

Resting on documents that disagree

The labelled sentences: 19 kinds of thing a client or colleague says

Every message and note in the corpus is one of these. The label says what each should trigger; the reason beside a sentence that triggers nothing is the part a reviewer reads first.

Sentence	Where	Should fire	Should discover
"Following up on our call. I'd suggest we move a year of spending out of cash into the ladder."	message, outbound	Correspondence not reviewed under written supervisory procedures	nothing
"Confirming Thursday at 10. Both of you, as you asked."	message, outbound	nothing	nothing
"The fund we discussed will return about 6% a year, which more than covers the gap."	message, outbound	Performance or testimonial content in advisory-capacity material	nothing
"Attached is the form. Sign where marked and send it back whenever convenient."	message, outbound	nothing	nothing
"I sit on the advisory board of the sponsor, so I can walk you through the terms myself." Outside business language, but the conduct rule is switched off at the firm layer in the shipped rule set, so nothing fires. The proposer sees it.	message, outbound	nothing	nothing
"Can we find time next week to talk through the pension?"	message, inbound	nothing	Retirement or pension decision
"This is not what I was told when I opened the account. I'm unhappy with how the fees were explained."	message, inbound	Unlogged client complaint	nothing
"We've just accepted an offer on the house. Completion is in six weeks."	message, inbound	nothing	Property sale or purchase
"My brother will be handling the paperwork for me from now on. Please send everything to him."	message, inbound	nothing	Someone new acting for the client
"Should I put the whole cash balance into crypto? A friend doubled his last month."	message, inbound	nothing	nothing
"Mentioned they are thinking about selling the second home next spring."	note	nothing	Property sale or purchase
"Asked whether the cash could go somewhere that earns a little more."	note	nothing	Cash the client wants working
"Daughter is getting married in the autumn; wants to help with the deposit on a flat."	note	nothing	Marriage, birth, divorce or a death Property sale or purchase
"Retiring at the end of next year; has not decided on the pension option."	note	nothing	Retirement or pension decision
"Moving to Lisbon for two years for work; unsure what that means for taxes."	note	nothing	Relocation or cross-border move
"Received an inheritance from an aunt; roughly four hundred thousand."	note	nothing	Inheritance or windfall
"Company is being acquired; the shares vest on close."	note	nothing	Company sale, vesting or funding round
"Wants to set something up for the grandchildren's school fees."	note	nothing	School or university fees
"Review; nothing changed"	contact	nothing	nothing

What the eval has caught

Recorded so the eval is not mistaken for a formality.

Found	Where	Fix
A client asking to talk through their pension was not read as a retirement candidate; 53 of 85 expected candidates were missed	Discovery extractor	The extractor now reads a request to talk through a pension as the decision it is
53 households had a liquidity target below their own minimum, and one spent a third of its assets a year	Sample generator	Spending is anchored to the portfolio at 1.5% to 5% a year; the target is never below the minimum
One household's day-90 concentration was negative	Sample generator	History is floored above zero

It was also seen failing on each of four deliberate breaks before it was recorded as a guard:

A complaint pattern with two of its phrases removed
The retirement extractor narrowed
One household's cash changed in the CSV
The review threshold moved from 180 to 200 days

The limit, stated

Recall of 100% on a fixed sentence set says the engines read every sentence in that set, not every way a client might say the same thing. A sentence a real book would contain and these engines would miss belongs in the labels with the finding expected, so the eval fails first and the engine is fixed second. When a model reads free text in production, it is scored by the same harness at the same level, and held-out phrasings the fixed patterns cannot read are where it has to beat them.

Next: How it works, the loop, every morning.