curated_feed.score

Turning a day of candidates into the model’s judgements.

A scorer produces the raw response text and nothing else. Persisting it, validating it and retrying live here, in one place, so that every backend gets the same treatment and a new one is a single small class.

Two backends so far, neither needing an API client or a key:

• StubScorer invents a response from publication dates alone. It exists to prove the stages after it — selection, rendering, publishing — against a real feed before a token is spent.

• FileScorer reads a response saved from a chat session, which is how the evaluation workflow drives the real pipeline.

The failure policy is decided here rather than at six in the morning: save the raw text before parsing, retry once with the validation error appended, and on a second failure publish nothing. The previously published feed stays valid and served, so a failed day costs one missing day.

Attributes

CODE_FENCE
NO_SUBSTANCE
MAX_ATTEMPTS
RESPONSE_STAGE

Exceptions

ScoreError

Raised when a usable response cannot be obtained.

Classes

Scorer	Anything that can produce a response for a day of candidates.
StubScorer	Grades the most recently published candidates, calling nothing.
FileScorer	Reads a response saved from a chat session.

Functions

score_day(→ curated_feed.models.Response)	Obtain a validated response for one day.
parse_response(→ curated_feed.models.Response)	Validate raw response text, tolerating a Markdown code fence.
read_response(→ curated_feed.models.Response)	Read a saved response artifact.
unfence(→ str)	The JSON inside a Markdown code fence, or the text unchanged.
make_scorer(→ Scorer)	The scorer named in configuration.
format_summary(→ str)	Human-readable run report.
build_parser(→ argparse.ArgumentParser)
main(→ int)

Module Contents

curated_feed.score.CODE_FENCE

curated_feed.score.NO_SUBSTANCE = 'no-substance'

curated_feed.score.MAX_ATTEMPTS = 2

curated_feed.score.RESPONSE_STAGE = 'response'

exception curated_feed.score.ScoreError

Bases: Exception

Raised when a usable response cannot be obtained.

class curated_feed.score.Scorer

Bases: Protocol

Anything that can produce a response for a day of candidates.

feedback carries the validation error from a previous attempt, and is None on the first. A backend that cannot act on it should say so rather than silently repeating itself.

name: str

generate(candidates: collections.abc.Sequence[curated_feed.models.Candidate], prompt: curated_feed.policy.Prompt, *, feedback: str | None = None) → str

class curated_feed.score.StubScorer(*, items: int)

Grades the most recently published candidates, calling nothing.

Judgement is not the point: publishing something Feedly can ingest is. The graded items therefore clear the floor on consequence alone, and are marked durable so the staleness adjustment cannot pull them back under it.

name = 'stub'

items

generate(candidates: collections.abc.Sequence[curated_feed.models.Candidate], prompt: curated_feed.policy.Prompt, *, feedback: str | None = None) → str

class curated_feed.score.FileScorer(path: pathlib.Path)

Reads a response saved from a chat session.

This is what connects the manual evaluation loop to the pipeline: export a day, paste it into a chat, save the JSON, and the real selection, rendering and publishing run against it.

name = 'file'

path

generate(candidates: collections.abc.Sequence[curated_feed.models.Candidate], prompt: curated_feed.policy.Prompt, *, feedback: str | None = None) → str

curated_feed.score.score_day(scorer: Scorer, candidates: collections.abc.Sequence[curated_feed.models.Candidate], prompt: curated_feed.policy.Prompt, *, raw_path: pathlib.Path) → curated_feed.models.Response

Obtain a validated response for one day.

The raw text of each attempt is written to raw_path before it is parsed, so a response that fails validation can still be read afterwards — fence and all, since what is saved is what arrived. The last attempt is the one left on disk.

curated_feed.score.parse_response(raw: str) → curated_feed.models.Response

Validate raw response text, tolerating a Markdown code fence.

curated_feed.score.read_response(path: pathlib.Path) → curated_feed.models.Response

Read a saved response artifact.

The artifact is the raw text the model returned, fence and all, because it is the audit trail for a day that went wrong. Everything that reads it back therefore has to be as tolerant as the parse that first accepted it — which is why this exists rather than a bare read_artifact(path, Response).

curated_feed.score.unfence(text: str) → str

The JSON inside a Markdown code fence, or the text unchanged.

Asking for bare JSON does not reliably get it: wrapping it in a fence is a deeply trained habit, and a response saved from a chat session almost always arrives fenced. Spending the day’s one retry on punctuation would be absurd, so the fence is simply removed here.

>>> unfence('```json\n{"graded": []}\n```')
'{"graded": []}'
>>> unfence('{"graded": []}')
'{"graded": []}'

curated_feed.score.make_scorer(config: curated_feed.config.Config, *, response_path: pathlib.Path | None) → Scorer

The scorer named in configuration.

The provider SDK is imported only when it is the one selected, so the offline backends — and every other command — stay free of it.

curated_feed.score.format_summary(response: curated_feed.models.Response, *, candidates: int, path: pathlib.Path, scorer: Scorer) → str

Human-readable run report.

curated_feed.score.build_parser() → argparse.ArgumentParser

curated_feed.score.main(argv: collections.abc.Sequence[str] | None = None) → int