curated_feed.paywall¶
Deciding whether a candidate’s page can actually be read.
The signal is schema.org’s isAccessibleForFree, and it is the only one. It is reliable because Google requires it of publishers who want paywalled content indexed, which is why it is on the page at all.
Read it in one direction only: present and false means paywalled, absent means free. Publishers mark the restriction, not the absence of one, so treating silence as evidence of a wall would gate almost every article in the feed.
Two things that look like signals and are not, so nobody adds them:
Body length. There is no cross-site threshold. A truncated article from a long-form magazine outruns a complete one from a wire service, so length measures house style rather than access.
The RSS subscriber marker. Where it exists at all it is channel boilerplate, and nothing survives build_record normalisation. Detection without a request is not available.
detect is pure and total: it answers for any string, and it has two answers. Whether a page was reached is the transport’s business, and that is where the third verdict is decided.
The stage runs after scoring, so it probes the graded records rather than every candidate: the ones the model dropped for having no substance are not worth a request to anybody’s server. Verdicts are appended to a log keyed by candidate id and never revisited, so replaying a day costs nothing and a re-run does not knock on the same door twice.
This fetches a full article page, and it must not be read as delivering the reading-effort scoring docs/ARCHITECTURE.md defers. It looks at the markup and throws the body away.
Attributes¶
Classes¶
Everything the end-of-run report needs. |
Functions¶
|
Whether a page's markup restricts it, in order of how publishers carry it. |
|
The graded candidates, most consequential first. |
|
The client one run probes with. |
|
Fetch one article page and read its markup. |
|
Probe up to max_checks candidates, and say how many were left unprobed. |
|
Where a runner's verdicts accumulate. |
|
Every verdict recorded so far, by candidate id. |
|
Probe what has not been probed before, and append what was learned. |
|
Human-readable run report. |
|
|
|
Module Contents¶
- curated_feed.paywall.PAYWALL_LOG = 'paywall.jsonl'¶
- curated_feed.paywall.ACCESSIBLE_KEY = 'isAccessibleForFree'¶
- curated_feed.paywall.FALSE_VALUES¶
- curated_feed.paywall.JSON_LD_TYPE = 'application/ld+json'¶
- curated_feed.paywall.GRAPH_KEY = '@graph'¶
- curated_feed.paywall.detect(html: str) str[source]¶
Whether a page’s markup restricts it, in order of how publishers carry it.
>>> detect('<script type="application/ld+json">{"isAccessibleForFree": false}</script>') 'paywalled' >>> detect('<meta name="isAccessibleForFree" content="no">') 'paywalled' >>> detect("<html><body><p>An article.</p></body></html>") 'free'
- class curated_feed.paywall.RunSummary[source]¶
Everything the end-of-run report needs.
- log_path: pathlib.Path¶
- verdicts: list[curated_feed.models.Verdict] = []¶
- curated_feed.paywall.probe_order(response: curated_feed.models.Response, candidates: collections.abc.Sequence[curated_feed.models.Candidate]) list[curated_feed.models.Candidate][source]¶
The graded candidates, most consequential first.
Ordered rather than merely collected so that max_checks truncates the least important tail. Consequence is a deliberate stand-in for the score: ranking properly here would be a second implementation of select.py’s arithmetic, and the two would drift.
An index the response invented is skipped. Selection records that as an anomaly later; this stage only has to avoid tripping over it.
- curated_feed.paywall.build_client(settings: curated_feed.config.PaywallSettings, *, user_agent: str) httpx.AsyncClient[source]¶
The client one run probes with.
A factory, so the stage’s one external dependency can be substituted the way fetch.py’s is. The User-Agent is passed in rather than read from settings, because an empty one there means “use the runner’s”, and this module is not where that is resolved.
- async curated_feed.paywall.check_one(client: httpx.AsyncClient, candidate: curated_feed.models.Candidate, *, max_retries: int, now: datetime.datetime) curated_feed.models.Verdict[source]¶
Fetch one article page and read its markup.
Every failure is unknown rather than an exception: a page that cannot be reached has said nothing about whether it can be read, and this stage must never cost the day its feed.
- curated_feed.paywall.check_all(candidates: collections.abc.Sequence[curated_feed.models.Candidate], config: curated_feed.config.Config, *, now: datetime.datetime) tuple[list[curated_feed.models.Verdict], int][source]¶
Probe up to max_checks candidates, and say how many were left unprobed.
The count is returned rather than logged away because a cap that truncates quietly reads as full coverage, and the day it matters is the day nobody is looking.
- curated_feed.paywall.log_path(state_dir: pathlib.Path) pathlib.Path[source]¶
Where a runner’s verdicts accumulate.
>>> log_path(Path("/state")).as_posix() '/state/paywall.jsonl'
- curated_feed.paywall.read_verdicts(path: pathlib.Path) dict[str, curated_feed.models.Verdict][source]¶
Every verdict recorded so far, by candidate id.
A missing file reads as an empty map, which is what makes the whole feature a no-op until a runner turns it on, and what lets every day already saved in the corpus replay exactly as it did.
A candidate probed twice keeps its later verdict, which is the one a repair run would have written.
- curated_feed.paywall.collect(config: curated_feed.config.Config, *, day, candidates, response, now: datetime.datetime) RunSummary[source]¶
Probe what has not been probed before, and append what was learned.
- curated_feed.paywall.format_summary(summary: RunSummary) str[source]¶
Human-readable run report.
- curated_feed.paywall.build_parser() argparse.ArgumentParser[source]¶