curated_feed.paywall
====================

.. py:module:: curated_feed.paywall

.. autoapi-nested-parse::

   Deciding whether a candidate's page can actually be read.

   The signal is schema.org's `isAccessibleForFree`, and it is the only one. It is
   reliable because Google requires it of publishers who want paywalled content
   indexed, which is why it is on the page at all.

   Read it in one direction only: **present and false means paywalled, absent
   means free.** Publishers mark the restriction, not the absence of one, so
   treating silence as evidence of a wall would gate almost every article in the
   feed.

   Two things that look like signals and are not, so nobody adds them:

   - **Body length.** There is no cross-site threshold. A truncated article from a
     long-form magazine outruns a complete one from a wire service, so length
     measures house style rather than access.
   - **The RSS subscriber marker.** Where it exists at all it is channel
     boilerplate, and nothing survives `build_record` normalisation. Detection
     without a request is not available.

   `detect` is pure and total: it answers for any string, and it has two answers.
   Whether a page was *reached* is the transport's business, and that is where the
   third verdict is decided.

   The stage runs after scoring, so it probes the graded records rather than every
   candidate: the ones the model dropped for having no substance are not worth a
   request to anybody's server. Verdicts are appended to a log keyed by candidate
   id and never revisited, so replaying a day costs nothing and a re-run does not
   knock on the same door twice.

   This fetches a full article page, and it must not be read as delivering the
   reading-effort scoring `docs/ARCHITECTURE.md` defers. It looks at the markup and
   throws the body away.



Attributes
----------

.. autoapisummary::

   curated_feed.paywall.PAYWALL_LOG
   curated_feed.paywall.ACCESSIBLE_KEY
   curated_feed.paywall.FALSE_VALUES
   curated_feed.paywall.JSON_LD_TYPE
   curated_feed.paywall.GRAPH_KEY


Classes
-------

.. autoapisummary::

   curated_feed.paywall.RunSummary


Functions
---------

.. autoapisummary::

   curated_feed.paywall.detect
   curated_feed.paywall.probe_order
   curated_feed.paywall.build_client
   curated_feed.paywall.check_one
   curated_feed.paywall.check_all
   curated_feed.paywall.log_path
   curated_feed.paywall.read_verdicts
   curated_feed.paywall.collect
   curated_feed.paywall.format_summary
   curated_feed.paywall.build_parser
   curated_feed.paywall.main


Module Contents
---------------

.. py:data:: PAYWALL_LOG
   :value: 'paywall.jsonl'


.. py:data:: ACCESSIBLE_KEY
   :value: 'isAccessibleForFree'


.. py:data:: FALSE_VALUES

.. py:data:: JSON_LD_TYPE
   :value: 'application/ld+json'


.. py:data:: GRAPH_KEY
   :value: '@graph'


.. py:function:: detect(html: str) -> str

   Whether a page's markup restricts it, in order of how publishers carry it.

   >>> detect('<script type="application/ld+json">{"isAccessibleForFree": false}</script>')
   'paywalled'
   >>> detect('<meta name="isAccessibleForFree" content="no">')
   'paywalled'
   >>> detect("<html><body><p>An article.</p></body></html>")
   'free'


.. py:class:: RunSummary

   Everything the end-of-run report needs.


   .. py:attribute:: log_path
      :type:  pathlib.Path


   .. py:attribute:: graded
      :type:  int
      :value: 0



   .. py:attribute:: cached
      :type:  int
      :value: 0



   .. py:attribute:: dropped_by_cap
      :type:  int
      :value: 0



   .. py:attribute:: verdicts
      :type:  list[curated_feed.models.Verdict]
      :value: []



   .. py:method:: counted(verdict: str) -> int


.. py:function:: probe_order(response: curated_feed.models.Response, candidates: collections.abc.Sequence[curated_feed.models.Candidate]) -> list[curated_feed.models.Candidate]

   The graded candidates, most consequential first.

   Ordered rather than merely collected so that `max_checks` truncates the
   least important tail. Consequence is a deliberate stand-in for the score:
   ranking properly here would be a second implementation of `select.py`'s
   arithmetic, and the two would drift.

   An index the response invented is skipped. Selection records that as an
   anomaly later; this stage only has to avoid tripping over it.


.. py:function:: build_client(settings: curated_feed.config.PaywallSettings, *, user_agent: str) -> httpx.AsyncClient

   The client one run probes with.

   A factory, so the stage's one external dependency can be substituted the
   way `fetch.py`'s is. The User-Agent is passed in rather than read from
   `settings`, because an empty one there means "use the runner's", and this
   module is not where that is resolved.


.. py:function:: check_one(client: httpx.AsyncClient, candidate: curated_feed.models.Candidate, *, max_retries: int, now: datetime.datetime) -> curated_feed.models.Verdict
   :async:


   Fetch one article page and read its markup.

   Every failure is `unknown` rather than an exception: a page that cannot be
   reached has said nothing about whether it can be read, and this stage must
   never cost the day its feed.


.. py:function:: check_all(candidates: collections.abc.Sequence[curated_feed.models.Candidate], config: curated_feed.config.Config, *, now: datetime.datetime) -> tuple[list[curated_feed.models.Verdict], int]

   Probe up to `max_checks` candidates, and say how many were left unprobed.

   The count is returned rather than logged away because a cap that truncates
   quietly reads as full coverage, and the day it matters is the day nobody is
   looking.


.. py:function:: log_path(state_dir: pathlib.Path) -> pathlib.Path

   Where a runner's verdicts accumulate.

   >>> log_path(Path("/state")).as_posix()
   '/state/paywall.jsonl'


.. py:function:: read_verdicts(path: pathlib.Path) -> dict[str, curated_feed.models.Verdict]

   Every verdict recorded so far, by candidate id.

   A missing file reads as an empty map, which is what makes the whole feature
   a no-op until a runner turns it on, and what lets every day already saved in
   the corpus replay exactly as it did.

   A candidate probed twice keeps its later verdict, which is the one a repair
   run would have written.


.. py:function:: collect(config: curated_feed.config.Config, *, day, candidates, response, now: datetime.datetime) -> RunSummary

   Probe what has not been probed before, and append what was learned.


.. py:function:: format_summary(summary: RunSummary) -> str

   Human-readable run report.


.. py:function:: build_parser() -> argparse.ArgumentParser

.. py:function:: main(argv: collections.abc.Sequence[str] | None = None) -> int