curated_feed.state
==================

.. py:module:: curated_feed.state

.. autoapi-nested-parse::

   Reading and writing everything the pipeline keeps on disk.

   Two kinds of file. The **corpus** is one JSONL per collection day, appended to
   and never rewritten. The **artifacts** are one JSON per stage per day, under the
   state directory, each the output of one stage and the input of the next.

   That every stage boundary is a file is what makes `--date` work: a saved
   candidate set can be re-scored, or a saved response re-selected, without
   refetching anything. Policy tuning is then a loop of seconds rather than a day.



Attributes
----------

.. autoapisummary::

   curated_feed.state.CORPUS_SUFFIX
   curated_feed.state.ARTIFACT_SUFFIX


Exceptions
----------

.. autoapisummary::

   curated_feed.state.StateError


Functions
---------

.. autoapisummary::

   curated_feed.state.corpus_path
   curated_feed.state.corpus_day
   curated_feed.state.latest_corpus_path
   curated_feed.state.resolve_corpus
   curated_feed.state.read_jsonl
   curated_feed.state.append_jsonl
   curated_feed.state.read_corpus
   curated_feed.state.append_corpus
   curated_feed.state.existing_ids
   curated_feed.state.artifact_path
   curated_feed.state.read_artifact
   curated_feed.state.write_artifact
   curated_feed.state.write_text_artifact


Module Contents
---------------

.. py:data:: CORPUS_SUFFIX
   :value: '.jsonl'


.. py:data:: ARTIFACT_SUFFIX
   :value: '.json'


.. py:exception:: StateError

   Bases: :py:obj:`Exception`


   Raised when a file the pipeline depends on is missing or unreadable.


.. py:function:: corpus_path(corpus_dir: pathlib.Path, day: datetime.date) -> pathlib.Path

   The corpus file for one collection day.

   >>> corpus_path(Path("/corpus"), date(2026, 8, 8)).as_posix()
   '/corpus/2026-08-08.jsonl'


.. py:function:: corpus_day(path: pathlib.Path) -> datetime.date

   The collection day a corpus file is named for.

   >>> corpus_day(Path("/corpus/2026-08-08.jsonl")).isoformat()
   '2026-08-08'


.. py:function:: latest_corpus_path(corpus_dir: pathlib.Path) -> pathlib.Path

   The most recent corpus file, by date-ordered filename.


.. py:function:: resolve_corpus(corpus_dir: pathlib.Path, day: datetime.date | None) -> tuple[pathlib.Path, datetime.date]

   The corpus file to work on, defaulting to the most recent collection day.


.. py:function:: read_jsonl[RecordT: pydantic.BaseModel](path: pathlib.Path, model: type[RecordT]) -> list[RecordT]

   Read and validate a JSONL file, skipping blank lines.

   A missing file reads as empty: the published log does not exist until the
   first feed is rendered, and that is not an error.


.. py:function:: append_jsonl(path: pathlib.Path, records: collections.abc.Iterable[pydantic.BaseModel]) -> int

   Append records as JSONL, creating the file and its directory if needed.

   Writes nothing when there is nothing to write: an empty corpus file would
   otherwise become the "latest" file that later commands pick up.


.. py:function:: read_corpus(path: pathlib.Path) -> list[curated_feed.models.Candidate]

   Read and validate a corpus file.


.. py:function:: append_corpus(path: pathlib.Path, candidates: collections.abc.Iterable[curated_feed.models.Candidate]) -> int

   Append candidates to a corpus file.


.. py:function:: existing_ids(corpus_dir: pathlib.Path) -> set[str]

   Every id already recorded in the corpus directory.

   Reads the id field alone rather than validating whole records: this runs over
   every line of every day collected so far, and only identity matters here.
   Junk lines are skipped, because a corrupt tail must not block a fetch.


.. py:function:: artifact_path(state_dir: pathlib.Path, stage: str, day: datetime.date) -> pathlib.Path

   Where one stage's output for one day lives.

   >>> artifact_path(Path("/state"), "response", date(2026, 8, 8)).as_posix()
   '/state/response/2026-08-08.json'


.. py:function:: read_artifact[ArtifactT: pydantic.BaseModel](path: pathlib.Path, model: type[ArtifactT]) -> ArtifactT

   Read and validate one stage artifact.


.. py:function:: write_artifact(path: pathlib.Path, artifact: pydantic.BaseModel) -> pathlib.Path

   Write one stage artifact, indented so its daily diff stays readable.


.. py:function:: write_text_artifact(path: pathlib.Path, text: str) -> pathlib.Path

   Write a stage artifact that is not a model, such as a raw response.

   The raw text is saved before parsing, so a response that fails validation can
   still be read afterwards.