Ingestion

Ingestion turns registered sources into searchable records. Choose its lifecycle explicitly so a growing document collection has predictable effects on application startup.

The in-memory store loses its indexes and records when the process restarts. Re-ingest before serving retrieval requests after every restart, either through startup ingestion or an explicit run. Merely registering the index again does not restore its data. PostgreSQL provides durable storage when that lifecycle is unsuitable.

Choose a start strategy

Select one strategy in the index configuration:

.ConfigureIngestion(ingestion => ingestion.BackgroundOnStartup())

Use OnStartup only when blocking host startup is intentional. If ingestion fails during that path, the host may never become ready. That tradeoff is acceptable for small samples and debugging, but production collections should usually use BackgroundOnStartup, a scheduled owner process, or an explicit management action.

For a scheduled index, a five-field expression selects the automatic trigger:

.ConfigureIngestion(ingestion => ingestion.Scheduled("0 * * * *"))

Check the host time zone when configuring the schedule. Scheduling is per process and does not supply distributed locks, leader election, or coordination between application instances. Decide which instance owns refreshes before enabling it across replicas.

Manage ingestion from RAG Management

Dashboard > RAG Management is an interactive way to run the same managed ingestion lifecycle. You can start an explicit run for any registered index, regardless of its automatic start strategy.

1. Open Dashboard > RAG Management and select company-policies from the registered indexes.
2. Inspect its readiness and active or last operation status.
3. Choose Start ingestion to refresh the index and monitor progress.
4. If the active run needs to stop, request cancellation through RAG Management and wait for the operation to reach its terminal state.
5. Check both the final operation result and index readiness before using the refreshed content.

Dashboard start, cancel, and status operations use IRagIngestionManager, the same coordinator used by application code below. They share conflict and cancellation handling. Indexes and their sources must already be registered in application startup code; the Dashboard does not create them.

The Product Support sample walkthrough shows this management surface using the sample's product-support index.

Run ingestion from application code

Resolve IRagIngestionManager through dependency injection in a service, job, or administrative handler. The following fragment assumes ingestionManager and cancellationToken were supplied by that caller:

var operation = await ingestionManager.StartAsync(
    "company-policies",
    cancellationToken);

var status = ingestionManager.GetStatus("company-policies");

StartAsync completes when the operation reaches a terminal state. The interface is in Runiq.AI.Rag.Runtime. An explicit run is supported for every strategy, including scheduled and startup indexes.

To stop an active operation from another administrative action:

await ingestionManager.CancelAsync("company-policies", cancellationToken);

Cancellation and conflicting runs are coordinated by the manager. Keep application ingestion commands on this shared path.

Separate readiness from operation status

A current ingestion operation and the availability of searchable content answer different questions. An index can retain usable content after a later refresh fails. Inspect readiness as well as the active or last operation when deciding what the UI should show.

Managed runtime status and the latest operation are held in memory and reset after a process restart. PostgreSQL's durable document and ingestion records are a separate persistence concern; they do not turn the manager's status into a permanent job history.

The Product Support sample walkthrough illustrates index readiness and retrieval through an agent conversation.

Reference: RAG ingestion lifecycle.