Evaluate AI applications¶

Scenario 1: Trace a function¶

You can add @instrument() before the function you need to trace. This automatically captures the inputs to the function, the outputs (return values), and the latency of execution. For example, the following code demonstrates tracing an answer_query function that automatically captures input query and the final response:

@instrument()
def answer_query(self, query: str) -> str:
    context_str = self.retrieve_context(query)
    return self.generate_completion(query, context_str)

Scenario 2: Trace a function with a specific span type¶

A span type specifies the nature of the function and improves the readability and understanding of the traces. For example, in a RAG application you can specify span type as “RETRIEVAL” for your search service (or retriever) and specify the span type as “GENERATION” for the LLM inference call. The following span types are supported:

• RETRIEVAL: Span type for retrieval or search functions

• GENERATION: Span type for model inference calls from an LLM

• RECORD_ROOT: Span type for the main function in your application

If you don’t specify a span type with the @instrument(), an UNKNOWN span type is assigned by default. To use span attributes, add the following import to your Python application.

from trulens.otel.semconv.trace import SpanAttributes

The following code snippet demonstrates tracing a RAG application. The span type must always be prefixed with SpanAttributes.SpanType.

@instrument(span_type=SpanAttributes.SpanType.RETRIEVAL)
def retrieve_context(self, query: str) -> list:
    """
    Retrieve relevant text from vector store.
    """
    return self.retrieve(query)

@instrument(span_type=SpanAttributes.SpanType.GENERATION)
def generate_completion(self, query: str, context_str: list) -> str:
    """
    Generate answer from context by calling an LLM.
    """
    return response

@instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT)
def answer_query(self, query: str) -> str:
    context_str = self.retrieve_context(query)
    return self.generate_completion(query, context_str)

Scenario 3: Trace a function and compute evaluations¶

In addition to providing span types, you must assign relevant parameters in your application to span attributes to compute the metrics. For example, to compute context relevance in a RAG application, you must assign the relevant query and retrieval results parameter to appropriate attributes RETRIEVAL.QUERY_TEXT and RETRIEVAL.RETRIEVED_CONTEXTS respectively. The attributes required to compute each individual metric can be found in the Metrics page.

The following span attributes are supported for each span type:

• RECORD_ROOT: INPUT, OUTPUT, GROUND_TRUTH_OUTPUT

• RETRIEVAL: QUERY_TEXT, RETRIEVED_CONTEXTS

• GENERATION: None

To use span attributes, you need to add the following import to your Python application.

from trulens.otel.semconv.trace import SpanAttributes

The following code snippet provides an example to compute context relevance for a retrieval service. The attributes must always follow the format SpanAttributes.<span type>.<attribute name> (e.g., SpanAttributes.RETRIEVAL.QUERY_TEXT).

@instrument(
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    attributes={
        SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
        SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
    }
)
def retrieve_context(self, query: str) -> list:
    """
    Retrieve relevant text from vector store.
    """
    return self.retrieve(query)

In the preceding example, query represents the input parameter to retrieve_context() and return represents the value returned. These are assigned to the attributes RETRIEVAL.QUERY_TEXT and RETRIEVAL.RETRIEVED_CONTEXTS to compute context relevance.

Run Configuration¶

A run is created from a RunConfig

run_config = RunConfig(
    run_name=run_name,
    description="desc",
    label="custom tag useful for grouping comparable runs",
    source_type="DATAFRAME",
    dataset_name="My test dataframe name",
    dataset_spec={
        "RETRIEVAL.QUERY_TEXT": "user_query_field",
        "RECORD_ROOT.INPUT": "user_query_field",
        "RECORD_ROOT.GROUND_TRUTH_OUTPUT": "golden_answer_field",
    },
    llm_judge_name: "mistral-large2"
)

• run_name: str: name of the run, should be unique under the same TruApp

• description: str (optional): string description of the run

• label: str (optional): label used to group run together

• source_type: str: specifies the source of the dataset. It can either be DATAFRAME for a python dataframe or TABLE for a user table in the Snowflake account.

• dataset_name: str: any arbitrary name specified by the user if source_type is DATAFRAME. Or, a valid Snowflake table name under the user’s account under current context (database and schema) or Snowflake fully-qualified name in the form of “database.schema.table_name”.

• dataset_spec: Dict[str, str]: a dictionary mapping supported span attributes to user’s column names in the dataframe or table. The allowed keys are span attributes as specified in the Dataset page and the allowed values are column names in the user’s specified dataframe or table. For example, “golden_answer_field” in the run config example above must be a valid column name

• llm_judge_name: str (Optional): name to use as LLM judges during LLM-based metric computation. Please see the models page for supported judges. If not specified, the default value is llama3.1-70b

run = tru_app.add_run(run_config=run_config)

Request Parameters:

• run_config: RunConfig: contains the configuration for the run.

Retrieve Run¶

Retrieves the run.

run = tru_app.get_run(run_name=run_name)

Request parameters:

• run_name: str: name of the run

View Run metadata¶

Describes the details of the run.

run.describe()

Invoke Run¶

You can invoke the run using the run.start() function. It reads the inputs from the dataset specified in the run configuration, invokes the application for each input, generates the traces, and ingests the information for storage in your Snowflake account. run.start() is a blocking call until the application is invoked for all inputs in your dataset and ingestion is completed or timed out.

run.start()  # if source_type is "TABLE"

run.start(input_df=user_input_df)  # if source_type is "DATAFRAME"

Request Parameters:

• input_df: DataFrame (Optional): is a pandas dataframe from the SDK. If the source_type in run configuration is specified as DATAFRAME, this field is mandatory. If the source_type is TABLE, this field is not required.

Compute metrics¶

You can start metric computations using run.compute_metrics() after the application is invoked and all traces are ingested. As long as the status of the run is INVOCATION_IN_PROGRESS, computation cannot be started. Once the status is INVOCATION_COMPLETED or INVOCATION_PARTIALLY_COMPLETED, run.compute_metrics() can be initiated. run.compute_metrics() is an asynchronous non-blocking function. You can call compute_metrics multiple times on the same run with a different set of metrics, and each call will trigger a new computation job. Note that metrics once computed cannot be re-computed again for the same run.

run.compute_metrics(metrics=[
    "coherence",
    "answer_relevance",
    "groundedness",
    "context_relevance",
    "correctness",
])

Request Parameters:

• metrics: List[str]: list of string names of the metrics listed in Metrics. The name of metrics should be specified in snake cases. i.e. Context Relevance should be specified as context_relevance.

Check Run Status¶

You can check the status of the run after it is in progress. The list of statuses are in Run Status section.

run.get_status()

Cancel Run¶

You can cancel an existing run using run.cancel(). This operation will prevent any future updates to the run, including run status and metadata fields.

run.cancel()

Delete Run¶

You can delete an existing run using run.delete(). This operation deletes the metadata associated with the run and the evaluation results cannot be accessed. However, the traces and evaluations generated as part of the runs are not deleted and remain stored. Please refer to Observability data section for more information about storage and deletion of evaluation and traces.

run.delete()

List Runs for an application¶

You can see the list of all available runs corresponding to a specific TruApp application object using the list_runs() function.

tru_app.list_runs()

Response:

Return a list of all Runs created under the tru_app.