Report Generation

The pypts framework includes an automated reporting system that generates a detailed CSV report of the execution flow and results of a recipe run. This process runs incrementally in the background.

Initialization

When a pypts recipe execution is initiated via the pts.run_pts function:

  1. A SimpleQueue named report_queue is created. This queue serves as the communication channel between the main recipe execution thread and the reporting thread.

  2. A report output directory is determined. Reports are written to ~/pts_reports regardless of the directory where the pypts application was launched. The directory is created if it doesn’t exist.

  3. A dedicated daemon thread is started, running the report.report_listener function. This function is passed the report_queue and the path to the output directory.

  4. The report_queue is passed to the recipe.Runtime object, making it accessible during recipe execution.

Sending Results to the Queue

As the recipe executes, each time a recipe.Step finishes its execution within the Step.run method:

  1. A recipe.StepResult object, containing details about the step’s execution (inputs, outputs, result status, errors, UUIDs, etc.), is created.

  2. Immediately after the post_run_step event is emitted, this StepResult object is placed onto the report_queue using runtime.report_queue.put(step_result).

Report Listener (report_listener)

The report_listener function runs continuously in its own thread, monitoring the report_queue. Its primary responsibilities are:

  1. Initialization: Upon starting, it instantiates a report.Report object, passing it the designated output directory. The Report object handles the creation and management of the actual report file (report.csv).

  2. Waiting for Results: It blocks, waiting for items to appear on the report_queue using result_queue.get().

  3. Processing Results: * If the received item is a StepResult object, it calls report_manager.add_step_result(item) to process and write the result to the report file. * If the received item is the special sentinel object report.STOP_LISTENER, it signifies the end of the recipe execution. * Any other unexpected item type is logged as a warning.

  4. Termination: Upon receiving the STOP_LISTENER sentinel, the listener loop terminates.

Report Manager (Report Class)

The report.Report class manages the actual file I/O for the CSV report:

  1. Initialization (`__init__`): * Takes the output directory path. * Creates the directory if needed. * Opens the report.csv file in write mode (‘w’), effectively overwriting any previous report in that location for the current run. * Creates a csv.DictWriter instance, configured with the predefined CSV headers. * Writes the header row to the CSV file.

  2. Adding Results (`add_step_result`): * Takes a StepResult object as input. * Uses internal helper functions (_result_to_dict, _flatten_single_result) to convert the potentially nested StepResult object into a flat dictionary suitable for a single CSV row. Complex data structures like inputs and outputs are JSON-serialized. * Copies any image files referenced in result.image_paths into <output_dir>/img/ (see Image Outputs below). * Writes the flattened dictionary as a row to the CSV file using the DictWriter. * Flushes the file buffer to ensure the data is written to disk promptly.

  3. Finalization (`finish_reports`): * Called by the report_listener just before it exits. * Closes the CSV file handle, ensuring all data is saved.

After the CSV is finalised, report_listener generates an HTML report (report_{timestamp}.html) from the CSV data. The HTML report includes:

  • A Run Context block (recipe name, file, serial number, pypts version).

  • A Summary with the total number of steps.

  • A colour-coded Details table (PASS = green, FAIL/ERROR = red, SKIP = yellow).

  • An Images section at the bottom, embedding any images returned by test steps (see below).

Stopping the Listener

When the main recipe execution completes in recipe.Recipe.run:

  1. Before returning the final results, it imports the report.STOP_LISTENER sentinel object.

  2. It places this sentinel onto the report_queue using runtime.report_queue.put(STOP_LISTENER).

  3. This signals the report_listener thread to stop waiting for more results, finalize the report, and exit.

Image Outputs

Test methods executed by PythonModuleStep can return image file paths (PNG, JPG, SVG, etc.) as part of their output dictionary. The framework automatically copies these files into the report directory and embeds them at the bottom of the HTML report with the corresponding step name and result as a caption.

How it works

  1. The test method returns a dictionary that includes a key mapped to an image file path.

  2. In the recipe’s output_mapping, that key is declared with type: image.

  3. When the step finishes, the framework stores the path in StepResult.image_paths.

  4. The Report.add_step_result() method copies the file to <output_dir>/img/<step_id>_<filename>.

  5. The relative path is recorded in the CSV (image_paths column).

  6. generate_html_report() embeds each image in an Images section at the bottom of the HTML report.

Example Python test method

import matplotlib.pyplot as plt
import tempfile, os

def run_signal_analysis(data=None, **kwargs):
    fig, ax = plt.subplots()
    ax.plot(data)
    ax.set_title("Signal")

    path = os.path.join(tempfile.gettempdir(), "signal_plot.png")
    fig.savefig(path)
    plt.close(fig)

    passed = max(data) < 1.0
    return {"chart": path, "passed": passed}

Corresponding recipe step

- steptype: PythonModuleStep
  step_name: Analyse Signal
  description: Analyse the signal and publish its plot.
  module: my_tests.py
  action_type: method
  method_name: run_signal_analysis
  input_mapping:
    data: {type: global, global_name: raw_signal}
  output_mapping:
    chart:
      type: image    # file is copied into the report and embedded in the HTML
    passed:
      type: passfail

Note

The type: image mapping does not affect the step’s pass/fail result — it only triggers file copying and HTML embedding. Combine it with passfail, equals, or range mappings on other output keys to get a verdict as usual.

Supported image formats: .png, .jpg, .jpeg, .gif, .bmp, .svg, .tiff, .webp.

Output directory structure after a run with images:

pts_reports/
├── report_2025-01-15_14h30.csv
├── report_2025-01-15_14h30.html
└── img/
    └── <step_id>_signal_plot.png