Tuning TIGRE for your computer#

Warning

This page is for advanced users. The changes below edit TIGRE’s CUDA source and need a rebuild. They give modest gains at best, and wrong values can make TIGRE fail. Try them only if speed is critical.

All the files below are in Common/CUDA. After each change, rebuild TIGRE (pip install . from the repository for Python, Compile for MATLAB) and time the operation again on data the size of your real problem.

Skip the free-memory check (largest gain)#

Before every CUDA call, TIGRE asks each GPU how much memory is free, to plan how to split the problem (GPUs and memory). This takes up to about 0.3 seconds per GPU per call. For a backprojection that takes around a second, that is a large share.

If you always run on the same, otherwise idle GPUs, you can replace the query with a fixed number. Each .cu file that launches GPU work ends with a function checkFreeMemory. Comment out its body and set the free memory, in bytes, directly:

*mem_GPU_global = 11000000000;  // for example, about 11 GB

Use about 95% of the GPU’s memory, and less if other programs (such as the desktop) use the GPU. A value larger than the memory actually free will make TIGRE fail. The function appears in Siddon_projection.cu, ray_interpolated_projection.cu, voxel_backprojection.cu, voxel_backprojection2.cu, tv_proximal.cu, GD_TV.cu and GD_AwTV.cu.

Kernel launch sizes (small gains)#

The constants below set how much work each GPU thread or block does. The defaults were tuned on an NVIDIA GTX 1080 Ti. Expect at most a few per cent from changing them, and often nothing. Change one at a time, by small steps.

Operation

File

Constants

Backprojection, FDK weights

voxel_backprojection.cu

PROJ_PER_KERNEL, VOXELS_PER_THREAD

Backprojection, matched weights

voxel_backprojection2.cu

PROJ_PER_KERNEL, VOXELS_PER_THREAD

Forward projection, Siddon

Siddon_projection.cu

PROJ_PER_BLOCK, PIXEL_SIZE_BLOCK

Forward projection, interpolated

ray_interpolated_projection.cu

PROJ_PER_BLOCK, PIXEL_SIZE_BLOCK

TV denoising

tv_proximal.cu

MAX_BUFFER, BLOCK_SIZE

TV gradient steps (ASD-POCS family)

GD_TV.cu, GD_AwTV.cu

MAX_BUFFER

For the forward projectors, a block has PIXEL_SIZE_BLOCK × PIXEL_SIZE_BLOCK × PROJ_PER_BLOCK threads, which must not exceed your GPU’s limit (1024 on current NVIDIA GPUs). For TV denoising, BLOCK_SIZE³ must not exceed it either; MAX_BUFFER usually has the larger effect.

A simple way to time each operation:

import time
import numpy as np
import tigre
from tigre.utilities import sample_loader
from tigre.utilities.im_3d_denoise import im3ddenoise

geo = tigre.geometry_default()          # use your own geometry here
angles = np.linspace(0, 2 * np.pi, 360, endpoint=False)
img = sample_loader.load_head_phantom(geo.nVoxel)
proj = tigre.Ax(img, geo, angles)

for name, op in [
    ("Ax", lambda: tigre.Ax(img, geo, angles)),
    ("Atb", lambda: tigre.Atb(proj, geo, angles)),
    ("TV", lambda: im3ddenoise(img, iter=50)),
]:
    t = time.perf_counter(); op(); print(name, time.perf_counter() - t)
geo = defaultGeometry();               % use your own geometry here
angles = linspace(0, 2*pi, 360);
img = headPhantom(geo.nVoxel);
proj = Ax(img, geo, angles);

tic; Ax(img, geo, angles); toc
tic; Atb(proj, geo, angles); toc
tic; im3DDenoise(img, 'TV', 50, 15); toc