benchmarking-ml-models

Model performance benchmarking.

Quick Benchmark

import time
import torch

# Warmup
for _ in range(10):
    model(sample_input)

# CUDA calls are asynchronous. Finish warmup before starting the clock.
use_cuda = torch.cuda.is_available()
if use_cuda:
    torch.cuda.synchronize()

# Benchmark
start = time.perf_counter()
for _ in range(100):
    model(sample_input)
if use_cuda:
    torch.cuda.synchronize()
elapsed = time.perf_counter() - start

print(f"Avg latency: {elapsed/100*1000:.2f}ms")

Metrics

| Metric | Description | Command | |--------|-------------|---------| | Latency | Inference time | time.time() | | Throughput | Samples/sec | samples / elapsed | | Memory | VRAM usage | torch.cuda.max_memory_allocated() | | Accuracy | Model quality | accuracy_score(y_true, y_pred) |

Benchmark Script

The bundled helper has no inference or dataset adapter. Its run and evaluate commands intentionally exit nonzero before printing or saving results. Do not treat endpoint health, sleeps, or generated labels as model measurements.

# From this Skill's directory; currently verifies the fail-closed contract.
bash scripts/ml-benchmark.sh run \
  --url localhost:8001 \
  --model langdetector \
  --input ./sample.wav \
  --batch-size 32 \
  --runs 100

Before enabling that command, add a real, versioned inference adapter and test that every recorded sample comes from a completed model response. Accuracy also requires a dataset adapter that reads ground-truth labels and predictions.

Output Format

## Benchmark Results: {model_name}

| Metric | Value |
|--------|-------|
| Latency (p50) | 15.2ms |
| Latency (p99) | 22.1ms |
| Throughput | 65 samples/sec |
| Memory | 4.2 GB |
| Accuracy | 92.3% |

### Configuration
- GPU: NVIDIA A100
- Batch size: 32
- Precision: FP16

Compare Models

results = {}
for model_name, model in models.items():
    results[model_name] = benchmark(model)

# Generate comparison table

Best Practices

• Always warmup before measuring
• Use torch.cuda.synchronize() for GPU
• Report p50/p99 latencies
• Document hardware configuration