create-inspect-task

9

niznik-dev

Create custom inspect-ai evaluation tasks through interacted, guided workflow.

run-experiment

9

niznik-dev

Execute the complete experimental workflow - model optimization followed by evaluation - for all runs in a scaffolded experiment. Use after scaffold-experiment to submit jobs to SLURM.

summarize-experiment

9

niznik-dev

Create a lightweight summary of experiment results from a completed (fine-tuned and evaluated) experiment. Use after run-experiment to capture key metrics from the experiment in textual form.

create-meeting-agenda

9

niznik-dev

Create weekly software meeting agenda in the wiki repo.

design-experiment

9

niznik-dev

Plan LLM fine-tuning and evaluation experiments. Use when the user wants to design a new experiment, plan training runs, or create an experiment_summary.yaml file.

scaffold-experiment

9

niznik-dev

Set up complete experimental infrastructure for all runs in a designed experiment. Orchestrates parallel generation of fine-tuning configs (via scaffold-torchtune) and evaluation configs (via scaffold-inspect). Use after design-experiment to prepare configs before running experiments.