A Custom Env for Your Real Task

Warm-up: the env contract

What you'll do

Build a custom environment for one of your own software development workflows. Instead of training on SearchQA's trivia questions, you'll define tasks that match what your agent actually does — and produce a skill that makes your daily work better.

The officeqa env is the simplest reference to copy from. We'll follow the same pattern, adapted to your domain.

Step 1 — Define your task domain

Pick a recurring workflow where your agent has a checkable correctness signal. Good candidates:

Step 2 — Create the data

You need 20–50 task items split into train/val/test. Each item is a dict with at minimum:

# Each item needs:
{
    "id": "unique_string",
    "task_type": "your_domain",
    # ... your domain-specific fields:
    "input": "...",       # what the agent receives
    "ground_truth": "..." # what correct looks like
}

Create the split directories:

mkdir -p ~/skillopt-repo/data/my_env_split/{train,val,test}

# Write your items as JSON — split ratio ~2:1:7 (train:val:test)
# train: items the optimizer sees during reflection
# val: the gate scores against these (NEVER trained on)
# test: final evaluation only

Step 3 — Implement the three files

dataloader.py

from skillopt.datasets.base import SplitDataLoader
from pathlib import Path
import json


class MyEnvDataLoader(SplitDataLoader):
    """Load items from JSON files in each split directory."""

    def load_split_items(self, split_path: str) -> list[dict]:
        json_files = sorted(Path(split_path).glob("*.json"))
        if not json_files:
            raise FileNotFoundError(f"No .json in {split_path}")
        with json_files[0].open() as f:
            return json.load(f)

rollout.py

from skillopt.model import chat_target
import os, json
from pathlib import Path


def _score(prediction: str, ground_truth: str) -> tuple[int, float]:
    """Your scoring function. Binary is simplest."""
    # Example: exact match (replace with your domain logic)
    match = prediction.strip() == ground_truth.strip()
    return (1 if match else 0), (1.0 if match else 0.0)


def run_batch(*, items, skill_content, out_root,
              workers=4, max_completion_tokens=4096) -> list[dict]:
    os.makedirs(out_root, exist_ok=True)
    results = []
    for item in items:
        prediction, _usage = chat_target(
            system=skill_content,
            user=item["input"],
            max_completion_tokens=max_completion_tokens,
        )
        hard, soft = _score(prediction, item["ground_truth"])
        results.append({
            "id": item["id"],
            "hard": hard,
            "soft": soft,
            "predicted_answer": prediction,
            "task_type": item.get("task_type", "my_env"),
        })
    Path(out_root, "rollouts.json").write_text(json.dumps(results, indent=2))
    return results

initial.md (seed skill)

# My Task Skill

You are a software development assistant. When given a task:
1. Read the input carefully
2. Produce the requested output
3. Follow the project's conventions

The seed skill can be minimal — the optimizer will grow it. But giving it a reasonable starting point (rather than empty) helps the first epoch.

Step 4 — Wire it up

Create the adapter (copy from officeqa/adapter.py and change the class names and dataloader import), then register it in scripts/train.py:

# In scripts/train.py → _register_builtins()
try:
    from skillopt.envs.my_env.adapter import MyEnvAdapter
    _ENV_REGISTRY["my_env"] = MyEnvAdapter
except ImportError:
    pass

Create a config at configs/my_env/default.yaml:

_base_: ../_base_/default.yaml

train:
  batch_size: 8
  num_epochs: 2

env:
  name: my_env
  skill_init: skillopt/envs/my_env/skills/initial.md
  split_mode: split_dir
  split_dir: data/my_env_split
  workers: 4
  max_completion_tokens: 4096

Step 5 — Train and evaluate

# Train
python scripts/train.py --config configs/my_env/default.yaml

# Evaluate the produced skill on held-out test
python scripts/evaluate.py \
  --config configs/my_env/default.yaml \
  --skill outputs/my_env/skills/best_skill.md \
  --split test

The best_skill.md that comes out is trained on your tasks. Read it with the rubric — and compare it to the generic seed skill you started with. The difference is what SkillOpt learned about your specific workflow.