GPU machine
A clean machine to work on over SSH.
What it runs
# setup: nothing to install # run nvidia-smi # then sky status <name> ssh <name>
Loading
What is on a GPU the moment it is up.
A clean machine to work on over SSH.
# setup: nothing to install # run nvidia-smi # then sky status <name> ssh <name>
Current PyTorch, checked against every GPU.
# setup (once per node)
pip install --upgrade torch torchvision torchaudio
# run
python -c "import torch; print('torch', torch.__version__, 'cuda', torch.cuda.is_available(), 'gpus', torch.cuda.device_count())"
# then
sky logs <name>
ssh <name>Jupyter Lab on the GPU, with a starter notebook that logs to Experiments.
# setup (once per node)
set -e
pip install --upgrade jupyterlab
# Kovara's experiment tracker: one Python file. The check below refuses any file but the one this task was written for.
curl -fsSL https://www.kovaragrid.com/sdk/kovara_track.py -o ~/kovara_track.py
echo "80f9618d4570c156b1c0c02b93ee1cf0b64b10e1e5c43762a8b13a9c1ab27180 $HOME/kovara_track.py" | sha256sum -c -
# A starter notebook: check the GPU, log a first run.
cat > ~/kovara_start.ipynb <<'KOVARA_NOTEBOOK'
{"cells":[{"cell_type":"markdown","metadata":{},"source":"# Your GPU notebook\n\nThis notebook runs on the GPU machine you launched. The machine is in your own cloud account and is billed by your provider until it is deleted.\n\nRuns you log from here appear in Kovara under **Experiments**."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"!nvidia-smi"},{"cell_type":"markdown","metadata":{},"source":"## Track a run\n\n`kovara_track.py` is already on this machine. When the machine was launched with tracking on, the key is set and the next cell starts a run. Without a key it says so and everything else still works."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"import kovara_track as kt\n\nrun = kt.init(project=\"notebooks\", config={\"learning_rate\": 1e-3})\nprint(run.url or \"This run is not being tracked.\")"},{"cell_type":"markdown","metadata":{},"source":"Log metrics from your training loop with `run.log({...}, step=step)`. The loop below logs a made-up curve so you can see it arrive; replace it with your own."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"import math\n\nfor step in range(200):\n run.log({\"demo/loss\": math.exp(-step / 40)}, step=step)"},{"cell_type":"markdown","metadata":{},"source":"Keep results with the run, register what you produced, and end the run when you are done."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"run.summary({\"note\": \"first notebook run\"})\n# run.log_model(\"my-model\", \"s3://my-bucket/models/my-model/v1\") # an address in your own storage\nrun.finish()"},{"cell_type":"markdown","metadata":{},"source":"Guide: https://www.kovaragrid.com/docs/experiments"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":5}
KOVARA_NOTEBOOK
# run
cd ~ && jupyter lab --no-browser --ip 127.0.0.1 --port 8888
# KOVARA_PROJECT=notebooks
# then
ssh -L 8888:localhost:8888 <name>
sky logs <name> # copy the URL with the token, open it on http://localhost:8888, then open kovara_start.ipynbLoRA fine-tuning of an open model on your data, logged to Experiments.
# setup (once per node)
set -e
pip install --upgrade torch transformers datasets peft accelerate
# Kovara's experiment tracker: one Python file. The check below refuses any file but the one this task was written for.
curl -fsSL https://www.kovaragrid.com/sdk/kovara_track.py -o ~/kovara_track.py
echo "80f9618d4570c156b1c0c02b93ee1cf0b64b10e1e5c43762a8b13a9c1ab27180 $HOME/kovara_track.py" | sha256sum -c -
# The job itself. Everything it does is in this file.
cat > ~/finetune.py <<'KOVARA_FINETUNE'
"""Kovara fine-tuning job: LoRA fine-tuning of an open model with Hugging Face Transformers and PEFT.
Written onto the machine by the "Fine-tune a model" launch template. Every setting is an environment variable:
MODEL base model on Hugging Face (default Qwen/Qwen2.5-0.5B-Instruct)
DATASET a dataset on Hugging Face, or a .jsonl / .json / .csv / .parquet file on the machine or a mounted bucket
DATASET_SPLIT which split of a Hugging Face dataset (default train)
MAX_EXAMPLES use at most this many examples, chosen at random; 0 for all (default 2000)
EPOCHS, LEARNING_RATE, BATCH_SIZE, GRADIENT_ACCUMULATION, MAX_SEQ_LENGTH, LORA_R, SEED
OUTPUT_URI a bucket mounted at /output; the adapter is written to /output/<name> and registered under this address
OUTPUT_DIR where the adapter is written instead (default /output/<name> with OUTPUT_URI, else ~/outputs/<name>)
MODEL_NAME the name the adapter gets in the registry (default: the base model's name with -adapter)
HF_TOKEN only for gated models and private datasets
A row of the dataset may be chat messages ("messages" or "conversations"), an instruction with an answer
("instruction"/"prompt"/"question" with "response"/"output"/"completion"/"answer", and optionally "context"/"input"),
or plain "text". Rows in another shape are skipped and counted.
"""
import os
import re
import sys
import time
DEFAULTS = {"MODEL": "Qwen/Qwen2.5-0.5B-Instruct", "DATASET": "databricks/databricks-dolly-15k", "DATASET_SPLIT": "train", "MAX_EXAMPLES": 2000,
"EPOCHS": 1.0, "LEARNING_RATE": 2e-4, "BATCH_SIZE": 8, "GRADIENT_ACCUMULATION": 2, "MAX_SEQ_LENGTH": 1024, "LORA_R": 16, "SEED": 42}
LIMITS = {"MAX_EXAMPLES": (0, 10_000_000), "EPOCHS": (0.01, 100), "LEARNING_RATE": (1e-7, 1e-1), "BATCH_SIZE": (1, 1024), "GRADIENT_ACCUMULATION": (1, 1024),
"MAX_SEQ_LENGTH": (16, 131072), "LORA_R": (1, 1024), "SEED": (0, 2**31 - 1)}
FILE_KINDS = {".jsonl": "json", ".json": "json", ".csv": "csv", ".parquet": "parquet"}
def settings(env):
"""The job's settings from the environment, checked. Raises SystemExit with a plain message on a bad value."""
out = {}
for key, default in DEFAULTS.items():
raw = (env.get(key) or "").strip()
if not raw:
out[key.lower()] = default
continue
if isinstance(default, str):
out[key.lower()] = raw
continue
try:
value = type(default)(raw)
except ValueError:
raise SystemExit("%s must be a number, not %r" % (key, raw))
low, high = LIMITS[key]
if not low <= value <= high:
raise SystemExit("%s must be between %s and %s, not %s" % (key, low, high, raw))
out[key.lower()] = value
name = (env.get("KOVARA_RUN_NAME") or "").strip() or "fine-tune-" + time.strftime("%Y%m%d-%H%M%S", time.gmtime())
out["name"] = name
uri = (env.get("OUTPUT_URI") or "").strip().rstrip("/")
# With a bucket to save to, the launch template mounts it at /output.
out["output_dir"] = os.path.expanduser((env.get("OUTPUT_DIR") or "").strip() or (("/output/" + name) if uri else os.path.join("~", "outputs", name)))
out["output_uri"] = (uri + "/" + name) if uri else ""
out["model_name"] = registry_name((env.get("MODEL_NAME") or "").strip() or out["model"].rsplit("/", 1)[-1] + "-adapter")
return out
def registry_name(text):
"""A registry name from any text: lower-case letters, digits and . _ -, starting with a letter or digit, at most 63."""
name = re.sub(r"[^a-z0-9._-]+", "-", text.lower()).strip("-._")[:63].rstrip("-._")
return name or "adapter"
def dataset_source(dataset):
"""("file", loader, path) for a data file, ("hub", None, id) for a Hugging Face dataset."""
lower = dataset.lower()
for ending, loader in FILE_KINDS.items():
if lower.endswith(ending):
return "file", loader, os.path.expanduser(dataset)
if dataset.startswith(("/", "~", ".")):
raise SystemExit("DATASET %r is a path but not a .jsonl, .json, .csv or .parquet file" % dataset)
return "hub", None, dataset
def _first(row, *names):
for name in names:
value = row.get(name)
if isinstance(value, str) and value.strip():
return value.strip()
return None
def to_example(row):
"""("chat", [messages]) or ("text", text) for one dataset row, or None when the row has no usable shape."""
turns = row.get("messages") or row.get("conversations")
if isinstance(turns, list) and turns:
roles = {"human": "user", "gpt": "assistant", "bot": "assistant", "model": "assistant"}
messages = []
for turn in turns:
if not isinstance(turn, dict):
return None
role = str(turn.get("role") or turn.get("from") or "").lower()
content = turn.get("content") if turn.get("content") is not None else turn.get("value")
if not isinstance(content, str) or not content.strip():
continue
messages.append({"role": roles.get(role, role if role in ("system", "user", "assistant") else "user"), "content": content.strip()})
return ("chat", messages) if any(m["role"] == "assistant" for m in messages) else None
prompt = _first(row, "instruction", "prompt", "question")
answer = _first(row, "response", "output", "completion", "answer")
if prompt and answer:
context = _first(row, "context", "input")
return "chat", [{"role": "user", "content": prompt + ("\n\n" + context if context else "")}, {"role": "assistant", "content": answer}]
text = _first(row, "text")
return ("text", text) if text else None
def render(example, tokenizer):
"""The training text of one example: the model's own chat format when it has one, a plain layout otherwise."""
kind, body = example
end = getattr(tokenizer, "eos_token", None) or ""
if kind == "text":
return body + end
if getattr(tokenizer, "chat_template", None):
return tokenizer.apply_chat_template(body, tokenize=False)
labels = {"system": "System", "user": "User", "assistant": "Assistant"}
return "\n\n".join("### %s:\n%s" % (labels[m["role"]], m["content"]) for m in body) + end
def train(cfg, run, tracker):
import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, DataCollatorForLanguageModeling, Trainer, TrainingArguments
from transformers.trainer_utils import get_last_checkpoint
token = os.environ.get("HF_TOKEN") or None
tokenizer = AutoTokenizer.from_pretrained(cfg["model"], token=token)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
kind, loader, source = dataset_source(cfg["dataset"])
data = load_dataset(loader, data_files=source, split="train") if kind == "file" else load_dataset(source, split=cfg["dataset_split"], token=token)
rows = len(data)
if cfg["max_examples"] and rows > cfg["max_examples"]:
data = data.shuffle(seed=cfg["seed"]).select(range(cfg["max_examples"]))
def encode(row):
example = to_example(row)
# A chat template writes the model's special tokens itself; adding them again would double the start token.
templated = example is not None and example[0] == "chat" and bool(getattr(tokenizer, "chat_template", None))
return tokenizer(render(example, tokenizer) if example else "", truncation=True, max_length=cfg["max_seq_length"], add_special_tokens=example is not None and not templated)
chosen = len(data)
data = data.map(encode, remove_columns=data.column_names).filter(lambda row: len(row["input_ids"]) > 1)
if len(data) == 0:
raise SystemExit("No usable examples in %s: rows need messages, an instruction with a response, or text" % cfg["dataset"])
parts = data.train_test_split(test_size=min(0.1, 200.0 / len(data)), seed=cfg["seed"]) if len(data) >= 200 else None
train_set, eval_set = (parts["train"], parts["test"]) if parts else (data, None)
cuda = torch.cuda.is_available()
bf16 = bool(cuda and torch.cuda.is_bf16_supported())
model = AutoModelForCausalLM.from_pretrained(cfg["model"], token=token, torch_dtype=torch.bfloat16 if bf16 else torch.float32)
model = get_peft_model(model, LoraConfig(r=cfg["lora_r"], lora_alpha=2 * cfg["lora_r"], lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM"))
run.update_config({"dataset_rows": rows, "examples": len(train_set), "eval_examples": len(eval_set) if eval_set else 0, "skipped_rows": chosen - len(data),
"trainable_parameters": sum(p.numel() for p in model.parameters() if p.requires_grad), "total_parameters": sum(p.numel() for p in model.parameters()),
"precision": "bf16" if bf16 else "float32"})
out = cfg["output_dir"]
os.makedirs(out, exist_ok=True)
args = TrainingArguments(
output_dir=out, num_train_epochs=cfg["epochs"], learning_rate=cfg["learning_rate"], per_device_train_batch_size=cfg["batch_size"],
per_device_eval_batch_size=cfg["batch_size"], gradient_accumulation_steps=cfg["gradient_accumulation"], lr_scheduler_type="cosine", warmup_ratio=0.03,
logging_steps=10, save_strategy="epoch", save_total_limit=2, eval_strategy="epoch" if eval_set else "no", bf16=bf16, report_to="none", seed=cfg["seed"])
trainer = Trainer(model=model, args=args, train_dataset=train_set, eval_dataset=eval_set,
data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False), callbacks=[tracker.hf_callback(run)])
started = time.time()
# A job that was interrupted (a spot machine taken away) continues from its last saved epoch when the output is on a bucket.
stats = trainer.train(resume_from_checkpoint=get_last_checkpoint(out))
final = os.path.join(out, "adapter")
model.save_pretrained(final)
tokenizer.save_pretrained(final)
result = {"train_loss": float(stats.training_loss), "train_seconds": round(time.time() - started), "adapter": (cfg["output_uri"] + "/adapter") if cfg["output_uri"] else final}
if eval_set is not None:
result["eval_loss"] = float(trainer.evaluate()["eval_loss"])
return result
def main():
cfg = settings(os.environ)
dataset_source(cfg["dataset"])
import kovara_track as tracker
public = {key: cfg[key] for key in ("model", "dataset", "dataset_split", "max_examples", "epochs", "learning_rate", "batch_size", "gradient_accumulation", "max_seq_length", "lora_r", "seed")}
run = tracker.init(project=os.environ.get("KOVARA_PROJECT") or "fine-tuning", name=cfg["name"], job_type="fine-tune", config=public)
try:
result = train(cfg, run, tracker)
except BaseException as error:
if type(error).__name__ == "OutOfMemoryError":
message = "Out of GPU memory. Lower BATCH_SIZE or MAX_SEQ_LENGTH, or choose a GPU with more memory."
print(message, file=sys.stderr)
else:
message = str(error)[:500] or type(error).__name__
run.summary({"error": message})
run.finish("stopped" if isinstance(error, KeyboardInterrupt) else "failed")
raise
run.summary(result)
if cfg["output_uri"]:
run.log_model(cfg["model_name"], result["adapter"], metadata={"base_model": cfg["model"], "dataset": cfg["dataset"], "lora_r": cfg["lora_r"]},
notes="LoRA adapter of %s, fine-tuned on %s" % (cfg["model"], cfg["dataset"]))
run.finish()
print("Adapter written to %s" % result["adapter"])
if not cfg["output_uri"]:
print("It is on this machine's disk only. Copy it off before the machine is deleted, or launch with a bucket to save to.")
if __name__ == "__main__":
main()
KOVARA_FINETUNE
# run
cd ~ && python finetune.py
# MODEL=Qwen/Qwen2.5-0.5B-Instruct
# DATASET=databricks/databricks-dolly-15k
# KOVARA_PROJECT=fine-tuning
# KOVARA_RUN_NAME={name}
# EPOCHS=1
# LEARNING_RATE=2e-4
# BATCH_SIZE=8
# MAX_SEQ_LENGTH=1024
# LORA_R=16
# MAX_EXAMPLES=2000
# then
sky logs <name> # follow the job
ssh <name> # the adapter is under ~/outputs, or in your bucketAn OpenAI-compatible server for an open model.
# setup (once per node) pip install --upgrade vllm # run vllm serve "$MODEL" --host 127.0.0.1 --port 8000 # MODEL=Qwen/Qwen2.5-7B-Instruct # then ssh -L 8000:localhost:8000 <name> curl http://localhost:8000/v1/models
The vLLM server as an autoscaled endpoint.
# setup (once per node) pip install --upgrade vllm # run vllm serve "$MODEL" --host 0.0.0.0 --port 8000 # MODEL=Qwen/Qwen2.5-7B-Instruct # then sky serve status <name> # shows the endpoint address curl http://<endpoint>/v1/models
Machines start from SkyPilot's default image for the cloud, with GPU drivers installed. To use your own environment, enter a public container image on Launch GPU under More options, written as docker:repository/name:tag.