Benchmarks only matter when they look like the work you actually hand to a model. Solar Pro 4 is scored on three evaluations that each end in a deliverable rather than a paragraph: a task completed in a shell, a policy found and applied across a tool-calling conversation, and an answer assembled from documents that do not fit in a chat window.
The three scores
| Evaluation | What it measures | Solar Pro 4 |
|---|---|---|
| Terminal-Bench v2.1 | Multi-step jobs completed in a live terminal | 57 |
| τ³-Banking | Multi-turn tool use against a large policy knowledge base | 23 |
| AA-LCR | Reasoning across roughly 100k tokens of reports and filings | 71 |
Each of these is a several-fold step over Solar Pro 3.
Terminal-Bench v2.1 = 57: work that ends in a shell
Terminal-Bench puts the model in front of real shell tasks and grades the final state of the machine, not the explanation. A score of 57 is about how often a multi-step job — inspect, edit, run, verify — actually lands. The failure mode this measures is the one that costs money in production: a model that writes plausible commands but never verifies them.
τ³-Banking = 23: multi-turn tool calls that follow policy
τ³-Banking is a tool-calling evaluation: the model has to find the right policy in a large knowledge base and act on it across several turns, with the tools returning partial information. This is the shape of most agent work — check a record, confirm a rule, perform the action. The score rewards finishing the transaction rather than answering the first turn quickly.
AA-LCR = 71: reasoning across long documents
AA-LCR asks questions whose answers are scattered across roughly 100k tokens of long documents. It rewards synthesizing evidence from several files at once and, importantly, refusing to guess when a clause or a number is not present. With a 512K context window and up to 128K output tokens, the whole set of files can stay in one session.
What these numbers do not tell you
- They measure specific task families. Your workload may be easier or harder.
- They do not price the work: a higher completion rate usually lowers total cost because failed runs are paid for twice.
- They are a reason to run your own evaluation, not a substitute for it.
Try it on your own task
Sign in on the home page and start with a multi-step job you already know the answer to — a document set with a question that requires two files, or a small terminal task. Then compare token pricing on the pricing page if you plan to run it from the API.
FAQ
Are these benchmark numbers the official ones?
They are the figures Solar Pro 4 reports for these three evaluations and are used consistently across this site, including the overview of the model.
What is a good score on Terminal-Bench v2.1?
The useful comparison is against the model you run today, on the same tasks. A several-fold step over the previous generation on terminal work is the claim; verify it with your own shell tasks.
How do I compare the scores with Solar Pro 3?
See Solar Pro 4 vs Solar Pro 3 for what changed between the two generations and what to check before migrating.
Where can I call the model directly?
The API guide covers the OpenAI-compatible endpoint, the model name, and the context limits.
