Skip to content

GPT-6.2 Sol and Astra review: should you switch?

CodeRabbit model evaluation: GPT-6.2 Sol and Astra.

It’s felt like a while since we had a new Astra to test. Our last GPT-6 Astra evaluation was a little over two months ago. OpenAI has kept shipping, including GPT-6 Cyber, while the improvements in Anthropic’s Opus 5.5 made returning to Claude feel like a good decision.

We’re asking agents to handle larger projects and work for hours at a time. Always-on agents such as Dots and Grok Bot can start delegated work from an event, then notify us when results are ready or our input is needed. As that work grows, token usage and the cost of finishing a task become harder to ignore.

OpenAI says its new GPT-6.2 Sol and Astra can handle demanding tasks with fewer output tokens. Much of the release conversation focused on image and video generation, so we wanted to see what that meant for software development. We evaluated two uses separately:

  • Code generation, where the models built and changed software.
  • Code review, where they inspected existing pull requests inside CodeRabbit’s pipeline.

Could either become our daily coding model or earn a place in our review harness?

Selected coding-related releases, September to November 2026. Summaries describe OpenAI’s announced focus, not comparable benchmark scores. Spacing does not represent elapsed time. OpenAI announcements and API changelog

Code review: benchmark results

We evaluated the final comments produced by CodeRabbit’s review pipeline, after verification and filtering. As we explained in how we bring a model online at CodeRabbit, finding a possible issue and deciding to publish it are separate jobs.

A known issue counts as caught when at least one regular actionable comment matches it. Nitpicks and outside-diff comments don’t qualify. Benchmark precision is the share of regular actionable comments that match known issues. Unmatched comments still need to be read to judge their accuracy.

Sol 6.2 versus Sol 6.1

We tested Sol on 81 known issues across 42 pull requests. The table brings together coverage, comment volume, recorded cost, and review time. The Sol 6.1 baseline also uses other models in the pipeline. The low- and high-reasoning runs replaced different models in different parts of that pipeline, so they don’t isolate the effect of reasoning level.

MeasureSol 6.1, high reasoningSol 6.2, low reasoningSol 6.2, high reasoning
Known issues caught, full 81-issue set58 of 81 (71.6%)59 of 81 (72.8%)60 of 81 (74.1%)
Regular actionable comments187202290
Total nitpicks111133
Benchmark precision35.3%32.7%25.9%
Mean recorded cost per review$0.772$0.830$1.146
Median full-review time7m 49s7m 44s7m 53s

Sol 6.2 at high reasoning caught two more known issues across the full set. Each configuration ran once, and the gain includes changes outside the part where we replaced Sol, so it does not establish a reliable model-only improvement.

Sol 6.2’s high-reasoning run had a recorded cost about 48% higher than the baseline, while median review time changed by four seconds. Frozen upstream calls used recorded responses at zero provider cost, and the remaining calls ran live. These figures describe replay-run costs and full-pipeline timings. They do not establish production review costs or the latency of code generation tasks.

Astra 6.2 versus Astra 6

MeasureGPT-6 AstraGPT-6.2 Astra
Known issues caught51 of 83 (61.5%)58 of 83 (69.9%)
Reported comments331847
Actionable-comment benchmark precision21.74%12.22%
Nitpick/assertive comments5215
Standard API input price, per million tokens$10$8
Cached API input price, per million tokens$1$0.40
Standard API output price, per million tokens$50$40

GPT-6.2 Astra’s 847 reported comments include 589 regular actionable comments and 258 findings in summary sections. GPT-6 Astra’s comment total has not been reconciled to those categories. Nitpick/assertive comments are separate. These results do not establish a model-only improvement between the Astra versions.

In the 83-issue benchmark, the baseline configuration used Sol 6.1 for most scored issues, alongside other models. It caught 54 of 83 known issues, compared with the Astra integration’s 58. Astra picked up 11 known issues the baseline missed and missed seven that the baseline caught, leaving a net gain of four.

GPT-6.2 Astra configuration’s median review time was 25 seconds longer than its baseline’s 5m 37s, about 7.4%. That measures the full pipeline, including verification, and cannot establish that Astra itself is slower. Only nine of 83 cases had all costs recorded, compared with 58 in that baseline, so the run cannot establish a reliable cost per completed review.

According to OpenAI’s pricing announcement, standard input and output API rates fell by 20%, and cached input by 60%. These rates exclude long-context premiums and service-tier adjustments. Lower token prices alone do not establish lower review costs, which also depend on delegated calls and retries.

More comments need to earn their place

With GPT-6.2 Astra, regular actionable comments increased from the baseline’s 253 to 589. Of those 589 comments, 72 matched known issues. Multiple comments can match the same issue, which is why that count differs from the 58 distinct issues caught. The remaining 517 comments may include useful problems outside the benchmark’s labels. Understanding their value requires reading them.

Sol 6.2’s high-reasoning configuration raises the same question. Comments increased from 187 to 290, about 55%, while the full pipeline caught two more known issues overall. We’d inspect the additional comments before treating the higher volume as a benefit.

Every additional comment asks someone to separate a useful finding from noise. A PR filled with low-value feedback makes that harder. Authors may eventually stop reading or addressing the comments, leaving useful findings ignored too.

Our next experiment would require the reviewer to explain what triggers a problem and what the author would observe before posting an inline comment. We’d then sample both the comments kept and those removed to check whether the filter discards useful findings. As our framework for evaluating AI code review tools explains, the goal is feedback that leads to a meaningful code change.

Code generation: our hands-on experience

We used both models to build interfaces and work on projects. These are observations from our coding sessions. We didn’t run a matched code-generation benchmark.

Astra often produced a useful first interface from a short prompt. It chose layouts without waiting for clarification. Prompt quality was still key when working within an existing design. On longer tasks, Astra returned to the original goal after detours and brought delegated investigations back into the work without us having to restate the brief.

Sol 6.2 needed more direction. Its initial UI designs weren’t an improvement over 6.1 in our sessions, and parts of the implementation remained rough even with a reference. Small changes sometimes prompted long pauses before implementation. It also repeatedly gave us incorrect configurations when setting up projects from scratch, so we had to verify that they actually started.

Recall and precision across our reviews

The chart below shows recall and precision results from this review and our Opus 5.5 and Fable 5.1 evaluations, ordered from highest to lowest reported recall. The benchmark sets and scoring definitions differ across reviews, so this is not a head-to-head comparison.

Select a model or point for exact values.
Precision (%)
Recall (%)
GPT-6.2 Sol, high reasoningRecall 74.1% Precision 25.9%

Should you switch for code review?

We’d keep the current review defaults while testing both models further. Astra’s additional catches came with a large increase in comments, and the integration needs a comparison with fixed routing. Sol’s small measured gain came with more comments and a higher recorded cost. Before switching, read the additional findings and retest the issues each configuration stopped catching.

We’re also working on our Fable 5.2 review.

About the authors

Share

Share on RedditShare on XShare on LinkedIn

Catch the latest, right in your inbox.

Add us to your feed.

GetStarted in2 clicks.