Michael JohnsonIn COBOL systems, every behavioral detail matters: a program may finish without an error yet round a sum differently or handle a missing record another way. On October 19, 2026, OpenAI published results from two programming evaluations focused on languages with less training data: the open MultiLangBench across 20 languages and a separate COBOL migration test.
The announcement’s practical value lies in the opportunity to examine the benchmark itself and compare the reported metrics with specific tasks: creating programs in languages underrepresented in training data and preserving legacy code behavior during migration.
Two evaluations, two different tasks
MultiLangBench tests whether a model can build a SQLite-compatible database engine from scratch in a given language. The task involves organizing code, managing state, handling errors, and adding features while preserving existing behavior. Results are measured by the share of tests passed; OpenAI also reports the cost of API calls.
To compare languages, the tests use a common check based on a database file: an implementation receives an input database and an SQL query, then the modified database is checked against the expected result. In the published benchmark, OpenAI adapted SQLite tests and reported an evaluation across 20 languages. Developers can explore the code and results in the MultiLangBench repository.
A separate evaluation focuses on migrating existing COBOL programs to C++, Java, Kotlin, Python, or Rust. Success here depends on precisely preserving observable behavior: OpenAI used hidden tests that require byte-for-byte identical output. The company specifically notes the importance of details such as rounding rules and handling missing records.
What the published figures mean
According to OpenAI, GPT‑6 Astra achieved an average pass rate of 87.4% across 20 languages in MultiLangBench at the highest reasoning level; the company reports 26.8% for GPT‑5.6 Sol. In the COBOL migration test, GPT‑6.1 Sol achieved 86.6% accuracy at the low reasoning level, compared with 5.4% for GPT‑5.6 Sol in the same mode. At a higher reasoning level, GPT‑6.1 Sol reached 97.9%.
These figures apply to the two test scenarios described. MultiLangBench evaluates the creation of a limited SQLite-compatible engine, while the COBOL test measures behavior preservation on a set of programs checked with hidden tests. The results can therefore help teams choose candidates for their own trials, but they are not a direct assessment of any corporate codebase. OpenAI described the methodology and results in its October 19 research post.
How to apply the benchmark to a modernization project
For a team considering a legacy system migration, a useful next step is to turn critical business rules into a testable set of behaviors. Assemble representative inputs and expected outputs, including boundary values, rounding, empty or missing records, and errors. Record the original program’s results, then run candidate models against the same set.
Compare more than just the share of successful checks. Track call costs, the number of manual fixes, and the types of discrepancies; separately check interfaces with external systems and dependencies that the test sample may not cover. This process shows where a model can help investigate and rewrite a module, and where critical logic should remain under manual oversight and specialized testing.
The October 19 announcement added a reproducible starting point for evaluating code models on languages with fewer available examples. For practical decisions, the most important step is to compare the open benchmark with the system’s own rules: migration suitability depends on behavior matching, not simply familiarity with the target language’s syntax.