Michael JohnsonA COBOL program can be ported to a modern language while preserving its syntax but changing its rules for calculating money. OpenAI devoted its October 19 post to this problem: the company reported results for its models on programming tests in COBOL, Fortran, and other languages with less training data, and released the MultiLangBench suite.
For teams maintaining legacy enterprise systems, the news is useful not only for its reported percentages but also for the two different kinds of evaluation. One measures how well models create programs from scratch in different languages; the other checks whether a COBOL port preserves observable behavior. These tasks require different tests and carry different implications for deployment risk.
Two tests, two engineering tasks
In MultiLangBench, a model must create a SQLite-compatible engine that accepts queries and modifies a database file. OpenAI reports the average percentage of tests passed across 20 languages: GPT‑6 Astra scored 87.4% at the highest reasoning level, while GPT‑5.6 Sol scored 26.8%. Each implementation is checked against the expected database contents; runtime and API cost are also evaluated.
The company published MultiLangBench on GitHub, including the testing infrastructure. The repository describes tests based on the SQLite TCL Tests suite and the rules for checking results. This lets developers inspect exactly which operations are included in the evaluation and then reproduce it with their own model configuration.
COBOL migration tests a different capability: rewriting an existing program in C++, Java, Kotlin, Python, or Rust while preserving its behavior on hidden tests. OpenAI reports 86.6% accuracy for GPT‑6.1 Sol at the low reasoning level, compared with 5.4% for GPT‑5.6 Sol in the same mode. At a higher reasoning level, GPT‑6.1 Sol reaches 97.9% on this evaluation.
Why a port requires checking business rules
OpenAI’s description emphasizes that compatibility depends on the source language’s semantics. For example, COBOL fixed-point arithmetic truncates extra decimal places unless rounding is explicitly specified. If the new implementation rounds differently, the program may run successfully while producing different financial results.
The migration test therefore measures whether outputs match for the cases being checked, while a team’s practical criteria should also cover its own calculation rules, data formats, and system dependencies. A useful first step is to choose a module with clear inputs and outputs, record reference results for real edge cases, and run the original and rewritten versions in parallel.
What the results mean for teams
OpenAI’s figures apply to the published datasets and reasoning settings. MultiLangBench evaluates the creation of a SQLite-compatible implementation, while the COBOL suite tests behavior preservation during porting; the scores describe these specific evaluations. They give teams a reference point for selecting models and a basis for their own reproducible comparisons, not a ready-made assessment of compatibility with any production system.
The practical takeaway is simple: test models on your own code and control cases, and consider cost alongside success rates. For COBOL modernization, checks for rounding, missing records, and other rules encoded in the legacy program’s behavior are especially important. The new results make this kind of testing more accessible; confirming equivalence remains part of the engineering process.