Last winter, an SAP consultant named Lao Zhang retired after 20 years at a manufacturing company in eastern China. He took with him more than just a badge—he carried the "living map" of hundreds of ABAP programs across the ECC system: which code block once fixed a bizarre tax code bug at 3 AM, which RFC interface connected to which authentication layer of the Golden Tax system, and which Z-table fields named SGTXT actually stored invoice remarks.
This knowledge was never written in any document. Now, it is gone.
Lao Zhang's story is not unique. SAP is everywhere—over 77% of the world's transaction revenue flows through SAP systems, and 87% of the Fortune 500 use them. SAP has announced that ECC (Business Suite 7) will end mainstream maintenance at the end of 2027, with extended maintenance available through the end of 2030 (SAP official). Companies still on ECC will sooner or later have to face migration to S/4HANA.
Why SAP Migration Is Different from Ordinary Code Migration
If you've done a framework upgrade for a web application, you might think "code migration is just code migration." But SAP migration is an entirely different species. You're not just updating familiar code to better patterns—you're reverse-engineering a complex business system that intertwines finance, supply chain, human resources, and tax compliance.
The code is the living documentation, and the people who could read it are gone.
The traditional solution was to assemble massive consultant teams, spending months or even years mapping out workflows. But the cost deterred most companies—so the systems just sat there, year after year, with the refrain "let's wait a bit longer."
AI is changing all of this. But to what extent, exactly? Imagination alone won't answer that—we decided to use data.
What ABAP-Bench measures
General-purpose coding benchmarks often cover code generation or GitHub issue resolution. ABAP-Bench focuses on SAP scenarios such as reading legacy programs, proposing S/4HANA rewrites, and assessing migration risks.
The tasks ask for concrete SAP objects, fields, or implementation steps. The scorer compares answers with rules and reference material. The protocol below documents how to replay the scores before examining the first GLM‑5 run.
Methods and reproducibility
ABAP-Bench uses controlled SAP scenarios to test whether a model can offer specific, checkable migration advice. This review covers the nine models in the April 2026 public archive. Their saved responses can be rescored task by task.
- Test set
- 60 tasks · 9 dimensions
- Full-set denominator
- 1,200 raw points
- Offline replay
- 540 / 540 exact matches
Sample and denominators
The fixed set runs from T01 to T60, at 20 points per task. The first eight dimensions each have six tasks and 120 possible points. Modern Ecosystem has 12 tasks and 240 points, so it contributes 20% of the overall score. Prompts are primarily in Chinese and draw on SAP migration scenarios and constructed code examples. The archived task file defines the exact prompts and assignments.
Task set, scorer, and archive versions
| Artifact | Protocol |
|---|---|
| Historical run · task set v4.0 | One stored answer per model and task. Three-layer scoring, with 40% rubric, 30% heuristic quality and 30% reference-answer coverage. The GLM‑5 score of 61 here comes from this run. |
| Frozen public release · Git tag v1.0.0 | This revision bundles the tasks, scorer and nine models’ saved answers. It is the replay baseline. The earlier v4.0.0 Git tag predates these per-task result files. |
| Optional judge and later code | The optional judge weights are 30% rubric, 20% heuristic quality, 20% reference-answer coverage, and 30% judge. The archive uses three layers. A later change on the main branch adds abaplint to some code tasks and can change the quality score. |
The scorer divides the rubric points by the task maximum and combines that fraction with two 0–1 subscores. The reference layer uses BM25 term similarity and key-concept coverage. The weighted fraction is scaled to 20, rounded and bounded to 0–20. The overall integer score is the sum of raw points divided by 1,200 and converted to a percentage. Each dimension uses its own denominator. Frozen scorer · abaplint change
Model calls and comparability
The GLM‑5 archive records April 3, 2026 (UTC), ZhipuAI, model ID glm-5 and a token limit of 8,000. The historical report and the frozen client use temperature 0.1, with one sample saved per task. Provider updates can change model weights or routing, and another call produces a new sample. Some conditions vary across models. The GLM‑4‑Plus archive, for example, records a 4,000-token limit. Frozen model configuration
The runner records API failures as zero-point rows with an error field. All nine public archives contain 60 rows and report no API errors. Report successful, failed and missing tasks separately in a new run and verify the denominator before comparing scores. The site also has a historical 12-model leaderboard. The replay here covers the nine models with public per-task files. Task-level archives · Runner
Replay the scores
Use Git and Python 3.9 or newer. This replay ran on Python 3.9.6. Download the replay script to your working directory and run these commands.
git clone --depth 1 --branch v1.0.0 https://github.com/fxp/abap-bench.git
python3 replay_scores.py ./abap-bench
The script checks the pinned revision, task IDs and point limits. It then reads the public JSON responses and uses the default three-layer scorer. An API key or a SAP environment is unnecessary. Expect 540 exact matches, zero differences and 733/1200 for GLM‑5, displayed as 61/100. See the replay output
Start a separate sampling run
The dry run should list 60 tasks, zhipuai, an 8,000-token limit and a 300-second timeout. For a live run, set ZHIPUAI_API_KEY in a separate checkout and run the second Python command. The current endpoint generates a new sample and writes it to that copy’s results/v4.0/. Record the model ID, provider, date, prompts and task revision, token limits, retries and failures, and report it as a new sample.
cd abap-bench
python3 -m src.run_benchmark --model glm-5 --dry-run
python3 -m src.run_benchmark --model glm-5
What the results support
These scores locate strengths and weaknesses on a fixed set of SAP tasks. Eight dimensions have six tasks each, and each task has one sample. Small gaps alone provide limited evidence of stable rank. The run has no human or current-workflow baseline. Constructed examples and version-specific prompts constrain the context. The tasks and reference answers are public, which leaves exposure during training unresolved. Quarterly task rotation is a maintenance proposal. Use the pinned set for replay.
The scorer approximates expert review. This evaluation does not establish independent human agreement, and it did not compile the answers in SAP or run business tests. Production use still calls for validation against the target release, authorizations and actual data.
First Evaluation: GLM-5
On April 3, 2026, GLM‑5 returned one archived response per task through the Zhipu API. The figures below aggregate the task records. The methods above document the model settings and replay steps.
Take a guess: what score can GLM-5 achieve?
60 SAP expert-level tasks, max score 100
GLM‑5 first-run score of 61 / 100
GLM‑5 scored 733/1,200 raw points. The 60 records total 8,670.52 seconds and 290,408 tokens. The latency card displays the sum of per-task durations in minutes.
View raw scores and denominators
| Dimension | Points | Maximum | Rate |
|---|---|---|---|
| Code Migration | 69 | 120 | 57.5% |
| Defect Discovery | 65 | 120 | 54.2% |
| Code Rewrite | 69 | 120 | 57.5% |
| China Compliance | 81 | 120 | 67.5% |
| Migration Risk | 77 | 120 | 64.2% |
| Security & Authorization | 76 | 120 | 63.3% |
| Architecture | 77 | 120 | 64.2% |
| Performance | 67 | 120 | 55.8% |
| Modern Ecosystem | 152 | 240 | 63.3% |
| Total | 733 | 1,200 | 61.1% |
The figures aggregate task records. The older JSON summary reports 59/120 for Defect Discovery, 8,223.17 seconds, and 283,552 tokens. Task records yield 65/120, 8,670.52 seconds, and 290,408 tokens. Both give 733 total points. GLM‑5 archive · Reconciliation output
Can you find this ABAP bug? — After the first LOOP AT gt_bkpf INTO DATA(ls_bkpf) loop ends, what will be the value of ls_bkpf?
DATA() inline-declared variable persists after the loop ends, retaining the value from the last iteration. GLM-5 correctly identified this bug and pointed out that it would cause "the credit memo/normal status judgment for all invoices to be completely wrong." This was one of the most surprising findings in the evaluation.
Surprising Moments
The CDS rewrite covered key rubric items
T03 · 16/20* ECC: N+1 query anti-pattern SELECT * FROM bkpf INTO TABLE @DATA(lt_bkpf) WHERE bukrs = @p_bukrs AND gjahr = @p_gjahr. LOOP AT lt_bkpf INTO DATA(ls_bkpf). SELECT * FROM bseg INTO TABLE @DATA(lt_bseg) WHERE bukrs = @ls_bkpf-bukrs AND belnr = @ls_bkpf-belnr AND gjahr = @ls_bkpf-gjahr. ENDLOOP.
* Illustrative rewrite: CDS view JOIN SELECT j~CompanyCode, j~AccountingDocument, j~FiscalYear, i~AccountingDocumentItem, i~GLAccount, i~AmountInCompanyCodeCurrency FROM I_JournalEntry AS j INNER JOIN I_JournalEntryItem AS i ON j~CompanyCode = i~CompanyCode AND j~AccountingDocument = i~AccountingDocument AND j~FiscalYear = i~FiscalYear WHERE j~CompanyCode = @p_bukrs AND j~FiscalYear = @p_gjahr INTO TABLE @DATA(lt_result).
The T03 answer used CDS joins, field mapping, and inline declarations. The snippets are editorial illustrations. T03 earned 20/20 on the rubric and 16/20 after weighting.
It identified a stale value after the loop
T02 · 11/20T02 covered VAT invoice processing. GLM‑5 identified how a stale ls_bkpf value after the loop could affect the classification.
The automated score was 11/20, with a rubric score of 16/20.
A high-scoring China Compliance answer
T16 · 17/20T16 covered SAP localization and payroll in China. GLM‑5 received full rubric and structure scores.
At 17/20, it was the highest-scoring task in the run.
The risk assessment covered key rubric items
T43 · 16/20T43 described an ECC landscape with 800 custom tables. The answer covered table types, compatibility checks, and ABAP Test Cockpit.
The task scored 16/20 and recorded 242.66 seconds of API latency.
In which dimension did GLM-5 score the lowest?
Notable Limitations
A score gap across two rewriting tasks
16 vs 5GLM‑5 scored 16/20 on the classic CDS rewrite (T03) and 5/20 on the RAP EML rewrite (T37).
T37 earned 0/20 on the rubric. The task-level records show where the scores diverged.
Defect Discovery contains six tasks
65/120The six Defect Discovery tasks total 65/120 (54.2%), the lowest dimension in this run.
The archive gives the individual answers and rubric matches for reviewing where points were lost.
Latency and score for two tasks
155s → 17pts 447s → 6ptsMean recorded latency across 60 tasks was about 145 seconds. T16 scored 17/20 at 155.13 seconds. T36 scored 6/20 at 447.35 seconds. The tasks differ in content, so these are examples of observed outcomes.
How to use these results
Outputs suitable for review
- Pre-migration risk assessment and roadmap drafts
- China-specific compliance mapping
- Architecture solution discussions (FI-CO / ACDOCA / CDS)
- Code rewrites for classic scenarios
Checks that need the target environment
- Production-grade code for cutting-edge paradigms
- Final confirmation of deep defects
- Implementation of performance optimizations
Use high-scoring responses as review checklists and verify the objects, fields, target versions, and business context. Estimating project impact requires measuring time, quality, and rework with and without model assistance.
What to validate next
The scores in this run replay exactly. The next round could repeat sampling on fresh tasks, collect independent SAP expert ratings, and check code in a specified S/4HANA environment. Reporting those outcomes alongside the automated scores would help assess model stability and deployment risk.