Skip to content
← DeepDive Experiments & Culture · v3 Updated 2026-11-19 中文
← ResearchRE · 0001 · 2026-05-09
AB
AI for Enterprise Software · Research

When AI Takes the
SAP Consultant's Seat

ABAP-Bench tests models on 60 SAP migration tasks. We replayed the archived scores task by task to see what GLM‑5’s 61 points tell us.

Data archived · April 2026 Task set v4.0
Summary

ABAP-Bench measures how concretely models can analyze, rewrite, and propose migrations for legacy SAP systems. This review pins the public scorer and 60 tasks and covers nine models.

Last winter, an SAP consultant named Lao Zhang retired after 20 years at a manufacturing company in eastern China. He took with him more than just a badge—he carried the "living map" of hundreds of ABAP programs across the ECC system: which code block once fixed a bizarre tax code bug at 3 AM, which RFC interface connected to which authentication layer of the Golden Tax system, and which Z-table fields named SGTXT actually stored invoice remarks.

This knowledge was never written in any document. Now, it is gone.

Lao Zhang's story is not unique. SAP is everywhere—over 77% of the world's transaction revenue flows through SAP systems, and 87% of the Fortune 500 use them. SAP has announced that ECC (Business Suite 7) will end mainstream maintenance at the end of 2027, with extended maintenance available through the end of 2030 (SAP official). Companies still on ECC will sooner or later have to face migration to S/4HANA.

People who understand these systems, like Lao Zhang, are leaving one by one.
· · ·
01

Why SAP Migration Is Different from Ordinary Code Migration

If you've done a framework upgrade for a web application, you might think "code migration is just code migration." But SAP migration is an entirely different species. You're not just updating familiar code to better patterns—you're reverse-engineering a complex business system that intertwines finance, supply chain, human resources, and tax compliance.

The code is the living documentation, and the people who could read it are gone.

The traditional solution was to assemble massive consultant teams, spending months or even years mapping out workflows. But the cost deterred most companies—so the systems just sat there, year after year, with the refrain "let's wait a bit longer."

AI is changing all of this. But to what extent, exactly? Imagination alone won't answer that—we decided to use data.

· · ·
02

What ABAP-Bench measures

General-purpose coding benchmarks often cover code generation or GitHub issue resolution. ABAP-Bench focuses on SAP scenarios such as reading legacy programs, proposing S/4HANA rewrites, and assessing migration risks.

The tasks ask for concrete SAP objects, fields, or implementation steps. The scorer compares answers with rules and reference material. The protocol below documents how to replay the scores before examining the first GLM‑5 run.

· · ·
03

Methods and reproducibility

ABAP-Bench uses controlled SAP scenarios to test whether a model can offer specific, checkable migration advice. This review covers the nine models in the April 2026 public archive. Their saved responses can be rescored task by task.

Test set
60 tasks · 9 dimensions
Full-set denominator
1,200 raw points
Offline replay
540 / 540 exact matches

Sample and denominators

The fixed set runs from T01 to T60, at 20 points per task. The first eight dimensions each have six tasks and 120 possible points. Modern Ecosystem has 12 tasks and 240 points, so it contributes 20% of the overall score. Prompts are primarily in Chinese and draw on SAP migration scenarios and constructed code examples. The archived task file defines the exact prompts and assignments.

Task set, scorer, and archive versions

ArtifactProtocol
Historical run · task set v4.0One stored answer per model and task. Three-layer scoring, with 40% rubric, 30% heuristic quality and 30% reference-answer coverage. The GLM‑5 score of 61 here comes from this run.
Frozen public release · Git tag v1.0.0This revision bundles the tasks, scorer and nine models’ saved answers. It is the replay baseline. The earlier v4.0.0 Git tag predates these per-task result files.
Optional judge and later codeThe optional judge weights are 30% rubric, 20% heuristic quality, 20% reference-answer coverage, and 30% judge. The archive uses three layers. A later change on the main branch adds abaplint to some code tasks and can change the quality score.

The scorer divides the rubric points by the task maximum and combines that fraction with two 0–1 subscores. The reference layer uses BM25 term similarity and key-concept coverage. The weighted fraction is scaled to 20, rounded and bounded to 0–20. The overall integer score is the sum of raw points divided by 1,200 and converted to a percentage. Each dimension uses its own denominator. Frozen scorer · abaplint change

Model calls and comparability

The GLM‑5 archive records April 3, 2026 (UTC), ZhipuAI, model ID glm-5 and a token limit of 8,000. The historical report and the frozen client use temperature 0.1, with one sample saved per task. Provider updates can change model weights or routing, and another call produces a new sample. Some conditions vary across models. The GLM‑4‑Plus archive, for example, records a 4,000-token limit. Frozen model configuration

The runner records API failures as zero-point rows with an error field. All nine public archives contain 60 rows and report no API errors. Report successful, failed and missing tasks separately in a new run and verify the denominator before comparing scores. The site also has a historical 12-model leaderboard. The replay here covers the nine models with public per-task files. Task-level archives · Runner

Replay the scores

Use Git and Python 3.9 or newer. This replay ran on Python 3.9.6. Download the replay script to your working directory and run these commands.

git clone --depth 1 --branch v1.0.0 https://github.com/fxp/abap-bench.git
python3 replay_scores.py ./abap-bench

The script checks the pinned revision, task IDs and point limits. It then reads the public JSON responses and uses the default three-layer scorer. An API key or a SAP environment is unnecessary. Expect 540 exact matches, zero differences and 733/1200 for GLM‑5, displayed as 61/100. See the replay output

Start a separate sampling run

The dry run should list 60 tasks, zhipuai, an 8,000-token limit and a 300-second timeout. For a live run, set ZHIPUAI_API_KEY in a separate checkout and run the second Python command. The current endpoint generates a new sample and writes it to that copy’s results/v4.0/. Record the model ID, provider, date, prompts and task revision, token limits, retries and failures, and report it as a new sample.

cd abap-bench
python3 -m src.run_benchmark --model glm-5 --dry-run
python3 -m src.run_benchmark --model glm-5

What the results support

These scores locate strengths and weaknesses on a fixed set of SAP tasks. Eight dimensions have six tasks each, and each task has one sample. Small gaps alone provide limited evidence of stable rank. The run has no human or current-workflow baseline. Constructed examples and version-specific prompts constrain the context. The tasks and reference answers are public, which leaves exposure during training unresolved. Quarterly task rotation is a maintenance proposal. Use the pinned set for replay.

The scorer approximates expert review. This evaluation does not establish independent human agreement, and it did not compile the answers in SAP or run business tests. Production use still calls for validation against the target release, authorizations and actual data.

· · ·
04

First Evaluation: GLM-5

On April 3, 2026, GLM‑5 returned one archived response per task through the Zhipu API. The figures below aggregate the task records. The methods above document the model settings and replay steps.

Take a guess: what score can GLM-5 achieve?

60 SAP expert-level tasks, max score 100

50
· · ·
05

GLM‑5 first-run score of 61 / 100

0 /100
Overall
0%
Highest dimension
0 min
Summed latency
0
Total tokens

GLM‑5 scored 733/1,200 raw points. The 60 records total 8,670.52 seconds and 290,408 tokens. The latency card displays the sum of per-task durations in minutes.

Highest China Compliance 67.5%
Tied second Migration Risk and Architecture 64.2%
Median Security & Authorization and Modern Ecosystem ≈ 63.3%
Lowest Defect Discovery 54.2%
Figure 1 · GLM‑5 scores normalized by dimension
View raw scores and denominators
DimensionPointsMaximumRate
Code Migration6912057.5%
Defect Discovery6512054.2%
Code Rewrite6912057.5%
China Compliance8112067.5%
Migration Risk7712064.2%
Security & Authorization7612063.3%
Architecture7712064.2%
Performance6712055.8%
Modern Ecosystem15224063.3%
Total7331,20061.1%

The figures aggregate task records. The older JSON summary reports 59/120 for Defect Discovery, 8,223.17 seconds, and 283,552 tokens. Task records yield 65/120, 8,670.52 seconds, and 290,408 tokens. Both give 733 total points. GLM‑5 archive · Reconciliation output

Interactive

Can you find this ABAP bug? — After the first LOOP AT gt_bkpf INTO DATA(ls_bkpf) loop ends, what will be the value of ls_bkpf?

Correct. ABAP's DATA() inline-declared variable persists after the loop ends, retaining the value from the last iteration. GLM-5 correctly identified this bug and pointed out that it would cause "the credit memo/normal status judgment for all invoices to be completely wrong." This was one of the most surprising findings in the evaluation.

Surprising Moments

1

The CDS rewrite covered key rubric items

T03 · 16/20
* ECC: N+1 query anti-pattern
SELECT * FROM bkpf INTO TABLE @DATA(lt_bkpf)
  WHERE bukrs = @p_bukrs AND gjahr = @p_gjahr.
LOOP AT lt_bkpf INTO DATA(ls_bkpf).
  SELECT * FROM bseg INTO TABLE @DATA(lt_bseg)
    WHERE bukrs = @ls_bkpf-bukrs
      AND belnr = @ls_bkpf-belnr
      AND gjahr = @ls_bkpf-gjahr.
ENDLOOP.
* Illustrative rewrite: CDS view JOIN
SELECT
  j~CompanyCode,
  j~AccountingDocument,
  j~FiscalYear,
  i~AccountingDocumentItem,
  i~GLAccount,
  i~AmountInCompanyCodeCurrency
FROM I_JournalEntry AS j
  INNER JOIN I_JournalEntryItem AS i
    ON j~CompanyCode = i~CompanyCode
   AND j~AccountingDocument = i~AccountingDocument
   AND j~FiscalYear = i~FiscalYear
WHERE j~CompanyCode = @p_bukrs
  AND j~FiscalYear = @p_gjahr
INTO TABLE @DATA(lt_result).

The T03 answer used CDS joins, field mapping, and inline declarations. The snippets are editorial illustrations. T03 earned 20/20 on the rubric and 16/20 after weighting.

2

It identified a stale value after the loop

T02 · 11/20

T02 covered VAT invoice processing. GLM‑5 identified how a stale ls_bkpf value after the loop could affect the classification.

The automated score was 11/20, with a rubric score of 16/20.

3

A high-scoring China Compliance answer

T16 · 17/20

T16 covered SAP localization and payroll in China. GLM‑5 received full rubric and structure scores.

At 17/20, it was the highest-scoring task in the run.

4

The risk assessment covered key rubric items

T43 · 16/20

T43 described an ECC landscape with 800 custom tables. The answer covered table types, compatibility checks, and ABAP Test Cockpit.

The task scored 16/20 and recorded 242.66 seconds of API latency.

Interactive

In which dimension did GLM-5 score the lowest?

Correct. GLM‑5 scored 65/120 (54.2%) on the six Defect Discovery tasks, its lowest dimension in this run.

Notable Limitations

1

A score gap across two rewriting tasks

16 vs 5

GLM‑5 scored 16/20 on the classic CDS rewrite (T03) and 5/20 on the RAP EML rewrite (T37).

T37 earned 0/20 on the rubric. The task-level records show where the scores diverged.

2

Defect Discovery contains six tasks

65/120

The six Defect Discovery tasks total 65/120 (54.2%), the lowest dimension in this run.

The archive gives the individual answers and rubric matches for reviewing where points were lost.

3

Latency and score for two tasks

155s → 17pts   447s → 6pts

Mean recorded latency across 60 tasks was about 145 seconds. T16 scored 17/20 at 155.13 seconds. T36 scored 6/20 at 447.35 seconds. The tasks differ in content, so these are examples of observed outcomes.

· · ·
06

How to use these results

Outputs suitable for review

  • Pre-migration risk assessment and roadmap drafts
  • China-specific compliance mapping
  • Architecture solution discussions (FI-CO / ACDOCA / CDS)
  • Code rewrites for classic scenarios

Checks that need the target environment

  • Production-grade code for cutting-edge paradigms
  • Final confirmation of deep defects
  • Implementation of performance optimizations

Use high-scoring responses as review checklists and verify the objects, fields, target versions, and business context. Estimating project impact requires measuring time, quality, and rework with and without model assistance.

· · ·
07

What to validate next

The scores in this run replay exactly. The next round could repeat sampling on fresh tasks, collect independent SAP expert ratings, and check code in a specified S/4HANA environment. Reporting those outcomes alongside the automated scores would help assess model stability and deployment risk.

Appendix

Evaluation Timeline

Historical GLM‑5 run
2026-04-03 · 60 tasks · 733/1,200
Offline score replay
2026-11-19 · 540 task scores matched

Full evaluation data and scoring engine are open source

github.com/fxp/abap-bench

View the historical v2.0 site leaderboard (12 models)

Revision history

3 versions
  1. v3 固定公开题集与评分提交,逐题核对 9 个模型的 540 条回答,修正 GLM‑5 维度分和耗时口径,补齐复评分脚本与证据边界。 View v3
  2. v2 删除“480 亿美元迁移市场”和“数十万家企业”两处查不到可靠出处的数字,改为 SAP 官方的 ECC 维护时间表(主流维护到 2027 年底,扩展维护到 2030 年底);中英文同步 View v2
  3. v1 回填结构化元数据(历史条目补齐 index.meta.json) View v1

Companion material