Last winter, an SAP consultant named Lao Zhang retired after 20 years at a manufacturing company in East China. He took with him more than just a badge—he carried the "living map" of hundreds of ABAP programs across the entire ECC system: which code once fixed a bizarre tax code bug at 3 AM, which RFC interface connected to which layer of authentication in the Golden Tax System, and which Z-table fields were named SGTXT but actually stored invoice remarks.
This knowledge was never written in any documentation. Now, it is gone.
Lao Zhang's story is not an isolated case. SAP is everywhere—over 77% of global transaction revenue flows through SAP systems, and 87% of the Fortune 500 use it. SAP has announced that ECC will end mainstream maintenance in 2027. Hundreds of thousands of enterprises worldwide must migrate to S/4HANA, creating a $48 billion migration market.
Why SAP Migration Is Different from General Code Migration
If you've done framework upgrades for web applications, you might think "code migration is just the same old thing." But SAP migration is an entirely different beast. You're not just updating familiar code to better patterns—you're reverse-engineering a complex business system that intertwines finance, supply chain, human resources, and tax compliance.
The code is the living documentation, and the people who could read it are gone.
The old solution was to assemble massive consultant teams, spending months or even years untangling workflows. But the cost deterred most companies—so the systems just kept getting pushed to "next year," year after year.
AI is changing all of this. But to what extent exactly? Imagination alone won't answer that—we decided to use data.
ABAP-Bench: Giving AI an SAP Consultant Certification Exam
Existing AI code evaluation benchmarks are of no help here. SWE-bench tests GitHub bug fixes, BigCodeBench tests general programming, and HumanEval only has algorithm problems. The SAP/ABAP domain is a complete blank spot on the AI evaluation map.
So we built one ourselves. ABAP-Bench—60 expert-authored SAP migration tasks, covering 9 key dimensions. Each task is worth 20 points, independently graded by a three-layer scoring engine. We gave AI an SAP consultant certification exam, and then graded it meticulously.
Building ABAP-Bench
Task Design
Every task originates from real SAP migration project scenarios, authored by domain experts. We don't accept vague answers: if the AI responds "recommend using a CDS view instead of the BKPF table"—zero points. You must specify I_JournalEntry, and that field BUKRS maps to CompanyCode.
The test code uses Chinese variable names (e.g., 发票号码, 含税金额), which are extremely rare in public code repositories—ABAP-Bench's "anti-cheat watermark."
Three-Layer Scoring Engine
Click each layer to learn the scoring mechanism:
I_JournalEntry correctly earns points; incorrectly claiming "BKPF is the primary table in S/4HANA" deducts 3 points—getting it wrong is worse than saying nothing.Preventing Evaluation "Expiration"
Every quarter, 10% of old questions are retired and replaced with new ones; tasks are anchored to specific versions of SAP Notes and regulatory documents. Like the college entrance exam changing its questions every year, ABAP-Bench's effectiveness won't degrade over time.
Engineering Quality
First Evaluation: GLM-5
We chose Zhipu GLM-5 as the first challenger—Zhipu AI's flagship reasoning model, featuring internal chain-of-thought reasoning capabilities. The evaluation covered the full lifecycle of SAP migration, with temperature fixed at 0.1 to ensure reproducibility. Then, we waited quietly for 145 minutes.
Take a guess: What score can GLM-5 achieve?
60 SAP expert-level questions, max score 100
Evaluation Results: 61 / 100
At first glance, it might seem like "just okay." But diving into specific tasks, a more vivid picture emerges—this variance itself is the most valuable discovery.
Can you find this ABAP bug? — After the first LOOP AT gt_bkpf INTO DATA(ls_bkpf) loop ends, what will the value of ls_bkpf be?
DATA() inline-declared variables continue to exist after the loop ends, retaining the value from the last iteration. GLM-5 correctly identified this bug and pointed out that it would cause "the credit memo/normal status judgment for all invoices to be completely wrong." This was one of the most surprising findings in the evaluation.
Surprising Moments
It wrote directly compilable S/4HANA code
T03 · 16/20Rewriting the ECC-era nested SELECTs (BKPF + BSEG, a classic N+1 disaster) into an S/4HANA-compatible version.
* ECC: N+1 query anti-pattern SELECT * FROM bkpf INTO TABLE @DATA(lt_bkpf) WHERE bukrs = @p_bukrs AND gjahr = @p_gjahr. LOOP AT lt_bkpf INTO DATA(ls_bkpf). SELECT * FROM bseg INTO TABLE @DATA(lt_bseg) WHERE bukrs = @ls_bkpf-bukrs AND belnr = @ls_bkpf-belnr AND gjahr = @ls_bkpf-gjahr. ENDLOOP.
* GLM-5: CDS View JOIN, single query SELECT j~CompanyCode, j~AccountingDocument, j~FiscalYear, i~AccountingDocumentItem, i~GLAccount, i~AmountInCompanyCodeCurrency FROM I_JournalEntry AS j INNER JOIN I_JournalEntryItem AS i ON j~CompanyCode = i~CompanyCode AND j~AccountingDocument = i~AccountingDocument AND j~FiscalYear = i~FiscalYear WHERE j~CompanyCode = @p_bukrs AND j~FiscalYear = @p_gjahr INTO TABLE @DATA(lt_result).
Correctly chose CDS view JOINs, precisely mapped field names (BUKRS → CompanyCode), and used modern inline declaration syntax. This code can compile and run directly.
Found hidden bugs like a meticulous code reviewer
T02 · 12/20Five bugs were planted in a VAT invoice processing program. GLM-5 precisely hit the most critical one: the residual value issue of variable ls_bkpf after the loop ends, and accurately described the business impact—"the credit memo/normal status judgment for all invoices is completely wrong, leading to severely distorted tax declaration data."
This isn't a textbook answer; it's an answer that truly understands ABAP runtime behavior and the VAT business context.
China-specific regulations are its home turf
T16 · 17/20The highest score across the board. Accurately identified the calculation rules for the five social insurances and one housing fund, specific fields for the cumulative withholding method for individual income tax, and the standard usage of the HR_READ_INFOTYPE function module. Full rubric score, full structural quality score.
VAT red invoice rules, the impact of fully digital e-invoices, PIPL data compliance—these answers are scattered across Chinese policy documents and deep within SAP configurations, taking even senior consultants half a day to look up.
Risk analysis comparable to an entry-level consultant
T43 · 16/20Evaluating the migration risk of an ECC system with 800 custom tables. GLM-5 identified the impact of Pool/Cluster tables, issues with Z-tables referencing deleted fields, and accurately mentioned SAP Readiness Check 2.0 and specific rule set names from ABAP Test Cockpit.
This kind of analysis previously required senior consultants days to complete. The AI delivered a qualified first draft in under 4 minutes.
In which dimension did GLM-5 score the lowest?
LOOP...ENDLOOP immediately followed by IF sy-subrc = 0. This intuition remains a uniquely human advantage for now.
Notable Limitations
Steady on familiar paths, stumbling on unfamiliar ones
16 vs 5T03 (classic CDS rewrite) 16/20, T37 (RAP EML rewrite) 5/20. Classic scenarios have abundant documentation; cutting-edge paradigms have scarce resources. GLM-5 got the direction right in T37 (I_Product, MODIFY ENTITIES), but had subtle errors in EML syntax.
Like a junior consultant who just started: they know which path to take, but aren't yet familiar with every stone along the way.
The most hidden bugs still belong to humans
5 bugs, hit 3The 5 bugs in T02 were ranked by concealment level. It hit the first 3, but missed the deepest 2—internal table/structure type confusion, and SY-SUBRC being unexpectedly overwritten in nested LOOPs.
AI can learn knowledge, but the intuition born from "having been burned" remains a uniquely human advantage for now.
Thinking longer doesn't mean answering better
155s → 17 pts 447s → 6 ptsThe reasoning model averaged 145 seconds per question. The highest-scoring T16 (17/20) took 155 seconds, while the lowest-scoring T36 (6/20) took 447 seconds. Thinking longer cannot compensate for knowledge blind spots.
Human + AI: The Optimal Solution for SAP Migration
ask where AI can save consultants the most time.
AI Can Handle This Today
- Pre-migration risk assessment and roadmap drafts
- China localization compliance review
- Architecture solution discussions (FI-CO / ACDOCA / CDS)
- Code rewriting for classic scenarios
Still Requires Human Oversight
- Production-grade code for cutting-edge paradigms
- Final confirmation of deep defects
- Implementation of performance optimizations
This is not a story of AI replacing people, but of AI changing how people work.
In the past, a senior consultant spent two weeks writing a migration assessment report. Now, AI generates a first draft in a few hours, and the consultant shifts from "writing from scratch" to "reviewing and optimizing"—efficiency increases several times over, and quality doesn't drop; it actually improves.
Conclusion: This Future Is Closer Than You Think
SAP migration has stalled for years; at its core, it's an economics problem: the cost of understanding legacy systems is too high.
AI is breaking down this cost barrier. 61/100 is not "barely passing"—it's a signal: AI in the SAP domain has crossed the "toy" stage and entered the starting point of "usable." It can write compilable S/4HANA code, find variable scope bugs, and produce migration risk reports with specific tools and steps.
The shortcomings it revealed—precision in cutting-edge syntax, capturing the deepest defects—point precisely to the next direction. These are not fundamental obstacles, but capability boundaries that can be continuously improved as training data enriches.
The $48 billion S/4HANA migration market can't wait, and consultants like Lao Zhang aren't coming back.
Fortunately, a new helper has arrived—it's not perfect, but it's available 24/7, won't forget a single field mapping, and gets a little better every day.