AMD Believes BAPCo SYSmark 2018 Benchmark Doesn’t Represent Typical Usage

To say that AMD Ryzen processors with “Zen” cores have slightly changed the mobile and desktop processor market would be an understatement. AMD’s latest 3rd Gen Ryzen 3000 series processors with Zen 2 cores have shown that AMD is a serious contender in the PC market. That means benchmarks are once again really important to help highlight the strengths of your CPU and the weaknesses of your competitors.

CPU benchmarks are inherently designed to show winners and losers and there is no shortage of benchmarks to run. Some benchmarks are purely synthetic while others are considered to be “real world benchmarks” that are based on applications running ‘normal’ workloads. AMD and Intel have both been producing and touting up a number of benchmarks since things have become more competitive. Intel has recently been claiming that their processors are better performers when it comes to “real world benchmarks” and often backs that claim with results from an application-based benchmark called BAPCo SYSMark 2018.

Legit Reviews reached out to AMD on July 18th, 2019 to find out their stance on SYSMark 2018 acknowledging the fact that AMD in the past said SYSmark was too heavily Intel optimized. That claim was quickly denied by BAPCo in a statement they released on the same day as AMD back in 2011.

After pestering AMD about SYSmark 2018 again we finally got an e-mail back, 7 days later, acknowledging they were working on the statement. It then took another five weeks to get an official statement from AMD. During that time we used SYSMark 2018 for an article between Intel and AMD desktop processors that got a tad bit more traffic and attention than anticipated. Our SYSmark 2018 results heavily favored Intel, so AMD’s statement about SYSMark 2018 doesn’t come as too big of a surprise.

AMD continues to believe that the BAPCo created SYSmark benchmarks, including the current 2018 version, are comprised of workloads that are not representative of typical usage, and are weighted in such a way as to generate results that are not demonstrative of real world workloads. We encourage those looking to evaluate system performance to consider testing the actual typical workload. Where that cannot be tested, ensure the selected benchmark tests are representative of the intended usage. – AMD PR on 08/29/2019

So, you have Intel heavily promoting the use of SYSmark as being a solid benchmark that uses real applications (as shown in the slide below from WCCFTech) and then you have AMD saying that it might not be too real world after all. (Legit Reviews contacted Intel and they confirmed these are real slides from a local Taiwan press session.)

That said the SYSmark 2018 Application Contribution to Overall Rating diagram found in the Sysmark 2018 whitepaper is rather complicated. Over a dozen applications are used in scoring with some making up less than 5% of the overall score and others making up 25%. How the scoring is broken down can most certainly impact the outcome of a benchmark that generates an overall score with subscores for three test scenarios.

The following charts shows the approximate contribution of each application to the SYSmark 2018version 1.0.0.37 scenario ratings on the calibration system (see section 2.7). Due to system implementation differences, this chart is accurate for the calibration system. These contributions will vary from one system to the next. – Bapco SYSmark 2018 whitepaper

We will have to look into this further, but for now we just wanted to get AMD’s stance out there when it comes to SYSmark 2018. We’ve been waiting six weeks to get the statement and are glad that we finally know what AMD thinks about this particular benchmark that has been much talked about lately.

Current BAPCo members include: Acer, ARCIntuition, Compal, Dell, Hewlett-Packard, Hitachi, Intel, Inventec, LC Future Center, Lenovo, Microsoft, Pegatron, Quanta Computer, Samsung, Toshiba, Western Digital, Wistron, and others. Notable names missing from this list include AMD, Qualcomm, and NVIDIA.