Experiences in lab peer review and procedure harmonization for inter-lab and tool-independent reproducibility in O-RAN testing
Repeatable and consistent testing for Open RAN is of great importance, particularly for a valid certification. But how to achieve that goal? To answer that question, the i14y Lab and VALOR have conducted a series of tests to see how the results would compare. The test consisted of six O-RAN ALLIANCE Test and Integration Focus Group (TIFG) end-to-end (E2E) Test Specification version 6.00 functional and performance test cases, such as user equipment (UE) registration, UE deregistration, and downlink/uplink peak throughput.
On May 19, the i14y Lab and VALOR, represented by
- Andreas Gladisch (Deutsche Telekom)
- Carsten Rossenhövel (EANTC)
- Ian C. Wong (VIAVI Solutions)
presented the procedures and discuss the results, learnings, challenges, and implications in a virtual International Open RAN Symposia (IORS) Virtual session.
An overview of the results can be found at the IORS page for the virtual session.
Here are the key results shared in the session:
The lab-to-lab comparisons showed variations within an acceptable range, but some required closer examination. Analyses of the results led to the following recommendations:
- It is critical to specify all components of the test setup. For example, the 5G Core can impact the system under test (SUT) performance and results.
- Specific equipment configuration parameters must be specified. For example, to compare system performance, the number of UEs must be identical.
- Test plans must specify the same technology. Variations in technology can produce very different results.
The following recommendations enable faster iteration and deployment of tested and validated 5G Open RAN components in real-world scenarios:
- Establish repeatable and consistent testing methodologies through standardized, reproducible test procedures.
- Benchmark performance across environments by comparing results from independent lab setups.
- Create a knowledge-sharing framework to share insights and benefit stakeholders.
- Support global standardization with O-RAN Alliance, TIP, and 3GPP to drive repeatable and consistent testing.
