Performance Testing a Video Streaming Platform

Background

On-demand video and live streaming video were two key scenarios used to performance test the platform. The identified scenarios were simulated by logging in as a user to the portal using the username and password. When logged in, the user viewed the on-demand video and live stream video.

Challenges

A challenge the team encountered was putting together the JMeter Framework to simulate the scenario. Putting together the script to simulate the live streaming video and on-demand videos was time-consuming. Despite the challenges, the Prolifics Testing Performance Test team was able to successfully execute the task.

The process

The planned tests were executed successfully by the Prolifics Testing Performance Test team, and these tests were coordinated between the client and the video hosting provider. The performance test results showed that the platform was able to manage 2.5K logins in 5 minutes. During the tests, the site was able to manage 2K users viewing the on-demand video and up to 2.5K users viewing the live streaming video.  

A general observation when the platform was under load was that the users experienced a few seconds delay in starting the live stream video and in loading the on-demand video. Apart from these delays, there were no other major errors or platform degradation.  

During the performance test, the client, platform provider, and Prolifics Testing users who were browsing the site, viewing the live stream and on-demand video were using their home networks, all did not notice any delays. However, a general observation noted was the quality of the videos depends on the network available at users' end.

Tests Summary

• 1K (Normal Load) and 2K (Peak Load) users test was run on the on-demand video.
• 2K (Peak Load) and 2.5K (Stress Test) users test was run on the live streaming video.

The initial test run with 1K users on the on-demand video showed the site was able to log in all the users successfully but was showing errors messages (429 – Too Many Attempts) for the Analytics API requests after login. This issue was evaluated by the video hosting provider, and it was confirmed that the security feature on the platform was blocking the number of requests during the test. As a result, the security feature was removed. The retests confirmed this issue was fixed after which the 1K users test did not show any errors for the Analytics API requests.

The platform was able to handle 1K user’s login without any issues during this test except for failing to send requests to Vimeo servers to view the on-demand video. The Vimeo server failed all the requests as the system detected a high number of requests from the single source IP (Known Issue). The Prolifics Testing team found a solution for this issue by accepting the captcha from the load generator machines and confirming the requests.

The 2K test run on the on-demand video showed the site was able to log in all the 2K users successfully. As a result, all 2K users were able to log in and view the on-demand video during the test. 

During the live stream, the site was able to handle the vast user load, which gathered between 2-2.5K users within 5 minutes. The platform performed efficiently and did not indicate any flaws or degradations. When users browsed the site, there was a slight delay in the live video on the broadcast page when the system was approaching overload. During the live streaming video test with the 2K users, Vimeo was able to accept the request and allow test users to view the live stream. However, Vimeo blocked the request when the user amount reached 2.5K.

The challenges of Load Testing streaming media involve complexity within the interaction between client and server, thus making it demanding to simulate in load tests. In general, various load and performance testing challenges will arise depending on the way you choose to deliver content to users (e.g., adaptive download vs. CDN).