gRPC Performance Analysis

Project for MyEpi

TL;DR

For my bachelor thesis, I designed a gRPC-based backend solution to replace Firebase, focusing on stream management, code generation, and authentication. Using MyEpi as a case study, I conducted a performance analysis that revealed key issues like resource leaks and database connection limitations. By addressing these, including implementing a global database stream, I optimized the system to handle 500 concurrent clients with significantly improved response times. The project earned me a top grade and deepened my understanding of data streaming and performance optimization.

To conclude my bachelor's degree, I wrote a thesis on replacing a Firebase backend solution with a custom gRPC-based backend. I based the thesis on a case study for a startup called MyEpi, which was building a platform for managing epilepsy. I later became the CTO of MyEpi, partly due to the work I did on this project. The thesis earned me a top grade, and my project supervisor recommended that I publish it.

Loading PDF...

Bachelor Project Report

Link

While the thesis covered a wide range of topics, including stream management, code generation, and authentication, the most interesting part was the performance analysis (Section V-D). The analysis uncovered several issues in the initial setup, requiring me to think strategically and critically to identify the root causes of the performance problems.

For performance testing, the setup involved multiple client processes. Each client subscribed to a stream and emitted 10 messages with random delays between 500 and 1500 milliseconds. Each message was echoed back to the client over the gRPC stream. Each process recorded the lowest, highest, and average round-trip message delay times. Tests were run for 25 iterations at various client counts.

Performance test, 100 clients, pool size 100 (x = iteration, y = milliseconds)

The first test run showed clear degradation in performance with each iteration, indicating a resource leak. This leak was traced back to an issue in the gRPC implementation, where client streams were not properly closed upon disconnection. After implementing a fix, the tests were rerun.

Performance test - Single iteration, variable client count (x = client count, y = milliseconds)

In the second test run, performance was measured for a single iteration at varying client counts. The results showed a sharp performance drop for tests exceeding 100 clients. This issue was traced to the MongoDB connection pool size, which defaults to 100.

Performance test - 100 clients, pool size 100 (x = iteration, y = milliseconds)

Performance test - 100 clients, pool size 2000 (x = iteration, y = milliseconds)

To test the impact of increasing the pool size, two tests were run, using pool sizes of 100 and 2000. The results showed significant performance and uniformity improvements with the larger pool size.

Performance test - 200 clients, pool size 550 (x = iteration, y = milliseconds)

Performance test - 100 clients, pool size 250 (x = iteration, y = milliseconds)

Further tests revealed that increasing the pool size could also negatively impact performance. As the number of connections increased, idle connections accumulated, degrading performance. It was deduced, that the optimal pool size matched the expected number of concurrent clients to balance connection availability and efficiency.

Performance test - 500 clients, pool size 750 (x = iteration, y = milliseconds)

Performance test - 500 clients, pool size 750, global stream modification (x = iteration, y = milliseconds)

The system was expected to handle up to 500 concurrent clients. However, initial tests showed unimpressive performance, with average response times around 800 milliseconds and worst-case times exceeding 4 seconds. To address this, an alternative setup was tested tested where a global database connection was established between the service and the database. Incoming data was filtered by the service and forwarded to the appropriate clients. This setup significantly improved performance, reducing average response times to around 100 milliseconds. Although worst-case timings still reached 1500 milliseconds during startup, this was deemed acceptable.

My bachelor thesis covered a wide range of topics and required me to leverage most of the hard skills I had acquired during my studies. The performance analysis was the most engaging part, helping me derive actionable insights, address critical issues, and gain a deeper understanding of the technologies I used.