Kraken prevents mobile performance regressions with Maestro
Kraken extended its Maestro test suite to run automated mobile performance benchmarks and alert on regressions.
Kraken has extended its Maestro end-to-end test suite to run automated mobile performance benchmarks aimed at detecting regressions earlier in development.
The company said it treats App Render Complete and Navigation Total Blocking Time as its primary performance vitals, and collects additional indicators such as render times, bundle sizes, network requests, frozen frames and memory usage to diagnose causes.
The company said it pipes the Performance API via react-native-performance to Sentry for real user monitoring and uses Rozenite in development, and that it extended Maestro to record key flows in CI, run them repeatedly across devices and calculate statistical significance to compare pull requests with the main branch.
The company said the initial setup exposed practical limits: automated benchmarks were often used only after regressions appeared in real user monitoring, release-branch comparisons were hard to triage because of many changes, and the App Render Complete metric proved network-dependent and non-deterministic, which could invalidate significance calculations.
The company said it addressed precision and variance by running benchmarks for every merge with an optimized pipeline that reduces turnaround to about 8 minutes, by recording and replaying network traffic and spreading runs across more devices, and by alerting when a metric regresses more than 10% for at least two consecutive runs; the firm said it plans to adapt sample sizes based on expected effect sizes to prevent regressions before merges.
The Recap
- Kraken extended Maestro to run automated mobile benchmarks continuously.
- Benchmarks run per merge with about 8 minutes turnaround.
- Alerts fire after more than 10% regression in two consecutive runs.
