Airgap Performance Test

Ran test against URL __redacted_customer_url__


Methodology

We loaded the site 100 times. In half of the cases, we blocked all outbound requests to cdn.transcend.io in order to simulate airgap not affecting the site's performance.

These tests are run using Google Lighthouse with no throttling on Github Actions executors. The tests are split across 20 machines in order to reduce noise from if certain CI executors are more performant than others.

See this guide to learn how these tests are run in a reproducible manner.

The p values mentioned below are from a Two-sample Kolmogorov–Smirnov test to test if the two distributions (with and without airgap enabled) are likely from the same distribution or not.

Please reach out if you would like any additional statistical testing performed.


Results

Overall Performance Score Metrics

Overall Performance Score distributions

With a p-value of 0.068, since the p-value is greater than .05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that the two sample datasets did not come from the same distribution.

In other words, we found no significant difference to this metric with airgap enabled.


First Contentful Paint Metrics

First Contentful Paint distributions

With a p-value of 0.179, since the p-value is greater than .05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that the two sample datasets did not come from the same distribution.

In other words, we found no significant difference to this metric with airgap enabled.


Largest Contentful Paint Metrics

Largest Contentful Paint distributions

With a p-value of 0.112, since the p-value is greater than .05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that the two sample datasets did not come from the same distribution.

In other words, we found no significant difference to this metric with airgap enabled.


First Meaningful Paint Metrics

First Meaningful Paint distributions

With a p-value of 0.001, since the p-value is less than .05, we reject the null hypothesis. We have sufficient evidence to say that the two sample datasets do not come from the same distribution.

In other words, we found a significant difference to this metric with airgap enabled.


Speed Index Metrics

Speed Index distributions

With a p-value of 0.549, since the p-value is greater than .05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that the two sample datasets did not come from the same distribution.

In other words, we found no significant difference to this metric with airgap enabled.


Time to Interactive Metrics

Time to Interactive distributions

With a p-value of 0.272, since the p-value is greater than .05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that the two sample datasets did not come from the same distribution.

In other words, we found no significant difference to this metric with airgap enabled.


Javascript Boot up Time Metrics

Javascript Boot up Time distributions

With a p-value of 0.012, since the p-value is less than .05, we reject the null hypothesis. We have sufficient evidence to say that the two sample datasets do not come from the same distribution.

In other words, we found a significant difference to this metric with airgap enabled.


Network Server Latency Metrics

Network Server Latency distributions

With a p-value of 0.396, since the p-value is greater than .05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that the two sample datasets did not come from the same distribution.

In other words, we found no significant difference to this metric with airgap enabled.