Quote:
Originally Posted by Nick
This is a common complaint, usually refuted by an appeal to advanced mathematical statistics

As an expert in advanced mathematics
, the
binomial test for 26 failures out of 335 at 4% failure rate gives a onesided
pvalue of 0.00114. This means that you can expect to observe 26 or more failures about 0.11% of the times.
Usually, in a single, nonrepeated experiment, a result is considered statistically significant if the pvalue is 0.05 or below. So yes, that seems a statistically unusually high failure rate, and worth investigating. There is usually a reporting bias, in that only 'unusual' results get reported to the devs, so the best thing would be collecting completely new data to test it.