Abstract
In this chapter, we are going to discuss statistics and how to apply it to benchmarking. You will learn many useful approaches and techniques to help you improve your benchmark design and analyze the results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For example, Scott’s normal reference rule, Rice rule, Freedman–Diaconis’ choice, Doane’s formula, square-root choice, Sturges’ formula, and others.
- 2.
It’s not the only outlier test; there are many other approaches: 6 sigma test, Chauvenet’s criterion, Grubbs’ test, Dixon’s Q test, Peirce’s criterion, and others.
- 3.
- 4.
For example, the classic box plot can be improved by additional information and transformed to the variable width box plot, the notched box plot, the vase plot, the bean plot, the bee swarm box plot, the highest density region box plot, the box-percentile plot, the letter-value box plot, or other kinds of box plot.
- 5.
- 6.
- 7.
- 8.
- 9.
There is an interesting blog called “Accidentally Quadratic” with stories about situations when an algorithm has the quadratic complexity, but it wasn’t obvious: https://accidentallyquadratic.tumblr.com/
- 10.
You can find an implementation of this method and other similar algorithms in the MathNet.Numerics NuGet package: see https://numerics.mathdotnet.com/
- 11.
You can find the most truthful explanation of how curve fitting works in real life here: https://xkcd.com/2048/
- 12.
We assume that the standard deviation is not changing significantly with additional iterations. The only exception from this rule supposes that new iterations take more time than the previous. In this case, the benchmark doesn’t have a steady state, and it doesn’t make sense to discuss its distribution.
- 13.
In [Lakens 2014b], Daniel Lakens describes an interesting effect called “bi-polar p-value disorder.”
- 14.
You can find more information about this effect in the following Wikipedia article: https://en.wikipedia.org/wiki/Confirmation_bias
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2019 Andrey Akinshin
About this chapter
Cite this chapter
Akinshin, A. (2019). Statistics for Performance Engineers. In: Pro .NET Benchmarking. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-4941-3_4
Download citation
DOI: https://doi.org/10.1007/978-1-4842-4941-3_4
Published:
Publisher Name: Apress, Berkeley, CA
Print ISBN: 978-1-4842-4940-6
Online ISBN: 978-1-4842-4941-3
eBook Packages: Professional and Applied ComputingApress Access BooksProfessional and Applied Computing (R0)