Robin's Blog

Calculating percentiles in Python – use numpy not scipy!

This is just a brief public service announcement reporting something that I’ve just found: np.percentile is lot faster than scipy.stats.scoreatpercentile – almost an order of magnitude faster in some cases.

Someone recently asked me why on earth I was using scoreatpercentile anyway – and it turns out that np.percentile was only added in numpy 1.7, which was released part-way through my PhD in Feb 2013, hence why the scipy function is used in some of my code.

In my code I frequently calculate percentiles from satellite images represented as large 2D numpy arrays – and the speed differences can be quite astounding:

Image size scoreatpercentile percentile speedup
100 595us 169us 3.5x
1000 84ms 13ms 6.5x
3000 927ms 104ms 9x
8000 8s 1s 8x

As you can see, we get 3-4 times speedup for even small arrays (100 x 100, so 10,000 elements), and up to 8-9 times speedup for large arrays (tens of millions of elements).

Anyway, the two functions have very similar signatures and options – the only thing missing from np.percentile is the ability to set hard upper or lower limits – so it should be fairly easy to switch over, and it’s worth it for the speed boost!

If you found this post useful, please consider buying me a coffee.
This post originally appeared on Robin's Blog.

Categorised as: Programming, Python

One Comment

  1. One big difference is that scipy does linear interpolation, and not numpy. Have you tried disabling linear interpolation (for instance putting interpolation_method=’lower’)?

Leave a Reply

Your email address will not be published. Required fields are marked *