Beware the Logarithms
Logarithms are great. They let you talk about incredibly wide ranges of numbers, and they transform multiplication into addition. Algorithms with logarithmic running times are so fast they might as well be constant time algorithms. Logarithmic scales on graphs can also make your results look much better. Let’s see how.
On a recent paper submission, I included a graph comparing Harlan to CUBLAS on a trivial vector addition benchmark. Below is a recreation of that graph.
The green dots represent the execution time for the Harlan program, while the blue dots indicate the same for the CUBLAS version. This graph clearly shows that the Harlan version takes about twice as long as the CUBLAS version. On the one hand, being slower than a highly tuned linear algebra library by only a factor of two isn’t bad. On the other hand, this graph doesn’t cast Harlan in a great light.
Let’s see what happens if we put the Y axis on a logarithmic scale.
This looks much better! Here it looks like we only have an additive factor overhead, and there’s really not that much room between the Harlan and CUBLAS points. We also get the nice logarithmic shape, which makes it look like the time doesn’t get that much worse as the vectors get larger.
As a third option, we can use a logarithmic scale on the X axis as well.
Now we see two lines, which suggests the execution time for each program increases linearly in the size of the vector, which is true. But now, the lines are essentially parallel, again with only an additive factor between the two. As we move to the right, the lines seem to converge, suggesting that for extremely large vectors, Harlan might even outperform CUBLAS. Not bad!
The point of all this is that I have presented the exact same data in all three cases, yet the choice of scales leads us visually to very different conclusions. In this case, I believe the linear-linear plot is the most honest. For other data sets, one of the other plots may be more appropriate. As an author, it’s important to choose visual presentations of data that faithfully highlight the key features of the data. Likewise, as a reader, it’s important to pay attention to the scales and understand what the shape of a graph means in light of those scales.