3/2/2024 0 Comments Lack of benchmark meaningFor this we’d need a new macro, 3-4 years ago I watched a video of great talk by some CS Professor who explains this and has developed a benchmarking tool that accounts for it (not for Julia, for command line applications). Indeed, we can even go further, and say whether the difference between two runs is statistically significant. In order to do this, it needs to randomly change the offset of stack and heap allocated memory between evaluations of the code. However, there’s a fix – we can make sure that the results from reflect this variability. Usually we have no control over this at runtime (though it’s possible to ensure that memory blocks have a particular alignment if needed). This can have a large impact on cache hits/misses and therefore performance. Different runs of the code will use memory blocks at different offsets from the start of a cache line. I believe that this issue is due to alignment between working memory and cache lines. I might randomly get a faster run for code that is in general slower. If I want to know whether a small code change leads to faster code, cannot reliably help me answer that. This makes results from lack actionability. In fact, there’s around a 30% difference between the slowest time in the first run and the fastest time in the next run! It’s clear that the histograms are therefore meaningless – they don’t reflect the real distribution of timings. There isn’t any overlap between the two histograms. There is a huge difference in the two outputs. Note there is no garbage collection at play here. Memory estimate: 0 bytes, allocs estimate: 0.īenchmarkTools.Trial: 119 samples with 1 evaluation. See the following two outputs, made consecutively, with zero code changes and no restarting Julia: BenchmarkTools.Trial: 161 samples with 1 evaluation. I see a huge disparity in the output of different runs of for the same code.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |