I don't believe that memory is the main reason for the speed issues. There is also the issue that in a compiled language, the compiler can turn function/method calls into explicit jump to a memory location or a lookup of the memory location in a v-table. In dynamic languages, that never happens. Instead, calls to functions are looking something up in a hashtable at runtime. While this is still O(1), it has a much larger constant and it likely to cause pipeline stalls.

The other issue, when compared to C/C++/Rust, is that Python programs have little control over how memory is organized and hence cache optimization isn't generally possible. Cache misses are very slow on modern architectures. The large memory you mentioned definitely contributes to this as well.