Mojo is faster using swap memory than regular Python is on RAM?
Has performance testing been done using Mojo with an NVMe swap file?
Given the number of threads in modern processors, it seems most scale problems are RAM-constrained. NVMe is 10x slower than DDR4 RAM, but 60x cheaper. For example, a single 4TB stick of NVMe costs $166 on Amazon right now.
If you combine this expansive capacity with the speed of Mojo, then it would effectively eliminate the "doesn't fit on my laptop" problem, and remove some of the need for distributed computing frameworks like Spark/Dask/Julia-Dagger.
1 Reply
Most real world applications in production actually spend most of their time moving memory around so slowing that part down even more by using an SSD instead of ordinary DRAM would not be a great idea most of the time.
If you want to work with huge dataframes and don't want to use distributed solutions you could check out Polars. If you use parquet or another disk-friendly file format you should be set. It'll only read the parts of the file it needs to in order to execute the query. It's written in Rust but has Python bindings so you can use it in Mojo.