it is because, the OS memory scheduler(Virtual Memory Manager (VMM) on Windows platforms.) kicks in, if it finds the RAM usage more than 50%, and swaps page to and fro HDD(secondary storage).
In my experience, if you are not using optimal memory(not running resource intensive...