^^ It's called Nvidia TurboCache or ATI HyperMemory
The GPU must have support for this feature - the gpu will then use system memory as a substitute to the dedicated physical memory of the gfx card
Say a gpu has 256MB physical memory with TurboCache/HyperMemory - the card supports upto 512MB - now because of TC/HM tech 256MB of the system memory will be allocated as the gpu memory to make the the 512MB vram buffer