my best guess is that rather than using programmatical methods for simulating physical happenings, a physx card uses hard circuitry to do that.
For ex., assume a collision. So, to simulate that, i write a C++ program which runs on my processor [Central proccesor].. which basically includes all the calculations required for an elastic collision simulation. Now, there is gonna be a lot of operations for this on the processor level. A PPU on the other hand, may have a processor instruction for collisions, so all these calculations are done on hard circuitry and since it is specialised to do it, it obviously will not require as many operations as a CPU does. The problem with this is that the PPU cannot be used for most other operations.
Now, as i said, this is my best guess, which basically stems from my ignorance.