Depth of Fields: In Photography, film or 3D games, Depth of field can be considered as the distance between the nearest and farthest object, present in the sharper (relatively sharper than the portion of the image which are blurred and can't be distinguished properly) portion of the image or Frame.
This effect adds realism and Movie like effects in 3D gaming. It also adds a depth sense in the game which we can understand that how far the object is from the Protagonist, who is in the focus of the camera, which is our eyes. How can we forget the creepy feelings in Metro 2033 where we can understand something is moving around in blurred, shabby areas, little far from the source of light but can't see it properly?
*i.imgur.com/METdT.jpg
If you look at the above image, then you will see that the palm which is away from the 1st palm which is in focus, is little shorter and little less sharper too. If we continue like this, the palm size will be getting smaller and more blurred and after some time, it will not be visible when our eyes meet the vanishing point. But until then, we can see palms with varying size and varying sharpness.
So, I guess you guys can relate it with the definition I've given. Here until the vanishing point, we can consider the image is sharper enough to distinguish different objects. So depth of field will be the distance between the 1st palm which is in focus and the last palm which is blurred maximum but still identifiable.
*i.imgur.com/dMgZ5.png
The area within the depth of field appears sharp, while the areas in front of and beyond the depth of field appear blurry.
How to Implement: To simulate this 3D sense of depth in a 2D image, we need to blurred and resized each of the distant objects in such a way so that our human eye can understand the depth in the image. resulting an 3D depth simulation in our brain. For this we need to apply separate level of blurriness an size to each of the objects in accordance to their distance from the object present in the focus.
Traditional Pixel Shader Based Approach:
1. Create a Blurred (Normally Down-scaled) version of the Frame
2. Get the distances of the Objects to be blurred from the object in focus.
3. Mix the blurred image with the Crisp image based on the distance from the focal plane to create a image version with DoF implemented.
4. Perform the same operation with the output image of the 1st image created with DoF. For each iteration, the output of the previous step will be the Input. The number of iterations depends upon the used algorithm.
5. Consider the output of the last iteration as the final image and discard all the other images.
Issues with the Approach:
Although the approach is not an heavy one, it has several issues and couple of them are mentioned below:
1. Fixed blur radius across scene, no variation with depth
2. Artifacts, created due to the DoF algorithm, along the edges are very hard and tricky to remove. It needs significant logic for each thread.
3. Blurring the objects just based on the distance from the focal plane is not a very good method. It ignores the position of the object with respect to a point camera, which is our eyes in Games and the light emitting from them. Even an object is far from the focal point, if it is thermally hot or radiator of intense light then it should not be as blurred as another object with same distance, not radiating any light. For example, a Lamp Post radiates high intensity light and a dark home, situated at roughly same distance from the focal point will be less blurred.
Scientific Model
Here we need understand one Optical concept, important in determining the DoF, both in Photography as well as 3D image creation, known as Circle of Confusion. Lets discuss it a little.
Definition from Wikipedia: In optics, a circle of confusion is an optical spot caused by a cone of light rays from a lens not coming to a perfect focus when imaging a point source. It is also known as disk of confusion, circle of indistinctness, blur circle, or blur spot.
*i.imgur.com/Sffq7.png
1st Image: Object situated before the focus point of the Lens
2nd Image: Object situated exactly at the focus point of the Lens
3rdImage: Object situated after the focus point of the Lens
In the above image we are describing the property of a perfect Lens which creates a blurred circle of light when a point object is not situated at the focus point. But in reality, no Lens is perfect and the object, situated exactly at the focal plane will create a circle or Spot rather than a single point.
Visual acuity: For most people, the closest comfortable viewing distance, termed the near distance for distinct vision (Ray 2000, 52), is approximately 25 cm. At this distance, a person with good vision can usually distinguish an image resolution of 5 line pairs per millimeter (lp/mm), equivalent to a CoC of 0.2 mm in the final image
*i.imgur.com/tLSc5.png
After that distance of 250mm, human eye cannot distinguish between a blurred circle and a very sharp circle that just fills the contact point of two adjoining planes. So Circle of Confusion can be a measure to get the Blur effect to be applied on different objects in an image to simulate a Depth of Field.
GPU Compute Based Solution: Here the algorithm creates a Heat Diffusion based approach to reduce the complexity.
Heat Diffusion Model:
1. The image intensities are taken from a Pin Hole Camera View
2. The heat diffuses or reduces gradually with the distance from the source or the Pin hole Camera (In Games, it is our eye)
3. If it creates an higher Circle of Confusion or having high Blur Radius for object B , it is assumed that the thermal conductivity B is higher and heat will pass through it. It means we don't need to paint it with sharp color and it will be Blurred when the image will be rendered.
4. Circle of Confusion increases with the distance from the source and as per the radius of the circle, the Blur effect is added to the object proportionately. So we are having objects with variable blur effect.
5. If the Circle of Confusion reaches 0 for an object then it is considered as complete insulator of heat. So it needs to be painted sharply for identifying.
By using this above mentioned Model, several parallel Processing based algorithm can be created which can produce, variable blur effects to different objects, situated at different distances from the focus.
Metro 2033 uses an algorithm known as Alternate Direction Implicit Solver and Hybrid Tri-diagonal Systems Solver which uses two pixel shader passes to create a Blurred version of the image and then perform a DirectCompute based algorithm to generate the final image with vivid Depth of field.
Dynamic Decompression of Textures:
This direct Compute based method is pretty simple compared to the other two, I've explained earlier.
Suppose I have huge number of documents which are needed to create my final resume and they are scattered here & there in my hard disc and taking a lot of disc space. So what I'll do, compress all of them in a single Zip or Rar or 7Z file and accessing the part which is needed. If I click on that file, that will be decompressed at run time and will be displayed to me.
There are plenty of games which needs to handle huge amount of textures to render different objects in screen. They might not be very exact but the amount of objects are very high. RTS games like Civilikzation V or StarCraft II are perfect example of it. Here the following DirectCompute based steps are used:-
1. Compress the Texture objects into a compressed file and place it to the Video Ram using DirectCompute based parallel processing
2. Dynamically decompress it as per the requirement by using DirectCompute based Decompressing algorithm.
By using this technique, Civilization V can compress texture file of 2GB into a 130MB format which helps it to keep multiple texture files into Video Ram. Now Video Ram can be accessed very fast by GPU due to its high speed and big memory Bus. So it can request the required file which gets decompressed on the fly to produce the final image.
In conventional way, the decompressed texture cannot be placed directly in the Video Ram due to its huge size. Here they need to switched among Virtual Memory (HDD) then to Main Memory (System Ram) and then to Video Ram ( GPU Memory), resulting huge I/O penalty.
Note: The information for the article are Gathered from Wikipedia, How Things Work website, Pixar Animation Studio and their developed algorithm, Nvidia DirectCompute Algorithm Showcase