Understanding the concepts of Neural Radiance Fields and Neural Rendering
In this article, we will start with an explanation of what is Computer Graphics (a very short introduction, with a brief on volume rendering and view synthesis). We will then have an overview of what is Neural Radiance Fields and how it is used in view synthesis.
Computer Graphics Introduction
Before we can jump on to Neural Radiance Fields it is good to be familiar with different areas of computer graphics. Computer Graphics (as per Wikipedia) is a branch of computer science that deals with generating images with the aid of computers. Now the question that might come is why focusing on only images and not videos? Well, the answer is simple a video is nothing but a time-series of images (FPS: Frame Per Second) in a sequence. So when you are running a game (let’s say Arkham Asylum) then, at that time your GPU computes scenes as an image. What you see next is your character (let’s say Batman) interacting with an environment (Jumping from Buildings, driving Batmobile, punching joker ) in terms of frames (a collection of images) that is being rendered per second for each of those scenes. So for simplicity, we can say that videos are nothing but several images in time duration (imagine it like a flipbook or flick book).
Now computer graphics is a very vast area in itself comprising of multiple topics like UI design, rendering, ray tracing, vector graphics, geometry processing, 3D modeling, image processing, shading, Material and texture, and so on. This is not a tutorial for computer graphics so please feel free to research the different topics. Every single topic mentioned above is an area of specialization in itself. I will rather be focusing on two things that too in the brief first one is rendering and another is view synthesis. So let’s get started.\
Neural Radiance Fields
NeRF or better known as Neural Radiance Fields is a state-of-the-art method that generates novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. The input can be provided as a blender model or a static set of images.
The input is provided as a continuous 5D function that outputs the radiance emitted in each direction (θ; Φ) at each point (x; y; z) in space, and a density at each point which acts like a differential opacity controlling how much radiance is accumulated by a ray passing through (x; y; z).
Representing Scenes as Neural Radiance Fields for View Synthesis: Click Here
Nvidia shows off AI model that turns a few dozen snapshots into a 3D-rendered scene: Read Here
Watch Here:
