Okay, here we go.
Neural Volume Rendering for Dynamic Scenes
NeRF has shown incredible view synthesis results, but it requires multi-view captures for STATIC scenes.
How can we achieve view synthesis for DYNAMIC scenes from a single video? Here is what I learned from several recent efforts.
Okay, here we go.
NeRF represents the scene as a 5D continuous volumetric scene function that maps the spatial position and viewing direction to color and density. It then projects the colors/densities to form an image with volume rendering.
Volumetric + Implicit -> Awesome!
Building on NeRF, one can extend it for handling dynamic scenes with two types of approaches.
A) 4D (or 6D with views) function.
One direct approach is to include TIME as an additional input to learn a DYNAMIC radiance field.
e.g., Video-NeRF, NSFF, NeRFlow
Inspired by non-rigid reconstruction methods, this type of approach learns a radiance field in a canonical frame (template) and predicts deformation for each frame to account for dynamics over time.
e.g., Nerfie, NR-NeRF, D-NeRF
All the methods use an MLP to encode the deformation field. But, how do they differ?
A) INPUT: How to encode the additional time dimension as input?
B) OUTPUT: How to parametrize the deformation field?
One can choose to use EXPLICIT conditioning by treating the frame index t as input.
Alternatively, one can use a learnable LATENT vector for each frame.
We can either use the MLP to predict
- dense 3D translation vectors (aka scene flow) or
- dense rigid motion field
More from Tech
The YouTube algorithm that I helped build in 2011 still recommends the flat earth theory by the *hundreds of millions*. This investigation by @RawStory shows some of the real-life consequences of this badly designed AI.
This spring at SxSW, @SusanWojcicki promised "Wikipedia snippets" on debated videos. But they didn't put them on flat earth videos, and instead @YouTube is promoting merchandising such as "NASA lies - Never Trust a Snake". 2/
A few example of flat earth videos that were promoted by YouTube #today:
https://t.co/TumQiX2tlj 3/
https://t.co/uAORIJ5BYX 4/
https://t.co/yOGZ0pLfHG 5/
Flat Earth conference attendees explain how they have been brainwashed by YouTube and Infowarshttps://t.co/gqZwGXPOoc
— Raw Story (@RawStory) November 18, 2018
This spring at SxSW, @SusanWojcicki promised "Wikipedia snippets" on debated videos. But they didn't put them on flat earth videos, and instead @YouTube is promoting merchandising such as "NASA lies - Never Trust a Snake". 2/
A few example of flat earth videos that were promoted by YouTube #today:
https://t.co/TumQiX2tlj 3/
https://t.co/uAORIJ5BYX 4/
https://t.co/yOGZ0pLfHG 5/
One of the best decisions I made during a very turbulent 2020 was to leave conventional coding behind and embrace the #nocode movement. @bubble made this a reality. Although my own journey thus far is premature, I’ve learned a lot so here’s a power thread on....
‘How I created @buildcamp sales funnel landing page in under 2hours’.
Preview here 👇
https://t.co/s9P5JodSHe
Power thread here 👇
1. Started with a vanilla bubble app ensuring that all styles and UI elements were removed. Created a new page called funnel and set the page size to 960px as this allows the page to render proportionately on both web and mobile when hitting responsive breakpoints.
2. Began dropping elements onto the page to ‘find the style’. These had to be closely aligned to our @buildcamp branding so included text, buttons and groups - nothing too heavy. Played around with a few fonts, colors and gradients and thus pinned down the following style guide.
3. Started to map out sections using groups as my ‘containers’ to hold the relevant information and imagery needed to pad out the sales pitch. At this point, they were merely blocks of color #ff6600 with reduced opacity set to 5% to ease page flair.
‘How I created @buildcamp sales funnel landing page in under 2hours’.
Preview here 👇
https://t.co/s9P5JodSHe
Power thread here 👇
1. Started with a vanilla bubble app ensuring that all styles and UI elements were removed. Created a new page called funnel and set the page size to 960px as this allows the page to render proportionately on both web and mobile when hitting responsive breakpoints.
2. Began dropping elements onto the page to ‘find the style’. These had to be closely aligned to our @buildcamp branding so included text, buttons and groups - nothing too heavy. Played around with a few fonts, colors and gradients and thus pinned down the following style guide.
3. Started to map out sections using groups as my ‘containers’ to hold the relevant information and imagery needed to pad out the sales pitch. At this point, they were merely blocks of color #ff6600 with reduced opacity set to 5% to ease page flair.