Artistic Video Chat Effects in Real-Time
We’ve been thinking that if GPU-accelerated, our effects could work well and be fast enough for server-based video chat and other streaming applications. As a test we implemented selected cartoon and painterly effects in CUDA (see examples below) using an NVIDIA GeForce GTX 480 Fermi card. Our effects are now rendering at real-time rates for 1280×720p HD, 18-22fps. We’ve also seen a comparable speedup for smaller frame sizes - 180-190fps for 320×240; 45-50fps for 720×480p SD.
At 18-22fps HD, our new CUDA-based effects are currently faster and support higher image resolutions than any fielded video chat application we’ve seen to date. Besides video chat, our new, faster effects could also work well for online video editing or other contexts where people might like to be playful and remain a bit anonymous, such as online dating. Another obvious real-time use will be to enhance replays in broadcast sports. They could also add real-time fun to interactive installations at art, music, social, and corporate events. We’re actively seeking new partnerships to bring these effects to the public.
Click on any icon below to pop the 1280×720p original videos (chosen at iStock and edited to simulate video chat) or corresponding Fermi-rendered demo videos. Our test effects were also selected, and in some cases designed, to work well for video chat. Cartoon is a CUDA reimplementation of our classic CPU Rototoon effect, also including Soft Outlines. Watercolor and Impasto are new effects, which will later be introduced in forthcoming GPU ToonIt. Impasto renders output with something of a similar appearance to that of the well-known Kuwahara image processing filter, while improving on it, but takes advantage of a new, previously unknown algorithm for edge-preserving smoothing.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
In our testing, we used CPU libraries to read and write each frame. Note that if we redid this with hardware IO instead, we’d expect to be significantly faster. Alternatively, we can obtain additional big speedups by directly integrating our effects into existing OpenGL pipelines. This would allow us to render to passed-in pointers to frame buffers that we could read from and write to without having to move data off and onto the card, which are among the slowest steps. Finally, even without the large speedups expected in next year’s Kepler Nvidia card, we can render HD video at more than real-time speeds right now by running in two GPU configurations, such as currently available in Amazon’s EC2 cloud computing through new cluster GPU instances.
ETA (12/4/10): Based on the specs of the latest GeForce GTX 580 Nvidia card, already out, we can get a 20% single-card speedup relative to our test results just by switching cards.