Real-Time Skin Cleanup for Video Chat and Broadcast

Some time ago we devised a new, unpublished, algorithm for edge-preserving smoothing. We discovered that it possessed the desirable property of being able to significantly reduce visible skin discolorations, blemishes, and wrinkles in facial photos and videos, while leaving minimal signs of blurring or other image manipulation. In other words, it produced results resembling hand-applied photo-reouching, but that could be applied automatically to each frame of a video. We showed our results to Digital Anarchy, who licensed it as the core skin-smoothing algorithm for their Beauty Box plugin for After Effects, Photoshop, and Final Cut Pro. Beauty Box went on to win multiple awards in 2010, including:

  • Mario award for New Technology at NAB 2010;
  • 5 star rating in Photoshop User;
  • Best Video Software Plugin award in Videomaker’s Magazine Best Video Products of the Year 2010.

We originally created this image-processing technology as CPU-only, but we’ve since ported it to CUDA-based Nvidia GPUs without loss of quality. We’ve now gotten it to run at the same speed ranges (tending towards the faster end) as our GPU-accelerated cartoon and painterly rendering effects (see http://toonamation.com/2010/11/28/artistic-video-chat-effects-in-real-time), meaning that it can be applied to enhance facial appearance in HD video in real-time.

Here’s some example output. Click on icons below to pop the 1280x720p original videos (chosen at iStock and edited as brief facial closeups) or corresponding post-process cleanup videos.

Original Clips:

Click to Play Click to Play Click to Play
     
Skinsmooth Level 1:

Click to Play Click to Play Click to Play
     
Skinsmooth Level 2:

Click to Play Click to Play Click to Play
     

For Skinsmooth Level 1 we selected settings to produce a pleasing amount of skin cleanup, while otherwise looking as naturalistic as possible (notice the retained hair detail and skin pores), so the videos wouldn’t look noticeably manipulated. Skinsmooth Level 2 enables somewhat more cleanup to resemble classic photo retouch or airbrushing. Many intermediate, plus stronger or weaker adjustments to degree/quality of smoothing are possible. For most applications, we’ll probably offer several preset setting combinations to support a range of user preferences. The effect gives best results for close-ups and medium shots, and settings can be further optimized by taking both shot type and image resolution into account.

We see a wide range of potential applications for this technology, including:

  1. Enhancing the appearance of participants with or without makeup in live or rapid turn-around broadcast tv, especially where there are HD facial closeups – including newscasts, interviews on talk, sports, and commentary shows, and on-scene interviews and reports for live-streaming.
  2. Improving the appearance of participants in business video conferences.
  3. Improving the appearance of self-conscious teens (or anyone with a skin condition/concern) in consumer-oriented video chat.
  4. Enhancing facial appearance in videos done for, and video chat associated with, online dating sites. Note there is already an industry to hand retouch photos submitted to such sites. We could do potentially do the same in high volume, plus offer the similar enhancements in real time for submitted videos.
  5. Allowing online facial photo/video cleanup for any non-government site where people are submitting photos/videos, or as a quick processing intermediary when people are sending photos/videos to each other through mobile devices.

We’re not primarily consumer-facing, so we’re looking to partner with companies in the above application areas who are. In order to run real-time our effects require a CUDA-enabled Nvidia card to be present somewhere along the transmission path, e.g., embedded in a camera, mobile device, intermediate server, display device, or computer at the broadcast or receiving end. We’re simultaneously exploring architectures assuming that our GPU-accelerated image processing can be supported offsite in the cloud, e.g., applying Amazon EC2’s new cluster GPU instances. We’re also expecting to conduct near future ports of our software to work on OpenCL-capable ARM processors for mobile devices as they become available.

Artistic Video Chat Effects in Real-Time

We’ve been thinking that if GPU-accelerated, our effects could work well and be fast enough for server-based video chat and other streaming applications. As a test we implemented selected cartoon and painterly effects in CUDA (see examples below) using an NVIDIA GeForce GTX 480 Fermi card. Our effects are now rendering at real-time rates for 1280x720p HD, 18-22fps. We’ve also seen a comparable speedup for smaller frame sizes – 180-190fps for 320×240; 45-50fps for 720x480p SD.

At 18-22fps HD, our new CUDA-based effects are currently faster and support higher image resolutions than any fielded video chat application we’ve seen to date. Besides video chat, our new, faster effects could also work well for online video editing or other contexts where people might like to be playful and remain a bit anonymous, such as online dating. Another obvious real-time use will be to enhance replays in broadcast sports. They could also add real-time fun to interactive installations at art, music, social, and corporate events. We’re actively seeking new partnerships to bring these effects to the public.

Click on any icon below to pop the 1280x720p original videos (chosen at iStock and edited to simulate video chat) or corresponding Fermi-rendered demo videos. Our test effects were also selected, and in some cases designed, to work well for video chat. Cartoon is a CUDA reimplementation of our classic CPU Rototoon effect, also including Soft Outlines. Watercolor and Impasto are new effects, which will later be introduced in forthcoming GPU ToonIt. Impasto renders output with something of a similar appearance to that of the well-known Kuwahara image processing filter, while improving on it, but takes advantage of a new, previously unknown algorithm for edge-preserving smoothing.

Original Clips:

Click to Play Click to Play Click to Play
     
Cartoon:

Click to Play Click to Play Click to Play
     
Watercolor:

Click to Play Click to Play Click to Play
     
Impasto:

Click to Play Click to Play Click to Play
     

In our testing, we used CPU libraries to read and write each frame. Note that if we redid this with hardware IO instead, we’d expect to be significantly faster. Alternatively, we can obtain additional big speedups by directly integrating our effects into existing OpenGL pipelines. This would allow us to render to passed-in pointers to frame buffers that we could read from and write to without having to move data off and onto the card, which are among the slowest steps. Finally, even without the large speedups expected in next year’s Kepler Nvidia card, we can render HD video at more than real-time speeds right now by running in two GPU configurations, such as currently available in Amazon’s EC2 cloud computing through new cluster GPU instances.

ETA (12/4/10): Based on the specs of the latest GeForce GTX 580 Nvidia card, already out, we can get a 20% single-card speedup relative to our test results just by switching cards.

Loads of Automatic Cartoon Effects

From the many cartoon, painterly, and artistic effects developed by Toonamation, we’ve pulled out a selection of the best at producing good-looking results automatically (without tweaking) for the widest range of videos. We think they’re well-suited for server-based user editing of web or mobile video clips. These effects aren’t real-time, but they are available as multi-threaded binary libraries for Windows or *NIX servers that are still quite fast. They also scale up cleanly for use with HD video, and will be great for editing video keepsakes for gift DVDs. We’ve been working on GPU-based acceleration using CUDA (to allow us better quality and a wider variety of effects than available through Flash) and have seen 7X speedups so far. By early spring of next year we expect to be able to offer real-time versions for many of these effects at 12-15 frames per second at web or mobile phone display resolutions – especially suitable for conversational and interactive video.

Click on any icon to pop the corresponding 640×360 demo video.

ETA (5/11/10): It’s taken a little longer than we expected, but we’ve completed a CUDA GPU port of the acrylic effect below, and I’ve recently clocked it at over 100fps at 320×240 resolution, 9-10fps at 720p. We’re in the process of porting the rest of the effects.

Original Clips:

Click to Play Click to Play Click to Play
     
Acrylic:

Click to Play Click to Play Click to Play
     
Airbrush:

Click to Play Click to Play Click to Play
     
Blacklight:

Click to Play Click to Play Click to Play
     
Cartoon:

Click to Play Click to Play Click to Play
     
Crayon:

Click to Play Click to Play Click to Play
     
Glow:

Click to Play Click to Play Click to Play
     
Heat Vision:

Click to Play Click to Play Click to Play
     
Night Vision:

Click to Play Click to Play Click to Play
     
Oilpaint:

Click to Play Click to Play Click to Play
     
Paint by Number:

Click to Play Click to Play Click to Play
     
Poster:

Click to Play Click to Play Click to Play
     
Scratchboard:

Click to Play Click to Play Click to Play
     
Screenprint:

Click to Play Click to Play Click to Play
     
Sepia:

Click to Play Click to Play Click to Play
     
Watercolor:

Click to Play Click to Play Click to Play
     
Woodcut:

Click to Play Click to Play Click to Play
     

Improved PosterPaint and New PaintByNumber

I recently figured out a way for us to improve the temporal coherence of PosterPaint. It involves selectively blending the original image back into the filter output in places where the original image was smoothest. This is a variation of the approach we use in Supersmart Blur, and it gives better results than our previous method of selectively blending PosterPaint output with a blurred version of itself. Our new method also works when used with PaintByNumber. PaintByNumber is a new effect we’ve developed that looks somewhat similar to PosterPaint, except the colors are more individually separated. Look for these enhancements to show up in future ToonIt releases.

Sample original video:
Click to Play

PosterPaint – Old method for temporal coherence correction:
Click to Play

PosterPaint – New method for temporal coherence correction:
Click to Play

PaintByNumber – Old method for temporal coherence correction:
Click to Play

PaintByNumber – New method for temporal coherence correction:
Click to Play

Technorati Tags: , , ,

ToonIt&#153 Bundled with Pinnacle Studio 14 Ultimate

ToonIt&#153 is now bundled with the latest versions of bestselling video editor Pinnacle Studio, Pinnacle Studio 14 Ultimate and Pinnacle Studio 14 Ultimate Collection. Pinnacle Studio 14 Ultimate and Ultimate Collection are available at online and offline software stores worldwide. If you’re a novice video editor, this is a great and inexpensive way for you to get access of the artistic power of ToonIt. If you like the Pinnacle and ToonIt combo, please vote them up and give positive comments at your online store, e.g., Amazon, so we can sell more copies and afford to develop even more cool ToonIt effects in the future. If you don’t like it (ToonIt specifically) or would like additional effects or controls, please let us know, e.g., in the comments below. Your feedback can help us keep making ToonIt better.

Anisotropic Kuwahara Filtering versus ToonIt

I heard about the new anisotropic Kuwahara filtering non-photorealistic rendering technique of Kyprianidis, Kang, and Döllner, unveiled earlier this month at NPAR 2009, and decided to compare it with Toonamation’s Gouache filter, publicly available in photo and video ToonIt. See Smooth Paint under http://toonamation.com/2008/03/26/pre-release-video-effects/ from March 2008 for a prerelease demo of Gouache using the same algorithm. For this comparison I set Gouache Quality to High, Spread to 2, Presharpen and Postsharpen to 0.0, and turned off Main Blur and Outlines.

Original:
Click to Play

Kyprianidis:
Click to Play

Gouache:
Click to Play

Click on the images above for a pop-up clip of the source video and clips of the respective filters applied to the same content (click on more than one and reposition to compare). The outputs of both filters look equally abstract and temporally coherent, but the Gouache filter from ToonIt produces results that appear more painterly and less smoothed and also retain more of important contrast details, especially around the face. The Kyprianidis algorithm is currently faster, although it is not yet out in publicly usable form. We are working on and plan to release a GPU implementation of Gouache, as well as our other filters, later this year or early next year.

The image and clips for this comparison were derived from the short film fatalité, produced at MIT. For the record, Toonamation’s Gouache doesn’t use structure tensors to generalize the Kuwahara filter.

Technorati Tags: , , , , , , , , , ,

Prerelease Acrylic Effect

We’ve developed a new effect called soft-borders that resembles a flat acrylic painting. Color regions are sometimes smoothly blended, sometimes provided strong transitions, for a painterly look that combines tight brushwork with loose washes. Soft-Borders renders faster than our well-know Rototoon (flat-shaded) effect. We’re currently working on CUDA-based speedup of all our effects, to be extended next year to OpenCL support for all GPU cards, and after GPU porting there’s a good chance that Soft-Borders will render in real-time or near real-time.

Here’s a demo clip applying Soft-Borders to a woman running in a field at twilight. It looks like it was lifted directly from a horror or suspense cartoon. You’ll have to add the zombies or evil mutants yourself, and the possible uses are limited only by your imagination. This look has great temporal coherence and darks are deftly handled and maintained without coalescing into pulsing blobs typical of posterize-based cartooning. We expect this to be a welcome addition to your toolbox in future ToonIts, and we’re also making it available for separate and standalone applications. Contact me through twitter DM or on LinkedIn if interested.

Click to Play

By the way, the royalty-free pre-transformed video clip was from iStockVideo, and royalty-free music track, Chase Pulse, was by Kevin MacLeod.

Technorati Tags: , , , , , , , , , , , ,

ToonIt&#153 on Sale in February

ToonIt! Photo for Mac Aperture 2.0 and higher is now available for trial or purchase at Digital Anarchy. It’s normally priced at $129, but is on launch sale for $99 until February 28, 2009. If you already have the Photoshop version of ToonIt, then the Aperture version is only $49. Also from Feb 10-20, Red Giant is doing a 35% off sale on all ex-DA products, including ToonIt. To get the discount, enter coupon code NLDA0209 on the Red Giant web site during purchase.

Toonamation Sampler on YouTube

Now that we’ve completed development work on the forthcoming ToonIt upgrade at Red Giant, I decided we should focus more on our marketing outreach, so I’ve started posting more demos to YouTube, including a sampler of our new effects. You can also check out at the sampler at much higher resolution here at http://toonamation.com/Toonamation_Sampler.mov. Oh yeah, the royalty-free original video was from iStockVideo, and royalty-free music track is called “Human” by Laurie Laptop.

001_roto_conte1.png

Automatic Sin City Look

I converted a clip from Heroes to look like it was drawn and animated by Frank Miller, creator of the Sin City graphic novel. This was a research test of some of the latest non-photorealistic rendering, image-processing tech we’ve developed here at Toonamation. Click the image below to see it as a popup video. This particular work also has a parody element that I’ll leave you to discover for yourselves.

Click to Play

Technorati Tags: , , , , , , , , , , , , , , , , , , , , , ,