Back when I worked at Framestore they had a very useful, in-house plugin for Shake called 'fractalBlur'. It's essentially just a blur combined with a noise filter so that the softened image does not have smooth gradients. It really helps to hide soft-edge mattes where the combined images have a lot of texture. I was working on Where the Wild Things Are at the time and every plate almost entirely consisted of heavy natural texture (forests, trees, fur etc.) and it became essential to use the fractalBlur on every single mask. Since then I frequently require this plugin and have not found an equivalent in Nuke, so thought I'd put one together.

THE PROBLEM

Take the examples above. Here the A and B plates that need to be combined have a lot of distinct texture. We create a matte using a roto shape and heavily blur it to hide the edges. But when the plates are merged the overlapping areas lose a lot of their texture - where the matte had a partial opacity the comped image looks soft (its tricky to see on a heavily compressed image on a website, but when working at film plate resolution it becomes very noticeable). A similar thing happens when doing cloning / paint work with a soft brush on an image that hasn't first been degrained - the painted areas will look soft compared to the untouched areas. When you are masking with a large, feathered roto, this is just a larger scale version of the same problem. Another thing to consider is that smooth, linear gradients never happen in nature. Even a clear blue sky is never just a simple, linear ramp from one shade of blue to another - there will always be some subtle irregularity. So when we use roto shapes / ramps as masks we draw attention to them, as the linear nature of a procedural blur / feather looks un-natural.

THE SOLUTION

To get around this we can break up the blurred areas by generating noise and multiplying it against the matte first. I've cranked up the contract in this case and got the grain size to something that roughly matches the general look of the plates. Even though the matte is a lot less gradated the blended edge is a lot less apparent. Ive wrapped this up in a Nuke group node and added some knobs to control the size of the noise used, contrast etc.

ADDITIONAL CONSIDERATIONS

One thing to note is that even though the noise can be animated in the sense of it evolving over time, it will not 'transform' to match the movement of your input. If your matte needs to move, then you need to apply your movement after the fractal blur, otherwise the texture of the blur will not lock to it and will appear to slide relative to the mask.

TEXTURE INPUT

Thanks again to Holger Hummel at Celluloid VFX who has expanded upon my original tool and now added the ability to plug in a image to use as the fractal texture.

In this example, we plug in this 'ice hockey background' image and with some tweaks we can generate a grass-like texture. You can click the image below to get the full 5k texture.

This has proven very handy for creating a holdout when comping an image into a grassy field.

The code in GitHub has been updated to v1.6 to reflect this. You can download the nuke script with the grassy field demo above here