Introduction to Anti-Aliasing

Hello Everyone,

So we’ve all heard about anti-aliasing, but what is it really?  In this tutorial I’ll be giving you a background on what anti-aliasing and sub-sampling mean in 3dsMax so that you can jump into a production setting with some understanding right off the bat.  We’ll start off by looking at anti-aliasing in the scanline renderer, followed by sub-sampling in the scanline renderer and mental ray, and finally we’ll review anti-aliasing in mental ray.

What is Anti-Aliasing and Sub-Pixel Sampling?

Rendering in 3dsMax is a very literal process.  The renderer takes samples of what the camera sees and converts them into pixels.   Only the geometry at the very center of the pixel is considered.  Sub-pixel sampling and anti-aliasing are two sides of the same coin.  One operates before the pixel is rendered, while the other operates afterward.  Sub-pixel sampling is the process of taking more than one sample per pixel and blending them together before the pixel is returned as part of the image.  Anti-aliasing is taking the pixels after they’ve been produced in the image and blending them together to remove harsh edges.

Notice the jagged edges in the version without anti-aliasing!

That said, sub-pixel sampling is a smarter but more intense process, so there needed to be a way to quickly take the edge off the final image.  That’s why we have anti-aliasing.  Note that anti-aliasing and sub-pixel sampling take time, and disabling them will make your renders go faster (though they aren’t as pretty).

But what methods of anti-aliasing and sub-pixel sampling are best?  There have been several revolutions on both fronts in 3dsMax.  mental ray and the scanline renederer have methods that each deliver a distinct (albeit subtle) flavor of image smoothing.  Let’s blast through examples of each and get acquainted with the options available.

Scanline Anti-Aliasing

What I’m going to do here is elaborate on the descriptions provided in the 3dsMax help file.  The fact is that whoever wrote this particular article didn’t really go into much detail about what each method looks like, so I’m going to fill in the gaps for you.  Each of these renders was taken at the resolution you see on your screen, and each method was left on default values.  I used a marble texture because it has plenty of hard edges to be smoothed.

Method	Description	Comments	Image
Area (default)	Computes antialiasing using a variable-size area filter. This is the original 3ds Max filter.	The original filter isn’t bad. Think of it as a gentle Gaussian blur.
Blackman	A 25-pixel filter that is sharp, but without edge enhancement.	This one is a little tighter than the Area method, but it doesn’t explicitly sharpen the image.
Blend	A blend between sharp area and Gaussian soften filters.	Wicked blurry for this resolution, but otherwise a nice smoothing method.  Notice the edges are still crisp.
Catmull-Rom	A 25-pixel reconstruction filter with a slight edge-enhancement effect.	Crisp with sharpened edges.  No options on this one.
Cook Variable	A general-purpose filter. Values of 1 to 2.5 are sharp; higher values blur the image.	A little blurrier, but the edges are retained in an overlay-like fashion, even for high values.
Cubic	A 25-pixel blurring filter based on a cubic spline.	Blurrier than Cook Variable without edge enhancement.
Mitchell-Netravali	Two-parameter filter; a trade-off of blurring, ringing, and anisotropy. If the ringing value is set higher than .5 it will impact the alpha channel of the image.	A more robust filter, the default values look like Catmull-Rom.
Plate Match/MAX R2	Uses the 3ds Max 2 method (no map filtering) to match camera and screen maps or matte/shadow elements to an unfiltered background image.	A legacy method.  It’s generally for matte-shadow composition.  See the F1 help file for more details.
Quadradic	A 9-pixel blurring filter based on a quadratic spline.	Similar to cubic; general blurring.
Sharp Quadratic	A sharp nine-pixel reconstruction filter from Nelson Max.	Not much to say; similar to the Quadradic shown above.
Soften	An adjustable Gaussian softening filter for mild blurring.	Gaussian blurring.  Similar to Cubic and Quadradic, this is general blurring without edge enhancement.
Video	A 25-pixel blurring filter optimized for NTSC and PAL video applications.	For use when rendering video with the 3dsMax scanline renderer.

Scanline Super-Sampling

Super-sampling in the scanline renderer is what you use to control the sub-pixel activity of the renderer. This is different from anti-aliasing because, while it controls edge blending, it does so through sub-pixel sampling rather than through pixel bleeding.

Here is where you can find the super-sampling controls.

Below I’ve included a table showing you example renders for each of these super-sampling methods. I’ve used the same example scene as before, and I’m using the default area anti-aliasing method. Again, I left the default values for each sub-sampler and these images were rendered using the 3dsMax scanline renderer at the resolution you see on your screen. Remember that using sub-sampling will slow down your renders considerably more than anti-aliasing!

Method	Description	Comments	Image
Adaptive Halton	Spaces samples along both X and Y axes according to a scattered, “quasi random” pattern. Depending on Quality, the number of samples can range from 4 to 40.	Similar to Hammersley, but provides a randomization that extends the uniform variant.
Adaptive Uniform	Spaces samples regularly, from a minimum quality of 4 samples to a maximum of 36. The pattern is not square, but skewed slightly to improve accuracy in the vertical and horizontal axes.	Provides a generally sharper outcome.
Hammersley	Spaces samples regularly along the X axis, but along the Y axis it spaces them according to a scattered, “quasi random” pattern. Depending on quality, the number of samples can range from 4 to 40.	Provides a generally smoother outcome. This method is not adaptive!
Max 2.5 Star	The sample at the center of the pixel is averaged with four samples surrounding it. The pattern is like the fives on dice. This is the super sampling method that was available in 3ds Max 2.5.	Provides the smoothest outcome. This method is not adaptive either.

The two adaptive methods are named so because they adapt to the change in pixel contrast while the renderer is operating. In areas of low contrast, the sub-sampler will back off and move more quickly, while in areas of high contrast, the sub-sampler will work more deeply. You’ll see a similar feature in the mental ray renderer soon.

mental ray Sub-Pixel Sampling

In mental ray, we don’t have super-sampling, we have “sub-pixel sampling”. This gives us a very robust level of control over how mental ray samples each pixel in our image.  Rather than rely on a forumla to handle sampling, we can simply tell mental ray the sampling levels for high contrast areas versus low contrast areas.  We do this with the Samples per Pixel controls in the “Render Setup” rollout, similar to the scanline methods.

These two pull down menus control the minimum and maximum sampling.

The sampling in mental ray is given as a number or a fraction.  The whole values indicate how many samples should be taken per pixel (i.e. a value of 1 is just one sample per pixel, while a value of 4 is 4 samples per pixel).  Fractional values indicate how many pixels can be filled with a single sample.

mental ray sub sampling at 1/4 and 4

You can probably already see the implications of such a system.  If you set the maximum and minimum sampling to 1/4, you’ll get a very blocky, but very fast render.  On the other hand, if you set the maximum and minimum sampling to 4, you’ll get a slow but clean render.

Without getting lost in the minutia of 3dsMax mental ray sub-sampling, I’ll also point out that there is a spatial contrast group just below the samples per pixel group.  This group controls how mental ray should chose between your minimum and maximum sampling levels based on the contrast across pixels.  By default this is set to [5, 5, 5, 5] which is just shy of a 1% difference across pixels when rendering.  However, you can change the threshold to a higher value if you want mental ray to bias itself toward the minimum (faster) sampling instead of the maximum (higher quality) sampling.

Anti-Aliasing in mental ray

Finally, let’s look at the anti-aliasing options in mental ray. Since there are only 5 flavors of smoothing, I’m going to include two sample images for each instead of just one. The first image will be using the default values, while the second image will show double the default values. For example, the box smoothing in mr is defaulted at width:1.0 and height:1.0. In image 2 for that method, the values are width:2.0 and height:2.0.

As with the 3dsMax scanline anti-aliasing examples, these images are all rendered at the resolutions you see on your screen. All other mental ray settings are left at defaults (including the sub-sampling levels of 1/4 minimum and 4 maximum).

Method	Description	Comments	Image	Image
Box filter	Sums all samples in the filter area with equal weight. This is the quickest sampling method.	Typical blurring; just blends the adjacent sub-pixels together.
Gauss filter	Weights the samples using a Gauss (bell) curve centered on the pixel.	The Gauss filter appears blurrier because it has a larger default size (3,3) than the box filter.
Triangle filter	Weights the samples using a pyramid centered on the pixel.	Generally yields crisper results.
Mitchell filter	Weights the samples using a curve (steeper than Gauss) centered on the pixel.	Generally considered the best filter in mental ray.
Lanczos filter	Weights the samples using a diminishing, but steep curve.	A fine filter that accentuates detail.

And that’s the rundown!I hope this tutorial has given you an idea of what anti-aliasing and sub-sampling are all about in 3dsMax. Just remember that it’s a quality/speed trade off like most things in computer graphics. You need to find the happy medium in order to come off a head. When in doubt, aim for faster (usually lower) values when testing, and quality (usually higher) values when doing your final output render.

Until next time, happy rendering!