Multisampling Anti-Aliasing: A Closeup View
Oversampling: The easiest way
The triangle setup only knows 2 different conditions per pixel: triangle covered or not. That's where a digitising error comes into play. Actually, the pixels on the triangle edge are just partially covered, but this is ignored. If the centre of the pixel is covered, the entire pixel is treated as a part of the triangle.
An effect of this digitising error is "stair" steps appearing, where a smooth edge is supposed to be. In general, this effect is inevitable because it is rendered onto a pixel raster (which, by the way, is very coarse comparing to the accuracy of the geometry data). The visibility of these steps could be reduced, for example by taking into account how much of the pixel is covered. I.e. if 50 per cent of a pixel were covered, the corresponding triangle's colour value would make up only 50 per cent of the final on-screen pixel. But to calculate the exact coverage and mix the colours accordingly is far beyond today's hardware, so in practice an approximation of the actual coverage is used. Hence every implementation of 3D anti-aliasing procedures nowadays is an approximation.
One way to achieve smoother edges is oversampling: The X and Y axes are scaled up by an integral factor, and after rendering this enlarged frame, it is filtered back down to the original resolution. This means that in this process of downfiltering the information of several framebuffer pixels makes it into one final on-screen pixel. That way "softer" edges with colour transitions are achieved.
2x2 oversampling: The internal resolution is higher.
After the downfiltering the edges are a bit smoother.
Whenever possible the filter kernel should cover whole pixels and not include fractions of pixels. A scaling by 1.5x1.5 for example also produces smoother edges, but the result isn't convincing:
Scaling by non-integral factors is disadvantageous.
Texture quality will also suffer from non-integral oversampling.
Supersampling: Can we get better smoothing for the same expense?
Oversampling is a supersampling method, but not vice versa. Supersampling can be made more efficient through repositioning of subpixels. Then terms like "edge equivalent resolution" become important. Why the edge smoothing can be improved through a different subpixel mask, is discussed later on in this article.
Such "sophisticated" supersampling needs special hardware support though, while oversampling can also be realised via the drivers alone. In this case the driver does not take over the anti-aliasing calculations (this is not "software anti-aliasing"!!), but guides the card through certain steps, required for the generation of primitive supersampling procedures.
Separating "oversampling" from "supersampling" is not the most common approach, one could simply distinguish between ordered grid supersampling (OGSS) and rotated grid supersampling (RGSS). But we find using the distinct term "oversampling" has its merits, since it describes the quite different approach of just scaling the axes, while supersampling also encompasses methods of higher quality at a given performance impact.
The more advanced so called "rotated grid supersampling" (the name refers to the layout of the subpixel mask) could also be realised with a bit of work solely via the drivers if the card supports multisample buffers. But with this method the required amount of geometry calculations is multiplied. Cards with support for multisample buffers usually also posses the special anti-aliasing hardware support we talked about earlier.
Supersampling has the disadvantage of occupying a whole pixel pipeline per subpixel. Multisampling can break this rule.