LearnOpenGL

Stencil-testing.html (16854B)

---

     <div id="content">
     <h1 id="content-title">Stencil testing</h1>
 <h1 id="content-url" style='display:none;'>Advanced-OpenGL/Stencil-testing</h1>
 <p>
   Once the fragment shader has processed the fragment a so called <def>stencil test</def> is executed that, just like the depth test, has the option to discard fragments. After that the remaining fragments are passed to the depth test where OpenGL could possibly discard even more fragments. The stencil test is based on the content of yet another buffer called the <def>stencil buffer</def> that we're allowed to update during rendering to achieve interesting effects.
 </p>
 
 <p>
   A stencil buffer (usually) contains <code>8</code> bits per <def>stencil value</def> that amounts to a total of <code>256</code> different stencil values per pixel. We can set these stencil values to values of our liking and we can discard or keep fragments whenever a particular fragment has a certain stencil value.
 </p>
 
 <note>
   Each windowing library needs to set up a stencil buffer for you. GLFW does this automatically so we don't have to tell GLFW to create one, but other windowing libraries may not create a stencil buffer by default so be sure to check your library's documentation.
 </note>
 
 <p>
   A simple example of a stencil buffer is shown below (pixels not-to-scale):
 </p>
 
 <img src="/img/advanced/stencil_buffer.png" class="clean" alt="A simple demonstration of a stencil buffer"/>
 
 <p>
   The stencil buffer is first cleared with zeros and then an open rectangle of <code>1</code>s is stored in the stencil buffer. The fragments of the scene are then only rendered (the others are discarded) wherever the stencil value of that fragment contains a <code>1</code>. 
 </p>
   
 <p>
 	Stencil buffer operations allow us to set the stencil buffer at specific values wherever we're rendering fragments. By changing the content of the stencil buffer while we're rendering,  we're <em>writing</em> to the stencil buffer. In the same (or following) frame(s) we can <em>read</em> these values to discard or pass certain fragments. When using stencil buffers you can get as crazy as you like, but the general outline is usually as follows:
 </p>
 
 <ul>
   <li>Enable writing to the stencil buffer.</li>
   <li>Render objects, updating the content of the stencil buffer.</li>
   <li>Disable writing to the stencil buffer.</li>
   <li>Render (other) objects, this time discarding certain fragments based on the content of the stencil buffer.</li>
 </ul>
 
 <p>
   By using the stencil buffer we can thus discard certain fragments based on the fragments of other drawn objects in the scene.
 </p>
   
 <p>
   You can enable stencil testing by enabling <var>GL_STENCIL_TEST</var>. From that point on, all rendering calls will influence the stencil buffer in one way or another.
 </p>
   
 <pre><code>
 <function id='60'>glEnable</function>(GL_STENCIL_TEST);    
 </code></pre>
 
 <p>
   Note that you also need to clear the stencil buffer each iteration just like the color and depth buffer:
 </p>
 
 <pre><code>
 <function id='10'>glClear</function>(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); 
 </code></pre>
   
 <p>
   Also, just like the depth testing's <fun><function id='65'>glDepthMask</function></fun> function, there is an equivalent function for the stencil buffer. The function <fun><function id='67'>glStencilMask</function></fun> allows us to set a bitmask that is <code>AND</code>ed with the stencil value about to be written to the buffer. By default this is set to a bitmask of all <code>1</code>s not affecting the output, but if we were to set this to <code>0x00</code> all the stencil values written to the buffer end up as <code>0</code>s. This is equivalent to depth testing's <fun><function id='65'>glDepthMask</function>(GL_FALSE)</fun>:
 </p>
   
 <pre><code>
 <function id='67'>glStencilMask</function>(0xFF); // each bit is written to the stencil buffer as is
 <function id='67'>glStencilMask</function>(0x00); // each bit ends up as 0 in the stencil buffer (disabling writes)
 </code></pre>
   
 <p>
   Most of the cases you'll only be using <code>0x00</code> or <code>0xFF</code> as the stencil mask, but it's good to know there are options to set custom bit-masks.
 </p>
   
 <h2>Stencil functions</h2>
 <p>
     Similar to depth testing, we have a certain amount of control over when a stencil test should pass or fail and how it should affect the stencil buffer. There are a total of two functions we can use to configure stencil testing: <fun><function id='68'>glStencilFunc</function></fun> and <fun><function id='69'>glStencilOp</function></fun>.
 </p>
   
 <p>
   The <fun><function id='68'>glStencilFunc</function>(GLenum func, GLint ref, GLuint mask)</fun> has three parameters:
 </p>
   
 <ul>
   <li><code>func</code>: sets the stencil test function that determines whether a fragment passes or is discarded. This test function is applied to the stored stencil value and the <fun><function id='68'>glStencilFunc</function></fun>'s <code>ref</code> value. Possible options are: <var>GL_NEVER</var>, <var>GL_LESS</var>, <var>GL_LEQUAL</var>, <var>GL_GREATER</var>, <var>GL_GEQUAL</var>, <var>GL_EQUAL</var>, <var>GL_NOTEQUAL</var> and <var>GL_ALWAYS</var>. The semantic meaning of these is similar to the depth buffer's functions.</li>
   <li><code>ref</code>: specifies the reference value for the stencil test. The stencil buffer's content is compared to this value.</li>
   <li><code>mask</code>: specifies a mask that is <code>AND</code>ed with both the reference value and the stored stencil value before the test compares them. Initially set to all <code>1</code>s.</li>
 </ul>
   
 <p>
    So in the case of the simple stencil example we've shown at the start, the function would be set to:
 </p>
 
 <pre class="cpp"><code>
 <function id='68'>glStencilFunc</function>(GL_EQUAL, 1, 0xFF)
 </code></pre>
 
 <p>
   This tells OpenGL that whenever the stencil value of a fragment is equal (<var>GL_EQUAL</var>) to the reference value <code>1</code>, the fragment passes the test and is drawn, otherwise discarded.  
 </p>
   
 <p>
   But <fun><function id='68'>glStencilFunc</function></fun> only describes whether OpenGL should pass or discard fragments based on the stencil buffer's content, not how we can actually update the buffer. That is where <fun><function id='69'>glStencilOp</function></fun> comes in.
 </p>
   
   <p>
     The <fun><function id='69'>glStencilOp</function>(GLenum sfail, GLenum dpfail, GLenum dppass)</fun> contains three options of which we can specify for each option what action to take:
   </p>
   
   <ul>
     <li><code>sfail</code>: action to take if the stencil test fails.</li>
     <li><code>dpfail</code>: action to take if the stencil test passes, but the depth test fails.</li>
     <li><code>dppass</code>: action to take if both the stencil and the depth test pass.</li>    
   </ul>
   
 <p>
   Then for each of the options you can take any of the following actions:
 </p>  
   
 <table>
   <tr>
   	<th>Action</th>
   	<th>Description</th>
   </tr>  
   <tr>
     <td><code>GL_KEEP</code></td>
  	<td>The currently stored stencil value is kept.</td>
   </tr>
   <tr>
     <td><code>GL_ZERO</code></td>
  	<td>The stencil value is set to <code>0</code>.</td>
   </tr>
   <tr>
     <td><code>GL_REPLACE</code></td>
  	<td>The stencil value is replaced with the reference value set with <fun><function id='68'>glStencilFunc</function></fun>.</td>
   </tr>
   <tr>
     <td><code>GL_INCR</code></td>
  	<td>The stencil value is increased by <code>1</code> if it is lower than the maximum value. </td>
   </tr><tr>
     <td><code>GL_INCR_WRAP</code></td>
  	<td>Same as <var>GL_INCR</var>, but wraps it back to <code>0</code> as soon as the maximum value is exceeded.</td>
   </tr> 
   <tr>
     <td><code>GL_DECR</code></td>
  	<td>The stencil value is decreased by <code>1</code> if it is higher than the minimum value.</td>
   </tr>
   <tr>
     <td><code>GL_DECR_WRAP</code></td>
  	<td>Same as <var>GL_DECR</var>, but wraps it to the maximum value if it ends up lower than <code>0</code>.</td>
   </tr>
   <tr>
     <td><code>GL_INVERT</code></td>
  	<td>Bitwise inverts the current stencil buffer value.</td>
   </tr>
 </table>
 
 <p>
   By default the <fun><function id='69'>glStencilOp</function></fun> function is set to <code>(GL_KEEP, GL_KEEP, GL_KEEP)</code> so whatever the outcome of any of the tests, the stencil buffer keeps its values. The default behavior does not update the stencil buffer, so if you want to write to the stencil buffer you need to specify at least one different action for any of the options.
 </p>
 
 <p>
   So using <fun><function id='68'>glStencilFunc</function></fun> and <fun><function id='69'>glStencilOp</function></fun> we can precisely specify when and how we want to update the stencil buffer and when to pass or discard fragments based on its content.
 </p>
   
 <h1>Object outlining</h1>
 <p>
    It would be unlikely if you completely understood how stencil testing works from the previous sections alone so we're going to demonstrate a particular useful feature that can be implemented with stencil testing alone called <def>object outlining</def>.
 </p>
 
 <img src="/img/advanced/stencil_object_outlining.png" class="clean" alt="An object outlined using stencil testing/buffer"/>
   
 <p>
   Object outlining does exactly what it says it does. For each object (or only one) we're creating a small colored border around the (combined) objects. This is a particular useful effect when you want to select units in a strategy game for example and need to show the user which of the units were selected. The routine for outlining your objects is as follows:
 </p>
 
 <ol>
   <li>Enable stencil writing.</li>
   <li>Set the stencil op to <var>GL_ALWAYS</var> before drawing the (to be outlined) objects, updating the stencil buffer with <code>1</code>s wherever the objects' fragments are rendered.</li>
   <li>Render the objects.</li>
   <li>Disable stencil writing and depth testing.</li>
   <li>Scale each of the objects by a small amount.</li>
   <li>Use a different fragment shader that outputs a single (border) color.</li>
   <li>Draw the objects again, but only if their fragments' stencil values are not equal to <code>1</code>.</li>
   <li>Enable depth testing again and restore stencil func to <var>GL_KEEP</var>.</li>
 </ol>
 
 <p>
   This process sets the content of the stencil buffer to <code>1</code>s for each of the object's fragments and when it's time to draw the borders, we draw scaled-up versions of the objects only where the stencil test passes. We're effectively discarding all the fragments of the scaled-up versions that are part of the original objects' fragments using the stencil buffer. 
 </p>
 
 <p>
   So we're first going to create a very basic fragment shader that outputs a border color. We simply set a hardcoded color value and call the shader <var>shaderSingleColor</var>:
 </p>
 
 <pre><code>
 void main()
 {
     FragColor = vec4(0.04, 0.28, 0.26, 1.0);
 }
 </code></pre>
 
 <p>
   Using the scene from the <a href="https://learnopengl.com/Advanced-OpenGL/Depth-testing" target="_blank">previous</a> chapter we're going to add object outlining to the two containers, so we'll leave the floor out of it. We want to first draw the floor, then the two containers (while writing to the stencil buffer), and then draw the scaled-up containers (while discarding the fragments that write over the previously drawn container fragments). 
 </p>
 
 <p>
   We first need to enable stencil testing: 
 </p>
 
 <pre class="cpp"><code>
 <function id='60'>glEnable</function>(GL_STENCIL_TEST);
 </code></pre>
 
 <p>
   And then in each frame we want to specify the action to take whenever any of the stencil tests succeed or fail:
 </p>
 
 <pre class="cpp"><code>
 <function id='69'>glStencilOp</function>(GL_KEEP, GL_KEEP, GL_REPLACE);  
 </code></pre>
 
 <p>
   If any of the tests fail we do nothing; we simply keep the currently stored value that is in the stencil buffer. If both the stencil test and the depth test succeed however, we want to replace the stored stencil value with the reference value set via <fun><function id='68'>glStencilFunc</function></fun> which we later set to <code>1</code>.
 </p>
 
 <p>
   We clear the stencil buffer to <code>0</code>s at the start of the frame and for the containers we update the stencil buffer to <code>1</code> for each fragment drawn:
 </p>
 
 <pre><code>
 <function id='69'>glStencilOp</function>(GL_KEEP, GL_KEEP, GL_REPLACE);  
 <function id='68'>glStencilFunc</function>(GL_ALWAYS, 1, 0xFF); // all fragments should pass the stencil test
 <function id='67'>glStencilMask</function>(0xFF); // enable writing to the stencil buffer
 normalShader.use();
 DrawTwoContainers();
 </code></pre>
 
 <p>
   By using <var>GL_REPLACE</var> as the stencil op function we make sure that each of the containers' fragments update the stencil buffer with a stencil value of <code>1</code>. Because the fragments always pass the stencil test, the stencil buffer is updated with the reference value wherever we've drawn them.
 </p>
 
 <p>
   Now that the stencil buffer is updated with <code>1</code>s where the containers were drawn we're going to draw the upscaled containers, but this time with the appropriate test function and disabling writes to the stencil buffer:
 </p>
 
 <pre><code>
 <function id='68'>glStencilFunc</function>(GL_NOTEQUAL, 1, 0xFF);
 <function id='67'>glStencilMask</function>(0x00); // disable writing to the stencil buffer
 glDisable(GL_DEPTH_TEST);
 shaderSingleColor.use(); 
 DrawTwoScaledUpContainers();
 </code></pre>
 
 <p>
   We set the stencil function to <var>GL_NOTEQUAL</var> to make sure that we're only drawing parts of the containers that are not equal to <code>1</code>. This way we only draw the part of the containers that are outside the previously drawn containers. Note that we also disable depth testing so the scaled up containers (e.g. the borders) do not get overwritten by the floor. Make sure to enable the depth buffer again once you're done.
 </p>
 
 <p>
   The total object outlining routine for our scene looks something like this:
 </p>
 
 <pre><code>
 <function id='60'>glEnable</function>(GL_DEPTH_TEST);
 <function id='69'>glStencilOp</function>(GL_KEEP, GL_KEEP, GL_REPLACE);  
   
 <function id='10'>glClear</function>(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); 
 
 <function id='67'>glStencilMask</function>(0x00); // make sure we don't update the stencil buffer while drawing the floor
 normalShader.use();
 DrawFloor()  
   
 <function id='68'>glStencilFunc</function>(GL_ALWAYS, 1, 0xFF); 
 <function id='67'>glStencilMask</function>(0xFF); 
 DrawTwoContainers();
   
 <function id='68'>glStencilFunc</function>(GL_NOTEQUAL, 1, 0xFF);
 <function id='67'>glStencilMask</function>(0x00); 
 glDisable(GL_DEPTH_TEST);
 shaderSingleColor.use(); 
 DrawTwoScaledUpContainers();
 <function id='67'>glStencilMask</function>(0xFF);
 <function id='68'>glStencilFunc</function>(GL_ALWAYS, 1, 0xFF);   
 <function id='60'>glEnable</function>(GL_DEPTH_TEST);  
 </code></pre>
 
 <p>
   As long as you understand the general idea behind stencil testing this shouldn't be too hard to understand. Otherwise try to carefully read the previous sections again and try to completely understand what each of the functions does now that you've seen an example of it can be used.
 </p>
 
 <p>
   The result of the outlining algorithm then looks like this:
 </p>
 
 
 <img src="/img/advanced/stencil_scene_outlined.png" class="clean" alt="3D scene with object outlining using a stencil buffer"/>
 
 <p>
   Check the source code <a href="/code_viewer_gh.php?code=src/4.advanced_opengl/2.stencil_testing/stencil_testing.cpp" target="_blank">here</a> to see the complete code of the object outlining algorithm.
 </p>
 
 <note>
   You can see that the borders overlap between both containers which is usually the effect that we want (think of strategy games where we want to select 10 units; merging borders is generally preferred). If you want a complete border per object you'd have to clear the stencil buffer per object and get a little creative with the depth buffer.
 </note>
 
 <p>
   The object outlining algorithm you've seen is commonly used in games to visualize selected objects (think of strategy games) and an algorithm like this can easily be implemented within a model class. You could set a boolean flag within the model class to draw either with borders or without. If you want to be creative you could even give the borders a more natural look with the help of post-processing filters like Gaussian Blur.
 </p>
 
 <p>
   Stencil testing has many more purposes (beside outlining objects) like drawing textures inside a rear-view mirror so it neatly fits into the mirror shape, or rendering real-time shadows with a stencil buffer technique called <def>shadow volumes</def>. Stencil buffers give us with yet another nice tool in our already extensive OpenGL toolkit.
 </p>       
 
     </div>