Displaying In-Memory Video using OpenGL

By Michael Portuesi, Nelson Bolyard, and Eric Bloch. Special thanks to Robert Tray and Terry Crane for some of the OpenGL information.

Introduction

A common question asked by beginning VL programmers is, "How do I display video using OpenGL?"

It is not surprising that people ask this question. There are few code examples in existence which demonstrate the use of VL and OpenGL in the same program. To compound matters, the subject is also not mentioned in the Iris Media Libraries Programming Guide, which was written around IRIS GL-based code examples.

Setting up OpenGL

The biggest problem that one encounters when trying to set up video display through OpenGL is that the video display is slow - much slower, in fact, than that produced by the IRIS GL-based code samples.

But video display through OpenGL does not have to be slow, if you perform the proper setup work. Here's the checklist:

/*
 * The original version of this code was developed by Allen Akin
 * (akin@sgi.com).  It has been modified for this example.
 */
void setupGL()
{
    /*
     * Is there a 24-bit visual? if not, we want to dither RGB.
     */
    Boolean doDither = TRUE;
    Display * dpy    = XtDisplay(getDeviceWidget());
    XVisualInfo vinfo;
    XVisualInfo *viList;
    int nitems;

    vinfo.depth = 24;
    viList = XGetVisualInfo(dpy, VisualDepthMask, &vinfo, &nitems);
    if (viList) {
        XFree(viList);
        doDither = FALSE;
    }

    /*
     * Disable stuff that's likely to slow down glDrawPixels.
     * (Omit as much of this as possible, when you know in advance
     * that the OpenGL state will already be set correctly.)
     */
    glDisable(GL_ALPHA_TEST);
    glDisable(GL_BLEND);
    glDisable(GL_DEPTH_TEST);
    if (!doDither) {
        glDisable(GL_DITHER);
    }
    glDisable(GL_FOG);
    glDisable(GL_LIGHTING);
    glDisable(GL_LOGIC_OP);
    glDisable(GL_STENCIL_TEST);
    glDisable(GL_TEXTURE_1D);
    glDisable(GL_TEXTURE_2D);
    glPixelTransferi(GL_MAP_COLOR, GL_FALSE);
    glPixelTransferi(GL_RED_SCALE, 1);
    glPixelTransferi(GL_RED_BIAS, 0);
    glPixelTransferi(GL_GREEN_SCALE, 1);
    glPixelTransferi(GL_GREEN_BIAS, 0);
    glPixelTransferi(GL_BLUE_SCALE, 1);
    glPixelTransferi(GL_BLUE_BIAS, 0);
    glPixelTransferi(GL_ALPHA_SCALE, 1);
    glPixelTransferi(GL_ALPHA_BIAS, 0);

    /*
     * Disable extensions that could slow down glDrawPixels.
     */
    const GLubyte* extString = glGetString(GL_EXTENSIONS);

    if (extString != NULL) {
       if (strstr((char*) extString, "GL_EXT_convolution") != NULL) {
           glDisable(GL_CONVOLUTION_1D_EXT);
           glDisable(GL_CONVOLUTION_2D_EXT);
           glDisable(GL_SEPARABLE_2D_EXT);
       }
       if (strstr((char*) extString, "GL_EXT_histogram") != NULL) {
           glDisable(GL_HISTOGRAM_EXT);
           glDisable(GL_MINMAX_EXT);
       }
       if (strstr((char*) extString, "GL_EXT_texture3D") != NULL) {
           glDisable(GL_TEXTURE_3D_EXT);
       }
    }
}

These steps are sufficient to ensure that video displays quickly. But there are further subtleties you must pay attention to in order to insure reliable transfers. These fine points, such as setting up VL events, and the proper way to respond to certain VL events, is discussed in detail elsewhere in this guide.

Displaying 8-bit Video using OpenGL

The above discussion assumes that you are displaying 24-bit video (VL packing VL_PACKING_RGB_8) to the screen. Some devices, such as vino, support 8-bit video-to-memory transfers (VL packing VL_PACKING_RGB_332_P).

The following method for displaying an 8-bit video stream using OpenGL is provided by Nelson Bolyard:

The trick for efficiently displaying 8-bit vino BGR233 images is quite involved, and entirely undocumented, as far as I know. Credit for the code below goes to Terry Crane. It involves using OpenGL's built-in "pixel mapping", which is another form of color mapping, that is separate and distinct from and in addition to the X-server's color mapping. Indy and Indigo2's XL graphics have hardware acceleration for this "pixel mapping" that translates dithered BGR233 into RGBA.

To use it, you first setup OpenGL's state machine with this code:

static void
FastUByteCItoRGBAPixelMap()
{
    GLint i;
    GLfloat constantAlpha = 1.0;
    GLfloat map[256];

    glPixelTransferf(GL_ALPHA_SCALE, 0.0);
    glPixelTransferf(GL_ALPHA_BIAS,  1.0);
    glPixelStorei(GL_UNPACK_ALIGNMENT, 1);

    /* define accelerated bgr233 to RGBA pixelmaps.  */
    for(i=0; i<256; i++)
        map[i] = (i&0x7)/7.0;
    glPixelMapfv(GL_PIXEL_MAP_I_TO_R, 256, map);
    for(i=0; i<256; i++)
        map[i] = ((i&0x38)>>3)/7.0;
    glPixelMapfv(GL_PIXEL_MAP_I_TO_G, 256, map);
    for(i=0; i<256; i++)
        map[i] = ((i&0xc0)>>6)/3.0;
    glPixelMapfv(GL_PIXEL_MAP_I_TO_B, 256, map);

    glPixelMapfv(GL_PIXEL_MAP_I_TO_A, 1, &constantAlpha);

    glPixelTransferi(GL_INDEX_SHIFT, 0);
    glPixelTransferi(GL_INDEX_OFFSET, 0);
    glPixelTransferi(GL_MAP_COLOR, GL_TRUE);
    glDisable(GL_DITHER);
}

and then you invoke glDrawPixels with the "format" GL_COLOR_INDEX.

Displaying Interlaced and/or YCrCb Video on O2 using OpenGL

There are two OpenGL extensions that facilitate the display of video on O2, the interlace extension (GL_SGIX_interlace) and the YCrCb extension (GL_SGIX_ycrcb).The first changes the semantics of calls to glDrawPixels (and related pixel and texture operations) so that video fields can be drawn directly to the graphics display. The second extends the set of pixel data formats to include YCrCb 4:2:2 (and 4:4:4).

GL_SGIX_interlace is (almost) completely implemented on O2. The extension defines an OpenGL state parameter, GL_INTERLACE_SGIX, which is GL_FALSE by default. When the parameter is GL_TRUE, glDrawPixels() will draw "every other line" on the screen. Roughly speaking, if your source buffer contains 100 lines, glDrawPixels with interlace enabled will draw line 1 of your buffer at line 1 of the screen, line 2 of your buffer on line 3 of the screen, ..., and line 100 of your buffer on line 199 of the screen. This extension interacts with glRasterPos* and glPixelZoom, which are commonly used to place, scale and/or re-orient video images. Note that when the zoom is not 1.0 (or -1.0), using glRasterPos* to correctly position the second field is tricky. Fortunately, using glBitmap to move the raster x and y positions works fine.

The following code draws 2 top-to-bottom NTSC fields with specified x and y zooms.


Besides O2, both the Reality Engine and Infinite Reality support this extension. Unfortunately, Impact does not.

GL_SGIX_ycrcb is only partially implemented on O2. It is supported for output only, which should be enough if all you want to do is display video on the screen. You can use it with glDrawPixels, but not glReadPixels. There is a further restriction in that the extension is only supported for some drawables. It is supported on windows and pixel buffers (GLXPbufferSGIX) but not on pixmaps (GLXPixmap). According to Terry Crane, because it is only partially implemented, you won't actually find GL_SGIX_ycrcb in the extension string on 02. According to Terry, the best way to determine if the extension is available is to see if the current OpenGL renderer is O2's graphics chip called CRIME. Sadly, to determine if it is available on a given drawable, the only method is to attempt to use it and see if the call to glDrawPixels fails (assumedly with GL_INVALID_ENUM).

Here's some code that checks for the presence of both extensions.

    glXMakeCurrent(dpy, window, ctxt);
    str = (const char *)glGetString( GL_EXTENSIONS );
    
    has_lace =  (int)(strstr(str, "GL_SGIX_interlace"));
    has_ycrcb = (int)(strstr(str, "GL_SGIX_ycrcb"));
        
    // If the GL_SGIX_ycrcb extension string isn't there
    // we check against the renderer
    str = (const char *)glGetString( GL_RENDERER );
    if (strstr(str, "CRIME"))
        has_ycrcb = 1;

Here's something else you might want to know about the YCrCb extension and its interaction with glPixelZoom. If the either of the factors you pass to glPixelZoom results in a non-intergral zoom, then glDrawPixel-ing YCrCb data will be very slow on O2 (because the zoom is done in software).

I don't know of any other platforms that currently support this extension.