The Pixel Rosetta Stone: Packings and Colorspaces

Assembled by Chris Pirazzi. Information provided by folks throughout the company, including Brian Beach, David Blythe, Nelson Bolyard, Terry Crane, Patricia Creek, Carolyn Curtis, Hilton Goldstein, Bent Hagemark, Angela Lai, Michael Minakami, Paul Ning, Michael Portuesi, Mark Segal, Paul Spencer, Mike Travis, Ashok Yerneni, and several mailing lists.

This document has two goals:

This document makes no attempt at generality. As new packings and colorspaces are born, or otherwise become useful to video developers, we'll change our descriptive mechanism to fit them. The sections are:

Introduction

Tower of Babel?

Your first question is probably "Which of these zillions of packings are useful?" Each of these packings has some reason to exist, but the three most commonly used packings are:

Packing vs. Colorspace

In our model, all images have four components, numbered 1 through 4.

A packing

A colorspace

In most SGI libraries, a single token encodes both colorspace and packing. Often, details of the colorspace are implicit. For the VL of divo and beyond, the two parameters are specified separately and explicitly, using VL_PACKING and VL_COLORSPACE. See How to Tell Which Colorspace You Have to understand which libraries use which colorspace when.

If you were wondering, 12-13 bit signed components are useful for storing rgba components which have been converted from the vyua color set. For example, this happens when you transfer in-memory RGB pixels to or from an external Rec. 601 digital video signal. Because of the way the color sets are defined, there exist vyua values which map outside of the normal rgba color cube. The rgba values must either be clamped, in which case information is lost, or the rgba component representation must be allowed to exceed the normal values. This is fully described in the most excellent Appendix "Color-Space Conversions" of the DIVO (Digital Video Option) XIO Board Owner's Guide.

Packings

How to Read the Packing Diagrams

In the diagrams below,

Pixel Packings, Sorted by Width

8 Bit Pixel Packings

Pixel 1
Byte 1
yyyyyyyy


Pixel 1
Byte 1
bbgggrrr


Pixel 1
Byte 1
rrrbbggg


Pixel 1
Byte 1
rrrgggbb


12 Bit Pixel Packings

Pixels 1-4
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6
uuuuuuuuyyyyyyyyvvvvvvvvyyyyyyyyyyyyyyyyyyyyyyyy
center top left center top right bottom left bottom right


16 Bit Pixel Packings

Pixel 1
Byte 1Byte 2
yyyyyyyyaaaaaaaa


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4
vvvvvvvvyyyyyyyyuuuuuuuuyyyyyyyy
left right


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4
uuuuuuuuyyyyyyyyvvvvvvvvyyyyyyyy
left right


Pixel 1
Byte 1Byte 2
arrrrrgggggbbbbb


Pixel 1
Byte 1Byte 2
rrrrrgggggbbbbba


Pixel 1
Byte 1Byte 2
rrrrrggggggbbbbb


20 Bit Pixel Packings

Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5
uuuuuuuuuuyyyyyyyyyyvvvvvvvvvvyyyyyyyyyy
left right


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5
vvvvvvvvvvyyyyyyyyyyuuuuuuuuuuyyyyyyyyyy
left right


24 Bit Pixel Packings

Pixel 1
Byte 1Byte 2Byte 3
rrrrrrrrggggggggbbbbbbbb
vvvvvvvvyyyyyyyyuuuuuuuu


Pixel 1
Byte 1Byte 2Byte 3
bbbbbbbbggggggggrrrrrrrr
uuuuuuuuyyyyyyyyvvvvvvvv


Pixel 1
Byte 1Byte 2Byte 3
rrrrrrggggggbbbbbbaaaaaa
vvvvvvyyyyyyuuuuuuaaaaaa


OpenGL-Like 32-bit Packings:

Pixel 1
Byte 1Byte 2Byte 3Byte 4
rrrrrrrrggggggggbbbbbbbbaaaaaaaa
vvvvvvvvyyyyyyyyuuuuuuuuaaaaaaaa


Pixel 1
Byte 1Byte 2Byte 3Byte 4
rrrrrrrrggggggggbbbbbbbbxxxxxxxx


IRIS GL-Like 32-bit Packings:

Pixel 1
Byte 1Byte 2Byte 3Byte 4
aaaaaaaabbbbbbbbggggggggrrrrrrrr
aaaaaaaauuuuuuuuyyyyyyyyvvvvvvvv


Pixel 1
Byte 1Byte 2Byte 3Byte 4
xxxxxxxxbbbbbbbbggggggggrrrrrrrr
xxxxxxxxuuuuuuuuyyyyyyyyvvvvvvvv


Unusual 32-bit Packings:

Pixel 1
Byte 1Byte 2Byte 3Byte 4
aaaaaaaarrrrrrrrggggggggbbbbbbbb


Pixel 1
Byte 1Byte 2Byte 3Byte 4
xxxxxxxxrrrrrrrrggggggggbbbbbbbb
xxxxxxxxvvvvvvvvyyyyyyyyuuuuuuuu


Pixel 1
Byte 1Byte 2Byte 3Byte 4
uuuuuuuuyyyyyyyyvvvvvvvvaaaaaaaa


4:4:4:4 10_10_10_2 32-bit Packings:

Pixel 1
Byte 1Byte 2Byte 3Byte 4
rrrrrrrrrrggggggggggbbbbbbbbbbaa
vvvvvvvvvvyyyyyyyyyyuuuuuuuuuuaa


Pixel 1
Byte 1Byte 2Byte 3Byte 4
aabbbbbbbbbbggggggggggrrrrrrrrrr
aauuuuuuuuuuyyyyyyyyyyvvvvvvvvvv


4:2:2:4 10_10_10_2 32-bit Packings:

Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
vvvvvvvvvvyyyyyyyyyyaaaaaaaaaa00uuuuuuuuuuyyyyyyyyyyaaaaaaaaaa00
left 00left right 00


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
uuuuuuuuuuyyyyyyyyyyaaaaaaaaaa00vvvvvvvvvvyyyyyyyyyyaaaaaaaaaa00
left 00left right 00


4:2:2 10_in_16 32-bit Packings:

Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
vvvvvvvvvvppppppyyyyyyyyyyppppppuuuuuuuuuuppppppyyyyyyyyyypppppp
left ppppppleft ppppppleft ppppppright pppppp


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
000000vvvvvvvvvv000000yyyyyyyyyy000000uuuuuuuuuu000000yyyyyyyyyy
000000left 000000left 000000left 000000right


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
uuuuuuuuuuppppppyyyyyyyyyyppppppvvvvvvvvvvppppppyyyyyyyyyypppppp
left ppppppleft ppppppleft ppppppright pppppp


Pixels 1-2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
000000uuuuuuuuuu000000yyyyyyyyyy000000vvvvvvvvvv000000yyyyyyyyyy
000000left 000000left 000000left 000000right


36 Bit Pixel Packings

Pixel 1Pixel 2
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8Byte 9
rrrrrrrrrrrrggggggggggggbbbbbbbbbbbbrrrrrrrrrrrrggggggggggggbbbbbbbbbbbb
vvvvvvvvvvvvyyyyyyyyyyyyuuuuuuuuuuuuvvvvvvvvvvvvyyyyyyyyyyyyuuuuuuuuuuuu


48 Bit Pixel Packings

Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6
rrrrrrrrrrppppppggggggggggppppppbbbbbbbbbbpppppp
vvvvvvvvvvppppppyyyyyyyyyyppppppuuuuuuuuuupppppp


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6
rrrrrrrrrrrrggggggggggggbbbbbbbbbbbbaaaaaaaaaaaa
vvvvvvvvvvvvyyyyyyyyyyyyuuuuuuuuuuuuaaaaaaaaaaaa


64 Bit Pixel Packings

Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
rrrrrrrrrrppppppggggggggggppppppbbbbbbbbbbppppppaaaaaaaaaapppppp
vvvvvvvvvvppppppyyyyyyyyyyppppppuuuuuuuuuuppppppaaaaaaaaaapppppp


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
000000rrrrrrrrrr000000gggggggggg000000bbbbbbbbbb000000aaaaaaaaaa
000000vvvvvvvvvv000000yyyyyyyyyy000000uuuuuuuuuu000000aaaaaaaaaa


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
aaaaaaaaaappppppbbbbbbbbbbppppppggggggggggpppppprrrrrrrrrrpppppp
aaaaaaaaaappppppuuuuuuuuuuppppppyyyyyyyyyyppppppvvvvvvvvvvpppppp


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
xxxxxxxxxxppppppbbbbbbbbbbppppppggggggggggpppppprrrrrrrrrrpppppp


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
rrrrrrrrrrrrppppggggggggggggppppbbbbbbbbbbbbppppaaaaaaaaaaaapppp


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
ssssrrrrrrrrrrrrssssggggggggggggssssbbbbbbbbbbbbssssaaaaaaaaaaaa


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
rrrrrrrrrrrrrpppgggggggggggggpppbbbbbbbbbbbbbpppaaaaaaaaaaaaappp


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
sssrrrrrrrrrrrrrsssgggggggggggggsssbbbbbbbbbbbbbsssaaaaaaaaaaaaa


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
rrrrrrrrrrrrrrrrggggggggggggggggbbbbbbbbbbbbbbbbaaaaaaaaaaaaaaaa


Pixel 1
Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
aaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbggggggggggggggggrrrrrrrrrrrrrrrr


Pixel Packings, Sorted by Library

OpenGL Pixel Packings

OpenGL can represent many more packings than we can list. The OpenGL packings which overlap with the other library packings, where: are:

IRIS GL Pixel Packings

IRIS GL can represent many more packings than we can list. The IRIS GL packings which overlap with the other library packings, where: are

Digital Media Library (DM) Pixel Packings

This list is complete as of IRIX 6.3:

Compression Library (CL) Pixel Packings

This list is complete as of IRIX 6.3:

Traditional VL Pixel Packings

This list is complete as of IRIX 6.3. Here are the ones we document: Here are the remaining tokens:

Memory-Only (divo and Beyond) VL Pixel Packings

Where Do the Memory-Only VL_PACKING Names Come From?

The naming scheme for the new memory-only VL_PACKING tokens (found in the VL of divo and beyond) works like this:

Sampling Pattern Definitions

4:4:4 and 4:4:4:4 Sampling

Some of the diagrams indicate 4:4:4 or 4:4:4:4 sampling. This video industry terminology simply means that each of your 3 or 4 components is sampled at every pixel. Here's an example with Y, Cr, and Cb (4:4:4 could also refer to R, G, B):

There is no particularly good reason why the terminology is 4:4:4 rather than 8:8:8 or any other N:N:N. N needed to be at least 4 in order to describe 4:1:1.

4:2:2 and 4:2:2:4 Sampling

Some of the diagrams indicate 4:2:2 sampling. These packings make sense only in the "vyua" colorspaces. 4:2:2 sampling is described by ITU-R BT.601-4 (Rec. 601). It means that for every two pixels, we get two luma samples (two Y's) but only one chroma sample (one sample of Cr and Cb, which together determine the chroma), like this:

The chroma samples belong at the same instant in space as the left Y sample (the chrominance samples and the left Y are "co-sited"). Our diagrams for 4:2:2 packings show the spatial location of each Y, Cr, or Cb component as "left" or "right." The first pixel of each video line is a "left" pixel.

Converting 4:4:4 video to 4:2:2 video is like converting 44.1kHz audio into 22.05kHz audio: just dropping every other Cr,Cb sample will yield extremely poor results. Real video devices which need to convert between 4:4:4 and 4:2:2 use carefully designed filters. The characteristics of the required filter are specified in Rec. 601.

4:2:2 sampled packings that also include alpha are called 4:2:2:4. There is one alpha value per pixel, like the Y value.

4:2:0 Sampling

Like 4:2:2 sampling, 4:2:0 sampling only makes sense for "vyua" packings. In this case, there is one chroma sample (one pair of Cr and Cb) for every four luminance samples (Y).

The MPEG I and H.261 video compression standards use this 4:2:0 sampling pattern:

The chroma samples occur at the spatial center of four luminance samples. The concept of "pixel" becomes difficult to define!

The 4:2:0 tokens DM_IMAGE_PACKING_CbYCrYYY. and CL_FORMAT_YCbCr422DC imply the above sampling pattern. CL_FORMAT_YCbCr422DC (for "Duplicate Chroma") represents 4:2:0 data redundantly as 4:2:2 data. See the document "CL_FORMAT_YCbCr422DC and DM_IMAGE_PACKING_CbYCrYYY" for an example. These are the tokens you use with SGI's software MPEG I codec. Currently (10/7/97) they are the only 4:2:0 tokens in any SGI library, because MPEG I and H.261 are currently the only shipped 4:2:0 codecs. Our 4:2:0 pixel packing diagrams show the spatial location of each Y, Cr, or Cb component as "top left," "top right," "bottom left," "bottom right," and "center," according to their position in the pattern above. The top left luminance sample of the image is a "top left" sample.

In addition to supporting 4:4:4 and Rec.601 4:2:2, the MPEG II video compression standard supports this 4:2:0 sampling pattern:

It differs from MPEG I/H.261 only in the position of the chroma samples. Why the MPEG committee changed it is beyond us. Perhaps they wanted to make conversion from Rec.601 4:2:2 to MPEG II 4:2:0 less computationally expensive.

If we had any MPEG II 4:2:0 packing diagrams above, we would label each component as "top left," "top right," "bottom left," "bottom right," and "middle left." The top left luminance sample of the image is a "top left" sample.

Unlike MPEG I or H.261, MPEG II understands the concept of fields (see Fields: Why Video Is Crucially Different from Graphics). Each row in the sampling pattern above is one picture line of an interleaved frame. The MPEG II spec defines how the rows of Cr,Cb samples coincide temporally with the rows of Y samples.

Not to be outdone, the 625-line DVC compression standard (not DVCPRO) specifies this 4:2:0 sampling pattern:

DVC is also field-aware, and again this diagram is meant to show an interleaved frame. In the 625-line DVC sampling pattern, Cr and Cb samples are no longer co-sited. Instead, we get a whole line of Cr samples from each field, then we get a whole line of Cb samples from each field. The top two lines of the standard DVC frame contain Cr samples.

4:1:1 Sampling

Another subsampling method with similar "compression" as 4:2:0 is 4:1:1. The 525-line DVC compression standard, and both the 525- and 625-line variants of the DVCPRO compression standard, use this pattern:

Four horizontally adjacent Y samples on the same line are paired with one Cr,Cb sample. The Cr,Cb sample and the "leftmost" Y sample are co-sited. The leftmost Y sample of each line is a "leftmost" sample. This method generally looks worse than 4:2:0 at the same cost, and it is not currently used by any SGI libraries. It seems like kind of a bummer that 525-line DVC uses it.

How to Tell Which Colorspace You Have

In most SGI libraries, a single token encodes both colorspace and packing. Often, details of the colorspace are implicit. For the VL of divo and beyond, the two parameters are specified separately and explicitly, using VL_PACKING and VL_COLORSPACE. This section tells you which libraries use which colorspace when.

For OpenGL, IRIS GL, DM, and CL:

For VL, using the traditional VL_PACKING tokens from IRIX 6.2:

The VL that comes with the divo product and beyond makes all of the parameters (packing, set of colors, range of components) explicit in a clean way:

When Dealing with JPEG Data (cosmo1, cosmo2, mvp+ICE, JFIF files, MV/QT files):