Introduction to UST and UST/MSC

This document describes the UST support in the VL, AL, GL, CL, MD, tserialio, DM, and other libraries (see What Are the SGI Video-Related Libraries?). 

What Is It?

• record or play audio, video, and/or MIDI in sync.
• perform field-accurate laydowns to and captures from a VTR.
• slave audio, video and/or MIDI playback to incoming timecode.
• implement highly accurate punch-in and punch-out capability via a MIDI or serial device.
• measure the total input-jack-to-output-jack delay of your real-time signal processing program.

 For a taste of why synchronization is hard and how the UST support solves the problem, see Why Use UST?.

Accuracy vs. Latency

• It will not affect the upper bound, lower bound, or average time it takes for your program to react to an incoming signal by producing a corresponding output signal.
• It will not make your program run any more or less often.
• It will not affect the amount of "advance warning" an output device needs in order to output your data on time.

The UST Clock

• has a 64-bit signed value (use the C typedef "stamp_t" from sys/time.h).
• has units of nanoseconds (ie, has 1ns numerical precision).
• is initialized to 0 at system startup.
• is never adjusted or reset while the system is running, unlike gettimeofday(3).
• is monotonically increasing, meaning that in any pair of UST measurements by a device or your software, the UST clock's value never decreases.
• has a resolution that is 1us or better (ie, takes on a new value at least once every 1us).
• has an accuracy or error that is ±100 parts per million or better, meaning that its actual rate over the long term differs at most that much from its nominal rate (as measured by your favorite cesium clock).
• has a wrap period of 292 years, so practical code does not have to worry about USTs wrapping.

How to Use UST

• When a piece of data such as a video field, an audio frame, or a MIDI event arrives at an input jack of the machine, an input subsystem will snap the value of the UST clock at that instant. The subsystem will then provide you with the data and give you a way to determine its UST. Your program doesn't have to be running at the instant the data arrives. You get the full accuracy of the input subsystem's USTs, which can have as fine a granularity as ±1.5us for some of SGI's audio subsystems, without needing to schedule your process to that granularity.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• Similarly, when you want to output a piece of data, the subsystem gives you a way to control the UST at which each piece of your data hits the output jack of the machine. Again, since you have provided the data and its UST ahead of time, your process does not have to be running at the crucial moment of output. Some data types (audio and video for example) inherently only let you output data at certain instants. The subsystem provides you with the possible choices in terms of UST.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 You can perform all of the timing and synchronization tasks listed above by simply comparing USTs you get from or give to different subsystems. For example:

• To record audio and video in sync, you use the UST support in the AL to pair incoming audio data with its USTs, and the UST support in the VL to pair incoming video data with its USTs. You can then compare the USTs from the AL and VL to determine how each frame of audio data coincided with each field/frame of video data at the audio and video jacks of the machine. This tells you how to create a movie file where the two streams are synchronized.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• To play audio and video data from a movie file in sync, you look at how the movie file wants each audio frame and video field/frame to line up in time, and schedule these items of data to go out at USTs which line up in the same way.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• To measure the input-jack-to-output-jack delay (or latency) of your real-time signal processing program, request the UST of a piece of input data, determine the UST at which your program outputs the processed version of that data, and subtract! If you wanted to increase the delay of your processing program to a precise value, you could add more and more buffering in your program and repeat the UST measurement until you reached the desired delay. Don't forget that the UST support just gives you a way to measure your latency; it won't inherently increase or reduce your program's latency.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

UST Timestamping in the MIDI Library

typedef struct __mdevent {
        char msg[4];
        char *sysexmsg;
        long long stamp;            /* time stamp in ns */
        int msglen;                 /* length of data, sysex only */
} MDevent;

{
  MDport port;
  /* ... open port ... */
  mdSetStampMode(port, MD_RELATIVESTAMP);
  mdSetStartPoint(port, 0, 0);
  /* ... send or receive MIDI ... */
}

 To receive data, you call mdReceive() and you get back an MDevent with the event and UST stamp filled in. To send data, you provide an MDevent to mdSend(), and the MIDI subsystem will output the specified MIDI message at the specified UST.

UST in the Timestamped Serial I/O Library

UST Timestamping for Video Library Input

 When you receive a field or frame in a DMbuffer using the O2 or cross-platform VL buffering API (see What Are the SGI Video-Related Libraries? for more info), you can pass the DMbuffer into dmBufferGetUSTMSCpair(). This returns a USTMSCpair structure. USTMSCpair.ust contains the UST of the field or frame. USTMSCpair.msc contains a coincident snap of a counter that increments once per buffer time (field or frame, depending on the time spacing of the data in your buffers). You can tell if you dropped any data by seeing if USTMSCpair.msc has incremented by more than 1. This MSC counter has some useful properties which we describe below along with UST/MSC.

 The VL also supports UST/MSC, another API for determining the UST of data which works for input and output. We describe UST/MSC below.

 We will explain how to distinguish dominant and non-dominant fields along with UST/MSC below.
 
 

UST Timestamping in Other Libraries

UST and gettimeofday()

 This function provides one possible way to relate the UST clocks of two different machines. See Synchronization Across Machines for more information.
 
 

UST for Sampled Data Types: UST/MSC

 As a result of this, you can think of your signal as having discrete "slots" which each hold one unit of data. Here's an example with an audio and video signal:
 
 

The UST of a slot is the UST of the synchronization point of the data in that slot (see How to Use UST above). Each slot in a given signal is numbered with an MSC ("media stream count" or "media sequence count").

 If your program inputs a sampled signal, your job is to read the data out of each slot. If your program outputs a sampled signal, your job is to place the right data into each slot. In both cases, you need a way to figure out the UST of any given slot.

 The UST/MSC support in the AL and VL consists of two operations which let you do this:
 
 

• alGetFrameNumber(), vlGetFrontierMSC(): This operation gives you a way to identify the MSC of the next piece of data you are about to read from an AL port or VL path, or the next piece of data you are about to write to an AL port or VL path. This MSC is called the "frontier MSC." We'll define the read and write operation for each library below.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 Until we say otherwise, this document will assume that your program reads or writes every slot. On input, this means that you read data out of your buffer frequently enough that the device always has room to deposit new data, so no data is ever dropped. On output, this means that you write data to your buffer frequently enough that the device will never starve for data, so the output will never glitch (e.g., audio click, video flash or pause). As we'll see later, the frontier MSC is also useful for detecting these error conditions.
 
 

• alGetFrameTime(), vlGetUSTMSCPair(): This operation returns a "UST/MSC pair" for your AL port or VL path. This pair has an MSC M and a UST U. M identifies a particular slot in the signal, and U is the UST of that slot. M could be any recent MSC. It's not necessarily the MSC that's crossing the jack "now," or the MSC that you just input or output. It's not necessarily the frontier MSC. It could even be an MSC that the machine has not yet input or has already output. The important part is that the library has paired together M and U so you can read them atomically.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

{
  double ust_per_msc = get_ust_per_msc();
  USTMSCpair pair = get_ust_msc_pair();
  stamp_t frontier_msc = get_frontier_msc();
  
  /* step 1: figure out which MSC you want a UST for */

  stamp_t desired_msc = frontier_msc;

  /* step 2: compute the UST of that MSC */

  stamp_t desired_ust = pair.ust + ust_per_msc*(desired_msc - pair.msc);
}

Determining the ust_per_msc figure can sometimes be tricky, because of various idiosyncrasies of audio hardware, video hardware, and video itself. Each library provides a simple way to get a nominal ust_per_msc figure. For any UST/MSC device, you can also measure ust_per_msc empirically instead of computing the nominal ust_per_msc figure. This often has accuracy advantages and works for any sync source. We will describe this more fully in Clocking and Clock Drift: Relating Multiple MSC or UST Clocks below.

 You don't have to compute ust_per_msc or get a UST/MSC pair every time you read or write data. Typically, applications compute ust_per_msc only once, and get a new UST/MSC pair every second or so.

 MSC values you get from a particular AL port or VL path are specific to that port/path only. For example, if you input the same video signal using two different VL paths, you must not assume that MSC 123 on one path refers to the same video field/frame as MSC 123 on the other path. When you need to compare the time of data on two different paths or ports, convert to UST.

 After giving the specifics of UST/MSC for each library, we will provide some pictorial examples of the above pseudocode.
 
 

UST/MSC in the Audio Library

 To get a nominal ust_per_msc figure, do this:
 
 

{
  ALpv pv;
  double samprate;
  double ust_per_msc;
  
  pv.param = AL_RATE;
  if (alGetParams(res, &pv, 1) < 0 || pv.value.ll <= 0)
    {
      /* cannot determine nominal sample-rate */
    }
  else
    {
      samprate = alFixedToDouble(pv.value.ll);
      ust_per_msc = 1.E9 / samprate;
    }
}

 The alGetParams() call will succeed but return a negative sampling rate if the nominal sample-rate cannot be determined. This only happens if the AL device's clock generator's master clock is an AES or ADAT digital signal which does not contain rate information (e.g. an AES source which has not set the rate bits in the subcode).

 You can measure ust_per_msc instead, which works in every case and gives you a more accurate figure. This will be described in Clocking and Clock Drift: Relating Multiple MSC or UST Clocks below.
 
 

UST/MSC in the Video Library

Dominant/Non-Dominant and F1/F2

 For VL_CAPTURE_FIELDS paths, the device may skip an even or odd number of fields during an input overflow, output underflow, or input signal problem. Therefore, you need a way to determine what field type you are currently enqueuing or dequeuing. This is currently only possible on vino, ev1, ev3, and mvp, where you can use any of DMediaInfo->sequence, USTMSCpair.msc, or an MSC from UST/MSC to determine the field type:
 
 

• (msc%2)==0 indicates an F1 field
• (msc%2)==1 indicates an F2 field

Jack-to-Jack Paths

Sample UST/MSC Calculation: Audio Input

This diagram shows your program sitting atop the audio subsystem (the AL and the audio hardware). We show an incoming audio signal as a stack of slots, each of which holds one frame of audio. Some of those frames have not entered the computer yet, and some are sitting in the computer waiting for you to read them. As in the diagram above, each slot has an MSC and a UST. The UST is the UST at which the data in that slot will hit, or did hit, the audio input jack of the machine.

 The next thing your program does is to read some data with alReadFrames:
 
 

The AL examples in this document will use reads and writes of 2-5 frames for typographical convenience. Typical programs transfer 10ms or more of audio frames with each alReadFrames() or alWriteFrames(). UST/MSC works the same either way.

 Now you have some data. The next thing you need to do is figure out the MSC of each of your frames. You do that with one call to alGetFrameNumber():
 
 

alGetFrameNumber() returns a frontier MSC of 80. This is the MSC of the next frame you're about to read from the port. Since audio frames come at regular intervals, and since we're assuming that we read the data out of every slot of the signal (ie, we never drop any audio frames on input), we then know that the MSCs of the 4 frames we just read are 76, 77, 78, and 79.

 Now that we know the MSC of each of our frames, we can figure out the UST of each of the frames using alGetFrameTime():

alGetFrameTime() returns a pair of numbers to us: MSC 98 and UST u. This tells us that MSC 98 hits the jack at UST u. We want to know when MSC 76, 77, 78, and 79 hit the jack. We find the UST of MSC 76 by taking u and adjusting it by (76-98) audio frame periods. The audio frame period T is (1/audio sampling rate) in nanoseconds. T is the same as ust_per_msc in the pseudocode above. You can use this simple extrapolation trick to find the UST of any recent MSC, and we do it for MSC 77, 78, and 79 above.

 It doesn't really matter which MSC alGetFrameTime() tells us about. It could return a UST/MSC pair for MSC 100, MSC 96, MSC 80, MSC 76, or any other recent MSC, and your code will still give you a UST typically within a few audio frames of accuracy (on some devices, the UST will be better than audio frame accurate). The reason you need a recent MSC (an MSC within a few seconds of the MSC currently hitting the jack) has to do with a secondary effect called clock drift, which will be explained later.
 
 

Sample UST/MSC Calculation: Audio Output

 Remember our assumption that your program fills every slot of the output signal with data. This means that every time the output device needs a new audio frame to output, you will have made one available with alWriteFrames(). If the output device begins to starve for data, we say that the buffer is "underflowing." When you first open an AL output port, there are no frames in the buffer, so the buffer is underflowing. The first thing you need to do, before attempting to use UST/MSC, is to get some audio frames in there! You need to enqueue at least enough silence to handle the worst-case amount of time between enqueuing the silence and taking the first UST measurement.

 Let's assume that you have opened an AL output port and you have written enough data to it to keep the port happy for a while. You want to determine which audio data you should write to the port next, so you want to know the UST of the next audio frame you write to the port:
 
 

Note that MSCs now decrease (data gets older) as you move down the diagram, since the data flows in the opposite direction. The slot shown in dotted lines is the next audio frame you will write to the AL. It's not an audio frame that is currently in the AL buffer.

 As with the input case, one call to alGetFrameNumber() tells you the MSC of each of your audio frames:
 
 

alGetFrameNumber() returns a frontier MSC of 65. This is the MSC of the next frame you're about to write to the port. So the MSCs of your four frames are 65, 66, 67, and 68.

 Be careful: In the input case, the audio frame with the frontier MSC is in the AL buffer. In the output case, the audio frame with the frontier MSC is in your buffer.

 Continuing to follow the pattern, we get a UST/MSC pair and use it to compute the UST of each frame we are about to write:

The explanation here is identical to that for input.

 Now we know the UST of each audio frame, so we can choose some data to output and proceed with the alWriteFrames().

UST/MSC: Using the Frontier MSC to Detect Errors

• On input, your AL or VL buffer never overflows. This means that you read data out of your buffer frequently enough that the device always has room to deposit new data, so no data is ever dropped.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• On output, your AL or VL buffer never underflows. This means that you write data to your buffer frequently enough that the device will never starve for data, so the output will never glitch (audio click, video flash or pause).

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 You can also use the frontier MSC to detect overflow and underflow conditions, and determine their precise length.

 Here's an AL example of how to detect overflow in an input buffer:

{     
  stamp_t newmsc, oldmsc=-1;
  
  /* this is your main data-reading loop, not a special one */
  while (1)
    {
      alReadFrames(port, buf, nframes);
     
      alGetFrameNumber(port, &newmsc);
      if (oldmsc > 0)
        {
          stamp_t M = (newmsc-oldmsc) - nframes;
          if (M != 0)
            printf("we overflowed for %lld MSCs!\n", M);
        }
      oldmsc = newmsc;
    }
}

 Use this basically identical code to detect underflow in an AL output buffer:

{     
  stamp_t newmsc, oldmsc=-1;
  
  /* this is your main data-writing loop, not a special one */
  while (1)
    {
      alWriteFrames(port, buf, nframes);
     
      alGetFrameNumber(port, &newmsc);
      if (oldmsc > 0)
        {
          stamp_t M = (newmsc-oldmsc) - nframes;
          if (M != 0)
            printf("we underflowed for %lld MSCs!\n", M);
        }
      oldmsc = newmsc;
    }
}

 This technique works just as well with the VL.

 For more information on how and why this works, check out UST/MSC: Using the Frontier MSC to Detect Errors.

UST/MSC in the Graphics Libraries

UST in the CL for cosmo1 and cosmo2

typedef struct {
    unsigned size;       /* size of compressed image in bytes */
    unsigned long long ustime;
    unsigned imagecount;
    unsigned status;     /* additional status information */
} CLimageInfo;

• Video-to-compression-to-memory: when you call clGetNextImageInfo() to get the size of the next compressed JPEG field, you can read the field's UST out of CLimageInfo.ustime and you can read a field counter out of CLimageInfo.imagecount. A jump in CLimageInfo.imagecount by more than 1 indicates dropped data. CLimageInfo is like DMediaInfo for video input.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• Memory-to-decompression-to-video: again you call clGetNextImageInfo(). CLimageInfo.ustime contains the UST at which the next JPEG field you enqueue will hit the output jack of the machine. If you miss a field on output, the UST will jump by one field time. CLimageInfo.imagecount will not. CLimageInfo.imagecount contains a delayed count of the number of fields you put into the CL buffer. If you want to determine how many images you missed on output, divide the jump in CLimageInfo.ustime by the field time (being careful to round to nearest, since the UST may have small amounts of jitter).

 The CL interface for cosmo1 and cosmo2 always deals in fields. The cosmo1 board is always F1 dominant (see Definitions: F1/F2, Interleave, Field Dominance, and More). You set cosmo2's dominance using VL_MGC_DOMINANCE_FIELD on the VL_CODEC node (the default is F1 dominance). Here is how to tell which of your fields is dominant and which is non-dominant:
 
 

• For cosmo2, the first field you enqueue or dequeue from the CL buffer after creating a CL compressor or decompressor is a dominant field. Subsequent fields you enqueue or dequeue will alternate between non-dominant and dominant. Therefore, if the CL buffer fills up on input or the CL device starves for data on output, cosmo2 will skip an even number of fields at the video jack to maintain the sequence.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• The cosmo1 device may skip an even or odd number of fields. You must determine the type of each field you input or output:

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• For cosmo1 video to compression to memory, look at the low bit of CLimageInfo.ustime. Yes, ustime, not imagecount. Yes, this bit normally is nanoseconds. Yes, this is a heinous hack. A 1 indicates a dominant field (which is always F1 for cosmo1). A 0 indicates a non-dominant field (F2 for cosmo1).

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• For cosmo1 memory to decompression to video, the first field you enqueue on the CL buffer will go out as a dominant field (which is always F1 for cosmo1). From then on, monitor jumps in CLimageInfo.ustime to see what kind of field you are addressing next. The ustime low bit hack will not work.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Clocking and Clock Drift: Relating Multiple MSC or UST Clocks

 If your software makes an assumption about the rate ratio of any two clocks, then it must assure that either:

• 1. the two clocks are driven by the same oscillator, or
• 2. the two clocks are driven by different oscillators, and the drift caused by the worst-case rate error of the two clocks over the worst-case time:
• a. is acceptable for your application, or
• b. is compensated for by code in your application.

Movie Playback Example

 Say your audio system and your video system are clocked off of the same oscillator (case 1). This would be the case if your video system and your audio system are locked to a common external blackburst signal, for example. If that oscillator is running a bit slow or fast compared to some other oscillator (say the oscillator in your wristwatch), it's no big deal: your audio system will not quite be running at 44100 Hz, but your video system will have the exact same error, so the audio and video you provide to them will stay in sync. This corresponds to case 1 above.

 Now say your audio system and your video system are not clocked off of the same oscillator (case 2). This is a common situation on lower-end SGI systems, where users may not have the knowledge, desire, or equipment to slave their audio and video devices to a common oscillator. Then the ratio of the rates of your audio and video system may not actually be 44100/50. As your program runs, the sound coming out the audio jack will slowly drift ahead of the image coming out the video jack, or vice versa. This drift will continue to worsen without bound as playback proceeds.

 To take a worst-case example for typical hardware, say the audio oscillator is 50 parts per million fast (44102.205 Hz) and the video oscillator is 50 parts per million slow (49.9975 Hz). After playing for 10 minutes, the video will have drifted 60ms (3 fields) behind the audio, which is quite noticeable. What's worse, since your program continues to send 44100 audio frames for every 50 video fields, but since the devices consume 44100 audio frames for every 49.995 (roughly) video fields, an unbounded amount of video data slowly builds up between your program and the video device (in this case that data will build up in a VL buffer). Eventually you will run out of memory.

 Say your application requires field accuracy (audio and video in sync within ±10ms). If your application never deals with movie files longer than 1 minute, then your audio and video will never drift apart more than 10ms in either direction, and no more than one extra field will collect in the VL buffer, so you have no problem. This is case 2a.

 If your application can deal with movies of any length, then you have case 2b. In this case, you cannot achieve any upper bound on synchronization error or memory usage. You must measure the actual ratio of audio frames per video field, and adjust the data coming out of the movie file so that it actually has that ratio going into the devices. Depending on the quality constraints, this may require sampling rate converting the audio data, dropping or duplicating video fields, or other tricks. UST/MSC gives you a way to measure the ratio. To do this, grab two audio UST/MSC pairs spaced a second or more apart (call them (aust1, amsc1) and (aust2, amsc2)). Also grab two video UST/MSC pairs spaced a second or more apart (call them (vust1, vmsc1) and (vust2, vmsc2)). The ratio of audio frames to video fields is:
 
 

((amsc2-amsc1)/(aust2-aust1)) / ((vmsc2-vmsc1)/(vust2-vust1)).

Other Examples

• If you are recording audio and video to a movie file, the movie file constraints you to storing 44100 audio frames per 50 video fields. If your input devices are not driven from a common clock, they may not produce data at this rate ratio. Depending on your synchronization requirement and maximum record time, you may need to tweak the data.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• If your program processes an input audio or video signal by producing a corresponding output signal, and the two AL ports or VL paths are not clocked off the same oscillator, then either your output buffer will starve (if the output is faster) or your input buffer will overflow (if the input is faster). Again you may have to tweak the incoming data to match the output rate.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• Whenever you use the UST/MSC API, you use code like this:

{
  double ust_per_msc = get_ust_per_msc();
  USTMSCpair pair = get_ust_msc_pair();
  stamp_t frontier_msc = get_frontier_msc();
  
  /* step 1: figure out which MSC you want a UST for */

  stamp_t desired_msc = frontier_msc;

  /* step 2: compute the UST of that MSC */

  stamp_t desired_ust = pair.ust + ust_per_msc*(desired_msc - pair.msc);
}

ust_per_msc is a rate ratio between the UST clock and a device's MSC clock. UST and MSC clocks are not driven off of the same oscillator in any current SGI system, so their rate ratio is uncertain. desired_ust will have an error which is proportional to the error in ust_per_msc and to the distance you are extrapolating the UST/MSC pair (desired_msc - pair.msc):
 
 
• You can reduce the error in ust_per_msc by measuring the actual rate ratio instead of using the nominal ust_per_msc figures described above. To do this, grab two UST/MSC pairs spaced a second or more apart (call them (ust1, msc1) and (ust2, msc2)), and compute:

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

ust_per_msc = (ust2-ust1) / (msc2-msc1).
You may want to measure ust_per_msc periodically for particularly long transfers.
 
 
• You can reduce (desired_msc - pair.msc) by remembering that alGetFrameTime() and vlGetUSTMSCPair() return a "recent" pair, whose UST is within a few seconds of the current UST. The less you have to extrapolate a UST/MSC pair, the more accurate your UST will be.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• If you want to relate the UST clock to the gettimeofday() clock, you can use dmGetUSTCurrentTimePair() as described above. These clocks are not derived from the same oscillator. Even if you assume the gettimeofday() clock is never adjusted by system administrators or network time daemons, you still cannot assume that 1 billion USTs represents the same amount of time as one timeval.tv_sec. You probably want to take multiple, sufficiently-spaced UST/Current Time pairs to determine the actual rate ratio.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

• If you want to relate the UST clocks of two different machines, keep in mind that their rate ratio may not be 1/1. We describe techniques for inter-machine synchronization in Synchronization Across Machines.

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Where's the Oscillator?