Introduction to SDD

for programmers who need to implement real-time data streams

The big picture

Engineers working in the field of Intelligent Transportation Systems (ITS) have embarked upon massive data-gathering projects, from installing thousands of traffic sensors in urban freeways, to giving buses and fleet vehicles the ability to transmit their location at all times. This kind of data is characterized by the following attributes:

Real-time updates provided at intervals ranging from 1 second to 10 minutes
Large volume of real-time data (thousands of sensors)
Description of the system (i.e. sensor lists or bus schedules) can also change over time.

The UW ITS Research Group is meeting the challenge of building applications for these systems. Several years of experience with building applications that receive and process real-time data have resulted in a free, open source software library that standardizes the way data streams are distributed. This library is called SDD, for the Self-Describing Data protocol that dictates the format of the data stream.

Is SDD right for me?

If your organization gathers real-time data, making it available as a Self-Describing Data stream offers you the following benefits:

Sharing the data with others. No matter what your data is, the standard SDD receiver program can be used on another computer to connect to your data stream and monitor it in real time. You do not need to distribute any customized client programs or data dictionaries.
Data fusion. You may have several different sources of data that need to be combined into a single stream. Each data source can provide its own SDD stream, and an SDD "operator" program can combine the multiple streams into a single high-level product. To give a concrete example, traffic management centers install systems that use loop signatures, lasers, video image processing, and vehicle probes, all with the final intent of estimating travel times. SDD allows each of these systems to provide ever-changing data to a central program that produces a single travel-time data stream, hiding the details of the divergent systems that fed into it.
Standardized applications. Encapsulated within each SDD stream is the Schema, a description of the related tables of data that the stream contains. If two different systems can provide an SDD stream with the same Schema, then an application that makes use of one stream can also be used with the other. For instance, an SDD stream for Seattle transit vehicle locations exists, and an application was developed whose input was the vehicle location stream, and whose output was a different stream of vehicle arrival predictions. If another city were to provide a similar vehicle location stream, the prediction application would work there too.

What's in the SDD software library?

The SDD software distribution provides a java class library which programmers can use to tap into existing SDD streams (using the SddReceiver class), or create new SDD streams (with the SddTransmitter class). Creating a new SDD stream is a more difficult task because it requires some level of integration with the existing system that is producing the real-time data. Either of the two following strategies can be used to accomplish this:

If you're comfortable with java programming, you can devise your own means of acquiring the real-time data from the source, and then use the SddTransmitter class to turn it into an SDD stream.
If you prefer C/C++ programming to java, then your existing system can be extended to provide the real-time data it collects to a standard program in the library called SddTranslator, which translates it to an SDD stream for you. This way, the interface between your system and SDD can be implemented in the environment familiar to you. We provide C source code you can use to transfer your data over a socket to SddTranslator.

More detailed information on accomplishing these tasks is available in the SDD Software Manual. If you're ready to get started, you can download SDD now. For more background information and reading, consult the ITS Backbone project pages.

ITS UW Home