Jitsi Architecture

10.1. Designing Jitsi

The three most important constraints that we had to keep in mind when designing Jitsi (at the time called SIP Communicator) were multi-protocol support, cross-platform operation, and developer-friendliness.

From a developer's perspective, being multi-protocol comes down to having a common interface for all protocols. In other words, when a user sends a message, our graphical user interface needs to always call the same sendMessage method regardless of whether the currently selected protocol actually uses a method called sendXmppMessage or sendSipMsg .

The fact that most of our code is written in Java satisfies, to a large degree, our second constraint: cross-platform operation. Still, there are things that the Java Runtime Environment (JRE) does not support or does not do the way we'd like it to, such as capturing video from your webcam. Therefore, we need to use DirectShow on Windows, QTKit on Mac OS X, and Video for Linux 2 on Linux. Just as with protocols, the parts of the code that control video calls cannot be bothered with these details (they are complicated enough as it is).

Finally, being developer-friendly means that it should be easy for people to add new features. There are millions of people using VoIP today in thousands of different ways; various service providers and server vendors come up with different use cases and ideas about new features. We have to make sure that it is easy for them to use Jitsi the way they want. Someone who needs to add something new should have to read and understand only those parts of the project they are modifying or extending. Similarly, one person's changes should have as little impact as possible on everyone else's work.

To sum up, we needed an environment where different parts of the code are relatively independent from each other. It had to be possible to easily replace some parts depending on the operating system; have others, like protocols, run in parallel and yet act the same; and it had to be possible to completely rewrite any one of those parts and have the rest of the code work without any changes. Finally, we wanted the ability to easily switch parts on and off, as well as the ability to download plugins over the Internet to our list.

We briefly considered writing our own framework, but soon dropped the idea. We were itching to start writing VoIP and IM code as soon as possible, and spending a couple of months on a plugin framework didn't seem that exciting. Someone suggested OSGi, and it seemed to be the perfect fit.

10.2. Jitsi and the OSGi Framework

People have written entire books about OSGi, so we're not going to go over everything the framework stands for. Instead we will only explain what it gives us and the way we use it in Jitsi.

Above everything else, OSGi is about modules. Features in OSGi applications are separated into bundles. An OSGi bundle is little more than a regular JAR file like the ones used to distribute Java libraries and applications. Jitsi is a collection of such bundles. There is one responsible for connecting to Windows Live Messenger, another one that does XMPP, yet another one that handles the GUI, and so on. All these bundles run together in an environment provided, in our case, by Apache Felix, an open source OSGi implementation.

All these modules need to work together. The GUI bundle needs to send messages via the protocol bundles, which in turn need to store them via the bundles handling message history. This is what OSGi services are for: they represent the part of a bundle that is visible to everyone else. An OSGi service is most often a group of Java interfaces that allow use of a specific functionality like logging, sending messages over the network, or retrieving the list of recent calls. The classes that actually implement the functionality are known as a service implementation. Most of them carry the name of the service interface they implement, with an "Impl" suffix at the end (e.g., ConfigurationServiceImpl ). The OSGi framework allows developers to hide service implementations and make sure that they are never visible outside the bundle they are in. This way, other bundles can only use them through the service interfaces.

Most bundles also have activators. Activators are simple interfaces that define a start and a stop method. Every time Felix loads or removes a bundle in Jitsi, it calls these methods so that the bundle can prepare to run or shut down. When calling these methods Felix passes them a parameter called BundleContext. The BundleContext gives bundles a way to connect to the OSGi environment. This way they can discover whatever OSGi service they need to use, or register one themselves ().

Figure 10.1: OSGi Bundle Activation

So let's see how this actually works. Imagine a service that persistently stores and retrieves properties. In Jitsi this is what we call the ConfigurationService and it looks like this:

package net.java.sip.communicator.service.configuration;

public interface ConfigurationService
{
  public void setProperty(String propertyName, Object property);
  public Object getProperty(String propertyName);
}

A very simple implementation of the ConfigurationService looks like this:

package net.java.sip.communicator.impl.configuration;

import java.util.*;
import net.java.sip.communicator.service.configuration.*;

public class ConfigurationServiceImpl implements ConfigurationService
{
  private final Properties properties = new Properties();

  public Object getProperty(String name)
  {
    return properties.get(name);
  }

  public void setProperty(String name, Object value)
  {
    properties.setProperty(name, value.toString());
  }
}

Notice how the service is defined in the net.java.sip.communicator.service package, while the implementation is in net.java.sip.communicator.impl . All services and implementations in Jitsi are separated under these two packages. OSGi allows bundles to only make some packages visible outside their own JAR, so the separation makes it easier for bundles to only export their service packages and keep their implementations hidden.

The last thing we need to do so that people can start using our implementation is to register it in the BundleContext and indicate that it provides an implementation of the ConfigurationService . Here's how this happens:

package net.java.sip.communicator.impl.configuration;

import org.osgi.framework.*;
import net.java.sip.communicator.service.configuration;

public class ConfigActivator implements BundleActivator
{
  public void start(BundleContext bc) throws Exception
  {
    bc.registerService(ConfigurationService.class.getName(), // service name
         new ConfigurationServiceImpl(), // service implementation
         null);
  }
}

Once the ConfigurationServiceImpl class is registered in the BundleContext , other bundles can start using it. Here's an example showing how some random bundle can use our configuration service:

package net.java.sip.communicator.plugin.randombundle;

import org.osgi.framework.*;
import net.java.sip.communicator.service.configuration.*;

public class RandomBundleActivator implements BundleActivator
{
  public void start(BundleContext bc) throws Exception
  {
    ServiceReference cRef = bc.getServiceReference(
                              ConfigurationService.class.getName());
    configService = (ConfigurationService) bc.getService(cRef);

    // And that's all! We have a reference to the service implementation
    // and we are ready to start saving properties:
    configService.setProperty("propertyName", "propertyValue");
  }
}

Once again, notice the package. In net.java.sip.communicator.plugin we keep bundles that use services defined by others but that neither export nor implement any themselves. Configuration forms are a good example of such plugins: They are additions to the Jitsi user interface that allow users to configure certain aspects of the application. When users change preferences, configuration forms interact with the ConfigurationService or directly with the bundles responsible for a feature. However, none of the other bundles ever need to interact with them in any way ().

Figure 10.2: Service Structure

10.3. Building and Running a Bundle

Now that we've seen how to write the code in a bundle, it's time to talk about packaging. When running, all bundles need to indicate three different things to the OSGi environment: the Java packages they make available to others (i.e. exported packages), the ones that they would like to use from others (i.e. imported packages), and the name of their BundleActivator class. Bundles do this through the manifest of the JAR file that they will be deployed in.

For the ConfigurationService that we defined above, the manifest file could look like this:

Bundle-Activator: net.java.sip.communicator.impl.configuration.ConfigActivator
Bundle-Name: Configuration Service Implementation
Bundle-Description: A bundle that offers configuration utilities
Bundle-Vendor: jitsi.org
Bundle-Version: 0.0.1
System-Bundle: yes
Import-Package: org.osgi.framework,
Export-Package: net.java.sip.communicator.service.configuration

After creating the JAR manifest, we are ready to create the bundle itself. In Jitsi we use Apache Ant to handle all build-related tasks. In order to add a bundle to the Jitsi build process, you need to edit the build.xml file in the root directory of the project. Bundle JARs are created at the bottom of the build.xml file, with bundle-xxx targets. In order to build our configuration service we need the following:

<target name="bundle-configuration">
  <jar destfile="${bundles.dest}/configuration.jar" manifest=
    "${src}/net/java/sip/communicator/impl/configuration/conf.manifest.mf" >

    <zipfileset dir="${dest}/net/java/sip/communicator/service/configuration"
        prefix="net/java/sip/communicator/service/configuration"/>
    <zipfileset dir="${dest}/net/java/sip/communicator/impl/configuration"
        prefix="net/java/sip/communicator/impl/configuration" />
  </jar>
</target>

As you can see, the Ant target simply creates a JAR file using our configuration manifest, and adds to it the configuration packages from the service and impl hierarchies. Now the only thing that we need to do is to make Felix load it.

We already mentioned that Jitsi is merely a collection of OSGi bundles. When a user executes the application, they actually start Felix with a list of bundles that it needs to load. You can find that list in our lib directory, inside a file called felix.client.run.properties . Felix starts bundles in the order defined by start levels: All those within a particular level are guaranteed to complete before bundles in subsequent levels start loading. Although you can't see this in the example code above, our configuration service stores properties in files so it needs to use our FileAccessService , shipped within the fileaccess.jar file. We'll therefore make sure that the ConfigurationService starts after the FileAccessService:

⋮    ⋮    ⋮
felix.auto.start.30= \
  reference:file:sc-bundles/fileaccess.jar

felix.auto.start.40= \
  reference:file:sc-bundles/configuration.jar \
  reference:file:sc-bundles/jmdnslib.jar \
  reference:file:sc-bundles/provdisc.jar \
⋮    ⋮    ⋮

If you look at the felix.client.run.properties file, you'll see a list of packages at the beginning:

org.osgi.framework.system.packages.extra= \
  apple.awt; \
  com.apple.cocoa.application; \
  com.apple.cocoa.foundation; \
  com.apple.eawt; \
⋮    ⋮    ⋮

The list tells Felix what packages it needs to make available to bundles from the system classpath. This means that packages that are on this list can be imported by bundles (i.e. added to their Import-Package manifest header) without any being exported by any other bundle. The list mostly contains packages that come from OS-specific JRE parts, and Jitsi developers rarely need to add new ones to it; in most cases packages are made available by bundles.

10.4. Protocol Provider Service

The ProtocolProviderService in Jitsi defines the way all protocol implementations behave. It is the interface that other bundles (like the user interface) use when they need to send and receive messages, make calls, and share files through the networks that Jitsi connects to.

The protocol service interfaces can all be found under the net.java.sip.communicator.service.protocol package. There are multiple implementations of the service, one per supported protocol, and all are stored in net.java.sip.communicator.impl.protocol.protocol_name .

Let's start with the service.protocol directory. The most prominent piece is the ProtocolProviderService interface. Whenever someone needs to perform a protocol-related task, they have to look up an implementation of that service in the BundleContext . The service and its implementations allow Jitsi to connect to any of the supported networks, to retrieve the connection status and details, and most importantly to obtain references to the classes that implement the actual communications tasks like chatting and making calls.

10.4.1. Operation Sets

As we mentioned earlier, the ProtocolProviderService needs to leverage the various communication protocols and their differences. While this is particularly simple for features that all protocols share, like sending a message, things get trickier for tasks that only some protocols support. Sometimes these differences come from the service itself: For example, most of the SIP services out there do not support server-stored contact lists, while this is a relatively well-supported feature with all other protocols. MSN and AIM are another good example: at one time neither of them offered the ability to send messages to offline users, while everyone else did. (This has since changed.)

The bottom line is our ProtocolProviderService needs to have a way of handling these differences so that other bundles, like the GUI, act accordingly; there's no point in adding a call button to an AIM contact if there's no way to actually make a call.

OperationSets to the rescue (). Unsurprisingly, they are sets of operations, and provide the interface that Jitsi bundles use to control the protocol implementations. The methods that you find in an operation set interface are all related to a particular feature. OperationSetBasicInstantMessaging , for instance, contains methods for creating and sending instant messages, and registering listeners that allow Jitsi to retrieve messages it receives. Another example, OperationSetPresence , has methods for querying the status of the contacts on your list and setting a status for yourself. So when the GUI updates the status it shows for a contact, or sends a message to a contact, it is first able to ask the corresponding provider whether they support presence and messaging. The methods that ProtocolProviderService defines for that purpose are:

public Map<String, OperationSet> getSupportedOperationSets();
public <T extends OperationSet> T getOperationSet(Class<T> opsetClass);

OperationSets have to be designed so that it is unlikely that a new protocol we add has support for only some of the operations defined in an OperationSet. For example, some protocols do not support server-stored contact lists even though they allow users to query each other's status. Therefore, rather than combining the presence management and buddy list retrieval features in OperationSetPresence , we also defined an OperationSetPersistentPresence which is only used with protocols that can store contacts online. On the other hand, we have yet to come across a protocol that only allows sending messages without receiving any, which is why things like sending and receiving messages can be safely combined.

Figure 10.3: Operation Sets

10.5. Media Service

When working with real-time communication over IP, there is one important thing to understand: protocols like SIP and XMPP, while recognized by many as the most common VoIP protocols, are not the ones that actually move voice and video over the Internet. This task is handled by the Real-time Transport Protocol (RTP). SIP and XMPP are only responsible for preparing everything that RTP needs, like determining the address where RTP packets need to be sent and negotiating the format that audio and video need to be encoded in (i.e. codec), etc. They also take care of things like locating users, maintaining their presence, making the phones ring, and many others. This is why protocols like SIP and XMPP are often referred to as signalling protocols.

What does this mean in the context of Jitsi? Well, first of all it means that you are not going to find any code manipulating audio or video flows in either the sip or jabber jitsi packages. This kind of code lives in our MediaService. The MediaService and its implementation are located in net.java.sip.communicator.service.neomedia and net.java.sip.communicator.impl.neomedia .

Why "neomedia"?

The "neo" in the neomedia package name indicates that it replaces a similar package that we used originally and that we then had to completely rewrite. This is actually how we came up with one of our rules of thumb: It is hardly ever worth it to spend a lot of time designing an application to be 100% future-proof. There is simply no way of taking everything into account, so you are bound to have to make changes later anyway. Besides, it is quite likely that a painstaking design phase will introduce complexities that you will never need because the scenarios you prepared for never happen.

In addition to the MediaService itself, there are two other interfaces that are particularly important: MediaDevice and MediaStream.

10.5.1. Capture, Streaming, and Playback

MediaDevices represent the capture and playback devices that we use during a call (). Your microphone and speakers, your headset and your webcam are all examples of such MediaDevices, but they are not the only ones. Desktop streaming and sharing calls in Jitsi capture video from your desktop, while a conference call uses an AudioMixer device in order to mix the audio we receive from the active participants. In all cases, MediaDevices represent only a single MediaType. That is, they can only be either audio or video but never both. This means that if, for example, you have a webcam with an integrated microphone, Jitsi sees it as two devices: one that can only capture video, and another one that can only capture sound.

Devices alone, however, are not enough to make a phone or a video call. In addition to playing and capturing media, one has to also be able to send it over the network. This is where MediaStreams come in. A MediaStream interface is what connects a MediaDevice to your interlocutor. It represents incoming and outgoing packets that you exchange with them within a call.

Just as with devices, one stream can be responsible for only one MediaType. This means that in the case of an audio/video call Jitsi has to create two separate media streams and then connect each to the corresponding audio or video MediaDevice.

Figure 10.4: Media Streams For Different Devices

public List<MediaDevice> getDevices(MediaType mediaType, MediaUseCase useCase);

public List<MediaFormat> getSupportedFormats();

// first create a stream connector telling the media service what sockets
// to use when transport media with RTP and flow control and statistics
// messages with RTCP
StreamConnector connector =  new DefaultStreamConnector(rtpSocket, rtcpSocket);
MediaStream stream = mediaService.createMediaStream(connector, device, control);

// A MediaStreamTarget indicates the address and ports where our
// interlocutor is expecting media. Different VoIP protocols have their
// own ways of exchanging this information
stream.setTarget(target);

// The MediaDirection parameter tells the stream whether it is going to be
// incoming, outgoing or both
stream.setDirection(direction);

// Then we set the stream format. We use the one that came
// first in the list returned in the session negotiation answer.
stream.setFormat(format);

// Finally, we are ready to actually start grabbing media from our
// media device and streaming it over the Internet
stream.start();

10.6. UI Service

So far we have covered parts of Jitsi that deal with protocols, sending and receiving messages and making calls. Above all, however, Jitsi is an application used by actual people and as such, one of its most important aspects is its user interface. Most of the time the user interface uses the services that all the other bundles in Jitsi expose. There are some cases, however, where things happen the other way around.

Plugins are the first example that comes to mind. Plugins in Jitsi often need to be able to interact with the user. This means they have to open, close, move or add components to existing windows and panels in the user interface. This is where our UIService comes into play. It allows for basic control over the main window in Jitsi and this is how our icons in the Mac OS X dock and the Windows notification area let users control the application.

In addition to simply playing with the contact list, plugins can also extend it. The plugin that implements support for chat encryption (OTR) in Jitsi is a good example for this. Our OTR bundle needs to register several GUI components in various parts of the user interface. It adds a padlock button in the chat window and a sub-section in the right-click menu of all contacts.

The good news is that it can do all this with just a few method calls. The OSGi activator for the OTR bundle, OtrActivator, contains the following lines:

Hashtable<String, String> filter = new Hashtable<String, String>();

// Register the right-click menu item.
filter(Container.CONTAINER_ID,
    Container.CONTAINER_CONTACT_RIGHT_BUTTON_MENU.getID());

bundleContext.registerService(PluginComponent.class.getName(),
    new OtrMetaContactMenu(Container.CONTAINER_CONTACT_RIGHT_BUTTON_MENU),
    filter);

// Register the chat window menu bar item.
filter.put(Container.CONTAINER_ID,
           Container.CONTAINER_CHAT_MENU_BAR.getID());

bundleContext.registerService(PluginComponent.class.getName(),
           new OtrMetaContactMenu(Container.CONTAINER_CHAT_MENU_BAR),
           filter);

As you can see, adding components to our graphical user interface simply comes down to registering OSGi services. On the other side of the fence, our UIService implementation is looking for implementations of its PluginComponent interface. Whenever it detects that a new implementation has been registered, it obtains a reference to it and adds it to the container indicated in the OSGi service filter.

Here's how this happens in the case of the right-click menu item. Within the UI bundle, the class that represents the right click menu, MetaContactRightButtonMenu, contains the following lines:

// Search for plugin components registered through the OSGI bundle context.
ServiceReference[] serRefs = null;

String osgiFilter = "("
    + Container.CONTAINER_ID
    + "="+Container.CONTAINER_CONTACT_RIGHT_BUTTON_MENU.getID()+")";

serRefs = GuiActivator.bundleContext.getServiceReferences(
        PluginComponent.class.getName(),
        osgiFilter);
// Go through all the plugins we found and add them to the menu.
for (int i = 0; i < serRefs.length; i ++)
{
    PluginComponent component = (PluginComponent) GuiActivator
        .bundleContext.getService(serRefs[i]);

    component.setCurrentContact(metaContact);

    if (component.getComponent() == null)
        continue;

    this.add((Component)component.getComponent());
}

And that's all there is to it. Most of the windows that you see within Jitsi do exactly the same thing: They look through the bundle context for services implementing the PluginComponent interface that have a filter indicating that they want to be added to the corresponding container. Plugins are like hitch-hikers holding up signs with the names of their destinations, making Jitsi windows the drivers who pick them up.

10.7. Lessons Learned

When we started work on SIP Communicator, one of the most common criticisms or questions we heard was: "Why are you using Java? Don't you know it's slow? You'd never be able to get decent quality for audio/video calls!" The "Java is slow" myth has even been repeated by potential users as a reason they stick with Skype instead of trying Jitsi. But the first lesson we've learned from our work on the project is that efficiency is no more of a concern with Java than it would have been with C++ or other native alternatives.

We won't pretend that the decision to choose Java was the result of rigorous analysis of all possible options. We simply wanted an easy way to build something that ran on Windows and Linux, and Java and the Java Media Framework seemed to offer one relatively easy way of doing so.

Throughout the years we haven't had many reasons to regret this decision. Quite the contrary: even though it doesn't make it completely transparent, Java does help portability and 90% of the code in SIP Communicator doesn't change from one OS to the next. This includes all the protocol stack implementations (e.g., SIP, XMPP, RTP, etc.) that are complex enough as they are. Not having to worry about OS specifics in such parts of the code has proven immensely useful.

Furthermore, Java's popularity has turned out to be very important when building our community. Contributors are a scarce resource as it is. People need to like the nature of the application, they need to find time and motivation—all of this is hard to muster. Not requiring them to learn a new language is, therefore, an advantage.

Contrary to most expectations, Java's presumed lack of speed has rarely been a reason to go native. Most of the time decisions to use native languages were driven by OS integration and how much access Java was giving us to OS-specific utilities. Below we discuss the three most important areas where Java fell short.

The important thing is that whenever we needed to choose a native solution, we could, and we did. This brings us to our point: Ever since we've started Jitsi we've fixed, added, or even entirely rewritten various parts of it because we wanted them to look, feel or perform better. However, we've never ever regretted any of the things we didn't get right the first time. When in doubt, we simply picked one of the available options and went with it. We could have waited until we knew better what we were doing, but if we had, there would be no Jitsi today.

10.8. Acknowledgments

Many thanks to Yana Stamcheva for creating all the diagrams in this chapter.