Yutani: The new compositor

In developing とあるOS, I've had a consistent focus on UI. Since early 2012, the OS has had a compositing window server, which was hacked together in roughly a week for a hackathon and presented at UIUC's Engineering Open House. Over the past two years, a number of improvements have been made to the compositor, including a rewrite of the graphics layer to use Cairo and the addition of animations. All the while, the compositor suffered from some major preformance issues, partly due to the overall design of the protocol, but mostly due to inabilities of the underlying kernel. Replacing the compositor with a new one, built from scratch with a well-designed API and a focus on performance had been an idea I had floating around in the back of my mind for quite a while, but it was not until April that I finally moved forward in implementing it.

For those unfamiliar, とあるOS looked something like this in March:

Screenshot circa March 2014

Make note of the framerate the gears application is running at. I believe the last screenshot I have of the old compositor is most likely this one from April 1st:

Screenshot circa April 2014

In building a new compositor, I wanted to focus on solving the more glaring issues in the old one. The first step to this was adding damage rects. Damage rects (short for rectangles) allow the compositor to track when and where clients have made changes to their windows and ensure that only the updated areas are redrawn - the old compositor would assume everything may have changed each time it needed to redraw the screen. In February, I built a prototype in SDL that ran on Linux, demonstrating a simple compositor. The prototype was titled Yutani, a reference to the Alien franchise and, through that, also a reference to Wayland (the megacorp in Alien is called Weyland-Yutani).

Yutani prototype, February 2014

The prototype couldn't do much - it didn't have any IPC, and the only thing that moved was the mouse cursor, but it did allow me to experiment with using Cairo's clipping functionality to implement damage rects. The prototype sat idle throughout March while I diverted my attention to implementing a module system in the kernel, and then worked on a release candidate for the year's April Fools Day. It wasn't until mid-April that I picked things up again.

Not wanting to have to completely redesign my client graphics libraries, Yutani needed to operate on the same basic principles as the existing compositor: Windows were stored as shared memory regions with 32-bit RGBA colors. The clients would use a communication pipe to send requests to the server (ie., create a new window, etc.) and receive events (ie., mouse moved, keyboard key pressed, etc.). There was a critical flaw in the IPC mechanism the old compositor used to communicate events: The server was unable to wait for a message from any client (the kernel does not support select(), even to this day). The workaround employed in the old compositor was to run a thread that would loop through each of the known clients, check if they had data to read, and then process it if so. In designing Yutani, there was an obvious need to rectify the problem. This resulted in the creation of a new kernel IPC mechanism which was dubbed "pex" (short for packet exchange). Pex operates similarly to Unix sockets in that a server can create an endpoint which multiple clients can then connect to. When the server goes to read from the endpoint, it receives individual packets that include header information describing where the packet came from. The server then responds by writing a similarly-formatted packet back out to the endpoint. Meanwhile, clients read and write raw packets, without headers. Since the server only has to read from a single file to receive data from all of its clients, an implementation of select() is not needed.

Yutani, alpha, April 2014

The first builds of Yutani just drew boxes, but I eventually ported a larger graphical application: the login manager.

Login manager under Yutani, April 2014

Next up on the ports list was the terminal.

Terminal under Yutani, April 2014

Eventually, the wallpaper and panel were also ported, but Yutani still lacked window management - it wasn't even possible to change focus.

Terminal, panel, wallpaper under Yutani

To support window management features quickly, code was ported from the old compositor for handling mouse events. With damage rects, window movement needed to be cleaned up:

Missing damage regions cause artifacts

Eventually, Yutani supported everything the old compositor supported, including window rotation:

Rotated window

With Yutani in a state of relative feature parity, the remaining apps were ported and the old compositor was removed from the git repository. It was now time to add more features. The most popularly requested feature was to add a window list to the panel. This required adding the ability to inform the compositor of the names of windows. An advertise-subscribe model was implemented, allowing the panel to subscribe to changes in windows, while clients would advertise their new titles. The panel would be informed not just of new window titles, but also when focus changed and when windows disappeared.

Window titles in panel

Expanding on this initial implementation, window icons were added to the advertisements - communicated as identifiers such as utilities-terminal which the panel would look for in /usr/share/icons/$SIZE/. The panel design went through a few different iterations on the drawing board before being implemented.

Panel with icons

The newest feature added to Yutani is support for global key bindings. An app, such as the panel, can request to steal or intercept specific key sequences, which allows for global bindings like Ctrl+Alt+T to open a terminal. This was further extended to support having the panel manage Alt+Tab to switch windows.

Alt-Tab capture from YouTube video

(That last screenshot links to a video of Alt-Tab in action.)

There's still a lot of things that can be added to Yutani. For now, I'm going to take a break from UI improvements to work on something a bit more useful: a network stack.