*This article is part of a series of blog posts covering the many different areas of work going on in Ubuntu right now. See the [introduction post here](https://archivedblog.jonobacon.com/2013/08/21/ubuntu-in-a-nutshell-series/) that links to all the articles*.

Unity is the graphical environment that we ship in a default Ubuntu installation. Released for the first time about three years ago, Unity is focused on simplicity and consistency across multiple devices. In this article I am going to cover the history of Unity and how Unity 8 is driving a new era of code and design convergence in Ubuntu.

Although Unity is a single graphical experience, you can think of it in three broad buckets:

1. Design – the visual design and interaction experience.
2. Platform – the core Unity platform software.
3. Services – a set of functions that Unity makes available to applications for integration and for content to be viewed.

Back when the design team started working on Unity the goal was to solve a key set of problems in the user experience of Linux and build a simple and efficient user interface, but to also build a set of design patterns that could scale across multiple form-factors.

Design is a complex process, and a process inspired by ideas but defined by practical real-world testing. As the team developed ideas, tested them, and iterated on them, these ideas were boiled into a set of common design patterns that were not merely intended for a desktop…but that could be applied to other form-factors too. This was a challenging prospect: to a reasonable extent, building design around a specific form-factor is much easier than producing scalable designs across these different form factors, and the level of design work that went into Unity was and continues to be incredible.

It is important to remember that this design effort was not limited to the launcher, panel, indicators and other common visible pieces that we associate as the building blocks of Unity; it was also the gestures, login manager, toolkit components, dash components, and more. The goal has always been to have a consistent design story across *everything* we do: Unity, Juju, conference booths, websites, and more, and thus the Unity work needed to be able to exist within this wider design story.

As the designs formed for Unity the development team kicked into action.

Back in the early days we had two different code-bases for Unity; a 3D and 2D code-base. The simple reason for this is that some OEMs wanted to ship Unity on hardware without 3D acceleration, so we decided to have two different branches and share as much code as possible between these branches so we could serve all potential OEMs.

These branches were quite different though. Unity 3D was written with Compiz and a graphical toolkit we created called NUX, and Unity 2D was written in Qt/QML. As you can imagine, this resulted in some duplication of effort and some deltas between 3D and 2D. The teams worked their socks off, but these technical limitations were causing some headaches.

As the goals to spin up our full convergence story across TV, Mobile, Tablet, and Desktop became more important it was clear that we needed to make a decision about these two different code-bases. After an extensive amount of discussion it was decided to focus explicitly on Qt/QML as our primary code-base, a decision that also matched our decision to [focus on Qt/QML as the core of our Ubuntu SDK](https://archivedblog.jonobacon.com/2013/08/22/ubuntu-in-a-nutshell-the-ubuntu-sdk-and-developer-story/).

This now brought technical consistency across our engineering teams: Qt/QML would form the bases for Unity, the Ubuntu SDK, new application development, our app developer community growth and more. I had been lobbying for a focus on Qt/QML for some years, so speaking personally I was delighted by this move. 🙂

A core benefit of Unity is the rich range of services that it provides. This includes services such as:

* The Launcher – showing running apps, progress bars, number pills, quicklists etc.
* The Messaging Menu – all of your messages in one place, irrespective of source or network.
* The HUD – search enabled for all your apps so you can find exactly what you need by finger or voice.

Another very significant service is The Dash. In previous versions of Unity the dash provided a place to search your local system and a limited set of online services. For Ubuntu 13.10 the dash has been extended to search 50+ services all from the home scope. The dash provides a consistent place to find and search for content, apps, information and more.

Again, it is important to remember that these services are not just useful for the desktop…they apply to all the different form-factors we are focusing on. This, tied together with our convergence-ready SDK means you can consume these services in your app and they run across all these different devices.

## Unity 8

Earlier I mentioned the decision to focus on Qt/QML as our core development focus, but that was not the only decision moving forward with Unity however. The goal was also to build true convergence into the core Unity code-base too. Our goal was to have a single Unity code-base and when you run it on a Desktop you get one experience, and when you run it on Phone you get another experience. This boils the full convergence story down into a single code-base, which also means that if you fix a bug in that code-base, the bug fix applies to all devices too.

This is *true* convergence: a single code-base with a scalable set of design patterns that can be deployed on a wide range of devices.

This focus has materialized in **Unity 8**; the next generation of Unity that is currently running in the Ubuntu Phone and Ubuntu Tablet images. On the desktop we are still running Unity 7 (based on Compiz and Nux) until Unity 8 has desktop features baked in.

If you want to see this convergence working, install Unity 8 on your Saucy desktop with:

sudo apt-get install unity8

Then run it with:

export UBUNTU_ICON_THEME=ubuntu-mobile
unity8 -mousetouch

This will load what looks like Ubuntu Phone on your desktop in a window. It should look like this:

Remember, Unity changes how it operates based on the screen size. To see this in action, increase the size of the window and you will now see something that looks like LightDM:

Re-size it smaller again and it looks like the phone interface again.

Another fun test is when at the phone size, slide from the left of the window to show the Launcher and click the Ubuntu button. Now click the ‘Search’ button and the search box takes up the full width of the window:

Now re-size the window to be much bigger and click the ‘Search’ button again; the search box now appears to the right of the window:

This is a subtle example of how Unity 8 adjusts the experience based upon the screen size and the goal is that we will make many such changes to optimize the Unity experience across these different form factors, but the core ingredients, technology, and focus on content is the same, just visible from different perspectives.

## Are We There Yet?

Today Unity 8 is running on the phone and tablet. Currently the vast majority of engineering focus is going into making everything work for Ubuntu Phone, but all of this engineering going into Unity 8 is built within the context of working on other devices too. As an example, although Unity 8 running on a full sized desktop screen looks like a mobile device running on a monitor (a native desktop UI will be added later), the core of the dash and all the system services will be desktop ready: they just need to be extended to support that screen profile.

In other words, although it might feel we are not working on the desktop, everything the engineering teams are working on is work we would need to do for the desktop in Unity 8 anyway, so it is all valuable work heading in a consistent direction.

Not only this but there is a far greater level of continuous integration and testing in Unity now than ever before. Every four hours there is a battery of tests run against the trunk code-base and if the tests pass a new package appears in the Ubuntu archive, giving you the opportunity to test it and keep up to date with the very latest in Unity. All feature planning and tracking is done publicly in blueprints and discussed at our [Ubuntu Developer Summits](https://uds.ubuntu.com).

The goal is that the phone user interface in Unity 8 will be mostly complete for Ubuntu 13.10, and then the focus will be on Desktop between 13.10 and 14.10 with the goal of shipping Unity 8 on the desktop in Ubuntu 14.10, thus spanning full convergence across all form-factors with this single Unity code-base and set of scalable design patterns.

This week from **Tues 27th – Thurs 29th August 2013** from **2pm – 8pm UTC** we will be running our online [Ubuntu Developer Summit](https://uds.ubuntu.com). This is an event we run every three months to discuss, debate, and plan the next three months of Ubuntu as well as wider goals.

All sessions will run using a combination of Google+ streaming video hangouts and IRC, and you can see the full schedule on [summit.ubuntu.com](https://summit.ubuntu.com). Consequently, for those who cannot attend or might miss certain sessions, all sessions will be available pre-recorded from the session pages when the session is complete.

We will kick off at **2pm UTC** with my normal intro and a keynote from [Mark Shuttleworth](https://markshuttleworth.com/), founder of Ubuntu.

*This article is part of a series of blog posts covering the many different areas of work going on in Ubuntu right now. See the [introduction post here](https://archivedblog.jonobacon.com/2013/08/21/ubuntu-in-a-nutshell-series/) that links to all the articles*.

In my [last article](https://archivedblog.jonobacon.com/2013/08/21/ubuntu-in-a-nutshell-app-upload-process/) I talked about the new app upload process, but today I am going to talk about how developers write apps in the first place.

For a long time our app developer story in Ubuntu has been quite fragmented. This has been due to a number of reasons:

* We have not had a single consistent platform that we ask developers to write to. We have traditionally supported GTK, Qt, and anything else that lives in the archive. This not only presents a inconsistent developer experience, but also an inconsistent user experience too.
* We lacked app design guidelines around how developers should build apps that look consistent on the platform.
* We didn’t have a single consistent developer portal and support network to provide the support and guidance app developers need to build awesome apps and get them into the platform.
* We also didn’t have a good answer for writing an app that can work across multiple form factors.
* Finally, we didn’t have a single consistent SDK that developers could use to write apps: they had to pick from a plethora of tools, with varying degrees of quality.

We tried to rectify some of these issues by recommending people write apps with Python and GTK, and we wrote a tool called Quickly to optimize this process. Quickly would generate a project and help with tasks such as editing, creating your UI, and generating a package, but quickly was a somewhat primitive and incomplete solution to the problem.

The work on Quickly also showcased some limitations in our tooling. At the time we recommended people write apps using GEdit, Glade, and GTK. Unfortunately, this collection of tools just didn’t compare favorably to the developer experience on Apple and Google’s platforms, despite the best efforts of the respective upstreams. We needed to provide an end-to-end SDK for developers that would take a developer from a new project through to submitting the app into the Ubuntu Software Center.

## Choosing a Technology

We set out to resolve these issues and build a consistent Ubuntu SDK.

The first decision we made was around which frameworks we wanted to support when developers write their apps. These frameworks needed to be highly efficient and able to converge across multiple devices. We finalized this list as:

* **Qt/QML** – native applications that can be run on any of the devices and adapt to the screen size.
* **HTML5** – web applications that can also adapt to the device with deep integration into the system services (e.g. messaging menu, launcher etc).
* **Online Services** – integration of web apps into the system services (e.g. messaging menu and unity integration).
* **OpenGL** – full OpenGL support for games.

Some time ago we decided to focus on Qt as a platform for not only building our SDK but building our convergence story too. Qt has many benefits:

* It provides a fast C++ library and toolkit as well as a neat higher-level declarative technology in the form of QML. This means that we have the power of C++ for system software (e.g. writing Unity) but app devs can write apps using a high-performance higher level technology that is easier to learn and faster to write apps with.
* Qt provides an awesome set of tools – an integrated IDE, debugger, designer and more.
* The Qt Creator IDE is very pluggable which means we could use it for our main IDE and use it for writing apps in HTML5 and OpenGL.
* Qt and QML documentation is *fantastic*.
* Qt has a strong eco-system surrounding it and lots of companies in that eco-system. This makes contracting out work and hiring much easier.
* Qt is a healthy upstream and very keen to work with those who consume it.

We also started looking into the best way in which we could support HTML5 developers. While the IDE decision had been made (Qt Creator) we also decided to invest in building Apache Cordova support into our SDK to make writing HTML5 as flexible as possible. This way you can either write a stock HTML5 app or use the cordova functionality…all accessible within the same IDE.

## The Ubuntu SDK

We formed the SDK team and started work. This work was broken into two areas.

Firstly, we started work on the app developer platform. This is largely identifying the needs of app developers for writing apps for Ubuntu devices, and ensuring we have support for those needs (which largely requires integrating that support and creating APIs). This has included:

* Building the Ubuntu Component set – a set of widgets that are usable in QML and HTML5 that developers can use to construct their apps.
* Application lifecycle (suspending apps to preserve battery life).
* Location Services.
* Multimedia and Music.
* Alarms.
* Calendar Integration (using Evolution Data Server).
* Sensor services (e.g. accelerometer).

This work is currently on-going and in various stages of completeness, but all of these platform APIs will be ready by the end of August and many apps are already consuming them. Remember, these services will be made available across all form factors.

The second piece was the SDK itself. This is tuning the Qt Creator IDE for our needs and ensuring it can be used to create QML, HTML5, and OpenGL apps. This work has touched on a number of different areas and has resulted in the following features:

* We have project templates for QML, HTML5 (Cordova), HTML5 (Stock), and Scopes – here you can easily generate a project to get started with.
* Source control integration for Bazaar and Git – this makes collaboration around an app easier.
* Device integration – with just a click of a button you can run your app on an Ubuntu device to test that it works correctly.
* Click package generation – generate a click package that you can use to upload to the Ubuntu Software Center.
* Ubuntu Component Showcase – browse all the different Ubuntu components and see the code for how to use them.
* Integrated documentation, IRC, design guidelines, and Ask Ubuntu support.

We rolled all of these features into the first Beta of the SDK which was released about a month ago and you can get started with it on [developer.ubuntu.com](https://developer.ubuntu.com).

## developer.ubuntu.com

Speaking of developer.ubuntu.com, we have invested significantly in making the site a central resource for all of your development needs.

Currently the site provides tutorials for building apps, API documentation, and a cookbook that brings together the top rated questions from Ask Ubuntu. The site provides a good spring-board for getting started.

We are however in the process of making a number of improvements to developer.ubuntu.com. This will include:

* Revised site navigation and structure to make it easier to use.
* Better and more clearly integrated API documentation.
* Wider API coverage.
* Cookbooks for all of the different app templates.
* Full integration of Juju Charm documentation and API.

We are expecting to have many of these improvements in place in the coming weeks.

## Are We There yet?

As we stand today we now have a powerful Ubuntu SDK with support for writing convergent apps in Qt/QML, HTML5, OpenGL, and writing Scopes that fit into the dash. You can go to [developer.ubuntu.com](https://developer.ubuntu.com) to find out more, install the SDK, and fine tutorials for getting started.

We are only just gettin started though. The 1.0 of the SDK will be released in October and expect to find more refinements, better integration, and more features as we understand the needs of our developers better and expand the platform.

As many of you will have seen, unfortunately the [Ubuntu Edge campaign](https://www.indiegogo.com/projects/ubuntu-edge) did not reach our goal of $32million. The final total reached was $12,812,776. I am hugely proud and thankful to everyone who pledged, supported the campaign, wrote about it, and helped to spread the word.

Some have described us not meeting the goal as a “failure”. I don’t see it that way. Let’s be honest: $32million was always an incredibly ambitious target. We would have liked to have done it for less money, but building a F1 superphone doesn’t come cheap (and remember that the $32million didn’t include any costs for software engineering and project management…Canonical were providing that for free). It was an ambitious target, but disrupting an industry is ambitious in itself, and we gave the crowd-funding campaign our best shot. The story does not end here though.

I am not surprised that we didn’t hit this ambitious $32million target, but I am surprised at what we did achieve. We broke all the crowd-funding records, garnered media attention across CNBC, Engadget, The Independent, TechCrunch, the BBC, T3, Stuff, The Verge, The Guardian, Wired, pandodaily, Fast Company, Forbes, The Telegraph and more. Every single person who put their support into the Ubuntu Edge campaign should be proud of their achievements and we are all thankful for your tremendous and inspiring support.

One thing to be critically clear about is that the Ubuntu convergence story does not end here. We are as voraciously excited and committed to bringing this Free Software convergence story to the world as ever before; our work with OEMs, Carriers, and ISVs continues apace. We have fantastic work going on across all fronts, and we are on track to have a 1.0 release of the Ubuntu Phone platform in October.

What this experience demonstrated to me more than anything was the passion and commitment of the Ubuntu family. We are a global and diverse family all united by a dream of what the future can look like, a future in which powerful, elegant technology is freely available to all, available in the devices people care about and use to learn, create, and live better lives. Our Ubuntu family is what makes us strong, and while we didn’t hit the $32million we saw yet another example of our family coming together as one and the wider industry getting a peek into our world and the technology we have to offer.

Onwards and upwards!

*This article is part of a series of blog posts covering the many different areas of work going on in Ubuntu right now. See the [introduction post here](https://archivedblog.jonobacon.com/2013/08/21/ubuntu-in-a-nutshell-series/) that links to all the articles*.

In this article I am going to discuss some improvements we making to significantly simply and speed up how app devs get their apps into Ubuntu.

You can think of an Ubuntu system as two layers of software:

1. The System – the core Operating System includes the system software itself that is required to boot the device, bring up services (e.g. networking/bluetooth/sound), and display the user interface.
2. The Applications – these are the applications that run on top of the system where you spend most of your time as a user.

When we started working on Ubuntu back in 2004 the system and the applications where closely intermingled. We basically synced the Debian archive with Ubuntu, applied a bunch of patches, and whatever was in the Debian and Ubuntu archives was available as applications.

There were some inherent problems with this approach:

* To get an app into Ubuntu you need to create a `.deb` file (a Debian package). Debian packages are really designed to be used by people who build Operating Systems, and not regular app developers.
* This approach meant that you could write an app with any software toolkit/platform that was in the archive. Unfortunately this results in a really inconsistent end-user experience – we have tried our best to support GTK and Qt apps, but these apps look and work quite differently, and it spreads the development team thinl trying to cater to all tastes.
* If an app developer wants to get an app into Ubuntu they have to either (a) be an Ubuntu developer with upload access or (b) convince a developer to get their app in. This doesn’t scale.

With these issues we tried to remedy the problem by creating the *Application Review Board*; a community group who would review packages with a simplified criteria. Unfortunately, despite the best efforts of the ARB, they were never really set up for success due to the technical limitations in the platform.

The primary technical limitation the ARB faced is that the Ubuntu system has traditionally lacked sand-boxing to protect it from naughty applications doing something…well, naughty. This meant that every app they reviewed would need a full code review for *every version submitted*. So, an app developer could release 1.0 and then release 1.1 a few weeks later and this would require two full code reviews. Not only this, but Debian packages have something called *maintainer scripts* that can theoretically do anything as root, so the packaging needed checking too.

For a group of volunteers with a growing list of submitted apps the ARB quickly fell behind. This resulted in frustration all around – for the ARB, for app devs, and for those of us working to refine and improve Ubuntu.

## The New App Upload Process

So, the previous system had the following primary limitations:

* Packaging was complicated.
* No app sandboxing, so apps needed a full code review.
* Maintainer scripts can do naughty things, so they needed a full review.
* Apps took a long time to be reviewed due to the technical limitations above.

We started building a new specification for the new process some time ago. My team [wrote the initial spec](https://wiki.ubuntu.com/AppDevUploadProcess) with input from a range of teams, and this has formed the foundation for the work that is reaching maturity now. Let’s cover how it works.

There are three primary pieces to this work, *click packages*, *sand-boxing*, and *uploading*.

## Click Packages

Firstly, remember how earlier I mentioned two layers in an Ubuntu system:

1. The System – the core Operating System includes the system software itself that is required to boot the device, bring up services (e.g. networking/bluetooth/sound), and display the user interface.
2. The Applications – these are the applications that run on top of the system where you spend most of your time as a user.

The Debian packaging format provides a remarkable range of features that are really designed for the former, for building Operating Systems. For the latter, app devs who simply want to package their code into something that can be loaded by users, we have created a new, simplified format for packages. It is called the *click* package format.

Click packages are simple to create: when you have your project open in the Ubuntu SDK you can press a button to generate a click package. Done! Click packages also don’t include maintainer scripts: they are simply not needed for the vast majority of apps, so, that already removes a security risk and something that needs reviewing.

A key benefit of click packages also is that they don’t include full dependency resolution. Many of you will be familiar with running `apt-get update` on your system. Well, this syncs the list of packages from the archive and figures out all the dependencies and how they map out (e.g. knowing that GIMP requires GTK, which in turn requires X). This takes quite some time and doesn’t scale to thousands of packages that getting updated all the time.

With a click package the software simply depends on the Ubuntu SDK. This means we don’t need to worry about all that complex dependency resolution: we know the dependency, the Ubuntu SDK. Additional dependencies can simply be bundled into the package. Also, instead of maintaining a list of packages on the system…they are on a web service. You need a connection to download the package anyway, so why not ask a service which packages are available?

## Sand-boxing

With a simple and efficient package format all set we next have the issue of sand-boxing.

This is more work than you might imagine. The team identified a big list of sand-boxing requirements that were needed for us to be sure an app could be run without a code review. This is not only protecting the system from inappropriate calls, but also handling issues with other components (e.g. sniffing keyboard events in X apps).

Well, the awesome Ubuntu Security Team has been working on full sand-boxing throughout the system and most of this work has been completed. This has also influenced some other design decisions: as an example, Mir (our new display server) is designed to not expose the keyboard event sniffing issue X has faced.

The basic result of this work is that the security team have an app called *evil app* that does a series of naughty things, and the sand-boxing protects the system from such naughtyness.

With this sand-boxing in place it essentially mitigates the need for a full code review, which was the primary bottleneck previously. This combined with click packages not having maintainer scripts and complex dependency chains makes reviews much easier and more efficient.

## Uploading

The process for uploading an app to Ubuntu is going to remain largely the same as it works generally pretty well. As before you can select a package to upload, set information about the app, add screenshots and more and then it will hit a review queue before it appears to Ubuntu users.

The good news is that because the review only really require a check of permissions, meta-data, and a few other simple bits (thanks to click packages and sand-boxing), the review can be done in under 15 minutes as opposed to multi-day code reviews. This means apps should go in more quickly.

Once your app is in the archive you will be able to search for it across all Ubuntu devices. Currently all apps will be displayed, but we are adding features to only show apps that will work on that device (e.g. only show apps with a phone UI on the phone).

## Are We There Yet?

So where do we stand with all this goodness? Fortunately, we are getting close.

The click package format is largely complete and we have tools for creating click packages in the SDK and installing them on a system.

Much of the sand-boxing work has been largely completed, although one caveat is that because we are working on Mir as our new display server, we are not investing in fixing keyboard sniffing in X (which would be a huge amount of work). This means that we won’t release this system on the desktop until Mir is on by default (around 14.10). Our existing system will be in place until then.

The app upload piece is online and largely complete (although going through a series of reviews), and the scope for browsing new click packages is on the mobile images although isn’t quite yet hooked up.

You can see an early video demo of this working below:

*Can’t see it? See it [here](https://www.youtube.com/watch?v=BjGAnV33GHU)!*

Our goal is to have this full system in place for mobile in 13.10, and for desktop in 14.10. This will all make getting apps into Ubuntu quicker, easier, and more rewarding than ever before.

There is some really awesome work going on right now in Ubuntu, and much of it is fixing and resolving issues and bottlenecks that have been an issue in Ubuntu for many years. Not only are we building an awesome convergence platform and cloud orchestration story, but we are re-building much of the core foundations to make these efforts more successful.

One of the challenges we have with all this great work is that even though everything is out in the open, some folks don’t have a crisp summary of the different pieces. So, I am kicking off a blog series that will summarize many of these different efforts as they stand.

My goal is to provide a overall summary of the work (not a huge wall of text) and when we expect it to be completed. I will regularly update this first post with a link to all the articles as I write them, so folks can point people to this post which will link to them all.

Enjoy!

## The Articles

* [App Upload Process](https://archivedblog.jonobacon.com/2013/08/21/ubuntu-in-a-nutshell-app-upload-process/)
* [The Ubuntu SDK and Developer Story](https://archivedblog.jonobacon.com/2013/08/22/ubuntu-in-a-nutshell-the-ubuntu-sdk-and-developer-story/)
* [Unity and Convergence](https://archivedblog.jonobacon.com/2013/08/27/ubuntu-in-a-nutshell-unity-and-convergence/)