Larry Price

And The Endless Cup Of Coffee

Snappy Libertine

| Comments

Libertine is software suite for runnin X11 apps in non-X11 environments and installing deb-based applications on a system without dpkg. Snappy is a package management system to confine applications from one another. Wouldn’t it be cool to run libertine as a snap?

Yes. Yes it would.

snapd

The first thing to install is snapd itself. You can find installation instructions for many Linux distros at snapcraft.io, but here’s the simple command if you’re on a debian-based operating system:

1
$ sudo apt install snapd

Ubuntu users may be surprised to find that snapd is already installed on their systems. snapd is the daemon for handling all things snappy: installing, removing, handling interface connections, etc.

lxd

We use lxd as our container backend for libertine in the snap. lxd is essentially a layer on top of lxc to give a better user experience. Fortunately for us, lxd has a snap all ready to go. Unfortunately, the snap version of lxd is incompatible with the deb-based version, so you’ll need to completely remove that before continuing. Skip this step if you never installed lxd:

1
2
3
4
$ sudo apt remove --purge lxd lxd-client
$ sudo zpool destroy lxd                 # if you use zfs
$ sudo ip link set lxdbr0 down           # take down the bridge (lxdbr0 is the default)
$ sudo brctl delbr lxdbr0                # delete the bridge

For installing, in-depth instructions can be found in this blog post by one of the lxd devs. In short, we’re going to create a new group called lxd, add ourselves to it, and then add our own user ID and group ID to map to root within the container.

1
2
3
4
5
6
$ sudo groupadd --system lxd                      # Create the group on your system
$ sudo usermod -G lxd -a $USER                    # Add the current user
$ newgrp lxd                                      # update current session with new group
$ echo root:`id --user ${USER}`:1 >> /etc/subuid  # Setup subuid to map correctly
$ echo root:`id --group ${USER}`:1 >> /etc/subgid # Setup subgid to map correctly
$ sudo snap install lxd                           # actually install the snap!

We also need to initialize lxd manually. For me, the defaults all work great. The important pieces here are setting up a new network bridge and a new filestore for lxd to use. You can optionally use zfs if you have it installed (zfsutils-linux should do it on Ubuntu). Generally, I just hit “return” as fast as the questions show up and everything turns out alright. If anything goes wrong, you may need to manually delete zpools, network bridges, or reinstall the lxd snap. No warranties here.

1
2
3
4
5
6
7
8
9
10
11
$ sudo lxd init
Do you want to configure a new storage pool (yes/no) [default=yes]?
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir or zfs) [default=zfs]:
Create a new ZFS pool (yes/no) [default=yes]?
Would you like to use an existing block device (yes/no) [default=no]?
Would you like LXD to be available over the network (yes/no) [default=no]?
Would you like stale cached images to be updated automatically (yes/no) [default=yes]?
Would you like to create a new network bridge (yes/no) [default=yes]?
What should the new bridge be called [default=lxdbr0]?
What IPv4 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]?

You should now be able to run lxd.lxc list without errors. It may warn you about running lxd init, but don’t worry about that if your initialization succeeded.

libertine

Now we’re onto the easy part. libertine is only available from edge channels in the app store, but we’re fairly close to having a version that we could push into more stable channels. For the latest and greatest libertine:

1
2
$ sudo snap install --edge libertine
$ sudo snap connect libertine:libertined-client libertine:libertined

If we want libertine to work fully, we need to jump through a couple of hoops. For starters, dbus-activation is not fully functional at this time for snaps. Lucky for us, we can fake this by either running the d-bus service manually (/snap/bin/libertined), or by adding this file to /usr/share/dbus-1/services/com.canonical.libertine.Service.service

/usr/share/dbus-1/services/com.canonical.libertine.Service.service
1
2
3
[D-BUS Service]
Name=com.canonical.libertine.Service
Exec=/snap/bin/libertine.libertined --cache-output

Personally, I always create the file, which will allow libertined to start automatically on the session bus whenever a user calls it. Hopefully d-bus activation will be fixed sooner rather than later, but this works fine for now.

Another issue is that existing deb-based libertine binaries may conflict with the snap binaries. We can fix this by adjusting PATH in our .bashrc file:

$HOME/.bashrc
1
2
# ...
export PATH=/snap/bin:$PATH

This will give higher priority to snap binaries (which should be the default, IMO). One more thing to fix before running full-force is to add an environment variable to /etc/environment such that the correct libertine binary is picked up in Unity 8:

/etc/environment
1
2
# ...
UBUNTU_APP_LAUNCH_LIBERTINE_LAUNCH=/snap/bin/libertine-launch

OK! Now we’re finally ready to start creating containers and installing packages:

1
2
3
4
$ libertine-container-manager create -i my-container
# ... (this could take a few minutes)
$ libertine-container-manager install-package -i my-container -p xterm
# ... (and any other packages you may want)

If you want to launch your apps in Unity 7 (why not?):

1
2
$ libertine-launch -i my-container xterm
# ... (lots of ourput, hopefully an open window!)

When running Unity 8, your apps should show up in the app drawer with all the other applications. This will all depend on libertined running, so make sure that it runs at startup!

I’ve been making a lot of improvements on the snap lately, especially as the ecosystem continues to mature. One day we plan for a much smoother experience, but this current setup will let us work out some of the kinks and find issues. If you want to switch back the deb-based libertine, you can just install it through apt and remove the change to /etc/environment.

2016 Retrospective

| Comments

This has been a unique year for me, and I wanted to quickly lay out what I’ve accomplished and where I think I’m going for the coming year. This is now officially a tradition at three posts (see: 2014 and 2015).

Revisiting 2016’s Goals

These are the goals I set for myself at the start of the year and how I met or missed them:

  • Spend Money
    • I wanted to become less of a miser and spend more money. Note that this is not for bragging purposes, just that I am naturally very frugal and hesitant to spend money. I think we did a pretty good job, though! For starters, I swapped out my commuter car for a tech-heavy crossover. We stayed in a really cool art-hotel in downtown Cincinnati in the Spring, drove across the country for a big Yellowstone trip in early Summer, stayed at the Indiana Dunes for a few days in the Fall, and took a brief trip to Madison, WI, shortly thereafter. I bought a nice sitting/standing desk and chair for my home office. I paid the entry fee to go to GenCon for a day. Our fridge, dishwasher, and furnace all died one weekend, and it ended with me buying upgraded, modern appliances. I’ve also been keeping the post office busy with plenty of orders off Amazon, and I’ve been trying to Kickstart more games that I think deserve attention. I also found a new hobby in homebrewing which has been a great use of fun-money.
  • Back Into Web
    • At the start of 2016, I though I really wanted to do more web work. Turns out I’ve done a 180 on this one, and I now work on primarily desktop/mobile with minimal web work, and I wouldn’t have it any other way for the time being.
  • Work With Others
    • In 2015, I worked alone a lot. In 2016, I joined a new company where I work remotely. Although I don’t necessarily see my coworkers every day, I am working on projects with multiple developers and communicating constantly with my teammates (and others) through chat. Working with others again has helped me grow socially and become a better engineer.
  • Become Part of an Open-Source Community
    • I really wanted to better integrate FLOSS into my life and my career, and I believe I’ve hit the jackpot. My job is to build popular open-source operating system Ubuntu, where all the code I write is public and generally GPLv3. My job is to write open-source software, and to interact with the community. I use my own software and report/fix bugs that affect me all the time.
  • Good Vibes
    • I was feeling a bit down at the end of 2015, and I wanted to be a more positive person in 2016. Although there were some depressing things going on worldwide in 2016, I am generally happier with my personal and professional life on a micro-level.

Surprise Victories

  • New Job
    • The big one! After almost four years at SEP, I transitioned to a new role at Canonical. I get to work on Ubuntu, one of my favorite open-source projects, as my full-time job. This also means I get to work remotely, dropping my commute from 30 miles in 2012 to 3 miles in 2014 to 50 feet in 2016. I’ve been having a great time working on software that’s in the public spotlight, working with the community, and traveling every few months to see my coworkers. With this new job, I’ve also had the opportunity to visit Europe for the first time this year.
  • Less Own-Time Work
    • Although I’ve hit pretty hard in the past that developers should do some learning outside of work, this year likely contained the least own-time work I’ve ever done. I’ve been finding joys in non-software hobbies and home maintenance, and working on Ubuntu as my full-time job has made me less needy for doing open-source work in my off-hours. I tend to respond to bugs and Github PRs at any hour, and I follow more technical people on social media than I used to. I think this stems from a satisfaction from the learning I do throughout the day, and the difficulty of separating work-life and home-life when one works at home.
  • FOSDEM
    • Yesterday, I learned that I’ll be giving a talk at FOSDEM. I’m very excited (and nervous) to give my first-ever conference talk in Brussels this February.
  • Homebrewing
    • I picked up homebrewing beer at the start of 2016, and I love it. I started with simple pre-made extract kits, but have worked my way up to creating my own all-grain recipes and labels. Brewing is a fun, tasty hobby giving me some creative, manual labor to break the mold of always doing computer work.
  • Books
  • Spotify
    • Spotify is very good for listening to whatever music I want anytime, especially now that I work at home. If you really like a band, you should still buy their music or merch through their website to make sure they get paid to keep existing.

2017 Goals

  • Local
    • I’d like to make sure I stay involved locally, especially as I continue to work from home. I’ve let my Golang group dwindle over the past few months, and I’d like to see us back up to our numbers at the start of 2016. If possible, I’d also like to attend other meetups and meet more local devs.
  • Linux Greybeard
    • This is a slow process, but I want to get better at using Linux in general. I’ve been surprised at how much I’ve learned about the low-level workings of Ubuntu over the past 9 months. I’m excited to see what I can learn over the next year, especially as I’ll likely move into a different codebase at some point this year.
  • More talking
    • I’m very excited to be giving a talk at FOSDEM this year, but I would enjoy doing such things more regularly. It doesn’t necessarily have to be at conferences, as I could do meetups much more easily. I need to try to get back into blogging more regularly. Additionally, I’ve recently been kicking around ideas for a discussion-based podcast on the worst parts of software development, although that may have already been done to death. Contact if interested.
  • Transition Web Tooling
    • I would like to switch over my analytics systems to use a personal Piwik instance, and I would love to replace the (hopefully unobtrusive) ads on this site with some kind of tip jar system. I would also like to update this blog to use a Let’s Encrypt certificate, as well as Ollert once I’ve been given full control.
  • Kegging
    • In my homebrewing, I’ve been bottling my beers. This is generally ok, but I think the beer would be consumed faster if I kegged it and could fill growlers for my friends and family. Getting started with kegging is expensive, requiring the purchase of kegs, tanks, parts, and some sort of refrigeration unit. By the end of the year, I intend to have a kegerator-style setup with the ability to stow and distribute from two kegs.
  • Moving
    • My wife is looking into graduate schools for Fall 2017, and has already been accepted by one. I’m currently assuming a big part of my life this Spring/Summer will be finding and adjusting to a new home.
  • Active Activism
    • I’ve complained a lot about our government and the way the world works on social media, at the “water cooler”, and privately to my wife, but it’s become obvious that passive activism isn’t good enough. Signing petitions and pledges are nice gestures, but are more meaningful when backed up by real action. I’d like to do something, though I’m not sure what at the moment. By the end of 2017, I would like to, at minimum, have a plan to donate, join, create, or generally be more involved.

Adieu, 2016

Major changes in my life, career, and the world at large have made 2016 a memorable year for me. I highly encourage you to reflect on the year you’ve had and think about what you can do to make 2017 great. Happy new year!

Confining a Snapped X11 Application

| Comments

Looking for the code? Look no further: https://github.com/larryprice/pingus-snap

In my last post, I demonstrated creating a snap package for an application available in the archive. I left that application unconfined, which is taboo in the long run if we want our system to be secure. In a few steps, we can add the necessary components to confine our pingus snap.

For reference, this is the original snapcraft.yaml file for creating a pingus snap, except that we’ve updated the confinement property to strict:

snapcraft.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
name: pingus
version: '0.1'
summary: Free Lemmings(TM) clone
description: |
    Pingus is a free clone of the popular Lemmings game.
    |
    Your goal is to guide a horde of penguins through a world full of obstacles
    and penguin traps to safety. Although penguins (unlike lemmings) are rather
    smart, they sometimes lack the necessary overview and now rely on you to
    save them.

grade: devel
confinement: strict

parts:
  archives:
    plugin: nil
    stage-packages:
      - pingus
  env:
    plugin: dump
    organize:
      pingus.wrapper: usr/bin/pingus

apps:
  pingus:
    command: pingus

If you’re feeling bold, you can build and install the snap from here, but be warned that this led me into an ncurses nightmare that I had to forcibly kill. That’s largely because pingus depends on X11, which is not available out-of-the-box once we’ve confined our snap. If we want to use X11, we’re going to need to connect to it using the snap-land concept of interfaces. Interfaces allow us to access shared resources and connections provided by the system or other snaps. There’s some terminology to grapple with here, but the bottom line is that a “slot” provides an interface which a “plug” connects to. You can see a big list of available interfaces with descriptions on the wiki. Our pingus app will “plug” into the X11 interface’s “slot”:

snapcraft.yaml
1
2
3
4
5
6
# ...
apps:
  pingus:
    command: pingus
    plugs:
      - x11

You can build and install the new snap with the --dangerous flag for your local confined snap. After that, you can verify the interface connection with the snap interfaces command:

1
2
3
4
5
6
7
8
9
$ snapcraft
$ sudo snap install --dangerous pingus_0.1_amd64.snap
pingus 0.1 installed
$ snap interfaces
Slot                     Plug
:alsa                    -
# ...
:upower-observe          -
:x11                     pingus

Now, when we run pingus… it works! Well, video works. If you want sound, we’ll also need the pulseaudio interface:

snapcraft.yaml
1
2
3
4
5
6
7
# ...
apps:
  pingus:
    command: pingus
    plugs:
      - x11
      - pulseaudio

Once again: build, install, and run… et voilà! Is it just me, or was that surprisingly painless? Of course, not all applications live such isolated lives. Make note that the x11 interface is supposed to be a transitional interface, meaning that we would rather our app fully transition to Mir or some alternative. To go a step further with this snap, we could create a snapcraft.yaml to build from source to get the absolute latest version of our app. At this point, we can change our grade property to stable and feel good about something that we could push to the store for review.

Any code you see here is free software. Find the project here: https://github.com/larryprice/pingus-snap

Building Snaps From Archived Packages

| Comments

If you haven’t heard, snaps are a new, modern packaging format made by the guys at Ubuntu. Snaps give every app a confined environment to live in, making desktops more secure and dependencies less of a hassle. One common way to create a snap is to simply use existing packages from the Ubuntu archives.

Let’s try to create a snap for the game pingus. pingus is a great little Lemmings clone that we can easily convert to a snap. We’ll start by installing the necessary dependencies for snap building (see the snapcraft website for more):

1
$ sudo apt install snapcraft

Now we can initialize a project directory with snapcraft:

1
2
$ mkdir -p pingus-snap && cd pingus-snap
$ snapcraft init

snapcraft init creates the following sample file to give us an idea of what we’ll need to provide.

snapcraft.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
name: my-snap-name # you probably want to 'snapcraft register <name>'
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil

Most of these values for our pingus snap should be obvious. The interesting markup here is in parts, which is where we’ll describe how to build our snap. We’ll start by taking advantage of the nil plugin to simply unpack the pingus deb from the archive. We define our list of debs to install in a list called stage-packages. We’ll also define another section, apps, to tell snapcraft what binaries we want to be able to execute. In our case, this will just be the pingus command. Here’s what my first draft looks like:

snapcraft.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
name: pingus
version: '0.1'
summary: Free Lemmings(TM) clone
description: |
    Pingus is a free clone of the popular Lemmings game.
    |
    Your goal is to guide a horde of penguins through a world full of obstacles
    and penguin traps to safety. Although penguins (unlike lemmings) are rather
    smart, they sometimes lack the necessary overview and now rely on you to
    save them.

grade: devel
confinement: devmode

parts:
  archives:
    plugin: nil
    stage-packages:
      - pingus

apps:
  pingus:
    command: usr/games/pingus

Nice, right? Building and installing our snap is easy:

1
2
3
$ snapcraft
$ sudo snap install --devmode pingus_0.1_amd64.snap
pingus 0.1 installed

We used devmode here because our app will be running unconfined (a topic for another blog post). Now, for the moment of truth! The snap tools automatically put our new app in PATH, so we can just run pingus:

1
2
$ pingus
/snap/pingus/x2/usr/games/pingus: 2: exec: /usr/lib/games/pingus/pingus: not found

¡Ay, caramba! We’ve run into a fairly common issue while snapping legacy software: hardcoded paths. Fortunately, the corresponding pingus executable is very simple. It’s trying to execute a command living in /usr/lib/games/pingus, which is not in our snap’s PATH. The easiest way to fix this is to fix the pingus executable. Since we don’t want to spend time modifying the upstream to use a relative path, we can create our own version of the pingus wrapper locally and copy it into our snap. The only change to this new wrapper will be prepending the snap’s install path $SNAP to the absolute paths:

pingus.wrapper
1
2
#!/bin/sh
exec $SNAP/usr/lib/games/pingus/pingus --datadir $SNAP/usr/share/games/pingus/data $@

Now we can update our yaml file with a new part called env which will use the dump plugin to copy our wrapper file into the snap. We’ll also update our command to call the wrapper:

snapcraft.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# ...

parts:
  archives:
    plugin: nil
    stage-packages:
      - pingus
  env:
    plugin: dump
    organize:
      pingus.wrapper: usr/bin/pingus

apps:
  pingus:
    command: pingus

When you run snapcraft this time, the env part will be built. After performing another install, you can run pingus, and you should be greeted with one of the best Lemmings clones available! Because we’re running unconfined in devmode, this all just works without any issues. I intend to write another blog post in the near future with the details on confining pingus, so look out for that soon. I may also go into detail on building more complex cases, such as building snaps from source and building custom plugins, or reviewing a case study such as the libertine snap.

For much, much more on snaps, be sure to visit snapcraft.io. If you’re looking for a published version of pingus as a snap, you can try sudo snap install --devmode --beta pingus-game, and you can run the game with pingus-game.pingus.

Source code available at https://github.com/larryprice/pingus-snap.

Clean Package Building With Pbuilder

| Comments

Whether I’m adding dependencies, updating package names, or creating new package spins, I always have issues testing my debian packages. Something will work locally, only to fail on jenkins under a clean environment. Fortunately, there’s a nifty tool called pbuilder that exists to help out in these situations. pbuilder uses a chroot to set up a clean environment to build packages, and can even be used to build packages for systems with architectures different from your own.

Note: All code samples were originally written from a machine running Ubuntu 16.10 64-bit. Your mileage may vary.

Clean builds for current distro

Given a typical debian-packaged project with a debian directory (control, rules, .install), you can use debuild to build a package from your local environment:

1
2
3
4
5
$ cd my-project
$ debuild
...
$ ls ../*.deb
my-project.deb

This works pretty well for sanity checks, but sometimes knowing your sane just isn’t quite enough. My development environment is filled with libraries and files installed in all kinds of weird ways and in all kinds of strange places, so there’s a good chance packages built successfully on my machine may not work on everyone’s machine. To solve this, I can install pbuilder and set up my first chroot:

1
2
3
4
$ # install pbuilder and its dependencies
$ sudo apt-get install pbuilder debootstrap devscripts
$ # create a chroot for your current distro with build-essential pre-installed
$ sudo pbuilder create --debootstrapopts --variant=buildd

Since I use debuild pretty frequently, I also rely on pdebuild which performs debuild inside of the clean chroot environment, temporarily installing the needed dependencies listed in the control file.

1
2
3
4
$ cd my-project
$ pdebuild
$ ls /var/cache/pbuilder/result/*.deb
my-project.deb

Alternatively, I could create the .dsc file and then use pbuilder to create the package from there:

1
2
3
4
5
6
7
8
$ # generate a dsc file however you like
$ cd my-project
$ bzr-builddeb -- -us -uc
$ cd ..
$ # use pbuilder to create package
$ sudo pbuilder build my-project.dsc
$ ls /var/cache/pbuilder/result/*.deb
my-project.deb

Clean cross builds

Let’s say that you need to build for an older distribution of Ubuntu on a weird architecture. For this example, let’s say vivid with armhf. We can use pbuilder-dist to verify and build our packages for other distros and architectures:

1
2
3
4
5
6
7
$ # create the chroot, once again with build-essential pre-installed
$ pbuilder-dist vivid armhf create --debootstrapopts --variant=buildd
$ # the above command could take a while, but once it's finished
$ # we can attempt to build our package using a .dsc file
$ pbuilder-dist vivid armhf build my-project-dsc
$ ls ~/pbuilder/vivid-armhf_result/*.deb
my-project.deb

Custom, persistent chroot changes

In some cases, you may need to enable other archives or install custom software in your chroot. In the case of our vivid-armhf chroot, let’s add the stable-overlay ppa which updates the outdated vivid with some more modern versions of packages.

1
2
3
4
5
6
7
8
9
$ # login to our vivid-armhf chroot, and save state when we're finished
$ # if --save-after-login is omitted, a throwaway chroot will be used
$ pbuilder vivid armhf login --save-after-login
(chroot) $ # install the package container add-apt-repository for convenience
(chroot) $ apt install software-properties-common
(chroot) $ add-apt-repository ppa:ci-train-ppa-service/stable-phone-overlay
(chroot) $ exit
$ # update packages in the chroot
$ pbuilder-dist vivid armhf update

pbuilder and chroots are powerful tools in the world of packaging and beyond. There are scripting utilities, as well as pre- and post-build hooks which can customize your builds. There are ways to speed up clean builds using local caches or other “cheats”. You could use the throwaway terminal abilities to create and destroy tiny worlds as you please. All of this is very similar to the utility which comes from using docker and lxc, though the underlying “container” is quite a bit different. Using pbuilder seems to have a much lower threshold for setup, so I prefer it over docker for clean build environments, but I believe docker/lxc to be the better tool for managing the creation of consistent virtual environments.

Further reading:

Pbuilder HowTo on the Ubuntu wiki Pbuilder tricks from the debian wiki