Category: Programming

Creating Flat Packages in OSX

I was recently involved with a daemon-style Java project that needed a simple installer solution for OSX. We didn’t require a huge amount of sophistication from the installer. So we opted for flat packages. Flat packages are simple, self-contained files that are easy to pass around.

Prior to Leopard (10.5), packages weren’t flat at all. They were bundles, which are just folders that OSX treats in a special way. Bundles contain a variety of files. Package bundles, for example, contain files used as part of the install process such as a bill of materials, the application payload, and a collection of localized resources among other things. Previously, if you wanted to distribute one of these packages, you’d have to put the package inside a DMG or tar it up.

Another interesting thing happened during the evolution of OSX packages. PackageMaker, a wart in the eyes of many developers and source control systems, was officially deprecated and dropped in Xcode 4.6 (and there was much rejoicing :-)). Developers were left with three terminal-based utilities that could be used to build packages – pkgbuild, productbuild, and pkgutil.

For more information on the evolution of OSX packages, I highly recommend reading this MacTech article which goes into the subject in some depth.

pkgbuild

pkgbuild is a tool for creating a very basic, bare-bones component package. The resulting package will contain the payload for a single component. It also supports preinstall/postinstall scripts. And that’s about it.

productbuild

productbuild is a tool for creating a “product archive”, which is kind of like a super-package. I like to think of it as a package of packages. But it’s really so much more. You can customize the UI, as well as explicitly declare installation prerequisites such as the minimum required RAM, OS version, etc. Even JavaScript is supported for some things. If you create apps for the Mac App store, this is the tool you’ll be using to build your packages.

pkgutil

pkgutil is kind of like a Swiss Army knife for packages. With pkgutil, you can rip a package apart and then put it back together again. It doesn’t matter if it’s a product archive or a component package. If it’s a pkg file, pkgutil can work with it. It also provides a number of options for examining package contents and performing tasks like package verification and repair.

I’m going to give a quick overview on how I use these three utilities to build the flat package for our product. This approach can be adapted for your own needs.

Simple Component Packages

If you want to create a very simple package for a single component application, the easiest tool for the job is pkgutil. A complete list of options for pkgutil can be found on the man page. But here are the ones that are important.

--root – This option specifies the location of the source content you want packaged up. This can be a path containing anything. Our own application is Java-based and is nothing more than a folder of JAR files. We simply point --root to the folder containing all of our JAR artifacts.

--component – You can use this option as an alternative to --root. With --component, you’re being a bit more specific by giving pkgbuild the path to a component bundle. This might be an app bundle, a browser plugin bundle, etc. Why would you use this option instead of --root? If pkgbuild knows it’s working with a bundle, it can infer some of the other configuration options it needs, such as package identifier and install location.

--identifier – The identifier is a unique string that identifies your component. The normal convention for the identifier resembles something like a backwards domain. You start with a TLD, then follow that with a company name or developer name, and then follow that with a product name. So an identifier for Roadie Assist (not an iOS or OSX app, but could be someday) might look like “com.shanekirk.roadieassist” (this is indeed the package name on the Google Play store). There’s certainly flexibility in how you name your package IDs. They just need to be unique. But don’t deviate too much from the convention. It’ll only cause confusion.

--version – This is a monotonically increasing number. The idea is that every time you release a new version of the component, you need to increase this value. Otherwise, the upgrade process may fail. If this value isn’t set, pkgutil defaults with a value to 0. (Tip: If your build system keeps track of build numbers, consider using that as your package version. If the whole thing is automated with scripts, you’ll probably never have to think too much about this value again.)

--install-location – This is where you want to place the component on the target system. For instance, if you’ve built an application bundle, you’ll probably want to place it in /Applications on the target machine.

--scripts – Packages support a number of scripts that are invoked during the install process. If you have a script called preinstall, this will be invoked before the component is installed. If you have a script called postinstall, it will be invoked after the component is installed. The --scripts option specifies where you’re storing these scripts. The scripts can be in any language so long as the files are marked executable and have the appropriate shebang indicating the path to the interpreter. If either of the scripts return a number other than 0, Installer assumes a failure occurred and the installation halts.

A typical invocation of pkgbuild might look like so…

pkgbuild --root ./stagedfiles --scripts ./scripts --identifier com.example.myapp --version 1 --install-location /Applications mypackage.pkg

For many folks, this may be all that’s needed. From within Finder, you can double click the resulting package and component can be installed without having to do much else. If you would like to customize the UI a bit, allow for a more “componentized” installation, or distribute your application through the Mac App store, you’ll need to use productbuild.

Product Archives with productbuild

Much of what you can do with pkgbuild you can also do with productbuild. But don’t be fooled. Product archives and component packages aren’t quite the same thing. Synthesizing a distribution from a product archive (package created with productbuild), for example, will fail, but doing the same thing with a component package (package created with pkgbuild) works just fine (have no idea what I just said? Don’t worry. You will soon.) With productbuild, you can also customize quite a bit of what the end user sees. If you peruse the man page for productbuild, you’ll find that many of the command-line options you used with pkgbuild are there for productbuild as well. But you’ll also see a few other things mentioned like plugins, distribution files, resource folders, etc.

Regardless of how you created your package, you’ll probably want to update the UI or perform more sophisticated prerequisites checking. To do this you’ll need to create a distribution file.

A distribution file is just an XML file. It’s used to configure both look and feel, as well as behavior, of the package installer. The idea is that you create this file and then use productbuild to create a new package from the old one and the new distribution file. Confused yet? Let’s work through an example.

productbuild example

Let’s assume we have a package called mypackage.pkg kicking around (if you’re reading closely, you’ll notice this is the same package I use as an example above while discussing pkgbuild.). Our goal is to change the title and background image shown to the user. So we need to create a distribution file and modify it to suit our needs. Fortunately for us, productbuild is able to synthesize a skeleton distribution file for us.

productbuild --synthesize --package ./mypackage.pkg distribution.xml

Note that this only works against component packages created with pkgbuild. If you try to do this against a pkg created by productbuild, it’ll result in an error. The resulting XML will look something like the following.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<installer-gui-script minSpecVersion="1">
    <pkg-ref id="com.example.myapp"/>
    <options customize="never" require-scripts="false"/>
    <choices-outline>
        <line choice="default">
            <line choice="com.example.myapp"/>
        </line>
    </choices-outline>
    <choice id="default"/>
    <choice id="com.example.myapp" visible="false">
        <pkg-ref id="com.example.myapp"/>
    </choice>
    <pkg-ref id="com.example.myapp" version="1" onConclusion="none">mypackage.pkg</pkg-ref>
</installer-gui-script>

So far, so good. Let’s first change our package title. We want the title to be “My Example Application”. Open the distribution XML with your favorite text editor. Within the installer-gui-script element, create a new child element called “title” and set the element’s character data appropriately. It should look something like this.

<title>My Example Application</title>

Now let’s create a new background image for our installer. The recommended size for the background image is 620×418. The background image I’ll be using for this is example is a simple PNG as shown below.

Create a new folder for your installer resources and place the image in there. We’ll call it the file background.png.

Now, let’s add the PNG reference to the distribution.xml. Open the XML file and within the root installer-gui-script element, create a new element called background. Add two attributes, one called “file” that simply names the file and one called “mime-type” that describes the image type. For example, your background element might look like so.

<background file="background.png" mime-type="image/png" />

Save your changes. Now we just need to create a new package with our modified distribution file. Using productbuild the command might look like so.

productbuild --distribution distribution.xml --resources ./myimages --package-path ./mypackage.pkg myimprovedpackage.pkg

This command produces a new package called myimprovedpackage.pkg that has the new modified background image. If you double click on it with Finder, you should see your new background image, but it should also install the new package just as it did the old one.

For a complete list of everything you can do to a distribution.xml, check out the distribution xml schema reference. You’ll find that there’s a lot of flexibility in what you can do to your packages. There are some things I always do, such as customize the welcome, license, readme pages, run JavaScript as part of the startup validation, and limit install locations to the root volume. But I’ll save discussion of those types of things for a future blog post.

Injecting Files Into Flat Packages

At some point in your journey with flat packages, you’ll find that you can’t get something into the package file that you really think should be there. This might be an extra script or even an Installer plugin (a topic that deserves its own blog entry). Both pkgbuild and productbuild promise to copy content out of the scripts or resources folders, but sometimes they just don’t. So we need to get these files into our package. That’s where pkgutil comes in.

pkgutil allows you to expand a package to a folder structure like so.

pkgutil --expand myimprovedpackage.pkg expandedcontent

This creates a folder called expandedcontent that contains the expanded content of the package. If you explore the folder structure, most of what you see will only make sense if you’re familiar with the structure of the original package bundles. I strongly advise you to read up on the history of the package format I mentioned in the beginning to be effective at shuffling stuff around within these folders.

Once you’ve got things where they need to be, then you can flatten everything again like so.

pkgutil --flatten expandedcontent myimprovedpackage.pkg

Conclusion

And that’s it. I realize I glossed over a lot of interesting stuff with regards to the distribution XML file. Experiment with it. I hope to write a followup to expand on some of these topics in closer detail. If you have questions, post them!

Introducing Roadie Assist

The Wee Lollies played shows for the better part of 2012. And there are few signs that we’ll be slowing things down for 2013. For all of last year, when it came to carting around all of my gear I frequently found myself in checklist hell. Every time we’d have a rehearsal or play a show, I’d tear off a little slip of paper and construct a gear checklist based on whatever old checklists happened to be littering my desk (and there were many!). I’d consult the list as I loaded all of my gear before a show to make sure I had everything. And then I’d consult it again after the show to make sure I didn’t leave anything behind. Many shows required different setups. And sometimes old gear was retired and new gear was added. So the checklists weren’t always the same. But as you might imagine, they were similar.

In the fall of last year, I got the idea for a checklist app for gigging musicians. I wanted to ditch the Post It notes and use something a bit more structured. I wanted something that’d allow me to record the use of gear, keep track of lost items, and be better organized when it came to keeping track of shows. Granted, there are already a LOT of general purpose Android checklist apps on the market. But I felt like there really needed to be an app specifically targeted for gigging musicians. So I decided to build one.

Thus Roadie Assist was born.

Roadie Assist is a fairly simple app. It allows you to supply information for all of your gear and upcoming gigs and then lets you associate that gear with gigs. Each gig has a load-in and load-out checklist that let’s you ensure you leave what you came with. If you lose an item, Roadie Assist will keep track of that too. If you supply address and contact information for a gig, Roadie Assist will allow you to navigate, email, or dial a number right from the app (assuming the device supports any of those features). And for quick gig creation, Roadie Assist features templates that allow you to create predefined gear lists that can be used as starting points for new gigs. I often use the template feature to differentiate between acoustic-only shows and shows where I need to haul in my electric guitars and effects.

If Roadie Assist interests you, pop over to the Google Play store and install it to your Android device. It’s only 99 cents. And if you find bugs or have ideas for “must-have” features, be sure to let me know!

-Shane

AsyncTask Missteps

AsyncTask comes up in conversation quite often around the office, usually because of a failed code review or some related bug that’s been discovered. It’s one of the most well understood classes on the Android platform, and yet it’s also one of the hardest classes to use correctly. Even advanced Android developers get it wrong.

In the following discussion, we’ll take a look at a few snippets of code, touch on some of their problems, and arrive (hopefully) at a pattern of AsyncTask usage that’s suitable for most cases.

Let’s start with a fairly typical example.

public class MyActivity extends Activity 
{
    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
 
        // Find views and assign them to member variables.
 
        new AsyncTask<void, void,="" string="">()
        {            
            @Override
            protected String doInBackground(Void... params)
            {
                // Do some long running task.
                return result;
            }
 
            @Override
            protected void onPostExecute(String result)
            {
                // Update UI
            }
        }.execute();
    }
    // Various View member variables.
}
</void,>

This seems fairly straightforward. When our Activity is created, we launch an AsyncTask to perform some background work, which is performed in doInBackground. When the background work is finished, doInBackground returns a String which is used by onPostExecute to safely update the UI.

We’re using the AsyncTask here exactly as I’ve seen it used in production code. UI elements are being updated safely. Work is being done in the background. We’re in no danger of an ANR message. So what exactly is the problem here?

The problem isn’t what happens inside the AsyncTask. The problem is what happens outside of it. Specifically, what happens if the Activity is destroyed while the AsyncTask is running?

Imagine this scenario – the Activity is created, the AsyncTask is instantiated and executed, and then the screen orientation changes. When screen orientation changes under normal circumstances, the original Activity is destroyed and a new one is created. This means another AsyncTask will be instantiated and the background work will start all over again.

But that’s not the worst of it. The original AsyncTask will continue to run until it’s finished. And once it’s finished, it’ll update UI elements that aren’t visible anymore. And because the original AsyncTask still has an implicit outerclass pointer to the original Activity, the original Activity can’t be garbage collected until the original AsyncTask is finished. Change the screen orientation a bunch of times on a resource intensive Activity and you risk running out of memory.

There’s also a limit on the number of AsyncTasks you can have running at once. For Activities with smaller footprints, you’ll most likely run out of AsyncTasks before you start to see memory problems appear.

Does that make you uncomfortable? It certainly should.

So how do we work around these issues? Let’s address these one at a time, starting with the simplest – the implicit pointer to the outer class. This can be solved by making the AsyncTask class static, which also means it can no longer be anonymous.

Removing Implicit Outerclass Pointers

public class MyActivity extends Activity 
{
    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
 
        // Find views and assign them to member variables.
 
        MyTask task = new MyTask();
        task.execute();
    }
 
    static class MyTask extends AsyncTask<void, void,="" string="">
    {
        @Override
        protected String doInBackground(Void... params)
        {
            // Do some long running task.            
            return result;
        }
 
        @Override
        protected void onPostExecute(String result)
        {
            // Need to update UI. But how?
        }
    }    
    // Various View member variables.
}
</void,>

So far so good. Our AsyncTask will no longer keep a reference to the Activity. But now we can’t update the UI when the task completes, which kind of defeats the purpose. How do we fix this? By giving back an Activity reference to the AsyncTask. “Wait, isn’t that what we tried to eliminate by making the class static?” Well, yes. But we wanted to remove the implicit reference, which is something we couldn’t change at runtime. With something a bit more explicit, we have direct control over when the reference comes and goes.

public class MyActivity extends Activity 
{
    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
 
        // Find views and assign them to member variables.
 
        m_task = new MyTask();
        m_task.m_activity = this;
        m_task.execute();
    }
 
    @Override
    public void onDestroy()
    {
        super.onDestroy();
        m_task.m_activity = null;
    }
 
    static class MyTask extends AsyncTask<void, void,="" string="">
    {
        @Override
        protected String doInBackground(Void... params)
        {
            // Do some long running task.            
            return result;
        }
 
        @Override
        protected void onPostExecute(String result)
        {
            if (m_activity != null)
            {
                // Update UI
            }
        }
 
        MyActivity m_activity = null;
    }
 
    private MyTask m_task = null;    
    // Various View member variables.
}
</void,>

That’s better. In the above example we remove the Activity reference in the AsyncTask when the Activity is destroyed. The AsyncTask can continue to run without preventing our Activity from being garbage collected.

Concurrent AsyncTask Limitations

Now let’s briefly discuss the problem of concurrent AsyncTasks. If you weren’t aware that there’s an upper limit to concurrent AsyncTasks, Google “AsyncTask limits” and you’ll find plenty of information about it. We need to control the lifetime of our AsyncTask instance. What that boils down to is canceling the task and letting the garbage collector deal with it. Canceling a task is a two-step process. First, we set the AsyncTask’s cancelled flag in the Activity’s onDestroy method.

m_task.cancel(false);

This alone doesn’t actually cancel the task. It has no effect on the background process if the task chooses to ignore it. So we also need to regularly check the return value of isCancelled() inside of our AsyncTask’s doInBackground method and bail out appropriately. This allows us to terminate that background process cleanly.

if (isCancelled())
{
    return null;
}

Notice that we passed false to the AsyncTask’s cancel method. If you pass true, it interrupts the thread and can leave things in a particular awkward state. See the Thread class’s interrupt method for details.

One AsyncTask To Rule Them All

We still have a problem of a new AsyncTask being created and executed every time our Activity is destroyed and recreated. In most cases, creating a new AsyncTask instance is unnecessary. We often only need the AsyncTask to query a database, make a web service request, or read some file once. So it’d be nice if we could leverage a single AsyncTask no matter how many times our Activity is temporarily destroyed/created.

I’ve seen some folks set the android:configChanges attribute in the Activity’s XML declaration inside the app manifest just to avoid the redundant AsyncTask issue. And while it’s certainly doable, it’s somewhat of an obscure solution. android:configChanges is meant to solve other problems. That approach may confuse other developers who will need to maintain your code. It also makes the UI a bit inflexible because the default behavior may actually be behavior you want.

I’ve also seen some folks address the issue using invisible Fragments. Again, it works. But it’s somewhat obscure if you’re using a Fragment oriented approach just to solve this issue. Make sure your Fragment usage is appropriate for your UI design. Never use a sledgehammer when a rubber mallet will do.

The approach I like best is mentioned in this StackOverflow thread and in Hervé Guihot’s book “Pro Android Apps Performance Optimization”.

public class MyActivity extends Activity 
{
    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
 
        // Find views and assign them to member variables.
 
        m_task = (MyTask) getLastNonConfigurationInstance();
        if (m_task != null)
        {
            m_task.m_activity = this;
        }
        else
        {
            m_task = new MyTask();
            m_task.m_activity = this;
            m_task.execute();
        }
    }
 
    @Override
    public void onDestroy()
    {
        super.onDestroy();
 
        m_task.m_activity = null;
 
        if (this.isFinishing())
            m_task.cancel(false);
    }
 
    @Override
    public Object onRetainNonConfigurationInstance() 
    {
        return m_task;
    }
 
 
    static class MyTask extends AsyncTask<void, void,="" string="">
    {
        @Override
        protected String doInBackground(Void... params)
        {
            // Do some long running task. We need to make sure
            // we peridically check the return value of isCancelled().
            return result;
        }
 
        @Override
        protected void onPostExecute(String result)
        {
            if (m_activity != null)
            {
                // Update UI
            }
        }
 
        MyActivity m_activity = null;
    }
 
    private MyTask m_task = null;
 
    // Various View member variables.
}
</void,>

The key piece here is the use of the onRetainNonConfigurationInstance and getLastNonConfigurationInstance methods to reuse an AsyncTask instance between build up and tear down of the Activity. Using this approach will always result in a single AsyncTask being created and used.

My code above differs a bit from that mentioned in the StackOverflow discussion and in Guihot’s book. One notable difference is that I check to see if the Activity is actually finishing in the onDestroy method. If it is, we cancel the task. isFinishing will return true if this Activity is being dismissed as a result of going backwards in the Activity stack.

Our Final Approach

There is still a bug here. Can you spot it? What happens if the Activity has been destroyed and recreated after the AsyncTask has completed? Our UI doesn’t get updated. The solution to this is fairly simple and illustrated in the final version of our Activity below.

public class MyActivity extends Activity 
{
    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
 
        // Find views and assign them to member variables.
 
        m_task = (MyTask) getLastNonConfigurationInstance();
        if (m_task != null)
        {
            m_task.m_activity = this;
            if (m_task.m_isFinished)
                m_task.updateUI();
        }
        else
        {
            m_task = new MyTask();
            m_task.m_activity = this;
            m_task.execute();
        }
    }
 
    @Override
    public void onDestroy()
    {
        super.onDestroy();
 
        m_task.m_activity = null;
 
        if (this.isFinishing())
            m_task.cancel(false);
    }
 
    @Override
    public Object onRetainNonConfigurationInstance() 
    {
        return m_task;
    }
 
 
    static class MyTask extends AsyncTask<void, void,="" string="">
    {
        @Override
        protected String doInBackground(Void... params)
        {
            // Do some long running task. We need to make sure
            // we peridically check the return value of isCancelled().
            return result;
        }
 
        @Override
        protected void onPostExecute(String result)
        {
            m_result = result;
            m_isFinished = true;
            updateUI();
        }
 
        public void updateUI()
        {
            if (m_activity != null)
            {
                // Update UI using m_result
            }            
        }
 
        // These should never be accessed from within doInBackground()
        MyActivity m_activity = null;
        boolean m_isFinished  = false;
        String m_result = null;
    }
 
    private MyTask m_task = null;
 
    // Various View member variables.
}
</void,>

I don’t claim that this pattern is a one-sized fits all solution. But I think it covers most cases quite well. There are a few things I avoided discussing here, such as dealing with progress dialogs and how to design tasks that can be used by multiple activities. I didn’t want to add too much complexity to the examples. And I think those sorts of things should be fairly intuitive once you’ve achieved a certain degree of comfort with AsyncTask