Leveraging Swift Code Coverage with Xcode 7

French speakers: the original version of this article is actually French, and you can read it on Xebia’s excellent blog.

Also, this article is the text version of the talk “Leveraging Xcode Code Coverage”, presented at the Mobile Optimized conference in Minsk, whose slides are available here.


One of the WWDC 2015 announcements that surprised interested us the most has definitely been the support for code coverage for the Swift language.
In this article we will understand the advantages of the new code coverage functionality introduced in Xcode 7 and how to integrate such KPI in our daily work.


Code coverage is a metric that measures the value of our tests by identifying what code is executed when running them and, above all, what portions of our project are untested.

How it works?

The production of code coverage information is done in two passes:

  1. At compile time, the compiler prepares the files for analysis
  2. At runtime, the lines of code affected by the tests are annotated in a specific file

Xcode code coverage before June 2015

Before WWDC 2015, only the Objective-C code coverage was supported by Apple’s tools, while Swift had been left behind. Also, the Objective-C support was sometimes inconsistent and required a few tricks to get the information.

How did it work?

The procedure necessary to retrieve the information was a variant of the one used by gcov, included in the gcc tools. Two settings had to be added to the Build Settings:

  1. Generate test coverage files, which corresponds to the -ftest-coverage gcc flag
  2. Instrument program flow, corresponding to -fprofile-arcs

The former allows the creation of the .gcno files, which contain the information needed to build the execution graph and reconstruct the line numbers.

The latter, Instrument program flow, deals with the creation oft the .gcda files, which contain the number of transitions on different arcs of the graph and other summary information.

In order to force the generation of these command line data, it was possible to use the following command:

Exploiting the data

A number of tools for reading and exporting reports exist, but in particular, we used the following:

  • Coverstory, a GUI tool to read files “.gcda” and “.gcno”
  • gcovr translates these files Cobertura XML format
  • lcov generates a visual report in a navigable HTML
  • and, especially, Slather.

Slather

Slather, developed by the SF-based company Venmo, exports the code coverage data in a number of different formats, including Gutter JSON, Cobertura XML, HTML and plain text. In addition, it integrates easily with other platforms, such as Jenkins, Travis CI, Circle CI and coveralls.

As said, one of the major assets of Slather is its ease of configuration and integration within a continuous integration system. Slather is open source, and available here at Venmo’s GitHub repo.

Swiftcov

A recent tool for collecting Swift coverage information is Swiftcov, developed by the guys at Realm. Swiftcov makes heavy use of LLDB breakpoints to detect which lines are affected by the execution of our tests.

Code coverage after June 2015

During the WWDC 2015 keynote, Apple announced that Xcode 7 would introduce support of code coverage for our beloved Swift.

How does it work?

A completely new format has been introduced, named profdata, thus making
gcov legacy – at least for what concerns projects developed with Apple’s development tools.

In other words, starting from the very first beta of Xcode 7, profdata is intended to replace completely gcov for both Swift and Objective-C.

In order to enable the setting, from Xcode 7, you will need to access the “scheme” setting and, in the “Test” tab, to tick the “Gather coverage data” checkbox.

Access the scheme editor, select the "Test" tab and enable the "Gather coverage data" checkbox

As for the command line, xcodebuild now ships with a new parameter, -enableCodeCoverage, which can be used as follows:

Once the tests run, coverage information is immediately available in Xcode, on the right side of the code editor (see image below) and, in particular, in the “Report Navigator”.

Inline code coverage information
Inline code coverage information
Code coverage information in the Report Navigator
The new Report Navigator

The Report Navigator shows in detail which classes are covered by our tests and, by expanding the selection, which methods are actually used.

Exploiting the data

Apple’s work hasn’t only consisted in enhancing Xcode but, also, in extending the features of the   llvm-cov command line tool, which allows working with .profdata format.

The llvm-cov show command, for instance, allows exporting plain text coverage information and outputs annotated source code files, which can be easily read and processed.

Slather (returns)

A recent Pull Request allows Slather to work with profdata files and convert them to other formats, thus enabling the integration with the other platforms supported by the tool.

Xcode Server

If you are thinking about setting up an automated integration system, aside from the excellent Jenkins, Travis or Circle CI, it is perhaps time to start taking into consideration Xcode Server, which is part of the OS X Server bundle, distributed free of charge by Apple.

With the new version of Xcode bots and Xcode Server, it is now possible to support code coverage values ​​and to display the results in a Web browser. The reports are also available in the “Big Screen” presentation, useful for presenting your content in a simplified yet effective overview.

In order to enable this workflow, you could follow the steps below:

  1. Install OS X server
  2. Enable “Xcode” and “Websites” services
  3. Create a new project and assign a Source Control Manager to it (such as git)
  4. In Xcode, create an Xcode bot under “Product > Create Bot”
  5. Select the frequency of integration and enable code coverage (see image below) next to the caption “Code Coverage”.
  6. Launch an integration
  7. Open the web browser at the host indicated by your instance of Mac OS.
Xcode Bot creation panel
Xcode Bot creation panel

Conclusion

Code coverage is very useful to keep under control your code base health status. Although it can not replace your developer confidence in well designed, well structured apps, this metric can help write better code by encouraging you to give yourself concrete goals day by day.

Also, and finally, the new tools offered by Apple can now allow you to keep under control these values ​​in minutes, with a simple and immediate configuration.

Carthage

Yesterday I talked about Carthage at CocoaHeads Paris. Here are the slides of my presentation, in French: https://speakerdeck.com/viteinfinite/carthage.

Even if I am not involved in the project, I believe Carthage has some great potential, I really like the minimalist approach and, in particular the fact that developers can keep control over what really happens when you add an external dependency.

The idea of using committed xcodeprojs to retrieve informations about the build is quite good, even if it obviously requires to have a shared scheme – which it’s not the case for the majority of project so far.

I’m looking forward to see more and more libraries supporting Carthage though even if for now, at least for client work, I’ll stick with CocoaPods.

 

Some opinions about Calabash-iOS

For what concerns iOS my colleagues and I have been using Calabash-iOS for a year now, with mixed feelings.
Here is a totally subjective opinion about Pros and Cons of Calabash-iOS

Pros:

  • Conciseness of the Gherkin language
  • Capability of querying webviews with CSS selectors
  • Access to all the object property values via Ruby
  • built-in Jenkins-Ready output (xml and HTML test reports, mainly)
  • Performance (on <= iOS 6)
  • It does not uniquely rely on accessibilityIdentifiers (as KIF does)
  • Actively mantained

Cons:

  • It’s not an Apple-backed project : functionalities change significantly from OS to OS
  • Test fail randomly under certain circumstances and, in particular, when dealing with repeated scrolling on a UIScrollView (due to a poor implementation of the scroll/swipe functions)
  • On iOS 7, it relies on UIAutomation, thus…
  • …terrible performance on iOS 7 (see above)
  • Once again, quite a Pain-in-the-ass to make it work on iOS 7
  • Finding the good query for your element usually requires much trial-and-error via the calabash-ios console (Frank provides a nice UI tool for that task, but it’s not merged into Calabash-iOS yet)

Truth to be told, at this stage, the performance impact of relying on UIAutomation is deal breaker for me, so, on the next project we’ll be using KIF, which appears to be also used in some Google projects.

How to resolve “Unable to start status bar server” when launching GHUnit Tests from command line after upgrading to XCode 5

After upgrading to XCode 5 my Jenkins Continuous Integration machine over at Xebia stopped executing command-line GHUnit tests under some apparently random conditions. The console output was as follows:

The reason of the exception looks pretty much the same I had to deal with some months ago (and solved in a previous post).

Well, it turns out that after the XCode upgrade, after quitting an iOS 5.x or iOS 6.x simulator instance, the SpringBoard daemon (along with many others) does not get removed, thus preventing our test target to instantiate a status bar. Interestingly enough, this behaviour does not occur when quitting an iOS 7.0 simulator.

That said, in order to fix the issue, I added the following line to the RunTests.sh script:

The complete RunTests.sh script is available at this public gist.

Introducing SMWheelControl

Today I am proud to announce the availability of a new Pod for iOS which will be included in my forthcoming app.

SMWheelControl

SMWheelControl is an iOS control allowing the selection of an item from a 360° spinning wheel with a smooth inertial rotation.

The code, released under the BSD 3-clause license, is loosely based on the tutorial “How To Create a Rotating Wheel Control with UIKit” published on this Ray Wenderlich’s post by Cesare Rocchi

As usual, you can insert it into your podfile by adding

Some howto follows.

Usage

Initialization and data source

Instantiate the control with a classical - (id)initWithFrame:(CGRect)rect and add a target as you usually do with a control, e.g.:

Then add a dataSource:

and implement the following methods (the dataSource should conform to the SMWheelControlDataSource):

For instance:

When the wheel ends snapping to the closest slice, if you added a target, then it will receive the event UIControlEventValueChanged, and the selector you specified will be executed.

About the Builder Pattern in Objective-C

When dealing with complex object creation in an iOS application you may feel the need for a cleaner and more fluent approach.

During my latest development my colleague @alexiskinsella set up a pattern based on providers in order to mimic the factory behaviour we often find in languages such as Java.

Recently, in the occasion of a side project, I am studying a slightly different approach, based on this article by Matthias Wessendorf.

What I am willing to do is an image builder which creates UIImages on which a GPUImageFilter is applied.

The result I came to is as follows:

The Builder itself (implementation)

The Configuration object (header)

 Using a Builder with an inline Configuration

This should make for a quite readable code with a reduced number of lines of code in the method that invokes the builder.

How to add custom environment variables in Xcode and CocoaPods

Let’s say we would like to share some Xcode environment variables across multiple targets in a project configured with CocoaPods. For instance, we would to use the same app version number across our development and distribution targets in order not to retype the same values for each version bump over and over again.

The solution I opted for was to:

1. Add a custom xcconfig file (for instance “SharedSettings.xcconfig”) from the Xcode menu

Screen Shot 2013-05-21 at 12.16.11 PM

2. In the project settings, change the configuration settings file to “SharedSettings”

Screen Shot 2013-05-21 at 12.18.43 PM

3. In settings file, in order to preserve your pod configuration, add

#include "Pods/Pods.xcconfig"
VERSION_NUMBER = 3.0

And it’s done: now you can use the environment variable $(VERSION_NUMBER), even in your plist file, without breaking the compatibility with CocoaPods. As a result, any future “pod install” will not break the configuration you have just created.

How to resolve “Unable to start status bar server. Failed to check into com.apple.UIKit.statusbarserver: unknown error code” exception

Ever wondered how to resolve the following error while executing GHUnitTest by Command Line?

*** Assertion failure in -[UIStatusBarServerThread main], /SourceCache/UIKit_Sim/UIKit-2372/UIStatusBarServer.m:96
*** Terminating app due to uncaught exception ‘NSInternalInconsistencyException’, reason: ‘Unable to start status bar server. Failed to check into com.apple.UIKit.statusbarserver: unknown error code’

Well, I did.

In my situation, the exception was caused by having an iPhone Simulator session opened while executing the CLI tests. So, in order to resolve the issue, I simply had to add

killall -s "iPhone Simulator" &> /dev/null
if [ $? -eq 0 ]; then
killall -m -KILL "iPhone Simulator"
fi

on top of the run script file (RunTests.sh) which is executed as last “build phase” of the build process.

EDIT: I wrote another post explaining how to solve the error after migrating to XCode 5.