heikomaass – Keep on learning…

Offline-Sync architecture

Syncing data for offline usage is simple. The native app only needs to download a snapshot of data which can be queried when the device is offline. But syncing back newly created and modified entities to a cloud based backend is really, really tricky.

In a current project we’ve found out that the following architectual rules lead to a clear implementation.

All your business is local

The whole app has to work locally. Each controller and service only reads and modifies local data. There are two exceptions: A dedicated sync service and an authentication service which handles the session token. Only these two services may communicate with the backend.

Client based identity generation

The unique identity of an entity will be generated by the app using an UUID generator. The backend will use this UUID as primary key for the received entity as well. The big advantage of this approach is simplicity: You don’t have to reassign entity ids after the sync process has completed.

Track changes using commands

If data is changed the local representation of the affected entity will be modified. In addition the responsible change command will be tracked in a separate database table. This pattern is called command sourcing. When the sync service is requested to sync back the data to the backend, it will iterate over the command table and send all the stored commands in the order of their appearance.
The backend should perform suitable validation of the command data, because the client request may not be trustworthy. In contrast to an event, a command might be rejected.

Quite simple, but the real interesting point is conflict resolution. This will be the topic for my next blog post.

Repositories with Realm

Realm 0.87 (for Android) came with a very important change: It introduced detached copies of RealmObjects. You can create a detached object using Realm.copyFromRealm. Any setter operation on a detached object won’t update the database so they can be called without opening a Realm transaction. In addition the detached objects are accessible from any thread.

We refactored our app code to use repositories and setup the following rules:

The UI only fetches realm objects via a repository (no more Realm code in Activities).
Repositories always return detached objects
Detached objects have to be saved using the save method in the repository.

Here’s a simple example repository:

public FooRepository {
 
    private Realm realm;
    
    public FooRepository(Realm realm) {
       this.realm = realm;  
    }
 
    public Foo getFooById(String fooId) {
        Foo foo = realm
                 .where(Foo.class)
                 .equalTo("id",fooId)
                 .findFirst();
        return realm.copyFromRealm(foo);
    }
    
    public void save(Foo detachedObjectToSave) {
       // isValid() always returns false on detached objects.
       if (object.isValid()) { 
          throw new IllegalArgumentException("received realm object is not a detached object");
       }

       realm.beginTransaction();
       realm.copyToRealmOrUpdate(object);
       realm.commitTransaction();
       // please note that this also could be saved 
       // in background using Realms asynchronous transactions.
    }
}

Apple Music evaluated

Since the start of Google Music in Germany I’m using this service and I’m quite happy with it. Last week Apple Music has arrived and I was eager to evaluate Apple’s interpretation of a streaming service.

In short, I’m disappointed regarding the usability of Apple Music. Here are my main complaints:

There is no listing history or “recent activity”, so you have to put songs to a playlist or add them to “My Music”.
If you are listing to a song, the link to its corresponding album is not obvious. You have to tap on the track summary in the context menu. But if you tap on the track summary in the main window a rating view appears. So tapping on the same content result into different actions.
The pause button is placed far too close to the “For you” tab button. So I accidentally stop the music when I want to visit the “For you” section.
Using the Offline Mode is hidden and overly complex: You have to activate the iCloud Music Libary in the Settings App. I had to google this, there was no hint in the app itself.
Last but not least the app crashed two times in the first ten minutes I’ve used it on my iPad Air. First impressions count.

By now I will stay at Google Music, it is a more mature service, but still I will evaluate the upcoming Apple Music Android App.

Bluetooth LE – GATT profile

Almost 9 years ago, I wrote a series of blog posts about Bluetooth and the J2ME based “Java APIs for Bluetooth Wireless Technology” (JABWT). JABWT and J2ME haven’t survived the advent of the mobile age, but Bluetooth has evolved quite well: The 2010 introduced Low Energy standard highly optimizes battery performance by sacrifing data throughput and latency. The standard has seen a rapid adoption rate: iOS supports Bluetooth LE since Version 5.0 and Android since Version 4.3. The official page shows an overview of supported devices and peripherals.

Bluetooth LE is not backward compatible to classical Bluetooth profiles or protocols. Instead of a stream based protocol (like RFCOMM) it provides a high level data exchange profile called GATT (Generic Attribute Profile) which stores data using hierarchical structured attributes. Conceptually GATT defines two roles: client and server. Typically a smartphone acts as a client while a peripheral (e.g. a temperature sensor) plays the server role. The server groups attributes into services, each of which can contain zero or more characteristics. A characteristic is a container for user data in a specific format (e.g. uint_32). So called “properties” define whether the characteristic is readable, writable and observable.

A characteristic may contain zero or more descriptors which store metadata. An important descriptor is the CCCD (Client Characteristic Configuration Descriptor) which enables or disables server-initiated updates. So the client can decide if he wants to get notified when the data of the characteristic changes.

The services and characteristics are identified with UUIDs. In order to optimize space in the data packages the Bluetooth specification supports shortened 16 and 32 bit UUIDs for predefined standard UUIDs. Otherwise custom 128 bit UUIDs have to be used. But still, the Bluetooth SIG offers the possibility to register own 16 bit UUIDs (see this registration form).

The following picture shows the (slightly simplified) structure of the standardized Battery Service which exposes the level of a battery within a device.

The service provides only one characteristic offering a CCCD descriptor so that clients may register for notifications. The format type is set to uint8 because the possible values of the characteristic range from 0 to 100. Due to the fact that the service and the characteristic is standardized by the Bluetooth SIG, 16 bit UUIDs are used.

The simplicity of the GATT profile was surely a key factor for the success of Bluetooth LE, but when you start building an application you’ll encounter some small glitches. For example the maximum amount of services and characteristics may be very limited on some chipsets (e.g. max. 7 characteristics per service). Although the specification defines a maximum size of 512 bytes for a characteristic value, some hardware implementations only allow a fraction of that size. So you should prototype your service very early against the Bluetooth LE chip you want to use.

Java 8 – Stream.iterate

Today I came to the conclusion that I have to change my fitness center. (The current one is great, but I pay for features I don’t actually use). I checked the contract to find out the period of notice: I signed the contract on 27-Feb-2012 for 52 weeks. After the first 52 weeks the contract is automatically extended by 28 weeks. The period of notice is 12 weeks before the end of each extension time. Finding out the period of notice by hand is quite cumbersome, so I’ve fired up my Java-IDE and played with the new Java 8 Stream (and Date) API. Here’s the result:

public class FCDismissal {

public static void main(String[] args) {
  LocalDate signDate = LocalDate.of(2012,02,27);
  LocalDate firstYear = signDate.plusWeeks(52);

  Optional<LocalDate> nextNoticeDate = 
  Stream.iterate(firstYear, 
                 FCDismissal::extendContract)
    .map(date -> date.minusWeeks(12))
    .filter(date -> date.isAfter(LocalDate.now()))
    .findFirst();

  nextNoticeDate.ifPresent(System.out::println);
}

static LocalDate extendContract(LocalDate date) {
  return date.plusWeeks(28);
}

}

The output of the main method is 2015-08-10, so I have to dismiss the contract before this date.

Let’s dig into the code: The iterate method creates an infinite sequential Stream with a seed and a function for the next element. In my case the seed is the date after the initial 52 weeks and a method reference to extendContract.

Each date is mapped to its corresponding period of notice.

 .map(date -> date.minusWeeks(12))

After that I filter out dates in the past using the filter method.

.filter(date -> date.isAfter(LocalDate.now()))

An infinite sequential Stream has to be bounded using a so called “Short-circuiting operation”. findFirst terminates the stream and returns an Optional, which will be printed out when present:

nextNoticeDate.ifPresent(System.out::println);

As you can see, the Java 8 Stream API allows a very powerful and efficient programming style.

JavaScript’s prototypal inheritance

In our current project we are using JavaScript for a component which should be deployed on iOS, Android and also on server side. JavaScript is a quite facinating language because it hides its true functional nature quite well and provides gotchas which traps the unwary.

In 2008, I’ve read the book “JavaScript – The good parts” from Douglas Crockford. In chapter 3 the beget method is introduced, which creates a new function with a given prototype:

if (typeof Object.beget !== 'function') {
   Object.beget = function(o) {
      var F = function() {};
      F.prototype = o;
      return new F();
    }
}

The advantages of the beget method:

The prototype chain is setup correctly
The programmer doesn’t have to use the new keyword in order to create a new object.

But why could the new keyword be so dangerous ?

If the programmer ommits the new keyword when calling a constructor function, no object will be created. Instead undefined will be returned. In addition the global context might be polluted, because the global context will become the invocation context (also known as this variable). The following example shows what happens when new is ommitted:

var Product = function(id) {
   this.id = id;
}

var p = Product(3);
console.log(p);  // --> undefined
console.log(id); // --> 3

So, the beget method prevents these errors and embraces prototypal inheritance. But there is an alternative: John Resig (and also others) presented a way to support pseudo-classical inheritance in JavaScript. I thought about to use it, but I came to the conclusion that it seems to be quite overengineered for our project. In addition the JavaScript community discussed the pseudo-classical inheritance controversial. Douglas Crockford wrote on his website:

The “super” idea is fairly important in the classical pattern, but it appears to be unnecessary in the prototypal and functional patterns. I now see my early attempts to support the classical model in JavaScript as a mistake.

In a comment on his own blog, John Resig sums it up quite well:

The usefulness of one method vs. another is debatable. If you’re just doing simple stuff then just use straight prototypal inheritance – there’s no reason to have any extra overhead. However when it comes time to stuff like super methods you’ll need some construct to make it happen.

So I wanted to use this beget method in my JavaScript code to use prototypal inheritance. But then I’ve found out, that this method found its way into the language itself as Object.create. Great 🙂

If you want to dig further into the prototype chain, consider the excellent book “You Don’t Know JS: this & Object Prototypes“.

MySQL, UTF-8 and Emoji

In a MySQL-based project we had a series of UncategorizedSQLExceptions in the log:

org.springframework.jdbc.UncategorizedSQLException:
### Error updating database.  
Cause: java.sql.SQLException: 
Incorrect string value: '\xF0\x9F\x98\x98' for column 'note' at row 1

These exceptions had occured because some users entered Emoji characters into a free text field. Emojis are encoded as UTF-8 character. Normally this isn’t a problem, but the MySQL “utf8” character set only supports UTF-8 characters with 3 bytes(!). So Emoji cannot be stored into a “utf8”-encoded MySQL database. 👎

The solution

MySQL 5.5 introduced an “utf8mb4” character set [1]. If you create new MySQL databases you should use this encoding right from the beginning. Otherwise you have to perform a migration afterwards which might take some time. ⌚ [2][3].

[1] http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html
[2] http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-upgrading.html
[3] https://mathiasbynens.be/notes/mysql-utf8mb4

Integrationtests with Groovy’s HTTP Builder

The HTTP Builder for Groovy simplyfies the creation and handling of HTTP requests. It is a wrapper around the Apache HttpClient adding a closure based API which increases readablitity and brevity.

I started to use the HTTP Builder in integration tests. Let’s take a look:

void test_myPage_shouldRedirectToLogin() {
  def baseURI = "http://servername"
  def httpBuilder = new HTTPBuilder(baseURI)
  // Make sure that HttpClient doesn't perform a redirect
  def dontHandleRedirectStrategy = [
    getRedirect : { request, response, context -> null},
    isRedirected : { request, response, context -> false}
  ]
  httpBuilder.client.setRedirectStrategy(dontHandleRedirectStrategy as RedirectStrategy)

  // Execute a GET request and expect a redirect
  httpBuilder.request(Method.GET, ContentType.TEXT) {
    req ->
      uri.path = '/webapp/de/de/myPage'
      response.success = { response, reader ->
        assertThat response.statusLine.statusCode, is(302)
        assertThat response.headers['Location'].value, startsWith("${baseURI}/webapp/login")
      }
      response.failure = { response, reader ->
        fail("Expected redirect but received ${response.statusLine} \n ${reader}")
      }
    }
  }

There are different overloaded versions of the request method. In the example I just use the variant which requires the HTTP Method and the expected content type. In the following configuration closure the request uri path is defined as well as the success and the failure closure. The failure closure will be executed when the status code is greater or equal than 400. Otherwise success will run. Each closure will be called with a HttpResponse object and a reader representing the response body.

Normally the Apache HttpClient will handle redirects transparently, but for this test I wanted to check if the redirect was correctly sent. So I created a new implementation of the RedirectStrategy interface using Groovy’s as operator.

HTTP Builder uses the Content Type to apply response parsing. When application/json is given, then the HTTP Builder will use the JsonSlurper to create a Map from the JSON content. For text/html the HTTP Builder provides a reader (as seen in our example).

In summary HTTP Builder is a convenient and easy to use HTTP framework. The resulting code is much more readable than when the Apache HttpClient is directly used.

HTTP 2.0 and Domain-Sharding

HTTP 2.0 (which is based on SPDY) improves latency and throughput through multiplexing of requests and responses onto one single TCP connection. The header and the data of one request/response cycle is represented by a stream which will be divided into several frames. Streams can have different priorities so that critical resources for rendering (e.g CSS files) can be sent earlier than lower prioritized resources (e.g. images).

The following figure shows an example of a HTTP 2.0 connection flow which multiplexes 6 streams:


       | --> [6] --------[5]-[5]-> | 
Client |                           | Server
       | <-- [1] [4] [2] [4] [2]-- |

Advantages of multiplexing

The multiplexing reduces perceived latency and allows constructs like server push of resources. In addition the browser doesn't have to open up several TCP connections for one origin. With HTTP 2.0 only one TCP connection per origin is needed. As a result only one connection is affected by the TCP "slow start" algorithm. (TCP connections speed up if the network is capable enough, see TCP Congestion Control). The TCP connection will be long-lived and so it is likely that it will reach its full speed (more precisely: its maximum congestion window size).

Domain sharding still useful ?

The Domain sharding technique was used to maximize the amount of parallel TCP connections. Most browers open up six TCP connections per origin. With domain sharding this limit can be increased. But due to the multiplexing technique of HTTP 2.0 this performance optimization doesn't make sense for a HTTP 2.0 connection. It actually might decrease performance on a HTTP 2.0 connection, because the browser has to perform additional DNS requests and manage more TCP connections.

Halbwertszeit von Wissen

Ein spannender Aspekt an der Software-Entwicklung ist die ungeheure Vielfalt an Konzepten, Technologien und Frameworks. Interessierte Software-Entwickler sind ständig dabei, Dinge neuzulernen, vorhandenes Wissen skeptisch zu validieren, und veraltetes Wissen (oder Ansichten) explizit “zu vergessen”.

IT-Wissen hat unterschiedliche Halbwertszeiten. Nehmen wir als Beispiel die Beherrschung der BlackBerry-API: Diese API ist an ein Produkt gebunden. Wenn das Produkt komplett auf eine neue Basis umgestellt wird oder gar stirbt, ist das vorhandene Wissen bzw. die Erfahrung mit der API auf einmal wertlos. Das gleiche gilt beispielsweise auch für Browser-“Optimierungen” (a.k.a Hacks) im Frontendbereich. Frontend-Entwickler, die sich bestens mit IE6 oder IE7-Hacks auskannten, können diese Informationen getrost vergessen, da das Produkt (der Internet Explorer 6,7) einfach nicht mehr relevant ist.

Das Einzige, was vom “Produkt”-Wissen bleibt, sind Konzepte, Strukturen und Ideen, die sich auf andere Gebiete übertragen lassen und die Fähigkeit, Probleme zu analysieren und schnell Lösungen zu finden.

Abseits von der Produktwelt ist das Wissen beständiger. Nehmen wir die HTTP-Spezifikation: HTTP 1.1 gibt es schon seit 1997, und wird wohl noch eine Weile bestehen (Auch wenn Google mit SPDY die Vorlage zu HTTP 2.0 geliefert hat). Von daher lohnt es, die Spezifikation wirklich komplett durchzuarbeiten, sofern man irgendwas mit Internet zu tun hat.

Andere Klassiker sind z.B. die 1989 veröffentliche Bash-Shell oder der wohl ewig nutzbare Vim-Editor. Wer sich da wirklich einarbeitet, kann Jahre davon profitieren.

Die Halbwertszeit von Wissen ist eine wichtige Variable bei der Zusammenstellung des eigenen “Knowledge-Portfolios“. Fundiertes Produkt-Wissen ist natürlich für die tägliche Arbeit unerlässlich, es darf aber nicht das Einzige sein, was man in das Portfolio übernimmt. Sehr wichtig ist es, Wissen mit langer Halbwertszeit zu identifizieren und so das eigene Portfolio langfristiger zu gestalten.