Casual Snippets insights on software development

Twitter4j extended mode

Recently Twitter increased the character limit from 140 to 280 characters. To get the full text of tweets via their API you need to explicitly set extended mode as a request parameter.

To enable this in the latest twitter4j version (4.0.6) you should use the setTweetModeExtended method in the ConfigurationBuilder:

twitter4j.conf.ConfigurationBuilder()
      .setOAuthConsumerKey(""))
      .setOAuthConsumerSecret("")
      .setOAuthAccessToken("")
      .setOAuthAccessTokenSecret("")
      .setTweetModeExtended(true)

If you are running an older twitter4j version there is a workaround here.

Note that for retweets you have to use getRetweetedStatus().getText() instead of just .getText().

Evicted dependencies with Selenium

Had a weird exception when I added a Selenium test to a project:

ThrowableException: Class org.openqa.selenium.support.ui.ExpectedConditions$6 does not implement the requested interface java.util.function.Function (FluentWait.java:209)

The error should be related to an outdated Guava version, but I didn’t have any Guava dependencies in my sbt project, or at least I thought I didn’t.

I added the sbt dependency graph plugin to check my dependency tree and there it was:

[info]  | +-org.seleniumhq.selenium:selenium-remote-driver:3.5.3
[info]  |   +-cglib:cglib-nodep:3.2.4
[info]  |   +-com.google.code.gson:gson:2.8.0
[info]  |   +-com.google.guava:guava:23.0 (evicted by: 23.1-android)
[info]  |   +-com.google.guava:guava:23.1-android

Guava 23.0 was being evicted by the 23.1-android version which I managed to trace back to the Bugsnag dependency:

[info]   +-com.bugsnag:bugsnag:3.1.3
[info]   | +-com.fasterxml.jackson.core:jackson-databind:2.9.1
[info]   | | +-com.fasterxml.jackson.core:jackson-annotations:2.9.0
[info]   | | +-com.fasterxml.jackson.core:jackson-core:2.9.1
[info]   | |
[info]   | +-com.google.guava:guava:23.1-android
[info]   | | +-com.google.code.findbugs:jsr305:1.3.9 (evicted by: 2.0.1)
[info]   | | +-com.google.code.findbugs:jsr305:2.0.1
[info]   | | +-com.google.errorprone:error_prone_annotations:2.0.18
[info]   | | +-com.google.j2objc:j2objc-annotations:1.1
[info]   | | +-org.codehaus.mojo:animal-sniffer-annotations:1.14

It was an easy fix now, I simply excluded Guava from Bugsnag in build.sbt and let it use the 23.0 version from selenium:

"org.seleniumhq.selenium" % "selenium-java" % "3.5.3" % "test",
"com.bugsnag" % "bugsnag" % "3.1.3" exclude("com.google.guava", "guava"),

Future.transform in Scala 2.12

Scala 2.12 added a few improvements to scala.concurrent.Future. Here I am going to take a quick look at the new transform and transformWith methods. First, let’s see the old method signature:

 def transform[S](s: (T) ⇒ S, f: (Throwable) ⇒ Throwable): Future[S]

Creates a new future by applying the ‘s’ function to the successful result of this future, or the ‘f’ function to the failed result.

The main shortcoming of this method is related to f. Even though we can handle the exception case, the signature Throwable ⇒ Throwable bounds the future to fail once an exception is thrown.

The new method is much more powerful in this regard:

 def transform[S](f: (Try[T]) ⇒ Try[S]): Future[S] 

Creates a new Future by applying the specified function to the result of this Future.

By providing a function Try[T] ⇒ Try[S] we can handle both Success and Failure cases and transform them in any way we like. For example, if we want a future to always complete successfully we can now write it like this:

  scala> val future = Future(throw new Exception("Oops"))

  scala> future.transform {
      case Success(_) => Try("OK")
      case Failure(_) => Try("KO")
  }

  res0: scala.concurrent.Future[String] = Future(Success(KO))

To achieve a similar behaviour in Scala 2.11 we would have to combine map and recover statements or use onComplete in a Promise.

The new transformWith unifies flatMap and recoverWith in a similar way. Here is the signature:

def transformWith[S](f: Try[T] => Future[S]): Future[S]

Creates a new Future by applying the specified function, which produces a Future, to the result of this Future.

If you want to transform your result and you need to return another Future instead of a Try, go for transformWith.

Make sure to check the blog posts from Viktor Klang for more information about Futures on Scala 2.12 and, as always, check the documentation for details.

Json4s custom serializers - the right way

Most examples of Json4s custom serializers use the following approach:

case class Animal(name: String, nrLegs: Int)

class AnimalSerializer extends CustomSerializer[Animal](format => ( {
  case JObject(
  JField("name", JString(name)) ::
    JField("nrLegs", JInt(nrLegs)) ::
    Nil
  ) => Animal(name, nrLegs.toInt)
}, {
  case animal: Animal =>
    ("name" -> animal.name) ~
      ("nrLegs" -> animal.nrLegs)
}
))

While this may sometimes work, it is not the correct way to deserialize JSON. The above code assumes the JSON is ordered, which is an incorrect assumption. A JSON object is, per definition, an unordered set of key/value pairs. Here is an example that breaks the deserializer:


scala> implicit val fmt = org.json4s.DefaultFormats + new AnimalSerializer()

scala> val cat = """{"name":"cat", "nrLegs": 42}"""

scala> val catLegsFirst = """{"nrLegs": 42, "name": "cat"}"""

scala> read[Animal](cat)
res2: Animal = Animal(cat,42)

scala> read[Animal](catLegsFirst)
org.json4s.package$MappingException: Can't convert JObject(List((nrLegs,JInt(42)), (name,JString(cat)))) to class Animal
  at org.json4s.CustomSerializer$$anonfun$deserialize$1.applyOrElse(Formats.scala:373)
  at org.json4s.CustomSerializer$$anonfun$deserialize$1.applyOrElse(Formats.scala:370)
  at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
  at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
  at scala.collection.AbstractMap.applyOrElse(Map.scala:59)
  at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
  at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:523)
  at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
  at org.json4s.Extraction$.extract(Extraction.scala:351)
  at org.json4s.Extraction$.extract(Extraction.scala:42)
  at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
  at org.json4s.jackson.Serialization$.read(Serialization.scala:50)
  ... 43 elided

If your application receives JSON from an external system, you can not predict the order of the received data and you should not rely on it. Here is an implementation that works regardless of the order. Note that the serializer code is the same as above, which is correct, I just changed the deserialization logic:

import org.json4s.CustomSerializer
import org.json4s.JsonAST._
import org.json4s.JsonDSL._

case class Animal(name: String, nrLegs: Int)

class AnimalSerializerUnordered extends CustomSerializer[Animal](format => ( {
  case jsonObj: JObject =>
    val name = (jsonObj \ "name").extract[String]
    val nrLegs = (jsonObj \ "nrLegs").extract[Int]

    Animal(name, nrLegs)
}, {
  case animal: Animal =>
    ("name" -> animal.name) ~
      ("nrLegs" -> animal.nrLegs)
}
))

Since each value is extracted independently, the order of the original object does not matter. You can see more details about Json4s value extraction here.

Running code when Play starts

Simple snippet that runs during Play 2.5.X initialization. Add your code to the initialize method below:

import javax.inject.{Inject, Singleton}

import com.google.inject.AbstractModule
import play.api.inject.ApplicationLifecycle

class Module extends AbstractModule {
  def configure(): Unit = bind(classOf[SystemGlobal]).asEagerSingleton()
}

@Singleton
class SystemGlobal @Inject()(appLifecycle: ApplicationLifecycle) {
  def initialize(): Unit = {

    println("Hello!")

  }

  initialize()
}

The Module class needs to be in your root package. If you create it anywhere else you have to register the class in application.conf.

You can see slightly more complex example here with a stop hook that runs on application exit. The Play website also has good documentation about replacing the old Global object functionality with dependency injection.