Scala: Lessons from the front line

thumb_justin by Justin Mann, Bubbly CTO

Over the course of human events, someone somewhere must fail, often repeatedly, for the rest of humanity to learn. Here at Bubbly, we have recently performed this noble service for humanity in our glorious search for improved server performance.  This is the blessed curse of a successful online service: hockey stick MAUs make the CEO ecstatic, but strike fear into the heart of even the most hardened engineer.

A long time ago in distant data center, specifically AWS’ Singapore location, our overworked m1.large Jetty server was straining under the load. We knew that this solution had reached the end of its useful life and needed a major rethink. So we did what any scrappy startup would do— look at other successful businesses and see what free options are available to solve our problem. Twitter & LinkedIn both recently switched to Scala for a portion of their server infrastructure and that solution seemed to fit our needs as well. We came up with a thorough plan for the server migration:

1)    Learn Scala
2)    Rewrite Server
3)    Profit!

With a solid plan, we began our quest with these simple words of wisdom:

“Everything is easy when you don’t understand it”.

Rewriting our API server in Scala was relatively easy. Scala is a very expressive language and maps very well with the functionality most API servers provide. Our biggest challenges were not faced until we deployed it.

Tracking failure was our first obstacle. Scala, like Java, uses exceptions to handle failure. We needed to know when one of these exceptions occurred. The obvious solution is to pay Typesafe or New Relic a bunch of money for server monitoring. But we are scrappy, highly skilled and humble engineers. Thus, we captured all exceptions and send them in email to ourselves. To be honest, we did sign up for New Relic for the free shirt and continue to use their free account.

This decision worked great except with futures. They go off on to random threads, run and die with an exception without telling us so we created the appropriately name safe future:

1
2
3
def safeFuture[T](request: Request[AnyContent])(p: Future[T]):Future[T] = {
p.recover(defaultFutureExceptionHandler(request))
}

This wraps every future with a default exception handler. Please for the love of everything holy, copy this code.

Now that our future is safe, what could possible go wrong? Java! Our server would randomly max at 100% cpu and never recover. After a lot of Google searching, we found that the problem is that Java 6 + Netty do not play well together. Another lesson to learn: do not use Java 6. Upgrade to Java 7.

The last lesson was the most crucial. It is the trap set by the creators of Scala. When using Scala, you must learn to not in any way, try to be creative. The language encourages the opposite with implicits, ability to override everything (+,-,.), and list parsing functions like zip. When used with restraint, it is a beautiful language that will massively speed development. But when it is used creatively, it becomes a curse.

Some simple advice:

  • Do not use implicits
  • Do not override operators
  • Case classes are better Tuples
    • It looks cool to return (1, “Hello”), but who knows what that is supposed to mean and what happens if the caller or callee change breaking the implicit contract
    • Yes, types are implied automatically but forcing the type is a great way to catch mistakes with compiler
1
2
3
4
List('g', 'o', 'o', 'd').zip(List('l', 'u', 'c', 'k')
).foldLeft[(String, String)] (("", "")) {
(acc, v) => (acc._1 + v._1, acc._2 + v._2)
}.productIterator.toList.mkString(" ")

Side note: Thank you for everyone who attended out first Bubbly TechTalk which I have discussed Scala. More to come!

5 thoughts on “Scala: Lessons from the front line

  1. Nirmalya Sengupta

    Do not use implicits‘ and ‘Do not overload operators‘ are very sound advices, IMHO. As a veteran ‘C++’ (and later years, ‘Java’) programmer (and let it be said, a newbie in the Scala world), I have been a party to and witness of enough mayhem caused by Operator overloading and default parameters (and global Enumerations). While learning Scala – which is undoubtedly a very expressive language especiall to those of us who are used to the verbiage of Java – I am constantly reminding myself to understand ‘implicit’ as a feature but to be extremely cautious while using it. Ditto for Operator Overloading.
    Case classes are better than Pairs – agreed. However, I am not sure if proliferation of Case classes is a hint of good design. I am just not knowledgeable enough to comment. May be, Scala Gurus can share their thoughts.

  2. Could you be more specific about what were your problems with implicits?

    “Do not use implicits” is broad and many things require other implicit things (futures need an implicit ExecutionContext).

    Also on overriding operators, do you mean overloading an operator for a “established” type? Or overuse of symbolic code? Both can be problematic but why would it be problematic to define sum for say orders and make it a symbolic method? `val total = Order + Order` is clear in intent.

    And I would also like to have a more qualified explanation about “list parsing functions like zip”. What’s wrong with that?

  3. Implicits, we had one case where we used an implicit to specify which DB to use, we had a read and a write connection. The code would compile, but would fail at runtime because the read connection was used for writing, or more challenging to discover, the write connection was used for a slow SELECT query. I could be more specific and say do not use implicit if you program can ever more than value, feel free to use implicit as an overly complex version of global state.

    Operator overloading, which is easier to understand SQS ! (1.second, “foo”) or SQS.queueMessage(1.second, “foo”). IMHO, operator overloading tends to reduce number of characters typed and increase complexity. In the above example, total = Order + Order, could mean anything and what is obvious to person writing the code is rarely obvious to the reader. Does this concat the orders, does it check if two orders are conflicting and throw an exception, can I subtract an order, it is summing the price of the order or do the order maintain some unique state like order id? I like using operators to perform tasks that are very simple and universally understood like adding two integers.

    List parsing functions, in most scala code you will see map, flatMap, foreach, for and few other simple list functions. When you use the more advanced list functions like zip it is easy to create confusion. Honestly, if I saw code using zip and the use case made sense then I would not have any complaints.

  4. Greetings, I believe your site may be having internet browser compatibility issues.
    Whenever I take a look at your website in Safari, it looks fine but when opening in I.E., it’s got some overlapping issues. I merely wanted to provide you with a quick heads up! Besides that, great site!

Thoughts? Comments? Something to share?