Bad things happens more often then we want. If you work on product running in production then it's just a matter of time when you'll see some process eating much more memory then it expected to. In a lucky case it's possible to track down a sequence of events that lead to a leak and reproduce it in a controlled environment with full power of debugging tools. However more often then not a leak is caused by something unknown that only happens in production and not on QA. That's exactly a kind of situation I found myself recently staring at 1.2 gigs Python process that normally consuming only 20-30 megs. People of Indonesia probably knew something when named one of their mythological monster leak.

Without any clues about what caused a leak, memory state of a process is the most obvious place to get some insight. It's possible to get full core dump of a process by sending SIGABRT with kill -SIGABRT <pid> or through GDB: gdb --pid=<pid> -ex generate-core-file. However Python code dumps are not easy to work with (at least for me).

There's some special tools designed to help with memory leaks in Python. I'm aware of Heapy and Meliae and probably there's some more. But these tools are intended to be used from a scope of a process under investigation. That's fine if it's possible to modify a source code and reproduce a memory leak. But it's not helping a lot with an existing process in faulty state.

Shooting for the moon I would like to have a Python's shell in the context of a troublesome process with access to either Heapy or Meliae toolset. Let's see what can be done to get there.

Attaching shell

One great tool to interact with running Python processes is Pyrasite. It's essentially the GDB wrapper that allows to run arbitrary code in the context of some Python process.

Installing Pyrasite doesn't take much. GDB is the only prerequisite. Version requirement on GitHub page is bit unclear: "gdb (version 7.3+ (or RHEL5+))". In practice it's been working fine for me on CentOS 5+, despite bundled GDB version 7.2. Package itself is available through PyPi, so pip/easy_install do their job nicely.

After installing Pyrasite it's taking just one command to connect to the running process:

$ sudo pyrasite-shell 27817
Pyrasite Shell 2.0
Connected to '/usr/bin/python27 bin/messages_rest_api 5637 8'
Python 2.7.3 (default, May 18 2012, 22:11:42)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)
>>>

sudo is required to get into someone else's process, but other then that it's just feels like magic. The following snippet can be used to make sure if it's really the process I'm looking for:

import sys, traceback

for thread, frame in sys._current_frames().items():
    print('Thread 0x%x' % thread)
    traceback.print_stack(frame)
    print()

Typing it in will print all threads that process should have plus one dedicated to this new shell:

Thread 0x7f5c341d8700
  File "/usr/lib64/python2.7/threading.py", line 524, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "<string>", line 167, in run
  File "/usr/lib64/python2.7/code.py", line 243, in interact
    more = self.push(line)
  File "/usr/lib64/python2.7/code.py", line 265, in push
    more = self.runsource(source, self.filename)
  File "/usr/lib64/python2.7/code.py", line 87, in runsource
    self.runcode(code)
  File "/usr/lib64/python2.7/code.py", line 103, in runcode
    exec code in self.locals
  File "<console>", line 3, in <module>
()

Now about actual profiling...

Attaching memory profiler

I prefer to have some debugging tools included to production deployments. Usually they include IPython, IPDB and Heapy. If they were around before process started they will be available for import immediately. It's not always the case, unfortunately. Heapy is default weapon-of-choice for memory profiling, so let's make it available in pyrasite shell.

No surprise, Heapy can be installed with standard tools like easy_install/pip. It's part of guppy package. However even after installing, it still will not be available in pyrasite shell. It's just not in the packages search path. In order to make it available, the package directory must be added to process' sys.path:

>>> from guppy import heapy
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: No module named guppy

>>> import sys
>>> sys.path.append("/usr/lib/python2.7/site-packages/guppy-0.1.10-py2.7-linux-x86_64.egg")
>>> from guppy import hpy
>>> hp = hpy()
>>> hp.heap()
Partition of a set of 208391 objects. Total size = 27694176 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  97049  47  8741208  32   8741208  32 str
     1  52668  25  4466568  16  13207776  48 tuple
     2    777   0  1831128   7  15038904  54 dict of module
     3   1988   1  1796392   6  16835296  61 type
     4  13807   7  1767296   6  18602592  67 types.CodeType
     5  13660   7  1639200   6  20241792  73 function
     6   1531   1  1573384   6  21815176  79 dict (no owner)
     7   1988   1  1507040   5  23322216  84 dict of type
     8   2586   1   406928   1  23729144  86 list
     9    372   0   390624   1  24119768  87 dict of class
<533 more rows. Type e.g. '_.more' to view.>

Now that's looks good. From that point it's possible to use full power of Heapy to track down the root cause. Other tools can be added in similar way. I found that Meliae while somewhat painful to install helps me better in complicated cases. There's good tutorial about this tool.

1. Monadic DI with Reader: is it that good?
2. Monadic DI with Reader: interfaces

The sole reason I can see that justifies efforts for exploring the Reader-based dependency injection versus a common constructor-based is that monadic DI builds a nice and comfortable immutable environment. Software components need not to be initialized and whole wiring of an application logic happens completely out-of-context. And I must admit that this reason is big enough to give it a try.

Let's consider a typical situation in software engineering: some interface, an interface's implementation and a configuration required for this implementation. For convenience sake I'll use a pair of host and port values as the configuration example:

trait FooConfiguration {
    val hostname: String
    val port: Int
}

Now the interface:

trait IFooService {
    def doFiz(a: Int): Reader[FooConfiguration, Int]
    def doBuzz(b: String): Reader[FooConfiguration, String]
}

Here's the first problem: Reader in return type of interface's operations requires a type of configuration as it's first type parameter. FooConfiguration is specific to some concrete implementation of IFooService. Some other implementation would use a different configuration, e.g. filename. We can dodge this problem by pulling configuration to the trait's type parameter:

trait IFooService[Conf] {
    def doFiz(a: Int): Reader[Conf, Int]
    def doBuzz(b: String): Reader[Conf, String]
}

...or using Scala's feature of abstract types:

trait IFooService {
    type Conf
    
    def doFiz(a: Int): Reader[Conf, Int]
    def doBuzz(b: String): Reader[Conf, String]
}

I prefer the abstract type approach because it's less intrusive and isolates implementation details better. Type parameters of lower-level interfaces will leak to their clients. Although they carry little of implementation specifics it's still a boilerplate code that no one is happy to see. I can also see ugly type lambdas in some distance at a type parameters direction. Here's an example how things can get bad for a configuration trait of some service on top of IFooService and his twin brother:

trait HighLevelServiceConfiguration[FooConf, BarConf] {
    val foo: IFooService[FooConf]
    val bar: IBarService[BarConf]
}

Each interface will add at least one type parameter and with an application building layers on top of other layers it will quickly get out of control.

One possible and trivial implementation of IFooService can looks like this:

object SomeFooService extends IFooService {
    // defining the concrete form of configuration required by the service
    trait Conf {
        val hostname: String
        val port: Int
    }
    
    def doFiz(a: Int): Reader[Conf, Int] = Reader { conf =>
        conf.port + a
    }
    
    def doBuzz(b: String): Reader[Conf, String] = Reader { conf =>
        conf.hostname + b
    }
}

Looks nice so far. There's some boilerplate code in a form of constructing new Reader instances for every interface method but depending on a point of view it can be either tolerated, or fixed by some sort of an implicit conversion. I'm not going to bother with it now, as there's much more troublesome things ahead.

Let's make a step back for a moment and see if different Readers can be transparently combined in same context. Let's assume that there's another interface IBarService with the implementation SomeBarService:

trait IBarService {
  type Conf

  def doBar(a: String): Reader[Conf, Int]
}

object SomeBarService extends IBarService {
  trait Conf {
    val filename: String
  }

  def doBar(a: String): Reader[Conf, Int] = Reader { conf =>
    (conf.filename + a).size
  }
}

Now between SomeBarService and SomeFooService I can write some nice for-comprehension:

val program = for {
  fiz <- SomeFooService.doFiz(1)
  bar <- SomeBarService.doBar("one")
} yield fiz + bar

val configuration = new SomeFooService.Conf with SomeBarService.Conf {
  val filename: String = "some_file"
  val hostname: String = "some_host"
  val port: Int = 5432
}

val result: Int = program(configuration)

Surprisingly enough this code will compile and the type of program value would be inferred as Reader[SomeFooService.Conf with SomeBarService.Conf, Int]. Now it's something to be both happy and worrying about. A happy part comes from the fact that compiler did it's job perfectly by inferring the correct value type. And worries are because that inferring was possible thanks to Reader's contravariance) on first of it's type arguments. It means that we have to use a class inheritance mechanic to build up the configuration that makes the program live and breathe.

The Scala 2.10 compiler will give you a warning about inferring existential type scalaz.Kleisli and propose to add import scala.language.existentials to get rid of this warning. Currently I don't understand how Scalaz managed to infer this type and don't know if this warning something to worry about. Hopefully I'll have some time soon to grok it.

In the next post I'll try to workaround some difficulties that comes from the need to use class inheritance to build up the configuration value. One problem that is particularly important is applying different configurations to the same services depending on some context. Constructor-based dependency injection makes it trivial by creating different stateful instances of services. It doesn't seems to be the case for Reader-based injection.

1. Monadic DI with Reader: is it that good?
2. Monadic DI with Reader: interfaces

This post will start my blog and also a series of thoughts about dependency injection in Scala. I want to explore this topic because I'm still confused about it even after spending multiple days digging into.

There is no lack of blog posts, slides and presentations that explain the idea of doing dependency injection in Scala through Reader monad. Here're just some of them for the reference:

• Dead-Simple Dependency Injection by Runar Bjarnason
• Dependency Injection Without the Gymnastics: video and PDF by Tony Morris & Runar Bjarnason
• Using Reader Monad for Dependency Injection and other StackOverflow questions
• and many, many more...

At the same time the development team of Twitter uses and recommends a simple and all-familiar constructor-based dependency injection. On the other hand there's more advanced technique in a form of cake pattern with it's own share of a well-grounded critique.

The constructor-based DI and the cake pattern are both well described in many sources. But all descriptions of the monadic DI approach that I have seen suffer from severe incompleteness. Here're just some problems:

• dependency injection is treated in a very narrow sense of making some objects available in specific contexts
• a configuration is always modeled as a single object without any form of separation per component
• presentations emphasize on transparent passing of a configuration rather than it's usage inside of functions
• almost no mentions about an additional complexity in a form of monad transformers all around

Among these problems the first one is most serious for me. After all, dependency injection isn't some trendy thing that we just must have in our code. The goal of dependency injection is to help with dependencies management of an application as it will grow bigger and bigger. There's two ways how dependency injection helps with it: pushing us into decoupling specific implementations from their interfaces and improving visibility into how components are interconnected. Without paying attention to an application structure it's absolutely possible and in fact very easy to end up with a poorly structured code and spaghetti-like dependencies using any DI framework or even without one. From a visibility standpoint, any person caring for keeping an application complexity under control would like to understand few things about the code easily:

• how many dependencies each specific application component have?
• how many components are depending on some specific component?
• how does a graph of dependencies looks like? Any loops? Is it tree-like or web-like?

Reader dependency injection in a form that's described in aforementioned sources makes it very difficult to answer these questions. In fact it's impossible to understand what are real dependencies of some component without going through it's code looking for all references of configuration passed through Reader. A global configuration accessible from every place whenever a developer wants makes the Service Locator anti-pattern, which is nothing I would like to see in my application.

I had a blitzkrieg plan to build a simple Reader-based solution for DI that would give better visibility into how application components are connected. Pretty soon, I realized that this task is far from a trivial one. I'm not done yet, but by merely trying I've got some good insights I'm going to share in my following posts. Stay tuned!