Disclaimer: I know this post requires diagrams. ;-)
I love AppEngine. I love it sou much that I wish to know its secret sauce. But, I don't have all that gigantic infrastructure it bas behind, so I must be happy with what I do have: a laptop.
And what I can do with a laptop? Experiment. Logic puzzles are my favorite distraction, and with a challenge so nice like scaling out similarly to how AppEngine does it, excites my mind.
Basically, I want it to distribute the CPU-load without me controlling the infrastructure. I don't want to fix a number of VMs, not even if I can change it on runtime. I want the infrastructure to do it for me.
First challenge (and maybe the key)
Running all the simulation environment in my laptop.
Other challenges I'll explore later:
- Resource monitoring and quotas
- Application services scalability
- Smart infrastructure
I think that finding a solution so lightweight that it could be demoed in my laptop was the key to get great performance. I evaluated many alternatives, leaving behind the heaviest and most used one among the big names: I won't be able to run full VMs. Not even small ones.
What other options do I have? Running naked processes isn't a nice thing if I want isolation. Here comes Linux containers. They're similar to Solaris Zones or BSD Jails. The processes running in a container share the kernel instance but none of the data, unless you specify it otherwise.
I also thought on using a User-mode kernel, but it would consume fixed amounts of memory per instance and I wouldn't be able to share other resources like the file system.
For example, to start a bash shell in a container named container1 you can run:
lxc-execute -n container1 /bin/bash
Note: mount a cgroup filesystem somewhere: mount -t cgroup none /cgroup
how much CPU time did it took to start up bash inside a container, and type exit and enter?
0.004s in my laptop. I can afford that time to create an isolated process =)
Also, you can specify some extra options via a configuration file (-f configfile):
lxc.utsname = container1 # hostname for the processes insde the container
lxc.network.type = veth # virtual ethernet card. There are other options such as VLANs, physical cards, etc
lxc.network.flags = up # flags for the NIC, "up" starts it up
lxc.network.link = br0 # bridge NIC outside the container (use brctl to create and configure it)
lxc.network.ipv4 = 10.4.0.2 # IP address for the container NIC
lxc.network.name = eth0 # name of the NIC for the # name of the NIC for the process inside the container
lxc.rootfs = /containers/container1 # root filesystem for the container
Nice, isn't it?
Infrastructure service I think I'll need:
- Routing
- Load balancing
- Web servers
- Queues
- Scalable storage
- Memcache
All services exposed to the instances (queues, storage, memcache) will be ran inside other containers, and web server instances should only be able to establish outgoing connections with internet or with their service instances.
As I cannot afford to maintain a custom API for each supported programming language/platform and keep the secret sauce for my own, what I plan is to expose publicly known services and tell the application developers to use the client library they like most.
Routing
A message routing system across nodes to make the aware of infrastructure changes. While this is a key piece to harmonize a multi-node geographically distributed data center, I don't require it right now.
I'm smoke-tested the use of RabbitMQ, and I felt comfortable.
I'll dive into routing in later posts.
Load balancer
When I first automatized the startup of the containers, I needed a load balancer to balance the load among them. My first try was HA-Proxy (the one used, for example, in stackoverflow.org). Maybe it was my fault, but I couldn't make it work fine.
I found that Nginx was very easy to setup, and to reload the configuration it was enough with sending the HUP signal to the process (killall -HUP nginx does the job in my tests).
I instantiate a load balancer for each hosted application. As each container has its own IP address, I transparently load-balance as if it were a server. My Nginx configuration files looks like this:
events {
worker_connections 4096;
}
http {
upstream app1 {
server 10.0.4.2; # container 1 IP address
server 10.0.4.3; # container 2 IP address
}
server
{
listen 10.0.4.1:80; # load balancer IP address
server_name app1.cloudy.com;
location / {
proxy_pass http://app1.cloudy.com;
}
}
}
Web server
One or many lightweight web servers to run different applications wrote in as many frameworks as possible.
For now, I'm using only non-optimized one (kinda good for this stage of the research). I'm using the xsp development web server that comes with mono to run ASP.NET applications (AppEngine runs only Python and Java).
Queuing
A queue system to let applications run asynchronous processing independently of how many instances are running. RabbitMQ (the same I plan to use to dispatch infrastructure messages) also supports queues. It appears that it will be easy to instantiate these processes as application services (also, inside containers) for application queues.
I'm also considering to use memcacheQ for simplicity.
Storage
A cluster-izable storage. I have nothing thought about this.
Memcache
A memcache service. I think this is fundamental and easy to offer. In fact, I can't think about having a load-balanced application without having a memcache.
The first service I'm deploying along with the web server instances is one memcached server instance per application. I know that because of fail-overs I'll need to deploy at least two instances in different places and restore the state from a persistent store in case of failure. In this case I think about using mecacheDB.
More about isolation
I need to isolate all interaction between applications. A process-level isolation is achieved through containers. Other aspects includes connectivity and storage.
Connectivity isolation is quite easy if every process runs with its own NIC card and IP address: a firewall outside the container will do the job.
Unfortunately, it's not very comfortable to create a own file system for each container and tricking the mountpoints with unionfs or aufs didn't work yet for me.
Currently, I'm creating a separate user for each container inside a shared filesystem and I'm looking forward to isolate all change-sensitive directories, like the /tmp, /var and /home to its own storage. An advantage is that they're volatile, I can trash them once the container is stopped.
No comments:
Post a Comment