YAPC::EU 2010 - The Renaissance of Perl - Large platform architecture in (mostly) perl

Large platform architecture in (mostly) perl - an illustrated tour

By Tomas Doran (‎t0m‎) from drinkers.pm
Date: Friday, 6 August 2010 10:30
Duration: 50 minutes
Target audience: Beginner
Language: English
Tags: architecture http lighttpd memcached mod_perl mogilefs mysql nginx ra scaling varnish web

I work for a company called state51 (www.state51.co.uk) in East London. We
provide a technology platform enabling digital distribution of music. This
platform is used by a number of companies to provide digital distribution
services for a large sector of the music industry (large companies and independents).

We store a significant fraction of a petabyte of music on behalf of our
clients. We integrate over 200 digital media services, to whom we deliver this
content (and the associated metadata) having encoded it in a wide variety of
formats. Our average constant data rate to the APPL AS (i.e. iTunes) is about
4MB/s 24 hours a day, 7 days a week, 365 days a year (1 song per second!).

We provide platform users with API access, allowing our customers to retrieve
their assets (in original or encoded form), and a mechanism to generate 1-use
URIs for that content that can be distributed to 3rd parties securely. We can
scale to delivering requested content at over 1Gb/s, supporting hundreds of
thousands of concurrent streams.

The platform started life in 1994, so we have a large legacy codebase, and have
to live and work with the mistakes and previous technology decisions from a 16
year history.

This talk will describe the architecture of our entire platform in depth. It
will focus on various important subsystems within the platform, explaining what
technology and architectures work well - covering the system constraints and
requirements, then going on to explain how the system is architected to fulfil
those requirements.

I will look into the evolution of each major component of the system, focusing
on how the system has had to change over it's lifetime to keep fulfilling
client needs. This includes correction or mitigation of architectural mistakes
made in the past

This talk will be of interest to anyone who would like to see the 'bigger
picture' of a mature perl based technology platform. It will aim to explain my
perspective on how to plan for changing business requirements, how to scale
production systems, how to architect those systems to be easy to monitor and
debug and how to achieve loose coupling and enable rapid change whilst keeping
the system running.

All of the concepts covered should be simple enough for a beginner to follow,
understand and learn from. Intermediate programmers who have experience
building complex applications should benefit from some of the design and
architecture patterns (and antipatterns) covered, as well as the wider view of
how to tie multiple loosely coupled applications together to form a cohesive
and reliable platform.

Advanced level programmers and architects should get a laugh or two from some
of the more stupid mistakes we've made and had to fix, as well as getting an
alternate view on the harder problems of large scale systems architecture.