. .

YAPC::EU 2010 - The Renaissance of Perl



Large platform architecture in (mostly) perl - an illustrated tour

Large platform architecture in (mostly) perl - an illustrated tour

By Tomas Doran (‎t0m‎) from drinkers.pm
Date: Friday, 6 August 2010 10:30
Duration: 50 minutes
Target audience: Beginner
Language: English
Tags: architecture http lighttpd memcached mod_perl mogilefs mysql nginx ra scaling varnish web

I work for a company called state51 (www.state51.co.uk) in East London. We
provide a technology platform enabling digital distribution of music. This
platform is used by a number of companies to provide digital distribution
services for a large sector of the music industry (large companies and independents).

We store a significant fraction of a petabyte of music on behalf of our
clients. We integrate over 200 digital media services, to whom we deliver this
content (and the associated metadata) having encoded it in a wide variety of
formats. Our average constant data rate to the APPL AS (i.e. iTunes) is about
4MB/s 24 hours a day, 7 days a week, 365 days a year (1 song per second!).

We provide platform users with API access, allowing our customers to retrieve
their assets (in original or encoded form), and a mechanism to generate 1-use
URIs for that content that can be distributed to 3rd parties securely. We can
scale to delivering requested content at over 1Gb/s, supporting hundreds of
thousands of concurrent streams.

The platform started life in 1994, so we have a large legacy codebase, and have
to live and work with the mistakes and previous technology decisions from a 16
year history.

This talk will describe the architecture of our entire platform in depth. It
will focus on various important subsystems within the platform, explaining what
technology and architectures work well - covering the system constraints and
requirements, then going on to explain how the system is architected to fulfil
those requirements.

I will look into the evolution of each major component of the system, focusing
on how the system has had to change over it's lifetime to keep fulfilling
client needs. This includes correction or mitigation of architectural mistakes
made in the past

This talk will be of interest to anyone who would like to see the 'bigger
picture' of a mature perl based technology platform. It will aim to explain my
perspective on how to plan for changing business requirements, how to scale
production systems, how to architect those systems to be easy to monitor and
debug and how to achieve loose coupling and enable rapid change whilst keeping
the system running.

All of the concepts covered should be simple enough for a beginner to follow,
understand and learn from. Intermediate programmers who have experience
building complex applications should benefit from some of the design and
architecture patterns (and antipatterns) covered, as well as the wider view of
how to tie multiple loosely coupled applications together to form a cohesive
and reliable platform.

Advanced level programmers and architects should get a laugh or two from some
of the more stupid mistakes we've made and had to fix, as well as getting an
alternate view on the harder problems of large scale systems architecture.

Attended by: Sébastien Aperghis-Tramoni (‎Maddingue‎), Michael Kröll (‎pepl‎), Marco Masetti (‎grubert‎), Alessio Bragadini (‎abragad‎), Léon Brocard (‎acme‎), Aaron Crane (‎arc‎), JJ Allen, Salve J. Nilsen (‎sjn‎), Leo Lapworth (‎Ranguard‎), Dagfinn Ilmari Mannsåker (‎ilmari‎), Tom Hukins, Diego Kuperman (‎diegok‎), Valerio Paolini (‎valdez‎), Poul Sørensen (‎poul‎), jonasbn, Ferruccio Zamuner (‎ferz‎), Emanuele Zeppieri (‎emazep‎), Chisel Wright, Florian Ragwitz (‎rafl‎), Damian Conway (‎damian‎), Henrik Andersen (‎HEM‎), Nelson Ferraz, Imre Saling (‎pelagic‎), Peter Makholm (‎brother‎), Alexander Hartmaier (‎abraxxa‎), Dave Rolsky (‎autarch‎), Anders Nielsen (‎anielsen‎), Adam Bartosik, Bogdan Lucaciu (‎zamolxes‎), Joel Bernstein (‎joel‎), David Faux, David Leadbeater (‎dg‎), Marco Fontani (‎mfontani‎), Michael Jemmeson (‎michael‎), Mark Morgan, Maciej Czekay (‎Bruno‎), Uwe Voelker (‎Perl-Uwe.com‎), Dai Okabayashi (‎bayashi‎), Steffen Mueller, Edoardo Sabadelli (‎viking-3‎), Adde Nilsson, Henrik Hald Nørgaard, Michele Valzelli, valerio crini, Rafiq Gemmail (‎Raf‎), Leon Timmermans (‎leont‎), Franck Cuny, Camille Maussang (‎cmaussan‎), Alex Muntada (‎alexm‎), Oliver Thieke (‎o-thieke‎), Markus Wichmann (‎telemorphix‎), J. Nick Koston (‎bdraco‎), Alan Haggai Alavi (‎alanhaggai‎), Gabi Hack (‎gabimuc‎), Squeeky, Marko Zagožen (‎mzagozen‎), Johannes Plunien (‎plu‎), Moritz Lenz (‎moritz‎), Andreas Vögele, Igor Komlew, Sawyer X, Jose Luis Martinez, Giel Goudsmit, Heinz Knutzen,

Our sponsors


  • Booking.Com
  • YAPC::Europe 2009
  • cPanel


  • Shadowcat
  • 123 people
  • Venda
  • Dada


  • YAPC::Europe Foundation
  • Smart Open Software
  • MySQL
  • Exonetric
  • pair Networks


  • GitHub
  • Italpro
  • Geoesse
  • $foo Perl Magazin
  • Lokku
  • Apress
  • No Starch Press
  • O'Reilly Media

Media Partners

  • Linux Magazine


  • Regione Toscana
  • Comune di Pisa