Disseminating information across a large set of processes is a classic problem in distributed systems. In this talk we will present and discuss several classes of epidemic solutions for achieving reliable broadcast across a large number of processes that can be found in the scientific literature, and briefly explore their strong and weak points. Then we will focus on a concrete approach based on the combination of different gossiping techniques such that implicit, albeit robust, spanning trees can be leveraged to disseminate information both reliably and efficiently. We further discuss how this technique which spawned from academia, ended up being used in Industry, as a part of the Cluster Metadata Subsystem of Basho's Riak. We'll explore some of the key differences between the original academic proposal and its applications on Riak and discuss the practical reasons that motivated these modifications. Finally, we will discuss some future techniques that Basho is currently exploring.

About João Leitão

About Jordan West

Linux developers have put a lot of effort into achieving parallel performance on multicore systems. How do these ideas apply to scalability in distributed systems? Distributed systems designers have created datatypes and protocols with varying semantics and consistency to perform wide-area transactions with low latency. Can these ideas be used in multicore databases to achieve scalability with many cores? This talk looks at the similarities and differences between executing transactions on multicore and distributed databases, and describes how ideas used in one area could be used in the other.

We built an in-memory, multicore database which achieves good parallel performance, even when many transactions conflict. A key contribution is breaking up a transaction into commutative and non-commutative operations, and executing commutative operations on contended data in parallel, while still using a concurrency control protocol for the rest of the transaction to ensure serializability. We suggest how these ideas could be applied to improve the throughput of distributed transactions.

We also share experiences discovering language-level scalability bottlenecks in Go when scaling up to 80 cores.

About Neha Narula

There are important distinctions between 'using' Riak as a storage backend and 'operating' Riak as a distributed database in the enterprise. While it's crucial for engineers to understand how the database itself operates, it's even more important for operators to be intimately familiar with the entire ecosystem in a global network. For the OpenX digital advertising and monetization platforms, Riak supports several different use cases across hundreds of nodes: local key/value reporting utilities, world-wide distribution of configurations, regional datastores used for delivery, and multi-datacenter cloud-storage. This talk will present some of the problems we've faced operating such large and diverse clusters, and how we approached solving them using configuration management, monitoring and statistics toolsets. Topics may include: mondemand stats collection, log aggregation with SumoLogic, Icinga monitoring, Salt Stack management, large object detection within clusters, and how we worked with Basho on one particular bitcask memory sawtooth issue within our largest clusters.

About Matt Davis

Knowledge of how to set up good benchmarks is invaluable in understanding performance of the system. Writing correct and useful benchmarks is hard, and verification of the results is difficult and prone to errors. When done right, benchmarks guide teams to improve the performance of their systems. When done wrong, hours of effort may result in a worse performing application, upset customers or worse! In this talk, we will discuss what you need to know to write better benchmarks for distributed systems. We will look at examples of bad benchmarks and learn about what biases can invalidate the measurements, in the hope of correctly applying our new-found skills and avoiding such pitfalls in the future

About Aysylu Greenberg

This talk explores an implementation of a service discovery service written in Erlang on top of riak ensemble. Features with this service and other strongly consistent key-value stores such as etcd and ZooKeeper will be compared and contrasted. We will also describe some of our experiences with the service in production as a means to configure other services

About Mark Allen

At Bitly we serve over 6 billion clicks/per month, record each one, and do all sorts of analysis on that data to provide useful information back to our users.

As a company we face two major challenges:

1. Not breaking the Internet - if Bitly links stop working, a non-trivial part of the Internet just became much less useful.
2. Moving fast - we need to continually adapt to change and build new tools to provide more value to our users.

This presents the eternal challenge of accomplishing both stability and speed.   In this talk we'll cover the processes, approaches, and tools we use to do over 20-30 deploys each day to over 400 servers without causing any major disruptions to service.

Some of the topics to be covered include:

• Benefits of Service Oriented Architecture
• Designing with ops in mind from day one
• The power of NSQ (realtime distributed messaging)
• Asynchronous > Synchronous (except when it isn't)
• Designing for predictability
• Route around failure (via hostpool)
• Automate + Monitor ALL THE THINGS
• Peer review for code + process

About Sean O'Connor

Starting in February 2012, we began building a distributed systems library, Sirius, that included an implementation of Paxos. The team working on it did not have extensive distributed systems experience, yet a working implementation was in production just a few short months later. Along the way, we employed some useful tactics and learned some lessons about building, deploying, and evolving distributed systems. In this talk we'll share several specific techniques we used (and production errors we encountered) that should generalize well to lots of particular implementation scenarios.

About Peter Cline

In League of Legends, just as in any competitive team game, communication is essential to success. Therefore, when building Chat for the game we had to make sure that the new service would be absolutely rock solid in every respect. This includes not only guaranteed message delivery and consistent presence propagation across the system, but also maintenance of the created social network graph.

In this talk I would like to present how we achieved linear scalability for Chat, improved its overall fault tolerance, and got ready for the new features we wanted to ship. I will also discuss in detail why we migrated our data from MySQL to Riak and how we used CRDTs to deal with conflicting object updates.

About Michal Ptaszek

Sherpa is Yahoo!'s massively parallel and geographically distributed hosted key-value store, supporting most of Yahoo!'s web scale applications. The existing deployment is based upon update-in-place index structures [B-Tree] and provides good performance for read intensive applications. But some of the newer applications requiring high write throughput and increasing adoption of SSDs with limited write capacity made us explore better indexing algorithms. Log Structure Merge Trees, we believed would provide the answer to both problems and we began our effort to create next generation Sherpa with LSM storage engine.

Typically in large established systems such fundamental changes are rarely easy, a series of careful performance benchmarks revealed many challenges and interesting architecture changes needed but in due course was rewarded with a 10X increase in write throughput for large insert workloads without sacrificing read latency. This talk with go through many of these challenges, lessons learnt and the changes we had to do to make this happen and this experience could be valuable to a practitioner in this space.

About Jayadev Chandrasekhar

Maybe you've heard of Mesos — that thing that you can run Hadoop on. I think it powers Twitter? Isn't it an Apache project, or something?

In this talk, we'll learn all about Mesos — what it is, how you can leverage it to simplify your infrastructure and reduce AWS/cloud computing costs, and why you should develop your next application on top of it. This talk will give you the tools you need to understand whether Mesos is the right fit for your infrastructure, and several starting points for learning more about Mesos.

About David Greenberg

Dataloop.IO is a SaaS based infrastructure monitoring startup. We provide analytics and alerting on time-series data in real-time. Our customers are pumping 10s (soon to be 100s) of thousands of metrics per second into our system for processing.

To support this we recently built a complex event processing system on top of Riak. The system is designed to scale horizontally and be completely masterless. The system allows us to define arbitrary stream processes (e.g. time-based metrics counters), with all processing state stored in Riak.

On top of this we built our time-series data store and our custom alerting system. The performance and flexibility of Riak, removes a lot of the complexity from our application.

This talk will cover the process we went through to build the system and why we build it the way we did.

About Colin Hemmings

One of the challenges of using Docker at scale is orchestrating Docker itself. In this talk we introduce Clocker - an Apache software licensed open source project which solves this problem.

Clocker uses Apache Brooklyn to deploy and manage a dynamic Docker cloud on your infrastructure of choice scaling it out as needed. However what is really compelling is that once your Docker cloud is spun up it can then be treated as a target location by Brooklyn enabling you to deploy and manage applications in it.

Bootstrapping a Docker cloud using Brooklyn enables you to take advantage of existing Brooklyn blueprints such as Riak, composing these with other applications and services to rapidly deploy and manage distributed applications.

Brooklyn handles both supply and demand - the cloud and the applications running in it.

About Alex Heneveld

One of the challenges of using Docker at scale is orchestrating Docker itself. In this talk we introduce Clocker - an Apache software licensed open source project which solves this problem.

Clocker uses Apache Brooklyn to deploy and manage a dynamic Docker cloud on your infrastructure of choice scaling it out as needed. However what is really compelling is that once your Docker cloud is spun up it can then be treated as a target location by Brooklyn enabling you to deploy and manage applications in it.

Bootstrapping a Docker cloud using Brooklyn enables you to take advantage of existing Brooklyn blueprints such as Riak, composing these with other applications and services to rapidly deploy and manage distributed applications.

Brooklyn handles both supply and demand - the cloud and the applications running in it.

About Alex Heneveld

About Duncan Johnston-Watt

One of the challenges of using Docker at scale is orchestrating Docker itself. In this talk we introduce Clocker - an Apache software licensed open source project which solves this problem.

Clocker uses Apache Brooklyn to deploy and manage a dynamic Docker cloud on your infrastructure of choice scaling it out as needed. However what is really compelling is that once your Docker cloud is spun up it can then be treated as a target location by Brooklyn enabling you to deploy and manage applications in it.

Bootstrapping a Docker cloud using Brooklyn enables you to take advantage of existing Brooklyn blueprints such as Riak, composing these with other applications and services to rapidly deploy and manage distributed applications.

Brooklyn handles both supply and demand - the cloud and the applications running in it.

About Duncan Johnston-Watt

Abstract is Coming Soon!

About Ines Sombra

Abstract is Coming Soon!

About Mahsa Najafzadeh

About Marten Mickos

About James Watters

How to validate distributed application workloads for object storage (primarily bulk storage of photo and video assets). This talk will include lessons learned in testing both RAID and object storage performance. We will examine the test environment used including a look at the Seagate hardware architecture, RAID configuration, network configuration, Linux configuration, and system tools used to set up RiakCS on a 6-node rack with 2PB of raw storage.

About Earl C Ruby III

About Mac Devine

Replacing the Big NHS Computer : A success story from the land of major Government IT Projects?

About Martin Sumner

About Dave McCrory