Agility™ for BLIMP

Introduction

In 2009 the East-Of-England Regional Development Agency (EEDA), a UK government organisation, had the innovative idea of creating a large-scale broadband testing engine which ran invisibly and periodically to build up a substantial and accurate national broadband performance database.

EEDA chose the Cloud for their deployment but it was of prime importance to them that the project would not tie them to a single cloud provider. They required a solution with the potential to manage their application executing on a variety of IT infrastructures (potentially simultaneously) including existing governmental IT infrastructure or alternative public clouds, and with the ability to take immediate advantage of improved service or reduced costs which providers might deliver in the future.

Arjuna's Agility product was chosen by EEDA because of the product’s unique focus on managing service delivered by a loosely connected cloud infrastructure i.e. a Federated Cloud. Agility was used to deploy and scale-out the project’s broadband measurement application on Amazon’s EC2 cloud and has been used to collect three million broadband measurement results. Additionally and critically Agility provided the required management capabilities necessary to allow the application to run on multiple infrastructures.

Laurence Ramsey of EEDA says "Arjuna's Agility product has enabled us to deliver a project that just wouldn't have been feasible using a traditional approach."

The Blimp application

In early 2008 it was predicted that in some regions of the UK, 50% of the population would not have access to acceptable broadband services for the foreseeable future. It is crucial for the UK to improve this situation if it wants to fulfil the goals of the government's 'Digital Britain' report [see http://www.bis.gov.uk/building-britains-digital-future]. The report sets the government a target to have 90% of the UK population connected to superfast broadband by mid 2017.

Traditional 'online broadband speed checking utilities' offer small scale tests of limited functionality and the results are not captured or recognised by UK Government and industry broadband investors. In 2009 EEDA had the innovative idea of creating a large-scale broadband testing engine which ran invisibly and periodically to build up a substantial and accurate national broadband performance database. The data collected was to be used to highlight areas where broadband service is poor and help direct resources to improve service.

To that end an application, BLIMP ('Broadband Line Information Monitoring Project'), was commissioned by EEDA. The application provided a service allowing consumers to register their broadband connection and view its performance over time using the console illustrated below. Through the use of BLIMP the project profiled terrestrial broadband connections over the entire country and highlighted areas that were consistently poor. The UK communications regulator Ofcom could then use the BLIMP results as evidence of poor service, and work with ISPs to improve the service in problem areas. The data is also available to other governmental bodies and will be of national importance in the development of next-generation broadband across the UK.

When EEDA considered their deployment options for BLIMP they faced problems that are similar to those experienced by many start-ups who launch a service with the potential for mass appeal. Firstly, EEDA did not wish to invest in the IT infrastructure required to trial their application prior to its proven efficacy, but at the same time needed to be capable of handling high loads if the project was an immediate success, This made the choice of a cloud computing solution an obvious one. However it was of prime importance to EEDA that a successful project should be future-proofed and not tie them to a single cloud provider. They therefore required a solution with the potential to manage BLIMP executing on a variety of IT infrastructures (potentially simultaneously) including a variety of existing governmental IT infrastructures or alternative public clouds, and with the ability to take immediate advantage of improved service or reduced costs which providers might deliver in the future.

Arjuna's Agility product was chosen by EEDA because of the product’s unique focus on managing service delivered by a loosely connected cloud infrastructure i.e. a Federated Cloud.

Using Agility to deploy and manage BLIMP on a single cloud

At the start of the deployment project an Agility server was installed on the chosen public cloud, Amazon’s EC2. Agility servers support a Policy framework which enables an application’s manager to deploy Policy Modules on demand. Policy Modules can both monitor and modify the organisation’s behaviour in order to reflect their organisation’s policy with regard to the running of the application.

In the BLIMP project, the first module to be deployed was designed to execute a single instance of the BLIMP application within an EC2 VM (this required only minor configuration changes to the BLIMP application) and direct performance test invocations appropriately. Once the initial deployment was proven to be successful, the Policy Module was replaced by one capable of monitoring the load exerted on the application by consumers running performance tests. The new Policy Module then dynamically provisioned the optimal number of 'on-demand' instances of the application within EC2, thus ensuring that all test requests were satisfied whilst keeping costs to a minimum.

As the project progressed it became clear that due to the granularity of EC2 provisioning (a VM per hour) cost-savings could be achieved by ensuring that once VMs were provisioned they would be effectively utilised for the entire hour. In order to achieve this, an additional Policy Module was deployed which staggered and smoothed the rate at which consumers ran tests.

Due to the billing model offered by Amazon EC2, it was also possible to use the data gathered by Agility in order to calculate the average cost of executing a test. This allowed EEDA to accurately budget for the use of the application based on how much test data they wished to capture, and for a Policy Module to limit the number of consumer tests which are run in order to ensure that any cost constraints are not exceeded.

Each of these modifications to existing policy, or additions of new policy was able to be made simply and without impacting the application or the availability of the service. Change is an inevitable aspect of any application’s lifecycle and the ability to reflect changing policy is essential if costly manual intervention is to be avoided.

Using Agility to manage BLIMP on a Federated Cloud

By deploying Arjuna Agility, the BLIMP project not only benefits from flexible Cloud management, it is also future proofed for developments in the Cloud Computing market. The project owners were aware that the Cloud infrastructure market is evolving and that a variety of cloud infrastructure offerings would likely be available in the future. In particular they were aware that the UK government was considering the creation of a governmental cloud for the use of public sector organisations such as their own. It was an important requirement for the owners therefore that their application should not be tied to a single cloud.

Agility can take advantage of access to resources offered by other organisations through it’s support for Service Agreements. Service Agreements enable organisations to be connected together in order to share service through cooperation. Such connected networks offer dynamic on-demand access to resources which although they are shared through cooperation, are ultimately controlled by the organisation which owns the resource i.e. they represent a Federated Cloud. As an example, the BLIMP application might run primarily on a governmental cloud, but use EC2 to handle scale-out for extreme spikes in load, and/or another public cloud to deliver continuity of service should the governmental or EC2 clouds suffer an outage.

Agility offers a common interface to any Cloud and can be deployed on many different and diverse Cloud environments. Agility allows the BLIMP application to seamlessly extend to a different Cloud provider without disruption to existing service. Furthermore, this can be done rapidly and on a regular basis in order to react to changes in service and pricing offered by Cloud providers.

Through its support for Policy and for Service Agreements, Agility allows BLIMP to react quickly to changes in strategic direction. For example, if the BLIMP service was to be offered to customers outside the UK, Service Agreements could be constructed with regional Clouds and a simple policy deployed that checks the originating location of the performance test and ensures that the appropriate regional infrastructure is utilised.