Category: cloud


By nature, when working with connectable resources in the cloud, the number and IP location of those resources can change at any point. A pain point is often managing the addresses with which to connect to these instances, so I spent a short amount of time doing something about it by creating a program in WPF which automatically retreives a list of instances for an Amazon EC2 account and allows connections over Remote Desktop Protocol (RDP).

CloudTerminal v0.2

After quickly realising then that there are many additional features which would also be useful in this area, I open sourced the project at http://cloudterminal.codeplex.com Special thanks for already contributing a beautiful logo to James Tenniswood.

To prevent over-engineering the tool and never coming up with a version I can use myself, let alone releasing, I decided to put some old skool agile methodology on the project and prioritise the features by how essential they are for each release. This roadmap is then published on CodePlex. Development of features in 0.2 is complete and a working copy can be installed via click-once.

0.1
- Retreive and display list of connections from EC2
- Connect and disconnect via RDP to any instance in list

0.2
- Show instance CPU history
- Store account keys in local configuration
- Optimise UX

0.3
- Allow multiple AWS accounts
- SSH connectivity, including private key storage
- Overlay instance details / commands on list select
- Add grid of instances, shown when no connections are active

0.4
- Allow Azure accounts with more appropriate list view of instances/services

0.5
- Test TCP connectivity before connecting. Offer option to open relevant remote cloud firewall port to client IP address.
- Allow instance / image / service specific credential saving for connections.

Awhile ago I wrote about developing a page to bring in some rudiementary cloud-watch data to measure and compare realtime trafffic of an ELB (Elastic Load Balancer) enabled website and the performance of the providing servers.

Part of my role at Condé Nast Digital is to become fixated on be aware of the performance of our public-facing web sites, and be able to pre-empt or respond quickly to any traffic spikes or performance issues. To that end, I spend some time thinking of new ways to visualize and explore this data for both myself and my team.

In wishful style, I’ve open-sourced a web app containing these visualizations, in the hope that others contribute in the form of ideas or code, or at least get some use out of it so I can more easily justify the late nights.

The AWSMonitor project on CodePlex explains each visualization and offers a roadmap, a forum to discuss new visualizations, and the code to download and run. The app is written in ASP.Net MVC3 and uses Razor views. The views use the javascript Google Visualization API to render graphs and gauges (favouring svg versions).

There are two main ways I use the visualizations in this app daily:

Infoporn – office displays
In true wired.co.uk style, I love to have screens of realtime data on show so that everyone can easily see what’s going on, both in terms of editorial content, new features, traffic and server performance.

The /elb/random view really shines here as it displays a new site from our list of load-balancers on AWS after each interval.

In this visualization we can see:

  • A graph comparing the traffic today (blue), yesterday (red), today -7 days (yellow) and today – 14 days (green)
  • A gauge showing the average CPU utilization for each server
  • A frame containing the site’s output

Problem Investigation
AWS ELB manages server health and will take servers out if the health check target responds with an error. When this happens, I like to see exactly what’s happening on each server. The /elb/{load-balancer-name}/preview shows what the site looks like for each server

This visualization also accepts a parameter that allows us to see a specific Url for each server.

View more information and download the app at CodePlex.

The Azure tools for Visual Studio have great support for creating web roles out of Web Application projects, but no built-in support for simple websites (that is, asp.net sites that are compiled at runtime). There are many scenerios where not needing to have a Web Application project are going to be the right decision for development and production, and the good news is that any IIS-servable directory can be packaged/deployed using simple commands from the Azure SDK.

The process of taking projects / code / files from from local to cloud with Azure is to:

  1. Prepare a local directory with a ready-to-run output of the application’s build.
  2. Prepare a service definiton file (describes the roles to package).
  3. Package into an Azure package file using CSPack.
  4. Upload to production or staging environments on Azure, alongside a service configuration file (determines how many instances and what kinds of storage are available to the roles).

The Visual Studio add-in can handle this for web roles using website application projects, but we have to do this manually for website projects.

Preparing the project

The best folder structure for working locally is to create a parent directory with each role as a subdirectory. The service definition/configuration and package files can then be stored in the parent. So here I’ve created a the default Web Site from Visual studio at TestWebsite1\WebRole1 and just to prove that we don’t need to add/modify anything from this default set-up I haven’t changed anything, except to add a hello-world default.aspx:

<h1>Hello World</h1>
    <p><b><%= Environment.MachineName %></b> running <i><%=Environment.OSVersion.VersionString %></i> on <i><%=DateTime.Now.ToString("s") %></i></p>

Packaging the project for Azure

First off, we need to create a service definiton file, which just describes which kind of roles are in our application, and what their requirements are. In this case, we only have a single WebRole (and we’ll give it some storage for fun), and I’ve called the file ServiceDefinition.csdef:

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="Azure1_umbraco" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WebRole name="WebRole1">
    <InputEndpoints>
      <InputEndpoint name="HttpIn" protocol="http" port="80" />
    </InputEndpoints>
    <ConfigurationSettings>
      <Setting name="DiagnosticsConnectionString" />
    </ConfigurationSettings>
    <LocalResources>
      <LocalStorage name="LocalStorage1" cleanOnRoleRecycle="false" sizeInMB="100" />
    </LocalResources>
  </WebRole>
</ServiceDefinition>

Next we use the CSPack tool to package up the site in to a single file (testwebsite1.cspkg) for upload to Azure. The tool will automatically look for the supporting files for each role under a directory with the role’s name:

cspack ServiceDefinition.csdef /out:testwebsite1.cspkg

We’ll also need to create a simple service configuration file (ServiceConfiguration.cscfg) which describes how many instances to give each role:

<?xml version="1.0"?>
<ServiceConfiguration serviceName="Azure1_umbraco" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
  <Role name="WebRole1">
    <Instances count="1" />
    <ConfigurationSettings>
      <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" />
    </ConfigurationSettings>
  </Role>
</ServiceConfiguration>

Deploying the website to Azure

Now that we’ve finished everything we need to do locally, we can move to the Azure online management tool, where we’ll need to create a new Hosted Service. Once that’s made, we select either the production or staging environment for the service and select Deploy…. All we need to do here is select the package and configuration files, and give the deployment a name:

Azure will spend some time uploading the package, after which your service will be in the Stopped state.

After clicking Run the site should be instantly available.

In a later post, you’ll see why I wanted to make sure we could use a web deployment without requiring a Web Application project.

In the past, EC2 virtual instances were started only from a machine image stored on S3, Amazon’s distributed object storage platform. In this model, instance storage is allocated to a new instance and the image is copied to the instance’s main instance storage device. In addition, you get a few extra storage devices depending on the instance type. The trouble is that whilst the instance may be given a lot of this instance storage, it is ephemeral – meaning that once you terminate the instance (or it experiences a failure), the data stored is permanently lost. Data that needs peristance is therefore usually stored by a user on S3, or by creating a disk from EBS, Amazon’s block-level peristent (but not distributed) storage platform.

Recently Amazon added the option to start an image from a copy of any EBS volume. Not only does this allow larger images (and therefore the introduction of Windows Server 2008) but, since the storage is persistent, instances can be stopped or started at will and the data will keep its state.

However, whilst Amazon’s command line utilities support it, very few GUI tools (including Amazon’s own web console) expose the ability to start an EBS-backed instance which also has it’s normal instance-store disks. These disks are actually very useful for scenarios where an application requires a large amount of temporary storage which you don’t want to have to pay for, or worry about volume management with the rest of your EBS volumes. The documentation for the command line also lacks a good explanation for how to do this for Windows servers. After some trial and error, I have found that these commands will work nicely:

Start a c1.medium instance from an EBS-backed private image including 1 ephemeral disk:

ec2-run-instances ami-4bebc03f -k myinstancekey -g my-security-group -b "xvdg=ephemeral0" -t c1.medium --availability-zone eu-west-1a

Start an m1.xlarge instance from an EBS-backed amazon image including 3 ephemeral disks:

ec2-run-instances ami-93ebc0e7 -k myinstancekey -g my-security-group -b "xvdg=ephemeral0" -b "xvdh=ephemeral1" -b "xvdi=ephemeral2" -t m1.xlarge --availability-zone eu-west-1a

When constructing these commands, you still need to know how many instance disks are available to each instance type, and what the windows device names are that you can attach them to (apparently limited to 10: xvdh through xvdp).

Follow

Get every new post delivered to your Inbox.