Upgrading Azure Storage Client Library to v2.0 from 1.7

I upgraded Azure Storage to version 2.0 from 1.7 and I've found a number of differences when using storage. I thought how I'd document how I upgraded these more awkward bits of Azure Storage in version 2.0.

DownloadByteArray has gone missing

For whatever reason DownloadByteArray has been taken from me. So has DownloadToFile, DownloadText, UploadFromFile, UploadByteArray, and UploadText

Without too much whinging I'm just going to get on and fix it. This is what was working PERFECTLY FINE in v1.7:

public byte[] GetBytes(string fileName)
{
    var blob = Container.GetBlobReference(fileName);
    return blob.DownloadByteArray();
}

And here is the code modified to account for the face that DownloadByteArray no longer exists in Azure Storage v2.0:

public byte[] GetBytes(string fileName)
{
    var blob = Container.GetBlockBlobReference(fileName);
    using (var ms = new MemoryStream())
    {
        blob.DownloadToStream(ms);
        ms.Position = 0;
        return ms.ToArray();
    }
}

How to get your CloudStorageAccount

Another apparently random change is that you can't get your storage account info in the same way as you used to. You used to be able to get it like this in Storage Client v1.7:

var storageAccountInfo = CloudStorageAccount.FromConfigurationSetting(configSetting);
var tableStorage = storageAccountInfo.CreateCloudTableClient();

But in Azure Storage v2.0 you must get it like this:

var storageAccountInfo = CloudStorageAccount.Parse(
            CloudConfigurationManager.GetSetting(configSetting));
var tableStorage = storageAccountInfo.CreateCloudTableClient();

Why?.. not sure. I have had problems with getting storage account information before so maybe this resolve that.

What happened to CreateTableIfNotExist?

Again, it's disappeared but who cares.. Oh you do? Right well let's fix that up. So, in Azure Storage Client v1.7 you did this:

var tableStorage = storageAccountInfo.CreateCloudTableClient();
tableStorage.CreateTableIfNotExist(tableName);

But now in Azure Storage Client Library v2.0 you must do this:

var tableStorage = storageAccountInfo.CreateCloudTableClient();
var table = tableStorage.GetTableReference(tableName);
table.CreateIfNotExists();

Attributes seem to have disappeared and LastModifiedUtc has gone

Another random change that possibly doesn't achieve anything other than making you refactor your code. This was my old code from Storage Library Client v1.7:

var blob = BlobService.FetchAttributes(FileName);
if (blob == null || blob.Attributes.Properties.LastModifiedUtc < DateTime.UtcNow.AddHours(-1))
{
    ...
}

But now it should read like this because thought it looks prettier (which it does in fairness).

var blob = BlobService.FetchAttributes(FileName);
if (blob == null || blob.Properties.LastModified < DateTimeOffset.UtcNow.AddHours(-1))
{
    ...
}

Change your development storage connection string

This is just a straight bug so that's excellent. I was getting a useless exception stating "The given key was not present in the dictionary" when trying to create a CloudStorageAccount reference. To resolve this change your development environment connection string from UseDevelopmentStorage=true to UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://127.0.0.1 then it will magically work.

Bitch and moan

Apologies for the whingy nature of this post, I'm quite a fan of Azure but I have wasted about 3-4 hours with this "upgrade" from Azure Storage Client Library 1.7 to 2.0. It's been incredibly frustrating particularly since there seems to be no obvious reason why these changes were made. I just can't believe the amount of breaking changes when I haven't really written that much Azure storage code.

Randomly taking out nice methods like DownloadByteArray and DownloadText is surely a step backwards no? Or randomly renaming CreateIfNotExist() to CreateIfNotExists()... what is the point in that?!

I remember when upgrading to ASP.NET 4 from 3.5, I spent very little time working through breaking changes and I have 100 times more .NET code than I do Azure Storage code. As well as that, I was well aware of the many improvements with that .NET version update, with this Azure Storage update I have no idea what I'm getting. No matter the improvements, it is just an Azure storage API and this number of breaking changes, often for the benefit of syntax niceties is just unnacceptable.

Oh, if you are still in pain doing this I have found a complete list of breaking changes in this update along with minimal explanations here.

Export and back up your SQL Azure databases nightly into blob storage

With Azure I have always believed, if you can do it with the Azure Management Portal then you can do it with a REST API. So I thought it would be a breeze to make an automated job to run every night to export and back up my SQL Azure database into a BACPAC file in blob storage. I was suprised to find scheduling bacpac exports of your SQL Azure databases is not documented in the Azure Service Management API. Maybe it is because the bacpac exporting and importing is in beta? Nevermind. I successfully have a worker role backing up my databases and here's how:

It is a REST API so you can't use nice WCF to handle all your POST data for you but there is a trick still to avoid writing out all your XML parameters by hand and instead strong typing a few classes.

Go to your worker role or console application and add a service reference to your particular DACWebService (it varies by region):

• North Central US: https://ch1prod-dacsvc.azure.com/DACWebService.svc
• South Central US: https://sn1prod-dacsvc.azure.com/DACWebService.svc
• North Europe: https://db3prod-dacsvc.azure.com/DACWebService.svc
• West Europe: https://am1prod-dacsvc.azure.com/DACWebService.svc
• East Asia: https://hkgprod-dacsvc.azure.com/DACWebService.svc
• Southeast Asia: https://sg1prod-dacsvc.azure.com/DACWebService.svc

Once you import this Service Reference you will have some new classes that will come in handy in the following code:

//these details are passed into my method but here is an example of what is needed
var dbServerName = "qwerty123.database.windows.net";
var dbName = "mydb";
var dbUserName = "myuser";
var dbPassword = "Password!";

//storage connection is in my ServiceConfig
//I know these the CloudStorageConnection can be obtained in one line of code
//but this way is necessary to be able to get the StorageAccessKey later
var storageConn = RoleEnvironment.GetConfigurationSettingValue("Storage.ConnectionString");
var storageAccount = CloudStorageAccount.Parse(storageConn);

//1. Get your blob storage credentials
var credentials= new BlobStorageAccessKeyCredentials();
//e.g. https://myStore.blob.core.windows.net/backups/mydb/2012-05-22.bacpac
credentials.Uri = string.Format("{0}backups/{1}/{2}.bacpac",
    storageAccount.BlobEndpoint,
    dbName,
    DateTime.UtcNow.ToString("yyyy-MM-dd"));
credentials.StorageAccessKey = ((StorageCredentialsAccountAndKey)storageAccount.Credentials)
                                   .Credentials.ExportBase64EncodedKey();

//2. Get the DB you want to back up
var connectionInfo = new ConnectionInfo();
connectionInfo.ServerName = dbServerName;
connectionInfo.DatabaseName = dbName;
connectionInfo.UserName = dbUserName;
connectionInfo.Password = dbPassword;

//3. Fill the object required for a successful POST
var export = new ExportInput();
export.BlobCredentials = credentials;
export.ConnectionInfo = connectionInfo;

//4. Create your request
var request = WebRequest.Create("https://am1prod-dacsvc.azure.com/DACWebService.svc/Export");
request.Method = "POST";
request.ContentType = "application/xml";
using (var stream = request.GetRequestStream())
{
    var dcs = new DataContractSerializer(typeof(ExportInput));
    dcs.WriteObject(stream, export);
}

//5. make the POST!
using (var response = (HttpWebResponse)request.GetResponse())
{
    if (response.StatusCode != HttpStatusCode.OK)
    {
        throw new HttpException((int)response.StatusCode, response.StatusDescription);
    }
}

This code would run in a scheduled task or worker role to be set for 2am each night for example. It is important you have appropriate logging and notifications in the event of failure.

Conclusion

This sends off the request to start the back up / export of database into a bacpac file. The success of this is no indication that the back up was successful, only the request submition. If your credentials are wrong you will get a 200 OK response but it the back up will fail silently later.

To see if it has been successful you can check on the status of your exports via the Azure Management Portal, or by waiting a short while and having a look in your blob storage.

I have not covered Importing because, really, exporting is the boring yet important activity that must happen regularly (such as nightly). Importing is the one you do on the odd occassion when there has been a disaster and the Azure Management Portal is well suited to such an occassion.

Cannot create database 'DevelopmentStorageDb20110816' in Storage Emulator Azure SDK

I have uncovered this problem before and this time I thought I would spend the time to crack it. When running the Storage Emulator from the Azure SDK or running DSINIT.exe for the first time it needs to initialise; part of this involves creating a new database. This was not running correctly due to a permissions problem which stated "Cannot create database 'DevelopmentStorageDb20110816' : CREATE DATABASE permission denied in database 'master'".

The full report is as follows:

Added reservation for http://127.0.0.1:10000/ in user account MSFT-123\MyUser.
Added reservation for http://127.0.0.1:10001/ in user account MSFT-123\MyUser.
Added reservation for http://127.0.0.1:10002/ in user account MSFT-123\MyUser.

Creating database DevelopmentStorageDb20110816...
Cannot create database 'DevelopmentStorageDb20110816' : CREATE DATABASE permission denied in database 'master'.

One or more initialization actions have failed. Resolve these errors before attempting to run the storage emulator again. These errors can occur if SQL Server was installed by someone other than the current user. Please refer to http://go.microsoft.com/fwlink/?LinkID=205140 for more details.

Grant permissions

The user this program is running as must be a sa with full permissions on the database. If this is not the case you can either change the user or GRANT permissions to the current user.

To change the user run the Windows Azure SDK Command Prompt (as administrator) and type 'DSINIT /?'. This will give you details on how to change user, which is to use the /user: argument.

Alternatively you could GRANT the permissions to your default user like so:

USE master

GRANT CREATE DATABASE TO "MYDOMAIN\MyUser"

Either of these solutions should solve your permissions problem with DSINIT.

Cannot GRANT permissions

Unfortunately my problems went further than this still. I believe this is to do with how many times I had installed a SQL Server Express in the past. How irritating is that?! You own the machine and yet you don't seem to have the permissions that reflect that!

Anyway, the solution to make this work once again was (unfortunately) to uninstall SQL Server Express and reinstall it again. This way you will be the owner of the SQL Server Express database engine and you will be able to create all the databases you wish, including your long awaited Storage Emulator database.

A better way to regain admin access

A better way to regain admin access was pointed out to me by a colleague, Michael Coates. You can either solve this loss of administrator on a SQL server by following this troubleshooting guide from MSDN (this worked for one commenter). Or you can run a batch script that will magically do it for you (this didn't work for one commenter). I have not done either of these so take this advice at your own risk. Remember, my way was to delete and reinstall the server so this cannot be more risky, surely?

"The Web Platform Installer could not start" fix

I was trying to download the Azure SDK version 1.5 and I was experiencing issues preventing the install because the Windows Platform installer could not load up and run properly. It instead opened a error message saying "The Web Platform Installer could not start. Please report the following error on the Web Platform Installer Forum. http:/forums.iis.net/1155.aspx".

The problem was that the Web Platform Installer needed to update but could not seem to manage it. This was causing the error to display and then crash without allowing me to install the original Azure SDK I was actually trying to install.

Microsoft Fix It fixed it

Microsoft Fix It is a tool from Microsoft that automatically finds the particular one you want in this scenario is the Diagnose and fix program installing and uninstalling problems automatically.

To do this download it and use it as follows:

• When the option to select "Detect problems and let me select the fixes to apply." appears, do that.
• say you are having a problem with "Installing"
• After a short wait scroll down to find "Microsoft Web Platform Installer 3.0" (or which ever version number you have) and select that.
• Next you should click "Yes, try uninstall" then click next and follow the instructions.

This should solve your issue with the Microsoft Web Platform Installer and allow you to install the Azure SDK or whatever it is you're trying to install.