Using SSL in the Azure Compute Emulator With A Cloud Service Web Role

December 8 2015

Here's what I had to do to get SSL working in VS2015 with a web role in a cloud service:

 

1. Make sure that the url itself is 127.0.0.1 when launching IIS Express. If it is localhost, it won't work.

2. Make sure that you bind to the azure dev fabric thumbprint that gets installed with the SDK. If you are using the 2.7 SDK, it is F8ACE24A36F93B006BFAF495F6C14FB827AC61A3

3. Make sure that the dev fabric cert is in the right place. EG, if you say it is in "MY" then make sure it is Personal when you are looking in certificate manager. Or, if you say it in root, make sure you copy it there

I explicitly remove the HTTP endpoint as well.

I've attached a sample project that works.

AzureCloudService1.zip (425.21 kb)

Using NLog For Diagnostic Logging In Windows Azure Cloud Services And Writing The Logs To Azure Table Storage

November 6 2015

I’ve never liked the default logging mechanisms in Azure for application event logging.  Parsing the WADDiagnosticInfrastructureLogsTable is always such a hassle. Events I’ve written are mixed in with all the other events that Azure is firing all the time, and everything in my event is jammed into a single field.

So, for my current Azure project, I switched to using NLog and love it!  I used the NLog extension for writing events to Azure table storage: https://github.com/abkonsta/NLog.Extensions.AzureTableStorage – you’ll notice that isn’t the master Git repository, but the master one https://github.com/harouny/NLog.Extensions.AzureTableStorage doesn’t have the right README.md updated!  But I’m using 1.1.3.2 and it works great.

I am setting the connection string via code so that I pick up the string based on the local config or cloud config, aka

var azureStorageTarget = (AzureTableStorageTarget)
    LogManager.Configuration.FindTargetByName("AzureTableStorage");
azureStorageTarget.ConnectionString = RoleEnvironment.GetConfigurationSettingValue
    ("StorageConnectionString");
LogManager.ReconfigExistingLoggers();

 

And wow so much better. Some of the overloads are so nifty, like the ability to just give the exception to the logger, or letting the logger do the string formatting.  Nice!

Below is a sample project showing both a web role and a worker role using NLog and logging to table storage.

NLog4AzureCloudService.zip (495.15 kb)

Encrypting and Decrypting A String Sent As A Querystring Parameter Using C#

June 24 2015

I recently needed to encrypt/decrypt strings sent as querystring parameters over the wire. The use case happens to be allowing people to unsubscribe from a newsletter by clicking on a hyperlink in their email. The server receives the email as a querystring. Obviously, I don’t want to expose a public service that takes an unencrypted email. So, I encrypt the email as part of the newsletter template. Then decrypt on the web server.

Not rocket science, but worth going over how I did it, as there were a few gotchas.

First, I generated an RSA crypto key as XML, using the code found here:

public class MyCrypto
{
    RSACryptoServiceProvider rsa = null;
    string publicPrivateKeyXML;
    string publicOnlyKeyXML;
    public void AssignNewKey()
    {
        const int PROVIDER_RSA_FULL = 1;
        const string CONTAINER_NAME = "KeyContainer";
        CspParameters cspParams;
        cspParams = new CspParameters(PROVIDER_RSA_FULL);
        cspParams.KeyContainerName = CONTAINER_NAME;
        cspParams.Flags = CspProviderFlags.UseMachineKeyStore;
        cspParams.ProviderName = "Microsoft Strong Cryptographic Provider";
        rsa = new RSACryptoServiceProvider(cspParams);
 
        //Pair of public and private key as XML string.
        //Do not share this to other party
        publicPrivateKeyXML = rsa.ToXmlString(true);
 
        //Private key in xml file, this string should be share to other parties
        publicOnlyKeyXML = rsa.ToXmlString(false);
         
    }
}

Then, to encrypt the string in such a way that it could be passed as a querystring, I had to make some changes. First, I changed the encoding to UTF8 and not ASCII. Second, I then base64 encode the string, Third, I URL encode the string:

public string EncryptAndEncode(string text)
{
    string encryptedText;
    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
    {
        rsa.FromXmlString(Resources.Resources.publicKeyXML);
        var bytes = rsa.Encrypt(Encoding.UTF8.GetBytes(text), true);
        encryptedText = Convert.ToBase64String(bytes);
    }
    return HttpUtility.UrlEncode(encryptedText);
}

On the decrypt side, because I’m getting the string as a method in my controller, the URL decoding is handled for me. So, the decrypt looks like this:

private string Decrypt(string text)
{
    string decryptedText;
    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
    {
        rsa.FromXmlString(Resources.Resources.publicPrivateKeyXML);
        var bytes = Convert.FromBase64String(text);
        decryptedText = Encoding.UTF8.GetString(rsa.Decrypt(bytes, true));
    }
    return decryptedText;
}

Installing Blog Engine 3.1 As A Virtual Application Under An MVC Website With Sql Server

May 14 2015

Hit a bunch of gotchas doing this; figure I’d share with the world how I got it working.

1. RUN THE UPDATED SQL SCRIPT

There’s a piece of SQL script that isn’t in the setup script and can only be found if you download the source … or copy/paste from below :)

ALTER TABLE dbo.be_Pages ADD
    SortOrder int NOT NULL CONSTRAINT DF_be_Pages_SortOrder DEFAULT 0
GO
CREATE NONCLUSTERED INDEX IX_be_Pages ON dbo.be_Pages
    (
    SortOrder
    ) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO

2. BEWARE WEB.CONFIG INHERITANCE

Because of web.config inheritance from the root application, all kinds of wacky stuff can go wrong. Easiest fix is to turn off inheritInChildApplications like this:

 

<location path="." inheritInChildApplications="false">
<system.web>
...
</system.web>
</location>

Remember to do it from system.web and system.webserver. And for appsettings, unless you do #3 below.

3. ADD SOME APPSETTINGS

Even if you add that element to appSettings, you still may need to add the following appSettings:

<appSettings>
  <add key="webpages:Enabled" value="true" />
  <add key="webpages:Version" value="3.0.0.0" />
...
</appSettings>

4. CONFIGURE MEMBERSHIP AND ROLEMANAGER

If you are using Sql Server, of course you change the provider up at the top of web.config.

<BlogEngine>
   <blogProvider defaultProvider="DbBlogProvider" fileStoreProvider="DbBlogProvider">
...
    </blogProvider>
</BlogEngine>

But don’t forget to configure membership and roleManager to use the database. They are deeper down in web.config:

<membership defaultProvider="DbMembershipProvider">
  <providers>
    <clear />
    <add name="XmlMembershipProvider" type="BlogEngine.Core.Providers.XmlMembershipProvider, BlogEngine.Core" description="XML membership provider" passwordFormat="Hashed" />
    <add name="SqlMembershipProvider" type="System.Web.Security.SqlMembershipProvider" connectionStringName="BlogEngine" applicationName="BlogEngine" />
    <add name="DbMembershipProvider" type="BlogEngine.Core.Providers.DbMembershipProvider, BlogEngine.Core" passwordFormat="Hashed" connectionStringName="BlogEngine" />
  </providers>
</membership>
<roleManager defaultProvider="DbRoleProvider" enabled="true" cacheRolesInCookie="false">
  <providers>
    <clear />
    <add name="XmlRoleProvider" type="BlogEngine.Core.Providers.XmlRoleProvider, BlogEngine.Core" description="XML role provider" />
    <add name="SqlRoleProvider" type="System.Web.Security.SqlRoleProvider" connectionStringName="BlogEngine" applicationName="BlogEngine" />
    <add name="DbRoleProvider" type="BlogEngine.Core.Providers.DbRoleProvider, BlogEngine.Core" connectionStringName="BlogEngine" />
  </providers>
</roleManager>

5. ADD A TRAILING SLASH WHEN YOU GO TO THE ADMIN

There’s a problem with the routing in the admin and you’ll get a blank page if you don’t add a trailing slash, aka http://locahost/blog/admin/ 

 

With those fixes, you should be able to get the best blogging platform written in .NET up and running!

BlogEngine.NET Provider Migration

December 11 2014

I recently migrated this blog (which runs on Blog Engine.NET 2.8) from the XMLProvider to the DBProvider. I followed the instructions here: http://www.nyveldt.com/blog/page/blogenginenet-provider-migration which almost worked, but I had to make a couple changes:

BlogService.Provider.FillCategories and BlogService.Provider.LoadSettings requires that you pass the current blog, so those lines just got changed to

BlogService.Provider.FillCategories(Blog.CurrentInstance)

BlogService.Provider.LoadSettings(Blog.CurrentInstance)

Then, I had to manually update the GUID of the blog itself in SQL. Basically, after you run the DB create script and run the migration page, you grab the id of your old blog and update the bd_blogs table:

UPDATE [dbo].[be_Blogs]
   SET [BlogId] = 'your new id'
WHERE [BlogId] = '27604F05-86AD-47EF-9E05-950BB762570C'
GO

And, walla, now I’m running on SQL Server!

Simple Unit Tests For HDInsight C# SDK

October 3 2014

I have been working on a project using the .NET SDK for Hadoop.  I wanted to add some unit tests to the project, so I ended up writing some fakes for HDInsightClient, JobSubmissionClientFactory and JobSubmissionClient. I was hoping I might be able to reuse some fakes from the SDK git repo, but it seems like their unit tests actually stand up an instance of Hadoop. I didn’t want to actually stand up an instance; I’m treating Hadoop like a black box and I’m more interested in getting code coverage on all the C# code around the calls to Hadoop.

For my fake of IHDInsightClient, I only implemented CreateCluster() and DeleteCluster(), nothing fancy.

I had to make my own interface and wrapper to have a factory that would make a JobSubmissionClient (which is the same thing that the SDK did for its cmdlets):

public interface IAzureHDInsightJobSubmissionClientFactory
{
    IJobSubmissionClient Create(IJobSubmissionClientCredential credentials);
}    
    

Then, for the service itself, I implement this interface using the static JobSubmissionClientFactory:

public class AzureHDInsightJobSubmissionClientFactory : IAzureHDInsightJobSubmissionClientFactory
{
    public IJobSubmissionClient Create(IJobSubmissionClientCredential credentials)
    {
        return JobSubmissionClientFactory.Connect(credentials);
    }
}

Whenever I need a JobSubmissionClient, I get one using my wrapper.

In the case of my fake, I have the factory return a new fake job submission client:

public class FakeJobSubmissionClientFactory : IAzureHDInsightJobSubmissionClientFactory
{
    public Microsoft.Hadoop.Client.IJobSubmissionClient Create(Microsoft.Hadoop.Client.IJobSubmissionClientCredential credentials)
    {
        return new FakeJobSubmissionClient();
    }
}

Finally, for my FakeJobSubmissionClient, I do need to fake the work that the job does in Hadoop. In this case, it writes a file to blob storage as a result of the Hive query it runs. So, since my fixture has a static reference to a fake blobClient, I was able to fake the work that Hadoop would do in my implementation of CreateHiveJob(HiveJobCreateParameters hiveJobCreateParameters).

With all these fakes, I then wired up dependency injection in my UnityContainer and I was good to go. And now I have much more confidence that future changes to this codebase won’t cause regressions.

Prototyping With Hive Using HDInsight

September 19 2014

I’ve been doing a lot of prototyping with Hive lately and I really wanted to use an emulator to do the work.  For awhile, I was trying to do the prototyping using Powershell but kept getting NotSupported exceptions. 

I finally decided to simply program against Hive itself which has turned out to work just great. I’m using Visual Studio to write my HQL as .sql files and getting a nice color coded editing experience, since HQL and SQL are fundamentally the same as far as keywords.

Then, I have a cmd line open with a Hive prompt and I run my HQL queries as follows:

hive> source c:\hdp\temp\test2.sql;

Working great! Fast and free – I like it…

Connecting To The Azure Storage Emulator From The HDInsight Emulator

September 18 2014

I was following the Getting Started instructions on using the HDInsight emulator and got stuck trying to connect to the Azure Storage Emulator:

 

hadoop fs -ls wasb://temp@storageemulator 
ls: `wasb://temp@storageemulator': No such file or directory

Turns out that you must have a trailing slash, like this:

hadoop fs -ls wasb://temp@storageemulator/ 

Maybe that’ll help someone out there…

A Simple Box.Com C# API Wrapper

August 25 2014

I had a need to access Box.com programmatically to do a daily pull of log files that were posted to Box from a third party service. At first I thought the Box .NET SDK would be helpful, but I quickly realized it is entirely oriented to be used by apps with a UI, not headless apps like this.  So, I dove into the documentation

My first stumble was that the Box developer token expires in one hour. I was hoping I’d be able to use it as a const for my headless server application, but no luck.

So, I need to actually generate an access_token and a refresh_token. The only way to do that is go through their UI. Thanks to a very helpful post on StackOverflow I was able to generate a code that could be used to generate both an access_token and refresh_token (which last for 60 days).

By persisting the refresh_token, you can write code that gets a fresh access_token programmatically. So, basically, my wrapper has a method called Bootstrap which you pass the code that you get from copy/pasting it out of the querystring. And then it has RefreshBoxToken, which gets a new access_token if the access_token is expired.

Then, there are two additional wrappers that actually do work which I called GetFolderByID and GetFileAsStream. GetFolderByID assumes you have the folderID, which you can figure out from the Box UI itself.  Then, with a little JSON.NET, you can parse the response and get the list of files as a JArray:

JObject jObject = JObject.Parse(folderContents);
JArray jArray = jObject["item_collection"]["entries"] as JArray;

Then, you’ve got the power do download files!

I wrapped both calls in a generic DoBoxAPICall method. Below is the entire class that encapsulates the logic:

using ExportConvivaLogsToHadoopWebJob.Properties;
using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

namespace ExportConvivaLogsToHadoopWebJob
{

    public static class BoxAPIHelper
    {
        private const string boxApiUrl = "https://api.box.com/2.0/";
        private const string boxClientId = "YOUR_ID";
        private const string boxClientSecret = "YOUR_SECRET";
        private static readonly HttpClient _httpClient = new HttpClient();
        private static int retryCount = 0;
        private static Stream DoBoxCall(string url, HttpMethod httpMethod)
        {
            Stream stream;
            var request = new HttpRequestMessage() { RequestUri = new Uri(url), Method = httpMethod };
            request.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Authorization", "Bearer " + Settings.Default.boxAccessToken);
            var response = _httpClient.SendAsync(request).Result;
            if (!response.IsSuccessStatusCode && response.StatusCode == System.Net.HttpStatusCode.Unauthorized)
            {
                if (retryCount < 2)
                {
                    RefreshBoxToken();
                    retryCount++;
                    stream = DoBoxCall(url, httpMethod);
                    return stream;
                }
                else
                {
                    throw new Exception("Failed to connect to Box.");
                }

            }
            retryCount = 0;

            return response.Content.ReadAsStreamAsync().Result;
        }
        private static void RefreshBoxToken()
        {
            using (var request = new HttpRequestMessage() { RequestUri = new Uri("https://www.box.com/api/oauth2/token"), Method = HttpMethod.Post })
            {
                HttpContent content = new FormUrlEncodedContent(new[] 
                { 
                 new KeyValuePair<string, string>("grant_type", "refresh_token"), 
                 new KeyValuePair<string, string>("refresh_token", Settings.Default.boxRefreshToken),
                 new KeyValuePair<string, string>("client_id", boxClientId),
                 new KeyValuePair<string, string>("client_secret", boxClientSecret)
                
                }


                );
                request.Content = content;
                using (var response = _httpClient.SendAsync(request).Result)
                {
                    if (!response.IsSuccessStatusCode)
                    {
                        throw new Exception("Box refresh token failed. A human needs to go to a browser and generate a fresh authorization code.");
                    }
                    JObject jObject = jObject = JObject.Parse(response.Content.ReadAsStringAsync().Result);
                    Settings.Default.boxAccessToken = (string)jObject["access_token"];
                    Settings.Default.boxRefreshToken = (string)jObject["refresh_token"];
                    Settings.Default.Save();
                }
            }



        }
        public static string GetFolderById(string folderId)
        {
            string url = string.Format("{0}folders/{1}", boxApiUrl, folderId);
            Stream stream = DoBoxCall(url, HttpMethod.Get);

            StreamReader reader = new StreamReader(stream);


            return reader.ReadToEnd();
        }
        public static void Bootstrap(string boxAccessCode)
        {
            using (var request = new HttpRequestMessage() { RequestUri = new Uri("https://www.box.com/api/oauth2/token"), Method = HttpMethod.Post })
            {
                HttpContent content = new FormUrlEncodedContent(new[] 
                { 
                 new KeyValuePair<string, string>("grant_type", "authorization_code"), 
                 new KeyValuePair<string, string>("code", boxAccessCode),
                 new KeyValuePair<string, string>("client_id", boxClientId),
                 new KeyValuePair<string, string>("client_secret", boxClientSecret)
                
                }


                );
                request.Content = content;
                var response = _httpClient.SendAsync(request).Result;
                if (response.IsSuccessStatusCode)
                {
                    JObject jObject = jObject = JObject.Parse(response.Content.ReadAsStringAsync().Result);
                    Settings.Default.boxAccessToken = (string)jObject["access_token"];
                    Settings.Default.boxRefreshToken = (string)jObject["refresh_token"];
                    Settings.Default.Save();

                }

            }

        }
        public static Stream GetFileAsStream(string fileId)
        {
            string url = string.Format("{0}files/{1}/content", boxApiUrl, fileId);
            return DoBoxCall(url, HttpMethod.Get);
        }
    }
}

Maybe that’ll help someone out there…

Beware: Hadoop C# SDK Inserts Line Breaks, Tabs That Break Your Queries

August 22 2014

After banging my head against the wall for many hours, I finally figured
out that .NET is adding escaped carriage returns, aka \r\n when the
queries are sent to HDInsight, which is causing the queries to fail. My code was loading the queries from files on disk like this:

string query = string.Empty;
using (var fs = new StreamReader("CreateTempTable.hql"))
{
    query = fs.ReadToEnd();
}

I figured this out by looking at userArgs
file in the templeton-hadoop directory to see what the jobs looked
like, and they appear like this:

"ADD JAR wasb:///user/jars/csv-serde-1.1.2-0.11.0-all.jar;
\r\nDROP TABLE IF EXISTS temp;\r\nCREATE EXTERNAL TABLE temp
(viewerId string, asset string, device_os string, country string, state 
string, city string, asn string, isp string, start_time_unix_time bigint,
startup_time_ms int) \r\nROW FORMAT serde 'com.bizo.hive.serde.csv.CSVSerde'
\r\nSTORED AS TEXTFILE LOCATION 'wasb:///temptable';\r\n\r\n               "

As you can see, the query is littered with escaped characters which causes the HIVE query to fail.
These same queries can be submitted via PowerShell no problem.

So, basically, I removed all the linebreaks in Notepad for my HQL and everything worked.