find a location for property in a new city
Showing posts with label Optimisation. Show all posts
Showing posts with label Optimisation. Show all posts

Monday, 28 May 2012

How to turn off logging on Combres

I have been using Combres for my .NET app to minify, combine and compress my JavaScript and CSS. So far I have found it awesome.

It really does do everything it claims. Minifing your scripts and CSS, then combines them all so that all the code is sent via one HTTP request (well one for CSS and one for JavaScript). It also handles GZip or deflate compression should the request accept it. It is easy to set up since it is now available through NuGet and using it will make YSlow smile when it scans your site.

Logging

One thing that is annoying though is that Combres seems to log everything it does. "Use content from content's cache for Resource x...Use content from content's cache for Resource y" This is fine in development but unnecessary in production so I wanted to turn it off but couldn't find how to do this in any documentation.

The way I managed to turn off logging was to find the line in the web.config that combres put in that looks like this:

<combres definitionUrl="~/App_Data/combres.xml" logProvider="Combres.Loggers.Log4NetLogger" />

You simply need to remove the logProvider so it looks like this:

<combres definitionUrl="~/App_Data/combres.xml" />

If you still want logging on your development environment you can simply remove the logProvider attribute in a web.config transform.

Follow britishdev on Twitter

Thursday, 19 May 2011

Image resizing, cropping and compression using .NET

If you need to resize an image or crop an image or compress an image or even do all three you will find you can do all of those things using the .NET framework's System.Drawing classes. In this tutorial I will talk you through the steps required to do all three to transform an large image to an image fit for your website.

This is the example scenario. You have been asked to make a page that enables a user to upload any image they want so that it can be manipulated to ensure it is 200px x 200px and compressed to a smaller file size. I won't talk you though how to make the uploading part as there are a plethora of blogs discussing that already. What I will discuss is how to resize crop and compress that image.

Say you have an image that is 1000px x 800px. There are 3 main steps to converting it to a compressed 200px x 200px image:

Resizing an image

To resize it the key is that it needs to keep the same aspect ratio whilst being a lot smaller.

So the expected outcome of this stage will be that the shortest of the two dimensions will be 200px, the other will be larger and the aspect ratio will remain the same. Begin code:
private byte[] GetCroppedImage(byte[] originalBytes, Size size, ImageFormat format)
{
    using (var streamOriginal = new MemoryStream(originalBytes))
    using (var imgOriginal = Image.FromStream(streamOriginal))
    {
        //get original width and height of the incoming image
        var originalWidth = imgOriginal.Width; // 1000
        var originalHeight = imgOriginal.Height; // 800

        //get the percentage difference in size of the dimension that will change the least
        var percWidth = ((float)size.Width / (float)originalWidth); // 0.2
        var percHeight = ((float)size.Height / (float)originalHeight); // 0.25
        var percentage = Math.Max(percHeight, percWidth); // 0.25

        //get the ideal width and height for the resize (to the next whole number)
        var width = (int)Math.Max(originalWidth * percentage, size.Width); // 250
        var height = (int)Math.Max(originalHeight * percentage, size.Height); // 200

        //actually resize it
        using (var resizedBmp = new Bitmap(width, height))
        {
            using (var graphics = Graphics.FromImage((Image)resizedBmp))
            {
                graphics.InterpolationMode = InterpolationMode.Default;
                graphics.DrawImage(imgOriginal, 0, 0, width, height);
            }
        }
    }
}

Cropping an image

After the last step you are left with an image that is 250px x 200px. As you will notice that is still an aspect ratio of 5:4 so it does not look squashed. However this still isn't the right size so you will now need to crop it.

You are now intending to cut off the excess from both sides to reduce the width whilst leaving the height the same. This will leave you with a 200px x 200px image. Code on:
private byte[] GetCroppedImage(byte[] originalBytes, Size size, ImageFormat format)
{
    using (var streamOriginal = new MemoryStream(originalBytes))
    using (var imgOriginal = Image.FromStream(streamOriginal))
    {
        //get original width and height of the incoming image
        var originalWidth = imgOriginal.Width; // 1000
        var originalHeight = imgOriginal.Height; // 800

        //get the percentage difference in size of the dimension that will change the least
        var percWidth = ((float)size.Width / (float)originalWidth); // 0.2
        var percHeight = ((float)size.Height / (float)originalHeight); // 0.25
        var percentage = Math.Max(percHeight, percWidth); // 0.25

        //get the ideal width and height for the resize (to the next whole number)
        var width = (int)Math.Max(originalWidth * percentage, size.Width); // 250
        var height = (int)Math.Max(originalHeight * percentage, size.Height); // 200

        //actually resize it
        using (var resizedBmp = new Bitmap(width, height))
        {
            using (var graphics = Graphics.FromImage((Image)resizedBmp))
            {
                graphics.InterpolationMode = InterpolationMode.Default;
                graphics.DrawImage(imgOriginal, 0, 0, width, height);
            }

            //work out the coordinates of the top left pixel for cropping
            var x = (width - size.Width) / 2; // 25
            var y = (height - size.Height) / 2; // 0

            //create the cropping rectangle
            var rectangle = new Rectangle(x, y, size.Width, size.Height); // 25, 0, 200, 200

            //crop
            using (var croppedBmp = resizedBmp.Clone(rectangle, resizedBmp.PixelFormat))
            {
            }
        }
    }
}

Compressing the image

You now have the image to the correct size but for the web it is not really optimised.

You now want to compress the image down from the current 4 bytes per pixel (32bit) image, which would be ~156KB (for a 200x200 image!). Time to compress:
private byte[] GetCroppedImage(byte[] originalBytes, Size size, ImageFormat format)
{
    using (var streamOriginal = new MemoryStream(originalBytes))
    using (var imgOriginal = Image.FromStream(streamOriginal))
    {
        //get original width and height of the incoming image
        var originalWidth = imgOriginal.Width; // 1000
        var originalHeight = imgOriginal.Height; // 800

        //get the percentage difference in size of the dimension that will change the least
        var percWidth = ((float)size.Width / (float)originalWidth); // 0.2
        var percHeight = ((float)size.Height / (float)originalHeight); // 0.25
        var percentage = Math.Max(percHeight, percWidth); // 0.25

        //get the ideal width and height for the resize (to the next whole number)
        var width = (int)Math.Max(originalWidth * percentage, size.Width); // 250
        var height = (int)Math.Max(originalHeight * percentage, size.Height); // 200

        //actually resize it
        using (var resizedBmp = new Bitmap(width, height))
        {
            using (var graphics = Graphics.FromImage((Image)resizedBmp))
            {
                graphics.InterpolationMode = InterpolationMode.Default;
                graphics.DrawImage(imgOriginal, 0, 0, width, height);
            }

            //work out the coordinates of the top left pixel for cropping
            var x = (width - size.Width) / 2; // 25
            var y = (height - size.Height) / 2; // 0

            //create the cropping rectangle
            var rectangle = new Rectangle(x, y, size.Width, size.Height); // 25, 0, 200, 200

            //crop
            using (var croppedBmp = resizedBmp.Clone(rectangle, resizedBmp.PixelFormat))
            using (var ms = new MemoryStream())
            {
                //get the codec needed
                var imgCodec = ImageCodecInfo.GetImageEncoders().First(c => c.FormatID == format.Guid);

                //make a paramater to adjust quality
                var codecParams = new EncoderParameters(1);

                //reduce to quality of 80 (from range of 0 (max compression) to 100 (no compression))
                codecParams.Param[0] = new EncoderParameter(Encoder.Quality, 80L);

                //save to the memorystream - convert it to an array and send it back as a byte[]
                croppedBmp.Save(ms, imgCodec, codecParams);
                return ms.ToArray();
            }
        }
    }
}

There you go. Your massive image is looking good to the correct size whilst being a much leaner version of former self. Congratulations!

Update

In line 24, you can see I have set the InterpoltionMode of the graphics object to InterpolationMode.HighQualityBicubic before doing the initial resize. I previously had this set to Default but I noticed that straight diagonal lines in resized images were coming out jagged and they didn't look pretty. HighQualityBicubic is meant to be "the best" form of interpolation according to Microsoft so I sided with that. I can see a big improvement in the quality so I would recommend it too.

Follow britishdev on Twitter

Monday, 28 March 2011

How to delete an entity by ID in Entity Framework

It may seem unnatural to delete Entities in Entity Framework. In the days of stored procedures I used to just pass an ID where as now you need to get the entity before deleting it. It seems inefficient - you're basically making two database hits when surely you can do it in one database call, right? Well it is difficult but entirely possible.

Since you do not have the entity you will need you will effectively be updating a detached entity which is a technique in itself. The trick here is that we are making a fake entity that shares the Entity Key (EF version of a primary key) with the one that needs deleting. Then we mark it as deleted and SaveChanges.

public void DeleteByID(object id)
{
    //make fake entity and set the ID to the id passed in
    var car = new Car();
    car.ID = id;

    using (var context = new MyDataContext())
    {
        //add this entity to object set
        context.CreateObjectSet<Car>().Attach(car);
        //mark it as deleted
        context.ObjectStateManager.ChangeObjectState(car, EntityState.Deleted);
        //save changed - effectively saying delete the entity with this ID
        context.SaveChanges();
    }
}

It is worth noting that I am following my own twist on the Repository pattern for data access so I have an "EntityRepository" that manages everything EF related. This is where I will make my generic "DeleteByID" method. My EntityRepository uses generics to specify the type of entities that are dealt with so this has to be completely reusable by any entity set (TEntity) or object context (TContext).

How to Delete by ID in a generic way

public void DeleteByID(object id)
{
    var item = new TEntity();
    var itemType = item.GetType();

    using (var context = new TContext())
    {
        //get the entity container - e.g. MyDataContext - this will have details of all its entities
        var entityContainer = context.MetadataWorkspace.GetEntityContainer(context.DefaultContainerName, DataSpace.CSpace);
        //get the name of the entity set that the type T belongs to - e.g. "Cars"
        var entitySetName = entityContainer.BaseEntitySets.First(b => b.ElementType.Name == itemType.Name).Name;
        //finding the entity key of this entity - e.g. Car.ID
        var primaryKey = context.CreateEntityKey(entitySetName, item).EntityKeyValues[0];
        //using Reflection to get and then set the property that acts as the entity's key
        itemType.GetProperty(primaryKey.Key).SetValue(item, id, null);
        //add this entity to object set
        context.CreateObjectSet<TEntity>().Attach(item);
        //mark it as deleted
        context.ObjectStateManager.ChangeObjectState(item, System.Data.EntityState.Deleted);
        //save changed - effectively saying delete the entity with this ID
        context.SaveChanges();
    }
}

These both translate to the following SQL:
delete from [dbo].[Cars]
where  ([ID] = 123)
And there it is - a reusable delete by ID function with no SELECT!

Follow britishdev on Twitter

Friday, 4 March 2011

Why you shouldn't use singleton DataContexts in Entity Framework

I have been struggling recently with putting an end to a problem caused my optimistically keeping the same DataContext alive in a singleton pattern. Let me tell you what I found and why I think it is bad.

Getting it wrong will cripple your site

The first and foremost reason why you shouldn't use it is that if you set it up wrong the repercussions are critical. Here's my horror story: after ages of head scratching, wondering why the CPUs are continuously nearing 100% and memory leaks are ubiquitous I finally figured out that DataContexts were not being disposed of. Of course this is by design - lovely singleton pattern...

Anyway, not only were they not being disposed of they were being instantiated with every access of the data. So we ended up with piles of open connections, which I assume were being terminated randomly by some critical process. This causes huge memory utilisation and huge CPU usage trying to reallocate memory.

Fine so it was implemented incorrectly, but it is practically impossible to see that this is happening during development without shelling out for a decent entity framework profiler (like Entity Profiler - it is very much worth it if you don't trust EF).

Data isn't in sync sometimes...

When we were using the singleton well the data often seemed out of sync with the database. It's as if the ObjectContext knew best and didn't really care what the database had to say. This is not wise when you are working on a web farm since your different servers will have a different view of what the data is in its memory.

The way to resolve this is to begin a new connection... In comes the problem of multiplying open and undisposed connections.

Where's the performance gain anyway?

Has anyone any idea of how expensive it is to open a DataContext connection? Answer: "A DataContext is lightweight and is not expensive to create" [citation]. So why is everyone so intent on saving the hassle of creating a DataContext?

You are probably saving a few 10s of milliseconds. The word micro optimisation springs to mind - in which case you probably shouldn't be using Entity Framework.

It allows poor coding

Having a DataContext that is never closed allows you to lazy load whenever you want. You may have left your Service and now be in your Controller or worse still, your View. Accessing the database from a View is asking for performance problems as I'm sure you didn't allow that intentionally. It is most likely because you forgot to eager load all the data you need to populate your view.

This also makes it hard to keep things nice and streamlined since you don't know where all your queries are coming from.

Conclusion

If you are a guru of Entity Framework and you think a Singleton DataContext is for you then by all means go for it. If you aren't convinced you need it and you're just experimenting you could encounter unexpected and serious problems. Don't use singleton because you think it sounds cool!

Also, during fixing up my problems I found the Entity Profiler invaluable. It tracks how many connections have been opened and closed (hopefully they will be equal). It also shows all the actual SQL that is being produced, the results, the query plan and even tips on how to sort your query out. Invaluable!

Follow britishdev on Twitter

Thursday, 3 March 2011

Visual Studio's load test and 400 Bad request errors

When trying some load testing with Visual Studio Ultimate edition. My load test received lots of 400 bad request errors. But why?

I had a look at my page that was claiming to be full of errors and it was fine. Even Fiddler reported that there were no 400's being returned from my page.

I took a look more closely at the URL that was returning the 400 and it was not the same as the one I put into the Web Performance test. It had suffixed my URL with %3C%25 - I had a look and this is URL encoding for <%. The 'start writing code' symbol in web forms pages/views.

Workaround to fix this 400 Bad Request %3C%25 problem

I routed about my View and found it could be a problem with the ad calls in the page. They make https ad calls so maybe this is the problem since I heard something about https not being supported properly in load test.

So I stripped the ad call parts out of my View and everything is working fine now.

Obviously this will now affect my page speed so not very accurate. Not impresses given the cost of Visual Studio Ultimate

Follow britishdev on Twitter

Friday, 28 January 2011

Caching. Absolute expiration or Sliding scale expiration?

You can set a cache with a temporal expiration in one of two ways:

Absolute expiration

This mean you set a time in the future in which it will expire e.g. DateTime.Now.AddMinutes(5). So the cache will remove the object from memory at that particular time in the future.

Sliding scale expiration

This means you set a length of time that the cache should keep the object in memory for from the last time it was accessed e.g. new TimeSpan(0, 0, 5, 0). That is 5 mins. So the object will be removed from cache if it hasn’t been accessed for 5mins or more.

Which temporal caching method to use?

I believe Absolute expiration is the one to use for most web page cases, unless there is a good reason not to. Imagine you are using a 5 minute sliding scale cache expiration on a popular page that gets say 50 hits/sec, chances are that this object will never expire from the cache.

How long to cache for?

Also avoiding huge expirations of hours or days on not outrageously expensive data would be an idea too. Although the benefits of large expirations are that you have to retrieve the data less often I think that reducing it from 50*60*5 = 15000 requests/5mins down to 1 request/5mins is efficient enough, without suffering from stagnant data.

Obviously, the length of how long to set your expiration for and your method of expiration is will depend on how expensive your data is and possibly other specific requirements but hopefully you can now make a more informed decision.

Follow britishdev on Twitter

Tuesday, 11 January 2011

Optimising Entity Framework - Don’t query from base entity

Be careful to query the derived type when dealing with an entity data model that involves inheritance. You can massively reduce query complexity by making sure you avoid querying the base type of your model.

Say there is a model with a base type of Person and three derived types of Customer, StaffMember, Client. It doesn't matter if your inheritance model is Table per Inheritance or Table per Type (TPT) but say for example it is Table per Hierarchy (TPH) and the condition for Customer is that an OrdersID column is not null. Also say there is a quality that of all Customers that they will have a guaranteed value of PersonType == 1.

So what is the difference between:

context.People.Where(p => p.MessageType == 1) and context.People.OfType()?

Aside from the better code the difference in generated SQL is massive. For some reason Entity Framework will generate SQL that works out which derived type each of the rows in the Person table are even though I'm asking for just the base type. This means unnecessarily joining on a number of tables if you’re using TPT or evaluating many conditional columns if you’re using TPH.

I tried the different approaches on quite a large Entity Data Model of mine. There is one base type with 5 derived types and a few other entities being included (joined). Using the first query from above the generated query was 28224 characters long. The revised query was 14177. That's half the size!

Using query length is a poor way of evaluating a query because most of EF queries are explicit query lengths which don’t actually add to complexity but believe me – in the case the saving of query length was almost exclusively down to removing pointless complexity.

Follow britishdev on Twitter