Search
Recent Tweets
Friday
Oct112013

A Remotely Managed Bing Image Search Wallpaper App - Part 2

Part 1 gave an overview introduction to the app and covered job scheduling, searching for images on Bing, and downloading images. Part 2 covers setting the wallpaper, remote app settings, remote logging and more.

Setting the Wallpaper

The switch wallpaper job is scheduled to start running later than the download job to give time for images to download. Plus it makes for a better prank to wait for the wallpaper change to set in later. The job starts by checking the app's enabled status and bailing if disabled. It then deletes old images according to app settings and the create date of the local images. Next if AppSettings.Instance.WallpaperOverrideUrl is set that image is downloaded and used as the background. Otherwise the job randomly picks one of the images downloaded from the Bing image search results.

The actual code that sets the wallpaper isn't that interesting; it's a modified version from this StackOverflow post. Finally the job uses the metadata manager to get the image URL where the local filename came from; that's used in the remote logging so we can see what image we set the wallpaper to.
internal class SwitchWallpaperJob : IJob
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();

	public const string OutputPathKey = "OutputPath";

	public async void Execute(IJobExecutionContext context)
	{
		try
		{
			if (!AppSettings.Instance.CheckStatus()) return;

			var path = AppSettings.ImagePath.FullName;
			Ensure.That(path, "path").IsNotNullOrWhiteSpace();

			ImageCleanup.Execute();

			var imageFile = await GetWallpaperImageFile(path);
			if (null == imageFile) return;

			//TODO: setting for wallpaper style or smart set via desktop & image size
			Logger.Troll("Changing wallpaper to {0}", imageFile.Name);
			Wallpaper.Set(imageFile.FullName, Wallpaper.Style.Centered);

			var meta = new MetadataManager().Get(imageFile.FullName);
			var changedTo = (null != meta) ? meta.RemoteLocation + " via term " 
				+ meta.Term : imageFile.Name;

			Logger.Troll("Changed wallpaper to {0}", changedTo);
		}
		catch (Exception ex)
		{
			Logger.Error(ex.ToString());
			throw new JobExecutionException(ex);
		}
	}

	private static async Task<FileInfo> GetWallpaperImageFile(string path)
	{
		if (!string.IsNullOrWhiteSpace(AppSettings.Instance.WallpaperOverrideUrl))
		{
			Logger.Troll("Using image override url {0}", 
				AppSettings.Instance.WallpaperOverrideUrl);
			var uri = new Uri(AppSettings.Instance.WallpaperOverrideUrl);
			var file = uri.Segments[uri.Segments.Length - 1];
			var ext = file.Substring(file.LastIndexOf(".") + 1);
			var outFilename = new FileInfo(Path.Combine(
				AppSettings.ImagePath.FullName, "Override." + ext));
			await ImageDownloader.DownloadImage(AppSettings.Instance.WallpaperOverrideUrl, 
				outFilename.FullName);
			return outFilename;
		}

		Logger.Troll("Enumerating files in {0}", path);
		var files = Directory.GetFiles(path, "*.jpg").ToList();

		if (!files.Any())
		{
			Logger.Troll("No images yet; will try again later");
			return null;
		}

		Logger.Troll("Found {0} wallpaper images in {1}", files.Count, path);
		var r = new Random();
		var randFile = new FileInfo(files[r.Next(0, files.Count)]);
		return randFile;
	}
}

Remote App Settings

The only app.config setting the app uses is ConfigSource which is expected to be a URL to retrieve the settings from. In this way we can change the wallpaper app behavior remotely which is useful when you're pranking someone or if you're using yourself to synchronize wallpaper settings between multiple computers. DropBox, SkyDrive, Google Drive and the like are simple solutions for hosting the file though that might trace the app back to you.

The config data is in JSON format and a sample follows. For pranking purposes I might use "Justin Bieber", "Justin Bieber Wallpaper", "Justin Bieber 2013" etc. in search terms but for testing or personal use, not so much. For debugging I set the all the job intervals to short amounts but use much longer durations for deployment.


Trying It Out

Images are downloaded to AppData\Local\WIO; WIO is the app name and stands for Windows Image Optimization :) and keeps the process name short and esoteric.



Also in this directory is the metadata index file, mapping image filenames to the source URL downloaded from, in BSON format with some base64 encoding

Fetching App Settings

When retrieving the settings, a simple check is done on the data returned from the HTTP call to see if it is JSON. If not, the code assumes it is encrypted. When using the app in a prank fashion, encrypting the data helps mask app activity if the target discovers the URL in the app config or notices the HTTP traffic. Of course encrypting the config makes changing settings more difficult, so the app also contains a FileEncrypt command line app to easily encrypt the data. Adding a Send To shortcut means only having to right-click a plain text JSON file to generate an encrypted version in the same directory.
	public enum AppStatus { Enabled, Paused, Disabled }

    public sealed class AppSettings
    {
        private static volatile AppSettings _instance;
        private static readonly object SyncRoot = new Object();

        private AppSettings()
        {
            this.Search = new SearchSettings();
            this.Job = new JobSettings();
            this.Log = new LogSettings();
        }

        public string ImageDeleteAfterTimespan { get; set; }
        public string WallpaperOverrideUrl { get; set; }

        [JsonConverter(typeof(StringEnumConverter))]
        public AppStatus Status { get; set; }

        public static AppSettings Instance
        {
            get
            {
                if (_instance == null)
                {
                    lock (SyncRoot)
                    {
                        _instance = new AppSettings();
                    }
                }
                return _instance;
            }
        }

        public static async Task<AppSettings> Load()
        {
            string configData;
            using (var client = new HttpClient())
            {
                configData = await client.GetStringAsync(
					ConfigurationManager.AppSettings["ConfigSource"]);

                var isJsonPlain = (configData.TrimStart().StartsWith("{"));
                if (!isJsonPlain)
                    configData = CryptoManager.Decrypt3DES(configData);
            }

            lock (SyncRoot)
                _instance = JsonConvert.DeserializeObject<AppSettings>(configData);

            return _instance;
        }

        public bool CheckStatus()
        {
            if (Status == AppStatus.Disabled)
            {
                Application.Exit();
                return false;
            }

            return Status == AppStatus.Enabled;
        }

        [JsonIgnore]
        public static DirectoryInfo ImagePath
        {
            get
            {
                var path = Path.Combine(Environment.GetFolderPath(
					Environment.SpecialFolder.LocalApplicationData), "WIO");
                var di = new DirectoryInfo(path);
                if (!di.Exists) di.Create();
                return di;
            }
        }

        public SearchSettings Search { get; set; }
        public JobSettings Job { get; set; }
        public LogSettings Log { get; set; }
    }

Obfuscating the Code

With punking a fellow IT coworker, I wanted to obfuscate the code so it'd be more difficult to figure out the app's logic should it be discovered and viewed in a disassembler. For that I used eazfuscator.

It was free to use for 30 days which was all I needed and it's available via NuGet. It slowed the build down some but it's only done in Release mode and usually that's only done at the very end.



Unfortunately when running in Release mode with the obfuscated code I received the below exception. Removing the obfuscation build step and running in Release mode removed the exception so that means the process changed the behavior of the code.
Quartz.SchedulerException was unhandled by user code
  HResult=-2146233088
  Message=Repeat Interval cannot be zero.
  Source=Quartz
  StackTrace:
       at Quartz.Impl.Triggers.SimpleTriggerImpl.Validate() in c:\Work\OpenSource\quartznet\src\Quartz\Impl\Triggers\SimpleTriggerImpl.cs:line 727
       at Quartz.Core.QuartzScheduler.ScheduleJob(IJobDetail jobDetail, ITrigger trigger) in c:\Work\OpenSource\quartznet\src\Quartz\Core\QuartzScheduler.cs:line 720
       at Quartz.Impl.StdScheduler.ScheduleJob(IJobDetail jobDetail, ITrigger trigger) in c:\Work\OpenSource\quartznet\src\Quartz\Impl\StdScheduler.cs:line 262
       at   .       ()
       at   . (Type  )
       at System.Collections.Generic.List`1.ForEach(Action`1 action)
       at   . ()
       at  . (Task`1  )
       at System.Threading.Tasks.ContinuationTaskFromResultTask`1.InnerInvoke()
       at System.Threading.Tasks.Task.Execute()
  InnerException:
This turned out to be an issue with the obfuscator not handling JSON serialization correctly. It was fixed in a later version of the tool but the latest version of the NuGet package was an older version. Any use of reflection can also be an issue so lesson learned - always fully regression test when using obfuscation. This tool now appears to be more commercialized but I applaud the quality and the after hours support I received when reporting an issue.

Quick Installation

To reduce the chance of getting caught, I created a simple install script that can be quickly run via a flash drive or from a network share. Sooner or later I knew my target would forget a workstation lock and this way even a quick bathroom or coffee trip was plenty of time.

First a post build event to copy the files needed for deployment to an Install\bin folder:

if not exist "$(SolutionDir)Install\bin" mkdir "$(SolutionDir)Install\bin"
del /f /q $(SolutionDir)Install\bin\*.*
xcopy /r /d /i /s /y /exclude:$(SolutionDir)Install\InstallStageExclude.txt
  $(TargetDir)*.* $(SolutionDir)Install\bin

Next the script would copy the contents to the appropriate location. Normally I'd use PowerShell but batch files are just faster with being able to double-click and run worry-free.



Remote Logging

Logging was needed to monitor the app to see what wallpaper was set on the victim's computer or to view problems if things weren't working correctly. Local computer logging didn't do any good as I wouldn't have access to the data later, so I evaluated a couple of cloud based logging solutions.

Loggr.net

Loggr.net is what I started with. It was intuitive and easy to use. Where it broke down for my needs was the API limits of the free account - 100 log records per day. So I ended up changing the code to only send important log events there based on log type (error, warning, custom). The code to send logging data wasn't bad, though it ended up being more than typical for more control over the logged events.


Loggly

Next I tried Loggly which allowed up to 200 MB/day for free. I found it's website UI to be a bit counter-intuitive though they were upgrading to a GEN 2 platform near the end of my usage. I liked the JSON logging and the command line style website. It offered a lot of searching functionality and from an API perspective it was easy to work with the log data programmatically. The code to send logging data was dead simple, though I was doing more with the Loggr.net API.



In Conclusion

Was it worth it?

Yes:
  • It's a fun prank that keeps on giving for you.
  • I have an automated, randomized source of wallpaper for myself, synced between computers.
  • Most importantly I got to play with different tech, learn new things and do some coding for fun.

Disclaimers

This was a fun educational experiment and one-time prank. I only offer the source code here; no binaries, setups, support, or API keys.

Should you use some or all of this or do something similar, keep in mind:
  • Depending on your use and legalese interpretation, you may be violating Bing's terms of use.
  • Web search is like a box of chocolates; you never know what you're gonna get.
  • Used in prank fashion, you may run the risk of offending someone with a random image, on top of already annoying him/her.
  • Used at work you may slow down someone's machine or the work network with downloading hundreds of high resolution images in a short time span.
Friday
Oct112013

A Remotely Managed Bing Image Search Wallpaper App - Part 1

At a previous job I made the mistake of leaving my computer unlocked on rare occasions and I ended up getting Biebered. We were constantly trolling each other and playing pranks which kept things fun. Changing someone's wallpaper is pretty basic so I decided to take it up a notch for striking back. One day I had too much time on my hands and started a remotely controlled wallpaper changing app powered by online image searches.

The Highlights

  • Powered by Bing Image Search API for variety
  • Windowless Windows background app
  • Remotely managed / configured / controlled
  • Silently changes victim's desktop wallpaper at scheduled intervals
  • Jobs scheduled with Quartz.net
  • HTTPS remote logging using both loggly.com and loggr.net
  • Encrypted configuration
  • Release compiled code obfuscation
  • Sets itself to run at Windows startup
  • Allows explicitly specifying an image URL to be used as background
  • Delete previously downloaded images by age according to remote config

Initial Decisions

App Type

I was pretty sure I would not be able to set the desktop wallpaper if the app was a Windows service but that didn't stop me from trying it :). Yeah it didn't work. In the old days you could just set the service property "Allow service to interact with desktop" but security is tighter with modern operating systems and that's a good thing. Maybe there's still a way to make that work but even if there was it probably wouldn't be worth the hassle. It would set off security alarms, it requires more hassle to install/uninstall, and similar functionality can be achieved with a Windows app that doesn't show any UI.

Image Sources

I considered different options for where to get the wallpaper images from including a preselected collection bundled with the app, reading from a network share, reading a URL from a remote config file, and using Google Image Search or Bing Image Search. I started out with Google's Image Search API and quickly found that to be a dead end, at least for any free version that didn't involve web scraping. Bing's Image Search API provided a free, quality, easy to use and diverse source of images. Later I decided to compliment that with the ability to override random images from a search with a specific image URL, if so desired.

App Lifecycle Skeleton

Rather than use any hidden form, the app spins up an ApplicationContext in the Main method of Program.cs with Application.Run(new AppContext()). On startup it first loads app settings from a remote source and then sets up a job schedule for work to be fired off later.
internal class AppContext : ApplicationContext
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();
	private readonly JobScheduler _scheduler = new JobScheduler();

	public AppContext()
	{
		AppSettings.Load().ContinueWith(AfterSettingsLoad);
	}

	private void AfterSettingsLoad(Task<AppSettings> task)
	{
		if (AppSettings.Instance.Status == AppStatus.Disabled)
		{
			Application.Exit();
			return;
		}

		Logger.Info("Setting up scheduler");
		RegisterAppForWindowsStartup();
		_scheduler.Setup();
		ImageCleanup.Execute();
	}

	protected override void ExitThreadCore()
	{
		Logger.Info("In ExitThreadCore");
		base.ExitThreadCore();
		AppTeardown();
	}

	private void AppTeardown()
	{
		Logger.Info("Tearing down app");
		if (null != _scheduler)
		{
			Logger.Info("Disposing scheduler");
			_scheduler.Dispose();
		}

		ImageCleanup.Execute();

		RegisterAppForWindowsStartup();
	}

	private static void RegisterAppForWindowsStartup()
	{
		if (!DebugMode.IsDebugging)
			WindowsStartup.Register();
	}
}

Job Scheduling

Job scheduling is done with Quartz.net and JobScheduler starts it's scheduler. It then scans the app's assembly for any scheduler classes and instantiates each to setup the scheduling for the different jobs.
internal class JobScheduler : DisposableObject
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();

	public void Setup()
	{
		Logger.Info("Creating job scheduler");
		ISchedulerFactory schedFact = new StdSchedulerFactory();
		Scheduler = schedFact.GetScheduler();
		Scheduler.Start();

		var types = Assembly.GetExecutingAssembly().GetTypes()
			.Where(x => x.BaseType == typeof(ScheduleBase)).ToList();
		Logger.Info("Found {0} schedules. Setting up each", types.Count);
		types.ForEach(t=> ((ScheduleBase) Activator.CreateInstance(t, Scheduler)).Setup());
		Logger.Debug("Schedules setup");
	}

	private IScheduler Scheduler { get; set; }
   
	protected override void DisposeManagedResources()
	{
		if (null != this.Scheduler)
			this.Scheduler.Shutdown(waitForJobsToComplete: false);
	}
}
There are jobs for refreshing remote app settings, downloading images from Bing Image Search and for changing the wallpaper. A sample job schedule builder for downloading images:
internal class DownloadSchedule : ScheduleBase
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();

	public DownloadSchedule(IScheduler scheduler) : base(scheduler)
	{
	}

	public override void Setup()
	{
		Logger.Info("Setting up download images job");

		if (!AppSettings.Instance.Search.Enabled)
		{
			Logger.Info("Search and download images isn't enabled; exiting");
			return;
		}

		var job = JobBuilder.Create<DownloadImagesJob>()
			.WithIdentity("downloadImages")
			.Build();

		var trigger = TriggerBuilder.Create()
			.WithIdentity("downloadImagesTrigger")
			.StartAt(AdjustOffset(DateBuilder.EvenMinuteDateAfterNow()))
			.WithSimpleSchedule(x =>
				x.WithIntervalInMinutes(AppSettings.Instance.Job.DownloadImagesIntervalMinutes)
				.RepeatForever())
			.Build();

		Scheduler.ScheduleJob(job, trigger);
		Logger.Info("Download job setup. Next fire time is {0}", GetNextFireTimeText(trigger));
	}
}

Downloading Images

Download Job

In the remote app settings there's a search section that defines the phrase(s) to search for with Bing Image Search, along with supporting data such as options and API credentials. The download job enumerates each search to be run and sets up each to be executed. TaskDelayer is based on this StackOverflow post and was added to prevent consuming too many resources at once in parallel or for too long continuously. If there is only one search to be run then it's a moot point.
internal class DownloadImagesJob : IJob
{
	//...
	public void Execute(IJobExecutionContext context)
	{
		try
		{
			if (!ShouldDownloadImages()) return;

			var outPath = AppSettings.ImagePath.FullName;
			Ensure.That(outPath, "outputPath").IsNotNullOrWhiteSpace();

			ImageCleanup.Execute();
			Logger.Info("Downloading images");
				
			if (AppSettings.Instance.Search.Queries.Count > 5)
				throw new InvalidOperationException("Please limit number of queries to 5");

			for (var q = 0; q < AppSettings.Instance.Search.Queries.Count; q++)
			{
				var search = AppSettings.Instance.Search.Queries[q];

				// try not to overwhelm system all at once, may draw too much attention
				var delaySeconds = q*AppSettings.Instance.Search.DelaySecondsBetweenSearches;
				var q1 = q;
				Logger.Info("Starting batch {0} for term {1} w/delay seconds {2}", 
					q1 + 1, search.Term, delaySeconds);

				TaskDelayer.RunDelayed(delaySeconds * 1024, () =>
				{
					var fetcher = new SearchImageFetcher(outPath);
					Logger.Info("Fetching images for {0}", search.Term);
					fetcher.Fetch(search.Term, search.Options).Wait();
					return fetcher;
				}).ContinueWith(t=> 
					Logger.Info("Finished batch {0} for term {1} w/delay seconds {2}", 
					q1 + 1, search.Term, delaySeconds));
			}
		}
		catch (Exception ex)
		{
			Logger.Error(ex.ToString());
			throw new JobExecutionException(ex);
		}
	}
	// ...
}

Searching for Images

The search image fetcher class begins by initializing the bing search container, downloaded from the .NET Framework C# Service Proxy Class Library link inside the Bing API Quick Start & Code. The Bing Data Search API allows 5,000 transactions per month for free; more than enough for my uses but keep that limit in mind.

The Fetch method takes any options specific to that search or the default if none were provided. It passes that to RunSearches to get the matching image URLs from Bing which are fed to DownloadImages for local download.
internal class SearchImageFetcher
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();
	private readonly BingSearchContainer _bingSearchContainer;
	private readonly string _outputPath;

	public SearchImageFetcher(string outputPath)
	{
		_outputPath = outputPath;
		Logger.Info("Creating Image Search client");

		_bingSearchContainer = new BingSearchContainer(
			new Uri(AppSettings.Instance.Search.ImageSearchUrl))
		{
			IgnoreMissingProperties = true,
			Timeout = AppSettings.Instance.Search.Timeout,
			Credentials = new NetworkCredential(AppSettings.Instance.Search.Username,
												AppSettings.Instance.Search.ApiKey)
		};
	}

	public async Task Fetch(string searchTerm, SearchOptions options = null)
	{
		Logger.Info("Inspecting output directory {0}", _outputPath);
		var dir = new DirectoryInfo(_outputPath);

		if (!dir.Exists) dir.Create();
		
		try
		{
			var searchOptions = options ?? AppSettings.Instance.Search.DefaultOptions;
			var results = RunSearches(searchTerm, searchOptions);

			Logger.Info("Image searches finished; {0} results", results.Count);
			await DownloadImages(results, searchTerm, searchOptions);
		}
		catch (Exception ex)
		{
			Logger.Error("Error fetching images for term '{0}' : {1}", searchTerm, 
				ex.ToString());
			throw;
		}
	}
	// ...
}
The RunSearches method invokes the bing image search method multiple times for paging, which required some customizations to BingSearchContainer.cs. I was unsure how to combine multiple filters from their documentation so I just used "Size:Large" in the settings.
private List<ImageResult> RunSearches(string searchTerm, SearchOptions options, 
int take = 50)
{
	var requests = options.Max/take;
	Logger.Info("Fetching images for term '{0}', options: {1}. Requests to make: {2}", 
		searchTerm, options, requests);
	var results = new List<ImageResult>();

	for (var i = 0; i < requests; i++)
	{
		var skip = i * take;
		Logger.Info("Setting up search. Query: {0}, Options: {1}, Skip: {2}", searchTerm, 
			options, skip);
		
		var query = _bingSearchContainer.Image(
			Query: searchTerm,
			Options: null,
			Market: null,
			Adult: options.Adult,
			Latitude: null,
			Longitude: null,
			ImageFilters: options.Filters,
			//ImageFilters:"Size:Height:768&Size:Width:1024", // how to combine multiple?
			top: 50, // 50 is the max we can request in one shot
			skip:skip);
		var currentResults = query.ToList();
		results.AddRange(currentResults);
	}

	return results;
}

Downloading Search Results

DownloadImages post-filters the bing results to exclude images smaller than the size specified in settings/options. Again it'd be better to specify that when querying Bing but I didn't see how to combine filters at first glance. Images are saved locally with the same name returned from Bing and are not downloaded again if already there. The MetadataManager associates the Bing image URL with the local filename and persists that to disk for later remote logging use when the wallpaper is changed.
private async Task DownloadImages(IEnumerable<ImageResult> results, string searchTerm, 
SearchOptions options)
{
	var sw = Stopwatch.StartNew();
	var filteredResults = results.Where(x => x.Width >= options.MinWidth 
		&& x.Height >= options.MinHeight).ToList();
	Logger.Info("Filtered result count: {0}", filteredResults.Count);
	var downloadCount = 0;

	var metadataMgr = new MetadataManager();
	foreach (var result in filteredResults)
	{
		var imageUrl = result.MediaUrl;
		var outputFilename = Path.Combine(_outputPath, string.Format("{0}.jpg", 
			result.ID.ToString("N")));

		// don't redownload image if it already exists locally, tho' it could have changed
		if (!File.Exists(outputFilename))
		{
			Logger.Debug("Image Url: {0}, destination: {1}", imageUrl, outputFilename);
			await ImageDownloader.DownloadImage(imageUrl, outputFilename);
			metadataMgr.Add(new Metadata
				{
					RemoteLocation = imageUrl,
					LocalLocation = outputFilename,
					Term = searchTerm
				});
			downloadCount++;
		}
		else 
			Logger.Debug("File already exists locally, not redownloading {0}", imageUrl);
	}

	metadataMgr.Save();
	sw.Stop();
	Logger.Info("Saved {0} images in {1:000.0} seconds", downloadCount, 
		sw.Elapsed.TotalSeconds);
}

Downloading an Image

ImageDownloader takes care of downloading a single image using WebClient; I'd probably use HttpClient if I was writing this today. I looked into throttling image downloads using SharpBits to utilize idle network bandwidth with Microsoft's Background Intelligent Transfer Service. Ultimatiely that was too much fuss and took far too long to download. I also researched utilizing this CodeProject ThrottledStream class but ultimiately I decided not to throttle downloads, mostly out of laziness.
class ImageDownloader
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();

	public static async Task DownloadImage(string url, string outputFilename)
	{
		using (var webClient = new WebClient())
		{
			try
			{
				Logger.Debug("Downloading {0} to {1}", url, outputFilename);
				var sw = Stopwatch.StartNew();
				var imageBytes = await webClient.DownloadDataTaskAsync(url);
				sw.Stop();
				Logger.Debug("Downloaded {0} bytes in {1:00.0} second(s)", 
					imageBytes.Length, sw.Elapsed.TotalSeconds);

				Logger.Debug("Writing image bytes to disk");
				var result = ImageWriter.Write(imageBytes, outputFilename);
				Logger.Debug("Image write complete with result: {0}", result);
			}
			catch (Exception ex)
			{
				Logger.Error("Error downloading image '{0}' to '{1}'. Likely corrupt "
					+ "and will be deleted. Error: {2}", url, outputFilename, ex.ToString());
				try
				{
					if (File.Exists(outputFilename))
						File.Delete(outputFilename);
				}
				catch (Exception inner)
				{
					Logger.Error(string.Format("Error deleting image '{0}': {1}", 
						outputFilename, inner));
				}
			}
		}
	}
}

Writing Image Bytes to Disk

I found that Image.Save() threw an exception for some large images so I attempted that first and if that failed I wrote the image bytes to disk in chunks using a Stream.
internal class ImageWriter
{
	private static readonly IAppLogger Logger = LoggerFactory.Create();

	public static bool Write(byte[] imageBytes, string outputFilename)
	{
		Logger.Debug("Creating memory stream from byte array of length {0}", 
			imageBytes.Length);
		using (var stream = new MemoryStream(imageBytes))
		{
			Logger.Debug("Creating image from stream");
			using (var image = Image.FromStream(stream))
			{
				Logger.Debug("Saving image {0} in Jpeg format", outputFilename);

				try
				{
					image.Save(outputFilename, ImageFormat.Jpeg);
					Logger.Info("Saved image {0}", outputFilename);
				}
				catch (Exception ex)
				{
					Logger.Error("Error saving {0}. Will attempt saving in chunks. Error was : {1}",
						outputFilename, ex.ToString());

					try
					{
						SaveImageInChunks(imageBytes, outputFilename);
					}
					catch (Exception inner)
					{
						Logger.Error("Saving image in chunks failed: {0}", inner);
					}
				}
			}
		}
		
		return true;
	}

	private static void SaveImageInChunks(byte[] imageBytes, string outputFilename)
	{
		// this is to handle a large image where Image.Save croaked
		using (Stream source = new MemoryStream(imageBytes))
		using (Stream dest = File.Create(outputFilename))
		{
			var buffer = new byte[1024];
			int bytes;
			while ((bytes = source.Read(buffer, 0, buffer.Length)) > 0)
			{
				dest.Write(buffer, 0, bytes);
			}
		}
	}
}

Part 2

Part 2 - remote app settings, setting the wallpaper, remote logging, source code and more...
Friday
May242013

ASP.NET NLog Sql Server Logging and Error Handling Part 2

Be sure to check out Part 1 as this post builds upon it and the two go hand in hand.

Series Overview

Part 1 - Setting up logging with ASP.NET MVC, NLog and SQL Server

Part 2 - Unhandled exception processing, building an error report, emailing errors, and custom error pages.

Custom Error Handling Attribute

Added in FilterConfig.RegisterGlobalFilters and bound in DiagnosticModule, AppErrorHandlerAttribute invokes reporting the unhandled exception and setting the error view to be displayed to the end user. An enableErrorPages appSetting controls whether any of this is done; for local debugging or a dev web server having this off might be desirable.
namespace NLogSql.Web.Infrastructure.ErrorHandling
{
    public class AppErrorHandlerAttribute : FilterAttribute, IExceptionFilter
    {
        [Inject]
        public IErrorReporter Reporter { get; set; }

        public void OnException(ExceptionContext exceptionContext)
        {
            if (exceptionContext.ExceptionHandled) return;

            if (ConfigurationManager.AppSettings["enableErrorPages"] == "false")
            {
                AppLogFactory.Create<AppErrorHandlerAttribute>().Error(
                    "Unexpected error. enableErrorPages is false, skipping detailed "
					+ "error gathering. Error was: {0}",
                    exceptionContext.Exception.ToString());
                return;
            }

            Ensure.That(Reporter, "Reporter").IsNotNull();
            Reporter.ReportException(exceptionContext);

            SetErrorViewResult(exceptionContext);
        }

        private static void SetErrorViewResult(ExceptionContext exceptionContext)
        {
            var statusCode = new HttpException(null, exceptionContext.Exception)
				.GetHttpCode();

            exceptionContext.Result = new ViewResult
            {
                ViewName = MVC.Shared.Views.ViewNames.Error,
                TempData = exceptionContext.Controller.TempData,
                //ViewData = new ViewDataDictionary<ErrorModel>(new ErrorModel())
            };

            exceptionContext.ExceptionHandled = true;
            exceptionContext.HttpContext.Response.Clear();
            exceptionContext.HttpContext.Response.StatusCode = statusCode;
            exceptionContext.HttpContext.Response.TrySkipIisCustomErrors = true;
        }
    }
}

Logging and Reporting the Error

In the same namespace the ErrorReporter class invokes generation of an error report and logs and emails the error report. The overload with customActivityMessage would generally be used with handled exceptions where it may still be desirable to report the exception in cases.
public class ErrorReporter : IErrorReporter
{
	private readonly ILog _log;

	public ErrorReporter(ILog log)
	{
		_log = Ensure.That(log, "log").IsNotNull().Value;
	}

	private string CustomActivityMessage { get; set; }

	public void ReportException(ControllerContext controllerContext, 
		Exception exception, string customActivityMessage = null)
	{
		this.CustomActivityMessage = customActivityMessage;
		ReportException(new ExceptionContext(controllerContext, exception));
	}

	public void ReportException(ExceptionContext exceptionContext)
	{
		var errorInfo = new ErrorReportInfo(exceptionContext, this.CustomActivityMessage);
		errorInfo.Generate();
		_log.Error("Unexpected error: {0}", errorInfo.ReportText);

		if (errorInfo.Errors.Any())
			_log.Error("Error generating error report. Original exception: {0}", 
				exceptionContext.Exception);

		// sending mail can be a little slow, don't delay end user seeing error page
		Task.Factory.StartNew(
		state =>
		{
			var errorReport = (ErrorReportInfo)state;
			DependencyResolver.Current.GetService<IErrorEmailer>()
				.SendErrorEmail(errorReport);
		},
		errorInfo).ContinueWith(t =>
		{
			if (null != t.Exception)
				_log.Error("Error sending email: " + t.Exception.ToString());    
		});
	}
}

Building the Error Report

Various diagnostic info classes are responsible for building different diagnostic "sub reports". Each inherits from DiagnosticInfoBase which is a glorified StringBuilder, with functionality to build both a plain text version of the report (logged to DB) as well as an HTML formatted version (used for emails).

The base class has safe appending and formatting functionality and ensures that any error in generating a part of the report doesn't stop the whole process.
A sample implementation:
public class FormInfo : DiagnosticInfoBase
{
	private readonly HttpRequestBase _request;

	public FormInfo(HttpRequestBase request)
	{
		_request = request;
	}

	protected override void GenerateReport()
	{
		StartTable();
		var keys = _request.Form.AllKeys.OrderBy(s => s).ToList();

		foreach (var name in keys)
		{
			var value = _request.Form[name];

			if (null != value && name.Contains("password", StringComparison.OrdinalIgnoreCase))
			{
				value = new string('*', value.Length);
			}

			AppendRow(name, value);
		}

		EndTable();
	}
}
The ErrorReportInfo class combines the output of each section for the full error report.



In the end this produces a sample HTML email report like this.

Sending the Email

Sending the email logs the result on failure or success,
public interface IErrorEmailer
{
	void SendErrorEmail(ErrorReportInfo errorInfo);
}

public class ErrorEmailer : IErrorEmailer
{
	private readonly IMailer _mailer;
	private readonly ILog _log;

	public ErrorEmailer(IMailer mailer, ILog log)
	{
		_mailer = Ensure.That(mailer, "mailer").IsNotNull().Value;
		_log = Ensure.That(log, "log").IsNotNull().Value;
	}

	public void SendErrorEmail(ErrorReportInfo errorInfo)
	{
		try
		{
			var subject = string.Format("{0} Error", 
				Assembly.GetExecutingAssembly().GetName().Name);

			if (null != errorInfo.Server && null != errorInfo.Location
				&& !string.IsNullOrWhiteSpace(errorInfo.Location.ControllerAction)
				&& !string.IsNullOrWhiteSpace(errorInfo.Server.HostName))
			{
				subject = string.Format("{0}: {1} - {2}", subject, 
					errorInfo.Server.HostName, errorInfo.Location.ControllerAction);
			}

			var to = AppSettings.Default.Email.ErrorMessageTo;
			_mailer.SendMail(to, subject, errorInfo.ReportHtml);

			_log.Info("Sent email: {0} to {1}", subject, to);
		}
		catch (Exception ex)
		{
			_log.Error("Error sending error report email: {0}", ex);
		}
	}
}

The Error View

At the end of AppErrorHandlerAttribute the SetErrorViewResult invoked the Views\Shared\Error.cshtml page. According to the HTTP status code, content such as text, images and styles differ. Separate error pages may have more advantages if there are a larger number of differences; a single page was less work here.
@section styles{
    <link rel="stylesheet" href="~/Content/Error.css"/>
}

@{
    var statusTitleMap = new Dictionary<int, string> {
             {404, "Something got lost in the shuffle"},
             {410, "Gone like yesterday"},
             {500, "Something go boom"}
         };
}

@section hero{
    <div id="error-body" class="container-fluid">
        <div class="row-fluid">
            <div class="span3">
                @{ var errorClass = (Response.StatusCode == 404 || Response.StatusCode == 410) ? "_" + Response.StatusCode : "_500";}
                <div id="errorImageBlock" class="@errorClass"></div>
            </div>
    
            <div class="span8 offset1">
                <div class="row-fluid">
                    <div class="span12 text-center" id="status-code">
                        @Response.StatusCode
                    </div>
                </div>
                <div class="row-fluid">
                    <div class="span12" id="well-this">
                        @statusTitleMap[Response.StatusCode]
                    </div>
                </div>
        
                <div class="row-fluid">
                    <div class="span11" id="message">
                    @switch (Response.StatusCode)
                    {
                        case 404:
                            @:We cannot find the page you are looking for. If you typed in the address, double check the spelling. If you got here by clicking a link, 
                            @: <a href="mailto:customerservice@domain.com?subject=Page%20Not%20Found">let us know</a>.
                            break;
                        case 410:
                            @:The page you are looking for is gone (permanently). If you feel you reached this page incorrectly, <a href="mailto:customerservice@domain.com?subject=Link%20Gone">let us know</a>.
                            break;
                        case 500:
                            @:Oh dear, something's gone wrong. Our team has already been alerted to the problem and will fix it as soon as possible! 
                            break;
                    }  
                    </div>
                </div>
            </div>
        </div>
    </div>
}

Testing Errors

At the bottom of the home page are some links to test out error functionality.





Handling Not Found and Gone Permanently

404s have some special handling in Global.asax.cs:
protected void Application_EndRequest()
{
	if (Context.Response.StatusCode == 404)
	{
		if (Request.RequestContext.RouteData.Values["fake404"] == null)
		{
			Response.Clear();

			var rd = new RouteData();
			rd.Values["controller"] = MVC.Error.Name;
			rd.Values["action"] = MVC.Error.ActionNames.NotFound;

			var c = (IController)DependencyResolver.Current.GetService<ErrorController>();
			Request.RequestContext.RouteData = rd;
			c.Execute(new RequestContext(new HttpContextWrapper(Context), rd));
		}
	}
}
That code makes me itch a bit and could be done better but there is a reason for it. If you're wondering why not just use custom error pages, the answer is that doing so for a 404 page ends up producing a 301 redirect to then 404 on the custom not found page. For SEO purposes that is usually considered a bad practice. If you don't have a public site or as much SEO concern, that may be acceptable but in this case it wasn't.

Error Controller

The error controller sets and logs the response status codes and returns the error view. For the Gone action, usually there'd be routes defined for legacy URLs that would direct to that action.
public partial class ErrorController : Controller
{
	private readonly ILog _log;

	public ErrorController(ILog log)
	{
		_log = Ensure.That(log, "log").IsNotNull().Value;
	}

	public virtual ActionResult NotFound()
	{
		Response.StatusCode = (int)HttpStatusCode.NotFound;
		RouteData.Values["fake404"] = true;
		_log.Write(LogType.Warn, new { Code = "404" }, 
			"404 Not Found for {0}", Request.Url);
		return View("Error");
	}

	public virtual ActionResult Gone()
	{
		Response.StatusCode = (int)HttpStatusCode.Gone;
		Response.Status = "410 Gone";
		Response.TrySkipIisCustomErrors = true;
		_log.Write(LogType.Warn, new {Code = "410"}, 
			"410 gone permanently for {0}", Request.Url);
		return View("Error");
	}
}

Future Enhancements

  • The diagnostic info report building classes are pretty quick and messy in spots and could use cleanup.
  • Error view work including better responsive design and user-acceptable images :)
  • Admin error view to inspect log data and errors
  • Use and integration of apps such as appfail.net and New Relic to monitor errors and performance. This app used those tools in conjunction with the custom error handling and logging functionality.

The Code

The code for this series is available at https://github.com/thnk2wn/NLogSql.Web